├── .gitignore ├── LICENSE ├── README.md ├── analysis ├── analysis.Rproj └── keras-cntk-benchmark.Rmd ├── keras_cntk_benchmark.py ├── logs ├── cifar10_cnn_cntk.csv ├── cifar10_cnn_tensorflow.csv ├── imdb_bidirectional_lstm_cntk.csv ├── imdb_bidirectional_lstm_tensorflow.csv ├── imdb_fasttext_cntk.csv ├── imdb_fasttext_tensorflow.csv ├── lstm_text_generation_cntk.csv ├── lstm_text_generation_tensorflow.csv ├── mnist_cnn_cntk.csv ├── mnist_cnn_tensorflow.csv ├── mnist_mlp_cntk.csv ├── mnist_mlp_tensorflow.csv ├── text_generator_keras_cntk.csv └── text_generator_keras_tensorflow.csv ├── test_files ├── CustomCallback.py ├── cifar10_cnn.py ├── imdb_bidirectional_lstm.py ├── imdb_fasttext.py ├── lstm_text_generation.py ├── mnist_cnn.py ├── mnist_mlp.py └── text_generator_keras.py └── v2 ├── analysis ├── .Rhistory ├── dl-cpu-gpu-1.png ├── dl-cpu-gpu-10.png ├── dl-cpu-gpu-11.png ├── dl-cpu-gpu-12.png ├── dl-cpu-gpu-2.png ├── dl-cpu-gpu-3.png ├── dl-cpu-gpu-4.png ├── dl-cpu-gpu-5.png ├── dl-cpu-gpu-6.png ├── dl-cpu-gpu-7.png ├── dl-cpu-gpu-8.png ├── dl-cpu-gpu-9.png └── gpu_benchmark_v2.Rmd ├── keras_cntk_benchmark.py ├── keras_cntk_benchmark_cpu.py ├── logs ├── cpu16 │ ├── cifar10_cnn_tensorflow.csv │ ├── imdb_bidirectional_lstm_tensorflow.csv │ ├── imdb_fasttext_tensorflow.csv │ ├── lstm_text_generation_tensorflow.csv │ ├── mnist_cnn_tensorflow.csv │ └── mnist_mlp_tensorflow.csv ├── cpu32 │ ├── cifar10_cnn_tensorflow.csv │ ├── imdb_bidirectional_lstm_tensorflow.csv │ ├── imdb_fasttext_tensorflow.csv │ ├── lstm_text_generation_tensorflow.csv │ ├── mnist_cnn_tensorflow.csv │ └── mnist_mlp_tensorflow.csv ├── k80 │ ├── cifar10_cnn_cntk.csv │ ├── cifar10_cnn_tensorflow.csv │ ├── imdb_bidirectional_lstm_cntk.csv │ ├── imdb_bidirectional_lstm_tensorflow.csv │ ├── imdb_fasttext_cntk.csv │ ├── imdb_fasttext_tensorflow.csv │ ├── lstm_text_generation_cntk.csv │ ├── lstm_text_generation_tensorflow.csv │ ├── mnist_cnn_cntk.csv │ ├── mnist_cnn_tensorflow.csv │ ├── mnist_mlp_cntk.csv │ ├── mnist_mlp_tensorflow.csv │ └── text_generator_keras_tensorflow.csv └── p100 │ ├── cifar10_cnn_cntk.csv │ ├── cifar10_cnn_tensorflow.csv │ ├── imdb_bidirectional_lstm_cntk.csv │ ├── imdb_bidirectional_lstm_tensorflow.csv │ ├── imdb_fasttext_cntk.csv │ ├── imdb_fasttext_tensorflow.csv │ ├── lstm_text_generation_cntk.csv │ ├── lstm_text_generation_tensorflow.csv │ ├── mnist_cnn_cntk.csv │ ├── mnist_cnn_tensorflow.csv │ ├── mnist_mlp_cntk.csv │ ├── mnist_mlp_tensorflow.csv │ ├── text_generator_keras_cntk.csv │ └── text_generator_keras_tensorflow.csv └── test_files ├── CustomCallback.py ├── cifar10_cnn.py ├── imdb_bidirectional_lstm.py ├── imdb_fasttext.py ├── lstm_text_generation.py ├── mnist_cnn.py └── mnist_mlp.py /.gitignore: -------------------------------------------------------------------------------- 1 | __pycache__ 2 | .Rproj.user 3 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2017 Max Woolf 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # keras-cntk-benchmark 2 | 3 | Repository to benchmark the performance of the CNTK backend on Keras vs. the performance of TensorFlow. This R Notebook is the complement to my blog post [Benchmarking CNTK on Keras: is it Better at Deep Learning than TensorFlow?](http://minimaxir.com/2017/06/keras-cntk/). 4 | 5 | ## Usage 6 | 7 | The repository uses my [keras-cntk-docker](https://github.com/minimaxir/keras-cntk-docker) container, and assumes necessary dependices for that are installed. 8 | 9 | `keras_cntk_benchmark.py` contains the benchmark script. 10 | 11 | `/test_files` contains the test files. 12 | 13 | `/logs` contains the performance output (CSV) for each test. 14 | 15 | `/analysis` contains the R Notebook of the logs used to create the interactive data visualizations in the blog post. (You can view the R Notebook on the Web here) 16 | 17 | ## Maintainer 18 | 19 | Max Woolf ([@minimaxir](http://minimaxir.com)) 20 | 21 | *Max's open-source projects are supported by his [Patreon](https://www.patreon.com/minimaxir). If you found this project helpful, any monetary contributions to the Patreon are appreciated and will be put to good creative use.* 22 | 23 | ## Credits 24 | 25 | Test scripts (aside from `text_generator_keras.py`) sourced from the official Keras repository/contributors. 26 | 27 | ## License 28 | 29 | MIT -------------------------------------------------------------------------------- /analysis/analysis.Rproj: -------------------------------------------------------------------------------- 1 | Version: 1.0 2 | 3 | RestoreWorkspace: Default 4 | SaveWorkspace: Default 5 | AlwaysSaveHistory: Default 6 | 7 | EnableCodeIndexing: Yes 8 | UseSpacesForTab: Yes 9 | NumSpacesForTab: 2 10 | Encoding: UTF-8 11 | 12 | RnwWeave: Sweave 13 | LaTeX: pdfLaTeX 14 | -------------------------------------------------------------------------------- /analysis/keras-cntk-benchmark.Rmd: -------------------------------------------------------------------------------- 1 | --- 2 | title: "Benchmarking CNTK on Keras: is it Better at Deep Learning than TensorFlow?" 3 | author: "Max Woolf (@minimaxir)" 4 | date: "2017-06-12" 5 | output: 6 | html_notebook: 7 | highlight: tango 8 | mathjax: null 9 | number_sections: yes 10 | theme: spacelab 11 | toc: yes 12 | toc_float: yes 13 | --- 14 | 15 | This R Notebook is the complement to my blog post [Benchmarking CNTK on Keras: is it Better at Deep Learning than TensorFlow?](http://minimaxir.com/2017/06/keras-cntk/). 16 | 17 | This notebook is licensed under the MIT License. If you use the code or data visualization designs contained within this notebook, it would be greatly appreciated if proper attribution is given back to this notebook and/or myself. Thanks! :) 18 | 19 | ```{r} 20 | library(readr) 21 | library(dplyr) 22 | library(ggplot2) 23 | library(plotly) 24 | library(boot) 25 | library(scales) 26 | library(htmlwidgets) 27 | 28 | sessionInfo() 29 | ``` 30 | 31 | ```{r} 32 | file_path <- "/Users/maxwoolf/Dropbox/PythonProjects/keras-cntk-benchmark/logs/" 33 | ``` 34 | 35 | CNTK is blue, TensorFlow is red. 36 | 37 | ```{r} 38 | framework_colors <- c(CNTK="#2980b9",TensorFlow="#c0392b") 39 | ``` 40 | 41 | # IMDb 42 | 43 | ## imdb_bidirectional_lstm 44 | 45 | ```{r} 46 | df_imdb_bi_cntk <- read_csv(paste0(file_path, "imdb_bidirectional_lstm_cntk.csv")) 47 | df_imdb_bi_tf <- read_csv(paste0(file_path, "imdb_bidirectional_lstm_tensorflow.csv")) 48 | ``` 49 | 50 | Test the bootstrapping functionality via `boot`. 51 | 52 | ```{r} 53 | elapsed_mean <- function(data, indices){ 54 | d <- data[indices,] 55 | mean(d$elapsed) 56 | } 57 | 58 | (df_imdb_bi_cntk %>% boot(elapsed_mean, 1000) %>% boot.ci(type="bca"))$bca[4:5] 59 | ``` 60 | 61 | ### Test Acc 62 | 63 | ```{r} 64 | df <- df_imdb_bi_cntk %>% mutate(framework = 'CNTK', epoch = epoch + 1) %>% union( 65 | df_imdb_bi_tf %>% mutate(framework = 'TensorFlow', epoch = epoch + 1) 66 | ) 67 | 68 | df 69 | ``` 70 | 71 | 72 | ```{r} 73 | plot <- ggplot(df, aes(epoch, val_acc, color=framework)) + 74 | geom_line() + 75 | geom_point(size=2) + 76 | theme_minimal(base_family="Source Sans Pro", base_size=14) + 77 | scale_y_continuous(labels = percent) + 78 | scale_color_manual(values=framework_colors) + 79 | labs(title = "Performance of Bidirectional LSTM Approach on IMDb Data", 80 | x = "Epoch", 81 | y = "Test Accuracy (Higher is Better)", 82 | color = "") 83 | 84 | plot 85 | plot %>% ggplotly(width="100%", height=400) 86 | plot %>% ggplotly(width="100%", height=400) %>% 87 | saveWidget("keras-1.html", selfcontained=F, libdir="plotly") 88 | ``` 89 | 90 | ### Speed 91 | 92 | ```{r} 93 | df_speed <- df %>% group_by(framework) %>% 94 | do({ 95 | boot <- boot.ci(boot(., elapsed_mean, 1000), type="bca")$bca 96 | data.frame( 97 | mean = mean(.$elapsed), 98 | low_ci = boot[4], 99 | high_ci = boot[5] 100 | ) 101 | }) %>% 102 | ungroup() %>% 103 | mutate(framework = factor(framework)) 104 | 105 | df_speed 106 | ``` 107 | 108 | 109 | ```{r} 110 | plot <- ggplot(df_speed, aes(x=framework, y=mean, ymin=low_ci, ymax=high_ci, fill=framework)) + 111 | geom_bar(stat="identity", width=0.25) + 112 | geom_errorbar(width = 0.25) + 113 | theme_minimal(base_family="Source Sans Pro", base_size=14) + 114 | scale_fill_manual(values=framework_colors) + 115 | labs(title = "Speed of Bidirectional LSTM Approach on IMDb Data", 116 | x = "Keras Backend", 117 | y = "Average Epoch Runtime (seconds)", 118 | fill = "") 119 | 120 | plot 121 | plot %>% ggplotly(tooltip=c("x", "y"), width="100%") 122 | plot %>% ggplotly(tooltip=c("x", "y"), width="100%", height=400) %>% 123 | saveWidget("keras-2.html", selfcontained=F, libdir="plotly") 124 | ``` 125 | 126 | ```{r} 127 | speed_imdb_bi = (df_speed$mean[2] / df_speed$mean[1]) %>% round(2) 128 | ``` 129 | 130 | 131 | CNTK runs at **`r speed_imdb_bi`**x the runtime of TensorFlow. 132 | 133 | ## FastText 134 | 135 | 136 | ```{r} 137 | df_imdb_ft_cntk <- read_csv(paste0(file_path, "imdb_fasttext_cntk.csv")) 138 | df_imdb_ft_tf <- read_csv(paste0(file_path, "imdb_fasttext_tensorflow.csv")) 139 | ``` 140 | 141 | ### Test Acc 142 | 143 | ```{r} 144 | df <- df_imdb_ft_cntk %>% mutate(framework = 'CNTK', epoch = epoch + 1) %>% union( 145 | df_imdb_ft_tf %>% mutate(framework = 'TensorFlow', epoch = epoch + 1) 146 | ) 147 | 148 | df 149 | ``` 150 | 151 | 152 | ```{r} 153 | plot <- ggplot(df, aes(epoch, val_acc, color=framework)) + 154 | geom_line() + 155 | geom_point(size=2) + 156 | theme_minimal(base_family="Source Sans Pro", base_size=14) + 157 | scale_y_continuous(labels = percent) + 158 | scale_color_manual(values=framework_colors) + 159 | labs(title = "Performance of fasttext Approach on IMDb Data", 160 | x = "Epoch", 161 | y = "Test Accuracy (Higher is Better)", 162 | color= "") 163 | 164 | plot %>% ggplotly() 165 | plot %>% ggplotly(width="100%", height=400) %>% 166 | saveWidget("keras-3.html", selfcontained=F, libdir="plotly") 167 | ``` 168 | 169 | ### Speed 170 | 171 | ```{r} 172 | df_speed <- df %>% group_by(framework) %>% 173 | do({ 174 | boot <- boot.ci(boot(., elapsed_mean, 1000), type="bca")$bca 175 | data.frame( 176 | mean = mean(.$elapsed), 177 | low_ci = boot[4], 178 | high_ci = boot[5] 179 | ) 180 | }) %>% 181 | ungroup() %>% 182 | mutate(framework = factor(framework)) 183 | 184 | df_speed 185 | ``` 186 | 187 | 188 | ```{r} 189 | plot <- ggplot(df_speed, aes(x=framework, y=mean, ymin=low_ci, ymax=high_ci, fill=framework)) + 190 | geom_bar(stat="identity", width=0.5) + 191 | geom_errorbar(width = 0.25) + 192 | theme_minimal(base_family="Source Sans Pro", base_size=14) + 193 | scale_fill_manual(values=framework_colors) + 194 | labs(title = "Speed of fasttext Approach on IMDb Data", 195 | x = "Keras Backend", 196 | y = "Average Epoch Runtime (seconds)", 197 | fill="") 198 | 199 | plot %>% ggplotly(tooltip=c("x", "y")) 200 | plot %>% ggplotly(tooltip=c("x", "y"), width="100%", height=400) %>% 201 | saveWidget("keras-4.html", selfcontained=F, libdir="plotly") 202 | ``` 203 | 204 | ```{r} 205 | speed_imdb_fasttext = (df_speed$mean[2] / df_speed$mean[1]) %>% round(2) 206 | ``` 207 | 208 | 209 | CNTK runs at **`r speed_imdb_fasttext`**x the runtime of TensorFlow. 210 | 211 | # MNIST 212 | 213 | ## MLP 214 | 215 | ```{r} 216 | df_mnist_mlp_cntk <- read_csv(paste0(file_path, "mnist_mlp_cntk.csv")) 217 | df_mnist_mlp_tf <- read_csv(paste0(file_path, "mnist_mlp_tensorflow.csv")) 218 | ``` 219 | 220 | ### Test Acc 221 | 222 | ```{r} 223 | df <- df_mnist_mlp_cntk %>% mutate(framework = 'CNTK', epoch = epoch + 1) %>% union( 224 | df_mnist_mlp_tf %>% mutate(framework = 'TensorFlow', epoch = epoch + 1) 225 | ) 226 | 227 | df 228 | ``` 229 | 230 | 231 | ```{r} 232 | plot <- ggplot(df, aes(epoch, val_acc, color=framework)) + 233 | geom_line() + 234 | geom_point(size=2) + 235 | theme_minimal(base_family="Source Sans Pro", base_size=14) + 236 | scale_color_manual(values=framework_colors) + 237 | scale_y_continuous(labels = percent) + 238 | labs(title = "Performance of MLP Approach on MNIST Data", 239 | x = "Epoch", 240 | y = "Test Accuracy (Higher is Better)", 241 | color="") 242 | 243 | plot %>% ggplotly() 244 | plot %>% ggplotly(width="100%", height=400) %>% 245 | saveWidget("keras-5.html", selfcontained=F, libdir="plotly") 246 | ``` 247 | 248 | ### Speed 249 | 250 | ```{r} 251 | df_speed <- df %>% group_by(framework) %>% 252 | do({ 253 | boot <- boot.ci(boot(., elapsed_mean, 1000), type="bca")$bca 254 | data.frame( 255 | mean = mean(.$elapsed), 256 | low_ci = boot[4], 257 | high_ci = boot[5] 258 | ) 259 | }) %>% 260 | ungroup() %>% 261 | mutate(framework = factor(framework)) 262 | 263 | df_speed 264 | ``` 265 | 266 | 267 | ```{r} 268 | plot <- ggplot(df_speed, aes(x=framework, y=mean, ymin=low_ci, ymax=high_ci, fill=framework)) + 269 | geom_bar(stat="identity", width=0.5) + 270 | geom_errorbar(width = 0.25) + 271 | theme_minimal(base_family="Source Sans Pro", base_size=14) + 272 | scale_fill_manual(values=framework_colors) + 273 | labs(title = "Speed of MLP Approach on MNIST Data", 274 | x = "Keras Backend", 275 | y = "Average Epoch Runtime (seconds)", 276 | fill="") 277 | 278 | plot %>% ggplotly(tooltip=c("x", "y")) 279 | plot %>% ggplotly(tooltip=c("x", "y"), width="100%", height=400) %>% 280 | saveWidget("keras-6.html", selfcontained=F, libdir="plotly") 281 | ``` 282 | 283 | ```{r} 284 | speed_mnist_mlp = (df_speed$mean[2] / df_speed$mean[1]) %>% round(2) 285 | ``` 286 | 287 | CNTK runs at **`r speed_mnist_mlp`**x the runtime of TensorFlow. 288 | 289 | ## CNN 290 | 291 | 292 | ```{r} 293 | df_mnist_cnn_cntk <- read_csv(paste0(file_path, "mnist_cnn_cntk.csv")) 294 | df_mnist_cnn_tf <- read_csv(paste0(file_path, "mnist_cnn_tensorflow.csv")) 295 | ``` 296 | 297 | ### Test Acc 298 | 299 | ```{r} 300 | df <- df_mnist_cnn_cntk %>% mutate(framework = 'CNTK', epoch = epoch + 1) %>% union( 301 | df_mnist_cnn_tf %>% mutate(framework = 'TensorFlow', epoch = epoch + 1) 302 | ) 303 | 304 | df 305 | ``` 306 | 307 | 308 | ```{r} 309 | plot <- ggplot(df, aes(epoch, val_acc, color=framework)) + 310 | geom_line() + 311 | geom_point(size=2) + 312 | theme_minimal(base_family="Source Sans Pro", base_size=14) + 313 | scale_y_continuous(labels = percent) + 314 | scale_color_manual(values=framework_colors) + 315 | labs(title = "Performance of CNN Approach on MNIST Data", 316 | x = "Epoch", 317 | y = "Test Accuracy (Higher is Better)", 318 | color="") 319 | 320 | plot %>% ggplotly() 321 | plot %>% ggplotly(width="100%", height=400) %>% 322 | saveWidget("keras-7.html", selfcontained=F, libdir="plotly") 323 | ``` 324 | 325 | ### Speed 326 | 327 | ```{r} 328 | df_speed <- df %>% group_by(framework) %>% 329 | do({ 330 | boot <- boot.ci(boot(., elapsed_mean, 1000), type="bca")$bca 331 | data.frame( 332 | mean = mean(.$elapsed), 333 | low_ci = boot[4], 334 | high_ci = boot[5] 335 | ) 336 | }) %>% 337 | ungroup() %>% 338 | mutate(framework = factor(framework)) 339 | 340 | df_speed 341 | ``` 342 | 343 | 344 | ```{r} 345 | plot <- ggplot(df_speed, aes(x=framework, y=mean, ymin=low_ci, ymax=high_ci, fill=framework)) + 346 | geom_bar(stat="identity", width=0.5) + 347 | geom_errorbar(width = 0.25) + 348 | theme_minimal(base_family="Source Sans Pro", base_size=14) + 349 | scale_fill_manual(values=framework_colors) + 350 | labs(title = "Speed of CNN Approach on MNIST Data", 351 | x = "Keras Backend", 352 | y = "Average Epoch Runtime (seconds)", 353 | fill = "") 354 | 355 | plot %>% ggplotly(tooltip=c("x", "y")) 356 | plot %>% ggplotly(tooltip=c("x", "y"), width="100%", height=400) %>% 357 | saveWidget("keras-8.html", selfcontained=F, libdir="plotly") 358 | ``` 359 | 360 | 361 | ```{r} 362 | speed_mnist_cnn = (df_speed$mean[2] / df_speed$mean[1]) %>% round(2) 363 | ``` 364 | 365 | CNTK runs at **`r speed_mnist_cnn`**x the runtime of TensorFlow. 366 | 367 | # CIFAR: 368 | 369 | ## CNN 370 | 371 | 372 | ```{r} 373 | df_cifar_cnn_cntk <- read_csv(paste0(file_path, "cifar10_cnn_cntk.csv")) 374 | df_cifar_cnn_tf <- read_csv(paste0(file_path, "cifar10_cnn_tensorflow.csv")) 375 | ``` 376 | 377 | ### Test Acc 378 | 379 | ```{r} 380 | df <- df_cifar_cnn_cntk %>% mutate(framework = 'CNTK', epoch = epoch + 1) %>% union( 381 | df_cifar_cnn_tf %>% mutate(framework = 'TensorFlow', epoch = epoch + 1) 382 | ) 383 | 384 | df 385 | ``` 386 | 387 | 388 | ```{r} 389 | plot <- ggplot(df, aes(epoch, val_acc, color=framework)) + 390 | geom_line() + 391 | geom_point(size=2) + 392 | theme_minimal(base_family="Source Sans Pro", base_size=14) + 393 | scale_y_continuous(labels = percent) + 394 | scale_color_manual(values=framework_colors) + 395 | labs(title = "Performance of CNN Approach on CIFAR-10 Data", 396 | x = "Epoch", 397 | y = "Test Accuracy (Higher is Better)", 398 | color="") 399 | 400 | plot %>% ggplotly() 401 | plot %>% ggplotly(width="100%", height=400) %>% 402 | saveWidget("keras-9.html", selfcontained=F, libdir="plotly") 403 | ``` 404 | 405 | ### Speed 406 | 407 | ```{r} 408 | df_speed <- df %>% group_by(framework) %>% 409 | do({ 410 | boot <- boot.ci(boot(., elapsed_mean, 1000), type="bca")$bca 411 | data.frame( 412 | mean = mean(.$elapsed), 413 | low_ci = boot[4], 414 | high_ci = boot[5] 415 | ) 416 | }) %>% 417 | ungroup() %>% 418 | mutate(framework = factor(framework)) 419 | 420 | df_speed 421 | ``` 422 | 423 | 424 | ```{r} 425 | plot <- ggplot(df_speed, aes(x=framework, y=mean, ymin=low_ci, ymax=high_ci, fill=framework)) + 426 | geom_bar(stat="identity", width=0.5) + 427 | geom_errorbar(width = 0.25) + 428 | theme_minimal(base_family="Source Sans Pro", base_size=14) + 429 | scale_fill_manual(values=framework_colors) + 430 | labs(title = "Speed of CNN Approach on CIFAR-10 Data", 431 | x = "Keras Backend", 432 | y = "Average Epoch Runtime (seconds)", 433 | fill = "") 434 | 435 | plot %>% ggplotly(tooltip=c("x", "y")) 436 | plot %>% ggplotly(tooltip=c("x", "y"), width="100%", height=400) %>% 437 | saveWidget("keras-10.html", selfcontained=F, libdir="plotly") 438 | ``` 439 | 440 | ```{r} 441 | speed_cifar_cnn = (df_speed$mean[2] / df_speed$mean[1]) %>% round(2) 442 | ``` 443 | 444 | CNTK runs at **`r speed_cifar_cnn`**x the runtime of TensorFlow.; not statistically significant 445 | 446 | # Text Generation 447 | 448 | ## CNN 449 | 450 | 451 | ```{r} 452 | df_gen_lstm_cntk <- read_csv(paste0(file_path, "lstm_text_generation_cntk.csv")) 453 | df_gen_lstm_tf <- read_csv(paste0(file_path, "lstm_text_generation_tensorflow.csv")) 454 | ``` 455 | 456 | ### Test Acc 457 | 458 | ```{r} 459 | df <- df_gen_lstm_cntk %>% mutate(framework = 'CNTK', epoch = epoch + 1) %>% union( 460 | df_gen_lstm_tf %>% mutate(framework = 'TensorFlow', epoch = epoch + 1) 461 | ) 462 | 463 | df 464 | ``` 465 | 466 | 467 | ```{r} 468 | plot <- ggplot(df, aes(epoch, loss, color=framework)) + 469 | geom_line() + 470 | geom_point(size=2) + 471 | theme_minimal(base_family="Source Sans Pro", base_size=14) + 472 | scale_color_manual(values=framework_colors) + 473 | labs(title = "Performance of Text Generation via LSTM", 474 | x = "Epoch", 475 | y = "Loss (Lower is Better)", 476 | color="") 477 | 478 | plot %>% ggplotly() 479 | plot %>% ggplotly(width="100%", height=400) %>% 480 | saveWidget("keras-11.html", selfcontained=F, libdir="plotly") 481 | ``` 482 | 483 | ### Speed 484 | 485 | ```{r} 486 | df_speed <- df %>% group_by(framework) %>% 487 | do({ 488 | boot <- boot.ci(boot(., elapsed_mean, 1000), type="bca")$bca 489 | data.frame( 490 | mean = mean(.$elapsed), 491 | low_ci = boot[4], 492 | high_ci = boot[5] 493 | ) 494 | }) %>% 495 | ungroup() %>% 496 | mutate(framework = factor(framework)) 497 | 498 | df_speed 499 | ``` 500 | 501 | 502 | ```{r} 503 | plot <- ggplot(df_speed, aes(x=framework, y=mean, ymin=low_ci, ymax=high_ci, fill=framework)) + 504 | geom_bar(stat="identity", width=0.5) + 505 | geom_errorbar(width = 0.25) + 506 | theme_minimal(base_family="Source Sans Pro", base_size=14) + 507 | scale_fill_manual(values=framework_colors) + 508 | labs(title = "Speed of CNN of Text Generation via LSTM", 509 | x = "Keras Backend", 510 | y = "Average Epoch Runtime (seconds)", 511 | fill = "") 512 | 513 | plot %>% ggplotly(tooltip=c("x", "y")) 514 | plot %>% ggplotly(tooltip=c("x", "y"), width="100%", height=400) %>% 515 | saveWidget("keras-12.html", selfcontained=F, libdir="plotly") 516 | ``` 517 | 518 | ```{r} 519 | speed_lstm_gen = (df_speed$mean[2] / df_speed$mean[1]) %>% round(2) 520 | ``` 521 | 522 | CNTK runs at **`r speed_lstm_gen`**x the runtime of TensorFlow. 523 | 524 | ## Custom Keras 525 | 526 | 527 | ```{r} 528 | df_gen_keras_cntk <- read_csv(paste0(file_path, "text_generator_keras_cntk.csv")) 529 | df_gen_keras_tf <- read_csv(paste0(file_path, "text_generator_keras_tensorflow.csv")) 530 | ``` 531 | 532 | ### Test Acc 533 | 534 | ```{r} 535 | df <- df_gen_keras_cntk %>% mutate(framework = 'CNTK', epoch = epoch + 1) %>% union( 536 | df_gen_keras_tf %>% mutate(framework = 'TensorFlow', epoch = epoch + 1) 537 | ) 538 | 539 | df 540 | ``` 541 | 542 | 543 | ```{r} 544 | plot <- ggplot(df, aes(epoch, loss, color=framework)) + 545 | geom_line() + 546 | geom_point(size=2) + 547 | theme_minimal(base_family="Source Sans Pro", base_size=14) + 548 | scale_color_manual(values=framework_colors) + 549 | labs(title = "Performance of Text Generation via Custom Keras Model", 550 | x = "Epoch", 551 | y = "Loss (Lower is Better)", 552 | color="") 553 | 554 | plot %>% ggplotly() 555 | plot %>% ggplotly(width="100%", height=400) %>% 556 | saveWidget("keras-13.html", selfcontained=F, libdir="plotly") 557 | ``` 558 | 559 | ### Speed 560 | 561 | ```{r} 562 | df_speed <- df %>% group_by(framework) %>% 563 | do({ 564 | boot <- boot.ci(boot(., elapsed_mean, 1000), type="bca")$bca 565 | data.frame( 566 | mean = mean(.$elapsed), 567 | low_ci = boot[4], 568 | high_ci = boot[5] 569 | ) 570 | }) %>% 571 | ungroup() %>% 572 | mutate(framework = factor(framework)) 573 | 574 | df_speed 575 | ``` 576 | 577 | 578 | ```{r} 579 | plot <- ggplot(df_speed, aes(x=framework, y=mean, ymin=low_ci, ymax=high_ci, fill=framework)) + 580 | geom_bar(stat="identity", width=0.5) + 581 | geom_errorbar(width = 0.25) + 582 | theme_minimal(base_family="Source Sans Pro", base_size=14) + 583 | scale_fill_manual(values=framework_colors) + 584 | labs(title = "Speed of Text Generation via Custom Keras Model", 585 | x = "Keras Backend", 586 | y = "Average Epoch Runtime (seconds)", 587 | fill = "") 588 | 589 | plot %>% ggplotly(tooltip=c("x", "y")) 590 | plot %>% ggplotly(tooltip=c("x", "y"), width="100%", height=400) %>% 591 | saveWidget("keras-14.html", selfcontained=F, libdir="plotly") 592 | ``` 593 | 594 | ```{r} 595 | speed_keras_gen = (df_speed$mean[2] / df_speed$mean[1]) %>% round(2) 596 | ``` 597 | 598 | CNTK runs at **`r speed_keras_gen`**x the runtime of TensorFlow. 599 | 600 | ## LICENSE 601 | 602 | MIT License 603 | 604 | Copyright (c) 2017 Max Woolf 605 | 606 | Permission is hereby granted, free of charge, to any person obtaining a copy 607 | of this software and associated documentation files (the "Software"), to deal 608 | in the Software without restriction, including without limitation the rights 609 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 610 | copies of the Software, and to permit persons to whom the Software is 611 | furnished to do so, subject to the following conditions: 612 | 613 | The above copyright notice and this permission notice shall be included in all 614 | copies or substantial portions of the Software. 615 | 616 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 617 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 618 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 619 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 620 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 621 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 622 | SOFTWARE. -------------------------------------------------------------------------------- /keras_cntk_benchmark.py: -------------------------------------------------------------------------------- 1 | import os 2 | 3 | if not os.path.exists('logs'): 4 | os.makedirs('logs') 5 | 6 | test_files = [f for f in os.listdir("test_files") if f.endswith('.py')] 7 | test_files.remove('CustomCallback.py') 8 | backends = ['cntk', 'tensorflow'] 9 | docker_cmd = "sudo nvidia-docker run -it --rm -v $(pwd)/:/keras --name keras" 10 | 11 | for test_file in test_files: 12 | for backend in backends: 13 | statement = docker_cmd + \ 14 | " -e KERAS_BACKEND='{}' minimaxir/keras-cntk python3 test_files/{}".format( 15 | backend, test_file) 16 | 17 | print(statement) 18 | print("{} + {}".format(test_file, backend)) 19 | 20 | os.system(statement) 21 | -------------------------------------------------------------------------------- /logs/cifar10_cnn_cntk.csv: -------------------------------------------------------------------------------- 1 | epoch,elapsed,loss,acc,val_loss,val_acc 2 | 0,40.2804057598114,1.86589310416,0.310919494238,1.56854267807,0.4301 3 | 1,38.76938056945801,1.58716095463,0.418587896254,1.4087799221,0.4885 4 | 2,38.28054165840149,1.4530920033,0.474003362152,1.32039558773,0.5267 5 | 3,38.47169303894043,1.365978489,0.50754482869,1.29151532001,0.5329 6 | 4,37.824854612350464,1.30230481873,0.536703490234,1.11646338196,0.6017 7 | 5,38.11517596244812,1.23871728796,0.559177873839,1.07639890728,0.6228 8 | 6,39.43211388587952,1.19138928132,0.57792987512,1.0280661087,0.6412 9 | 7,40.955573081970215,1.14424348561,0.593780019212,1.00708253002,0.6423 10 | 8,41.23397469520569,1.10805962317,0.607829010567,0.94424066782,0.6695 11 | 9,43.67655897140503,1.07311177464,0.621397694524,0.922811874294,0.6745 12 | 10,40.5250768661499,1.04233114686,0.633205251361,0.924589164639,0.675 13 | 11,40.468372106552124,1.0190941199,0.642811399296,0.873791015053,0.6939 14 | 12,41.350709438323975,0.994231730575,0.649955971822,0.843022844696,0.7043 15 | 13,40.430147647857666,0.971577167587,0.657740954211,0.837625859547,0.7066 16 | 14,40.023507833480835,0.953034550438,0.664645373039,0.854900911999,0.6976 17 | 15,40.85829567909241,0.932977740176,0.672750560359,0.848833426762,0.7036 18 | 16,42.565085887908936,0.921098662079,0.677933877682,0.781946517563,0.7328 19 | 17,40.20765733718872,0.906880486825,0.682596861992,0.801780790329,0.7259 20 | 18,40.36050057411194,0.888273588958,0.690982228626,0.783959844875,0.7279 21 | 19,41.97719216346741,0.886328784312,0.690241754723,0.756659758663,0.74 22 | -------------------------------------------------------------------------------- /logs/cifar10_cnn_tensorflow.csv: -------------------------------------------------------------------------------- 1 | epoch,elapsed,loss,acc,val_loss,val_acc 2 | 0,44.4142632484436,1.84825661691,0.317701664533,1.55345717812,0.4381 3 | 1,43.91377115249634,1.57259917091,0.422550432277,1.3669194725,0.5066 4 | 2,41.560434103012085,1.45258072355,0.474923951329,1.26068198357,0.5545 5 | 3,39.29543375968933,1.3620472317,0.51232788985,1.23445290813,0.5603 6 | 4,43.3046441078186,1.29324564703,0.538564681396,1.1094232811,0.6146 7 | 5,37.7802791595459,1.22538556482,0.564121037464,1.05347865505,0.6296 8 | 6,37.343838930130005,1.17121979784,0.58359349984,1.00312870846,0.6458 9 | 7,42.24616265296936,1.13273182215,0.600444284342,1.03691489449,0.6396 10 | 8,37.8385648727417,1.09194743103,0.61201168748,0.93775783968,0.6652 11 | 9,42.083117723464966,1.05933193331,0.627241434518,0.925485464287,0.6786 12 | 10,38.1989860534668,1.03492718792,0.636047070125,0.888392238903,0.6943 13 | 11,37.42759847640991,1.00790112171,0.644232308678,0.86097441721,0.7035 14 | 12,37.0068302154541,0.987320181711,0.653318123599,0.850618889523,0.7055 15 | 13,37.2577383518219,0.958732662848,0.665325808517,0.876722399044,0.699 16 | 14,36.35597038269043,0.953116552879,0.665525936599,0.816950187969,0.7159 17 | 15,36.11813402175903,0.934621452331,0.673491034262,0.814759126282,0.7171 18 | 16,38.07388377189636,0.918741381649,0.678314121037,0.788503902817,0.7327 19 | 17,38.167693853378296,0.90242383553,0.685338616715,0.760356424427,0.7348 20 | 18,39.817323207855225,0.887211844664,0.690662023695,0.762179770756,0.737 21 | 19,35.947898864746094,0.88304429815,0.695365033622,0.754483118725,0.7453 22 | -------------------------------------------------------------------------------- /logs/imdb_bidirectional_lstm_cntk.csv: -------------------------------------------------------------------------------- 1 | epoch,elapsed,loss,acc,val_loss,val_acc 2 | 0,157.65192556381226,0.424538787689,0.80056,0.337316939507,0.85196 3 | 1,149.98708653450012,0.225951759443,0.91224,0.362363909025,0.84492 4 | 2,153.018324136734,0.136436212782,0.95056,0.595512974324,0.83352 5 | 3,148.85844206809998,0.0919926258466,0.96828,0.584490527191,0.83544 6 | -------------------------------------------------------------------------------- /logs/imdb_bidirectional_lstm_tensorflow.csv: -------------------------------------------------------------------------------- 1 | epoch,elapsed,loss,acc,val_loss,val_acc 2 | 0,286.8201458454132,0.408391364622,0.81016,0.357212373309,0.84356 3 | 1,276.42249488830566,0.221626912638,0.91284,0.359876287594,0.84584 4 | 2,272.2098560333252,0.125970403229,0.95416,0.40103153791,0.836 5 | 3,271.13646626472473,0.0795135529992,0.97244,0.682387446499,0.83432 6 | -------------------------------------------------------------------------------- /logs/imdb_fasttext_cntk.csv: -------------------------------------------------------------------------------- 1 | epoch,elapsed,loss,acc,val_loss,val_acc 2 | 0,69.57469654083252,0.580163868866,0.78716,0.432025263309,0.85992 3 | 1,69.46677255630493,0.276750185275,0.9316,0.299230588417,0.89428 4 | 2,69.42021584510803,0.13589290071,0.97252,0.260243431845,0.90168 5 | 3,69.445716381073,0.0729293511319,0.989,0.242731614876,0.904 6 | 4,69.44344305992126,0.0408660746157,0.99524,0.234989104524,0.90636 7 | -------------------------------------------------------------------------------- /logs/imdb_fasttext_tensorflow.csv: -------------------------------------------------------------------------------- 1 | epoch,elapsed,loss,acc,val_loss,val_acc 2 | 0,59.124223947525024,0.580220301666,0.78712,0.431095582581,0.86024 3 | 1,58.11960506439209,0.276470099492,0.93084,0.298824888411,0.89416 4 | 2,58.11721467971802,0.136108452978,0.97232,0.260095517864,0.90144 5 | 3,58.11217999458313,0.0731679415584,0.98872,0.242653308477,0.90384 6 | 4,58.05756497383118,0.0410324006414,0.9952,0.234965501885,0.90684 7 | -------------------------------------------------------------------------------- /logs/lstm_text_generation_cntk.csv: -------------------------------------------------------------------------------- 1 | epoch,elapsed,loss,acc,val_loss,val_acc 2 | 0,49.866777181625366,2.00440777019,,, 3 | 1,46.70109510421753,1.64802151896,,, 4 | 2,45.76371359825134,1.55469932735,,, 5 | 3,45.96822810173035,1.50813395339,,, 6 | 4,45.72452354431152,1.47896788624,,, 7 | 5,46.282110929489136,1.45615055283,,, 8 | 6,45.693716287612915,1.44040952227,,, 9 | 7,45.93787097930908,1.42692558066,,, 10 | 8,45.73245286941528,1.41612182536,,, 11 | 9,45.919697284698486,1.40472281286,,, 12 | -------------------------------------------------------------------------------- /logs/lstm_text_generation_tensorflow.csv: -------------------------------------------------------------------------------- 1 | epoch,elapsed,loss,acc,val_loss,val_acc 2 | 0,90.75324249267578,2.01015738712,,, 3 | 1,88.81532311439514,1.66685410016,,, 4 | 2,87.67203569412231,1.55497078145,,, 5 | 3,87.93380308151245,1.50765904726,,, 6 | 4,86.99751901626587,1.47890785292,,, 7 | 5,86.36205768585205,1.45558360526,,, 8 | 6,86.4939694404602,1.44119170534,,, 9 | 7,85.7814691066742,1.42519511551,,, 10 | 8,86.92407774925232,1.41509616279,,, 11 | 9,88.44729614257812,1.40604660672,,, 12 | -------------------------------------------------------------------------------- /logs/mnist_cnn_cntk.csv: -------------------------------------------------------------------------------- 1 | epoch,elapsed,loss,acc,val_loss,val_acc 2 | 0,17.028452396392822,0.331873047519,0.898616666698,0.0799730597906,0.9768 3 | 1,15.757854223251343,0.119243453189,0.965333333365,0.0572860743888,0.9827 4 | 2,15.715027809143066,0.0892291597664,0.974000000064,0.0451378938406,0.9853 5 | 3,15.743610143661499,0.0748125498141,0.9779,0.0407408931152,0.9865 6 | 4,15.732864379882812,0.0653675059686,0.9804,0.0358311031747,0.9882 7 | 5,15.70543098449707,0.0591745432953,0.982600000064,0.0366426306298,0.9883 8 | 6,15.691295385360718,0.0534284668783,0.984483333397,0.0309565464654,0.9899 9 | 7,15.71666431427002,0.0471314096669,0.986133333333,0.0301007628715,0.9899 10 | 8,15.754114389419556,0.0456862730414,0.986616666698,0.0292966813193,0.9894 11 | 9,15.834892749786377,0.0435060719932,0.986983333397,0.0319727587268,0.9895 12 | 10,15.795220375061035,0.0391366513461,0.987783333397,0.0308343384376,0.9893 13 | 11,15.759152173995972,0.0370258491874,0.988966666698,0.0307478069558,0.9892 14 | -------------------------------------------------------------------------------- /logs/mnist_cnn_tensorflow.csv: -------------------------------------------------------------------------------- 1 | epoch,elapsed,loss,acc,val_loss,val_acc 2 | 0,12.282670974731445,0.335521505817,0.898966666698,0.0807156630933,0.9745 3 | 1,10.951014995574951,0.114269672016,0.966316666794,0.0534493427228,0.983 4 | 2,10.91591739654541,0.0859377976596,0.97473333346,0.0393288518414,0.9866 5 | 3,10.940671443939209,0.0709846737752,0.979333333397,0.0354606549665,0.9884 6 | 4,10.95503568649292,0.0609362660487,0.981883333429,0.0343359194011,0.989 7 | 5,11.110825061798096,0.056472609514,0.983233333333,0.0322479530187,0.9891 8 | 6,11.012576341629028,0.0490786953568,0.985200000064,0.0279548041783,0.9901 9 | 7,11.0181565284729,0.0478446786627,0.985583333365,0.0297413421015,0.9899 10 | 8,11.022599220275879,0.0436080189114,0.986983333365,0.0274853931204,0.9911 11 | 9,11.00107479095459,0.0417839774718,0.987200000095,0.0267874110861,0.9911 12 | 10,10.964780330657959,0.0383102151384,0.98825,0.024798527691,0.9918 13 | 11,11.016568422317505,0.0370451306949,0.988950000064,0.0256737682713,0.9916 14 | -------------------------------------------------------------------------------- /logs/mnist_mlp_cntk.csv: -------------------------------------------------------------------------------- 1 | epoch,elapsed,loss,acc,val_loss,val_acc 2 | 0,3.0137956142425537,0.242186372741,0.92466666673,0.128266207066,0.9584 3 | 1,2.86456036567688,0.102701489309,0.968683333365,0.102003812627,0.9709 4 | 2,2.872704267501831,0.0749231881797,0.977033333333,0.0853028113315,0.976 5 | 3,2.89451265335083,0.0600253074278,0.980850000032,0.0716481169874,0.9802 6 | 4,2.812457799911499,0.0493220958566,0.984933333397,0.0829220459462,0.9789 7 | 5,2.7403883934020996,0.0432435727214,0.987050000064,0.0696924929704,0.9831 8 | 6,2.715404987335205,0.0370951452812,0.989116666698,0.0686377533145,0.984 9 | 7,2.7048842906951904,0.034766115479,0.9898,0.0718466625467,0.9842 10 | 8,2.72576904296875,0.0316673108642,0.990466666698,0.0746223774353,0.9835 11 | 9,2.746661424636841,0.0283266301582,0.991716666667,0.0925723486873,0.9825 12 | 10,2.7482779026031494,0.0256678766826,0.992183333397,0.0870764082674,0.9823 13 | 11,2.716982126235962,0.0235901254994,0.993200000032,0.0965560917189,0.9819 14 | 12,2.728381633758545,0.0236752535511,0.993466666667,0.0906421001895,0.9842 15 | 13,2.7470028400421143,0.0227728806624,0.994050000032,0.102537439931,0.9832 16 | 14,2.737339496612549,0.0199852184341,0.994683333333,0.103263656684,0.9839 17 | 15,2.730595111846924,0.0213927870471,0.994383333397,0.0954030246086,0.9835 18 | 16,2.708587169647217,0.0202152497799,0.994616666667,0.103561137967,0.9838 19 | 17,2.7262139320373535,0.0191261547333,0.995133333333,0.103973092545,0.9825 20 | 18,2.7134830951690674,0.0179221526908,0.99556666673,0.111110156224,0.9833 21 | 19,2.715505599975586,0.0184923034692,0.995150000064,0.133161038316,0.9796 22 | -------------------------------------------------------------------------------- /logs/mnist_mlp_tensorflow.csv: -------------------------------------------------------------------------------- 1 | epoch,elapsed,loss,acc,val_loss,val_acc 2 | 0,3.9881763458251953,0.247517457684,0.923883333397,0.108368017657,0.9664 3 | 1,3.3139400482177734,0.102551094838,0.968400000064,0.0761688729202,0.9773 4 | 2,3.340587615966797,0.0730028645714,0.977733333333,0.0794973087247,0.9788 5 | 3,3.296327590942383,0.060755754816,0.982200000064,0.0894505322253,0.9764 6 | 4,3.307112216949463,0.0506240921368,0.984816666667,0.072089218424,0.981 7 | 5,3.3594954013824463,0.0436414991476,0.98698333346,0.0881698190795,0.9788 8 | 6,3.327594757080078,0.0383693618786,0.988400000095,0.0933947281386,0.9792 9 | 7,3.3208186626434326,0.0337913486809,0.990033333365,0.0768017513516,0.9823 10 | 8,3.337339162826538,0.0317978569403,0.990733333429,0.0845677987231,0.9823 11 | 9,3.4492383003234863,0.0311804633704,0.99146666673,0.0741520002239,0.984 12 | 10,3.4310672283172607,0.0258404452243,0.99245,0.091043373453,0.9851 13 | 11,3.3845322132110596,0.0279774191281,0.992266666698,0.0965806756109,0.982 14 | 12,3.3983795642852783,0.0245766369906,0.993333333333,0.0925275212025,0.9838 15 | 13,3.3495523929595947,0.023266648576,0.993800000064,0.0939913036475,0.9843 16 | 14,3.297057628631592,0.0209815669522,0.994400000064,0.0927997081935,0.9855 17 | 15,3.303403615951538,0.0218027054908,0.994183333333,0.094612107863,0.9841 18 | 16,3.3102893829345703,0.0190222535619,0.995033333397,0.108995261153,0.9818 19 | 17,3.32236647605896,0.0220266610151,0.99461666673,0.105040867896,0.9837 20 | 18,3.3431410789489746,0.0200786361082,0.995333333333,0.110549809264,0.9832 21 | 19,3.301548719406128,0.0189747179001,0.995516666667,0.108222102677,0.9839 22 | -------------------------------------------------------------------------------- /logs/text_generator_keras_cntk.csv: -------------------------------------------------------------------------------- 1 | epoch,elapsed,loss 2 | 0,102.9405415058136,2.00351653112 3 | 1,99.47491836547852,1.72485264239 4 | 2,99.23683285713196,1.60276587148 5 | 3,101.15853762626648,1.51993696254 6 | 4,99.94689559936523,1.46022373265 7 | 5,99.71121883392334,1.41141036682 8 | 6,100.9410355091095,1.37179022428 9 | 7,103.16774296760559,1.33587299931 10 | 8,99.7122688293457,1.3062139421 11 | 9,99.85623931884766,1.27900991821 12 | -------------------------------------------------------------------------------- /logs/text_generator_keras_tensorflow.csv: -------------------------------------------------------------------------------- 1 | epoch,elapsed,loss 2 | 0,111.58463478088379,2.00852505003 3 | 1,110.27159023284912,1.73037641162 4 | 2,107.60015559196472,1.60431502849 5 | 3,108.13198494911194,1.52106773144 6 | 4,108.11294674873352,1.45870362669 7 | 5,109.10827541351318,1.40930885724 8 | 6,107.15311169624329,1.36821400507 9 | 7,109.48617720603943,1.33404665547 10 | 8,109.08394646644592,1.30228422424 11 | 9,109.05903244018555,1.27453109394 12 | -------------------------------------------------------------------------------- /test_files/CustomCallback.py: -------------------------------------------------------------------------------- 1 | import sys 2 | import csv 3 | import os 4 | import time 5 | from keras import backend as K 6 | from keras.callbacks import Callback 7 | 8 | 9 | class EpochStatsLogger(Callback): 10 | 11 | def on_train_begin(self, logs={}): 12 | filename = os.path.basename(sys.argv[0])[:-3] 13 | backend = K.backend() 14 | self.f = open('logs/{}_{}.csv'.format(filename, backend), 'w') 15 | self.log_writer = csv.writer(self.f) 16 | self.log_writer.writerow(['epoch', 'elapsed', 'loss', 17 | 'acc', 'val_loss', 'val_acc']) 18 | 19 | def on_train_end(self, logs={}): 20 | self.f.close() 21 | 22 | def on_epoch_begin(self, epoch, logs={}): 23 | self.start_time = time.time() 24 | 25 | def on_epoch_end(self, epoch, logs={}): 26 | self.log_writer.writerow([epoch, time.time() - self.start_time, 27 | logs.get('loss'), 28 | logs.get('acc'), 29 | logs.get('val_loss'), 30 | logs.get('val_acc')]) 31 | -------------------------------------------------------------------------------- /test_files/cifar10_cnn.py: -------------------------------------------------------------------------------- 1 | '''Train a simple deep CNN on the CIFAR10 small images dataset. 2 | 3 | GPU run command with Theano backend (with TensorFlow, the GPU is automatically used): 4 | THEANO_FLAGS=mode=FAST_RUN,device=gpu,floatx=float32 python cifar10_cnn.py 5 | 6 | It gets down to 0.65 test logloss in 25 epochs, and down to 0.55 after 50 epochs. 7 | (it's still underfitting at that point, though). 8 | ''' 9 | 10 | from __future__ import print_function 11 | import keras 12 | from keras.datasets import cifar10 13 | from keras.preprocessing.image import ImageDataGenerator 14 | from keras.models import Sequential 15 | from keras.layers import Dense, Dropout, Activation, Flatten 16 | from keras.layers import Conv2D, MaxPooling2D 17 | 18 | from CustomCallback import EpochStatsLogger 19 | logger = EpochStatsLogger() 20 | 21 | 22 | batch_size = 32 23 | num_classes = 10 24 | epochs = 20 25 | data_augmentation = True 26 | 27 | # The data, shuffled and split between train and test sets: 28 | (x_train, y_train), (x_test, y_test) = cifar10.load_data() 29 | print('x_train shape:', x_train.shape) 30 | print(x_train.shape[0], 'train samples') 31 | print(x_test.shape[0], 'test samples') 32 | 33 | # Convert class vectors to binary class matrices. 34 | y_train = keras.utils.to_categorical(y_train, num_classes) 35 | y_test = keras.utils.to_categorical(y_test, num_classes) 36 | 37 | model = Sequential() 38 | 39 | model.add(Conv2D(32, (3, 3), padding='same', 40 | input_shape=x_train.shape[1:])) 41 | model.add(Activation('relu')) 42 | model.add(Conv2D(32, (3, 3))) 43 | model.add(Activation('relu')) 44 | model.add(MaxPooling2D(pool_size=(2, 2))) 45 | model.add(Dropout(0.25)) 46 | 47 | model.add(Conv2D(64, (3, 3), padding='same')) 48 | model.add(Activation('relu')) 49 | model.add(Conv2D(64, (3, 3))) 50 | model.add(Activation('relu')) 51 | model.add(MaxPooling2D(pool_size=(2, 2))) 52 | model.add(Dropout(0.25)) 53 | 54 | model.add(Flatten()) 55 | model.add(Dense(512)) 56 | model.add(Activation('relu')) 57 | model.add(Dropout(0.5)) 58 | model.add(Dense(num_classes)) 59 | model.add(Activation('softmax')) 60 | 61 | # initiate RMSprop optimizer 62 | opt = keras.optimizers.rmsprop(lr=0.0001, decay=1e-6) 63 | 64 | # Let's train the model using RMSprop 65 | model.compile(loss='categorical_crossentropy', 66 | optimizer=opt, 67 | metrics=['accuracy']) 68 | 69 | x_train = x_train.astype('float32') 70 | x_test = x_test.astype('float32') 71 | x_train /= 255 72 | x_test /= 255 73 | 74 | if not data_augmentation: 75 | print('Not using data augmentation.') 76 | model.fit(x_train, y_train, 77 | batch_size=batch_size, 78 | epochs=epochs, 79 | validation_data=(x_test, y_test), 80 | shuffle=True) 81 | else: 82 | print('Using real-time data augmentation.') 83 | # This will do preprocessing and realtime data augmentation: 84 | datagen = ImageDataGenerator( 85 | featurewise_center=False, # set input mean to 0 over the dataset 86 | samplewise_center=False, # set each sample mean to 0 87 | featurewise_std_normalization=False, # divide inputs by std of the dataset 88 | samplewise_std_normalization=False, # divide each input by its std 89 | zca_whitening=False, # apply ZCA whitening 90 | rotation_range=0, # randomly rotate images in the range (degrees, 0 to 180) 91 | width_shift_range=0.1, # randomly shift images horizontally (fraction of total width) 92 | height_shift_range=0.1, # randomly shift images vertically (fraction of total height) 93 | horizontal_flip=True, # randomly flip images 94 | vertical_flip=False) # randomly flip images 95 | 96 | # Compute quantities required for feature-wise normalization 97 | # (std, mean, and principal components if ZCA whitening is applied). 98 | datagen.fit(x_train) 99 | 100 | # Fit the model on the batches generated by datagen.flow(). 101 | model.fit_generator(datagen.flow(x_train, y_train, 102 | batch_size=batch_size), 103 | steps_per_epoch=x_train.shape[0] // batch_size, 104 | epochs=epochs, 105 | validation_data=(x_test, y_test), 106 | callbacks=[logger]) 107 | -------------------------------------------------------------------------------- /test_files/imdb_bidirectional_lstm.py: -------------------------------------------------------------------------------- 1 | '''Train a Bidirectional LSTM on the IMDB sentiment classification task. 2 | 3 | Output after 4 epochs on CPU: ~0.8146 4 | Time per epoch on CPU (Core i7): ~150s. 5 | ''' 6 | 7 | from __future__ import print_function 8 | import numpy as np 9 | 10 | from keras.preprocessing import sequence 11 | from keras.models import Sequential 12 | from keras.layers import Dense, Dropout, Embedding, LSTM, Bidirectional 13 | from keras.datasets import imdb 14 | 15 | from CustomCallback import EpochStatsLogger 16 | logger = EpochStatsLogger() 17 | 18 | 19 | 20 | max_features = 20000 21 | # cut texts after this number of words 22 | # (among top max_features most common words) 23 | maxlen = 100 24 | batch_size = 32 25 | 26 | print('Loading data...') 27 | (x_train, y_train), (x_test, y_test) = imdb.load_data(num_words=max_features) 28 | print(len(x_train), 'train sequences') 29 | print(len(x_test), 'test sequences') 30 | 31 | print('Pad sequences (samples x time)') 32 | x_train = sequence.pad_sequences(x_train, maxlen=maxlen) 33 | x_test = sequence.pad_sequences(x_test, maxlen=maxlen) 34 | print('x_train shape:', x_train.shape) 35 | print('x_test shape:', x_test.shape) 36 | y_train = np.array(y_train) 37 | y_test = np.array(y_test) 38 | 39 | model = Sequential() 40 | model.add(Embedding(max_features, 128, input_length=maxlen)) 41 | model.add(Bidirectional(LSTM(64))) 42 | model.add(Dropout(0.5)) 43 | model.add(Dense(1, activation='sigmoid')) 44 | 45 | # try using different optimizers and different optimizer configs 46 | model.compile('adam', 'binary_crossentropy', metrics=['accuracy']) 47 | 48 | print('Train...') 49 | model.fit(x_train, y_train, 50 | batch_size=batch_size, 51 | epochs=4, 52 | validation_data=[x_test, y_test], 53 | callbacks=[logger]) 54 | -------------------------------------------------------------------------------- /test_files/imdb_fasttext.py: -------------------------------------------------------------------------------- 1 | '''This example demonstrates the use of fasttext for text classification 2 | 3 | Based on Joulin et al's paper: 4 | 5 | Bags of Tricks for Efficient Text Classification 6 | https://arxiv.org/abs/1607.01759 7 | 8 | Results on IMDB datasets with uni and bi-gram embeddings: 9 | Uni-gram: 0.8813 test accuracy after 5 epochs. 8s/epoch on i7 cpu. 10 | Bi-gram : 0.9056 test accuracy after 5 epochs. 2s/epoch on GTx 980M gpu. 11 | ''' 12 | 13 | from __future__ import print_function 14 | import numpy as np 15 | 16 | from keras.preprocessing import sequence 17 | from keras.models import Sequential 18 | from keras.layers import Dense 19 | from keras.layers import Embedding 20 | from keras.layers import GlobalAveragePooling1D 21 | from keras.datasets import imdb 22 | 23 | from CustomCallback import EpochStatsLogger 24 | logger = EpochStatsLogger() 25 | 26 | 27 | def create_ngram_set(input_list, ngram_value=2): 28 | """ 29 | Extract a set of n-grams from a list of integers. 30 | 31 | >>> create_ngram_set([1, 4, 9, 4, 1, 4], ngram_value=2) 32 | {(4, 9), (4, 1), (1, 4), (9, 4)} 33 | 34 | >>> create_ngram_set([1, 4, 9, 4, 1, 4], ngram_value=3) 35 | [(1, 4, 9), (4, 9, 4), (9, 4, 1), (4, 1, 4)] 36 | """ 37 | return set(zip(*[input_list[i:] for i in range(ngram_value)])) 38 | 39 | 40 | def add_ngram(sequences, token_indice, ngram_range=2): 41 | """ 42 | Augment the input list of list (sequences) by appending n-grams values. 43 | 44 | Example: adding bi-gram 45 | >>> sequences = [[1, 3, 4, 5], [1, 3, 7, 9, 2]] 46 | >>> token_indice = {(1, 3): 1337, (9, 2): 42, (4, 5): 2017} 47 | >>> add_ngram(sequences, token_indice, ngram_range=2) 48 | [[1, 3, 4, 5, 1337, 2017], [1, 3, 7, 9, 2, 1337, 42]] 49 | 50 | Example: adding tri-gram 51 | >>> sequences = [[1, 3, 4, 5], [1, 3, 7, 9, 2]] 52 | >>> token_indice = {(1, 3): 1337, (9, 2): 42, (4, 5): 2017, (7, 9, 2): 2018} 53 | >>> add_ngram(sequences, token_indice, ngram_range=3) 54 | [[1, 3, 4, 5, 1337], [1, 3, 7, 9, 2, 1337, 2018]] 55 | """ 56 | new_sequences = [] 57 | for input_list in sequences: 58 | new_list = input_list[:] 59 | for i in range(len(new_list) - ngram_range + 1): 60 | for ngram_value in range(2, ngram_range + 1): 61 | ngram = tuple(new_list[i:i + ngram_value]) 62 | if ngram in token_indice: 63 | new_list.append(token_indice[ngram]) 64 | new_sequences.append(new_list) 65 | 66 | return new_sequences 67 | 68 | # Set parameters: 69 | # ngram_range = 2 will add bi-grams features 70 | ngram_range = 2 71 | max_features = 20000 72 | maxlen = 400 73 | batch_size = 32 74 | embedding_dims = 50 75 | epochs = 5 76 | 77 | print('Loading data...') 78 | (x_train, y_train), (x_test, y_test) = imdb.load_data(num_words=max_features) 79 | print(len(x_train), 'train sequences') 80 | print(len(x_test), 'test sequences') 81 | print('Average train sequence length: {}'.format(np.mean(list(map(len, x_train)), dtype=int))) 82 | print('Average test sequence length: {}'.format(np.mean(list(map(len, x_test)), dtype=int))) 83 | 84 | if ngram_range > 1: 85 | print('Adding {}-gram features'.format(ngram_range)) 86 | # Create set of unique n-gram from the training set. 87 | ngram_set = set() 88 | for input_list in x_train: 89 | for i in range(2, ngram_range + 1): 90 | set_of_ngram = create_ngram_set(input_list, ngram_value=i) 91 | ngram_set.update(set_of_ngram) 92 | 93 | # Dictionary mapping n-gram token to a unique integer. 94 | # Integer values are greater than max_features in order 95 | # to avoid collision with existing features. 96 | start_index = max_features + 1 97 | token_indice = {v: k + start_index for k, v in enumerate(ngram_set)} 98 | indice_token = {token_indice[k]: k for k in token_indice} 99 | 100 | # max_features is the highest integer that could be found in the dataset. 101 | max_features = np.max(list(indice_token.keys())) + 1 102 | 103 | # Augmenting x_train and x_test with n-grams features 104 | x_train = add_ngram(x_train, token_indice, ngram_range) 105 | x_test = add_ngram(x_test, token_indice, ngram_range) 106 | print('Average train sequence length: {}'.format(np.mean(list(map(len, x_train)), dtype=int))) 107 | print('Average test sequence length: {}'.format(np.mean(list(map(len, x_test)), dtype=int))) 108 | 109 | print('Pad sequences (samples x time)') 110 | x_train = sequence.pad_sequences(x_train, maxlen=maxlen) 111 | x_test = sequence.pad_sequences(x_test, maxlen=maxlen) 112 | print('x_train shape:', x_train.shape) 113 | print('x_test shape:', x_test.shape) 114 | 115 | print('Build model...') 116 | model = Sequential() 117 | 118 | # we start off with an efficient embedding layer which maps 119 | # our vocab indices into embedding_dims dimensions 120 | model.add(Embedding(max_features, 121 | embedding_dims, 122 | input_length=maxlen)) 123 | 124 | # we add a GlobalAveragePooling1D, which will average the embeddings 125 | # of all words in the document 126 | model.add(GlobalAveragePooling1D()) 127 | 128 | # We project onto a single unit output layer, and squash it with a sigmoid: 129 | model.add(Dense(1, activation='sigmoid')) 130 | 131 | model.compile(loss='binary_crossentropy', 132 | optimizer='adam', 133 | metrics=['accuracy']) 134 | 135 | model.fit(x_train, y_train, 136 | batch_size=batch_size, 137 | epochs=epochs, 138 | validation_data=(x_test, y_test), 139 | callbacks=[logger]) 140 | -------------------------------------------------------------------------------- /test_files/lstm_text_generation.py: -------------------------------------------------------------------------------- 1 | '''Example script to generate text from Nietzsche's writings. 2 | 3 | At least 20 epochs are required before the generated text 4 | starts sounding coherent. 5 | 6 | It is recommended to run this script on GPU, as recurrent 7 | networks are quite computationally intensive. 8 | 9 | If you try this script on new data, make sure your corpus 10 | has at least ~100k characters. ~1M is better. 11 | ''' 12 | 13 | from __future__ import print_function 14 | from keras.models import Sequential 15 | from keras.layers import Dense, Activation 16 | from keras.layers import LSTM 17 | from keras.optimizers import RMSprop 18 | from keras.utils.data_utils import get_file 19 | import numpy as np 20 | import random 21 | import sys 22 | 23 | from CustomCallback import EpochStatsLogger 24 | logger = EpochStatsLogger() 25 | 26 | 27 | path = get_file('nietzsche.txt', origin='https://s3.amazonaws.com/text-datasets/nietzsche.txt') 28 | text = open(path).read().lower() 29 | print('corpus length:', len(text)) 30 | 31 | chars = sorted(list(set(text))) 32 | print('total chars:', len(chars)) 33 | char_indices = dict((c, i) for i, c in enumerate(chars)) 34 | indices_char = dict((i, c) for i, c in enumerate(chars)) 35 | 36 | # cut the text in semi-redundant sequences of maxlen characters 37 | maxlen = 40 38 | step = 3 39 | sentences = [] 40 | next_chars = [] 41 | for i in range(0, len(text) - maxlen, step): 42 | sentences.append(text[i: i + maxlen]) 43 | next_chars.append(text[i + maxlen]) 44 | print('nb sequences:', len(sentences)) 45 | 46 | print('Vectorization...') 47 | X = np.zeros((len(sentences), maxlen, len(chars)), dtype=np.bool) 48 | y = np.zeros((len(sentences), len(chars)), dtype=np.bool) 49 | for i, sentence in enumerate(sentences): 50 | for t, char in enumerate(sentence): 51 | X[i, t, char_indices[char]] = 1 52 | y[i, char_indices[next_chars[i]]] = 1 53 | 54 | 55 | # build the model: a single LSTM 56 | print('Build model...') 57 | model = Sequential() 58 | model.add(LSTM(128, input_shape=(maxlen, len(chars)), implementation=2)) 59 | model.add(Dense(len(chars))) 60 | model.add(Activation('softmax')) 61 | 62 | optimizer = RMSprop(lr=0.01) 63 | model.compile(loss='categorical_crossentropy', optimizer=optimizer) 64 | 65 | 66 | def sample(preds, temperature=1.0): 67 | # helper function to sample an index from a probability array 68 | preds = np.asarray(preds).astype('float64') 69 | preds = np.log(preds) / temperature 70 | exp_preds = np.exp(preds) 71 | preds = exp_preds / np.sum(exp_preds) 72 | probas = np.random.multinomial(1, preds, 1) 73 | return np.argmax(probas) 74 | 75 | # train the model, output generated text after each iteration 76 | for iteration in [1]: 77 | print() 78 | print('-' * 50) 79 | print('Iteration', iteration) 80 | model.fit(X, y, 81 | batch_size=128, 82 | epochs=10, 83 | callbacks=[logger]) 84 | 85 | start_index = random.randint(0, len(text) - maxlen - 1) 86 | 87 | for diversity in [0.5]: 88 | print() 89 | print('----- diversity:', diversity) 90 | 91 | generated = '' 92 | sentence = text[start_index: start_index + maxlen] 93 | generated += sentence 94 | print('----- Generating with seed: "' + sentence + '"') 95 | sys.stdout.write(generated) 96 | 97 | for i in range(400): 98 | x = np.zeros((1, maxlen, len(chars))) 99 | for t, char in enumerate(sentence): 100 | x[0, t, char_indices[char]] = 1. 101 | 102 | preds = model.predict(x, verbose=0)[0] 103 | next_index = sample(preds, diversity) 104 | next_char = indices_char[next_index] 105 | 106 | generated += next_char 107 | sentence = sentence[1:] + next_char 108 | 109 | sys.stdout.write(next_char) 110 | sys.stdout.flush() 111 | print() 112 | -------------------------------------------------------------------------------- /test_files/mnist_cnn.py: -------------------------------------------------------------------------------- 1 | '''Trains a simple convnet on the MNIST dataset. 2 | 3 | Gets to 99.25% test accuracy after 12 epochs 4 | (there is still a lot of margin for parameter tuning). 5 | 16 seconds per epoch on a GRID K520 GPU. 6 | ''' 7 | 8 | from __future__ import print_function 9 | import keras 10 | from keras.datasets import mnist 11 | from keras.models import Sequential 12 | from keras.layers import Dense, Dropout, Flatten 13 | from keras.layers import Conv2D, MaxPooling2D 14 | from keras import backend as K 15 | 16 | from CustomCallback import EpochStatsLogger 17 | logger = EpochStatsLogger() 18 | 19 | 20 | batch_size = 128 21 | num_classes = 10 22 | epochs = 12 23 | 24 | # input image dimensions 25 | img_rows, img_cols = 28, 28 26 | 27 | # the data, shuffled and split between train and test sets 28 | (x_train, y_train), (x_test, y_test) = mnist.load_data() 29 | 30 | if K.image_data_format() == 'channels_first': 31 | x_train = x_train.reshape(x_train.shape[0], 1, img_rows, img_cols) 32 | x_test = x_test.reshape(x_test.shape[0], 1, img_rows, img_cols) 33 | input_shape = (1, img_rows, img_cols) 34 | else: 35 | x_train = x_train.reshape(x_train.shape[0], img_rows, img_cols, 1) 36 | x_test = x_test.reshape(x_test.shape[0], img_rows, img_cols, 1) 37 | input_shape = (img_rows, img_cols, 1) 38 | 39 | x_train = x_train.astype('float32') 40 | x_test = x_test.astype('float32') 41 | x_train /= 255 42 | x_test /= 255 43 | print('x_train shape:', x_train.shape) 44 | print(x_train.shape[0], 'train samples') 45 | print(x_test.shape[0], 'test samples') 46 | 47 | # convert class vectors to binary class matrices 48 | y_train = keras.utils.to_categorical(y_train, num_classes) 49 | y_test = keras.utils.to_categorical(y_test, num_classes) 50 | 51 | model = Sequential() 52 | model.add(Conv2D(32, kernel_size=(3, 3), 53 | activation='relu', 54 | input_shape=input_shape)) 55 | model.add(Conv2D(64, (3, 3), activation='relu')) 56 | model.add(MaxPooling2D(pool_size=(2, 2))) 57 | model.add(Dropout(0.25)) 58 | model.add(Flatten()) 59 | model.add(Dense(128, activation='relu')) 60 | model.add(Dropout(0.5)) 61 | model.add(Dense(num_classes, activation='softmax')) 62 | 63 | model.compile(loss=keras.losses.categorical_crossentropy, 64 | optimizer=keras.optimizers.Adadelta(), 65 | metrics=['accuracy']) 66 | 67 | model.fit(x_train, y_train, 68 | batch_size=batch_size, 69 | epochs=epochs, 70 | verbose=1, 71 | validation_data=(x_test, y_test), 72 | callbacks=[logger]) 73 | score = model.evaluate(x_test, y_test, verbose=0) 74 | print('Test loss:', score[0]) 75 | print('Test accuracy:', score[1]) 76 | -------------------------------------------------------------------------------- /test_files/mnist_mlp.py: -------------------------------------------------------------------------------- 1 | '''Trains a simple deep NN on the MNIST dataset. 2 | 3 | Gets to 98.40% test accuracy after 20 epochs 4 | (there is *a lot* of margin for parameter tuning). 5 | 2 seconds per epoch on a K520 GPU. 6 | ''' 7 | 8 | from __future__ import print_function 9 | 10 | import keras 11 | from keras.datasets import mnist 12 | from keras.models import Sequential 13 | from keras.layers import Dense, Dropout 14 | from keras.optimizers import RMSprop 15 | 16 | from CustomCallback import EpochStatsLogger 17 | logger = EpochStatsLogger() 18 | 19 | 20 | batch_size = 128 21 | num_classes = 10 22 | epochs = 20 23 | 24 | # the data, shuffled and split between train and test sets 25 | (x_train, y_train), (x_test, y_test) = mnist.load_data() 26 | 27 | x_train = x_train.reshape(60000, 784) 28 | x_test = x_test.reshape(10000, 784) 29 | x_train = x_train.astype('float32') 30 | x_test = x_test.astype('float32') 31 | x_train /= 255 32 | x_test /= 255 33 | print(x_train.shape[0], 'train samples') 34 | print(x_test.shape[0], 'test samples') 35 | 36 | # convert class vectors to binary class matrices 37 | y_train = keras.utils.to_categorical(y_train, num_classes) 38 | y_test = keras.utils.to_categorical(y_test, num_classes) 39 | 40 | model = Sequential() 41 | model.add(Dense(512, activation='relu', input_shape=(784,))) 42 | model.add(Dropout(0.2)) 43 | model.add(Dense(512, activation='relu')) 44 | model.add(Dropout(0.2)) 45 | model.add(Dense(10, activation='softmax')) 46 | 47 | model.summary() 48 | 49 | model.compile(loss='categorical_crossentropy', 50 | optimizer=RMSprop(), 51 | metrics=['accuracy']) 52 | 53 | history = model.fit(x_train, y_train, 54 | batch_size=batch_size, 55 | epochs=epochs, 56 | verbose=1, 57 | validation_data=(x_test, y_test), 58 | callbacks=[logger]) 59 | score = model.evaluate(x_test, y_test, verbose=0) 60 | print('Test loss:', score[0]) 61 | print('Test accuracy:', score[1]) 62 | -------------------------------------------------------------------------------- /test_files/text_generator_keras.py: -------------------------------------------------------------------------------- 1 | from keras.models import Model 2 | from keras.layers import Dense, Activation, Embedding 3 | from keras.layers import LSTM, Input, Bidirectional 4 | from keras.layers.merge import concatenate 5 | from keras.optimizers import RMSprop, Adam, SGD 6 | from keras.utils.data_utils import get_file 7 | from keras.layers.normalization import BatchNormalization 8 | from keras.callbacks import Callback, ModelCheckpoint 9 | from keras.utils import plot_model 10 | import numpy as np 11 | import random 12 | import sys 13 | import csv 14 | import os 15 | import h5py 16 | import time 17 | from keras import backend as K 18 | from keras.callbacks import Callback 19 | 20 | 21 | class EpochStatsLogger(Callback): 22 | 23 | def on_train_begin(self, logs={}): 24 | filename = os.path.basename(sys.argv[0])[:-3] 25 | backend = K.backend() 26 | self.f = open('logs/{}_{}.csv'.format(filename, backend), 'w') 27 | self.log_writer = csv.writer(self.f) 28 | self.log_writer.writerow(['epoch', 'elapsed', 'loss']) 29 | 30 | def on_train_end(self, logs={}): 31 | self.f.close() 32 | 33 | def on_epoch_begin(self, epoch, logs={}): 34 | self.start_time = time.time() 35 | 36 | def on_epoch_end(self, epoch, logs={}): 37 | self.log_writer.writerow([epoch, time.time() - self.start_time, 38 | logs.get('main_out_loss')]) 39 | 40 | logger = EpochStatsLogger() 41 | 42 | embeddings_path = "glove.840B.300d-char.txt" 43 | embedding_dim = 300 44 | batch_size = 128 45 | use_pca = False 46 | lr = 0.001 47 | lr_decay = 1e-4 48 | maxlen = 40 49 | consume_less = 2 # 0 for cpu, 2 for gpu 50 | 51 | path = get_file('nietzsche.txt', origin='https://s3.amazonaws.com/text-datasets/nietzsche.txt') 52 | text = open(path).read().lower() 53 | print('corpus length:', len(text)) 54 | 55 | chars = sorted(list(set(text))) 56 | print('total chars:', len(chars)) 57 | char_indices = dict((c, i) for i, c in enumerate(chars)) 58 | indices_char = dict((i, c) for i, c in enumerate(chars)) 59 | 60 | # cut the text in semi-redundant sequences of maxlen characters 61 | 62 | step = 3 63 | sentences = [] 64 | next_chars = [] 65 | for i in range(0, len(text) - maxlen, step): 66 | sentences.append(text[i: i + maxlen]) 67 | next_chars.append(text[i + maxlen]) 68 | print('nb sequences:', len(sentences)) 69 | 70 | 71 | print('Vectorization...') 72 | X = np.zeros((len(sentences), maxlen), dtype=np.int) 73 | y = np.zeros((len(sentences), len(chars)), dtype=np.bool) 74 | for i, sentence in enumerate(sentences): 75 | for t, char in enumerate(sentence): 76 | X[i, t] = char_indices[char] 77 | y[i, char_indices[next_chars[i]]] = 1 78 | 79 | 80 | print('Build model...') 81 | main_input = Input(shape=(maxlen,)) 82 | embedding_layer = Embedding( 83 | len(chars), embedding_dim, input_length=maxlen) 84 | 85 | embedded = embedding_layer(main_input) 86 | 87 | # RNN Layer 88 | rnn = LSTM(128, implementation=consume_less)(embedded) 89 | 90 | aux_output = Dense(len(chars))(rnn) 91 | aux_output = Activation('softmax', name='aux_out')(aux_output) 92 | 93 | # Hidden Layers 94 | hidden_1 = Dense(512, use_bias=False)(rnn) 95 | hidden_1 = BatchNormalization()(hidden_1) 96 | hidden_1 = Activation('relu')(hidden_1) 97 | 98 | hidden_2 = Dense(256, use_bias=False)(hidden_1) 99 | hidden_2 = BatchNormalization()(hidden_2) 100 | hidden_2 = Activation('relu')(hidden_2) 101 | 102 | main_output = Dense(len(chars))(hidden_2) 103 | main_output = Activation('softmax', name='main_out')(main_output) 104 | 105 | model = Model(inputs=main_input, outputs=[main_output, aux_output]) 106 | 107 | optimizer = Adam(lr=lr, decay=lr_decay) 108 | model.compile(loss='categorical_crossentropy', 109 | optimizer=optimizer, loss_weights=[1., 0.2]) 110 | model.summary() 111 | 112 | # plot_model(model, to_file='model.png', show_shapes=True) 113 | 114 | 115 | def sample(preds, temperature=1.0): 116 | # helper function to sample an index from a probability array 117 | preds = np.asarray(preds).astype('float64') 118 | preds = np.log(preds) / temperature 119 | exp_preds = np.exp(preds) 120 | preds = exp_preds / np.sum(exp_preds) 121 | probas = np.random.multinomial(1, preds, 1) 122 | return np.argmax(probas) 123 | 124 | 125 | for iteration in [1]: 126 | print() 127 | print('-' * 50) 128 | print('Iteration', iteration) 129 | 130 | model.fit(X, [y, y], batch_size=batch_size, 131 | epochs=10, callbacks=[logger]) 132 | 133 | start_index = random.randint(0, len(text) - maxlen - 1) 134 | 135 | for diversity in [0.5]: 136 | print() 137 | print('----- diversity:', diversity) 138 | 139 | generated = '' 140 | sentence = text[start_index: start_index + maxlen] 141 | generated += sentence 142 | print('----- Generating with seed: "' + sentence + '"') 143 | sys.stdout.write(generated) 144 | 145 | for i in range(1200): 146 | x = np.zeros((1, maxlen), dtype=np.int) 147 | for t, char in enumerate(sentence): 148 | x[0, t] = char_indices[char] 149 | 150 | preds = model.predict(x, verbose=0)[0][0] 151 | next_index = sample(preds, diversity) 152 | next_char = indices_char[next_index] 153 | 154 | generated += next_char 155 | sentence = sentence[1:] + next_char 156 | 157 | sys.stdout.write(next_char) 158 | sys.stdout.flush() 159 | print() 160 | 161 | -------------------------------------------------------------------------------- /v2/analysis/.Rhistory: -------------------------------------------------------------------------------- 1 | install.packages(c("backports", "BH", "broom", "car", "caret", "crayon", "curl", "data.table", "debugme", "devtools", "dplyr", "geosphere", "glue", "gridExtra", "hms", "httr", "keras", "lava", "lazyeval", "lme4", "lubridate", "Matrix", "mgcv", "openssl", "PKI", "psych", "purrr", "quantreg", "raster", "Rcpp", "RcppEigen", "reticulate", "rlang", "rmarkdown", "rsconnect", "rstudioapi", "scales", "shiny", "stringi", "tensorflow", "tfruns", "tibble", "tidyr", "tidyselect", "tidyverse", "withr")) 2 | library(scales) 3 | library(tidyverse) 4 | library(RColorBrewer) 5 | sessionInfo() 6 | theme_set(theme_minimal(base_size=9, base_family="Source Sans Pro") + 7 | theme(plot.title = element_text(size=11, family="Source Sans Pro Bold"), 8 | axis.title.x = element_blank(), 9 | axis.title.y = element_text(family="Source Sans Pro Semibold"), 10 | plot.caption = element_text(size=6, color="#969696"), 11 | axis.text.x = element_text(angle = 45, vjust = 0.75, size = 7), 12 | legend.position="none")) 13 | relative <- function(x) { 14 | lab <- paste0(sprintf("%.2f", x), 'x') 15 | } 16 | standard_cost_hr <- 0.0475 17 | k80_cost_hr <- 0.450 18 | p100_cost_hr <- 1.460 19 | cpu16_cost_hr <- 0.120 20 | k80_cost_s <- (k80_cost_hr + standard_cost_hr) / 3600 21 | p100_cost_s <- (p100_cost_hr + standard_cost_hr) / 3600 22 | cpu16_cost_s <- cpu16_cost_hr / 3600 23 | cpu32_cost_s <- cpu16_cost_s * 2 24 | # works like a Python dict 25 | costs <- c(gpu=gpu_cost_s, cpu8=cpu8_cost_s, cpu16=cpu16_cost_s, cpu32=cpu32_cost_s, cpu64=cpu64_cost_s) 26 | base_label <- 'k80-tensorflow' 27 | label_split <- strsplit(base_label, "-") 28 | label-split 29 | label_split 30 | label_split[[1]] 31 | base_label <- 'k80-tensorflow' 32 | label_split <- str_split(base_label, "-")[[1]] 33 | label_split 34 | # frameworks <- c('tensorflow','cntk') 35 | # platforms <- c("p100", "k80", "cpu32", "cpu16") 36 | labels <- c('p100-tensorflow','p100-cntk', 'k80-tensorflow','k80-cntk', 'cpu32-tensorflow','cpu16-tensorflow') 37 | base_label %in% labels 38 | labels %in% base_label 39 | !(labels %in% base_label) 40 | labels[!(labels %in% base_label)] 41 | # frameworks <- c('tensorflow','cntk') 42 | # platforms <- c("p100", "k80", "cpu32", "cpu16") 43 | labels <- c('p100-tensorflow','p100-cntk', 'k80-tensorflow','k80-cntk', 'cpu32-tensorflow','cpu16-tensorflow') 44 | process_test_data <- function(file_name) { 45 | base_label <- 'k80-tensorflow' 46 | label_split <- str_split(base_label, "-")[[1]] 47 | results <- read_csv(sprintf("../logs/%s/%s_%s.csv", label_split[1], file_name, label_split[2])) %>% 48 | mutate(platform = label_split[1], framework = label_split[2]) %>% 49 | group_by(platform, framework) %>% 50 | summarize(total_time = sum(elapsed), 51 | total_cost = total_time * costs[label_split[1]]) 52 | base_total_time <- results %>% pull(total_time) 53 | base_total_cost <- results %>% pull(total_cost) 54 | labels_tf <- labels[!(labels %in% base_label)] 55 | for(i in 1:length(labels_tf)) { 56 | label = labels_tf[i] 57 | label_split <- str_split(label, "-")[[1]] 58 | temp_df <- read_csv(sprintf("../logs/%s/%s_%s.csv", label_split[1], file_name, label_split[2])) %>% 59 | mutate(platform = label_split[1], framework = label_split[2]) %>% 60 | group_by(platform, framework) %>% 61 | summarize(total_time = sum(elapsed), 62 | total_cost = total_time * costs[label_split[1]]) 63 | results <- results %>% bind_rows(temp_df) 64 | } 65 | # Normalize 66 | results <- results %>% 67 | mutate(total_time_norm = total_time / base_total_time, 68 | total_cost_norm = total_cost / base_total_cost 69 | ) 70 | return(results) 71 | } 72 | process_test_data('cifar10_cnn') 73 | standard_cost_hr <- 0.0475 74 | k80_cost_hr <- 0.450 75 | p100_cost_hr <- 1.460 76 | cpu16_cost_hr <- 0.120 77 | k80_cost_s <- (k80_cost_hr + standard_cost_hr) / 3600 78 | p100_cost_s <- (p100_cost_hr + standard_cost_hr) / 3600 79 | cpu16_cost_s <- cpu16_cost_hr / 3600 80 | cpu32_cost_s <- cpu16_cost_s * 2 81 | # works like a Python dict 82 | costs <- c(k80=k80_cost_s, p100=p100_cost_s, cpu16=cpu16_cost_s, cpu32=cpu32_cost_s) 83 | # frameworks <- c('tensorflow','cntk') 84 | # platforms <- c("p100", "k80", "cpu32", "cpu16") 85 | labels <- c('p100-tensorflow','p100-cntk', 'k80-tensorflow','k80-cntk', 'cpu32-tensorflow','cpu16-tensorflow') 86 | process_test_data <- function(file_name) { 87 | base_label <- 'k80-tensorflow' 88 | label_split <- str_split(base_label, "-")[[1]] 89 | results <- read_csv(sprintf("../logs/%s/%s_%s.csv", label_split[1], file_name, label_split[2])) %>% 90 | mutate(platform = label_split[1], framework = label_split[2]) %>% 91 | group_by(platform, framework) %>% 92 | summarize(total_time = sum(elapsed), 93 | total_cost = total_time * costs[label_split[1]]) 94 | base_total_time <- results %>% pull(total_time) 95 | base_total_cost <- results %>% pull(total_cost) 96 | labels_tf <- labels[!(labels %in% base_label)] 97 | for(i in 1:length(labels_tf)) { 98 | label = labels_tf[i] 99 | label_split <- str_split(label, "-")[[1]] 100 | temp_df <- read_csv(sprintf("../logs/%s/%s_%s.csv", label_split[1], file_name, label_split[2])) %>% 101 | mutate(platform = label_split[1], framework = label_split[2]) %>% 102 | group_by(platform, framework) %>% 103 | summarize(total_time = sum(elapsed), 104 | total_cost = total_time * costs[label_split[1]]) 105 | results <- results %>% bind_rows(temp_df) 106 | } 107 | # Normalize 108 | results <- results %>% 109 | mutate(total_time_norm = total_time / base_total_time, 110 | total_cost_norm = total_cost / base_total_cost 111 | ) 112 | return(results) 113 | } 114 | process_test_data('cifar10_cnn') 115 | # frameworks <- c('tensorflow','cntk') 116 | # platforms <- c("p100", "k80", "cpu32", "cpu16") 117 | labels <- c('p100-tensorflow','p100-cntk', 'k80-tensorflow','k80-cntk', 'cpu32-tensorflow','cpu16-tensorflow') 118 | process_test_data <- function(file_name) { 119 | base_label <- 'k80-tensorflow' 120 | label_split <- str_split(base_label, "-")[[1]] 121 | results <- read_csv(sprintf("../logs/%s/%s_%s.csv", label_split[1], file_name, label_split[2]), col_types = cols()) %>% 122 | mutate(platform = label_split[1], framework = label_split[2]) %>% 123 | group_by(platform, framework) %>% 124 | summarize(total_time = sum(elapsed), 125 | total_cost = total_time * costs[label_split[1]]) 126 | base_total_time <- results %>% pull(total_time) 127 | base_total_cost <- results %>% pull(total_cost) 128 | labels_tf <- labels[!(labels %in% base_label)] 129 | for(i in 1:length(labels_tf)) { 130 | label = labels_tf[i] 131 | label_split <- str_split(label, "-")[[1]] 132 | temp_df <- read_csv(sprintf("../logs/%s/%s_%s.csv", label_split[1], file_name, label_split[2]), col_types = cols()) %>% 133 | mutate(platform = label_split[1], framework = label_split[2]) %>% 134 | group_by(platform, framework) %>% 135 | summarize(total_time = sum(elapsed), 136 | total_cost = total_time * costs[label_split[1]]) 137 | results <- results %>% bind_rows(temp_df) 138 | } 139 | # Normalize 140 | results <- results %>% 141 | mutate(total_time_norm = total_time / base_total_time, 142 | total_cost_norm = total_cost / base_total_cost 143 | ) 144 | return(results) 145 | } 146 | process_test_data('cifar10_cnn') 147 | df_imdb_lstm <- process_test_data("imdb_bidirectional_lstm") 148 | df_imdb_lstm 149 | df_imdb_fasttext <- process_test_data("imdb_fasttext") 150 | df_imdb_fasttext 151 | df_mnist_mlp <- process_test_data("mnist_mlp") 152 | df_mnist_mlp 153 | df_mnist_cnn <- process_test_data("mnist_cnn") 154 | df_mnist_cnn 155 | df_cifar10_cnn <- process_test_data("cifar10_cnn") 156 | df_cifar10_cnn 157 | df_lstm_text <- process_test_data("lstm_text_generation") 158 | df_lstm_text 159 | df_custom_text <- process_test_data("text_generator_keras") 160 | -------------------------------------------------------------------------------- /v2/analysis/dl-cpu-gpu-1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/minimaxir/keras-cntk-benchmark/5b95f4640f371e0b532f0a14f6bb05466f41eef1/v2/analysis/dl-cpu-gpu-1.png -------------------------------------------------------------------------------- /v2/analysis/dl-cpu-gpu-10.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/minimaxir/keras-cntk-benchmark/5b95f4640f371e0b532f0a14f6bb05466f41eef1/v2/analysis/dl-cpu-gpu-10.png -------------------------------------------------------------------------------- /v2/analysis/dl-cpu-gpu-11.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/minimaxir/keras-cntk-benchmark/5b95f4640f371e0b532f0a14f6bb05466f41eef1/v2/analysis/dl-cpu-gpu-11.png -------------------------------------------------------------------------------- /v2/analysis/dl-cpu-gpu-12.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/minimaxir/keras-cntk-benchmark/5b95f4640f371e0b532f0a14f6bb05466f41eef1/v2/analysis/dl-cpu-gpu-12.png -------------------------------------------------------------------------------- /v2/analysis/dl-cpu-gpu-2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/minimaxir/keras-cntk-benchmark/5b95f4640f371e0b532f0a14f6bb05466f41eef1/v2/analysis/dl-cpu-gpu-2.png -------------------------------------------------------------------------------- /v2/analysis/dl-cpu-gpu-3.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/minimaxir/keras-cntk-benchmark/5b95f4640f371e0b532f0a14f6bb05466f41eef1/v2/analysis/dl-cpu-gpu-3.png -------------------------------------------------------------------------------- /v2/analysis/dl-cpu-gpu-4.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/minimaxir/keras-cntk-benchmark/5b95f4640f371e0b532f0a14f6bb05466f41eef1/v2/analysis/dl-cpu-gpu-4.png -------------------------------------------------------------------------------- /v2/analysis/dl-cpu-gpu-5.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/minimaxir/keras-cntk-benchmark/5b95f4640f371e0b532f0a14f6bb05466f41eef1/v2/analysis/dl-cpu-gpu-5.png -------------------------------------------------------------------------------- /v2/analysis/dl-cpu-gpu-6.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/minimaxir/keras-cntk-benchmark/5b95f4640f371e0b532f0a14f6bb05466f41eef1/v2/analysis/dl-cpu-gpu-6.png -------------------------------------------------------------------------------- /v2/analysis/dl-cpu-gpu-7.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/minimaxir/keras-cntk-benchmark/5b95f4640f371e0b532f0a14f6bb05466f41eef1/v2/analysis/dl-cpu-gpu-7.png -------------------------------------------------------------------------------- /v2/analysis/dl-cpu-gpu-8.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/minimaxir/keras-cntk-benchmark/5b95f4640f371e0b532f0a14f6bb05466f41eef1/v2/analysis/dl-cpu-gpu-8.png -------------------------------------------------------------------------------- /v2/analysis/dl-cpu-gpu-9.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/minimaxir/keras-cntk-benchmark/5b95f4640f371e0b532f0a14f6bb05466f41eef1/v2/analysis/dl-cpu-gpu-9.png -------------------------------------------------------------------------------- /v2/analysis/gpu_benchmark_v2.Rmd: -------------------------------------------------------------------------------- 1 | --- 2 | title: "Benchmarking Modern GPUs for Maximum Cloud Cost Efficiency in Deep Learning" 3 | author: "Max Woolf (@minimaxir)" 4 | date: "2017-11-28" 5 | output: 6 | html_notebook: 7 | highlight: tango 8 | mathjax: null 9 | number_sections: yes 10 | toc: yes 11 | --- 12 | 13 | This R Notebook is the complement to my blog post [Benchmarking Modern GPUs for Maximum Cloud Cost Efficiency in Deep Learning](http://minimaxir.com/2017/11/benchmark-gpus/). 14 | 15 | This notebook is licensed under the MIT License. If you use the code or data visualization designs contained within this notebook, it would be greatly appreciated if proper attribution is given back to this notebook and/or myself. Thanks! :) 16 | 17 | # Setup 18 | 19 | ```{r} 20 | library(scales) 21 | library(tidyverse) 22 | library(RColorBrewer) 23 | 24 | sessionInfo() 25 | ``` 26 | 27 | Set ggplot2 theme. 28 | 29 | ```{r} 30 | theme_set(theme_minimal(base_size=9, base_family="Source Sans Pro") + 31 | theme(plot.title = element_text(size=11, family="Source Sans Pro Bold"), 32 | axis.title.x = element_blank(), 33 | axis.title.y = element_blank(), 34 | plot.subtitle = element_text(family="Source Sans Pro Semibold", color="#969696"), 35 | plot.caption = element_text(size=6, color="#969696"), 36 | axis.text.x = element_text(size = 7), 37 | legend.position="none")) 38 | ``` 39 | 40 | ```{r} 41 | relative <- function(x) { 42 | lab <- paste0(sprintf("%.2f", x), 'x') 43 | } 44 | ``` 45 | 46 | Set known price rates from [Google Compute Engine Pricing](https://cloud.google.com/compute/pricing). 47 | 48 | ```{r} 49 | standard_cost_hr <- 0.0475 50 | k80_cost_hr <- 0.450 51 | p100_cost_hr <- 1.460 52 | cpu16_cost_hr <- 0.120 53 | ``` 54 | 55 | Derive the remaining rates, in seconds. 56 | 57 | ```{r} 58 | k80_cost_s <- (k80_cost_hr + standard_cost_hr) / 3600 59 | p100_cost_s <- (p100_cost_hr + standard_cost_hr) / 3600 60 | cpu16_cost_s <- cpu16_cost_hr / 3600 61 | cpu32_cost_s <- cpu16_cost_s * 2 62 | 63 | # works like a Python dict 64 | costs <- c(k80=k80_cost_s, p100=p100_cost_s, cpu16=cpu16_cost_s, cpu32=cpu32_cost_s) 65 | ``` 66 | 67 | # Analysis 68 | 69 | Create a helper function to return the results for all permutations of a given test file name. 70 | 71 | ```{r} 72 | # frameworks <- c('tensorflow','cntk') 73 | # platforms <- c("p100", "k80", "cpu32", "cpu16") 74 | labels <- c('p100-tensorflow','p100-cntk', 'k80-tensorflow','k80-cntk', 'cpu32-tensorflow','cpu16-tensorflow') 75 | 76 | 77 | process_test_data <- function(file_name) { 78 | base_label <- 'k80-tensorflow' 79 | label_split <- str_split(base_label, "-")[[1]] 80 | label_str <- paste(label_split[1], label_split[2], sep="\n") 81 | results <- read_csv(sprintf("../logs/%s/%s_%s.csv", label_split[1], file_name, label_split[2]), col_types = cols()) %>% 82 | mutate(platform = label_split[1], framework = label_split[2]) %>% 83 | group_by(platform, framework) %>% 84 | summarize(total_time = sum(elapsed), 85 | total_cost = total_time * costs[label_split[1]]) %>% 86 | mutate(label = label_str) 87 | 88 | base_total_time <- results %>% pull(total_time) 89 | base_total_cost <- results %>% pull(total_cost) 90 | 91 | labels_tf <- labels[!(labels %in% base_label)] 92 | 93 | for(i in 1:length(labels_tf)) { 94 | label = labels_tf[i] 95 | label_split <- str_split(label, "-")[[1]] 96 | label_str <- paste(label_split[1], label_split[2], sep="\n") 97 | 98 | temp_df <- read_csv(sprintf("../logs/%s/%s_%s.csv", label_split[1], file_name, label_split[2]), col_types = cols()) %>% 99 | mutate(platform = label_split[1], 100 | framework = label_split[2]) %>% 101 | group_by(platform, framework) %>% 102 | summarize(total_time = sum(elapsed), 103 | total_cost = total_time * costs[label_split[1]]) %>% 104 | ungroup() %>% 105 | mutate(label = label_str) 106 | 107 | results <- results %>% bind_rows(temp_df) 108 | 109 | } 110 | 111 | # Normalize 112 | results_final <- results %>% 113 | as_tibble() %>% 114 | mutate(total_time_norm = total_time / base_total_time, 115 | total_cost_norm = total_cost / base_total_cost, 116 | label = as_factor(label) 117 | ) 118 | 119 | return(results_final) 120 | 121 | } 122 | 123 | process_test_data('cifar10_cnn') 124 | ``` 125 | 126 | ## IMDB Bidirectional LSTM 127 | 128 | ```{r} 129 | df_imdb_lstm <- process_test_data("imdb_bidirectional_lstm") 130 | 131 | df_imdb_lstm 132 | ``` 133 | 134 | ```{r} 135 | plot <- ggplot(df_imdb_lstm, aes(x=fct_rev(label), y=total_time_norm, fill=platform)) + 136 | geom_bar(stat="identity") + 137 | geom_hline(yintercept = 1, linetype="dashed", color="#1a1a1a") + 138 | geom_text(aes(label = relative(total_time_norm), color=platform), vjust=-0.2, family="Source Sans Pro Bold", size=3) + 139 | 140 | scale_y_continuous(labels = relative) + 141 | scale_fill_hue(l=50) + 142 | scale_color_hue(l=50) + 143 | labs(title = "Benchmarking Speed of Training Bidirectional LSTMs", 144 | x = "Platform", 145 | subtitle = "Total Model Training Time, Relative to TensorFlow on K80 GPU", 146 | caption = "Max Woolf — minimaxir.com") 147 | 148 | ggsave("dl-cpu-gpu-1.png", plot, width=4, height=3) 149 | ``` 150 | 151 | ![](dl-cpu-gpu-1.png) 152 | 153 | ```{r} 154 | plot <- ggplot(df_imdb_lstm, aes(x=fct_rev(label), y=total_cost_norm, fill=platform)) + 155 | geom_bar(stat="identity") + 156 | geom_hline(yintercept = 1, linetype="dashed", color="#1a1a1a") + 157 | geom_text(aes(label = relative(total_cost_norm), color=platform), vjust=-0.2, family="Source Sans Pro Bold", size=3) + 158 | 159 | scale_y_continuous(labels = relative) + 160 | scale_fill_hue(l=50) + 161 | scale_color_hue(l=50) + 162 | labs(title = "Benchmarking Cost of Training Bidirectional LSTMs", 163 | x = "Platform", 164 | subtitle = "Total Model Training Cost, Relative to TensorFlow on K80 GPU", 165 | caption = "Max Woolf — minimaxir.com") 166 | 167 | ggsave("dl-cpu-gpu-2.png", plot, width=4, height=3) 168 | ``` 169 | 170 | ![](dl-cpu-gpu-2.png) 171 | 172 | ## IMDB Fasttext 173 | 174 | ```{r} 175 | df_imdb_fasttext <- process_test_data("imdb_fasttext") 176 | 177 | df_imdb_fasttext 178 | ``` 179 | 180 | ```{r} 181 | plot <- ggplot(df_imdb_fasttext, aes(x=fct_rev(label), y=total_time_norm, fill=platform)) + 182 | geom_bar(stat="identity") + 183 | geom_hline(yintercept = 1, linetype="dashed", color="#1a1a1a") + 184 | geom_text(aes(label = relative(total_time_norm), color=platform), vjust=-0.2, family="Source Sans Pro Bold", size=3) + 185 | 186 | scale_y_continuous(labels = relative) + 187 | scale_fill_hue(l=50) + 188 | scale_color_hue(l=50) + 189 | labs(title = "Benchmarking Speed of Training fasttext", 190 | x = "Platform", 191 | subtitle = "Total Model Training Time, Relative to TensorFlow on K80 GPU", 192 | caption = "Max Woolf — minimaxir.com") 193 | 194 | ggsave("dl-cpu-gpu-3.png", plot, width=4, height=3) 195 | ``` 196 | 197 | ![](dl-cpu-gpu-3.png) 198 | 199 | ```{r} 200 | plot <- ggplot(df_imdb_fasttext, aes(x=fct_rev(label), y=total_cost_norm, fill=platform)) + 201 | geom_bar(stat="identity") + 202 | geom_hline(yintercept = 1, linetype="dashed", color="#1a1a1a") + 203 | geom_text(aes(label = relative(total_cost_norm), color=platform), vjust=-0.2, family="Source Sans Pro Bold", size=3) + 204 | 205 | scale_y_continuous(labels = relative) + 206 | scale_fill_hue(l=50) + 207 | scale_color_hue(l=50) + 208 | labs(title = "Benchmarking Cost of Training fasttext", 209 | x = "Platform", 210 | subtitle = "Total Model Training Cost, Relative to TensorFlow on K80 GPU", 211 | caption = "Max Woolf — minimaxir.com") 212 | 213 | ggsave("dl-cpu-gpu-4.png", plot, width=4, height=3) 214 | ``` 215 | 216 | ![](dl-cpu-gpu-4.png) 217 | 218 | ## MNIST MLP 219 | 220 | ```{r} 221 | df_mnist_mlp <- process_test_data("mnist_mlp") 222 | 223 | df_mnist_mlp 224 | ``` 225 | 226 | ```{r} 227 | plot <- ggplot(df_mnist_mlp, aes(x=fct_rev(label), y=total_time_norm, fill=platform)) + 228 | geom_bar(stat="identity") + 229 | geom_hline(yintercept = 1, linetype="dashed", color="#1a1a1a") + 230 | geom_text(aes(label = relative(total_time_norm), color=platform), vjust=-0.2, family="Source Sans Pro Bold", size=3) + 231 | 232 | scale_y_continuous(labels = relative) + 233 | scale_fill_hue(l=50) + 234 | scale_color_hue(l=50) + 235 | labs(title = "Benchmarking Speed of Training MLP for MNIST", 236 | x = "Platform", 237 | subtitle = "Total Model Training Time, Relative to TensorFlow on K80 GPU", 238 | caption = "Max Woolf — minimaxir.com") 239 | 240 | ggsave("dl-cpu-gpu-5.png", plot, width=4, height=3) 241 | ``` 242 | 243 | ![](dl-cpu-gpu-5.png) 244 | 245 | ```{r} 246 | plot <- ggplot(df_mnist_mlp, aes(x=fct_rev(label), y=total_cost_norm, fill=platform)) + 247 | geom_bar(stat="identity") + 248 | geom_hline(yintercept = 1, linetype="dashed", color="#1a1a1a") + 249 | geom_text(aes(label = relative(total_cost_norm), color=platform), vjust=-0.2, family="Source Sans Pro Bold", size=3) + 250 | 251 | scale_y_continuous(labels = relative) + 252 | scale_fill_hue(l=50) + 253 | scale_color_hue(l=50) + 254 | labs(title = "Benchmarking Cost of Training MLP for MNIST", 255 | x = "Platform", 256 | subtitle = "Total Model Training Cost, Relative to TensorFlow on K80 GPU", 257 | caption = "Max Woolf — minimaxir.com") 258 | 259 | ggsave("dl-cpu-gpu-6.png", plot, width=4, height=3) 260 | ``` 261 | 262 | ![](dl-cpu-gpu-6.png) 263 | 264 | ## MNIST CNN 265 | 266 | ```{r} 267 | df_mnist_cnn <- process_test_data("mnist_cnn") 268 | 269 | df_mnist_cnn 270 | ``` 271 | 272 | ```{r} 273 | plot <- ggplot(df_mnist_cnn, aes(x=fct_rev(label), y=total_time_norm, fill=platform)) + 274 | geom_bar(stat="identity") + 275 | geom_hline(yintercept = 1, linetype="dashed", color="#1a1a1a") + 276 | geom_text(aes(label = relative(total_time_norm), color=platform), vjust=-0.2, family="Source Sans Pro Bold", size=3) + 277 | 278 | scale_y_continuous(labels = relative) + 279 | scale_fill_hue(l=50) + 280 | scale_color_hue(l=50) + 281 | labs(title = "Benchmarking Speed of Training CNN for MNIST", 282 | x = "Platform", 283 | subtitle = "Total Model Training Time, Relative to TensorFlow on K80 GPU", 284 | caption = "Max Woolf — minimaxir.com") 285 | 286 | ggsave("dl-cpu-gpu-7.png", plot, width=4, height=3) 287 | ``` 288 | 289 | ![](dl-cpu-gpu-7.png) 290 | 291 | ```{r} 292 | plot <- ggplot(df_mnist_cnn, aes(x=fct_rev(label), y=total_cost_norm, fill=platform)) + 293 | geom_bar(stat="identity") + 294 | geom_hline(yintercept = 1, linetype="dashed", color="#1a1a1a") + 295 | geom_text(aes(label = relative(total_cost_norm), color=platform), vjust=-0.2, family="Source Sans Pro Bold", size=3) + 296 | 297 | scale_y_continuous(labels = relative) + 298 | scale_fill_hue(l=50) + 299 | scale_color_hue(l=50) + 300 | labs(title = "Benchmarking Cost of Training CNN for MNIST", 301 | x = "Platform", 302 | subtitle = "Total Model Training Cost, Relative to TensorFlow on K80 GPU", 303 | caption = "Max Woolf — minimaxir.com") 304 | 305 | ggsave("dl-cpu-gpu-8.png", plot, width=4, height=3) 306 | ``` 307 | 308 | ![](dl-cpu-gpu-8.png) 309 | 310 | ## CIFAR10 Deep CNN + MLP 311 | 312 | ```{r} 313 | df_cifar10_cnn <- process_test_data("cifar10_cnn") 314 | 315 | df_cifar10_cnn 316 | ``` 317 | 318 | ```{r} 319 | plot <- ggplot(df_cifar10_cnn, aes(x=fct_rev(label), y=total_time_norm, fill=platform)) + 320 | geom_bar(stat="identity") + 321 | geom_hline(yintercept = 1, linetype="dashed", color="#1a1a1a") + 322 | geom_text(aes(label = relative(total_time_norm), color=platform), vjust=-0.2, family="Source Sans Pro Bold", size=3) + 323 | 324 | scale_y_continuous(labels = relative) + 325 | scale_fill_hue(l=50) + 326 | scale_color_hue(l=50) + 327 | labs(title = "Benchmark Speed of Training MLP + CNN for CIFAR10", 328 | x = "Platform", 329 | subtitle = "Total Model Training Time, Relative to TensorFlow on K80 GPU", 330 | caption = "Max Woolf — minimaxir.com") 331 | 332 | ggsave("dl-cpu-gpu-9.png", plot, width=4, height=3) 333 | ``` 334 | 335 | ![](dl-cpu-gpu-9.png) 336 | 337 | ```{r} 338 | plot <- ggplot(df_cifar10_cnn, aes(x=fct_rev(label), y=total_cost_norm, fill=platform)) + 339 | geom_bar(stat="identity") + 340 | geom_hline(yintercept = 1, linetype="dashed", color="#1a1a1a") + 341 | geom_text(aes(label = relative(total_cost_norm), color=platform), vjust=-0.2, family="Source Sans Pro Bold", size=3) + 342 | 343 | scale_y_continuous(labels = relative) + 344 | scale_fill_hue(l=50) + 345 | scale_color_hue(l=50) + 346 | labs(title = "Benchmark Cost of Training MLP + CNN for CIFAR10", 347 | x = "Platform", 348 | subtitle = "Total Model Training Cost, Relative to TensorFlow on K80 GPU", 349 | caption = "Max Woolf — minimaxir.com") 350 | 351 | ggsave("dl-cpu-gpu-10.png", plot, width=4, height=3) 352 | ``` 353 | 354 | ![](dl-cpu-gpu-10.png) 355 | 356 | ## LSTM Text Generation 357 | 358 | ```{r} 359 | df_lstm_text <- process_test_data("lstm_text_generation") 360 | 361 | df_lstm_text 362 | ``` 363 | 364 | ```{r} 365 | plot <- ggplot(df_lstm_text, aes(x=fct_rev(label), y=total_time_norm, fill=platform)) + 366 | geom_bar(stat="identity") + 367 | geom_hline(yintercept = 1, linetype="dashed", color="#1a1a1a") + 368 | geom_text(aes(label = relative(total_time_norm), color=platform), vjust=-0.2, family="Source Sans Pro Bold", size=3) + 369 | 370 | scale_y_continuous(labels = relative) + 371 | scale_fill_hue(l=50) + 372 | scale_color_hue(l=50) + 373 | labs(title = "Benchmarking Speed of Training LSTM Text Generator", 374 | x = "Platform", 375 | subtitle = "Total Model Training Time, Relative to TensorFlow on K80 GPU", 376 | caption = "Max Woolf — minimaxir.com") 377 | 378 | ggsave("dl-cpu-gpu-11.png", plot, width=4, height=3) 379 | ``` 380 | 381 | ![](dl-cpu-gpu-11.png) 382 | 383 | ```{r} 384 | plot <- ggplot(df_lstm_text, aes(x=fct_rev(label), y=total_cost_norm, fill=platform)) + 385 | geom_bar(stat="identity") + 386 | geom_hline(yintercept = 1, linetype="dashed", color="#1a1a1a") + 387 | geom_text(aes(label = relative(total_cost_norm), color=platform), vjust=-0.2, family="Source Sans Pro Bold", size=3) + 388 | 389 | scale_y_continuous(labels = relative) + 390 | scale_fill_hue(l=50) + 391 | scale_color_hue(l=50) + 392 | labs(title = "Benchmarking Cost of Training LSTM Text Generator", 393 | x = "Platform", 394 | subtitle = "Total Model Training Cost, Relative to TensorFlow on K80 GPU", 395 | caption = "Max Woolf — minimaxir.com") 396 | 397 | ggsave("dl-cpu-gpu-12.png", plot, width=4, height=3) 398 | ``` 399 | 400 | ![](dl-cpu-gpu-12.png) 401 | 402 | ## LICENSE 403 | 404 | MIT License 405 | 406 | Copyright (c) 2017 Max Woolf 407 | 408 | Permission is hereby granted, free of charge, to any person obtaining a copy 409 | of this software and associated documentation files (the "Software"), to deal 410 | in the Software without restriction, including without limitation the rights 411 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 412 | copies of the Software, and to permit persons to whom the Software is 413 | furnished to do so, subject to the following conditions: 414 | 415 | The above copyright notice and this permission notice shall be included in all 416 | copies or substantial portions of the Software. 417 | 418 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 419 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 420 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 421 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 422 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 423 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 424 | SOFTWARE. -------------------------------------------------------------------------------- /v2/keras_cntk_benchmark.py: -------------------------------------------------------------------------------- 1 | import os 2 | import argparse 3 | 4 | parser = argparse.ArgumentParser() 5 | parser.add_argument("backend", nargs="?", type=str, default='all') 6 | args = parser.parse_args() 7 | 8 | if not os.path.exists('logs'): 9 | os.makedirs('logs') 10 | 11 | test_files = [f for f in os.listdir("test_files") if f.endswith('.py')] 12 | test_files.remove('CustomCallback.py') 13 | backends = ['cntk', 'tensorflow'] if args.backend == 'all' else ['tensorflow'] 14 | docker_cmd = "sudo nvidia-docker run -it --rm -v $(pwd)/:/keras --name keras" 15 | 16 | for test_file in test_files: 17 | for backend in backends: 18 | statement = docker_cmd + \ 19 | " -e KERAS_BACKEND='{}' minimaxir/keras-cntk python3 test_files/{}".format( 20 | backend, test_file) 21 | 22 | print(statement) 23 | print("{} + {}".format(test_file, backend)) 24 | 25 | os.system(statement) 26 | -------------------------------------------------------------------------------- /v2/keras_cntk_benchmark_cpu.py: -------------------------------------------------------------------------------- 1 | import os 2 | 3 | if not os.path.exists('logs'): 4 | os.makedirs('logs') 5 | 6 | test_files = [f for f in os.listdir("test_files") if f.endswith('.py')] 7 | test_files.remove('CustomCallback.py') 8 | backends = ['tensorflow'] 9 | docker_cmd = "sudo docker run -it --rm -v $(pwd)/:/keras --name keras" 10 | 11 | for test_file in test_files: 12 | for backend in backends: 13 | statement = docker_cmd + \ 14 | " -e KERAS_BACKEND='{}' minimaxir/keras-cntk:cpu python3 test_files/{}".format( 15 | backend, test_file) 16 | 17 | print(statement) 18 | print("{} + {}".format(test_file, backend)) 19 | 20 | os.system(statement) 21 | -------------------------------------------------------------------------------- /v2/logs/cpu16/cifar10_cnn_tensorflow.csv: -------------------------------------------------------------------------------- 1 | epoch,elapsed,loss,acc,val_loss,val_acc 2 | 0,77.85155463218689,1.86742397778,0.31274,1.545931534,0.4362 3 | 1,77.76814413070679,1.5661743671,0.42772,1.38367985096,0.4965 4 | 2,77.72126793861389,1.4450737096,0.47636,1.37010936718,0.5122 5 | 3,77.75143957138062,1.36734116772,0.50942,1.20524418736,0.573 6 | 4,77.7372932434082,1.30145150261,0.53386,1.13496678677,0.5996 7 | 5,77.75758814811707,1.24348509216,0.55658,1.08993252621,0.6166 8 | 6,77.69909715652466,1.18948881876,0.5765,1.03709352264,0.634 9 | 7,77.69506740570068,1.14151015396,0.59332,0.994248121071,0.651 10 | 8,77.81793189048767,1.10269918896,0.60682,0.986319536781,0.6562 11 | 9,77.87699484825134,1.07335769493,0.62348,0.980085507011,0.655 12 | 10,77.62610292434692,1.04862807915,0.6291,0.908432744789,0.6771 13 | 11,77.47897148132324,1.02069468899,0.641,0.865981108475,0.6969 14 | 12,77.48874258995056,0.998484708862,0.6483,0.890801371479,0.6885 15 | 13,77.59840536117554,0.97504091238,0.6568,0.834773423767,0.7091 16 | 14,77.49337720870972,0.960188118687,0.66176,0.811638911915,0.7176 17 | 15,77.55315899848938,0.94426899641,0.66764,0.84489747448,0.7118 18 | 16,77.68613696098328,0.930000379086,0.67668,0.8113323946,0.718 19 | 17,78.07343792915344,0.913533308887,0.68104,0.827635403919,0.7178 20 | 18,78.26746797561646,0.903662760849,0.68434,0.759495596504,0.7385 21 | 19,78.31115770339966,0.888804786091,0.69028,0.771488968754,0.733 22 | -------------------------------------------------------------------------------- /v2/logs/cpu16/imdb_bidirectional_lstm_tensorflow.csv: -------------------------------------------------------------------------------- 1 | epoch,elapsed,loss,acc,val_loss,val_acc 2 | 0,81.08267283439636,0.419936220913,0.80852,0.360823537784,0.85212 3 | 1,79.86627626419067,0.224093766255,0.913,0.417138373342,0.83764 4 | 2,79.49575710296631,0.12942130576,0.95296,0.500116160693,0.83728 5 | 3,78.66546535491943,0.0722072500691,0.97532,0.543437700028,0.83248 6 | -------------------------------------------------------------------------------- /v2/logs/cpu16/imdb_fasttext_tensorflow.csv: -------------------------------------------------------------------------------- 1 | epoch,elapsed,loss,acc,val_loss,val_acc 2 | 0,207.3773708343506,0.58186328022,0.7944,0.4324565312,0.85716 3 | 1,202.64179587364197,0.278605595074,0.93224,0.300852227392,0.89136 4 | 2,201.96806478500366,0.136882821631,0.97188,0.261535477505,0.9006 5 | 3,201.77582001686096,0.0736134610081,0.9882,0.243202977667,0.90384 6 | 4,201.63202595710754,0.0413228863817,0.99536,0.236758649368,0.90412 7 | -------------------------------------------------------------------------------- /v2/logs/cpu16/lstm_text_generation_tensorflow.csv: -------------------------------------------------------------------------------- 1 | epoch,elapsed,loss,acc,val_loss,val_acc 2 | 0,85.77240228652954,1.97452336302,,, 3 | 1,85.10480523109436,1.62679852929,,, 4 | 2,85.08443212509155,1.54008023527,,, 5 | 3,85.06283211708069,1.49445700385,,, 6 | 4,85.0007951259613,1.46522903293,,, 7 | 5,84.11580300331116,1.4441173019,,, 8 | 6,83.5541365146637,1.42873444225,,, 9 | 7,82.92258906364441,1.41432186194,,, 10 | 8,82.90804624557495,1.40431713995,,, 11 | 9,82.88181328773499,1.39510649326,,, 12 | -------------------------------------------------------------------------------- /v2/logs/cpu16/mnist_cnn_tensorflow.csv: -------------------------------------------------------------------------------- 1 | epoch,elapsed,loss,acc,val_loss,val_acc 2 | 0,47.27238154411316,0.326829688934,0.901400000032,0.0749106484288,0.9763 3 | 1,46.865362882614136,0.113179657129,0.966483333302,0.0534839995213,0.9836 4 | 2,46.89046573638916,0.0859067355633,0.9743,0.0407762795322,0.9867 5 | 3,46.86016583442688,0.0726060126389,0.979033333333,0.0369127379933,0.9873 6 | 4,46.77301812171936,0.0600206892361,0.981733333302,0.0342449276675,0.9881 7 | 5,46.83618688583374,0.0554004737874,0.983733333302,0.0332615607391,0.9884 8 | 6,46.75807595252991,0.0513681407422,0.984883333333,0.0302465125337,0.9894 9 | 7,46.81679844856262,0.0477050659845,0.985816666698,0.0291918482735,0.9906 10 | 8,46.75765776634216,0.0434242632816,0.987350000032,0.027686886591,0.9899 11 | 9,46.90495848655701,0.0406392022093,0.987466666667,0.0287437992523,0.9897 12 | 10,46.90260577201843,0.0393369773795,0.987933333333,0.0285187766067,0.991 13 | 11,46.851301193237305,0.0359615796708,0.989149999968,0.0335145690527,0.9894 14 | -------------------------------------------------------------------------------- /v2/logs/cpu16/mnist_mlp_tensorflow.csv: -------------------------------------------------------------------------------- 1 | epoch,elapsed,loss,acc,val_loss,val_acc 2 | 0,4.18695330619812,0.245993348845,0.924300000032,0.0977674564928,0.9697 3 | 1,3.945624589920044,0.105463405552,0.967883333302,0.0891822993651,0.9736 4 | 2,3.950881004333496,0.0765103109707,0.977333333302,0.0712863502568,0.9798 5 | 3,3.947377920150757,0.0608818912735,0.981883333365,0.092673603113,0.9758 6 | 4,3.9435343742370605,0.0495421472768,0.984700000032,0.0917445602114,0.9774 7 | 5,3.9409899711608887,0.0444461905805,0.98735,0.0743368053343,0.9802 8 | 6,3.9451606273651123,0.0379309795558,0.98875,0.0875477740738,0.9796 9 | 7,3.9459903240203857,0.0339267388372,0.989933333333,0.0800390482787,0.9825 10 | 8,3.949697256088257,0.0309562060685,0.991066666667,0.0749688516981,0.9835 11 | 9,3.9518189430236816,0.0293886809506,0.991200000032,0.0885325074587,0.9816 12 | 10,3.9533474445343018,0.0243559247977,0.992716666667,0.0889133313083,0.9828 13 | 11,3.95377516746521,0.0251947002812,0.992649999968,0.087899348662,0.9832 14 | 12,3.9507482051849365,0.0259302256187,0.992866666667,0.0977519898633,0.9819 15 | 13,3.948826313018799,0.0251612996625,0.9933,0.103621034013,0.983 16 | 14,3.944530725479126,0.0210150492429,0.994266666635,0.104736464325,0.9823 17 | 15,3.9507839679718018,0.02066708467,0.994566666667,0.117916404959,0.9817 18 | 16,3.9574639797210693,0.019710186212,0.994816666667,0.102070035719,0.984 19 | 17,3.9510512351989746,0.0182930554452,0.995283333333,0.103051655137,0.9847 20 | 18,3.9530203342437744,0.0197115965941,0.994749999968,0.109398936373,0.9849 21 | 19,3.961052417755127,0.0185151252578,0.995233333302,0.109457576773,0.9837 22 | -------------------------------------------------------------------------------- /v2/logs/cpu32/cifar10_cnn_tensorflow.csv: -------------------------------------------------------------------------------- 1 | epoch,elapsed,loss,acc,val_loss,val_acc 2 | 0,54.02229976654053,1.87044228924,0.31428,1.52794161396,0.4507 3 | 1,52.93217349052429,1.54952944912,0.43668,1.3436529562,0.5211 4 | 2,52.90557265281677,1.41780606434,0.49056,1.24718640823,0.5584 5 | 3,52.808589935302734,1.32052189369,0.5293,1.15437825623,0.5925 6 | 4,52.787386417388916,1.24885207905,0.5553,1.10655418015,0.6078 7 | 5,52.77370238304138,1.18498669773,0.57848,1.01651653538,0.6481 8 | 6,52.72513699531555,1.13590287663,0.59842,1.02259638004,0.6448 9 | 7,52.75477337837219,1.09761239166,0.61026,0.978209474659,0.6573 10 | 8,52.694387674331665,1.06202650055,0.62504,0.972441997147,0.6523 11 | 9,52.61835861206055,1.03042896042,0.63672,0.877878312111,0.6918 12 | 10,52.65453577041626,1.0049589279,0.64636,0.86827571888,0.6971 13 | 11,52.629932165145874,0.980816236916,0.65738,0.922544854832,0.677 14 | 12,52.659178733825684,0.955915523243,0.66398,0.844506147957,0.7077 15 | 13,52.61152672767639,0.944649474697,0.6688,0.816082772255,0.7155 16 | 14,52.498136043548584,0.92224991106,0.67602,0.815828598499,0.713 17 | 15,52.437281131744385,0.908104412556,0.68136,0.806905603409,0.7167 18 | 16,52.48247051239014,0.893034098854,0.68904,0.756306199837,0.7337 19 | 17,52.46162796020508,0.881056600552,0.69342,0.776272151947,0.7325 20 | 18,52.4447717666626,0.867293370457,0.69742,0.753894430065,0.7389 21 | 19,52.45112109184265,0.857023642273,0.70114,0.734237711716,0.7504 22 | -------------------------------------------------------------------------------- /v2/logs/cpu32/imdb_bidirectional_lstm_tensorflow.csv: -------------------------------------------------------------------------------- 1 | epoch,elapsed,loss,acc,val_loss,val_acc 2 | 0,79.57440185546875,0.420550251656,0.80848,0.349865781097,0.85204 3 | 1,77.37651443481445,0.223759819846,0.91272,0.409191547909,0.83856 4 | 2,77.09731793403625,0.125411113153,0.95536,0.435959491405,0.82792 5 | 3,76.39198470115662,0.0723444526505,0.97424,0.593988825978,0.83168 6 | -------------------------------------------------------------------------------- /v2/logs/cpu32/imdb_fasttext_tensorflow.csv: -------------------------------------------------------------------------------- 1 | epoch,elapsed,loss,acc,val_loss,val_acc 2 | 0,186.74625539779663,0.581868059416,0.7944,0.432452156181,0.8572 3 | 1,186.4736843109131,0.278603171916,0.93224,0.300851381159,0.89136 4 | 2,186.69903683662415,0.136883081198,0.97188,0.261536180801,0.9006 5 | 3,186.50076055526733,0.0736141765332,0.9882,0.243203435717,0.90388 6 | 4,186.4354681968689,0.041323501116,0.99536,0.236759076309,0.90412 7 | -------------------------------------------------------------------------------- /v2/logs/cpu32/lstm_text_generation_tensorflow.csv: -------------------------------------------------------------------------------- 1 | epoch,elapsed,loss,acc,val_loss,val_acc 2 | 0,75.53100109100342,1.98017068317,,, 3 | 1,74.93195295333862,1.63062357811,,, 4 | 2,74.84714531898499,1.54411474574,,, 5 | 3,74.78328275680542,1.50051357694,,, 6 | 4,74.26676487922668,1.47245821504,,, 7 | 5,74.08651185035706,1.45123158858,,, 8 | 6,73.89953446388245,1.43490475079,,, 9 | 7,74.04923582077026,1.42201335389,,, 10 | 8,74.26634120941162,1.41067897976,,, 11 | 9,72.93687295913696,1.39998660249,,, 12 | -------------------------------------------------------------------------------- /v2/logs/cpu32/mnist_cnn_tensorflow.csv: -------------------------------------------------------------------------------- 1 | epoch,elapsed,loss,acc,val_loss,val_acc 2 | 0,29.555381298065186,0.351053794481,0.893016666667,0.0780364168294,0.9765 3 | 1,28.702430725097656,0.114997463159,0.966733333302,0.0537732117772,0.9825 4 | 2,28.713917016983032,0.0851574542512,0.974333333333,0.0409160562579,0.9858 5 | 3,28.757818460464478,0.0705007861892,0.979416666698,0.0368437049543,0.9875 6 | 4,28.725544929504395,0.0639422024667,0.981200000032,0.0393348312979,0.9875 7 | 5,28.55361580848694,0.0553135386363,0.9834,0.0333812963224,0.9887 8 | 6,28.73941946029663,0.0519399103021,0.984816666635,0.0335396190858,0.988 9 | 7,28.662410974502563,0.0466181039562,0.986266666698,0.0304649807509,0.9895 10 | 8,28.744566679000854,0.0436511786158,0.987416666698,0.0294436884086,0.9898 11 | 9,28.54804539680481,0.0401053435301,0.987733333302,0.027865035728,0.99 12 | 10,28.729073762893677,0.0384579607084,0.988350000032,0.0324349100173,0.9897 13 | 11,28.683627128601074,0.0355422070161,0.989149999968,0.0277269085781,0.9909 14 | -------------------------------------------------------------------------------- /v2/logs/cpu32/mnist_mlp_tensorflow.csv: -------------------------------------------------------------------------------- 1 | epoch,elapsed,loss,acc,val_loss,val_acc 2 | 0,3.480964422225952,0.24634060665,0.923483333333,0.0994600394979,0.9689 3 | 1,3.186957359313965,0.10335813783,0.968600000032,0.081832732258,0.9755 4 | 2,3.0772931575775146,0.0751472705811,0.977416666667,0.0707948827654,0.9805 5 | 3,3.183433771133423,0.0624530684138,0.981683333365,0.0746616420452,0.9796 6 | 4,3.2244069576263428,0.0507344550878,0.984266666635,0.0783962795715,0.9807 7 | 5,3.2843751907348633,0.0436649174872,0.986816666635,0.0781278628245,0.9815 8 | 6,3.3099708557128906,0.03701409194,0.989283333302,0.0849756316754,0.9808 9 | 7,3.317821741104126,0.0347389107019,0.989716666635,0.0815079435548,0.9811 10 | 8,3.31480073928833,0.0321399734666,0.990566666667,0.0934977438043,0.9813 11 | 9,3.3313019275665283,0.0300871940791,0.99125,0.0815630925168,0.9836 12 | 10,3.3056399822235107,0.0277138383774,0.992083333333,0.102807439225,0.9798 13 | 11,3.3021066188812256,0.0251516443846,0.992966666635,0.0927888089603,0.9829 14 | 12,3.3133082389831543,0.0242939277526,0.993466666667,0.0881624193069,0.9828 15 | 13,3.305758237838745,0.0234689490666,0.993283333333,0.101330824429,0.984 16 | 14,3.2813072204589844,0.0221701166269,0.993683333333,0.0898127955079,0.9841 17 | 15,3.295175313949585,0.0208720814542,0.9944,0.105507926576,0.9821 18 | 16,3.3379170894622803,0.0180380731309,0.995183333302,0.110105264291,0.9844 19 | 17,3.4448604583740234,0.0205052753267,0.994999999968,0.109315524949,0.9836 20 | 18,3.402825117111206,0.0189037017223,0.995466666635,0.106839768716,0.9837 21 | 19,3.345201015472412,0.0168582454929,0.995766666635,0.110746482389,0.9835 22 | -------------------------------------------------------------------------------- /v2/logs/k80/cifar10_cnn_cntk.csv: -------------------------------------------------------------------------------- 1 | epoch,elapsed,loss,acc,val_loss,val_acc 2 | 0,40.2804057598114,1.86589310416,0.310919494238,1.56854267807,0.4301 3 | 1,38.76938056945801,1.58716095463,0.418587896254,1.4087799221,0.4885 4 | 2,38.28054165840149,1.4530920033,0.474003362152,1.32039558773,0.5267 5 | 3,38.47169303894043,1.365978489,0.50754482869,1.29151532001,0.5329 6 | 4,37.824854612350464,1.30230481873,0.536703490234,1.11646338196,0.6017 7 | 5,38.11517596244812,1.23871728796,0.559177873839,1.07639890728,0.6228 8 | 6,39.43211388587952,1.19138928132,0.57792987512,1.0280661087,0.6412 9 | 7,40.955573081970215,1.14424348561,0.593780019212,1.00708253002,0.6423 10 | 8,41.23397469520569,1.10805962317,0.607829010567,0.94424066782,0.6695 11 | 9,43.67655897140503,1.07311177464,0.621397694524,0.922811874294,0.6745 12 | 10,40.5250768661499,1.04233114686,0.633205251361,0.924589164639,0.675 13 | 11,40.468372106552124,1.0190941199,0.642811399296,0.873791015053,0.6939 14 | 12,41.350709438323975,0.994231730575,0.649955971822,0.843022844696,0.7043 15 | 13,40.430147647857666,0.971577167587,0.657740954211,0.837625859547,0.7066 16 | 14,40.023507833480835,0.953034550438,0.664645373039,0.854900911999,0.6976 17 | 15,40.85829567909241,0.932977740176,0.672750560359,0.848833426762,0.7036 18 | 16,42.565085887908936,0.921098662079,0.677933877682,0.781946517563,0.7328 19 | 17,40.20765733718872,0.906880486825,0.682596861992,0.801780790329,0.7259 20 | 18,40.36050057411194,0.888273588958,0.690982228626,0.783959844875,0.7279 21 | 19,41.97719216346741,0.886328784312,0.690241754723,0.756659758663,0.74 22 | -------------------------------------------------------------------------------- /v2/logs/k80/cifar10_cnn_tensorflow.csv: -------------------------------------------------------------------------------- 1 | epoch,elapsed,loss,acc,val_loss,val_acc 2 | 0,49.124043226242065,1.89704900368,0.30378,1.58163899021,0.4346 3 | 1,36.77831292152405,1.59887711494,0.41582,1.38911455631,0.5092 4 | 2,38.10988092422485,1.46357470089,0.4715,1.30308621178,0.5361 5 | 3,36.00877809524536,1.36823048454,0.50922,1.23947635441,0.5536 6 | 4,36.98551416397095,1.29094060871,0.54022,1.12344844437,0.6061 7 | 5,37.96749949455261,1.22307422348,0.56584,1.03874421005,0.6376 8 | 6,42.040690183639526,1.16834425901,0.58518,0.996676991463,0.6505 9 | 7,41.78765845298767,1.12156461182,0.60418,0.993986828804,0.6515 10 | 8,41.891352891922,1.07958936325,0.61814,0.916416798973,0.6821 11 | 9,35.74237895011902,1.04375737343,0.63156,0.888623579597,0.6903 12 | 10,37.371594190597534,1.01520398106,0.6433,0.838888675785,0.7092 13 | 11,41.38497304916382,0.989567587891,0.65382,0.876924628162,0.6955 14 | 12,37.094807386398315,0.967014650192,0.66118,0.811079662514,0.7203 15 | 13,35.44521903991699,0.944818131447,0.671,0.818454336452,0.7146 16 | 14,35.04994797706604,0.929915652943,0.67574,0.800851525497,0.7218 17 | 15,35.3215765953064,0.918187829552,0.68084,0.797418560505,0.7207 18 | 16,41.93111062049866,0.905485151711,0.68404,0.762911274624,0.7401 19 | 17,42.20401453971863,0.890715201759,0.692,0.759130449009,0.7405 20 | 18,35.532843351364136,0.883756470575,0.69474,0.729190686798,0.7464 21 | 19,37.32469415664673,0.867535569239,0.69846,0.732829962444,0.7464 22 | -------------------------------------------------------------------------------- /v2/logs/k80/imdb_bidirectional_lstm_cntk.csv: -------------------------------------------------------------------------------- 1 | epoch,elapsed,loss,acc,val_loss,val_acc 2 | 0,157.65192556381226,0.424538787689,0.80056,0.337316939507,0.85196 3 | 1,149.98708653450012,0.225951759443,0.91224,0.362363909025,0.84492 4 | 2,153.018324136734,0.136436212782,0.95056,0.595512974324,0.83352 5 | 3,148.85844206809998,0.0919926258466,0.96828,0.584490527191,0.83544 6 | -------------------------------------------------------------------------------- /v2/logs/k80/imdb_bidirectional_lstm_tensorflow.csv: -------------------------------------------------------------------------------- 1 | epoch,elapsed,loss,acc,val_loss,val_acc 2 | 0,45.43592882156372,0.422017547169,0.79972,0.347733076477,0.85248 3 | 1,42.77916216850281,0.226469208074,0.91212,0.442988648129,0.84384 4 | 2,42.54819297790527,0.126160395252,0.95536,0.474205054178,0.8424 5 | 3,42.65998125076294,0.0630990765588,0.97912,0.582901168704,0.83 6 | -------------------------------------------------------------------------------- /v2/logs/k80/imdb_fasttext_cntk.csv: -------------------------------------------------------------------------------- 1 | epoch,elapsed,loss,acc,val_loss,val_acc 2 | 0,69.57469654083252,0.580163868866,0.78716,0.432025263309,0.85992 3 | 1,69.46677255630493,0.276750185275,0.9316,0.299230588417,0.89428 4 | 2,69.42021584510803,0.13589290071,0.97252,0.260243431845,0.90168 5 | 3,69.445716381073,0.0729293511319,0.989,0.242731614876,0.904 6 | 4,69.44344305992126,0.0408660746157,0.99524,0.234989104524,0.90636 7 | -------------------------------------------------------------------------------- /v2/logs/k80/imdb_fasttext_tensorflow.csv: -------------------------------------------------------------------------------- 1 | epoch,elapsed,loss,acc,val_loss,val_acc 2 | 0,58.07392382621765,0.581614620399,0.79432,0.432139319954,0.85748 3 | 1,57.231138467788696,0.278327860069,0.93232,0.300754867039,0.89132 4 | 2,57.29234170913696,0.136762644334,0.97188,0.261495333652,0.90064 5 | 3,57.18907856941223,0.0735536189556,0.9882,0.243184900665,0.90388 6 | 4,57.09803009033203,0.0412910348547,0.99536,0.236750949664,0.90408 7 | -------------------------------------------------------------------------------- /v2/logs/k80/lstm_text_generation_cntk.csv: -------------------------------------------------------------------------------- 1 | epoch,elapsed,loss,acc,val_loss,val_acc 2 | 0,52.20422434806824,2.0447495056,,, 3 | 1,52.46811246871948,1.67135673964,,, 4 | 2,52.51628661155701,1.57640326793,,, 5 | 3,51.93128800392151,1.53340146365,,, 6 | 4,52.9999635219574,1.50317605586,,, 7 | 5,53.395265102386475,1.48386206895,,, 8 | 6,52.549546003341675,1.46864373226,,, 9 | 7,52.671395778656006,1.45513973705,,, 10 | 8,52.618128061294556,1.44479656297,,, 11 | 9,52.85879993438721,1.43656654531,,, 12 | -------------------------------------------------------------------------------- /v2/logs/k80/lstm_text_generation_tensorflow.csv: -------------------------------------------------------------------------------- 1 | epoch,elapsed,loss,acc,val_loss,val_acc 2 | 0,27.236011743545532,1.9519188328,,, 3 | 1,24.843777418136597,1.61629407054,,, 4 | 2,25.028794288635254,1.53332218693,,, 5 | 3,25.119277000427246,1.48881616057,,, 6 | 4,25.30532479286194,1.46209246144,,, 7 | 5,25.087280750274658,1.44108218506,,, 8 | 6,25.057835340499878,1.42560435041,,, 9 | 7,25.0188148021698,1.41358763832,,, 10 | 8,24.99008274078369,1.40277466295,,, 11 | 9,25.010312795639038,1.39460252792,,, 12 | -------------------------------------------------------------------------------- /v2/logs/k80/mnist_cnn_cntk.csv: -------------------------------------------------------------------------------- 1 | epoch,elapsed,loss,acc,val_loss,val_acc 2 | 0,17.028452396392822,0.331873047519,0.898616666698,0.0799730597906,0.9768 3 | 1,15.757854223251343,0.119243453189,0.965333333365,0.0572860743888,0.9827 4 | 2,15.715027809143066,0.0892291597664,0.974000000064,0.0451378938406,0.9853 5 | 3,15.743610143661499,0.0748125498141,0.9779,0.0407408931152,0.9865 6 | 4,15.732864379882812,0.0653675059686,0.9804,0.0358311031747,0.9882 7 | 5,15.70543098449707,0.0591745432953,0.982600000064,0.0366426306298,0.9883 8 | 6,15.691295385360718,0.0534284668783,0.984483333397,0.0309565464654,0.9899 9 | 7,15.71666431427002,0.0471314096669,0.986133333333,0.0301007628715,0.9899 10 | 8,15.754114389419556,0.0456862730414,0.986616666698,0.0292966813193,0.9894 11 | 9,15.834892749786377,0.0435060719932,0.986983333397,0.0319727587268,0.9895 12 | 10,15.795220375061035,0.0391366513461,0.987783333397,0.0308343384376,0.9893 13 | 11,15.759152173995972,0.0370258491874,0.988966666698,0.0307478069558,0.9892 14 | -------------------------------------------------------------------------------- /v2/logs/k80/mnist_cnn_tensorflow.csv: -------------------------------------------------------------------------------- 1 | epoch,elapsed,loss,acc,val_loss,val_acc 2 | 0,12.174193859100342,0.329419441716,0.900866666667,0.0809970830321,0.973 3 | 1,10.744781970977783,0.116141914745,0.965650000032,0.0542017631002,0.9819 4 | 2,10.696205615997314,0.0860263949235,0.974699999968,0.0477514946811,0.9849 5 | 3,10.691241025924683,0.0736939989765,0.978416666635,0.0426409541197,0.9852 6 | 4,10.70249629020691,0.0630857757131,0.981150000032,0.0343508055815,0.9887 7 | 5,10.730011224746704,0.0550646511614,0.983066666667,0.0359650225936,0.9885 8 | 6,10.696717500686646,0.0520215339363,0.984466666698,0.0321831733613,0.9889 9 | 7,10.737751722335815,0.0456735002846,0.986133333333,0.029996500011,0.9901 10 | 8,10.710510015487671,0.0438249525006,0.987016666698,0.0289622863797,0.991 11 | 9,10.713500738143921,0.0404004510393,0.987549999968,0.0271875864503,0.9911 12 | 10,10.700608491897583,0.0381636970873,0.988233333333,0.0262187655427,0.9918 13 | 11,10.645448446273804,0.0361841987362,0.988766666635,0.0274093576288,0.9916 14 | -------------------------------------------------------------------------------- /v2/logs/k80/mnist_mlp_cntk.csv: -------------------------------------------------------------------------------- 1 | epoch,elapsed,loss,acc,val_loss,val_acc 2 | 0,3.0137956142425537,0.242186372741,0.92466666673,0.128266207066,0.9584 3 | 1,2.86456036567688,0.102701489309,0.968683333365,0.102003812627,0.9709 4 | 2,2.872704267501831,0.0749231881797,0.977033333333,0.0853028113315,0.976 5 | 3,2.89451265335083,0.0600253074278,0.980850000032,0.0716481169874,0.9802 6 | 4,2.812457799911499,0.0493220958566,0.984933333397,0.0829220459462,0.9789 7 | 5,2.7403883934020996,0.0432435727214,0.987050000064,0.0696924929704,0.9831 8 | 6,2.715404987335205,0.0370951452812,0.989116666698,0.0686377533145,0.984 9 | 7,2.7048842906951904,0.034766115479,0.9898,0.0718466625467,0.9842 10 | 8,2.72576904296875,0.0316673108642,0.990466666698,0.0746223774353,0.9835 11 | 9,2.746661424636841,0.0283266301582,0.991716666667,0.0925723486873,0.9825 12 | 10,2.7482779026031494,0.0256678766826,0.992183333397,0.0870764082674,0.9823 13 | 11,2.716982126235962,0.0235901254994,0.993200000032,0.0965560917189,0.9819 14 | 12,2.728381633758545,0.0236752535511,0.993466666667,0.0906421001895,0.9842 15 | 13,2.7470028400421143,0.0227728806624,0.994050000032,0.102537439931,0.9832 16 | 14,2.737339496612549,0.0199852184341,0.994683333333,0.103263656684,0.9839 17 | 15,2.730595111846924,0.0213927870471,0.994383333397,0.0954030246086,0.9835 18 | 16,2.708587169647217,0.0202152497799,0.994616666667,0.103561137967,0.9838 19 | 17,2.7262139320373535,0.0191261547333,0.995133333333,0.103973092545,0.9825 20 | 18,2.7134830951690674,0.0179221526908,0.99556666673,0.111110156224,0.9833 21 | 19,2.715505599975586,0.0184923034692,0.995150000064,0.133161038316,0.9796 22 | -------------------------------------------------------------------------------- /v2/logs/k80/mnist_mlp_tensorflow.csv: -------------------------------------------------------------------------------- 1 | epoch,elapsed,loss,acc,val_loss,val_acc 2 | 0,3.6109423637390137,0.247495934212,0.924183333302,0.110450588347,0.9662 3 | 1,3.204498529434204,0.102795934665,0.968716666635,0.094417254135,0.9735 4 | 2,3.2131006717681885,0.0745966853221,0.977616666698,0.0887141774173,0.9753 5 | 3,3.225198984146118,0.0598167552054,0.98155,0.0739951520626,0.9798 6 | 4,3.2391326427459717,0.0494218067067,0.985350000032,0.0899514354088,0.9784 7 | 5,3.208841323852539,0.0432826337673,0.9868,0.085172977964,0.98 8 | 6,3.223283290863037,0.0397618806694,0.988566666667,0.0782685480784,0.9812 9 | 7,3.220470428466797,0.0366584119746,0.989733333302,0.0912196785451,0.9778 10 | 8,3.2216684818267822,0.0325212095627,0.990766666698,0.0855749940706,0.982 11 | 9,3.204092502593994,0.0297829642881,0.991283333365,0.0960519233256,0.9809 12 | 10,3.2223355770111084,0.027934777708,0.991866666667,0.0832015748993,0.9834 13 | 11,3.210875988006592,0.0238112516167,0.993,0.0901135490063,0.9813 14 | 12,3.2074482440948486,0.0216355573042,0.993616666635,0.106991900264,0.9808 15 | 13,3.2245447635650635,0.0232860148008,0.99335,0.0941657002897,0.9829 16 | 14,3.230410099029541,0.0214499377137,0.994433333365,0.0957102910411,0.9838 17 | 15,3.2193119525909424,0.0205901038301,0.994416666635,0.112226306082,0.9815 18 | 16,3.1905248165130615,0.0207331340988,0.994350000032,0.0948941006666,0.9839 19 | 17,3.157679796218872,0.0206543681536,0.9946,0.109549365188,0.9831 20 | 18,3.144972801208496,0.0175945445394,0.995266666667,0.107800236263,0.9839 21 | 19,3.1662495136260986,0.0166272717292,0.995633333333,0.106415643081,0.9833 22 | -------------------------------------------------------------------------------- /v2/logs/k80/text_generator_keras_tensorflow.csv: -------------------------------------------------------------------------------- 1 | epoch,elapsed,loss 2 | 0,111.58463478088379,2.00852505003 3 | 1,110.27159023284912,1.73037641162 4 | 2,107.60015559196472,1.60431502849 5 | 3,108.13198494911194,1.52106773144 6 | 4,108.11294674873352,1.45870362669 7 | 5,109.10827541351318,1.40930885724 8 | 6,107.15311169624329,1.36821400507 9 | 7,109.48617720603943,1.33404665547 10 | 8,109.08394646644592,1.30228422424 11 | 9,109.05903244018555,1.27453109394 12 | -------------------------------------------------------------------------------- /v2/logs/p100/cifar10_cnn_cntk.csv: -------------------------------------------------------------------------------- 1 | epoch,elapsed,loss,acc,val_loss,val_acc 2 | 0,35.41434669494629,1.85064567204,0.31648,1.55237310982,0.4353 3 | 1,31.950220346450806,1.56490099796,0.42612,1.38878220882,0.4976 4 | 2,31.92563033103943,1.45004784824,0.47502,1.28888796883,0.5351 5 | 3,25.42246723175049,1.36171865784,0.51282,1.19842368088,0.5767 6 | 4,25.96535897254944,1.29402110649,0.53838,1.18807289639,0.5922 7 | 5,26.026352882385254,1.23774277431,0.56026,1.11626679058,0.6099 8 | 6,26.050200939178467,1.18892648937,0.5811,1.01768425789,0.6447 9 | 7,26.626814603805542,1.14580399773,0.59272,0.991455718422,0.653 10 | 8,26.433076858520508,1.10264527672,0.60814,0.951521513176,0.6679 11 | 9,25.982987880706787,1.07188716293,0.6232,0.911626050568,0.6848 12 | 10,25.96177864074707,1.04138446806,0.63384,0.89799665451,0.6866 13 | 11,25.93162441253662,1.01115448589,0.64484,0.878210019493,0.6949 14 | 12,25.3612380027771,0.992895291023,0.65298,0.879528224182,0.6942 15 | 13,25.32689380645752,0.965735113487,0.66124,0.827446404362,0.7112 16 | 14,25.15267300605774,0.945130259552,0.66892,0.85607030077,0.7093 17 | 15,26.034178256988525,0.931635525646,0.6734,0.792795105076,0.724 18 | 16,26.00099754333496,0.909194149475,0.6834,0.798944708729,0.7226 19 | 17,25.99183225631714,0.896552643204,0.68804,0.779070177364,0.7312 20 | 18,25.93670153617859,0.892344875412,0.68732,0.741453601837,0.7438 21 | 19,26.523309230804443,0.880185710945,0.6916,0.747760982847,0.7436 22 | -------------------------------------------------------------------------------- /v2/logs/p100/cifar10_cnn_tensorflow.csv: -------------------------------------------------------------------------------- 1 | epoch,elapsed,loss,acc,val_loss,val_acc 2 | 0,49.01321721076965,1.82819095566,0.32652,1.51143625755,0.4554 3 | 1,26.299399852752686,1.53063364708,0.44304,1.35778492489,0.5125 4 | 2,26.108940839767456,1.41696154484,0.4907,1.24468954601,0.5638 5 | 3,26.511279344558716,1.33163410278,0.52446,1.19055610847,0.5819 6 | 4,27.311219453811646,1.25793643539,0.5536,1.09417643681,0.6156 7 | 5,27.148319482803345,1.19907695816,0.5761,1.02416698647,0.6434 8 | 6,27.31009340286255,1.14295100914,0.59452,0.987831526947,0.655 9 | 7,27.286990642547607,1.10113928558,0.6106,0.955387758827,0.6621 10 | 8,27.39044737815857,1.06871204445,0.62318,0.936024359131,0.6722 11 | 9,27.425236701965332,1.03589128574,0.63484,0.89375239563,0.6907 12 | 10,27.417020082473755,1.01060265057,0.6435,0.860002017307,0.7043 13 | 11,26.605664014816284,0.980788215103,0.6572,0.834214721298,0.709 14 | 12,26.04123091697693,0.961328783283,0.66216,0.824520044041,0.7057 15 | 13,26.346101760864258,0.944694975166,0.67126,0.811857182598,0.7166 16 | 14,27.222757816314697,0.921794348392,0.67494,0.800992681885,0.7185 17 | 15,27.398495197296143,0.908797464924,0.68118,0.787717997932,0.7294 18 | 16,27.325636625289917,0.89373969099,0.68764,0.811040874195,0.7173 19 | 17,27.321394205093384,0.882563320885,0.69192,0.775371003819,0.7356 20 | 18,27.381813287734985,0.869415559778,0.69648,0.755828210831,0.7448 21 | 19,27.334288120269775,0.862537681389,0.7023,0.755700751495,0.7383 22 | -------------------------------------------------------------------------------- /v2/logs/p100/imdb_bidirectional_lstm_cntk.csv: -------------------------------------------------------------------------------- 1 | epoch,elapsed,loss,acc,val_loss,val_acc 2 | 0,140.7444498538971,0.411184919176,0.80652,0.349638678818,0.8474 3 | 1,139.38370513916016,0.233582675409,0.91064,0.383020885577,0.84112 4 | 2,139.56415176391602,0.140347567888,0.94916,0.402713240271,0.83496 5 | 3,140.05550503730774,0.0726930761048,0.97632,0.625321593827,0.83592 6 | -------------------------------------------------------------------------------- /v2/logs/p100/imdb_bidirectional_lstm_tensorflow.csv: -------------------------------------------------------------------------------- 1 | epoch,elapsed,loss,acc,val_loss,val_acc 2 | 0,30.462503910064697,0.415879525814,0.8024,0.343477629547,0.8504 3 | 1,26.8399920463562,0.223236217868,0.91364,0.440553866968,0.84548 4 | 2,26.827641248703003,0.128415624348,0.9532,0.478915286465,0.83968 5 | 3,26.835662126541138,0.065707326307,0.97864,0.5827703141,0.83636 6 | -------------------------------------------------------------------------------- /v2/logs/p100/imdb_fasttext_cntk.csv: -------------------------------------------------------------------------------- 1 | epoch,elapsed,loss,acc,val_loss,val_acc 2 | 0,21.906052589416504,0.581958141413,0.7854,0.432687903767,0.85692 3 | 1,21.779340505599976,0.278528164072,0.9316,0.300865051823,0.89328 4 | 2,21.756834506988525,0.136962437005,0.97188,0.260597912827,0.90088 5 | 3,21.727710247039795,0.073471989522,0.98836,0.24298554038,0.90348 6 | 4,21.75581383705139,0.0414372600555,0.99548,0.235887133183,0.90508 7 | -------------------------------------------------------------------------------- /v2/logs/p100/imdb_fasttext_tensorflow.csv: -------------------------------------------------------------------------------- 1 | epoch,elapsed,loss,acc,val_loss,val_acc 2 | 0,21.647926807403564,0.581616509399,0.79432,0.432141548386,0.85748 3 | 1,19.836503744125366,0.278329746819,0.93232,0.300755511436,0.89132 4 | 2,19.67049479484558,0.136763426201,0.97188,0.261495608397,0.90064 5 | 3,19.835765600204468,0.0735540029907,0.9882,0.243185018597,0.90388 6 | 4,19.75249934196472,0.0412912451863,0.99536,0.236750996633,0.90408 7 | -------------------------------------------------------------------------------- /v2/logs/p100/lstm_text_generation_cntk.csv: -------------------------------------------------------------------------------- 1 | epoch,elapsed,loss,acc,val_loss,val_acc 2 | 0,47.82776474952698,2.00362462429,,, 3 | 1,53.1002676486969,1.64092446608,,, 4 | 2,52.41587448120117,1.54931462139,,, 5 | 3,45.20899248123169,1.50335526846,,, 6 | 4,44.457520961761475,1.4744956731,,, 7 | 5,43.986284017562866,1.45631904351,,, 8 | 6,44.94143271446228,1.44166715756,,, 9 | 7,45.481956243515015,1.43076751851,,, 10 | 8,50.373347759246826,1.42095098996,,, 11 | 9,52.73934292793274,1.41320039304,,, 12 | -------------------------------------------------------------------------------- /v2/logs/p100/lstm_text_generation_tensorflow.csv: -------------------------------------------------------------------------------- 1 | epoch,elapsed,loss,acc,val_loss,val_acc 2 | 0,23.13133668899536,1.97328571831,,, 3 | 1,19.93032670021057,1.62274533157,,, 4 | 2,19.70904779434204,1.53389883612,,, 5 | 3,19.720826625823975,1.48772088915,,, 6 | 4,16.772226572036743,1.45757675289,,, 7 | 5,17.743033170700073,1.43406329195,,, 8 | 6,19.841793298721313,1.41791005005,,, 9 | 7,19.903658390045166,1.40318179163,,, 10 | 8,20.02936577796936,1.39290764529,,, 11 | 9,16.534003257751465,1.38486074138,,, 12 | -------------------------------------------------------------------------------- /v2/logs/p100/mnist_cnn_cntk.csv: -------------------------------------------------------------------------------- 1 | epoch,elapsed,loss,acc,val_loss,val_acc 2 | 0,11.82216763496399,0.326736208165,0.901100000064,0.0785006817609,0.9764 3 | 1,5.071880578994751,0.113105143237,0.96716666673,0.0542550528556,0.9817 4 | 2,5.054255723953247,0.0859758155485,0.974833333333,0.0434627683289,0.9857 5 | 3,5.0534508228302,0.0715467862387,0.978483333365,0.0382352160536,0.9867 6 | 4,5.053097486495972,0.0619546099683,0.98181666673,0.0362876011433,0.9878 7 | 5,5.0498645305633545,0.0559698135475,0.98355,0.0328970663309,0.9897 8 | 6,5.052006483078003,0.0519534332429,0.984533333333,0.0341924524756,0.9883 9 | 7,5.051746368408203,0.047631043035,0.98615,0.0341497630327,0.9895 10 | 8,5.052472352981567,0.0453360629926,0.986233333397,0.0288171676521,0.9905 11 | 9,5.052483797073364,0.040593683285,0.98796666673,0.0306379469483,0.9908 12 | 10,5.044263601303101,0.0395454575708,0.988466666698,0.0339219108873,0.9886 13 | 11,5.038207054138184,0.0368747893045,0.988700000064,0.0276270765387,0.9905 14 | -------------------------------------------------------------------------------- /v2/logs/p100/mnist_cnn_tensorflow.csv: -------------------------------------------------------------------------------- 1 | epoch,elapsed,loss,acc,val_loss,val_acc 2 | 0,8.561296939849854,0.336880172348,0.898750000032,0.0754725995719,0.9764 3 | 1,5.159795045852661,0.113624921974,0.966349999968,0.0506873977751,0.9843 4 | 2,5.157105922698975,0.084952218008,0.974766666667,0.04235566618,0.9853 5 | 3,5.156136989593506,0.0713759913186,0.978800000032,0.0391565415795,0.9874 6 | 4,5.149126291275024,0.0619579523116,0.981666666667,0.0364029486202,0.9883 7 | 5,5.148672342300415,0.0554124212801,0.983249999968,0.0344643597597,0.9883 8 | 6,5.213907480239868,0.0503088297675,0.984833333333,0.0311649445834,0.9898 9 | 7,5.207034349441528,0.0448308366547,0.986316666635,0.0323342735426,0.9892 10 | 8,5.161535024642944,0.0433097507134,0.98725,0.0284608206159,0.9905 11 | 9,5.148395299911499,0.0382533318217,0.988666666635,0.0290173692139,0.9894 12 | 10,5.197574138641357,0.038663071763,0.988633333333,0.0347576850406,0.9891 13 | 11,5.147228002548218,0.0359459894394,0.989416666635,0.02766190141,0.9907 14 | -------------------------------------------------------------------------------- /v2/logs/p100/mnist_mlp_cntk.csv: -------------------------------------------------------------------------------- 1 | epoch,elapsed,loss,acc,val_loss,val_acc 2 | 0,2.197521448135376,0.245876102718,0.924500000032,0.0945696546674,0.9734 3 | 1,2.063632011413574,0.100857123989,0.96961666673,0.0970140635038,0.9722 4 | 2,2.0604915618896484,0.0739718242367,0.97761666673,0.0756785837076,0.98 5 | 3,2.097670316696167,0.0590243987307,0.981916666698,0.0984727152434,0.9749 6 | 4,2.1107234954833984,0.0515879580369,0.98455,0.0824642045647,0.9786 7 | 5,2.1193559169769287,0.0435750776829,0.98695,0.0756245337702,0.9822 8 | 6,2.109248399734497,0.0384008791779,0.98875,0.0753379534573,0.9823 9 | 7,2.1172096729278564,0.0353053958925,0.989733333397,0.0954459385332,0.9802 10 | 8,2.1065123081207275,0.0340137117852,0.990566666667,0.0842726349457,0.9822 11 | 9,2.1022870540618896,0.029356737094,0.991550000064,0.0919254814055,0.9832 12 | 10,2.1025261878967285,0.0267601940444,0.992533333365,0.113275160324,0.9802 13 | 11,2.0926437377929688,0.0249215214085,0.99265,0.0983570981265,0.9821 14 | 12,2.1015501022338867,0.0238423580886,0.993116666667,0.116123755859,0.98 15 | 13,2.1001813411712646,0.0237690269651,0.99351666673,0.115184633363,0.98 16 | 14,2.0978128910064697,0.0237733567451,0.993900000064,0.112987927809,0.981 17 | 15,2.0512855052948,0.0210668670031,0.994366666698,0.101834090287,0.9828 18 | 16,2.083719253540039,0.0202812521284,0.994633333333,0.11438970448,0.9824 19 | 17,2.0293850898742676,0.0209481597225,0.9949,0.10272702564,0.9838 20 | 18,2.0259127616882324,0.0189315928977,0.99526666673,0.122217949466,0.9815 21 | 19,2.029282569885254,0.0196373662193,0.995333333333,0.109695772326,0.9842 22 | -------------------------------------------------------------------------------- /v2/logs/p100/mnist_mlp_tensorflow.csv: -------------------------------------------------------------------------------- 1 | epoch,elapsed,loss,acc,val_loss,val_acc 2 | 0,4.271389484405518,0.247067463994,0.925033333302,0.105097129568,0.9674 3 | 1,2.676290273666382,0.101676740394,0.969466666698,0.0820931939911,0.9754 4 | 2,2.692317247390747,0.0760153124849,0.977083333333,0.0674006863666,0.9786 5 | 3,2.6823151111602783,0.0597352749076,0.981966666667,0.085734071897,0.9762 6 | 4,2.6592445373535156,0.0500145974994,0.984816666635,0.0779916787123,0.98 7 | 5,2.7165632247924805,0.0431384113956,0.986966666698,0.0717088033024,0.9826 8 | 6,2.6778101921081543,0.0382615744246,0.988533333302,0.0727083372459,0.983 9 | 7,2.6621932983398438,0.0347755475556,0.989583333302,0.0930764522769,0.9806 10 | 8,2.6892311573028564,0.0320931165359,0.990733333333,0.0802420222337,0.983 11 | 9,2.693310499191284,0.0287805814259,0.99145,0.0813027971034,0.9824 12 | 10,2.7231364250183105,0.0282789430404,0.99215,0.0966503628195,0.9821 13 | 11,2.731013536453247,0.0249852054702,0.993233333333,0.0963109616383,0.9834 14 | 12,2.6238136291503906,0.0250728149685,0.992899999968,0.0964100754891,0.9808 15 | 13,2.6369152069091797,0.0238675231889,0.993533333333,0.093276767934,0.9841 16 | 14,2.660953998565674,0.0210459832778,0.994033333333,0.0959468315524,0.9835 17 | 15,2.6416075229644775,0.0208935926063,0.994083333333,0.10254487436,0.983 18 | 16,2.667837142944336,0.0197679711693,0.994699999968,0.117201488869,0.9815 19 | 17,2.699467420578003,0.0210190677994,0.994416666667,0.114609972224,0.9812 20 | 18,2.7163631916046143,0.0210674966768,0.994533333333,0.100782306998,0.985 21 | 19,2.729889154434204,0.01670999035,0.995583333302,0.106166922863,0.9839 22 | -------------------------------------------------------------------------------- /v2/logs/p100/text_generator_keras_cntk.csv: -------------------------------------------------------------------------------- 1 | epoch,elapsed,loss 2 | 0,102.9405415058136,2.00351653112 3 | 1,99.47491836547852,1.72485264239 4 | 2,99.23683285713196,1.60276587148 5 | 3,101.15853762626648,1.51993696254 6 | 4,99.94689559936523,1.46022373265 7 | 5,99.71121883392334,1.41141036682 8 | 6,100.9410355091095,1.37179022428 9 | 7,103.16774296760559,1.33587299931 10 | 8,99.7122688293457,1.3062139421 11 | 9,99.85623931884766,1.27900991821 12 | -------------------------------------------------------------------------------- /v2/logs/p100/text_generator_keras_tensorflow.csv: -------------------------------------------------------------------------------- 1 | epoch,elapsed,loss 2 | 0,111.58463478088379,2.00852505003 3 | 1,110.27159023284912,1.73037641162 4 | 2,107.60015559196472,1.60431502849 5 | 3,108.13198494911194,1.52106773144 6 | 4,108.11294674873352,1.45870362669 7 | 5,109.10827541351318,1.40930885724 8 | 6,107.15311169624329,1.36821400507 9 | 7,109.48617720603943,1.33404665547 10 | 8,109.08394646644592,1.30228422424 11 | 9,109.05903244018555,1.27453109394 12 | -------------------------------------------------------------------------------- /v2/test_files/CustomCallback.py: -------------------------------------------------------------------------------- 1 | import sys 2 | import csv 3 | import os 4 | import time 5 | from keras import backend as K 6 | from keras.callbacks import Callback 7 | 8 | 9 | class EpochStatsLogger(Callback): 10 | 11 | def on_train_begin(self, logs={}): 12 | filename = os.path.basename(sys.argv[0])[:-3] 13 | backend = K.backend() 14 | self.f = open('logs/{}_{}.csv'.format(filename, backend), 'w') 15 | self.log_writer = csv.writer(self.f) 16 | self.log_writer.writerow(['epoch', 'elapsed', 'loss', 17 | 'acc', 'val_loss', 'val_acc']) 18 | 19 | def on_train_end(self, logs={}): 20 | self.f.close() 21 | 22 | def on_epoch_begin(self, epoch, logs={}): 23 | self.start_time = time.time() 24 | 25 | def on_epoch_end(self, epoch, logs={}): 26 | self.log_writer.writerow([epoch, time.time() - self.start_time, 27 | logs.get('loss'), 28 | logs.get('acc'), 29 | logs.get('val_loss'), 30 | logs.get('val_acc')]) 31 | -------------------------------------------------------------------------------- /v2/test_files/cifar10_cnn.py: -------------------------------------------------------------------------------- 1 | '''Train a simple deep CNN on the CIFAR10 small images dataset. 2 | 3 | GPU run command with Theano backend (with TensorFlow, the GPU is automatically used): 4 | THEANO_FLAGS=mode=FAST_RUN,device=gpu,floatx=float32 python cifar10_cnn.py 5 | 6 | It gets down to 0.65 test logloss in 25 epochs, and down to 0.55 after 50 epochs. 7 | (it's still underfitting at that point, though). 8 | ''' 9 | 10 | from __future__ import print_function 11 | import keras 12 | from keras.datasets import cifar10 13 | from keras.preprocessing.image import ImageDataGenerator 14 | from keras.models import Sequential 15 | from keras.utils import multi_gpu_model 16 | from keras.layers import Dense, Dropout, Activation, Flatten 17 | from keras.layers import Conv2D, MaxPooling2D 18 | from keras import backend as K 19 | 20 | from CustomCallback import EpochStatsLogger 21 | logger = EpochStatsLogger() 22 | 23 | 24 | batch_size = 32 25 | num_classes = 10 26 | epochs = 20 27 | data_augmentation = True 28 | 29 | # The data, shuffled and split between train and test sets: 30 | (x_train, y_train), (x_test, y_test) = cifar10.load_data() 31 | print('x_train shape:', x_train.shape) 32 | print(x_train.shape[0], 'train samples') 33 | print(x_test.shape[0], 'test samples') 34 | 35 | # Convert class vectors to binary class matrices. 36 | y_train = keras.utils.to_categorical(y_train, num_classes) 37 | y_test = keras.utils.to_categorical(y_test, num_classes) 38 | 39 | model = Sequential() 40 | 41 | model.add(Conv2D(32, (3, 3), padding='same', 42 | input_shape=x_train.shape[1:])) 43 | model.add(Activation('relu')) 44 | model.add(Conv2D(32, (3, 3))) 45 | model.add(Activation('relu')) 46 | model.add(MaxPooling2D(pool_size=(2, 2))) 47 | model.add(Dropout(0.25)) 48 | 49 | model.add(Conv2D(64, (3, 3), padding='same')) 50 | model.add(Activation('relu')) 51 | model.add(Conv2D(64, (3, 3))) 52 | model.add(Activation('relu')) 53 | model.add(MaxPooling2D(pool_size=(2, 2))) 54 | model.add(Dropout(0.25)) 55 | 56 | model.add(Flatten()) 57 | model.add(Dense(512)) 58 | model.add(Activation('relu')) 59 | model.add(Dropout(0.5)) 60 | model.add(Dense(num_classes)) 61 | model.add(Activation('softmax')) 62 | 63 | # initiate RMSprop optimizer 64 | opt = keras.optimizers.rmsprop(lr=0.0001, decay=1e-6) 65 | 66 | if K.backend() == 'tensorflow' and len(K.tensorflow_backend._get_available_gpus()) > 1: 67 | print("Using Multi-GPU Model") 68 | model = multi_gpu_model(model, gpus=len(K.tensorflow_backend._get_available_gpus())) 69 | 70 | # Let's train the model using RMSprop 71 | model.compile(loss='categorical_crossentropy', 72 | optimizer=opt, 73 | metrics=['accuracy']) 74 | 75 | x_train = x_train.astype('float32') 76 | x_test = x_test.astype('float32') 77 | x_train /= 255 78 | x_test /= 255 79 | 80 | if not data_augmentation: 81 | print('Not using data augmentation.') 82 | model.fit(x_train, y_train, 83 | batch_size=batch_size, 84 | epochs=epochs, 85 | validation_data=(x_test, y_test), 86 | shuffle=True) 87 | else: 88 | print('Using real-time data augmentation.') 89 | # This will do preprocessing and realtime data augmentation: 90 | datagen = ImageDataGenerator( 91 | featurewise_center=False, # set input mean to 0 over the dataset 92 | samplewise_center=False, # set each sample mean to 0 93 | featurewise_std_normalization=False, # divide inputs by std of the dataset 94 | samplewise_std_normalization=False, # divide each input by its std 95 | zca_whitening=False, # apply ZCA whitening 96 | rotation_range=0, # randomly rotate images in the range (degrees, 0 to 180) 97 | width_shift_range=0.1, # randomly shift images horizontally (fraction of total width) 98 | height_shift_range=0.1, # randomly shift images vertically (fraction of total height) 99 | horizontal_flip=True, # randomly flip images 100 | vertical_flip=False) # randomly flip images 101 | 102 | # Compute quantities required for feature-wise normalization 103 | # (std, mean, and principal components if ZCA whitening is applied). 104 | datagen.fit(x_train) 105 | 106 | # Fit the model on the batches generated by datagen.flow(). 107 | model.fit_generator(datagen.flow(x_train, y_train, 108 | batch_size=batch_size), 109 | steps_per_epoch=x_train.shape[0] // batch_size, 110 | epochs=epochs, 111 | validation_data=(x_test, y_test), 112 | callbacks=[logger]) 113 | -------------------------------------------------------------------------------- /v2/test_files/imdb_bidirectional_lstm.py: -------------------------------------------------------------------------------- 1 | '''Train a Bidirectional LSTM on the IMDB sentiment classification task. 2 | 3 | Output after 4 epochs on CPU: ~0.8146 4 | Time per epoch on CPU (Core i7): ~150s. 5 | ''' 6 | 7 | from __future__ import print_function 8 | import numpy as np 9 | 10 | from keras.preprocessing import sequence 11 | from keras.models import Sequential 12 | from keras.layers import Dense, Dropout, Embedding, LSTM, CuDNNLSTM, Bidirectional 13 | from keras.datasets import imdb 14 | from keras import backend as K 15 | from keras.utils import multi_gpu_model 16 | 17 | from CustomCallback import EpochStatsLogger 18 | logger = EpochStatsLogger() 19 | 20 | 21 | 22 | max_features = 20000 23 | # cut texts after this number of words 24 | # (among top max_features most common words) 25 | maxlen = 100 26 | batch_size = 32 27 | 28 | print('Loading data...') 29 | (x_train, y_train), (x_test, y_test) = imdb.load_data(num_words=max_features) 30 | print(len(x_train), 'train sequences') 31 | print(len(x_test), 'test sequences') 32 | 33 | print('Pad sequences (samples x time)') 34 | x_train = sequence.pad_sequences(x_train, maxlen=maxlen) 35 | x_test = sequence.pad_sequences(x_test, maxlen=maxlen) 36 | print('x_train shape:', x_train.shape) 37 | print('x_test shape:', x_test.shape) 38 | y_train = np.array(y_train) 39 | y_test = np.array(y_test) 40 | 41 | model = Sequential() 42 | model.add(Embedding(max_features, 128, input_length=maxlen)) 43 | if K.backend() == 'tensorflow' and len(K.tensorflow_backend._get_available_gpus()) > 0: 44 | print("Using CuDNNLSTM") 45 | model.add(Bidirectional(CuDNNLSTM(64))) 46 | else: 47 | model.add(Bidirectional(LSTM(64))) 48 | model.add(Dropout(0.5)) 49 | model.add(Dense(1, activation='sigmoid')) 50 | 51 | if K.backend() == 'tensorflow' and len(K.tensorflow_backend._get_available_gpus()) > 1: 52 | print("Using Multi-GPU Model") 53 | model = multi_gpu_model(model, gpus=len(K.tensorflow_backend._get_available_gpus())) 54 | 55 | # try using different optimizers and different optimizer configs 56 | model.compile('adam', 'binary_crossentropy', metrics=['accuracy']) 57 | 58 | print('Train...') 59 | model.fit(x_train, y_train, 60 | batch_size=batch_size, 61 | epochs=4, 62 | validation_data=[x_test, y_test], 63 | callbacks=[logger]) 64 | -------------------------------------------------------------------------------- /v2/test_files/imdb_fasttext.py: -------------------------------------------------------------------------------- 1 | '''This example demonstrates the use of fasttext for text classification 2 | 3 | Based on Joulin et al's paper: 4 | 5 | Bags of Tricks for Efficient Text Classification 6 | https://arxiv.org/abs/1607.01759 7 | 8 | Results on IMDB datasets with uni and bi-gram embeddings: 9 | Uni-gram: 0.8813 test accuracy after 5 epochs. 8s/epoch on i7 cpu. 10 | Bi-gram : 0.9056 test accuracy after 5 epochs. 2s/epoch on GTx 980M gpu. 11 | ''' 12 | 13 | from __future__ import print_function 14 | import numpy as np 15 | 16 | from keras.preprocessing import sequence 17 | from keras.models import Sequential 18 | from keras.layers import Dense 19 | from keras.layers import Embedding 20 | from keras.layers import GlobalAveragePooling1D 21 | from keras.datasets import imdb 22 | from keras.utils import multi_gpu_model 23 | from keras import backend as K 24 | 25 | from CustomCallback import EpochStatsLogger 26 | logger = EpochStatsLogger() 27 | 28 | 29 | def create_ngram_set(input_list, ngram_value=2): 30 | """ 31 | Extract a set of n-grams from a list of integers. 32 | 33 | >>> create_ngram_set([1, 4, 9, 4, 1, 4], ngram_value=2) 34 | {(4, 9), (4, 1), (1, 4), (9, 4)} 35 | 36 | >>> create_ngram_set([1, 4, 9, 4, 1, 4], ngram_value=3) 37 | [(1, 4, 9), (4, 9, 4), (9, 4, 1), (4, 1, 4)] 38 | """ 39 | return set(zip(*[input_list[i:] for i in range(ngram_value)])) 40 | 41 | 42 | def add_ngram(sequences, token_indice, ngram_range=2): 43 | """ 44 | Augment the input list of list (sequences) by appending n-grams values. 45 | 46 | Example: adding bi-gram 47 | >>> sequences = [[1, 3, 4, 5], [1, 3, 7, 9, 2]] 48 | >>> token_indice = {(1, 3): 1337, (9, 2): 42, (4, 5): 2017} 49 | >>> add_ngram(sequences, token_indice, ngram_range=2) 50 | [[1, 3, 4, 5, 1337, 2017], [1, 3, 7, 9, 2, 1337, 42]] 51 | 52 | Example: adding tri-gram 53 | >>> sequences = [[1, 3, 4, 5], [1, 3, 7, 9, 2]] 54 | >>> token_indice = {(1, 3): 1337, (9, 2): 42, (4, 5): 2017, (7, 9, 2): 2018} 55 | >>> add_ngram(sequences, token_indice, ngram_range=3) 56 | [[1, 3, 4, 5, 1337], [1, 3, 7, 9, 2, 1337, 2018]] 57 | """ 58 | new_sequences = [] 59 | for input_list in sequences: 60 | new_list = input_list[:] 61 | for i in range(len(new_list) - ngram_range + 1): 62 | for ngram_value in range(2, ngram_range + 1): 63 | ngram = tuple(new_list[i:i + ngram_value]) 64 | if ngram in token_indice: 65 | new_list.append(token_indice[ngram]) 66 | new_sequences.append(new_list) 67 | 68 | return new_sequences 69 | 70 | # Set parameters: 71 | # ngram_range = 2 will add bi-grams features 72 | ngram_range = 2 73 | max_features = 20000 74 | maxlen = 400 75 | batch_size = 32 76 | embedding_dims = 50 77 | epochs = 5 78 | 79 | print('Loading data...') 80 | (x_train, y_train), (x_test, y_test) = imdb.load_data(num_words=max_features) 81 | print(len(x_train), 'train sequences') 82 | print(len(x_test), 'test sequences') 83 | print('Average train sequence length: {}'.format(np.mean(list(map(len, x_train)), dtype=int))) 84 | print('Average test sequence length: {}'.format(np.mean(list(map(len, x_test)), dtype=int))) 85 | 86 | if ngram_range > 1: 87 | print('Adding {}-gram features'.format(ngram_range)) 88 | # Create set of unique n-gram from the training set. 89 | ngram_set = set() 90 | for input_list in x_train: 91 | for i in range(2, ngram_range + 1): 92 | set_of_ngram = create_ngram_set(input_list, ngram_value=i) 93 | ngram_set.update(set_of_ngram) 94 | 95 | # Dictionary mapping n-gram token to a unique integer. 96 | # Integer values are greater than max_features in order 97 | # to avoid collision with existing features. 98 | start_index = max_features + 1 99 | token_indice = {v: k + start_index for k, v in enumerate(ngram_set)} 100 | indice_token = {token_indice[k]: k for k in token_indice} 101 | 102 | # max_features is the highest integer that could be found in the dataset. 103 | max_features = np.max(list(indice_token.keys())) + 1 104 | 105 | # Augmenting x_train and x_test with n-grams features 106 | x_train = add_ngram(x_train, token_indice, ngram_range) 107 | x_test = add_ngram(x_test, token_indice, ngram_range) 108 | print('Average train sequence length: {}'.format(np.mean(list(map(len, x_train)), dtype=int))) 109 | print('Average test sequence length: {}'.format(np.mean(list(map(len, x_test)), dtype=int))) 110 | 111 | print('Pad sequences (samples x time)') 112 | x_train = sequence.pad_sequences(x_train, maxlen=maxlen) 113 | x_test = sequence.pad_sequences(x_test, maxlen=maxlen) 114 | print('x_train shape:', x_train.shape) 115 | print('x_test shape:', x_test.shape) 116 | 117 | print('Build model...') 118 | model = Sequential() 119 | 120 | # we start off with an efficient embedding layer which maps 121 | # our vocab indices into embedding_dims dimensions 122 | model.add(Embedding(max_features, 123 | embedding_dims, 124 | input_length=maxlen)) 125 | 126 | # we add a GlobalAveragePooling1D, which will average the embeddings 127 | # of all words in the document 128 | model.add(GlobalAveragePooling1D()) 129 | 130 | # We project onto a single unit output layer, and squash it with a sigmoid: 131 | model.add(Dense(1, activation='sigmoid')) 132 | 133 | if K.backend() == 'tensorflow' and len(K.tensorflow_backend._get_available_gpus()) > 1: 134 | print("Using Multi-GPU Model") 135 | model = multi_gpu_model(model, gpus=len(K.tensorflow_backend._get_available_gpus())) 136 | 137 | model.compile(loss='binary_crossentropy', 138 | optimizer='adam', 139 | metrics=['accuracy']) 140 | 141 | model.fit(x_train, y_train, 142 | batch_size=batch_size, 143 | epochs=epochs, 144 | validation_data=(x_test, y_test), 145 | callbacks=[logger]) 146 | -------------------------------------------------------------------------------- /v2/test_files/lstm_text_generation.py: -------------------------------------------------------------------------------- 1 | '''Example script to generate text from Nietzsche's writings. 2 | 3 | At least 20 epochs are required before the generated text 4 | starts sounding coherent. 5 | 6 | It is recommended to run this script on GPU, as recurrent 7 | networks are quite computationally intensive. 8 | 9 | If you try this script on new data, make sure your corpus 10 | has at least ~100k characters. ~1M is better. 11 | ''' 12 | 13 | from __future__ import print_function 14 | from keras.models import Sequential 15 | from keras.layers import Dense, Activation 16 | from keras.layers import LSTM, CuDNNLSTM 17 | from keras.utils import multi_gpu_model 18 | from keras.optimizers import RMSprop 19 | from keras.utils.data_utils import get_file 20 | from keras import backend as K 21 | import numpy as np 22 | import random 23 | import sys 24 | 25 | from CustomCallback import EpochStatsLogger 26 | logger = EpochStatsLogger() 27 | 28 | 29 | path = get_file('nietzsche.txt', origin='https://s3.amazonaws.com/text-datasets/nietzsche.txt') 30 | text = open(path).read().lower() 31 | print('corpus length:', len(text)) 32 | 33 | chars = sorted(list(set(text))) 34 | print('total chars:', len(chars)) 35 | char_indices = dict((c, i) for i, c in enumerate(chars)) 36 | indices_char = dict((i, c) for i, c in enumerate(chars)) 37 | 38 | # cut the text in semi-redundant sequences of maxlen characters 39 | maxlen = 40 40 | step = 3 41 | sentences = [] 42 | next_chars = [] 43 | for i in range(0, len(text) - maxlen, step): 44 | sentences.append(text[i: i + maxlen]) 45 | next_chars.append(text[i + maxlen]) 46 | print('nb sequences:', len(sentences)) 47 | 48 | print('Vectorization...') 49 | X = np.zeros((len(sentences), maxlen, len(chars)), dtype=np.bool) 50 | y = np.zeros((len(sentences), len(chars)), dtype=np.bool) 51 | for i, sentence in enumerate(sentences): 52 | for t, char in enumerate(sentence): 53 | X[i, t, char_indices[char]] = 1 54 | y[i, char_indices[next_chars[i]]] = 1 55 | 56 | 57 | # build the model: a single LSTM 58 | print('Build model...') 59 | model = Sequential() 60 | if K.backend() == 'tensorflow' and len(K.tensorflow_backend._get_available_gpus()) > 0: 61 | print("Using CuDNNLSTM") 62 | model.add(CuDNNLSTM(128, input_shape=(maxlen, len(chars)))) 63 | else: 64 | model.add(LSTM(128, input_shape=(maxlen, len(chars)))) 65 | model.add(Dense(len(chars))) 66 | model.add(Activation('softmax')) 67 | 68 | optimizer = RMSprop(lr=0.01) 69 | if K.backend() == 'tensorflow' and len(K.tensorflow_backend._get_available_gpus()) > 1: 70 | print("Using Multi-GPU Model") 71 | model = multi_gpu_model(model, gpus=len(K.tensorflow_backend._get_available_gpus())) 72 | model.compile(loss='categorical_crossentropy', optimizer=optimizer) 73 | 74 | 75 | def sample(preds, temperature=1.0): 76 | # helper function to sample an index from a probability array 77 | preds = np.asarray(preds).astype('float64') 78 | preds = np.log(preds) / temperature 79 | exp_preds = np.exp(preds) 80 | preds = exp_preds / np.sum(exp_preds) 81 | probas = np.random.multinomial(1, preds, 1) 82 | return np.argmax(probas) 83 | 84 | # train the model, output generated text after each iteration 85 | for iteration in [1]: 86 | print() 87 | print('-' * 50) 88 | print('Iteration', iteration) 89 | model.fit(X, y, 90 | batch_size=128, 91 | epochs=10, 92 | callbacks=[logger]) 93 | 94 | start_index = random.randint(0, len(text) - maxlen - 1) 95 | 96 | for diversity in [0.5]: 97 | print() 98 | print('----- diversity:', diversity) 99 | 100 | generated = '' 101 | sentence = text[start_index: start_index + maxlen] 102 | generated += sentence 103 | print('----- Generating with seed: "' + sentence + '"') 104 | sys.stdout.write(generated) 105 | 106 | for i in range(400): 107 | x = np.zeros((1, maxlen, len(chars))) 108 | for t, char in enumerate(sentence): 109 | x[0, t, char_indices[char]] = 1. 110 | 111 | preds = model.predict(x, verbose=0)[0] 112 | next_index = sample(preds, diversity) 113 | next_char = indices_char[next_index] 114 | 115 | generated += next_char 116 | sentence = sentence[1:] + next_char 117 | 118 | sys.stdout.write(next_char) 119 | sys.stdout.flush() 120 | print() 121 | -------------------------------------------------------------------------------- /v2/test_files/mnist_cnn.py: -------------------------------------------------------------------------------- 1 | '''Trains a simple convnet on the MNIST dataset. 2 | 3 | Gets to 99.25% test accuracy after 12 epochs 4 | (there is still a lot of margin for parameter tuning). 5 | 16 seconds per epoch on a GRID K520 GPU. 6 | ''' 7 | 8 | from __future__ import print_function 9 | import keras 10 | from keras.datasets import mnist 11 | from keras.models import Sequential 12 | from keras.layers import Dense, Dropout, Flatten 13 | from keras.layers import Conv2D, MaxPooling2D 14 | from keras import backend as K 15 | from keras.utils import multi_gpu_model 16 | 17 | from CustomCallback import EpochStatsLogger 18 | logger = EpochStatsLogger() 19 | 20 | 21 | batch_size = 128 22 | num_classes = 10 23 | epochs = 12 24 | 25 | # input image dimensions 26 | img_rows, img_cols = 28, 28 27 | 28 | # the data, shuffled and split between train and test sets 29 | (x_train, y_train), (x_test, y_test) = mnist.load_data() 30 | 31 | if K.image_data_format() == 'channels_first': 32 | x_train = x_train.reshape(x_train.shape[0], 1, img_rows, img_cols) 33 | x_test = x_test.reshape(x_test.shape[0], 1, img_rows, img_cols) 34 | input_shape = (1, img_rows, img_cols) 35 | else: 36 | x_train = x_train.reshape(x_train.shape[0], img_rows, img_cols, 1) 37 | x_test = x_test.reshape(x_test.shape[0], img_rows, img_cols, 1) 38 | input_shape = (img_rows, img_cols, 1) 39 | 40 | x_train = x_train.astype('float32') 41 | x_test = x_test.astype('float32') 42 | x_train /= 255 43 | x_test /= 255 44 | print('x_train shape:', x_train.shape) 45 | print(x_train.shape[0], 'train samples') 46 | print(x_test.shape[0], 'test samples') 47 | 48 | # convert class vectors to binary class matrices 49 | y_train = keras.utils.to_categorical(y_train, num_classes) 50 | y_test = keras.utils.to_categorical(y_test, num_classes) 51 | 52 | model = Sequential() 53 | model.add(Conv2D(32, kernel_size=(3, 3), 54 | activation='relu', 55 | input_shape=input_shape)) 56 | model.add(Conv2D(64, (3, 3), activation='relu')) 57 | model.add(MaxPooling2D(pool_size=(2, 2))) 58 | model.add(Dropout(0.25)) 59 | model.add(Flatten()) 60 | model.add(Dense(128, activation='relu')) 61 | model.add(Dropout(0.5)) 62 | model.add(Dense(num_classes, activation='softmax')) 63 | 64 | if K.backend() == 'tensorflow' and len(K.tensorflow_backend._get_available_gpus()) > 1: 65 | print("Using Multi-GPU Model") 66 | model = multi_gpu_model(model, gpus=len(K.tensorflow_backend._get_available_gpus())) 67 | 68 | model.compile(loss=keras.losses.categorical_crossentropy, 69 | optimizer=keras.optimizers.Adadelta(), 70 | metrics=['accuracy']) 71 | 72 | model.fit(x_train, y_train, 73 | batch_size=batch_size, 74 | epochs=epochs, 75 | verbose=1, 76 | validation_data=(x_test, y_test), 77 | callbacks=[logger]) 78 | score = model.evaluate(x_test, y_test, verbose=0) 79 | print('Test loss:', score[0]) 80 | print('Test accuracy:', score[1]) 81 | -------------------------------------------------------------------------------- /v2/test_files/mnist_mlp.py: -------------------------------------------------------------------------------- 1 | '''Trains a simple deep NN on the MNIST dataset. 2 | 3 | Gets to 98.40% test accuracy after 20 epochs 4 | (there is *a lot* of margin for parameter tuning). 5 | 2 seconds per epoch on a K520 GPU. 6 | ''' 7 | 8 | from __future__ import print_function 9 | 10 | import keras 11 | from keras.datasets import mnist 12 | from keras.models import Sequential 13 | from keras.layers import Dense, Dropout 14 | from keras.optimizers import RMSprop 15 | from keras.utils import multi_gpu_model 16 | from keras import backend as K 17 | 18 | from CustomCallback import EpochStatsLogger 19 | logger = EpochStatsLogger() 20 | 21 | 22 | batch_size = 128 23 | num_classes = 10 24 | epochs = 20 25 | 26 | # the data, shuffled and split between train and test sets 27 | (x_train, y_train), (x_test, y_test) = mnist.load_data() 28 | 29 | x_train = x_train.reshape(60000, 784) 30 | x_test = x_test.reshape(10000, 784) 31 | x_train = x_train.astype('float32') 32 | x_test = x_test.astype('float32') 33 | x_train /= 255 34 | x_test /= 255 35 | print(x_train.shape[0], 'train samples') 36 | print(x_test.shape[0], 'test samples') 37 | 38 | # convert class vectors to binary class matrices 39 | y_train = keras.utils.to_categorical(y_train, num_classes) 40 | y_test = keras.utils.to_categorical(y_test, num_classes) 41 | 42 | model = Sequential() 43 | model.add(Dense(512, activation='relu', input_shape=(784,))) 44 | model.add(Dropout(0.2)) 45 | model.add(Dense(512, activation='relu')) 46 | model.add(Dropout(0.2)) 47 | model.add(Dense(10, activation='softmax')) 48 | 49 | model.summary() 50 | 51 | if K.backend() == 'tensorflow' and len(K.tensorflow_backend._get_available_gpus()) > 1: 52 | print("Using Multi-GPU Model") 53 | model = multi_gpu_model(model, gpus=len(K.tensorflow_backend._get_available_gpus())) 54 | 55 | model.compile(loss='categorical_crossentropy', 56 | optimizer=RMSprop(), 57 | metrics=['accuracy']) 58 | 59 | history = model.fit(x_train, y_train, 60 | batch_size=batch_size, 61 | epochs=epochs, 62 | verbose=1, 63 | validation_data=(x_test, y_test), 64 | callbacks=[logger]) 65 | score = model.evaluate(x_test, y_test, verbose=0) 66 | print('Test loss:', score[0]) 67 | print('Test accuracy:', score[1]) 68 | --------------------------------------------------------------------------------