├── .gitignore ├── install_vcpkg_dependencies.bat ├── readme.md ├── run_numbers.ps1 └── plot.py /.gitignore: -------------------------------------------------------------------------------- 1 | .vs 2 | x64/ 3 | measurements/ 4 | -------------------------------------------------------------------------------- /install_vcpkg_dependencies.bat: -------------------------------------------------------------------------------- 1 | vcpkg.exe install --triplet x64-windows date imgui boost-variant2 boost-optional boost-any fmt outcome glm doctest -------------------------------------------------------------------------------- /readme.md: -------------------------------------------------------------------------------- 1 | # C++ Library Include Times (CPP-LIT :fire:) 2 | This repo answers the question: how much time is added to my compile time by a single inclusion of `
`? Featuring *all* C++ Standard Library headers, their C++20 [module versions](https://docs.microsoft.com/en-us/cpp/cpp/modules-cpp?view=vs-2019), `windows.h` and a couple of other third party libraries. All times are for Visual Studio 2019 16.11.0 Preview 2.0. The red entries are new C++20 headers. 3 | 4 | ![results](https://user-images.githubusercontent.com/6044318/123312578-d5f5dc80-d528-11eb-841f-a03f76a5ff4d.png) 5 | 6 | ## Interpretation & notes 7 | The numbers are measured with care, but are easy to misinterpret. Note: 8 | 9 | - This analysis was done by including the headers into `.cpp` files. That accurately measures inclusion time, but is only a lower limit for actual **compile times**. The measurement code doesn't *do* anything with those headers. The cost of heavy template usage for example might outweigh the cost of merely including a header. 10 | 11 | - Headers can themselves include other headers. Since each header is at most included once (`#pragma once`) per TU, the include time of any header depends on whether any of its sub-includes have been included before. The worst case is when no other headers were included before and the best case is when all sub-headers were already included. This analysis compares zero includes vs the include of any header - so it's the **worst case**. This is a somewhat realistic scenario for sparse use of the standard library. Not so much when there's already a lenghty include list. 12 | 13 | - When compiling a project with **multiple translation units** and with [`/MP` enabled](https://docs.microsoft.com/en-us/cpp/build/reference/mp-build-with-multiple-processes), MSVC compiles TUs in parallel. In such cases, the include times above can be misleading. Example: the time for one `` is around half a second for one thread. During that half second, the other n-1 cores can compile other translation units, reducing the actual added include time to about *t*/*n*, with *t* being the include time as listed and *n* being the number of threads compiling. 14 | 15 | - Especially with some of the **third party libraries**, the situation is more complex than can be summarized with a single value. Some libraries offer different versions with better compile performance, like [spdlog](https://github.com/gabime/spdlog), which explicitly recommends against the use of the single-header version which was used here. Others like [GLM](https://github.com/g-truc/glm) are modularized: I used the heavy `glm.hpp` - using a smaller subset *will* be faster. This was done for simplicity and not to portray any of those libraries in a bad light. 16 | 17 | - There is no use of PCH, header units or any form of caching. The tests were done on a fast SSD and a Ryzen 3800X. All tests were done with a warmup phase and on an otherwise idle system, so real-world numbers will probably come down higher than this. 18 | 19 | - The standard library in **module** form was so fast to include that it can barely be measured or seen, at least in comparison with the rest. That's not a mistake. 20 | 21 | ## Libraries 22 | `windows.h [LAM]` refers to the common 23 | ``` 24 | #define WIN32_LEAN_AND_MEAN 25 | #include 26 | ``` 27 | 28 | - [Tracy](https://github.com/wolfpld/tracy) v0.7.8, obviously with the `TRACY_ENABLE` define. 29 | - [spdlog](https://github.com/gabime/spdlog) v1.8.5 using header-only version with only `spdlog.h` included. Note that the readme recommends to use the static lib version instead for faster compile times. 30 | - [{fmt}](https://github.com/fmtlib/fmt) v7.1.3 including only `fmt/core.h`. 31 | - [JSON for Modern C++](https://github.com/nlohmann/json) v3.9.1. Note that this is split into the main header (nl_json - `json.hpp`) and the forward include header (nl_json_fwd - `json_fwd.hpp`). The latter is what you would include more often. 32 | - [GLM](https://github.com/g-truc/glm) v0.9.9.8. GLM is modular, this repo measures the include of `glm.hpp` - which might be more than what would typically include. 33 | - [vulkan.h](https://www.lunarg.com/vulkan-sdk/) and `vulkan.hpp` (not to be confused!) from Vulkan SDK v1.2.162.1. 34 | - All boost libraries are v1.76.0. Note that [Boost.JSON](https://www.boost.org/doc/libs/1_76_0/libs/json/doc/html/index.html) is being measured in its header-only mode. 35 | - [stb](https://github.com/nothings/stb) headers are from 2020-09-14. 36 | - [EnTT](https://github.com/skypjack/entt) v3.5.0. 37 | 38 | ## Methodology 39 | All reported times are based on release builds. The complete compile command without includes is: 40 | 41 | ``` 42 | cl.exe /O2 /GL /Oi /MD /D NDEBUG /std:c++latest /experimental:module /EHsc /nologo /link /MACHINE:X64 /LTCG:incremental 43 | ``` 44 | 45 | To measure the cost of an include, the difference was taken between a build with that include and one without. All measurements were taken after a warm-up phase and with lots of data points. The plotted errors are the standard deviations of those numbers. The results are computed as a difference. Error propagation tells us the resulting standard deviation is: 46 | 47 | σ = sqrt(σ_A² + σ_B²) 48 | 49 | The sources being compiled consist of 10 identical translation units with the resulting time being divided by 10 to get the individual cost. 50 | -------------------------------------------------------------------------------- /run_numbers.ps1: -------------------------------------------------------------------------------- 1 | # $vcvars_dir = "C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Auxiliary\Build\vcvars64.bat" 2 | $vcvars_dir = "C:\Program Files (x86)\Microsoft Visual Studio\2019\Preview\VC\Auxiliary\Build\vcvars64.bat" 3 | $vcpkg_dir = "C:\inc\vcpkg-master" 4 | $tracy_dir = "..\common_libs\tracy" 5 | $vulkan_dir = "C:\VulkanSDK\1.2.162.1\Include" 6 | $boost_dir = "C:\boost_1_76_0" 7 | 8 | 9 | function Invoke-CmdScript { 10 | param( 11 | [String] $script_path 12 | ) 13 | Write-Output "running script $script_path" 14 | $cmdLine = """$script_path"" $args & set" 15 | & $Env:SystemRoot\system32\cmd.exe /c $cmdLine | 16 | select-string '^([^=]*)=(.*)$' | foreach-object { 17 | $varName = $_.Matches[0].Groups[1].Value 18 | $varValue = $_.Matches[0].Groups[2].Value 19 | set-item Env:$varName $varValue 20 | } 21 | } 22 | if((-Not (Test-Path env:cpp_lit_invoked_vcvars)) -And (-Not $env:cpp_lit_invoked_vcvars)){ 23 | Invoke-CmdScript -script_path $vcvars_dir 24 | # prevent running that script more than once per session. It's slow and there's an issue with multiple invokations 25 | $env:cpp_lit_invoked_vcvars = $true 26 | } 27 | 28 | 29 | $include_statement = "/I" + $vcpkg_dir + "\installed\x64-windows\include " + "/I" + $tracy_dir + " /I" + $vulkan_dir + " /I" + $boost_dir + " " 30 | 31 | function Setup-Tus{ 32 | Param( 33 | [Parameter(Mandatory=$true)] $tu_count 34 | ) 35 | For ($i=0; $i -lt $tu_count; $i++){ 36 | Copy-Item build_project\template.cpp -Destination ("build_project\tu{0}.cpp" -f $i) 37 | } 38 | } 39 | 40 | function del_main{ 41 | while($true){ 42 | try{ 43 | Remove-Item -Path main.* 44 | Remove-Item -Path tu*.obj 45 | return 46 | } 47 | catch{ 48 | Write-Host "error, retrying" -ForegroundColor Red 49 | } 50 | } 51 | } 52 | 53 | function Invoke-Meas{ 54 | Param( 55 | [Parameter(Mandatory=$true)] $description, 56 | [Parameter(Mandatory=$true)] $inc, 57 | [Parameter(Mandatory=$true)] $repeats, 58 | [Parameter(Mandatory=$false)] [string[]]$defines, 59 | [Parameter(Mandatory=$true)] $tu_count 60 | ) 61 | 62 | $extra_defines_str = "" 63 | foreach ($d in $defines){ 64 | $extra_defines_str += "/D " + $d + " " 65 | } 66 | $sources_str = "build_project/main.cpp" 67 | For ($i=0; $i -lt $tu_count; $i++){ 68 | $sources_str += " build_project\tu{0}.cpp" -f $i 69 | } 70 | 71 | $cl_command = "CL " + $include_statement + "/O2 /GL /Oi /MD /D NDEBUG " + $extra_defines_str + " /std:c++latest /experimental:module /EHsc /nologo " + $sources_str + " /link /MACHINE:X64 /LTCG:incremental" 72 | 73 | For ($i=0; $i -lt $repeats; $i++){ 74 | Write-Host -NoNewLine ("`rdescription: {0}, inc: {1}, defines: {2}, i:{3}/{4}" -f $description, $inc, [system.String]::Join(" ", $defines), $i, $repeats) -ForegroundColor DarkGreen 75 | if($description -eq "warmup"){ 76 | Invoke-Expression $cl_command | out-null 77 | } 78 | else{ 79 | Measure-Command { 80 | Invoke-Expression $cl_command 81 | } | Out-File -FilePath ("measurements\$description-$inc-" + $tu_count + ".txt") -Append -Encoding utf8 82 | } 83 | if(-Not(Test-Path "main.exe")){ 84 | Write-Host ("no main.exe! desc: {0}, inc: {1}. Printing output, then exiting" -f $description, $inc) -ForegroundColor Red 85 | Invoke-Expression $cl_command 86 | del_main 87 | exit 88 | } 89 | 90 | del_main 91 | } 92 | Write-Host "" 93 | } 94 | 95 | $std_headers = "algorithm","any","array","atomic","barrier","bit","bitset","cassert","cctype","cerrno","cfenv","cfloat","charconv","chrono","cinttypes","climits","clocale","cmath","compare","complex","concepts","condition_variable","coroutine","csetjmp","csignal","cstdarg","cstddef","cstdint","cstdio","cstdlib","cstring","ctime","cuchar","cwchar","cwctype","deque","exception","execution","filesystem","format","forward_list","fstream","functional","future","initializer_list","iomanip","ios","iosfwd","iostream","istream","iterator","latch","limits","list","locale","map","memory","memory_resource","mutex","new","numbers","numeric","optional","ostream","queue","random","ranges","ratio","regex","scoped_allocator","semaphore","set","shared_mutex","source_location","span","sstream","stack","stdexcept","stop_token","streambuf","string","string_view","syncstream","system_error","thread","tuple","type_traits","typeindex","typeinfo","unordered_map","unordered_set","utility","valarray","variant","vector","version" 96 | 97 | $std_modules = "std_regex","std_filesystem","std_memory","std_threading","std_core" 98 | 99 | $third_party_libs = @() 100 | # $third_party_libs += "windows","windows_mal" 101 | # $third_party_libs += "tracy" 102 | # $third_party_libs += "vulkan" 103 | # $third_party_libs += "vulkanhpp" 104 | # $third_party_libs += "spdlog" 105 | # $third_party_libs += "fmt" 106 | # $third_party_libs += "nl_json_fwd","nl_json" 107 | # $third_party_libs += "glm" 108 | # $third_party_libs += "stb_image" 109 | # $third_party_libs += "stb_image_write" 110 | $third_party_libs += "entt" 111 | 112 | # $third_party_libs += "boost_json" 113 | # $third_party_libs += "boost_variant" 114 | # $third_party_libs += "boost_variant2" 115 | 116 | 117 | # Clean measurements output dir 118 | # if(Test-Path measurements){ 119 | # Remove-Item measurements -Recurse 120 | # } 121 | new-item -Name measurements -ItemType directory -Force | out-null 122 | 123 | Write-Host "Start:" (get-date).ToString('T') -ForegroundColor DarkGreen 124 | Setup-Tus -tu_count 10 125 | 126 | # Invoke-Meas -description "warmup" -inc "warmup" -repeats 5 -defines @() -tu_count 50 127 | # Invoke-Meas -description "special" -inc "baseline" -repeats 40 -defines @("no_std") -tu_count 50 128 | # Invoke-Meas -description "special" -inc "baseline" -repeats 40 -defines @("no_std") -tu_count 50 129 | # Invoke-Meas -description "std" -inc "filesystem" -repeats 40 -defines @("no_std", "i_filesystem") -tu_count 1 130 | # Invoke-Meas -description "std" -inc "filesystem" -repeats 40 -defines @("no_std", "i_filesystem") -tu_count 1 131 | 132 | Invoke-Meas -description "warmup" -inc "warmup" -repeats 40 -defines @() -tu_count 10 133 | # Invoke-Meas -description "special" -inc "baseline" -repeats 40 -defines @() -tu_count 10 134 | 135 | $normal_repeat_n = 10 136 | 137 | 138 | 139 | # Foreach($header in $std_headers){ 140 | # $def = "i_{0}" -f $header 141 | # Invoke-Meas -description "std" -inc $header -repeats $normal_repeat_n -defines @($def) -tu_count 10 142 | # } 143 | 144 | # Foreach($header in $std_modules){ 145 | # $def = "i_{0}" -f $header 146 | # Invoke-Meas -description "std_modules" -inc $header -repeats $normal_repeat_n -defines @($def) -tu_count 10 147 | # } 148 | 149 | 150 | Foreach($header in $third_party_libs){ 151 | $def = "i_{0}" -f $header 152 | Invoke-Meas -description "third_party" -inc $header -repeats $normal_repeat_n -defines @($def) -tu_count 10 153 | } 154 | 155 | Write-Host "End:" (get-date).ToString('T') -ForegroundColor DarkGreen 156 | -------------------------------------------------------------------------------- /plot.py: -------------------------------------------------------------------------------- 1 | import pathlib 2 | import re 3 | import json 4 | from collections import defaultdict 5 | import numpy as np 6 | import matplotlib as mpl 7 | import matplotlib.pyplot as plt 8 | from collections import namedtuple 9 | from scipy.stats import norm 10 | 11 | Measurement = namedtuple('Measurement', ['mean', 'std']) 12 | 13 | 14 | def get_2sigma_filtered(data): 15 | mu, std = norm.fit(data) 16 | lower = mu - 2*std 17 | upper = mu + 2*std 18 | data = data[(data < upper) & (lower < data)] 19 | return data 20 | 21 | 22 | # Mean absolute deviation https://en.wikipedia.org/wiki/Average_absolute_deviation#Mean_absolute_deviation_around_the_mean 23 | def get_MAD(data): 24 | mean = np.mean(data) 25 | return np.sum(np.abs(data-mean)) / data.size 26 | 27 | 28 | def get_times_from_txt(filename, ignore_count): 29 | ms_regex = re.compile(r'TotalMilliseconds\s?:\s?(\d*[,.]\d*)') 30 | with open(filename) as file: 31 | times = np.zeros(shape=(0)) 32 | for line in file.readlines(): 33 | match = ms_regex.match(line.strip()) 34 | if match is None: 35 | continue 36 | if ignore_count > 0: 37 | ignore_count -= 1 38 | continue 39 | ms_str = match.group(1).replace(",", ".") 40 | ms = float(ms_str) 41 | times = np.append(times, ms) 42 | times = get_2sigma_filtered(times) 43 | return times 44 | 45 | def get_time_and_std_from_txt(filename, ignore_count): 46 | times = get_times_from_txt(filename, ignore_count) 47 | return Measurement(np.mean(times), np.std(times, dtype=np.float64)) 48 | 49 | 50 | def get_file_data(ignore_count=1): 51 | measurement_fn_regex = re.compile(r'([\w_]+)-([\w_]+)-(\d+)\.txt') 52 | file_data = defaultdict(lambda: defaultdict(lambda: defaultdict(lambda: defaultdict(int)))) 53 | for path in pathlib.Path('measurements').iterdir(): 54 | if not path.is_file(): 55 | continue 56 | filename = path.name 57 | match = measurement_fn_regex.match(filename) 58 | if match is None: 59 | continue 60 | category, lib, tu_count = match.group(1, 2, 3) 61 | if category == "std_modules": 62 | lib = lib.replace("std_", "std.") 63 | tu_count = int(tu_count) 64 | file_data[category][lib] = get_time_and_std_from_txt(path, ignore_count) 65 | # print(json.dumps(file_data, indent=4)) 66 | return file_data 67 | 68 | 69 | def get_pretty_name(description): 70 | replacements = { 71 | "windows": "windows.h", 72 | "windows_mal": "windows.h [LAM]", 73 | "tracy": "Tracy.hpp", 74 | "doctest": "doctest/doctest.h", 75 | "spdlog": "spdlog/spdlog.h", 76 | "fmt": "fmt/core.h", 77 | "nl_json": "nlohmann/json.hpp", 78 | "nl_json_fwd": "nlohmann/json_fwd.hpp", 79 | "glm": "glm/glm.hpp", 80 | "vulkan": "vulkan/vulkan.h", 81 | "vulkanhpp": "vulkan/vulkan.hpp", 82 | "boost_json": "boost/json.hpp", 83 | "boost_variant": "boost/variant.hpp", 84 | "boost_variant2": "boost/variant2/variant.hpp", 85 | "stb_image": "stb_image.h", 86 | "stb_image_write": "stb_image_write.h", 87 | "entt": "entt/entt.hpp" 88 | } 89 | if description in replacements.keys(): 90 | return replacements[description] 91 | else: 92 | return description 93 | 94 | def get_raw_labels(categories, file_data): 95 | labels = [] 96 | for category in categories: 97 | for name in file_data[category].keys(): 98 | labels.append(name) 99 | return labels 100 | 101 | def get_labels(raw_labels): 102 | labels = [] 103 | is_std = True 104 | for raw_label in raw_labels: 105 | if is_std: 106 | raw_label = "<{}>".format(raw_label) 107 | labels.append(get_pretty_name(raw_label)) 108 | if raw_label == "": 109 | is_std = False 110 | max_len = max([len(label) for label in labels]) 111 | labels = [(label+" ").ljust(max_len+1, '—') for label in labels] 112 | 113 | return labels 114 | 115 | 116 | def get_positions(categories, file_data): 117 | current_pos = 0.0 118 | positions = [] 119 | for category in categories: 120 | for _ in file_data[category].keys(): 121 | positions.append(current_pos) 122 | current_pos -= 1.0 123 | current_pos -= 1.0 124 | 125 | return np.array(positions) 126 | 127 | 128 | def get_addition_error(a, b): 129 | return np.sqrt(a*a + b*b) 130 | 131 | tu_count = 10 132 | cpp_20_headers = ["concepts", "coroutines", "compare", "version", "source_location", "format", "semaphore", "span", "ranges", "bit", "numbers", "syncstream", "stop_token", "latch", "barrier"] 133 | 134 | def get_worst(category, file_data): 135 | sort_data = np.empty([0, 2]) 136 | for res in file_data[category].values(): 137 | worst_time = (res.mean - file_data["special"]["baseline"].mean) / tu_count 138 | worst_time_std = get_addition_error(res.std/tu_count, file_data["special"]["baseline"].std/tu_count) 139 | sort_data = np.append(sort_data, np.array([[worst_time, worst_time_std]]), axis=0) 140 | return sort_data 141 | 142 | 143 | def main_plot(): 144 | file_data = get_file_data() 145 | categories = ["std", "std_modules", "third_party"] 146 | raw_labels = get_raw_labels(categories, file_data) 147 | labels = get_labels(raw_labels) 148 | positions = get_positions(categories, file_data) 149 | 150 | worst_data = np.empty([0, 2]) 151 | for category in categories: 152 | worst_data = np.append(worst_data, get_worst(category, file_data), axis=0) 153 | 154 | max_pos = np.max(np.abs(positions)) 155 | fig = plt.figure(figsize=(10, 2 + 0.12 * max_pos)) 156 | ax = fig.add_subplot() 157 | 158 | bar_height = 0.3 159 | 160 | _ = ax.barh(y=positions, width=worst_data[:, 0], height=bar_height, color="tab:orange") 161 | _ = ax.barh(y=positions, left=worst_data[:, 0] - worst_data[:, 1], width=2 * worst_data[:, 1], height=bar_height/2, color="blue", alpha=0.5) 162 | 163 | _ = plt.yticks(positions, labels, fontfamily="monospace", horizontalalignment='left') 164 | 165 | for i, label in enumerate(raw_labels): 166 | if label in cpp_20_headers: 167 | ax.get_yticklabels()[i].set_color("red") 168 | 169 | ax.grid(axis='x', alpha=0.2) 170 | ax.get_yaxis().set_tick_params(pad=160) 171 | ax.get_yaxis().set_tick_params(length=0) 172 | ax.set_axisbelow(True) 173 | ax.spines["right"].set_visible(False) 174 | ax.spines["top"].set_visible(False) 175 | ax.spines["bottom"].set_visible(False) 176 | ax.spines["left"].set_visible(False) 177 | 178 | _ = ax.axvline(x=0, linewidth=0.5, color="black", zorder=1) 179 | _ = ax.set_xlabel("Include time [ms]") 180 | ax.margins(0) 181 | _ = ax.set_ylim(ax.get_ylim()[0]-1, ax.get_ylim()[1]+1) 182 | 183 | ax2 = ax.twiny() 184 | _ = ax2.set_xlabel("Include time [ms]") 185 | ax2.set_xlim(ax.get_xlim()) 186 | ax2.spines = ax.spines 187 | 188 | fig.tight_layout() 189 | fig.savefig("lit.png") 190 | 191 | 192 | def disable_top_right_spines(ax): 193 | ax.spines["right"].set_visible(False) 194 | ax.spines["top"].set_visible(False) 195 | 196 | 197 | if __name__ == "__main__": 198 | main_plot() 199 | --------------------------------------------------------------------------------