├── .gitignore ├── Benchmarkable.Test ├── BenchmarkTests.cs ├── Benchmarkable.Test.csproj └── packages.config ├── Benchmarkable.sln ├── Benchmarkable ├── Benchmark.cs ├── BenchmarkResults.cs ├── BenchmarkSettings.cs ├── Benchmarkable.csproj ├── Benchmarkable.nuspec ├── JustClause.cs ├── Properties │ └── AssemblyInfo.cs ├── Result.cs ├── Run.cs ├── TDistribution.cs ├── ThisClause.cs └── packages.config ├── Examples ├── Examples.csproj ├── Program.cs ├── Properties │ └── AssemblyInfo.cs ├── app.config └── packages.config ├── LICENSE └── README.md /.gitignore: -------------------------------------------------------------------------------- 1 | ## Ignore Visual Studio temporary files, build results, and 2 | ## files generated by popular Visual Studio add-ons. 3 | 4 | # User-specific files 5 | *.suo 6 | *.user 7 | *.userosscache 8 | *.sln.docstates 9 | 10 | # User-specific files (MonoDevelop/Xamarin Studio) 11 | *.userprefs 12 | 13 | # Build results 14 | [Dd]ebug/ 15 | [Dd]ebugPublic/ 16 | [Rr]elease/ 17 | [Rr]eleases/ 18 | x64/ 19 | x86/ 20 | bld/ 21 | [Bb]in/ 22 | [Oo]bj/ 23 | [Ll]og/ 24 | 25 | # Visual Studio 2015 cache/options directory 26 | .vs/ 27 | # Uncomment if you have tasks that create the project's static files in wwwroot 28 | #wwwroot/ 29 | 30 | # MSTest test Results 31 | [Tt]est[Rr]esult*/ 32 | [Bb]uild[Ll]og.* 33 | 34 | # NUNIT 35 | *.VisualState.xml 36 | TestResult.xml 37 | 38 | # Build Results of an ATL Project 39 | [Dd]ebugPS/ 40 | [Rr]eleasePS/ 41 | dlldata.c 42 | 43 | # DNX 44 | project.lock.json 45 | artifacts/ 46 | 47 | *_i.c 48 | *_p.c 49 | *_i.h 50 | *.ilk 51 | *.meta 52 | *.obj 53 | *.pch 54 | *.pdb 55 | *.pgc 56 | *.pgd 57 | *.rsp 58 | *.sbr 59 | *.tlb 60 | *.tli 61 | *.tlh 62 | *.tmp 63 | *.tmp_proj 64 | *.log 65 | *.vspscc 66 | *.vssscc 67 | .builds 68 | *.pidb 69 | *.svclog 70 | *.scc 71 | 72 | # Chutzpah Test files 73 | _Chutzpah* 74 | 75 | # Visual C++ cache files 76 | ipch/ 77 | *.aps 78 | *.ncb 79 | *.opendb 80 | *.opensdf 81 | *.sdf 82 | *.cachefile 83 | *.VC.db 84 | *.VC.VC.opendb 85 | 86 | # Visual Studio profiler 87 | *.psess 88 | *.vsp 89 | *.vspx 90 | *.sap 91 | 92 | # TFS 2012 Local Workspace 93 | $tf/ 94 | 95 | # Guidance Automation Toolkit 96 | *.gpState 97 | 98 | # ReSharper is a .NET coding add-in 99 | _ReSharper*/ 100 | *.[Rr]e[Ss]harper 101 | *.DotSettings.user 102 | 103 | # JustCode is a .NET coding add-in 104 | .JustCode 105 | 106 | # TeamCity is a build add-in 107 | _TeamCity* 108 | 109 | # DotCover is a Code Coverage Tool 110 | *.dotCover 111 | 112 | # NCrunch 113 | _NCrunch_* 114 | .*crunch*.local.xml 115 | nCrunchTemp_* 116 | 117 | # MightyMoose 118 | *.mm.* 119 | AutoTest.Net/ 120 | 121 | # Web workbench (sass) 122 | .sass-cache/ 123 | 124 | # Installshield output folder 125 | [Ee]xpress/ 126 | 127 | # DocProject is a documentation generator add-in 128 | DocProject/buildhelp/ 129 | DocProject/Help/*.HxT 130 | DocProject/Help/*.HxC 131 | DocProject/Help/*.hhc 132 | DocProject/Help/*.hhk 133 | DocProject/Help/*.hhp 134 | DocProject/Help/Html2 135 | DocProject/Help/html 136 | 137 | # Click-Once directory 138 | publish/ 139 | 140 | # Publish Web Output 141 | *.[Pp]ublish.xml 142 | *.azurePubxml 143 | # TODO: Comment the next line if you want to checkin your web deploy settings 144 | # but database connection strings (with potential passwords) will be unencrypted 145 | *.pubxml 146 | *.publishproj 147 | 148 | # Microsoft Azure Web App publish settings. Comment the next line if you want to 149 | # checkin your Azure Web App publish settings, but sensitive information contained 150 | # in these scripts will be unencrypted 151 | PublishScripts/ 152 | 153 | # NuGet Packages 154 | *.nupkg 155 | # The packages folder can be ignored because of Package Restore 156 | **/packages/* 157 | # except build/, which is used as an MSBuild target. 158 | !**/packages/build/ 159 | # Uncomment if necessary however generally it will be regenerated when needed 160 | #!**/packages/repositories.config 161 | # NuGet v3's project.json files produces more ignoreable files 162 | *.nuget.props 163 | *.nuget.targets 164 | 165 | # Microsoft Azure Build Output 166 | csx/ 167 | *.build.csdef 168 | 169 | # Microsoft Azure Emulator 170 | ecf/ 171 | rcf/ 172 | 173 | # Windows Store app package directories and files 174 | AppPackages/ 175 | BundleArtifacts/ 176 | Package.StoreAssociation.xml 177 | _pkginfo.txt 178 | 179 | # Visual Studio cache files 180 | # files ending in .cache can be ignored 181 | *.[Cc]ache 182 | # but keep track of directories ending in .cache 183 | !*.[Cc]ache/ 184 | 185 | # Others 186 | ClientBin/ 187 | ~$* 188 | *~ 189 | *.dbmdl 190 | *.dbproj.schemaview 191 | *.pfx 192 | *.publishsettings 193 | node_modules/ 194 | orleans.codegen.cs 195 | 196 | # Since there are multiple workflows, uncomment next line to ignore bower_components 197 | # (https://github.com/github/gitignore/pull/1529#issuecomment-104372622) 198 | #bower_components/ 199 | 200 | # RIA/Silverlight projects 201 | Generated_Code/ 202 | 203 | # Backup & report files from converting an old project file 204 | # to a newer Visual Studio version. Backup files are not needed, 205 | # because we have git ;-) 206 | _UpgradeReport_Files/ 207 | Backup*/ 208 | UpgradeLog*.XML 209 | UpgradeLog*.htm 210 | 211 | # SQL Server files 212 | *.mdf 213 | *.ldf 214 | 215 | # Business Intelligence projects 216 | *.rdl.data 217 | *.bim.layout 218 | *.bim_*.settings 219 | 220 | # Microsoft Fakes 221 | FakesAssemblies/ 222 | 223 | # GhostDoc plugin setting file 224 | *.GhostDoc.xml 225 | 226 | # Node.js Tools for Visual Studio 227 | .ntvs_analysis.dat 228 | 229 | # Visual Studio 6 build log 230 | *.plg 231 | 232 | # Visual Studio 6 workspace options file 233 | *.opt 234 | 235 | # Visual Studio LightSwitch build output 236 | **/*.HTMLClient/GeneratedArtifacts 237 | **/*.DesktopClient/GeneratedArtifacts 238 | **/*.DesktopClient/ModelManifest.xml 239 | **/*.Server/GeneratedArtifacts 240 | **/*.Server/ModelManifest.xml 241 | _Pvt_Extensions 242 | 243 | # Paket dependency manager 244 | .paket/paket.exe 245 | paket-files/ 246 | 247 | # FAKE - F# Make 248 | .fake/ 249 | 250 | # JetBrains Rider 251 | .idea/ 252 | *.sln.iml 253 | -------------------------------------------------------------------------------- /Benchmarkable.Test/BenchmarkTests.cs: -------------------------------------------------------------------------------- 1 | using Benchmarkable; 2 | using NUnit.Framework; 3 | using System; 4 | using System.Threading; 5 | 6 | namespace BenchmarkIt.Test 7 | { 8 | [TestFixture ()] 9 | public class Test 10 | { 11 | [Test ()] 12 | public void TestSingle () 13 | { 14 | var result = Benchmark.Just.This(() => {}); 15 | 16 | Assert.IsTrue(result.Length == 1); 17 | Assert.AreEqual("Test 1", result[0].Label); 18 | Assert.Greater(result[0].OperationsPerSecond, 00); 19 | Assert.Greater(result[0].Variance, 0); 20 | Assert.Greater(result[0].BatchSize, 0); 21 | Assert.Greater(result[0].Error, 0); 22 | 23 | 24 | // Make sure this doesn't throw an exception 25 | result.Print(); 26 | } 27 | 28 | [Test()] 29 | public void TestSingleWithException() 30 | { 31 | Assert.Catch(() => Benchmark.Just.This(() => { throw new NullReferenceException(); })); 32 | } 33 | 34 | [Test()] 35 | public void ThrowsExceptionIfNoMethods() 36 | { 37 | var bench = new Benchmark(); 38 | 39 | Assert.Catch(() => bench.Run()); 40 | } 41 | 42 | [Test()] 43 | public void TestSingleWithLabel() 44 | { 45 | var result = Benchmark.Just.This(() => { }, "My test"); 46 | 47 | 48 | Assert.IsTrue(result.Length == 1); 49 | Assert.AreEqual("My test", result[0].Label); 50 | Assert.IsTrue(result[0].OperationsPerSecond > 0); 51 | Assert.IsTrue(result[0].Variance > 0); 52 | } 53 | 54 | [Test ()] 55 | public void TestDouble () 56 | { 57 | var start = DateTime.Now; 58 | var result = Benchmark.This(() => { }) 59 | .Against(() => { Thread.Sleep(250); }); 60 | 61 | Assert.AreEqual("Test 1", result[0].Label); 62 | Assert.AreEqual("Test 2", result[1].Label); 63 | Assert.Greater(result[0].OperationsPerSecond, result[1].OperationsPerSecond); 64 | } 65 | 66 | [Test()] 67 | public void TestDoubleWithLabel() 68 | { 69 | var result = Benchmark.This(() => { }, "my test 1") 70 | .Against(() => { Thread.Sleep(250); }, "my test 2"); 71 | 72 | Assert.AreEqual("my test 1", result[0].Label); 73 | Assert.AreEqual("my test 2", result[1].Label); 74 | Assert.IsTrue(result[0].OperationsPerSecond > result[1].OperationsPerSecond); 75 | } 76 | 77 | [Test()] 78 | public void TestMultipleWithLabels() 79 | { 80 | var result = Benchmark.These(new (Action, string)[] 81 | { 82 | (() => { }, "my test 1" ), 83 | (() => { }, "my test 2" ), 84 | (() => { }, "my test 3" ), 85 | }); 86 | 87 | Assert.AreEqual(result.Length, 3); 88 | Assert.Greater(result[0].OperationsPerSecond, 0); 89 | Assert.Greater(result[1].OperationsPerSecond, 0); 90 | Assert.Greater(result[2].OperationsPerSecond, 0); 91 | Assert.AreEqual("my test 1", result[0].Label); 92 | Assert.AreEqual("my test 2", result[1].Label); 93 | Assert.AreEqual("my test 3", result[2].Label); 94 | } 95 | 96 | [Test()] 97 | public void TestExplicitUse() 98 | { 99 | var benchmark = new Benchmark(); 100 | benchmark.Add(() => { }, "my test 1"); 101 | 102 | var result = benchmark.Run(); 103 | 104 | Assert.AreEqual(result.Length, 1); 105 | Assert.Greater(result[0].OperationsPerSecond, 0); 106 | Assert.AreEqual("my test 1", result[0].Label); 107 | } 108 | } 109 | } 110 | 111 | -------------------------------------------------------------------------------- /Benchmarkable.Test/Benchmarkable.Test.csproj: -------------------------------------------------------------------------------- 1 |  2 | 3 | 4 | Debug 5 | AnyCPU 6 | {37F4CA2B-D3B1-4F73-B0BC-2F76CB2453A4} 7 | Library 8 | BenchmarkIt.Test 9 | BenchmarkIt.Test 10 | v4.6.1 11 | 12 | 13 | 14 | true 15 | full 16 | false 17 | bin\Debug 18 | DEBUG; 19 | prompt 20 | 4 21 | false 22 | 23 | 24 | full 25 | true 26 | bin\Release 27 | prompt 28 | 4 29 | false 30 | 31 | 32 | 33 | ..\packages\NUnit.3.6.1\lib\net45\nunit.framework.dll 34 | 35 | 36 | 37 | ..\packages\System.ValueTuple.4.3.0\lib\netstandard1.0\System.ValueTuple.dll 38 | 39 | 40 | 41 | 42 | 43 | 44 | 45 | 46 | {68B3ADDC-3F9E-4205-A471-21D553966F7F} 47 | Benchmarkable 48 | 49 | 50 | 51 | 52 | 53 | 54 | 55 | 56 | -------------------------------------------------------------------------------- /Benchmarkable.Test/packages.config: -------------------------------------------------------------------------------- 1 |  2 | 3 | 4 | 5 | -------------------------------------------------------------------------------- /Benchmarkable.sln: -------------------------------------------------------------------------------- 1 |  2 | Microsoft Visual Studio Solution File, Format Version 12.00 3 | # Visual Studio 15 4 | VisualStudioVersion = 15.0.26403.7 5 | MinimumVisualStudioVersion = 10.0.40219.1 6 | Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Examples", "Examples\Examples.csproj", "{8B4B91C2-69C0-4A28-B2DB-F2B4F58C49B3}" 7 | EndProject 8 | Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Benchmarkable", "Benchmarkable\Benchmarkable.csproj", "{68B3ADDC-3F9E-4205-A471-21D553966F7F}" 9 | EndProject 10 | Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Benchmarkable.Test", "Benchmarkable.Test\Benchmarkable.Test.csproj", "{37F4CA2B-D3B1-4F73-B0BC-2F76CB2453A4}" 11 | EndProject 12 | Global 13 | GlobalSection(SolutionConfigurationPlatforms) = preSolution 14 | Debug|Any CPU = Debug|Any CPU 15 | Debug|x86 = Debug|x86 16 | Release|Any CPU = Release|Any CPU 17 | Release|x86 = Release|x86 18 | EndGlobalSection 19 | GlobalSection(ProjectConfigurationPlatforms) = postSolution 20 | {8B4B91C2-69C0-4A28-B2DB-F2B4F58C49B3}.Debug|Any CPU.ActiveCfg = Debug|Any CPU 21 | {8B4B91C2-69C0-4A28-B2DB-F2B4F58C49B3}.Debug|Any CPU.Build.0 = Debug|Any CPU 22 | {8B4B91C2-69C0-4A28-B2DB-F2B4F58C49B3}.Debug|x86.ActiveCfg = Debug|x86 23 | {8B4B91C2-69C0-4A28-B2DB-F2B4F58C49B3}.Debug|x86.Build.0 = Debug|x86 24 | {8B4B91C2-69C0-4A28-B2DB-F2B4F58C49B3}.Release|Any CPU.ActiveCfg = Release|Any CPU 25 | {8B4B91C2-69C0-4A28-B2DB-F2B4F58C49B3}.Release|Any CPU.Build.0 = Release|Any CPU 26 | {8B4B91C2-69C0-4A28-B2DB-F2B4F58C49B3}.Release|x86.ActiveCfg = Release|x86 27 | {8B4B91C2-69C0-4A28-B2DB-F2B4F58C49B3}.Release|x86.Build.0 = Release|x86 28 | {68B3ADDC-3F9E-4205-A471-21D553966F7F}.Debug|Any CPU.ActiveCfg = Debug|Any CPU 29 | {68B3ADDC-3F9E-4205-A471-21D553966F7F}.Debug|Any CPU.Build.0 = Debug|Any CPU 30 | {68B3ADDC-3F9E-4205-A471-21D553966F7F}.Debug|x86.ActiveCfg = Debug|x86 31 | {68B3ADDC-3F9E-4205-A471-21D553966F7F}.Debug|x86.Build.0 = Debug|x86 32 | {68B3ADDC-3F9E-4205-A471-21D553966F7F}.Release|Any CPU.ActiveCfg = Release|Any CPU 33 | {68B3ADDC-3F9E-4205-A471-21D553966F7F}.Release|Any CPU.Build.0 = Release|Any CPU 34 | {68B3ADDC-3F9E-4205-A471-21D553966F7F}.Release|x86.ActiveCfg = Release|x86 35 | {68B3ADDC-3F9E-4205-A471-21D553966F7F}.Release|x86.Build.0 = Release|x86 36 | {37F4CA2B-D3B1-4F73-B0BC-2F76CB2453A4}.Debug|Any CPU.ActiveCfg = Debug|Any CPU 37 | {37F4CA2B-D3B1-4F73-B0BC-2F76CB2453A4}.Debug|Any CPU.Build.0 = Debug|Any CPU 38 | {37F4CA2B-D3B1-4F73-B0BC-2F76CB2453A4}.Debug|x86.ActiveCfg = Debug|Any CPU 39 | {37F4CA2B-D3B1-4F73-B0BC-2F76CB2453A4}.Debug|x86.Build.0 = Debug|Any CPU 40 | {37F4CA2B-D3B1-4F73-B0BC-2F76CB2453A4}.Release|Any CPU.ActiveCfg = Release|Any CPU 41 | {37F4CA2B-D3B1-4F73-B0BC-2F76CB2453A4}.Release|Any CPU.Build.0 = Release|Any CPU 42 | {37F4CA2B-D3B1-4F73-B0BC-2F76CB2453A4}.Release|x86.ActiveCfg = Release|Any CPU 43 | {37F4CA2B-D3B1-4F73-B0BC-2F76CB2453A4}.Release|x86.Build.0 = Release|Any CPU 44 | EndGlobalSection 45 | GlobalSection(SolutionProperties) = preSolution 46 | HideSolutionNode = FALSE 47 | EndGlobalSection 48 | GlobalSection(MonoDevelopProperties) = preSolution 49 | StartupItem = Examples\Examples.csproj 50 | EndGlobalSection 51 | EndGlobal 52 | -------------------------------------------------------------------------------- /Benchmarkable/Benchmark.cs: -------------------------------------------------------------------------------- 1 | using System; 2 | using System.Collections.Generic; 3 | using System.Diagnostics; 4 | using System.Linq; 5 | 6 | namespace Benchmarkable 7 | { 8 | /// 9 | /// Provides benchmarks of functions 10 | /// 11 | public class Benchmark 12 | { 13 | private List<(Action action, string label)> actions = new List<(Action, string)>(); 14 | private BenchmarkSettings settings; 15 | 16 | /// 17 | /// The default settings used to configure a benchmark. 18 | /// 19 | public static BenchmarkSettings Settings { get; } = new BenchmarkSettings(); 20 | 21 | /// 22 | /// Create a new benchmark 23 | /// 24 | public Benchmark() 25 | { 26 | settings = Settings; 27 | } 28 | 29 | /// 30 | /// Create a new benchmark with the given settings 31 | /// 32 | /// 33 | public Benchmark(BenchmarkSettings settings) 34 | { 35 | this.settings = settings; 36 | } 37 | 38 | /// 39 | /// Run the configured benchmark and return the results 40 | /// 41 | /// Results of the benchmark 42 | /// Thrown when no methods have been added to benchmark 43 | public BenchmarkResults Run() 44 | { 45 | if (actions.Count == 0) 46 | { 47 | throw new ArgumentOutOfRangeException("Can not run a benchmark when no methods to benchmark have been added. Call Add(Action) first"); 48 | } 49 | 50 | var results = actions.Select(x => RunAction(x.action, x.label)); 51 | return new BenchmarkResults(results.ToArray()); 52 | } 53 | 54 | /// 55 | /// Benchmark just one method 56 | /// 57 | /// Benchmark.Just.This(()=> {}, "my benchmark") 58 | public static JustClause Just { get; } = new JustClause(); 59 | 60 | /// 61 | /// Benchmark two methods against each other. 62 | /// 63 | /// First method 64 | /// A label for this method 65 | /// 66 | /// Benchmark.This(method).Against(otherMethod) 67 | /// 68 | /// 69 | public static ThisClause This(Action action, string label = null) 70 | { 71 | var benchmark = new Benchmark(); 72 | benchmark.Add(action, label); 73 | return new ThisClause(benchmark); 74 | } 75 | 76 | /// 77 | /// Benchmark a number of methods against each other 78 | /// 79 | /// An array of methods to benchmark. Each method requires a label 80 | /// 81 | /// Benchmark.These(new (Action, string)[] 82 | /// { 83 | /// (() => { }, "my test 1" ), 84 | /// (() => { }, "my test 2" ), 85 | /// (() => { }, "my test 3" ), 86 | /// }) 87 | /// 88 | /// 89 | public static BenchmarkResults These((Action, string)[] actions) 90 | { 91 | var benchmark = new Benchmark(); 92 | foreach (var action in actions) 93 | { 94 | benchmark.Add(action.Item1, action.Item2); 95 | } 96 | return benchmark.Run(); 97 | } 98 | 99 | /// 100 | /// Add an action to benchmark 101 | /// 102 | /// Add(() => Math.Sqrt(1.23), "Sqare root benchmark") 103 | /// The method to benchmark 104 | /// A name for this benchmark 105 | public void Add(Action action, string label = null) 106 | { 107 | actions.Add( 108 | (action, 109 | label ?? $"Test {actions.Count + 1}")); 110 | } 111 | 112 | private Result RunAction(Action action, string label) 113 | { 114 | // Inspiration taken fro 115 | // http://monsur.hossa.in/2012/12/11/benchmarkjs.html 116 | // http://ejohn.org/blog/javascript-benchmark-quality/ 117 | // http://ejohn.org/apps/measure/ 118 | 119 | var (batchSize, batchTime) = CalculateBatchSize(action); 120 | 121 | double error = double.MaxValue; 122 | var runTicks = new List(); 123 | var runs = new List(); 124 | var start = DateTime.Now; 125 | 126 | // Always run it 2 times at least so we can calculate the standard deviation 127 | while ((error > settings.MinimumErrorToAccept || runTicks.Count <= 2)) 128 | { 129 | // Give the test as good a chance as possible of avoiding garbage collection 130 | GC.Collect(); 131 | GC.WaitForPendingFinalizers(); 132 | GC.Collect(); 133 | 134 | var sw = Stopwatch.StartNew(); 135 | for (var i = 0; i < batchSize; i++) 136 | { 137 | action(); 138 | } 139 | sw.Stop(); 140 | 141 | runTicks.Add(sw.ElapsedTicks); 142 | var runStatistics = CalculateRunStatistics(runTicks, batchSize); 143 | error = runStatistics.Error; 144 | 145 | if ((DateTime.Now - start).TotalMilliseconds > settings.MaxTime) 146 | { 147 | runStatistics.ExceededMaxTime = true; 148 | runs.Add(runStatistics); 149 | break; 150 | } 151 | 152 | runs.Add(runStatistics); 153 | Write($"Error: {error}\tS.D: {runStatistics.StandardDeviation}"); 154 | } 155 | 156 | Write($"Time ran for: {(DateTime.Now - start).TotalMilliseconds}"); 157 | 158 | var result = new Result() 159 | { 160 | Label = label, 161 | BatchSize = batchSize, 162 | BatchTime = batchTime, 163 | Runs = runs 164 | }; 165 | return result; 166 | } 167 | 168 | /// 169 | /// Calculate statistics for the current series of batch timings 170 | /// 171 | /// 172 | /// 173 | /// 174 | private Run CalculateRunStatistics(List values, int batchSize) 175 | { 176 | // Only use the last 10 values 177 | var valuesToUse = values.Skip(Math.Max(0, values.Count() - settings.BatchesToWorkAcross)).ToArray(); 178 | var ticks = values.Last(); 179 | 180 | // TODO: Not shitty implementation, use a rolling average and standard deviation 181 | double mean = valuesToUse.Average(); 182 | double variance = valuesToUse.Average(v => Math.Pow(v - mean, 2)); 183 | double deviation = Math.Sqrt(variance); 184 | 185 | double msSpentOnRun = ticks / (double)Stopwatch.Frequency * 1000d; 186 | double upgradeFactor = 1000 / msSpentOnRun; 187 | double operationsPerSecond = batchSize * upgradeFactor; 188 | 189 | int currTDistributionValue = Math.Min(settings.BatchesToWorkAcross, values.Count()); 190 | double standardErrorsMean = deviation / Math.Sqrt(valuesToUse.Count()) * TDistribution.Values[currTDistributionValue]; 191 | double error = Math.Max((standardErrorsMean / mean) * 100, 0); 192 | 193 | return new Run() 194 | { 195 | MeanTicks = mean, 196 | StandardDeviation = deviation, 197 | Variance = variance, 198 | StandardErrorsMean = standardErrorsMean, 199 | Error = error, 200 | OperationsPerSecond = operationsPerSecond, 201 | Ticks = ticks 202 | }; 203 | } 204 | 205 | /// 206 | /// Calculates how many times we should run our action per batch 207 | /// 208 | private (int batchSize, int batchTime) CalculateBatchSize(Action action) 209 | { 210 | int count = 0; 211 | var sw = Stopwatch.StartNew(); 212 | while (sw.ElapsedMilliseconds < settings.InitialBatchTime) 213 | { 214 | action(); 215 | count++; 216 | } 217 | 218 | // TODO: Calculate a good batch size based on how long we're taking to run 219 | // So slow running actions don't need to be run many times, but fast might need 220 | // running a lot for a shorter period of time. For now just give back whatevers in the settings 221 | // Potential approach could be to run for BatchSizeTime and if the number is below a threshold we give back some 222 | // other number 223 | return (count, settings.InitialBatchTime); 224 | } 225 | 226 | private void Write(string message) 227 | { 228 | if (settings.Verbose) 229 | { 230 | Console.WriteLine(message); 231 | } 232 | } 233 | } 234 | 235 | 236 | } 237 | 238 | -------------------------------------------------------------------------------- /Benchmarkable/BenchmarkResults.cs: -------------------------------------------------------------------------------- 1 | using System; 2 | using System.Collections.Generic; 3 | using System.Linq; 4 | 5 | namespace Benchmarkable 6 | { 7 | public class BenchmarkResults 8 | { 9 | private Result[] results; 10 | 11 | internal BenchmarkResults(Result[] results) 12 | { 13 | this.results = results; 14 | } 15 | 16 | public Result this[int i] 17 | { 18 | get 19 | { 20 | return results[i]; 21 | } 22 | } 23 | 24 | public int Length { get { return results.Length; } } 25 | 26 | // TODO: Clean this up, it's gross 27 | public void Print() 28 | { 29 | var fastest = results.OrderBy(x => x.OperationsPerSecond).Last(); 30 | 31 | var output = new List(); 32 | var longestLabel = 0; 33 | var fastestRow = 0; 34 | 35 | foreach (var result in results) 36 | { 37 | var amountSlower = fastest.OperationsPerSecond / result.OperationsPerSecond; 38 | 39 | output.Add(new string[] { 40 | result.Label, 41 | result.Runs.Count().ToString(), 42 | SencibleDouble(result.OperationsPerSecond) + " +/-" + SencibleDouble(result.Error) + "%", 43 | SencibleDouble(amountSlower) + "x" 44 | }); 45 | 46 | longestLabel = Math.Max(longestLabel, result.Label.Length); 47 | fastestRow = amountSlower == 0.0d ? output.Count() - 1 : fastestRow; 48 | } 49 | 50 | PrintOutput(output, longestLabel, fastestRow); 51 | } 52 | 53 | private void PrintOutput(List output, int longestLabel, int fastestRow) 54 | { 55 | var prevColor = Console.ForegroundColor; 56 | 57 | //output.Add(String.Format("{0,-10}|{1,-10}|{2,-25}|{3,-15}", "Label", "Runs", "Ops/Sec", "% Slower")); 58 | //output.Add(String.Format("{0,-10}+{0,-10}+{1,-25}+{2,-15}", new String('-', 10), new String('-', 25), new String('-', 15))); 59 | var formatString = $"{{0,-{longestLabel+2}}}|{{1,-10}}|{{2,-25}}|{{3,-15}}"; 60 | 61 | Console.WriteLine(String.Format(formatString, "Label", "Runs", "Ops/Sec", "Times slower")); 62 | Console.WriteLine(String.Format(formatString.Replace('|', '+'), new String('-', longestLabel + 2), new String('-', 10), new String('-', 25), new String('-', 15))); 63 | 64 | // This is purpoesfully split as at some point it would be good to offer a more comprehencive output 65 | for (var i = 0; i < output.Count(); i++) 66 | { 67 | if (i == fastestRow) 68 | { 69 | Console.ForegroundColor = ConsoleColor.Green; 70 | } 71 | 72 | Console.WriteLine(String.Format(formatString, output[i])); 73 | Console.ForegroundColor = prevColor; 74 | } 75 | } 76 | 77 | private string SencibleDouble(double input) 78 | { 79 | if (input > 1000) 80 | { 81 | return String.Format("{0:n0}", input); 82 | } 83 | return String.Format("{0:n3}", input); 84 | } 85 | } 86 | } -------------------------------------------------------------------------------- /Benchmarkable/BenchmarkSettings.cs: -------------------------------------------------------------------------------- 1 | using System; 2 | 3 | namespace Benchmarkable 4 | { 5 | /// 6 | /// Settings that control the running of a benchmark. 7 | /// 8 | public class BenchmarkSettings 9 | { 10 | /// 11 | /// The minimum amount of time a batch should take to run 12 | /// 13 | public int InitialBatchTime { get; set; } = 500; 14 | 15 | /// 16 | /// The acceptable MSE of the last 10 (default) batches. 17 | /// 18 | public double MinimumErrorToAccept { get; set; } = 1.0d; 19 | 20 | /// 21 | /// The number of batches to keep and to calculate statistcs for 22 | /// 23 | private int batchesToWorkAcross = 10; 24 | public int BatchesToWorkAcross 25 | { 26 | get 27 | { 28 | return batchesToWorkAcross; 29 | } 30 | set 31 | { 32 | if (value > TDistribution.Values.Length) 33 | { 34 | throw new ArgumentOutOfRangeException($"Maximum value of batches is {TDistribution.Values.Length} as that is the number of t-distrubtion values stored"); 35 | } 36 | batchesToWorkAcross = value; 37 | } 38 | } 39 | 40 | /// 41 | /// Maximum amount of time to benchmark for if we haven't got to a standard deviation within our acceptable amount 42 | /// 43 | public int MaxTime { get; set; } = 5000; 44 | 45 | /// 46 | /// Write out information as the benchmark runs 47 | /// 48 | public bool Verbose { get; set; } = false; 49 | } 50 | } -------------------------------------------------------------------------------- /Benchmarkable/Benchmarkable.csproj: -------------------------------------------------------------------------------- 1 |  2 | 3 | 4 | Debug 5 | x86 6 | {68B3ADDC-3F9E-4205-A471-21D553966F7F} 7 | Library 8 | Benchmarkable 9 | Benchmarkable 10 | SAK 11 | SAK 12 | SAK 13 | SAK 14 | v4.6.1 15 | 16 | 17 | 18 | true 19 | full 20 | false 21 | bin\Debug 22 | DEBUG; 23 | prompt 24 | 4 25 | true 26 | x86 27 | false 28 | 29 | 30 | full 31 | true 32 | bin\Release 33 | prompt 34 | 4 35 | true 36 | x86 37 | false 38 | 39 | 40 | AnyCPU 41 | bin\Debug\ 42 | 4 43 | false 44 | false 45 | 46 | 47 | AnyCPU 48 | bin\Release\ 49 | 4 50 | false 51 | true 52 | false 53 | 54 | 55 | 56 | 57 | ..\packages\System.ValueTuple.4.3.0\lib\netstandard1.0\System.ValueTuple.dll 58 | 59 | 60 | 61 | 62 | 63 | 64 | 65 | 66 | 67 | 68 | 69 | 70 | 71 | 72 | 73 | 74 | 75 | -------------------------------------------------------------------------------- /Benchmarkable/Benchmarkable.nuspec: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | Benchmarkable 5 | $version$ 6 | Easy benchmarks that are statistically accurate 7 | Joey Ciechanowicz 8 | joeyciechanowicz 9 | https://github.com/bodyloss/Benchmarkable/blob/master/LICENSE 10 | http://github.com/bodyloss/Benchmarkable 11 | http://ICON_URL_HERE_OR_DELETE_THIS_LINE 12 | false 13 | Benchmark tool that aims to provide low error benchmark results using statistical methods 14 | Copyright 2017 15 | benchmark profile statistics fluent timing performance 16 | 17 | 18 | -------------------------------------------------------------------------------- /Benchmarkable/JustClause.cs: -------------------------------------------------------------------------------- 1 | using System; 2 | 3 | namespace Benchmarkable 4 | { 5 | public class JustClause 6 | { 7 | public BenchmarkResults This(Action action, string label = null) 8 | { 9 | var benchmark = new Benchmark(); 10 | benchmark.Add(action, label); 11 | return benchmark.Run(); 12 | } 13 | } 14 | } -------------------------------------------------------------------------------- /Benchmarkable/Properties/AssemblyInfo.cs: -------------------------------------------------------------------------------- 1 | using System.Reflection; 2 | using System.Runtime.CompilerServices; 3 | 4 | // Information about this assembly is defined by the following attributes. 5 | // Change them to the values specific to your project. 6 | 7 | [assembly: AssemblyTitle ("Benchmarkable")] 8 | [assembly: AssemblyDescription ("Benchmark utility")] 9 | [assembly: AssemblyConfiguration ("")] 10 | [assembly: AssemblyCompany ("")] 11 | [assembly: AssemblyProduct ("")] 12 | [assembly: AssemblyCopyright ("Joey Ciechanowicz")] 13 | [assembly: AssemblyTrademark ("")] 14 | [assembly: AssemblyCulture ("")] 15 | 16 | // The assembly version has the format "{Major}.{Minor}.{Build}.{Revision}". 17 | // The form "{Major}.{Minor}.*" will automatically update the build and revision, 18 | // and "{Major}.{Minor}.{Build}.*" will update just the revision. 19 | 20 | [assembly: AssemblyVersion ("1.0.0")] -------------------------------------------------------------------------------- /Benchmarkable/Result.cs: -------------------------------------------------------------------------------- 1 | using System.Collections.Generic; 2 | 3 | namespace Benchmarkable 4 | { 5 | public class Result 6 | { 7 | public string Label { get; internal set; } 8 | public int BatchSize { get; internal set; } 9 | public int BatchTime { get; internal set; } 10 | public IList Runs { get; internal set; } 11 | 12 | public double OperationsPerSecond 13 | { 14 | get 15 | { 16 | return Runs[Runs.Count - 1].OperationsPerSecond; 17 | } 18 | } 19 | 20 | public double MeanTicksPerBatch 21 | { 22 | get 23 | { 24 | return Runs[Runs.Count - 1].MeanTicks; 25 | } 26 | } 27 | 28 | public double Variance 29 | { 30 | get 31 | { 32 | return Runs[Runs.Count - 1].Variance; 33 | } 34 | } 35 | 36 | public double StandardDeviation 37 | { 38 | get 39 | { 40 | return Runs[Runs.Count - 1].StandardDeviation; 41 | } 42 | } 43 | 44 | public double StandardErrorsMean 45 | { 46 | get 47 | { 48 | return Runs[Runs.Count - 1].StandardErrorsMean; 49 | } 50 | } 51 | 52 | public double Error 53 | { 54 | get 55 | { 56 | return Runs[Runs.Count - 1].Error; 57 | } 58 | } 59 | } 60 | } -------------------------------------------------------------------------------- /Benchmarkable/Run.cs: -------------------------------------------------------------------------------- 1 | namespace Benchmarkable 2 | { 3 | // This class is a bit confusing. It actually stores the current values for all 4 | // its props, as calculated across all previous runs. So MeanTicks is the mean ticks across all batches (or last 10) 5 | public class Run 6 | { 7 | public long Ticks { get; internal set; } 8 | public double MeanTicks { get; internal set; } 9 | public double Variance { get; internal set; } 10 | public double StandardDeviation { get; internal set; } 11 | public double StandardErrorsMean { get; internal set; } 12 | public double Error { get; internal set; } 13 | public double OperationsPerSecond { get; internal set; } 14 | public bool ExceededMaxTime { get; internal set; } = false; 15 | } 16 | } -------------------------------------------------------------------------------- /Benchmarkable/TDistribution.cs: -------------------------------------------------------------------------------- 1 | using System; 2 | using System.Collections.Generic; 3 | using System.Linq; 4 | using System.Text; 5 | using System.Threading.Tasks; 6 | 7 | namespace Benchmarkable 8 | { 9 | class TDistribution 10 | { 11 | // We want 95% correction 12 | internal static double[] Values = 13 | { 14 | 12.71, 15 | 4.303, 16 | 3.182, 17 | 2.776, 18 | 2.571, 19 | 2.447, 20 | 2.365, 21 | 2.306, 22 | 2.262, 23 | 2.228, 24 | 2.201, 25 | 2.179, 26 | 2.160, 27 | 2.145, 28 | 2.131, 29 | 2.120, 30 | 2.110, 31 | 2.101, 32 | 2.093, 33 | 2.086, 34 | 2.080, 35 | 2.074, 36 | 2.069, 37 | 2.064, 38 | 2.060, 39 | 2.056, 40 | 2.052, 41 | 2.048, 42 | 2.045, 43 | 2.042 44 | }; 45 | } 46 | } -------------------------------------------------------------------------------- /Benchmarkable/ThisClause.cs: -------------------------------------------------------------------------------- 1 | using System; 2 | 3 | namespace Benchmarkable 4 | { 5 | public class ThisClause 6 | { 7 | private Benchmark benchmark; 8 | 9 | internal ThisClause(Benchmark benchmark) 10 | { 11 | this.benchmark = benchmark; 12 | } 13 | 14 | /// 15 | /// Method to benchmark against 16 | /// 17 | /// Second method 18 | /// Label for the method 19 | /// Results of the benchmark 20 | public BenchmarkResults Against(Action action, string label = null) 21 | { 22 | benchmark.Add(action, label); 23 | return benchmark.Run(); 24 | } 25 | } 26 | } -------------------------------------------------------------------------------- /Benchmarkable/packages.config: -------------------------------------------------------------------------------- 1 |  2 | 3 | 4 | -------------------------------------------------------------------------------- /Examples/Examples.csproj: -------------------------------------------------------------------------------- 1 |  2 | 3 | 4 | Debug 5 | x86 6 | {8B4B91C2-69C0-4A28-B2DB-F2B4F58C49B3} 7 | Exe 8 | Examples 9 | Examples 10 | v4.6.1 11 | 12 | 13 | 14 | true 15 | full 16 | false 17 | bin\Debug 18 | DEBUG; 19 | prompt 20 | 4 21 | true 22 | x86 23 | false 24 | 25 | 26 | full 27 | true 28 | bin\Release 29 | prompt 30 | 4 31 | true 32 | x86 33 | false 34 | 35 | 36 | AnyCPU 37 | bin\Debug\ 38 | 4 39 | false 40 | false 41 | 42 | 43 | AnyCPU 44 | bin\Release\ 45 | 4 46 | false 47 | false 48 | 49 | 50 | 51 | 52 | ..\packages\System.ValueTuple.4.3.0\lib\netstandard1.0\System.ValueTuple.dll 53 | 54 | 55 | 56 | 57 | 58 | 59 | 60 | 61 | 62 | 63 | 64 | 65 | 66 | {68b3addc-3f9e-4205-a471-21d553966f7f} 67 | Benchmarkable 68 | 69 | 70 | -------------------------------------------------------------------------------- /Examples/Program.cs: -------------------------------------------------------------------------------- 1 | using System; 2 | using Benchmarkable; 3 | using System.Text.RegularExpressions; 4 | 5 | namespace Examples 6 | { 7 | class MainClass 8 | { 9 | public static void Main (string[] args) 10 | { 11 | Benchmark.Settings.Verbose = true; 12 | 13 | var haystack = "abcdef"; 14 | Benchmark.This(() => haystack.Contains("ef"), "string.Contains") 15 | .Against(() => new Regex("ef").IsMatch(haystack), "string.IndexOf") 16 | .Print(); 17 | 18 | // Single method 19 | Benchmark.Just.This(() => { }) 20 | .Print(); 21 | 22 | // Two methods 23 | Benchmark.This(() => { }) 24 | .Against(() => { }) 25 | .Print(); 26 | 27 | // Multiple methods 28 | Benchmark.These(new(Action, string)[]{ 29 | (() => Math.Sin(1.23), "Sin1"), 30 | (() => Math.Sin(1.23), "Sin2"), 31 | (() => Math.Sin(1.23), "Sin3"), 32 | (() => Math.Sin(1.23), "Sin4"), 33 | (() => Math.Sin(1.23), "Sin5"), 34 | }).Print(); 35 | 36 | // Explicit usage 37 | var benchmark = new Benchmark(); 38 | benchmark.Add(() => { }, "Label"); 39 | benchmark.Add(() => { }, "Label 2"); 40 | var results = benchmark.Run(); 41 | results.Print(); 42 | 43 | Console.Read(); 44 | } 45 | } 46 | } 47 | -------------------------------------------------------------------------------- /Examples/Properties/AssemblyInfo.cs: -------------------------------------------------------------------------------- 1 | using System.Reflection; 2 | using System.Runtime.CompilerServices; 3 | 4 | // Information about this assembly is defined by the following attributes. 5 | // Change them to the values specific to your project. 6 | 7 | [assembly: AssemblyTitle ("Examples")] 8 | [assembly: AssemblyDescription ("")] 9 | [assembly: AssemblyConfiguration ("")] 10 | [assembly: AssemblyCompany ("")] 11 | [assembly: AssemblyProduct ("")] 12 | [assembly: AssemblyCopyright ("joeyciechanowicz")] 13 | [assembly: AssemblyTrademark ("")] 14 | [assembly: AssemblyCulture ("")] 15 | 16 | // The assembly version has the format "{Major}.{Minor}.{Build}.{Revision}". 17 | // The form "{Major}.{Minor}.*" will automatically update the build and revision, 18 | // and "{Major}.{Minor}.{Build}.*" will update just the revision. 19 | 20 | [assembly: AssemblyVersion ("1.0.*")] 21 | 22 | // The following attributes are used to specify the signing key for the assembly, 23 | // if desired. See the Mono documentation for more information about signing. 24 | 25 | //[assembly: AssemblyDelaySign(false)] 26 | //[assembly: AssemblyKeyFile("")] 27 | 28 | -------------------------------------------------------------------------------- /Examples/app.config: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | -------------------------------------------------------------------------------- /Examples/packages.config: -------------------------------------------------------------------------------- 1 |  2 | 3 | 4 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | The MIT License (MIT) 2 | 3 | Copyright (c) 2014 Joey Ciechanowicz 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | Benchmarkable 2 | =========== 3 | 4 | Simple easy .NET benchmarking for little bits of code that uses probability to give you accurate results regardless of how fast your CPU is. When you benchmark a method it will do a sample run to 5 | see how many iterations per batch it should run of your method. Then it will run batch after batch until the error percentage is below 1% (the default, this can be changed though). 6 | 7 | The idea is that even if your CPU is insanely fast or really slow you will still get a statistically accurate result. 8 | 9 | ## Install 10 | Run the following command in the Package Manager Console (NuGet). 11 | ```bash 12 | PM> Install-Package Benchmarkable 13 | ``` 14 | Or clone and include Benchmarkable.csproj directly 15 | 16 | ## Use 17 | Lets say you wanted to see if string.Contains was faster or slower than using a regular expression 18 | ```csharp 19 | var haystack = "abcdef"; 20 | Benchmark.This(() => haystack.Contains("ef"), "string.Contains") 21 | .Against(() => new Regex("ef").IsMatch(haystack), "Regex.IsMatch") 22 | .Print(); 23 | ``` 24 | ``` 25 | Label |Runs |Ops/Sec |Times slower 26 | -----------------+----------+-------------------------+--------------- 27 | string.Contains |27 |13,773,246 +/-3.460% |1.000x 28 | Regex.IsMatch |10 |459,500 +/-9.674% |29.974x 29 | ``` 30 | 31 | ### Single method 32 | ```csharp 33 | Benchmark.Just.This(() => Math.Sin(1.23)) 34 | .Print(); 35 | ``` 36 | 37 | ### Two methods 38 | ```csharp 39 | Benchmark.This(() => { }) 40 | .Against(() => { }, "With a label") 41 | .Print(); 42 | ``` 43 | 44 | ### Multiple methods 45 | ```csharp 46 | Benchmark.These(new(Action, string)[]{ 47 | (() => Math.Sin(1.23), "Sin1"), 48 | (() => Math.Sin(1.23), "Sin2"), 49 | (() => Math.Sin(1.23), "Sin3"), 50 | (() => Math.Sin(1.23), "Sin4"), 51 | (() => Math.Sin(1.23), "Sin5"), 52 | }).Print(); 53 | ``` 54 | _Note: You have to provide labels for each method you want to benchmark when using this syntax_ 55 | 56 | ### Explicit declaration 57 | ```csharp 58 | var benchmark = new Benchmark(); 59 | benchmark.Add(() => { }, "Label"); 60 | benchmark.Add(() => { }, "Label 2"); 61 | var results = benchmark.Run(); 62 | results.Print(); 63 | ``` 64 | 65 | ### Benchmark.Settings 66 | The settings must be changed *before* creating a benchmark. 67 | 68 | | Value | Default | Description | 69 | |-------------------------------------------|---------|---------------------------------------------------------------------------------------------------------| 70 | | `Benchmark.Settings.InitialBatchTime` | 500ms | The time to run a benchmark method for initially to calculate the batch size | 71 | | `Benchmark.Settings.MinimumErrorToAccept` | 1.0 | Error % over the last 10 (default) batches to accept as a valid benchmark result | 72 | | `Benchmark.Settings.BatchesToWorkAcross` | 10 | Number of batches to calculate statistics across (and thus error%) | 73 | | `Benchmark.Settings.MaxTime` | 5000ms | Maximum time to run a single method for before terminating, this is if the error % has not yet been met | 74 | | `Benchmark.Settings.Verbose` | false | Print outinformation as a benchmark is run. | 75 | 76 | 77 | ### Contributing 78 | Any and all help is greatly appreciated. Especially around the statistical elements. 79 | 80 | # TODO 81 | * Printing of results, needs to be cleaned up made more versatile. Maybe some lightweight templates 82 | * Automatically calculate the batch size to one that will lower the error % as fast as possible. 83 | --------------------------------------------------------------------------------