├── .gitignore ├── .gitlab-ci.yml ├── CODEOWNERS ├── CONTRIBUTING ├── CONTRIBUTIONS.md ├── Changelog.md ├── LICENSE.md ├── MismatchedConfigTooltip.png ├── README.md ├── TestConfigWithMismatchedItems.png ├── Third Party Notices.md ├── UnityPerformanceBenchmarkComponents.png ├── UnityPerformanceBenchmarkReport.png ├── UnityPerformanceBenchmarkReporter.sln ├── UnityPerformanceBenchmarkReporter.sln.DotSettings.user ├── UnityPerformanceBenchmarkReporter ├── ESupportedFileTypes.cs ├── Entities │ ├── Data.cs │ ├── PerformanceTestResult.cs │ ├── PerformanceTestRun.cs │ ├── PerformanceTestRunResult.cs │ ├── SampleGroup.cs │ ├── SampleGroupResult.cs │ ├── TestResult.cs │ └── TestState.cs ├── ExtensionMethods.cs ├── IParser.cs ├── OptionsParser.cs ├── PerformanceBenchmark.cs ├── PerformanceTestRunProcessor.cs ├── Program.cs ├── Report │ ├── Chart.bundle.js │ ├── ReportWriter.cs │ ├── UnityLogo.png │ ├── help-hover.png │ ├── help.png │ ├── styles.css │ └── warning.png ├── TestResultJsonParser.cs ├── TestResultXmlParser.cs ├── TestRunMetadataProcessor.cs ├── UnityPerformanceBenchmarkReporter.csproj └── UnityPerformanceBenchmarkReporter.csproj.user ├── UnityPerformanceBenchmarkReporterTests ├── ExtensionMethodsTests.cs ├── PerformanceBenchmarkTestsBase.cs ├── PerformanceBenchmarkTestsJson.cs ├── PerformanceBenchmarkTestsXML.cs ├── TestData │ ├── BaselineJson │ │ └── baseline.json │ ├── BaselineJson2 │ │ └── baseline2.json │ ├── Baselines │ │ └── baseline.xml │ ├── Baselines2 │ │ └── baseline2.xml │ ├── Results │ │ ├── results.xml │ │ └── results2.xml │ ├── ResultsJson │ │ ├── results.json │ │ └── resultscopy.json │ ├── ResultsJson2 │ │ └── results2.json │ ├── baseline.json │ ├── baseline.xml │ ├── baseline2.json │ ├── baseline2.xml │ ├── results.json │ ├── results.xml │ ├── results2.json │ └── results2.xml └── UnityPerformanceBenchmarkReporterTests.csproj ├── dotnet-install.ps1 └── latest_custom_UnityPerformanceBenchmark_2020-06-29_01-08-13-481.html /.gitignore: -------------------------------------------------------------------------------- 1 | .vs/ 2 | .idea/ 3 | */bin/ 4 | */obj/ 5 | */Properties/launchSettings.json 6 | -------------------------------------------------------------------------------- /.gitlab-ci.yml: -------------------------------------------------------------------------------- 1 | stages: 2 | - build 3 | build:win: 4 | stage: build 5 | tags: 6 | - buildfarm 7 | - windows 8 | - "10" 9 | before_script: 10 | - echo "Installing dotnetcore-sdk" 11 | - 'powershell -Command "Unblock-File dotnet-install.ps1 && powershell -Command ".\dotnet-install.ps1 -InstallDir C:\Program Files\dotnet"' 12 | script: 13 | - echo "Building solution" 14 | - '"C:\Program Files\dotnet\dotnet.exe" build' 15 | # - echo "Running tests" 16 | # - cd UnityPerformanceBenchmarkReporterTests 17 | # - '"C:\Program Files\dotnet\dotnet.exe" test --logger "trx;LogFileName=UnityPerformanceBenchmarkReporterTests.trx"' 18 | after_script: 19 | - C:\Users\builduser\post_build_script.bat 20 | -------------------------------------------------------------------------------- /CODEOWNERS: -------------------------------------------------------------------------------- 1 | # PerformanceBenchmarkReporter codeowners 2 | 3 | # Global rule: 4 | * @seanstolberg-unity -------------------------------------------------------------------------------- /CONTRIBUTING: -------------------------------------------------------------------------------- 1 | # PR review process 2 | 3 | - Any PR must have an entry in the corresponding changelog in a separate commit (CHANGELOG.MD file) 4 | - Changelog follow these guidelines: https://github.com/Unity-Technologies/PostProcessing/blob/v2/CHANGELOG.md 5 | - Each release branch (2018.1, 2018.2...) have a unique Changelog file 6 | - when backporting, don't backport the changelog commit but update the branch changelog manually 7 | - (optional) add reviewver from doc team 8 | - Any more complex description of a change with future need to go in a release note file -------------------------------------------------------------------------------- /CONTRIBUTIONS.md: -------------------------------------------------------------------------------- 1 | # Contributions 2 | 3 | ## If you are interested in contributing, here are some ground rules: 4 | * Talk to us before doing the work -- we love contributions, but we might already be working on the same thing, or we might have different opinions on how it should be implemented. 5 | 6 | ## All contributions are subject to the [Unity Contribution Agreement(UCA)](https://unity3d.com/legal/licenses/Unity_Contribution_Agreement) 7 | By making a pull request, you are confirming agreement to the terms and conditions of the UCA, including that your Contributions are your original creation and that you have complete right and authority to make your Contributions. 8 | 9 | ## Once you have a change ready following these ground rules. Simply make a pull request in Github -------------------------------------------------------------------------------- /Changelog.md: -------------------------------------------------------------------------------- 1 | # Changelog 2 | 3 | All notable changes to this package will be documented in this file. 4 | 5 | The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/) 6 | and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.html). 7 | 8 | ## [1.5.1] - 2023-05-02 9 | 10 | ### Fixed 11 | 12 | - Fixed missing Changelog 13 | - Fixed outdated command line help docs 14 | 15 | ## [1.5.0] - 2023-04-25 16 | 17 | ### Fixed 18 | 19 | - Fixed Standard Deveation getting added twice. 20 | 21 | ### Added 22 | 23 | - Added ignore= command line option for passing in a list of metrics by name to ignore during compariosn. 24 | 25 | ### Changed 26 | 27 | - Updated .Net Target Version to include .Net 6 and .Net 7 28 | 29 | ## [1.4.0] - 2023-04-13 30 | 31 | ### Added 32 | 33 | - Added support marking a sample group as having "A Known Issue". Logging these in a seperate results category and not returning 1. 34 | 35 | ## [1.3.0] - 2023-04-13 36 | 37 | ### Fixed 38 | 39 | - Fixed missing data members in V2 Data Format 40 | 41 | ### Added 42 | 43 | - Added support for parsing json formatted perf data 44 | - Added fileformat= command line option for setting the file format 45 | - Added dataversion= command line option for setting the version V1 or V2 orf the files to be processed for Json Format 46 | 47 | ## [1.2.0] - 2021-05-17 48 | 49 | ### Changed 50 | 51 | - Updated .Net Target to .Net Core 3.1 52 | - Updated chart.js to V2.9.4 53 | 54 | ## [1.0.1] - 2020-05-29 55 | 56 | ### Fixed 57 | 58 | - Fix for date format from epoch unix s to ms 59 | 60 | ## [1.0.0] - 2020-05-08 61 | 62 | ### Fixed 63 | 64 | - Fixed "Show Failed Tests Only" not being togglable in report. 65 | 66 | ### Added 67 | 68 | - Adds support for V2 Perf Data Format 69 | 70 | ### Changed 71 | 72 | - Updated .Net Target to .Net Core 3.0 73 | 74 | This is the first release of _Unity Performance Benchmark Reporter_. 75 | -------------------------------------------------------------------------------- /LICENSE.md: -------------------------------------------------------------------------------- 1 | Copyright © 2018 Unity Technologies ApS 2 | 3 | Licensed under the Unity Companion License for Unity-dependent projects--see [Unity Companion License](http://www.unity3d.com/legal/licenses/Unity_Companion_License). 4 | 5 | Unless expressly provided otherwise, the Software under this license is made available strictly on an “AS IS” BASIS WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED. Please review the license for details on these and other terms and conditions. -------------------------------------------------------------------------------- /MismatchedConfigTooltip.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Unity-Technologies/PerformanceBenchmarkReporter/c403bfed95d9c1c14b644943a8ea662abb7d483e/MismatchedConfigTooltip.png -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | The Unity Performance Benchmark tool enables partners and developers to establish benchmark samples and measurements using the Performance Testing package, then use these benchmark values to compare subsequent performance test results in an html output utilizing graphical visualizations. 2 | 3 | Please see the [Performance Benchmark Reporter wiki](https://github.com/Unity-Technologies/PerformanceBenchmarkReporter/wiki) in this project for more detailed description and usage. 4 | -------------------------------------------------------------------------------- /TestConfigWithMismatchedItems.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Unity-Technologies/PerformanceBenchmarkReporter/c403bfed95d9c1c14b644943a8ea662abb7d483e/TestConfigWithMismatchedItems.png -------------------------------------------------------------------------------- /Third Party Notices.md: -------------------------------------------------------------------------------- 1 | This package contains third-party software components governed by the license(s) indicated below: 2 | 3 | Component Name: Chart.js 4 | 5 | License Type: MIT 6 | 7 | The MIT License (MIT) 8 | 9 | Copyright (c) 2018 Chart.js Contributors 10 | 11 | Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: 12 | 13 | The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. -------------------------------------------------------------------------------- /UnityPerformanceBenchmarkComponents.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Unity-Technologies/PerformanceBenchmarkReporter/c403bfed95d9c1c14b644943a8ea662abb7d483e/UnityPerformanceBenchmarkComponents.png -------------------------------------------------------------------------------- /UnityPerformanceBenchmarkReport.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Unity-Technologies/PerformanceBenchmarkReporter/c403bfed95d9c1c14b644943a8ea662abb7d483e/UnityPerformanceBenchmarkReport.png -------------------------------------------------------------------------------- /UnityPerformanceBenchmarkReporter.sln: -------------------------------------------------------------------------------- 1 |  2 | Microsoft Visual Studio Solution File, Format Version 12.00 3 | # Visual Studio 15 4 | VisualStudioVersion = 15.0.27428.2027 5 | MinimumVisualStudioVersion = 10.0.40219.1 6 | Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "UnityPerformanceBenchmarkReporter", "UnityPerformanceBenchmarkReporter\UnityPerformanceBenchmarkReporter.csproj", "{3E9C48B1-7CA9-4E55-B3EF-0EE91FF8BF37}" 7 | EndProject 8 | Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "UnityPerformanceBenchmarkReporterTests", "UnityPerformanceBenchmarkReporterTests\UnityPerformanceBenchmarkReporterTests.csproj", "{4BB0BB51-6DDE-4ED2-8F41-51943FA74844}" 9 | EndProject 10 | Global 11 | GlobalSection(SolutionConfigurationPlatforms) = preSolution 12 | Debug|Any CPU = Debug|Any CPU 13 | Release|Any CPU = Release|Any CPU 14 | EndGlobalSection 15 | GlobalSection(ProjectConfigurationPlatforms) = postSolution 16 | {3E9C48B1-7CA9-4E55-B3EF-0EE91FF8BF37}.Debug|Any CPU.ActiveCfg = Debug|Any CPU 17 | {3E9C48B1-7CA9-4E55-B3EF-0EE91FF8BF37}.Debug|Any CPU.Build.0 = Debug|Any CPU 18 | {3E9C48B1-7CA9-4E55-B3EF-0EE91FF8BF37}.Release|Any CPU.ActiveCfg = Release|Any CPU 19 | {3E9C48B1-7CA9-4E55-B3EF-0EE91FF8BF37}.Release|Any CPU.Build.0 = Release|Any CPU 20 | {4BB0BB51-6DDE-4ED2-8F41-51943FA74844}.Debug|Any CPU.ActiveCfg = Debug|Any CPU 21 | {4BB0BB51-6DDE-4ED2-8F41-51943FA74844}.Debug|Any CPU.Build.0 = Debug|Any CPU 22 | {4BB0BB51-6DDE-4ED2-8F41-51943FA74844}.Release|Any CPU.ActiveCfg = Release|Any CPU 23 | {4BB0BB51-6DDE-4ED2-8F41-51943FA74844}.Release|Any CPU.Build.0 = Release|Any CPU 24 | EndGlobalSection 25 | GlobalSection(SolutionProperties) = preSolution 26 | HideSolutionNode = FALSE 27 | EndGlobalSection 28 | GlobalSection(ExtensibilityGlobals) = postSolution 29 | SolutionGuid = {DB9F077A-8AD5-4C3A-84C6-1E9E4658D66F} 30 | EndGlobalSection 31 | EndGlobal 32 | -------------------------------------------------------------------------------- /UnityPerformanceBenchmarkReporter.sln.DotSettings.user: -------------------------------------------------------------------------------- 1 |  2 | SOLUTION 3 | 2 -------------------------------------------------------------------------------- /UnityPerformanceBenchmarkReporter/ESupportedFileTypes.cs: -------------------------------------------------------------------------------- 1 | namespace UnityPerformanceBenchmarkReporter.Entities 2 | { 3 | public enum ESupportedFileTypes 4 | { 5 | json, 6 | xml 7 | } 8 | } -------------------------------------------------------------------------------- /UnityPerformanceBenchmarkReporter/Entities/Data.cs: -------------------------------------------------------------------------------- 1 | using System; 2 | using System.Collections.Generic; 3 | 4 | namespace UnityPerformanceBenchmarkReporter.Entities.New 5 | { 6 | [Serializable] 7 | public class TestResult 8 | { 9 | public string Name; 10 | public string Version; 11 | public List Categories = new List(); 12 | public List SampleGroups = new List(); 13 | } 14 | 15 | [Serializable] 16 | public class SampleGroup 17 | { 18 | public string Name; 19 | public SampleUnit Unit; 20 | public bool IncreaseIsBetter; 21 | public double Threshold = 0.15; 22 | public List Samples = new List(); 23 | public double Min; 24 | public double Max; 25 | public double Median; 26 | public double Average; 27 | public double StandardDeviation; 28 | public double Sum; 29 | public bool ContainsKnownIssue; 30 | public string KnownIssueDetails = ""; 31 | 32 | public SampleGroup(string name, SampleUnit unit, bool increaseIsBetter) 33 | { 34 | Name = name; 35 | Unit = unit; 36 | IncreaseIsBetter = increaseIsBetter; 37 | } 38 | } 39 | 40 | [Serializable] 41 | public class Run 42 | { 43 | public string TestSuite; 44 | public long Date; 45 | public Player Player; 46 | public Hardware Hardware; 47 | public Editor Editor; 48 | public List Dependencies = new List(); 49 | public List Results = new List(); 50 | } 51 | 52 | [Serializable] 53 | public class Editor 54 | { 55 | public string Version; 56 | public string Branch; 57 | public string Changeset; 58 | public int Date; 59 | } 60 | 61 | [Serializable] 62 | public class Hardware 63 | { 64 | public string OperatingSystem; 65 | public string DeviceModel; 66 | public string DeviceName; 67 | public string ProcessorType; 68 | public int ProcessorCount; 69 | public string GraphicsDeviceName; 70 | public int SystemMemorySizeMB; 71 | 72 | public string XrModel ; 73 | public string XrDevice; 74 | } 75 | 76 | [Serializable] 77 | public class Player 78 | { 79 | public string Platform; 80 | public bool Development; 81 | public int ScreenWidth; 82 | public int ScreenHeight; 83 | public int ScreenRefreshRate; 84 | public bool Fullscreen; 85 | public int Vsync; 86 | public int AntiAliasing; 87 | public string ColorSpace; 88 | public string AnisotropicFiltering; 89 | public string BlendWeights; 90 | public string GraphicsApi; 91 | public bool Batchmode; 92 | public string RenderThreadingMode; 93 | public bool GpuSkinning; 94 | 95 | // strings because values are editor only enums 96 | public string ScriptingBackend; 97 | public string AndroidTargetSdkVersion; 98 | public string AndroidBuildSystem; 99 | public string BuildTarget; 100 | public string StereoRenderingPath; 101 | } 102 | 103 | public enum SampleUnit 104 | { 105 | Nanosecond, 106 | Microsecond, 107 | Millisecond, 108 | Second, 109 | Byte, 110 | Kilobyte, 111 | Megabyte, 112 | Gigabyte, 113 | Undefined 114 | } 115 | } 116 | -------------------------------------------------------------------------------- /UnityPerformanceBenchmarkReporter/Entities/PerformanceTestResult.cs: -------------------------------------------------------------------------------- 1 | using System; 2 | using System.Collections.Generic; 3 | 4 | namespace UnityPerformanceBenchmarkReporter.Entities 5 | { 6 | [Serializable] 7 | public class PerformanceTestResult 8 | { 9 | public string TestName; 10 | public List TestCategories; 11 | public string TestVersion; 12 | public double StartTime; 13 | public double EndTime; 14 | public List SampleGroups; 15 | } 16 | } -------------------------------------------------------------------------------- /UnityPerformanceBenchmarkReporter/Entities/PerformanceTestRun.cs: -------------------------------------------------------------------------------- 1 | using System; 2 | using System.Collections.Generic; 3 | 4 | namespace UnityPerformanceBenchmarkReporter.Entities 5 | { 6 | [Serializable] 7 | public class PerformanceTestRun 8 | { 9 | public PlayerSystemInfo PlayerSystemInfo; 10 | public EditorVersion EditorVersion; 11 | public BuildSettings BuildSettings; 12 | public ScreenSettings ScreenSettings; 13 | public QualitySettings QualitySettings; 14 | public PlayerSettings PlayerSettings; 15 | public ProjectVersion ProjectVersion; 16 | 17 | public string TestProject; 18 | public string TestSuite; 19 | public double StartTime; 20 | public double EndTime; 21 | public List Results = new List(); 22 | 23 | public JobMetaData JobMetaData ; 24 | public object Dependencies; 25 | } 26 | 27 | [Serializable] 28 | public class ProjectVersion 29 | { 30 | public string ProjectName { get; set; } 31 | public string Branch { get; set; } 32 | public string Changeset { get; set; } 33 | public DateTime Date { get; set; } 34 | } 35 | 36 | [Serializable] 37 | public class PlayerSystemInfo 38 | { 39 | public string OperatingSystem; 40 | public string DeviceModel; 41 | public string DeviceName; 42 | public string ProcessorType; 43 | public int ProcessorCount; 44 | public string GraphicsDeviceName; 45 | public int SystemMemorySize; 46 | public string XrModel; 47 | public string XrDevice; 48 | } 49 | 50 | [Serializable] 51 | public class EditorVersion 52 | { 53 | public string FullVersion; 54 | public int DateSeconds; 55 | public string Branch; 56 | public int RevisionValue; 57 | } 58 | 59 | [Serializable] 60 | public class BuildSettings 61 | { 62 | public string Platform; 63 | public string BuildTarget; 64 | public bool DevelopmentPlayer; 65 | public string AndroidBuildSystem; 66 | } 67 | 68 | [Serializable] 69 | public class ScreenSettings 70 | { 71 | public int ScreenWidth; 72 | public int ScreenHeight; 73 | public int ScreenRefreshRate; 74 | public bool Fullscreen; 75 | } 76 | 77 | [Serializable] 78 | public class QualitySettings 79 | { 80 | public int Vsync; 81 | public int AntiAliasing; 82 | public string ColorSpace; 83 | public string AnisotropicFiltering; 84 | public string BlendWeights; 85 | } 86 | 87 | [Serializable] 88 | public class PlayerSettings 89 | { 90 | public string ScriptingBackend; 91 | public bool VrSupported; 92 | public bool MtRendering; 93 | public bool GraphicsJobs; 94 | public bool GpuSkinning; 95 | public string GraphicsApi; 96 | public string Batchmode; 97 | //public int StaticBatching; TODO 98 | //public int DynamicBatching; TODO 99 | public string StereoRenderingPath; 100 | public string RenderThreadingMode; 101 | public string AndroidMinimumSdkVersion; 102 | public string AndroidTargetSdkVersion; 103 | public List EnabledXrTargets; 104 | 105 | public string ScriptingRuntimeVersion; 106 | } 107 | 108 | 109 | [Serializable] 110 | public class Yamato 111 | { 112 | public string JobFriendlyName { get; set; } 113 | public string JobName { get; set; } 114 | public string JobId { get; set; } 115 | public string ProjectId { get; set; } 116 | public string ProjectName { get; set; } 117 | public object WorkDir { get; set; } 118 | public object JobOwnerEmail { get; set; } 119 | } 120 | 121 | [Serializable] 122 | public class JobMetaData 123 | { 124 | public Yamato Yamato { get; set; } 125 | public object Bokken { get; set; } 126 | } 127 | } -------------------------------------------------------------------------------- /UnityPerformanceBenchmarkReporter/Entities/PerformanceTestRunResult.cs: -------------------------------------------------------------------------------- 1 | using System; 2 | using System.Collections.Generic; 3 | 4 | namespace UnityPerformanceBenchmarkReporter.Entities 5 | { 6 | [Serializable] 7 | public class PerformanceTestRunResult 8 | { 9 | public PlayerSystemInfo PlayerSystemInfo; 10 | public EditorVersion EditorVersion; 11 | public BuildSettings BuildSettings; 12 | public ScreenSettings ScreenSettings; 13 | public QualitySettings QualitySettings; 14 | public PlayerSettings PlayerSettings; 15 | public string TestSuite; 16 | public DateTime StartTime; 17 | public List TestResults = new List(); 18 | public bool IsBaseline; 19 | public string ResultName; 20 | 21 | public bool TestRunMetadataExists() 22 | { 23 | return PlayerSystemInfo != null 24 | || PlayerSettings != null 25 | || QualitySettings != null 26 | || ScreenSettings != null 27 | || BuildSettings != null 28 | || EditorVersion != null; 29 | } 30 | } 31 | } -------------------------------------------------------------------------------- /UnityPerformanceBenchmarkReporter/Entities/SampleGroup.cs: -------------------------------------------------------------------------------- 1 | using System; 2 | using System.Collections.Generic; 3 | 4 | namespace UnityPerformanceBenchmarkReporter.Entities 5 | { 6 | [Serializable] 7 | public class SampleGroup 8 | { 9 | public List Samples; 10 | public double Min; 11 | public double Max; 12 | public double Median; 13 | public double Average; 14 | public double StandardDeviation; 15 | public double PercentileValue; 16 | public double Sum; 17 | public int Zeroes; 18 | public int SampleCount; 19 | public SampleGroupDefinition Definition; 20 | } 21 | 22 | [Serializable] 23 | public class SampleGroupDefinition 24 | { 25 | public string Name; 26 | public SampleUnit SampleUnit; 27 | public AggregationType AggregationType = AggregationType.Median; 28 | public double Threshold; 29 | public bool IncreaseIsBetter; 30 | public double Percentile; 31 | 32 | public bool FailOnBaseline; 33 | public bool ContainsKnownIssue; 34 | public string KnownIssueDetails = ""; 35 | } 36 | 37 | public enum AggregationType 38 | { 39 | Average = 0, 40 | Min = 1, 41 | Max = 2, 42 | Median = 3, 43 | Percentile = 4 44 | } 45 | 46 | public enum SampleUnit 47 | { 48 | Nanosecond, 49 | Microsecond, 50 | Millisecond, 51 | Second, 52 | Byte, 53 | Kilobyte, 54 | Megabyte, 55 | Gigabyte, 56 | None 57 | } 58 | 59 | } -------------------------------------------------------------------------------- /UnityPerformanceBenchmarkReporter/Entities/SampleGroupResult.cs: -------------------------------------------------------------------------------- 1 |  2 | namespace UnityPerformanceBenchmarkReporter.Entities 3 | { 4 | public class SampleGroupResult 5 | { 6 | public string SampleGroupName; 7 | public string SampleUnit; 8 | public double AggregatedValue; 9 | public double BaselineValue; 10 | public double Threshold; 11 | public bool IncreaseIsBetter; 12 | public string AggregationType; 13 | public double Percentile; 14 | public bool Regressed; 15 | public bool Progressed; 16 | public bool RegressedKnown; 17 | public bool ContainsKnownIssue; 18 | public string KnownIssueDetails; 19 | public double Min; 20 | public double Max; 21 | public double Median; 22 | public double Average; 23 | public double StandardDeviation; 24 | public double PercentileValue; 25 | public double Sum; 26 | public int Zeroes; 27 | public int SampleCount; 28 | } 29 | } 30 | -------------------------------------------------------------------------------- /UnityPerformanceBenchmarkReporter/Entities/TestResult.cs: -------------------------------------------------------------------------------- 1 | using System.Collections.Generic; 2 | 3 | namespace UnityPerformanceBenchmarkReporter.Entities 4 | { 5 | public class TestResult 6 | { 7 | public string TestName; 8 | public List TestCategories; 9 | public string TestVersion; 10 | public int State; 11 | public List SampleGroupResults; 12 | } 13 | } 14 | -------------------------------------------------------------------------------- /UnityPerformanceBenchmarkReporter/Entities/TestState.cs: -------------------------------------------------------------------------------- 1 |  2 | namespace UnityPerformanceBenchmarkReporter.Entities 3 | { 4 | public enum TestState 5 | { 6 | Inconclusive = 0, 7 | NotRunnable = 1, 8 | Skipped = 2, 9 | Ignored = 3, 10 | Success = 4, 11 | Failure = 5, 12 | Error = 6, 13 | Cancelled = 7 14 | } 15 | } 16 | -------------------------------------------------------------------------------- /UnityPerformanceBenchmarkReporter/ExtensionMethods.cs: -------------------------------------------------------------------------------- 1 | using System; 2 | using System.Globalization; 3 | 4 | namespace UnityPerformanceBenchmarkReporter 5 | { 6 | public static class ExtensionMethods 7 | { 8 | public static double TruncToSigFig(this double d, uint digits) 9 | { 10 | double truncated; 11 | 12 | if(d == 0) 13 | { 14 | truncated = 0; 15 | } 16 | else 17 | { 18 | var s = Convert.ToString(d, CultureInfo.InvariantCulture); 19 | var parts = s.Split('.'); 20 | if (parts.Length <= 1 || parts[1].Length <= digits) 21 | { 22 | truncated = d; 23 | } 24 | else 25 | { 26 | var newSigDigits = parts[1].Substring(0, (int) digits); 27 | var truncString = string.Format("{0}.{1}", parts[0], newSigDigits); 28 | truncated = Convert.ToDouble(truncString); 29 | } 30 | } 31 | 32 | return truncated; 33 | } 34 | } 35 | } 36 | -------------------------------------------------------------------------------- /UnityPerformanceBenchmarkReporter/IParser.cs: -------------------------------------------------------------------------------- 1 | using UnityPerformanceBenchmarkReporter.Entities; 2 | 3 | namespace UnityPerformanceBenchmarkReporter 4 | { 5 | public interface IParser 6 | { 7 | public PerformanceTestRun Parse(string path,int version); 8 | } 9 | } -------------------------------------------------------------------------------- /UnityPerformanceBenchmarkReporter/OptionsParser.cs: -------------------------------------------------------------------------------- 1 | using System; 2 | using System.Collections.Generic; 3 | using System.Linq; 4 | using Mono.Options; 5 | 6 | namespace UnityPerformanceBenchmarkReporter 7 | { 8 | public class OptionsParser 9 | { 10 | private bool help; 11 | private readonly string about = "The Unity Performance Benchmark Reporter enables the comparison of performance metric baselines and subsequent performance metrics (as generated using the Unity Test Runner with the Unity Performance Testing Extension) for use in test run pipelines and/or for generating an html report utilizing graphical visualizations."; 12 | 13 | private readonly string learnMore = 14 | "To learn more about the Unity Performance Benchmark Reporter visit the Unity Performance Benchmark Reporter GitHub wiki at https://github.com/Unity-Technologies/PerformanceBenchmarkReporter/wiki."; 15 | 16 | private readonly string commandLineOptionFormat = 17 | "// Command line option format\r\n--results=... [--baseline=\"Path to a baseline XML filename\"] [--reportdirpath=\"Path to where the report will be written\"]"; 18 | 19 | private readonly string example1 = "// Run reporter with one performance test result .xml file\r\n --results=\"G:\\My Drive\\XRPerfRuns\\results\\results.xml\""; 20 | private readonly string example2 = "// Run reporter with one performance test result .json file\r\n--format=json --results=\"G:\\My Drive\\XRPerfRuns\\results\\results.json\""; 21 | private readonly string example3 = "// Run reporter with one performance test result .json file which is in v1 format \r\n--dataversion=1 --format=json --results=\"G:\\My Drive\\XRPerfRuns\\results\\results.json\""; 22 | private readonly string example4 = "// Run reporter against a directory containing one or more performance test result files of the same file type\r\n --format=json --results=\"G:\\My Drive\\XRPerfRuns\\results\" "; 23 | private readonly string example5 = "// Run reporter against a directory containing one or more performance test result files, and a baseline result file. (File extension can be xml or json but both must be the same)\r\n--results=\"G:\\My Drive\\XRPerfRuns\\results\" --baseline=\"G:\\My Drive\\XRPerfRuns\\baselines\\baseline.xml\" "; 24 | 25 | 26 | public enum ResultType 27 | { 28 | Test, 29 | Baseline 30 | } 31 | 32 | public void ParseOptions(PerformanceBenchmark performanceBenchmark, IEnumerable args) 33 | { 34 | var os = GetOptions(performanceBenchmark); 35 | 36 | try 37 | { 38 | var remaining = os.Parse(args); 39 | 40 | if (help) 41 | { 42 | ShowHelp(string.Empty, os); 43 | } 44 | 45 | if (!performanceBenchmark.ResultFilePaths.Any() && !performanceBenchmark.ResultDirectoryPaths.Any()) 46 | { 47 | ShowHelp("Missing required option --results=(filePath|directoryPath)", os); 48 | } 49 | 50 | if (remaining.Any()) 51 | { 52 | var errorMessage = string.Format("Unknown option: '{0}.\r\n'", remaining[0]); 53 | ShowHelp(errorMessage, os); 54 | } 55 | } 56 | catch (Exception e) 57 | { 58 | ShowHelp(string.Format("Error encountered while parsing option: {0}.\r\n", e.Message), os); 59 | } 60 | } 61 | 62 | private OptionSet GetOptions(PerformanceBenchmark performanceBenchmark) 63 | { 64 | var optionsSet = new OptionSet(); 65 | optionsSet.Add("?|help|h", "Prints out the options.", option => help = option != null); 66 | optionsSet.Add("dataversion|version=", "Sets Expected Perf Data Version for Results and Baseline Files (1 = V1 2 = V2). Versions of Unity Perf Framework 2.0 or newer will use the V2 data format. If no arg is provided we assume the format is V2", version => performanceBenchmark.SetDataVersion(version)); 67 | optionsSet.Add("fileformat|format=", "Sets Expected File Format for Results and Baseline Files. If no arg is provided we assume the format is XML", filtype => performanceBenchmark.SetFileType(filtype)); 68 | optionsSet.Add( 69 | "results|testresultsxmlsource=", 70 | "REQUIRED - Path to a test result filename OR directory. Directories are searched resursively. You can repeat this option with multiple result file or directory paths.", 71 | xmlsource => performanceBenchmark.AddSourcePath(xmlsource, "results", ResultType.Test)); 72 | optionsSet.Add( 73 | "baseline|baselinexmlsource:", "OPTIONAL - Path to a baseline filename.", 74 | xmlsource => performanceBenchmark.AddSourcePath(xmlsource, "baseline", ResultType.Baseline)); 75 | optionsSet.Add( 76 | "report|reportdirpath:", "OPTIONAL - Path to where the report will be written. Default is current working directory.", 77 | performanceBenchmark.AddReportDirPath); 78 | optionsSet.Add("ignore:", 79 | "send in ; seperated list of metric names. These metrics are ignored and not compared even if they both exist on baseline and results", 80 | option => performanceBenchmark.SetIgnoredMetrics(option)); 81 | optionsSet.Add("failonbaseline", 82 | "Enable return '1' by the reporter if a baseline is passed in and one or more matching configs is out of threshold. Disabled is default. Use option to enable, or use option and append '-' to explicitly disable.", 83 | option => performanceBenchmark.FailOnBaseline = option != null); 84 | return optionsSet; 85 | } 86 | 87 | private void ShowHelp(string message, OptionSet optionSet) 88 | { 89 | if (!string.IsNullOrEmpty(message)) 90 | { 91 | Console.ForegroundColor = ConsoleColor.Red; 92 | Console.Error.WriteLine(message); 93 | Console.ResetColor(); 94 | } 95 | 96 | Console.WriteLine(about + "\r\n"); 97 | Console.WriteLine(learnMore + "\r\n"); 98 | Console.WriteLine("Usage is:" + "\r\n"); 99 | Console.WriteLine(commandLineOptionFormat + "\r\n"); 100 | Console.WriteLine(example1 + "\r\n"); 101 | Console.WriteLine(example2 + "\r\n"); 102 | Console.WriteLine(example3 + "\r\n"); 103 | Console.WriteLine(example4 + "\r\n"); 104 | Console.WriteLine(example5 + "\r\n"); 105 | Console.WriteLine("Options: \r\n"); 106 | optionSet.WriteOptionDescriptions(Console.Error); 107 | Environment.Exit(-1); 108 | } 109 | } 110 | } 111 | -------------------------------------------------------------------------------- /UnityPerformanceBenchmarkReporter/PerformanceBenchmark.cs: -------------------------------------------------------------------------------- 1 | using System; 2 | using System.Collections.Generic; 3 | using System.ComponentModel; 4 | using System.IO; 5 | using System.Linq; 6 | using UnityPerformanceBenchmarkReporter.Entities; 7 | 8 | namespace UnityPerformanceBenchmarkReporter 9 | { 10 | public class PerformanceBenchmark 11 | { 12 | private readonly PerformanceTestRunProcessor performanceTestRunProcessor = new PerformanceTestRunProcessor(); 13 | 14 | public readonly TestRunMetadataProcessor TestRunMetadataProcessor; 15 | private ESupportedFileTypes fileExtension = ESupportedFileTypes.xml; 16 | public ESupportedFileTypes FileType { get { return fileExtension; } } 17 | 18 | public int DataVersion { get; private set; } = 2; 19 | public PerformanceBenchmark(Dictionary configFieldNames = null) 20 | { 21 | // Default significant figures to use for non-integer metrics if user doesn't specify another value. 22 | // Most values are in milliseconds or a count of something, so using more often creates an artificial baseline 23 | // failure based on insignificant digits equating to a microsecond, or less, time difference. The Unity Profiler only shows 24 | // up to 2 significant figures for milliseconds as well, so this is what folks are used to working with. 25 | SigFig = 2; 26 | TestRunMetadataProcessor = new TestRunMetadataProcessor(configFieldNames); 27 | } 28 | 29 | public HashSet ResultFilePaths { get; } = new HashSet(); 30 | public HashSet ResultDirectoryPaths { get; } = new HashSet(); 31 | public HashSet BaselineFilePaths { get; } = new HashSet(); 32 | public uint SigFig { get; } 33 | public string ReportDirPath { get; private set; } 34 | public bool FailOnBaseline { get; set; } 35 | 36 | 37 | public bool BaselineResultFilesExist => BaselineFilePaths.Any(); 38 | 39 | public bool ResultFilesExist => ResultFilePaths.Any() || ResultDirectoryPaths.Any(); 40 | 41 | public List IgnoredMetrics = new List(); 42 | 43 | 44 | public void AddPerformanceTestRunResults( 45 | IParser testResultParser, 46 | List performanceTestRunResults, 47 | List testResults, 48 | List baselineTestResults) 49 | { 50 | AddTestResults(testResultParser, performanceTestRunResults, testResults, baselineTestResults, 51 | ResultDirectoryPaths, ResultFilePaths); 52 | } 53 | 54 | public void AddBaselinePerformanceTestRunResults( 55 | IParser testResultParser, 56 | List baselinePerformanceTestRunResults, 57 | List baselineTestResults) 58 | { 59 | AddTestResults(testResultParser, baselinePerformanceTestRunResults, baselineTestResults, 60 | baselineTestResults, null, BaselineFilePaths, true); 61 | } 62 | 63 | private void AddTestResults( 64 | IParser testResultParser, 65 | List testRunResults, 66 | List testResults, 67 | List baselineTestResults, 68 | HashSet directoryPaths, 69 | HashSet fileNamePaths, 70 | bool isBaseline = false) 71 | { 72 | if (!isBaseline && directoryPaths != null && directoryPaths.Any()) 73 | { 74 | foreach (var directory in directoryPaths) 75 | { 76 | var fileNames = GetAllFileNames(directory); 77 | 78 | foreach (var fileName in fileNames) 79 | { 80 | fileNamePaths.Add(fileName); 81 | } 82 | } 83 | } 84 | 85 | if (fileNamePaths.Any()) 86 | { 87 | var perfTestRuns = new List>(); 88 | 89 | foreach (var fileNamePath in fileNamePaths) 90 | { 91 | var performanceTestRun = testResultParser.Parse(fileNamePath, DataVersion); 92 | if (performanceTestRun != null && performanceTestRun.Results.Any()) 93 | { 94 | perfTestRuns.Add( 95 | new KeyValuePair(fileNamePath, performanceTestRun)); 96 | } 97 | } 98 | 99 | perfTestRuns.Sort((run1, run2) => string.Compare(run1.Key, run2.Key, StringComparison.Ordinal)); 100 | var resultFilesOrderedByResultName = perfTestRuns.ToArray(); 101 | 102 | for (var i = 0; i < resultFilesOrderedByResultName.Length; i++) 103 | { 104 | var performanceTestRun = 105 | testResultParser.Parse(resultFilesOrderedByResultName[i].Key, DataVersion); 106 | 107 | if (performanceTestRun != null && performanceTestRun.Results.Any()) 108 | { 109 | var results = performanceTestRunProcessor.GetTestResults(performanceTestRun); 110 | if (!results.Any()) 111 | { 112 | Console.ForegroundColor = ConsoleColor.Yellow; 113 | Console.WriteLine("No performance test data found to report in: {0}", 114 | resultFilesOrderedByResultName[i].Key); 115 | Console.ResetColor(); 116 | continue; 117 | } 118 | 119 | testResults.AddRange(results); 120 | 121 | performanceTestRunProcessor.UpdateTestResultsBasedOnBaselineResults(baselineTestResults, IgnoredMetrics, testResults, SigFig); 122 | 123 | TestRunMetadataProcessor.ProcessMetadata(performanceTestRun, resultFilesOrderedByResultName[i].Key); 124 | 125 | var performanceTestRunResult = performanceTestRunProcessor.CreateTestRunResult( 126 | performanceTestRun, 127 | results, 128 | Path.GetFileNameWithoutExtension(resultFilesOrderedByResultName[i].Key), 129 | isBaseline); 130 | testRunResults.Add(performanceTestRunResult); 131 | } 132 | } 133 | } 134 | } 135 | 136 | public void SetIgnoredMetrics(string metrics) 137 | { 138 | IgnoredMetrics = metrics.Split(';').ToList(); 139 | } 140 | 141 | public void SetDataVersion(string version) 142 | { 143 | if (int.TryParse(version, out int result)) 144 | { 145 | if (result > 0 && result < 3) 146 | DataVersion = result; 147 | else 148 | throw new ArgumentException($"{version} is not a valid data format version. Please pass 1 or 2"); 149 | } 150 | else 151 | { 152 | throw new ArgumentException($"{version} is not a valid data format version"); 153 | } 154 | } 155 | 156 | public void SetFileType(string filetype) 157 | { 158 | if (String.IsNullOrEmpty(filetype)) 159 | return; 160 | 161 | if (Enum.TryParse(filetype, true, out ESupportedFileTypes result)) 162 | { 163 | fileExtension = result; 164 | } 165 | else 166 | { 167 | throw new ArgumentException($"{filetype} is not a valid file format"); 168 | } 169 | } 170 | 171 | private IEnumerable GetAllFileNames(string directory) 172 | { 173 | var dir = new DirectoryInfo(directory); 174 | var FileNames = dir.GetFiles("*" + fileExtension, SearchOption.AllDirectories) 175 | .Select(f => f.FullName); 176 | return FileNames; 177 | } 178 | 179 | public void AddSourcePath(string sourcePath, string optionName, OptionsParser.ResultType resultType) 180 | { 181 | System.Console.WriteLine($" Adding Source Path : {sourcePath}"); 182 | System.Console.WriteLine($""); 183 | 184 | if (string.IsNullOrEmpty(sourcePath)) 185 | { 186 | throw new ArgumentNullException(sourcePath); 187 | } 188 | 189 | if (string.IsNullOrEmpty(optionName)) 190 | { 191 | throw new ArgumentNullException(optionName); 192 | } 193 | 194 | if (sourcePath.EndsWith(fileExtension.ToString())) 195 | { 196 | ProcessAsFile(sourcePath, optionName, resultType); 197 | } 198 | else 199 | { 200 | ProcessAsDirectory(sourcePath, optionName, resultType); 201 | } 202 | 203 | 204 | 205 | 206 | } 207 | 208 | private void ProcessAsDirectory(string sourcePath, string optionName, OptionsParser.ResultType resultType) 209 | { 210 | if (!Directory.Exists(sourcePath)) 211 | { 212 | throw new ArgumentException(string.Format("{0} directory `{1}` cannot be found", optionName, 213 | sourcePath)); 214 | } 215 | 216 | var fileNames = GetAllFileNames(sourcePath).ToArray(); 217 | if (!fileNames.Any()) 218 | { 219 | throw new ArgumentException(string.Format("{0} directory `{1}` doesn't contain any {2} files.", 220 | optionName, 221 | sourcePath, FileType)); 222 | } 223 | 224 | switch (resultType) 225 | { 226 | case OptionsParser.ResultType.Test: 227 | ResultDirectoryPaths.Add(sourcePath); 228 | break; 229 | case OptionsParser.ResultType.Baseline: 230 | foreach (var filename in fileNames) 231 | { 232 | BaselineFilePaths.Add(filename); 233 | } 234 | 235 | break; 236 | default: 237 | throw new InvalidEnumArgumentException(resultType.ToString()); 238 | } 239 | } 240 | 241 | private void ProcessAsFile(string sourcePath, string optionName, OptionsParser.ResultType resultType) 242 | { 243 | if (!File.Exists(sourcePath)) 244 | { 245 | throw new ArgumentException(string.Format("{0} file `{1}` cannot be found", optionName, sourcePath)); 246 | } 247 | 248 | switch (resultType) 249 | { 250 | case OptionsParser.ResultType.Test: 251 | ResultFilePaths.Add(sourcePath); 252 | break; 253 | case OptionsParser.ResultType.Baseline: 254 | BaselineFilePaths.Add(sourcePath); 255 | break; 256 | default: 257 | throw new InvalidEnumArgumentException(resultType.ToString()); 258 | } 259 | } 260 | 261 | public void AddReportDirPath(string reportDirectoryPath) 262 | { 263 | ReportDirPath = reportDirectoryPath; 264 | } 265 | } 266 | } -------------------------------------------------------------------------------- /UnityPerformanceBenchmarkReporter/PerformanceTestRunProcessor.cs: -------------------------------------------------------------------------------- 1 | using System; 2 | using System.Collections.Generic; 3 | using System.Linq; 4 | using UnityPerformanceBenchmarkReporter.Entities; 5 | 6 | namespace UnityPerformanceBenchmarkReporter 7 | { 8 | internal enum MeasurementResult 9 | { 10 | Neutral = 0, 11 | Regression = 1, 12 | Progression = 2, 13 | RegressionKnown = 3 14 | } 15 | 16 | public class PerformanceTestRunProcessor 17 | { 18 | public List GetTestResults( 19 | PerformanceTestRun performanceTestRun) 20 | { 21 | var mergedTestExecutions = MergeTestExecutions(performanceTestRun); 22 | var performanceTestResults = new List(); 23 | foreach (var testName in mergedTestExecutions.Keys) 24 | { 25 | var performanceTestResult = new TestResult 26 | { 27 | TestName = testName, 28 | TestCategories = performanceTestRun.Results.First(r => r.TestName == testName).TestCategories, 29 | TestVersion = performanceTestRun.Results.First(r => r.TestName == testName).TestVersion, 30 | State = (int)TestState.Success, 31 | SampleGroupResults = new List() 32 | }; 33 | foreach (var sampleGroup in mergedTestExecutions[testName]) 34 | { 35 | var sampleGroupResult = new SampleGroupResult 36 | { 37 | SampleGroupName = sampleGroup.Definition.Name, 38 | SampleUnit = sampleGroup.Definition.SampleUnit.ToString(), 39 | IncreaseIsBetter = sampleGroup.Definition.IncreaseIsBetter, 40 | Threshold = sampleGroup.Definition.Threshold, 41 | AggregationType = sampleGroup.Definition.AggregationType.ToString(), 42 | Percentile = sampleGroup.Definition.Percentile, 43 | Min = sampleGroup.Min, 44 | Max = sampleGroup.Max, 45 | Median = sampleGroup.Median, 46 | Average = sampleGroup.Average, 47 | StandardDeviation = sampleGroup.StandardDeviation, 48 | PercentileValue = sampleGroup.PercentileValue, 49 | ContainsKnownIssue = sampleGroup.Definition.ContainsKnownIssue, 50 | KnownIssueDetails = sampleGroup.Definition.KnownIssueDetails, 51 | Sum = sampleGroup.Sum, 52 | Zeroes = sampleGroup.Zeroes, 53 | SampleCount = sampleGroup.SampleCount, 54 | BaselineValue = -1, 55 | AggregatedValue = GetAggregatedSampleValue(sampleGroup) 56 | }; 57 | 58 | performanceTestResult.SampleGroupResults.Add(sampleGroupResult); 59 | } 60 | performanceTestResults.Add(performanceTestResult); 61 | } 62 | return performanceTestResults; 63 | } 64 | 65 | public void UpdateTestResultsBasedOnBaselineResults(List baselineTestResults, List ignoredMetrics, 66 | List testResults, uint sigfig) 67 | { 68 | foreach (var testResult in testResults) 69 | { 70 | // If the baseline results doesn't have a matching TestName for this result, skip it 71 | if (baselineTestResults.All(r => r.TestName != testResult.TestName)) continue; 72 | 73 | // Get the corresponding baseline testname samplegroupresults for this result's testname 74 | var baselineSampleGroupResults = baselineTestResults.First(r => r.TestName == testResult.TestName).SampleGroupResults; 75 | 76 | foreach (var sampleGroupResult in testResult.SampleGroupResults) 77 | { 78 | // if we have a corresponding baseline samplegroupname in this sampleGroupResult, compare them 79 | if ( baselineSampleGroupResults.Any(sg => sg.SampleGroupName == sampleGroupResult.SampleGroupName && !ignoredMetrics.Contains(sampleGroupResult.SampleGroupName)) ) 80 | { 81 | // Get the baselineSampleGroupResult that corresponds to this SampleGroupResults sample group name 82 | var baselineSampleGroupResult = baselineSampleGroupResults.First(sg => sg.SampleGroupName == sampleGroupResult.SampleGroupName); 83 | 84 | // update this samplegroupresults baselinevalue and threshold to be that of the baselinesamplegroup so we can perform an accurate assessement of 85 | // whether or not a regression has occurred. 86 | sampleGroupResult.BaselineValue = baselineSampleGroupResult.AggregatedValue; 87 | sampleGroupResult.Threshold = baselineSampleGroupResult.Threshold; 88 | sampleGroupResult.StandardDeviation = baselineSampleGroupResult.StandardDeviation; 89 | sampleGroupResult.IncreaseIsBetter = baselineSampleGroupResult.IncreaseIsBetter; 90 | 91 | var res = DeterminePerformanceResult(sampleGroupResult, sigfig); 92 | 93 | if (res == MeasurementResult.Regression) 94 | { 95 | sampleGroupResult.Regressed = true; 96 | sampleGroupResult.Progressed = false; 97 | sampleGroupResult.RegressedKnown = false; 98 | } 99 | else if (res == MeasurementResult.Progression) 100 | { 101 | sampleGroupResult.Regressed = false; 102 | sampleGroupResult.Progressed = true; 103 | sampleGroupResult.RegressedKnown = false; 104 | }else if(res == MeasurementResult.RegressionKnown){ 105 | sampleGroupResult.Regressed = true; 106 | sampleGroupResult.Progressed = false; 107 | sampleGroupResult.RegressedKnown = true; 108 | } 109 | } 110 | } 111 | 112 | if (testResult.SampleGroupResults.Any(r => r.Regressed && r.RegressedKnown == false)) 113 | { 114 | testResult.State = (int)TestState.Failure; 115 | } 116 | } 117 | } 118 | 119 | private Dictionary> MergeTestExecutions(PerformanceTestRun performanceTestRun) 120 | { 121 | var mergedTestExecutions = new Dictionary>(); 122 | var testNames = performanceTestRun.Results.Select(te => te.TestName).Where(t => !String.IsNullOrEmpty(t)).Distinct().ToList(); 123 | foreach (var testName in testNames) 124 | { 125 | var executions = performanceTestRun.Results.Where(te => te.TestName == testName); 126 | var sampleGroups = new List(); 127 | foreach (var execution in executions) 128 | { 129 | foreach (var sampleGroup in execution.SampleGroups) 130 | { 131 | if (sampleGroups.Any(sg => sg.Definition.Name == sampleGroup.Definition.Name)) 132 | { 133 | sampleGroups.First(sg => sg.Definition.Name == sampleGroup.Definition.Name).Samples 134 | .AddRange(sampleGroup.Samples); 135 | } 136 | else 137 | { 138 | sampleGroups.Add(sampleGroup); 139 | } 140 | } 141 | } 142 | 143 | mergedTestExecutions.Add(testName, sampleGroups); 144 | } 145 | return mergedTestExecutions; 146 | } 147 | 148 | private double GetAggregatedSampleValue(SampleGroup sampleGroup) 149 | { 150 | double aggregatedSampleValue; 151 | switch (sampleGroup.Definition.AggregationType) 152 | { 153 | case AggregationType.Average: 154 | aggregatedSampleValue = sampleGroup.Average; 155 | break; 156 | case AggregationType.Min: 157 | aggregatedSampleValue = sampleGroup.Min; 158 | break; 159 | case AggregationType.Max: 160 | aggregatedSampleValue = sampleGroup.Max; 161 | break; 162 | case AggregationType.Median: 163 | aggregatedSampleValue = sampleGroup.Median; 164 | break; 165 | case AggregationType.Percentile: 166 | aggregatedSampleValue = sampleGroup.PercentileValue; 167 | break; 168 | default: 169 | throw new ArgumentOutOfRangeException(string.Format("Unhandled aggregation type {0}", sampleGroup.Definition.AggregationType)); 170 | } 171 | return aggregatedSampleValue; 172 | } 173 | 174 | private MeasurementResult DeterminePerformanceResult(SampleGroupResult sampleGroup, uint sigFig) 175 | { 176 | var measurementResult = MeasurementResult.Neutral; 177 | 178 | var baselineval = sampleGroup.BaselineValue;// + sampleGroup.StandardDeviation ; //TODO Add flag to use standard deviation or not 179 | var positiveThresholdValue = baselineval + (baselineval * sampleGroup.Threshold); 180 | var negativeThresholdValue = baselineval - (baselineval * sampleGroup.Threshold); 181 | 182 | 183 | if (sampleGroup.IncreaseIsBetter) 184 | { 185 | if (sampleGroup.AggregatedValue.TruncToSigFig(sigFig) < negativeThresholdValue.TruncToSigFig(sigFig)) 186 | { 187 | measurementResult = MeasurementResult.Regression; 188 | 189 | if(sampleGroup.ContainsKnownIssue) 190 | measurementResult = MeasurementResult.RegressionKnown; 191 | } 192 | if (sampleGroup.AggregatedValue.TruncToSigFig(sigFig) > positiveThresholdValue.TruncToSigFig(sigFig)) 193 | { 194 | measurementResult = MeasurementResult.Progression; 195 | } 196 | } 197 | else 198 | { 199 | if (sampleGroup.AggregatedValue.TruncToSigFig(sigFig) > positiveThresholdValue.TruncToSigFig(sigFig)) 200 | { 201 | measurementResult = MeasurementResult.Regression; 202 | 203 | if(sampleGroup.ContainsKnownIssue) 204 | measurementResult = MeasurementResult.RegressionKnown; 205 | } 206 | if (sampleGroup.AggregatedValue.TruncToSigFig(sigFig) < negativeThresholdValue.TruncToSigFig(sigFig)) 207 | { 208 | measurementResult = MeasurementResult.Progression; 209 | } 210 | } 211 | return measurementResult; 212 | } 213 | 214 | public PerformanceTestRunResult CreateTestRunResult(PerformanceTestRun runResults, 215 | List testResults, string resultName, bool isBaseline = false) 216 | { 217 | var performanceTestRunResult = new PerformanceTestRunResult 218 | { 219 | ResultName = resultName, 220 | IsBaseline = isBaseline, 221 | TestSuite = runResults.TestSuite, 222 | StartTime = 223 | new DateTime(1970, 1, 1, 0, 0, 0, DateTimeKind.Utc).AddMilliseconds( 224 | runResults.StartTime), 225 | TestResults = testResults, 226 | PlayerSystemInfo = runResults.PlayerSystemInfo, 227 | EditorVersion = runResults.EditorVersion, 228 | BuildSettings = runResults.BuildSettings, 229 | ScreenSettings = runResults.ScreenSettings, 230 | QualitySettings = runResults.QualitySettings, 231 | PlayerSettings = runResults.PlayerSettings 232 | }; 233 | return performanceTestRunResult; 234 | } 235 | } 236 | } 237 | -------------------------------------------------------------------------------- /UnityPerformanceBenchmarkReporter/Program.cs: -------------------------------------------------------------------------------- 1 | using System; 2 | using System.Collections.Generic; 3 | using System.Linq; 4 | using UnityPerformanceBenchmarkReporter.Entities; 5 | using UnityPerformanceBenchmarkReporter.Report; 6 | 7 | namespace UnityPerformanceBenchmarkReporter 8 | { 9 | internal class Program 10 | { 11 | private static int indentLevel; 12 | 13 | private static readonly Dictionary ExcludedConfigFieldNames = new Dictionary 14 | { 15 | {typeof(EditorVersion), new [] 16 | { 17 | "DateSeconds", 18 | "RevisionValue" 19 | }}, 20 | {typeof(BuildSettings), new [] 21 | { 22 | "DevelopmentPlayer" 23 | }}, 24 | {typeof(PlayerSystemInfo), new [] 25 | { 26 | "XrModel" 27 | }}, 28 | {typeof(PlayerSettings), new [] 29 | { 30 | "MtRendering", // Hidden because we have a calculated field, RenderThreadingMode, that provides a more succinct value (SingleThreaded, MultiThreaded, GfxJobs) 31 | "GraphicsJobs", // Hidden because we have a calculated field, RenderThreadingMode, that provides a more succinct value (SingleThreaded, MultiThreaded, GfxJobs) 32 | "VrSupported", // Hidden because this value doesn't seem to be coming through as 'True' when it should be true. 33 | "AndroidMinimumSdkVersion", 34 | "AndroidTargetSdkVersion" 35 | }} 36 | }; 37 | 38 | private static int Main(string[] args) 39 | { 40 | var aggregateTestRunResults = new List(); 41 | var baselinePerformanceTestRunResults = new List(); 42 | var baselineTestResults = new List(); 43 | var performanceTestRunResults = new List(); 44 | var testResults = new List(); 45 | var performanceBenchmark = new PerformanceBenchmark(ExcludedConfigFieldNames); 46 | var optionsParser = new OptionsParser(); 47 | 48 | optionsParser.ParseOptions(performanceBenchmark, args); 49 | 50 | IParser testResultParser = null; 51 | 52 | if (performanceBenchmark.FileType == ESupportedFileTypes.json) 53 | { 54 | testResultParser = new TestResultJsonParser(); 55 | } 56 | else if (performanceBenchmark.FileType == ESupportedFileTypes.xml) 57 | { 58 | testResultParser = new TestResultXmlParser(); 59 | } 60 | 61 | if (performanceBenchmark.BaselineResultFilesExist) 62 | { 63 | performanceBenchmark.AddBaselinePerformanceTestRunResults(testResultParser, baselinePerformanceTestRunResults, baselineTestResults); 64 | 65 | if (baselinePerformanceTestRunResults.Any()) 66 | { 67 | aggregateTestRunResults.AddRange(baselinePerformanceTestRunResults); 68 | } 69 | else 70 | { 71 | Environment.Exit(1); 72 | } 73 | } 74 | 75 | if (performanceBenchmark.ResultFilesExist) 76 | { 77 | performanceBenchmark.AddPerformanceTestRunResults(testResultParser, performanceTestRunResults, testResults, baselineTestResults); 78 | 79 | if (performanceTestRunResults.Any()) 80 | { 81 | aggregateTestRunResults.AddRange(performanceTestRunResults); 82 | } 83 | else 84 | { 85 | Environment.Exit(1); 86 | } 87 | } 88 | 89 | var performanceTestResults = new PerformanceTestRunResult[0]; 90 | 91 | // If we have a baseline 92 | if (aggregateTestRunResults.Any(a => a.IsBaseline)) 93 | { 94 | // Insert the baseline in the front of the array results; this way we can display the baseline first in the report 95 | Array.Resize(ref performanceTestResults, 1); 96 | performanceTestResults[0] = aggregateTestRunResults.First(a => a.IsBaseline); 97 | } 98 | 99 | var nonBaselineTestRunResults = aggregateTestRunResults.Where(a => !a.IsBaseline).ToList(); 100 | 101 | nonBaselineTestRunResults.Sort((run1, run2) => string.Compare(run1.ResultName, run2.ResultName, StringComparison.Ordinal)); 102 | 103 | 104 | foreach (var performanceTestRunResult in nonBaselineTestRunResults) 105 | { 106 | Array.Resize(ref performanceTestResults, performanceTestResults.Length + 1); 107 | performanceTestResults[performanceTestResults.Length - 1] = performanceTestRunResult; 108 | } 109 | 110 | performanceBenchmark.TestRunMetadataProcessor.PerformFinalMetadataUpdate(performanceBenchmark); 111 | 112 | var reportWriter = new ReportWriter(performanceBenchmark.TestRunMetadataProcessor); 113 | 114 | reportWriter.WriteReport( 115 | performanceTestResults, 116 | performanceBenchmark.SigFig, 117 | performanceBenchmark.ReportDirPath, 118 | performanceBenchmark.BaselineResultFilesExist); 119 | WriteProgressedTestsAndMetricsToConsole(performanceTestResults, performanceBenchmark); 120 | WriteRegressedKnownTestsAndMetricsToConsole(performanceTestResults, performanceBenchmark); 121 | int result = WriteFailedTestsAndMetricsToConsole(performanceTestResults, performanceBenchmark); 122 | WriteLine($"Finished with Result {result}"); 123 | return result; 124 | } 125 | 126 | private static int WriteFailedTestsAndMetricsToConsole(PerformanceTestRunResult[] performanceTestResults, PerformanceBenchmark performanceBenchmark) 127 | { 128 | var failedTestsExist = performanceTestResults.SelectMany(ptr => ptr.TestResults) 129 | .Any(tr => tr.State == (int)TestState.Failure); 130 | if (failedTestsExist) 131 | { 132 | WriteLine("FAILURE: One ore more performance test metric aggregations is out of threshold from the baseline value. REGRESSIONS!"); 133 | WriteLine("-------------------------------------"); 134 | WriteLine(" Performance tests with failed metrics"); 135 | WriteLine("-------------------------------------"); 136 | foreach (var performanceTestRunResult in performanceTestResults) 137 | { 138 | var failedTests = performanceTestRunResult.TestResults.Where(tr => tr.State == (int)TestState.Failure); 139 | if (failedTests.Any()) 140 | { 141 | foreach (var failedTest in failedTests) 142 | { 143 | ++indentLevel; 144 | WriteLine("{0}", failedTest.TestName); 145 | 146 | var regressedSgs = failedTest.SampleGroupResults.Where(sgr => sgr.Regressed); 147 | foreach (var sampleGroupResult in regressedSgs) 148 | { 149 | WriteLine("----"); 150 | WriteLine("Metric : {0}", sampleGroupResult.SampleGroupName); 151 | WriteLine("Aggregation : {0}", sampleGroupResult.AggregationType); 152 | WriteLine("Failed Value : {0,8:F2}", sampleGroupResult.AggregatedValue); 153 | WriteLine("Baseline Value: {0,8:F2}", sampleGroupResult.BaselineValue); 154 | WriteLine("Threshold % : {0,8:F2}", sampleGroupResult.Threshold); 155 | WriteLine("Actual Diff % : {0,8:F2}", Math.Abs(sampleGroupResult.AggregatedValue - sampleGroupResult.BaselineValue) / sampleGroupResult.BaselineValue); 156 | } 157 | --indentLevel; 158 | WriteLine("\r\n"); 159 | } 160 | } 161 | } 162 | } 163 | return performanceBenchmark.FailOnBaseline && failedTestsExist ? 1 : 0; 164 | } 165 | 166 | private static void WriteRegressedKnownTestsAndMetricsToConsole(PerformanceTestRunResult[] performanceTestResults, PerformanceBenchmark performanceBenchmark) 167 | { 168 | bool loggedHeader = false; 169 | var passedTestsExist = performanceTestResults.SelectMany(ptr => ptr.TestResults) 170 | .Any(tr => tr.State == (int)TestState.Success); 171 | if (passedTestsExist) 172 | { 173 | 174 | foreach (var performanceTestRunResult in performanceTestResults) 175 | { 176 | var passedTests = performanceTestRunResult.TestResults.Where(tr => tr.State == (int)TestState.Success); 177 | if (passedTests.Any()) 178 | { 179 | foreach (var tests in passedTests) 180 | { 181 | if (tests.SampleGroupResults.Any(sgr => sgr.RegressedKnown && sgr.Regressed)) 182 | { 183 | if (!loggedHeader) 184 | { 185 | loggedHeader = true; 186 | WriteLine("Info: One ore more performance test metric aggregations is out of threshold from the baseline value. KNOWN REGRESSIONS!"); 187 | WriteLine("-------------------------------------"); 188 | WriteLine(" Performance tests with Known Regressions metrics"); 189 | WriteLine("-------------------------------------"); 190 | } 191 | 192 | ++indentLevel; 193 | WriteLine("{0}", tests.TestName); 194 | 195 | var progressedSgs = tests.SampleGroupResults.Where(sgr => sgr.Progressed); 196 | foreach (var sampleGroupResult in progressedSgs) 197 | { 198 | WriteLine("----"); 199 | WriteLine("Metric : {0}", sampleGroupResult.SampleGroupName); 200 | WriteLine("Aggregation : {0}", sampleGroupResult.AggregationType); 201 | WriteLine("New Value : {0,8:F2}", sampleGroupResult.AggregatedValue); 202 | WriteLine("Baseline Value: {0,8:F2}", sampleGroupResult.BaselineValue); 203 | WriteLine("Threshold % : {0,8:F2}", sampleGroupResult.Threshold); 204 | WriteLine("Actual Diff % : {0,8:F2}", Math.Abs(sampleGroupResult.BaselineValue - sampleGroupResult.AggregatedValue) / sampleGroupResult.BaselineValue); 205 | WriteLine($"Known Issue: {sampleGroupResult.KnownIssueDetails}"); 206 | } 207 | --indentLevel; 208 | WriteLine("\r\n"); 209 | } 210 | } 211 | } 212 | } 213 | } 214 | 215 | } 216 | 217 | private static void WriteProgressedTestsAndMetricsToConsole(PerformanceTestRunResult[] performanceTestResults, PerformanceBenchmark performanceBenchmark) 218 | { 219 | bool loggedHeader = false; 220 | var passedTestsExist = performanceTestResults.SelectMany(ptr => ptr.TestResults) 221 | .Any(tr => tr.State == (int)TestState.Success); 222 | if (passedTestsExist) 223 | { 224 | 225 | foreach (var performanceTestRunResult in performanceTestResults) 226 | { 227 | var passedTests = performanceTestRunResult.TestResults.Where(tr => tr.State == (int)TestState.Success); 228 | if (passedTests.Any()) 229 | { 230 | foreach (var tests in passedTests) 231 | { 232 | if (tests.SampleGroupResults.Any(sgr => sgr.Progressed)) 233 | { 234 | if (!loggedHeader) 235 | { 236 | loggedHeader = true; 237 | WriteLine("Info: One ore more performance test metric aggregations is out of threshold from the baseline value. PROGRESSIONS!"); 238 | WriteLine("-------------------------------------"); 239 | WriteLine(" Performance tests with Progressed metrics"); 240 | WriteLine("-------------------------------------"); 241 | } 242 | 243 | ++indentLevel; 244 | WriteLine("{0}", tests.TestName); 245 | 246 | var progressedSgs = tests.SampleGroupResults.Where(sgr => sgr.Progressed); 247 | foreach (var sampleGroupResult in progressedSgs) 248 | { 249 | WriteLine("----"); 250 | WriteLine("Metric : {0}", sampleGroupResult.SampleGroupName); 251 | WriteLine("Aggregation : {0}", sampleGroupResult.AggregationType); 252 | WriteLine("New Value : {0,8:F2}", sampleGroupResult.AggregatedValue); 253 | WriteLine("Baseline Value: {0,8:F2}", sampleGroupResult.BaselineValue); 254 | WriteLine("Threshold % : {0,8:F2}", sampleGroupResult.Threshold); 255 | WriteLine("Actual Diff % : {0,8:F2}", Math.Abs(sampleGroupResult.BaselineValue - sampleGroupResult.AggregatedValue) / sampleGroupResult.BaselineValue); 256 | } 257 | --indentLevel; 258 | WriteLine("\r\n"); 259 | } 260 | } 261 | } 262 | } 263 | } 264 | 265 | } 266 | 267 | private static void WriteLine(string format, params object[] args) 268 | { 269 | Console.Write(new string('\t', indentLevel)); 270 | Console.WriteLine(format, args); 271 | } 272 | } 273 | } 274 | -------------------------------------------------------------------------------- /UnityPerformanceBenchmarkReporter/Report/UnityLogo.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Unity-Technologies/PerformanceBenchmarkReporter/c403bfed95d9c1c14b644943a8ea662abb7d483e/UnityPerformanceBenchmarkReporter/Report/UnityLogo.png -------------------------------------------------------------------------------- /UnityPerformanceBenchmarkReporter/Report/help-hover.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Unity-Technologies/PerformanceBenchmarkReporter/c403bfed95d9c1c14b644943a8ea662abb7d483e/UnityPerformanceBenchmarkReporter/Report/help-hover.png -------------------------------------------------------------------------------- /UnityPerformanceBenchmarkReporter/Report/help.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Unity-Technologies/PerformanceBenchmarkReporter/c403bfed95d9c1c14b644943a8ea662abb7d483e/UnityPerformanceBenchmarkReporter/Report/help.png -------------------------------------------------------------------------------- /UnityPerformanceBenchmarkReporter/Report/styles.css: -------------------------------------------------------------------------------- 1 | .body * { 2 | font-family: 'Arial', 'Helvetica', 'Helvetica Neue', sans-serif !important; 3 | } 4 | 5 | h1 { 6 | font-size: 3vh; 7 | overflow-wrap: break-word; 8 | } 9 | 10 | h3 { 11 | font-size: 1.5vh; 12 | overflow-wrap: break-word; 13 | } 14 | 15 | h5 { 16 | font-size: 1vh; 17 | overflow-wrap: break-word; 18 | } 19 | 20 | p { 21 | line-height: 25% 22 | } 23 | 24 | th { 25 | border-bottom-style: solid; 26 | border-bottom-color: lightgray; 27 | } 28 | 29 | .targetvalue { 30 | color: black; 31 | } 32 | 33 | .help { 34 | width: 100%; 35 | } 36 | 37 | .toggleconfigwrapper { 38 | display: inline-flex; 39 | align-items: flex-start; 40 | width: 100%; 41 | } 42 | 43 | .helpwrapper { 44 | display: inline; 45 | background-image: url("help.png"); 46 | height: 4vh; 47 | width: 4vh; 48 | background-repeat: no-repeat; 49 | float: right; 50 | background-size: 100%; 51 | padding: 10px; 52 | } 53 | 54 | .helpwrapper:hover { 55 | background-image: url("help-hover.png"); 56 | } 57 | 58 | .titletable { 59 | height: 80%; 60 | width: 100%; 61 | padding: 5px 5px 5px 5px; 62 | } 63 | 64 | .logocell { 65 | padding: 10px; 66 | background-color: black; 67 | } 68 | 69 | .logo { 70 | width: 30%; 71 | vertical-align: bottom; 72 | max-width: 400px; 73 | } 74 | 75 | .warning { 76 | width: 2vh; 77 | vertical-align: middle; 78 | max-width: 200px; 79 | margin: 5px; 80 | } 81 | 82 | .titlecell { 83 | padding-left: 10px; 84 | background-color: black; 85 | } 86 | 87 | .title { 88 | font-family: 'Arial', 'Helvetica', 'Helvetica Neue', sans-serif; 89 | color: white; 90 | text-align: left; 91 | vertical-align: bottom 92 | } 93 | 94 | .testconfigtable { 95 | padding: 5px 5px 5px 5px; 96 | width: 100%; 97 | } 98 | 99 | .testconfig { 100 | display: none 101 | } 102 | 103 | 104 | 105 | .testnamecell { 106 | background-color: lightgray; 107 | padding: 0px 5px 0px 5px; 108 | } 109 | 110 | .testname { 111 | color: black; 112 | } 113 | 114 | .initialbutton { 115 | font-size: 1.25vh; 116 | background-color: #2196F3; 117 | border: none; 118 | color: white; 119 | padding: 15px 25px; 120 | cursor: pointer; 121 | border-radius: 4px; 122 | font-weight: bold; 123 | margin: 5px 5px 5px 0px; 124 | } 125 | 126 | .button { 127 | font-size: 1.25vh; 128 | background-color: #3e6892; 129 | border: none; 130 | color: white; 131 | padding: 15px 25px; 132 | cursor: pointer; 133 | border-radius: 4px; 134 | font-weight: bold; 135 | margin: 5px 5px 5px 0px; 136 | } 137 | 138 | div > .configwarning { 139 | font-size: 1.25vh; 140 | font-weight: bold; 141 | visibility: hidden; 142 | width: 120px; 143 | background-color: #f28034; 144 | color: white; 145 | text-align: left; 146 | border-radius: 4px; 147 | z-index: 1; 148 | top: -5px; 149 | left: 105%; 150 | padding: 10px 5px 5px 5px; 151 | font-family: 'Arial', 'Helvetica', 'Helvetica Neue', sans-serif; 152 | margin: 5px; 153 | } 154 | 155 | div:hover > .configwarning { 156 | visibility: visible; 157 | } 158 | 159 | 160 | .button:hover { 161 | background-color: #2196F3 !important; 162 | } 163 | 164 | .flex { 165 | display: flex; 166 | } 167 | 168 | .buttondiv { 169 | display: inline; 170 | } 171 | 172 | .initialbutton:hover { 173 | background-color: #2196F3 !important; 174 | } 175 | 176 | 177 | .buttonheader { 178 | font-size: 1.25vh; 179 | font-weight: bold; 180 | padding: 5px 5px 5px 0px; 181 | } 182 | 183 | .showedfailedtests { 184 | padding: 5px 5px 5px 0px; 185 | display: inline-flex; 186 | align-items: flex-start; 187 | width: 100%; 188 | } 189 | 190 | 191 | .systeminfo > div { 192 | flex: 1 1 160px; 193 | margin: 5px; 194 | } 195 | 196 | .systeminfowide > div { 197 | flex: 1 1 160px; 198 | margin: 5px; 199 | } 200 | 201 | .typename { 202 | text-align: left; 203 | font-weight: bold; 204 | font-size: 1.35vh; 205 | background-color: lightgray; 206 | color: black; 207 | font-family: 'Arial', 'Helvetica', 'Helvetica Neue', sans-serif; 208 | padding: 5px 5px 5px 10px; 209 | } 210 | 211 | .typenamewarning { 212 | text-align: left; 213 | font-weight: bold; 214 | font-size: 1.35vh; 215 | background-color: #f28034; 216 | color: white; 217 | font-family: 'Arial', 'Helvetica', 'Helvetica Neue', sans-serif; 218 | padding: 5px 5px 5px 10px; 219 | } 220 | 221 | .fieldgroup { 222 | display: inline-block !important; 223 | border: 1px solid lightgray; 224 | border-radius: 4px; 225 | margin: 5px; 226 | vertical-align: top; 227 | } 228 | 229 | .fieldgroupwarning { 230 | display: inline-block !important; 231 | border: 1px solid #f28034; 232 | border-radius: 4px; 233 | margin: 5px; 234 | vertical-align: top; 235 | } 236 | 237 | .fieldname { 238 | font-weight: bold; 239 | font-size: 1.25vh; 240 | overflow-wrap: break-word; 241 | font-family: 'Arial', 'Helvetica', 'Helvetica Neue', sans-serif; 242 | padding: 4px; 243 | } 244 | 245 | .fieldnamewarning { 246 | background-color: #f28034; 247 | color: white; 248 | font-weight: bold; 249 | font-size: 1.25vh; 250 | overflow-wrap: break-word; 251 | font-family: 'Arial', 'Helvetica', 'Helvetica Neue', sans-serif; 252 | padding: 4px; 253 | } 254 | 255 | .fieldvalue { 256 | font-size: 1.1vh; 257 | padding: 0px 4px 4px 4px; 258 | } 259 | 260 | .fieldvaluewarning { 261 | font-size: 1.1vh; 262 | color: #f28034; 263 | } 264 | 265 | .warningtable { 266 | padding: 5px 5px 5px 5px; 267 | width: 100%; 268 | text-align: left; 269 | } 270 | 271 | .warningtable > tbody > tr > td { 272 | padding: 5px; 273 | } 274 | 275 | .warningtable > tbody > tr:nth-child(even) { 276 | background-color: #f2f2f2; 277 | } 278 | 279 | pre { 280 | white-space: pre-wrap; /* Since CSS 2.1 */ 281 | white-space: -moz-pre-wrap; /* Mozilla, since 1999 */ 282 | white-space: -pre-wrap; /* Opera 4-6 */ 283 | white-space: -o-pre-wrap; /* Opera 7 */ 284 | word-wrap: break-word; /* Internet Explorer 5.5+ */ 285 | margin: 5px; 286 | } 287 | 288 | .containerLabel { 289 | display: block; 290 | position: relative; 291 | padding-left: 35px; 292 | margin-bottom: 12px; 293 | cursor: pointer; 294 | font-size: 1.25vh; 295 | font-weight: bold; 296 | -webkit-user-select: none; 297 | -moz-user-select: none; 298 | -ms-user-select: none; 299 | user-select: none; 300 | font-family: 'Arial', 'Helvetica', 'Helvetica Neue', sans-serif; 301 | } 302 | 303 | /* Hide the browser's default checkbox */ 304 | .containerLabel input { 305 | position: absolute; 306 | opacity: 0; 307 | cursor: pointer; 308 | } 309 | 310 | /* Create a custom checkbox */ 311 | .checkmark { 312 | position: absolute; 313 | top: 0; 314 | left: 0; 315 | height: 4vh; 316 | width: 4vh; 317 | max-height: 24px; 318 | max-width: 24px; 319 | background-color: #eee; 320 | border-radius: 4px; 321 | } 322 | 323 | /* On mouse-over, add a grey background color */ 324 | .containerLabel:hover input ~ .checkmark { 325 | background-color: #2196F3; 326 | } 327 | 328 | /* When the checkbox is checked, add a blue background */ 329 | .containerLabel input:checked ~ .checkmark { 330 | background-color: #2196F3; 331 | } 332 | 333 | /* Create the checkmark/indicator (hidden when not checked) */ 334 | .checkmark:after { 335 | content: ""; 336 | position: absolute; 337 | display: none; 338 | } 339 | 340 | /* Show the checkmark when checked */ 341 | .containerLabel input:checked ~ .checkmark:after { 342 | display: block; 343 | } 344 | 345 | /* Style the checkmark/indicator */ 346 | .containerLabel .checkmark:after { 347 | left: 9px; 348 | top: 5px; 349 | width: 5px; 350 | height: 10px; 351 | border: solid white; 352 | border-width: 0 3px 3px 0; 353 | -webkit-transform: rotate(45deg); 354 | -ms-transform: rotate(45deg); 355 | transform: rotate(45deg); 356 | } 357 | 358 | .disabledContainerLabel { 359 | font-size: 1.25vh; 360 | display: block; 361 | font-weight: bold; 362 | position: relative; 363 | padding-left: 35px; 364 | margin-bottom: 12px; 365 | cursor: pointer; 366 | -webkit-user-select: none; 367 | -moz-user-select: none; 368 | -ms-user-select: none; 369 | user-select: none; 370 | font-family: 'Arial', 'Helvetica', 'Helvetica Neue', sans-serif; 371 | } 372 | 373 | /* Hide the browser's default checkbox */ 374 | .disabledContainerLabel input { 375 | position: absolute; 376 | opacity: 0; 377 | cursor: pointer; 378 | } 379 | 380 | .disabledContainerLabel .tooltiptext { 381 | font-size: 1.25vh; 382 | visibility: hidden; 383 | width: 120px; 384 | background-color: #2196F3; 385 | color: #fff; 386 | text-align: left; 387 | border-radius: 4px; 388 | padding: 5px 0; 389 | z-index: 1; 390 | top: -5px; 391 | left: 105%; 392 | padding: 10px 5px 5px 5px; 393 | font-family: 'Arial', 'Helvetica', 'Helvetica Neue', sans-serif; 394 | margin: 5px; 395 | } 396 | 397 | .disabledContainerLabel:hover .tooltiptext { 398 | visibility: visible; 399 | } 400 | 401 | .visualizationTable { 402 | width: 100%; 403 | font-family: 'Arial', 'Helvetica', 'Helvetica Neue', sans-serif; 404 | padding: 5px 5px 5px 5px; 405 | } 406 | 407 | .statMethodTable { 408 | width: 100%; 409 | border: none; 410 | font-family: 'Arial', 'Helvetica', 'Helvetica Neue', sans-serif; 411 | padding: 5px 5px 5px 5px; 412 | } 413 | 414 | .chartcell { 415 | padding: 5px 5px 5px 10px; 416 | border-width: 2px 0px 0px 0px; 417 | border-style: solid; 418 | border-color: darkgray; 419 | } 420 | 421 | .container { 422 | position: relative; 423 | width: 99%; 424 | height: 18vh; 425 | font-family: 'Arial', 'Helvetica', 'Helvetica Neue', sans-serif; 426 | min-height: 200px; 427 | } 428 | 429 | .divider { 430 | width: 5px; 431 | height: auto; 432 | display: inline-block; 433 | } 434 | -------------------------------------------------------------------------------- /UnityPerformanceBenchmarkReporter/Report/warning.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Unity-Technologies/PerformanceBenchmarkReporter/c403bfed95d9c1c14b644943a8ea662abb7d483e/UnityPerformanceBenchmarkReporter/Report/warning.png -------------------------------------------------------------------------------- /UnityPerformanceBenchmarkReporter/TestResultJsonParser.cs: -------------------------------------------------------------------------------- 1 | using System; 2 | using System.Collections.Generic; 3 | using System.IO; 4 | using System.Linq; 5 | using System.Text.RegularExpressions; 6 | using System.Xml.Linq; 7 | using Newtonsoft.Json; 8 | using UnityPerformanceBenchmarkReporter.Entities; 9 | using UnityPerformanceBenchmarkReporter.Entities.New; 10 | 11 | namespace UnityPerformanceBenchmarkReporter 12 | { 13 | public class TestResultJsonParser : IParser 14 | { 15 | public PerformanceTestRun Parse(string path, int version) 16 | { 17 | string report = ""; 18 | try 19 | { 20 | using (StreamReader reader = new StreamReader(path)) 21 | { 22 | string stream = reader.ReadToEnd().Trim(); 23 | 24 | // json wrawrapped in invalid [], removed for valid json format 25 | if (stream[0] == '[' && stream[stream.Length - 1] == ']') 26 | report = stream.Substring(1, stream.Length - 2); 27 | else 28 | report = stream; 29 | } 30 | } 31 | catch (System.Exception) 32 | { 33 | 34 | throw; 35 | } 36 | 37 | switch (version) 38 | { 39 | case 1: 40 | return ParseJsonV1(report); 41 | case 2: 42 | return ParseJsonV2(report); 43 | default: 44 | return null; 45 | } 46 | } 47 | 48 | private static PerformanceTestRun ParseJsonV1(string json) 49 | { 50 | 51 | PerformanceTestRun result; 52 | 53 | try 54 | { 55 | result = JsonConvert.DeserializeObject(json); 56 | var path = result.PlayerSettings.StereoRenderingPath; 57 | result.PlayerSettings.StereoRenderingPath = GetStereoPath(path,result.PlayerSettings.AndroidTargetSdkVersion); 58 | } 59 | catch (System.Exception) 60 | { 61 | 62 | throw; 63 | } 64 | 65 | 66 | return result; 67 | } 68 | 69 | private static PerformanceTestRun ParseJsonV2(string json) 70 | { 71 | 72 | Run run = null; 73 | try 74 | { 75 | run = JsonConvert.DeserializeObject(json); 76 | } 77 | catch (System.Exception) 78 | { 79 | throw; 80 | } 81 | 82 | if (run != null) 83 | { 84 | var testRun = new PerformanceTestRun() 85 | { 86 | BuildSettings = new BuildSettings() 87 | { 88 | Platform = run.Player.Platform, 89 | BuildTarget = run.Player.BuildTarget, 90 | DevelopmentPlayer = true, 91 | AndroidBuildSystem = run.Player.AndroidBuildSystem 92 | }, 93 | EditorVersion = new EditorVersion() 94 | { 95 | Branch = run.Editor.Branch, 96 | DateSeconds = run.Editor.Date, 97 | FullVersion = $"{run.Editor.Version} ({run.Editor.Changeset})", 98 | RevisionValue = 0 99 | }, 100 | PlayerSettings = new PlayerSettings() 101 | { 102 | GpuSkinning = run.Player.GpuSkinning, 103 | GraphicsApi = run.Player.GraphicsApi, 104 | RenderThreadingMode = run.Player.RenderThreadingMode, 105 | ScriptingBackend = run.Player.ScriptingBackend, 106 | AndroidTargetSdkVersion = run.Player.AndroidTargetSdkVersion, 107 | EnabledXrTargets = new List(), 108 | ScriptingRuntimeVersion = "", 109 | StereoRenderingPath = GetStereoPath(run.Player.StereoRenderingPath, run.Player.AndroidTargetSdkVersion) 110 | }, 111 | QualitySettings = new QualitySettings() 112 | { 113 | Vsync = run.Player.Vsync, 114 | AntiAliasing = run.Player.AntiAliasing, 115 | AnisotropicFiltering = run.Player.AnisotropicFiltering, 116 | BlendWeights = run.Player.BlendWeights, 117 | ColorSpace = run.Player.ColorSpace 118 | }, 119 | ScreenSettings = new ScreenSettings() 120 | { 121 | Fullscreen = run.Player.Fullscreen, 122 | ScreenHeight = run.Player.ScreenHeight, 123 | ScreenWidth = run.Player.ScreenWidth, 124 | ScreenRefreshRate = run.Player.ScreenRefreshRate 125 | }, 126 | PlayerSystemInfo = new Entities.PlayerSystemInfo() 127 | { 128 | DeviceModel = run.Hardware.DeviceModel, 129 | DeviceName = run.Hardware.DeviceName, 130 | OperatingSystem = run.Hardware.OperatingSystem, 131 | ProcessorCount = run.Hardware.ProcessorCount, 132 | ProcessorType = run.Hardware.ProcessorType, 133 | GraphicsDeviceName = run.Hardware.GraphicsDeviceName, 134 | SystemMemorySize = run.Hardware.SystemMemorySizeMB, 135 | XrDevice = run.Hardware.XrDevice, 136 | XrModel = run.Hardware.XrModel 137 | }, 138 | StartTime = run.Date, 139 | TestSuite = run.TestSuite, 140 | Results = new List() 141 | }; 142 | 143 | testRun.EndTime = DateTime.Now.ToUniversalTime() 144 | .Subtract(new DateTime(1970, 1, 1, 0, 0, 0, DateTimeKind.Utc)) 145 | .TotalMilliseconds; 146 | 147 | foreach (var res in run.Results) 148 | { 149 | var pt = new PerformanceTestResult() 150 | { 151 | TestCategories = res.Categories, 152 | TestName = res.Name, 153 | TestVersion = res.Version, 154 | SampleGroups = res.SampleGroups.Select(sg => new Entities.SampleGroup 155 | { 156 | Samples = sg.Samples, 157 | Average = sg.Average, 158 | Max = sg.Max, 159 | Median = sg.Median, 160 | Min = sg.Min, 161 | Sum = sg.Sum, 162 | StandardDeviation = sg.StandardDeviation, 163 | SampleCount = sg.Samples.Count, 164 | Definition = new SampleGroupDefinition() 165 | { 166 | Name = sg.Name, 167 | SampleUnit = (Entities.SampleUnit)sg.Unit, 168 | IncreaseIsBetter = sg.IncreaseIsBetter, 169 | Threshold = sg.Threshold, 170 | ContainsKnownIssue = sg.ContainsKnownIssue, 171 | KnownIssueDetails = sg.KnownIssueDetails 172 | } 173 | }).ToList() 174 | }; 175 | testRun.Results.Add(pt); 176 | } 177 | 178 | return testRun; 179 | } 180 | 181 | return null; 182 | } 183 | 184 | /// This allows us to get the stereo mode from a custom data string as well as the normal stereo mode setting 185 | /// This is due to a bug in the perf framework where the stereo mode always displays multipass 186 | /// We pass this custom string in as AndroidTargetSDKVersion 187 | /// The second parameter takes in this string compares the values and overwrites the stereo mode if the mode in the custom string eists and is different 188 | private static string GetStereoPath(string stereoModeString, string miscDataString) 189 | { 190 | 191 | if(string.IsNullOrWhiteSpace(stereoModeString)) 192 | { 193 | Console.WriteLine("Stereo Mode String Is Null"); 194 | return ""; 195 | } 196 | 197 | 198 | Regex stereoModeRegex = new Regex(@"stereorenderingmode=([^|]*)"); 199 | var match = stereoModeRegex.Match(miscDataString); 200 | if(match.Success) 201 | { 202 | 203 | if(match.Groups.Count <= 1) 204 | return stereoModeString; 205 | 206 | if(stereoModeString.ToLower() == match.Value.ToLower()) 207 | { 208 | return stereoModeString; 209 | } 210 | else 211 | { 212 | return match.Groups[1].Value; 213 | } 214 | } 215 | else 216 | { 217 | Console.WriteLine("Failed To Parse Custom Data String for StereoMode"); 218 | return stereoModeString; 219 | } 220 | 221 | 222 | } 223 | } 224 | } 225 | -------------------------------------------------------------------------------- /UnityPerformanceBenchmarkReporter/TestResultXmlParser.cs: -------------------------------------------------------------------------------- 1 | using System; 2 | using System.Collections.Generic; 3 | using System.IO; 4 | using System.Linq; 5 | using System.Text.RegularExpressions; 6 | using System.Xml.Linq; 7 | using Newtonsoft.Json; 8 | using UnityPerformanceBenchmarkReporter.Entities; 9 | using UnityPerformanceBenchmarkReporter.Entities.New; 10 | 11 | namespace UnityPerformanceBenchmarkReporter 12 | { 13 | public class TestResultXmlParser : IParser 14 | { 15 | public PerformanceTestRun Parse(string path,int version) 16 | { 17 | var xmlDocument = XDocument.Load(path); 18 | return Parse(xmlDocument); 19 | } 20 | 21 | private static PerformanceTestRun Parse(XDocument xmlDocument) 22 | { 23 | var output = xmlDocument.Descendants("output"); 24 | var xElements = output as XElement[] ?? output.ToArray(); 25 | 26 | if (!xElements.Any()) 27 | { 28 | return null; 29 | } 30 | 31 | var run = DeserializeMetadata(xElements) ?? DeserializeMetadataV2(xElements); 32 | 33 | if (run == null) 34 | { 35 | return null; 36 | } 37 | 38 | run.EndTime = DateTime.Now.ToUniversalTime() 39 | .Subtract(new DateTime(1970, 1, 1, 0, 0, 0, DateTimeKind.Utc)) 40 | .TotalMilliseconds; 41 | 42 | DeserializeTestResults(xElements, run); 43 | DeserializeTestResultsV2(xElements, run); 44 | 45 | return run; 46 | } 47 | 48 | private static void DeserializeTestResults(IEnumerable output, PerformanceTestRun run) 49 | { 50 | foreach (var element in output) 51 | { 52 | foreach (var line in element.Value.Split('\n')) 53 | { 54 | var json = GetJsonFromHashtag("performancetestresult", line); 55 | if (json == null) continue; 56 | 57 | var result = JsonConvert.DeserializeObject(json); 58 | run.Results.Add(result); 59 | } 60 | } 61 | } 62 | 63 | private static void DeserializeTestResultsV2(IEnumerable output, PerformanceTestRun run) 64 | { 65 | foreach (var element in output) 66 | { 67 | foreach (var line in element.Value.Split('\n')) 68 | { 69 | var json = GetJsonFromHashtag("performancetestresult2", line); 70 | if (json == null) 71 | { 72 | continue; 73 | } 74 | 75 | var result = TryDeserializePerformanceTestResultJsonObject(json); 76 | if (result != null) 77 | { 78 | var pt = new PerformanceTestResult() 79 | { 80 | TestCategories = result.Categories, 81 | TestName = result.Name, 82 | TestVersion = result.Version, 83 | SampleGroups = result.SampleGroups.Select(sg => new Entities.SampleGroup 84 | { 85 | Samples = sg.Samples, 86 | Average = sg.Average, 87 | Max = sg.Max, 88 | Median = sg.Median, 89 | Min = sg.Min, 90 | Sum = sg.Sum, 91 | StandardDeviation = sg.StandardDeviation, 92 | SampleCount = sg.Samples.Count, 93 | 94 | Definition = new SampleGroupDefinition() 95 | { 96 | Name = sg.Name, 97 | SampleUnit = (Entities.SampleUnit)sg.Unit, 98 | IncreaseIsBetter = sg.IncreaseIsBetter, 99 | Threshold = sg.Threshold, 100 | ContainsKnownIssue = sg.ContainsKnownIssue, 101 | KnownIssueDetails = sg.KnownIssueDetails 102 | } 103 | }).ToList() 104 | }; 105 | run.Results.Add(pt); 106 | } 107 | } 108 | } 109 | } 110 | 111 | private static PerformanceTestRun DeserializeMetadata(IEnumerable output) 112 | { 113 | return (output 114 | .SelectMany(element => element.Value.Split('\n'), 115 | (element, line) => GetJsonFromHashtag("performancetestruninfo", line)) 116 | .Where(json => json != null) 117 | .Select(JsonConvert.DeserializeObject)).FirstOrDefault(); 118 | } 119 | 120 | private static PerformanceTestRun DeserializeMetadataV2(IEnumerable output) 121 | { 122 | foreach (var element in output) 123 | { 124 | var pattern = @"##performancetestruninfo2:(.+)\n"; 125 | var regex = new Regex(pattern); 126 | var matches = regex.Match(element.Value); 127 | if (!matches.Success) continue; 128 | if (matches.Groups.Count == 0) continue; 129 | 130 | if (matches.Groups[1].Captures.Count > 1) 131 | { 132 | throw new Exception("Performance test run had multiple hardware and player settings, there should only be one."); 133 | } 134 | 135 | var json = matches.Groups[1].Value; 136 | if (string.IsNullOrEmpty(json)) 137 | { 138 | throw new Exception("Performance test run has incomplete hardware and player settings."); 139 | } 140 | 141 | var result = TryDeserializePerformanceTestRunJsonObject(json); 142 | 143 | var run = new PerformanceTestRun() 144 | { 145 | BuildSettings = new BuildSettings() 146 | { 147 | Platform = result.Player.Platform, 148 | BuildTarget = result.Player.BuildTarget, 149 | DevelopmentPlayer = true, 150 | AndroidBuildSystem = result.Player.AndroidBuildSystem 151 | }, 152 | EditorVersion = new EditorVersion() 153 | { 154 | Branch = result.Editor.Branch, 155 | DateSeconds = result.Editor.Date, 156 | FullVersion = $"{result.Editor.Version} ({result.Editor.Changeset})", 157 | RevisionValue = 0 158 | }, 159 | PlayerSettings = new PlayerSettings() 160 | { 161 | GpuSkinning = result.Player.GpuSkinning, 162 | GraphicsApi = result.Player.GraphicsApi, 163 | RenderThreadingMode = result.Player.RenderThreadingMode, 164 | ScriptingBackend = result.Player.ScriptingBackend, 165 | AndroidTargetSdkVersion = result.Player.AndroidTargetSdkVersion, 166 | EnabledXrTargets = new List(), 167 | ScriptingRuntimeVersion = "", 168 | StereoRenderingPath = result.Player.StereoRenderingPath 169 | }, 170 | QualitySettings = new QualitySettings() 171 | { 172 | Vsync = result.Player.Vsync, 173 | AntiAliasing = result.Player.AntiAliasing, 174 | AnisotropicFiltering = result.Player.AnisotropicFiltering, 175 | BlendWeights = result.Player.BlendWeights, 176 | ColorSpace = result.Player.ColorSpace 177 | }, 178 | ScreenSettings = new ScreenSettings() 179 | { 180 | Fullscreen = result.Player.Fullscreen, 181 | ScreenHeight = result.Player.ScreenHeight, 182 | ScreenWidth = result.Player.ScreenWidth, 183 | ScreenRefreshRate = result.Player.ScreenRefreshRate 184 | }, 185 | PlayerSystemInfo = new Entities.PlayerSystemInfo() 186 | { 187 | DeviceModel = result.Hardware.DeviceModel, 188 | DeviceName = result.Hardware.DeviceName, 189 | OperatingSystem = result.Hardware.OperatingSystem, 190 | ProcessorCount = result.Hardware.ProcessorCount, 191 | ProcessorType = result.Hardware.ProcessorType, 192 | GraphicsDeviceName = result.Hardware.GraphicsDeviceName, 193 | SystemMemorySize = result.Hardware.SystemMemorySizeMB, 194 | XrDevice = "", 195 | XrModel = "" 196 | }, 197 | StartTime = result.Date, 198 | TestSuite = result.TestSuite, 199 | Results = new List() 200 | }; 201 | 202 | return run; 203 | } 204 | 205 | return null; 206 | } 207 | 208 | private static Run TryDeserializePerformanceTestRunJsonObject(string json) 209 | { 210 | try 211 | { 212 | return JsonConvert.DeserializeObject(json); 213 | } 214 | catch (Exception e) 215 | { 216 | Console.WriteLine(e.Message); 217 | } 218 | 219 | return null; 220 | } 221 | 222 | private static Entities.New.TestResult TryDeserializePerformanceTestResultJsonObject(string json) 223 | { 224 | try 225 | { 226 | return JsonConvert.DeserializeObject(json); 227 | } 228 | catch (Exception e) 229 | { 230 | Console.WriteLine(e.Message); 231 | } 232 | 233 | return null; 234 | } 235 | 236 | private static string GetJsonFromHashtag(string tag, string line) 237 | { 238 | if (!line.Contains($"##{tag}:")) return null; 239 | var jsonStart = line.IndexOf('{'); 240 | var openBrackets = 0; 241 | var stringIndex = jsonStart; 242 | while (openBrackets > 0 || stringIndex == jsonStart) 243 | { 244 | var character = line[stringIndex]; 245 | if (character == '{') 246 | { 247 | openBrackets++; 248 | } 249 | 250 | if (character == '}') 251 | { 252 | openBrackets--; 253 | } 254 | 255 | stringIndex++; 256 | } 257 | 258 | var jsonEnd = stringIndex; 259 | return line.Substring(jsonStart, jsonEnd - jsonStart); 260 | } 261 | } 262 | } 263 | -------------------------------------------------------------------------------- /UnityPerformanceBenchmarkReporter/TestRunMetadataProcessor.cs: -------------------------------------------------------------------------------- 1 | using System; 2 | using System.Collections; 3 | using System.Collections.Generic; 4 | using System.Linq; 5 | using System.Reflection; 6 | using System.Runtime.InteropServices; 7 | using System.Text; 8 | using System.Text.RegularExpressions; 9 | using UnityPerformanceBenchmarkReporter.Entities; 10 | 11 | namespace UnityPerformanceBenchmarkReporter 12 | { 13 | public class TypeMetadata 14 | { 15 | public TypeMetadata(Type thisType) 16 | { 17 | Type = thisType; 18 | } 19 | 20 | public List FieldGroups { get; } = new List(); 21 | 22 | public int NullResultCount { get; set; } 23 | 24 | public int ValidResultCount { get; set; } 25 | 26 | public bool HasMismatches 27 | { 28 | get { return FieldGroups.Any(g => g.HasMismatches); } 29 | } 30 | 31 | public Type Type { get; } 32 | 33 | public string TypeName => Type.Name; 34 | } 35 | 36 | public class FieldGroup 37 | { 38 | // using an array instead of list so we can preserve ordering for display purposes 39 | public FieldValue[] Values = new FieldValue[0]; 40 | 41 | public FieldGroup(string fieldName) 42 | { 43 | FieldName = fieldName; 44 | } 45 | 46 | public string FieldName { get; } 47 | 48 | public bool HasMismatches 49 | { 50 | get { return Values.Any(v => v.IsMismatched); } 51 | } 52 | } 53 | 54 | public class FieldValue 55 | { 56 | private readonly char pathSeperator = RuntimeInformation.IsOSPlatform(OSPlatform.Windows) ? '\\' : '/'; 57 | 58 | public bool IsMismatched; 59 | 60 | public FieldValue(string resultFilePath, string value) 61 | { 62 | if (!string.IsNullOrEmpty(resultFilePath)) 63 | { 64 | var pathParts = resultFilePath.Split(pathSeperator); 65 | ResultFileDirectory = string.Join(pathSeperator, pathParts.Take(pathParts.Length - 1)); 66 | ResultFileName = pathParts[pathParts.Length - 1]; 67 | } 68 | 69 | Value = value; 70 | } 71 | 72 | public string Value { get; } 73 | public string ResultFileDirectory { get; } 74 | 75 | public string ResultFileName { get; } 76 | } 77 | 78 | public class TestRunMetadataProcessor 79 | { 80 | private readonly string[] androidOnlyMetadata = 81 | { 82 | "AndroidBuildSystem" 83 | }; 84 | 85 | private readonly string[] builtInXrOnlyMetadata = 86 | { 87 | "EnabledXrTargets" 88 | }; 89 | 90 | private readonly Dictionary excludedConfigFieldNames; 91 | private static readonly string MetadataNotAvailable = "Metadata not available"; 92 | 93 | private readonly Type[] metadataTypes = 94 | { 95 | typeof(PlayerSystemInfo), 96 | typeof(PlayerSettings), 97 | typeof(ScreenSettings), 98 | typeof(QualitySettings), 99 | typeof(BuildSettings), 100 | typeof(EditorVersion) 101 | }; 102 | 103 | public readonly List TypeMetadata = new List(); 104 | 105 | private bool isAndroid; 106 | 107 | public bool BuiltInVrExists { get; private set; } 108 | 109 | public class ExtractField 110 | { 111 | public string ExtractedFieldName; 112 | public Regex ExtractionRegex; 113 | public string ValueExtracted; 114 | } 115 | 116 | private readonly Dictionary> extraMetadataExtractFields = 117 | new Dictionary> 118 | { 119 | { 120 | "ScriptingRuntimeVersion", 121 | new List 122 | { 123 | new ExtractField 124 | { 125 | ExtractedFieldName = "OculusPluginVersion", 126 | ExtractionRegex = new Regex("OculusPluginVersion\\|([^|]*)", 127 | RegexOptions.Compiled | RegexOptions.IgnoreCase) 128 | }, 129 | new ExtractField 130 | { 131 | ExtractedFieldName = "DeviceRuntimeVersion", 132 | ExtractionRegex = 133 | new Regex("deviceruntimeversion\\|[^/]*/[^/]*/[^/]*/[^/]*/([0-9]*\\.[0-9]*\\.[0-9]*):", 134 | RegexOptions.Compiled | RegexOptions.IgnoreCase) 135 | }, 136 | new ExtractField 137 | { 138 | ExtractedFieldName = "XrSdkName", 139 | ExtractionRegex = new Regex("XrsdkName\\|([^|]*)", 140 | RegexOptions.Compiled | RegexOptions.IgnoreCase) 141 | }, 142 | new ExtractField 143 | { 144 | ExtractedFieldName = "XrSdkVersion", 145 | ExtractionRegex = new Regex("XrSdkVersion\\|([^|]*)", 146 | RegexOptions.Compiled | RegexOptions.IgnoreCase) 147 | }, 148 | new ExtractField 149 | { 150 | ExtractedFieldName = "XrSdkRevision", 151 | ExtractionRegex = new Regex("XrSdkRevision\\|([^|]*)", 152 | RegexOptions.Compiled | RegexOptions.IgnoreCase) 153 | }, 154 | new ExtractField 155 | { 156 | ExtractedFieldName = "XrSdkRevisionDate", 157 | ExtractionRegex = new Regex("XrSdkRevisionDate\\|([^|]*)", 158 | RegexOptions.Compiled | RegexOptions.IgnoreCase) 159 | }, 160 | new ExtractField 161 | { 162 | ExtractedFieldName = "XrSdkBranch", 163 | ExtractionRegex = new Regex("XrSdkBranch\\|([^|]*)", 164 | RegexOptions.Compiled | RegexOptions.IgnoreCase) 165 | }, 166 | new ExtractField 167 | { 168 | ExtractedFieldName = "XrManagementVersion", 169 | ExtractionRegex = new Regex("XrManagementVersion\\|([^|]*)", 170 | RegexOptions.Compiled | RegexOptions.IgnoreCase) 171 | }, 172 | new ExtractField 173 | { 174 | ExtractedFieldName = "XrManagementRevision", 175 | ExtractionRegex = new Regex("XrManagementRevision\\|([^|]*)", 176 | RegexOptions.Compiled | RegexOptions.IgnoreCase) 177 | }, 178 | new ExtractField 179 | { 180 | ExtractedFieldName = "DeviceUniqueId", 181 | ExtractionRegex = new Regex("deviceuniqueid\\|([^|]*)", 182 | RegexOptions.Compiled | RegexOptions.IgnoreCase) 183 | }, 184 | new ExtractField 185 | { 186 | ExtractedFieldName = "Username", 187 | ExtractionRegex = new Regex("username\\|([^|]*)", 188 | RegexOptions.Compiled | RegexOptions.IgnoreCase) 189 | }, 190 | new ExtractField 191 | { 192 | ExtractedFieldName = "RenderPipeline", 193 | ExtractionRegex = new Regex("renderpipeline\\|([^|]*)", 194 | RegexOptions.Compiled | RegexOptions.IgnoreCase) 195 | }, 196 | new ExtractField 197 | { 198 | ExtractedFieldName = "FfrLevel", 199 | ExtractionRegex = new Regex("ffrlevel\\|([^|]*)", 200 | RegexOptions.Compiled | RegexOptions.IgnoreCase) 201 | }, 202 | new ExtractField 203 | { 204 | ExtractedFieldName = "TestsBranch", 205 | ExtractionRegex = new Regex("testsbranch\\|([^|]*)", 206 | RegexOptions.Compiled | RegexOptions.IgnoreCase) 207 | }, 208 | new ExtractField 209 | { 210 | ExtractedFieldName = "TestsRev", 211 | ExtractionRegex = new Regex("testsrev\\|([^|]*)", 212 | RegexOptions.Compiled | RegexOptions.IgnoreCase) 213 | }, 214 | new ExtractField 215 | { 216 | ExtractedFieldName = "TestsRevDate", 217 | ExtractionRegex = new Regex("testsrevdate\\|([^|]*)", 218 | RegexOptions.Compiled | RegexOptions.IgnoreCase) 219 | }, 220 | new ExtractField 221 | { 222 | ExtractedFieldName = "PerfTestsPackageName", 223 | ExtractionRegex = new Regex("PerfTestsPackageName\\|([^|]*)", 224 | RegexOptions.Compiled | RegexOptions.IgnoreCase) 225 | }, 226 | new ExtractField 227 | { 228 | ExtractedFieldName = "PerfTestsVersion", 229 | ExtractionRegex = new Regex("PerfTestsVersion\\|([^|]*)", 230 | RegexOptions.Compiled | RegexOptions.IgnoreCase) 231 | }, 232 | new ExtractField 233 | { 234 | ExtractedFieldName = "PerfTestsRevision", 235 | ExtractionRegex = new Regex("PerfTestsRevision\\|([^|]*)", 236 | RegexOptions.Compiled | RegexOptions.IgnoreCase) 237 | }, 238 | new ExtractField 239 | { 240 | ExtractedFieldName = "AndroidTargetArchitecture", 241 | ExtractionRegex = new Regex("AndroidTargetArchitecture\\|([^|]*)", 242 | RegexOptions.Compiled | RegexOptions.IgnoreCase) 243 | }, 244 | new ExtractField 245 | { 246 | ExtractedFieldName = "UrpVersion", 247 | ExtractionRegex = new Regex("UrpVersion\\|([^|]*)", 248 | RegexOptions.Compiled | RegexOptions.IgnoreCase) 249 | }, 250 | new ExtractField 251 | { 252 | ExtractedFieldName = "UrpRevision", 253 | ExtractionRegex = new Regex("UrpRevision\\|([^|]*)", 254 | RegexOptions.Compiled | RegexOptions.IgnoreCase) 255 | } 256 | } 257 | } 258 | }; 259 | 260 | public TestRunMetadataProcessor(Dictionary excludedFieldNames) 261 | { 262 | excludedConfigFieldNames = excludedFieldNames; 263 | } 264 | 265 | /// 266 | /// Assumes first performanceTestRun should be used to compare all other performanceTestRuns against. 267 | /// 268 | /// 269 | /// 270 | public void ProcessMetadata(PerformanceTestRun performanceTestRun, string xmlFileNamePath) 271 | { 272 | SetIsBuiltInVr(new[] {performanceTestRun}); 273 | SetIsAndroid(new[] {performanceTestRun}); 274 | 275 | foreach (var metadataType in metadataTypes) 276 | { 277 | var typeMetadata = TypeMetadata.Any(tm => tm.Type == metadataType) 278 | ? TypeMetadata.First(m => m.Type == metadataType) 279 | : null; 280 | 281 | // If metadataType doesn't exist in our TypeMetadata list, add it 282 | if (typeMetadata == null) 283 | { 284 | typeMetadata = new TypeMetadata(metadataType); 285 | TypeMetadata.Add(typeMetadata); 286 | } 287 | 288 | var fieldInfos = performanceTestRun.GetType().GetFields(); 289 | 290 | // If this metadataType is completely missing from the perf test run, mark it as such and move on 291 | if (fieldInfos.Any(f => f.FieldType == metadataType)) 292 | { 293 | var fieldInfo = fieldInfos.First(f => f.FieldType == metadataType); 294 | 295 | object obj = null; 296 | GetFieldInfoValue(performanceTestRun, metadataType, ref obj, fieldInfo); 297 | 298 | // If null, we're missing metadata for this performanceTestRun 299 | if (obj == null) 300 | { 301 | typeMetadata.NullResultCount++; 302 | 303 | // But we already have results for this metadataType, 304 | // add an empty "missing value" entry for it each FieldGroup 305 | if (typeMetadata.ValidResultCount > 0) 306 | { 307 | foreach (var fieldGroup in typeMetadata.FieldGroups) 308 | { 309 | BackfillFieldGroupValuesForMissingMetadata(xmlFileNamePath, fieldGroup, typeMetadata); 310 | } 311 | } 312 | 313 | continue; 314 | } 315 | 316 | var fieldsToProcess = GetFieldsToProcess(metadataType, obj.GetType().GetFields()); 317 | 318 | // if we have valid field metadata to process 319 | if (fieldsToProcess.Length > 0) 320 | { 321 | ProcessMetaData(xmlFileNamePath, fieldsToProcess, typeMetadata, metadataType, obj); 322 | } 323 | } 324 | } 325 | } 326 | 327 | private void ProcessMetaData(string xmlFileNamePath, FieldInfo[] fieldsToProcess, TypeMetadata typeMetadata, 328 | Type metadataType, object obj) 329 | { 330 | foreach (var field in fieldsToProcess) 331 | { 332 | var fieldName = field.Name; 333 | if (!typeMetadata.FieldGroups.Any(fg => fg.FieldName.Equals(fieldName))) 334 | { 335 | typeMetadata.FieldGroups.Add(new FieldGroup(fieldName)); 336 | } 337 | 338 | var thisFieldGroup = typeMetadata.FieldGroups.First(fg => fg.FieldName.Equals(fieldName)); 339 | 340 | 341 | if (extraMetadataExtractFields.ContainsKey(fieldName)) 342 | { 343 | foreach (var extractField in extraMetadataExtractFields[fieldName]) 344 | { 345 | FieldGroup newFieldGroup; 346 | if (!typeMetadata.FieldGroups.Any(fg => fg.FieldName.Equals(extractField.ExtractedFieldName))) 347 | { 348 | newFieldGroup = new FieldGroup(extractField.ExtractedFieldName); 349 | typeMetadata.FieldGroups.Add(newFieldGroup); 350 | } 351 | else 352 | { 353 | newFieldGroup = 354 | typeMetadata.FieldGroups.First(fg => 355 | fg.FieldName.Equals(extractField.ExtractedFieldName)); 356 | } 357 | 358 | var value = GetValueBasedOnType(metadataType, field, obj); 359 | 360 | extractField.ValueExtracted = ExtractValue(extractField.ExtractionRegex, value); 361 | 362 | InsertFieldValueWithBackfill(xmlFileNamePath, newFieldGroup, typeMetadata, 363 | extractField.ValueExtracted); 364 | DetermineIfMismatchExists(typeMetadata, newFieldGroup); 365 | } 366 | 367 | InsertFieldValueWithBackfill(xmlFileNamePath, thisFieldGroup, typeMetadata, MetadataNotAvailable); 368 | DetermineIfMismatchExists(typeMetadata, thisFieldGroup); 369 | } 370 | else 371 | { 372 | var thisValue = GetValueBasedOnType(metadataType, field, obj); 373 | InsertFieldValueWithBackfill(xmlFileNamePath, thisFieldGroup, typeMetadata, thisValue); 374 | DetermineIfMismatchExists(typeMetadata, thisFieldGroup); 375 | } 376 | } 377 | 378 | foreach (var fieldGroup in typeMetadata.FieldGroups.Where(fg => 379 | fg.Values.Length < typeMetadata.ValidResultCount + 1)) 380 | { 381 | while (fieldGroup.Values.Length < typeMetadata.ValidResultCount + 1) 382 | { 383 | InsertFieldValue(xmlFileNamePath, fieldGroup, MetadataNotAvailable, isMismatched: true); 384 | } 385 | } 386 | 387 | typeMetadata.ValidResultCount++; 388 | } 389 | 390 | private static void DetermineIfMismatchExists(TypeMetadata typeMetadata, FieldGroup thisFieldGroup) 391 | { 392 | // fieldGroup.Values is sorted by result name; the first element in this array 393 | // is considered to be the reference point, regardless if it's a "baseline" or not. 394 | if (typeMetadata.FieldGroups.Any(fg => fg.FieldName.Equals(thisFieldGroup.FieldName)) && 395 | thisFieldGroup.Values.Length > 0 && thisFieldGroup.Values[thisFieldGroup.Values.Length - 1].Value != 396 | thisFieldGroup.Values[0].Value) 397 | { 398 | thisFieldGroup.Values[thisFieldGroup.Values.Length - 1].IsMismatched = true; 399 | } 400 | } 401 | 402 | private string ExtractValue(Regex regex, string value) 403 | { 404 | var matches = regex.Matches(value); 405 | var matchValue = matches.Count > 0 ? matches[0].Groups[1].Value : MetadataNotAvailable; 406 | 407 | return matchValue; 408 | } 409 | 410 | private void InsertFieldValueWithBackfill(string xmlFileNamePath, FieldGroup thisFieldGroup, 411 | TypeMetadata typeMetadata, 412 | string value) 413 | { 414 | // We want to keep the values array length consistent with the number of results, even for results 415 | // that are missing metadata. We do that here. 416 | BackfillFieldGroupValuesForMissingMetadata(xmlFileNamePath, thisFieldGroup, typeMetadata); 417 | 418 | InsertFieldValue(xmlFileNamePath, thisFieldGroup, value); 419 | } 420 | 421 | private static void InsertFieldValue(string xmlFileNamePath, FieldGroup thisFieldGroup, string value, 422 | bool isMismatched = false) 423 | { 424 | Array.Resize(ref thisFieldGroup.Values, thisFieldGroup.Values.Length + 1); 425 | thisFieldGroup.Values[thisFieldGroup.Values.Length - 1] = 426 | new FieldValue(xmlFileNamePath, value) 427 | { 428 | IsMismatched = isMismatched 429 | }; 430 | } 431 | 432 | private void BackfillFieldGroupValuesForMissingMetadata(string xmlFileNamePath, FieldGroup fieldGroup, 433 | TypeMetadata typeMetadata) 434 | { 435 | if (fieldGroup.Values.Length < typeMetadata.ValidResultCount + typeMetadata.NullResultCount) 436 | { 437 | while (fieldGroup.Values.Length < typeMetadata.ValidResultCount + typeMetadata.NullResultCount) 438 | { 439 | Array.Resize(ref fieldGroup.Values, fieldGroup.Values.Length + 1); 440 | fieldGroup.Values[fieldGroup.Values.Length - 1] = 441 | new FieldValue(xmlFileNamePath, MetadataNotAvailable); 442 | 443 | // fieldGroup.Values is sorted by result name; the first element in this array 444 | // is considered to be the reference point, regardless if it's a "baseline" or not. 445 | if (fieldGroup.Values[fieldGroup.Values.Length - 1].Value != fieldGroup.Values[0].Value) 446 | { 447 | fieldGroup.Values[fieldGroup.Values.Length - 1].IsMismatched = true; 448 | } 449 | } 450 | } 451 | } 452 | 453 | private void GetFieldInfoValue(PerformanceTestRun performanceTestRun, Type metadataType, ref object obj, 454 | FieldInfo fieldInfo) 455 | { 456 | if (metadataType == typeof(PlayerSystemInfo)) 457 | { 458 | obj = (PlayerSystemInfo) fieldInfo.GetValue(performanceTestRun); 459 | } 460 | 461 | if (metadataType == typeof(PlayerSettings)) 462 | { 463 | obj = (PlayerSettings) fieldInfo.GetValue(performanceTestRun); 464 | } 465 | 466 | if (metadataType == typeof(ScreenSettings)) 467 | { 468 | obj = (ScreenSettings) fieldInfo.GetValue(performanceTestRun); 469 | } 470 | 471 | if (metadataType == typeof(QualitySettings)) 472 | { 473 | obj = (QualitySettings) fieldInfo.GetValue(performanceTestRun); 474 | } 475 | 476 | if (metadataType == typeof(BuildSettings)) 477 | { 478 | obj = (BuildSettings) fieldInfo.GetValue(performanceTestRun); 479 | } 480 | 481 | if (metadataType == typeof(EditorVersion)) 482 | { 483 | obj = (EditorVersion) fieldInfo.GetValue(performanceTestRun); 484 | } 485 | } 486 | 487 | private string GetValueBasedOnType(Type metadataType, FieldInfo field, object obj) 488 | { 489 | string value = null; 490 | if (metadataType == typeof(PlayerSystemInfo)) 491 | { 492 | value = GetValue(field, obj); 493 | } 494 | 495 | if (metadataType == typeof(PlayerSettings)) 496 | { 497 | value = GetValue(field, obj); 498 | } 499 | 500 | if (metadataType == typeof(ScreenSettings)) 501 | { 502 | value = GetValue(field, obj); 503 | } 504 | 505 | if (metadataType == typeof(QualitySettings)) 506 | { 507 | value = GetValue(field, obj); 508 | } 509 | 510 | if (metadataType == typeof(BuildSettings)) 511 | { 512 | value = GetValue(field, obj); 513 | } 514 | 515 | if (metadataType == typeof(EditorVersion)) 516 | { 517 | value = GetValue(field, obj); 518 | } 519 | 520 | return value; 521 | } 522 | 523 | private string GetValue(FieldInfo field, object obj) 524 | { 525 | if (IsIEnumerableFieldType(field)) 526 | return ConvertIEnumberableToString(field, (T) obj); 527 | var value = field.GetValue((T) obj); 528 | if (value == null) 529 | { 530 | value = MetadataNotAvailable; 531 | } 532 | else 533 | { 534 | if (value.GetType() != typeof(string)) 535 | { 536 | value = Convert.ToString(value); 537 | if ((string) value == string.Empty) 538 | { 539 | value = MetadataNotAvailable; 540 | } 541 | } 542 | else 543 | { 544 | if ((string) value == string.Empty) 545 | { 546 | value = MetadataNotAvailable; 547 | } 548 | } 549 | } 550 | 551 | return (string) value; 552 | } 553 | 554 | private FieldInfo[] GetFieldsToProcess(Type metadataType, FieldInfo[] fields) 555 | { 556 | // Derive a subset of fields to process from validFieldNames 557 | var excludedFieldNames = 558 | excludedConfigFieldNames != null && excludedConfigFieldNames.ContainsKey(metadataType) 559 | ? excludedConfigFieldNames[metadataType] 560 | : null; 561 | var validFieldNames = GetValidFieldNames(excludedFieldNames, fields.Select(f => f.Name).ToArray()); 562 | var fieldsToProcess = fields.Join(validFieldNames, f => f.Name, s => s, (field, vField) => field).ToArray(); 563 | return fieldsToProcess; 564 | } 565 | 566 | private string ConvertIEnumberableToString(FieldInfo field, T thisObject) 567 | { 568 | var sb = new StringBuilder(); 569 | var fieldValues = (List) (IEnumerable) field.GetValue(thisObject); 570 | 571 | if (fieldValues != null && fieldValues.Any()) 572 | { 573 | foreach (var enumerable in fieldValues) 574 | { 575 | sb.Append(enumerable + ","); 576 | } 577 | 578 | if (sb.ToString().EndsWith(',')) 579 | { 580 | // trim trailing comma 581 | sb.Length--; 582 | } 583 | } 584 | else 585 | { 586 | sb.Append("None"); 587 | } 588 | 589 | return sb.ToString(); 590 | } 591 | 592 | private bool IsIEnumerableFieldType(FieldInfo field) 593 | { 594 | return typeof(IEnumerable).IsAssignableFrom(field.FieldType) && field.FieldType != typeof(string); 595 | } 596 | 597 | private string[] GetValidFieldNames(string[] excludedFieldNames, IEnumerable fieldNames) 598 | { 599 | var validFieldNames = fieldNames as string[] ?? fieldNames.ToArray(); 600 | 601 | if (excludedFieldNames != null && excludedFieldNames.Any()) 602 | { 603 | validFieldNames = validFieldNames.Where(k1 => excludedFieldNames.All(k2 => k2 != k1)).ToArray(); 604 | } 605 | 606 | if (!BuiltInVrExists) 607 | { 608 | validFieldNames = validFieldNames.Where(k1 => builtInXrOnlyMetadata.All(k2 => k2 != k1)).ToArray(); 609 | } 610 | 611 | if (!isAndroid) 612 | { 613 | validFieldNames = validFieldNames.Where(k1 => androidOnlyMetadata.All(k2 => k2 != k1)).ToArray(); 614 | } 615 | 616 | return validFieldNames; 617 | } 618 | 619 | private void SetIsBuiltInVr(PerformanceTestRun[] performanceTestRuns) 620 | { 621 | foreach (var performanceTestRun in performanceTestRuns) 622 | { 623 | BuiltInVrExists = BuiltInVrExists || performanceTestRun.PlayerSettings.EnabledXrTargets != null && performanceTestRun.PlayerSettings.EnabledXrTargets.Any(); 624 | } 625 | } 626 | 627 | private void SetIsAndroid(PerformanceTestRun[] performanceTestRuns) 628 | { 629 | foreach (var performanceTestRun in performanceTestRuns) 630 | { 631 | isAndroid = isAndroid || performanceTestRun.BuildSettings != null && 632 | performanceTestRun.BuildSettings.Platform.Equals("Android"); 633 | } 634 | } 635 | 636 | public void PerformFinalMetadataUpdate(PerformanceBenchmark performanceBenchmark) 637 | { 638 | // The keys in the extraMetadataExtractFields structure have additional embedded metadata that we extract out. 639 | // This renders the raw, unextracted, value of this field unusable, so we discard it 640 | foreach (var metadataName in extraMetadataExtractFields.Keys) 641 | { 642 | foreach (var typeMetadata in performanceBenchmark.TestRunMetadataProcessor.TypeMetadata) 643 | { 644 | typeMetadata.FieldGroups.RemoveAll(fg => fg.FieldName.Equals(metadataName)); 645 | } 646 | } 647 | } 648 | } 649 | } 650 | 651 | -------------------------------------------------------------------------------- /UnityPerformanceBenchmarkReporter/UnityPerformanceBenchmarkReporter.csproj: -------------------------------------------------------------------------------- 1 |  2 | 3 | 4 | Exe 5 | UnityPerformanceBenchmarkReporter.Program 6 | true 7 | netcoreapp3.1;net6.0;net7.0 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | -------------------------------------------------------------------------------- /UnityPerformanceBenchmarkReporter/UnityPerformanceBenchmarkReporter.csproj.user: -------------------------------------------------------------------------------- 1 |  2 | 3 | 4 | true 5 | 6 | -------------------------------------------------------------------------------- /UnityPerformanceBenchmarkReporterTests/ExtensionMethodsTests.cs: -------------------------------------------------------------------------------- 1 | using NUnit.Framework; 2 | using UnityPerformanceBenchmarkReporter; 3 | 4 | namespace UnityPerformanceBenchmarkReporterTests 5 | { 6 | public class ExtensionMethodsTests 7 | { 8 | [Test] 9 | public void VerifyZeroValue() 10 | { 11 | // Arrange 12 | double d = 0; 13 | 14 | // Act 15 | var truncatedD = d.TruncToSigFig(2); 16 | 17 | // Assert 18 | Assert.AreEqual(0, truncatedD, 0); 19 | } 20 | 21 | [Test] 22 | public void VerifyNonZeroValue_GreaterThan1_0SigFigs() 23 | { 24 | // Arrange 25 | double d = 1.58888299999999993; 26 | 27 | // Act 28 | var truncatedD = d.TruncToSigFig(0); 29 | 30 | // Assert 31 | Assert.AreEqual(1, truncatedD, 0); 32 | } 33 | 34 | [Test] 35 | public void VerifyNonZeroValue_LessThan1_0SigFigs() 36 | { 37 | // Arrange 38 | double d = 0.58888299999999993; 39 | 40 | // Act 41 | var truncatedD = d.TruncToSigFig(0); 42 | 43 | // Assert 44 | Assert.AreEqual(0, truncatedD, 0); 45 | } 46 | 47 | [Test] 48 | public void VerifyNonZeroValue_GreaterThan1_1SigFigs() 49 | { 50 | // Arrange 51 | double d = 1.58888299999999993; 52 | 53 | // Act 54 | var truncatedD = d.TruncToSigFig(1); 55 | 56 | // Assert 57 | Assert.AreEqual(1.5, truncatedD, 0); 58 | } 59 | 60 | [Test] 61 | public void VerifyNonZeroValue_LessThan1_1SigFigs() 62 | { 63 | // Arrange 64 | double d = 0.58888299999999993; 65 | 66 | // Act 67 | var truncatedD = d.TruncToSigFig(1); 68 | 69 | // Assert 70 | Assert.AreEqual(0.5, truncatedD, 0); 71 | } 72 | } 73 | } -------------------------------------------------------------------------------- /UnityPerformanceBenchmarkReporterTests/PerformanceBenchmarkTestsBase.cs: -------------------------------------------------------------------------------- 1 | using System.IO; 2 | using UnityPerformanceBenchmarkReporter; 3 | 4 | namespace UnityPerformanceBenchmarkReporterTests 5 | { 6 | public class PerformanceBenchmarkTestsBase 7 | { 8 | protected PerformanceBenchmark PerformanceBenchmark; 9 | private static readonly string TestDataDirectoryName = "TestData"; 10 | 11 | protected string EnsureFullPath(string directoryOrFileName) 12 | { 13 | var currentDirectory = Directory.GetCurrentDirectory(); 14 | return !directoryOrFileName.StartsWith(TestDataDirectoryName) 15 | ? Path.Combine(currentDirectory, TestDataDirectoryName, directoryOrFileName) 16 | : Path.Combine(currentDirectory, directoryOrFileName); 17 | } 18 | } 19 | } -------------------------------------------------------------------------------- /UnityPerformanceBenchmarkReporterTests/PerformanceBenchmarkTestsJson.cs: -------------------------------------------------------------------------------- 1 | using System.Collections.Generic; 2 | using System.Linq; 3 | using NUnit.Framework; 4 | using UnityPerformanceBenchmarkReporter; 5 | using UnityPerformanceBenchmarkReporter.Entities; 6 | 7 | namespace UnityPerformanceBenchmarkReporterTests 8 | { 9 | public class PerformanceBenchmarkTestsJson : PerformanceBenchmarkTestsBase 10 | { 11 | private OptionsParser optionsParser; 12 | private IParser testResultJsonParser; 13 | private List performanceTestRunResults; 14 | private List testResults; 15 | private List baselinePerformanceTestRunResults; 16 | private List baselineTestResults; 17 | 18 | [SetUp] 19 | public void Setup() 20 | { 21 | optionsParser = new OptionsParser(); 22 | PerformanceBenchmark = new PerformanceBenchmark(); 23 | testResultJsonParser = new TestResultJsonParser(); 24 | performanceTestRunResults = new List(); 25 | testResults = new List(); 26 | baselinePerformanceTestRunResults = new List(); 27 | baselineTestResults = new List(); 28 | } 29 | 30 | 31 | [Test] 32 | public void VerifyV1_AddPerformanceTestRunResults() 33 | { 34 | // Arrange 35 | var resultJsonFilePath = EnsureFullPath("results.json"); 36 | var args = new[] { "--format=json", string.Format("--testresultsxmlsource={0}", resultJsonFilePath), "--version=1" }; 37 | optionsParser.ParseOptions(PerformanceBenchmark, args); 38 | 39 | // Act 40 | PerformanceBenchmark.AddPerformanceTestRunResults(testResultJsonParser, performanceTestRunResults, testResults, new List()); 41 | 42 | // Assert 43 | Assert.IsTrue(PerformanceBenchmark.ResultFilesExist); 44 | AssertCorrectResultJsonFilePaths(new[] { resultJsonFilePath }); 45 | Assert.NotNull(testResults); 46 | Assert.IsTrue(testResults.Count > 0); 47 | Assert.NotNull(performanceTestRunResults); 48 | Assert.IsTrue(performanceTestRunResults.Count > 0); 49 | } 50 | 51 | [Test] 52 | public void VerifyV1_AddPerformanceTestRunResults_TwoResultFiles() 53 | { 54 | // Arrange 55 | var resultJsonFilePath = EnsureFullPath("results.json"); 56 | var resultFileName2 = EnsureFullPath("results.json"); 57 | var args = new[] 58 | { 59 | "--format=json", 60 | string.Format("--testresultsxmlsource={0}", resultJsonFilePath), 61 | string.Format("--testresultsxmlsource={0}", resultFileName2) 62 | , "--version=1" 63 | }; 64 | optionsParser.ParseOptions(PerformanceBenchmark, args); 65 | 66 | // Act 67 | PerformanceBenchmark.AddPerformanceTestRunResults(testResultJsonParser, performanceTestRunResults, testResults, new List()); 68 | 69 | // Assert 70 | Assert.IsTrue(PerformanceBenchmark.ResultFilesExist); 71 | AssertCorrectResultJsonFilePaths(new[] { resultJsonFilePath, resultFileName2 }); 72 | Assert.NotNull(testResults); 73 | Assert.IsTrue(testResults.Count > 0); 74 | Assert.NotNull(performanceTestRunResults); 75 | Assert.IsTrue(performanceTestRunResults.Count > 0); 76 | } 77 | 78 | [Test] 79 | public void VerifyV1_AddPerformanceTestRunResults_OneResultFiles_OneResultDirectory() 80 | { 81 | // Arrange 82 | var resultJsonFilePath = EnsureFullPath("results.json"); 83 | var resultsJsonDir = EnsureFullPath("ResultsJson"); 84 | var args = new[] 85 | { 86 | "--format=json", 87 | string.Format("--testresultsxmlsource={0}", resultJsonFilePath), 88 | string.Format("--testresultsxmlsource={0}", resultsJsonDir) 89 | , "--version=1" 90 | }; 91 | optionsParser.ParseOptions(PerformanceBenchmark, args); 92 | 93 | // Act 94 | PerformanceBenchmark.AddPerformanceTestRunResults(testResultJsonParser, performanceTestRunResults, testResults, new List()); 95 | 96 | // Assert 97 | Assert.IsTrue(PerformanceBenchmark.ResultFilesExist); 98 | AssertCorrectResultsJsonDirectoryPaths(new[] { resultsJsonDir }); 99 | AssertCorrectResultJsonFilePaths(new[] { resultJsonFilePath }); 100 | Assert.NotNull(testResults); 101 | Assert.IsTrue(testResults.Count > 0); 102 | Assert.NotNull(performanceTestRunResults); 103 | Assert.IsTrue(performanceTestRunResults.Count > 0); 104 | } 105 | 106 | [Test] 107 | public void VerifyV1_AddBaselinePerformanceTestRunResults() 108 | { 109 | // Arrange 110 | var resultJsonFilePath = EnsureFullPath("results.json"); 111 | var baselineJsonFilePath = EnsureFullPath("baseline.json"); 112 | var args = new[] 113 | { 114 | "--format=Json", 115 | string.Format("--testresultsxmlsource={0}", resultJsonFilePath), 116 | string.Format("--baselinexmlsource={0}", baselineJsonFilePath) 117 | , "--version=1" 118 | }; 119 | optionsParser.ParseOptions(PerformanceBenchmark, args); 120 | 121 | // Act 122 | PerformanceBenchmark.AddBaselinePerformanceTestRunResults(testResultJsonParser, baselinePerformanceTestRunResults, baselineTestResults); 123 | 124 | // Assert 125 | Assert.IsTrue(PerformanceBenchmark.BaselineResultFilesExist); 126 | AssertCorrectBaselineJsonFilePaths(new[] { baselineJsonFilePath }); 127 | AssertCorrectResultJsonFilePaths(new[] { resultJsonFilePath }); 128 | Assert.NotNull(baselineTestResults); 129 | Assert.IsTrue(baselineTestResults.Count > 0); 130 | Assert.NotNull(baselinePerformanceTestRunResults); 131 | Assert.IsTrue(baselinePerformanceTestRunResults.Count > 0); 132 | } 133 | 134 | [Test] 135 | public void VerifyV1_AddBaselinePerformanceTestRunResultsDirectory() 136 | { 137 | // Arrange 138 | var resultJsonFilePath = EnsureFullPath("ResultsJson"); 139 | var baselineJsonFilePath = EnsureFullPath("BaselineJson"); 140 | var args = new[] 141 | { 142 | "--format=Json", 143 | string.Format("--testresultsxmlsource={0}", resultJsonFilePath), 144 | string.Format("--baselinexmlsource={0}", baselineJsonFilePath) 145 | , "--version=1" 146 | }; 147 | optionsParser.ParseOptions(PerformanceBenchmark, args); 148 | 149 | // Act 150 | PerformanceBenchmark.AddBaselinePerformanceTestRunResults(testResultJsonParser, baselinePerformanceTestRunResults, baselineTestResults); 151 | 152 | // Assert 153 | Assert.IsTrue(PerformanceBenchmark.BaselineResultFilesExist); 154 | Assert.NotNull(baselineTestResults); 155 | Assert.IsTrue(baselineTestResults.Count > 0); 156 | Assert.NotNull(baselinePerformanceTestRunResults); 157 | Assert.IsTrue(baselinePerformanceTestRunResults.Count > 0); 158 | } 159 | 160 | [Test] 161 | public void VerifyV1_Verify_AddBaselineAndNonBaselinePerformanceTestRunResults() 162 | { 163 | // Arrange 164 | var resultJsonFilePath = EnsureFullPath("results.json"); 165 | var baselineJsonFilePath = EnsureFullPath("baseline.json"); 166 | var args = new[] 167 | { 168 | "--format=Json", 169 | string.Format("--testresultsxmlsource={0}", resultJsonFilePath), 170 | string.Format("--baselinexmlsource={0}", baselineJsonFilePath) 171 | , "--version=1" 172 | }; 173 | optionsParser.ParseOptions(PerformanceBenchmark, args); 174 | 175 | // Act 176 | PerformanceBenchmark.AddBaselinePerformanceTestRunResults(testResultJsonParser, baselinePerformanceTestRunResults, baselineTestResults); 177 | PerformanceBenchmark.AddPerformanceTestRunResults(testResultJsonParser, performanceTestRunResults, testResults, new List()); 178 | 179 | // Assert 180 | Assert.IsTrue(PerformanceBenchmark.ResultFilesExist); 181 | AssertCorrectResultJsonFilePaths(new[] { resultJsonFilePath }); 182 | Assert.NotNull(testResults); 183 | Assert.IsTrue(testResults.Count > 0); 184 | Assert.NotNull(performanceTestRunResults); 185 | Assert.IsTrue(performanceTestRunResults.Count > 0); 186 | 187 | Assert.IsTrue(PerformanceBenchmark.BaselineResultFilesExist); 188 | AssertCorrectBaselineJsonFilePaths(new[] { baselineJsonFilePath }); 189 | Assert.NotNull(baselineTestResults); 190 | Assert.IsTrue(baselineTestResults.Count > 0); 191 | Assert.NotNull(baselinePerformanceTestRunResults); 192 | Assert.IsTrue(baselinePerformanceTestRunResults.Count > 0); 193 | } 194 | 195 | public void VerifyV2_AddPerformanceTestRunResults() 196 | { 197 | // Arrange 198 | var resultJsonFilePath = EnsureFullPath("baseline2.json"); 199 | var args = new[] { string.Format("--format=Json", "--testresultsxmlsource={0}", resultJsonFilePath), "--version=2" }; 200 | optionsParser.ParseOptions(PerformanceBenchmark, args); 201 | 202 | // Act 203 | PerformanceBenchmark.AddPerformanceTestRunResults(testResultJsonParser, performanceTestRunResults, testResults, new List()); 204 | 205 | // Assert 206 | Assert.IsTrue(PerformanceBenchmark.ResultFilesExist); 207 | AssertCorrectResultJsonFilePaths(new[] { resultJsonFilePath }); 208 | Assert.NotNull(testResults); 209 | Assert.IsTrue(testResults.Count > 0); 210 | Assert.NotNull(performanceTestRunResults); 211 | Assert.IsTrue(performanceTestRunResults.Count > 0); 212 | } 213 | 214 | [Test] 215 | public void VerifyV2_AddPerformanceTestRunResults_TwoResultFiles() 216 | { 217 | // Arrange 218 | var resultJsonFilePath = EnsureFullPath("ResultsJson2/results2.json"); 219 | var resultFileName2 = EnsureFullPath("results2.json"); 220 | var args = new[] 221 | { 222 | "--format=Json", 223 | string.Format("--testresultsxmlsource={0}", resultJsonFilePath), 224 | string.Format("--testresultsxmlsource={0}", resultFileName2) 225 | , "--version=2" 226 | }; 227 | optionsParser.ParseOptions(PerformanceBenchmark, args); 228 | 229 | // Act 230 | PerformanceBenchmark.AddPerformanceTestRunResults(testResultJsonParser, performanceTestRunResults, testResults, new List()); 231 | 232 | // Assert 233 | Assert.IsTrue(PerformanceBenchmark.ResultFilesExist); 234 | AssertCorrectResultJsonFilePaths(new[] { resultJsonFilePath, resultFileName2 }); 235 | Assert.NotNull(testResults); 236 | Assert.IsTrue(testResults.Count > 0); 237 | Assert.NotNull(performanceTestRunResults); 238 | Assert.IsTrue(performanceTestRunResults.Count > 0); 239 | } 240 | 241 | [Test] 242 | public void VerifyV2_AddPerformanceTestRunResults_OneResultFiles_OneResultDirectory() 243 | { 244 | // Arrange 245 | var resultJsonFilePath = EnsureFullPath("results2.json"); 246 | var resultsJsonDir = EnsureFullPath("ResultsJson2"); 247 | var args = new[] 248 | { 249 | "--format=Json", 250 | string.Format("--testresultsxmlsource={0}", resultJsonFilePath), 251 | string.Format("--testresultsxmlsource={0}", resultsJsonDir) 252 | , "--version=2" 253 | }; 254 | optionsParser.ParseOptions(PerformanceBenchmark, args); 255 | 256 | // Act 257 | PerformanceBenchmark.AddPerformanceTestRunResults(testResultJsonParser, performanceTestRunResults, testResults, new List()); 258 | 259 | // Assert 260 | Assert.IsTrue(PerformanceBenchmark.ResultFilesExist); 261 | AssertCorrectResultsJsonDirectoryPaths(new[] { resultsJsonDir }); 262 | AssertCorrectResultJsonFilePaths(new[] { resultJsonFilePath }); 263 | Assert.NotNull(testResults); 264 | Assert.IsTrue(testResults.Count > 0); 265 | Assert.NotNull(performanceTestRunResults); 266 | Assert.IsTrue(performanceTestRunResults.Count > 0); 267 | } 268 | 269 | [Test] 270 | public void VerifyV2_AddBaselinePerformanceTestRunResults() 271 | { 272 | // Arrange 273 | var resultJsonFilePath = EnsureFullPath("results2.json"); 274 | var baselineJsonFilePath = EnsureFullPath("baseline2.json"); 275 | var args = new[] 276 | { 277 | "--format=Json", 278 | string.Format("--testresultsxmlsource={0}", resultJsonFilePath), 279 | string.Format("--baselinexmlsource={0}", baselineJsonFilePath) 280 | , "--version=2" 281 | }; 282 | optionsParser.ParseOptions(PerformanceBenchmark, args); 283 | 284 | // Act 285 | PerformanceBenchmark.AddBaselinePerformanceTestRunResults(testResultJsonParser, baselinePerformanceTestRunResults, baselineTestResults); 286 | 287 | // Assert 288 | Assert.IsTrue(PerformanceBenchmark.BaselineResultFilesExist); 289 | AssertCorrectBaselineJsonFilePaths(new[] { baselineJsonFilePath }); 290 | AssertCorrectResultJsonFilePaths(new[] { resultJsonFilePath }); 291 | Assert.NotNull(baselineTestResults); 292 | Assert.IsTrue(baselineTestResults.Count > 0); 293 | Assert.NotNull(baselinePerformanceTestRunResults); 294 | Assert.IsTrue(baselinePerformanceTestRunResults.Count > 0); 295 | } 296 | 297 | [Test] 298 | public void VerifyV2_AddBaselinePerformanceTestRunResultsDirectory() 299 | { 300 | // Arrange 301 | var resultJsonFilePath = EnsureFullPath("results2.json"); 302 | var baselineJsonFilePath = EnsureFullPath("baseline2.json"); 303 | var args = new[] 304 | { 305 | "--format=Json", 306 | string.Format("--testresultsxmlsource={0}", resultJsonFilePath), 307 | string.Format("--baselinexmlsource={0}", baselineJsonFilePath) 308 | , "--version=2" 309 | }; 310 | optionsParser.ParseOptions(PerformanceBenchmark, args); 311 | 312 | // Act 313 | PerformanceBenchmark.AddBaselinePerformanceTestRunResults(testResultJsonParser, baselinePerformanceTestRunResults, baselineTestResults); 314 | 315 | // Assert 316 | Assert.IsTrue(PerformanceBenchmark.BaselineResultFilesExist); 317 | Assert.NotNull(baselineTestResults); 318 | Assert.IsTrue(baselineTestResults.Count > 0); 319 | Assert.NotNull(baselinePerformanceTestRunResults); 320 | Assert.IsTrue(baselinePerformanceTestRunResults.Count > 0); 321 | } 322 | 323 | [Test] 324 | public void VerifyV2_Verify_AddBaselineAndNonBaselinePerformanceTestRunResults() 325 | { 326 | // Arrange 327 | var resultJsonFilePath = EnsureFullPath("results2.json"); 328 | var baselineJsonFilePath = EnsureFullPath("baseline2.json"); 329 | var args = new[] 330 | { 331 | "--format=Json", 332 | string.Format("--testresultsxmlsource={0}", resultJsonFilePath), 333 | string.Format("--baselinexmlsource={0}", baselineJsonFilePath) 334 | , "--version=2" 335 | }; 336 | optionsParser.ParseOptions(PerformanceBenchmark, args); 337 | 338 | // Act 339 | PerformanceBenchmark.AddBaselinePerformanceTestRunResults(testResultJsonParser, baselinePerformanceTestRunResults, baselineTestResults); 340 | PerformanceBenchmark.AddPerformanceTestRunResults(testResultJsonParser, performanceTestRunResults, testResults, new List()); 341 | 342 | // Assert 343 | Assert.IsTrue(PerformanceBenchmark.ResultFilesExist); 344 | AssertCorrectResultJsonFilePaths(new[] { resultJsonFilePath }); 345 | Assert.NotNull(testResults); 346 | Assert.IsTrue(testResults.Count > 0); 347 | Assert.NotNull(performanceTestRunResults); 348 | Assert.IsTrue(performanceTestRunResults.Count > 0); 349 | 350 | Assert.IsTrue(PerformanceBenchmark.BaselineResultFilesExist); 351 | AssertCorrectBaselineJsonFilePaths(new[] { baselineJsonFilePath }); 352 | Assert.NotNull(baselineTestResults); 353 | Assert.IsTrue(baselineTestResults.Count > 0); 354 | Assert.NotNull(baselinePerformanceTestRunResults); 355 | Assert.IsTrue(baselinePerformanceTestRunResults.Count > 0); 356 | } 357 | 358 | private void AssertCorrectBaselineJsonFilePaths(string[] baselineJsonFilePaths) 359 | { 360 | foreach (var baselineJsonFilePath in baselineJsonFilePaths) 361 | { 362 | Assert.IsFalse(PerformanceBenchmark.ResultFilePaths.Any(f => f.Equals(baselineJsonFilePath))); 363 | Assert.IsTrue(PerformanceBenchmark.BaselineFilePaths.Any(f => f.Equals(baselineJsonFilePath))); 364 | } 365 | } 366 | 367 | private void AssertCorrectResultJsonFilePaths(string[] resultFileNames) 368 | { 369 | foreach (var resultJsonFilePath in resultFileNames) 370 | { 371 | Assert.IsTrue(PerformanceBenchmark.ResultFilePaths.Contains(resultJsonFilePath)); 372 | Assert.IsFalse(PerformanceBenchmark.BaselineFilePaths.Contains(resultJsonFilePath)); 373 | } 374 | } 375 | 376 | private void AssertCorrectResultsJsonDirectoryPaths(string[] resultsJsonDirPaths) 377 | { 378 | foreach (var resultJsonDirPath in resultsJsonDirPaths) 379 | { 380 | Assert.IsTrue(PerformanceBenchmark.ResultDirectoryPaths.Contains(resultJsonDirPath)); 381 | } 382 | } 383 | } 384 | 385 | } 386 | -------------------------------------------------------------------------------- /UnityPerformanceBenchmarkReporterTests/PerformanceBenchmarkTestsXML.cs: -------------------------------------------------------------------------------- 1 | using System.Collections.Generic; 2 | using System.Linq; 3 | using NUnit.Framework; 4 | using UnityPerformanceBenchmarkReporter; 5 | using UnityPerformanceBenchmarkReporter.Entities; 6 | 7 | namespace UnityPerformanceBenchmarkReporterTests 8 | { 9 | public class PerformanceBenchmarkTestsXML : PerformanceBenchmarkTestsBase 10 | { 11 | private OptionsParser optionsParser; 12 | private IParser testResultXmlParser; 13 | private List performanceTestRunResults; 14 | private List testResults; 15 | private List baselinePerformanceTestRunResults; 16 | private List baselineTestResults; 17 | 18 | [SetUp] 19 | public void Setup() 20 | { 21 | optionsParser = new OptionsParser(); 22 | PerformanceBenchmark = new PerformanceBenchmark(); 23 | testResultXmlParser = new TestResultXmlParser(); 24 | performanceTestRunResults = new List(); 25 | testResults = new List(); 26 | baselinePerformanceTestRunResults = new List(); 27 | baselineTestResults = new List(); 28 | } 29 | 30 | 31 | [Test] 32 | public void Verify_AddPerformanceTestRunResults() 33 | { 34 | // Arrange 35 | var resultXmlFilePath = EnsureFullPath("results.xml"); 36 | var args = new[] { string.Format("--testresultsxmlsource={0}", resultXmlFilePath) }; 37 | optionsParser.ParseOptions(PerformanceBenchmark, args); 38 | 39 | // Act 40 | PerformanceBenchmark.AddPerformanceTestRunResults(testResultXmlParser, performanceTestRunResults, testResults, new List()); 41 | 42 | // Assert 43 | Assert.IsTrue(PerformanceBenchmark.ResultFilesExist); 44 | AssertCorrectResultXmlFilePaths(new[] { resultXmlFilePath }); 45 | Assert.NotNull(testResults); 46 | Assert.IsTrue(testResults.Count > 0); 47 | Assert.NotNull(performanceTestRunResults); 48 | Assert.IsTrue(performanceTestRunResults.Count > 0); 49 | } 50 | 51 | [Test] 52 | public void Verify_AddPerformanceTestRunResults_TwoResultFiles() 53 | { 54 | // Arrange 55 | var resultXmlFilePath = EnsureFullPath("results.xml"); 56 | var resultFileName2 = EnsureFullPath("results2.xml"); 57 | var args = new[] 58 | { 59 | string.Format("--testresultsxmlsource={0}", resultXmlFilePath), 60 | string.Format("--testresultsxmlsource={0}", resultFileName2) 61 | }; 62 | optionsParser.ParseOptions(PerformanceBenchmark, args); 63 | 64 | // Act 65 | PerformanceBenchmark.AddPerformanceTestRunResults(testResultXmlParser, performanceTestRunResults, testResults, new List()); 66 | 67 | // Assert 68 | Assert.IsTrue(PerformanceBenchmark.ResultFilesExist); 69 | AssertCorrectResultXmlFilePaths(new[] { resultXmlFilePath, resultFileName2 }); 70 | Assert.NotNull(testResults); 71 | Assert.IsTrue(testResults.Count > 0); 72 | Assert.NotNull(performanceTestRunResults); 73 | Assert.IsTrue(performanceTestRunResults.Count > 0); 74 | } 75 | 76 | [Test] 77 | public void Verify_AddPerformanceTestRunResults_OneResultFiles_OneResultDirectory() 78 | { 79 | // Arrange 80 | var resultXmlFilePath = EnsureFullPath("results.xml"); 81 | var resultsXmlDir = EnsureFullPath("Results"); 82 | var args = new[] 83 | { 84 | string.Format("--testresultsxmlsource={0}", resultXmlFilePath), 85 | string.Format("--testresultsxmlsource={0}", resultsXmlDir) 86 | }; 87 | optionsParser.ParseOptions(PerformanceBenchmark, args); 88 | 89 | // Act 90 | PerformanceBenchmark.AddPerformanceTestRunResults(testResultXmlParser, performanceTestRunResults, testResults, new List()); 91 | 92 | // Assert 93 | Assert.IsTrue(PerformanceBenchmark.ResultFilesExist); 94 | AssertCorrectResultsXmlDirectoryPaths(new[] { resultsXmlDir }); 95 | AssertCorrectResultXmlFilePaths(new[] { resultXmlFilePath }); 96 | Assert.NotNull(testResults); 97 | Assert.IsTrue(testResults.Count > 0); 98 | Assert.NotNull(performanceTestRunResults); 99 | Assert.IsTrue(performanceTestRunResults.Count > 0); 100 | } 101 | 102 | [Test] 103 | public void Verify_AddBaselinePerformanceTestRunResults() 104 | { 105 | // Arrange 106 | var resultXmlFilePath = EnsureFullPath("results.xml"); 107 | var baselineXmlFilePath = EnsureFullPath("baseline.xml"); 108 | var args = new[] 109 | { 110 | string.Format("--testresultsxmlsource={0}", resultXmlFilePath), 111 | string.Format("--baselinexmlsource={0}", baselineXmlFilePath) 112 | }; 113 | optionsParser.ParseOptions(PerformanceBenchmark, args); 114 | 115 | // Act 116 | PerformanceBenchmark.AddBaselinePerformanceTestRunResults(testResultXmlParser, baselinePerformanceTestRunResults, baselineTestResults); 117 | 118 | // Assert 119 | Assert.IsTrue(PerformanceBenchmark.BaselineResultFilesExist); 120 | AssertCorrectBaselineXmlFilePaths(new[] { baselineXmlFilePath }); 121 | AssertCorrectResultXmlFilePaths(new[] { resultXmlFilePath }); 122 | Assert.NotNull(baselineTestResults); 123 | Assert.IsTrue(baselineTestResults.Count > 0); 124 | Assert.NotNull(baselinePerformanceTestRunResults); 125 | Assert.IsTrue(baselinePerformanceTestRunResults.Count > 0); 126 | } 127 | 128 | [Test] 129 | public void Verify_AddBaselinePerformanceTestRunResultsDirectory() 130 | { 131 | // Arrange 132 | var resultXmlFilePath = EnsureFullPath("results.xml"); 133 | var baselineXmlFilePath = EnsureFullPath("Baselines"); 134 | var args = new[] 135 | { 136 | string.Format("--testresultsxmlsource={0}", resultXmlFilePath), 137 | string.Format("--baselinexmlsource={0}", baselineXmlFilePath) 138 | }; 139 | optionsParser.ParseOptions(PerformanceBenchmark, args); 140 | 141 | // Act 142 | PerformanceBenchmark.AddBaselinePerformanceTestRunResults(testResultXmlParser, baselinePerformanceTestRunResults, baselineTestResults); 143 | 144 | // Assert 145 | Assert.IsTrue(PerformanceBenchmark.BaselineResultFilesExist); 146 | Assert.NotNull(baselineTestResults); 147 | Assert.IsTrue(baselineTestResults.Count > 0); 148 | Assert.NotNull(baselinePerformanceTestRunResults); 149 | Assert.IsTrue(baselinePerformanceTestRunResults.Count > 0); 150 | } 151 | 152 | [Test] 153 | public void Verify_Verify_AddBaselineAndNonBaselinePerformanceTestRunResults() 154 | { 155 | // Arrange 156 | var resultXmlFilePath = EnsureFullPath("results.xml"); 157 | var baselineXmlFilePath = EnsureFullPath("baseline.xml"); 158 | var args = new[] 159 | { 160 | string.Format("--testresultsxmlsource={0}", resultXmlFilePath), 161 | string.Format("--baselinexmlsource={0}", baselineXmlFilePath) 162 | }; 163 | optionsParser.ParseOptions(PerformanceBenchmark, args); 164 | 165 | // Act 166 | PerformanceBenchmark.AddBaselinePerformanceTestRunResults(testResultXmlParser, baselinePerformanceTestRunResults, baselineTestResults); 167 | PerformanceBenchmark.AddPerformanceTestRunResults(testResultXmlParser, performanceTestRunResults, testResults, new List()); 168 | 169 | // Assert 170 | Assert.IsTrue(PerformanceBenchmark.ResultFilesExist); 171 | AssertCorrectResultXmlFilePaths(new[] { resultXmlFilePath }); 172 | Assert.NotNull(testResults); 173 | Assert.IsTrue(testResults.Count > 0); 174 | Assert.NotNull(performanceTestRunResults); 175 | Assert.IsTrue(performanceTestRunResults.Count > 0); 176 | 177 | Assert.IsTrue(PerformanceBenchmark.BaselineResultFilesExist); 178 | AssertCorrectBaselineXmlFilePaths(new[] { baselineXmlFilePath }); 179 | Assert.NotNull(baselineTestResults); 180 | Assert.IsTrue(baselineTestResults.Count > 0); 181 | Assert.NotNull(baselinePerformanceTestRunResults); 182 | Assert.IsTrue(baselinePerformanceTestRunResults.Count > 0); 183 | } 184 | 185 | private void AssertCorrectBaselineXmlFilePaths(string[] baselineXmlFilePaths) 186 | { 187 | foreach (var baselineXmlFilePath in baselineXmlFilePaths) 188 | { 189 | Assert.IsFalse(PerformanceBenchmark.ResultFilePaths.Any(f => f.Equals(baselineXmlFilePath))); 190 | Assert.IsTrue(PerformanceBenchmark.BaselineFilePaths.Any(f => f.Equals(baselineXmlFilePath))); 191 | } 192 | } 193 | 194 | private void AssertCorrectResultXmlFilePaths(string[] resultFileNames) 195 | { 196 | foreach (var resultXmlFilePath in resultFileNames) 197 | { 198 | Assert.IsTrue(PerformanceBenchmark.ResultFilePaths.Contains(resultXmlFilePath)); 199 | Assert.IsFalse(PerformanceBenchmark.BaselineFilePaths.Contains(resultXmlFilePath)); 200 | } 201 | } 202 | 203 | private void AssertCorrectResultsXmlDirectoryPaths(string[] resultsXmlDirPaths) 204 | { 205 | foreach (var resultXmlDirPath in resultsXmlDirPaths) 206 | { 207 | Assert.IsTrue(PerformanceBenchmark.ResultDirectoryPaths.Contains(resultXmlDirPath)); 208 | } 209 | } 210 | } 211 | } 212 | -------------------------------------------------------------------------------- /UnityPerformanceBenchmarkReporterTests/UnityPerformanceBenchmarkReporterTests.csproj: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | false 6 | 7 | netcoreapp3.1;net6.0;net7.0 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | PreserveNewest 23 | 24 | 25 | PreserveNewest 26 | 27 | 28 | PreserveNewest 29 | 30 | 31 | PreserveNewest 32 | 33 | 34 | PreserveNewest 35 | 36 | 37 | PreserveNewest 38 | 39 | 40 | PreserveNewest 41 | 42 | 43 | PreserveNewest 44 | 45 | 46 | PreserveNewest 47 | 48 | 49 | PreserveNewest 50 | 51 | 52 | PreserveNewest 53 | 54 | 55 | PreserveNewest 56 | 57 | 58 | PreserveNewest 59 | 60 | 61 | PreserveNewest 62 | 63 | 64 | PreserveNewest 65 | 66 | 67 | PreserveNewest 68 | 69 | 70 | PreserveNewest 71 | 72 | 73 | 74 | -------------------------------------------------------------------------------- /dotnet-install.ps1: -------------------------------------------------------------------------------- 1 | # 2 | # Copyright (c) .NET Foundation and contributors. All rights reserved. 3 | # Licensed under the MIT license. See LICENSE file in the project root for full license information. 4 | # 5 | 6 | <# 7 | .SYNOPSIS 8 | Installs dotnet cli 9 | .DESCRIPTION 10 | Installs dotnet cli. If dotnet installation already exists in the given directory 11 | it will update it only if the requested version differs from the one already installed. 12 | .PARAMETER Channel 13 | Default: LTS 14 | Download from the Channel specified. Possible values: 15 | - Current - most current release 16 | - LTS - most current supported release 17 | - 2-part version in a format A.B - represents a specific release 18 | examples: 2.0; 1.0 19 | - Branch name 20 | examples: release/2.0.0; Master 21 | .PARAMETER Version 22 | Default: latest 23 | Represents a build version on specific channel. Possible values: 24 | - latest - most latest build on specific channel 25 | - coherent - most latest coherent build on specific channel 26 | coherent applies only to SDK downloads 27 | - 3-part version in a format A.B.C - represents specific version of build 28 | examples: 2.0.0-preview2-006120; 1.1.0 29 | .PARAMETER InstallDir 30 | Default: %LocalAppData%\Microsoft\dotnet 31 | Path to where to install dotnet. Note that binaries will be placed directly in a given directory. 32 | .PARAMETER Architecture 33 | Default: - this value represents currently running OS architecture 34 | Architecture of dotnet binaries to be installed. 35 | Possible values are: , x64 and x86 36 | .PARAMETER SharedRuntime 37 | This parameter is obsolete and may be removed in a future version of this script. 38 | The recommended alternative is '-Runtime dotnet'. 39 | 40 | Default: false 41 | Installs just the shared runtime bits, not the entire SDK. 42 | This is equivalent to specifying `-Runtime dotnet`. 43 | .PARAMETER Runtime 44 | Installs just a shared runtime, not the entire SDK. 45 | Possible values: 46 | - dotnet - the Microsoft.NETCore.App shared runtime 47 | - aspnetcore - the Microsoft.AspNetCore.App shared runtime 48 | .PARAMETER DryRun 49 | If set it will not perform installation but instead display what command line to use to consistently install 50 | currently requested version of dotnet cli. In example if you specify version 'latest' it will display a link 51 | with specific version so that this command can be used deterministicly in a build script. 52 | It also displays binaries location if you prefer to install or download it yourself. 53 | .PARAMETER NoPath 54 | By default this script will set environment variable PATH for the current process to the binaries folder inside installation folder. 55 | If set it will display binaries location but not set any environment variable. 56 | .PARAMETER Verbose 57 | Displays diagnostics information. 58 | .PARAMETER AzureFeed 59 | Default: https://dotnetcli.azureedge.net/dotnet 60 | This parameter typically is not changed by the user. 61 | It allows changing the URL for the Azure feed used by this installer. 62 | .PARAMETER UncachedFeed 63 | This parameter typically is not changed by the user. 64 | It allows changing the URL for the Uncached feed used by this installer. 65 | .PARAMETER FeedCredential 66 | Used as a query string to append to the Azure feed. 67 | It allows changing the URL to use non-public blob storage accounts. 68 | .PARAMETER ProxyAddress 69 | If set, the installer will use the proxy when making web requests 70 | .PARAMETER ProxyUseDefaultCredentials 71 | Default: false 72 | Use default credentials, when using proxy address. 73 | .PARAMETER SkipNonVersionedFiles 74 | Default: false 75 | Skips installing non-versioned files if they already exist, such as dotnet.exe. 76 | .PARAMETER NoCdn 77 | Disable downloading from the Azure CDN, and use the uncached feed directly. 78 | #> 79 | [cmdletbinding()] 80 | param( 81 | [string]$Channel="LTS", 82 | [string]$Version="Latest", 83 | [string]$InstallDir="", 84 | [string]$Architecture="", 85 | [ValidateSet("dotnet", "aspnetcore", IgnoreCase = $false)] 86 | [string]$Runtime, 87 | [Obsolete("This parameter may be removed in a future version of this script. The recommended alternative is '-Runtime dotnet'.")] 88 | [switch]$SharedRuntime, 89 | [switch]$DryRun, 90 | [switch]$NoPath, 91 | [string]$AzureFeed="https://dotnetcli.azureedge.net/dotnet", 92 | [string]$UncachedFeed="https://dotnetcli.blob.core.windows.net/dotnet", 93 | [string]$FeedCredential, 94 | [string]$ProxyAddress, 95 | [switch]$ProxyUseDefaultCredentials, 96 | [switch]$SkipNonVersionedFiles, 97 | [switch]$NoCdn 98 | ) 99 | 100 | Set-StrictMode -Version Latest 101 | $ErrorActionPreference="Stop" 102 | $ProgressPreference="SilentlyContinue" 103 | 104 | if ($NoCdn) { 105 | $AzureFeed = $UncachedFeed 106 | } 107 | 108 | $BinFolderRelativePath="" 109 | 110 | if ($SharedRuntime -and (-not $Runtime)) { 111 | $Runtime = "dotnet" 112 | } 113 | 114 | # example path with regex: shared/1.0.0-beta-12345/somepath 115 | $VersionRegEx="/\d+\.\d+[^/]+/" 116 | $OverrideNonVersionedFiles = !$SkipNonVersionedFiles 117 | 118 | function Say($str) { 119 | Write-Host "dotnet-install: $str" 120 | } 121 | 122 | function Say-Verbose($str) { 123 | Write-Verbose "dotnet-install: $str" 124 | } 125 | 126 | function Say-Invocation($Invocation) { 127 | $command = $Invocation.MyCommand; 128 | $args = (($Invocation.BoundParameters.Keys | foreach { "-$_ `"$($Invocation.BoundParameters[$_])`"" }) -join " ") 129 | Say-Verbose "$command $args" 130 | } 131 | 132 | function Invoke-With-Retry([ScriptBlock]$ScriptBlock, [int]$MaxAttempts = 3, [int]$SecondsBetweenAttempts = 1) { 133 | $Attempts = 0 134 | 135 | while ($true) { 136 | try { 137 | return $ScriptBlock.Invoke() 138 | } 139 | catch { 140 | $Attempts++ 141 | if ($Attempts -lt $MaxAttempts) { 142 | Start-Sleep $SecondsBetweenAttempts 143 | } 144 | else { 145 | throw 146 | } 147 | } 148 | } 149 | } 150 | 151 | function Get-Machine-Architecture() { 152 | Say-Invocation $MyInvocation 153 | 154 | # possible values: AMD64, IA64, x86 155 | return $ENV:PROCESSOR_ARCHITECTURE 156 | } 157 | 158 | # TODO: Architecture and CLIArchitecture should be unified 159 | function Get-CLIArchitecture-From-Architecture([string]$Architecture) { 160 | Say-Invocation $MyInvocation 161 | 162 | switch ($Architecture.ToLower()) { 163 | { $_ -eq "" } { return Get-CLIArchitecture-From-Architecture $(Get-Machine-Architecture) } 164 | { ($_ -eq "amd64") -or ($_ -eq "x64") } { return "x64" } 165 | { $_ -eq "x86" } { return "x86" } 166 | default { throw "Architecture not supported. If you think this is a bug, please report it at https://github.com/dotnet/cli/issues" } 167 | } 168 | } 169 | 170 | function Get-Version-Info-From-Version-Text([string]$VersionText) { 171 | Say-Invocation $MyInvocation 172 | 173 | $Data = @($VersionText.Split([char[]]@(), [StringSplitOptions]::RemoveEmptyEntries)); 174 | 175 | $VersionInfo = @{} 176 | $VersionInfo.CommitHash = $Data[0].Trim() 177 | $VersionInfo.Version = $Data[1].Trim() 178 | return $VersionInfo 179 | } 180 | 181 | function Load-Assembly([string] $Assembly) { 182 | try { 183 | Add-Type -Assembly $Assembly | Out-Null 184 | } 185 | catch { 186 | # On Nano Server, Powershell Core Edition is used. Add-Type is unable to resolve base class assemblies because they are not GAC'd. 187 | # Loading the base class assemblies is not unnecessary as the types will automatically get resolved. 188 | } 189 | } 190 | 191 | function GetHTTPResponse([Uri] $Uri) 192 | { 193 | Invoke-With-Retry( 194 | { 195 | 196 | $HttpClient = $null 197 | 198 | try { 199 | # HttpClient is used vs Invoke-WebRequest in order to support Nano Server which doesn't support the Invoke-WebRequest cmdlet. 200 | Load-Assembly -Assembly System.Net.Http 201 | 202 | if(-not $ProxyAddress) { 203 | try { 204 | # Despite no proxy being explicitly specified, we may still be behind a default proxy 205 | $DefaultProxy = [System.Net.WebRequest]::DefaultWebProxy; 206 | if($DefaultProxy -and (-not $DefaultProxy.IsBypassed($Uri))) { 207 | $ProxyAddress = $DefaultProxy.GetProxy($Uri).OriginalString 208 | $ProxyUseDefaultCredentials = $true 209 | } 210 | } catch { 211 | # Eat the exception and move forward as the above code is an attempt 212 | # at resolving the DefaultProxy that may not have been a problem. 213 | $ProxyAddress = $null 214 | Say-Verbose("Exception ignored: $_.Exception.Message - moving forward...") 215 | } 216 | } 217 | 218 | if($ProxyAddress) { 219 | $HttpClientHandler = New-Object System.Net.Http.HttpClientHandler 220 | $HttpClientHandler.Proxy = New-Object System.Net.WebProxy -Property @{Address=$ProxyAddress;UseDefaultCredentials=$ProxyUseDefaultCredentials} 221 | $HttpClient = New-Object System.Net.Http.HttpClient -ArgumentList $HttpClientHandler 222 | } 223 | else { 224 | 225 | $HttpClient = New-Object System.Net.Http.HttpClient 226 | } 227 | # Default timeout for HttpClient is 100s. For a 50 MB download this assumes 500 KB/s average, any less will time out 228 | # 10 minutes allows it to work over much slower connections. 229 | $HttpClient.Timeout = New-TimeSpan -Minutes 10 230 | $Response = $HttpClient.GetAsync("${Uri}${FeedCredential}").Result 231 | if (($Response -eq $null) -or (-not ($Response.IsSuccessStatusCode))) { 232 | # The feed credential is potentially sensitive info. Do not log FeedCredential to console output. 233 | $ErrorMsg = "Failed to download $Uri." 234 | if ($Response -ne $null) { 235 | $ErrorMsg += " $Response" 236 | } 237 | 238 | throw $ErrorMsg 239 | } 240 | 241 | return $Response 242 | } 243 | finally { 244 | if ($HttpClient -ne $null) { 245 | $HttpClient.Dispose() 246 | } 247 | } 248 | }) 249 | } 250 | 251 | 252 | function Get-Latest-Version-Info([string]$AzureFeed, [string]$Channel, [bool]$Coherent) { 253 | Say-Invocation $MyInvocation 254 | 255 | $VersionFileUrl = $null 256 | if ($Runtime -eq "dotnet") { 257 | $VersionFileUrl = "$UncachedFeed/Runtime/$Channel/latest.version" 258 | } 259 | elseif ($Runtime -eq "aspnetcore") { 260 | $VersionFileUrl = "$UncachedFeed/aspnetcore/Runtime/$Channel/latest.version" 261 | } 262 | elseif (-not $Runtime) { 263 | if ($Coherent) { 264 | $VersionFileUrl = "$UncachedFeed/Sdk/$Channel/latest.coherent.version" 265 | } 266 | else { 267 | $VersionFileUrl = "$UncachedFeed/Sdk/$Channel/latest.version" 268 | } 269 | } 270 | else { 271 | throw "Invalid value for `$Runtime" 272 | } 273 | 274 | $Response = GetHTTPResponse -Uri $VersionFileUrl 275 | $StringContent = $Response.Content.ReadAsStringAsync().Result 276 | 277 | switch ($Response.Content.Headers.ContentType) { 278 | { ($_ -eq "application/octet-stream") } { $VersionText = $StringContent } 279 | { ($_ -eq "text/plain") } { $VersionText = $StringContent } 280 | { ($_ -eq "text/plain; charset=UTF-8") } { $VersionText = $StringContent } 281 | default { throw "``$Response.Content.Headers.ContentType`` is an unknown .version file content type." } 282 | } 283 | 284 | $VersionInfo = Get-Version-Info-From-Version-Text $VersionText 285 | 286 | return $VersionInfo 287 | } 288 | 289 | 290 | function Get-Specific-Version-From-Version([string]$AzureFeed, [string]$Channel, [string]$Version) { 291 | Say-Invocation $MyInvocation 292 | 293 | switch ($Version.ToLower()) { 294 | { $_ -eq "latest" } { 295 | $LatestVersionInfo = Get-Latest-Version-Info -AzureFeed $AzureFeed -Channel $Channel -Coherent $False 296 | return $LatestVersionInfo.Version 297 | } 298 | { $_ -eq "coherent" } { 299 | $LatestVersionInfo = Get-Latest-Version-Info -AzureFeed $AzureFeed -Channel $Channel -Coherent $True 300 | return $LatestVersionInfo.Version 301 | } 302 | default { return $Version } 303 | } 304 | } 305 | 306 | function Get-Download-Link([string]$AzureFeed, [string]$SpecificVersion, [string]$CLIArchitecture) { 307 | Say-Invocation $MyInvocation 308 | 309 | if ($Runtime -eq "dotnet") { 310 | $PayloadURL = "$AzureFeed/Runtime/$SpecificVersion/dotnet-runtime-$SpecificVersion-win-$CLIArchitecture.zip" 311 | } 312 | elseif ($Runtime -eq "aspnetcore") { 313 | $PayloadURL = "$AzureFeed/aspnetcore/Runtime/$SpecificVersion/aspnetcore-runtime-$SpecificVersion-win-$CLIArchitecture.zip" 314 | } 315 | elseif (-not $Runtime) { 316 | $PayloadURL = "$AzureFeed/Sdk/$SpecificVersion/dotnet-sdk-$SpecificVersion-win-$CLIArchitecture.zip" 317 | } 318 | else { 319 | throw "Invalid value for `$Runtime" 320 | } 321 | 322 | Say-Verbose "Constructed primary payload URL: $PayloadURL" 323 | 324 | return $PayloadURL 325 | } 326 | 327 | function Get-LegacyDownload-Link([string]$AzureFeed, [string]$SpecificVersion, [string]$CLIArchitecture) { 328 | Say-Invocation $MyInvocation 329 | 330 | if (-not $Runtime) { 331 | $PayloadURL = "$AzureFeed/Sdk/$SpecificVersion/dotnet-dev-win-$CLIArchitecture.$SpecificVersion.zip" 332 | } 333 | elseif ($Runtime -eq "dotnet") { 334 | $PayloadURL = "$AzureFeed/Runtime/$SpecificVersion/dotnet-win-$CLIArchitecture.$SpecificVersion.zip" 335 | } 336 | else { 337 | return $null 338 | } 339 | 340 | Say-Verbose "Constructed legacy payload URL: $PayloadURL" 341 | 342 | return $PayloadURL 343 | } 344 | 345 | function Get-User-Share-Path() { 346 | Say-Invocation $MyInvocation 347 | 348 | $InstallRoot = $env:DOTNET_INSTALL_DIR 349 | if (!$InstallRoot) { 350 | $InstallRoot = "$env:LocalAppData\Microsoft\dotnet" 351 | } 352 | return $InstallRoot 353 | } 354 | 355 | function Resolve-Installation-Path([string]$InstallDir) { 356 | Say-Invocation $MyInvocation 357 | 358 | if ($InstallDir -eq "") { 359 | return Get-User-Share-Path 360 | } 361 | return $InstallDir 362 | } 363 | 364 | function Get-Version-Info-From-Version-File([string]$InstallRoot, [string]$RelativePathToVersionFile) { 365 | Say-Invocation $MyInvocation 366 | 367 | $VersionFile = Join-Path -Path $InstallRoot -ChildPath $RelativePathToVersionFile 368 | Say-Verbose "Local version file: $VersionFile" 369 | 370 | if (Test-Path $VersionFile) { 371 | $VersionText = cat $VersionFile 372 | Say-Verbose "Local version file text: $VersionText" 373 | return Get-Version-Info-From-Version-Text $VersionText 374 | } 375 | 376 | Say-Verbose "Local version file not found." 377 | 378 | return $null 379 | } 380 | 381 | function Is-Dotnet-Package-Installed([string]$InstallRoot, [string]$RelativePathToPackage, [string]$SpecificVersion) { 382 | Say-Invocation $MyInvocation 383 | 384 | $DotnetPackagePath = Join-Path -Path $InstallRoot -ChildPath $RelativePathToPackage | Join-Path -ChildPath $SpecificVersion 385 | Say-Verbose "Is-Dotnet-Package-Installed: Path to a package: $DotnetPackagePath" 386 | return Test-Path $DotnetPackagePath -PathType Container 387 | } 388 | 389 | function Get-Absolute-Path([string]$RelativeOrAbsolutePath) { 390 | # Too much spam 391 | # Say-Invocation $MyInvocation 392 | 393 | return $ExecutionContext.SessionState.Path.GetUnresolvedProviderPathFromPSPath($RelativeOrAbsolutePath) 394 | } 395 | 396 | function Get-Path-Prefix-With-Version($path) { 397 | $match = [regex]::match($path, $VersionRegEx) 398 | if ($match.Success) { 399 | return $entry.FullName.Substring(0, $match.Index + $match.Length) 400 | } 401 | 402 | return $null 403 | } 404 | 405 | function Get-List-Of-Directories-And-Versions-To-Unpack-From-Dotnet-Package([System.IO.Compression.ZipArchive]$Zip, [string]$OutPath) { 406 | Say-Invocation $MyInvocation 407 | 408 | $ret = @() 409 | foreach ($entry in $Zip.Entries) { 410 | $dir = Get-Path-Prefix-With-Version $entry.FullName 411 | if ($dir -ne $null) { 412 | $path = Get-Absolute-Path $(Join-Path -Path $OutPath -ChildPath $dir) 413 | if (-Not (Test-Path $path -PathType Container)) { 414 | $ret += $dir 415 | } 416 | } 417 | } 418 | 419 | $ret = $ret | Sort-Object | Get-Unique 420 | 421 | $values = ($ret | foreach { "$_" }) -join ";" 422 | Say-Verbose "Directories to unpack: $values" 423 | 424 | return $ret 425 | } 426 | 427 | # Example zip content and extraction algorithm: 428 | # Rule: files if extracted are always being extracted to the same relative path locally 429 | # .\ 430 | # a.exe # file does not exist locally, extract 431 | # b.dll # file exists locally, override only if $OverrideFiles set 432 | # aaa\ # same rules as for files 433 | # ... 434 | # abc\1.0.0\ # directory contains version and exists locally 435 | # ... # do not extract content under versioned part 436 | # abc\asd\ # same rules as for files 437 | # ... 438 | # def\ghi\1.0.1\ # directory contains version and does not exist locally 439 | # ... # extract content 440 | function Extract-Dotnet-Package([string]$ZipPath, [string]$OutPath) { 441 | Say-Invocation $MyInvocation 442 | 443 | Load-Assembly -Assembly System.IO.Compression.FileSystem 444 | Set-Variable -Name Zip 445 | try { 446 | $Zip = [System.IO.Compression.ZipFile]::OpenRead($ZipPath) 447 | 448 | $DirectoriesToUnpack = Get-List-Of-Directories-And-Versions-To-Unpack-From-Dotnet-Package -Zip $Zip -OutPath $OutPath 449 | 450 | foreach ($entry in $Zip.Entries) { 451 | $PathWithVersion = Get-Path-Prefix-With-Version $entry.FullName 452 | if (($PathWithVersion -eq $null) -Or ($DirectoriesToUnpack -contains $PathWithVersion)) { 453 | $DestinationPath = Get-Absolute-Path $(Join-Path -Path $OutPath -ChildPath $entry.FullName) 454 | $DestinationDir = Split-Path -Parent $DestinationPath 455 | $OverrideFiles=$OverrideNonVersionedFiles -Or (-Not (Test-Path $DestinationPath)) 456 | if ((-Not $DestinationPath.EndsWith("\")) -And $OverrideFiles) { 457 | New-Item -ItemType Directory -Force -Path $DestinationDir | Out-Null 458 | [System.IO.Compression.ZipFileExtensions]::ExtractToFile($entry, $DestinationPath, $OverrideNonVersionedFiles) 459 | } 460 | } 461 | } 462 | } 463 | finally { 464 | if ($Zip -ne $null) { 465 | $Zip.Dispose() 466 | } 467 | } 468 | } 469 | 470 | function DownloadFile([Uri]$Uri, [string]$OutPath) { 471 | if ($Uri -notlike "http*") { 472 | Say-Verbose "Copying file from $Uri to $OutPath" 473 | Copy-Item $Uri.AbsolutePath $OutPath 474 | return 475 | } 476 | 477 | $Stream = $null 478 | 479 | try { 480 | $Response = GetHTTPResponse -Uri $Uri 481 | $Stream = $Response.Content.ReadAsStreamAsync().Result 482 | $File = [System.IO.File]::Create($OutPath) 483 | $Stream.CopyTo($File) 484 | $File.Close() 485 | } 486 | finally { 487 | if ($Stream -ne $null) { 488 | $Stream.Dispose() 489 | } 490 | } 491 | } 492 | 493 | function Prepend-Sdk-InstallRoot-To-Path([string]$InstallRoot, [string]$BinFolderRelativePath) { 494 | $BinPath = Get-Absolute-Path $(Join-Path -Path $InstallRoot -ChildPath $BinFolderRelativePath) 495 | if (-Not $NoPath) { 496 | Say "Adding to current process PATH: `"$BinPath`". Note: This change will not be visible if PowerShell was run as a child process." 497 | $env:path = "$BinPath;" + $env:path 498 | } 499 | else { 500 | Say "Binaries of dotnet can be found in $BinPath" 501 | } 502 | } 503 | 504 | $CLIArchitecture = Get-CLIArchitecture-From-Architecture $Architecture 505 | $SpecificVersion = Get-Specific-Version-From-Version -AzureFeed $AzureFeed -Channel $Channel -Version $Version 506 | $DownloadLink = Get-Download-Link -AzureFeed $AzureFeed -SpecificVersion $SpecificVersion -CLIArchitecture $CLIArchitecture 507 | $LegacyDownloadLink = Get-LegacyDownload-Link -AzureFeed $AzureFeed -SpecificVersion $SpecificVersion -CLIArchitecture $CLIArchitecture 508 | 509 | if ($DryRun) { 510 | Say "Payload URLs:" 511 | Say "Primary - $DownloadLink" 512 | if ($LegacyDownloadLink) { 513 | Say "Legacy - $LegacyDownloadLink" 514 | } 515 | Say "Repeatable invocation: .\$($MyInvocation.Line)" 516 | exit 0 517 | } 518 | 519 | $InstallRoot = Resolve-Installation-Path $InstallDir 520 | Say-Verbose "InstallRoot: $InstallRoot" 521 | 522 | if ($Runtime -eq "dotnet") { 523 | $assetName = ".NET Core Runtime" 524 | $dotnetPackageRelativePath = "shared\Microsoft.NETCore.App" 525 | } 526 | elseif ($Runtime -eq "aspnetcore") { 527 | $assetName = "ASP.NET Core Runtime" 528 | $dotnetPackageRelativePath = "shared\Microsoft.AspNetCore.App" 529 | } 530 | elseif (-not $Runtime) { 531 | $assetName = ".NET Core SDK" 532 | $dotnetPackageRelativePath = "sdk" 533 | } 534 | else { 535 | throw "Invalid value for `$Runtime" 536 | } 537 | 538 | # Check if the SDK version is already installed. 539 | $isAssetInstalled = Is-Dotnet-Package-Installed -InstallRoot $InstallRoot -RelativePathToPackage $dotnetPackageRelativePath -SpecificVersion $SpecificVersion 540 | if ($isAssetInstalled) { 541 | Say "$assetName version $SpecificVersion is already installed." 542 | Prepend-Sdk-InstallRoot-To-Path -InstallRoot $InstallRoot -BinFolderRelativePath $BinFolderRelativePath 543 | exit 0 544 | } 545 | 546 | New-Item -ItemType Directory -Force -Path $InstallRoot | Out-Null 547 | 548 | $installDrive = $((Get-Item $InstallRoot).PSDrive.Name); 549 | $free = Get-CimInstance -Class win32_logicaldisk | where Deviceid -eq "${installDrive}:" 550 | if ($free.Freespace / 1MB -le 100 ) { 551 | Say "There is not enough disk space on drive ${installDrive}:" 552 | exit 0 553 | } 554 | 555 | $ZipPath = [System.IO.Path]::combine([System.IO.Path]::GetTempPath(), [System.IO.Path]::GetRandomFileName()) 556 | Say-Verbose "Zip path: $ZipPath" 557 | Say "Downloading link: $DownloadLink" 558 | try { 559 | DownloadFile -Uri $DownloadLink -OutPath $ZipPath 560 | } 561 | catch { 562 | Say "Cannot download: $DownloadLink" 563 | if ($LegacyDownloadLink) { 564 | $DownloadLink = $LegacyDownloadLink 565 | $ZipPath = [System.IO.Path]::combine([System.IO.Path]::GetTempPath(), [System.IO.Path]::GetRandomFileName()) 566 | Say-Verbose "Legacy zip path: $ZipPath" 567 | Say "Downloading legacy link: $DownloadLink" 568 | DownloadFile -Uri $DownloadLink -OutPath $ZipPath 569 | } 570 | else { 571 | throw "Could not download $assetName version $SpecificVersion" 572 | } 573 | } 574 | 575 | Say "Extracting zip from $DownloadLink" 576 | Extract-Dotnet-Package -ZipPath $ZipPath -OutPath $InstallRoot 577 | 578 | # Check if the SDK version is now installed; if not, fail the installation. 579 | $isAssetInstalled = Is-Dotnet-Package-Installed -InstallRoot $InstallRoot -RelativePathToPackage $dotnetPackageRelativePath -SpecificVersion $SpecificVersion 580 | if (!$isAssetInstalled) { 581 | throw "$assetName version $SpecificVersion failed to install with an unknown error." 582 | } 583 | 584 | Remove-Item $ZipPath 585 | 586 | Prepend-Sdk-InstallRoot-To-Path -InstallRoot $InstallRoot -BinFolderRelativePath $BinFolderRelativePath 587 | 588 | Say "Installation finished" 589 | exit 0 590 | -------------------------------------------------------------------------------- /latest_custom_UnityPerformanceBenchmark_2020-06-29_01-08-13-481.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | Unity Performance Benchmark Report 6 | 7 | 8 | 15 | 281 | 282 | 283 | 284 | 285 | 286 |

Performance Benchmark Report

287 | 290 | 318 |
288 |
Mismatched test configurations present
 289 |
291 |
292 |
PlayerSystemInfo
293 | 
294 | 
OperatingSystem
Android OS 7.0 / API-24 (NRD90M/G930AUCS4BQH1)
DeviceModel
samsung SAMSUNG-SM-G930A
DeviceName
SAMSUNG-SM-G930A
ProcessorType
ARM64 FP ASIMD AES
ProcessorCount
4
GraphicsDeviceName
Adreno (TM) 530
SystemMemorySize
3417
XrDevice
Metadata not available

295 |
296 |
PlayerSettings
297 | 
298 | 
ScriptingBackend
Mono2x
OculusPluginVersion
Metadata not available
DeviceRuntimeVersion
Metadata not available
XrSdkName
Metadata not available
XrSdkVersion
Metadata not available
XrSdkRevision
Metadata not available
XrSdkRevisionDate
Metadata not available
XrSdkBranch
Metadata not available
XrManagementVersion
Metadata not available
XrManagementRevision
Metadata not available
DeviceUniqueId
881d5e2c1b201c0e3f5331494fd1ea10dcf68810
Username
seans
RenderPipeline
BuiltInRenderer
FfrLevel
Metadata not available
TestsBranch
Metadata not available
TestsRev
Metadata not available
TestsRevDate
Metadata not available
PerfTestsPackageName
Metadata not available
PerfTestsVersion
Metadata not available
PerfTestsRevision
Metadata not available
AndroidTargetArchitecture
Metadata not available
UrpVersion
Metadata not available
UrpRevision
Metadata not available
GpuSkinning
True
GraphicsApi
ValueResult FilePath
OpenGLES32019.3_Perf_Android_BuiltInRP_OpenGLES3_Linear_Mono.xmlD:\PerfResults\dev
Vulkan2019.3_Perf_Android_BuiltInRP_Vulkan_Linear_Mono.xmlD:\PerfResults\dev
StereoRenderingPath
MultiPass
RenderThreadingMode
MultiThreaded

299 |
300 |
ScreenSettings
301 | 
302 | 
ScreenWidth
1920
ScreenHeight
1080
ScreenRefreshRate
60
Fullscreen
True

303 |
304 |
QualitySettings
305 | 
306 | 
Vsync
1
AntiAliasing
4
ColorSpace
Linear
AnisotropicFiltering
Enable
BlendWeights
TwoBones

307 |
308 |
BuildSettings
309 | 
310 | 
Platform
Android
BuildTarget
Android
AndroidBuildSystem
Gradle

311 |
312 |
EditorVersion
313 | 
314 | 
FullVersion
2019.3.0f6 (27ab2135bccf)
Branch
2019.3/staging

315 |
316 |
317 |
319 | 324 |
320 |
325 | 326 | 327 | 328 | 329 | 332 | 333 | 334 | 335 | 338 | 339 | 340 | 341 | 344 |

Metric:

FrameTime

330 | 331 |

Test Name:

OculusStatsTests_StatsTest_SimpleSceneWithLitCube

336 | 337 |

Test Name:

OculusStatsTests_StatsTest_SimpleSceneWithManyLitCubes

342 | 343 |
345 | 346 | 347 | --------------------------------------------------------------------------------