├── .config └── tsaoptions.json ├── .gitattributes ├── .github ├── ISSUE_TEMPLATE │ ├── bug-report.md │ ├── config.yml │ └── feature-request.md ├── labeler.yml ├── linters │ ├── .linkspector.yml │ └── .markdown-lint.yml └── workflows │ ├── check-markdown-links.yml │ ├── labeler.yml │ └── lint-code-base.yml ├── .gitignore ├── CODE-OF-CONDUCT.md ├── CODEOWNERS ├── Documentation ├── VMR-re-bootstrapping.md ├── boostrap-new-os.md ├── bootstrapping-guidelines.md ├── ci-platform-coverage-guidelines.md ├── eliminating-pre-builts.md ├── leak-detection.md ├── package-dependency-flow.md ├── packaging-installation.md ├── patching-guidelines.md ├── planning │ ├── arcade-powered-source-build │ │ ├── README.md │ │ ├── img │ │ │ ├── official-leaves.dot │ │ │ ├── official-leaves.svg │ │ │ ├── rough-pipeline-official.png │ │ │ ├── rough-pipeline-validate.png │ │ │ ├── spec-build-1-general.png │ │ │ ├── spec-build-2-bootstrap.png │ │ │ └── spec-build-3-n-1.png │ │ ├── in-arcade.md │ │ ├── incremental-official-chunked.md │ │ ├── incremental-official.md │ │ ├── intermediate-nupkg.md │ │ ├── source-build-in-pipeline.md │ │ └── speculative-build.md │ └── multi-sdk-band-support.md ├── poison-report-format.md ├── sourcebuild-in-repos │ ├── README.md │ ├── adding-features.md │ ├── build-info.md │ ├── img │ │ └── ci-job.png │ ├── new-dependencies.md │ ├── new-repo.md │ └── update-dependencies.md └── system-requirements.md ├── LICENSE.txt └── README.md /.config/tsaoptions.json: -------------------------------------------------------------------------------- 1 | { 2 | "instanceUrl": "https://devdiv.visualstudio.com/", 3 | "template": "TFSDEVDIV", 4 | "projectName": "DEVDIV", 5 | "areaPath": "DevDiv\\NET Fundamentals\\.NET Acquisition\\Source-build", 6 | "iterationPath": "DevDiv", 7 | "notificationAliases": [ "dnsbcgalerts@microsoft.com" ], 8 | "repositoryName":"source-build", 9 | "codebaseName": "source-build" 10 | } 11 | -------------------------------------------------------------------------------- /.gitattributes: -------------------------------------------------------------------------------- 1 | * text=auto 2 | 3 | # patch files need lf line-endings, even on Windows 4 | *.patch text eol=lf 5 | *.sh text eol=lf 6 | -------------------------------------------------------------------------------- /.github/ISSUE_TEMPLATE/bug-report.md: -------------------------------------------------------------------------------- 1 | --- 2 | name: 🐞 Bug report 3 | about: Create a report about something that is not working 4 | labels: 'bug' 5 | --- 6 | 7 | 14 | 15 | ### Describe the Bug 16 | 17 | 18 | 19 | ### Steps to Reproduce 20 | 21 | 28 | 29 | ### Other Information 30 | 31 | 36 | -------------------------------------------------------------------------------- /.github/ISSUE_TEMPLATE/config.yml: -------------------------------------------------------------------------------- 1 | blank_issues_enabled: true 2 | contact_links: 3 | - name: Discussions 4 | url: https://github.com/dotnet/source-build/discussions 5 | about: Is your issue perhaps more of a discussion or question? Please start a new discussion. 6 | - name: Issue with .NET runtime or core .NET libraries 7 | url: https://github.com/dotnet/runtime/issues/new/choose 8 | about: Please open issues relating to the .NET runtime or core .NET libraries in dotnet/runtime. 9 | - name: Issue with ASP.NET Core 10 | url: https://github.com/dotnet/efcore/issues/new/choose 11 | about: Please open issues relating to ASP.NET Core in dotnet/aspnetcore. 12 | - name: Issue with .NET SDK 13 | url: https://github.com/dotnet/sdk/issues/new/choose 14 | about: Please open issues relating to the .NET SDK itself in dotnet/sdk. 15 | - name: Issue with Roslyn compiler 16 | url: https://github.com/dotnet/roslyn/issues/new/choose 17 | about: Please open issues relating to the Roslyn .NET compiler in dotnet/roslyn. 18 | - name: Issue with other .NET components 19 | url: https://github.com/dotnet/core/blob/master/Documentation/core-repos.md 20 | about: Having issues with other .NET components? Please find the correct location where to open an issue. 21 | -------------------------------------------------------------------------------- /.github/ISSUE_TEMPLATE/feature-request.md: -------------------------------------------------------------------------------- 1 | --- 2 | name: 💡 Feature request 3 | about: Suggest an idea for this project 4 | labels: 'enhancement' 5 | --- 6 | 7 | ### Describe the Problem 8 | 9 | 13 | 14 | ### Describe the Solution 15 | 16 | 17 | 18 | ### Additional Context 19 | 20 | 21 | 22 | -------------------------------------------------------------------------------- /.github/labeler.yml: -------------------------------------------------------------------------------- 1 | # Add 'untriaged' label to any issue that gets opened 2 | untriaged: 3 | - '/.*/' 4 | -------------------------------------------------------------------------------- /.github/linters/.linkspector.yml: -------------------------------------------------------------------------------- 1 | aliveStatusCodes: 2 | - 200 3 | - 406 4 | dirs: 5 | - . 6 | - .github 7 | excludedDirs: 8 | - eng/readme-templates 9 | ignorePatterns: 10 | - pattern: "^https://github.com/dotnet/sdk/tree/main/src/SourceBuild/patches" 11 | - pattern: "^https://dotnetcli.blob.core.windows.net/source-built-artifacts/sdks/" 12 | - pattern: "^https://dotnetcli.blob.core.windows.net/source-built-artifacts/assets/" 13 | - pattern: "^https://www.reddit.com/r/archlinux/comments/cx64r5/the_state_of_net_core_on_arch/" 14 | useGitIgnore: true 15 | -------------------------------------------------------------------------------- /.github/linters/.markdown-lint.yml: -------------------------------------------------------------------------------- 1 | default: true 2 | MD013: # line length 3 | code_block_line_length: 256 4 | heading_line_length: 96 5 | line_length: 96 6 | MD033: false # no inline HTML 7 | -------------------------------------------------------------------------------- /.github/workflows/check-markdown-links.yml: -------------------------------------------------------------------------------- 1 | name: 'Check Markdown Links' 2 | 3 | on: 4 | pull_request: 5 | paths: 6 | - "**/*.md" 7 | 8 | permissions: 9 | pull-requests: read 10 | 11 | jobs: 12 | check-markdown-links: 13 | runs-on: ubuntu-22.04 14 | 15 | steps: 16 | - name: Checkout code 17 | uses: actions/checkout@v4 18 | 19 | - name: Check markdown links 20 | uses: umbrelladocs/action-linkspector@v1 21 | with: 22 | config_file: .github/linters/.linkspector.yml 23 | fail_on_error: true 24 | filter_mode: nofilter 25 | -------------------------------------------------------------------------------- /.github/workflows/labeler.yml: -------------------------------------------------------------------------------- 1 | name: "Issue Labeler" 2 | on: 3 | issues: 4 | types: [opened] 5 | 6 | permissions: 7 | issues: write 8 | contents: read 9 | 10 | jobs: 11 | triage: 12 | runs-on: ubuntu-latest 13 | steps: 14 | - uses: github/issue-labeler@v3.4 15 | with: 16 | configuration-path: .github/labeler.yml 17 | enable-versioned-regex: 0 18 | repo-token: ${{ github.token }} 19 | -------------------------------------------------------------------------------- /.github/workflows/lint-code-base.yml: -------------------------------------------------------------------------------- 1 | name: Lint Code Base 2 | 3 | on: 4 | pull_request: null 5 | 6 | permissions: {} 7 | 8 | jobs: 9 | run-lint: 10 | runs-on: ubuntu-latest 11 | 12 | permissions: 13 | contents: read 14 | packages: read 15 | # To report GitHub Actions status checks 16 | statuses: write 17 | 18 | steps: 19 | - name: Checkout code 20 | uses: actions/checkout@v4 21 | with: 22 | # Full git history is needed to get a proper list of changed files within `super-linter` 23 | fetch-depth: 0 24 | - name: Lint Code Base 25 | uses: github/super-linter@v6 # https://github.com/github/super-linter 26 | env: 27 | DEFAULT_BRANCH: main 28 | FILTER_REGEX_EXCLUDE: eng/common/.*|eng/readme-templates/.* 29 | GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} 30 | VALIDATE_ALL_CODEBASE: false 31 | VALIDATE_MARKDOWN: true 32 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | syntax: glob 2 | 3 | ### VisualStudio ### 4 | 5 | # Tool Runtime Dir 6 | /[Tt]ools/ 7 | .dotnet/ 8 | artifacts 9 | .packages/ 10 | 11 | # User-specific files 12 | *.suo 13 | *.user 14 | *.userosscache 15 | *.sln.docstates 16 | 17 | # Build results 18 | [Dd]ebug/ 19 | [Dd]ebugPublic/ 20 | [Rr]elease/ 21 | [Rr]eleases/ 22 | x64/ 23 | x86/ 24 | build/ 25 | bld/ 26 | [Bb]in/ 27 | [Oo]bj/ 28 | msbuild.log 29 | msbuild.err 30 | msbuild.wrn 31 | *.binlog 32 | 33 | # Cross building rootfs 34 | cross/rootfs/ 35 | 36 | # Visual Studio 2015 37 | .vs/ 38 | 39 | # Visual Studio 2015 Pre-CTP6 40 | *.sln.ide 41 | *.ide/ 42 | 43 | # MSTest test Results 44 | [Tt]est[Rr]esult*/ 45 | [Bb]uild[Ll]og.* 46 | 47 | #NUNIT 48 | *.VisualState.xml 49 | TestResult.xml 50 | 51 | # Build Results of an ATL Project 52 | [Dd]ebugPS/ 53 | [Rr]eleasePS/ 54 | dlldata.c 55 | 56 | *_i.c 57 | *_p.c 58 | *_i.h 59 | *.ilk 60 | *.meta 61 | *.obj 62 | *.pch 63 | *.pdb 64 | *.pgc 65 | *.pgd 66 | *.rsp 67 | *.sbr 68 | *.tlb 69 | *.tli 70 | *.tlh 71 | *.tmp 72 | *.tmp_proj 73 | *.log 74 | *.vspscc 75 | *.vssscc 76 | .builds 77 | *.pidb 78 | *.svclog 79 | *.scc 80 | 81 | # Chutzpah Test files 82 | _Chutzpah* 83 | 84 | # Visual C++ cache files 85 | ipch/ 86 | *.aps 87 | *.ncb 88 | *.opendb 89 | *.opensdf 90 | *.sdf 91 | *.cachefile 92 | *.VC.db 93 | 94 | # Visual Studio profiler 95 | *.psess 96 | *.vsp 97 | *.vspx 98 | 99 | # TFS 2012 Local Workspace 100 | $tf/ 101 | 102 | # Guidance Automation Toolkit 103 | *.gpState 104 | 105 | # ReSharper is a .NET coding add-in 106 | _ReSharper*/ 107 | *.[Rr]e[Ss]harper 108 | *.DotSettings.user 109 | 110 | # JustCode is a .NET coding addin-in 111 | .JustCode 112 | 113 | # TeamCity is a build add-in 114 | _TeamCity* 115 | 116 | # DotCover is a Code Coverage Tool 117 | *.dotCover 118 | 119 | # NCrunch 120 | _NCrunch_* 121 | .*crunch*.local.xml 122 | 123 | # MightyMoose 124 | *.mm.* 125 | AutoTest.Net/ 126 | 127 | # Web workbench (sass) 128 | .sass-cache/ 129 | 130 | # Installshield output folder 131 | [Ee]xpress/ 132 | 133 | # DocProject is a documentation generator add-in 134 | DocProject/buildhelp/ 135 | DocProject/Help/*.HxT 136 | DocProject/Help/*.HxC 137 | DocProject/Help/*.hhc 138 | DocProject/Help/*.hhk 139 | DocProject/Help/*.hhp 140 | DocProject/Help/Html2 141 | DocProject/Help/html 142 | 143 | # Click-Once directory 144 | publish/ 145 | 146 | # Publish Web Output 147 | *.[Pp]ublish.xml 148 | *.azurePubxml 149 | *.pubxml 150 | *.publishproj 151 | 152 | # NuGet Packages 153 | *.nuget.props 154 | *.nuget.targets 155 | *.nupkg 156 | **/packages/* 157 | 158 | # NuGet package restore lockfiles 159 | project.lock.json 160 | 161 | # Windows Azure Build Output 162 | csx/ 163 | *.build.csdef 164 | 165 | # Windows Store app package directory 166 | AppPackages/ 167 | 168 | # Others 169 | *.Cache 170 | ClientBin/ 171 | [Ss]tyle[Cc]op.* 172 | ~$* 173 | *.dbmdl 174 | *.dbproj.schemaview 175 | *.pfx 176 | *.publishsettings 177 | node_modules/ 178 | *.metaproj 179 | *.metaproj.tmp 180 | 181 | # RIA/Silverlight projects 182 | Generated_Code/ 183 | 184 | # Backup & report files from converting an old project file 185 | # to a newer Visual Studio version. Backup files are not needed, 186 | # because we have git ;-) 187 | _UpgradeReport_Files/ 188 | Backup*/ 189 | UpgradeLog*.XML 190 | UpgradeLog*.htm 191 | 192 | # SQL Server files 193 | *.mdf 194 | *.ldf 195 | 196 | # Business Intelligence projects 197 | *.rdl.data 198 | *.bim.layout 199 | *.bim_*.settings 200 | 201 | # Microsoft Fakes 202 | FakesAssemblies/ 203 | 204 | ### MonoDevelop ### 205 | 206 | *.pidb 207 | *.userprefs 208 | 209 | ### Windows ### 210 | 211 | # Windows image file caches 212 | Thumbs.db 213 | ehthumbs.db 214 | 215 | # Folder config file 216 | Desktop.ini 217 | 218 | # Recycle Bin used on file shares 219 | $RECYCLE.BIN/ 220 | 221 | # Windows Installer files 222 | *.cab 223 | *.msi 224 | *.msm 225 | *.msp 226 | 227 | # Windows shortcuts 228 | *.lnk 229 | 230 | ### Linux ### 231 | 232 | *~ 233 | 234 | # KDE directory preferences 235 | .directory 236 | 237 | ### OSX ### 238 | 239 | .DS_Store 240 | .AppleDouble 241 | .LSOverride 242 | 243 | # Icon must end with two \r 244 | Icon 245 | 246 | # Thumbnails 247 | ._* 248 | 249 | # Files that might appear on external disk 250 | .Spotlight-V100 251 | .Trashes 252 | 253 | # Directories potentially created on remote AFP share 254 | .AppleDB 255 | .AppleDesktop 256 | Network Trash Folder 257 | Temporary Items 258 | .apdisk 259 | 260 | # vim temporary files 261 | [._]*.s[a-w][a-z] 262 | [._]s[a-w][a-z] 263 | *.un~ 264 | Session.vim 265 | .netrwhist 266 | *~ 267 | 268 | # Visual Studio Code 269 | .vscode/ 270 | 271 | # Private test configuration and binaries. 272 | config.ps1 273 | **/IISApplications 274 | 275 | ### source-build ### 276 | prebuilt/ 277 | msbuild.binlog 278 | testing-smoke/ 279 | 280 | # Exists temporarily during build. Interferes with VersionTools' 281 | # UpdateDependenciesAndSubmitPullRequest: if no upgrades are performed, the 282 | # sanity check fails when "git status" shows this file. 283 | msbuild.ProjectImports.zip 284 | Build_*.ProjectImports.zip 285 | -------------------------------------------------------------------------------- /CODE-OF-CONDUCT.md: -------------------------------------------------------------------------------- 1 | # Code of Conduct 2 | 3 | This project has adopted the code of conduct defined by the Contributor Covenant 4 | to clarify expected behavior in our community. 5 | 6 | For more information, see the [.NET Foundation Code of 7 | Conduct](https://dotnetfoundation.org/code-of-conduct). 8 | -------------------------------------------------------------------------------- /CODEOWNERS: -------------------------------------------------------------------------------- 1 | # Users referenced in this file will automatically be requested as reviewers for PRs that modify the given paths. 2 | # See https://help.github.com/articles/about-code-owners/ 3 | 4 | * @MichaelSimons @mthalman 5 | -------------------------------------------------------------------------------- /Documentation/VMR-re-bootstrapping.md: -------------------------------------------------------------------------------- 1 | # How to re-bootstrap the toolset used to build the VMR 2 | 3 | .NET utilizes itself to build. Therefore, in order to build .NET from source, 4 | you first need to acquire or build a bootstrapping .NET SDK and other tooling 5 | such as [Arcade](https://github.com/dotnet/arcade). Re-bootstrapping is the term 6 | used to describe when the bootstrapped toolset need to be updated. This document 7 | describes the steps to re-bootstrap the VMR. 8 | 9 | ## When is it appropriate to re-bootstrap? 10 | 11 | As part of the release process, the toolset is updated (e.g. PRs are created via 12 | the release automation). Outside of a release, re-bootstrapping is only 13 | permitted during preview releases. It is not allowed during RC, GA, or servicing 14 | releases. The reason it is not allowed during non-preview releases is because of 15 | the negative impact it has on Linux distro maintainers who source build .NET. It 16 | is often a long and time consuming process for them to re-bootstrap. It is 17 | likely to cause significant delays in the release/availability of .NET within 18 | the distros that are source built. 19 | 20 | ## Why is re-bootstrap necessary? 21 | 22 | Re-bootstrapping is necessary when .NET takes a dependency on new functionality 23 | added within the bootstrap toolset. For example suppose a new compiler feature 24 | is added. In order for a repo to take a dependency on the new feature, a 25 | re-bootstrap would be necessary. The implication of this, and the restrictions 26 | of when re-bootstrapping is allowed, means that repos should, in general, wait 27 | to take a dependency on a new toolset feature until after that feature has been 28 | released. 29 | 30 | ## Steps to re-bootstrap 31 | 32 | ### Automated 33 | 34 | > [!IMPORTANT] 35 | > The re-bootstrap pipeline uploads the artifacts to the official blob storage, 36 | > so do not use this pipeline for testing of any kind. To test stage 2 failures, 37 | > please refer to [this 38 | > documentation](bootstrapping-guidelines.md#building-on-a-supported-platform-using-rid-known-to-net). 39 | 40 | You can re-bootstrap the VMR using [this 41 | pipeline](https://dev.azure.com/dnceng/internal/_build?definitionId=1371). The 42 | pipeline will upload the artifacts & open the corresponding re-bootstrap PR. 43 | 44 | ### Manual 45 | 46 | In case the automated re-bootstrapping pipeline is unavailable, you can manually 47 | re-bootstrap the VMR: 48 | 49 | 1. Update previous source-build artifacts 50 | 1. Find a 51 | [dotnet-source-build](https://dev.azure.com/dnceng/internal/_build?definitionId=1219) 52 | run with the desired changes. 53 | 1. If a rebootstrap is needed quickly and it is not feasibly to wait for 54 | a 55 | [dotnet-source-build](https://dev.azure.com/dnceng/internal/_build?definitionId=1219) 56 | run, you can also use the artifacts from a 57 | [dotnet-source-build-lite](https://dev.azure.com/dnceng/internal/_build?definitionId=1299) 58 | run. 59 | 1. Retrieve the built SDKs and private source-built artifacts archives, from 60 | the following legs: 61 | 1. Alpine\_Online_MsftSdk_x64 62 | 1. CentOSStream\_Online_MsftSdk_x64 63 | 1. Upload the SDKs to the [source build sdk blob 64 | storage](https://dotnetcli.blob.core.windows.net/source-built-artifacts/sdks/) 65 | 1. Upload the private source-built artifacts archives to the [source build 66 | assets blob 67 | storage](https://dotnetcli.blob.core.windows.net/source-built-artifacts/assets/) 68 | 1. Update .NET SDK 69 | 1. Find the 70 | [dotnet-sdk-official-ci](https://dev.azure.com/dnceng/internal/_build?definitionId=140) 71 | build that best matches the dotnet-source-build. The following is the 72 | suggested order of precedence for finding the best match. 73 | 1. A build from the same commit. 74 | 1. From the 75 | [dotnet-source-build](https://dev.azure.com/dnceng/internal/_build?definitionId=1219), 76 | look at the build's installer tag. 77 | 1. From a VMR commit, you can find the corresponding installer 78 | commit by looking at the 79 | [source-manifest.json](https://github.com/dotnet/dotnet/blob/main/src/source-manifest.json). 80 | 1. The next passing build after the same commit. 81 | 1. In the odd case where the are no passing builds after the commit, you 82 | can try using an earlier passing build. 83 | 1. Retrieve the built SDK version from the build. 84 | 1. Update the dotnet version in the 85 | [global.json](https://github.com/dotnet/sdk/blob/main/src/SourceBuild/content/global.json). 86 | 1. Update arcade 87 | 1. Lookup the arcade commit and version. From a VMR commit, you can find the 88 | corresponding arcade commit/version by looking at the 89 | [source-manifest.json](https://github.com/dotnet/dotnet/blob/main/src/source-manifest.json). 90 | 1. Update the arcade SDK version in the 91 | [global.json](https://github.com/dotnet/sdk/blob/main/src/SourceBuild/content/global.json). 92 | 1. Update the arcade dependency commit and version in the 93 | [Version.Details.xml](https://github.com/dotnet/sdk/blob/main/src/SourceBuild/content/eng/Version.Details.xml). 94 | 1. Update private source-built SDK and artifacts versions 95 | 1. Update `PrivateSourceBuiltSdkVersion` and 96 | `PrivateSourceBuiltArtifactsVersion` in the 97 | [Versions.props](https://github.com/dotnet/sdk/blob/main/src/SourceBuild/content/eng/Versions.props). 98 | 99 | [Tracking issue for automating this 100 | process.](https://github.com/dotnet/source-build/issues/4246) 101 | -------------------------------------------------------------------------------- /Documentation/boostrap-new-os.md: -------------------------------------------------------------------------------- 1 | # Adding support for a new OS 2 | 3 | ## Bootstrap CLI 4 | 5 | When adding support for a new OS, both the native (C++) and managed (C#) code 6 | components in the coreclr, corefx and core-setup repos need to be compiled. 7 | Making the native components build on a new OS is a quite straightforward 8 | process and the tools like CMake, Clang C/C++ compiler, python and awk that are 9 | required for the build are available for almost all platforms. But for the 10 | managed code compilation, we have a chicken and egg problem. A .NET CLI 11 | toolchain is needed to build managed components in the coreclr, corefx and 12 | core-setup repos, but it is not available for the target platform yet. The way 13 | to solve this problem is to create a bootstrap CLI by taking an existing CLI as 14 | a "seed" and replacing native components in it by the native components that we 15 | build for the new platform. The seed CLI we use has to be for the same processor 16 | architecture as our new platform or we have to use a seed cli that we build from 17 | sources on another platform that is already supported and opt for not 18 | crossgenning the managed components. 19 | 20 | ## When is bootstrapping necessary? 21 | 22 | Bootstrapping is necessary when: 23 | 24 | - Your available version of the SDK cannot build your desired version of the 25 | SDK. 26 | - You have no version of the SDK available for your platform. 27 | - You are building the SDK for a non-x64 platform. 28 | 29 | In practice, most maintainers will bootstrap once every major version (e.g. 30 | 6.0.100, 8.0.100, etc). If you bootstrap these initial versions once you will 31 | generally be able to build 6.0.101 with 6.0.100, 6.0.102 with 6.0.101, 8.0.101 32 | with 8.0.100, and so on. The source-build team attempts to avoid requiring 33 | bootstrapping more than once per major version but sometimes this is unavoidable, 34 | often when we move from one feature band to another, or if a major language 35 | feature is added, or if there are issues discovered with a given version of the 36 | SDK after release. These types of situations will be announced on the 37 | source-build [discussions 38 | page](https://github.com/dotnet/source-build/discussions). 39 | 40 | ### Choosing the seed CLI 41 | 42 | While the bootstrap CLI can be built for any version of the seed CLI, it is 43 | important to pick a version of CLI that can be used to build current corefx, 44 | coreclr and core-setup repos. From time to time, a change in the CLI causes it 45 | to not to be usable for compiling one of the repos without fixes in the msbuild 46 | project files. So to stay on the safe side, the best practice is to pick a 47 | version used by one of the three repos. This version can be found in the 48 | `DotnetCLIVersion.txt` file in the root of each repo. 49 | 50 | ### Getting the seed CLI 51 | 52 | After choosing the seed CLI version as described in the previous paragraph, it 53 | can be used to construct an URL to download the .tar.gz file with the seed CLI 54 | itself. The way to construct the URL can be found in the `init-tools.sh` file in 55 | the root of each repo. Currently, it is constructed as follows: 56 | 57 | ```bash 58 | https://builds.dotnet.microsoft.com/dotnet/Sdk/${__DOTNET_TOOLS_VERSION}/dotnet-sdk-${__DOTNET_TOOLS_VERSION}-${__PKG_RID}-${__PKG_ARCH}.tar.gz 59 | ``` 60 | 61 | The `${__DOTNET_TOOLS_VERSION}` is replaced by the seed CLI version, the 62 | `${__PKG_RID}` by the RID of the current platform and `${__PKG_ARCH}` by the 63 | architecture of the current platform. So for tools version `2.0.0` on linux 64 | distro with x64 architecture where portable dotnet core can be used, the URL is 65 | `https://builds.dotnet.microsoft.com/dotnet/Sdk/2.0.0/dotnet-sdk-2.0.0-linux-x64.tar.gz` 66 | To download it, `wget` or `curl` tools can be used. Once the file is downloaded, 67 | create a new folder and untar the file into it. 68 | So e.g. for the file mentioned above, use: 69 | 70 | ```bash 71 | tar -xf dotnet-sdk-2.0.0-linux-x64.tar.gz 72 | ``` 73 | 74 | ### Choosing RID for the new OS 75 | 76 | The new RID represents your target OS. It is used by developers to target that 77 | OS. The RID format is `.`. The name is lower case and should 78 | match the target OS. The version is optional and should be used if it is 79 | expected that in future versions the libraries that .NET Core depends on will 80 | not be binary compatible with the first supported version. So for example for 81 | FreeBSD 11, the RID would likely be `freebsd.11`. For OS that has 82 | `/etc/os-release` file, the RID needs to match the `$ID.$VERSION_ID` extracted 83 | from that file. 84 | 85 | ### Prerequisites 86 | 87 | The following libraries and tools need to be installed in order to build the 88 | bootstrap CLI. The precise names of the packages are OS specific. 89 | 90 | #### Tools 91 | 92 | - Clang version 3.5 or higher, 3.9 recommended 93 | - CMake version 2.8.12 or higher 94 | - Python version 2.7 95 | - AWK 96 | - SED 97 | 98 | #### Development libraries 99 | 100 | - lttng-ust 101 | - openssl or libressl 102 | - krb5 103 | - curl 104 | - liblldb 105 | - icu 106 | 107 | For versions earlier than .NET Core 2.1, following dependencies are also 108 | required: 109 | 110 | - libunwind 111 | - libuuid 112 | 113 | ### Building the bootstrap CLI 114 | 115 | There is a bash script file that automatizes most of the process of building the 116 | bootstrap CLI end to end. The bash script is located at 117 | `dotnet/source-build/scripts/bootstrap/buildbootstrapcli.sh`. It first creates a 118 | folder named by the new target RID, clones the coreclr, corefx and core-setup 119 | repos into it and checks out the same commit of each of the repos as the one 120 | that was used to build the seed CLI. This first step is skipped if the coreclr, 121 | corefx and core-setup folders already exist. This is important so that the 122 | sources can be modified to fix possible build issues and to target the new RID. 123 | 124 | The script needs to be passed several arguments. The target architecture, the 125 | build configuration, the target OS, the new RID and path to the folder with the 126 | untared seed CLI. There is also an optional option to specify the version of the 127 | Clang compiler to use to compile the native code. There is also an option to 128 | pass in a System.Private.CoreLib.dll built elsewhere. This is useful if you are 129 | building debug configuration, since the seed CLI that comes from Azure is built 130 | for release configuration and the release version of System.Private.CoreLib.dll 131 | is not compatible with debug version of libcoreclr.so. 132 | 133 | The build configuration (debug or release) specified here *must* match with the 134 | configuration of the seed CLI. Otherwise everything will build fine but you will 135 | get a cryptic error as soon as coreclr tries to load a dll. 136 | 137 | Here is the summary of the options that you get from running the script with 138 | `--help` option: 139 | 140 | ```text 141 | Usage: buildbootstrapcli.sh [BuildType] -rid -seedcli [-os ] [-clang ] [-corelib ] 142 | 143 | Options: 144 | BuildType Type of build (-debug, -release), default: -release 145 | -clang Override of the version of clang compiler to use 146 | -corelib Path to System.Private.CoreLib.dll, default: use the System.Private.CoreLib.dll from the seed CLI 147 | -os Operating system (used for corefx build), default: Linux 148 | -rid Runtime identifier including the architecture part (e.g. rhel.6-x64) 149 | -seedcli Seed CLI used to generate the target CLI 150 | -outputpath Optional output directory to contain the generated cli and cloned repos, default: 151 | ``` 152 | 153 | So, for example, when we were creating bootstrap CLI for RHEL / CentOS 6, the 154 | command was: 155 | 156 | ```bash 157 | ./buildbootstrapcli.sh -rid rhel.6-x64 -os Linux -seedcli ~/seed-cli 158 | ``` 159 | 160 | After running the script, check the console output. If the last line printed is 161 | `**** Bootstrap CLI was successfully built ****`, then everything went fine and 162 | the bootstrap CLI is ready. You can find it in the 163 | `-/dotnetcli` subfolder. So for the example command above, it 164 | would be `rhel.6-x64/dotnetcli`. If there were build errors, they need to be 165 | looked into and fixed. After that run the `buildbootstrapcli.sh` with the same 166 | arguments again. Repeat until everything builds. 167 | 168 | ### Testing the bootstrap CLI 169 | 170 | The easiest way to test the bootstrap CLI that was just created is to create, 171 | build and run a "hello world" console application. Create a new folder for the 172 | application and then run `/your/path/to/bootstrap/dotnetcli/dotnet new console` 173 | followed by `/your/path/to/bootstrap/dotnetcli/dotnet run`. If both of these 174 | succeed and the second prints "Hello world!", then everything went ok and you 175 | have a working bootstrap CLI. If there are issues in either of these steps, they 176 | need to be debugged and the culprits figured out. 177 | 178 | ### Troubleshooting 179 | 180 | TODO: describe how to do debug build and replace System.Private.CoreLib.dll 181 | 182 | #### Using bootstrap CLI on platforms with different OS or architecture 183 | 184 | If the bootstrap CLI was created for OS other than the one the seed cli supports 185 | (e.g. FreeBSD) or the target architecture of the target OS differs from the 186 | architecture of the bootstrap cli (e.g. ARM64), the managed assemblies in the 187 | bootstrap CLI cannot be loaded, since they contain native code for the target OS 188 | and architecture. Fortunately, they also contain the original IL code, so they 189 | can be re-crossgened for the target OS and architecture. Or it is possible to 190 | instruct the CoreCLR runtime to ignore the native code and use the IL by setting 191 | environment variables `COMPlus_ZapDisable=1` and `COMPlus_ReadyToRun=0`. That 192 | can be useful during the bringup. But ultimately, re-crossgening should be made 193 | to improve startup performance. 194 | 195 | ## Use the bootstrap CLI to build source-build 196 | 197 | Now that you have an SDK that works on the new OS, you can build all of the .NET 198 | Core source code. To do this, first tar up the boostrap CLI: 199 | 200 | ```bash 201 | tar -czf ~/dotnet.tar --directory /your/path/to/bootstrap/dotnetcli "." 202 | ``` 203 | 204 | Next, the ILASM tool that you built locally needs to be copied out so it can be 205 | used. 206 | 207 | ```bash 208 | mkdir ~/ilasm 209 | cp dotnet/source-build/scripts/bootstrap/RID/coreclr/bin/Product/RID.Release/* ~/ilasm 210 | cp /your/path/to/bootstrap/dotnetcli/shared/Microsoft.NETCore.App/VERSION/System.Private.CoreLib.dll ~/ilasm 211 | ``` 212 | 213 | TODO: make this next step better - possibly by adding a new build parameter that 214 | takes the path to ILASM. Now, we can initialize our build tools with the 215 | bootstrap CLI and the ILASM tool. 216 | 217 | ```bash 218 | DotNetBootstrapCliTarPath=~/dotnet.tar SOURCE_BUILD_SKIP_SUBMODULE_CHECK=1 ./init-tools.sh 219 | # Copy ILASM into the Tools directory 220 | cp ~/ilasm/* dotnet/source-build/Tools/ilasm/ 221 | ``` 222 | 223 | Now you can build the whole product using the toolset you just bootstrapped: 224 | 225 | ```bash 226 | SOURCE_BUILD_SKIP_SUBMODULE_CHECK=1 ./build.sh 227 | ``` 228 | -------------------------------------------------------------------------------- /Documentation/bootstrapping-guidelines.md: -------------------------------------------------------------------------------- 1 | # Bootstrapping Guidelines to Build .NET from Source 2 | 3 | .NET utilizes itself to build therefore in order to build .NET from source, you 4 | first need to acquire or build a bootstrapping .NET SDK. This document provides 5 | guidance around acquiring and building this bootstrapping .NET SDK. 6 | 7 | The version of the SDK used to source build .NET is referred to as "N-1" (e.g. 8 | 8.0.100). The version of the SDK produced by source build is referred to as "N" 9 | (e.g. 8.0.101). The previous SDK (e.g. N-1) supplies the tools required to 10 | build. 11 | 12 | For new major versions or new platforms, you need to acquire or build the 13 | bootstrapping SDK as you cannot use a previous source-built SDK. This is to say 14 | you cannot use a 8.0 version of the SDK to build a 9.0 SDK. 15 | 16 | Bootstrapping typically requires an exception in the distro packaging guidelines 17 | (e.g. [Fedora Bootstrapping 18 | Guidelines](https://docs.fedoraproject.org/en-US/packaging-guidelines/#bootstrapping)). 19 | 20 | Refer to the [build 21 | instructions](https://github.com/dotnet/installer/blob/main/README.md#build-net-from-source-source-build) 22 | to review how to build the .NET SDK from source. 23 | 24 | ## Scenarios 25 | 26 | There are three major scenarios for bootstrapping: 27 | 28 | 1. [Building on a supported platform (Using RID known to 29 | .NET)](#building-on-a-supported-platform-using-rid-known-to-net) 30 | 1. [Building for New OS (Using a RID unknown to 31 | .NET)](#building-for-new-os-using-a-rid-unknown-to-net) 32 | 1. [Building for New Architecture (Using a RID unknown to 33 | .NET)](#building-for-new-architecture-using-a-rid-unknown-to-net) 34 | 35 | ## Building on a supported platform (Using RID known to .NET) 36 | 37 | To find out if your platform is supported you must first determine its 38 | [RID](https://learn.microsoft.com/en-us/dotnet/core/rid-catalog). You can then 39 | check if it's supported by looking at the RID graph in the 40 | [runtime.json](https://github.com/dotnet/runtime/blob/main/src/libraries/Microsoft.NETCore.Platforms/src/runtime.json). 41 | 42 | Building .NET for the first time is a two stage process: 43 | 44 | **Stage 1:** Build bootstrapping .NET SDK for the targeted platform. 45 | 46 | 1. Download a platform-native portable Microsoft-built version of the dotnet SDK 47 | for bootstrapping as well as the previously-source-built package archive. 48 | 49 | ``` bash 50 | ./prep-source-build.sh 51 | ``` 52 | 53 | 1. Build the source built .NET SDK. 54 | 55 | ``` bash 56 | ./build.sh --source-only 57 | ``` 58 | 59 | **Stage 2:** Use the source-built .NET SDK and source-built artifacts created in 60 | stage 1 to build .NET SDK from source. 61 | 62 | 1. Extract your freshly-built stage 1 SDK to a convenient location. 63 | 64 | ``` bash 65 | tar -ozxf //artifacts/assets/Release/dotnet-sdk---tar.gz -C 66 | ``` 67 | 68 | 1. Extract your freshly-built stage 1 source-built artifacts to a convenient 69 | location. 70 | 71 | ```bash 72 | tar -ozxf //artifacts/assets/Release/Private.SourceBuilt.Artifacts.--.tar.gz -C 73 | ``` 74 | 75 | 1. Prep the build. 76 | 77 | ```bash 78 | ./prep-source-build.sh --no-sdk --no-artifacts --no-bootstrap --with-sdk --with-packages 79 | ``` 80 | 81 | 1. Build the source built .NET SDK. 82 | 83 | ``` bash 84 | ./build.sh --source-only --with-sdk --with-packages 85 | ``` 86 | 87 | ## Building for New OS (Using a RID unknown to .NET) 88 | 89 | Building for an OS that Microsoft does not currently build the SDK for is 90 | possible but requires more work. If [Microsoft 91 | produces](https://dotnet.microsoft.com/en-us/download/dotnet) a portable SDK for 92 | your platform (e.g. amd64 and arm64), you can follow the two-step process below. 93 | 94 | **RIDs:** 95 | 96 | The RID graph or runtime fallback graph is a list of RIDs that are compatible 97 | with each other. You can see the list of supported RIDs and the RID graph in the 98 | [runtime.json](https://github.com/dotnet/runtime/blob/main/src/libraries/Microsoft.NETCore.Platforms/src/runtime.json). 99 | Learn more about RID catalog 100 | [here](https://learn.microsoft.com/en-us/dotnet/core/rid-catalog#linux-rids). 101 | 102 | If a compatible RID is found, you can use a compatible supported OS as host to 103 | build. Choose a host with same processor architecture as that of the new 104 | targeted platform. If you choose this option, the RID of the resulting SDK will 105 | be that of the host. If this is acceptable follow the instructions in [Building 106 | on a supported platform (Using RID known to 107 | .NET)](#building-on-a-supported-platform-using-rid-known-to-net) using a 108 | compatible host OS. 109 | 110 | If no compatible RID is found or you want a RID specific SDK use the folloring 111 | the steps (works for .NET 6, but requires validation for .NET 7): 112 | 113 | **Stage 0:** 114 | 115 | 1. Get Microsoft portable SDK. 116 | 1. Update the RID graph (runtime.json) in the Microsoft-built portable SDK with 117 | the same changes you will make below to add your new RID to the RID graph. 118 | This should include a fallback to the portable RID (linux-x64 or similar). 119 | 120 | **Stage 1:** 121 | 122 | 1. Update the RID graph in source with the same changes made in Stage 0. For an 123 | example, see . 124 | 1. Build with Stage 0 SDK using `--with-sdk` with your modified portable SDK. 125 | See the Stage 1 instructions in [Building on a supported platform (Using RID 126 | known to .NET)](#building-on-a-supported-platform-using-rid-known-to-net). 127 | 128 | **Stage 2:** 129 | 130 | 1. Now you have a RID-specific SDK that knows about your new RID, build with 131 | Stage 1 SDK as done in [Building on a supported platform (Using RID known to 132 | .NET)](#building-on-a-supported-platform-using-rid-known-to-net). 133 | 134 | ## Building for New Architecture (Using a RID unknown to .NET) 135 | 136 | Building for an architecture that Microsoft does not currently build the SDK for 137 | is possible but requires more work. 138 | 139 | **RIDs:** 140 | 141 | The RID graph or runtime fallback graph is a list of RIDs that are compatible 142 | with each other. You can see the list of supported RIDs and the RID graph in the 143 | [runtime.json](https://github.com/dotnet/runtime/blob/main/src/libraries/Microsoft.NETCore.Platforms/src/runtime.json). 144 | Learn more about RID catalog in 145 | [here](https://learn.microsoft.com/en-us/dotnet/core/rid-catalog#linux-rids). 146 | 147 | You will need to update the RID graph to include your new platform and runtime 148 | IDs. See or 149 | for examples. 150 | 151 | Building for unsupported architectures require cross-compilaton on the supported 152 | platform. Determine the compatible host to build which provides 153 | cross-compilation toolchain. [IBM has 154 | published](https://community.ibm.com/community/user/powerdeveloper/blogs/sapana-khemkar/2023/01/13/cross-build-dotnet7-on-x86-ibm-power?CommunityKey=8cc2a1f0-6307-48cb-9178-ace50920244e) 155 | a detailed description of how they successfully built .NET 7 for IBM Power. 156 | 157 | While this is a more complicated scenario that may differ from platform to 158 | platform, the steps will be roughly: 159 | 160 | **Stage 0:** 161 | 162 | 1. Cross compile an SDK (using prebuilts) on x64 for target platform (this 163 | process may be quite long and involved and include setting up a rootfs for 164 | your architecture). 165 | 2. Cross compile the runtime repo (on x64 for target platform, generally done as 166 | part of previous step) and save the nuget packages, use these to augment the 167 | Microsoft-built previously-source-built archive. 168 | 169 | **Stage 1:** 170 | 171 | 1. Use the cross-compiled SDK and augmented previously-source-built-archive to 172 | build a stage 1 SDK. See the Stage 1 instructions in [Building on a 173 | supported platform (Using RID known to 174 | .NET)](#building-on-a-supported-platform-using-rid-known-to-net). 175 | 176 | **Stage 2:** 177 | 178 | 1. Use your stage 1 SDK to build a stage 2 SDK, pointing it to the SDK and 179 | previously-source-built archives from stage 1. See the Stage 2 instructions 180 | in [Building on a supported platform (Using RID known to 181 | .NET)](#building-on-a-supported-platform-using-rid-known-to-net). 182 | 183 | ## Building a Servicing Release of .NET 184 | 185 | Building a subsequent or servicing version of .NET requires that you have source 186 | built the previous version of .NET available as descibed in one of the [building 187 | scenarios](#scenarios). Once you have a previous verion of the .NET SDK 188 | available, all you have to do is run the following build command. 189 | 190 | ``` bash 191 | ./build.sh --source-only --with-sdk --with-packages 192 | ``` 193 | -------------------------------------------------------------------------------- /Documentation/ci-platform-coverage-guidelines.md: -------------------------------------------------------------------------------- 1 | # Guidelines for Platforms Tested in CI 2 | 3 | This document contains the guidelines for which platforms (OS and architectures) 4 | to test in the source build CI. 5 | 6 | ## Distro Families 7 | 8 | 1. Prefer testing base distros of families over derivatives. 9 | 1. Prioritize testing distros that source build .NET. 10 | 1. Use CentOS Stream instead of Red Hat because it is the free alternative. 11 | 1. Include a permutation of distros for the supported C standard library 12 | implementations (e.g. glibc and musl). 13 | 1. For each C standard library implementation, include at least one distro that 14 | uses the minimum supported version of this library. 15 | 1. Only test [distros that are officially supported by 16 | .NET](https://github.com/dotnet/core/blob/main/os-lifecycle-policy.md#net-supported-os-policy). 17 | Community supported distros will not be covered. 18 | 19 | ## Distro Versions 20 | 21 | 1. Prefer testing latest LTS version. If LTS is unsupported, use the latest 22 | version instead. 23 | 1. Drop distro versions that will be or are nearing EOL on the .NET release day. 24 | 1. Stop updating the test matrix in the last 6 months of support for a .NET 25 | version as new distro versions are released. 26 | 27 | ## Architectures 28 | 29 | 1. Amd64 30 | 1. Arm64 31 | 32 | ## Permutations 33 | 34 | It is not the intent to test every permutation of distro family, distro version, 35 | and architecture. Rather, smart decisions should be made to best utilize 36 | resources. 37 | 38 | The following distro versions will be included in the [CI 39 | matrix](https://github.com/dotnet/sdk/blob/main/eng/pipelines/templates/stages/vmr-build.yml): 40 | 41 | 1. CentOS Stream - Latest version (amd64) 42 | 1. Fedora - Latest version (amd64) 43 | 1. Ubuntu - Latest LTS version at the time of the release (amd64/arm64) 44 | 1. Alpine - Latest version (amd64) 45 | 1. AlmaLinux - Oldest version (targets lowest glibc version) (amd64) 46 | 47 | When updating the distro versions in the CI matrix as new versions are released 48 | and older versions reach EOL: 49 | 50 | 1. Update `main` to the newer version one to two months prior to the GA/EOL date. 51 | This is done to flush out any issues and to avoid destabilizing the servicing 52 | branches. 53 | 1. At the GA/EOL date, update the servicing branches. 54 | -------------------------------------------------------------------------------- /Documentation/eliminating-pre-builts.md: -------------------------------------------------------------------------------- 1 | # Eliminating pre-builts in .NET repositories 2 | 3 | This is a detailed guide on how to eliminate pre-builts in a source-buildable 4 | repository. It is primarily intended for developers contributing to the `dotnet` 5 | organization. 6 | 7 | ## Table of content 8 | 9 | - [What is a Prebuilt](#what-is-a-prebuilt) 10 | - [Elimitating pre-builts](#eliminating-pre-builts) 11 | - [Allowed exceptions](#allowed-exceptions) 12 | - [Contacts](#contacts) 13 | 14 | ## What is a Prebuilt 15 | 16 | _Source-build_ is a process of building a given product on a single machine from 17 | source with no internet access. 18 | 19 | By definition, _pre-builts_ are dependencies that are not built from source, 20 | such as reference packages, nuget packages and built tools. _Build from source_ 21 | points to any package produced during the _current source-build_ with the 22 | exception of dependencies that are picked up from the host distro such as 23 | `cmake`. In layman terms, this means that packages from `nuget.org`, Microsoft 24 | builds or other non-source-built binaries cannot be used for source-building a 25 | given repository. 26 | 27 | When onboarding a repository to source-build or adding a new dependency to a 28 | source-buildable one, the contributor runs the risk of introducing a new 29 | pre-built to the product. To protect against this and catch any new pre-builts 30 | as soon as possible, Arcade source-build infrastructure provides _pre-built 31 | detection_ - MSBuild logic responsible for veryfing that no used dependency is a 32 | pre-built. In case one is discovered (for example, during a PR pipeline), the 33 | build will fail with an appropriate message somewhat similar to the following: 34 | 35 | ```text 36 | 3 new packages used not in baseline! See report at ./artifacts/source-build/self/prebuilt-report/baseline-comparison.xml for more information. Package IDs are: 37 | Microsoft.Bcl.AsyncInterfaces.8.0.0-alpha.1.22557.11 38 | Microsoft.Build.16.7.0 39 | Microsoft.Build.Framework.14.3.0 40 | ``` 41 | 42 | Pre-built detection identifies the source of dependencies used to build the 43 | repository. These dependencies include not only direct dependencies, but also 44 | build tooling as well as dangling dependencies - packages downloaded / used by 45 | tooling during the build process and not referenced by the project itself. 46 | Dependencies retrieved from external sources that are not explicitly excluded 47 | from pre-built detection will be flagged as pre-builts. 48 | 49 | ## Eliminating pre-builts 50 | 51 | When altering the dependecy tree of a repository, specifically adding or 52 | updating dependencies, there is a posibility that a new pre-built is introduced, 53 | failing the build and blocking any merge. This can be resolved by identifying 54 | what exactly is the pre-built and following the approriate steps listed below. 55 | 56 | To check if new pre-builts were introduce, the repository needs to be 57 | source-built first. This can be done through the following command: 58 | 59 | ```sh 60 | ./build.sh --sb 61 | ``` 62 | 63 | If a new unhandled pre-built is found, the build will fail with a detailed 64 | exception pointing to the exact package / version of the dependency that caused 65 | the failure. Additional information can also be found in the generated pre-built 66 | detection reports, located in the 67 | `./artifacts/source-build/self/prebuild-report` directory, specifically in the 68 | `./prebuild-usage.xml` file. The information in question can, for example, be 69 | the path to the project that is referencing the dependency. 70 | 71 | With this information retrieved, the [adding a new source-build 72 | dependency](https://github.com/dotnet/source-build/blob/main/Documentation/sourcebuild-in-repos/new-dependencies.md#adding-dependencies) 73 | documentation should be referred to as the main guide for resolving the 74 | pre-built. 75 | 76 | ### Pre-built through a transitive dependency 77 | 78 | During a project dependency updated, a new pre-built might be introduced by a 79 | new or updated transitive dependency. While the Arcade tooling will highlight 80 | the name and version of the pre-built in the build exception as well as the 81 | project that restored the dependency in question, it will not point out where 82 | exactly in the dependency tree it is. This means that it's up to the developer 83 | to identify what type of dependency they are dealing with (direct or transitive) 84 | and choose the correct way of handling the issue. In case of a transitive 85 | dependency, it might be hard to identify the relationship that is bringing in 86 | the pre-built into the project, especially if the developer has limited 87 | knowledge of the project or code-base in general. 88 | 89 | Arcade source-build infrastructure helps accomplish this this by pointing out 90 | the `project.assets.json` file that is referencing the pre-built in the 91 | `./artifacts/source-build/self/prebuild-report/prebuild-usage.xml` report file. 92 | A `project.assets.json` file is a NuGet restore process artifact that contains a 93 | resolved dependency tree for a specific project. Every package that was restored 94 | by a given project is mentioned there with links between the dependencies, 95 | allowing the reader to identify transitive dependencies and the direct 96 | dependencies referencing them. 97 | 98 | Example of a pre-built caused by a transitive dependency and corresponding 99 | entries in files mentioned above: 100 | 101 | Exception identifying the pre-built: 102 | 103 | ```text 104 | 1 new packages used not in baseline! See report at ./artifacts/source-build/self/prebuilt-report/baseline-comparison.xml for more information. Package IDs are: 105 | System.Text.Json.8.0.0 106 | ``` 107 | 108 | Entry in prebuilt-usage.xml: 109 | 110 | ```xml 111 | 112 | 113 | 114 | 115 | 116 | ``` 117 | 118 | Entry in project.assets.json: 119 | 120 | ```json 121 | { 122 | "targets": { 123 | "net8.0": { 124 | "Microsoft.Extensions.DependencyModel/8.0.0": { 125 | "type": "package", 126 | "dependencies": { 127 | "System.Text.Json": "8.0.0" 128 | } 129 | } 130 | } 131 | } 132 | } 133 | ``` 134 | 135 | In this example, `Microsoft.Extensions.DependencyModel` would be the direct 136 | dependency causing the `System.Text.Json` pre-built. 137 | 138 | ## Allowed exceptions 139 | 140 | The list of permitted pre-builts can be found in the 141 | `./eng/SourceBuildPrebuiltBaseline.xml` file in the root of the repository. It 142 | contains package information of pre-builts that for one reason or another are 143 | allowed in the source-build of the repository. 144 | 145 | Any new addition to the pre-built exception list must be signed-off by a member 146 | of the `@dotnet/source-build` team. 147 | 148 | A common example of a exception that is present in several .NET repositories is 149 | an [_intermediate 150 | package_](https://github.com/dotnet/source-build/blob/main/Documentation/planning/arcade-powered-source-build/intermediate-nupkg.md). 151 | When a repository utilizes an intermediate package, it will be excluded from 152 | pre-built detection with the following declaration in the above-mentioned file: 153 | 154 | ```xml 155 | 156 | 157 | 158 | 159 | 160 | ``` 161 | 162 | With this ignore pattern in place, pre-built detection will not mark any .NET 163 | intermediate package as long as it conforms to the naming in the pattern. 164 | 165 | In cases where a specific package needs to be excluded from pre-built detection 166 | (for example, to not block the introduction of changes until a source-build 167 | acceptable solution for the pre-built is provided), the developer can directly 168 | specify the name / version of the depedency: 169 | 170 | ```xml 171 | 172 | 173 | 174 | 175 | ``` 176 | 177 | If a new pre-built is encountered, pre-built detection will also generate a new 178 | version of the baseline file by adding the dependency that contains the 179 | pre-built to the existing baseline. The new file can be found at 180 | `./artifacts/source-build/self/prebuild-report/generated-new-baseline.xml`. 181 | 182 | ## Contacts 183 | 184 | For any questions or additional information about this document, pre-builts or 185 | source-build in general, please create an 186 | [issue](https://github.com/dotnet/source-build/issues) or open a 187 | [discussion](https://github.com/dotnet/source-build/discussions) in the 188 | [source-build](https://github.com/dotnet/source-build) repository. 189 | -------------------------------------------------------------------------------- /Documentation/leak-detection.md: -------------------------------------------------------------------------------- 1 | # Leak detection (poisoning) 2 | 3 | ## Overview 4 | 5 | Source-build includes a mechanism for *poisoning* its input files, and then for 6 | checking for that poison in the build output. This allows us to ensure that all 7 | output files were built during the build rather than copied directly from the 8 | input, which would be an illegal prebuilt usage. 9 | 10 | ## Before the build 11 | 12 | Before the build, the MarkAndCatalogFiles task runs. This does a few things: 13 | 14 | - Record the hash of every file in the source-build binary input directories 15 | (prebuilts, previously-source-built, and reference-packages). If the file is 16 | a zip, tarball, or nupkg, unpack it and do the same thing recursively. 17 | - For managed DLLs, either bare or in an archive, add a custom attribute that 18 | marks the file as poisoned. 19 | - For nupkgs, drop a `.poisoned` file. 20 | - Repack the poisoned assemblies and extra files, removing nupkg signatures so 21 | they don't fail verification. 22 | - Replace the binary inputs with these new packages and archives. 23 | - Record the hash of each file again, to make sure we will be able to track 24 | output files whether they were used poisoned or unpoisoned. 25 | 26 | ## During the build 27 | 28 | There's no change in source-build operation in poisoning mode during the build. 29 | 30 | **Note**: During the build of the source-build-reference-packages repository 31 | (regardless of poisoning mode), reference packages have the 32 | `System.Reflection.AssemblyMetadataAttribute("source", 33 | "source-build-reference-packages")` attribute injected into their respective 34 | reference assemblies. Leak detection flags all assemblies with this attribute. 35 | 36 | ## After the build 37 | 38 | After the build, the CheckForPoison task is run on the source-build output 39 | directory. It again unpacks any archives and packages recursively, and checks 40 | for the three kinds of markers (AssemblyAttribute, Hash, and NupkgFile) injected 41 | before the build and the source-build-reference-packages attribute added during 42 | the build. It then writes out a [report](poison-report-format.md) that details 43 | everything that was found. 44 | -------------------------------------------------------------------------------- /Documentation/package-dependency-flow.md: -------------------------------------------------------------------------------- 1 | # Package Dependency Flow 2 | 3 | This document describes how package dependencies are handled within source 4 | build. It describes the mechanisms that exist to control which package versions 5 | are used. 6 | 7 | ## Origins of Packages 8 | 9 | A source build must be self-contained, meaning the entire product must be built 10 | from source in an offline environment. To achieve this, all packages 11 | dependencies must be satisfied by one of the following: 12 | 13 | ### Source-Build-Reference-Packages 14 | 15 | The first repo that is built as part of source build is 16 | [source-build-reference-packages](https://github.com/dotnet/source-build-reference-packages). 17 | This repo contains all of the reference packages used to build the product. The 18 | repo contains [tooling to generate new reference 19 | packages](https://github.com/dotnet/source-build-reference-packages?tab=readme-ov-file#adding-new-packages). 20 | 21 | ### Current Source Built Packages 22 | 23 | This refers to all of the packages produced in the current build. The set of 24 | packages available to each repo varies based on its build order. For example the 25 | msbuild repo can take a dependency on the current version of 26 | Microsoft.CodeAnalysis from roslyn because roslyn builds before msbuild. 27 | Conversely, since roslyn builds before msbuild, roslyn cannot take a dependency 28 | on the current version of Microsoft.Build; it can only take a dependency on a 29 | previously released version. 30 | 31 | ### Previous Source Built Packages 32 | 33 | Because the .NET product uses itself to build, the .NET source build product 34 | must be [bootstrapped](./bootstrapping-guidelines.md). This process allows the 35 | packages from the previous source build release to be used to build the next 36 | version of the product. This provides a means for breaking the product's 37 | circular dependencies. For example repos like 38 | [arcade](https://github.com/dotnet/arcade) can self-reference its previous 39 | version to produce the next version. 40 | 41 | When referencing previous source built packages, it is important to not leak 42 | these previously built packages into the resulting packages/product. This is 43 | considered a [poison leak](./leak-detection.md) and is not permitted during a 44 | source build as it breaks the notion of building the product entirely from 45 | source. This hinders the ability to service the product. 46 | 47 | ## Package Versions 48 | 49 | Package dependencies defined using [Arcade's dependency 50 | patterns](https://github.com/dotnet/arcade/blob/main/Documentation/Darc.md) will 51 | get lifted dynamically during a source build if the following conditions are 52 | met: 53 | 54 | 1. The dependency is declared in the Version.Details.xml file. 55 | 56 | **Version.Details.xml** 57 | 58 | ```xml 59 | ... 60 | 61 | https://github.com/dotnet/command-line-api 62 | 02fe27cd6a9b001c8feb7938e6ef4b3799745759 63 | 64 | ... 65 | ``` 66 | 67 | 1. A corresponding version property is defined in the Versions.props. 68 | 69 | **Versions.props** 70 | 71 | ```xml 72 | ... 73 | 2.0.0-beta4.24068.1 74 | ... 75 | ``` 76 | 77 | 1. A repository reference is defined in the [VMR's project dependency 78 | graph](https://github.com/dotnet/dotnet/tree/main/repo-projects). This 79 | reference does not have to be direct, it can be transitive. 80 | 81 | **{VMR repo project}.proj** 82 | 83 | ```xml 84 | ... 85 | 86 | ... 87 | ``` 88 | 89 | When these conditions are met during a source build, the infrastructure will 90 | scan the Version.Details.xml file and dynamically create two new Versions.props 91 | files containing updated version properties for all non-pinned dependencies. 92 | 93 | **PackageVersions.Previous.props:** This will contain version properties with 94 | the package versions from the [previous release of source 95 | build](#previous-source-built-packages). If a new package exists that has never 96 | been released before, it will not have a version property defined. 97 | 98 | ```xml 99 | ... 100 | 2.0.0-beta3 101 | ... 102 | ``` 103 | 104 | **PackageVersions.Current.props:** This will contain version properties with the 105 | package versions from the [current source 106 | build](#current-source-built-packages). If a package comes from a repo that has 107 | not been built yet, it will not have a version property defined. 108 | 109 | ```xml 110 | ... 111 | 2.0.0-beta4 112 | ... 113 | ``` 114 | 115 | These two version.props files get imported by the arcade source build 116 | infrastructure after the repo's Version.props file. Therefore the repo's 117 | Versions.props property versions get overridden by the source build versions. In 118 | the case of the `SystemCommandLineVersion` example, the current source build 119 | version, 2.0.0-beta4, would win. All msbuild references (e.g. project 120 | PackageReferences) to these Versions.props properties pick up the newer 121 | versions. This is known as package version lifting since it lifts the originally 122 | defined package version to the current source built version. This behavior only 123 | applies to source build in the context of the 124 | [VMR](https://github.com/dotnet/dotnet) (see also [Repo Level Source 125 | Builds](#repo-level-source-builds)). 126 | 127 | ### Transitive Version Properties 128 | 129 | Transitive version properties in your Versions.props file may not work as 130 | intended with source build. 131 | 132 | #### Versions.props 133 | 134 | ```xml 135 | ... 136 | 17.7.0-preview-23217-02 137 | $(MicrosoftBuildFrameworkVersion) 138 | ... 139 | ``` 140 | 141 | #### Version.Details.xml 142 | 143 | ```xml 144 | ... 145 | 146 | https://github.com/dotnet/msbuild 147 | 2cbc8b6aef648cf21c6a68a0dab7fe09a614e475 148 | 149 | 150 | ... 151 | ``` 152 | 153 | In this case source build will override the `MicrosoftBuildFrameworkVersion` to 154 | the latest version but the `MicrosoftBuildVersion` will remain set to 155 | `17.7.0-preview-23217-02` because of the property evaluation order. If the 156 | desired behavior is for `MicrosoftBuildVersion` to be set to the same value as 157 | `MicrosoftBuildFrameworkVersion` for source build, then you either need to 158 | declare the Microsoft.Build dependency in the Version.Details.xml file or move 159 | the `MicrosoftBuildVersion` assignment outside of the Versions.props file. 160 | 161 | ### Repo Level Source Builds 162 | 163 | The source build package lifting mechanism is not applicable when building 164 | individual repos in source build mode because it doesn't have the context of the 165 | other product repos or previous source build release. In repo source build mode, 166 | the versions of the packages declared in the Versions.props are used (see also 167 | [backlog issue](https://github.com/dotnet/source-build/issues/3562)). 168 | -------------------------------------------------------------------------------- /Documentation/packaging-installation.md: -------------------------------------------------------------------------------- 1 | # .NET Core Packaging and Installation 2 | 3 | This document helps you install or package a .NET Core SDK built using 4 | source-build. 5 | 6 | The SDK built by source-build is generally not portable. That means it will work 7 | on the same operating system where it was built. It will not work on older 8 | operating systems. It may work on newer operating systems. 9 | 10 | The built SDK is generally located at 11 | `artifacts/${ARCHITECTURE}/Release/dotnet-sdk-${SDK_VERSION}-${RUNTIME_ID}.tar.gz`. 12 | 13 | ## Using the SDK directly (install per-user) 14 | 15 | To use the SDK directly, unpack the SDK to any directory and then use the 16 | `dotnet` executable from it. 17 | 18 | You can find more details on how to manually install an SDK from a tarball at 19 | [manually installing an 20 | SDK](https://docs.microsoft.com/en-us/dotnet/core/install/linux-ubuntu#manual-install). 21 | The same steps listed should work for all Linux distributions where `bash` (or 22 | `sh`) is the default shell. 23 | 24 | ## Installing the SDK globally 25 | 26 | If you want to install the SDK globally (and not per user), here are some 27 | suggestions. 28 | 29 | 1. Extract the tarball to a distribution-appropriate location such as 30 | `/usr/local/lib64/dotnet/`, `/usr/local/lib/dotnet/`, or 31 | `/usr/local/lib/x86_64-linux-gnu/dotnet/`. 32 | 33 | 2. Create a symlink from `/usr/local/bin/dotnet` (or an equivalent location 34 | available in `$PATH`) to the `dotnet` binary in the SDK that you installed in 35 | the previous step. For example: 36 | 37 | ```bash 38 | ln -s /usr/local/lib64/dotnet/dotnet /usr/local/bin/dotnet 39 | ``` 40 | 41 | Now users can simply run `dotnet` and it will work. 42 | 43 | 3. Create an `/etc/dotnet/install_location` file and add the path of the SDK 44 | directory in there. The file should contain a single line like this: 45 | 46 | ```bash 47 | /usr/local/lib64/dotnet/ 48 | ``` 49 | 50 | This file is used by [.NET Core to find the SDK/Runtime 51 | location](https://github.com/dotnet/designs/blob/master/accepted/2020/install-locations.md) 52 | 53 | 4. Define `DOTNET_ROOT` and update `PATH` by saving the following as 54 | `/etc/profile.d/dotnet-local.sh` (or equivalent) 55 | 56 | ```bash 57 | # Set location for AppHost lookup 58 | [ -z "$DOTNET_ROOT" ] && export DOTNET_ROOT=/usr/local/lib64/dotnet 59 | 60 | # Add dotnet tools directory to PATH 61 | DOTNET_TOOLS_PATH="$HOME/.dotnet/tools" 62 | case "$PATH" in 63 | *"$DOTNET_TOOLS_PATH"* ) true ;; 64 | * ) PATH="$PATH:$DOTNET_TOOLS_PATH" ;; 65 | esac 66 | ``` 67 | 68 | Make sure to adjust the paths to match what you used on your system. 69 | 70 | This snippet should work in `sh` (including `dash`) and `bash`. You may need 71 | to adapt it, or use something entirely different, for other shells. 72 | 73 | This allows apphost-lookup to work via `DOTNET_ROOT` and allows users to 74 | easily use dotnet tools directly after a `dotnet tool install`. 75 | 76 | ## Creating a Linux distribution package 77 | 78 | If you want to create a Linux distribution package (`rpm`, `deb`) out of the 79 | source-build SDK here are some suggestions. 80 | 81 | See [.NET Core distribution 82 | packaging](https://docs.microsoft.com/en-us/dotnet/core/distribution-packaging) 83 | for information on suggested packages, subpackages, name and contents. 84 | 85 | This is the minimal amount of content you need to package up: 86 | 87 | 1. Extract the tarball to the distribution appropriate location such as 88 | `/usr/lib64/dotnet/`, `/usr/lib/dotnet/`, or 89 | `/usr/lib/x86_64-linux-gnu/dotnet/`. 90 | 91 | 2. Create a symlink from `/usr/bin/dotnet` (or equivalent) to the `dotnet` 92 | binary in the SDK that you installed in the previous step. For example: 93 | 94 | ```bash 95 | ln -s /usr/lib64/dotnet/dotnet /usr/bin/dotnet 96 | ``` 97 | 98 | Now users can simply run `dotnet` and it will work. 99 | 100 | 3. Create an `/etc/dotnet/install_location` file and add the path of the SDK 101 | directory in there. The file should contain a single line like this: 102 | 103 | ```bash 104 | /usr/lib64/dotnet 105 | ``` 106 | 107 | This file is used by [.NET Core to find the SDK/Runtime 108 | location](https://github.com/dotnet/designs/blob/master/accepted/2020/install-locations.md). 109 | 110 | 4. Define `DOTNET_ROOT` and update `PATH` by saving the following as 111 | `/etc/profile.d/dotnet.sh` (or equivalent) 112 | 113 | ```bash 114 | # Set location for AppHost lookup 115 | [ -z "$DOTNET_ROOT" ] && export DOTNET_ROOT=/usr/lib64/dotnet 116 | 117 | # Add dotnet tools directory to PATH 118 | DOTNET_TOOLS_PATH="$HOME/.dotnet/tools" 119 | case "$PATH" in 120 | *"$DOTNET_TOOLS_PATH"* ) true ;; 121 | * ) PATH="$PATH:$DOTNET_TOOLS_PATH" ;; 122 | esac 123 | ``` 124 | 125 | Make sure to adjust the paths to match what the distribution policies. 126 | 127 | This snippet should work in `sh` (including `dash`) and `bash`. You may need 128 | to adapt it, or use something entirely different, for other shells. 129 | 130 | This allows apphost-lookup to work via `DOTNET_ROOT` and allows users to 131 | easily use dotnet tools directly after a `dotnet tool install`. 132 | 133 | There are other optional things you can do: 134 | 135 | - .NET Core source repositories include man pages. You can search for them and 136 | package them up: `find -iname '*.1' -exec cp {} /usr/share/man/man1/ \;` 137 | 138 | - .NET Core includes 139 | [bash-completion](https://github.com/dotnet/cli/blob/master/scripts/register-completions.bash) 140 | and 141 | [zsh-completion](https://github.com/dotnet/cli/blob/master/scripts/register-completions.zsh) 142 | scripts. The copies in the source code you used to build the SDK should be the 143 | latest version. See [how to enable tab completion for .NET Core 144 | cli](https://docs.microsoft.com/en-us/dotnet/core/tools/enable-tab-autocomplete) 145 | for more information. 146 | 147 | ## Resources and references 148 | 149 | - [.NET Core distribution 150 | packaging](https://docs.microsoft.com/en-us/dotnet/core/distribution-packaging) 151 | - [Fedora .NET Core 3.1 152 | package](https://src.fedoraproject.org/rpms/dotnet3.1/tree/master) 153 | - [.NET Core - ArchWiki](https://wiki.archlinux.org/index.php/.NET_Core) 154 | - [State of .NET Core on Arch - a discussion between a few distribution package 155 | maintainers](https://www.reddit.com/r/archlinux/comments/cx64r5/the_state_of_net_core_on_arch/) 156 | -------------------------------------------------------------------------------- /Documentation/patching-guidelines.md: -------------------------------------------------------------------------------- 1 | # Patching Guidelines 2 | 3 | There are times when the source build product will have build errors, functional 4 | defects, poison leaks, prebuilts, etc. Ideally, the underlying issues would be 5 | addressed at the source of the problem (e.g. within the product repos) and flow 6 | into source build. The reality is that repo fixes can take a long time to flow 7 | into source build. This can block source build development (e.g. builds, tests, 8 | feature development, etc.). To avoid these delays source patches are applied 9 | within source-build to the repo source. These patches are intended to be 10 | ephemeral until the repo fixes flow in. 11 | 12 | This document provides guidance for creating and managing source build patches. 13 | 14 | ## Creating Patches 15 | 16 | To create a repo patch file, first commit your changes to the source repo (e.g. 17 | runtime, aspnetcore) as normal, then run this command inside the repo to 18 | generate a patch file inside the repo: 19 | 20 | ```sh 21 | git format-patch --zero-commit --no-signature -1 22 | ``` 23 | 24 | Then, move the patch file into the `src/SourceBuild/patches/` directory of 25 | the following repo: 26 | 27 | * [.NET 9.0+] 28 | [sdk](https://github.com/dotnet/sdk/tree/main/src/SourceBuild/patches) 29 | * [.NET 8.0] 30 | [installer](https://github.com/dotnet/installer/tree/main/src/SourceBuild/patches) 31 | 32 | If an existing directory for the repo does not already exist, you will need to 33 | create one. 34 | 35 | > If you define `PATCH_DIR` to point at the `patches` directory, you can use 36 | > `-o` to place the patch file directly in the right directory: 37 | > 38 | > ```sh 39 | > git format-patch --zero-commit --no-signature -1 -o "$PATCH_DIR/" 40 | > ``` 41 | 42 | ## Applying Patches 43 | 44 | To apply a patch, or multiple patches, use `git am` while inside the target 45 | repo. For example, to apply *all* `sdk` patches onto a fresh clone of the `sdk` 46 | repository that has already been checked out to the correct commit, use: 47 | 48 | ```sh 49 | git am "$PATCH_DIR/sdk/*" 50 | ``` 51 | 52 | This creates a Git commit with the patch contents, so you can easily amend a 53 | patch or create a new commit on top that you can be sure will apply cleanly. 54 | 55 | > **Note:** The VMR has all of the `src/SourceBuild/patches` applied. This is 56 | done as part of the [synchronization 57 | process](https://github.com/dotnet/arcade/blob/main/Documentation/UnifiedBuild/VMR-Design-And-Operation.md#source-build-patches). 58 | 59 | ## Patch Guidelines 60 | 61 | 1. **Documentation** 62 | 63 | The patch file name and `Subject` should indicate the purpose of the patch. 64 | It is often helpful to explain the problem/build issue the patch addresses. 65 | 66 | 1. **Naming Conventions** 67 | 68 | After generating the patch file, the numeric prefix of the filename may need 69 | to be changed. By convention, new patches should be one number above the 70 | largest number that already exists in the patch file directory. 71 | 72 | > **Note:** If there's a gap in the number sequence, do not fix it 73 | (generally speaking), to avoid unnecessary diffs and potential merge 74 | conflicts. 75 | 76 | 1. **Backport** 77 | 78 | All patches should have an issue or PR opened to address/backport the patch 79 | before openings PRs that add new patches. All patches are required to 80 | include a link in its `Subject` to the backport issue/PR in the following 81 | format: 82 | 83 | `Backport: ` 84 | 85 | Example: 86 | 87 | ``` text 88 | From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001 89 | From: John Doe 90 | Date: Mon, 17 Jul 2023 15:10:02 +0000 91 | Subject: [PATCH] Updates to work with the latest command-line APIs 92 | 93 | Backport: https://github.com/dotnet/sourcelink/pull/1068 94 | ``` 95 | 96 | 1. **Consult Repo Experts** 97 | 98 | It is a good practive to include repo owners or subject matter experts when 99 | authoring patches and include them as a required reviewer. 100 | 101 | ## Resolving Patch Conflicts 102 | 103 | Dependency flow PRs into the installer repo can cause patch conflicts (e.g. the 104 | patch will fail to apply). These conflicts come in two forms. 105 | 106 | 1. The patch was backported - when this happen, the backported patch will need 107 | to be deleted. 108 | 109 | 1. The code being changed or surrounding code was changed - when this happens, 110 | the patch will need to be updated. This can be done by manually updating the 111 | patch or by re-applying the changes to the repo and recreating the patch. 112 | 113 | ## Unified Build Plans 114 | 115 | The [Unified 116 | Build](https://github.com/dotnet/arcade/blob/main/Documentation/UnifiedBuild/README.md) 117 | project will add support for source edits in the 118 | [VMR](https://github.com/dotnet/dotnet). This will eliminate the need for 119 | patches as the required changes can be directly made in the VMR. All changes 120 | made to the VMR will automatically flow to the associated repos. 121 | -------------------------------------------------------------------------------- /Documentation/planning/arcade-powered-source-build/README.md: -------------------------------------------------------------------------------- 1 | # ArPow (arcade-powered source-build) 2 | 3 | Source-build exists as a process outside the Microsoft build, taking place after 4 | the Microsoft build is complete. This creates a significant amount of catch-up 5 | work to keep .NET buildable from source. Integration into the normal build 6 | process is necessary to make source-build sustainable. 7 | 8 | This is a user story in .NET 6.0: 9 | [dotnet/core#5448](https://github.com/dotnet/core/issues/5448) 10 | 11 | > *Definitions*: 12 | > 13 | > * **Official builds** run in Azure DevOps and produce the signed bits released 14 | > by Microsoft. These run on a daily or per-commit basis depending on repo and 15 | > point in time. 16 | > * **PR validation** builds run on each pull request submitted to a repository, 17 | > and are typically required to succeed to merge. There are also **rolling 18 | > builds** that run either periodically or after each merge that run 19 | > lower-priority and/or slower tests. 20 | 21 | ArPow consists of two main goals with some key benefits: 22 | 23 | 1. When the .NET Core SDK **official build** completes, its artifacts include 24 | validated, ship-ready source-build outputs, in addition to the Microsoft 25 | build outputs. 26 | 27 | * This way, we know that if a given SDK commit has a Microsoft build 28 | available, it is also buildable from source. 29 | 30 | * If an error occurs during source-build, it is visible to the same 31 | stakeholders as a Microsoft build failure, not just the source-build team. 32 | 33 | * Breaks in source-build can be detected and fixed immediately by repo 34 | owners, just as any other build failure. This is opposed to the traditional 35 | workflow: source-build maintainers detect the issue weeks to months after 36 | the fact, and must fix it using patches. 37 | 38 | * With ArPow and sufficient automated testing, there is zero delay between 39 | Microsoft build completion and source-build readiness. The traditional 40 | delay no longer needs to be considered by the Microsoft release process 41 | owners when figuring out a release timeline. 42 | 43 | * Rebuilding the SDK close to a release day doesn't cause setbacks to 44 | source-build maintainers by throwing away manual build uptake work. 45 | 46 | 2. Every repo involved in source-build validates source-buildability in its **PR 47 | validation** build. 48 | 49 | * This allows developers and release owners to understand the source-build 50 | impact of changes, reducing the frequency the source-build servicing team 51 | has to root-cause and patch over problems. 52 | 53 | * Gives us a place to add additional source-build-specific tests for common 54 | problems, such as prebuilt usage regressions. 55 | 56 | This doc is about where we can start, what incremental progress would look like, 57 | and the vision. 58 | 59 | ## Starting point: SBRP (dotnet/source-build-reference-packages) 60 | 61 | Status: Complete. 62 | 63 | We start by putting ArPow build infrastructure into SBRP, a repository owned by 64 | the source-build team that has no upstream dependencies. This is done using the 65 | Arcade SDK's extension points and custom Azure DevOps templates. Both are 66 | integrated into Arcade after we see them working in SBRP. This lets us quickly 67 | prototype the core ArPow functionality. 68 | 69 | There are a few key parts of the infrastructure to prototype: 70 | 71 | ### Build integration 72 | 73 | The official build and PR validation build pipelines work, with a reasonable API 74 | for other repositories to adopt. 75 | 76 | * [source-build-in-pipeline.md] 77 | 78 | ### Intermediate nupkg outputs/inputs 79 | 80 | Every ArPow-enabled repository creates a NuGet package containing its build 81 | outputs, for downstream repositories to use. On the other side, the Arcade SDK 82 | infrastructure needs to be able to restore and use source-built intermediate 83 | nupkg content. 84 | 85 | SBRP ultimately won't use the restore-side infrastructure because it has no 86 | upstreams, however, it can be developed here as a prototype. 87 | 88 | * [intermediate-nupkg.md] 89 | * [Create proof of concept: the source-build intermediate nupkg format 90 | #1543](https://github.com/dotnet/source-build/issues/1543) 91 | * [Use SBRP intermediate nupkg to build SBRP 92 | #1636](https://github.com/dotnet/source-build/issues/1636) 93 | 94 | ### Prebuilt usage tracking 95 | 96 | We aren't eliminating instances of prebuilt usage at this stage in the process, 97 | but the tooling needs to work in the context of an Arcade-powered build. 98 | 99 | * [Include prebuilt reports in intermediate nupkgs 100 | #1725](https://github.com/dotnet/source-build/issues/1725) 101 | 102 | ### Managed-only and RID-specific builds 103 | 104 | Some .NET repositories only produce managed code outputs, and others produce 105 | RID-specific outputs such as native binaries. To reduce the burden of ArPow on 106 | small managed-only repositories, we build only one RID and use the same results 107 | for all RIDs downstream. All repositories must be buildable under any RID, but 108 | for simple managed-only repositories, this is a reasonable assumption. The 109 | templates and Arcade SDK need to handle this. 110 | 111 | * [Add mechanism to restore RID-specific intermediate nupkgs 112 | #1722](https://github.com/dotnet/source-build/issues/1722) 113 | 114 | ### Integrating tooling into Arcade 115 | 116 | The ArPow tooling can be implemented inside the Arcade SDK's MSBuild extension 117 | points, initially. Before onboarding any extra repos, it should be integrated 118 | into the Arcade SDK proper to reuse the code. 119 | 120 | * [in-arcade.md] 121 | * [Source code location in Arcade SDK 122 | (`Microsoft.DotNet.Arcade.Sdk/tools/SourceBuild`)](https://github.com/dotnet/arcade/tree/master/src/Microsoft.DotNet.Arcade.Sdk/tools/SourceBuild) 123 | * [Source-build-specific MSBuild tool source in Arcade SDK 124 | (`Microsoft.DotNet.SourceBuild`)](https://github.com/dotnet/arcade/tree/master/src/Microsoft.DotNet.SourceBuild) 125 | 126 | ## Incremental progress 127 | 128 | ### Adding ArPow to the build graph 129 | 130 | When we have ArPow SBRP complete, we start adding ArPow to repositories in the 131 | .NET SDK dependency graph. 132 | 133 | Working from the bottom (leaves) upward (towards dotnet/installer), we onboard 134 | repos to ArPow: consume intermediate nupkgs from upstreams, run ArPow PR 135 | validation, and produce intermediate nupkgs in the official build. For more 136 | details about this process, see: 137 | 138 | * [incremental-official.md] 139 | 140 | It is possible to instead only implement official source-build in a handful of 141 | repos. This segments the build into chunks, which must be coherent. This idea is 142 | discussed in [incremental-official-chunked.md], and is not recommended. 143 | 144 | Some repos required to build .NET aren't maintained by Microsoft, and don't have 145 | official builds that inject artifacts into the Arcade dependency flow system 146 | (Darc/BAR/Maestro++). It isn't feasible to add ArPow to these repos in the same 147 | way. These repos will continue being built by dotnet/source-build, and 148 | dotnet/source-build becomes a leaf in the build graph instead of an 149 | orchestration repo. 150 | 151 | ### The speculative SDK 152 | 153 | It's difficult to validate that a PR won't break downstream repos. This problem 154 | is shared by source-build and the Microsoft build. "Speculative builds" have 155 | been proposed to try to help with this, but would be very difficult to implement 156 | in the Microsoft build. It may be more reasonable in the context of 157 | source-build: all builds happen on a single machine, so the problem is focused 158 | on figuring out a build graph rather than organizing dozens of machines in a 159 | build lab and flowing bits across a network. 160 | 161 | This is also necessary in source-build to validate several distro maintenance 162 | scenarios: by making a PR, is it still possible to run a bootstrap build of the 163 | .NET Core SDK? Can .NET Core SDK version N be built using SDK N-1? 164 | 165 | This can be developed in parallel to other efforts. See [speculative-build.md] 166 | for more info about speculative builds. 167 | 168 | ### Adding unit tests 169 | 170 | Unit tests are typically disabled in dotnet/source-build, because the test 171 | infrastructure isn't built from source, and it increases build time. We should 172 | run tests on the source-build product to catch bugs. However, this isn't 173 | critical for the sustainability goals of this plan. 174 | 175 | ## End result 176 | 177 | When all of this is working, the official Microsoft build of the .NET Core SDK 178 | also produces tarballs that distro maintainers can use to build it from source. 179 | Contributors in each repo use checks in PR validation to verify each change is 180 | compatible with source-build requirements, and if validation runs into a 181 | problem, they are able to reproduce the build locally using an Arcade build 182 | command. 183 | 184 | [in-arcade.md]: in-arcade.md 185 | [incremental-official-chunked.md]: incremental-official-chunked.md 186 | [incremental-official.md]: incremental-official.md 187 | [source-build-in-pipeline.md]: source-build-in-pipeline.md 188 | [speculative-build.md]: speculative-build.md 189 | [intermediate-nupkg.md]: intermediate-nupkg.md 190 | [intermediate nupkgs]: intermediate-nupkg.md 191 | 192 | --- 193 | 194 | ## Revisions 195 | 196 | **2021-02-17** dagood 197 | Updated this doc to align with the current ArPow implementation plan. The old 198 | implementation plan can be found at: 199 | [2bf686a9](https://github.com/dotnet/source-build/tree/2bf686a91560911477885ae139909bbf10b7ca98/Documentation/planning/arcade-powered-source-build). 200 | The plan has shifted towards purely incremental changes, with no end-to-end 201 | ArPow source-build working until completion. 202 | 203 | **2021-02-11** dagood 204 | Added a plan to use "supplemental intermediate nupkgs" to handle situations 205 | where the [intermediate nupkgs] are too large to publish to Azure DevOps. 206 | 207 | **2020-07-15** dagood 208 | Removed the plan to test every added intermediate nupkg all the way downstream 209 | in dotnet/installer. Looking at it after some hands-on work has been done, we 210 | don't think this end-to-end integration test is actually feasible. Not running 211 | these tests creates some uncertainty, but it seems acceptable. We will likely 212 | end up with a backlog of unknown issues to work through once we start building a 213 | tarball in dotnet/installer. They *may* have been avoided with testing. We don't 214 | expect this will be disruptive enough to make it worth trying to more 215 | exhaustively find some way to get end-to-end testing feasible. 216 | -------------------------------------------------------------------------------- /Documentation/planning/arcade-powered-source-build/img/official-leaves.dot: -------------------------------------------------------------------------------- 1 | digraph { 2 | node [color="lightsteelblue1", style=filled]; 3 | rankdir=BT; 4 | 5 | "sourcelink"[color="gold"]; 6 | "xliff-tasks"[color="gold"]; 7 | "newtonsoft-json901"[color="gold"]; 8 | "linker"[color="gold"]; 9 | "application-insights"[color="gold"]; 10 | "netcorecli-fsc"[color="gold"]; 11 | 12 | "sourcelink" -> "arcade" 13 | "arcade" -> "aspnet-extensions" 14 | "core-setup" -> "aspnet-extensions" 15 | "roslyn" -> "aspnet-extensions" 16 | "arcade" -> "aspnet-xdt" 17 | "arcade" -> "aspnetcore-tooling" 18 | "aspnet-extensions" -> "aspnetcore-tooling" 19 | "core-setup" -> "aspnetcore-tooling" 20 | "msbuild" -> "aspnetcore-tooling" 21 | "roslyn" -> "aspnetcore-tooling" 22 | "arcade" -> "aspnetcore" 23 | "aspnetcore-tooling" -> "aspnetcore" 24 | "core-setup" -> "aspnetcore" 25 | "msbuild" -> "aspnetcore" 26 | "roslyn" -> "aspnetcore" 27 | "arcade" -> "cli" 28 | "application-insights" -> "cli" 29 | "clicommandlineparser" -> "cli" 30 | "core-setup" -> "cli" 31 | "corefx" -> "cli" 32 | "msbuild" -> "cli" 33 | "nuget-client" -> "cli" 34 | "newtonsoft-json" -> "cli" 35 | "newtonsoft-json901" -> "cli" 36 | "roslyn" -> "cli" 37 | "sdk" -> "cli" 38 | "standard" -> "cli" 39 | "templating" -> "cli" 40 | "websdk" -> "cli" 41 | "xliff-tasks" -> "cli" 42 | "arcade" -> "clicommandlineparser" 43 | "newtonsoft-json" -> "common" 44 | "aspnetcore" -> "core-sdk" 45 | "newtonsoft-json" -> "core-sdk" 46 | "newtonsoft-json901" -> "core-sdk" 47 | "toolset" -> "core-sdk" 48 | "coreclr" -> "core-setup-portable" 49 | "corefx" -> "core-setup-portable" 50 | "newtonsoft-json" -> "core-setup-portable" 51 | "newtonsoft-json901" -> "core-setup-portable" 52 | "roslyn" -> "core-setup-portable" 53 | "standard" -> "core-setup-portable" 54 | "coreclr" -> "core-setup" 55 | "corefx" -> "core-setup" 56 | "newtonsoft-json" -> "core-setup" 57 | "newtonsoft-json901" -> "core-setup" 58 | "roslyn" -> "core-setup" 59 | "standard" -> "core-setup" 60 | "linker" -> "coreclr-portable" 61 | "roslyn" -> "coreclr-portable" 62 | "linker" -> "coreclr" 63 | "roslyn" -> "coreclr" 64 | "arcade" -> "corefx-portable" 65 | "linker" -> "corefx-portable" 66 | "coreclr" -> "corefx-portable" 67 | "standard" -> "corefx-portable" 68 | "arcade" -> "corefx" 69 | "linker" -> "corefx" 70 | "coreclr" -> "corefx" 71 | "standard" -> "corefx" 72 | "arcade" -> "fsharp" 73 | "newtonsoft-json901" -> "fsharp" 74 | "newtonsoft-json" -> "fsharp" 75 | "xliff-tasks" -> "fsharp" 76 | "arcade" -> "msbuild" 77 | "corefx" -> "msbuild" 78 | "roslyn" -> "msbuild" 79 | "arcade" -> "newtonsoft-json" 80 | "newtonsoft-json" -> "nuget-client" 81 | "common" -> "nuget-client" 82 | "msbuild" -> "nuget-client" 83 | "arcade" -> "roslyn" 84 | "xliff-tasks" -> "roslyn" 85 | "arcade" -> "sdk" 86 | "xliff-tasks" -> "sdk" 87 | "core-setup" -> "sdk" 88 | "msbuild" -> "sdk" 89 | "newtonsoft-json" -> "sdk" 90 | "newtonsoft-json901" -> "sdk" 91 | "nuget-client" -> "sdk" 92 | "arcade" -> "standard" 93 | "clicommandlineparser" -> "templating" 94 | "newtonsoft-json" -> "templating" 95 | "aspnetcore-tooling" -> "toolset" 96 | "cli" -> "toolset" 97 | "fsharp" -> "toolset" 98 | "netcorecli-fsc" -> "toolset" 99 | "newtonsoft-json" -> "toolset" 100 | "newtonsoft-json901" -> "toolset" 101 | "vstest" -> "toolset" 102 | "websdk" -> "toolset" 103 | "aspnet-extensions" -> "vstest" 104 | "core-setup" -> "vstest" 105 | "newtonsoft-json" -> "vstest" 106 | "arcade" -> "websdk" 107 | "aspnetcore" -> "websdk" 108 | "aspnet-xdt" -> "websdk" 109 | "newtonsoft-json" -> "websdk" 110 | } 111 | -------------------------------------------------------------------------------- /Documentation/planning/arcade-powered-source-build/img/official-leaves.svg: -------------------------------------------------------------------------------- 1 | 2 | 4 | 6 | 7 | 9 | 10 | %3 11 | 12 | 13 | sourcelink 14 | 15 | sourcelink 16 | 17 | 18 | arcade 19 | 20 | arcade 21 | 22 | 23 | sourcelink->arcade 24 | 25 | 26 | 27 | 28 | xliff-tasks 29 | 30 | xliff-tasks 31 | 32 | 33 | roslyn 34 | 35 | roslyn 36 | 37 | 38 | xliff-tasks->roslyn 39 | 40 | 41 | 42 | 43 | cli 44 | 45 | cli 46 | 47 | 48 | xliff-tasks->cli 49 | 50 | 51 | 52 | 53 | sdk 54 | 55 | sdk 56 | 57 | 58 | xliff-tasks->sdk 59 | 60 | 61 | 62 | 63 | fsharp 64 | 65 | fsharp 66 | 67 | 68 | xliff-tasks->fsharp 69 | 70 | 71 | 72 | 73 | newtonsoft-json901 74 | 75 | newtonsoft-json901 76 | 77 | 78 | core-setup 79 | 80 | core-setup 81 | 82 | 83 | newtonsoft-json901->core-setup 84 | 85 | 86 | 87 | 88 | newtonsoft-json901->cli 89 | 90 | 91 | 92 | 93 | newtonsoft-json901->sdk 94 | 95 | 96 | 97 | 98 | core-sdk 99 | 100 | core-sdk 101 | 102 | 103 | newtonsoft-json901->core-sdk 104 | 105 | 106 | 107 | 108 | toolset 109 | 110 | toolset 111 | 112 | 113 | newtonsoft-json901->toolset 114 | 115 | 116 | 117 | 118 | core-setup-portable 119 | 120 | core-setup-portable 121 | 122 | 123 | newtonsoft-json901->core-setup-portable 124 | 125 | 126 | 127 | 128 | newtonsoft-json901->fsharp 129 | 130 | 131 | 132 | 133 | linker 134 | 135 | linker 136 | 137 | 138 | corefx 139 | 140 | corefx 141 | 142 | 143 | linker->corefx 144 | 145 | 146 | 147 | 148 | coreclr 149 | 150 | coreclr 151 | 152 | 153 | linker->coreclr 154 | 155 | 156 | 157 | 158 | coreclr-portable 159 | 160 | coreclr-portable 161 | 162 | 163 | linker->coreclr-portable 164 | 165 | 166 | 167 | 168 | corefx-portable 169 | 170 | corefx-portable 171 | 172 | 173 | linker->corefx-portable 174 | 175 | 176 | 177 | 178 | application-insights 179 | 180 | application-insights 181 | 182 | 183 | application-insights->cli 184 | 185 | 186 | 187 | 188 | netcorecli-fsc 189 | 190 | netcorecli-fsc 191 | 192 | 193 | netcorecli-fsc->toolset 194 | 195 | 196 | 197 | 198 | aspnet-extensions 199 | 200 | aspnet-extensions 201 | 202 | 203 | arcade->aspnet-extensions 204 | 205 | 206 | 207 | 208 | arcade->roslyn 209 | 210 | 211 | 212 | 213 | aspnet-xdt 214 | 215 | aspnet-xdt 216 | 217 | 218 | arcade->aspnet-xdt 219 | 220 | 221 | 222 | 223 | aspnetcore-tooling 224 | 225 | aspnetcore-tooling 226 | 227 | 228 | arcade->aspnetcore-tooling 229 | 230 | 231 | 232 | 233 | msbuild 234 | 235 | msbuild 236 | 237 | 238 | arcade->msbuild 239 | 240 | 241 | 242 | 243 | aspnetcore 244 | 245 | aspnetcore 246 | 247 | 248 | arcade->aspnetcore 249 | 250 | 251 | 252 | 253 | arcade->cli 254 | 255 | 256 | 257 | 258 | clicommandlineparser 259 | 260 | clicommandlineparser 261 | 262 | 263 | arcade->clicommandlineparser 264 | 265 | 266 | 267 | 268 | arcade->corefx 269 | 270 | 271 | 272 | 273 | newtonsoft-json 274 | 275 | newtonsoft-json 276 | 277 | 278 | arcade->newtonsoft-json 279 | 280 | 281 | 282 | 283 | arcade->sdk 284 | 285 | 286 | 287 | 288 | standard 289 | 290 | standard 291 | 292 | 293 | arcade->standard 294 | 295 | 296 | 297 | 298 | websdk 299 | 300 | websdk 301 | 302 | 303 | arcade->websdk 304 | 305 | 306 | 307 | 308 | arcade->corefx-portable 309 | 310 | 311 | 312 | 313 | arcade->fsharp 314 | 315 | 316 | 317 | 318 | aspnet-extensions->aspnetcore-tooling 319 | 320 | 321 | 322 | 323 | vstest 324 | 325 | vstest 326 | 327 | 328 | aspnet-extensions->vstest 329 | 330 | 331 | 332 | 333 | core-setup->aspnet-extensions 334 | 335 | 336 | 337 | 338 | core-setup->aspnetcore-tooling 339 | 340 | 341 | 342 | 343 | core-setup->aspnetcore 344 | 345 | 346 | 347 | 348 | core-setup->cli 349 | 350 | 351 | 352 | 353 | core-setup->sdk 354 | 355 | 356 | 357 | 358 | core-setup->vstest 359 | 360 | 361 | 362 | 363 | roslyn->aspnet-extensions 364 | 365 | 366 | 367 | 368 | roslyn->core-setup 369 | 370 | 371 | 372 | 373 | roslyn->aspnetcore-tooling 374 | 375 | 376 | 377 | 378 | roslyn->msbuild 379 | 380 | 381 | 382 | 383 | roslyn->aspnetcore 384 | 385 | 386 | 387 | 388 | roslyn->cli 389 | 390 | 391 | 392 | 393 | roslyn->coreclr 394 | 395 | 396 | 397 | 398 | roslyn->core-setup-portable 399 | 400 | 401 | 402 | 403 | roslyn->coreclr-portable 404 | 405 | 406 | 407 | 408 | aspnet-xdt->websdk 409 | 410 | 411 | 412 | 413 | aspnetcore-tooling->aspnetcore 414 | 415 | 416 | 417 | 418 | aspnetcore-tooling->toolset 419 | 420 | 421 | 422 | 423 | msbuild->aspnetcore-tooling 424 | 425 | 426 | 427 | 428 | msbuild->aspnetcore 429 | 430 | 431 | 432 | 433 | msbuild->cli 434 | 435 | 436 | 437 | 438 | nuget-client 439 | 440 | nuget-client 441 | 442 | 443 | msbuild->nuget-client 444 | 445 | 446 | 447 | 448 | msbuild->sdk 449 | 450 | 451 | 452 | 453 | aspnetcore->websdk 454 | 455 | 456 | 457 | 458 | aspnetcore->core-sdk 459 | 460 | 461 | 462 | 463 | cli->toolset 464 | 465 | 466 | 467 | 468 | clicommandlineparser->cli 469 | 470 | 471 | 472 | 473 | templating 474 | 475 | templating 476 | 477 | 478 | clicommandlineparser->templating 479 | 480 | 481 | 482 | 483 | corefx->core-setup 484 | 485 | 486 | 487 | 488 | corefx->msbuild 489 | 490 | 491 | 492 | 493 | corefx->cli 494 | 495 | 496 | 497 | 498 | corefx->core-setup-portable 499 | 500 | 501 | 502 | 503 | nuget-client->cli 504 | 505 | 506 | 507 | 508 | nuget-client->sdk 509 | 510 | 511 | 512 | 513 | newtonsoft-json->core-setup 514 | 515 | 516 | 517 | 518 | newtonsoft-json->cli 519 | 520 | 521 | 522 | 523 | newtonsoft-json->nuget-client 524 | 525 | 526 | 527 | 528 | newtonsoft-json->sdk 529 | 530 | 531 | 532 | 533 | newtonsoft-json->templating 534 | 535 | 536 | 537 | 538 | newtonsoft-json->websdk 539 | 540 | 541 | 542 | 543 | common 544 | 545 | common 546 | 547 | 548 | newtonsoft-json->common 549 | 550 | 551 | 552 | 553 | newtonsoft-json->core-sdk 554 | 555 | 556 | 557 | 558 | newtonsoft-json->toolset 559 | 560 | 561 | 562 | 563 | newtonsoft-json->core-setup-portable 564 | 565 | 566 | 567 | 568 | newtonsoft-json->fsharp 569 | 570 | 571 | 572 | 573 | newtonsoft-json->vstest 574 | 575 | 576 | 577 | 578 | sdk->cli 579 | 580 | 581 | 582 | 583 | standard->core-setup 584 | 585 | 586 | 587 | 588 | standard->cli 589 | 590 | 591 | 592 | 593 | standard->corefx 594 | 595 | 596 | 597 | 598 | standard->core-setup-portable 599 | 600 | 601 | 602 | 603 | standard->corefx-portable 604 | 605 | 606 | 607 | 608 | templating->cli 609 | 610 | 611 | 612 | 613 | websdk->cli 614 | 615 | 616 | 617 | 618 | websdk->toolset 619 | 620 | 621 | 622 | 623 | common->nuget-client 624 | 625 | 626 | 627 | 628 | toolset->core-sdk 629 | 630 | 631 | 632 | 633 | coreclr->core-setup 634 | 635 | 636 | 637 | 638 | coreclr->corefx 639 | 640 | 641 | 642 | 643 | coreclr->core-setup-portable 644 | 645 | 646 | 647 | 648 | coreclr->corefx-portable 649 | 650 | 651 | 652 | 653 | fsharp->toolset 654 | 655 | 656 | 657 | 658 | vstest->toolset 659 | 660 | 661 | 662 | 663 | 664 | -------------------------------------------------------------------------------- /Documentation/planning/arcade-powered-source-build/img/rough-pipeline-official.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/dotnet/source-build/6bd71d2e5cc898d224b05f737c25b6f6b68c5c50/Documentation/planning/arcade-powered-source-build/img/rough-pipeline-official.png -------------------------------------------------------------------------------- /Documentation/planning/arcade-powered-source-build/img/rough-pipeline-validate.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/dotnet/source-build/6bd71d2e5cc898d224b05f737c25b6f6b68c5c50/Documentation/planning/arcade-powered-source-build/img/rough-pipeline-validate.png -------------------------------------------------------------------------------- /Documentation/planning/arcade-powered-source-build/img/spec-build-1-general.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/dotnet/source-build/6bd71d2e5cc898d224b05f737c25b6f6b68c5c50/Documentation/planning/arcade-powered-source-build/img/spec-build-1-general.png -------------------------------------------------------------------------------- /Documentation/planning/arcade-powered-source-build/img/spec-build-2-bootstrap.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/dotnet/source-build/6bd71d2e5cc898d224b05f737c25b6f6b68c5c50/Documentation/planning/arcade-powered-source-build/img/spec-build-2-bootstrap.png -------------------------------------------------------------------------------- /Documentation/planning/arcade-powered-source-build/img/spec-build-3-n-1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/dotnet/source-build/6bd71d2e5cc898d224b05f737c25b6f6b68c5c50/Documentation/planning/arcade-powered-source-build/img/spec-build-3-n-1.png -------------------------------------------------------------------------------- /Documentation/planning/arcade-powered-source-build/in-arcade.md: -------------------------------------------------------------------------------- 1 | # Source-build as a part of the Arcade build 2 | 3 | Source-build does a lot of stuff. A sampler: 4 | 5 | 1. Clones the upstream repos to form a layout. 6 | 1. Sets up package cache and source-built intermediate output directories. 7 | 1. Orchestrates build order. 8 | 1. Provides a number of source-build-specific args to build commands. 9 | 1. Runs extra targets before and after each build to fill in missing 10 | source-build compatibility, if necessary. 11 | 1. Builds tarballs--`.tar.gz` file containing everything necessary to build a 12 | specific version of the SDK from source without prebuilts. 13 | 1. Maintains scripts specific to the tarballs to enable scenario like 14 | bootstrapping. 15 | 1. Track prebuilt usage to ensure repos don't pull in dependencies not built 16 | from source. 17 | 1. Automatically decompile and recompile certain dependencies used as prebuilts 18 | so that they can be maintained as source code even if we don't have the 19 | original source code. 20 | 21 | The main doc discusses why this needs to be done in Arcade. We should also aim 22 | at all of this being doable via a single `./build.sh [...]` command. This allows 23 | source-build to be easily reproducible with a local build, something the project 24 | has been sorely missing and bars all but the most committed and awesome 25 | contributors. 26 | 27 | ## Additions to the `build.sh` procedure 28 | 29 | It seems that source-build can fit into the existing Arcade extension points. 30 | 31 | ### Tool Restore 32 | 33 | This point is normally used to restore tooling that may impact the ordinary 34 | restore by adding MSBuild targets. Source-build can add a step to get 35 | source-build dependencies, with several options: 36 | 37 | 1. Use transport package. Restores the [intermediate nupkg] and sets it up as a 38 | restore source. 39 | 2. Bring-your-own dependencies. A set of args allow the user (or automation) to 40 | point to existing intermediates on disk. 41 | 3. Build dependencies from source. Instead of downloading intermediates, clone 42 | the repos and build them all. 43 | 44 | Any of those options allows the source-build of the current repo to continue 45 | afterwards with source-built intermediates. Then, source-build sets up 46 | RestoreSources, a Package Version Props, a blob root, and any repo-specific 47 | special requirements. 48 | 49 | At this point in the build, we should also have all we need to create a 50 | source-buildable tarball. We should aim to allow the user to create it as early 51 | as possible and cancel the rest of the build, as they may be a distro maintainer 52 | who doesn't actually want to build the repo right now. Unfortunately, there are 53 | some known issues blocking us from creating a source-buildable tarball without 54 | building the product first, tracked by 55 | [source-build#831](https://github.com/dotnet/source-build/issues/831). Until we 56 | find solutions, tarball creation must be done in Publish instead. 57 | 58 | ### Restore 59 | 60 | We need to be careful that no machine-wide caches are used here. Caching a 61 | source-built nupkg may cause unexpected and incredibly hard to detect errors 62 | later on. 63 | 64 | ### Build/Pack 65 | 66 | The slew of source-build args normally passed to the build command should be 67 | moved to Arcade as much as possible. Many of these apply to the Build and Pack 68 | parts of the build process. 69 | 70 | ### Publish 71 | 72 | We should add a simple target to Publish that ensures no artifacts are published 73 | directly, only the [intermediate nupkg] wrapper. This prevents NuGet feed 74 | pollution/conflict vs. the non-source-built packages. 75 | 76 | ### New "AfterSourceBuild" project 77 | 78 | We need a place to put [intermediate nupkg] creation and prebuilt usage 79 | detection. The intermediate nupkg should contain the prebuilt report, so these 80 | steps are related. 81 | 82 | It's important to put prebuilt usage detection as the very last step, to be sure 83 | no prebuilts were used, even by Publish. Not all package restores and artifact 84 | downloads happen during the Arcade SDK's Restore phase. 85 | 86 | Creating an intermediate nupkg involves gathering together all shipped artifacts 87 | (nupkgs and tarballs) into a nupkg. The nupkg the needs to be uploaded to the 88 | build pipeline along with a manifest to indicate to Arcade how to publish it. 89 | 90 | ## Supporting source-built SDKs 91 | 92 | We need to build using a source-built Arcade SDK. For bootstrap and N-1 flows, 93 | we also need to use a source-built .NET Core SDK. We can get a source-built SDK 94 | in an [intermediate nupkg]. The problem is that by the time we've restored the 95 | intermediate nupkg, we're already inside a process using a prebuilt SDK, so we 96 | need to launch a new build. A few alternative ways how to do this (best first): 97 | 98 | 1. Reclone the current repo somewhere in `artifacts/`, then run `build.sh` with 99 | extra parameters pointing to the restored SDK. 100 | * This seems like a complex way to do it, however it ensures a clean repo and 101 | keeps the logic inside MSBuild code. 102 | * We already need to be able to clone upstreams in `artifacts` for the "build 103 | dependencies from source" option. This can leverage that infrastructure. 104 | The `darc clone` command provided by Arcade should fulfill this role. 105 | * It may not be necessary to create a fresh clone: this can be investigated 106 | as a perf optimization. 107 | 2. Have the user first call `./build.sh [...]` with some args that bring down 108 | the new SDK. Then the user calls a second command to use that SDK. 109 | * Not bad in official builds. However, harder to reproduce locally. 110 | * We could have an additional script to automate the two calls. This is 111 | better. We may still need to worry about cleanup logic between the first 112 | and second call. It is still a departure from the norm of just calling 113 | build.sh once. 114 | 3. Add functionality to `/eng/common/tool.sh` directly to support intermediate 115 | nupkgs before entering any SDK code. 116 | * Reinvents a lot of NuGet caching/download/restore logic. 117 | * Needs to be implemented in tool.sh and tool.ps1 for parity. 118 | 119 | Option (1) has been implemented in the Arcade SDK, using `git` commands. 120 | [dotnet/arcade#5116](https://github.com/dotnet/arcade/issues/5116) 121 | 122 | ## Special per-repo handling 123 | 124 | Some repos may have some infra in source-build that is incompatible with arcade. 125 | Any functionality like this should be added to `eng/SourceBuild.targets`. 126 | 127 | ## Non-Microsoft-maintained repositories 128 | 129 | Repos not maintained by Microsoft or the .NET Foundation likely don't use the 130 | Arcade toolset. The simple approach is to fork the repo and add the Arcade 131 | wrapper to allow it to seamlessly take part in the .NET Core SDK source-build. 132 | 133 | A "soft" way to fork is to maintain a repo with a single submodule pointing to 134 | the non-Microsoft repo. This wraps it without forking the original repo's 135 | history. The intent would be very clear. However, it would be more difficult to 136 | maintain patches, if any are necessary to add Arcade. 137 | 138 | [intermediate nupkg]: intermediate-nupkg.md 139 | -------------------------------------------------------------------------------- /Documentation/planning/arcade-powered-source-build/incremental-official-chunked.md: -------------------------------------------------------------------------------- 1 | # Chunked approach to including source-build in official builds 2 | 3 | Instead of adding source-build to all official builds from the bottom up, it is 4 | possible to separate the repos into chunks. Each chunk is several repos with one 5 | "builder" repo that builds the entire chunk from source. This has some 6 | significant restrictions: 7 | 8 | * The builder must use all repos in the chunk as upstreams. 9 | * The builder needs to know which version to build, so it needs a dependency. 10 | * The repos in a chunk need to be coherent. 11 | * The builder won't build multiple versions of the same repo, so all 12 | references to repos within the chunk need to be on the same version. 13 | * When a downstream repo takes a dependency on any repo in the chunk, it must 14 | have a coherent parent dependency on the builder. 15 | * If the downstream repo takes a dependency on a repo that isn't part of a 16 | coherent build, source-build intermediates won't be available. 17 | 18 | These coherency and upstream restrictions apply to the Microsoft build as well, 19 | because it shares the same dependency graph. This will slow down dependency flow 20 | in some cases. For example, a new package may need to flow through several repos 21 | to maintain coherency rather than being uptaken immediately by a downstream 22 | repo, due to the fake dependencies put in place to make the chunks work. 23 | 24 | ## Suggestion: avoid it 25 | 26 | A chunked approach *could* be a backup if it ends up being extremely costly to 27 | implement source-build in every official build. This is not expected. 28 | -------------------------------------------------------------------------------- /Documentation/planning/arcade-powered-source-build/incremental-official.md: -------------------------------------------------------------------------------- 1 | # Incremental improvements to source-build in official builds 2 | 3 | We can add official source-build to repos one at a time. This incrementally 4 | creates a tree of cached builds. 5 | 6 | ## Dependency leaves 7 | 8 | The following diagram is the source-build intermediate output flow in 3.1, with 9 | leaf nodes colored gold and the Core-SDK (dotnet/installer) product at the top: 10 | 11 | > ![Source build leaves](img/official-leaves.svg) 12 | > [source (img/official-leaves.dot)](img/official-leaves.dot) 13 | 14 | To build up the tree of cached builds, source-build must be distributed amongst 15 | the repos. We should do this by repeatedly adding source-build functionality to 16 | leaf repos (removing completed repos from the graph) until every repo runs 17 | source-build. 18 | 19 | > Note: the .NET Core build does have circular dependencies, so technically no 20 | > repos are pure leaves. For example, compiling C# in the leaves above requires 21 | > a .NET Core SDK toolset. Source-build cuts the dependencies as shown above and 22 | > uses a bootstrap flow to avoid prebuilts. The bootstrap flow is out of the 23 | > scope of this incremental official build doc. See ["The speculative SDK" in 24 | > README.md](README.md#the-speculative-sdk). 25 | 26 | ## Leaf first 27 | 28 | Reasons to implement ArPow in leaves first: 29 | 30 | * This most closely resembles an actual source-build. Either the repo has no 31 | dependencies, or it gets all its dependencies from upstream ArPow builds. 32 | * If there are differences in the source-built artifacts from the upstreams, 33 | they are exercised. 34 | * Fewer or no prebuilts are used. 35 | 36 | ## Exceptions to leaf-first 37 | 38 | Some repos have very high impact to source-build and may introduce unexpected 39 | problems, such as dotnet/runtime. It is useful to implement ArPow in these repos 40 | even if upstreams aren't ready yet. For example, working on dotnet/runtime 41 | before it was a leaf exposed issues with artifact size. 42 | 43 | This makes sense for dotnet/runtime, but not for dotnet/aspnetcore. The behavior 44 | of aspnetcore depends a lot on the non-portable source-build RID, and needs 45 | artifacts to be available specifically for that RID. Building aspnetcore for 46 | `centos.7-x64` requires runtime artifacts with that non-portable RID, which are 47 | only built by source-build. 48 | -------------------------------------------------------------------------------- /Documentation/planning/arcade-powered-source-build/intermediate-nupkg.md: -------------------------------------------------------------------------------- 1 | # The source-build intermediate NuGet package 2 | 3 | To transfer source-built prebuilts from one repo to the next, we need a 4 | transport format. NuGet packages are a proven method to do so, including 5 | built-in caching, secure restore and push supported by Arcade, and versioning. 6 | 7 | ## The monster package 8 | 9 | We can't publish individual source-built NuGet packages (nupkgs) to the feed 10 | directly. They would conflict with Microsoft-built packages with the same IDs 11 | and versions. These packages also have a reasonable risk of accidentally being 12 | used by devs not familiar with source-build, poisoning the NuGet caches on their 13 | systems. 14 | 15 | To address both of these, source-build should assemble a monster nupkg that 16 | contains all the nupkgs and other artifacts produced by the build. 17 | 18 | * `source-build.dotnet-runtime.linux-x64.5.0.0-alpha.1.12345.nupkg/` 19 | * `Microsoft.NETCore.App.Ref.5.0.0-alpha.1.12345.nupkg` 20 | * `Microsoft.NETCore.App.Runtime.linux-x64.5.0.0-alpha.1.12345.nupkg` 21 | * `dotnet-runtime-5.0.0-alpha.1.12345-linux-x64.tar.gz` 22 | * ... 23 | 24 | The version number here is the same as the official build's version number. This 25 | makes the official build produce consistent artifacts and prevents overlap in 26 | the same way it's prevented in existing official builds. The nupkg should always 27 | have an unstable version because it's a transport package, helping to avoid 28 | complicated feed management allowing mutation for stabilized rebuilds. 29 | 30 | Official builds publish the monster nupkg, and downstream repos restore the 31 | monster nupkg then use the contents. Note that if an official build fails to 32 | produce its Microsoft build outputs *or* the source-build monster nupkg, it must 33 | fail altogether. This ensures downstream repos have uninterrupted access to the 34 | source-built intermediates. 35 | 36 | ## Too large? 37 | 38 | Some repositories produce artifacts that would be too large for Azure DevOps 39 | feeds to handle if put into a single intermediate package. To solve this, some 40 | repos need to produce "supplemental intermediate nupkgs". 41 | 42 | A "supplemental intermediate nupkg" uses the same naming convention as 43 | intermediate nupkgs, but inserts an extra name between the repo name and the 44 | RID. 45 | 46 | * `Microsoft.SourceBuild.Intermediate.runtime.Libraries.linux.x64/6.0.0-foo` 47 | * Contains the dotnet/runtime platform extensions libraries. 48 | * E.g. `System.IO.Pipelines.6.0.0-foo.nupkg`, 49 | `System.Numerics.Tensors.6.0.0-foo.nupkg`, ... 50 | * `Microsoft.SourceBuild.Intermediate.runtime.Crossgen2Pack.linux.x64/6.0.0-foo` 51 | * Contains the crossgen2 framework pack: 52 | * `Microsoft.NETCore.App.Crossgen2.linux-x64.6.0.0-foo.nupkg` 53 | * `Microsoft.SourceBuild.Intermediate.runtime.Crossgen2Pack.Archive.linux.x64/6.0.0-foo` 54 | * This stores the `tar.gz` version of the crossgen2 framework pack: 55 | * `dotnet-crossgen2-6.0.0-foo-linux-x64.tar.gz` 56 | * Note: Calling it `.Archive` rather than `.Tarball` gives us room to handle 57 | `zip` files, to allow for Windows support in the future. 58 | * Crossgen2 is huge (~167 MB), so we need one supplemental nupkg for each 59 | artifact: one for the nupkg, another for the archive. 60 | 61 | The names of the supplemental categories are based on the contents, but the 62 | exact names are arbitrary. The goal is that it's clear enough that downstream 63 | users will know what it contains. 64 | 65 | For example, it would be fine to give the `Crossgen2Pack.Archive` supplemental 66 | nupkg a name based on the artifact's filename, say, `dotnet-crossgen2`. 67 | Arcade-powered source-build doesn't require a specific naming style. In this 68 | case, making `Crossgen2Pack` and `Crossgen2Pack.Archive` similar seems 69 | reasonable, since these are simply the same bits packaged different ways. Naming 70 | is ultimately up to the project owner. 71 | 72 | In general, the name should be concise: long NuGet package IDs can cause path 73 | length problems. 74 | 75 | ### Contents 76 | 77 | Each nupkg should either be < 20 MB, or contain only one artifact. This is a 78 | heuristic to use to decide when to produce supplemental nupkgs and how to split 79 | them up. 80 | 81 | The non-supplemental intermediate nupkg contains: 82 | 83 | * The prebuilt report. 84 | * Any artifacts that are < 20 MB total and that all downstreams use. 85 | * [New!] A list of supplemental intermediate nupkg names produced by the repo. 86 | * This shouldn't be read by any tooling. (Avoid complicated implementation.) 87 | * This makes it easier to figure out what supplemental nupkgs to include when 88 | working on a downstream repo. Just restore the main one and look at the list 89 | to see what to add. 90 | 91 | The supplemental intermediate nupkgs only contain one or more artifacts--no 92 | reports, and no list of supplemental nupkg names. The single *non-supplemental* 93 | nupkg is always restored if *any supplemental* nupkg is restored, so the 94 | report/metadata is always available to someone investigating a downstream build 95 | that uses supplemental nupkgs. There is no need to deliver redundant data. 96 | 97 | The supplemental packages of dotnet/runtime will have approximately these 98 | download sizes: 99 | 100 | Name | Download size (approx) 101 | -- | -- 102 | RefPack | 1 MB 103 | RefPack.Archive | 1 MB 104 | RuntimePack | 107 MB 105 | RuntimePack.Archive | 93 MB 106 | HostPack | 47 MB 107 | HostPack.Archive | 47 MB 108 | Crossgen2Pack | 167 MB 109 | Crossgen2Pack.Archive | 167 MB 110 | Libraries | 6 MB 111 | CoreCLR (ilasm/ildasm/testhost) | 6 MB 112 | 113 | For context, some other repositories' intermediate nupkg total download size 114 | that do not need any supplemental nupkgs: 115 | 116 | * Arcade's intermediate nupkg is ~12 MB. 117 | * SourceLink's intermediate nupkg is < 1 MB. 118 | * xliff-tasks's intermediate nupkg is < 100 kB. 119 | 120 | ### Alternatives to manually split supplemental nupkgs 121 | 122 | There are a few alternate solutions that aren't suitable: 123 | 124 | * Split just enough for the nupkg to fit on Azure DevOps, and always restore 125 | every supplemental package. 126 | * This is direct workaround for the immediate problem: Azure DevOps has a 500 127 | MB limit on nupkg size (as of writing), and our intermediate nupkg for 128 | dotnet/runtime is ~1 GB. 129 | * This simplifies the design: the supplemental packages don't need meaningful 130 | names, and there are no extra lines needed in `eng/Version.Details.xml`. 131 | * This creates significant concern for CI performance, because every build 132 | downstream of dotnet/runtime will have to download all of that data, whether 133 | or not it's needed. 134 | * We have an opportunity to avoid a significant perf regression by rejecting 135 | this bare minimum approach. 136 | 137 | * Split artifacts into one artifact per package. 138 | * This absolutely minimizes downloading artifacts that aren't needed. 139 | * This will either: 140 | * Significantly bloat the `eng/Version.Details.xml` file, specifying every 141 | single required artifact. This makes it less maintainable and more brittle 142 | to upstream changes. 143 | * Require much more complicated intermediate nupkg restore logic to figure 144 | out which supplemental intermediate nupkgs are necessary for the build. 145 | * This doesn't seem like it has major benefits vs. establishing a reasonable 146 | splitting heuristic. This is rejected to keep arcade-powered source-build 147 | simple. 148 | 149 | ## Addressability with different build args? 150 | 151 | With a generic build cache, it's typically important for the build inputs to be 152 | captured and incorporated in how the cache is addressed later. The build and its 153 | artifacts must be deterministic, or else the distro maintainer ends up with 154 | something different and we've failed to do useful validation. 155 | 156 | However, the source-build intermediate packages are not a generic build cache. A 157 | given official build version number is *never* built twice by Microsoft, let 158 | alone with unpredictably varying arguments. That immutability is automatically 159 | shared by the source-build official build. 160 | -------------------------------------------------------------------------------- /Documentation/planning/arcade-powered-source-build/source-build-in-pipeline.md: -------------------------------------------------------------------------------- 1 | # Source-build's place in a build pipeline 2 | 3 | For each repo, source-build must occur in the same official build pipeline that 4 | produces the Microsoft product. This is needed to guarantee the version is the 5 | same and source-built intermediates are available for uptake by downstream 6 | repos. 7 | 8 | ## Official build 9 | 10 | > ![Official builds](img/rough-pipeline-official.png) 11 | > Blue: Existing 12 | > Black: Source-build additions. 13 | 14 | In official builds, it is important for source-build to be reliable, and not 15 | extend the build time. This means only the necessary steps are performed to 16 | produce the source-build intermediates: minimal to no validation. 17 | 18 | The source-build job produces artifacts that are published by the Arcade stages, 19 | like any other artifacts. This will be facilitated by yaml templates in Arcade, 20 | but may require action for each repo to add. 21 | 22 | The official build also generates source-buildable tarballs for distro 23 | maintainers, for quick and easy access. These are only needed for the Core-SDK 24 | repo in particular. A full SDK is necessary to be buildable from source via the 25 | bootstrap flow, so there is no significant benefit to producing tarballs in 26 | other official builds. 27 | 28 | ## PR validation and rolling builds 29 | 30 | > ![Pipeline validation](img/rough-pipeline-validate.png) 31 | > Blue: Existing. 32 | > Black: Basic source-build. Always validate. 33 | > Green: Extra validation steps (necessary coverage per repo varies). 34 | 35 | There are a variety of extra ways to validate source-build. For example, 36 | speculatively building an SDK with the current PR's changes (see 37 | [speculative-build.md](speculative-build.md)), testing the bootstrap flow, etc. 38 | 39 | Depending on the value each type of coverage adds to a specific repo, the extra 40 | validation steps need to be configured. They may change depending on the risk of 41 | certain breaks per repo, the state of the release cycle, the set of changed 42 | files, or manual contributor request. For example, a `master` branch is likely 43 | to hit API breaks frequently and not work speculatively, but a servicing branch 44 | may find value in always confirming there are no such breaks. 45 | 46 | For rolling builds and PR validation, there may be extra pipelines to run these 47 | steps, to avoid interfering with dev workflows. 48 | -------------------------------------------------------------------------------- /Documentation/planning/arcade-powered-source-build/speculative-build.md: -------------------------------------------------------------------------------- 1 | # Speculative build: what would an SDK look like with my changes? 2 | 3 | As a dev working on a repo upstream from the SDK, it would be desirable to be 4 | able to build a full SDK that incorporates my changes to see if the SDK's 5 | behavior is changed as expected. It is also convenient to test changes in a full 6 | end-to-end environment. 7 | 8 | This has additional uses in the context of source-build. In order to build from 9 | source, we need to be able to bootstrap: build a .NET Core SDK, then build the 10 | entire product again using that .NET Core SDK. Without speculatively building an 11 | SDK, bootstrapping can only be tested in the Core-SDK repository. In some cases, 12 | this is too late to actually fix any problems that are encountered. We need to 13 | push validation for this scenario as far upstream as we can. 14 | 15 | Speculative builds are complex. They have been considered before for the 16 | Microsoft build, but haven't been done. It may be significantly easier to do in 17 | source-build, because the entire build can happen on a single machine, rather 18 | than every supported platform (requiring transporting artifacts across 19 | machines). 20 | 21 | ## How to orchestrate a speculative build 22 | 23 | > ![Speculative build](img/spec-build-1-general.png) 24 | 25 | The red circle is the current build. For example, dotnet/aspnetcore on `master`. 26 | The black circles are the builds that were used for intermediates. 27 | 28 | First, we need to choose a speculative root build. We choose Core-SDK because it 29 | produces the SDK, and the latest version because it's most likely what we want 30 | to test against. This results in the blue graph. Note that because we have to 31 | make a choice here, there are many possible speculative builds we could perform, 32 | and they might not all work! 33 | 34 | Then, we replace all the nodes in the speculative root graph with our local 35 | build graph nodes, based on repository identity. This forms the third graph in 36 | the diagram. 37 | 38 | Finally, we rebuild the remaining nodes in the speculative graph, using our 39 | local build outputs. This gives us a speculative SDK as an output. 40 | 41 | ## Validation: speculative bootstrap 42 | 43 | > ![Speculative bootstrap](img/spec-build-2-bootstrap.png) 44 | > (The big arrow means *from* and/or *to* many nodes.) 45 | 46 | We can use the output SDK and intermediates from our bootstrap build to try to 47 | build the product again, using the same set of repositories/commits. If this is 48 | successful, we have validated the bootstrap build scenario. 49 | 50 | ## Validation: speculative N-1 51 | 52 | > ![Speculative N-1](img/spec-build-3-n-1.png) 53 | > (The big arrow means *from* and/or *to* many nodes.) 54 | 55 | If we instead use the output SDK and intermediates from an N-1 build, instead of 56 | building the local speculative graph, we validate the N-1 scenario. 57 | -------------------------------------------------------------------------------- /Documentation/planning/multi-sdk-band-support.md: -------------------------------------------------------------------------------- 1 | # Source-Build Support for Multiple SDKs 2 | 3 | This document serves as the design planning document for how .NET source build 4 | will support multiple SDK feature bands. More generally, this support could be 5 | described as "partial VMR support". This is the implementation plan for 6 | [Managing SDK Bands Unified Build 7 | feature](https://github.com/dotnet/arcade/blob/main/Documentation/UnifiedBuild/VMR-Managing-SDK-Bands.md). 8 | 9 | ## Terminology for this document 10 | 11 | - **VMR** - Virtual Monolithic Repository - A projection of the sources of a set 12 | of input repo and branch combinations onto another repository. 13 | - **SDK Feature Bands** - SDK functionality, especially around tooling, may 14 | evolve signficantly during the servicing lifetime of a .NET major release. 15 | This evolution is primarily to support new Visual Studio scenarios. To support 16 | this, .NET uses a series of SDK "bands" that correspond to VS releases and the 17 | .NET runtime GA release. They are denoted by diferences in the first digit of 18 | the patch numbers of the .NET SDK. All patch numbers have 3 digits. Fore more 19 | information, please see [official 20 | documentation](https://learn.microsoft.com/en-us/dotnet/core/releases-and-support#feature-bands-sdk-only). 21 | - **Component** - A logical collection of sources in a VMR, mapped from an input 22 | source repo, which build together to produce a set of binaries that relate in 23 | functionality. Examples include 24 | [runtime](https://github.com/dotnet/dotnet/tree/main/src/runtime), 25 | [roslyn](https://github.com/dotnet/dotnet/tree/main/src/roslyn) or 26 | [sdk](https://github.com/dotnet/dotnet/tree/main/src/sdk). 27 | - **Previously source-built (PSB)** - Artifacts from a previous 28 | servicing/preview iteration, or from a bootstrap build. These may be used when 29 | building the current dotnet VMR branch, but may not be bundled into any 30 | output. 31 | 32 | ## Overview and General Approach 33 | 34 | The general approach to supporting more than one SDK band is to **not** view the 35 | build of additional SDK feature bands as being any different from any other 36 | source build. Each build of the product combines a set of inputs (previously 37 | source-built artifacts + source) to produce a set of outputs that can ship to 38 | customers. When building two completely different major versions of .NET, the 39 | set of inputs is different (e.g. different source, 8.0 SDK vs. 9.0 SDK). Some of 40 | the previously source-built artifacts come from .NET, some may come from the 41 | source-built package ecosystem (e.g. icu or clang/llvm). If a repo were to be 42 | eliminated from the input sources, the binaries previously built from those 43 | sources would now need to come from previously-source built binaries. 44 | 45 | The primary restriction on building multiple SDK feature bands is that they must 46 | not differ in the shared runtimes. For N bands, there must be a single runtime 47 | artifact. Thus, we can view a source-build of N SDK feature bands as: 48 | 49 | - As a single build of one runtime, producing a set of source-built outputs and 50 | **then** 51 | - Building N SDK feature bands **using** 52 | - The *outputs* of the runtime build combined with additional previously 53 | source-built artifacts (e.g. a matching band SDK, required tools and 54 | libraries) and source, to produce N SDK feature bands that meet source-build 55 | requirements. 56 | 57 | The runtime build requires a 1xx band SDK to build **and** redistributes 58 | functionality from tooling that ships in the 1xx band (e.g. it has dependencies 59 | on roslyn and roslyn-analyzers). Furthermore, most changes to a .NET major 60 | version occur when only one SDK band exists. Therefore it makes practical sense 61 | at this point to include the source of the 1xx SDK and the shared runtime in the 62 | same dotnet VMR branch, and to produce all artifacts related to 1xx, including 63 | the runtime, in its build. *Newer bands will be branches of the dotnet VMR that 64 | contain **only** the components that **differ** from the 1xx band.* Only those 65 | components that differ. When building the newer band, the outputs of the 1xx 66 | build would be fed into the newer band build, in a similar manner as previously 67 | source built artifacts are today. If the source for a component does not exist 68 | in the dotnet VMR, it is simply not built, and obviously no outputs would be 69 | produced for that component. Thus any downstream dependencies would be forced to 70 | use the versions provided by input source built artifacts, typically from the 71 | 1xx source build. If no such version exists, then the input would fail to be 72 | found or restored from online sources and be reported as a prebuilt. 73 | 74 | This approach fits naturally into the source-build methodology. Generally, 75 | additional SDK feature bands may be represented as "subsetted dotnet VMRs", 76 | since use of a subsetted dotnet VMR is not restricted to SDKs only. 77 | 78 | ### Conceptual splitting of 'previously source-built' artifact types 79 | 80 | Today, the term "previously source-built artifacts' refers to those artifacts 81 | that were built in a previous product build iteration, typically from the 82 | previous month. For example, they would refer to the outputs of 8.0.1's build, 83 | when building 8.0.2. When building a subsetted dotnet VMR, two sets of input 84 | sets of source-built artifacts are required. While similar in some ways, they 85 | should be kept distinct. They are referred to as: 86 | 87 | - **Previously source-built (PSB)** - Artifacts from a previous 88 | servicing/preview iteration, or from a bootstrap build. These may be used when 89 | building the current dotnet VMR branch, but may not be bundled into any 90 | output. 91 | - **Current source-built (CSB)** - Artifacts just built, e.g. from a 1xx dotnet 92 | VMR build. These may be used **and** bundled into the build outputs, since 93 | logically they are from the same source-build iteration. 94 | 95 | ### Supported input artifact combinations and validation thereof 96 | 97 | Not all toolsets can be used to build all dotnet VMR branches. SDK feature bands 98 | are driven by VS development, which moves forward between .NET releases. Those 99 | components of the .NET SDK introduce new functionality, which may require 100 | support from the SDK. Prior to the release of the corresponding SDK feature 101 | band, components of newer SDKs ocassionally take dependencies on SDK features 102 | introduced in the same band, and ocassionally deficiences in older SDK bands may 103 | require SDK upgrades. Therefore, it is possible that the 1xx SDK will not 104 | successfully build *all* Nxx+ bands from the same .NET major version. This may 105 | mean a bootstrap of the initial release of a new non-1xx SDK band. This is an 106 | unfortunate reality. 107 | 108 | To set expectations and reduce potential distro maintainer pain, the following 109 | statements are made: 110 | 111 | - The N.0.1xx dotnet VMR branch containing the runtime (1xx) is only gauranteed 112 | to build with the 1xx SDK feature band. 113 | - A SDK feature band (e.g. 2xx) *may* be buildable with an older band, typically 114 | the 1xx band. However this is not gauranteed. E.g. `8.0.200` may be buildable 115 | with `8.0.103`. 116 | - A newer SDK feature band (e.g. 2xx) will *always* be able to build itself. 117 | E.g. `8.0.200` will be able to build `8.0.200`. `8.0.105` can build `8.0.105`. 118 | - A new release of an SDK feature band will *always* be buildable using the 119 | previous release of the same feature band. E.g. `8.0.201` will be buildable 120 | with `8.0.200`, and `8.0.102` will be buildable with `8.0.101`. 121 | 122 | From these statements. we can deduce that the initial release of a newer band 123 | (n00) may require a bootstrap, or may be buildable using an older band. 124 | 125 | To avoid confusion, Microsoft will: 126 | 127 | - Document the set of supported build combinations, including required inputs 128 | for all active SDK feature bands. This provides a guide for distro 129 | maintainers. 130 | - Validate those supported build combinations. 131 | - Validate 'stage2' builds (that a new SDK can build itself). 132 | 133 | ## Changes required for the source-build infrastructure 134 | 135 | This section details changes to the source build infrastructure to support 136 | building of subsetted dotnet VMRs. 137 | 138 | ### Scripting changes 139 | 140 | Since conceptually CSB and PSB inputs are different, the scripting should 141 | reflect this. Furthermore, since NuGet and non-NuGet archives are treated 142 | similarly within the source-build infrastructure, we should take this 143 | opportunity to make changes to the names of existing switches: 144 | 145 | - `--with-packages` becomes `--with-previous-artifacts`, reflecting that it 146 | represents only previously source built artifacts that may be used in the 147 | build, but not redistributed with outputs. When poisoning is enabled, this set 148 | is poisoned. `--with-previous-artifacts` should be viewed as functionally 149 | equivalent to `--with-packages` 150 | - `--with-current-artifacts` - This switch is added. It points to a directory 151 | that contains input NuGet and non-NuGet assets that may be redistributed in 152 | the current build. Typically this set would come from the 1xx build. 153 | 154 | In addition, we should add a script that prepares the input artifacts for the 155 | build of a particular dotnet VMR branch, given a set of input directories from 156 | other builds/other previously source built artifacts. See 157 | [below](#what-artifacts-should-be-passed-with---with-current-artifacts--with-previous-artifacts) 158 | for an explanation of the input artifacts. 159 | 160 | #### Improvements to input artifact handling 161 | 162 | `--with-previous-packages`/`--with-packages` takes an input directory. With this 163 | effort, the infrastructure should be changed to allow 164 | `--with-previous-artifacts` and `--with-current-artifacts` to take a tarball as 165 | input, in addition to a directory. The build scripting would extract a tarball 166 | to a temporary location. 167 | 168 | ### Changes to PackageVersion.props flow 169 | 170 | When determining what version of a package or asset that a component should 171 | build against, there are 3 distinct sets in source-build today, listed in order 172 | of precedence from highest to lowest: 173 | 174 | - 1. A version just produced by a component built earlier in the build (if 175 | available). 176 | - 1. A version in the previously source-built packages (if available). 177 | - 1. The version in the checked in `Versions.props` for a repo. 178 | 179 | With the addition of a new input set (CSB), we now have an additional set. 180 | Changes should be made such that a PackageVersions.props file is generated for 181 | this new set, and is imported in the correct place based on its precedence. 182 | 183 | - 1. A version just produced by a component built earlier in the build (if 184 | available). 185 | - 1. A version in the current source-built packages (if available). 186 | - 1. A version in the previously source-built packages (if available). 187 | - 1. The version in the checked in `Versions.props` for a repo. 188 | 189 | ### Filtering builds of components 190 | 191 | The dotnet VMR defines a set of projects (under `repo-projects`) that correspond 192 | to each component of the build. These projects also define a build dependency 193 | graph of these components via the `RepositoryReference` ItemGroup. *Note: there 194 | are a few projects that do not correspond directly to a component, but instead 195 | serve as general join points in the build graph.*. Source build invokes a root 196 | build (`dotnet.proj`), which evaluates the dependency graph and invokes the 197 | MSBuild task on dependent projects. To avoid maintenance complexity, we should 198 | avoid altering this dependency graph or removing project files when a component 199 | should not be built. Instead, it should simply be filtered out and ignored as if 200 | the node in the graph was inactive. 201 | 202 | When filtering out components, we do not want to build a filtered subproject at 203 | all. `AfterTargets=Build`, `BeforeTargets=Build` are common in both component 204 | projects as well as the shared targets. We do not want these invoked at all, or 205 | else a large variety of special casing per-target will be required. Instead, 206 | each individual project should determine whether it should be built by setting a 207 | property. By default, this property would be set based on the whether the source 208 | for the component exists. Therefore, a user has two options for explicitly 209 | excluding or including a project: 210 | 211 | - Explicitly setting the property value in the project file 212 | - Removing the source code altogether from the branch. 213 | 214 | ```xml 215 | // Default 216 | // repo-projects/Directory.Build.props 217 | true 218 | false 219 | 220 | // Explicit setting 221 | // repo-projects/package-source-build.proj 222 | false 223 | ``` 224 | 225 | Given this set of properties, a new target can be added that will determine the 226 | active dependency projects (those `RepositoryReference` items that should be 227 | built) given the original set of a particular project's dependencies. Then, only 228 | active dependency projects would be built. Example code from the 229 | Proof-of-Concept is shown below. 230 | 231 | ```xml 232 | 233 | 234 | <_AllDependencyProjects Include="@(RepositoryReference -> '%(Identity).proj')"> 235 | %(Identity) 236 | 237 | 238 | 239 | 243 | 245 | 246 | 247 | 248 | 249 | 250 | %(RepositoryName) 251 | 252 | 253 | 254 | 255 | 258 | 259 | 262 | 263 | 264 | 265 | ``` 266 | 267 | ### Choosing the correct source-built MSBuild SDKs 268 | 269 | There is one wrinkle to the simple graph filtering approach. Most .NET projects 270 | have dependencies on the arcade toolset and associated MSBuild SDK. This 271 | dependency is handled specially by source-build as it helps bootstrap a repo's 272 | build process. There are three potential arcade versions available: 273 | 274 | - The bootstrap arcade version coming in via the previously source-built 275 | packages + artifacts (PSB). 276 | - The bootstrap arcade version coming in via the current source-built packages + 277 | artifacts (CSB). 278 | - The arcade built during the current source build invocation. 279 | 280 | The 'bootstrap' version of arcade is the version of arcade present in CSB, or if 281 | not present, in PSB. Repositories built **before** arcade (and arcade itself) is 282 | built must choose the bootstrap arcade and know its location and version. Today, 283 | repos explicitly choose this via the `UseBootstrapArcade` property, which causes 284 | a series of environment variables to be set up for the Arcade SDK. If arcade is 285 | filtered from the graph, however, then a repo must **always** use the bootstrap 286 | arcade version. *In this case, the arcade should have been produced by the 1xx 287 | band build.* To achieve this, the UseBootstrapArcade property must be 288 | dynamically set, since the graph is dynamically filtered, based on whether 289 | arcade appears in the filtered input dependency projects. Then, the appropriate 290 | environment variables like `_InitializeToolset` would be set in a target based 291 | on its value. 292 | 293 | Example code from the PoC: 294 | 295 | ```xml 296 | // From SetSourceBuiltSdkOverrides target in repo-projects/Directory.Build.targets. 297 | 298 | 299 | 300 | 301 | 302 | true 303 | 304 | 305 | 306 | 307 | 308 | 309 | 310 | ``` 311 | 312 | ***Note: Today it is necessary to explicitly set UseBootstrapArcade for those 313 | projects where arcade is not available (arcade, 314 | source-build-reference-packages). It may be possible to remove this requirement, 315 | simply relying on the repo dependency graph to give the correct answer.*** 316 | 317 | ### Input asset locations 318 | 319 | The input assets required for the .NET build come in two primary forms: 320 | 321 | - NuGet assets 322 | - Non-NuGet assets (typically `.zip` or `.tar.gz` archives) 323 | 324 | Source build handles NuGet assets using a set of local NuGet feeds (which are 325 | just directories of NuGet packages). It handles non-Nuget assets by using one of 326 | these local NuGet feeds as a shared storage location where repos producing 327 | archives (e.g. runtime, sdk, aspnetcore) place their outputs (`blob-feed` dir). 328 | Downstream repos that need these archives copy the files from that location. 329 | They locate specific file names based on set naming patterns combined with 330 | package input versions. For instance, the installer repo might find the runtime 331 | archive for linux-x64 by looking for `dotnet-runtime--linux-x64.tar.gz`. When building a subsetted dotnet VMR, just as the 333 | shared location may not contain all the input packages, it may not contain all 334 | of the input archives. For NuGet packages, this is handled reasonably 335 | transparently. Given a set of potential input feeds, NuGet simply checks each 336 | one. So if arcade wasn't built and version 1.2.3 isn't available in the shared 337 | `blob-feed` dir, NuGet will look in the previously source-built artifacts feed. 338 | 339 | Because CSBs are separated from PSBs when building a subsetted dotnet VMR, 340 | source-build needs to be altered to add new CSB location as a NuGet input to 341 | each component build. In addition, builds of subsetted dotnet VMRs need a way to 342 | locate non-NuGet artifacts from the CSBs, without mixing them into the shared 343 | `blob-feed` where component builds currently look. Repositories should be 344 | altered to accept an additional location where they may find input artifacts. 345 | This has precedent, as it is the same logic used in installer, aspnetcore, 346 | windowsdesktop, and others in Microsoft's non-source-build scenarios. These 347 | repos will check a public official source for binaries first, then a public 348 | build artifacts location, and finally an internal build artifacts location, if 349 | the appropriate credentials are available. 350 | 351 | For example, `installer.proj` might add the following logic. dotnet/installer 352 | then uses `AdditionalBaseUrl` when available, adding it as a location for 353 | downloading assets: 354 | 355 | ```xml 356 | $(BuildCommandArgs) /p:AdditionalBaseUrl=file:%2F%2F$(AdditionalInputPackagesPath) 357 | ``` 358 | 359 | In the above example, `AdditionalInputPackagesPath` is the current source-built 360 | packages (CSB) path (provided by `--with-current-assets`). This path is also 361 | added as a NuGet feed for each component. 362 | 363 | ### Gathering assets for delivery to customers 364 | 365 | To gather assets for delivery to customers, the outputs present in each of the 366 | `artifacts//` (e.g. `artifacts/x64/Release`) should be used. For 367 | each build, the artifacts present in the output directory are only those 368 | artifacts that were produced in that build. 369 | 370 | ## Changes to distro maintainer workflow 371 | 372 | A distro maintainer wishing to support a newer band, either in addition to or in 373 | place of an older band, would see some change in workflow. First off, it is 374 | important to remember that the 1xx band will still be required. This is for two 375 | reasons: 376 | 377 | - The 1xx SDK is the only SDK that is supported for building the runtime 378 | components. Because tooling differs between the bands, using a newer band to 379 | build the runtime is likely to expose new warnings or issues. 380 | - There are components of the 1xx band that are used in the runtime. There are 381 | "sdk-like" components present in the runtime (analyzers, generators, etc.) 382 | that are dependent upon "sdk" functionality. Substituting in a 2xx aligned 383 | roslyn, for instance, would result in a product that differs from Microsoft's 384 | build. 385 | 386 | ### What artifacts should be passed with --with-current-artifacts/--with-previous-artifacts? 387 | 388 | When building a full dotnet VMR branch (no skipped components), only 389 | --with-previous-artifacts should be passed. When building a subsetted dotnet VMR 390 | branch (e.g. 2xx, 3xx, etc.), both `--with-current-artifacts` and 391 | `--with-previous-artifacts` should be passed. 392 | 393 | `--with-current-artifacts` - The NuGet and non-NuGet archive outputs of the 1xx 394 | dotnet VMR branch build that is logically part of the same product iteration. 395 | `--with-previous-artifacts` - The NuGet outputs of the previous dotnet VMR 396 | branch build(s) that are logically part of the previous product iteration. This 397 | set may encompass more than one set of build outputs. 398 | 399 | For an illustration of artifact flow, please see the 400 | [Appendix](#conceptual-diagram-of-source-build-artifact-flow) 401 | 402 | ### Previews and major release GA builds 403 | 404 | A distro maintainer sees no change here. Until a few months **after** GA, only 405 | the 1xx band ships. The workflow remains the same as today. Clone the dotnet VMR 406 | branch/tag in question, build using previously source-built artifacts and SDKs. 407 | 408 | ### A distro maintainer wishing to only support the 1xx SDK 409 | 410 | A distro maintainer sees no change here. 411 | 412 | ### A distro maintainer wishing to build the initial release (n00) of a Nxx+ band SDK 413 | 414 | The initial release of a Nxx+ band SDK should be straightforward. It is unlikely 415 | (though possible) that the initial Nxx band would depend on a Nxx SDK to build. 416 | If this is not the case, then a distro maintainer performs the following 417 | actions, and no bootstrap will be required: 418 | 419 | 1. **Checkout** 1xx branch: 420 | 2. **Build** using `./build.sh --with-sdk 421 | --with-previous-artifacts `. 422 | 3. **Gather** 1xx build outputs from `artifacts/x64/Release` = ``. 424 | 4. **Checkout** Nxx branch 425 | 5. **Build** using `./build.sh --with-sdk 426 | --with-previous-artifacts --with-current-artifacts ` 428 | 429 | If a Nxx SDK is required to build the initial Nxx SDK, then the following 430 | workflow is used: 431 | 432 | 1. **Checkout** 1xx branch: 433 | 2. **Build** using `./build.sh --with-sdk 434 | --with-previous-artifacts `. 435 | 3. **Gather** 1xx build outputs from `artifacts/x64/Release` = ``. 437 | 4. **Checkout** Nxx branch 438 | 5. **Prep** using `./prep.sh` 439 | 6. **Build** with bootstrap SDK using `./build.sh --with-previous-artifacts --with-current-artifacts ` 441 | 7. **Gather** Nxx bootstrap inputs - ` + ` 442 | = `` 443 | 8. **Build** using `./build.sh --with-sdk --with-current-artifacts 444 | --with-previous-artifacts ` 445 | 446 | ### A distro maintainer wishing to support non-1xx SDKs (monthly workflow) 447 | 448 | A distro maintainer who to support the Nxx (N>1) SDK(s) still needs to build the 449 | 1xx branch to obtain the runtime and an updated 1xx SDK for use the next month. 450 | Each non 1xx SDK is built with its corresponding SDK 451 | 452 | 1. **Checkout** 1xx branch: 453 | 2. **Build** using `./build.sh --with-sdk 454 | --with-previous-artifacts `. 455 | 3. **Gather** 1xx build outputs from `artifacts/x64/Release` = ``. 457 | 458 | The following additional steps may be done in parallel. For each SDK band Nxx: 459 | 460 | 1. **Gather** Previous Nxx inputs - ` + ` 461 | = `` 462 | 2. **Checkout** Nxx branch 463 | 3. **Build** using `./build.sh --with-sdk 464 | --with-previous-artifacts --with-current-artifacts ` 466 | 467 | ## Poison and Prebuilt detection 468 | 469 | Poisoning and prebuilt workflows may need slight tweaks. 470 | 471 | - The poisoning infrastructure will not poison CSB inputs, but should continue 472 | to poison PSBs. 473 | - Prebuilt detection should not view CSB inputs as prebuilts. 474 | 475 | ## Potential issues and mitigations 476 | 477 | There are some potential issues that may require tweaks to this approach, source 478 | fixes, additional guidance, etc. 479 | 480 | ### A toolset input to a newer band does not work with the PSB from that same band 481 | 482 | When preparing a newer band's input artifacts, there would be up to 3 versions 483 | of some components. For example: 484 | 485 | - PSB 1xx Microsoft.NET.Compilers.Toolset @ 4.7.1 486 | - CSB 1xx Microsoft.NET.Compilers.Toolset @ 4.7.2 487 | - PSB 2xx Microsoft.NET.Compilers.Toolset @ 4.8.0 488 | 489 | In this scenario, the *newest* version of Microsoft.NET.Compilers.Toolset will 490 | get used as input to any repo taking a dependency on it, before the 2xx 491 | dotnet/roslyn is built. If this particular repo has an explicit dependency on 492 | 4.7.x, then transparently updating to 4.8.x might cause a break. In the current 493 | source build infrastructure, the only way to avoid that transparent upgrade 494 | would be to not use the PSB 2xx inputs with the 2xx SDK build. But of course, 495 | the 2xx inputs would be required to provide supporting packages that the SDK 496 | depends on. 497 | 498 | If this scenario ends up becoming a common problem, I think we could alter the 499 | source build PVP infrastructure to be smarter about version selection, 500 | attempting to match on best "MAJOR.MINOR" when multiple input versions are 501 | available. 502 | 503 | ## Appendix 504 | 505 | ### Conceptual diagram of source-build artifact flow 506 | 507 | Below is an illustration of source-build currently uses various types of input 508 | artifacts: 509 | 510 | ```mermaid 511 | flowchart TD 512 | previouslysourcebuilt[("previousy source built (--with-packages/prep.sh)")] 513 | currentsourcebuilt[("current live source built (built in current build invocation)")] 514 | startbuild["start build"]-.->arcade["Arcade build"]-.->aspnetcore["aspnetcore build"]-.->sdk["sdk build"]-.->preparefinaloutputs["gather source built artifacts"]-.->finishbuid["finish build"] 515 | previouslysourcebuilt-->arcade 516 | previouslysourcebuilt-->aspnetcore 517 | previouslysourcebuilt-->sdk 518 | currentsourcebuilt-->aspnetcore 519 | currentsourcebuilt-->sdk 520 | arcade-->currentsourcebuilt 521 | aspnetcore-->currentsourcebuilt 522 | sdk-->currentsourcebuilt 523 | currentsourcebuilt-->preparefinaloutputs 524 | 525 | subgraph versionselection["version selection"] 526 | current["current source built version"]-->|overrides|previous["previously source built version"]-->|overrides|repoCheckedInDefault["repo checked in eng/Versions.props"] 527 | end 528 | ``` 529 | 530 | What this proposal is doing is extending this concept of another set of 531 | "current" artifacts that come from another build invocation: 532 | 533 | ```mermaid 534 | flowchart TD 535 | previouslysourcebuilt[("previousy source built (--with-packages/prep.sh)")] 536 | currentlivesourcebuilt[("current source built (built in current build invocation)")] 537 | currentsourcebuilt[("current source built (--with-current-packages)")] 538 | startbuild["start build"]-.->roslyn["roslyn build"]-.->sdk["sdk build"]-.->preparefinaloutputs["gather source built artifacts"]-.->finishbuid["finish build"] 539 | previouslysourcebuilt-->roslyn 540 | previouslysourcebuilt-->sdk 541 | currentsourcebuilt-->roslyn 542 | currentlivesourcebuilt-->sdk 543 | currentsourcebuilt-->sdk 544 | roslyn-->currentlivesourcebuilt 545 | sdk-->currentlivesourcebuilt 546 | currentlivesourcebuilt-->preparefinaloutputs 547 | 548 | subgraph versionselection["version selection"] 549 | currentlive["current live source built version"]-->|overrides|current["current source built version"]-->|overrides|previous["previously source built version"]-->|overrides|repoCheckedInDefault["repo checked in eng/Versions.props"] 550 | end 551 | ``` 552 | -------------------------------------------------------------------------------- /Documentation/poison-report-format.md: -------------------------------------------------------------------------------- 1 | # Interpreting the poison report 2 | 3 | This is an explanation of the poison report and its interpretation. For an 4 | overview of poisoning, [see this document](leak-detection.md). The poison 5 | report looks something like this: 6 | 7 | ```xml 8 | 9 | 10 | Hash 11 | 9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08 12 | 13 | 14 | 15 | AssemblyAttribute 16 | cf80cd8aed482d5d1527d7dc72fceff84e6326592848447d2dc0b0e87dfc9a90 17 | 18 | 19 | 3961408c6033157f4c612b56247b35c3ad28dac0b26f95a6fe55315f440a48eb 20 | SourceBuildReferenceAssembly 21 | 22 | 23 | NupkgFile 24 | e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 25 | 26 | 27 | ``` 28 | 29 | The first thing to notice is the different `Type`s. These can help you track 30 | down where the leak came from. One match can have multiple types, e.g. 31 | `AssemblyAttribute,Hash`. 32 | 33 | ## Failure types 34 | 35 | ### AssemblyAttribute 36 | 37 | This means that we detected a custom attribute that source-build adds to each 38 | prebuilt and previously-source-built assembly before the build. This is a 39 | *definite* leak - there will not be any false positives. However, this may not 40 | be cause for failure - some categories of assemblies may leak without causing 41 | problems: 42 | 43 | - Localization assemblies, e.g. .../langcode/Assembly.Resources.dll 44 | - Reference assemblies (these should not ship, but also will very rarely cause 45 | problems). 46 | - Test or resource assemblies - these should be removed. 47 | - N-1 (previously-source-built) assemblies included in shipping packages to 48 | break cycles - this is expected behavior. Usually this use case is covered by 49 | reference packages but in rare cases we need to actually run an N-1 binary in 50 | the build. 51 | 52 | ### SourceBuildReferenceAssembly 53 | 54 | This means that a reference assembly originating from the 55 | source-build-reference-packages repo was leaked. 56 | 57 | ### Hash 58 | 59 | Hash failures mean that a file matches the catalog of hashes that we take of 60 | existing files before the build. These are commonly false positives due to 61 | shipping static files not produced by the build such as: 62 | 63 | - Text files (licenses, templates, etc) 64 | - Empty files (\_.\_) 65 | 66 | However, if this is a DLL this is probably a legitimate leak. 67 | 68 | ### NupkgFile 69 | 70 | This type of failure means that a nupkg was copied wholesale to the final 71 | output. This is a definite leak, but very uncommon and likely easy to trace in 72 | the binlogs. 73 | 74 | ## Other metadata 75 | 76 | ### \ 77 | 78 | This is the location in the output where the leak was detected. The first 79 | component of the path is an archive or package name, e.g. `dotnet-sdk-6.0.100` 80 | is `dotnet-sdk-6.0.100.tar.gz` in the `artifacts/` directory. Components 81 | after that are the file's path within the archive or nupkg. 82 | 83 | ### \ 84 | 85 | This is the hash of the failing file. This is useful to track down the correct 86 | file if there are multiple with the same name. It is included in all cases, 87 | even when the failure type is not `Hash`. 88 | 89 | ### \ 90 | 91 | When available, this describes the file in the source-build *input* that matches 92 | the failing file. For instance: `` means that the 96 | failing `System.Text.Encoding.CodePages.dll` matches the hash of the file 97 | `lib/netcoreapp3.1/System.Text.Encoding.CodePages.dll` in the 98 | previously-source-built package System.Text.Encoding version 1.2.3. This can 99 | help track down the exact *input* file that made it to the final output. 100 | -------------------------------------------------------------------------------- /Documentation/sourcebuild-in-repos/README.md: -------------------------------------------------------------------------------- 1 | # Repo owner's handbook for source-build 2 | 3 | These docs aim to be a collection of useful information for repo owners who work 4 | with and participate in source-build. 5 | 6 | ## Index 7 | 8 | * [Adding new dependencies](new-dependencies.md) 9 | * [Updating dependencies](update-dependencies.md) 10 | * [Considerations when adding features](adding-features.md) 11 | * [General notes on the source-build build](build-info.md) 12 | * [Adding new repositories](new-repo.md) 13 | -------------------------------------------------------------------------------- /Documentation/sourcebuild-in-repos/adding-features.md: -------------------------------------------------------------------------------- 1 | # New features 2 | 3 | New features are great! They also have source-build considerations though. .NET 4 | is no longer just multi-platform but also multi-distribution: there's a Linux 5 | SDK produced by Microsoft, but there's also a Linux SDK produced by Red Hat, one 6 | produced by Fedora, and one that anyone in the community can build himself from 7 | source. 8 | 9 | ## Things to consider 10 | 11 | New features, or expansions of current features, should act sensibly across 12 | Windows, Linux, and OSX. This also involves taking into account the limitations 13 | and conventions of the different platforms - for instance, on Linux, it's 14 | typical for the .NET SDK to be installed by root and therefore be unchangeable 15 | by normal users, so installing global tools needs to take into account the fact 16 | that the user might not be able to add anything to the SDK directories (see 17 | [this docker global tools 18 | issue](https://github.com/dotnet/dotnet-docker/issues/520) and [a similar 19 | installer issue](https://github.com/dotnet/installer/issues/7069)). 20 | 21 | New features also need to be compatible across all distributions of .NET - for 22 | instance, Fedora and Debian cannot distribute *any* non-open-source code, even 23 | samples, tests, or minor additions to licenses like restricting the field that 24 | the code can be used in. This includes any code or packages that are used to 25 | build product code as well. Microsoft generally prefers the MIT license. 26 | 27 | One example of a licensing issue that source-build found was with dotnet-check 28 | and dotnet-format: these tools were brought into the product build but had a 29 | dependency that was not licensed compatibly - even without the source-build 30 | aspect. We had to scramble to replace the dependency before the release. 31 | Dependencies need to be carefully checked for license compatibility as well. 32 | 33 | ## Resources 34 | 35 | Fedora's approved open-source licenses can be found [on their 36 | wiki](https://fedoraproject.org/wiki/Licensing:Main#Good_Licenses), or you can 37 | check on the [OSI-approved list of 38 | licenses](https://opensource.org/licenses/alphabetical). 39 | 40 | If you would like some of our distribution maintainer partners to review your 41 | new feature you can ping @dotnet/distro-maintainers on a PR or issue. 42 | -------------------------------------------------------------------------------- /Documentation/sourcebuild-in-repos/build-info.md: -------------------------------------------------------------------------------- 1 | # Source-build build info 2 | 3 | This is a collection of notes about how source-build can differ in general from 4 | your repo's build and what kind of issues that can create. 5 | 6 | ## Single-version and single-RID build 7 | 8 | Source-build is required to build on a single machine with no internet access. 9 | This means that we build targeting a single RID, usually the non-portable RID 10 | for the build machines (like rhel.7-x64). We do support building portable 11 | (linux-x64) as well - this is useful for bootstrapping new distributions. 12 | 13 | Source-build cannot build with any *prebuilts*. This is our term for any 14 | package that comes from *outside the current source-build*. This means that 15 | everything that ships out of source-build and everything that is used to build 16 | those shipping products must come from the source-build in progress. Packages 17 | from nuget.org, Microsoft builds, or other unrelated source-builds cannot be 18 | used in source-build except in a limited bootstrapping process. 19 | 20 | Source-build supplies a *previously-source-built* set of packages for this 21 | bootstrapping process. This is one way we have of breaking cycles in the build. 22 | However, none of these packages can make it to the final build output. This also 23 | means that your repo should be buildable with the immediately previous version 24 | of the SDK than you are building for; i.e., if you are building for 8.0.103, 25 | everything should be buildable with the 8.0.102 SDK. 26 | 27 | We also only build one version of each repo. This means that if your repo turns 28 | production of some packages on and off, for instance, if you only produce 29 | packages if they are changed, source-build will need a workaround to force all 30 | packages to be produced. Additionally, we can only supply one version of each 31 | package to a repo. This is injected into the `$({PackageName}PackageVersion)` 32 | variables, e.g. SystemReflectionMetadataPackageVersion. One exception is 33 | reference-only packages - 34 | [dotnet/source-build-reference-packages](https://github.com/dotnet/source-build-reference-packages) 35 | produces multiple versions and these can be hard-coded or use a 36 | `$(ReferenceVersion)` property or similar if you don't need 37 | source-build to change them. 38 | 39 | ## Platform-specific packages 40 | 41 | Packages that require components or packages only available on some other 42 | operating system than the building OS cannot be built in source-build. These 43 | should use `true` or other 44 | options to be excluded from the source-build. For instance, if a project 45 | depends on a package that can only be built on Windows, it will need to be 46 | disabled or worked around in source-build. As an example, [Roslyn 47 | removes](https://github.com/dotnet/roslyn/blob/b999a65c8b0feeccb2b58da3d7a6e80e5f08feab/src/Workspaces/Core/Portable/Storage/PersistentStorageExtensions.cs#L23) 48 | a small performance improvement when building for source-build because it 49 | requires a component that isn't available. 50 | -------------------------------------------------------------------------------- /Documentation/sourcebuild-in-repos/img/ci-job.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/dotnet/source-build/6bd71d2e5cc898d224b05f737c25b6f6b68c5c50/Documentation/sourcebuild-in-repos/img/ci-job.png -------------------------------------------------------------------------------- /Documentation/sourcebuild-in-repos/new-dependencies.md: -------------------------------------------------------------------------------- 1 | # Adding a new dependency 2 | 3 | ## Basics 4 | 5 | When adding a new dependency, there are a few steps common between any type of 6 | dependency. If you use darc, `darc add-dependency` will take care of this 7 | process for you. Otherwise: 8 | 9 | 1. Add the package and a default version to eng/Versions.props. The default 10 | version property should be named `PackageVersion`, e.g. 11 | `SystemCollectionsImmutablePackageVersion`. 12 | 1. Add the package and a repo reference to eng/Version.Details.xml. This should 13 | include a SourceBuild metadata entry, looking something like: 14 | 15 | ```xml 16 | 19 | https://github.com/dotnet/roslyn 20 | 0167599e0e1634ea3ed8d0e41390a3c0d9b3e4e9 21 | 22 | 23 | ``` 24 | 25 | 1. Set up dependency flow from the foreign repo to your repo. 26 | 27 | In general, you should aim to use one version of each package. If you are using 28 | a package as reference-only, it is possible to use multiple versions, but only 29 | one implementation version of each package will be used - source-build will 30 | override it to the version that is being built in this version of the SDK. 31 | 32 | The source-build metadata is important - this tells source-build which repo 33 | package contains the specific nupkg you want. 34 | 35 | Another uncommon case that can cause problems is when the repo `` 36 | version and the source-built intermediate package version don't match. In this 37 | case, a direct dependency should be added to the source-build intermediate 38 | package and the CoherentParentDependency attribute should be set to the repo 39 | that consumes the dependency. For an example, see [installer's F# 40 | dependency](https://github.com/dotnet/installer/blob/ba1739a2363b1062f03ea386ec67174c6468d3b2/eng/Version.Details.xml#L128). 41 | You can find the version needed by running `darc get-build` or using 42 | [BAR](https://aka.ms/bar). 43 | 44 | ## Adding dependencies 45 | 46 | Source build classifies dependencies in the following ways 47 | 48 | 1. .NET - a dependency on a component from the .NET org - e.g. dotnet/runtime 49 | 1. Microsoft - a dependency on a component from the Microsoft org - e.g. 50 | microsoft/vstest 51 | 1. External - a dependency on a component outside of Microsoft/.NET - e.g. 52 | JamesNK/Newtonsoft.Json 53 | 54 | The following checklist can be used to determine how to handle each type of 55 | dependency and the nuances it may have. 56 | 57 | 1. Are you already using a package from the same repo and is the new dependency 58 | already source-built? You can determine this by checking if the repo has a 59 | SourceBuild leg building, e.g. runtime's is `runtime-dev-innerloop (Build 60 | Linux x64 release SourceBuild)`. 61 | 62 | 1. This is the simplest case. You can add new dependencies from repos you 63 | are already using freely. Note that you need to use the same version for 64 | all packages from the repo. 65 | 1. Is the repo already built in source-build including the specific package you 66 | want? 67 | 1. Add the dependency using [`darc 68 | add-dependency`](https://github.com/dotnet/arcade/blob/main/Documentation/Darc.md#add-dependency) 69 | as normal, then add the [source-build metadata](#basics) as above. 70 | 1. Is the repo already built in source-build but the specific package is not? 71 | 1. There's probably an issue with source-building this package. Please talk 72 | to a [source-build team 73 | member](https://github.com/orgs/dotnet/teams/source-build) about 74 | why that is and whether we can fix it. 75 | 1. Is this a repo that uses Arcade to build? 76 | 1. Does the foreign repo depend on your repo, directly or indirectly? i.e. 77 | would adding the dependency create a cycle? 78 | 1. This isn't necessarily a deal-breaker - it can sometimes be worked 79 | around with reference-only packages. Please contact a [source-build 80 | team 81 | member](https://github.com/orgs/dotnet/teams/source-build) to 82 | discuss. 83 | 1. Does the foreign repo publish to BAR? 84 | 1. If not, please contact them to get them publishing to BAR in an 85 | appropriate channel. 86 | 1. If neither of these caveats apply you should be in good shape. Follow the 87 | instructions under "Basics" above. 88 | 1. Dependencies that have no code (e.g. SDKs with just props and targets) can 89 | usually be added using the [source-build text-only-package 90 | process](https://github.com/dotnet/source-build-reference-packages/tree/main/src/textOnlyPackages/src). 91 | If the package is not already included there, you can [open a 92 | PR](https://github.com/dotnet/source-build-reference-packages/pulls) or [file 93 | an issue](https://github.com/dotnet/source-build/issues/new/choose) to include 94 | it. 95 | 1. Source-build has in the past used Arcade shims to allow non-Arcade repos to 96 | build appropriate packages for source-build. Please [log an 97 | issue](https://github.com/dotnet/source-build/issues/new/choose) to determine 98 | if this is a workable approach for your foreign repo. 99 | 1. We build some external dependencies in the 100 | [dotnet/source-build-externals](https://github.com/dotnet/source-build-externals) 101 | repo. Good targets for this generally have very few if any dependencies and 102 | very simple build processes. [Please log an 103 | issue](https://github.com/dotnet/source-build/issues/new/choose) to get the 104 | process started. 105 | 106 | ## Deciding what version to use 107 | 108 | - If you are using the package as reference-only and want the version to be 109 | pinned, use a literal version number in the csprojs that reference the 110 | project. You can also set up a reference-only package version variable in 111 | eng/Versions.props, for instance 112 | `1.2.3` in addition to 113 | `4.5.6`. Also verify 114 | that the package is available in 115 | [source-build-reference-packages](https://github.com/dotnet/source-build-reference-packages), 116 | and if not, [file a source-build 117 | issue](https://github.com/dotnet/source-build/issues). 118 | - If you are using the package in the actual build or want the version to be 119 | updated whenever the foreign repo publishes to your channel, use the version 120 | number property set up in eng/Versions.props. When performing a source-build, 121 | the version number will get updated to the current source-built version. 122 | - If you are using an external or non-Arcade package, please coordinate as much 123 | as possible with other teams using that package. Each package-version is 124 | essentially maintained as a separate concern, so something like repo A 125 | requiring Newtonsoft.Json 9.0.1 and repo B requiring 12.0.2 essentially 126 | doubles the source-build team's work. 127 | -------------------------------------------------------------------------------- /Documentation/sourcebuild-in-repos/new-repo.md: -------------------------------------------------------------------------------- 1 | # Adding a new repository to source build 2 | 3 | This document describes the overall process of onboarding new repos onto source 4 | build. 5 | 6 | 1. [Source Build Configuration](#source-build-configuration) 7 | 1. [Setup CI](#setup-ci) 8 | 1. [Source build repos and the VMR](#source-build-repos-and-the-vmr) 9 | 1. [Validate](#validate) 10 | 11 | ## Source Build Configuration 12 | 13 | ### Trying it out locally 14 | 15 | If a repo passes build options through to the common arcade build script, a 16 | source build can be triggered via the following command. 17 | 18 | ```bash 19 | ./build.sh -sb 20 | ``` 21 | 22 | > Note: [source build is not supported on 23 | Windows](https://github.com/dotnet/source-build/issues/1190), only Linux and 24 | macOS. 25 | 26 | ### Excluding components 27 | 28 | It is not always necessary or correct to include all repo components in source 29 | build. Components should be excluded if they are not required for the platform 30 | being source-built. Including them expands the source build graph and may not 31 | be possible because of licensing. Examples include tests, Windows components 32 | (remember source build currently only supports Linux and macOS), etc. To exlcude 33 | these components use the `DotNetBuildSourceOnly` msbuild property to 34 | conditionally exclude. 35 | 36 | ```code 37 | Condition="'$(DotNetBuildSourceOnly)' != 'true'" 38 | ``` 39 | 40 | See the [Unified Build Controls](https://github.com/dotnet/arcade/blob/main/Documentation/UnifiedBuild/Unified-Build-Controls.md) 41 | for more options on how to exclude components from source build. 42 | 43 | ## Setup CI 44 | 45 | Source build should run in PR validation builds, to prevent regressions. 46 | 47 | Source build CI can be activated with a single flag in the ordinary Arcade SDK 48 | jobs template for easy consumption. If a repo can't simply use the Arcade SDK 49 | jobs template, more granular templates are also available. 50 | 51 | See for the 52 | current template source code. The inline comments in the `parameters:` section 53 | in those files are the most up to date docs, maintained with higher priority 54 | than this general onboarding doc. 55 | 56 | ### `eng/common/templates/jobs/jobs.yml` opt-in switch 57 | 58 | The simplest way to onboard. This approach applies if the repo already uses the 59 | `eng/common/templates/jobs/jobs.yml` template. 60 | 61 | To opt in: 62 | 63 | 1. Set `enableSourceBuild: true` 64 | 65 | Set `enableSourceBuild: true` in the template parameters. 66 | 67 | This should look something like [this sourcelink 68 | implementation:](https://github.com/dotnet/sourcelink/blob/dfe619dc722be42d475595c755c958afe6177554/azure-pipelines.yml#L40) 69 | 70 | ```yaml 71 | stages: 72 | - stage: build 73 | displayName: Build 74 | jobs: 75 | - template: /eng/common/templates/jobs/jobs.yml 76 | parameters: 77 | enablePublishUsingPipelines: true 78 | enablePublishBuildArtifacts: true 79 | enablePublishBuildAssets: true 80 | enableSourceBuild: true 81 | artifacts: 82 | publish: 83 | artifacts: true 84 | manifests: true 85 | ``` 86 | 87 | 1. Specify platforms (if repo is not managed-only) 88 | 89 | A repo is managed-only if `eng/SourceBuild.props` contains 90 | `true`. If this is true, 91 | this step is not necessary. Otherwise, specify `sourceBuildParameters` in 92 | the `jobs.yml` template's parameters like this: 93 | 94 | ```yaml 95 | sourceBuildParameters: 96 | platforms: 97 | - name: 'CentosStream8_Portable' 98 | container: 'mcr.microsoft.com/dotnet-buildtools/prereqs:centos-stream8' 99 | - name: 'CentosStream8' 100 | nonPortable: true 101 | container: 'mcr.microsoft.com/dotnet-buildtools/prereqs:centos-stream8' 102 | ``` 103 | 104 | #### End result 105 | 106 | Submit the changes above in a PR and include 107 | [@source-build](https://github.com/orgs/dotnet/teams/source-build) 108 | as a reviewer. The jobs (or job, if managed-only) are automatically be added to 109 | CI in the existing pipeline alongside existing jobs, with a name like `Build 110 | Source-Build ()`: 111 | 112 | ![Build jobs image with source build leg](img/ci-job.png) 113 | 114 | Once this PR works, run a mock official build (AKA "validation build") in your 115 | official build definition. The usual workflow is to push a 116 | `dev//` branch to the AzDO repo and then queue a build 117 | on that branch. This makes sure that merging the PR won't immediately break the 118 | official build: `enableSourceBuild: true` does add job(s) to the official build, 119 | not just PR validation. 120 | 121 | If the PR is green, but merging it produces red PR or Official builds, 122 | immediately let the source build team know about the failure and revert the 123 | source build PR to unblock dev work. 124 | 125 | ### Advanced: granular templates 126 | 127 | If the repo doesn't use the basic Arcade jobs template, or has advanced job 128 | templating infra built on top of the Arcade jobs template, the simple 129 | `enableSourceBuild` flag might not work out. There are a few more granular 130 | templates to use in this case. 131 | 132 | Look at the documentation in each YAML file itself to figure out how to use it 133 | properly, and if it fits the scenario. This list is only an overview. 134 | 135 | #### `eng/common/templates/jobs/source-build.yml` 136 | 137 | This is one level deeper than `eng/common/templates/jobs/jobs.yml`. It is a 138 | `jobs` template that produces just the set of source build jobs based on the 139 | specified `platforms`. Or, just one job with the default platform, if 140 | managed-only. 141 | 142 | #### `eng/common/templates/job/source-build.yml` 143 | 144 | This template defines a single `job` that runs source build on a 145 | specifiedplatform. 146 | 147 | #### `eng/common/templates/steps/source-build.yml` 148 | 149 | This template defines the build `steps` for a single source build job. This is 150 | the most granular template, and may be useful if some repo-specific preamble or 151 | cleanup steps are required, or if the repo already has job matrix templates and 152 | this just happens to fit in nicely. 153 | 154 | ## Source build repos and the VMR 155 | 156 | Another effect of adding a new source build repository is that its sources will 157 | be synchronized into the [Virtual Monolithic Repository of 158 | .NET](https://github.com/dotnet/dotnet). The VMR is then where the official 159 | source build happens from. 160 | 161 | In order for the sources of the new repo to by synchronized into the VMR, the 162 | repo needs to be registered in the [`source-mappings.json` 163 | file](https://github.com/dotnet/dotnet/blob/main/src/source-mappings.json) which 164 | tells the tooling where from and which sources should be synchronized. Please 165 | open a PR in [`dotnet/dotnet`](https://github.com/dotnet/dotnet) and add 166 | your repository into `src/source-mappings.json`. 167 | 168 | ### Cloaking (filtering) the repository sources 169 | 170 | When creating the `source-mappings.json` record for the new repo, there is a 171 | possibility of filtering which sources get synchronized into the VMR. The VMR 172 | should only really contain plain text-based files as it is consumed by 3rd party 173 | .NET distro builders who do not accept any non-text sources (e.g. binaries). 174 | When registering the repository into the VMR, it is a good time to consider 175 | which files are required for it to build and only synchronize those. Commonly, 176 | repositories contain binaries that are required for testing or similar purposes. 177 | Files like these should not be synchronized into the VMR. Another common 178 | scenario is that the repo has multiple products/ship vehicles and only a subset 179 | is needed for the source-built .NET scenario. 180 | 181 | ## Validate 182 | 183 | Once the code flows to the VMR, the PR validation legs will ensure that no 184 | [prebuilts](https://github.com/dotnet/source-build/blob/main/Documentation/eliminating-pre-builts.md#what-is-a-prebuilt) 185 | were added to the system and everything is functioning correctly. 186 | If you need help on addressing any prebuilds, reach out to @source-build. 187 | -------------------------------------------------------------------------------- /Documentation/sourcebuild-in-repos/update-dependencies.md: -------------------------------------------------------------------------------- 1 | # Updating a dependency 2 | 3 | In most cases, dependency flow should automatically update the packages you 4 | depend on. There are some special considerations and things to keep in mind 5 | when doing manual updates. If you are manually updating a version, also see 6 | [the doc on new dependencies](new-dependencies.md) for special considerations. 7 | 8 | ## Internal packages 9 | 10 | If you are manually updating a package, please make sure it's from a compatible 11 | branch (e.g. runtime release/8.0 to sdk release/8.0.1xx). 12 | Package versions that you are updating to should be source-built in their 13 | respective repos. If the version you need is produced in a branch that is not 14 | yet source-build-compatible please let the [source-build 15 | team](https://github.com/orgs/dotnet/teams/source-build) know. 16 | 17 | Another common error we see is updating eng/Versions.props but not 18 | eng/Version.Details.xml. This causes problems for source-build because we 19 | depend on these files being updated in lockstep. Please prefer updating with 20 | Darc - it takes care of these issues - or, if a manual update really is 21 | necessary, make sure you update both files. 22 | 23 | Example Darc command: `darc update-dependencies --name MyCoolPackage -v 1.2.3` 24 | 25 | ## External packages 26 | 27 | Updating a non-Microsoft (e.g. Newtonsoft.Json 9.0.1 to 13.0.1) or non-Arcade 28 | (e.g. ApplicationInsights) package is typically equivalent to adding the new 29 | version all over again. Please [log an 30 | issue](https://github.com/dotnet/source-build/issues/new/choose) to discuss 31 | this. You can check if the external package you want is already included in 32 | source-build in the 33 | [source-build-externals](https://github.com/dotnet/source-build-externals) repo. 34 | 35 | ## Splitting, combining, and moving packages 36 | 37 | These operations often end up causing source-build issues. A typical case of 38 | what can happen is 39 | 40 | 1. Package A is produced in version 1. 41 | 1. In version 2, Package A is split into Package A1 and Package A2. 42 | 1. A downstream repo is never updated to take the split packages. 43 | 1. The version 2 source-build succeeds because Package A exists in the 44 | [previously-source-built archive](build-info.md#single-version-and-single-rid-build), 45 | but no new version of Package A is produced. 46 | 1. Source-build version 3 fails because Package A no longer exists in the 47 | previously-source-built archive either. 48 | 49 | So please keep in mind consumers of your packages and try to keep them informed 50 | of these kinds of changes. If you are the consuming repo, consider tracking PRs 51 | in the repos that produce your dependencies. 52 | -------------------------------------------------------------------------------- /Documentation/system-requirements.md: -------------------------------------------------------------------------------- 1 | # System Requirements to Source-Build 2 | 3 | This document provides the system requirements to source build the .NET SDK for 4 | a targeted platform. 5 | 6 | ## Operating System 7 | 8 | ### Linux 9 | 10 | * [Toolchain 11 | Setup](https://github.com/dotnet/runtime/blob/main/docs/workflow/requirements/linux-requirements.md#toolchain-setup) 12 | * [Preconfigured Container 13 | Images](https://github.com/dotnet/dotnet-buildtools-prereqs-docker) - These 14 | images are used by 15 | [CI](https://github.com/dotnet/dotnet/blob/main/src/sdk/eng/pipelines/templates/stages/vmr-build.yml) 16 | to build and test source-build. 17 | * [Distros Source Building 18 | .NET](https://github.com/dotnet/source-build#net-in-linux-distributions) 19 | 20 | ### MacOS 21 | 22 | MacOS is not currently supported: [Tracking 23 | Issue](https://github.com/dotnet/source-build/issues/2909). However, community 24 | users have created a [Homebrew 25 | project](https://github.com/Homebrew/homebrew-core/blob/master/Formula/d/dotnet.rb) 26 | to build .NET for OSX. Please feel free to open new issues in individual repos 27 | or in source-build for OSX issues. 28 | 29 | ### Windows 30 | 31 | Windows is not currently supported. [Tracking 32 | Issue](https://github.com/dotnet/source-build/issues/2910) 33 | 34 | ## Hardware 35 | 36 | ### Disk Space 37 | 38 | 80 GB of space is required for a typical build. You can reduced this down to ~30 39 | GB if you build with the `clean-while-building` option. This might increase over 40 | time, so consider this to be a minimum bar. 41 | 42 | ### Memory 43 | 44 | A minimum of 8 GB of memory is recommended. 45 | 46 | ### Architectures 47 | 48 | #### Officially Supported 49 | 50 | * ARM64 51 | * x64 52 | 53 | #### Community Supported 54 | 55 | * ARM32 56 | * s390x 57 | * ppc64le 58 | 59 | ## Network 60 | 61 | The following assets will need to be downloaded in order to build. 62 | 63 | * Source: ~300 MB 64 | * SDK: ~200 MB 65 | * Artifacts: ~1 GB 66 | -------------------------------------------------------------------------------- /LICENSE.txt: -------------------------------------------------------------------------------- 1 | The MIT License (MIT) 2 | 3 | Copyright (c) .NET Foundation and Contributors 4 | 5 | All rights reserved. 6 | 7 | Permission is hereby granted, free of charge, to any person obtaining a copy 8 | of this software and associated documentation files (the "Software"), to deal 9 | in the Software without restriction, including without limitation the rights 10 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 11 | copies of the Software, and to permit persons to whom the Software is 12 | furnished to do so, subject to the following conditions: 13 | 14 | The above copyright notice and this permission notice shall be included in all 15 | copies or substantial portions of the Software. 16 | 17 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 18 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 19 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 20 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 21 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 22 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 23 | SOFTWARE. 24 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # .NET Source-Build 2 | 3 | Please use [GitHub 4 | discussions](https://github.com/dotnet/source-build/discussions) to see 5 | announcements, ask questions, make suggestions, and share information with other 6 | members of the source-build community. 7 | 8 | This repo is the starting point for building .NET from source. It contains 9 | documentation, tools, and is used for issue tracking. 10 | 11 | ## Prerequisites 12 | 13 | * [Build system requirements](Documentation/system-requirements.md) 14 | 15 | ## Building .NET 16 | 17 | Follow the instructions in [dotnet/dotnet's 18 | README](https://github.com/dotnet/dotnet/blob/main/README.md#dev-instructions) 19 | to build .NET from source. 20 | 21 | ## Source-build goals 22 | 23 | There are two primary goals of the source-build effort: 24 | 25 | 1. Increase .NET adoption by focusing on making .NET available everywhere 26 | 27 | If .NET was available everywhere - including Linux distributions and package 28 | managers like Homebrew - as a first class citizen, it would make .NET a more 29 | attractive option for developers who might otherwise look at other languages 30 | or runtimes. Users would be more likely to start using and keep using .NET if 31 | .NET is available on their development and release platforms of choice. 32 | 33 | To achieve this, we try to make it easier for community and partner 34 | maintainers to build and release .NET for their platforms. We need to make 35 | sure source-build satisfies the official packaging rules of commonly used 36 | Linux distributions, such as 37 | [Fedora](https://fedoraproject.org/wiki/Packaging:Guidelines) and 38 | [Debian](https://www.debian.org/doc/manuals/maint-guide/build.en.html). Many 39 | Linux distributions have similar rules. These rules tend to have three main 40 | principles: 41 | 42 | * Limited or no network access 43 | * Consistent reproducibility 44 | * Source code for everything 45 | 46 | 2. Make maintenance of the .NET product easier 47 | 48 | The current way of making changes to .NET during a servicing release is to 49 | make changes to individual product repositories and then adjust the 50 | dependency versions to flow the changes to the next set of repositories. This 51 | is repeated until all the repositories are updated. If there's an issue 52 | discovered late in the release cycle, the fixes and the dependency updates 53 | need to be re-done quickly, which becomes difficult. It's also difficult to 54 | verify that the issue is fixed in the final product. It would be much easier 55 | to make and test product-wide changes if we could make atomic changes to one 56 | repository and be able to build the whole product based on that source code 57 | at once. 58 | 59 | In addition, getting source-build fully functional would provide everyone 60 | with a place to try changes that would otherwise require a lot of 61 | coordination between multiple repositories - such as landing features that 62 | require changes to both the runtime and the SDK. 63 | 64 | For more details about this Unified Build, see [this 65 | overview](https://github.com/dotnet/arcade/blob/main/Documentation/UnifiedBuild/Overview.md). 66 | 67 | Source-build can help achieve both these goals by making it easier for everyone 68 | to build and release the entire .NET product end-to-end. 69 | 70 | ## What does the source-build infrastructure do? 71 | 72 | Source-build solves common challenges that most developers encounter when trying 73 | to build the whole .NET SDK from source. 74 | 75 | * .NET is composed of many repositories that need to be built at a specific 76 | combination of commits. 77 | * Each repository's build output needs to flow into the next repository's build. 78 | * By default, most .NET repositories download prebuilt binary dependencies from 79 | online sources. These are forbidden by typical Linux distribution rules, and 80 | interfere with build output flow. 81 | * Some of the binary build dependencies are under a proprietary license, making 82 | it difficult to build everything without taking accidental dependencies on 83 | non-Open Source code. Many of our community members and partners want to build 84 | an Open Source .NET only. 85 | * Flowing the build output automatically also means we can make and test changes 86 | that require coordination across a number of repositories. 87 | * Nearly all .NET repositories require the .NET SDK to build. This is a circular 88 | dependency, which presents a bootstrapping problem, whether that's bringing up 89 | new architectures (eg, riscv), or support new operating systems (eg, FreeBSD). 90 | * Microsoft controls the SDKs used by `dotnet-install.sh` scripts and 91 | `Microsoft.*` and `runtime.*` nuget packages at nuget.org; source-build makes 92 | it possible for the community to bring up the platforms without Microsoft 93 | having to accept/bless things. 94 | 95 | ## Comprehensive Guidelines 96 | 97 | * [Bootstrapping new distro and architecture 98 | guidelines](Documentation/bootstrapping-guidelines.md) 99 | * [Distribution packaging 100 | guidelines](https://learn.microsoft.com/dotnet/core/distribution-packaging) 101 | 102 | ## .NET in Linux Distributions 103 | 104 | | Distro | Package Feed | Maintainer | 105 | |---|---|---| 106 | | Alpine | [Community](https://pkgs.alpinelinux.org/packages?name=dotnet*&branch=v3.16&repo=&arch=&maintainer=) | [@ayakael](https://github.com/ayakael) | 107 | | Arch Linux | [Community](https://archlinux.org/packages/?q=dotnet)
[Arch User Repo](https://aur.archlinux.org/packages?K=dotnet) | [@alucryd](https://github.com/alucryd) | 108 | | CentOS Stream | [CentOS Stream Mirror](http://mirror.stream.centos.org/9-stream/AppStream/x86_64/os/Packages/) | [@omajid](https://github.com/omajid) | 109 | | [Fedora](https://fedoraproject.org/wiki/DotNet) | [Default](https://packages.fedoraproject.org/search?query=dotnet) | [@omajid](https://github.com/omajid) | 110 | | Homebrew | [Formula](https://formulae.brew.sh/formula/dotnet) | [@asbjornu](https://github.com/asbjornu) | 111 | | [Red Hat Enterprise Linux](https://developers.redhat.com/products/dotnet/overview) | [Default](https://access.redhat.com/documentation/en-us/net/8.0) | [@omajid](https://github.com/omajid) | 112 | | [Ubuntu](https://canonical.com/blog/install-dotnet-on-ubuntu) | [Default](https://packages.ubuntu.com/search?suite=default§ion=all&arch=any&keywords=dotnet&searchon=names)
[Personal Package Archives](https://launchpad.net/ubuntu/+ppas?name_filter=dotnet) | [@mirespace](https://github.com/mirespace) | 113 | 114 | ## Support 115 | 116 | .NET source build supports the following: 117 | 118 | 1. Linux only, Windows and MacOS is not supported 119 | 1. 1xx SDK feature band only (e.g. 8.0.1xx) 120 | 121 | For the latest information about Source-Build support for new .NET versions, 122 | please check our [GitHub Discussions 123 | page](https://github.com/dotnet/source-build/discussions) for announcements. 124 | 125 | * [More information about .NET 126 | Versioning](https://docs.microsoft.com/en-us/dotnet/core/versions/) 127 | * [More information about .NET Support 128 | Policies](https://dotnet.microsoft.com/en-us/platform/support/policy/dotnet-core) 129 | 130 | ## License 131 | 132 | This repo is licensed under the [MIT](LICENSE.txt) license. 133 | --------------------------------------------------------------------------------