├── README.adoc ├── SUMMARY.adoc ├── big-picture.md ├── design-notes.md ├── git-how.md ├── projects-user.md └── pull-request.md /README.adoc: -------------------------------------------------------------------------------- 1 | = Draft FreeBSD Git docs 2 | 3 | :doctype: article 4 | :description: A first draft of FreeBSD Git docs 5 | :author: Warner Losh 6 | :email: imp@FreeBSD.org 7 | :source-highlighter: rouge 8 | :rouge-style: github 9 | :icons: font 10 | :toc: preamble 11 | 12 | 13 | These changes have been pushed into the FreeBSD tree. See https://docs.freebsd.org/en/articles/committers-guide/#git-primer[the committer's guide]. 14 | 15 | Here's the few reamaining that didn't fit in: 16 | 17 | * link:git-why.md[FreeBSD moving to Git: Why] 18 | * link:projects-user.md[A brief note about Subversion Projects and User branches] 19 | * link:pull-request.md[How to deal with Pull Requests] 20 | * link:big-picture.md[Big Picture] 21 | * link:design-notes.md[Design Notes] 22 | 23 | -------------------------------------------------------------------------------- /SUMMARY.adoc: -------------------------------------------------------------------------------- 1 | # Summary 2 | 3 | :doctype: article 4 | :description: A first draft of FreeBSD Git docs 5 | :author: Warner Losh 6 | :email: imp@FreeBSD.org 7 | :source-highlighter: rouge 8 | :rouge-style: github 9 | :icons: font 10 | :toc: preamble 11 | 12 | Most of this has been merged into https://docs.freebsd.org/en/articles/committers-guide/#git-primer[the committer's guide]. 13 | 14 | Here's the few reamaining that didn't fit in: 15 | 16 | * link:git-why.md[FreeBSD moving to Git: Why] 17 | * link:projects-user.md[A brief note about Subversion Projects and User branches] 18 | * link:pull-request.md[How to deal with Pull Requests] 19 | * link:big-picture.md[Big Picture] 20 | * link:design-notes.md[Design Notes] 21 | -------------------------------------------------------------------------------- /big-picture.md: -------------------------------------------------------------------------------- 1 | # Big Picture Overview of FreeBSD's git migration 2 | 3 | ## Schedule 4 | 5 | | Date | Event | 6 | |------|-------| 7 | | December 5th, 2020 | Doc repo transition started | 8 | | December 19th, 2020 | Src repo transition to start | 9 | | End of March, 2021 | Ports repo slated to transition to git | 10 | 11 | ## Main Design Points 12 | 13 | The first phase of FreeBSD's migration to git focuses on replacing 14 | Subversion with Git. This is a modest goal where the project's work 15 | flow will remain much as it is today, but with Subversion replaced 16 | with Git. The bug tracking (bugzilla) and code review (phabricator) 17 | will remain in place. We'll self-host the git repo and manage 18 | the first layer of mirrors ourselves. We'll publish the repository to 19 | github, gitlab and other places that are useful for our developers, 20 | users and contributors. 21 | 22 | The project will strive to maintain a linear history in the new src 23 | repo. Developers will be expected to land their changes as a logical 24 | series of commits, much as has become the norm in other projects. For 25 | the most part, developers are doing this today, so using a tool (git) 26 | that facilitates this will help increase compliance and improve code 27 | quality. 28 | 29 | MFCs will be done as cherry picks from the main repo, with some 30 | decoration added to the commit messages to help us track MFCs. Git 31 | provides most of what's needed to track cherry picks, but fails when 32 | the cherry picks are squashed as has been the project custom (since 33 | current is for new features to mature, stable should have those 34 | features committed all at once). The details of this will likely 35 | evolve in the months after the conversion. 36 | 37 | Vendor branches will be the one exception to the linear history 38 | requirement. The vendor branch in git is similar to how it was in 39 | subversion: a tree of files that are similar to the way they are 40 | distributed naively. All the merge commits from the old vendor trees 41 | have been converted to git branches. Just like with subversion, the 42 | vendor branches are not fetched by default, at least until they are 43 | bootstrapped. Like the conversion from CVS to Subversion, vendor 44 | branches will need to be bootstrapped the first time there's a new 45 | import. vendor branches are then merged to the main branch with `git 46 | subtree merge`. Since we're not using submodules, users of the tree 47 | need not install submodules, recursively grab them or put up with any 48 | of the logistical issues submodules bring to the table. 49 | 50 | Hashes changing from beta github tree. Due to a large number of bad 51 | translations that have happened when exporting from subversion to 52 | github, we're rerunning the conversion process. This will necessarily 53 | create new hashes for the entire tree. The errors that prompted this 54 | decision were such that every time we had to do anything with history, 55 | such as extract a program for its own repo, the errors would get in 56 | the way and create a lot of extra work. They also manifested 57 | themselves in other ways that caused confusion. Rather than put up 58 | with these quirks for all time, and waste resources forever on it, we 59 | decided to redo the conversion. This also allowed us to correct a 60 | few names that were wrong and fix other less important artifacts of 61 | the translation. Fortunately, git has a number of tools that allow 62 | easy transition from one stream of hashes to another. 63 | 64 | When we converted to Subversion, there were no good places to host 65 | collaborative work. The project filled the gap by creating two 66 | hierarchies in the subversion repository: the projects hierarchy for 67 | work that would likely be merged, and the user hierarchy for work that 68 | would not. Since that time, services like github, gitlab, sourceforge, codeberg, 69 | project.net, and others have sprung up to provide collaboration 70 | points. As such, in the first phase, we're converting the old projects 71 | and user branches. But they will become read only. And they won't be 72 | in the default revs that are fetched (you'll have to opt-in to get 73 | them). Over the past few years, traffic has dropped significantly in 74 | these branches as the work has already migrated to other places. Once 75 | the initial phase is done, the project may allow creation of 'forks' 76 | or the hosting of other repositories in the project's git hosting 77 | environment. It's an open question as to whether the work to create 78 | and manage this infrastructure provides enough benefit. 79 | 80 | The name of the default branch will be `main` in keeping with the new 81 | git default name. The beta mirror at github used the old default of 82 | `master` as its default branch. Users migrating from that will need 83 | to adjust their tooling. 84 | 85 | The plan for github is still working out the details. The basic plan 86 | is to save the current refs/branches/tags in a separate name space to 87 | allow current users to refer to the old names. We'll then push the new 88 | repo to github. By default, new users will get only the new stuff, 89 | while old users won't have the old stuff deleted for a while. This 90 | should provide for an orderly transition. 91 | 92 | The ports tree needs to make the switch just before a quarterly branch 93 | to avoid needing to mirror changes to subversion. Since there are a 94 | number of details to work out, they are planning to make the switch in 95 | March to allow time to consolidate their plans. 96 | 97 | The docs tree has been converted. We're in the process of mopping up 98 | from the different issues related to this that have come to light. So 99 | far it's the usual thing you'd expect: the automated web site building 100 | script needed to be changed to pull from git instead of subversion, 101 | for example. 102 | 103 | Release engineering plans to do future stable/12 releases from 104 | subversion to ensure maximum compatibility with current users 105 | (especially $FreeBSD$ tags). Starting with 13.0, releases will be cut 106 | from git. To facilitate re@'s building from subversion, and to support 107 | our current users and downstreams using FreeBSD via subversion, we've 108 | built a git to subversion exporter. 109 | 110 | The $FreeBSD$ tag is being phased out. Git provides no way to 111 | implement it in a sane and helpful way. We're recommending the project 112 | keep $FreeBSD$ in the source tree at least until stable/12 reaches end 113 | of life.. 114 | 115 | Git uses hashes instead of a sequentially increasing number. The 116 | conversion will include notes (that must be fetched separately) to 117 | allow a git revision to map to a subversion revision. A script is 118 | planned for the tools tree in the src repo. 119 | -------------------------------------------------------------------------------- /design-notes.md: -------------------------------------------------------------------------------- 1 | # Design notes 2 | 3 | I've abstracted the README from git_conv here. 4 | It was written by uqs@. 5 | This is still quite rough, but contains useful information, even in its currently lightly-edited form from there. 6 | More revision will be forthcoming. 7 | 8 | ## Gimme the repo! 9 | 10 | ``` 11 | git clone https://cgit-beta.freebsd.org/src.git && cd src 12 | git config --add remote.freebsd.fetch '+refs/notes/*:refs/notes/*' && git fetch 13 | ``` 14 | 15 | Same for the `doc` and `ports` repos. You can expect some things to be different. 16 | Most importantly you should check whether branch and mergepoints (project branches) are there and make sense. 17 | 18 | Neither `vendor`, `user` or `projects` branches are "visible" by default. 19 | `backups` refs are deleted branches and `cvs2svn` contains some of the detritus left over from the CVS days and the cvs2svn conversion. 20 | The `internal` namespace for now only has the `access` and `mentors` file, detailing when people got their various commit bits. 21 | 22 | ``` 23 | git config --add remote.freebsd.fetch '+refs/vendor/*:refs/vendor/*' 24 | git config --add remote.freebsd.fetch '+refs/projects/*:refs/projects/*' 25 | git config --add remote.freebsd.fetch '+refs/user/*:refs/user/*' 26 | git config --add remote.freebsd.fetch '+refs/backups/*:refs/backups/*' 27 | git config --add remote.freebsd.fetch '+refs/cvs2svn/*:refs/cvs2svn/*' 28 | git config --add remote.freebsd.fetch '+refs/internal/*:refs/internal/*' 29 | git fetch 30 | ``` 31 | 32 | Note that `internal`, `projects` and `user` branches also exist for the `doc` repo and `ports` has `projects` and `releng` as well. 33 | cvs2svn and backups are exclusive to the `src` repo though. 34 | 35 | - `user/` branches are never merged back into `master` 36 | - MFHs into `user/` or `projects/` branches are just cherry-picks to keep `git log --graph` somewhat readable and as merges wouldn't convey any useful information, really. 37 | - `vendor` **tags** were never flattened post-creation, as that would advance them off of the mainline branch and make them invisible to a simple `git log` 38 | - `release/1.0_cvs` et al. are snapshots of the checked out CVS source code, including expanded $Id$ tags. 39 | - no other keywords are expanded. 40 | - various vendor-foo suffixes have been collapsed into 1 vendor namespace, except for a few vendors where merging the userland and kernel bits is not straightforward due to how they interleave with the merge and branch history. 41 | - some branches have their history "extended", that is, commits under the `cvs2svn` area were properly attached. 42 | - ... and most of these commits have actually been inlined directly into the mainline tree to keep the history more "linear" and associate the commit with the original author and commit message. 43 | 44 | ## How to analyze the results 45 | 46 | Here are some tips that were deemed useful in making sense of resulting history. 47 | Please add a "graph notes log" alias to your `.gitconfig`: 48 | ``` 49 | [alias] 50 | gnlog = log --graph --pretty=format:'%Cred%h %C(green)%t %Creset %C(red)%ad %Creset-%C(yellow)%d%Creset %s %n %N %-GG' --date=short 51 | ``` 52 | 53 | ### Show the full tree of a vendor area 54 | 55 | ``` 56 | git gnlog `git show-ref|grep vendor/sendmail|cut -d" " -f1` 57 | ``` 58 | 59 | ### Find a certain SVN revision on master 60 | 61 | ``` 62 | git show -p `git log --format=%h --notes --grep=revision=294706$` 63 | ``` 64 | 65 | ### Show how/where/when a vendor branch was merged into master over time 66 | 67 | ``` 68 | git gnlog vendor/zstd/dist master 69 | ``` 70 | (but you'll need to search in the massive output for where the vendor branch is being merged, if you know a better way to represent this, please let us know!) 71 | 72 | ### Look for commits with more than 5 parents and log them 73 | 74 | ``` 75 | git log --format='%H %P' --all|awk '{if (NF > 5) { print NF " " $0}}'|sort -rn|cut -d" " -f2|xargs -n1 -I% git snlog -n 1 % 76 | ``` 77 | -------------------------------------------------------------------------------- /git-how.md: -------------------------------------------------------------------------------- 1 | # How FreeBSD implements Git 2 | 3 | ## Initial repositories 4 | 5 | There will be three initial repositories. 6 | 7 | The `doc` repository will contain all the project's documentation. 8 | 9 | The `src` repository will contain the base OS source code. 10 | 11 | The `ports` repository will contain the ports system. 12 | 13 | Note: There are discussions about prefixing freebsd- to the front of those. 14 | 15 | In addition to these main repositories, a number of specialized 16 | repositories exist to coordinate work on things like packages and 17 | graphics drivers. These repositories have their own rules and are 18 | organized in a way that makes the best sense for the group working on 19 | these things. Their organization is beyond the scope of this document. 20 | 21 | ## The `doc` repository 22 | 23 | The documentation repository ('doc') is the simplest repository. It has 24 | no branches (large scale collaboration projects are done with 25 | forks). It has a number of tags that describe the doc tree at various 26 | releases. 27 | 28 | The doc tree contains the content for the FreeBSD handbook, the web 29 | site and different incidental articles about the project and its 30 | operation. 31 | 32 | The repository uses a 'rebase' model. In this model, there are no 33 | merge commits, only curated patches that have been rebased forward to 34 | the tip of the tree and are 'fast forward'able and contain no merge 35 | commits. Force commits are forbidden. 36 | 37 | This may undergo changes after the conversion to asciidoc completes. 38 | 39 | ## The `ports` repository 40 | 41 | The ports repository has all the files we use to generate our 42 | packages. It's structure is currently being debated. 43 | 44 | ## The `src` repository 45 | 46 | The 'src' repository contains all the source code needed to build 47 | FreeBSD. 48 | 49 | There are four kinds of branches in this repository. 50 | 51 | The 'main' branch contains the mainline development of the operating 52 | system. This is "FreeBSD CURRENT" and all new work is expected to come 53 | in here when ready. 54 | 55 | The `stable/\*` branches are the projects stable, release branches. We 56 | create a new one every couple of years and maintain it for a 57 | while. Changes that are mature are cherry picked from the `main` 58 | branch into these branches (though sometimes direct commits happen, or 59 | the commits need to be tweaked). 60 | 61 | The `releng/\*` branches are branches for the release engineer to 62 | create releases for the project. 63 | 64 | The `vendor/\*` branches are vendor branches. These are used to manage 65 | importing new software and help track local changes not yet in 66 | upstream (or that will never be in upstream). 67 | 68 | -------------------------------------------------------------------------------- /projects-user.md: -------------------------------------------------------------------------------- 1 | # A brief note about Subversion Projects and User branches 2 | 3 | In the conversion to git, we plan on discontinuing the projects and 4 | user branches. This should have been communicated sooner, but has not 5 | been. 6 | 7 | The historical projects and user branches will be converted to git and 8 | be read only. You'll not be able to push commits to them once we've 9 | made the conversion. These are lightly used today, and it is simpler 10 | to discontinue this service than to deal with the logistical issues 11 | having them in the tree would cause. 12 | 13 | To not pollute the branch namespace, they will be not 14 | fetched by default. One needs to add the following to the remote 15 | for `freebsd` 16 | ``` 17 | fetch = +refs/projects/*:refs/projects/* 18 | fetch = +refs/user/*:refs/user/* 19 | ``` 20 | to fetch them. You can also choose to fetch a smaller portion of 21 | that namespace if you like. 22 | 23 | Since they are in a separate namespace, you'll need to add `ref/` 24 | to the front of them name. To re-create your old user branch: 25 | ``` 26 | % git checkout -b masq ref/user/imp/masq 27 | ``` 28 | This will create a local branch named masq. But you'll be unable to 29 | push these branches to FreeBSD's src repo. 30 | 31 | We've done this to simplify the FreeBSD repository. Going forward, it 32 | will just be for project branches. Collaborative works will be done in 33 | forked repositories. There are several sites that can host this. At the 34 | present we have no plans to offer hosting services for this 35 | purpose. However, if there's demand those plans can change in the future. 36 | Most of this work has already moved to github forks and other platforms hosted 37 | elsewhere, however. 38 | 39 | In the mean time, github.com, gitlab.com and sourceforge offer free hosting services 40 | anything should you need this. launchpad.net also has some interesting hosting 41 | options as well. 42 | -------------------------------------------------------------------------------- /pull-request.md: -------------------------------------------------------------------------------- 1 | # How to deal with Pull Requests 2 | 3 | Various Git collaboration tools allow random contributors to send us pull 4 | requests (PR, yes, unfortunately the same abbreviation as Problem Report). With 5 | Git, these are comparatively easy to pull down, inspect and test, as well as 6 | rebase onto our `main` branch and push to our source of truth. 7 | 8 | However, we cannot use the integrated 1-click buttons on these tools, as we see 9 | them as read-only mirrors and the pushes need to go directly to FreeBSD's 10 | infrastructure instead. 11 | 12 | Note that we disallow merges (except for vendor merges) so we want a "linear 13 | history" and need to do the equivalent of a "Rebase and merge". 14 | 15 | # GitHub 16 | 17 | As we had the legacy `master` branch on GitHub for quite some time, there are 18 | old PRs that are based on them. If you pull down anything before PR #449, 19 | you'll likely get a commit that's based on `master` instead of `main`. 20 | 21 | Add GitHub as a remote (only showing -src here), add a refspec to pull down 22 | all pull requests, fetch them, and clean up after our "legacy" master 23 | conversion. This is a one-time step and only needed for GitHub. 24 | ``` 25 | % git remote add github https://github.com/freebsd/freebsd-src 26 | % git config --add remote.github.fetch "+refs/pull/*:refs/remotes/github/pull/*" 27 | % git fetch github && git gc 28 | ``` 29 | 30 | Or fetch a very specific PR, only. 31 | ``` 32 | % git fetch github "refs/pull/422/*:refs/remotes/github/pull/422/*" 33 | remote: Enumerating objects: 19, done. 34 | remote: Counting objects: 100% (19/19), done. 35 | remote: Total 23 (delta 19), reused 19 (delta 19), pack-reused 4 36 | Unpacking objects: 100% (23/23), 6.23 KiB | 102.00 KiB/s, done. 37 | From github.com:freebsd/freebsd-src 38 | * [new ref] refs/pull/422/head -> github/pull/422/head 39 | * [new ref] refs/pull/422/merge -> github/pull/422/merge 40 | % git show-ref | grep /pull/ 41 | 63c82ec5c3833b50e45c31c59a04d7f9a827abb8 refs/remotes/github/pull/422/head 42 | 1d1d776bd0f036130446f3d331aef3dd131ef8a3 refs/remotes/github/pull/422/merge 43 | ``` 44 | 45 | We can ignore the merge head, as we want to pick the (potentially many) commits 46 | off of the head and rebase them onto our `main`. 47 | 48 | # GitLab et al. 49 | 50 | ``` 51 | % git remote add gitlab https://gitlab.com/FreeBSD/freebsd-src.git 52 | % git config --add remote.gitlab.fetch "+refs/merge-requests/*:refs/remotes/gitlab/merge-requests/*" 53 | % git fetch gitlab 54 | ``` 55 | 56 | Adapt this to wherever you got the pull or merge request from. 57 | 58 | # Common ways to integrate the Pull Request 59 | 60 | ## Patches in a Git repository 61 | 62 | We'll take the GitHub example, but GitLab and co. will be just the same. 63 | 64 | Run `git log github/pull/422/head` and you see 3 commits by a contributor, 65 | followed by commits that all come from "foo ". This is an 66 | indication that it's on the legacy `master` branch. 67 | 68 | While you can use `rebase A..B --onto C`, yours truly is often confused by the 69 | ordering of the arguments and a series of cherry-picks is likely easier. So in 70 | this case, let's create a new branch and see if stuff applies cleanly on top of `main`. 71 | 72 | ``` 73 | % git checkout -b pull_422 freebsd/main 74 | % git cherry-pick github/pull/422/head~2 75 | Auto-merging sys/sys/errno.h 76 | Auto-merging sys/kern/imgact_elf.c 77 | [pull_422 df45283b47f9] * Remove redundant code in __elfN(map_insert) * Do more check on the executable file. * Add a new error return constant ESUCCESS to make the return code more clear. 78 | Author: Wuyang Chung 79 | Date: Wed Jan 22 19:05:14 2020 +0800 80 | 2 files changed, 36 insertions(+), 99 deletions(-) 81 | % git cherry-pick github/pull/422/head~1 82 | Auto-merging sys/kern/imgact_elf.c 83 | [pull_422 07bb52c328c8] Modify comment. 84 | Author: Wuyang Chung 85 | Date: Wed Jan 22 19:10:33 2020 +0800 86 | 1 file changed, 1 insertion(+), 1 deletion(-) 87 | % git cherry-pick github/pull/422/head 88 | Auto-merging sys/sys/errno.h 89 | [pull_422 0a27e19be827] Add #ifndef _POSIX_SOURCE around ESUCCESS. 90 | Author: Wuyang Chung 91 | Date: Fri Jan 24 16:31:41 2020 +0800 92 | 1 file changed, 2 insertions(+) 93 | 94 | ``` 95 | 96 | ## Other patches 97 | 98 | Various means exist to submit patches that do not haved metadata associated with 99 | them, including the patch author, e.g., Bugzilla. Previously a committer would 100 | commit the received patch and typically add the contributor's name and e-mail 101 | address in the message metadata, in the form of a "Submitted by" line. 102 | 103 | Going forward, the procedure is generally the same with exception of the 104 | contributor metadata. Patches should have the author set to the contributor's 105 | information that would have previously appeared in the "Submitted by" tag, and 106 | the "Submitted by" tag should be entirely omitted. 107 | 108 | ``` 109 | % git checkout -b unattributed_patch freebsd/main 110 | % fetch "https://.../foo.diff" 111 | % patch < foo.diff 112 | ... 113 | % git commit --author="Submitter Name " ... 114 | ``` 115 | 116 | ## Massaging the commit history and content 117 | 118 | Inspect the commits with `git log -p --format=fuller`, note that Author is the 119 | contributor and Committer will be you. AuthorDate will also be way in the past, 120 | this is fine. If the Author fields contains egregious swear words or the like, 121 | maybe get them to change this. 122 | 123 | In our case, there's a trailing whitespace introduced by the commit, something 124 | you might want to fix and `git commit --amend` or use `git rebase -i` to squash 125 | things. Especially since there are 2 trivial changes in a commit and an ifdef 126 | wrapper, this is a case of squashing things down into 1 commit. There are 127 | multiple ways to do this. 128 | 129 | ``` 130 | % git rebase -i HEAD~10 131 | 132 | % vim sys/kern/imgact_elf.c 133 | % git commit --amend sys/kern/imgact_elf.c 134 | ``` 135 | 136 | You should amend the commit message with a pointer to the review that might've 137 | happened on the website, similar to phabricator. As a simple pull request 138 | number is ambiguous, please use the full URL. 139 | 140 | ``` 141 | Pull Request: https://github.com/freebsd/freebsd-src/pull/422 142 | ``` 143 | 144 | ## Double check your work 145 | 146 | We now have a single commit (without trailing whitespace), that looks like so: 147 | 148 | ``` 149 | % git log -n1 --format=fuller 150 | commit 61ea2b75b055bd7ef6edf0224adefaf457ce4c4a (HEAD -> pull_422) 151 | Author: Wuyang Chung 152 | AuthorDate: 2020-01-22 19:05:14 +0800 153 | Commit: Ulrich Spörlein 154 | CommitDate: 2021-01-06 14:14:38 +0100 155 | 156 | * Remove redundant code in __elfN(map_insert) 157 | * Do more check on the executable file. 158 | * Add a new error return constant ESUCCESS to make the return code more clear. 159 | 160 | Also add #ifndef _POSIX_SOURCE around ESUCCESS. 161 | 162 | % git log --graph --oneline 163 | ``` 164 | 165 | ## Pushing upstream 166 | 167 | If that passes all the build and runtime checks, it can be pushed to FreeBSD, 168 | but since time has passed, we first need to rebase onto `origin/main` again. As 169 | we branched off of `origin/main`, this is a simple `git pull --rebase` away 170 | (pull always implies fetching first). 171 | 172 | --------------------------------------------------------------------------------