├── .github ├── issue_template.md └── pull_request_template.md ├── .gitignore ├── LICENSE.txt ├── README.md ├── cherrytree ├── __init__.py ├── bin │ └── cherrytree ├── branch.py ├── classes.py └── github_utils.py ├── requirements-dev.txt ├── requirements.txt └── setup.py /.github/issue_template.md: -------------------------------------------------------------------------------- 1 | ## Expected Behavior 2 | 3 | 4 | ## Actual Behavior 5 | 6 | 7 | ## Steps to Reproduce the Problem 8 | 9 | 1. 10 | 1. 11 | 1. 12 | 13 | ## Specifications 14 | 15 | - Version: 16 | - Platform: 17 | - Subsystem: 18 | -------------------------------------------------------------------------------- /.github/pull_request_template.md: -------------------------------------------------------------------------------- 1 | ## Description of the change 2 | 3 | > Description here 4 | 5 | ## Type of change 6 | - [ ] Bug fix (non-breaking change that fixes an issue) 7 | - [ ] New feature (non-breaking change that adds functionality) 8 | - [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected) 9 | 10 | ## Related issues 11 | 12 | > Fix [#1]() 13 | 14 | ## Checklists 15 | 16 | ### Development 17 | 18 | - [ ] Lint rules pass locally 19 | - [ ] The code changed/added as part of this pull request has been covered with tests 20 | - [ ] All tests related to the changed code pass in development 21 | 22 | ### Code review 23 | 24 | - [ ] This pull request has a descriptive title and information useful to a reviewer. There may be a screenshot or screencast attached 25 | - [ ] "Ready for review" label attached to the PR and reviewers mentioned in a comment 26 | - [ ] Changes have been reviewed by at least one other engineer 27 | - [ ] Issue from task tracker has a link to this pull request 28 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | *.pyc 2 | cherrytree.egg-info/ 3 | yarn-error.log 4 | _modules 5 | changelog.sh 6 | .DS_Store 7 | .coverage 8 | _build 9 | _static 10 | _images 11 | _modules 12 | env_py3 13 | envpy3 14 | envp3 15 | .eggs 16 | build 17 | *.db 18 | tmp 19 | local_config.py 20 | env 21 | dist 22 | app.db 23 | *.bak 24 | .idea 25 | *.sqllite 26 | .vscode 27 | .python-version 28 | .tox 29 | dump.rdb 30 | 31 | # Node.js, webpack artifacts 32 | *.entry.js 33 | *.js.map 34 | node_modules 35 | npm-debug.log* 36 | 37 | # IntelliJ 38 | *.iml 39 | venv 40 | @eaDir/ 41 | 42 | # docker 43 | /Dockerfile 44 | /docker-build.sh 45 | /docker-compose.yml 46 | /docker-entrypoint.sh 47 | /docker-init.sh 48 | -------------------------------------------------------------------------------- /LICENSE.txt: -------------------------------------------------------------------------------- 1 | Apache License 2 | Version 2.0, January 2004 3 | http://www.apache.org/licenses/ 4 | 5 | TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 6 | 7 | 1. Definitions. 8 | 9 | "License" shall mean the terms and conditions for use, reproduction, 10 | and distribution as defined by Sections 1 through 9 of this document. 11 | 12 | "Licensor" shall mean the copyright owner or entity authorized by 13 | the copyright owner that is granting the License. 14 | 15 | "Legal Entity" shall mean the union of the acting entity and all 16 | other entities that control, are controlled by, or are under common 17 | control with that entity. For the purposes of this definition, 18 | "control" means (i) the power, direct or indirect, to cause the 19 | direction or management of such entity, whether by contract or 20 | otherwise, or (ii) ownership of fifty percent (50%) or more of the 21 | outstanding shares, or (iii) beneficial ownership of such entity. 22 | 23 | "You" (or "Your") shall mean an individual or Legal Entity 24 | exercising permissions granted by this License. 25 | 26 | "Source" form shall mean the preferred form for making modifications, 27 | including but not limited to software source code, documentation 28 | source, and configuration files. 29 | 30 | "Object" form shall mean any form resulting from mechanical 31 | transformation or translation of a Source form, including but 32 | not limited to compiled object code, generated documentation, 33 | and conversions to other media types. 34 | 35 | "Work" shall mean the work of authorship, whether in Source or 36 | Object form, made available under the License, as indicated by a 37 | copyright notice that is included in or attached to the work 38 | (an example is provided in the Appendix below). 39 | 40 | "Derivative Works" shall mean any work, whether in Source or Object 41 | form, that is based on (or derived from) the Work and for which the 42 | editorial revisions, annotations, elaborations, or other modifications 43 | represent, as a whole, an original work of authorship. For the purposes 44 | of this License, Derivative Works shall not include works that remain 45 | separable from, or merely link (or bind by name) to the interfaces of, 46 | the Work and Derivative Works thereof. 47 | 48 | "Contribution" shall mean any work of authorship, including 49 | the original version of the Work and any modifications or additions 50 | to that Work or Derivative Works thereof, that is intentionally 51 | submitted to Licensor for inclusion in the Work by the copyright owner 52 | or by an individual or Legal Entity authorized to submit on behalf of 53 | the copyright owner. For the purposes of this definition, "submitted" 54 | means any form of electronic, verbal, or written communication sent 55 | to the Licensor or its representatives, including but not limited to 56 | communication on electronic mailing lists, source code control systems, 57 | and issue tracking systems that are managed by, or on behalf of, the 58 | Licensor for the purpose of discussing and improving the Work, but 59 | excluding communication that is conspicuously marked or otherwise 60 | designated in writing by the copyright owner as "Not a Contribution." 61 | 62 | "Contributor" shall mean Licensor and any individual or Legal Entity 63 | on behalf of whom a Contribution has been received by Licensor and 64 | subsequently incorporated within the Work. 65 | 66 | 2. Grant of Copyright License. Subject to the terms and conditions of 67 | this License, each Contributor hereby grants to You a perpetual, 68 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 69 | copyright license to reproduce, prepare Derivative Works of, 70 | publicly display, publicly perform, sublicense, and distribute the 71 | Work and such Derivative Works in Source or Object form. 72 | 73 | 3. Grant of Patent License. Subject to the terms and conditions of 74 | this License, each Contributor hereby grants to You a perpetual, 75 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 76 | (except as stated in this section) patent license to make, have made, 77 | use, offer to sell, sell, import, and otherwise transfer the Work, 78 | where such license applies only to those patent claims licensable 79 | by such Contributor that are necessarily infringed by their 80 | Contribution(s) alone or by combination of their Contribution(s) 81 | with the Work to which such Contribution(s) was submitted. If You 82 | institute patent litigation against any entity (including a 83 | cross-claim or counterclaim in a lawsuit) alleging that the Work 84 | or a Contribution incorporated within the Work constitutes direct 85 | or contributory patent infringement, then any patent licenses 86 | granted to You under this License for that Work shall terminate 87 | as of the date such litigation is filed. 88 | 89 | 4. Redistribution. You may reproduce and distribute copies of the 90 | Work or Derivative Works thereof in any medium, with or without 91 | modifications, and in Source or Object form, provided that You 92 | meet the following conditions: 93 | 94 | (a) You must give any other recipients of the Work or 95 | Derivative Works a copy of this License; and 96 | 97 | (b) You must cause any modified files to carry prominent notices 98 | stating that You changed the files; and 99 | 100 | (c) You must retain, in the Source form of any Derivative Works 101 | that You distribute, all copyright, patent, trademark, and 102 | attribution notices from the Source form of the Work, 103 | excluding those notices that do not pertain to any part of 104 | the Derivative Works; and 105 | 106 | (d) If the Work includes a "NOTICE" text file as part of its 107 | distribution, then any Derivative Works that You distribute must 108 | include a readable copy of the attribution notices contained 109 | within such NOTICE file, excluding those notices that do not 110 | pertain to any part of the Derivative Works, in at least one 111 | of the following places: within a NOTICE text file distributed 112 | as part of the Derivative Works; within the Source form or 113 | documentation, if provided along with the Derivative Works; or, 114 | within a display generated by the Derivative Works, if and 115 | wherever such third-party notices normally appear. The contents 116 | of the NOTICE file are for informational purposes only and 117 | do not modify the License. You may add Your own attribution 118 | notices within Derivative Works that You distribute, alongside 119 | or as an addendum to the NOTICE text from the Work, provided 120 | that such additional attribution notices cannot be construed 121 | as modifying the License. 122 | 123 | You may add Your own copyright statement to Your modifications and 124 | may provide additional or different license terms and conditions 125 | for use, reproduction, or distribution of Your modifications, or 126 | for any such Derivative Works as a whole, provided Your use, 127 | reproduction, and distribution of the Work otherwise complies with 128 | the conditions stated in this License. 129 | 130 | 5. Submission of Contributions. Unless You explicitly state otherwise, 131 | any Contribution intentionally submitted for inclusion in the Work 132 | by You to the Licensor shall be under the terms and conditions of 133 | this License, without any additional terms or conditions. 134 | Notwithstanding the above, nothing herein shall supersede or modify 135 | the terms of any separate license agreement you may have executed 136 | with Licensor regarding such Contributions. 137 | 138 | 6. Trademarks. This License does not grant permission to use the trade 139 | names, trademarks, service marks, or product names of the Licensor, 140 | except as required for reasonable and customary use in describing the 141 | origin of the Work and reproducing the content of the NOTICE file. 142 | 143 | 7. Disclaimer of Warranty. Unless required by applicable law or 144 | agreed to in writing, Licensor provides the Work (and each 145 | Contributor provides its Contributions) on an "AS IS" BASIS, 146 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 147 | implied, including, without limitation, any warranties or conditions 148 | of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A 149 | PARTICULAR PURPOSE. You are solely responsible for determining the 150 | appropriateness of using or redistributing the Work and assume any 151 | risks associated with Your exercise of permissions under this License. 152 | 153 | 8. Limitation of Liability. In no event and under no legal theory, 154 | whether in tort (including negligence), contract, or otherwise, 155 | unless required by applicable law (such as deliberate and grossly 156 | negligent acts) or agreed to in writing, shall any Contributor be 157 | liable to You for damages, including any direct, indirect, special, 158 | incidental, or consequential damages of any character arising as a 159 | result of this License or out of the use or inability to use the 160 | Work (including but not limited to damages for loss of goodwill, 161 | work stoppage, computer failure or malfunction, or any and all 162 | other commercial damages or losses), even if such Contributor 163 | has been advised of the possibility of such damages. 164 | 165 | 9. Accepting Warranty or Additional Liability. While redistributing 166 | the Work or Derivative Works thereof, You may choose to offer, 167 | and charge a fee for, acceptance of support, warranty, indemnity, 168 | or other liability obligations and/or rights consistent with this 169 | License. However, in accepting such obligations, You may act only 170 | on Your own behalf and on Your sole responsibility, not on behalf 171 | of any other Contributor, and only if You agree to indemnify, 172 | defend, and hold each Contributor harmless for any liability 173 | incurred by, or claims asserted against, such Contributor by reason 174 | of your accepting any such warranty or additional liability. 175 | 176 | END OF TERMS AND CONDITIONS 177 | 178 | APPENDIX: How to apply the Apache License to your work. 179 | 180 | To apply the Apache License to your work, attach the following 181 | boilerplate notice, with the fields enclosed by brackets "[]" 182 | replaced with your own identifying information. (Don't include 183 | the brackets!) The text should be enclosed in the appropriate 184 | comment syntax for the file format. We also recommend that a 185 | file or class name and description of purpose be included on the 186 | same "printed page" as the copyright notice for easier 187 | identification within third-party archives. 188 | 189 | Copyright [yyyy] [name of copyright owner] 190 | 191 | Licensed under the Apache License, Version 2.0 (the "License"); 192 | you may not use this file except in compliance with the License. 193 | You may obtain a copy of the License at 194 | 195 | http://www.apache.org/licenses/LICENSE-2.0 196 | 197 | Unless required by applicable law or agreed to in writing, software 198 | distributed under the License is distributed on an "AS IS" BASIS, 199 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 200 | See the License for the specific language governing permissions and 201 | limitations under the License. 202 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Cherry Tree 2 | 3 | 4 | 5 | Cherry Tree is a set of tools that were originally designed to help 6 | build releases for 7 | [Apache Superset](https://github.com/apache/incubator-superset), 8 | but can be use for any other project 9 | that wants to implement a similar workflow. 10 | 11 | Ideas behind `cherrytree` include: 12 | * Github label driven development / releases 13 | * Make release from specifying base reference and Github labels 14 | * Apply cherries in correct order 15 | * Follow a base reference + cherries approach 16 | 17 | ## An example build file 18 | 19 | `cherrytree` offers tooling to both: 20 | 1. validate that all cherries apply cleanly before baking release 21 | 2. craft a branch in a target repo from a "bake file" 22 | 23 | Here's an example : 24 | 25 | `$ cherrytree bake -r apache/superset -l v1.3 1.3` 26 | ``` 27 | 🍒🌳🍒 CherryTree 28 | Running in dryrun mode, all changes will be rolled back 29 | Base ref is 69c5cd792296167d503403455b7e434fb3fedcd6 30 | 2090 commits skipped due to missing PRs 31 | Fetching labeled PRs: "v1.3" (113 labels found) 32 | 113 PRs found 33 | Fetching all branches 34 | Checking out base branch: 1.3 35 | Recreating and checking out temporary branch: __tmp_branch 36 | skip-applied #16233: fix(dashboard): cross filter chart highlight when filters badge ic... 37 | apply-ok #16193: refactor: external metadata fetch API [DRY-RUN] 38 | skip-applied #16251: chore: bump superset-ui packages to 0.17.84 39 | skip-applied #16253: fix: Homepage dashboard examples tab does not show user created ob... 40 | skip-applied #16259: fix: pivot columns with ints for name 41 | skip-applied #16260: fix: check roles before fetching reports 42 | apply-ok #16293: fix(sqlite): week grain refer to day of week [DRY-RUN] 43 | apply-ok #16167: feat: Adding Rockset db engine spec [DRY-RUN] 44 | apply-ok #16369: fix: call external metadata endpoint with correct rison object [DRY-RUN] 45 | apply-ok #16299: fix: copy to Clipboard order [DRY-RUN] 46 | apply-ok #16416: feat: add Shillelagh DB engine spec [DRY-RUN] 47 | apply-ok #16460: fix(native-filters): handle null values in value filter [DRY-RUN] 48 | apply-ok #16464: fix: prevent page crash when chart can't render [DRY-RUN] 49 | apply-ok #16468: fix(native-filters): handle undefined control value gracefully [DRY-RUN] 50 | apply-ok #16515: fix: Pin snowflake-sqlalchemy to 1.2.4 [DRY-RUN] 51 | apply-ok #16372: fix: ensure setting operator to `None` (#16371) [DRY-RUN] 52 | apply-ok #16526: fix: Set correct comparison operator for snowflake-sqlalchemy pinning [DRY-RUN] 53 | apply-ok #16482: fix: can't drop column when name overlap [DRY-RUN] 54 | apply-ok #16412: fix: Support Jinja template functions in global async queries [DRY-RUN] 55 | apply-ok #16573: fix: impersonate user label/tooltip [DRY-RUN] 56 | apply-ok #16594: feat: Experimental cross-filter plugins [DRY-RUN] 57 | apply-ok #16592: fix: Remove export CSV in old filter box [DRY-RUN] 58 | 59 | Summary: 60 | 0 successful cherries 61 | 17 dry-run cherries 62 | ``` 63 | 64 | ## Available options 65 | 66 | `cherrytree bake --help` 67 | 68 | ``` 69 | 🍒🌳🍒 CherryTree 70 | Usage: cherrytree bake [OPTIONS] RELEASE_BRANCH 71 | 72 | Applies cherries to release 73 | 74 | Options: 75 | -t, --target-branch TEXT target branch for baking. Leave empty for 76 | dry run 77 | -m, --main_branch TEXT name of branch containing cherries, usually 78 | `master` or `main` 79 | -r, --repo TEXT The name of the main repo. Example: 80 | apache/superset [required] 81 | -l, --label TEXT Name of label to use for cherry picking. 82 | Supports multiple labels, e.g. `-l Label1 -l 83 | Label2` 84 | -b, --blocking-label TEXT Name of labels to block cherry picking 85 | operation. Supports multiple labels, e.g. 86 | `-b Blocker1 -b Blocker2` 87 | -pr, --pull-request TEXT Pull request id to add to list of cherries 88 | to pick. Supports multiple ids, e.g. `-pr 89 | 1234 -pr 5678` 90 | -nd, --no-dryrun Should cherries be committed to target 91 | branch 92 | -e, --error-mode [break|dryrun|skip] 93 | What to do in case of an error. `skip` skips 94 | conflicted cherries, `dryrun` reverts to 95 | dryrun for subsequent prs and `break` stops 96 | cherry picking. [default: skip] 97 | -f, --force-rebuild-target Forcefully remove target branch before 98 | applying cherries. Only relevant when using 99 | `--target-branch` 100 | -at, --access-token TEXT GitHub access token. If left undefined, will 101 | use the GITHUB_TOKEN environment variable 102 | --help Show this message and exit. 103 | ``` 104 | -------------------------------------------------------------------------------- /cherrytree/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/apache-superset/cherrytree/b93af8755f49bae392cd493c26265361beb885bb/cherrytree/__init__.py -------------------------------------------------------------------------------- /cherrytree/bin/cherrytree: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | import os 3 | import sys 4 | from typing import Tuple, List, Optional 5 | 6 | import click 7 | 8 | from cherrytree.branch import CherryTreeBranch 9 | from cherrytree.github_utils import check_if_branch_exists 10 | 11 | sys.path.insert(0, os.path.abspath("..")) 12 | 13 | 14 | @click.group() 15 | def cli(): 16 | click.secho("🍒🌳🍒 CherryTree", fg="cyan") 17 | 18 | 19 | @cli.command() 20 | @click.argument("release-branch") 21 | @click.option( 22 | "--target-branch", "-t", help="target branch for baking. Leave empty for dry run" 23 | ) 24 | @click.option( 25 | "--main_branch", 26 | "-m", 27 | help="name of branch containing cherries, usually `master` or `main`", 28 | default="master", 29 | ) 30 | @click.option( 31 | "--repo", 32 | "-r", 33 | help='The name of the main repo. Example: apache/superset', 34 | required=True, 35 | ) 36 | @click.option( 37 | "--label", 38 | "-l", 39 | multiple=True, 40 | help='Name of label to use for cherry picking. Supports multiple labels, e.g. `-l Label1 -l Label2`', 41 | ) 42 | @click.option( 43 | "--blocking-label", 44 | "-b", 45 | multiple=True, 46 | help='Name of labels to block cherry picking operation. Supports multiple labels, e.g. `-b Blocker1 -b Blocker2`', 47 | ) 48 | @click.option( 49 | "--pull-request", 50 | "-pr", 51 | multiple=True, 52 | help='Pull request id to add to list of cherries to pick. Supports multiple ids, e.g. `-pr 1234 -pr 5678`', 53 | ) 54 | @click.option( 55 | "--no-dryrun", "-nd", 56 | is_flag=True, 57 | help="Should cherries be committed to target branch", 58 | show_default=True, 59 | default=False, 60 | ) 61 | @click.option( 62 | "--error-mode", 63 | "-e", 64 | help="What to do in case of an error. `skip` skips conflicted cherries, " 65 | "`dryrun` reverts to dryrun for subsequent prs and `break` stops " 66 | "cherry picking.", 67 | type=click.Choice(["break", "dryrun", "skip"], case_sensitive=False), 68 | default="skip", 69 | show_default=True, 70 | ) 71 | @click.option( 72 | "--force-rebuild-target", 73 | "-f", 74 | is_flag=True, 75 | help="Forcefully remove target branch before applying cherries. Only relevant " 76 | "when using `--target-branch`", 77 | show_default=True, 78 | default=False, 79 | ) 80 | @click.option( 81 | "--access-token", 82 | "-at", 83 | multiple=False, 84 | help="GitHub access token. If left undefined, will use the GITHUB_TOKEN " 85 | "environment variable", 86 | ) 87 | def bake( 88 | release_branch: str, 89 | main_branch: str, 90 | repo: str, 91 | target_branch: str, 92 | label: Tuple[str], 93 | blocking_label: Tuple[str], 94 | pull_request: Tuple[int], 95 | no_dryrun: bool, 96 | error_mode: str, 97 | force_rebuild_target: bool, 98 | access_token: Optional[str], 99 | ): 100 | """Applies cherries to release""" 101 | if no_dryrun: 102 | click.secho( 103 | "Running in non-dryrun mode, all changes will be committed!", fg="red" 104 | ) 105 | if force_rebuild_target and target_branch: 106 | click.secho( 107 | f"Forcefully rebuilding target branch `{target_branch}` " 108 | f"from base branch `{release_branch}`!", fg="red" 109 | ) 110 | else: 111 | click.secho( 112 | "Running in dryrun mode, all changes will be rolled back", fg="cyan" 113 | ) 114 | if no_dryrun and error_mode == "dryrun": 115 | click.secho( 116 | "In case of conflict: skip conflicted cherry, " 117 | "revert to dryrun mode for subsequent cherries", fg="cyan" 118 | ) 119 | elif no_dryrun and error_mode == "skip": 120 | click.secho( 121 | "In case of conflict: skip conflicted cherry, " 122 | "continue applying subsequent cherries", fg="cyan" 123 | ) 124 | elif no_dryrun: 125 | click.secho( 126 | "In case of conflict: stop applying cherries, " 127 | "revert to dry-run for subsequent cherries", fg="cyan" 128 | ) 129 | 130 | if force_rebuild_target is False and target_branch and check_if_branch_exists(target_branch): 131 | release_branch = target_branch 132 | pull_requests: List[int] = [] 133 | for id_ in pull_request: 134 | try: 135 | pull_requests.append(int(id_)) 136 | except ValueError: 137 | click.secho(f"Non-numeric pull request id: {id_}", fg="red") 138 | exit(1) 139 | cherry_tree = CherryTreeBranch( 140 | repo=repo, 141 | release_branch=release_branch, 142 | main_branch=main_branch, 143 | labels=list(label), 144 | blocking_labels=list(blocking_label), 145 | pull_requests=pull_requests, 146 | access_token=access_token, 147 | ) 148 | cherry_tree.apply_cherries( 149 | target_branch=target_branch, 150 | dryrun=not no_dryrun, 151 | error_mode=error_mode, 152 | force_rebuild_target=force_rebuild_target, 153 | ) 154 | 155 | 156 | if __name__ == "__main__": 157 | cli() 158 | -------------------------------------------------------------------------------- /cherrytree/branch.py: -------------------------------------------------------------------------------- 1 | from collections import OrderedDict 2 | from datetime import datetime, timezone 3 | from typing import Dict, List, Optional 4 | 5 | import click 6 | from git import Commit 7 | from github.Issue import Issue 8 | 9 | from cherrytree.github_utils import ( 10 | commit_pr_number, 11 | deduplicate_prs, 12 | get_access_token, 13 | get_issue, 14 | get_issues_from_labels, 15 | git_get_current_head, 16 | get_git_repo, 17 | os_system, 18 | truncate_str, 19 | ) 20 | from cherrytree.classes import ( 21 | Cherry, 22 | CherryTreeExecutionException, 23 | CommitSummary, 24 | ) 25 | 26 | SHORT_SHA_LEN = 12 27 | TMP_BRANCH = "__tmp_branch" 28 | 29 | 30 | class CherryTreeBranch: 31 | """Represents a release branch""" 32 | 33 | repo: str 34 | release_branch: str 35 | main_branch: str 36 | labels: List[str] 37 | blocking_labels: List[str] 38 | branch_commits: Dict[str, Dict[int, Commit]] 39 | missing_pull_requests: List[Issue] 40 | pull_requests: List[int] 41 | cherries: List[Cherry] 42 | blocking_pr_ids: List[int] 43 | 44 | def __init__( 45 | self, 46 | repo: str, 47 | release_branch: str, 48 | main_branch: str, 49 | labels: List[str], 50 | blocking_labels: List[str], 51 | pull_requests: List[int], 52 | access_token: Optional[str], 53 | ): 54 | self.repo = repo 55 | self.labels = labels 56 | self.blocking_labels = blocking_labels 57 | self.pull_requests = pull_requests 58 | self.missing_pull_requests = [] 59 | self.release_branch = release_branch 60 | self.main_branch = main_branch 61 | self.git_repo = get_git_repo() 62 | self.base_ref = self.get_base() 63 | self.blocking_pr_ids = [] 64 | try: 65 | self.access_token = get_access_token(access_token) 66 | except NotImplementedError: 67 | click.secho( 68 | f"No access token provided. Either provide one via the --access-token " 69 | f"parameter, or set the GITHUB_TOKEN env variable", fg="red") 70 | exit(1) 71 | 72 | click.secho(f"Base ref is {self.base_ref}", fg="cyan") 73 | 74 | 75 | self.branches = {} 76 | self.branch_commits = {} 77 | skipped_commits = 0 78 | for branch in (self.main_branch, self.release_branch): 79 | commits = OrderedDict() 80 | self.branch_commits[branch] = commits 81 | for commit in self.git_repo.iter_commits(branch): 82 | pr_number = commit_pr_number(commit) 83 | if pr_number is None: 84 | skipped_commits += 1 85 | else: 86 | commits[pr_number] = commit 87 | if skipped_commits: 88 | click.secho( 89 | f"{skipped_commits} commits skipped due to missing PRs", fg="yellow" 90 | ) 91 | 92 | # add all PRs that should be cherries 93 | prs: List[Issue] = [] 94 | for label in self.labels: 95 | click.secho(f'Fetching labeled PRs: "{label}"', fg="cyan", nl=False) 96 | new_prs = get_issues_from_labels( 97 | repo=self.repo, 98 | access_token=self.access_token, 99 | label=label, 100 | prs_only=True, 101 | ) 102 | click.secho(f' ({len(new_prs)} labels found)', fg="cyan") 103 | prs += new_prs 104 | 105 | for pull_request in pull_requests: 106 | prs.append(get_issue(self.repo, self.access_token, pull_request)) 107 | prs = deduplicate_prs(prs) 108 | 109 | # add all PRs that are flagged as blocking 110 | for label in self.blocking_labels: 111 | click.secho( 112 | f'Fetching labeled PRs marked as blocking: "{label}"', 113 | fg="cyan", 114 | nl=False, 115 | ) 116 | blocking_prs = get_issues_from_labels( 117 | repo=self.repo, 118 | access_token=self.access_token, 119 | label=label, 120 | prs_only=True, 121 | ) 122 | click.secho(f' ({len(blocking_prs)} blocking labels found)', fg="cyan") 123 | self.blocking_pr_ids += [pr.number for pr in blocking_prs] 124 | prs = deduplicate_prs(prs) 125 | now = datetime.now(tz=timezone.utc) 126 | prs.sort( 127 | key=lambda x: x.closed_at if x.closed_at is not None else now, 128 | ) 129 | click.secho(f"{len(prs)} PRs found", fg="cyan") 130 | self.cherries = [] 131 | for pr in prs: 132 | main_commit = self.branch_commits[self.main_branch].get(pr.number) 133 | applied_commit = self.branch_commits[self.release_branch].get(pr.number) 134 | if main_commit is None and pr.closed_at is not None: 135 | # skip closed PRs that haven't been merged 136 | continue 137 | cherry = Cherry( 138 | commit=main_commit, 139 | pr=pr, 140 | is_applied=True if applied_commit is not None else False, 141 | ) 142 | self.cherries.append(cherry) 143 | 144 | def get_base(self) -> str: 145 | base_commits = self.git_repo.merge_base(self.main_branch, self.release_branch) 146 | if len(base_commits) < 1: 147 | raise Exception("No common ancestor found!") 148 | elif len(base_commits) > 1: 149 | raise Exception("Multiple common ancestors found!?") 150 | return base_commits[0].hexsha 151 | 152 | def apply_cherries( 153 | self, 154 | target_branch: Optional[str], 155 | dryrun: bool, 156 | error_mode: str, 157 | force_rebuild_target: bool, 158 | ): 159 | error = False 160 | current_head = git_get_current_head() 161 | click.secho("Fetching all branches", fg="cyan") 162 | os_system("git fetch --all") 163 | click.secho(f"Checking out base branch: {self.release_branch}", fg="cyan") 164 | os_system(f"git checkout {self.release_branch}") 165 | 166 | if target_branch is None and dryrun: 167 | target_branch = TMP_BRANCH 168 | click.secho( 169 | f"Recreating and checking out temporary branch: {target_branch}", 170 | fg="cyan", 171 | ) 172 | os_system(f"git branch -D {target_branch}", raise_on_error=False) 173 | os_system(f"git checkout -b {target_branch}") 174 | elif (target_branch is None or target_branch == self.release_branch) and not dryrun: 175 | # base and target are the same - no need to recheckout 176 | target_branch = self.release_branch 177 | else: 178 | os_system(f"git branch {target_branch}", raise_on_error=False) 179 | if force_rebuild_target: 180 | click.secho(f"Recreating target branch: {target_branch}", fg="cyan") 181 | os_system(f"git branch -D {target_branch}", raise_on_error=False) 182 | os_system(f"git branch {target_branch}") 183 | click.secho(f"Checking out target branch: {target_branch}", fg="cyan") 184 | os_system(f"git checkout {target_branch}") 185 | 186 | applied_cherries: List[Cherry] = [] 187 | applied_dryrun_cherries: List[Cherry] = [] 188 | blocking_cherries: List[Cherry] = [] 189 | conflicted_cherries: List[CommitSummary] = [] 190 | open_cherries: List[Cherry] = [] 191 | 192 | base_sha = self.git_repo.head.commit.hexsha 193 | for cherry in self.cherries: 194 | pr = cherry.pr 195 | commit = cherry.commit 196 | if commit is None: 197 | click.secho( 198 | truncate_str(f"error-open #{pr.number}: {pr.title}"), fg="red" 199 | ) 200 | open_cherries.append(cherry) 201 | error = True 202 | continue 203 | sha = cherry.commit.hexsha 204 | if cherry.is_applied: 205 | click.secho( 206 | truncate_str(f"skip-applied #{pr.number}: {pr.title}"), fg="yellow" 207 | ) 208 | continue 209 | if cherry.pr.number in self.blocking_pr_ids: 210 | click.secho( 211 | truncate_str(f"error-blocking #{pr.number}: {pr.title}"), fg="red" 212 | ) 213 | blocking_cherries.append(cherry) 214 | error = True 215 | if error_mode == "dryrun": 216 | dryrun = True 217 | elif error_mode == "break": 218 | break 219 | try: 220 | os_system(f"git cherry-pick -x {sha}") 221 | if dryrun: 222 | applied_dryrun_cherries.append(cherry) 223 | else: 224 | applied_cherries.append(cherry) 225 | click.secho( 226 | truncate_str(f"apply-ok #{pr.number}: {pr.title}"), 227 | fg="green", 228 | nl=False, 229 | ) 230 | if dryrun: 231 | # os_system(f"git reset --hard HEAD~1") 232 | click.secho(" [DRY-RUN]", fg="cyan") 233 | else: 234 | base_sha = cherry.commit.hexsha 235 | click.echo() 236 | 237 | except CherryTreeExecutionException: 238 | os_system("git cherry-pick --abort") 239 | try: 240 | # try to ff to see if cherry was already applied 241 | os_system(f"git cherry-pick --ff {sha}") 242 | click.secho(f"skip-empty #{pr.number}: {pr.title}", fg="yellow") 243 | except CherryTreeExecutionException: 244 | click.secho( 245 | truncate_str(f"error-conflict #{pr.number}: {pr.title}"), 246 | fg="red", 247 | ) 248 | # These need to be put into a wrapper to avoid re-hitting the 249 | # GH API later 250 | conflicted_cherries.append(CommitSummary( 251 | pr_number=pr.number, 252 | pr_title=pr.title, 253 | sha=commit.hexsha, 254 | author=pr.user.login, 255 | merged_by=pr.closed_by.login, 256 | )) 257 | os_system("git cherry-pick --abort") 258 | error = True 259 | if error_mode == "dryrun": 260 | dryrun = True 261 | elif error_mode == "break": 262 | break 263 | 264 | if dryrun: 265 | os_system(f"git reset --hard {base_sha}") 266 | os_system(f"git checkout {current_head}") 267 | if target_branch == TMP_BRANCH: 268 | os_system(f"git branch -D {target_branch}") 269 | 270 | if blocking_cherries: 271 | click.echo() 272 | click.secho( 273 | f"{len(blocking_cherries)} open PRs that need to be cleared first:", 274 | fg="red", 275 | ) 276 | for cherry in blocking_cherries: 277 | pr = cherry.pr 278 | click.echo(f"#{pr.number} (author: {pr.user.login}): {pr.title}") 279 | 280 | if open_cherries: 281 | click.echo() 282 | click.secho( 283 | f"{len(open_cherries)} open PRs that need to be merged:", 284 | fg="red", 285 | ) 286 | for cherry in open_cherries: 287 | pr = cherry.pr 288 | click.echo(f"#{pr.number} (author: {pr.user.login}): {pr.title}") 289 | 290 | if conflicted_cherries: 291 | click.echo() 292 | click.secho( 293 | f"{len(conflicted_cherries)} " 294 | "PRs that need to be manually cherried due to conflicts:", 295 | fg="red", 296 | ) 297 | for commit in conflicted_cherries: 298 | click.echo( 299 | f"#{commit.pr_number} (sha: {commit.sha[:12]}, " 300 | f"author: {commit.author}, " 301 | f"merged by: {commit.merged_by}): " 302 | f"{truncate_str(commit.pr_title, 30)}" 303 | ) 304 | 305 | click.echo() 306 | click.secho(f"Summary:", fg="cyan") 307 | click.secho( 308 | f"{len(applied_cherries)} successful cherries", fg="cyan", 309 | ) 310 | if applied_dryrun_cherries: 311 | click.secho( 312 | f"{len(applied_dryrun_cherries)} dry-run cherries", fg="cyan", 313 | ) 314 | if blocking_cherries: 315 | click.secho( 316 | f"{len(blocking_cherries)} blocking cherries", fg="cyan", 317 | ) 318 | if conflicted_cherries: 319 | click.secho( 320 | f"{len(conflicted_cherries)} conflicts", fg="cyan", 321 | ) 322 | if open_cherries: 323 | click.secho( 324 | f"{len(open_cherries)} open PRs", fg="cyan", 325 | ) 326 | if error: 327 | exit(1) 328 | -------------------------------------------------------------------------------- /cherrytree/classes.py: -------------------------------------------------------------------------------- 1 | from dataclasses import dataclass 2 | from typing import Optional 3 | 4 | from git import Commit 5 | from github.Issue import Issue 6 | 7 | 8 | @dataclass 9 | class Cherry: 10 | commit: Optional[Commit] 11 | pr: Issue 12 | is_applied: bool 13 | 14 | 15 | @dataclass 16 | class CommitSummary: 17 | """ 18 | This dataclass is needed to speed up processing 19 | """ 20 | pr_number: int 21 | pr_title: str 22 | sha: str 23 | author: str 24 | merged_by: str 25 | 26 | 27 | class CherryTreeExecutionException(Exception): 28 | pass 29 | -------------------------------------------------------------------------------- /cherrytree/github_utils.py: -------------------------------------------------------------------------------- 1 | import os 2 | import re 3 | from collections import OrderedDict 4 | from typing import Generator, List, Optional, Reversible 5 | 6 | import delegator 7 | from git import Commit 8 | from git.exc import InvalidGitRepositoryError 9 | from git.repo import Repo 10 | from github import Github 11 | from github.Label import Label 12 | from github.Issue import Issue 13 | from github.GithubException import UnknownObjectException 14 | from github.Repository import Repository 15 | 16 | from cherrytree.classes import CherryTreeExecutionException 17 | 18 | # PRs are either of form "Merge pull request #nnn from..." or "...(#nnn)" 19 | PR_REGEX = re.compile(r"(^Merge pull request #(\d+) from|\(#(\d+)\)$)") 20 | 21 | 22 | def get_github_instance(access_token: str) -> Github: 23 | return Github(access_token) 24 | 25 | 26 | def get_access_token(access_token: Optional[str]) -> str: 27 | if access_token: 28 | return access_token 29 | 30 | access_token = os.environ.get("GITHUB_TOKEN") 31 | if not access_token: 32 | raise NotImplementedError("Env var 'GITHUB_TOKEN' is missing") 33 | 34 | return access_token 35 | 36 | 37 | def get_repo(repo: str, access_token: str) -> Repository: 38 | g = get_github_instance(access_token) 39 | return g.get_repo(repo) 40 | 41 | 42 | def get_issues_from_labels( 43 | repo: str, 44 | access_token: str, 45 | label: str, 46 | prs_only: bool = False, 47 | ) -> List[Issue]: 48 | label_objects: List[Label] = [] 49 | gh_repo = get_repo(repo, access_token) 50 | try: 51 | label_objects.append(gh_repo.get_label(label)) 52 | except UnknownObjectException: 53 | # unknown label 54 | return [] 55 | issues = gh_repo.get_issues(labels=label_objects, state="all") 56 | if prs_only: 57 | return [o for o in issues if o.pull_request] 58 | return [o for o in issues] 59 | 60 | 61 | def get_issue(repo: str, access_token: str, id_: int) -> Optional[Issue]: 62 | gh_repo = get_repo(repo, access_token) 63 | try: 64 | return gh_repo.get_issue(id_) 65 | except UnknownObjectException: 66 | # unknown id 67 | return None 68 | 69 | 70 | def get_commits(repo: str, access_token: str, branch: str, since=None): 71 | """Get commit objects from a branch, over a limited period""" 72 | gh_repo = get_repo(repo, access_token) 73 | branch_object = gh_repo.get_branch(branch) 74 | sha = branch_object.commit.sha 75 | if since: 76 | commits = gh_repo.get_commits(sha=sha, since=since) 77 | else: 78 | commits = gh_repo.get_commits(sha=sha) 79 | return commits 80 | 81 | 82 | def commit_pr_number(commit: Commit) -> Optional[int]: 83 | """Given a commit object, returns the PR number""" 84 | res = PR_REGEX.search(commit.summary) 85 | if res: 86 | groups = res.groups() 87 | return int(groups[1] or groups[2]) 88 | return None 89 | 90 | 91 | def get_commit_pr_map(commits: Reversible[Commit]): 92 | """Given a list of commits and prs, returns a map of pr_number to commit""" 93 | d = OrderedDict() 94 | for commit in reversed(commits): 95 | pr_number = commit_pr_number(commit) 96 | if pr_number: 97 | d[pr_number] = commit 98 | return d 99 | 100 | 101 | def truncate_str(value: str, width: int = 90) -> str: 102 | cont_str = "..." 103 | trunc_value = value[: width - len(cont_str)].strip() 104 | if len(trunc_value) < len(value.strip()): 105 | trunc_value = f"{trunc_value}{cont_str}" 106 | return f"{trunc_value:<{width}}" 107 | 108 | 109 | def git_get_current_head() -> str: 110 | output = os_system("git status | head -1") 111 | match = re.match("(?:HEAD detached at|On branch) (.*)", output) 112 | if not match: 113 | return "" 114 | return match.group(1) 115 | 116 | 117 | def os_system(cmd, raise_on_error=True) -> str: 118 | p = delegator.run(cmd) 119 | if raise_on_error and p.return_code != 0: 120 | raise CherryTreeExecutionException(p.err) 121 | return p.out 122 | 123 | 124 | def check_if_branch_exists(branch: str) -> bool: 125 | current_head = git_get_current_head() 126 | try: 127 | os_system(f"git checkout {branch}") 128 | except CherryTreeExecutionException: 129 | return False 130 | os_system(f"git checkout {current_head}") 131 | return True 132 | 133 | 134 | def deduplicate_prs(prs: List[Issue]) -> List[Issue]: 135 | pr_set = set() 136 | ret: List[Issue] = [] 137 | for pr in prs: 138 | if pr.number not in pr_set: 139 | ret.append(pr) 140 | pr_set.add(pr.number) 141 | return ret 142 | 143 | 144 | def get_git_repo() -> Repo: 145 | """ 146 | Find the path containing the git repo. Start by checking the current working 147 | directory, and proceed up the directory tree if a git repo can't be found. 148 | 149 | returns: Paath to closest git repo 150 | raises FileNotFoundError: if no git repo is found in the current path 151 | """ 152 | def _traverse_dirs(path: str) -> Generator[str, None, None]: 153 | # first yield the current directory 154 | yield path 155 | # then start yielding parents until we reach the root 156 | while True: 157 | parent = os.path.dirname(path) 158 | if path != parent: 159 | yield parent 160 | path = parent 161 | else: 162 | break 163 | 164 | cwd = os.getcwd() 165 | for dir_ in _traverse_dirs(cwd): 166 | try: 167 | repo = Repo(dir_) 168 | return repo 169 | except InvalidGitRepositoryError: 170 | pass 171 | raise FileNotFoundError("No git repo found in path: {}". format(cwd)) 172 | -------------------------------------------------------------------------------- /requirements-dev.txt: -------------------------------------------------------------------------------- 1 | black 2 | flake8 3 | pycodestyle 4 | pip-tools 5 | -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | # 2 | # This file is autogenerated by pip-compile with python 3.8 3 | # To update, run: 4 | # 5 | # pip-compile --output-file=requirements.txt setup.py 6 | # 7 | certifi==2022.12.7 8 | # via requests 9 | cffi==1.15.0 10 | # via pynacl 11 | charset-normalizer==2.0.12 12 | # via requests 13 | click==8.1.2 14 | # via cherrytree (setup.py) 15 | delegator-py==0.1.1 16 | # via cherrytree (setup.py) 17 | deprecated==1.2.13 18 | # via pygithub 19 | gitdb==4.0.9 20 | # via gitpython 21 | gitpython==3.1.30 22 | # via cherrytree (setup.py) 23 | idna==3.3 24 | # via requests 25 | pexpect==4.8.0 26 | # via delegator-py 27 | ptyprocess==0.7.0 28 | # via pexpect 29 | pycparser==2.21 30 | # via cffi 31 | pygithub==1.55 32 | # via cherrytree (setup.py) 33 | pyjwt==2.4.0 34 | # via pygithub 35 | pynacl==1.5.0 36 | # via pygithub 37 | python-dateutil==2.8.2 38 | # via cherrytree (setup.py) 39 | pyyaml==6.0 40 | # via cherrytree (setup.py) 41 | requests==2.27.1 42 | # via pygithub 43 | six==1.16.0 44 | # via python-dateutil 45 | smmap==5.0.0 46 | # via gitdb 47 | termcolor==1.1.0 48 | # via yaspin 49 | urllib3==1.26.9 50 | # via requests 51 | wrapt==1.14.0 52 | # via deprecated 53 | yaspin==2.1.0 54 | # via cherrytree (setup.py) 55 | -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | import io 2 | import os 3 | 4 | from setuptools import find_packages, setup 5 | 6 | VERSION = '2.0.1' 7 | REPO = 'https://github.com/apache-superset/cherrytree' 8 | BASE_DIR = os.path.abspath(os.path.dirname(__file__)) 9 | 10 | with io.open('README.md', encoding='utf-8') as f: 11 | long_description = f.read() 12 | 13 | 14 | setup( 15 | name='cherrytree', 16 | description=( 17 | 'A build tool to pick cherry, bake release branches, and power ' 18 | 'label-driven development'), 19 | long_description=long_description, 20 | long_description_content_type='text/markdown', 21 | version=VERSION, 22 | packages=find_packages(), 23 | include_package_data=True, 24 | zip_safe=False, 25 | scripts=['cherrytree/bin/cherrytree'], 26 | install_requires=[ 27 | 'click', 28 | 'pygithub', 29 | 'python-dateutil', 30 | 'GitPython', 31 | 'delegator.py', 32 | 'pyyaml', 33 | 'yaspin', 34 | ], 35 | author='Maxime Beauchemin', 36 | author_email='maximebeauchemin@gmail.com', 37 | url=REPO, 38 | download_url= REPO + '/tarball/' + VERSION, 39 | classifiers=[ 40 | 'Programming Language :: Python :: 3.6', 41 | 'Programming Language :: Python :: 3.7', 42 | 'Programming Language :: Python :: 3.8', 43 | 'Programming Language :: Python :: 3.9', 44 | ], 45 | ) 46 | --------------------------------------------------------------------------------