├── .gitignore ├── LICENSE ├── README.md ├── docs └── checklist.md ├── requirements └── base.txt ├── scripts └── label_updates │ ├── README.md │ ├── add_update_labels.py │ ├── github_specs │ └── template_github_specs.json │ ├── label_specs │ ├── label_input_dataverse-org.txt │ ├── label_input_eyedata.txt │ ├── label_input_geoconnect.txt │ ├── label_input_milestone-reader.txt │ ├── label_input_phthisis-ravens.txt │ ├── label_input_plaid.txt │ ├── label_input_private-zelig.txt │ ├── label_input_shared-dataverse-information.txt │ └── sample_label_input.txt │ └── make_md_table.py └── src ├── __init__.py ├── github_issues ├── __init__.py ├── github_issue_maker.py ├── label_helper.py ├── label_map.py ├── md_translate.py ├── migration_manager.py ├── milestone_helper.py ├── templates │ ├── comment.md │ ├── description.md │ └── related_issues.md └── user_map_helper.py ├── redmine_ticket ├── __init__.py ├── redmine_issue_downloader.py ├── redmine_issue_updater.py └── templates │ └── description_with_github_link.md ├── settings ├── Workbook1.xlsx ├── __init__.py ├── base.py ├── local_sample.py ├── sample_label_map.csv ├── sample_label_map2.csv ├── sample_milestone_map.csv ├── sample_milestone_map.xlsx ├── sample_milestone_map2.csv └── sample_user_map.csv └── utils ├── __init__.py └── msg_util.py /.gitignore: -------------------------------------------------------------------------------- 1 | scripts/label_updates/add_labels.py 2 | scripts/label_updates/github_specs/github*json 3 | src/settings/local_geoconnect.* 4 | test_files/* 5 | src/settings/local.py 6 | working_files/* 7 | *.idea 8 | # Byte-compiled / optimized / DLL files 9 | __pycache__/ 10 | *.py[cod] 11 | 12 | # C extensions 13 | *.so 14 | 15 | # Distribution / packaging 16 | .Python 17 | env/ 18 | bin/ 19 | build/ 20 | develop-eggs/ 21 | dist/ 22 | eggs/ 23 | lib/ 24 | lib64/ 25 | parts/ 26 | sdist/ 27 | var/ 28 | *.egg-info/ 29 | .installed.cfg 30 | *.egg 31 | 32 | # Installer logs 33 | pip-log.txt 34 | pip-delete-this-directory.txt 35 | 36 | # Unit test / coverage reports 37 | htmlcov/ 38 | .tox/ 39 | .coverage 40 | .cache 41 | nosetests.xml 42 | coverage.xml 43 | 44 | # Translations 45 | *.mo 46 | 47 | # Mr Developer 48 | .mr.developer.cfg 49 | .project 50 | .pydevproject 51 | 52 | # Rope 53 | .ropeproject 54 | 55 | # Django stuff: 56 | *.log 57 | *.pot 58 | 59 | # Sphinx documentation 60 | docs/_build/ 61 | 62 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | The MIT License (MIT) 2 | 3 | Copyright (c) 2014 Institute for Quantitative Social Science 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | ## redmine2github 2 | 3 | Scripts to migrate redmine tickets to github issues. This is for a 1-time move--so it's a mix of automation and manual decisions. e.g., Get-it-done, but make the process repeatable. 4 | 5 | ### Setup with [virtualenvwrapper](http://virtualenvwrapper.readthedocs.org/en/latest/install.html#basic-installation) 6 | 7 | * ```mkvirtualenv redmine_move``` 8 | * ```pip install -r requirements/base.txt``` 9 | 10 | ### Make a config file 11 | 12 | * Copy "/src/settings/local_sample.py to "/src/settings/local.py" and fill in variables 13 | * Fill in your redmine and github credentials 14 | * For the Redmine API key, see ["You can find your API key on your account page..."](http://www.redmine.org/projects/redmine/wiki/Rest_api#Authentication) 15 | * For github, enter your credentials. 16 | (To do, use a [Personal API token](https://github.com/blog/1509-personal-api-tokens)) 17 | 18 | ### Workflow 19 | 20 | #### (1) Download your open redmine issues 21 | 22 | * Each issue is saved as a .json file, including relevant "children", "journals", "watchers", "relations" 23 | * The file naming convention is by ticket issue number. 24 | * e.g. Issue 387 becomes "00387.json" 25 | * Files are saved to the following directory: 26 | * from settings/local.py: (REDMINE_ISSUES_DIRECTORY)/(current date in 'YYYY_MMDD' format) 27 | * e.g. ../working_files/redmine_issues/2014-0709/(json files here) 28 | ../working_files/redmine_issues/2014-0709/03982.json 29 | ../working_files/redmine_issues/2014-0709/04050.json 30 | * A json dict is saved to "issues_list.json" that maps the redmine issue number to the issue subject. For example: 31 | * e.g. ../working_files/redmine_issues/2014-0709/issues_list.json 32 | 33 | ```javascript 34 | { 35 | "03982": "Create Genomics Metadata Block", 36 | "04050": "Additional Astronomy FITS File Metadata Support for Units", 37 | "04051": "Metadata: Astronomy - Changes to Ingest and Display of Resolution Elements", 38 | "04072": "Edit Dataverse: Checking/unchecking use facets from Host Dataverse undoes any changes in the rest of the form." 39 | } 40 | 41 | ``` 42 | 43 | #### Example of downloading redmine issues 44 | 45 | + cd into the src/redmine_ticket directory 46 | + update the bottom of the "redmine_issue_downloader.py" file 47 | + Currently looks something like this: 48 | 49 | ```python 50 | if __name__=='__main__': 51 | from settings.base import REDMINE_SERVER, REDMINE_API_KEY, REDMINE_ISSUES_DIRECTORY 52 | #rn = RedmineIssueDownloader(REDMINE_SERVER, REDMINE_API_KEY, 'dvn', REDMINE_ISSUES_DIRECTORY) 53 | rn = RedmineIssueDownloader(REDMINE_SERVER, REDMINE_API_KEY, 1, REDMINE_ISSUES_DIRECTORY) 54 | rn.download_tickets() 55 | ``` 56 | 57 | + run it: 58 | 59 | ``` 60 | ../redmine2github/src/redmine_ticket> python redmine_issue_downloader.py 61 | ``` 62 | 63 | 64 | 65 | #### (2) Migrate your issues to a github repository 66 | 67 | 68 | --- 69 | 70 | Note 1: Once added, a github issue **cannot be deleted**. Therefore, to test your migration, create a new "scratch" github repository. Once you're satisfied that the migration works, delete the scratch repository. 71 | 72 | --- 73 | 74 | 75 | --- 76 | 77 | Note 2: The current [GitHub API limit](https://developer.github.com/v3/rate_limit/) is 5,000/day. Adding each issue may have 1-n API calls. Plan appropriately. 78 | 79 | + 1 API Call: Create issue with labels, milestones, assignee 80 | + This process creates a json file mapping { Redmine issue number : GitHub issue number} 81 | + 0-n API Calls for comments: A single API call is used to transfer each comment 82 | + 2 API Calls for related issues (optional): After all issues are moved 83 | + Call 1: Read each GitHub issue 84 | + At the bottom of the description, use the Redmine->GitHub issue number mapping to add related issue numbers and child issue numbers 85 | + Call 2: Update the GitHub description 86 | 87 | 88 | --- 89 | 90 | 91 | #### Quick script 92 | 93 | + cd into the src/github_issues directory 94 | + update the bottom of the "migration_manage.py" file 95 | + Currently looks something like this: 96 | 97 | ```python 98 | if __name__=='__main__': 99 | # e.g. json_input_directory, where you downloaded the redmine JSON 100 | # json_input_directory="some_dir/working_files/redmine_issues/2014-0709/" 101 | # 102 | json_input_directory = os.path.join(REDMINE_ISSUES_DIRECTORY, '2014-0702') 103 | 104 | kwargs = dict(include_comments=True\ 105 | , include_assignee=False\ 106 | , redmine_issue_start_number=4123\ 107 | , redmine_issue_end_number=4134\ 108 | , user_mapping_filename=USER_MAP_FILE # optional 109 | , label_mapping_filename=LABEL_MAP_FILE # optional 110 | , milestone_mapping_filename=MILESTONE_MAP_FILE # optional 111 | ) 112 | mm = MigrationManager(json_input_directory, **kwargs) 113 | mm.migrate_issues() 114 | ``` 115 | 116 | + run it: 117 | 118 | ``` 119 | ../redmine2github/src/github_issues>python migration_manager.py 120 | ``` 121 | 122 | 123 | 124 | 125 | #### Label Map Notes 126 | 127 | The label map is optional. It allows you to assign label names and colors by creating a label map file. 128 | 129 | + See [Sample Label Map, sample_label_map.csv](https://github.com/IQSS/redmine2github/blob/master/src/settings/sample_label_map.csv) 130 | 131 | --- 132 | 133 | + The [**redmine_type** column](https://github.com/IQSS/redmine2github/blob/master/src/settings/sample_label_map.csv) is for user convenience only, it is ignored by the program. So watch out for name collisions 134 | 135 | --- 136 | 137 | + Pertains to Redmine name values in fields **status, tracker, priority, or custom_fields** 138 | + If no map is specified in the [MigrationManager kwargs](https://github.com/IQSS/redmine2github/blob/master/src/github_issues/migration_manager.py#L127): 139 | * The status, tracker, priority, or custom_fields names in Redmine issues are made into GitHub labels. See "def get_label_names" in the [label_helper.py file](https://github.com/IQSS/redmine2github/blob/master/src/github_issues/label_helper.py) 140 | * A Redmine status of "New" would turn into label "Status: New" 141 | * A Redmine tracker of "Feature" would turn into label "Tracker: Feature" 142 | * A Redmine priority of "Urgent" would turn into label "Priority: Urgent" 143 | * A Redmine custom field of "UX/UI" turns into label "Component: UX/UI" 144 | * If no map is specified, then newly created labels will not have a color 145 | 146 | **Map Notes** - How is the map used? 147 | 148 | 149 | + The map is specfied in the [settings/local.py file](https://github.com/IQSS/redmine2github/blob/master/src/settings/local_sample.py#L32) under LABEL_MAP_FILE 150 | + If a status, tracker, priority, or custom_field name in a Redmine ticket is NOT found in the map, that name value will NOT be moved to GitHub 151 | + The map file is "dumb." If you would like to map more than one status name to a single status label, simply repeat it. 152 | + In the example below, the "redmine_name"s "In Design" and "In Dev" are _both_ mapped to the label named "Status 3: In Design/Dev" 153 | + For repeated github_label_names, make sure they're the same:) 154 | ```csv 155 | redmine_type, redmine_name, github_label_name, github_label_color 156 | status, In Design, Status 3: In Design/Dev,996600 157 | status, In Dev, Status 3: In Design/Dev,996600 158 | ``` 159 | 160 | + When the map is read, the values are trimmed. e.g. ", In Design ," would become "In Design" with leading/trailing spaces removed 161 | 162 | 163 | -------------------------------------------------------------------------------- /docs/checklist.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/IQSS/redmine2github/57d50bcf10952bcdd87174ac208e8dfebccec8ae/docs/checklist.md -------------------------------------------------------------------------------- /requirements/base.txt: -------------------------------------------------------------------------------- 1 | pygithub3>=0.5 2 | python-redmine==0.8.2 3 | Jinja2>=2.7.3 4 | requests==2.4 5 | -------------------------------------------------------------------------------- /scripts/label_updates/README.md: -------------------------------------------------------------------------------- 1 | # Add Colored Labels to GitHub Issues via API 2 | 3 | ## Create a github_specs.json file 4 | 5 | * Example file contents 6 | 7 | ```json 8 | { 9 | "REPO_OWNER_NAME": "IQSS", 10 | "REPO_NAME": "dataverse.org", 11 | "GITHUB_AUTH_USERNAME": "my_github_username", 12 | "GITHUB_PERSONAL_API_TOKEN_OR_PASSWORD": "abcd-abcd-abcd-abcd-abcd-abcd-abcd" 13 | } 14 | ``` 15 | 16 | * Template file: (github_specs/template_github_specs.json) 17 | * ```GITHUB_PERSONAL_API_TOKEN_OR_PASSWORD``` - See [Github instructions to create a Personal API Token](https://github.com/blog/1509-personal-api-tokens) 18 | 19 | ## Create a label specs file 20 | 21 | Sample label specs file: ["sample_label_input.txt"](https://github.com/IQSS/redmine2github/blob/master/scripts/label_updates/label_specs/sample_label_input.txt) 22 | 23 | 24 | ### Label Guide / Colors 25 | 26 | The Label Name/Color table below shows the label names and colors for: 27 | 28 | * Priority 29 | * Status 30 | * Type 31 | * Components - Add specific component names for your file 32 | 33 | 34 | ### Label Name/Color table 35 | 36 | + [See label colors in Dataverse](https://github.com/IQSS/dataverse/labels) 37 | 38 | 39 | |Label Name|Label Color| 40 | |------------|------------| 41 | |Priority: Critical|ff9900| 42 | |Priority: High|e11d21| 43 | |Priority: Medium|cc6666| 44 | |Status: Design|66ff66| 45 | |Status: Dev|66cc66| 46 | |Status: QA|336633| 47 | |Type: Bug|fef2c0| 48 | |Type: Feature|fef2c0| 49 | |Type: Suggestion|fef2c0| 50 | |Component: (component name 1)|c7def8| 51 | |Component: (component name 2)|c7def8| 52 | 53 | Notes: 54 | + All Types have the same color 55 | + All Components have the same color 56 | 57 | ### Run label making scripts 58 | 59 | The [add_update_labels.py](https://github.com/IQSS/redmine2github/blob/master/scripts/label_updates/add_update_labels.py) script in this directory may be used to script in your labels via the GitHub API. 60 | 61 | 1. Fill in the "MANDATORY VARIABLES" at the top. 62 | 1. Create a issue map similar to 63 | 1. Run the script against the file: 64 | 65 | ```python 66 | $ python add_update_labels.py github_specs.json sample_label_input.txt 67 | ``` 68 | -------------------------------------------------------------------------------- /scripts/label_updates/add_update_labels.py: -------------------------------------------------------------------------------- 1 | import os, sys 2 | from os.path import isfile, isdir, join 3 | import json 4 | import requests 5 | 6 | def msg(s): print (s) 7 | def dashes(): msg(40*'-') 8 | def msgt(s): dashes(); msg(s); dashes() 9 | def msgx(s): msgt('Error. Exiting'); msg(s); dashes(); sys.exit(0) 10 | 11 | 12 | """ 13 | ## FILL IN THE FOLLOWING MANDATORY VARIABLES 14 | REPO_NAME = 'github repository name' 15 | REPO_OWNER_NAME = 'github repository owner name' 16 | 17 | GITHUB_AUTH_USERNAME = 'github-username' 18 | # For A GitHub personal access token: https://github.com/settings/applications#personal-access-tokens 19 | GITHUB_PERSONAL_API_TOKEN_OR_PASSWORD = '' 20 | ## end: neeeded MANDATORY VARIABLES 21 | 22 | """ 23 | class GithubLabelMaker: 24 | 25 | GITHUB_SPEC_ATTRIBUTE_NAMES = [ 'REPO_NAME', 'REPO_OWNER_NAME', 'GITHUB_AUTH_USERNAME', 'GITHUB_PERSONAL_API_TOKEN_OR_PASSWORD' ] 26 | 27 | def __init__(self, github_specs_filename, label_specs_filename): 28 | 29 | assert isfile(github_specs_filename), "The github_specs file was not found: %s" % github_specs_filename 30 | assert isfile(label_specs_filename), "The label specs file was not found: %s" % label_specs_filename 31 | 32 | self.load_github_specs_from_json_file(github_specs_filename) 33 | self.add_labels(label_specs_filename) 34 | 35 | 36 | def load_github_specs_from_json_file(self, github_specs_fname): 37 | """ 38 | Create and load class attributes for each of the GITHUB_SPEC_ATTRIBUTE_NAMES 39 | 40 | e.g. self.REPO_NAME = 'something in JSON file' 41 | """ 42 | assert isfile(github_specs_fname), "The github_specs file was not found: %s" % github_specs_filename 43 | 44 | try: 45 | d = json.loads(open(github_specs_fname, 'rU').read()) 46 | except: 47 | msgx('Could not parse JSON in file: %s' % github_specs_fname) 48 | 49 | for attr in self.GITHUB_SPEC_ATTRIBUTE_NAMES: 50 | if not d.has_key(attr): 51 | msgx('Value not found for "%s". The github specs file "%s" must have a "%s"' % (attr, github_specs_fname, attr )) 52 | #print ('loaded: %s->%s' % (attr, self.__dict__[attr])) 53 | self.__dict__[attr] = d[attr] 54 | 55 | 56 | def get_github_auth(self): 57 | return (self.GITHUB_AUTH_USERNAME, self.GITHUB_PERSONAL_API_TOKEN_OR_PASSWORD) 58 | 59 | def get_label_url(self, label_name): 60 | assert label_name is not None, "Label name cannot be None" 61 | 62 | return 'https://api.github.com/repos/%s/%s/labels/%s' % (self.REPO_OWNER_NAME, self.REPO_NAME, label_name ) 63 | 64 | def get_create_label_url(self): 65 | return 'https://api.github.com/repos/%s/%s/labels' % (self.REPO_OWNER_NAME, self.REPO_NAME ) 66 | 67 | 68 | def add_labels(self, label_fname): 69 | assert isfile(label_fname), 'File not found [%s]' % label_fname 70 | 71 | flines = open(label_fname, 'rU').readlines() 72 | 73 | # strip lines and skip comments 74 | # 75 | flines = [x.strip() for x in flines if len(x.strip()) > 0 and not x.strip()[:1]=='#'] 76 | 77 | # split each line into a tuple: (label_name, label_color) 78 | # 79 | info_lines = [x.split('|') for x in flines if len(x.split('|'))==2] 80 | 81 | # Create each label--or update the color if label name exists and color is different 82 | # 83 | cnt = 0 84 | for label_info in info_lines: 85 | run_update_color = False 86 | 87 | if not len(label_info) == 2: 88 | continue # shouldn't happen 89 | 90 | 91 | label_info = [x.strip() for x in label_info] # strip each value 92 | label_name, label_color = label_info # split the tuple into 2 vals 93 | label_color = label_color.lower() # color in lowercase 94 | 95 | 96 | if len(label_name) == 0: 97 | msgx('The label name is blank!\nFrom line with "%s"' % ('|'.join(label_info))) 98 | 99 | if not len(label_color)==6: 100 | msgx('This label color should be 6 characters: "%s"\nFrom line with "%s"' % (label_color, 101 | '|'.join(label_info))) 102 | 103 | cnt += 1 104 | msgt('(%s) Create label [%s] with color [%s]' % (cnt, label_name, label_color)) 105 | 106 | 107 | github_label_url = self.get_label_url(label_name) 108 | msg('github_label_url: %s' % github_label_url) 109 | 110 | r = requests.get(github_label_url, auth=self.get_github_auth()) 111 | 112 | if r.status_code == 200: 113 | label_json = json.loads(r.text) 114 | if label_json.get('color', '').lower() == label_color: 115 | msg('Label already exists with same color!') 116 | continue 117 | else: 118 | msg('Label exists BUT different color...') 119 | run_update_color = True 120 | 121 | data = dict(name=label_name, color=label_color) 122 | if run_update_color: 123 | r = requests.patch(get_label_url, data=json.dumps(data), auth=self.get_github_auth()) 124 | else: 125 | #create_label_url = 'https://api.github.com/repos/%s/%s/labels' % (REPO_OWNER_NAME, REPO_NAME ) 126 | r = requests.post(self.get_create_label_url(), data=json.dumps(data), auth=self.get_github_auth()) 127 | 128 | 129 | if r.status_code == 200: 130 | msg('Label color updated!') 131 | elif r.status_code == 201: 132 | msg('Label created!') 133 | else: 134 | msg('Label failed!') 135 | msg(r.text) 136 | 137 | 138 | 139 | def show_instructions(): 140 | dashes() 141 | msg("""Please run this script from the command line as follows: 142 | 143 | >python label_update.py [github_specs.json] [label input filename] 144 | example: >python label_update.py github_specs_template.json label_input_dataverse-org.txt 145 | 146 | ----------------------------- 147 | -- Label Input file -- 148 | ----------------------------- 149 | 150 | Each line of the label input file includes: 151 | (1) label name 152 | (2) label color (hex) 153 | 154 | This info is separated by a '|' delimiter. 155 | 156 | Format: "Label Name|Hex Color" 157 | 158 | Notes: 159 | - The file has no header line. 160 | - Blank lines and comment lines ('#') are ignored 161 | - Each line is trimmed 162 | - Label names and colors are trimmed 163 | 164 | Example file contents: 165 | 166 | Priority: Critical|ff9900 167 | Priority: High|e11d21 168 | # Priority: Medium|cc6666 -- this line is ignored 169 | Status: Design|66ff66 170 | Status: Dev|66cc66 171 | Status: QA|336633 172 | 173 | 174 | """) 175 | 176 | if __name__=='__main__': 177 | if len(sys.argv) == 3: 178 | GithubLabelMaker(sys.argv[1], sys.argv[2]) 179 | #add_labels(sys.argv[1]) 180 | else: 181 | show_instructions() 182 | -------------------------------------------------------------------------------- /scripts/label_updates/github_specs/template_github_specs.json: -------------------------------------------------------------------------------- 1 | { 2 | "REPO_OWNER_NAME": "--github-owner-name-- e.g. IQSS", 3 | "REPO_NAME": "--repo-name-- e.g. dataverse.org", 4 | "GITHUB_AUTH_USERNAME": "--github-username--", 5 | "GITHUB_PERSONAL_API_TOKEN_OR_PASSWORD": "-- see: https://github.com/blog/1509-personal-api-tokens -- " 6 | } -------------------------------------------------------------------------------- /scripts/label_updates/label_specs/label_input_dataverse-org.txt: -------------------------------------------------------------------------------- 1 | Component: Content|c7def8 2 | Component: UI|c7def8 3 | Component: Deployment|c7def8 4 | Priority: Critical|ff9900 5 | Priority: High|e11d21 6 | Priority: Medium|cc6666 7 | Status: Design|66ff66 8 | Status: Dev|66cc66 9 | Status: QA|336633 10 | Type: Bug|fef2c0 11 | Type: Feature|fef2c0 12 | Type: Suggestion|fef2c0 -------------------------------------------------------------------------------- /scripts/label_updates/label_specs/label_input_eyedata.txt: -------------------------------------------------------------------------------- 1 | Component: Search|c7def8 2 | Component: Visualize|c7def8 3 | Component: Content|c7def8 4 | Priority: Critical|ff9900 5 | Priority: High|e11d21 6 | Priority: Medium|cc6666 7 | Status: Design|66ff66 8 | Status: Dev|66cc66 9 | Status: QA|336633 10 | Type: Bug|fef2c0 11 | Type: Feature|fef2c0 12 | Type: Suggestion|fef2c0 -------------------------------------------------------------------------------- /scripts/label_updates/label_specs/label_input_geoconnect.txt: -------------------------------------------------------------------------------- 1 | Component: GeoConnect|c7def8 2 | Component: WorldMap|c7def8 3 | Component: Dataverse|c7def8 4 | Priority: Critical|ff9900 5 | Priority: High|e11d21 6 | Priority: Medium|cc6666 7 | Status: Design|66ff66 8 | Status: Dev|66cc66 9 | Status: QA|336633 10 | Type: Bug|fef2c0 11 | Type: Feature|fef2c0 12 | Type: Suggestion|fef2c0 -------------------------------------------------------------------------------- /scripts/label_updates/label_specs/label_input_milestone-reader.txt: -------------------------------------------------------------------------------- 1 | Component: Functionality|c7def8 2 | Component: Content|c7def8 3 | Component: UI|c7def8 4 | Component: Deployment|c7def8 5 | Priority: Critical|ff9900 6 | Priority: High|e11d21 7 | Priority: Medium|cc6666 8 | Status: Design|66ff66 9 | Status: Dev|66cc66 10 | Status: QA|336633 11 | Type: Bug|fef2c0 12 | Type: Feature|fef2c0 13 | Type: Suggestion|fef2c0 -------------------------------------------------------------------------------- /scripts/label_updates/label_specs/label_input_phthisis-ravens.txt: -------------------------------------------------------------------------------- 1 | Component: ShinyServer|c7def8 2 | Component: Functionality|c7def8 3 | Component: Content|c7def8 4 | Component: UI|c7def8 5 | Component: Deployment|c7def8 6 | Priority: Critical|ff9900 7 | Priority: High|e11d21 8 | Priority: Medium|cc6666 9 | Status: Design|66ff66 10 | Status: Dev|66cc66 11 | Status: QA|336633 12 | Type: Bug|fef2c0 13 | Type: Feature|fef2c0 14 | Type: Suggestion|fef2c0 -------------------------------------------------------------------------------- /scripts/label_updates/label_specs/label_input_plaid.txt: -------------------------------------------------------------------------------- 1 | Component: File API|c7def8 2 | Component: Search API|c7def8 3 | Component: Analysis API|c7def8 4 | Priority: Critical|ff9900 5 | Priority: High|e11d21 6 | Priority: Medium|cc6666 7 | Status: Design|66ff66 8 | Status: Dev|66cc66 9 | Status: QA|336633 10 | Type: Bug|fef2c0 11 | Type: Feature|fef2c0 12 | Type: Suggestion|fef2c0 -------------------------------------------------------------------------------- /scripts/label_updates/label_specs/label_input_private-zelig.txt: -------------------------------------------------------------------------------- 1 | Component: High Level|c7def8 2 | Component: Testing|c7def8 3 | Component: UX/UI|c7def8 4 | Priority: Critical|ff9900 5 | Priority: High|e11d21 6 | Priority: Medium|cc6666 7 | Status: Design|66ff66 8 | Status: Dev|66cc66 9 | Status: QA|336633 10 | Type: Bug|fef2c0 11 | Type: Feature|fef2c0 12 | Type: Suggestion|fef2c0 -------------------------------------------------------------------------------- /scripts/label_updates/label_specs/label_input_shared-dataverse-information.txt: -------------------------------------------------------------------------------- 1 | Component: Validation|c7def8 2 | Component: Data Model|c7def8 3 | Component: Testing|c7def8 4 | Priority: Critical|ff9900 5 | Priority: High|e11d21 6 | Priority: Medium|cc6666 7 | Status: Design|66ff66 8 | Status: Dev|66cc66 9 | Status: QA|336633 10 | Type: Bug|fef2c0 11 | Type: Feature|fef2c0 12 | Type: Suggestion|fef2c0 -------------------------------------------------------------------------------- /scripts/label_updates/label_specs/sample_label_input.txt: -------------------------------------------------------------------------------- 1 | Component: Content|c7def8 2 | Component: UI|c7def8 3 | Component: Deployment|c7def8 4 | Priority: Critical|ff9900 5 | Priority: High|e11d21 6 | Priority: Medium|cc6666 7 | Status: Design|66ff66 8 | Status: Dev|66cc66 9 | Status: QA|336633 10 | Type: Bug|fef2c0 11 | Type: Feature|fef2c0 12 | Type: Suggestion|fef2c0 -------------------------------------------------------------------------------- /scripts/label_updates/make_md_table.py: -------------------------------------------------------------------------------- 1 | import os, sys 2 | 3 | """ 4 | Create something like this for github markdown 5 | 6 | |Label Name|Label Color| 7 | |------------|------------| 8 | |Priority: Critical|ff9900| 9 | |Priority: High|e11d21| 10 | |Priority: Medium|cc6666| 11 | |Status: Design|66ff66| 12 | |Status: Dev|66cc66| 13 | |Status: QA|336633| 14 | |Type: Bug|fef2c0| 15 | |Type: Feature|fef2c0| 16 | |Type: Suggestion|fef2c0| 17 | |Component: (component name 1)|c7def8| 18 | |Component: (component name 2)|c7def8| 19 | 20 | """ 21 | def msg(s): print (s) 22 | def dashes(): msg(40*'-') 23 | def msgt(s): dashes(); msg(s); dashes() 24 | def msgx(s): msgt('Error. Exiting'); msg(s); dashes(); sys.exit(0) 25 | 26 | def get_md_trow(*col_str_vals): 27 | trow = '|%s|' % ('|'.join(col_str_vals)) 28 | return trow 29 | 30 | def get_trow_split(num_cols): 31 | dash_list = [ '-'*12 for x in range(0, num_cols)] 32 | return get_md_trow(*dash_list) 33 | return trow 34 | 35 | def make_md_table(label_fname): 36 | if not os.path.isfile(label_fname): 37 | msgx('File not found [%s]' % label_fname) 38 | 39 | flines = open(label_fname, 'r').readlines() 40 | 41 | # strip lines and skip comments 42 | flines = [x.strip() for x in flines if len(x.strip()) > 0 and not x.strip()[:1]=='#'] 43 | 44 | # split lines into two parts 45 | info_lines = [x.split('|') for x in flines if len(x.split('|'))==2] 46 | 47 | cnt = 0 48 | tlines = [] 49 | tlines.append(get_md_trow('Label Name', 'Label Color')) 50 | tlines.append(get_trow_split(2)) 51 | for label_info in info_lines: 52 | run_update_color = False 53 | if not len(label_info) == 2: 54 | continue # shouldn't happen 55 | label_info = [x.strip() for x in label_info] 56 | label_name, label_color = label_info 57 | label_color = label_color.lower() 58 | 59 | if not len(label_name) > 0: 60 | msgx('The label name is blank!\nFrom line with "%s"' % ('|'.join(label_info))) 61 | if not len(label_color)==6: 62 | msgx('This label color should be 6 characters: "%s"\nFrom line with "%s"' % (label_color, 63 | '|'.join(label_info))) 64 | 65 | cnt += 1 66 | 67 | md_row = get_md_trow(label_name, label_color) 68 | tlines.append(md_row) 69 | print '\n'.join(tlines) 70 | 71 | 72 | if __name__=='__main__': 73 | if len(sys.argv) == 2: 74 | make_md_table(sys.argv[1]) 75 | else: 76 | msg('\nPlease use a label input file:') 77 | msg('\n >python make_md_table.py label_input.txt\n') 78 | -------------------------------------------------------------------------------- /src/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/IQSS/redmine2github/57d50bcf10952bcdd87174ac208e8dfebccec8ae/src/__init__.py -------------------------------------------------------------------------------- /src/github_issues/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/IQSS/redmine2github/57d50bcf10952bcdd87174ac208e8dfebccec8ae/src/github_issues/__init__.py -------------------------------------------------------------------------------- /src/github_issues/github_issue_maker.py: -------------------------------------------------------------------------------- 1 | from __future__ import print_function 2 | import os 3 | import sys 4 | import json 5 | 6 | 7 | if __name__=='__main__': 8 | SRC_ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) 9 | sys.path.append(SRC_ROOT) 10 | 11 | from datetime import datetime 12 | import requests 13 | from jinja2 import Template 14 | from jinja2 import Environment, PackageLoader 15 | 16 | from utils.msg_util import * 17 | from github_issues.md_translate import translate_for_github 18 | from github_issues.milestone_helper import MilestoneHelper 19 | from github_issues.label_helper import LabelHelper 20 | import csv 21 | 22 | from settings.base import get_github_auth, REDMINE_SERVER 23 | 24 | import pygithub3 25 | 26 | class GithubIssueMaker: 27 | """ 28 | Given a Redmine issue in JSON format, create a GitHub issue. 29 | These issues should be moved from Redmine in order of issue.id. This will allow mapping of Redmine issue ID's against newly created Github issued IDs. e.g., can translate related issues numbers, etc. 30 | """ 31 | ISSUE_STATE_CLOSED = ['Rejected', 'Closed', 'Resolved'] 32 | 33 | def __init__(self, user_map_helper=None, label_mapping_filename=None, milestone_mapping_filename=None): 34 | self.github_conn = None 35 | self.comments_service = None 36 | self.milestone_manager = MilestoneHelper(milestone_mapping_filename) 37 | self.label_helper = LabelHelper(label_mapping_filename) 38 | self.jinja_env = Environment(loader=PackageLoader('github_issues', 'templates')) 39 | self.user_map_helper = user_map_helper 40 | 41 | def get_comments_service(self): 42 | if self.comments_service is None: 43 | self.comments_service = pygithub3.services.issues.Comments(**get_github_auth()) 44 | 45 | return self.comments_service 46 | 47 | 48 | def get_github_conn(self): 49 | 50 | if self.github_conn is None: 51 | self.github_conn = pygithub3.Github(**get_github_auth()) 52 | return self.github_conn 53 | 54 | def format_name_for_github(self, author_name, include_at_sign=True): 55 | """ 56 | (1) Try the user map 57 | (2) If no match, return the name 58 | """ 59 | if not author_name: 60 | return None 61 | 62 | if self.user_map_helper: 63 | github_name = self.user_map_helper.get_github_user(author_name, include_at_sign) 64 | if github_name is not None: 65 | return github_name 66 | return author_name 67 | 68 | 69 | def get_redmine_assignee_name(self, redmine_issue_dict): 70 | """ 71 | If a redmine user has a github account mapped, add the person as the assignee 72 | 73 | "assigned_to": { 74 | "id": 4, 75 | "name": "Philip Durbin" 76 | }, 77 | /cc @kneath @jresig 78 | """ 79 | if not type(redmine_issue_dict) is dict: 80 | return None 81 | 82 | redmine_name = redmine_issue_dict.get('assigned_to', {}).get('name', None) 83 | if redmine_name is None: 84 | return None 85 | 86 | return redmine_name 87 | 88 | 89 | def get_assignee(self, redmine_issue_dict): 90 | """ 91 | If a redmine user has a github account mapped, add the person as the assignee 92 | 93 | "assigned_to": { 94 | "id": 4, 95 | "name": "Philip Durbin" 96 | }, 97 | /cc @kneath @jresig 98 | """ 99 | if not type(redmine_issue_dict) is dict: 100 | return None 101 | 102 | redmine_name = redmine_issue_dict.get('assigned_to', {}).get('name', None) 103 | if redmine_name is None: 104 | return None 105 | 106 | github_username = self.format_name_for_github(redmine_name, include_at_sign=False) 107 | 108 | return github_username 109 | 110 | 111 | def update_github_issue_with_related(self, redmine_json_fname, redmine2github_issue_map): 112 | """ 113 | Update a GitHub issue with related tickets as specfied in Redmine 114 | 115 | - Read the current github description 116 | - Add related notes to the bottom of description 117 | - Update the description 118 | 119 | "relations": [ 120 | { 121 | "delay": null, 122 | "issue_to_id": 4160, 123 | "issue_id": 4062, 124 | "id": 438, 125 | "relation_type": "relates" 126 | }, 127 | { 128 | "delay": null, 129 | "issue_to_id": 3643, 130 | "issue_id": 4160, 131 | "id": 439, 132 | "relation_type": "relates" 133 | } 134 | ], 135 | "id": 4160, 136 | """ 137 | if not os.path.isfile(redmine_json_fname): 138 | msgx('ERROR. update_github_issue_with_related. file not found: %s' % redmine_json_fname) 139 | 140 | #msg('issue map: %s' % redmine2github_issue_map) 141 | 142 | json_str = open(redmine_json_fname, 'rU').read() 143 | rd = json.loads(json_str) # The redmine issue as a python dict 144 | #msg('rd: %s' % rd) 145 | 146 | if rd.get('relations', None) is None: 147 | msg('no relations') 148 | return 149 | 150 | redmine_issue_num = rd.get('id', None) 151 | if redmine_issue_num is None: 152 | return 153 | 154 | github_issue_num = redmine2github_issue_map.get(str(redmine_issue_num), None) 155 | if github_issue_num is None: 156 | msg('Redmine issue not in nap') 157 | return 158 | 159 | 160 | # Related tickets under 'relations' 161 | # 162 | github_related_tickets = [] 163 | original_related_tickets = [] 164 | for rel in rd.get('relations'): 165 | issue_to_id = rel.get('issue_to_id', None) 166 | if issue_to_id is None: 167 | continue 168 | if rd.get('id') == issue_to_id: # skip relations pointing to this ticket 169 | continue 170 | 171 | original_related_tickets.append(issue_to_id) 172 | related_github_issue_num = redmine2github_issue_map.get(str(issue_to_id), None) 173 | msg(related_github_issue_num) 174 | if related_github_issue_num: 175 | github_related_tickets.append(related_github_issue_num) 176 | github_related_tickets.sort() 177 | original_related_tickets.sort() 178 | # 179 | # end: Related tickets under 'relations' 180 | 181 | 182 | # Related tickets under 'children' 183 | # 184 | # "children": [{ "tracker": {"id": 2, "name": "Feature" }, "id": 3454, "subject": "Icons in results and facet" }, ...] 185 | # 186 | github_child_tickets = [] 187 | original_child_tickets = [] 188 | 189 | child_ticket_info = rd.get('children', []) 190 | if child_ticket_info: 191 | for ctick in child_ticket_info: 192 | 193 | child_id = ctick.get('id', None) 194 | if child_id is None: 195 | continue 196 | 197 | original_child_tickets.append(child_id) 198 | child_github_issue_num = redmine2github_issue_map.get(str(child_id), None) 199 | 200 | msg(child_github_issue_num) 201 | if child_github_issue_num: 202 | github_child_tickets.append(child_github_issue_num) 203 | original_child_tickets.sort() 204 | github_child_tickets.sort() 205 | # 206 | # end: Related tickets under 'children' 207 | 208 | 209 | # 210 | # Update github issue with related and child tickets 211 | # 212 | # 213 | if len(original_related_tickets) == 0 and len(original_child_tickets)==0: 214 | return 215 | 216 | # Format related ticket numbers 217 | # 218 | original_issues_formatted = [ """[%s](%s)""" % (x, self.format_redmine_issue_link(x)) for x in original_related_tickets] 219 | original_issues_str = ', '.join(original_issues_formatted) 220 | 221 | related_issues_formatted = [ '#%d' % x for x in github_related_tickets] 222 | related_issue_str = ', '.join(related_issues_formatted) 223 | msg('Redmine related issues: %s' % original_issues_str) 224 | msg('Github related issues: %s' % related_issue_str) 225 | 226 | 227 | # Format children ticket numbers 228 | # 229 | original_children_formatted = [ """[%s](%s)""" % (x, self.format_redmine_issue_link(x)) for x in original_child_tickets] 230 | original_children_str = ', '.join(original_children_formatted) 231 | 232 | github_children_formatted = [ '#%d' % x for x in github_child_tickets] 233 | github_children_str = ', '.join(github_children_formatted) 234 | msg('Redmine sub-issues: %s' % original_children_str) 235 | msg('Github sub-issues: %s' % github_children_str) 236 | 237 | try: 238 | issue = self.get_github_conn().issues.get(number=github_issue_num) 239 | except pygithub3.exceptions.NotFound: 240 | msg('Issue not found!') 241 | return 242 | 243 | template = self.jinja_env.get_template('related_issues.md') 244 | 245 | template_params = { 'original_description' : issue.body\ 246 | , 'original_issues' : original_issues_str\ 247 | , 'related_issues' : related_issue_str\ 248 | , 'child_issues_original' : original_children_str\ 249 | , 'child_issues_github' : github_children_str\ 250 | 251 | } 252 | 253 | updated_description = template.render(template_params) 254 | 255 | issue = self.get_github_conn().issues.update(number=github_issue_num, data={'body':updated_description}) 256 | 257 | msg('Issue updated!')#' % issue.body) 258 | 259 | 260 | def format_redmine_issue_link(self, issue_id): 261 | if issue_id is None: 262 | return None 263 | 264 | return os.path.join(REDMINE_SERVER, 'issues', '%d' % issue_id) 265 | 266 | 267 | def close_github_issue(self, github_issue_num): 268 | 269 | if not github_issue_num: 270 | return False 271 | msgt('Close issue: %s' % github_issue_num) 272 | 273 | try: 274 | issue = self.get_github_conn().issues.get(number=github_issue_num) 275 | except pygithub3.exceptions.NotFound: 276 | msg('Issue not found!') 277 | return False 278 | 279 | if issue.state in self.ISSUE_STATE_CLOSED: 280 | msg('Already closed') 281 | return True 282 | 283 | updated_issue = self.get_github_conn().issues.update(number=github_issue_num, data={'state': 'closed' }) 284 | if not updated_issue: 285 | msg('Failed to close issue') 286 | return False 287 | 288 | if updated_issue.state in self.ISSUE_STATE_CLOSED: 289 | msg('Issue closed') 290 | return True 291 | 292 | msg('Failed to close issue') 293 | return False 294 | 295 | 296 | 297 | def make_github_issue(self, redmine_json_fname, **kwargs): 298 | """ 299 | Create a GitHub issue from JSON for a Redmine issue. 300 | 301 | - Format the GitHub description to include original redmine info: author, link back to redmine ticket, etc 302 | - Add/Create Labels 303 | - Add/Create Milestones 304 | """ 305 | if not os.path.isfile(redmine_json_fname): 306 | msgx('ERROR. make_github_issue. file not found: %s' % redmine_json_fname) 307 | 308 | include_comments = kwargs.get('include_comments', True) 309 | include_assignee = kwargs.get('include_assignee', True) 310 | 311 | json_str = open(redmine_json_fname, 'rU').read() 312 | rd = json.loads(json_str) # The redmine issue as a python dict 313 | 314 | #msg(json.dumps(rd, indent=4)) 315 | msg('Attempt to create issue: [#%s][%s]' % (rd.get('id'), rd.get('subject') )) 316 | 317 | # (1) Format the github issue description 318 | # 319 | # 320 | template = self.jinja_env.get_template('description.md') 321 | 322 | author_name = rd.get('author', {}).get('name', None) 323 | author_github_username = self.format_name_for_github(author_name) 324 | 325 | desc_dict = {'description' : translate_for_github(rd.get('description', 'no description'))\ 326 | , 'redmine_link' : self.format_redmine_issue_link(rd.get('id'))\ 327 | , 'redmine_issue_num' : rd.get('id')\ 328 | , 'start_date' : rd.get('start_date', None)\ 329 | , 'author_name' : author_name\ 330 | , 'author_github_username' : author_github_username\ 331 | , 'redmine_assignee' : self.get_redmine_assignee_name(rd) 332 | } 333 | 334 | description_info = template.render(desc_dict) 335 | 336 | # 337 | # (2) Create the dictionary for the GitHub issue--for the github API 338 | # 339 | #self.label_helper.clear_labels(151) 340 | github_issue_dict = { 'title': rd.get('subject')\ 341 | , 'body' : description_info\ 342 | , 'labels' : self.label_helper.get_label_names_from_issue(rd) 343 | } 344 | 345 | milestone_number = self.milestone_manager.get_create_milestone(rd) 346 | if milestone_number: 347 | github_issue_dict['milestone'] = milestone_number 348 | 349 | if include_assignee: 350 | assignee = self.get_assignee(rd) 351 | if assignee: 352 | github_issue_dict['assignee'] = assignee 353 | 354 | msg( github_issue_dict) 355 | 356 | # 357 | # (3) Create the issue on github 358 | # 359 | 360 | issue_obj = self.get_github_conn().issues.create(github_issue_dict) 361 | #issue_obj = self.get_github_conn().issues.update(151, github_issue_dict) 362 | 363 | msgt('Github issue created: %s' % issue_obj.number) 364 | msg('issue id: %s' % issue_obj.id) 365 | msg('issue url: %s' % issue_obj.html_url) 366 | 367 | 368 | # Map the new github Issue number to the redmine issue number 369 | # 370 | #redmine2github_id_map.update({ rd.get('id', 'unknown') : issue_obj.number }) 371 | 372 | #print( redmine2github_id_map) 373 | 374 | # 375 | # (4) Add the redmine comments (journals) as github comments 376 | # 377 | if include_comments: 378 | journals = rd.get('journals', None) 379 | if journals: 380 | self.add_comments_for_issue(issue_obj.number, journals) 381 | 382 | 383 | # 384 | # (5) Should this issue be closed? 385 | # 386 | if self.is_redmine_issue_closed(rd): 387 | self.close_github_issue(issue_obj.number) 388 | 389 | return issue_obj.number 390 | 391 | 392 | def is_redmine_issue_closed(self, redmine_issue_dict): 393 | """ 394 | "status": { 395 | "id": 5, 396 | "name": "Completed" 397 | }, 398 | """ 399 | if not type(redmine_issue_dict) == dict: 400 | return False 401 | 402 | status_info = redmine_issue_dict.get('status', None) 403 | if not status_info: 404 | return False 405 | 406 | if status_info.has_key('name') and status_info.get('name', None) in self.ISSUE_STATE_CLOSED: 407 | return True 408 | 409 | return False 410 | 411 | 412 | 413 | def add_comments_for_issue(self, issue_num, journals): 414 | """ 415 | Add comments 416 | """ 417 | if journals is None: 418 | msg('no journals') 419 | return 420 | 421 | comment_template = self.jinja_env.get_template('comment.md') 422 | 423 | for j in journals: 424 | notes = j.get('notes', None) 425 | if not notes: 426 | continue 427 | 428 | author_name = j.get('user', {}).get('name', None) 429 | author_github_username = self.format_name_for_github(author_name) 430 | 431 | note_dict = { 'description' : translate_for_github(notes)\ 432 | , 'note_date' : j.get('created_on', None)\ 433 | , 'author_name' : author_name\ 434 | , 'author_github_username' : author_github_username\ 435 | } 436 | comment_info = comment_template.render(note_dict) 437 | 438 | comment_obj = None 439 | try: 440 | comment_obj = self.get_comments_service().create(issue_num, comment_info) 441 | except requests.exceptions.HTTPError as e: 442 | msgt('Error creating comment: %s' % e.message) 443 | continue 444 | 445 | if comment_obj: 446 | dashes() 447 | msg('comment created') 448 | 449 | msg('comment id: %s' % comment_obj.id) 450 | msg('api issue_url: %s' % comment_obj.issue_url) 451 | msg('api comment url: %s' % comment_obj.url) 452 | msg('html_url: %s' % comment_obj.html_url) 453 | #msg(dir(comment_obj)) 454 | 455 | 456 | if __name__=='__main__': 457 | #auth = dict(login=GITHUB_LOGIN, password=GITHUB_PASSWORD_OR_PERSONAL_ACCESS_TOKEN, repo=GITHUB_TARGET_REPOSITORY, user=GITHUB_TARGET_USERNAME) 458 | #milestone_service = pygithub3.services.issues.Milestones(**auth) 459 | #comments_service = pygithub3.services.issues.Comments(**auth) 460 | #fname = 03385.json' 461 | #gm.make_github_issue(fname, {}) 462 | 463 | import time 464 | 465 | issue_filename = '/Users/rmp553/Documents/iqss-git/redmine2github/working_files/redmine_issues/2014-0702/04156.json' 466 | gm = GithubIssueMaker() 467 | for x in range(100, 170): 468 | gm.close_github_issue(x) 469 | #gm.make_github_issue(issue_filename, {}) 470 | 471 | sys.exit(0) 472 | root_dir = '/Users/rmp553/Documents/iqss-git/redmine2github/working_files/redmine_issues/2014-0702/' 473 | 474 | cnt =0 475 | for fname in os.listdir(root_dir): 476 | if fname.endswith('.json'): 477 | 478 | num = int(fname.replace('.json', '')) 479 | if num < 3902: continue 480 | msg('Add issue from: %s' % fname) 481 | cnt+=1 482 | fullname = os.path.join(root_dir, fname) 483 | gm.make_github_issue(fullname, {}) 484 | if cnt == 150: 485 | break 486 | 487 | if cnt%50 == 0: 488 | msg('sleep 2 secs') 489 | time.sleep(2) 490 | 491 | #sys.exit(0) 492 | 493 | 494 | 495 | 496 | 497 | 498 | -------------------------------------------------------------------------------- /src/github_issues/label_helper.py: -------------------------------------------------------------------------------- 1 | import os 2 | import sys 3 | 4 | if __name__=='__main__': 5 | SRC_ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) 6 | sys.path.append(SRC_ROOT) 7 | 8 | import requests 9 | from utils.msg_util import * 10 | from settings.base import GITHUB_LOGIN, GITHUB_PASSWORD_OR_PERSONAL_ACCESS_TOKEN, GITHUB_TARGET_USERNAME, GITHUB_TARGET_REPOSITORY 11 | import json 12 | from github_issues.label_map import LabelMap 13 | 14 | 15 | class LabelHelper: 16 | 17 | def __init__(self, label_map_filename=None): 18 | """The add label to issue seems broken in pygithub3, just use this for now""" 19 | self.auth = (GITHUB_LOGIN, GITHUB_PASSWORD_OR_PERSONAL_ACCESS_TOKEN) 20 | 21 | self.label_map_filename = label_map_filename 22 | self.label_map = None 23 | self.using_label_map = False 24 | self.load_map() 25 | 26 | def load_map(self): 27 | """ 28 | If a label_map_filename is specified, load it up! 29 | """ 30 | if self.label_map_filename is None: 31 | self.using_label_map = False # a bit redundant 32 | return 33 | 34 | self.label_map = LabelMap(self.label_map_filename) 35 | self.using_label_map = True 36 | 37 | self.make_update_map_labels() 38 | 39 | 40 | def make_update_map_labels(self): 41 | """ 42 | Go through the label make and make sure they all exist, with the appropriate colors 43 | """ 44 | msgt('Match label map names/color to GitHub') 45 | 46 | for label_info in self.label_map.get_label_info_objects(): 47 | 48 | msg('\nCheck label: %s %s' % (label_info.github_label_name, label_info.github_label_color)) 49 | 50 | # (1) try to get label 51 | # 52 | msg(' (1) Try to retrieve label') 53 | label_url = 'https://api.github.com/repos/%s/%s/labels/%s' % (GITHUB_TARGET_USERNAME, GITHUB_TARGET_REPOSITORY, label_info.github_label_name) 54 | req = requests.get(label_url, auth=self.auth) 55 | msg('url: %s' % label_url) 56 | msg('status: %s' % req.status_code) 57 | 58 | # (2) Label exists 59 | if req.status_code == 200: 60 | github_label_info = req.json() 61 | msg('-- label exists\n %s' % github_label_info) 62 | #print label_info 63 | 64 | # (2a) Color matches! -- all done 65 | if github_label_info.get('color', 'nope') == label_info.github_label_color: 66 | msg('-- color matches!') 67 | continue 68 | 69 | # (2b) Color doesn't match -- update color 70 | msg(' (2b) Try to update label color') 71 | label_url = 'https://api.github.com/repos/%s/%s/labels/%s' % (GITHUB_TARGET_USERNAME, GITHUB_TARGET_REPOSITORY, label_info.github_label_name) 72 | data = dict(name=label_info.github_label_name\ 73 | , color=label_info.github_label_color) 74 | req = requests.patch(label_url, data=json.dumps(data), auth=self.auth) 75 | if req.status_code == 200: 76 | msg(' Color updated!') 77 | msg(req.text) 78 | continue 79 | msgx('Color updated failed!') 80 | 81 | # (3) Create new label with color 82 | msg(' (3) Try to create label') 83 | label_url = 'https://api.github.com/repos/%s/%s/labels' % (GITHUB_TARGET_USERNAME, GITHUB_TARGET_REPOSITORY) 84 | data = dict(name=label_info.github_label_name\ 85 | , color=label_info.github_label_color) 86 | req = requests.post(label_url, data=json.dumps(data), auth=self.auth) 87 | msg(req.text) 88 | msg(req.status_code) 89 | if req.status_code in [ 200, 201]: 90 | msg('Label created with color!') 91 | continue 92 | else: 93 | msgx('Create label with color failed!') 94 | 95 | 96 | def clear_labels(self, issue_id): 97 | msgt('Clear Labels for an Issue. Issue: [%s]' % (issue_id)) 98 | #DELETE /repos/:owner/:repo/issues/:number/labels 99 | label_url = 'https://api.github.com/repos/%s/%s/issues/%s/labels' % (GITHUB_TARGET_USERNAME, GITHUB_TARGET_REPOSITORY, issue_id) 100 | req = requests.delete(label_url, auth=self.auth) 101 | msg('labels deleted!') 102 | 103 | 104 | 105 | def add_labels_to_issue(self, issue_id, labels=[]): 106 | msg('Add Labels to Issue. Issue: [%s] Labels: [%s]' % (issue_id, labels)) 107 | 108 | if not issue_id or not type(labels) in [list, tuple]: 109 | return 110 | 111 | if len(labels) == 0: 112 | return 113 | 114 | label_url = 'https://api.github.com/repos/%s/%s/issues/%s/labels' % (GITHUB_TARGET_USERNAME, GITHUB_TARGET_REPOSITORY, issue_id) 115 | 116 | #labels = json.dumps(['invalid', 'bug', 'enhancement', 'duplicate'])#['Bug', 'invalid']) 117 | labels_for_call = json.dumps(labels) 118 | msg('labels: %s' % labels_for_call) 119 | 120 | req = requests.post(label_url, auth=self.auth, data=labels_for_call) 121 | 122 | msg('result: %s' % req.text) 123 | 124 | 125 | 126 | def get_label_from_id_name(self, label_info_dict, key_name=None, label_prefix='', non_formatted=False): 127 | """ 128 | Expects a dict in one of 2 formats (where key_name is "status"): 129 | Format 1: 130 | "status": { "id": 1, "name": "New" }, 131 | 132 | Format 2: 133 | { "id":3, "name":"UX/UI Component"} 134 | 135 | """ 136 | if not type(label_info_dict) is dict: 137 | return None 138 | 139 | # For Format 1 above 140 | if key_name is not None: 141 | label_info_dict = label_info_dict.get(key_name, None) 142 | if label_info_dict is None: 143 | return None 144 | 145 | if label_info_dict.has_key('id') and label_info_dict.has_key('name'): 146 | if non_formatted: 147 | return label_info_dict['name'] 148 | 149 | if label_prefix: 150 | return '%s %s' % (label_prefix, label_info_dict['name']) 151 | 152 | return label_info_dict['name'] 153 | 154 | return None 155 | 156 | 157 | def get_label_names_based_on_map(self, redmine_issue_dict): 158 | """ 159 | We're using the label map, so before using a label, create it with the appropriate color. 160 | 161 | If the label doesn't appear in the map, then discard it 162 | """ 163 | label_names = self.get_label_names(redmine_issue_dict, non_formatted=True) 164 | 165 | if len(label_names) == 0: 166 | return [] 167 | 168 | mapped_label_names = [] 169 | 170 | for name in label_names: 171 | github_label_name = self.label_map.get_github_label_from_redmine_name(name) 172 | print 'name', name, 'github_label_name: ', github_label_name 173 | if github_label_name: 174 | mapped_label_names.append(github_label_name) 175 | #msgx('blah') 176 | return mapped_label_names 177 | 178 | def get_label_names_from_issue(self, redmine_issue_dict): 179 | if self.using_label_map is True: 180 | return self.get_label_names_based_on_map(redmine_issue_dict) 181 | 182 | return self.get_label_names(redmine_issue_dict) 183 | 184 | 185 | def get_label_names(self, redmine_issue_dict, non_formatted=False): 186 | """ 187 | Read a redmine issue and a make a list of formatted label names for 188 | - status 189 | - tracker 190 | - priority 191 | - component 192 | - category 193 | :returns: list with formatted label names 194 | """ 195 | if not type(redmine_issue_dict) is dict: 196 | return [] 197 | 198 | label_names = [] 199 | 200 | # Add status 201 | status_label_name = self.get_label_from_id_name(redmine_issue_dict, 'status', 'Status:', non_formatted) 202 | if status_label_name: 203 | label_names.append(status_label_name) 204 | 205 | # Add tracker 206 | tracker_label_name = self.get_label_from_id_name(redmine_issue_dict, 'tracker', 'Tracker:', non_formatted) 207 | if tracker_label_name: 208 | label_names.append(tracker_label_name) 209 | 210 | # Add priority 211 | priority_label_name = self.get_label_from_id_name(redmine_issue_dict, 'priority', 'Priority:', non_formatted) 212 | if priority_label_name: 213 | label_names.append(priority_label_name) 214 | 215 | # Add category 216 | category_label_name = self.get_label_from_id_name(redmine_issue_dict, 'category', 'Category:', non_formatted) 217 | if category_label_name: 218 | label_names.append(category_label_name) 219 | 220 | # Add component 221 | # "custom_fields": [ 222 | # { 223 | # "id": 1, 224 | # "value": "0", 225 | # "name": "Usability Testing" 226 | # } 227 | # ], 228 | # 229 | custom_fields = redmine_issue_dict.get('custom_fields', None) 230 | if custom_fields and len(custom_fields) > 0: 231 | for cf_dict in custom_fields: 232 | component_label_name = self.get_label_from_id_name(cf_dict, None, 'Component:', non_formatted) 233 | if component_label_name: 234 | label_names.append(component_label_name) 235 | 236 | return label_names 237 | 238 | if __name__=='__main__': 239 | from settings.base import LABEL_MAP_FILE 240 | LabelHelper(LABEL_MAP_FILE) # load a new label color map 241 | 242 | 243 | """ 244 | #labels_service = pygithub3.services.issues.Labels(**auth) 245 | #auth = dict(login=GITHUB_LOGIN, password=GITHUB_PASSWORD_OR_PERSONAL_ACCESS_TOKEN) 246 | 247 | #pygithub3.services.issues.Labels 248 | 249 | # works! 250 | label_info = dict(name='Week of World Cup', color="006699") 251 | #issues_service.create(label_info)#, user=USER, repo=REPO) 252 | 253 | #existing_label = labels_service.get('Bug') 254 | #label_names = ['Bug', ] 255 | labels = json.dumps(['invalid', 'bug', 'enhancement', 'duplicate'])#['Bug', 'invalid']) 256 | 257 | labels_service.add_to_issue(1, labels=labels) 258 | #*label_names) 259 | #labels_service.add_to_issue(1, labels=[('label1', 'ffcc00')], user=USER, repo=REPO) 260 | 261 | """ 262 | -------------------------------------------------------------------------------- /src/github_issues/label_map.py: -------------------------------------------------------------------------------- 1 | import csv 2 | import os 3 | from utils.msg_util import * 4 | 5 | class LabelInfo: 6 | ATTR_NAMES = """redmine_type redmine_name github_label_name github_label_color""".split() 7 | 8 | def __init__(self, row): 9 | if not row or not len(row) == 4: 10 | msgx('Expected 4 values in this row: %s' % row) 11 | 12 | for idx, item in enumerate(row): 13 | self.__dict__[self.ATTR_NAMES[idx]] = item.strip() 14 | 15 | def get_label_dict_info(self): 16 | return { self.redmine_name : self} 17 | 18 | class LabelMap: 19 | 20 | def __init__(self, label_map_fname): 21 | self.label_map_fname = label_map_fname 22 | #self.map_lookup = {} # redmine_type : { redmine_name : LabelInfo } 23 | self.label_lookup = {} # { redmine_name : LabelInfo } 24 | 25 | self.load_map_lookup() 26 | 27 | def get_key_count(self): 28 | return len(self.map_lookup) 29 | 30 | def get_label_info_objects(self): 31 | if not type(self.label_lookup) is dict: 32 | return None 33 | 34 | return self.label_lookup.values() 35 | 36 | def load_map_lookup(self): 37 | if not os.path.isfile(self.label_map_fname): 38 | msgx('Error: user name file not found: %s' % self.label_map_fname) 39 | 40 | msgt('Loading label map: %s' % self.label_map_fname) 41 | with open(self.label_map_fname, 'rU') as csvfile: 42 | map_reader = csv.reader(csvfile, delimiter=',')#, quotechar='|') 43 | row_num = 0 44 | for row in map_reader: 45 | row_num += 1 46 | if row_num == 1: continue # skip header row 47 | if len(row) == 0: continue 48 | if row[0].startswith('#'): continue 49 | label_info = LabelInfo(row) 50 | 51 | self.label_lookup.update(label_info.get_label_dict_info()) 52 | 53 | msg('Label dict loaded as follows.\nRemember: the "redmine_type" column in the .csv is ignored by the system--it is only for user convenience') 54 | dashes() 55 | for redmine_name, label_info in self.label_lookup.items(): 56 | msg('[%s] -> [%s][%s]' % (redmine_name, label_info.github_label_name, label_info.github_label_color)) 57 | 58 | 59 | def get_github_label_from_redmine_name(self, redmine_name): 60 | if not redmine_name: 61 | return None 62 | 63 | label_info = self.label_lookup.get(redmine_name.strip(), None) 64 | if label_info: 65 | return label_info.github_label_name 66 | return None 67 | -------------------------------------------------------------------------------- /src/github_issues/md_translate.py: -------------------------------------------------------------------------------- 1 | 2 | 3 | def get_translate_dict(): 4 | d = {} 5 | for hlevel in range(1, 7): 6 | d['h%s.' % hlevel] = '#' * hlevel # e.g. d['h2.'] = '##' 7 | d['\n# '] = '\n1. ' # lists 8 | d['
'] = '```'  # code block
 9 |     d['
'] = '\n```' # code block 10 | return d 11 | 12 | def translate_for_github(content): 13 | if not content: 14 | return None 15 | 16 | for k, v in get_translate_dict().items(): 17 | content = content.replace(k, v) 18 | 19 | return content 20 | -------------------------------------------------------------------------------- /src/github_issues/migration_manager.py: -------------------------------------------------------------------------------- 1 | import os, sys 2 | 3 | if __name__=='__main__': 4 | SRC_ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) 5 | sys.path.append(SRC_ROOT) 6 | 7 | import time 8 | import re 9 | import json 10 | from settings.base import get_github_auth, REDMINE_ISSUES_DIRECTORY, USER_MAP_FILE, LABEL_MAP_FILE, MILESTONE_MAP_FILE, REDMINE_TO_GITHUB_MAP_FILE 11 | 12 | 13 | from github_issues.user_map_helper import UserMapHelper 14 | from github_issues.github_issue_maker import GithubIssueMaker 15 | from utils.msg_util import * 16 | 17 | 18 | class MigrationManager: 19 | """Move the files to github""" 20 | 21 | def __init__(self, redmine_json_directory, redmine2github_map_file, **kwargs): 22 | 23 | self.redmine_json_directory = redmine_json_directory 24 | 25 | # Keep track of redmine issue #'s and related github issue #'s 26 | self.redmine2github_map_file = redmine2github_map_file 27 | 28 | 29 | self.include_comments = kwargs.get('include_comments', True) 30 | self.include_assignee = kwargs.get('include_assignee', True) 31 | 32 | 33 | self.user_mapping_filename = kwargs.get('user_mapping_filename', None) 34 | self.label_mapping_filename = kwargs.get('label_mapping_filename', None) 35 | self.milestone_mapping_filename = kwargs.get('milestone_mapping_filename', None) 36 | 37 | # Start loading with issue number (int) based on json file name 38 | self.redmine_issue_start_number = kwargs.get('redmine_issue_start_number', 0) 39 | 40 | # (optional) STOP loading at issue number (int) based on json file name. The stop issue number itself IS loaded 41 | # None = go to the end 42 | self.redmine_issue_end_number = kwargs.get('redmine_issue_end_number', None) 43 | 44 | def does_redmine_json_directory_exist(self): 45 | if not os.path.isdir(self.redmine_json_directory): 46 | return False 47 | return True 48 | 49 | def get_redmine_json_fnames(self): 50 | if not self.does_redmine_json_directory_exist(): 51 | msgx('ERROR: Directory does not exist: %s' % self.redmine_json_directory) 52 | 53 | pat ='^\d{1,10}\.json$' 54 | fnames = [x for x in os.listdir(self.redmine_json_directory) if re.match(pat, x)] 55 | fnames.sort() 56 | return fnames 57 | 58 | 59 | def sanity_check(self): 60 | # Is there a redmine JSON file directory with JSON files? 61 | fnames = self.get_redmine_json_fnames() 62 | if len(fnames)==0: 63 | msgx('ERROR: Directory [%s] does contain any .json files' % self.redmine_json_directory) 64 | 65 | for mapping_filename in [self.user_mapping_filename, self.label_mapping_filename, self.milestone_mapping_filename ]: 66 | if mapping_filename: # This mapping files may be None 67 | if not os.path.isfile(mapping_filename): 68 | msgx('ERROR: Mapping file not found [%s]' % mapping_filename) 69 | 70 | 71 | if not os.path.isdir(os.path.dirname(self.redmine2github_map_file)): 72 | msgx('ERROR: Directory not found for redmine2github_map_file [%s]' % self.redmine2github_map_file) 73 | 74 | 75 | if not type(self.redmine_issue_start_number) is int: 76 | msgx('ERROR: The start issue number is not an integer [%s]' % self.redmine_issue_start_number) 77 | 78 | if not type(self.redmine_issue_end_number) in (None, int): 79 | msgx('ERROR: The end issue number must be an integer of None [%s]' % self.redmine_issue_end_number) 80 | 81 | if type(self.redmine_issue_end_number) is int: 82 | if not self.redmine_issue_end_number >= self.redmine_issue_start_number: 83 | msgx('ERROR: The end issue number [%s] must greater than or equal to the start issue number [%s]' % (self.redmine_issue_end_number, self.redmine_issue_start_number)) 84 | 85 | 86 | def get_user_map_helper(self): 87 | if not self.user_mapping_filename: 88 | return None 89 | 90 | user_map_helper = UserMapHelper(self.user_mapping_filename) 91 | 92 | if user_map_helper.get_key_count() == 0: 93 | msgx('ERROR. get_user_map_helper. No names found in user map: %s' % self.user_mapping_filename) 94 | 95 | return user_map_helper 96 | 97 | 98 | def save_dict_to_file(self, d): 99 | 100 | d_str = json.dumps(d) 101 | fh = open(self.redmine2github_map_file, 'w') 102 | fh.write(d_str) 103 | fh.close() 104 | 105 | 106 | def get_dict_from_map_file(self): 107 | 108 | # new dict, file doesn't exist yet 109 | if not os.path.isfile(self.redmine2github_map_file): 110 | return {} # {redmine issue # : github issue #} 111 | 112 | fh = open(self.redmine2github_map_file, 'rU') 113 | content = fh.read() 114 | fh.close() 115 | 116 | # let it blow up if incorrect 117 | return json.loads(content) 118 | 119 | 120 | def migrate_related_tickets(self): 121 | """ 122 | After github issues are already migrated, go back and udpate the descriptions to include "related tickets 123 | 124 | 125 | """ 126 | 127 | gm = GithubIssueMaker() 128 | 129 | issue_cnt = 0 130 | redmine2github_issue_map = self.get_dict_from_map_file() 131 | 132 | for json_fname in self.get_redmine_json_fnames(): 133 | 134 | # Pull the issue number from the file name 135 | redmine_issue_num = int(json_fname.replace('.json', '')) 136 | 137 | # Start processing at or after redmine_issue_START_number 138 | if not redmine_issue_num >= self.redmine_issue_start_number: 139 | msg('Skipping Redmine issue: %s (start at %s)' % (redmine_issue_num, self.redmine_issue_start_number )) 140 | continue # skip Attempt to create issue 141 | # his 142 | 143 | # Don't process after the redmine_issue_END_number 144 | if self.redmine_issue_end_number: 145 | if redmine_issue_num > self.redmine_issue_end_number: 146 | print redmine_issue_num, self.redmine_issue_end_number 147 | break 148 | 149 | issue_cnt += 1 150 | 151 | msgt('(%s) Loading redmine issue: [%s] from file [%s]' % (issue_cnt, redmine_issue_num, json_fname)) 152 | 153 | json_fname_fullpath = os.path.join(self.redmine_json_directory, json_fname) 154 | 155 | gm.update_github_issue_with_related(json_fname_fullpath, redmine2github_issue_map) 156 | 157 | 158 | 159 | def migrate_issues(self): 160 | 161 | self.sanity_check() 162 | 163 | # Load a map if a filename was passed to the constructor 164 | # 165 | user_map_helper = self.get_user_map_helper() # None is ok 166 | # Note: for self.label_mapping_filename, None is ok 167 | gm = GithubIssueMaker(user_map_helper=user_map_helper\ 168 | , label_mapping_filename=self.label_mapping_filename\ 169 | , milestone_mapping_filename=self.milestone_mapping_filename 170 | ) 171 | 172 | # Iterate through json files 173 | issue_cnt = 0 174 | 175 | mapping_dict = self.get_dict_from_map_file() # { redmine issue : github issue } 176 | for json_fname in self.get_redmine_json_fnames(): 177 | 178 | # Pull the issue number from the file name 179 | redmine_issue_num = int(json_fname.replace('.json', '')) 180 | 181 | # Start processing at or after redmine_issue_START_number 182 | if not redmine_issue_num >= self.redmine_issue_start_number: 183 | msg('Skipping Redmine issue: %s (start at %s)' % (redmine_issue_num, self.redmine_issue_start_number )) 184 | continue # skip tAttempt to create issue 185 | # his 186 | 187 | # Don't process after the redmine_issue_END_number 188 | if self.redmine_issue_end_number: 189 | if redmine_issue_num > self.redmine_issue_end_number: 190 | print redmine_issue_num, self.redmine_issue_end_number 191 | break 192 | 193 | issue_cnt += 1 194 | 195 | msgt('(%s) Loading redmine issue: [%s] from file [%s]' % (issue_cnt, redmine_issue_num, json_fname)) 196 | json_fname_fullpath = os.path.join(self.redmine_json_directory, json_fname) 197 | gm_kwargs = { 'include_assignee' : self.include_assignee \ 198 | , 'include_comments' : self.include_comments \ 199 | } 200 | github_issue_number = gm.make_github_issue(json_fname_fullpath, **gm_kwargs) 201 | 202 | if github_issue_number: 203 | mapping_dict.update({ redmine_issue_num : github_issue_number}) 204 | self.save_dict_to_file(mapping_dict) 205 | 206 | if issue_cnt % 50 == 0: 207 | msgt('sleep 1 seconds....') 208 | time.sleep(1) 209 | 210 | if __name__=='__main__': 211 | json_input_directory = os.path.join(REDMINE_ISSUES_DIRECTORY, '2014-1224') 212 | 213 | kwargs = dict(include_comments=True\ 214 | , redmine_issue_start_number=1\ 215 | , redmine_issue_end_number=5000\ 216 | #, user_mapping_filename=USER_MAP_FILE # optional 217 | , include_assignee=False # Optional. Assignee must be in the github repo and USER_MAP_FILE above 218 | , label_mapping_filename=LABEL_MAP_FILE # optional 219 | #, milestone_mapping_filename=MILESTONE_MAP_FILE # optional 220 | ) 221 | mm = MigrationManager(json_input_directory\ 222 | , REDMINE_TO_GITHUB_MAP_FILE\ 223 | , **kwargs) 224 | 225 | #------------------------------------------------- 226 | # Run 1 - migrate issues from redmine to github 227 | #------------------------------------------------- 228 | mm.migrate_issues() 229 | 230 | #------------------------------------------------- 231 | # Run 2 - Using the issues maps created in Run 1 (redmine issue num -> new github issue num), 232 | # update github issues to include tickets to related tickets 233 | # 234 | #------------------------------------------------- 235 | #mm.migrate_related_tickets() 236 | 237 | 238 | 239 | -------------------------------------------------------------------------------- /src/github_issues/milestone_helper.py: -------------------------------------------------------------------------------- 1 | from __future__ import print_function 2 | import os 3 | import sys 4 | import json 5 | import csv 6 | 7 | if __name__=='__main__': 8 | SRC_ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) 9 | sys.path.append(SRC_ROOT) 10 | 11 | from datetime import datetime 12 | 13 | from jinja2 import Template 14 | from jinja2 import Environment, PackageLoader 15 | 16 | from utils.msg_util import * 17 | from github_issues.md_translate import translate_for_github 18 | 19 | 20 | #from settings.base import GITHUB_LOGIN, GITHUB_PASSWORD_OR_PERSONAL_ACCESS_TOKEN, GITHUB_TARGET_REPOSITORY, GITHUB_TARGET_USERNAME 21 | from settings.base import get_github_auth #GITHUB_LOGIN, GITHUB_PASSWORD_OR_PERSONAL_ACCESS_TOKEN, GITHUB_TARGET_REPOSITORY, GITHUB_TARGET_USERNAME 22 | from settings.base import REDMINE_SERVER#, REDMINE_API_KEY, REDMINE_ISSUES_DIRECTORY 23 | 24 | import pygithub3 25 | from datetime import datetime 26 | 27 | class MilestoneInfo: 28 | ATTR_NAMES = """redmine_milestone name due_date""".split() 29 | 30 | def __init__(self, row): 31 | if not row or not len(row) == 3: 32 | msgx('Expected 3 values in this row: %s' % row) 33 | 34 | for idx, item in enumerate(row): 35 | self.__dict__[self.ATTR_NAMES[idx]] = item.strip() 36 | 37 | if self.due_date == 'None': 38 | self.due_date = None 39 | else: 40 | self.due_date = datetime.strptime(self.due_date, '%Y-%m%d') 41 | 42 | def get_label_dict_info(self): 43 | return { self.redmine_milestone : self} 44 | 45 | class MilestoneHelper: 46 | """ 47 | Certain redmine attributes, such as "fixed_version", will be translated into milestones 48 | """ 49 | 50 | def __init__(self, milestone_mapping_filename=None): 51 | self.github_conn = None 52 | self.milestone_service = None 53 | self.milestone_mapping_filename = milestone_mapping_filename 54 | 55 | self.milestone_lookup = {} # { redmine_name : LabelInfo } 56 | self.using_milestone_map = False 57 | 58 | self.load_milestone_lookup() 59 | 60 | self.get_github_conn() 61 | 62 | 63 | def load_milestone_lookup(self): 64 | if self.milestone_mapping_filename is None: 65 | self.using_milestone_map = False 66 | return 67 | 68 | self.using_milestone_map = True 69 | if not os.path.isfile(self.milestone_mapping_filename): 70 | msgx('Error: user name file not found: %s' % self.milestone_mapping_filename) 71 | 72 | msgt('Loading milestone map: %s' % self.milestone_mapping_filename) 73 | with open(self.milestone_mapping_filename, 'rU') as csvfile: 74 | map_reader = csv.reader(csvfile, delimiter=',')#, quotechar='|') 75 | row_num = 0 76 | for row in map_reader: 77 | row_num += 1 78 | if row_num == 1: continue # skip header row 79 | if len(row) == 0: continue 80 | if row[0].startswith('#'): continue 81 | milestone_info = MilestoneInfo(row) 82 | 83 | self.milestone_lookup.update(milestone_info.get_label_dict_info()) 84 | 85 | msg('Label dict loaded as follows.\nRemember: the "redmine_type" column in the .csv is ignored by the system--it is only for user convenience') 86 | dashes() 87 | for redmine_name, milestone_info in self.milestone_lookup.items(): 88 | msg('[%s] -> [%s][%s]' % (redmine_name, milestone_info.name, milestone_info.due_date)) 89 | 90 | 91 | 92 | def get_github_conn(self): 93 | 94 | if self.github_conn is None: 95 | #auth = dict(login=GITHUB_LOGIN, password=GITHUB_PASSWORD_OR_PERSONAL_ACCESS_TOKEN, repo=GITHUB_TARGET_REPOSITORY, user=GITHUB_TARGET_USERNAME) 96 | self.github_conn = pygithub3.Github(**get_github_auth()) 97 | return self.github_conn 98 | 99 | def get_create_milestone_number(self, title): 100 | """Given a milestone title, retrieve the milestone number. 101 | If the milestone doesn't exist, then create it and return the new number 102 | """ 103 | if not title: 104 | return None 105 | 106 | mnum = self.get_mile_stone_number(title) 107 | if mnum: 108 | return mnum 109 | 110 | 111 | mstone = self.get_milestones_service().create({'title': title}) 112 | 113 | return mstone.number 114 | 115 | def get_mile_stone_number(self, title): 116 | """Given a milestone title, retrieve the milestone number. 117 | 118 | :param title: str, the title of the milestone 119 | :returns: int or None. The milestone number or None, if the milestone is not found 120 | """ 121 | 122 | if not title: 123 | return None 124 | 125 | milestones = self.get_milestones_service().list() 126 | """ 127 | for page in milestones: 128 | print('--page--') 129 | for resource in page: 130 | print('--resource--') 131 | print (resource) 132 | print (resource.number) 133 | print (resource.title) 134 | return 135 | """ 136 | for page in milestones: 137 | for resource in page: 138 | if resource.title == title: 139 | return resource.number 140 | return None 141 | 142 | 143 | def get_milestones_service(self): 144 | 145 | if self.milestone_service is None: 146 | self.milestone_service = pygithub3.services.issues.Milestones(**get_github_auth()) 147 | #labels_service = pygithub3.services.issues.Labels(**auth) 148 | # #labels_service = pygithub3.services.issues.Labels(**auth) 149 | #pygithub3.services.issues.Comments(**config) 150 | 151 | return self.milestone_service 152 | 153 | 154 | def get_create_milestone(self, redmine_issue_dict): 155 | # Add milestones! 156 | # 157 | # "fixed_version": { 158 | # "id": 96, 159 | # "name": "4.0 - review for weekly assignment" 160 | # }, 161 | # 162 | if not type(redmine_issue_dict) is dict: 163 | return None 164 | 165 | fixed_version = redmine_issue_dict.get('fixed_version', {}) 166 | if not fixed_version.has_key('name'): 167 | return None 168 | 169 | 170 | mstone_name = fixed_version['name'] 171 | msg('Milestone: %s' % mstone_name) 172 | if mstone_name: 173 | if self.using_milestone_map: 174 | mstone_info = self.milestone_lookup.get(mstone_name, None) 175 | if mstone_info is None: 176 | msgt('Milestone not found in map: %s' % mstone_name) 177 | mstone_name = mstone_name # Use original name 178 | else: 179 | mstone_name = mstone_info.name 180 | 181 | milestone_number = self.get_create_milestone_number(mstone_name) 182 | if not milestone_number: 183 | msgx('Milestone number not found for: [%s]' % mstone_name) 184 | 185 | return milestone_number 186 | 187 | return None 188 | 189 | # Add milestone to issue 190 | # mstone_dict = { 'milestone' : milestone_number} 191 | # print(mstone_dict) 192 | # issue_obj = self.get_github_conn().issues.update(issue_obj.number, mstone_dict) 193 | 194 | 195 | 196 | if __name__=='__main__': 197 | mstones = MilestoneHelper() 198 | #print(mstones.get_create_milestone_number('Pre-Alpha')) 199 | #print(mstones.get_mile_stone_number('Pre-Alpha')) 200 | print(mstones.get_create_milestone_number('We did it')) 201 | -------------------------------------------------------------------------------- /src/github_issues/templates/comment.md: -------------------------------------------------------------------------------- 1 | --- 2 | 3 | Original Redmine Comment 4 | {% if author_name %}Author Name: **{{ author_name }}** {% if author_github_username %}({{ author_github_username }}){% endif %}{% endif %} 5 | {% if note_date %}Original Date: {{ note_date }}{% endif %} 6 | 7 | --- 8 | 9 | {{ description }} 10 | -------------------------------------------------------------------------------- /src/github_issues/templates/description.md: -------------------------------------------------------------------------------- 1 | --- 2 | 3 | 4 | {% if author_name %}Author Name: **{{ author_name }}** {% if author_github_username %}({{ author_github_username }}){% endif %}{% endif %} 5 | {% if redmine_link %}Original Redmine Issue: {{ redmine_issue_num }}, {{redmine_link}}{% endif %} 6 | {% if start_date %}Original Date: {{ start_date }}{% endif %} 7 | {% if redmine_assignee %}Original Assignee: {{ redmine_assignee }}{% endif %} 8 | 9 | --- 10 | 11 | {{ description }} 12 | 13 | 14 | 15 | -------------------------------------------------------------------------------- /src/github_issues/templates/related_issues.md: -------------------------------------------------------------------------------- 1 | {{ original_description }} 2 | 3 | {% if related_issues or original_issues %} 4 | 5 | --- 6 | 7 | {% if related_issues %}Related issue(s): {{ related_issues }}{% endif %} 8 | {% if original_issues %}Redmine related issue(s): {{ original_issues }}{% endif %} 9 | 10 | --- 11 | {% endif %} 12 | 13 | {% if child_issues_github or child_issues_original %} 14 | --- 15 | 16 | {% if child_issues_github %}Child issue(s): {{ child_issues_github }}{% endif %} 17 | {% if child_issues_original %}Redmine child issue(s): {{ child_issues_original }}{% endif %} 18 | 19 | --- 20 | 21 | {% endif %} 22 | 23 | -------------------------------------------------------------------------------- /src/github_issues/user_map_helper.py: -------------------------------------------------------------------------------- 1 | import csv 2 | import os 3 | from utils.msg_util import * 4 | 5 | class UserMapHelper: 6 | 7 | def __init__(self, user_map_fname): 8 | self.user_map_fname = user_map_fname 9 | self.map_lookup = {} # Redmine username : Github Account 10 | self.load_map_lookup() 11 | 12 | def get_key_count(self): 13 | return len(self.map_lookup) 14 | 15 | def load_map_lookup(self): 16 | if not os.path.isfile(self.user_map_fname): 17 | msgx('Error: user name file not found: %s' % self.user_map_fname) 18 | 19 | with open(self.user_map_fname, 'rU') as csvfile: 20 | map_reader = csv.reader(csvfile, delimiter=',')#, quotechar='|') 21 | row_num = 0 22 | for row in map_reader: 23 | row_num +=1 24 | if row_num == 1: continue # skip header row 25 | if len(row) == 0: continue 26 | if row[0].startswith('#'): continue 27 | 28 | if len(row) ==2: 29 | if row[1].strip(): 30 | self.map_lookup[row[0].strip()] = row[1].strip() 31 | 32 | msgt('User map loaded with %s names' % len(self.map_lookup)) 33 | for k, v in self.map_lookup.items(): 34 | msg('[%s] -> [%s]' % (k, v)) 35 | 36 | def get_github_user(self, redmine_name, with_github_at=True): 37 | if not redmine_name: 38 | return None 39 | 40 | github_name = self.map_lookup.get(redmine_name.strip(), None) 41 | if github_name is None: 42 | return None 43 | 44 | if with_github_at: 45 | return '@' + github_name 46 | 47 | return github_name -------------------------------------------------------------------------------- /src/redmine_ticket/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/IQSS/redmine2github/57d50bcf10952bcdd87174ac208e8dfebccec8ae/src/redmine_ticket/__init__.py -------------------------------------------------------------------------------- /src/redmine_ticket/redmine_issue_downloader.py: -------------------------------------------------------------------------------- 1 | from __future__ import print_function 2 | import os 3 | from os.path import dirname, join, abspath, isdir 4 | import sys 5 | import json 6 | import requests 7 | try: 8 | from urlparse import urljoin 9 | except: 10 | from urllib.parse import urljoin # python 3.x 11 | 12 | # http://python-redmine.readthedocs.org/ 13 | from redmine import Redmine 14 | 15 | if __name__=='__main__': 16 | SRC_ROOT = dirname(dirname(abspath(__file__))) 17 | sys.path.append(SRC_ROOT) 18 | 19 | from datetime import datetime 20 | from utils.msg_util import * 21 | 22 | class RedmineIssueDownloader: 23 | """ 24 | For a given Redmine project, download the issues in JSON format 25 | """ 26 | 27 | #TIME_FORMAT_STRING = '%Y-%m%d-%H%M' 28 | TIME_FORMAT_STRING = '%Y-%m%d' 29 | 30 | # Redmine tickets are written to JSON files with the naming convention "(issue id).json" 31 | # For file sorting, preceding zeros are tacked on. 32 | # Example: If ZERO_PADDING_LEVEL=5: 33 | # issue #375 is written to file "00375.json" 34 | # issue #1789 is written to file "01789.json" 35 | # issue #2 is written to file "00002.json" 36 | # 37 | # If your issue numbers go beyond 99,999 then increase the ZERO_PADDING_LEVEL 38 | # 39 | ZERO_PADDING_LEVEL = 5 40 | 41 | def __init__(self, redmine_server, redmine_api_key, project_name_or_identifier, issues_base_directory, **kwargs): 42 | """ 43 | Constructor 44 | 45 | :param redmine_server: str giving the url of the redmine server. e.g. https://redmine.myorg.edu/ 46 | :param redmine_api_key: str with a redmine api key 47 | :param project_name_or_identifier: str or int with either the redmine project id or project identifier 48 | :param issues_base_directory: str, directory to download the redmine issues in JSON format. Directory will be crated 49 | :param specific_tickets_to_download: optional, list of specific ticket numbers to download. e.g. [2215, 2216, etc] 50 | """ 51 | self.redmine_server = redmine_server 52 | self.redmine_api_key = redmine_api_key 53 | self.project_name_or_identifier = project_name_or_identifier 54 | self.issues_base_directory = issues_base_directory 55 | self.issue_status = kwargs.get('issue_status', '*') # values 'open', 'closed', '*' 56 | 57 | self.specific_tickets_to_download = kwargs.get('specific_tickets_to_download', None) 58 | 59 | self.redmine_conn = None 60 | self.redmine_project = None 61 | 62 | self.issue_dirname = join(self.issues_base_directory\ 63 | , datetime.today().strftime(RedmineIssueDownloader.TIME_FORMAT_STRING)\ 64 | ) 65 | 66 | self.setup() 67 | 68 | def setup(self): 69 | self.connect_to_redmine() 70 | if not isdir(self.issue_dirname): 71 | os.makedirs(self.issue_dirname) 72 | msgt('Directory created: %s' % self.issue_dirname) 73 | 74 | def connect_to_redmine(self): 75 | self.redmine_conn = Redmine(self.redmine_server, key=self.redmine_api_key) 76 | self.redmine_project = self.redmine_conn.project.get(self.project_name_or_identifier) 77 | msg('Connected to server [%s] project [%s]' % (self.redmine_server, self.project_name_or_identifier)) 78 | 79 | 80 | def get_issue_count(self): 81 | msgt('get_issue_count') 82 | 83 | issue_query_str = 'issues.json?project_id=%s&limit=1&status_id=%s' \ 84 | % (self.project_name_or_identifier, self.issue_status) 85 | 86 | url = urljoin(self.redmine_server, issue_query_str) 87 | 88 | msg('Issue count url: %s' % url) 89 | 90 | # Note: Auth purposely uses the API KEY "as a username with a random password via HTTP Basic authentication" 91 | # from: http://www.redmine.org/projects/redmine/wiki/Rest_api 92 | # 93 | auth = (self.redmine_api_key, 'random-pw') 94 | r = requests.get(url, auth=auth) 95 | if not r.status_code == 200: 96 | msgt('Error!') 97 | msg(r.text) 98 | raise Exception("Request for issue count failed! Status code: %s\nUrl: %s\nAuth:%s" % (r.status_code, url, auth)) 99 | 100 | msg('Convert result to JSON') 101 | try: 102 | data = r.json() # Let it blow up 103 | except: 104 | msgt('Error!') 105 | msg('Data from request (as text): %s' % r.text) 106 | raise Exception('Failed to convert issue count data to JSON.\nUrl: %s\nAuth:%s" % (url, auth)') 107 | 108 | if not data.has_key('total_count'): 109 | msgx('Total count not found in data: \n[%s]' % data) 110 | 111 | return data['total_count'] 112 | 113 | """ 114 | from __future__ import print_function 115 | import requests 116 | 117 | project_id = 'dvn' 118 | redmine_api_key = 'some-key' 119 | url = 'https://redmine.hmdc.harvard.edu/issues.json?project_id=%s&limit=1' % project_id 120 | 121 | #--------------------- 122 | # Alternative 1 123 | #--------------------- 124 | auth = (redmine_api_key, 'random-pw') 125 | r = requests.get(url, auth=auth) 126 | print (r.text) 127 | print (r.status_code) 128 | data = r.json() 129 | print (data['total_count']) 130 | 131 | #--------------------- 132 | # Alternative 2 133 | #--------------------- 134 | url2 = '%s&key=%s' % (url, redmine_api_key) 135 | r = requests.get(url2) 136 | print (r.text) 137 | print (r.status_code) 138 | data = r.json() 139 | print (data['total_count']) 140 | 141 | 142 | """ 143 | 144 | def write_issue_list(self, issue_fname, issue_dict): 145 | if issue_fname is None or not type(issue_dict) == dict: 146 | msgx('ERROR: write_issue_list, issue_fname is None or issue_dict not dict') 147 | return 148 | fh = open(issue_fname, 'w') 149 | fh.write(json.dumps(issue_dict)) 150 | fh.close() 151 | msg('file updated: %s' % issue_fname) 152 | 153 | def show_project_info(self): 154 | msg(self.redmine_project._attributes) 155 | 156 | 157 | def download_tickets2(self): 158 | """ 159 | fyi: Retrieving total count via regular api, not python redmine package 160 | """ 161 | issue_dict = {} 162 | issue_fname = join(self.issue_dirname, 'issue_list.json') 163 | msg('Gathering issue information.... (may take a minute)') 164 | 165 | ticket_cnt = self.get_issue_count() 166 | 167 | RECORD_RETRIEVAL_SIZE = 100 168 | 169 | num_loops = ticket_cnt / RECORD_RETRIEVAL_SIZE 170 | extra_recs = ticket_cnt % RECORD_RETRIEVAL_SIZE 171 | if extra_recs > 0: 172 | num_loops+=1 173 | #num_loops=3 174 | msg('num_loops: %d' % num_loops) 175 | msg('extra_recs: %d' % extra_recs) 176 | 177 | cnt = 0 178 | for loop_num in range(0, num_loops): 179 | start_record = loop_num * RECORD_RETRIEVAL_SIZE 180 | end_record = (loop_num+1) * RECORD_RETRIEVAL_SIZE 181 | 182 | msgt('Retrieve records via idx (skip last): %s - %s' % (start_record, end_record)) 183 | 184 | # limit of 100 is returning 125 185 | rec_cnt = 0 186 | for item in self.redmine_conn.issue.filter(project_id=self.project_name_or_identifier, status_id=self.issue_status, sort='id', offset=start_record)[:RECORD_RETRIEVAL_SIZE]: #, limit=RECORD_RETRIEVAL_SIZE): #[start_record:end_record] 187 | rec_cnt +=1 188 | cnt +=1 189 | msg('(%s) %s - %s' % (rec_cnt, item.id, item.subject)) 190 | 191 | if self.specific_tickets_to_download is not None: 192 | # only download specific tickets 193 | # 194 | if item.id in self.specific_tickets_to_download: 195 | self.save_single_issue(item) 196 | issue_dict[self.pad_issue_id(item.id)] = item.subject 197 | continue # go to next item 198 | else: 199 | # Get all tickets 200 | # 201 | self.save_single_issue(item) 202 | issue_dict[self.pad_issue_id(item.id)] = item.subject 203 | if rec_cnt == RECORD_RETRIEVAL_SIZE: 204 | break 205 | #continue 206 | #self.save_single_issue(item) 207 | self.write_issue_list(issue_fname, issue_dict) 208 | 209 | 210 | def pad_issue_id(self, issue_id): 211 | if issue_id is None: 212 | msgx('ERROR. pad_issue_id. The "issue_id" is None') 213 | 214 | return ('%s' % issue_id).zfill(self.ZERO_PADDING_LEVEL) 215 | 216 | def save_single_issue(self, single_issue): 217 | """ 218 | Write a single issue object to a file using JSON format 219 | 220 | :param single_issue: Issue object 221 | """ 222 | if single_issue is None: 223 | msgx('ERROR. download_single_issue. The "single_issue" is None') 224 | 225 | ## FIX: Expensive adjustment -- to pull out full relation and journal info 226 | json_str = self.get_single_issue(single_issue.id) # another call to redmine 227 | 228 | #json_str = json.dumps(single_issue._attributes, indent=4) 229 | 230 | fullpath = join(self.issue_dirname, self.pad_issue_id(single_issue.id) + '.json') 231 | open(fullpath, 'w').write(json_str) 232 | msg('Ticket retrieved: %s' % fullpath) 233 | 234 | 235 | 236 | def process_files(self, issues_dirname=None): 237 | if issues_dirname is None: 238 | issues_dirname = self.issue_dirname 239 | 240 | tracker_info = [] 241 | status_info = [] 242 | priority_info = [] 243 | 244 | fnames = [x for x in os.listdir(issues_dirname) if x.endswith('.json')] 245 | for fname in fnames: 246 | content = open(join(issues_dirname, fname), 'rU').read() 247 | d = json.loads(content) 248 | 249 | 250 | # Tracker Info 251 | tracker = d.get('tracker', None) 252 | if tracker: 253 | tracker_str = '%s|%s' % (tracker['id'], tracker['name']) 254 | if not tracker_str in tracker_info: 255 | tracker_info.append(tracker_str) 256 | # Status Info 257 | status = d.get('status', None) 258 | if status: 259 | status_str = '%s|%s' % (status['id'], status['name']) 260 | if not status_str in status_info: 261 | status_info.append(status_str) 262 | 263 | # Priority Info 264 | priority = d.get('priority', None) 265 | if priority: 266 | priority_str = '%s|%s' % (priority['id'], priority['name']) 267 | if not priority_str in priority_info: 268 | priority_info.append(priority_str) 269 | #print d.keys() 270 | msg(tracker_info) 271 | msg(status_info) 272 | msg(priority_info) 273 | 274 | 275 | def get_single_issue(self, issue_id): 276 | """ 277 | Download a single issue 278 | 279 | :param ticket_id: int of issue id in redmine 280 | :returns: json string with issue information 281 | """ 282 | # test using .issue.get 283 | issue = self.redmine_conn.issue.get(issue_id, include='children,journals,watchers,relations') 284 | json_str = json.dumps(issue._attributes, indent=4) 285 | msg('Issue retrieved: %s' % issue_id) 286 | return json_str 287 | 288 | 289 | if __name__=='__main__': 290 | from settings.base import REDMINE_SERVER, REDMINE_PROJECT_ID, REDMINE_API_KEY, REDMINE_ISSUES_DIRECTORY 291 | #rn = RedmineIssueDownloader(REDMINE_SERVER, REDMINE_API_KEY, 'dvn', REDMINE_ISSUES_DIRECTORY) 292 | #Only import some specific tickets 293 | #kwargs = dict(specific_tickets_to_download=[1371, 1399, 1843, 2214, 2215, 2216, 3362, 3387, 3397, 3400, 3232, 3271, 3305, 3426, 3425, 3313, 3208]) 294 | rn = RedmineIssueDownloader(REDMINE_SERVER, REDMINE_API_KEY, REDMINE_PROJECT_ID, REDMINE_ISSUES_DIRECTORY) 295 | rn.download_tickets2() 296 | 297 | msg(rn.get_issue_count()) 298 | #rn.show_project_info() 299 | #rn.process_files() 300 | #msg(rn.get_single_issue(3232)) 301 | 302 | """ 303 | import json 304 | c = open('issue_list2.txt', 'rU').read() 305 | d = json.loads(c) 306 | print(len(d)) 307 | """ 308 | -------------------------------------------------------------------------------- /src/redmine_ticket/redmine_issue_updater.py: -------------------------------------------------------------------------------- 1 | from __future__ import print_function 2 | import os 3 | from os.path import dirname, join, abspath, isdir 4 | import sys 5 | import json 6 | import urllib2 7 | 8 | try: 9 | from urlparse import urljoin 10 | except: 11 | from urllib.parse import urljoin # python 3.x 12 | 13 | # http://python-redmine.readthedocs.org/ 14 | from redmine import Redmine 15 | 16 | if __name__=='__main__': 17 | SRC_ROOT = dirname(dirname(abspath(__file__))) 18 | sys.path.append(SRC_ROOT) 19 | 20 | 21 | from jinja2 import Template 22 | from jinja2 import Environment, PackageLoader 23 | 24 | from utils.msg_util import * 25 | from settings.base import GITHUB_TARGET_REPOSITORY, GITHUB_TARGET_USERNAME, get_gethub_issue_url 26 | from redmine_ticket.redmine_issue_downloader import RedmineIssueDownloader 27 | 28 | class RedmineIssueUpdater: 29 | """ 30 | For a Redmine project moved to github, update the redmine ticket with the github link 31 | """ 32 | 33 | #TIME_FORMAT_STRING = '%Y-%m%d-%H%M' 34 | TIME_FORMAT_STRING = '%Y-%m%d' 35 | 36 | # Redmine tickets are written to JSON files with the naming convention "(issue id).json" 37 | # For file sorting, preceding zeros are tacked on. 38 | # Example: If ZERO_PADDING_LEVEL=5: 39 | # issue #375 is written to file "00375.json" 40 | # issue #1789 is written to file "01789.json" 41 | # issue #2 is written to file "00002.json" 42 | # 43 | # If your issue numbers go beyond 99,999 then increase the ZERO_PADDING_LEVEL 44 | # 45 | def __init__(self, redmine_server, redmine_api_key, project_name_or_identifier, issues_dirname, redmine2github_id_map_filename): 46 | """ 47 | Constructor 48 | 49 | :param redmine_server: str giving the url of the redmine server. e.g. https://redmine.myorg.edu/ 50 | :param redmine_api_key: str with a redmine api key 51 | :param project_name_or_identifier: str or int with either the redmine project id or project identifier 52 | :param issues_base_directory: str, directory to download the redmine issues in JSON format. Directory will be crated 53 | """ 54 | self.redmine_server = redmine_server 55 | self.redmine_api_key = redmine_api_key 56 | self.project_name_or_identifier = project_name_or_identifier 57 | self.issue_dirname = issues_dirname 58 | msg('redmine2github_id_map_filename: %s' % redmine2github_id_map_filename) 59 | self.redmine2github_id_map = json.loads(open(redmine2github_id_map_filename, 'rU').read()) 60 | 61 | self.redmine_conn = None 62 | self.redmine_project = None 63 | 64 | self.jinja_env = Environment(loader=PackageLoader('redmine_ticket', 'templates')) 65 | 66 | self.setup() 67 | 68 | def setup(self): 69 | self.connect_to_redmine() 70 | if not isdir(self.issue_dirname): 71 | msgx('Directory doesn\'t exist: %s' % self.issue_dirname) 72 | 73 | 74 | def connect_to_redmine(self): 75 | self.redmine_conn = Redmine(self.redmine_server, key=self.redmine_api_key) 76 | self.redmine_project = self.redmine_conn.project.get(self.project_name_or_identifier) 77 | msg('Connected to server [%s] project [%s]' % (self.redmine_server, self.project_name_or_identifier)) 78 | 79 | 80 | def update_tickets(self): 81 | 82 | redmine_keys = self.redmine2github_id_map.keys() 83 | redmine_keys.sort() 84 | redmine_keys = set(redmine_keys) 85 | 86 | ticket_cnt = 0 87 | for redmine_issue_num in redmine_keys: 88 | ticket_cnt +=1 89 | 90 | msgt('(%s) Updating redmine ticket: %s' % (ticket_cnt, redmine_issue_num)) 91 | github_issue_id = self.redmine2github_id_map.get(redmine_issue_num) 92 | msg('github_issue_id: %s' % github_issue_id) 93 | 94 | fname = redmine_issue_num.zfill(RedmineIssueDownloader.ZERO_PADDING_LEVEL) + '.json' 95 | redmine_issue_fname = os.path.join(self.issue_dirname, fname) 96 | if not os.path.isfile(redmine_issue_fname): 97 | msgx('file not found: %s' % redmine_issue_fname) 98 | redmine_issue_dict = json.loads(open(redmine_issue_fname, 'rU').read()) 99 | 100 | github_issue_url = get_gethub_issue_url(github_issue_id) 101 | 102 | template = self.jinja_env.get_template('description_with_github_link.md') 103 | 104 | original_description = redmine_issue_dict.get('description', None) 105 | #if not original_description: 106 | # msgx('Description not found in file: %s' % redmine_issue_fname) 107 | 108 | template_params = { 'original_description' : original_description\ 109 | , 'github_issue_id' : github_issue_id\ 110 | , 'github_repo' : GITHUB_TARGET_REPOSITORY\ 111 | , 'github_username' : GITHUB_TARGET_USERNAME\ 112 | , 'github_issue_url' : github_issue_url\ 113 | } 114 | 115 | updated_description = template.render(template_params) 116 | 117 | #msg(updated_description) 118 | 119 | update_params = dict(project_id=self.project_name_or_identifier\ 120 | , description=updated_description\ 121 | ) 122 | 123 | 124 | updated_record = self.redmine_conn.issue.update(resource_id=int(redmine_issue_num)\ 125 | , **update_params) 126 | 127 | dashes() 128 | 129 | if updated_record is True: 130 | msg('-----> Updated!') 131 | else: 132 | msgx('Updated Failed!') 133 | 134 | 135 | 136 | 137 | 138 | if __name__=='__main__': 139 | from settings.base import REDMINE_SERVER, REDMINE_API_KEY, REDMINE_ISSUES_DIRECTORY, REDMINE_TO_GITHUB_MAP_FILE 140 | 141 | issues_dir = os.path.join(REDMINE_ISSUES_DIRECTORY, '2014-0902') 142 | #rn = RedmineIssueDownloader(REDMINE_SERVER, REDMINE_API_KEY, 'dvn', REDMINE_ISSUES_DIRECTORY) 143 | rn = RedmineIssueUpdater(REDMINE_SERVER, REDMINE_API_KEY, 1, issues_dir, REDMINE_TO_GITHUB_MAP_FILE) 144 | rn.update_tickets() -------------------------------------------------------------------------------- /src/redmine_ticket/templates/description_with_github_link.md: -------------------------------------------------------------------------------- 1 | {% if github_issue_url %} 2 | 3 | h1. !! Ticket moved to GitHub: "{{github_username}}/{{github_repo}}/{{ github_issue_id }}":{{ github_issue_url }} 4 | {% endif %} 5 | 6 | 7 | {% if original_description %} 8 | ---- 9 | 10 | 11 | {{ original_description }} 12 | {% endif %} -------------------------------------------------------------------------------- /src/settings/Workbook1.xlsx: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/IQSS/redmine2github/57d50bcf10952bcdd87174ac208e8dfebccec8ae/src/settings/Workbook1.xlsx -------------------------------------------------------------------------------- /src/settings/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/IQSS/redmine2github/57d50bcf10952bcdd87174ac208e8dfebccec8ae/src/settings/__init__.py -------------------------------------------------------------------------------- /src/settings/base.py: -------------------------------------------------------------------------------- 1 | 2 | import settings.local as config 3 | 4 | # 5 | # Redmine API information 6 | # https://redmine.hmdc.harvard.edu 7 | # 8 | REDMINE_SERVER = config.REDMINE_SERVER 9 | REDMINE_PROJECT_ID = config.REDMINE_PROJECT_ID 10 | REDMINE_API_KEY = config.REDMINE_API_KEY 11 | 12 | 13 | # 14 | # GitHub API information 15 | # https://github.com/blog/1509-personal-api-tokens 16 | # 17 | GITHUB_SERVER = config.GITHUB_SERVER 18 | GITHUB_LOGIN = config.GITHUB_LOGIN 19 | GITHUB_PASSWORD_OR_PERSONAL_ACCESS_TOKEN = config.GITHUB_PASSWORD_OR_PERSONAL_ACCESS_TOKEN 20 | 21 | GITHUB_TARGET_REPOSITORY = config.GITHUB_TARGET_REPOSITORY 22 | GITHUB_TARGET_USERNAME = config.GITHUB_TARGET_USERNAME 23 | 24 | 25 | # 26 | # Working files directory 27 | # 28 | WORKING_FILES_DIRECTORY = config.WORKING_FILES_DIRECTORY 29 | REDMINE_ISSUES_DIRECTORY = config.REDMINE_ISSUES_DIRECTORY 30 | 31 | # JSON file mapping { redmine issue # : github issue # } 32 | REDMINE_TO_GITHUB_MAP_FILE = config.REDMINE_TO_GITHUB_MAP_FILE 33 | 34 | # (optional) csv file mapping Redmine users to github users. 35 | # Manually created. Doesn't check for name collisions 36 | # example, see settings/sample_user_map.csv 37 | USER_MAP_FILE = config.USER_MAP_FILE 38 | 39 | # (optional) csv file mapping Redmine status, tracker, priority, and custom fields names to github labels. 40 | # Manually created. Doesn't check for name collisions 41 | # example, see settings/sample_label_map.csv 42 | LABEL_MAP_FILE = config.LABEL_MAP_FILE 43 | 44 | # (optional) csv file mapping Redmine "target version" to GitHub milestones. 45 | # Manually created. Doesn't check for name collisions 46 | # example, see settings/sample_milestone_map.csv 47 | MILESTONE_MAP_FILE = config.MILESTONE_MAP_FILE 48 | 49 | 50 | def get_gethub_issue_url(issue_id=None): 51 | """ 52 | Used by the "redmine_issue_updater" to add links back to the original redmine tickets 53 | """ 54 | issue_url = 'https://github.com/%s/%s/issues' % (GITHUB_TARGET_USERNAME, GITHUB_TARGET_REPOSITORY) 55 | 56 | if issue_id: 57 | return '%s/%s' % (issue_url, issue_id) 58 | return issue_url 59 | 60 | def get_github_auth(): 61 | return dict(login=GITHUB_LOGIN, password=GITHUB_PASSWORD_OR_PERSONAL_ACCESS_TOKEN, repo=GITHUB_TARGET_REPOSITORY, user=GITHUB_TARGET_USERNAME) 62 | -------------------------------------------------------------------------------- /src/settings/local_sample.py: -------------------------------------------------------------------------------- 1 | import getpass 2 | from os.path import abspath, dirname, join 3 | import sys 4 | 5 | PROJECT_ROOT = dirname(dirname(dirname(abspath(__file__)))) 6 | sys.path.append(PROJECT_ROOT) 7 | 8 | # 9 | # Redmine API information 10 | # 11 | REDMINE_SERVER = 'https://redmine.my-org.edu' 12 | REDMINE_PROJECT_ID = 'PROJ_ID' # Found in project url: http://redmine.my-org.edu/projects/PROJ_ID 13 | 14 | # See http://www.redmine.org/projects/redmine/wiki/Rest_api#Authentication 15 | # "You can find your API key on your account page..." 16 | REDMINE_API_KEY = 'my-api-key from remdine account page' 17 | 18 | GITHUB_SERVER = 'https://api.github.com' 19 | GITHUB_LOGIN = 'github username' 20 | GITHUB_PASSWORD_OR_PERSONAL_ACCESS_TOKEN = getpass.getpass('Enter github pw:') 21 | 22 | GITHUB_TARGET_REPOSITORY = 'test-issue-migrate' 23 | GITHUB_TARGET_USERNAME = 'target-repo-github-username' 24 | 25 | WORKING_FILES_DIRECTORY = join(PROJECT_ROOT, 'working_files') 26 | REDMINE_ISSUES_DIRECTORY = join(WORKING_FILES_DIRECTORY, 'redmine_issues') 27 | 28 | # JSON file mapping { redmine issue # : github issue # } 29 | REDMINE_TO_GITHUB_MAP_FILE = join(WORKING_FILES_DIRECTORY, 'redmine2github_issue_map.json') 30 | 31 | # (optional) csv file mapping Redmine users to github users. 32 | # Manually created. Doesn't check for name collisions 33 | # example, see settings/sample_user_map.csv 34 | USER_MAP_FILE = join(WORKING_FILES_DIRECTORY, 'redmine2github_user_map.csv') 35 | 36 | # (optional) csv file mapping Redmine status, tracker, priority, and custom fields names to github labels. 37 | # Manually created. Doesn't check for name collisions 38 | # example, see settings/sample_label_map.csv 39 | LABEL_MAP_FILE = join(WORKING_FILES_DIRECTORY, 'redmine2github_label_map.csv') 40 | 41 | # (optional) csv file mapping Redmine "target version" to GitHub milestones. 42 | # Manually created. Doesn't check for name collisions 43 | # example, see settings/sample_milestone_map.csv 44 | MILESTONE_MAP_FILE = join(WORKING_FILES_DIRECTORY, 'redmine2github_milestone_map.csv') 45 | 46 | def get_github_auth(): 47 | return dict(login=GITHUB_LOGIN, password=GITHUB_PASSWORD_OR_PERSONAL_ACCESS_TOKEN, repo=GITHUB_TARGET_REPOSITORY, user=GITHUB_TARGET_USERNAME) 48 | -------------------------------------------------------------------------------- /src/settings/sample_label_map.csv: -------------------------------------------------------------------------------- 1 | redmine_type, redmine_name, github_label_name, github_label_color status, New, Status 1: New, FF6600 status, In Review, Status 2: In Review, CC6600 status, In Design, Status 3: In Design/Dev,996600 status, In Dev, Status 3: In Design/Dev,996600 status, In Code Review, Status 4: In Code Review,666600 status, In QA, Status 5: In QA,009933 status, Completed, Status 6: Completed,009933 status, Rejected, Status 7: Rejected, 00FF99 priority, Low, Priority 5: Low, 009933 priority, Normal, Priority 4: Normal, 666600 priority, High, Priority 3: High, 996600 priority, Urgent, Priority 2: Urgent, FF6600 priority, Immediate, Priority 1: Immediate, FF0000 tracker, Bug, Tracker 1: Bug, FF0000 tracker, Feature, Tracker 2: Feature, FF6600 tracker, Suggestion, Tracker 3: Suggestion, 996600 tracker, Support, Tracker 4: Support, 666600 tracker, Documentation, Tracker 5 Documentation,009933 Category, API, Category: API, 00CC99 Category, File Upload & Handling, Category: File Upload & Handling, 009999 Category, high-level, Category: High-level, 006699 Category, Metadata, Category: Metadata, 003399 Category, Migration, Category: Migration, 000099 Category, Search/Browse, Category: Search/Browse,3300CC Category, UX & Category Upgrade, Category: UX & Category Upgrade, 6600FF -------------------------------------------------------------------------------- /src/settings/sample_label_map2.csv: -------------------------------------------------------------------------------- 1 | redmine_type, redmine_name, github_label_name, github_label_color 2 | status, New, Status: Design, 66ff66 3 | status, In Review, Status: Design, 66ff66 4 | status, In Design, Status: Design, 66ff66 5 | status, In Dev, Status: Dev, 66cc66 6 | status, In Code Review, Status: QA, 336633 7 | status, In QA, Status: QA, 336633 8 | priority, Blank, Priority: Critical, ff9900 9 | priority, High, Priority: High, e11d21 10 | priority, Urgent, Priority: High, e11d21 11 | priority, Immediate, Priority: High, e11d21 12 | priority, Low, Priority: Medium, cc6666 13 | priority, Normal, Priority: Medium, cc6666 14 | tracker, Bug, Type: Bug, fef2c0 15 | tracker, Feature, Type: Feature, fef2c0 16 | tracker, Support, Type: Feature, fef2c0 17 | tracker, Documentation, Type: Feature, fef2c0 18 | tracker, Suggestion, Type: Suggestion, fef2c0 19 | Component, API, Component: API, 3399ff 20 | Component, File Upload & Handling, Component: File Upload & Handling, c7def8 21 | Component, high-level, Component: High-level, c7def8 22 | Component, Metadata, Component: Metadata, c7def8 23 | Component, Migration, Component: Migration, c7def8 24 | Component, Search/Browse, Component: Search/Browse, c7def8 25 | Component, UX & Component Upgrade, Component: UX & Component Upgrade, c7def8 26 | -------------------------------------------------------------------------------- /src/settings/sample_milestone_map.csv: -------------------------------------------------------------------------------- 1 | redmine_milestone, github_milestone_name, due_date_yyyy_mm_dd 4.0 - week of March 10, Dataverse 4.0: Beta 1, 2014-0601 4.0 - week of March 17, Dataverse 4.0: Beta 1, 2014-0601 4.0 - week of March 24, Dataverse 4.0: Beta 1, 2014-0601 4.0 - week of March 31, Dataverse 4.0: Beta 1, 2014-0601 4.0 - week of April 7, Dataverse 4.0: Beta 1, 2014-0601 4.0 - week of April 14, Dataverse 4.0: Beta 1, 2014-0601 4.0 - week of April 21, Dataverse 4.0: Beta 1, 2014-0601 4.0 - week of April 28, Dataverse 4.0: Beta 1, 2014-0601 4.0 - Beta1, Dataverse 4.0: Beta 1, 2014-0601 4.0 - week of May 12, Dataverse 4.0: Beta 2, 2014-0601 4.0-Beta 1.1, Dataverse 4.0: Beta 1, 2014-0601 4.0 - week of May 19, Dataverse 4.0: Beta 1, 2014-0601 4.0 - week of May 26, Dataverse 4.0: Beta 1, 2014-0601 4.0 - week of June 2, Dataverse 4.0: Beta 1, 2014-0601 4.0 - week of June 9, Dataverse 4.0: Beta 1, 2014-0601 4.0 - week of June 16, Dataverse 4.0: Beta 1, 2014-0601 4.0 - week of June 23, Dataverse 4.0: Beta 1, 2014-0601 4.0 - Final, Dataverse 4.0: Beta 1, 2014-0601 3.6 patch, Dataverse 3.6: Patch, 2014-0601 4.0 - Triage, Dataverse 4.0 Triage, 2014-0610 4.0 - review for post 4.0, Dataverse 4.0: Beta 1, 2014-0601 4.0 - review for weekly assignment, Dataverse 4.0: Beta 1, 2014-0601 4.1, Dataverse 4.0: Release, 2014-1027 "Future: Ingest, File Upload", Future - Dataverse 4.x, None Future: Internationalization, Future - Dataverse 4.x, None "Future: Performance, Downloads", Future - Dataverse 4.x, None Future: Security / Shibboleth, Future - Dataverse 4.x, None Future: Subset / Analysis separation, Future - Dataverse 4.x, None Future: World Map integration, Future - Dataverse 4.x, None Review Post 4.0, Future - Dataverse 4.x, None Review: Clean up issues, Future - Dataverse 4.x, None Review: Harvesting / new data, Future - Dataverse 4.x, None Review: before each new release, Future - Dataverse 4.x, None Review: for 4.1, Future - Dataverse 4.x, None -------------------------------------------------------------------------------- /src/settings/sample_milestone_map.xlsx: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/IQSS/redmine2github/57d50bcf10952bcdd87174ac208e8dfebccec8ae/src/settings/sample_milestone_map.xlsx -------------------------------------------------------------------------------- /src/settings/sample_milestone_map2.csv: -------------------------------------------------------------------------------- 1 | redmine_milestone, github_milestone_name, due_date_yyyy_mm_dd 4.0 - week of March 10, Dataverse 4.0: Beta 1, 2014-0601 4.0 - week of March 17, Dataverse 4.0: Beta 1, 2014-0601 4.0 - week of March 24, Dataverse 4.0: Beta 1, 2014-0601 4.0 - week of March 31, Dataverse 4.0: Beta 1, 2014-0601 4.0 - week of April 7, Dataverse 4.0: Beta 1, 2014-0601 4.0 - week of April 14, Dataverse 4.0: Beta 1, 2014-0601 4.0 - week of April 21, Dataverse 4.0: Beta 1, 2014-0601 4.0 - week of April 28, Dataverse 4.0: Beta 1, 2014-0601 4.0 - Beta1, Dataverse 4.0: Beta 1, 2014-0601 4.0 - week of May 12,Dataverse 4.0: In Review, None 4.0-Beta 1.1,Dataverse 4.0: In Review, None 4.0 - week of May 19,Dataverse 4.0: In Review, None 4.0 - week of May 26,Dataverse 4.0: In Review, None 4.0 - week of June 2,Dataverse 4.0: In Review, None 4.0 - week of June 9,Dataverse 4.0: In Review, None 4.0 - week of June 16,Dataverse 4.0: In Review, None 4.0 - week of June 23,Dataverse 4.0: In Review, None 4.0 - Final,Dataverse 4.0: In Review, None 3.6 patch,Dataverse 3.6: Patch, None 4.0 - Triage,Dataverse 4.0: In Review, None 4.0 - review for post 4.0,Dataverse 4.0: In Review, None 4.0 - review for weekly assignment,Dataverse 4.0: In Review, None 4.1,Dataverse 4.0: In Review, None "Future: Ingest, File Upload",Future: Dataverse 4.x, None Future: Internationalization,Future: Dataverse 4.x, None "Future: Performance, Downloads",Future: Dataverse 4.x, None Future: Security / Shibboleth,Future: Dataverse 4.x, None Future: Subset / Analysis separation,Future: Dataverse 4.x, None Future: World Map integration,Future: Dataverse 4.x, None Review Post 4.0,In Review, None Review: Clean up issues,In Review, None Review: Harvesting / new data,In Review, None Review: before each new release,In Review, None Review: for 4.1,In Review, None -------------------------------------------------------------------------------- /src/settings/sample_user_map.csv: -------------------------------------------------------------------------------- 1 | Redmine User, Github User 2 | Gustavo Durand,scolapasta 3 | Bob Treacy,rtreacy 4 | Elda Sotiri,esotiri 5 | Eleni Castro,posixeleni 6 | Elizabeth Quigley,eaquigley 7 | Ellen Kraffmiller,ekraffmiller 8 | Helena Caminal, 9 | Kevin Condon,kcondon 10 | Leonid Andreev,landreev 11 | Michael Bar-Sinai,michbarsinai 12 | Mike Heppler,mheppler 13 | Naomi Day,thenaomiday 14 | Noel Castillo, 15 | Philip Durbin,pdurbin 16 | Raman Prasad,raprasad 17 | Steve Kraffmiller,sekmiller 18 | Xiangqing Yang,xyang02 19 | Merce Crosas,mcrosas 20 | Sonia Barbosa, -------------------------------------------------------------------------------- /src/utils/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/IQSS/redmine2github/57d50bcf10952bcdd87174ac208e8dfebccec8ae/src/utils/__init__.py -------------------------------------------------------------------------------- /src/utils/msg_util.py: -------------------------------------------------------------------------------- 1 | from __future__ import print_function 2 | 3 | import sys 4 | # Just the print statement... 5 | 6 | def msg(s): print(s) 7 | def dashes(): msg(40*'-') 8 | def msgt(s): dashes(); msg(s); dashes() 9 | def msgx(s): msgt(s); sys.exit(0) 10 | --------------------------------------------------------------------------------