├── LICENSE ├── README.md ├── giggity ├── __init__.py ├── giggity.py └── utils │ ├── emaildump.py │ ├── repodump.py │ └── repos-to-hamburglar.sh ├── requirements.txt └── setup.py /LICENSE: -------------------------------------------------------------------------------- 1 | Apache License 2 | Version 2.0, January 2004 3 | http://www.apache.org/licenses/ 4 | 5 | TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 6 | 7 | 1. Definitions. 8 | 9 | "License" shall mean the terms and conditions for use, reproduction, 10 | and distribution as defined by Sections 1 through 9 of this document. 11 | 12 | "Licensor" shall mean the copyright owner or entity authorized by 13 | the copyright owner that is granting the License. 14 | 15 | "Legal Entity" shall mean the union of the acting entity and all 16 | other entities that control, are controlled by, or are under common 17 | control with that entity. For the purposes of this definition, 18 | "control" means (i) the power, direct or indirect, to cause the 19 | direction or management of such entity, whether by contract or 20 | otherwise, or (ii) ownership of fifty percent (50%) or more of the 21 | outstanding shares, or (iii) beneficial ownership of such entity. 22 | 23 | "You" (or "Your") shall mean an individual or Legal Entity 24 | exercising permissions granted by this License. 25 | 26 | "Source" form shall mean the preferred form for making modifications, 27 | including but not limited to software source code, documentation 28 | source, and configuration files. 29 | 30 | "Object" form shall mean any form resulting from mechanical 31 | transformation or translation of a Source form, including but 32 | not limited to compiled object code, generated documentation, 33 | and conversions to other media types. 34 | 35 | "Work" shall mean the work of authorship, whether in Source or 36 | Object form, made available under the License, as indicated by a 37 | copyright notice that is included in or attached to the work 38 | (an example is provided in the Appendix below). 39 | 40 | "Derivative Works" shall mean any work, whether in Source or Object 41 | form, that is based on (or derived from) the Work and for which the 42 | editorial revisions, annotations, elaborations, or other modifications 43 | represent, as a whole, an original work of authorship. For the purposes 44 | of this License, Derivative Works shall not include works that remain 45 | separable from, or merely link (or bind by name) to the interfaces of, 46 | the Work and Derivative Works thereof. 47 | 48 | "Contribution" shall mean any work of authorship, including 49 | the original version of the Work and any modifications or additions 50 | to that Work or Derivative Works thereof, that is intentionally 51 | submitted to Licensor for inclusion in the Work by the copyright owner 52 | or by an individual or Legal Entity authorized to submit on behalf of 53 | the copyright owner. For the purposes of this definition, "submitted" 54 | means any form of electronic, verbal, or written communication sent 55 | to the Licensor or its representatives, including but not limited to 56 | communication on electronic mailing lists, source code control systems, 57 | and issue tracking systems that are managed by, or on behalf of, the 58 | Licensor for the purpose of discussing and improving the Work, but 59 | excluding communication that is conspicuously marked or otherwise 60 | designated in writing by the copyright owner as "Not a Contribution." 61 | 62 | "Contributor" shall mean Licensor and any individual or Legal Entity 63 | on behalf of whom a Contribution has been received by Licensor and 64 | subsequently incorporated within the Work. 65 | 66 | 2. Grant of Copyright License. Subject to the terms and conditions of 67 | this License, each Contributor hereby grants to You a perpetual, 68 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 69 | copyright license to reproduce, prepare Derivative Works of, 70 | publicly display, publicly perform, sublicense, and distribute the 71 | Work and such Derivative Works in Source or Object form. 72 | 73 | 3. Grant of Patent License. Subject to the terms and conditions of 74 | this License, each Contributor hereby grants to You a perpetual, 75 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 76 | (except as stated in this section) patent license to make, have made, 77 | use, offer to sell, sell, import, and otherwise transfer the Work, 78 | where such license applies only to those patent claims licensable 79 | by such Contributor that are necessarily infringed by their 80 | Contribution(s) alone or by combination of their Contribution(s) 81 | with the Work to which such Contribution(s) was submitted. If You 82 | institute patent litigation against any entity (including a 83 | cross-claim or counterclaim in a lawsuit) alleging that the Work 84 | or a Contribution incorporated within the Work constitutes direct 85 | or contributory patent infringement, then any patent licenses 86 | granted to You under this License for that Work shall terminate 87 | as of the date such litigation is filed. 88 | 89 | 4. Redistribution. You may reproduce and distribute copies of the 90 | Work or Derivative Works thereof in any medium, with or without 91 | modifications, and in Source or Object form, provided that You 92 | meet the following conditions: 93 | 94 | (a) You must give any other recipients of the Work or 95 | Derivative Works a copy of this License; and 96 | 97 | (b) You must cause any modified files to carry prominent notices 98 | stating that You changed the files; and 99 | 100 | (c) You must retain, in the Source form of any Derivative Works 101 | that You distribute, all copyright, patent, trademark, and 102 | attribution notices from the Source form of the Work, 103 | excluding those notices that do not pertain to any part of 104 | the Derivative Works; and 105 | 106 | (d) If the Work includes a "NOTICE" text file as part of its 107 | distribution, then any Derivative Works that You distribute must 108 | include a readable copy of the attribution notices contained 109 | within such NOTICE file, excluding those notices that do not 110 | pertain to any part of the Derivative Works, in at least one 111 | of the following places: within a NOTICE text file distributed 112 | as part of the Derivative Works; within the Source form or 113 | documentation, if provided along with the Derivative Works; or, 114 | within a display generated by the Derivative Works, if and 115 | wherever such third-party notices normally appear. The contents 116 | of the NOTICE file are for informational purposes only and 117 | do not modify the License. You may add Your own attribution 118 | notices within Derivative Works that You distribute, alongside 119 | or as an addendum to the NOTICE text from the Work, provided 120 | that such additional attribution notices cannot be construed 121 | as modifying the License. 122 | 123 | You may add Your own copyright statement to Your modifications and 124 | may provide additional or different license terms and conditions 125 | for use, reproduction, or distribution of Your modifications, or 126 | for any such Derivative Works as a whole, provided Your use, 127 | reproduction, and distribution of the Work otherwise complies with 128 | the conditions stated in this License. 129 | 130 | 5. Submission of Contributions. Unless You explicitly state otherwise, 131 | any Contribution intentionally submitted for inclusion in the Work 132 | by You to the Licensor shall be under the terms and conditions of 133 | this License, without any additional terms or conditions. 134 | Notwithstanding the above, nothing herein shall supersede or modify 135 | the terms of any separate license agreement you may have executed 136 | with Licensor regarding such Contributions. 137 | 138 | 6. Trademarks. This License does not grant permission to use the trade 139 | names, trademarks, service marks, or product names of the Licensor, 140 | except as required for reasonable and customary use in describing the 141 | origin of the Work and reproducing the content of the NOTICE file. 142 | 143 | 7. Disclaimer of Warranty. Unless required by applicable law or 144 | agreed to in writing, Licensor provides the Work (and each 145 | Contributor provides its Contributions) on an "AS IS" BASIS, 146 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 147 | implied, including, without limitation, any warranties or conditions 148 | of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A 149 | PARTICULAR PURPOSE. You are solely responsible for determining the 150 | appropriateness of using or redistributing the Work and assume any 151 | risks associated with Your exercise of permissions under this License. 152 | 153 | 8. Limitation of Liability. In no event and under no legal theory, 154 | whether in tort (including negligence), contract, or otherwise, 155 | unless required by applicable law (such as deliberate and grossly 156 | negligent acts) or agreed to in writing, shall any Contributor be 157 | liable to You for damages, including any direct, indirect, special, 158 | incidental, or consequential damages of any character arising as a 159 | result of this License or out of the use or inability to use the 160 | Work (including but not limited to damages for loss of goodwill, 161 | work stoppage, computer failure or malfunction, or any and all 162 | other commercial damages or losses), even if such Contributor 163 | has been advised of the possibility of such damages. 164 | 165 | 9. Accepting Warranty or Additional Liability. While redistributing 166 | the Work or Derivative Works thereof, You may choose to offer, 167 | and charge a fee for, acceptance of support, warranty, indemnity, 168 | or other liability obligations and/or rights consistent with this 169 | License. However, in accepting such obligations, You may act only 170 | on Your own behalf and on Your sole responsibility, not on behalf 171 | of any other Contributor, and only if You agree to indemnify, 172 | defend, and hold each Contributor harmless for any liability 173 | incurred by, or claims asserted against, such Contributor by reason 174 | of your accepting any such warranty or additional liability. 175 | 176 | END OF TERMS AND CONDITIONS 177 | 178 | 179 | Copyright 2019 Adam Musciano 180 | 181 | Licensed under the Apache License, Version 2.0 (the "License"); 182 | you may not use this file except in compliance with the License. 183 | You may obtain a copy of the License at 184 | 185 | http://www.apache.org/licenses/LICENSE-2.0 186 | 187 | Unless required by applicable law or agreed to in writing, software 188 | distributed under the License is distributed on an "AS IS" BASIS, 189 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 190 | See the License for the specific language governing permissions and 191 | limitations under the License. 192 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Giggity - grab hierarchical data about a github organization, user, or repo 2 | 3 |

4 | 5 |

6 | 7 | Get information about an organization, user, or repo on github. Stores all data in a json file, organized in a tree of dictionaries for easy database transfer or data analysis. All done through the github api, with or without authentication (authentication highly recommended). 8 | 9 | **Warning:** Github API still supports v3 (what this script uses), however they are making the shift to graphql in v4. [Here](https://developer.github.com/v4/guides/migrating-from-rest/) is a post on migrating from v3. I don't know if I will update this script before it becomes deprecated, but beware. 10 | 11 | ## Setup 12 | 13 | `pip3 install giggity` 14 | 15 | 16 | or clone the repository and run: 17 | 18 | `pip3 install -r requirements.txt` 19 | 20 | 21 | ## Operation 22 | 23 | ``` 24 | giggity.py [-h] [-v] [-a] [-u] [-o] [-O OUTPUT] path 25 | 26 | positional arguments: 27 | path name of organization or user (or url of repository) 28 | 29 | optional arguments: 30 | -h, --help show this help message and exit 31 | -v, --verbose increase output verbosity 32 | -a, --authenticate allows github authentication to avoid ratelimiting 33 | -u, --user denotes that given input is a user 34 | -o, --org denotes that given input is an organization 35 | -O OUTPUT, --outfile OUTPUT 36 | location to put generated json file 37 | 38 | ``` 39 | 40 | **Example of Scraping a User** 41 | 42 | python3 giggity.py -a -O needmorecowbell.json -v -u needmorecowbell 43 | 44 | - This will ask for authentication credentials, put the program into verbose mode, scrape github for the user needmorecowbell, then put the results into needmorecowbell.json 45 | 46 | **Example of Scraping an Organization** 47 | 48 | python3 giggity.py -a -o github -O github.json 49 | 50 | - This will ask for authentication, scrape the github organization on github, then put out the results in github.json 51 | 52 | **Giggity as a Module** 53 | 54 | - giggity can also be used as a module -- all data is stored within orgTree as a nested dict. 55 | 56 | ```python 57 | from giggity import giggity 58 | 59 | g = giggity("username","password") 60 | data = g.getUsers("organization-name", followers=True) 61 | 62 | print("List of users in organization: ") 63 | for user, info in data.items(): 64 | print(user) 65 | 66 | data = g.getEmails("username", verbose=True) # Get any emails found 67 | ``` 68 | 69 | **Other examples of how to use giggity are available in the util folder.** 70 | 71 | **Example Output** 72 | 73 | 74 | When `python3 giggity.py -a -u geohot -O output.json` is used.. 75 | 76 | output.json contains: 77 | 78 | ```json 79 | { 80 | "emails": [ 81 | "george@comma.ai", 82 | "other emails taken out of example" 83 | 84 | ], 85 | "names": [ 86 | "Charles Ellis", 87 | "George Hotz" 88 | ], 89 | "repos": { 90 | "ORB_SLAM2": { 91 | "created_at": "2017-04-08T00:21:13Z", 92 | "description": "ORBSLAM2 running on Mac OS X cause I was on a plane and bored and maybe useful for someone?", 93 | "fork": true, 94 | "name": "ORB_SLAM2", 95 | "updated_at": "2018-10-22T23:51:28Z", 96 | "url": "https://github.com/geohot/ORB_SLAM2" 97 | }, 98 | { Many more repositories cut out for the example }, 99 | "xnu-deps-linux": { 100 | "created_at": "2013-10-02T00:36:29Z", 101 | "description": "Mig/cctools for Linux combo makefile thingy", 102 | "fork": true, 103 | "name": "xnu-deps-linux", 104 | "updated_at": "2016-05-01T16:04:45Z", 105 | "url": "https://github.com/geohot/xnu-deps-linux" 106 | } 107 | } 108 | } 109 | ``` 110 | -------------------------------------------------------------------------------- /giggity/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/needmorecowbell/giggity/83fd780671ad9c0e99c26447311474c6b4e696ef/giggity/__init__.py -------------------------------------------------------------------------------- /giggity/giggity.py: -------------------------------------------------------------------------------- 1 | import getpass 2 | import argparse 3 | import requests 4 | from nested_lookup import nested_lookup 5 | import json 6 | 7 | class giggity(): 8 | 9 | def __init__(self, auth_usr="", auth_pss="", depth=1): 10 | self.depth = depth 11 | 12 | self.orgTree = {} 13 | self.header ={'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36'} 14 | self.auth = (auth_usr,auth_pss) 15 | 16 | def getUsers(self, org, verbose=False, followers=False): 17 | if(verbose): 18 | print("Sraping organization: "+org) 19 | 20 | page=1 21 | isEmpty=False; 22 | tree = {} 23 | 24 | while(not isEmpty): 25 | r = requests.get("https://api.github.com/orgs/"+org+"/members?page="+str(page)+"&per_page=100", headers=self.header, auth=self.auth) 26 | result= r.json() 27 | page+=1 28 | if(len(result)==0): 29 | isEmpty=True 30 | 31 | elif("message" in result): 32 | print("Organization not found") 33 | else: 34 | for account in result: 35 | 36 | #Remove unnecessary items 37 | account.pop('node_id', None) 38 | account.pop('avatar_url',None) 39 | account.pop('gravatar_id',None) 40 | account.pop('url',None) 41 | account.pop('followers_url', None) 42 | account.pop('following_url',None) 43 | account.pop('gists_url',None) 44 | account.pop('starred_url',None) 45 | account.pop('subscriptions_url', None) 46 | account.pop('organizations_url',None) 47 | account.pop('repos_url',None) 48 | account.pop('events_url',None) 49 | account.pop('received_events_url',None) 50 | 51 | 52 | #This is where you would branch out and do additional searches on an account 53 | account["repos"]= self.getRepos(account["login"], verbose) 54 | #ie, get all user repos, then tie them back into the data structure before appending it to tree 55 | 56 | if(followers): 57 | account["followers"]= self.getFollowers(account["login"],verbose) 58 | 59 | account["emails"] = self.getEmails(account["login"],verbose) 60 | 61 | account["names"] = self.getNames(account["login"], verbose) 62 | tree[account["login"]] = account 63 | 64 | 65 | self.orgTree["users"] = tree 66 | return tree 67 | 68 | def getFollowers(self, user, depth=0, verbose=False): 69 | page=1 70 | isEmpty=False; 71 | tree= {} 72 | 73 | while (not isEmpty): 74 | r = requests.get( 75 | "https://api.github.com/users/" + user + "/followers?page=" + str(page) + "&per_page=100", 76 | headers=self.header, auth=self.auth) 77 | result = r.json() 78 | page += 1 79 | 80 | if (len(result) == 0): 81 | isEmpty = True 82 | 83 | elif ("message" in result): 84 | print("User not found") 85 | else: 86 | for account in result: 87 | #Remove unnecessary items 88 | account.pop('node_id', None) 89 | account.pop('avatar_url',None) 90 | account.pop('gravatar_id',None) 91 | account.pop('url',None) 92 | account.pop('followers_url', None) 93 | account.pop('following_url',None) 94 | account.pop('gists_url',None) 95 | account.pop('starred_url',None) 96 | account.pop('subscriptions_url', None) 97 | account.pop('organizations_url',None) 98 | account.pop('repos_url',None) 99 | account.pop('events_url',None) 100 | account.pop('received_events_url',None) 101 | 102 | #This is where additional queries for followers will go (get names, emails, etc) 103 | account["emails"] = self.getEmails(account["login"],verbose) 104 | account["names"] = self.getNames(account["login"], verbose) 105 | 106 | depth -= 1 107 | if depth >= 0: 108 | account["followers"] = self.getFollowers(account["login"], depth, verbose) 109 | depth += 1 110 | 111 | # add user's data as branch of main tree 112 | tree[account["login"]] = account 113 | 114 | if(verbose): 115 | print("["+user+"] Number of followers found: "+str(len(tree.items()))) 116 | 117 | 118 | return tree 119 | 120 | def getEmails(self, user, verbose=False): 121 | 122 | page=1 123 | isEmpty=False; 124 | emails= [] 125 | 126 | while(not isEmpty): 127 | r = requests.get("https://api.github.com/users/"+user+"/events/public?page="+str(page)+"&per_page=100", headers=self.header, auth=self.auth) 128 | result = r.json() 129 | page+=1 130 | if("message" in result): 131 | if("pagination" in result["message"]): 132 | isEmpty=True 133 | else: 134 | print("Error: "+result["message"]) 135 | isEmpty=True 136 | 137 | elif(len(result)==0): 138 | isEmpty=True 139 | else: 140 | for event in result: 141 | res = set(nested_lookup("email",event)) 142 | 143 | if(len(res)>0): #append any new emails 144 | emails= emails+ list(res) 145 | 146 | emails = list(set(emails)) # remove duplicates 147 | 148 | if(verbose): 149 | print("["+user+"] Number of emails found: "+str(len(emails))) 150 | 151 | return emails 152 | 153 | def getNames(self, user, verbose=False): 154 | 155 | page=1 156 | isEmpty=False; 157 | names= [] # avoid duplicates with a set 158 | 159 | while(not isEmpty): 160 | r = requests.get("https://api.github.com/users/"+user+"/events/public?page="+str(page)+"&per_page=100", headers=self.header, auth=self.auth) 161 | result = r.json() 162 | page+=1 163 | 164 | if("message" in result): 165 | if("pagination" in result["message"]): 166 | isEmpty = True 167 | else: 168 | print("Error: "+result["message"]) 169 | isEmpty=True 170 | 171 | elif(len(result)==0): 172 | isEmpty=True 173 | else: 174 | for event in result: 175 | res = nested_lookup("author",event) 176 | for author in res: 177 | name= nested_lookup("name",author) 178 | # res = set(nested_lookup("name",res)) 179 | if(len(res)>0): 180 | names= names + name 181 | 182 | names = list(set(names)) 183 | if(verbose): 184 | print("["+user+"] Number of names found: "+str(len(names))) 185 | 186 | return names 187 | 188 | def getRepos(self, user, org=False, verbose=False): 189 | 190 | page=1 191 | isEmpty=False; 192 | repoTree= {} 193 | 194 | while(not isEmpty): 195 | if(org): 196 | r = requests.get("https://api.github.com/orgs/"+user+"/repos?page="+str(page)+"&per_page=100", headers=self.header, auth=self.auth) 197 | else: 198 | r = requests.get("https://api.github.com/users/"+user+"/repos?page="+str(page)+"&per_page=100", headers=self.header, auth=self.auth) 199 | result = r.json() 200 | page+=1 201 | 202 | if("message" in result): 203 | print("User Not Found") 204 | elif(len(result)==0): 205 | isEmpty=True 206 | else: 207 | if(verbose): 208 | print("["+user+"] Number of repositories found: "+str(len(result))) 209 | 210 | for repo in result: 211 | tree = { 212 | "name": repo["name"], 213 | "url":repo["html_url"], 214 | "fork":repo["fork"], 215 | "description":repo["description"], 216 | "created_at":repo["created_at"], 217 | "updated_at":repo["updated_at"], 218 | } 219 | 220 | repoTree[repo["name"]] = tree 221 | 222 | return repoTree 223 | 224 | def getTree(self): 225 | return json.dumps(self.orgTree,indent=4, sort_keys=True) 226 | 227 | def writeToFile(self, filepath): 228 | with open(filepath , 'w') as outfile: 229 | json.dump(self.orgTree, outfile, indent=4 , sort_keys=True) 230 | 231 | def main(): 232 | print(""" 233 | 234 | __ _(_) __ _ __ _(_) |_ _ _ 235 | / _` | |/ _` |/ _` | | __| | | | 236 | | (_| | | (_| | (_| | | |_| |_| | 237 | \__, |_|\__, |\__, |_|\__|\__, | 238 | |___/ |___/ |___/ |___/ 239 | """) 240 | print("") 241 | # Parse arguments. 242 | parser = argparse.ArgumentParser() 243 | parser.add_argument("-v", "--verbose", help="increase output verbosity", action="store_true") 244 | parser.add_argument("-a", "--authenticate", help="allows github authentication to avoid ratelimiting",action="store_true") 245 | parser.add_argument("-u", "--user",help="denotes that given input is a user",action="store_true") 246 | parser.add_argument("-d", "--depth", help="indicates the depth for extracting followers info") 247 | parser.add_argument("-o", "--org",help="denotes that given input is an organization",action="store_true") 248 | parser.add_argument("-f", "--followers",help="adds followers entry to each account",action="store_true") 249 | parser.add_argument("-O", "--outfile", dest="output", help="location to put generated json file") 250 | parser.add_argument("path", help="name of organization or user (or url of repository)") 251 | 252 | args = parser.parse_args() 253 | 254 | outfile= "results.json" 255 | if(args.path is None): 256 | print("No input given") 257 | exit() 258 | 259 | target = args.path 260 | 261 | if(args.output is not None): 262 | outfile= args.output 263 | 264 | if(args.authenticate): 265 | user = input("Enter Github Username: ") 266 | psswd = getpass.getpass("Enter Github Password: ") 267 | if (args.depth): 268 | g = giggity(user, psswd, int(args.depth)) 269 | else: 270 | g = giggity(user, psswd) 271 | else: 272 | g = giggity() 273 | 274 | if(args.user): 275 | tree={} 276 | tree["repos"]= g.getRepos(target, verbose=args.verbose) 277 | tree["emails"]= g.getEmails(target, verbose=args.verbose) 278 | tree["names"] = g.getNames(target, verbose=args.verbose) 279 | 280 | if(args.followers): 281 | tree["followers"] = g.getFollowers(target, int(args.depth), verbose=args.verbose) 282 | 283 | with open(outfile , 'w') as out: 284 | json.dump(tree, out, indent=4 , sort_keys=True) 285 | 286 | if(args.org): 287 | tree = {} 288 | tree["repos"] = g.getRepos(target, org=True, verbose=True ) 289 | 290 | tree["users"]= g.getUsers(target, verbose=args.verbose, followers=args.followers) 291 | with open(outfile , 'w') as out: 292 | json.dump(tree, out, indent=4 , sort_keys=True) 293 | 294 | g.getUsers(target, args.verbose, args.followers) 295 | g.writeToFile(outfile) 296 | 297 | print("Scraping Complete, file available at: "+outfile) 298 | 299 | 300 | 301 | 302 | if __name__ == '__main__': 303 | main() 304 | -------------------------------------------------------------------------------- /giggity/utils/emaildump.py: -------------------------------------------------------------------------------- 1 | ### 2 | # Scrapes all emails from an organization 3 | # 4 | ### 5 | 6 | 7 | import json 8 | 9 | def getEmails(d): 10 | emails = [] 11 | for user, info in d["users"].items(): 12 | emails = emails +info["emails"] 13 | if("followers" in info.keys()): # get emails of followers too 14 | for follower, finfo in info["followers"].items(): 15 | emails = emails + finfo["emails"] 16 | 17 | return list(set(emails)) 18 | 19 | 20 | d = json.loads(open("results.json").read()) 21 | 22 | emails = getEmails(d) 23 | 24 | 25 | for email in emails: 26 | print(email) 27 | 28 | print("Emails: "+str(len(emails))) 29 | -------------------------------------------------------------------------------- /giggity/utils/repodump.py: -------------------------------------------------------------------------------- 1 | ### 2 | # Scrapes all repo urls from an organization 3 | # TIP: This would be a great precursor to a hamburglar.py script 4 | ### 5 | 6 | 7 | import json 8 | 9 | def getRepos(d): 10 | repos = [] 11 | for user, info in d["users"].items(): 12 | for repo, repoVal in info["repos"].items(): 13 | repos.append(repoVal["url"]) 14 | 15 | return list(set(repos)) 16 | 17 | 18 | d = json.loads(open("results.json").read()) 19 | repos = getRepos(d) 20 | 21 | 22 | 23 | for repo in repos: 24 | print(repo) 25 | 26 | #print("Repos found: "+str(len(repos))) 27 | -------------------------------------------------------------------------------- /giggity/utils/repos-to-hamburglar.sh: -------------------------------------------------------------------------------- 1 | # setup hamburglar.py before running 2 | # put inside blank directory with repodump.py, and results.json, and hamburglar.py 3 | # add rule directory path if using yara 4 | 5 | python3 repodump.py >> repos.txt; 6 | mkdir out; 7 | for f in `cat repos.txt`; do 8 | # Yara Rule checking 9 | # python3 hamburglar.py -y rules/ -g $f; 10 | 11 | # Regex rule checking 12 | python3 hamburglar.py -g $f; 13 | mv *.json out; 14 | done 15 | 16 | rm repos.txt 17 | 18 | 19 | -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | requests==2.20.0 2 | nested_lookup==0.2.12 3 | -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | import setuptools 2 | 3 | with open("README.md", "r") as fh: 4 | long_description = fh.read() 5 | 6 | setuptools.setup( 7 | name="giggity", 8 | version="1.0.1", 9 | author="Adam Musciano", 10 | author_email="amusciano@gmail.com", 11 | description="CLI Git Sleuth Tool", 12 | long_description=long_description, 13 | long_description_content_type="text/markdown", 14 | url="https://github.com/needmorecowbell/giggity", 15 | packages=setuptools.find_packages(), 16 | classifiers=[ 17 | "Programming Language :: Python :: 3", 18 | "License :: OSI Approved :: MIT License", 19 | ], 20 | install_requires=[ 21 | 'requests==2.20.0', 22 | 'nested_lookup==0.2.12' 23 | ], 24 | entry_points = { 25 | 'console_scripts': ['giggity = giggity.giggity:main'], 26 | }, 27 | python_requires='>=3.6', 28 | ), 29 | 30 | --------------------------------------------------------------------------------