├── .gitignore ├── LICENSE ├── MANIFEST.in ├── README.md ├── bin ├── cmu-course-api └── cmu-fce-api ├── cmu_course_api ├── __init__.py ├── aggregate.py ├── parse_descs.py ├── parse_fces.py └── parse_schedules.py └── setup.py /.gitignore: -------------------------------------------------------------------------------- 1 | 2 | # Created by https://www.gitignore.io/api/python 3 | 4 | ### Python ### 5 | # Byte-compiled / optimized / DLL files 6 | __pycache__/ 7 | *.py[cod] 8 | *$py.class 9 | 10 | # C extensions 11 | *.so 12 | 13 | # Distribution / packaging 14 | .Python 15 | env/ 16 | build/ 17 | develop-eggs/ 18 | dist/ 19 | downloads/ 20 | eggs/ 21 | .eggs/ 22 | lib/ 23 | lib64/ 24 | parts/ 25 | sdist/ 26 | var/ 27 | *.egg-info/ 28 | .installed.cfg 29 | *.egg 30 | 31 | # PyInstaller 32 | # Usually these files are written by a python script from a template 33 | # before PyInstaller builds the exe, so as to inject date/other infos into it. 34 | *.manifest 35 | *.spec 36 | 37 | # Installer logs 38 | pip-log.txt 39 | pip-delete-this-directory.txt 40 | 41 | # Unit test / coverage reports 42 | htmlcov/ 43 | .tox/ 44 | .coverage 45 | .coverage.* 46 | .cache 47 | nosetests.xml 48 | coverage.xml 49 | *,cover 50 | .hypothesis/ 51 | 52 | # Translations 53 | *.mo 54 | *.pot 55 | 56 | # Django stuff: 57 | *.log 58 | local_settings.py 59 | 60 | # Flask stuff: 61 | instance/ 62 | .webassets-cache 63 | 64 | # Scrapy stuff: 65 | .scrapy 66 | 67 | # Sphinx documentation 68 | docs/_build/ 69 | 70 | # PyBuilder 71 | target/ 72 | 73 | # IPython Notebook 74 | .ipynb_checkpoints 75 | 76 | # pyenv 77 | .python-version 78 | 79 | # celery beat schedule file 80 | celerybeat-schedule 81 | 82 | # dotenv 83 | .env 84 | 85 | # virtualenv 86 | venv/ 87 | ENV/ 88 | 89 | # Spyder project settings 90 | .spyderproject 91 | 92 | # Rope project settings 93 | .ropeproject 94 | 95 | # Ignore test output 96 | *.json 97 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | The MIT License (MIT) 2 | 3 | Copyright (c) 2014 ScottyLabs 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | 23 | -------------------------------------------------------------------------------- /MANIFEST.in: -------------------------------------------------------------------------------- 1 | recursive-include cmu_course_api/data * 2 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Course API 2 | This is a command line tool for retrieving course information. For an updated RESTful API, check out [Course API v2](https://github.com/ScottyLabs/course-api-v2) 3 | 4 | This project is structured as a number of scraper scripts which parse 5 | information from various sources including the Course Catalog, the 6 | Schedule of Classes, and the site for FCEs. 7 | 8 | ## Setup 9 | 10 | The Scottylabs Course API is now available as a pip package! All you need to do to install is run: 11 | 12 | ``` 13 | $ pip3 install cmu-course-api 14 | ``` 15 | 16 | ## Course Schedules & Descriptions Usage 17 | 18 | To get course schedules and descriptions from the command line, run: 19 | 20 | ``` 21 | $ cmu-course-api [SEMESTER] [OUTFILE] 22 | ``` 23 | 24 | `SEMESTER` is the school semester for which you wish to retrieve scheduling data. It must be one of S, M1, M2, or F. 25 | 26 | `OUTFILE` is a path to write the output JSON to. 27 | 28 | Alternatively, you can use the course API in your Python 3 projects: 29 | 30 | ```python 31 | import cmu_course_api 32 | 33 | data = cmu_course_api.get_course_data(semester) 34 | ``` 35 | 36 | Then, `data` will contain the course information as a Python object. 37 | 38 | See [Course output format](#course-output-format) for details. 39 | 40 | ## FCEs Usage 41 | 42 | To parse FCE data, download the relevant data set from the [CMU FCE website](https://cmu.smartevals.com) by logging in, clicking "See Results from Past Years", then the Excel icon at the lop left of the table. Make sure to choose CSV format. Place all files at the top level of a folder. 43 | 44 | Parse the data by running the following from the command line: 45 | 46 | ``` 47 | $ cmu-fce-api [FOLDER] [OUTFILE] 48 | ``` 49 | 50 | `FOLDER` is the folder containing CSV files to parse. 51 | 52 | `OUTFILE` is a path to write the output JSON to. 53 | 54 | Alternatively, you can use the FCE API in your Python 3 projects: 55 | 56 | ```python 57 | import cmu_course_api 58 | 59 | fces = cmu_course_api.parse_fces(csvpath) 60 | ``` 61 | 62 | Then `fces` will contain the FCE data in the file `csvpath`. 63 | 64 | See [FCE output format](#fce-output-format) for details. 65 | 66 | ## Minification 67 | 68 | By default, all output data is stored as a minified JSON file. To get human readable JSON, use the command (for output file `out.json`): 69 | 70 | ``` 71 | $ python -m json.tool out.json 72 | ``` 73 | 74 | ## Course output format 75 | 76 | Scraped data is output in the following form: 77 | 78 | ``` 79 | { 80 | "courses": { 81 | ..., 82 | "15-122": { 83 | "name": "Principles of Imperative Computation", 84 | "department": "Computer Science", 85 | "units": 10.0, 86 | "desc": "For students with a basic understanding of programming...", 87 | "prereqs": "15-112", 88 | "prereqs_obj": {"invert": false, "reqs_list": [["15-112"]] }, 89 | "coreqs": "15-151 and 21-127", 90 | "coreqs_obj": {"invert": false, "reqs_list": [["21-127"],["15-151"]] }, 91 | "lectures": , 92 | "sections": 93 | }, 94 | ... 95 | }, 96 | "rundate": "2016-05-27", 97 | "semester": "Spring 2016" 98 | } 99 | ``` 100 | 101 | Field | Type | Description 102 | -----------|------------|------------ 103 | courses | {} | Object containing course information, with course numbers as keys 104 | name | String | Course name 105 | department | String | Department name 106 | units | float | Units awarded by course. Null if the number of units is not specified or variable (common for independent study) 107 | desc | String | Course description 108 | prereqs | String | Course prerequisites as a string 109 | prereqs_obj| Object | Course prerequisites as an object representation 110 | coreqs | String | Course corequisites as a string 111 | coreqs_obj | Object | Course corequisites as an object representation 112 | lectures | {} | Lectures for this semester. See the [Meetings section](#meetings) for more info. 113 | sections | {} | Sections for this semester. See the [Meetings section](#meetings) for more info. 114 | rundate | String | Date that this JSON blob was generated in ISO format (YYYY-MM-DD). 115 | semester | String | Semester that this data's schedules represent. 116 | 117 | ### Prerequisites/Corequisites Object Representation: 118 | 119 | The fields prereqs_obj and coreqs_obj are object representations of a course's prerequisites/corequisites respectively. Each of these objects has a field "invert", which 120 | is a boolean, and a field "reqs_list" which is a 2-dimensional list representation of the requisites. If a course does not have any prerequisites/corequisites then the 121 | fields of the corresponding object will be null. 122 | 123 | Field | Type | Description 124 | ------------|------------|------------ 125 | invert | Boolean | Boolean that indicates whether the reqs_list logic is inverted or not. 126 | reqs_list | [[String]] | 2-dimensional list representation of prerequisites/corequisites 127 | 128 | In most cases, courses will have requisites with the invert field equal to false, this will be the primary representation. Under the primary representation, the elements 129 | inside the inner lists operate under 'or' logic while the inner lists with respect to other inner lists operate under 'and' logic. If the invert field is true then the 130 | primary representation is reversed. 131 | 132 | ######Invert = false (Primary Representation): 133 | 134 | [ [ A ] ] => "A" 135 | 136 | [ [ A, B ] ] => "A or B" 137 | 138 | [ [ A ], [ B ] ] => "A and B" 139 | 140 | [ [ A, B ], [ C ] ] => "(A or B) and C" 141 | 142 | [ [ A, B ], [ C ], [ D, E, F ] ] => "(A or B) and C and (D or E or F)" 143 | 144 | 145 | ###### Invert = true: 146 | 147 | [ [ A ] ] => "A" 148 | 149 | [ [ A, B ] ] => "A and B" 150 | 151 | [ [ A ], [ B ] ] => "A or B" 152 | 153 | [ [ A, B ], [ C ] ] => "(A and B) or C" 154 | 155 | [ [ A, B ], [ C ], [ D, E, F ] ] => "(A and B) or C or (D and E and F)" 156 | 157 | 158 | ###### Examples: 159 | 160 | {"invert": false, "reqs_list": [ [ "15-213", "18-243" ], [ "18-370", "18-396" ] ] } => "(15-213 or 18-243) and (18-370 or 18-396)" 161 | 162 | {"invert": true,"reqs_list": [ [ "18-320", "18-300" ], [ "18-402" ] ] } => "(18-320 and 18-300) or 18-402" 163 | 164 | ### Meetings 165 | 166 | A meeting has the form: 167 | 168 | ``` 169 | { 170 | "instructors": [ 171 | "Kosbie, David", 172 | "Andersen, David" 173 | ], 174 | "name": "Lec 1", 175 | "times": [ 176 | ..., 177 |