├── .github ├── FUNDING.yml └── workflows │ └── nuitka.yml ├── .gitignore ├── .pylintrc ├── LICENSE ├── README.md ├── build.py ├── fire.ico ├── mypy.ini ├── pyro ├── Anonymizer.py ├── Application.py ├── BuildFacade.py ├── CaseInsensitiveList.py ├── CommandArguments.py ├── Comparators.py ├── Constant.py ├── Constants.py ├── Enums │ ├── Event.py │ └── ProcessState.py ├── PackageManager.py ├── PapyrusProject.py ├── PapyrusProject.xsd ├── PathHelper.py ├── Performance │ ├── CompileData.py │ ├── PackageData.py │ └── ZippingData.py ├── PexHeader.py ├── PexReader.py ├── PexTypes.py ├── ProcessManager.py ├── ProjectBase.py ├── ProjectOptions.py ├── PyroArgumentParser.py ├── PyroRawDescriptionHelpFormatter.py ├── Remotes │ ├── BitbucketRemote.py │ ├── GenericRemote.py │ ├── GitHubRemote.py │ ├── GiteaRemote.py │ ├── RemoteBase.py │ └── RemoteUri.py ├── StringTemplate.py ├── TimeElapsed.py ├── XmlHelper.py ├── XmlRoot.py ├── __main__.py └── setup.py ├── requirements-dev.txt ├── requirements.txt └── tools ├── bsarch.exe └── bsarch.license.txt /.github/FUNDING.yml: -------------------------------------------------------------------------------- 1 | ko_fi: letsplaywithfire 2 | -------------------------------------------------------------------------------- /.github/workflows/nuitka.yml: -------------------------------------------------------------------------------- 1 | name: GitHub CI 2 | 3 | on: 4 | push: 5 | paths: 6 | - 'build.py' 7 | - 'Pipfile' 8 | - '.github/workflows/nuitka.yml' 9 | - 'pyro/**' 10 | - 'tools/**' 11 | branches: 12 | - master 13 | 14 | jobs: 15 | build: 16 | 17 | runs-on: windows-latest 18 | 19 | steps: 20 | - uses: actions/checkout@v4 21 | - name: Set up Visual Studio environment 22 | if: success() 23 | uses: seanmiddleditch/gha-setup-vsdevenv@master 24 | - name: Set up Python 3.12.4 (x64) 25 | if: success() 26 | uses: actions/setup-python@v5 27 | with: 28 | python-version: 3.12.4 29 | architecture: x64 30 | cache: pip 31 | - name: Create virtual environment 32 | if: success() 33 | run: python -m venv env 34 | - name: Install requirements 35 | if: success() 36 | run: pip install -r requirements.txt 37 | - name: Run build script 38 | if: success() 39 | run: python D:\a\pyro\pyro\build.py --no-zip 40 | - name: Upload artifact 41 | if: success() 42 | uses: actions/upload-artifact@v4 43 | with: 44 | name: pyro-master-${{ github.event.repository.pushed_at }} 45 | path: D:\a\pyro\pyro\pyro.dist 46 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | .idea/ 2 | .mypy_cache/ 3 | .vscode/ 4 | __pycache__/ 5 | bin/ 6 | dev/ 7 | pyro/dist/ 8 | pyro/logs/ 9 | pyro/out/ 10 | pyro/temp/ 11 | pyro/tools/ 12 | pyro.build/ 13 | pyro.dist/ 14 | test_data/ 15 | *.bat 16 | *.flg 17 | *.pex 18 | *.psc 19 | .secrets -------------------------------------------------------------------------------- /.pylintrc: -------------------------------------------------------------------------------- 1 | [MASTER] 2 | jobs=4 3 | extension-pkg-whitelist=lxml 4 | 5 | [MESSAGES CONTROL] 6 | disable= 7 | import-outside-toplevel, 8 | invalid-name, 9 | missing-class-docstring, 10 | missing-function-docstring, 11 | missing-module-docstring, 12 | too-few-public-methods, 13 | too-many-branches, 14 | too-many-instance-attributes, 15 | too-many-return-statements, 16 | too-many-statements, 17 | protected-access, 18 | unnecessary-comprehension, 19 | unnecessary-pass 20 | 21 | [FORMAT] 22 | max-line-length=260 23 | logging-format-style=fstr 24 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Copyright (c) 2019 fireundubh 2 | 3 | Mozilla Public License Version 2.0 4 | ================================== 5 | 6 | 1. Definitions 7 | -------------- 8 | 9 | 1.1. "Contributor" 10 | means each individual or legal entity that creates, contributes to 11 | the creation of, or owns Covered Software. 12 | 13 | 1.2. "Contributor Version" 14 | means the combination of the Contributions of others (if any) used 15 | by a Contributor and that particular Contributor's Contribution. 16 | 17 | 1.3. "Contribution" 18 | means Covered Software of a particular Contributor. 19 | 20 | 1.4. "Covered Software" 21 | means Source Code Form to which the initial Contributor has attached 22 | the notice in Exhibit A, the Executable Form of such Source Code 23 | Form, and Modifications of such Source Code Form, in each case 24 | including portions thereof. 25 | 26 | 1.5. "Incompatible With Secondary Licenses" 27 | means 28 | 29 | (a) that the initial Contributor has attached the notice described 30 | in Exhibit B to the Covered Software; or 31 | 32 | (b) that the Covered Software was made available under the terms of 33 | version 1.1 or earlier of the License, but not also under the 34 | terms of a Secondary License. 35 | 36 | 1.6. "Executable Form" 37 | means any form of the work other than Source Code Form. 38 | 39 | 1.7. "Larger Work" 40 | means a work that combines Covered Software with other material, in 41 | a separate file or files, that is not Covered Software. 42 | 43 | 1.8. "License" 44 | means this document. 45 | 46 | 1.9. "Licensable" 47 | means having the right to grant, to the maximum extent possible, 48 | whether at the time of the initial grant or subsequently, any and 49 | all of the rights conveyed by this License. 50 | 51 | 1.10. "Modifications" 52 | means any of the following: 53 | 54 | (a) any file in Source Code Form that results from an addition to, 55 | deletion from, or modification of the contents of Covered 56 | Software; or 57 | 58 | (b) any new file in Source Code Form that contains any Covered 59 | Software. 60 | 61 | 1.11. "Patent Claims" of a Contributor 62 | means any patent claim(s), including without limitation, method, 63 | process, and apparatus claims, in any patent Licensable by such 64 | Contributor that would be infringed, but for the grant of the 65 | License, by the making, using, selling, offering for sale, having 66 | made, import, or transfer of either its Contributions or its 67 | Contributor Version. 68 | 69 | 1.12. "Secondary License" 70 | means either the GNU General Public License, Version 2.0, the GNU 71 | Lesser General Public License, Version 2.1, the GNU Affero General 72 | Public License, Version 3.0, or any later versions of those 73 | licenses. 74 | 75 | 1.13. "Source Code Form" 76 | means the form of the work preferred for making modifications. 77 | 78 | 1.14. "You" (or "Your") 79 | means an individual or a legal entity exercising rights under this 80 | License. For legal entities, "You" includes any entity that 81 | controls, is controlled by, or is under common control with You. For 82 | purposes of this definition, "control" means (a) the power, direct 83 | or indirect, to cause the direction or management of such entity, 84 | whether by contract or otherwise, or (b) ownership of more than 85 | fifty percent (50%) of the outstanding shares or beneficial 86 | ownership of such entity. 87 | 88 | 2. License Grants and Conditions 89 | -------------------------------- 90 | 91 | 2.1. Grants 92 | 93 | Each Contributor hereby grants You a world-wide, royalty-free, 94 | non-exclusive license: 95 | 96 | (a) under intellectual property rights (other than patent or trademark) 97 | Licensable by such Contributor to use, reproduce, make available, 98 | modify, display, perform, distribute, and otherwise exploit its 99 | Contributions, either on an unmodified basis, with Modifications, or 100 | as part of a Larger Work; and 101 | 102 | (b) under Patent Claims of such Contributor to make, use, sell, offer 103 | for sale, have made, import, and otherwise transfer either its 104 | Contributions or its Contributor Version. 105 | 106 | 2.2. Effective Date 107 | 108 | The licenses granted in Section 2.1 with respect to any Contribution 109 | become effective for each Contribution on the date the Contributor first 110 | distributes such Contribution. 111 | 112 | 2.3. Limitations on Grant Scope 113 | 114 | The licenses granted in this Section 2 are the only rights granted under 115 | this License. No additional rights or licenses will be implied from the 116 | distribution or licensing of Covered Software under this License. 117 | Notwithstanding Section 2.1(b) above, no patent license is granted by a 118 | Contributor: 119 | 120 | (a) for any code that a Contributor has removed from Covered Software; 121 | or 122 | 123 | (b) for infringements caused by: (i) Your and any other third party's 124 | modifications of Covered Software, or (ii) the combination of its 125 | Contributions with other software (except as part of its Contributor 126 | Version); or 127 | 128 | (c) under Patent Claims infringed by Covered Software in the absence of 129 | its Contributions. 130 | 131 | This License does not grant any rights in the trademarks, service marks, 132 | or logos of any Contributor (except as may be necessary to comply with 133 | the notice requirements in Section 3.4). 134 | 135 | 2.4. Subsequent Licenses 136 | 137 | No Contributor makes additional grants as a result of Your choice to 138 | distribute the Covered Software under a subsequent version of this 139 | License (see Section 10.2) or under the terms of a Secondary License (if 140 | permitted under the terms of Section 3.3). 141 | 142 | 2.5. Representation 143 | 144 | Each Contributor represents that the Contributor believes its 145 | Contributions are its original creation(s) or it has sufficient rights 146 | to grant the rights to its Contributions conveyed by this License. 147 | 148 | 2.6. Fair Use 149 | 150 | This License is not intended to limit any rights You have under 151 | applicable copyright doctrines of fair use, fair dealing, or other 152 | equivalents. 153 | 154 | 2.7. Conditions 155 | 156 | Sections 3.1, 3.2, 3.3, and 3.4 are conditions of the licenses granted 157 | in Section 2.1. 158 | 159 | 3. Responsibilities 160 | ------------------- 161 | 162 | 3.1. Distribution of Source Form 163 | 164 | All distribution of Covered Software in Source Code Form, including any 165 | Modifications that You create or to which You contribute, must be under 166 | the terms of this License. You must inform recipients that the Source 167 | Code Form of the Covered Software is governed by the terms of this 168 | License, and how they can obtain a copy of this License. You may not 169 | attempt to alter or restrict the recipients' rights in the Source Code 170 | Form. 171 | 172 | 3.2. Distribution of Executable Form 173 | 174 | If You distribute Covered Software in Executable Form then: 175 | 176 | (a) such Covered Software must also be made available in Source Code 177 | Form, as described in Section 3.1, and You must inform recipients of 178 | the Executable Form how they can obtain a copy of such Source Code 179 | Form by reasonable means in a timely manner, at a charge no more 180 | than the cost of distribution to the recipient; and 181 | 182 | (b) You may distribute such Executable Form under the terms of this 183 | License, or sublicense it under different terms, provided that the 184 | license for the Executable Form does not attempt to limit or alter 185 | the recipients' rights in the Source Code Form under this License. 186 | 187 | 3.3. Distribution of a Larger Work 188 | 189 | You may create and distribute a Larger Work under terms of Your choice, 190 | provided that You also comply with the requirements of this License for 191 | the Covered Software. If the Larger Work is a combination of Covered 192 | Software with a work governed by one or more Secondary Licenses, and the 193 | Covered Software is not Incompatible With Secondary Licenses, this 194 | License permits You to additionally distribute such Covered Software 195 | under the terms of such Secondary License(s), so that the recipient of 196 | the Larger Work may, at their option, further distribute the Covered 197 | Software under the terms of either this License or such Secondary 198 | License(s). 199 | 200 | 3.4. Notices 201 | 202 | You may not remove or alter the substance of any license notices 203 | (including copyright notices, patent notices, disclaimers of warranty, 204 | or limitations of liability) contained within the Source Code Form of 205 | the Covered Software, except that You may alter any license notices to 206 | the extent required to remedy known factual inaccuracies. 207 | 208 | 3.5. Application of Additional Terms 209 | 210 | You may choose to offer, and to charge a fee for, warranty, support, 211 | indemnity or liability obligations to one or more recipients of Covered 212 | Software. However, You may do so only on Your own behalf, and not on 213 | behalf of any Contributor. You must make it absolutely clear that any 214 | such warranty, support, indemnity, or liability obligation is offered by 215 | You alone, and You hereby agree to indemnify every Contributor for any 216 | liability incurred by such Contributor as a result of warranty, support, 217 | indemnity or liability terms You offer. You may include additional 218 | disclaimers of warranty and limitations of liability specific to any 219 | jurisdiction. 220 | 221 | 4. Inability to Comply Due to Statute or Regulation 222 | --------------------------------------------------- 223 | 224 | If it is impossible for You to comply with any of the terms of this 225 | License with respect to some or all of the Covered Software due to 226 | statute, judicial order, or regulation then You must: (a) comply with 227 | the terms of this License to the maximum extent possible; and (b) 228 | describe the limitations and the code they affect. Such description must 229 | be placed in a text file included with all distributions of the Covered 230 | Software under this License. Except to the extent prohibited by statute 231 | or regulation, such description must be sufficiently detailed for a 232 | recipient of ordinary skill to be able to understand it. 233 | 234 | 5. Termination 235 | -------------- 236 | 237 | 5.1. The rights granted under this License will terminate automatically 238 | if You fail to comply with any of its terms. However, if You become 239 | compliant, then the rights granted under this License from a particular 240 | Contributor are reinstated (a) provisionally, unless and until such 241 | Contributor explicitly and finally terminates Your grants, and (b) on an 242 | ongoing basis, if such Contributor fails to notify You of the 243 | non-compliance by some reasonable means prior to 60 days after You have 244 | come back into compliance. Moreover, Your grants from a particular 245 | Contributor are reinstated on an ongoing basis if such Contributor 246 | notifies You of the non-compliance by some reasonable means, this is the 247 | first time You have received notice of non-compliance with this License 248 | from such Contributor, and You become compliant prior to 30 days after 249 | Your receipt of the notice. 250 | 251 | 5.2. If You initiate litigation against any entity by asserting a patent 252 | infringement claim (excluding declaratory judgment actions, 253 | counter-claims, and cross-claims) alleging that a Contributor Version 254 | directly or indirectly infringes any patent, then the rights granted to 255 | You by any and all Contributors for the Covered Software under Section 256 | 2.1 of this License shall terminate. 257 | 258 | 5.3. In the event of termination under Sections 5.1 or 5.2 above, all 259 | end user license agreements (excluding distributors and resellers) which 260 | have been validly granted by You or Your distributors under this License 261 | prior to termination shall survive termination. 262 | 263 | ************************************************************************ 264 | * * 265 | * 6. Disclaimer of Warranty * 266 | * ------------------------- * 267 | * * 268 | * Covered Software is provided under this License on an "as is" * 269 | * basis, without warranty of any kind, either expressed, implied, or * 270 | * statutory, including, without limitation, warranties that the * 271 | * Covered Software is free of defects, merchantable, fit for a * 272 | * particular purpose or non-infringing. The entire risk as to the * 273 | * quality and performance of the Covered Software is with You. * 274 | * Should any Covered Software prove defective in any respect, You * 275 | * (not any Contributor) assume the cost of any necessary servicing, * 276 | * repair, or correction. This disclaimer of warranty constitutes an * 277 | * essential part of this License. No use of any Covered Software is * 278 | * authorized under this License except under this disclaimer. * 279 | * * 280 | ************************************************************************ 281 | 282 | ************************************************************************ 283 | * * 284 | * 7. Limitation of Liability * 285 | * -------------------------- * 286 | * * 287 | * Under no circumstances and under no legal theory, whether tort * 288 | * (including negligence), contract, or otherwise, shall any * 289 | * Contributor, or anyone who distributes Covered Software as * 290 | * permitted above, be liable to You for any direct, indirect, * 291 | * special, incidental, or consequential damages of any character * 292 | * including, without limitation, damages for lost profits, loss of * 293 | * goodwill, work stoppage, computer failure or malfunction, or any * 294 | * and all other commercial damages or losses, even if such party * 295 | * shall have been informed of the possibility of such damages. This * 296 | * limitation of liability shall not apply to liability for death or * 297 | * personal injury resulting from such party's negligence to the * 298 | * extent applicable law prohibits such limitation. Some * 299 | * jurisdictions do not allow the exclusion or limitation of * 300 | * incidental or consequential damages, so this exclusion and * 301 | * limitation may not apply to You. * 302 | * * 303 | ************************************************************************ 304 | 305 | 8. Litigation 306 | ------------- 307 | 308 | Any litigation relating to this License may be brought only in the 309 | courts of a jurisdiction where the defendant maintains its principal 310 | place of business and such litigation shall be governed by laws of that 311 | jurisdiction, without reference to its conflict-of-law provisions. 312 | Nothing in this Section shall prevent a party's ability to bring 313 | cross-claims or counter-claims. 314 | 315 | 9. Miscellaneous 316 | ---------------- 317 | 318 | This License represents the complete agreement concerning the subject 319 | matter hereof. If any provision of this License is held to be 320 | unenforceable, such provision shall be reformed only to the extent 321 | necessary to make it enforceable. Any law or regulation which provides 322 | that the language of a contract shall be construed against the drafter 323 | shall not be used to construe this License against a Contributor. 324 | 325 | 10. Versions of the License 326 | --------------------------- 327 | 328 | 10.1. New Versions 329 | 330 | Mozilla Foundation is the license steward. Except as provided in Section 331 | 10.3, no one other than the license steward has the right to modify or 332 | publish new versions of this License. Each version will be given a 333 | distinguishing version number. 334 | 335 | 10.2. Effect of New Versions 336 | 337 | You may distribute the Covered Software under the terms of the version 338 | of the License under which You originally received the Covered Software, 339 | or under the terms of any subsequent version published by the license 340 | steward. 341 | 342 | 10.3. Modified Versions 343 | 344 | If you create software not governed by this License, and you want to 345 | create a new license for such software, you may create and use a 346 | modified version of this License if you rename the license and remove 347 | any references to the name of the license steward (except to note that 348 | such modified license differs from this License). 349 | 350 | 10.4. Distributing Source Code Form that is Incompatible With Secondary 351 | Licenses 352 | 353 | If You choose to distribute Source Code Form that is Incompatible With 354 | Secondary Licenses under the terms of this version of the License, the 355 | notice described in Exhibit B of this License must be attached. 356 | 357 | Exhibit A - Source Code Form License Notice 358 | ------------------------------------------- 359 | 360 | This Source Code Form is subject to the terms of the Mozilla Public 361 | License, v. 2.0. If a copy of the MPL was not distributed with this 362 | file, You can obtain one at http://mozilla.org/MPL/2.0/. 363 | 364 | If it is not possible or desirable to put the notice in a particular 365 | file, then You may include the notice in a location (such as a LICENSE 366 | file in a relevant directory) where a recipient would be likely to look 367 | for such a notice. 368 | 369 | You may add additional accurate notices of copyright ownership. 370 | 371 | Exhibit B - "Incompatible With Secondary Licenses" Notice 372 | --------------------------------------------------------- 373 | 374 | This Source Code Form is "Incompatible With Secondary Licenses", as 375 | defined by the Mozilla Public License, v. 2.0. -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | **Pyro** is a parallelized incremental build system for _Skyrim Classic_ (TESV), _Skyrim Special Edition_ (SSE), and _Fallout 4_ (FO4) projects. Pyro provides mod authors with an all-in-one tool for compiling Papyrus scripts, packaging BSA and BA2 archives, and preparing builds for distribution. Pyro can be integrated as an external tool into any IDE, allowing mod authors to "Instant Build" projects with a single hotkey. 2 | 3 | 4 | ## Binaries 5 | 6 | Latest build from master branch: 7 | 8 | [![](https://github.com/fireundubh/pyro/workflows/GitHub%20CI/badge.svg)](https://github.com/fireundubh/pyro/actions) 9 | 10 | See also: [Releases](https://github.com/fireundubh/pyro/releases) 11 | 12 | 13 | ## Documentation 14 | 15 | External link: [Pyro Wiki](https://wiki.fireundubh.com/pyro) 16 | -------------------------------------------------------------------------------- /build.py: -------------------------------------------------------------------------------- 1 | import argparse 2 | import glob 3 | import logging 4 | import os 5 | import shutil 6 | import subprocess 7 | import sys 8 | import zipfile 9 | 10 | from pyro.Comparators import endswith 11 | 12 | 13 | class Application: 14 | logging.basicConfig(level=logging.INFO, format='%(asctime)s [%(levelname).4s] %(message)s') 15 | log: logging.Logger = logging.getLogger('pyro') 16 | 17 | def __init__(self, args: argparse.Namespace) -> None: 18 | self.root_path: str = os.path.dirname(__file__) 19 | 20 | self.local_venv_path: str = args.local_venv_path 21 | self.package_name: str = 'pyro' 22 | self.no_zip: bool = args.no_zip 23 | self.vcvars64_path: str = args.vcvars64_path 24 | 25 | self.dist_path: str = os.path.join(self.root_path, f'{self.package_name}.dist') 26 | self.root_tools_path: str = os.path.join(self.root_path, 'tools') 27 | self.dist_tools_path: str = os.path.join(self.dist_path, 'tools') 28 | 29 | self.site_path: str = os.path.join(self.dist_path, 'site') 30 | self.zip_path: str = os.path.join(self.root_path, 'bin', f'{self.package_name}.zip') 31 | 32 | self.dist_glob_pattern: str = os.path.join(self.dist_path, r'**\*') 33 | 34 | self.icon_path: str = os.path.join(self.root_path, 'fire.ico') 35 | 36 | pyro_version: str = '1.0.0.0' 37 | 38 | self.nuitka_args: list = [ 39 | 'python', '-m', 'nuitka', 40 | '--standalone', 'pyro', 41 | '--include-package=pyro', 42 | '--assume-yes-for-downloads', 43 | '--plugin-enable=multiprocessing', 44 | '--plugin-enable=pkg-resources', 45 | '--msvc=14.3', 46 | '--disable-ccache', 47 | '--windows-company-name=fireundubh', 48 | f'--windows-product-name={self.package_name.capitalize()}', 49 | f'--windows-file-version={pyro_version}', 50 | f'--windows-product-version={pyro_version}', 51 | '--windows-file-description=https://github.com/fireundubh/pyro', 52 | '--windows-icon-from-ico=fire.ico' 53 | ] 54 | 55 | def __setattr__(self, key: str, value: object) -> None: 56 | # sanitize paths 57 | if isinstance(value, str) and endswith(key, 'path', ignorecase=True): 58 | value = os.path.normpath(value) 59 | # normpath converts empty paths to os.curdir which we don't want 60 | if value == os.curdir: 61 | value = '' 62 | super(Application, self).__setattr__(key, value) 63 | 64 | def _build_zip_archive(self) -> None: 65 | os.makedirs(os.path.dirname(self.zip_path), exist_ok=True) 66 | 67 | files: list = [f for f in glob.glob(self.dist_glob_pattern, recursive=True) 68 | if os.path.isfile(f)] 69 | 70 | with zipfile.ZipFile(self.zip_path, 'w') as z: 71 | for f in files: 72 | z.write(f, os.path.join(self.package_name, os.path.relpath(f, self.dist_path)), 73 | compress_type=zipfile.ZIP_STORED) 74 | Application.log.info(f'Added file to archive: {f}') 75 | 76 | @staticmethod 77 | def exec_process(cmd: list, env: dict) -> int: 78 | try: 79 | process = subprocess.Popen(cmd, stdout=subprocess.PIPE, shell=True, universal_newlines=True, env=env) 80 | while process.poll() is None: 81 | line: str = process.stdout.readline().strip() 82 | if line: 83 | Application.log.info(line) 84 | except FileNotFoundError: 85 | Application.log.error(f'Cannot run command: {cmd}') 86 | return 1 87 | return 0 88 | 89 | def run(self) -> int: 90 | if not sys.platform == 'win32': 91 | Application.log.error('Cannot build Pyro with Nuitka on a non-Windows platform') 92 | sys.exit(1) 93 | 94 | if self.vcvars64_path: 95 | if not os.path.exists(self.vcvars64_path) or not endswith(self.vcvars64_path, '.bat', ignorecase=True): 96 | Application.log.error('Cannot build Pyro with MSVC compiler because VsDevCmd path is invalid') 97 | sys.exit(1) 98 | 99 | Application.log.info(f'Using project path: "{self.root_path}"') 100 | 101 | Application.log.warning(f'Cleaning: "{self.dist_path}"') 102 | shutil.rmtree(self.dist_path, ignore_errors=True) 103 | os.makedirs(self.dist_path, exist_ok=True) 104 | 105 | fail_state: int = 0 106 | 107 | env_log_path: str = '' 108 | environ: dict = os.environ.copy() 109 | 110 | if self.vcvars64_path: 111 | try: 112 | process = subprocess.Popen(f'%comspec% /C "{self.vcvars64_path}"', stdout=subprocess.PIPE, shell=True, universal_newlines=True) 113 | except FileNotFoundError: 114 | fail_state = 1 115 | 116 | # noinspection PyUnboundLocalVariable 117 | while process.poll() is None: 118 | line = process.stdout.readline().strip() 119 | Application.log.info(line) 120 | 121 | if 'post-execution' in line: 122 | _, env_log_path = line.split(' to ') 123 | process.terminate() 124 | 125 | Application.log.info(f'Loading environment: "{env_log_path}"') 126 | 127 | with open(env_log_path, encoding='utf-8') as f: 128 | lines: list = f.read().splitlines() 129 | 130 | for line in lines: 131 | key, value = line.split('=', maxsplit=1) 132 | environ[key] = value 133 | 134 | if fail_state == 0: 135 | fail_state = self.exec_process(self.local_venv_path.split(' '), environ) 136 | 137 | if fail_state == 0: 138 | fail_state = self.exec_process(self.nuitka_args, environ) 139 | 140 | if fail_state == 0 and not os.path.exists(self.dist_path) or f'{self.package_name}.exe' not in os.listdir(self.dist_path): 141 | Application.log.error(f'Cannot find {self.package_name}.exe in folder or folder does not exist: {self.dist_path}') 142 | fail_state = 1 143 | 144 | if fail_state == 0: 145 | Application.log.info('Copying schemas...') 146 | for schema_file_name in ['PapyrusProject.xsd']: 147 | shutil.copy2(os.path.join(self.root_path, self.package_name, schema_file_name), 148 | os.path.join(self.dist_path, schema_file_name)) 149 | 150 | Application.log.info('Copying tools...') 151 | os.makedirs(self.dist_tools_path, exist_ok=True) 152 | 153 | for tool_file_name in ['bsarch.exe', 'bsarch.license.txt']: 154 | shutil.copy2(os.path.join(self.root_tools_path, tool_file_name), 155 | os.path.join(self.dist_tools_path, tool_file_name)) 156 | 157 | if not self.no_zip: 158 | Application.log.info('Building archive...') 159 | self._build_zip_archive() 160 | 161 | Application.log.info(f'Wrote archive: "{self.zip_path}"') 162 | 163 | Application.log.info('Build complete.') 164 | 165 | return fail_state 166 | 167 | Application.log.error(f'Failed to execute command: {" ".join(self.nuitka_args)}') 168 | 169 | Application.log.warning(f'Resetting: "{self.dist_path}"') 170 | shutil.rmtree(self.dist_path, ignore_errors=True) 171 | 172 | return fail_state 173 | 174 | 175 | if __name__ == '__main__': 176 | _parser = argparse.ArgumentParser(description='Pyro Build Script') 177 | 178 | _parser.add_argument('--local-venv-path', 179 | action='store', default=os.path.join(os.path.dirname(sys.executable), 'activate.bat'), 180 | help='path to local venv activate script') 181 | 182 | _parser.add_argument('--no-zip', 183 | action='store_true', default=False, 184 | help='do not create zip archive') 185 | 186 | _parser.add_argument('--vcvars64-path', 187 | action='store', default='', 188 | help='path to visual studio developer command prompt') 189 | 190 | Application(_parser.parse_args()).run() 191 | -------------------------------------------------------------------------------- /fire.ico: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fireundubh/pyro/82a6faf6f6a79e40a5fbf705e03d6c655cff40b7/fire.ico -------------------------------------------------------------------------------- /mypy.ini: -------------------------------------------------------------------------------- 1 | [mypy] 2 | ; ----------------------------------------------------------------------------- 3 | ; IMPORT DISCOVERY 4 | ; ----------------------------------------------------------------------------- 5 | 6 | ; Enables PEP 420 style namespace packages. (bool, default False) 7 | ;namespace_packages = False 8 | 9 | ; Suppress error messages about unresolvable imports. (bool, default False) 10 | ignore_missing_imports = True 11 | 12 | ; ----------------------------------------------------------------------------- 13 | ; PLATFORM CONFIGURATION 14 | ; ----------------------------------------------------------------------------- 15 | 16 | ; Specify Python version used to parse and check target program. (string) 17 | python_version = 3.12 18 | 19 | ; Specify platform for target program. (string, default sys.platform) 20 | platform = win32 21 | 22 | ; ----------------------------------------------------------------------------- 23 | ; DISALLOW DYNAMIC TYPING 24 | ; ----------------------------------------------------------------------------- 25 | 26 | ; Disallow all expressions in module that have type Any. (bool, default False) 27 | disallow_any_expr = False 28 | 29 | ; Disallow functions with Any in signatures after decorator transformation. (bool, default False) 30 | disallow_any_decorated = False 31 | 32 | ; Disallow explicit Any in annotations and generic type parameters. (bool, default False) 33 | disallow_any_explicit = True 34 | 35 | ; Disallow generic types that do not specify explicit type parameters. (bool, default False) 36 | disallow_any_generics = False 37 | 38 | ; Disallow subclassing value of type Any. (bool, default False) 39 | disallow_subclassing_any = False 40 | 41 | ; ----------------------------------------------------------------------------- 42 | ; UNTYPED DEFINITIONS AND CALLS 43 | ; ----------------------------------------------------------------------------- 44 | 45 | ; Disallow functions without annotations from annotated functions. (bool, default False) 46 | disallow_untyped_calls = True 47 | 48 | ; Disallow functions without annotations or with incomplete annotations. (bool, default False) 49 | disallow_untyped_defs = True 50 | 51 | ; Disallow functions with incomplete annotations. (bool, default False) 52 | disallow_incomplete_defs = True 53 | 54 | ; Typecheck interior of functions without annotations. (bool, default False) 55 | check_untyped_defs = True 56 | 57 | ; ----------------------------------------------------------------------------- 58 | ; NONE AND OPTIONAL HANDLING 59 | ; ----------------------------------------------------------------------------- 60 | 61 | ; Treat None-default arguments by not implicitly making their type Optional. (bool, default False) 62 | no_implicit_optional = True 63 | 64 | ; Enable strict Optional checks. If False, treat None as compatible with every type. (bool, default True) 65 | strict_optional = True 66 | 67 | ; ----------------------------------------------------------------------------- 68 | ; CONFIGURING WARNINGS 69 | ; ----------------------------------------------------------------------------- 70 | 71 | ; Warn about casting an expression to inferred type. (bool, default False) 72 | warn_redundant_casts = True 73 | 74 | ; Show warning for missing return statements on some execution paths. (bool, default True) 75 | warn_no_return = True 76 | 77 | ; Show warning when code is inferred as unreachable or redundant. (bool, default False) 78 | warn_unreachable = True 79 | 80 | ; ----------------------------------------------------------------------------- 81 | ; MISCELLANEOUS STRICTNESS FLAGS 82 | ; ----------------------------------------------------------------------------- 83 | 84 | ; Prohibit equality, identity, and container checks between non-overlapping types. (bool, default False) 85 | strict_equality = True 86 | -------------------------------------------------------------------------------- /pyro/Anonymizer.py: -------------------------------------------------------------------------------- 1 | import logging 2 | import os 3 | import random 4 | import string 5 | import sys 6 | 7 | from pyro.PexHeader import PexHeader 8 | from pyro.PexReader import PexReader 9 | 10 | from pyro.Comparators import endswith 11 | 12 | 13 | class Anonymizer: 14 | log: logging.Logger = logging.getLogger('pyro') 15 | 16 | @staticmethod 17 | def _randomize_str(size: int, uppercase: bool = False) -> str: 18 | charset = string.ascii_uppercase if uppercase else string.ascii_lowercase 19 | return ''.join(random.choice(charset) for _ in range(size)) 20 | 21 | @staticmethod 22 | def anonymize_script(path: str) -> None: 23 | """ 24 | Obfuscates script path, user name, and computer name in compiled script 25 | """ 26 | try: 27 | header: PexHeader = PexReader.get_header(path) 28 | except ValueError: 29 | Anonymizer.log.error(f'Cannot anonymize script due to unknown file magic: "{path}"') 30 | sys.exit(1) 31 | 32 | file_path: str = header.script_path.value 33 | user_name: str = header.user_name.value 34 | computer_name: str = header.computer_name.value 35 | 36 | if '.' not in file_path: 37 | Anonymizer.log.warning(f'Cannot anonymize script again: "{path}"') 38 | return 39 | 40 | if not endswith(file_path, '.psc', ignorecase=True): 41 | Anonymizer.log.error(f'Cannot anonymize script due to invalid file extension: "{path}"') 42 | sys.exit(1) 43 | 44 | if not len(file_path) > 0: 45 | Anonymizer.log.error(f'Cannot anonymize script due to zero-length file path: "{path}"') 46 | sys.exit(1) 47 | 48 | if not len(user_name) > 0: 49 | Anonymizer.log.error(f'Cannot anonymize script due to zero-length user name: "{path}"') 50 | sys.exit(1) 51 | 52 | if not len(computer_name) > 0: 53 | Anonymizer.log.error(f'Cannot anonymize script due to zero-length computer name: "{path}"') 54 | sys.exit(1) 55 | 56 | with open(path, mode='r+b') as f: 57 | f.seek(header.script_path.offset, os.SEEK_SET) 58 | f.write(bytes(Anonymizer._randomize_str(header.script_path_size.value), encoding='ascii')) 59 | 60 | f.seek(header.user_name.offset, os.SEEK_SET) 61 | f.write(bytes(Anonymizer._randomize_str(header.user_name_size.value), encoding='ascii')) 62 | 63 | f.seek(header.computer_name.offset, os.SEEK_SET) 64 | f.write(bytes(Anonymizer._randomize_str(header.computer_name_size.value, True), encoding='ascii')) 65 | 66 | Anonymizer.log.info(f'Anonymized "{path}"...') 67 | -------------------------------------------------------------------------------- /pyro/Application.py: -------------------------------------------------------------------------------- 1 | import argparse 2 | import logging 3 | import os 4 | import sys 5 | 6 | from pyro.Enums.Event import (BuildEvent, 7 | ImportEvent, 8 | CompileEvent, 9 | AnonymizeEvent, 10 | PackageEvent, 11 | ZipEvent) 12 | from pyro.BuildFacade import BuildFacade 13 | from pyro.Comparators import startswith 14 | from pyro.PapyrusProject import PapyrusProject 15 | from pyro.PathHelper import PathHelper 16 | from pyro.PexReader import PexReader 17 | from pyro.ProjectOptions import ProjectOptions 18 | 19 | 20 | class Application: 21 | logging.basicConfig(stream=sys.stdout, level=logging.DEBUG, format='%(asctime)s [%(levelname).4s] %(message)s') 22 | log = logging.getLogger('pyro') 23 | 24 | args: argparse.Namespace 25 | 26 | def __init__(self, parser: argparse.ArgumentParser) -> None: 27 | self.parser = parser 28 | 29 | self.args = self.parser.parse_args() 30 | 31 | if self.args.show_help: 32 | self.parser.print_help() 33 | sys.exit(1) 34 | 35 | # set up log levels 36 | log_level_argv = self.args.log_level.upper() 37 | log_level = getattr(logging, log_level_argv, logging.DEBUG) 38 | self.log.setLevel(log_level) 39 | 40 | Application.log.debug(f'Set log level to: {log_level_argv}') 41 | 42 | self.args.input_path = self._try_fix_input_path(self.args.input_path or self.args.input_path_deprecated) 43 | 44 | if not self.args.create_project and not os.path.isfile(self.args.input_path): 45 | Application.log.error(f'Cannot load nonexistent PPJ at given path: "{self.args.input_path}"') 46 | sys.exit(1) 47 | 48 | @staticmethod 49 | def _try_fix_input_path(input_path: str) -> str: 50 | if not input_path: 51 | Application.log.error('required argument missing: -i INPUT.ppj') 52 | sys.exit(1) 53 | 54 | if startswith(input_path, 'file:', ignorecase=True): 55 | full_path = PathHelper.url2pathname(input_path) 56 | input_path = os.path.normpath(full_path) 57 | 58 | if not os.path.isabs(input_path): 59 | cwd = os.getcwd() 60 | Application.log.info(f'Using working directory: "{cwd}"') 61 | 62 | input_path = os.path.join(cwd, input_path) 63 | 64 | Application.log.info(f'Using input path: "{input_path}"') 65 | 66 | return input_path 67 | 68 | @staticmethod 69 | def _validate_project_file(ppj: PapyrusProject) -> None: 70 | if ppj.imports_node is None and \ 71 | (ppj.scripts_node is not None or ppj.folders_node is not None): 72 | Application.log.error('Cannot proceed without imports defined in project') 73 | sys.exit(1) 74 | 75 | if ppj.options.package and ppj.packages_node is None: 76 | Application.log.error('Cannot proceed with Package enabled without Packages defined in project') 77 | sys.exit(1) 78 | 79 | if ppj.options.zip and ppj.zip_files_node is None: 80 | Application.log.error('Cannot proceed with Zip enabled without ZipFile defined in project') 81 | sys.exit(1) 82 | 83 | @staticmethod 84 | def _validate_project_paths(ppj: PapyrusProject) -> None: 85 | compiler_path = ppj.get_compiler_path() 86 | if not compiler_path or not os.path.isfile(compiler_path): 87 | Application.log.error('Cannot proceed without compiler path') 88 | sys.exit(1) 89 | 90 | flags_path = ppj.get_flags_path() 91 | if not flags_path: 92 | Application.log.error('Cannot proceed without flags path') 93 | sys.exit(1) 94 | 95 | if not ppj.options.game_type: 96 | Application.log.error('Cannot determine game type from arguments or Papyrus Project') 97 | sys.exit(1) 98 | 99 | if not os.path.isabs(flags_path) and \ 100 | not any([os.path.isfile(os.path.join(import_path, flags_path)) for import_path in ppj.import_paths]): 101 | Application.log.error('Cannot proceed without flags file in any import folder') 102 | sys.exit(1) 103 | 104 | def run(self) -> int: 105 | """ 106 | Entry point 107 | """ 108 | _, extension = os.path.splitext(os.path.basename(self.args.input_path).casefold()) 109 | 110 | if extension == '.pex': 111 | header = PexReader.dump(self.args.input_path) 112 | Application.log.info(f'Dumping: "{self.args.input_path}"\n{header}') 113 | sys.exit(0) 114 | elif extension not in ('.ppj', '.pyroproject'): 115 | Application.log.error('Cannot proceed without PPJ file path') 116 | sys.exit(1) 117 | 118 | options = ProjectOptions(self.args.__dict__) 119 | ppj = PapyrusProject(options) 120 | 121 | self._validate_project_file(ppj) 122 | 123 | if ppj.scripts_node is not None or ppj.folders_node is not None or ppj.remote_paths: 124 | ppj.try_initialize_remotes() 125 | 126 | if ppj.use_pre_import_event: 127 | ppj.try_run_event(ImportEvent.PRE) 128 | 129 | ppj.try_populate_imports() 130 | 131 | if ppj.use_post_import_event: 132 | ppj.try_run_event(ImportEvent.POST) 133 | 134 | ppj.try_set_game_type() 135 | ppj.find_missing_scripts() 136 | ppj.try_set_game_path() 137 | 138 | self._validate_project_paths(ppj) 139 | 140 | Application.log.info('Imports found:') 141 | for path in ppj.import_paths: 142 | Application.log.info(f'+ "{path}"') 143 | 144 | Application.log.info('Scripts found:') 145 | for _, path in ppj.psc_paths.items(): 146 | Application.log.info(f'+ "{path}"') 147 | 148 | build = BuildFacade(ppj) 149 | 150 | # bsarch path is not set until BuildFacade initializes 151 | if ppj.options.package and not os.path.isfile(ppj.options.bsarch_path): 152 | Application.log.error('Cannot proceed with Package enabled without valid BSArch path') 153 | sys.exit(1) 154 | 155 | if ppj.use_pre_build_event: 156 | ppj.try_run_event(BuildEvent.PRE) 157 | 158 | if build.scripts_count > 0: 159 | if ppj.use_pre_compile_event: 160 | ppj.try_run_event(CompileEvent.PRE) 161 | 162 | build.try_compile() 163 | 164 | if ppj.use_post_compile_event: 165 | ppj.try_run_event(CompileEvent.POST) 166 | 167 | if ppj.options.anonymize: 168 | if build.get_compile_data().failed_count == 0 or ppj.options.ignore_errors: 169 | if ppj.use_pre_anonymize_event: 170 | ppj.try_run_event(AnonymizeEvent.PRE) 171 | 172 | build.try_anonymize() 173 | 174 | if ppj.use_post_anonymize_event: 175 | ppj.try_run_event(AnonymizeEvent.POST) 176 | else: 177 | Application.log.error(f'Cannot anonymize scripts because {build.get_compile_data().failed_count} scripts failed to compile') 178 | sys.exit(build.get_compile_data().failed_count) 179 | else: 180 | Application.log.info('Cannot anonymize scripts because Anonymize is disabled in project') 181 | 182 | if ppj.options.package: 183 | if build.get_compile_data().failed_count == 0 or ppj.options.ignore_errors: 184 | if ppj.use_pre_package_event: 185 | ppj.try_run_event(PackageEvent.PRE) 186 | 187 | build.try_pack() 188 | 189 | if ppj.use_post_package_event: 190 | ppj.try_run_event(PackageEvent.POST) 191 | else: 192 | Application.log.error(f'Cannot create Packages because {build.get_compile_data().failed_count} scripts failed to compile') 193 | sys.exit(build.get_compile_data().failed_count) 194 | elif ppj.packages_node is not None: 195 | Application.log.info('Cannot create Packages because Package is disabled in project') 196 | 197 | if ppj.options.zip: 198 | if build.get_compile_data().failed_count == 0 or ppj.options.ignore_errors: 199 | if ppj.use_pre_zip_event: 200 | ppj.try_run_event(ZipEvent.PRE) 201 | 202 | build.try_zip() 203 | 204 | if ppj.use_post_zip_event: 205 | ppj.try_run_event(ZipEvent.POST) 206 | else: 207 | Application.log.error(f'Cannot create ZipFile because {build.get_compile_data().failed_count} scripts failed to compile') 208 | sys.exit(build.get_compile_data().failed_count) 209 | elif ppj.zip_files_node is not None: 210 | Application.log.info('Cannot create ZipFile because Zip is disabled in project') 211 | 212 | if build.scripts_count > 0: 213 | Application.log.info(build.get_compile_data().to_string() if build.get_compile_data().success_count > 0 else 'No scripts were compiled.') 214 | 215 | if ppj.packages_node is not None: 216 | Application.log.info(build.package_data.to_string() if build.package_data.file_count > 0 else 'No files were packaged.') 217 | 218 | if ppj.zip_files_node is not None: 219 | Application.log.info(build.zipping_data.to_string() if build.zipping_data.file_count > 0 else 'No files were zipped.') 220 | 221 | Application.log.info('DONE!') 222 | 223 | if ppj.use_post_build_event and build.get_compile_data().failed_count == 0: 224 | ppj.try_run_event(BuildEvent.POST) 225 | 226 | return build.get_compile_data().failed_count 227 | -------------------------------------------------------------------------------- /pyro/BuildFacade.py: -------------------------------------------------------------------------------- 1 | import logging 2 | import multiprocessing 3 | import os 4 | import sys 5 | import time 6 | from typing import Union 7 | from copy import deepcopy 8 | 9 | import psutil 10 | 11 | from pyro.Anonymizer import Anonymizer 12 | from pyro.PackageManager import PackageManager 13 | from pyro.PapyrusProject import PapyrusProject 14 | from pyro.PathHelper import PathHelper 15 | from pyro.Performance.CompileData import (CompileData, 16 | CompileDataCaprica) 17 | from pyro.Performance.PackageData import PackageData 18 | from pyro.Performance.ZippingData import ZippingData 19 | from pyro.PexReader import PexReader 20 | from pyro.ProcessManager import ProcessManager 21 | from pyro.Enums.ProcessState import ProcessState 22 | 23 | from pyro.Comparators import (endswith, 24 | startswith) 25 | 26 | 27 | class BuildFacade: 28 | log: logging.Logger = logging.getLogger('pyro') 29 | 30 | ppj: PapyrusProject 31 | 32 | compile_data: CompileData = CompileData() 33 | compile_data_caprica: CompileData = CompileDataCaprica() 34 | package_data: PackageData = PackageData() 35 | zipping_data: ZippingData = ZippingData() 36 | 37 | def __init__(self, ppj: PapyrusProject) -> None: 38 | self.ppj = ppj 39 | 40 | self.scripts_count = len(self.ppj.psc_paths) 41 | 42 | # WARN: if methods are renamed and their respective option names are not, this will break. 43 | options: dict = deepcopy(self.ppj.options.__dict__) 44 | 45 | for key in options: 46 | if key in ('args', 'input_path', 'anonymize', 'package', 'zip', 'zip_compression'): 47 | continue 48 | if startswith(key, ('ignore_', 'no_', 'force_', 'create_', 'resolve_'), ignorecase=True): 49 | continue 50 | if endswith(key, '_token', ignorecase=True): 51 | continue 52 | setattr(self.ppj.options, key, getattr(self.ppj, f'get_{key}')()) 53 | 54 | def _find_modified_scripts(self) -> list: 55 | pex_paths: list = [] 56 | 57 | for object_name, script_path in self.ppj.psc_paths.items(): 58 | script_name, _ = os.path.splitext(os.path.basename(script_path)) 59 | 60 | # if pex exists, compare time_t in pex header with psc's last modified timestamp 61 | pex_match: list = [pex_path for pex_path in self.ppj.pex_paths 62 | if endswith(pex_path, f'{script_name}.pex', ignorecase=True)] 63 | if not pex_match: 64 | continue 65 | 66 | pex_path: str = pex_match[0] 67 | if not os.path.isfile(pex_path): 68 | continue 69 | 70 | try: 71 | header = PexReader.get_header(pex_path) 72 | except ValueError: 73 | BuildFacade.log.error(f'Cannot determine compilation time due to unknown magic: "{pex_path}"') 74 | sys.exit(1) 75 | 76 | psc_last_modified: float = os.path.getmtime(script_path) 77 | pex_last_compiled: float = float(header.compilation_time.value) 78 | 79 | # if psc is older than the pex 80 | if psc_last_modified < pex_last_compiled: 81 | pex_paths.append(pex_path) 82 | 83 | return PathHelper.uniqify(pex_paths) 84 | 85 | @staticmethod 86 | def _limit_priority() -> None: 87 | process = psutil.Process(os.getpid()) 88 | process.nice(psutil.BELOW_NORMAL_PRIORITY_CLASS if sys.platform == 'win32' else 19) 89 | 90 | def get_compile_data(self) -> Union[CompileData, CompileDataCaprica]: 91 | using_caprica = endswith(self.ppj.get_compiler_path(), 'Caprica.exe', ignorecase=True) 92 | return self.compile_data_caprica if using_caprica else self.compile_data 93 | 94 | def try_compile(self) -> None: 95 | """Builds and passes commands to Papyrus Compiler""" 96 | using_caprica = endswith(self.ppj.get_compiler_path(), 'Caprica.exe', ignorecase=True) 97 | 98 | compile_data = self.get_compile_data() 99 | 100 | compile_data.command_count, commands = self.ppj.build_commands() 101 | 102 | compile_data.time.start_time = time.time() 103 | 104 | if using_caprica or self.ppj.options.no_parallel or compile_data.command_count == 1: 105 | for command in commands: 106 | BuildFacade.log.debug(f'Command: {command}') 107 | if ProcessManager.run_compiler(command) == ProcessState.SUCCESS: 108 | compile_data.success_count += 1 109 | 110 | elif compile_data.command_count > 0: 111 | multiprocessing.freeze_support() 112 | worker_limit = min(compile_data.command_count, self.ppj.options.worker_limit) 113 | with multiprocessing.Pool(processes=worker_limit, 114 | initializer=BuildFacade._limit_priority) as pool: 115 | for state in pool.imap(ProcessManager.run_compiler, commands): 116 | if state == ProcessState.SUCCESS: 117 | compile_data.success_count += 1 118 | pool.close() 119 | pool.join() 120 | 121 | compile_data.time.end_time = time.time() 122 | 123 | # caprica success = all files compiled 124 | if using_caprica and compile_data.success_count > 0: 125 | compile_data.scripts_count = compile_data.command_count 126 | compile_data.success_count = compile_data.success_count 127 | 128 | def try_anonymize(self) -> None: 129 | """Obfuscates identifying metadata in compiled scripts""" 130 | scripts: list = self._find_modified_scripts() 131 | 132 | if not scripts and not self.ppj.missing_scripts and not self.ppj.options.no_incremental_build: 133 | BuildFacade.log.error('Cannot anonymize compiled scripts because no source scripts were modified') 134 | else: 135 | # these are absolute paths. there's no reason to manipulate them. 136 | for pex_path in self.ppj.pex_paths: 137 | if not os.path.isfile(pex_path): 138 | BuildFacade.log.error(f'Cannot locate file to anonymize: "{pex_path}"') 139 | sys.exit(1) 140 | 141 | Anonymizer.anonymize_script(pex_path) 142 | 143 | def try_pack(self) -> None: 144 | """Generates BSA/BA2 packages for project""" 145 | self.package_data.time.start_time = time.time() 146 | package_manager = PackageManager(self.ppj) 147 | package_manager.create_packages() 148 | self.package_data.time.end_time = time.time() 149 | self.package_data.file_count = package_manager.includes 150 | 151 | def try_zip(self) -> None: 152 | """Generates ZIP file for project""" 153 | self.zipping_data.time.start_time = time.time() 154 | package_manager = PackageManager(self.ppj) 155 | package_manager.create_zip() 156 | self.zipping_data.time.end_time = time.time() 157 | self.zipping_data.file_count = package_manager.includes 158 | -------------------------------------------------------------------------------- /pyro/CaseInsensitiveList.py: -------------------------------------------------------------------------------- 1 | from collections import UserList 2 | 3 | 4 | class CaseInsensitiveList(UserList): 5 | """ 6 | Simple list type for storing and comparing strings case-insensitively 7 | """ 8 | def __contains__(self, item: object) -> bool: 9 | if isinstance(item, str): 10 | return any(item.casefold() == x.casefold() for x in self.data) 11 | return item in self.data 12 | 13 | def append(self, item: object) -> None: 14 | self.data.append(item.casefold() if isinstance(item, str) else item) 15 | -------------------------------------------------------------------------------- /pyro/CommandArguments.py: -------------------------------------------------------------------------------- 1 | class CommandArguments: 2 | def __init__(self) -> None: 3 | self._items: list = [] 4 | 5 | def append(self, value: str, *, key: str = '', enquote_value: bool = False) -> None: 6 | if enquote_value: 7 | self._items.append(f'-{key}="{value}"' if key else f'"{value}"') 8 | else: 9 | self._items.append(value) 10 | 11 | def clear(self) -> None: 12 | self._items.clear() 13 | 14 | def join(self, delimiter: str = ' ') -> str: 15 | return delimiter.join(self._items) 16 | -------------------------------------------------------------------------------- /pyro/Comparators.py: -------------------------------------------------------------------------------- 1 | import os 2 | from typing import Optional, Union 3 | 4 | from lxml import etree 5 | 6 | 7 | def startswith(a_source: str, a_prefix: Union[str, tuple], 8 | a_start: Optional[int] = None, a_end: Optional[int] = None, /, ignorecase: bool = False) -> bool: 9 | 10 | source = a_source[a_start:a_end] 11 | 12 | prefixes = a_prefix if isinstance(a_prefix, tuple) else (a_prefix,) 13 | 14 | for prefix in prefixes: 15 | source_prefix = source[:len(prefix)] 16 | 17 | if not ignorecase: 18 | if source_prefix == prefix: 19 | return True 20 | else: 21 | if source_prefix.casefold() == prefix.casefold(): 22 | return True 23 | 24 | return False 25 | 26 | 27 | def endswith(a_source: str, a_suffix: Union[str, tuple], 28 | a_start: Optional[int] = None, a_end: Optional[int] = None, /, ignorecase: bool = False) -> bool: 29 | 30 | source = a_source[a_start:a_end] 31 | 32 | suffixes = a_suffix if isinstance(a_suffix, tuple) else (a_suffix,) 33 | 34 | for suffix in suffixes: 35 | source_suffix = source[-len(suffix):] 36 | 37 | if not ignorecase: 38 | if source_suffix == suffix: 39 | return True 40 | else: 41 | if source_suffix.casefold() == suffix.casefold(): 42 | return True 43 | 44 | return False 45 | 46 | 47 | def is_command_node(node: etree.ElementBase) -> bool: 48 | return node is not None and endswith(node.tag, 'Command') and node.text is not None 49 | 50 | 51 | def is_folder_node(node: etree.ElementBase) -> bool: 52 | return node is not None and endswith(node.tag, 'Folder') and node.text is not None 53 | 54 | 55 | def is_import_node(node: etree.ElementBase) -> bool: 56 | return node is not None and endswith(node.tag, 'Import') and node.text is not None 57 | 58 | 59 | def is_include_node(node: etree.ElementBase) -> bool: 60 | return node is not None and endswith(node.tag, 'Include') and node.text is not None 61 | 62 | 63 | def is_match_node(node: etree.ElementBase) -> bool: 64 | return node is not None and endswith(node.tag, 'Match') and node.text is not None 65 | 66 | 67 | def is_package_node(node: etree.ElementBase) -> bool: 68 | return node is not None and endswith(node.tag, 'Package') 69 | 70 | 71 | def is_script_node(node: etree.ElementBase) -> bool: 72 | return node is not None and endswith(node.tag, 'Script') and node.text is not None 73 | 74 | 75 | def is_variable_node(node: etree.ElementBase) -> bool: 76 | return node is not None and endswith(node.tag, 'Variable') 77 | 78 | 79 | def is_zipfile_node(node: etree.ElementBase) -> bool: 80 | return node is not None and endswith(node.tag, 'ZipFile') 81 | 82 | 83 | def is_namespace_path(node: etree.ElementBase) -> bool: 84 | return not os.path.isabs(node.text) and ':' in node.text 85 | -------------------------------------------------------------------------------- /pyro/Constant.py: -------------------------------------------------------------------------------- 1 | from dataclasses import dataclass 2 | 3 | 4 | @dataclass 5 | class Constant: 6 | @classmethod 7 | def items(cls) -> list: 8 | return list((k, getattr(cls, k)) for k in cls.__annotations__) 9 | 10 | @classmethod 11 | def keys(cls) -> list: 12 | return list(k for k in cls.__annotations__) 13 | 14 | @classmethod 15 | def values(cls) -> list: 16 | return list(getattr(cls, k) for k in cls.__annotations__) 17 | 18 | @classmethod 19 | def get(cls, field_name: str) -> str: 20 | uc_field_name: str = field_name.upper() 21 | return getattr(cls, uc_field_name) if uc_field_name in cls.keys() else '' 22 | -------------------------------------------------------------------------------- /pyro/Constants.py: -------------------------------------------------------------------------------- 1 | from pyro.Constant import Constant 2 | 3 | 4 | class FlagsName(Constant): 5 | FO4: str = 'Institute_Papyrus_Flags.flg' 6 | SF1: str = 'Starfield_Papyrus_Flags.flg' 7 | SSE: str = 'TESV_Papyrus_Flags.flg' 8 | TES5: str = 'TESV_Papyrus_Flags.flg' 9 | 10 | 11 | class GameName(Constant): 12 | FO4: str = 'Fallout 4' 13 | SF1: str = 'Starfield' 14 | SSE: str = 'Skyrim Special Edition' 15 | TES5: str = 'Skyrim' 16 | 17 | 18 | class GameType(Constant): 19 | FO4: str = 'fo4' 20 | SF1: str = 'sf1' 21 | SSE: str = 'sse' 22 | TES5: str = 'tes5' 23 | 24 | 25 | class XmlAttributeName(Constant): 26 | ANONYMIZE: str = 'Anonymize' 27 | COMPRESSION: str = 'Compression' 28 | DESCRIPTION: str = 'Description' 29 | EXCLUDE: str = 'Exclude' 30 | FINAL: str = 'Final' 31 | FLAGS: str = 'Flags' 32 | GAME: str = 'Game' 33 | IN: str = 'In' 34 | NAME: str = 'Name' 35 | NO_RECURSE: str = 'NoRecurse' 36 | OPTIMIZE: str = 'Optimize' 37 | OUTPUT: str = 'Output' 38 | PACKAGE: str = 'Package' 39 | PATH: str = 'Path' 40 | RELEASE: str = 'Release' 41 | ROOT_DIR: str = 'RootDir' 42 | USE_IN_BUILD: str = 'UseInBuild' 43 | VALUE: str = 'Value' 44 | ZIP: str = 'Zip' 45 | 46 | 47 | class XmlTagName(Constant): 48 | FOLDER: str = 'Folder' 49 | FOLDERS: str = 'Folders' 50 | IMPORT: str = 'Import' 51 | IMPORTS: str = 'Imports' 52 | INCLUDE: str = 'Include' 53 | MATCH: str = 'Match' 54 | PACKAGE: str = 'Package' 55 | PACKAGES: str = 'Packages' 56 | PAPYRUS_PROJECT: str = 'PapyrusProject' 57 | POST_BUILD_EVENT: str = 'PostBuildEvent' 58 | POST_IMPORT_EVENT: str = 'PostImportEvent' 59 | POST_COMPILE_EVENT: str = 'PostCompileEvent' 60 | POST_ANONYMIZE_EVENT: str = 'PostAnonymizeEvent' 61 | POST_PACKAGE_EVENT: str = 'PostPackageEvent' 62 | POST_ZIP_EVENT: str = 'PostZipEvent' 63 | PRE_BUILD_EVENT: str = 'PreBuildEvent' 64 | PRE_IMPORT_EVENT: str = 'PreImportEvent' 65 | PRE_COMPILE_EVENT: str = 'PreCompileEvent' 66 | PRE_ANONYMIZE_EVENT: str = 'PreAnonymizeEvent' 67 | PRE_PACKAGE_EVENT: str = 'PrePackageEvent' 68 | PRE_ZIP_EVENT: str = 'PreZipEvent' 69 | SCRIPTS: str = 'Scripts' 70 | VARIABLES: str = 'Variables' 71 | ZIP_FILE: str = 'ZipFile' 72 | ZIP_FILES: str = 'ZipFiles' 73 | 74 | 75 | __all__ = ['FlagsName', 'GameName', 'GameType', 'XmlAttributeName', 'XmlTagName'] 76 | -------------------------------------------------------------------------------- /pyro/Enums/Event.py: -------------------------------------------------------------------------------- 1 | from enum import Enum 2 | from typing import Union 3 | 4 | 5 | class BuildEvent(Enum): 6 | PRE = 0 7 | POST = 1 8 | 9 | 10 | class ImportEvent(Enum): 11 | PRE = 0 12 | POST = 1 13 | 14 | 15 | class CompileEvent(Enum): 16 | PRE = 0 17 | POST = 1 18 | 19 | 20 | class AnonymizeEvent(Enum): 21 | PRE = 0 22 | POST = 1 23 | 24 | 25 | class PackageEvent(Enum): 26 | PRE = 0 27 | POST = 1 28 | 29 | 30 | class ZipEvent(Enum): 31 | PRE = 0 32 | POST = 1 33 | 34 | 35 | Event = Union[BuildEvent, ImportEvent, CompileEvent, AnonymizeEvent, PackageEvent, ZipEvent] 36 | -------------------------------------------------------------------------------- /pyro/Enums/ProcessState.py: -------------------------------------------------------------------------------- 1 | from enum import Enum 2 | 3 | 4 | class ProcessState(Enum): 5 | SUCCESS = 0 6 | FAILURE = 1 7 | INTERRUPTED = 2 8 | ERRORS = 3 9 | -------------------------------------------------------------------------------- /pyro/PackageManager.py: -------------------------------------------------------------------------------- 1 | import logging 2 | import os 3 | import shutil 4 | import sys 5 | import typing 6 | import zipfile 7 | 8 | from lxml import etree 9 | from wcmatch import (glob, 10 | wcmatch) 11 | 12 | from pyro.CommandArguments import CommandArguments 13 | from pyro.Comparators import (endswith, 14 | is_include_node, 15 | is_match_node, 16 | is_package_node, 17 | is_zipfile_node, 18 | startswith) 19 | from pyro.CaseInsensitiveList import CaseInsensitiveList 20 | from pyro.Constants import (GameType, 21 | XmlAttributeName) 22 | from pyro.PapyrusProject import PapyrusProject 23 | from pyro.ProcessManager import ProcessManager 24 | from pyro.ProjectOptions import ProjectOptions 25 | 26 | 27 | class PackageManager: 28 | log: logging.Logger = logging.getLogger('pyro') 29 | 30 | ppj: PapyrusProject 31 | options: ProjectOptions 32 | pak_extension: str = '' 33 | zip_extension: str = '' 34 | 35 | DEFAULT_GLFLAGS = glob.NODIR | glob.MATCHBASE | glob.SPLIT | glob.REALPATH | glob.FOLLOW | glob.IGNORECASE | glob.MINUSNEGATE 36 | DEFAULT_WCFLAGS = wcmatch.SYMLINKS | wcmatch.IGNORECASE | wcmatch.MINUSNEGATE 37 | 38 | COMPRESS_TYPE = {'store': 0, 'deflate': 8} 39 | 40 | includes: int = 0 41 | 42 | def __init__(self, ppj: PapyrusProject) -> None: 43 | self.ppj = ppj 44 | self.options = ppj.options 45 | 46 | self.pak_extension = '.ba2' if self.options.game_type == GameType.FO4 else '.bsa' 47 | self.zip_extension = '.zip' 48 | 49 | @staticmethod 50 | def _can_compress_package(containing_folder: str) -> bool: 51 | flags = wcmatch.RECURSIVE | wcmatch.IGNORECASE 52 | 53 | # voices bad because bethesda no likey 54 | for _ in wcmatch.WcMatch(containing_folder, '*.fuz', flags=flags).imatch(): 55 | return False 56 | 57 | # sounds bad because bethesda no likey 58 | for _ in wcmatch.WcMatch(containing_folder, '*.wav|*.xwm', flags=flags).imatch(): 59 | return False 60 | 61 | # strings bad because wrye bash no likey 62 | for _ in wcmatch.WcMatch(containing_folder, '*.*strings', flags=flags).imatch(): 63 | return False 64 | 65 | return True 66 | 67 | @staticmethod 68 | def _check_write_permission(file_path: str) -> None: 69 | if os.path.isfile(file_path): 70 | try: 71 | open(file_path, 'a').close() 72 | except PermissionError: 73 | PackageManager.log.error(f'Cannot create file without write permission to: "{file_path}"') 74 | sys.exit(1) 75 | 76 | @staticmethod 77 | def _match(root_dir: str, file_pattern: str, *, exclude_pattern: str = '', user_path: str = '', no_recurse: bool = False) -> typing.Generator: 78 | user_flags = wcmatch.RECURSIVE if not no_recurse else 0x0 79 | matcher = wcmatch.WcMatch(root_dir, file_pattern, 80 | exclude_pattern=exclude_pattern, 81 | flags=PackageManager.DEFAULT_WCFLAGS | user_flags) 82 | 83 | matcher.on_reset() 84 | matcher._skipped = 0 85 | for file_path in matcher._walk(): 86 | yield file_path, user_path 87 | 88 | @staticmethod 89 | def _generate_include_paths(includes_node: etree.ElementBase, root_path: str, zip_mode: bool = False) -> typing.Generator: 90 | for include_node in filter(is_include_node, includes_node): 91 | attr_no_recurse: bool = include_node.get(XmlAttributeName.NO_RECURSE) == 'True' 92 | attr_path: str = include_node.get(XmlAttributeName.PATH).strip() 93 | search_path: str = include_node.text 94 | 95 | if not search_path: 96 | PackageManager.log.error(f'Include path at line {include_node.sourceline} in project file is empty') 97 | sys.exit(1) 98 | 99 | if not zip_mode and startswith(search_path, os.pardir): 100 | PackageManager.log.error(f'Include paths cannot start with "{os.pardir}"') 101 | sys.exit(1) 102 | 103 | if startswith(search_path, os.curdir): 104 | search_path = search_path.replace(os.curdir, root_path, 1) 105 | 106 | # fix invalid pattern with leading separator 107 | if not zip_mode and startswith(search_path, (os.path.sep, os.path.altsep)): 108 | search_path = '**' + search_path 109 | 110 | if '\\' in search_path: 111 | search_path = search_path.replace('\\', '/') 112 | 113 | # populate files list using glob patterns or relative paths 114 | if '*' in search_path: 115 | for include_path in glob.iglob(search_path, 116 | root_dir=root_path, 117 | flags=PackageManager.DEFAULT_GLFLAGS | glob.GLOBSTAR if not attr_no_recurse else 0x0): 118 | yield os.path.join(root_path, include_path), attr_path 119 | 120 | elif not os.path.isabs(search_path): 121 | test_path = os.path.normpath(os.path.join(root_path, search_path)) 122 | if os.path.isfile(test_path): 123 | yield test_path, attr_path 124 | elif os.path.isdir(test_path): 125 | yield from PackageManager._match(test_path, '*.*', 126 | user_path=attr_path, 127 | no_recurse=attr_no_recurse) 128 | else: 129 | for include_path in glob.iglob(search_path, 130 | root_dir=root_path, 131 | flags=PackageManager.DEFAULT_GLFLAGS | glob.GLOBSTAR if not attr_no_recurse else 0x0): 132 | yield os.path.join(root_path, include_path), attr_path 133 | 134 | # populate files list using absolute paths 135 | else: 136 | if not zip_mode and root_path not in search_path: 137 | PackageManager.log.error(f'Cannot include path outside RootDir: "{search_path}"') 138 | sys.exit(1) 139 | 140 | search_path = os.path.abspath(os.path.normpath(search_path)) 141 | 142 | if os.path.isfile(search_path): 143 | yield search_path, attr_path 144 | else: 145 | yield from PackageManager._match(search_path, '*.*', 146 | user_path=attr_path, 147 | no_recurse=attr_no_recurse) 148 | 149 | for match_node in filter(is_match_node, includes_node): 150 | attr_in: str = match_node.get(XmlAttributeName.IN).strip() 151 | attr_no_recurse: bool = match_node.get(XmlAttributeName.NO_RECURSE) == 'True' # type: ignore 152 | attr_exclude: str = match_node.get(XmlAttributeName.EXCLUDE).strip() 153 | attr_path: str = match_node.get(XmlAttributeName.PATH).strip() # type: ignore 154 | 155 | in_path: str = os.path.normpath(attr_in) 156 | 157 | if in_path == os.pardir or startswith(in_path, os.pardir): 158 | in_path = in_path.replace(os.pardir, os.path.normpath(os.path.join(root_path, os.pardir)), 1) 159 | elif in_path == os.curdir or startswith(in_path, os.curdir): 160 | in_path = in_path.replace(os.curdir, root_path, 1) 161 | 162 | if not os.path.isabs(in_path): 163 | in_path = os.path.join(root_path, in_path) 164 | elif zip_mode and root_path not in in_path: 165 | PackageManager.log.error(f'Cannot match path outside RootDir: "{in_path}"') 166 | sys.exit(1) 167 | 168 | if not os.path.isdir(in_path): 169 | PackageManager.log.error(f'Cannot match path that does not exist or is not a directory: "{in_path}"') 170 | sys.exit(1) 171 | 172 | match_text: str = match_node.text 173 | 174 | if startswith(match_text, '.'): 175 | PackageManager.log.error(f'Match pattern at line {match_node.sourceline} in project file is not a valid wildcard pattern') 176 | sys.exit(1) 177 | 178 | yield from PackageManager._match(in_path, match_text, 179 | exclude_pattern=attr_exclude, 180 | user_path=attr_path, 181 | no_recurse=attr_no_recurse) 182 | 183 | def _fix_package_extension(self, package_name: str) -> str: 184 | if not endswith(package_name, ('.ba2', '.bsa'), ignorecase=True): 185 | return f'{package_name}{self.pak_extension}' 186 | return f'{os.path.splitext(package_name)[0]}{self.pak_extension}' 187 | 188 | def _fix_zip_extension(self, zip_name: str) -> str: 189 | if not endswith(zip_name, '.zip', ignorecase=True): 190 | return f'{zip_name}{self.zip_extension}' 191 | return f'{os.path.splitext(zip_name)[0]}{self.zip_extension}' 192 | 193 | def _try_resolve_project_relative_path(self, path: str) -> str: 194 | if os.path.isabs(path): 195 | return path 196 | 197 | test_path: str = os.path.normpath(os.path.join(self.ppj.project_path, path)) 198 | 199 | return test_path if os.path.isdir(test_path) else '' 200 | 201 | def build_commands(self, containing_folder: str, output_path: str) -> str: 202 | """ 203 | Builds command for creating package with BSArch 204 | """ 205 | arguments = CommandArguments() 206 | 207 | arguments.append(self.options.bsarch_path, enquote_value=True) 208 | arguments.append('pack') 209 | arguments.append(containing_folder, enquote_value=True) 210 | arguments.append(output_path, enquote_value=True) 211 | 212 | compressed_package = PackageManager._can_compress_package(containing_folder) 213 | 214 | flags = wcmatch.RECURSIVE | wcmatch.IGNORECASE 215 | 216 | if self.options.game_type == GameType.FO4 or self.options.game_type == GameType.SF1: 217 | for _ in wcmatch.WcMatch(containing_folder, '!*.dds', flags=flags).imatch(): 218 | arguments.append('-fo4') 219 | break 220 | else: 221 | arguments.append('-fo4dds') 222 | elif self.options.game_type == GameType.SSE: 223 | arguments.append('-sse') 224 | 225 | if not compressed_package: 226 | # SSE crashes when uncompressed BSA has Embed Filenames flag and contains textures 227 | for _ in wcmatch.WcMatch(containing_folder, '*.dds', flags=flags).imatch(): 228 | arguments.append('-af:0x3') 229 | break 230 | else: 231 | arguments.append('-tes5') 232 | 233 | # binary identical files share same data to preserve space 234 | arguments.append('-share') 235 | 236 | if compressed_package: 237 | arguments.append('-z') 238 | 239 | return arguments.join() 240 | 241 | def create_packages(self) -> None: 242 | # clear temporary data 243 | if os.path.isdir(self.options.temp_path): 244 | shutil.rmtree(self.options.temp_path, ignore_errors=True) 245 | 246 | # ensure package path exists 247 | if not os.path.isdir(self.options.package_path): 248 | os.makedirs(self.options.package_path, exist_ok=True) 249 | 250 | file_names = CaseInsensitiveList() 251 | 252 | for i, package_node in enumerate(filter(is_package_node, self.ppj.packages_node)): 253 | attr_file_name: str = package_node.get(XmlAttributeName.NAME) 254 | 255 | # noinspection PyProtectedMember 256 | root_dir: str = self.ppj._get_path(package_node.get(XmlAttributeName.ROOT_DIR), 257 | relative_root_path=self.ppj.project_path, 258 | fallback_path=[self.ppj.project_path, os.path.basename(attr_file_name)]) 259 | 260 | # prevent clobbering files previously created in this session 261 | if attr_file_name in file_names: 262 | attr_file_name = f'{self.ppj.project_name} ({i})' 263 | 264 | if attr_file_name not in file_names: 265 | file_names.append(attr_file_name) 266 | 267 | attr_file_name = self._fix_package_extension(attr_file_name) 268 | 269 | file_path: str = os.path.join(self.options.package_path, attr_file_name) 270 | 271 | self._check_write_permission(file_path) 272 | 273 | PackageManager.log.info(f'Creating "{attr_file_name}"...') 274 | 275 | for source_path, attr_path in self._generate_include_paths(package_node, root_dir): 276 | if os.path.isabs(source_path): 277 | relpath: str = os.path.relpath(source_path, root_dir) 278 | else: 279 | relpath: str = source_path # type: ignore 280 | source_path = os.path.join(self.ppj.project_path, source_path) 281 | 282 | adj_relpath = os.path.normpath(os.path.join(attr_path, relpath)) 283 | 284 | PackageManager.log.debug(f'+ "{adj_relpath.casefold()}"') 285 | 286 | target_path: str = os.path.join(self.options.temp_path, adj_relpath) 287 | 288 | # fix target path if user passes a deeper package root (RootDir) 289 | if endswith(source_path, '.pex', ignorecase=True) and not startswith(relpath, 'scripts', ignorecase=True): 290 | target_path = os.path.join(self.options.temp_path, 'Scripts', relpath) 291 | 292 | os.makedirs(os.path.dirname(target_path), exist_ok=True) 293 | shutil.copy2(source_path, target_path) 294 | 295 | self.includes += 1 296 | 297 | # run bsarch 298 | command: str = self.build_commands(self.options.temp_path, file_path) 299 | ProcessManager.run_bsarch(command) 300 | 301 | # clear temporary data 302 | if os.path.isdir(self.options.temp_path): 303 | shutil.rmtree(self.options.temp_path, ignore_errors=True) 304 | 305 | def create_zip(self) -> None: 306 | # ensure zip output path exists 307 | if not os.path.isdir(self.options.zip_output_path): 308 | os.makedirs(self.options.zip_output_path, exist_ok=True) 309 | 310 | file_names = CaseInsensitiveList() 311 | 312 | for i, zip_node in enumerate(filter(is_zipfile_node, self.ppj.zip_files_node)): 313 | attr_file_name: str = zip_node.get(XmlAttributeName.NAME) 314 | 315 | # prevent clobbering files previously created in this session 316 | if attr_file_name in file_names: 317 | attr_file_name = f'{attr_file_name} ({i})' 318 | 319 | if attr_file_name not in file_names: 320 | file_names.append(attr_file_name) 321 | 322 | attr_file_name = self._fix_zip_extension(attr_file_name) 323 | 324 | file_path: str = os.path.join(self.options.zip_output_path, attr_file_name) 325 | 326 | self._check_write_permission(file_path) 327 | 328 | compress_str: str = self.options.zip_compression or zip_node.get(XmlAttributeName.COMPRESSION) 329 | 330 | try: 331 | compress_type = self.COMPRESS_TYPE[compress_str.casefold()] 332 | except KeyError: 333 | PackageManager.log.error(f'"{compress_str}" is not a valid compression type, defaulting to STORE') 334 | compress_type = 0 335 | 336 | root_dir: str = self.ppj._get_path(zip_node.get(XmlAttributeName.ROOT_DIR), 337 | relative_root_path=self.ppj.project_path, 338 | fallback_path='') 339 | 340 | if root_dir: 341 | PackageManager.log.info(f'Creating "{attr_file_name}"...') 342 | 343 | try: 344 | with zipfile.ZipFile(file_path, mode='w', compression=compress_type) as z: 345 | for include_path, attr_path in self._generate_include_paths(zip_node, root_dir, True): 346 | if not attr_path: 347 | if root_dir in include_path: 348 | arcname = os.path.relpath(include_path, root_dir) 349 | else: 350 | # just add file to zip root 351 | arcname = os.path.basename(include_path) 352 | else: 353 | _, attr_file_name = os.path.split(include_path) 354 | arcname = attr_file_name if attr_path == os.curdir else os.path.join(attr_path, attr_file_name) 355 | 356 | PackageManager.log.debug('+ "{}"'.format(arcname)) 357 | z.write(include_path, arcname, compress_type=compress_type) 358 | 359 | self.includes += 1 360 | 361 | PackageManager.log.info(f'Wrote ZIP file: "{file_path}"') 362 | except PermissionError: 363 | PackageManager.log.error(f'Cannot open ZIP file for writing: "{file_path}"') 364 | sys.exit(1) 365 | else: 366 | PackageManager.log.error(f'Cannot resolve RootDir path to existing folder: "{root_dir}"') 367 | sys.exit(1) 368 | -------------------------------------------------------------------------------- /pyro/PapyrusProject.py: -------------------------------------------------------------------------------- 1 | import configparser 2 | import hashlib 3 | import io 4 | import os 5 | import sys 6 | import time 7 | import typing 8 | 9 | from lxml import etree 10 | from wcmatch import wcmatch 11 | 12 | from pyro.Enums.Event import (Event, 13 | BuildEvent, 14 | ImportEvent, 15 | CompileEvent, 16 | AnonymizeEvent, 17 | PackageEvent, 18 | ZipEvent) 19 | from pyro.CommandArguments import CommandArguments 20 | from pyro.Comparators import (endswith, 21 | is_folder_node, 22 | is_import_node, 23 | is_script_node, 24 | is_variable_node, 25 | is_namespace_path, 26 | startswith) 27 | from pyro.Constants import (GameName, 28 | GameType, 29 | XmlAttributeName, 30 | XmlTagName) 31 | from pyro.PathHelper import PathHelper 32 | from pyro.PexReader import PexReader 33 | from pyro.ProcessManager import ProcessManager 34 | from pyro.ProjectBase import ProjectBase 35 | from pyro.ProjectOptions import ProjectOptions 36 | from pyro.Remotes.RemoteBase import RemoteBase 37 | from pyro.Remotes.GenericRemote import GenericRemote 38 | from pyro.XmlHelper import XmlHelper 39 | from pyro.XmlRoot import XmlRoot 40 | 41 | 42 | class PapyrusProject(ProjectBase): 43 | ppj_root: XmlRoot 44 | folders_node: etree.ElementBase = None 45 | imports_node: etree.ElementBase = None 46 | packages_node: etree.ElementBase = None 47 | scripts_node: etree.ElementBase = None 48 | zip_files_node: etree.ElementBase = None 49 | pre_build_node: etree.ElementBase = None 50 | post_build_node: etree.ElementBase = None 51 | pre_import_node: etree.ElementBase = None 52 | post_import_node: etree.ElementBase = None 53 | pre_compile_node: etree.ElementBase = None 54 | post_compile_node: etree.ElementBase = None 55 | pre_anonymize_node: etree.ElementBase = None 56 | post_anonymize_node: etree.ElementBase = None 57 | pre_package_node: etree.ElementBase = None 58 | post_package_node: etree.ElementBase = None 59 | pre_zip_node: etree.ElementBase = None 60 | post_zip_node: etree.ElementBase = None 61 | 62 | remote: RemoteBase 63 | remote_schemas: tuple = ('https:', 'http:') 64 | 65 | zip_file_name: str = '' 66 | zip_root_path: str = '' 67 | 68 | missing_scripts: dict = {} 69 | pex_paths: list = [] 70 | psc_paths: dict = {} 71 | 72 | def __init__(self, options: ProjectOptions) -> None: 73 | super(PapyrusProject, self).__init__(options) 74 | 75 | xml_parser: etree.XMLParser = etree.XMLParser(remove_blank_text=True, remove_comments=True) 76 | 77 | if self.options.create_project: 78 | sys.exit(1) 79 | 80 | # strip comments from raw text because lxml.etree.XMLParser does not remove XML-unsupported comments 81 | # e.g., '>' 82 | xml_document: io.StringIO = XmlHelper.strip_xml_comments(self.options.input_path) 83 | 84 | project_xml: etree.ElementTree = etree.parse(xml_document, xml_parser) 85 | 86 | self.ppj_root = XmlRoot(project_xml) 87 | 88 | schema: etree.XMLSchema = XmlHelper.validate_schema(self.ppj_root.ns, self.program_path) 89 | 90 | if schema: 91 | try: 92 | schema.assertValid(project_xml) 93 | except etree.DocumentInvalid as e: 94 | PapyrusProject.log.error(f'Failed to validate XML Schema.{os.linesep}\t{e}') 95 | sys.exit(1) 96 | else: 97 | PapyrusProject.log.info('Successfully validated XML Schema.') 98 | 99 | # variables need to be parsed before nodes are updated 100 | variables_node = self.ppj_root.find(XmlTagName.VARIABLES) 101 | if variables_node is not None: 102 | self._parse_variables(variables_node) 103 | 104 | # we need to parse all attributes after validating and before we do anything else 105 | # options can be overridden by arguments when the BuildFacade is initialized 106 | self._update_attributes(self.ppj_root.node) 107 | 108 | if self.options.resolve_project: 109 | xml_output = etree.tostring(self.ppj_root.node, encoding='utf-8', xml_declaration=True, pretty_print=True) 110 | PapyrusProject.log.debug(f'Resolved PPJ. Text output:{os.linesep * 2}{xml_output.decode()}') 111 | sys.exit(1) 112 | 113 | self.options.flags_path = self.ppj_root.get(XmlAttributeName.FLAGS) 114 | self.options.output_path = self.ppj_root.get(XmlAttributeName.OUTPUT) 115 | 116 | if self.options.output_path and not os.path.isabs(self.options.output_path): 117 | self.options.output_path = self.get_output_path() 118 | 119 | def bool_attr(element: etree.Element, attr_name: str) -> bool: 120 | return element is not None and element.get(attr_name) == 'True' 121 | 122 | self.optimize = bool_attr(self.ppj_root, XmlAttributeName.OPTIMIZE) 123 | self.release = bool_attr(self.ppj_root, XmlAttributeName.RELEASE) 124 | self.final = bool_attr(self.ppj_root, XmlAttributeName.FINAL) 125 | 126 | self.options.anonymize = bool_attr(self.ppj_root, XmlAttributeName.ANONYMIZE) 127 | self.options.package = bool_attr(self.ppj_root, XmlAttributeName.PACKAGE) 128 | self.options.zip = bool_attr(self.ppj_root, XmlAttributeName.ZIP) 129 | 130 | self.imports_node = self.ppj_root.find(XmlTagName.IMPORTS) 131 | self.scripts_node = self.ppj_root.find(XmlTagName.SCRIPTS) 132 | self.folders_node = self.ppj_root.find(XmlTagName.FOLDERS) 133 | self.packages_node = self.ppj_root.find(XmlTagName.PACKAGES) 134 | self.zip_files_node = self.ppj_root.find(XmlTagName.ZIP_FILES) 135 | 136 | self.pre_build_node = self.ppj_root.find(XmlTagName.PRE_BUILD_EVENT) 137 | self.use_pre_build_event = bool_attr(self.pre_build_node, XmlAttributeName.USE_IN_BUILD) 138 | 139 | self.post_build_node = self.ppj_root.find(XmlTagName.POST_BUILD_EVENT) 140 | self.use_post_build_event = bool_attr(self.post_build_node, XmlAttributeName.USE_IN_BUILD) 141 | 142 | self.pre_import_node = self.ppj_root.find(XmlTagName.PRE_IMPORT_EVENT) 143 | self.use_pre_import_event = bool_attr(self.pre_import_node, XmlAttributeName.USE_IN_BUILD) 144 | 145 | self.post_import_node = self.ppj_root.find(XmlTagName.POST_IMPORT_EVENT) 146 | self.use_post_import_event = bool_attr(self.post_import_node, XmlAttributeName.USE_IN_BUILD) 147 | 148 | self.pre_compile_node = self.ppj_root.find(XmlTagName.PRE_COMPILE_EVENT) 149 | self.use_pre_compile_event = bool_attr(self.pre_compile_node, XmlAttributeName.USE_IN_BUILD) 150 | 151 | self.post_compile_node = self.ppj_root.find(XmlTagName.POST_COMPILE_EVENT) 152 | self.use_post_compile_event = bool_attr(self.post_compile_node, XmlAttributeName.USE_IN_BUILD) 153 | 154 | self.pre_anonymize_node = self.ppj_root.find(XmlTagName.PRE_ANONYMIZE_EVENT) 155 | self.use_pre_anonymize_event = bool_attr(self.pre_anonymize_node, XmlAttributeName.USE_IN_BUILD) 156 | 157 | self.post_anonymize_node = self.ppj_root.find(XmlTagName.POST_ANONYMIZE_EVENT) 158 | self.use_post_anonymize_event = bool_attr(self.post_anonymize_node, XmlAttributeName.USE_IN_BUILD) 159 | 160 | self.pre_package_node = self.ppj_root.find(XmlTagName.PRE_PACKAGE_EVENT) 161 | self.use_pre_package_event = bool_attr(self.pre_package_node, XmlAttributeName.USE_IN_BUILD) 162 | 163 | self.post_package_node = self.ppj_root.find(XmlTagName.POST_PACKAGE_EVENT) 164 | self.use_post_package_event = bool_attr(self.post_package_node, XmlAttributeName.USE_IN_BUILD) 165 | 166 | self.pre_zip_node = self.ppj_root.find(XmlTagName.PRE_ZIP_EVENT) 167 | self.use_pre_zip_event = bool_attr(self.pre_zip_node, XmlAttributeName.USE_IN_BUILD) 168 | 169 | self.post_zip_node = self.ppj_root.find(XmlTagName.POST_ZIP_EVENT) 170 | self.use_post_zip_event = bool_attr(self.post_zip_node, XmlAttributeName.USE_IN_BUILD) 171 | 172 | if self.options.package and self.packages_node is not None: 173 | if not self.options.package_path: 174 | self.options.package_path = self.packages_node.get(XmlAttributeName.OUTPUT) 175 | 176 | if self.options.zip and self.zip_files_node is not None: 177 | if not self.options.zip_output_path: 178 | self.options.zip_output_path = self.zip_files_node.get(XmlAttributeName.OUTPUT) 179 | 180 | def try_initialize_remotes(self) -> None: 181 | # initialize remote if needed 182 | if self.remote_paths: 183 | if not self.options.remote_temp_path: 184 | self.options.remote_temp_path = self.get_remote_temp_path() 185 | 186 | if self.options.worker_limit == 0: 187 | self.options.worker_limit = self.get_worker_limit() 188 | 189 | self.remote = GenericRemote(access_token=self.options.access_token, 190 | worker_limit=self.options.worker_limit, 191 | force_overwrite=self.options.force_overwrite) 192 | 193 | if not self.options.access_token: 194 | cfg_parser = configparser.ConfigParser() 195 | cfg_path = os.path.join(self.program_path, '.secrets') 196 | 197 | try: 198 | parsed_files = cfg_parser.read(cfg_path) 199 | except configparser.DuplicateSectionError: 200 | PapyrusProject.log.error('Cannot proceed while ".secrets" contains duplicate sections') 201 | sys.exit(1) 202 | except configparser.MissingSectionHeaderError: 203 | PapyrusProject.log.error('Cannot proceed while ".secrets" contains no sections') 204 | sys.exit(1) 205 | 206 | if cfg_path in parsed_files: 207 | self.remote = GenericRemote(config=cfg_parser, 208 | worker_limit=self.options.worker_limit, 209 | force_overwrite=self.options.force_overwrite) 210 | 211 | # validate remote paths 212 | for path in self.remote_paths: 213 | if not self.remote.validate_url(path): 214 | PapyrusProject.log.error(f'Cannot proceed while node contains invalid URL: "{path}"') 215 | sys.exit(1) 216 | 217 | def try_populate_imports(self) -> None: 218 | # we need to populate the list of import paths before we try to determine the game type 219 | # because the game type can be determined from import paths 220 | self.import_paths = self._get_import_paths() 221 | if not self.import_paths: 222 | PapyrusProject.log.error('Failed to build list of import paths') 223 | sys.exit(1) 224 | 225 | # remove project path, if added by user 226 | self.import_paths = [p for p in self.import_paths if not startswith(p, self.project_path, ignorecase=True)] 227 | 228 | # prepend project path 229 | self.import_paths.insert(0, self.project_path) 230 | 231 | self.psc_paths = self._get_psc_paths() 232 | if not self.psc_paths: 233 | PapyrusProject.log.error('Failed to build list of script paths') 234 | sys.exit(1) 235 | 236 | def try_set_game_type(self) -> None: 237 | # we need to set the game type after imports are populated but before pex paths are populated 238 | # allow xml to set game type but defer to passed argument 239 | if self.options.game_type not in GameType.values(): 240 | attr_game_type: str = self.ppj_root.get(XmlAttributeName.GAME, default='') 241 | self.options.game_type = GameType.get(attr_game_type) 242 | 243 | if self.options.game_type: 244 | PapyrusProject.log.info(f'Using game type: {GameName.get(attr_game_type)} (determined from Papyrus Project)') 245 | 246 | if not self.options.game_type: 247 | self.options.game_type = self.get_game_type() 248 | 249 | if not self.options.game_type: 250 | PapyrusProject.log.error('Cannot determine game type from arguments or Papyrus Project') 251 | sys.exit(1) 252 | 253 | def find_missing_scripts(self) -> None: 254 | # get expected pex paths - these paths may not exist and that is okay! 255 | self.pex_paths = self._get_pex_paths() 256 | 257 | # these are relative paths to psc scripts whose pex counterparts are missing 258 | self.missing_scripts: dict = self._find_missing_script_paths() 259 | 260 | def try_set_game_path(self) -> None: 261 | # game type must be set before we call this 262 | if not self.options.game_path: 263 | self.options.game_path = self.get_game_path(self.options.game_type) 264 | 265 | @property 266 | def remote_paths(self) -> list: 267 | """ 268 | Collects list of remote paths from Import and Folder nodes 269 | """ 270 | results: list = [] 271 | 272 | if self.imports_node is not None: 273 | results.extend([node.text for node in filter(is_import_node, self.imports_node) 274 | if startswith(node.text, self.remote_schemas, ignorecase=True)]) 275 | 276 | if self.folders_node is not None: 277 | results.extend([node.text for node in filter(is_folder_node, self.folders_node) 278 | if startswith(node.text, self.remote_schemas, ignorecase=True)]) 279 | 280 | return results 281 | 282 | def _parse_variables(self, variables_node: etree.ElementBase) -> None: 283 | reserved_characters: tuple = ('!', '#', '^', '&', '*') 284 | 285 | for node in filter(is_variable_node, variables_node): 286 | key, value = node.get(XmlAttributeName.NAME, default=''), node.get(XmlAttributeName.VALUE, default='') 287 | 288 | if any([not key, not value]): 289 | continue 290 | 291 | if not key.isalnum(): 292 | PapyrusProject.log.error(f'The name of the variable "{key}" must be an alphanumeric string.') 293 | sys.exit(1) 294 | 295 | if any(c in reserved_characters for c in value): 296 | PapyrusProject.log.error(f'The value of the variable "{key}" contains a reserved character.') 297 | sys.exit(1) 298 | 299 | self.variables.update({key: value}) 300 | 301 | self.variables.update({ 302 | 'UNIXTIME': str(int(time.time())), 303 | 'PROGRAM_PATH': self.program_path, 304 | 'PROJECT_PATH': self.project_path, 305 | 'O_BSARCH_PATH': self.options.bsarch_path, 306 | 'O_COMPILER_PATH': self.options.compiler_path, 307 | 'O_COMPILER_CONFIG_PATH': self.options.compiler_config_path, 308 | 'O_FLAGS_PATH': self.options.flags_path, 309 | 'O_GAME_PATH': self.options.game_path, 310 | 'O_LOG_PATH': self.options.log_path, 311 | 'O_OUTPUT_PATH': self.options.output_path, 312 | 'O_PACKAGE_PATH': self.options.package_path, 313 | 'O_REGISTRY_PATH': self.options.registry_path, 314 | 'O_REMOTE_TEMP_PATH': self.options.remote_temp_path, 315 | 'O_TEMP_PATH': self.options.temp_path, 316 | 'O_ZIP_OUTPUT_PATH': self.options.zip_output_path 317 | }) 318 | 319 | # allow variables to reference other variables 320 | for key, value in self.variables.items(): 321 | self.variables.update({key: self.parse(value)}) 322 | 323 | # complete round trip so that order does not matter 324 | for key in reversed(self.variables.keys()): 325 | value = self.variables[key] 326 | self.variables.update({key: self.parse(value)}) 327 | 328 | def _update_attributes(self, parent_node: etree.ElementBase) -> None: 329 | """Updates attributes of element tree with missing attributes and default values""" 330 | ppj_bool_keys = [ 331 | XmlAttributeName.OPTIMIZE, 332 | XmlAttributeName.RELEASE, 333 | XmlAttributeName.FINAL, 334 | XmlAttributeName.ANONYMIZE, 335 | XmlAttributeName.PACKAGE, 336 | XmlAttributeName.ZIP 337 | ] 338 | 339 | other_bool_keys = [ 340 | XmlAttributeName.NO_RECURSE, 341 | XmlAttributeName.USE_IN_BUILD 342 | ] 343 | 344 | for node in parent_node.getiterator(): 345 | if node.text: 346 | node.text = self.parse(node.text.strip()) 347 | 348 | tag = node.tag.replace('{%s}' % self.ppj_root.ns, '') 349 | 350 | if tag == XmlTagName.PAPYRUS_PROJECT: 351 | if XmlAttributeName.GAME not in node.attrib: 352 | node.set(XmlAttributeName.GAME, '') 353 | if XmlAttributeName.FLAGS not in node.attrib: 354 | node.set(XmlAttributeName.FLAGS, self.options.flags_path) 355 | if XmlAttributeName.OUTPUT not in node.attrib: 356 | node.set(XmlAttributeName.OUTPUT, self.options.output_path) 357 | for key in ppj_bool_keys: 358 | if key not in node.attrib: 359 | node.set(key, 'False') 360 | 361 | elif tag == XmlTagName.PACKAGES: 362 | if XmlAttributeName.OUTPUT not in node.attrib: 363 | node.set(XmlAttributeName.OUTPUT, self.options.package_path) 364 | 365 | elif tag == XmlTagName.PACKAGE: 366 | if XmlAttributeName.NAME not in node.attrib: 367 | node.set(XmlAttributeName.NAME, self.project_name) 368 | if XmlAttributeName.ROOT_DIR not in node.attrib: 369 | node.set(XmlAttributeName.ROOT_DIR, self.project_path) 370 | 371 | elif tag in (XmlTagName.FOLDER, XmlTagName.INCLUDE, XmlTagName.MATCH): 372 | if XmlAttributeName.NO_RECURSE not in node.attrib: 373 | node.set(XmlAttributeName.NO_RECURSE, 'False') 374 | if tag in (XmlTagName.INCLUDE, XmlTagName.MATCH): 375 | if XmlAttributeName.PATH not in node.attrib: 376 | node.set(XmlAttributeName.PATH, '') 377 | if tag == XmlTagName.MATCH: 378 | if XmlAttributeName.IN not in node.attrib: 379 | node.set(XmlAttributeName.IN, os.curdir) 380 | if XmlAttributeName.EXCLUDE not in node.attrib: 381 | node.set(XmlAttributeName.EXCLUDE, '') 382 | 383 | elif tag == XmlTagName.ZIP_FILES: 384 | if XmlAttributeName.OUTPUT not in node.attrib: 385 | node.set(XmlAttributeName.OUTPUT, self.options.zip_output_path) 386 | 387 | elif tag == XmlTagName.ZIP_FILE: 388 | if XmlAttributeName.NAME not in node.attrib: 389 | node.set(XmlAttributeName.NAME, self.project_name) 390 | if XmlAttributeName.ROOT_DIR not in node.attrib: 391 | node.set(XmlAttributeName.ROOT_DIR, self.project_path) 392 | if XmlAttributeName.COMPRESSION not in node.attrib: 393 | node.set(XmlAttributeName.COMPRESSION, 'deflate') 394 | else: 395 | node.set(XmlAttributeName.COMPRESSION, node.get(XmlAttributeName.COMPRESSION).casefold()) 396 | 397 | elif tag in (XmlTagName.PRE_BUILD_EVENT, XmlTagName.POST_BUILD_EVENT, 398 | XmlTagName.PRE_IMPORT_EVENT, XmlTagName.POST_IMPORT_EVENT): 399 | if XmlAttributeName.DESCRIPTION not in node.attrib: 400 | node.set(XmlAttributeName.DESCRIPTION, '') 401 | if XmlAttributeName.USE_IN_BUILD not in node.attrib: 402 | node.set(XmlAttributeName.USE_IN_BUILD, 'True') 403 | 404 | # parse values 405 | for key, value in node.attrib.items(): 406 | value = value.casefold() in ('true', '1') if key in ppj_bool_keys + other_bool_keys else self.parse(value) 407 | node.set(key, str(value)) 408 | 409 | def _calculate_object_name(self, psc_path: str) -> str: 410 | return PathHelper.calculate_relative_object_name(psc_path, self.import_paths) 411 | 412 | def _find_missing_script_paths(self) -> dict: 413 | """Returns list of script paths for compiled scripts that do not exist""" 414 | results: dict = {} 415 | 416 | for object_name, script_path in self.psc_paths.items(): 417 | if endswith(object_name, '.pex', ignorecase=True): 418 | pex_path = os.path.join(self.get_output_path(), object_name) 419 | elif endswith(object_name, '.psc', ignorecase=True): 420 | pex_path = os.path.join(self.get_output_path(), object_name.replace('.psc', '.pex')) 421 | else: 422 | pex_path = os.path.join(self.get_output_path(), object_name + '.pex') 423 | 424 | if not os.path.isfile(pex_path) and script_path not in results: 425 | results[object_name] = script_path 426 | 427 | return results 428 | 429 | def _get_import_paths(self) -> list: 430 | """Returns absolute import paths from Papyrus Project""" 431 | results: list = [] 432 | 433 | if self.imports_node is None: 434 | return [] 435 | 436 | for import_node in filter(is_import_node, self.imports_node): 437 | import_path: str = import_node.text 438 | 439 | if startswith(import_path, self.remote_schemas, ignorecase=True): 440 | local_path = self._get_remote_path(import_node) 441 | PapyrusProject.log.info(f'Adding import path from remote: "{local_path}"...') 442 | results.append(local_path) 443 | continue 444 | 445 | if import_path == os.pardir or startswith(import_path, os.pardir): 446 | import_path = import_path.replace(os.pardir, os.path.normpath(os.path.join(self.project_path, os.pardir)), 1) 447 | elif import_path == os.curdir or startswith(import_path, os.curdir): 448 | import_path = import_path.replace(os.curdir, self.project_path, 1) 449 | 450 | # relative import paths should be relative to the project 451 | if not os.path.isabs(import_path): 452 | import_path = os.path.join(self.project_path, import_path) 453 | 454 | import_path = os.path.normpath(import_path) 455 | 456 | if os.path.isdir(import_path): 457 | results.append(import_path) 458 | else: 459 | PapyrusProject.log.error(f'Import path does not exist: "{import_path}"') 460 | sys.exit(1) 461 | 462 | return PathHelper.uniqify(results) 463 | 464 | def _get_pex_paths(self) -> list: 465 | """ 466 | Returns absolute paths to compiled scripts that may not exist yet in output folder 467 | """ 468 | pex_paths: list = [] 469 | 470 | for object_name, script_path in self.psc_paths.items(): 471 | pex_path = os.path.join(self.options.output_path, os.path.basename(script_path).replace('.psc', '.pex')) 472 | 473 | # do not check if file exists, we do that in _find_missing_script_paths for a different reason 474 | if pex_path not in pex_paths: 475 | pex_paths.append(pex_path) 476 | 477 | return pex_paths 478 | 479 | def get_object_item(self, path: str) -> tuple: 480 | object_name = path if not os.path.isabs(path) else self._calculate_object_name(path) 481 | object_path = path if endswith(path, '.psc', ignorecase=True) else f'{path}.psc' 482 | return object_name, object_path 483 | 484 | def _get_psc_paths(self) -> dict: 485 | """Returns script paths from Folders and Scripts nodes""" 486 | object_names: dict = {} 487 | 488 | # populate object names dictionary 489 | if self.folders_node is not None: 490 | for path in self._get_script_paths_from_folders_node(): 491 | k, v = self.get_object_item(path) 492 | object_names[k] = v 493 | 494 | if self.scripts_node is not None: 495 | for path in self._get_script_paths_from_scripts_node(): 496 | k, v = self.get_object_item(path) 497 | object_names[k] = v 498 | 499 | # convert user paths to absolute paths 500 | for k, v in object_names.items(): 501 | # ignore existing absolute paths 502 | if os.path.isabs(v) and os.path.isfile(v): 503 | continue 504 | 505 | # try to add existing import-relative paths 506 | for import_path in self.import_paths: 507 | if not os.path.isabs(import_path): 508 | import_path = os.path.join(self.project_path, import_path) 509 | 510 | # test for shallow matches 511 | test_path = os.path.join(import_path, v) 512 | if os.path.isfile(test_path): 513 | object_names[k] = test_path 514 | break 515 | 516 | # go deep 517 | matcher = wcmatch.WcMatch(import_path, '*.psc', flags=wcmatch.IGNORECASE | wcmatch.RECURSIVE) 518 | 519 | flag = False 520 | 521 | for f in matcher.imatch(): 522 | if endswith(f, os.path.basename(v), ignorecase=True): 523 | object_names[k] = f 524 | flag = True 525 | break 526 | 527 | if flag: 528 | break 529 | 530 | PapyrusProject.log.info(f'{len(object_names)} unique script paths resolved to absolute paths.') 531 | 532 | return object_names 533 | 534 | def _get_remote_path(self, node: etree.ElementBase) -> str: 535 | import_path: str = node.text 536 | 537 | url_hash = hashlib.sha1(import_path.encode()).hexdigest()[:8] 538 | temp_path = os.path.join(self.options.remote_temp_path, url_hash) 539 | 540 | if self.options.force_overwrite or not os.path.isdir(temp_path): 541 | try: 542 | for message in self.remote.fetch_contents(import_path, temp_path): 543 | if message: 544 | if not startswith(message, 'Failed to load'): 545 | PapyrusProject.log.info(message) 546 | else: 547 | PapyrusProject.log.error(message) 548 | sys.exit(1) 549 | except PermissionError as e: 550 | PapyrusProject.log.error(e) 551 | sys.exit(1) 552 | 553 | if endswith(import_path, '.git', ignorecase=True): 554 | url_path = self.remote.create_local_path(import_path[:-4]) 555 | else: 556 | url_path = self.remote.create_local_path(import_path) 557 | 558 | local_path = os.path.join(temp_path, url_path) 559 | 560 | matcher = wcmatch.WcMatch(local_path, '*.psc', flags=wcmatch.IGNORECASE | wcmatch.RECURSIVE) 561 | 562 | for f in matcher.imatch(): 563 | return os.path.dirname(f) 564 | 565 | return local_path 566 | 567 | @staticmethod 568 | def try_fix_namespace_path(node: etree.ElementBase) -> None: 569 | if is_namespace_path(node): 570 | node.text = node.text.replace(':', os.sep) 571 | 572 | # noinspection DuplicatedCode 573 | def _get_script_paths_from_folders_node(self) -> typing.Generator: 574 | """Returns script paths from the Folders element array""" 575 | for folder_node in filter(is_folder_node, self.folders_node): 576 | self.try_fix_namespace_path(folder_node) 577 | 578 | attr_no_recurse: bool = folder_node.get(XmlAttributeName.NO_RECURSE) == 'True' 579 | 580 | folder_path: str = folder_node.text 581 | 582 | # handle . and .. in path 583 | if folder_path == os.pardir or startswith(folder_path, os.pardir): 584 | folder_path = folder_path.replace(os.pardir, os.path.normpath(os.path.join(self.project_path, os.pardir)), 1) 585 | yield from PathHelper.find_script_paths_from_folder(folder_path, 586 | no_recurse=attr_no_recurse) 587 | continue 588 | 589 | if folder_path == os.curdir or startswith(folder_path, os.curdir): 590 | folder_path = folder_path.replace(os.curdir, self.project_path, 1) 591 | yield from PathHelper.find_script_paths_from_folder(folder_path, 592 | no_recurse=attr_no_recurse) 593 | continue 594 | 595 | if startswith(folder_path, self.remote_schemas, ignorecase=True): 596 | local_path = self._get_remote_path(folder_node) 597 | PapyrusProject.log.info(f'Adding import path from remote: "{local_path}"...') 598 | self.import_paths.insert(0, local_path) 599 | PapyrusProject.log.info(f'Adding folder path from remote: "{local_path}"...') 600 | yield from PathHelper.find_script_paths_from_folder(local_path, 601 | no_recurse=attr_no_recurse) 602 | continue 603 | 604 | folder_path = os.path.normpath(folder_path) 605 | 606 | # try to add absolute path 607 | if os.path.isabs(folder_path) and os.path.isdir(folder_path): 608 | yield from PathHelper.find_script_paths_from_folder(folder_path, 609 | no_recurse=attr_no_recurse) 610 | continue 611 | 612 | # try to add import-relative folder path 613 | for import_path in self.import_paths: 614 | test_path = os.path.join(import_path, folder_path) 615 | if os.path.isdir(test_path): 616 | yield from PathHelper.find_script_paths_from_folder(test_path, 617 | no_recurse=attr_no_recurse) 618 | 619 | # noinspection DuplicatedCode 620 | def _get_script_paths_from_scripts_node(self) -> typing.Generator: 621 | """Returns script paths from the Scripts node""" 622 | for script_node in filter(is_script_node, self.scripts_node): 623 | self.try_fix_namespace_path(script_node) 624 | 625 | script_path: str = script_node.text 626 | 627 | if script_path == os.pardir or script_path == os.curdir: 628 | PapyrusProject.log.error(f'Script path at line {script_node.sourceline} in project file is not a file path') 629 | sys.exit(1) 630 | 631 | # handle . and .. in path 632 | if startswith(script_path, os.pardir): 633 | script_path = script_path.replace(os.pardir, os.path.normpath(os.path.join(self.project_path, os.pardir)), 1) 634 | elif startswith(script_path, os.curdir): 635 | script_path = script_path.replace(os.curdir, self.project_path, 1) 636 | 637 | if os.path.isdir(script_path): 638 | PapyrusProject.log.error(f'Script path at line {script_node.sourceline} in project file is not a file path') 639 | sys.exit(1) 640 | 641 | yield os.path.normpath(script_path) 642 | 643 | def _try_exclude_unmodified_scripts(self) -> dict: 644 | psc_paths: dict = {} 645 | 646 | for object_name, script_path in self.psc_paths.items(): 647 | if endswith(object_name, '.pex', ignorecase=True): 648 | script_name = object_name 649 | elif endswith(object_name, '.psc', ignorecase=True): 650 | script_name = object_name.replace('.psc', '.pex') 651 | else: 652 | script_name = object_name + '.pex' 653 | 654 | # if pex exists, compare time_t in pex header with psc's last modified timestamp 655 | matching_path: str = '' 656 | for pex_path in self.pex_paths: 657 | if endswith(pex_path, script_name, ignorecase=True): 658 | matching_path = pex_path 659 | break 660 | 661 | if not os.path.isfile(matching_path): 662 | continue 663 | 664 | try: 665 | header = PexReader.get_header(matching_path) 666 | except ValueError: 667 | PapyrusProject.log.error(f'Cannot determine compilation time due to unknown magic: "{matching_path}"') 668 | sys.exit(1) 669 | 670 | compiled_time: int = header.compilation_time.value 671 | if os.path.getmtime(script_path) < compiled_time: 672 | continue 673 | 674 | if script_path not in psc_paths: 675 | psc_paths[object_name] = script_path 676 | 677 | return psc_paths 678 | 679 | def build_commands(self) -> tuple[int, list]: 680 | """ 681 | Builds list of commands for compiling scripts 682 | """ 683 | commands: list = [] 684 | 685 | arguments = CommandArguments() 686 | 687 | if self.options.no_incremental_build: 688 | psc_paths: dict = self.psc_paths 689 | else: 690 | psc_paths = self._try_exclude_unmodified_scripts() 691 | 692 | # add .psc scripts whose .pex counterparts do not exist 693 | for object_name, script_path in self.missing_scripts.items(): 694 | if object_name not in psc_paths.keys(): 695 | psc_paths[object_name] = script_path 696 | 697 | # do not try to compile nothing 698 | if psc_paths is None or psc_paths == {}: 699 | return 0, [] 700 | 701 | if endswith(self.get_compiler_path(), 'Caprica.exe', ignorecase=True): 702 | arguments.append(self.get_compiler_path(), enquote_value=True) 703 | 704 | object_names = ';'.join(psc_paths.keys()) 705 | 706 | with open(self.get_compiler_config_path(), encoding='utf-8') as f: 707 | options = f.read().splitlines() 708 | 709 | # disable parallel compilation if the user overrides the default 710 | if self.options.no_parallel and 'parallel-compile=1' in options: 711 | for i, option in enumerate(options): 712 | if startswith(option, 'parallel-compile', ignorecase=True): 713 | options.pop(i) 714 | break 715 | 716 | use_config_file_for_input_paths = False 717 | 718 | if len(object_names) > 32486: # 32766 total - 280 chars for all arguments 719 | use_config_file_for_input_paths = True 720 | options.append(f'input-file={object_names.strip()}\n') 721 | 722 | config_dir_path = os.path.dirname(self.get_compiler_config_path()) 723 | config_file_path = os.path.join(config_dir_path, f'caprica_{str(int(time.time()))}.cfg') 724 | 725 | with open(config_file_path, mode='w', encoding='utf-8') as f: 726 | f.write('\n'.join(options)) 727 | 728 | arguments.append(config_file_path, key='-config-file', enquote_value=True) 729 | 730 | # caprica defaults to starfield 731 | game_name = 'starfield' 732 | if self.options.game_type == GameType.FO4: 733 | game_name = 'fallout4' 734 | elif self.options.game_type in [GameType.TES5, GameType.SSE]: 735 | game_name = 'skyrim' 736 | 737 | arguments.append(game_name, key='g', enquote_value=True) 738 | 739 | arguments.append(self.get_flags_path(), key='f', enquote_value=True) 740 | arguments.append(';'.join(self.import_paths), key='i', enquote_value=True) 741 | arguments.append(self.get_output_path(), key='o', enquote_value=True) 742 | 743 | if not use_config_file_for_input_paths: 744 | arguments.append(object_names, enquote_value=True) 745 | 746 | else: 747 | for object_name, script_path in psc_paths.items(): 748 | arguments.clear() 749 | arguments.append(self.get_compiler_path(), enquote_value=True) 750 | arguments.append(object_name if self.options.game_type == GameType.FO4 else script_path, enquote_value=True) 751 | arguments.append(self.get_flags_path(), key='f', enquote_value=True) 752 | arguments.append(';'.join(self.import_paths), key='i', enquote_value=True) 753 | arguments.append(self.get_output_path(), key='o', enquote_value=True) 754 | 755 | if self.options.game_type in [GameType.FO4, GameType.SF1]: 756 | if self.release: 757 | arguments.append('-release') 758 | 759 | if self.final: 760 | arguments.append('-final') 761 | 762 | if self.optimize: 763 | arguments.append('-op') 764 | 765 | arg_s = arguments.join() 766 | commands.append(arg_s) 767 | 768 | return len(psc_paths.keys()), commands 769 | 770 | def try_run_event(self, event: Event) -> None: 771 | if event == ImportEvent.PRE: 772 | ProcessManager.run_event(self.pre_import_node, self.project_path) 773 | elif event == ImportEvent.POST: 774 | ProcessManager.run_event(self.post_import_node, self.project_path) 775 | elif event == BuildEvent.PRE: 776 | ProcessManager.run_event(self.pre_build_node, self.project_path) 777 | elif event == BuildEvent.POST: 778 | ProcessManager.run_event(self.post_build_node, self.project_path) 779 | elif event == CompileEvent.PRE: 780 | ProcessManager.run_event(self.pre_compile_node, self.project_path) 781 | elif event == CompileEvent.POST: 782 | ProcessManager.run_event(self.post_compile_node, self.project_path) 783 | elif event == AnonymizeEvent.PRE: 784 | ProcessManager.run_event(self.pre_anonymize_node, self.project_path) 785 | elif event == AnonymizeEvent.POST: 786 | ProcessManager.run_event(self.post_anonymize_node, self.project_path) 787 | elif event == PackageEvent.PRE: 788 | ProcessManager.run_event(self.pre_package_node, self.project_path) 789 | elif event == PackageEvent.POST: 790 | ProcessManager.run_event(self.post_package_node, self.project_path) 791 | elif event == ZipEvent.PRE: 792 | ProcessManager.run_event(self.pre_zip_node, self.project_path) 793 | elif event == ZipEvent.POST: 794 | ProcessManager.run_event(self.post_zip_node, self.project_path) 795 | else: 796 | raise NotImplementedError 797 | -------------------------------------------------------------------------------- /pyro/PapyrusProject.xsd: -------------------------------------------------------------------------------- 1 |  2 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | 45 | 46 | 47 | 48 | 49 | 50 | 51 | 52 | 53 | 54 | 55 | 56 | 57 | 58 | 59 | 60 | 61 | 62 | 63 | 64 | 65 | 66 | 67 | 68 | 69 | 70 | 71 | 72 | 73 | 74 | 75 | 76 | 77 | 78 | 79 | 80 | 81 | 82 | 83 | 84 | 85 | 86 | 87 | 88 | 89 | 90 | 91 | 92 | 93 | 94 | 95 | 96 | 97 | 98 | 99 | 100 | 101 | 102 | 103 | 104 | 105 | 106 | 107 | 108 | 109 | 110 | 111 | 112 | 113 | 114 | 115 | 116 | 117 | 118 | 119 | 120 | 121 | 122 | 123 | 124 | 125 | 126 | 127 | 128 | 129 | 130 | 131 | 132 | 133 | 134 | 135 | 136 | 137 | 138 | 139 | 140 | 141 | 142 | 143 | 144 | 145 | 146 | 147 | 148 | 149 | 150 | 151 | 152 | 153 | 154 | 155 | 156 | 157 | 158 | 159 | 160 | 161 | 162 | 163 | 164 | 165 | 166 | 167 | -------------------------------------------------------------------------------- /pyro/PathHelper.py: -------------------------------------------------------------------------------- 1 | import os 2 | from typing import (Generator, 3 | Iterable, 4 | Optional) 5 | from urllib.parse import (unquote_plus, 6 | urlparse) 7 | 8 | from wcmatch import wcmatch 9 | 10 | from pyro.Comparators import (endswith, 11 | startswith) 12 | 13 | 14 | class PathHelper: 15 | @staticmethod 16 | def calculate_absolute_script_path(object_name: str, import_paths: list) -> str: 17 | for import_path in reversed(PathHelper.uniqify(import_paths)): 18 | if not os.path.isabs(import_path): 19 | import_path = os.path.join(os.getcwd(), import_path) 20 | 21 | import_path = os.path.normpath(import_path) 22 | 23 | file_path = os.path.join(import_path, object_name) 24 | 25 | if not endswith(file_path, '.psc', ignorecase=True): 26 | file_path += '.psc' 27 | 28 | if os.path.isfile(file_path): 29 | return file_path 30 | 31 | return '' 32 | 33 | @staticmethod 34 | def calculate_relative_object_name(script_path: str, import_paths: list) -> str: 35 | """Returns import-relative path from absolute path""" 36 | # reverse the list to find the best import path 37 | file_name = os.path.basename(script_path) 38 | 39 | for import_path in reversed(PathHelper.uniqify(import_paths)): 40 | if not os.path.isabs(import_path): 41 | import_path = os.path.join(os.getcwd(), import_path) 42 | 43 | import_path = os.path.normpath(import_path) 44 | 45 | if len(script_path) > len(import_path) and startswith(script_path, import_path, ignorecase=True): 46 | file_name = script_path[len(import_path):] 47 | if file_name[0] == '\\' or file_name[0] == '/': 48 | file_name = file_name[1:] 49 | break 50 | 51 | return file_name 52 | 53 | @staticmethod 54 | def find_script_paths_from_folder(root_dir: str, *, no_recurse: bool, matcher: Optional[wcmatch.WcMatch] = None) -> Generator: 55 | """Yields existing script paths starting from absolute folder path""" 56 | if not matcher: 57 | user_flags = wcmatch.RECURSIVE if not no_recurse else 0x0 58 | matcher = wcmatch.WcMatch(root_dir, '*.psc', flags=wcmatch.IGNORECASE | user_flags) 59 | for script_path in matcher.imatch(): 60 | yield script_path 61 | 62 | @staticmethod 63 | def uniqify(items: Iterable) -> list: 64 | """Returns ordered list without duplicates""" 65 | return list(dict.fromkeys(items)) 66 | 67 | @staticmethod 68 | def url2pathname(url_path: str) -> str: 69 | """Returns normalized unquoted path from URL""" 70 | url = urlparse(url_path) 71 | 72 | netloc: str = url.netloc 73 | path: str = url.path 74 | 75 | if netloc and startswith(netloc, '/'): 76 | netloc = netloc[1:] 77 | 78 | if path and startswith(path, '/'): 79 | path = path[1:] 80 | 81 | return os.path.normpath(unquote_plus(os.path.join(netloc, path))) 82 | -------------------------------------------------------------------------------- /pyro/Performance/CompileData.py: -------------------------------------------------------------------------------- 1 | from dataclasses import (dataclass, 2 | field) 3 | 4 | from pyro.TimeElapsed import TimeElapsed 5 | 6 | 7 | @dataclass 8 | class CompileData: 9 | time: TimeElapsed = field(init=False, default_factory=TimeElapsed) 10 | scripts_count: int = field(init=False, default_factory=int) 11 | success_count: int = field(init=False, default_factory=int) 12 | command_count: int = field(init=False, default_factory=int) 13 | 14 | def __post_init__(self) -> None: 15 | self.time = TimeElapsed() 16 | 17 | @property 18 | def failed_count(self) -> int: 19 | return self.command_count - self.success_count 20 | 21 | def to_string(self) -> str: 22 | raw_time, avg_time = ('{0:.3f}s'.format(t) # type: ignore 23 | for t in (self.time.value(), self.time.average(self.success_count))) 24 | 25 | return f'Compile time: ' \ 26 | f'{raw_time} ({avg_time}/script) - ' \ 27 | f'{self.success_count} succeeded, ' \ 28 | f'{self.failed_count} failed ' \ 29 | f'({self.scripts_count} scripts)' 30 | 31 | @dataclass 32 | class CompileDataCaprica(CompileData): 33 | @property 34 | def failed_count(self) -> int: 35 | return 1 if self.success_count == 0 else 0 36 | 37 | def to_string(self) -> str: 38 | raw_time = '{0:.3f}s'.format(self.time.value()) 39 | avg_time = '{0:.4f}s'.format(self.time.average(self.scripts_count)) 40 | 41 | return f'Compile time: ' \ 42 | f'{raw_time} ({avg_time}/script) - ' \ 43 | f'({self.scripts_count} scripts)' 44 | -------------------------------------------------------------------------------- /pyro/Performance/PackageData.py: -------------------------------------------------------------------------------- 1 | from dataclasses import (dataclass, 2 | field) 3 | 4 | from pyro.TimeElapsed import TimeElapsed 5 | 6 | 7 | @dataclass 8 | class PackageData: 9 | time: TimeElapsed = field(init=False, default_factory=TimeElapsed) 10 | file_count: int = field(init=False, default_factory=int) 11 | 12 | def __post_init__(self) -> None: 13 | self.time = TimeElapsed() 14 | 15 | def to_string(self) -> str: 16 | raw_time, avg_time = ('{0:.3f}s'.format(t) 17 | for t in (self.time.value(), self.time.average(self.file_count))) 18 | 19 | return f'Package time: ' \ 20 | f'{raw_time} ({avg_time}/file, {self.file_count} files)' 21 | -------------------------------------------------------------------------------- /pyro/Performance/ZippingData.py: -------------------------------------------------------------------------------- 1 | from dataclasses import (dataclass, 2 | field) 3 | 4 | from pyro.TimeElapsed import TimeElapsed 5 | 6 | 7 | @dataclass 8 | class ZippingData: 9 | time: TimeElapsed = field(init=False, default_factory=TimeElapsed) 10 | file_count: int = field(init=False, default_factory=int) 11 | 12 | def __post_init__(self) -> None: 13 | self.time = TimeElapsed() 14 | 15 | def to_string(self) -> str: 16 | raw_time, avg_time = ('{0:.3f}s'.format(t) 17 | for t in (self.time.value(), self.time.average(self.file_count))) 18 | 19 | return f'Zipping time: ' \ 20 | f'{raw_time} ({avg_time}/file, {self.file_count} files)' 21 | -------------------------------------------------------------------------------- /pyro/PexHeader.py: -------------------------------------------------------------------------------- 1 | from dataclasses import dataclass, field 2 | from typing import (IO, # type: ignore 3 | Literal) 4 | 5 | from pyro.PexTypes import PexInt 6 | from pyro.PexTypes import PexStr 7 | 8 | 9 | @dataclass 10 | class PexHeader: 11 | size: int = field(init=False, default=0) 12 | endianness: Literal['little', 'big'] = field(init=False, default='little') 13 | 14 | magic: PexInt = field(init=False, default_factory=PexInt) 15 | major_version: PexInt = field(init=False, default_factory=PexInt) 16 | minor_version: PexInt = field(init=False, default_factory=PexInt) 17 | game_id: PexInt = field(init=False, default_factory=PexInt) 18 | compilation_time: PexInt = field(init=False, default_factory=PexInt) 19 | script_path_size: PexInt = field(init=False, default_factory=PexInt) 20 | script_path: PexStr = field(init=False, default_factory=PexStr) 21 | user_name_size: PexInt = field(init=False, default_factory=PexInt) 22 | user_name: PexStr = field(init=False, default_factory=PexStr) 23 | computer_name_size: PexInt = field(init=False, default_factory=PexInt) 24 | computer_name: PexStr = field(init=False, default_factory=PexStr) 25 | 26 | def read(self, f: IO, name: str, length: int) -> None: 27 | """Reads a set of bytes and their offset to an attribute by name""" 28 | try: 29 | obj = getattr(self, name) 30 | except AttributeError: 31 | # this is just a guard against developer error 32 | raise AttributeError(f'Attribute "{name}" does not exist in PexHeader') 33 | else: 34 | obj.offset, obj.data = f.tell(), f.read(length) 35 | 36 | if isinstance(obj, PexInt): 37 | obj.value = int.from_bytes(obj.data, self.endianness, signed=False) 38 | elif isinstance(obj, PexStr): 39 | try: 40 | obj.value = obj.data.decode('ascii') 41 | except UnicodeDecodeError: 42 | obj.value = obj.data.decode('utf-8') 43 | -------------------------------------------------------------------------------- /pyro/PexReader.py: -------------------------------------------------------------------------------- 1 | import binascii 2 | import json 3 | import os 4 | 5 | from pyro.PexHeader import PexHeader 6 | from pyro.PexTypes import PexInt, PexStr 7 | 8 | 9 | class PexReader: 10 | @staticmethod 11 | def get_header(path: str) -> PexHeader: 12 | header = PexHeader() 13 | 14 | with open(path, mode='rb') as f: 15 | f.seek(0, os.SEEK_SET) 16 | 17 | header.read(f, 'magic', 4) 18 | 19 | if header.magic.value == 0xFA57C0DE: # Fallout 4 20 | header.endianness = 'little' 21 | elif header.magic.value == 0xDEC057FA: # Skyrim LE/SE 22 | header.endianness = 'big' 23 | else: 24 | raise ValueError(f'Cannot determine endianness from file magic in "{path}"') 25 | 26 | header.read(f, 'major_version', 1) 27 | header.read(f, 'minor_version', 1) 28 | header.read(f, 'game_id', 2) 29 | header.read(f, 'compilation_time', 8) 30 | header.read(f, 'script_path_size', 2) 31 | header.read(f, 'script_path', header.script_path_size.value) 32 | header.read(f, 'user_name_size', 2) 33 | header.read(f, 'user_name', header.user_name_size.value) 34 | header.read(f, 'computer_name_size', 2) 35 | header.read(f, 'computer_name', header.computer_name_size.value) 36 | header.size = f.tell() 37 | 38 | return header 39 | 40 | @staticmethod 41 | def dump(file_path: str) -> str: 42 | header = PexReader.get_header(file_path).__dict__ 43 | 44 | for key, value in header.items(): 45 | if not isinstance(value, (PexInt, PexStr)): 46 | continue 47 | 48 | header[key] = value.__dict__ 49 | 50 | for k, v in header[key].items(): 51 | if not isinstance(v, bytes): 52 | continue 53 | 54 | bytes2hex = binascii.hexlify(v).decode('ascii').upper() 55 | header[key][k] = ' '.join(bytes2hex[i:i + 2] for i in range(0, len(bytes2hex), 2)) 56 | 57 | return json.dumps(header, indent=4) 58 | -------------------------------------------------------------------------------- /pyro/PexTypes.py: -------------------------------------------------------------------------------- 1 | class PexData: 2 | offset: int 3 | data: bytes 4 | value: object 5 | 6 | 7 | class PexInt(PexData): 8 | value: int 9 | 10 | 11 | class PexStr(PexData): 12 | value: str 13 | -------------------------------------------------------------------------------- /pyro/ProcessManager.py: -------------------------------------------------------------------------------- 1 | import logging 2 | import os 3 | import re 4 | import subprocess 5 | from decimal import Decimal 6 | 7 | from lxml import etree 8 | 9 | from pyro.Enums.ProcessState import ProcessState 10 | 11 | from pyro.Comparators import (is_command_node, 12 | startswith) 13 | from pyro.Constants import XmlAttributeName 14 | 15 | 16 | class ProcessManager: 17 | log: logging.Logger = logging.getLogger('pyro') 18 | 19 | @staticmethod 20 | def _format_time(hours: Decimal, minutes: Decimal, seconds: Decimal) -> str: 21 | if hours.compare(0) == 1 and minutes.compare(0) == 1 and seconds.compare(0) == 1: 22 | return f'{hours}h {minutes}m {seconds}s' 23 | if hours.compare(0) == 0 and minutes.compare(0) == 1 and seconds.compare(0) == 1: 24 | return f'{minutes}m {seconds}s' 25 | if hours.compare(0) == 0 and minutes.compare(0) == 0 and seconds.compare(0) == 1: 26 | return f'{seconds}s' 27 | return f'{hours}h {minutes}m {seconds}s' 28 | 29 | @staticmethod 30 | def run_event(event_node: etree.ElementBase, project_path: str) -> None: 31 | if event_node is None: 32 | return 33 | 34 | ProcessManager.log.info(event_node.get(XmlAttributeName.DESCRIPTION)) 35 | 36 | ws: re.Pattern = re.compile('[ \t\n\r]+') 37 | 38 | environ: dict = os.environ.copy() 39 | # command: str = ' && '.join( 40 | # ws.sub(' ', node.text) 41 | # for node in filter(is_command_node, event_node) 42 | # ) 43 | 44 | for node in filter(is_command_node, event_node): 45 | command = ws.sub(' ', node.text) 46 | ProcessManager.run_command(command, project_path, environ) 47 | 48 | @staticmethod 49 | def run_command(command: str, cwd: str, env: dict) -> ProcessState: 50 | try: 51 | process = subprocess.Popen(command, 52 | stdout=subprocess.PIPE, 53 | stderr=subprocess.STDOUT, 54 | universal_newlines=True, 55 | shell=True, 56 | cwd=cwd, 57 | env=env) 58 | except WindowsError as e: 59 | ProcessManager.log.error(f'Cannot create process because: {e.strerror}') 60 | return ProcessState.FAILURE 61 | 62 | try: 63 | while process.poll() is None: 64 | if (line := process.stdout.readline().strip()) != '': 65 | ProcessManager.log.info(line) 66 | 67 | except KeyboardInterrupt: 68 | try: 69 | process.terminate() 70 | except OSError: 71 | ProcessManager.log.error('Process interrupted by user.') 72 | return ProcessState.INTERRUPTED 73 | 74 | return ProcessState.SUCCESS 75 | 76 | @staticmethod 77 | def run_bsarch(command: str) -> ProcessState: 78 | """ 79 | Creates bsarch process and logs output to console 80 | 81 | :param command: Command to execute, including absolute path to executable and its arguments 82 | :return: ProcessState (SUCCESS, FAILURE, INTERRUPTED, ERRORS) 83 | """ 84 | try: 85 | process = subprocess.Popen(command, 86 | stdout=subprocess.PIPE, 87 | stderr=subprocess.STDOUT, 88 | universal_newlines=True) 89 | except WindowsError as e: 90 | ProcessManager.log.error(f'Cannot create process because: {e.strerror}') 91 | return ProcessState.FAILURE 92 | 93 | exclusions = ( 94 | '*', 95 | '[', 96 | 'Archive Flags', 97 | 'Bit', 98 | 'BSArch', 99 | 'Compressed', 100 | 'Done', 101 | 'Embed', 102 | 'Files', 103 | 'Format', 104 | 'Packer', 105 | 'Retain', 106 | 'Startup', 107 | 'Version', 108 | 'XBox', 109 | 'XMem' 110 | ) 111 | 112 | try: 113 | while process.poll() is None: 114 | if (line := process.stdout.readline().strip()) != '': 115 | if startswith(line, exclusions): 116 | continue 117 | 118 | if startswith(line, 'Packing'): 119 | package_path = line.split(':', 1)[1].strip() 120 | ProcessManager.log.info(f'Packaging folder "{package_path}"...') 121 | continue 122 | 123 | if startswith(line, 'Archive Name'): 124 | archive_path = line.split(':', 1)[1].strip() 125 | ProcessManager.log.info(f'Building "{archive_path}"...') 126 | continue 127 | 128 | ProcessManager.log.info(line) 129 | 130 | except KeyboardInterrupt: 131 | try: 132 | process.terminate() 133 | except OSError: 134 | ProcessManager.log.error('Process interrupted by user.') 135 | return ProcessState.INTERRUPTED 136 | 137 | return ProcessState.SUCCESS 138 | 139 | @staticmethod 140 | def run_compiler(command: str) -> ProcessState: 141 | """ 142 | Creates compiler process and logs output to console 143 | 144 | :param command: Command to execute, including absolute path to executable and its arguments 145 | :return: ProcessState (SUCCESS, FAILURE, INTERRUPTED, ERRORS) 146 | """ 147 | command_size = len(command) 148 | 149 | if command_size > 32766: 150 | ProcessManager.log.error(f'Cannot create process because command exceeds max length: {command_size}') 151 | return ProcessState.FAILURE 152 | 153 | try: 154 | process = subprocess.Popen(command, 155 | stdout=subprocess.PIPE, 156 | stderr=subprocess.STDOUT, 157 | universal_newlines=True) 158 | except WindowsError as e: 159 | ProcessManager.log.error(f'Cannot create process because: {e.strerror}') 160 | return ProcessState.FAILURE 161 | 162 | exclusions = ( 163 | 'Assembly', 164 | 'Batch', 165 | 'Compilation', 166 | 'Copyright', 167 | 'Failed', 168 | 'No output', 169 | 'Papyrus', 170 | 'Starting' 171 | ) 172 | 173 | line_error = re.compile(r'(.*)(\(-?\d*\.?\d+,-?\d*\.?\d+\)):\s+(.*)') 174 | 175 | error_count = 0 176 | 177 | try: 178 | while process.poll() is None: 179 | if (line := process.stdout.readline().strip()) != '': 180 | if startswith(line, exclusions): 181 | continue 182 | 183 | if (match := line_error.search(line)) is not None: 184 | path, location, message = match.groups() 185 | head, tail = os.path.split(path) 186 | ProcessManager.log.error(f'COMPILATION FAILED: ' 187 | f'{os.path.basename(head)}\\{tail}{location}: {message}') 188 | process.terminate() 189 | error_count += 1 190 | 191 | elif startswith(line, ('Error', 'Fatal Error'), ignorecase=True): 192 | ProcessManager.log.error(line) 193 | process.terminate() 194 | error_count += 1 195 | 196 | elif 'error(s)' not in line: 197 | ProcessManager.log.info(line) 198 | 199 | except KeyboardInterrupt: 200 | try: 201 | process.terminate() 202 | except OSError: 203 | ProcessManager.log.error('Process interrupted by user.') 204 | return ProcessState.INTERRUPTED 205 | 206 | return ProcessState.SUCCESS if error_count == 0 else ProcessState.ERRORS 207 | -------------------------------------------------------------------------------- /pyro/ProjectBase.py: -------------------------------------------------------------------------------- 1 | import logging 2 | import os 3 | import sys 4 | from typing import Union 5 | 6 | from pyro.Comparators import (endswith, 7 | startswith) 8 | from pyro.Constants import (FlagsName, 9 | GameName, 10 | GameType) 11 | from pyro.ProjectOptions import ProjectOptions 12 | from pyro.StringTemplate import StringTemplate 13 | 14 | 15 | class ProjectBase: 16 | log: logging.Logger = logging.getLogger('pyro') 17 | 18 | options: ProjectOptions 19 | 20 | variables: dict = {} 21 | 22 | program_path: str = '' 23 | project_name: str = '' 24 | project_path: str = '' 25 | 26 | import_paths: list = [] 27 | 28 | final: bool = False 29 | optimize: bool = False 30 | release: bool = False 31 | 32 | def __init__(self, options: ProjectOptions) -> None: 33 | self.options = options 34 | 35 | self.program_path = os.path.dirname(__file__) 36 | if endswith(sys.argv[0], ('pyro', '.exe')): 37 | self.program_path = os.path.abspath(os.path.join(self.program_path, os.pardir)) 38 | 39 | self.project_name = os.path.splitext(os.path.basename(self.options.input_path))[0] 40 | self.project_path = os.path.dirname(self.options.input_path) 41 | os.chdir(self.project_path) 42 | 43 | def __setattr__(self, key: str, value: object) -> None: 44 | if isinstance(value, str) and endswith(key, 'path'): 45 | if os.altsep in value: 46 | value = os.path.normpath(value) 47 | elif isinstance(value, list) and endswith(key, 'paths'): 48 | value = [os.path.normpath(path) if path != os.curdir else path for path in value] 49 | super(ProjectBase, self).__setattr__(key, value) 50 | 51 | @staticmethod 52 | def _get_path(path: str, *, relative_root_path: str, fallback_path: Union[str, list]) -> str: 53 | """ 54 | Returns absolute path from path or fallback path if path empty or unset 55 | 56 | :param path: A relative or absolute path 57 | :param relative_root_path: Absolute path to directory to join with relative path 58 | :param fallback_path: Absolute path to return if path empty or unset 59 | """ 60 | if path or startswith(path, (os.curdir, os.pardir)): 61 | path = os.path.expanduser(os.path.expandvars(path)) 62 | return os.path.normpath(path) if os.path.isabs(path) else os.path.normpath(os.path.join(relative_root_path, path)) 63 | if isinstance(fallback_path, list): 64 | return os.path.abspath(os.path.join(*fallback_path)) 65 | return fallback_path 66 | 67 | def parse(self, value: str) -> str: 68 | """Expands string tokens and environment variables, and returns the parsed string""" 69 | t = StringTemplate(value) 70 | try: 71 | retval = os.path.expanduser(os.path.expandvars(t.substitute(self.variables))) 72 | if os.path.isabs(retval): 73 | retval = os.path.normpath(retval) 74 | return retval 75 | except KeyError as e: 76 | ProjectBase.log.error(f'Failed to parse variable "{e.args[0]}" in "{value}". Is the variable name correct?') 77 | sys.exit(1) 78 | 79 | # build arguments 80 | def get_worker_limit(self) -> int: 81 | """ 82 | Returns worker limit from arguments 83 | 84 | Used by: BuildFacade 85 | """ 86 | if self.options.worker_limit > 0: 87 | return self.options.worker_limit 88 | try: 89 | cpu_count = os.cpu_count() # can be None if indeterminate 90 | if cpu_count is None: 91 | raise ValueError('The number of CPUs in the system is indeterminate') 92 | except (NotImplementedError, ValueError): 93 | return 2 94 | else: 95 | return cpu_count 96 | 97 | # compiler arguments 98 | def get_compiler_path(self) -> str: 99 | """ 100 | Returns absolute compiler path from arguments 101 | 102 | Used by: BuildFacade 103 | """ 104 | return self._get_path(self.options.compiler_path, 105 | relative_root_path=os.getcwd(), 106 | fallback_path=[self.options.game_path, 'Papyrus Compiler', 'PapyrusCompiler.exe']) 107 | 108 | def get_compiler_config_path(self) -> str: 109 | """Returns absolute compiler config file path from arguments""" 110 | return self._get_path(self.options.compiler_config_path, 111 | relative_root_path=os.getcwd(), 112 | fallback_path=[self.program_path, 'tools', 'caprica.cfg']) 113 | 114 | def get_flags_path(self) -> str: 115 | """ 116 | Returns absolute flags path or flags file name from arguments or game path 117 | 118 | Used by: BuildFacade 119 | """ 120 | if self.options.flags_path: 121 | if startswith(self.options.flags_path, tuple(FlagsName.values()), ignorecase=True): 122 | return self.options.flags_path 123 | if os.path.isabs(self.options.flags_path): 124 | return self.options.flags_path 125 | return os.path.join(self.project_path, self.options.flags_path) 126 | 127 | if self.options.game_path: 128 | if endswith(self.options.game_path, GameName.FO4, ignorecase=True): 129 | return FlagsName.FO4 130 | if endswith(self.options.game_path, GameName.SF1, ignorecase=True): 131 | return FlagsName.SF1 132 | 133 | return FlagsName.TES5 134 | 135 | def get_output_path(self) -> str: 136 | """ 137 | Returns absolute output path from arguments 138 | 139 | Used by: BuildFacade 140 | """ 141 | return self._get_path(self.options.output_path, 142 | relative_root_path=self.project_path, 143 | fallback_path=[self.program_path, 'out']) 144 | 145 | # game arguments 146 | def get_game_path(self, game_type: str = '') -> str: 147 | """ 148 | Returns absolute game path from arguments or Windows Registry 149 | 150 | Used by: BuildFacade, ProjectBase 151 | """ 152 | if self.options.game_path: 153 | if os.path.isabs(self.options.game_path): 154 | return self.options.game_path 155 | return os.path.join(os.getcwd(), self.options.game_path) 156 | 157 | if sys.platform == 'win32': 158 | return self.get_installed_path(game_type) 159 | 160 | raise FileNotFoundError('Cannot determine game path') 161 | 162 | def get_registry_path(self, game_type: str = '') -> str: 163 | """Returns path to game installed path in Windows Registry from game type""" 164 | if not self.options.registry_path: 165 | game_type = self.options.game_type if not game_type else game_type 166 | if game_type in GameType.values(): 167 | game_name = GameName.get(game_type) 168 | else: 169 | raise KeyError('Cannot determine registry path from game type') 170 | if startswith(game_name, 'Starfield'): 171 | return r'HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Uninstall\Steam App 1716740\InstallLocation' 172 | if startswith(game_name, 'Fallout'): 173 | game_name = game_name.replace(' ', '') 174 | return rf'HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Bethesda Softworks\{game_name}\Installed Path' 175 | return self.options.registry_path.replace('/', '\\') 176 | 177 | def get_installed_path(self, game_type: str = '') -> str: 178 | """ 179 | Returns path to game installed path in Windows Registry 180 | 181 | Used by: BuildFacade, ProjectBase 182 | """ 183 | import winreg 184 | 185 | registry_path, registry_type = self.options.registry_path, winreg.HKEY_LOCAL_MACHINE 186 | 187 | game_type = self.options.game_type if not game_type else game_type 188 | 189 | if not registry_path: 190 | registry_path = self.get_registry_path(game_type) 191 | 192 | hkey, key_path = registry_path.split(os.sep, maxsplit=1) 193 | key_head, key_tail = os.path.split(key_path) 194 | 195 | # fix absolute registry paths, if needed 196 | if any(hkey == value for value in ('HKCU', 'HKEY_CURRENT_USER')): 197 | registry_type = winreg.HKEY_CURRENT_USER 198 | 199 | try: 200 | registry_key = winreg.OpenKey(registry_type, key_head, 0, winreg.KEY_READ) 201 | reg_value, _ = winreg.QueryValueEx(registry_key, key_tail) 202 | winreg.CloseKey(registry_key) 203 | except WindowsError: 204 | ProjectBase.log.error(f'Installed Path for {game_type} ' 205 | f'does not exist in Windows Registry. Run the game launcher once, then try again.') 206 | sys.exit(1) 207 | 208 | # noinspection PyUnboundLocalVariable 209 | if not os.path.isdir(reg_value): 210 | ProjectBase.log.error(f'Installed Path for {game_type} does not exist: {reg_value}') 211 | sys.exit(1) 212 | 213 | return reg_value 214 | 215 | # bsarch arguments 216 | def get_bsarch_path(self) -> str: 217 | """Returns absolute bsarch path from arguments""" 218 | return self._get_path(self.options.bsarch_path, 219 | relative_root_path=os.getcwd(), 220 | fallback_path=[self.program_path, 'tools', 'bsarch.exe']) 221 | 222 | def get_package_path(self) -> str: 223 | """Returns absolute package path from arguments""" 224 | return self._get_path(self.options.package_path, 225 | relative_root_path=self.project_path, 226 | fallback_path=[self.program_path, 'dist']) 227 | 228 | def get_temp_path(self) -> str: 229 | """Returns absolute package temp path from arguments""" 230 | return self._get_path(self.options.temp_path, 231 | relative_root_path=os.getcwd(), 232 | fallback_path=[self.program_path, 'temp']) 233 | 234 | # zip arguments 235 | def get_zip_output_path(self) -> str: 236 | """Returns absolute zip output path from arguments""" 237 | return self._get_path(self.options.zip_output_path, 238 | relative_root_path=self.project_path, 239 | fallback_path=[self.program_path, 'dist']) 240 | 241 | # remote arguments 242 | def get_remote_temp_path(self) -> str: 243 | return self._get_path(self.options.remote_temp_path, 244 | relative_root_path=self.project_path, 245 | fallback_path=[self.program_path, 'remote']) 246 | 247 | @staticmethod 248 | def _get_game_type_from_path(path: str) -> str: 249 | parts: list = path.casefold().split(os.sep) 250 | if GameName.SF1.casefold() in parts: 251 | return GameType.SF1 252 | if GameName.SF1.casefold().replace(' ', '') in parts: 253 | return GameType.SF1 254 | if GameName.FO4.casefold() in parts: 255 | return GameType.FO4 256 | if GameName.FO4.casefold().replace(' ', '') in parts: 257 | return GameType.FO4 258 | if GameName.SSE.casefold() in parts: 259 | return GameType.SSE 260 | if GameName.TES5.casefold() in parts: 261 | return GameType.TES5 262 | return '' 263 | 264 | # program arguments 265 | def get_game_type(self) -> str: 266 | """Returns game type from arguments or Papyrus Project""" 267 | if self.options.game_type: 268 | return self.options.game_type 269 | 270 | if self.options.game_path: 271 | for game_type, game_name in GameName.items(): 272 | if endswith(self.options.game_path, game_name, ignorecase=True): 273 | ProjectBase.log.info(f'Using game type: {game_name} (determined from game path)') 274 | return GameType.get(game_type) 275 | 276 | if self.options.registry_path: 277 | game_type = self._get_game_type_from_path(self.options.registry_path) 278 | if game_type: 279 | ProjectBase.log.info(f'Using game type: {GameName.get(game_type)} (determined from registry path)') 280 | return game_type 281 | 282 | if self.import_paths: 283 | for import_path in reversed(self.import_paths): 284 | game_type = self._get_game_type_from_path(import_path) 285 | if game_type: 286 | ProjectBase.log.info(f'Using game type: {GameName.get(game_type)} (determined from import paths)') 287 | return game_type 288 | 289 | if self.options.flags_path: 290 | if endswith(self.options.flags_path, FlagsName.SF1, ignorecase=True): 291 | ProjectBase.log.info(f'Using game type: {GameName.FO4} (determined from flags path)') 292 | return GameType.SF1 293 | 294 | if endswith(self.options.flags_path, FlagsName.FO4, ignorecase=True): 295 | ProjectBase.log.info(f'Using game type: {GameName.FO4} (determined from flags path)') 296 | return GameType.FO4 297 | 298 | if endswith(self.options.flags_path, FlagsName.TES5, ignorecase=True): 299 | try: 300 | self.get_game_path('sse') 301 | except FileNotFoundError: 302 | ProjectBase.log.info(f'Using game type: {GameName.TES5} (determined from flags path)') 303 | return GameType.TES5 304 | else: 305 | ProjectBase.log.info(f'Using game type: {GameName.SSE} (determined from flags path)') 306 | return GameType.SSE 307 | 308 | raise AssertionError('Cannot determine game type from game path, registry path, import paths, or flags path') 309 | 310 | def get_log_path(self) -> str: 311 | """Returns absolute log path from arguments""" 312 | return self._get_path(self.options.log_path, 313 | relative_root_path=os.getcwd(), 314 | fallback_path=[self.program_path, 'logs']) 315 | -------------------------------------------------------------------------------- /pyro/ProjectOptions.py: -------------------------------------------------------------------------------- 1 | import os 2 | 3 | from dataclasses import dataclass, field 4 | 5 | from pyro.Comparators import endswith 6 | 7 | 8 | @dataclass 9 | class ProjectOptions: 10 | args: dict = field(repr=False, default_factory=dict) 11 | anonymize: bool = field(init=False, default_factory=bool) 12 | package: bool = field(init=False, default_factory=bool) 13 | zip: bool = field(init=False, default_factory=bool) 14 | 15 | # required arguments 16 | input_path: str = field(init=False, default_factory=str) 17 | 18 | # build arguments 19 | ignore_errors: bool = field(init=False, default_factory=bool) 20 | no_implicit_imports: bool = field(init=False, default_factory=bool) 21 | no_incremental_build: bool = field(init=False, default_factory=bool) 22 | no_parallel: bool = field(init=False, default_factory=bool) 23 | worker_limit: int = field(init=False, default_factory=int) 24 | 25 | # game arguments 26 | game_type: str = field(init=False, default_factory=str) 27 | game_path: str = field(init=False, default_factory=str) 28 | registry_path: str = field(init=False, default_factory=str) 29 | 30 | # compiler arguments 31 | compiler_path: str = field(init=False, default_factory=str) 32 | flags_path: str = field(init=False, default_factory=str) 33 | output_path: str = field(init=False, default_factory=str) 34 | 35 | # bsarch arguments 36 | bsarch_path: str = field(init=False, default_factory=str) 37 | package_path: str = field(init=False, default_factory=str) 38 | temp_path: str = field(init=False, default_factory=str) 39 | 40 | # zip arguments 41 | zip_compression: str = field(init=False, default_factory=str) 42 | zip_output_path: str = field(init=False, default_factory=str) 43 | 44 | # remote arguments 45 | access_token: str = field(init=False, default_factory=str) 46 | force_overwrite: bool = field(init=False, default_factory=bool) 47 | remote_temp_path: str = field(init=False, default_factory=str) 48 | 49 | # program arguments 50 | log_path: str = field(init=False, default_factory=str) 51 | create_project: bool = field(init=False, default_factory=bool) 52 | resolve_project: bool = field(init=False, default_factory=bool) 53 | 54 | def __post_init__(self) -> None: 55 | for attr_key, attr_value in self.__dict__.items(): 56 | if attr_key == 'args': 57 | continue 58 | arg_value = self.args.get(attr_key) 59 | if arg_value and arg_value != attr_value: 60 | setattr(self, attr_key, arg_value) 61 | 62 | def __setattr__(self, key: str, value: object) -> None: 63 | if value and isinstance(value, str): 64 | # sanitize paths 65 | if endswith(key, 'path', ignorecase=True) and os.altsep in value: 66 | value = os.path.normpath(value) 67 | if key in ('game_type', 'zip_compression'): 68 | value = value.casefold() 69 | 70 | super(ProjectOptions, self).__setattr__(key, value) 71 | -------------------------------------------------------------------------------- /pyro/PyroArgumentParser.py: -------------------------------------------------------------------------------- 1 | from argparse import ArgumentParser 2 | 3 | 4 | class PyroArgumentParser(ArgumentParser): 5 | def format_help(self) -> str: 6 | formatter = self._get_formatter() 7 | 8 | # description 9 | formatter.add_text(self.description) 10 | 11 | # usage 12 | formatter.add_usage(self.usage, self._actions, 13 | self._mutually_exclusive_groups) 14 | 15 | # positionals, optionals and user-defined groups 16 | for action_group in self._action_groups: 17 | formatter.start_section(action_group.title) 18 | formatter.add_text(action_group.description) 19 | # noinspection PyProtectedMember 20 | formatter.add_arguments(action_group._group_actions) 21 | formatter.end_section() 22 | 23 | # epilog 24 | formatter.add_text(self.epilog) 25 | 26 | # determine help from format above 27 | return formatter.format_help() 28 | -------------------------------------------------------------------------------- /pyro/PyroRawDescriptionHelpFormatter.py: -------------------------------------------------------------------------------- 1 | import argparse 2 | 3 | 4 | class PyroRawDescriptionHelpFormatter(argparse.RawDescriptionHelpFormatter): 5 | def __init__(self, prog: str, indent_increment: int = 2, max_help_position: int = 35, width: int = 80) -> None: 6 | super().__init__(prog, indent_increment, max_help_position, width) 7 | 8 | def _format_action_invocation(self, action: argparse.Action) -> str: 9 | """ 10 | Remove metavars from printing (w/o extra space before comma) 11 | and support tuple metavars for positional arguments 12 | """ 13 | _print_metavar = False 14 | 15 | if not action.option_strings: 16 | default = self._get_default_metavar_for_positional(action) 17 | metavar = self._metavar_formatter(action, default)(1) 18 | return ', '.join(metavar) 19 | 20 | parts: list = [] 21 | 22 | if action.nargs == 0: 23 | parts.extend(action.option_strings) 24 | else: 25 | default = self._get_default_metavar_for_optional(action) 26 | args_string = f' {self._format_args(action, default)}' if _print_metavar else '' 27 | for option_string in action.option_strings: 28 | parts.append(f'{option_string}{args_string}') 29 | 30 | return ', '.join(parts) 31 | 32 | def _get_default_metavar_for_optional(self, action: argparse.Action) -> str: 33 | return '' 34 | 35 | 36 | class PyroRawTextHelpFormatter(argparse.RawTextHelpFormatter, PyroRawDescriptionHelpFormatter): 37 | pass 38 | -------------------------------------------------------------------------------- /pyro/Remotes/BitbucketRemote.py: -------------------------------------------------------------------------------- 1 | import json 2 | import os 3 | import sys 4 | import urllib.error 5 | from http import HTTPStatus 6 | from typing import Generator 7 | from urllib.request import Request 8 | from urllib.request import urlopen 9 | 10 | from pyro.Comparators import endswith 11 | from pyro.Remotes.RemoteBase import RemoteBase 12 | 13 | 14 | class BitbucketRemote(RemoteBase): 15 | def _fetch_payloads(self, request_url: str) -> Generator: 16 | """ 17 | Recursively generates payloads from paginated responses 18 | """ 19 | request = Request(request_url) 20 | 21 | try: 22 | response = urlopen(request, timeout=30) 23 | except urllib.error.HTTPError as e: 24 | status: HTTPStatus = HTTPStatus(e.code) 25 | yield 'Failed to load remote: "%s" (%s %s)' % (request_url, e.code, status.phrase) 26 | sys.exit(1) 27 | 28 | if response.status != 200: 29 | status: HTTPStatus = HTTPStatus(response.status) # type: ignore 30 | yield 'Failed to load remote: "%s" (%s %s)' % (request_url, response.status, status.phrase) 31 | sys.exit(1) 32 | 33 | payload: dict = json.loads(response.read().decode('utf-8')) 34 | 35 | yield payload 36 | 37 | if 'next' in payload: 38 | yield from self._fetch_payloads(payload['next']) 39 | 40 | def fetch_contents(self, url: str, output_path: str) -> Generator: 41 | """ 42 | Downloads files from URL to output path 43 | """ 44 | request_url = self.extract_request_args(url) 45 | 46 | script_count: int = 0 47 | 48 | for payload in self._fetch_payloads(request_url.url): 49 | for payload_object in payload['values']: 50 | payload_object_type = payload_object['type'] 51 | 52 | target_path = os.path.normpath(os.path.join(output_path, request_url.owner, request_url.repo, payload_object['path'])) 53 | 54 | download_url = payload_object['links']['self']['href'] 55 | 56 | if payload_object_type == 'commit_file': 57 | # we only care about scripts 58 | if not endswith(download_url, '.psc', ignorecase=True): 59 | continue 60 | 61 | file_response = urlopen(download_url, timeout=30) 62 | 63 | if file_response.status != 200: 64 | yield f'Failed to download ({file_response.status}): "{download_url}"' 65 | continue 66 | 67 | os.makedirs(os.path.dirname(target_path), exist_ok=True) 68 | 69 | with open(target_path, mode='w+b') as f: 70 | f.write(file_response.read()) 71 | 72 | script_count += 1 73 | 74 | elif payload_object_type == 'commit_directory': 75 | yield from self.fetch_contents(download_url, output_path) 76 | 77 | if script_count > 0: 78 | yield f'Downloaded {script_count} scripts from "{request_url.url}"' 79 | -------------------------------------------------------------------------------- /pyro/Remotes/GenericRemote.py: -------------------------------------------------------------------------------- 1 | from typing import Generator 2 | from urllib.parse import urlparse 3 | 4 | from pyro.Comparators import endswith 5 | from pyro.Remotes.RemoteBase import RemoteBase 6 | from pyro.Remotes.BitbucketRemote import BitbucketRemote 7 | from pyro.Remotes.GiteaRemote import GiteaRemote 8 | from pyro.Remotes.GitHubRemote import GitHubRemote 9 | 10 | 11 | class GenericRemote(RemoteBase): 12 | def fetch_contents(self, url: str, output_path: str) -> Generator: 13 | """ 14 | Downloads files from URL to output path 15 | """ 16 | parsed_url = urlparse(url) 17 | 18 | schemeless_url = url.removeprefix(f'{parsed_url.scheme}://') # type: ignore 19 | 20 | if endswith(parsed_url.netloc, 'github.com', ignorecase=True): 21 | if self.config or not self.access_token: # type: ignore 22 | self.access_token = self.find_access_token(schemeless_url) 23 | if not self.access_token: 24 | raise PermissionError('Cannot download from GitHub remote without access token') 25 | github = GitHubRemote(access_token=self.access_token, 26 | worker_limit=self.worker_limit, 27 | force_overwrite=self.force_overwrite) 28 | yield from github.fetch_contents(url, output_path) 29 | elif endswith(parsed_url.netloc, 'bitbucket.org', ignorecase=True): 30 | bitbucket = BitbucketRemote() 31 | yield from bitbucket.fetch_contents(url, output_path) 32 | else: 33 | if self.config or not self.access_token: 34 | self.access_token = self.find_access_token(schemeless_url) 35 | if not self.access_token: 36 | raise PermissionError('Cannot download from Gitea remote without access token') 37 | gitea = GiteaRemote(access_token=self.access_token, 38 | worker_limit=self.worker_limit, 39 | force_overwrite=self.force_overwrite) 40 | yield from gitea.fetch_contents(url, output_path) 41 | -------------------------------------------------------------------------------- /pyro/Remotes/GitHubRemote.py: -------------------------------------------------------------------------------- 1 | import hashlib 2 | import json 3 | import multiprocessing 4 | import os 5 | import sys 6 | import urllib.error 7 | from http import HTTPStatus 8 | from typing import (Generator, 9 | Optional, 10 | Union) 11 | from urllib.request import (Request, 12 | urlopen) 13 | 14 | from pyro.Comparators import endswith 15 | from pyro.Remotes.RemoteBase import RemoteBase 16 | 17 | 18 | class GitHubRemote(RemoteBase): 19 | @staticmethod 20 | def download_file(file: tuple) -> Optional[str]: 21 | url, target_path = file 22 | 23 | file_response = urlopen(url, timeout=30) 24 | 25 | if file_response.status != 200: 26 | return f'Failed to download ({file_response.status}): "{url}"' 27 | 28 | os.makedirs(os.path.dirname(target_path), exist_ok=True) 29 | 30 | with open(target_path, mode='wb') as f: 31 | f.write(file_response.read()) 32 | 33 | return None 34 | 35 | def fetch_contents(self, url: str, output_path: str) -> Generator: 36 | """ 37 | Downloads files from URL to output path 38 | """ 39 | request_url = self.extract_request_args(url) 40 | 41 | request = Request(request_url.url) 42 | request.add_header('Authorization', f'token {self.access_token}') 43 | 44 | try: 45 | response = urlopen(request, timeout=30) 46 | except urllib.error.HTTPError as e: 47 | status: HTTPStatus = HTTPStatus(e.code) 48 | yield 'Failed to load remote: "%s" (%s %s)' % (request_url.url, e.code, status.phrase) 49 | sys.exit(1) 50 | 51 | if response.status != 200: 52 | status: HTTPStatus = HTTPStatus(response.status) # type: ignore 53 | yield 'Failed to load remote: "%s" (%s %s)' % (request_url.url, response.status, status.phrase) 54 | sys.exit(1) 55 | 56 | payload_objects: Union[dict, list] = json.loads(response.read().decode('utf-8')) 57 | 58 | if 'contents_url' in payload_objects: 59 | branch = payload_objects['default_branch'] # type: ignore 60 | contents_url = payload_objects['contents_url'].replace('{+path}', f'?ref={branch}') # type: ignore 61 | yield from self.fetch_contents(contents_url, output_path) 62 | return 63 | 64 | scripts: list = [] 65 | 66 | for payload_object in payload_objects: 67 | target_path = os.path.normpath(os.path.join(output_path, request_url.owner, request_url.repo, payload_object['path'])) 68 | 69 | if not self.force_overwrite and os.path.isfile(target_path): 70 | with open(target_path, mode='rb') as f: 71 | data = f.read() 72 | sha1 = hashlib.sha1(b'blob %s\x00%s' % (len(data), data.decode())) # type: ignore 73 | 74 | if sha1.hexdigest() == payload_object['sha']: 75 | continue 76 | 77 | download_url = payload_object['download_url'] 78 | 79 | # handle folders 80 | if not download_url: 81 | yield from self.fetch_contents(payload_object['url'], output_path) 82 | continue 83 | 84 | # we only care about scripts and flags files 85 | if not (payload_object['type'] == 'file' and endswith(payload_object['name'], ('.flg', '.psc'), ignorecase=True)): 86 | continue 87 | 88 | scripts.append((download_url, target_path)) 89 | 90 | script_count: int = len(scripts) 91 | 92 | if script_count == 0: 93 | return 94 | 95 | multiprocessing.freeze_support() 96 | worker_limit: int = min(script_count, self.worker_limit) 97 | with multiprocessing.Pool(processes=worker_limit) as pool: 98 | for download_result in pool.imap_unordered(self.download_file, scripts): 99 | yield download_result 100 | pool.close() 101 | pool.join() 102 | 103 | if script_count > 0: 104 | yield f'Downloaded {script_count} scripts from "{request_url.url}"' 105 | -------------------------------------------------------------------------------- /pyro/Remotes/GiteaRemote.py: -------------------------------------------------------------------------------- 1 | import hashlib 2 | import json 3 | import multiprocessing 4 | import os 5 | import sys 6 | import urllib.error 7 | from http import HTTPStatus 8 | from typing import (Generator, 9 | Optional, 10 | Union) 11 | from urllib.request import (Request, 12 | urlopen) 13 | 14 | from pyro.Comparators import endswith 15 | from pyro.Remotes.RemoteBase import RemoteBase 16 | 17 | 18 | class GiteaRemote(RemoteBase): 19 | @staticmethod 20 | def download_file(file: tuple) -> Optional[str]: 21 | url, target_path = file 22 | 23 | file_response = urlopen(url, timeout=30) 24 | 25 | if file_response.status != 200: 26 | return f'Failed to download ({file_response.status}): "{url}"' 27 | 28 | os.makedirs(os.path.dirname(target_path), exist_ok=True) 29 | 30 | with open(target_path, mode='wb') as f: 31 | f.write(file_response.read()) 32 | 33 | return None 34 | 35 | def fetch_contents(self, url: str, output_path: str) -> Generator: 36 | """ 37 | Downloads files from URL to output path 38 | """ 39 | request_url = self.extract_request_args(url) 40 | 41 | request = Request(request_url.url) 42 | request.add_header('Authorization', f'token {self.access_token}') 43 | 44 | try: 45 | response = urlopen(request, timeout=30) 46 | except urllib.error.HTTPError as e: 47 | status: HTTPStatus = HTTPStatus(e.code) 48 | yield 'Failed to load remote: "%s" (%s %s)' % (request_url.url, e.code, status.phrase) 49 | sys.exit(1) 50 | 51 | if response.status != 200: 52 | status: HTTPStatus = HTTPStatus(response.status) # type: ignore 53 | yield 'Failed to load remote: "%s" (%s %s)' % (request_url.url, response.status, status.phrase) 54 | sys.exit(1) 55 | 56 | payload_objects: Union[dict, list] = json.loads(response.read().decode('utf-8')) 57 | 58 | scripts: list = [] 59 | 60 | for payload_object in payload_objects: 61 | target_path = os.path.normpath(os.path.join(output_path, request_url.owner, request_url.repo, payload_object['path'])) 62 | 63 | if not self.force_overwrite and os.path.isfile(target_path): 64 | with open(target_path, mode='rb') as f: 65 | data = f.read() 66 | sha1 = hashlib.sha1(b'blob %s\x00%s' % (len(data), data.decode())) # type: ignore 67 | 68 | if sha1.hexdigest() == payload_object['sha']: 69 | continue 70 | 71 | download_url = payload_object['download_url'] 72 | 73 | # handle folders 74 | if not download_url: 75 | yield from self.fetch_contents(payload_object['url'], output_path) 76 | continue 77 | 78 | # we only care about scripts and flags files 79 | if not (payload_object['type'] == 'file' and endswith(payload_object['name'], ('.flg', '.psc'), ignorecase=True)): 80 | continue 81 | 82 | scripts.append((download_url, target_path)) 83 | 84 | script_count: int = len(scripts) 85 | 86 | if script_count == 0: 87 | return 88 | 89 | multiprocessing.freeze_support() 90 | worker_limit: int = min(script_count, self.worker_limit) 91 | with multiprocessing.Pool(processes=worker_limit) as pool: 92 | for download_result in pool.imap_unordered(self.download_file, scripts): 93 | yield download_result 94 | pool.close() 95 | pool.join() 96 | 97 | if script_count > 0: 98 | yield f'Downloaded {script_count} scripts from "{request_url.url}"' 99 | -------------------------------------------------------------------------------- /pyro/Remotes/RemoteBase.py: -------------------------------------------------------------------------------- 1 | import configparser 2 | import os 3 | from typing import Generator, Optional 4 | from urllib.error import HTTPError 5 | from urllib.parse import urlparse 6 | 7 | from pyro.Comparators import startswith 8 | from pyro.Remotes.RemoteUri import RemoteUri 9 | 10 | 11 | class RemoteBase: 12 | access_token: str = '' 13 | 14 | url_patterns: dict = { 15 | 'api.github.com': (3, 0), # pop 'contents', 'repos' 16 | 'github.com': (3, 2), # pop branch, 'tree' 17 | 'api.bitbucket.org': (5, 4, 1, 0), # pop branch, 'src', 'repositories', '2.0' 18 | 'bitbucket.org': (3, 2) # pop branch, 'src' 19 | } 20 | 21 | def __init__(self, *, config: configparser.ConfigParser = None, access_token: str = '', worker_limit: int = -1, force_overwrite: bool = False) -> None: 22 | self.config = config 23 | self.access_token = access_token 24 | self.worker_limit = worker_limit 25 | self.force_overwrite = force_overwrite 26 | 27 | def create_local_path(self, url: str) -> str: 28 | """ 29 | Creates relative local path from URL 30 | """ 31 | parsed_url = urlparse(url) 32 | 33 | netloc = parsed_url.netloc 34 | 35 | if netloc in self.url_patterns: 36 | url_path_parts = parsed_url.path.split('/') 37 | url_path_parts.pop(0) if not url_path_parts[0] else None # pop empty space 38 | 39 | if netloc == 'github.com' and len(url_path_parts) == 2: 40 | return os.sep.join(url_path_parts) 41 | 42 | for i in self.url_patterns[netloc]: 43 | url_path_parts.pop(i) 44 | 45 | url_path = os.sep.join(url_path_parts) 46 | else: 47 | url_path_parts = parsed_url.path.split('/') 48 | url_path_parts.pop(0) if not url_path_parts[0] else None # pop empty space 49 | url_path = os.sep.join(url_path_parts) 50 | 51 | return url_path 52 | 53 | @staticmethod 54 | def extract_request_args(url: str) -> RemoteUri: 55 | """ 56 | Extracts (owner, repo, request_url) from URL 57 | """ 58 | result = RemoteUri() 59 | 60 | data = urlparse(url) 61 | 62 | # remove '.git' from clone url, if suffix exists 63 | result.data = data._asdict() 64 | result.data['path'] = result.data['path'].replace('.git', '') 65 | 66 | netloc, path, query = result.data['netloc'], result.data['path'], result.data['query'] 67 | 68 | # store branch if specified 69 | query_params = query.split('&') 70 | for param in query_params: 71 | if startswith(param, 'ref', ignorecase=True): 72 | _, branch = param.split('=') 73 | result.branch = branch 74 | break 75 | 76 | url_path_parts = path.split('/') 77 | url_path_parts.pop(0) if not url_path_parts[0] else None # pop empty space 78 | 79 | request_url = '' 80 | 81 | if netloc == 'github.com': 82 | if len(url_path_parts) == 2: 83 | request_url = f'https://api.{netloc}/repos{path}' 84 | elif 'tree/' in path: # example: /fireundubh/LibFire/tree/master 85 | result.branch = url_path_parts.pop(3) # pop 'master' (or any other branch) 86 | url_path_parts.pop(2) if url_path_parts[2] == 'tree' else None # pop 'tree' 87 | url_path_parts.insert(2, 'contents') 88 | url_path = '/'.join(url_path_parts) 89 | request_url = f'https://api.{netloc}/repos/{url_path}?ref={result.branch}' 90 | elif netloc == 'api.github.com': 91 | if startswith(path, '/repos'): 92 | url_path_parts.pop() if not url_path_parts[len(url_path_parts) - 1] else None # pop empty space 93 | url_path_parts.pop() if url_path_parts[len(url_path_parts) - 1] == 'contents' else None # pop 'contents' 94 | url_path_parts.pop(0) # pop 'repos' 95 | request_url = url 96 | elif netloc == 'bitbucket.org': 97 | request_url = 'https://api.bitbucket.org/2.0/repositories{}{}'.format(path, query) 98 | elif netloc == 'api.bitbucket.org': 99 | request_url = url 100 | else: 101 | if 'contents' not in path: 102 | q = f'?ref={result.branch}' if result.branch else '' 103 | request_url = f'https://{netloc}/api/v1/repos{path}/contents{q}' 104 | else: 105 | request_url = url 106 | 107 | if 'api/v1' not in path: 108 | result.owner = url_path_parts[0] 109 | result.repo = url_path_parts[1] 110 | else: 111 | result.owner = url_path_parts[3] 112 | result.repo = url_path_parts[4] 113 | 114 | result.url = request_url 115 | 116 | return result 117 | 118 | @staticmethod 119 | def validate_url(url: str) -> bool: 120 | """ 121 | Tests whether URL has scheme, netloc, and path 122 | """ 123 | try: 124 | result = urlparse(url) 125 | except HTTPError: 126 | return False 127 | else: 128 | return all([result.scheme, result.netloc, result.path]) 129 | 130 | def find_access_token(self, schemeless_url: str) -> Optional[str]: 131 | if self.config: 132 | for section_url in self.config.sections(): 133 | if section_url.casefold() in schemeless_url.casefold(): 134 | token = self.config.get(section_url, 'access_token', fallback=None) 135 | if token: 136 | return os.path.expanduser(os.path.expandvars(token)) 137 | return None 138 | 139 | def fetch_contents(self, url: str, output_path: str) -> Generator: 140 | """ 141 | Downloads files from URL to output path 142 | """ 143 | pass 144 | -------------------------------------------------------------------------------- /pyro/Remotes/RemoteUri.py: -------------------------------------------------------------------------------- 1 | from dataclasses import dataclass, field 2 | 3 | 4 | @dataclass 5 | class RemoteUri: 6 | owner: str = field(init=False, default_factory=str) 7 | repo: str = field(init=False, default_factory=str) 8 | branch: str = field(init=False, default_factory=str) 9 | data: dict = field(init=False, default=dict) # type: ignore 10 | url: str = field(init=False, default_factory=str) 11 | -------------------------------------------------------------------------------- /pyro/StringTemplate.py: -------------------------------------------------------------------------------- 1 | import re as _re 2 | from string import Template 3 | 4 | 5 | class StringTemplate(Template): 6 | # noinspection PyClassVar 7 | delimiter: str = '@' # type: ignore 8 | idpattern = '([_a-z][_a-z0-9]*)' 9 | flags = _re.IGNORECASE | _re.ASCII 10 | -------------------------------------------------------------------------------- /pyro/TimeElapsed.py: -------------------------------------------------------------------------------- 1 | from decimal import Context, Decimal, ROUND_DOWN 2 | 3 | 4 | class TimeElapsed: 5 | start_time: float = 0.0 6 | end_time: float = 0.0 7 | 8 | def __init__(self) -> None: 9 | self._context = Context(prec=4, rounding=ROUND_DOWN) 10 | self.start_time = 0.0 11 | self.end_time = 0.0 12 | 13 | def average(self, dividend: int) -> int: 14 | if dividend == 0: 15 | return round(Decimal(0), 8) 16 | value = self.value() 17 | if value.compare(0) == 0: 18 | return round(Decimal(0), 8) 19 | return round(value / Decimal(dividend, self._context), 8) 20 | 21 | def value(self) -> Decimal: 22 | return Decimal(self.end_time) - Decimal(self.start_time) 23 | -------------------------------------------------------------------------------- /pyro/XmlHelper.py: -------------------------------------------------------------------------------- 1 | import io 2 | import os 3 | import re 4 | import typing 5 | 6 | from lxml import etree 7 | 8 | 9 | class XmlHelper: 10 | @staticmethod 11 | def strip_xml_comments(path: str) -> io.StringIO: 12 | with open(path, encoding='utf-8') as f: 13 | xml_document: str = f.read() 14 | comments_pattern = re.compile('()', flags=re.DOTALL) 15 | xml_document = comments_pattern.sub('', xml_document) 16 | return io.StringIO(xml_document) 17 | 18 | @staticmethod 19 | def validate_schema(namespace: str, program_path: str) -> typing.Optional[etree.XMLSchema]: 20 | if not namespace: 21 | return None 22 | 23 | schema_path = os.path.join(program_path, namespace) 24 | if not os.path.isfile(schema_path): 25 | raise FileExistsError(f'Schema file does not exist: "{schema_path}"') 26 | 27 | schema = etree.parse(schema_path) 28 | return etree.XMLSchema(schema) 29 | -------------------------------------------------------------------------------- /pyro/XmlRoot.py: -------------------------------------------------------------------------------- 1 | from typing import Any 2 | 3 | from lxml import etree 4 | 5 | 6 | class XmlRoot: 7 | node: etree.ElementBase = None 8 | ns: str = '' 9 | 10 | def __init__(self, element_tree: etree.ElementTree) -> None: 11 | self.node = element_tree.getroot() 12 | 13 | nsmap, prefix = self.node.nsmap, self.node.prefix 14 | self.ns = nsmap[prefix] if prefix in nsmap else '' 15 | 16 | def find(self, key: str) -> etree.ElementBase: 17 | path = key if not self.ns else f'ns:{key}', {'ns': self.ns} 18 | return self.node.find(*path) 19 | 20 | def get(self, key: str, default: Any = None) -> Any: # type: ignore 21 | return self.node.get(key, default) 22 | -------------------------------------------------------------------------------- /pyro/__main__.py: -------------------------------------------------------------------------------- 1 | import os 2 | import sys 3 | from argparse import SUPPRESS 4 | 5 | from pyro.Application import Application 6 | from pyro.PyroArgumentParser import PyroArgumentParser 7 | from pyro.PyroRawDescriptionHelpFormatter import PyroRawTextHelpFormatter 8 | 9 | if __name__ == '__main__': 10 | # noinspection PyTypeChecker 11 | _parser = PyroArgumentParser(add_help=False, 12 | formatter_class=PyroRawTextHelpFormatter, 13 | description=os.linesep.join([ 14 | '-' * 80, 15 | ''.join(c.center(3) for c in 'PYRO').center(80), 16 | '-' * 53 + ' github.com/fireundubh/pyro' 17 | ])) 18 | 19 | _required_arguments = _parser.add_argument_group('required arguments') 20 | _required_arguments_flex = _required_arguments.add_mutually_exclusive_group() 21 | _required_arguments_flex.add_argument('input_path', nargs='?', 22 | action='store', type=str, 23 | help='relative or absolute path to file\n' 24 | '(if relative, must be relative to current working directory)') 25 | # deprecated argument format (retained to avoid breaking change) 26 | _required_arguments_flex.add_argument('-i', '--input-path', dest='input_path_deprecated', 27 | action='store', type=str, 28 | help=SUPPRESS) 29 | 30 | _build_arguments = _parser.add_argument_group('build arguments') 31 | _build_arguments.add_argument('--ignore-errors', 32 | action='store_true', default=False, 33 | help='ignore compiler errors during build') 34 | # deprecated argument (functionality no longer exists, retained arg to avoid breaking change) 35 | _build_arguments.add_argument('--no-implicit-imports', dest='no_implicit_imports_deprecated', 36 | action='store_true', default=False, 37 | help=SUPPRESS) 38 | _build_arguments.add_argument('--no-incremental-build', 39 | action='store_true', default=False, 40 | help='do not build incrementally') 41 | _build_arguments.add_argument('--no-parallel', 42 | action='store_true', default=False, 43 | help='do not parallelize compilation') 44 | _build_arguments.add_argument('--worker-limit', 45 | action='store', type=int, 46 | help='max workers for parallel compilation\n' 47 | '(usually set automatically to processor count)') 48 | 49 | _compiler_arguments = _parser.add_argument_group('compiler arguments') 50 | _compiler_arguments.add_argument('--compiler-path', 51 | action='store', type=str, 52 | help='relative or absolute path to PapyrusCompiler.exe\n' 53 | '(if relative, must be relative to current working directory)') 54 | _compiler_arguments.add_argument('--compiler-config-path', 55 | action='store', type=str, 56 | help='relative or absolute path to compiler config file\n' 57 | '(if relative, must be relative to current working directory)') 58 | _compiler_arguments.add_argument('--flags-path', 59 | action='store', type=str, 60 | help='relative or absolute path to Papyrus Flags file\n' 61 | '(if relative, must be relative to project)') 62 | _compiler_arguments.add_argument('--output-path', 63 | action='store', type=str, 64 | help='relative or absolute path to output folder\n' 65 | '(if relative, must be relative to project)') 66 | 67 | _game_arguments = _parser.add_argument_group('game arguments') 68 | _game_arguments.add_argument('-g', '--game-type', 69 | action='store', type=str, 70 | choices=('fo4', 'tes5', 'sf1', 'sse'), 71 | help='set game type (choices: fo4, tes5, sf1, sse)') 72 | 73 | _game_path_arguments = _game_arguments.add_mutually_exclusive_group() 74 | _game_path_arguments.add_argument('--game-path', 75 | action='store', type=str, 76 | help='relative or absolute path to game install directory\n' 77 | '(if relative, must be relative to current working directory)') 78 | if sys.platform == 'win32': 79 | _game_path_arguments.add_argument('--registry-path', 80 | action='store', type=str, 81 | help='path to Installed Path key in Windows Registry') 82 | 83 | _bsarch_arguments = _parser.add_argument_group('bsarch arguments') 84 | _bsarch_arguments.add_argument('--bsarch-path', 85 | action='store', type=str, 86 | help='relative or absolute path to bsarch.exe\n' 87 | '(if relative, must be relative to current working directory)') 88 | _bsarch_arguments.add_argument('--package-path', 89 | action='store', type=str, 90 | help='relative or absolute path to bsa/ba2 output folder\n' 91 | '(if relative, must be relative to project)') 92 | _bsarch_arguments.add_argument('--temp-path', 93 | action='store', type=str, 94 | help='relative or absolute path to temp folder\n' 95 | '(if relative, must be relative to current working directory)') 96 | 97 | _zip_arguments = _parser.add_argument_group('zip arguments') 98 | _zip_arguments.add_argument('--zip-compression', 99 | action='store', type=str, 100 | choices=('store', 'deflate'), 101 | help='set compression method (choices: store, deflate)') 102 | _zip_arguments.add_argument('--zip-output-path', 103 | action='store', type=str, 104 | help='relative or absolute path to zip output folder\n' 105 | '(if relative, must be relative to project)') 106 | 107 | _remote_arguments = _parser.add_argument_group('remote arguments') 108 | _remote_arguments.add_argument('--access-token', 109 | action='store', type=str, 110 | help='access token for GitHub remotes\n(must have public_repo access scope)') 111 | _remote_arguments.add_argument('--force-overwrite', 112 | action='store_true', 113 | help='download remote files and overwrite existing files\n' 114 | '(default: do not redownload unchanged files)') 115 | _remote_arguments.add_argument('--remote-temp-path', 116 | action='store', type=str, 117 | help='relative or absolute path to temp folder for remote files\n' 118 | '(if relative, must be relative to project)') 119 | 120 | _project_arguments = _parser.add_argument_group('project arguments') 121 | _project_arguments.add_argument('--create-project', 122 | action='store_true', 123 | help='generate project from current directory') 124 | _project_arguments.add_argument('--resolve-project', 125 | action='store_true', 126 | help='resolve variables and paths in project file') 127 | 128 | _program_arguments = _parser.add_argument_group('program arguments') 129 | _program_arguments.add_argument('--log-level', dest='log_level', 130 | action='store', type=str, default='debug', 131 | choices=('all', 'debug', 'info', 'warn', 'error', 'fatal'), 132 | help='show help and exit') 133 | _program_arguments.add_argument('--help', dest='show_help', 134 | action='store_true', default=False, 135 | help='show help and exit') 136 | 137 | Application(_parser).run() 138 | -------------------------------------------------------------------------------- /pyro/setup.py: -------------------------------------------------------------------------------- 1 | from setuptools import find_packages, setup 2 | 3 | setup( 4 | name='Pyro', 5 | description='A parallelized incremental build system for TES5, SSE, and FO4 projects', 6 | author='fireundubh', 7 | author_email='fireundubh@gmail.com', 8 | license='MIT License', 9 | packages=find_packages(), 10 | zip_safe=False, 11 | classifiers=[ 12 | 'Programming Language :: Python :: 3', 13 | 'License :: OSI Approved :: MIT License', 14 | 'Operating System :: OS Independent', 15 | ], 16 | ) 17 | -------------------------------------------------------------------------------- /requirements-dev.txt: -------------------------------------------------------------------------------- 1 | mypy 2 | pylint 3 | -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | lxml>=5.2.2 2 | nuitka>=2.3.10 3 | psutil==5.9.3 4 | wcmatch>=8.5.2 5 | ordered-set>=4.1.0 6 | -------------------------------------------------------------------------------- /tools/bsarch.exe: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fireundubh/pyro/82a6faf6f6a79e40a5fbf705e03d6c655cff40b7/tools/bsarch.exe -------------------------------------------------------------------------------- /tools/bsarch.license.txt: -------------------------------------------------------------------------------- 1 | Mozilla Public License Version 2.0 2 | ================================== 3 | 4 | 1. Definitions 5 | -------------- 6 | 7 | 1.1. "Contributor" 8 | means each individual or legal entity that creates, contributes to 9 | the creation of, or owns Covered Software. 10 | 11 | 1.2. "Contributor Version" 12 | means the combination of the Contributions of others (if any) used 13 | by a Contributor and that particular Contributor's Contribution. 14 | 15 | 1.3. "Contribution" 16 | means Covered Software of a particular Contributor. 17 | 18 | 1.4. "Covered Software" 19 | means Source Code Form to which the initial Contributor has attached 20 | the notice in Exhibit A, the Executable Form of such Source Code 21 | Form, and Modifications of such Source Code Form, in each case 22 | including portions thereof. 23 | 24 | 1.5. "Incompatible With Secondary Licenses" 25 | means 26 | 27 | (a) that the initial Contributor has attached the notice described 28 | in Exhibit B to the Covered Software; or 29 | 30 | (b) that the Covered Software was made available under the terms of 31 | version 1.1 or earlier of the License, but not also under the 32 | terms of a Secondary License. 33 | 34 | 1.6. "Executable Form" 35 | means any form of the work other than Source Code Form. 36 | 37 | 1.7. "Larger Work" 38 | means a work that combines Covered Software with other material, in 39 | a separate file or files, that is not Covered Software. 40 | 41 | 1.8. "License" 42 | means this document. 43 | 44 | 1.9. "Licensable" 45 | means having the right to grant, to the maximum extent possible, 46 | whether at the time of the initial grant or subsequently, any and 47 | all of the rights conveyed by this License. 48 | 49 | 1.10. "Modifications" 50 | means any of the following: 51 | 52 | (a) any file in Source Code Form that results from an addition to, 53 | deletion from, or modification of the contents of Covered 54 | Software; or 55 | 56 | (b) any new file in Source Code Form that contains any Covered 57 | Software. 58 | 59 | 1.11. "Patent Claims" of a Contributor 60 | means any patent claim(s), including without limitation, method, 61 | process, and apparatus claims, in any patent Licensable by such 62 | Contributor that would be infringed, but for the grant of the 63 | License, by the making, using, selling, offering for sale, having 64 | made, import, or transfer of either its Contributions or its 65 | Contributor Version. 66 | 67 | 1.12. "Secondary License" 68 | means either the GNU General Public License, Version 2.0, the GNU 69 | Lesser General Public License, Version 2.1, the GNU Affero General 70 | Public License, Version 3.0, or any later versions of those 71 | licenses. 72 | 73 | 1.13. "Source Code Form" 74 | means the form of the work preferred for making modifications. 75 | 76 | 1.14. "You" (or "Your") 77 | means an individual or a legal entity exercising rights under this 78 | License. For legal entities, "You" includes any entity that 79 | controls, is controlled by, or is under common control with You. For 80 | purposes of this definition, "control" means (a) the power, direct 81 | or indirect, to cause the direction or management of such entity, 82 | whether by contract or otherwise, or (b) ownership of more than 83 | fifty percent (50%) of the outstanding shares or beneficial 84 | ownership of such entity. 85 | 86 | 2. License Grants and Conditions 87 | -------------------------------- 88 | 89 | 2.1. Grants 90 | 91 | Each Contributor hereby grants You a world-wide, royalty-free, 92 | non-exclusive license: 93 | 94 | (a) under intellectual property rights (other than patent or trademark) 95 | Licensable by such Contributor to use, reproduce, make available, 96 | modify, display, perform, distribute, and otherwise exploit its 97 | Contributions, either on an unmodified basis, with Modifications, or 98 | as part of a Larger Work; and 99 | 100 | (b) under Patent Claims of such Contributor to make, use, sell, offer 101 | for sale, have made, import, and otherwise transfer either its 102 | Contributions or its Contributor Version. 103 | 104 | 2.2. Effective Date 105 | 106 | The licenses granted in Section 2.1 with respect to any Contribution 107 | become effective for each Contribution on the date the Contributor first 108 | distributes such Contribution. 109 | 110 | 2.3. Limitations on Grant Scope 111 | 112 | The licenses granted in this Section 2 are the only rights granted under 113 | this License. No additional rights or licenses will be implied from the 114 | distribution or licensing of Covered Software under this License. 115 | Notwithstanding Section 2.1(b) above, no patent license is granted by a 116 | Contributor: 117 | 118 | (a) for any code that a Contributor has removed from Covered Software; 119 | or 120 | 121 | (b) for infringements caused by: (i) Your and any other third party's 122 | modifications of Covered Software, or (ii) the combination of its 123 | Contributions with other software (except as part of its Contributor 124 | Version); or 125 | 126 | (c) under Patent Claims infringed by Covered Software in the absence of 127 | its Contributions. 128 | 129 | This License does not grant any rights in the trademarks, service marks, 130 | or logos of any Contributor (except as may be necessary to comply with 131 | the notice requirements in Section 3.4). 132 | 133 | 2.4. Subsequent Licenses 134 | 135 | No Contributor makes additional grants as a result of Your choice to 136 | distribute the Covered Software under a subsequent version of this 137 | License (see Section 10.2) or under the terms of a Secondary License (if 138 | permitted under the terms of Section 3.3). 139 | 140 | 2.5. Representation 141 | 142 | Each Contributor represents that the Contributor believes its 143 | Contributions are its original creation(s) or it has sufficient rights 144 | to grant the rights to its Contributions conveyed by this License. 145 | 146 | 2.6. Fair Use 147 | 148 | This License is not intended to limit any rights You have under 149 | applicable copyright doctrines of fair use, fair dealing, or other 150 | equivalents. 151 | 152 | 2.7. Conditions 153 | 154 | Sections 3.1, 3.2, 3.3, and 3.4 are conditions of the licenses granted 155 | in Section 2.1. 156 | 157 | 3. Responsibilities 158 | ------------------- 159 | 160 | 3.1. Distribution of Source Form 161 | 162 | All distribution of Covered Software in Source Code Form, including any 163 | Modifications that You create or to which You contribute, must be under 164 | the terms of this License. You must inform recipients that the Source 165 | Code Form of the Covered Software is governed by the terms of this 166 | License, and how they can obtain a copy of this License. You may not 167 | attempt to alter or restrict the recipients' rights in the Source Code 168 | Form. 169 | 170 | 3.2. Distribution of Executable Form 171 | 172 | If You distribute Covered Software in Executable Form then: 173 | 174 | (a) such Covered Software must also be made available in Source Code 175 | Form, as described in Section 3.1, and You must inform recipients of 176 | the Executable Form how they can obtain a copy of such Source Code 177 | Form by reasonable means in a timely manner, at a charge no more 178 | than the cost of distribution to the recipient; and 179 | 180 | (b) You may distribute such Executable Form under the terms of this 181 | License, or sublicense it under different terms, provided that the 182 | license for the Executable Form does not attempt to limit or alter 183 | the recipients' rights in the Source Code Form under this License. 184 | 185 | 3.3. Distribution of a Larger Work 186 | 187 | You may create and distribute a Larger Work under terms of Your choice, 188 | provided that You also comply with the requirements of this License for 189 | the Covered Software. If the Larger Work is a combination of Covered 190 | Software with a work governed by one or more Secondary Licenses, and the 191 | Covered Software is not Incompatible With Secondary Licenses, this 192 | License permits You to additionally distribute such Covered Software 193 | under the terms of such Secondary License(s), so that the recipient of 194 | the Larger Work may, at their option, further distribute the Covered 195 | Software under the terms of either this License or such Secondary 196 | License(s). 197 | 198 | 3.4. Notices 199 | 200 | You may not remove or alter the substance of any license notices 201 | (including copyright notices, patent notices, disclaimers of warranty, 202 | or limitations of liability) contained within the Source Code Form of 203 | the Covered Software, except that You may alter any license notices to 204 | the extent required to remedy known factual inaccuracies. 205 | 206 | 3.5. Application of Additional Terms 207 | 208 | You may choose to offer, and to charge a fee for, warranty, support, 209 | indemnity or liability obligations to one or more recipients of Covered 210 | Software. However, You may do so only on Your own behalf, and not on 211 | behalf of any Contributor. You must make it absolutely clear that any 212 | such warranty, support, indemnity, or liability obligation is offered by 213 | You alone, and You hereby agree to indemnify every Contributor for any 214 | liability incurred by such Contributor as a result of warranty, support, 215 | indemnity or liability terms You offer. You may include additional 216 | disclaimers of warranty and limitations of liability specific to any 217 | jurisdiction. 218 | 219 | 4. Inability to Comply Due to Statute or Regulation 220 | --------------------------------------------------- 221 | 222 | If it is impossible for You to comply with any of the terms of this 223 | License with respect to some or all of the Covered Software due to 224 | statute, judicial order, or regulation then You must: (a) comply with 225 | the terms of this License to the maximum extent possible; and (b) 226 | describe the limitations and the code they affect. Such description must 227 | be placed in a text file included with all distributions of the Covered 228 | Software under this License. Except to the extent prohibited by statute 229 | or regulation, such description must be sufficiently detailed for a 230 | recipient of ordinary skill to be able to understand it. 231 | 232 | 5. Termination 233 | -------------- 234 | 235 | 5.1. The rights granted under this License will terminate automatically 236 | if You fail to comply with any of its terms. However, if You become 237 | compliant, then the rights granted under this License from a particular 238 | Contributor are reinstated (a) provisionally, unless and until such 239 | Contributor explicitly and finally terminates Your grants, and (b) on an 240 | ongoing basis, if such Contributor fails to notify You of the 241 | non-compliance by some reasonable means prior to 60 days after You have 242 | come back into compliance. Moreover, Your grants from a particular 243 | Contributor are reinstated on an ongoing basis if such Contributor 244 | notifies You of the non-compliance by some reasonable means, this is the 245 | first time You have received notice of non-compliance with this License 246 | from such Contributor, and You become compliant prior to 30 days after 247 | Your receipt of the notice. 248 | 249 | 5.2. If You initiate litigation against any entity by asserting a patent 250 | infringement claim (excluding declaratory judgment actions, 251 | counter-claims, and cross-claims) alleging that a Contributor Version 252 | directly or indirectly infringes any patent, then the rights granted to 253 | You by any and all Contributors for the Covered Software under Section 254 | 2.1 of this License shall terminate. 255 | 256 | 5.3. In the event of termination under Sections 5.1 or 5.2 above, all 257 | end user license agreements (excluding distributors and resellers) which 258 | have been validly granted by You or Your distributors under this License 259 | prior to termination shall survive termination. 260 | 261 | ************************************************************************ 262 | * * 263 | * 6. Disclaimer of Warranty * 264 | * ------------------------- * 265 | * * 266 | * Covered Software is provided under this License on an "as is" * 267 | * basis, without warranty of any kind, either expressed, implied, or * 268 | * statutory, including, without limitation, warranties that the * 269 | * Covered Software is free of defects, merchantable, fit for a * 270 | * particular purpose or non-infringing. The entire risk as to the * 271 | * quality and performance of the Covered Software is with You. * 272 | * Should any Covered Software prove defective in any respect, You * 273 | * (not any Contributor) assume the cost of any necessary servicing, * 274 | * repair, or correction. This disclaimer of warranty constitutes an * 275 | * essential part of this License. No use of any Covered Software is * 276 | * authorized under this License except under this disclaimer. * 277 | * * 278 | ************************************************************************ 279 | 280 | ************************************************************************ 281 | * * 282 | * 7. Limitation of Liability * 283 | * -------------------------- * 284 | * * 285 | * Under no circumstances and under no legal theory, whether tort * 286 | * (including negligence), contract, or otherwise, shall any * 287 | * Contributor, or anyone who distributes Covered Software as * 288 | * permitted above, be liable to You for any direct, indirect, * 289 | * special, incidental, or consequential damages of any character * 290 | * including, without limitation, damages for lost profits, loss of * 291 | * goodwill, work stoppage, computer failure or malfunction, or any * 292 | * and all other commercial damages or losses, even if such party * 293 | * shall have been informed of the possibility of such damages. This * 294 | * limitation of liability shall not apply to liability for death or * 295 | * personal injury resulting from such party's negligence to the * 296 | * extent applicable law prohibits such limitation. Some * 297 | * jurisdictions do not allow the exclusion or limitation of * 298 | * incidental or consequential damages, so this exclusion and * 299 | * limitation may not apply to You. * 300 | * * 301 | ************************************************************************ 302 | 303 | 8. Litigation 304 | ------------- 305 | 306 | Any litigation relating to this License may be brought only in the 307 | courts of a jurisdiction where the defendant maintains its principal 308 | place of business and such litigation shall be governed by laws of that 309 | jurisdiction, without reference to its conflict-of-law provisions. 310 | Nothing in this Section shall prevent a party's ability to bring 311 | cross-claims or counter-claims. 312 | 313 | 9. Miscellaneous 314 | ---------------- 315 | 316 | This License represents the complete agreement concerning the subject 317 | matter hereof. If any provision of this License is held to be 318 | unenforceable, such provision shall be reformed only to the extent 319 | necessary to make it enforceable. Any law or regulation which provides 320 | that the language of a contract shall be construed against the drafter 321 | shall not be used to construe this License against a Contributor. 322 | 323 | 10. Versions of the License 324 | --------------------------- 325 | 326 | 10.1. New Versions 327 | 328 | Mozilla Foundation is the license steward. Except as provided in Section 329 | 10.3, no one other than the license steward has the right to modify or 330 | publish new versions of this License. Each version will be given a 331 | distinguishing version number. 332 | 333 | 10.2. Effect of New Versions 334 | 335 | You may distribute the Covered Software under the terms of the version 336 | of the License under which You originally received the Covered Software, 337 | or under the terms of any subsequent version published by the license 338 | steward. 339 | 340 | 10.3. Modified Versions 341 | 342 | If you create software not governed by this License, and you want to 343 | create a new license for such software, you may create and use a 344 | modified version of this License if you rename the license and remove 345 | any references to the name of the license steward (except to note that 346 | such modified license differs from this License). 347 | 348 | 10.4. Distributing Source Code Form that is Incompatible With Secondary 349 | Licenses 350 | 351 | If You choose to distribute Source Code Form that is Incompatible With 352 | Secondary Licenses under the terms of this version of the License, the 353 | notice described in Exhibit B of this License must be attached. 354 | 355 | Exhibit A - Source Code Form License Notice 356 | ------------------------------------------- 357 | 358 | This Source Code Form is subject to the terms of the Mozilla Public 359 | License, v. 2.0. If a copy of the MPL was not distributed with this 360 | file, You can obtain one at http://mozilla.org/MPL/2.0/. 361 | 362 | If it is not possible or desirable to put the notice in a particular 363 | file, then You may include the notice in a location (such as a LICENSE 364 | file in a relevant directory) where a recipient would be likely to look 365 | for such a notice. 366 | 367 | You may add additional accurate notices of copyright ownership. 368 | 369 | Exhibit B - "Incompatible With Secondary Licenses" Notice 370 | --------------------------------------------------------- 371 | 372 | This Source Code Form is "Incompatible With Secondary Licenses", as 373 | defined by the Mozilla Public License, v. 2.0. 374 | --------------------------------------------------------------------------------