├── .gitignore ├── COPYING.md ├── README.md ├── Urcheon ├── Action.py ├── Bsp.py ├── Default.py ├── Esquirel.py ├── FileSystem.py ├── Game.py ├── IqmConfig.py ├── Map.py ├── MapCompiler.py ├── Pak.py ├── Parallelism.py ├── Profile.py ├── Repository.py ├── Texset.py ├── Ui.py ├── Urcheon.py └── __init__.py ├── bin ├── esquirel └── urcheon ├── doc └── cute-granger.512.png ├── flake.nix ├── profile ├── file │ ├── common.conf │ ├── daemon.conf │ ├── unrealarena.conf │ └── unvanquished.conf ├── game │ ├── daemon.conf │ ├── unrealarena.conf │ └── unvanquished.conf ├── map │ ├── common.conf │ ├── daemon.conf │ ├── smokinguns.conf │ ├── unrealarena.conf │ └── unvanquished.conf ├── prevrun │ ├── daemon.prevrun │ └── unvanquished.prevrun ├── sloth │ └── daemon.sloth └── slothrun │ ├── daemon.slothrun │ └── unvanquished.slothrun ├── requirements.txt └── samples └── unvanquished └── entity_substitution.csv /.gitignore: -------------------------------------------------------------------------------- 1 | *.swp 2 | __pycache__ 3 | test/ 4 | -------------------------------------------------------------------------------- /COPYING.md: -------------------------------------------------------------------------------- 1 | Copyright (c) 2015 Thomas Debesse . 2 | 3 | Permission to use, copy, modify, and/or distribute this software for any purpose with or without fee is hereby granted, provided that the above copyright notice and this permission notice appear in all copies. 4 | 5 | The software is provided "as is" and the author disclaims all warranties with regard to this software including all implied warranties of merchantability and fitness. in no event shall the author be liable for any special, direct, indirect, or consequential damages or any damages whatsoever resulting from loss of use, data or profits, whether in an action of contract, negligence or other tortious action, arising out of or in connection with the use or performance of this software. 6 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | Urcheon 2 | ======= 3 | 4 | 5 | ![Cute Granger](doc/cute-granger.512.png) 6 | _My lovely granger needs a tender knight to care for his little flower._ 7 | 8 | 9 | Description 10 | ----------- 11 | 12 | ℹ️ Urcheon is purposed to manage and build source directories to produce game packages like Dæmon engine `dpk` packages or id Tech engines `pk3` or `pk4` packages. 13 | 14 | The primary usage of this toolset is to build of [Unvanquished](http://unvanquished.net) game media files. It was initially developed and tested against the files from [Interstellar Oasis](https://github.com/interstellar-oasis/interstellar-oasis). 15 | 16 | ℹ️ The Esquirel tool is also shipped with Urcheon, which is a tool to do some common editions on `.map` and `.bsp` files. Esquirel is a bit id Tech 3 map and bsp format centric at this time. 17 | 18 | 19 | How to run Urcheon and Esquirel 20 | ------------------------------- 21 | 22 | 💡️ You may want to install the [DaemonMediaAuthoringKit](https://github.com/DaemonEngine/DaemonMediaAuthoringKit). The _DaemonMediaAuthoringKit_ provides a convenient way to install Urcheon with its dependencies alongside some other usual edition tools (like the NetRadiant level editor). This makes easy to set-up a complete production environment. 23 | 24 | If you're installing Urcheon separately, for example if you already own all tools required by Urcheon (see [Dependencies](#dependencies)), once you cloned this repository you can add this `bin/` directory to your `$PATH` environment variable to run them easily. 25 | 26 | 27 | Urcheon help 28 | ------------ 29 | 30 | ℹ️ Urcheon is a game data package builder and packager. It converts files, compile maps, and produce distributables packages from them. 31 | 32 | 33 | Type `urcheon --help` for generic help. 34 | 35 | 36 | ### Creating a package source 37 | 38 | So you want to make a package, in this example we create a simple resource package that contains a text file. We will name this package `res-helloworld`. The DPK specifications reserves the `_` character as a version separator (can't be used in package name or version string). 39 | 40 | We create a folder named `res-helloworld_src.dpkdir` and enter it: 41 | 42 | ```sh 43 | mkdir res-helloworld_src.dpkdir 44 | cd res-helloworld_src.dpkdir 45 | ``` 46 | 47 | This will be a package for the Unvanquished game so let's configure it this way: 48 | 49 | ```sh 50 | mkdir .urcheon 51 | echo 'unvanquished' > .urcheon/game.txt 52 | ``` 53 | 54 | We need the package to ship some content, let's add a simple text file: 55 | 56 | ```sh 57 | mkdir about 58 | echo 'Hello world!' > about/helloworld.txt 59 | ``` 60 | 61 | 62 | ### Basic package tutorial 63 | 64 | Now we can build the package, this will produce another dpkdir you can use with your game, with files being either copied, converted or compiled given the need. 65 | 66 | It will be stored as `res-helloworld_src.dpkdir/build/_pakdir/pkg/res-helloworld_test.dpkdir`, you can tell the game engine to use `res-helloworld_src.dpkdir/build/_pakdir/pkg` as a pakpath to be able to find the package (example: `daemon -pakpath res-helloworld_src.dpkdir/build/_pakdir/pkg`). 67 | 68 | ```sh 69 | urcheon build 70 | ``` 71 | 72 | Then we can produce the distributable `dpk` package. 73 | 74 | It will be stored as `res-helloworld_src.dpkdir/build/pkg/map-castle_.dpk`. The version will be computed by Urcheon (see below). 75 | 76 | You can tell the game engine to use `res-helloworld_src.dpkdir/build/_pakdir/pkg` as a pakpath to be able to find the package (example: `daemon -pakpath res-helloworld_src.dpkdir/build/_pakdir/pkg`). 77 | 78 | ```sh 79 | urcheon package 80 | ``` 81 | 82 | We can also pass the dpkdir path to Urcheon, this way: 83 | 84 | ```sh 85 | cd .. 86 | urcheon build res-helloworld_src.dpkdir 87 | urcheon package res-helloworld_src.dpkdir 88 | ``` 89 | 90 | 91 | ### Building in an arbitrary folder 92 | 93 | As you noticed with our previous example, the built files were produced within the source dpkdir, with this layout: 94 | 95 | ``` 96 | res-helloworld_src.dpkdir/build/_pakdir/pkg/res-helloworld_test.dpkdir 97 | res-helloworld_src.dpkdir/build/pkg/res-helloworld_.dpk 98 | ``` 99 | 100 | You may not want this, especially if you want to build many package and want to get a single build directory. You can use the `--build-prefix ` option to change that, like that: 101 | 102 | ```sh 103 | urcheon --build-prefix build build res-helloworld_src.dpkdir 104 | urcheon --build-prefix build package res-helloworld_src.dpkdir 105 | ``` 106 | 107 | You get: 108 | 109 | ``` 110 | build/_pakdir/pkg/res-helloworld_test.dpkdir 111 | build/pkg/res-helloworld_.dpk 112 | ``` 113 | 114 | 115 | ### Special case of prepared dpkdir 116 | 117 | Some packages need to be prepared before being built. This is because some third-party software requires some files to exist in source directories, for example map editors and compilers. 118 | 119 | Urcheon can produce `.shader` material files or model formats and others in source directory with the `prepare` command, for such package, the build and package routine is: 120 | 121 | ```sh 122 | urcheon prepare 123 | urcheon build 124 | urcheon package 125 | ``` 126 | 127 | 128 | ### Package collection tutorial 129 | 130 | A package collection is a folder containing a `src` subdirectory full of source dpkdirs. 131 | 132 | Let's create a package collection, enter it and create the basic layout: 133 | 134 | ```sh 135 | mkdir PackageCollection 136 | cd PackageCollection 137 | ``` 138 | 139 | To tell Urcheon this folder is a package collection, you just need to create the `.urcheon/collection.txt` file, it just has to exist, en empty file is enough: 140 | 141 | ```sh 142 | mkdir .urcheon 143 | touch .urcheon/collection.txt 144 | ``` 145 | 146 | Then we create two packages, they must be stored in a subdirectory named `src`: 147 | 148 | ```sh 149 | mkdir pkg 150 | 151 | mkdir pkg/res-package1_src.dpkdir 152 | mkdir pkg/res-package1_src.dpkdir/.urcheon 153 | echo 'unvanquished' > pkg/res_package1_src.dpkdir/.urcheon/game.txt 154 | mkdir pkg/res-package1_src.dpkdir/about 155 | echo 'Package 1' > pkg/res_package1_src.dpkdir/about/package1.txt 156 | 157 | mkdir pkg/res-package2_src.dpkdir 158 | mkdir pkg/res-package2_src.dpkdir/.urcheon 159 | echo 'unvanquished' > pkg/res_package2_src.dpkdir/.urcheon/game.txt 160 | mkdir pkg/res-package2_src.dpkdir/about 161 | echo 'Package 2' > pkg/res_package1_src.dpkdir/about/package2.txt 162 | 163 | urcheon build pkg/*.dpkdir 164 | urcheon package pkg/*.dpkdir 165 | ``` 166 | 167 | You'll get this layout: 168 | 169 | ``` 170 | build/_pakdir/pkg/res-package1_test.dpkdir 171 | build/_pakdir/pkg/res-package2_test.dpkdir 172 | build/pkg/res-package1_.dpk 173 | build/pkg/res-package2_.dpk 174 | ``` 175 | 176 | You'll be able to use `build/_pakdir/pkg` or `build/pkg` as pakpath to make the game engine able to find those packages, example: `daemon -pakpath PackageCollection/build/_pakdir/pkg` or `daemon -pakpath PackageCollection/build/pkg`. 177 | 178 | 179 | ### Delta package building 180 | 181 | Urcheon can produce partial packages relying on older versions of the same package. To be able to do that you need to have your dpkdirs stored in git repositories with version computed from git repositories. 182 | 183 | You pass the old reference with the `--reference` build option followed by the git reference (for example a git tag), this way: 184 | 185 | ```sh 186 | urcheon build --reference 187 | urcheon package 188 | ``` 189 | 190 | 191 | ### Dealing with multiple collections 192 | 193 | When building dpkdirs from a collection requiring dpkdirs from another collection, one can set the `PAKPATH` environment variable this way (the separator is `;` on Windows and `:` on every other operating system): 194 | 195 | ```sh 196 | export PAKPATH='Collection1/pkg:Collection2/pkg:Collection3/pkg' 197 | ``` 198 | 199 | or (Windows): 200 | 201 | ```cmd 202 | set PAKPATH='Collection1/pkg;Collection2/pkg;Collection3/pkg' 203 | ``` 204 | 205 | 206 | ### Real life examples 207 | 208 | Here we clone the [UnvanquishedAssets](https://github.com/UnvanquishedAssets/UnvanquishedAssets) repository, prepare, build and package it: 209 | 210 | ```sh 211 | git clone --recurse-submodules \ 212 | https://github.com/UnvanquishedAssets/UnvanquishedAssets.git 213 | 214 | urcheon prepare UnvanquishedAssets/pkg/*.dpkdir 215 | urcheon build UnvanquishedAssets/pkg/*.dpkdir 216 | urcheon package UnvanquishedAssets/pkg/*.dpkdir 217 | ``` 218 | 219 | We can load the Unvanquished game with the stock plat23 map this way: 220 | 221 | ```sh 222 | daemon -pakpath UnvanquishedAssets/build/pkg +devmap plat23 223 | ``` 224 | 225 | 226 | Here we build and package delta Unvanquished packages for `res-` and `tex-` packages, only shipping files modified since Unvanquished 0.52.0, and full packages for map ones: 227 | 228 | ```sh 229 | urcheon prepare UnvanquishedAssets/pkg/*.dpkdir 230 | urcheon build --reference unvanquished/0.54.1 \ 231 | UnvanquishedAssets/pkg/res-*.dpkdir \ 232 | UnvanquishedAssets/pkg/tex-*.dpkdir 233 | urcheon build UnvanquishedAssets/pkg/map-*.dpkdir 234 | urcheon package UnvanquishedAssets/pkg/*.dpkdir 235 | ``` 236 | 237 | Here we clone the [InterstellarOasis](https://github.com/InterstellarOasis/InterstellarOasis) and build it. 238 | 239 | Since it needs to access dpkdirs from UnvanquishedAssets, we set `UnvanquishedAssets/pkg` as a pakpath using the `PAKPATH` environment variable. 240 | 241 | We also need the `UnvanquishedAsset/pkg` folder to be prepared, but there is no need to prepare `InterstellarOasis/pkg`, only build and package it: 242 | 243 | ```sh 244 | git clone --recurse-submodules \ 245 | https://github.com/InterstellarOasis/InterstellarOasis.git 246 | 247 | export PAKPATH=UnvanquishedAssets/pkg 248 | 249 | urcheon prepare UnvanquishedAssets/pkg/*.dpkdir 250 | urcheon build InterstellarOasis/pkg/*.dpkdir 251 | urcheon package InterstellarOasis/pkg/*.dpkdir 252 | ``` 253 | 254 | Given both `UnvanquishedAssets` and `InterstellarOasis` are built, one can load the Unvanquished game with the third-party atcshd map this way: 255 | 256 | ```sh 257 | daemon -pakpath UnvanquishedAssets/build/pkg InterstellarOasis/build/pkg +devmap atcshd 258 | ``` 259 | 260 | 261 | ### DPK version computation 262 | 263 | Urcheon knows how to write the DPK version string, computing it if needed. 264 | 265 | The recommended way is to store the dpkdir as a git repository, preferably one repository per dpkdir. Doing this unlock all abilities of Urcheon. It will be able to compute versions from git tag and do delta paks (partial DPK relying on older versions of it). 266 | 267 | If the dpkdir is not a git repository, Urcheon provides two ways to set the version string. 268 | 269 | One way is to write the version string in the `.urcheon/version.txt` file, like this: 270 | 271 | ```sh 272 | echo '0.1' > .urcheon/version.txt 273 | ``` 274 | 275 | Urcheon does not implement delta packaging when doing this way (it may be implementable though). 276 | 277 | Another way is to set the version string in the dpkdir name. 278 | 279 | For example: `res-helloworld_0.1.dpkdir` 280 | 281 | This is the least recommended method if you care about version control. Urcheon will never implement delta packaging for this (it's an unsolvable problem). 282 | 283 | 284 | ### More about Urcheon abilities 285 | 286 | As we seen, this tool can prepare the assets (sloth-driven material generation, iqe compilation), build them (asset compression, bspdir merge, map compilation), then package them 287 | 288 | Each file type (lightmap, skybox, texture, model…) is recognized thanks to some profiles you can extend or modify, picking the optimal compression format for each kind. 289 | 290 | If needed, you can write explicit rules for some specific files to force some format or blacklist some files. 291 | 292 | The Urcheon tool becomes more powerful when used in git-tracked asset repositories: it can build partial package given any given git reference (for example to build a package that contains only things since the previous release tag), and it can automatically computes the package version using tags, commits date, and commit id. 293 | 294 | Urcheon also allows to define per-map compilation profile. 295 | 296 | The asset conversion and compression pass is heavily parallelized to speed-up the process. 297 | 298 | 299 | ## More about Urcheon options and commands 300 | 301 | Type `urcheon --help` for help about generic options. 302 | 303 | 304 | ### The `discover` command 305 | 306 | This is an optional and not recommended command, you can use it if you want or need to not rely on automatic action lists. This stage produces your action lists, do not forget to use `-n` or `--no-auto` options on `prepare` and `build` stages later! 307 | 308 | In most case, you don't need it. If you need it, it means you have to fix or extend file detection profiles. 309 | 310 | This stage is not recommended since it will add a lot of noise to your git history each time you add or remove files. 311 | 312 | This can be used to debug the automatic action list generation (what Urcheon decides to do for each file). 313 | 314 | Type `urcheon discover --help` for help about the specific `discover` command options. 315 | 316 | 317 | ### The `prepare` command 318 | 319 | This is an optional stage to prepare your source directory, it is needed when you have to produce files to feed your map editor or your map compiler, like material files or models. If your texture package is `sloth` driven, you must define a `slothrun` file per texture set and use the `prepare` stage. 320 | 321 | If you need to prepare your source, always call this stage before the `build` one. 322 | 323 | Type `urcheon prepare --help` for help about the specific `prepare` command options. 324 | 325 | 326 | ### The `build` command 327 | 328 | This stage is required, it produces for you a testable pakdir with final formats: compressed textures, compiled map etc. If your assets are tracked in a git repository, you can build a partial pakdir using the `-r` or `--reference` options followed by an arbitrary past git reference (tag, commit…) 329 | 330 | You can set a `PAKPATH` environment variable to declare multiple directories containing other pakdir, it's needed if your package relies on other packages that are not in the current collection. The format is like the good old `PATH` environment variable: pathes separated with semicolons on Windows and colons on every other operating system. 331 | 332 | If you're building a partial `dpk` package, an extra entry containing your previous package version is added to the `DEPS` file automatically. 333 | 334 | You must call this stage before the `package` one. 335 | 336 | Type `urcheon build --help` for help about the specific `build` command options. 337 | 338 | 339 | ### The `package` command 340 | 341 | This stage produces a pak file from your previously built pakdir. Urcheon automatically writes the version string of the produced pak and if your game supports `dpk` format it will automatically rewrites your `DEPS` file with versions from other pakdirs found in `PAKPATH`. 342 | 343 | Type `urcheon package --help` for help about the specific `package` command options. 344 | 345 | 346 | ### The `clean` command 347 | 348 | This stage is convenient to clean stuff, it has multiple options if you don't want to clean-up everything. 349 | 350 | This will delete built files from the source dpkdir if prepared, and from the `build/_pakdir/pkg` and `build/pkg` folders: 351 | 352 | ```sh 353 | urcheon clean pkg/ 354 | ``` 355 | 356 | You can clean those folders selectively: 357 | 358 | ```sh 359 | urcheon clean --source pkg/ 360 | urcheon clean --test pkg/ 361 | urcheon clean --package pkg/ 362 | ``` 363 | 364 | Those special options also exist, here to only clean `build/_pakdir/pkg` and `build/pkg`: 365 | 366 | ```sh 367 | urcheon clean --build pkg/ 368 | ``` 369 | 370 | This will only delete built maps from `build/_pakdir/pkg` (keeping every other build files: 371 | 372 | ```sh 373 | urcheon clean --maps pkg/ 374 | ``` 375 | 376 | Type `urcheon clean --help` for help about the specific `clean` command options. 377 | 378 | 379 | ### Dependencies 380 | 381 | 💡️ The [DaemonMediaAuthoringKit](https://github.com/DaemonEngine/DaemonMediaAuthoringKit) makes easy to set-up a complete production environment with Urcheon, its dependencies, and other tools. 382 | 383 | These are the Python3 modules you will need to run `urcheon`: `argparse`, `colorama`, `pillow`, `psutil`, and `toml` >= 0.9.0. 384 | 385 | The `urcheon` tool relies on: 386 | 387 | - [`q3map2` from NetRadiant](https://gitlab.com/xonotic/netradiant), the one maintained by the Xonotic team, to compile maps (the one from GtkRadiant is lacking required features); 388 | - [Sloth](https://github.com/Unvanquished/Sloth) if you need it to generate shader files; 389 | - [`cwebp` from Google](https://developers.google.com/speed/webp/docs/cwebp) to convert images to webp format; 390 | - [`crunch` from Dæmon](https://github.com/DaemonEngine/crunch) to convert images to crn format (the one from BinomialLLC is not compatible and the one from Unity lacks required features); 391 | - [`opusenc` from Xiph](http://opus-codec.org) to convert sound files to opus format; 392 | - [`iqmtool` from FTE QuakeWorld](https://sourceforge.net/p/fteqw/code/HEAD/tree/trunk/iqm/) to convert iqe models (the `iqm` one from Sauerbraten is lacking required features). 393 | - [`sloth`](https://github.com/DaemonEngine/Sloth/) to generate .shader material files. 394 | 395 | To summarize: 396 | 397 | * Python3 modules: `argparse colorama pillow psutil toml>=0.9.0` 398 | * Third party tools: `crunch cwebp iqmtool opusenc q3map2 sloth.py` 399 | 400 | 401 | Esquirel help 402 | ------------- 403 | 404 | ℹ️ Esquirel is a tool to inspect `.map` and `.bsp` files and apply some modifications to them. 405 | 406 | Type `esquirel --help` for generic help. 407 | 408 | Esquirel offers multiple commands. 409 | 410 | 411 | ### The `map` command 412 | 413 | It allows to parse some maps (id Tech 3 format only supported at this time): de-numberize them for better diff, export entities as seen in bsp, or substitutes entity keywords using some substitution list you can write yourself. 414 | 415 | Example: 416 | 417 | ```sh 418 | esquirel map --input-map file.map \ 419 | --substitute-keywords substitution.csv \ 420 | --disable-numbering \ 421 | --output-map file.map 422 | ``` 423 | 424 | This `esquirel` call updates obsolete entities keywords using the `substitution.csv` list, disabling the entity numbering to make lately diffing easier. 425 | 426 | Type `esquirel map --help` about the specific `map` command options. 427 | 428 | 429 | ### The `bsp` command 430 | 431 | It allows to edit some bsp (id Tech 3 format only supported at this time): import/export texture lists (this way you can rename them or tweak their surface flags), import/export entities, import/export lightmaps (this way you can repaint them by hand or use them as compressed external instead of internal one), or print some statistics. The best part in the `bsp` stage is the ability to convert a `bsp` to a `bspdir` that contains one file per lump, and some of them are stored in editable text format. These `bspdir` are mergeable back as a new `bsp`, allowing many modification or fixes to maps you lost source for. It allows easy maintenance or port to other games. 432 | 433 | Example: 434 | 435 | ```sh 436 | esquirel bsp --input-bsp level.bsp \ 437 | --list-lumps \ 438 | --output-bspdir level.bspdir 439 | ``` 440 | 441 | This `esquirel` call converts a `bsp` file to a `bspdir` directory, printing some lump statistics at the same time. 442 | 443 | The reverse operation is: 444 | 445 | ```sh 446 | esquirel bsp --input-bspdir level.bspdir \ 447 | --list-lumps \ 448 | --output-bsp level.bsp 449 | ``` 450 | 451 | Type `esquirel bsp --help` about the specific `bsp` command options. 452 | 453 | 454 | Warning 455 | ------- 456 | 457 | No warranty is given, use this at your own risk. It can make you awesome in space if used inconsiderately. 458 | 459 | 460 | Author 461 | ------ 462 | 463 | Thomas Debesse 464 | 465 | 466 | Copyright 467 | --------- 468 | 469 | This toolbox is distributed under the highly permissive and laconic [ISC License](COPYING.md). 470 | 471 | 472 | Trivia 473 | ------ 474 | 475 | _Esquirel is the Englo-Norman word for “squirrel”, from the Old French “escurel” who displaced Middle English “aquerne”._ 476 | 477 | _Urcheon is the Middle English term for “hedgehog”, used to refer the related ordinary in heraldry._ 478 | -------------------------------------------------------------------------------- /Urcheon/Bsp.py: -------------------------------------------------------------------------------- 1 | #! /usr/bin/env python3 2 | #-*- coding: UTF-8 -*- 3 | 4 | ### Legal 5 | # 6 | # Author: Thomas DEBESSE 7 | # License: ISC 8 | # 9 | 10 | from Urcheon import Map 11 | from Urcheon import Ui 12 | import __main__ as m 13 | import argparse 14 | import glob 15 | import json 16 | import logging 17 | import os 18 | import struct 19 | import sys 20 | from collections import OrderedDict 21 | from logging import debug 22 | from PIL import Image 23 | 24 | class Lump(): 25 | bsp_parser_dict = None 26 | 27 | def readBspDirLump(self, dir_name, lump_name): 28 | file_list = glob.glob(dir_name + os.path.sep + lump_name + os.path.extsep + "*") 29 | 30 | if len(file_list) > 1: 31 | # TODO: handle that 32 | Ui.error("more than one " + lump_name + " lump in bspdir") 33 | if len(file_list) == 0: 34 | # TODO: warning? 35 | return 36 | 37 | file_path = file_list[0] 38 | file_ext = os.path.splitext(file_path)[-1][1:] 39 | file_name = os.path.splitext(os.path.basename(file_path))[0] 40 | 41 | if file_ext == "bin": 42 | if file_name in self.bsp_parser_dict["lump_name_list"]: 43 | blob_file = open(file_path, "rb") 44 | self.importLump(blob_file.read()) 45 | blob_file.close() 46 | else: 47 | Ui.error("unknown lump file: " + file_name) 48 | 49 | elif not self.validateExtension(file_ext): 50 | Ui.error("unknown lump format: " + file_path) 51 | 52 | if file_ext == "d": 53 | self.readDir(file_path) 54 | else: 55 | self.readFile(file_path) 56 | 57 | 58 | class Blob(Lump): 59 | blob_stream = None 60 | 61 | def isEmpty(self): 62 | return not self.blob_stream 63 | 64 | def readFile(self, file_name): 65 | blob_file = open(file_name, "rb") 66 | self.importLump(blob_file.read()) 67 | blob_file.close() 68 | return True 69 | 70 | def writeFile(self, file_name): 71 | blob_file = open(file_name, "wb") 72 | blob_file.write(self.exportLump()) 73 | blob_file.close() 74 | 75 | def writeBspDirLump(self, dir_name, lump_name): 76 | self.writeFile(dir_name + os.path.sep + lump_name + os.path.extsep + "bin") 77 | 78 | def importLump(self, blob): 79 | self.blob_stream = blob 80 | 81 | def exportLump(self): 82 | return self.blob_stream 83 | 84 | 85 | class Q3Entities(Lump): 86 | entities_as_map = None 87 | 88 | def isEmpty(self): 89 | return not self.entities_as_map 90 | 91 | def validateExtension(self, file_ext): 92 | return file_ext == "txt" 93 | 94 | def readFile(self, file_name): 95 | entities_file = open(file_name, "rb") 96 | self.importLump(entities_file.read()) 97 | entities_file.close() 98 | return True 99 | 100 | def writeFile(self, file_name): 101 | entities_file = open(file_name, "wb") 102 | entities_file.write(self.exportLump().split(b'\0', 1)[0]) 103 | entities_file.close() 104 | return True 105 | 106 | def writeBspDirLump(self, dir_name, lump_name): 107 | self.writeFile(dir_name + os.path.sep + lump_name + os.path.extsep + "txt") 108 | 109 | def printString(self): 110 | print(bytes.decode(self.exportLump().split(b'\0', 1)[0])) 111 | 112 | def printList(self): 113 | print("*** Entities") 114 | i = 0 115 | for entity in self.entities_as_map.entity_list: 116 | string = "" 117 | for thing in entity.thing_list: 118 | if isinstance(thing, Map.KeyValue): 119 | string += "\"" + thing.key + "\": \"" + thing.value + "\", " 120 | 121 | print(str(i) + ": [" + string[:-2] + "]") 122 | i += 1 123 | 124 | print("") 125 | return True 126 | 127 | def printSoundList(self): 128 | print("*** Entities") 129 | i = 0 130 | for entity in self.entities_as_map.entity_list: 131 | found = False 132 | for thing in entity.thing_list: 133 | if isinstance(thing, Map.KeyValue): 134 | for sound_keyword in Map.q3_sound_keyword_list: 135 | if thing.key.lower() == sound_keyword.lower(): 136 | print(str(i) + ": " + thing.value + " [" + sound_keyword + "]") 137 | i += 1 138 | 139 | print("") 140 | return True 141 | 142 | def substituteKeywords(self, substitution): 143 | self.entities_as_map.substituteKeywords(substitution) 144 | 145 | def lowerCaseFilePaths(self): 146 | self.entities_as_map.lowerCaseFilePaths() 147 | 148 | def importLump(self, blob): 149 | self.entity_list = [] 150 | entities_bstring = blob.split(b'\0', 1)[0] 151 | 152 | self.entities_as_map = Map.Map() 153 | self.entities_as_map.numbering_enabled = False 154 | self.entities_as_map.readBlob(entities_bstring) 155 | 156 | def exportLump(self): 157 | blob = b'' 158 | blob += self.entities_as_map.exportFile().encode() 159 | blob += b'\0' 160 | return blob 161 | 162 | 163 | class Q3Textures(Lump): 164 | texture_list = None 165 | 166 | def int2bstr(self, i): 167 | return "{0:b}".format(i).zfill(30) 168 | 169 | def bstr2int(self, s): 170 | return int(s, 2) 171 | 172 | def isEmpty(self): 173 | return not self.texture_list 174 | 175 | def validateExtension(self, file_ext): 176 | return file_ext == "csv" 177 | 178 | def readFile(self, file_name): 179 | # TODO: check 180 | textures_file = open(file_name, 'rb') 181 | 182 | textures_file_bstring = textures_file.read() 183 | self.texture_list = [] 184 | 185 | for texture_line_bstring in textures_file_bstring.split(b'\n'): 186 | # TODO: check 3 comma minimum 187 | # TODO: allow string path with comma 188 | if texture_line_bstring != b'': 189 | bstring_list = texture_line_bstring.split(b',') 190 | flags = self.bstr2int(bstring_list[0]) 191 | contents = self.bstr2int(bstring_list[1]) 192 | name = bytes.decode(bstring_list[2]) 193 | self.texture_list.append({"name": name, "flags": flags, "contents": contents}) 194 | 195 | textures_file.close() 196 | 197 | return True 198 | 199 | def writeFile(self, file_name): 200 | textures_string = "" 201 | for i in range(0, len(self.texture_list)): 202 | textures_string += self.int2bstr(self.texture_list[i]["flags"]) + "," 203 | textures_string += self.int2bstr(self.texture_list[i]["contents"]) + "," 204 | textures_string += self.texture_list[i]["name"] + "\n" 205 | 206 | # TODO: check 207 | textures_file = open(file_name, "wb") 208 | textures_file.write(textures_string.encode()) 209 | textures_file.close() 210 | 211 | def writeBspDirLump(self, dir_name, lump_name): 212 | self.writeFile(dir_name + os.path.sep + lump_name + os.path.extsep + "csv") 213 | 214 | def printList(self): 215 | # TODO: check 216 | 217 | print("*** Textures:") 218 | for i in range(0, len(self.texture_list)): 219 | print(str(i) + ": " + self.texture_list[i]["name"] + " [" + self.int2bstr(self.texture_list[i]["flags"]) + ", " + self.int2bstr(self.texture_list[i]["contents"]) + "]") 220 | print("") 221 | 222 | def lowerCaseFilePaths(self): 223 | textures_count = len(self.texture_list) 224 | 225 | for i in range(0, textures_count): 226 | self.texture_list[i]["name"] = self.texture_list[i]["name"].lower() 227 | 228 | def importLump(self, blob): 229 | # TODO: check exists 230 | 231 | # 64 bytes string name 232 | # 4 bytes integer flags 233 | # 4 bytes integer contents 234 | textures_count = int(len(blob) / 72) 235 | 236 | self.texture_list = [] 237 | 238 | # TODO: check 239 | 240 | for i in range(0, textures_count): 241 | offset = i * 72 242 | bstring = blob[offset:offset + 64] 243 | name = bytes.decode(bstring.split(b'\0', 1)[0]) 244 | flags, contents = struct.unpack(' 1: 477 | # TODO: handling 478 | Ui.error("more than one " + lump_name + " lump in bspdir") 479 | if len(file_list) == 0: 480 | # TODO: warning? 481 | continue 482 | 483 | file_path = file_list[0] 484 | file_ext = os.path.splitext(file_path)[-1][1:] 485 | file_name = os.path.splitext(os.path.basename(file_path))[0] 486 | 487 | lump = self.bsp_parser_dict["lump_dict"][lump_name]() 488 | lump.bsp_parser_dict = self.bsp_parser_dict 489 | 490 | lump.readBspDirLump(dir_name, lump_name) 491 | self.lump_dict[lump_name] = lump.exportLump() 492 | 493 | self.lump_directory[lump_name] = {} 494 | self.lump_directory[lump_name]["offset"] = None 495 | self.lump_directory[lump_name]["length"] = None 496 | 497 | def printFileName(self): 498 | print("*** File:") 499 | print(self.bsp_file_name) 500 | print("") 501 | 502 | 503 | def substituteKeywords(self, substitution): 504 | for lump_name in ["entities"]: 505 | if lump_name in self.lump_dict: 506 | lump = self.bsp_parser_dict["lump_dict"][lump_name]() 507 | lump.importLump(self.lump_dict[lump_name]) 508 | lump.substituteKeywords(substitution) 509 | self.lump_dict[lump_name] = lump.exportLump() 510 | 511 | def lowerCaseFilePaths(self): 512 | for lump_name in ["entities", "textures"]: 513 | if lump_name in self.lump_dict: 514 | lump = self.bsp_parser_dict["lump_dict"][lump_name]() 515 | lump.importLump(self.lump_dict[lump_name]) 516 | lump.lowerCaseFilePaths() 517 | self.lump_dict[lump_name] = lump.exportLump() 518 | 519 | 520 | def readLumpList(self): 521 | self.lump_directory = {} 522 | 523 | # TODO: check 524 | 525 | larger_offset = 0 526 | ql_advertisements_offset = 0 527 | for lump_name in self.bsp_parser_dict["lump_name_list"]: 528 | # FIXME: q3 centric 529 | # 4 bytes string magic number (IBSP) 530 | # 4 bytes integer version 531 | # 4 bytes integer lump offset 532 | # 4 bytes integer lump size 533 | self.bsp_file.seek(8 + (self.bsp_parser_dict["lump_name_list"].index(lump_name) * 8)) 534 | 535 | self.lump_directory[lump_name] = {} 536 | 537 | offset, length = struct.unpack(' larger_offset: 557 | larger_offset = offset 558 | ql_advertisements_offset = offset + length 559 | 560 | self.lump_directory[lump_name]["offset"], self.lump_directory[lump_name]["length"] = (offset, length) 561 | self.lump_dict[lump_name] = None 562 | 563 | def printLumpList(self): 564 | # TODO: check 565 | 566 | print("*** Lumps:") 567 | for i in range(0, len(self.bsp_parser_dict["lump_name_list"])): 568 | lump_name = self.bsp_parser_dict["lump_name_list"][i] 569 | if lump_name in self.lump_directory: 570 | if not self.lump_directory[lump_name]["offset"]: 571 | # bspdir, length is also unknown 572 | print(str(i) + ": " + lump_name ) 573 | else: 574 | print(str(i) + ": " + lump_name + " [" + str(self.lump_directory[lump_name]["offset"]) + ", " + str(self.lump_directory[lump_name]["length"]) + "]") 575 | print("") 576 | 577 | def readLump(self, lump_name): 578 | # TODO: check 579 | 580 | # 4 bytes string magic number (IBSP) 581 | # 4 bytes integer version 582 | # 4 bytes integer lump offset 583 | # 4 bytes integer lump size 584 | self.bsp_file.seek(8 + (self.bsp_parser_dict["lump_name_list"].index(lump_name) * 8)) 585 | offset, length = struct.unpack(' 7 | # License: ISC 8 | # 9 | 10 | 11 | import logging 12 | import os.path 13 | import sys 14 | 15 | profile_dir = "profile" 16 | 17 | legacy_pakinfo_dir = ".pakinfo" 18 | legacy_setinfo_dir = ".setinfo" 19 | repository_config_dir = ".urcheon" 20 | 21 | cache_dir = ".cache" 22 | 23 | legacy_paktrace_dir = ".paktrace" 24 | paktrace_dir = os.path.join(cache_dir, "urcheon", "paktrace") 25 | paktrace_file_ext = ".json" 26 | 27 | default_base = "common" 28 | 29 | game_profile_dir = "game" 30 | game_profile_ext = ".conf" 31 | 32 | map_profile_dir = "map" 33 | map_profile_ext = ".conf" 34 | 35 | sloth_profile_dir = "sloth" 36 | sloth_profile_ext = ".sloth" 37 | 38 | prevrun_profile_dir = "prevrun" 39 | prevrun_profile_ext = ".prevrun" 40 | 41 | slothrun_profile_dir = "slothrun" 42 | slothrun_profile_ext = ".slothrun" 43 | 44 | file_profile_dir = "file" 45 | file_profile_ext = ".conf" 46 | action_list_dir = "action" 47 | action_list_ext = ".txt" 48 | 49 | ignore_list_file = "ignore.txt" 50 | 51 | base_dir = "pkg" 52 | 53 | build_prefix = "build" 54 | build_parent_dir = "_pakdir" 55 | build_root_prefix = os.path.join(build_prefix, build_parent_dir) 56 | build_base_prefix = os.path.join(build_root_prefix, base_dir) 57 | 58 | package_prefix = build_prefix 59 | package_parent_dir = "" 60 | package_root_prefix = os.path.join(package_prefix, package_parent_dir).rstrip("/") 61 | package_base_prefix = os.path.join(package_root_prefix, base_dir) 62 | 63 | prefix_dir = os.path.realpath(os.path.join(os.path.dirname(os.path.realpath(__file__)), "..")) 64 | 65 | # HACK: if installed in lib/python3/dist-packages 66 | if os.path.basename(prefix_dir) == "dist-packages": 67 | prefix_dir = os.path.realpath(os.path.join(os.path.dirname(os.path.realpath(__file__)), "../../../..")) 68 | 69 | for sub_dir in [".", "share/Urcheon"]: 70 | share_dir = os.path.realpath(os.path.join(prefix_dir, sub_dir)) 71 | if os.path.isdir(os.path.join(share_dir, profile_dir)): 72 | break 73 | 74 | def getCollectionConfigDir(source_dir): 75 | config_dir = os.path.join(source_dir, repository_config_dir) 76 | legacy_config_dir = os.path.join(source_dir, legacy_setinfo_dir) 77 | 78 | if os.path.isdir(config_dir): 79 | logging.debug("Found collection configuration directory: " + config_dir) 80 | 81 | elif os.path.isdir(legacy_config_dir): 82 | logging.debug("Found legacy collection configuration directory: " + legacy_config_dir) 83 | config_dir = legacy_config_dir 84 | 85 | return config_dir 86 | 87 | def getPakConfigDir(source_dir): 88 | config_dir = os.path.abspath(os.path.join(source_dir, repository_config_dir)) 89 | legacy_config_dir = os.path.abspath(os.path.join(source_dir, legacy_pakinfo_dir)) 90 | 91 | if os.path.isdir(config_dir): 92 | logging.debug("Found package configuration directory: " + config_dir) 93 | elif os.path.isdir(legacy_config_dir): 94 | logging.debug("Found legacy package configuration directory: " + config_dir) 95 | config_dir = legacy_config_dir 96 | 97 | return config_dir 98 | 99 | def getPakTraceDir(build_dir): 100 | cache_dir = os.path.abspath(os.path.join(build_dir, paktrace_dir)) 101 | legacy_cache_dir = os.path.abspath(os.path.join(build_dir, legacy_paktrace_dir)) 102 | 103 | if os.path.isdir(cache_dir): 104 | logging.debug("Found paktrace cache directory: " + cache_dir) 105 | elif os.path.isdir(legacy_cache_dir): 106 | logging.debug("Found legacy paktrace cache directory: " + cache_dir) 107 | cache_dir = legacy_cache_dir 108 | 109 | return cache_dir 110 | -------------------------------------------------------------------------------- /Urcheon/Esquirel.py: -------------------------------------------------------------------------------- 1 | #! /usr/bin/env python3 2 | #-*- coding: UTF-8 -*- 3 | 4 | ### Legal 5 | # 6 | # Author: Thomas DEBESSE 7 | # License: ISC 8 | # 9 | 10 | 11 | from Urcheon import Bsp 12 | from Urcheon import Map 13 | import argparse 14 | import logging 15 | from logging import debug 16 | import sys 17 | 18 | 19 | def main(): 20 | description="Esquirel is a gentle intendant for my lovely granger's garden." 21 | parser = argparse.ArgumentParser(description=description) 22 | 23 | parser.add_argument("-D", "--debug", help="print debug information", action="store_true") 24 | 25 | subparsers = parser.add_subparsers(help='contexts') 26 | subparsers.required = True 27 | 28 | map_parser = subparsers.add_parser('map', help='inspect or edit a map file') 29 | Map.add_arguments(map_parser) 30 | map_parser.set_defaults(func=Map.main) 31 | 32 | bsp_parser = subparsers.add_parser('bsp', help='inspect or edit a bsp file') 33 | Bsp.add_arguments(bsp_parser) 34 | bsp_parser.set_defaults(func=Bsp.main) 35 | 36 | args = parser.parse_args() 37 | 38 | if args.debug: 39 | logging.basicConfig(stream=sys.stderr, level=logging.DEBUG) 40 | debug("Debug logging activated") 41 | debug("args: " + str(args)) 42 | 43 | args.func(args) 44 | -------------------------------------------------------------------------------- /Urcheon/FileSystem.py: -------------------------------------------------------------------------------- 1 | #! /usr/bin/env python3 2 | #-*- coding: UTF-8 -*- 3 | 4 | ### Legal 5 | # 6 | # Author: Thomas DEBESSE 7 | # License: ISC 8 | # 9 | 10 | 11 | from Urcheon import Ui 12 | import logging 13 | import os 14 | 15 | 16 | def cleanRemoveFile(file_name): 17 | os.remove(file_name) 18 | dir_name = os.path.dirname(file_name) 19 | removeEmptyDir(dir_name) 20 | 21 | 22 | def removeEmptyDir(dir_name): 23 | if os.path.isdir(dir_name): 24 | if os.listdir(dir_name) == []: 25 | os.rmdir(dir_name) 26 | 27 | 28 | def makeFileSubdirs(file_name): 29 | os.makedirs(os.path.dirname(file_name), exist_ok=True) 30 | 31 | 32 | def getNewer(file_path_list): 33 | # TODO: remove files that does not exist before checking 34 | 35 | if file_path_list == []: 36 | Ui.error("can't find newer file if file list is empty") 37 | 38 | newer_path = file_path_list[0] 39 | 40 | for file_path in file_path_list: 41 | if os.stat(file_path).st_mtime > os.stat(newer_path).st_mtime: 42 | newer_path = file_path 43 | 44 | logging.debug("newer file: " + newer_path) 45 | return newer_path 46 | 47 | 48 | def isSameTimestamp(file_path, reference_path): 49 | if not os.path.isfile(file_path): 50 | logging.debug("file does not exist, can't have same timestamp: " + file_path) 51 | return False 52 | 53 | if os.stat(file_path).st_mtime == os.stat(reference_path).st_mtime: 54 | logging.debug("timestamp for file “" + file_path + "” is the same to reference one: " + reference_path) 55 | return True 56 | else: 57 | logging.debug("timestap for file “" + file_path + "”is not same to reference one: " + reference_path) 58 | return False 59 | 60 | def isDifferentTimestamp(file_path, reference_path): 61 | return not isSame(file_path, reference_path) 62 | -------------------------------------------------------------------------------- /Urcheon/Game.py: -------------------------------------------------------------------------------- 1 | #! /usr/bin/env python3 2 | #-*- coding: UTF-8 -*- 3 | 4 | ### Legal 5 | # 6 | # Author: Thomas DEBESSE 7 | # License: ISC 8 | # 9 | 10 | 11 | from Urcheon import Default 12 | from Urcheon import FileSystem 13 | from Urcheon import Profile 14 | from Urcheon import Repository 15 | from Urcheon import Ui 16 | from collections import OrderedDict 17 | import logging 18 | import os 19 | import toml 20 | 21 | # FIXME: do we need OrderedDict toml constructor here? 22 | 23 | 24 | class Game(): 25 | def __init__(self, source_tree): 26 | self.source_dir = source_tree.dir 27 | self.profile_fs = Profile.Fs(self.source_dir) 28 | 29 | self.key_dict = {} 30 | 31 | self.read(source_tree.game_name) 32 | 33 | self.pak_format = self.requireKey("pak") 34 | self.pak_ext = os.path.extsep + self.pak_format 35 | self.pakdir_ext = self.pak_ext + "dir" 36 | 37 | 38 | def read(self, profile_name): 39 | profile_name = os.path.join(Default.game_profile_dir, profile_name + Default.game_profile_ext) 40 | profile_path = self.profile_fs.getPath(profile_name) 41 | 42 | if not profile_path: 43 | Ui.error("game profile file not found: " + profile_name) 44 | 45 | logging.debug("reading game profile file " + profile_path) 46 | profile_file = open(profile_path, "r") 47 | profile_dict = toml.load(profile_file, _dict=OrderedDict) 48 | profile_file.close() 49 | 50 | if "_init_" in profile_dict.keys(): 51 | logging.debug("found “_init_” section in game profile: " + profile_path) 52 | if "extend" in profile_dict["_init_"].keys(): 53 | game_name = profile_dict["_init_"]["extend"] 54 | logging.debug("found “extend” instruction in “_init_” section: " + game_name) 55 | logging.debug("loading parent game profile") 56 | self.read(game_name) 57 | 58 | del profile_dict["_init_"] 59 | 60 | # only one section supported at this time, let's keep it simple 61 | if "config" in profile_dict.keys(): 62 | logging.debug("config found in game profile file: " + profile_path) 63 | self.key_dict = profile_dict["config"] 64 | 65 | 66 | def requireKey(self, key_name): 67 | # TODO: strip quotes 68 | if key_name in self.key_dict.keys(): 69 | return self.key_dict[key_name] 70 | else: 71 | Ui.error("key not found in pak config: " + key_name) 72 | 73 | 74 | def getKey(self, key_name): 75 | # TODO: strip quotes 76 | if key_name in self.key_dict.keys(): 77 | return self.key_dict[key_name] 78 | else: 79 | return None 80 | -------------------------------------------------------------------------------- /Urcheon/IqmConfig.py: -------------------------------------------------------------------------------- 1 | #! /usr/bin/env python3 2 | #-*- coding: UTF-8 -*- 3 | 4 | ### Legal 5 | # 6 | # Author: Thomas DEBESSE 7 | # License: ISC 8 | # 9 | 10 | import os 11 | import re 12 | 13 | class File(): 14 | def __init__(self): 15 | self.line_list = None 16 | self.output_pattern = re.compile(r"^[ \t]*output[ \t]*(?P.*)$") 17 | self.scene_pattern = re.compile(r"^[ \t]*scene[ \t]*(?P.*)$") 18 | 19 | def readFile(self, file_name): 20 | config_file = open(file_name, "r") 21 | 22 | self.line_list = config_file.readlines() 23 | config_file.close() 24 | 25 | def translate(self, scene_dir, output_dir): 26 | translated_line_list = [] 27 | 28 | for line in self.line_list: 29 | match = self.scene_pattern.match(line) 30 | if match: 31 | line = "scene " + os.path.join(scene_dir, match.group("scene")) 32 | match = self.output_pattern.match(line) 33 | if match: 34 | line = "output " + os.path.join(output_dir, match.group("output")) 35 | translated_line_list.append(line) 36 | 37 | self.line_list = translated_line_list 38 | 39 | def writeFile(self, file_name): 40 | config_string = "\n".join(self.line_list) 41 | config_file = open(file_name, "w") 42 | config_file.write(config_string) 43 | config_file.close() 44 | -------------------------------------------------------------------------------- /Urcheon/Map.py: -------------------------------------------------------------------------------- 1 | #! /usr/bin/env python3 2 | #-*- coding: UTF-8 -*- 3 | 4 | ### Legal 5 | # 6 | # Author: Thomas DEBESSE 7 | # License: ISC 8 | # 9 | 10 | from Urcheon import Ui 11 | import __main__ as m 12 | import argparse 13 | import logging 14 | import os 15 | import re 16 | import sys 17 | from collections import OrderedDict 18 | from logging import debug 19 | 20 | # see https://github.com/Unvanquished/Unvanquished/blob/master/src/gamelogic/game/g_spawn.cpp 21 | q3_sound_keyword_list = [ 22 | "noise", 23 | "sound1to2", 24 | "sound2to1", 25 | "soundPos1", 26 | "soundPos2", 27 | ] 28 | 29 | # Two kinds of Quake 3 brush formats are known: 30 | # - Q3 Brush, also known as “Axial Projection” or AP, sometime referred to as Q3 Legacy Brush, the historical and most used format, seems to be derived from Quake format. 31 | # - Q3 BP Brush, also known as “Brush Primitives” or BP, sometime referred to as “Alternate Texture Projection”, also very old but less old (introduced in Q3Radiant 192). 32 | 33 | # For Quake 3 brush and patch format, see: 34 | # see https://web.archive.org/web/20160316213335/http://forums.ubergames.net/topic/2658-understanding-the-quake-3-map-format/ 35 | 36 | class Map(): 37 | def __init__(self): 38 | self.entity_list = None 39 | 40 | # write entity numbers or not 41 | self.numbering_enabled = True 42 | 43 | def readFile(self, file_name): 44 | input_map_file = open(file_name, "rb") 45 | 46 | map_bstring = input_map_file.read() 47 | 48 | self.readBlob(map_bstring) 49 | 50 | input_map_file.close() 51 | 52 | def readBlob(self, map_bstring): 53 | map_lines = str.splitlines(bytes.decode(map_bstring)) 54 | 55 | in_entity = False 56 | in_shape = False 57 | 58 | in_q3brush = False 59 | 60 | start_q3bpbrush = False 61 | in_q3bpbrush = False 62 | 63 | start_q3patch = False 64 | in_q3patch = False 65 | 66 | in_q3patch_material = False 67 | 68 | start_q3patch_matrix = False 69 | in_q3patch_matrix = False 70 | 71 | self.entity_list = [] 72 | 73 | # empty line 74 | empty_line_pattern = re.compile(r"^[ \t]*$") 75 | 76 | # generic comment 77 | generic_comment_pattern = re.compile("[ \t]*//.*$") 78 | 79 | # entity comment 80 | # // entity num 81 | # entity_comment_pattern = re.compile(r"^[ \t]*//[ \t]+entity[ \t]+(?P[0-9]+)[ \t]*$") 82 | 83 | # block opening 84 | # { 85 | block_opening_pattern = re.compile(r"^[ \t]*{[ \t]*$") 86 | 87 | # keyvalue pair 88 | # "key" "value" 89 | keyvalue_pattern = re.compile(r"^[ \t]*\"(?P[^\"]*)\"[ \t]+\"(?P[^\"]*)\"[ \t]*$") 90 | 91 | # shape comment 92 | # // brush num 93 | # shape_comment_pattern = re.compile(r"^[ \t]*//[ \t]+brush[ \t]+(?P[0-9]+)[ \t]*$") 94 | 95 | # q3 brush plane line 96 | # coord, textures 97 | # ( coord0_x coord0_y coord0_z ) ( coord1_x coord1_y coord1_z ) ( coord2_x coord2_y coord2_z ) material shift_x shift_y rotation scale_x scale_y flags_content flags_surface value 98 | q3brush_plane_pattern = re.compile(r""" 99 | [ \t]*\([ \t]* 100 | (?P-?[0-9.]+)[ \t]+ 101 | (?P-?[0-9.]+)[ \t]+ 102 | (?P-?[0-9.]+)[ \t]* 103 | \)[ \t]* 104 | \([ \t]* 105 | (?P-?[0-9.]+)[ \t]+ 106 | (?P-?[0-9.]+)[ \t]+ 107 | (?P-?[0-9.]+)[ \t]* 108 | \)[ \t]* 109 | \([ \t]* 110 | (?P-?[0-9.]+)[ \t]+ 111 | (?P-?[0-9.]+)[ \t]+ 112 | (?P-?[0-9.]+)[ \t]* 113 | \)[ \t]* 114 | (?P[^ \t]+)[ \t]+ 115 | (?P-?[0-9.]+)[ \t]+ 116 | (?P-?[0-9.]+)[ \t]+ 117 | (?P-?[0-9.]+)[ \t]+ 118 | (?P-?[0-9.]+)[ \t]+ 119 | (?P-?[0-9.]+)[ \t]+ 120 | (?P[0-9]+)[ \t]+ 121 | (?P[0-9]+)[ \t]+ 122 | (?P[0-9]+) 123 | [ \t]*$ 124 | """, re.VERBOSE) 125 | 126 | # q3 bp brush start 127 | # brushDef 128 | q3bpbrush_start_pattern = re.compile(r"^[ \t]*brushDef[ \t]*$") 129 | 130 | # q3 bp brush plane line 131 | # coord, textures 132 | # ( coord0_x coord0_y coord0_z ) ( coord1_x coord1_y coord1_z ) ( coord2_x coord2_y coord2_z ) ( ( texdef_xx texdef_yx texdef_tx ) ( texdef_xy texdef_yy texdef_ty ) ) material flag_content flag_surface value 133 | q3bpbrush_plane_pattern = re.compile(r""" 134 | [ \t]*\([ \t]* 135 | (?P-?[0-9.]+)[ \t]+ 136 | (?P-?[0-9.]+)[ \t]+ 137 | (?P-?[0-9.]+)[ \t]* 138 | \)[ \t]* 139 | \([ \t]* 140 | (?P-?[0-9.]+)[ \t]+ 141 | (?P-?[0-9.]+)[ \t]+ 142 | (?P-?[0-9.]+)[ \t]* 143 | \)[ \t]* 144 | \([ \t]* 145 | (?P-?[0-9.]+)[ \t]+ 146 | (?P-?[0-9.]+)[ \t]+ 147 | (?P-?[0-9.]+)[ \t]* 148 | \)[ \t]* 149 | \([ \t]* 150 | \([ \t]* 151 | (?P-?[0-9.]+)[ \t]+ 152 | (?P-?[0-9.]+)[ \t]+ 153 | (?P-?[0-9.]+)[ \t]* 154 | \)[ \t]* 155 | \([ \t]* 156 | (?P-?[0-9.]+)[ \t]+ 157 | (?P-?[0-9.]+)[ \t]+ 158 | (?P-?[0-9.]+)[ \t]* 159 | \)[ \t]* 160 | \)[ \t]* 161 | (?P[^ \t]+)[ \t]+ 162 | (?P[0-9]+)[ \t]+ 163 | (?P[0-9]+)[ \t]+ 164 | (?P[0-9]+) 165 | [ \t]*$ 166 | """, re.VERBOSE) 167 | 168 | # q3 patch start 169 | # patchDef2 170 | q3patch_start_pattern = re.compile(r"^[ \t]*patchDef2[ \t]*$") 171 | 172 | # q3 patch material 173 | # somename 174 | q3patch_material_pattern = re.compile(r"^[ \t]*(?P[^ \t]+)[ \t]*$") 175 | 176 | # vertex matrix info 177 | # ( width height reserved0 reserved1 reserved2 ) 178 | q3patch_vertex_q3patch_matrix_info_pattern = re.compile(r""" 179 | ^[ \t]*\([ \t]* 180 | (?P[0-9]+)[ \t]+ 181 | (?P[0-9]+)[ \t]+ 182 | (?P[0-9]+)[ \t]+ 183 | (?P[0-9]+)[ \t]+ 184 | (?P[0-9]+)[ \t]* 185 | \)[ \t]*$ 186 | """, re.VERBOSE) 187 | 188 | # vertex matrix opening 189 | # ( 190 | q3patch_vertex_q3patch_matrix_opening_pattern = re.compile(r"^[ \t]*\([ \t]*$") 191 | 192 | # vertex line 193 | q3patch_vertex_line_pattern = re.compile(r""" 194 | ^[ \t]*\([ \t]* 195 | (?P\([ \t]*[ \t0-9.\(\)-]+[ \t]*\))[ \t]* 196 | \)[ \t]*$ 197 | """, re.VERBOSE) 198 | 199 | # vertex list 200 | q3patch_vertex_list_pattern = re.compile(r""" 201 | ^[ \t]*\([ \t]* 202 | (?P-?[0-9.]+)[ \t]* 203 | (?P-?[0-9.]+)[ \t]* 204 | (?P-?[0-9.]+)[ \t]* 205 | (?P-?[0-9.]+)[ \t]* 206 | (?P-?[0-9.]+)[ \t]* 207 | \)[ \t]* 208 | (?P\(?[ \t]*[ \t0-9().-]*[ \t]*\)?) 209 | [ \t]*$ 210 | """, re.VERBOSE) 211 | 212 | # vertex matrix ending 213 | # ) 214 | q3patch_vertex_q3patch_matrix_ending_pattern = re.compile(r"^[ \t]*\)[ \t]*$") 215 | 216 | # block ending 217 | # } 218 | block_ending_pattern = re.compile(r"^[ \t]*}[ \t]*$") 219 | 220 | entity_num = -1 221 | 222 | for line in map_lines: 223 | debug("Reading: " + line) 224 | 225 | # Empty lines 226 | if empty_line_pattern.match(line): 227 | debug("Empty line") 228 | continue 229 | 230 | # Comments 231 | if generic_comment_pattern.match(line): 232 | debug("Comment") 233 | continue 234 | 235 | # Entity start 236 | if not in_entity: 237 | match = block_opening_pattern.match(line) 238 | if match: 239 | entity_num += 1 240 | debug("Start Entity #" + str(entity_num)) 241 | self.entity_list.append(Entity()) 242 | in_entity = True 243 | shape_num = -1 244 | continue 245 | 246 | # In Entity 247 | 248 | if not in_q3brush and not start_q3bpbrush and not in_q3bpbrush and not start_q3patch and not in_q3patch: 249 | # We can only find KeyValue or Shape opening block at this point 250 | if not in_shape: 251 | # KeyValue pair 252 | match = keyvalue_pattern.match(line) 253 | if match: 254 | key = match.group("key") 255 | value = match.group("value") 256 | debug("KeyValue pair [“" + key + "”, “" + value + "”]") 257 | self.entity_list[-1].thing_list.append(KeyValue()) 258 | self.entity_list[-1].thing_list[-1].key = key 259 | self.entity_list[-1].thing_list[-1].value = value 260 | continue 261 | 262 | # Shape start 263 | match = block_opening_pattern.match(line) 264 | if match: 265 | shape_num += 1 266 | debug("Start Shape #" + str(shape_num)) 267 | in_shape = True 268 | continue 269 | 270 | # Brush/Patch start 271 | else: # in_shape 272 | if q3bpbrush_start_pattern.match(line): 273 | debug("Start Q3 BP Brush") 274 | self.entity_list[-1].thing_list.append(Q3BPBrush()) 275 | in_q3bpbrush = False 276 | start_q3bpbrush = True 277 | continue 278 | 279 | if q3patch_start_pattern.match(line): 280 | debug("Start Q3 Patch") 281 | self.entity_list[-1].thing_list.append(Q3Patch()) 282 | in_shape = False 283 | start_q3patch = True 284 | continue 285 | 286 | # if we are not a brush or patch, and not a ending brush or patch (ending shape) 287 | if not block_ending_pattern.match(line): 288 | debug("In Q3Brush") 289 | self.entity_list[-1].thing_list.append(Q3Brush()) 290 | in_shape = False 291 | in_q3brush = True 292 | # do not continue! this line must be read one more time! 293 | # this is brush content! 294 | 295 | # Q3 Patch opening 296 | if start_q3patch and not in_q3patch: 297 | if block_opening_pattern.match(line): 298 | debug("In Q3 Patch") 299 | start_q3patch = False 300 | in_q3patch = True 301 | in_q3patch_material = True 302 | continue 303 | 304 | # Q3 BP Brush opening 305 | if start_q3bpbrush and not in_q3bpbrush: 306 | if block_opening_pattern.match(line): 307 | debug("In Q3 BP Brush") 308 | start_q3bpbrush = False 309 | in_q3bpbrush = True 310 | continue 311 | 312 | # Q3Brush content 313 | if in_q3brush: 314 | 315 | # Plane content 316 | match = q3brush_plane_pattern.match(line) 317 | if match: 318 | debug("Add Plane to Q3Brush") 319 | plane = OrderedDict() 320 | plane["coord0_x"] = match.group("coord0_x") 321 | plane["coord0_y"] = match.group("coord0_y") 322 | plane["coord0_z"] = match.group("coord0_z") 323 | plane["coord1_x"] = match.group("coord1_x") 324 | plane["coord1_y"] = match.group("coord1_y") 325 | plane["coord1_z"] = match.group("coord1_z") 326 | plane["coord2_x"] = match.group("coord2_x") 327 | plane["coord2_y"] = match.group("coord2_y") 328 | plane["coord2_z"] = match.group("coord2_z") 329 | plane["material"] = match.group("material") 330 | plane["shift_x"] = match.group("shift_x") 331 | plane["shift_y"] = match.group("shift_y") 332 | plane["rotation"] = match.group("rotation") 333 | plane["scale_x"] = match.group("scale_x") 334 | plane["scale_y"] = match.group("scale_y") 335 | plane["flag_content"] = match.group("flag_content") 336 | plane["flag_surface"] = match.group("flag_surface") 337 | plane["value"] = match.group("value") 338 | 339 | self.entity_list[-1].thing_list[-1].plane_list.append(plane) 340 | continue 341 | 342 | # Q3Brush End 343 | if block_ending_pattern.match(line): 344 | debug("End Q3Brush") 345 | in_q3brush = False 346 | continue 347 | 348 | # Q3 BP Brush content 349 | if in_q3bpbrush: 350 | # Plane content 351 | match = q3bpbrush_plane_pattern.match(line) 352 | if match: 353 | plane = OrderedDict() 354 | plane["coord0_x"] = match.group("coord0_x") 355 | plane["coord0_y"] = match.group("coord0_y") 356 | plane["coord0_z"] = match.group("coord0_z") 357 | plane["coord1_x"] = match.group("coord1_x") 358 | plane["coord1_y"] = match.group("coord1_y") 359 | plane["coord1_z"] = match.group("coord1_z") 360 | plane["coord2_x"] = match.group("coord2_x") 361 | plane["coord2_y"] = match.group("coord2_y") 362 | plane["coord2_z"] = match.group("coord2_z") 363 | plane["texdef_xx"] = match.group("texdef_xx") 364 | plane["texdef_yx"] = match.group("texdef_yx") 365 | plane["texdef_tx"] = match.group("texdef_tx") 366 | plane["texdef_xy"] = match.group("texdef_xy") 367 | plane["texdef_yy"] = match.group("texdef_yy") 368 | plane["texdef_ty"] = match.group("texdef_ty") 369 | plane["material"] = match.group("material") 370 | plane["flag_content"] = match.group("flag_content") 371 | plane["flag_surface"] = match.group("flag_surface") 372 | plane["value"] = match.group("value") 373 | 374 | self.entity_list[-1].thing_list[-1].plane_list.append(plane) 375 | continue 376 | 377 | # Q3 BP Brush End 378 | if block_ending_pattern.match(line): 379 | debug("End Q3Brush") 380 | in_q3bpbrush = False 381 | in_shape = True 382 | continue 383 | 384 | # Q3 Patch content 385 | if in_q3patch: 386 | # Q3 Patch material 387 | if in_q3patch_material: 388 | match = q3patch_material_pattern.match(line) 389 | if match: 390 | debug("Add Material name to Q3 Patch") 391 | self.entity_list[-1].thing_list[-1].q3patch_material = match.group("material") 392 | in_q3patch_material = False 393 | in_q3patch_matrix_info = True 394 | continue 395 | 396 | if in_q3patch_matrix_info: 397 | match = q3patch_vertex_q3patch_matrix_info_pattern.match(line) 398 | if match: 399 | debug("Add Vertex matrix info to Q3 Patch") 400 | self.entity_list[-1].thing_list[-1].q3patch_vertex_q3patch_matrix_info["width"] = match.group("width") 401 | self.entity_list[-1].thing_list[-1].q3patch_vertex_q3patch_matrix_info["height"] = match.group("height") 402 | self.entity_list[-1].thing_list[-1].q3patch_vertex_q3patch_matrix_info["reserved0"] = match.group("reserved0") 403 | self.entity_list[-1].thing_list[-1].q3patch_vertex_q3patch_matrix_info["reserved1"] = match.group("reserved1") 404 | self.entity_list[-1].thing_list[-1].q3patch_vertex_q3patch_matrix_info["reserved2"] = match.group("reserved2") 405 | in_q3patch_matrix_info = False 406 | start_q3patch_matrix = True 407 | continue 408 | 409 | if start_q3patch_matrix: 410 | if q3patch_vertex_q3patch_matrix_opening_pattern.match(line): 411 | start_q3patch_matrix = False 412 | in_q3patch_matrix = True 413 | continue 414 | 415 | if in_q3patch_matrix: 416 | if not q3patch_vertex_q3patch_matrix_ending_pattern.match(line): 417 | match = q3patch_vertex_line_pattern.match(line) 418 | if match: 419 | debug("Add line to patch") 420 | q3patch_vertex_list = [] 421 | q3patch_vertex_list_string = match.group("q3patch_vertex_list") 422 | 423 | debug("Reading substring: " + q3patch_vertex_list_string) 424 | match = q3patch_vertex_list_pattern.match(q3patch_vertex_list_string) 425 | while match: 426 | debug("Add vertex to patch line") 427 | vertex = {} 428 | vertex["origin_x"] = match.group("origin_x") 429 | vertex["origin_y"] = match.group("origin_y") 430 | vertex["origin_z"] = match.group("origin_z") 431 | vertex["texcoord_x"] = match.group("texcoord_x") 432 | vertex["texcoord_y"] = match.group("texcoord_y") 433 | 434 | q3patch_vertex_list.append(vertex) 435 | 436 | remaining = match.group("remaining") 437 | debug("Reading substring: " + remaining) 438 | match = q3patch_vertex_list_pattern.match(remaining) 439 | 440 | self.entity_list[-1].thing_list[-1].q3patch_vertex_q3patch_matrix.append(q3patch_vertex_list) 441 | 442 | continue 443 | 444 | if q3patch_vertex_q3patch_matrix_ending_pattern.match(line): 445 | in_q3patch_matrix = False 446 | continue 447 | 448 | # Q3 Patch End 449 | if block_ending_pattern.match(line): 450 | debug("End Q3 Patch") 451 | in_q3patch = False 452 | in_shape = True 453 | continue 454 | 455 | if in_shape: 456 | # Shape End 457 | if block_ending_pattern.match(line): 458 | debug("End Shape") 459 | in_shape = False 460 | continue 461 | 462 | # Entity End 463 | if block_ending_pattern.match(line): 464 | debug("End Entity") 465 | in_entity = False 466 | continue 467 | 468 | # No match 469 | Ui.error("Unknown line: " + line) 470 | 471 | # an empty file is not an error 472 | 473 | def exportFile(self, bsp_entities_only=False): 474 | if self.entity_list == None: 475 | Ui.error("No map loaded") 476 | 477 | numbering_enabled = self.numbering_enabled and not bsp_entities_only 478 | 479 | map_string = "" 480 | model_count = 0 481 | 482 | for i in range(0, len(self.entity_list)): 483 | debug("Exporting Entity #" + str(i)) 484 | 485 | entity_string = "" 486 | 487 | if len(self.entity_list[i].thing_list) > 0: 488 | entity_printable = True 489 | has_classname = False 490 | has_shape = False; 491 | is_model = False; 492 | keyvalue_count = 0 493 | shape_count = 0 494 | 495 | thing_list = self.entity_list[i].thing_list 496 | 497 | if bsp_entities_only: 498 | thing_list.reverse() 499 | 500 | for thing in thing_list: 501 | if isinstance(thing, KeyValue): 502 | if bsp_entities_only: 503 | if thing.key == "classname": 504 | if thing.value in [ "func_group", "light", "misc_model" ]: 505 | entity_printable = False 506 | continue 507 | 508 | if thing.key == "classname": 509 | has_classname = True 510 | if has_classname and has_shape: 511 | is_model = True 512 | 513 | debug("Exporting KeyValue pair") 514 | 515 | entity_string += "\"" + thing.key + "\" \"" + thing.value + "\"" + "\n" 516 | keyvalue_count += 1 517 | 518 | continue 519 | 520 | if isinstance(thing, Shape): 521 | has_shape = True 522 | 523 | if has_classname and has_shape: 524 | is_model = True 525 | 526 | if bsp_entities_only: 527 | continue 528 | 529 | shape = thing 530 | 531 | if numbering_enabled: 532 | entity_string += "// brush " + str(shape_count) + "\n" 533 | 534 | entity_string += "{\n" 535 | debug("Exporting Shape #" + str(shape_count)) 536 | 537 | if isinstance(shape, Q3Brush): 538 | debug("Exporting Q3Brush") 539 | 540 | for plane in shape.plane_list: 541 | entity_string += "( " 542 | entity_string += plane["coord0_x"] + " " 543 | entity_string += plane["coord0_y"] + " " 544 | entity_string += plane["coord0_z"] + " " 545 | entity_string += ") " 546 | entity_string += "( " 547 | entity_string += plane["coord1_x"] + " " 548 | entity_string += plane["coord1_y"] + " " 549 | entity_string += plane["coord1_z"] + " " 550 | entity_string += ") " 551 | entity_string += "( " 552 | entity_string += plane["coord2_x"] + " " 553 | entity_string += plane["coord2_y"] + " " 554 | entity_string += plane["coord2_z"] + " " 555 | entity_string += ") " 556 | entity_string += plane["material"] + " " 557 | entity_string += plane["shift_x"] + " " 558 | entity_string += plane["shift_y"] + " " 559 | entity_string += plane["rotation"] + " " 560 | entity_string += plane["scale_x"] + " " 561 | entity_string += plane["scale_y"] + " " 562 | entity_string += plane["flag_content"] + " " 563 | entity_string += plane["flag_surface"] + " " 564 | entity_string += plane["value"] 565 | entity_string += "\n" 566 | 567 | elif isinstance(shape, Q3BPBrush): 568 | debug("Exporting Q3 BP Brush") 569 | entity_string += "brushDef\n" 570 | entity_string += "{\n" 571 | 572 | for plane in shape.plane_list: 573 | entity_string += "( " 574 | entity_string += plane["coord0_x"] + " " 575 | entity_string += plane["coord0_y"] + " " 576 | entity_string += plane["coord0_z"] + " " 577 | entity_string += ") " 578 | entity_string += "( " 579 | entity_string += plane["coord1_x"] + " " 580 | entity_string += plane["coord1_y"] + " " 581 | entity_string += plane["coord1_z"] + " " 582 | entity_string += ") " 583 | entity_string += "( " 584 | entity_string += plane["coord2_x"] + " " 585 | entity_string += plane["coord2_y"] + " " 586 | entity_string += plane["coord2_z"] + " " 587 | entity_string += ") " 588 | entity_string += "( " 589 | entity_string += "( " 590 | entity_string += plane["texdef_xx"] + " " 591 | entity_string += plane["texdef_yx"] + " " 592 | entity_string += plane["texdef_tx"] + " " 593 | entity_string += ") " 594 | entity_string += "( " 595 | entity_string += plane["texdef_xy"] + " " 596 | entity_string += plane["texdef_yy"] + " " 597 | entity_string += plane["texdef_ty"] + " " 598 | entity_string += ") " 599 | entity_string += ") " 600 | entity_string += plane["material"] + " " 601 | entity_string += plane["flag_content"] + " " 602 | entity_string += plane["flag_surface"] + " " 603 | entity_string += plane["value"] 604 | entity_string += "\n" 605 | 606 | entity_string += "}\n" 607 | 608 | elif isinstance(shape, Q3Patch): 609 | debug("Exporting Q3 Patch") 610 | entity_string += "patchDef2\n" 611 | entity_string += "{\n" 612 | entity_string += shape.q3patch_material + "\n" 613 | entity_string += "( " 614 | entity_string += shape.q3patch_vertex_q3patch_matrix_info["width"] + " " 615 | entity_string += shape.q3patch_vertex_q3patch_matrix_info["height"] + " " 616 | entity_string += shape.q3patch_vertex_q3patch_matrix_info["reserved0"] + " " 617 | entity_string += shape.q3patch_vertex_q3patch_matrix_info["reserved1"] + " " 618 | entity_string += shape.q3patch_vertex_q3patch_matrix_info["reserved2"] + " " 619 | entity_string += ")\n" 620 | entity_string += "(\n" 621 | 622 | for q3patch_vertex_line in shape.q3patch_vertex_q3patch_matrix: 623 | entity_string += "( " 624 | for vertex in q3patch_vertex_line: 625 | entity_string += "( " 626 | entity_string += vertex["origin_x"] + " " 627 | entity_string += vertex["origin_y"] + " " 628 | entity_string += vertex["origin_z"] + " " 629 | entity_string += vertex["texcoord_x"] + " " 630 | entity_string += vertex["texcoord_y"] + " " 631 | entity_string += ") " 632 | entity_string += ")\n" 633 | entity_string += ")\n" 634 | entity_string += "}\n" 635 | 636 | else: 637 | Ui.error("Unknown Entity Shape") 638 | return False 639 | 640 | entity_string += "}\n" 641 | shape_count += 1 642 | 643 | continue 644 | 645 | Ui.error("Unknown Entity Thing") 646 | 647 | if numbering_enabled: 648 | entity_string += "// entity " + str(i) + "\n" 649 | 650 | if entity_printable: 651 | map_string += "{\n" 652 | 653 | if is_model: 654 | if model_count > 0: 655 | map_string += "\"model\" \"*" + str(model_count) + "\"\n" 656 | 657 | model_count += 1 658 | 659 | map_string += entity_string 660 | map_string += "}\n" 661 | 662 | return map_string 663 | 664 | def writeFile(self, file_name): 665 | map_string = self.exportFile() 666 | if map_string: 667 | input_map_file = open(file_name, 'wb') 668 | input_map_file.write(str.encode(map_string)) 669 | input_map_file.close() 670 | 671 | def exportBspEntities(self): 672 | return self.exportFile(bsp_entities_only=True) 673 | 674 | def writeBspEntities(self, file_name): 675 | bsp_entities_string = self.exportBspEntities() 676 | if bsp_entities_string: 677 | bsp_entities_file = open(file_name, 'wb') 678 | bsp_entities_file.write(str.encode(bsp_entities_string)) 679 | bsp_entities_file.close() 680 | 681 | def substituteKeywords(self, substitution): 682 | if not self.entity_list: 683 | Ui.error("No map loaded") 684 | 685 | for entity in self.entity_list: 686 | entity.substituteKeys(substitution) 687 | entity.substituteValues(substitution) 688 | 689 | def lowerCaseFilePaths(self): 690 | for entity in self.entity_list: 691 | for thing in entity.thing_list: 692 | if isinstance(thing, KeyValue): 693 | if thing.key in [ "model", "targetShaderName" ] + q3_sound_keyword_list: 694 | thing.value = thing.value.lower() 695 | 696 | elif isinstance(thing, Q3Brush): 697 | for plane in thing.plane_list: 698 | plane["material"] = plane["material"].lower() 699 | 700 | elif isinstance(thing, Q3BPBrush): 701 | for plane in thing.plane_list: 702 | plane["material"] = plane["material"].lower() 703 | 704 | elif isinstance(thing, Q3Patch): 705 | thing.q3patch_material = thing.q3patch_material.lower() 706 | 707 | class Entity(): 708 | def __init__(self): 709 | self.thing_list = [] 710 | 711 | def substituteKeys(self, substitution): 712 | for old_key, new_key in substitution.key_dict.items(): 713 | # rename the key in place 714 | for thing in self.thing_list: 715 | if isinstance(thing, KeyValue): 716 | if str.lower(thing.key) == str.lower(old_key): 717 | thing.key = new_key 718 | 719 | def substituteValues(self, substitution): 720 | for old_value, new_value in substitution.value_dict.items(): 721 | for thing in self.thing_list: 722 | if isinstance(thing, KeyValue): 723 | if str.lower(thing.value) == str.lower(old_value): 724 | thing.value = new_value 725 | 726 | class KeyValue(): 727 | def __init__(self): 728 | self.key = "" 729 | self.value = "" 730 | 731 | class Shape(): 732 | pass 733 | 734 | class Q3Brush(Shape): 735 | def __init__(self): 736 | self.plane_list = [] 737 | 738 | class Q3BPBrush(Shape): 739 | def __init__(self): 740 | self.plane_list = [] 741 | 742 | class Q3Patch(Shape): 743 | def __init__(self): 744 | self.q3patch_material = None 745 | self.q3patch_vertex_q3patch_matrix_info = {} 746 | self.q3patch_vertex_q3patch_matrix = [] 747 | 748 | class KeyValueSubstitution(): 749 | def __init__(self): 750 | self.key_dict = {} 751 | self.value_dict = {} 752 | 753 | 754 | def readFile(self, file_name): 755 | substitution_file = open(file_name, "rb") 756 | 757 | if not substitution_file: 758 | Ui.error("failed to open file: " + file_name) 759 | 760 | substitution_bstring = substitution_file.read() 761 | substitution_file.close() 762 | 763 | substitution_pattern = re.compile(r""" 764 | ^[ \t]* 765 | (?Pkey|value)[ \t]*,[ \t]* 766 | "(?P[^\"]*)"[ \t]*,[ \t]* 767 | "(?P[^\"]*)"[ \t]*$ 768 | """, re.VERBOSE) 769 | 770 | substitution_lines = str.splitlines(bytes.decode(substitution_bstring)) 771 | 772 | for line in substitution_lines: 773 | debug("Reading line: " + line) 774 | match = substitution_pattern.match(line) 775 | if match: 776 | debug("Matched") 777 | value_type = match.group("value_type") 778 | old_value = match.group("old_value") 779 | new_value = match.group("new_value") 780 | if value_type == "key": 781 | debug("Add Key Substitution [ " + old_value + ", " + new_value + " ]") 782 | self.key_dict[old_value] = new_value 783 | elif value_type == "value": 784 | debug("Add Value Substitution [ " + old_value + ", " + new_value + " ]") 785 | self.value_dict[old_value] = new_value 786 | 787 | def add_arguments(parser): 788 | parser.add_argument("-im", "--input-map", dest="input_map_file", metavar="FILENAME", help="read from .map file %(metavar)s") 789 | parser.add_argument("-oe", "--output-bsp-entities", dest="output_bsp_entities", metavar="FILENAME", help="dump entities to .bsp entities format to .txt file %(metavar)s") 790 | parser.add_argument("-sk", "--substitute-keywords", dest="substitute_keywords", metavar="FILENAME", help="use entity keyword substitution rules from .csv file %(metavar)s") 791 | parser.add_argument("-Lf", "--lowercase-filepaths", dest="lowercase_filepaths", help="lowercase file paths", action="store_true") 792 | parser.add_argument("-dn", "--disable-numbering", dest="disable_numbering", help="disable entity and shape numbering", action="store_true") 793 | parser.add_argument("-om", "--output-map", dest="output_map_file", metavar="FILENAME", help="write to .map file %(metavar)s") 794 | 795 | def main(args=None): 796 | if not args: 797 | description="Esquirel bsp is a map parser for my lovely granger." 798 | parser = argparse.ArgumentParser(description=description) 799 | parser.add_argument("-D", "--debug", help="print debug information", action="store_true") 800 | add_arguments(parser) 801 | args = parser.parse_args() 802 | 803 | map = Map() 804 | 805 | if args.input_map_file: 806 | map.readFile(args.input_map_file) 807 | 808 | debug("File " + args.input_map_file + " read") 809 | 810 | if args.substitute_keywords: 811 | substitution = KeyValueSubstitution() 812 | substitution.readFile(args.substitute_keywords) 813 | map.substituteKeywords(substitution) 814 | 815 | if args.lowercase_filepaths: 816 | map.lowerCaseFilePaths() 817 | 818 | if args.disable_numbering: 819 | map.numbering_enabled = False 820 | 821 | if args.output_bsp_entities: 822 | map.writeBspEntities(args.output_bsp_entities) 823 | 824 | if args.output_map_file: 825 | map.writeFile(args.output_map_file) 826 | 827 | if __name__ == "__main__": 828 | main() 829 | -------------------------------------------------------------------------------- /Urcheon/MapCompiler.py: -------------------------------------------------------------------------------- 1 | #! /usr/bin/env python3 2 | #-*- coding: UTF-8 -*- 3 | 4 | ### Legal 5 | # 6 | # Author: Thomas DEBESSE 7 | # License: ISC 8 | # 9 | 10 | 11 | from Urcheon import Default 12 | from Urcheon import Parallelism 13 | from Urcheon import Profile 14 | from Urcheon import Repository 15 | from Urcheon import Ui 16 | from collections import OrderedDict 17 | import logging 18 | import re 19 | import shutil 20 | import subprocess 21 | import sys 22 | import tempfile 23 | import time 24 | import os 25 | import toml 26 | 27 | 28 | class Config(): 29 | def __init__(self, source_tree, map_path=None): 30 | self.source_dir = source_tree.dir 31 | self.game_name = source_tree.game_name 32 | 33 | self.profile_fs = Profile.Fs(self.source_dir) 34 | self.profile_dict = OrderedDict() 35 | 36 | self.default_profile = None 37 | self.keep_source = True 38 | self.q3map2_config = {} 39 | 40 | 41 | config_path = None 42 | 43 | # try loading map config first 44 | if map_path: 45 | map_base = os.path.splitext(os.path.basename(map_path))[0] 46 | map_config_path = os.path.join(Default.map_profile_dir, map_base + Default.map_profile_ext) 47 | 48 | if self.profile_fs.isFile(map_config_path): 49 | config_path = map_config_path 50 | 51 | # TODO: name config based on pak name instead of game name 52 | 53 | # if no map config, try loading game map config 54 | if not config_path and self.game_name: 55 | game_config_path = os.path.join(Default.map_profile_dir, self.game_name + Default.map_profile_ext) 56 | 57 | if self.profile_fs.isFile(game_config_path): 58 | config_path = game_config_path 59 | 60 | # if no map config and no game config, try loading default one 61 | if not config_path: 62 | default_config_path = os.path.join(Default.map_profile_dir, Default.default_base + Default.map_profile_ext) 63 | 64 | if self.profile_fs.isFile(default_config_path): 65 | config_path = default_config_path 66 | 67 | # well, if it occurs it means there is missing files installation dir 68 | if not config_path: 69 | Ui.error("missing map compiler config") 70 | 71 | self.readConfig(config_path) 72 | 73 | def readConfig(self, config_file_name, is_parent=False): 74 | config_path = self.profile_fs.getPath(config_file_name) 75 | 76 | logging.debug("reading map config: " + config_path) 77 | config_file = open(config_path, "r") 78 | config_dict = toml.load(config_file, _dict=OrderedDict) 79 | config_file.close() 80 | 81 | if "_init_" in config_dict.keys(): 82 | logging.debug("found “_init_” section in map profile: " + config_file_name) 83 | if "extend" in config_dict["_init_"].keys(): 84 | extend_game_name = config_dict["_init_"]["extend"] 85 | logging.debug("found “extend” instruction in “_init_” section: " + extend_game_name) 86 | logging.debug("loading parent game map config") 87 | 88 | if extend_game_name == "${game}": 89 | extend_game_name = self.game_name 90 | 91 | game_config_path = os.path.join(Default.map_profile_dir, extend_game_name + Default.map_profile_ext) 92 | self.readConfig(game_config_path, is_parent=True) 93 | 94 | if "default" in config_dict["_init_"].keys(): 95 | default = config_dict["_init_"]["default"] 96 | logging.debug("found “default” instruction in “_init_” section: " + default) 97 | self.default_profile = default 98 | 99 | if "source" in config_dict["_init_"].keys(): 100 | keep_source = config_dict["_init_"]["source"] 101 | logging.debug("found “source” instruction in “_init_” section: " + str(keep_source)) 102 | self.keep_source = keep_source 103 | 104 | del config_dict["_init_"] 105 | 106 | if "_q3map2_" in config_dict.keys(): 107 | for key in config_dict["_q3map2_"].keys(): 108 | value = config_dict["_q3map2_"][key] 109 | if value == "${game}": 110 | value = self.game_name 111 | self.q3map2_config[key] = value 112 | del config_dict["_q3map2_"] 113 | 114 | logging.debug("build profiles found: " + ", ".join(config_dict.keys())) 115 | 116 | for profile_name in config_dict.keys(): 117 | logging.debug("build profile found: " + profile_name) 118 | 119 | # overwrite parent profile 120 | self.profile_dict[profile_name] = OrderedDict() 121 | 122 | for build_stage in config_dict[profile_name].keys(): 123 | logging.debug("found build stage in “" + profile_name + "” profile: " + build_stage) 124 | 125 | profile_build_stage_dict = OrderedDict() 126 | config_stage_dict = config_dict[profile_name][build_stage] 127 | 128 | if "tool" in config_stage_dict.keys(): 129 | if isinstance(config_stage_dict["tool"], str): 130 | logging.debug("found tool, “" + build_stage + "” stage will run: " + config_stage_dict["tool"]) 131 | profile_build_stage_dict["tool"] = config_stage_dict["tool"] 132 | else: 133 | logging.error("in map build profile stage, \"tool\" key must be a string") 134 | else: 135 | logging.error("missing tool in “" + build_stage + "” stage in profile: " + profile_name) 136 | 137 | if "after" in config_stage_dict.keys(): 138 | if isinstance(config_stage_dict["after"], str): 139 | logging.debug("found prerequisite, stage “" + build_stage + "” must run after: " + config_stage_dict["after"]) 140 | profile_build_stage_dict["prerequisites"] = [config_stage_dict["after"]] 141 | elif isinstance(config_stage_dict["after"], list): 142 | logging.debug("found prerequisites, stage “" + build_stage + "” must run after: " + ", ".join(config_stage_dict["after"])) 143 | profile_build_stage_dict["prerequisites"] = config_stage_dict["after"] 144 | else: 145 | logging.error("in map build profile stage, \"after\" key must be a string or a list") 146 | else: 147 | profile_build_stage_dict["prerequisites"] = [] 148 | 149 | if "options" in config_stage_dict.keys(): 150 | if isinstance(config_stage_dict["options"], str): 151 | logging.debug("found options for “" + build_stage + "” stage: " + config_stage_dict["options"]) 152 | profile_build_stage_dict["options"] = config_stage_dict["options"].split(" ") 153 | else: 154 | logging.error("in map build profile stage, \"options\" key must be a string") 155 | else: 156 | profile_build_stage_dict["options"] = [] 157 | 158 | self.profile_dict[profile_name][build_stage] = profile_build_stage_dict 159 | 160 | default_prerequisite_dict = { 161 | "vis": ["bsp"], 162 | "light": ["vis"], 163 | "minimap": ["vis"], 164 | "nav": ["vis"], 165 | } 166 | 167 | for profile_name in self.profile_dict.keys(): 168 | # HACK: vis stage is optional 169 | if "vis" not in self.profile_dict[profile_name].keys() \ 170 | and "bsp" in self.profile_dict[profile_name].keys(): 171 | self.profile_dict[profile_name]["vis"] = { 172 | "tool": "dummy", 173 | "options": [], 174 | } 175 | 176 | # set default prerequisites 177 | for stage_name in self.profile_dict[profile_name].keys(): 178 | is_prerequisites_empty = False 179 | if "prerequisites" in self.profile_dict[profile_name][stage_name].keys(): 180 | if self.profile_dict[profile_name][stage_name]["prerequisites"] == []: 181 | is_prerequisites_empty = True 182 | else: 183 | # FIXME: isn't always set? 184 | self.profile_dict[profile_name][stage_name]["prerequisites"] = [] 185 | is_prerequisites_empty = True 186 | 187 | if is_prerequisites_empty: 188 | if stage_name in default_prerequisite_dict.keys(): 189 | for prerequisite_stage_name in default_prerequisite_dict[stage_name]: 190 | if stage_name in self.profile_dict[profile_name].keys(): 191 | self.profile_dict[profile_name][stage_name]["prerequisites"].append(prerequisite_stage_name) 192 | 193 | if is_parent: 194 | return 195 | 196 | 197 | def requireDefaultProfile(self): 198 | if not self.default_profile: 199 | Ui.error("no default map profile found, cannot compile map") 200 | return self.default_profile 201 | 202 | 203 | def printConfig(self): 204 | # TODO: order it? 205 | print(toml.dumps(self.profile_dict)) 206 | 207 | 208 | class Compiler(): 209 | def __init__(self, source_tree, map_profile=None, is_parallel=True): 210 | self.source_tree = source_tree 211 | self.source_dir = source_tree.dir 212 | self.map_profile = map_profile 213 | self.is_parallel = is_parallel 214 | 215 | if not map_profile: 216 | # TODO: test it 217 | map_config = Config(self.source_tree) 218 | map_profile = map_config.requireDefaultProfile() 219 | 220 | self.map_profile = map_profile 221 | 222 | # TODO: set something else for quiet and verbose mode 223 | self.subprocess_stdout = None 224 | self.subprocess_stderr = None 225 | 226 | 227 | def compile(self, map_path, build_prefix, stage_done=[]): 228 | self.map_path = map_path 229 | self.build_prefix = build_prefix 230 | stage_name = None 231 | stage_option_list = [] 232 | self.pakpath_list = [] 233 | 234 | tool_dict = { 235 | "q3map2": self.q3map2, 236 | "copy": self.copy, 237 | "dummy": self.dummy, 238 | } 239 | 240 | prt_handle, self.prt_path = tempfile.mkstemp(suffix="_q3map2" + os.path.extsep + "prt") 241 | srf_handle, self.srf_path = tempfile.mkstemp(suffix="_q3map2" + os.path.extsep + "srf") 242 | # close them since they will be written and read by another program 243 | os.close(prt_handle) 244 | os.close(srf_handle) 245 | 246 | logging.debug("building " + self.map_path + " to prefix: " + self.build_prefix) 247 | os.makedirs(self.build_prefix, exist_ok=True) 248 | 249 | self.map_config = Config(self.source_tree, map_path=map_path) 250 | self.pakpath_list = self.source_tree.pak_vfs.listPakPath() 251 | 252 | build_stage_dict = self.map_config.profile_dict[self.map_profile] 253 | 254 | # FIXME: if default profile is not set but map profile is set on command line 255 | # this fails on recursion (transient dir processing) 256 | if self.map_profile not in self.map_config.profile_dict.keys(): 257 | Ui.error("unknown map profile: " + self.map_profile) 258 | 259 | # list(…) because otherwise: 260 | # AttributeError: 'odict_keys' object has no attribute 'remove' 261 | stage_list = list(build_stage_dict.keys()) 262 | 263 | # remove from todo list 264 | # stages that are already marked as done 265 | # by actions like copy_bsp or merge_bsp 266 | for stage_name in stage_done: 267 | if stage_name in stage_list: 268 | stage_list.remove(stage_name) 269 | 270 | subprocess_dict = {} 271 | 272 | # loop until all the stages are done 273 | while stage_list != []: 274 | for stage_name in stage_list: 275 | # if stage started (ended or not), skip it 276 | if stage_name in subprocess_dict.keys(): 277 | # if stage ended, remove it from the todo list 278 | if not subprocess_dict[stage_name].is_alive(): 279 | # join dead thread to raise thread exceptions 280 | Parallelism.joinDeadThreads(list(subprocess_dict.values())) 281 | 282 | del subprocess_dict[stage_name] 283 | stage_list.remove(stage_name) 284 | continue 285 | 286 | logging.debug("found stage: " + stage_name) 287 | 288 | stage_option_list = build_stage_dict[stage_name] 289 | 290 | tool_name = stage_option_list["tool"] 291 | logging.debug("tool name: " + tool_name) 292 | 293 | if not tool_name in tool_dict: 294 | Ui.error("unknown tool name: " + tool_name) 295 | 296 | prerequisite_list = stage_option_list["prerequisites"] 297 | logging.debug("stage prerequisites: " + str(prerequisite_list)) 298 | 299 | # if there is at least one stage not done that is known as prerequisite, pass this stage 300 | if not set(stage_list).isdisjoint(prerequisite_list): 301 | continue 302 | 303 | # otherwise run the stage 304 | Ui.laconic("Building " + self.map_path + ", stage: " + stage_name) 305 | 306 | option_list = stage_option_list["options"] 307 | logging.debug("stage options: " + str(option_list)) 308 | 309 | if tool_name == "q3map2": 310 | option_list = ["-v"] + option_list 311 | # default game 312 | if not "-game" in option_list: 313 | if "game" in self.map_config.q3map2_config.keys(): 314 | option_list = ["-game", self.map_config.q3map2_config["game"]] + option_list 315 | 316 | subprocess_dict[stage_name] = Parallelism.Thread(target=tool_dict[tool_name], args=(option_list,)) 317 | subprocess_dict[stage_name].start() 318 | 319 | # wait for this task to finish if sequential build 320 | if not self.is_parallel: 321 | subprocess_dict[stage_name].join() 322 | 323 | # join dead thread to raise thread exceptions 324 | Parallelism.joinDeadThreads(list(subprocess_dict.values())) 325 | 326 | # no need to loop at full cpu speed 327 | time.sleep(.05) 328 | 329 | # when the last stage is running, find it and wait for it 330 | for stage_name in subprocess_dict.keys(): 331 | subprocess_dict[stage_name].join() 332 | 333 | if os.path.isfile(self.prt_path): 334 | os.remove(self.prt_path) 335 | 336 | if os.path.isfile(self.srf_path): 337 | os.remove(self.srf_path) 338 | 339 | 340 | def dummy(self, option_list): 341 | pass 342 | 343 | 344 | def q3map2(self, option_list, tool_name="q3map2"): 345 | map_base = os.path.splitext(os.path.basename(self.map_path))[0] 346 | lightmapdir_path = os.path.join(self.build_prefix, map_base) 347 | bsp_path = os.path.join(self.build_prefix, map_base + os.path.extsep + "bsp") 348 | 349 | # FIXME: needed for some advanced lightstyle (generated q3map2_ shader) 350 | # q3map2 is not able to create the “scripts/” directory itself 351 | scriptdir_path = os.path.realpath(os.path.join(self.build_prefix, "..", "scripts")) 352 | os.makedirs(scriptdir_path, exist_ok=True) 353 | 354 | thread_option_list = ["-threads", str(Parallelism.countCPU())] 355 | 356 | pakpath_option_list = ["-fs_nobasepath", "-fs_nohomepath", "-fs_nomagicpath"] 357 | 358 | # FIXME: is os.path.abspath() needed? 359 | pakpath_option_list += ["-fs_pakpath", self.source_dir] 360 | 361 | for pakpath in self.pakpath_list: 362 | # FIXME: is os.path.abspath() needed? 363 | pakpath_option_list += ["-fs_pakpath", pakpath] 364 | 365 | extended_option_list = [] 366 | 367 | # bsp stage is the one that calls -bsp, etc. 368 | for stage in ["bsp", "vis", "light", "minimap", "nav"]: 369 | if "-" + stage in option_list: 370 | stage_name = stage 371 | logging.debug("stage name: " + stage_name) 372 | 373 | if "-bsp" in option_list: 374 | extended_option_list = ["-prtfile", self.prt_path, "-srffile", self.srf_path, "-bspfile", bsp_path, "-leaktest", "-custinfoparms"] 375 | source_path = self.map_path 376 | elif "-vis" in option_list: 377 | extended_option_list = ["-prtfile", self.prt_path, "-saveprt"] 378 | source_path = bsp_path 379 | elif "-light" in option_list: 380 | extended_option_list = ["-srffile", self.srf_path, "-bspfile", bsp_path, "-lightmapdir", lightmapdir_path] 381 | source_path = self.map_path 382 | elif "-nav" in option_list: 383 | source_path = bsp_path 384 | elif "-minimap" in option_list: 385 | source_path = bsp_path 386 | else: 387 | extended_option_list = ["-prtfile", self.prt_path, "-srffile", self.srf_path, "-bspfile", bsp_path] 388 | # TODO: define the name somewhere 389 | Ui.warning("unknown q3map2 stage in command line, remind that -bsp is required by Urcheon: " + " ".join(option_list)) 390 | source_path = self.map_path 391 | 392 | command_list = [tool_name] + option_list + thread_option_list + pakpath_option_list + extended_option_list + [source_path] 393 | logging.debug("call list: " + str(command_list)) 394 | Ui.verbose("Build command: " + " ".join(command_list)) 395 | 396 | if subprocess.call(command_list, stdout=self.subprocess_stdout, stderr=self.subprocess_stderr) != 0: 397 | Ui.error("command failed: '" + "' '".join(command_list) + "'") 398 | 399 | # keep map source 400 | if "-bsp" in option_list and self.map_config.keep_source: 401 | self.copy([]) 402 | 403 | 404 | def copy(self, option_list): 405 | Ui.laconic("Copying map source: " + self.map_path) 406 | source_path = os.path.join(self.source_dir, self.map_path) 407 | if os.path.isfile(source_path): 408 | copy_path = os.path.join(self.build_prefix, os.path.basename(self.map_path)) 409 | shutil.copyfile(source_path, copy_path) 410 | shutil.copystat(source_path, copy_path) 411 | -------------------------------------------------------------------------------- /Urcheon/Pak.py: -------------------------------------------------------------------------------- 1 | #! /usr/bin/env python3 2 | #-*- coding: UTF-8 -*- 3 | 4 | ### Legal 5 | # 6 | # Author: Thomas DEBESSE 7 | # License: ISC 8 | # 9 | 10 | 11 | from Urcheon import Action 12 | from Urcheon import Default 13 | from Urcheon import FileSystem 14 | from Urcheon import Game 15 | from Urcheon import MapCompiler 16 | from Urcheon import Parallelism 17 | from Urcheon import Repository 18 | from Urcheon import Ui 19 | import __main__ as m 20 | import argparse 21 | import logging 22 | import os 23 | import sys 24 | import tempfile 25 | import time 26 | import zipfile 27 | from collections import OrderedDict 28 | from datetime import datetime 29 | from operator import attrgetter 30 | 31 | 32 | class MultiRunner(): 33 | def __init__(self, source_dir_list, args): 34 | self.source_dir_list = source_dir_list 35 | self.args = args 36 | 37 | self.runner_dict = { 38 | "prepare": Builder, 39 | "build": Builder, 40 | "package": Packager, 41 | } 42 | 43 | def run(self): 44 | cpu_count = Parallelism.countCPU() 45 | runner_thread_list = [] 46 | 47 | for source_dir in self.source_dir_list: 48 | # FIXME: because of this code Urcheon must run within package set directory 49 | Ui.notice(self.args.stage_name + " from: " + source_dir) 50 | source_dir = os.path.realpath(source_dir) 51 | 52 | source_tree = Repository.Tree(source_dir, game_name=self.args.game_name) 53 | 54 | runner = self.runner_dict[self.args.stage_name](source_tree, self.args) 55 | 56 | if self.args.no_parallel: 57 | runner.run() 58 | else: 59 | runner_thread = Parallelism.Thread(target=runner.run) 60 | runner_thread_list.append(runner_thread) 61 | 62 | while len(runner_thread_list) > cpu_count: 63 | # join dead thread early to raise thread exceptions early 64 | # forget ended threads 65 | runner_thread_list = Parallelism.joinDeadThreads(runner_thread_list) 66 | 67 | runner_thread.start() 68 | 69 | # wait for all remaining threads ending 70 | Parallelism.joinThreads(runner_thread_list) 71 | 72 | 73 | class Builder(): 74 | def __init__(self, source_tree, args, is_nested=False, disabled_action_list=[], file_list=[]): 75 | 76 | self.source_tree = source_tree 77 | self.source_dir = source_tree.dir 78 | self.pak_name = source_tree.pak_name 79 | self.pak_format = source_tree.pak_format 80 | self.game_name = source_tree.game_name 81 | self.is_nested = is_nested 82 | 83 | self.stage_name = args.stage_name 84 | 85 | if is_nested: 86 | self.keep_dust = False 87 | else: 88 | self.keep_dust = args.keep_dust 89 | 90 | action_list = Action.List(source_tree, self.stage_name, disabled_action_list=disabled_action_list) 91 | 92 | if self.stage_name == "prepare": 93 | self.test_dir = self.source_dir 94 | 95 | self.since_reference = None 96 | self.no_auto_actions = False 97 | self.clean_map = False 98 | self.map_profile = None 99 | 100 | # FIXME: currently the prepare stage 101 | # can't be parallel (for example SlothRun task 102 | # needs all PrevRun tasks to be finished first) 103 | # btw all packages can be prepared in parallel 104 | self.is_parallel = False 105 | else: 106 | if is_nested: 107 | self.test_dir = args.test_dir 108 | else: 109 | self.test_dir = source_tree.pak_config.getTestDir(args) 110 | 111 | if is_nested: 112 | self.since_reference = False 113 | self.no_auto_actions = False 114 | self.clean_map = False 115 | self.map_profile = None 116 | 117 | self.is_parallel = not args.no_parallel 118 | else: 119 | self.since_reference = args.since_reference 120 | self.no_auto_actions = args.no_auto_actions 121 | self.clean_map = args.clean_map 122 | self.map_profile = args.map_profile 123 | self.is_parallel = not args.no_parallel 124 | 125 | if self.pak_format == "dpk": 126 | self.deleted = Repository.Deleted(self.source_tree, self.test_dir, self.stage_name) 127 | self.deps = Repository.Deps(self.source_tree, self.test_dir) 128 | 129 | if not is_nested: 130 | if self.pak_format == "dpk": 131 | deleted_action_list = self.deleted.getActions() 132 | action_list.readActions(action_list=deleted_action_list) 133 | 134 | action_list.readActions() 135 | 136 | if not file_list: 137 | # FIXME: only if one package? 138 | # same reference for multiple packages 139 | # makes sense when using tags 140 | 141 | # NOTE: already prepared file can be seen as source again, but there may be no easy way to solve it 142 | if self.since_reference: 143 | file_repo = Repository.Git(self.source_dir, self.pak_format) 144 | file_list = file_repo.listFilesSinceReference(self.since_reference) 145 | 146 | # also look for untracked files 147 | untracked_file_list = file_repo.listUntrackedFiles() 148 | for file_name in untracked_file_list: 149 | if file_name not in file_list: 150 | logging.debug("found untracked file “" + file_name + "”") 151 | # FIXME: next loop will look for prepared files for it, which makes no sense, 152 | # is it harmful? 153 | file_list.append(file_name) 154 | 155 | # also look for files produced with “prepare” command 156 | # from files modified since this reference 157 | paktrace = Repository.Paktrace(source_tree, self.source_dir) 158 | input_file_dict = paktrace.getFileDict()["input"] 159 | for file_path in file_list: 160 | logging.debug("looking for prepared files for “" + str(file_path) + "”") 161 | logging.debug("looking for prepared files for “" + file_path + "”") 162 | if file_path in input_file_dict.keys(): 163 | for input_file_path in input_file_dict[file_path]: 164 | if not os.path.exists(os.path.join(self.source_dir, input_file_path)): 165 | logging.debug("missing prepared files for “" + file_path + "”: " + input_file_path) 166 | else: 167 | logging.debug("found prepared files for “" + file_path + "”: " + input_file_path) 168 | file_list.append(input_file_path) 169 | else: 170 | file_list = source_tree.listFiles() 171 | 172 | if not self.no_auto_actions: 173 | action_list.computeActions(file_list) 174 | 175 | self.action_list = action_list 176 | 177 | self.game_profile = Game.Game(source_tree) 178 | 179 | if not self.map_profile: 180 | map_config = MapCompiler.Config(source_tree) 181 | self.map_profile = map_config.requireDefaultProfile() 182 | 183 | 184 | def run(self): 185 | if self.source_dir == self.test_dir: 186 | Ui.print("Preparing: " + self.source_dir) 187 | else: 188 | Ui.print("Building “" + self.source_dir + "” as: " + self.test_dir) 189 | 190 | # TODO: check if not a directory 191 | if os.path.isdir(self.test_dir): 192 | logging.debug("found build dir: " + self.test_dir) 193 | else: 194 | logging.debug("create build dir: " + self.test_dir) 195 | os.makedirs(self.test_dir, exist_ok=True) 196 | 197 | if not self.is_nested and not self.keep_dust: 198 | clean_dust = True 199 | else: 200 | clean_dust = False 201 | 202 | if clean_dust: 203 | # do not read paktrace from temporary directories 204 | # do not read paktrace if dust will be kept 205 | paktrace = Repository.Paktrace(self.source_tree, self.test_dir) 206 | previous_file_list = paktrace.listAll() 207 | 208 | if self.clean_map or clean_dust: 209 | cleaner = Cleaner(self.source_tree) 210 | 211 | if self.clean_map: 212 | cleaner.cleanMap(self.test_dir) 213 | 214 | cpu_count = Parallelism.countCPU() 215 | action_thread_list = [] 216 | produced_unit_list = [] 217 | 218 | main_process = Parallelism.getProcess() 219 | 220 | for action_type in Action.list(): 221 | for file_path in self.action_list.active_action_dict[action_type.keyword]: 222 | # no need to use multiprocessing module to manage task contention, since each task will call its own process 223 | # using threads on one core is faster, and it does not prevent tasks to be able to use other cores 224 | 225 | # the is_nested argument is there to tell action to not do specific stuff because of recursion 226 | action = action_type(self.source_tree, self.test_dir, file_path, self.stage_name, map_profile=self.map_profile, is_nested=self.is_nested) 227 | 228 | # check if task is already done (usually comparing timestamps the make way) 229 | if action.isDone(): 230 | produced_unit_list.extend(action.getOldProducedUnitList()) 231 | continue 232 | 233 | if not self.is_parallel or not action_type.is_parallel: 234 | # tasks are run sequentially but they can 235 | # use multiple threads themselves 236 | thread_count = cpu_count 237 | else: 238 | # this compute is super slow because of process.children() 239 | child_thread_count = Parallelism.countChildThread(main_process) 240 | thread_count = max(1, cpu_count - child_thread_count) 241 | 242 | action.thread_count = thread_count 243 | 244 | if not self.is_parallel or not action_type.is_parallel: 245 | # sequential build explicitely requested (like in recursion) 246 | # or action that can't be run concurrently to others (like MergeBsp) 247 | produced_unit_list.extend(action.run()) 248 | else: 249 | # do not use >= in case of there is some extra thread we don't think about 250 | # it's better to spawn an extra one than looping forever 251 | while child_thread_count > cpu_count: 252 | # no need to loop at full cpu speed 253 | time.sleep(.05) 254 | child_thread_count = Parallelism.countChildThread(main_process) 255 | pass 256 | 257 | # join dead thread early to raise thread exceptions early 258 | # forget ended threads 259 | action_thread_list = Parallelism.joinDeadThreads(action_thread_list) 260 | 261 | action.thread_count = max(2, cpu_count - child_thread_count) 262 | 263 | # wrapper does: produced_unit_list.extend(action.run()) 264 | action_thread = Parallelism.Thread(target=self.threadExtendRes, args=(action.run, (), produced_unit_list)) 265 | action_thread_list.append(action_thread) 266 | action_thread.start() 267 | 268 | # join dead thread early to raise thread exceptions early 269 | # forget ended threads 270 | action_thread_list = Parallelism.joinDeadThreads(action_thread_list) 271 | 272 | # wait for all threads to end, otherwise it will start packaging next 273 | # package while the building task for the current one is not ended 274 | # and well, we now have to read that list to purge old files, so we 275 | # must wait 276 | Parallelism.joinThreads(action_thread_list) 277 | 278 | # Handle symbolic links. 279 | for action_type in Action.list(): 280 | for file_path in self.action_list.active_action_dict[action_type.keyword]: 281 | action = action_type(self.source_tree, self.test_dir, file_path, self.stage_name, action_list=self.action_list, map_profile=self.map_profile, is_nested=self.is_nested) 282 | 283 | # TODO: check for symbolic link to missing or deleted files. 284 | produced_unit_list.extend(action.symlink()) 285 | 286 | # deduplication 287 | unit_list = [] 288 | deleted_file_list = [] 289 | produced_file_list = [] 290 | for unit in produced_unit_list: 291 | if unit == []: 292 | continue 293 | 294 | logging.debug("unit: " + str(unit)) 295 | head = unit["head"] 296 | body = unit["body"] 297 | action = unit["action"] 298 | 299 | if action == "ignore": 300 | continue 301 | 302 | if action == "delete": 303 | deleted_file_list.append( head ) 304 | 305 | if head not in produced_file_list: 306 | produced_file_list.append(head) 307 | 308 | for part in body: 309 | if part not in produced_file_list: 310 | # FIXME: only if action was not “ignore” 311 | produced_file_list.append(part) 312 | 313 | # if multiple calls produce the same files (like merge_bsp) 314 | # FIXME: that can't work, this is probably a leftover 315 | # or we may have to do “if head in body” instead. 316 | # See https://github.com/DaemonEngine/Urcheon/issues/48 317 | if head in unit: 318 | continue 319 | 320 | unit_list.append(unit) 321 | 322 | produced_unit_list = unit_list 323 | 324 | if self.stage_name == "build" and not self.is_nested: 325 | if self.pak_format == "dpk": 326 | is_deleted = False 327 | 328 | if self.since_reference: 329 | Ui.laconic("looking for deleted files") 330 | # Unvanquished game did not support DELETED file until after 0.52.1. 331 | workaround_no_delete = self.source_tree.game_name == "unvanquished" and self.since_reference in ["unvanquished/0.52.1", "v0.52.1"] 332 | 333 | git_repo = Repository.Git(self.source_dir, "dpk", workaround_no_delete=workaround_no_delete) 334 | 335 | previous_version = git_repo.computeVersion(self.since_reference, named_reference=True) 336 | self.deps.set(self.pak_name, previous_version) 337 | 338 | for deleted_file in git_repo.getDeletedFileList(self.since_reference): 339 | if deleted_file not in deleted_file_list: 340 | is_deleted = True 341 | deleted_file_list.append(deleted_file) 342 | 343 | if deleted_file_list: 344 | is_deleted = True 345 | for deleted_file in deleted_file_list: 346 | self.deleted.set(self.pak_name, deleted_file) 347 | 348 | if self.deleted.read(): 349 | is_deleted = True 350 | 351 | if is_deleted: 352 | deleted_part_list = self.deleted.translate() 353 | 354 | # TODO: No need to mark as DELETED a file from the same 355 | # package if it does not depend on itself. 356 | # TODO: A way to not translate DELETED files may be needed 357 | # in some cases. 358 | 359 | # If flamer.jpg producing flamer.crn was replaced 360 | # by flamer.png also producing flamer.crn, the 361 | # flamer.crn file will be listed as deleted 362 | # while it will be shipped, but built from another 363 | # source file, so we must check deleted files 364 | # aren't built in other way to avoid listing 365 | # as deleted a file that is actually shipped. 366 | for deleted_part_dict in deleted_part_list: 367 | deleted_part = deleted_part_dict["file_path"] 368 | 369 | is_built = False 370 | 371 | if deleted_part_dict["pak_name"] == self.pak_name: 372 | if deleted_part.startswith(Default.repository_config_dir + os.path.sep): 373 | continue 374 | 375 | if deleted_part.startswith(Default.legacy_pakinfo_dir + os.path.sep): 376 | continue 377 | 378 | if deleted_part in produced_file_list: 379 | is_built = True 380 | Ui.laconic(deleted_part + ": do nothing because it is produced by another source file.") 381 | self.deleted.removePart(self.pak_name, deleted_part) 382 | 383 | if not is_built: 384 | Ui.laconic(deleted_part + ": will mark as deleted.") 385 | 386 | # Writing DELETED file. 387 | for deleted_part in deleted_part_list: 388 | self.deleted.set(self.source_tree.pak_name, deleted_part) 389 | 390 | is_deleted = self.deleted.write() 391 | 392 | if is_deleted: 393 | unit = { 394 | "head": "DELETED", 395 | "body": [ "DELETED" ], 396 | } 397 | 398 | produced_unit_list.append(unit) 399 | else: 400 | # Remove DELETED leftover from partial build. 401 | self.deps.remove(self.test_dir) 402 | 403 | is_deps = False 404 | 405 | # add itself to DEPS if partial build, 406 | # also look for deleted files 407 | if self.since_reference: 408 | is_deps = True 409 | 410 | if self.deps.read(): 411 | is_deps = True 412 | 413 | if is_deps: 414 | # translating DEPS file 415 | self.deps.translateTest() 416 | self.deps.write() 417 | 418 | unit = { 419 | "head": "DEPS", 420 | "body": [ "DEPS" ], 421 | } 422 | 423 | produced_unit_list.append(unit) 424 | else: 425 | # Remove DEPS leftover from partial build. 426 | self.deps.remove(self.test_dir) 427 | 428 | logging.debug("produced unit list:" + str(produced_unit_list)) 429 | 430 | # do not clean-up if building from temporary directories 431 | # or if user asked to not clean-up 432 | if clean_dust: 433 | cleaner.cleanDust(self.test_dir, produced_unit_list, previous_file_list) 434 | 435 | return produced_unit_list 436 | 437 | def threadExtendRes(self, func, args, res): 438 | # magic: only works if res is a mutable object (like a list) 439 | res.extend(func(*args)) 440 | 441 | 442 | class Packager(): 443 | # TODO: reuse paktraces, do not walk for file,s 444 | def __init__(self, source_tree, args): 445 | self.source_tree = source_tree 446 | 447 | self.source_dir = source_tree.dir 448 | self.pak_vfs = source_tree.pak_vfs 449 | self.pak_config = source_tree.pak_config 450 | self.pak_format = source_tree.pak_format 451 | 452 | self.allow_dirty = args.allow_dirty 453 | self.no_compress = args.no_compress 454 | self.merge_dir = args.merge_dir 455 | 456 | self.test_dir = self.pak_config.getTestDir(args) 457 | self.pak_file = self.pak_config.getPakFile(args) 458 | 459 | self.temp_pak_file = self.getTempPakFile() 460 | 461 | self.game_profile = Game.Game(source_tree) 462 | 463 | if self.pak_format == "dpk": 464 | self.deleted = Repository.Deleted(source_tree, self.test_dir, None) 465 | self.deps = Repository.Deps(source_tree, self.test_dir) 466 | 467 | def getTempPakFile(self): 468 | # TODO: add a specific cleaning function for those files 469 | # to clean them from older builds if something aborted packaging 470 | temp_pak_file_dirname = os.path.dirname(self.pak_file) 471 | temp_pak_file_basename = "." + os.path.basename(self.pak_file) + ".temp" 472 | return os.path.join(temp_pak_file_dirname, temp_pak_file_basename) 473 | 474 | def createSubdirs(self, pak_file): 475 | pak_subdir = os.path.dirname(pak_file) 476 | if pak_subdir == "": 477 | pak_subdir = "." 478 | 479 | if os.path.isdir(pak_subdir): 480 | logging.debug("found pak subdir: " + pak_subdir) 481 | else: 482 | logging.debug("create pak subdir: " + pak_subdir) 483 | os.makedirs(pak_subdir, exist_ok=True) 484 | 485 | def addToPak(self, pak_zipfile, full_path, file_path): 486 | # TODO: add a mechanism to know if VFS supports 487 | # symbolic links in packages or not. 488 | # Dæmon's DPK VFS is supporting symbolic links. 489 | # DarkPlaces' PK3 VFS is supporting symbolic links. 490 | # Others may not. 491 | is_symlink_supported = True 492 | if is_symlink_supported and os.path.islink(full_path): 493 | Ui.print("add symlink to package " + os.path.basename(self.pak_file) + ": " + file_path) 494 | 495 | # TODO: Remove this test when Urcheon deletes extra 496 | # files in build directory. Currently a deleted but not 497 | # committed file is kept. 498 | if os.path.exists(full_path): 499 | # FIXME: getmtime reads realpath datetime, not symbolic link datetime. 500 | file_date_time = (datetime.fromtimestamp(os.path.getmtime(full_path))) 501 | 502 | # See https://stackoverflow.com/a/61795576/9131399 503 | attrs = ('year', 'month', 'day', 'hour', 'minute', 'second') 504 | file_date_time_tuple = attrgetter(*attrs)(file_date_time) 505 | 506 | # See https://stackoverflow.com/a/60691331/9131399 507 | zip_info = zipfile.ZipInfo(file_path, date_time=file_date_time_tuple) 508 | zip_info.create_system = 3 509 | 510 | file_permissions = 0o777 511 | file_permissions |= 0xA000 512 | zip_info.external_attr = file_permissions << 16 513 | 514 | target_path = os.readlink(full_path) 515 | pak_zipfile.writestr(zip_info, target_path) 516 | else: 517 | Ui.print("add file to package " + os.path.basename(self.pak_file) + ": " + file_path) 518 | pak_zipfile.write(full_path, arcname=file_path) 519 | 520 | def run(self): 521 | if not os.path.isdir(self.test_dir): 522 | Ui.error("test pakdir not built: " + self.test_dir) 523 | 524 | source_repository = Repository.Git(self.source_dir, self.pak_format) 525 | if source_repository.isGit() and source_repository.isDirty(): 526 | if self.allow_dirty: 527 | Ui.warning("Dirty repository: " + self.source_dir) 528 | else: 529 | Ui.error("Dirty repository isn't allowed to be packaged (use --allow-dirty to override): " + self.source_dir, silent=True) 530 | 531 | Ui.print("Packaging “" + self.test_dir + "” as: " + self.pak_file) 532 | 533 | self.createSubdirs(self.pak_file) 534 | logging.debug("opening: " + self.temp_pak_file) 535 | 536 | if self.no_compress: 537 | # why zlib.Z_NO_COMPRESSION not defined? 538 | zipfile.zlib.Z_DEFAULT_COMPRESSION = 0 539 | else: 540 | # maximum compression 541 | zipfile.zlib.Z_DEFAULT_COMPRESSION = zipfile.zlib.Z_BEST_COMPRESSION 542 | 543 | paktrace_dir = Default.getPakTraceDir(self.test_dir) 544 | relative_paktrace_dir = os.path.relpath(paktrace_dir, self.test_dir) 545 | 546 | paktrace = Repository.Paktrace(self.source_tree, self.test_dir) 547 | built_file_list = paktrace.listAll() 548 | 549 | for dir_name, subdir_name_list, file_name_list in os.walk(self.test_dir): 550 | for file_name in file_name_list: 551 | rel_dir_name = os.path.relpath(dir_name, self.test_dir) 552 | 553 | full_path = os.path.join(dir_name, file_name) 554 | file_path = os.path.relpath(full_path, self.test_dir) 555 | 556 | # ignore paktrace files 557 | if file_path.startswith(relative_paktrace_dir + os.path.sep): 558 | continue 559 | 560 | # ignore DELETED and DEPS file, will add it later 561 | if self.pak_format == "dpk" and file_path in Repository.dpk_special_files: 562 | continue 563 | 564 | if file_path not in built_file_list: 565 | Ui.warning("extraneous file, will not package: " + file_path) 566 | 567 | test_file_list = [] 568 | 569 | for file_path in built_file_list: 570 | full_path = os.path.join(self.test_dir, file_path) 571 | 572 | if not os.path.exists(full_path): 573 | Ui.error("Missing " + full_path) 574 | 575 | file_dict = { 576 | "full_path": full_path, 577 | "file_path": file_path, 578 | } 579 | 580 | test_file_list.append(file_dict) 581 | 582 | merge_file_list = [] 583 | 584 | if self.merge_dir: 585 | for dir_name, subdir_name_list, file_name_list in os.walk(self.merge_dir): 586 | for file_name in file_name_list: 587 | full_path = os.path.join(dir_name, file_name) 588 | file_path = os.path.relpath(full_path, self.merge_dir) 589 | 590 | # unsupported paktrace files 591 | if file_path.startswith(relative_paktrace_dir + os.path.sep): 592 | Ui.error("Merging urcheon-built dpkdir is not supported", silent=True) 593 | 594 | # unsupported DELETED and DEPS file 595 | if self.pak_format == "dpk" and file_path in Repository.dpk_special_files: 596 | Ui.error("Merging urcheon-built dpkdir is not supported", silent=True) 597 | 598 | file_dict = { 599 | "full_path": full_path, 600 | "file_path": file_path, 601 | } 602 | 603 | merge_file_list.append(file_dict) 604 | 605 | # FIXME: if only the DEPS file is modified, the package will 606 | # not be created (it should be). 607 | if not test_file_list and not merge_file_list: 608 | Ui.print("Not writing empty package: " + self.pak_file) 609 | return 610 | 611 | pak_zipfile = zipfile.ZipFile(self.temp_pak_file, "w", zipfile.ZIP_DEFLATED) 612 | 613 | for file_dict in test_file_list: 614 | self.addToPak(pak_zipfile, file_dict["full_path"], file_dict["file_path"]) 615 | 616 | if self.merge_dir: 617 | Ui.print("Merging " + self.merge_dir + " directory") 618 | for file_dict in merge_file_list: 619 | self.addToPak(pak_zipfile, file_dict["full_path"], file_dict["file_path"]) 620 | 621 | if self.pak_format == "dpk": 622 | # Writing DELETED file. 623 | deleted_file_path = self.deleted.get_test_path() 624 | if os.path.isfile(deleted_file_path): 625 | pak_zipfile.write(deleted_file_path, arcname="DELETED") 626 | 627 | # Translating DEPS file. 628 | if self.deps.read(deps_dir=self.test_dir): 629 | self.deps.translateRelease(self.pak_vfs) 630 | 631 | deps_temp_dir = tempfile.mkdtemp() 632 | deps_temp_file = self.deps.write(deps_dir=deps_temp_dir) 633 | Ui.print("add file to package " + os.path.basename(self.pak_file) + ": DEPS") 634 | pak_zipfile.write(deps_temp_file, arcname="DEPS") 635 | 636 | logging.debug("close: " + self.temp_pak_file) 637 | pak_zipfile.close() 638 | 639 | if source_repository.isGit(): 640 | repo_date = int(source_repository.getDate("HEAD")) 641 | os.utime(self.temp_pak_file, (repo_date, repo_date)) 642 | 643 | # remove existing file (do not write in place) to force the game engine to reread the file 644 | # maybe the renaming of the temp file already makes sure the file is not the same one anyway 645 | if os.path.isfile(self.pak_file): 646 | logging.debug("remove existing package: " + self.pak_file) 647 | os.remove(self.pak_file) 648 | 649 | logging.debug("Renaming “" + self.temp_pak_file +"” as: " + self.pak_file) 650 | 651 | os.rename(self.temp_pak_file, self.pak_file) 652 | 653 | Ui.laconic("Package written: " + self.pak_file) 654 | 655 | 656 | class Cleaner(): 657 | def __init__(self, source_tree): 658 | 659 | self.pak_name = source_tree.pak_name 660 | 661 | self.game_profile = Game.Game(source_tree) 662 | 663 | 664 | def cleanTest(self, test_dir): 665 | for dir_name, subdir_name_list, file_name_list in os.walk(test_dir): 666 | for file_name in file_name_list: 667 | that_file = os.path.join(dir_name, file_name) 668 | Ui.laconic("clean: " + that_file) 669 | os.remove(that_file) 670 | FileSystem.removeEmptyDir(dir_name) 671 | for dir_name in subdir_name_list: 672 | that_dir = dir_name + os.path.sep + dir_name 673 | FileSystem.removeEmptyDir(that_dir) 674 | FileSystem.removeEmptyDir(dir_name) 675 | FileSystem.removeEmptyDir(test_dir) 676 | 677 | 678 | def cleanPak(self, install_dir): 679 | for dir_name, subdir_name_list, file_name_list in os.walk(install_dir): 680 | for file_name in file_name_list: 681 | if file_name.startswith(self.pak_name) and file_name.endswith(self.game_profile.pak_ext): 682 | pak_file = os.path.join(dir_name, file_name) 683 | Ui.laconic("clean: " + pak_file) 684 | os.remove(pak_file) 685 | FileSystem.removeEmptyDir(dir_name) 686 | FileSystem.removeEmptyDir(install_dir) 687 | 688 | 689 | def cleanMap(self, test_dir): 690 | # TODO: use paktrace abilities? 691 | for dir_name, subdir_name_list, file_name_list in os.walk(test_dir): 692 | for file_name in file_name_list: 693 | if dir_name.split("/")[-1:] == ["maps"] and file_name.endswith(os.path.extsep + "bsp"): 694 | bsp_file = os.path.join(dir_name, file_name) 695 | Ui.laconic("clean: " + bsp_file) 696 | os.remove(bsp_file) 697 | FileSystem.removeEmptyDir(dir_name) 698 | 699 | if dir_name.split("/")[-1:] == ["maps"] and file_name.endswith(os.path.extsep + "map"): 700 | map_file = os.path.join(dir_name, file_name) 701 | Ui.laconic("clean: " + map_file) 702 | os.remove(map_file) 703 | FileSystem.removeEmptyDir(dir_name) 704 | 705 | if dir_name.split("/")[-2:-1] == ["maps"] and file_name.startswith("lm_"): 706 | lightmap_file = os.path.join(dir_name, file_name) 707 | Ui.laconic("clean: " + lightmap_file) 708 | os.remove(lightmap_file) 709 | FileSystem.removeEmptyDir(dir_name) 710 | 711 | if dir_name.split("/")[-1:] == ["maps"] and file_name.endswith(os.path.extsep + "navMesh"): 712 | navmesh_file = os.path.join(dir_name, file_name) 713 | Ui.laconic("clean: " + navmesh_file) 714 | os.remove(navmesh_file) 715 | FileSystem.removeEmptyDir(dir_name) 716 | 717 | if dir_name.split("/")[-1:] == ["minimaps"]: 718 | minimap_file = os.path.join(dir_name, file_name) 719 | Ui.laconic("clean: " + minimap_file) 720 | os.remove(minimap_file) 721 | FileSystem.removeEmptyDir(dir_name) 722 | 723 | FileSystem.removeEmptyDir(test_dir) 724 | 725 | 726 | def cleanDust(self, test_dir, produced_unit_list, previous_file_list): 727 | # TODO: remove extra files that are not tracked in paktraces? 728 | # FIXME: reuse produced_file_list from build() 729 | produced_file_list = [] 730 | head_list = [] 731 | for unit in produced_unit_list: 732 | head_list.append(unit["head"]) 733 | produced_file_list.extend(unit["body"]) 734 | 735 | for file_name in previous_file_list: 736 | if file_name not in produced_file_list: 737 | dust_file_path = os.path.normpath(os.path.join(test_dir, file_name)) 738 | Ui.laconic("clean dust file: " + file_name) 739 | dust_file_fullpath = os.path.realpath(dust_file_path) 740 | 741 | if not os.path.isfile(dust_file_fullpath): 742 | # if you're there, it's because you are debugging a crash 743 | continue 744 | 745 | FileSystem.cleanRemoveFile(dust_file_fullpath) 746 | 747 | paktrace_dir = Default.getPakTraceDir(test_dir) 748 | 749 | if os.path.isdir(paktrace_dir): 750 | logging.debug("look for dust in directory: " + paktrace_dir) 751 | for dir_name, subdir_name_list, file_name_list in os.walk(paktrace_dir): 752 | dir_name = os.path.relpath(dir_name, test_dir) 753 | logging.debug("found paktrace dir: " + dir_name) 754 | 755 | for file_name in file_name_list: 756 | file_path = os.path.join(dir_name, file_name) 757 | file_path = os.path.normpath(file_path) 758 | 759 | relative_paktrace_dir = os.path.relpath(paktrace_dir, test_dir) 760 | trace_file = os.path.relpath(file_path, relative_paktrace_dir) 761 | head_name=trace_file[:-len(Default.paktrace_file_ext)] 762 | 763 | if head_name not in head_list: 764 | Ui.print("clean dust paktrace: " + file_path) 765 | dust_paktrace_path = os.path.normpath(os.path.join(test_dir, file_path)) 766 | dust_paktrace_fullpath = os.path.realpath(dust_paktrace_path) 767 | FileSystem.cleanRemoveFile(dust_paktrace_fullpath) 768 | -------------------------------------------------------------------------------- /Urcheon/Parallelism.py: -------------------------------------------------------------------------------- 1 | #! /usr/bin/env python3 2 | #-*- coding: UTF-8 -*- 3 | 4 | ### Legal 5 | # 6 | # Author: Thomas DEBESSE 7 | # License: ISC 8 | # 9 | 10 | import psutil 11 | import subprocess 12 | import threading 13 | 14 | 15 | getProcess = psutil.Process 16 | 17 | 18 | def countCPU(): 19 | # Reuse computed value 20 | if not hasattr(countCPU, "count"): 21 | countCPU.count = psutil.cpu_count() 22 | 23 | return countCPU.count 24 | 25 | 26 | def countThread(process): 27 | # process can disappear between the time 28 | # this function is called and the num_threads() 29 | # one is called, in this case return 0 30 | try: 31 | return process.num_threads() 32 | except (psutil.NoSuchProcess, psutil.ZombieProcess): 33 | return 0 34 | 35 | 36 | def countChildThread(process): 37 | # process can disappear between the time 38 | # this function is called and the children() 39 | # one is called, in this case return 0 40 | try: 41 | thread_count = 0 42 | # process.children() is super slow 43 | for subprocess in process.children(recursive=True): 44 | thread_count = thread_count + countThread(subprocess) 45 | return thread_count 46 | except (psutil.NoSuchProcess, psutil.ZombieProcess): 47 | return 0 48 | 49 | 50 | def joinDeadThreads(thread_list): 51 | for thread in thread_list: 52 | if not thread.is_alive() and thread._started.is_set(): 53 | thread.join() 54 | thread_list.remove(thread) 55 | 56 | return thread_list 57 | 58 | 59 | def joinThreads(thread_list): 60 | for thread in thread_list: 61 | if not thread._started.is_set(): 62 | thread.start() 63 | thread.join() 64 | 65 | 66 | # this extends threading.Thread to transmit exceptions 67 | # back to the parent, best used with joinDeadThreads() 68 | # on active thread list to raise exceptions early 69 | class Thread(threading.Thread): 70 | def run(self): 71 | self._exception = None 72 | 73 | try: 74 | self._return = self._target(*self._args, **self._kwargs) 75 | except BaseException as exception: 76 | self._exception = exception 77 | 78 | def join(self): 79 | super(Thread, self).join() 80 | 81 | if self._exception: 82 | raise self._exception 83 | 84 | return self._return 85 | -------------------------------------------------------------------------------- /Urcheon/Profile.py: -------------------------------------------------------------------------------- 1 | #! /usr/bin/env python3 2 | #-*- coding: UTF-8 -*- 3 | 4 | ### Legal 5 | # 6 | # Author: Thomas DEBESSE 7 | # License: ISC 8 | # 9 | 10 | 11 | from Urcheon import Default 12 | from Urcheon import Ui 13 | import logging 14 | import os 15 | import sys 16 | 17 | 18 | class Fs(): 19 | def __init__(self, source_dir): 20 | self.file_dict = {} 21 | 22 | profile_dir = os.path.join(Default.share_dir, Default.profile_dir) 23 | self.walk(profile_dir) 24 | 25 | config_dir = Default.getPakConfigDir(source_dir) 26 | self.walk(config_dir) 27 | 28 | logging.debug("files found: " + str(self.file_dict)) 29 | 30 | def walk(self, dir_path): 31 | full_dir_path = os.path.abspath(dir_path) 32 | for dir_name, subdir_name_list, file_name_list in os.walk(full_dir_path): 33 | rel_dir_path = os.path.relpath(dir_name, full_dir_path) 34 | for file_name in file_name_list: 35 | rel_file_path = os.path.normpath(os.path.join(rel_dir_path, file_name)) 36 | full_file_path = os.path.join(dir_name, file_name) 37 | self.file_dict[rel_file_path] = full_file_path 38 | 39 | def isFile(self, file_path): 40 | return file_path in self.file_dict.keys() 41 | 42 | def getPath(self, file_path): 43 | if self.isFile(file_path): 44 | return self.file_dict[file_path] 45 | else: 46 | return None 47 | 48 | def print(self): 49 | for file_path in self.file_dict.keys(): 50 | print(file_path + " → " + self.file_dict[file_path]) 51 | -------------------------------------------------------------------------------- /Urcheon/Texset.py: -------------------------------------------------------------------------------- 1 | #! /usr/bin/env python3 2 | #-*- coding: UTF-8 -*- 3 | 4 | ### Legal 5 | # 6 | # Author: Thomas DEBESSE 7 | # License: ISC 8 | # 9 | 10 | 11 | from Urcheon import Default 12 | from Urcheon import FileSystem 13 | from Urcheon import Profile 14 | from Urcheon import Repository 15 | from Urcheon import Ui 16 | from collections import OrderedDict 17 | import logging 18 | import os 19 | import shutil 20 | import subprocess 21 | import tempfile 22 | import toml 23 | 24 | # FIXME: do we need OrderedDict toml constructor here? 25 | 26 | 27 | class PrevRun(): 28 | def __init__(self, source_tree, preview_profile_path): 29 | # BEWARE: this is not the repository source dir, but the texture source dir! 30 | self.source_dir = source_tree.dir 31 | self.game_name = source_tree.game_name 32 | 33 | self.source_dir_fullpath = os.path.realpath(self.source_dir) 34 | preview_profile_fullpath = os.path.join(self.source_dir_fullpath, preview_profile_path) 35 | 36 | self.profile_fs = Profile.Fs(self.source_dir) 37 | self.prevrun_dict = {} 38 | 39 | self.read(preview_profile_path, real_path = True) 40 | 41 | logging.debug("reading preview profile file: " + preview_profile_fullpath) 42 | preview_profile_file = open(preview_profile_fullpath, "r") 43 | prevrun_dict = toml.load(preview_profile_file, _dict=OrderedDict) 44 | preview_profile_file.close() 45 | 46 | if "dir" not in self.prevrun_dict.keys(): 47 | Ui.error("missing config section: dir") 48 | 49 | if "source" not in self.prevrun_dict["dir"].keys(): 50 | Ui.error("missing “dir” key: dir") 51 | 52 | self.source_dir_name = self.prevrun_dict["dir"]["source"] 53 | self.source_dir_fullpath = os.path.realpath(os.path.join(self.source_dir, self.source_dir_name)) 54 | 55 | if "preview" not in self.prevrun_dict["dir"].keys(): 56 | Ui.error("missing “dir” key: preview") 57 | 58 | self.preview_dir_name = self.prevrun_dict["dir"]["preview"] 59 | self.preview_dir_fullpath = os.path.realpath(os.path.join(self.source_dir, self.preview_dir_name)) 60 | 61 | if "suffix" not in self.prevrun_dict.keys(): 62 | Ui.error("missing config section: suffix") 63 | 64 | if "source" not in self.prevrun_dict["suffix"].keys(): 65 | Ui.error("missing “suffix” key: source") 66 | 67 | suffix_value = self.prevrun_dict["suffix"]["source"] 68 | if isinstance(suffix_value, str): 69 | self.source_suf = [suffix_value] 70 | elif isinstance(suffix_value, list): 71 | self.source_suf = suffix_value 72 | else: 73 | Ui.error("suffix must be a string or a list of string") 74 | 75 | if "preview" not in self.prevrun_dict["suffix"].keys(): 76 | Ui.error("missing “suffix” key: preview") 77 | 78 | self.preview_suf = self.prevrun_dict["suffix"]["preview"] 79 | 80 | self.preview_downscale = False 81 | if "format" in self.prevrun_dict.keys(): 82 | if "downscale" in self.prevrun_dict["format"].keys(): 83 | self.preview_downscale = self.prevrun_dict["format"]["downscale"] 84 | 85 | 86 | def run(self): 87 | source_list = self.walk() 88 | 89 | preview_list = [] 90 | for preview_source_name in source_list: 91 | preview_list.append(self.convert(preview_source_name)) 92 | 93 | return preview_list 94 | 95 | 96 | def read(self, prevrun_profile, real_path=False): 97 | if not real_path: 98 | prevrun_profile_name = os.path.join(Default.prevrun_profile_dir, prevrun_profile + Default.prevrun_profile_ext) 99 | prevrun_profile_fullpath = self.profile_fs.getPath(prevrun_profile_name) 100 | else: 101 | prevrun_profile_fullpath = os.path.realpath(os.path.join(self.source_dir, prevrun_profile)) 102 | 103 | if not os.path.isfile(prevrun_profile_fullpath): 104 | Ui.error("prevrun profile file not found: " + prevrun_profile_fullpath) 105 | 106 | logging.debug("reading prevrun profile file: " + prevrun_profile_fullpath) 107 | prevrun_profile_file = open(prevrun_profile_fullpath, "r") 108 | prevrun_dict = toml.load(prevrun_profile_file, _dict=OrderedDict) 109 | prevrun_profile_file.close() 110 | 111 | if "_init_" in prevrun_dict.keys(): 112 | logging.debug("found “_init_” section in prevrun profile file: " + prevrun_profile_fullpath) 113 | if "extend" in prevrun_dict["_init_"].keys(): 114 | parent_prevrun_name = prevrun_dict["_init_"]["extend"] 115 | 116 | if parent_prevrun_name == "${game}": 117 | parent_prevrun_name = self.game_name 118 | 119 | logging.debug("found “extend” instruction in “_init_” section: " + parent_prevrun_name) 120 | logging.debug("loading parent prevrun profile file") 121 | self.read(parent_prevrun_name) 122 | 123 | del prevrun_dict["_init_"] 124 | 125 | for section in prevrun_dict.keys(): 126 | # if two section names collide, the child win 127 | self.prevrun_dict[section] = prevrun_dict[section] 128 | 129 | 130 | def print(self): 131 | logging.debug(str(self.prevrun_dict)) 132 | print(toml.dumps(self.prevrun_dict)) 133 | 134 | 135 | def walk(self): 136 | source_list = [] 137 | 138 | for dir_name, subdir_name_list, file_name_list in os.walk(self.source_dir_fullpath): 139 | dir_relpath = os.path.relpath(dir_name, self.source_dir) 140 | 141 | logging.debug("dir_name: " + dir_name + ", subdir_name_list: " + str(subdir_name_list) + ", file_name_list: " + str(file_name_list)) 142 | 143 | for file_name in file_name_list: 144 | file_ext = os.path.splitext(file_name)[1] 145 | if file_ext in [ ".bmp", ".jpg", ".jpeg", ".png", ".tga", ".webp" ]: 146 | base_name = file_name[:-len(file_ext)] 147 | for source_suffix in self.source_suf: 148 | if base_name.endswith(source_suffix): 149 | source_path = os.path.normpath(os.path.join(dir_relpath, file_name)) 150 | logging.debug("preview source texture found: " + source_path) 151 | 152 | preview_path = self.getPreviewPath(source_path) 153 | if preview_path: 154 | source_list.append(source_path) 155 | 156 | return source_list 157 | 158 | 159 | def getPreviewPath(self, source_path): 160 | file_ext = os.path.splitext(source_path)[1] 161 | source_basename = os.path.basename(source_path[:-len(file_ext)]) 162 | 163 | suffix_found = False; 164 | for source_suffix in self.source_suf: 165 | if source_basename.endswith(source_suffix): 166 | suffix_found = True 167 | break; 168 | 169 | if not suffix_found: 170 | Ui.error("suffix not found for preview: " + source_path) 171 | 172 | preview_basename = source_basename[:-len(source_suffix)] + self.preview_suf 173 | 174 | preview_name = preview_basename + os.path.extsep + "jpg" 175 | preview_path = os.path.normpath(os.path.join(self.preview_dir_name, preview_name)) 176 | 177 | if preview_path == source_path: 178 | logging.debug("will reuse source as preview") 179 | return None 180 | 181 | for file_ext in [ ".bmp", ".jpg", ".jpeg", ".png", ".tga" ]: 182 | # FIXME: it preserves existing preview but also prevent 183 | # to overwrite the ones we generate 184 | existing_preview_name = preview_basename + file_ext 185 | existing_preview_path = os.path.normpath(os.path.join(self.preview_dir_fullpath, existing_preview_name)) 186 | logging.debug("testing preview: " + existing_preview_path) 187 | if os.path.isfile(existing_preview_path): 188 | logging.debug("will reuse preview: " + existing_preview_path) 189 | return None 190 | 191 | logging.debug("will generate preview: " + preview_path) 192 | return preview_path 193 | 194 | 195 | def convert(self, source_path): 196 | preview_path = self.getPreviewPath(source_path) 197 | 198 | if not preview_path: 199 | logging.debug("will reuse itself as preview for: " + source_path) 200 | return 201 | 202 | preview_fullpath = os.path.realpath(os.path.join(self.source_dir, preview_path)) 203 | source_fullpath = os.path.realpath(os.path.join(self.source_dir, source_path)) 204 | 205 | if FileSystem.isSameTimestamp(preview_fullpath, source_fullpath): 206 | Ui.print("Unmodified file, do nothing: " + source_path) 207 | return preview_path 208 | 209 | FileSystem.makeFileSubdirs(preview_fullpath) 210 | 211 | command_list = [ "convert" ] 212 | command_list += [ source_fullpath ] 213 | command_list += [ "-quality", "50", "-background", "magenta", "-alpha", "remove", "-alpha", "off" ] 214 | 215 | if self.preview_downscale: 216 | command_list += [ "-resize", "256x256>" ] 217 | 218 | command_list += [ preview_fullpath ] 219 | 220 | Ui.laconic("Generate preview: " + source_path) 221 | 222 | logging.debug("convert command list: " + str(command_list)) 223 | logging.debug("convert command line: '" + str("' '".join(command_list)) + "'") 224 | 225 | # TODO: set something else in verbose mode 226 | subprocess_stdout = subprocess.DEVNULL 227 | subprocess_stderr = subprocess.STDOUT 228 | if subprocess.call(command_list, stdout=subprocess_stdout, stderr=subprocess_stderr) != 0: 229 | Ui.error("command failed: '" + "' '".join(command_list) + "'") 230 | 231 | shutil.copystat(source_fullpath, preview_fullpath) 232 | 233 | return preview_path 234 | 235 | 236 | class SlothRun(): 237 | def __init__(self, source_tree, slothrun_file_path): 238 | self.source_dir = source_tree.dir 239 | self.game_name = source_tree.game_name 240 | 241 | self.slothrun_file_path = os.path.normpath(os.path.relpath(slothrun_file_path, self.source_dir)) 242 | 243 | self.profile_fs = Profile.Fs(self.source_dir) 244 | self.slothrun_dict = {} 245 | 246 | self.read(self.slothrun_file_path, real_path=True) 247 | 248 | self.texture_source_dir_list = None 249 | 250 | if "dir" not in self.slothrun_dict.keys(): 251 | Ui.error("missing slothrun section: dir") 252 | 253 | if "source" not in self.slothrun_dict["dir"].keys(): 254 | Ui.error("missing key in “dir” slothrun section: source") 255 | 256 | texture_source_dir_key = self.slothrun_dict["dir"]["source"] 257 | 258 | if not isinstance(texture_source_dir_key, list): 259 | # value must always be a list, if there is only one string, put it in list 260 | self.texture_source_dir_list = [ texture_source_dir_key ] 261 | else: 262 | # TODO: missing directory 263 | self.texture_source_dir_list = texture_source_dir_key 264 | 265 | self.sloth_config = None 266 | 267 | if "sloth" in self.slothrun_dict.keys(): 268 | if "config" in self.slothrun_dict["sloth"].keys(): 269 | self.sloth_config = self.slothrun_dict["sloth"]["config"] 270 | 271 | self.shader_filename = None 272 | self.shader_namespace = None 273 | self.shader_header = None 274 | 275 | if "shader" not in self.slothrun_dict.keys(): 276 | Ui.error("missing slothrun section: shader") 277 | 278 | if "filename" not in self.slothrun_dict["shader"].keys(): 279 | Ui.error("missing key in “shader” slothrun section: filename") 280 | 281 | self.shader_filename = self.slothrun_dict["shader"]["filename"] 282 | 283 | if "header" in self.slothrun_dict["shader"].keys(): 284 | self.shader_header = self.slothrun_dict["shader"]["header"] 285 | 286 | if "namespace" not in self.slothrun_dict["shader"].keys(): 287 | Ui.error("missing key in “shader” slothrun section: namespace") 288 | 289 | self.shader_namespace = self.slothrun_dict["shader"]["namespace"] 290 | 291 | logging.debug("found slothrun directores: " + str(self.texture_source_dir_list)) 292 | 293 | default_texture_suffix_dict = { 294 | "normal": "_n", 295 | "diffuse": "_d", 296 | "height": "_h", 297 | "specular": "_s", 298 | "addition": "_a", 299 | "preview": "_p", 300 | } 301 | 302 | self.texture_suffix_dict = {} 303 | for suffix in default_texture_suffix_dict.keys(): 304 | if suffix in self.slothrun_dict["texture"].keys(): 305 | self.texture_suffix_dict[suffix] = self.slothrun_dict["texture"][suffix] 306 | else: 307 | self.texture_suffix_dict[suffix] = default_texture_suffix_dict[suffix] 308 | 309 | 310 | def run(self): 311 | sloth_list = self.walk() 312 | logging.debug("sloth list: " + str(sloth_list)) 313 | self.sloth() 314 | 315 | 316 | def read(self, slothrun_profile, real_path=False): 317 | if not real_path: 318 | slothrun_profile_name = os.path.join(Default.slothrun_profile_dir, slothrun_profile + Default.slothrun_profile_ext) 319 | slothrun_profile_fullpath = self.profile_fs.getPath(slothrun_profile_name) 320 | else: 321 | slothrun_profile_fullpath = os.path.realpath(os.path.join(self.source_dir, slothrun_profile)) 322 | 323 | if not os.path.isfile(slothrun_profile_fullpath): 324 | Ui.error("slothrun profile file not found: " + slothrun_profile_fullpath) 325 | 326 | logging.debug("reading slothrun profile file: " + slothrun_profile_fullpath) 327 | slothrun_profile_file = open(slothrun_profile_fullpath, "r") 328 | slothrun_dict = toml.load(slothrun_profile_file, _dict=OrderedDict) 329 | slothrun_profile_file.close() 330 | 331 | if "_init_" in slothrun_dict.keys(): 332 | logging.debug("found “_init_” section in slothrun profile file: " + slothrun_profile_fullpath) 333 | if "extend" in slothrun_dict["_init_"].keys(): 334 | parent_slothrun_name = slothrun_dict["_init_"]["extend"] 335 | 336 | if parent_slothrun_name == "${game}": 337 | parent_slothrun_name = self.game_name 338 | 339 | logging.debug("found “extend” instruction in “_init_” section: " + parent_slothrun_name) 340 | logging.debug("loading parent slothrun profile file") 341 | self.read(parent_slothrun_name) 342 | 343 | del slothrun_dict["_init_"] 344 | 345 | for section in slothrun_dict.keys(): 346 | # if two section names collide, the child win 347 | self.slothrun_dict[section] = slothrun_dict[section] 348 | 349 | 350 | def print(self): 351 | logging.debug(str(self.slothrun_dict)) 352 | print(toml.dumps(self.slothrun_dict)) 353 | 354 | 355 | def walk(self): 356 | for texture_source_dir in self.texture_source_dir_list: 357 | sloth_list = [] 358 | 359 | for dir_name, subdir_name_list, file_name_list in os.walk(texture_source_dir): 360 | dir_relpath = os.path.relpath(dir_name, self.source_dir) 361 | 362 | logging.debug("dir_name: " + dir_name + ", subdir_name_list: " + str(subdir_name_list) + ", file_name_list: " + str(file_name_list)) 363 | 364 | for file_name in file_name_list: 365 | file_ext = os.path.splitext(file_name)[1] 366 | 367 | if file_ext == Default.sloth_profile_ext: 368 | sloth_name = os.path.normpath(os.path.join(dir_relpath, file_name)) 369 | logging.debug("sloth file found: " + sloth_name) 370 | sloth_list.append(sloth_name) 371 | 372 | return sloth_list 373 | 374 | 375 | def getStatReference(self): 376 | sourcedir_file_list = [] 377 | for file_path in [ self.slothrun_file_path ] + self.sloth_list + self.preview_source_list: 378 | full_path = os.path.realpath(os.path.join(self.source_dir, file_path)) 379 | sourcedir_file_list.append(full_path) 380 | 381 | # TODO: check also slothrun and sloth files in .urcheon and profiles directories 382 | file_reference_list = sourcedir_file_list 383 | file_reference = FileSystem.getNewer(file_reference_list) 384 | 385 | return file_reference 386 | 387 | 388 | def setTimeStamp(self): 389 | shader_path = os.path.join(self.source_dir, self.shader_name) 390 | shader_fullpath = os.path.realpath(shader_path) 391 | 392 | file_reference = self.getStatReference() 393 | shutil.copystat(file_reference, shader_fullpath) 394 | 395 | 396 | # always sloth after preview generation 397 | def sloth(self): 398 | shader_path = os.path.join(self.source_dir, self.shader_filename) 399 | shader_fullpath = os.path.realpath(shader_path) 400 | 401 | # HACK: never check because multiple files produces on reference 402 | # we can detect added files, but not removed files yet 403 | # if FileSystem.isSameTimestamp(shader_fullpath, file_reference): 404 | # logging.debug("unmodified slothrun, skipping sloth generation") 405 | # return 406 | 407 | FileSystem.makeFileSubdirs(shader_fullpath) 408 | 409 | for command_name in [ "sloth", "sloth.py", None ]: 410 | if command_name == None: 411 | Ui.error("Sloth utility not found") 412 | elif shutil.which(command_name) != None: 413 | break 414 | 415 | command_list = [ command_name ] 416 | 417 | if self.sloth_config: 418 | sloth_profile_name = os.path.join(Default.sloth_profile_dir, self.sloth_config + Default.sloth_profile_ext) 419 | sloth_profile_path = self.profile_fs.getPath(sloth_profile_name) 420 | if sloth_profile_path: 421 | command_list += [ "-f", sloth_profile_path ] 422 | 423 | if "diffuse" in self.texture_suffix_dict.keys(): 424 | command_list += [ "--diffuse", self.texture_suffix_dict["diffuse"] ] 425 | 426 | if "normal" in self.texture_suffix_dict.keys(): 427 | command_list += [ "--normal", self.texture_suffix_dict["normal"] ] 428 | 429 | if "height" in self.texture_suffix_dict.keys(): 430 | command_list += [ "--height", self.texture_suffix_dict["height"] ] 431 | 432 | if "specular" in self.texture_suffix_dict.keys(): 433 | command_list += [ "--specular", self.texture_suffix_dict["specular"] ] 434 | 435 | if "addition" in self.texture_suffix_dict.keys(): 436 | command_list += [ "--addition", self.texture_suffix_dict["addition"] ] 437 | 438 | if "preview" in self.texture_suffix_dict.keys(): 439 | command_list += [ "--preview", self.texture_suffix_dict["preview"] ] 440 | 441 | sloth_header_file = None 442 | if self.shader_header: 443 | header_handle, sloth_header_file = tempfile.mkstemp(suffix="sloth_header" + os.path.extsep + "txt") 444 | os.write(header_handle, str.encode(self.shader_header)) 445 | os.close(header_handle) 446 | 447 | # TODO: write file 448 | 449 | command_list += [ "--header", sloth_header_file ] 450 | 451 | command_list += [ "--root", self.shader_namespace ] 452 | 453 | command_list += [ "--out", shader_fullpath ] 454 | 455 | for texture_source_dir in self.texture_source_dir_list: 456 | command_list += [ os.path.realpath(os.path.join(self.source_dir, texture_source_dir)) ] 457 | 458 | logging.debug("sloth command list: " + str(command_list)) 459 | logging.debug("sloth command line: '" + str("' '".join(command_list)) + "'") 460 | 461 | Ui.laconic("Sloth shader: " + self.slothrun_file_path) 462 | 463 | # TODO: set something else in verbose mode 464 | subprocess_stdout = subprocess.DEVNULL 465 | subprocess_stderr = None 466 | if subprocess.call(command_list, stdout=subprocess_stdout, stderr=subprocess_stderr) != 0: 467 | Ui.error("command failed: '" + "' '".join(command_list) + "'") 468 | 469 | if sloth_header_file: 470 | os.remove(sloth_header_file) 471 | -------------------------------------------------------------------------------- /Urcheon/Ui.py: -------------------------------------------------------------------------------- 1 | #! /usr/bin/env python3 2 | #-*- coding: UTF-8 -*- 3 | 4 | ### Legal 5 | # 6 | # Author: Thomas DEBESSE 7 | # License: ISC 8 | # 9 | 10 | 11 | import sys 12 | from colorama import Fore, Style, init 13 | 14 | 15 | # keep an eye on the default Python's print function 16 | _print = print 17 | 18 | verbosity = None 19 | 20 | def laconic(message): 21 | if sys.stdout.isatty(): 22 | message = Fore.GREEN + message + Style.RESET_ALL 23 | _print(message) 24 | 25 | def print(message): 26 | if verbosity != "laconic": 27 | if sys.stdout.isatty(): 28 | message = Fore.GREEN + message + Style.RESET_ALL 29 | _print(message) 30 | 31 | def verbose(message): 32 | if verbosity == verbose: 33 | if sys.stdout.isatty(): 34 | message = Style.DIM + message + Style.RESET_ALL 35 | 36 | _print(message) 37 | 38 | def warning(message): 39 | message = "Warning: " + message 40 | 41 | if sys.stdout.isatty(): 42 | message = Fore.YELLOW + message + Style.RESET_ALL 43 | 44 | _print(message) 45 | 46 | def help(message, exit=False): 47 | message = "Help: " + message 48 | 49 | if sys.stdout.isatty(): 50 | message = Fore.MAGENTA + message + Style.RESET_ALL 51 | 52 | _print(message) 53 | 54 | if exit: 55 | raise SystemExit() 56 | 57 | def notice(message): 58 | message = "Notice: " + message 59 | 60 | if sys.stdout.isatty(): 61 | message = Fore.CYAN + message + Style.RESET_ALL 62 | 63 | _print(message) 64 | 65 | def error(message, silent=False, exit=True): 66 | _message = message 67 | message = "Error: " + message 68 | 69 | if sys.stdout.isatty(): 70 | message = Fore.RED + message + Style.RESET_ALL 71 | _print(message) 72 | 73 | if exit: 74 | if silent: 75 | raise sys.exit(1) 76 | else: 77 | raise ValueError(_message) 78 | -------------------------------------------------------------------------------- /Urcheon/Urcheon.py: -------------------------------------------------------------------------------- 1 | #! /usr/bin/env python3 2 | #-*- coding: UTF-8 -*- 3 | 4 | ### Legal 5 | # 6 | # Author: Thomas DEBESSE 7 | # License: ISC 8 | # 9 | 10 | 11 | from Urcheon import Action 12 | from Urcheon import Default 13 | from Urcheon import Pak 14 | from Urcheon import Repository 15 | from Urcheon import Ui 16 | import argparse 17 | import logging 18 | import os 19 | import sys 20 | 21 | def discover(args): 22 | source_dir_list = args.source_dir 23 | 24 | for source_dir in source_dir_list: 25 | Ui.notice("discover from: " + source_dir) 26 | source_dir = os.path.realpath(source_dir) 27 | 28 | source_tree = Repository.Tree(source_dir, game_name=args.game_name) 29 | 30 | # TODO: find a way to update "prepare" actions too 31 | file_list = source_tree.listFiles() 32 | 33 | action_list = Action.List(source_tree, "build") 34 | action_list.updateActions(action_list) 35 | 36 | def prepare(args): 37 | args.__dict__.update(stage_name="prepare") 38 | 39 | source_dir_list = args.source_dir 40 | 41 | is_parallel = not args.no_parallel 42 | multi_runner = Pak.MultiRunner(source_dir_list, args) 43 | multi_runner.run() 44 | 45 | def build(args): 46 | args.__dict__.update(stage_name="build") 47 | 48 | source_dir_list = args.source_dir 49 | 50 | if args.test_dir and len(source_dir_list) > 1: 51 | Ui.error("--pakdir can't be used while building more than one source directory", silent=True) 52 | 53 | multi_runner = Pak.MultiRunner(source_dir_list, args) 54 | multi_runner.run() 55 | 56 | def package(args): 57 | args.__dict__.update(stage_name="package") 58 | 59 | source_dir_list = args.source_dir 60 | 61 | if args.test_dir and len(source_dir_list) > 1: 62 | Ui.error("--pakdir can't be used while packaging more than one source directory", silent=True) 63 | 64 | if args.pak_name and len(source_dir_list) > 1: 65 | Ui.error("--pakname can't be used while packaging more than one source directory", silent=True) 66 | 67 | if args.pak_file and len(source_dir_list) > 1: 68 | Ui.error("--pak can't be used while packaging more than one source directory", silent=True) 69 | 70 | is_parallel = not args.no_parallel 71 | multi_runner = Pak.MultiRunner(source_dir_list, args) 72 | multi_runner.run() 73 | 74 | def clean(args): 75 | clean_all = args.clean_all 76 | 77 | if not args.clean_source \ 78 | and not args.clean_map \ 79 | and not args.clean_build \ 80 | and not args.clean_package \ 81 | and not args.clean_all: 82 | clean_all = True 83 | 84 | source_dir_list = args.source_dir 85 | 86 | if args.test_dir and len(source_dir_list) > 1: 87 | Ui.error("--test-dir can't be used while cleaning more than one source directory", silent=True) 88 | 89 | for source_dir in source_dir_list: 90 | Ui.notice("clean from: " + source_dir) 91 | source_dir = os.path.realpath(source_dir) 92 | 93 | source_tree = Repository.Tree(source_dir, game_name=args.game_name) 94 | 95 | cleaner = Pak.Cleaner(source_tree) 96 | 97 | if args.clean_map: 98 | pak_config = Repository.Config(source_tree) 99 | # Do not use pak_name 100 | test_dir = pak_config.getTestDir(args) 101 | cleaner.cleanMap(test_dir) 102 | 103 | if args.clean_source or clean_all: 104 | paktrace = Repository.Paktrace(source_tree, source_dir) 105 | previous_file_list = paktrace.listAll() 106 | cleaner.cleanDust(source_dir, [], previous_file_list) 107 | 108 | if args.clean_build or clean_all: 109 | pak_config = Repository.Config(source_tree) 110 | test_dir = pak_config.getTestDir(args) 111 | cleaner.cleanTest(test_dir) 112 | 113 | if args.clean_package or clean_all: 114 | package_config = Repository.Config(source_tree) 115 | package_dir = package_config.getPackageBasePrefix(args) 116 | cleaner.cleanPak(package_dir) 117 | 118 | def main(): 119 | description="Urcheon is a tender knight who takes care of my lovely granger's little flower." 120 | parser = argparse.ArgumentParser(description=description) 121 | 122 | parser.add_argument("-D", "--debug", dest="debug", help="print debug information", action="store_true") 123 | parser.add_argument("-v", "--verbose", dest="verbose", help="print verbose information", action="store_true") 124 | parser.add_argument("-l", "--laconic", dest="laconic", help="print laconic information", action="store_true") 125 | parser.add_argument("-g", "--game", dest="game_name", metavar="GAMENAME", help="use game %(metavar)s game profile, example: unvanquished") 126 | 127 | parser.add_argument("-C", "--change-directory", dest="change_directory", metavar="DIRNAME", default=".", help="run Urcheon in %(metavar)s directory, default: %(default)s") 128 | 129 | parser.add_argument("--build-prefix", dest="build_prefix", metavar="DIRNAME", help="write build in %(metavar)s prefix, example: " + Default.build_prefix) 130 | parser.add_argument("--build-root-prefix", dest="build_root_prefix", metavar="DIRNAME", help="write test pkg folder in %(metavar)s prefix, example: " + Default.build_root_prefix) 131 | parser.add_argument("--build-base-prefix", dest="build_base_prefix", metavar="DIRNAME", help="write test pakdir in %(metavar)s prefix, example: " + Default.build_base_prefix) 132 | parser.add_argument("--package-prefix", dest="package_prefix", metavar="DIRNAME", help="write package in %(metavar)s prefix, example: " + Default.package_prefix + ", default: same as build prefix") 133 | parser.add_argument("--package-root-prefix", dest="package_root_prefix", metavar="DIRNAME", help="write package pkg folder in %(metavar)s prefix, example: " + Default.package_root_prefix) 134 | parser.add_argument("--package-base-prefix", dest="package_base_prefix", metavar="DIRNAME", help="write package in %(metavar)s prefix, example: " + Default.package_base_prefix) 135 | # FIXME: check on Windows if / works 136 | parser.add_argument("--pakname", dest="pak_name", metavar="STRING", help="use %(metavar)s as pak name with optional prefix, example: dev/nightly will produce build/pkg/dev/nightly_.dpk") 137 | parser.add_argument("--pakprefix", dest="pakprefix", metavar="DIRNAME", help="package release pak in %(metavar)s subdirectory, example: nightly will write build/pkg/nightly/_.dpk ") 138 | parser.add_argument("--pakdir", dest="test_dir", metavar="DIRNAME", help="use directory %(metavar)s as pakdir") 139 | parser.add_argument("--pak", dest="pak_file", metavar="FILENAME", help="build release pak as %(metavar)s file") 140 | parser.add_argument("--version-suffix", dest="version_suffix", metavar="STRING", default=None, help="version suffix string, default: %(default)s") 141 | parser.add_argument("-np", "--no-parallel", dest="no_parallel", help="process tasks sequentially (disable parallel multitasking)", action="store_true") 142 | 143 | subparsers = parser.add_subparsers(help='commands') 144 | subparsers.required = True 145 | 146 | # Discover 147 | discover_parser = subparsers.add_parser('discover', help='discover a package (do not use)') 148 | discover_parser.set_defaults(func=discover) 149 | 150 | discover_parser.add_argument("source_dir", nargs="*", metavar="DIRNAME", default=".", help="discover %(metavar)s directory, default: %(default)s") 151 | 152 | # Prepare 153 | prepare_parser = subparsers.add_parser('prepare', help='prepare a pakdir') 154 | prepare_parser.set_defaults(func=prepare) 155 | 156 | prepare_parser.add_argument("-n", "--no-auto-actions", dest="no_auto_actions", help="do not compute actions at build time", action="store_true") 157 | prepare_parser.add_argument("-k", "--keep", dest="keep_dust", help="keep dust from previous build", action="store_true") 158 | prepare_parser.add_argument("source_dir", nargs="*", metavar="DIRNAME", default=".", help="prepare %(metavar)s directory, default: %(default)s") 159 | 160 | # Build 161 | build_parser = subparsers.add_parser('build', help='build a pakdir') 162 | build_parser.set_defaults(func=build) 163 | 164 | build_parser.add_argument("-mp", "--map-profile", dest="map_profile", metavar="PROFILE", help="build map with %(metavar)s profile, default: %(default)s") 165 | build_parser.add_argument("-n", "--no-auto", dest="no_auto_actions", help="do not compute actions", action="store_true") 166 | build_parser.add_argument("-k", "--keep", dest="keep_dust", help="keep dust from previous build", action="store_true") 167 | build_parser.add_argument("-cm", "--clean-map", dest="clean_map", help="clean previous map build", action="store_true") 168 | build_parser.add_argument("-r", "--reference", dest="since_reference", metavar="REFERENCE", help="build partial pakdir since given reference") 169 | build_parser.add_argument("source_dir", nargs="*", metavar="DIRNAME", default=".", help="build from %(metavar)s directory, default: %(default)s") 170 | 171 | # Package 172 | package_parser = subparsers.add_parser('package', help='package a pak') 173 | package_parser.set_defaults(func=package) 174 | 175 | package_parser.add_argument("-ad", "--allow-dirty", dest="allow_dirty", help="allow to package from repositories with uncommitted files", action="store_true") 176 | package_parser.add_argument("-nc", "--no-compress", dest="no_compress", help="package without compression", action="store_true") 177 | package_parser.add_argument("--merge-directory", dest="merge_dir", metavar="DIRNAME", help="add files from the directory to the archive") 178 | package_parser.add_argument("source_dir", nargs="*", metavar="DIRNAME", default=".", help="package from %(metavar)s directory, default: %(default)s") 179 | 180 | # Clean 181 | clean_parser = subparsers.add_parser('clean', help='clean pakdir and pak') 182 | clean_parser.set_defaults(func=clean) 183 | 184 | clean_parser.add_argument("-a", "--all", dest="clean_all", help="clean all (default)", action="store_true") 185 | clean_parser.add_argument("-s", "--source", dest="clean_source", help="clean source pakdir", action="store_true") 186 | clean_parser.add_argument("-b", "--build", dest="clean_build", help="clean built pakdir", action="store_true") 187 | clean_parser.add_argument("-m", "--map", dest="clean_map", help="clean map build", action="store_true") 188 | clean_parser.add_argument("-p", "--package", dest="clean_package", help="clean packaged pak", action="store_true") 189 | clean_parser.add_argument("source_dir", nargs="*", metavar="DIRNAME", default=[ "." ], help="clean %(metavar)s directory, default: .") 190 | 191 | args = parser.parse_args() 192 | 193 | if args.debug: 194 | logging.basicConfig(stream=sys.stderr, level=logging.DEBUG) 195 | logging.debug("Debug logging activated") 196 | logging.debug("args: " + str(args)) 197 | 198 | if args.laconic: 199 | Ui.verbosity = "laconic" 200 | elif args.verbose: 201 | Ui.verbosity = "verbose" 202 | 203 | os.chdir(args.change_directory) 204 | 205 | args.func(args) 206 | 207 | if __name__ == "__main__": 208 | main() 209 | -------------------------------------------------------------------------------- /Urcheon/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/DaemonEngine/Urcheon/6a8bbc74366e9bebc869582793604cd1915d818f/Urcheon/__init__.py -------------------------------------------------------------------------------- /bin/esquirel: -------------------------------------------------------------------------------- 1 | #! /usr/bin/env python3 2 | #-*- coding: UTF-8 -*- 3 | 4 | ### Legal 5 | # 6 | # Author: Thomas DEBESSE 7 | # License: ISC 8 | # 9 | 10 | import os.path 11 | import sys 12 | 13 | prefix_dir = os.path.realpath(os.path.join(os.path.dirname(os.path.realpath(__file__)), "..")) 14 | 15 | for sub_dir in [".", "lib/python3/dist-packages"]: 16 | module_path = os.path.realpath(os.path.join(prefix_dir, sub_dir)) 17 | if os.path.isfile(os.path.join(module_path, "Urcheon", "Esquirel.py")): 18 | sys.path.append(module_path) 19 | break 20 | 21 | from Urcheon import Esquirel 22 | 23 | if __name__ == "__main__": 24 | Esquirel.main() 25 | -------------------------------------------------------------------------------- /bin/urcheon: -------------------------------------------------------------------------------- 1 | #! /usr/bin/env python3 2 | #-*- coding: UTF-8 -*- 3 | 4 | ### Legal 5 | # 6 | # Author: Thomas DEBESSE 7 | # License: ISC 8 | # 9 | 10 | import os.path 11 | import sys 12 | 13 | prefix_dir = os.path.realpath(os.path.join(os.path.dirname(os.path.realpath(__file__)), "..")) 14 | 15 | for sub_dir in [".", "lib/python3/dist-packages"]: 16 | module_path = os.path.realpath(os.path.join(prefix_dir, sub_dir)) 17 | if os.path.isfile(os.path.join(module_path, "Urcheon", "Urcheon.py")): 18 | sys.path.append(module_path) 19 | break 20 | 21 | from Urcheon import Urcheon 22 | 23 | if __name__ == "__main__": 24 | Urcheon.main() 25 | -------------------------------------------------------------------------------- /doc/cute-granger.512.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/DaemonEngine/Urcheon/6a8bbc74366e9bebc869582793604cd1915d818f/doc/cute-granger.512.png -------------------------------------------------------------------------------- /flake.nix: -------------------------------------------------------------------------------- 1 | # This file allows building and running the software with the Nix package 2 | # manager, used in NixOS or on another distribution. 3 | 4 | { 5 | description = "a toolset to manage and build `pk3` or `dpk` source directories"; 6 | 7 | inputs = { 8 | nixpkgs.url = "flake:nixpkgs"; 9 | 10 | crunch.url = "github:DaemonEngine/crunch"; 11 | crunch.inputs.nixpkgs.follows = "nixpkgs"; 12 | 13 | netradiant.url = "gitlab:xonotic/netradiant"; 14 | netradiant.inputs.nixpkgs.follows = "nixpkgs"; 15 | 16 | sloth.url = "github:Unvanquished/Sloth"; 17 | sloth.inputs.nixpkgs.follows = "nixpkgs"; 18 | }; 19 | 20 | outputs = { self, nixpkgs, crunch, netradiant, sloth }: 21 | let 22 | lib = nixpkgs.legacyPackages.x86_64-linux.lib; 23 | in { 24 | 25 | packages = lib.mapAttrs (system: pkgs: { 26 | iqmtool = 27 | pkgs.stdenv.mkDerivation { 28 | name = "iqmtool"; 29 | src = pkgs.fetchsvn { 30 | url = "http://svn.code.sf.net/p/fteqw/code/trunk/iqm/"; 31 | rev = 6258; 32 | sha256 = "sha256-ddRG4waOSDNfw0OlAnQAFRfdF4caXVefVZWXAvUaszQ="; 33 | }; 34 | 35 | buildInputs = with pkgs; [ 36 | gcc gnumake 37 | ]; 38 | 39 | installPhase = '' 40 | if [ -f iqmtool ]; then 41 | install -Dm0755 iqmtool -T $out/bin/iqmtool 42 | else 43 | install -Dm0755 iqm -T $out/bin/iqmtool 44 | fi 45 | ''; 46 | }; 47 | 48 | urcheon = pkgs.python3.pkgs.buildPythonPackage { 49 | name = "urcheon"; 50 | 51 | src = pkgs.lib.cleanSource ./.; 52 | 53 | format = "other"; 54 | 55 | buildInputs = [ 56 | (pkgs.python3.withPackages 57 | (ps: [ ps.colorama ps.psutil ps.toml ps.pillow ])) 58 | ]; 59 | 60 | propagatedBuildInputs = with pkgs; [ 61 | netradiant.packages."${system}".quake-tools 62 | crunch.defaultPackage."${system}" 63 | sloth.defaultPackage."${system}" 64 | opusTools 65 | libwebp 66 | ]; 67 | 68 | installPhase = '' 69 | runHook preInstall 70 | 71 | mkdir $out/ 72 | cp -ra bin/ profile/ Urcheon/ $out/ 73 | 74 | runHook postInstall 75 | ''; 76 | }; 77 | 78 | crunch = crunch.defaultPackage."${system}"; 79 | sloth = sloth.defaultPackage."${system}"; 80 | } // netradiant.packages."${system}") nixpkgs.legacyPackages; 81 | 82 | apps = lib.mapAttrs (system: pkgs: { 83 | iqmtool = { 84 | type = "app"; 85 | program = "${self.packages."${system}".iqmtool}/bin/iqmtool"; 86 | }; 87 | 88 | urcheon = { 89 | type = "app"; 90 | program = "${self.packages."${system}".urcheon}/bin/urcheon"; 91 | }; 92 | 93 | esquirel = { 94 | type = "app"; 95 | program = "${self.packages."${system}".urcheon}/bin/esquirel"; 96 | }; 97 | 98 | crunch = crunch.defaultApp."${system}"; 99 | sloth = sloth.defaultApp."${system}"; 100 | }) nixpkgs.legacyPackages; 101 | 102 | }; 103 | } 104 | -------------------------------------------------------------------------------- /profile/file/common.conf: -------------------------------------------------------------------------------- 1 | # Author: Thomas DEBESSE 2 | # License: ISC 3 | 4 | [common_pixmap_editor] 5 | file_ext = [ 6 | ".xcf", 7 | ".xcf.gz", 8 | ".xcf.bz2", 9 | ".xcf.xz", 10 | ".psd", 11 | ".ora", 12 | ] 13 | description = "Pixmap Editor File" 14 | build = "ignore" 15 | 16 | [common_gimp_curves] 17 | file_ext = ".curves" 18 | description = "Gimp curves file" 19 | build = "ignore" 20 | 21 | [common_vector_picture] 22 | file_ext = [ 23 | ".svg", 24 | ".svgz", 25 | ] 26 | description = "Vector picture" 27 | build = "ignore" 28 | 29 | [common_metada_sidecar] 30 | file_ext = ".vorbiscomment" 31 | description = "Metadata Sidecar" 32 | build = "ignore" 33 | 34 | [common_pixmap] 35 | file_ext = [ 36 | ".jpg", 37 | ".jpeg", 38 | ".png", 39 | ".tga", 40 | ".bmp", 41 | ".webp", 42 | ".crn", 43 | ".dds", 44 | ] 45 | description = "Texture" 46 | build = "copy" 47 | 48 | [common_sound] 49 | file_ext = [ 50 | ".wav", 51 | ".flac", 52 | ".ogg", 53 | ".opus", 54 | ] 55 | description = "Sound File" 56 | build = "copy" 57 | 58 | [common_script] 59 | file_ext = [ 60 | ".shader", 61 | ".particle", 62 | ".trail", 63 | ] 64 | dir_ancestor_name = "scripts" 65 | description = "Common Script" 66 | build = "copy" 67 | 68 | [common_model] 69 | file_ext = [ 70 | ".ase", 71 | ".iqm", 72 | ".md3", 73 | ".md5anim", 74 | ".md5mesh", 75 | ".qc", 76 | ] 77 | description = "Common Model File" 78 | build = "copy" 79 | 80 | [common_iqe_model] 81 | inherit = "common_model" 82 | file_ext = ".iqe" 83 | description = "Common IQE Model File" 84 | prepare = "compile_iqm" 85 | build = "ignore" 86 | 87 | [common_iqe_config_model] 88 | file_ext = ".iqe.cfg" 89 | description = "Common IQE Command File" 90 | prepare = "ignore" 91 | build = "ignore" 92 | 93 | [common_model_source] 94 | file_ext = [ 95 | ".blend", 96 | ] 97 | description = "Common Model Source" 98 | build = "ignore" 99 | 100 | [common_text] 101 | file_ext = [ 102 | ".txt", 103 | ".md", 104 | ] 105 | description = "Common Text File" 106 | build = "copy" 107 | 108 | [common_prevrun] 109 | file_ext = ".prevrun" 110 | description = "Common PrevRun Description File" 111 | prepare = "run_prevrun" 112 | 113 | [common_slothrun] 114 | file_ext = ".slothrun" 115 | description = "Common SlothRun Description File" 116 | prepare = "run_slothrun" 117 | 118 | [common_sloth] 119 | file_ext = ".sloth" 120 | description = "Common Sloth Description File" 121 | # handled by slothrun 122 | build = "ignore" 123 | 124 | [common_sloth_Feature] 125 | inherit = "common_text" 126 | file_prefix = "sloth-" 127 | file_ext = ".txt" 128 | description = "Common Sloth Feature File" 129 | # handled by slothrun 130 | build = "ignore" 131 | 132 | [common_readme] 133 | inherit = "common_text" 134 | file_base = "README" 135 | description = "Common ReadMe File" 136 | -------------------------------------------------------------------------------- /profile/file/daemon.conf: -------------------------------------------------------------------------------- 1 | # Author: Thomas DEBESSE 2 | # License: ISC 3 | 4 | [_init_] 5 | extend = "common" 6 | 7 | [daemon_script] 8 | inherit = "common_script" 9 | dir_ancestor_name = "scripts" 10 | description = "Script" 11 | 12 | [daemon_texture] 13 | inherit = "common_pixmap" 14 | dir_ancestor_name = [ 15 | "textures", 16 | "lights", 17 | "models", 18 | "gfx", 19 | "env", 20 | ] 21 | description = "Texture" 22 | build = "convert_crn" 23 | 24 | [daemon_crn] 25 | inherit = "daemon_texture" 26 | file_ext = [ 27 | ".crn", 28 | ".dds", 29 | ] 30 | build = "copy" 31 | 32 | [daemon_ui_texture] 33 | inherit = "common_pixmap" 34 | dir_ancestor_name = [ 35 | "emoticons", 36 | "icons", 37 | "ui", 38 | ] 39 | description = "User interface texture" 40 | build = "convert_crn" 41 | 42 | [daemon_ui_crn] 43 | inherit = "daemon_ui_texture" 44 | file_ext = [ 45 | ".crn", 46 | ".dds", 47 | ] 48 | build = "copy" 49 | 50 | [daemon_skybox] 51 | inherit = "daemon_texture" 52 | file_suffix = [ 53 | "_bk", 54 | "_dn", 55 | "_ft", 56 | "_lf", 57 | "_rt", 58 | "_up", 59 | ] 60 | dir_ancestor_name = [ "textures", "env" ] 61 | description = "Skybox" 62 | build = "convert_lossy_webp" 63 | 64 | [daemon_jpg_skybox] 65 | inherit = "daemon_skybox" 66 | file_ext = ".jpg" 67 | build = "copy" 68 | 69 | [daemon_lightmap] 70 | inherit = "common_pixmap" 71 | file_prefix = "lm_" 72 | dir_ancestor_name = "maps" 73 | description = "LightMap" 74 | build = "convert_lossless_webp" 75 | 76 | [daemon_preview] 77 | inherit = "daemon_texture" 78 | file_suffix = "_p" 79 | description = "Editor preview image" 80 | build = "convert_low_jpg" 81 | 82 | [daemon_jpg_preview] 83 | inherit = "daemon_preview" 84 | file_ext = ".jpg" 85 | build = "copy" 86 | 87 | [daemon_normalmap] 88 | inherit = "daemon_texture" 89 | file_suffix = [ 90 | "_n", 91 | # those are non-standards 92 | "_norm", 93 | "_normal", 94 | ] 95 | description = "Normal map" 96 | build = "convert_normalized_crn" 97 | 98 | [daemon_crn_normalmap] 99 | inherit = "daemon_normalmap" 100 | file_ext = [ 101 | ".crn", 102 | ".dds", 103 | ] 104 | build = "copy" 105 | 106 | [daemon_normalheightmap] 107 | inherit = "daemon_texture" 108 | file_suffix = [ 109 | "_nh", 110 | ] 111 | description = "NormalHeight map" 112 | build = "convert_lossy_webp" 113 | 114 | [daemon_crn_normalheightmap] 115 | inherit = "daemon_normalheightmap" 116 | file_ext = [ 117 | ".crn", 118 | ".dds", 119 | ] 120 | build = "copy" 121 | 122 | [daemon_minimap_sidecar] 123 | file_ext = ".minimap" 124 | dir_ancestor_name = "minimaps" 125 | description = "MiniMap sidecar" 126 | build = "copy" 127 | 128 | [daemon_minimap_image] 129 | inherit = "common_pixmap" 130 | dir_ancestor_name = "minimaps" 131 | description = "MiniMap image" 132 | build = "convert_crn" 133 | 134 | [daemon_crn_minimap_image] 135 | inherit = "daemon_minimap_image" 136 | file_ext = [ 137 | ".crn", 138 | ".dds", 139 | ] 140 | build = "copy" 141 | 142 | [daemon_gfx] 143 | inherit = "daemon_texture" 144 | dir_ancestor_name = "gfx" 145 | description = "Graphical effect" 146 | 147 | [daemon_colorgrade] 148 | inherit = "daemon_gfx" 149 | dir_parent_name = "cgrading" 150 | description = "ColorGrade" 151 | build = "convert_lossless_webp" 152 | 153 | [daemon_colorgrade_bis] 154 | inherit = "daemon_gfx" 155 | file_base = [ 156 | "cgrading", 157 | "colorgrading", 158 | ] 159 | description = "ColorGrade" 160 | build = "convert_lossless_webp" 161 | 162 | [daemon_arena] 163 | file_ext = ".arena" 164 | dir_ancestor_name = "meta" 165 | description = "Arena file" 166 | build = "copy" 167 | 168 | [daemon_levelshot] 169 | inherit = "common_pixmap" 170 | dir_ancestor_name = "meta" 171 | description = "LevelShot" 172 | build = "convert_crn" 173 | 174 | [daemon_crn_levelshot] 175 | inherit= "daemon_levelshot" 176 | file_ext = [ 177 | ".crn", 178 | ".dds", 179 | ] 180 | build = "copy" 181 | 182 | [daemon_about] 183 | dir_ancestor_name = "about" 184 | description = "About file" 185 | build = "copy" 186 | 187 | [daemon_map] 188 | file_ext = "map" 189 | dir_ancestor_name = "maps" 190 | description = "Map" 191 | build = "compile_bsp" 192 | 193 | [daemon_bspdir_lump] 194 | dir_ancestor_name = "maps" 195 | dir_father_ext = ".bspdir" 196 | description = "BSP Lump" 197 | build = "merge_bsp" 198 | 199 | [daemon_bspdir_text_lump] 200 | inherit = "daemon_bspdir_lump" 201 | file_ext = [ 202 | ".txt", 203 | ".csv", 204 | ] 205 | description = "BSP Editable Lump" 206 | build = "merge_bsp" 207 | 208 | [daemon_bspdir_blob_lump] 209 | inherit = "daemon_bspdir_lump" 210 | file_ext = [ 211 | ".bin", 212 | ] 213 | dir_ancestor_name = "maps" 214 | dir_father_ext = ".bspdir" 215 | description = "BSP Blob Lump" 216 | build = "merge_bsp" 217 | 218 | [daemon_bspdir_lightmap] 219 | inherit = "daemon_lightmap" 220 | dir_ancestor_name = "maps" 221 | dir_father_name = "lightmaps.d" 222 | dir_grandfather_ext = ".bspdir" 223 | description = "BSP LightMap" 224 | build = "merge_bsp" 225 | 226 | [daemon_bsp] 227 | file_ext = ".bsp" 228 | description = "BSP File" 229 | build = "copy_bsp" 230 | 231 | [daemon_navmesh] 232 | dir_ancestor_name = "maps" 233 | file_ext = ".navMesh" 234 | description = "Navigation Mesh" 235 | build = "copy" 236 | 237 | [daemon_sound] 238 | inherit = "common_sound" 239 | description = "Sound File" 240 | build = "convert_opus" 241 | 242 | [daemon_nullwav] 243 | inherit = "daemon_sound" 244 | file_ext = ".wav" 245 | file_base = "null" 246 | description = "Common NULL Sound File" 247 | build = "copy" 248 | 249 | [daemon_opus] 250 | inherit = "daemon_sound" 251 | file_ext = ".opus" 252 | description = "Opus Sound File" 253 | build = "copy" 254 | 255 | [daemon_vorbis] 256 | inherit = "daemon_sound" 257 | file_ext = ".ogg" 258 | description = "Vorbis Sound File" 259 | build = "copy" 260 | 261 | [daemon_model] 262 | inherit = "common_model" 263 | dir_ancestor_name = "models" 264 | description = "Model File" 265 | 266 | [daemon_iqe_model] 267 | inherit = "common_iqe_model" 268 | dir_ancestor_name = "models" 269 | description = "IQE Model File" 270 | build = "compile_iqm" 271 | -------------------------------------------------------------------------------- /profile/file/unrealarena.conf: -------------------------------------------------------------------------------- 1 | # Author: Thomas DEBESSE 2 | # License: ISC 3 | 4 | [_init_] 5 | extend = "daemon" 6 | -------------------------------------------------------------------------------- /profile/file/unvanquished.conf: -------------------------------------------------------------------------------- 1 | # Author: Thomas DEBESSE 2 | # License: ISC 3 | 4 | [_init_] 5 | extend = "daemon" 6 | -------------------------------------------------------------------------------- /profile/game/daemon.conf: -------------------------------------------------------------------------------- 1 | # Author: Thomas DEBESSE 2 | # License: CC0 1.0 3 | 4 | [config] 5 | pak = "dpk" 6 | -------------------------------------------------------------------------------- /profile/game/unrealarena.conf: -------------------------------------------------------------------------------- 1 | # Author: Thomas DEBESSE 2 | # License: CC0 1.0 3 | 4 | [config] 5 | pak = "pk3" 6 | -------------------------------------------------------------------------------- /profile/game/unvanquished.conf: -------------------------------------------------------------------------------- 1 | # Author: Thomas DEBESSE 2 | # License: CC0 1.0 3 | 4 | [_init_] 5 | extend = "daemon" 6 | -------------------------------------------------------------------------------- /profile/map/common.conf: -------------------------------------------------------------------------------- 1 | # Author: Thomas DEBESSE 2 | # License: CC0 1.0 3 | 4 | [_init_] 5 | source = true 6 | default = "release" 7 | 8 | [_q3map2_] 9 | game="${game}" 10 | 11 | [copy] 12 | copy = { tool="copy" } 13 | -------------------------------------------------------------------------------- /profile/map/daemon.conf: -------------------------------------------------------------------------------- 1 | # Author: Thomas DEBESSE 2 | # License: CC0 1.0 3 | 4 | [_init_] 5 | extend = "common" 6 | 7 | [nometa] 8 | bsp = { tool="q3map2", options="-bsp -keeplights" } 9 | 10 | [novis] 11 | bsp = { tool="q3map2", options="-bsp -keeplights -meta -fastmeta" } 12 | 13 | [nolight] 14 | bsp = { tool="q3map2", options="-bsp -keeplights -meta -fastmeta" } 15 | vis = { tool="q3map2", options="-vis -fast" } 16 | 17 | [nodeluxe] 18 | bsp = { tool="q3map2", options="-bsp -keeplights -meta -fastmeta" } 19 | vis = { tool="q3map2", options="-vis -fast" } 20 | light = { tool="q3map2", options="-light -nocollapse -faster -fastallocate -patchshadows -lightmapsize 1024 -external" } 21 | 22 | [nosample] 23 | bsp = { tool="q3map2", options="-bsp -keeplights -meta -fastmeta" } 24 | vis = { tool="q3map2", options="-vis -fast" } 25 | light = { tool="q3map2", options="-light -nocollapse -faster -fastallocate -patchshadows -deluxe -lightmapsize 1024 -external" } 26 | 27 | [nobounce] 28 | bsp = { tool="q3map2", options="-bsp -keeplights -meta -fastmeta -maxarea -samplesize 16" } 29 | vis = { tool="q3map2", options="-vis" } 30 | light = { tool="q3map2", options="-light -nocollapse -faster -fastallocate -dirty -dirtscale 0.8 -dirtdepth 32 -patchshadows -samples 2 -samplesize 16 -randomsamples -deluxe -lightmapsize 1024 -external" } 31 | 32 | [fast] 33 | bsp = { tool="q3map2", options="-bsp -keeplights -meta -fastmeta -maxarea -samplesize 16" } 34 | vis = { tool="q3map2", options="-vis" } 35 | light = { tool="q3map2", options="-light -nocollapse -faster -fastbounce -fastallocate -nobouncestore -shade -dirty -dirtscale 0.8 -dirtdepth 32 -patchshadows -samples 2 -samplesize 16 -randomsamples -bouncegrid -bounce 1 -deluxe -lightmapsize 1024 -external" } 36 | 37 | [release] 38 | bsp = { tool="q3map2", options="-bsp -keeplights -meta -maxarea -samplesize 8" } 39 | vis = { tool="q3map2", options="-vis" } 40 | light = { tool="q3map2", options="-light -nocollapse -fastbounce -fastallocate -nobouncestore -shade -dirty -dirtscale 0.8 -dirtdepth 32 -patchshadows -samples 3 -samplesize 8 -randomsamples -bouncegrid -bounce 16 -deluxe -lightmapsize 1024 -external" } 41 | 42 | [extreme] 43 | bsp = { tool="q3map2", options="-bsp -keeplights -meta -maxarea -samplesize 8" } 44 | vis = { tool="q3map2", options="-vis" } 45 | light = { tool="q3map2", options="-light -nocollapse -fastallocate -nobouncestore -shade -dirty -dirtscale 0.8 -dirtdepth 32 -patchshadows -samples 3 -samplesize 8 -randomsamples -bouncegrid -bounce 16 -deluxe -lightmapsize 1024 -external" } 46 | -------------------------------------------------------------------------------- /profile/map/smokinguns.conf: -------------------------------------------------------------------------------- 1 | # Author: Thomas DEBESSE 2 | # License: CC0 1.0 3 | 4 | [_init_] 5 | extend = "daemon" 6 | -------------------------------------------------------------------------------- /profile/map/unrealarena.conf: -------------------------------------------------------------------------------- 1 | # Author: Thomas DEBESSE 2 | # License: CC0 1.0 3 | 4 | [_init_] 5 | extend = "daemon" 6 | -------------------------------------------------------------------------------- /profile/map/unvanquished.conf: -------------------------------------------------------------------------------- 1 | # Author: Thomas DEBESSE 2 | # License: CC0 1.0 3 | 4 | [_init_] 5 | extend = "daemon" 6 | 7 | [nobounce] 8 | bsp = { tool="q3map2", options="-bsp -keeplights -meta -fastmeta -maxarea -samplesize 16" } 9 | vis = { tool="q3map2", options="-vis" } 10 | light = { tool="q3map2", options="-light -nocollapse -faster -fastallocate -dirty -dirtscale 0.8 -dirtdepth 32 -patchshadows -samples 2 -samplesize 16 -randomsamples -deluxe -lightmapsize 1024 -external" } 11 | minimap = { tool="q3map2", options="-minimap" } 12 | 13 | [fast] 14 | bsp = { tool="q3map2", options="-bsp -keeplights -meta -fastmeta -maxarea -samplesize 16" } 15 | vis = { tool="q3map2", options="-vis" } 16 | light = { tool="q3map2", options="-light -nocollapse -faster -fastbounce -fastallocate -nobouncestore -shade -dirty -dirtscale 0.8 -dirtdepth 32 -patchshadows -samples 2 -samplesize 16 -randomsamples -bouncegrid -bounce 1 -deluxe -lightmapsize 1024 -external" } 17 | minimap = { tool="q3map2", options="-minimap" } 18 | 19 | [release] 20 | bsp = { tool="q3map2", options="-bsp -keeplights -meta -maxarea -samplesize 8" } 21 | vis = { tool="q3map2", options="-vis" } 22 | light = { tool="q3map2", options="-light -nocollapse -fastbounce -fastallocate -nobouncestore -shade -dirty -dirtscale 0.8 -dirtdepth 32 -patchshadows -samples 3 -samplesize 8 -randomsamples -bouncegrid -bounce 16 -deluxe -lightmapsize 1024 -external" } 23 | minimap = { tool="q3map2", options="-minimap" } 24 | 25 | [extreme] 26 | bsp = { tool="q3map2", options="-bsp -keeplights -meta -maxarea -samplesize 8" } 27 | vis = { tool="q3map2", options="-vis" } 28 | light = { tool="q3map2", options="-light -nocollapse -fastallocate -nobouncestore -shade -dirty -dirtscale 0.8 -dirtdepth 32 -patchshadows -samples 3 -samplesize 8 -randomsamples -bouncegrid -bounce 16 -deluxe -lightmapsize 1024 -external" } 29 | minimap = { tool="q3map2", options="-minimap" } 30 | -------------------------------------------------------------------------------- /profile/prevrun/daemon.prevrun: -------------------------------------------------------------------------------- 1 | [suffix] 2 | source = "_d" 3 | preview = "_p" 4 | -------------------------------------------------------------------------------- /profile/prevrun/unvanquished.prevrun: -------------------------------------------------------------------------------- 1 | [_init_] 2 | extend = "daemon" 3 | -------------------------------------------------------------------------------- /profile/sloth/daemon.sloth: -------------------------------------------------------------------------------- 1 | [options] 2 | 3 | # Unvanquished uses the daemon engine 4 | renderer = daemon 5 | 6 | # netradiant can read compressed texture, so diffuse can be used as preview, with alpha channel 7 | editorOpacity = 0.5 8 | 9 | # q3map2 can read compressed textures, so sloth does not have to precalculate light colors 10 | precalcColors = off 11 | 12 | # q3map2 can read compressed textures, so it can calculate alpha shadows 13 | alphaShadows = on 14 | 15 | # Set default light colors & intensities for lights with custom colors 16 | colors = white:ffffff orange:ffba60 blue:7bb3ff red:ff6c61 17 | customLights = 1500 3000 6000 18 | colorBlendExp = 1.2 19 | 20 | # Set default light intensities for lights with predefined color 21 | predefLights = 0 200 400 22 | -------------------------------------------------------------------------------- /profile/slothrun/daemon.slothrun: -------------------------------------------------------------------------------- 1 | # Author: Thomas DEBESSE 2 | # License: CC0 1.0 3 | 4 | [shader] 5 | header = """ 6 | For modifications, it is recommended that you copy this file into your map's 7 | file space and, inside the shader name (but not in the pathes to the texture 8 | maps!), replace "shared" with your map's short name. If you do so, please 9 | remove this header. 10 | 11 | This file has been generated automatically with Sloth, which can be found at 12 | https://github.com/Unvanquished/Sloth 13 | """ 14 | 15 | [sloth] 16 | config = "daemon" 17 | 18 | [texture] 19 | diffuse = "_d" 20 | normal = "_n" 21 | height = "_h" 22 | specular = "_s" 23 | addition = "_a" 24 | preview = "_p" 25 | -------------------------------------------------------------------------------- /profile/slothrun/unvanquished.slothrun: -------------------------------------------------------------------------------- 1 | # Author: Thomas DEBESSE 2 | # License: CC0 1.0 3 | 4 | [_init_] 5 | extend = "daemon" 6 | -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | argparse 2 | colorama 3 | pillow 4 | psutil 5 | toml>=0.9.0 6 | -------------------------------------------------------------------------------- /samples/unvanquished/entity_substitution.csv: -------------------------------------------------------------------------------- 1 | key, "colorGrade", "gradingTexture" 2 | key, "targetShaderName", "shader" 3 | key, "targetShaderNewName", "replacement" 4 | key, "team", "group" 5 | --------------------------------------------------------------------------------