├── RELEASES.md ├── LICENSE ├── TODO.md ├── README.md └── omwllf.py /RELEASES.md: -------------------------------------------------------------------------------- 1 | # OMWLLF - OpenMW Leveled List Fixer 2 | 3 | ### v1.0 - released 2018-05-14 4 | 5 | Functionality has been stable for a while, with minimal problems. 6 | 7 | - added code to work with non-default windows `My Documents` directories 8 | - added command-line options to: 9 | - specify a non-default config file (actually, this already existed, but was undocumented) 10 | - specify a non-default output directory 11 | - specify a name for the ouput module 12 | - updated README 13 | - added this RELEASES file 14 | 15 | 16 | 17 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Copyright (c) 2016-2017, John Melesky 2 | 3 | Permission to use, copy, modify, and/or distribute this software for any 4 | purpose with or without fee is hereby granted, provided that the above 5 | copyright notice and this permission notice appear in all copies. 6 | 7 | THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES 8 | WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF 9 | MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR 10 | ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES 11 | WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN 12 | ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF 13 | OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. 14 | 15 | -------------------------------------------------------------------------------- /TODO.md: -------------------------------------------------------------------------------- 1 | ### To-do list 2 | 3 | This is brainstorming more than a real plan. 4 | 5 | - Make it even faster. 6 | - Profile it and see if there are any parsing bits that can be improved 7 | - Recollection is that moving to the generator method improved speed significantly, but should still check 8 | - Consider multiprocessing -- the parsing itself does not require sequence at all 9 | - i/o might be the bottleneck, though 10 | - Improve the readability and usefulness of the output, generally 11 | - Improve the mod description text 12 | - Add flags for custom output directories, custom mod names. 13 | - If we do this, we'll have to add in either: 14 | - mechanically editing the openmw.cfg in-place, or 15 | - outputting "add this 'data=' line to your ...", etc. 16 | - GUI version? Should be relatively straightforward with tkinter 17 | - Wrap the script into a .exe and .app download for windows/osx 18 | 19 | Other ideas: 20 | 21 | - Conflict detection is pretty easy, but is it worthwhile for this tool? 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # OMWLLF has been archived 2 | 3 | I'm so glad this tool was useful for as long as it was, but ultimately it was a stopgap until more sophisticated stuff came along. The [Delta Plugin](https://gitlab.com/bmwinger/delta-plugin) tool handles leveled list merging extremely well, and I recommend using it instead of `omwllf`. Thanks, @bmwinger, for writing an excellent tool. 4 | 5 | The readme remains below, and the `omwllf` code will remain available, but this repository is now archived and there will be no further changes. Thanks to all of you who used it! 6 | 7 | 8 | # OMWLLF - OpenMW Leveled List Fixer - v1.0 9 | 10 | This is a utility written specifically for [OpenMW](http://openmw.org/) users who want to use lots of mods, and don't want to wrestle with using MW Classic tools to merge their leveled lists. 11 | 12 | ## How to use this 13 | 14 | First, make sure you have python (version 3.3 or higher) installed on your system and reachable. 15 | 16 | Second, make sure the script itself (`omwllf.py`) is downloaded and available. You can download it from github at https://github.com/jmelesky/omwllf 17 | 18 | Then, [install your mods in the OpenMW way](https://wiki.openmw.org/index.php?title=Mod_installation), adding `data` lines to your `openmw.cfg`. 19 | 20 | Make sure to start the launcher and enable all the appropriate `.esm`, `.esp`, and `.omwaddon` files. Drag them around to the appropriate load order. 21 | 22 | Then, run `omwllf.py` from a command line (Terminal in OS X, Command Prompt in Windows, etc). This should create a new `.omwaddon` module, and give you instructions on how to enable it. 23 | 24 | Open the Launcher, drag the new module to the bottom (it should be loaded last), and enable it. 25 | 26 | Finally, start OpenMW with your new, merged leveled lists. 27 | 28 | ## Advanced usage 29 | 30 | Having everything automatic is darn handy, but some of us have multiple config files and setups. OpenMW can handle that, so `omwllf` should be able to, too. 31 | 32 | There are some useful command-line arguments: 33 | 34 | - `-c` (or `--configfile`), which allows you to specify a specific config file to use 35 | - `-d` (or `--moddir`), where you can set the directory in which to put the new mod 36 | - `-m` (or `--modname`), which lets you set the name of the new mod (I like the default of `OMW Mod - .omwaddon`, but to each their own) 37 | 38 | All of those are optional (obviously), but when you need them, you need them. 39 | 40 | ## HELP! 41 | 42 | Are you having a problem? I can only fix it if I know about it. You can [file an issue](https://github.com/jmelesky/omwllf/issues) on the github project. I'm also trying to be available on the [OpenMW General Discussion forum](https://forum.openmw.org/viewforum.php?f=2), and sometimes on the [#openmw irc channel](https://webchat.freenode.net/?channels=openmw&uio=OT10cnVlde). 43 | 44 | ## Thanks 45 | 46 | * Resources for understanding MW file formats: 47 | * http://www.mwmythicmods.com/argent/tech/tute.html 48 | * http://www.mwmythicmods.com/tutorials/MorrowindESPFormat.html 49 | * The much-more-fully-functioning tool for classic TES3: 50 | * https://github.com/john-moonsugar/tes3cmd 51 | 52 | -------------------------------------------------------------------------------- /omwllf.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | 3 | from struct import pack, unpack 4 | from datetime import date 5 | from pathlib import Path 6 | import os.path 7 | import argparse 8 | import sys 9 | import re 10 | 11 | 12 | configFilename = 'openmw.cfg' 13 | configPaths = { 'linux': '~/.config/openmw', 14 | 'freebsd': '~/.config/openmw', 15 | 'darwin': '~/Library/Preferences/openmw' } 16 | 17 | modPaths = { 'linux': '~/.local/share/openmw/data', 18 | 'freebsd': '~/.local/share/openmw/data', 19 | 'darwin': '~/Library/Application Support/openmw/data' } 20 | 21 | 22 | def packLong(i): 23 | # little-endian, "standard" 4-bytes (old 32-bit systems) 24 | return pack(' l: 32 | # still need to null-terminate 33 | return bs[:(l-1)] + bytes(1) 34 | else: 35 | return bs + bytes(l - len(bs)) 36 | 37 | def parseString(ba): 38 | i = ba.find(0) 39 | return ba[:i].decode(encoding='ascii', errors='ignore') 40 | 41 | def parseNum(ba): 42 | return int.from_bytes(ba, 'little') 43 | 44 | def parseFloat(ba): 45 | return unpack('f', ba)[0] 46 | 47 | def parseLEV(rec): 48 | levrec = {} 49 | sr = rec['subrecords'] 50 | 51 | levrec['type'] = rec['type'] 52 | levrec['name'] = parseString(sr[0]['data']) 53 | levrec['calcfrom'] = parseNum(sr[1]['data']) 54 | levrec['chancenone'] = parseNum(sr[2]['data']) 55 | levrec['file'] = os.path.basename(rec['fullpath']) 56 | 57 | # Apparently, you can have LEV records that end before 58 | # the INDX subrecord. Found those in Tamriel_Data.esm 59 | if len(sr) > 3: 60 | listcount = parseNum(sr[3]['data']) 61 | listitems = [] 62 | 63 | for i in range(0,listcount*2,2): 64 | itemid = parseString(sr[4+i]['data']) 65 | itemlvl = parseNum(sr[5+i]['data']) 66 | listitems.append((itemlvl, itemid)) 67 | 68 | levrec['items'] = listitems 69 | else: 70 | levrec['items'] = [] 71 | 72 | return levrec 73 | 74 | def parseTES3(rec): 75 | tesrec = {} 76 | sr = rec['subrecords'] 77 | tesrec['version'] = parseFloat(sr[0]['data'][0:4]) 78 | tesrec['filetype'] = parseNum(sr[0]['data'][4:8]) 79 | tesrec['author'] = parseString(sr[0]['data'][8:40]) 80 | tesrec['desc'] = parseString(sr[0]['data'][40:296]) 81 | tesrec['numrecords'] = parseNum(sr[0]['data'][296:300]) 82 | 83 | masters = [] 84 | for i in range(1, len(sr), 2): 85 | mastfile = parseString(sr[i]['data']) 86 | mastsize = parseNum(sr[i+1]['data']) 87 | masters.append((mastfile, mastsize)) 88 | 89 | tesrec['masters'] = masters 90 | return tesrec 91 | 92 | def pullSubs(rec, subtype): 93 | return [ s for s in rec['subrecords'] if s['type'] == subtype ] 94 | 95 | def readHeader(ba): 96 | header = {} 97 | header['type'] = ba[0:4].decode() 98 | header['length'] = int.from_bytes(ba[4:8], 'little') 99 | return header 100 | 101 | def readSubRecord(ba): 102 | sr = {} 103 | sr['type'] = ba[0:4].decode() 104 | sr['length'] = int.from_bytes(ba[4:8], 'little') 105 | endbyte = 8 + sr['length'] 106 | sr['data'] = ba[8:endbyte] 107 | return (sr, ba[endbyte:]) 108 | 109 | def readRecords(filename): 110 | fh = open(filename, 'rb') 111 | while True: 112 | headerba = fh.read(16) 113 | if headerba is None or len(headerba) < 16: 114 | return None 115 | 116 | record = {} 117 | header = readHeader(headerba) 118 | record['type'] = header['type'] 119 | record['length'] = header['length'] 120 | record['subrecords'] = [] 121 | # stash the filename here (a bit hacky, but useful) 122 | record['fullpath'] = filename 123 | 124 | remains = fh.read(header['length']) 125 | 126 | while len(remains) > 0: 127 | (subrecord, restofbytes) = readSubRecord(remains) 128 | record['subrecords'].append(subrecord) 129 | remains = restofbytes 130 | 131 | yield record 132 | 133 | def oldGetRecords(filename, rectype): 134 | return ( r for r in readRecords(filename) if r['type'] == rectype ) 135 | 136 | def getRecords(filename, rectypes): 137 | numtypes = len(rectypes) 138 | retval = [ [] for x in range(numtypes) ] 139 | for r in readRecords(filename): 140 | if r['type'] in rectypes: 141 | for i in range(numtypes): 142 | if r['type'] == rectypes[i]: 143 | retval[i].append(r) 144 | return retval 145 | 146 | def packStringSubRecord(lbl, strval): 147 | str_bs = packString(strval) + bytes(1) 148 | l = packLong(len(str_bs)) 149 | return packString(lbl) + l + str_bs 150 | 151 | def packIntSubRecord(lbl, num, numsize=4): 152 | # This is interesting. The 'pack' function from struct works fine like this: 153 | # 154 | # >>> pack('>> fs = '>> pack(fs, 200) 161 | # Traceback (most recent call last): 162 | # File "", line 1, in 163 | # struct.error: repeat count given without format specifier 164 | # 165 | # This is as of Python 3.5.2 166 | 167 | num_bs = b'' 168 | if numsize == 4: 169 | # "standard" 4-byte longs, little-endian 170 | num_bs = pack(' 1: 283 | mergeables[k] = candidates[k] 284 | 285 | return mergeables 286 | 287 | 288 | def mergeLists(lls): 289 | # last one gets priority for list-level attributes 290 | last = lls[-1] 291 | newLev = { 'type': last['type'], 292 | 'name': last['name'], 293 | 'calcfrom': last['calcfrom'], 294 | 'chancenone': last['chancenone'] } 295 | 296 | allItems = [] 297 | for l in lls: 298 | allItems += l['items'] 299 | 300 | newLev['files'] = [ x['file'] for x in lls ] 301 | newLev['file'] = ', '.join(newLev['files']) 302 | 303 | 304 | # This ends up being a bit tricky, but it prevents us 305 | # from overloading lists with the same stuff. 306 | # 307 | # This is needed, because the original leveled lists 308 | # contain multiple entries for some creatures/items, and 309 | # that gets reproduced in many plugins. 310 | # 311 | # If we just added and sorted, then the more plugins you 312 | # have, the less often you'd see plugin content. This 313 | # method prevents the core game content from overwhelming 314 | # plugin contents. 315 | 316 | allUniques = [ x for x in set(allItems) ] 317 | allUniques.sort() 318 | 319 | newList = [] 320 | 321 | for i in allUniques: 322 | newCount = max([ x['items'].count(i) for x in lls ]) 323 | newList += [i] * newCount 324 | 325 | newLev['items'] = newList 326 | 327 | return newLev 328 | 329 | 330 | def mergeAllLists(alllists): 331 | mergeables = mergeableLists(alllists) 332 | 333 | merged = [] 334 | 335 | for k in mergeables: 336 | merged.append(mergeLists(mergeables[k])) 337 | 338 | return merged 339 | 340 | 341 | def readCfg(cfg): 342 | # first, open the file and pull all 'data' and 'content' lines, in order 343 | 344 | data_dirs = [] 345 | mods = [] 346 | with open(cfg, 'r') as f: 347 | for l in f.readlines(): 348 | # match of form "blah=blahblah" 349 | m = re.search(r'^(.*)=(.*)$', l) 350 | if m: 351 | varname = m.group(1).strip() 352 | # get rid of not only whitespace, but also surrounding quotes 353 | varvalue = m.group(2).strip().strip('\'"') 354 | if varname == 'data': 355 | data_dirs.append(varvalue) 356 | elif varname == 'content': 357 | mods.append(varvalue) 358 | 359 | # we've got the basenames of the mods, but not the full paths 360 | # and we have to search through the data_dirs to find them 361 | fp_mods = [] 362 | for m in mods: 363 | for p in data_dirs: 364 | full_path = os.path.join(p, m) 365 | if os.path.exists(full_path): 366 | fp_mods.append(full_path) 367 | break 368 | 369 | print("Config file parsed...") 370 | 371 | return fp_mods 372 | 373 | def dumplists(cfg): 374 | llists = [] 375 | fp_mods = readCfg(cfg) 376 | 377 | for f in fp_mods: 378 | [ ppTES3(parseTES3(x)) for x in oldGetRecords(f, 'TES3') ] 379 | 380 | for f in fp_mods: 381 | llists += [ parseLEV(x) for x in oldGetRecords(f, 'LEVI') ] 382 | 383 | for f in fp_mods: 384 | llists += [ parseLEV(x) for x in oldGetRecords(f, 'LEVC') ] 385 | 386 | for l in llists: 387 | ppLEV(l) 388 | 389 | 390 | def main(cfg, outmoddir, outmod): 391 | fp_mods = readCfg(cfg) 392 | 393 | # first, let's grab the "raw" records from the files 394 | 395 | (rtes3, rlevi, rlevc) = ([], [], []) 396 | for f in fp_mods: 397 | print("Parsing '%s' for relevant records" % f) 398 | (rtes3t, rlevit, rlevct) = getRecords(f, ('TES3', 'LEVI', 'LEVC')) 399 | rtes3 += rtes3t 400 | rlevi += rlevit 401 | rlevc += rlevct 402 | 403 | # next, parse the tes3 records so we can get a list 404 | # of master files required by all our mods 405 | 406 | tes3list = [ parseTES3(x) for x in rtes3 ] 407 | 408 | masters = {} 409 | for t in tes3list: 410 | for m in t['masters']: 411 | masters[m[0]] = m[1] 412 | 413 | master_list = [ (k,v) for (k,v) in masters.items() ] 414 | 415 | # now, let's parse the levi and levc records into 416 | # mergeable lists, then merge them 417 | 418 | # creature lists 419 | clist = [ parseLEV(x) for x in rlevc ] 420 | levc = mergeAllLists(clist) 421 | 422 | # item lists 423 | ilist = [ parseLEV(x) for x in rlevi ] 424 | levi = mergeAllLists(ilist) 425 | 426 | 427 | # now build the binary representation of 428 | # the merged lists. 429 | # along the way, build up the module 430 | # description for the new merged mod, out 431 | # of the names of mods that had lists 432 | 433 | llist_bc = b'' 434 | pluginlist = [] 435 | for x in levi + levc: 436 | # ppLEV(x) 437 | llist_bc += packLEV(x) 438 | pluginlist += x['files'] 439 | plugins = set(pluginlist) 440 | moddesc = "Merged leveled lists from: %s" % ', '.join(plugins) 441 | 442 | # finally, build the binary form of the 443 | # TES3 record, and write the whole thing 444 | # out to disk 445 | 446 | if not os.path.exists(outmoddir): 447 | p = Path(outmoddir) 448 | p.mkdir(parents=True) 449 | 450 | with open(outmod, 'wb') as f: 451 | f.write(packTES3(moddesc, len(levi + levc), master_list)) 452 | f.write(llist_bc) 453 | 454 | # And give some hopefully-useful instructions 455 | 456 | modShortName = os.path.basename(outmod) 457 | print("\n\n****************************************") 458 | print(" Great! I think that worked. When you next start the OpenMW Launcher, look for a module named %s. Make sure of the following things:" % modShortName) 459 | print(" 1. %s is at the bottom of the list. Drag it to the bottom if it's not. It needs to load last." % modShortName) 460 | print(" 2. %s is checked (enabled)" % modShortName) 461 | print(" 3. Any other OMWLLF mods are *un*checked. Loading them might not cause problems, but probably will") 462 | print("\n") 463 | print(" Then, go ahead and start the game! Your leveled lists should include adjustments from all relevant enabled mods") 464 | print("\n") 465 | 466 | 467 | if __name__ == '__main__': 468 | parser = argparse.ArgumentParser() 469 | 470 | parser.add_argument('-c', '--conffile', type = str, default = None, 471 | action = 'store', required = False, 472 | help = 'Conf file to use. Optional. By default, attempts to use the default conf file location.') 473 | 474 | parser.add_argument('-d', '--moddir', type = str, default = None, 475 | action = 'store', required = False, 476 | help = 'Directory to store the new module in. By default, attempts to use the default work directory for OpenMW-CS') 477 | 478 | parser.add_argument('-m', '--modname', type = str, default = None, 479 | action = 'store', required = False, 480 | help = 'Name of the new module to create. By default, this is "OMWLLF Mod - .omwaddon.') 481 | 482 | parser.add_argument('--dumplists', default = False, 483 | action = 'store_true', required = False, 484 | help = 'Instead of generating merged lists, dump all leveled lists in the conf mods. Used for debugging') 485 | 486 | p = parser.parse_args() 487 | 488 | 489 | # determine the conf file to use 490 | confFile = '' 491 | if p.conffile: 492 | confFile = p.conffile 493 | else: 494 | pl = sys.platform 495 | if pl in configPaths: 496 | baseDir = os.path.expanduser(configPaths[pl]) 497 | confFile = os.path.join(baseDir, configFilename) 498 | elif pl == 'win32': 499 | # this is ugly. first, imports that only work properly on windows 500 | from ctypes import * 501 | import ctypes.wintypes 502 | 503 | buf = create_unicode_buffer(ctypes.wintypes.MAX_PATH) 504 | 505 | # opaque arguments. they are, roughly, for our purposes: 506 | # - an indicator of folder owner (0 == current user) 507 | # - an id for the type of folder (5 == 'My Documents') 508 | # - an indicator for user to call from (0 same as above) 509 | # - a bunch of flags for different things 510 | # (if you want, for example, to get the default path 511 | # instead of the actual path, or whatnot) 512 | # 0 == current stuff 513 | # - the variable to hold the return value 514 | 515 | windll.shell32.SHGetFolderPathW(0, 5, 0, 0, buf) 516 | 517 | # pull out the return value and construct the rest 518 | baseDir = os.path.join(buf.value, 'My Games', 'OpenMW') 519 | confFile = os.path.join(baseDir, configFilename) 520 | else: 521 | print("Sorry, I don't recognize the platform '%s'. You can try specifying the conf file using the '-c' flag." % p) 522 | sys.exit(1) 523 | 524 | baseModDir = '' 525 | if p.moddir: 526 | baseModDir = p.moddir 527 | else: 528 | pl = sys.platform 529 | if pl in configPaths: 530 | baseModDir = os.path.expanduser(modPaths[pl]) 531 | elif pl == 'win32': 532 | # this is ugly in exactly the same ways as above. 533 | # see there for more information 534 | 535 | from ctypes import * 536 | import ctypes.wintypes 537 | 538 | buf = create_unicode_buffer(ctypes.wintypes.MAX_PATH) 539 | 540 | windll.shell32.SHGetFolderPathW(0, 5, 0, 0, buf) 541 | 542 | baseDir = os.path.join(buf.value, 'My Games', 'OpenMW') 543 | baseModDir = os.path.join(baseDir, 'data') 544 | else: 545 | print("Sorry, I don't recognize the platform '%s'. You can try specifying the conf file using the '-c' flag." % p) 546 | sys.exit(1) 547 | 548 | 549 | if not os.path.exists(confFile): 550 | print("Sorry, the conf file '%s' doesn't seem to exist." % confFile) 551 | sys.exit(1) 552 | 553 | modName = '' 554 | if p.modname: 555 | modName = p.modname 556 | else: 557 | modName = 'OMWLLF Mod - %s.omwaddon' % date.today().strftime('%Y-%m-%d') 558 | 559 | modFullPath = os.path.join(baseModDir, modName) 560 | 561 | if p.dumplists: 562 | dumplists(confFile) 563 | else: 564 | main(confFile, baseModDir, modFullPath) 565 | 566 | 567 | 568 | 569 | 570 | # regarding the windows path detection: 571 | # 572 | # "SHGetFolderPath" is deprecated in favor of "SHGetKnownFolderPath", but 573 | # >>> windll.shell32.SHGetKnownFolderPath('{FDD39AD0-238F-46AF-ADB4-6C85480369C7}', 0, 0, buf2) 574 | # -2147024894 575 | 576 | 577 | --------------------------------------------------------------------------------