├── .gitignore ├── FORMATS.md ├── LICENSE ├── MANIFEST.in ├── README.md ├── setup.py └── starbound ├── __init__.py ├── btreedb5.py ├── cliexport.py ├── cliregion.py ├── clirepair.py ├── sbasset6.py └── sbon.py /.gitignore: -------------------------------------------------------------------------------- 1 | *.pyc 2 | dist 3 | build 4 | py_starbound.egg-info 5 | -------------------------------------------------------------------------------- /FORMATS.md: -------------------------------------------------------------------------------- 1 | # Starbound data formats 2 | 3 | This document is intended to describe Starbound's various data structures. 4 | 5 | - [File formats](#file-formats) 6 | - [SBON](#sbon) 7 | - [Celestial data](#celestial-data) 8 | - [World data](#world-data) 9 | 10 | ## File formats 11 | 12 | Starbound uses regular JSON and Lua files for some things, but this 13 | document will only focus on the custom file formats. 14 | 15 | - [BTreeDB5](#btreedb5) 16 | - [SBAsset6](#sbasset6) 17 | - [SBVJ01](#sbvj01) 18 | 19 | ### BTreeDB5 20 | 21 | A B-tree database format which enables quick scanning and updating. 22 | It's used by Starbound to save world and universe data. 23 | 24 | #### Header 25 | 26 | The header consists of 512 bytes, representing the following fields: 27 | 28 | | Field # | Type | Description | 29 | | ------: | ----------- | -------------------------------------- | 30 | | 1 | `char[8]` | The string "BTreeDB5" | 31 | | 2 | `int32` | Byte size of blocks (see below) | 32 | | 3 | `char[16]` | The name of the database (null padded) | 33 | | 4 | `int32` | Byte size of index keys | 34 | | 5 | `bool` | Whether to use root node #2 instead | 35 | | 6 | `int32` | Free node #1 block index | 36 | | – | `byte[4]` | Unknown | 37 | | 7 | `int32` | Offset in file of end of free block #1 | 38 | | 8 | `int32` | Root node #1 block index | 39 | | 9 | `boolean` | Whether root node #1 is a leaf | 40 | | 10 | `int32` | Free node #2 block index | 41 | | – | `byte[4]` | Unknown | 42 | | 11 | `int32` | Offset in file of end of free block #2 | 43 | | 12 | `int32` | Root node #2 block index | 44 | | 13 | `boolean` | Whether root node #2 is a leaf | 45 | | – | `byte[445]` | Unused bytes | 46 | 47 | In the BTreeDB4 format there was also a "free node is dirty" boolean 48 | which is not accounted for above. It may be one of the "Unknown" 49 | values. 50 | 51 | #### Blocks 52 | 53 | The most primitive structure in the BTreeDB5 format is the block. It's 54 | a chunk of bytes of a fixed size (defined in the header) which plays a 55 | certain role in the database. 56 | 57 | A lot of fields in the BTreeDB5 format references blocks by their index 58 | which means an offset of `header_size + index * block_size`. 59 | 60 | ##### Root nodes 61 | 62 | The root node is the entry point when scanning for a specific key. The 63 | root node can be either an index block or a leaf block, depending on 64 | how large the database is. Usually, it will be an index block. 65 | 66 | Since BTreeDB5 databases are meant to be updated on the fly, the root 67 | node may alternate to allow for transactional updates to the index. 68 | If the _alternate root block index_ flag is true, the alternate root 69 | index should be used for entry instead when scanning for a key. 70 | 71 | ##### Index block 72 | 73 | The index block always starts with the characters `II`. 74 | 75 | ##### Leaf block 76 | 77 | The leaf block always starts with the characters `LL`. 78 | 79 | ##### Free block 80 | 81 | The free block always starts with the characters `FF`. 82 | 83 | Free blocks may be either brand new blocks (after growing the file), or 84 | reclaimed blocks that were no longer in use. 85 | 86 | #### Scanning for a key 87 | 88 | This section will contain information on how to retrieve a value from a 89 | BTreeDB5 database. 90 | 91 | ### SBAsset6 92 | 93 | A large file that contains an index pointing to many small files within 94 | it. The main asset file (`packed.pak`) and mods are of this type. 95 | 96 | #### Header 97 | 98 | The header for SBAsset6 is very straightforward: 99 | 100 | | Field # | Type | Description | 101 | | ------: | --------- | --------------------- | 102 | | 1 | `char[8]` | The string "SBAsset6" | 103 | | 2 | `uint64` | Metadata offset | 104 | 105 | The metadata offset points to another location in the file where the 106 | metadata can be read. Seek to that point in the file and find: 107 | 108 | | Field # | Type | Description | 109 | | ------: | ----------- | ---------------------------- | 110 | | 1 | `char[5]` | The string "INDEX" | 111 | | 2 | SBON map | Information about the file | 112 | | 3 | SBON varint | Number of files in the index | 113 | | 4 + 3n | SBON string | SBON UTF-8 encoded string | 114 | | 5 + 3n | `uint64` | Offset where file starts | 115 | | 6 + 3n | `uint64` | Length of file | 116 | 117 | Once the index has been parsed into memory, it can be used to seek to 118 | various files in the SBAsset6 file. 119 | 120 | ### SBVJ01 121 | 122 | Versioned JSON-like data. Used for player data files and the like. The 123 | data structures themselves use a custom binary form of JSON which will 124 | be referred to as "SBON" in this document. 125 | 126 | The file structure is simply the string `"SBVJ01"` followed by a single 127 | versioned JSON object (see below). 128 | 129 | ## SBON 130 | 131 | (I'm calling this "Starbound Binary Object Notation", but don't know 132 | what the Starbound developers call it internally.) 133 | 134 | This format is similar to other binary formats for JSON (e.g., BSON). 135 | SBON is used in most file formats to represent complex data such as 136 | metadata and entities. 137 | 138 | ### Data types 139 | 140 | - Variable length integer (also known as [VLQ][vlq]) 141 | - Bytes (varint for length + the bytes) 142 | - String (bytes with UTF-8 encoding) 143 | - List (varint for count, dynamic for values) 144 | - Map (varint for count, string/dynamic pairs for entries) 145 | - Dynamic (byte for type + value) 146 | - `0x01`: Nil value 147 | - `0x02`: Double-precision float (a.k.a. `double`) 148 | - `0x03`: Boolean 149 | - `0x04`: Signed varint (see below) 150 | - `0x05`: String 151 | - `0x06`: List 152 | - `0x07`: Map 153 | 154 | #### Varint 155 | 156 | A variable length (in bytes) integer, also known as [VLQ][vlq]. As long as the most significant bit is set read the next byte and concatenate its 7 other bits with the 7 bits of the previous bytes. The resulting string of bits is the binary representation of the number. 157 | 158 | The purposes of this data type is to allow (common) lower values 0...127 to only use up one byte, 128...16383 two bytes, and so on. 159 | 160 | #### Signed varint 161 | 162 | A signed varint is just a regular varint, except that the least significant bit (the very last bit in the data stream) is used to represent the sign of the number. If the bit is 1, the number should be considered negative and also have one subtracted from its value (because there is no negative 0). If the bit is 0, the number is positive. In both cases, the least significant bit should not be considered part of the number. 163 | 164 | ### Versioned JSON 165 | 166 | Starbound has a data structure known as "versioned JSON" which consists 167 | of SBON. In addition to arbitrary data, it also holds a name and a 168 | version. 169 | 170 | Most complex data structures are represented as versioned JSON and may 171 | have Lua scripts that upgrade older versions to the current one. 172 | 173 | | Field | Type | Description | 174 | | ------------- | ------------ | ------------------------------------------- | 175 | | Name | SBON string | The name or type of the data structure. | 176 | | Is versioned? | `bool` | Flag indicating that there’s a version. | 177 | | Version | `int32` | Version (only if previous field is `true`). | 178 | | Data | SBON dynamic | The data itself, usually a map. | 179 | 180 | ## Celestial data 181 | 182 | Celestial files are BTreeDB5 databases that contain generated 183 | information about the universe. 184 | 185 | Little is currently known about this format as the keys are hashes of 186 | some key that has not yet been reverse engineered. 187 | 188 | ## World data 189 | 190 | World files (this includes the player ship) are BTreeDB5 databases that 191 | contain the metadata, entities, and tile regions of the world. 192 | 193 | ### Regions 194 | 195 | Regions are indexed by _type_, _X_, and _Y_ values. The _type_ value is 196 | `1` for tile data and `2` for entity data. There's a special key, 197 | {0, 0, 0} which points to the world metadata. All values are gzip 198 | deflated and must be inflated before they can be read. 199 | 200 | The BTreeDB5 key for a region is represented in binary as a byte for 201 | _type_ followed by two shorts for the _X_ and _Y_ coordinates. 202 | 203 | The X axis goes from left to right and the Y axis goes from down to up. 204 | 205 | #### World metadata 206 | 207 | Once inflated, the {0, 0, 0} value starts with two integers (8 bytes) 208 | holding the number of tiles along the X and Y axes, followed by an SBON 209 | data structure containing all the world's metadata. 210 | 211 | #### Tile data 212 | 213 | The {1, X, Y} value contains three bytes followed by the data for 32×32 214 | tiles. The purpose of the three bytes is currently unknown. 215 | 216 | A single tile is made up of 30 bytes of binary data: 217 | 218 | | Field # | Bytes | Type | Description | 219 | | ------: | ----: | --------- | ------------------------------- | 220 | | 1 | 1–2 | `int16` | Foreground material¹ | 221 | | 2 | 3 | `uint8` | Foreground hue shift | 222 | | 3 | 4 | `uint8` | Foreground color variant | 223 | | 4 | 5–6 | `int16` | Foreground mod | 224 | | 5 | 7 | `uint8` | Foreground mod hue shift | 225 | | 6 | 8–9 | `int16` | Background material¹ | 226 | | 7 | 10 | `uint8` | Background hue shift | 227 | | 8 | 11 | `uint8` | Background color variant | 228 | | 9 | 12–13 | `int16` | Background mod | 229 | | 10 | 14 | `uint8` | Background mod hue shift | 230 | | 11 | 15 | `uint8` | Liquid | 231 | | 12 | 16–19 | `float` | Liquid level | 232 | | 13 | 20–23 | `float` | Liquid pressure | 233 | | 14 | 24 | `bool` | Liquid is infinite | 234 | | 15 | 25 | `uint8` | Collision map² | 235 | | 16 | 26–27 | `uint16` | Dungeon ID³ | 236 | | 17 | 28 | `uint8` | "Biome"⁴ | 237 | | 18 | 29 | `uint8` | "Environment Biome"⁴ | 238 | | 19 | 30 | `bool` | Indestructible (tree/vine base) | 239 | | 20 | 31 | `unknown` | Unknown? | 240 | 241 | ¹ Refers to a material by its id. Additional constants: 242 | 243 | | Constant | Meaning | 244 | | -------: | --------------------------------------- | 245 | | -36 | Unknown (seen on ships) | 246 | | -9 | Unknown (possibly dungeon related) | 247 | | -8 | Unknown (possibly dungeon related) | 248 | | -7 | Unknown (possibly dungeon related) | 249 | | -3 | Not placeable | 250 | | -2 | Not generated (or outside world bounds) | 251 | | -1 | Empty | 252 | 253 | ² Used by the game to block the player's movement. Constants: 254 | 255 | | Constant | Meaning | 256 | | -------: | --------------------------------------------- | 257 | | 1 | Empty space | 258 | | 2 | Platform (floor that player can pass through) | 259 | | 3 | Dynamic (e.g., closed door) | 260 | | 5 | Solid | 261 | 262 | ³ Dungeon info is stored in the world metadata. Additional constants: 263 | 264 | | Constant | Meaning | 265 | | -------: | ---------------------------- | 266 | | 65,531 | Tile removed by player | 267 | | 65,532 | Tile placed by player | 268 | | 65,533 | Microdungeon | 269 | | 65,535 | Not associated with anything | 270 | 271 | ⁴ Unverified, simply quoting from this page: http://seancode.com/galileo/format/tile.html 272 | 273 | #### Entity data 274 | 275 | Values for {2, X, Y} keys are a sequence of various entities in the 276 | region at (X, Y). Each entity is a versioned JSON object. 277 | 278 | The data, once inflated, consists of an SBON varint for the count and 279 | then the versioned JSON objects, one after the other. 280 | 281 | [vlq]: https://en.wikipedia.org/wiki/Variable-length_quantity 282 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | The MIT License (MIT) 2 | 3 | Copyright (c) 2016 Blixt 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy of 6 | this software and associated documentation files (the "Software"), to deal in 7 | the Software without restriction, including without limitation the rights to 8 | use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of 9 | the Software, and to permit persons to whom the Software is furnished to do so, 10 | subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS 17 | FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR 18 | COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER 19 | IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN 20 | CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 21 | -------------------------------------------------------------------------------- /MANIFEST.in: -------------------------------------------------------------------------------- 1 | include FORMATS.md 2 | include README.md 3 | include LICENSE 4 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Starbound utilities for Python 2 | 3 | This is a library to parse Starbound's file formats which are used to 4 | store worlds, player characters, assets, etc. 5 | 6 | Feel free to contribute either via submitting pull requests or writing 7 | up issues with suggestions and/or bugs. 8 | 9 | ## File & data formats 10 | 11 | Check out [FORMATS.md](./FORMATS.md) for technical information on 12 | Starbound's file and data formats. 13 | 14 | ## Installation 15 | 16 | py-starbound can be installed (either to your system, user account, or 17 | virtualenv) using the usual `setup.py` script: 18 | 19 | ```bash 20 | $ python setup.py install 21 | ``` 22 | 23 | After installation, the commandline utilities (described below) should 24 | be available in your `$PATH` can can be run like any other app: 25 | 26 | ```bash 27 | $ pystarbound-export [args] 28 | $ pystarbound-region [args] 29 | ``` 30 | 31 | If you wish to run these utilities from the git checkout itself (without 32 | installing first), the syntax is slightly more verbose: 33 | 34 | ```bash 35 | $ python -m starbound.cliexport [args] 36 | $ python -m starbound.cliregion [args] 37 | ``` 38 | 39 | ## Command line utilities 40 | 41 | ### Extracting `.pak` files 42 | 43 | You can use the `pystarbound-export` script to extract all the files in a `.pak` 44 | (or `.modpak`) file. 45 | 46 | Example: 47 | 48 | ```bash 49 | $ pystarbound-export -d assets /Starbound/assets/packed.pak 50 | ``` 51 | 52 | Or from the git checkout directly: 53 | 54 | ```bash 55 | $ python -m starbound.cliexport -d assets /Starbound/assets/packed.pak 56 | ``` 57 | 58 | ### Getting world info 59 | 60 | If you want information about a region in a world (planet or ship), you 61 | can use the `region.py` script. For example, here's how to pretty print 62 | the tiles in a region: 63 | 64 | ```bash 65 | $ pystarbound-region /Starbound/storage/universe/-382912739_-582615456_-73870035_3.world 66 | World size: 3000×2000 67 | Spawn point: (1224.0, 676.0) 68 | Outputting region: (37, 21) 69 | Outputting value: foreground_material 70 | ``` 71 | 72 | Or from the git checkout directly: 73 | 74 | ```bash 75 | $ python -m starbound.cliregion /Starbound/storage/universe/-382912739_-582615456_-73870035_3.world 76 | ``` 77 | 78 | Outputs something like this: 79 | 80 | ![](http://i.imgur.com/b4ZitYX.png) 81 | 82 | If you don't provide X and Y coordinates after the path, it will 83 | default to the region that the spawn point is in. 84 | 85 | You can also output specific tile values (instead of the foreground) 86 | using `--value-index` (or `-v`): 87 | 88 | ```bash 89 | $ pystarbound-region --value-index=12 /Starbound/storage/universe/-382912739_-582615456_-73870035_3.world 69 27 90 | World size: 3000×2000 91 | Spawn point: (1224.0, 676.0) 92 | Outputting region: (69, 27) 93 | Outputting value: liquid_pressure 94 | ``` 95 | 96 | Outputs something like this: 97 | 98 | ![](http://i.imgur.com/XZ3OYTO.png) 99 | 100 | And here's how to print the entities in a region: 101 | 102 | ```bash 103 | $ pystarbound-region --entities /Starbound/storage/universe/-382912739_-582615456_-73870035_3.world 69 27 104 | World size: 3000×2000 105 | Spawn point: (1224.0, 676.0) 106 | Outputting region: (69, 27) 107 | 108 | [ 109 | [ 110 | "ObjectEntity", 111 | 8, 112 | { 113 | "direction": "left", 114 | "inputWireNodes": [], 115 | "interactive": true, 116 | "name": "wiringstation", 117 | "orientationIndex": 0, 118 | "outputWireNodes": [], 119 | "parameters": { 120 | "owner": "916d5878483e3a40d10467dc419982c2" 121 | }, 122 | "scriptStorage": {}, 123 | ... 124 | ``` 125 | 126 | ## Using the Python package 127 | 128 | The Python package lets you read data from Starbound's various file 129 | formats. The classes and functions expect file objects to read from. 130 | 131 | You can use the `mmap` package to improve performance for large files, 132 | such as `packed.pak` and world files. 133 | 134 | ### Example: Reading a player file 135 | 136 | Here's how to print the name of a player: 137 | 138 | ```python 139 | import starbound 140 | 141 | with open('player/11475cedd80ead373c19a91de2e2c4d3.player', 'rb') as fh: 142 | player = starbound.read_sbvj01(fh) 143 | print('Hello, {}!'.format(player.data['identity']['name'])) 144 | ``` 145 | 146 | ### Example: World files 147 | 148 | In the following example the `mmap` package is used for faster access: 149 | 150 | ```python 151 | import mmap, starbound 152 | 153 | with open('universe/43619853_198908799_-9440367_6_3.world', 'rb') as fh: 154 | mm = mmap.mmap(fh.fileno(), 0, access=mmap.ACCESS_READ) 155 | 156 | world = starbound.World(mm) 157 | world.read_metadata() 158 | 159 | print('World size: {}×{}'.format(world.width, world.height)) 160 | x, y = world.metadata['playerStart'] 161 | print('Player spawns at ({}, {})'.format(x, y)) 162 | 163 | # Regions consist of 32×32 tiles. 164 | rx, ry = x // 32, y // 32 165 | print('An entity: {}'.format(world.get_entities(rx, ry)[0])) 166 | ``` 167 | 168 | ### Example: Easy access to various world attributes 169 | 170 | A vast amount of information about loaded Worlds is available via the 171 | `metadata` attribute (as seen in the above section), but some 172 | information is also abstracted out into an `info` attribute. For instance: 173 | 174 | ```python 175 | world = starbound.World(fh) 176 | print('World Name: {}'.format(world.info.name)) 177 | print('World Description: {}'.format(world.info.description)) 178 | print('World Coordinates: ({}, {})'.format(world.info.coords[0], world.info.coords[1])) 179 | ``` 180 | 181 | The full list of attributes currently available are: 182 | 183 | | Attribute | Description | 184 | | -------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- | 185 | | `biomes` | The full set of biomes found on the world. This should be a complete list, regardless of how much of the world has been explored. | 186 | | `coords` | World coordinates, as a tuple. The first two elements are the in-map coordinates of the system, the third is effectively random but describes the world itself. | 187 | | `description` | The internal description of the world. Will often include text describing the tier of the world. | 188 | | `dungeons` | The full set of dungeons found on the world. This should be a complete list, regardless of how much of the world has been explored. | 189 | | `name` | The name of the world. Will often include Starbound coloration markup. | 190 | | `size` | A tuple describing the width and height of the world. | 191 | | `world_biomes` | A set of the main biome IDs of the world, of the sort reported in the ingame navigation screen. | 192 | 193 | ### Example: Finding an entity by UUID/ID 194 | 195 | Many entities in Starbound, such as bookmarked flags, mech beacons, 196 | quest markers, etc, have UUIDs or IDs which the game can use to find 197 | where they are in the map without having to have all regions loaded. 198 | Player bookmark UUIDs can be found in the `player.data['universeMap']` 199 | dict, underneath `teleportBookmarks`. One object type which does 200 | _not_ use UUIDs is a level's mech beacon, which instead uses the magic 201 | string `mechbeacon`. To find the ingame coordinates for a level's 202 | beacon (if one is present), this can be used: 203 | 204 | ```python 205 | mechbeacon_coords = world.get_entity_uuid_coords('mechbeacon') 206 | if mechbeacon_coords: 207 | print('Mech beacon found at ({}, {})'.format(*mechbeacon_coords)) 208 | else: 209 | print('No mech beacon in level!') 210 | ``` 211 | 212 | ### Example: Getting assets from `packed.pak` 213 | 214 | Starbound keeps most of the assets (images, configuration files, 215 | dungeons, etc.) in a file called `packed.pak`. This file uses a special 216 | format which can be read by py-starbound, as you can see below. 217 | 218 | ```python 219 | import starbound 220 | 221 | with open('assets/packed.pak', 'rb') as fh: 222 | package = starbound.SBAsset6(fh) 223 | 224 | # Print the contents of a file in the asset package. 225 | print(package.get('/lighting.config')) 226 | ``` 227 | 228 | ### Example: Modifying Starbound files 229 | 230 | Currently, only the SBVJ01 file format can be written by py-starbound. 231 | This means player files, client context files, and the statistics file. 232 | 233 | Here's an example that renames a player (WARNING: Always back up files 234 | before writing to them!): 235 | 236 | ```python 237 | import starbound 238 | 239 | with open('player/420ed511f83b3760dead42a173339b3e.player', 'r+b') as fh: 240 | player = starbound.read_sbvj01(fh) 241 | 242 | old_name = player.data['identity']['name'] 243 | new_name = old_name.encode('rot13') 244 | player.data['identity']['name'] = new_name 245 | print('Updating name: {} -> {}'.format(old_name, new_name)) 246 | 247 | # Go back to the beginning of the file and write the updated data. 248 | fh.seek(0) 249 | starbound.write_sbvj01(fh, player) 250 | # If the file got shorter, truncate away the remaining content. 251 | fh.truncate() 252 | ``` 253 | 254 | ## License 255 | 256 | [MIT License](./LICENSE) 257 | -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # -*- coding: utf-8 -*- 3 | 4 | from setuptools import find_packages, setup 5 | from starbound import __version__ 6 | 7 | 8 | def readme(): 9 | with open('README.md') as f: 10 | return f.read() 11 | 12 | 13 | setup( 14 | name='py-starbound', 15 | version=__version__, 16 | packages=find_packages(), 17 | include_package_data=True, 18 | license='MIT License', 19 | description='Python package for working with Starbound files.', 20 | long_description=readme(), 21 | long_description_content_type='text/markdown', 22 | url='https://github.com/blixt/py-starbound', 23 | author='Blixt', 24 | author_email='me@blixt.nyc', 25 | # Shouldn't have any deps other than Python itself 26 | install_requires=[ 27 | ], 28 | # https://pypi.python.org/pypi?%3Aaction=list_classifiers 29 | classifiers=[ 30 | 'Development Status :: 5 - Production/Stable', 31 | 'Environment :: Console', 32 | 'Intended Audience :: Developers', 33 | 'Intended Audience :: End Users/Desktop', 34 | 'License :: OSI Approved :: MIT License', 35 | 'Natural Language :: English', 36 | 'Operating System :: OS Independent', 37 | 'Programming Language :: Python', 38 | 'Programming Language :: Python :: 2', 39 | 'Programming Language :: Python :: 3', 40 | 'Topic :: Games/Entertainment', 41 | 'Topic :: Utilities', 42 | ], 43 | entry_points={ 44 | 'console_scripts': [ 45 | 'pystarbound-region = starbound.cliregion:main', 46 | 'pystarbound-repair = starbound.clirepair:main', 47 | 'pystarbound-export = starbound.cliexport:main', 48 | ], 49 | }, 50 | ) 51 | -------------------------------------------------------------------------------- /starbound/__init__.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | 3 | from collections import namedtuple 4 | import hashlib 5 | import io 6 | import struct 7 | import zlib 8 | 9 | from . import sbon 10 | from .btreedb5 import BTreeDB5 11 | from .sbasset6 import SBAsset6 12 | 13 | __version__ = '1.0.0' 14 | 15 | # Override range with xrange when running Python 2.x. 16 | try: 17 | range = xrange 18 | except: 19 | pass 20 | 21 | 22 | # Utility descriptor for memoized properties. 23 | class lazyproperty(object): 24 | def __init__(self, fget): 25 | self.fget = fget 26 | self.__doc__ = fget.__doc__ 27 | self.propname = '_lazyproperty_{}'.format(self.fget.__name__) 28 | 29 | def __delete__(self, obj): 30 | if hasattr(obj, self.propname): 31 | delattr(obj, self.propname) 32 | 33 | def __get__(self, obj, objtype=None): 34 | if obj is None: 35 | return self 36 | if not hasattr(obj, self.propname): 37 | setattr(obj, self.propname, self.fget(obj)) 38 | return getattr(obj, self.propname) 39 | 40 | def __set__(self, obj, value): 41 | setattr(obj, self.propname, value) 42 | 43 | 44 | class CelestialChunks(BTreeDB5): 45 | def get(self, key): 46 | key = hashlib.sha256(key.encode('utf-8')).digest() 47 | data = super(CelestialChunks, self).get(key) 48 | data = zlib.decompress(data) 49 | stream = io.BytesIO(data) 50 | return read_versioned_json(stream) 51 | 52 | def read_header(self): 53 | super(CelestialChunks, self).read_header() 54 | assert self.name == 'Celestial2', 'Invalid header' 55 | 56 | 57 | Tile = namedtuple('Tile', [ 58 | 'foreground_material', 59 | 'foreground_hue_shift', 60 | 'foreground_variant', 61 | 'foreground_mod', 62 | 'foreground_mod_hue_shift', 63 | 'background_material', 64 | 'background_hue_shift', 65 | 'background_variant', 66 | 'background_mod', 67 | 'background_mod_hue_shift', 68 | 'liquid', 69 | 'liquid_level', 70 | 'liquid_pressure', 71 | 'liquid_infinite', 72 | 'collision', 73 | 'dungeon_id', 74 | 'biome', 75 | 'biome_2', 76 | 'indestructible', 77 | ]) 78 | 79 | 80 | VersionedJSON = namedtuple('VersionedJSON', ['name', 'version', 'data']) 81 | 82 | 83 | class World(BTreeDB5): 84 | @lazyproperty 85 | def info(self): 86 | if not hasattr(self, 'metadata'): 87 | self.read_metadata() 88 | return WorldInfo(self.metadata) 89 | 90 | def get(self, layer, x, y): 91 | # World keys are based on a layer followed by X and Y coordinates. 92 | data = super(World, self).get(struct.pack('>BHH', layer, x, y)) 93 | return zlib.decompress(data) 94 | 95 | def get_all_regions_with_tiles(self): 96 | """ 97 | Generator which yields a set of (rx, ry) tuples which describe 98 | all regions for which the world has tile data 99 | """ 100 | for key in self.get_all_keys(): 101 | (layer, rx, ry) = struct.unpack('>BHH', key) 102 | if layer == 1: 103 | yield (rx, ry) 104 | 105 | def get_entities(self, x, y): 106 | stream = io.BytesIO(self.get(2, x, y)) 107 | count = sbon.read_varint(stream) 108 | return [read_versioned_json(stream) for _ in range(count)] 109 | 110 | def get_entity_uuid_coords(self, uuid): 111 | """ 112 | Returns the coordinates of the given entity UUID inside this world, or 113 | `None` if the UUID is not found. 114 | """ 115 | if uuid in self._entity_to_region_map: 116 | coords = self._entity_to_region_map[uuid] 117 | entities = self.get_entities(*coords) 118 | for entity in entities: 119 | if 'uniqueId' in entity.data and entity.data['uniqueId'] == uuid: 120 | return tuple(entity.data['tilePosition']) 121 | return None 122 | 123 | def get_tiles(self, x, y): 124 | stream = io.BytesIO(self.get(1, x, y)) 125 | # TODO: Figure out what this means. 126 | unknown = stream.read(3) 127 | # There are 1024 (32x32) tiles in a region. 128 | return [self.read_tile(stream) for _ in range(1024)] 129 | 130 | def read_header(self): 131 | super(World, self).read_header() 132 | assert self.name == 'World4', 'Not a World4 file' 133 | 134 | def read_metadata(self): 135 | # World metadata is held at a special layer/x/y combination. 136 | stream = io.BytesIO(self.get(0, 0, 0)) 137 | self.width, self.height = struct.unpack('>ii', stream.read(8)) 138 | name, version, data = read_versioned_json(stream) 139 | assert name == 'WorldMetadata', 'Invalid world data' 140 | self.metadata = data 141 | self.metadata_version = version 142 | 143 | @classmethod 144 | def read_tile(cls, stream): 145 | values = struct.unpack('>hBBhBhBBhBBffBBHBB?x', stream.read(31)) 146 | return Tile(*values) 147 | 148 | @lazyproperty 149 | def _entity_to_region_map(self): 150 | """ 151 | A dict whose keys are the UUIDs (or just IDs, in some cases) of 152 | entities, and whose values are the `(rx, ry)` coordinates in which that 153 | entity can be found. This can be used to easily locate particular 154 | entities inside the world. 155 | """ 156 | entity_to_region = {} 157 | for key in self.get_all_keys(): 158 | layer, rx, ry = struct.unpack('>BHH', key) 159 | if layer != 4: 160 | continue 161 | stream = io.BytesIO(self.get(layer, rx, ry)) 162 | num_entities = sbon.read_varint(stream) 163 | for _ in range(num_entities): 164 | uuid = sbon.read_string(stream) 165 | if uuid in entity_to_region: 166 | raise ValueError('Duplicate UUID {}'.format(uuid)) 167 | entity_to_region[uuid] = (rx, ry) 168 | return entity_to_region 169 | 170 | 171 | class WorldInfo(object): 172 | """ 173 | Convenience class to provide some information about a World without having 174 | to know which keys to look at. 175 | """ 176 | 177 | def __init__(self, metadata): 178 | self.metadata = metadata 179 | 180 | @property 181 | def biomes(self): 182 | """ 183 | Returns a set of all biomes found in the world. This should be a 184 | complete list even if the world isn't fully-explored. 185 | """ 186 | return self._worldParameters.biomes 187 | 188 | @property 189 | def coords(self): 190 | """ 191 | The coordinates of the system. The first two elements of the tuple will 192 | be the `(x, y)` coordinates in the universe map, and the third is 193 | largely useless. 194 | """ 195 | return self._celestialParameters.coords 196 | 197 | @property 198 | def description(self): 199 | """ 200 | A description of the world - will include a "Tier" ranking for 201 | planets/moons. 202 | """ 203 | return self._celestialParameters.description 204 | 205 | @property 206 | def dungeons(self): 207 | """ 208 | Returns a set of all dungeons found in the world. This should be a 209 | complete list even if the world isn't fully-explored. 210 | """ 211 | return self._worldParameters.dungeons 212 | 213 | @property 214 | def name(self): 215 | """ 216 | The name of the world. Note that this will often include coloration 217 | markup. 218 | """ 219 | return self._celestialParameters.name 220 | 221 | @lazyproperty 222 | def size(self): 223 | """ 224 | The size of the world, as a tuple. 225 | """ 226 | return tuple(self.metadata.get('worldTemplate', {})['size']) 227 | 228 | @property 229 | def world_biomes(self): 230 | """ 231 | A set of main biomes which define the world as a whole. This will be a 232 | much shorter list than the full list of biomes found in the world -- 233 | generally only a couple of entries. 234 | """ 235 | return self._celestialParameters.biomes 236 | 237 | @lazyproperty 238 | def _celestialParameters(self): 239 | t = namedtuple('celestialParameters', 'name description coords biomes') 240 | name = None 241 | description = None 242 | coords = None 243 | biomes = set() 244 | cp = self.metadata.get('worldTemplate', {}).get('celestialParameters') 245 | if cp: 246 | name = cp.get('name') 247 | if 'parameters' in cp: 248 | description = cp['parameters'].get('description') 249 | if 'terrestrialType' in cp['parameters']: 250 | biomes.update(cp['parameters']['terrestrialType']) 251 | if 'coordinate' in cp and 'location' in cp['coordinate']: 252 | coords = tuple(cp['coordinate']['location']) 253 | return t(name, description, coords, biomes) 254 | 255 | @lazyproperty 256 | def _worldParameters(self): 257 | t = namedtuple('worldParameters', 'biomes dungeons') 258 | biomes = set() 259 | dungeons = set() 260 | wp = self.metadata.get('worldTemplate', {}).get('worldParameters') 261 | if wp: 262 | SCAN_LAYERS = [ 263 | ('atmosphereLayer', False), 264 | ('coreLayer', False), 265 | ('spaceLayer', False), 266 | ('subsurfaceLayer', False), 267 | ('surfaceLayer', False), 268 | ('undergroundLayers', True), 269 | ] 270 | for name, is_list in SCAN_LAYERS: 271 | if name not in wp: 272 | continue 273 | layers = wp[name] if is_list else [wp[name]] 274 | for layer in layers: 275 | dungeons.update(layer['dungeons']) 276 | for label in ['primaryRegion', 'primarySubRegion']: 277 | biomes.add(layer[label]['biome']) 278 | for label in ['secondaryRegions', 'secondarySubRegions']: 279 | for inner_region in layer[label]: 280 | biomes.add(inner_region['biome']) 281 | return t(biomes, dungeons) 282 | 283 | 284 | def read_sbvj01(stream): 285 | assert stream.read(6) == b'SBVJ01', 'Invalid header' 286 | return read_versioned_json(stream) 287 | 288 | 289 | def read_versioned_json(stream): 290 | name = sbon.read_string(stream) 291 | # The object only has a version if the following bool is true. 292 | if stream.read(1) == b'\x00': 293 | version = None 294 | else: 295 | version, = struct.unpack('>i', stream.read(4)) 296 | data = sbon.read_dynamic(stream) 297 | return VersionedJSON(name, version, data) 298 | 299 | 300 | def write_sbvj01(stream, vj): 301 | stream.write(b'SBVJ01') 302 | write_versioned_json(stream, vj) 303 | 304 | 305 | def write_versioned_json(stream, vj): 306 | sbon.write_string(stream, vj.name) 307 | if vj.version is None: 308 | stream.write(struct.pack('>b', 0)) 309 | else: 310 | stream.write(struct.pack('>bi', 1, vj.version)) 311 | sbon.write_dynamic(stream, vj.data) 312 | -------------------------------------------------------------------------------- /starbound/btreedb5.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | 3 | import binascii 4 | import io 5 | import struct 6 | 7 | from starbound import sbon 8 | 9 | 10 | # Override range with xrange when running Python 2.x. 11 | try: 12 | range = xrange 13 | except: 14 | pass 15 | 16 | 17 | HEADER = '>8si16si?ixxxxii?ixxxxii?445x' 18 | HEADER_SIZE = struct.calcsize(HEADER) 19 | # Constants for the different block types. 20 | FREE = b'FF' 21 | INDEX = b'II' 22 | LEAF = b'LL' 23 | 24 | 25 | class BTreeDB5(object): 26 | def __init__(self, stream): 27 | self.stream = stream 28 | 29 | def get(self, key): 30 | if not hasattr(self, 'key_size'): 31 | self.read_header() 32 | assert len(key) == self.key_size, 'Invalid key length' 33 | # Traverse the B-tree until we reach a leaf. 34 | offset = HEADER_SIZE + self.block_size * self.root_block 35 | entry_size = self.key_size + 4 36 | s = self.stream 37 | while True: 38 | s.seek(offset) 39 | block_type = s.read(2) 40 | if block_type != INDEX: 41 | break 42 | # Read the index header and scan for the closest key. 43 | lo, (_, hi, block) = 0, struct.unpack('>Bii', s.read(9)) 44 | offset += 11 45 | while lo < hi: 46 | mid = (lo + hi) // 2 47 | s.seek(offset + entry_size * mid) 48 | if key < s.read(self.key_size): 49 | hi = mid 50 | else: 51 | lo = mid + 1 52 | if lo > 0: 53 | s.seek(offset + entry_size * (lo - 1) + self.key_size) 54 | block, = struct.unpack('>i', s.read(4)) 55 | offset = HEADER_SIZE + self.block_size * block 56 | assert block_type == LEAF, 'Did not reach a leaf' 57 | # Scan leaves for the key, then read the data. 58 | reader = LeafReader(self) 59 | num_keys, = struct.unpack('>i', reader.read(4)) 60 | for i in range(num_keys): 61 | cur_key = reader.read(self.key_size) 62 | length = sbon.read_varint(reader) 63 | if key == cur_key: 64 | return reader.read(length) 65 | reader.seek(length, 1) 66 | # None of the keys in the leaf node matched. 67 | raise KeyError(binascii.hexlify(key)) 68 | 69 | def get_all_keys(self, start=None): 70 | """ 71 | A generator which yields a list of all valid keys starting at the 72 | given `start` offset. If `start` is `None`, we will start from 73 | the root of the tree. 74 | """ 75 | s = self.stream 76 | if not start: 77 | start = HEADER_SIZE + self.block_size * self.root_block 78 | s.seek(start) 79 | block_type = s.read(2) 80 | if block_type == LEAF: 81 | reader = LeafReader(self) 82 | num_keys = struct.unpack('>i', reader.read(4))[0] 83 | for _ in range(num_keys): 84 | cur_key = reader.read(self.key_size) 85 | # We to a tell/seek here so that the user can read from 86 | # the file while this loop is still being run 87 | cur_pos = s.tell() 88 | yield cur_key 89 | s.seek(cur_pos) 90 | length = sbon.read_varint(reader) 91 | reader.seek(length, 1) 92 | elif block_type == INDEX: 93 | (_, num_keys, first_child) = struct.unpack('>Bii', s.read(9)) 94 | children = [first_child] 95 | for _ in range(num_keys): 96 | # Skip the key field. 97 | _ = s.read(self.key_size) 98 | # Read pointer to the child block. 99 | next_child = struct.unpack('>i', s.read(4))[0] 100 | children.append(next_child) 101 | for child_loc in children: 102 | for key in self.get_all_keys(HEADER_SIZE + self.block_size * child_loc): 103 | yield key 104 | elif block_type == FREE: 105 | pass 106 | else: 107 | raise Exception('Unhandled block type: {}'.format(block_type)) 108 | 109 | def read_header(self): 110 | self.stream.seek(0) 111 | data = struct.unpack(HEADER, self.stream.read(HEADER_SIZE)) 112 | assert data[0] == b'BTreeDB5', 'Invalid header' 113 | self.block_size = data[1] 114 | self.name = data[2].rstrip(b'\0').decode('utf-8') 115 | self.key_size = data[3] 116 | self.use_other_root = data[4] 117 | self.free_block_1 = data[5] 118 | self.free_block_1_end = data[6] 119 | self.root_block_1 = data[7] 120 | self.root_block_1_is_leaf = data[8] 121 | self.free_block_2 = data[9] 122 | self.free_block_2_end = data[10] 123 | self.root_block_2 = data[11] 124 | self.root_block_2_is_leaf = data[12] 125 | 126 | @property 127 | def root_block(self): 128 | return self.root_block_2 if self.use_other_root else self.root_block_1 129 | 130 | @property 131 | def root_block_is_leaf(self): 132 | if self.use_other_root: 133 | return self.root_block_2_is_leaf 134 | else: 135 | return self.root_block_1_is_leaf 136 | 137 | def swap_root(self): 138 | self.use_other_root = not self.use_other_root 139 | 140 | 141 | class LeafReader(object): 142 | def __init__(self, db): 143 | # The stream offset must be right after an "LL" marker. 144 | self.db = db 145 | self.offset = 2 146 | 147 | def read(self, size=-1): 148 | if size < 0: 149 | raise NotImplemented('Can only read specific amount') 150 | with io.BytesIO() as data: 151 | for length in self._traverse(size): 152 | data.write(self.db.stream.read(length)) 153 | return data.getvalue() 154 | 155 | def seek(self, offset, whence=0): 156 | if whence != 1 or offset < 0: 157 | raise NotImplemented('Can only seek forward relatively') 158 | for length in self._traverse(offset): 159 | self.db.stream.seek(length, 1) 160 | 161 | def _traverse(self, length): 162 | block_end = self.db.block_size - 4 163 | while True: 164 | if self.offset + length <= block_end: 165 | yield length 166 | self.offset += length 167 | break 168 | delta = block_end - self.offset 169 | yield delta 170 | block, = struct.unpack('>i', self.db.stream.read(4)) 171 | assert block >= 0, 'Could not traverse to next block' 172 | self.db.stream.seek(HEADER_SIZE + self.db.block_size * block) 173 | assert self.db.stream.read(2) == LEAF, 'Did not reach a leaf' 174 | self.offset = 2 175 | length -= delta 176 | -------------------------------------------------------------------------------- /starbound/cliexport.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # -*- coding: utf-8 -*- 3 | 4 | import mmap 5 | import optparse 6 | import os 7 | import sys 8 | import time 9 | 10 | import starbound 11 | 12 | 13 | def main(): 14 | p = optparse.OptionParser('Usage: %prog ') 15 | p.add_option('-d', '--destination', dest='path', 16 | help='Destination directory') 17 | options, arguments = p.parse_args() 18 | # Validate the arguments. 19 | if len(arguments) != 1: 20 | p.error('Only one argument is supported (package path)') 21 | package_path = arguments[0] 22 | base = options.path if options.path else '.' 23 | # Load the assets file and its index. 24 | start = time.clock() 25 | with open(package_path, 'rb') as fh: 26 | mm = mmap.mmap(fh.fileno(), 0, access=mmap.ACCESS_READ) 27 | package = starbound.SBAsset6(mm) 28 | print('Loading index...') 29 | # Get the paths from the index in the database. 30 | package.read_index() 31 | print('Index loaded. Extracting {} files...'.format(package.file_count)) 32 | # Start extracting everything. 33 | num_files = 0 34 | percentage_count = max(package.file_count // 100, 1) 35 | for path in package.index: 36 | dest_path = base + path 37 | dir_path = os.path.dirname(dest_path) 38 | if not os.path.exists(dir_path): 39 | os.makedirs(dir_path) 40 | try: 41 | data = package.get(path) 42 | except: 43 | # Break the dots in case std{out,err} are the same tty: 44 | sys.stdout.write('\n') 45 | sys.stdout.flush() 46 | print >>sys.stderr, 'W: Failed to read', path 47 | continue 48 | with open(dest_path, 'wb') as file: 49 | file.write(data) 50 | num_files += 1 51 | if not num_files % percentage_count: 52 | sys.stdout.write('.') 53 | sys.stdout.flush() 54 | elapsed = time.clock() - start 55 | print('') 56 | print('Extracted {} files in {:.1f} seconds.'.format(num_files, elapsed)) 57 | 58 | 59 | if __name__ == '__main__': 60 | main() 61 | -------------------------------------------------------------------------------- /starbound/cliregion.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # -*- coding: utf-8 -*- 3 | 4 | from __future__ import print_function 5 | 6 | import hashlib 7 | import json 8 | import mmap 9 | import optparse 10 | import signal 11 | 12 | import starbound 13 | 14 | 15 | try: 16 | # Don't break on pipe signal. 17 | signal.signal(signal.SIGPIPE, signal.SIG_DFL) 18 | except: 19 | # Probably a Windows machine. 20 | pass 21 | 22 | 23 | def main(): 24 | p = optparse.OptionParser('Usage: %prog [ ]') 25 | p.add_option('-e', '--entities', dest='entities', 26 | action='store_true', default=False, 27 | help='Output entity data instead of tile data') 28 | p.add_option('-v', '--value-index', dest='value_index', 29 | type=int, default=0, 30 | help='The value in the tile data to output') 31 | options, arguments = p.parse_args() 32 | # Get the path and coordinates from arguments. 33 | if len(arguments) == 1: 34 | path = arguments[0] 35 | x, y = None, None 36 | elif len(arguments) == 3: 37 | path, x, y = arguments 38 | x, y = int(x), int(y) 39 | else: 40 | p.error('Incorrect number of arguments') 41 | # Load up the world file. 42 | with open(path, 'rb') as fh: 43 | mm = mmap.mmap(fh.fileno(), 0, access=mmap.ACCESS_READ) 44 | world = starbound.World(mm) 45 | world.read_metadata() 46 | spawn = world.metadata['playerStart'] 47 | # Default coordinates to spawn point. 48 | if x is None or y is None: 49 | x, y = int(spawn[0] / 32), int(spawn[1] / 32) 50 | # Print world metadata. 51 | print('World size: {}×{}'.format(world.width, world.height)) 52 | print('Metadata version: {}'.format(world.metadata_version)) 53 | if spawn: 54 | print('Spawn point: ({}, {})'.format(spawn[0], spawn[1])) 55 | print('') 56 | # Print either entities or tile data depending on options. 57 | if options.entities: 58 | entities = [{'type': e.name, 'version': e.version, 'data': e.data} 59 | for e in world.get_entities(x, y)] 60 | print('Entities in region ({}, {}):'.format(x, y)) 61 | print(json.dumps(entities, indent=2, separators=(',', ': '), sort_keys=True)) 62 | else: 63 | try: 64 | print('Tiles ({}) in region ({}, {}):'.format( 65 | starbound.Tile._fields[options.value_index], x, y)) 66 | except: 67 | print('Unsupported value index! Pick one of these indices:') 68 | index = 0 69 | for field in starbound.Tile._fields: 70 | print('> {} ({})'.format(index, field)) 71 | index += 1 72 | else: 73 | pretty_print_tiles(world, x, y, options.value_index) 74 | 75 | 76 | _fraction_to_string = ( 77 | (1.0 / 2, '½'), 78 | (1.0 / 3, '⅓'), 79 | (1.0 / 4, '¼'), 80 | (1.0 / 5, '⅕'), 81 | (1.0 / 6, '⅙'), 82 | (1.0 / 8, '⅛'), 83 | (2.0 / 3, '⅔'), 84 | (2.0 / 5, '⅖'), 85 | (3.0 / 4, '¾'), 86 | (3.0 / 5, '⅗'), 87 | (3.0 / 8, '⅜'), 88 | (4.0 / 5, '⅘'), 89 | (5.0 / 6, '⅚'), 90 | (5.0 / 8, '⅝'), 91 | (7.0 / 8, '⅞'), 92 | (1.0 / 100, '.'), 93 | ) 94 | 95 | 96 | def fraction_to_string(number): 97 | fraction = number - int(number) 98 | string = '?' 99 | min_diff = 1.0 100 | for value, character in _fraction_to_string: 101 | diff = abs(fraction - value) 102 | if diff < min_diff: 103 | min_diff = diff 104 | string = character 105 | return string 106 | 107 | 108 | def get_colors(value): 109 | # More complicated due to Python 2/3 support. 110 | b = hashlib.md5(str(value).encode('utf-8')).digest()[1] 111 | x = ord(b) if isinstance(b, str) else b 112 | if x < 16: 113 | return x, 15 if x < 8 else 0 114 | elif x < 232: 115 | return x, 15 if (x - 16) % 36 < 18 else 0 116 | else: 117 | return x, 15 if x < 244 else 0 118 | 119 | 120 | def pretty_print_tiles(world, x, y, index=0): 121 | lines = [] 122 | line = '' 123 | for i, tile in enumerate(world.get_tiles(x, y)): 124 | # Create a new line after every 32 tiles. 125 | if i > 0 and i % 32 == 0: 126 | lines.append(line) 127 | line = '' 128 | value = tile[index] 129 | # Create a uniquely colored block with the tile value. 130 | if isinstance(value, float): 131 | v = '{:02X}{}'.format(int(value), fraction_to_string(value)) 132 | else: 133 | v = '{:03X}'.format(value) 134 | if len(v) > 3: 135 | v = '-…' + v[-1] if value < 0 else '…' + v[-2:] 136 | bg, fg = get_colors(value) 137 | line += '\033[48;5;{:d}m\033[38;5;{:d}m{}\033[000m'.format(bg, fg, v) 138 | lines.append(line) 139 | print('\n'.join(reversed(lines))) 140 | print('') 141 | 142 | 143 | if __name__ == '__main__': 144 | main() 145 | -------------------------------------------------------------------------------- /starbound/clirepair.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # -*- coding: utf-8 -*- 3 | 4 | from __future__ import print_function 5 | 6 | import io 7 | import math 8 | import optparse 9 | import os 10 | import os.path 11 | import signal 12 | import struct 13 | import zlib 14 | 15 | import starbound 16 | import starbound.btreedb5 17 | 18 | 19 | try: 20 | # Don't break on pipe signal. 21 | signal.signal(signal.SIGPIPE, signal.SIG_DFL) 22 | except: 23 | # Probably a Windows machine. 24 | pass 25 | 26 | 27 | # Override range with xrange when running Python 2.x. 28 | try: 29 | range = xrange 30 | except: 31 | pass 32 | 33 | 34 | def main(): 35 | p = optparse.OptionParser('Usage: %prog [options] ') 36 | p.add_option('-f', '--force', dest='force', 37 | action='store_true', default=False, 38 | help='ignore some errors') 39 | p.add_option('-o', '--output', dest='output', 40 | help='where to output repaired world (defaults to input file ' 41 | 'path with .repaired added to the end)') 42 | p.add_option('-w', '--blank-world', dest='world', 43 | help='the blank .world file that was created in place of the ' 44 | '.fail one (for metadata recovery)') 45 | options, arguments = p.parse_args() 46 | # Get the path from arguments. 47 | if len(arguments) != 1: 48 | p.error('incorrect number of arguments') 49 | try: 50 | fh = open(arguments[0], 'rb') 51 | file_size = os.fstat(fh.fileno()).st_size 52 | world = starbound.World(fh) 53 | except Exception as e: 54 | p.error('could not open fail file ({})'.format(e)) 55 | # Output path (defaults to fail file + .repaired). 56 | if options.output: 57 | out_name = options.output 58 | else: 59 | out_name = arguments[0] + '.repaired' 60 | # Ensure the user doesn't accidentally overwrite existing files. 61 | if os.path.isfile(out_name): 62 | if options.force: 63 | print('warning: overwriting existing file') 64 | else: 65 | p.error('"{}" already exists'.format(out_name)) 66 | # Allow user to use the fresh world for metadata (which should be the same). 67 | if options.world: 68 | fail_name = os.path.basename(arguments[0]) 69 | world_name = os.path.basename(options.world) 70 | if fail_name[:len(world_name)] != world_name: 71 | if options.force: 72 | print('warning: .fail and .world filenames do not match') 73 | else: 74 | p.error('.fail and .world filenames do not match') 75 | try: 76 | blank_world = starbound.World(open(options.world, 'rb')) 77 | except Exception as e: 78 | p.error('could not open blank world ({})'.format(e)) 79 | # This dict will contain all the keys and their data. 80 | data = dict() 81 | try: 82 | world.read_metadata() 83 | metadata, version = world.metadata, world.metadata_version 84 | except Exception as e: 85 | if options.world: 86 | try: 87 | print('warning: restoring metadata using blank world') 88 | blank_world.read_metadata() 89 | metadata, version = blank_world.metadata, blank_world.metadata_version 90 | except Exception as e: 91 | p.error('failed to restore metadata ({})'.format(e)) 92 | else: 93 | p.error('metadata section is corrupt ({})'.format(e)) 94 | try: 95 | size = metadata['worldTemplate']['size'] 96 | except Exception as e: 97 | size = [-1, -1] 98 | print('warning: failed to read world size ({})'.format(e)) 99 | regions_x = int(math.ceil(size[0] / 32)) 100 | regions_y = int(math.ceil(size[1] / 32)) 101 | print('attempting to recover {}×{} regions...'.format(regions_x, regions_y)) 102 | block_count = int((file_size - starbound.btreedb5.HEADER_SIZE) / world.block_size) 103 | blocks_per_percent = block_count // 100 + 1 104 | nodes_recovered = 0 105 | percent = 0 106 | # Find all leaves and try to read them individually. 107 | for index in range(block_count): 108 | if index % blocks_per_percent == 0: 109 | print('{}% ({} nodes recovered)'.format(percent, nodes_recovered)) 110 | percent += 1 111 | # Seek to the block and only process it if it's a leaf. 112 | world.stream.seek(starbound.btreedb5.HEADER_SIZE + world.block_size * index) 113 | if world.stream.read(2) != starbound.btreedb5.LEAF: 114 | continue 115 | stream = starbound.btreedb5.LeafReader(world) 116 | try: 117 | num_keys, = struct.unpack('>i', stream.read(4)) 118 | except Exception as e: 119 | print('failed to read keys of leaf block #{}: {}'.format(index, e)) 120 | continue 121 | # Ensure that the number of keys makes sense, otherwise skip the leaf. 122 | if num_keys > 100: 123 | continue 124 | for i in range(num_keys): 125 | try: 126 | cur_key = stream.read(world.key_size) 127 | cur_data = starbound.sbon.read_bytes(stream) 128 | except Exception as e: 129 | print('could not read key/data: {}'.format(e)) 130 | break 131 | layer, x, y = struct.unpack('>BHH', cur_key) 132 | # Skip this leaf if we encounter impossible indexes. 133 | if layer == 0 and (x != 0 or y != 0): 134 | break 135 | if layer not in (0, 1, 2) or x >= regions_x or y >= regions_y: 136 | break 137 | result = None 138 | if cur_key in data: 139 | # Duplicates should be checked up against the index, which always wins. 140 | # TODO: Make this code run again. 141 | try: 142 | #result = world.get(layer, x, y) 143 | result = None 144 | except Exception: 145 | world.swap_root() 146 | try: 147 | #result = world.get(layer, x, y) 148 | result = None 149 | except Exception: 150 | pass 151 | world.swap_root() 152 | # Use the data from this leaf if not using the index. 153 | if not result: 154 | try: 155 | result = zlib.decompress(cur_data) 156 | except Exception as e: 157 | print('broken leaf node: {}'.format(e)) 158 | continue 159 | # Validate the data before storing it. 160 | try: 161 | if layer == 0: 162 | temp_stream = io.BytesIO(result) 163 | temp_stream.seek(8) 164 | name, _, _ = starbound.read_versioned_json(temp_stream) 165 | assert name == 'WorldMetadata', 'broken world metadata' 166 | elif layer == 1: 167 | assert len(result) == 3 + 32 * 32 * 30, 'broken region data' 168 | elif layer == 2: 169 | temp_stream = io.BytesIO(result) 170 | for _ in range(starbound.sbon.read_varint(temp_stream)): 171 | starbound.read_versioned_json(temp_stream) 172 | except Exception as e: 173 | print('invalid key data: {}'.format(e)) 174 | continue 175 | # Count the node the first time it's stored. 176 | if cur_key not in data: 177 | nodes_recovered += 1 178 | data[cur_key] = zlib.compress(result) 179 | METADATA_KEY = b'\x00\x00\x00\x00\x00' 180 | # Ensure that the metadata key is in the data. 181 | if METADATA_KEY not in data: 182 | if options.world: 183 | try: 184 | data[METADATA_KEY] = blank_world.get(0, 0, 0) 185 | except Exception: 186 | p.error('failed to recover metadata from alternate world') 187 | else: 188 | if options.force: 189 | try: 190 | data[METADATA_KEY] = world.get(0, 0, 0) 191 | print('warning: using partially recovered metadata') 192 | except Exception: 193 | p.error('failed to recover partial metadata') 194 | else: 195 | p.error('failed to recover metadata; use -w to load metadata ' 196 | 'from another world, or -f to attempt partial recovery') 197 | print('done! {} nodes recovered'.format(nodes_recovered)) 198 | print('creating BTree database...') 199 | # Try not to exceed this number of keys per leaf. 200 | LEAF_KEYS_TRESHOLD = 10 201 | # Try not to exceed this size for a leaf. 202 | LEAF_SIZE_TRESHOLD = world.block_size * .8 203 | # Fill indexes up to this ratio. 204 | INDEX_FILL = .9 205 | # 6 is the number of bytes used for signature + next block pointer. 206 | LEAF_BYTES = world.block_size - 6 207 | # 11 is the number of bytes in the index header. 208 | INDEX_BYTES = world.block_size - 11 209 | # Maximum number of keys that can go into an index. 210 | INDEX_MAX_KEYS = int(INDEX_BYTES // (world.key_size + 4) * INDEX_FILL) 211 | # The data of individual blocks will be stored in this list. 212 | blocks = [] 213 | buffer = io.BytesIO() 214 | 215 | # This will create an initial leaf and connect it to following leaves which 216 | # will all contain the data currently in the buffer. 217 | def dump_buffer(): 218 | buffer_size = buffer.tell() 219 | buffer.seek(0) 220 | block_data = b'LL' + struct.pack('>i', num_keys) + buffer.read(LEAF_BYTES - 4) 221 | while buffer.tell() < buffer_size: 222 | blocks.append(block_data + struct.pack('>i', len(blocks) + 1)) 223 | block_data = b'LL' + buffer.read(LEAF_BYTES) 224 | blocks.append(block_data.ljust(world.block_size - 4, b'\x00') + struct.pack('>i', -1)) 225 | # Empty the buffer. 226 | buffer.seek(0) 227 | buffer.truncate() 228 | 229 | # The number of keys that will be stored in the next created leaf. 230 | num_keys = 0 231 | # Map of key range to leaf block pointer. 232 | range_to_leaf = dict() 233 | # All the keys, sorted (important). 234 | keys = sorted(data) 235 | # Build all the leaf blocks. 236 | min_key = None 237 | for key in keys: 238 | if not num_keys: 239 | # Remember the first key of the leaf. 240 | min_key = key 241 | buffer.write(key) 242 | starbound.sbon.write_bytes(buffer, data[key]) 243 | num_keys += 1 244 | # Empty buffer once one of the tresholds is reached. 245 | if num_keys >= LEAF_KEYS_TRESHOLD or buffer.tell() >= LEAF_SIZE_TRESHOLD: 246 | range_to_leaf[(min_key, key)] = len(blocks) 247 | dump_buffer() 248 | num_keys = 0 249 | # Empty any remaining data in the buffer. 250 | if buffer.tell(): 251 | range_to_leaf[(min_key, key)] = len(blocks) 252 | dump_buffer() 253 | print('created {} blocks containing world data'.format(len(blocks))) 254 | 255 | def build_index_level(range_to_block, level=0): 256 | # Get a list of ranges that this index level needs to point to. 257 | index_ranges = sorted(range_to_block) 258 | # The new list of ranges that the next level of indexes can use. 259 | new_ranges = dict() 260 | for i in range(0, len(index_ranges), INDEX_MAX_KEYS): 261 | ranges = index_ranges[i:i + INDEX_MAX_KEYS] 262 | min_key, _ = ranges[0] 263 | _, max_key = ranges[-1] 264 | left_block = range_to_block[ranges.pop(0)] 265 | index_data = io.BytesIO() 266 | index_data.write(b'II' + struct.pack('>Bii', level, len(ranges), left_block)) 267 | for key_range in ranges: 268 | index_data.write(key_range[0] + struct.pack('>i', range_to_block[key_range])) 269 | new_ranges[(min_key, max_key)] = len(blocks) 270 | blocks.append(index_data.getvalue().ljust(world.block_size, b'\x00')) 271 | print('- created {} index(es) for level {}'.format(len(new_ranges), level)) 272 | return new_ranges 273 | 274 | # Build the indexes in multiple levels up to a single root node. 275 | print('creating root node...') 276 | root_is_leaf = True 277 | level = 0 278 | current_index = range_to_leaf 279 | while len(current_index) > 1: 280 | current_index = build_index_level(current_index, level) 281 | root_is_leaf = False 282 | level += 1 283 | root_node = list(current_index.values())[0] 284 | # Also build an alternative root node. 285 | print('creating alternate root node...') 286 | alternate_root_is_leaf = True 287 | level = 0 288 | current_index = range_to_leaf 289 | while len(current_index) > 1: 290 | current_index = build_index_level(current_index, level) 291 | alternate_root_is_leaf = False 292 | level += 1 293 | alternate_root_node = list(current_index.values())[0] 294 | # The last two blocks will be free blocks. 295 | blocks.append(b'FF\xFF\xFF\xFF\xFF' + b'\x00' * (world.block_size - 6)) 296 | blocks.append(b'FF\xFF\xFF\xFF\xFF' + b'\x00' * (world.block_size - 6)) 297 | print('writing all the data to disk...') 298 | with open(out_name, 'wb') as f: 299 | header = struct.pack( 300 | starbound.btreedb5.HEADER, 301 | b'BTreeDB5', 302 | world.block_size, 303 | world.name.encode('utf-8') + b'\x00' * (16 - len(world.name)), 304 | world.key_size, 305 | False, 306 | len(blocks) - 1, 307 | 14282, # XXX: Unknown value! 308 | root_node, 309 | root_is_leaf, 310 | len(blocks) - 2, 311 | 14274, # XXX: Unknown value! 312 | alternate_root_node, 313 | alternate_root_is_leaf) 314 | f.write(header) 315 | for block in blocks: 316 | f.write(block) 317 | print('done!') 318 | 319 | 320 | if __name__ == '__main__': 321 | main() 322 | -------------------------------------------------------------------------------- /starbound/sbasset6.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | 3 | from collections import namedtuple 4 | import struct 5 | 6 | from starbound import sbon 7 | 8 | 9 | # Override range with xrange when running Python 2.x. 10 | try: 11 | range = xrange 12 | except: 13 | pass 14 | 15 | 16 | HEADER = '>8sQ' 17 | HEADER_SIZE = struct.calcsize(HEADER) 18 | 19 | 20 | IndexEntry = namedtuple('IndexEntry', ['offset', 'length']) 21 | 22 | 23 | class SBAsset6(object): 24 | def __init__(self, stream): 25 | self.stream = stream 26 | 27 | def get(self, path): 28 | if not hasattr(self, 'index'): 29 | self.read_index() 30 | offset, length = self.index[path.lower()] 31 | self.stream.seek(offset) 32 | return self.stream.read(length) 33 | 34 | def read_header(self): 35 | self.stream.seek(0) 36 | data = struct.unpack(HEADER, self.stream.read(HEADER_SIZE)) 37 | assert data[0] == b'SBAsset6', 'Invalid header' 38 | self.metadata_offset = data[1] 39 | # Read the metadata as well. 40 | self.stream.seek(self.metadata_offset) 41 | assert self.stream.read(5) == b'INDEX', 'Invalid index data' 42 | self.metadata = sbon.read_map(self.stream) 43 | self.file_count = sbon.read_varint(self.stream) 44 | # Store the offset of where the file index starts. 45 | self.index_offset = self.stream.tell() 46 | 47 | def read_index(self): 48 | if not hasattr(self, 'index_offset'): 49 | self.read_header() 50 | self.stream.seek(self.index_offset) 51 | self.index = {} 52 | for i in range(self.file_count): 53 | path = sbon.read_string(self.stream).lower() 54 | offset, length = struct.unpack('>QQ', self.stream.read(16)) 55 | self.index[path] = IndexEntry(offset, length) 56 | -------------------------------------------------------------------------------- /starbound/sbon.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | 3 | import struct 4 | import sys 5 | 6 | 7 | if sys.version >= '3': 8 | _int_type = int 9 | _str_type = str 10 | 11 | def _byte(x): 12 | return bytes((x,)) 13 | 14 | def _items(d): 15 | return d.items() 16 | else: 17 | _int_type = (int, long) 18 | _str_type = basestring 19 | range = xrange 20 | 21 | def _byte(x): 22 | return chr(x) 23 | 24 | def _items(d): 25 | return d.iteritems() 26 | 27 | 28 | def read_bytes(stream): 29 | length = read_varint(stream) 30 | return stream.read(length) 31 | 32 | 33 | def read_dynamic(stream): 34 | type_id = ord(stream.read(1)) 35 | if type_id == 1: 36 | return None 37 | elif type_id == 2: 38 | return struct.unpack('>d', stream.read(8))[0] 39 | elif type_id == 3: 40 | return stream.read(1) != b'\0' 41 | elif type_id == 4: 42 | return read_varint_signed(stream) 43 | elif type_id == 5: 44 | return read_string(stream) 45 | elif type_id == 6: 46 | return read_list(stream) 47 | elif type_id == 7: 48 | return read_map(stream) 49 | raise ValueError('Unknown dynamic type 0x%02X' % type_id) 50 | 51 | 52 | def read_list(stream): 53 | length = read_varint(stream) 54 | return [read_dynamic(stream) for _ in range(length)] 55 | 56 | 57 | def read_map(stream): 58 | length = read_varint(stream) 59 | value = dict() 60 | for _ in range(length): 61 | key = read_string(stream) 62 | value[key] = read_dynamic(stream) 63 | return value 64 | 65 | 66 | def read_string(stream): 67 | return read_bytes(stream).decode('utf-8') 68 | 69 | 70 | def read_varint(stream): 71 | """Read while the most significant bit is set, then put the 7 least 72 | significant bits of all read bytes together to create a number. 73 | 74 | """ 75 | value = 0 76 | while True: 77 | byte = ord(stream.read(1)) 78 | if not byte & 0b10000000: 79 | return value << 7 | byte 80 | value = value << 7 | (byte & 0b01111111) 81 | 82 | 83 | def read_varint_signed(stream): 84 | value = read_varint(stream) 85 | # Least significant bit represents the sign. 86 | if value & 1: 87 | return -(value >> 1) - 1 88 | else: 89 | return value >> 1 90 | 91 | 92 | def write_bytes(stream, value): 93 | write_varint(stream, len(value)) 94 | stream.write(value) 95 | 96 | 97 | def write_dynamic(stream, value): 98 | if value is None: 99 | stream.write(b'\x01') 100 | elif isinstance(value, float): 101 | stream.write(b'\x02') 102 | stream.write(struct.pack('>d', value)) 103 | elif isinstance(value, bool): 104 | stream.write(b'\x03\x01' if value else b'\x03\x00') 105 | elif isinstance(value, _int_type): 106 | stream.write(b'\x04') 107 | write_varint_signed(stream, value) 108 | elif isinstance(value, _str_type): 109 | stream.write(b'\x05') 110 | write_string(stream, value) 111 | elif isinstance(value, list): 112 | stream.write(b'\x06') 113 | write_list(stream, value) 114 | elif isinstance(value, dict): 115 | stream.write(b'\x07') 116 | write_map(stream, value) 117 | else: 118 | raise ValueError('Cannot write value %r' % (value,)) 119 | 120 | 121 | def write_list(stream, value): 122 | write_varint(stream, len(value)) 123 | for v in value: 124 | write_dynamic(stream, v) 125 | 126 | 127 | def write_map(stream, value): 128 | write_varint(stream, len(value)) 129 | for k, v in _items(value): 130 | write_string(stream, k) 131 | write_dynamic(stream, v) 132 | 133 | 134 | def write_string(stream, value): 135 | write_bytes(stream, value.encode('utf-8')) 136 | 137 | 138 | def write_varint(stream, value): 139 | buf = _byte(value & 0b01111111) 140 | value >>= 7 141 | while value: 142 | buf = _byte(value & 0b01111111 | 0b10000000) + buf 143 | value >>= 7 144 | stream.write(buf) 145 | 146 | 147 | def write_varint_signed(stream, value): 148 | write_varint(stream, (-(value + 1) << 1 | 1) if value < 0 else (value << 1)) 149 | --------------------------------------------------------------------------------