├── .gitignore
├── example-mod-settings.dat
├── LICENSE-0BSD.txt
├── README.md
└── factorio_data.py


/.gitignore:
--------------------------------------------------------------------------------
1 | __pycache__/
2 | *.pyc
3 | roundtrip-mod-settings.*
4 | example-mod-settings.json
5 | 


--------------------------------------------------------------------------------
/example-mod-settings.dat:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/whitequark/factorio-data-codec/HEAD/example-mod-settings.dat


--------------------------------------------------------------------------------
/LICENSE-0BSD.txt:
--------------------------------------------------------------------------------
 1 | Copyright (C) Catherine <whitequark@whitequark.org>
 2 | 
 3 | Permission to use, copy, modify, and/or distribute this software for
 4 | any purpose with or without fee is hereby granted.
 5 | 
 6 | THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
 7 | WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
 8 | MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
 9 | ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
10 | WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN
11 | AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT
12 | OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
13 | 
14 | 


--------------------------------------------------------------------------------
/README.md:
--------------------------------------------------------------------------------
 1 | Factorio data codec
 2 | ===================
 3 | 
 4 | The game [Factorio](https://www.factorio.com/) uses a custom binary file format to save certain game data, including the `mod-settings.dat` file. While these settings may be modified via the in-game UI, this isn't possible when running a dedicated server: the `mod-settings.dat` file must be pre-loaded with the correct values during world generation.
 5 | 
 6 | This repository contains a codec for Factorio game data (implemented in Python) that can convert the `mod-settings.dat` file to JSON and back, making it possible to adjust the settings without having to create and upload a local world or hex-edit the file.
 7 | 
 8 | 
 9 | Usage
10 | -----
11 | 
12 | To decode the settings, run in Command Prompt or PowerShell:
13 | 
14 | ```doscon
15 | > python factorio_data.py c:\...\mods\mod-settings.dat
16 | Reading DAT file 'c:\...\mods\mod-settings.dat'...
17 | Writing JSON file 'c:\...\mods\mod-settings.json'...
18 | ```
19 | 
20 | Edit the `mod-settings.json` file in Notepad or any other text editor.
21 | 
22 | To re-encode the settings, run in Command Prompt or PowerShell:
23 | 
24 | ```doscon
25 | > python factorio_data.py c:\...\mods\mod-settings.json
26 | Reading JSON file 'c:\...\mods\mod-settings.json'...
27 | Writing DAT file 'c:\...\mods\mod-settings.dat'...
28 | ```
29 | 
30 | If no modifications were made, the resulting DAT file must be bit-for-bit identical to the original DAT file. (If it is not, please file an issue on this repository, and attach the problematic DAT file for investigation.)
31 | 
32 | On Windows, running the `python` command may cause Windows Store to open and prompt you to install Python. Do so. Otherwise, if you don't have Python installed, [download and install it first](https://www.python.org/downloads/).
33 | 
34 | 
35 | Curiosity
36 | ---------
37 | 
38 | The `mod-settings.dat` file format is very similar to several other Factorio file formats, including the `level-init.dat` and `script.dat` files from saved games. However, these files contain additional data that isn't handled by this codec, and so it cannot decode them to JSON.
39 | 
40 | 
41 | License
42 | -------
43 | 
44 | [0-clause BSD](LICENSE-0BSD.txt).


--------------------------------------------------------------------------------
/factorio_data.py:
--------------------------------------------------------------------------------
  1 | import typing
  2 | import enum
  3 | import struct
  4 | import json
  5 | 
  6 | 
  7 | class ImmutableString:
  8 |     value: typing.Union[None, bytes]
  9 | 
 10 |     def __init__(self, value: typing.Union[None, bytes]):
 11 |         self.value = value
 12 | 
 13 |     @classmethod
 14 |     def load(cls, stream: typing.BinaryIO):
 15 |         is_none, = struct.unpack("<B", stream.read(1))
 16 | 
 17 |         if is_none:
 18 |             return cls(None)
 19 | 
 20 |         else:
 21 |             value_len, = struct.unpack("<B", stream.read(1))
 22 |             if value_len == 0xff:
 23 |                 value_len, = struct.unpack("<I", stream.read(4))
 24 | 
 25 |             return cls(stream.read(value_len))
 26 | 
 27 |     def save(self, stream: typing.BinaryIO):
 28 |         stream.write(struct.pack("<B", self.value is None))
 29 | 
 30 |         if self.value is not None:
 31 |             if len(self.value) >= 0xff:
 32 |                 stream.write(struct.pack("<BI", 0xff, len(self.value)))
 33 |             else:
 34 |                 stream.write(struct.pack("<B", len(self.value)))
 35 | 
 36 |             stream.write(self.value)
 37 | 
 38 |     def __eq__(self, other):
 39 |         return self.value == other.value
 40 | 
 41 |     def __repr__(self):
 42 |         return f"ImmutableString({self.value!r})"
 43 | 
 44 | 
 45 | class PropertyTree:
 46 |     class Type(enum.Enum):
 47 |         Null        = 0 # originally `None`
 48 |         Bool        = 1
 49 |         Number      = 2
 50 |         String      = 3
 51 |         List        = 4
 52 |         Dictionary  = 5
 53 | 
 54 |     key: ImmutableString
 55 |     value: typing.Union[None, bool, float, ImmutableString, list]
 56 |     type: Type
 57 |     any_type: bool
 58 | 
 59 |     def __init__(self,
 60 |             key: typing.Union[None, ImmutableString],
 61 |             value: typing.Union[bool, float, ImmutableString, list],
 62 |             type: Type,
 63 |             any_type: bool = False
 64 |     ):
 65 |         if key is None:
 66 |             key = ImmutableString(None)
 67 | 
 68 |         self.key = key
 69 |         self.value = value
 70 |         self.type = type
 71 |         self.any_type = any_type
 72 | 
 73 |     @classmethod
 74 |     def load(cls, stream: typing.BinaryIO):
 75 |         value_type_raw, any_type, = struct.unpack("<BB", stream.read(2))
 76 |         value_type = cls.Type(value_type_raw)
 77 | 
 78 |         if value_type == cls.Type.Null:
 79 |             value = None
 80 | 
 81 |         if value_type == cls.Type.Bool:
 82 |             value, = struct.unpack("<B", stream.read(1))
 83 |             value = bool(value)
 84 | 
 85 |         if value_type == cls.Type.Number:
 86 |             value, = struct.unpack("<d", stream.read(8))
 87 | 
 88 |         if value_type == cls.Type.String:
 89 |             value = ImmutableString.load(stream)
 90 | 
 91 |         if value_type in (cls.Type.List, cls.Type.Dictionary):
 92 |             count, = struct.unpack("<I", stream.read(4))
 93 |             value = []
 94 | 
 95 |             for _ in range(count):
 96 |                 key = ImmutableString.load(stream)
 97 |                 item = cls.load(stream)
 98 |                 item.key = key
 99 |                 value.append(item)
100 | 
101 |         return cls(None, value, value_type, bool(any_type))
102 | 
103 |     def save(self, stream: typing.BinaryIO):
104 |         stream.write(struct.pack("<BB", self.type.value, self.any_type))
105 | 
106 |         if self.type == self.Type.Null:
107 |             pass
108 | 
109 |         if self.type == self.Type.Bool:
110 |             stream.write(struct.pack("<B", self.value))
111 | 
112 |         if self.type == self.Type.Number:
113 |             stream.write(struct.pack("<d", self.value))
114 | 
115 |         if self.type == self.Type.String:
116 |             self.value.save(stream)
117 | 
118 |         if self.type in (self.Type.List, self.Type.Dictionary):
119 |             stream.write(struct.pack("<I", len(self.value)))
120 | 
121 |             for item in self.value:
122 |                 item.key.save(stream)
123 |                 item.save(stream)
124 | 
125 |     def __eq__(self, other):
126 |         return (self.key == other.key and self.value == other.value and
127 |                 self.type == other.type and self.any_type == other.any_type)
128 | 
129 |     def __repr__(self):
130 |         return f"PropertyTree({self.key.value!r}, {self.value!r}, {self.type!r}, anyType={self.any_type!r})"
131 | 
132 | 
133 | class ModSettings:
134 |     data: PropertyTree
135 |     version: typing.Tuple[int, int, int, int]
136 |     has_quality: bool
137 | 
138 |     def __init__(self, data: PropertyTree, version: typing.Tuple[int, int, int, int], has_quality: bool):
139 |         self.data = data
140 |         self.version = version
141 |         self.has_quality = has_quality
142 | 
143 |     @classmethod
144 |     def load(cls, stream: typing.BinaryIO):
145 |         # Factorio 1.1.110 becomes (1, 1, 110, 0)
146 |         version = struct.unpack("<HHHH", stream.read(8))
147 |         has_quality, = struct.unpack("<B", stream.read(1))
148 |         data = PropertyTree.load(stream)
149 | 
150 |         if version < (0, 18, 0, 0):
151 |             raise Exception(f"Cannot load settings from Factorio {version!r}: settings version too low")
152 | 
153 |         return cls(data, version, bool(has_quality))
154 | 
155 |     def save(self, stream: typing.BinaryIO):
156 |         stream.write(struct.pack("<HHHH", *self.version))
157 |         stream.write(struct.pack("<B", self.has_quality))
158 |         self.data.save(stream)
159 | 
160 |     def __eq__(self, other):
161 |         return self.data == other.data and self.version == other.version and self.has_quality == other.has_quality
162 | 
163 |     def __repr__(self):
164 |         return f"ModSettings({self.data!r}, version={self.version!r}, has_quality={self.has_quality!r})"
165 | 
166 | 
167 | class JSONEncoder(json.JSONEncoder):
168 |     def default(self, obj):
169 |         if isinstance(obj, ImmutableString):
170 |             if obj.value is None:
171 |                 return None
172 |             else:
173 |                 return obj.value.decode()
174 | 
175 |         if isinstance(obj, PropertyTree):
176 |             if obj.type == PropertyTree.Type.Null:
177 |                 return None
178 | 
179 |             if obj.type == PropertyTree.Type.Bool:
180 |                 return obj.value
181 | 
182 |             if obj.type == PropertyTree.Type.Number:
183 |                 return obj.value # this might not round-trip if it gets rounded
184 | 
185 |             if obj.type == PropertyTree.Type.String:
186 |                 return obj.value
187 | 
188 |             if obj.type == PropertyTree.Type.List:
189 |                 return [item for item in obj.value]
190 | 
191 |             if obj.type == PropertyTree.Type.Dictionary:
192 |                 return {item.key.value.decode(): item for item in obj.value}
193 | 
194 |         if isinstance(obj, ModSettings):
195 |             return {
196 |                 "!type": "ModSettings",
197 |                 "version": obj.version,
198 |                 "has_quality": obj.has_quality,
199 |                 "data": obj.data
200 |             }
201 | 
202 |         return super().default(obj)
203 | 
204 | 
205 | class JSONDecoder(json.JSONDecoder):
206 |     def __init__(self, strict=True):
207 |         super().__init__(object_hook=self.object_hook, strict=strict)
208 | 
209 |     def object_hook(self, obj):
210 |         if obj is None:
211 |             return PropertyTree(None, None, PropertyTree.Type.Null)
212 | 
213 |         if isinstance(obj, bool):
214 |             return PropertyTree(None, obj, PropertyTree.Type.Bool)
215 | 
216 |         if isinstance(obj, float):
217 |             return PropertyTree(None, obj, PropertyTree.Type.Number)
218 | 
219 |         if isinstance(obj, str):
220 |             return PropertyTree(None, ImmutableString(obj.encode()), PropertyTree.Type.String)
221 | 
222 |         if isinstance(obj, list):
223 |             return PropertyTree(None, obj, PropertyTree.Type.List)
224 | 
225 |         if isinstance(obj, dict):
226 |             if "!type" in obj:
227 |                 if obj["!type"] == "ModSettings":
228 |                     return ModSettings(obj["data"], (*obj["version"],), obj["has_quality"])
229 |                 else:
230 |                     raise Exception(f"Unknown object type {obj['!type']}")
231 | 
232 |             else:
233 |                 items = []
234 |                 for key, value in obj.items():
235 |                     item = self.object_hook(value)
236 |                     item.key = ImmutableString(key.encode())
237 |                     items.append(item)
238 | 
239 |                 return PropertyTree(None, items, PropertyTree.Type.Dictionary)
240 | 
241 |         if isinstance(obj, PropertyTree):
242 |             return obj
243 | 
244 |         raise NotImplementedError(f"Cannot convert {obj!r}")
245 | 
246 | 
247 | def selftest():
248 |     with open("example-mod-settings.dat", "rb") as selftest_dat:
249 |         settings_1 = ModSettings.load(selftest_dat)
250 |     json_data = json.dumps(settings_1, indent=2, cls=JSONEncoder)
251 |     settings_2 = json.loads(json_data, cls=JSONDecoder)
252 |     with open("roundtrip-mod-settings.dat", "wb") as roundtrip_dat:
253 |         settings_2.save(roundtrip_dat)
254 | 
255 |     assert settings_1 == settings_2
256 |     assert open("example-mod-settings.dat", "rb").read() == open("roundtrip-mod-settings.dat", "rb").read()
257 | 
258 | 
259 | def main():
260 |     import sys
261 |     import argparse
262 |     import io
263 | 
264 |     parser = argparse.ArgumentParser(description="""
265 |     Converts Factorio settings in `mod-settings.dat` to JSON and back.
266 |     """)
267 |     parser.add_argument("input", type=argparse.FileType("rb"),
268 |         help="Input file (.dat or .json).")
269 |     parser.add_argument("output",  type=argparse.FileType("wb"), nargs='?',
270 |         help="Output file (.json or .dat). Based on the input filename if omitted.")
271 | 
272 |     args = parser.parse_args()
273 | 
274 |     if args.input.name.endswith(".dat"):
275 |         print(f"Reading DAT file '{args.input.name}'...", file=sys.stderr)
276 |         mod_settings = ModSettings.load(args.input)
277 |         default_output_name = args.input.name[:-4] + ".json"
278 | 
279 |     elif args.input.name.endswith(".json"):
280 |         print(f"Reading JSON file '{args.input.name}'...", file=sys.stderr)
281 |         mod_settings = json.load(args.input, cls=JSONDecoder)
282 |         default_output_name = args.input.name[:-5] + ".dat"
283 | 
284 |     else:
285 |         print(f"Input filename '{args.input.name}' does not end with .dat or .json.", file=sys.stderr)
286 |         sys.exit(1)
287 | 
288 |     output = args.output
289 |     if output is None:
290 |         output = open(default_output_name, "wb")
291 | 
292 |     if output.name.endswith(".dat"):
293 |         print(f"Writing DAT file '{output.name}'...", file=sys.stderr)
294 |         mod_settings.save(output)
295 | 
296 |     elif output.name.endswith(".json"):
297 |         print(f"Writing JSON file '{output.name}'...", file=sys.stderr)
298 |         text_output = io.TextIOWrapper(output, encoding="utf-8")
299 |         json.dump(mod_settings, text_output, indent=4, cls=JSONEncoder)
300 | 
301 |     else:
302 |         print(f"Output filename '{output.name}' does not end with .dat or .json.", file=sys.stderr)
303 |         sys.exit(1)
304 | 
305 | 
306 | if __name__ == "__main__":
307 |     # selftest()
308 |     main()
309 | 


--------------------------------------------------------------------------------