├── .gitignore
├── README.md
├── config.json
├── lib
├── client.py
├── config.py
├── downloader.py
├── item_chain.py
├── reader.py
└── writer.py
├── main.py
└── requirements.txt
/.gitignore:
--------------------------------------------------------------------------------
1 | /.idea/
2 | /assets/
3 | /patches/
4 | /dumps/
5 | /.vscode/
6 | __pycache__
7 |
--------------------------------------------------------------------------------
/README.md:
--------------------------------------------------------------------------------
1 |
2 |
(better) SC Downloader
3 |
4 |
5 | ### Description
6 | File downloader for all (probably) games from Supercell. It directly receives all the latest data from the server and downloads it.
7 | [This sc-downloader by vorono4ka was used as a base](https://github.com/Vorono4ka/sc-downloader), some code from it is still here. But it has been greatly improved:
8 | - Added support for latest game servers iterations
9 | - Added threading to improve downloading speed
10 | - Added file patching feature
11 | - Added file verify feature
12 |
13 | ### HOW TO USE
14 | You need to have at least Python 3.9 for this to work.
15 | If you want to run script for the first time, then install all modules with this command in cmd
16 | ```
17 | pip install -r requirements.txt
18 | ```
19 | Basic usage is very simple, just run script with a command like
20 | ```
21 | py main.py
22 | ```
23 | After that, select the game server you need. All assets will begin downloading into ```assets/{Server name}/```
24 |
25 | ## In case if you need more
26 | There are additional flags here too.
27 |
28 | - ```--hash``` You can download assets directly using the version hash. Assets will be downloaded to a folder with the same name as hash. Example ```py main.py --hash=SomeVersionHash```
29 |
30 | - ```--repair-mode``` and ```--strict-repair-mode``` is just flags.
31 | Normal mode checks if files exist and if not, downloads them. Useful if: You have downloaded apk or ipa of the game, you already have almost all the assets. You can unpack these assets into the folder of the desired server and run script with this flag, it will download all files that may not be in your assets like background textures or music.
32 | Strict mode checks all files based on their content and this can be a bit long. Useful if: You accidentally somehow replaced a file or its content. Run script with this flag and its contents will be restored.
33 |
34 | ## Patches
35 | Patches are a very useful feature if you just need to get new files from the latest update.
36 | It compares previous version and current one, and copies all new or changed files to ```patches/{Server name}/{old version name} {new version name}/```
37 | If detailed patches are enabled in the config, then the patch will be divided into 3 parts, new files, changed files and deleted files.
38 | To enable or disable this feature, you can look in ```config.json```.
39 |
40 | ## Config
41 | ```config.json``` contains all settings for managing servers, threading and patches.
42 | - ```servers``` are stored in a dictionary, key of which means name and value of key means address of the game server. You can easily add your own server.
43 | - ```auto_update``` means whether files should be updated automatically. Otherwise, every time there is an optional update, script will ask about files updating.
44 | - ```make_patches``` and ```make_detailed_patches``` explained in patches description
45 | - ```max_workers``` or maximum count of threads. Each thread downloads its own list of files and this parameter sets maximum number of threads that can work at one time. Be careful, a large number of threads can either speed up or slow down process, for the most part it all depends on your PC. A large number of threads is not recommended on weak PCs.
46 | - ```worker_max_items``` sets the number of how many files one worker can process. This is made for large folders like folders with 3d graphics or sounds. This folders will be divided into pieces whose size depends on this value, and each piece will be processed by a new worker.
47 | - ```save_dump``` mostly needed for debugging messages from server. just don't touch it.
48 |
49 |
--------------------------------------------------------------------------------
/config.json:
--------------------------------------------------------------------------------
1 | {
2 | "save_dump": false,
3 | "auto_update": false,
4 | "make_patches": true,
5 | "make_detailed_patches": false,
6 | "max_workers": 12,
7 | "worker_max_items": 50,
8 | "servers": {
9 | "BrawlStarsPROD": "game.brawlstarsgame.com",
10 | "BrawlStarsCN": "52.83.179.16"
11 | }
12 | }
--------------------------------------------------------------------------------
/lib/client.py:
--------------------------------------------------------------------------------
1 | import os
2 | from socket import socket, create_connection
3 | import json
4 | from struct import unpack
5 | from .writer import Writer
6 | from .reader import Reader
7 | import zlib
8 |
9 | from enum import Enum
10 |
11 |
12 | class HelloServerResponse(Enum):
13 | Success = 7
14 | NeedUpdate = 8
15 |
16 |
17 | class Client:
18 | def __init__(self, assets_path: str) -> None:
19 | self.assets_path = assets_path
20 | self.fingerprint_filepath = os.path.join(assets_path, "fingerprint.json")
21 |
22 | self.socket: socket = None
23 | self.major = 0
24 | self.build = 0
25 | self.revision = 0
26 | self.dump = False
27 | self.assets_url = ""
28 | self.assets_url_2 = ""
29 | self.content_url = ""
30 |
31 | self.fingerprint: dict = {}
32 | if os.path.exists(self.fingerprint_filepath):
33 | data_file = open(self.fingerprint_filepath, "rb")
34 | self.fingerprint = json.load(data_file)
35 | data_file.close()
36 |
37 | self.major, self.build, self.revision = self.content_version
38 |
39 | @property
40 | def content_version(self) -> list[int, int, int]:
41 | if self.fingerprint:
42 | version = str(self.fingerprint["version"]).split(".")
43 | return [int(num) for num in version]
44 |
45 | return [0, 0, 0]
46 |
47 | @property
48 | def content_hash(self) -> str:
49 | if self.fingerprint:
50 | return self.fingerprint.get("sha") or ""
51 |
52 | return ""
53 |
54 | def handle_packet(self) -> bytes:
55 | header = self.socket.recv(7)
56 | packet_length = int.from_bytes(header[2:5], "big")
57 |
58 | received_data = b""
59 | while packet_length > 0:
60 | chunk = self.socket.recv(packet_length)
61 | if not chunk:
62 | raise EOFError
63 | received_data += chunk
64 | packet_length -= len(chunk)
65 |
66 | return received_data
67 |
68 | def send_packet(self, id: int, data: bytes) -> bytes:
69 | packet = Writer()
70 | packet.writeUShort(id)
71 | packet.buffer += len(data).to_bytes(3, "big")
72 | packet.writeUShort(0)
73 | packet.buffer += data
74 | self.socket.send(packet.buffer)
75 |
76 | return self.handle_packet()
77 |
78 | def disconnect(self) -> None:
79 | self.socket.close()
80 |
81 | def connect(self, address: str) -> HelloServerResponse:
82 | self.socket = create_connection((address, 9339))
83 |
84 | # HelloMessage
85 | stream = Writer()
86 |
87 | stream.writeUInt32(0) # Protocol Version
88 | stream.writeUInt32(11) # Key Version
89 | stream.writeUInt32(self.major) # major
90 | stream.writeUInt32(self.revision) # revision
91 | stream.writeUInt32(self.build) # build (minor)
92 | stream.writeString("") # ContentHash
93 | stream.writeUInt32(2) # DeviceType
94 | stream.writeUInt32(2) # AppStore
95 |
96 | server_data_buffer = self.send_packet(10100, stream.buffer)
97 | server_data_stream = Reader(server_data_buffer)
98 |
99 | if self.dump:
100 | os.makedirs("dumps/", exist_ok=True)
101 | open(f"dumps/{address}.{self.major}.{self.build}.10100.bin", "wb").write(
102 | server_data_buffer
103 | )
104 |
105 | status_code = server_data_stream.readUInt32()
106 |
107 | if status_code == 7:
108 | server_data_stream.readUInt32()
109 | server_data_stream.readUInt32()
110 |
111 | self.content_url = server_data_stream.readString()
112 |
113 | server_data_stream.readUInt32()
114 |
115 | serialized_fingerprint = server_data_stream.readString()
116 |
117 | # If decompressed data length is 0 then decompress data with zlib
118 | if len(serialized_fingerprint) == 0:
119 | # Skip zeros bytes
120 | server_data_stream.seek(5, 1)
121 |
122 | # decompressInMySQLFormat
123 | compressed_data_length = server_data_stream.readUInt32()
124 | # For some reason decompressed size is in Little Endian
125 | decompressed_data_length = unpack(" None:
7 | self.short_name = name
8 | self.server_address = address
9 |
10 | class Config:
11 | def __init__(self, filepath: str) -> None:
12 | file = open(filepath, "rb")
13 | data: dict = json.load(file)
14 | file.close()
15 |
16 | # self.server_specific_data: dict = data["server_specific_data"]
17 | self.save_dump = True if data.get("save_dump") else False
18 | self.auto_update = True if data.get("auto_update") else False
19 | self.make_patches = True if data.get("make_patches") else False
20 | self.make_detailed_patches = (
21 | True if data.get("make_detailed_patches") else False
22 | )
23 | self.max_workers = data.get("max_workers") or 1
24 | self.worker_max_items = data.get("worker_max_items") or 1
25 |
26 | servers_data: dict = data.get("servers")
27 | self.servers: list[ServerDescriptor] = []
28 |
29 | # Session Settings
30 | parser = argparse.ArgumentParser(
31 | prog="SC-Downloader", description="Asset Downloader for Supercell Games"
32 | )
33 |
34 | parser.add_argument(
35 | "--hash",
36 | type=str,
37 | help="Specify your version hash by which you want to download assets",
38 | default=None,
39 | )
40 |
41 | parser.add_argument(
42 | "--asset-servers",
43 | help="You can provide your own links to asset servers",
44 | nargs="+",
45 | default=None,
46 | )
47 |
48 | parser.add_argument(
49 | "--repair-mode",
50 | action=argparse.BooleanOptionalAction,
51 | help="Checks if all files exist and loads them if they are missing",
52 | default=False,
53 | )
54 |
55 | parser.add_argument(
56 | "--strict-repair-mode",
57 | action=argparse.BooleanOptionalAction,
58 | help="Checks file content and if it is corrupted, downloads it again. Much slower than default mode.",
59 | default=False,
60 | )
61 |
62 | args = parser.parse_args()
63 |
64 | self.custom_hash: str = "" or args.hash
65 | self.asset_servers_override = args.asset_servers
66 |
67 | self.strict_repair: bool = args.strict_repair_mode
68 | self.repair: bool = args.repair_mode or self.strict_repair
69 |
70 | # Server specific variables
71 | self.status_code_size = 4 # int
72 | self.variable_schema: list = []
73 |
74 | for server in servers_data:
75 | self.servers.append(ServerDescriptor(server, servers_data[server]))
76 |
77 | def load_server_specific_data(self, name: str) -> None:
78 | data = self.server_specific_data
79 | pass
80 |
--------------------------------------------------------------------------------
/lib/downloader.py:
--------------------------------------------------------------------------------
1 | from .item_chain import ItemChain, Item
2 | from threading import Thread
3 | import os
4 | import posixpath
5 | import requests
6 | from hashlib import sha1
7 |
8 | class DownloaderWorker(Thread):
9 | def __init__(
10 | self,
11 | content_hash: str,
12 | assets_urls: list[str],
13 | assets_path: str,
14 | assets_basepath: str,
15 | folder: ItemChain,
16 | ) -> None:
17 | Thread.__init__(self)
18 | self.is_working = True
19 | self.assets_basepath = assets_basepath
20 | self.assets_path = assets_path
21 | self.folder = folder
22 | self.content_hash = content_hash
23 | self.assets_urls = assets_urls
24 |
25 | @staticmethod
26 | def download_file(urls: list[str], conent_hash: str, filepath: str) -> bytes or int:
27 | request: requests.Response = None
28 | for url in urls:
29 | request = requests.get(
30 | f"{url}/{conent_hash}/{filepath}"
31 | )
32 | if request.status_code == 200: break
33 |
34 | # Final writing to file
35 | if request.status_code == 200:
36 | return request.content
37 | else:
38 | return request.status_code
39 |
40 | def run(self):
41 | """
42 | The function downloads files from multiple URLs and saves them to a specified directory,
43 | providing status messages along the way.
44 | :return: The code is returning either None or stopping the execution of the function if the
45 | `self.is_working` flag is False.
46 | """
47 |
48 | for item in self.folder.items:
49 | if not self.is_working: return
50 |
51 | if isinstance(item, ItemChain):
52 | continue
53 |
54 | base_filepath = posixpath.join(self.assets_basepath, item.name)
55 |
56 | server_response = DownloaderWorker.download_file(self.assets_urls, self.content_hash, base_filepath)
57 |
58 | if (isinstance(server_response, bytes)):
59 | with open(os.path.join(self.assets_path, base_filepath), "wb") as file:
60 | file.write(server_response)
61 |
62 | self.message(f"Downloaded {base_filepath}")
63 | else:
64 | self.message(f"Failed to download \"{base_filepath}\" with code {server_response}")
65 |
66 | self.message("Done")
67 | self.is_working = False
68 |
69 | def message(self, text: str):
70 | """
71 | The function "message" prints a formatted message with the name of the object and the provided
72 | text.
73 |
74 | :param text: The `text` parameter is a string that represents the message that you want to print
75 | :type text: str
76 | """
77 |
78 | print(f"[{self.name}] {text}")
79 |
80 | def DownloaderDecorator(function):
81 | def decorator(*args, **kwargs):
82 | try:
83 | return function(*args, **kwargs)
84 | except KeyboardInterrupt:
85 | args[0].stop_all_workers()
86 | exit(0)
87 |
88 | return decorator
89 |
90 | class Downloader:
91 | def __init__(self,
92 | content_urls: list[str],
93 | content_hash: str,
94 | output_folder: str,
95 | max_workers=8,
96 | worker_max_items=50,
97 | strict_level = 0) -> None:
98 | self.workers: list[DownloaderWorker] = []
99 | self.max_workers = max_workers
100 | self.worker_max_items = worker_max_items
101 | self.output_folder = output_folder
102 | self.content_urls = content_urls
103 | self.content_hash = content_hash
104 | self.strict_level = strict_level
105 |
106 | @staticmethod
107 | def add_unlisted_items(folder: ItemChain):
108 | """
109 | The function adds specific items to a given chain
110 |
111 | :param folder: The parameter "folder" is of type ItemChain
112 | :type folder: ItemChain
113 | """
114 |
115 | # Fingerprint itself
116 | folder.items.append(Item("fingerprint.json", ""))
117 |
118 | # Version info
119 | folder.items.append(Item("version.number", ""))
120 |
121 | def check_workers_status(self) -> bool:
122 | """
123 | The function checks the status of workers and returns True if all workers are not working,
124 | otherwise it returns False.
125 | :return: a boolean value. It returns True if there are no workers in the list, and False
126 | otherwise.
127 | """
128 |
129 | i = 0
130 | while len(self.workers) > i:
131 | worker = self.workers[i]
132 |
133 | if not worker.is_working:
134 | del self.workers[i]
135 | else:
136 | i += 1
137 |
138 | if len(self.workers) == 0:
139 | return True
140 |
141 | return False
142 |
143 | def wait_for_workers(self) -> None:
144 | """
145 | The function waits until all workers have finished their tasks.
146 | """
147 |
148 | while True:
149 | if self.check_workers_status():
150 | break
151 |
152 | def add_worker(self, basepath: str, chain: ItemChain) -> bool:
153 | """
154 | The function adds a worker to the download queue if the maximum number of workers has not been
155 | reached.
156 |
157 | :param basepath: The `basepath` parameter is a string that represents the base path where the
158 | downloaded files will be saved. It is used to determine the location where the downloaded files
159 | will be stored
160 | :type basepath: str
161 | :param chain: The `chain` parameter is an instance of the `ItemChain` class. It represents a
162 | chain of items that need to be downloaded
163 | :type chain: ItemChain
164 | :return: a boolean value. If the length of the `self.workers` list is less than the maximum
165 | number of workers (`self.max_workers`), the function will return `True`. Otherwise, it will
166 | return `False`.
167 | """
168 |
169 | if len(self.workers) >= self.max_workers:
170 | return False
171 |
172 | worker = DownloaderWorker(
173 | self.content_hash,
174 | self.content_urls,
175 | self.output_folder,
176 | basepath,
177 | chain
178 | )
179 |
180 | print(f"[Main] {chain.name or 'Assets'} folder added to download queue")
181 | worker.start()
182 | self.workers.append(worker)
183 |
184 | return True
185 |
186 | def stop_all_workers(self):
187 | """
188 | The function stops all workers by setting their "is_working" attribute to False and waits for
189 | them to finish their current tasks.
190 | """
191 |
192 | for worker in self.workers:
193 | worker.is_working = False
194 |
195 | self.wait_for_workers()
196 |
197 | @DownloaderDecorator
198 | def download(self, folder: ItemChain, basepath: str = "") -> None:
199 | """
200 | The `download` function takes a folder and downloads its contents, splitting them into worker
201 | chunks to be downloaded concurrently.
202 |
203 | :param folder: The `folder` parameter is an instance of the `ItemChain` class, which represents
204 | a collection of items. Each item can be either a file or a subfolder
205 | :type folder: ItemChain
206 | :param basepath: The `basepath` parameter is a string that represents the base path where the
207 | items will be downloaded. It is used to create the directory structure for the downloaded items
208 | :type basepath: str
209 | """
210 |
211 | # Folders prepare
212 | current_dir = os.path.join(self.output_folder, basepath)
213 | os.makedirs(
214 | current_dir, exist_ok=True
215 | )
216 |
217 | # worker_max_items sorting & existing files removing
218 | worker_chunks: list[ItemChain] = []
219 |
220 | i = 0
221 | temp_chunk = ItemChain(folder.name)
222 |
223 | for item in folder.items:
224 | if isinstance(item, ItemChain): continue
225 |
226 | asset_path = os.path.join(current_dir, item.name)
227 |
228 | valid_file = False
229 | if (self.strict_level >= 1):
230 | valid_file = os.path.exists(asset_path) and len(item.hash) != 0
231 |
232 | if (self.strict_level >= 2):
233 | if (valid_file):
234 | with open(asset_path, "rb") as file:
235 | digest = sha1(file.read())
236 | valid_file = digest.hexdigest() == item.hash
237 |
238 | if (valid_file): continue
239 |
240 | if (i >= self.worker_max_items):
241 | i = 0
242 | worker_chunks.append(temp_chunk)
243 | temp_chunk = ItemChain(folder.name)
244 |
245 | temp_chunk.items.append(item)
246 | i += 1
247 |
248 | # Append last small chunk
249 | if (len(temp_chunk.items) != 0):
250 | worker_chunks.append(temp_chunk)
251 |
252 |
253 | for worker_chunk in worker_chunks:
254 | while True:
255 | self.check_workers_status()
256 | if self.add_worker(basepath, worker_chunk):
257 | break
258 |
259 | for item in folder.items:
260 | if isinstance(item, Item):
261 | continue
262 |
263 | self.download(item, posixpath.join(basepath, item.name))
264 |
265 | @DownloaderDecorator
266 | def download_folder(self, folder: ItemChain) -> None:
267 | """
268 | The function `download_folder` downloads a folder and waits for the download to finish.
269 |
270 | :param folder: The `folder` parameter is of type `ItemChain`. It represents a folder or
271 | directory that needs to be downloaded
272 | :type folder: ItemChain
273 | """
274 |
275 | print("Downloading...")
276 | self.download(folder)
277 | self.wait_for_workers()
278 | print("Downloading is finished")
279 |
280 | @DownloaderDecorator
281 | def download_fingerprint(self, fingerprint: dict) -> None:
282 | """
283 | The function downloads a folder and its contents based on a given fingerprint.
284 |
285 | :param fingerprint: The `fingerprint` parameter is a dictionary that represents fingerprint data.
286 | :type fingerprint: dict
287 | """
288 |
289 | root = ItemChain.from_fingerprint(fingerprint)
290 | Downloader.add_unlisted_items(root)
291 | self.download_folder(root)
292 |
293 |
--------------------------------------------------------------------------------
/lib/item_chain.py:
--------------------------------------------------------------------------------
1 | from __future__ import annotations
2 | import os
3 |
4 |
5 | ''' Representation of "File" or asset with hash and name '''
6 | class Item:
7 | def __init__(self, name: str, hash: str) -> None:
8 | self.name = name
9 | self.hash = hash
10 |
11 | ''' Representation of "Folder" or "Chain of asset files" '''
12 | class ItemChain:
13 | def __init__(self, name: str, *args) -> None:
14 | self.name = name
15 | self.items: list[Item or ItemChain] = list(args)
16 |
17 | def get(self, name: str) -> Item or ItemChain or None:
18 | """
19 | The function "get" searches for an item with a given name in a list of items and returns the
20 | item if found, otherwise it returns None.
21 |
22 | :param name: A string representing the name of the item to search for
23 | :type name: str
24 | :return: an instance of the class "Item" or "ItemChain" if an item with the specified name is found in the list
25 | of items. If no matching item is found, it returns "None".
26 | """
27 | for item in self.items:
28 | if item.name == name:
29 | return item
30 |
31 | return None
32 |
33 | def get_chain(self, chain_names: list[str], auto_create=False) -> ItemChain or None:
34 | """
35 | The `get_chain` function retrieves a specific chain from a list of chain names, optionally
36 | creating the chain if it doesn't exist.
37 |
38 | :param chain_names: A list of strings representing the names of the chains to be retrieved
39 | :type chain_names: list[str]
40 | :param auto_create: The `auto_create` parameter is a boolean flag that determines whether a new
41 | `ItemChain` should be automatically created if the specified chain name does not exist. If
42 | `auto_create` is set to `True`, a new `ItemChain` will be created and added to the `items` list,
43 | defaults to False (optional)
44 | :return: The function `get_chain` returns an instance of `ItemChain` or `None`.
45 | """
46 | if (len(chain_names) == 0): return self
47 |
48 | result_item = None
49 | for chain_name in chain_names:
50 | iterable_item: ItemChain = result_item if result_item is not None else self
51 | iterable_item_result: ItemChain or None = None
52 |
53 | for item in iterable_item.items:
54 | if not isinstance(item, ItemChain):
55 | continue
56 |
57 | if item.name == chain_name:
58 | iterable_item_result = item
59 | break
60 |
61 | if iterable_item_result is None:
62 | if auto_create:
63 | result_item = ItemChain(chain_name)
64 | iterable_item.items.append(result_item)
65 | else:
66 | return None
67 | else:
68 | result_item = iterable_item_result
69 |
70 | return result_item
71 |
72 | @staticmethod
73 | def from_fingerprint(data: dict):
74 | """
75 | The `from_fingerprint` function takes in a dictionary of file descriptors and creates a
76 | hierarchical structure of folders and files based on the file paths and hashes provided.
77 |
78 | :param data: The `data` parameter is a dictionary that contains information about asset files.
79 | :type data: dict
80 | :return: an instance of the ItemChain class, which represents a hierarchical structure of items
81 | (files and folders) based on the provided fingerprint data.
82 | """
83 | root = ItemChain("")
84 |
85 | files: list[dict] = data["files"]
86 |
87 | for descriptor in files:
88 | name = str(descriptor["file"])
89 | hash = str(descriptor["sha"])
90 |
91 | folder = root
92 |
93 | basename = os.path.dirname(name)
94 | if (basename):
95 | folder_name_chain = os.path.normpath(basename).split(os.sep)
96 | else:
97 | folder_name_chain = []
98 |
99 | folder: ItemChain = root.get_chain(folder_name_chain, True)
100 | folder.items.append(Item(os.path.basename(name), hash))
101 |
102 | return root
103 |
--------------------------------------------------------------------------------
/lib/reader.py:
--------------------------------------------------------------------------------
1 | from io import BufferedReader, BytesIO
2 | from struct import unpack
3 |
4 |
5 | class Reader(BufferedReader):
6 | def __init__(self, initial_bytes: bytes):
7 | super(Reader, self).__init__(BytesIO(initial_bytes))
8 |
9 | def readUInt64(self) -> int:
10 | return unpack('>Q', self.read(8))[0]
11 |
12 | def readInt64(self) -> int:
13 | return unpack('>q', self.read(8))[0]
14 |
15 | def readUInt32(self) -> int:
16 | return unpack('>I', self.read(4))[0]
17 |
18 | def readInt32(self) -> int:
19 | return unpack('>i', self.read(4))[0]
20 |
21 | def readUInt16(self) -> int:
22 | return unpack('>H', self.read(2))[0]
23 |
24 | def readInt16(self) -> int:
25 | return unpack('>h', self.read(2))[0]
26 |
27 | def readUInt8(self) -> int:
28 | return unpack('>B', self.read(1))[0]
29 |
30 | def readInt8(self) -> int:
31 | return unpack('>b', self.read(1))[0]
32 |
33 | readULong = readUInt64
34 | readLong = readInt64
35 |
36 | readUShort = readUInt16
37 | readShort = readInt16
38 |
39 | readUByte = readUInt8
40 | readByte = readInt8
41 |
42 | def readChar(self, length: int = 1) -> str:
43 | return self.read(length).decode('utf-8')
44 |
45 | def readString(self) -> str:
46 | length = self.readUInt32()
47 | if length == 2**32 - 1 or length == 0:
48 | return ''
49 | else:
50 | return self.readChar(length)
51 |
--------------------------------------------------------------------------------
/lib/writer.py:
--------------------------------------------------------------------------------
1 | from struct import pack
2 |
3 |
4 | class Writer:
5 | def __init__(self):
6 | super(Writer, self).__init__()
7 | self.buffer = b''
8 |
9 | def writeUInt64(self, integer: int):
10 | self.buffer += pack('>Q', integer)
11 |
12 | def writeInt64(self, integer: int):
13 | self.buffer += pack('>q', integer)
14 |
15 | def writeUInt32(self, integer: int):
16 | self.buffer += pack('>I', integer)
17 |
18 | def writeInt32(self, integer: int):
19 | self.buffer += pack('>i', integer)
20 |
21 | def writeUInt16(self, integer: int):
22 | self.buffer += pack('>H', integer)
23 |
24 | def writeInt16(self, integer: int):
25 | self.buffer += pack('>h', integer)
26 |
27 | def writeUInt8(self, integer: int):
28 | self.buffer += pack('>B', integer)
29 |
30 | def writeInt8(self, integer: int):
31 | self.buffer += pack('>b', integer)
32 |
33 | writeULong = writeUInt64
34 | writeLong = writeInt64
35 |
36 | writeUShort = writeUInt16
37 | writeShort = writeInt16
38 |
39 | writeUByte = writeUInt8
40 | writeByte = writeInt8
41 |
42 | def writeString(self, string: str):
43 | encoded = string.encode('utf-8')
44 | self.writeUInt32(len(encoded))
45 | self.buffer += encoded
46 |
--------------------------------------------------------------------------------
/main.py:
--------------------------------------------------------------------------------
1 | import posixpath
2 | from typing import Any
3 | from lib.client import Client, HelloServerResponse
4 | from lib.config import Config
5 | from lib.downloader import Downloader, DownloaderWorker
6 | from lib.item_chain import ItemChain, Item
7 | import os
8 | from shutil import move as fmove
9 | from shutil import copyfile as fcopy
10 |
11 | class ScDownloader:
12 | def __init__(self) -> None:
13 | self.config = Config("config.json")
14 |
15 | # Servers
16 | print("Choose server to connect: ")
17 |
18 | for i, descriptor in enumerate(self.config.servers):
19 | print(f'{i}. "{descriptor.short_name}": {descriptor.server_address}"')
20 |
21 | server_index = int(input("\nServer index: "))
22 | self.active_server = self.config.servers[server_index]
23 |
24 | self.assets_path = f"assets/{self.config.custom_hash or self.active_server.short_name}/"
25 | self.patches_path = f"patches/{self.active_server.short_name}"
26 |
27 | os.makedirs(self.assets_path, exist_ok=True)
28 | if (self.config.make_patches):
29 | os.makedirs(self.patches_path, exist_ok=True)
30 |
31 | # Downloading by hash stuff
32 | if (self.config.custom_hash):
33 | self.config.asset_servers_override = self.config.asset_servers_override or self.get_latest_asset_servers()
34 |
35 | hash_fingerprint = DownloaderWorker.download_file(
36 | self.config.asset_servers_override,
37 | self.config.custom_hash,
38 | "fingerprint.json"
39 | )
40 |
41 | if (isinstance(hash_fingerprint, int)):
42 | raise Exception(f"Failed to get fingerprint.json by hash {self.config.custom_hash}. Request failed with code {hash_fingerprint}")
43 |
44 | os.makedirs(self.assets_path, exist_ok=True)
45 | with open(os.path.join(self.assets_path, "fingerprint.json"), "wb") as file:
46 | file.write(hash_fingerprint)
47 |
48 | self.client = Client(self.assets_path)
49 | self.client.dump = self.config.save_dump
50 |
51 | @staticmethod
52 | def ask_question_bool(question: str) -> bool:
53 | """
54 | The function `ask_question_bool` prompts the user with a question and returns a boolean value
55 | based on their response.
56 |
57 | :param question: A string representing the question that will be asked to the user
58 | :type question: str
59 | :return: The function `ask_question_bool` returns a boolean value.
60 | """
61 |
62 | answer = str(input(f"{question} (yes/no): ")).lower()
63 |
64 | if answer.startswith("y"): return True
65 | if answer.startswith("n"): return False
66 |
67 | if answer.isdigit():
68 | return int(answer) >= 1
69 |
70 | @staticmethod
71 | def make_patch_chain(current: ItemChain, latest: ItemChain) -> list[ItemChain, ItemChain, ItemChain]:
72 | """
73 | The `make_patch_chain` function takes in two `ItemChain` objects, `current` and `latest`, and
74 | returns a list of three `ItemChain` objects representing new files, changed files, and deleted
75 | files between the two chains.
76 |
77 | :param current: The `current` parameter is an instance of the `ItemChain` class, which
78 | represents the current state of a chain of items. It contains a list of items
79 | :type current: ItemChain
80 | :param latest: The "latest" parameter is an instance of the ItemChain class, which represents
81 | the most recent version of a chain of items. It contains a collection of items, where each item
82 | can be either a file or a folder. The items in the "latest" chain may have been added, modified
83 | or deleted
84 | :type latest: ItemChain
85 | :return: The function `make_patch_chain` returns a list containing three `ItemChain` objects:
86 | `new_files_result`, `changed_files_result`, and `deleted_files_result`.
87 | """
88 |
89 | def make_chain(current: ItemChain, new: ItemChain) -> list[ItemChain, ItemChain, ItemChain]:
90 | current_items_names = [item.name for item in current.items]
91 |
92 | new_files_result = ItemChain(current.name)
93 | changed_files_result = ItemChain(current.name)
94 | deleted_files_result = ItemChain(current.name)
95 | deleted_files_indices = [True for _ in current.items]
96 |
97 | new_files_result = ItemChain(current.name)
98 | for new_item in new.items:
99 | # New Files
100 | if new_item.name not in current_items_names:
101 | new_files_result.items.append(new_item)
102 | continue
103 |
104 | current_item_index = current_items_names.index(new_item.name)
105 | current_item = current.items[current_item_index]
106 | deleted_files_indices[current_item_index] = False
107 |
108 | if isinstance(new_item, Item):
109 | # Changed Files
110 | if (new_item.hash != current_item.hash):
111 | changed_files_result.items.append(new_item)
112 | else:
113 | # Folder Processing
114 | new_files_chain, changed_files_chain, deleted_files_chain = make_chain(current_item, new_item)
115 |
116 | if (len(new_files_chain.items) != 0):
117 | new_files_result.items.append(new_files_chain)
118 |
119 | if (len(changed_files_chain.items) != 0):
120 | changed_files_result.items.append(changed_files_chain)
121 |
122 | if (len(deleted_files_chain.items) != 0):
123 | deleted_files_result.items.append(deleted_files_chain)
124 |
125 | for i in range(len(current.items)):
126 | current_item = current.items[i]
127 | is_deleted = deleted_files_indices[i]
128 |
129 | if (is_deleted):
130 | deleted_files_result.items.append(current_item)
131 |
132 | return [new_files_result, changed_files_result, deleted_files_result]
133 | return make_chain(current, latest)
134 |
135 | def download_all(self):
136 | """
137 | Downloads all files from client fingerprint to `self.client.assets_path`
138 | """
139 |
140 | asset_servers_urls = self.config.asset_servers_override or \
141 | [self.client.assets_url, self.client.assets_url_2, self.client.content_url]
142 |
143 | downloader = Downloader(
144 | asset_servers_urls,
145 | self.client.content_hash,
146 | self.client.assets_path,
147 | self.config.max_workers,
148 | self.config.worker_max_items,
149 | int(self.config.repair) + int(self.config.strict_repair),
150 | )
151 | downloader.download_fingerprint(self.client.fingerprint)
152 |
153 | def get_latest_client(self) -> Client:
154 | """
155 | The function `get_latest_client` returns client with the latest data.
156 | :return: an instance of the Client class.
157 | """
158 | client_latest = Client("")
159 | client_latest.major = client_latest.build = client_latest.revision = 0
160 | client_latest.connect(self.active_server.server_address)
161 |
162 | return client_latest
163 |
164 | def get_latest_asset_servers(self) -> list[str]:
165 | """
166 | The function `get_latest_asset_servers` returns a list of asset servers from the latest client.
167 | :return: a list of strings, specifically the assets URLs of the latest client.
168 | """
169 | client = self.get_latest_client()
170 |
171 | return [client.assets_url, client.assets_url_2, client.content_url]
172 |
173 | def check_update(self) -> tuple[bool, Client]:
174 | """
175 | The function `check_update` compares the content version of the current client with the content
176 | version of the latest client on the active server and returns a tuple indicating whether they
177 | are different and the latest client.
178 | :return: a tuple containing two values. The first value is a boolean indicating whether the
179 | client version is different from the latest version, and the second value is an instance of the
180 | Client class representing the latest client version.
181 | """
182 | client_latest = self.get_latest_client()
183 | is_different = False in [client_latest.content_version[i] <= self.client.content_version[i] for i in range(3)]
184 | # print(f"Current version is {client_latest.content_version}, Server Version is {self.client.content_version}")
185 | return (is_different, client_latest)
186 |
187 | def make_update(self, latest_client: Client or None = None) -> None:
188 | """
189 | The function `make_update` is responsible for updating the client's assets by downloading new
190 | and changed files, and deleting unnecessary files.
191 |
192 | :param latest_client: The `latest_client` parameter is an instance of the `Client` class or
193 | `None`. It represents the latest version of the client that needs to be updated. If it is
194 | `None`, the `check_update()` method is called to get the latest client version
195 | :type latest_client: Client or None
196 | :return: The function does not return anything. It has a return type annotation of `None`.
197 | """
198 |
199 | if latest_client is None:
200 | latest_client = self.get_latest_client()
201 |
202 | print("Updating...")
203 | downloader = Downloader(
204 | [latest_client.assets_url, latest_client.assets_url_2, latest_client.content_url],
205 | latest_client.content_hash,
206 | self.client.assets_path,
207 | self.config.max_workers,
208 | self.config.worker_max_items,
209 | )
210 |
211 | latest_chain = ItemChain.from_fingerprint(latest_client.fingerprint)
212 | current_chain = ItemChain.from_fingerprint(self.client.fingerprint)
213 |
214 | new_files, changed_files, deleted_files = ScDownloader.make_patch_chain(current_chain, latest_chain)
215 |
216 | if (len(new_files.items) == 0):
217 | print("There are no new files here")
218 | else:
219 | print("New Files: ")
220 | downloader.download_folder(new_files)
221 |
222 | print("Downloading changed files")
223 | Downloader.add_unlisted_items(changed_files)
224 | downloader.download_folder(changed_files)
225 |
226 | print("Deleting unnecessary files")
227 |
228 | # Some prepares for patching
229 | old_version = ".".join([str(num) for num in self.client.content_version])
230 | new_version = ".".join([str(num) for num in latest_client.content_version])
231 | patch_name = f"{old_version} {new_version}"
232 | patch_path = os.path.join(self.patches_path, patch_name)
233 | deleted_patch_path = os.path.join(patch_path, "deleted")
234 | changed_patch_path = os.path.join(patch_path, "changed")
235 | new_patch_path = os.path.join(patch_path, "new")
236 |
237 | def remove_files(folder: ItemChain, basepath: str = ""):
238 | # Macro optimization to avoid calling makedirs for each file
239 | destination_basepath: str or None = None
240 | if (self.config.make_detailed_patches):
241 | destination_basepath = os.path.join(deleted_patch_path, basepath)
242 | os.makedirs(destination_basepath, exist_ok=True)
243 |
244 | for item in folder.items:
245 | if isinstance(item, ItemChain):
246 | remove_files(item, posixpath.join(basepath, item.name))
247 | else:
248 | asset_path = os.path.join(self.client.assets_path, basepath, item.name)
249 |
250 | if (self.config.make_detailed_patches):
251 | asset_destination = os.path.join(destination_basepath, item.name)
252 |
253 | try:
254 | fmove(asset_path, asset_destination)
255 | except FileNotFoundError:
256 | print(f"Failed to move file: {os.path.normpath(asset_path)} -> {os.path.normpath(asset_destination)}")
257 |
258 | else:
259 | os.remove(asset_path)
260 |
261 | def move_files(folder: ItemChain, output_path: str, basepath: str = ""):
262 | destination_basepath = os.path.join(output_path, basepath)
263 | os.makedirs(destination_basepath, exist_ok=True)
264 |
265 | for item in folder.items:
266 | if isinstance(item, ItemChain):
267 | move_files(item, output_path, posixpath.join(basepath, item.name))
268 | else:
269 | asset_path = os.path.join(self.client.assets_path, basepath, item.name)
270 | asset_destination = os.path.join(destination_basepath, item.name)
271 |
272 | try:
273 | fcopy(asset_path, asset_destination)
274 | except FileNotFoundError:
275 | print(f"Failed to copy file: {os.path.normpath(asset_path)} -> {os.path.normpath(asset_destination)}")
276 |
277 |
278 | remove_files(deleted_files) # Deleted Files Move
279 | if (not self.config.make_patches): return
280 |
281 | move_files(new_files, new_patch_path if self.config.make_detailed_patches else patch_path) # New Files Copy
282 | move_files(changed_files, changed_patch_path if self.config.make_detailed_patches else patch_path) # Changed Files Copy
283 |
284 | def make_connect(self) -> bool:
285 | status = self.client.connect(self.active_server.server_address)
286 |
287 | if status == HelloServerResponse.Success:
288 | print(f"Successfully connected to {self.active_server.short_name}")
289 | return True
290 | elif status == HelloServerResponse.NeedUpdate:
291 | print(
292 | f"Successfully connected to {self.active_server.short_name} but server requires update. Updating..."
293 | )
294 | self.make_update()
295 | return True
296 |
297 | return False
298 |
299 | def __call__(self, *args: Any, **kwds: Any) -> Any:
300 | """
301 | The function checks if the client is connected to a server, updates the server if necessary,
302 | downloads assets if it's the first connection, and checks for updates and downloads them if
303 | requested.
304 | """
305 |
306 | major, _, _ = self.client.content_version
307 |
308 | # Downloading from scratch
309 | if major == 0:
310 | print("Detected first connection. This can take a little bit long time. Downloading all assets...")
311 | if (not self.make_connect()): return
312 | self.download_all()
313 | return
314 |
315 | if not self.client.fingerprint:
316 | if (not self.make_connect()): return
317 |
318 | # User hash handling
319 | if (self.config.custom_hash):
320 | print(f"Downloading assets using hash {self.config.custom_hash}")
321 | self.download_all()
322 | return
323 |
324 | if (self.config.repair):
325 | self.download_all()
326 | return
327 |
328 | # Update Check
329 | is_update_available, client_latest = self.check_update()
330 |
331 | if is_update_available:
332 | is_update_granted = self.config.auto_update
333 |
334 | old_version = ".".join([str(num) for num in self.client.content_version])
335 | new_version = ".".join([str(num) for num in client_latest.content_version])
336 |
337 | if (self.config.auto_update):
338 | print(f"New update found {old_version} -> {new_version}")
339 | else:
340 | is_update_granted = ScDownloader.ask_question_bool(f"New update found {old_version} -> {new_version}. Do you want download it?")
341 |
342 | if (is_update_granted):
343 | self.make_update(client_latest)
344 |
345 | else:
346 | print("All files are ok and do not require updates")
347 |
348 | if __name__ == "__main__":
349 | downloader = ScDownloader()
350 | downloader()
--------------------------------------------------------------------------------
/requirements.txt:
--------------------------------------------------------------------------------
1 | Requests
2 |
--------------------------------------------------------------------------------