├── .gitignore ├── README.md ├── config.json ├── lib ├── client.py ├── config.py ├── downloader.py ├── item_chain.py ├── reader.py └── writer.py ├── main.py └── requirements.txt /.gitignore: -------------------------------------------------------------------------------- 1 | /.idea/ 2 | /assets/ 3 | /patches/ 4 | /dumps/ 5 | /.vscode/ 6 | __pycache__ 7 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 |

2 |

(better) SC Downloader

3 |

4 | 5 | ### Description 6 | File downloader for all (probably) games from Supercell. It directly receives all the latest data from the server and downloads it. 7 | [This sc-downloader by vorono4ka was used as a base](https://github.com/Vorono4ka/sc-downloader), some code from it is still here. But it has been greatly improved: 8 | - Added support for latest game servers iterations 9 | - Added threading to improve downloading speed 10 | - Added file patching feature 11 | - Added file verify feature 12 | 13 | ### HOW TO USE 14 | You need to have at least Python 3.9 for this to work. 15 | If you want to run script for the first time, then install all modules with this command in cmd 16 | ``` 17 | pip install -r requirements.txt 18 | ``` 19 | Basic usage is very simple, just run script with a command like 20 | ``` 21 | py main.py 22 | ``` 23 | After that, select the game server you need. All assets will begin downloading into ```assets/{Server name}/``` 24 | 25 | ## In case if you need more 26 | There are additional flags here too. 27 | 28 | - ```--hash``` You can download assets directly using the version hash. Assets will be downloaded to a folder with the same name as hash. Example ```py main.py --hash=SomeVersionHash``` 29 | 30 | - ```--repair-mode``` and ```--strict-repair-mode``` is just flags. 31 | Normal mode checks if files exist and if not, downloads them. Useful if: You have downloaded apk or ipa of the game, you already have almost all the assets. You can unpack these assets into the folder of the desired server and run script with this flag, it will download all files that may not be in your assets like background textures or music. 32 | Strict mode checks all files based on their content and this can be a bit long. Useful if: You accidentally somehow replaced a file or its content. Run script with this flag and its contents will be restored. 33 | 34 | ## Patches 35 | Patches are a very useful feature if you just need to get new files from the latest update. 36 | It compares previous version and current one, and copies all new or changed files to ```patches/{Server name}/{old version name} {new version name}/``` 37 | If detailed patches are enabled in the config, then the patch will be divided into 3 parts, new files, changed files and deleted files. 38 | To enable or disable this feature, you can look in ```config.json```. 39 | 40 | ## Config 41 | ```config.json``` contains all settings for managing servers, threading and patches. 42 | - ```servers``` are stored in a dictionary, key of which means name and value of key means address of the game server. You can easily add your own server. 43 | - ```auto_update``` means whether files should be updated automatically. Otherwise, every time there is an optional update, script will ask about files updating. 44 | - ```make_patches``` and ```make_detailed_patches``` explained in patches description 45 | - ```max_workers``` or maximum count of threads. Each thread downloads its own list of files and this parameter sets maximum number of threads that can work at one time. Be careful, a large number of threads can either speed up or slow down process, for the most part it all depends on your PC. A large number of threads is not recommended on weak PCs. 46 | - ```worker_max_items``` sets the number of how many files one worker can process. This is made for large folders like folders with 3d graphics or sounds. This folders will be divided into pieces whose size depends on this value, and each piece will be processed by a new worker. 47 | - ```save_dump``` mostly needed for debugging messages from server. just don't touch it. 48 | 49 | -------------------------------------------------------------------------------- /config.json: -------------------------------------------------------------------------------- 1 | { 2 | "save_dump": false, 3 | "auto_update": false, 4 | "make_patches": true, 5 | "make_detailed_patches": false, 6 | "max_workers": 12, 7 | "worker_max_items": 50, 8 | "servers": { 9 | "BrawlStarsPROD": "game.brawlstarsgame.com", 10 | "BrawlStarsCN": "52.83.179.16" 11 | } 12 | } -------------------------------------------------------------------------------- /lib/client.py: -------------------------------------------------------------------------------- 1 | import os 2 | from socket import socket, create_connection 3 | import json 4 | from struct import unpack 5 | from .writer import Writer 6 | from .reader import Reader 7 | import zlib 8 | 9 | from enum import Enum 10 | 11 | 12 | class HelloServerResponse(Enum): 13 | Success = 7 14 | NeedUpdate = 8 15 | 16 | 17 | class Client: 18 | def __init__(self, assets_path: str) -> None: 19 | self.assets_path = assets_path 20 | self.fingerprint_filepath = os.path.join(assets_path, "fingerprint.json") 21 | 22 | self.socket: socket = None 23 | self.major = 0 24 | self.build = 0 25 | self.revision = 0 26 | self.dump = False 27 | self.assets_url = "" 28 | self.assets_url_2 = "" 29 | self.content_url = "" 30 | 31 | self.fingerprint: dict = {} 32 | if os.path.exists(self.fingerprint_filepath): 33 | data_file = open(self.fingerprint_filepath, "rb") 34 | self.fingerprint = json.load(data_file) 35 | data_file.close() 36 | 37 | self.major, self.build, self.revision = self.content_version 38 | 39 | @property 40 | def content_version(self) -> list[int, int, int]: 41 | if self.fingerprint: 42 | version = str(self.fingerprint["version"]).split(".") 43 | return [int(num) for num in version] 44 | 45 | return [0, 0, 0] 46 | 47 | @property 48 | def content_hash(self) -> str: 49 | if self.fingerprint: 50 | return self.fingerprint.get("sha") or "" 51 | 52 | return "" 53 | 54 | def handle_packet(self) -> bytes: 55 | header = self.socket.recv(7) 56 | packet_length = int.from_bytes(header[2:5], "big") 57 | 58 | received_data = b"" 59 | while packet_length > 0: 60 | chunk = self.socket.recv(packet_length) 61 | if not chunk: 62 | raise EOFError 63 | received_data += chunk 64 | packet_length -= len(chunk) 65 | 66 | return received_data 67 | 68 | def send_packet(self, id: int, data: bytes) -> bytes: 69 | packet = Writer() 70 | packet.writeUShort(id) 71 | packet.buffer += len(data).to_bytes(3, "big") 72 | packet.writeUShort(0) 73 | packet.buffer += data 74 | self.socket.send(packet.buffer) 75 | 76 | return self.handle_packet() 77 | 78 | def disconnect(self) -> None: 79 | self.socket.close() 80 | 81 | def connect(self, address: str) -> HelloServerResponse: 82 | self.socket = create_connection((address, 9339)) 83 | 84 | # HelloMessage 85 | stream = Writer() 86 | 87 | stream.writeUInt32(0) # Protocol Version 88 | stream.writeUInt32(11) # Key Version 89 | stream.writeUInt32(self.major) # major 90 | stream.writeUInt32(self.revision) # revision 91 | stream.writeUInt32(self.build) # build (minor) 92 | stream.writeString("") # ContentHash 93 | stream.writeUInt32(2) # DeviceType 94 | stream.writeUInt32(2) # AppStore 95 | 96 | server_data_buffer = self.send_packet(10100, stream.buffer) 97 | server_data_stream = Reader(server_data_buffer) 98 | 99 | if self.dump: 100 | os.makedirs("dumps/", exist_ok=True) 101 | open(f"dumps/{address}.{self.major}.{self.build}.10100.bin", "wb").write( 102 | server_data_buffer 103 | ) 104 | 105 | status_code = server_data_stream.readUInt32() 106 | 107 | if status_code == 7: 108 | server_data_stream.readUInt32() 109 | server_data_stream.readUInt32() 110 | 111 | self.content_url = server_data_stream.readString() 112 | 113 | server_data_stream.readUInt32() 114 | 115 | serialized_fingerprint = server_data_stream.readString() 116 | 117 | # If decompressed data length is 0 then decompress data with zlib 118 | if len(serialized_fingerprint) == 0: 119 | # Skip zeros bytes 120 | server_data_stream.seek(5, 1) 121 | 122 | # decompressInMySQLFormat 123 | compressed_data_length = server_data_stream.readUInt32() 124 | # For some reason decompressed size is in Little Endian 125 | decompressed_data_length = unpack(" None: 7 | self.short_name = name 8 | self.server_address = address 9 | 10 | class Config: 11 | def __init__(self, filepath: str) -> None: 12 | file = open(filepath, "rb") 13 | data: dict = json.load(file) 14 | file.close() 15 | 16 | # self.server_specific_data: dict = data["server_specific_data"] 17 | self.save_dump = True if data.get("save_dump") else False 18 | self.auto_update = True if data.get("auto_update") else False 19 | self.make_patches = True if data.get("make_patches") else False 20 | self.make_detailed_patches = ( 21 | True if data.get("make_detailed_patches") else False 22 | ) 23 | self.max_workers = data.get("max_workers") or 1 24 | self.worker_max_items = data.get("worker_max_items") or 1 25 | 26 | servers_data: dict = data.get("servers") 27 | self.servers: list[ServerDescriptor] = [] 28 | 29 | # Session Settings 30 | parser = argparse.ArgumentParser( 31 | prog="SC-Downloader", description="Asset Downloader for Supercell Games" 32 | ) 33 | 34 | parser.add_argument( 35 | "--hash", 36 | type=str, 37 | help="Specify your version hash by which you want to download assets", 38 | default=None, 39 | ) 40 | 41 | parser.add_argument( 42 | "--asset-servers", 43 | help="You can provide your own links to asset servers", 44 | nargs="+", 45 | default=None, 46 | ) 47 | 48 | parser.add_argument( 49 | "--repair-mode", 50 | action=argparse.BooleanOptionalAction, 51 | help="Checks if all files exist and loads them if they are missing", 52 | default=False, 53 | ) 54 | 55 | parser.add_argument( 56 | "--strict-repair-mode", 57 | action=argparse.BooleanOptionalAction, 58 | help="Checks file content and if it is corrupted, downloads it again. Much slower than default mode.", 59 | default=False, 60 | ) 61 | 62 | args = parser.parse_args() 63 | 64 | self.custom_hash: str = "" or args.hash 65 | self.asset_servers_override = args.asset_servers 66 | 67 | self.strict_repair: bool = args.strict_repair_mode 68 | self.repair: bool = args.repair_mode or self.strict_repair 69 | 70 | # Server specific variables 71 | self.status_code_size = 4 # int 72 | self.variable_schema: list = [] 73 | 74 | for server in servers_data: 75 | self.servers.append(ServerDescriptor(server, servers_data[server])) 76 | 77 | def load_server_specific_data(self, name: str) -> None: 78 | data = self.server_specific_data 79 | pass 80 | -------------------------------------------------------------------------------- /lib/downloader.py: -------------------------------------------------------------------------------- 1 | from .item_chain import ItemChain, Item 2 | from threading import Thread 3 | import os 4 | import posixpath 5 | import requests 6 | from hashlib import sha1 7 | 8 | class DownloaderWorker(Thread): 9 | def __init__( 10 | self, 11 | content_hash: str, 12 | assets_urls: list[str], 13 | assets_path: str, 14 | assets_basepath: str, 15 | folder: ItemChain, 16 | ) -> None: 17 | Thread.__init__(self) 18 | self.is_working = True 19 | self.assets_basepath = assets_basepath 20 | self.assets_path = assets_path 21 | self.folder = folder 22 | self.content_hash = content_hash 23 | self.assets_urls = assets_urls 24 | 25 | @staticmethod 26 | def download_file(urls: list[str], conent_hash: str, filepath: str) -> bytes or int: 27 | request: requests.Response = None 28 | for url in urls: 29 | request = requests.get( 30 | f"{url}/{conent_hash}/{filepath}" 31 | ) 32 | if request.status_code == 200: break 33 | 34 | # Final writing to file 35 | if request.status_code == 200: 36 | return request.content 37 | else: 38 | return request.status_code 39 | 40 | def run(self): 41 | """ 42 | The function downloads files from multiple URLs and saves them to a specified directory, 43 | providing status messages along the way. 44 | :return: The code is returning either None or stopping the execution of the function if the 45 | `self.is_working` flag is False. 46 | """ 47 | 48 | for item in self.folder.items: 49 | if not self.is_working: return 50 | 51 | if isinstance(item, ItemChain): 52 | continue 53 | 54 | base_filepath = posixpath.join(self.assets_basepath, item.name) 55 | 56 | server_response = DownloaderWorker.download_file(self.assets_urls, self.content_hash, base_filepath) 57 | 58 | if (isinstance(server_response, bytes)): 59 | with open(os.path.join(self.assets_path, base_filepath), "wb") as file: 60 | file.write(server_response) 61 | 62 | self.message(f"Downloaded {base_filepath}") 63 | else: 64 | self.message(f"Failed to download \"{base_filepath}\" with code {server_response}") 65 | 66 | self.message("Done") 67 | self.is_working = False 68 | 69 | def message(self, text: str): 70 | """ 71 | The function "message" prints a formatted message with the name of the object and the provided 72 | text. 73 | 74 | :param text: The `text` parameter is a string that represents the message that you want to print 75 | :type text: str 76 | """ 77 | 78 | print(f"[{self.name}] {text}") 79 | 80 | def DownloaderDecorator(function): 81 | def decorator(*args, **kwargs): 82 | try: 83 | return function(*args, **kwargs) 84 | except KeyboardInterrupt: 85 | args[0].stop_all_workers() 86 | exit(0) 87 | 88 | return decorator 89 | 90 | class Downloader: 91 | def __init__(self, 92 | content_urls: list[str], 93 | content_hash: str, 94 | output_folder: str, 95 | max_workers=8, 96 | worker_max_items=50, 97 | strict_level = 0) -> None: 98 | self.workers: list[DownloaderWorker] = [] 99 | self.max_workers = max_workers 100 | self.worker_max_items = worker_max_items 101 | self.output_folder = output_folder 102 | self.content_urls = content_urls 103 | self.content_hash = content_hash 104 | self.strict_level = strict_level 105 | 106 | @staticmethod 107 | def add_unlisted_items(folder: ItemChain): 108 | """ 109 | The function adds specific items to a given chain 110 | 111 | :param folder: The parameter "folder" is of type ItemChain 112 | :type folder: ItemChain 113 | """ 114 | 115 | # Fingerprint itself 116 | folder.items.append(Item("fingerprint.json", "")) 117 | 118 | # Version info 119 | folder.items.append(Item("version.number", "")) 120 | 121 | def check_workers_status(self) -> bool: 122 | """ 123 | The function checks the status of workers and returns True if all workers are not working, 124 | otherwise it returns False. 125 | :return: a boolean value. It returns True if there are no workers in the list, and False 126 | otherwise. 127 | """ 128 | 129 | i = 0 130 | while len(self.workers) > i: 131 | worker = self.workers[i] 132 | 133 | if not worker.is_working: 134 | del self.workers[i] 135 | else: 136 | i += 1 137 | 138 | if len(self.workers) == 0: 139 | return True 140 | 141 | return False 142 | 143 | def wait_for_workers(self) -> None: 144 | """ 145 | The function waits until all workers have finished their tasks. 146 | """ 147 | 148 | while True: 149 | if self.check_workers_status(): 150 | break 151 | 152 | def add_worker(self, basepath: str, chain: ItemChain) -> bool: 153 | """ 154 | The function adds a worker to the download queue if the maximum number of workers has not been 155 | reached. 156 | 157 | :param basepath: The `basepath` parameter is a string that represents the base path where the 158 | downloaded files will be saved. It is used to determine the location where the downloaded files 159 | will be stored 160 | :type basepath: str 161 | :param chain: The `chain` parameter is an instance of the `ItemChain` class. It represents a 162 | chain of items that need to be downloaded 163 | :type chain: ItemChain 164 | :return: a boolean value. If the length of the `self.workers` list is less than the maximum 165 | number of workers (`self.max_workers`), the function will return `True`. Otherwise, it will 166 | return `False`. 167 | """ 168 | 169 | if len(self.workers) >= self.max_workers: 170 | return False 171 | 172 | worker = DownloaderWorker( 173 | self.content_hash, 174 | self.content_urls, 175 | self.output_folder, 176 | basepath, 177 | chain 178 | ) 179 | 180 | print(f"[Main] {chain.name or 'Assets'} folder added to download queue") 181 | worker.start() 182 | self.workers.append(worker) 183 | 184 | return True 185 | 186 | def stop_all_workers(self): 187 | """ 188 | The function stops all workers by setting their "is_working" attribute to False and waits for 189 | them to finish their current tasks. 190 | """ 191 | 192 | for worker in self.workers: 193 | worker.is_working = False 194 | 195 | self.wait_for_workers() 196 | 197 | @DownloaderDecorator 198 | def download(self, folder: ItemChain, basepath: str = "") -> None: 199 | """ 200 | The `download` function takes a folder and downloads its contents, splitting them into worker 201 | chunks to be downloaded concurrently. 202 | 203 | :param folder: The `folder` parameter is an instance of the `ItemChain` class, which represents 204 | a collection of items. Each item can be either a file or a subfolder 205 | :type folder: ItemChain 206 | :param basepath: The `basepath` parameter is a string that represents the base path where the 207 | items will be downloaded. It is used to create the directory structure for the downloaded items 208 | :type basepath: str 209 | """ 210 | 211 | # Folders prepare 212 | current_dir = os.path.join(self.output_folder, basepath) 213 | os.makedirs( 214 | current_dir, exist_ok=True 215 | ) 216 | 217 | # worker_max_items sorting & existing files removing 218 | worker_chunks: list[ItemChain] = [] 219 | 220 | i = 0 221 | temp_chunk = ItemChain(folder.name) 222 | 223 | for item in folder.items: 224 | if isinstance(item, ItemChain): continue 225 | 226 | asset_path = os.path.join(current_dir, item.name) 227 | 228 | valid_file = False 229 | if (self.strict_level >= 1): 230 | valid_file = os.path.exists(asset_path) and len(item.hash) != 0 231 | 232 | if (self.strict_level >= 2): 233 | if (valid_file): 234 | with open(asset_path, "rb") as file: 235 | digest = sha1(file.read()) 236 | valid_file = digest.hexdigest() == item.hash 237 | 238 | if (valid_file): continue 239 | 240 | if (i >= self.worker_max_items): 241 | i = 0 242 | worker_chunks.append(temp_chunk) 243 | temp_chunk = ItemChain(folder.name) 244 | 245 | temp_chunk.items.append(item) 246 | i += 1 247 | 248 | # Append last small chunk 249 | if (len(temp_chunk.items) != 0): 250 | worker_chunks.append(temp_chunk) 251 | 252 | 253 | for worker_chunk in worker_chunks: 254 | while True: 255 | self.check_workers_status() 256 | if self.add_worker(basepath, worker_chunk): 257 | break 258 | 259 | for item in folder.items: 260 | if isinstance(item, Item): 261 | continue 262 | 263 | self.download(item, posixpath.join(basepath, item.name)) 264 | 265 | @DownloaderDecorator 266 | def download_folder(self, folder: ItemChain) -> None: 267 | """ 268 | The function `download_folder` downloads a folder and waits for the download to finish. 269 | 270 | :param folder: The `folder` parameter is of type `ItemChain`. It represents a folder or 271 | directory that needs to be downloaded 272 | :type folder: ItemChain 273 | """ 274 | 275 | print("Downloading...") 276 | self.download(folder) 277 | self.wait_for_workers() 278 | print("Downloading is finished") 279 | 280 | @DownloaderDecorator 281 | def download_fingerprint(self, fingerprint: dict) -> None: 282 | """ 283 | The function downloads a folder and its contents based on a given fingerprint. 284 | 285 | :param fingerprint: The `fingerprint` parameter is a dictionary that represents fingerprint data. 286 | :type fingerprint: dict 287 | """ 288 | 289 | root = ItemChain.from_fingerprint(fingerprint) 290 | Downloader.add_unlisted_items(root) 291 | self.download_folder(root) 292 | 293 | -------------------------------------------------------------------------------- /lib/item_chain.py: -------------------------------------------------------------------------------- 1 | from __future__ import annotations 2 | import os 3 | 4 | 5 | ''' Representation of "File" or asset with hash and name ''' 6 | class Item: 7 | def __init__(self, name: str, hash: str) -> None: 8 | self.name = name 9 | self.hash = hash 10 | 11 | ''' Representation of "Folder" or "Chain of asset files" ''' 12 | class ItemChain: 13 | def __init__(self, name: str, *args) -> None: 14 | self.name = name 15 | self.items: list[Item or ItemChain] = list(args) 16 | 17 | def get(self, name: str) -> Item or ItemChain or None: 18 | """ 19 | The function "get" searches for an item with a given name in a list of items and returns the 20 | item if found, otherwise it returns None. 21 | 22 | :param name: A string representing the name of the item to search for 23 | :type name: str 24 | :return: an instance of the class "Item" or "ItemChain" if an item with the specified name is found in the list 25 | of items. If no matching item is found, it returns "None". 26 | """ 27 | for item in self.items: 28 | if item.name == name: 29 | return item 30 | 31 | return None 32 | 33 | def get_chain(self, chain_names: list[str], auto_create=False) -> ItemChain or None: 34 | """ 35 | The `get_chain` function retrieves a specific chain from a list of chain names, optionally 36 | creating the chain if it doesn't exist. 37 | 38 | :param chain_names: A list of strings representing the names of the chains to be retrieved 39 | :type chain_names: list[str] 40 | :param auto_create: The `auto_create` parameter is a boolean flag that determines whether a new 41 | `ItemChain` should be automatically created if the specified chain name does not exist. If 42 | `auto_create` is set to `True`, a new `ItemChain` will be created and added to the `items` list, 43 | defaults to False (optional) 44 | :return: The function `get_chain` returns an instance of `ItemChain` or `None`. 45 | """ 46 | if (len(chain_names) == 0): return self 47 | 48 | result_item = None 49 | for chain_name in chain_names: 50 | iterable_item: ItemChain = result_item if result_item is not None else self 51 | iterable_item_result: ItemChain or None = None 52 | 53 | for item in iterable_item.items: 54 | if not isinstance(item, ItemChain): 55 | continue 56 | 57 | if item.name == chain_name: 58 | iterable_item_result = item 59 | break 60 | 61 | if iterable_item_result is None: 62 | if auto_create: 63 | result_item = ItemChain(chain_name) 64 | iterable_item.items.append(result_item) 65 | else: 66 | return None 67 | else: 68 | result_item = iterable_item_result 69 | 70 | return result_item 71 | 72 | @staticmethod 73 | def from_fingerprint(data: dict): 74 | """ 75 | The `from_fingerprint` function takes in a dictionary of file descriptors and creates a 76 | hierarchical structure of folders and files based on the file paths and hashes provided. 77 | 78 | :param data: The `data` parameter is a dictionary that contains information about asset files. 79 | :type data: dict 80 | :return: an instance of the ItemChain class, which represents a hierarchical structure of items 81 | (files and folders) based on the provided fingerprint data. 82 | """ 83 | root = ItemChain("") 84 | 85 | files: list[dict] = data["files"] 86 | 87 | for descriptor in files: 88 | name = str(descriptor["file"]) 89 | hash = str(descriptor["sha"]) 90 | 91 | folder = root 92 | 93 | basename = os.path.dirname(name) 94 | if (basename): 95 | folder_name_chain = os.path.normpath(basename).split(os.sep) 96 | else: 97 | folder_name_chain = [] 98 | 99 | folder: ItemChain = root.get_chain(folder_name_chain, True) 100 | folder.items.append(Item(os.path.basename(name), hash)) 101 | 102 | return root 103 | -------------------------------------------------------------------------------- /lib/reader.py: -------------------------------------------------------------------------------- 1 | from io import BufferedReader, BytesIO 2 | from struct import unpack 3 | 4 | 5 | class Reader(BufferedReader): 6 | def __init__(self, initial_bytes: bytes): 7 | super(Reader, self).__init__(BytesIO(initial_bytes)) 8 | 9 | def readUInt64(self) -> int: 10 | return unpack('>Q', self.read(8))[0] 11 | 12 | def readInt64(self) -> int: 13 | return unpack('>q', self.read(8))[0] 14 | 15 | def readUInt32(self) -> int: 16 | return unpack('>I', self.read(4))[0] 17 | 18 | def readInt32(self) -> int: 19 | return unpack('>i', self.read(4))[0] 20 | 21 | def readUInt16(self) -> int: 22 | return unpack('>H', self.read(2))[0] 23 | 24 | def readInt16(self) -> int: 25 | return unpack('>h', self.read(2))[0] 26 | 27 | def readUInt8(self) -> int: 28 | return unpack('>B', self.read(1))[0] 29 | 30 | def readInt8(self) -> int: 31 | return unpack('>b', self.read(1))[0] 32 | 33 | readULong = readUInt64 34 | readLong = readInt64 35 | 36 | readUShort = readUInt16 37 | readShort = readInt16 38 | 39 | readUByte = readUInt8 40 | readByte = readInt8 41 | 42 | def readChar(self, length: int = 1) -> str: 43 | return self.read(length).decode('utf-8') 44 | 45 | def readString(self) -> str: 46 | length = self.readUInt32() 47 | if length == 2**32 - 1 or length == 0: 48 | return '' 49 | else: 50 | return self.readChar(length) 51 | -------------------------------------------------------------------------------- /lib/writer.py: -------------------------------------------------------------------------------- 1 | from struct import pack 2 | 3 | 4 | class Writer: 5 | def __init__(self): 6 | super(Writer, self).__init__() 7 | self.buffer = b'' 8 | 9 | def writeUInt64(self, integer: int): 10 | self.buffer += pack('>Q', integer) 11 | 12 | def writeInt64(self, integer: int): 13 | self.buffer += pack('>q', integer) 14 | 15 | def writeUInt32(self, integer: int): 16 | self.buffer += pack('>I', integer) 17 | 18 | def writeInt32(self, integer: int): 19 | self.buffer += pack('>i', integer) 20 | 21 | def writeUInt16(self, integer: int): 22 | self.buffer += pack('>H', integer) 23 | 24 | def writeInt16(self, integer: int): 25 | self.buffer += pack('>h', integer) 26 | 27 | def writeUInt8(self, integer: int): 28 | self.buffer += pack('>B', integer) 29 | 30 | def writeInt8(self, integer: int): 31 | self.buffer += pack('>b', integer) 32 | 33 | writeULong = writeUInt64 34 | writeLong = writeInt64 35 | 36 | writeUShort = writeUInt16 37 | writeShort = writeInt16 38 | 39 | writeUByte = writeUInt8 40 | writeByte = writeInt8 41 | 42 | def writeString(self, string: str): 43 | encoded = string.encode('utf-8') 44 | self.writeUInt32(len(encoded)) 45 | self.buffer += encoded 46 | -------------------------------------------------------------------------------- /main.py: -------------------------------------------------------------------------------- 1 | import posixpath 2 | from typing import Any 3 | from lib.client import Client, HelloServerResponse 4 | from lib.config import Config 5 | from lib.downloader import Downloader, DownloaderWorker 6 | from lib.item_chain import ItemChain, Item 7 | import os 8 | from shutil import move as fmove 9 | from shutil import copyfile as fcopy 10 | 11 | class ScDownloader: 12 | def __init__(self) -> None: 13 | self.config = Config("config.json") 14 | 15 | # Servers 16 | print("Choose server to connect: ") 17 | 18 | for i, descriptor in enumerate(self.config.servers): 19 | print(f'{i}. "{descriptor.short_name}": {descriptor.server_address}"') 20 | 21 | server_index = int(input("\nServer index: ")) 22 | self.active_server = self.config.servers[server_index] 23 | 24 | self.assets_path = f"assets/{self.config.custom_hash or self.active_server.short_name}/" 25 | self.patches_path = f"patches/{self.active_server.short_name}" 26 | 27 | os.makedirs(self.assets_path, exist_ok=True) 28 | if (self.config.make_patches): 29 | os.makedirs(self.patches_path, exist_ok=True) 30 | 31 | # Downloading by hash stuff 32 | if (self.config.custom_hash): 33 | self.config.asset_servers_override = self.config.asset_servers_override or self.get_latest_asset_servers() 34 | 35 | hash_fingerprint = DownloaderWorker.download_file( 36 | self.config.asset_servers_override, 37 | self.config.custom_hash, 38 | "fingerprint.json" 39 | ) 40 | 41 | if (isinstance(hash_fingerprint, int)): 42 | raise Exception(f"Failed to get fingerprint.json by hash {self.config.custom_hash}. Request failed with code {hash_fingerprint}") 43 | 44 | os.makedirs(self.assets_path, exist_ok=True) 45 | with open(os.path.join(self.assets_path, "fingerprint.json"), "wb") as file: 46 | file.write(hash_fingerprint) 47 | 48 | self.client = Client(self.assets_path) 49 | self.client.dump = self.config.save_dump 50 | 51 | @staticmethod 52 | def ask_question_bool(question: str) -> bool: 53 | """ 54 | The function `ask_question_bool` prompts the user with a question and returns a boolean value 55 | based on their response. 56 | 57 | :param question: A string representing the question that will be asked to the user 58 | :type question: str 59 | :return: The function `ask_question_bool` returns a boolean value. 60 | """ 61 | 62 | answer = str(input(f"{question} (yes/no): ")).lower() 63 | 64 | if answer.startswith("y"): return True 65 | if answer.startswith("n"): return False 66 | 67 | if answer.isdigit(): 68 | return int(answer) >= 1 69 | 70 | @staticmethod 71 | def make_patch_chain(current: ItemChain, latest: ItemChain) -> list[ItemChain, ItemChain, ItemChain]: 72 | """ 73 | The `make_patch_chain` function takes in two `ItemChain` objects, `current` and `latest`, and 74 | returns a list of three `ItemChain` objects representing new files, changed files, and deleted 75 | files between the two chains. 76 | 77 | :param current: The `current` parameter is an instance of the `ItemChain` class, which 78 | represents the current state of a chain of items. It contains a list of items 79 | :type current: ItemChain 80 | :param latest: The "latest" parameter is an instance of the ItemChain class, which represents 81 | the most recent version of a chain of items. It contains a collection of items, where each item 82 | can be either a file or a folder. The items in the "latest" chain may have been added, modified 83 | or deleted 84 | :type latest: ItemChain 85 | :return: The function `make_patch_chain` returns a list containing three `ItemChain` objects: 86 | `new_files_result`, `changed_files_result`, and `deleted_files_result`. 87 | """ 88 | 89 | def make_chain(current: ItemChain, new: ItemChain) -> list[ItemChain, ItemChain, ItemChain]: 90 | current_items_names = [item.name for item in current.items] 91 | 92 | new_files_result = ItemChain(current.name) 93 | changed_files_result = ItemChain(current.name) 94 | deleted_files_result = ItemChain(current.name) 95 | deleted_files_indices = [True for _ in current.items] 96 | 97 | new_files_result = ItemChain(current.name) 98 | for new_item in new.items: 99 | # New Files 100 | if new_item.name not in current_items_names: 101 | new_files_result.items.append(new_item) 102 | continue 103 | 104 | current_item_index = current_items_names.index(new_item.name) 105 | current_item = current.items[current_item_index] 106 | deleted_files_indices[current_item_index] = False 107 | 108 | if isinstance(new_item, Item): 109 | # Changed Files 110 | if (new_item.hash != current_item.hash): 111 | changed_files_result.items.append(new_item) 112 | else: 113 | # Folder Processing 114 | new_files_chain, changed_files_chain, deleted_files_chain = make_chain(current_item, new_item) 115 | 116 | if (len(new_files_chain.items) != 0): 117 | new_files_result.items.append(new_files_chain) 118 | 119 | if (len(changed_files_chain.items) != 0): 120 | changed_files_result.items.append(changed_files_chain) 121 | 122 | if (len(deleted_files_chain.items) != 0): 123 | deleted_files_result.items.append(deleted_files_chain) 124 | 125 | for i in range(len(current.items)): 126 | current_item = current.items[i] 127 | is_deleted = deleted_files_indices[i] 128 | 129 | if (is_deleted): 130 | deleted_files_result.items.append(current_item) 131 | 132 | return [new_files_result, changed_files_result, deleted_files_result] 133 | return make_chain(current, latest) 134 | 135 | def download_all(self): 136 | """ 137 | Downloads all files from client fingerprint to `self.client.assets_path` 138 | """ 139 | 140 | asset_servers_urls = self.config.asset_servers_override or \ 141 | [self.client.assets_url, self.client.assets_url_2, self.client.content_url] 142 | 143 | downloader = Downloader( 144 | asset_servers_urls, 145 | self.client.content_hash, 146 | self.client.assets_path, 147 | self.config.max_workers, 148 | self.config.worker_max_items, 149 | int(self.config.repair) + int(self.config.strict_repair), 150 | ) 151 | downloader.download_fingerprint(self.client.fingerprint) 152 | 153 | def get_latest_client(self) -> Client: 154 | """ 155 | The function `get_latest_client` returns client with the latest data. 156 | :return: an instance of the Client class. 157 | """ 158 | client_latest = Client("") 159 | client_latest.major = client_latest.build = client_latest.revision = 0 160 | client_latest.connect(self.active_server.server_address) 161 | 162 | return client_latest 163 | 164 | def get_latest_asset_servers(self) -> list[str]: 165 | """ 166 | The function `get_latest_asset_servers` returns a list of asset servers from the latest client. 167 | :return: a list of strings, specifically the assets URLs of the latest client. 168 | """ 169 | client = self.get_latest_client() 170 | 171 | return [client.assets_url, client.assets_url_2, client.content_url] 172 | 173 | def check_update(self) -> tuple[bool, Client]: 174 | """ 175 | The function `check_update` compares the content version of the current client with the content 176 | version of the latest client on the active server and returns a tuple indicating whether they 177 | are different and the latest client. 178 | :return: a tuple containing two values. The first value is a boolean indicating whether the 179 | client version is different from the latest version, and the second value is an instance of the 180 | Client class representing the latest client version. 181 | """ 182 | client_latest = self.get_latest_client() 183 | is_different = False in [client_latest.content_version[i] <= self.client.content_version[i] for i in range(3)] 184 | # print(f"Current version is {client_latest.content_version}, Server Version is {self.client.content_version}") 185 | return (is_different, client_latest) 186 | 187 | def make_update(self, latest_client: Client or None = None) -> None: 188 | """ 189 | The function `make_update` is responsible for updating the client's assets by downloading new 190 | and changed files, and deleting unnecessary files. 191 | 192 | :param latest_client: The `latest_client` parameter is an instance of the `Client` class or 193 | `None`. It represents the latest version of the client that needs to be updated. If it is 194 | `None`, the `check_update()` method is called to get the latest client version 195 | :type latest_client: Client or None 196 | :return: The function does not return anything. It has a return type annotation of `None`. 197 | """ 198 | 199 | if latest_client is None: 200 | latest_client = self.get_latest_client() 201 | 202 | print("Updating...") 203 | downloader = Downloader( 204 | [latest_client.assets_url, latest_client.assets_url_2, latest_client.content_url], 205 | latest_client.content_hash, 206 | self.client.assets_path, 207 | self.config.max_workers, 208 | self.config.worker_max_items, 209 | ) 210 | 211 | latest_chain = ItemChain.from_fingerprint(latest_client.fingerprint) 212 | current_chain = ItemChain.from_fingerprint(self.client.fingerprint) 213 | 214 | new_files, changed_files, deleted_files = ScDownloader.make_patch_chain(current_chain, latest_chain) 215 | 216 | if (len(new_files.items) == 0): 217 | print("There are no new files here") 218 | else: 219 | print("New Files: ") 220 | downloader.download_folder(new_files) 221 | 222 | print("Downloading changed files") 223 | Downloader.add_unlisted_items(changed_files) 224 | downloader.download_folder(changed_files) 225 | 226 | print("Deleting unnecessary files") 227 | 228 | # Some prepares for patching 229 | old_version = ".".join([str(num) for num in self.client.content_version]) 230 | new_version = ".".join([str(num) for num in latest_client.content_version]) 231 | patch_name = f"{old_version} {new_version}" 232 | patch_path = os.path.join(self.patches_path, patch_name) 233 | deleted_patch_path = os.path.join(patch_path, "deleted") 234 | changed_patch_path = os.path.join(patch_path, "changed") 235 | new_patch_path = os.path.join(patch_path, "new") 236 | 237 | def remove_files(folder: ItemChain, basepath: str = ""): 238 | # Macro optimization to avoid calling makedirs for each file 239 | destination_basepath: str or None = None 240 | if (self.config.make_detailed_patches): 241 | destination_basepath = os.path.join(deleted_patch_path, basepath) 242 | os.makedirs(destination_basepath, exist_ok=True) 243 | 244 | for item in folder.items: 245 | if isinstance(item, ItemChain): 246 | remove_files(item, posixpath.join(basepath, item.name)) 247 | else: 248 | asset_path = os.path.join(self.client.assets_path, basepath, item.name) 249 | 250 | if (self.config.make_detailed_patches): 251 | asset_destination = os.path.join(destination_basepath, item.name) 252 | 253 | try: 254 | fmove(asset_path, asset_destination) 255 | except FileNotFoundError: 256 | print(f"Failed to move file: {os.path.normpath(asset_path)} -> {os.path.normpath(asset_destination)}") 257 | 258 | else: 259 | os.remove(asset_path) 260 | 261 | def move_files(folder: ItemChain, output_path: str, basepath: str = ""): 262 | destination_basepath = os.path.join(output_path, basepath) 263 | os.makedirs(destination_basepath, exist_ok=True) 264 | 265 | for item in folder.items: 266 | if isinstance(item, ItemChain): 267 | move_files(item, output_path, posixpath.join(basepath, item.name)) 268 | else: 269 | asset_path = os.path.join(self.client.assets_path, basepath, item.name) 270 | asset_destination = os.path.join(destination_basepath, item.name) 271 | 272 | try: 273 | fcopy(asset_path, asset_destination) 274 | except FileNotFoundError: 275 | print(f"Failed to copy file: {os.path.normpath(asset_path)} -> {os.path.normpath(asset_destination)}") 276 | 277 | 278 | remove_files(deleted_files) # Deleted Files Move 279 | if (not self.config.make_patches): return 280 | 281 | move_files(new_files, new_patch_path if self.config.make_detailed_patches else patch_path) # New Files Copy 282 | move_files(changed_files, changed_patch_path if self.config.make_detailed_patches else patch_path) # Changed Files Copy 283 | 284 | def make_connect(self) -> bool: 285 | status = self.client.connect(self.active_server.server_address) 286 | 287 | if status == HelloServerResponse.Success: 288 | print(f"Successfully connected to {self.active_server.short_name}") 289 | return True 290 | elif status == HelloServerResponse.NeedUpdate: 291 | print( 292 | f"Successfully connected to {self.active_server.short_name} but server requires update. Updating..." 293 | ) 294 | self.make_update() 295 | return True 296 | 297 | return False 298 | 299 | def __call__(self, *args: Any, **kwds: Any) -> Any: 300 | """ 301 | The function checks if the client is connected to a server, updates the server if necessary, 302 | downloads assets if it's the first connection, and checks for updates and downloads them if 303 | requested. 304 | """ 305 | 306 | major, _, _ = self.client.content_version 307 | 308 | # Downloading from scratch 309 | if major == 0: 310 | print("Detected first connection. This can take a little bit long time. Downloading all assets...") 311 | if (not self.make_connect()): return 312 | self.download_all() 313 | return 314 | 315 | if not self.client.fingerprint: 316 | if (not self.make_connect()): return 317 | 318 | # User hash handling 319 | if (self.config.custom_hash): 320 | print(f"Downloading assets using hash {self.config.custom_hash}") 321 | self.download_all() 322 | return 323 | 324 | if (self.config.repair): 325 | self.download_all() 326 | return 327 | 328 | # Update Check 329 | is_update_available, client_latest = self.check_update() 330 | 331 | if is_update_available: 332 | is_update_granted = self.config.auto_update 333 | 334 | old_version = ".".join([str(num) for num in self.client.content_version]) 335 | new_version = ".".join([str(num) for num in client_latest.content_version]) 336 | 337 | if (self.config.auto_update): 338 | print(f"New update found {old_version} -> {new_version}") 339 | else: 340 | is_update_granted = ScDownloader.ask_question_bool(f"New update found {old_version} -> {new_version}. Do you want download it?") 341 | 342 | if (is_update_granted): 343 | self.make_update(client_latest) 344 | 345 | else: 346 | print("All files are ok and do not require updates") 347 | 348 | if __name__ == "__main__": 349 | downloader = ScDownloader() 350 | downloader() -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | Requests 2 | --------------------------------------------------------------------------------