├── .gitignore ├── README.md ├── arcade_extract_ccf.py ├── arcade_extract_gng.py ├── arcade_utilities.py ├── brrencode3.py ├── ccfarchive.py ├── configurationfile.py ├── game_specific_patches.py ├── gensave.py ├── huf8.py ├── lz77.py ├── lzh8.py ├── n64crc.py ├── n64save.py ├── neogeo_acm.py ├── neogeo_cmc_gfx.py ├── neogeo_cmc_m1.py ├── neogeo_convert.py ├── neogeo_decrypt.py ├── neogeo_readme.txt ├── neogeo_sma.py ├── neogeo_smc.py ├── nes_extract.py ├── romchu.py ├── rso.py ├── snesrestore.py ├── tgcd_extract.py ├── tgsave.py ├── u8archive.py └── wiimetadata.py /.gitignore: -------------------------------------------------------------------------------- 1 | __pycache__ 2 | .vscode 3 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | vcromclaim 2 | ========== 3 | 4 | Intro 5 | ----- 6 | vcromclaim is a program to extract game ROMs from Wii Virtual Console games. 7 | It does this by analyzing an extracted Wii NAND filesystem, locating the ROMs, 8 | and extracting them. It automatically detects and decompresses compressed ROMs. 9 | It also extracts the game manual and save files for each Virtual Console game 10 | it encounters. 11 | 12 | Features 13 | -------- 14 | * Extracts virtually all NES/Famicom/Disk System, SNES, PC Engine / 15 | TurboGrafx16 / TurboGrafx CD, Mega Drive/Genesis, Master System, and 16 | Nintendo 64 games without fail! 17 | * Extracts several Neo Geo games, including encrypted games, along with the AES/MVS 18 | BIOS, so that they are playable in MAME 19 | * NAM-1975 20 | * Magician Lord 21 | * King of the Monsters 22 | * Spinmaster 23 | * Neo Turf Masters 24 | * Metal Slug 25 | * Real Bout Fatal Fury Special 26 | * Magical Drop 3 27 | * Shock Troopers 28 | * Metal Slug 2 29 | * The King of Fighters '98 30 | * The Last Blade 2 31 | * Shock Troopers 2 32 | * Metal Slug X 33 | * Metal Slug 3 34 | * Metal Slug 4 35 | * ...and support for many other Neo Geo games can be added relatively easily. 36 | Lots of caveats though - see below! 37 | * Can extract the arcade games Ghosts'N Goblins and Space Harrier. 38 | * Automatically extracts the built-in manuals in VC games. 39 | * Automatically extracts saves for most formats. 40 | * Cross-platform - compatible with Linux, Windows, Mac OS X, and any other 41 | platform supported by Python. Some games may require additional libraries to extract. 42 | * If the game/platform you want to extract is missing - please submit a bug to let me 43 | know there is demand for it! 44 | 45 | Requirements 46 | ------------ 47 | * [Python](http://python.org) 3.11.4 or newer. Older versions may work but 48 | are not tested or supported. 49 | * A NAND dump dumped by [BootMii](http://bootmii.org) and extracted by 50 | [ShowMiiWads](http://code.google.com/p/showmiiwads) or [nandextract](http://github.com/Plombo/showmiiwads) 51 | * Additional requirements applies for some Neo Geo games, see [neogeo_reame.txt](neogeo_readme.txt) 52 | 53 | Usage 54 | ----- 55 | The program is run by executing wiimetadata.py: 56 | 57 | python wiimetadata.py nand_directory 58 | 59 | Known Issues/Caveats 60 | -------------------- 61 | * ALL SYSTEMS: A lot of games have been modified for VC for various reasons. Same games may 62 | simply just behave differently from the original games, some games may not work properly 63 | in accurate emulators or on real hardware. Very often checksums will not be accurate. 64 | Known instances: 65 | * Removal of flashing graphics: 66 | * In Magical Drop 3, Tower character's flashing animation has been removed. 67 | * FDS games have been customized to let the VC emulator automatically switch disk 68 | side. You might get strange behaviour when the game normally would ask you to change the 69 | disk. Depending on emulator, just changing disk may work as usual. 70 | * Bio Miracle Bokutte Upa: Flashing "wait" screen 71 | * Zelda no Densetsu: "Press start" is shown, nothing happens if you do 72 | * Shadow of the Ninja (NES) - 2 bytes are different, causing the intro to 73 | glitch and freeze in accurate emulators. 74 | * Content changes: 75 | * Ogre Battle 64: "JIHAD" was renamed "LASER" for obvious reasons 76 | * TURBOGRAFX CD: CD audio is extracted in the wrong speed, because the quality is 77 | higher than required (48 kHz instead of 44.1kHz). Music in all games will play too slow 78 | in Mednafen, and a few games (like Super Air Zonk) are completely broken. 79 | Manually reencodeing the OGG files to 44.1kHz WAVE with e.g. Audacity should fix all this. 80 | * COMMODORE 64 games and most ARCADE games cannot be extracted at this time. 81 | * NEO GEO: Because of the way Neo Geo ROMs are made, part of the extraction 82 | process has to be hardcoded separately for each game. If your game is not 83 | supported, it might be trivial to expand neogeo_convert.py to include support 84 | for your game - please create a bug and I can try to implement it. 85 | * NEO GEO: For AES encrypted games, you will need to install PyCryptodome (or PyCrypto). 86 | Install it by typing in terminal: 87 | pip install pycryptodome 88 | * NEO GEO: About the system ROMs... 89 | NEO GEO games always require a set of them. Either the MVS (arcade) or AES (home) ROMs, 90 | each available in many different versions, and regional variations (jap/us/eu). 91 | Depending on what ROM is used, the game will automatically display different language, 92 | and different content. The MVS ROM will enable system menu, some generic "how to play" 93 | screens, "winners don't do drugs", etc. The AES ROMs are lighter and does not provide any 94 | of that. Also games ran with MVS ROMs will typically show "insert coin", but not when ran with 95 | AES ROMs. 96 | 97 | The Wii games comes bundled with weird system ROMs. Some games randomly comes shippd with 98 | MVS ROMs, others with AES ROMs. All are shipped with the Japanese ROMs. 99 | The system ROMs contain a few flags that tell the game what region/system it is in. 100 | Annoyingly, instead of shipping the correct system ROMs, all of them are instead patched 101 | to tell the game it is e.g. US AES. 102 | 103 | Also the MVS ROM sets are incomplete. They are missing the SFIX ROM, which contains all of the 104 | graphics used in system menu and how-to-play screens. (To allow the games to run, this tool 105 | creates a dummy SFIX file, but system menu etc is basically unusable.) 106 | 107 | This tool will extract whatever ROM was included. It will export an original version that 108 | is as close to original as possible, AND it will also create a number of different patches 109 | to simulate different Wii behaviours. You may have to experiment and try different ROM 110 | versions, to get the experience you want. Some combinations of games+system ROM may not work, 111 | or have audio problems. 112 | 113 | * If you want to have an US/EU Wii-like experience, use "XXX-patched-to-aes-us" or "XXX-patched-to-aes-eu" 114 | * If you want ARCADE-like experience, use "jp-mvs" or "XXX-patched-to-mvs-XX" 115 | * If you want the game to give a HOME CONSOLE experience, use "aes-jp" or "XXX-patched-to-aes-XX" 116 | * If you want to have an experience as ACCURATE as possible, use the "mvs-jp" or "aes-jp" 117 | * If you want the game to be in ENGLISH, use "XXX-patched-to-XX-us" or "XXX-patched-to-XX-eu" 118 | * If you want the game to be in JAPANESE, use "mvs-jp", "aes-jp" or "XXX-patched-to-XX-jp" 119 | 120 | 121 | 122 | Credits 123 | ------- 124 | * [Bryan Cain](https://github.com/Plombo) - author of vcromclaim 125 | * [JanErikGunnar](https://github.com/JanErikGunnar) - added extraction of Famicom FDS, 126 | TurboGrafx CD, Neo Geo and arcade games. 127 | * [Euan Forrester](https://github.com/euan-forrester) - TurboGrafx save file exporting 128 | * [hcs](http://hcs64.com) - author of C decompression code for Huf8, LZH8, and 129 | romchu, all of which I (Bryan) ported to Python for vcromextract. 130 | * [Hector Martin (marcan)](http://marcansoft.com/blog) - original author of the 131 | Python LZ77 decompression code, which I (Plombo) heavily modified and expanded for 132 | vcromextract. 133 | * [sepalani](https://github.com/sepalani/librso/blob/master/rvl/rso.py) - author of librso, 134 | with some reverse engineering done for RSO file format 135 | * [Bregalad](http://www.romhacking.net/community/1067) - author of BRRTools, 136 | a Java program on which the BRR encoder in vcromclaim was based. 137 | * qwikrazor87 - author of PCE CD Tools, of which the TG CD data decompression 138 | was based. 139 | * [blastar](http://www.yaronet.com/topics/185388-ngfx-neogeoneogeocd-graphicseditor) - author of NGFX, 140 | a very good Neo Geo graphics editor that was useful in creating the open SFIX substitute. 141 | * [ZOINKITY](https://pastebin.com/hcRjjTWg) - author of N64.py, containing the cart CRC code 142 | * [The Neo Geo Development Wiki](https://wiki.neogeodev.org) - very useful for extracting Neo Geo roms 143 | * [MAME](https://www.mamedev.org/) - the source code was very useful in extracting 144 | arcade and Neo Geo roms 145 | * [HxD](https://mh-nexus.de/en/hxd/) - great, free hex editor 146 | * [WiiBrew](https://wiibrew.org) - invaluable for any Wii development 147 | 148 | -------------------------------------------------------------------------------- /arcade_extract_ccf.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | 3 | #Extracts VC Arcade games from a CCF archive. 4 | 5 | import os, os.path 6 | 7 | from configurationfile import getConfiguration 8 | from rso import rso 9 | from arcade_utilities import getPart, getStripes, getBitStripe 10 | 11 | def extract_arcade_ccf(ccfArchive, outputFolder): 12 | config = ccfArchive.find('config') 13 | 14 | architecture = getConfiguration(config, "console.machine_arch") 15 | 16 | if architecture == 'sharrier': 17 | extract_SHARRIER(ccfArchive, create_rom_folder(outputFolder,'sharrier1')) 18 | #print_metadata(ccfArchive, config) 19 | return True 20 | else: 21 | print('Found unfamiliar game. Extraction scripts need to be updated to be able to extract this game.') 22 | # #print_metadata(ccfArchive, config) 23 | return False 24 | 25 | def extract_SHARRIER(ccfArchive, outputFolder): 26 | 27 | # https://github.com/mamedev/mame/blob/master/src/mame/drivers/segahang.cpp 28 | 29 | def convert_roadgfx(roadInput): 30 | 31 | # input = vc version format 32 | # output = original ROM format 33 | 34 | # this would probably be very similar for Super Hangon which use the same Arcade hardware 35 | 36 | outputHalfLength = int(len(roadInput) / 8) 37 | outputFullLength = int(outputHalfLength * 2) 38 | BITS_IN_BYTE = 8 39 | 40 | # real rom has twice as many "rows", but each "row" has 8x bytes as the VC ROM 41 | returnValue = bytearray(outputFullLength) 42 | 43 | for outputByteOffset in range(0, outputHalfLength): 44 | inputByteOffset = outputByteOffset * BITS_IN_BYTE 45 | explodedByte = roadInput[inputByteOffset : inputByteOffset+BITS_IN_BYTE] 46 | 47 | returnValue[outputByteOffset] = ( 48 | ((explodedByte[0] & 0x1) << 7) | 49 | ((explodedByte[1] & 0x1) << 6) | 50 | ((explodedByte[2] & 0x1) << 5) | 51 | ((explodedByte[3] & 0x1) << 4) | 52 | ((explodedByte[4] & 0x1) << 3) | 53 | ((explodedByte[5] & 0x1) << 2) | 54 | ((explodedByte[6] & 0x1) << 1) | 55 | ((explodedByte[7] & 0x1) ) 56 | ) 57 | 58 | returnValue[outputHalfLength + outputByteOffset] = ( 59 | ((explodedByte[0] & 0x2) << 6) | 60 | ((explodedByte[1] & 0x2) << 5) | 61 | ((explodedByte[2] & 0x2) << 4) | 62 | ((explodedByte[3] & 0x2) << 3) | 63 | ((explodedByte[4] & 0x2) << 2) | 64 | ((explodedByte[5] & 0x2) << 1) | 65 | ((explodedByte[6] & 0x2) ) | 66 | ((explodedByte[7] & 0x2) >> 1) 67 | ) 68 | 69 | return returnValue 70 | 71 | moduleFile = ccfArchive.find('sharrier.rso') 72 | module = rso(moduleFile) 73 | 74 | f = open(os.path.join(outputFolder, 'sharrier.rso'), 'wb') 75 | moduleFile.seek(0) 76 | f.write(moduleFile.read()) 77 | f.close() 78 | moduleFile.seek(0) 79 | 80 | #for export in module.getAllExports(): 81 | # print(" -- Export " + export) 82 | 83 | # The separate ROM files has been merged, we need to splice them so that MAME can load them. 84 | # Basically this is doing the reverse of what MAME does when loading the ROMs 85 | 86 | #maincpu = 68000 code 87 | cpu1 = get_rom_file(module, 'sharrier_rom_cpu1', 0x40000) 88 | save_rom_file(getStripes(getPart(cpu1,0,0x10000),[0,2]), outputFolder, 'epr-7188.ic97') 89 | save_rom_file(getStripes(getPart(cpu1,0,0x10000),[1,3]), outputFolder, 'epr-7184.ic84') 90 | save_rom_file(getStripes(getPart(cpu1,1,0x10000),[0,2]), outputFolder, 'epr-7189.ic98') 91 | save_rom_file(getStripes(getPart(cpu1,1,0x10000),[1,3]), outputFolder, 'epr-7185.ic85') 92 | save_rom_file(getStripes(getPart(cpu1,2,0x10000),[0,2]), outputFolder, 'epr-7190.ic99') 93 | save_rom_file(getStripes(getPart(cpu1,2,0x10000),[1,3]), outputFolder, 'epr-7186.ic86') 94 | save_rom_file(getStripes(getPart(cpu1,3,0x10000),[0,2]), outputFolder, 'epr-7191.ic100') 95 | save_rom_file(getStripes(getPart(cpu1,3,0x10000),[1,3]), outputFolder, 'epr-7187.ic87') 96 | 97 | #subcpu = second 68000 CPU 98 | cpu2 = get_rom_file(module, 'sharrier_rom_cpu2', 0x10000) 99 | save_rom_file(getStripes(cpu2,[0,2]), outputFolder, 'epr-7182.ic54') 100 | save_rom_file(getStripes(cpu2,[1,3]), outputFolder, 'epr-7183.ic67') 101 | 102 | #gfx1 = tiles 103 | # These are 3 bits per pixel. one bit is in each rom. 104 | # Then the hardware or emulator read from the three separate roms to build every pixel. 105 | # To speed things up on the Wii, the three roms has been merged into one linear rom. 106 | # We need to split them up again with getBitStripe 107 | # Lots of help from MAME source code to figure this out 108 | gfx1 = get_rom_file(module, 'sharrier_rom_grp1', 0x20000) 109 | save_rom_file(getBitStripe(gfx1, 0), outputFolder, 'epr-7196.ic31') 110 | save_rom_file(getBitStripe(gfx1, 1), outputFolder, 'epr-7197.ic46') 111 | save_rom_file(getBitStripe(gfx1, 3), outputFolder, 'epr-7198.ic60') 112 | #This has data, not sure what it is though 113 | #save_rom_file(getBitStripe(gfx1, 2), outputFolder, 'should-be-empty-but-is-not') 114 | 115 | #sprites 116 | sprites = get_rom_file(module, 'sharrier_rom_grp2', 0x100000) 117 | save_rom_file(getStripes(getPart(sprites,0,0x20000),[3]), outputFolder, 'epr-7230.ic36') 118 | save_rom_file(getStripes(getPart(sprites,0,0x20000),[2]), outputFolder, 'epr-7222.ic28') 119 | save_rom_file(getStripes(getPart(sprites,0,0x20000),[1]), outputFolder, 'epr-7214.ic18') 120 | save_rom_file(getStripes(getPart(sprites,0,0x20000),[0]), outputFolder, 'epr-7206.ic8') 121 | save_rom_file(getStripes(getPart(sprites,1,0x20000),[3]), outputFolder, 'epr-7229.ic35') 122 | save_rom_file(getStripes(getPart(sprites,1,0x20000),[2]), outputFolder, 'epr-7221.ic27') 123 | save_rom_file(getStripes(getPart(sprites,1,0x20000),[1]), outputFolder, 'epr-7213.ic17') 124 | save_rom_file(getStripes(getPart(sprites,1,0x20000),[0]), outputFolder, 'epr-7205.ic7') 125 | save_rom_file(getStripes(getPart(sprites,2,0x20000),[3]), outputFolder, 'epr-7228.ic34') 126 | save_rom_file(getStripes(getPart(sprites,2,0x20000),[2]), outputFolder, 'epr-7220.ic26') 127 | save_rom_file(getStripes(getPart(sprites,2,0x20000),[1]), outputFolder, 'epr-7212.ic16') 128 | save_rom_file(getStripes(getPart(sprites,2,0x20000),[0]), outputFolder, 'epr-7204.ic6') 129 | save_rom_file(getStripes(getPart(sprites,3,0x20000),[3]), outputFolder, 'epr-7227.ic33') 130 | save_rom_file(getStripes(getPart(sprites,3,0x20000),[2]), outputFolder, 'epr-7219.ic25') 131 | save_rom_file(getStripes(getPart(sprites,3,0x20000),[1]), outputFolder, 'epr-7211.ic15') 132 | save_rom_file(getStripes(getPart(sprites,3,0x20000),[0]), outputFolder, 'epr-7203.ic5') 133 | save_rom_file(getStripes(getPart(sprites,4,0x20000),[3]), outputFolder, 'epr-7226.ic32') 134 | save_rom_file(getStripes(getPart(sprites,4,0x20000),[2]), outputFolder, 'epr-7218.ic24') 135 | save_rom_file(getStripes(getPart(sprites,4,0x20000),[1]), outputFolder, 'epr-7210.ic14') 136 | save_rom_file(getStripes(getPart(sprites,4,0x20000),[0]), outputFolder, 'epr-7202.ic4') 137 | save_rom_file(getStripes(getPart(sprites,5,0x20000),[3]), outputFolder, 'epr-7225.ic31') 138 | save_rom_file(getStripes(getPart(sprites,5,0x20000),[2]), outputFolder, 'epr-7217.ic23') 139 | save_rom_file(getStripes(getPart(sprites,5,0x20000),[1]), outputFolder, 'epr-7209.ic13') 140 | save_rom_file(getStripes(getPart(sprites,5,0x20000),[0]), outputFolder, 'epr-7201.ic3') 141 | save_rom_file(getStripes(getPart(sprites,6,0x20000),[3]), outputFolder, 'epr-7224.ic30') 142 | save_rom_file(getStripes(getPart(sprites,6,0x20000),[2]), outputFolder, 'epr-7216.ic22') 143 | save_rom_file(getStripes(getPart(sprites,6,0x20000),[1]), outputFolder, 'epr-7208.ic12') 144 | save_rom_file(getStripes(getPart(sprites,6,0x20000),[0]), outputFolder, 'epr-7200.ic2') 145 | save_rom_file(getStripes(getPart(sprites,7,0x20000),[3]), outputFolder, 'epr-7223.ic29') 146 | save_rom_file(getStripes(getPart(sprites,7,0x20000),[2]), outputFolder, 'epr-7215.ic21') 147 | save_rom_file(getStripes(getPart(sprites,7,0x20000),[1]), outputFolder, 'epr-7207.ic11') 148 | save_rom_file(getStripes(getPart(sprites,7,0x20000),[0]), outputFolder, 'epr-7199.ic1') 149 | 150 | #gfx3 = road gfx - BROKEN! 151 | gfx3 = get_rom_file(module, 'sharrier_rom_grp3', 0x20000) 152 | save_rom_file(convert_roadgfx(gfx3), outputFolder, 'epr-7181.ic2') 153 | 154 | #soundcpu = sound CPU 155 | soundcpu = get_rom_file(module, 'sharrier_rom_cpu3', 0x008000) 156 | save_rom_file(getPart(soundcpu,0,0x004000), outputFolder, 'epr-7234.ic73') 157 | save_rom_file(getPart(soundcpu,1,0x004000), outputFolder, 'epr-7233.ic72') 158 | 159 | #pcm = Sega PCM sound data - NOTE: the two roms are stacked in reverse order vs the actual address range 160 | pcm = get_rom_file(module, 'sharrier_rom_pcm', 0x10000) 161 | save_rom_file(getPart(pcm,1,0x008000), outputFolder, 'epr-7231.ic5') 162 | save_rom_file(getPart(pcm,0,0x008000), outputFolder, 'epr-7232.ic6') 163 | 164 | #sprites:zoom = zoom table - OK 165 | save_rom_file(get_rom_file(module, 'sharrier_rom_prom', 0x2000), outputFolder, 'epr-6844.ic123') 166 | 167 | 168 | #TODO: create extract_ARCHITECTURE functions for other games 169 | 170 | 171 | #def print_metadata(ccfArchive, config): 172 | 173 | #Look for the module most specific to the game, and for exports that refers to roms 174 | 175 | # modules = string.split(getConfiguration(config, 'modules')) 176 | # for module in modules: 177 | # moduleFilename = module + '.rso' 178 | # print(' - Module ' + moduleFilename + ':') 179 | # moduleFile = ccfArchive.find(moduleFilename) 180 | # module = rso(moduleFile) 181 | # for export in module.getAllExports(): 182 | # print(" -- Export " + export) 183 | 184 | 185 | def create_rom_folder(parentFolder, romFolderName): 186 | newFolder = os.path.join(parentFolder, romFolderName) 187 | if not os.path.lexists(newFolder): 188 | os.makedirs(newFolder) 189 | return newFolder 190 | 191 | 192 | def get_rom_file(inputModule, inputName, size): 193 | return inputModule.getExport(inputName, size) 194 | 195 | def save_rom_file(romData, outputFolder, outputName): 196 | outputFile = open(os.path.join(outputFolder, outputName), 'wb') 197 | outputFile.write(romData) 198 | outputFile.close() 199 | 200 | 201 | 202 | 203 | 204 | 205 | 206 | 207 | 208 | 209 | 210 | 211 | 212 | -------------------------------------------------------------------------------- /arcade_extract_gng.py: -------------------------------------------------------------------------------- 1 | #FACTS about Ghosts n Goblins on Wii Virtual Console: 2 | # - When played on Wii, it looks like the English verison of the game. 3 | # - The Wii emulator also skips the glitching boot up screens and the RAM/ROM self-tests that are normal on real hardware. 4 | # Not sure how it is done, possibly the emulator has a built-in save-state that is always loaded. 5 | 6 | # - The ROMs stored in the Wii version is identical to "makaimurg" in MAME (Makai-Mura, Japanese revision G). It is NOT a clean match of any other MAME sets. 7 | # - 2 tiny roms at 256 bytes each are completely missing, but all other files have correct checksums. 8 | # - The missing files are not used by MAME, according to MAME's source code, so the game should be as accurate as it gets despite the checksum warning on startup. 9 | # - The rom set is the JAPANESE version of the game! 10 | 11 | # - The differences between Japanese and English versions of the game are very small, most notably the title screen is different. 12 | # - The Japanese ROMs even have the English title screen included in the ROM, probably there is just some flags in the P ROM deciding region. 13 | # - It seems the Wii emulator is patching the P ROMs. 2 of the 3 PROMs have a total of 10-20 changed bytes scattered throughout, 14 | # which probably change the title screen to English, possibly other things like reduce flickering. 15 | # - Applying the patches to the PROMs mentioned above will cause the self-test to fail ("ROM BAD"). Probably the P ROMs have a checksum that is not patched on Wii 16 | 17 | # - So, to get the English version as played on Wii would probably require us to 18 | # (1) Repeat the patches that are applied by Wii emulator 19 | # (2) Further patch the ROMs with correct checksum 20 | 21 | 22 | # ABOUT SAVE DATA: 23 | # Wii stores the high scores as part of save data. 24 | # MAME does NOT load or save high scores in this game. 25 | # Not sure if the real hardware keeps high score when power is off. 26 | # Since MAME does not read the save data, I don't know what format it should be exported to, so I don't export it. 27 | # But if needed, it should be very easy to export it. It is not compressed or anything. It might need to be truncated though, most of the file is blank. 28 | 29 | 30 | 31 | # ROM hashes taken from: 32 | # https://github.com/mamedev/mame/blob/master/src/mame/capcom/gng.cpp 33 | 34 | 35 | 36 | 37 | import os, hashlib, sys 38 | import lz77 39 | 40 | 41 | 42 | 43 | def create_rom_folder(parentFolder, romFolderName): 44 | newFolder = os.path.join(parentFolder, romFolderName) 45 | if not os.path.lexists(newFolder): 46 | os.makedirs(newFolder) 47 | return newFolder 48 | 49 | 50 | def export_rom(output_folder, file_name, dol_bytearray, index, length): 51 | outputfile = open(os.path.join(output_folder, file_name), 'wb') 52 | outputfile.write(dol_bytearray[index : index+length]) 53 | outputfile.close() 54 | 55 | 56 | def export_roms_from_dol(dol_bytearray, output_parent_folder): 57 | 58 | output_folder = create_rom_folder(output_parent_folder, 'makaimurg') 59 | 60 | # these two files does not exist in Wii, and according to mame source code, mame does not use them, but wont start without them 61 | dummy_files = bytearray(0x100) 62 | export_rom(output_folder, 'tbp24s10.14k', dummy_files, 0, 0x100) 63 | export_rom(output_folder, '63s141.2e', dummy_files, 0, 0x100) 64 | 65 | print("Scanning file for ROMs...") 66 | 67 | # This method is obviously terribly slow, but should be more reliable if there are different version of the game (different regions?) 68 | # and should be easier to adapt to other games 69 | 70 | for i in range(0, len(dol_bytearray)): 71 | 72 | digest = hashlib.sha1(dol_bytearray[i:i+0x8000]).hexdigest() 73 | if digest == '23a0a17d0abc4b084ffeba90266ef455361771cc': 74 | export_rom(output_folder, 'mj03g.bin', dol_bytearray, i, 0x8000) 75 | elif digest == '7694a981a6196d77fd2279fc34042b4cfb40c054': 76 | export_rom(output_folder, 'mj05g.bin', dol_bytearray, i, 0x8000) 77 | elif digest == '7ef9ec5c2072e21c787a6bbf700033f50c759c1d': 78 | export_rom(output_folder, 'gg2.bin', dol_bytearray, i, 0x8000) 79 | 80 | 81 | digest = hashlib.sha1(dol_bytearray[i:i+0x4000]).hexdigest() 82 | 83 | if digest == '07f7cf788810a1425016e016ce3579adb3253ac7': 84 | export_rom(output_folder, 'mj04g.bin', dol_bytearray, i, 0x4000) 85 | elif digest == '0a1518e19a2e0a4cc3dde4b9568202ea911b5ece': 86 | export_rom(output_folder, 'gg1.bin', dol_bytearray, i, 0x4000) 87 | elif digest == 'f9d77eee5e2738b7e83ba02fcc55dd480391479f': 88 | export_rom(output_folder, 'gg11.bin', dol_bytearray, i, 0x4000) 89 | elif digest == '8434c994cc55d2586641f3b90b6b15fd65dfb67c': 90 | export_rom(output_folder, 'gg10.bin', dol_bytearray, i, 0x4000) 91 | elif digest == 'bbb1fba0eb19471f66d29526fa8423ccb047bd63': 92 | export_rom(output_folder, 'gg9.bin', dol_bytearray, i, 0x4000) 93 | elif digest == '1c42fa02cb27b35d10c3f7f036005e747f9f6b79': 94 | export_rom(output_folder, 'gg8.bin', dol_bytearray, i, 0x4000) 95 | elif digest == '1947f159189b3a53f1251d8653b6e7c65c91fc3c': 96 | export_rom(output_folder, 'gg7.bin', dol_bytearray, i, 0x4000) 97 | elif digest == '944da1ce29a18bf0fc8deff78bceacba0bf23a07': 98 | export_rom(output_folder, 'gg6.bin', dol_bytearray, i, 0x4000) 99 | elif digest == '13e5a38a134bd7cfa16c63a18fa332c6d66b9345': 100 | export_rom(output_folder, 'gng13.n4', dol_bytearray, i, 0x4000) 101 | elif digest == '9e06012bcd82f98fad43de666ef9a75979d940ab': 102 | export_rom(output_folder, 'gg16.bin', dol_bytearray, i, 0x4000) 103 | elif digest == 'e3a1421d465b87148ffa94f5673b2307f0246afe': 104 | export_rom(output_folder, 'gg15.bin', dol_bytearray, i, 0x4000) 105 | elif digest == 'af207f9ee2f93a0cf9cf25cfe72b0fdfe55481b8': 106 | export_rom(output_folder, 'gng16.l4', dol_bytearray, i, 0x4000) 107 | elif digest == 'cb641c25bb04b970b2cbeca41adb792bbe142fb5': 108 | export_rom(output_folder, 'gg13.bin', dol_bytearray, i, 0x4000) 109 | elif digest == '3f129ca6d695548b659955fe538584bd9ac2ff17': 110 | export_rom(output_folder, 'gg12.bin', dol_bytearray, i, 0x4000) 111 | 112 | # these files don't seem to exist, and mame does not use them 113 | #digest = hashlib.sha1(dol_bytearray[i:i+0x100]).hexdigest() 114 | #if digest == 'bafd4108708f66cd7b280e47152b108f3e254fc9': 115 | # export_rom(output_folder, 'tbp24s10.14k', dol_bytearray, i, 0x100) 116 | #elif digest == '5018c3950b675af58db499e2883ecbc55419b491': 117 | # export_rom(output_folder, '63s141.2e', dol_bytearray, i, 0x100) 118 | 119 | if (i % 10240 == 0): 120 | sys.stdout.write("\r %5.2f%%" % ((100 * i) / (len(dol_bytearray)+1))) 121 | sys.stdout.flush() 122 | 123 | print() # start a new line after the progress counter 124 | 125 | 126 | def export_roms_from_lzh_compressed_dol(lzh_dol_file, output_parent_folder): 127 | print("Warning: Save data (highscores) are not exported because MAME does not persist them.") 128 | 129 | data = lz77.decompress_nonN64(lzh_dol_file) 130 | export_roms_from_dol(bytearray(data), output_parent_folder) 131 | -------------------------------------------------------------------------------- /arcade_utilities.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | 3 | import struct 4 | 5 | # Functions to split and splice ROMs. 6 | 7 | 8 | 9 | #Utilities 10 | 11 | def getAsymmetricPart(fileData, start, length): 12 | retVal = fileData[ start : (start+length) ] 13 | assert len(retVal) == length 14 | return retVal 15 | 16 | 17 | # e.g. if file data is 4 mb, and lengthInKb = 1024, then index 2 would retrieve the third mb of the region 18 | def getPart(fileData, index, length): 19 | retVal = fileData[ index*length : index*length + length ] 20 | assert len(retVal) == length 21 | return retVal 22 | 23 | # E.g. to get the first half of a ROM, call with partIndex = 0, partCount = 2. To get second half, call partIndex 1, partCount = 2. 24 | # fileData = the file data of the region. 25 | # partIndex = the part to retrieve. (0-based index) 26 | # partCount = the total number of parts. 27 | # The size of the returned value will be len(fileData) / partCount. 28 | def getPartByDivision(fileData, partIndex, partCount): 29 | assert partCount > 0 30 | assert partIndex >= 0 31 | assert partIndex < partCount 32 | 33 | partSize = int(len(fileData) / partCount) 34 | 35 | retVal = fileData[ partIndex*partSize : partIndex*partSize + partSize ] 36 | assert len(retVal) == partSize 37 | return retVal 38 | 39 | 40 | #stripes = [0] = get byte 0, 4, 8 etc 41 | #stripes = [0,1] = get byte 0,1,4,5,8,9 etc 42 | #stripes = [0,2] = get byte 0,2,4,6,8,10 etc 43 | #stripes = [1,3] = get byte 1,3,5,7,9,11 etc 44 | #stripes = [1,2,3,4] = get all bytes 45 | def getStripes(fileData, stripes): 46 | retVal = bytearray() 47 | for i in range(0, len(fileData), 4): 48 | for j in stripes: 49 | retVal.append(fileData[i+j]) 50 | return retVal 51 | 52 | def getBitStripe(fileData, romIndex): 53 | # Lots of help from mame source code to figure this out 54 | 55 | retVal = b'' 56 | #fileData is 4x the size of the striped roms. 57 | for i in range(0, len(fileData), 4): 58 | fourByteInteger = struct.unpack('>I', fileData[i:i+4])[0] 59 | singleOutputByte = 0 60 | 61 | for j in range(0,8): 62 | #outputByte = bit x1, x10, x100, x1000 for rom 1, x2, x20, x200 for rom 2, etc 63 | #bit x8, x80 etc are always 0 so stripe 4 can be skipped 64 | 65 | if (fourByteInteger & ((0x1+romIndex) << j*4)) > 0: 66 | singleOutputByte |= (0x80 >> j) 67 | 68 | assert singleOutputByte <= 0xFF 69 | 70 | retVal += struct.pack('>B', singleOutputByte) 71 | return retVal 72 | 73 | 74 | 75 | def pad(fileData, totalLength): 76 | assert totalLength >= len(fileData) 77 | return fileData + b'\xFF'*(totalLength-len(fileData)) 78 | 79 | 80 | 81 | #bitmangling functions converted from: 82 | # https://github.com/mamedev/mame/blob/519cd8b99af8264e4117e3061b0e2391902cfc02/src/lib/util/coretmpl.h 83 | 84 | # \brief Extract a single bit from an integer 85 | # 86 | # Extracts a single bit from an integer into the least significant bit 87 | # position. 88 | # 89 | # \param [in] x The integer to extract the bit from. 90 | # \param [in] n The bit to extract, where zero is the least 91 | # significant bit of the input. 92 | # \return Zero if the specified bit is unset, or one if it is set. 93 | # \sa bitswap 94 | #template constexpr T BIT(T x, U n) noexcept { return (x >> n) & T(1); } 95 | def get_bit(x, n): 96 | return (x >> n) & 1 97 | 98 | 99 | # \brief Extract bits in arbitrary order 100 | # 101 | # Extracts bits from an integer. Specify the bits in the order they 102 | # should be arranged in the output, from most significant to least 103 | # significant. The extracted bits will be packed into a right-aligned 104 | # field in the output. 105 | # 106 | # \param [in] val The integer to extract bits from. 107 | # \param [in] b The first bit to extract from the input 108 | # extract, where zero is the least significant bit of the input. 109 | # This bit will appear in the most significant position of the 110 | # right-aligned output field. 111 | # \param [in] c The remaining bits to extract, where zero is the 112 | # least significant bit of the input. 113 | # \return The extracted bits packed into a right-aligned field. 114 | #template constexpr T bitswap(T val, U b, V... c) noexcept 115 | def bitswap(val, c): 116 | 117 | #if constexpr (sizeof...(c) > 0U) 118 | if len(c) > 1: 119 | #return (BIT(val, b) << sizeof...(c)) | bitswap(val, c...); 120 | return (get_bit(val, c[0]) << (len(c)-1)) | bitswap(val, c[1:]) 121 | else: 122 | # return BIT(val, b); 123 | return get_bit(val, c[0]) 124 | 125 | -------------------------------------------------------------------------------- /brrencode3.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | # Author: Bryan Cain 3 | # Based on but heavily modified from BRRTools by Bregalad (written in Java) 4 | # Date: January 16, 2011 5 | # Description: Encodes 16-bit signed PCM data to SNES BRR format. 6 | 7 | import wave 8 | import struct 9 | 10 | class BRREncoder(object): 11 | def __init__(self, pcm): 12 | self.pcm_owner = False 13 | #self.brr_owner = False 14 | 15 | if type(pcm) == type(''): 16 | pcm = open(pcm, 'rb') 17 | self.pcm_owner = True 18 | #if type(brr) == type(''): 19 | # brr = open(brr, 'wb') 20 | # self.brr_owner = True 21 | 22 | self.pcm = pcm 23 | #self.brr = brr 24 | self.p1 = 0 25 | self.p2 = 0 26 | 27 | # clamps value to a signed short 28 | def sshort(self, n): 29 | if n > 0x7FFF: return (n - 0x10000) 30 | elif n < -0x8000: return n & 0x7FFF 31 | else: return n 32 | 33 | # short clamp_16(int n) 34 | def clamp_16(self, n): 35 | if n > 0x7FFF: return (0x7FFF - (n>>24)) 36 | else: return n 37 | 38 | # void ADPCMBlockMash(short[] PCMData) 39 | def ADPCMBlockMash(self, PCMData): 40 | smin=0 41 | kmin=0 42 | dmin=2**31 43 | 44 | for s in range(13, 0, -1): 45 | for k in range(4): 46 | d = self.ADPCMMash(s, k, PCMData, False) 47 | if d < dmin: 48 | kmin = k # Memorize the filter, shift values with smaller error 49 | dmin = d 50 | smin = s 51 | if dmin == 0.0: break 52 | if dmin == 0.0: break 53 | 54 | self.BRRBuffer[0] = (smin<<4)|(kmin<<2) 55 | self.ADPCMMash(smin, kmin, PCMData, True) 56 | 57 | # double ADPCMMash(int shiftamount, int filter, short[] PCMData, boolean write) 58 | def ADPCMMash(self, shiftamount, filter, PCMData, write): 59 | d2=0.0 60 | vlin=0 61 | l1 = self.p1 62 | l2 = self.p2 63 | step = 1<> 1 71 | vlin += (-l1) >> 5 72 | elif filter == 2: 73 | vlin = l1 74 | vlin += (-(l1 +(l1>>1)))>>5 75 | vlin -= l2 >> 1 76 | vlin += l2 >> 5 77 | else: 78 | vlin = l1 79 | vlin += (-(l1+(l1<<2) + (l1<<3)))>>7 80 | vlin -= l2>>1 81 | vlin += (l2+(l2>>1))>>4 82 | 83 | d = (PCMData[i]>>1) - vlin # Difference between linear prediction and current sample 84 | da = abs(d) 85 | 86 | if da > 16384 and da < 32768: 87 | d = d - 32768 * ( d >> 24 ) # Take advantage of wrapping 88 | dp = d + (step << 2) + (step >> 2) 89 | c = 0 90 | if dp > 0: 91 | if step > 1: 92 | c = int(dp /(step>>1)) 93 | else: 94 | c = dp<<1 95 | if c > 15: 96 | c = 15 97 | c -= 8 98 | dp = (c<<(shiftamount-1)) # quantized estimate of samp - vlin 99 | # edge case, if caller even wants to use it */ 100 | if shiftamount > 12: 101 | dp = ( dp >> 14 ) & ~0x7FF 102 | c &= 0x0f # mask to 4 bits 103 | l2 = l1 # shift history 104 | l1 = self.sshort(self.clamp_16(vlin + dp)*2) 105 | d = PCMData[i]-l1 106 | d2 += float(d)*d # update square-error 107 | 108 | if write: # if we want output, put it in proper place */ 109 | self.BRRBuffer[(i>>1)+1] |= c<<(4-((i&0x01)<<2)) 110 | 111 | if write: 112 | self.p2 = l2 113 | self.p1 = l1 114 | 115 | return d2 116 | 117 | # encodes the entire PCM file to BRR 118 | def encode(self): 119 | self.BRRBuffer = [0, 0, 0, 0, 0, 0, 0, 0, 0] # byte[9] 120 | 121 | #wav = wave.open(wav, 'rb') 122 | #if wav.getsampwidth() != 2: raise ValueError('must be 16 bits per sample') 123 | pcm = self.pcm 124 | #brr = self.brr 125 | self.p1 = 0 126 | self.p2 = 0 127 | 128 | samples2 = pcm.read(32) # the PCM samples in VC PCM files are misaligned for some reason 129 | while len(samples2) == 32: 130 | samples2 = struct.unpack('>16h', samples2) 131 | self.BRRBuffer = [0, 0, 0, 0, 0, 0, 0, 0, 0] 132 | self.ADPCMBlockMash(samples2) 133 | #brr.write(struct.pack('9B', *self.BRRBuffer)) 134 | #samples2 = wav.readframes(16) 135 | samples2 = pcm.read(32) 136 | 137 | #wav.close() 138 | if self.pcm_owner: pcm.close() 139 | #if self.brr_owner: brr.close() 140 | 141 | # offset: PCM offset (measured in samples, NOT in bytes) 142 | # returns: 9-byte BRR block 143 | def encode_block(self, offset): 144 | # read PCM - the PCM samples in VC PCM files are misaligned for some reason 145 | self.pcm.seek(offset * 2) 146 | samples2 = self.pcm.read(32) 147 | 148 | if len(samples2) != 32: 149 | raise ValueError('invalid PCM offset %d (file offset %d)' % (offset, offset*2)) 150 | 151 | samples2 = struct.unpack('>16h', samples2) 152 | 153 | # encode to BRR 154 | self.BRRBuffer = [0, 0, 0, 0, 0, 0, 0, 0, 0] 155 | self.ADPCMBlockMash(samples2) 156 | 157 | return struct.pack('9B', *self.BRRBuffer) 158 | 159 | 160 | if __name__ == '__main__': 161 | import sys 162 | if len(sys.argv) != 3: 163 | print('Usage: %s input.pcm' % sys.argv[0]) 164 | #enc = BRREncoder(sys.argv[1]) 165 | #enc.encode() 166 | print('Wrote file %s' % sys.argv[2]) 167 | 168 | -------------------------------------------------------------------------------- /ccfarchive.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | # Author: Bryan Cain (Plombo) 3 | # Date: December 27, 2010 4 | # Description: Reads Wii CCF archives, which contain Genesis and Master System ROMs. 5 | 6 | import struct 7 | import zlib 8 | from io import BytesIO 9 | 10 | class CCFArchive(object): 11 | # archive: a file-like object containing the CCF archive, OR the path to a CCF archive 12 | def __init__(self, archive): 13 | if type(archive) == type(''): 14 | self.file = open(archive, 'rb') 15 | else: 16 | self.file = archive 17 | self.files = [] 18 | self.readheader() 19 | 20 | def readheader(self): 21 | magic, zeroes1, rootnode_offset, numfiles, zeroes2 = struct.unpack('<4s12sII8s', self.file.read(32)) 22 | assert magic == b'CCF\0' 23 | assert zeroes1 == 12 * b'\0' 24 | assert rootnode_offset == 0x20 25 | assert zeroes2 == 8 * b'\0' 26 | for i in range(numfiles): 27 | fd = FileDescriptor(self.file) 28 | self.files.append(fd) 29 | 30 | def hasfile(self, path): 31 | for f in self.files: 32 | if f.name == path: return True 33 | return False 34 | 35 | def getfile(self, path): 36 | assert self.hasfile(path) 37 | fd = None 38 | for f in self.files: 39 | if f.name == path: fd = f 40 | return self.getfile2(fd) 41 | 42 | def getfile2(self, fd): 43 | self.file.seek(fd.data_offset * 32) 44 | string = self.file.read(fd.size) 45 | if fd.compressed: 46 | string = zlib.decompress(string) 47 | assert len(string) == fd.decompressed_size 48 | return BytesIO(string) 49 | 50 | # returns the requested file, even if the name is cut off inside the archive 51 | def find(self, name): 52 | for fd in self.files: 53 | if name.startswith(fd.name.rstrip()) or fd.name.startswith(name.rstrip()): return self.getfile2(fd) 54 | return None 55 | 56 | class FileDescriptor(object): 57 | # f: a file-like object of a CCF file at the position of this file descriptor 58 | def __init__(self, f): 59 | self.name, self.data_offset, self.size, self.decompressed_size = struct.unpack('<20sIII', f.read(32)) 60 | self.name = (self.name[0:self.name.find(b'\0')]).decode('ascii') 61 | self.compressed = (self.size != self.decompressed_size) 62 | 63 | if __name__ == '__main__': 64 | import os 65 | arc = CCFArchive(os.getenv('HOME') + '/wii/spinball/data.ccf') 66 | arc.getfile('SonicSpinball_USA.S') 67 | 68 | 69 | -------------------------------------------------------------------------------- /configurationfile.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | 3 | #utility to work with config files such as "config.ini" for TG16 games, and "config" for MasterSystem/Genesis/some Arcade games 4 | 5 | def getConfiguration(file, key): 6 | file.seek(0) 7 | keyPart = key.encode('ascii', 'strict') + b'=' 8 | for line in file: 9 | if line.startswith(keyPart): 10 | return line[len(keyPart):].strip(b'/\\\"\0\r\n').decode('ascii','strict') 11 | return None 12 | -------------------------------------------------------------------------------- /game_specific_patches.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | 3 | import hashlib 4 | 5 | # Dirty hacks to fix specific games 6 | 7 | # WARNING: For legal reasons, we cannot ADD any hardcoded data that is not alreday in the ROM. That would be piracy. 8 | # We can only delete, move, copy, convert etc data already in the ROM. 9 | # Likewise should we not document any actual data as found in any ROM. 10 | 11 | 12 | 13 | def patch_mario_tennis(romByteArray): 14 | print('Patching Mario Tennis') 15 | 16 | # Game contains an extra instruction in the code with an invalid opcode. 17 | # The instruction causes hardware and accurate emulators to crash. 18 | # This fixes that. 19 | 20 | romByteArray[(0x1070):(0x1070 + 0xC)] = romByteArray[(0x1070 + 0x04):(0x1070 + 0x04 + 0xC)] 21 | return romByteArray 22 | 23 | def patch_specific_games(romByteArray): 24 | hexDigest = hashlib.sha1(romByteArray).hexdigest() 25 | #print(hexDigest) 26 | 27 | funcs = { 28 | '36bcbb5b9b5592d05482ac677a0c54df51b122a1': patch_mario_tennis 29 | } 30 | if hexDigest in funcs.keys(): 31 | return funcs[hexDigest](romByteArray) 32 | else: 33 | return romByteArray 34 | -------------------------------------------------------------------------------- /gensave.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | # Author: Bryan Cain 3 | # Date: January 22, 2011 4 | # Description: Converts VC Genesis and VC Master System saves to the .srm/.ssm format used by Gens/GS/Kega Fusion. 5 | # The save format used by Genesis VC games was reverse engineered by Bryan Cain. 6 | 7 | import struct 8 | 9 | # src, dest: filesystem paths 10 | # genesis: 11 | # True if the output file is a .srm file for a Genesis game. 12 | # False if the output file is a .ssm file for a Master System game. 13 | def convert(src, dest, genesis): 14 | infile = open(src, 'rb') 15 | outfile = open(dest, 'wb') 16 | 17 | SIZE_SIZE = 4 18 | COMPOUND_DATA_MAGIC = b'compound data\x00\x00\x00' 19 | COMPOUND_DATA_MAGIC_SIZE = len(COMPOUND_DATA_MAGIC) 20 | SRAM_MAGIC = b'SRAM' 21 | SRAM_MAGIC_SIZE = len(SRAM_MAGIC) 22 | 23 | # MW4 = monster world IV, genesis NTSC 24 | # PS = phantasy star, sms NTSC 25 | # Those two games are confirmed working in GENS and Kega Fusion (latest versions as of july 2018). 26 | # the original reverse engineering was probably of some other games, which seem to have a somewhat different format. 27 | # i've tried to preserve the original functionality but have not been able to test it so I might have broken it. 28 | 29 | # read "VCSD" 30 | assert infile.read(4) == b'VCSD' 31 | 32 | # read four bytes = size 33 | totalSize = struct.unpack(' 0 35 | 36 | # read 4 or 5 bytes: 5 bytes 0x12C0A21004 for MW4, 5 bytes 0x0A53124504 for PS, 4 bytes for some games (which?) 37 | headSize = 4 38 | infile.seek(4,1) 39 | 40 | # then skip bytes until we get to "SRAM" magic word 41 | while infile.read(SRAM_MAGIC_SIZE) != SRAM_MAGIC and headSize < totalSize: 42 | infile.seek(-3,1) #step 3 bytes back and look again 43 | headSize = headSize + 1 44 | assert headSize == 4 or headSize == 5 45 | 46 | # read four bytes = size 47 | innerSize = struct.unpack('' + str(actualSramSize) + 'B', sramData) 75 | 76 | if (genesis): 77 | outType = 'H' 78 | else: 79 | outType = 'B' 80 | 81 | outfile.write(struct.pack('>' + str(actualSramSize) + outType, *intData)) 82 | 83 | outfile.close() 84 | infile.close() 85 | 86 | if __name__ == '__main__': 87 | import sys 88 | if len(sys.argv) != 3: 89 | sys.stderr.write('Usage: %s infile outfile\n' % sys.argv[0]) 90 | sys.exit(1) 91 | convert(sys.argv[1], sys.argv[2], True) 92 | print('Done') 93 | 94 | -------------------------------------------------------------------------------- /huf8.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | # Author: Bryan Cain (translated to Python from the "puff8" program by hcs, written in C) 3 | # Date: January 17, 2011 4 | # Description: Decompresses Nintendo's Huf8 compression used in Virtual Console games. 5 | 6 | import os, struct 7 | from array import array 8 | 9 | def decompress(infile, outfile): 10 | infile.seek(0, os.SEEK_END) 11 | #file_length = infile.tell() 12 | infile.seek(0, os.SEEK_SET) 13 | 14 | # read header 15 | magic_declength, symbol_count = struct.unpack('> 8 19 | symbol_count += 1 20 | 21 | # read decode table 22 | decode_table_size = symbol_count * 2 - 1 23 | decode_table = array('B', infile.read(decode_table_size)) 24 | 25 | ''' 26 | print("encoded size = %ld bytes (%d header + %ld body)" % ( 27 | file_length, 5 + decode_table_size, 28 | file_length - (5 + decode_table_size))) 29 | print("decoded size = %ld bytes" % decoded_length) 30 | ''' 31 | 32 | # decode 33 | bits = 0 34 | bits_left = 0 35 | table_offset = 0 36 | bytes_decoded = 0 37 | 38 | while bytes_decoded < decoded_length: 39 | if bits_left == 0: 40 | bits = struct.unpack("= decode_table_size: 49 | raise ValueError("reading past end of decode table") 50 | 51 | if ((not current_bit and (decode_table[table_offset] & 0x80)) or 52 | ( current_bit and (decode_table[table_offset] & 0x40))): 53 | outfile.write(decode_table[next_offset].to_bytes(1, byteorder='big')) 54 | bytes_decoded += 1 55 | # print("%02x" % decode_table[next_offset]) 56 | next_offset = 0 57 | 58 | if next_offset == table_offset: 59 | raise ValueError("infinite loop in Huf8 decompression") 60 | table_offset = next_offset 61 | bits_left -= 1 62 | bits <<= 1 63 | 64 | if __name__ == "__main__": 65 | import sys 66 | if len(sys.argv) != 3: 67 | sys.stderr.write("Usage: %s infile outfile\n" % sys.argv[0]) 68 | 69 | infile = open(sys.argv[1], "rb") 70 | outfile = open(sys.argv[2], "wb") 71 | 72 | decompress(infile, outfile) 73 | outfile.close() 74 | 75 | -------------------------------------------------------------------------------- /lz77.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | # Author: Bryan Cain (Plombo) 3 | # Original WiiLZ77 class by Hector Martin (marcan) 4 | # Date: December 30, 2010 5 | # Description: Decompresses LZ77-encoded files and compressed N64 ROMs. 6 | 7 | import sys, os, struct 8 | from array import array 9 | from io import BytesIO 10 | import romchu 11 | 12 | def decompress_lz77_lzss(file, inputOffset, outputLength): 13 | 14 | #print("Decompressing LZ77/LZSS") 15 | 16 | dout = array('B', b'\0' * outputLength) 17 | file.seek(inputOffset) 18 | outputOffset = 0 19 | 20 | while outputOffset < outputLength: 21 | flags = file.read(1)[0] 22 | 23 | for i in range(8): 24 | if flags & 0x80: 25 | info = struct.unpack(">H", file.read(2))[0] 26 | num = 3 + (info>>12) 27 | disp = info & 0xFFF 28 | ptr = outputOffset - disp - 1 29 | for i in range(num): 30 | dout[outputOffset] = dout[ptr] 31 | ptr += 1 32 | outputOffset += 1 33 | if outputOffset >= outputLength: 34 | break 35 | else: 36 | dout[outputOffset] = file.read(1)[0] 37 | outputOffset += 1 38 | flags <<= 1 39 | if outputOffset >= outputLength: 40 | break 41 | 42 | return dout 43 | 44 | def decompress_lz77_11(file, inputOffset, outputLength): 45 | #print("Decompressing LZ77 mode 11"() 46 | 47 | dout = array('B', b'\0'*outputLength) 48 | 49 | file.seek(inputOffset) 50 | outputOffset = 0 51 | 52 | 53 | while outputOffset < outputLength: 54 | 55 | flags = file.read(1)[0] 56 | 57 | for i in range(7, -1, -1): 58 | if (flags & (1< 0: 59 | info = struct.unpack(">H", file.read(2))[0] 60 | ptr, num = 0, 0 61 | if info < 0x2000: 62 | if info >= 0x1000: 63 | info2 = struct.unpack(">H", file.read(2))[0] 64 | ptr = outputOffset - (info2 & 0xFFF) - 1 65 | num = (((info & 0xFFF) << 4) | (info2 >> 12)) + 273 66 | else: 67 | info2 = file.read(1)[0] 68 | ptr = outputOffset - (((info & 0xF) << 8) | info2) - 1 69 | num = ((info&0xFF0)>>4) + 17 70 | else: 71 | ptr = outputOffset - (info & 0xFFF) - 1 72 | num = (info>>12) + 1 73 | for i in range(num): 74 | dout[outputOffset] = dout[ptr] 75 | outputOffset += 1 76 | ptr += 1 77 | if outputOffset >= outputLength: 78 | break 79 | else: 80 | dout[outputOffset] = file.read(1)[0] 81 | outputOffset += 1 82 | 83 | if outputOffset >= outputLength: 84 | break 85 | 86 | return dout 87 | 88 | def decompress_romchu(file, inputOffset, outputLength): 89 | # LZ77+Huffman (romchu) 90 | return romchu.decompress(file, inputOffset, outputLength) 91 | 92 | def decompress_n64(file): 93 | 94 | file.seek(0) 95 | 96 | # This header has a 30 bit size of the uncompressed file, and 2 bit flag (0x1 and 0x2 being known) 97 | # It has reversed byte order compared to the non-n64 header. 98 | inputOffset = 4 99 | hdr = struct.unpack(">I", file.read(4))[0] 100 | uncompressed_length = hdr>>2 101 | compression_type = hdr & 0x3 102 | 103 | if compression_type == 0x1: return decompress_lz77_lzss(file, inputOffset, uncompressed_length) 104 | elif compression_type == 0x2: return decompress_romchu(file, inputOffset, uncompressed_length) 105 | else: raise ValueError("Unsupported compression method %d"%compression_type) 106 | 107 | def decompress_nonN64(file): 108 | # This header MAY have magic word "LZ77" 109 | # Then it has a 24 bit size of the uncompressed file, and 8 bits fla (0x10 and 0x11 being known) 110 | # It has reversed byte order compared to the n64 header. 111 | 112 | hdr = file.read(4) 113 | if hdr != "LZ77": 114 | file.seek(0) 115 | lz77offset = file.tell() 116 | inputOffset = lz77offset + 4 117 | 118 | file.seek(lz77offset) 119 | 120 | hdr = struct.unpack(">8 122 | compression_type = hdr & 0xFF 123 | 124 | if compression_type == 0x11: return decompress_lz77_11(file, inputOffset, uncompressed_length) 125 | elif compression_type == 0x10: return decompress_lz77_lzss(file, inputOffset, uncompressed_length) 126 | else: raise ValueError("Unsupported compression method %d"%compression_type) 127 | 128 | 129 | if __name__ == '__main__': 130 | import time 131 | f = open(sys.argv[1], 'rb') 132 | 133 | start = time.process_time() 134 | if (sys.argv[2] == 'True'): 135 | du = decompress_n64(f) 136 | else: 137 | du = decompress_nonN64(f) 138 | 139 | end = time.process_time() 140 | print('Time: %.2f seconds' % (end - start)) 141 | 142 | f2 = open(sys.argv[2], 'wb') 143 | f2.write(''.join(du)) 144 | f2.close() 145 | f.close() 146 | 147 | -------------------------------------------------------------------------------- /lzh8.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | # Author: Bryan Cain 3 | # Original software (lzh8_dec 0.8) written in C by hcs 4 | # Date: January 17, 2011 5 | # Description: Decompresses Nintendo's N64 romc compression, type 2 (LZ77+Huffman) 6 | 7 | ''' 8 | LZH8 decompressor 9 | 10 | An implementation of LZSS, with symbols stored via two Huffman codes: 11 | - one for backreference lengths and literal bytes (8 bits each) 12 | - one for backreference displacement lengths (bits - 1) 13 | 14 | Layout of the compression: 15 | 16 | 0x00: 0x40 (LZH8 identifier) 17 | 0x01-0x03: uncompressed size (little endian) 18 | 0x04-0x07: optional 32-bit size if 0x01-0x03 is 0 19 | followed by: 20 | 21 | 9-bit prefix coding tree table (for literal bytes and backreference lengths) 22 | 0x00-0x01: Tree table size in 32-bit words, -1 23 | 0x02-: Bit packed 9-bit inner nodes and leaves, stored as in Huff8 24 | Total size: 2 ^ (leaf count + 1) 25 | 26 | 5-bit prefix coding tree table (for backreference displacement lengths) 27 | 0x00: Tree table size in 32-bit words, -1 28 | 0x01-: Bit packed 5-bit inner nodes and leaves, stored as in Huff8 29 | Total size: 2 ^ (leaf count + 1) 30 | 31 | Followed by compressed data bitstream: 32 | 1) Get a symbol from the 9-bit tree, if < 0x100 is a literal byte, repeat 1. 33 | 2) If 1 wasn't a literal byte, symbol - 0x100 + 3 is the backreference length 34 | 3) Get a symbol from the 5-bit tree, this is the length of the backreference 35 | displacement. 36 | 3a) If displacement length is zero, displacement is zero 37 | 3b) If displacement length is one, displacement is one 38 | 3c) If displacement length is > 1, displacement is next displen-1 bits, 39 | with an extra 1 on the front (normalized). 40 | 41 | Reverse engineered by hcs. 42 | ''' 43 | 44 | import sys, os, struct 45 | from array import array 46 | 47 | VERSION = "0.8" 48 | 49 | # debug output options 50 | SHOW_SYMBOLS = 0 51 | SHOW_FREQUENCIES = 0 52 | SHOW_TREE = 0 53 | SHOW_TABLE = 0 54 | 55 | # constants 56 | LENBITS = 9 57 | DISPBITS = 5 58 | LENCNT = (1 << LENBITS) 59 | DISPCNT = (1 << DISPBITS) 60 | 61 | # globals 62 | input_offset = 0 63 | bit_pool = 0 # uint8_t 64 | bits_left = 0 65 | 66 | # read MSB->LSB order 67 | '''static inline uint16_t get_next_bits( 68 | FILE *infile, 69 | long * const offset_p, 70 | uint8_t * const bit_pool_p, 71 | int * const bits_left_p, 72 | const int bit_count)''' 73 | def get_next_bits(infile, bit_count): 74 | global input_offset, bit_pool, bits_left 75 | 76 | offset_p = input_offset 77 | bit_pool_p = bit_pool 78 | bits_left_p = bits_left 79 | 80 | out_bits = 0 81 | num_bits_produced = 0 82 | while num_bits_produced < bit_count: 83 | if bits_left_p == 0: 84 | infile.seek(offset_p) 85 | bit_pool_p = struct.unpack(" (bit_count - num_bits_produced): 91 | bits_this_round = bit_count - num_bits_produced 92 | else: 93 | bits_this_round = bits_left_p 94 | 95 | out_bits <<= bits_this_round 96 | out_bits |= (bit_pool_p >> (bits_left_p - bits_this_round)) & ((1 << bits_this_round) - 1) 97 | 98 | bits_left_p -= bits_this_round 99 | num_bits_produced += bits_this_round 100 | 101 | input_offset = offset_p 102 | bit_pool = bit_pool_p 103 | bits_left = bits_left_p 104 | 105 | return out_bits 106 | 107 | # void analyze_LZH8(FILE *infile, FILE *outfile, long file_length) 108 | def decompress(infile): 109 | global input_offset, bit_pool, bits_left 110 | input_offset = 0 111 | bit_pool = 0 112 | bits_left = 0 113 | 114 | # determine input file size 115 | infile.seek(0, os.SEEK_END) 116 | #file_length = infile.tell() 117 | 118 | # read header 119 | infile.seek(input_offset) 120 | header = struct.unpack("> 8 123 | #if uncompressed_length == 0: 124 | # uncompressed_length = struct.unpack("= length_decode_table_size: 143 | break 144 | length_decode_table[i] = get_next_bits(infile, LENBITS) 145 | i += 1 146 | #if SHOW_TABLE: print("%ld: %d" % (i-1, length_decode_table[i-1])) 147 | input_offset = start_input_offset + length_table_bytes 148 | bits_left = 0 149 | #if SHOW_TABLE: print("done at 0x%lx" % input_offset) 150 | 151 | # allocate backreference displacement length decode table 152 | infile.seek(input_offset) 153 | displen_table_bytes = (struct.unpack("= length_decode_table_size: 164 | break 165 | displen_decode_table[i] = get_next_bits(infile, DISPBITS) 166 | i += 1 167 | #if SHOW_TABLE: print("%ld: %d" % (i-1, displen_decode_table[bit_pool = 0 # uint8_ti-1])) 168 | input_offset = start_input_offset + displen_table_bytes 169 | bits_left = 0 170 | 171 | #if SHOW_TABLE: print("done at 0x%lx" % input_offset) 172 | 173 | bytes_decoded = 0 174 | 175 | # main decode loop 176 | while bytes_decoded < uncompressed_length: 177 | length_table_offset = 1 178 | 179 | # get next backreference length or literal byte 180 | while True: 181 | next_length_child = get_next_bits(infile, 1) 182 | length_node_payload = length_decode_table[length_table_offset] & 0x7F 183 | next_length_table_offset = (int(length_table_offset / 2) * 2) + (length_node_payload + 1) * 2 + bool(next_length_child) 184 | next_length_child_isleaf = length_decode_table[length_table_offset] & (0x100 >> next_length_child) 185 | 186 | if next_length_child_isleaf: 187 | length = length_decode_table[next_length_table_offset] 188 | 189 | if 0x100 > length: 190 | # literal byte 191 | outbuf[bytes_decoded] = length 192 | bytes_decoded += 1 193 | else: 194 | # backreference 195 | length = (length & 0xFF) + 3 196 | displen_table_offset = 1 197 | 198 | # get backreference displacement length 199 | while True: 200 | next_displen_child = get_next_bits(infile, 1) 201 | displen_node_payload = displen_decode_table[displen_table_offset] & 0x7 202 | next_displen_table_offset = (int(displen_table_offset / 2) * 2) + (displen_node_payload + 1) * 2 + bool(next_displen_child) 203 | next_displen_child_isleaf = displen_decode_table[displen_table_offset] & (0x10 >> next_displen_child) 204 | 205 | if next_displen_child_isleaf: 206 | displen = displen_decode_table[next_displen_table_offset] 207 | displacement = 0 208 | 209 | if displen != 0: 210 | displacement = 1 # normalized 211 | 212 | # collect the bits 213 | #for (uint16_t i = displen-1; i > 0; i--) 214 | for i in range(displen-1, 0, -1): 215 | displacement *= 2 216 | next_bit = get_next_bits(infile, 1) 217 | 218 | displacement |= next_bit 219 | 220 | # apply backreference 221 | #for (long i = 0; i < length && bytes_decoded < uncompressed_length; bytes_decoded ++, i ++) 222 | for i in range(length): 223 | outbuf[bytes_decoded] = outbuf[bytes_decoded - displacement - 1] 224 | bytes_decoded += 1 225 | if bytes_decoded >= uncompressed_length: break 226 | 227 | break # break out of displen tree traversal loop 228 | else: 229 | assert next_displen_table_offset != displen_table_offset # stuck in a loop somehow 230 | displen_table_offset = next_displen_table_offset 231 | # end of displen tree traversal loop 232 | # end of if backreference !(0x100 > length)*/ 233 | break # break out of length tree traversal loop 234 | else: 235 | assert next_length_table_offset != length_table_offset # "stuck in a loop somehow" 236 | length_table_offset = next_length_table_offset 237 | # end of length tree traversal 238 | # end of main decode loop 239 | 240 | return outbuf 241 | 242 | 243 | if __name__ == "__main__": 244 | if len(sys.argv) != 3: 245 | print("lzh8_dec %s\n" % VERSION) 246 | print("Usage: %s infile outfile" % sys.argv[0]) 247 | sys.exit(1) 248 | 249 | # open file 250 | infile = open(sys.argv[1], "rb") 251 | outfile = open(sys.argv[2], "wb") 252 | 253 | # decompress 254 | print("Decompressing") 255 | infile.seek(0, os.SEEK_SET) 256 | output = decompress(infile) 257 | 258 | print("Writing to file") 259 | outfile.write(output) 260 | 261 | outfile.close() 262 | infile.close() 263 | 264 | sys.exit(0) 265 | 266 | 267 | -------------------------------------------------------------------------------- /n64crc.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | # Original author: ZOINKITY 3 | # Original version? https://pastebin.com/hcRjjTWg 4 | # CRC part copied and downgraded to Python 2 by JanErikGunnar 5 | 6 | import struct 7 | 8 | class Cart(): 9 | cic_names = { 10 | "starf":0x3F, 11 | "lylat":0x3F, 12 | "mario":0x3F, 13 | "diddy":0x78, 14 | "aleck":0xAC, 15 | "zelda":0x91, 16 | "yoshi":0x85, 17 | "ddipl":0xDD, 18 | "dddev":0xDD, 19 | "ddusa":0xDE, 20 | } 21 | 22 | def __init__(self, rom): 23 | """Sets ROM data, unswaps it if swapped, 24 | and extracts some handy header data. 25 | 26 | Expects rom to be a byte-like object.""" 27 | self.rom = bytearray(rom) 28 | self.programCounter = struct.unpack(">I",rom[8:12])[0] # int.from_bytes(data[8:12], byteorder='big') 29 | 30 | @staticmethod 31 | def cic2seed(cic): 32 | """Returns (name, seed) for . 33 | name: the normalized name of the chip 34 | seed: unencoded seed value. 35 | should be a str, matching the serial or short name of the chip. 36 | cic types: 37 | 'aleck', '5101' 38 | 'starf', '6101' 39 | 'lylat', '7102' 40 | 'mario', '6102', '7101' 41 | 'diddy', '6103', '7103' 42 | 'ddipl', '8303' 43 | 'dddev' 44 | 'ddusa' 45 | 'zelda', '6105', '7105' 46 | 'yoshi', '6106', '7106' 47 | """ 48 | # Normalize the names. (kinda makes the dict pointless...) 49 | cic = cic.lower() 50 | if cic in ("6102", "7101", "mario"): 51 | cic = "mario" 52 | elif cic in ("6101", "starf"): 53 | cic = "starf" 54 | elif cic in ("7102", "lylat"): 55 | cic = "starf" 56 | elif cic in ("6103", "7103", "diddy"): 57 | cic = "diddy" 58 | elif cic in ("6105", "7105", "zelda"): 59 | cic = "zelda" 60 | elif cic in ("6106", "7106", "yoshi"): 61 | cic = "yoshi" 62 | elif cic in ("5101", 'aleck'): 63 | cic = 'aleck' 64 | elif cic in ("8303", "ddipl"): 65 | cic = "ddipl" 66 | elif cic in ("8302", "8301", "dddev"): 67 | cic = "dddev" 68 | elif cic in ("ddusa"): 69 | cic = "ddusa" 70 | else: 71 | raise TypeError("Unknown CIC type {}.".format(cic)) 72 | return cic, Cart.cic_names.get(cic) 73 | 74 | @property 75 | def crc(self): 76 | """Returns CRC in rom as a tuple of integers.""" 77 | u = struct.unpack(">I",self.rom[16:20])[0] # int.from_bytes(self.rom[16:20], byteorder='big') 78 | l = struct.unpack(">I",self.rom[20:24])[0] # int.from_bytes(self.rom[20:24], byteorder='big') 79 | return (u, l) 80 | 81 | def calccrc(self, cic='mario', fix=False, seed=None, base=0x1000, seel=None): 82 | """Recalculates the CRC based on the CIC chip version given. 83 | Set fix to True to revise the crc in self.rom. 84 | 85 | cic types: 86 | 'aleck', '5101' 87 | 'starf', '6101' 88 | 'lylat', '7102' 89 | 'mario', '6102', '7101' 90 | 'diddy', '6103', '7103' 91 | 'ddipl', '8303' 92 | 'dddev' 93 | 'ddusa' 94 | 'zelda', '6105', '7105' 95 | 'yoshi', '6106', '7106' 96 | Setting seed overrides the normal seed value for these types. 97 | If you need to use one of the particular cic algorithms set the 98 | string version of the name in cic. Now that the Aleck64 99 | CIC has been identified this will probably not be necessary 100 | unless some repro cic chips start using different values.""" 101 | def rol(v, n): 102 | return (v % 0x100000000)>>n 103 | 104 | # The algo slightly changes with certain cics. 105 | cic, s = self.cic2seed(cic) 106 | # Pull the seed byte and generate the seed from it. 107 | # Note even if seed is set you'll have to pass a cic "name" to get funky types out. 108 | if seed is not None: 109 | s = seed 110 | if cic in ('diddy', 'yoshi', 'aleck'): 111 | seed = 0x6C078965 * s 112 | elif cic in ('ddipl', 'ddusa'): 113 | seed = 0x2E90EDD * s 114 | elif cic == 'dddev': 115 | seed = 0x260BCD5 * s 116 | else: 117 | seed = 0x5D588B65 * s 118 | seed+= 1 119 | seed&=0xFFFFFFFF 120 | r1, r2, r3, r4, r5, r6 = seed, seed, seed, seed, seed, seed 121 | 122 | # I wish there was a less horrifying way to do this... 123 | from array import array 124 | if seel is None: 125 | if cic == 'aleck': 126 | seel = 0x3FE000 if self.programCounter == 0x80100400 else 0x100000 127 | elif cic in ('ddipl', 'dddev', 'ddusa'): 128 | seel = 0xA0000 129 | else: 130 | seel = 0x100000 131 | seel += base 132 | l = min(seel, len(self.rom)) 133 | 134 | #in Python 3, bytes(7) returns a null array of length 7 135 | #in Python 3, bytes(7) returns a 136 | m = array("L", struct.unpack(">" + str(int((l-base)/4)) + "L", self.rom[base:l]) + tuple(bytearray(seel - l))) 137 | #Python3: m = array("L", self.rom[base:l] + bytes(seel - l)) 138 | #Python3: m.byteswap() 139 | 140 | # Zelda updates the second word a different way... 141 | if cic == 'zelda': 142 | from itertools import cycle 143 | n = array("L", struct.unpack(">" + str(int((0x850-0x750)/4)) + "L", self.rom[0x750:0x850])) 144 | #Python3: n = array("L", self.rom[0x750:0x850]) 145 | #Python3: n.byteswap() 146 | n = cycle(n) 147 | 148 | # Read each word as an integer. 149 | for i in m: 150 | v = (r1+i) & 0xFFFFFFFF 151 | if v < r1: 152 | if cic in ('ddipl', 'ddusa'): 153 | r2^=r3 154 | ## if cic == 'dddev': 155 | ## pass # invalid opcode in v1.0: 014A0000 156 | else: 157 | r2+=1 158 | v = i & 0x1F 159 | a = (i<I", r1) #r1.to_bytes(4, 'big') 214 | self.rom[20:24] = struct.pack(">I", r4) #r4.to_bytes(4, 'big') 215 | return (r1,r4) 216 | 217 | def bootstrapcrc(self, cic='mario', seed=None): 218 | if seed is None: 219 | cic, seed = self.cic2seed(cic) 220 | 221 | def trimult(v1, v2, v3): 222 | if not v2: 223 | v2 = v3 224 | l = v1 * v2 225 | u = (l>>32) & 0xFFFFFFFF 226 | l &= 0xFFFFFFFF 227 | u -= l 228 | if not u: 229 | u = v1 230 | return (u & 0xFFFFFFFF) 231 | 232 | def rol(v, n): 233 | return (v % 0x100000000)>>n 234 | 235 | data = struct.unpack(">1008L", self.rom[0x40:0x1000]) 236 | 237 | seed *= 0x6C078965 238 | seed += 1 239 | seed ^= data[0] 240 | seed &= 0xFFFFFFFF 241 | regs = [seed, seed, seed, seed, seed, seed, seed, seed, 242 | seed, seed, seed, seed, seed, seed, seed, seed,] 243 | count = 0 #S1 244 | cur = data[0] #S4 245 | # First half of the algorithm: read in bootcode. 246 | while True: 247 | prev = cur #S4 248 | cur = data[count] #S0 249 | count += 1 250 | v = (0x3EF - count) & 0xFFFFFFFF 251 | regs[0] += trimult(v, cur, count) 252 | regs[0] &= 0xFFFFFFFF 253 | regs[1] = trimult(regs[1], cur, count) 254 | regs[2] ^= cur 255 | regs[3] += trimult(cur + 5, 0x6C078965, count) 256 | regs[3] &= 0xFFFFFFFF 257 | # BFC00250 258 | if prev < cur: 259 | regs[9] = trimult(regs[9], cur, count) 260 | else: 261 | regs[9] += cur 262 | regs[9] &= 0xFFFFFFFF 263 | # BFC00288 264 | b = prev & 0x1F 265 | u = cur << (32 - b) 266 | l = rol(cur, b) 267 | roll1 = (u | l) & 0xFFFFFFFF #S5 268 | regs[4] += roll1 269 | regs[4] &= 0xFFFFFFFF 270 | b = prev & 0x1F 271 | u = cur << b 272 | l = rol(cur, 32 - b) 273 | v = (u | l) & 0xFFFFFFFF 274 | regs[7] = trimult(regs[7], v, count) 275 | # BFC002C0 276 | if cur < regs[6]: 277 | regs[6] += regs[3] 278 | regs[6] ^= cur + count 279 | else: 280 | regs[6] ^= regs[4] + cur 281 | regs[6] &= 0xFFFFFFFF 282 | # BFC002FC 283 | b = prev >> 0x1B 284 | u = cur << b 285 | l = rol(cur, 32 - b) 286 | roll2 = (u | l) & 0xFFFFFFFF #S2 287 | regs[5] += roll2 288 | regs[5] &= 0xFFFFFFFF 289 | u = cur << (32 - b) 290 | l = rol(cur, b) 291 | v = (u | l) & 0xFFFFFFFF 292 | regs[8] = trimult(regs[8], v, count) 293 | if count == 0x3F0: 294 | break 295 | future = data[count] #S3 296 | # BFC00340 297 | regs[15] = trimult(regs[15], roll2, count) 298 | b = rol(cur, 0x1B) 299 | u = future << b 300 | l = rol(future, 32 - b) 301 | v = (u | l) & 0xFFFFFFFF 302 | regs[15] = trimult(regs[15], v, count) 303 | # BFC00374 304 | regs[14] = trimult(regs[14], roll1, count) 305 | b = cur & 0x1F #S2 306 | u = future << (32 - b) 307 | l = rol(future, b) 308 | v = (u | l) & 0xFFFFFFFF 309 | regs[14] = trimult(regs[14], v, count) 310 | # BFC003A4 311 | u = cur << (32 - b) 312 | l = rol(cur, b) 313 | v = (u | l) & 0xFFFFFFFF 314 | regs[13] += v 315 | b = future & 0x1F 316 | u = future << (32 - b) 317 | l = rol(future, b) 318 | v = (u | l) & 0xFFFFFFFF 319 | regs[13] += v 320 | regs[13] &= 0xFFFFFFFF 321 | # BFC003DC 322 | regs[10] += cur 323 | regs[10] = trimult(regs[10] & 0xFFFFFFFF, future, count) 324 | regs[11] = trimult(regs[11] ^ cur, future, count) 325 | regs[12] += regs[8] ^ cur 326 | regs[12] &= 0xFFFFFFFF 327 | # BFC00420 328 | buf = [regs[0], regs[0], regs[0], regs[0]] 329 | for i in range(16): 330 | cur = regs[i] 331 | b = cur & 0x1F 332 | u = cur << (32 - b) 333 | l = rol(cur, b) 334 | v = (u | l) & 0xFFFFFFFF 335 | buf[0] += v 336 | buf[0] &= 0xFFFFFFFF 337 | if cur < buf[0]: 338 | buf[1] += cur 339 | buf[1] &= 0xFFFFFFFF 340 | else: 341 | buf[1] = trimult(buf[1], cur, i) 342 | # BFC00494 343 | if (cur & 3) in (0, 3): 344 | buf[2] += cur 345 | buf[2] &= 0xFFFFFFFF 346 | else: 347 | buf[2] = trimult(buf[2], cur, i) 348 | # BFC004CC 349 | if cur & 1: 350 | buf[3] ^= cur 351 | else: 352 | buf[3] = trimult(buf[3], cur, i) 353 | # BFC00504 354 | v = trimult(buf[0], buf[1], 16) 355 | b = buf[2] ^ buf[3] 356 | return (v & 0xFFFFFFFF, b & 0xFFFFFFFF) 357 | 358 | def getASMvalue(self, upper, lower=None): 359 | """If and are given, 360 | reads the address given by a LUI+ADDIU or LUI+ORI pair. 361 | If upper is None only the lower half will be read. 362 | If only is given, 363 | reads the address in a J or JAL instruction, | 80000000.""" 364 | if lower is None: 365 | u = struct.unpack(">H",self.rom[upper+2:upper+4])[0] & 0x3FFFFFF # int.from_bytes(self.rom[upper+2:upper+4], 'big') & 0x3FFFFFF 366 | u<<=2 367 | return u | 0x80000000 368 | if upper is None: 369 | u = 0 370 | else: 371 | u = struct.unpack(">H",self.rom[upper+2:upper+4])[0] <<16 #int.from_bytes(self.rom[upper+2:upper+4], 'big')<<16 372 | 373 | if self.rom[lower]&0x10: 374 | l = struct.unpack(">H",self.rom[upper+2:upper+4])[0] # int.from_bytes(self.rom[lower+2:lower+4], 'big', signed=False) 375 | else: 376 | l = struct.unpack(">h",self.rom[upper+2:upper+4])[0] # int.from_bytes(self.rom[lower+2:lower+4], 'big', signed=True) 377 | return u+l 378 | 379 | def setASMvalue(self, value, upper, lower=None): 380 | """If and are given, 381 | sets the LUI+ADDIU or LUI+ORI pair to value. 382 | If upper is None only the lower half will be set. 383 | If only is given, 384 | sets value as the address of a J or JAL instruction.""" 385 | if lower is None: 386 | value>>=2 387 | value&=0x3FFFFFF 388 | u = self.rom[upper] & 0xFC 389 | value|= u<<24 390 | self.rom[upper:upper+4] = struct.pack(">I", value) #value.to_bytes(4, 'big') 391 | else: 392 | u = value>>16 393 | l = value&0xFFFF 394 | if not self.rom[lower]&0x10: 395 | u += bool(l&0x8000) 396 | if upper is not None: 397 | self.rom[upper+2:upper+4] = struct.pack(">H", u) # u.to_bytes(2, 'big') 398 | self.rom[lower+2:lower+4] = struct.pack(">H", l) # l.to_bytes(2, 'big') 399 | 400 | def updateN64Crc(romdata): 401 | cart = Cart(romdata) 402 | 403 | cic = { 404 | 0:"starf", 405 | 0xD5:"lylat", 406 | 0xDE:"mario", 407 | 0xDB:"diddy", 408 | 0xE4:"aleck", 409 | 0x14:"zelda", 410 | 0xEC:"yoshi", 411 | }.get(cart.rom[0x173], 'mario') 412 | if cic == "starf": 413 | if cart.rom[0x16F]==0xDB: 414 | cic = "ddipl" if cart.rom[0x5F3] == 0x21 else "ddusa" 415 | elif cart.rom[0x16F]==0xD9: 416 | cic = "dddev" 417 | cic = Cart.cic2seed(cic) 418 | print("Updating the checksum...") 419 | print("CIC type\t%s" % cic[0]) 420 | if cic[0] in ('ddipl', 'dddev', 'ddusa'): 421 | print("internal\t{:08X} {:08X} {:08X} {:08X} {:08X} {:08X}".format(cart.getASMvalue(0x608, 0x60C), cart.getASMvalue(0x618, 0x61C), cart.getASMvalue(0x628, 0x62C), cart.getASMvalue(0x638, 0x63C), cart.getASMvalue(0x648, 0x64C), cart.getASMvalue(0x658, 0x65C))) 422 | print("calculated\t{:08X} {:08X} {:08X} {:08X} {:08X} {:08X}".format(*cart.calccrc(cic[0], fix=True))) 423 | else: 424 | print("internal\t{:08X} {:08X}".format(*cart.crc)) 425 | print("calculated\t{:08X} {:08X}".format(*cart.calccrc(cic[0], fix=True))) 426 | 427 | return cart.rom 428 | -------------------------------------------------------------------------------- /n64save.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | # Author: Bryan Cain 3 | # Date: January 22, 2011 4 | # Description: Converts Virtual Console N64 saves to Mupen64Plus N64 saves. 5 | # The save formats used by N64 Virtual Console games were reverse engineered by Bryan Cain. 6 | 7 | 8 | 9 | # EXAMPLES: 10 | # Mario Golf (North America) = 4e415545 = NMFE 11 | # Save file created in Project64: ".sra", >= 27 KiB (seems to cut after the last written byte) 12 | # Save file created in Wii: "RAM_MMFE", 48KiB (where the P64 file ends, the Wii file is padded with 0xAA) 13 | # Project64 is fine with using the 48 KiB Wii file as it is, so that is how we export it 14 | # Actual hardware: 32KiB battery-backed SRAM 15 | 16 | 17 | 18 | 19 | import os, shutil, struct 20 | 21 | # Converts (byte-swaps) Nintendo N64 SRAM and/or Flash RAM saves to little endian 22 | # SRAM and/or Flash RAM saves that can be used by Mupen64Plus and other emulators. 23 | def convert_sram(src, name, size): 24 | # determine output extensions 25 | if size == 32*1024: 26 | ext = '.sra' 27 | if size == 48*1024: 28 | ext = '.sra' 29 | elif size == 128*1024: 30 | ext = '.fla' 31 | elif size == 256*1024: 32 | ext = '.fla' # this might be the wrong extension, fix if needed 33 | 34 | # copy original file as a big-endian save file 35 | shutil.copy2(src, name+'.be'+ext) 36 | 37 | # open files 38 | infile = open(src, 'rb') 39 | outfile = open(name+'.le'+ext, 'wb') 40 | 41 | # byte-swap file 42 | while True: 43 | data = infile.read(8192) 44 | if len(data) != 8192: 45 | if len(data) != 0: raise ValueError('SRAM save file size should be a multiple of 8 KB') 46 | break 47 | 48 | intdata = struct.unpack('>2048I', data) 49 | outfile.write(struct.pack('<2048I', *intdata)) 50 | 51 | outfile.close() 52 | infile.close() 53 | 54 | # Converts (truncates) Nintendo N64 EEPROM saves to the appropriate size so they 55 | # can be used with Mupen64Plus and other N64 emulators. 56 | def convert_eeprom(src, name): 57 | infile = open(src, 'rb') 58 | outfile = open(name + '.eep', 'wb') 59 | 60 | data = infile.read(2048) 61 | if len(data) != 2048: raise ValueError('EEPROM save file size should be at least 2 KB') 62 | outfile.write(data) 63 | 64 | outfile.close() 65 | infile.close() 66 | 67 | def convert(src, name): 68 | f = open(src, 'rb') 69 | f.seek(0, os.SEEK_END) 70 | size = f.tell() 71 | f.close() 72 | 73 | if size in (4*1024, 16*1024): convert_eeprom(src, name) 74 | elif size in (32*1024, 48*1024, 128*1024, 256*1024): convert_sram(src, name, size) 75 | else: raise ValueError('unknown save type (size=%d bytes)' % size) 76 | 77 | if __name__ == '__main__': 78 | import sys 79 | if len(sys.argv) != 3: 80 | sys.stderr.write('Usage: %s infile outfile\n' % sys.argv[0]) 81 | sys.exit(1) 82 | convert(sys.argv[1], sys.argv[2]) 83 | print('Done') 84 | 85 | -------------------------------------------------------------------------------- /neogeo_acm.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | 3 | import sys 4 | 5 | # Decompress ACM files, used by some Neo Geo games on the Virtual Console. 6 | # Reverse engineered by JanErikGunnar using Dolphin. 7 | 8 | # The files are used to compress C ROMs for Neo Geo games. 9 | # The Virtual Console emulator decompress on demand, because the C ROMs are too large to fit in the Wii RAM. 10 | # Hence, the algorithm is very simple. 11 | # This implementation however, is not very optimized :) 12 | 13 | # 0x0: magic word ACM, followed by null padding. 14 | # 0x10: Number of blocks in block table, e.g. 0x0000 0400. 15 | # The output file size will be this value * 0x1 0000. 16 | # 0x14: Always 0001 followed by null padding? 17 | # 0x20: Translation table 18 | # Fixed length 19 | # For 0 .. 0xFF: 20 | # 0x00: either 0 or 1 21 | # 0x01: If value at 0x00 is 0x00, this is the output byte. 22 | # if value at 0x00 is 0x01, this is another entry in this table. Recursively replace the byte with values from that entry instead. 23 | # 0x02: similar to 0x00 24 | # 0x03: similar to 0x01 25 | # 0x420: Block table 26 | # For each block: 27 | # 0x0: (4 bytes) address of block start (4 bytes), from top of file 28 | # 0x4: (2 bytes) length of compressed data in block (from start of block) 29 | # 0x6: (2 bytes) always 0xFFFF 30 | # [number of blocks in block table * 8 + 0x420]: Blocks of data 31 | # For each block (starting at positions specified in block table): 32 | # [0 - length of compressed data]: First the compressed data (has the length specified in block table) 33 | # [length of compressed data - ceil(length of compressed data/8)]: Flags determining whether trans 34 | # Each byte in the first block has a corresponding bit in the second. 35 | # If the bit in the second block is 1, the byte in the first block should be replaced with two or more values as defined by the translation table. 36 | # If the bit is the second block is 0, the byte in the first block should be used as is. 37 | 38 | 39 | # Once decompressed, the data revealed is still different from the original roms, maybe to reduce the load required of the Wii CPU to read the sprites? 40 | # We have to convert it back to the Neo Geo sprite data structure. 41 | 42 | # Very useful: 43 | # https://wiki.neogeodev.org/index.php?title=Sprite_graphics_format 44 | 45 | 46 | 47 | import struct 48 | 49 | 50 | def decompressAcm(inputAcmStr): 51 | 52 | def translateValue(byteValue): 53 | translationtableEntryPos = TRANSLATION_TABLE_POS + (byteValue << 2) 54 | char0 = inputArray[translationtableEntryPos+0] 55 | char1 = inputArray[translationtableEntryPos+1] 56 | char2 = inputArray[translationtableEntryPos+2] 57 | char3 = inputArray[translationtableEntryPos+3] 58 | 59 | if char0 == 0x1: 60 | retVal = translateValue(char1) 61 | elif char0 == 0x0: 62 | retVal = bytearray([char1]) 63 | else: 64 | raise ValueError(char0) 65 | 66 | if char2 == 0x1: 67 | retVal.extend(translateValue(char3)) 68 | elif char2 == 0x0: 69 | retVal.append(char3) 70 | else: 71 | raise ValueError(char2) 72 | 73 | return retVal 74 | 75 | inputArray = bytearray(inputAcmStr) 76 | outputPosition = 0 77 | 78 | blockCount = struct.unpack('>I', inputArray[0x10:0x14])[0] 79 | 80 | outputArray = bytearray(blockCount * 0x10000) 81 | TRANSLATION_TABLE_POS = 0x20 82 | BLOCK_TABLE_POS = 0x420 83 | BLOCK_TABLE_ENTRY_SIZE = 0x8 84 | 85 | 86 | #perform translations all at once for faster processing of the compressed data 87 | translations = [translateValue(i) for i in range(0x00,0x100)] 88 | 89 | print("Decompressing ACM blocks...") 90 | 91 | for blockIndex in range(0, blockCount): 92 | assert blockIndex < blockCount 93 | 94 | blockTableEntryPos = BLOCK_TABLE_POS + BLOCK_TABLE_ENTRY_SIZE*blockIndex 95 | blockTableEntry = struct.unpack('>IHxx', inputArray[blockTableEntryPos:blockTableEntryPos+8]) 96 | compressedDataStart = blockTableEntry[0] 97 | compressedDataLength = blockTableEntry[1] 98 | flagStart = compressedDataStart + compressedDataLength 99 | 100 | for compressedDataIndex in range(0, compressedDataLength): 101 | 102 | # read the flag after the compressed data 103 | useTranslation = ( 104 | ( 105 | ( 106 | inputArray[flagStart + int(compressedDataIndex / 8)] 107 | ) >> (7 - (compressedDataIndex % 8)) 108 | ) & 0x1 109 | ) == 0x1 110 | 111 | if useTranslation: 112 | translatedBytes = translations[inputArray[compressedDataStart + compressedDataIndex]] 113 | for translatedByteIndex in range(0, len(translatedBytes)): 114 | outputArray[outputPosition] = translatedBytes[translatedByteIndex] 115 | outputPosition = outputPosition + 1 116 | else: 117 | outputArray[outputPosition] = inputArray[compressedDataStart + compressedDataIndex] 118 | outputPosition = outputPosition + 1 119 | 120 | 121 | sys.stdout.write("\r %d of %d (%5.2f%%)" % (blockIndex+1, blockCount, 100.0 * ((blockIndex+1) / blockCount))) 122 | sys.stdout.flush() 123 | 124 | 125 | assert outputPosition == blockCount * 0x10000 126 | 127 | print() # start a new line after the progress counter 128 | 129 | return convertToRegularSpriteData(outputArray) 130 | 131 | 132 | 133 | # Takes the CROM data that was decompressed from "ACM", and gives it in the normal Neo Geo format. (bitplanes 0,1,2,3 in one string, in that order) 134 | def convertToRegularSpriteData(inputByteArray): 135 | assert (len(inputByteArray) % 0x80) == 0 136 | 137 | outputByteArray = bytearray(b'\x00' * len(inputByteArray)) 138 | 139 | print ("Converting sprites...") 140 | 141 | for spritePosition in range(0, len(inputByteArray), 0x80): 142 | # Each sprite is 16x16 = 256 pixels 143 | # Each sprite is stored as 128 bytes, both in input and output 144 | # Each pixel is always 4 bits (16 different colours). 145 | # In input (the decompressed ACM file), each byte represents two pixels. 146 | # In output (that matches the memory layout of the NG), each byte represents 1 bit = 1/4 of 8 pixels. 147 | # Note that the output file then needs to be split into C1,C2,C3... files, as all other VC NG games 148 | 149 | # read four bytes from the input file, convert them to four bytes in the output file. 150 | 151 | for inputIndex in range (0x00, 0x80, 0x04): 152 | 153 | # Read 8 pixels from 4 bytes. The & operation is not needed, but below we only use the four last bits. 154 | inputBytes = inputByteArray[spritePosition + inputIndex : spritePosition + inputIndex + 4] 155 | 156 | ip = [ 157 | ((inputBytes[0]) >> 4), # & 0x0F = 00001111 158 | ((inputBytes[0]) ), # & 0x0F 159 | ((inputBytes[1]) >> 4), # & 0x0F 160 | ((inputBytes[1]) ), # & 0x0F 161 | ((inputBytes[2]) >> 4), # & 0x0F 162 | ((inputBytes[2]) ), # & 0x0F 163 | ((inputBytes[3]) >> 4), # & 0x0F 164 | ((inputBytes[3]) ) # & 0x0F 165 | ] 166 | 167 | # ACTUAL row within the sprite: 168 | row = (inputIndex >> 3) # 0-F 169 | #rowWithinBlock = row % 0x8 170 | 171 | #outputIndex = prependrows * 0x4 + rowWithinBlock * 0x4 + 0 172 | if (inputIndex % 0x8 == 0): # whether the four pixels is in the left half of the sprite or not 173 | #outputIndex = 2*0x8*0x4 + row * 0x4 + 0 174 | outputIndex = 0x40 + row * 0x4 175 | else: 176 | outputIndex = row * 0x4 177 | 178 | # Convert the bits to the bytes in the output format. 179 | 180 | outputByteArray[spritePosition + outputIndex : spritePosition + outputIndex + 4] =[ 181 | # 0 everything except bitplane 0 for all 8 pixels, store in byte 0 182 | ( 183 | ((ip[0] & 0x1) << 7) | 184 | ((ip[1] & 0x1) << 6) | 185 | ((ip[2] & 0x1) << 5) | 186 | ((ip[3] & 0x1) << 4) | 187 | ((ip[4] & 0x1) << 3) | 188 | ((ip[5] & 0x1) << 2) | 189 | ((ip[6] & 0x1) << 1) | 190 | ((ip[7] & 0x1) ) 191 | ), 192 | 193 | # 0 everything except bitplane 1 for all 8 pixels, store in byte 1 194 | ( 195 | ((ip[0] & 0x2) << 6) | 196 | ((ip[1] & 0x2) << 5) | 197 | ((ip[2] & 0x2) << 4) | 198 | ((ip[3] & 0x2) << 3) | 199 | ((ip[4] & 0x2) << 2) | 200 | ((ip[5] & 0x2) << 1) | 201 | ((ip[6] & 0x2) ) | 202 | ((ip[7] & 0x2) >> 1) 203 | ), 204 | 205 | # 0 everything except bitplane 2 for all 8 pixels, store in byte 2 206 | ( 207 | ((ip[0] & 0x4) << 5) | 208 | ((ip[1] & 0x4) << 4) | 209 | ((ip[2] & 0x4) << 3) | 210 | ((ip[3] & 0x4) << 2) | 211 | ((ip[4] & 0x4) << 1) | 212 | ((ip[5] & 0x4) ) | 213 | ((ip[6] & 0x4) >> 1) | 214 | ((ip[7] & 0x4) >> 2) 215 | ), 216 | 217 | # 0 everything except bitplane 3 for all 8 pixels, store in byte 3 218 | ( 219 | ((ip[0] & 0x8) << 4) | 220 | ((ip[1] & 0x8) << 3) | 221 | ((ip[2] & 0x8) << 2) | 222 | ((ip[3] & 0x8) << 1) | 223 | ((ip[4] & 0x8) ) | 224 | ((ip[5] & 0x8) >> 1) | 225 | ((ip[6] & 0x8) >> 2) | 226 | ((ip[7] & 0x8) >> 3) 227 | )] 228 | 229 | if spritePosition % (0x8000) == 0 or spritePosition+0x80 == len(inputByteArray): 230 | sys.stdout.write("\r %d of %d (%5.2f%%)" % (int((spritePosition) / 0x80)+1, int(len(inputByteArray) / 0x80), 100.0 * (spritePosition+0x80) / len(inputByteArray))) 231 | sys.stdout.flush() 232 | 233 | print() # start a new line after the progress counter 234 | 235 | return outputByteArray -------------------------------------------------------------------------------- /neogeo_cmc_m1.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | 3 | 4 | # Python re-implementation of https://github.com/mamedev/mame/blob/master/src/devices/bus/neogeo/prot_cmc.cpp (M1 part) 5 | # but also with encryption, so that MAME can decrypt it 6 | 7 | # WARNING: minor adjustments to the input and output. 8 | # this converts back and forth between the Wii ROM and an original ROM, while 9 | # mame converts from an original ROM to the address space expected by the game code/hardware. 10 | 11 | 12 | import arcade_utilities 13 | 14 | 15 | 16 | #static const uint8_t m1_address_8_15_xor[256] = { 17 | m1_address_8_15_xor = [ 18 | 0x0a, 0x72, 0xb7, 0xaf, 0x67, 0xde, 0x1d, 0xb1, 0x78, 0xc4, 0x4f, 0xb5, 0x4b, 0x18, 0x76, 0xdd, 19 | 0x11, 0xe2, 0x36, 0xa1, 0x82, 0x03, 0x98, 0xa0, 0x10, 0x5f, 0x3f, 0xd6, 0x1f, 0x90, 0x6a, 0x0b, 20 | 0x70, 0xe0, 0x64, 0xcb, 0x9f, 0x38, 0x8b, 0x53, 0x04, 0xca, 0xf8, 0xd0, 0x07, 0x68, 0x56, 0x32, 21 | 0xae, 0x1c, 0x2e, 0x48, 0x63, 0x92, 0x9a, 0x9c, 0x44, 0x85, 0x41, 0x40, 0x09, 0xc0, 0xc8, 0xbf, 22 | 0xea, 0xbb, 0xf7, 0x2d, 0x99, 0x21, 0xf6, 0xba, 0x15, 0xce, 0xab, 0xb0, 0x2a, 0x60, 0xbc, 0xf1, 23 | 0xf0, 0x9e, 0xd5, 0x97, 0xd8, 0x4e, 0x14, 0x9d, 0x42, 0x4d, 0x2c, 0x5c, 0x2b, 0xa6, 0xe1, 0xa7, 24 | 0xef, 0x25, 0x33, 0x7a, 0xeb, 0xe7, 0x1b, 0x6d, 0x4c, 0x52, 0x26, 0x62, 0xb6, 0x35, 0xbe, 0x80, 25 | 0x01, 0xbd, 0xfd, 0x37, 0xf9, 0x47, 0x55, 0x71, 0xb4, 0xf2, 0xff, 0x27, 0xfa, 0x23, 0xc9, 0x83, 26 | 0x17, 0x39, 0x13, 0x0d, 0xc7, 0x86, 0x16, 0xec, 0x49, 0x6f, 0xfe, 0x34, 0x05, 0x8f, 0x00, 0xe6, 27 | 0xa4, 0xda, 0x7b, 0xc1, 0xf3, 0xf4, 0xd9, 0x75, 0x28, 0x66, 0x87, 0xa8, 0x45, 0x6c, 0x20, 0xe9, 28 | 0x77, 0x93, 0x7e, 0x3c, 0x1e, 0x74, 0xf5, 0x8c, 0x3e, 0x94, 0xd4, 0xc2, 0x5a, 0x06, 0x0e, 0xe8, 29 | 0x3d, 0xa9, 0xb2, 0xe3, 0xe4, 0x22, 0xcf, 0x24, 0x8e, 0x6b, 0x8a, 0x8d, 0x84, 0x4a, 0xd2, 0x91, 30 | 0x88, 0x79, 0x57, 0xa5, 0x0f, 0xcd, 0xb9, 0xac, 0x3b, 0xaa, 0xb3, 0xd1, 0xee, 0x31, 0x81, 0x7c, 31 | 0xd7, 0x89, 0xd3, 0x96, 0x43, 0xc5, 0xc6, 0xc3, 0x69, 0x7f, 0x46, 0xdf, 0x30, 0x5b, 0x6e, 0xe5, 32 | 0x08, 0x95, 0x9b, 0xfb, 0xb8, 0x58, 0x0c, 0x61, 0x50, 0x5d, 0x3a, 0xa2, 0x29, 0x12, 0xfc, 0x51, 33 | 0x7d, 0x1a, 0x02, 0x65, 0x54, 0x5e, 0x19, 0xcc, 0xdc, 0xdb, 0x73, 0xed, 0xad, 0x59, 0x2f, 0xa3, 34 | ] 35 | 36 | #static const uint8_t m1_address_0_7_xor[256] = { 37 | m1_address_0_7_xor = [ 38 | 0xf4, 0xbc, 0x02, 0xf7, 0x2c, 0x3d, 0xe8, 0xd9, 0x50, 0x62, 0xec, 0xbd, 0x53, 0x73, 0x79, 0x61, 39 | 0x00, 0x34, 0xcf, 0xa2, 0x63, 0x28, 0x90, 0xaf, 0x44, 0x3b, 0xc5, 0x8d, 0x3a, 0x46, 0x07, 0x70, 40 | 0x66, 0xbe, 0xd8, 0x8b, 0xe9, 0xa0, 0x4b, 0x98, 0xdc, 0xdf, 0xe2, 0x16, 0x74, 0xf1, 0x37, 0xf5, 41 | 0xb7, 0x21, 0x81, 0x01, 0x1c, 0x1b, 0x94, 0x36, 0x09, 0xa1, 0x4a, 0x91, 0x30, 0x92, 0x9b, 0x9a, 42 | 0x29, 0xb1, 0x38, 0x4d, 0x55, 0xf2, 0x56, 0x18, 0x24, 0x47, 0x9d, 0x3f, 0x80, 0x1f, 0x22, 0xa4, 43 | 0x11, 0x54, 0x84, 0x0d, 0x25, 0x48, 0xee, 0xc6, 0x59, 0x15, 0x03, 0x7a, 0xfd, 0x6c, 0xc3, 0x33, 44 | 0x5b, 0xc4, 0x7b, 0x5a, 0x05, 0x7f, 0xa6, 0x40, 0xa9, 0x5d, 0x41, 0x8a, 0x96, 0x52, 0xd3, 0xf0, 45 | 0xab, 0x72, 0x10, 0x88, 0x6f, 0x95, 0x7c, 0xa8, 0xcd, 0x9c, 0x5f, 0x32, 0xae, 0x85, 0x39, 0xac, 46 | 0xe5, 0xd7, 0xfb, 0xd4, 0x08, 0x23, 0x19, 0x65, 0x6b, 0xa7, 0x93, 0xbb, 0x2b, 0xbf, 0xb8, 0x35, 47 | 0xd0, 0x06, 0x26, 0x68, 0x3e, 0xdd, 0xb9, 0x69, 0x2a, 0xb2, 0xde, 0x87, 0x45, 0x58, 0xff, 0x3c, 48 | 0x9e, 0x7d, 0xda, 0xed, 0x49, 0x8c, 0x14, 0x8e, 0x75, 0x2f, 0xe0, 0x6e, 0x78, 0x6d, 0x20, 0xd2, 49 | 0xfa, 0x2d, 0x51, 0xcc, 0xc7, 0xe7, 0x1d, 0x27, 0x97, 0xfc, 0x31, 0xdb, 0xf8, 0x42, 0xe3, 0x99, 50 | 0x5e, 0x83, 0x0e, 0xb4, 0x2e, 0xf6, 0xc0, 0x0c, 0x4c, 0x57, 0xb6, 0x64, 0x0a, 0x17, 0xa3, 0xc1, 51 | 0x77, 0x12, 0xfe, 0xe6, 0x8f, 0x13, 0x71, 0xe4, 0xf9, 0xad, 0x9f, 0xce, 0xd5, 0x89, 0x7e, 0x0f, 52 | 0xc2, 0x86, 0xf3, 0x67, 0xba, 0x60, 0x43, 0xc9, 0x04, 0xb3, 0xb0, 0x1e, 0xb5, 0xc8, 0xeb, 0xa5, 53 | 0x76, 0xea, 0x5c, 0x82, 0x1a, 0x4f, 0xaa, 0xca, 0xe1, 0x0b, 0x4e, 0xcb, 0x6a, 0xef, 0xd1, 0xd6, 54 | ] 55 | 56 | 57 | 58 | # The CMC50 hardware does a checksum of the first 64kb of the M1 rom, and 59 | # uses this checksum as the basis of the key with which to decrypt the rom */ 60 | #uint16_t cmc_prot_device::generate_cs16(uint8_t *rom, int size) 61 | def generate_cs16(rom, size): 62 | 63 | #uint16_t cs16 = 0x0000; 64 | cs16 = 0x0000 65 | 66 | #for (int i = 0; i < size; i++) 67 | for i in range(0, size): 68 | cs16 += rom[i] 69 | 70 | return cs16 & 0xffff 71 | 72 | 73 | 74 | #int cmc_prot_device::m1_address_scramble(int address, uint16_t key) 75 | def m1_address_scramble(address, key): 76 | #const int p1[8][16] = { 77 | p1 = [ 78 | [15,14,10,7,1,2,3,8,0,12,11,13,6,9,5,4], 79 | [7,1,8,11,15,9,2,3,5,13,4,14,10,0,6,12], 80 | [8,6,14,3,10,7,15,1,4,0,2,5,13,11,12,9], 81 | [2,8,15,9,3,4,11,7,13,6,0,10,1,12,14,5], 82 | [1,13,6,15,14,3,8,10,9,4,7,12,5,2,0,11], 83 | [11,15,3,4,7,0,9,2,6,14,12,1,8,5,10,13], 84 | [10,5,13,8,6,15,1,14,11,9,3,0,12,7,4,2], 85 | [9,3,7,0,2,12,4,11,14,10,5,8,15,13,1,6], 86 | ] 87 | 88 | #int block = (address >> 16) & 7; 89 | block = (address >> 16) & 7 90 | 91 | #int aux = address & 0xffff; 92 | aux = address & 0xffff 93 | 94 | #aux ^= bitswap<16>(key,12,0,2,4,8,15,7,13,10,1,3,6,11,9,14,5); 95 | aux ^= arcade_utilities.bitswap(key,[12,0,2,4,8,15,7,13,10,1,3,6,11,9,14,5]) 96 | 97 | #aux = bitswap<16>(aux, 98 | # p1[block][15], p1[block][14], p1[block][13], p1[block][12], 99 | # p1[block][11], p1[block][10], p1[block][9], p1[block][8], 100 | # p1[block][7], p1[block][6], p1[block][5], p1[block][4], 101 | # p1[block][3], p1[block][2], p1[block][1], p1[block][0]); 102 | aux = arcade_utilities.bitswap(aux, 103 | [p1[block][15], p1[block][14], p1[block][13], p1[block][12], 104 | p1[block][11], p1[block][10], p1[block][9], p1[block][8], 105 | p1[block][7], p1[block][6], p1[block][5], p1[block][4], 106 | p1[block][3], p1[block][2], p1[block][1], p1[block][0]]) 107 | 108 | #aux ^= m1_address_0_7_xor[(aux >> 8) & 0xff]; 109 | aux ^= m1_address_0_7_xor[(aux >> 8) & 0xff] 110 | 111 | #aux ^= m1_address_8_15_xor[aux & 0xff] << 8; 112 | aux ^= m1_address_8_15_xor[aux & 0xff] << 8 113 | 114 | #aux = bitswap<16>(aux, 7,15,14,6,5,13,12,4,11,3,10,2,9,1,8,0); 115 | aux = arcade_utilities.bitswap(aux, [7,15,14,6,5,13,12,4,11,3,10,2,9,1,8,0]) 116 | 117 | return (block << 16) | aux 118 | 119 | 120 | 121 | #void cmc_prot_device::cmc50_m1_decrypt(uint8_t* romcrypt, uint32_t romcrypt_size, uint8_t* romaudio, uint32_t romaudio_size) { 122 | def cmc50_m1_decrypt_or_encrypt(input_rom, encrypt): 123 | 124 | entire_m1_size = 0x20000 125 | used_m1_size = 0x10000 126 | empty_m1_size = 0x10000 127 | 128 | assert len(input_rom) == entire_m1_size 129 | for i in range(used_m1_size, entire_m1_size): 130 | assert input_rom[i] == 0xFF 131 | 132 | #uint8_t* rom = romcrypt; 133 | #size_t rom_size = 0x80000; 134 | #uint8_t* rom2 = romaudio; 135 | 136 | #std::vector buffer(rom_size); 137 | output_data = bytearray(used_m1_size) 138 | 139 | #uint16_t key = generate_cs16(rom, 0x10000); 140 | key = generate_cs16(input_rom, used_m1_size) 141 | 142 | #for (uint32_t i = 0; i < rom_size; i++) 143 | for i in range(0, used_m1_size): 144 | #buffer[i] = rom[m1_address_scramble(i,key)]; 145 | if encrypt: 146 | output_data[m1_address_scramble(i, key)] = input_rom[i] 147 | else: 148 | output_data[i] = input_rom[m1_address_scramble(i, key)] 149 | 150 | #memcpy(rom, &buffer[0], rom_size); 151 | #memcpy(rom2,rom, 0x10000); 152 | #memcpy(rom2 + 0x10000, rom, 0x80000); 153 | #rom2[0x10000:0x90000] = rom[0:0x80000] 154 | return output_data[0:used_m1_size] + b'\xFF' * empty_m1_size 155 | 156 | 157 | 158 | def encrypt_cmc50_m1(unencrypted_m1rom): 159 | print("Encrypting M1 ROM with original CMC50 encryption...") 160 | return cmc50_m1_decrypt_or_encrypt(unencrypted_m1rom, True) 161 | 162 | def decrypt_cmc50_m1(encrypted_m1rom): 163 | print("Decrypting M1 ROM with CMC50 encryption...") 164 | return cmc50_m1_decrypt_or_encrypt(encrypted_m1rom, False) 165 | -------------------------------------------------------------------------------- /neogeo_decrypt.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | 3 | #from neogeo_keys import keys 4 | import os, hashlib 5 | from u8archive import U8Archive 6 | 7 | try: 8 | from Crypto.Cipher import AES 9 | except: 10 | pass 11 | 12 | 13 | 14 | def tryGetU8Archive(path): 15 | try: 16 | u8arc = U8Archive(path) 17 | if not u8arc: 18 | return None 19 | else: 20 | return u8arc 21 | except AssertionError: 22 | return None 23 | 24 | 25 | def get_banner(folder): 26 | # find the banner.bin file, which may be in any of the U8Archives in the folder 27 | for app in os.listdir(folder): 28 | app_path = os.path.join(folder, app) 29 | u8arc = tryGetU8Archive(app_path) 30 | if u8arc: 31 | for file in u8arc.files: 32 | if file.name == 'banner.bin': 33 | file_as_bytesIO = u8arc.getfile(file.path) 34 | file_as_bytearray = bytearray(file_as_bytesIO.read()) 35 | file_as_bytesIO.close() 36 | return file_as_bytearray 37 | 38 | raise ValueError 39 | 40 | def xor_bytearray(ba1, ba2): 41 | assert len(ba1) == len(ba2) 42 | out = bytearray(len(ba1)) 43 | for i in range(0, len(out)): 44 | out[i] = ba1[i] ^ ba2[i] 45 | return out 46 | 47 | 48 | 49 | def scramble_16_bytes(input_data): 50 | # 8019a5c4--8019a630 51 | data_1 = bytearray(16) 52 | for i in range(0,16): 53 | data_1[i] = input_data[i] ^ 0xFF 54 | 55 | # 8019a644--8019a6b0 56 | data_2 = bytearray(16) 57 | work_byte = 0xFF 58 | for i in range(0,16): 59 | work_byte ^= data_1[i] 60 | data_2[i] = work_byte 61 | 62 | # 8019a6c0--8019a764 63 | data_3 = bytearray(16) 64 | for i in range(0,16): 65 | work_byte = data_2[i] 66 | for j in [0,3,3]: 67 | shift_amount = (work_byte >> j) & 7 68 | work_byte = ((work_byte << (8-shift_amount)) | (work_byte >> shift_amount)) & 0xFF 69 | data_3[i] = work_byte 70 | 71 | # 8019a77c--8019a834 72 | data_4 = bytearray(16) 73 | carry_over_to_next_byte = 0 74 | for i in range(0,16): 75 | shift_amount = (data_3[i] >> (carry_over_to_next_byte & 7)) & 7 76 | shifted = carry_over_to_next_byte | ((data_3[i] << shift_amount) & 0xFF) 77 | data_4[i] = shifted 78 | carry_over_to_next_byte = shifted >> (8-shift_amount) 79 | 80 | # 8019a844--8019a8d0 81 | data_5 = bytearray(16) 82 | last_input_byte = 0 83 | for i in range(0,16): 84 | data_5[i] = (last_input_byte + data_4[i]) & 0xFF 85 | last_input_byte = data_4[i] 86 | 87 | return data_5 88 | 89 | 90 | def get_aes_key(folder, cr00_key): 91 | return scramble_16_bytes( 92 | hashlib.md5( 93 | xor_bytearray( 94 | hashlib.md5( 95 | get_banner(folder) 96 | ).digest(), 97 | cr00_key 98 | ), 99 | ).digest() 100 | ) 101 | 102 | 103 | # titleIdString must be the 8 byte string identifying each Wii title, e.g. '421a2b3c' 104 | # fileString input must be a string containing the content of the encrypted romfile, starting with CR00. 105 | # will return a tuple - either (true, decryptedData) or (false, encryptedData) 106 | # note that the decryptedData will still be compressed (zipped), similar to some other neo geo games without encryption. 107 | def decrypt_neogeo(sourceFolderPath, titleIdString, fileString): 108 | # encrypted files starts with a magic word 109 | assert fileString[0:4] == b'CR00' 110 | 111 | # the next 16 bytes are used as input for key generation. 112 | key = get_aes_key(sourceFolderPath, bytearray(fileString[4:0x14])) 113 | 114 | # the rest of the files is AES-CBC-128 encrypted. as such it consists of 16 byte (128bit) blocks. 115 | assert (len(fileString) - 0x14) % 0x10 == 0 116 | 117 | encryptedString = fileString[0x14:] 118 | assert len(fileString) == 0x14+len(encryptedString) 119 | 120 | #if (titleIdString in keys): 121 | # if key == unhexlify(keys[titleIdString]): 122 | # print("FOUND MATCHING KEY!!!!!!!!!! SUCCESS",key, keys[titleIdString]) 123 | # else: 124 | # print("NOT MATCHING! :(",key, keys[titleIdString]) 125 | #else: 126 | # print("key not found in good list, assuming it's OK") 127 | 128 | # The IV is just zeroes. 129 | # If the IV is wrong, only the first block (16 bytes) will be decrypted incorrectly, 130 | # but that's bad enough for the decompression to fail 131 | zeroIv = b'\x00'*16 132 | 133 | assert len(key) == 16 134 | 135 | try: 136 | cipher = AES.new(key, AES.MODE_CBC, iv=zeroIv) 137 | decryptedString = cipher.decrypt(encryptedString) 138 | except: 139 | print("Got the key, but AES decryption failed. Make sure PyCryptodome or PyCrypto is installed.") 140 | return (False, encryptedString) 141 | 142 | assert len(decryptedString) == len(encryptedString) 143 | 144 | print("Found the AES key and decrypted the ROMs") 145 | 146 | return (True, decryptedString) 147 | -------------------------------------------------------------------------------- /neogeo_readme.txt: -------------------------------------------------------------------------------- 1 | UPDATE: This is not really needed anymore, since the scripts now automatically can regenerate the AES key. 2 | Leaving it since it MAY be useful if some games uses a different method to generate the key. 3 | 4 | 5 | 6 | --- 7 | 8 | The following are instructions that are necessary to be able to extract encrypted Neo Geo games. 9 | Roughly half of the Neo Geo virtual console games are encrypted. 10 | Vcromclaim will tell you if a game is encrypted, and refer to this file. 11 | 12 | Note that "encryption" in this file refers to the Wii specific encrption. 13 | If does NOT refer to the original encryption that some games originally had, like Metal Slug 3 and Metal Slug 4. 14 | 15 | The steps are, roughly: 16 | 1. Run ShowMiiWads to pack the game as a Wad 17 | 2. Run the Wad in Dolphin 18 | 3. Use Dolphin's debugging tools to get the key from the emulated RAM (it is different for each game) 19 | 4. Add the key to neogeo_key.py 20 | 5. Install PyCryptodome 21 | 6. Run vcromclaim as for any other games. 22 | 23 | These instructions are not very user friendly - FEEL FREE TO IMPROVE THEM AND SUBMIT PR! 24 | 25 | 26 | See further below for technical details about the encryption. 27 | 28 | 29 | DETAILED INSTRUCTIONS 30 | 31 | 32 | 33 | Install and run ShowMiiWads- https://wiibrew.org/wiki/ShowMiiWads 34 | Options --> Change NAND backup path --> Select the location of your Wii NAND files (the same as you specify as source for vcromclaim) 35 | You should see all of the games in the NAND backup 36 | Find the game you need the encryption key for 37 | In the left most column, there is 'xxxxxxxx.tik'. Add the 8 characters xxxxxxxx to a new line in neogeo_keys.py. 38 | Right click and select Pack Wad. Save this file somewhere convenient. 39 | 40 | 41 | 42 | Install and run Dolphin - http://www.dolphin-emulator.com/ 43 | Make sure the views "Code", "Registers" and "Memory" are open. 44 | Load the wad file (drag & drop or File -> Open) for the wad you packed. 45 | When the game says "Now loading...", hit the pause button in Dolphin. 46 | (Note: the timing kind of matters. You may have to start over from here trying hitting pause at various times). 47 | 48 | 49 | In the Memory view, there are two text boxes at the top, the second one is "Value". 50 | In this box, type: 812b0000 51 | Make sure "Hex" is selected in the radio button. 52 | Make sure "U32" is selected as data type. 53 | Click Find next and/or Find previous. There should be exactly two matches: One on a 80xxxxxx address and one on a c0xxxxxx address. 54 | Click Find next and/or Find previous so that the 80xxxxxx match is highlighted. 55 | Right click on the highlighted address 80xxxxxx (the leftmost column) and select copy address. 56 | In Code, in the textbox Search address, paste the address you copied and hit Enter. 57 | The highlighted line in the Code view should be: 58 | 59 | 80xxxxxx lwz r9, 0 (r11) 60 | followed by: 61 | 80xxxxyy rlwinm r12, r10, 16, 0, 15 (0000ffff) 62 | 63 | This is one of the instructions that read part of the encryption key from RAM during decryption. 64 | 65 | Right click the "80xxxxxx lwz" line and pick "Run to here". 66 | Click Play to resume execution. 67 | It will pause again once the execution is at this line which will be very quick). 68 | If the game starts (beyond Now loading), it's too late, the decryption has already been done, if so, stop emulation completely and start over Try hitting pause earlier. 69 | 70 | Now, look at the Registers view. 71 | Look for r11. This is the CPU register that holds the address to RAM where the key is stored. 72 | There will be *another* 80xxxxxx address. Double click it, select it and copy it. 73 | (Note - you might get the wrong key, this code is run with a few different keys, only one of the keys is correct. 74 | You may want to experiment pausing emulation earlier and repeatedly pressing play until the value in r11 changes.) 75 | 76 | Back to the Memory view. In the top text box (Search Address), paste the last address. 77 | The address will be highlighted. 78 | The 4 blocks of characters next to the address is the encryption key! 79 | 80 | Open the neogeo_keys.py file and add the encryption key (whithout spaces) to it. 81 | 82 | NOTE: that the memory view will move around if you try to click on the data, 83 | so the easiest way to copy the entire value is to manually copy it. 84 | 85 | NOTE: Since neogeo_keys.py is Python, be careful to use the correct whitespace on each line. 86 | 87 | Finally, install PyCryptodome by typing in terminal: 88 | pip install pycryptodome 89 | Read more about it here if necessary: https://pypi.org/project/pycryptodome/ 90 | PyCrypto should also work. 91 | 92 | NOTE: If game extraction crash with an unclear error message, you probably got the wrong key. Make sure you copied the key correctly or look for another key as described above. 93 | 94 | 95 | TECHNICAL DETAILS ABOUT THE KEYS AND NEO GEO VIRTUAL CONSOLE ENCRYPTION 96 | 97 | 98 | The encrypted games are packed similarly to non-encrypted games: 99 | There is one DOL file (the emulator), and one U8 archive. 100 | The U8 archive contains several files, one of them contains all of the ROMs, the other are config files etc. 101 | The ROMs file can either be plain, compressed or compressed and then encrypted. 102 | For the encrypted files, the first 20 bytes is a header, the rest is the encrypted data. 103 | It is encrypted using 128-bit (16 byte blocks) AES in CBC mode. 104 | The IV is just zeroes. 105 | 106 | The key is the problem... It is constructed using the following: (UPDATED) 107 | - Bytes 4-19 (0-based) of the encrypted file (unique for every game) 108 | - The entire content of "banner.bin" within one of the game's U8 archives (unique for each game) 109 | - A 448 block of "random" data in the DOL file (probably same for every game) 110 | - 16 bytes of data constructed by some code (probably same for every game) 111 | - A few other bytes of random data (probably the same for every game) 112 | 113 | Between each merging of the above data, A TON of bitshifting, byteswapping, xoring etc etc is done. 114 | 115 | So it's definitly POSSIBLE to automatically rebuild the encryption keys from a NAND backup. 116 | However, I can't justify spending the time reverse-engineering this. UPDATE: I did... :) 117 | -------------------------------------------------------------------------------- /neogeo_sma.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | 3 | 4 | import arcade_utilities 5 | 6 | # decryption converted from: https://github.com/mamedev/mame/blob/519cd8b99af8264e4117e3061b0e2391902cfc02/src/devices/bus/neogeo/prot_sma.cpp 7 | # added reversed code for encryption 8 | 9 | #void sma_prot_device::mslug3_decrypt_68k(uint8_t* base){ 10 | def mslug3_decrypt_68k(encrypted_rom_including_green): 11 | 12 | #MAP OF INPUT: 13 | # 0 - c0000 = EMPTY 14 | # c0000 - 100000 = "green" SMA ROM 15 | # 100000 - 900000 = encrypted P ROMs 16 | # this is how the roms are initially loaded in mame 17 | 18 | #MAP OF OUTPUT: 19 | # 0 - 900000 = the unencrypted code, in a way that can be executed. 20 | 21 | print("Decrypting P ROM with SMA encryption...") 22 | 23 | #uint16_t *rom = (uint16_t *)(base + 0x100000); 24 | output_everything = bytearray(encrypted_rom_including_green) 25 | rom = output_everything[0x100000:] 26 | 27 | # swap data lines on the whole ROMs 28 | #for (int i = 0; i < 0x800000/2; i++) 29 | for i in range(0, int(0x800000), 2): 30 | #rom[i] = bitswap<16>(rom[i],4,11,14,3,1,13,0,7,2,8,12,15,10,9,5,6); 31 | two_bytes_int = int.from_bytes(rom[i : i+2],'little') 32 | two_bytes_int = arcade_utilities.bitswap(two_bytes_int,[4,11,14,3,1,13,0,7,2,8,12,15,10,9,5,6]) 33 | rom[i : i+2] = two_bytes_int.to_bytes(2, 'little') 34 | 35 | output_everything[0x100000:0x100000+len(rom)] = rom # 0x800000 * b'\xFF 36 | 37 | 38 | 39 | # swap address lines & relocate fixed part 40 | #rom = (uint16_t *)base; 41 | rom = output_everything 42 | 43 | #for (int i = 0; i < 0x0c0000/2; i++) 44 | for i in range (0, int(0x0c0000/2)): 45 | #rom[i] = rom[0x5d0000/2 + bitswap<19>(i,18,15,2,1,13,3,0,9,6,16,4,11,5,7,12,17,14,10,8)]; 46 | swapped_address = int(0x5d0000) + arcade_utilities.bitswap(i,[18,15,2,1,13,3,0,9,6,16,4,11,5,7,12,17,14,10,8]) * 2 47 | rom[i*2+0] = rom[swapped_address + 0] 48 | rom[i*2+1] = rom[swapped_address + 1] 49 | 50 | # swap address lines for the banked part 51 | #rom = (uint16_t *)(base + 0x100000); 52 | output_everything = rom 53 | rom = output_everything[0x100000 :] 54 | 55 | #for (int i = 0; i < 0x800000/2; i += 0x10000/2) { 56 | for i in range(0, int(0x800000), int(0x10000)): 57 | 58 | #uint16_t buffer[0x10000/2]; 59 | #memcpy(buffer, &rom[i], 0x10000); 60 | buffer = rom[i : i + 0x10000] 61 | 62 | #for (int j = 0; j < 0x10000/2; j++) 63 | for j in range(0, int(0x10000/2)): 64 | #rom[i+j] = buffer[bitswap<15>(j,2,11,0,14,6,4,13,8,9,3,10,7,5,12,1)]; 65 | swapped_address = arcade_utilities.bitswap(j,[2,11,0,14,6,4,13,8,9,3,10,7,5,12,1]) * 2 66 | rom[i + j*2 : i + j*2 + 2] = buffer[swapped_address : swapped_address + 2] 67 | #} 68 | 69 | output_everything[0x100000 :] = rom 70 | return output_everything 71 | #} 72 | 73 | 74 | #void sma_prot_device::mslug3_decrypt_68k(uint8_t* base){ 75 | def mslug3_encrypt_68k(decrypted_rom_including_green): 76 | 77 | #MAP OF INPUT: 78 | # 0 - 900000 = the unencrypted code, in a way that can be executed. this is the output of mslug3_decrypt_68k and also how it was delivered as a P ROM on Wii. 79 | 80 | #MAP OF OUTPUT: 81 | # 0 - c0000 = UNDEFINED - should be same as input, which means it is decrypted garbage and should be ignored. 82 | # c0000 - 100000 = "green" SMA ROM, untouched, same input as output 83 | # 100000 - 900000 = encrypted actual P ROMs 84 | # this is how the roms are initially loaded in mame 85 | 86 | print("Encrypting P ROM with SMA encryption...") 87 | 88 | output_everything = bytearray(decrypted_rom_including_green) 89 | 90 | rom = output_everything[0x100000:] 91 | 92 | for i in range(0, int(0x800000), int(0x10000)): 93 | buffer = rom[i : i + 0x10000] 94 | 95 | for j in range(0, int(0x10000/2)): 96 | swapped_address = arcade_utilities.bitswap(j,[2,11,0,14,6,4,13,8,9,3,10,7,5,12,1]) * 2 97 | rom[i + swapped_address : i + swapped_address + 2] = buffer[j*2 : j*2 + 2] 98 | 99 | output_everything[0x100000:] = rom 100 | 101 | # swap address lines & relocate fixed part 102 | rom = output_everything 103 | 104 | for i in range (0, int(0x0c0000/2)): 105 | swapped_address = int(0x5d0000) + arcade_utilities.bitswap(i,[18,15,2,1,13,3,0,9,6,16,4,11,5,7,12,17,14,10,8])*2 106 | rom[swapped_address : swapped_address + 1] = rom[i*2 : i*2 + 1] 107 | 108 | output_everything = rom 109 | rom = output_everything[0x100000 :] 110 | 111 | 112 | # swap data lines on the whole ROMs 113 | for i in range(0, int(0x800000), 2): 114 | two_bytes_int = int.from_bytes(rom[i : i+2],'little') 115 | two_bytes_int = arcade_utilities.bitswap(two_bytes_int,[4,13,10,5,14,3,2,6,8,0,1,15,12,7,11,9]) 116 | rom[i : i+2] = two_bytes_int.to_bytes(2, 'little') 117 | 118 | output_everything[0x100000:] = rom 119 | return output_everything 120 | #} 121 | 122 | -------------------------------------------------------------------------------- /neogeo_smc.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | 3 | #Metal Slug 4 has some weird byteswapping not seen elsewhere, maybe related to SMC encryption or something. 4 | 5 | 6 | def swap_8_byte_pairs(input_rom): 7 | value = 8 8 | 9 | output_rom = input_rom 10 | 11 | size = len(input_rom) 12 | 13 | buffer = bytearray(value) 14 | 15 | for h in range(0, int(size), int(value)): 16 | 17 | buffer[0:value] = output_rom[h:h+value] 18 | 19 | for i in range(0, int(value), int(value/2)): 20 | 21 | swapInputOffset = i ^ int(value/2) 22 | 23 | for j in range(0, int(value/2)): 24 | output_rom[h + i + j] = buffer[swapInputOffset + j] 25 | 26 | return output_rom 27 | -------------------------------------------------------------------------------- /nes_extract.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | # Author: Bryan Cain (Plombo) 3 | # Extracts an NES ROM or FDS image from a 00000001.app file from an NES Virtual Console game. 4 | 5 | import sys, struct, hashlib 6 | import copy 7 | from array import array 8 | from io import BytesIO 9 | import lz77 10 | 11 | 12 | NES_HEADER_MAGIC_WORD = b'NES\x1a' 13 | 14 | #VCI file format used by Virtual console 15 | VCI_DISK_SIDE_LENGTH = 0x10000 16 | 17 | #.fds file format used by common emulators 18 | #OUTPUT FORMAT: (fds format as described here: https://wiki.nesdev.com/w/index.php/Family_Computer_Disk_System) 19 | #Byte 00-03 = FDS\x1a 20 | FDS_MAGIC_WORD = b'FDS\x1A' 21 | FDS_MAGIC_WORD_LENGTH = len(FDS_MAGIC_WORD) 22 | #Byte 04 = number of sides (0x01, 0x02) 23 | FDS_SIDE_COUNT_POSITION = 0x04 24 | FDS_SIDE_SIZE = 0x01 25 | #Byte 05-0F = 0x00 26 | FDS_HEADER_LENGTH = 0x10 27 | #DISK SIDE BLOCK (one for each side in byte 04) 28 | #0000-FFDC = image of one disk side, should start with \x00*NINTENDO-HVC. Disk gaps between files excluded. The end of the image is padded with zeroes. 29 | FDS_DISK_SIDE_LENGTH = 0x0FFDC 30 | 31 | #original data on disks/ROMs 32 | FDS_SIDE_HEADER_MAGIC_WORD = b'\x01*NINTENDO-HVC*' 33 | FDS_BIOS_HEADER_MAGIC_WORD = b'\x00\x38\x4C\xC6\xC6\xC6\x64\x38\x00\x18\x38\x18\x18\x18\x18\x7E' 34 | 35 | 36 | def extract_fds_bios_from_app(app1, tryLZ77 = True): 37 | fdsBiosOffset = find_fds_bios_in_app(app1) 38 | if fdsBiosOffset >= 0: 39 | app1.seek(fdsBiosOffset) 40 | fileData = app1.read(0x2000) # = 8 KiB 41 | fileHash = hashlib.md5(fileData) 42 | print("Found FDS BIOS ROM with hash: " + fileHash.hexdigest()) 43 | 44 | return BytesIO(fileData) 45 | 46 | if tryLZ77: 47 | try: 48 | return extract_fds_bios_from_app(BytesIO(lz77.decompress_lz77_11(app1, 4, 5 * 1024 * 1024)), False) 49 | except IndexError: 50 | return None 51 | 52 | else: 53 | return None 54 | 55 | 56 | # return: 57 | # result = 0 (neither cartridge or FDS), 1 (cartridge), or 2 (FDS) 58 | # output = the rom/disk image data as BytesIO with iNES or FDS header 59 | def extract_nes_file_from_app(app1, tryLZ77 = True): 60 | 61 | cartridgeRomOffset = find_cartridge_header_in_app(app1) 62 | if cartridgeRomOffset >= 0: 63 | return (1, get_cartridge_in_app(app1, cartridgeRomOffset)) 64 | 65 | fdsImageOffset = find_fds_header_in_app(app1) 66 | if fdsImageOffset >= 0: 67 | return (2, BytesIO(get_vci_image_as_fds_image(get_vci_body_in_app(app1, fdsImageOffset)))) 68 | 69 | # some app files are compressed. decompress the entire file. 70 | # it seems the ROMs are decompressed properly even if we do not start the decompression at the ROM's start position. 71 | if tryLZ77: 72 | #f = open('compressed-NES', 'wb') 73 | #app1.seek(0) 74 | #f.write(app1.read()) 75 | #app1.seek(0) 76 | #f.close() 77 | 78 | try: 79 | #try to autodetect format 80 | (result, output) = extract_nes_file_from_app(BytesIO(lz77.decompress_nonN64(app1)), False) 81 | return (result, output) 82 | except ValueError: 83 | try: 84 | #try brutally decompressing the entire file 85 | (result, output) = extract_nes_file_from_app(BytesIO(lz77.decompress_lz77_11(app1, 4, 5 * 1024 * 1024)), False) 86 | return (result, output) 87 | except IndexError: 88 | return (0,None) 89 | 90 | else: 91 | return (0,None) 92 | 93 | 94 | def find_cartridge_header_in_app(inputFile): 95 | # The NES header is: (source = http://wiki.nesdev.com/w/index.php/INES ) 96 | # Byte 0-4 = 'NES\x1a' 97 | # Byte 5-A = header data 98 | # Byte B-F = 0x00 99 | 100 | position = 0 101 | while True: 102 | inputFile.seek(position) 103 | position = inputFile.read().find(NES_HEADER_MAGIC_WORD) 104 | if position < 0: 105 | #no header found. return -1. 106 | return position 107 | else: 108 | #check if header byte B-F is zeroes, to skip some false positives 109 | inputFile.seek(position + 0xB) 110 | if inputFile.read(5) == b'\x00\x00\x00\x00\x00': 111 | # most likely a NES header 112 | return position 113 | else: 114 | # not a NES header - keep searching 115 | position += 1 116 | 117 | 118 | def find_fds_header_in_app(inputFile): 119 | inputFile.seek(0) 120 | return inputFile.read().find(FDS_SIDE_HEADER_MAGIC_WORD) 121 | 122 | def get_cartridge_in_app(inputFile, start): 123 | # NES ROM found; calculate size and extract it 124 | inputFile.seek(start+4) 125 | # This assumes the ROMs doesn't have any of the optional stuff (trainer, file name, etc) 126 | 127 | size = 16 # 16-byte header, 128-byte title data (optional footer) 128 | size += 16 * 1024 * struct.unpack(">B", inputFile.read(1))[0] # next byte: number of PRG banks, 16KB each 129 | size += 8 * 1024 * struct.unpack(">B", inputFile.read(1))[0] # next byte: number of CHR banks, 8KB each 130 | inputFile.seek(start) 131 | return BytesIO(inputFile.read(size)) 132 | 133 | # Returns the body as an array - all sides of the game, but with no header, in the Wii format (VCI) 134 | def get_vci_body_in_app(inputFile, start): 135 | #NES VC FDS images are prefixed with VCI header. 136 | #However, this header is also used on some ROMs, and the header does not indicate whether the data is a cart rom or fds image. 137 | #So, just ignore the VCI header and look for the headers from the disk images... 138 | 139 | #For reference, the ignored VCI header is as follows: 140 | #Byte 00-03 = VCI\x1a (Virtual Console Image?) 141 | #Byte 04-08 = ?????? (often 0x00 but sometimes different values) 142 | #Byte 09 = number of disk sides (0x01, 0x02) but is e.g. 0x04 for NES Open Tournament Golf (cart game) 143 | #Byte 0A-2F = ?????? (often 0x00 but sometimes different values) 144 | #DISK SIDE BLOCK (repeated for each disk side) 145 | #0000-FFFF = 146 | # Image of one disk side, should start with \x01*NINTENDO-HVC. 147 | # Disk gaps between files excluded (just like in FDS format). 148 | # The end of the disk is just zeroes, just like with FDS format, however, the VC format has longer padding. 149 | 150 | outputArray = array('B', []) 151 | sideCounter = 0 152 | while True: 153 | vciSideStart = start + sideCounter*VCI_DISK_SIDE_LENGTH 154 | 155 | inputFile.seek(vciSideStart) 156 | if inputFile.read(len(FDS_SIDE_HEADER_MAGIC_WORD)) == FDS_SIDE_HEADER_MAGIC_WORD: 157 | inputFile.seek(vciSideStart) 158 | outputArray.extend(array('B', inputFile.read(VCI_DISK_SIDE_LENGTH))) 159 | sideCounter += 1 160 | else: 161 | break 162 | 163 | return outputArray 164 | 165 | 166 | def get_vci_image_as_fds_image(vciImage): 167 | #convert the VCI image to the FDS file format. 168 | 169 | numberOfSides = int(len(vciImage) / VCI_DISK_SIDE_LENGTH) 170 | 171 | outputArray = array('B', FDS_MAGIC_WORD) 172 | outputArray.extend([numberOfSides]) 173 | outputArray.extend(array('B', b'\0' * (FDS_HEADER_LENGTH - FDS_MAGIC_WORD_LENGTH - FDS_SIDE_SIZE)) ) 174 | 175 | for sideIndex in range(0, numberOfSides): 176 | outputArray.extend(get_vci_side_as_fds_side(vciImage, sideIndex*VCI_DISK_SIDE_LENGTH)) 177 | 178 | return outputArray 179 | 180 | 181 | def get_vci_side_as_fds_side(vciImage, vciSideStart): 182 | #Within the disk sides, there is one difference between the VC format and the FDS format: 183 | #The VC format includes a 2 byte checksum (or at least padding where the checksum used to be) after each block 184 | #We need to parse the side, block by block, and skip the checksums. 185 | 186 | #checksum is NOT included in these lengths. 187 | BLOCK_1_LENGTH = 0x38 #Side header 188 | BLOCK_1_HEADER = b'\x01' 189 | 190 | BLOCK_2_LENGTH = 0x02 #Number of files on the side - though some disks lie to improve load times. have to scan the entire side! 191 | BLOCK_2_HEADER = b'\x02' 192 | 193 | BLOCK_3_LENGTH = 0x10 #File header 194 | BLOCK_3_HEADER = b'\x03' 195 | BLOCK_3_SIZE_OF_BLOCK_4_POSITION = 0xD # Position of little endian 2 byte value that defines the size of the block 4 data (excluding header and checksum) 196 | BLOCK_3_SIZE_OF_BLOCK_4_SIZE = 0x2 197 | 198 | BLOCK_4_HEADER = b'\x04' 199 | BLOCK_4_HEADER_LENGTH = 0x1 200 | #Blocks are 1,2,[(3,4),(3,4),(3,4),...]. after the last file is padded with 0 till the end of the side. 201 | 202 | CHECKSUM_LENGTH = 0x02 203 | 204 | inputPosition = vciSideStart 205 | side = get_fds_block_from_vci_image(vciImage, inputPosition, BLOCK_1_LENGTH, BLOCK_1_HEADER) 206 | inputPosition += BLOCK_1_LENGTH + CHECKSUM_LENGTH 207 | 208 | side.extend(get_fds_block_from_vci_image(vciImage, inputPosition, BLOCK_2_LENGTH, BLOCK_2_HEADER)) 209 | inputPosition += BLOCK_2_LENGTH + CHECKSUM_LENGTH 210 | 211 | while is_fds_block_in_vci_image(vciImage, inputPosition, BLOCK_3_HEADER): 212 | block3 = get_fds_block_from_vci_image(vciImage, inputPosition, BLOCK_3_LENGTH, BLOCK_3_HEADER) 213 | block4length = BLOCK_4_HEADER_LENGTH + struct.unpack(' 0): 263 | # The actual data payload is the rest of the file. This is the same format as FCEUX etc use. 264 | # Perhaps the file is padded just like FDS saves are, but the current solutions seems to be working OK. 265 | infile = open(originalFilePath, 'rb') 266 | outfile = open(outputPathWithoutExtension + '.sav', 'wb') 267 | infile.seek(64) 268 | outfile.write(infile.read()) 269 | outfile.close() 270 | infile.close() 271 | return True 272 | else: #fds 273 | # The body is a format very similar to a IPS patch: 274 | # 3 bytes of offset (I), referring to a position in the VCI image file (excluding VCI header). 275 | # 2 bytes of data length (N) 276 | # The N bytes to apply at offset (I). 277 | # (Repeated) 278 | # 0xFF indicates end of file. 279 | # padded with 0x00 up to 128KB. (so total file size is 128KB + 64 byte for the header) 280 | # the above should be the same as the payload size in the header. 281 | 282 | # NOTE: additionally, the savefile can simply be 128KB + 64 bytes but have nothing but 0s, if nothing has been saved in the game. 283 | 284 | #Since VCI is a format that is not used by common emulators, we are not exporting the disk images in that format. 285 | #We could either export an IPS file with the offsets adjusted to the converted FDS file, or we can just apply this savefile on the 286 | # VCI file and convert the patched VCI file to .FDS format as a savefile. 287 | 288 | infile = open(originalFilePath, 'rb') 289 | 290 | firstFourBytes = infile.read(4) 291 | if firstFourBytes == b'\x00\x00\x00\x00': 292 | print('Save file was found, but it was empty. This is normal if nothing has been saved in the game.') 293 | return False 294 | elif firstFourBytes == b'EBDM': 295 | infile.seek(40) 296 | savePayloadSize = struct.unpack('>I', infile.read(4))[0] 297 | assert savePayloadSize >= 1 298 | 299 | infile.seek(64) 300 | patch = array('B', infile.read(savePayloadSize)) 301 | 302 | patchedFdsImage = get_vci_image_as_fds_image(apply_patch(get_vci_body_in_app(appFile, find_fds_header_in_app(appFile)), patch)) 303 | outfile = open(outputPathWithoutExtension + '.withsavedata.fds', 'wb') # if we name it .fds, it will overwrite the pristine file 304 | outfile.write(patchedFdsImage) 305 | else: 306 | raise ValueError 307 | 308 | #passes arrays, returns arrays 309 | def apply_patch(originalFile, patch): 310 | 311 | patchedFile = copy.deepcopy(originalFile) 312 | 313 | i = 0 314 | while True: 315 | #end of patch data - should not happen without seeing the terminator first 316 | assert i < len(patch) 317 | 318 | if patch[i] == 0xFF: 319 | #terminator - check that we are also at the end of the patch data 320 | assert i == len(patch)-1 321 | break 322 | else: 323 | #get patch offset 324 | byteArray = array('B', b'\x00') 325 | byteArray.extend(patch[i:i+3]) 326 | offset = struct.unpack('>I', array('B', byteArray))[0] 327 | assert offset >= 0 328 | 329 | length = struct.unpack('>H', patch[i+3 : i+5])[0] 330 | assert length > 0 331 | 332 | assert offset+length <= len(patchedFile) 333 | 334 | data = patch[i+5 : i+5+length] 335 | 336 | patchedFile[offset:offset+length] = data 337 | 338 | i += 3 + 2 + length 339 | 340 | return patchedFile 341 | 342 | 343 | def find_fds_bios_in_app(inputFile): 344 | inputFile.seek(0) 345 | return inputFile.read().find(FDS_BIOS_HEADER_MAGIC_WORD) 346 | 347 | if __name__ == '__main__': 348 | if len(sys.argv) != 3: 349 | sys.exit('Usage: %s input.app output' % sys.argv[0]) 350 | f = open(sys.argv[1], 'rb') 351 | result, output = extract_nes_file_from_app(f) 352 | f.close() 353 | if result <= 0: 354 | print("No rom or disk image found in file!") 355 | else: 356 | if result == 1: 357 | filename = sys.argv[2] + ".nes" 358 | elif result == 2: 359 | filename = sys.argv[2] + ".fds" 360 | 361 | f2 = open(filename, 'wb') 362 | f2.write(output.read()) 363 | f2.close() 364 | print('Done!') 365 | 366 | -------------------------------------------------------------------------------- /romchu.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | # Author: Bryan Cain 3 | # Original software (romchu 0.3) written in C by hcs 4 | # Date: January 17, 2011 5 | # Description: Decompresses Nintendo's N64 romc compression, type 2 (LZ77+Huffman) 6 | 7 | import sys, struct, time 8 | from array import array 9 | 10 | VERSION = "0.6" 11 | 12 | class backref(object): 13 | def __init__(self): 14 | self.bits = 0 15 | self.base = 0 16 | 17 | #backref backref_len[0x1D], backref_disp[0x1E]; 18 | backref_len, backref_disp = [], [] 19 | for i in range(0x1D): 20 | backref_len.append(backref()) 21 | for i in range(0x1E): 22 | backref_disp.append(backref()) 23 | 24 | #easiest to call through lz77.py 25 | def decompress(infile, inputOffset, nominal_size): 26 | #print("Decompressing Romchu") 27 | 28 | #block_mult = 0x10000 - unused 29 | block_count = 0 30 | out_offset = 0 31 | 32 | #header parsing moved to lz77.py 33 | # read header 34 | #infile.seek(0) 35 | #head_buf = infile.read(4) 36 | #bs = init_bitstream(head_buf, 0, 4*8) 37 | 38 | #nominal_size = head_buf[0] 39 | #nominal_size *= 0x100 40 | #nominal_size |= head_buf[1] 41 | #nominal_size *= 0x100 42 | #nominal_size |= head_buf[2] 43 | #nominal_size *= 0x40 44 | #nominal_size |= head_buf[3] >> 2 45 | #romc_type = head_buf[3] & 0x3 46 | 47 | #if romc_type != 2: 48 | # raise ValueError("Expected type 2 romc, got %d\n" % romc_type) 49 | 50 | #free_bitstream(bs) 51 | 52 | infile.seek(inputOffset) 53 | 54 | # initialize backreference lookup tables 55 | for i in range(8): 56 | backref_len[i].bits = 0 57 | backref_len[i].base = i 58 | 59 | i = 8 60 | for scale in range(1, 6): 61 | k = (1<<(scale+2)) 62 | while k < (1<<(scale+3)): 63 | backref_len[i].bits = scale 64 | backref_len[i].base = k 65 | k += (1< 0: 121 | read_size += 1 122 | 123 | #this is not needed in Python 124 | '''if read_size > len(payload_buf): 125 | raise ValueError("payload too large")''' 126 | 127 | payload_buf = infile.read(read_size) 128 | 129 | # attempt to parse... 130 | if compression_flag: 131 | # read table 1 size 132 | tab1_offset = 0 133 | bs = init_bitstream(payload_buf, tab1_offset, payload_bytes*8+payload_bits) 134 | tab1_size = get_bits(bs, 16) 135 | #free_bitstream(bs) 136 | 137 | # load table 1 138 | bs = init_bitstream(payload_buf, tab1_offset + 2, tab1_size) 139 | table1 = load_table(bs, 0x11D) 140 | #free_bitstream(bs) 141 | 142 | # read table 2 size 143 | tab2_offset = int(tab1_offset + 2 + (tab1_size+7) / 8) 144 | bs = init_bitstream(payload_buf, tab2_offset, 2*8) 145 | tab2_size = get_bits(bs, 16) 146 | #free_bitstream(bs) 147 | 148 | # load table 2 149 | bs = init_bitstream(payload_buf, tab2_offset + 2, tab2_size) 150 | table2 = load_table(bs, 0x1E) 151 | #free_bitstream(bs) 152 | 153 | # decode body 154 | body_offset = int(tab2_offset + 2 + (tab2_size+7) / 8) 155 | body_size = payload_bytes*8 + payload_bits - body_offset*8 156 | bs = init_bitstream(payload_buf, body_offset, body_size) 157 | 158 | while (bs.bits_left + bs.first_byte_bits) != 0: 159 | symbol = huf_lookup(bs, table1) 160 | 161 | if symbol < 0x100: 162 | # byte literal 163 | #unsigned char b = symbol; 164 | b = symbol 165 | assert out_offset <= nominal_size # generated too many bytes 166 | out_buf[out_offset] = b 167 | out_offset += 1 168 | else: 169 | # backreference 170 | len_bits = backref_len[symbol-0x100].bits 171 | length = backref_len[symbol-0x100].base 172 | if len_bits > 0: 173 | length += get_bits(bs, len_bits) 174 | length += 3 175 | 176 | symbol2 = huf_lookup(bs, table2) 177 | 178 | disp_bits = backref_disp[symbol2].bits 179 | disp = backref_disp[symbol2].base 180 | if disp_bits > 0: 181 | disp += get_bits(bs, disp_bits) 182 | disp += 1 183 | 184 | assert disp <= out_offset # backreference too far 185 | assert (out_offset + length) <= nominal_size # generated too many bytes 186 | 187 | #for i in range(length): 188 | count = 0 189 | while count < length: 190 | #for i in range(length): 191 | out_buf[out_offset] = out_buf[out_offset-disp] 192 | out_offset += 1 193 | count += 1 194 | 195 | #free_table(table1) 196 | #free_table(table2) 197 | #free_bitstream(bs) 198 | else: # not compression_flag 199 | assert (out_offset + payload_bytes) <= nominal_size # generated too many bytes 200 | out_buf[out_offset:out_offset+payload_bytes] = array('B', payload_buf[0:payload_bytes]) 201 | out_offset += payload_bytes 202 | 203 | block_count += 1 204 | sys.stdout.write("\rDecompressed %d of %d bytes [%x/%x] (%5.2f%%)" % (out_offset, nominal_size, out_offset, nominal_size, 100.0 * out_offset / nominal_size)) 205 | sys.stdout.flush() 206 | 207 | print() # start a new line after the progress counter 208 | assert out_offset == nominal_size # size mismatch 209 | 210 | return out_buf 211 | 212 | 213 | # bitstream reader 214 | class bitstream(object): 215 | def __init__(self): 216 | #const unsigned char* pool 217 | #long bits_left 218 | #uint8_t first_byte 219 | #int first_byte_bits 220 | self.pool = None 221 | self.bits_left = 0 222 | self.first_byte = 0 223 | self.first_byte_bits = 0 224 | self.index = 0 225 | 226 | # struct bitstream *init_bitstream(const unsigned char *pool, unsigned long pool_size) 227 | def init_bitstream(pool_buf, pool_start, pool_size): 228 | bs = bitstream() 229 | 230 | bs.pool = array('B', pool_buf[pool_start:pool_start+pool_size]) 231 | bs.bits_left = pool_size 232 | bs.first_byte_bits = 0 233 | 234 | # check that padding bits are 0 (to ensure we aren't ignoring anything) 235 | if pool_size % 8: 236 | if bs.pool[int(pool_size/8)] & ~((1<<(pool_size%8))-1): 237 | raise ValueError("nonzero padding at end of bitstream") 238 | 239 | return bs 240 | 241 | # uint32_t get_bits(struct bitstream *bs, int bits) 242 | def get_bits(bs, bits): 243 | accum = 0 244 | 245 | if bits > 32: 246 | raise ValueError("get_bits() supports max 32") 247 | if bits > (bs.bits_left + bs.first_byte_bits): 248 | raise ValueError("get_bits() underflow") 249 | 250 | count = 0 251 | #for i in range(bits): 252 | while count < bits: 253 | if bs.first_byte_bits == 0: 254 | #print(bs.bits_left, len(bs.pool), i) 255 | bs.first_byte = bs.pool[bs.index] 256 | bs.index += 1 257 | if bs.bits_left >= 8: 258 | bs.first_byte_bits = 8 259 | bs.bits_left -= 8 260 | else: 261 | bs.first_byte_bits = bs.bits_left 262 | bs.bits_left = 0 263 | 264 | accum >>= 1 265 | accum |= (bs.first_byte & 1) << 31 266 | bs.first_byte >>= 1 267 | bs.first_byte_bits -= 1 268 | count += 1 269 | 270 | return accum >> (32-bits) 271 | 272 | def free_bitstream(bs): 273 | pass 274 | 275 | # Huffman code handling 276 | '''class hufnode_inner(object): 277 | def __init__(self): 278 | self.left, self.right = 0, 0 279 | 280 | class hufnode_leaf(object): 281 | def __init__(self): 282 | self.symbol = 0 283 | 284 | class hufnode_union(object): 285 | def __init__(self): 286 | self.inner = hufnode_inner() 287 | self.leaf = hufnode_leaf()''' 288 | 289 | class hufnode(object): 290 | def __init__(self): 291 | self.is_leaf = False 292 | self.symbol = 0 293 | self.left = 0 294 | self.right = 0 295 | #self.u = hufnode_union() 296 | 297 | class huftable(object): 298 | def __init__(self): 299 | self.symbols = 0 300 | self.t = [] 301 | 302 | # struct huftable *load_table(struct bitstream *bs, int symbols) 303 | def load_table(bs, symbols): 304 | len_count = [0] * 32 305 | codes = [0] * 32 306 | length_of = [0] * symbols 307 | i = 0 308 | 309 | while i < symbols: 310 | if get_bits(bs, 1): 311 | # run of equal lengths 312 | count = get_bits(bs, 7) + 2 313 | length = get_bits(bs, 5) 314 | 315 | len_count[length] += count 316 | for j in range(count): 317 | length_of[i] = length 318 | i += 1 319 | else: 320 | # set of inequal lengths 321 | count = get_bits(bs, 7) + 1 322 | 323 | for j in range(count): 324 | length = get_bits(bs, 5) 325 | length_of[i] = length 326 | len_count[length] += 1 327 | i += 1 328 | 329 | assert (bs.bits_left + bs.first_byte_bits) == 0 # did not exhaust bitstream reading table 330 | 331 | # compute the first canonical Hufman code for each length 332 | accum = 0 333 | for i in range(1, 32): 334 | accum = (accum + len_count[i-1]) << 1 335 | codes[i] = accum 336 | 337 | # determine codes and build a tree 338 | ht = huftable() 339 | ht.symbols = symbols 340 | for i in range(symbols * 2): 341 | node = hufnode() 342 | node.is_leaf = 0 343 | node.left = 0 344 | node.right = 0 345 | ht.t.append(node) 346 | 347 | next_free_node = 1 348 | for i in range(symbols): 349 | cur = 0 350 | if length_of[i] == 0: 351 | # 0 length indicates absent symbol 352 | continue 353 | 354 | #for (int j = length_of[i]-1; j >= 0; j --) 355 | for j in range(length_of[i]-1, -1, -1): 356 | #next = 0 # shouldn't be necessary 357 | assert not ht.t[cur].is_leaf # oops, walked onto a leaf 358 | 359 | if codes[length_of[i]] & (1<= 8: 391 | bs.first_byte_bits = 8 392 | bs.bits_left -= 8 393 | else: 394 | bs.first_byte_bits = bs.bits_left 395 | bs.bits_left = 0 396 | 397 | #if get_bits(bs, 1): 398 | if bs.first_byte & 1: 399 | # 1 == right 400 | cur = ht.t[cur].right 401 | else: 402 | cur = ht.t[cur].left 403 | 404 | bs.first_byte >>= 1 405 | bs.first_byte_bits -= 1 406 | 407 | return ht.t[cur].symbol 408 | 409 | 410 | -------------------------------------------------------------------------------- /rso.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | 3 | # Utilities to extract stuff from RSO files. 4 | # RSO files are like SO files in Linux or DLL files in Windows, so they contain mostly code but sometimes other binary data. 5 | # In some VC Arcade games, some RSO files contain the system specific emulation code, as well as the emulated ROMs. 6 | 7 | # Invaluable source: https://github.com/sepalani/librso/blob/master/rvl/rso.py 8 | 9 | # The actual binary data is in the "Sections". 10 | # The externals table lists different offset within different sections to make available to use by other code. (including ROMs) 11 | # The internals table is a bit unknown. 12 | 13 | # Unless otherwise mentioned, all "offset" is relative to the beginning of the file. 14 | # It seems that all offsets are aligned to 4 bytes, there is null padding between file portions with odd lenghts. 15 | # Unless otherwise mentioned, each value is 4 bytes 16 | 17 | #HEADER: 18 | 19 | #Info 20 | #headerValues[0] = Next RSO Entry 21 | #headerValues[1] = Previous RSO Entry 22 | #headerValues[2] = Section Count 23 | #headerValues[3] = Section Table Offset 24 | #headerValues[4] = Name Offset 25 | #headerValues[5] = Name Size 26 | #headerValues[6] = Version 27 | #headerValues[7] = BSS Section Size 28 | 29 | #SectionInfo 30 | #headerValues[8] = 4x1 byte values - Has Prolog, Has Epilog, Has Unresolved, Has BSS 31 | #headerValues[9] = Prolog Offset 32 | #headerValues[10] = Epilog Offset 33 | #headerValues[11] = Unresolved Offset 34 | 35 | #RelocationTables: 36 | #headerValues[12] = Internals Relocation Table Offset 37 | #headerValues[13] = Internals Relocation Table Size 38 | #headerValues[14] = Externals Relocation Table Offset 39 | #headerValues[15] = Externals Relocation Table Size 40 | 41 | #Exports: 42 | #headerValues[16] = Exports Offset 43 | #headerValues[17] = Exports Size 44 | #headerValues[18] = Exports Name Offset 45 | 46 | #Imports 47 | #headerValues[19] = Imports Offset 48 | #headerValues[20] = Imports Size 49 | #headerValues[21] = Imports Name Offset 50 | 51 | #end of header, the rest of the file is different portions, whose positions is given in the header, or in other tables. 52 | 53 | #SECTION TABLE: 54 | #Starts at position "Section Table Offset" in header 55 | #For each section (0 - section count-1): 56 | # Section offset 57 | # Section length, in bytes 58 | #Some sections have both 0 offset and 0 length. Ignoring them. 59 | #Some sections have both 0 offset but a positive length. Not sure what this means. Ignoring them. 60 | 61 | #SECTION: 62 | #Starts at the position and has the length determined by the values in the section table. 63 | #The content is the actual binary data. Most often Wii binary code, but sometimes ROM content. 64 | 65 | #NAME: 66 | #The name of the entire library. 67 | #Starts at the position and has the length specified in header. 68 | 69 | #EXPORTS TABLE: 70 | #Contains the list of export points in the different sections. 71 | #Starts at position "Exports Offset" in header. The size in bytes is specified in header. The number of entries is the size in bytes / 16. 72 | #For each entry: 73 | #Name offset (offset of the name in the Exports Name table, relative to the start of the export name table) 74 | #Data Offset (relative to section start, i.e. 0=first byte in the section) 75 | #Section index (0-section count-1) 76 | #??? (large value, not referenced elsewhere in the file) 77 | 78 | #EXPORT NAMES: 79 | #Starts at position "Exports Name Offset" in header. Size is not specified anywhere. 80 | #Contains a number of nullterminated names without any alignment. 81 | #The position of a name is specified in the exports table. 82 | 83 | #EXTERNALS RELOCATION TABLE 84 | #Starts point and Size specified at "Externals Relocation Table Offset" and "Size" in header. 85 | #Not sure what this is. 86 | 87 | #IMPORTS TABLE, IMPORT NAMES, INTERNALS RELOCATION TABLE: 88 | #Similar to EXTERNALS counterparts. 89 | 90 | 91 | import struct 92 | 93 | class rso(object): 94 | def __init__(self, inputFile): 95 | self.file = inputFile 96 | 97 | def getExport(self, name, dataLength): 98 | self.file.seek(0) 99 | headerValues = struct.unpack(">22I", self.file.read(22*4)) 100 | 101 | exportTableOffset = headerValues[16] 102 | exportTableLength = headerValues[17] 103 | exportNamesOffset = headerValues[18] 104 | 105 | for exportTableEntryPosition in range(0, exportTableLength, 4*4): 106 | 107 | self.file.seek(exportTableOffset + exportTableEntryPosition) 108 | exportTableEntry = struct.unpack(">4I", self.file.read(4*4)) 109 | 110 | namePosition = exportTableEntry[0] 111 | dataPosition = exportTableEntry[1] 112 | sectionIndex = exportTableEntry[2] 113 | 114 | nameCandidate = self.readNullTerminatedString(exportNamesOffset + namePosition) 115 | 116 | if name == nameCandidate: 117 | 118 | sectionTableEntry = self.getSectionTableEntry(sectionIndex) 119 | 120 | sectionOffset = sectionTableEntry[0] 121 | sectionLength = sectionTableEntry[1] 122 | 123 | assert dataPosition + dataLength <= sectionLength 124 | 125 | self.file.seek(sectionOffset + dataPosition) 126 | 127 | data = self.file.read(dataLength) 128 | 129 | assert len(data) == dataLength 130 | return data 131 | 132 | def getSectionTableEntry(self, sectionIndex): 133 | self.file.seek(0) 134 | headerValues = struct.unpack(">22I", self.file.read(22*4)) 135 | 136 | sectionTableOffset = headerValues[3] 137 | sectionCount = headerValues[2] 138 | 139 | assert sectionIndex < sectionCount 140 | 141 | self.file.seek(sectionTableOffset + sectionIndex*2*4) 142 | return struct.unpack(">2I", self.file.read(2*4)) 143 | 144 | 145 | def getAllExports(self): 146 | self.file.seek(0) 147 | headerValues = struct.unpack(">22I", self.file.read(22*4)) 148 | 149 | exportTableOffset = headerValues[16] 150 | exportTableLength = headerValues[17] 151 | exportNamesOffset = headerValues[18] 152 | 153 | exports = [] 154 | 155 | for exportTableEntryOffset in range(0, exportTableLength, 4*4): 156 | 157 | self.file.seek(exportTableOffset + exportTableEntryOffset) 158 | exportTableEntry = struct.unpack(">4I", self.file.read(4*4)) 159 | 160 | nameOffset = exportTableEntry[0] 161 | 162 | name = self.readNullTerminatedString(exportNamesOffset + nameOffset) 163 | 164 | exports.append(name) 165 | 166 | return exports 167 | 168 | def readNullTerminatedString(self, startOffset): 169 | self.file.seek(startOffset) 170 | ba = bytearray() 171 | while True: 172 | b = self.file.read(1) 173 | 174 | if b[0] == 0: 175 | return ba.decode('ascii') 176 | else: 177 | ba.extend(b) 178 | 179 | 180 | 181 | 182 | 183 | 184 | 185 | 186 | 187 | 188 | 189 | 190 | 191 | 192 | 193 | 194 | 195 | 196 | 197 | -------------------------------------------------------------------------------- /snesrestore.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | # Author: Bryan Cain 3 | # Date: December 31, 2010 4 | # Description: Attempts to restore the original sound samples to SNES ROMs in 5 | # Virtual Console games, which use uncompressed PCM data separate from the ROM. 6 | 7 | import os 8 | import sys 9 | import struct 10 | from brrencode3 import BRREncoder 11 | from io import BytesIO 12 | 13 | # vcrom: file-like object for the original VC ROM 14 | # brr: file-like object containing the BRR-compressed audio samples 15 | # Precondition: brr is the correct size 16 | def restore_brr_samples(vcrom, pcm): 17 | # read the samples from the input ROM into memory (TODO: check file size first) 18 | vcrom.seek(0) 19 | inbytes = vcrom.read() 20 | samplestart = inbytes.find(b'PCMF') # samples start with first instance of the string "PCMF" 21 | inbytes = inbytes[samplestart:] 22 | 23 | # initialize output ROM in memory as a BytesIO (pseudo-file) 24 | rom = BytesIO() 25 | rom.write(inbytes) 26 | 27 | # read the input BRR samples 28 | #brr.seek(0) 29 | #brrdata = brr.read() 30 | #lastbrroffset = None 31 | 32 | enc = BRREncoder(pcm) 33 | lastpcmoffset = 0 34 | 35 | # misc. variables 36 | #goodrom = open('SNADNE1.667', 'rb') 37 | wrong = 0 38 | controlwrong = 0 39 | indices = [] 40 | 41 | while inbytes.find(b'PCMF') >= 0: 42 | index = inbytes.find(b'PCMF') 43 | #filepos = samplestart + index 44 | 45 | # error checking to prevent infinite loops 46 | #assert index not in indices 47 | #indices.append(index) 48 | 49 | pcmf, pcmoffset = struct.unpack('<4sI', inbytes[index:index+8]) 50 | pcmoffset &= 0xffffff 51 | if pcmoffset % 16 or pcmoffset < lastpcmoffset: 52 | #print('%08x: unexpected offset %d' % (filepos, pcmoffset)) 53 | pcmoffset = lastpcmoffset + 16 54 | #else: 55 | # brroffset = 9 * (brroffset >> 4) 56 | 57 | # read the BRR sample 58 | #brr.seek(brroffset) 59 | #brrsample = brr.read(9) 60 | 61 | # read and encode the BRR block 62 | brrsample = bytearray(enc.encode_block(pcmoffset)) 63 | 64 | # error checking for invalid BRR offsets 65 | if len(brrsample) != 9: 66 | raise ValueError('Invalid BRR offset') 67 | 68 | # set the END bit in the BRR sample if it is set in the PCMF block 69 | if inbytes[index+7] & 1: 70 | brrsample[0] = (brrsample[0] | 1) 71 | 72 | # set the LOOP bit in the BRR sample if it is set in the PCMF block 73 | if inbytes[index+7] & 2: 74 | brrsample[0] = (brrsample[0] | 2) 75 | 76 | # checks whether sample matches the original ROM, when the original ROM is available (for debugging purposes) 77 | '''goodrom.seek(samplestart + index) 78 | grsample = goodrom.read(9) 79 | if brrsample != grsample: 80 | wrong += 1 81 | sys.stdout.write('%08x: ' % filepos) 82 | #if brrdata.find(grsample) >= 0: 83 | # print('wrong sample, correct BRR offset is %08x' % brrdata.find(grsample)) 84 | if brrsample[1:] == grsample[1:]: 85 | if abs(ord(brrsample[0]) - ord(grsample[0])) <= 3: 86 | controlwrong += 1 87 | print('SPC700 control bits differ') 88 | else: 89 | print('flags are different') 90 | else: 91 | print('sample encoded differently?') ''' 92 | 93 | rom.seek(index) 94 | rom.write(brrsample) 95 | inbytes = rom.getvalue() 96 | lastpcmoffset = pcmoffset 97 | 98 | #print('%d wrong samples' % wrong) 99 | #print('%d differences in SPC700 control bits' % controlwrong) 100 | 101 | rom.close() 102 | vcrom.seek(0) 103 | return vcrom.read(samplestart) + inbytes 104 | 105 | if __name__ == '__main__': 106 | import time 107 | 108 | if len(sys.argv) != 4: 109 | print('Usage: snesrestore game.rom game.pcm output.smc') 110 | sys.exit(1) 111 | 112 | vcrom = open(sys.argv[1], 'rb') 113 | pcm = open(sys.argv[2], 'rb') 114 | 115 | '''# encode raw PCM in SNES BRR format 116 | print('Encoding audio as BRR') 117 | brr = BytesIO() 118 | enc = BRREncoder(pcm, brr) 119 | enc.encode() 120 | pcm.close()''' 121 | 122 | # encode and inject BRR sound data into the ROM 123 | print('Encoding and restoring BRR audio data to ROM') 124 | start = time.process_time() 125 | string = restore_brr_samples(vcrom, pcm) 126 | end = time.process_time() 127 | print('Time: %.2f seconds' % (end - start)) 128 | 129 | # write to file 130 | output = open(sys.argv[3], "wb") 131 | output.write(string) 132 | output.close() 133 | pcm.close() 134 | #print('done') 135 | 136 | -------------------------------------------------------------------------------- /tgcd_extract.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | 3 | import os.path, struct, zlib 4 | from io import BytesIO, StringIO 5 | from configurationfile import getConfiguration 6 | 7 | 8 | #TODO: audio clearly runs to slow in Mednafen (may cause secondary issues, like music/sound starting at the wrong positions) 9 | # Exported audio files are 48khz. Maybe Mednafen probably plays this in 44khz (the CD-A frequency). 10 | # Can be worked-around by using any tool to convert the OGG files to CD-A or any other format that Mednafen can support 11 | # and updating the Cue sheet accordingly. 12 | #TODO: Extracted super air zonk freeze on BIOS screen in Mednafen, a raw disk dump (2352 bytes/sector) of the same game plays fine. 13 | # Maybe there is a problem with the CD-ROM overhead (VC emulator ignores it or recreates the headers/footers differntly than Mednafen?) 14 | 15 | 16 | 17 | def extract_tgcd(u8InputFile, outputFolder): 18 | 19 | # The data tracks = bin files are extracted as is, but padded or truncated to the length specified in HCD file. 20 | # cue file instructs emulator to read files as MODE1/2048. 21 | 22 | # Missing bin files (mentioned in HCD file but not existing) have replacment null-filled 23 | # ".dummy" files of the size specified in the HCD file. 24 | 25 | # Audio files are in OGG format and copied as is. 26 | 27 | # Missing audio files are replaced by null-filled ".dummy" files of the size specified in the HCD file. 28 | 29 | 30 | hcdFileName = getHcdFileName(extractFile(u8InputFile, "config.ini")) 31 | hcdFileContent = extractFile(u8InputFile, hcdFileName) 32 | assert hcdFileContent is not None 33 | 34 | cueFileContent = StringIO() 35 | 36 | #saveFile(hcdFileContent, "debug.hcd", outputFolder) 37 | #saveFile(extractFile(u8InputFile, "config.ini"), "debug.ini", outputFolder) 38 | 39 | for trackDescription in getTrackDescriptionsFromHcdFile(hcdFileContent): 40 | 41 | 42 | 43 | # It is normal that the file mentioned in HCD file does not exist. 44 | # First track is often an audio track with a voice asking the user to stop playing the disc in a CD player. 45 | # One of the last tracks is often a copy of the first data track. 46 | 47 | if trackDescription.dataTrack: 48 | 49 | if fileExists(u8InputFile, trackDescription.fileName): 50 | data = decompressBinaryFile(extractFile(u8InputFile,trackDescription.fileName)) 51 | fileName = trackDescription.fileName 52 | else: 53 | data = BytesIO() 54 | fileName = getWithOtherFileExtension(trackDescription.fileName, "dummy") 55 | saveFile(getLengthCorrectedData(data, 2048, trackDescription.sectorCount), fileName, outputFolder) 56 | cueFileContent.write(createCueDirective(trackDescription, fileName)) 57 | 58 | else: #Audio track 59 | 60 | if fileExists(u8InputFile, trackDescription.fileName): 61 | saveFile(extractFile(u8InputFile,trackDescription.fileName), trackDescription.fileName, outputFolder) 62 | cueFileContent.write(createCueDirective(trackDescription, trackDescription.fileName)) 63 | else: 64 | dummyFileName = getWithOtherFileExtension(trackDescription.fileName, "dummy") 65 | saveFile(getLengthCorrectedData(BytesIO(), 2352, trackDescription.sectorCount), dummyFileName, outputFolder) 66 | cueFileContent.write(createDummyAudioCueDirective(trackDescription, dummyFileName)) 67 | 68 | 69 | saveFile(cueFileContent, getWithOtherFileExtension(hcdFileName, "cue"), outputFolder) 70 | 71 | saveFile(extractFile(u8InputFile, "syscard3P.pce"), "syscard3P.pce", outputFolder) 72 | 73 | 74 | 75 | 76 | def fileExists(u8InputFile, fileName): 77 | return u8InputFile.hasfile(fileName) 78 | 79 | def extractFile(u8InputFile, fileName): 80 | return u8InputFile.getfile(fileName) 81 | 82 | def getHcdFileName(configFileContent): 83 | romname = getConfiguration(configFileContent, "ROM") 84 | if romname: 85 | return romname 86 | else: 87 | raise ValueError 88 | 89 | def getTrackDescriptionsFromHcdFile(hcdFileContentBytesIO): 90 | #convert BytesIO to stringIO 91 | hcdBytes = hcdFileContentBytesIO.read() 92 | # Convert to a "unicode" object 93 | text_obj = hcdBytes.decode('ascii') 94 | hcdFileContentStringIO = StringIO(text_obj) 95 | 96 | hcdFileContentStringIO.seek(0) 97 | returnArray = [] 98 | counter = 1 99 | previousEndSector = 0 100 | for line in hcdFileContentStringIO: 101 | trackDescription = TrackDescription(line, counter, previousEndSector) 102 | returnArray.append(trackDescription) 103 | previousEndSector = trackDescription.endSector 104 | counter += 1 105 | assert len(returnArray) > 0 106 | return returnArray 107 | 108 | def decompressBinaryFile(compressedBinaryFileContent): 109 | #Based on qwikrazor87's decomp.py! 110 | 111 | compressedBinaryFileContent.seek(0) 112 | entries = struct.unpack(' targetSize: 182 | newData = StringIO() 183 | data.seek(0) 184 | newData.write(data.read(targetSize)) 185 | return newData 186 | else: 187 | return data 188 | 189 | 190 | 191 | 192 | 193 | 194 | class TrackDescription(object): 195 | def __init__(self, hcdLine, trackNumber, previousEndSector): 196 | splitLine = hcdLine.split(",") 197 | 198 | assert trackNumber >= 0 199 | assert trackNumber <= 99 200 | self.trackNumber = trackNumber 201 | 202 | assert len(splitLine[2]) > 0 203 | self.fileName = splitLine[2] 204 | 205 | normalizedTrackType = splitLine[1].strip() 206 | assert normalizedTrackType == "audio" or normalizedTrackType == "code" 207 | self.dataTrack = normalizedTrackType == "code" # True = data, False = audiotrack 208 | 209 | # start sectors seems to be offset by 150 sectors 210 | self.startSector = int(splitLine[3]) - 150 211 | assert (self.startSector >= 0) 212 | 213 | self.sectorCount = int(splitLine[4]) 214 | assert (self.sectorCount >= 0) 215 | 216 | self.endSector = self.startSector + self.sectorCount #NON-INCLUSIVE 217 | assert (self.endSector >= self.startSector) 218 | 219 | # Calculate the pregap. 220 | # In Cue/Bin, each track starts immediatelly after the next track. 221 | # In HCD, the start sector of each track is specified. 222 | # We calculate pregap for the ISO to start the new track at the correct sector. 223 | 224 | # Sometimes this track is positioned before the last track's start+length. 225 | # I've only seen it once, track 3 on Castlevania: RoB is about 15 frames (0,1 or 0,2 sec) too long. 226 | # It is probably a bug on that game. 227 | #if (self.startSector - previousEndSector < 0): 228 | # print("Warning: Start sector less than previous end sector.") 229 | # print(self.trackNumber) 230 | # print(self.fileName) 231 | # print(self.startSector) 232 | # print(self.sectorCount) 233 | # print(previousEndSector) 234 | 235 | self.pregap = max(0, self.startSector - previousEndSector) 236 | assert (self.pregap) >= 0 237 | -------------------------------------------------------------------------------- /tgsave.py: -------------------------------------------------------------------------------- 1 | from io import BytesIO 2 | import os 3 | 4 | # Wii/TurboGrafx save format reverse engineered by Euan Forrester ( https://github.com/euan-forrester/ ) and JanErikGunnar 5 | 6 | # Also with help from: 7 | # https://blackfalcongames.net/?p=190 8 | # http://blockos.github.io/HuDK/doc/files/include/bram-s.html 9 | # https://www.lorenzomoretti.com/wp-content/files/chapter_0.txt 10 | # https://github.com/asterick/TurboSharp/blob/master/Text/pcetech.txt 11 | 12 | def bitwise_xor_bytes(a, b): 13 | result_int = int.from_bytes(a, byteorder="big", signed = True) ^ int.from_bytes(b, byteorder="big", signed = True) 14 | return result_int.to_bytes(max(len(a), len(b)), byteorder="big", signed = True) 15 | 16 | def bitwise_not_bytes(a): 17 | result_int = ~int.from_bytes(a, byteorder="big", signed = True) 18 | return result_int.to_bytes(len(a), byteorder="big", signed = True) 19 | 20 | def unmangle_tgsave(inputMangledFilePath, outputPlainFileFolder): 21 | 22 | # mangled format: 23 | # 4 bytes magic word "$PCE" 24 | # 4 bytes IV (random bytes, different for each game or save?) 25 | # 4 bytes purpose unknown (unused IV?) 26 | # 4 bytes padding, always 0 27 | # the rest of the file is the content of an 8 KiB BRAM, each block of 4 bytes is xored with the NOT previous unmangled (the first block being xored with the IV) 28 | # After demangling the 8 KiB: 29 | # 4 bytes magic word "HUBM" 30 | # 2 bytes "Pointer to the first byte after BRAM." 31 | # BRAM starts at 0x8000 32 | # Wii emulates an 8 KiB BRAM 0 0x2000 33 | # That + endiannes = these two bytes are 0x00A0 34 | # ... the rest of the bram content 35 | # 16 unknown bytes (checksums?) 36 | 37 | # We need to unmangle this (remove the Wii header/footer, undo the xoring) to get a file usable in emulators. 38 | # It's uncertain how emulators will handle the 8 KiB BRAM. 39 | # All official hardware only had 2 KiB. 40 | # We export both the original, and a copy converted to 2 KiB. 41 | 42 | inputFile = open(inputMangledFilePath, 'rb') 43 | plain8kMemcard = BytesIO() 44 | 45 | magicWord = inputFile.read(4) 46 | assert magicWord == b'$PCE' 47 | 48 | # special case for first iteration 49 | previousMangledBlock = bitwise_not_bytes(inputFile.read(4)) 50 | 51 | inputFile.read(4) 52 | padding = inputFile.read(4) 53 | assert padding == b'\x00\x00\x00\x00' 54 | 55 | plain8kMemcard = BytesIO() 56 | 57 | for i in range(0x0, 0x2000, 4): 58 | mangledBlock = inputFile.read(4) 59 | assert len(mangledBlock) == 4 60 | 61 | plainBlock = bitwise_xor_bytes(mangledBlock, bitwise_not_bytes(previousMangledBlock)) 62 | 63 | if i == 0: 64 | assert plainBlock == b'HUBM' 65 | elif i == 4: 66 | assert plainBlock[0] == 0x00 and plainBlock[1] == 0xa0 67 | 68 | plain8kMemcard.write(plainBlock) 69 | 70 | previousMangledBlock = mangledBlock 71 | 72 | inputFile.read(16) # unknown (checksum?) 73 | beyondEof = inputFile.read(1) 74 | assert len(beyondEof) == 0 75 | 76 | output = open(os.path.join(outputPlainFileFolder, "8k.sav"), 'wb') 77 | output.write(plain8kMemcard.getvalue()) 78 | output.close() 79 | 80 | 81 | 82 | # convert the 8k memcard to a 2k memcard by truncating it, and changing byte 2 and 3 from $A000 to $8800 83 | # the bytes are pointers to the byte after the memory card. The memory card starts at 0x8000. 84 | # 8800-8000 = 0800 (2 KiB) 85 | # A000-8000 = 2000 (8 KiB) 86 | 87 | 88 | plain2kMemcard = BytesIO() 89 | plain8kMemcard.seek(0) 90 | 91 | 92 | for i in range(0x0, 0x800, 2): 93 | block = plain8kMemcard.read(2) 94 | assert len(block) == 2 95 | 96 | if i == 4: 97 | assert block == b'\x00\xa0' 98 | plain2kMemcard.write(b'\x00\x88') 99 | else: 100 | plain2kMemcard.write(block) 101 | 102 | #make sure the rest of the 8 KiB memory card is empty 103 | for i in range(0x800, 0x2000, 4): 104 | emptyBlock = plain8kMemcard.read(4) 105 | assert emptyBlock == b'\x00\x00\x00\x00' 106 | 107 | 108 | output = open(os.path.join(outputPlainFileFolder, "2k.sav"), 'wb') 109 | output.write(plain2kMemcard.getvalue()) 110 | output.close() 111 | 112 | print("Extracted BRAM to 2k.sav and 8k.sav") 113 | -------------------------------------------------------------------------------- /u8archive.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | # Author: Bryan Cain (Plombo) 3 | # Date: December 27, 2010 4 | # Description: Reads Wii U8 archives. 5 | 6 | import os, struct, posixpath 7 | from io import BytesIO 8 | import lz77, huf8, lzh8 9 | import re 10 | 11 | class U8Archive(object): 12 | # archive can be a string (filesystem path) or file-like object 13 | def __init__(self, archive): 14 | if type(archive) == str: 15 | #print(archive) 16 | self.file = open(archive, 'rb') 17 | else: 18 | self.file = archive 19 | assert self.file 20 | self.files = [] 21 | self.readheader() 22 | 23 | def readheader(self): 24 | magic, rootnode_offset, header_size, data_offset = tuple(struct.unpack('>IIII', self.file.read(16))) 25 | #print('IS A U8?') 26 | #print(magic) 27 | #print(0x55aa382d) 28 | 29 | assert magic == 0x55aa382d 30 | assert rootnode_offset == 0x20 31 | assert self.file.read(16) == 16 * b'\0' 32 | 33 | root = Node(self.file, rootnode_offset) 34 | root.path = '' 35 | path = '' 36 | curdirs = [root.size] 37 | dirnames = [''] 38 | filenum = 1 39 | while curdirs: 40 | node = Node(self.file, rootnode_offset + 12 * root.size) 41 | node.path = posixpath.join(path, node.name) 42 | filenum += 1 43 | if node.type == 0x100: 44 | # change current path if this is a directory 45 | path = node.path 46 | #print(node.name, node.path) 47 | curdirs.append(node.size) 48 | dirnames.append(node.name) 49 | else: self.files.append(node) 50 | 51 | while curdirs and filenum >= curdirs[len(curdirs)-1]: 52 | #print('done with ' + dirnames.pop() + ' at %d' % filenum) 53 | path = posixpath.dirname(path) 54 | curdirs.pop() 55 | 56 | # closes the physical file associated with this archive 57 | def close(self): 58 | self.file.close() 59 | 60 | # returns True if this archive has a file with the given path 61 | def hasfile(self, path): 62 | f = self.getfile(path) 63 | if f: 64 | f.close() 65 | return True 66 | else: 67 | return False 68 | 69 | # returns a file-like object (actually a BytesIO object) for the specified 70 | # file; detects and decompresses compressed files (LZ77/Huf8/LZH8) automatically! 71 | # path: file name (string) or actual file node, but NOT node path :D 72 | def getfile(self, path): 73 | for node in self.files: 74 | if node == path or (type(path) == str and node.name.endswith(path)): 75 | if node == path: 76 | path = node.name 77 | self.file.seek(node.data_offset) 78 | file = BytesIO(self.file.read(node.size)) 79 | if path.startswith("LZ77"): 80 | try: 81 | decompressed_file = BytesIO(lz77.decompress_nonN64(file)) 82 | file.close() 83 | return decompressed_file 84 | except (ValueError, IndexError): 85 | print("LZ77 decompression of '%s' failed" % path) 86 | print('Dumping compressed file to %s' % path) 87 | f2 = open(path, "wb") 88 | f2.write(file.read()) 89 | f2.close() 90 | file.close() 91 | return None 92 | elif path.startswith("Huf8"): 93 | try: 94 | decompressed_file = BytesIO() 95 | huf8.decompress(file, decompressed_file) 96 | file.close() 97 | decompressed_file.seek(0) 98 | return decompressed_file 99 | except Exception: 100 | print("Huf8 decompression of '%s' failed" % path) 101 | print("Dumping compressed file to %s" % path) 102 | f2 = open(path, "wb") 103 | f2.write(file.read()) 104 | f2.close() 105 | file.close() 106 | return None 107 | elif path.startswith("LZH8"): 108 | try: 109 | decompressed_file = BytesIO() 110 | decompressed_file.write(lzh8.decompress(file)) 111 | decompressed_file.seek(0) 112 | file.close() 113 | return decompressed_file 114 | except Exception: 115 | print("LZH8 decompression of '%s' failed" % path) 116 | print("Dumping compressed file to %s" % path) 117 | f2 = open(path, "wb") 118 | file.seek(0) 119 | f2.write(file.read()) 120 | f2.close() 121 | file.close() 122 | return None 123 | else: 124 | return file 125 | return None 126 | 127 | # finds a file with the given name, accounting for compression prefixes like "LZ77", "Huf8", etc. 128 | def findfile(self, name): 129 | for f in self.files: 130 | names = (name, "LZ77"+name, "LZ77_"+name, "Huf8"+name, "Huf8_"+name, "LZH8"+name, "LZH8_"+name) 131 | if f.name in names: return f.name 132 | return None 133 | 134 | def findfilebyregex(self, reExpression): 135 | for f in self.files: 136 | if re.match(reExpression, f.name): return f.name 137 | return None 138 | 139 | def extract(self, dest): 140 | if not os.path.lexists(dest): os.makedirs(dest) 141 | for node in self.files: 142 | if node.name in ('', '.'): continue 143 | if node.type == 0x100: 144 | os.makedirs(os.path.join(dest, node.path)) 145 | #print('created dir %s' % os.path.join(dest, node.path)) 146 | else: 147 | #print(node.path) 148 | path = os.path.join(dest, node.path) 149 | if not os.path.lexists(os.path.dirname(path)): os.makedirs(os.path.dirname(path)) 150 | f = open(path, 'wb') 151 | contents = self.getfile(node) 152 | contents.seek(0) 153 | f.write(contents.read()) 154 | f.close() 155 | #print('extracted file %s' % os.path.join(dest, node.path)) 156 | 157 | # file node object 158 | class Node(object): 159 | def __init__(self, arcfile, stringoffset): 160 | self.rawdata = arcfile.read(12) 161 | arcpos = arcfile.tell() 162 | chunk1, self.data_offset, self.size = tuple(struct.unpack('>III', self.rawdata)) 163 | self.type = chunk1 >> 16 164 | self.name_offset = chunk1 & 0xffffff 165 | 166 | # the root node has a name_offset of 0 167 | if not self.name_offset: return 168 | 169 | # no sane file name should be more than 64 bytes; if one is, string.index() will throw an exception 170 | arcfile.seek(stringoffset + self.name_offset) 171 | binaryName = arcfile.read(64) 172 | self.name = (binaryName[0:binaryName.index(b'\0')]).decode('ascii') 173 | #print(self.name) 174 | arcfile.seek(arcpos) 175 | 176 | if __name__ == '__main__': 177 | # Quick functionality test and sanity check; will only work on my (Plombo's) computer without a path change 178 | import os, os.path 179 | arc = U8Archive(os.path.join(os.getenv('HOME'), 'wii/ssb/00000005.app')) 180 | print(arc.hasfile('romc')) 181 | print(len(arc.getfile('romc').read())) 182 | 183 | -------------------------------------------------------------------------------- /wiimetadata.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | # Author: Bryan Cain (Plombo) 3 | # Date: December 27, 2010 4 | # Description: Reads Wii title metadata from an extracted NAND dump. 5 | # Thanks to Leathl for writing Wii.cs in ShowMiiWads, which was an important 6 | # reference in writing this program. 7 | 8 | import os, os.path, struct, shutil, re, zlib, lzma 9 | from io import BytesIO 10 | import gensave, n64save 11 | from u8archive import U8Archive 12 | from ccfarchive import CCFArchive 13 | import lz77 14 | from nes_extract import extract_nes_file_from_app, extract_fds_bios_from_app, convert_nes_save_data 15 | from snesrestore import restore_brr_samples 16 | from neogeo_decrypt import decrypt_neogeo 17 | from neogeo_convert import convert_neogeo 18 | from arcade_extract_ccf import extract_arcade_ccf 19 | import arcade_extract_gng 20 | from tgcd_extract import extract_tgcd 21 | from tgsave import unmangle_tgsave 22 | from configurationfile import getConfiguration 23 | from n64crc import updateN64Crc 24 | import game_specific_patches 25 | 26 | # rom: file-like object 27 | # path: string (filesystem path) 28 | def writerom(rom, path): 29 | f = open(path, 'wb') 30 | rom.seek(0) 31 | f.write(rom.read()) 32 | f.close() 33 | rom.seek(0) 34 | 35 | class RomExtractor(object): 36 | # file extensions for ROMs (not applicable for all formats) 37 | extensions = { 38 | 'Nintendo 64': '.z64', 39 | 'Genesis': '.gen', 40 | 'Master System': '.sms', 41 | 'SNES': '.smc', 42 | 'TurboGrafx16': '.pce' 43 | } 44 | 45 | def __init__(self, id, name, channeltype, nand): 46 | self.id = id 47 | self.name = name 48 | self.channeltype = channeltype 49 | self.nand = nand 50 | 51 | def ensure_folder_exists(self, outputFolderName): 52 | if not os.path.lexists(outputFolderName): 53 | os.makedirs(outputFolderName) 54 | 55 | def extract(self): 56 | content = os.path.join(self.nand.path, 'title', '00010001', self.id, 'content') 57 | rom_extracted = False 58 | manual_extracted = False 59 | 60 | for app in os.listdir(content): 61 | if not app.endswith('.app'): continue 62 | app = os.path.join(content, app) 63 | if self.extractrom(app, os.path.join(self.channeltype, self.name), app[-10:], self.id): rom_extracted = True 64 | if self.extractmanual(app, os.path.join(self.channeltype, self.name, 'manual')): manual_extracted = True 65 | 66 | if rom_extracted and manual_extracted: return 67 | elif rom_extracted: print('Unable to extract manual.') 68 | elif manual_extracted: print('Unable to extract ROM.') 69 | else: print('Unable to extract ROM and manual.') 70 | 71 | # Actually extract the ROM 72 | # Currently works for almost all NES, SNES, N64, TG16, Master System, and Genesis ROMs. 73 | def extractrom(self, u8path, gameOutputPath, name, id): 74 | funcs = { 75 | 'Nintendo 64': self.extractrom_n64, 76 | 'Genesis': self.extractrom_sega, 77 | 'Master System': self.extractrom_sega, 78 | 'NES': self.extractrom_nes, 79 | 'SNES': self.extractrom_snes, 80 | 'TurboGrafx16': self.extractrom_tg16, 81 | 'TurboGrafxCD': self.extractrom_tgcd, 82 | 'Neo Geo': self.extractrom_neogeo, 83 | 'Arcade': self.extractrom_arcade 84 | } 85 | 86 | if self.channeltype == 'NES': 87 | #NES roms are NOT packages in U8 archives. 88 | u8arc = self.tryGetU8Archive(u8path) 89 | if u8arc: 90 | return False 91 | else: 92 | u8arc = u8path 93 | elif self.channeltype == 'Arcade': 94 | #SOME arcade games are packed in U8 archives 95 | u8arc = u8path 96 | else: 97 | u8arc = self.tryGetU8Archive(u8path) 98 | 99 | #x = open(u8path, 'rb') 100 | #self.ensure_folder_exists(gameOutputPath) 101 | #f = open(os.path.join(gameOutputPath, 'file_' + name),'wb') 102 | #f.write(x.read()) 103 | #f.close() 104 | 105 | if not u8arc: 106 | return False 107 | 108 | self.ensure_folder_exists(gameOutputPath) 109 | 110 | if self.channeltype in funcs.keys(): 111 | return funcs[self.channeltype](u8arc, gameOutputPath, self.name, id) 112 | else: 113 | return False 114 | 115 | def extractrom_nes(self, u8path, outputPath, filenameWithoutExtension, id): 116 | if not os.path.exists(u8path): return False 117 | 118 | f = open(u8path, 'rb') 119 | result, output = extract_nes_file_from_app(f) 120 | 121 | hasExportedSaveData = False 122 | if result == 1 or result == 2: 123 | saveFilePath = self.getsavefile('savedata.bin') 124 | if saveFilePath != None: 125 | try: 126 | hasExportedSaveData = convert_nes_save_data(saveFilePath, os.path.join(outputPath, self.name), f) 127 | except: 128 | print('Failed to extract save file(s)') 129 | pass 130 | 131 | 132 | # fdsBios = extract_fds_bios_from_app(f) 133 | 134 | f.close() 135 | 136 | if result == 1: 137 | # nes rom 138 | 139 | # make sure save/SRAM flag is set if the game has save data - not sure which games this is used for? 140 | # Original behaviour in vcromclaim was to force the flag IF there was save data, otherwise leave it. 141 | # That is not complete because user maybe never saved anything. 142 | # Have extended to include a check for some games known to have SRAM. 143 | 144 | sramFlagAlreadySet = output.getvalue()[6] & 2 145 | 146 | if sramFlagAlreadySet: 147 | # WORST CASE: some games incorrectly indicate to emulator that they use SRAM. 148 | # If it causes problems - extend logic to clear flags of games known NOT to have SRAM? 149 | print('Leaving SRAM flag ON.') 150 | else: 151 | # Incomplete list of games known to have SRAM: Kirby's Adventure, Zelda 1 and 2, Star Tropics 1 and 2, Final Fantasy, Wario's Woods, NES Open Tournament Golf 152 | # All of those except Kirby's Adventure has been seen to have the SRAM flag set though. 153 | knownToHaveSram = id in ['46413845', '46414b45', '46413945', '46433645', '46455245', '46464145', '46414d45', '46415045'] 154 | 155 | # Flag NOT set correctly! Not sure if this causes any problems, not sure if emulators actually care about this flag. 156 | if knownToHaveSram or hasExportedSaveData: 157 | print('Changing SRAM flag from OFF to ON! It is OFF in extracted ROM header, but we found saved data, or the game is known to have SRAM.') 158 | output.seek(6) 159 | output.write((output.getvalue()[6] | 2).to_bytes(1, 'little')) 160 | else: 161 | print('Leaving SRAM flag OFF, because nothing suggest it is wrong.') 162 | 163 | filename = os.path.join(outputPath, filenameWithoutExtension + ".nes") 164 | 165 | print('Got ROM: %s' % filename) 166 | 167 | elif result == 2: 168 | # FDS 169 | 170 | filename = os.path.join(outputPath, filenameWithoutExtension + ".fds") 171 | 172 | print('Got FDS image: %s' % filename) 173 | 174 | else: 175 | return False 176 | 177 | writerom(output, filename) 178 | 179 | # if fdsBios != None: 180 | # print('Extracted FDS BIOS') 181 | # writerom(fdsBios, os.path.join(outputPath, "DISKSYS.ROM")) 182 | # 183 | if hasExportedSaveData: 184 | print('Extracted save data') 185 | 186 | return True 187 | 188 | def extractrom_n64(self, arc, outputPath, filenameWithoutExtension, id): 189 | filename = os.path.join(outputPath, filenameWithoutExtension + self.extensions[self.channeltype]) 190 | if arc.hasfile('rom'): 191 | rom = arc.getfile('rom') 192 | outfile = open(filename, 'wb') 193 | outfile.write(updateN64Crc(game_specific_patches.patch_specific_games(rom.read()))) 194 | outfile.close() 195 | print('Got ROM: %s' % filename) 196 | elif arc.hasfile('romc'): 197 | rom = arc.getfile('romc') 198 | print('Decompressing ROM: %s (this could take a minute or two)' % filename) 199 | try: 200 | outfile = open(filename, 'wb') 201 | outfile.write(updateN64Crc(game_specific_patches.patch_specific_games(lz77.decompress_n64(rom)))) 202 | outfile.close() 203 | print('Got ROM: %s' % filename) 204 | except IndexError: # unknown compression - something besides LZSS and romchu? 205 | print('Decompression failed: unknown compression type') 206 | outfile.close() 207 | os.remove(filename) 208 | return False 209 | else: return False 210 | 211 | # extract save file 212 | savepath = self.extractsave(outputPath) 213 | if savepath: print('Extracted save file(s)') 214 | else: print('Failed to extract save file(s)') 215 | 216 | return True 217 | 218 | def extractrom_sega(self, arc, outputPath, filenameWithoutExtension, id): 219 | filename = os.path.join(outputPath, filenameWithoutExtension + self.extensions[self.channeltype]) 220 | if arc.hasfile('data.ccf'): 221 | ccf = CCFArchive(arc.getfile('data.ccf')) 222 | 223 | if ccf.hasfile('config'): 224 | romfilename = getConfiguration(ccf.getfile('config'), 'romfile') 225 | else: 226 | return False 227 | 228 | if romfilename: 229 | rom = ccf.find(romfilename) 230 | writerom(rom, filename) 231 | print('Got ROM: %s' % filename) 232 | 233 | if self.extractsave(outputPath): 234 | print('Extracted save to %s.srm' % self.name) 235 | else: 236 | print('No save file found') 237 | 238 | return True 239 | else: 240 | print('ROM filename not specified in config') 241 | return False 242 | 243 | def extractrom_tg16(self, arc, outputPath, filenameWithoutExtension, id): 244 | #for node in arc.files: 245 | # print(node.name) 246 | config = arc.getfile('config.ini') 247 | #writerom(config, os.path.join(outputPath, "config.ini")) 248 | #savetemplate = arc.getfile('savedata.tpl') 249 | #writerom(savetemplate, os.path.join(outputPath, "savedata.tpl")) 250 | 251 | if not config: 252 | print('config.ini not found') 253 | return False 254 | 255 | path = getConfiguration(config, "ROM") 256 | 257 | if not path: 258 | print('ROM filename not specified in config.ini') 259 | return False 260 | 261 | rom = arc.getfile(path) 262 | 263 | if rom: 264 | filename = os.path.join(outputPath, filenameWithoutExtension + self.extensions[self.channeltype]) 265 | writerom(rom, filename) 266 | print('Got ROM: %s' % filename) 267 | self.extractsave(outputPath) 268 | return True 269 | 270 | return False 271 | 272 | def extractrom_tgcd(self, arc, outputPath, filenameWithoutExtension, id): 273 | if (arc.hasfile("config.ini")): 274 | extract_tgcd(arc, outputPath) 275 | print("Extracted TurboGrafx CD image") 276 | self.extractsave(outputPath) 277 | return True 278 | else: 279 | return False 280 | 281 | def extractrom_snes(self, arc, outputPath, filenameWithoutExtension, id): 282 | filename = os.path.join(outputPath, filenameWithoutExtension + self.extensions[self.channeltype]) 283 | extracted = False 284 | 285 | # try to find the original ROM first 286 | for f in arc.files: 287 | path = f.path.split('.') 288 | if len(path) == 2 and path[0].startswith('SN') and path[1].isdigit(): 289 | print('Found original ROM: %s' % f.path) 290 | rom = arc.getfile(f.path) 291 | writerom(rom, filename) 292 | print('Got ROM: %s' % filename) 293 | 294 | extracted = True 295 | 296 | # if original ROM not present, try to create a playable ROM by recreating and injecting the original sounds 297 | if not extracted: 298 | for f in arc.files: 299 | path = f.path.split('.') 300 | if len(path) == 2 and path[1] == 'rom': 301 | print("Recreating original ROM from %s" % f.path) 302 | vcrom = arc.getfile(f.path) 303 | if not vcrom: print("Error in reading ROM file %s" % f.path); return False 304 | 305 | # find raw PCM data 306 | pcm = None 307 | for f2 in arc.files: 308 | path2 = f2.path.split('.') 309 | if len(path2) == 2 and path2[1] == 'pcm': 310 | pcm = arc.getfile(f2.path) 311 | if not pcm: print('Error: PCM audio data not found'); return False 312 | 313 | '''# encode raw PCM in SNES BRR format 314 | print('Encoding audio as BRR') 315 | brr = BytesIO() 316 | enc = BRREncoder(pcm, brr) 317 | enc.encode() 318 | pcm.close()''' 319 | 320 | # inject BRR audio into the ROM 321 | print('Encoding and restoring BRR audio data to ROM') 322 | romdata = restore_brr_samples(vcrom, pcm) 323 | vcrom.close() 324 | pcm.close() 325 | 326 | # write the recreated ROM to disk 327 | f = open(filename, 'wb') 328 | f.write(romdata) 329 | f.close() 330 | print('Got ROM: %s' % filename) 331 | extracted = True 332 | 333 | # extract save data (but don't overwrite existing save data) 334 | if extracted: 335 | srm = filename[0:filename.rfind('.smc')] + '.srm' 336 | if os.path.lexists(srm): print('Not overwriting existing save data') 337 | elif self.extractsave(outputPath): print('Extracted save data to %s' % srm) 338 | else: print('Could not extract save data') 339 | 340 | return extracted 341 | 342 | 343 | def extractrom_neogeo(self, arc, outputPath, filenameWithoutExtension, id): 344 | foundRom = False 345 | for file in arc.files: 346 | #print(file.name) 347 | 348 | #arcFile = arc.getfile(file.path) 349 | #f = open(os.path.join(outputPath, 'arc_' + file.name),'wb') 350 | #f.write(arcFile.read()) 351 | #f.close() 352 | 353 | #PROBABLY. game.bin = not compressed, .z = zlib compressed, .xz = LZMA/XZ compressed 354 | #encryption may be applied on any file, and is applied after compression (decrypt before decompressing) 355 | 356 | if file.name == "game.bin" or file.name == "game.bin.z" or file.name == "game.bin.xz": 357 | 358 | rom = arc.getfile(file.path) 359 | rom.seek(0) 360 | entireFile = rom.read() 361 | 362 | giveUp = False 363 | 364 | if (entireFile[0:4] == b'CR00'): 365 | #f = open(os.path.join(outputPath, 'encrypted_' + file.name),'wb') 366 | #f.write(entireFile) 367 | #f.close() 368 | 369 | (success, output) = decrypt_neogeo(os.path.join(self.nand.path, 'title', '00010001', self.id, 'content'), self.id, entireFile) 370 | entireFile = output 371 | 372 | if not success: 373 | outputFileName = "game.bin.encrypted" 374 | print("Exporting encrypted ROMs without decrypting them") 375 | giveUp = True 376 | 377 | if not giveUp: 378 | if file.name == "game.bin.xz": 379 | assert entireFile[0:4] == b'\x5D\x00\x00\x80' 380 | print("Decompressing LZMA (XZ) file") 381 | decomp = lzma.LZMADecompressor(lzma.FORMAT_AUTO, None, None) 382 | entireFile = decomp.decompress(entireFile) 383 | 384 | elif file.name == "game.bin.z": 385 | assert entireFile[0] == 0x78 386 | print("Decompressing using zlib") 387 | entireFile = zlib.decompress(entireFile) 388 | 389 | elif file.name != "game.bin": 390 | #game.bin = unencrypted, uncompress game 391 | #any other file name: do nothing 392 | outputFileName = file.name 393 | giveUp = True 394 | 395 | convert_neogeo(BytesIO(entireFile), outputPath) 396 | print("Extracted ROM files (some BIOS files may be missing)") 397 | else: 398 | writerom(BytesIO(entireFile), os.path.join(outputPath, outputFileName)) 399 | print("Files extracted but further processing is required.") 400 | 401 | if self.extractsave(outputPath): 402 | print("Exported memory card with save file") 403 | else: 404 | print("No save data found") 405 | 406 | foundRom = True 407 | 408 | # This is just the contents of a formatted 2KB memory card without any saves on it. Probably useless to everyone. 409 | #elif file.name == "memcard.dat": 410 | # rom = arc.getfile(file.path) 411 | # print('Got default (empty) save data') 412 | # writerom(rom, os.path.join(outputFolderName, "memcard.empty.dat")) 413 | 414 | #This probably contains the DIP switch settings of the game, or maybe flags for the emulator 415 | #elif file.name == "config.dat": 416 | # rom = arc.getfile(file.path) 417 | # writerom(rom, os.path.join(outputFolderName, "config.dat")) 418 | 419 | #else: other files are probably useless 420 | # rom = arc.getfile(file.path) 421 | # writerom(rom, os.path.join(outputFolderName, file.name)) 422 | 423 | 424 | return foundRom 425 | 426 | def extractrom_arcade(self, appFilePath, outputPath, filenameWithoutExtension, id): 427 | foundRom = False 428 | 429 | #print("file in app:" + appFilePath) 430 | 431 | u8arc = self.tryGetU8Archive(appFilePath) 432 | if not u8arc: 433 | #print("It is NOT a U8 archive! First bytes:") 434 | inFile = open(appFilePath, 'rb') 435 | 436 | inFile.seek(0) 437 | if (inFile.read(1))[0] == 0x11: 438 | 439 | #print("The first byte is 11 so it is probably compressed LZ77") 440 | 441 | if id == '45353445': #ghosts n goblins 442 | arcade_extract_gng.export_roms_from_lzh_compressed_dol(inFile, outputPath) 443 | foundRom = True 444 | # else ignore the file 445 | #else: 446 | #print("The first byte is unknown, don't know what to do with this file") 447 | 448 | inFile.close() 449 | 450 | else: 451 | #print("It IS a U8 archive! Content:") 452 | 453 | #for file in u8arc.files: 454 | # print(file.path) 455 | 456 | if u8arc.hasfile('data.ccf'): 457 | ccf = CCFArchive(u8arc.getfile('data.ccf')) 458 | if ccf.hasfile('config'): 459 | foundRom = extract_arcade_ccf(ccf, outputPath) 460 | 461 | if foundRom: 462 | print("Got ROMs") 463 | 464 | return foundRom 465 | 466 | def tryGetU8Archive(self, path): 467 | try: 468 | u8arc = U8Archive(path) 469 | if not u8arc: 470 | return None 471 | else: 472 | return u8arc 473 | except AssertionError: 474 | return None 475 | 476 | def getsavefile(self, expectedFileName): 477 | datadir = os.path.join(self.nand.path, 'title', '00010001', self.id, 'data') 478 | datafiles = os.listdir(datadir) 479 | for filename in datafiles: 480 | path = os.path.join(datadir, filename) 481 | if filename == expectedFileName: 482 | return path 483 | 484 | return None 485 | 486 | # copy save file, doing any necessary conversions to common emulator formats 487 | def extractsave(self, outputPath): 488 | datadir = os.path.join(self.nand.path, 'title', '00010001', self.id, 'data') 489 | datafiles = os.listdir(datadir) 490 | 491 | for filename in datafiles: 492 | path = os.path.join(datadir, filename) 493 | if (self.channeltype == 'TurboGrafxCD' or self.channeltype == 'TurboGrafx16') and filename == 'pcengine.bup': 494 | unmangle_tgsave(path, outputPath) 495 | return True 496 | elif filename == 'savedata.bin': 497 | if self.channeltype == 'SNES': 498 | # VC SNES saves are standard SRM files 499 | outpath = os.path.join(outputPath, self.name + '.srm') 500 | shutil.copy2(path, outpath) 501 | return True 502 | #elif self.channeltype == 'NES': #not used because FDS games requires the app file 503 | #return convert_nes_save_data(path, self.name) 504 | elif self.channeltype == 'Genesis': 505 | # VC Genesis saves use a slightly different format from 506 | # the one used by Gens/GS and other emulators 507 | outpath = os.path.join(outputPath, self.name + '.srm') 508 | gensave.convert(path, outpath, True) 509 | return True 510 | elif self.channeltype == 'Master System': 511 | # VC Genesis saves use a slightly different format from 512 | # the one used by Gens/GS and other emulators 513 | outpath = os.path.join(outputPath, self.name + '.ssm') 514 | gensave.convert(path, outpath, False) 515 | return True 516 | elif filename == 'savefile.dat' and self.channeltype == 'Neo Geo': 517 | # VC Neo Geo saves are memory card images, can be opened as is by mame 518 | shutil.copy2(path, os.path.join(outputPath, "memorycard.bin")) 519 | return True 520 | elif filename.startswith('EEP_') or filename.startswith('RAM_'): 521 | assert self.channeltype == 'Nintendo 64' 522 | n64save.convert(path, os.path.join(outputPath, self.name)) 523 | return True 524 | 525 | return False 526 | 527 | def extractmanual(self, u8path, manualOutputPath): 528 | try: 529 | u8arc = U8Archive(u8path) 530 | if not u8arc: return False 531 | except AssertionError: 532 | return False 533 | 534 | man = None 535 | try: 536 | if u8arc.findfile('emanual.arc'): 537 | man = U8Archive(u8arc.getfile(u8arc.findfile('emanual.arc'))) 538 | elif u8arc.findfile('html.arc'): 539 | man = U8Archive(u8arc.getfile(u8arc.findfile('html.arc'))) 540 | elif u8arc.findfile('man.arc'): 541 | man = U8Archive(u8arc.getfile(u8arc.findfile('man.arc'))) 542 | elif u8arc.findfile('data.ccf'): 543 | ccf = CCFArchive(u8arc.getfile(u8arc.findfile('data.ccf'))) 544 | man = U8Archive(ccf.getfile('man.arc')) 545 | elif u8arc.findfile('htmlc.arc'): 546 | manc = u8arc.getfile(u8arc.findfile('htmlc.arc')) 547 | print('Decompressing manual: htmlc.arc') 548 | man = U8Archive(BytesIO(lz77.decompress_n64(manc))) 549 | elif u8arc.findfilebyregex('.+_manual_.+\\.arc\\.lz77$'): 550 | # E.g. makaimura_manual_usa.arc.lz77 (Arcade Ghosts n Goblins) 551 | manc = u8arc.getfile(u8arc.findfilebyregex('.+_manual_.+\\.arc\\.lz77$')) 552 | man = U8Archive(BytesIO(lz77.decompress_nonN64(manc))) 553 | manc.close() 554 | except AssertionError: pass 555 | 556 | if man: 557 | self.ensure_folder_exists(manualOutputPath) 558 | man.extract(manualOutputPath) 559 | print('Extracted manual to ' + manualOutputPath) 560 | return True 561 | 562 | return False 563 | 564 | class NandDump(object): 565 | # path: path on filesystem to the extracted NAND dump 566 | def __init__(self, path): 567 | self.path = path + '/' 568 | 569 | def scantickets(self): 570 | tickets = os.listdir(os.path.join(self.path, 'ticket', '00010001')) 571 | for ticket in tickets: 572 | id = ticket.rstrip('.tik') 573 | content = os.path.join('title', '00010001', id, 'content') 574 | title = os.path.join(content, 'title.tmd') 575 | if(os.path.exists(os.path.join(self.path, title))): 576 | appname = self.getappname(title) 577 | if not appname: continue 578 | #print(title, content + appname) 579 | name = self.gettitle(os.path.join(content, appname), id) 580 | channeltype = self.channeltype(ticket) 581 | if name and channeltype: 582 | print('%s: %s (ID: %s)' % (channeltype, name, id)) 583 | ext = RomExtractor(id, name, channeltype, self) 584 | ext.extract() 585 | print() 586 | 587 | # Returns a string denoting the channel type. Returns None if it's not a VC game. 588 | def channeltype(self, ticket): 589 | 590 | f = open(os.path.join(self.path, 'ticket', '00010001', ticket), 'rb') 591 | f.seek(0x1dc) 592 | thistype = struct.unpack('>I', f.read(4))[0] 593 | if thistype != 0x10001: return None 594 | f.seek(0x221) 595 | if struct.unpack('>B', f.read(1))[0] != 1: return None 596 | f.seek(0x1e0) 597 | ident = f.read(2) 598 | 599 | # TODO: support the commented game types 600 | # http://wiibrew.org/wiki/Title_database 601 | 602 | if ident[0] == ord('F'): return 'NES' 603 | elif ident[0] == ord('J'): return 'SNES' 604 | elif ident[0] == ord('L'): return 'Master System' 605 | elif ident[0] == ord('M'): return 'Genesis' 606 | elif ident[0] == ord('N'): return 'Nintendo 64' 607 | elif ident[0] == ord('P'): return 'TurboGrafx16' 608 | elif ident[0] == ord('E') and ident[1] == ord('A'): return 'Neo Geo' #E.g. Neo Turf Master 609 | elif ident[0] == ord('E') and ident[1] == ord('B'): return 'Neo Geo' #E.g. Spin Master, RFBB Special 610 | elif ident[0] == ord('E') and ident[1] == ord('C'): return 'Neo Geo' #E.g. Shock Troopers 2, NAM-1975 611 | elif ident[0] == ord('E'): return 'Arcade' #E.g. E5 = Ghosts'n' Goblins, E6 = Space Harrier 612 | elif ident[0] == ord('Q'): return 'TurboGrafxCD' 613 | #elif ident[0] == 'C': return 'Commodore 64' 614 | #elif ident[0] == 'X': return 'MSX' 615 | else: return None 616 | 617 | # Returns the path to the 00.app file containing the game's title 618 | # Precondition: the file denoted by "title" exists on the filesystem 619 | def getappname(self, title): 620 | f = open(os.path.join(self.path, title), 'rb') 621 | f.seek(0x1de) 622 | count = struct.unpack('>H', f.read(2))[0] 623 | f.seek(0x1e4) 624 | appname = None 625 | for i in range(count): 626 | info = struct.unpack('>IHHQ', f.read(16)) 627 | f.read(20) 628 | if info[1] == 0: 629 | appname = '%08x.app' % info[0] 630 | return appname 631 | 632 | # Gets title (in English) from a 00.app file 633 | def gettitle(self, path, defaultTitle): 634 | path = os.path.join(self.path, path) 635 | if not os.path.exists(path): return None 636 | f = open(path, 'rb') 637 | data = f.read() 638 | f.close() 639 | index = data.find(b'IMET') 640 | if index < 0: return None 641 | engindex = index + 29 + 84 642 | title = data[engindex:engindex+84] 643 | 644 | # Format the title properly 645 | title = title.strip(b'\0') 646 | while title.find(b'\0\0\0') >= 0: title = title.replace(b'\0\0\0', b'\0\0') 647 | title = title.replace(b'\0\0', b' - ') 648 | title = title.replace(b'\0', b'') 649 | title = title.replace(b':', b' - ') 650 | 651 | # Replace some characters 652 | title = re.sub(b'!\x60', b'I', title) # e.g. Ys Book I&II 653 | title = re.sub(b'!\x61', b'II', title) # e.g. Zelda II 654 | title = re.sub(b'!\x62', b'III', title) # e.g. Ninja Gaiden III 655 | title = re.sub(b' \x19', b'\'', title) # e.g. Indiana Jones' GA 656 | 657 | # Delete any characters that are not known to be safe 658 | title = re.sub(b'[^A-Za-z0-9\\-\\!\\_\\&\\\'\\. ]', b'', title) 659 | 660 | # more than one consequtive spaces --> one space 661 | while title.find(b' ') >= 0: title = title.replace(b' ', b' ') 662 | 663 | # Delete any mix of "." and space at beginning or end of string - they are valid in filenames, but not always as head or tail 664 | title = re.sub(b'(^[\\s.]*)|([\\s.]*$)', b'', title) 665 | 666 | # If we stripped everything (maybe can happen on japanese titles?), fall back to using defaultTitle 667 | if len(title) <= 0: 668 | title = defaultTitle 669 | 670 | return title.decode('ascii') 671 | 672 | if __name__ == '__main__': 673 | import sys 674 | nand = NandDump(sys.argv[1]) 675 | nand.scantickets() 676 | 677 | --------------------------------------------------------------------------------