.
675 |
--------------------------------------------------------------------------------
/README.md:
--------------------------------------------------------------------------------
1 | # mtk_fw_tools
2 |
3 | Set of tools to unpack/repack Mediatek firmware. Work in progress.
4 |
5 | The primary focus of developing these tools was to decompress the ALICE partition from Mediatek firmware blobs in order to inspect/reverse engineer the software on various devices (smart watches, GSM trackers, cheap phones, etc.).
6 |
7 | A typical firmware dump consists of:
8 | + Internal bootloader
9 | + External bootloader (EXT_BOOTLOADER)
10 | + Kernel (ROM)
11 | + User partition (VIVA header)
12 | + ZIMGE partition (ZIMAGE_ER - LZMA compressed resources)
13 | + BOOT_ZIMAGE partition (usually empty)
14 | + DCMCMP partition (LZMA compressed parts)
15 | + ALICE partition (ALICE_1/ALICE_2 - main firmware, compressed)
16 |
17 | The main (non-kernel) firmware of the device seems to lie in the ALICE partition which is range encoded and bitpacked ARM instructions. Until now this encoding was not public.
18 |
19 | Briefly, the encoder performs the following steps:
20 |
21 | 1. Read instructions from ALICE.bin (16-bits ARM Thumb)
22 | 2. Translate BL/BLX addresses
23 | 3. Add instructions, in order of appearance, to a binary tree
24 | 4. Generate dictionary (histogram)
25 | 5. Range encode instructions
26 | 6. Bitpack range encoded instructions
27 | 7. Generate mapping table (24-bit pointers to individual blocks), low byte unknown
28 | 8. Postprocess, prepend header, append mapping table and dictionary, etc
29 |
30 | The decoder must do the reverse. In short, we read the compressed data as a bit string, looking up each instruction in the dictionary to retrieve the uncompressed version as we proceed. The mapping table tells us when we have reached a block of $blocksize (typically 64 bytes, 32 instructions), at which point we skip to the next start-of-byte and proceed. Presumably this lets the firmware decode segments of the code as needed, without loading the entire image into memory.
31 |
32 | # Tools
33 |
34 | + alice.py - pack ALICE partition (not working 100% yet, use ALICE.exe instead)
35 | + unalice.py - unpack ALICE partition (working for ALICE_1, ALICE_2 partition types except for some minor issues)
36 |
37 | # Usage
38 |
39 | ## Requirements
40 | python (or python3)
41 | python-bitstring (or python3-bitstring)
42 |
43 | If you have the firmware of your device, open it in a hex editor and search for the ALICE_1 or ALICE_2 string.
44 |
45 |
46 | ...
47 | 0017FEB0 DF 5A 8A AC 22 D7 FE 6F 01 6E 98 E8 79 36 7C 50 .Z.."..o.n..y6|P
48 | 0017FEC0 82 5E 25 DD 26 B9 20 A1 67 17 F9 2D CD 54 EC 08 .^%.&. .g..-.T..
49 | 0017FED0 E0 A8 06 1D A7 88 DB 9C 61 92 6E 75 1B 3D 1D 00 ........a.nu.=..
50 | 0017FEE0 41 4C 49 43 45 5F 32 00 08 FF 17 10 D0 92 2C 10 ALICE_2.......,.
51 | 0017FEF0 0C 6A 2D 10 04 00 06 00 07 00 08 00 0A 00 0B 00 .j-.............
52 | 0017FF00 0C 00 09 01 40 00 FF FF E8 28 7D 15 2C 5B 2D 68 ....@....(}.,[-h
53 | 0017FF10 3F D5 F6 DF E0 BB 8C CA C2 D6 38 1E 05 33 9F CB ?.........8..3..
54 | 0017FF20 04 96 CC 6D 35 7E F0 F4 20 3C B8 46 0E 79 0E 27 ...m5~.. <.F.y.'
55 | 0017FF30 9D 87 A6 4E 78 E2 03 B0 42 82 16 0C CC 19 98 33 ...Nx...B......3
56 | ...
57 |
58 |
59 | Cut the section out using `dd` (0x17FEE0 == 1572576):
60 |
61 | ```
62 | $ dd if=firmware.bin of=ALICE bs=1 skip=1572576
63 | ```
64 |
65 | Run unalice on the resulting file:
66 |
67 | ```
68 | $ python3 unalice.py ALICE
69 | ```
70 |
71 | Load the resulting `alice-py.bin` into your favourite disassembler!
72 |
73 | If BL/BLX targets seem to not make sense in the disassembler, try using the `-t` option with `unalice.py`.
74 |
--------------------------------------------------------------------------------
/alice.py:
--------------------------------------------------------------------------------
1 | #!/usr/bin/python3
2 |
3 | '''
4 | Alice compress
5 |
6 | Encode and pack Mediatek ALICE.bin into compressed firmware component
7 |
8 | ALICE.exe encoder steps
9 | 1. Read instructions from ALICE.bin (16-bits ARM Thumb)
10 | 2. Translate BL/BLX addresses
11 | 3. Add instructions, in order of appearance, to binary tree
12 | 4. Generate dictionary (histogram)
13 | 5. Range encode instructions
14 | 6. Bitpack range encoded instructions
15 | 7. Generate mapping table (24-bit pointers to individual blocks), low byte unknown
16 | 8. Postprocess, prepend header, append mapping table and dictionary, etc
17 |
18 | ALICE is range encoded (range registers contained in header) and then
19 | bitpacked. Each packed instruction has a 3-bit prefix denoting the range from
20 | which it comes and thus implicitly its length.
21 |
22 | Sets of packed instructions are divided into blocks, the size of which is
23 | contained in the header. When we encounter the end of a block, we must pad
24 | with zeros until the next byte offset.
25 |
26 | Requirements:
27 | python3
28 |
29 | Copyright 2018 Donn Morrison donn.morrison@gmail.com
30 |
31 | TODO:
32 | - determine correct sorting of dictionary histogram
33 | - dynamically generate range registers
34 | - implement ALICE_1 encoding?
35 | - testing
36 |
37 | This program is free software: you can redistribute it and/or modify
38 | it under the terms of the GNU General Public License as published by
39 | the Free Software Foundation, either version 3 of the License, or
40 | (at your option) any later version.
41 |
42 | This program is distributed in the hope that it will be useful,
43 | but WITHOUT ANY WARRANTY; without even the implied warranty of
44 | MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
45 | GNU General Public License for more details.
46 |
47 | You should have received a copy of the GNU General Public License
48 | along with this program. If not, see .
49 | '''
50 |
51 | import sys
52 | import struct
53 | from collections import Counter
54 |
55 | def translate_bl_blx():
56 | global buff
57 | ptr = 0 # ptr can be equiv to PC
58 | bl_count = 0
59 | blx_count = 0
60 | while ptr < len(buff)/2-1:
61 | if (ptr+1) % 32 == 0:
62 | ptr += 1
63 | continue
64 |
65 | instr = buff[ptr*2] | (buff[ptr*2+1] << 8)
66 | instr2 = buff[ptr*2+2] | (buff[ptr*2+3] << 8)
67 |
68 | if (instr & 0xf800) == 0xf000:
69 | if (instr2 & 0xf800) == 0xf800: # bit 12 = 1, BL instruction
70 | upbits = 0xf800
71 | bl_count += 1
72 | elif (instr2 & 0xf800) == 0xe800: # bit 12 = 0, BLX instruction
73 | upbits = 0xe800
74 | blx_count += 1
75 | else:
76 | ptr += 1
77 | continue
78 |
79 | if instr & 0x400: # if J2 bit is set
80 | # shift imm11 left 11 bits, add lower bits from imm10 + sign
81 | # multiply by two, subtract 0x7ffffffe?
82 | v10 = 2 * (ptr + ((instr & 0x7ff) << 0x0b) + (instr2 & 0x7ff)) - 0x7ffffffe
83 | else:
84 | # shift imm11 left 11 bits, add lower bits from imm10 + sign
85 | # multiply by two, add 2
86 | v10 = 2 * (ptr + ((instr & 0x7ff) << 0x0b) + (instr2 & 0x7ff)) + 0x00000002
87 |
88 | # print("-translated type 0x%04x from 0x%08x to 0x%08x"%(upbits, ((instr & 0x7ff) << 0x0b) + (instr2 & 0x7ff), v10))
89 | # print("%d 0x%08x"%(ptr,((instr & 0x7ff) << 0x0b) + (instr2 & 0x7ff)))
90 |
91 | # reassemble branch target
92 | instr = (v10 >> 0x0c) & 0x7ff | 0xf000 # high bits
93 | instr2 = (v10 >> 1) & 0x7ff | upbits # low bits
94 |
95 | # print("-translated 0x%04x to 0x%04x at 0x%x, diff = %d"%(buff[ptr*2] | buff[ptr*2+1] << 8, instr, ptr*2, (buff[ptr*2] | buff[ptr*2+1] << 8) - instr))
96 | buff[ptr*2] = instr & 0xff
97 | buff[ptr*2+1] = (instr >> 8) & 0xff
98 | # print("-translated 0x%04x to 0x%04x at 0x%x, diff = %d"%(buff[ptr*2+2] | buff[ptr*2+3] << 8, instr2, ptr*2+2, (buff[ptr*2+2] | buff[ptr*2+3] << 8) - instr2))
99 | buff[ptr*2+2] = instr2 & 0xff
100 | buff[ptr*2+3] = (instr2 >> 8) & 0xff
101 |
102 | ptr += 1
103 | ptr += 1
104 |
105 | print("translated %d bl and %d blx instructions"%(bl_count*2, blx_count*2))
106 |
107 | def bitpack(length, instr):
108 | print("in bitpack 0x%02x 0x%08x"%(length, instr))
109 | global buff, length_remain, ptr, count
110 |
111 | instr_part = instr & 0xff
112 | if length != 0:
113 | if length > 8:
114 | bitpack(length - 8, instr >> 8)
115 | length = 8
116 |
117 | length_part = length - length_remain - 1
118 | print("length_part: %d = %d - %d - 1"%(length_part, length, length_remain))
119 |
120 | if length_part < 0:
121 | # shift left
122 | print("shifting 0x%02x << %d = 0x%02x"%(instr_part, -length_part, (instr_part << -length_part) & 0xff))
123 | print("%d b %02x | %02x -> %02x"%(ptr, buff[ptr], (instr_part << -length_part) & 0xff, (buff[ptr] | (instr_part << -length_part) & 0xff) & 0xff))
124 | buff[ptr] |= (instr_part << -length_part) & 0xff
125 | print("length_remain: %d = %d - %d"%(length_remain - length, length_remain, length))
126 | length_remain -= length
127 | elif length_part == 0:
128 | # no shift
129 | print("no shift 0x%02x"%(instr_part))
130 | print("%d a %02x | %02x -> %02x"%(ptr, buff[ptr], instr_part, (buff[ptr] | instr_part) & 0xff))
131 | buff[ptr] |= instr_part
132 | ptr += 1
133 | buff.append(0x00)
134 | print("bitpack 1st buff.append")
135 | length_remain = 7
136 | print("length_remain: 7")
137 | elif length_part > 0:
138 | # shift right
139 | print("shifting 0x%02x >> %d = 0x%02x"%(instr_part, length_part, (instr_part >> length_part) & 0xff))
140 | print("%d c %02x | %02x -> %02x"%(ptr, buff[ptr], (instr_part >> length_part) & 0xff, (buff[ptr] | (instr_part >> length_part)) & 0xff))
141 | buff[ptr] |= (instr_part >> length_part) & 0xff
142 | ptr += 1
143 | buff.append(0x00)
144 | print("bitpack 2nd buff.append")
145 | # shift left into new byte
146 | print("shifting 0x%02x << %d = 0x%02x"%(instr_part, 8 - length_part, instr_part << ((8 - length_part) & 0xff) & 0xff))
147 | print("%d d %02x | %02x -> %02x"%(ptr, buff[ptr], instr_part << (8 - length_part & 0xff) & 0xff, (buff[ptr] | instr_part << (8 - length_part & 0xff) & 0xff) & 0xff))
148 | buff[ptr] |= instr_part << ((8 - length_part) & 0xff) & 0xff
149 | length_remain = 7 - length_part
150 | print("length_remain: %d = 7 - %d"%(length_remain, length_part))
151 |
152 | # Read ALICE.bin
153 |
154 | f = open(sys.argv[1], "rb")
155 | buff = bytearray(f.read())
156 | f.close()
157 |
158 | # Translate BL and BLX instructions
159 | ptr = 0
160 | translate_bl_blx()
161 |
162 | f = open("translated-py.bin", "wb")
163 | f.write(buff)
164 | f.close()
165 |
166 | # Generate histogram
167 | # Need to make sure it is sorted in the same way ALICE.exe sorts, that being
168 | # that first by frequency, then for instructions in the same frequency bin,
169 | # possibly by instruction value, first location in ALICE.bin, or something
170 | # else? Most likely ALICE.exe gets the order from the way the BST is
171 | # traversed.
172 |
173 | # Construct fake ALICE.bin with a desired histogram and try to match
174 | # the output.
175 | instrs=[bytes(buff[i*2:i*2+2]) for i in range(int(len(buff)/2))]
176 | hist=Counter(instrs)
177 | shist=sorted(hist.items(), key=lambda x: (-x[1], x[0]))
178 | shist_f=[freq for instr,freq in shist]
179 | shist=[instr for instr,freq in shist]
180 |
181 | # Generate magic vector, funked instructions dictionary (before_encode.bin)
182 | range_regs = [0x0, 0x10, 0x50, 0xd0, 0x1d0, 0x3d0, 0xbd0, 0x1bd0]
183 | #range_regs = [0x0, 0x02, 0x06, 0x0a, 0x0e, 0x12, 0x16, 0x1e]
184 | codes = [0x07, 0x09, 0x0A, 0x0B, 0x0C, 0x0E, 0x0F, 0x13]
185 | starts = [0x0, 0x40, 0x100, 0x300, 0x800, 0x2800, 0x6000, 0x70000]
186 |
187 | magic = bytearray(0x10000)
188 | fshist = dict()
189 |
190 | for r in range(len(range_regs)-1):
191 | instrnr = 0
192 | for i in range(range_regs[r], range_regs[r+1]):
193 | instr_idx = struct.unpack(" 0 and length_remain != 0x07:
251 | length_remain = 0x07
252 | ptr += 1
253 | buff.append(0x00)
254 | print("buff.append")
255 | # Do some stuff here...
256 |
257 | # TODO Need to add some padding here
258 |
259 | f=open("alice-py", "wb")
260 | len = f.write(buff)
261 | print("wrote alice-py %d bytes"%len)
262 | f.close()
263 |
--------------------------------------------------------------------------------
/unalice.py:
--------------------------------------------------------------------------------
1 | #!/usr/bin/python3
2 |
3 | '''
4 | Alice decompress
5 |
6 | Unpack Mediatek compressed ALICE firmware component
7 |
8 | ALICE is range encoded (range registers contained in header) and then
9 | bitpacked. Each packed instruction has a 3-bit prefix denoting the range from
10 | which it comes and thus implicitly its length.
11 |
12 | Sets of packed instructions are divided into blocks, the size of which is
13 | contained in the header. When we encounter the end of a block, we must skip any
14 | remaining bits in the current byte and move to the next start-of-byte.
15 |
16 | As each range encoded instruction is unpacked, we look it up in the dictionary
17 | appended to the end of ALICE to reveal the original instruction.
18 |
19 | Requirements:
20 | python
21 | bitstring for python https://github.com/scott-griffiths/bitstring
22 |
23 | Copyright 2018 Donn Morrison donn.morrison@gmail.com
24 |
25 | TODO:
26 | - find correct EOF and stop decoding
27 |
28 | This program is free software: you can redistribute it and/or modify
29 | it under the terms of the GNU General Public License as published by
30 | the Free Software Foundation, either version 3 of the License, or
31 | (at your option) any later version.
32 |
33 | This program is distributed in the hope that it will be useful,
34 | but WITHOUT ANY WARRANTY; without even the implied warranty of
35 | MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
36 | GNU General Public License for more details.
37 |
38 | You should have received a copy of the GNU General Public License
39 | along with this program. If not, see .
40 | '''
41 |
42 | import math
43 | import os
44 | import sys
45 | import struct
46 | import getopt
47 | from bitstring import BitArray
48 |
49 | def bitunpack():
50 | global fout, instrdict, range_regs, bitbuff, mappings, alicebin, blocksize
51 | bitptr = 0
52 | numblocks = 0
53 | byteswritten = 0
54 | lastblockbitptr = 0
55 | matchedblocks = 0
56 | lastblock = False
57 | lastinstr = None
58 |
59 | # Range register dependent vectors:
60 | # starts 000, 001, ..., 111
61 | # prefixes are starts shifted by respective range reg
62 | # lengths are range encoded instruction lengths + starts
63 | # range_regs_pow dictionary boundaries (ranges)
64 | starts = range(0,8) # each 3 bits long
65 | prefixes = [start << r for start,r in zip(starts, range_regs)]
66 | lengths = [r + 3 for r in range_regs]
67 | range_regs_pow = [0] + [int(math.pow(2, r)) for r in range_regs[0:-1]]
68 |
69 | while bitptr < ((mappings[-1])[0] + mappings[-1][1])*8 and bitptr < len(bitbuff): # mapping table addr + len
70 | # Check if we've done a block
71 | if blocksize != 0 and (byteswritten % blocksize) == 0:
72 | # print("--- hit a block at %d bytes written, compressed index 0x%08x, rem %d, %d"%(byteswritten, int(bitptr/8),bitptr%8, bitptr))
73 | sys.stdout.flush()
74 | # FFW to the next byte offset
75 | if bitptr%8 != 0:
76 | bitptr = bitptr + (8 - (bitptr%8))
77 |
78 | # Check every even block against mapping table
79 | if numblocks%2 == 0:
80 | # print("--- corresponding maptable addr 0x%08x len %d"%(mappings[int(numblocks/2)][0], mappings[int(numblocks/2)][1]))
81 | if int(bitptr/8) in [m for m,l in mappings]:
82 | matchedblocks += 1
83 | # print(" --- bitptr in mapping table! matchedblocks = %d"%(matchedblocks))
84 | # else:
85 | # print(" --- bitptr 0x%08x NOT in mapping table!"%(int(bitptr/8)))
86 | # print("--- ptr forwarded to next byte %d, numblocks = %d"%(bitptr, numblocks))
87 | # print("--- distance to start of previous block %d bits (%d bytes)"%(bitptr-lastblockbitptr,int((bitptr-lastblockbitptr)/8)))
88 | lastblockbitptr = bitptr
89 | numblocks += 1
90 | #if math.ceil(numblocks/2) == len(mappings) -1:
91 | # print("--- at pos %02f"%((numblocks/2) / float(len(mappings)-1)))
92 |
93 | # FIXME EOF detection is a hack based on observations.
94 | # We first check if we're near the end of the compressed region,
95 | # then lookahead for low 1 counts in the bit buffer, or observed
96 | # EOF instruction sequence (0xeaff, 0x0000)
97 | if (numblocks/2) / float(len(mappings)-1) > 0.999: # Somewhere near the end?
98 | # We've reached possibly the last complete block
99 | print("--- Possible last block, now scanning for EOF signature instructions")
100 | lastblock = True
101 | # print("next 64 bits: %s"%(format(bitbuff[bitptr:bitptr+64].uint, '#066b')))
102 | # print("contains %d ones"%(bin(bitbuff[bitptr:bitptr+64].uint)[2:].count('1')))
103 |
104 | # Look for instruction header
105 | for s,l in zip(starts,lengths):
106 | if bitbuff[bitptr:bitptr+3].uint == s:
107 | # Fetch the range encoded instruction
108 | instr = bitbuff[bitptr:bitptr+l].uint
109 | # print("%d (0x%x,%d): fetched instruction 0x%08x and prefix 0x%x, length %d"%(bitptr, int(bitptr/8), bitptr%8, instr, prefixes[starts.index(s)], l))
110 |
111 | # FIXME EOF detection is a hack based on observations.
112 | # We first check if we're near the end of the compressed region,
113 | # then lookahead for low 1 counts in the bit buffer, or observed
114 | # EOF instruction sequence (0xeaff, 0x0000)
115 | if (lastblock and bin(bitbuff[bitptr:bitptr+64].uint)[2:].count('1') < 2):
116 | print("--- Last block, mostly zero bits left (< 2 of 64). Stopping.")
117 | return
118 | if (lastblock and instr == 1 and s == 0 and lastinstr == 0xeaff): # If we're left with mostly zeros, probably at end
119 | print("--- Last block, end instructions detected (0xeaff, 0x0000). Stopping.")
120 | return
121 | # If encoded, look up in dictionary
122 | if s != 0x7:
123 | # Find the range (have to sum previous ranges to get correct index)
124 | low = sum(range_regs_pow[0:starts.index(s)+1])
125 | # Subtract the instruction prefix
126 | instridx = instr - prefixes[starts.index(s)]
127 | # This is the index into the range_reg subrange
128 | originstr = instrdict[low + instridx]
129 | else:
130 | # Not encoded, simply extract the instruction
131 | originstr = instr & 0xffff
132 | # print("original instruction 0x%04x written at 0x%08x"%(originstr,byteswritten))
133 | lastinstr = originstr
134 | decomp = struct.pack("> 0x0b) & 0x7ff | 0xf000 # high bits
182 | instr2 = v10 & 0x7ff | upbits # low bits
183 |
184 | # print("-translated 0x%04x to 0x%04x at 0x%x, diff = %d"%(buff[ptr*2] | buff[ptr*2+1] << 8, instr, ptr*2, (buff[ptr*2] | buff[ptr*2+1] << 8) - instr))
185 | buff[ptr*2] = instr & 0xff
186 | buff[ptr*2+1] = (instr >> 8) & 0xff
187 | # print("-translated 0x%04x to 0x%04x at 0x%x, diff = %d"%(buff[ptr*2+2] | buff[ptr*2+3] << 8, instr2, ptr*2+2, (buff[ptr*2+2] | buff[ptr*2+3] << 8) - instr2))
188 | buff[ptr*2+2] = instr2 & 0xff
189 | buff[ptr*2+3] = (instr2 >> 8) & 0xff
190 |
191 | ptr += 1
192 | ptr += 1
193 |
194 | print("translated %d bl and %d blx instructions"%(bl_count*2, blx_count*2))
195 |
196 | # ALICE
197 | # -t ALICE
198 |
199 | notranslate = 0
200 | if len(sys.argv) == 3 and sys.argv[1] == "-t":
201 | alicefile = sys.argv[2]
202 | notranslate = 1
203 | elif len(sys.argv) == 2 and sys.argv[1] != "-t":
204 | alicefile = sys.argv[1]
205 | elif len(sys.argv) < 2 or len(sys.argv) > 3:
206 | print("usage: unalice.py [-t] ")
207 | print(" -t disable bl/blx addr translation (required for some images)")
208 | sys.exit()
209 |
210 | f = open(alicefile, "rb")
211 | magic = f.read(7)
212 | if magic == b'ALICE_1':
213 | alice_version = 1
214 | elif magic == b'ALICE_2':
215 | alice_version = 2
216 | else:
217 | print("found %s, expected ALICE_2, quitting."%(magic))
218 | sys.exit(1)
219 | print("found %s magic"%(magic))
220 |
221 | header_size = 40 # ALICE_2 or ALICE_1 with full header
222 | if alice_version == 1:
223 | f.seek(36) # Check ALICE_1
224 | endbytes = f.read(4)
225 | if endbytes != b'\x00\x00\xff\xff':
226 | header_size = 36 # ALICE_1 with short header
227 |
228 | f.seek(8)
229 | base, mapping_offset, dict_offset = struct.unpack(">26) + (3*int(blocksize/2) >> 3)) + 1 # FIXME hardcoded 26
273 | print("mapping entry 0x%08x addr 0x%08x len %d"%(mapping, addr, length))
274 | mappings.append((addr, length))
275 | reads += 4
276 |
277 | print("mappings length: %d"%(len(mappings)))
278 |
279 | #mappingsbits = [a + (((b>>0x1a) + 3*(blocksize/2) >> 3) + 1) for a,b in mappings]
280 | #print(mappingsbits)
281 |
282 | print("last nonzero mapping: 0x%08x, len = %d"%(mappings[-2][0], mappings[-2][1]))
283 |
284 | reads = 0
285 | instrdict = []
286 | while reads < filesize - dict_offset:
287 | instr = struct.unpack("