├── .gitignore ├── .travis.yml ├── MANIFEST.in ├── README.rst ├── lz4tools ├── __init__.py ├── __main__.py ├── lz4file.py └── lz4tar.py ├── lz4toolsCli ├── setup.py ├── src ├── lz4.c ├── lz4.h ├── lz4frame.c ├── lz4frame.h ├── lz4frame_static.h ├── lz4hc.c ├── lz4hc.h ├── python-lz4f.c ├── python-lz4f.h ├── xxhash.c └── xxhash.h └── tests └── test.py /.gitignore: -------------------------------------------------------------------------------- 1 | # Build objects 2 | nose* 3 | build/ 4 | dist/ 5 | *egg-info/ 6 | *.so 7 | *.pyc 8 | *__pycache__/ 9 | # Edit objects 10 | *.swp 11 | -------------------------------------------------------------------------------- /.travis.yml: -------------------------------------------------------------------------------- 1 | arch: 2 | - amd64 3 | - ppc64le 4 | language: python 5 | python: 6 | # - 2.6 7 | - 2.7 8 | - 3.3 9 | - 3.4 10 | matrix: 11 | allow_failures: 12 | - python: 3.3 #EOL 13 | install: 14 | script: python setup.py test 15 | os: 16 | - linux 17 | # - osx 18 | -------------------------------------------------------------------------------- /MANIFEST.in: -------------------------------------------------------------------------------- 1 | include README.rst 2 | include src/*.h 3 | -------------------------------------------------------------------------------- /README.rst: -------------------------------------------------------------------------------- 1 | ================== 2 | Lz4tools and Lz4f 3 | ================== 4 | 5 | .. image:: https://travis-ci.org/darkdragn/lz4tools.svg?branch=master 6 | 7 | Overview 8 | -------- 9 | This package consists of three parts: 10 | 11 | 1. lz4f - C-Module containing python bindings for all lz4frame functions. 12 | 2. lz4tools - a zipfile-like file wrapper class and tarfile-like class for lz4 compressed files. 13 | 3. lz4toolsCli - a quick cli for using lz4tools static functions. 14 | 15 | Before going any further, I recommend reading up on lz4 at: 16 | http://lz4.github.io/lz4/ 17 | 18 | It is an awesome compression algorithm and I can't thank Yann Collet enough for 19 | putting together the C implementation and lz4frame. 20 | 21 | Usage 22 | ----- 23 | Cli: 24 | New addition, there is now a simple Cli for anyone wishing these 25 | capabilities would exist directly on the command line. 26 | 27 | .. code:: bash 28 | 29 | usage: lz4toolsCli [-h] [-f] [-t] [-d] [-i] [-bs {4,5,6,7}] [-bm {0,1}] 30 | input [output] 31 | 32 | positional arguments: 33 | input The targeted input. 34 | output Optional output target. 35 | 36 | optional arguments: 37 | -h, --help show this help message and exit 38 | -f Compress file. Default action if input is a file. 39 | -t Compress directory to .tar.lz4. Default action if input is a 40 | directory 41 | -d Decompress file. Default action if the file ends in .lz4. 42 | -i Print frame information from the file's header. 43 | -bs {4,5,6,7} Specify blkSizeId. Valid values are 4-7. Default value is 7. 44 | -bm {0,1} Specify blkMode. 0 = Chained blocks. 1 = Independent blocks 45 | Default value is 0. 46 | 47 | .. 48 | 49 | C-Module / Bindings: 50 | I would recommend against using lz4f directly except in debug/testing 51 | situations. If necessary, a compress or decompress operation first needs a 52 | context that will be used with all lz4f functions: 53 | 54 | **lz4f compression the hard way:** 55 | 56 | .. code:: python 57 | 58 | >>> import lz4f 59 | >>> inputFile = open('fileIn', 'rb') 60 | >>> cCtx = lz4f.createCompContext 61 | >>> header = lz4f.compressBegin(cCtx) 62 | >>> data = lz4f.compressUpdate(inputFile.read(), cCtx) 63 | >>> end = lz4f.compressEnd(cCtx) 64 | >>> with open('output.lz4', 'wb') as out: 65 | ... out.write(header) 66 | ... out.write(data) 67 | ... out.write(end) 68 | ... out.flush() 69 | ... out.close() 70 | >>> lz4f.freeCompContext(cCtx) 71 | >>> inputFile.close() 72 | >>> del header, data, end 73 | 74 | **lz4f compression the easy way:** 75 | 76 | .. code:: python 77 | 78 | >>> import lz4f 79 | >>> with open('output.lz4', 'wb') as out: 80 | ... with open('fileIn', 'rb') as inFile: 81 | ... out.write(lz4f.compressFrame(inFile.read()) 82 | ... out.flush() 83 | ... out.close() 84 | 85 | **Advantages and disadvantages:** 86 | The easy way takes more ram. It reads the 87 | contents of the file into a buffer, passes it and compresses it all in one 88 | go. With the hard way you can have it read as little or as much as you 89 | like. For instance, you can break up the input into 64kiB chunks. 90 | Each chunk could be read, compressed and dropped to disk to conserve ram. 91 | 92 | ---- 93 | 94 | Lz4Tools Module: 95 | The lz4file class is currently read only. Right now it is a bit rough around 96 | the edges, however over the next couple of weeks, I will finish adding some 97 | docstrings, and such to make it more user friendly. As soon as I get a chance 98 | I will make it write capable. The easiest way to use it is with either the 99 | open or openTar methods. That's right! There is a lz4Tar class in the module 100 | that is a subclass of tarfile. 101 | 102 | **lz4tools tar example:** 103 | 104 | .. code:: 105 | 106 | >>> import lz4tools 107 | >>> lz4tools.compressTarDefault('src') 108 | >>> testTar = lz4tools.openTar('src.tar.lz4') 109 | >>> testTar.list() 110 | -rwxr-xr-x darkdragn/darkdragn 0 2014-10-02 23:06:09 src/ 111 | -rw-r--r-- darkdragn/darkdragn 29905 2014-09-16 18:29:45 src/lz4hc.c 112 | -rw-r--r-- darkdragn/darkdragn 6781 2014-09-16 18:29:45 src/ xxhash.h 113 | -rw-r--r-- darkdragn/darkdragn 25662 2014-09-16 18:29:45 src/ xxhash.c 114 | -rw-rw-r-- darkdragn/darkdragn 13894 2014-10-02 20:22:09 src/lz4frame.h 115 | -rw-rw-r-- darkdragn/darkdragn 46241 2014-10-02 20:22:09 src/lz4.c 116 | -rw-r--r-- darkdragn/darkdragn 8832 2014-09-16 18:29:45 src/lz4hc.h 117 | -rw-rw-r-- darkdragn/darkdragn 11734 2014-10-02 23:06:08 src/python-lz4f.c 118 | -rw-rw-r-- darkdragn/darkdragn 2554 2014-10-02 20:22:09 src/python-lz4f.h 119 | -rw-r--r-- darkdragn/darkdragn 14882 2014-09-18 01:28:06 src/lz4.h 120 | -rw-rw-r-- darkdragn/darkdragn 50141 2014-10-02 23:04:05 src/lz4frame.c 121 | 122 | 123 | **lz4tools file example:** 124 | 125 | .. code:: python 126 | 127 | >>> import lz4tools 128 | >>> lz4tools.compressFileDefault('setup.py') 129 | >>> testFile = lz4tools.open('setup.py.lz4') 130 | >>> testFile.blkDict 131 | {0: {'decomp_e': 1445, 'compressed_begin': 7, 'blkSize': 923}} 132 | >>> testFile.seek(1002) 133 | >>> print testFile.read() 134 | test_suite = "nose.collector", 135 | keywords = ['lz4', 'lz4frame', 'lz4file', 'lz4tar'], 136 | classifiers=[ 137 | 'Development Status :: 5 - Production/Stable', 138 | 'License :: OSI Approved :: BSD License', 139 | 'Intended Audience :: Developers', 140 | 'Programming Language :: C', 141 | 'Programming Language :: Python', 142 | 'Programming Language :: Python :: 2.6', 143 | 'Programming Language :: Python :: 2.7', 144 | ], 145 | ) 146 | 147 | And thus ends the brief tutorial. 148 | 149 | Notes 150 | _____ 151 | Version: 152 | The first two digits of the version will always correspond with the version 153 | of lz4 that is included. Current version is r124, thus 1.2. The next digit 154 | is correspond to milestone improvements. Example: Once lz4file supports write. 155 | The last digit will be slight improvements. Usually some contextual error, 156 | or syntax error. Perhaps even a quick fix for python3.4, since I don't use it 157 | often, if an issue is brought to my attention, I will provide a quick fix 158 | as quickly as possible. 159 | -------------------------------------------------------------------------------- /lz4tools/__init__.py: -------------------------------------------------------------------------------- 1 | import lz4f 2 | import os 3 | import sys 4 | 5 | if sys.version_info.major >= 3: 6 | import builtins as __builtin__ 7 | from .lz4file import Lz4File 8 | from .lz4tar import Lz4Tar 9 | from io import BytesIO as StringIO 10 | else: 11 | import __builtin__ 12 | from lz4file import Lz4File 13 | from lz4tar import Lz4Tar 14 | from StringIO import StringIO 15 | 16 | 17 | __all__ = ['lz4file', 'lz4tar'] 18 | 19 | 20 | def compressFileDefault(name, overwrite=False, outname=None, prefs=None): 21 | """ 22 | :type string: name - name of file to compress 23 | :type bool: overwrite - overwrite destination 24 | :type string: outname - name for compressed file, not required. 25 | Default will be '.'.join([name, 'lz4']) 26 | Generic compress method for a file. Adds .lz4 to original file name for 27 | output, unless outname is provided. 28 | 29 | ***NOTE*** No longer uses compressFrame. This is now large file safe! 30 | It will now read the input in 64Kb chunks. 31 | """ 32 | if not outname: 33 | outname = '.'.join([name, 'lz4']) 34 | if os.path.exists(outname): 35 | if not overwrite: 36 | print('File Exists!') 37 | return 38 | print('Overwrite authorized') 39 | if not os.path.exists(name): 40 | print('Unable to locate the original file. Please check filename.') 41 | return 42 | cCtx = lz4f.createCompContext() 43 | header = lz4f.compressBegin(cCtx, prefs) 44 | with __builtin__.open(outname, 'wb') as out: 45 | out.write(header) 46 | with __builtin__.open(name, 'rb') as infile: 47 | while True: 48 | decompData = infile.read((64*(1 << 10))) 49 | if not decompData: 50 | break 51 | compData = lz4f.compressUpdate(decompData, cCtx) 52 | out.write(compData) 53 | out.write(lz4f.compressEnd(cCtx)) 54 | out.flush() 55 | out.close() 56 | lz4f.freeCompContext(cCtx) 57 | 58 | 59 | def compressTarDefault(name, overwrite=None, outname=None, prefs=None): 60 | """ 61 | :type string: dirName - the name of the dir to tar 62 | :type bool: overwrite - overwrite destination 63 | Generic compress method for creating .tar.lz4 from a dir. 64 | 65 | ***WARNING*** Currently uses StringIO object until lz4file supports write. 66 | Avoid using for large directories, it will consume quite a bit of RAM. 67 | """ 68 | if not outname: 69 | outname = '.'.join([name.rstrip('/'), 'tar', 'lz4']) 70 | if not os.path.exists(name): 71 | print('Unable to locate the directory to compress.') 72 | return 73 | buff = StringIO() 74 | tarbuff = Lz4Tar.open(fileobj=buff, mode='w') 75 | tarbuff.add(name) 76 | tarbuff.close() 77 | buff.seek(0) 78 | cCtx = lz4f.createCompContext() 79 | header = lz4f.compressBegin(cCtx, prefs) 80 | with __builtin__.open(outname, 'wb') as out: 81 | out.write(header) 82 | while True: 83 | decompData = buff.read((64*(1 << 10))) 84 | if not decompData: 85 | break 86 | compData = lz4f.compressUpdate(decompData, cCtx) 87 | out.write(compData) 88 | out.write(lz4f.compressEnd(cCtx)) 89 | out.flush() 90 | lz4f.freeCompContext(cCtx) 91 | del tarbuff, buff 92 | 93 | 94 | def decompressFileDefault(name, overwrite=False, outname=None): 95 | """ 96 | :type string: name - name of file to decompress 97 | :type bool: overwrite - overwrite destination 98 | :type string: outname - name for decompressed file, not required. 99 | Default will be '.'.join([name, 'lz4']) 100 | Generic decompress method for a file. Removes .lz4 to original file name 101 | for output, unless outname is provided. 102 | 103 | ***WARNING*** Currently uses lz4f.compressFrame, which will read the entire 104 | original file into memory, then pass to c-module for compression. Avoid 105 | using this for large files until migrated to advCompress functions. 106 | """ 107 | if not outname: 108 | outname = name.replace('.lz4', '') 109 | if outname == name: 110 | print(''.join(['File does not contain .lz4 extension. ', 111 | 'Please provide outname.'])) 112 | return 113 | if os.path.exists(outname) and not overwrite: 114 | print(''.join(['Output file exists! Please authorize overwrite or', 115 | ' specify a different outfile name.'])) 116 | infile = Lz4File.open(name) 117 | infile.decompress(outname) 118 | 119 | 120 | def getFileInfo(name): 121 | """ 122 | :type string: name - name of file to examine 123 | Returns a dict object containing the file's header information. 124 | """ 125 | if not os.path.exists(name): 126 | print('Unable to locate the file') 127 | return 128 | dCtx = lz4f.createDecompContext() 129 | with __builtin__.open(name, 'rb') as inFile: 130 | header = inFile.read(7) 131 | return lz4f.getFrameInfo(header, dCtx) 132 | 133 | 134 | def open(name=None, fileObj=None): 135 | """ Alias for Lz4File.open() """ 136 | return Lz4File.open(name, fileObj) 137 | 138 | 139 | def openTar(name=None, fileObj=None): 140 | """ Alias for Lz4Tar.open() """ 141 | return Lz4Tar.lz4open(name, 'r', fileObj) 142 | -------------------------------------------------------------------------------- /lz4tools/__main__.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | 3 | import argparse 4 | import os 5 | import sys 6 | 7 | try: 8 | import lz4tools 9 | except ImportError: 10 | import __init__ as lz4tools 11 | 12 | 13 | def compFile(): 14 | return lz4tools.compressFileDefault(res.input, outname=res.output) 15 | 16 | 17 | def compDir(): 18 | return lz4tools.compressTarDefault(res.input, outname=res.output) 19 | 20 | 21 | def decompFile(): 22 | return lz4tools.decompressFileDefault(res.input, outname=res.output) 23 | 24 | 25 | def outErr(): 26 | return sys.stdout.write(''.join(['Please specify only ony of the ', 27 | 'comp/decomp options'])) 28 | 29 | parser = argparse.ArgumentParser() 30 | 31 | parser.add_argument('-f', action='store_true', dest='file', default=False, 32 | help='Compress file. Default action if input is a file.') 33 | parser.add_argument('-t', action='store_true', dest='tar', default=False, 34 | help=''.join(['Compress directory to .tar.lz4. ', 35 | 'Default action if input is a directory'])) 36 | parser.add_argument('-d', action='store_true', dest='decomp', default=False, 37 | help=''.join(['Decompress file. Default action if the', 38 | 'file ends in .lz4.'])) 39 | parser.add_argument('-i', action='store_true', dest='info', default=False, 40 | help=''.join(['Print frame information from the file\'s ', 41 | 'header.'])) 42 | parser.add_argument('-bs', action='store', dest='blkSizeId', default=7, 43 | type=int, help=''.join(['Specify blkSizeId. Valid values', 44 | 'are 4-7. Default value is 7.']), 45 | choices=range(4, 8)) 46 | parser.add_argument('-bm', action='store', dest='blkMode', default=0, type=int, 47 | help=''.join(['Specify blkMode. 0 = Chained blocks. ', 48 | '1 = Independent blocks Default value is', 49 | '0.']), choices=[0, 1]) 50 | parser.add_argument('input', action='store', help='The targeted input.') 51 | parser.add_argument('output', action='store', nargs='?', default=None, 52 | help='Optional output target.') 53 | if len(sys.argv) == 1: 54 | parser.print_help() 55 | sys.exit() 56 | res = parser.parse_args() 57 | 58 | prefs = lz4tools.lz4f.makePrefs(res.blkSizeId, res.blkMode) 59 | kwargs = {'name': res.input, 'outname': res.output, 'prefs': prefs} 60 | 61 | 62 | def compFile(): 63 | return lz4tools.compressFileDefault(**kwargs) 64 | 65 | 66 | def compDir(): 67 | return lz4tools.compressTarDefault(**kwargs) 68 | 69 | 70 | def decompFile(): 71 | return lz4tools.decompressFileDefault(res.input, outname=res.output) 72 | 73 | 74 | def getInfo(): 75 | return lz4tools.getFileInfo(res.input) 76 | 77 | 78 | def outErr(): 79 | return sys.stdout.write(''.join(['Please specify only ony of the ', 80 | 'comp/decomp options'])) 81 | 82 | 83 | if res.info: 84 | print(getInfo()) 85 | elif res.file: 86 | if res.tar or res.decomp: 87 | outErr() 88 | compFile() 89 | elif res.tar: 90 | if res.decomp: 91 | outErr() 92 | compDir() 93 | elif res.decomp: 94 | decompFile() 95 | elif res.input.endswith('lz4'): 96 | decompFile() 97 | elif os.path.isfile(res.input): 98 | compFile() 99 | elif os.path.isdir(res.input): 100 | compDir() 101 | else: 102 | parser.print_help() 103 | -------------------------------------------------------------------------------- /lz4tools/lz4file.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/python 2 | 3 | import binascii 4 | import lz4f 5 | import struct 6 | import sys 7 | 8 | if sys.version_info.major >= 3: 9 | from builtins import open as _open 10 | else: 11 | from __builtin__ import open as _open 12 | 13 | 14 | class Lz4File: 15 | def __init__(self, name, fileObj=None, seekable=True): 16 | parseMagic = lambda x: binascii.hexlify(x[:4]) 17 | self.name = name 18 | if fileObj: 19 | self.fileObj = fileObj 20 | self.compEnd = self.tell_end() 21 | else: 22 | self.fileObj = _open(name) 23 | self.compEnd = self.tell_end() 24 | self.header = fileObj.read(7) 25 | if parseMagic(self.header) == b'04224d18': 26 | self.dCtx = lz4f.createDecompContext() 27 | self.fileInfo = lz4f.getFrameInfo(self.header, self.dCtx) 28 | self.blkSizeID = self.fileInfo.get('blkSize') 29 | else: 30 | raise IOError 31 | if seekable: 32 | try: 33 | self.load_blocks() 34 | except: 35 | print('Unable to load blockDict. Possibly not a lz4 file.') 36 | raise IOError 37 | 38 | @classmethod 39 | def open(cls, name=None, fileObj=None, seekable=True): 40 | if not name and not fileObj: 41 | sys.stderr.write('Nothing to open!') 42 | if not fileObj: 43 | fileObj = _open(name, 'rb') 44 | return cls(name, fileObj, seekable) 45 | 46 | def close(self): 47 | self.fileObj.close() 48 | 49 | def decompress(self, outName): 50 | """ 51 | :type string: outName 52 | Generic decompress function. Will decompress the entire file to 53 | outName. 54 | """ 55 | writeOut = _open(outName, 'wb') 56 | for blk in self.blkDict.values(): 57 | out = self.read_block(blk=blk) 58 | writeOut.write(out) 59 | writeOut.flush() 60 | writeOut.close() 61 | 62 | def get_block_size(self): 63 | """ 64 | :type file: fileObj - At least seeked to 7 (past header) 65 | Static method to determine next block's blockSize. 66 | """ 67 | size = struct.unpack(' 0: 83 | data = self.read_block(blkSize, setCur=False) 84 | total += len(data) 85 | self.blkDict.update({blkNum: {'comp_begin': pos, 86 | 'decomp_e': total, 'blkSize': blkSize}}) 87 | blkNum += 1 88 | if not self.fileObj.tell() == self.compEnd: 89 | blkSize = self.get_block_size() 90 | else: 91 | break 92 | pos = self.fileObj.tell() 93 | self.end = total-1 94 | del data, total 95 | self.curBlk = 0 96 | self.decomp = self.read_block(blk=self.blkDict.get(0)) 97 | self.seek(0) 98 | 99 | def read(self, size=None): 100 | """ 101 | :type int: size 102 | File read-like function. If passed a size, it only reads those bytes. 103 | If not passed a size, it reads the entire file, from the current 104 | position. 105 | """ 106 | out = bytes() 107 | decompOld = self.decompPos 108 | if size == 0: 109 | return '' 110 | if self.pos == self.end: 111 | return "" 112 | if not size or self.pos+size > self.end: 113 | size = self.end-self.pos 114 | newPos = self.pos+size 115 | if self.decompPos+size > -1: 116 | out += self.decomp[decompOld:] 117 | size = size - len(out) 118 | self.pos = self.curBlkData.get('decomp_e') 119 | self.curBlk += 1 120 | self.decomp = self.read_block(blk=self.curBlkData, setCur=False) 121 | out += self.read(size) 122 | else: 123 | out += self.decomp[decompOld:decompOld+size] 124 | self.pos = newPos 125 | return out 126 | return out 127 | 128 | def read_block(self, blkSize=None, blk=None, setCur=True): 129 | """ 130 | :type int: blkSize - returned from get_block_size() 131 | :type dict: blk - entry from blkDict 132 | :type bool: setCur - update current blk var 133 | 134 | Reads the next block, unless provided a blk from blkDict. If provided 135 | a blk, it will read that specific block. 136 | """ 137 | if blk: 138 | self.fileObj.seek(blk.get('comp_begin')) 139 | blkSize = blk.get('blkSize') 140 | if not blkSize: 141 | blkSize = self.get_block_size() 142 | if blkSize == 0: 143 | return '' 144 | if setCur: 145 | try: 146 | iteritems = self.blkDict.iteritems 147 | except AttributeError: 148 | iteritems = self.blkDict.items 149 | self.curBlk = [num for num, b in iteritems() 150 | if self.fileObj.tell() == b.get('comp_begin')][0] 151 | if (self.fileObj.tell() + blkSize + 8) == self.compEnd: 152 | blkSize += 8 153 | regen = True 154 | compData = self.fileObj.read(blkSize) 155 | resultDict = lz4f.decompressFrame(compData, self.dCtx, self.blkSizeID) 156 | if 'regen' in locals(): 157 | self._regenDCTX() 158 | return resultDict.get('decomp') 159 | 160 | def seek(self, offset, whence=0): 161 | """ 162 | :type int: offset 163 | File seek-like function. Accepts offset. Whence for future improvement, 164 | but not yet implemented. 165 | """ 166 | thisBlk = int() 167 | if not offset: 168 | blk = self.blkDict.get(0) 169 | else: 170 | try: 171 | iteritems = self.blkDict.iteritems 172 | except AttributeError: 173 | iteritems = self.blkDict.items 174 | thisBlk, blk = [[num, b] for num, b in iteritems() 175 | if offset < b.get('decomp_e')][0] 176 | if self.curBlk == thisBlk: 177 | self.pos = offset 178 | else: 179 | self.curBlk = thisBlk 180 | self.decomp = self.read_block(blk=blk, setCur=False) 181 | self.pos = offset 182 | 183 | def tell(self): 184 | """ 185 | Returns the current position in the 'decompressed' data. 186 | """ 187 | return self.pos 188 | 189 | def tell_end(self): 190 | """ 191 | :type file: fileObj 192 | Determine the end of the compressed file. 193 | """ 194 | pos = self.fileObj.tell() 195 | self.fileObj.seek(0, 2) 196 | end = self.fileObj.tell() 197 | self.fileObj.seek(pos) 198 | return end 199 | 200 | def _regenDCTX(self): 201 | """ 202 | Regenerate the decompression context. 203 | """ 204 | try: 205 | lz4f.freeDecompContext(self.dCtx) 206 | del self.dCtx 207 | self.dCtx = lz4f.createDecompContext() 208 | frameInfo = lz4f.getFrameInfo(self.header, self.dCtx) 209 | lz4f.disableChecksum(self.dCtx) 210 | except AttributeError: 211 | self.dCtx = lz4f.createDecompContext() 212 | frameInfo = lz4f.getFrameInfo(self.header, self.dCtx) 213 | pass 214 | del frameInfo 215 | 216 | @property 217 | def decompPos(self): 218 | return self.pos - self.curBlkData.get('decomp_e') 219 | 220 | @property 221 | def curBlkData(self): 222 | return self.blkDict.get(self.curBlk) 223 | 224 | def seekable(self): 225 | if self.blkDict: 226 | return True 227 | return False 228 | -------------------------------------------------------------------------------- /lz4tools/lz4tar.py: -------------------------------------------------------------------------------- 1 | import sys 2 | import tarfile 3 | if sys.version_info.major >= 3: 4 | from .lz4file import Lz4File 5 | else: 6 | from lz4file import Lz4File 7 | 8 | 9 | class Lz4Tar(tarfile.TarFile): 10 | @classmethod 11 | def lz4open(cls, name, mode='r', fileobj=None, **kwargs): 12 | try: 13 | import lz4tools 14 | Lz4File = lz4tools.Lz4File 15 | except (ImportError, AttributeError): 16 | raise CompressionError("Lz4file module is not available") 17 | if name and fileobj is None: 18 | try: 19 | fileobj = Lz4File.open(name) 20 | except IOError: 21 | raise ReadError('Not a lz4 file') 22 | elif not name and not fileobj: 23 | print('Unable to open without a name or fileobj') 24 | return 25 | if not name and hasattr(fileobj, 'name'): 26 | name = fileobj.name 27 | try: 28 | t = cls(None, mode, fileobj) 29 | except: 30 | fileobj.close() 31 | t._extfileobj = False 32 | return t 33 | tarfile.TarFile.OPEN_METH.update({'lz4': 'lz4open'}) 34 | tarfile.TarFile.lz4open = lz4open 35 | OPEN_METH = { 36 | "tar": "taropen", # uncompressed tar 37 | "gz": "gzopen", # gzip compressed tar 38 | "bz2": "bz2open", # bzip2 compressed tar 39 | "lz4": "lz4open" # lz4 compressed tar 40 | } 41 | -------------------------------------------------------------------------------- /lz4toolsCli: -------------------------------------------------------------------------------- 1 | lz4tools/__main__.py -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | 3 | 4 | from setuptools import setup, Extension 5 | 6 | VERSION = (1, 3, 1, 2) 7 | VERSION_STR = ".".join([str(x) for x in VERSION]) 8 | ECA = [ 9 | "-std=c99", 10 | "-O3", 11 | "-Wall", 12 | "-Wundef", 13 | "-DVERSION=\"%s\"" % VERSION_STR, 14 | "-DLZ4_VERSION=\"r123\"", 15 | ] 16 | 17 | setup( 18 | name='lz4tools', 19 | version=VERSION_STR, 20 | description="LZ4Frame Bindings and tools for Python", 21 | license='BSD', 22 | long_description=open('README.rst', 'r').read(), 23 | author='Christopher Jackson', 24 | author_email='darkdragn.cj@gmail.com', 25 | url='https://github.com/darkdragn/lz4file', 26 | packages=['lz4tools'], 27 | scripts=['lz4toolsCli'], 28 | ext_modules=[ 29 | Extension('lz4f', [ 30 | 'src/lz4.c', 31 | 'src/lz4hc.c', 32 | 'src/lz4frame.c', 33 | 'src/python-lz4f.c', 34 | 'src/xxhash.c' 35 | ], extra_compile_args=[ 36 | "-std=c99", 37 | "-O3", 38 | "-Wall", 39 | "-W", 40 | "-Wundef", 41 | "-DVERSION=\"%s\"" % VERSION_STR, 42 | "-DLZ4_VERSION=\"r124\"", 43 | ])], 44 | setup_requires=["nose>=1.0"], 45 | test_suite="nose.collector", 46 | keywords=['lz4', 'lz4frame', 'lz4file', 'lz4tar'], 47 | classifiers=[ 48 | 'Development Status :: 5 - Production/Stable', 49 | 'License :: OSI Approved :: BSD License', 50 | 'Intended Audience :: Developers', 51 | 'Programming Language :: C', 52 | 'Programming Language :: Python', 53 | 'Programming Language :: Python :: 2', 54 | 'Programming Language :: Python :: 2.6', 55 | 'Programming Language :: Python :: 2.7', 56 | 'Programming Language :: Python :: 3', 57 | 'Programming Language :: Python :: 3.2', 58 | 'Programming Language :: Python :: 3.3', 59 | 'Programming Language :: Python :: 3.4', 60 | ], 61 | ) 62 | -------------------------------------------------------------------------------- /src/lz4.c: -------------------------------------------------------------------------------- 1 | /* 2 | LZ4 - Fast LZ compression algorithm 3 | Copyright (C) 2011-2015, Yann Collet. 4 | 5 | BSD 2-Clause License (http://www.opensource.org/licenses/bsd-license.php) 6 | 7 | Redistribution and use in source and binary forms, with or without 8 | modification, are permitted provided that the following conditions are 9 | met: 10 | 11 | * Redistributions of source code must retain the above copyright 12 | notice, this list of conditions and the following disclaimer. 13 | * Redistributions in binary form must reproduce the above 14 | copyright notice, this list of conditions and the following disclaimer 15 | in the documentation and/or other materials provided with the 16 | distribution. 17 | 18 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 19 | "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT 20 | LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR 21 | A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT 22 | OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, 23 | SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT 24 | LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, 25 | DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY 26 | THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT 27 | (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 28 | OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 29 | 30 | You can contact the author at : 31 | - LZ4 source repository : https://github.com/Cyan4973/lz4 32 | - LZ4 public forum : https://groups.google.com/forum/#!forum/lz4c 33 | */ 34 | 35 | 36 | /************************************** 37 | * Tuning parameters 38 | **************************************/ 39 | /* 40 | * HEAPMODE : 41 | * Select how default compression functions will allocate memory for their hash table, 42 | * in memory stack (0:default, fastest), or in memory heap (1:requires malloc()). 43 | */ 44 | #define HEAPMODE 0 45 | 46 | /* 47 | * ACCELERATION_DEFAULT : 48 | * Select "acceleration" for LZ4_compress_fast() when parameter value <= 0 49 | */ 50 | #define ACCELERATION_DEFAULT 1 51 | 52 | 53 | /************************************** 54 | * CPU Feature Detection 55 | **************************************/ 56 | /* 57 | * LZ4_FORCE_SW_BITCOUNT 58 | * Define this parameter if your target system or compiler does not support hardware bit count 59 | */ 60 | #if defined(_MSC_VER) && defined(_WIN32_WCE) /* Visual Studio for Windows CE does not support Hardware bit count */ 61 | # define LZ4_FORCE_SW_BITCOUNT 62 | #endif 63 | 64 | 65 | /************************************** 66 | * Includes 67 | **************************************/ 68 | #include "lz4.h" 69 | 70 | 71 | /************************************** 72 | * Compiler Options 73 | **************************************/ 74 | #ifdef _MSC_VER /* Visual Studio */ 75 | # define FORCE_INLINE static __forceinline 76 | # include 77 | # pragma warning(disable : 4127) /* disable: C4127: conditional expression is constant */ 78 | # pragma warning(disable : 4293) /* disable: C4293: too large shift (32-bits) */ 79 | #else 80 | # if defined(__STDC_VERSION__) && (__STDC_VERSION__ >= 199901L) /* C99 */ 81 | # if defined(__GNUC__) || defined(__clang__) 82 | # define FORCE_INLINE static inline __attribute__((always_inline)) 83 | # else 84 | # define FORCE_INLINE static inline 85 | # endif 86 | # else 87 | # define FORCE_INLINE static 88 | # endif /* __STDC_VERSION__ */ 89 | #endif /* _MSC_VER */ 90 | 91 | /* LZ4_GCC_VERSION is defined into lz4.h */ 92 | #if (LZ4_GCC_VERSION >= 302) || (__INTEL_COMPILER >= 800) || defined(__clang__) 93 | # define expect(expr,value) (__builtin_expect ((expr),(value)) ) 94 | #else 95 | # define expect(expr,value) (expr) 96 | #endif 97 | 98 | #define likely(expr) expect((expr) != 0, 1) 99 | #define unlikely(expr) expect((expr) != 0, 0) 100 | 101 | 102 | /************************************** 103 | * Memory routines 104 | **************************************/ 105 | #include /* malloc, calloc, free */ 106 | #define ALLOCATOR(n,s) calloc(n,s) 107 | #define FREEMEM free 108 | #include /* memset, memcpy */ 109 | #define MEM_INIT memset 110 | 111 | 112 | /************************************** 113 | * Basic Types 114 | **************************************/ 115 | #if defined (__STDC_VERSION__) && (__STDC_VERSION__ >= 199901L) /* C99 */ 116 | # include 117 | typedef uint8_t BYTE; 118 | typedef uint16_t U16; 119 | typedef uint32_t U32; 120 | typedef int32_t S32; 121 | typedef uint64_t U64; 122 | #else 123 | typedef unsigned char BYTE; 124 | typedef unsigned short U16; 125 | typedef unsigned int U32; 126 | typedef signed int S32; 127 | typedef unsigned long long U64; 128 | #endif 129 | 130 | 131 | /************************************** 132 | * Reading and writing into memory 133 | **************************************/ 134 | #define STEPSIZE sizeof(size_t) 135 | 136 | static unsigned LZ4_64bits(void) { return sizeof(void*)==8; } 137 | 138 | static unsigned LZ4_isLittleEndian(void) 139 | { 140 | const union { U32 i; BYTE c[4]; } one = { 1 }; /* don't use static : performance detrimental */ 141 | return one.c[0]; 142 | } 143 | 144 | 145 | static U16 LZ4_read16(const void* memPtr) 146 | { 147 | U16 val16; 148 | memcpy(&val16, memPtr, 2); 149 | return val16; 150 | } 151 | 152 | static U16 LZ4_readLE16(const void* memPtr) 153 | { 154 | if (LZ4_isLittleEndian()) 155 | { 156 | return LZ4_read16(memPtr); 157 | } 158 | else 159 | { 160 | const BYTE* p = (const BYTE*)memPtr; 161 | return (U16)((U16)p[0] + (p[1]<<8)); 162 | } 163 | } 164 | 165 | static void LZ4_writeLE16(void* memPtr, U16 value) 166 | { 167 | if (LZ4_isLittleEndian()) 168 | { 169 | memcpy(memPtr, &value, 2); 170 | } 171 | else 172 | { 173 | BYTE* p = (BYTE*)memPtr; 174 | p[0] = (BYTE) value; 175 | p[1] = (BYTE)(value>>8); 176 | } 177 | } 178 | 179 | static U32 LZ4_read32(const void* memPtr) 180 | { 181 | U32 val32; 182 | memcpy(&val32, memPtr, 4); 183 | return val32; 184 | } 185 | 186 | static U64 LZ4_read64(const void* memPtr) 187 | { 188 | U64 val64; 189 | memcpy(&val64, memPtr, 8); 190 | return val64; 191 | } 192 | 193 | static size_t LZ4_read_ARCH(const void* p) 194 | { 195 | if (LZ4_64bits()) 196 | return (size_t)LZ4_read64(p); 197 | else 198 | return (size_t)LZ4_read32(p); 199 | } 200 | 201 | 202 | static void LZ4_copy4(void* dstPtr, const void* srcPtr) { memcpy(dstPtr, srcPtr, 4); } 203 | 204 | static void LZ4_copy8(void* dstPtr, const void* srcPtr) { memcpy(dstPtr, srcPtr, 8); } 205 | 206 | /* customized version of memcpy, which may overwrite up to 7 bytes beyond dstEnd */ 207 | static void LZ4_wildCopy(void* dstPtr, const void* srcPtr, void* dstEnd) 208 | { 209 | BYTE* d = (BYTE*)dstPtr; 210 | const BYTE* s = (const BYTE*)srcPtr; 211 | BYTE* e = (BYTE*)dstEnd; 212 | do { LZ4_copy8(d,s); d+=8; s+=8; } while (d>3); 258 | # elif (defined(__clang__) || (LZ4_GCC_VERSION >= 304)) && !defined(LZ4_FORCE_SW_BITCOUNT) 259 | return (__builtin_ctzll((U64)val) >> 3); 260 | # else 261 | static const int DeBruijnBytePos[64] = { 0, 0, 0, 0, 0, 1, 1, 2, 0, 3, 1, 3, 1, 4, 2, 7, 0, 2, 3, 6, 1, 5, 3, 5, 1, 3, 4, 4, 2, 5, 6, 7, 7, 0, 1, 2, 3, 3, 4, 6, 2, 6, 5, 5, 3, 4, 5, 6, 7, 1, 2, 4, 6, 4, 4, 5, 7, 2, 6, 5, 7, 6, 7, 7 }; 262 | return DeBruijnBytePos[((U64)((val & -(long long)val) * 0x0218A392CDABBD3FULL)) >> 58]; 263 | # endif 264 | } 265 | else /* 32 bits */ 266 | { 267 | # if defined(_MSC_VER) && !defined(LZ4_FORCE_SW_BITCOUNT) 268 | unsigned long r; 269 | _BitScanForward( &r, (U32)val ); 270 | return (int)(r>>3); 271 | # elif (defined(__clang__) || (LZ4_GCC_VERSION >= 304)) && !defined(LZ4_FORCE_SW_BITCOUNT) 272 | return (__builtin_ctz((U32)val) >> 3); 273 | # else 274 | static const int DeBruijnBytePos[32] = { 0, 0, 3, 0, 3, 1, 3, 0, 3, 2, 2, 1, 3, 2, 0, 1, 3, 3, 1, 2, 2, 2, 2, 0, 3, 1, 2, 0, 1, 0, 1, 1 }; 275 | return DeBruijnBytePos[((U32)((val & -(S32)val) * 0x077CB531U)) >> 27]; 276 | # endif 277 | } 278 | } 279 | else /* Big Endian CPU */ 280 | { 281 | if (LZ4_64bits()) 282 | { 283 | # if defined(_MSC_VER) && defined(_WIN64) && !defined(LZ4_FORCE_SW_BITCOUNT) 284 | unsigned long r = 0; 285 | _BitScanReverse64( &r, val ); 286 | return (unsigned)(r>>3); 287 | # elif (defined(__clang__) || (LZ4_GCC_VERSION >= 304)) && !defined(LZ4_FORCE_SW_BITCOUNT) 288 | return (__builtin_clzll((U64)val) >> 3); 289 | # else 290 | unsigned r; 291 | if (!(val>>32)) { r=4; } else { r=0; val>>=32; } 292 | if (!(val>>16)) { r+=2; val>>=8; } else { val>>=24; } 293 | r += (!val); 294 | return r; 295 | # endif 296 | } 297 | else /* 32 bits */ 298 | { 299 | # if defined(_MSC_VER) && !defined(LZ4_FORCE_SW_BITCOUNT) 300 | unsigned long r = 0; 301 | _BitScanReverse( &r, (unsigned long)val ); 302 | return (unsigned)(r>>3); 303 | # elif (defined(__clang__) || (LZ4_GCC_VERSION >= 304)) && !defined(LZ4_FORCE_SW_BITCOUNT) 304 | return (__builtin_clz((U32)val) >> 3); 305 | # else 306 | unsigned r; 307 | if (!(val>>16)) { r=2; val>>=8; } else { r=0; val>>=24; } 308 | r += (!val); 309 | return r; 310 | # endif 311 | } 312 | } 313 | } 314 | 315 | static unsigned LZ4_count(const BYTE* pIn, const BYTE* pMatch, const BYTE* pInLimit) 316 | { 317 | const BYTE* const pStart = pIn; 318 | 319 | while (likely(pIn compression run slower on incompressible data */ 344 | 345 | 346 | /************************************** 347 | * Local Structures and types 348 | **************************************/ 349 | typedef struct { 350 | U32 hashTable[HASH_SIZE_U32]; 351 | U32 currentOffset; 352 | U32 initCheck; 353 | const BYTE* dictionary; 354 | BYTE* bufferStart; /* obsolete, used for slideInputBuffer */ 355 | U32 dictSize; 356 | } LZ4_stream_t_internal; 357 | 358 | typedef enum { notLimited = 0, limitedOutput = 1 } limitedOutput_directive; 359 | typedef enum { byPtr, byU32, byU16 } tableType_t; 360 | 361 | typedef enum { noDict = 0, withPrefix64k, usingExtDict } dict_directive; 362 | typedef enum { noDictIssue = 0, dictSmall } dictIssue_directive; 363 | 364 | typedef enum { endOnOutputSize = 0, endOnInputSize = 1 } endCondition_directive; 365 | typedef enum { full = 0, partial = 1 } earlyEnd_directive; 366 | 367 | 368 | /************************************** 369 | * Local Utils 370 | **************************************/ 371 | int LZ4_versionNumber (void) { return LZ4_VERSION_NUMBER; } 372 | int LZ4_compressBound(int isize) { return LZ4_COMPRESSBOUND(isize); } 373 | int LZ4_sizeofState() { return LZ4_STREAMSIZE; } 374 | 375 | 376 | 377 | /******************************** 378 | * Compression functions 379 | ********************************/ 380 | 381 | static U32 LZ4_hashSequence(U32 sequence, tableType_t const tableType) 382 | { 383 | if (tableType == byU16) 384 | return (((sequence) * 2654435761U) >> ((MINMATCH*8)-(LZ4_HASHLOG+1))); 385 | else 386 | return (((sequence) * 2654435761U) >> ((MINMATCH*8)-LZ4_HASHLOG)); 387 | } 388 | 389 | static const U64 prime5bytes = 889523592379ULL; 390 | static U32 LZ4_hashSequence64(size_t sequence, tableType_t const tableType) 391 | { 392 | const U32 hashLog = (tableType == byU16) ? LZ4_HASHLOG+1 : LZ4_HASHLOG; 393 | const U32 hashMask = (1<> (40 - hashLog)) & hashMask; 395 | } 396 | 397 | static U32 LZ4_hashSequenceT(size_t sequence, tableType_t const tableType) 398 | { 399 | if (LZ4_64bits()) 400 | return LZ4_hashSequence64(sequence, tableType); 401 | return LZ4_hashSequence((U32)sequence, tableType); 402 | } 403 | 404 | static U32 LZ4_hashPosition(const void* p, tableType_t tableType) { return LZ4_hashSequenceT(LZ4_read_ARCH(p), tableType); } 405 | 406 | static void LZ4_putPositionOnHash(const BYTE* p, U32 h, void* tableBase, tableType_t const tableType, const BYTE* srcBase) 407 | { 408 | switch (tableType) 409 | { 410 | case byPtr: { const BYTE** hashTable = (const BYTE**)tableBase; hashTable[h] = p; return; } 411 | case byU32: { U32* hashTable = (U32*) tableBase; hashTable[h] = (U32)(p-srcBase); return; } 412 | case byU16: { U16* hashTable = (U16*) tableBase; hashTable[h] = (U16)(p-srcBase); return; } 413 | } 414 | } 415 | 416 | static void LZ4_putPosition(const BYTE* p, void* tableBase, tableType_t tableType, const BYTE* srcBase) 417 | { 418 | U32 h = LZ4_hashPosition(p, tableType); 419 | LZ4_putPositionOnHash(p, h, tableBase, tableType, srcBase); 420 | } 421 | 422 | static const BYTE* LZ4_getPositionOnHash(U32 h, void* tableBase, tableType_t tableType, const BYTE* srcBase) 423 | { 424 | if (tableType == byPtr) { const BYTE** hashTable = (const BYTE**) tableBase; return hashTable[h]; } 425 | if (tableType == byU32) { U32* hashTable = (U32*) tableBase; return hashTable[h] + srcBase; } 426 | { U16* hashTable = (U16*) tableBase; return hashTable[h] + srcBase; } /* default, to ensure a return */ 427 | } 428 | 429 | static const BYTE* LZ4_getPosition(const BYTE* p, void* tableBase, tableType_t tableType, const BYTE* srcBase) 430 | { 431 | U32 h = LZ4_hashPosition(p, tableType); 432 | return LZ4_getPositionOnHash(h, tableBase, tableType, srcBase); 433 | } 434 | 435 | FORCE_INLINE int LZ4_compress_generic( 436 | void* const ctx, 437 | const char* const source, 438 | char* const dest, 439 | const int inputSize, 440 | const int maxOutputSize, 441 | const limitedOutput_directive outputLimited, 442 | const tableType_t tableType, 443 | const dict_directive dict, 444 | const dictIssue_directive dictIssue, 445 | const U32 acceleration) 446 | { 447 | LZ4_stream_t_internal* const dictPtr = (LZ4_stream_t_internal*)ctx; 448 | 449 | const BYTE* ip = (const BYTE*) source; 450 | const BYTE* base; 451 | const BYTE* lowLimit; 452 | const BYTE* const lowRefLimit = ip - dictPtr->dictSize; 453 | const BYTE* const dictionary = dictPtr->dictionary; 454 | const BYTE* const dictEnd = dictionary + dictPtr->dictSize; 455 | const size_t dictDelta = dictEnd - (const BYTE*)source; 456 | const BYTE* anchor = (const BYTE*) source; 457 | const BYTE* const iend = ip + inputSize; 458 | const BYTE* const mflimit = iend - MFLIMIT; 459 | const BYTE* const matchlimit = iend - LASTLITERALS; 460 | 461 | BYTE* op = (BYTE*) dest; 462 | BYTE* const olimit = op + maxOutputSize; 463 | 464 | U32 forwardH; 465 | size_t refDelta=0; 466 | 467 | /* Init conditions */ 468 | if ((U32)inputSize > (U32)LZ4_MAX_INPUT_SIZE) return 0; /* Unsupported input size, too large (or negative) */ 469 | switch(dict) 470 | { 471 | case noDict: 472 | default: 473 | base = (const BYTE*)source; 474 | lowLimit = (const BYTE*)source; 475 | break; 476 | case withPrefix64k: 477 | base = (const BYTE*)source - dictPtr->currentOffset; 478 | lowLimit = (const BYTE*)source - dictPtr->dictSize; 479 | break; 480 | case usingExtDict: 481 | base = (const BYTE*)source - dictPtr->currentOffset; 482 | lowLimit = (const BYTE*)source; 483 | break; 484 | } 485 | if ((tableType == byU16) && (inputSize>=LZ4_64Klimit)) return 0; /* Size too large (not within 64K limit) */ 486 | if (inputSize> LZ4_skipTrigger); 508 | 509 | if (unlikely(forwardIp > mflimit)) goto _last_literals; 510 | 511 | match = LZ4_getPositionOnHash(h, ctx, tableType, base); 512 | if (dict==usingExtDict) 513 | { 514 | if (match<(const BYTE*)source) 515 | { 516 | refDelta = dictDelta; 517 | lowLimit = dictionary; 518 | } 519 | else 520 | { 521 | refDelta = 0; 522 | lowLimit = (const BYTE*)source; 523 | } 524 | } 525 | forwardH = LZ4_hashPosition(forwardIp, tableType); 526 | LZ4_putPositionOnHash(ip, h, ctx, tableType, base); 527 | 528 | } while ( ((dictIssue==dictSmall) ? (match < lowRefLimit) : 0) 529 | || ((tableType==byU16) ? 0 : (match + MAX_DISTANCE < ip)) 530 | || (LZ4_read32(match+refDelta) != LZ4_read32(ip)) ); 531 | } 532 | 533 | /* Catch up */ 534 | while ((ip>anchor) && (match+refDelta > lowLimit) && (unlikely(ip[-1]==match[refDelta-1]))) { ip--; match--; } 535 | 536 | { 537 | /* Encode Literal length */ 538 | unsigned litLength = (unsigned)(ip - anchor); 539 | token = op++; 540 | if ((outputLimited) && (unlikely(op + litLength + (2 + 1 + LASTLITERALS) + (litLength/255) > olimit))) 541 | return 0; /* Check output limit */ 542 | if (litLength>=RUN_MASK) 543 | { 544 | int len = (int)litLength-RUN_MASK; 545 | *token=(RUN_MASK<= 255 ; len-=255) *op++ = 255; 547 | *op++ = (BYTE)len; 548 | } 549 | else *token = (BYTE)(litLength< matchlimit) limit = matchlimit; 570 | matchLength = LZ4_count(ip+MINMATCH, match+MINMATCH, limit); 571 | ip += MINMATCH + matchLength; 572 | if (ip==limit) 573 | { 574 | unsigned more = LZ4_count(ip, (const BYTE*)source, matchlimit); 575 | matchLength += more; 576 | ip += more; 577 | } 578 | } 579 | else 580 | { 581 | matchLength = LZ4_count(ip+MINMATCH, match+MINMATCH, matchlimit); 582 | ip += MINMATCH + matchLength; 583 | } 584 | 585 | if ((outputLimited) && (unlikely(op + (1 + LASTLITERALS) + (matchLength>>8) > olimit))) 586 | return 0; /* Check output limit */ 587 | if (matchLength>=ML_MASK) 588 | { 589 | *token += ML_MASK; 590 | matchLength -= ML_MASK; 591 | for (; matchLength >= 510 ; matchLength-=510) { *op++ = 255; *op++ = 255; } 592 | if (matchLength >= 255) { matchLength-=255; *op++ = 255; } 593 | *op++ = (BYTE)matchLength; 594 | } 595 | else *token += (BYTE)(matchLength); 596 | } 597 | 598 | anchor = ip; 599 | 600 | /* Test end of chunk */ 601 | if (ip > mflimit) break; 602 | 603 | /* Fill table */ 604 | LZ4_putPosition(ip-2, ctx, tableType, base); 605 | 606 | /* Test next position */ 607 | match = LZ4_getPosition(ip, ctx, tableType, base); 608 | if (dict==usingExtDict) 609 | { 610 | if (match<(const BYTE*)source) 611 | { 612 | refDelta = dictDelta; 613 | lowLimit = dictionary; 614 | } 615 | else 616 | { 617 | refDelta = 0; 618 | lowLimit = (const BYTE*)source; 619 | } 620 | } 621 | LZ4_putPosition(ip, ctx, tableType, base); 622 | if ( ((dictIssue==dictSmall) ? (match>=lowRefLimit) : 1) 623 | && (match+MAX_DISTANCE>=ip) 624 | && (LZ4_read32(match+refDelta)==LZ4_read32(ip)) ) 625 | { token=op++; *token=0; goto _next_match; } 626 | 627 | /* Prepare next loop */ 628 | forwardH = LZ4_hashPosition(++ip, tableType); 629 | } 630 | 631 | _last_literals: 632 | /* Encode Last Literals */ 633 | { 634 | const size_t lastRun = (size_t)(iend - anchor); 635 | if ((outputLimited) && ((op - (BYTE*)dest) + lastRun + 1 + ((lastRun+255-RUN_MASK)/255) > (U32)maxOutputSize)) 636 | return 0; /* Check output limit */ 637 | if (lastRun >= RUN_MASK) 638 | { 639 | size_t accumulator = lastRun - RUN_MASK; 640 | *op++ = RUN_MASK << ML_BITS; 641 | for(; accumulator >= 255 ; accumulator-=255) *op++ = 255; 642 | *op++ = (BYTE) accumulator; 643 | } 644 | else 645 | { 646 | *op++ = (BYTE)(lastRun<= LZ4_compressBound(inputSize)) 663 | { 664 | if (inputSize < LZ4_64Klimit) 665 | return LZ4_compress_generic(state, source, dest, inputSize, 0, notLimited, byU16, noDict, noDictIssue, acceleration); 666 | else 667 | return LZ4_compress_generic(state, source, dest, inputSize, 0, notLimited, LZ4_64bits() ? byU32 : byPtr, noDict, noDictIssue, acceleration); 668 | } 669 | else 670 | { 671 | if (inputSize < LZ4_64Klimit) 672 | return LZ4_compress_generic(state, source, dest, inputSize, maxOutputSize, limitedOutput, byU16, noDict, noDictIssue, acceleration); 673 | else 674 | return LZ4_compress_generic(state, source, dest, inputSize, maxOutputSize, limitedOutput, LZ4_64bits() ? byU32 : byPtr, noDict, noDictIssue, acceleration); 675 | } 676 | } 677 | 678 | 679 | int LZ4_compress_fast(const char* source, char* dest, int inputSize, int maxOutputSize, int acceleration) 680 | { 681 | #if (HEAPMODE) 682 | void* ctxPtr = ALLOCATOR(1, sizeof(LZ4_stream_t)); /* malloc-calloc always properly aligned */ 683 | #else 684 | LZ4_stream_t ctx; 685 | void* ctxPtr = &ctx; 686 | #endif 687 | 688 | int result = LZ4_compress_fast_extState(ctxPtr, source, dest, inputSize, maxOutputSize, acceleration); 689 | 690 | #if (HEAPMODE) 691 | FREEMEM(ctxPtr); 692 | #endif 693 | return result; 694 | } 695 | 696 | 697 | int LZ4_compress_default(const char* source, char* dest, int inputSize, int maxOutputSize) 698 | { 699 | return LZ4_compress_fast(source, dest, inputSize, maxOutputSize, 1); 700 | } 701 | 702 | 703 | /* hidden debug function */ 704 | /* strangely enough, gcc generates faster code when this function is uncommented, even if unused */ 705 | int LZ4_compress_fast_force(const char* source, char* dest, int inputSize, int maxOutputSize, int acceleration) 706 | { 707 | LZ4_stream_t ctx; 708 | 709 | LZ4_resetStream(&ctx); 710 | 711 | if (inputSize < LZ4_64Klimit) 712 | return LZ4_compress_generic(&ctx, source, dest, inputSize, maxOutputSize, limitedOutput, byU16, noDict, noDictIssue, acceleration); 713 | else 714 | return LZ4_compress_generic(&ctx, source, dest, inputSize, maxOutputSize, limitedOutput, LZ4_64bits() ? byU32 : byPtr, noDict, noDictIssue, acceleration); 715 | } 716 | 717 | 718 | /******************************** 719 | * destSize variant 720 | ********************************/ 721 | 722 | static int LZ4_compress_destSize_generic( 723 | void* const ctx, 724 | const char* const src, 725 | char* const dst, 726 | int* const srcSizePtr, 727 | const int targetDstSize, 728 | const tableType_t tableType) 729 | { 730 | const BYTE* ip = (const BYTE*) src; 731 | const BYTE* base = (const BYTE*) src; 732 | const BYTE* lowLimit = (const BYTE*) src; 733 | const BYTE* anchor = ip; 734 | const BYTE* const iend = ip + *srcSizePtr; 735 | const BYTE* const mflimit = iend - MFLIMIT; 736 | const BYTE* const matchlimit = iend - LASTLITERALS; 737 | 738 | BYTE* op = (BYTE*) dst; 739 | BYTE* const oend = op + targetDstSize; 740 | BYTE* const oMaxLit = op + targetDstSize - 2 /* offset */ - 8 /* because 8+MINMATCH==MFLIMIT */ - 1 /* token */; 741 | BYTE* const oMaxMatch = op + targetDstSize - (LASTLITERALS + 1 /* token */); 742 | BYTE* const oMaxSeq = oMaxLit - 1 /* token */; 743 | 744 | U32 forwardH; 745 | 746 | 747 | /* Init conditions */ 748 | if (targetDstSize < 1) return 0; /* Impossible to store anything */ 749 | if ((U32)*srcSizePtr > (U32)LZ4_MAX_INPUT_SIZE) return 0; /* Unsupported input size, too large (or negative) */ 750 | if ((tableType == byU16) && (*srcSizePtr>=LZ4_64Klimit)) return 0; /* Size too large (not within 64K limit) */ 751 | if (*srcSizePtr> LZ4_skipTrigger); 774 | 775 | if (unlikely(forwardIp > mflimit)) 776 | goto _last_literals; 777 | 778 | match = LZ4_getPositionOnHash(h, ctx, tableType, base); 779 | forwardH = LZ4_hashPosition(forwardIp, tableType); 780 | LZ4_putPositionOnHash(ip, h, ctx, tableType, base); 781 | 782 | } while ( ((tableType==byU16) ? 0 : (match + MAX_DISTANCE < ip)) 783 | || (LZ4_read32(match) != LZ4_read32(ip)) ); 784 | } 785 | 786 | /* Catch up */ 787 | while ((ip>anchor) && (match > lowLimit) && (unlikely(ip[-1]==match[-1]))) { ip--; match--; } 788 | 789 | { 790 | /* Encode Literal length */ 791 | unsigned litLength = (unsigned)(ip - anchor); 792 | token = op++; 793 | if (op + ((litLength+240)/255) + litLength > oMaxLit) 794 | { 795 | /* Not enough space for a last match */ 796 | op--; 797 | goto _last_literals; 798 | } 799 | if (litLength>=RUN_MASK) 800 | { 801 | unsigned len = litLength - RUN_MASK; 802 | *token=(RUN_MASK<= 255 ; len-=255) *op++ = 255; 804 | *op++ = (BYTE)len; 805 | } 806 | else *token = (BYTE)(litLength< oMaxMatch) 824 | { 825 | /* Match description too long : reduce it */ 826 | matchLength = (15-1) + (oMaxMatch-op) * 255; 827 | } 828 | //printf("offset %5i, matchLength%5i \n", (int)(ip-match), matchLength + MINMATCH); 829 | ip += MINMATCH + matchLength; 830 | 831 | if (matchLength>=ML_MASK) 832 | { 833 | *token += ML_MASK; 834 | matchLength -= ML_MASK; 835 | while (matchLength >= 255) { matchLength-=255; *op++ = 255; } 836 | *op++ = (BYTE)matchLength; 837 | } 838 | else *token += (BYTE)(matchLength); 839 | } 840 | 841 | anchor = ip; 842 | 843 | /* Test end of block */ 844 | if (ip > mflimit) break; 845 | if (op > oMaxSeq) break; 846 | 847 | /* Fill table */ 848 | LZ4_putPosition(ip-2, ctx, tableType, base); 849 | 850 | /* Test next position */ 851 | match = LZ4_getPosition(ip, ctx, tableType, base); 852 | LZ4_putPosition(ip, ctx, tableType, base); 853 | if ( (match+MAX_DISTANCE>=ip) 854 | && (LZ4_read32(match)==LZ4_read32(ip)) ) 855 | { token=op++; *token=0; goto _next_match; } 856 | 857 | /* Prepare next loop */ 858 | forwardH = LZ4_hashPosition(++ip, tableType); 859 | } 860 | 861 | _last_literals: 862 | /* Encode Last Literals */ 863 | { 864 | size_t lastRunSize = (size_t)(iend - anchor); 865 | if (op + 1 /* token */ + ((lastRunSize+240)/255) /* litLength */ + lastRunSize /* literals */ > oend) 866 | { 867 | /* adapt lastRunSize to fill 'dst' */ 868 | lastRunSize = (oend-op) - 1; 869 | lastRunSize -= (lastRunSize+240)/255; 870 | } 871 | ip = anchor + lastRunSize; 872 | 873 | if (lastRunSize >= RUN_MASK) 874 | { 875 | size_t accumulator = lastRunSize - RUN_MASK; 876 | *op++ = RUN_MASK << ML_BITS; 877 | for(; accumulator >= 255 ; accumulator-=255) *op++ = 255; 878 | *op++ = (BYTE) accumulator; 879 | } 880 | else 881 | { 882 | *op++ = (BYTE)(lastRunSize<= LZ4_compressBound(*srcSizePtr)) /* compression success is guaranteed */ 899 | { 900 | return LZ4_compress_fast_extState(state, src, dst, *srcSizePtr, targetDstSize, 1); 901 | } 902 | else 903 | { 904 | if (*srcSizePtr < LZ4_64Klimit) 905 | return LZ4_compress_destSize_generic(state, src, dst, srcSizePtr, targetDstSize, byU16); 906 | else 907 | return LZ4_compress_destSize_generic(state, src, dst, srcSizePtr, targetDstSize, LZ4_64bits() ? byU32 : byPtr); 908 | } 909 | } 910 | 911 | 912 | int LZ4_compress_destSize(const char* src, char* dst, int* srcSizePtr, int targetDstSize) 913 | { 914 | #if (HEAPMODE) 915 | void* ctx = ALLOCATOR(1, sizeof(LZ4_stream_t)); /* malloc-calloc always properly aligned */ 916 | #else 917 | LZ4_stream_t ctxBody; 918 | void* ctx = &ctxBody; 919 | #endif 920 | 921 | int result = LZ4_compress_destSize_extState(ctx, src, dst, srcSizePtr, targetDstSize); 922 | 923 | #if (HEAPMODE) 924 | FREEMEM(ctx); 925 | #endif 926 | return result; 927 | } 928 | 929 | 930 | 931 | /******************************** 932 | * Streaming functions 933 | ********************************/ 934 | 935 | LZ4_stream_t* LZ4_createStream(void) 936 | { 937 | LZ4_stream_t* lz4s = (LZ4_stream_t*)ALLOCATOR(8, LZ4_STREAMSIZE_U64); 938 | LZ4_STATIC_ASSERT(LZ4_STREAMSIZE >= sizeof(LZ4_stream_t_internal)); /* A compilation error here means LZ4_STREAMSIZE is not large enough */ 939 | LZ4_resetStream(lz4s); 940 | return lz4s; 941 | } 942 | 943 | void LZ4_resetStream (LZ4_stream_t* LZ4_stream) 944 | { 945 | MEM_INIT(LZ4_stream, 0, sizeof(LZ4_stream_t)); 946 | } 947 | 948 | int LZ4_freeStream (LZ4_stream_t* LZ4_stream) 949 | { 950 | FREEMEM(LZ4_stream); 951 | return (0); 952 | } 953 | 954 | 955 | #define HASH_UNIT sizeof(size_t) 956 | int LZ4_loadDict (LZ4_stream_t* LZ4_dict, const char* dictionary, int dictSize) 957 | { 958 | LZ4_stream_t_internal* dict = (LZ4_stream_t_internal*) LZ4_dict; 959 | const BYTE* p = (const BYTE*)dictionary; 960 | const BYTE* const dictEnd = p + dictSize; 961 | const BYTE* base; 962 | 963 | if ((dict->initCheck) || (dict->currentOffset > 1 GB)) /* Uninitialized structure, or reuse overflow */ 964 | LZ4_resetStream(LZ4_dict); 965 | 966 | if (dictSize < (int)HASH_UNIT) 967 | { 968 | dict->dictionary = NULL; 969 | dict->dictSize = 0; 970 | return 0; 971 | } 972 | 973 | if ((dictEnd - p) > 64 KB) p = dictEnd - 64 KB; 974 | dict->currentOffset += 64 KB; 975 | base = p - dict->currentOffset; 976 | dict->dictionary = p; 977 | dict->dictSize = (U32)(dictEnd - p); 978 | dict->currentOffset += dict->dictSize; 979 | 980 | while (p <= dictEnd-HASH_UNIT) 981 | { 982 | LZ4_putPosition(p, dict->hashTable, byU32, base); 983 | p+=3; 984 | } 985 | 986 | return dict->dictSize; 987 | } 988 | 989 | 990 | static void LZ4_renormDictT(LZ4_stream_t_internal* LZ4_dict, const BYTE* src) 991 | { 992 | if ((LZ4_dict->currentOffset > 0x80000000) || 993 | ((size_t)LZ4_dict->currentOffset > (size_t)src)) /* address space overflow */ 994 | { 995 | /* rescale hash table */ 996 | U32 delta = LZ4_dict->currentOffset - 64 KB; 997 | const BYTE* dictEnd = LZ4_dict->dictionary + LZ4_dict->dictSize; 998 | int i; 999 | for (i=0; ihashTable[i] < delta) LZ4_dict->hashTable[i]=0; 1002 | else LZ4_dict->hashTable[i] -= delta; 1003 | } 1004 | LZ4_dict->currentOffset = 64 KB; 1005 | if (LZ4_dict->dictSize > 64 KB) LZ4_dict->dictSize = 64 KB; 1006 | LZ4_dict->dictionary = dictEnd - LZ4_dict->dictSize; 1007 | } 1008 | } 1009 | 1010 | 1011 | int LZ4_compress_fast_continue (LZ4_stream_t* LZ4_stream, const char* source, char* dest, int inputSize, int maxOutputSize, int acceleration) 1012 | { 1013 | LZ4_stream_t_internal* streamPtr = (LZ4_stream_t_internal*)LZ4_stream; 1014 | const BYTE* const dictEnd = streamPtr->dictionary + streamPtr->dictSize; 1015 | 1016 | const BYTE* smallest = (const BYTE*) source; 1017 | if (streamPtr->initCheck) return 0; /* Uninitialized structure detected */ 1018 | if ((streamPtr->dictSize>0) && (smallest>dictEnd)) smallest = dictEnd; 1019 | LZ4_renormDictT(streamPtr, smallest); 1020 | if (acceleration < 1) acceleration = ACCELERATION_DEFAULT; 1021 | 1022 | /* Check overlapping input/dictionary space */ 1023 | { 1024 | const BYTE* sourceEnd = (const BYTE*) source + inputSize; 1025 | if ((sourceEnd > streamPtr->dictionary) && (sourceEnd < dictEnd)) 1026 | { 1027 | streamPtr->dictSize = (U32)(dictEnd - sourceEnd); 1028 | if (streamPtr->dictSize > 64 KB) streamPtr->dictSize = 64 KB; 1029 | if (streamPtr->dictSize < 4) streamPtr->dictSize = 0; 1030 | streamPtr->dictionary = dictEnd - streamPtr->dictSize; 1031 | } 1032 | } 1033 | 1034 | /* prefix mode : source data follows dictionary */ 1035 | if (dictEnd == (const BYTE*)source) 1036 | { 1037 | int result; 1038 | if ((streamPtr->dictSize < 64 KB) && (streamPtr->dictSize < streamPtr->currentOffset)) 1039 | result = LZ4_compress_generic(LZ4_stream, source, dest, inputSize, maxOutputSize, limitedOutput, byU32, withPrefix64k, dictSmall, acceleration); 1040 | else 1041 | result = LZ4_compress_generic(LZ4_stream, source, dest, inputSize, maxOutputSize, limitedOutput, byU32, withPrefix64k, noDictIssue, acceleration); 1042 | streamPtr->dictSize += (U32)inputSize; 1043 | streamPtr->currentOffset += (U32)inputSize; 1044 | return result; 1045 | } 1046 | 1047 | /* external dictionary mode */ 1048 | { 1049 | int result; 1050 | if ((streamPtr->dictSize < 64 KB) && (streamPtr->dictSize < streamPtr->currentOffset)) 1051 | result = LZ4_compress_generic(LZ4_stream, source, dest, inputSize, maxOutputSize, limitedOutput, byU32, usingExtDict, dictSmall, acceleration); 1052 | else 1053 | result = LZ4_compress_generic(LZ4_stream, source, dest, inputSize, maxOutputSize, limitedOutput, byU32, usingExtDict, noDictIssue, acceleration); 1054 | streamPtr->dictionary = (const BYTE*)source; 1055 | streamPtr->dictSize = (U32)inputSize; 1056 | streamPtr->currentOffset += (U32)inputSize; 1057 | return result; 1058 | } 1059 | } 1060 | 1061 | 1062 | /* Hidden debug function, to force external dictionary mode */ 1063 | int LZ4_compress_forceExtDict (LZ4_stream_t* LZ4_dict, const char* source, char* dest, int inputSize) 1064 | { 1065 | LZ4_stream_t_internal* streamPtr = (LZ4_stream_t_internal*)LZ4_dict; 1066 | int result; 1067 | const BYTE* const dictEnd = streamPtr->dictionary + streamPtr->dictSize; 1068 | 1069 | const BYTE* smallest = dictEnd; 1070 | if (smallest > (const BYTE*) source) smallest = (const BYTE*) source; 1071 | LZ4_renormDictT((LZ4_stream_t_internal*)LZ4_dict, smallest); 1072 | 1073 | result = LZ4_compress_generic(LZ4_dict, source, dest, inputSize, 0, notLimited, byU32, usingExtDict, noDictIssue, 1); 1074 | 1075 | streamPtr->dictionary = (const BYTE*)source; 1076 | streamPtr->dictSize = (U32)inputSize; 1077 | streamPtr->currentOffset += (U32)inputSize; 1078 | 1079 | return result; 1080 | } 1081 | 1082 | 1083 | int LZ4_saveDict (LZ4_stream_t* LZ4_dict, char* safeBuffer, int dictSize) 1084 | { 1085 | LZ4_stream_t_internal* dict = (LZ4_stream_t_internal*) LZ4_dict; 1086 | const BYTE* previousDictEnd = dict->dictionary + dict->dictSize; 1087 | 1088 | if ((U32)dictSize > 64 KB) dictSize = 64 KB; /* useless to define a dictionary > 64 KB */ 1089 | if ((U32)dictSize > dict->dictSize) dictSize = dict->dictSize; 1090 | 1091 | memmove(safeBuffer, previousDictEnd - dictSize, dictSize); 1092 | 1093 | dict->dictionary = (const BYTE*)safeBuffer; 1094 | dict->dictSize = (U32)dictSize; 1095 | 1096 | return dictSize; 1097 | } 1098 | 1099 | 1100 | 1101 | /******************************* 1102 | * Decompression functions 1103 | *******************************/ 1104 | /* 1105 | * This generic decompression function cover all use cases. 1106 | * It shall be instantiated several times, using different sets of directives 1107 | * Note that it is essential this generic function is really inlined, 1108 | * in order to remove useless branches during compilation optimization. 1109 | */ 1110 | FORCE_INLINE int LZ4_decompress_generic( 1111 | const char* const source, 1112 | char* const dest, 1113 | int inputSize, 1114 | int outputSize, /* If endOnInput==endOnInputSize, this value is the max size of Output Buffer. */ 1115 | 1116 | int endOnInput, /* endOnOutputSize, endOnInputSize */ 1117 | int partialDecoding, /* full, partial */ 1118 | int targetOutputSize, /* only used if partialDecoding==partial */ 1119 | int dict, /* noDict, withPrefix64k, usingExtDict */ 1120 | const BYTE* const lowPrefix, /* == dest if dict == noDict */ 1121 | const BYTE* const dictStart, /* only if dict==usingExtDict */ 1122 | const size_t dictSize /* note : = 0 if noDict */ 1123 | ) 1124 | { 1125 | /* Local Variables */ 1126 | const BYTE* ip = (const BYTE*) source; 1127 | const BYTE* const iend = ip + inputSize; 1128 | 1129 | BYTE* op = (BYTE*) dest; 1130 | BYTE* const oend = op + outputSize; 1131 | BYTE* cpy; 1132 | BYTE* oexit = op + targetOutputSize; 1133 | const BYTE* const lowLimit = lowPrefix - dictSize; 1134 | 1135 | const BYTE* const dictEnd = (const BYTE*)dictStart + dictSize; 1136 | const size_t dec32table[] = {4, 1, 2, 1, 4, 4, 4, 4}; 1137 | const size_t dec64table[] = {0, 0, 0, (size_t)-1, 0, 1, 2, 3}; 1138 | 1139 | const int safeDecode = (endOnInput==endOnInputSize); 1140 | const int checkOffset = ((safeDecode) && (dictSize < (int)(64 KB))); 1141 | 1142 | 1143 | /* Special cases */ 1144 | if ((partialDecoding) && (oexit> oend-MFLIMIT)) oexit = oend-MFLIMIT; /* targetOutputSize too high => decode everything */ 1145 | if ((endOnInput) && (unlikely(outputSize==0))) return ((inputSize==1) && (*ip==0)) ? 0 : -1; /* Empty output buffer */ 1146 | if ((!endOnInput) && (unlikely(outputSize==0))) return (*ip==0?1:-1); 1147 | 1148 | 1149 | /* Main Loop */ 1150 | while (1) 1151 | { 1152 | unsigned token; 1153 | size_t length; 1154 | const BYTE* match; 1155 | 1156 | /* get literal length */ 1157 | token = *ip++; 1158 | if ((length=(token>>ML_BITS)) == RUN_MASK) 1159 | { 1160 | unsigned s; 1161 | do 1162 | { 1163 | s = *ip++; 1164 | length += s; 1165 | } 1166 | while (likely((endOnInput)?ip(partialDecoding?oexit:oend-MFLIMIT)) || (ip+length>iend-(2+1+LASTLITERALS))) ) 1174 | || ((!endOnInput) && (cpy>oend-COPYLENGTH))) 1175 | { 1176 | if (partialDecoding) 1177 | { 1178 | if (cpy > oend) goto _output_error; /* Error : write attempt beyond end of output buffer */ 1179 | if ((endOnInput) && (ip+length > iend)) goto _output_error; /* Error : read attempt beyond end of input buffer */ 1180 | } 1181 | else 1182 | { 1183 | if ((!endOnInput) && (cpy != oend)) goto _output_error; /* Error : block decoding must stop exactly there */ 1184 | if ((endOnInput) && ((ip+length != iend) || (cpy > oend))) goto _output_error; /* Error : input must be consumed */ 1185 | } 1186 | memcpy(op, ip, length); 1187 | ip += length; 1188 | op += length; 1189 | break; /* Necessarily EOF, due to parsing restrictions */ 1190 | } 1191 | LZ4_wildCopy(op, ip, cpy); 1192 | ip += length; op = cpy; 1193 | 1194 | /* get offset */ 1195 | match = cpy - LZ4_readLE16(ip); ip+=2; 1196 | if ((checkOffset) && (unlikely(match < lowLimit))) goto _output_error; /* Error : offset outside destination buffer */ 1197 | 1198 | /* get matchlength */ 1199 | length = token & ML_MASK; 1200 | if (length == ML_MASK) 1201 | { 1202 | unsigned s; 1203 | do 1204 | { 1205 | if ((endOnInput) && (ip > iend-LASTLITERALS)) goto _output_error; 1206 | s = *ip++; 1207 | length += s; 1208 | } while (s==255); 1209 | if ((safeDecode) && unlikely((size_t)(op+length)<(size_t)op)) goto _output_error; /* overflow detection */ 1210 | } 1211 | length += MINMATCH; 1212 | 1213 | /* check external dictionary */ 1214 | if ((dict==usingExtDict) && (match < lowPrefix)) 1215 | { 1216 | if (unlikely(op+length > oend-LASTLITERALS)) goto _output_error; /* doesn't respect parsing restriction */ 1217 | 1218 | if (length <= (size_t)(lowPrefix-match)) 1219 | { 1220 | /* match can be copied as a single segment from external dictionary */ 1221 | match = dictEnd - (lowPrefix-match); 1222 | memmove(op, match, length); op += length; 1223 | } 1224 | else 1225 | { 1226 | /* match encompass external dictionary and current segment */ 1227 | size_t copySize = (size_t)(lowPrefix-match); 1228 | memcpy(op, dictEnd - copySize, copySize); 1229 | op += copySize; 1230 | copySize = length - copySize; 1231 | if (copySize > (size_t)(op-lowPrefix)) /* overlap within current segment */ 1232 | { 1233 | BYTE* const endOfMatch = op + copySize; 1234 | const BYTE* copyFrom = lowPrefix; 1235 | while (op < endOfMatch) *op++ = *copyFrom++; 1236 | } 1237 | else 1238 | { 1239 | memcpy(op, lowPrefix, copySize); 1240 | op += copySize; 1241 | } 1242 | } 1243 | continue; 1244 | } 1245 | 1246 | /* copy repeated sequence */ 1247 | cpy = op + length; 1248 | if (unlikely((op-match)<8)) 1249 | { 1250 | const size_t dec64 = dec64table[op-match]; 1251 | op[0] = match[0]; 1252 | op[1] = match[1]; 1253 | op[2] = match[2]; 1254 | op[3] = match[3]; 1255 | match += dec32table[op-match]; 1256 | LZ4_copy4(op+4, match); 1257 | op += 8; match -= dec64; 1258 | } else { LZ4_copy8(op, match); op+=8; match+=8; } 1259 | 1260 | if (unlikely(cpy>oend-12)) 1261 | { 1262 | if (cpy > oend-LASTLITERALS) goto _output_error; /* Error : last LASTLITERALS bytes must be literals */ 1263 | if (op < oend-8) 1264 | { 1265 | LZ4_wildCopy(op, match, oend-8); 1266 | match += (oend-8) - op; 1267 | op = oend-8; 1268 | } 1269 | while (opprefixSize = (size_t) dictSize; 1342 | lz4sd->prefixEnd = (const BYTE*) dictionary + dictSize; 1343 | lz4sd->externalDict = NULL; 1344 | lz4sd->extDictSize = 0; 1345 | return 1; 1346 | } 1347 | 1348 | /* 1349 | *_continue() : 1350 | These decoding functions allow decompression of multiple blocks in "streaming" mode. 1351 | Previously decoded blocks must still be available at the memory position where they were decoded. 1352 | If it's not possible, save the relevant part of decoded data into a safe buffer, 1353 | and indicate where it stands using LZ4_setStreamDecode() 1354 | */ 1355 | int LZ4_decompress_safe_continue (LZ4_streamDecode_t* LZ4_streamDecode, const char* source, char* dest, int compressedSize, int maxOutputSize) 1356 | { 1357 | LZ4_streamDecode_t_internal* lz4sd = (LZ4_streamDecode_t_internal*) LZ4_streamDecode; 1358 | int result; 1359 | 1360 | if (lz4sd->prefixEnd == (BYTE*)dest) 1361 | { 1362 | result = LZ4_decompress_generic(source, dest, compressedSize, maxOutputSize, 1363 | endOnInputSize, full, 0, 1364 | usingExtDict, lz4sd->prefixEnd - lz4sd->prefixSize, lz4sd->externalDict, lz4sd->extDictSize); 1365 | if (result <= 0) return result; 1366 | lz4sd->prefixSize += result; 1367 | lz4sd->prefixEnd += result; 1368 | } 1369 | else 1370 | { 1371 | lz4sd->extDictSize = lz4sd->prefixSize; 1372 | lz4sd->externalDict = lz4sd->prefixEnd - lz4sd->extDictSize; 1373 | result = LZ4_decompress_generic(source, dest, compressedSize, maxOutputSize, 1374 | endOnInputSize, full, 0, 1375 | usingExtDict, (BYTE*)dest, lz4sd->externalDict, lz4sd->extDictSize); 1376 | if (result <= 0) return result; 1377 | lz4sd->prefixSize = result; 1378 | lz4sd->prefixEnd = (BYTE*)dest + result; 1379 | } 1380 | 1381 | return result; 1382 | } 1383 | 1384 | int LZ4_decompress_fast_continue (LZ4_streamDecode_t* LZ4_streamDecode, const char* source, char* dest, int originalSize) 1385 | { 1386 | LZ4_streamDecode_t_internal* lz4sd = (LZ4_streamDecode_t_internal*) LZ4_streamDecode; 1387 | int result; 1388 | 1389 | if (lz4sd->prefixEnd == (BYTE*)dest) 1390 | { 1391 | result = LZ4_decompress_generic(source, dest, 0, originalSize, 1392 | endOnOutputSize, full, 0, 1393 | usingExtDict, lz4sd->prefixEnd - lz4sd->prefixSize, lz4sd->externalDict, lz4sd->extDictSize); 1394 | if (result <= 0) return result; 1395 | lz4sd->prefixSize += originalSize; 1396 | lz4sd->prefixEnd += originalSize; 1397 | } 1398 | else 1399 | { 1400 | lz4sd->extDictSize = lz4sd->prefixSize; 1401 | lz4sd->externalDict = (BYTE*)dest - lz4sd->extDictSize; 1402 | result = LZ4_decompress_generic(source, dest, 0, originalSize, 1403 | endOnOutputSize, full, 0, 1404 | usingExtDict, (BYTE*)dest, lz4sd->externalDict, lz4sd->extDictSize); 1405 | if (result <= 0) return result; 1406 | lz4sd->prefixSize = originalSize; 1407 | lz4sd->prefixEnd = (BYTE*)dest + originalSize; 1408 | } 1409 | 1410 | return result; 1411 | } 1412 | 1413 | 1414 | /* 1415 | Advanced decoding functions : 1416 | *_usingDict() : 1417 | These decoding functions work the same as "_continue" ones, 1418 | the dictionary must be explicitly provided within parameters 1419 | */ 1420 | 1421 | FORCE_INLINE int LZ4_decompress_usingDict_generic(const char* source, char* dest, int compressedSize, int maxOutputSize, int safe, const char* dictStart, int dictSize) 1422 | { 1423 | if (dictSize==0) 1424 | return LZ4_decompress_generic(source, dest, compressedSize, maxOutputSize, safe, full, 0, noDict, (BYTE*)dest, NULL, 0); 1425 | if (dictStart+dictSize == dest) 1426 | { 1427 | if (dictSize >= (int)(64 KB - 1)) 1428 | return LZ4_decompress_generic(source, dest, compressedSize, maxOutputSize, safe, full, 0, withPrefix64k, (BYTE*)dest-64 KB, NULL, 0); 1429 | return LZ4_decompress_generic(source, dest, compressedSize, maxOutputSize, safe, full, 0, noDict, (BYTE*)dest-dictSize, NULL, 0); 1430 | } 1431 | return LZ4_decompress_generic(source, dest, compressedSize, maxOutputSize, safe, full, 0, usingExtDict, (BYTE*)dest, (const BYTE*)dictStart, dictSize); 1432 | } 1433 | 1434 | int LZ4_decompress_safe_usingDict(const char* source, char* dest, int compressedSize, int maxOutputSize, const char* dictStart, int dictSize) 1435 | { 1436 | return LZ4_decompress_usingDict_generic(source, dest, compressedSize, maxOutputSize, 1, dictStart, dictSize); 1437 | } 1438 | 1439 | int LZ4_decompress_fast_usingDict(const char* source, char* dest, int originalSize, const char* dictStart, int dictSize) 1440 | { 1441 | return LZ4_decompress_usingDict_generic(source, dest, 0, originalSize, 0, dictStart, dictSize); 1442 | } 1443 | 1444 | /* debug function */ 1445 | int LZ4_decompress_safe_forceExtDict(const char* source, char* dest, int compressedSize, int maxOutputSize, const char* dictStart, int dictSize) 1446 | { 1447 | return LZ4_decompress_generic(source, dest, compressedSize, maxOutputSize, endOnInputSize, full, 0, usingExtDict, (BYTE*)dest, (const BYTE*)dictStart, dictSize); 1448 | } 1449 | 1450 | 1451 | /*************************************************** 1452 | * Obsolete Functions 1453 | ***************************************************/ 1454 | /* obsolete compression functions */ 1455 | int LZ4_compress_limitedOutput(const char* source, char* dest, int inputSize, int maxOutputSize) { return LZ4_compress_default(source, dest, inputSize, maxOutputSize); } 1456 | int LZ4_compress(const char* source, char* dest, int inputSize) { return LZ4_compress_default(source, dest, inputSize, LZ4_compressBound(inputSize)); } 1457 | int LZ4_compress_limitedOutput_withState (void* state, const char* src, char* dst, int srcSize, int dstSize) { return LZ4_compress_fast_extState(state, src, dst, srcSize, dstSize, 1); } 1458 | int LZ4_compress_withState (void* state, const char* src, char* dst, int srcSize) { return LZ4_compress_fast_extState(state, src, dst, srcSize, LZ4_compressBound(srcSize), 1); } 1459 | int LZ4_compress_limitedOutput_continue (LZ4_stream_t* LZ4_stream, const char* src, char* dst, int srcSize, int maxDstSize) { return LZ4_compress_fast_continue(LZ4_stream, src, dst, srcSize, maxDstSize, 1); } 1460 | int LZ4_compress_continue (LZ4_stream_t* LZ4_stream, const char* source, char* dest, int inputSize) { return LZ4_compress_fast_continue(LZ4_stream, source, dest, inputSize, LZ4_compressBound(inputSize), 1); } 1461 | 1462 | /* 1463 | These function names are deprecated and should no longer be used. 1464 | They are only provided here for compatibility with older user programs. 1465 | - LZ4_uncompress is totally equivalent to LZ4_decompress_fast 1466 | - LZ4_uncompress_unknownOutputSize is totally equivalent to LZ4_decompress_safe 1467 | */ 1468 | int LZ4_uncompress (const char* source, char* dest, int outputSize) { return LZ4_decompress_fast(source, dest, outputSize); } 1469 | int LZ4_uncompress_unknownOutputSize (const char* source, char* dest, int isize, int maxOutputSize) { return LZ4_decompress_safe(source, dest, isize, maxOutputSize); } 1470 | 1471 | 1472 | /* Obsolete Streaming functions */ 1473 | 1474 | int LZ4_sizeofStreamState() { return LZ4_STREAMSIZE; } 1475 | 1476 | static void LZ4_init(LZ4_stream_t_internal* lz4ds, BYTE* base) 1477 | { 1478 | MEM_INIT(lz4ds, 0, LZ4_STREAMSIZE); 1479 | lz4ds->bufferStart = base; 1480 | } 1481 | 1482 | int LZ4_resetStreamState(void* state, char* inputBuffer) 1483 | { 1484 | if ((((size_t)state) & 3) != 0) return 1; /* Error : pointer is not aligned on 4-bytes boundary */ 1485 | LZ4_init((LZ4_stream_t_internal*)state, (BYTE*)inputBuffer); 1486 | return 0; 1487 | } 1488 | 1489 | void* LZ4_create (char* inputBuffer) 1490 | { 1491 | void* lz4ds = ALLOCATOR(8, LZ4_STREAMSIZE_U64); 1492 | LZ4_init ((LZ4_stream_t_internal*)lz4ds, (BYTE*)inputBuffer); 1493 | return lz4ds; 1494 | } 1495 | 1496 | char* LZ4_slideInputBuffer (void* LZ4_Data) 1497 | { 1498 | LZ4_stream_t_internal* ctx = (LZ4_stream_t_internal*)LZ4_Data; 1499 | int dictSize = LZ4_saveDict((LZ4_stream_t*)LZ4_Data, (char*)ctx->bufferStart, 64 KB); 1500 | return (char*)(ctx->bufferStart + dictSize); 1501 | } 1502 | 1503 | /* Obsolete streaming decompression functions */ 1504 | 1505 | int LZ4_decompress_safe_withPrefix64k(const char* source, char* dest, int compressedSize, int maxOutputSize) 1506 | { 1507 | return LZ4_decompress_generic(source, dest, compressedSize, maxOutputSize, endOnInputSize, full, 0, withPrefix64k, (BYTE*)dest - 64 KB, NULL, 64 KB); 1508 | } 1509 | 1510 | int LZ4_decompress_fast_withPrefix64k(const char* source, char* dest, int originalSize) 1511 | { 1512 | return LZ4_decompress_generic(source, dest, 0, originalSize, endOnOutputSize, full, 0, withPrefix64k, (BYTE*)dest - 64 KB, NULL, 64 KB); 1513 | } 1514 | 1515 | #endif /* LZ4_COMMONDEFS_ONLY */ 1516 | 1517 | -------------------------------------------------------------------------------- /src/lz4.h: -------------------------------------------------------------------------------- 1 | /* 2 | LZ4 - Fast LZ compression algorithm 3 | Header File 4 | Copyright (C) 2011-2015, Yann Collet. 5 | 6 | BSD 2-Clause License (http://www.opensource.org/licenses/bsd-license.php) 7 | 8 | Redistribution and use in source and binary forms, with or without 9 | modification, are permitted provided that the following conditions are 10 | met: 11 | 12 | * Redistributions of source code must retain the above copyright 13 | notice, this list of conditions and the following disclaimer. 14 | * Redistributions in binary form must reproduce the above 15 | copyright notice, this list of conditions and the following disclaimer 16 | in the documentation and/or other materials provided with the 17 | distribution. 18 | 19 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 20 | "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT 21 | LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR 22 | A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT 23 | OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, 24 | SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT 25 | LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, 26 | DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY 27 | THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT 28 | (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 29 | OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 30 | 31 | You can contact the author at : 32 | - LZ4 source repository : https://github.com/Cyan4973/lz4 33 | - LZ4 public forum : https://groups.google.com/forum/#!forum/lz4c 34 | */ 35 | #pragma once 36 | 37 | #if defined (__cplusplus) 38 | extern "C" { 39 | #endif 40 | 41 | /* 42 | * lz4.h provides block compression functions, and gives full buffer control to programmer. 43 | * If you need to generate inter-operable compressed data (respecting LZ4 frame specification), 44 | * and can let the library handle its own memory, please use lz4frame.h instead. 45 | */ 46 | 47 | /************************************** 48 | * Version 49 | **************************************/ 50 | #define LZ4_VERSION_MAJOR 1 /* for breaking interface changes */ 51 | #define LZ4_VERSION_MINOR 7 /* for new (non-breaking) interface capabilities */ 52 | #define LZ4_VERSION_RELEASE 1 /* for tweaks, bug-fixes, or development */ 53 | #define LZ4_VERSION_NUMBER (LZ4_VERSION_MAJOR *100*100 + LZ4_VERSION_MINOR *100 + LZ4_VERSION_RELEASE) 54 | int LZ4_versionNumber (void); 55 | 56 | /************************************** 57 | * Tuning parameter 58 | **************************************/ 59 | /* 60 | * LZ4_MEMORY_USAGE : 61 | * Memory usage formula : N->2^N Bytes (examples : 10 -> 1KB; 12 -> 4KB ; 16 -> 64KB; 20 -> 1MB; etc.) 62 | * Increasing memory usage improves compression ratio 63 | * Reduced memory usage can improve speed, due to cache effect 64 | * Default value is 14, for 16KB, which nicely fits into Intel x86 L1 cache 65 | */ 66 | #define LZ4_MEMORY_USAGE 14 67 | 68 | 69 | /************************************** 70 | * Simple Functions 71 | **************************************/ 72 | 73 | int LZ4_compress_default(const char* source, char* dest, int sourceSize, int maxDestSize); 74 | int LZ4_decompress_safe (const char* source, char* dest, int compressedSize, int maxDecompressedSize); 75 | 76 | /* 77 | LZ4_compress_default() : 78 | Compresses 'sourceSize' bytes from buffer 'source' 79 | into already allocated 'dest' buffer of size 'maxDestSize'. 80 | Compression is guaranteed to succeed if 'maxDestSize' >= LZ4_compressBound(sourceSize). 81 | It also runs faster, so it's a recommended setting. 82 | If the function cannot compress 'source' into a more limited 'dest' budget, 83 | compression stops *immediately*, and the function result is zero. 84 | As a consequence, 'dest' content is not valid. 85 | This function never writes outside 'dest' buffer, nor read outside 'source' buffer. 86 | sourceSize : Max supported value is LZ4_MAX_INPUT_VALUE 87 | maxDestSize : full or partial size of buffer 'dest' (which must be already allocated) 88 | return : the number of bytes written into buffer 'dest' (necessarily <= maxOutputSize) 89 | or 0 if compression fails 90 | 91 | LZ4_decompress_safe() : 92 | compressedSize : is the precise full size of the compressed block. 93 | maxDecompressedSize : is the size of destination buffer, which must be already allocated. 94 | return : the number of bytes decompressed into destination buffer (necessarily <= maxDecompressedSize) 95 | If destination buffer is not large enough, decoding will stop and output an error code (<0). 96 | If the source stream is detected malformed, the function will stop decoding and return a negative result. 97 | This function is protected against buffer overflow exploits, including malicious data packets. 98 | It never writes outside output buffer, nor reads outside input buffer. 99 | */ 100 | 101 | 102 | /************************************** 103 | * Advanced Functions 104 | **************************************/ 105 | #define LZ4_MAX_INPUT_SIZE 0x7E000000 /* 2 113 929 216 bytes */ 106 | #define LZ4_COMPRESSBOUND(isize) ((unsigned)(isize) > (unsigned)LZ4_MAX_INPUT_SIZE ? 0 : (isize) + ((isize)/255) + 16) 107 | 108 | /* 109 | LZ4_compressBound() : 110 | Provides the maximum size that LZ4 compression may output in a "worst case" scenario (input data not compressible) 111 | This function is primarily useful for memory allocation purposes (destination buffer size). 112 | Macro LZ4_COMPRESSBOUND() is also provided for compilation-time evaluation (stack memory allocation for example). 113 | Note that LZ4_compress_default() compress faster when dest buffer size is >= LZ4_compressBound(srcSize) 114 | inputSize : max supported value is LZ4_MAX_INPUT_SIZE 115 | return : maximum output size in a "worst case" scenario 116 | or 0, if input size is too large ( > LZ4_MAX_INPUT_SIZE) 117 | */ 118 | int LZ4_compressBound(int inputSize); 119 | 120 | /* 121 | LZ4_compress_fast() : 122 | Same as LZ4_compress_default(), but allows to select an "acceleration" factor. 123 | The larger the acceleration value, the faster the algorithm, but also the lesser the compression. 124 | It's a trade-off. It can be fine tuned, with each successive value providing roughly +~3% to speed. 125 | An acceleration value of "1" is the same as regular LZ4_compress_default() 126 | Values <= 0 will be replaced by ACCELERATION_DEFAULT (see lz4.c), which is 1. 127 | */ 128 | int LZ4_compress_fast (const char* source, char* dest, int sourceSize, int maxDestSize, int acceleration); 129 | 130 | 131 | /* 132 | LZ4_compress_fast_extState() : 133 | Same compression function, just using an externally allocated memory space to store compression state. 134 | Use LZ4_sizeofState() to know how much memory must be allocated, 135 | and allocate it on 8-bytes boundaries (using malloc() typically). 136 | Then, provide it as 'void* state' to compression function. 137 | */ 138 | int LZ4_sizeofState(void); 139 | int LZ4_compress_fast_extState (void* state, const char* source, char* dest, int inputSize, int maxDestSize, int acceleration); 140 | 141 | 142 | /* 143 | LZ4_compress_destSize() : 144 | Reverse the logic, by compressing as much data as possible from 'source' buffer 145 | into already allocated buffer 'dest' of size 'targetDestSize'. 146 | This function either compresses the entire 'source' content into 'dest' if it's large enough, 147 | or fill 'dest' buffer completely with as much data as possible from 'source'. 148 | *sourceSizePtr : will be modified to indicate how many bytes where read from 'source' to fill 'dest'. 149 | New value is necessarily <= old value. 150 | return : Nb bytes written into 'dest' (necessarily <= targetDestSize) 151 | or 0 if compression fails 152 | */ 153 | int LZ4_compress_destSize (const char* source, char* dest, int* sourceSizePtr, int targetDestSize); 154 | 155 | 156 | /* 157 | LZ4_decompress_fast() : 158 | originalSize : is the original and therefore uncompressed size 159 | return : the number of bytes read from the source buffer (in other words, the compressed size) 160 | If the source stream is detected malformed, the function will stop decoding and return a negative result. 161 | Destination buffer must be already allocated. Its size must be a minimum of 'originalSize' bytes. 162 | note : This function fully respect memory boundaries for properly formed compressed data. 163 | It is a bit faster than LZ4_decompress_safe(). 164 | However, it does not provide any protection against intentionally modified data stream (malicious input). 165 | Use this function in trusted environment only (data to decode comes from a trusted source). 166 | */ 167 | int LZ4_decompress_fast (const char* source, char* dest, int originalSize); 168 | 169 | /* 170 | LZ4_decompress_safe_partial() : 171 | This function decompress a compressed block of size 'compressedSize' at position 'source' 172 | into destination buffer 'dest' of size 'maxDecompressedSize'. 173 | The function tries to stop decompressing operation as soon as 'targetOutputSize' has been reached, 174 | reducing decompression time. 175 | return : the number of bytes decoded in the destination buffer (necessarily <= maxDecompressedSize) 176 | Note : this number can be < 'targetOutputSize' should the compressed block to decode be smaller. 177 | Always control how many bytes were decoded. 178 | If the source stream is detected malformed, the function will stop decoding and return a negative result. 179 | This function never writes outside of output buffer, and never reads outside of input buffer. It is therefore protected against malicious data packets 180 | */ 181 | int LZ4_decompress_safe_partial (const char* source, char* dest, int compressedSize, int targetOutputSize, int maxDecompressedSize); 182 | 183 | 184 | /*********************************************** 185 | * Streaming Compression Functions 186 | ***********************************************/ 187 | #define LZ4_STREAMSIZE_U64 ((1 << (LZ4_MEMORY_USAGE-3)) + 4) 188 | #define LZ4_STREAMSIZE (LZ4_STREAMSIZE_U64 * sizeof(long long)) 189 | /* 190 | * LZ4_stream_t 191 | * information structure to track an LZ4 stream. 192 | * important : init this structure content before first use ! 193 | * note : only allocated directly the structure if you are statically linking LZ4 194 | * If you are using liblz4 as a DLL, please use below construction methods instead. 195 | */ 196 | typedef struct { long long table[LZ4_STREAMSIZE_U64]; } LZ4_stream_t; 197 | 198 | /* 199 | * LZ4_resetStream 200 | * Use this function to init an allocated LZ4_stream_t structure 201 | */ 202 | void LZ4_resetStream (LZ4_stream_t* streamPtr); 203 | 204 | /* 205 | * LZ4_createStream will allocate and initialize an LZ4_stream_t structure 206 | * LZ4_freeStream releases its memory. 207 | * In the context of a DLL (liblz4), please use these methods rather than the static struct. 208 | * They are more future proof, in case of a change of LZ4_stream_t size. 209 | */ 210 | LZ4_stream_t* LZ4_createStream(void); 211 | int LZ4_freeStream (LZ4_stream_t* streamPtr); 212 | 213 | /* 214 | * LZ4_loadDict 215 | * Use this function to load a static dictionary into LZ4_stream. 216 | * Any previous data will be forgotten, only 'dictionary' will remain in memory. 217 | * Loading a size of 0 is allowed. 218 | * Return : dictionary size, in bytes (necessarily <= 64 KB) 219 | */ 220 | int LZ4_loadDict (LZ4_stream_t* streamPtr, const char* dictionary, int dictSize); 221 | 222 | /* 223 | * LZ4_compress_fast_continue 224 | * Compress buffer content 'src', using data from previously compressed blocks as dictionary to improve compression ratio. 225 | * Important : Previous data blocks are assumed to still be present and unmodified ! 226 | * 'dst' buffer must be already allocated. 227 | * If maxDstSize >= LZ4_compressBound(srcSize), compression is guaranteed to succeed, and runs faster. 228 | * If not, and if compressed data cannot fit into 'dst' buffer size, compression stops, and function returns a zero. 229 | */ 230 | int LZ4_compress_fast_continue (LZ4_stream_t* streamPtr, const char* src, char* dst, int srcSize, int maxDstSize, int acceleration); 231 | 232 | /* 233 | * LZ4_saveDict 234 | * If previously compressed data block is not guaranteed to remain available at its memory location 235 | * save it into a safer place (char* safeBuffer) 236 | * Note : you don't need to call LZ4_loadDict() afterwards, 237 | * dictionary is immediately usable, you can therefore call LZ4_compress_fast_continue() 238 | * Return : saved dictionary size in bytes (necessarily <= dictSize), or 0 if error 239 | */ 240 | int LZ4_saveDict (LZ4_stream_t* streamPtr, char* safeBuffer, int dictSize); 241 | 242 | 243 | /************************************************ 244 | * Streaming Decompression Functions 245 | ************************************************/ 246 | 247 | #define LZ4_STREAMDECODESIZE_U64 4 248 | #define LZ4_STREAMDECODESIZE (LZ4_STREAMDECODESIZE_U64 * sizeof(unsigned long long)) 249 | typedef struct { unsigned long long table[LZ4_STREAMDECODESIZE_U64]; } LZ4_streamDecode_t; 250 | /* 251 | * LZ4_streamDecode_t 252 | * information structure to track an LZ4 stream. 253 | * init this structure content using LZ4_setStreamDecode or memset() before first use ! 254 | * 255 | * In the context of a DLL (liblz4) please prefer usage of construction methods below. 256 | * They are more future proof, in case of a change of LZ4_streamDecode_t size in the future. 257 | * LZ4_createStreamDecode will allocate and initialize an LZ4_streamDecode_t structure 258 | * LZ4_freeStreamDecode releases its memory. 259 | */ 260 | LZ4_streamDecode_t* LZ4_createStreamDecode(void); 261 | int LZ4_freeStreamDecode (LZ4_streamDecode_t* LZ4_stream); 262 | 263 | /* 264 | * LZ4_setStreamDecode 265 | * Use this function to instruct where to find the dictionary. 266 | * Setting a size of 0 is allowed (same effect as reset). 267 | * Return : 1 if OK, 0 if error 268 | */ 269 | int LZ4_setStreamDecode (LZ4_streamDecode_t* LZ4_streamDecode, const char* dictionary, int dictSize); 270 | 271 | /* 272 | *_continue() : 273 | These decoding functions allow decompression of multiple blocks in "streaming" mode. 274 | Previously decoded blocks *must* remain available at the memory position where they were decoded (up to 64 KB) 275 | In the case of a ring buffers, decoding buffer must be either : 276 | - Exactly same size as encoding buffer, with same update rule (block boundaries at same positions) 277 | In which case, the decoding & encoding ring buffer can have any size, including very small ones ( < 64 KB). 278 | - Larger than encoding buffer, by a minimum of maxBlockSize more bytes. 279 | maxBlockSize is implementation dependent. It's the maximum size you intend to compress into a single block. 280 | In which case, encoding and decoding buffers do not need to be synchronized, 281 | and encoding ring buffer can have any size, including small ones ( < 64 KB). 282 | - _At least_ 64 KB + 8 bytes + maxBlockSize. 283 | In which case, encoding and decoding buffers do not need to be synchronized, 284 | and encoding ring buffer can have any size, including larger than decoding buffer. 285 | Whenever these conditions are not possible, save the last 64KB of decoded data into a safe buffer, 286 | and indicate where it is saved using LZ4_setStreamDecode() 287 | */ 288 | int LZ4_decompress_safe_continue (LZ4_streamDecode_t* LZ4_streamDecode, const char* source, char* dest, int compressedSize, int maxDecompressedSize); 289 | int LZ4_decompress_fast_continue (LZ4_streamDecode_t* LZ4_streamDecode, const char* source, char* dest, int originalSize); 290 | 291 | 292 | /* 293 | Advanced decoding functions : 294 | *_usingDict() : 295 | These decoding functions work the same as 296 | a combination of LZ4_setStreamDecode() followed by LZ4_decompress_x_continue() 297 | They are stand-alone. They don't need nor update an LZ4_streamDecode_t structure. 298 | */ 299 | int LZ4_decompress_safe_usingDict (const char* source, char* dest, int compressedSize, int maxDecompressedSize, const char* dictStart, int dictSize); 300 | int LZ4_decompress_fast_usingDict (const char* source, char* dest, int originalSize, const char* dictStart, int dictSize); 301 | 302 | 303 | 304 | /************************************** 305 | * Obsolete Functions 306 | **************************************/ 307 | /* Deprecate Warnings */ 308 | /* Should these warnings messages be a problem, 309 | it is generally possible to disable them, 310 | with -Wno-deprecated-declarations for gcc 311 | or _CRT_SECURE_NO_WARNINGS in Visual for example. 312 | You can also define LZ4_DEPRECATE_WARNING_DEFBLOCK. */ 313 | #ifndef LZ4_DEPRECATE_WARNING_DEFBLOCK 314 | # define LZ4_DEPRECATE_WARNING_DEFBLOCK 315 | # define LZ4_GCC_VERSION (__GNUC__ * 100 + __GNUC_MINOR__) 316 | # if (LZ4_GCC_VERSION >= 405) || defined(__clang__) 317 | # define LZ4_DEPRECATED(message) __attribute__((deprecated(message))) 318 | # elif (LZ4_GCC_VERSION >= 301) 319 | # define LZ4_DEPRECATED(message) __attribute__((deprecated)) 320 | # elif defined(_MSC_VER) 321 | # define LZ4_DEPRECATED(message) __declspec(deprecated(message)) 322 | # else 323 | # pragma message("WARNING: You need to implement LZ4_DEPRECATED for this compiler") 324 | # define LZ4_DEPRECATED(message) 325 | # endif 326 | #endif /* LZ4_DEPRECATE_WARNING_DEFBLOCK */ 327 | 328 | /* Obsolete compression functions */ 329 | /* These functions are planned to start generate warnings by r131 approximately */ 330 | int LZ4_compress (const char* source, char* dest, int sourceSize); 331 | int LZ4_compress_limitedOutput (const char* source, char* dest, int sourceSize, int maxOutputSize); 332 | int LZ4_compress_withState (void* state, const char* source, char* dest, int inputSize); 333 | int LZ4_compress_limitedOutput_withState (void* state, const char* source, char* dest, int inputSize, int maxOutputSize); 334 | int LZ4_compress_continue (LZ4_stream_t* LZ4_streamPtr, const char* source, char* dest, int inputSize); 335 | int LZ4_compress_limitedOutput_continue (LZ4_stream_t* LZ4_streamPtr, const char* source, char* dest, int inputSize, int maxOutputSize); 336 | 337 | /* Obsolete decompression functions */ 338 | /* These function names are completely deprecated and must no longer be used. 339 | They are only provided here for compatibility with older programs. 340 | - LZ4_uncompress is the same as LZ4_decompress_fast 341 | - LZ4_uncompress_unknownOutputSize is the same as LZ4_decompress_safe 342 | These function prototypes are now disabled; uncomment them only if you really need them. 343 | It is highly recommended to stop using these prototypes and migrate to maintained ones */ 344 | /* int LZ4_uncompress (const char* source, char* dest, int outputSize); */ 345 | /* int LZ4_uncompress_unknownOutputSize (const char* source, char* dest, int isize, int maxOutputSize); */ 346 | 347 | /* Obsolete streaming functions; use new streaming interface whenever possible */ 348 | LZ4_DEPRECATED("use LZ4_createStream() instead") void* LZ4_create (char* inputBuffer); 349 | LZ4_DEPRECATED("use LZ4_createStream() instead") int LZ4_sizeofStreamState(void); 350 | LZ4_DEPRECATED("use LZ4_resetStream() instead") int LZ4_resetStreamState(void* state, char* inputBuffer); 351 | LZ4_DEPRECATED("use LZ4_saveDict() instead") char* LZ4_slideInputBuffer (void* state); 352 | 353 | /* Obsolete streaming decoding functions */ 354 | LZ4_DEPRECATED("use LZ4_decompress_safe_usingDict() instead") int LZ4_decompress_safe_withPrefix64k (const char* src, char* dst, int compressedSize, int maxDstSize); 355 | LZ4_DEPRECATED("use LZ4_decompress_fast_usingDict() instead") int LZ4_decompress_fast_withPrefix64k (const char* src, char* dst, int originalSize); 356 | 357 | 358 | #if defined (__cplusplus) 359 | } 360 | #endif 361 | -------------------------------------------------------------------------------- /src/lz4frame.h: -------------------------------------------------------------------------------- 1 | /* 2 | LZ4 auto-framing library 3 | Header File 4 | Copyright (C) 2011-2015, Yann Collet. 5 | BSD 2-Clause License (http://www.opensource.org/licenses/bsd-license.php) 6 | 7 | Redistribution and use in source and binary forms, with or without 8 | modification, are permitted provided that the following conditions are 9 | met: 10 | 11 | * Redistributions of source code must retain the above copyright 12 | notice, this list of conditions and the following disclaimer. 13 | * Redistributions in binary form must reproduce the above 14 | copyright notice, this list of conditions and the following disclaimer 15 | in the documentation and/or other materials provided with the 16 | distribution. 17 | 18 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 19 | "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT 20 | LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR 21 | A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT 22 | OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, 23 | SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT 24 | LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, 25 | DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY 26 | THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT 27 | (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 28 | OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 29 | 30 | You can contact the author at : 31 | - LZ4 source repository : https://github.com/Cyan4973/lz4 32 | - LZ4 public forum : https://groups.google.com/forum/#!forum/lz4c 33 | */ 34 | 35 | /* LZ4F is a stand-alone API to create LZ4-compressed frames 36 | * fully conformant to specification v1.5.1. 37 | * All related operations, including memory management, are handled by the library. 38 | * You don't need lz4.h when using lz4frame.h. 39 | * */ 40 | 41 | #pragma once 42 | 43 | #if defined (__cplusplus) 44 | extern "C" { 45 | #endif 46 | 47 | /************************************** 48 | * Includes 49 | **************************************/ 50 | #include /* size_t */ 51 | 52 | 53 | /************************************** 54 | * Error management 55 | * ************************************/ 56 | typedef size_t LZ4F_errorCode_t; 57 | 58 | unsigned LZ4F_isError(LZ4F_errorCode_t code); 59 | const char* LZ4F_getErrorName(LZ4F_errorCode_t code); /* return error code string; useful for debugging */ 60 | 61 | 62 | /************************************** 63 | * Frame compression types 64 | * ************************************/ 65 | //#define LZ4F_DISABLE_OBSOLETE_ENUMS 66 | #ifndef LZ4F_DISABLE_OBSOLETE_ENUMS 67 | # define LZ4F_OBSOLETE_ENUM(x) ,x 68 | #else 69 | # define LZ4F_OBSOLETE_ENUM(x) 70 | #endif 71 | 72 | typedef enum { 73 | LZ4F_default=0, 74 | LZ4F_max64KB=4, 75 | LZ4F_max256KB=5, 76 | LZ4F_max1MB=6, 77 | LZ4F_max4MB=7 78 | LZ4F_OBSOLETE_ENUM(max64KB = LZ4F_max64KB) 79 | LZ4F_OBSOLETE_ENUM(max256KB = LZ4F_max256KB) 80 | LZ4F_OBSOLETE_ENUM(max1MB = LZ4F_max1MB) 81 | LZ4F_OBSOLETE_ENUM(max4MB = LZ4F_max4MB) 82 | } LZ4F_blockSizeID_t; 83 | 84 | typedef enum { 85 | LZ4F_blockLinked=0, 86 | LZ4F_blockIndependent 87 | LZ4F_OBSOLETE_ENUM(blockLinked = LZ4F_blockLinked) 88 | LZ4F_OBSOLETE_ENUM(blockIndependent = LZ4F_blockIndependent) 89 | } LZ4F_blockMode_t; 90 | 91 | typedef enum { 92 | LZ4F_noContentChecksum=0, 93 | LZ4F_contentChecksumEnabled 94 | LZ4F_OBSOLETE_ENUM(noContentChecksum = LZ4F_noContentChecksum) 95 | LZ4F_OBSOLETE_ENUM(contentChecksumEnabled = LZ4F_contentChecksumEnabled) 96 | } LZ4F_contentChecksum_t; 97 | 98 | typedef enum { 99 | LZ4F_frame=0, 100 | LZ4F_skippableFrame 101 | LZ4F_OBSOLETE_ENUM(skippableFrame = LZ4F_skippableFrame) 102 | } LZ4F_frameType_t; 103 | 104 | #ifndef LZ4F_DISABLE_OBSOLETE_ENUMS 105 | typedef LZ4F_blockSizeID_t blockSizeID_t; 106 | typedef LZ4F_blockMode_t blockMode_t; 107 | typedef LZ4F_frameType_t frameType_t; 108 | typedef LZ4F_contentChecksum_t contentChecksum_t; 109 | #endif 110 | 111 | typedef struct { 112 | LZ4F_blockSizeID_t blockSizeID; /* max64KB, max256KB, max1MB, max4MB ; 0 == default */ 113 | LZ4F_blockMode_t blockMode; /* blockLinked, blockIndependent ; 0 == default */ 114 | LZ4F_contentChecksum_t contentChecksumFlag; /* noContentChecksum, contentChecksumEnabled ; 0 == default */ 115 | LZ4F_frameType_t frameType; /* LZ4F_frame, skippableFrame ; 0 == default */ 116 | unsigned long long contentSize; /* Size of uncompressed (original) content ; 0 == unknown */ 117 | unsigned reserved[2]; /* must be zero for forward compatibility */ 118 | } LZ4F_frameInfo_t; 119 | 120 | typedef struct { 121 | LZ4F_frameInfo_t frameInfo; 122 | int compressionLevel; /* 0 == default (fast mode); values above 16 count as 16; values below 0 count as 0 */ 123 | unsigned autoFlush; /* 1 == always flush (reduce need for tmp buffer) */ 124 | unsigned reserved[4]; /* must be zero for forward compatibility */ 125 | } LZ4F_preferences_t; 126 | 127 | 128 | /*********************************** 129 | * Simple compression function 130 | * *********************************/ 131 | size_t LZ4F_compressFrameBound(size_t srcSize, const LZ4F_preferences_t* preferencesPtr); 132 | 133 | size_t LZ4F_compressFrame(void* dstBuffer, size_t dstMaxSize, const void* srcBuffer, size_t srcSize, const LZ4F_preferences_t* preferencesPtr); 134 | /* LZ4F_compressFrame() 135 | * Compress an entire srcBuffer into a valid LZ4 frame, as defined by specification v1.5.1 136 | * The most important rule is that dstBuffer MUST be large enough (dstMaxSize) to ensure compression completion even in worst case. 137 | * You can get the minimum value of dstMaxSize by using LZ4F_compressFrameBound() 138 | * If this condition is not respected, LZ4F_compressFrame() will fail (result is an errorCode) 139 | * The LZ4F_preferences_t structure is optional : you can provide NULL as argument. All preferences will be set to default. 140 | * The result of the function is the number of bytes written into dstBuffer. 141 | * The function outputs an error code if it fails (can be tested using LZ4F_isError()) 142 | */ 143 | 144 | 145 | 146 | /********************************** 147 | * Advanced compression functions 148 | **********************************/ 149 | typedef struct LZ4F_cctx_s* LZ4F_compressionContext_t; /* must be aligned on 8-bytes */ 150 | 151 | typedef struct { 152 | unsigned stableSrc; /* 1 == src content will remain available on future calls to LZ4F_compress(); avoid saving src content within tmp buffer as future dictionary */ 153 | unsigned reserved[3]; 154 | } LZ4F_compressOptions_t; 155 | 156 | /* Resource Management */ 157 | 158 | #define LZ4F_VERSION 100 159 | LZ4F_errorCode_t LZ4F_createCompressionContext(LZ4F_compressionContext_t* cctxPtr, unsigned version); 160 | LZ4F_errorCode_t LZ4F_freeCompressionContext(LZ4F_compressionContext_t cctx); 161 | /* LZ4F_createCompressionContext() : 162 | * The first thing to do is to create a compressionContext object, which will be used in all compression operations. 163 | * This is achieved using LZ4F_createCompressionContext(), which takes as argument a version and an LZ4F_preferences_t structure. 164 | * The version provided MUST be LZ4F_VERSION. It is intended to track potential version differences between different binaries. 165 | * The function will provide a pointer to a fully allocated LZ4F_compressionContext_t object. 166 | * If the result LZ4F_errorCode_t is not zero, there was an error during context creation. 167 | * Object can release its memory using LZ4F_freeCompressionContext(); 168 | */ 169 | 170 | 171 | /* Compression */ 172 | 173 | size_t LZ4F_compressBegin(LZ4F_compressionContext_t cctx, void* dstBuffer, size_t dstMaxSize, const LZ4F_preferences_t* prefsPtr); 174 | /* LZ4F_compressBegin() : 175 | * will write the frame header into dstBuffer. 176 | * dstBuffer must be large enough to accommodate a header (dstMaxSize). Maximum header size is 15 bytes. 177 | * The LZ4F_preferences_t structure is optional : you can provide NULL as argument, all preferences will then be set to default. 178 | * The result of the function is the number of bytes written into dstBuffer for the header 179 | * or an error code (can be tested using LZ4F_isError()) 180 | */ 181 | 182 | size_t LZ4F_compressBound(size_t srcSize, const LZ4F_preferences_t* prefsPtr); 183 | /* LZ4F_compressBound() : 184 | * Provides the minimum size of Dst buffer given srcSize to handle worst case situations. 185 | * Different preferences can produce different results. 186 | * prefsPtr is optional : you can provide NULL as argument, all preferences will then be set to cover worst case. 187 | * This function includes frame termination cost (4 bytes, or 8 if frame checksum is enabled) 188 | */ 189 | 190 | size_t LZ4F_compressUpdate(LZ4F_compressionContext_t cctx, void* dstBuffer, size_t dstMaxSize, const void* srcBuffer, size_t srcSize, const LZ4F_compressOptions_t* cOptPtr); 191 | /* LZ4F_compressUpdate() 192 | * LZ4F_compressUpdate() can be called repetitively to compress as much data as necessary. 193 | * The most important rule is that dstBuffer MUST be large enough (dstMaxSize) to ensure compression completion even in worst case. 194 | * You can get the minimum value of dstMaxSize by using LZ4F_compressBound(). 195 | * If this condition is not respected, LZ4F_compress() will fail (result is an errorCode). 196 | * LZ4F_compressUpdate() doesn't guarantee error recovery, so you have to reset compression context when an error occurs. 197 | * The LZ4F_compressOptions_t structure is optional : you can provide NULL as argument. 198 | * The result of the function is the number of bytes written into dstBuffer : it can be zero, meaning input data was just buffered. 199 | * The function outputs an error code if it fails (can be tested using LZ4F_isError()) 200 | */ 201 | 202 | size_t LZ4F_flush(LZ4F_compressionContext_t cctx, void* dstBuffer, size_t dstMaxSize, const LZ4F_compressOptions_t* cOptPtr); 203 | /* LZ4F_flush() 204 | * Should you need to generate compressed data immediately, without waiting for the current block to be filled, 205 | * you can call LZ4_flush(), which will immediately compress any remaining data buffered within cctx. 206 | * Note that dstMaxSize must be large enough to ensure the operation will be successful. 207 | * LZ4F_compressOptions_t structure is optional : you can provide NULL as argument. 208 | * The result of the function is the number of bytes written into dstBuffer 209 | * (it can be zero, this means there was no data left within cctx) 210 | * The function outputs an error code if it fails (can be tested using LZ4F_isError()) 211 | */ 212 | 213 | size_t LZ4F_compressEnd(LZ4F_compressionContext_t cctx, void* dstBuffer, size_t dstMaxSize, const LZ4F_compressOptions_t* cOptPtr); 214 | /* LZ4F_compressEnd() 215 | * When you want to properly finish the compressed frame, just call LZ4F_compressEnd(). 216 | * It will flush whatever data remained within compressionContext (like LZ4_flush()) 217 | * but also properly finalize the frame, with an endMark and a checksum. 218 | * The result of the function is the number of bytes written into dstBuffer (necessarily >= 4 (endMark), or 8 if optional frame checksum is enabled) 219 | * The function outputs an error code if it fails (can be tested using LZ4F_isError()) 220 | * The LZ4F_compressOptions_t structure is optional : you can provide NULL as argument. 221 | * A successful call to LZ4F_compressEnd() makes cctx available again for next compression task. 222 | */ 223 | 224 | 225 | /*********************************** 226 | * Decompression functions 227 | ***********************************/ 228 | 229 | typedef struct LZ4F_dctx_s* LZ4F_decompressionContext_t; /* must be aligned on 8-bytes */ 230 | 231 | typedef struct { 232 | unsigned stableDst; /* guarantee that decompressed data will still be there on next function calls (avoid storage into tmp buffers) */ 233 | unsigned reserved[3]; 234 | } LZ4F_decompressOptions_t; 235 | 236 | 237 | /* Resource management */ 238 | 239 | LZ4F_errorCode_t LZ4F_createDecompressionContext(LZ4F_decompressionContext_t* dctxPtr, unsigned version); 240 | LZ4F_errorCode_t LZ4F_freeDecompressionContext(LZ4F_decompressionContext_t dctx); 241 | /* LZ4F_createDecompressionContext() : 242 | * The first thing to do is to create an LZ4F_decompressionContext_t object, which will be used in all decompression operations. 243 | * This is achieved using LZ4F_createDecompressionContext(). 244 | * The version provided MUST be LZ4F_VERSION. It is intended to track potential breaking differences between different versions. 245 | * The function will provide a pointer to a fully allocated and initialized LZ4F_decompressionContext_t object. 246 | * The result is an errorCode, which can be tested using LZ4F_isError(). 247 | * dctx memory can be released using LZ4F_freeDecompressionContext(); 248 | * The result of LZ4F_freeDecompressionContext() is indicative of the current state of decompressionContext when being released. 249 | * That is, it should be == 0 if decompression has been completed fully and correctly. 250 | */ 251 | 252 | 253 | /* Decompression */ 254 | 255 | size_t LZ4F_getFrameInfo(LZ4F_decompressionContext_t dctx, 256 | LZ4F_frameInfo_t* frameInfoPtr, 257 | const void* srcBuffer, size_t* srcSizePtr); 258 | /* LZ4F_getFrameInfo() 259 | * This function decodes frame header information (such as max blockSize, frame checksum, etc.). 260 | * Its usage is optional : you can start by calling directly LZ4F_decompress() instead. 261 | * The objective is to extract frame header information, typically for allocation purposes. 262 | * LZ4F_getFrameInfo() can also be used anytime *after* starting decompression, on any valid LZ4F_decompressionContext_t. 263 | * The result is *copied* into an existing LZ4F_frameInfo_t structure which must be already allocated. 264 | * The number of bytes read from srcBuffer will be provided within *srcSizePtr (necessarily <= original value). 265 | * The function result is an hint of how many srcSize bytes LZ4F_decompress() expects for next call, 266 | * or an error code which can be tested using LZ4F_isError() 267 | * (typically, when there is not enough src bytes to fully decode the frame header) 268 | * You are expected to resume decompression from where it stopped (srcBuffer + *srcSizePtr) 269 | */ 270 | 271 | size_t LZ4F_decompress(LZ4F_decompressionContext_t dctx, 272 | void* dstBuffer, size_t* dstSizePtr, 273 | const void* srcBuffer, size_t* srcSizePtr, 274 | const LZ4F_decompressOptions_t* dOptPtr); 275 | /* LZ4F_decompress() 276 | * Call this function repetitively to regenerate data compressed within srcBuffer. 277 | * The function will attempt to decode *srcSizePtr bytes from srcBuffer, into dstBuffer of maximum size *dstSizePtr. 278 | * 279 | * The number of bytes regenerated into dstBuffer will be provided within *dstSizePtr (necessarily <= original value). 280 | * 281 | * The number of bytes read from srcBuffer will be provided within *srcSizePtr (necessarily <= original value). 282 | * If number of bytes read is < number of bytes provided, then decompression operation is not completed. 283 | * It typically happens when dstBuffer is not large enough to contain all decoded data. 284 | * LZ4F_decompress() must be called again, starting from where it stopped (srcBuffer + *srcSizePtr) 285 | * The function will check this condition, and refuse to continue if it is not respected. 286 | * 287 | * dstBuffer is supposed to be flushed between each call to the function, since its content will be overwritten. 288 | * dst arguments can be changed at will with each consecutive call to the function. 289 | * 290 | * The function result is an hint of how many srcSize bytes LZ4F_decompress() expects for next call. 291 | * Schematically, it's the size of the current (or remaining) compressed block + header of next block. 292 | * Respecting the hint provides some boost to performance, since it does skip intermediate buffers. 293 | * This is just a hint, you can always provide any srcSize you want. 294 | * When a frame is fully decoded, the function result will be 0 (no more data expected). 295 | * If decompression failed, function result is an error code, which can be tested using LZ4F_isError(). 296 | * 297 | * After a frame is fully decoded, dctx can be used again to decompress another frame. 298 | */ 299 | 300 | void LZ4F_disableChecksum(LZ4F_decompressionContext_t decompressionContext); 301 | 302 | #if defined (__cplusplus) 303 | } 304 | #endif 305 | -------------------------------------------------------------------------------- /src/lz4frame_static.h: -------------------------------------------------------------------------------- 1 | /* 2 | LZ4 auto-framing library 3 | Header File for static linking only 4 | Copyright (C) 2011-2015, Yann Collet. 5 | 6 | BSD 2-Clause License (http://www.opensource.org/licenses/bsd-license.php) 7 | 8 | Redistribution and use in source and binary forms, with or without 9 | modification, are permitted provided that the following conditions are 10 | met: 11 | 12 | * Redistributions of source code must retain the above copyright 13 | notice, this list of conditions and the following disclaimer. 14 | * Redistributions in binary form must reproduce the above 15 | copyright notice, this list of conditions and the following disclaimer 16 | in the documentation and/or other materials provided with the 17 | distribution. 18 | 19 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 20 | "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT 21 | LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR 22 | A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT 23 | OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, 24 | SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT 25 | LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, 26 | DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY 27 | THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT 28 | (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 29 | OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 30 | 31 | You can contact the author at : 32 | - LZ4 source repository : https://github.com/Cyan4973/lz4 33 | - LZ4 public forum : https://groups.google.com/forum/#!forum/lz4c 34 | */ 35 | 36 | #pragma once 37 | 38 | #if defined (__cplusplus) 39 | extern "C" { 40 | #endif 41 | 42 | /* lz4frame_static.h should be used solely in the context of static linking. 43 | * It contains definitions which may still change overtime. 44 | * Never use it in the context of DLL linking. 45 | * */ 46 | 47 | 48 | /************************************** 49 | * Includes 50 | **************************************/ 51 | #include "lz4frame.h" 52 | 53 | 54 | /************************************** 55 | * Error management 56 | * ************************************/ 57 | #define LZ4F_LIST_ERRORS(ITEM) \ 58 | ITEM(OK_NoError) ITEM(ERROR_GENERIC) \ 59 | ITEM(ERROR_maxBlockSize_invalid) ITEM(ERROR_blockMode_invalid) ITEM(ERROR_contentChecksumFlag_invalid) \ 60 | ITEM(ERROR_compressionLevel_invalid) \ 61 | ITEM(ERROR_headerVersion_wrong) ITEM(ERROR_blockChecksum_unsupported) ITEM(ERROR_reservedFlag_set) \ 62 | ITEM(ERROR_allocation_failed) \ 63 | ITEM(ERROR_srcSize_tooLarge) ITEM(ERROR_dstMaxSize_tooSmall) \ 64 | ITEM(ERROR_frameHeader_incomplete) ITEM(ERROR_frameType_unknown) ITEM(ERROR_frameSize_wrong) \ 65 | ITEM(ERROR_srcPtr_wrong) \ 66 | ITEM(ERROR_decompressionFailed) \ 67 | ITEM(ERROR_headerChecksum_invalid) ITEM(ERROR_contentChecksum_invalid) \ 68 | ITEM(ERROR_maxCode) 69 | 70 | //#define LZ4F_DISABLE_OLD_ENUMS 71 | #ifndef LZ4F_DISABLE_OLD_ENUMS 72 | #define LZ4F_GENERATE_ENUM(ENUM) LZ4F_##ENUM, ENUM = LZ4F_##ENUM, 73 | #else 74 | #define LZ4F_GENERATE_ENUM(ENUM) LZ4F_##ENUM, 75 | #endif 76 | typedef enum { LZ4F_LIST_ERRORS(LZ4F_GENERATE_ENUM) } LZ4F_errorCodes; /* enum is exposed, to handle specific errors; compare function result to -enum value */ 77 | 78 | 79 | #if defined (__cplusplus) 80 | } 81 | #endif 82 | -------------------------------------------------------------------------------- /src/lz4hc.c: -------------------------------------------------------------------------------- 1 | /* 2 | LZ4 HC - High Compression Mode of LZ4 3 | Copyright (C) 2011-2015, Yann Collet. 4 | 5 | BSD 2-Clause License (http://www.opensource.org/licenses/bsd-license.php) 6 | 7 | Redistribution and use in source and binary forms, with or without 8 | modification, are permitted provided that the following conditions are 9 | met: 10 | 11 | * Redistributions of source code must retain the above copyright 12 | notice, this list of conditions and the following disclaimer. 13 | * Redistributions in binary form must reproduce the above 14 | copyright notice, this list of conditions and the following disclaimer 15 | in the documentation and/or other materials provided with the 16 | distribution. 17 | 18 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 19 | "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT 20 | LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR 21 | A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT 22 | OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, 23 | SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT 24 | LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, 25 | DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY 26 | THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT 27 | (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 28 | OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 29 | 30 | You can contact the author at : 31 | - LZ4 source repository : https://github.com/Cyan4973/lz4 32 | - LZ4 public forum : https://groups.google.com/forum/#!forum/lz4c 33 | */ 34 | 35 | 36 | 37 | /************************************** 38 | * Tuning Parameter 39 | **************************************/ 40 | static const int LZ4HC_compressionLevel_default = 9; 41 | 42 | 43 | /************************************** 44 | * Includes 45 | **************************************/ 46 | #include "lz4hc.h" 47 | 48 | 49 | /************************************** 50 | * Local Compiler Options 51 | **************************************/ 52 | #if defined(__GNUC__) 53 | # pragma GCC diagnostic ignored "-Wunused-function" 54 | #endif 55 | 56 | #if defined (__clang__) 57 | # pragma clang diagnostic ignored "-Wunused-function" 58 | #endif 59 | 60 | 61 | /************************************** 62 | * Common LZ4 definition 63 | **************************************/ 64 | #define LZ4_COMMONDEFS_ONLY 65 | #include "lz4.c" 66 | 67 | 68 | /************************************** 69 | * Local Constants 70 | **************************************/ 71 | #define DICTIONARY_LOGSIZE 16 72 | #define MAXD (1<> ((MINMATCH*8)-HASH_LOG)) 106 | //#define DELTANEXTU16(p) chainTable[(p) & MAXD_MASK] /* flexible, MAXD dependent */ 107 | #define DELTANEXTU16(p) chainTable[(U16)(p)] /* faster */ 108 | 109 | static U32 LZ4HC_hashPtr(const void* ptr) { return HASH_FUNCTION(LZ4_read32(ptr)); } 110 | 111 | 112 | 113 | /************************************** 114 | * HC Compression 115 | **************************************/ 116 | static void LZ4HC_init (LZ4HC_Data_Structure* hc4, const BYTE* start) 117 | { 118 | MEM_INIT((void*)hc4->hashTable, 0, sizeof(hc4->hashTable)); 119 | MEM_INIT(hc4->chainTable, 0xFF, sizeof(hc4->chainTable)); 120 | hc4->nextToUpdate = 64 KB; 121 | hc4->base = start - 64 KB; 122 | hc4->end = start; 123 | hc4->dictBase = start - 64 KB; 124 | hc4->dictLimit = 64 KB; 125 | hc4->lowLimit = 64 KB; 126 | } 127 | 128 | 129 | /* Update chains up to ip (excluded) */ 130 | FORCE_INLINE void LZ4HC_Insert (LZ4HC_Data_Structure* hc4, const BYTE* ip) 131 | { 132 | U16* chainTable = hc4->chainTable; 133 | U32* HashTable = hc4->hashTable; 134 | const BYTE* const base = hc4->base; 135 | const U32 target = (U32)(ip - base); 136 | U32 idx = hc4->nextToUpdate; 137 | 138 | while(idx < target) 139 | { 140 | U32 h = LZ4HC_hashPtr(base+idx); 141 | size_t delta = idx - HashTable[h]; 142 | if (delta>MAX_DISTANCE) delta = MAX_DISTANCE; 143 | DELTANEXTU16(idx) = (U16)delta; 144 | HashTable[h] = idx; 145 | idx++; 146 | } 147 | 148 | hc4->nextToUpdate = target; 149 | } 150 | 151 | 152 | FORCE_INLINE int LZ4HC_InsertAndFindBestMatch (LZ4HC_Data_Structure* hc4, /* Index table will be updated */ 153 | const BYTE* ip, const BYTE* const iLimit, 154 | const BYTE** matchpos, 155 | const int maxNbAttempts) 156 | { 157 | U16* const chainTable = hc4->chainTable; 158 | U32* const HashTable = hc4->hashTable; 159 | const BYTE* const base = hc4->base; 160 | const BYTE* const dictBase = hc4->dictBase; 161 | const U32 dictLimit = hc4->dictLimit; 162 | const U32 lowLimit = (hc4->lowLimit + 64 KB > (U32)(ip-base)) ? hc4->lowLimit : (U32)(ip - base) - (64 KB - 1); 163 | U32 matchIndex; 164 | const BYTE* match; 165 | int nbAttempts=maxNbAttempts; 166 | size_t ml=0; 167 | 168 | /* HC4 match finder */ 169 | LZ4HC_Insert(hc4, ip); 170 | matchIndex = HashTable[LZ4HC_hashPtr(ip)]; 171 | 172 | while ((matchIndex>=lowLimit) && (nbAttempts)) 173 | { 174 | nbAttempts--; 175 | if (matchIndex >= dictLimit) 176 | { 177 | match = base + matchIndex; 178 | if (*(match+ml) == *(ip+ml) 179 | && (LZ4_read32(match) == LZ4_read32(ip))) 180 | { 181 | size_t mlt = LZ4_count(ip+MINMATCH, match+MINMATCH, iLimit) + MINMATCH; 182 | if (mlt > ml) { ml = mlt; *matchpos = match; } 183 | } 184 | } 185 | else 186 | { 187 | match = dictBase + matchIndex; 188 | if (LZ4_read32(match) == LZ4_read32(ip)) 189 | { 190 | size_t mlt; 191 | const BYTE* vLimit = ip + (dictLimit - matchIndex); 192 | if (vLimit > iLimit) vLimit = iLimit; 193 | mlt = LZ4_count(ip+MINMATCH, match+MINMATCH, vLimit) + MINMATCH; 194 | if ((ip+mlt == vLimit) && (vLimit < iLimit)) 195 | mlt += LZ4_count(ip+mlt, base+dictLimit, iLimit); 196 | if (mlt > ml) { ml = mlt; *matchpos = base + matchIndex; } /* virtual matchpos */ 197 | } 198 | } 199 | matchIndex -= DELTANEXTU16(matchIndex); 200 | } 201 | 202 | return (int)ml; 203 | } 204 | 205 | 206 | FORCE_INLINE int LZ4HC_InsertAndGetWiderMatch ( 207 | LZ4HC_Data_Structure* hc4, 208 | const BYTE* const ip, 209 | const BYTE* const iLowLimit, 210 | const BYTE* const iHighLimit, 211 | int longest, 212 | const BYTE** matchpos, 213 | const BYTE** startpos, 214 | const int maxNbAttempts) 215 | { 216 | U16* const chainTable = hc4->chainTable; 217 | U32* const HashTable = hc4->hashTable; 218 | const BYTE* const base = hc4->base; 219 | const U32 dictLimit = hc4->dictLimit; 220 | const BYTE* const lowPrefixPtr = base + dictLimit; 221 | const U32 lowLimit = (hc4->lowLimit + 64 KB > (U32)(ip-base)) ? hc4->lowLimit : (U32)(ip - base) - (64 KB - 1); 222 | const BYTE* const dictBase = hc4->dictBase; 223 | U32 matchIndex; 224 | int nbAttempts = maxNbAttempts; 225 | int delta = (int)(ip-iLowLimit); 226 | 227 | 228 | /* First Match */ 229 | LZ4HC_Insert(hc4, ip); 230 | matchIndex = HashTable[LZ4HC_hashPtr(ip)]; 231 | 232 | while ((matchIndex>=lowLimit) && (nbAttempts)) 233 | { 234 | nbAttempts--; 235 | if (matchIndex >= dictLimit) 236 | { 237 | const BYTE* matchPtr = base + matchIndex; 238 | if (*(iLowLimit + longest) == *(matchPtr - delta + longest)) 239 | if (LZ4_read32(matchPtr) == LZ4_read32(ip)) 240 | { 241 | int mlt = MINMATCH + LZ4_count(ip+MINMATCH, matchPtr+MINMATCH, iHighLimit); 242 | int back = 0; 243 | 244 | while ((ip+back>iLowLimit) 245 | && (matchPtr+back > lowPrefixPtr) 246 | && (ip[back-1] == matchPtr[back-1])) 247 | back--; 248 | 249 | mlt -= back; 250 | 251 | if (mlt > longest) 252 | { 253 | longest = (int)mlt; 254 | *matchpos = matchPtr+back; 255 | *startpos = ip+back; 256 | } 257 | } 258 | } 259 | else 260 | { 261 | const BYTE* matchPtr = dictBase + matchIndex; 262 | if (LZ4_read32(matchPtr) == LZ4_read32(ip)) 263 | { 264 | size_t mlt; 265 | int back=0; 266 | const BYTE* vLimit = ip + (dictLimit - matchIndex); 267 | if (vLimit > iHighLimit) vLimit = iHighLimit; 268 | mlt = LZ4_count(ip+MINMATCH, matchPtr+MINMATCH, vLimit) + MINMATCH; 269 | if ((ip+mlt == vLimit) && (vLimit < iHighLimit)) 270 | mlt += LZ4_count(ip+mlt, base+dictLimit, iHighLimit); 271 | while ((ip+back > iLowLimit) && (matchIndex+back > lowLimit) && (ip[back-1] == matchPtr[back-1])) back--; 272 | mlt -= back; 273 | if ((int)mlt > longest) { longest = (int)mlt; *matchpos = base + matchIndex + back; *startpos = ip+back; } 274 | } 275 | } 276 | matchIndex -= DELTANEXTU16(matchIndex); 277 | } 278 | 279 | return longest; 280 | } 281 | 282 | 283 | typedef enum { noLimit = 0, limitedOutput = 1 } limitedOutput_directive; 284 | 285 | #define LZ4HC_DEBUG 0 286 | #if LZ4HC_DEBUG 287 | static unsigned debug = 0; 288 | #endif 289 | 290 | FORCE_INLINE int LZ4HC_encodeSequence ( 291 | const BYTE** ip, 292 | BYTE** op, 293 | const BYTE** anchor, 294 | int matchLength, 295 | const BYTE* const match, 296 | limitedOutput_directive limitedOutputBuffer, 297 | BYTE* oend) 298 | { 299 | int length; 300 | BYTE* token; 301 | 302 | #if LZ4HC_DEBUG 303 | if (debug) printf("literal : %u -- match : %u -- offset : %u\n", (U32)(*ip - *anchor), (U32)matchLength, (U32)(*ip-match)); 304 | #endif 305 | 306 | /* Encode Literal length */ 307 | length = (int)(*ip - *anchor); 308 | token = (*op)++; 309 | if ((limitedOutputBuffer) && ((*op + (length>>8) + length + (2 + 1 + LASTLITERALS)) > oend)) return 1; /* Check output limit */ 310 | if (length>=(int)RUN_MASK) { int len; *token=(RUN_MASK< 254 ; len-=255) *(*op)++ = 255; *(*op)++ = (BYTE)len; } 311 | else *token = (BYTE)(length<>8) + (1 + LASTLITERALS) > oend)) return 1; /* Check output limit */ 323 | if (length>=(int)ML_MASK) { *token+=ML_MASK; length-=ML_MASK; for(; length > 509 ; length-=510) { *(*op)++ = 255; *(*op)++ = 255; } if (length > 254) { length-=255; *(*op)++ = 255; } *(*op)++ = (BYTE)length; } 324 | else *token += (BYTE)(length); 325 | 326 | /* Prepare next loop */ 327 | *ip += matchLength; 328 | *anchor = *ip; 329 | 330 | return 0; 331 | } 332 | 333 | 334 | static int LZ4HC_compress_generic ( 335 | void* ctxvoid, 336 | const char* source, 337 | char* dest, 338 | int inputSize, 339 | int maxOutputSize, 340 | int compressionLevel, 341 | limitedOutput_directive limit 342 | ) 343 | { 344 | LZ4HC_Data_Structure* ctx = (LZ4HC_Data_Structure*) ctxvoid; 345 | const BYTE* ip = (const BYTE*) source; 346 | const BYTE* anchor = ip; 347 | const BYTE* const iend = ip + inputSize; 348 | const BYTE* const mflimit = iend - MFLIMIT; 349 | const BYTE* const matchlimit = (iend - LASTLITERALS); 350 | 351 | BYTE* op = (BYTE*) dest; 352 | BYTE* const oend = op + maxOutputSize; 353 | 354 | unsigned maxNbAttempts; 355 | int ml, ml2, ml3, ml0; 356 | const BYTE* ref=NULL; 357 | const BYTE* start2=NULL; 358 | const BYTE* ref2=NULL; 359 | const BYTE* start3=NULL; 360 | const BYTE* ref3=NULL; 361 | const BYTE* start0; 362 | const BYTE* ref0; 363 | 364 | 365 | /* init */ 366 | if (compressionLevel > g_maxCompressionLevel) compressionLevel = g_maxCompressionLevel; 367 | if (compressionLevel < 1) compressionLevel = LZ4HC_compressionLevel_default; 368 | maxNbAttempts = 1 << (compressionLevel-1); 369 | ctx->end += inputSize; 370 | 371 | ip++; 372 | 373 | /* Main Loop */ 374 | while (ip < mflimit) 375 | { 376 | ml = LZ4HC_InsertAndFindBestMatch (ctx, ip, matchlimit, (&ref), maxNbAttempts); 377 | if (!ml) { ip++; continue; } 378 | 379 | /* saved, in case we would skip too much */ 380 | start0 = ip; 381 | ref0 = ref; 382 | ml0 = ml; 383 | 384 | _Search2: 385 | if (ip+ml < mflimit) 386 | ml2 = LZ4HC_InsertAndGetWiderMatch(ctx, ip + ml - 2, ip + 1, matchlimit, ml, &ref2, &start2, maxNbAttempts); 387 | else ml2 = ml; 388 | 389 | if (ml2 == ml) /* No better match */ 390 | { 391 | if (LZ4HC_encodeSequence(&ip, &op, &anchor, ml, ref, limit, oend)) return 0; 392 | continue; 393 | } 394 | 395 | if (start0 < ip) 396 | { 397 | if (start2 < ip + ml0) /* empirical */ 398 | { 399 | ip = start0; 400 | ref = ref0; 401 | ml = ml0; 402 | } 403 | } 404 | 405 | /* Here, start0==ip */ 406 | if ((start2 - ip) < 3) /* First Match too small : removed */ 407 | { 408 | ml = ml2; 409 | ip = start2; 410 | ref =ref2; 411 | goto _Search2; 412 | } 413 | 414 | _Search3: 415 | /* 416 | * Currently we have : 417 | * ml2 > ml1, and 418 | * ip1+3 <= ip2 (usually < ip1+ml1) 419 | */ 420 | if ((start2 - ip) < OPTIMAL_ML) 421 | { 422 | int correction; 423 | int new_ml = ml; 424 | if (new_ml > OPTIMAL_ML) new_ml = OPTIMAL_ML; 425 | if (ip+new_ml > start2 + ml2 - MINMATCH) new_ml = (int)(start2 - ip) + ml2 - MINMATCH; 426 | correction = new_ml - (int)(start2 - ip); 427 | if (correction > 0) 428 | { 429 | start2 += correction; 430 | ref2 += correction; 431 | ml2 -= correction; 432 | } 433 | } 434 | /* Now, we have start2 = ip+new_ml, with new_ml = min(ml, OPTIMAL_ML=18) */ 435 | 436 | if (start2 + ml2 < mflimit) 437 | ml3 = LZ4HC_InsertAndGetWiderMatch(ctx, start2 + ml2 - 3, start2, matchlimit, ml2, &ref3, &start3, maxNbAttempts); 438 | else ml3 = ml2; 439 | 440 | if (ml3 == ml2) /* No better match : 2 sequences to encode */ 441 | { 442 | /* ip & ref are known; Now for ml */ 443 | if (start2 < ip+ml) ml = (int)(start2 - ip); 444 | /* Now, encode 2 sequences */ 445 | if (LZ4HC_encodeSequence(&ip, &op, &anchor, ml, ref, limit, oend)) return 0; 446 | ip = start2; 447 | if (LZ4HC_encodeSequence(&ip, &op, &anchor, ml2, ref2, limit, oend)) return 0; 448 | continue; 449 | } 450 | 451 | if (start3 < ip+ml+3) /* Not enough space for match 2 : remove it */ 452 | { 453 | if (start3 >= (ip+ml)) /* can write Seq1 immediately ==> Seq2 is removed, so Seq3 becomes Seq1 */ 454 | { 455 | if (start2 < ip+ml) 456 | { 457 | int correction = (int)(ip+ml - start2); 458 | start2 += correction; 459 | ref2 += correction; 460 | ml2 -= correction; 461 | if (ml2 < MINMATCH) 462 | { 463 | start2 = start3; 464 | ref2 = ref3; 465 | ml2 = ml3; 466 | } 467 | } 468 | 469 | if (LZ4HC_encodeSequence(&ip, &op, &anchor, ml, ref, limit, oend)) return 0; 470 | ip = start3; 471 | ref = ref3; 472 | ml = ml3; 473 | 474 | start0 = start2; 475 | ref0 = ref2; 476 | ml0 = ml2; 477 | goto _Search2; 478 | } 479 | 480 | start2 = start3; 481 | ref2 = ref3; 482 | ml2 = ml3; 483 | goto _Search3; 484 | } 485 | 486 | /* 487 | * OK, now we have 3 ascending matches; let's write at least the first one 488 | * ip & ref are known; Now for ml 489 | */ 490 | if (start2 < ip+ml) 491 | { 492 | if ((start2 - ip) < (int)ML_MASK) 493 | { 494 | int correction; 495 | if (ml > OPTIMAL_ML) ml = OPTIMAL_ML; 496 | if (ip + ml > start2 + ml2 - MINMATCH) ml = (int)(start2 - ip) + ml2 - MINMATCH; 497 | correction = ml - (int)(start2 - ip); 498 | if (correction > 0) 499 | { 500 | start2 += correction; 501 | ref2 += correction; 502 | ml2 -= correction; 503 | } 504 | } 505 | else 506 | { 507 | ml = (int)(start2 - ip); 508 | } 509 | } 510 | if (LZ4HC_encodeSequence(&ip, &op, &anchor, ml, ref, limit, oend)) return 0; 511 | 512 | ip = start2; 513 | ref = ref2; 514 | ml = ml2; 515 | 516 | start2 = start3; 517 | ref2 = ref3; 518 | ml2 = ml3; 519 | 520 | goto _Search3; 521 | } 522 | 523 | /* Encode Last Literals */ 524 | { 525 | int lastRun = (int)(iend - anchor); 526 | if ((limit) && (((char*)op - dest) + lastRun + 1 + ((lastRun+255-RUN_MASK)/255) > (U32)maxOutputSize)) return 0; /* Check output limit */ 527 | if (lastRun>=(int)RUN_MASK) { *op++=(RUN_MASK< 254 ; lastRun-=255) *op++ = 255; *op++ = (BYTE) lastRun; } 528 | else *op++ = (BYTE)(lastRun<base = NULL; 571 | ((LZ4HC_Data_Structure*)LZ4_streamHCPtr)->compressionLevel = (unsigned)compressionLevel; 572 | } 573 | 574 | int LZ4_loadDictHC (LZ4_streamHC_t* LZ4_streamHCPtr, const char* dictionary, int dictSize) 575 | { 576 | LZ4HC_Data_Structure* ctxPtr = (LZ4HC_Data_Structure*) LZ4_streamHCPtr; 577 | if (dictSize > 64 KB) 578 | { 579 | dictionary += dictSize - 64 KB; 580 | dictSize = 64 KB; 581 | } 582 | LZ4HC_init (ctxPtr, (const BYTE*)dictionary); 583 | if (dictSize >= 4) LZ4HC_Insert (ctxPtr, (const BYTE*)dictionary +(dictSize-3)); 584 | ctxPtr->end = (const BYTE*)dictionary + dictSize; 585 | return dictSize; 586 | } 587 | 588 | 589 | /* compression */ 590 | 591 | static void LZ4HC_setExternalDict(LZ4HC_Data_Structure* ctxPtr, const BYTE* newBlock) 592 | { 593 | if (ctxPtr->end >= ctxPtr->base + 4) 594 | LZ4HC_Insert (ctxPtr, ctxPtr->end-3); /* Referencing remaining dictionary content */ 595 | /* Only one memory segment for extDict, so any previous extDict is lost at this stage */ 596 | ctxPtr->lowLimit = ctxPtr->dictLimit; 597 | ctxPtr->dictLimit = (U32)(ctxPtr->end - ctxPtr->base); 598 | ctxPtr->dictBase = ctxPtr->base; 599 | ctxPtr->base = newBlock - ctxPtr->dictLimit; 600 | ctxPtr->end = newBlock; 601 | ctxPtr->nextToUpdate = ctxPtr->dictLimit; /* match referencing will resume from there */ 602 | } 603 | 604 | static int LZ4_compressHC_continue_generic (LZ4HC_Data_Structure* ctxPtr, 605 | const char* source, char* dest, 606 | int inputSize, int maxOutputSize, limitedOutput_directive limit) 607 | { 608 | /* auto-init if forgotten */ 609 | if (ctxPtr->base == NULL) 610 | LZ4HC_init (ctxPtr, (const BYTE*) source); 611 | 612 | /* Check overflow */ 613 | if ((size_t)(ctxPtr->end - ctxPtr->base) > 2 GB) 614 | { 615 | size_t dictSize = (size_t)(ctxPtr->end - ctxPtr->base) - ctxPtr->dictLimit; 616 | if (dictSize > 64 KB) dictSize = 64 KB; 617 | 618 | LZ4_loadDictHC((LZ4_streamHC_t*)ctxPtr, (const char*)(ctxPtr->end) - dictSize, (int)dictSize); 619 | } 620 | 621 | /* Check if blocks follow each other */ 622 | if ((const BYTE*)source != ctxPtr->end) 623 | LZ4HC_setExternalDict(ctxPtr, (const BYTE*)source); 624 | 625 | /* Check overlapping input/dictionary space */ 626 | { 627 | const BYTE* sourceEnd = (const BYTE*) source + inputSize; 628 | const BYTE* dictBegin = ctxPtr->dictBase + ctxPtr->lowLimit; 629 | const BYTE* dictEnd = ctxPtr->dictBase + ctxPtr->dictLimit; 630 | if ((sourceEnd > dictBegin) && ((const BYTE*)source < dictEnd)) 631 | { 632 | if (sourceEnd > dictEnd) sourceEnd = dictEnd; 633 | ctxPtr->lowLimit = (U32)(sourceEnd - ctxPtr->dictBase); 634 | if (ctxPtr->dictLimit - ctxPtr->lowLimit < 4) ctxPtr->lowLimit = ctxPtr->dictLimit; 635 | } 636 | } 637 | 638 | return LZ4HC_compress_generic (ctxPtr, source, dest, inputSize, maxOutputSize, ctxPtr->compressionLevel, limit); 639 | } 640 | 641 | int LZ4_compress_HC_continue (LZ4_streamHC_t* LZ4_streamHCPtr, const char* source, char* dest, int inputSize, int maxOutputSize) 642 | { 643 | if (maxOutputSize < LZ4_compressBound(inputSize)) 644 | return LZ4_compressHC_continue_generic ((LZ4HC_Data_Structure*)LZ4_streamHCPtr, source, dest, inputSize, maxOutputSize, limitedOutput); 645 | else 646 | return LZ4_compressHC_continue_generic ((LZ4HC_Data_Structure*)LZ4_streamHCPtr, source, dest, inputSize, maxOutputSize, noLimit); 647 | } 648 | 649 | 650 | /* dictionary saving */ 651 | 652 | int LZ4_saveDictHC (LZ4_streamHC_t* LZ4_streamHCPtr, char* safeBuffer, int dictSize) 653 | { 654 | LZ4HC_Data_Structure* streamPtr = (LZ4HC_Data_Structure*)LZ4_streamHCPtr; 655 | int prefixSize = (int)(streamPtr->end - (streamPtr->base + streamPtr->dictLimit)); 656 | if (dictSize > 64 KB) dictSize = 64 KB; 657 | if (dictSize < 4) dictSize = 0; 658 | if (dictSize > prefixSize) dictSize = prefixSize; 659 | memmove(safeBuffer, streamPtr->end - dictSize, dictSize); 660 | { 661 | U32 endIndex = (U32)(streamPtr->end - streamPtr->base); 662 | streamPtr->end = (const BYTE*)safeBuffer + dictSize; 663 | streamPtr->base = streamPtr->end - endIndex; 664 | streamPtr->dictLimit = endIndex - dictSize; 665 | streamPtr->lowLimit = endIndex - dictSize; 666 | if (streamPtr->nextToUpdate < streamPtr->dictLimit) streamPtr->nextToUpdate = streamPtr->dictLimit; 667 | } 668 | return dictSize; 669 | } 670 | 671 | 672 | /*********************************** 673 | * Deprecated Functions 674 | ***********************************/ 675 | /* Deprecated compression functions */ 676 | /* These functions are planned to start generate warnings by r131 approximately */ 677 | int LZ4_compressHC(const char* src, char* dst, int srcSize) { return LZ4_compress_HC (src, dst, srcSize, LZ4_compressBound(srcSize), 0); } 678 | int LZ4_compressHC_limitedOutput(const char* src, char* dst, int srcSize, int maxDstSize) { return LZ4_compress_HC(src, dst, srcSize, maxDstSize, 0); } 679 | int LZ4_compressHC2(const char* src, char* dst, int srcSize, int cLevel) { return LZ4_compress_HC (src, dst, srcSize, LZ4_compressBound(srcSize), cLevel); } 680 | int LZ4_compressHC2_limitedOutput(const char* src, char* dst, int srcSize, int maxDstSize, int cLevel) { return LZ4_compress_HC(src, dst, srcSize, maxDstSize, cLevel); } 681 | int LZ4_compressHC_withStateHC (void* state, const char* src, char* dst, int srcSize) { return LZ4_compress_HC_extStateHC (state, src, dst, srcSize, LZ4_compressBound(srcSize), 0); } 682 | int LZ4_compressHC_limitedOutput_withStateHC (void* state, const char* src, char* dst, int srcSize, int maxDstSize) { return LZ4_compress_HC_extStateHC (state, src, dst, srcSize, maxDstSize, 0); } 683 | int LZ4_compressHC2_withStateHC (void* state, const char* src, char* dst, int srcSize, int cLevel) { return LZ4_compress_HC_extStateHC(state, src, dst, srcSize, LZ4_compressBound(srcSize), cLevel); } 684 | int LZ4_compressHC2_limitedOutput_withStateHC (void* state, const char* src, char* dst, int srcSize, int maxDstSize, int cLevel) { return LZ4_compress_HC_extStateHC(state, src, dst, srcSize, maxDstSize, cLevel); } 685 | int LZ4_compressHC_continue (LZ4_streamHC_t* ctx, const char* src, char* dst, int srcSize) { return LZ4_compress_HC_continue (ctx, src, dst, srcSize, LZ4_compressBound(srcSize)); } 686 | int LZ4_compressHC_limitedOutput_continue (LZ4_streamHC_t* ctx, const char* src, char* dst, int srcSize, int maxDstSize) { return LZ4_compress_HC_continue (ctx, src, dst, srcSize, maxDstSize); } 687 | 688 | 689 | /* Deprecated streaming functions */ 690 | /* These functions currently generate deprecation warnings */ 691 | int LZ4_sizeofStreamStateHC(void) { return LZ4_STREAMHCSIZE; } 692 | 693 | int LZ4_resetStreamStateHC(void* state, char* inputBuffer) 694 | { 695 | if ((((size_t)state) & (sizeof(void*)-1)) != 0) return 1; /* Error : pointer is not aligned for pointer (32 or 64 bits) */ 696 | LZ4HC_init((LZ4HC_Data_Structure*)state, (const BYTE*)inputBuffer); 697 | ((LZ4HC_Data_Structure*)state)->inputBuffer = (BYTE*)inputBuffer; 698 | return 0; 699 | } 700 | 701 | void* LZ4_createHC (char* inputBuffer) 702 | { 703 | void* hc4 = ALLOCATOR(1, sizeof(LZ4HC_Data_Structure)); 704 | if (hc4 == NULL) return NULL; /* not enough memory */ 705 | LZ4HC_init ((LZ4HC_Data_Structure*)hc4, (const BYTE*)inputBuffer); 706 | ((LZ4HC_Data_Structure*)hc4)->inputBuffer = (BYTE*)inputBuffer; 707 | return hc4; 708 | } 709 | 710 | int LZ4_freeHC (void* LZ4HC_Data) 711 | { 712 | FREEMEM(LZ4HC_Data); 713 | return (0); 714 | } 715 | 716 | int LZ4_compressHC2_continue (void* LZ4HC_Data, const char* source, char* dest, int inputSize, int compressionLevel) 717 | { 718 | return LZ4HC_compress_generic (LZ4HC_Data, source, dest, inputSize, 0, compressionLevel, noLimit); 719 | } 720 | 721 | int LZ4_compressHC2_limitedOutput_continue (void* LZ4HC_Data, const char* source, char* dest, int inputSize, int maxOutputSize, int compressionLevel) 722 | { 723 | return LZ4HC_compress_generic (LZ4HC_Data, source, dest, inputSize, maxOutputSize, compressionLevel, limitedOutput); 724 | } 725 | 726 | char* LZ4_slideInputBufferHC(void* LZ4HC_Data) 727 | { 728 | LZ4HC_Data_Structure* hc4 = (LZ4HC_Data_Structure*)LZ4HC_Data; 729 | int dictSize = LZ4_saveDictHC((LZ4_streamHC_t*)LZ4HC_Data, (char*)(hc4->inputBuffer), 64 KB); 730 | return (char*)(hc4->inputBuffer + dictSize); 731 | } 732 | -------------------------------------------------------------------------------- /src/lz4hc.h: -------------------------------------------------------------------------------- 1 | /* 2 | LZ4 HC - High Compression Mode of LZ4 3 | Header File 4 | Copyright (C) 2011-2015, Yann Collet. 5 | BSD 2-Clause License (http://www.opensource.org/licenses/bsd-license.php) 6 | 7 | Redistribution and use in source and binary forms, with or without 8 | modification, are permitted provided that the following conditions are 9 | met: 10 | 11 | * Redistributions of source code must retain the above copyright 12 | notice, this list of conditions and the following disclaimer. 13 | * Redistributions in binary form must reproduce the above 14 | copyright notice, this list of conditions and the following disclaimer 15 | in the documentation and/or other materials provided with the 16 | distribution. 17 | 18 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 19 | "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT 20 | LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR 21 | A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT 22 | OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, 23 | SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT 24 | LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, 25 | DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY 26 | THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT 27 | (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 28 | OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 29 | 30 | You can contact the author at : 31 | - LZ4 source repository : https://github.com/Cyan4973/lz4 32 | - LZ4 public forum : https://groups.google.com/forum/#!forum/lz4c 33 | */ 34 | #pragma once 35 | 36 | 37 | #if defined (__cplusplus) 38 | extern "C" { 39 | #endif 40 | 41 | /***************************** 42 | * Includes 43 | *****************************/ 44 | #include /* size_t */ 45 | 46 | 47 | /************************************** 48 | * Block Compression 49 | **************************************/ 50 | int LZ4_compress_HC (const char* src, char* dst, int srcSize, int maxDstSize, int compressionLevel); 51 | /* 52 | LZ4_compress_HC : 53 | Destination buffer 'dst' must be already allocated. 54 | Compression completion is guaranteed if 'dst' buffer is sized to handle worst circumstances (data not compressible) 55 | Worst size evaluation is provided by function LZ4_compressBound() (see "lz4.h") 56 | srcSize : Max supported value is LZ4_MAX_INPUT_SIZE (see "lz4.h") 57 | compressionLevel : Recommended values are between 4 and 9, although any value between 0 and 16 will work. 58 | 0 means "use default value" (see lz4hc.c). 59 | Values >16 behave the same as 16. 60 | return : the number of bytes written into buffer 'dst' 61 | or 0 if compression fails. 62 | */ 63 | 64 | 65 | /* Note : 66 | Decompression functions are provided within LZ4 source code (see "lz4.h") (BSD license) 67 | */ 68 | 69 | 70 | int LZ4_sizeofStateHC(void); 71 | int LZ4_compress_HC_extStateHC(void* state, const char* src, char* dst, int srcSize, int maxDstSize, int compressionLevel); 72 | /* 73 | LZ4_compress_HC_extStateHC() : 74 | Use this function if you prefer to manually allocate memory for compression tables. 75 | To know how much memory must be allocated for the compression tables, use : 76 | int LZ4_sizeofStateHC(); 77 | 78 | Allocated memory must be aligned on 8-bytes boundaries (which a normal malloc() will do properly). 79 | 80 | The allocated memory can then be provided to the compression functions using 'void* state' parameter. 81 | LZ4_compress_HC_extStateHC() is equivalent to previously described function. 82 | It just uses externally allocated memory for stateHC. 83 | */ 84 | 85 | 86 | /************************************** 87 | * Streaming Compression 88 | **************************************/ 89 | #define LZ4_STREAMHCSIZE 262192 90 | #define LZ4_STREAMHCSIZE_SIZET (LZ4_STREAMHCSIZE / sizeof(size_t)) 91 | typedef struct { size_t table[LZ4_STREAMHCSIZE_SIZET]; } LZ4_streamHC_t; 92 | /* 93 | LZ4_streamHC_t 94 | This structure allows static allocation of LZ4 HC streaming state. 95 | State must then be initialized using LZ4_resetStreamHC() before first use. 96 | 97 | Static allocation should only be used in combination with static linking. 98 | If you want to use LZ4 as a DLL, please use construction functions below, which are future-proof. 99 | */ 100 | 101 | 102 | LZ4_streamHC_t* LZ4_createStreamHC(void); 103 | int LZ4_freeStreamHC (LZ4_streamHC_t* streamHCPtr); 104 | /* 105 | These functions create and release memory for LZ4 HC streaming state. 106 | Newly created states are already initialized. 107 | Existing state space can be re-used anytime using LZ4_resetStreamHC(). 108 | If you use LZ4 as a DLL, use these functions instead of static structure allocation, 109 | to avoid size mismatch between different versions. 110 | */ 111 | 112 | void LZ4_resetStreamHC (LZ4_streamHC_t* streamHCPtr, int compressionLevel); 113 | int LZ4_loadDictHC (LZ4_streamHC_t* streamHCPtr, const char* dictionary, int dictSize); 114 | 115 | int LZ4_compress_HC_continue (LZ4_streamHC_t* streamHCPtr, const char* src, char* dst, int srcSize, int maxDstSize); 116 | 117 | int LZ4_saveDictHC (LZ4_streamHC_t* streamHCPtr, char* safeBuffer, int maxDictSize); 118 | 119 | /* 120 | These functions compress data in successive blocks of any size, using previous blocks as dictionary. 121 | One key assumption is that previous blocks (up to 64 KB) remain read-accessible while compressing next blocks. 122 | There is an exception for ring buffers, which can be smaller 64 KB. 123 | Such case is automatically detected and correctly handled by LZ4_compress_HC_continue(). 124 | 125 | Before starting compression, state must be properly initialized, using LZ4_resetStreamHC(). 126 | A first "fictional block" can then be designated as initial dictionary, using LZ4_loadDictHC() (Optional). 127 | 128 | Then, use LZ4_compress_HC_continue() to compress each successive block. 129 | It works like LZ4_compress_HC(), but use previous memory blocks as dictionary to improve compression. 130 | Previous memory blocks (including initial dictionary when present) must remain accessible and unmodified during compression. 131 | As a reminder, size 'dst' buffer to handle worst cases, using LZ4_compressBound(), to ensure success of compression operation. 132 | 133 | If, for any reason, previous data blocks can't be preserved unmodified in memory during next compression block, 134 | you must save it to a safer memory space, using LZ4_saveDictHC(). 135 | Return value of LZ4_saveDictHC() is the size of dictionary effectively saved into 'safeBuffer'. 136 | */ 137 | 138 | 139 | 140 | /************************************** 141 | * Deprecated Functions 142 | **************************************/ 143 | /* Deprecate Warnings */ 144 | /* Should these warnings messages be a problem, 145 | it is generally possible to disable them, 146 | with -Wno-deprecated-declarations for gcc 147 | or _CRT_SECURE_NO_WARNINGS in Visual for example. 148 | You can also define LZ4_DEPRECATE_WARNING_DEFBLOCK. */ 149 | #ifndef LZ4_DEPRECATE_WARNING_DEFBLOCK 150 | # define LZ4_DEPRECATE_WARNING_DEFBLOCK 151 | # define LZ4_GCC_VERSION (__GNUC__ * 100 + __GNUC_MINOR__) 152 | # if (LZ4_GCC_VERSION >= 405) || defined(__clang__) 153 | # define LZ4_DEPRECATED(message) __attribute__((deprecated(message))) 154 | # elif (LZ4_GCC_VERSION >= 301) 155 | # define LZ4_DEPRECATED(message) __attribute__((deprecated)) 156 | # elif defined(_MSC_VER) 157 | # define LZ4_DEPRECATED(message) __declspec(deprecated(message)) 158 | # else 159 | # pragma message("WARNING: You need to implement LZ4_DEPRECATED for this compiler") 160 | # define LZ4_DEPRECATED(message) 161 | # endif 162 | #endif // LZ4_DEPRECATE_WARNING_DEFBLOCK 163 | 164 | /* compression functions */ 165 | /* these functions are planned to trigger warning messages by r131 approximately */ 166 | int LZ4_compressHC (const char* source, char* dest, int inputSize); 167 | int LZ4_compressHC_limitedOutput (const char* source, char* dest, int inputSize, int maxOutputSize); 168 | int LZ4_compressHC2 (const char* source, char* dest, int inputSize, int compressionLevel); 169 | int LZ4_compressHC2_limitedOutput (const char* source, char* dest, int inputSize, int maxOutputSize, int compressionLevel); 170 | int LZ4_compressHC_withStateHC (void* state, const char* source, char* dest, int inputSize); 171 | int LZ4_compressHC_limitedOutput_withStateHC (void* state, const char* source, char* dest, int inputSize, int maxOutputSize); 172 | int LZ4_compressHC2_withStateHC (void* state, const char* source, char* dest, int inputSize, int compressionLevel); 173 | int LZ4_compressHC2_limitedOutput_withStateHC(void* state, const char* source, char* dest, int inputSize, int maxOutputSize, int compressionLevel); 174 | int LZ4_compressHC_continue (LZ4_streamHC_t* LZ4_streamHCPtr, const char* source, char* dest, int inputSize); 175 | int LZ4_compressHC_limitedOutput_continue (LZ4_streamHC_t* LZ4_streamHCPtr, const char* source, char* dest, int inputSize, int maxOutputSize); 176 | 177 | /* Streaming functions following the older model; should no longer be used */ 178 | LZ4_DEPRECATED("use LZ4_createStreamHC() instead") void* LZ4_createHC (char* inputBuffer); 179 | LZ4_DEPRECATED("use LZ4_saveDictHC() instead") char* LZ4_slideInputBufferHC (void* LZ4HC_Data); 180 | LZ4_DEPRECATED("use LZ4_freeStreamHC() instead") int LZ4_freeHC (void* LZ4HC_Data); 181 | LZ4_DEPRECATED("use LZ4_compress_HC_continue() instead") int LZ4_compressHC2_continue (void* LZ4HC_Data, const char* source, char* dest, int inputSize, int compressionLevel); 182 | LZ4_DEPRECATED("use LZ4_compress_HC_continue() instead") int LZ4_compressHC2_limitedOutput_continue (void* LZ4HC_Data, const char* source, char* dest, int inputSize, int maxOutputSize, int compressionLevel); 183 | LZ4_DEPRECATED("use LZ4_createStreamHC() instead") int LZ4_sizeofStreamStateHC(void); 184 | LZ4_DEPRECATED("use LZ4_resetStreamHC() instead") int LZ4_resetStreamStateHC(void* state, char* inputBuffer); 185 | 186 | 187 | #if defined (__cplusplus) 188 | } 189 | #endif 190 | -------------------------------------------------------------------------------- /src/python-lz4f.c: -------------------------------------------------------------------------------- 1 | /* 2 | * Copyright (c) 2014, Christopher Jackson 3 | * All rights reserved. 4 | * 5 | * Redistribution and use in source and binary forms, with or without 6 | * modification, are permitted provided that the following conditions are met: 7 | * 8 | * 1. Redistributions of source code must retain the above copyright notice, 9 | * this list of conditions and the following disclaimer. 10 | * 11 | * 2. Redistributions in binary form must reproduce the above copyright notice, 12 | * this list of conditions and the following disclaimer in the documentation 13 | * and/or other materials provided with the distribution. 14 | * 15 | * 3. Neither the name of Christopher Jackson nor the names of its contributors 16 | * may be used to endorse or promote products derived from this software 17 | * without specific prior written permission. 18 | * 19 | * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" 20 | * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 21 | * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE 22 | * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE 23 | * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR 24 | * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF 25 | * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS 26 | * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN 27 | * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) 28 | * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE 29 | * POSSIBILITY OF SUCH DAMAGE. 30 | */ 31 | #include 32 | #include 33 | #include 34 | #include 35 | #include 36 | #include "lz4.h" 37 | #include "lz4hc.h" 38 | #include "lz4frame.h" 39 | #include "python-lz4f.h" 40 | #include "structmember.h" 41 | 42 | #if PY_MAJOR_VERSION >= 3 43 | #define PyInt_FromSize_t(x) PyLong_FromSize_t(x) 44 | #endif 45 | 46 | #define CHECK(cond, ...) if (LZ4F_isError(cond)) { printf("%s%s", "Error => ", LZ4F_getErrorName(cond)); goto _output_error; } 47 | 48 | static int LZ4S_GetBlockSize_FromBlockId (int id) { return (1 << (8 + (2 * id))); } 49 | 50 | /* Compression methods */ 51 | 52 | static PyObject *py_lz4f_createCompCtx(PyObject *self, PyObject *args) { 53 | PyObject *result; 54 | LZ4F_compressionContext_t cCtx; 55 | size_t err; 56 | 57 | (void)self; 58 | (void)args; 59 | 60 | err = LZ4F_createCompressionContext(&cCtx, LZ4F_VERSION); 61 | CHECK(err, "Allocation failed (error %i)", (int)err); 62 | result = PyCapsule_New(cCtx, NULL, NULL); 63 | 64 | return result; 65 | _output_error: 66 | return Py_None; 67 | } 68 | 69 | static PyObject *py_lz4f_freeCompCtx(PyObject *self, PyObject *args) { 70 | PyObject *py_cCtx; 71 | LZ4F_compressionContext_t cCtx; 72 | 73 | (void)self; 74 | if (!PyArg_ParseTuple(args, "O", &py_cCtx)) { 75 | return NULL; 76 | } 77 | 78 | cCtx = (LZ4F_compressionContext_t)PyCapsule_GetPointer(py_cCtx, NULL); 79 | LZ4F_freeCompressionContext(cCtx); 80 | 81 | Py_RETURN_NONE; 82 | } 83 | 84 | static PyObject *py_lz4f_compressFrame(PyObject *self, PyObject *args) { 85 | PyObject *result; 86 | const char* source; 87 | char* dest; 88 | int src_size; 89 | size_t dest_size; 90 | size_t final_size; 91 | size_t ssrc_size; 92 | 93 | (void)self; 94 | if (!PyArg_ParseTuple(args, "s#", &source, &src_size)) { 95 | return NULL; 96 | } 97 | 98 | ssrc_size = (size_t)src_size; 99 | dest_size = LZ4F_compressFrameBound(ssrc_size, NULL); 100 | dest = (char*)malloc(dest_size); 101 | 102 | final_size = LZ4F_compressFrame(dest, dest_size, source, ssrc_size, NULL); 103 | result = PyBytes_FromStringAndSize(dest, final_size); 104 | 105 | free(dest); 106 | 107 | return result; 108 | } 109 | 110 | 111 | static PyObject *py_lz4f_makePrefs(PyObject *self, PyObject *args, PyObject *keywds) { 112 | LZ4F_frameInfo_t frameInfo; 113 | LZ4F_preferences_t* prefs; 114 | PyObject *result = PyDict_New(); 115 | static char *kwlist[] = {"blockSizeID", "blockMode", "chkFlag" 116 | "autoFlush"}; 117 | unsigned int blkID=7; 118 | unsigned int blkMode=1; 119 | unsigned int chkSumFlag=0; 120 | // unsigned int compLevel=0; //For future expansion 121 | unsigned int autoFlush=0; 122 | 123 | (void)self; 124 | if (!PyArg_ParseTupleAndKeywords(args, keywds, "|IIII", kwlist, &blkID, 125 | &blkMode, &chkSumFlag, &autoFlush)) { 126 | return NULL; 127 | } 128 | 129 | //fprintf(stdout, "blkID: %u As self.", blkID); 130 | prefs = calloc(1,sizeof(LZ4F_preferences_t)); 131 | frameInfo = (LZ4F_frameInfo_t){blkID, blkMode, chkSumFlag, 0, 0, {0,0}}; 132 | prefs->frameInfo = frameInfo; 133 | prefs->autoFlush = autoFlush; 134 | result = PyCapsule_New(prefs, NULL, NULL); 135 | 136 | return result; 137 | //_output_error: 138 | // return Py_None; 139 | } 140 | 141 | static PyObject *py_lz4f_compressBegin(PyObject *self, PyObject *args) { 142 | char* dest; 143 | LZ4F_compressionContext_t cCtx; 144 | LZ4F_preferences_t prefs = {{7, 0, 0, 0, 0, {0}}, 0, 0, {0}}; 145 | LZ4F_preferences_t* prefsPtr = &prefs; 146 | PyObject *result; 147 | PyObject *py_cCtx; 148 | PyObject *py_prefsPtr = Py_None; 149 | size_t dest_size; 150 | size_t final_size; 151 | 152 | (void)self; 153 | if (!PyArg_ParseTuple(args, "O|O", &py_cCtx, &py_prefsPtr)) { 154 | return NULL; 155 | } 156 | 157 | cCtx = (LZ4F_compressionContext_t)PyCapsule_GetPointer(py_cCtx, NULL); 158 | dest_size = 19; 159 | dest = (char*)malloc(dest_size); 160 | if (py_prefsPtr != Py_None) { 161 | prefsPtr = (LZ4F_preferences_t*)PyCapsule_GetPointer(py_prefsPtr, NULL); 162 | } 163 | 164 | final_size = LZ4F_compressBegin(cCtx, dest, dest_size, prefsPtr); 165 | CHECK(final_size); 166 | result = PyBytes_FromStringAndSize(dest, final_size); 167 | 168 | free(dest); 169 | 170 | return result; 171 | _output_error: 172 | return Py_None; 173 | } 174 | 175 | static PyObject *py_lz4f_compressUpdate(PyObject *self, PyObject *args) { 176 | const char* source; 177 | char* dest; 178 | int src_size; 179 | LZ4F_compressionContext_t cCtx; 180 | PyObject *result; 181 | PyObject *py_cCtx; 182 | size_t dest_size; 183 | size_t final_size; 184 | size_t ssrc_size; 185 | 186 | (void)self; 187 | if (!PyArg_ParseTuple(args, "s#O", &source, &src_size, &py_cCtx)) { 188 | return NULL; 189 | } 190 | 191 | cCtx = (LZ4F_compressionContext_t)PyCapsule_GetPointer(py_cCtx, NULL); 192 | ssrc_size = (size_t)src_size; 193 | dest_size = LZ4F_compressBound(ssrc_size, (LZ4F_preferences_t *)cCtx); 194 | dest = (char*)malloc(dest_size); 195 | 196 | final_size = LZ4F_compressUpdate(cCtx, dest, dest_size, source, ssrc_size, NULL); 197 | CHECK(final_size); 198 | result = PyBytes_FromStringAndSize(dest, final_size); 199 | 200 | free(dest); 201 | 202 | return result; 203 | _output_error: 204 | return Py_None; 205 | } 206 | 207 | static PyObject *py_lz4f_compressEnd(PyObject *self, PyObject *args) { 208 | char* dest; 209 | LZ4F_compressionContext_t cCtx; 210 | PyObject *result; 211 | PyObject *py_cCtx; 212 | size_t dest_size; 213 | size_t final_size; 214 | 215 | (void)self; 216 | if (!PyArg_ParseTuple(args, "O", &py_cCtx)) { 217 | return NULL; 218 | } 219 | 220 | cCtx = (LZ4F_compressionContext_t)PyCapsule_GetPointer(py_cCtx, NULL); 221 | dest_size = LZ4F_compressBound(0, (LZ4F_preferences_t *)cCtx); 222 | dest = (char*)malloc(dest_size); 223 | 224 | final_size = LZ4F_compressEnd(cCtx, dest, dest_size, NULL); 225 | CHECK(final_size); 226 | result = PyBytes_FromStringAndSize(dest, final_size); 227 | 228 | free(dest); 229 | 230 | return result; 231 | _output_error: 232 | return Py_None; 233 | } 234 | 235 | 236 | /* Decompression methods */ 237 | static PyObject *py_lz4f_createDecompCtx(PyObject *self, PyObject *args) { 238 | PyObject *result; 239 | LZ4F_decompressionContext_t dCtx; 240 | size_t err; 241 | 242 | (void)self; 243 | (void)args; 244 | 245 | err = LZ4F_createDecompressionContext(&dCtx, LZ4F_VERSION); 246 | CHECK(LZ4F_isError(err), "Allocation failed (error %i)", (int)err); 247 | result = PyCapsule_New(dCtx, NULL, NULL); 248 | 249 | return result; 250 | _output_error: 251 | return Py_None; 252 | } 253 | 254 | static PyObject *py_lz4f_freeDecompCtx(PyObject *self, PyObject *args) { 255 | PyObject *py_dCtx; 256 | LZ4F_decompressionContext_t dCtx; 257 | 258 | (void)self; 259 | if (!PyArg_ParseTuple(args, "O", &py_dCtx)) { 260 | return NULL; 261 | } 262 | 263 | dCtx = (LZ4F_decompressionContext_t)PyCapsule_GetPointer(py_dCtx, NULL); 264 | LZ4F_freeDecompressionContext(dCtx); 265 | 266 | Py_RETURN_NONE; 267 | } 268 | 269 | static PyObject *py_lz4f_getFrameInfo(PyObject *self, PyObject *args) { 270 | const char *source; 271 | int src_size; 272 | LZ4F_decompressionContext_t dCtx; 273 | LZ4F_frameInfo_t frameInfo; 274 | PyObject *blkSize; 275 | PyObject *blkMode; 276 | PyObject *contChkFlag; 277 | PyObject *py_dCtx; 278 | PyObject *result = PyDict_New(); 279 | size_t ssrc_size; 280 | size_t err; 281 | 282 | (void)self; 283 | if (!PyArg_ParseTuple(args, "s#O", &source, &src_size, &py_dCtx)) { 284 | return NULL; 285 | } 286 | 287 | dCtx = (LZ4F_decompressionContext_t)PyCapsule_GetPointer(py_dCtx, NULL); 288 | ssrc_size = (size_t)src_size; 289 | 290 | err = LZ4F_getFrameInfo(dCtx, &frameInfo, (unsigned char*)source, &ssrc_size); 291 | CHECK(LZ4F_isError(err), "Failed getting frameInfo. (error %i)", (int)err); 292 | 293 | blkSize = PyInt_FromSize_t(frameInfo.blockSizeID); 294 | blkMode = PyInt_FromSize_t(frameInfo.blockMode); 295 | contChkFlag = PyInt_FromSize_t(frameInfo.contentChecksumFlag); 296 | PyDict_SetItemString(result, "blkSize", blkSize); 297 | PyDict_SetItemString(result, "blkMode", blkMode); 298 | PyDict_SetItemString(result, "chkFlag", contChkFlag); 299 | 300 | 301 | return result; 302 | _output_error: 303 | return Py_None; 304 | } 305 | 306 | static PyObject *py_lz4f_disableChecksum(PyObject *self, PyObject *args) { 307 | PyObject *py_dCtx; 308 | LZ4F_decompressionContext_t dCtx; 309 | 310 | (void)self; 311 | if (!PyArg_ParseTuple(args, "O", &py_dCtx)) { 312 | return NULL; 313 | } 314 | 315 | dCtx = (LZ4F_decompressionContext_t)PyCapsule_GetPointer(py_dCtx, NULL); 316 | LZ4F_disableChecksum(dCtx); 317 | 318 | Py_RETURN_NONE; 319 | } 320 | 321 | static PyObject *py_lz4f_decompress(PyObject *self, PyObject *args, PyObject *keywds) { 322 | const char* source; 323 | char* dest; 324 | LZ4F_decompressionContext_t dCtx; 325 | int src_size; 326 | PyObject *decomp; 327 | PyObject *next; 328 | PyObject *py_dCtx; 329 | PyObject *result = PyDict_New(); 330 | size_t ssrc_size; 331 | size_t dest_size; 332 | size_t err; 333 | static char *kwlist[] = {"source", "dCtx", "blkSizeID"}; 334 | unsigned int blkID=7; 335 | 336 | (void)self; 337 | if (!PyArg_ParseTupleAndKeywords(args, keywds, "s#O|i", kwlist, &source, 338 | &src_size, &py_dCtx, &blkID)) { 339 | return NULL; 340 | } 341 | 342 | dest_size = LZ4S_GetBlockSize_FromBlockId(blkID); 343 | dCtx = (LZ4F_decompressionContext_t)PyCapsule_GetPointer(py_dCtx, NULL); 344 | ssrc_size = (size_t)src_size; 345 | 346 | dest = (char*)malloc(dest_size); 347 | err = LZ4F_decompress(dCtx, dest, &dest_size, source, &ssrc_size, NULL); 348 | CHECK(LZ4F_isError(err), "Failed getting frameInfo. (error %i)", (int)err); 349 | //fprintf(stdout, "Dest_size: %zu Error Code:%zu \n", dest_size, err); 350 | 351 | decomp = PyBytes_FromStringAndSize(dest, dest_size); 352 | next = PyInt_FromSize_t(err); 353 | PyDict_SetItemString(result, "decomp", decomp); 354 | PyDict_SetItemString(result, "next", next); 355 | 356 | Py_XDECREF(decomp); 357 | Py_XDECREF(next); 358 | free(dest); 359 | 360 | return result; 361 | _output_error: 362 | return Py_None; 363 | } 364 | 365 | static PyMethodDef Lz4fMethods[] = { 366 | {"createCompContext", py_lz4f_createCompCtx, METH_VARARGS, CCCTX_DOCSTRING}, 367 | {"compressFrame", py_lz4f_compressFrame, METH_VARARGS, COMPF_DOCSTRING}, 368 | {"makePrefs", (PyCFunction)py_lz4f_makePrefs, METH_VARARGS | METH_KEYWORDS, MKPFS_DOCSTRING}, 369 | {"compressBegin", py_lz4f_compressBegin, METH_VARARGS, COMPB_DOCSTRING}, 370 | {"compressUpdate", py_lz4f_compressUpdate, METH_VARARGS, COMPU_DOCSTRING}, 371 | {"compressEnd", py_lz4f_compressEnd, METH_VARARGS, COMPE_DOCSTRING}, 372 | {"freeCompContext", py_lz4f_freeCompCtx, METH_VARARGS, FCCTX_DOCSTRING}, 373 | {"createDecompContext", py_lz4f_createDecompCtx, METH_VARARGS, CDCTX_DOCSTRING}, 374 | {"freeDecompContext", py_lz4f_freeDecompCtx, METH_VARARGS, FDCTX_DOCSTRING}, 375 | {"getFrameInfo", py_lz4f_getFrameInfo, METH_VARARGS, GETFI_DOCSTRING}, 376 | {"decompressFrame", (PyCFunction)py_lz4f_decompress, METH_VARARGS | METH_KEYWORDS, DCOMP_DOCSTRING}, 377 | {"disableChecksum", py_lz4f_disableChecksum, METH_VARARGS, DCHKS_DOCSTRING}, 378 | {NULL, NULL, 0, NULL} 379 | }; 380 | 381 | struct module_state { 382 | PyObject *error; 383 | }; 384 | 385 | #if PY_MAJOR_VERSION >= 3 386 | #define GETSTATE(m) ((struct module_state*)PyModule_GetState(m)) 387 | #else 388 | #define GETSTATE(m) (&_state) 389 | static struct module_state _state; 390 | #endif 391 | 392 | #if PY_MAJOR_VERSION >= 3 393 | 394 | static int myextension_traverse(PyObject *m, visitproc visit, void *arg) { 395 | Py_VISIT(GETSTATE(m)->error); 396 | return 0; 397 | } 398 | 399 | static int myextension_clear(PyObject *m) { 400 | Py_CLEAR(GETSTATE(m)->error); 401 | return 0; 402 | } 403 | 404 | 405 | static struct PyModuleDef moduledef = { 406 | PyModuleDef_HEAD_INIT, 407 | "lz4f", 408 | NULL, 409 | sizeof(struct module_state), 410 | Lz4fMethods, 411 | NULL, 412 | myextension_traverse, 413 | myextension_clear, 414 | NULL 415 | }; 416 | 417 | #define INITERROR return NULL 418 | PyObject *PyInit_lz4f(void) 419 | 420 | #else 421 | #define INITERROR return 422 | void initlz4f(void) 423 | 424 | #endif 425 | { 426 | #if PY_MAJOR_VERSION >= 3 427 | PyObject *module = PyModule_Create(&moduledef); 428 | #else 429 | PyObject *module = Py_InitModule("lz4f", Lz4fMethods); 430 | #endif 431 | struct module_state *state = NULL; 432 | 433 | if (module == NULL) { 434 | INITERROR; 435 | } 436 | state = GETSTATE(module); 437 | 438 | state->error = PyErr_NewException("lz4.Error", NULL, NULL); 439 | if (state->error == NULL) { 440 | Py_DECREF(module); 441 | INITERROR; 442 | } 443 | 444 | PyModule_AddStringConstant(module, "VERSION", VERSION); 445 | PyModule_AddStringConstant(module, "__version__", VERSION); 446 | PyModule_AddStringConstant(module, "LZ4_VERSION", LZ4_VERSION); 447 | 448 | #if PY_MAJOR_VERSION >= 3 449 | return module; 450 | #endif 451 | } 452 | -------------------------------------------------------------------------------- /src/python-lz4f.h: -------------------------------------------------------------------------------- 1 | /* 2 | * Copyright (c) 2014, Christoper Jackson 3 | * All rights reserved. 4 | * 5 | * Redistribution and use in source and binary forms, with or without 6 | * modification, are permitted provided that the following conditions are met: 7 | * 8 | * 1. Redistributions of source code must retain the above copyright notice, 9 | * this list of conditions and the following disclaimer. 10 | * 11 | * 2. Redistributions in binary form must reproduce the above copyright notice, 12 | * this list of conditions and the following disclaimer in the documentation 13 | * and/or other materials provided with the distribution. 14 | * 15 | * 3. Neither the name of Christoper Jackson nor the names of its contributors may be 16 | * used to endorse or promote products derived from this software without 17 | * specific prior written permission. 18 | * 19 | * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" 20 | * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 21 | * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE 22 | * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE 23 | * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR 24 | * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF 25 | * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS 26 | * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN 27 | * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) 28 | * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE 29 | * POSSIBILITY OF SUCH DAMAGE. 30 | */ 31 | 32 | #include "Python.h" 33 | 34 | static PyObject *py_lz4f_createCompCtx(PyObject *self, PyObject *args); 35 | static PyObject *py_lz4f_compressFrame(PyObject *self, PyObject *args); 36 | static PyObject *py_lz4f_compressBegin(PyObject *self, PyObject *args); 37 | static PyObject *py_lz4f_compressUpdate(PyObject *self, PyObject *args); 38 | static PyObject *py_lz4f_compressEnd(PyObject *self, PyObject *args); 39 | static PyObject *py_lz4f_freeCompCtx(PyObject *self, PyObject *args); 40 | static PyObject *py_lz4f_createDecompCtx(PyObject *self, PyObject *args); 41 | static PyObject *py_lz4f_freeDecompCtx(PyObject *self, PyObject *args); 42 | static PyObject *py_lz4f_getFrameInfo(PyObject *self, PyObject *args); 43 | static PyObject *py_lz4f_decompress(PyObject *self, PyObject *args, PyObject *keywds); 44 | static PyObject *py_lz4f_disableChecksum(PyObject *self, PyObject *args); 45 | 46 | PyMODINIT_FUNC initlz4f(void); 47 | 48 | #define CTX_DOCSTRING "context, to be used with all respective functions." 49 | #define CCCTX_DOCSTRING "::rtype cCtx::\nGenerates a compression " CTX_DOCSTRING 50 | #define COMPF_DOCSTRING "(source)\n :type string: source\n::rtype string::\n" \ 51 | "Accepts a string, and compresses the string in one go, returning the compressed string. \n" \ 52 | "This generates a header, compressed data and endmark, making the result ready to be written to file." 53 | #define MKPFS_DOCSTRING "(blockSizeID=7, blockMode=1, chkFlag=0, autoFlush=0)\n:type int: blockSizeID\n:type int: blockMode\n" \ 54 | ":type int: chkFlag\n:type int: autoFlush\n::rtype PyCapsule:\n" \ 55 | "All accepted arguments are keyword args. Valid entries for blockSizeID are 4-7. Valid values for the\n" \ 56 | "remaining optional arguments are 0-1." 57 | #define COMPB_DOCSTRING "(cCtx)\n:type cCtx: cCtx\n::rtype string::\n" \ 58 | "Accepts a compression context as a PyCObject. Returns a frame header, based on context variables." 59 | #define COMPU_DOCSTRING "(source, cCtx)\n:type string: source\n:type cCtx: cCtx\n::rtype string::\n" \ 60 | "Accepts a string, and a compression context. Returns the string as a compressed block, if the\n" \ 61 | "block is filled. If not, it will return a blank string and hold the compressed data until the\n" \ 62 | "block is filled, flush is called or compressEnd is called." 63 | #define COMPE_DOCSTRING "(cCtx)\n:type cCtx: cCtx\n::rtype string::\n" \ 64 | "Accepts a compression context as a PyCObject. Flushed the holding buffer, applies endmark and if\n" \ 65 | "applicable will generate a checksum. Returns a string." 66 | #define FCCTX_DOCSTRING "(cCtx)\n:type cCtx: cCtx\n::NO RETURN::\nFrees a compression context, passed as a PyCObject." 67 | #define CDCTX_DOCSTRING "::rtype dCtx::\nGenerates a decompression " CTX_DOCSTRING 68 | #define FDCTX_DOCSTRING "(dCtx)\n:type dCtx: dCtx\n::NO RETURN::\nFrees a decompression context, passed as a PyCObject." 69 | #define GETFI_DOCSTRING "(header, dctx)\n:type string: header\n:type dCtx: dCtx\n" \ 70 | "::rtype dict:: {'chkFlag': int, 'blkSize': int, 'blkMode': int}\n" \ 71 | "Accepts a string, which should be the first 7 bytes of a lz4 file, the 'header,' and a dCtx PyCObject.\n" \ 72 | "Returns a dictionary object containing the frame info for the given header." 73 | #define DCHKS_DOCSTRING "(dCtx)\n:type dCtx: dCtx\nDisables the checksum portion of a the frameInfo struct in the dCtx. \n" \ 74 | "This is required for arbitrary seeking of a lz4 file. Without this, decompress will error out if blocks\n" \ 75 | "are read out of order." 76 | #define DCOMP_DOCSTRING "(source, dCtx, blkSizeID = 7)\n:type string: source\n:type dCtx: dCtx\n:type int: blkSizeID\n" \ 77 | "::rtype dict :: {'decomp': '', 'next': ''}\n" \ 78 | "Accepts required source string and decompressionContext, returns a dictionary containing the uncompressed\n" \ 79 | "data if block was complete and next block size. If block was incomplete, returns characters remaining to\n" \ 80 | "complete block. Raises an exception if any error occurs." 81 | 82 | #if defined(_WIN32) && defined(_MSC_VER) 83 | # define inline __inline 84 | # if _MSC_VER >= 1600 85 | # include 86 | # else /* _MSC_VER >= 1600 */ 87 | typedef signed char int8_t; 88 | typedef signed short int16_t; 89 | typedef signed int int32_t; 90 | typedef unsigned char uint8_t; 91 | typedef unsigned short uint16_t; 92 | typedef unsigned int uint32_t; 93 | # endif /* _MSC_VER >= 1600 */ 94 | #endif 95 | 96 | #if defined(__SUNPRO_C) || defined(__hpux) || defined(_AIX) 97 | #define inline 98 | #endif 99 | 100 | #ifdef __linux 101 | #define inline __inline 102 | #endif 103 | -------------------------------------------------------------------------------- /src/xxhash.c: -------------------------------------------------------------------------------- 1 | /* 2 | xxHash - Fast Hash algorithm 3 | Copyright (C) 2012-2015, Yann Collet 4 | 5 | BSD 2-Clause License (http://www.opensource.org/licenses/bsd-license.php) 6 | 7 | Redistribution and use in source and binary forms, with or without 8 | modification, are permitted provided that the following conditions are 9 | met: 10 | 11 | * Redistributions of source code must retain the above copyright 12 | notice, this list of conditions and the following disclaimer. 13 | * Redistributions in binary form must reproduce the above 14 | copyright notice, this list of conditions and the following disclaimer 15 | in the documentation and/or other materials provided with the 16 | distribution. 17 | 18 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 19 | "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT 20 | LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR 21 | A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT 22 | OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, 23 | SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT 24 | LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, 25 | DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY 26 | THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT 27 | (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 28 | OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 29 | 30 | You can contact the author at : 31 | - xxHash source repository : https://github.com/Cyan4973/xxHash 32 | */ 33 | 34 | 35 | /************************************** 36 | * Tuning parameters 37 | **************************************/ 38 | /* Unaligned memory access is automatically enabled for "common" CPU, such as x86. 39 | * For others CPU, the compiler will be more cautious, and insert extra code to ensure aligned access is respected. 40 | * If you know your target CPU supports unaligned memory access, you want to force this option manually to improve performance. 41 | * You can also enable this parameter if you know your input data will always be aligned (boundaries of 4, for U32). 42 | */ 43 | #if defined(__ARM_FEATURE_UNALIGNED) || defined(__i386) || defined(_M_IX86) || defined(__x86_64__) || defined(_M_X64) 44 | # define XXH_USE_UNALIGNED_ACCESS 1 45 | #endif 46 | 47 | /* XXH_ACCEPT_NULL_INPUT_POINTER : 48 | * If the input pointer is a null pointer, xxHash default behavior is to trigger a memory access error, since it is a bad pointer. 49 | * When this option is enabled, xxHash output for null input pointers will be the same as a null-length input. 50 | * By default, this option is disabled. To enable it, uncomment below define : 51 | */ 52 | /* #define XXH_ACCEPT_NULL_INPUT_POINTER 1 */ 53 | 54 | /* XXH_FORCE_NATIVE_FORMAT : 55 | * By default, xxHash library provides endian-independant Hash values, based on little-endian convention. 56 | * Results are therefore identical for little-endian and big-endian CPU. 57 | * This comes at a performance cost for big-endian CPU, since some swapping is required to emulate little-endian format. 58 | * Should endian-independance be of no importance for your application, you may set the #define below to 1. 59 | * It will improve speed for Big-endian CPU. 60 | * This option has no impact on Little_Endian CPU. 61 | */ 62 | #define XXH_FORCE_NATIVE_FORMAT 0 63 | 64 | 65 | /************************************** 66 | * Compiler Specific Options 67 | ***************************************/ 68 | #ifdef _MSC_VER /* Visual Studio */ 69 | # pragma warning(disable : 4127) /* disable: C4127: conditional expression is constant */ 70 | # define FORCE_INLINE static __forceinline 71 | #else 72 | # if defined (__STDC_VERSION__) && __STDC_VERSION__ >= 199901L /* C99 */ 73 | # ifdef __GNUC__ 74 | # define FORCE_INLINE static inline __attribute__((always_inline)) 75 | # else 76 | # define FORCE_INLINE static inline 77 | # endif 78 | # else 79 | # define FORCE_INLINE static 80 | # endif /* __STDC_VERSION__ */ 81 | #endif 82 | 83 | 84 | /************************************** 85 | * Includes & Memory related functions 86 | ***************************************/ 87 | #include "xxhash.h" 88 | /* Modify the local functions below should you wish to use some other memory routines */ 89 | /* for malloc(), free() */ 90 | #include 91 | static void* XXH_malloc(size_t s) { return malloc(s); } 92 | static void XXH_free (void* p) { free(p); } 93 | /* for memcpy() */ 94 | #include 95 | static void* XXH_memcpy(void* dest, const void* src, size_t size) { return memcpy(dest,src,size); } 96 | 97 | 98 | /************************************** 99 | * Basic Types 100 | ***************************************/ 101 | #if defined (__STDC_VERSION__) && __STDC_VERSION__ >= 199901L /* C99 */ 102 | # include 103 | typedef uint8_t BYTE; 104 | typedef uint16_t U16; 105 | typedef uint32_t U32; 106 | typedef int32_t S32; 107 | typedef uint64_t U64; 108 | #else 109 | typedef unsigned char BYTE; 110 | typedef unsigned short U16; 111 | typedef unsigned int U32; 112 | typedef signed int S32; 113 | typedef unsigned long long U64; 114 | #endif 115 | 116 | static U32 XXH_read32(const void* memPtr) 117 | { 118 | U32 val32; 119 | memcpy(&val32, memPtr, 4); 120 | return val32; 121 | } 122 | 123 | static U64 XXH_read64(const void* memPtr) 124 | { 125 | U64 val64; 126 | memcpy(&val64, memPtr, 8); 127 | return val64; 128 | } 129 | 130 | 131 | 132 | /****************************************** 133 | * Compiler-specific Functions and Macros 134 | ******************************************/ 135 | #define GCC_VERSION (__GNUC__ * 100 + __GNUC_MINOR__) 136 | 137 | /* Note : although _rotl exists for minGW (GCC under windows), performance seems poor */ 138 | #if defined(_MSC_VER) 139 | # define XXH_rotl32(x,r) _rotl(x,r) 140 | # define XXH_rotl64(x,r) _rotl64(x,r) 141 | #else 142 | # define XXH_rotl32(x,r) ((x << r) | (x >> (32 - r))) 143 | # define XXH_rotl64(x,r) ((x << r) | (x >> (64 - r))) 144 | #endif 145 | 146 | #if defined(_MSC_VER) /* Visual Studio */ 147 | # define XXH_swap32 _byteswap_ulong 148 | # define XXH_swap64 _byteswap_uint64 149 | #elif GCC_VERSION >= 403 150 | # define XXH_swap32 __builtin_bswap32 151 | # define XXH_swap64 __builtin_bswap64 152 | #else 153 | static U32 XXH_swap32 (U32 x) 154 | { 155 | return ((x << 24) & 0xff000000 ) | 156 | ((x << 8) & 0x00ff0000 ) | 157 | ((x >> 8) & 0x0000ff00 ) | 158 | ((x >> 24) & 0x000000ff ); 159 | } 160 | static U64 XXH_swap64 (U64 x) 161 | { 162 | return ((x << 56) & 0xff00000000000000ULL) | 163 | ((x << 40) & 0x00ff000000000000ULL) | 164 | ((x << 24) & 0x0000ff0000000000ULL) | 165 | ((x << 8) & 0x000000ff00000000ULL) | 166 | ((x >> 8) & 0x00000000ff000000ULL) | 167 | ((x >> 24) & 0x0000000000ff0000ULL) | 168 | ((x >> 40) & 0x000000000000ff00ULL) | 169 | ((x >> 56) & 0x00000000000000ffULL); 170 | } 171 | #endif 172 | 173 | 174 | /*************************************** 175 | * Architecture Macros 176 | ***************************************/ 177 | typedef enum { XXH_bigEndian=0, XXH_littleEndian=1 } XXH_endianess; 178 | #ifndef XXH_CPU_LITTLE_ENDIAN /* XXH_CPU_LITTLE_ENDIAN can be defined externally, for example using a compiler switch */ 179 | static const int one = 1; 180 | # define XXH_CPU_LITTLE_ENDIAN (*(const char*)(&one)) 181 | #endif 182 | 183 | 184 | /***************************** 185 | * Memory reads 186 | *****************************/ 187 | typedef enum { XXH_aligned, XXH_unaligned } XXH_alignment; 188 | 189 | FORCE_INLINE U32 XXH_readLE32_align(const void* ptr, XXH_endianess endian, XXH_alignment align) 190 | { 191 | if (align==XXH_unaligned) 192 | return endian==XXH_littleEndian ? XXH_read32(ptr) : XXH_swap32(XXH_read32(ptr)); 193 | else 194 | return endian==XXH_littleEndian ? *(const U32*)ptr : XXH_swap32(*(const U32*)ptr); 195 | } 196 | 197 | FORCE_INLINE U32 XXH_readLE32(const void* ptr, XXH_endianess endian) 198 | { 199 | return XXH_readLE32_align(ptr, endian, XXH_unaligned); 200 | } 201 | 202 | FORCE_INLINE U64 XXH_readLE64_align(const void* ptr, XXH_endianess endian, XXH_alignment align) 203 | { 204 | if (align==XXH_unaligned) 205 | return endian==XXH_littleEndian ? XXH_read64(ptr) : XXH_swap64(XXH_read64(ptr)); 206 | else 207 | return endian==XXH_littleEndian ? *(const U64*)ptr : XXH_swap64(*(const U64*)ptr); 208 | } 209 | 210 | FORCE_INLINE U64 XXH_readLE64(const void* ptr, XXH_endianess endian) 211 | { 212 | return XXH_readLE64_align(ptr, endian, XXH_unaligned); 213 | } 214 | 215 | 216 | /*************************************** 217 | * Macros 218 | ***************************************/ 219 | #define XXH_STATIC_ASSERT(c) { enum { XXH_static_assert = 1/(!!(c)) }; } /* use only *after* variable declarations */ 220 | 221 | 222 | /*************************************** 223 | * Constants 224 | ***************************************/ 225 | #define PRIME32_1 2654435761U 226 | #define PRIME32_2 2246822519U 227 | #define PRIME32_3 3266489917U 228 | #define PRIME32_4 668265263U 229 | #define PRIME32_5 374761393U 230 | 231 | #define PRIME64_1 11400714785074694791ULL 232 | #define PRIME64_2 14029467366897019727ULL 233 | #define PRIME64_3 1609587929392839161ULL 234 | #define PRIME64_4 9650029242287828579ULL 235 | #define PRIME64_5 2870177450012600261ULL 236 | 237 | 238 | /***************************** 239 | * Simple Hash Functions 240 | *****************************/ 241 | FORCE_INLINE U32 XXH32_endian_align(const void* input, size_t len, U32 seed, XXH_endianess endian, XXH_alignment align) 242 | { 243 | const BYTE* p = (const BYTE*)input; 244 | const BYTE* bEnd = p + len; 245 | U32 h32; 246 | #define XXH_get32bits(p) XXH_readLE32_align(p, endian, align) 247 | 248 | #ifdef XXH_ACCEPT_NULL_INPUT_POINTER 249 | if (p==NULL) 250 | { 251 | len=0; 252 | bEnd=p=(const BYTE*)(size_t)16; 253 | } 254 | #endif 255 | 256 | if (len>=16) 257 | { 258 | const BYTE* const limit = bEnd - 16; 259 | U32 v1 = seed + PRIME32_1 + PRIME32_2; 260 | U32 v2 = seed + PRIME32_2; 261 | U32 v3 = seed + 0; 262 | U32 v4 = seed - PRIME32_1; 263 | 264 | do 265 | { 266 | v1 += XXH_get32bits(p) * PRIME32_2; 267 | v1 = XXH_rotl32(v1, 13); 268 | v1 *= PRIME32_1; 269 | p+=4; 270 | v2 += XXH_get32bits(p) * PRIME32_2; 271 | v2 = XXH_rotl32(v2, 13); 272 | v2 *= PRIME32_1; 273 | p+=4; 274 | v3 += XXH_get32bits(p) * PRIME32_2; 275 | v3 = XXH_rotl32(v3, 13); 276 | v3 *= PRIME32_1; 277 | p+=4; 278 | v4 += XXH_get32bits(p) * PRIME32_2; 279 | v4 = XXH_rotl32(v4, 13); 280 | v4 *= PRIME32_1; 281 | p+=4; 282 | } 283 | while (p<=limit); 284 | 285 | h32 = XXH_rotl32(v1, 1) + XXH_rotl32(v2, 7) + XXH_rotl32(v3, 12) + XXH_rotl32(v4, 18); 286 | } 287 | else 288 | { 289 | h32 = seed + PRIME32_5; 290 | } 291 | 292 | h32 += (U32) len; 293 | 294 | while (p+4<=bEnd) 295 | { 296 | h32 += XXH_get32bits(p) * PRIME32_3; 297 | h32 = XXH_rotl32(h32, 17) * PRIME32_4 ; 298 | p+=4; 299 | } 300 | 301 | while (p> 15; 309 | h32 *= PRIME32_2; 310 | h32 ^= h32 >> 13; 311 | h32 *= PRIME32_3; 312 | h32 ^= h32 >> 16; 313 | 314 | return h32; 315 | } 316 | 317 | 318 | unsigned XXH32 (const void* input, size_t len, unsigned seed) 319 | { 320 | #if 0 321 | /* Simple version, good for code maintenance, but unfortunately slow for small inputs */ 322 | XXH32_state_t state; 323 | XXH32_reset(&state, seed); 324 | XXH32_update(&state, input, len); 325 | return XXH32_digest(&state); 326 | #else 327 | XXH_endianess endian_detected = (XXH_endianess)XXH_CPU_LITTLE_ENDIAN; 328 | 329 | # if !defined(XXH_USE_UNALIGNED_ACCESS) 330 | if ((((size_t)input) & 3) == 0) /* Input is 4-bytes aligned, leverage the speed benefit */ 331 | { 332 | if ((endian_detected==XXH_littleEndian) || XXH_FORCE_NATIVE_FORMAT) 333 | return XXH32_endian_align(input, len, seed, XXH_littleEndian, XXH_aligned); 334 | else 335 | return XXH32_endian_align(input, len, seed, XXH_bigEndian, XXH_aligned); 336 | } 337 | # endif 338 | 339 | if ((endian_detected==XXH_littleEndian) || XXH_FORCE_NATIVE_FORMAT) 340 | return XXH32_endian_align(input, len, seed, XXH_littleEndian, XXH_unaligned); 341 | else 342 | return XXH32_endian_align(input, len, seed, XXH_bigEndian, XXH_unaligned); 343 | #endif 344 | } 345 | 346 | FORCE_INLINE U64 XXH64_endian_align(const void* input, size_t len, U64 seed, XXH_endianess endian, XXH_alignment align) 347 | { 348 | const BYTE* p = (const BYTE*)input; 349 | const BYTE* bEnd = p + len; 350 | U64 h64; 351 | #define XXH_get64bits(p) XXH_readLE64_align(p, endian, align) 352 | 353 | #ifdef XXH_ACCEPT_NULL_INPUT_POINTER 354 | if (p==NULL) 355 | { 356 | len=0; 357 | bEnd=p=(const BYTE*)(size_t)32; 358 | } 359 | #endif 360 | 361 | if (len>=32) 362 | { 363 | const BYTE* const limit = bEnd - 32; 364 | U64 v1 = seed + PRIME64_1 + PRIME64_2; 365 | U64 v2 = seed + PRIME64_2; 366 | U64 v3 = seed + 0; 367 | U64 v4 = seed - PRIME64_1; 368 | 369 | do 370 | { 371 | v1 += XXH_get64bits(p) * PRIME64_2; 372 | p+=8; 373 | v1 = XXH_rotl64(v1, 31); 374 | v1 *= PRIME64_1; 375 | v2 += XXH_get64bits(p) * PRIME64_2; 376 | p+=8; 377 | v2 = XXH_rotl64(v2, 31); 378 | v2 *= PRIME64_1; 379 | v3 += XXH_get64bits(p) * PRIME64_2; 380 | p+=8; 381 | v3 = XXH_rotl64(v3, 31); 382 | v3 *= PRIME64_1; 383 | v4 += XXH_get64bits(p) * PRIME64_2; 384 | p+=8; 385 | v4 = XXH_rotl64(v4, 31); 386 | v4 *= PRIME64_1; 387 | } 388 | while (p<=limit); 389 | 390 | h64 = XXH_rotl64(v1, 1) + XXH_rotl64(v2, 7) + XXH_rotl64(v3, 12) + XXH_rotl64(v4, 18); 391 | 392 | v1 *= PRIME64_2; 393 | v1 = XXH_rotl64(v1, 31); 394 | v1 *= PRIME64_1; 395 | h64 ^= v1; 396 | h64 = h64 * PRIME64_1 + PRIME64_4; 397 | 398 | v2 *= PRIME64_2; 399 | v2 = XXH_rotl64(v2, 31); 400 | v2 *= PRIME64_1; 401 | h64 ^= v2; 402 | h64 = h64 * PRIME64_1 + PRIME64_4; 403 | 404 | v3 *= PRIME64_2; 405 | v3 = XXH_rotl64(v3, 31); 406 | v3 *= PRIME64_1; 407 | h64 ^= v3; 408 | h64 = h64 * PRIME64_1 + PRIME64_4; 409 | 410 | v4 *= PRIME64_2; 411 | v4 = XXH_rotl64(v4, 31); 412 | v4 *= PRIME64_1; 413 | h64 ^= v4; 414 | h64 = h64 * PRIME64_1 + PRIME64_4; 415 | } 416 | else 417 | { 418 | h64 = seed + PRIME64_5; 419 | } 420 | 421 | h64 += (U64) len; 422 | 423 | while (p+8<=bEnd) 424 | { 425 | U64 k1 = XXH_get64bits(p); 426 | k1 *= PRIME64_2; 427 | k1 = XXH_rotl64(k1,31); 428 | k1 *= PRIME64_1; 429 | h64 ^= k1; 430 | h64 = XXH_rotl64(h64,27) * PRIME64_1 + PRIME64_4; 431 | p+=8; 432 | } 433 | 434 | if (p+4<=bEnd) 435 | { 436 | h64 ^= (U64)(XXH_get32bits(p)) * PRIME64_1; 437 | h64 = XXH_rotl64(h64, 23) * PRIME64_2 + PRIME64_3; 438 | p+=4; 439 | } 440 | 441 | while (p> 33; 449 | h64 *= PRIME64_2; 450 | h64 ^= h64 >> 29; 451 | h64 *= PRIME64_3; 452 | h64 ^= h64 >> 32; 453 | 454 | return h64; 455 | } 456 | 457 | 458 | unsigned long long XXH64 (const void* input, size_t len, unsigned long long seed) 459 | { 460 | #if 0 461 | /* Simple version, good for code maintenance, but unfortunately slow for small inputs */ 462 | XXH64_state_t state; 463 | XXH64_reset(&state, seed); 464 | XXH64_update(&state, input, len); 465 | return XXH64_digest(&state); 466 | #else 467 | XXH_endianess endian_detected = (XXH_endianess)XXH_CPU_LITTLE_ENDIAN; 468 | 469 | # if !defined(XXH_USE_UNALIGNED_ACCESS) 470 | if ((((size_t)input) & 7)==0) /* Input is aligned, let's leverage the speed advantage */ 471 | { 472 | if ((endian_detected==XXH_littleEndian) || XXH_FORCE_NATIVE_FORMAT) 473 | return XXH64_endian_align(input, len, seed, XXH_littleEndian, XXH_aligned); 474 | else 475 | return XXH64_endian_align(input, len, seed, XXH_bigEndian, XXH_aligned); 476 | } 477 | # endif 478 | 479 | if ((endian_detected==XXH_littleEndian) || XXH_FORCE_NATIVE_FORMAT) 480 | return XXH64_endian_align(input, len, seed, XXH_littleEndian, XXH_unaligned); 481 | else 482 | return XXH64_endian_align(input, len, seed, XXH_bigEndian, XXH_unaligned); 483 | #endif 484 | } 485 | 486 | /**************************************************** 487 | * Advanced Hash Functions 488 | ****************************************************/ 489 | 490 | /*** Allocation ***/ 491 | typedef struct 492 | { 493 | U64 total_len; 494 | U32 seed; 495 | U32 v1; 496 | U32 v2; 497 | U32 v3; 498 | U32 v4; 499 | U32 mem32[4]; /* defined as U32 for alignment */ 500 | U32 memsize; 501 | } XXH_istate32_t; 502 | 503 | typedef struct 504 | { 505 | U64 total_len; 506 | U64 seed; 507 | U64 v1; 508 | U64 v2; 509 | U64 v3; 510 | U64 v4; 511 | U64 mem64[4]; /* defined as U64 for alignment */ 512 | U32 memsize; 513 | } XXH_istate64_t; 514 | 515 | 516 | XXH32_state_t* XXH32_createState(void) 517 | { 518 | XXH_STATIC_ASSERT(sizeof(XXH32_state_t) >= sizeof(XXH_istate32_t)); /* A compilation error here means XXH32_state_t is not large enough */ 519 | return (XXH32_state_t*)XXH_malloc(sizeof(XXH32_state_t)); 520 | } 521 | XXH_errorcode XXH32_freeState(XXH32_state_t* statePtr) 522 | { 523 | XXH_free(statePtr); 524 | return XXH_OK; 525 | } 526 | 527 | XXH64_state_t* XXH64_createState(void) 528 | { 529 | XXH_STATIC_ASSERT(sizeof(XXH64_state_t) >= sizeof(XXH_istate64_t)); /* A compilation error here means XXH64_state_t is not large enough */ 530 | return (XXH64_state_t*)XXH_malloc(sizeof(XXH64_state_t)); 531 | } 532 | XXH_errorcode XXH64_freeState(XXH64_state_t* statePtr) 533 | { 534 | XXH_free(statePtr); 535 | return XXH_OK; 536 | } 537 | 538 | 539 | /*** Hash feed ***/ 540 | 541 | XXH_errorcode XXH32_reset(XXH32_state_t* state_in, U32 seed) 542 | { 543 | XXH_istate32_t* state = (XXH_istate32_t*) state_in; 544 | state->seed = seed; 545 | state->v1 = seed + PRIME32_1 + PRIME32_2; 546 | state->v2 = seed + PRIME32_2; 547 | state->v3 = seed + 0; 548 | state->v4 = seed - PRIME32_1; 549 | state->total_len = 0; 550 | state->memsize = 0; 551 | return XXH_OK; 552 | } 553 | 554 | XXH_errorcode XXH64_reset(XXH64_state_t* state_in, unsigned long long seed) 555 | { 556 | XXH_istate64_t* state = (XXH_istate64_t*) state_in; 557 | state->seed = seed; 558 | state->v1 = seed + PRIME64_1 + PRIME64_2; 559 | state->v2 = seed + PRIME64_2; 560 | state->v3 = seed + 0; 561 | state->v4 = seed - PRIME64_1; 562 | state->total_len = 0; 563 | state->memsize = 0; 564 | return XXH_OK; 565 | } 566 | 567 | 568 | FORCE_INLINE XXH_errorcode XXH32_update_endian (XXH32_state_t* state_in, const void* input, size_t len, XXH_endianess endian) 569 | { 570 | XXH_istate32_t* state = (XXH_istate32_t *) state_in; 571 | const BYTE* p = (const BYTE*)input; 572 | const BYTE* const bEnd = p + len; 573 | 574 | #ifdef XXH_ACCEPT_NULL_INPUT_POINTER 575 | if (input==NULL) return XXH_ERROR; 576 | #endif 577 | 578 | state->total_len += len; 579 | 580 | if (state->memsize + len < 16) /* fill in tmp buffer */ 581 | { 582 | XXH_memcpy((BYTE*)(state->mem32) + state->memsize, input, len); 583 | state->memsize += (U32)len; 584 | return XXH_OK; 585 | } 586 | 587 | if (state->memsize) /* some data left from previous update */ 588 | { 589 | XXH_memcpy((BYTE*)(state->mem32) + state->memsize, input, 16-state->memsize); 590 | { 591 | const U32* p32 = state->mem32; 592 | state->v1 += XXH_readLE32(p32, endian) * PRIME32_2; 593 | state->v1 = XXH_rotl32(state->v1, 13); 594 | state->v1 *= PRIME32_1; 595 | p32++; 596 | state->v2 += XXH_readLE32(p32, endian) * PRIME32_2; 597 | state->v2 = XXH_rotl32(state->v2, 13); 598 | state->v2 *= PRIME32_1; 599 | p32++; 600 | state->v3 += XXH_readLE32(p32, endian) * PRIME32_2; 601 | state->v3 = XXH_rotl32(state->v3, 13); 602 | state->v3 *= PRIME32_1; 603 | p32++; 604 | state->v4 += XXH_readLE32(p32, endian) * PRIME32_2; 605 | state->v4 = XXH_rotl32(state->v4, 13); 606 | state->v4 *= PRIME32_1; 607 | p32++; 608 | } 609 | p += 16-state->memsize; 610 | state->memsize = 0; 611 | } 612 | 613 | if (p <= bEnd-16) 614 | { 615 | const BYTE* const limit = bEnd - 16; 616 | U32 v1 = state->v1; 617 | U32 v2 = state->v2; 618 | U32 v3 = state->v3; 619 | U32 v4 = state->v4; 620 | 621 | do 622 | { 623 | v1 += XXH_readLE32(p, endian) * PRIME32_2; 624 | v1 = XXH_rotl32(v1, 13); 625 | v1 *= PRIME32_1; 626 | p+=4; 627 | v2 += XXH_readLE32(p, endian) * PRIME32_2; 628 | v2 = XXH_rotl32(v2, 13); 629 | v2 *= PRIME32_1; 630 | p+=4; 631 | v3 += XXH_readLE32(p, endian) * PRIME32_2; 632 | v3 = XXH_rotl32(v3, 13); 633 | v3 *= PRIME32_1; 634 | p+=4; 635 | v4 += XXH_readLE32(p, endian) * PRIME32_2; 636 | v4 = XXH_rotl32(v4, 13); 637 | v4 *= PRIME32_1; 638 | p+=4; 639 | } 640 | while (p<=limit); 641 | 642 | state->v1 = v1; 643 | state->v2 = v2; 644 | state->v3 = v3; 645 | state->v4 = v4; 646 | } 647 | 648 | if (p < bEnd) 649 | { 650 | XXH_memcpy(state->mem32, p, bEnd-p); 651 | state->memsize = (int)(bEnd-p); 652 | } 653 | 654 | return XXH_OK; 655 | } 656 | 657 | XXH_errorcode XXH32_update (XXH32_state_t* state_in, const void* input, size_t len) 658 | { 659 | XXH_endianess endian_detected = (XXH_endianess)XXH_CPU_LITTLE_ENDIAN; 660 | 661 | if ((endian_detected==XXH_littleEndian) || XXH_FORCE_NATIVE_FORMAT) 662 | return XXH32_update_endian(state_in, input, len, XXH_littleEndian); 663 | else 664 | return XXH32_update_endian(state_in, input, len, XXH_bigEndian); 665 | } 666 | 667 | 668 | 669 | FORCE_INLINE U32 XXH32_digest_endian (const XXH32_state_t* state_in, XXH_endianess endian) 670 | { 671 | const XXH_istate32_t* state = (const XXH_istate32_t*) state_in; 672 | const BYTE * p = (const BYTE*)state->mem32; 673 | const BYTE* bEnd = (const BYTE*)(state->mem32) + state->memsize; 674 | U32 h32; 675 | 676 | if (state->total_len >= 16) 677 | { 678 | h32 = XXH_rotl32(state->v1, 1) + XXH_rotl32(state->v2, 7) + XXH_rotl32(state->v3, 12) + XXH_rotl32(state->v4, 18); 679 | } 680 | else 681 | { 682 | h32 = state->seed + PRIME32_5; 683 | } 684 | 685 | h32 += (U32) state->total_len; 686 | 687 | while (p+4<=bEnd) 688 | { 689 | h32 += XXH_readLE32(p, endian) * PRIME32_3; 690 | h32 = XXH_rotl32(h32, 17) * PRIME32_4; 691 | p+=4; 692 | } 693 | 694 | while (p> 15; 702 | h32 *= PRIME32_2; 703 | h32 ^= h32 >> 13; 704 | h32 *= PRIME32_3; 705 | h32 ^= h32 >> 16; 706 | 707 | return h32; 708 | } 709 | 710 | 711 | U32 XXH32_digest (const XXH32_state_t* state_in) 712 | { 713 | XXH_endianess endian_detected = (XXH_endianess)XXH_CPU_LITTLE_ENDIAN; 714 | 715 | if ((endian_detected==XXH_littleEndian) || XXH_FORCE_NATIVE_FORMAT) 716 | return XXH32_digest_endian(state_in, XXH_littleEndian); 717 | else 718 | return XXH32_digest_endian(state_in, XXH_bigEndian); 719 | } 720 | 721 | 722 | FORCE_INLINE XXH_errorcode XXH64_update_endian (XXH64_state_t* state_in, const void* input, size_t len, XXH_endianess endian) 723 | { 724 | XXH_istate64_t * state = (XXH_istate64_t *) state_in; 725 | const BYTE* p = (const BYTE*)input; 726 | const BYTE* const bEnd = p + len; 727 | 728 | #ifdef XXH_ACCEPT_NULL_INPUT_POINTER 729 | if (input==NULL) return XXH_ERROR; 730 | #endif 731 | 732 | state->total_len += len; 733 | 734 | if (state->memsize + len < 32) /* fill in tmp buffer */ 735 | { 736 | XXH_memcpy(((BYTE*)state->mem64) + state->memsize, input, len); 737 | state->memsize += (U32)len; 738 | return XXH_OK; 739 | } 740 | 741 | if (state->memsize) /* some data left from previous update */ 742 | { 743 | XXH_memcpy(((BYTE*)state->mem64) + state->memsize, input, 32-state->memsize); 744 | { 745 | const U64* p64 = state->mem64; 746 | state->v1 += XXH_readLE64(p64, endian) * PRIME64_2; 747 | state->v1 = XXH_rotl64(state->v1, 31); 748 | state->v1 *= PRIME64_1; 749 | p64++; 750 | state->v2 += XXH_readLE64(p64, endian) * PRIME64_2; 751 | state->v2 = XXH_rotl64(state->v2, 31); 752 | state->v2 *= PRIME64_1; 753 | p64++; 754 | state->v3 += XXH_readLE64(p64, endian) * PRIME64_2; 755 | state->v3 = XXH_rotl64(state->v3, 31); 756 | state->v3 *= PRIME64_1; 757 | p64++; 758 | state->v4 += XXH_readLE64(p64, endian) * PRIME64_2; 759 | state->v4 = XXH_rotl64(state->v4, 31); 760 | state->v4 *= PRIME64_1; 761 | p64++; 762 | } 763 | p += 32-state->memsize; 764 | state->memsize = 0; 765 | } 766 | 767 | if (p+32 <= bEnd) 768 | { 769 | const BYTE* const limit = bEnd - 32; 770 | U64 v1 = state->v1; 771 | U64 v2 = state->v2; 772 | U64 v3 = state->v3; 773 | U64 v4 = state->v4; 774 | 775 | do 776 | { 777 | v1 += XXH_readLE64(p, endian) * PRIME64_2; 778 | v1 = XXH_rotl64(v1, 31); 779 | v1 *= PRIME64_1; 780 | p+=8; 781 | v2 += XXH_readLE64(p, endian) * PRIME64_2; 782 | v2 = XXH_rotl64(v2, 31); 783 | v2 *= PRIME64_1; 784 | p+=8; 785 | v3 += XXH_readLE64(p, endian) * PRIME64_2; 786 | v3 = XXH_rotl64(v3, 31); 787 | v3 *= PRIME64_1; 788 | p+=8; 789 | v4 += XXH_readLE64(p, endian) * PRIME64_2; 790 | v4 = XXH_rotl64(v4, 31); 791 | v4 *= PRIME64_1; 792 | p+=8; 793 | } 794 | while (p<=limit); 795 | 796 | state->v1 = v1; 797 | state->v2 = v2; 798 | state->v3 = v3; 799 | state->v4 = v4; 800 | } 801 | 802 | if (p < bEnd) 803 | { 804 | XXH_memcpy(state->mem64, p, bEnd-p); 805 | state->memsize = (int)(bEnd-p); 806 | } 807 | 808 | return XXH_OK; 809 | } 810 | 811 | XXH_errorcode XXH64_update (XXH64_state_t* state_in, const void* input, size_t len) 812 | { 813 | XXH_endianess endian_detected = (XXH_endianess)XXH_CPU_LITTLE_ENDIAN; 814 | 815 | if ((endian_detected==XXH_littleEndian) || XXH_FORCE_NATIVE_FORMAT) 816 | return XXH64_update_endian(state_in, input, len, XXH_littleEndian); 817 | else 818 | return XXH64_update_endian(state_in, input, len, XXH_bigEndian); 819 | } 820 | 821 | 822 | 823 | FORCE_INLINE U64 XXH64_digest_endian (const XXH64_state_t* state_in, XXH_endianess endian) 824 | { 825 | const XXH_istate64_t * state = (const XXH_istate64_t *) state_in; 826 | const BYTE * p = (const BYTE*)state->mem64; 827 | const BYTE* bEnd = (const BYTE*)state->mem64 + state->memsize; 828 | U64 h64; 829 | 830 | if (state->total_len >= 32) 831 | { 832 | U64 v1 = state->v1; 833 | U64 v2 = state->v2; 834 | U64 v3 = state->v3; 835 | U64 v4 = state->v4; 836 | 837 | h64 = XXH_rotl64(v1, 1) + XXH_rotl64(v2, 7) + XXH_rotl64(v3, 12) + XXH_rotl64(v4, 18); 838 | 839 | v1 *= PRIME64_2; 840 | v1 = XXH_rotl64(v1, 31); 841 | v1 *= PRIME64_1; 842 | h64 ^= v1; 843 | h64 = h64*PRIME64_1 + PRIME64_4; 844 | 845 | v2 *= PRIME64_2; 846 | v2 = XXH_rotl64(v2, 31); 847 | v2 *= PRIME64_1; 848 | h64 ^= v2; 849 | h64 = h64*PRIME64_1 + PRIME64_4; 850 | 851 | v3 *= PRIME64_2; 852 | v3 = XXH_rotl64(v3, 31); 853 | v3 *= PRIME64_1; 854 | h64 ^= v3; 855 | h64 = h64*PRIME64_1 + PRIME64_4; 856 | 857 | v4 *= PRIME64_2; 858 | v4 = XXH_rotl64(v4, 31); 859 | v4 *= PRIME64_1; 860 | h64 ^= v4; 861 | h64 = h64*PRIME64_1 + PRIME64_4; 862 | } 863 | else 864 | { 865 | h64 = state->seed + PRIME64_5; 866 | } 867 | 868 | h64 += (U64) state->total_len; 869 | 870 | while (p+8<=bEnd) 871 | { 872 | U64 k1 = XXH_readLE64(p, endian); 873 | k1 *= PRIME64_2; 874 | k1 = XXH_rotl64(k1,31); 875 | k1 *= PRIME64_1; 876 | h64 ^= k1; 877 | h64 = XXH_rotl64(h64,27) * PRIME64_1 + PRIME64_4; 878 | p+=8; 879 | } 880 | 881 | if (p+4<=bEnd) 882 | { 883 | h64 ^= (U64)(XXH_readLE32(p, endian)) * PRIME64_1; 884 | h64 = XXH_rotl64(h64, 23) * PRIME64_2 + PRIME64_3; 885 | p+=4; 886 | } 887 | 888 | while (p> 33; 896 | h64 *= PRIME64_2; 897 | h64 ^= h64 >> 29; 898 | h64 *= PRIME64_3; 899 | h64 ^= h64 >> 32; 900 | 901 | return h64; 902 | } 903 | 904 | 905 | unsigned long long XXH64_digest (const XXH64_state_t* state_in) 906 | { 907 | XXH_endianess endian_detected = (XXH_endianess)XXH_CPU_LITTLE_ENDIAN; 908 | 909 | if ((endian_detected==XXH_littleEndian) || XXH_FORCE_NATIVE_FORMAT) 910 | return XXH64_digest_endian(state_in, XXH_littleEndian); 911 | else 912 | return XXH64_digest_endian(state_in, XXH_bigEndian); 913 | } 914 | 915 | 916 | -------------------------------------------------------------------------------- /src/xxhash.h: -------------------------------------------------------------------------------- 1 | /* 2 | xxHash - Extremely Fast Hash algorithm 3 | Header File 4 | Copyright (C) 2012-2015, Yann Collet. 5 | 6 | BSD 2-Clause License (http://www.opensource.org/licenses/bsd-license.php) 7 | 8 | Redistribution and use in source and binary forms, with or without 9 | modification, are permitted provided that the following conditions are 10 | met: 11 | 12 | * Redistributions of source code must retain the above copyright 13 | notice, this list of conditions and the following disclaimer. 14 | * Redistributions in binary form must reproduce the above 15 | copyright notice, this list of conditions and the following disclaimer 16 | in the documentation and/or other materials provided with the 17 | distribution. 18 | 19 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 20 | "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT 21 | LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR 22 | A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT 23 | OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, 24 | SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT 25 | LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, 26 | DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY 27 | THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT 28 | (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 29 | OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 30 | 31 | You can contact the author at : 32 | - xxHash source repository : https://github.com/Cyan4973/xxHash 33 | */ 34 | 35 | /* Notice extracted from xxHash homepage : 36 | 37 | xxHash is an extremely fast Hash algorithm, running at RAM speed limits. 38 | It also successfully passes all tests from the SMHasher suite. 39 | 40 | Comparison (single thread, Windows Seven 32 bits, using SMHasher on a Core 2 Duo @3GHz) 41 | 42 | Name Speed Q.Score Author 43 | xxHash 5.4 GB/s 10 44 | CrapWow 3.2 GB/s 2 Andrew 45 | MumurHash 3a 2.7 GB/s 10 Austin Appleby 46 | SpookyHash 2.0 GB/s 10 Bob Jenkins 47 | SBox 1.4 GB/s 9 Bret Mulvey 48 | Lookup3 1.2 GB/s 9 Bob Jenkins 49 | SuperFastHash 1.2 GB/s 1 Paul Hsieh 50 | CityHash64 1.05 GB/s 10 Pike & Alakuijala 51 | FNV 0.55 GB/s 5 Fowler, Noll, Vo 52 | CRC32 0.43 GB/s 9 53 | MD5-32 0.33 GB/s 10 Ronald L. Rivest 54 | SHA1-32 0.28 GB/s 10 55 | 56 | Q.Score is a measure of quality of the hash function. 57 | It depends on successfully passing SMHasher test set. 58 | 10 is a perfect score. 59 | 60 | A 64-bits version, named XXH64, is available since r35. 61 | It offers much better speed, but for 64-bits applications only. 62 | Name Speed on 64 bits Speed on 32 bits 63 | XXH64 13.8 GB/s 1.9 GB/s 64 | XXH32 6.8 GB/s 6.0 GB/s 65 | */ 66 | 67 | #pragma once 68 | 69 | #if defined (__cplusplus) 70 | extern "C" { 71 | #endif 72 | 73 | 74 | /***************************** 75 | * Definitions 76 | *****************************/ 77 | #include /* size_t */ 78 | typedef enum { XXH_OK=0, XXH_ERROR } XXH_errorcode; 79 | 80 | 81 | /***************************** 82 | * Namespace Emulation 83 | *****************************/ 84 | /* Motivations : 85 | 86 | If you need to include xxHash into your library, 87 | but wish to avoid xxHash symbols to be present on your library interface 88 | in an effort to avoid potential name collision if another library also includes xxHash, 89 | 90 | you can use XXH_NAMESPACE, which will automatically prefix any symbol from xxHash 91 | with the value of XXH_NAMESPACE (so avoid to keep it NULL, and avoid numeric values). 92 | 93 | Note that no change is required within the calling program : 94 | it can still call xxHash functions using their regular name. 95 | They will be automatically translated by this header. 96 | */ 97 | #ifdef XXH_NAMESPACE 98 | # define XXH_CAT(A,B) A##B 99 | # define XXH_NAME2(A,B) XXH_CAT(A,B) 100 | # define XXH32 XXH_NAME2(XXH_NAMESPACE, XXH32) 101 | # define XXH64 XXH_NAME2(XXH_NAMESPACE, XXH64) 102 | # define XXH32_createState XXH_NAME2(XXH_NAMESPACE, XXH32_createState) 103 | # define XXH64_createState XXH_NAME2(XXH_NAMESPACE, XXH64_createState) 104 | # define XXH32_freeState XXH_NAME2(XXH_NAMESPACE, XXH32_freeState) 105 | # define XXH64_freeState XXH_NAME2(XXH_NAMESPACE, XXH64_freeState) 106 | # define XXH32_reset XXH_NAME2(XXH_NAMESPACE, XXH32_reset) 107 | # define XXH64_reset XXH_NAME2(XXH_NAMESPACE, XXH64_reset) 108 | # define XXH32_update XXH_NAME2(XXH_NAMESPACE, XXH32_update) 109 | # define XXH64_update XXH_NAME2(XXH_NAMESPACE, XXH64_update) 110 | # define XXH32_digest XXH_NAME2(XXH_NAMESPACE, XXH32_digest) 111 | # define XXH64_digest XXH_NAME2(XXH_NAMESPACE, XXH64_digest) 112 | #endif 113 | 114 | 115 | /***************************** 116 | * Simple Hash Functions 117 | *****************************/ 118 | 119 | unsigned int XXH32 (const void* input, size_t length, unsigned seed); 120 | unsigned long long XXH64 (const void* input, size_t length, unsigned long long seed); 121 | 122 | /* 123 | XXH32() : 124 | Calculate the 32-bits hash of sequence "length" bytes stored at memory address "input". 125 | The memory between input & input+length must be valid (allocated and read-accessible). 126 | "seed" can be used to alter the result predictably. 127 | This function successfully passes all SMHasher tests. 128 | Speed on Core 2 Duo @ 3 GHz (single thread, SMHasher benchmark) : 5.4 GB/s 129 | XXH64() : 130 | Calculate the 64-bits hash of sequence of length "len" stored at memory address "input". 131 | Faster on 64-bits systems. Slower on 32-bits systems. 132 | */ 133 | 134 | 135 | 136 | /***************************** 137 | * Advanced Hash Functions 138 | *****************************/ 139 | typedef struct { long long ll[ 6]; } XXH32_state_t; 140 | typedef struct { long long ll[11]; } XXH64_state_t; 141 | 142 | /* 143 | These structures allow static allocation of XXH states. 144 | States must then be initialized using XXHnn_reset() before first use. 145 | 146 | If you prefer dynamic allocation, please refer to functions below. 147 | */ 148 | 149 | XXH32_state_t* XXH32_createState(void); 150 | XXH_errorcode XXH32_freeState(XXH32_state_t* statePtr); 151 | 152 | XXH64_state_t* XXH64_createState(void); 153 | XXH_errorcode XXH64_freeState(XXH64_state_t* statePtr); 154 | 155 | /* 156 | These functions create and release memory for XXH state. 157 | States must then be initialized using XXHnn_reset() before first use. 158 | */ 159 | 160 | 161 | XXH_errorcode XXH32_reset (XXH32_state_t* statePtr, unsigned seed); 162 | XXH_errorcode XXH32_update (XXH32_state_t* statePtr, const void* input, size_t length); 163 | unsigned int XXH32_digest (const XXH32_state_t* statePtr); 164 | 165 | XXH_errorcode XXH64_reset (XXH64_state_t* statePtr, unsigned long long seed); 166 | XXH_errorcode XXH64_update (XXH64_state_t* statePtr, const void* input, size_t length); 167 | unsigned long long XXH64_digest (const XXH64_state_t* statePtr); 168 | 169 | /* 170 | These functions calculate the xxHash of an input provided in multiple smaller packets, 171 | as opposed to an input provided as a single block. 172 | 173 | XXH state space must first be allocated, using either static or dynamic method provided above. 174 | 175 | Start a new hash by initializing state with a seed, using XXHnn_reset(). 176 | 177 | Then, feed the hash state by calling XXHnn_update() as many times as necessary. 178 | Obviously, input must be valid, meaning allocated and read accessible. 179 | The function returns an error code, with 0 meaning OK, and any other value meaning there is an error. 180 | 181 | Finally, you can produce a hash anytime, by using XXHnn_digest(). 182 | This function returns the final nn-bits hash. 183 | You can nonetheless continue feeding the hash state with more input, 184 | and therefore get some new hashes, by calling again XXHnn_digest(). 185 | 186 | When you are done, don't forget to free XXH state space, using typically XXHnn_freeState(). 187 | */ 188 | 189 | 190 | #if defined (__cplusplus) 191 | } 192 | #endif 193 | -------------------------------------------------------------------------------- /tests/test.py: -------------------------------------------------------------------------------- 1 | import lz4f 2 | import lz4tools 3 | import os 4 | import unittest 5 | 6 | 7 | class TestLZ4File(unittest.TestCase): 8 | 9 | def test_1_writeTar(self): 10 | lz4tools.compressTarDefault('src') 11 | self.assertTrue(os.path.exists('src.tar.lz4')) 12 | 13 | def test_2_readTar(self): 14 | testTar = lz4tools.openTar('src.tar.lz4') 15 | count = testTar.getmembers() 16 | testTar.close() 17 | dircount = 1 18 | for root, dirs, files in os.walk('src'): 19 | dircount += len(dirs) 20 | dircount += len(files) 21 | self.assertEqual(dircount, len(count)) 22 | os.remove('src.tar.lz4') 23 | 24 | def test_3_prefs(self): 25 | cCtx = lz4f.createCompContext() 26 | dCtx = lz4f.createDecompContext() 27 | prefs = lz4f.makePrefs(5) 28 | header = lz4f.compressBegin(cCtx, prefs) 29 | frameInfo = lz4f.getFrameInfo(header, dCtx) 30 | self.assertEqual(5, frameInfo.get('blkSize')) 31 | 32 | 33 | if __name__ == '__main__': 34 | unittest.main() 35 | --------------------------------------------------------------------------------