├── CMakeLists.txt ├── README ├── pngwolf.cxx └── sevenzip920.patch /CMakeLists.txt: -------------------------------------------------------------------------------- 1 | cmake_minimum_required(VERSION 2.7) 2 | 3 | include_directories(zlib 7zip/CPP galib) 4 | 5 | add_executable(pngwolf 6 | pngwolf.cxx 7 | galib/ga/GA1DArrayGenome.C 8 | galib/ga/GAAllele.C 9 | galib/ga/GABaseGA.C 10 | galib/ga/gabincvt.C 11 | galib/ga/GAGenome.C 12 | galib/ga/GAIncGA.C 13 | galib/ga/GAParameter.C 14 | galib/ga/GAPopulation.C 15 | galib/ga/garandom.C 16 | galib/ga/gaerror.C 17 | galib/ga/GAScaling.C 18 | galib/ga/GASelector.C 19 | galib/ga/GAStatistics.C 20 | zlib/adler32.c 21 | zlib/compress.c 22 | zlib/crc32.c 23 | zlib/deflate.c 24 | zlib/gzclose.c 25 | zlib/gzlib.c 26 | zlib/gzread.c 27 | zlib/gzwrite.c 28 | zlib/infback.c 29 | zlib/inffast.c 30 | zlib/inflate.c 31 | zlib/inftrees.c 32 | zlib/trees.c 33 | zlib/uncompr.c 34 | zlib/zutil.c 35 | 7zip/CPP/7zip/Common/InBuffer.cpp 36 | 7zip/CPP/7zip/Common/OutBuffer.cpp 37 | 7zip/CPP/7zip/Common/StreamObjects.cpp 38 | 7zip/CPP/7zip/Common/StreamUtils.cpp 39 | 7zip/CPP/7zip/Compress/BitlDecoder.cpp 40 | 7zip/CPP/7zip/Compress/DeflateDecoder.cpp 41 | 7zip/CPP/7zip/Compress/DeflateEncoder.cpp 42 | 7zip/CPP/7zip/Compress/LzOutWindow.cpp 43 | 7zip/CPP/7zip/Compress/ZlibDecoder.cpp 44 | 7zip/CPP/7zip/Compress/ZlibEncoder.cpp 45 | 7zip/C/Alloc.c 46 | 7zip/C/HuffEnc.c 47 | 7zip/C/LzFind.c 48 | 7zip/C/Sort.c 49 | ) 50 | -------------------------------------------------------------------------------- /README: -------------------------------------------------------------------------------- 1 | `pngwolf` is a tool to minimize the size of PNG image files. There are 2 | a number of factors that affect the size of PNG image files, such as 3 | the number of colors in the image and whether the image data is stored 4 | as RGBA data or in the form of references to a color palette. The main 5 | factor is the quality of the Deflate compression used to compress the 6 | image data, which is in turn affected by the quality of the compressor 7 | and how well the data to be compressed is arranged. 8 | 9 | The PNG format supports a number of scanline filters that transform the 10 | image data by relating nearby pixels mathematically. Choosing the right 11 | filters for each scanline can make the image data more compressible. It 12 | is, however, infeasible for non-trivial images to find the best filters 13 | so typical encoders rely on a couple of heuristics to find good filters. 14 | 15 | `pngwolf` employs a genetic algorithm to find better filter combinations 16 | than traditional heuristics. It derives a couple of filter combinations 17 | heuristically, adds a couple of random combinations, and then looks how 18 | well each combination compresses. Two very different combinations may 19 | compress similarily well, for instance, one combination may be very good 20 | for the first couple of scanlines, while the other may be very good for 21 | the last couple of scanlines. So taking the beginning of one combination 22 | and the tail of the other to make a new one may result in a combination 23 | that compresses better then the original two. 24 | 25 | That is, in essence, what `pngwolf` does, over and over again. Further, 26 | the most widely used PNG encoders use the zlib library for compression. 27 | The zlib library favours speed over compression ratio in some cases, so 28 | whatever filters are selected to aid compression, the result with zlib 29 | may not be the smallest possible. The 7-Zip library by Igor Pavlov has 30 | a Deflate encoder that favours size over speed at certain settings. So, 31 | `pngwolf` attempts to make use of both: a fast zlib setting is used to 32 | estimate how well some filter combination aids compression, and when it 33 | gets bored, it uses 7-Zip to generate the final result. 34 | 35 | Doing this `pngwolf` is able to compress some images better than other 36 | optimizers (like `OptiPNG`, `AdvanceCOMP`, `pngcrush`, and `pngout`), 37 | either because it finds better filter combinations then they do, or be- 38 | cause it uses 7-Zip's Deflate implementation (`AdvanceCOMP` uses that 39 | aswell, although an older version which sometimes performs better and 40 | sometimes worse, for reasons yet to be studied). It does not attempt to 41 | make other optimizations, like converting indexed images to RGB format. 42 | 43 | None of the tools mentioned, including `pngwolf` follow any kind of ho- 44 | listic approach to PNG optimization, so to get the best results they 45 | need to be used in combination (and sometimes applying them repeatedly 46 | or in different orders provides the best results). As far as I can tell 47 | most other tools do not try to preserve the filter combination in the 48 | original image, so `pngwolf` should usually be used last or second-to- 49 | last in the optimization process. 50 | 51 | For images that are already optimized using all the other tools, there 52 | is about `1%` further reduction to be expected from `pngwolf` for suit- 53 | able images. Still, it should be rare to find images on the Web that 54 | `pngwolf` cannot compress a little bit further. 55 | 56 | The tool suffers from the lack of a Deflate encoder that makes it easy 57 | store the results of data analysis (where are duplicate substrings in 58 | the data) and combine them (if you recall the earlier example where it 59 | takes the head of one combination and the tail of another, an encoder 60 | would not have to analyze all of the two parts again, only where they 61 | overlap). So it can often take a long time (as in minutes) to find the 62 | best results. 63 | 64 | Regardless of the performance deficiency `pngwolf` is well-suited as a 65 | research tool to come up with better heuristics for filter selection, 66 | or to extend the genetic algorithm approach to other aspects of PNG op- 67 | timization (the main thing being considered is re-arranging the entries 68 | in color palettes so the image data compresses better). The tool logs 69 | extensive information in a YAML-based machine-readable format while it 70 | attempts to optimize images which should aid in that. 71 | 72 | It also addresses two (other) user-interface issues I had in using the 73 | other tools, namely it allows you to make it stop trying to find better 74 | optimizations at well-specified points (such as the total time used), 75 | and if you start an optimization run but grow impatient and abort the 76 | program, results should not get lost, but should be stored anyway. 77 | 78 | To compile `pngwolf` you need three additional libraries: 79 | 80 | * GAlib http://lancet.mit.edu/ga/dist/ 81 | * 7-Zip http://www.7-zip.org/download.html ("7-Zip Source code") 82 | * zlib http://zlib.net/ 83 | 84 | Put these into `galib`, `7zip`, and `zlib` sub-directories into the 85 | directory where pngwolf.cxx is located, and then either use the CMake 86 | utility (http://www.cmake.org/) on the `CMakeLists.txt`, or simply 87 | specify all the files specified in `CMakeLists.txt` as input to your 88 | compiler. The latter would look like: 89 | 90 | % gcc -I7zip/CPP -Igalib pngwolf.cxx galib/ga/GA1DArrayGenome.C 91 | galib/ga/GAAllele.C ... -lstdc++ -o pngwolf 92 | 93 | If you are using 7-Zip 9.20 there are two bugs in 7-Zip that prevent 94 | gcc building https://sourceforge.net/support/tracker.php?aid=3200655 95 | it. To address that, apply the patch `sevenzip920.patch` like so: 96 | 97 | % patch -p 0 < sevenzip920.patch 98 | 99 | I've done this successfully with Visual C++ 2010 and Cygwin gcc 4.3.4. 100 | -------------------------------------------------------------------------------- /pngwolf.cxx: -------------------------------------------------------------------------------- 1 | //////////////////////////////////////////////////////////////////// 2 | // 3 | // pngwolf - Optimize PNG file size by genetically finding filters 4 | // 5 | // Copyright (C) 2008-2011 Bjoern Hoehrmann 6 | // 7 | // This program is free software; you can redistribute it and/or 8 | // modify it under the terms of the GNU General Public License 9 | // as published by the Free Software Foundation; either version 2 10 | // of the License, or (at your option) any later version. 11 | // 12 | // This program is distributed in the hope that it will be useful, 13 | // but WITHOUT ANY WARRANTY; without even the implied warranty of 14 | // MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 15 | // GNU General Public License for more details. 16 | // 17 | // $Id$ 18 | // 19 | //////////////////////////////////////////////////////////////////// 20 | 21 | #ifdef _MSC_VER 22 | #include 23 | #pragma comment(lib, "ws2_32.lib") 24 | #else 25 | #include 26 | #endif 27 | 28 | #include "Common/MyWindows.h" 29 | #include "Common/MyInitGuid.h" 30 | #include "7zip/IStream.h" 31 | #include "7zip/Compress/ZlibEncoder.h" 32 | #include "7zip/Common/FileStreams.h" 33 | #include "7zip/Common/InBuffer.h" 34 | #include "7zip/Common/StreamObjects.h" 35 | 36 | #include 37 | #include 38 | #include 39 | #include 40 | #include 41 | #include 42 | #include 43 | #include 44 | #include 45 | #include 46 | #include 47 | #include 48 | #include 49 | #include 50 | #include 51 | #include 52 | #include 53 | #include 54 | 55 | #ifndef SIZE_MAX 56 | #define SIZE_MAX ((size_t)-1) 57 | #endif 58 | 59 | #ifdef _MSC_VER 60 | #pragma warning(push, 4) 61 | #pragma warning(disable: 4996) 62 | #pragma warning(disable: 4100) 63 | #endif 64 | 65 | //////////////////////////////////////////////////////////////////// 66 | // Miscellaneous structures and types 67 | //////////////////////////////////////////////////////////////////// 68 | typedef enum { 69 | None = 0, 70 | Sub = 1, 71 | Up = 2, 72 | Avg = 3, 73 | Paeth = 4 74 | } PngFilter; 75 | 76 | typedef struct PngChunk PngChunk; 77 | struct PngChunk { 78 | uint32_t size; 79 | uint32_t type; 80 | uint32_t crc32; 81 | std::vector data; 82 | }; 83 | 84 | typedef struct IhdrChunk IhdrChunk; 85 | struct IhdrChunk { 86 | uint32_t width; 87 | uint32_t height; 88 | uint32_t depth; 89 | uint32_t color; 90 | uint32_t comp; 91 | uint32_t filter; 92 | uint32_t interlace; 93 | }; 94 | 95 | typedef GA1DArrayAlleleGenome PngFilterGenome; 96 | 97 | class Deflater { 98 | public: 99 | virtual std::vector deflate(const std::vector&) = 0; 100 | }; 101 | 102 | class PngWolf { 103 | public: 104 | // IHDR data 105 | IhdrChunk ihdr; 106 | 107 | // Derived IHDR data 108 | size_t scanline_width; 109 | size_t scanline_delta; 110 | 111 | // The input image as list of chunks 112 | std::list chunks; 113 | 114 | // Urfilter 115 | std::vector original_filters; 116 | 117 | // Filters; TODO: who owns them? 118 | std::map genomes; 119 | 120 | std::vector best_genomes; 121 | 122 | // ... 123 | GAPopulation initial_pop; 124 | 125 | // Command line options 126 | unsigned max_stagnate_time; 127 | unsigned max_time; 128 | unsigned max_evaluations; 129 | unsigned max_deflate; 130 | size_t population_size; 131 | const char* in_path; 132 | const char* out_path; 133 | bool verbose_analysis; 134 | bool verbose_summary; 135 | bool verbose_genomes; 136 | bool exclude_singles; 137 | bool exclude_original; 138 | bool exclude_heuristic; 139 | bool exclude_experiment1; 140 | bool exclude_experiment2; 141 | bool exclude_experiment3; 142 | bool exclude_experiment4; 143 | bool normalize_alpha; 144 | bool even_if_bigger; 145 | bool auto_mpass; 146 | bool bigger_is_better; 147 | int zlib_level; 148 | int zlib_windowBits; 149 | int zlib_memLevel; 150 | int zlib_strategy; 151 | int szip_pass; 152 | int szip_fast; 153 | int szip_cycl; 154 | 155 | // 156 | Deflater* deflate_fast; 157 | Deflater* deflate_good; 158 | 159 | // User input 160 | bool should_abort; 161 | 162 | // Keeping track of time 163 | time_t program_begun_at; 164 | time_t search_begun_at; 165 | time_t last_improvement_at; 166 | time_t last_step_at; 167 | time_t done_deflating_at; 168 | 169 | // IDAT 170 | std::vector original_inflated; 171 | std::vector original_deflated; 172 | std::vector original_unfiltered; 173 | 174 | // 175 | std::map > flt_singles; 176 | 177 | // 178 | std::map invis_colors; 179 | 180 | // 181 | unsigned nth_generation; 182 | unsigned genomes_evaluated; 183 | 184 | // 185 | std::vector best_inflated; 186 | std::vector best_deflated; 187 | 188 | // The genetic algorithm 189 | GAGeneticAlgorithm* ga; 190 | 191 | // Logging 192 | void log_analysis(); 193 | void log_critter(PngFilterGenome* curr_best); 194 | void log_summary(); 195 | void log_genome(PngFilterGenome* ge); 196 | 197 | // Various 198 | bool read_file(); 199 | bool save_file(); 200 | bool save_best_idat(const char* path); 201 | bool save_original_idat(const char* path); 202 | bool save_idat(const char* path, std::vector& deflated, std::vector& inflated); 203 | void init_filters(); 204 | void run(); 205 | void recompress(); 206 | std::vector refilter(const PngFilterGenome& ge); 207 | 208 | // Constructor 209 | PngWolf() : 210 | should_abort(false), 211 | nth_generation(0), 212 | genomes_evaluated(0), 213 | done_deflating_at(0), 214 | deflate_fast(NULL), 215 | deflate_good(NULL) 216 | {} 217 | 218 | ~PngWolf() { 219 | // TODO: This should probably delete the genomes, both 220 | // the ge_ ones and the ones in the best_genomes vector 221 | } 222 | 223 | }; 224 | 225 | struct DeflateZlib : public Deflater { 226 | public: 227 | std::vector deflate(const std::vector& inflated) { 228 | 229 | if (deflateReset(&strm) != Z_OK) { 230 | // TODO: ... 231 | abort(); 232 | } 233 | 234 | strm.next_in = (Bytef*)&inflated[0]; 235 | strm.avail_in = inflated.size(); 236 | 237 | size_t max = deflateBound(&strm, inflated.size()); 238 | std::vector new_deflated(max); 239 | 240 | strm.next_out = (Bytef*)&new_deflated[0]; 241 | strm.avail_out = max; 242 | 243 | // TODO: aborting here probably leaks memory 244 | if (::deflate(&strm, Z_FINISH) != Z_STREAM_END) 245 | abort(); 246 | 247 | new_deflated.resize(max - strm.avail_out); 248 | 249 | return new_deflated; 250 | } 251 | 252 | DeflateZlib(int level, int windowBits, int memLevel, int strategy) { 253 | strm.zalloc = Z_NULL; 254 | strm.zfree = Z_NULL; 255 | strm.opaque = Z_NULL; 256 | 257 | if (deflateInit2(&strm, level, Z_DEFLATED, 258 | windowBits, memLevel, strategy) != Z_OK) { 259 | // TODO: 260 | abort(); 261 | } 262 | } 263 | 264 | z_stream strm; 265 | }; 266 | 267 | struct Deflate7zip : public Deflater { 268 | public: 269 | std::vector deflate(const std::vector& inflated) { 270 | 271 | NCompress::NZlib::CEncoder c; 272 | PROPID algoProp = NCoderPropID::kAlgorithm; 273 | PROPID passProp = NCoderPropID::kNumPasses; 274 | PROPID fastProp = NCoderPropID::kNumFastBytes; 275 | PROPID cyclProp = NCoderPropID::kMatchFinderCycles; 276 | 277 | PROPVARIANT v; 278 | v.vt = VT_UI4; 279 | 280 | // TODO: figure out what to do with errors here 281 | 282 | c.Create(); 283 | 284 | NCompress::NDeflate::NEncoder::CCOMCoder* d = 285 | c.DeflateEncoderSpec; 286 | 287 | v.ulVal = szip_algo; 288 | if (d->SetCoderProperties(&algoProp, &v, 1) != S_OK) { 289 | } 290 | 291 | v.ulVal = szip_pass; 292 | if (d->SetCoderProperties(&passProp, &v, 1) != S_OK) { 293 | } 294 | 295 | v.ulVal = szip_fast; 296 | if (d->SetCoderProperties(&fastProp, &v, 1) != S_OK) { 297 | } 298 | 299 | v.ulVal = szip_cycl; 300 | if (d->SetCoderProperties(&cyclProp, &v, 1) != S_OK) { 301 | } 302 | 303 | CBufInStream* in_buf = new CBufInStream; 304 | 305 | // TODO: find a way to use a a fixed buffer since we know 306 | // the maximum size for it and don't use more than one. It 307 | // might also be a good idea to keep the other objects for 308 | // all the passes through this to avoid re-allocations and 309 | // the possible failures that might go along with them. 310 | CDynBufSeqOutStream* out_buf = new CDynBufSeqOutStream; 311 | in_buf->Init((const Byte*)&inflated[0], inflated.size()); 312 | CMyComPtr in(in_buf); 313 | CMyComPtr out(out_buf); 314 | 315 | if (c.Code(in, out, NULL, NULL, NULL) != S_OK) { 316 | } 317 | 318 | std::vector deflated(out_buf->GetSize()); 319 | memcpy(&deflated[0], out_buf->GetBuffer(), deflated.size()); 320 | 321 | return deflated; 322 | } 323 | 324 | Deflate7zip(int pass, int fast, int cycl) : 325 | szip_pass(pass), 326 | szip_fast(fast), 327 | szip_cycl(cycl), 328 | szip_algo(1) { 329 | } 330 | 331 | int szip_pass; 332 | int szip_fast; 333 | int szip_cycl; 334 | int szip_algo; 335 | }; 336 | 337 | static const char PNG_MAGIC[] = "\x89\x50\x4E\x47\x0D\x0A\x1A\x0A"; 338 | static const uint32_t IDAT_TYPE = 0x49444154; 339 | static const uint32_t IHDR_TYPE = 0x49484452; 340 | static const uint32_t IEND_TYPE = 0x49454e44; 341 | 342 | //////////////////////////////////////////////////////////////////// 343 | // Global PngWolf instance 344 | //////////////////////////////////////////////////////////////////// 345 | static PngWolf wolf; 346 | 347 | //////////////////////////////////////////////////////////////////// 348 | // PNG Scanline Filters 349 | //////////////////////////////////////////////////////////////////// 350 | 351 | unsigned char paeth_predictor(unsigned char a, unsigned char b, unsigned char c) { 352 | unsigned int p = a + b - c; 353 | unsigned int pa = abs((int)(p - a)); 354 | unsigned int pb = abs((int)(p - b)); 355 | unsigned int pc = abs((int)(p - c)); 356 | 357 | if (pa <= pb && pa <= pc) 358 | return a; 359 | 360 | if (pb <= pc) 361 | return b; 362 | 363 | return c; 364 | } 365 | 366 | void filter_row_none(unsigned char* src, unsigned char* dst, size_t row, size_t pwidth, size_t bytes) { 367 | size_t xix = row * bytes + 1; 368 | memcpy(dst + xix, src + xix, bytes - 1); 369 | } 370 | 371 | void filter_row_sub(unsigned char* src, unsigned char* dst, size_t row, size_t pwidth, size_t bytes) { 372 | size_t xix = row * bytes + 1; 373 | size_t aix = xix; 374 | size_t end = (row+1)*bytes; 375 | 376 | for (; xix < row * bytes + 1 + pwidth; ++xix) 377 | dst[xix] = src[xix]; 378 | 379 | for (; xix < end; ++xix, ++aix) 380 | dst[xix] = src[xix] - src[aix]; 381 | } 382 | 383 | void filter_row_up(unsigned char* src, unsigned char* dst, size_t row, size_t pwidth, size_t bytes) { 384 | size_t xix = row * bytes + 1; 385 | size_t bix = xix - bytes; 386 | size_t end = (row+1)*bytes; 387 | 388 | if (row == 0) { 389 | memcpy(dst + 1, src + 1, bytes - 1); 390 | return; 391 | } 392 | 393 | for (; xix < end; ++xix, ++bix) 394 | dst[xix] = src[xix] - src[bix]; 395 | } 396 | 397 | void filter_row_avg(unsigned char* src, unsigned char* dst, size_t row, size_t pwidth, size_t bytes) { 398 | size_t xix = row * bytes + 1; 399 | size_t bix = xix - bytes; 400 | size_t aix = xix; 401 | size_t end = (row+1)*bytes; 402 | 403 | if (row == 0) { 404 | for (; xix < row * bytes + 1 + pwidth; ++xix) 405 | dst[xix] = src[xix]; 406 | 407 | for (; xix < end; ++xix, ++aix) 408 | dst[xix] = src[xix] - (src[aix] >> 1); 409 | 410 | return; 411 | } 412 | 413 | for (; xix < row * bytes + 1 + pwidth; ++xix, ++bix) 414 | dst[xix] = src[xix] - (src[bix] >> 1); 415 | 416 | for (; xix < end; ++xix, ++aix, ++bix) 417 | dst[xix] = src[xix] - ((src[aix] + src[bix]) >> 1); 418 | } 419 | 420 | void filter_row_paeth(unsigned char* src, unsigned char* dst, size_t row, size_t pwidth, size_t bytes) { 421 | size_t xix = row * bytes + 1; 422 | size_t aix = xix; 423 | size_t bix = xix - bytes; 424 | size_t cix = xix - bytes; 425 | size_t end = (row+1)*bytes; 426 | 427 | if (row == 0) { 428 | for (; xix < row * bytes + 1 + pwidth; ++xix) 429 | dst[xix] = src[xix]; 430 | 431 | for (; xix < end; ++xix, ++aix) 432 | dst[xix] = src[xix] - paeth_predictor(src[aix], 0 , 0); 433 | 434 | return; 435 | } 436 | 437 | // TODO: this should not change pwidth 438 | for (; pwidth > 0; --pwidth, ++xix, ++bix) 439 | dst[xix] = src[xix] - paeth_predictor(0, src[bix] , 0); 440 | 441 | for (; xix < end; ++xix, ++aix, ++bix, ++cix) 442 | dst[xix] = src[xix] - paeth_predictor(src[aix], src[bix], src[cix]); 443 | } 444 | 445 | void unfilter_row_sub(unsigned char* idat, size_t row, size_t pwidth, size_t bytes) { 446 | size_t xix = row * bytes + 1; 447 | size_t aix = xix; 448 | size_t end = (row+1)*bytes; 449 | 450 | xix += pwidth; 451 | while (xix < end) 452 | idat[xix++] += idat[aix++]; 453 | } 454 | 455 | void unfilter_row_up(unsigned char* idat, size_t row, size_t pwidth, size_t bytes) { 456 | size_t xix = row * bytes + 1; 457 | size_t bix = xix - bytes; 458 | size_t end = (row+1)*bytes; 459 | 460 | if (row == 0) 461 | return; 462 | while (xix < end) 463 | idat[xix++] += idat[bix++]; 464 | } 465 | 466 | void unfilter_row_avg(unsigned char* idat, size_t row, size_t pwidth, size_t bytes) { 467 | size_t xix = row * bytes + 1; 468 | size_t bix = xix - bytes; 469 | size_t end = (row+1)*bytes; 470 | size_t aix; 471 | 472 | if (row == 0) { 473 | size_t aix = xix; 474 | xix += pwidth; 475 | while (xix < end) 476 | idat[xix++] += idat[aix++] >> 1; 477 | return; 478 | } 479 | 480 | aix = xix; 481 | for (; pwidth > 0; --pwidth) 482 | idat[xix++] += idat[bix++] >> 1; 483 | 484 | while (xix < end) 485 | idat[xix++] += (idat[aix++] + idat[bix++]) >> 1; 486 | } 487 | 488 | void unfilter_row_paeth(unsigned char* idat, size_t row, size_t pwidth, size_t bytes) { 489 | size_t xix = row * bytes + 1; 490 | size_t bix = xix - bytes; 491 | size_t aix, cix; 492 | size_t end = (row+1)*bytes; 493 | 494 | if (row == 0) { 495 | size_t aix = xix; 496 | xix += pwidth; 497 | while (xix < end) 498 | idat[xix++] += paeth_predictor(idat[aix++], 0 , 0); 499 | return; 500 | } 501 | 502 | aix = xix; 503 | cix = aix - bytes; 504 | 505 | for (; pwidth > 0; --pwidth) 506 | idat[xix++] += paeth_predictor(0, idat[bix++] , 0); 507 | 508 | while (xix < end) 509 | idat[xix++] += paeth_predictor(idat[aix++], idat[bix++] , idat[cix++]); 510 | } 511 | 512 | void unfilter_idat(unsigned char* idat, size_t rows, size_t pwidth, size_t bytes) { 513 | size_t row; 514 | for (row = 0; row < rows; ++row) { 515 | switch(idat[row*bytes]) { 516 | case 0: 517 | break; 518 | case 1: 519 | unfilter_row_sub(idat, row, pwidth, bytes); 520 | break; 521 | case 2: 522 | unfilter_row_up(idat, row, pwidth, bytes); 523 | break; 524 | case 3: 525 | unfilter_row_avg(idat, row, pwidth, bytes); 526 | break; 527 | case 4: 528 | unfilter_row_paeth(idat, row, pwidth, bytes); 529 | break; 530 | default: 531 | assert(!"bad filter type"); 532 | } 533 | idat[row*bytes] = 0; 534 | } 535 | } 536 | 537 | void filter_idat(unsigned char* src, unsigned char* dst, const PngFilterGenome& filter, size_t pwidth, size_t bytes) { 538 | for (int row = 0; row < filter.size(); ++row) { 539 | switch(filter.gene(row)) { 540 | case 0: 541 | filter_row_none(src, dst, row, pwidth, bytes); 542 | break; 543 | case 1: 544 | filter_row_sub(src, dst, row, pwidth, bytes); 545 | break; 546 | case 2: 547 | filter_row_up(src, dst, row, pwidth, bytes); 548 | break; 549 | case 3: 550 | filter_row_avg(src, dst, row, pwidth, bytes); 551 | break; 552 | case 4: 553 | filter_row_paeth(src, dst, row, pwidth, bytes); 554 | break; 555 | default: 556 | assert(!"bad filter type"); 557 | } 558 | // TODO: check that src uses the `none` filter 559 | dst[row*bytes] = (unsigned char)filter.gene(row); 560 | } 561 | } 562 | 563 | //////////////////////////////////////////////////////////////////// 564 | // Signal handlers 565 | //////////////////////////////////////////////////////////////////// 566 | #ifdef _MSC_VER 567 | BOOL WINAPI console_event_handler(DWORD Event) { 568 | switch (Event) { 569 | case CTRL_C_EVENT: 570 | case CTRL_BREAK_EVENT: 571 | case CTRL_CLOSE_EVENT: 572 | case CTRL_LOGOFF_EVENT: 573 | case CTRL_SHUTDOWN_EVENT: 574 | wolf.should_abort = true; 575 | return TRUE; 576 | } 577 | return FALSE; 578 | } 579 | #else 580 | void sigint_handler(int signum) { 581 | wolf.should_abort = true; 582 | } 583 | #endif 584 | 585 | //////////////////////////////////////////////////////////////////// 586 | // Genome Helpers 587 | //////////////////////////////////////////////////////////////////// 588 | template <> PngFilter 589 | GAAlleleSet::allele() const { 590 | return (PngFilter)GARandomInt(lower(), upper()); 591 | } 592 | 593 | std::vector inflate_zlib(std::vector& deflated) { 594 | z_stream strm; 595 | strm.zalloc = Z_NULL; 596 | strm.zfree = Z_NULL; 597 | strm.opaque = Z_NULL; 598 | strm.next_in = (Bytef*)&deflated[0]; 599 | strm.avail_in = deflated.size(); 600 | std::vector inflated; 601 | std::vector temp(65535); 602 | 603 | if (inflateInit(&strm) != Z_OK) 604 | goto error; 605 | 606 | do { 607 | strm.avail_out = temp.size(); 608 | strm.next_out = (Bytef*)&temp[0]; 609 | int ret = inflate(&strm, Z_NO_FLUSH); 610 | 611 | // TODO: going to `error` here probably leaks some memory 612 | // but it would be freed when exiting the process, so this 613 | // is mostly important when turning this into a library. 614 | if (ret != Z_STREAM_END && ret != Z_OK) 615 | goto error; 616 | 617 | size_t have = temp.size() - strm.avail_out; 618 | inflated.insert(inflated.end(), 619 | temp.begin(), temp.begin() + have); 620 | 621 | } while (strm.avail_out == 0); 622 | 623 | if (inflateEnd(&strm) != Z_OK) 624 | goto error; 625 | 626 | return inflated; 627 | 628 | error: 629 | // TODO: ... 630 | abort(); 631 | return inflated; 632 | } 633 | 634 | std::vector PngWolf::refilter(const PngFilterGenome& ge) { 635 | std::vector refiltered(original_unfiltered.size()); 636 | 637 | filter_idat((unsigned char*)&original_unfiltered[0], 638 | (unsigned char*)&refiltered[0], ge, 639 | scanline_delta, scanline_width); 640 | 641 | return refiltered; 642 | } 643 | 644 | float Evaluator(GAGenome& genome) { 645 | PngFilterGenome& ge = 646 | (PngFilterGenome&)genome; 647 | 648 | // TODO: wolf should be user data, not a global 649 | if (wolf.should_abort) 650 | return FLT_MAX; 651 | 652 | wolf.genomes_evaluated++; 653 | 654 | if (wolf.flt_singles.begin() == wolf.flt_singles.end()) { 655 | // TODO: ... 656 | abort(); 657 | } 658 | 659 | // TODO: it would be better to do this incrementally. 660 | 661 | std::vector filtered(wolf.original_unfiltered.size()); 662 | 663 | for (int row = 0; row < ge.size(); ++row) { 664 | size_t pos = wolf.scanline_width * row; 665 | memcpy(&filtered[pos], 666 | &wolf.flt_singles[ge.gene(row)][pos], wolf.scanline_width); 667 | } 668 | 669 | std::vector deflated = wolf.deflate_fast->deflate(filtered); 670 | 671 | return float(deflated.size()); 672 | } 673 | 674 | //////////////////////////////////////////////////////////////////// 675 | // Helper 676 | //////////////////////////////////////////////////////////////////// 677 | unsigned sum_abs(unsigned c1, unsigned char c2) { 678 | return c1 + (c2 < 128 ? c2 : 256 - c2); 679 | } 680 | 681 | //////////////////////////////////////////////////////////////////// 682 | // Logging 683 | //////////////////////////////////////////////////////////////////// 684 | void PngWolf::log_genome(PngFilterGenome* ge) { 685 | for (int gix = 0; gix < ge->size(); ++gix) { 686 | if (gix % 72 == 0) 687 | fprintf(stdout, "\n "); 688 | fprintf(stdout, "%1d", ge->gene(gix)); 689 | } 690 | fprintf(stdout, "\n"); 691 | } 692 | 693 | void PngWolf::log_summary() { 694 | 695 | int diff = original_deflated.size() - best_deflated.size(); 696 | 697 | if (verbose_summary) { 698 | fprintf(stdout, "best filter sequence found:"); 699 | log_genome(best_genomes.back()); 700 | fprintf(stdout, "" 701 | "best zlib deflated idat size: %0.0f\n" 702 | "total time spent optimizing: %0.0f\n" 703 | "number of genomes evaluated: %u\n" 704 | "size of 7zip deflated data: %u\n" 705 | "size difference to original: %d\n", 706 | best_genomes.back()->score(), 707 | difftime(time(NULL), program_begun_at), 708 | genomes_evaluated, 709 | best_deflated.size(), 710 | -diff); 711 | } 712 | 713 | if (diff >= 0) 714 | fprintf(stdout, "# %u bytes smaller\n", diff); 715 | else 716 | fprintf(stdout, "# %u bytes bigger\n", abs(diff)); 717 | 718 | fflush(stdout); 719 | } 720 | 721 | void PngWolf::log_analysis() { 722 | 723 | fprintf(stdout, "---\n" 724 | "# %u x %u pixels at depth %u (mode %u) with IDAT %u bytes (%u deflated)\n", 725 | ihdr.width, ihdr.height, ihdr.depth, ihdr.color, 726 | original_inflated.size(), original_deflated.size()); 727 | 728 | if (!verbose_analysis) 729 | return; 730 | 731 | fprintf(stdout, "" 732 | "image file path: %s\n" 733 | "width in pixels: %u\n" 734 | "height in pixels: %u\n" 735 | "color mode: %u\n" 736 | "color bit depth: %u\n" 737 | "interlaced: %u\n" 738 | "scanline width: %u\n" 739 | "scanline delta: %u\n" 740 | "inflated idat size: %u\n" 741 | "deflated idat size: %u\n" 742 | "chunks present: ", 743 | this->in_path, 744 | this->ihdr.width, 745 | this->ihdr.height, 746 | this->ihdr.color, 747 | this->ihdr.depth, 748 | this->ihdr.interlace, 749 | this->scanline_width, 750 | this->scanline_delta, 751 | this->original_inflated.size(), 752 | this->original_deflated.size()); 753 | 754 | std::list::iterator c_it; 755 | 756 | for (c_it = chunks.begin(); c_it != chunks.end(); ++c_it) { 757 | // TODO: check that this is broken on bad endianess systems 758 | // Also, since no validation is performed for the types, it 759 | // is possible to break the YAML output with bad files, but 760 | // that does not seem all that important at the moment. 761 | fprintf(stdout, "%c", (c_it->type >> 24)); 762 | fprintf(stdout, "%c", (c_it->type >> 16)); 763 | fprintf(stdout, "%c", (c_it->type >> 8)); 764 | fprintf(stdout, "%c", (c_it->type >> 0)); 765 | fprintf(stdout, " "); 766 | } 767 | 768 | if (ihdr.color == 6 && ihdr.depth == 8) { 769 | fprintf(stdout, "\ninvisible colors:\n"); 770 | std::map::iterator it; 771 | uint32_t total = 0; 772 | 773 | // TODO: htonl is probably not right here 774 | for (it = invis_colors.begin(); it != invis_colors.end(); ++it) { 775 | fprintf(stdout, " - %08X # %u times\n", htonl(it->first), it->second); 776 | total += it->second; 777 | } 778 | 779 | bool skip = invis_colors.size() == 1 780 | && invis_colors.begin()->first == 0x00000000; 781 | 782 | fprintf(stdout, " # %u pixels (%0.2f%%) are fully transparent\n", 783 | total, (double)total / ((double)ihdr.width * (double)ihdr.height)); 784 | 785 | if (invis_colors.size() > 0 && !skip) 786 | fprintf(stdout, " # --normalize-alpha changes them into transparent black\n"); 787 | } else { 788 | fprintf(stdout, "\n"); 789 | } 790 | 791 | fprintf(stdout, "" 792 | "zlib deflated idat sizes:\n" 793 | " original filter: %0.0f\n" 794 | " none: %0.0f\n" 795 | " sub: %0.0f\n" 796 | " up: %0.0f\n" 797 | " avg: %0.0f\n" 798 | " paeth: %0.0f\n" 799 | " deflate scanline: %0.0f\n" 800 | " distinct bytes: %0.0f\n" 801 | " distinct bigrams: %0.0f\n" 802 | " incremental: %0.0f\n" 803 | " basic heuristic: %0.0f\n", 804 | this->genomes["original"]->score(), 805 | this->genomes["all set to none"]->score(), 806 | this->genomes["all set to sub"]->score(), 807 | this->genomes["all set to up"]->score(), 808 | this->genomes["all set to avg"]->score(), 809 | this->genomes["all set to paeth"]->score(), 810 | this->genomes["deflate scanline"]->score(), 811 | this->genomes["distinct bytes"]->score(), 812 | this->genomes["distinct bigrams"]->score(), 813 | this->genomes["incremental"]->score(), 814 | this->genomes["heuristic"]->score()); 815 | 816 | fprintf(stdout, "original filters:"); 817 | log_genome(this->genomes["original"]); 818 | fprintf(stdout, "basic heuristic filters:"); 819 | log_genome(this->genomes["heuristic"]); 820 | fprintf(stdout, "deflate scanline filters:"); 821 | log_genome(this->genomes["deflate scanline"]); 822 | fprintf(stdout, "distinct bytes filters:"); 823 | log_genome(this->genomes["distinct bytes"]); 824 | fprintf(stdout, "distinct bigrams filters:"); 825 | log_genome(this->genomes["distinct bigrams"]); 826 | fprintf(stdout, "incremental filters:"); 827 | log_genome(this->genomes["incremental"]); 828 | 829 | fflush(stdout); 830 | } 831 | 832 | void PngWolf::log_critter(PngFilterGenome* curr_best) { 833 | PngFilterGenome* prev_best = best_genomes.back(); 834 | 835 | if (!this->verbose_genomes) { 836 | fprintf(stdout, "" 837 | "- zlib deflated idat size: %7u # %+5d bytes %+4.0f seconds\n", 838 | unsigned(curr_best->score()), 839 | signed(curr_best->score() - initial_pop.best().score()), 840 | difftime(time(NULL), program_begun_at)); 841 | return; 842 | } 843 | 844 | fprintf(stdout, "" 845 | " ##########################################################################\n" 846 | "- zlib deflated idat size: %7u # %+5d bytes %+4.0f seconds since previous\n" 847 | " ##########################################################################\n" 848 | " zlib bytes since previous improvement: %+d\n" 849 | " zlib bytes since first generation: %+d\n" 850 | " seconds since program launch: %+0.0f\n" 851 | " seconds since previous improvement: %+0.0f\n" 852 | " current generation is the nth: %u\n" 853 | " number of genomes evaluated: %u\n" 854 | " best filters so far:", 855 | unsigned(curr_best->score()), 856 | signed(curr_best->score() - prev_best->score()), 857 | difftime(time(NULL), last_improvement_at), 858 | signed(curr_best->score() - prev_best->score()), 859 | signed(curr_best->score() - best_genomes.front()->score()), 860 | difftime(time(NULL), program_begun_at), 861 | difftime(time(NULL), last_improvement_at), 862 | nth_generation, 863 | genomes_evaluated); 864 | 865 | log_genome(curr_best); 866 | fflush(stdout); 867 | }; 868 | 869 | void PngWolf::init_filters() { 870 | 871 | GAAlleleSet allele(None, Paeth); 872 | PngFilterGenome ge(ihdr.height, allele, Evaluator); 873 | 874 | // Copy the Urcritter to all the critters we want to hold 875 | // on to for anlysis and for the initial population. 876 | 877 | // TODO: Can clone fail? What do we do then? 878 | 879 | genomes["all set to avg"] = (PngFilterGenome*)ge.clone(); 880 | genomes["all set to none"] = (PngFilterGenome*)ge.clone(); 881 | genomes["all set to sub"] = (PngFilterGenome*)ge.clone(); 882 | genomes["all set to up"] = (PngFilterGenome*)ge.clone(); 883 | genomes["all set to paeth"] = (PngFilterGenome*)ge.clone(); 884 | genomes["original"] = (PngFilterGenome*)ge.clone(); 885 | genomes["heuristic"] = (PngFilterGenome*)ge.clone(); 886 | genomes["deflate scanline"] = (PngFilterGenome*)ge.clone(); 887 | genomes["distinct bytes"] = (PngFilterGenome*)ge.clone(); 888 | genomes["distinct bigrams"] = (PngFilterGenome*)ge.clone(); 889 | genomes["incremental"] = (PngFilterGenome*)ge.clone(); 890 | 891 | for (int i = 0; i < ge.size(); ++i) { 892 | genomes["original"]->gene(i, original_filters[i]); 893 | genomes["all set to avg"]->gene(i, Avg); 894 | genomes["all set to sub"]->gene(i, Sub); 895 | genomes["all set to none"]->gene(i, None); 896 | genomes["all set to paeth"]->gene(i, Paeth); 897 | genomes["all set to up"]->gene(i, Up); 898 | } 899 | 900 | flt_singles[None] = refilter(*genomes["all set to none"]); 901 | flt_singles[Sub] = refilter(*genomes["all set to sub"]); 902 | flt_singles[Up] = refilter(*genomes["all set to up"]); 903 | flt_singles[Avg] = refilter(*genomes["all set to avg"]); 904 | flt_singles[Paeth] = refilter(*genomes["all set to paeth"]); 905 | 906 | typedef std::map< PngFilter, std::vector >::iterator flt_iter; 907 | 908 | // TODO: for bigger_is_better it might make sense to have a 909 | // function for the comparisons and set that so heuristics 910 | // also work towards making the selection worse. 911 | 912 | for (int row = 0; row < ge.size(); ++row) { 913 | size_t best_sum = SIZE_MAX; 914 | PngFilter best_flt = None; 915 | 916 | // "The following simple heuristic has performed well in 917 | // early tests: compute the output scanline using all five 918 | // filters, and select the filter that gives the smallest 919 | // sum of absolute values of outputs. (Consider the output 920 | // bytes as signed differences for this test.) This method 921 | // usually outperforms any single fixed filter choice." as 922 | // per . 923 | 924 | // Note that I've found this to be incorrect, as far as 925 | // typical RGB and RGBA images found on the web go, using 926 | // None for all scanlines outperforms the heuristic in 57% 927 | // of the cases. Even if you carefully check whether they 928 | // should really be stored as indexed images, there is not 929 | // much evidence to support "usually". A better heuristic 930 | // would be applying the heuristic and None to all and use 931 | // the combination that performs better. 932 | 933 | for (flt_iter fi = flt_singles.begin(); fi != flt_singles.end(); ++fi) { 934 | std::vector::iterator scanline = 935 | flt_singles[fi->first].begin() + row * scanline_width; 936 | 937 | size_t sum = std::accumulate(scanline + 1, 938 | scanline + scanline_width, 0, sum_abs); 939 | 940 | // If, for this scanline, the current filter is better 941 | // then the previous best filter, we memorize this filter, 942 | // otherwise this filter can be disregarded for the line. 943 | if (sum >= best_sum) 944 | continue; 945 | 946 | best_sum = sum; 947 | best_flt = fi->first; 948 | } 949 | 950 | genomes["heuristic"]->gene(row, (PngFilter)best_flt); 951 | } 952 | 953 | // As an experimental heuristic, this compresses each scanline 954 | // individually and picks the filter that compresses the line 955 | // best. This may be a useful clue for the others, but tests 956 | // suggests this might interfere in cases where zlib is a poor 957 | // estimator, tuning genomes too much for zlib instead of 7zip. 958 | // Generally this should be expected to perform poorly for very 959 | // small images. In the standard Alexa 1000 sample it performs 960 | // better than the specification's heuristic in 73% of cases; 961 | // files would be around 3% (median) and 4% (mean) smaller. 962 | 963 | for (int row = 0; row < ge.size(); ++row) { 964 | size_t best_sum = SIZE_MAX; 965 | PngFilter best_flt = None; 966 | 967 | for (flt_iter fi = flt_singles.begin(); fi != flt_singles.end(); ++fi) { 968 | std::vector::iterator scanline = 969 | flt_singles[fi->first].begin() + row * scanline_width; 970 | 971 | std::vector line(scanline, scanline + scanline_width); 972 | size_t sum = deflate_fast->deflate(line).size(); 973 | 974 | if (sum >= best_sum) 975 | continue; 976 | 977 | best_sum = sum; 978 | best_flt = fi->first; 979 | } 980 | 981 | genomes["deflate scanline"]->gene(row, (PngFilter)best_flt); 982 | } 983 | 984 | // unigram heuristic 985 | for (int row = 0; row < ge.size(); ++row) { 986 | size_t best_sum = SIZE_MAX; 987 | PngFilter best_flt = None; 988 | 989 | for (flt_iter fi = flt_singles.begin(); fi != flt_singles.end(); ++fi) { 990 | std::bitset<65536> seen; 991 | std::vector::iterator it; 992 | std::vector::iterator scanline = 993 | flt_singles[fi->first].begin() + row * scanline_width; 994 | 995 | for (it = scanline; it < scanline + scanline_width; ++it) 996 | seen.set(uint8_t(*it)); 997 | 998 | size_t sum = seen.count(); 999 | 1000 | if (sum >= best_sum) 1001 | continue; 1002 | 1003 | best_sum = sum; 1004 | best_flt = fi->first; 1005 | } 1006 | 1007 | genomes["distinct bytes"]->gene(row, (PngFilter)best_flt); 1008 | } 1009 | 1010 | // bigram heuristic 1011 | for (int row = 0; row < ge.size(); ++row) { 1012 | size_t best_sum = SIZE_MAX; 1013 | PngFilter best_flt = None; 1014 | 1015 | for (flt_iter fi = flt_singles.begin(); fi != flt_singles.end(); ++fi) { 1016 | std::bitset<65536> seen; 1017 | std::vector::iterator it; 1018 | std::vector::iterator scanline = 1019 | flt_singles[fi->first].begin() + row * scanline_width; 1020 | 1021 | for (it = scanline + 1; it < scanline + scanline_width; ++it) 1022 | seen.set((uint8_t(*(it - 1)) << 8) | uint8_t(*it)); 1023 | 1024 | size_t sum = seen.count(); 1025 | 1026 | if (sum >= best_sum) 1027 | continue; 1028 | 1029 | best_sum = sum; 1030 | best_flt = fi->first; 1031 | } 1032 | 1033 | genomes["distinct bigrams"]->gene(row, (PngFilter)best_flt); 1034 | } 1035 | 1036 | z_stream strm; 1037 | strm.zalloc = Z_NULL; 1038 | strm.zfree = Z_NULL; 1039 | strm.opaque = Z_NULL; 1040 | 1041 | if (deflateInit2(&strm, zlib_level, Z_DEFLATED, 1042 | zlib_windowBits, zlib_memLevel, 1043 | zlib_strategy) != Z_OK) { 1044 | abort(); 1045 | } 1046 | 1047 | size_t max = deflateBound(&strm, original_inflated.size()); 1048 | std::vector strm_deflated(max); 1049 | strm.next_out = (Bytef*)&strm_deflated[0]; 1050 | strm.avail_out = max; 1051 | 1052 | for (int row = 0; row < ge.size(); ++row) { 1053 | size_t pos = row * scanline_width; 1054 | size_t best_sum = INT_MAX; 1055 | PngFilter best_flt = None; 1056 | 1057 | for (flt_iter fi = flt_singles.begin(); fi != flt_singles.end(); ++fi) { 1058 | z_stream here; 1059 | 1060 | if (deflateCopy(&here, &strm) != Z_OK) { 1061 | } 1062 | 1063 | here.next_in = (Bytef*)&flt_singles[fi->first][pos]; 1064 | here.avail_in = scanline_width; 1065 | 1066 | int status = deflate(&here, Z_FINISH); 1067 | if (status != Z_STREAM_END && status != Z_OK) { 1068 | } 1069 | 1070 | size_t sum = max - here.avail_out; 1071 | 1072 | deflateEnd(&here); 1073 | 1074 | if (sum >= best_sum) 1075 | continue; 1076 | 1077 | best_sum = sum; 1078 | best_flt = fi->first; 1079 | } 1080 | 1081 | genomes["incremental"]->gene(row, (PngFilter)best_flt); 1082 | strm.next_in = (Bytef*)&flt_singles[(PngFilter)best_flt][pos]; 1083 | strm.avail_in = scanline_width; 1084 | 1085 | if (deflate(&strm, Z_NO_FLUSH) != Z_OK) { 1086 | } 1087 | 1088 | } 1089 | 1090 | deflateEnd(&strm); 1091 | 1092 | // As initial population this uses, by default, the filters in the 1093 | // original image, the filters derived by the heuristic proposed by 1094 | // the PNG specification, the unary filter selections where every 1095 | // scanline uses the same filter, and a couple of random ones re- 1096 | // required to fill the population to the requested size. With some 1097 | // Genetic Algorithms results vary a lot depending on the filters 1098 | // in the initial population. For instance, improvements are hard to 1099 | // find with some GAs if the heuristic filter selection is included. 1100 | 1101 | // TODO: for now this uses copies but there is no deallocator for 1102 | // the originals, at some point I should figure out how to own the 1103 | // critters. Maybe the Wolf should have a second population that 1104 | // owns them, that way the default deallocator should handle them. 1105 | 1106 | typedef std::map::iterator ge_iter; 1107 | for (ge_iter i = genomes.begin(); i != genomes.end(); ++i) 1108 | i->second->evaluate(); 1109 | 1110 | if (!exclude_singles) { 1111 | initial_pop.add(*this->genomes["all set to none"]); 1112 | initial_pop.add(*this->genomes["all set to sub"]); 1113 | initial_pop.add(*this->genomes["all set to up"]); 1114 | initial_pop.add(*this->genomes["all set to avg"]); 1115 | initial_pop.add(*this->genomes["all set to paeth"]); 1116 | } 1117 | 1118 | if (!exclude_original) 1119 | initial_pop.add(*this->genomes["original"]); 1120 | 1121 | if (!exclude_heuristic) 1122 | initial_pop.add(*this->genomes["heuristic"]); 1123 | 1124 | if (!exclude_experiment1) 1125 | initial_pop.add(*this->genomes["deflate scanline"]); 1126 | 1127 | if (!exclude_experiment2) 1128 | initial_pop.add(*this->genomes["distinct bytes"]); 1129 | 1130 | if (!exclude_experiment3) 1131 | initial_pop.add(*this->genomes["distinct bigrams"]); 1132 | 1133 | if (!exclude_experiment4) 1134 | initial_pop.add(*this->genomes["incremental"]); 1135 | 1136 | // If all standard genomes have been excluded a randomized one has 1137 | // to be added so the population knows how to make more genomes. 1138 | if (initial_pop.size() == 0) { 1139 | PngFilterGenome clone(*genomes["original"]); 1140 | clone.initialize(); 1141 | initial_pop.add(clone); 1142 | } 1143 | 1144 | // This adds random critters to the initial population. Very low 1145 | // values for the population size generally lead to insufficient 1146 | // genetic diversity so improvements are rarely found, while with 1147 | // very high values evolution takes too much time. I've found the 1148 | // value here works okay-ish for reasonably sized images. There 1149 | // is the option to make this configurable, but there is not much 1150 | // evidence the value makes that much of a difference. 1151 | initial_pop.size(population_size); 1152 | 1153 | // This defines ordering by score (idat size in our case). Lower 1154 | // idat size is better than higher idat size, setting accordingly. 1155 | initial_pop.order(GAPopulation::LOW_IS_BEST); 1156 | 1157 | if (bigger_is_better) 1158 | initial_pop.order(GAPopulation::HIGH_IS_BEST); 1159 | } 1160 | 1161 | //////////////////////////////////////////////////////////////////// 1162 | // Experiment 1163 | //////////////////////////////////////////////////////////////////// 1164 | void PngWolf::run() { 1165 | 1166 | // With what few samples I have used in testing, GAIncrementalGA 1167 | // works very well with the other options I've used, it finds im- 1168 | // provements fairly reliably while other Algorithms have trouble 1169 | // to find improvements after a certain number of generations. 1170 | GAIncrementalGA ga(initial_pop); 1171 | 1172 | // There is no particular reason to use the tournament selector, 1173 | // but the general concept seems sound for our purposes here, and 1174 | // I've found this to work better than some others in my simple 1175 | // tests, better than selecting by Rank for instance. 1176 | ga.selector(GATournamentSelector()); 1177 | 1178 | // I am not entirely sure I understand the scaling concept in the 1179 | // GALib library, I initially did not realize fitness and score 1180 | // weren't the same thing in GALib, so I turned scaling off to 1181 | // make "fitness" the same as the "score". After gaining a better 1182 | // understanding of "scaling" I tried a couple of other things, 1183 | // but no scaling still seemed to work best for my samples. 1184 | ga.scaling(GANoScaling()); 1185 | 1186 | // This is currently not used and maybe should not be used as the 1187 | // scoping slash ownership is unclear. It would work only during 1188 | // run() which is a bit non-intuitive, so TODO: maybe remove this. 1189 | this->ga = &ga; 1190 | 1191 | ga.crossover(PngFilterGenome::TwoPointCrossover); 1192 | 1193 | best_genomes.push_back((PngFilterGenome*) 1194 | ga.population().best().clone()); 1195 | 1196 | fprintf(stdout, "---\n"); 1197 | 1198 | if (ihdr.height == 1) 1199 | goto after_while; 1200 | 1201 | while (!should_abort) { 1202 | 1203 | double since_start = difftime(time(NULL), program_begun_at); 1204 | double since_last = difftime(time(NULL), last_improvement_at); 1205 | size_t deflated = genomes_evaluated * original_inflated.size(); 1206 | 1207 | if (max_time > 0 && max_time < since_start) 1208 | break; 1209 | 1210 | if (max_evaluations > 0 && max_evaluations < genomes_evaluated) 1211 | break; 1212 | 1213 | if (max_stagnate_time > 0 && max_stagnate_time < since_last) 1214 | break; 1215 | 1216 | if (max_deflate > 0 && max_deflate < deflated / (1024*1024)) 1217 | break; 1218 | 1219 | nth_generation++; 1220 | 1221 | ga.step(); 1222 | last_step_at = time(NULL); 1223 | 1224 | if (should_abort) 1225 | break; 1226 | 1227 | PngFilterGenome& new_best = 1228 | (PngFilterGenome&)ga.population().best(); 1229 | 1230 | if (!bigger_is_better) 1231 | if (new_best.score() >= best_genomes.back()->score()) 1232 | continue; 1233 | 1234 | if (bigger_is_better) 1235 | if (new_best.score() <= best_genomes.back()->score()) 1236 | continue; 1237 | 1238 | log_critter(&new_best); 1239 | 1240 | last_improvement_at = time(NULL); 1241 | best_genomes.push_back((PngFilterGenome*)new_best.clone()); 1242 | } 1243 | 1244 | after_while: 1245 | 1246 | // Since we intercept CTRL+C the user should get some feedback 1247 | // on that as soon as possible, so the log header comes here. 1248 | fprintf(stdout, "---\n"); 1249 | fflush(stdout); 1250 | } 1251 | 1252 | void PngWolf::recompress() { 1253 | best_inflated = refilter(*best_genomes.back()); 1254 | best_deflated = deflate_good->deflate(best_inflated); 1255 | 1256 | // In my test sample in 1.66% of cases, using a high zlib level, 1257 | // zlib is able to produce smaller output than 7-Zip. So for the 1258 | // case where users do choose a high setting for zlib, reward 1259 | // them by using zlib instead to recompress. Since zlib is fast, 1260 | // this recompression should not be much of a performance hit. 1261 | 1262 | // TODO: This should be noted in the verbose output, otherwise 1263 | // this would make 7zip appear better than it is. In the longer 1264 | // term perhaps the output should simply say what estimator and 1265 | // what compressor was used and give the respective sizes. 1266 | 1267 | if (best_deflated.size() > best_genomes.back()->score()) { 1268 | best_deflated = deflate_fast->deflate(best_inflated); 1269 | } 1270 | 1271 | // TODO: Doing this here is a bit of an hack, and doing it 1272 | // should also be logged in the verbose output. Main problem 1273 | // is separation of things you'd put into a library and what 1274 | // is really more part of the command line application. Right 1275 | // now run() should really do this, but then you could not 1276 | // abort 7zip easily. Also not sure what --best-idat-to ought 1277 | // to do here. Might end up exposing a step() method and let 1278 | // the command line part do logging and other things. 1279 | 1280 | if (best_deflated.size() > original_deflated.size() && !even_if_bigger) { 1281 | best_genomes.push_back(genomes["original"]); 1282 | best_inflated = original_inflated; 1283 | best_deflated = original_deflated; 1284 | } 1285 | 1286 | done_deflating_at = time(NULL); 1287 | } 1288 | 1289 | bool PngWolf::read_file() { 1290 | 1291 | char fileMagic[8]; 1292 | unsigned int iend_chunk_count = 0; 1293 | size_t expected; 1294 | 1295 | std::ifstream in; 1296 | in.exceptions(std::ios::badbit | std::ios::failbit); 1297 | in.open(in_path, std::ios::binary | std::ios::in); 1298 | in.read(fileMagic, 8); 1299 | 1300 | if (memcmp(fileMagic, PNG_MAGIC, 8) != 0) 1301 | goto error; 1302 | 1303 | while (!in.eof()) { 1304 | PngChunk chunk; 1305 | 1306 | in.read((char*)&chunk.size, sizeof(chunk.size)); 1307 | chunk.size = ntohl(chunk.size); 1308 | 1309 | in.read((char*)&chunk.type, sizeof(chunk.type)); 1310 | chunk.type = ntohl(chunk.type); 1311 | 1312 | if (chunk.size > 0) { 1313 | chunk.data.resize(chunk.size); 1314 | in.read((char*)&chunk.data[0], chunk.size); 1315 | } 1316 | 1317 | in.read((char*)&chunk.crc32, sizeof(chunk.crc32)); 1318 | 1319 | chunk.crc32 = ntohl(chunk.crc32); 1320 | 1321 | // IHDR 1322 | if (chunk.type == IHDR_TYPE && chunk.size == 13) { 1323 | 1324 | // TODO: This does not check that this is the first and only 1325 | // IHDR chunk in the file even though only one IHDR is allowed. 1326 | 1327 | memcpy(&ihdr.width, &chunk.data.at(0), sizeof(ihdr.width)); 1328 | ihdr.width = ntohl(ihdr.width); 1329 | 1330 | memcpy(&ihdr.height, &chunk.data.at(4), sizeof(ihdr.height)); 1331 | ihdr.height = ntohl(ihdr.height); 1332 | 1333 | ihdr.depth = chunk.data.at(8); 1334 | ihdr.color = chunk.data.at(9); 1335 | ihdr.comp = chunk.data.at(10); 1336 | ihdr.filter = chunk.data.at(11); 1337 | ihdr.interlace = chunk.data.at(12); 1338 | } 1339 | 1340 | // IDAT 1341 | if (chunk.type == IDAT_TYPE) 1342 | original_deflated.insert(original_deflated.end(), 1343 | chunk.data.begin(), chunk.data.end()); 1344 | 1345 | // IEND 1346 | if (chunk.type == IEND_TYPE) 1347 | iend_chunk_count++; 1348 | 1349 | chunks.push_back(chunk); 1350 | 1351 | // Peek so the eof check works as expected. 1352 | in.peek(); 1353 | } 1354 | 1355 | in.close(); 1356 | 1357 | // We can't do anything if there is no image data in the input. 1358 | if (original_deflated.size() == 0) 1359 | goto error; 1360 | 1361 | // For simplicity, we rely on the image having only exactly one 1362 | // IEND chunk (as mandated by the specification) so inserting a 1363 | // new IDAT chunk is simple. Note that there are other possible 1364 | // errors with the chunk arrangement that are not checked for, 1365 | // but the worst that would happen is that a broken image is re- 1366 | // written into a new similarily broken image, which is fine. 1367 | if (iend_chunk_count != 1) 1368 | goto error; 1369 | 1370 | // PNG does not allow images with zero height or width, and at 1371 | // the time of writing, only filter mode zero was permitted. 1372 | if (ihdr.width == 0 || ihdr.height == 0 || ihdr.filter != 0) 1373 | goto error; 1374 | 1375 | // At the time of writing, compression level zero was the only 1376 | // valid one, and color modes could not exceed six; interlaced 1377 | // images are not supported, mainly because the author did not 1378 | // bother to implement Adam7 when the goal is to minimize size. 1379 | // Futher checks on the color mode are performed later. TODO: 1380 | // since interlaced images are not supported, that may merit a 1381 | // specific error message pointing that design decision out. 1382 | if (ihdr.comp != 0 || ihdr.interlace != 0 || ihdr.color > 6) 1383 | goto error; 1384 | 1385 | // PNG does not allow bit depths below one or above 16 1386 | if (ihdr.depth == 0 || ihdr.depth > 16) 1387 | goto error; 1388 | 1389 | // PNG bit depths must be a power of two 1390 | if ((ihdr.depth - 1) & ihdr.depth) 1391 | goto error; 1392 | 1393 | static const uint32_t channel_map[] = { 1394 | 1, 0, 3, 1, 2, 0, 4 1395 | }; 1396 | 1397 | // This validates generally permissable color modes. It does not 1398 | // fully check whether the combination of bith depth and color 1399 | // mode is permitted by the specification. TODO: Maybe it should. 1400 | if (channel_map[ihdr.color] < 1) 1401 | goto error; 1402 | 1403 | original_inflated = inflate_zlib(original_deflated); 1404 | 1405 | expected = ihdr.height * int(ceil( 1406 | float(((ihdr.width * channel_map[ihdr.color] * ihdr.depth + 8) / 8.0f)) 1407 | )); 1408 | 1409 | if (expected != original_inflated.size()) 1410 | goto error; 1411 | 1412 | scanline_width = expected / ihdr.height; 1413 | scanline_delta = channel_map[ihdr.color] * 1414 | int(ceil(float(ihdr.depth / 8.0))); 1415 | 1416 | for (size_t ix = 0; ix < ihdr.height; ++ix) { 1417 | unsigned char filter = original_inflated.at(ix * scanline_width); 1418 | 1419 | // Abort when the image uses an unsupported filter type 1420 | if (filter > 4) 1421 | goto error; 1422 | 1423 | original_filters.push_back((PngFilter)filter); 1424 | } 1425 | 1426 | // TODO: copy properly here 1427 | original_unfiltered.resize(original_inflated.size()); 1428 | memcpy(&original_unfiltered[0], 1429 | &original_inflated[0], original_inflated.size()); 1430 | 1431 | unfilter_idat((unsigned char*)&original_unfiltered[0], 1432 | ihdr.height, scanline_delta, scanline_width); 1433 | 1434 | // Creates a histogram of the fully transparent pixels in the 1435 | // image. It is apparently common for graphics programs to 1436 | // keep the color values of fully transparent pixels around, 1437 | // but this is rarely desired and makes compression harder, so 1438 | // we tell users about that and offer to normalize the pixels. 1439 | 1440 | // TODO: Now that it is modified, original_unfiltered is the 1441 | // wrong name for the attribute. 1442 | 1443 | if (ihdr.color == 6 && ihdr.depth == 8) { 1444 | uint32_t pixel; 1445 | uint32_t zero = 0; 1446 | for (uint32_t row = 0; row < ihdr.height; ++row) { 1447 | for (uint32_t col = 1; col < scanline_width; col += 4) { 1448 | size_t pos = row * scanline_width + col; 1449 | 1450 | if (original_unfiltered[pos + 3] != 0) 1451 | continue; 1452 | 1453 | memcpy(&pixel, &original_unfiltered[pos], 4); 1454 | invis_colors[pixel]++; 1455 | 1456 | if (normalize_alpha) 1457 | memcpy(&original_unfiltered[pos], &zero, 4); 1458 | } 1459 | } 1460 | } 1461 | 1462 | // For very large images the highest 7-Zip setting requires too 1463 | // much time to be worth the saved bytes, especially as `pngout` 1464 | // performs better for such images, at least if they are highly 1465 | // redundant, anyway, so this option allows picking the highest 1466 | // setting for small images while not requiring users to wait a 1467 | // very long time for the compressed result. TODO: maybe the 1468 | // base value should configurable. 1469 | if (auto_mpass) { 1470 | double times = double(original_inflated.size()) / (64*1024.f); 1471 | szip_pass = 16 - std::max(1, int(floor(times))); 1472 | } 1473 | 1474 | return false; 1475 | 1476 | error: 1477 | 1478 | return true; 1479 | } 1480 | 1481 | //////////////////////////////////////////////////////////////////// 1482 | // Save data 1483 | //////////////////////////////////////////////////////////////////// 1484 | bool PngWolf::save_original_idat(const char* path) { 1485 | return save_idat(path, original_deflated, original_inflated); 1486 | } 1487 | 1488 | bool PngWolf::save_best_idat(const char* path) { 1489 | return save_idat(path, best_deflated, best_inflated); 1490 | } 1491 | 1492 | bool PngWolf::save_idat(const char* path, std::vector& deflated, std::vector& inflated) { 1493 | 1494 | // TODO: when there is a simple inflate() function, make this 1495 | // assert when deflated and inflated to not mach each other. 1496 | // Or alternatively make some IDAT struct that has both and 1497 | // then require its use for this function. 1498 | 1499 | FILE* out = fopen(path, "wb"); 1500 | 1501 | static const uint8_t GZIP_HEADER[] = { 1502 | 0x1f, 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0x03 1503 | }; 1504 | 1505 | if (out == NULL) 1506 | return true; 1507 | 1508 | if (fwrite(GZIP_HEADER, sizeof(GZIP_HEADER), 1, out) != 1) { 1509 | } 1510 | 1511 | if (fwrite(&deflated[2], deflated.size() - 6, 1, out) != 1) { 1512 | } 1513 | 1514 | // TODO: endianess? 1515 | 1516 | uint32_t crc = crc32(0L, Z_NULL, 0); 1517 | crc = crc32(crc, (Bytef*)&inflated[0], inflated.size()); 1518 | 1519 | if (fwrite(&crc, sizeof(crc), 1, out) != 1) { 1520 | } 1521 | 1522 | uint32_t size = inflated.size(); 1523 | if (fwrite(&size, sizeof(size), 1, out) != 1) { 1524 | } 1525 | 1526 | if (fclose(out) != 0) { 1527 | } 1528 | 1529 | return false; 1530 | } 1531 | 1532 | bool PngWolf::save_file() { 1533 | 1534 | // Create new list of chunks with old IDATs removed, and when 1535 | // IEND is seen, insert a new IDAT chunk and the IEND chunk. 1536 | // Then proceed with serializing the whole new list to a file. 1537 | 1538 | std::list new_chunks; 1539 | std::list::iterator i; 1540 | 1541 | for (i = chunks.begin(); i != chunks.end(); ++i) { 1542 | if (i->type == IDAT_TYPE) 1543 | continue; 1544 | 1545 | if (i->type != IEND_TYPE) { 1546 | new_chunks.push_back(*i); 1547 | continue; 1548 | } 1549 | 1550 | PngChunk new_idat; 1551 | new_idat.type = IDAT_TYPE; 1552 | new_idat.data = best_deflated; 1553 | new_idat.size = new_idat.data.size(); 1554 | 1555 | uint32_t idat_type_network = htonl(new_idat.type); 1556 | 1557 | new_idat.crc32 = crc32(0L, Z_NULL, 0); 1558 | new_idat.crc32 = crc32(new_idat.crc32, 1559 | (const Bytef*)&idat_type_network, 1560 | sizeof(idat_type_network)); 1561 | 1562 | new_idat.crc32 = crc32(new_idat.crc32, 1563 | (const Bytef*)&new_idat.data[0], 1564 | new_idat.data.size()); 1565 | 1566 | new_chunks.push_back(new_idat); 1567 | new_chunks.push_back(*i); 1568 | } 1569 | 1570 | // TODO: it might be nice to not overwrite existing files, but 1571 | // as a rule, if there are separate parameters for in and out, 1572 | // that might not provide the best usability for users. 1573 | 1574 | FILE* out = fopen(out_path, "wb"); 1575 | 1576 | if (out == NULL) 1577 | return true; 1578 | 1579 | if (fwrite(PNG_MAGIC, 8, 1, out) != 1) { 1580 | } 1581 | 1582 | for (i = new_chunks.begin(); i != new_chunks.end(); ++i) { 1583 | uint32_t size = htonl(i->size); 1584 | uint32_t type = htonl(i->type); 1585 | uint32_t crc32 = htonl(i->crc32); 1586 | 1587 | // TODO: does this merit handling write errors? 1588 | 1589 | if (fwrite(&size, sizeof(size), 1, out) != 1) { 1590 | } 1591 | 1592 | if (fwrite(&type, sizeof(type), 1, out) != 1) { 1593 | } 1594 | 1595 | if (i->data.size() > 0) { 1596 | if (fwrite(&i->data[0], i->data.size(), 1, out) != 1) { 1597 | } 1598 | } 1599 | 1600 | if (fwrite(&crc32, sizeof(crc32), 1, out) != 1) { 1601 | } 1602 | } 1603 | 1604 | if (fclose(out) != 0) { 1605 | } 1606 | 1607 | return false; 1608 | } 1609 | 1610 | //////////////////////////////////////////////////////////////////// 1611 | // Help! 1612 | //////////////////////////////////////////////////////////////////// 1613 | void 1614 | help(void) { 1615 | fprintf(stdout, "%s", 1616 | " -----------------------------------------------------------------------------\n" 1617 | " Usage: pngwolf --in=file.png --out=file.png \n" 1618 | " -----------------------------------------------------------------------------\n" 1619 | " --in= The PNG input image \n" 1620 | " --out= The PNG output file (defaults to not saving!)\n" 1621 | " --original-idat-to= Save original IDAT data in a gzip container \n" 1622 | " --best-idat-to= Save best IDAT data in a gzip container \n" 1623 | " --exclude-singles Exclude single-filter genomes from population\n" 1624 | " --exclude-original Exclude the filters of the input image \n" 1625 | " --exclude-heuristic Exclude the heuristically generated filters \n" 1626 | " --exclude-experiments Exclude experimental heuristics \n" 1627 | " --population-size= Size of the population. Defaults to 19. \n" 1628 | " --max-time= Timeout after seconds. (default: 0, disabled)\n" 1629 | " --max-stagnate-time= Give up if no improvement is found (d: 5) \n" 1630 | " --max-deflate= Give up after deflating this many megabytes \n" 1631 | " --max-evaluations= Give up after evaluating this many genomes \n" 1632 | " --zlib-level= zlib estimator compression level (default: 5)\n" 1633 | " --zlib-strategy= zlib estimator strategy (default: 0) \n" 1634 | " --zlib-window= zlib estimator window bits (default: 15) \n" 1635 | " --zlib-memlevel= zlib estimator memory level (default: 8) \n" 1636 | " --7zip-mfb= 7zip fast bytes 3..258 (default: 258) \n" 1637 | " --7zip-mpass= 7zip passes 0..15 (d: 2; > ~ slower, smaller)\n" 1638 | " --7zip-mmc= 7zip match finder cycles (d: 258) \n" 1639 | " --verbose-analysis More details in initial image analysis \n" 1640 | " --verbose-summary More details in optimization summary \n" 1641 | " --verbose-genomes More details when improvements are found \n" 1642 | " --verbose Shorthand for all verbosity options \n" 1643 | " --normalize-alpha For RGBA, make fully transparent pixels black\n" 1644 | " --even-if-bigger Otherwise the original is copied if it's best\n" 1645 | " --bigger-is-better Find filter sequences that compress worse \n" 1646 | " --info Just print out verbose analysis and exit \n" 1647 | " --help Print this help page and exit \n" 1648 | " -----------------------------------------------------------------------------\n" 1649 | " To reduce the file size of PNG images `pngwolf` uses a genetic algorithm for \n" 1650 | " finding the best scanline filter for each scanline in the image. It does not \n" 1651 | " implement any other optimization techniques (a future version may attempt to \n" 1652 | " use a similar approach to find a good arrangement of color palette entries). \n" 1653 | " \n" 1654 | " To approximate the quality of a filter combination it compresses IDAT chunks \n" 1655 | " using `zlib` and ultimately uses the Deflate encoder in `7-Zip` to store the \n" 1656 | " output image. It is slow because it recompresses the IDAT data fully for all \n" 1657 | " filter combinations even if only minor changes are made or if two filter com-\n" 1658 | " binations are merged, as `zlib` has no built-in support for caching analysis \n" 1659 | " data. Send mail if you know of a freely available encoder that supports that.\n" 1660 | " -----------------------------------------------------------------------------\n" 1661 | " Output images should be saved even if you send SIGINT (~CTRL+C) to `pngwolf`.\n" 1662 | " The machine-readable progress report format is based on YAML http://yaml.org/\n" 1663 | " -----------------------------------------------------------------------------\n" 1664 | " Uses http://zlib.net/ and http://lancet.mit.edu/ga/ and http://www.7-zip.org/\n" 1665 | " -----------------------------------------------------------------------------\n" 1666 | " http://bjoern.hoehrmann.de/pngwolf/ (c) 2008-2011 http://bjoern.hoehrmann.de/\n" 1667 | ""); 1668 | } 1669 | 1670 | //////////////////////////////////////////////////////////////////// 1671 | // main 1672 | //////////////////////////////////////////////////////////////////// 1673 | int 1674 | main(int argc, char *argv[]) { 1675 | 1676 | bool argHelp = false; 1677 | bool argVerboseAnalysis = false; 1678 | bool argVerboseSummary = false; 1679 | bool argVerboseGenomes = false; 1680 | bool argExcludeSingles = false; 1681 | bool argExcludeOriginal = false; 1682 | bool argExcludeHeuristic = false; 1683 | bool argExcludeExperiment1 = false; 1684 | bool argExcludeExperiment2 = false; 1685 | bool argExcludeExperiment3 = false; 1686 | bool argExcludeExperiment4 = false; 1687 | bool argInfo = false; 1688 | bool argNormalizeAlpha = false; 1689 | bool argEvenIfBigger = false; 1690 | bool argAutoMpass = false; 1691 | bool argBiggerIsBetter = false; 1692 | const char* argPng = NULL; 1693 | const char* argOut = NULL; 1694 | const char* argBestIdatTo = NULL; 1695 | const char* argOriginalIdatTo = NULL; 1696 | int argMaxTime = 0; 1697 | int argMaxStagnateTime = 5; 1698 | int argMaxEvaluations = 0; 1699 | int argMaxDeflate = 0; 1700 | int argPopulationSize = 19; 1701 | int argZlibLevel = 5; 1702 | int argZlibStrategy = 0; 1703 | int argZlibMemlevel = 8; 1704 | int argZlibWindow = 15; 1705 | int arg7zipFastBytes = 258; 1706 | int arg7zipPasses = 2; 1707 | int arg7zipCycles = 258; 1708 | 1709 | bool argOkay = true;; 1710 | 1711 | #ifndef _MSC_VER 1712 | sig_t old_handler; 1713 | #endif 1714 | 1715 | // Parse command line parameters 1716 | for (int ax = 1; ax < argc; ++ax) { 1717 | size_t nlen; 1718 | 1719 | const char* s = argv[ax]; 1720 | const char* value; 1721 | 1722 | // boolean options 1723 | if (strcmp("--help", s) == 0) { 1724 | argHelp = 1; 1725 | break; 1726 | 1727 | } else if (strcmp("--verbose-analysis", s) == 0) { 1728 | argVerboseAnalysis = true; 1729 | continue; 1730 | 1731 | } else if (strcmp("--verbose-summary", s) == 0) { 1732 | argVerboseSummary = true; 1733 | continue; 1734 | 1735 | } else if (strcmp("--verbose-genomes", s) == 0) { 1736 | argVerboseGenomes = true; 1737 | continue; 1738 | 1739 | } else if (strcmp("--exclude-original", s) == 0) { 1740 | argExcludeOriginal = true; 1741 | continue; 1742 | 1743 | } else if (strcmp("--exclude-singles", s) == 0) { 1744 | argExcludeSingles = true; 1745 | continue; 1746 | 1747 | } else if (strcmp("--exclude-heuristic", s) == 0) { 1748 | argExcludeHeuristic = true; 1749 | continue; 1750 | 1751 | } else if (strcmp("--verbose", s) == 0) { 1752 | argVerboseAnalysis = true; 1753 | argVerboseSummary = true; 1754 | argVerboseGenomes = true; 1755 | continue; 1756 | 1757 | } else if (strcmp("--info", s) == 0) { 1758 | argInfo = true; 1759 | argVerboseAnalysis = true; 1760 | continue; 1761 | 1762 | } else if (strcmp("--normalize-alpha", s) == 0) { 1763 | argNormalizeAlpha = true; 1764 | continue; 1765 | 1766 | } else if (strcmp("--even-if-bigger", s) == 0) { 1767 | argEvenIfBigger = true; 1768 | continue; 1769 | 1770 | } else if (strcmp("--bigger-is-better", s) == 0) { 1771 | argBiggerIsBetter = true; 1772 | continue; 1773 | 1774 | } else if (strcmp("--exclude-experiments", s) == 0) { 1775 | argExcludeExperiment1 = true; 1776 | argExcludeExperiment2 = true; 1777 | argExcludeExperiment3 = true; 1778 | argExcludeExperiment4 = true; 1779 | continue; 1780 | 1781 | } 1782 | 1783 | value = strchr(s, '='); 1784 | 1785 | if (value == NULL) { 1786 | argOkay = false; 1787 | break; 1788 | } 1789 | 1790 | nlen = value++ - s; 1791 | 1792 | // --name=value options 1793 | if (strncmp("--in", s, nlen) == 0) { 1794 | argPng = value; 1795 | 1796 | } else if (strncmp("--out", s, nlen) == 0) { 1797 | argOut = value; 1798 | 1799 | } else if (strncmp("--best-idat-to", s, nlen) == 0) { 1800 | argBestIdatTo = value; 1801 | 1802 | } else if (strncmp("--original-idat-to", s, nlen) == 0) { 1803 | argOriginalIdatTo = value; 1804 | 1805 | } else if (strncmp("--max-time", s, nlen) == 0) { 1806 | argMaxTime = atoi(value); 1807 | 1808 | } else if (strncmp("--max-stagnate-time", s, nlen) == 0) { 1809 | argMaxStagnateTime = atoi(value); 1810 | 1811 | } else if (strncmp("--max-deflate", s, nlen) == 0) { 1812 | argMaxDeflate = atoi(value); 1813 | 1814 | } else if (strncmp("--max-evaluations", s, nlen) == 0) { 1815 | argMaxEvaluations = atoi(value); 1816 | 1817 | } else if (strncmp("--population-size", s, nlen) == 0) { 1818 | argPopulationSize = atoi(value); 1819 | 1820 | } else if (strncmp("--zlib-level", s, nlen) == 0) { 1821 | argZlibLevel = atoi(value); 1822 | argOkay &= argZlibLevel >= 0; 1823 | argOkay &= argZlibLevel <= 9; 1824 | 1825 | } else if (strncmp("--zlib-memlevel", s, nlen) == 0) { 1826 | argZlibMemlevel = atoi(value); 1827 | argOkay &= argZlibMemlevel >= 1; 1828 | argOkay &= argZlibMemlevel <= 9; 1829 | 1830 | } else if (strncmp("--zlib-window", s, nlen) == 0) { 1831 | argZlibWindow = atoi(value); 1832 | argOkay &= argZlibWindow >= 8; 1833 | argOkay &= argZlibWindow <= 15; 1834 | 1835 | } else if (strncmp("--zlib-strategy", s, nlen) == 0) { 1836 | argZlibStrategy = atoi(value); 1837 | argOkay &= argZlibStrategy == Z_DEFAULT_STRATEGY 1838 | || argZlibStrategy == Z_FILTERED 1839 | || argZlibStrategy == Z_HUFFMAN_ONLY 1840 | || argZlibStrategy == Z_RLE; 1841 | 1842 | } else if (strncmp("--7zip-mfb", s, nlen) == 0) { 1843 | arg7zipFastBytes = atoi(value); 1844 | argOkay &= arg7zipFastBytes >= 3; 1845 | argOkay &= arg7zipFastBytes <= 258; 1846 | 1847 | } else if (strncmp("--7zip-mpass", s, nlen) == 0) { 1848 | if (strcmp(value, "auto") == 0) { 1849 | argAutoMpass = true; 1850 | } else { 1851 | arg7zipPasses = atoi(value); 1852 | argOkay &= arg7zipPasses >= 1; 1853 | argOkay &= arg7zipPasses <= 15; 1854 | } 1855 | 1856 | } else if (strncmp("--7zip-mmc", s, nlen) == 0) { 1857 | arg7zipCycles = atoi(value); 1858 | 1859 | } else { 1860 | // TODO: error 1861 | argHelp = 1; 1862 | } 1863 | } 1864 | 1865 | if (argHelp || argPng == NULL || !argOkay) { 1866 | help(); 1867 | return EXIT_SUCCESS; 1868 | } 1869 | 1870 | DeflateZlib fast(argZlibLevel, argZlibWindow, argZlibMemlevel, argZlibStrategy); 1871 | Deflate7zip good(arg7zipPasses, arg7zipFastBytes, arg7zipCycles); 1872 | 1873 | wolf.szip_cycl = arg7zipCycles; 1874 | wolf.szip_fast = arg7zipFastBytes; 1875 | wolf.szip_pass = arg7zipPasses; 1876 | wolf.zlib_level = argZlibLevel; 1877 | wolf.zlib_memLevel = argZlibMemlevel; 1878 | wolf.zlib_strategy = argZlibStrategy; 1879 | wolf.zlib_windowBits = argZlibWindow; 1880 | wolf.deflate_fast = &fast; 1881 | wolf.deflate_good = &good; 1882 | wolf.in_path = argPng; 1883 | wolf.max_deflate = argMaxDeflate; 1884 | wolf.max_evaluations = argMaxEvaluations; 1885 | wolf.verbose_analysis = argVerboseAnalysis; 1886 | wolf.verbose_genomes = argVerboseGenomes; 1887 | wolf.verbose_summary = argVerboseSummary; 1888 | wolf.exclude_heuristic = argExcludeHeuristic; 1889 | wolf.exclude_original = argExcludeOriginal; 1890 | wolf.exclude_singles = argExcludeSingles; 1891 | wolf.exclude_experiment1 = argExcludeExperiment1; 1892 | wolf.exclude_experiment2 = argExcludeExperiment2; 1893 | wolf.exclude_experiment3 = argExcludeExperiment3; 1894 | wolf.exclude_experiment4 = argExcludeExperiment4; 1895 | wolf.population_size = argPopulationSize; 1896 | wolf.max_stagnate_time = argMaxStagnateTime; 1897 | wolf.last_step_at = time(NULL); 1898 | wolf.last_improvement_at = time(NULL); 1899 | wolf.program_begun_at = time(NULL); 1900 | wolf.max_time = argMaxTime; 1901 | wolf.out_path = argOut; 1902 | wolf.normalize_alpha = argNormalizeAlpha; 1903 | wolf.even_if_bigger = argEvenIfBigger; 1904 | wolf.auto_mpass = argAutoMpass; 1905 | wolf.bigger_is_better = argBiggerIsBetter; 1906 | 1907 | // TODO: ... 1908 | try { 1909 | if (wolf.read_file()) 1910 | goto error; 1911 | } catch (...) { 1912 | goto error; 1913 | } 1914 | 1915 | if (argOriginalIdatTo != NULL) 1916 | if (wolf.save_original_idat(argOriginalIdatTo)) 1917 | goto out_error; 1918 | 1919 | wolf.init_filters(); 1920 | wolf.log_analysis(); 1921 | 1922 | if (argInfo) 1923 | goto done; 1924 | 1925 | wolf.search_begun_at = time(NULL); 1926 | 1927 | #ifdef _MSC_VER 1928 | SetConsoleCtrlHandler(console_event_handler, TRUE); 1929 | #else 1930 | old_handler = signal(SIGINT, sigint_handler); 1931 | #endif 1932 | 1933 | wolf.run(); 1934 | 1935 | // Uninstall the SIGINT interceptor to allow users to abort 1936 | // the possibly very slow recompression step. This may lead 1937 | // to users unintentionally hit CTRL+C twice, but there is 1938 | // not much to avoid that, other than setting a timer with 1939 | // some grace perdiod which strikes me as too complicated. 1940 | 1941 | #ifdef _MSC_VER 1942 | SetConsoleCtrlHandler(console_event_handler, FALSE); 1943 | #else 1944 | signal(SIGINT, old_handler); 1945 | #endif 1946 | 1947 | wolf.recompress(); 1948 | 1949 | if (wolf.out_path) 1950 | if (wolf.save_file()) 1951 | goto out_error; 1952 | 1953 | if (argBestIdatTo != NULL) 1954 | if (wolf.save_best_idat(argBestIdatTo)) 1955 | goto out_error; 1956 | 1957 | wolf.log_summary(); 1958 | 1959 | done: 1960 | 1961 | return EXIT_SUCCESS; 1962 | 1963 | error: 1964 | if (wolf.ihdr.interlace) { 1965 | fprintf(stderr, "Interlaced images are not supported\n"); 1966 | return EXIT_FAILURE; 1967 | } 1968 | 1969 | fprintf(stderr, "Some error occured while reading the input file\n"); 1970 | return EXIT_FAILURE; 1971 | 1972 | out_error: 1973 | fprintf(stderr, "Some error occured while writing an output file\n"); 1974 | return EXIT_FAILURE; 1975 | 1976 | } 1977 | 1978 | // TODO: There are probably integer overflow issues with really, 1979 | // really big image files. Really, really big image ones. Biiig. 1980 | -------------------------------------------------------------------------------- /sevenzip920.patch: -------------------------------------------------------------------------------- 1 | diff -pur 7zip/C/Alloc.c 7zip/C/Alloc.c 2 | --- 7zip/C/Alloc.c 2008-09-24 14:51:47.000000000 +0200 3 | +++ 7zip/C/Alloc.c 2011-03-14 01:52:44.916600000 +0100 4 | @@ -6,7 +6,6 @@ Public domain */ 5 | #ifdef _WIN32 6 | #include 7 | #endif 8 | -#include 9 | 10 | #include "Alloc.h" 11 | 12 | diff -pur 7zip/C/Alloc.h 7zip/C/Alloc.h 13 | --- 7zip/C/Alloc.h 2009-02-07 07:44:34.000000000 +0100 14 | +++ 7zip/C/Alloc.h 2011-03-14 01:52:38.246600000 +0100 15 | @@ -5,6 +5,7 @@ 16 | #define __COMMON_ALLOC_H 17 | 18 | #include 19 | +#include 20 | 21 | #ifdef __cplusplus 22 | extern "C" { 23 | diff -pur 7zip/CPP/7zip/Common/StreamObjects.cpp 7zip/CPP/7zip/Common/StreamObjects.cpp 24 | --- 7zip/CPP/7zip/Common/StreamObjects.cpp 2010-11-02 18:12:26.000000000 +0100 25 | +++ 7zip/CPP/7zip/Common/StreamObjects.cpp 2011-03-05 18:44:51.044851100 +0100 26 | @@ -78,7 +78,7 @@ Byte *CDynBufSeqOutStream::GetBufPtrForW 27 | void CDynBufSeqOutStream::CopyToBuffer(CByteBuffer &dest) const 28 | { 29 | dest.SetCapacity(_size); 30 | - memcpy(dest, _buffer, _size); 31 | + memcpy(dest, _buffer.operator const unsigned char *(), _size); 32 | } 33 | 34 | STDMETHODIMP CDynBufSeqOutStream::Write(const void *data, UInt32 size, UInt32 *processedSize) 35 | --------------------------------------------------------------------------------