├── .github └── workflows │ └── crystal.yml ├── .gitignore ├── CHANGELOG.md ├── LICENSE ├── README.md ├── shard.yml ├── spec ├── crzt_spec.cr ├── sizer_spec.cr ├── spec_helper.cr └── writer_spec.cr └── src ├── cr_zip_tricks.cr ├── crc32_writer.cr ├── offset_io.cr ├── sizer.cr ├── streamer.cr ├── version.cr └── writer.cr /.github/workflows/crystal.yml: -------------------------------------------------------------------------------- 1 | name: Crystal CI 2 | 3 | on: 4 | push: 5 | pull_request: 6 | schedule: 7 | - cron: '0 0 * * *' 8 | 9 | jobs: 10 | test: 11 | strategy: 12 | fail-fast: false 13 | matrix: 14 | os: [ubuntu-latest, macos-latest] 15 | crystal: [latest, nightly] 16 | runs-on: ${{ matrix.os }} 17 | steps: 18 | - name: Install Crystal 19 | uses: oprypin/install-crystal@v1 20 | with: 21 | crystal: ${{ matrix.crystal }} 22 | - name: Check out repository code 23 | uses: actions/checkout@v2 24 | - name: Install dependencies 25 | run: shards install 26 | - name: Run tests 27 | run: crystal spec 28 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | /docs/ 2 | /lib/ 3 | /bin/ 4 | /.shards/ 5 | *.dwarf 6 | 7 | # Libraries don't need dependency lock 8 | # Dependencies will be locked in application that uses them 9 | /shard.lock 10 | -------------------------------------------------------------------------------- /CHANGELOG.md: -------------------------------------------------------------------------------- 1 | ## 0.3.0 2 | * Change required Crystal version to >= 0.35.1 3 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | The MIT License (MIT) 2 | 3 | Copyright (c) 2018 Julik Tarkhanov 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in 13 | all copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN 21 | THE SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # cr_zip_tricks 2 | 3 | [![Crystal CI](https://github.com/WeTransfer/cr_zip_tricks/actions/workflows/crystal.yml/badge.svg)](https://github.com/WeTransfer/cr_zip_tricks/actions/workflows/crystal.yml) 4 | 5 | An alternate ZIP writer for Crystal, ported from [zip_tricks for Ruby](https://github.com/WeTransfer/zip_tricks) 6 | 7 | ## Installation 8 | 9 | 1. Add the dependency to your `shard.yml`: 10 | ```yaml 11 | dependencies: 12 | cr_zip_tricks: 13 | github: WeTransfer/cr_zip_tricks 14 | ``` 15 | 2. Run `shards install` 16 | 17 | ## Usage 18 | 19 | Archiving to any IO: 20 | 21 | ```crystal 22 | require "zip_tricks" 23 | 24 | ZipTricks::Streamer.archive(STDOUT) do |s| 25 | s.add_deflated("deflated.txt") do |sink| 26 | sink << "Hello stranger! This is a chunk of text that is going to compress. Well." 27 | end 28 | 29 | s.add_stored("stored.txt") do |sink| 30 | sink << "Goodbye stranger!" 31 | end 32 | end 33 | 34 | ``` 35 | 36 | Sizing an archive before creation, to the byte: 37 | 38 | ```crystal 39 | require "cr_zip_tricks" 40 | 41 | size = ZipTricks::Sizer.size do |s| 42 | s.predeclare_entry(filename: "deflated1.txt", uncompressed_size: 8969887, compressed_size: 1245, use_data_descriptor: true) 43 | s.predeclare_entry(filename: "deflated2.txt", uncompressed_size: 4568, compressed_size: 4065, use_data_descriptor: true) 44 | end 45 | size #=> 5641 46 | ``` 47 | 48 | ## Using it with Kemal or other web app skeleton 49 | 50 | Here is a Kemal app that outputs itself, 1000 times, compressed: 51 | 52 | ```crystal 53 | require "kemal" 54 | require "cr_zip_tricks" 55 | 56 | get "/quine.zip" do |env| 57 | env.response.headers["Content-Type"] = "application/octet-stream" 58 | env.response.headers["Content-Disposition"] = "attachment" 59 | ZipTricks::Streamer.archive(env.response) do |s| 60 | 1000.times do |i| 61 | s.add_deflated("cr_download_server_%05d.cr" % i) do |sink| 62 | File.open(__FILE__, "rb") do |f| 63 | IO.copy(f, sink) 64 | end 65 | end 66 | end 67 | end 68 | end 69 | 70 | Kemal.run 71 | ``` 72 | 73 | ## Development 74 | 75 | TODO: Write development instructions here 76 | 77 | ## Contributing 78 | 79 | 1. Fork it () 80 | 2. Create your feature branch (`git checkout -b my-new-feature`) 81 | 3. Commit your changes (`git commit -am 'Add some feature'`) 82 | 4. Push to the branch (`git push origin my-new-feature`) 83 | 5. Create a new Pull Request 84 | 85 | ## Contributors 86 | 87 | - [Julik Tarkhanov](https://github.com/julik) - creator and maintainer 88 | -------------------------------------------------------------------------------- /shard.yml: -------------------------------------------------------------------------------- 1 | name: cr_zip_tricks 2 | version: 0.3.0 3 | authors: ["Julik Tarkhanov "] 4 | description: "An alternate ZIP writer for Crystal, ported from zip_tricks for Ruby" 5 | crystal: ">= 0.35.1" 6 | license: MIT 7 | -------------------------------------------------------------------------------- /spec/crzt_spec.cr: -------------------------------------------------------------------------------- 1 | require "./spec_helper" 2 | 3 | describe ZipTricks do 4 | it "works" do 5 | true.should eq(true) 6 | end 7 | end 8 | -------------------------------------------------------------------------------- /spec/sizer_spec.cr: -------------------------------------------------------------------------------- 1 | require "./spec_helper" 2 | 3 | describe ZipTricks::Sizer do 4 | describe ".size" do 5 | it "sizes the archive with all sorts of entries" do 6 | size = ZipTricks::Sizer.size do |s| 7 | s.predeclare_entry(filename: "deflated1.txt", uncompressed_size: 8969887, compressed_size: 1245, use_data_descriptor: true) 8 | s.predeclare_entry(filename: "deflated12.txt", uncompressed_size: 4568, compressed_size: 4065, use_data_descriptor: true) 9 | end 10 | size.should eq(5641) 11 | end 12 | 13 | it "sizes an empty archive" do 14 | size = ZipTricks::Sizer.size do |s| 15 | end 16 | size.should eq(57) 17 | end 18 | end 19 | end 20 | -------------------------------------------------------------------------------- /spec/spec_helper.cr: -------------------------------------------------------------------------------- 1 | require "spec" 2 | require "../src/cr_zip_tricks" 3 | -------------------------------------------------------------------------------- /spec/writer_spec.cr: -------------------------------------------------------------------------------- 1 | require "./spec_helper" 2 | 3 | class ByteReader 4 | def initialize(io : IO) 5 | @io = io 6 | @io.rewind 7 | end 8 | 9 | def read_1b 10 | slice = read_n(1) 11 | slice[0] 12 | end 13 | 14 | def read_2b 15 | @io.read_bytes(UInt16, format = IO::ByteFormat::LittleEndian) 16 | end 17 | 18 | def read_2c 19 | # reads a binary string of 2 bytes 20 | @io.read_bytes(UInt16, format = IO::ByteFormat::LittleEndian) 21 | end 22 | 23 | def read_4b 24 | @io.read_bytes(UInt32, format = IO::ByteFormat::LittleEndian) 25 | end 26 | 27 | def read_8b 28 | @io.read_bytes(UInt64, format = IO::ByteFormat::LittleEndian) 29 | end 30 | 31 | def read_4b_signed 32 | @io.read_bytes(Int32, format = IO::ByteFormat::LittleEndian) 33 | end 34 | 35 | def read_string_of(n) 36 | @io.read_string(bytesize: n) 37 | end 38 | 39 | def read_n(n) 40 | slice = Bytes.new(n) 41 | @io.read_fully(slice) 42 | slice 43 | end 44 | end 45 | 46 | describe ZipTricks::Writer do 47 | describe "#write_local_file_header" do 48 | it "writes the local file header for an entry that does not require Zip64" do 49 | buf = IO::Memory.new 50 | mtime = Time.utc(2016, 7, 17, 13, 48) 51 | 52 | ZipTricks::Writer.new.write_local_file_header(io: buf, 53 | filename: "foo.bin", 54 | compressed_size: 768, 55 | uncompressed_size: 901, 56 | crc32: 456, 57 | gp_flags: 12, 58 | mtime: mtime, 59 | storage_mode: 8) 60 | 61 | br = ByteReader.new(buf) 62 | br.read_4b.should eq(0x04034b50) # Signature 63 | br.read_2b.should eq(20) # Version needed to extract 64 | br.read_2b.should eq(12) # gp flags 65 | br.read_2b.should eq(8) # storage mode 66 | br.read_2b.should eq(28_160) # DOS time 67 | br.read_2b.should eq(18_673) # DOS date 68 | br.read_4b.should eq(456) # CRC32 69 | br.read_4b.should eq(768) # compressed size 70 | br.read_4b.should eq(901) # uncompressed size 71 | br.read_2b.should eq(7) # filename size 72 | br.read_2b.should eq(9) # extra fields size 73 | 74 | br.read_string_of(7).should eq("foo.bin") # extra fields size 75 | 76 | br.read_2b.should eq(0x5455) # Extended timestamp extra tag 77 | br.read_2b.should eq(5) # Size of the timestamp extra 78 | br.read_1b.should eq(128) # The timestamp flag 79 | 80 | ext_mtime = br.read_4b_signed 81 | ext_mtime.should eq(1_468_763_280) # The mtime encoded as a 4byte uint 82 | 83 | parsed_time = Time.unix(ext_mtime) 84 | parsed_time.year.should eq(2_016) 85 | end 86 | 87 | it "writes the local file header for an entry that does require Zip64 based \ 88 | on uncompressed size (with the Zip64 extra)" do 89 | buf = IO::Memory.new 90 | mtime = Time.utc(2016, 7, 17, 13, 48) 91 | 92 | ZipTricks::Writer.new.write_local_file_header(io: buf, 93 | filename: "foo.bin", 94 | gp_flags: 12, 95 | crc32: 456, 96 | compressed_size: 768, 97 | uncompressed_size: (0xFFFFFFFF + 1), 98 | mtime: mtime, 99 | storage_mode: 8) 100 | 101 | br = ByteReader.new(buf) 102 | br.read_4b.should eq(0x04034b50) # Signature 103 | br.read_2b.should eq(45) # Version needed to extract 104 | br.read_2b.should eq(12) # gp flags 105 | br.read_2b.should eq(8) # storage mode 106 | br.read_2b.should eq(28_160) # DOS time 107 | br.read_2b.should eq(18_673) # DOS date 108 | br.read_4b.should eq(456) # CRC32 109 | br.read_4b.should eq(0xFFFFFFFF) # compressed size 110 | br.read_4b.should eq(0xFFFFFFFF) # uncompressed size 111 | br.read_2b.should eq(7) # filename size 112 | br.read_2b.should eq(29) # extra fields size (Zip64 + extended timestamp) 113 | br.read_string_of(7).should eq("foo.bin") # extra fields size 114 | 115 | # buf.should_not be_eof 116 | 117 | br.read_2b.should eq(1) # Zip64 extra tag 118 | br.read_2b.should eq(16) # Size of the Zip64 extra payload 119 | br.read_8b.should eq(0xFFFFFFFF + 1) # uncompressed size 120 | br.read_8b.should eq(768) # compressed size 121 | end 122 | 123 | it "writes the local file header for an entry that does require Zip64 based \ 124 | on compressed size (with the Zip64 extra)" do 125 | buf = IO::Memory.new 126 | mtime = Time.utc(2016, 7, 17, 13, 48) 127 | 128 | ZipTricks::Writer.new.write_local_file_header(io: buf, 129 | gp_flags: 12, 130 | crc32: 456, 131 | compressed_size: 0xFFFFFFFF + 1, 132 | uncompressed_size: 768, 133 | mtime: mtime, 134 | filename: "foo.bin", 135 | storage_mode: 8) 136 | 137 | br = ByteReader.new(buf) 138 | br.read_4b.should eq(0x04034b50) # Signature 139 | br.read_2b.should eq(45) # Version needed to extract 140 | br.read_2b.should eq(12) # gp flags 141 | br.read_2b.should eq(8) # storage mode 142 | br.read_2b.should eq(28_160) # DOS time 143 | br.read_2b.should eq(18_673) # DOS date 144 | br.read_4b.should eq(456) # CRC32 145 | br.read_4b.should eq(0xFFFFFFFF) # compressed size 146 | br.read_4b.should eq(0xFFFFFFFF) # uncompressed size 147 | br.read_2b.should eq(7) # filename size 148 | br.read_2b.should eq(29) # extra fields size 149 | br.read_string_of(7).should eq("foo.bin") # extra fields size 150 | 151 | # buf.should_not be_eof 152 | 153 | br.read_2b.should eq(1) # Zip64 extra tag 154 | br.read_2b.should eq(16) # Size of the Zip64 extra payload 155 | br.read_8b.should eq(768) # uncompressed size 156 | br.read_8b.should eq(0xFFFFFFFF + 1) # compressed size 157 | end 158 | end 159 | 160 | describe "#write_data_descriptor" do 161 | it "writes 4-byte sizes into the data descriptor for standard file sizes" do 162 | buf = IO::Memory.new 163 | 164 | ZipTricks::Writer.new.write_data_descriptor(io: buf, crc32: 123, compressed_size: 89_821, uncompressed_size: 990_912) 165 | 166 | br = ByteReader.new(buf) 167 | br.read_4b.should eq(0x08074b50) # Signature 168 | br.read_4b.should eq(123) # CRC32 169 | br.read_4b.should eq(89_821) # compressed size 170 | br.read_4b.should eq(990_912) # uncompressed size 171 | # buf.should be_eof 172 | end 173 | 174 | it "writes 8-byte sizes into the data descriptor for Zip64 compressed file size" do 175 | buf = IO::Memory.new 176 | 177 | ZipTricks::Writer.new.write_data_descriptor(io: buf, 178 | crc32: 123, 179 | compressed_size: (0xFFFFFFFF + 1), 180 | uncompressed_size: 990_912) 181 | 182 | br = ByteReader.new(buf) 183 | br.read_4b.should eq(0x08074b50) # Signature 184 | br.read_4b.should eq(123) # CRC32 185 | br.read_8b.should eq(0xFFFFFFFF + 1) # compressed size 186 | br.read_8b.should eq(990_912) # uncompressed size 187 | # buf.should be_eof 188 | end 189 | 190 | it "writes 8-byte sizes into the data descriptor for Zip64 uncompressed file size" do 191 | buf = IO::Memory.new 192 | 193 | ZipTricks::Writer.new.write_data_descriptor(io: buf, 194 | crc32: 123, 195 | compressed_size: 123, 196 | uncompressed_size: 0xFFFFFFFF + 1) 197 | 198 | br = ByteReader.new(buf) 199 | br.read_4b.should eq(0x08074b50) # Signature 200 | br.read_4b.should eq(123) # CRC32 201 | br.read_8b.should eq(123) # compressed size 202 | br.read_8b.should eq(0xFFFFFFFF + 1) # uncompressed size 203 | # buf.should be_eof 204 | end 205 | end 206 | 207 | describe "#write_central_directory_file_header" do 208 | it "writes the file header for a small-ish entry" do 209 | buf = IO::Memory.new 210 | 211 | ZipTricks::Writer.new.write_central_directory_file_header(io: buf, 212 | local_file_header_location: 898_921, 213 | gp_flags: 555, 214 | storage_mode: 23, 215 | compressed_size: 901, 216 | uncompressed_size: 909_102, 217 | mtime: Time.utc(2016, 2, 2, 14, 0), 218 | crc32: 89_765, 219 | filename: "a-file.txt") 220 | 221 | br = ByteReader.new(buf) 222 | br.read_4b.should eq(0x02014b50) # Central directory entry sig 223 | br.read_2b.should eq(820) # version made by 224 | br.read_2b.should eq(20) # version need to extract 225 | br.read_2b.should eq(555) # general purpose bit flag (explicitly 226 | # set to bogus value to ensure we pass it through) 227 | br.read_2b.should eq(23) # compression method (explicitly set to bogus value) 228 | br.read_2b.should eq(28_672) # last mod file time 229 | br.read_2b.should eq(18_498) # last mod file date 230 | br.read_4b.should eq(89_765) # crc32 231 | br.read_4b.should eq(901) # compressed size 232 | br.read_4b.should eq(909_102) # uncompressed size 233 | br.read_2b.should eq(10) # filename length 234 | br.read_2b.should eq(9) # extra field length 235 | br.read_2b.should eq(0) # file comment 236 | br.read_2b.should eq(0) # disk number, must be blanked to the 237 | # maximum value because of The Unarchiver bug 238 | br.read_2b.should eq(0) # internal file attributes 239 | br.read_4b.should eq(2_175_008_768) # external file attributes 240 | br.read_4b.should eq(898_921) # relative offset of local header 241 | br.read_string_of(10).should eq("a-file.txt") # the filename 242 | end 243 | 244 | it "writes the file header for an entry that contains an empty directory" do 245 | buf = IO::Memory.new 246 | 247 | ZipTricks::Writer.new.write_central_directory_file_header(io: buf, 248 | local_file_header_location: 898_921, 249 | gp_flags: 555, 250 | storage_mode: 23, 251 | compressed_size: 0, 252 | uncompressed_size: 0, 253 | mtime: Time.utc(2016, 2, 2, 14, 0), 254 | crc32: 544, 255 | filename: "this-is-here-directory/") 256 | 257 | br = ByteReader.new(buf) 258 | br.read_4b.should eq(0x02014b50) # Central directory entry sig 259 | br.read_2b.should eq(820) # version made by 260 | br.read_2b.should eq(20) # version need to extract 261 | br.read_2b.should eq(555) # general purpose bit flag (explicitly 262 | # set to bogus value to ensure we pass it through) 263 | br.read_2b.should eq(23) # compression method (explicitly set to bogus value) 264 | br.read_2b.should eq(28_672) # last mod file time 265 | br.read_2b.should eq(18_498) # last mod file date 266 | br.read_4b.should eq(544) # crc32 267 | br.read_4b.should eq(0) # compressed size 268 | br.read_4b.should eq(0) # uncompressed size 269 | br.read_2b.should eq(23) # filename length 270 | br.read_2b.should eq(9) # extra field length 271 | br.read_2b.should eq(0) # file comment 272 | br.read_2b.should eq(0) # disk number (0, first disk) 273 | br.read_2b.should eq(0) # internal file attributes 274 | br.read_4b.should eq(1_106_051_072) # external file attributes 275 | br.read_4b.should eq(898_921) # relative offset of local header 276 | br.read_string_of(23).should eq("this-is-here-directory/") # the filename 277 | end 278 | 279 | it "writes the file header for an entry that requires Zip64 extra because of \ 280 | the uncompressed size" do 281 | buf = IO::Memory.new 282 | 283 | ZipTricks::Writer.new.write_central_directory_file_header(io: buf, 284 | local_file_header_location: 898_921, 285 | gp_flags: 555, 286 | storage_mode: 23, 287 | compressed_size: 901, 288 | uncompressed_size: 0xFFFFFFFFF + 3, 289 | mtime: Time.utc(2016, 2, 2, 14, 0), 290 | crc32: 89_765, 291 | filename: "a-file.txt") 292 | 293 | br = ByteReader.new(buf) 294 | br.read_4b.should eq(0x02014b50) # Central directory entry sig 295 | br.read_2b.should eq(820) # version made by 296 | br.read_2b.should eq(45) # version need to extract 297 | br.read_2b.should eq(555) # general purpose bit flag 298 | # (explicitly set to bogus value 299 | # to ensure we pass it through) 300 | br.read_2b.should eq(23) # compression method (explicitly 301 | # set to bogus value) 302 | br.read_2b.should eq(28_672) # last mod file time 303 | br.read_2b.should eq(18_498) # last mod file date 304 | br.read_4b.should eq(89_765) # crc32 305 | br.read_4b.should eq(0xFFFFFFFF) # compressed size 306 | br.read_4b.should eq(0xFFFFFFFF) # uncompressed size 307 | br.read_2b.should eq(10) # filename length 308 | br.read_2b.should eq(41) # extra field length 309 | br.read_2b.should eq(0) # file comment 310 | br.read_2b.should eq(0xFFFF) # disk number, must be blanked to the maximum value 311 | br.read_2b.should eq(0) # internal file attributes 312 | br.read_4b.should eq(2_175_008_768) # external file attributes 313 | br.read_4b.should eq(0xFFFFFFFF) # relative offset of local header 314 | br.read_string_of(10).should eq("a-file.txt") # the filename 315 | 316 | br.read_2b.should eq(1) # Zip64 extra tag 317 | br.read_2b.should eq(28) # Size of the Zip64 extra payload 318 | br.read_8b.should eq(0xFFFFFFFFF + 3) # uncompressed size 319 | br.read_8b.should eq(901) # compressed size 320 | br.read_8b.should eq(898_921) # local file header location 321 | end 322 | 323 | it "writes the file header for an entry that requires Zip64 extra because of \ 324 | the compressed size" do 325 | buf = IO::Memory.new 326 | 327 | ZipTricks::Writer.new.write_central_directory_file_header(io: buf, 328 | local_file_header_location: 898_921, 329 | gp_flags: 555, 330 | storage_mode: 23, 331 | compressed_size: 0xFFFFFFFFF + 3, 332 | # the worst compression scheme in the universe 333 | uncompressed_size: 901, 334 | mtime: Time.utc(2016, 2, 2, 14, 0), 335 | crc32: 89_765, 336 | filename: "a-file.txt") 337 | 338 | br = ByteReader.new(buf) 339 | br.read_4b.should eq(0x02014b50) # Central directory entry sig 340 | br.read_2b.should eq(820) # version made by 341 | br.read_2b.should eq(45) # version need to extract 342 | br.read_2b.should eq(555) # general purpose bit flag (explicitly 343 | # set to bogus value to ensure we pass it through) 344 | br.read_2b.should eq(23) # compression method (explicitly set to bogus value) 345 | br.read_2b.should eq(28_672) # last mod file time 346 | br.read_2b.should eq(18_498) # last mod file date 347 | br.read_4b.should eq(89_765) # crc32 348 | br.read_4b.should eq(0xFFFFFFFF) # compressed size 349 | br.read_4b.should eq(0xFFFFFFFF) # uncompressed size 350 | br.read_2b.should eq(10) # filename length 351 | br.read_2b.should eq(41) # extra field length 352 | br.read_2b.should eq(0) # file comment 353 | br.read_2b.should eq(0xFFFF) # disk number, must be blanked to the 354 | # maximum value because of The Unarchiver bug 355 | br.read_2b.should eq(0) # internal file attributes 356 | br.read_4b.should eq(2_175_008_768) # external file attributes 357 | br.read_4b.should eq(0xFFFFFFFF) # relative offset of local header 358 | br.read_string_of(10).should eq("a-file.txt") # the filename 359 | 360 | # buf.should_not be_eof 361 | br.read_2b.should eq(1) # Zip64 extra tag 362 | br.read_2b.should eq(28) # Size of the Zip64 extra payload 363 | br.read_8b.should eq(901) # uncompressed size 364 | br.read_8b.should eq(0xFFFFFFFFF + 3) # compressed size 365 | br.read_8b.should eq(898_921) # local file header location 366 | end 367 | 368 | it "writes the file header for an entry that requires Zip64 extra because of \ 369 | the local file header offset being beyound 4GB" do 370 | buf = IO::Memory.new 371 | 372 | ZipTricks::Writer.new.write_central_directory_file_header(io: buf, 373 | local_file_header_location: 0xFFFFFFFFF + 1, 374 | gp_flags: 555, 375 | storage_mode: 23, 376 | compressed_size: 8_981, 377 | # the worst compression scheme in the universe 378 | uncompressed_size: 819_891, 379 | mtime: Time.utc(2016, 2, 2, 14, 0), 380 | crc32: 89_765, 381 | filename: "a-file.txt") 382 | 383 | br = ByteReader.new(buf) 384 | br.read_4b.should eq(0x02014b50) # Central directory entry sig 385 | br.read_2b.should eq(820) # version made by 386 | br.read_2b.should eq(45) # version need to extract 387 | br.read_2b.should eq(555) # general purpose bit flag (explicitly 388 | # set to bogus value to ensure we pass it through) 389 | br.read_2b.should eq(23) # compression method (explicitly set to bogus value) 390 | br.read_2b.should eq(28_672) # last mod file time 391 | br.read_2b.should eq(18_498) # last mod file date 392 | br.read_4b.should eq(89_765) # crc32 393 | br.read_4b.should eq(0xFFFFFFFF) # compressed size 394 | br.read_4b.should eq(0xFFFFFFFF) # uncompressed size 395 | br.read_2b.should eq(10) # filename length 396 | br.read_2b.should eq(41) # extra field length 397 | br.read_2b.should eq(0) # file comment 398 | br.read_2b.should eq(0xFFFF) # disk number, must be blanked to the 399 | # maximum value because of The Unarchiver bug 400 | br.read_2b.should eq(0) # internal file attributes 401 | br.read_4b.should eq(2_175_008_768) # external file attributes 402 | br.read_4b.should eq(0xFFFFFFFF) # relative offset of local header 403 | br.read_string_of(10).should eq("a-file.txt") # the filename 404 | 405 | # buf.should_not be_eof 406 | br.read_2b.should eq(1) # Zip64 extra tag 407 | br.read_2b.should eq(28) # Size of the Zip64 extra payload 408 | br.read_8b.should eq(819_891) # uncompressed size 409 | br.read_8b.should eq(8_981) # compressed size 410 | br.read_8b.should eq(0xFFFFFFFFF + 1) # local file header location 411 | end 412 | end 413 | 414 | describe "#write_end_of_central_directory" do 415 | it "writes out the EOCD with all markers for a small ZIP file with just a few entries" do 416 | buf = IO::Memory.new 417 | 418 | num_files = rand(8..190) 419 | ZipTricks::Writer.new.write_end_of_central_directory(io: buf, 420 | start_of_central_directory_location: 9_091_211, 421 | central_directory_size: 9_091, 422 | num_files_in_archive: num_files, comment: "xyz") 423 | 424 | br = ByteReader.new(buf) 425 | br.read_4b.should eq(0x06054b50) # EOCD signature 426 | br.read_2b.should eq(0) # number of this disk 427 | br.read_2b.should eq(0) # number of the disk with the EOCD record 428 | br.read_2b.should eq(num_files) # number of files on this disk 429 | br.read_2b.should eq(num_files) # number of files in central directory 430 | # total (for all disks) 431 | br.read_4b.should eq(9_091) # size of the central directory (cdir records for all files) 432 | br.read_4b.should eq(9_091_211) # start of central directory offset from 433 | # the beginning of file/disk 434 | 435 | comment_length = br.read_2b 436 | comment_length.should eq(3) 437 | 438 | br.read_string_of(comment_length).should match(/xyz/) 439 | end 440 | 441 | it "writes out the custom comment" do 442 | buf = IO::Memory.new 443 | comment = "Ohai mate" 444 | ZipTricks::Writer.new.write_end_of_central_directory(io: buf, 445 | start_of_central_directory_location: 9_091_211, 446 | central_directory_size: 9_091, 447 | num_files_in_archive: 4, 448 | comment: comment) 449 | # 450 | # size_and_comment = buf[((comment.bytesize + 2) * -1)..-1] 451 | # comment_size = size_and_comment.unpack("v")[0] 452 | # comment_size.should eq(comment.bytesize) 453 | end 454 | 455 | it "writes out the Zip64 EOCD as well if the central directory is located \ 456 | beyound 4GB in the archive" do 457 | buf = IO::Memory.new 458 | 459 | num_files = rand(8..190) 460 | ZipTricks::Writer.new.write_end_of_central_directory(io: buf, 461 | start_of_central_directory_location: 0xFFFFFFFF + 3, 462 | central_directory_size: 9091, 463 | num_files_in_archive: num_files) 464 | 465 | br = ByteReader.new(buf) 466 | 467 | br.read_4b.should eq(0x06064b50) # Zip64 EOCD signature 468 | br.read_8b.should eq(44) # Zip64 EOCD record size 469 | br.read_2b.should eq(820) # Version made by 470 | br.read_2b.should eq(45) # Version needed to extract 471 | br.read_4b.should eq(0) # Number of this disk 472 | br.read_4b.should eq(0) # Number of the disk with the Zip64 EOCD record 473 | br.read_8b.should eq(num_files) # Number of entries in the central 474 | # directory of this disk 475 | br.read_8b.should eq(num_files) # Number of entries in the central 476 | # directories of all disks 477 | br.read_8b.should eq(9_091) # Central directory size 478 | br.read_8b.should eq(0xFFFFFFFF + 3) # Start of central directory location 479 | 480 | br.read_4b.should eq(0x07064b50) # Zip64 EOCD locator signature 481 | br.read_4b.should eq(0) # Number of the disk with the EOCD locator signature 482 | br.read_8b.should eq((0xFFFFFFFF + 3) + 9_091) # Where the Zip64 EOCD record starts 483 | br.read_4b.should eq(1) # Total number of disks 484 | 485 | # Then the usual EOCD record 486 | br.read_4b.should eq(0x06054b50) # EOCD signature 487 | br.read_2b.should eq(0) # number of this disk 488 | br.read_2b.should eq(0) # number of the disk with the EOCD record 489 | br.read_2b.should eq(0xFFFF) # number of files on this disk 490 | br.read_2b.should eq(0xFFFF) # number of files in central directory 491 | # total (for all disks) 492 | br.read_4b.should eq(0xFFFFFFFF) # size of the central directory 493 | # (cdir records for all files) 494 | br.read_4b.should eq(0xFFFFFFFF) # start of central directory offset 495 | # from the beginning of file/disk 496 | 497 | comment_length = br.read_2b 498 | comment_length.should_not eq(0) 499 | 500 | br.read_string_of(comment_length).should match(/zip_tricks/i) 501 | end 502 | 503 | it "writes out the Zip64 EOCD if the archive has more than 0xFFFF files" do 504 | buf = IO::Memory.new 505 | 506 | ZipTricks::Writer.new.write_end_of_central_directory(io: buf, 507 | start_of_central_directory_location: 123, 508 | central_directory_size: 9_091, 509 | num_files_in_archive: 0xFFFF + 1, comment: "") 510 | 511 | br = ByteReader.new(buf) 512 | 513 | br.read_4b.should eq(0x06064b50) # Zip64 EOCD signature 514 | br.read_8b 515 | br.read_2b 516 | br.read_2b 517 | br.read_4b 518 | br.read_4b 519 | br.read_8b.should eq(0xFFFF + 1) # Number of entries in the central 520 | # directory of this disk 521 | br.read_8b.should eq(0xFFFF + 1) # Number of entries in the central 522 | # directories of all disks 523 | end 524 | 525 | it "writes out the Zip64 EOCD if the central directory size exceeds 0xFFFFFFFF" do 526 | buf = IO::Memory.new 527 | 528 | ZipTricks::Writer.new.write_end_of_central_directory(io: buf, 529 | start_of_central_directory_location: 123, 530 | central_directory_size: 0xFFFFFFFF + 2, 531 | num_files_in_archive: 5, comment: "Foooo") 532 | 533 | br = ByteReader.new(buf) 534 | 535 | br.read_4b.should eq(0x06064b50) # Zip64 EOCD signature 536 | br.read_8b 537 | br.read_2b 538 | br.read_2b 539 | br.read_4b 540 | br.read_4b 541 | br.read_8b.should eq(5) # Number of entries in the central directory of this disk 542 | br.read_8b.should eq(5) # Number of entries in the central directories of all disks 543 | end 544 | end 545 | end 546 | -------------------------------------------------------------------------------- /src/cr_zip_tricks.cr: -------------------------------------------------------------------------------- 1 | module ZipTricks 2 | end 3 | 4 | require "./version" 5 | require "./streamer" 6 | require "./sizer" 7 | require "./writer" 8 | -------------------------------------------------------------------------------- /src/crc32_writer.cr: -------------------------------------------------------------------------------- 1 | require "digest/crc32" 2 | 3 | class ZipTricks::CRC32Writer < IO 4 | getter count = 0_u32 5 | getter crc32 = Digest::CRC32.initial 6 | getter io : IO 7 | 8 | def initialize(io : IO) 9 | @io = io 10 | end 11 | 12 | # Does nothing but must be implemented since Crystal does not differentiate 13 | # between Writesr/Readers 14 | def read(slice : Bytes) 15 | raise IO::Error.new "Can't read from CRC32Writer" 16 | end 17 | 18 | def write(slice : Bytes) : Nil 19 | return if slice.empty? 20 | @crc32 = Digest::CRC32.update(slice, @crc32) 21 | @io.write(slice) 22 | nil 23 | end 24 | end 25 | -------------------------------------------------------------------------------- /src/offset_io.cr: -------------------------------------------------------------------------------- 1 | class ZipTricks::OffsetIO < IO 2 | def initialize(any_io : IO) 3 | @io = any_io 4 | @offset = 0_u64 5 | end 6 | 7 | def offset 8 | @offset 9 | end 10 | 11 | def advance(by : Int) 12 | @offset += by 13 | end 14 | 15 | # Does nothing but must be implemented since Crystal does not differentiate 16 | # between Writesr/Readers 17 | def read(slice : Bytes) 18 | raise IO::Error.new "Can't read from CRC32Writer" 19 | end 20 | 21 | def write(slice : Bytes) : Nil 22 | @io.write(slice) 23 | @offset += slice.size 24 | nil 25 | end 26 | end 27 | -------------------------------------------------------------------------------- /src/sizer.cr: -------------------------------------------------------------------------------- 1 | require "./streamer" 2 | 3 | class ZipTricks::Sizer 4 | private class NullIO < IO 5 | def read(slice : Bytes) 6 | raise IO::Error.new "Can't read from NullIO" 7 | end 8 | 9 | def write(slice : Bytes) : Nil 10 | nil 11 | end 12 | end 13 | 14 | def self.size 15 | streamer = ZipTricks::Streamer.new(NullIO.new) 16 | sizer = new(streamer) 17 | 18 | yield(sizer) 19 | 20 | streamer.finish 21 | streamer.bytesize 22 | end 23 | 24 | def initialize(streamer : ZipTricks::Streamer) 25 | @streamer = streamer 26 | end 27 | 28 | def predeclare_entry(filename : String, uncompressed_size : Int, compressed_size : Int, use_data_descriptor : Bool = false) 29 | @streamer.predeclare_entry(filename: filename, 30 | uncompressed_size: uncompressed_size, 31 | compressed_size: compressed_size, 32 | use_data_descriptor: use_data_descriptor, 33 | crc32: 0, 34 | storage_mode: 0) 35 | @streamer.advance(compressed_size) 36 | if use_data_descriptor 37 | @streamer.write_data_descriptor_for_last_entry 38 | end 39 | end 40 | end 41 | -------------------------------------------------------------------------------- /src/streamer.cr: -------------------------------------------------------------------------------- 1 | require "./writer" 2 | require "./offset_io" 3 | require "./crc32_writer" 4 | require "compress/deflate" 5 | 6 | class ZipTricks::Streamer 7 | STORED = 0 8 | DEFLATED = 8 9 | 10 | class DuplicateFilename < ArgumentError 11 | end 12 | 13 | class Entry 14 | property filename = "" 15 | property entry_offset_in_file = 0_u64 16 | property crc32 = Digest::CRC32.initial 17 | property uncompressed_size = 0_u64 18 | property compressed_size = 0_u64 19 | property use_data_descriptor = false 20 | property storage_mode = 0 # Stored 21 | 22 | # Get the general purpose flags for the entry. We care about is the EFS 23 | # bit (bit 11) which should be set if the filename is UTF8. If it is, we need to set the 24 | # bit so that the unarchiving application knows that the filename in the archive is UTF-8 25 | # encoded, and not some DOS default. For ASCII entries it does not matter. 26 | # Additionally, we care about bit 3 which toggles the use of the postfix data descriptor. 27 | 28 | def gp_flags 29 | flag = 0b00000000000 30 | flag |= 0b100000000000 # if @requires_efs_flag # bit 11 31 | flag |= 0x0008 if @use_data_descriptor # bit 3 32 | flag 33 | end 34 | end 35 | 36 | def initialize(io : IO) 37 | @raw_io = io 38 | @io = ZipTricks::OffsetIO.new(@raw_io) 39 | @filenames = Set(String).new 40 | @entries = Array(Entry).new 41 | @writer = ZipTricks::Writer.new 42 | end 43 | 44 | def self.archive(io : IO) 45 | streamer = new(io) 46 | yield streamer 47 | streamer.finish 48 | end 49 | 50 | def finish 51 | write_central_directory 52 | @filenames.clear 53 | @entries.clear 54 | end 55 | 56 | def predeclare_entry(filename : String, uncompressed_size : Int, compressed_size : Int, crc32 : Int, storage_mode : Int, use_data_descriptor : Bool = false) 57 | entry = Entry.new 58 | entry.filename = filename 59 | entry.use_data_descriptor = false 60 | entry.storage_mode = storage_mode 61 | entry.entry_offset_in_file = @io.offset 62 | entry.use_data_descriptor = use_data_descriptor 63 | entry.uncompressed_size = uncompressed_size.to_u64 64 | entry.compressed_size = uncompressed_size.to_u64 65 | entry.crc32 = crc32.to_u32 66 | 67 | check_dupe_filename!(filename) 68 | @entries << entry 69 | write_local_entry_header(entry) 70 | end 71 | 72 | def add_stored(filename : String) 73 | predeclare_entry(filename, uncompressed_size: 0, compressed_size: 0, crc32: 0, storage_mode: STORED, use_data_descriptor: true) 74 | sizer = ZipTricks::OffsetIO.new(@io) 75 | checksum = ZipTricks::CRC32Writer.new(sizer) 76 | 77 | yield checksum # for writing, the caller can write to it as an IO 78 | 79 | last_entry = @entries.last 80 | last_entry.uncompressed_size = sizer.offset 81 | last_entry.compressed_size = sizer.offset 82 | last_entry.crc32 = checksum.crc32 83 | write_data_descriptor_for_last_entry 84 | end 85 | 86 | def add_deflated(filename : String) 87 | predeclare_entry(filename, uncompressed_size: 0, compressed_size: 0, crc32: 0, storage_mode: DEFLATED, use_data_descriptor: true) 88 | # The "IO sandwich" 89 | compressed_sizer = ZipTricks::OffsetIO.new(@io) 90 | flater_io = Compress::Deflate::Writer.new(compressed_sizer) 91 | uncompressed_sizer = ZipTricks::OffsetIO.new(flater_io) 92 | checksum = ZipTricks::CRC32Writer.new(uncompressed_sizer) 93 | 94 | yield checksum # for writing, the caller can write to it as an IO 95 | 96 | flater_io.close # To finish generating the deflated block 97 | last_entry = @entries.last 98 | last_entry.uncompressed_size = uncompressed_sizer.offset 99 | last_entry.compressed_size = compressed_sizer.offset 100 | last_entry.crc32 = checksum.crc32 101 | write_data_descriptor_for_last_entry 102 | end 103 | 104 | def write_data_descriptor_for_last_entry 105 | entry = @entries[-1] 106 | @writer.write_data_descriptor(io: @io, 107 | compressed_size: entry.compressed_size, 108 | uncompressed_size: entry.uncompressed_size, 109 | crc32: entry.crc32) 110 | end 111 | 112 | def write_local_entry_header(entry) 113 | @writer.write_local_file_header(io: @io, 114 | filename: entry.filename, 115 | compressed_size: entry.compressed_size, 116 | uncompressed_size: entry.uncompressed_size, 117 | crc32: entry.crc32, 118 | gp_flags: entry.gp_flags, 119 | mtime: Time.utc, 120 | storage_mode: entry.storage_mode) 121 | end 122 | 123 | def advance(by) 124 | @io.advance(by) 125 | end 126 | 127 | def bytesize 128 | @io.offset 129 | end 130 | 131 | def write_central_directory 132 | cdir_starts_at = @io.offset 133 | @entries.each do |entry| 134 | @writer.write_central_directory_file_header(io: @io, 135 | filename: entry.filename, 136 | compressed_size: entry.compressed_size, 137 | uncompressed_size: entry.uncompressed_size, 138 | crc32: entry.crc32, 139 | gp_flags: entry.gp_flags, 140 | mtime: Time.utc, 141 | storage_mode: entry.storage_mode, 142 | local_file_header_location: entry.entry_offset_in_file) 143 | end 144 | cdir_ends_at = @io.offset 145 | cdir_size = cdir_ends_at - cdir_starts_at 146 | @writer.write_end_of_central_directory(io: @io, 147 | start_of_central_directory_location: cdir_starts_at, 148 | central_directory_size: @io.offset - cdir_starts_at, 149 | num_files_in_archive: @entries.size) 150 | end 151 | 152 | private def check_dupe_filename!(filename) 153 | if @filenames.includes?(filename) 154 | raise(DuplicateFilename.new("The archive already contains an entry named #{filename.inspect}")) 155 | else 156 | @filenames.add(filename) 157 | end 158 | end 159 | end 160 | -------------------------------------------------------------------------------- /src/version.cr: -------------------------------------------------------------------------------- 1 | module ZipTricks 2 | VERSION = "0.3.0" 3 | end 4 | -------------------------------------------------------------------------------- /src/writer.cr: -------------------------------------------------------------------------------- 1 | require "./version" 2 | 3 | class ZipTricks::Writer 4 | # All of these are aliased to Int even though they do not have the same 5 | # capacity internally - this is done to prevent callers from havint downcast 6 | # to the very-specific-terrific Int subtype manually. We are not doing Golang here. 7 | # We will however protect from overflows in the writing routines 8 | alias ZipLocation = Int 9 | alias ZipFilesize = Int 10 | alias ZipCRC32 = Int 11 | alias ZipGpFlags = Int 12 | alias ZipStorageMode = Int 13 | 14 | FOUR_BYTE_MAX_UINT = UInt32::MAX 15 | TWO_BYTE_MAX_UINT = UInt16::MAX 16 | EIGHT_BYTE_MAX_UINT = UInt64::MAX 17 | CRZT_COMMENT = "Written using cr_zip_tricks v.#{ZipTricks::VERSION}" 18 | VERSION_NEEDED_TO_EXTRACT = 20 19 | VERSION_NEEDED_TO_EXTRACT_ZIP64 = 45 20 | 21 | # A combination of the VERSION_MADE_BY low byte and the OS type high byte 22 | # VERSION_MADE_BY = 52 23 | # os_type = 3 # UNIX 24 | # [VERSION_MADE_BY, os_type].pack('CC') 25 | MADE_BY_SIGNATURE = Bytes[52, 3] 26 | 27 | def file_external_attrs 28 | # These need to be set so that the unarchived files do not become executable on UNIX, for 29 | # security purposes. Strictly speaking we would want to make this user-customizable, 30 | # but for now just putting in sane defaults will do. For example, Trac with zipinfo does this: 31 | # zipinfo.external_attr = 0644 << 16L # permissions -r-wr--r--. 32 | # We snatch the incantations from Rubyzip for this. 33 | unix_perms = 0o644 34 | file_type_file = 0o10 35 | ((file_type_file << 12 | (unix_perms & 0o7777)) << 16).to_u32! 36 | end 37 | 38 | def dir_external_attrs 39 | # Applies permissions to an empty directory. 40 | unix_perms = 0o755 41 | file_type_dir = 0o04 42 | ((file_type_dir << 12 | (unix_perms & 0o7777)) << 16).to_u32! 43 | end 44 | 45 | private def to_binary_dos_time(t : Time) 46 | (t.hour << 11) | 47 | (t.minute << 5) | 48 | (t.second // 2) 49 | end 50 | 51 | private def to_binary_dos_date(t : Time) 52 | ((t.year - 1980) << 9) | 53 | (t.month << 5) | 54 | t.day 55 | end 56 | 57 | def write_uint8_le(io : IO, val : Int) 58 | if val < UInt8::MIN || val > UInt8::MAX 59 | raise(ArgumentError.new("Unable to fit #{val} into uint8")) 60 | end 61 | io.write_bytes(val.to_u8, IO::ByteFormat::LittleEndian) 62 | end 63 | 64 | def write_uint16_le(io : IO, val : Int) 65 | if val < UInt16::MIN || val > UInt16::MAX 66 | raise(ArgumentError.new("Unable to fit #{val} into uint16")) 67 | end 68 | io.write_bytes(val.to_u16, IO::ByteFormat::LittleEndian) 69 | end 70 | 71 | def write_uint32_le(io : IO, val : Int) 72 | if val < UInt32::MIN || val > UInt32::MAX 73 | raise(ArgumentError.new("Unable to fit #{val} into uint32")) 74 | end 75 | io.write_bytes(val.to_u32, IO::ByteFormat::LittleEndian) 76 | end 77 | 78 | def write_int32_le(io : IO, val : Int) 79 | if val < Int32::MIN || val > Int32::MAX 80 | raise(ArgumentError.new("Unable to fit #{val} into int32")) 81 | end 82 | io.write_bytes(val.to_i32, IO::ByteFormat::LittleEndian) 83 | end 84 | 85 | def write_uint64_le(io : IO, val : Int) 86 | if UInt64::MIN < 0 || val > UInt64::MAX 87 | raise(ArgumentError.new("Unable to fit #{val} into uint64")) 88 | end 89 | io.write_bytes(val.to_u64, IO::ByteFormat::LittleEndian) 90 | end 91 | 92 | def write_zip64_extra_for_local_file_header(io : IO, compressed_size : ZipFilesize, uncompressed_size : ZipFilesize) 93 | write_uint16_le(io, 0x0001) # Tag for the extra field 94 | write_uint16_le(io, 16) # Size of the extra field 95 | write_uint64_le(io, uncompressed_size) # Original uncompressed size 96 | write_uint64_le(io, compressed_size) # Size of compressed data 97 | end 98 | 99 | def write_local_file_header(io : IO, filename : String, compressed_size : ZipFilesize, uncompressed_size : ZipFilesize, crc32 : ZipCRC32, gp_flags : ZipGpFlags, mtime : Time, storage_mode : ZipStorageMode) 100 | requires_zip64 = (compressed_size > FOUR_BYTE_MAX_UINT || uncompressed_size > FOUR_BYTE_MAX_UINT) 101 | 102 | write_uint32_le(io, 0x04034b50) 103 | if requires_zip64 104 | write_uint16_le(io, VERSION_NEEDED_TO_EXTRACT_ZIP64) 105 | else 106 | write_uint16_le(io, VERSION_NEEDED_TO_EXTRACT) 107 | end 108 | 109 | write_uint16_le(io, gp_flags) # general purpose bit flag 2 bytes 110 | write_uint16_le(io, storage_mode) # compression method 2 bytes 111 | write_uint16_le(io, to_binary_dos_time(mtime)) # last mod file time 2 bytes 112 | write_uint16_le(io, to_binary_dos_date(mtime)) # last mod file date 2 bytes 113 | write_uint32_le(io, crc32) # CRC32 4 bytes 114 | 115 | # compressed size 4 bytes 116 | # uncompressed size 4 bytes 117 | if requires_zip64 118 | write_uint32_le(io, FOUR_BYTE_MAX_UINT) 119 | write_uint32_le(io, FOUR_BYTE_MAX_UINT) 120 | else 121 | write_uint32_le(io, compressed_size) 122 | write_uint32_le(io, uncompressed_size) 123 | end 124 | 125 | # Filename should not be longer than 0xFFFF otherwise this wont fit here 126 | write_uint16_le(io, filename.bytesize) 127 | 128 | extra_fields_io = IO::Memory.new 129 | 130 | # Interesting tidbit: 131 | # https://social.technet.microsoft.com/Forums/windows/en-US/6a60399f-2879-4859-b7ab-6ddd08a70948 132 | # TL;DR of it is: Windows 7 Explorer _will_ open Zip64 entries. However, it desires to have the 133 | # Zip64 extra field as _the first_ extra field. 134 | if requires_zip64 135 | write_zip64_extra_for_local_file_header(extra_fields_io, compressed_size, uncompressed_size) 136 | end 137 | write_timestamp_extra_field(extra_fields_io, mtime) 138 | 139 | write_uint16_le(io, extra_fields_io.size) # extra field length 2 bytes 140 | extra_fields_io.rewind 141 | 142 | io.write(filename.encode("utf-8")) # file name (variable size) 143 | IO.copy(extra_fields_io, io) 144 | end 145 | 146 | def write_central_directory_file_header(io : IO, filename : String, compressed_size : ZipFilesize, uncompressed_size : ZipFilesize, crc32 : ZipCRC32, gp_flags : ZipGpFlags, mtime : Time, storage_mode : ZipStorageMode, local_file_header_location : ZipLocation) 147 | # At this point if the header begins somewhere beyound 0xFFFFFFFF we _have_ to record the offset 148 | # of the local file header as a zip64 extra field, so we give up, give in, you loose, love will always win... 149 | add_zip64 = (local_file_header_location > FOUR_BYTE_MAX_UINT) || (compressed_size > FOUR_BYTE_MAX_UINT) || (uncompressed_size > FOUR_BYTE_MAX_UINT) 150 | 151 | # Compose extra fields 152 | extra_fields_io = IO::Memory.new 153 | if add_zip64 154 | write_zip64_extra_for_central_directory_file_header(extra_fields_io, uncompressed_size, compressed_size, local_file_header_location) 155 | end 156 | write_timestamp_extra_field(extra_fields_io, mtime) 157 | extra_fields_io.rewind 158 | 159 | write_uint32_le(io, 0x02014b50) # central directory entry file header signature 4 bytes (0x02014b50) 160 | io.write(MADE_BY_SIGNATURE) # version made by 2 bytes 161 | write_uint16_le(io, add_zip64 ? VERSION_NEEDED_TO_EXTRACT_ZIP64 : VERSION_NEEDED_TO_EXTRACT) # version needed to extract 2 bytes 162 | 163 | write_uint16_le(io, gp_flags) # general purpose bit flag 2 bytes 164 | write_uint16_le(io, storage_mode) # compression method 2 bytes 165 | write_uint16_le(io, to_binary_dos_time(mtime)) # last mod file time 2 bytes 166 | write_uint16_le(io, to_binary_dos_date(mtime)) # last mod file date 2 bytes 167 | write_uint32_le(io, crc32) # crc-32 4 bytes 168 | 169 | write_uint32_le(io, add_zip64 ? FOUR_BYTE_MAX_UINT : compressed_size) 170 | write_uint32_le(io, add_zip64 ? FOUR_BYTE_MAX_UINT : uncompressed_size) 171 | 172 | # Filename should not be longer than 0xFFFF otherwise this wont fit here 173 | write_uint16_le(io, filename.bytesize) # file name length 2 bytes 174 | write_uint16_le(io, extra_fields_io.size) # extra field length 2 bytes 175 | write_uint16_le(io, 0) # file comment length 2 bytes 176 | 177 | # For The Unarchiver < 3.11.1 this field has to be set to the overflow value if zip64 is used 178 | # because otherwise it does not properly advance the pointer when reading the Zip64 extra field 179 | # https://bitbucket.org/WAHa_06x36/theunarchiver/pull-requests/2/bug-fix-for-zip64-extra-field-parser/diff 180 | write_uint16_le(io, add_zip64 ? TWO_BYTE_MAX_UINT : 0) # disk number start 2 bytes 181 | write_uint16_le(io, 0) # internal file attributes 2 bytes 182 | 183 | # Because the add_empty_directory method will create a directory with a trailing "/", 184 | # this check can be used to assign proper permissions to the created directory. 185 | # external file attributes 4 bytes 186 | exattrs = filename.ends_with?('/') ? dir_external_attrs : file_external_attrs 187 | write_uint32_le(io, exattrs) 188 | 189 | entry_header_offset = add_zip64 ? FOUR_BYTE_MAX_UINT : local_file_header_location 190 | write_uint32_le(io, entry_header_offset) # relative offset of local header 4 bytes 191 | 192 | io.write(filename.encode("utf-8")) # file name (variable size) 193 | 194 | IO.copy(extra_fields_io, io) # extra field (variable size) 195 | # (empty) # file comment (variable size) 196 | end 197 | 198 | def write_zip64_extra_for_central_directory_file_header(io : IO, uncompressed_size : Int, compressed_size : Int, local_file_header_location : ZipLocation) 199 | write_uint16_le(io, 0x0001) # 2 bytes Tag for this "extra" block type 200 | write_uint16_le(io, 28) # 2 bytes Size of this "extra" block. For us it will always be 28 201 | write_uint64_le(io, uncompressed_size) # 8 bytes Size of uncompressed data 202 | write_uint64_le(io, compressed_size) # 8 bytes Size of compressed data 203 | write_uint64_le(io, local_file_header_location) # 8 bytes Local file header location in file 204 | write_uint32_le(io, 0) # 4 bytes Number of the disk on which this file starts 205 | end 206 | 207 | def write_end_of_central_directory(io : IO, start_of_central_directory_location : ZipLocation, central_directory_size : ZipLocation, num_files_in_archive : ZipLocation, comment : String = CRZT_COMMENT) 208 | zip64_eocdr_offset = start_of_central_directory_location.to_u64 + central_directory_size.to_u64 209 | zip64_required = central_directory_size > FOUR_BYTE_MAX_UINT || 210 | start_of_central_directory_location > FOUR_BYTE_MAX_UINT || 211 | zip64_eocdr_offset > FOUR_BYTE_MAX_UINT || 212 | num_files_in_archive > TWO_BYTE_MAX_UINT 213 | 214 | # Then, if zip64 is used 215 | if zip64_required 216 | # [zip64 end of central directory record] 217 | # zip64 end of central dir 218 | write_uint32_le(io, 0x06064b50) # signature 4 bytes (0x06064b50) 219 | write_uint64_le(io, 44) # size of zip64 end of central 220 | # directory record 8 bytes 221 | # (this is ex. the 12 bytes of the signature and the size value itself). 222 | # Without the extensible data sector (which we are not using) 223 | # it is always 44 bytes. 224 | io.write(MADE_BY_SIGNATURE) # version made by 2 bytes 225 | write_uint16_le(io, VERSION_NEEDED_TO_EXTRACT_ZIP64) # version needed to extract 2 bytes 226 | write_uint32_le(io, 0) # number of this disk 4 bytes 227 | write_uint32_le(io, 0) # number of the disk with the start of the central directory 4 bytes 228 | write_uint64_le(io, num_files_in_archive) # total number of entries in the central directory on this disk 8 bytes 229 | write_uint64_le(io, num_files_in_archive) # total number of entries in the archive total 8 bytes 230 | write_uint64_le(io, central_directory_size) # size of the central directory 8 bytes 231 | 232 | # offset of start of central directory with respect to the starting disk number 8 bytes 233 | write_uint64_le(io, start_of_central_directory_location) 234 | # zip64 extensible data sector (variable size), blank for us 235 | 236 | # [zip64 end of central directory locator] 237 | write_uint32_le(io, 0x07064b50) # zip64 end of central dir locator signature 4 bytes (0x07064b50) 238 | write_uint32_le(io, 0) # number of the disk with the start of the zip64 end of central directory 4 bytes 239 | write_uint64_le(io, zip64_eocdr_offset) # relative offset of the zip64 240 | # end of central directory record 8 bytes 241 | # (note: "relative" is actually "from the start of the file") 242 | write_uint32_le(io, 1) # total number of disks 4 bytes 243 | end 244 | 245 | # Then the end of central directory record: 246 | write_uint32_le(io, 0x06054b50) # end of central dir signature 4 bytes (0x06054b50) 247 | write_uint16_le(io, 0) # number of this disk 2 bytes 248 | write_uint16_le(io, 0) # number of the disk with the 249 | # start of the central directory 2 bytes 250 | 251 | num_entries = zip64_required ? TWO_BYTE_MAX_UINT : num_files_in_archive 252 | write_uint16_le(io, num_entries) # total number of entries in the central directory on this disk 2 bytes 253 | write_uint16_le(io, num_entries) # total number of entries in the central directory 2 bytes 254 | 255 | write_uint32_le(io, zip64_required ? FOUR_BYTE_MAX_UINT : central_directory_size) # size of the central directory 4 bytes 256 | write_uint32_le(io, zip64_required ? FOUR_BYTE_MAX_UINT : start_of_central_directory_location) # offset of start of central directory with respect to the starting disk number 4 bytes 257 | 258 | # Sneak in the default comment 259 | write_uint16_le(io, comment.bytesize) # .ZIP file comment length 2 bytes 260 | io.write(comment.encode("utf-8")) # .ZIP file comment (variable size) 261 | end 262 | 263 | def write_data_descriptor(io : IO, compressed_size : ZipFilesize, uncompressed_size : ZipFilesize, crc32 : ZipCRC32) 264 | write_uint32_le(io, 0x08074b50) # Although not originally assigned a signature, the value 265 | # 0x08074b50 has commonly been adopted as a signature value 266 | # for the data descriptor record. 267 | write_uint32_le(io, crc32) # crc-32 4 bytes 268 | 269 | # If one of the sizes is above 0xFFFFFFF use ZIP64 lengths (8 bytes) instead. A good unarchiver 270 | # will decide to unpack it as such if it finds the Zip64 extra for the file in the central directory. 271 | # So also use the opportune moment to switch the entry to Zip64 if needed 272 | requires_zip64 = (compressed_size > FOUR_BYTE_MAX_UINT || uncompressed_size > FOUR_BYTE_MAX_UINT) 273 | 274 | # compressed size 4 bytes, or 8 bytes for ZIP64 275 | # uncompressed size 4 bytes, or 8 bytes for ZIP64 276 | if requires_zip64 277 | write_uint64_le(io, compressed_size) 278 | write_uint64_le(io, uncompressed_size) 279 | else 280 | write_uint32_le(io, compressed_size) 281 | write_uint32_le(io, uncompressed_size) 282 | end 283 | end 284 | 285 | # Writes the extended timestamp information field. The spec defines 2 286 | # different formats - the one for the local file header can also accomodate the 287 | # atime and ctime, whereas the one for the central directory can only take 288 | # the mtime - and refers the reader to the local header extra to obtain the 289 | # remaining times 290 | def write_timestamp_extra_field(io : IO, mtime : Time) 291 | # Local-header version: 292 | # 293 | # Value Size Description 294 | # ----- ---- ----------- 295 | # (time) 0x5455 Short tag for this extra block type ("UT") 296 | # TSize Short total data size for this block 297 | # Flags Byte info bits 298 | # (ModTime) Long time of last modification (UTC/GMT) 299 | # (AcTime) Long time of last access (UTC/GMT) 300 | # (CrTime) Long time of original creation (UTC/GMT) 301 | # 302 | # Central-header version: 303 | # 304 | # Value Size Description 305 | # ----- ---- ----------- 306 | # (time) 0x5455 Short tag for this extra block type ("UT") 307 | # TSize Short total data size for this block 308 | # Flags Byte info bits (refers to local header!) 309 | # (ModTime) Long time of last modification (UTC/GMT) 310 | # 311 | # The lower three bits of Flags in both headers indicate which time- 312 | # stamps are present in the LOCAL extra field: 313 | # 314 | # bit 0 if set, modification time is present 315 | # bit 1 if set, access time is present 316 | # bit 2 if set, creation time is present 317 | # bits 3-7 reserved for additional timestamps; not set 318 | flags = 0b10000000 # Set bit 1 only to indicate only mtime is present 319 | write_uint16_le(io, 0x5455) # tag for this extra block type ("UT") 320 | write_uint16_le(io, 1 + 4) # # the size of this block (1 byte used for the Flag + 1 long used for the timestamp) 321 | write_uint8_le(io, flags) # encode a single byte 322 | write_int32_le(io, mtime.to_unix) # Use a signed long, not the unsigned one used by the rest of the ZIP spec. 323 | end 324 | end 325 | --------------------------------------------------------------------------------