├── .github └── workflows │ └── rust.yml ├── .gitignore ├── CODE_OF_CONDUCT.md ├── CONTRIBUTING.md ├── Cargo.lock ├── Cargo.toml ├── LICENSE ├── NEWS.md ├── README.md ├── doc └── protocol-notes.md ├── src ├── bin │ └── rsyn.rs ├── client.rs ├── connection.rs ├── flist.rs ├── lib.rs ├── localtree.rs ├── mux.rs ├── options.rs ├── statistics.rs ├── sums.rs └── varint.rs └── tests └── interop.rs /.github/workflows/rust.yml: -------------------------------------------------------------------------------- 1 | name: rust 2 | 3 | on: [push, pull_request] 4 | 5 | jobs: 6 | build: 7 | runs-on: ${{ matrix.os }} 8 | timeout-minutes: 30 9 | strategy: 10 | matrix: 11 | os: [ubuntu-latest, macos-latest, windows-latest] 12 | env: 13 | RUST_BACKTRACE: 1 14 | steps: 15 | - name: Install rsync from Chocolatey 16 | if: ${{ matrix.os == 'windows-latest' }} 17 | run: choco install rsync 18 | - uses: actions/checkout@v2 19 | - name: Build 20 | run: cargo build --all --verbose --all-targets 21 | - name: Unit tests 22 | run: cargo test --lib --bins 23 | - name: Integration tests 24 | # TODO: Fix these to test against external rsync on Windows. 25 | # TODO: At least, run integration tests that don't depend on an external rsync. 26 | if: ${{ matrix.os != 'windows-latest' }} 27 | run: cargo test --tests 28 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | /target 2 | /rsyn.log 3 | mutants.out* 4 | -------------------------------------------------------------------------------- /CODE_OF_CONDUCT.md: -------------------------------------------------------------------------------- 1 | # Google Open Source Community Guidelines 2 | 3 | At Google, we recognize and celebrate the creativity and collaboration of open 4 | source contributors and the diversity of skills, experiences, cultures, and 5 | opinions they bring to the projects and communities they participate in. 6 | 7 | Every one of Google's open source projects and communities are inclusive 8 | environments, based on treating all individuals respectfully, regardless of 9 | gender identity and expression, sexual orientation, disabilities, 10 | neurodiversity, physical appearance, body size, ethnicity, nationality, race, 11 | age, religion, or similar personal characteristic. 12 | 13 | We value diverse opinions, but we value respectful behavior more. 14 | 15 | Respectful behavior includes: 16 | 17 | * Being considerate, kind, constructive, and helpful. 18 | * Not engaging in demeaning, discriminatory, harassing, hateful, sexualized, or 19 | physically threatening behavior, speech, and imagery. 20 | * Not engaging in unwanted physical contact. 21 | 22 | Some Google open source projects [may adopt][] an explicit project code of 23 | conduct, which may have additional detailed expectations for participants. Most 24 | of those projects will use our [modified Contributor Covenant][]. 25 | 26 | [may adopt]: https://opensource.google/docs/releasing/preparing/#conduct 27 | [modified Contributor Covenant]: https://opensource.google/docs/releasing/template/CODE_OF_CONDUCT/ 28 | 29 | ## Resolve peacefully 30 | 31 | We do not believe that all conflict is necessarily bad; healthy debate and 32 | disagreement often yields positive results. However, it is never okay to be 33 | disrespectful. 34 | 35 | If you see someone behaving disrespectfully, you are encouraged to address the 36 | behavior directly with those involved. Many issues can be resolved quickly and 37 | easily, and this gives people more control over the outcome of their dispute. 38 | If you are unable to resolve the matter for any reason, or if the behavior is 39 | threatening or harassing, report it. We are dedicated to providing an 40 | environment where participants feel welcome and safe. 41 | 42 | ## Reporting problems 43 | 44 | Some Google open source projects may adopt a project-specific code of conduct. 45 | In those cases, a Google employee will be identified as the Project Steward, 46 | who will receive and handle reports of code of conduct violations. In the event 47 | that a project hasn’t identified a Project Steward, you can report problems by 48 | emailing opensource@google.com. 49 | 50 | We will investigate every complaint, but you may not receive a direct response. 51 | We will use our discretion in determining when and how to follow up on reported 52 | incidents, which may range from not taking action to permanent expulsion from 53 | the project and project-sponsored spaces. We will notify the accused of the 54 | report and provide them an opportunity to discuss it before any action is 55 | taken. The identity of the reporter will be omitted from the details of the 56 | report supplied to the accused. In potentially harmful situations, such as 57 | ongoing harassment or threats to anyone's safety, we may take action without 58 | notice. 59 | 60 | *This document was adapted from the [IndieWeb Code of Conduct][] and can also 61 | be found at .* 62 | 63 | [IndieWeb Code of Conduct]: https://indieweb.org/code-of-conduct 64 | -------------------------------------------------------------------------------- /CONTRIBUTING.md: -------------------------------------------------------------------------------- 1 | # How to Contribute 2 | 3 | We'd love to accept your patches and contributions to this project. There are 4 | just a few small guidelines you need to follow. 5 | 6 | ## Contributor License Agreement 7 | 8 | Contributions to this project must be accompanied by a Contributor License 9 | Agreement. You (or your employer) retain the copyright to your contribution; 10 | this simply gives us permission to use and redistribute your contributions as 11 | part of the project. Head over to to see 12 | your current agreements on file or to sign a new one. 13 | 14 | You generally only need to submit a CLA once, so if you've already submitted one 15 | (even if it was for a different project), you probably don't need to do it 16 | again. 17 | 18 | ## Code reviews 19 | 20 | All submissions, including submissions by project members, require review. We 21 | use GitHub pull requests for this purpose. Consult 22 | [GitHub Help](https://help.github.com/articles/about-pull-requests/) for more 23 | information on using pull requests. 24 | 25 | ## Community Guidelines 26 | 27 | This project follows [Google's Open Source Community 28 | Guidelines](https://opensource.google/conduct/). 29 | -------------------------------------------------------------------------------- /Cargo.lock: -------------------------------------------------------------------------------- 1 | # This file is automatically @generated by Cargo. 2 | # It is not intended for manual editing. 3 | version = 3 4 | 5 | [[package]] 6 | name = "aho-corasick" 7 | version = "0.7.18" 8 | source = "registry+https://github.com/rust-lang/crates.io-index" 9 | checksum = "1e37cfd5e7657ada45f742d6e99ca5788580b5c529dc78faf11ece6dc702656f" 10 | dependencies = [ 11 | "memchr", 12 | ] 13 | 14 | [[package]] 15 | name = "ansi_term" 16 | version = "0.11.0" 17 | source = "registry+https://github.com/rust-lang/crates.io-index" 18 | checksum = "ee49baf6cb617b853aa8d93bf420db2383fab46d314482ca2803b40d5fde979b" 19 | dependencies = [ 20 | "winapi", 21 | ] 22 | 23 | [[package]] 24 | name = "anyhow" 25 | version = "1.0.44" 26 | source = "registry+https://github.com/rust-lang/crates.io-index" 27 | checksum = "61604a8f862e1d5c3229fdd78f8b02c68dcf73a4c4b05fd636d12240aaa242c1" 28 | 29 | [[package]] 30 | name = "atty" 31 | version = "0.2.14" 32 | source = "registry+https://github.com/rust-lang/crates.io-index" 33 | checksum = "d9b39be18770d11421cdb1b9947a45dd3f37e93092cbf377614828a319d5fee8" 34 | dependencies = [ 35 | "hermit-abi", 36 | "libc", 37 | "winapi", 38 | ] 39 | 40 | [[package]] 41 | name = "autocfg" 42 | version = "1.0.1" 43 | source = "registry+https://github.com/rust-lang/crates.io-index" 44 | checksum = "cdb031dd78e28731d87d56cc8ffef4a8f36ca26c38fe2de700543e627f8a464a" 45 | 46 | [[package]] 47 | name = "bitflags" 48 | version = "1.3.2" 49 | source = "registry+https://github.com/rust-lang/crates.io-index" 50 | checksum = "bef38d45163c2f1dde094a7dfd33ccf595c92905c8f8f4fdc18d06fb1037718a" 51 | 52 | [[package]] 53 | name = "block-buffer" 54 | version = "0.7.3" 55 | source = "registry+https://github.com/rust-lang/crates.io-index" 56 | checksum = "c0940dc441f31689269e10ac70eb1002a3a1d3ad1390e030043662eb7fe4688b" 57 | dependencies = [ 58 | "block-padding", 59 | "byte-tools", 60 | "byteorder", 61 | "generic-array", 62 | ] 63 | 64 | [[package]] 65 | name = "block-padding" 66 | version = "0.1.5" 67 | source = "registry+https://github.com/rust-lang/crates.io-index" 68 | checksum = "fa79dedbb091f449f1f39e53edf88d5dbe95f895dae6135a8d7b881fb5af73f5" 69 | dependencies = [ 70 | "byte-tools", 71 | ] 72 | 73 | [[package]] 74 | name = "byte-tools" 75 | version = "0.3.1" 76 | source = "registry+https://github.com/rust-lang/crates.io-index" 77 | checksum = "e3b5ca7a04898ad4bcd41c90c5285445ff5b791899bb1b0abdd2a2aa791211d7" 78 | 79 | [[package]] 80 | name = "byteorder" 81 | version = "1.4.3" 82 | source = "registry+https://github.com/rust-lang/crates.io-index" 83 | checksum = "14c189c53d098945499cdfa7ecc63567cf3886b3332b312a5b4585d8d3a6a610" 84 | 85 | [[package]] 86 | name = "cfg-if" 87 | version = "0.1.10" 88 | source = "registry+https://github.com/rust-lang/crates.io-index" 89 | checksum = "4785bdd1c96b2a846b2bd7cc02e86b6b3dbf14e7e53446c4f54c92a361040822" 90 | 91 | [[package]] 92 | name = "cfg-if" 93 | version = "1.0.0" 94 | source = "registry+https://github.com/rust-lang/crates.io-index" 95 | checksum = "baf1de4339761588bc0619e3cbc0120ee582ebb74b53b4efbf79117bd2da40fd" 96 | 97 | [[package]] 98 | name = "chrono" 99 | version = "0.4.19" 100 | source = "registry+https://github.com/rust-lang/crates.io-index" 101 | checksum = "670ad68c9088c2a963aaa298cb369688cf3f9465ce5e2d4ca10e6e0098a1ce73" 102 | dependencies = [ 103 | "libc", 104 | "num-integer", 105 | "num-traits", 106 | "time", 107 | "winapi", 108 | ] 109 | 110 | [[package]] 111 | name = "clap" 112 | version = "2.33.3" 113 | source = "registry+https://github.com/rust-lang/crates.io-index" 114 | checksum = "37e58ac78573c40708d45522f0d80fa2f01cc4f9b4e2bf749807255454312002" 115 | dependencies = [ 116 | "ansi_term", 117 | "atty", 118 | "bitflags", 119 | "strsim", 120 | "term_size", 121 | "textwrap", 122 | "unicode-width", 123 | "vec_map", 124 | ] 125 | 126 | [[package]] 127 | name = "colored" 128 | version = "1.9.3" 129 | source = "registry+https://github.com/rust-lang/crates.io-index" 130 | checksum = "f4ffc801dacf156c5854b9df4f425a626539c3a6ef7893cc0c5084a23f0b6c59" 131 | dependencies = [ 132 | "atty", 133 | "lazy_static", 134 | "winapi", 135 | ] 136 | 137 | [[package]] 138 | name = "crossbeam" 139 | version = "0.7.3" 140 | source = "registry+https://github.com/rust-lang/crates.io-index" 141 | checksum = "69323bff1fb41c635347b8ead484a5ca6c3f11914d784170b158d8449ab07f8e" 142 | dependencies = [ 143 | "cfg-if 0.1.10", 144 | "crossbeam-channel", 145 | "crossbeam-deque", 146 | "crossbeam-epoch", 147 | "crossbeam-queue", 148 | "crossbeam-utils", 149 | ] 150 | 151 | [[package]] 152 | name = "crossbeam-channel" 153 | version = "0.4.4" 154 | source = "registry+https://github.com/rust-lang/crates.io-index" 155 | checksum = "b153fe7cbef478c567df0f972e02e6d736db11affe43dfc9c56a9374d1adfb87" 156 | dependencies = [ 157 | "crossbeam-utils", 158 | "maybe-uninit", 159 | ] 160 | 161 | [[package]] 162 | name = "crossbeam-deque" 163 | version = "0.7.4" 164 | source = "registry+https://github.com/rust-lang/crates.io-index" 165 | checksum = "c20ff29ded3204c5106278a81a38f4b482636ed4fa1e6cfbeef193291beb29ed" 166 | dependencies = [ 167 | "crossbeam-epoch", 168 | "crossbeam-utils", 169 | "maybe-uninit", 170 | ] 171 | 172 | [[package]] 173 | name = "crossbeam-epoch" 174 | version = "0.8.2" 175 | source = "registry+https://github.com/rust-lang/crates.io-index" 176 | checksum = "058ed274caafc1f60c4997b5fc07bf7dc7cca454af7c6e81edffe5f33f70dace" 177 | dependencies = [ 178 | "autocfg", 179 | "cfg-if 0.1.10", 180 | "crossbeam-utils", 181 | "lazy_static", 182 | "maybe-uninit", 183 | "memoffset", 184 | "scopeguard", 185 | ] 186 | 187 | [[package]] 188 | name = "crossbeam-queue" 189 | version = "0.2.3" 190 | source = "registry+https://github.com/rust-lang/crates.io-index" 191 | checksum = "774ba60a54c213d409d5353bda12d49cd68d14e45036a285234c8d6f91f92570" 192 | dependencies = [ 193 | "cfg-if 0.1.10", 194 | "crossbeam-utils", 195 | "maybe-uninit", 196 | ] 197 | 198 | [[package]] 199 | name = "crossbeam-utils" 200 | version = "0.7.2" 201 | source = "registry+https://github.com/rust-lang/crates.io-index" 202 | checksum = "c3c7c73a2d1e9fc0886a08b93e98eb643461230d5f1925e4036204d5f2e261a8" 203 | dependencies = [ 204 | "autocfg", 205 | "cfg-if 0.1.10", 206 | "lazy_static", 207 | ] 208 | 209 | [[package]] 210 | name = "digest" 211 | version = "0.8.1" 212 | source = "registry+https://github.com/rust-lang/crates.io-index" 213 | checksum = "f3d0c8c8752312f9713efd397ff63acb9f85585afbf179282e720e7704954dd5" 214 | dependencies = [ 215 | "generic-array", 216 | ] 217 | 218 | [[package]] 219 | name = "fake-simd" 220 | version = "0.1.2" 221 | source = "registry+https://github.com/rust-lang/crates.io-index" 222 | checksum = "e88a8acf291dafb59c2d96e8f59828f3838bb1a70398823ade51a84de6a6deed" 223 | 224 | [[package]] 225 | name = "fern" 226 | version = "0.6.0" 227 | source = "registry+https://github.com/rust-lang/crates.io-index" 228 | checksum = "8c9a4820f0ccc8a7afd67c39a0f1a0f4b07ca1725164271a64939d7aeb9af065" 229 | dependencies = [ 230 | "colored", 231 | "log", 232 | ] 233 | 234 | [[package]] 235 | name = "generic-array" 236 | version = "0.12.4" 237 | source = "registry+https://github.com/rust-lang/crates.io-index" 238 | checksum = "ffdf9f34f1447443d37393cc6c2b8313aebddcd96906caf34e54c68d8e57d7bd" 239 | dependencies = [ 240 | "typenum", 241 | ] 242 | 243 | [[package]] 244 | name = "getrandom" 245 | version = "0.2.3" 246 | source = "registry+https://github.com/rust-lang/crates.io-index" 247 | checksum = "7fcd999463524c52659517fe2cea98493cfe485d10565e7b0fb07dbba7ad2753" 248 | dependencies = [ 249 | "cfg-if 1.0.0", 250 | "libc", 251 | "wasi", 252 | ] 253 | 254 | [[package]] 255 | name = "heck" 256 | version = "0.3.3" 257 | source = "registry+https://github.com/rust-lang/crates.io-index" 258 | checksum = "6d621efb26863f0e9924c6ac577e8275e5e6b77455db64ffa6c65c904e9e132c" 259 | dependencies = [ 260 | "unicode-segmentation", 261 | ] 262 | 263 | [[package]] 264 | name = "hermit-abi" 265 | version = "0.1.19" 266 | source = "registry+https://github.com/rust-lang/crates.io-index" 267 | checksum = "62b467343b94ba476dcb2500d242dadbb39557df889310ac77c5d99100aaac33" 268 | dependencies = [ 269 | "libc", 270 | ] 271 | 272 | [[package]] 273 | name = "hex" 274 | version = "0.4.3" 275 | source = "registry+https://github.com/rust-lang/crates.io-index" 276 | checksum = "7f24254aa9a54b5c858eaee2f5bccdb46aaf0e486a595ed5fd8f86ba55232a70" 277 | 278 | [[package]] 279 | name = "lazy_static" 280 | version = "1.4.0" 281 | source = "registry+https://github.com/rust-lang/crates.io-index" 282 | checksum = "e2abad23fbc42b3700f2f279844dc832adb2b2eb069b2df918f455c4e18cc646" 283 | 284 | [[package]] 285 | name = "libc" 286 | version = "0.2.105" 287 | source = "registry+https://github.com/rust-lang/crates.io-index" 288 | checksum = "869d572136620d55835903746bcb5cdc54cb2851fd0aeec53220b4bb65ef3013" 289 | 290 | [[package]] 291 | name = "log" 292 | version = "0.4.14" 293 | source = "registry+https://github.com/rust-lang/crates.io-index" 294 | checksum = "51b9bbe6c47d51fc3e1a9b945965946b4c44142ab8792c50835a980d362c2710" 295 | dependencies = [ 296 | "cfg-if 1.0.0", 297 | ] 298 | 299 | [[package]] 300 | name = "maybe-uninit" 301 | version = "2.0.0" 302 | source = "registry+https://github.com/rust-lang/crates.io-index" 303 | checksum = "60302e4db3a61da70c0cb7991976248362f30319e88850c487b9b95bbf059e00" 304 | 305 | [[package]] 306 | name = "md4" 307 | version = "0.8.0" 308 | source = "registry+https://github.com/rust-lang/crates.io-index" 309 | checksum = "a4030c65cf2aab7ada769cae7d1e7159f8d034d6ded4f39afba037f094bfd9a1" 310 | dependencies = [ 311 | "block-buffer", 312 | "digest", 313 | "fake-simd", 314 | "opaque-debug", 315 | ] 316 | 317 | [[package]] 318 | name = "memchr" 319 | version = "2.4.1" 320 | source = "registry+https://github.com/rust-lang/crates.io-index" 321 | checksum = "308cc39be01b73d0d18f82a0e7b2a3df85245f84af96fdddc5d202d27e47b86a" 322 | 323 | [[package]] 324 | name = "memoffset" 325 | version = "0.5.6" 326 | source = "registry+https://github.com/rust-lang/crates.io-index" 327 | checksum = "043175f069eda7b85febe4a74abbaeff828d9f8b448515d3151a14a3542811aa" 328 | dependencies = [ 329 | "autocfg", 330 | ] 331 | 332 | [[package]] 333 | name = "num-integer" 334 | version = "0.1.44" 335 | source = "registry+https://github.com/rust-lang/crates.io-index" 336 | checksum = "d2cc698a63b549a70bc047073d2949cce27cd1c7b0a4a862d08a8031bc2801db" 337 | dependencies = [ 338 | "autocfg", 339 | "num-traits", 340 | ] 341 | 342 | [[package]] 343 | name = "num-traits" 344 | version = "0.2.14" 345 | source = "registry+https://github.com/rust-lang/crates.io-index" 346 | checksum = "9a64b1ec5cda2586e284722486d802acf1f7dbdc623e2bfc57e65ca1cd099290" 347 | dependencies = [ 348 | "autocfg", 349 | ] 350 | 351 | [[package]] 352 | name = "opaque-debug" 353 | version = "0.2.3" 354 | source = "registry+https://github.com/rust-lang/crates.io-index" 355 | checksum = "2839e79665f131bdb5782e51f2c6c9599c133c6098982a54c794358bf432529c" 356 | 357 | [[package]] 358 | name = "ppv-lite86" 359 | version = "0.2.15" 360 | source = "registry+https://github.com/rust-lang/crates.io-index" 361 | checksum = "ed0cfbc8191465bed66e1718596ee0b0b35d5ee1f41c5df2189d0fe8bde535ba" 362 | 363 | [[package]] 364 | name = "proc-macro-error" 365 | version = "1.0.4" 366 | source = "registry+https://github.com/rust-lang/crates.io-index" 367 | checksum = "da25490ff9892aab3fcf7c36f08cfb902dd3e71ca0f9f9517bea02a73a5ce38c" 368 | dependencies = [ 369 | "proc-macro-error-attr", 370 | "proc-macro2", 371 | "quote", 372 | "syn", 373 | "version_check", 374 | ] 375 | 376 | [[package]] 377 | name = "proc-macro-error-attr" 378 | version = "1.0.4" 379 | source = "registry+https://github.com/rust-lang/crates.io-index" 380 | checksum = "a1be40180e52ecc98ad80b184934baf3d0d29f979574e439af5a55274b35f869" 381 | dependencies = [ 382 | "proc-macro2", 383 | "quote", 384 | "version_check", 385 | ] 386 | 387 | [[package]] 388 | name = "proc-macro2" 389 | version = "1.0.30" 390 | source = "registry+https://github.com/rust-lang/crates.io-index" 391 | checksum = "edc3358ebc67bc8b7fa0c007f945b0b18226f78437d61bec735a9eb96b61ee70" 392 | dependencies = [ 393 | "unicode-xid", 394 | ] 395 | 396 | [[package]] 397 | name = "quote" 398 | version = "1.0.10" 399 | source = "registry+https://github.com/rust-lang/crates.io-index" 400 | checksum = "38bc8cc6a5f2e3655e0899c1b848643b2562f853f114bfec7be120678e3ace05" 401 | dependencies = [ 402 | "proc-macro2", 403 | ] 404 | 405 | [[package]] 406 | name = "rand" 407 | version = "0.8.4" 408 | source = "registry+https://github.com/rust-lang/crates.io-index" 409 | checksum = "2e7573632e6454cf6b99d7aac4ccca54be06da05aca2ef7423d22d27d4d4bcd8" 410 | dependencies = [ 411 | "libc", 412 | "rand_chacha", 413 | "rand_core", 414 | "rand_hc", 415 | ] 416 | 417 | [[package]] 418 | name = "rand_chacha" 419 | version = "0.3.1" 420 | source = "registry+https://github.com/rust-lang/crates.io-index" 421 | checksum = "e6c10a63a0fa32252be49d21e7709d4d4baf8d231c2dbce1eaa8141b9b127d88" 422 | dependencies = [ 423 | "ppv-lite86", 424 | "rand_core", 425 | ] 426 | 427 | [[package]] 428 | name = "rand_core" 429 | version = "0.6.3" 430 | source = "registry+https://github.com/rust-lang/crates.io-index" 431 | checksum = "d34f1408f55294453790c48b2f1ebbb1c5b4b7563eb1f418bcfcfdbb06ebb4e7" 432 | dependencies = [ 433 | "getrandom", 434 | ] 435 | 436 | [[package]] 437 | name = "rand_hc" 438 | version = "0.3.1" 439 | source = "registry+https://github.com/rust-lang/crates.io-index" 440 | checksum = "d51e9f596de227fda2ea6c84607f5558e196eeaf43c986b724ba4fb8fdf497e7" 441 | dependencies = [ 442 | "rand_core", 443 | ] 444 | 445 | [[package]] 446 | name = "redox_syscall" 447 | version = "0.2.10" 448 | source = "registry+https://github.com/rust-lang/crates.io-index" 449 | checksum = "8383f39639269cde97d255a32bdb68c047337295414940c68bdd30c2e13203ff" 450 | dependencies = [ 451 | "bitflags", 452 | ] 453 | 454 | [[package]] 455 | name = "regex" 456 | version = "1.5.4" 457 | source = "registry+https://github.com/rust-lang/crates.io-index" 458 | checksum = "d07a8629359eb56f1e2fb1652bb04212c072a87ba68546a04065d525673ac461" 459 | dependencies = [ 460 | "aho-corasick", 461 | "memchr", 462 | "regex-syntax", 463 | ] 464 | 465 | [[package]] 466 | name = "regex-syntax" 467 | version = "0.6.25" 468 | source = "registry+https://github.com/rust-lang/crates.io-index" 469 | checksum = "f497285884f3fcff424ffc933e56d7cbca511def0c9831a7f9b5f6153e3cc89b" 470 | 471 | [[package]] 472 | name = "remove_dir_all" 473 | version = "0.5.3" 474 | source = "registry+https://github.com/rust-lang/crates.io-index" 475 | checksum = "3acd125665422973a33ac9d3dd2df85edad0f4ae9b00dafb1a05e43a9f5ef8e7" 476 | dependencies = [ 477 | "winapi", 478 | ] 479 | 480 | [[package]] 481 | name = "rsyn" 482 | version = "0.0.2" 483 | dependencies = [ 484 | "anyhow", 485 | "chrono", 486 | "crossbeam", 487 | "fern", 488 | "hex", 489 | "lazy_static", 490 | "log", 491 | "md4", 492 | "regex", 493 | "shell-words", 494 | "structopt", 495 | "tempfile", 496 | "unix_mode", 497 | ] 498 | 499 | [[package]] 500 | name = "scopeguard" 501 | version = "1.1.0" 502 | source = "registry+https://github.com/rust-lang/crates.io-index" 503 | checksum = "d29ab0c6d3fc0ee92fe66e2d99f700eab17a8d57d1c1d3b748380fb20baa78cd" 504 | 505 | [[package]] 506 | name = "shell-words" 507 | version = "1.0.0" 508 | source = "registry+https://github.com/rust-lang/crates.io-index" 509 | checksum = "b6fa3938c99da4914afedd13bf3d79bcb6c277d1b2c398d23257a304d9e1b074" 510 | 511 | [[package]] 512 | name = "strsim" 513 | version = "0.8.0" 514 | source = "registry+https://github.com/rust-lang/crates.io-index" 515 | checksum = "8ea5119cdb4c55b55d432abb513a0429384878c15dde60cc77b1c99de1a95a6a" 516 | 517 | [[package]] 518 | name = "structopt" 519 | version = "0.3.25" 520 | source = "registry+https://github.com/rust-lang/crates.io-index" 521 | checksum = "40b9788f4202aa75c240ecc9c15c65185e6a39ccdeb0fd5d008b98825464c87c" 522 | dependencies = [ 523 | "clap", 524 | "lazy_static", 525 | "structopt-derive", 526 | ] 527 | 528 | [[package]] 529 | name = "structopt-derive" 530 | version = "0.4.18" 531 | source = "registry+https://github.com/rust-lang/crates.io-index" 532 | checksum = "dcb5ae327f9cc13b68763b5749770cb9e048a99bd9dfdfa58d0cf05d5f64afe0" 533 | dependencies = [ 534 | "heck", 535 | "proc-macro-error", 536 | "proc-macro2", 537 | "quote", 538 | "syn", 539 | ] 540 | 541 | [[package]] 542 | name = "syn" 543 | version = "1.0.80" 544 | source = "registry+https://github.com/rust-lang/crates.io-index" 545 | checksum = "d010a1623fbd906d51d650a9916aaefc05ffa0e4053ff7fe601167f3e715d194" 546 | dependencies = [ 547 | "proc-macro2", 548 | "quote", 549 | "unicode-xid", 550 | ] 551 | 552 | [[package]] 553 | name = "tempfile" 554 | version = "3.2.0" 555 | source = "registry+https://github.com/rust-lang/crates.io-index" 556 | checksum = "dac1c663cfc93810f88aed9b8941d48cabf856a1b111c29a40439018d870eb22" 557 | dependencies = [ 558 | "cfg-if 1.0.0", 559 | "libc", 560 | "rand", 561 | "redox_syscall", 562 | "remove_dir_all", 563 | "winapi", 564 | ] 565 | 566 | [[package]] 567 | name = "term_size" 568 | version = "0.3.2" 569 | source = "registry+https://github.com/rust-lang/crates.io-index" 570 | checksum = "1e4129646ca0ed8f45d09b929036bafad5377103edd06e50bf574b353d2b08d9" 571 | dependencies = [ 572 | "libc", 573 | "winapi", 574 | ] 575 | 576 | [[package]] 577 | name = "textwrap" 578 | version = "0.11.0" 579 | source = "registry+https://github.com/rust-lang/crates.io-index" 580 | checksum = "d326610f408c7a4eb6f51c37c330e496b08506c9457c9d34287ecc38809fb060" 581 | dependencies = [ 582 | "term_size", 583 | "unicode-width", 584 | ] 585 | 586 | [[package]] 587 | name = "time" 588 | version = "0.1.43" 589 | source = "registry+https://github.com/rust-lang/crates.io-index" 590 | checksum = "ca8a50ef2360fbd1eeb0ecd46795a87a19024eb4b53c5dc916ca1fd95fe62438" 591 | dependencies = [ 592 | "libc", 593 | "winapi", 594 | ] 595 | 596 | [[package]] 597 | name = "typenum" 598 | version = "1.14.0" 599 | source = "registry+https://github.com/rust-lang/crates.io-index" 600 | checksum = "b63708a265f51345575b27fe43f9500ad611579e764c79edbc2037b1121959ec" 601 | 602 | [[package]] 603 | name = "unicode-segmentation" 604 | version = "1.8.0" 605 | source = "registry+https://github.com/rust-lang/crates.io-index" 606 | checksum = "8895849a949e7845e06bd6dc1aa51731a103c42707010a5b591c0038fb73385b" 607 | 608 | [[package]] 609 | name = "unicode-width" 610 | version = "0.1.9" 611 | source = "registry+https://github.com/rust-lang/crates.io-index" 612 | checksum = "3ed742d4ea2bd1176e236172c8429aaf54486e7ac098db29ffe6529e0ce50973" 613 | 614 | [[package]] 615 | name = "unicode-xid" 616 | version = "0.2.2" 617 | source = "registry+https://github.com/rust-lang/crates.io-index" 618 | checksum = "8ccb82d61f80a663efe1f787a51b16b5a51e3314d6ac365b08639f52387b33f3" 619 | 620 | [[package]] 621 | name = "unix_mode" 622 | version = "0.1.3" 623 | source = "registry+https://github.com/rust-lang/crates.io-index" 624 | checksum = "35abed4630bb800f02451a7428205d1f37b8e125001471bfab259beee6a587ed" 625 | 626 | [[package]] 627 | name = "vec_map" 628 | version = "0.8.2" 629 | source = "registry+https://github.com/rust-lang/crates.io-index" 630 | checksum = "f1bddf1187be692e79c5ffeab891132dfb0f236ed36a43c7ed39f1165ee20191" 631 | 632 | [[package]] 633 | name = "version_check" 634 | version = "0.9.3" 635 | source = "registry+https://github.com/rust-lang/crates.io-index" 636 | checksum = "5fecdca9a5291cc2b8dcf7dc02453fee791a280f3743cb0905f8822ae463b3fe" 637 | 638 | [[package]] 639 | name = "wasi" 640 | version = "0.10.2+wasi-snapshot-preview1" 641 | source = "registry+https://github.com/rust-lang/crates.io-index" 642 | checksum = "fd6fbd9a79829dd1ad0cc20627bf1ed606756a7f77edff7b66b7064f9cb327c6" 643 | 644 | [[package]] 645 | name = "winapi" 646 | version = "0.3.9" 647 | source = "registry+https://github.com/rust-lang/crates.io-index" 648 | checksum = "5c839a674fcd7a98952e593242ea400abe93992746761e38641405d28b00f419" 649 | dependencies = [ 650 | "winapi-i686-pc-windows-gnu", 651 | "winapi-x86_64-pc-windows-gnu", 652 | ] 653 | 654 | [[package]] 655 | name = "winapi-i686-pc-windows-gnu" 656 | version = "0.4.0" 657 | source = "registry+https://github.com/rust-lang/crates.io-index" 658 | checksum = "ac3b87c63620426dd9b991e5ce0329eff545bccbbb34f3be09ff6fb6ab51b7b6" 659 | 660 | [[package]] 661 | name = "winapi-x86_64-pc-windows-gnu" 662 | version = "0.4.0" 663 | source = "registry+https://github.com/rust-lang/crates.io-index" 664 | checksum = "712e227841d057c1ee1cd2fb22fa7e5a5461ae8e48fa2ca79ec42cfc1931183f" 665 | -------------------------------------------------------------------------------- /Cargo.toml: -------------------------------------------------------------------------------- 1 | [package] 2 | authors = ["Martin Pool "] 3 | categories = [ 4 | "filesystem", 5 | "network-programming", 6 | ] 7 | description = "[pre-alpha] Wire-compatible rsync client" 8 | edition = "2018" 9 | license = "Apache-2.0" 10 | name = "rsyn" 11 | publish = true 12 | version = "0.0.2" 13 | repository = "https://github.com/sourcefrog/rsyn" 14 | homepage = "https://github.com/sourcefrog/rsyn/blob/master/README.md" 15 | readme = "README.md" 16 | 17 | [dependencies] 18 | anyhow = "1.0.28" 19 | chrono = "0.4.11" 20 | crossbeam = "0.7.3" 21 | hex = "0.4.2" 22 | lazy_static = "1.4.0" 23 | log = "0.4" 24 | md4 = "0.8.0" 25 | regex = "1.3.7" 26 | shell-words = "1.0.0" 27 | tempfile = "3.1.0" 28 | 29 | [dependencies.unix_mode] 30 | version = "0.1.3" 31 | # path = "../unix_mode" 32 | 33 | [dependencies.fern] 34 | version = "0.6" 35 | features = [ 36 | "colored", 37 | ] 38 | 39 | [dependencies.structopt] 40 | features = [ 41 | "wrap_help", 42 | "suggestions", 43 | ] 44 | version = "0.3" 45 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | 2 | Apache License 3 | Version 2.0, January 2004 4 | http://www.apache.org/licenses/ 5 | 6 | TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 7 | 8 | 1. Definitions. 9 | 10 | "License" shall mean the terms and conditions for use, reproduction, 11 | and distribution as defined by Sections 1 through 9 of this document. 12 | 13 | "Licensor" shall mean the copyright owner or entity authorized by 14 | the copyright owner that is granting the License. 15 | 16 | "Legal Entity" shall mean the union of the acting entity and all 17 | other entities that control, are controlled by, or are under common 18 | control with that entity. For the purposes of this definition, 19 | "control" means (i) the power, direct or indirect, to cause the 20 | direction or management of such entity, whether by contract or 21 | otherwise, or (ii) ownership of fifty percent (50%) or more of the 22 | outstanding shares, or (iii) beneficial ownership of such entity. 23 | 24 | "You" (or "Your") shall mean an individual or Legal Entity 25 | exercising permissions granted by this License. 26 | 27 | "Source" form shall mean the preferred form for making modifications, 28 | including but not limited to software source code, documentation 29 | source, and configuration files. 30 | 31 | "Object" form shall mean any form resulting from mechanical 32 | transformation or translation of a Source form, including but 33 | not limited to compiled object code, generated documentation, 34 | and conversions to other media types. 35 | 36 | "Work" shall mean the work of authorship, whether in Source or 37 | Object form, made available under the License, as indicated by a 38 | copyright notice that is included in or attached to the work 39 | (an example is provided in the Appendix below). 40 | 41 | "Derivative Works" shall mean any work, whether in Source or Object 42 | form, that is based on (or derived from) the Work and for which the 43 | editorial revisions, annotations, elaborations, or other modifications 44 | represent, as a whole, an original work of authorship. For the purposes 45 | of this License, Derivative Works shall not include works that remain 46 | separable from, or merely link (or bind by name) to the interfaces of, 47 | the Work and Derivative Works thereof. 48 | 49 | "Contribution" shall mean any work of authorship, including 50 | the original version of the Work and any modifications or additions 51 | to that Work or Derivative Works thereof, that is intentionally 52 | submitted to Licensor for inclusion in the Work by the copyright owner 53 | or by an individual or Legal Entity authorized to submit on behalf of 54 | the copyright owner. For the purposes of this definition, "submitted" 55 | means any form of electronic, verbal, or written communication sent 56 | to the Licensor or its representatives, including but not limited to 57 | communication on electronic mailing lists, source code control systems, 58 | and issue tracking systems that are managed by, or on behalf of, the 59 | Licensor for the purpose of discussing and improving the Work, but 60 | excluding communication that is conspicuously marked or otherwise 61 | designated in writing by the copyright owner as "Not a Contribution." 62 | 63 | "Contributor" shall mean Licensor and any individual or Legal Entity 64 | on behalf of whom a Contribution has been received by Licensor and 65 | subsequently incorporated within the Work. 66 | 67 | 2. Grant of Copyright License. Subject to the terms and conditions of 68 | this License, each Contributor hereby grants to You a perpetual, 69 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 70 | copyright license to reproduce, prepare Derivative Works of, 71 | publicly display, publicly perform, sublicense, and distribute the 72 | Work and such Derivative Works in Source or Object form. 73 | 74 | 3. Grant of Patent License. Subject to the terms and conditions of 75 | this License, each Contributor hereby grants to You a perpetual, 76 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 77 | (except as stated in this section) patent license to make, have made, 78 | use, offer to sell, sell, import, and otherwise transfer the Work, 79 | where such license applies only to those patent claims licensable 80 | by such Contributor that are necessarily infringed by their 81 | Contribution(s) alone or by combination of their Contribution(s) 82 | with the Work to which such Contribution(s) was submitted. If You 83 | institute patent litigation against any entity (including a 84 | cross-claim or counterclaim in a lawsuit) alleging that the Work 85 | or a Contribution incorporated within the Work constitutes direct 86 | or contributory patent infringement, then any patent licenses 87 | granted to You under this License for that Work shall terminate 88 | as of the date such litigation is filed. 89 | 90 | 4. Redistribution. You may reproduce and distribute copies of the 91 | Work or Derivative Works thereof in any medium, with or without 92 | modifications, and in Source or Object form, provided that You 93 | meet the following conditions: 94 | 95 | (a) You must give any other recipients of the Work or 96 | Derivative Works a copy of this License; and 97 | 98 | (b) You must cause any modified files to carry prominent notices 99 | stating that You changed the files; and 100 | 101 | (c) You must retain, in the Source form of any Derivative Works 102 | that You distribute, all copyright, patent, trademark, and 103 | attribution notices from the Source form of the Work, 104 | excluding those notices that do not pertain to any part of 105 | the Derivative Works; and 106 | 107 | (d) If the Work includes a "NOTICE" text file as part of its 108 | distribution, then any Derivative Works that You distribute must 109 | include a readable copy of the attribution notices contained 110 | within such NOTICE file, excluding those notices that do not 111 | pertain to any part of the Derivative Works, in at least one 112 | of the following places: within a NOTICE text file distributed 113 | as part of the Derivative Works; within the Source form or 114 | documentation, if provided along with the Derivative Works; or, 115 | within a display generated by the Derivative Works, if and 116 | wherever such third-party notices normally appear. The contents 117 | of the NOTICE file are for informational purposes only and 118 | do not modify the License. You may add Your own attribution 119 | notices within Derivative Works that You distribute, alongside 120 | or as an addendum to the NOTICE text from the Work, provided 121 | that such additional attribution notices cannot be construed 122 | as modifying the License. 123 | 124 | You may add Your own copyright statement to Your modifications and 125 | may provide additional or different license terms and conditions 126 | for use, reproduction, or distribution of Your modifications, or 127 | for any such Derivative Works as a whole, provided Your use, 128 | reproduction, and distribution of the Work otherwise complies with 129 | the conditions stated in this License. 130 | 131 | 5. Submission of Contributions. Unless You explicitly state otherwise, 132 | any Contribution intentionally submitted for inclusion in the Work 133 | by You to the Licensor shall be under the terms and conditions of 134 | this License, without any additional terms or conditions. 135 | Notwithstanding the above, nothing herein shall supersede or modify 136 | the terms of any separate license agreement you may have executed 137 | with Licensor regarding such Contributions. 138 | 139 | 6. Trademarks. This License does not grant permission to use the trade 140 | names, trademarks, service marks, or product names of the Licensor, 141 | except as required for reasonable and customary use in describing the 142 | origin of the Work and reproducing the content of the NOTICE file. 143 | 144 | 7. Disclaimer of Warranty. Unless required by applicable law or 145 | agreed to in writing, Licensor provides the Work (and each 146 | Contributor provides its Contributions) on an "AS IS" BASIS, 147 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 148 | implied, including, without limitation, any warranties or conditions 149 | of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A 150 | PARTICULAR PURPOSE. You are solely responsible for determining the 151 | appropriateness of using or redistributing the Work and assume any 152 | risks associated with Your exercise of permissions under this License. 153 | 154 | 8. Limitation of Liability. In no event and under no legal theory, 155 | whether in tort (including negligence), contract, or otherwise, 156 | unless required by applicable law (such as deliberate and grossly 157 | negligent acts) or agreed to in writing, shall any Contributor be 158 | liable to You for damages, including any direct, indirect, special, 159 | incidental, or consequential damages of any character arising as a 160 | result of this License or out of the use or inability to use the 161 | Work (including but not limited to damages for loss of goodwill, 162 | work stoppage, computer failure or malfunction, or any and all 163 | other commercial damages or losses), even if such Contributor 164 | has been advised of the possibility of such damages. 165 | 166 | 9. Accepting Warranty or Additional Liability. While redistributing 167 | the Work or Derivative Works thereof, You may choose to offer, 168 | and charge a fee for, acceptance of support, warranty, indemnity, 169 | or other liability obligations and/or rights consistent with this 170 | License. However, in accepting such obligations, You may act only 171 | on Your own behalf and on Your sole responsibility, not on behalf 172 | of any other Contributor, and only if You agree to indemnify, 173 | defend, and hold each Contributor harmless for any liability 174 | incurred by, or claims asserted against, such Contributor by reason 175 | of your accepting any such warranty or additional liability. 176 | 177 | END OF TERMS AND CONDITIONS 178 | 179 | APPENDIX: How to apply the Apache License to your work. 180 | 181 | To apply the Apache License to your work, attach the following 182 | boilerplate notice, with the fields enclosed by brackets "[]" 183 | replaced with your own identifying information. (Don't include 184 | the brackets!) The text should be enclosed in the appropriate 185 | comment syntax for the file format. We also recommend that a 186 | file or class name and description of purpose be included on the 187 | same "printed page" as the copyright notice for easier 188 | identification within third-party archives. 189 | 190 | Copyright [yyyy] [name of copyright owner] 191 | 192 | Licensed under the Apache License, Version 2.0 (the "License"); 193 | you may not use this file except in compliance with the License. 194 | You may obtain a copy of the License at 195 | 196 | http://www.apache.org/licenses/LICENSE-2.0 197 | 198 | Unless required by applicable law or agreed to in writing, software 199 | distributed under the License is distributed on an "AS IS" BASIS, 200 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 201 | See the License for the specific language governing permissions and 202 | limitations under the License. 203 | -------------------------------------------------------------------------------- /NEWS.md: -------------------------------------------------------------------------------- 1 | # rsyn release notes 2 | 3 | ## 0.0.2 (NOT RELEASED YET) 4 | 5 | Various API changes, including: 6 | 7 | * Transfer operations return a new `Summary` object including counters of how 8 | much work was done, and of non-fatal errors. 9 | 10 | ## 0.0.1 (2020-05-13) 11 | 12 | Features: 13 | 14 | * List a directory from rsync, either as a local subprocess or over ssh. 15 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Wire-compatible rsync client in Rust 2 | 3 | [![crates.io](https://img.shields.io/crates/v/rsyn.svg)](https://crates.io/crates/rsyn) 4 | [![docs.rs](https://docs.rs/rsyn/badge.svg)](https://docs.rs/rsyn) 5 | [![Tests](https://github.com/sourcefrog/rsyn/workflows/rust/badge.svg)](https://github.com/sourcefrog/rsyn/actions?query=workflow%3Arust) 6 | 7 | `rsyn` reimplements part of the rsync network protocol in pure Rust. (It's 8 | "rsync with no C.") 9 | 10 | **NOTE: `rsyn` is an incomplete and inactive experiment. It's not currently a 11 | useful replacement for rsync.** 12 | 13 | rsyn supports protocol version 27, which is supported by rsync versions from 14 | 2.6.0 (released in 2004) onwards, and by openrsync. 15 | 16 | ## Install 17 | 18 | 1. [Install Rust](https://rustup.rs/). 19 | 20 | 2. Run 21 | 22 | ```shell 23 | cargo install rsyn 24 | ``` 25 | 26 | To run the interoperability tests (with `cargo test`) you'll need a copy of 27 | rsync installed. 28 | 29 | ## Usage 30 | 31 | `rsyn DIR` prints a recursive listing of the given local directory, by launching 32 | an rsync subprocess and controlling it over a pair of pipes. 33 | 34 | `rsyn USER@HOST:DIR` or `rsyn HOST:DIR` lists a remote directory, connecting to 35 | the rsync server over SSH. 36 | 37 | ## Roadmap 38 | 39 | Progress so far: 40 | 41 | - [x] List a local directory from a local subprocess. 42 | 43 | - [x] List a directory over SSH. 44 | 45 | Intended next steps are: 46 | 47 | - [ ] Copy a directory from rsync over SSH into an empty local directory. 48 | 49 | - [ ] Copy a directory from rsync into a local directory, skipping already 50 | up-to-date files, but downloading the full content of missing or 51 | out-of-date files. 52 | 53 | - [ ] Connect to an rsync daemon (`rsync://`): these talk a different 54 | introductory protocol before starting the main rsync protocol. Support 55 | downloads with the limitations above. 56 | 57 | - [ ] Support incremental rolling-sum and checksum file transfers: the actual 58 | "rsync algorithm". 59 | 60 | - [ ] Support the commonly-used `-a` option. 61 | 62 | - [ ] Upload a directory to rsync over SSH. 63 | 64 | Below this point the ordering is less certain but some options are: 65 | 66 | - [ ] Act as a server for rsync+ssh. In particular, use this to test rsyn 67 | against itself, as well as against rsync. 68 | 69 | - [ ] Act as an `rsync://` daemon. 70 | 71 | - [ ] Support some more selected command line options. 72 | 73 | ## Why do this? 74 | 75 | rsync does by-hand parsing of a complicated binary network protocol in C. 76 | Although that was a reasonble option in the 90s, today it looks dangerous. 77 | Fuzzers find cases where a malicious peer can crash rsync, and worse may be 78 | possible. 79 | 80 | The rsync C code is quite convoluted, with many interacting options and 81 | parameters stored in global variables affecting many different parts of the 82 | control flow, including how structures are encoded and decoded. 83 | 84 | rsync is still fairly widely deployed, and does a good job. A safer 85 | interoperable implementation could be useful. 86 | 87 | And, personally: I contributed to rsync many years ago, and it's interesting to 88 | revisit the space with better tools, and with more experience, and see if I can 89 | do better. 90 | 91 | ## Goals 92 | 93 | - rsyn will interoperate with recent versions of upstream "tridge" rsync, over 94 | (first) rsync+ssh or (later) `rsync://`. 95 | 96 | - rsyn will support commonly-used rsync options and scenarios. The most 97 | important are to transfer files recursively, with mtimes and permissions, with 98 | exclusion patterns. 99 | 100 | - rsyn will offer a clean public library Rust API through which transfers can be 101 | initiated and observed in-process. As is usual for Rust libraries, the API is 102 | not guaranteed to be stable before 1.0. 103 | 104 | - Every command line option in rsyn should have the same meaning as in rsync. 105 | 106 | It's OK if some of the many rsync options are not supported. 107 | 108 | The exception is that rsyn-specific options will start with `--Z` to 109 | distinguish them and avoid collisions. 110 | 111 | - rsyn's test suite should demonstrate interoperability by automatically testing 112 | rsyn against rsync. (Later versions might demonstrate compatibility against 113 | various different versions of rsync, and maybe also against openrsync.) 114 | 115 | - rsyn should have no `unsafe` blocks. (The underlying Rust libraries have some 116 | trusted implementation code and link in some C code.) 117 | 118 | - rsyn will run on Linux, macOS, Windows, and other Unixes, in both 64-bit and 119 | (if the OS supports it) 32-bit mode. 120 | 121 | rsyn will use Rust concurrency structures that are supported everywhere, 122 | rather than rsync's creative application of Unix-isms such as sockets shared 123 | between multiple processes. 124 | 125 | - rsyn should be safe even against an arbitrarily malicious peer. 126 | 127 | In particular, paths received from the peer should be carefully validated to 128 | prevent 129 | [path traversal bugs](https://cwe.mitre.org/data/definitions/1219.html). 130 | 131 | - rsyn should show comparable performance to rsync, in terms of throughput, CPU, 132 | and memory. 133 | 134 | - rsyn should have good test coverage: both unit tests and interoperability 135 | tests. 136 | 137 | - rsyn code should be clean and understandable Rust code. (The rsync code is now 138 | quite convoluted.) rsyn will use Rust type checking to prevent illegal or 139 | unsafe states. Interacting options should be factored into composed types, 140 | rather than forests of `if` statements. 141 | 142 | ### Non-goals 143 | 144 | - rsyn will not necessarily support every single option and feature in rsync. 145 | 146 | rsync has a lot of options, which (at least in the rsync codebase) interact in 147 | complicated ways. Some seem to have niche audiences, or to be obsolete, such 148 | as special support for `rsh` or HP-UX `remsh`. 149 | 150 | - rsyn speaks the protocol defined by rsync's implementation, and does not 151 | aspire to evolve the protocol or to add rsyn-specific upgrades. 152 | 153 | rsync's protocol is already fairly weird and complicated, and was built for a 154 | different environment than exists today. Dramatically new features, in my 155 | view, are better off in a clean-slate protocol. 156 | 157 | - rsyn need not address security weaknesses in the rsync protocol. 158 | 159 | rsync's block-hashing, file-hashing, and daemon mode authentication use MD4, 160 | which is not advisable today. This can't be unilaterally changed by rsyn while 161 | keeping compatibility. 162 | 163 | For sensitive data or writable directories, or really any traffic over 164 | less-than-fully-trusted networks, I'd strongly recommend running rsync over 165 | SSH. 166 | 167 | - rsyn need not generate exactly identical text/log output. 168 | 169 | ## More docs 170 | 171 | * [Release notes](NEWS.md) 172 | 173 | ## Acknowledgements 174 | 175 | Thanks to [Tridge](https://www.samba.org/~tridge/) for his brilliant and 176 | generous mentorship and contributions to open source. 177 | 178 | This project would have been far harder without Kristaps Dzonsons's 179 | documentation of the rsync protocol in the 180 | [openrsync](https://github.com/kristapsdz/openrsync) project. 181 | 182 | ## License 183 | 184 | [Apache 2.0](LICENSE). 185 | 186 | ## Contributing 187 | 188 | I'd love to accept patches to this project. Please read the 189 | [contribution guidelines](CONTRIBUTING.md) and 190 | [code of conduct](CODE_OF_CONDUCT.md). 191 | 192 | ## Disclaimer 193 | 194 | This is not an official Google project. It is not supported by Google, and 195 | Google specifically disclaims all warranties as to its quality, merchantability, 196 | or fitness for a particular purpose. 197 | -------------------------------------------------------------------------------- /doc/protocol-notes.md: -------------------------------------------------------------------------------- 1 | # rsync protocol notes 2 | 3 | ## rsync (C) structure 4 | 5 | Older rsync code, with fewer features, is easier to follow. 6 | 7 | rsyn is trying to support protocol 27, from 2004, the same as openrsync. 8 | 9 | The receiver process forks off a child (in `do_recv`) where the child receives 10 | the files (in `recv_files`) and the parent generates (in `generate_files`.) 11 | 12 | ## Protocol negotiation 13 | 14 | Protocol version is selected (in `exchange_version` and `setup_protocol`) as 15 | the minimum of the protocols offered by the client and server, which makes 16 | sense. 17 | 18 | There is a separate text-mode greeting, including a protocol version, in 19 | `exchange_protocols`, that sends a text string like `"@RSYNCD: %d.%d\n"`. It 20 | seems this is only used in the bare-TCP daemon (in `clientserver.c`) not over 21 | SSH or locally. In `start_inband_exchange` the client sends authentication and 22 | the module and args that it wants to use. 23 | 24 | There's also a concept of "subprotocols", and comments indicate perhaps this is 25 | for pre-release builds. This might not be deployed widely enough to worry 26 | about? This is also handled in `check_sub_protocol`. It basically seems to 27 | downgrade to the prior protocol if the peer offers a subprotocol version that 28 | the the local process doesn't support. 29 | 30 | `compat.c` looks at the `client_info` string to both determine a protocol, and 31 | to find some compatibility flags. This only ever seems to get set from 32 | `shell_cmd`, and that in turn seems to only come from the `--rsh` command line 33 | option. I don't understand how it could end up with the values this seems to 34 | expect, unless perhaps it's passed as a hack in the daemon protocol, without 35 | really representing the rsh command? 36 | 37 | I guess this is set for daemon connections from arguments constructed in 38 | `server_options`. 39 | 40 | ## varint encoding 41 | 42 | The openrsync docs say that a 8-byte long is preceded by a maximum integer, but 43 | it's actually preceded by `(int32) 0xffff_ffff`, in other words -1. (See 44 | `read_longint`. 45 | 46 | In addition to this encoding, there's also `read_varlong` which seems to read a 47 | genuinely-variable length encoding. 48 | -------------------------------------------------------------------------------- /src/bin/rsyn.rs: -------------------------------------------------------------------------------- 1 | // Copyright 2020 Google LLC 2 | // 3 | // Licensed under the Apache License, Version 2.0 (the "License"); 4 | // you may not use this file except in compliance with the License. 5 | // You may obtain a copy of the License at 6 | // 7 | // http://www.apache.org/licenses/LICENSE-2.0 8 | // 9 | // Unless required by applicable law or agreed to in writing, software 10 | // distributed under the License is distributed on an "AS IS" BASIS, 11 | // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | // See the License for the specific language governing permissions and 13 | // limitations under the License. 14 | 15 | //! Command-line program for rsyn, an rsync client in Rust. 16 | 17 | use std::path::PathBuf; 18 | 19 | use anyhow::Context; 20 | use fern::colors::{Color, ColoredLevelConfig}; 21 | #[allow(unused_imports)] 22 | use log::{debug, error, info, trace, warn}; 23 | use structopt::StructOpt; 24 | 25 | use rsyn::{Client, LocalTree, Options, Result}; 26 | 27 | #[derive(Debug, StructOpt)] 28 | #[structopt()] 29 | /// [pre-alpha] Wire-compatible rsync client in Rust. 30 | /// 31 | /// With one PATH argument, lists the contents of that directory. 32 | struct Opt { 33 | /// Location, directory, or file to copy. 34 | source: String, 35 | 36 | /// Local directory to copy to. 37 | destination: Option, 38 | 39 | /// File to send log/debug messages. 40 | #[structopt(long, env = "RSYN_LOG_FILE")] 41 | log_file: Option, 42 | 43 | /// Shell command to run to start rsync server. 44 | #[structopt(long, env = "RSYN_RSYNC_PATH")] 45 | rsync_path: Option, 46 | 47 | /// Shell command to open a connection to a remote server (default is ssh). 48 | #[structopt(long, short = "e", env = "RSYN_RSH")] 49 | rsh: Option, 50 | 51 | /// Recurse into directories. 52 | #[structopt(long, short = "r")] 53 | recursive: bool, 54 | 55 | /// List files, don't copy them. 56 | #[structopt(long)] 57 | list_only: bool, 58 | 59 | /// Be more verbose. 60 | #[structopt(short = "v", parse(from_occurrences))] 61 | verbose: u32, 62 | } 63 | 64 | impl Opt { 65 | /// Convert command-line options to protocol options. 66 | fn to_options(&self) -> Options { 67 | Options { 68 | recursive: self.recursive, 69 | list_only: self.list_only, 70 | verbose: self.verbose, 71 | rsync_command: self.rsync_path.as_ref().map(|p| { 72 | shell_words::split(&p).expect("Failed to split shell words from rsync_command") 73 | }), 74 | ssh_command: self.rsh.as_ref().map(|p| { 75 | shell_words::split(&p).expect("Failed to split shell words from ssh_command") 76 | }), 77 | } 78 | } 79 | } 80 | 81 | fn main() -> Result<()> { 82 | let opt = Opt::from_args(); 83 | 84 | configure_logging(&opt)?; 85 | 86 | let mut client = Client::from_str(&opt.source).expect("Failed to parse path"); 87 | *client.mut_options() = opt.to_options(); 88 | if let Some(destination) = opt.destination { 89 | let (_file_list, _summary) = client.download(&mut LocalTree::new(&destination))?; 90 | } else { 91 | let (file_list, _summary) = client.list_files()?; 92 | for entry in file_list { 93 | println!("{}", &entry) 94 | } 95 | } 96 | debug!("That's all folks!"); 97 | Ok(()) 98 | } 99 | 100 | // Configure the logger: send everything to the log file (if there is one), and 101 | // send info and above to the console. 102 | fn configure_logging(opt: &Opt) -> Result<()> { 103 | // TODO: Maybe an option to turn this up to 'trace' verbosity? It's a bit 104 | // loud to have on by default. 105 | let mut to_file = 106 | fern::Dispatch::new() 107 | .level(log::LevelFilter::Debug) 108 | .format(move |out, message, record| { 109 | out.finish(format_args!( 110 | "[{}] [{:<6}] [{:<20}] {}", 111 | chrono::Local::now().format("%b %d %H:%M:%S%.3f"), 112 | record.level(), 113 | record.target(), 114 | message, 115 | )) 116 | }); 117 | if let Some(ref log_file) = opt.log_file { 118 | to_file = to_file.chain(fern::log_file(log_file).context("Failed to open log file")?); 119 | } 120 | 121 | let colors = ColoredLevelConfig::new() 122 | .debug(Color::Cyan) 123 | .trace(Color::Magenta); 124 | let console_level = match opt.verbose { 125 | 0 => log::LevelFilter::Warn, 126 | 1 => log::LevelFilter::Info, 127 | 2 => log::LevelFilter::Debug, 128 | _ => log::LevelFilter::Trace, 129 | }; 130 | let to_console = fern::Dispatch::new() 131 | .format(move |out, message, record| { 132 | out.finish(format_args!( 133 | "[{:<6}] {}: {}", 134 | colors.color(record.level()), 135 | record.target(), 136 | message 137 | )) 138 | }) 139 | .level(console_level) 140 | .chain(std::io::stderr()); 141 | 142 | fern::Dispatch::new() 143 | .chain(to_console) 144 | .chain(to_file) 145 | .apply() 146 | .expect("Failed to configure logger"); 147 | Ok(()) 148 | } 149 | 150 | #[cfg(test)] 151 | mod test { 152 | use super::*; 153 | 154 | #[test] 155 | fn rsync_path_option() { 156 | let opt = Opt::from_iter(&[ 157 | "rsyn", 158 | "--rsync-path=rsync --wibble --wobble", 159 | "-vv", 160 | "/example", 161 | ]); 162 | assert_eq!( 163 | opt.rsync_path.as_deref().unwrap(), 164 | "rsync --wibble --wobble" 165 | ); 166 | let options = opt.to_options(); 167 | assert_eq!( 168 | options.rsync_command.unwrap(), 169 | ["rsync", "--wibble", "--wobble"] 170 | ); 171 | } 172 | 173 | #[test] 174 | fn rsh_option() { 175 | let opt = Opt::from_iter(&["rsyn", "--rsh=ssh -OFoo -OBar=123 -v -A", "-vv", "/example"]); 176 | assert!(opt.rsync_path.is_none()); 177 | let options = opt.to_options(); 178 | assert!(options.rsync_command.is_none()); 179 | assert_eq!( 180 | options.ssh_command.unwrap(), 181 | ["ssh", "-OFoo", "-OBar=123", "-v", "-A"] 182 | ); 183 | } 184 | } 185 | -------------------------------------------------------------------------------- /src/client.rs: -------------------------------------------------------------------------------- 1 | // Copyright 2020 Google LLC 2 | // 3 | // Licensed under the Apache License, Version 2.0 (the "License"); 4 | // you may not use this file except in compliance with the License. 5 | // You may obtain a copy of the License at 6 | // 7 | // http://www.apache.org/licenses/LICENSE-2.0 8 | // 9 | // Unless required by applicable law or agreed to in writing, software 10 | // distributed under the License is distributed on an "AS IS" BASIS, 11 | // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | // See the License for the specific language governing permissions and 13 | // limitations under the License. 14 | 15 | //! A client that connects to an rsync server. 16 | 17 | use std::ffi::OsString; 18 | use std::path::Path; 19 | use std::process::{Command, Stdio}; 20 | 21 | use anyhow::Context; 22 | use lazy_static::lazy_static; 23 | #[allow(unused_imports)] 24 | use log::{debug, error, info, trace, warn}; 25 | use regex::Regex; 26 | 27 | use crate::connection::Connection; 28 | use crate::{FileList, LocalTree, Options, Result, Summary}; 29 | 30 | /// SSH command name, to start it as a subprocess. 31 | const DEFAULT_SSH_COMMAND: &str = "ssh"; 32 | /// rsync command name, to start it as a subprocess either locally or remotely. 33 | const DEFAULT_RSYNC_COMMAND: &str = "rsync"; 34 | 35 | /// A client for an rsync server. 36 | /// 37 | /// The client is built with information about the location of the server and 38 | /// what options to use, and then transfer operations can be invoked. 39 | /// 40 | /// Clients can be parsed from strings: 41 | /// ``` 42 | /// let client = rsyn::Client::from_str("rsync.example.com::module") 43 | /// .expect("Parse failed"); 44 | /// ``` 45 | /// 46 | /// Or constructed: 47 | /// ``` 48 | /// let client = rsyn::Client::local("./src"); 49 | /// let client = rsyn::Client::ssh(Some("user"), "host.example.com", "./src"); 50 | /// ``` 51 | #[derive(Eq, PartialEq, Clone, Debug)] 52 | pub struct Client { 53 | /// Root path to pass to the server. 54 | path: OsString, 55 | 56 | /// How to start the SSH transport, if applicable. 57 | ssh: Option, 58 | 59 | /// Use the rsync daemon wrapper protocol. 60 | /// 61 | /// This can be done either over bare TCP, or wrapped in SSH. 62 | /// (See "USING RSYNC-DAEMON FEATURES VIA A REMOTE-SHELL CONNECTION" in the 63 | /// rsync manual.) 64 | daemon: Option, 65 | 66 | /// Protocol / remote command line options. 67 | options: Options, 68 | } 69 | 70 | #[derive(Clone, Eq, PartialEq, Debug)] 71 | struct Daemon { 72 | user: Option, 73 | host: String, 74 | port: Option, 75 | } 76 | 77 | /// Describes how to start an SSH subprocess. 78 | #[derive(Clone, Eq, PartialEq, Debug)] 79 | struct Ssh { 80 | user: Option, 81 | host: String, 82 | } 83 | 84 | impl Client { 85 | /// Builds a `Client` that, when connected, starts an `rsync --server` subprocess 86 | /// on the local machine. 87 | /// 88 | /// This is primarily useful for testing, or copying files locally. 89 | pub fn local>(path: P) -> Client { 90 | Client { 91 | path: path.as_ref().as_os_str().into(), 92 | ssh: None, 93 | daemon: None, 94 | options: Options::default(), 95 | } 96 | } 97 | 98 | /// Builds a `Client` that will connect to an rsync server over ssh. 99 | /// 100 | /// This will run an external SSH process, defaulting to `ssh`, controlled 101 | /// by `Options.ssh_command`. 102 | /// 103 | /// If `user` is None, ssh's default username, typically the same as the 104 | /// local user, has effect. 105 | /// 106 | /// `path` is the path on the remote host. 107 | pub fn ssh(user: Option<&str>, host: &str, path: &str) -> Client { 108 | Client { 109 | path: path.into(), 110 | ssh: Some(Ssh { 111 | user: user.map(String::from), 112 | host: host.into(), 113 | }), 114 | daemon: None, 115 | options: Options::default(), 116 | } 117 | } 118 | 119 | /// Mutably borrow this client's `Options`. 120 | pub fn mut_options(&mut self) -> &mut Options { 121 | &mut self.options 122 | } 123 | 124 | /// Replace this client's `Options`. 125 | pub fn set_options(&mut self, options: Options) -> &mut Self { 126 | self.options = options; 127 | self 128 | } 129 | 130 | /// Set the `recursive` option. 131 | pub fn set_recursive(&mut self, recursive: bool) -> &mut Self { 132 | self.options.recursive = recursive; 133 | self 134 | } 135 | 136 | /// Set the `verbose` option. 137 | pub fn set_verbose(&mut self, verbose: u32) -> &mut Self { 138 | self.options.verbose = verbose; 139 | self 140 | } 141 | 142 | /// Builds the arguments to start a connection subcommand, including the 143 | /// command name. 144 | fn build_args(&self) -> Vec { 145 | let mut v = Vec::::new(); 146 | let mut push_str = |s: &str| v.push(s.into()); 147 | if let Some(ref ssh) = self.ssh { 148 | if let Some(args) = &self.options.ssh_command { 149 | for arg in args { 150 | push_str(arg) 151 | } 152 | } else { 153 | push_str(DEFAULT_SSH_COMMAND) 154 | } 155 | if let Some(ref user) = ssh.user { 156 | push_str("-l"); 157 | push_str(user); 158 | } 159 | push_str(&ssh.host); 160 | }; 161 | if let Some(rsync_command) = &self.options.rsync_command { 162 | for arg in rsync_command { 163 | push_str(arg) 164 | } 165 | } else { 166 | push_str(DEFAULT_RSYNC_COMMAND) 167 | } 168 | push_str("--server"); 169 | push_str("--sender"); 170 | if self.options.verbose > 0 { 171 | let mut o = "-".to_string(); 172 | for _ in 0..self.options.verbose { 173 | o.push('v'); 174 | } 175 | push_str(&o); 176 | } 177 | if self.options.list_only { 178 | push_str("--list-only") 179 | } 180 | if self.options.recursive { 181 | push_str("-r") 182 | } 183 | if self.path.is_empty() { 184 | push_str(".") 185 | } else { 186 | v.push(self.path.clone()) 187 | } 188 | v 189 | } 190 | 191 | /// List files from the remote server. 192 | /// 193 | /// This implicitly sets the `list_only` option. 194 | pub fn list_files(&mut self) -> Result<(FileList, Summary)> { 195 | self.download(&mut LocalTree::new("/dev/null")) // TODO: Clean LocalTree::null() 196 | } 197 | 198 | /// Download from the server into a local tree. 199 | pub fn download(&mut self, local_tree: &mut LocalTree) -> Result<(FileList, Summary)> { 200 | self.connect() 201 | .context("Failed to connect")? 202 | .receive(local_tree) 203 | .context("Failed to list files") 204 | } 205 | 206 | /// Opens a connection using the previously configured destination and options. 207 | /// 208 | /// The `Client` can be opened any number of times, but each `Connection` 209 | /// can only do a single operation. 210 | fn connect(&self) -> Result { 211 | if self.daemon.is_some() { 212 | todo!("daemon mode is not implemented yet"); 213 | } 214 | let mut args = self.build_args(); 215 | info!("Run connection command {:?}", &args); 216 | let mut command = Command::new(args.remove(0)); 217 | command.args(args); 218 | command.stdin(Stdio::piped()); 219 | command.stdout(Stdio::piped()); 220 | let mut child = command 221 | .spawn() 222 | .with_context(|| format!("Failed to launch rsync subprocess {:?}", command))?; 223 | 224 | let r = Box::new(child.stdout.take().expect("Child has no stdout")); 225 | let w = Box::new(child.stdin.take().expect("Child has no stdin")); 226 | 227 | Connection::handshake(r, w, child, self.options.clone()) 228 | } 229 | 230 | /// Builds a Client from a path, URL, or SFTP-like path. 231 | /// 232 | /// ``` 233 | /// let client = rsyn::Client::from_str("rsync.example.com::module") 234 | /// .expect("Parse failed"); 235 | /// ``` 236 | #[allow(clippy::should_implement_trait)] 237 | // This isn't in FromStr because construction doesn't seem exactly like 238 | // parsing, and because this avoids clients needing to import FromStr. 239 | pub fn from_str(s: &str) -> Result { 240 | lazy_static! { 241 | static ref SFTP_RE: Regex = Regex::new( 242 | r"^(?x) 243 | ((?P[^@:]+)@)? 244 | (?P[^:@]+): 245 | (?P:)? # maybe a second colon, to indicate --daemon 246 | (?P.*) # path; may be absolute or relative 247 | $", 248 | ) 249 | .unwrap(); 250 | static ref URL_RE: Regex = Regex::new( 251 | r"^(?x) 252 | rsync:// 253 | ((?P[^@:]+)@)? 254 | (?P[^:/]+) 255 | (:(?P\d+))? 256 | / 257 | (?P.*) 258 | $", 259 | ) 260 | .unwrap(); 261 | } 262 | if let Some(caps) = URL_RE.captures(s) { 263 | Ok(Client { 264 | daemon: Some(Daemon { 265 | host: caps["host"].into(), 266 | user: caps.name("user").map(|m| m.as_str().to_string()), 267 | port: caps.name("port").map(|p| p.as_str().parse().unwrap()), 268 | }), 269 | path: caps["path"].into(), 270 | ssh: None, 271 | options: Options::default(), 272 | }) 273 | } else if let Some(caps) = SFTP_RE.captures(s) { 274 | if caps.name("colon").is_some() { 275 | Ok(Client { 276 | path: caps["path"].into(), 277 | daemon: Some(Daemon { 278 | user: caps.name("user").map(|m| m.as_str().to_string()), 279 | host: caps["host"].into(), 280 | port: None, 281 | }), 282 | ssh: None, 283 | options: Options::default(), 284 | }) 285 | } else { 286 | Ok(Client { 287 | path: caps["path"].into(), 288 | ssh: Some(Ssh { 289 | user: caps.name("user").map(|m| m.as_str().to_string()), 290 | host: caps["host"].into(), 291 | }), 292 | daemon: None, 293 | options: Options::default(), 294 | }) 295 | } 296 | } else { 297 | // Assume it's just a path. 298 | Ok(Client { 299 | path: s.into(), 300 | ssh: None, 301 | daemon: None, 302 | options: Options::default(), 303 | }) 304 | } 305 | } 306 | } 307 | 308 | #[cfg(test)] 309 | mod test { 310 | use super::*; 311 | 312 | #[test] 313 | fn parse_sftp_style_without_user() { 314 | let client = Client::from_str("bilbo:/home/www").unwrap(); 315 | assert_eq!( 316 | client, 317 | Client { 318 | ssh: Some(Ssh { 319 | user: None, 320 | host: "bilbo".into(), 321 | }), 322 | path: "/home/www".into(), 323 | daemon: None, 324 | options: Options::default(), 325 | } 326 | ); 327 | } 328 | 329 | #[test] 330 | fn parse_sftp_style_with_user() { 331 | let client = Client::from_str("mbp@bilbo:/home/www").unwrap(); 332 | assert_eq!( 333 | client, 334 | Client { 335 | ssh: Some(Ssh { 336 | user: Some("mbp".to_string()), 337 | host: "bilbo".to_string(), 338 | }), 339 | path: "/home/www".into(), 340 | daemon: None, 341 | options: Options::default(), 342 | } 343 | ); 344 | } 345 | 346 | #[test] 347 | fn parse_daemon_simple() { 348 | let client = Client::from_str("rsync.samba.org::foo").unwrap(); 349 | assert_eq!( 350 | client, 351 | Client { 352 | path: "foo".into(), 353 | ssh: None, 354 | daemon: Some(Daemon { 355 | host: "rsync.samba.org".into(), 356 | user: None, 357 | port: None, 358 | }), 359 | options: Options::default(), 360 | } 361 | ); 362 | } 363 | 364 | #[test] 365 | fn parse_daemon_with_user() { 366 | let client = Client::from_str("rsync@rsync.samba.org::meat/bread/wine").unwrap(); 367 | assert_eq!( 368 | client, 369 | Client { 370 | path: "meat/bread/wine".into(), 371 | ssh: None, 372 | daemon: Some(Daemon { 373 | host: "rsync.samba.org".into(), 374 | user: Some("rsync".into()), 375 | port: None, 376 | }), 377 | options: Options::default(), 378 | } 379 | ); 380 | } 381 | 382 | #[test] 383 | fn parse_rsync_url() { 384 | let client = Client::from_str("rsync://rsync.samba.org/foo").unwrap(); 385 | assert_eq!( 386 | client, 387 | Client { 388 | path: "foo".into(), 389 | ssh: None, 390 | daemon: Some(Daemon { 391 | host: "rsync.samba.org".into(), 392 | user: None, 393 | port: None, 394 | }), 395 | options: Options::default(), 396 | } 397 | ); 398 | } 399 | 400 | #[test] 401 | fn parse_rsync_url_with_username() { 402 | let client = Client::from_str("rsync://anon@rsync.samba.org/foo").unwrap(); 403 | assert_eq!( 404 | client, 405 | Client { 406 | path: "foo".into(), 407 | ssh: None, 408 | daemon: Some(Daemon { 409 | host: "rsync.samba.org".into(), 410 | user: Some("anon".into()), 411 | port: None, 412 | }), 413 | options: Options::default(), 414 | } 415 | ); 416 | } 417 | 418 | #[test] 419 | fn parse_rsync_url_with_username_and_port() { 420 | let client = 421 | Client::from_str("rsync://anon@rsync.samba.org:8370/alpha/beta/gamma").unwrap(); 422 | assert_eq!( 423 | client, 424 | Client { 425 | path: "alpha/beta/gamma".into(), 426 | ssh: None, 427 | daemon: Some(Daemon { 428 | host: "rsync.samba.org".into(), 429 | user: Some("anon".into()), 430 | port: Some(8370), 431 | }), 432 | options: Options::default(), 433 | } 434 | ); 435 | } 436 | 437 | #[test] 438 | fn parse_simple_path() { 439 | let client = Client::from_str("/usr/local/foo").unwrap(); 440 | assert_eq!( 441 | client, 442 | Client { 443 | path: "/usr/local/foo".into(), 444 | ssh: None, 445 | daemon: None, 446 | options: Options::default(), 447 | } 448 | ); 449 | } 450 | 451 | #[test] 452 | fn build_local_args() { 453 | let args = Client::local("./src").set_recursive(true).build_args(); 454 | assert_eq!(args, vec!["rsync", "--server", "--sender", "-r", "./src"],); 455 | } 456 | 457 | #[test] 458 | fn build_local_args_with_rsync_path() { 459 | let args = Client::local("testdir") 460 | .set_options(Options { 461 | rsync_command: Some(vec!["/opt/rsync/rsync-3.1415".to_owned()]), 462 | ..Options::default() 463 | }) 464 | .build_args(); 465 | assert_eq!( 466 | args, 467 | ["/opt/rsync/rsync-3.1415", "--server", "--sender", "testdir"], 468 | ); 469 | } 470 | 471 | #[test] 472 | fn build_local_args_verbose() { 473 | let mut client = Client::local("./src"); 474 | client.set_verbose(3); 475 | let args = client.build_args(); 476 | assert_eq!(args, ["rsync", "--server", "--sender", "-vvv", "./src"],); 477 | } 478 | 479 | #[test] 480 | fn build_ssh_args() { 481 | // Actually running SSH is a bit hard to test hermetically, but let's 482 | // at least check the command lines are plausible. 483 | 484 | let client = Client::ssh(None, "samba.org", "/home/mbp"); 485 | let args = client.build_args(); 486 | assert_eq!( 487 | args, 488 | [ 489 | "ssh", 490 | "samba.org", 491 | "rsync", 492 | "--server", 493 | "--sender", 494 | "/home/mbp" 495 | ], 496 | ); 497 | } 498 | 499 | #[test] 500 | fn build_ssh_args_with_user() { 501 | let mut client = Client::ssh(Some("mbp"), "samba.org", "/home/mbp"); 502 | { 503 | let mut options = client.mut_options(); 504 | options.recursive = true; 505 | options.list_only = true; 506 | } 507 | let args = client.build_args(); 508 | assert_eq!( 509 | args, 510 | [ 511 | "ssh", 512 | "-l", 513 | "mbp", 514 | "samba.org", 515 | "rsync", 516 | "--server", 517 | "--sender", 518 | "--list-only", 519 | "-r", 520 | "/home/mbp" 521 | ], 522 | ); 523 | } 524 | 525 | #[test] 526 | fn build_ssh_args_with_ssh_command() { 527 | let ssh_args = ["/opt/openssh/ssh", "-A", "-DFoo=bar qux"] 528 | .iter() 529 | .map(|s| s.to_string()) 530 | .collect(); 531 | let args = Client::from_str("mbp@bilbo:/home/www") 532 | .unwrap() 533 | .set_options(Options { 534 | ssh_command: Some(ssh_args), 535 | ..Options::default() 536 | }) 537 | .build_args(); 538 | assert_eq!( 539 | args, 540 | [ 541 | "/opt/openssh/ssh", 542 | "-A", 543 | "-DFoo=bar qux", 544 | "-l", 545 | "mbp", 546 | "bilbo", 547 | "rsync", 548 | "--server", 549 | "--sender", 550 | "/home/www", 551 | ] 552 | ); 553 | } 554 | 555 | /// SSH with no path should say '.', typically to look in the home 556 | /// directory. 557 | #[test] 558 | fn build_ssh_args_for_default_directory() { 559 | let mut client = Client::from_str("example-host:").unwrap(); 560 | client.mut_options().list_only = true; 561 | let args = client.build_args(); 562 | assert_eq!( 563 | args, 564 | [ 565 | "ssh", 566 | "example-host", 567 | "rsync", 568 | "--server", 569 | "--sender", 570 | "--list-only", 571 | "." 572 | ], 573 | ); 574 | } 575 | 576 | /// Daemon mode is not implemented yet. 577 | #[test] 578 | #[should_panic] 579 | fn daemon_connection_unimplemented() { 580 | Client::from_str("rsync.example.com::example") 581 | .unwrap() 582 | .connect() 583 | .unwrap(); 584 | } 585 | } 586 | -------------------------------------------------------------------------------- /src/connection.rs: -------------------------------------------------------------------------------- 1 | // Copyright 2020 Google LLC 2 | // 3 | // Licensed under the Apache License, Version 2.0 (the "License"); 4 | // you may not use this file except in compliance with the License. 5 | // You may obtain a copy of the License at 6 | // 7 | // http://www.apache.org/licenses/LICENSE-2.0 8 | // 9 | // Unless required by applicable law or agreed to in writing, software 10 | // distributed under the License is distributed on an "AS IS" BASIS, 11 | // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | // See the License for the specific language governing permissions and 13 | // limitations under the License. 14 | 15 | //! A connection to an rsync server. 16 | 17 | #![allow(unused_imports)] 18 | 19 | use std::convert::TryInto; 20 | use std::io; 21 | use std::io::prelude::*; 22 | use std::io::ErrorKind; 23 | use std::path::Path; 24 | use std::process::{Child, Command, Stdio}; 25 | 26 | use anyhow::{bail, Context, Result}; 27 | use crossbeam::thread; 28 | #[allow(unused_imports)] 29 | use log::{debug, error, info, trace, warn}; 30 | use md4::{Digest, Md4}; 31 | 32 | use crate::flist::{read_file_list, FileEntry, FileList}; 33 | use crate::mux::DemuxRead; 34 | use crate::sums::SumHead; 35 | use crate::varint::{ReadVarint, WriteVarint}; 36 | use crate::{LocalTree, Options, ServerStatistics, Summary}; 37 | 38 | const MY_PROTOCOL_VERSION: i32 = 27; 39 | 40 | /// Connection to an rsync server. 41 | /// 42 | /// Due to the protocol definition, only one transfer (list, send, or receive) 43 | /// can be done per connection. 44 | pub(crate) struct Connection { 45 | rv: ReadVarint, 46 | wv: WriteVarint, 47 | 48 | /// Mutually-agreed rsync protocol version number. 49 | protocol_version: i32, 50 | 51 | /// Permutation to checksums, pushed as a le i32 at the start of file MD4s. 52 | checksum_seed: i32, 53 | 54 | /// The child process carrying this connection. 55 | child: Child, 56 | 57 | /// Connection options, corresponding to a subset of rsync command-line options. 58 | /// 59 | /// The options affect which fields are present or not on the wire. 60 | options: Options, 61 | } 62 | 63 | impl Connection { 64 | /// Start a new connection, by doing the rsync handshake protocol. 65 | /// 66 | /// The public interface is through `Client`. 67 | pub(crate) fn handshake( 68 | r: Box, 69 | w: Box, 70 | child: Child, 71 | options: Options, 72 | ) -> Result { 73 | let mut wv = WriteVarint::new(w); 74 | let mut rv = ReadVarint::new(r); 75 | 76 | wv.write_i32(MY_PROTOCOL_VERSION)?; 77 | let remote_protocol_version = rv.read_i32().unwrap(); 78 | if remote_protocol_version < MY_PROTOCOL_VERSION { 79 | bail!( 80 | "server protocol version {} is too old", 81 | remote_protocol_version 82 | ); 83 | } 84 | // The server and client agree to use the minimum supported version, 85 | // which will now be ours, because we refuse to accept anything 86 | // older. 87 | 88 | let checksum_seed = rv.read_i32().unwrap(); 89 | debug!( 90 | "Connected to server version {}, checksum_seed {:#x}", 91 | remote_protocol_version, checksum_seed 92 | ); 93 | let protocol_version = std::cmp::min(MY_PROTOCOL_VERSION, remote_protocol_version); 94 | debug!("Agreed protocol version {}", protocol_version); 95 | 96 | // Server-to-client is multiplexed; client-to-server is not. 97 | // Pull back the underlying stream and wrap it in a demuxed varint 98 | // encoder. 99 | let rv = ReadVarint::new(Box::new(DemuxRead::new(rv.take()))); 100 | 101 | Ok(Connection { 102 | rv, 103 | wv, 104 | protocol_version, 105 | checksum_seed, 106 | child, 107 | options, 108 | }) 109 | } 110 | 111 | /// Receive files from the server to the given LocalTree. 112 | pub fn receive(mut self, local_tree: &mut LocalTree) -> Result<(FileList, Summary)> { 113 | // Analogous to rsync/receiver.c recv_files(). 114 | // let max_phase = if self.protocol_version >= 29 { 2 } else { 1 }; 115 | let max_phase = 2; 116 | let mut summary = Summary::default(); 117 | 118 | send_empty_exclusions(&mut self.wv)?; 119 | let file_list = read_file_list(&mut self.rv)?; 120 | // TODO: With -o, get uid list. 121 | // TODO: With -g, get gid list. 122 | 123 | if self.protocol_version < 30 { 124 | let io_error_count = self 125 | .rv 126 | .read_i32() 127 | .context("Failed to read server error count")?; 128 | if io_error_count > 0 { 129 | warn!("Server reports {} IO errors", io_error_count); 130 | } 131 | summary.server_flist_io_error_count = io_error_count; 132 | } 133 | 134 | // Server stops here if there were no files. 135 | if file_list.is_empty() { 136 | info!("Server returned no files, so we're done"); 137 | // TODO: Maybe write one -1 here? 138 | self.shutdown(&mut summary)?; 139 | return Ok((file_list, summary)); 140 | } 141 | 142 | for phase in 1..=max_phase { 143 | debug!("Start phase {}", phase); 144 | if phase == 1 && !self.options.list_only { 145 | self.receive_files(&file_list, local_tree, &mut summary)?; 146 | } else { 147 | self.wv 148 | .write_i32(-1) 149 | .context("Failed to send phase transition")?; 150 | assert_eq!(self.rv.read_i32()?, -1); 151 | } 152 | } 153 | 154 | debug!("Send end of sequence"); 155 | self.wv 156 | .write_i32(-1) 157 | .context("Failed to send end-of-sequence marker")?; 158 | // TODO: In later versions (which?) read an end-of-sequence marker? 159 | summary.server_stats = read_server_statistics(&mut self.rv, self.protocol_version) 160 | .context("Failed to read server statistics")?; 161 | 162 | // TODO: In later versions, send a final -1 marker. 163 | self.shutdown(&mut summary)?; 164 | info!("{:#?}", summary); 165 | Ok((file_list, summary)) 166 | } 167 | 168 | /// Download all regular files. 169 | /// 170 | /// Includes sending requests for them (with no basis) and receiving the data. 171 | fn receive_files( 172 | &mut self, 173 | file_list: &[FileEntry], 174 | local_tree: &mut LocalTree, 175 | summary: &mut Summary, 176 | ) -> Result<()> { 177 | // compare to `recv_generator` in generator.c. 178 | assert!(!file_list.is_empty()); 179 | let rv = &mut self.rv; 180 | let wv = &mut self.wv; 181 | let checksum_seed = self.checksum_seed; 182 | thread::scope(|scope| { 183 | scope 184 | .builder() 185 | .name("rsyn_receiver".to_owned()) 186 | .spawn(|_| receive_offered_files(rv, checksum_seed, file_list, local_tree, summary)) 187 | .expect("Failed to spawn receiver thread"); 188 | generate_files(wv, file_list).unwrap(); 189 | }) 190 | .unwrap(); 191 | debug!("receive_files done"); 192 | Ok(()) // TODO: Handle errors from threads correctly 193 | } 194 | 195 | /// Shut down this connection, consuming the object. 196 | /// 197 | /// This isn't the drop method, because it only makes sense to do after 198 | /// the protocol has reached the natural end. 199 | fn shutdown(self, summary: &mut Summary) -> Result<()> { 200 | let Connection { 201 | rv, 202 | wv, 203 | protocol_version: _, 204 | checksum_seed: _, 205 | mut child, 206 | options: _, 207 | } = self; 208 | 209 | rv.check_for_eof()?; 210 | drop(wv); 211 | 212 | // TODO: Should we timeout after a while? 213 | // TODO: Map rsync return codes to messages. 214 | let child_exit_status = child.wait()?; 215 | summary.child_exit_status = Some(child_exit_status); 216 | info!("Child process exited: {}", child_exit_status); 217 | 218 | Ok(()) 219 | } 220 | } 221 | 222 | fn read_server_statistics(rv: &mut ReadVarint, protocol_version: i32) -> Result { 223 | Ok(ServerStatistics { 224 | total_bytes_read: rv.read_i64()?, 225 | total_bytes_written: rv.read_i64()?, 226 | total_file_size: rv.read_i64()?, 227 | flist_build_time: if protocol_version >= 29 { 228 | Some(rv.read_i64()?) 229 | } else { 230 | None 231 | }, 232 | flist_xfer_time: if protocol_version >= 29 { 233 | Some(rv.read_i64()?) 234 | } else { 235 | None 236 | }, 237 | }) 238 | } 239 | 240 | fn send_empty_exclusions(wv: &mut WriteVarint) -> Result<()> { 241 | wv.write_i32(0).context("Failed to send exclusion list") 242 | } 243 | 244 | fn generate_files(wv: &mut WriteVarint, file_list: &[FileEntry]) -> Result<()> { 245 | for (idx, entry) in file_list.iter().enumerate().filter(|(_idx, e)| e.is_file()) { 246 | debug!( 247 | "Send request for file idx {}, name {:?}", 248 | idx, 249 | entry.name_lossy_string() 250 | ); 251 | wv.write_i32(idx.try_into().unwrap())?; 252 | SumHead::zero().write(wv)?; 253 | } 254 | debug!("Generator done"); 255 | wv.write_i32(-1) 256 | .context("Failed to send phase transition")?; 257 | Ok(()) 258 | } 259 | 260 | /// Receive files from the sender until it sends an end-of-phase marker. 261 | fn receive_offered_files( 262 | rv: &mut ReadVarint, 263 | checksum_seed: i32, 264 | file_list: &[FileEntry], 265 | local_tree: &mut LocalTree, 266 | summary: &mut Summary, 267 | ) -> Result<()> { 268 | // Files normally return in the order the receiver requests them, but this isn't guaranteed. 269 | // And if the sender fails to open the file, it just doesn't send any message, it just 270 | // continues to the next one. 271 | loop { 272 | let remote_idx = rv.read_i32()?; 273 | if remote_idx == -1 { 274 | debug!("Received end-of-phase marker"); 275 | return Ok(()); 276 | } 277 | let idx = remote_idx as usize; 278 | if idx >= file_list.len() { 279 | summary.invalid_file_index_count += 1; 280 | error!("Remote file index {} is out of range", remote_idx) 281 | } 282 | receive_file(rv, checksum_seed, &file_list[idx], local_tree, summary)?; 283 | summary.files_received += 1; 284 | } 285 | } 286 | 287 | fn receive_file( 288 | rv: &mut ReadVarint, 289 | checksum_seed: i32, 290 | entry: &FileEntry, 291 | _local_tree: &LocalTree, 292 | summary: &mut Summary, 293 | ) -> Result<()> { 294 | // Like |receive_data|. 295 | let name = entry.name_lossy_string(); 296 | info!("Receive {:?}", name); 297 | let sums = SumHead::read(rv)?; 298 | trace!("Got sums for {:?}: {:?}", name, sums); 299 | let mut hasher = Md4::new(); 300 | hasher.input(checksum_seed.to_le_bytes()); 301 | loop { 302 | // TODO: Specially handle data for deflate mode. 303 | // Like rsync |simple_recv_token|. 304 | let t = rv.read_i32()?; 305 | if t == 0 { 306 | break; 307 | } else if t < 0 { 308 | todo!("Block copy reference") 309 | } else { 310 | let t = t.try_into().unwrap(); 311 | let content = rv.read_byte_string(t)?; 312 | assert_eq!(content.len(), t); 313 | summary.literal_bytes_received += content.len(); 314 | hasher.input(content); 315 | // TODO: Write it to the local tree. 316 | } 317 | } 318 | let remote_md4 = rv.read_byte_string(crate::MD4_SUM_LENGTH)?; 319 | let local_md4 = hasher.result(); 320 | if local_md4[..] != remote_md4[..] { 321 | // TODO: Remember the error, but don't bail out. Try again in phase 2. 322 | summary.whole_file_sum_mismatch_count += 1; 323 | error!( 324 | "MD4 mismatch for {:?}: sender {}, receiver {}", 325 | name, 326 | hex::encode(remote_md4), 327 | hex::encode(local_md4) 328 | ); 329 | } else { 330 | debug!( 331 | "Completed file {:?} with matching MD4 {}", 332 | name, 333 | hex::encode(&remote_md4) 334 | ); 335 | } 336 | Ok(()) 337 | } 338 | -------------------------------------------------------------------------------- /src/flist.rs: -------------------------------------------------------------------------------- 1 | // Copyright 2020 Google LLC 2 | // 3 | // Licensed under the Apache License, Version 2.0 (the "License"); 4 | // you may not use this file except in compliance with the License. 5 | // You may obtain a copy of the License at 6 | // 7 | // http://www.apache.org/licenses/LICENSE-2.0 8 | // 9 | // Unless required by applicable law or agreed to in writing, software 10 | // distributed under the License is distributed on an "AS IS" BASIS, 11 | // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | // See the License for the specific language governing permissions and 13 | // limitations under the License. 14 | 15 | //! File lists and entries. 16 | 17 | use std::convert::TryInto; 18 | use std::fmt; 19 | 20 | use anyhow::{bail, Context}; 21 | use chrono::{Local, TimeZone}; 22 | 23 | #[allow(unused_imports)] 24 | use log::{debug, error, info, trace, warn}; 25 | 26 | use crate::varint::ReadVarint; 27 | use crate::Result; 28 | 29 | // const STATUS_TOP_LEVEL_DIR: u8 = 0x01; 30 | const STATUS_REPEAT_MODE: u8 = 0x02; 31 | // const STATUS_REPEAT_UID: u8 = 0x08; 32 | // const STATUS_REPEAT_GID: u8 = 0x08; 33 | const STATUS_REPEAT_PARTIAL_NAME: u8 = 0x20; 34 | const STATUS_LONG_NAME: u8 = 0x40; 35 | const STATUS_REPEAT_MTIME: u8 = 0x80; 36 | 37 | type ByteString = Vec; 38 | 39 | /// Description of a single file (or directory or symlink etc). 40 | /// 41 | /// The `Display` trait formats an entry like in `ls -l`, and like in rsync 42 | /// directory listings. 43 | #[derive(Clone, Debug, PartialEq, Eq)] 44 | pub struct FileEntry { 45 | // Corresponds to rsync |file_struct|. 46 | /// Name of this file, as a byte string. 47 | name: Vec, 48 | 49 | /// Length of the file, in bytes. 50 | pub file_len: u64, 51 | 52 | /// Unix mode, containing the file type and permissions. 53 | pub mode: u32, 54 | 55 | /// Modification time, in seconds since the Unix epoch. 56 | mtime: u32, 57 | 58 | /// If this is a symlink, the target. 59 | link_target: Option, 60 | // TODO: Other file_struct fields. 61 | // TODO: Work out what |basedir| is and maybe include that. 62 | } 63 | 64 | impl FileEntry { 65 | /// Returns the file name, as a byte string, in the (remote) OS's encoding. 66 | /// 67 | /// rsync doesn't constrain the encoding, so this will typically, but not 68 | /// necessarily be UTF-8. 69 | pub fn name_bytes(&self) -> &[u8] { 70 | &self.name 71 | } 72 | 73 | /// Returns the file name, with un-decodable bytes converted to Unicode 74 | /// replacement characters. 75 | /// 76 | /// For the common case of UTF-8 names, this is simply the name, but 77 | /// if the remote end uses a different encoding the name may be mangled. 78 | /// 79 | /// This is suitable for printing, but might not be suitable for use as a 80 | /// destination file name. 81 | pub fn name_lossy_string(&self) -> std::borrow::Cow<'_, str> { 82 | String::from_utf8_lossy(&self.name) 83 | } 84 | 85 | /// Returns true if this entry describes a plain file. 86 | pub fn is_file(&self) -> bool { 87 | unix_mode::is_file(self.mode) 88 | } 89 | 90 | /// Returns true if this entry describes a directory. 91 | pub fn is_dir(&self) -> bool { 92 | unix_mode::is_dir(self.mode) 93 | } 94 | 95 | /// Returns true if this entry describes a symlink. 96 | pub fn is_symlink(&self) -> bool { 97 | unix_mode::is_symlink(self.mode) 98 | } 99 | 100 | /// Returns the modification time, in seconds since the Unix epoch. 101 | pub fn unix_mtime(&self) -> u32 { 102 | self.mtime 103 | } 104 | 105 | /// Returns the modification time as a chrono::DateTime associated to the 106 | /// local timezone. 107 | pub fn mtime(&self) -> chrono::DateTime { 108 | Local.timestamp(self.mtime as i64, 0) 109 | } 110 | } 111 | 112 | /// Display this entry in a format like that of `ls`, and like `rsync` uses in 113 | /// listing directories: 114 | /// 115 | /// ```text 116 | /// drwxr-x--- 420 2020-05-02 07:25:17 rsyn 117 | /// ``` 118 | /// 119 | /// The modification time is shown in the local timezone. 120 | impl fmt::Display for FileEntry { 121 | fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { 122 | write!( 123 | f, 124 | "{:08} {:11} {:19} {}", 125 | unix_mode::to_string(self.mode), 126 | self.file_len, 127 | self.mtime().format("%Y-%m-%d %H:%M:%S"), 128 | self.name_lossy_string(), 129 | ) 130 | } 131 | } 132 | 133 | /// A list of files returned from a server. 134 | pub type FileList = Vec; 135 | 136 | /// Reads a file list, and then cleans and sorts it. 137 | pub(crate) fn read_file_list(rv: &mut ReadVarint) -> Result { 138 | // Corresponds to rsync |receive_file_entry|. 139 | // TODO: Support receipt of uid and gid with -o, -g. 140 | // TODO: Support devices, links, etc. 141 | // TODO: Sort order changes in different protocol versions. 142 | 143 | let mut file_list = Vec::new(); 144 | while let Some(entry) = receive_file_entry(rv, file_list.last())? { 145 | file_list.push(entry) 146 | } 147 | debug!("End of file list"); 148 | sort_and_dedupe(&mut file_list); 149 | Ok(file_list) 150 | } 151 | 152 | fn receive_file_entry( 153 | rv: &mut ReadVarint, 154 | previous: Option<&FileEntry>, 155 | ) -> Result> { 156 | let status = rv 157 | .read_u8() 158 | .context("Failed to read file entry status byte")?; 159 | trace!("File list status {:#x}", status); 160 | if status == 0 { 161 | return Ok(None); 162 | } 163 | 164 | let inherit_name_bytes = if (status & STATUS_REPEAT_PARTIAL_NAME) != 0 { 165 | rv.read_u8() 166 | .context("Failed to read inherited name bytes")? as usize 167 | } else { 168 | 0 169 | }; 170 | 171 | let name_len = if status & STATUS_LONG_NAME != 0 { 172 | rv.read_i32()? as usize 173 | } else { 174 | rv.read_u8()? as usize 175 | }; 176 | let mut name = rv.read_byte_string(name_len)?; 177 | if inherit_name_bytes > 0 { 178 | let mut new_name = previous.unwrap().name.clone(); 179 | new_name.truncate(inherit_name_bytes); 180 | new_name.append(&mut name); 181 | name = new_name; 182 | } 183 | trace!(" filename: {:?}", String::from_utf8_lossy(&name)); 184 | assert!(!name.is_empty()); 185 | validate_name(&name)?; 186 | 187 | let file_len: u64 = rv 188 | .read_i64()? 189 | .try_into() 190 | .context("Received negative file_len")?; 191 | trace!(" file_len: {}", file_len); 192 | 193 | let mtime = if status & STATUS_REPEAT_MTIME == 0 { 194 | rv.read_i32()? as u32 195 | } else { 196 | previous.unwrap().mtime 197 | }; 198 | trace!(" mtime: {}", mtime); 199 | 200 | let mode = if status & STATUS_REPEAT_MODE == 0 { 201 | rv.read_i32()? as u32 202 | } else { 203 | previous.unwrap().mode 204 | }; 205 | trace!(" mode: {:#o}", mode); 206 | 207 | // TODO: If the relevant options are set, read uid, gid, device, link target. 208 | 209 | Ok(Some(FileEntry { 210 | name, 211 | file_len, 212 | mtime, 213 | mode, 214 | link_target: None, 215 | })) 216 | } 217 | 218 | /// Check that this name is safe to handle, and doesn't seem to include an escape from the 219 | /// directory. 220 | /// 221 | /// The resulting path should only ever be used as relative to a destination directory. 222 | /// 223 | fn validate_name(name: &[u8]) -> Result<()> { 224 | // Compare to rsync |clean_fname| and |sanitize_path|, although this does not 225 | // yet have the behavior of mapping into a pseudo-chroot directory, and it 226 | // only treats bad names as errors. 227 | // 228 | // TODO: Also look for special device files on Windows? 229 | let printable = || String::from_utf8_lossy(name); 230 | if name.is_empty() { 231 | bail!("Invalid name: empty"); 232 | } 233 | if name[0] == b'/' { 234 | bail!("Invalid name: absolute: {:?}", printable()); 235 | } 236 | for part in name.split(|b| *b == b'/') { 237 | if part.is_empty() || part == b".." { 238 | bail!( 239 | "Unsafe file path {:?}: this is either mischief by the sender or a bug", 240 | printable() 241 | ); 242 | } 243 | } 244 | Ok(()) 245 | } 246 | 247 | fn sort_and_dedupe(file_list: &mut Vec) { 248 | // Compare to rsync `file_compare`. 249 | 250 | // In the rsync protocol the receiver gets a list of files from the server in 251 | // arbitrary order, and then is required to sort them into the same order 252 | // as the server, so they can use the same index numbers to refer to identify 253 | // files. (It's a bit strange.) 254 | // 255 | // The ordering varies per protocol version but in protocol 27 it's essentially 256 | // strcmp. (The rsync code is a bit complicated by storing the names split 257 | // into directory and filename.) 258 | file_list.sort_unstable_by(|a, b| a.name.cmp(&b.name)); 259 | debug!("File list sort done"); 260 | let len_before = file_list.len(); 261 | file_list.dedup_by(|a, b| a.name == b.name); 262 | let removed = len_before - file_list.len(); 263 | if removed > 0 { 264 | debug!("{} duplicate file list entries removed", removed) 265 | } 266 | for (i, entry) in file_list.iter().enumerate() { 267 | debug!("[{:8}] {:?}", i, entry.name_lossy_string()) 268 | } 269 | } 270 | 271 | #[cfg(test)] 272 | mod test { 273 | use super::*; 274 | use regex::Regex; 275 | 276 | #[test] 277 | fn file_entry_display_like_ls() { 278 | let entry = FileEntry { 279 | mode: 0o0040750, 280 | file_len: 420, 281 | mtime: 1588429517, 282 | name: b"rsyn".to_vec(), 283 | link_target: None, 284 | }; 285 | // The mtime is in the local timezone, and we need the tests to pass 286 | // regardless of timezone. Rust Chrono doesn't seem to provide a way 287 | // to override it for testing. Let's just assert that the pattern is 288 | // plausible. 289 | // 290 | // This does assume there are no timezones with a less-than-whole minute 291 | // offset. (There are places like South Australia with a fractional-hour offset. 292 | let entry_display = format!("{}", entry); 293 | assert!( 294 | Regex::new(r"drwxr-x--- 420 2020-05-0[123] \d\d:\d\d:17 rsyn") 295 | .unwrap() 296 | .is_match(&entry_display), 297 | "{:?} doesn't match expected format", 298 | entry_display 299 | ); 300 | } 301 | 302 | // TODO: Test reading and decoding from an varint stream. 303 | 304 | /// Examples from verbose output of rsync 2.6.1. 305 | #[test] 306 | fn ordering_examples() { 307 | const EXAMPLE: &[&[u8]] = &[ 308 | b"./", 309 | b".git/", 310 | b".git/HEAD", 311 | b".github/", 312 | b".github/workflows/", 313 | b".github/workflows/rust.yml", 314 | b".gitignore", 315 | b"CONTRIBUTING.md", 316 | b"src/", 317 | b"src/lib.rs", 318 | ]; 319 | let clean: Vec = EXAMPLE 320 | .iter() 321 | .map(|name| FileEntry { 322 | mode: 0o0040750, 323 | file_len: 420, 324 | mtime: 1588429517, 325 | name: name.to_vec(), 326 | link_target: None, 327 | }) 328 | .collect(); 329 | let mut messy = clean.clone(); 330 | messy.reverse(); 331 | messy.extend_from_slice(clean.as_slice()); 332 | sort_and_dedupe(&mut messy); 333 | assert_eq!(&messy, &clean); 334 | } 335 | 336 | #[test] 337 | fn validate_name() { 338 | use super::validate_name; 339 | assert!(validate_name(b".").is_ok()); 340 | assert!(validate_name(b"./ok").is_ok()); 341 | assert!(validate_name(b"easy").is_ok()); 342 | assert!(validate_name(b"../../naughty").is_err()); 343 | assert!(validate_name(b"still/not/../ok").is_err()); 344 | } 345 | } 346 | -------------------------------------------------------------------------------- /src/lib.rs: -------------------------------------------------------------------------------- 1 | // Copyright 2020 Google LLC 2 | // 3 | // Licensed under the Apache License, Version 2.0 (the "License"); 4 | // you may not use this file except in compliance with the License. 5 | // You may obtain a copy of the License at 6 | // 7 | // http://www.apache.org/licenses/LICENSE-2.0 8 | // 9 | // Unless required by applicable law or agreed to in writing, software 10 | // distributed under the License is distributed on an "AS IS" BASIS, 11 | // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | // See the License for the specific language governing permissions and 13 | // limitations under the License. 14 | 15 | #![deny(unsafe_code)] 16 | #![warn(missing_docs)] 17 | #![warn(future_incompatible)] 18 | #![warn(rust_2018_idioms)] 19 | // rustdoc::private_doc_tests is a nice idea but unfortunately warns on types republished 20 | // by `pub use`. 21 | // https://github.com/rust-lang/rust/issues/72081 22 | #![allow(rustdoc::private_doc_tests)] 23 | // MAYBE: warn(missing-doc-code-examples) but covering everything isn't a 24 | // priority yet. 25 | // Match on Ord isn't any easier to read. 26 | #![allow(clippy::comparison_chain)] 27 | 28 | //! A wire-compatible rsync client in Rust. 29 | //! 30 | //! Messages are sent to [`log`](https://docs.rs/log/) and a log destination 31 | //! may optionally be configured by clients. 32 | //! 33 | //! Use the [`Client`](struct.Client.html) type to list or transfer files: 34 | //! 35 | //! ``` 36 | //! // Open a connection to a local rsync server, and list the source directory. 37 | //! use rsyn::{Client, Options}; 38 | //! 39 | //! let mut client = Client::local("./src"); 40 | //! client.set_recursive(true); 41 | //! let (flist, _summary) = client.list_files()?; 42 | //! 43 | //! // We can see the `lib.rs` in the listing. 44 | //! assert!(flist.iter().any(|fe| 45 | //! fe.name_lossy_string().ends_with("lib.rs"))); 46 | //! # rsyn::Result::Ok(()) 47 | //! ``` 48 | 49 | mod client; 50 | mod connection; 51 | mod flist; 52 | mod localtree; 53 | mod mux; 54 | mod options; 55 | mod statistics; 56 | mod sums; 57 | mod varint; 58 | 59 | pub use client::Client; 60 | pub use flist::{FileEntry, FileList}; 61 | pub use localtree::LocalTree; 62 | pub use options::Options; 63 | pub use statistics::{ServerStatistics, Summary}; 64 | 65 | /// General Result type from rsyn APIs. 66 | pub type Result = anyhow::Result; 67 | 68 | const MD4_SUM_LENGTH: usize = 16; 69 | -------------------------------------------------------------------------------- /src/localtree.rs: -------------------------------------------------------------------------------- 1 | // Copyright 2020 Google LLC 2 | // 3 | // Licensed under the Apache License, Version 2.0 (the "License"); 4 | // you may not use this file except in compliance with the License. 5 | // You may obtain a copy of the License at 6 | // 7 | // http://www.apache.org/licenses/LICENSE-2.0 8 | // 9 | // Unless required by applicable law or agreed to in writing, software 10 | // distributed under the License is distributed on an "AS IS" BASIS, 11 | // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | // See the License for the specific language governing permissions and 13 | // limitations under the License. 14 | 15 | //! Facade for local-filesystem operations. 16 | 17 | use std::path::{Path, PathBuf}; 18 | 19 | use anyhow::Context; 20 | use tempfile::NamedTempFile; 21 | 22 | use crate::Result; 23 | 24 | /// A filesystem tree local to this process. 25 | /// 26 | /// The local tree is the destination for downloads and the source for uploads. 27 | /// 28 | /// All local IO is funneled through this layer so that it can be observed 29 | /// and so filenames can be checked. (And perhaps later, applications can provide 30 | /// new implementations that don't literally use the local filesystem.) 31 | pub struct LocalTree { 32 | root: PathBuf, 33 | } 34 | 35 | /// A file being written into the local tree. 36 | /// 37 | /// It becomes visible under its name only when it's persisted. 38 | #[derive(Debug)] 39 | pub struct WriteFile { 40 | final_path: PathBuf, 41 | temp: NamedTempFile, 42 | } 43 | 44 | impl LocalTree { 45 | /// Construct a new LocalTree addressing a local directory. 46 | pub fn new>(root: P) -> LocalTree { 47 | LocalTree { root: root.into() } 48 | } 49 | 50 | /// Open a file for write. 51 | /// 52 | /// The result, a `WriteFile` can be used as `std::io::Write`, but must then be finalized 53 | /// before the results are committed to the final file name. 54 | /// 55 | /// `path` is the relative path. 56 | pub fn write_file>(&self, path: &P) -> Result { 57 | let final_path = self.root.join(path.as_ref()); 58 | // Store the temporary file in its subdirectory, not in the root. 59 | let temp = NamedTempFile::new_in(final_path.parent().unwrap())?; 60 | Ok(WriteFile { final_path, temp }) 61 | } 62 | } 63 | 64 | impl WriteFile { 65 | /// Finish writing to this file and store it to its permanent location. 66 | pub fn finalize(self) -> Result<()> { 67 | let WriteFile { temp, final_path } = self; 68 | temp.persist(&final_path) 69 | .with_context(|| format!("Failed to persist temporary file to {:?}", final_path))?; 70 | Ok(()) 71 | } 72 | 73 | /// The full path to which this file will eventually be written. 74 | pub fn final_path(&self) -> &Path { 75 | &self.final_path 76 | } 77 | } 78 | 79 | impl std::io::Write for WriteFile { 80 | fn write(&mut self, buf: &[u8]) -> std::io::Result { 81 | self.temp.write(buf) 82 | } 83 | 84 | fn flush(&mut self) -> std::io::Result<()> { 85 | self.temp.flush() 86 | } 87 | 88 | fn write_vectored(&mut self, bufs: &[std::io::IoSlice<'_>]) -> std::io::Result { 89 | self.temp.write_vectored(bufs) 90 | } 91 | 92 | fn write_all(&mut self, buf: &[u8]) -> std::io::Result<()> { 93 | self.temp.write_all(buf) 94 | } 95 | } 96 | 97 | #[cfg(test)] 98 | mod test { 99 | use super::*; 100 | use std::fs; 101 | use std::fs::File; 102 | use std::io::prelude::*; 103 | 104 | #[test] 105 | fn write_a_file() { 106 | let tempdir = tempfile::Builder::new() 107 | .prefix("rsyn_localtree_write_a_file") 108 | .tempdir() 109 | .unwrap(); 110 | let lt = LocalTree::new(tempdir.path()); 111 | let mut f = lt.write_file(&"hello").unwrap(); 112 | let final_path = tempdir.path().join("hello"); 113 | 114 | // File does not yet exist until it's finalized. 115 | assert!(fs::metadata(&final_path).is_err()); 116 | 117 | writeln!(f, "The answer is: {}", 42).unwrap(); 118 | f.finalize().unwrap(); 119 | assert!(fs::metadata(&final_path).unwrap().is_file()); 120 | let mut content = String::new(); 121 | File::open(&final_path) 122 | .unwrap() 123 | .read_to_string(&mut content) 124 | .unwrap(); 125 | assert_eq!(content, "The answer is: 42\n"); 126 | } 127 | 128 | #[test] 129 | fn dropped_file_is_discarded() { 130 | let tempdir = tempfile::Builder::new() 131 | .prefix("rsyn_localtree_dropped_file_is_discarded") 132 | .tempdir() 133 | .unwrap(); 134 | let lt = LocalTree::new(tempdir.path()); 135 | let mut f = lt.write_file(&"hello").unwrap(); 136 | let final_path = f.final_path().to_owned(); 137 | f.write_all("some content".as_bytes()).unwrap(); 138 | drop(f); 139 | // File does not yet exist 140 | assert!(fs::metadata(&final_path).is_err()); 141 | // Can also drop the LocalTree but not the tempdir, and the file still 142 | // does not exist. 143 | drop(lt); 144 | assert!(fs::metadata(tempdir.path()).unwrap().is_dir()); 145 | assert!(fs::metadata(&final_path).is_err()); 146 | } 147 | } 148 | -------------------------------------------------------------------------------- /src/mux.rs: -------------------------------------------------------------------------------- 1 | // Copyright 2020 Google LLC 2 | // 3 | // Licensed under the Apache License, Version 2.0 (the "License"); 4 | // you may not use this file except in compliance with the License. 5 | // You may obtain a copy of the License at 6 | // 7 | // http://www.apache.org/licenses/LICENSE-2.0 8 | // 9 | // Unless required by applicable law or agreed to in writing, software 10 | // distributed under the License is distributed on an "AS IS" BASIS, 11 | // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | // See the License for the specific language governing permissions and 13 | // limitations under the License. 14 | 15 | //! Length-prefixed, typed, packets multiplexed onto a byte stream. 16 | //! 17 | //! The main function of these is to allow remote error/message strings 18 | //! to be mixed in with normal data transfer. 19 | //! 20 | //! This format is used only from the remote server to the client. 21 | 22 | use std::io; 23 | use std::io::prelude::*; 24 | 25 | #[allow(unused_imports)] 26 | use log::{debug, error, info, trace, warn}; 27 | 28 | // TODO: Handle other message types from rsync `read_a_msg`. 29 | const TAG_DATA: u8 = 7; 30 | const TAG_FATAL: u8 = 1; 31 | 32 | pub struct DemuxRead { 33 | /// Underlying stream. 34 | r: Box, 35 | /// Amount of data from previous packet remaining to read out. 36 | current_packet_len: usize, 37 | } 38 | 39 | impl Read for DemuxRead { 40 | fn read(&mut self, buf: &mut [u8]) -> io::Result { 41 | if self.current_packet_len == 0 { 42 | self.current_packet_len = self.read_header_consume_messages()?; 43 | } 44 | let max_len = std::cmp::min(buf.len(), self.current_packet_len); 45 | let read_len = self.r.read(&mut buf[..max_len])?; 46 | self.current_packet_len -= read_len; 47 | Ok(read_len) 48 | } 49 | } 50 | 51 | impl DemuxRead { 52 | /// Construct a new packet demuxer, wrapping an underlying Read (typically 53 | /// a pipe). 54 | pub fn new(r: Box) -> DemuxRead { 55 | DemuxRead { 56 | r, 57 | current_packet_len: 0, 58 | } 59 | } 60 | 61 | /// Return the length of the next real data block. 62 | /// 63 | /// Read and print out any messages from the remote end, without returning 64 | /// them. 65 | /// 66 | /// Returns Ok(0) for a clean EOF before the start of the packet. 67 | fn read_header_consume_messages(&mut self) -> io::Result { 68 | loop { 69 | // Read a length-prefixed packet from peer. 70 | let mut h = [0u8; 4]; 71 | if let Err(e) = self.r.read_exact(&mut h) { 72 | match e.kind() { 73 | io::ErrorKind::UnexpectedEof => { 74 | debug!("Clean eof before mux packet"); 75 | return Ok(0); 76 | } 77 | _ => return Err(e), 78 | } 79 | } 80 | 81 | // debug!("got envelope header {{{}}}", hex::encode(&h)); 82 | let h = u32::from_le_bytes(h); 83 | let tag = (h >> 24) as u8; 84 | let len = (h & 0xff_ffff) as usize; 85 | trace!("Read envelope tag {:#04x} length {:#x}", tag, len); 86 | if tag == TAG_DATA { 87 | if len == 0 { 88 | return Err(io::Error::new( 89 | io::ErrorKind::InvalidData, 90 | "Zero-length data packet received", 91 | )); 92 | } 93 | return Ok(len); 94 | } 95 | 96 | // A human-readable message: read and display it here. 97 | let mut message = vec![0; len]; 98 | self.r.read_exact(&mut message)?; 99 | info!("REMOTE: {}", String::from_utf8_lossy(&message).trim_end()); 100 | if tag == TAG_FATAL { 101 | return Err(io::Error::new( 102 | io::ErrorKind::ConnectionAborted, 103 | "Remote signalled fatal error", 104 | )); 105 | } 106 | } 107 | } 108 | } 109 | 110 | // MAYBE: Add buffering and flushing, so that every single write is 111 | // not sent as a single packet. 112 | 113 | /// Translate a stream of bytes into length-prefixed packets. 114 | /// 115 | /// This is only used from the server to the client, and 116 | /// at the moment rsyn only acts as a client, so this is never used. 117 | #[allow(unused)] 118 | pub struct MuxWrite { 119 | w: Box, 120 | } 121 | 122 | impl MuxWrite { 123 | #[allow(unused)] 124 | pub fn new(w: Box) -> MuxWrite { 125 | MuxWrite { w } 126 | } 127 | } 128 | 129 | impl Write for MuxWrite { 130 | fn write(&mut self, buf: &[u8]) -> io::Result { 131 | // TODO: Break large buffers into multiple packets instead of erroring. 132 | let l = buf.len(); 133 | assert!( 134 | l < 0x0ff_ffff, 135 | "Data length {:#x} is too much for one packet", 136 | l 137 | ); 138 | let l: u32 = l as u32 | ((TAG_DATA as u32) << 24); 139 | let h = l.to_le_bytes(); 140 | self.w 141 | .write_all(&h) 142 | .expect("failed to write envelope header"); 143 | self.w 144 | .write_all(buf) 145 | .expect("failed to write envelope body"); 146 | trace!("Send envelope tag {:#x} data {}", l, hex::encode(buf)); 147 | Ok(buf.len()) 148 | } 149 | 150 | fn flush(&mut self) -> io::Result<()> { 151 | self.w.flush() 152 | } 153 | } 154 | -------------------------------------------------------------------------------- /src/options.rs: -------------------------------------------------------------------------------- 1 | // Copyright 2020 Google LLC 2 | // 3 | // Licensed under the Apache License, Version 2.0 (the "License"); 4 | // you may not use this file except in compliance with the License. 5 | // You may obtain a copy of the License at 6 | // 7 | // http://www.apache.org/licenses/LICENSE-2.0 8 | // 9 | // Unless required by applicable law or agreed to in writing, software 10 | // distributed under the License is distributed on an "AS IS" BASIS, 11 | // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | // See the License for the specific language governing permissions and 13 | // limitations under the License. 14 | 15 | //! Command-line options controlling the local and remote processes. 16 | 17 | #[allow(unused_imports)] 18 | use log::{debug, error, info, trace, warn}; 19 | 20 | /// Command-line options controlling the local and remote processes. 21 | /// 22 | /// These are held inside a [`Client`](struct.Client.html), 23 | /// and are passed to the remote side when the connection is opened. 24 | /// 25 | /// ``` 26 | /// use rsyn::{Client, Options}; 27 | /// let mut client = Client::from_str("rsync.example.com::mod").unwrap(); 28 | /// client.set_options(Options { 29 | /// verbose: 2, 30 | /// recursive: true, 31 | /// .. Options::default() 32 | /// }); 33 | /// ``` 34 | #[derive(Clone, Eq, PartialEq, Debug, Default)] 35 | pub struct Options { 36 | /// Recurse into directories. 37 | pub recursive: bool, 38 | 39 | /// Command to run to start the rsync server, typically remotely. 40 | /// 41 | /// May be multiple words, which will be passed as separate shell arguments. 42 | /// 43 | /// If unset, just "rsync". 44 | pub rsync_command: Option>, 45 | 46 | /// Command to open a connection to the remote server. 47 | /// 48 | /// May be multiple words to include options, which will be passed as separate 49 | /// shell arguments. 50 | /// 51 | /// If unset, just "ssh". 52 | pub ssh_command: Option>, 53 | 54 | /// Only list files, don't transfer contents. 55 | /// 56 | /// In some cases the server will infer this. 57 | pub list_only: bool, 58 | 59 | /// Be verbose. 60 | /// 61 | /// (This is passed to the server to encourage it to be verbose too.) 62 | pub verbose: u32, 63 | } 64 | -------------------------------------------------------------------------------- /src/statistics.rs: -------------------------------------------------------------------------------- 1 | // Copyright 2020 Google LLC 2 | // 3 | // Licensed under the Apache License, Version 2.0 (the "License"); 4 | // you may not use this file except in compliance with the License. 5 | // You may obtain a copy of the License at 6 | // 7 | // http://www.apache.org/licenses/LICENSE-2.0 8 | // 9 | // Unless required by applicable law or agreed to in writing, software 10 | // distributed under the License is distributed on an "AS IS" BASIS, 11 | // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | // See the License for the specific language governing permissions and 13 | // limitations under the License. 14 | 15 | //! Statistics/counter structs. 16 | 17 | /// Description of what happened during a transfer. 18 | #[derive(Clone, Eq, PartialEq, Debug, Default)] 19 | pub struct Summary { 20 | /// Server reported this many errors while building the file count. 21 | /// (Typically, "permission denied" on a subdirectory.) 22 | pub server_flist_io_error_count: i32, 23 | 24 | /// Statistics sent from the server. 25 | pub server_stats: crate::ServerStatistics, 26 | 27 | /// If a child process was used for the connection and it has exited, 28 | /// it's exit status. 29 | pub child_exit_status: Option, 30 | 31 | /// Number of invalid file indexes received. Should be 0. 32 | pub invalid_file_index_count: usize, 33 | 34 | /// Number of times the whole-file MD4 did not match. 35 | pub whole_file_sum_mismatch_count: usize, 36 | 37 | /// Number of literal bytes (rather than references to the old file) received. 38 | pub literal_bytes_received: usize, 39 | 40 | /// Number of files received. 41 | pub files_received: usize, 42 | } 43 | 44 | /// Statistics from a remote server about how much work it did. 45 | #[derive(Clone, Eq, PartialEq, Debug, Default)] 46 | pub struct ServerStatistics { 47 | // The rsync(1) man page has some description of these. 48 | /// Total bytes sent over the network from the client to the server. 49 | pub total_bytes_read: i64, 50 | /// Total bytes sent over the network from the server to the client, 51 | /// ignoring any text messages. 52 | pub total_bytes_written: i64, 53 | /// The sum of the size of all file sizes in the transfer. This does not 54 | /// count directories or special files, but does include the size of 55 | /// symlinks. 56 | pub total_file_size: i64, 57 | /// The number of seconds spent by the server building a file list. 58 | pub flist_build_time: Option, 59 | /// The number of seconds the server spent sending the file list to the 60 | /// client. 61 | pub flist_xfer_time: Option, 62 | // TODO: More fields in at least some protocol versions. 63 | } 64 | -------------------------------------------------------------------------------- /src/sums.rs: -------------------------------------------------------------------------------- 1 | // Copyright 2020 Google LLC 2 | // 3 | // Licensed under the Apache License, Version 2.0 (the "License"); 4 | // you may not use this file except in compliance with the License. 5 | // You may obtain a copy of the License at 6 | // 7 | // http://www.apache.org/licenses/LICENSE-2.0 8 | // 9 | // Unless required by applicable law or agreed to in writing, software 10 | // distributed under the License is distributed on an "AS IS" BASIS, 11 | // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | // See the License for the specific language governing permissions and 13 | // limitations under the License. 14 | 15 | //! A collection of strong and weak sums for a single file, from which deltas 16 | //! can be generated. 17 | 18 | use crate::varint::{ReadVarint, WriteVarint}; 19 | use crate::Result; 20 | 21 | #[derive(Debug)] 22 | pub(crate) struct SumHead { 23 | // like rsync |sum_struct|. 24 | count: i32, 25 | blength: i32, 26 | s2length: i32, 27 | remainder: i32, 28 | } 29 | 30 | impl SumHead { 31 | /// Create an empty SumHead describing an empty or absent file. 32 | pub(crate) fn zero() -> Self { 33 | SumHead { 34 | count: 0, 35 | blength: 0, 36 | s2length: 0, 37 | remainder: 0, 38 | } 39 | } 40 | 41 | pub fn read(rv: &mut ReadVarint) -> Result { 42 | // TODO: Encoding varies per protocol version. 43 | // TODO: Assertions about the values? 44 | Ok(SumHead { 45 | count: rv.read_i32()?, 46 | blength: rv.read_i32()?, 47 | s2length: rv.read_i32()?, 48 | remainder: rv.read_i32()?, 49 | }) 50 | } 51 | 52 | pub fn write(&self, wv: &mut WriteVarint) -> Result<()> { 53 | wv.write_i32(self.count)?; 54 | wv.write_i32(self.blength)?; 55 | wv.write_i32(self.s2length)?; 56 | wv.write_i32(self.remainder)?; 57 | Ok(()) 58 | } 59 | } 60 | -------------------------------------------------------------------------------- /src/varint.rs: -------------------------------------------------------------------------------- 1 | // Copyright 2020 Google LLC 2 | // 3 | // Licensed under the Apache License, Version 2.0 (the "License"); 4 | // you may not use this file except in compliance with the License. 5 | // You may obtain a copy of the License at 6 | // 7 | // http://www.apache.org/licenses/LICENSE-2.0 8 | // 9 | // Unless required by applicable law or agreed to in writing, software 10 | // distributed under the License is distributed on an "AS IS" BASIS, 11 | // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | // See the License for the specific language governing permissions and 13 | // limitations under the License. 14 | 15 | //! Read and write rsync's integer encoding scheme: u8, i32, i64, and byte strings. 16 | 17 | use std::io; 18 | use std::io::prelude::*; 19 | 20 | use anyhow::{Context, Error, Result}; 21 | #[allow(unused_imports)] 22 | use log::{debug, error, info, trace, warn}; 23 | 24 | /// Read rsync data types from a wrapped stream. 25 | pub struct ReadVarint { 26 | r: Box, 27 | } 28 | 29 | impl ReadVarint { 30 | pub fn new(r: Box) -> ReadVarint { 31 | ReadVarint { r } 32 | } 33 | 34 | pub fn read_u8(&mut self) -> io::Result { 35 | let mut b = [0u8]; 36 | self.r.read_exact(&mut b).and(Ok(b[0])) 37 | } 38 | 39 | /// Read a known-length byte string into a newly allocated buffer. 40 | /// 41 | /// Always returns the exact size, or an error. 42 | pub fn read_byte_string(&mut self, len: usize) -> io::Result> { 43 | let mut buf = vec![0; len]; 44 | self.r.read_exact(&mut buf).and(Ok(buf)) 45 | } 46 | 47 | pub fn read_i32(&mut self) -> io::Result { 48 | let mut buf = [0; 4]; 49 | self.r.read_exact(&mut buf)?; 50 | let v = i32::from_le_bytes(buf); 51 | trace!("Read {:#x}i32", v); 52 | Ok(v) 53 | } 54 | 55 | pub fn read_i64(&mut self) -> io::Result { 56 | let v = self.read_i32()?; 57 | let v = if v != -1 { 58 | v as i64 59 | } else { 60 | let mut buf = [0; 8]; 61 | self.r.read_exact(&mut buf)?; 62 | i64::from_le_bytes(buf) 63 | }; 64 | trace!("Read {:#x}i64", v); 65 | Ok(v) 66 | } 67 | 68 | /// Return the underlying stream, consuming this wrapper. 69 | pub fn take(self) -> Box { 70 | self.r 71 | } 72 | 73 | /// Destructively test that this is at the end of the input. 74 | /// 75 | /// Returns an error either on an IO error, or if there's any remaining 76 | /// data. The remaining data will be unreachable (and corrupt.) Returns Ok(()) 77 | /// on a clean end. 78 | #[allow(unused)] 79 | pub fn check_for_eof(mut self) -> Result<()> { 80 | match self.read_u8().and(Ok(())) { 81 | Ok(_) => Err(Error::msg("Data found when we expected end of stream")), 82 | Err(e) if e.kind() == io::ErrorKind::UnexpectedEof => Ok(()), 83 | e => e.context("Error looking for end of stream"), 84 | } 85 | } 86 | } 87 | 88 | /// Write rsync low-level protocol variable integers. 89 | pub struct WriteVarint { 90 | w: Box, 91 | } 92 | 93 | impl WriteVarint { 94 | pub fn new(w: Box) -> WriteVarint { 95 | WriteVarint { w } 96 | } 97 | 98 | pub fn write_i32(&mut self, v: i32) -> io::Result<()> { 99 | trace!("Send {:#x}i32", v); 100 | self.w.write_all(&v.to_le_bytes()) 101 | } 102 | 103 | #[allow(unused)] 104 | pub fn write_u8(&mut self, v: u8) -> io::Result<()> { 105 | trace!("Send {:#x}u8", v); 106 | self.w.write_all(&[v]) 107 | } 108 | } 109 | 110 | #[cfg(test)] 111 | mod test { 112 | use super::*; 113 | 114 | fn make_rv(s: &'static [u8]) -> ReadVarint { 115 | ReadVarint::new(Box::new(s)) 116 | } 117 | 118 | #[test] 119 | fn read_i64() { 120 | let mut rv = make_rv(&[0x10, 0, 0, 0]); 121 | assert_eq!(rv.read_i64().unwrap(), 0x10); 122 | 123 | let mut rv = make_rv(&[ 124 | 0xff, 0xff, 0xff, 0xff, 0x00, 0x11, 0x22, 0x33, 0x44, 0x55, 0x66, 0x77, 125 | ]); 126 | assert_eq!(rv.read_i64().unwrap(), 0x7766554433221100); 127 | rv.check_for_eof().unwrap(); 128 | } 129 | } 130 | -------------------------------------------------------------------------------- /tests/interop.rs: -------------------------------------------------------------------------------- 1 | // Copyright 2020 Google LLC 2 | // 3 | // Licensed under the Apache License, Version 2.0 (the "License"); 4 | // you may not use this file except in compliance with the License. 5 | // You may obtain a copy of the License at 6 | // 7 | // http://www.apache.org/licenses/LICENSE-2.0 8 | // 9 | // Unless required by applicable law or agreed to in writing, software 10 | // distributed under the License is distributed on an "AS IS" BASIS, 11 | // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | // See the License for the specific language governing permissions and 13 | // limitations under the License. 14 | 15 | //! Test this library's compatibility by running original Tridge rsync. 16 | //! 17 | //! This requires 'rsync' be available on the path. 18 | 19 | use std::fmt; 20 | use std::fs::{create_dir, File}; 21 | 22 | use anyhow::Result; 23 | use chrono::prelude::*; 24 | 25 | use rsyn::{Client, Options}; 26 | 27 | /// List files from a newly-created temporary directory. 28 | #[test] 29 | fn list_files() { 30 | install_test_logger(); 31 | 32 | let tmp = tempfile::Builder::new() 33 | .prefix("rsyn_interop_list_files") 34 | .tempdir() 35 | .unwrap(); 36 | File::create(tmp.path().join("a")).unwrap(); 37 | File::create(tmp.path().join("b")).unwrap(); 38 | create_dir(tmp.path().join("subdir")).unwrap(); 39 | File::create(tmp.path().join("subdir").join("galah")).unwrap(); 40 | 41 | let mut client = Client::local(tmp.path()); 42 | client.set_recursive(true); 43 | let (flist, summary) = client.list_files().unwrap(); 44 | 45 | assert_eq!(flist.len(), 5); 46 | let names: Vec = flist 47 | .iter() 48 | .map(|fe| fe.name_lossy_string().into_owned()) 49 | .collect(); 50 | // Names should already be sorted. 51 | assert_eq!(names[0], "."); 52 | assert_eq!(names[1], "a"); 53 | assert_eq!(names[2], "b"); 54 | assert_eq!(names[3], "subdir"); 55 | assert_eq!(names[4], "subdir/galah"); 56 | 57 | // Check file types. 58 | assert!(flist[0].is_dir()); 59 | assert!(!flist[0].is_file()); 60 | assert!( 61 | flist[1].is_file(), 62 | "expected {:?} would be a file", 63 | &flist[1] 64 | ); 65 | assert!(flist[2].is_file()); 66 | assert!(flist[3].is_dir()); 67 | assert!(flist[4].is_file()); 68 | 69 | // Check mtimes. We don't control them precisely, but they should be close 70 | // to the current time. (Probably within a couple of seconds, but allow 71 | // some slack for debugging, thrashing machines, etc.) 72 | let now = Local::now(); 73 | assert!((now - flist[0].mtime()).num_minutes() < 5); 74 | assert!((now - flist[1].mtime()).num_minutes() < 5); 75 | 76 | // All the files are empty. 77 | assert_eq!(summary.server_stats.total_file_size, 0); 78 | } 79 | 80 | /// Only on Unix, check we can list a directory containing a symlink, and see 81 | /// the symlink. 82 | #[cfg(unix)] 83 | #[test] 84 | fn list_symlink() -> rsyn::Result<()> { 85 | install_test_logger(); 86 | 87 | let tmp = tempfile::Builder::new() 88 | .prefix("rsyn_interop_list_symlink") 89 | .tempdir()?; 90 | std::os::unix::fs::symlink("dangling link", tmp.path().join("a link"))?; 91 | 92 | let mut client = Client::local(tmp.path()); 93 | client.mut_options().list_only = true; 94 | let (flist, _summary) = client.list_files()?; 95 | 96 | assert_eq!(flist.len(), 2); 97 | assert_eq!(flist[0].name_lossy_string(), "."); 98 | assert_eq!(flist[1].name_lossy_string(), "a link"); 99 | 100 | assert!(!flist[0].is_symlink()); 101 | assert!(flist[1].is_symlink()); 102 | 103 | Ok(()) 104 | } 105 | 106 | /// Only on Unix: list `/etc`, a good natural source of files with different 107 | /// permissions, including some probably not readable to the non-root 108 | /// user running this test. 109 | #[cfg(unix)] 110 | #[test] 111 | fn list_files_etc() -> Result<()> { 112 | install_test_logger(); 113 | let mut client = Client::local("/etc"); 114 | client.set_options(Options { 115 | recursive: true, 116 | list_only: true, 117 | ..Options::default() 118 | }); 119 | let (flist, _summary) = client.list_files()?; 120 | assert_eq!( 121 | flist 122 | .iter() 123 | .filter(|e| e.name_lossy_string() == "passwd" 124 | && e.is_file() 125 | && (e.mode & 0o777 == 0o644)) 126 | .count(), 127 | 1 128 | ); 129 | Ok(()) 130 | } 131 | 132 | /// Only on Unix: list `/dev`, a good source of devices and unusual files. 133 | #[cfg(unix)] 134 | #[test] 135 | fn list_files_dev() -> Result<()> { 136 | install_test_logger(); 137 | let mut client = Client::local("/dev"); 138 | client.set_options(Options { 139 | recursive: true, 140 | list_only: true, 141 | ..Options::default() 142 | }); 143 | let (flist, _summary) = client.list_files()?; 144 | assert_eq!( 145 | flist 146 | .iter() 147 | .filter(|e| e.name_lossy_string() == "null" 148 | && !e.is_file() 149 | && unix_mode::is_char_device(e.mode) 150 | && (e.mode & 0o777 == 0o666)) 151 | .count(), 152 | 1 153 | ); 154 | Ok(()) 155 | } 156 | 157 | fn install_test_logger() { 158 | // This works, but leaks out of the normally-captured test stdout, because 159 | // the way Rust catches output only affects the main thread. 160 | // 161 | // Therefore, it's just off by default. 162 | // 163 | // https://github.com/rust-lang/rust/issues/42474 164 | if false { 165 | let _ = fern::Dispatch::new() 166 | .format(format_log) 167 | .level(log::LevelFilter::Debug) 168 | .chain(fern::Output::call(|record| println!("{}", record.args()))) 169 | .apply(); 170 | } 171 | } 172 | 173 | /// Format a `log::Record`. 174 | fn format_log(out: fern::FormatCallback<'_>, args: &fmt::Arguments<'_>, record: &log::Record<'_>) { 175 | out.finish(format_args!( 176 | "[{:<30}][{}] {}", 177 | record.target(), 178 | record.level().to_string().chars().next().unwrap(), 179 | args 180 | )) 181 | } 182 | --------------------------------------------------------------------------------