├── .gitignore ├── LICENSE ├── ReadMe.md ├── keeper.sln ├── keeper ├── CLIparser.cpp ├── CLIparser.h ├── ChaChaFilter.h ├── CommonDefinitions.h ├── ConsoleLogger.cpp ├── ConsoleLogger.h ├── DbCompressTest.cpp ├── FileIO.cpp ├── FileIO.h ├── FilesTransformer.cpp ├── FilesTransformer.h ├── GlobalUtils.cpp ├── GlobalUtils.h ├── PathEncoder.cpp ├── PathEncoder.h ├── PathEncoderTest.cpp ├── TaskBackup.cpp ├── TaskBackup.h ├── TaskContext.cpp ├── TaskContext.h ├── TaskDumpDB.cpp ├── TaskDumpDB.h ├── TaskPurge.cpp ├── TaskPurge.h ├── TaskRestore.cpp ├── TaskRestore.h ├── TestMain.cpp ├── TnxGuard.cpp ├── TnxGuard.h ├── WildCardNameChecker.cpp ├── WildCardNameChecker.h ├── WildCardNamesCheckerTest.cpp ├── basen.hpp ├── bocu1.cpp ├── bocu1.h ├── fpaq.cpp ├── fpaq.h ├── keeper.cpp ├── keeper.rc ├── keeper.vcxproj ├── keeper.vcxproj.filters ├── keeper.vcxproj.user ├── resource.h ├── stdafx.cpp ├── stdafx.h └── targetver.h └── logo_buro.png /.gitignore: -------------------------------------------------------------------------------- 1 | # Compiled Object files 2 | *.slo 3 | *.lo 4 | *.o 5 | *.obj 6 | 7 | # Precompiled Headers 8 | *.gch 9 | *.pch 10 | 11 | # Compiled Dynamic libraries 12 | *.so 13 | *.dylib 14 | *.dll 15 | 16 | # Fortran module files 17 | *.mod 18 | *.smod 19 | 20 | # Compiled Static libraries 21 | *.lai 22 | *.la 23 | *.a 24 | *.lib 25 | 26 | # Executables 27 | *.exe 28 | *.out 29 | *.app 30 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | GNU AFFERO GENERAL PUBLIC LICENSE 2 | Version 3, 19 November 2007 3 | 4 | Copyright (C) 2007 Free Software Foundation, Inc. 5 | Everyone is permitted to copy and distribute verbatim copies 6 | of this license document, but changing it is not allowed. 7 | 8 | Preamble 9 | 10 | The GNU Affero General Public License is a free, copyleft license for 11 | software and other kinds of works, specifically designed to ensure 12 | cooperation with the community in the case of network server software. 13 | 14 | The licenses for most software and other practical works are designed 15 | to take away your freedom to share and change the works. By contrast, 16 | our General Public Licenses are intended to guarantee your freedom to 17 | share and change all versions of a program--to make sure it remains free 18 | software for all its users. 19 | 20 | When we speak of free software, we are referring to freedom, not 21 | price. Our General Public Licenses are designed to make sure that you 22 | have the freedom to distribute copies of free software (and charge for 23 | them if you wish), that you receive source code or can get it if you 24 | want it, that you can change the software or use pieces of it in new 25 | free programs, and that you know you can do these things. 26 | 27 | Developers that use our General Public Licenses protect your rights 28 | with two steps: (1) assert copyright on the software, and (2) offer 29 | you this License which gives you legal permission to copy, distribute 30 | and/or modify the software. 31 | 32 | A secondary benefit of defending all users' freedom is that 33 | improvements made in alternate versions of the program, if they 34 | receive widespread use, become available for other developers to 35 | incorporate. Many developers of free software are heartened and 36 | encouraged by the resulting cooperation. However, in the case of 37 | software used on network servers, this result may fail to come about. 38 | The GNU General Public License permits making a modified version and 39 | letting the public access it on a server without ever releasing its 40 | source code to the public. 41 | 42 | The GNU Affero General Public License is designed specifically to 43 | ensure that, in such cases, the modified source code becomes available 44 | to the community. It requires the operator of a network server to 45 | provide the source code of the modified version running there to the 46 | users of that server. Therefore, public use of a modified version, on 47 | a publicly accessible server, gives the public access to the source 48 | code of the modified version. 49 | 50 | An older license, called the Affero General Public License and 51 | published by Affero, was designed to accomplish similar goals. This is 52 | a different license, not a version of the Affero GPL, but Affero has 53 | released a new version of the Affero GPL which permits relicensing under 54 | this license. 55 | 56 | The precise terms and conditions for copying, distribution and 57 | modification follow. 58 | 59 | TERMS AND CONDITIONS 60 | 61 | 0. Definitions. 62 | 63 | "This License" refers to version 3 of the GNU Affero General Public License. 64 | 65 | "Copyright" also means copyright-like laws that apply to other kinds of 66 | works, such as semiconductor masks. 67 | 68 | "The Program" refers to any copyrightable work licensed under this 69 | License. Each licensee is addressed as "you". "Licensees" and 70 | "recipients" may be individuals or organizations. 71 | 72 | To "modify" a work means to copy from or adapt all or part of the work 73 | in a fashion requiring copyright permission, other than the making of an 74 | exact copy. The resulting work is called a "modified version" of the 75 | earlier work or a work "based on" the earlier work. 76 | 77 | A "covered work" means either the unmodified Program or a work based 78 | on the Program. 79 | 80 | To "propagate" a work means to do anything with it that, without 81 | permission, would make you directly or secondarily liable for 82 | infringement under applicable copyright law, except executing it on a 83 | computer or modifying a private copy. Propagation includes copying, 84 | distribution (with or without modification), making available to the 85 | public, and in some countries other activities as well. 86 | 87 | To "convey" a work means any kind of propagation that enables other 88 | parties to make or receive copies. Mere interaction with a user through 89 | a computer network, with no transfer of a copy, is not conveying. 90 | 91 | An interactive user interface displays "Appropriate Legal Notices" 92 | to the extent that it includes a convenient and prominently visible 93 | feature that (1) displays an appropriate copyright notice, and (2) 94 | tells the user that there is no warranty for the work (except to the 95 | extent that warranties are provided), that licensees may convey the 96 | work under this License, and how to view a copy of this License. If 97 | the interface presents a list of user commands or options, such as a 98 | menu, a prominent item in the list meets this criterion. 99 | 100 | 1. Source Code. 101 | 102 | The "source code" for a work means the preferred form of the work 103 | for making modifications to it. "Object code" means any non-source 104 | form of a work. 105 | 106 | A "Standard Interface" means an interface that either is an official 107 | standard defined by a recognized standards body, or, in the case of 108 | interfaces specified for a particular programming language, one that 109 | is widely used among developers working in that language. 110 | 111 | The "System Libraries" of an executable work include anything, other 112 | than the work as a whole, that (a) is included in the normal form of 113 | packaging a Major Component, but which is not part of that Major 114 | Component, and (b) serves only to enable use of the work with that 115 | Major Component, or to implement a Standard Interface for which an 116 | implementation is available to the public in source code form. A 117 | "Major Component", in this context, means a major essential component 118 | (kernel, window system, and so on) of the specific operating system 119 | (if any) on which the executable work runs, or a compiler used to 120 | produce the work, or an object code interpreter used to run it. 121 | 122 | The "Corresponding Source" for a work in object code form means all 123 | the source code needed to generate, install, and (for an executable 124 | work) run the object code and to modify the work, including scripts to 125 | control those activities. However, it does not include the work's 126 | System Libraries, or general-purpose tools or generally available free 127 | programs which are used unmodified in performing those activities but 128 | which are not part of the work. For example, Corresponding Source 129 | includes interface definition files associated with source files for 130 | the work, and the source code for shared libraries and dynamically 131 | linked subprograms that the work is specifically designed to require, 132 | such as by intimate data communication or control flow between those 133 | subprograms and other parts of the work. 134 | 135 | The Corresponding Source need not include anything that users 136 | can regenerate automatically from other parts of the Corresponding 137 | Source. 138 | 139 | The Corresponding Source for a work in source code form is that 140 | same work. 141 | 142 | 2. Basic Permissions. 143 | 144 | All rights granted under this License are granted for the term of 145 | copyright on the Program, and are irrevocable provided the stated 146 | conditions are met. This License explicitly affirms your unlimited 147 | permission to run the unmodified Program. The output from running a 148 | covered work is covered by this License only if the output, given its 149 | content, constitutes a covered work. This License acknowledges your 150 | rights of fair use or other equivalent, as provided by copyright law. 151 | 152 | You may make, run and propagate covered works that you do not 153 | convey, without conditions so long as your license otherwise remains 154 | in force. You may convey covered works to others for the sole purpose 155 | of having them make modifications exclusively for you, or provide you 156 | with facilities for running those works, provided that you comply with 157 | the terms of this License in conveying all material for which you do 158 | not control copyright. Those thus making or running the covered works 159 | for you must do so exclusively on your behalf, under your direction 160 | and control, on terms that prohibit them from making any copies of 161 | your copyrighted material outside their relationship with you. 162 | 163 | Conveying under any other circumstances is permitted solely under 164 | the conditions stated below. Sublicensing is not allowed; section 10 165 | makes it unnecessary. 166 | 167 | 3. Protecting Users' Legal Rights From Anti-Circumvention Law. 168 | 169 | No covered work shall be deemed part of an effective technological 170 | measure under any applicable law fulfilling obligations under article 171 | 11 of the WIPO copyright treaty adopted on 20 December 1996, or 172 | similar laws prohibiting or restricting circumvention of such 173 | measures. 174 | 175 | When you convey a covered work, you waive any legal power to forbid 176 | circumvention of technological measures to the extent such circumvention 177 | is effected by exercising rights under this License with respect to 178 | the covered work, and you disclaim any intention to limit operation or 179 | modification of the work as a means of enforcing, against the work's 180 | users, your or third parties' legal rights to forbid circumvention of 181 | technological measures. 182 | 183 | 4. Conveying Verbatim Copies. 184 | 185 | You may convey verbatim copies of the Program's source code as you 186 | receive it, in any medium, provided that you conspicuously and 187 | appropriately publish on each copy an appropriate copyright notice; 188 | keep intact all notices stating that this License and any 189 | non-permissive terms added in accord with section 7 apply to the code; 190 | keep intact all notices of the absence of any warranty; and give all 191 | recipients a copy of this License along with the Program. 192 | 193 | You may charge any price or no price for each copy that you convey, 194 | and you may offer support or warranty protection for a fee. 195 | 196 | 5. Conveying Modified Source Versions. 197 | 198 | You may convey a work based on the Program, or the modifications to 199 | produce it from the Program, in the form of source code under the 200 | terms of section 4, provided that you also meet all of these conditions: 201 | 202 | a) The work must carry prominent notices stating that you modified 203 | it, and giving a relevant date. 204 | 205 | b) The work must carry prominent notices stating that it is 206 | released under this License and any conditions added under section 207 | 7. This requirement modifies the requirement in section 4 to 208 | "keep intact all notices". 209 | 210 | c) You must license the entire work, as a whole, under this 211 | License to anyone who comes into possession of a copy. This 212 | License will therefore apply, along with any applicable section 7 213 | additional terms, to the whole of the work, and all its parts, 214 | regardless of how they are packaged. This License gives no 215 | permission to license the work in any other way, but it does not 216 | invalidate such permission if you have separately received it. 217 | 218 | d) If the work has interactive user interfaces, each must display 219 | Appropriate Legal Notices; however, if the Program has interactive 220 | interfaces that do not display Appropriate Legal Notices, your 221 | work need not make them do so. 222 | 223 | A compilation of a covered work with other separate and independent 224 | works, which are not by their nature extensions of the covered work, 225 | and which are not combined with it such as to form a larger program, 226 | in or on a volume of a storage or distribution medium, is called an 227 | "aggregate" if the compilation and its resulting copyright are not 228 | used to limit the access or legal rights of the compilation's users 229 | beyond what the individual works permit. Inclusion of a covered work 230 | in an aggregate does not cause this License to apply to the other 231 | parts of the aggregate. 232 | 233 | 6. Conveying Non-Source Forms. 234 | 235 | You may convey a covered work in object code form under the terms 236 | of sections 4 and 5, provided that you also convey the 237 | machine-readable Corresponding Source under the terms of this License, 238 | in one of these ways: 239 | 240 | a) Convey the object code in, or embodied in, a physical product 241 | (including a physical distribution medium), accompanied by the 242 | Corresponding Source fixed on a durable physical medium 243 | customarily used for software interchange. 244 | 245 | b) Convey the object code in, or embodied in, a physical product 246 | (including a physical distribution medium), accompanied by a 247 | written offer, valid for at least three years and valid for as 248 | long as you offer spare parts or customer support for that product 249 | model, to give anyone who possesses the object code either (1) a 250 | copy of the Corresponding Source for all the software in the 251 | product that is covered by this License, on a durable physical 252 | medium customarily used for software interchange, for a price no 253 | more than your reasonable cost of physically performing this 254 | conveying of source, or (2) access to copy the 255 | Corresponding Source from a network server at no charge. 256 | 257 | c) Convey individual copies of the object code with a copy of the 258 | written offer to provide the Corresponding Source. This 259 | alternative is allowed only occasionally and noncommercially, and 260 | only if you received the object code with such an offer, in accord 261 | with subsection 6b. 262 | 263 | d) Convey the object code by offering access from a designated 264 | place (gratis or for a charge), and offer equivalent access to the 265 | Corresponding Source in the same way through the same place at no 266 | further charge. You need not require recipients to copy the 267 | Corresponding Source along with the object code. If the place to 268 | copy the object code is a network server, the Corresponding Source 269 | may be on a different server (operated by you or a third party) 270 | that supports equivalent copying facilities, provided you maintain 271 | clear directions next to the object code saying where to find the 272 | Corresponding Source. Regardless of what server hosts the 273 | Corresponding Source, you remain obligated to ensure that it is 274 | available for as long as needed to satisfy these requirements. 275 | 276 | e) Convey the object code using peer-to-peer transmission, provided 277 | you inform other peers where the object code and Corresponding 278 | Source of the work are being offered to the general public at no 279 | charge under subsection 6d. 280 | 281 | A separable portion of the object code, whose source code is excluded 282 | from the Corresponding Source as a System Library, need not be 283 | included in conveying the object code work. 284 | 285 | A "User Product" is either (1) a "consumer product", which means any 286 | tangible personal property which is normally used for personal, family, 287 | or household purposes, or (2) anything designed or sold for incorporation 288 | into a dwelling. In determining whether a product is a consumer product, 289 | doubtful cases shall be resolved in favor of coverage. For a particular 290 | product received by a particular user, "normally used" refers to a 291 | typical or common use of that class of product, regardless of the status 292 | of the particular user or of the way in which the particular user 293 | actually uses, or expects or is expected to use, the product. A product 294 | is a consumer product regardless of whether the product has substantial 295 | commercial, industrial or non-consumer uses, unless such uses represent 296 | the only significant mode of use of the product. 297 | 298 | "Installation Information" for a User Product means any methods, 299 | procedures, authorization keys, or other information required to install 300 | and execute modified versions of a covered work in that User Product from 301 | a modified version of its Corresponding Source. The information must 302 | suffice to ensure that the continued functioning of the modified object 303 | code is in no case prevented or interfered with solely because 304 | modification has been made. 305 | 306 | If you convey an object code work under this section in, or with, or 307 | specifically for use in, a User Product, and the conveying occurs as 308 | part of a transaction in which the right of possession and use of the 309 | User Product is transferred to the recipient in perpetuity or for a 310 | fixed term (regardless of how the transaction is characterized), the 311 | Corresponding Source conveyed under this section must be accompanied 312 | by the Installation Information. But this requirement does not apply 313 | if neither you nor any third party retains the ability to install 314 | modified object code on the User Product (for example, the work has 315 | been installed in ROM). 316 | 317 | The requirement to provide Installation Information does not include a 318 | requirement to continue to provide support service, warranty, or updates 319 | for a work that has been modified or installed by the recipient, or for 320 | the User Product in which it has been modified or installed. Access to a 321 | network may be denied when the modification itself materially and 322 | adversely affects the operation of the network or violates the rules and 323 | protocols for communication across the network. 324 | 325 | Corresponding Source conveyed, and Installation Information provided, 326 | in accord with this section must be in a format that is publicly 327 | documented (and with an implementation available to the public in 328 | source code form), and must require no special password or key for 329 | unpacking, reading or copying. 330 | 331 | 7. Additional Terms. 332 | 333 | "Additional permissions" are terms that supplement the terms of this 334 | License by making exceptions from one or more of its conditions. 335 | Additional permissions that are applicable to the entire Program shall 336 | be treated as though they were included in this License, to the extent 337 | that they are valid under applicable law. If additional permissions 338 | apply only to part of the Program, that part may be used separately 339 | under those permissions, but the entire Program remains governed by 340 | this License without regard to the additional permissions. 341 | 342 | When you convey a copy of a covered work, you may at your option 343 | remove any additional permissions from that copy, or from any part of 344 | it. (Additional permissions may be written to require their own 345 | removal in certain cases when you modify the work.) You may place 346 | additional permissions on material, added by you to a covered work, 347 | for which you have or can give appropriate copyright permission. 348 | 349 | Notwithstanding any other provision of this License, for material you 350 | add to a covered work, you may (if authorized by the copyright holders of 351 | that material) supplement the terms of this License with terms: 352 | 353 | a) Disclaiming warranty or limiting liability differently from the 354 | terms of sections 15 and 16 of this License; or 355 | 356 | b) Requiring preservation of specified reasonable legal notices or 357 | author attributions in that material or in the Appropriate Legal 358 | Notices displayed by works containing it; or 359 | 360 | c) Prohibiting misrepresentation of the origin of that material, or 361 | requiring that modified versions of such material be marked in 362 | reasonable ways as different from the original version; or 363 | 364 | d) Limiting the use for publicity purposes of names of licensors or 365 | authors of the material; or 366 | 367 | e) Declining to grant rights under trademark law for use of some 368 | trade names, trademarks, or service marks; or 369 | 370 | f) Requiring indemnification of licensors and authors of that 371 | material by anyone who conveys the material (or modified versions of 372 | it) with contractual assumptions of liability to the recipient, for 373 | any liability that these contractual assumptions directly impose on 374 | those licensors and authors. 375 | 376 | All other non-permissive additional terms are considered "further 377 | restrictions" within the meaning of section 10. If the Program as you 378 | received it, or any part of it, contains a notice stating that it is 379 | governed by this License along with a term that is a further 380 | restriction, you may remove that term. If a license document contains 381 | a further restriction but permits relicensing or conveying under this 382 | License, you may add to a covered work material governed by the terms 383 | of that license document, provided that the further restriction does 384 | not survive such relicensing or conveying. 385 | 386 | If you add terms to a covered work in accord with this section, you 387 | must place, in the relevant source files, a statement of the 388 | additional terms that apply to those files, or a notice indicating 389 | where to find the applicable terms. 390 | 391 | Additional terms, permissive or non-permissive, may be stated in the 392 | form of a separately written license, or stated as exceptions; 393 | the above requirements apply either way. 394 | 395 | 8. Termination. 396 | 397 | You may not propagate or modify a covered work except as expressly 398 | provided under this License. Any attempt otherwise to propagate or 399 | modify it is void, and will automatically terminate your rights under 400 | this License (including any patent licenses granted under the third 401 | paragraph of section 11). 402 | 403 | However, if you cease all violation of this License, then your 404 | license from a particular copyright holder is reinstated (a) 405 | provisionally, unless and until the copyright holder explicitly and 406 | finally terminates your license, and (b) permanently, if the copyright 407 | holder fails to notify you of the violation by some reasonable means 408 | prior to 60 days after the cessation. 409 | 410 | Moreover, your license from a particular copyright holder is 411 | reinstated permanently if the copyright holder notifies you of the 412 | violation by some reasonable means, this is the first time you have 413 | received notice of violation of this License (for any work) from that 414 | copyright holder, and you cure the violation prior to 30 days after 415 | your receipt of the notice. 416 | 417 | Termination of your rights under this section does not terminate the 418 | licenses of parties who have received copies or rights from you under 419 | this License. If your rights have been terminated and not permanently 420 | reinstated, you do not qualify to receive new licenses for the same 421 | material under section 10. 422 | 423 | 9. Acceptance Not Required for Having Copies. 424 | 425 | You are not required to accept this License in order to receive or 426 | run a copy of the Program. Ancillary propagation of a covered work 427 | occurring solely as a consequence of using peer-to-peer transmission 428 | to receive a copy likewise does not require acceptance. However, 429 | nothing other than this License grants you permission to propagate or 430 | modify any covered work. These actions infringe copyright if you do 431 | not accept this License. Therefore, by modifying or propagating a 432 | covered work, you indicate your acceptance of this License to do so. 433 | 434 | 10. Automatic Licensing of Downstream Recipients. 435 | 436 | Each time you convey a covered work, the recipient automatically 437 | receives a license from the original licensors, to run, modify and 438 | propagate that work, subject to this License. You are not responsible 439 | for enforcing compliance by third parties with this License. 440 | 441 | An "entity transaction" is a transaction transferring control of an 442 | organization, or substantially all assets of one, or subdividing an 443 | organization, or merging organizations. If propagation of a covered 444 | work results from an entity transaction, each party to that 445 | transaction who receives a copy of the work also receives whatever 446 | licenses to the work the party's predecessor in interest had or could 447 | give under the previous paragraph, plus a right to possession of the 448 | Corresponding Source of the work from the predecessor in interest, if 449 | the predecessor has it or can get it with reasonable efforts. 450 | 451 | You may not impose any further restrictions on the exercise of the 452 | rights granted or affirmed under this License. For example, you may 453 | not impose a license fee, royalty, or other charge for exercise of 454 | rights granted under this License, and you may not initiate litigation 455 | (including a cross-claim or counterclaim in a lawsuit) alleging that 456 | any patent claim is infringed by making, using, selling, offering for 457 | sale, or importing the Program or any portion of it. 458 | 459 | 11. Patents. 460 | 461 | A "contributor" is a copyright holder who authorizes use under this 462 | License of the Program or a work on which the Program is based. The 463 | work thus licensed is called the contributor's "contributor version". 464 | 465 | A contributor's "essential patent claims" are all patent claims 466 | owned or controlled by the contributor, whether already acquired or 467 | hereafter acquired, that would be infringed by some manner, permitted 468 | by this License, of making, using, or selling its contributor version, 469 | but do not include claims that would be infringed only as a 470 | consequence of further modification of the contributor version. For 471 | purposes of this definition, "control" includes the right to grant 472 | patent sublicenses in a manner consistent with the requirements of 473 | this License. 474 | 475 | Each contributor grants you a non-exclusive, worldwide, royalty-free 476 | patent license under the contributor's essential patent claims, to 477 | make, use, sell, offer for sale, import and otherwise run, modify and 478 | propagate the contents of its contributor version. 479 | 480 | In the following three paragraphs, a "patent license" is any express 481 | agreement or commitment, however denominated, not to enforce a patent 482 | (such as an express permission to practice a patent or covenant not to 483 | sue for patent infringement). To "grant" such a patent license to a 484 | party means to make such an agreement or commitment not to enforce a 485 | patent against the party. 486 | 487 | If you convey a covered work, knowingly relying on a patent license, 488 | and the Corresponding Source of the work is not available for anyone 489 | to copy, free of charge and under the terms of this License, through a 490 | publicly available network server or other readily accessible means, 491 | then you must either (1) cause the Corresponding Source to be so 492 | available, or (2) arrange to deprive yourself of the benefit of the 493 | patent license for this particular work, or (3) arrange, in a manner 494 | consistent with the requirements of this License, to extend the patent 495 | license to downstream recipients. "Knowingly relying" means you have 496 | actual knowledge that, but for the patent license, your conveying the 497 | covered work in a country, or your recipient's use of the covered work 498 | in a country, would infringe one or more identifiable patents in that 499 | country that you have reason to believe are valid. 500 | 501 | If, pursuant to or in connection with a single transaction or 502 | arrangement, you convey, or propagate by procuring conveyance of, a 503 | covered work, and grant a patent license to some of the parties 504 | receiving the covered work authorizing them to use, propagate, modify 505 | or convey a specific copy of the covered work, then the patent license 506 | you grant is automatically extended to all recipients of the covered 507 | work and works based on it. 508 | 509 | A patent license is "discriminatory" if it does not include within 510 | the scope of its coverage, prohibits the exercise of, or is 511 | conditioned on the non-exercise of one or more of the rights that are 512 | specifically granted under this License. You may not convey a covered 513 | work if you are a party to an arrangement with a third party that is 514 | in the business of distributing software, under which you make payment 515 | to the third party based on the extent of your activity of conveying 516 | the work, and under which the third party grants, to any of the 517 | parties who would receive the covered work from you, a discriminatory 518 | patent license (a) in connection with copies of the covered work 519 | conveyed by you (or copies made from those copies), or (b) primarily 520 | for and in connection with specific products or compilations that 521 | contain the covered work, unless you entered into that arrangement, 522 | or that patent license was granted, prior to 28 March 2007. 523 | 524 | Nothing in this License shall be construed as excluding or limiting 525 | any implied license or other defenses to infringement that may 526 | otherwise be available to you under applicable patent law. 527 | 528 | 12. No Surrender of Others' Freedom. 529 | 530 | If conditions are imposed on you (whether by court order, agreement or 531 | otherwise) that contradict the conditions of this License, they do not 532 | excuse you from the conditions of this License. If you cannot convey a 533 | covered work so as to satisfy simultaneously your obligations under this 534 | License and any other pertinent obligations, then as a consequence you may 535 | not convey it at all. For example, if you agree to terms that obligate you 536 | to collect a royalty for further conveying from those to whom you convey 537 | the Program, the only way you could satisfy both those terms and this 538 | License would be to refrain entirely from conveying the Program. 539 | 540 | 13. Remote Network Interaction; Use with the GNU General Public License. 541 | 542 | Notwithstanding any other provision of this License, if you modify the 543 | Program, your modified version must prominently offer all users 544 | interacting with it remotely through a computer network (if your version 545 | supports such interaction) an opportunity to receive the Corresponding 546 | Source of your version by providing access to the Corresponding Source 547 | from a network server at no charge, through some standard or customary 548 | means of facilitating copying of software. This Corresponding Source 549 | shall include the Corresponding Source for any work covered by version 3 550 | of the GNU General Public License that is incorporated pursuant to the 551 | following paragraph. 552 | 553 | Notwithstanding any other provision of this License, you have 554 | permission to link or combine any covered work with a work licensed 555 | under version 3 of the GNU General Public License into a single 556 | combined work, and to convey the resulting work. The terms of this 557 | License will continue to apply to the part which is the covered work, 558 | but the work with which it is combined will remain governed by version 559 | 3 of the GNU General Public License. 560 | 561 | 14. Revised Versions of this License. 562 | 563 | The Free Software Foundation may publish revised and/or new versions of 564 | the GNU Affero General Public License from time to time. Such new versions 565 | will be similar in spirit to the present version, but may differ in detail to 566 | address new problems or concerns. 567 | 568 | Each version is given a distinguishing version number. If the 569 | Program specifies that a certain numbered version of the GNU Affero General 570 | Public License "or any later version" applies to it, you have the 571 | option of following the terms and conditions either of that numbered 572 | version or of any later version published by the Free Software 573 | Foundation. If the Program does not specify a version number of the 574 | GNU Affero General Public License, you may choose any version ever published 575 | by the Free Software Foundation. 576 | 577 | If the Program specifies that a proxy can decide which future 578 | versions of the GNU Affero General Public License can be used, that proxy's 579 | public statement of acceptance of a version permanently authorizes you 580 | to choose that version for the Program. 581 | 582 | Later license versions may give you additional or different 583 | permissions. However, no additional obligations are imposed on any 584 | author or copyright holder as a result of your choosing to follow a 585 | later version. 586 | 587 | 15. Disclaimer of Warranty. 588 | 589 | THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY 590 | APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT 591 | HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY 592 | OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, 593 | THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR 594 | PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM 595 | IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF 596 | ALL NECESSARY SERVICING, REPAIR OR CORRECTION. 597 | 598 | 16. Limitation of Liability. 599 | 600 | IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING 601 | WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS 602 | THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY 603 | GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE 604 | USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF 605 | DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD 606 | PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS), 607 | EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF 608 | SUCH DAMAGES. 609 | 610 | 17. Interpretation of Sections 15 and 16. 611 | 612 | If the disclaimer of warranty and limitation of liability provided 613 | above cannot be given local legal effect according to their terms, 614 | reviewing courts shall apply local law that most closely approximates 615 | an absolute waiver of all civil liability in connection with the 616 | Program, unless a warranty or assumption of liability accompanies a 617 | copy of the Program in return for a fee. 618 | 619 | END OF TERMS AND CONDITIONS 620 | 621 | How to Apply These Terms to Your New Programs 622 | 623 | If you develop a new program, and you want it to be of the greatest 624 | possible use to the public, the best way to achieve this is to make it 625 | free software which everyone can redistribute and change under these terms. 626 | 627 | To do so, attach the following notices to the program. It is safest 628 | to attach them to the start of each source file to most effectively 629 | state the exclusion of warranty; and each file should have at least 630 | the "copyright" line and a pointer to where the full notice is found. 631 | 632 | 633 | Copyright (C) 634 | 635 | This program is free software: you can redistribute it and/or modify 636 | it under the terms of the GNU Affero General Public License as published 637 | by the Free Software Foundation, either version 3 of the License, or 638 | (at your option) any later version. 639 | 640 | This program is distributed in the hope that it will be useful, 641 | but WITHOUT ANY WARRANTY; without even the implied warranty of 642 | MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 643 | GNU Affero General Public License for more details. 644 | 645 | You should have received a copy of the GNU Affero General Public License 646 | along with this program. If not, see . 647 | 648 | Also add information on how to contact you by electronic and paper mail. 649 | 650 | If your software can interact with users remotely through a computer 651 | network, you should also make sure that it provides a way for users to 652 | get its source. For example, if your program is a web application, its 653 | interface could display a "Source" link that leads users to an archive 654 | of the code. There are many ways you could offer source, and different 655 | solutions will be better for different programs; see section 13 for the 656 | specific requirements. 657 | 658 | You should also get your employer (if you work as a programmer) or school, 659 | if any, to sign a "copyright disclaimer" for the program, if necessary. 660 | For more information on this, and how to apply and follow the GNU AGPL, see 661 | . 662 | -------------------------------------------------------------------------------- /ReadMe.md: -------------------------------------------------------------------------------- 1 | # BURO - freeware command line incremental backup utility for MS Windows.

# 2 |

3 | BURO supports data compression and encryption. You can access all your files and it's previous versions with your favorite file manager.

4 | 5 | [Click here to download](https://github.com/sapozhnikov/keeper/releases "Downloads page") 6 | >I'll be very appreciated if someone will help me to correct this translation. Dmitriy. 7 | ## How to backup ## 8 | Simple example how to backup single folder:

9 | >buro --backup --srcdir="" --dstdir=""

10 | 11 | If *source directory* contains files: 12 | >file1
13 | >file2
14 | >...
15 | >fileN

16 | 17 | new repository will be created at the *destination directory*: 18 | >
19 | >>file1
20 | >>file2
21 | >>...
22 | >>fileN
23 | 24 | >buro.db

25 | 26 | If you change *file1* and run the same command again, *file1* will be moved to the folder, named as current time, and changed file will be copied to *mirror* folder. Example: 27 | ><2017-08-31T00-37-12.048\>

28 | >>file1 <-old file

29 | > 30 | >
31 | >>file1 <- newer file
32 | >>file2
33 | >>...
34 | >>fileN
35 | 36 | >buro.db

37 | 38 | If you want to enable compression, add argument "--compress". You can set compression ratio from 1 (faster) to 9 (slower):

39 | >buro --backup --srcdir="" --dstdir="" --compress=9

40 | 41 | Extension ".bz2" will be added to the names at *destination directory*:

42 | >file1.bz2
43 | >file2.bz2
44 | >...
45 | >fileN.bz2

46 | 47 | If you want to protect your files with password, add argument *--password*:

48 | >buro --backup --srcdir="" --dstdir="" --password=""

49 | 50 | Extension ".encrypted" will be added to the names at *destination directory* and files will be encrypted with the strong encryption algorithm:

51 | >file1.encrypted
52 | >file2.encrypted
53 | >...
54 | >fileN.encrypted

55 | 56 | With argument *--encryptnames* file names will encrypted and looks like a garbage:
57 | >DXJKXWIXMQ
58 | >DXJKXWIXQY
59 | >...
60 | >DXJKXWLFKE
61 | 62 | You can combine *--compress* and *--password* arguments.

63 | ## How to restore ## 64 | 65 | Restoring is simple:

66 | >buro --restore --srcdir="" --dstdir=""

67 | 68 | Argument *srcdir* is pointing to the repository folder. Latest state of the files will be restored. If you want to restore files state at the given time, define it with argument *--timestamp*. Time's format is "YYYY-MM-DD 69 | HH:mm:SS.SSS".

70 | Do not forget to define password with argument *--password*, if backup was protected with password, or restoring will fail.

71 | ## How to delete old diffs ## 72 | You can delete old diffs to save disk space:

73 | >buro --purge --srcdir="" --older=

74 | 75 | Argument *srcdir* is pointing to the repository folder. You can define the date (format is "YYYY-MM-DD HH:mm:SS.SSS") or period (format is "PxYxMxDTxHxMxS"). For example, older than 1 month is "P1M", older than 1 minute is "PT1M", older than 1 year is "P1Y", one month and 5 days is "P1M5D".

76 | 77 | ### Filtering ### 78 | It is possible to exclude some directories or files while backuping or restoring. Wildmasks "*" and "?" are supported. You can define more than one mask, just separate them with symbol ";". If argument *--include* is defined, only matched files will be proceeded. If argument *--exclude* is defined, matched files will be skipped. Examples: 79 | >buro --backup --srcdir="" --dstdir="" --exclude="\*.tmp;\*.bak"

80 | >buro --restore --srcdir="" --dstdir="" --include="documents\*"

81 | 82 |

83 | -------------------------------------------------------------------------------- /keeper.sln: -------------------------------------------------------------------------------- 1 |  2 | Microsoft Visual Studio Solution File, Format Version 12.00 3 | # Visual Studio 15 4 | VisualStudioVersion = 15.0.28307.779 5 | MinimumVisualStudioVersion = 10.0.40219.1 6 | Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "keeper", "keeper\keeper.vcxproj", "{D42BBE5E-51FF-48A9-A338-2FB442AC63B7}" 7 | EndProject 8 | Global 9 | GlobalSection(SolutionConfigurationPlatforms) = preSolution 10 | Debug|x64 = Debug|x64 11 | Debug|x86 = Debug|x86 12 | Doctest debug x86|x64 = Doctest debug x86|x64 13 | Doctest debug x86|x86 = Doctest debug x86|x86 14 | Release|x64 = Release|x64 15 | Release|x86 = Release|x86 16 | EndGlobalSection 17 | GlobalSection(ProjectConfigurationPlatforms) = postSolution 18 | {D42BBE5E-51FF-48A9-A338-2FB442AC63B7}.Debug|x64.ActiveCfg = Debug|x64 19 | {D42BBE5E-51FF-48A9-A338-2FB442AC63B7}.Debug|x64.Build.0 = Debug|x64 20 | {D42BBE5E-51FF-48A9-A338-2FB442AC63B7}.Debug|x86.ActiveCfg = Debug|Win32 21 | {D42BBE5E-51FF-48A9-A338-2FB442AC63B7}.Debug|x86.Build.0 = Debug|Win32 22 | {D42BBE5E-51FF-48A9-A338-2FB442AC63B7}.Doctest debug x86|x64.ActiveCfg = Doctest debug x86|x64 23 | {D42BBE5E-51FF-48A9-A338-2FB442AC63B7}.Doctest debug x86|x64.Build.0 = Doctest debug x86|x64 24 | {D42BBE5E-51FF-48A9-A338-2FB442AC63B7}.Doctest debug x86|x86.ActiveCfg = Doctest debug x86|Win32 25 | {D42BBE5E-51FF-48A9-A338-2FB442AC63B7}.Doctest debug x86|x86.Build.0 = Doctest debug x86|Win32 26 | {D42BBE5E-51FF-48A9-A338-2FB442AC63B7}.Release|x64.ActiveCfg = Release|x64 27 | {D42BBE5E-51FF-48A9-A338-2FB442AC63B7}.Release|x64.Build.0 = Release|x64 28 | {D42BBE5E-51FF-48A9-A338-2FB442AC63B7}.Release|x86.ActiveCfg = Release|Win32 29 | {D42BBE5E-51FF-48A9-A338-2FB442AC63B7}.Release|x86.Build.0 = Release|Win32 30 | EndGlobalSection 31 | GlobalSection(SolutionProperties) = preSolution 32 | HideSolutionNode = FALSE 33 | EndGlobalSection 34 | GlobalSection(ExtensibilityGlobals) = postSolution 35 | SolutionGuid = {3E5A1040-EDE5-499D-8D10-D244BF544A5B} 36 | EndGlobalSection 37 | EndGlobal 38 | -------------------------------------------------------------------------------- /keeper/CLIparser.cpp: -------------------------------------------------------------------------------- 1 | #include "stdafx.h" 2 | #include 3 | #include 4 | 5 | #pragma warning(push) 6 | #pragma warning(disable:4459) 7 | #include 8 | #pragma warning(pop) 9 | #include "TaskContext.h" 10 | 11 | //WARNING: timestamps as std::string, cause BOOST libs, period as std::string cause RegEx libs 12 | 13 | using namespace boost::program_options; 14 | using namespace ConsoleLogger; 15 | 16 | class WrongArguments final {}; 17 | 18 | //returns 'true' if its OK to proceed the task 19 | bool ParseCLITask(keeper::TaskContext& ctx, int argc, wchar_t *argv[]) 20 | { 21 | options_description commonDesc("Allowed options"); 22 | commonDesc.add_options() 23 | ("backup,B", "backup files") 24 | ("restore,R", "restore files from backup") 25 | ("purge,P", "erase old changes") 26 | ("dumpdb,D", "dump database"); 27 | 28 | options_description otherOptionsDesc("Other options"); 29 | otherOptionsDesc.add_options() 30 | ("verbose,V", "verbose mode") 31 | ("printloglevel", "add log level to messages") 32 | ("printtimestamp", "add timestamp to messages") 33 | ("password", value(), "password for accessing files") 34 | ("encryptnames", "encrypt file names") 35 | ("include,I", wvalue(), "mask to include files (only matched files will be included, if specified)") 36 | ("exclude,E", wvalue(), "mask to exclude files"); 37 | 38 | options_description backupDesc("Backup options"); 39 | backupDesc.add_options() 40 | ("srcdir,S", wvalue(), "full path to directory to backup") 41 | ("dstdir,D", wvalue(), "full path to directory there data will be stored") 42 | ("compress", value()->implicit_value(5), "compress files (1-faster, ..., 9-slower)") 43 | ("threads", value(), "number of threads used for compression. if threads=0, than number of threads will be equal to number of CPU cores"); 44 | 45 | options_description restoreDesc("Restore options"); 46 | restoreDesc.add_options() 47 | ("srcdir,S", wvalue(), "full path to directory there is data stored") 48 | ("dstdir,D", wvalue(), "full path to directory there data will be written") 49 | ("timestamp,T", value(), "restore files state at the given time, or the latest state, if not defined (format \"YYYY-MM-DD HH:mm:SS.SSS\")"); 50 | 51 | options_description dumpDesc("Dump DB options"); 52 | dumpDesc.add_options() 53 | ("srcdir,S", wvalue(), "full path to directory there is data stored"); 54 | 55 | options_description purgeDesc("Purge options"); 56 | purgeDesc.add_options() 57 | ("srcdir,S", wvalue(), "full path to directory there is data stored") 58 | ("older,O", value(), "delete changes older than date (format \"YYYY-MM-DD HH:mm:SS.SSS\") or period (format P1Y2M3DT4H5M6S)"); 59 | 60 | try 61 | { 62 | variables_map varMap; 63 | //preparsing task 64 | wparsed_options preparsed = wcommand_line_parser(argc, argv).options(commonDesc).allow_unregistered().run(); 65 | store(preparsed, varMap); 66 | notify(varMap); 67 | 68 | //----------------- 69 | keeper::Task task = keeper::Task::Undetected; 70 | if (varMap.count("backup")) 71 | task = keeper::Task::Backup; 72 | 73 | if (varMap.count("restore")) 74 | { 75 | if (task != keeper::Task::Undetected) 76 | throw WrongArguments(); 77 | else 78 | task = keeper::Task::Restore; 79 | } 80 | 81 | if (varMap.count("dumpdb")) 82 | { 83 | if (task != keeper::Task::Undetected) 84 | throw WrongArguments(); 85 | else 86 | task = keeper::Task::DumpDB; 87 | } 88 | 89 | if (varMap.count("purge")) 90 | { 91 | if (task != keeper::Task::Undetected) 92 | throw WrongArguments(); 93 | else 94 | task = keeper::Task::Purge; 95 | } 96 | 97 | if (task == keeper::Task::Undetected) 98 | throw WrongArguments(); 99 | 100 | variables_map varMapOther; 101 | wparsed_options otherParsed = wcommand_line_parser(argc, argv).options(otherOptionsDesc).allow_unregistered().run(); 102 | store(otherParsed, varMapOther); 103 | notify(varMapOther); 104 | 105 | if (varMapOther.count("verbose")) 106 | { 107 | ConsoleLogger::SetLogLevel(ConsoleLogger::LogLevel::verbose); 108 | } 109 | 110 | if (varMapOther.count("printloglevel")) 111 | { 112 | ConsoleLogger::SetAttrLogLevelVisible(true); 113 | } 114 | 115 | if (varMapOther.count("printtimestamp")) 116 | { 117 | ConsoleLogger::SetAttrTimestampVisible(true); 118 | } 119 | 120 | if (varMapOther.count("encryptnames")) 121 | { 122 | ctx.isEncryptedFileNames = true; 123 | } 124 | 125 | if (varMapOther.count("password")) 126 | { 127 | ctx.DbPassword = varMapOther["password"].as(); 128 | } 129 | 130 | if (varMapOther.count("include")) 131 | { 132 | ctx.NamesChecker.AddIncludeMask(varMapOther["include"].as()); 133 | } 134 | 135 | if (varMapOther.count("exclude")) 136 | { 137 | ctx.NamesChecker.AddExcludeMask(varMapOther["exclude"].as()); 138 | } 139 | 140 | if (ctx.isEncryptedFileNames && ctx.DbPassword.empty()) 141 | { 142 | LOG_FATAL() << "Password missed" << std::endl; 143 | throw WrongArguments(); 144 | } 145 | 146 | //backup 147 | if (task == keeper::Task::Backup) 148 | { 149 | options_description tmpOptions; 150 | tmpOptions.add(commonDesc).add(otherOptionsDesc).add(backupDesc); 151 | variables_map varMapBackup; 152 | store(parse_command_line(argc, argv, tmpOptions), varMapBackup); 153 | notify(varMapBackup); 154 | 155 | if (varMapBackup.count("srcdir")) 156 | ctx.SetSourceDirectory(varMapBackup["srcdir"].as()); 157 | else 158 | throw WrongArguments(); 159 | 160 | if (varMapBackup.count("dstdir")) 161 | ctx.SetDestinationDirectory(varMapBackup["dstdir"].as()); 162 | else 163 | throw WrongArguments(); 164 | 165 | if (varMapBackup.count("compress")) 166 | ctx.CompressionLevel = varMapBackup["compress"].as(); 167 | else 168 | ctx.CompressionLevel = 0; 169 | 170 | if (ctx.CompressionLevel != 0) 171 | { 172 | if (varMapBackup.count("threads")) 173 | { 174 | ctx.Threads = varMapBackup["threads"].as(); 175 | if (ctx.Threads == 0) 176 | ctx.Threads = std::thread::hardware_concurrency(); 177 | } 178 | else 179 | ctx.Threads = 1; 180 | } 181 | 182 | ctx.Task = keeper::Task::Backup; 183 | 184 | return true; 185 | } 186 | 187 | //restore 188 | if (task == keeper::Task::Restore) 189 | { 190 | options_description tmpOptions; 191 | tmpOptions.add(commonDesc).add(otherOptionsDesc).add(restoreDesc); 192 | variables_map varMapRestore; 193 | store(parse_command_line(argc, argv, tmpOptions), varMapRestore); 194 | notify(varMapRestore); 195 | 196 | if (varMapRestore.count("srcdir")) 197 | ctx.SetSourceDirectory(varMapRestore["srcdir"].as()); 198 | else 199 | throw WrongArguments(); 200 | 201 | if (varMapRestore.count("dstdir")) 202 | ctx.SetDestinationDirectory(varMapRestore["dstdir"].as()); 203 | else 204 | throw WrongArguments(); 205 | 206 | if (varMapRestore.count("timestamp")) 207 | { 208 | if (!ctx.SetRestoreTimeStamp(varMapRestore["timestamp"].as())) //string! 209 | throw WrongArguments(); 210 | } 211 | else 212 | ctx.SetRestoreTimeStamp(std::string()); 213 | 214 | ctx.Task = keeper::Task::Restore; 215 | 216 | return true; 217 | } 218 | 219 | //dump 220 | if (task == keeper::Task::DumpDB) 221 | { 222 | options_description tmpOptions; 223 | tmpOptions.add(commonDesc).add(otherOptionsDesc).add(dumpDesc); 224 | variables_map varMapDumpDB; 225 | store(parse_command_line(argc, argv, tmpOptions), varMapDumpDB); 226 | notify(varMapDumpDB); 227 | 228 | if (varMapDumpDB.count("srcdir")) 229 | ctx.SetSourceDirectory(varMapDumpDB["srcdir"].as()); 230 | else 231 | throw WrongArguments(); 232 | 233 | ctx.Task = keeper::Task::DumpDB; 234 | 235 | return true; 236 | } 237 | 238 | //purge 239 | if (task == keeper::Task::Purge) 240 | { 241 | options_description tmpOptions; 242 | tmpOptions.add(commonDesc).add(otherOptionsDesc).add(purgeDesc); 243 | variables_map varMapPurge; 244 | store(parse_command_line(argc, argv, tmpOptions), varMapPurge); 245 | notify(varMapPurge); 246 | 247 | if (varMapPurge.count("srcdir")) 248 | ctx.SetSourceDirectory(varMapPurge["srcdir"].as()); 249 | else 250 | throw WrongArguments(); 251 | 252 | if (varMapPurge.count("older")) 253 | { 254 | if (!ctx.SetPurgeTimestampFromArg(varMapPurge["older"].as())) 255 | throw WrongArguments(); 256 | } 257 | 258 | ctx.Task = keeper::Task::Purge; 259 | 260 | return true; 261 | } 262 | } 263 | catch (WrongArguments) 264 | { 265 | //show help 266 | std::cout << commonDesc << std::endl; 267 | std::cout << backupDesc << std::endl; 268 | std::cout << restoreDesc << std::endl; 269 | std::cout << purgeDesc << std::endl; 270 | std::cout << dumpDesc << std::endl; 271 | std::cout << otherOptionsDesc << std::endl; 272 | 273 | return false; 274 | } 275 | catch (...) 276 | { 277 | LOG_FATAL() << "Error while parsing arguments" << std::endl; 278 | } 279 | 280 | return false; 281 | } -------------------------------------------------------------------------------- /keeper/CLIparser.h: -------------------------------------------------------------------------------- 1 | #pragma once 2 | 3 | bool ParseCLITask(keeper::TaskContext& ctx, int argc, wchar_t *argv[]); -------------------------------------------------------------------------------- /keeper/ChaChaFilter.h: -------------------------------------------------------------------------------- 1 | #pragma once 2 | #include 3 | #include 4 | #include 5 | 6 | using namespace boost::iostreams; 7 | 8 | class ChaCha20filter 9 | { 10 | public: 11 | typedef char char_type; 12 | struct category : output_filter_tag, flushable_tag, multichar_tag { }; 13 | 14 | explicit ChaCha20filter(const byte* key, const byte* nonce) 15 | { 16 | memcpy(key_, key, crypto_stream_chacha20_KEYBYTES); 17 | memcpy(nonce_, nonce, crypto_stream_chacha20_NONCEBYTES); 18 | } 19 | 20 | ~ChaCha20filter() 21 | { 22 | sodium_memzero(key_, crypto_stream_chacha20_KEYBYTES); 23 | sodium_memzero(nonce_, crypto_stream_chacha20_NONCEBYTES); 24 | } 25 | 26 | ChaCha20filter& operator=(const ChaCha20filter&) = delete; 27 | 28 | //WARNING!!! it's writes to the input buffer!!! 29 | template 30 | std::streamsize write(Sink& dest, const char* s, std::streamsize n) 31 | { 32 | int64_t inlen = int(n); 33 | 34 | crypto_stream_chacha20_xor_ic((unsigned char*)s, (unsigned char*)s, inlen, nonce_, counter_, key_); 35 | counter_ += inlen; 36 | 37 | boost::iostreams::write(dest, s, inlen); 38 | return n; 39 | } 40 | 41 | template 42 | bool flush(Sink& /*dest*/) 43 | { 44 | if (!isFlushed_) 45 | { 46 | //nothing to do 47 | isFlushed_ = true; 48 | } 49 | return true; 50 | } 51 | 52 | private: 53 | 54 | unsigned char key_[crypto_stream_chacha20_KEYBYTES]; 55 | unsigned char nonce_[crypto_stream_chacha20_NONCEBYTES]; 56 | uint64_t counter_ = 0; 57 | 58 | bool isFlushed_ = false; 59 | }; 60 | -------------------------------------------------------------------------------- /keeper/CommonDefinitions.h: -------------------------------------------------------------------------------- 1 | #pragma once 2 | #ifdef _WIN64 3 | const char * const MESSAGE_ABOUT = "BURO - backup utility v2.0/W64 by Dmitriy Sapozhnikov"; 4 | #else 5 | const char * const MESSAGE_ABOUT = "BURO - backup utility v2.0/W32 by 2017 Dmitriy Sapozhnikov"; 6 | #endif 7 | const wchar_t * const MIRROR_SUB_DIR = L"mirror\\"; 8 | const wchar_t * const ENV_SUB_DIR = L"env\\"; 9 | const wchar_t * const MAIN_DB_FILE = L"buro.db"; 10 | const char * const EVENTS_DB_TABLE = "events"; 11 | const char * const CONFIG_DB_TABLE = "config"; 12 | const wchar_t * const NAME_SUFFIX_COMPRESSED = L".xz"; 13 | const wchar_t * const NAME_SUFFIX_ENCRYPTED = L".encrypted"; 14 | const wchar_t MASKS_SEPARATOR = L';'; 15 | const unsigned int LOG_FILE_SIZE = 128 * 1024 * 1024; //DB log max file size 16 | const unsigned int LOG_BUF_SIZE = 1024 * 1024; //DB log buffer 17 | const unsigned int ENV_CACHE_SIZE = 16 * 1024 * 1024; //DB memory pool 18 | const unsigned int MAX_PATH_LENGTH = 32767; 19 | 20 | template int sgn(T val) { 21 | return (T(0) < val) - (val < T(0)); 22 | } 23 | 24 | enum class FileEventType 25 | { 26 | Deleted, 27 | Added, 28 | Changed 29 | }; 30 | 31 | // TODO: change to serialization 32 | #pragma pack(push, 1) 33 | struct DbFileEvent 34 | { 35 | int64_t EventTimeStamp; 36 | byte FileEvent; 37 | WIN32_FILE_ATTRIBUTE_DATA FileAttributes; 38 | }; 39 | #pragma pack(pop) -------------------------------------------------------------------------------- /keeper/ConsoleLogger.cpp: -------------------------------------------------------------------------------- 1 | #include "stdafx.h" 2 | #include "ConsoleLogger.h" 3 | #include "GlobalUtils.h" 4 | 5 | namespace ConsoleLogger 6 | { 7 | void SetLogLevel(LogLevel lvl) 8 | { 9 | LogConfig::GetInstance().CurrentLogLevel = lvl; 10 | } 11 | 12 | void ConsoleLogger::SetAttrLogLevelVisible(bool b) 13 | { 14 | LogConfig::GetInstance().AddLogLevelName = b; 15 | } 16 | 17 | void ConsoleLogger::SetAttrTimestampVisible(bool b) 18 | { 19 | LogConfig::GetInstance().AddTimestamp = b; 20 | } 21 | 22 | LogLevel ConsoleLogger::GetMaxLogLevelPrinted() 23 | { 24 | return LogConfig::GetInstance().MaxLogLevelPrinted; 25 | } 26 | 27 | LogTrace& LOG_TRACE() 28 | { 29 | LogTrace& inst = LogTrace::GetInstance(); 30 | inst.PrintLogEntryAttributes(); 31 | return inst; 32 | } 33 | 34 | LogDebug& LOG_DEBUG() 35 | { 36 | LogDebug& inst = LogDebug::GetInstance(); 37 | inst.PrintLogEntryAttributes(); 38 | return inst; 39 | } 40 | 41 | LogVerbose& LOG_VERBOSE() 42 | { 43 | LogVerbose& inst = LogVerbose::GetInstance(); 44 | inst.PrintLogEntryAttributes(); 45 | return inst; 46 | } 47 | 48 | LogInfo& LOG_INFO() 49 | { 50 | LogInfo& inst = LogInfo::GetInstance(); 51 | inst.PrintLogEntryAttributes(); 52 | return inst; 53 | } 54 | 55 | LogWarning& LOG_WARNING() 56 | { 57 | LogWarning& inst = LogWarning::GetInstance(); 58 | inst.PrintLogEntryAttributes(); 59 | return inst; 60 | } 61 | 62 | LogError& LOG_ERROR() 63 | { 64 | LogError& inst = LogError::GetInstance(); 65 | inst.PrintLogEntryAttributes(); 66 | return inst; 67 | } 68 | 69 | LogFatal& LOG_FATAL() 70 | { 71 | LogFatal& inst = LogFatal::GetInstance(); 72 | inst.PrintLogEntryAttributes(); 73 | return inst; 74 | } 75 | 76 | LogMandatory& CONSOLE() 77 | { 78 | //LogMandatory::GetInstance().PrintLogEntryAttributes(); 79 | return LogMandatory::GetInstance(); 80 | } 81 | } -------------------------------------------------------------------------------- /keeper/ConsoleLogger.h: -------------------------------------------------------------------------------- 1 | #pragma once 2 | 3 | // Simple console logger by Dmitriy Sapozhnikov 4 | // 5 | // Usage example: 6 | // 7 | // using namespace ConsoleLogger; 8 | // SetLogLevel(LogLevel::info); 9 | // LOG_WARNING() << "some warning text"; 10 | // 11 | // Error and Fatal messages goes to STDERR, others goes to STDOUT 12 | 13 | #include 14 | #include 15 | #include 16 | 17 | namespace ConsoleLogger 18 | { 19 | //Levels of logging 20 | typedef enum 21 | { 22 | trace, 23 | debug, 24 | verbose, 25 | info, //default 26 | warning, 27 | error, 28 | fatal, 29 | mandatory 30 | } LogLevel; 31 | 32 | static char* LogLevelName[] = { 33 | "TRACE", 34 | "DEBUG", 35 | "VERBOSE", 36 | "INFO", 37 | "WARNING", 38 | "ERROR", 39 | "FATAL", 40 | "MANDATORY" 41 | }; 42 | 43 | //Singleton class which stores logger config 44 | class LogConfig 45 | { 46 | public: 47 | LogLevel CurrentLogLevel; 48 | LogLevel MaxLogLevelPrinted = LogLevel::trace; // keep it for program return code 49 | bool AddTimestamp = false; 50 | bool AddLogLevelName = false; 51 | 52 | //singleton stuff 53 | public: 54 | static LogConfig& GetInstance() 55 | { 56 | static LogConfig s; 57 | return s; 58 | } 59 | private: 60 | LogConfig() : CurrentLogLevel(LogLevel::info) 61 | {} 62 | LogConfig(LogConfig const&) = delete; 63 | LogConfig& operator= (LogConfig const&) = delete; 64 | }; 65 | 66 | 67 | void SetLogLevel(LogLevel lvl); 68 | void SetAttrLogLevelVisible(bool b); 69 | void SetAttrTimestampVisible(bool b); 70 | LogLevel GetMaxLogLevelPrinted(); 71 | 72 | //Base class for other loggers 73 | template 74 | class BaseLogger final 75 | { 76 | public: 77 | template 78 | BaseLogger& operator << (const T& arg) 79 | { 80 | if (IsLogLevelEnough()) 81 | OSTREAM << arg; 82 | return *this; 83 | } 84 | 85 | template <> 86 | BaseLogger& operator << (const std::wstring& arg) 87 | { 88 | if (IsLogLevelEnough()) 89 | OSTREAM << convertWStringToString(arg).c_str(); 90 | return *this; 91 | } 92 | 93 | BaseLogger& operator << (const wchar_t* p) 94 | { 95 | if (IsLogLevelEnough()) 96 | OSTREAM << convertPWCharToString(p).c_str(); 97 | return *this; 98 | } 99 | 100 | //specification for stream manipulators 101 | BaseLogger& operator<<(std::ostream& (*os)(std::ostream&)) 102 | { 103 | if (IsLogLevelEnough()) 104 | OSTREAM << os; 105 | return *this; 106 | } 107 | 108 | void PrintLogEntryAttributes() 109 | { 110 | if (!IsLogLevelEnough()) 111 | return; 112 | 113 | LogConfig& config = LogConfig::GetInstance(); 114 | 115 | if (config.AddTimestamp) 116 | (*this) << '[' << keeper::PTimeToWstring(boost::posix_time::microsec_clock::local_time()) << ']'; 117 | if (config.AddLogLevelName) 118 | (*this) << '[' << LogLevelName[LOGLEVEL] << ']'; 119 | } 120 | 121 | private: 122 | bool IsLogLevelEnough() 123 | { 124 | #pragma warning(suppress: 4127) 125 | if (LOGLEVEL == LogLevel::mandatory) 126 | return true; 127 | else 128 | { 129 | LogConfig& inst = LogConfig::GetInstance(); 130 | if (inst.MaxLogLevelPrinted < LOGLEVEL) 131 | inst.MaxLogLevelPrinted = LOGLEVEL; 132 | return inst.CurrentLogLevel <= LOGLEVEL; 133 | } 134 | } 135 | 136 | std::string convertWStringToString(const std::wstring& source) 137 | { 138 | if (source.empty()) 139 | return std::string(); 140 | 141 | int length = ::WideCharToMultiByte(CP_OEMCP, 0, source.data(), (int)source.length(), NULL, 0, NULL, NULL); 142 | if (length <= 0) 143 | return std::string(); 144 | 145 | std::vector buffer(length); 146 | ::WideCharToMultiByte(CP_OEMCP, 0, source.data(), (int)source.length(), buffer.data(), length, NULL, NULL); 147 | 148 | return std::string(buffer.begin(), buffer.end()); 149 | } 150 | 151 | std::string convertPWCharToString(const wchar_t* p) 152 | { 153 | if (p == nullptr) 154 | return std::string(); 155 | size_t sourceLength = std::wcslen(p); 156 | if (sourceLength == 0) 157 | return std::string(); 158 | 159 | int length = ::WideCharToMultiByte(CP_OEMCP, 0, p, sourceLength, NULL, 0, NULL, NULL); 160 | if (length <= 0) 161 | return std::string(); 162 | 163 | std::vector buffer(length); 164 | ::WideCharToMultiByte(CP_OEMCP, 0, p, sourceLength, buffer.data(), length, NULL, NULL); 165 | 166 | return std::string(buffer.begin(), buffer.end()); 167 | } 168 | 169 | //singleton stuff 170 | public: 171 | static BaseLogger& GetInstance() 172 | { 173 | static BaseLogger log; 174 | return log; 175 | } 176 | private: 177 | BaseLogger() {} 178 | BaseLogger(BaseLogger const&) = delete; 179 | BaseLogger& operator= (BaseLogger const&) = delete; 180 | }; 181 | 182 | typedef BaseLogger LogTrace; 183 | typedef BaseLogger LogDebug; 184 | typedef BaseLogger LogVerbose; 185 | typedef BaseLogger LogInfo; 186 | typedef BaseLogger LogWarning; 187 | typedef BaseLogger LogError; 188 | typedef BaseLogger LogFatal; 189 | typedef BaseLogger LogMandatory; 190 | 191 | LogTrace& LOG_TRACE(); 192 | LogDebug& LOG_DEBUG(); 193 | LogVerbose& LOG_VERBOSE(); 194 | LogInfo& LOG_INFO(); 195 | LogWarning& LOG_WARNING(); 196 | LogError& LOG_ERROR(); 197 | LogFatal& LOG_FATAL(); 198 | LogMandatory& CONSOLE(); 199 | } -------------------------------------------------------------------------------- /keeper/DbCompressTest.cpp: -------------------------------------------------------------------------------- 1 | #include "stdafx.h" 2 | #include "doctest\doctest.h" 3 | #include "TaskContext.h" 4 | #include "TaskPurge.h" 5 | 6 | TEST_CASE("DbCompress") 7 | { 8 | //just local test 9 | /* 10 | keeper::TaskContext ctx; 11 | ctx.SetSourceDirectory(L"D:\\\\test"); 12 | ctx.Task = keeper::Task::Purge; 13 | CHECK(ctx.OpenDatabase() == true); 14 | 15 | ctx.CompressDatabase(); 16 | */ 17 | } -------------------------------------------------------------------------------- /keeper/FileIO.cpp: -------------------------------------------------------------------------------- 1 | #include "stdafx.h" 2 | #include "FileIO.h" 3 | #include "GlobalUtils.h" 4 | 5 | using namespace ConsoleLogger; 6 | using namespace boost::filesystem; 7 | 8 | namespace keeper 9 | { 10 | namespace FileIO 11 | { 12 | std::string GetLastErrorAsString() 13 | { 14 | //Get the error message, if any. 15 | DWORD errorMessageID = ::GetLastError(); 16 | if (errorMessageID == 0) 17 | return std::string(); //No error message has been recorded 18 | 19 | LPSTR messageBuffer = nullptr; 20 | size_t size = FormatMessageA(FORMAT_MESSAGE_ALLOCATE_BUFFER | FORMAT_MESSAGE_FROM_SYSTEM | FORMAT_MESSAGE_IGNORE_INSERTS, 21 | NULL, errorMessageID, MAKELANGID(LANG_NEUTRAL, SUBLANG_DEFAULT), (LPSTR)&messageBuffer, 0, NULL); 22 | 23 | std::string message(messageBuffer, size); 24 | 25 | //Free the buffer. 26 | LocalFree(messageBuffer); 27 | 28 | return StrAnsiToOEM(message); 29 | } 30 | 31 | bool CopySingleFile(const boost::filesystem::path& source, const boost::filesystem::path& destination, bool failIfExists) 32 | { 33 | boost::system::error_code errorCode; 34 | 35 | LOG_VERBOSE() << "Copying file from " << source.wstring() << " to " << destination.wstring() << std::endl; 36 | 37 | boost::filesystem::create_directories(boost::filesystem::path(destination).parent_path()); 38 | if (failIfExists) 39 | boost::filesystem::copy_file(source, destination, boost::filesystem::copy_option::fail_if_exists, errorCode); 40 | else 41 | boost::filesystem::copy_file(source, destination, errorCode); 42 | 43 | if (!errorCode) 44 | return true; 45 | else 46 | { 47 | LOG_ERROR() << "Can't copy file from " << source.wstring() << " to " << destination.wstring() << " : " << StrAnsiToOEM(errorCode.message()) << std::endl; 48 | return false; 49 | } 50 | } 51 | 52 | //remove R/O attribute from file or directory with subdirs and files 53 | bool RemoveReadOnlyAttribute(const boost::filesystem::path& fileName) 54 | { 55 | boost::system::error_code errorCode; 56 | 57 | permissions(fileName, status(fileName).permissions() | owner_write | group_write | others_write, errorCode); 58 | if (errorCode) 59 | { 60 | LOG_ERROR() << "Can't change file attributes: " << fileName.wstring() << " : " << StrAnsiToOEM(errorCode.message()) << std::endl; 61 | return false; 62 | } 63 | 64 | if (is_directory(fileName)) 65 | { 66 | recursive_directory_iterator dirIterator(fileName); 67 | while (true) 68 | { 69 | if (dirIterator == recursive_directory_iterator()) 70 | break; 71 | permissions(dirIterator->path(), status(dirIterator->path()).permissions() | owner_write | group_write | others_write, errorCode); 72 | if (errorCode) 73 | { 74 | LOG_ERROR() << "Can't change file attributes: " << fileName.wstring() << " : " << StrAnsiToOEM(errorCode.message()) << std::endl; 75 | return false; 76 | } 77 | 78 | dirIterator++; 79 | } 80 | } 81 | return true; 82 | } 83 | 84 | bool MoveSingleFile(const boost::filesystem::path& source, const boost::filesystem::path& destination) 85 | { 86 | boost::system::error_code errorCode; 87 | 88 | LOG_VERBOSE() << "Moving file from " << source.wstring() << " to " << destination.wstring() << std::endl; 89 | 90 | boost::filesystem::create_directories(boost::filesystem::path(destination).parent_path(), errorCode); 91 | if (errorCode) 92 | { 93 | LOG_ERROR() << "Can't create directories for " << destination.wstring() << " : " << StrAnsiToOEM(errorCode.message()) << std::endl; 94 | //return false; 95 | } 96 | 97 | boost::filesystem::rename(source, destination, errorCode); 98 | if (errorCode) 99 | { 100 | LOG_ERROR() << "Can't move file from " << source.wstring() << " to " << destination.wstring() << " : " << StrAnsiToOEM(errorCode.message()) << std::endl; 101 | return false; 102 | } 103 | return true; 104 | } 105 | 106 | bool RemoveDir(const boost::filesystem::path& dir) 107 | { 108 | boost::system::error_code errorCode; 109 | 110 | LOG_VERBOSE() << "Deleting directory " << dir.wstring() << std::endl; 111 | 112 | RemoveReadOnlyAttribute(dir); 113 | boost::filesystem::remove_all(dir, errorCode); 114 | if (errorCode) 115 | { 116 | LOG_ERROR() << "Can't delete directory " << dir.wstring() << " : " << StrAnsiToOEM(errorCode.message()) << std::endl; 117 | return false; 118 | } 119 | 120 | return true; 121 | } 122 | 123 | bool CreateDir(const boost::filesystem::path& dir) 124 | { 125 | boost::system::error_code errorCode; 126 | 127 | LOG_VERBOSE() << "Creating directory " << dir.wstring() << std::endl; 128 | 129 | //boost::filesystem::create_directories(dir.parent_path()); //! 130 | boost::filesystem::create_directories(dir, errorCode); 131 | if (!errorCode) 132 | return true; 133 | else 134 | { 135 | LOG_ERROR() << "Can't create directory " << dir.wstring() << " : " << StrAnsiToOEM(errorCode.message()) << std::endl; 136 | return false; 137 | } 138 | } 139 | 140 | bool CopyDir(const boost::filesystem::path& source, const boost::filesystem::path& destination) 141 | { 142 | boost::system::error_code errorCode; 143 | 144 | LOG_VERBOSE() << "Copying directory from " << source.wstring() << " to " << destination.wstring() << std::endl; 145 | 146 | boost::filesystem::copy_directory(source, destination, errorCode); 147 | if (!errorCode) 148 | return true; 149 | else 150 | { 151 | LOG_ERROR() << "Can't copy directory from " << source.wstring() << " to " << destination.wstring() << " : " << StrAnsiToOEM(errorCode.message()) << std::endl; 152 | return false; 153 | } 154 | } 155 | } 156 | } -------------------------------------------------------------------------------- /keeper/FileIO.h: -------------------------------------------------------------------------------- 1 | #pragma once 2 | #include 3 | 4 | namespace keeper 5 | { 6 | namespace FileIO 7 | { 8 | bool CopySingleFile(const boost::filesystem::path& source, const boost::filesystem::path& destination, bool failIfExists = false); 9 | bool MoveSingleFile(const boost::filesystem::path& source, const boost::filesystem::path& destination); 10 | bool RemoveDir(const boost::filesystem::path& dir); 11 | bool CreateDir(const boost::filesystem::path& dir); 12 | bool CopyDir(const boost::filesystem::path& source, const boost::filesystem::path& destination); 13 | } 14 | } -------------------------------------------------------------------------------- /keeper/FilesTransformer.cpp: -------------------------------------------------------------------------------- 1 | #include "stdafx.h" 2 | #include "FilesTransformer.h" 3 | #include "CommonDefinitions.h" 4 | #include "FileIO.h" 5 | #include 6 | #include 7 | #include 8 | #include 9 | 10 | using namespace ConsoleLogger; 11 | 12 | namespace keeper 13 | { 14 | FilesTransformer::FilesTransformer(const TaskContext & ctx) : 15 | pathEncoder(ctx), 16 | ctx_(ctx) 17 | { 18 | isFileCompressed = (ctx.CompressionLevel != 0); 19 | isFileEncrypted = !ctx.DbPassword.empty(); 20 | isFileNamesEncrypted = ctx.isEncryptedFileNames; 21 | memcpy(FileEncryptKey, ctx.FileEncodeKey_, crypto_stream_chacha20_KEYBYTES); 22 | } 23 | 24 | FilesTransformer::~FilesTransformer() 25 | { 26 | sodium_memzero(FileEncryptKey, crypto_stream_chacha20_KEYBYTES); 27 | } 28 | 29 | std::wstring FilesTransformer::GetTransformedName(const std::wstring & name, bool isDirectory) 30 | { 31 | if (isFileNamesEncrypted) 32 | return pathEncoder.encode(name); 33 | if (!isDirectory) 34 | { 35 | if (isFileEncrypted) 36 | return name + NAME_SUFFIX_ENCRYPTED; 37 | if (isFileCompressed) 38 | return name + NAME_SUFFIX_COMPRESSED; 39 | } 40 | return name; 41 | } 42 | 43 | std::wstring FilesTransformer::GetOriginalName(const std::wstring & name, bool isDirectory) 44 | { 45 | auto const static LenSuffixEncrypted = wcslen(NAME_SUFFIX_ENCRYPTED); 46 | auto const static LenSuffixCompressed = wcslen(NAME_SUFFIX_COMPRESSED); 47 | 48 | if (isFileNamesEncrypted) 49 | return pathEncoder.decode(name); 50 | 51 | if (!isDirectory) 52 | { 53 | if (isFileEncrypted) 54 | { 55 | if ((name.length() < LenSuffixEncrypted) || 56 | (name.substr(name.length() - LenSuffixEncrypted, name.length()) != NAME_SUFFIX_ENCRYPTED)) 57 | throw std::exception("Bad file name"); 58 | return name.substr(0, name.length() - LenSuffixEncrypted); 59 | } 60 | else 61 | if (isFileCompressed) 62 | { 63 | if ((name.length() < LenSuffixCompressed) || 64 | (name.substr(name.length() - LenSuffixCompressed, name.length()) != NAME_SUFFIX_COMPRESSED)) 65 | throw std::exception("Bad file name"); 66 | return name.substr(0, name.length() - LenSuffixCompressed); 67 | } 68 | } 69 | 70 | return name; 71 | } 72 | 73 | bool FilesTransformer::TransformFile(const std::wstring & pathFrom, const std::wstring & pathTo) 74 | { 75 | if (!isFileCompressed && !isFileEncrypted) 76 | { 77 | return FileIO::CopySingleFile(pathFrom, pathTo); 78 | } 79 | else 80 | { 81 | LOG_VERBOSE() << "Storing file " << pathFrom << " to " << pathTo << std::endl; 82 | 83 | std::ifstream inFile(pathFrom, std::ios::binary | std::ios::in); 84 | if (!inFile.is_open()) 85 | return false; 86 | 87 | boost::filesystem::create_directories(boost::filesystem::path(pathTo).parent_path()); 88 | std::ofstream outFile(pathTo, std::ios::binary | std::ios::out | std::ios::trunc); 89 | if (isFileEncrypted) 90 | { 91 | //random nonce for each file 92 | randombytes_buf(FileEncryptNonce, crypto_stream_chacha20_NONCEBYTES); 93 | outFile.write(reinterpret_cast(FileEncryptNonce), crypto_stream_chacha20_NONCEBYTES); 94 | } 95 | boost::iostreams::filtering_streambuf outStream; 96 | boost::iostreams::lzma_params lzmaParams; 97 | if (isFileCompressed) 98 | { 99 | lzmaParams.level = ctx_.CompressionLevel; 100 | lzmaParams.threads = ctx_.Threads; 101 | outStream.push(boost::iostreams::lzma_compressor(lzmaParams)); 102 | } 103 | if (isFileEncrypted) 104 | outStream.push(ChaCha20filter(FileEncryptKey, FileEncryptNonce)); 105 | outStream.push(outFile); 106 | 107 | boost::iostreams::copy(inFile, outStream); 108 | 109 | return true; 110 | } 111 | } 112 | 113 | bool FilesTransformer::RestoreFile(const std::wstring & pathFrom, const std::wstring & pathTo) 114 | { 115 | if (!isFileCompressed && !isFileEncrypted) 116 | { 117 | return FileIO::CopySingleFile(pathFrom, pathTo); 118 | } 119 | else 120 | { 121 | LOG_VERBOSE() << "Restoring file from " << pathFrom << " to " << pathTo << std::endl; 122 | 123 | std::ifstream inFile(pathFrom, std::ios::binary | std::ios::in); 124 | if (!inFile.is_open()) 125 | return false; 126 | 127 | if (isFileEncrypted) 128 | { 129 | inFile.read(reinterpret_cast(FileEncryptNonce), crypto_stream_chacha20_NONCEBYTES); 130 | if (inFile.gcount() < crypto_stream_chacha20_NONCEBYTES) 131 | return false; 132 | } 133 | boost::iostreams::filtering_streambuf outStream; 134 | 135 | boost::filesystem::create_directories(boost::filesystem::path(pathTo).parent_path()); 136 | std::ofstream outFile(pathTo, std::ios::binary | std::ios::out | std::ios::trunc); 137 | if (isFileEncrypted) 138 | outStream.push(ChaCha20filter(FileEncryptKey, FileEncryptNonce)); 139 | if (isFileCompressed) 140 | outStream.push(boost::iostreams::lzma_decompressor()); 141 | outStream.push(outFile); 142 | 143 | boost::iostreams::copy(inFile, outStream); 144 | return true; 145 | } 146 | } 147 | } -------------------------------------------------------------------------------- /keeper/FilesTransformer.h: -------------------------------------------------------------------------------- 1 | #pragma once 2 | #include "TaskContext.h" 3 | #include "PathEncoder.h" 4 | #include "ChaChaFilter.h" 5 | 6 | namespace keeper 7 | { 8 | //class implements encrypting & decrypting of files & filenames 9 | class FilesTransformer 10 | { 11 | public: 12 | FilesTransformer() = delete; 13 | explicit FilesTransformer(const TaskContext& ctx); 14 | ~FilesTransformer(); 15 | std::wstring GetTransformedName(const std::wstring& name, bool isDirectory); 16 | std::wstring GetOriginalName(const std::wstring& name, bool isDirectory); 17 | bool TransformFile(const std::wstring& pathFrom, const std::wstring& pathTo/*, bool isMoveFile = false*/); 18 | bool RestoreFile(const std::wstring& pathFrom, const std::wstring& pathTo); 19 | private: 20 | const TaskContext& ctx_; 21 | bool isFileCompressed; 22 | bool isFileEncrypted; 23 | bool isFileNamesEncrypted; 24 | PathEncoder pathEncoder; 25 | byte FileEncryptKey[crypto_stream_chacha20_KEYBYTES]; 26 | byte FileEncryptNonce[crypto_stream_chacha20_NONCEBYTES]; 27 | }; 28 | } -------------------------------------------------------------------------------- /keeper/GlobalUtils.cpp: -------------------------------------------------------------------------------- 1 | #include "stdafx.h" 2 | #include 3 | #include 4 | #include "GlobalUtils.h" 5 | #include "ConsoleLogger.h" 6 | 7 | using namespace boost::posix_time; 8 | using namespace ConsoleLogger; 9 | 10 | namespace keeper 11 | { 12 | static const ptime myEpoch(boost::gregorian::date(1970, boost::gregorian::Jan, 1)); 13 | 14 | int64_t ConvertPtimeToMillisec(boost::posix_time::ptime time) 15 | { 16 | time_duration myTimeFromEpoch = time - myEpoch; 17 | int64_t myTimeAsInt = myTimeFromEpoch.total_milliseconds(); 18 | 19 | return myTimeAsInt; 20 | } 21 | 22 | boost::posix_time::ptime ConvertMillisecToPtime(int64_t millisec) 23 | { 24 | ptime result = myEpoch + boost::posix_time::milliseconds(millisec); 25 | 26 | return result; 27 | } 28 | 29 | std::string WstringToUTF8(const std::wstring& str) 30 | { 31 | using namespace std; 32 | wstring_convert> utf8_conv; 33 | string resultUTF8(utf8_conv.to_bytes(str)); 34 | 35 | return resultUTF8; 36 | } 37 | 38 | std::wstring StringToWstring(const std::string && str) 39 | { 40 | std::wstring wresult(str.begin(), str.end()); 41 | return wresult; 42 | } 43 | 44 | std::wstring PTimeToWstringSafeSymbols(boost::posix_time::ptime time) 45 | { 46 | boost::gregorian::date dt = time.date(); 47 | auto timePart = time.time_of_day(); 48 | std::wostringstream stm; 49 | const auto w2 = std::setw(2); 50 | stm << std::setfill(L'0') << std::setw(4) << dt.year() << L'-' << w2 << static_cast(dt.month()) << L'-' << w2 << dt.day() << L'T' 51 | << w2 << timePart.hours() << L'-' << w2 << timePart.minutes() << L'-' << w2 << timePart.seconds() << L'.' << std::setw(3) << int(timePart.fractional_seconds()/1000); 52 | 53 | return stm.str(); 54 | } 55 | 56 | std::wstring PTimeToWstring(boost::posix_time::ptime time) 57 | { 58 | boost::gregorian::date dt = time.date(); 59 | auto timePart = time.time_of_day(); 60 | std::wostringstream stm; 61 | const auto w2 = std::setw(2); 62 | stm << std::setfill(L'0') << std::setw(4) << dt.year() << L'-' << w2 << static_cast(dt.month()) << L'-' << w2 << dt.day() << L'T' 63 | << w2 << timePart.hours() << L':' << w2 << timePart.minutes() << L':' << w2 << timePart.seconds() << L'.' << std::setw(3) << int(timePart.fractional_seconds() / 1000); 64 | 65 | return stm.str(); 66 | } 67 | 68 | std::wstring SystemTimeToWstring(const SYSTEMTIME & utc) 69 | { 70 | std::wostringstream stm; 71 | const auto w2 = std::setw(2); 72 | stm << std::setfill(L'0') << std::setw(4) << utc.wYear << L'-' << w2 << utc.wMonth << L'-' << w2 << utc.wDay << L'T' 73 | << w2 << utc.wHour << L':' << w2 << utc.wMinute << L':' << w2 << utc.wSecond << L'.' << std::setw(3) << utc.wMilliseconds; 74 | 75 | return stm.str(); 76 | } 77 | 78 | long match_duration(const std::string& input, const std::regex& re) 79 | { 80 | std::smatch match; 81 | std::regex_search(input, match, re); 82 | if (match.empty()) 83 | return 0; // Pattern don't match 84 | 85 | double durArgs[6] = { 0,0,0,0,0,0 }; // years, months, days, hours, minutes, seconds 86 | 87 | std::string matchStr; 88 | for (size_t i = 0; (i < match.size()) && (i < 6); i++) 89 | { 90 | if (match[i + 1].matched) 91 | { 92 | matchStr = match[i + 1]; 93 | matchStr.pop_back(); // remove last character. 94 | durArgs[i] = std::stod(matchStr); 95 | } 96 | } 97 | 98 | long duration = long( 99 | 60 * 60 * 24 * 365.25 * durArgs[0] + // years 100 | 60 * 60 * 24 * 31 * durArgs[1] + // months 101 | 60 * 60 * 24 * durArgs[2] + // days 102 | 60 * 60 * durArgs[3] + // hours 103 | 60 * durArgs[4] + // minutes 104 | durArgs[5]); // seconds 105 | 106 | return duration; 107 | } 108 | 109 | boost::posix_time::time_duration StringToDuration(const std::string & str) 110 | { 111 | time_duration result = not_a_date_time; 112 | 113 | if (str.empty()) 114 | return result; 115 | 116 | long secs; 117 | std::regex rshort("^((?!T).)*$"); 118 | if (std::regex_match(str, rshort)) // no T (Time) exist 119 | { 120 | std::regex r("P([[:d:]]+Y)?([[:d:]]+M)?([[:d:]]+D)?"); 121 | secs = match_duration(str, r); 122 | } 123 | else 124 | { 125 | std::regex r("P([[:d:]]+Y)?([[:d:]]+M)?([[:d:]]+D)?T([[:d:]]+H)?([[:d:]]+M)?([[:d:]]+S|[[:d:]]+\\.[[:d:]]+S)?"); 126 | secs = match_duration(str, r); 127 | } 128 | 129 | if (secs <= 0) 130 | return result; 131 | else 132 | result = seconds(secs); 133 | 134 | return result; 135 | } 136 | 137 | std::wstring DbtToWstring(const Dbt & dbt) 138 | { 139 | return std::wstring(static_cast(dbt.get_data()), dbt.get_size() / sizeof(wchar_t)); 140 | } 141 | 142 | Dbt WstringToDbt(const std::wstring & str) //is it safe to pass??? 143 | { 144 | Dbt dbt((void*)str.c_str(), int(sizeof(wchar_t)*str.length())); //dirty hack 145 | dbt.set_flags(DB_DBT_READONLY); 146 | return dbt; 147 | } 148 | 149 | void keeper::CheckDbResult(int result) 150 | { 151 | if ((result == 0) || (result == DB_NOTFOUND)) 152 | return; 153 | LOG_FATAL() << "DB error: " << result << std::endl; 154 | throw std::exception(); 155 | } 156 | 157 | std::string keeper::PasswordToKey(const std::string & password) 158 | { 159 | byte key[crypto_box_SEEDBYTES]; //32 160 | byte salt[crypto_pwhash_SALTBYTES] = { 0xBE,0x28,0x74,0x25,0x7E,0xDA,0x9F,0x3C,0x1C,0x19,0xBE,0x38,0x0A,0x3A,0x2C,0x7B }; //16 161 | 162 | if (crypto_pwhash(key, crypto_box_SEEDBYTES, password.c_str(), password.length(), salt, 163 | crypto_pwhash_OPSLIMIT_INTERACTIVE, crypto_pwhash_MEMLIMIT_INTERACTIVE, 164 | crypto_pwhash_ALG_DEFAULT) != 0) 165 | { 166 | LOG_FATAL() << "Can't derive key from password" << std::endl; 167 | throw; 168 | } 169 | std::ostringstream s; 170 | //s.fill('0'); 171 | s << std::setfill('0') << std::hex; 172 | //s.width(2); 173 | for (int i = 0; i < crypto_box_SEEDBYTES; i++) 174 | s << std::setw(2) << int(key[i]); 175 | 176 | return s.str(); 177 | } 178 | 179 | std::string keeper::StrAnsiToOEM(const std::string& str) 180 | { 181 | std::unique_ptr buff = std::make_unique(str.length() + 1); 182 | CharToOemBuffA(str.c_str(), buff.get(), DWORD(str.length() + 1)); 183 | std::string result(buff.get()); 184 | return result; 185 | } 186 | } -------------------------------------------------------------------------------- /keeper/GlobalUtils.h: -------------------------------------------------------------------------------- 1 | #pragma once 2 | //#include 3 | 4 | namespace keeper 5 | { 6 | int64_t ConvertPtimeToMillisec(boost::posix_time::ptime time); 7 | boost::posix_time::ptime ConvertMillisecToPtime(int64_t millisec); 8 | std::string WstringToUTF8(const std::wstring& str); 9 | std::wstring StringToWstring(const std::string&& str); 10 | 11 | //manual implenemtation of "to_iso_wstring" to avoid different locales 12 | std::wstring PTimeToWstringSafeSymbols(boost::posix_time::ptime time); 13 | std::wstring PTimeToWstring(boost::posix_time::ptime time); 14 | std::wstring SystemTimeToWstring(const SYSTEMTIME& utc); 15 | boost::posix_time::time_duration StringToDuration(const std::string& str); 16 | 17 | std::wstring DbtToWstring(const Dbt& dbt); 18 | Dbt WstringToDbt(const std::wstring& str); 19 | 20 | void CheckDbResult(int result); 21 | 22 | std::string PasswordToKey(const std::string& password); 23 | 24 | std::string StrAnsiToOEM(const std::string& str); 25 | } -------------------------------------------------------------------------------- /keeper/PathEncoder.cpp: -------------------------------------------------------------------------------- 1 | #include "stdafx.h" 2 | #include "PathEncoder.h" 3 | #include 4 | #include 5 | #include "basen.hpp" 6 | #include "fpaq.h" 7 | 8 | using namespace std; 9 | using namespace keeper; 10 | 11 | wstring PathEncoder::BinToString(byte* buf, size_t length) 12 | { 13 | wstring outputString; 14 | outputString.reserve(size_t(length * 1.7)); //160% data grown for base32 15 | bn::encode_b32(buf, buf + length, back_inserter(outputString)); 16 | return outputString; 17 | } 18 | 19 | size_t PathEncoder::StringToBin(const std::wstring & s, byte * buf) 20 | { 21 | //decode_b32 don't work with unicode, so, convert it to OEM 22 | int length = ::WideCharToMultiByte(CP_OEMCP, 0, s.c_str(), s.size(), NULL, 0, NULL, NULL); 23 | if (length <= 0) 24 | return 0; 25 | std::vector buffer(length); 26 | ::WideCharToMultiByte(CP_OEMCP, 0, s.c_str(), s.size(), buffer.data(), length, NULL, NULL); 27 | std::string narrowedStr(buffer.begin(), buffer.end()); 28 | 29 | assert(s.length() == narrowedStr.length()); 30 | 31 | std::vector tmpBuf; 32 | tmpBuf.reserve(size_t(length / 1.5)); //160% data grown for base32 33 | bn::decode_b32(narrowedStr.c_str(), narrowedStr.c_str() + narrowedStr.size(), back_inserter(tmpBuf)); 34 | memcpy(buf, tmpBuf.data(), tmpBuf.size()); 35 | return tmpBuf.size(); 36 | } 37 | 38 | //PathEncoder::PathEncoder(const byte * key, const byte * iv) 39 | PathEncoder::PathEncoder(const TaskContext& ctx) 40 | { 41 | //fixme: possibly OutOfRange error with very long filename 42 | compressBuffer_ = make_unique(compressBufferSize_); 43 | bocu1Buffer_ = make_unique(compressBufferSize_); 44 | charsBuffer_ = make_unique(MAX_PATH_LENGTH * sizeof(wchar_t)); 45 | 46 | memcpy(key_, ctx.NamesEncodeKey_, crypto_stream_chacha20_KEYBYTES); 47 | memcpy(nonce_, ctx.NamesEncodeNonce_, crypto_stream_chacha20_NONCEBYTES); 48 | } 49 | 50 | PathEncoder::~PathEncoder() 51 | { 52 | sodium_memzero(key_, crypto_stream_chacha20_KEYBYTES); 53 | sodium_memzero(nonce_, crypto_stream_chacha20_NONCEBYTES); 54 | } 55 | 56 | std::wstring PathEncoder::GetCachedName(const std::wstring & name) 57 | { 58 | for (auto it = NameCache.begin(); it != NameCache.end(); it++) 59 | { 60 | if (get<0>(*it) == name) 61 | { 62 | //move cache match to the top of cache 63 | NameCache.emplace_front(move(*it)); 64 | NameCache.erase(it); 65 | return get<1>(NameCache.front()); 66 | } 67 | } 68 | return std::wstring(); 69 | } 70 | 71 | void PathEncoder::AddNameToCache(const std::wstring & cachedKey, const std::wstring & cachedName) 72 | { 73 | NameCache.push_front(CacheDictPair(cachedKey, cachedName)); 74 | if (NameCache.size() > NameCacheSize) 75 | NameCache.resize(NameCacheSize); 76 | } 77 | 78 | std::wstring PathEncoder::encode(const std::wstring & path) 79 | { 80 | vector fileNames, fileNamesEncoded; 81 | 82 | boost::split(fileNames, path, [](wchar_t ch) -> bool { return ch == L'\\'; }); 83 | if (fileNames.size() > NameCacheSize) 84 | NameCacheSize = fileNames.size(); 85 | std::wstring cachedName; 86 | for (const auto& fileName : fileNames) 87 | { 88 | cachedName = GetCachedName(fileName); 89 | if (cachedName.empty()) 90 | { 91 | //compressing UCS-2 string 92 | //Unicode -> BOCU-1 93 | unsigned int lengthCompressed = BOCU1::writeString(fileName.c_str(), fileName.length(), bocu1Buffer_.get()); 94 | //BOCU-1 -> fpaq compressing 95 | lengthCompressed = fpaq::fpaqCompress(bocu1Buffer_.get(), lengthCompressed, compressBuffer_.get()); 96 | //encrypt 97 | crypto_stream_chacha20_xor_ic(compressBuffer_.get(), compressBuffer_.get(), lengthCompressed, nonce_, 0, key_); 98 | //convert to BASE32 string 99 | cachedName = BinToString(compressBuffer_.get(), lengthCompressed); 100 | AddNameToCache(fileName, cachedName); 101 | } 102 | fileNamesEncoded.push_back(cachedName); 103 | } 104 | 105 | return boost::join(fileNamesEncoded, L"\\"); 106 | } 107 | 108 | std::wstring PathEncoder::decode(const std::wstring & path) 109 | { 110 | vector fileNamesEncoded, fileNamesDecoded; 111 | 112 | boost::split(fileNamesEncoded, path, [](wchar_t ch) -> bool { return ch == L'\\'; }); 113 | if (fileNamesEncoded.size() > NameCacheSize) 114 | NameCacheSize = fileNamesEncoded.size(); 115 | std::wstring cachedName; 116 | for (const auto& fileName : fileNamesEncoded) 117 | { 118 | cachedName = GetCachedName(fileName); 119 | if (cachedName.empty()) 120 | { 121 | //BASE32 string to binary 122 | auto lengthBin = StringToBin(fileName, compressBuffer_.get()); 123 | //decrypt 124 | crypto_stream_chacha20_xor_ic(compressBuffer_.get(), compressBuffer_.get(), lengthBin, nonce_, 0, key_); 125 | //decompress 126 | unsigned int lengthDecompressed = fpaq::fpaqDecompress(compressBuffer_.get(), lengthBin, bocu1Buffer_.get()); 127 | lengthDecompressed = BOCU1::readString(bocu1Buffer_.get(), lengthDecompressed, charsBuffer_.get()); 128 | //convert to string 129 | assert((lengthDecompressed & 1) == 0); 130 | cachedName = std::wstring(charsBuffer_.get(), lengthDecompressed); 131 | AddNameToCache(fileName, cachedName); 132 | } 133 | fileNamesDecoded.push_back(cachedName); 134 | } 135 | return boost::join(fileNamesDecoded, L"\\"); 136 | } 137 | -------------------------------------------------------------------------------- /keeper/PathEncoder.h: -------------------------------------------------------------------------------- 1 | #pragma once 2 | #include 3 | #include 4 | #include "TaskContext.h" 5 | #include "bocu1.h" 6 | 7 | typedef std::tuple CacheDictPair; 8 | 9 | class PathEncoder 10 | { 11 | public: 12 | PathEncoder() = delete; 13 | PathEncoder(const PathEncoder&) = delete; 14 | //PathEncoder(const byte* key, const byte* iv); 15 | PathEncoder(const keeper::TaskContext& ctx); 16 | ~PathEncoder(); 17 | std::wstring encode(const std::wstring& path); 18 | std::wstring decode(const std::wstring& path); 19 | private: 20 | std::wstring BinToString(byte* buf, size_t length); 21 | size_t StringToBin(const std::wstring& s, byte* buf); 22 | 23 | static constexpr size_t compressBufferSize_ = MAX_PATH_LENGTH * sizeof(wchar_t) * 2; //k=2 just to make sure 24 | std::unique_ptr compressBuffer_; 25 | std::unique_ptr bocu1Buffer_; 26 | std::unique_ptr charsBuffer_; 27 | 28 | byte key_[crypto_stream_chacha20_KEYBYTES]; 29 | byte nonce_[crypto_stream_chacha20_NONCEBYTES]; 30 | 31 | //encoded names cache 32 | size_t NameCacheSize = 8; 33 | //ATTENTION: same cache for encoding and decoding 34 | std::list NameCache; 35 | std::wstring GetCachedName(const std::wstring& name); 36 | void AddNameToCache(const std::wstring& cachedKey, const std::wstring& cachedName); 37 | }; -------------------------------------------------------------------------------- /keeper/PathEncoderTest.cpp: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sapozhnikov/keeper/b83c5c3bb4852ca4a52311854076a2015b112a13/keeper/PathEncoderTest.cpp -------------------------------------------------------------------------------- /keeper/TaskBackup.cpp: -------------------------------------------------------------------------------- 1 | #include "stdafx.h" 2 | #include 3 | #include "GlobalUtils.h" 4 | #include "TaskBackup.h" 5 | #include 6 | #include "ConsoleLogger.h" 7 | #include "FileIO.h" 8 | #include "FilesTransformer.h" 9 | 10 | using namespace std; 11 | using namespace boost::filesystem; 12 | using namespace ConsoleLogger; 13 | 14 | TaskBackup::TaskBackup(keeper::TaskContext& ctx) : 15 | ctx_(ctx), 16 | dirIterator_(ctx_.GetSourceDirectory()), 17 | //initialDirectory_(ctx_.GetSourceDirectory()), 18 | skipPathCharsCount_((unsigned int)ctx_.GetSourceDirectory().length()) 19 | { 20 | currentTimeStamp_ = boost::posix_time::microsec_clock::local_time(); 21 | timestamp64_ = keeper::ConvertPtimeToMillisec(currentTimeStamp_); 22 | } 23 | 24 | TaskBackup::~TaskBackup() 25 | { 26 | } 27 | 28 | void TaskBackup::Run() 29 | { 30 | boost::system::error_code errorCode; 31 | //bool isFirstBackup = false; 32 | 33 | bool isFirstBackup = false; 34 | if (!ctx_.OpenDatabase()) 35 | { 36 | LOG_INFO() << "Creating new repository" << std::endl; 37 | isFirstBackup = true; 38 | ctx_.CreateDatabase(); 39 | } 40 | 41 | mirrorPath_ = ctx_.GetDestinationDirectory() + MIRROR_SUB_DIR; 42 | if (!boost::filesystem::exists(mirrorPath_)) 43 | { 44 | //boost::filesystem::create_directory(mirrorPath_); 45 | if (!keeper::FileIO::CreateDir(mirrorPath_)) 46 | { 47 | LOG_FATAL() << "Can't create directory " << mirrorPath_ << std::endl; 48 | return; 49 | } 50 | } 51 | 52 | storeOldPath_ = ctx_.GetDestinationDirectory() + keeper::PTimeToWstringSafeSymbols(currentTimeStamp_) + L"\\"; 53 | 54 | //ctx_.GetMainDB().cursor(nullptr, &mainCursor_, 0); 55 | 56 | keeper::FilesTransformer transformer(ctx_); 57 | keeper::WildCardNameChecker& namesChecker = ctx_.NamesChecker; 58 | bool NamesFilteringEnabled = namesChecker.IsFilteringEnabled; 59 | 60 | //start copy process 61 | boost::system::error_code errCode; 62 | while (true) 63 | { 64 | if (dirIterator_ == recursive_directory_iterator()) 65 | break; 66 | const wstring& sourceFullPath = dirIterator_->path().wstring(); 67 | relativePath_ = sourceFullPath.substr(skipPathCharsCount_, skipPathCharsCount_ - sourceFullPath.length()); 68 | 69 | bool isDirectory = is_directory(dirIterator_->path(), errCode); 70 | if (errCode.value() != boost::system::errc::success) 71 | { 72 | LOG_ERROR() << "Can't get info about " << sourceFullPath << std::endl; 73 | ++dirIterator_; 74 | continue; 75 | } 76 | 77 | if (NamesFilteringEnabled) 78 | { 79 | if (!namesChecker.IsFitPattern(relativePath_)) 80 | { 81 | if (isDirectory) 82 | dirIterator_.no_push(); 83 | ++dirIterator_; 84 | continue; 85 | } 86 | } 87 | 88 | //skip symlinks 89 | bool isSymlink = is_symlink(dirIterator_->path(), errCode); 90 | if (errCode.value() != boost::system::errc::success) 91 | { 92 | LOG_ERROR() << "Can't get info about " << sourceFullPath << std::endl; 93 | ++dirIterator_; 94 | continue; 95 | } 96 | 97 | if (isSymlink) 98 | { 99 | LOG_INFO() << "Skipping symlink \"" << dirIterator_->path().wstring() << "\"" << endl; 100 | if (isDirectory) 101 | dirIterator_.no_push(); 102 | ++dirIterator_; 103 | continue; 104 | } 105 | 106 | transformedPath_ = transformer.GetTransformedName(relativePath_, isDirectory); 107 | 108 | if (isDirectory) 109 | { 110 | //directory handling 111 | if (isFirstBackup) 112 | { 113 | if (keeper::FileIO::CopyDir(sourceFullPath, mirrorPath_ + transformedPath_)) 114 | DbAddEvent(sourceFullPath, relativePath_, FileEventType::Added); 115 | } 116 | else 117 | { 118 | //do not move existing folder to storage 119 | 120 | //check if exists 121 | if (!boost::filesystem::exists(mirrorPath_ + transformedPath_, errorCode)) //or DB check? 122 | { 123 | //copy to mirror 124 | if (keeper::FileIO::CopyDir(sourceFullPath, mirrorPath_ + transformedPath_)) 125 | DbAddEvent(sourceFullPath, relativePath_, FileEventType::Changed); 126 | } 127 | } 128 | } 129 | else 130 | { 131 | //file handling 132 | if (isFirstBackup) 133 | { 134 | //just copy the file 135 | if (transformer.TransformFile(sourceFullPath, mirrorPath_ + transformedPath_)) 136 | DbAddEvent(sourceFullPath, relativePath_, FileEventType::Added); 137 | } 138 | else 139 | { 140 | //compare file attrs with DB record 141 | TaskBackup::FileState fileState = GetFileState(); 142 | switch (fileState) 143 | { 144 | case FileState::New: 145 | //just copy the file 146 | if (transformer.TransformFile(sourceFullPath, mirrorPath_ + transformedPath_)) 147 | DbAddEvent(sourceFullPath, relativePath_, FileEventType::Added); 148 | break; 149 | 150 | case FileState::Same: 151 | //do nothing 152 | break; 153 | 154 | case FileState::Changed: 155 | //move to storage 156 | if (keeper::FileIO::MoveSingleFile(mirrorPath_ + transformedPath_, storeOldPath_ + transformedPath_)) 157 | { 158 | //copy new version 159 | if (transformer.TransformFile(sourceFullPath, mirrorPath_ + transformedPath_)) 160 | DbAddEvent(sourceFullPath, relativePath_, FileEventType::Changed); 161 | } 162 | break; 163 | default: 164 | break; 165 | } 166 | } 167 | } 168 | dirIterator_.increment(errCode); 169 | while(errCode) 170 | { 171 | if (isDirectory) //failed to enter directory 172 | { 173 | LOG_ERROR() << "No access to " << dirIterator_->path().wstring() << std::endl; 174 | dirIterator_.no_push(); 175 | dirIterator_.increment(errCode); 176 | } 177 | else 178 | throw std::runtime_error(errCode.message()); 179 | 180 | if (dirIterator_ == recursive_directory_iterator()) 181 | break; 182 | } 183 | } 184 | 185 | //search for deleted files 186 | if (!isFirstBackup) 187 | { 188 | Dbc* mainCursor = nullptr; 189 | ctx_.GetMainDB().cursor(nullptr, &mainCursor, 0); 190 | BOOST_SCOPE_EXIT_ALL(&) 191 | { 192 | if (mainCursor != nullptr) 193 | mainCursor->close(); 194 | }; 195 | 196 | Dbt keyMain, dataMain; 197 | // 198 | vector> filesToMove; 199 | 200 | int result = mainCursor->get(&keyMain, &dataMain, DB_FIRST); 201 | keeper::CheckDbResult(result); 202 | 203 | while (true) 204 | { 205 | if (result == DB_NOTFOUND) //end of DB 206 | break; 207 | 208 | if (result == 0) 209 | { 210 | //move to the last record with the same key 211 | while (true) 212 | { 213 | result = mainCursor->get(&keyMain, &dataMain, DB_NEXT_DUP); 214 | keeper::CheckDbResult(result); 215 | if (result == DB_NOTFOUND) 216 | break; 217 | } 218 | DbFileEvent* pEvent = static_cast(dataMain.get_data()); 219 | 220 | if ((static_cast(pEvent->FileEvent) != FileEventType::Deleted) && 221 | (pEvent->EventTimeStamp != timestamp64_)) 222 | { 223 | relativePath_ = keeper::DbtToWstring(keyMain); 224 | //check if file exists 225 | if (!boost::filesystem::exists(ctx_.GetSourceDirectory() + relativePath_)) 226 | { 227 | bool isDirectory = (pEvent->FileAttributes.dwFileAttributes & FILE_ATTRIBUTE_DIRECTORY) != 0; 228 | //if add DB record now, the cursor will skip some records, so keep names for later 229 | filesToMove.push_back(tuple(isDirectory, relativePath_)); 230 | } 231 | } 232 | } 233 | result = mainCursor->get(&keyMain, &dataMain, DB_NEXT_NODUP); 234 | keeper::CheckDbResult(result); 235 | }//while 236 | 237 | //the easyest way to move directories with subdirs and files in right order is sort them 238 | sort(filesToMove.begin(), filesToMove.end(), 239 | [](const auto& t1, const auto& t2){ 240 | const wstring& path1 = std::get(t1); 241 | const wstring& path2 = std::get(t2); 242 | if (path1.length() == path2.length()) 243 | return std::get(t1) < std::get(t2); 244 | else 245 | return path1.length() < path2.length(); 246 | }); 247 | 248 | for (const auto& fileInfo : filesToMove) 249 | { 250 | //move from mirror folder to delta folder 251 | transformedPath_ = transformer.GetTransformedName(std::get(fileInfo), std::get(fileInfo)); 252 | auto deletedFile = mirrorPath_ + transformedPath_; 253 | if (boost::filesystem::exists(deletedFile)) 254 | { 255 | keeper::FileIO::MoveSingleFile(deletedFile, storeOldPath_ + transformedPath_); 256 | } 257 | 258 | //write event to DB 259 | DbAddEvent(std::wstring(), std::get(fileInfo), FileEventType::Deleted); 260 | } 261 | } //if (!isFirstBackup) 262 | } 263 | 264 | void TaskBackup::DbAddEvent(const std::wstring& fullPath, const std::wstring& relativePath, FileEventType ev) 265 | { 266 | //relative path as the key 267 | Dbt key = keeper::WstringToDbt(relativePath); 268 | 269 | //properties as the value 270 | DbFileEvent fileDbEvent; 271 | 272 | if (ev != FileEventType::Deleted) 273 | { 274 | if (!GetFileAttributesEx(fullPath.c_str(), GetFileExInfoStandard, &fileDbEvent.FileAttributes)) 275 | throw std::exception("Can't get file attributes"); 276 | } 277 | else 278 | { 279 | fileDbEvent.FileAttributes = { 0 }; 280 | } 281 | 282 | fileDbEvent.EventTimeStamp = timestamp64_; 283 | fileDbEvent.FileEvent = static_cast(ev); 284 | 285 | Dbt data((void*)&fileDbEvent, sizeof(DbFileEvent)); 286 | 287 | ctx_.GetMainDB().put(nullptr, &key, &data, 0); 288 | } 289 | 290 | TaskBackup::FileState TaskBackup::GetFileState() 291 | { 292 | //check if exists in mirror folder, ignore DB, files are more important 293 | if (!boost::filesystem::exists(mirrorPath_ + transformedPath_)) 294 | return TaskBackup::FileState::New; 295 | 296 | //compare attributes from source file with DB record 297 | 298 | //get the last event 299 | Dbt key = keeper::WstringToDbt(relativePath_); 300 | Dbt data; 301 | Dbc* mainCursor = nullptr; 302 | BOOST_SCOPE_EXIT_ALL(&) 303 | { 304 | if (mainCursor != nullptr) 305 | mainCursor->close(); 306 | }; 307 | 308 | ctx_.GetMainDB().cursor(nullptr, &mainCursor, 0); 309 | int result = mainCursor->get(&key, &data, DB_SET); 310 | keeper::CheckDbResult(result); 311 | if (result == DB_NOTFOUND) 312 | return TaskBackup::FileState::New; //something wrong 313 | 314 | while (true) 315 | { 316 | result = mainCursor->get(&key, &data, DB_NEXT_DUP); 317 | keeper::CheckDbResult(result); 318 | if (result == DB_NOTFOUND) 319 | break; 320 | } 321 | DbFileEvent* pEvent = static_cast(data.get_data()); //TODO: data size check 322 | 323 | if (static_cast(pEvent->FileEvent) == FileEventType::Deleted) 324 | return TaskBackup::FileState::New; 325 | 326 | if (pEvent->EventTimeStamp == timestamp64_) 327 | return TaskBackup::FileState::Same; //something wrong 328 | 329 | const wstring& sourceFullPath = dirIterator_->path().wstring(); 330 | 331 | WIN32_FILE_ATTRIBUTE_DATA sourceAttributes; 332 | 333 | if (!GetFileAttributesEx(sourceFullPath.c_str(), GetFileExInfoStandard, &sourceAttributes)) 334 | throw std::exception("Can't get file attributes"); 335 | 336 | if ((sourceAttributes.nFileSizeHigh == pEvent->FileAttributes.nFileSizeHigh) && 337 | (sourceAttributes.nFileSizeLow == pEvent->FileAttributes.nFileSizeLow) && 338 | (sourceAttributes.ftLastWriteTime.dwHighDateTime == pEvent->FileAttributes.ftLastWriteTime.dwHighDateTime) && 339 | (sourceAttributes.ftLastWriteTime.dwLowDateTime == pEvent->FileAttributes.ftLastWriteTime.dwLowDateTime)) 340 | return TaskBackup::FileState::Same; 341 | else 342 | return TaskBackup::FileState::Changed; 343 | } 344 | -------------------------------------------------------------------------------- /keeper/TaskBackup.h: -------------------------------------------------------------------------------- 1 | #pragma once 2 | #include 3 | #include "TaskContext.h" 4 | 5 | class TaskBackup final 6 | { 7 | public: 8 | TaskBackup(keeper::TaskContext& ctx); 9 | ~TaskBackup(); 10 | 11 | void Run(); 12 | 13 | private: 14 | enum class FileState 15 | { 16 | Error, 17 | New, 18 | Same, 19 | Changed, 20 | Deleted 21 | }; 22 | 23 | keeper::TaskContext& ctx_; 24 | boost::filesystem::recursive_directory_iterator dirIterator_; 25 | unsigned int skipPathCharsCount_ = 0; 26 | std::wstring mirrorPath_; 27 | std::wstring relativePath_; 28 | std::wstring transformedPath_; 29 | 30 | std::wstring storeOldPath_; 31 | 32 | void DbAddEvent(const std::wstring& fullPath, const std::wstring& relativePath, FileEventType ev); 33 | 34 | boost::posix_time::ptime currentTimeStamp_; 35 | int64_t timestamp64_; 36 | 37 | FileState GetFileState(); 38 | }; 39 | 40 | -------------------------------------------------------------------------------- /keeper/TaskContext.cpp: -------------------------------------------------------------------------------- 1 | #include "stdafx.h" 2 | #include "TaskContext.h" 3 | #include "GlobalUtils.h" 4 | #include "ConsoleLogger.h" 5 | #include 6 | 7 | using namespace ConsoleLogger; 8 | extern DbEnv* BuroEnv; 9 | 10 | namespace keeper 11 | { 12 | static char* PARAM_COMPRESSION = "COMPRESSION RATIO"; 13 | static char* PARAM_FILES_ENCODE_KEY = "FILES ENCODE KEY"; 14 | static char* PARAM_NAMES_ENCODE = "NAMES ENCODE"; 15 | static char* PARAM_NAMES_ENCODE_KEY = "NAMES ENCODE KEY"; 16 | static char* PARAM_NAMES_ENCODE_NONCE = "NAMES ENCODE NONCE"; 17 | 18 | void NormalizePath(std::wstring& path) 19 | { 20 | //make path extended length by default 21 | //TODO: skip it for Win10 and newer 22 | if ((path[0] != L'\\') || 23 | (path[1] != L'\\') || 24 | (path[2] != L'?') || 25 | (path[3] != L'\\')) 26 | path = L"\\\\?\\" + path; 27 | 28 | if (path[path.length() - 1] != L'\\') 29 | path.append(L"\\"); 30 | } 31 | 32 | keeper::TaskContext::TaskContext() 33 | { 34 | //envDirectory_ = std::wstring(); 35 | } 36 | 37 | keeper::TaskContext::~TaskContext() 38 | { 39 | sodium_memzero(FileEncodeKey_, crypto_stream_chacha20_KEYBYTES); 40 | sodium_memzero(NamesEncodeKey_, crypto_stream_chacha20_KEYBYTES); 41 | sodium_memzero(NamesEncodeNonce_, crypto_stream_chacha20_NONCEBYTES); 42 | 43 | if (eventsDb_) 44 | eventsDb_->close(0); 45 | if (configDb_) 46 | configDb_->close(0); 47 | if (env_) 48 | env_->close(0); 49 | 50 | if (eventsDb_) 51 | delete(eventsDb_); 52 | if (configDb_) 53 | delete(configDb_); 54 | if (env_) 55 | delete(env_); 56 | BuroEnv = nullptr; 57 | } 58 | 59 | bool TaskContext::SetSourceDirectory(std::wstring SourceDirectory) 60 | { 61 | sourceDirectory_ = SourceDirectory; 62 | NormalizePath(sourceDirectory_); 63 | 64 | return true; 65 | } 66 | 67 | bool TaskContext::SetDestinationDirectory(std::wstring DestinationDirectory) 68 | { 69 | destinationDirectory_ = DestinationDirectory; 70 | NormalizePath(destinationDirectory_); 71 | 72 | return true; 73 | } 74 | 75 | const std::wstring& TaskContext::GetSourceDirectory() const 76 | { 77 | return sourceDirectory_; 78 | } 79 | 80 | const std::wstring& TaskContext::GetDestinationDirectory() const 81 | { 82 | return destinationDirectory_; 83 | } 84 | 85 | bool TaskContext::SetRestoreTimeStamp(std::string timestamp) 86 | { 87 | if (timestamp == std::string()) 88 | { 89 | restoreTimeStamp_ = boost::posix_time::not_a_date_time; 90 | return true; 91 | } 92 | 93 | try 94 | { 95 | restoreTimeStamp_ = boost::posix_time::time_from_string(timestamp); 96 | } 97 | catch (...) 98 | { 99 | restoreTimeStamp_ = boost::posix_time::not_a_date_time; 100 | return false; 101 | } 102 | 103 | return true; 104 | } 105 | 106 | bool TaskContext::SetPurgeTimeStampFromDate(std::string timestamp) 107 | { 108 | if (timestamp == std::string()) 109 | { 110 | purgeTimeStamp_ = boost::posix_time::not_a_date_time; 111 | return true; 112 | } 113 | 114 | try 115 | { 116 | purgeTimeStamp_ = boost::posix_time::time_from_string(timestamp); 117 | } 118 | catch (...) 119 | { 120 | purgeTimeStamp_ = boost::posix_time::not_a_date_time; 121 | return false; 122 | } 123 | 124 | return true; 125 | } 126 | 127 | bool TaskContext::SetPurgeTimeStampFromDuration(std::string timestamp) 128 | { 129 | auto duration = StringToDuration(timestamp); 130 | if (duration == boost::posix_time::not_a_date_time) 131 | return false; 132 | auto currentTimeStamp = boost::posix_time::microsec_clock::local_time(); 133 | purgeTimeStamp_ = currentTimeStamp - duration; 134 | 135 | return true; 136 | } 137 | 138 | bool TaskContext::SetPurgeTimestampFromArg(std::string timestamp) 139 | { 140 | if (timestamp[0] == L'P') 141 | return SetPurgeTimeStampFromDuration(timestamp); 142 | else 143 | return SetPurgeTimeStampFromDate(timestamp); 144 | } 145 | 146 | const boost::posix_time::ptime & TaskContext::GetPurgeTimeStamp() const 147 | { 148 | return purgeTimeStamp_; 149 | } 150 | 151 | const boost::posix_time::ptime & TaskContext::GetRestoreTimeStamp() const 152 | { 153 | return restoreTimeStamp_; 154 | } 155 | 156 | //generate DB path from args according task type 157 | std::wstring TaskContext::GenerateDbPath() 158 | { 159 | switch (Task) 160 | { 161 | case keeper::Task::Backup: 162 | return GetDestinationDirectory() + MAIN_DB_FILE; 163 | case keeper::Task::Purge: 164 | case keeper::Task::Restore: 165 | case keeper::Task::DumpDB: 166 | return GetSourceDirectory() + MAIN_DB_FILE; 167 | default: 168 | return std::wstring(); 169 | } 170 | } 171 | 172 | std::wstring keeper::TaskContext::GenerateEnvPath() 173 | { 174 | switch (Task) 175 | { 176 | case keeper::Task::Backup: 177 | return GetDestinationDirectory() + ENV_SUB_DIR; 178 | case keeper::Task::Purge: 179 | case keeper::Task::Restore: 180 | case keeper::Task::DumpDB: 181 | return GetSourceDirectory() + ENV_SUB_DIR; 182 | default: 183 | return std::wstring(); 184 | } 185 | } 186 | 187 | void EnvErrHandler(const DbEnv *, const char *errpfx, const char *errstr) 188 | { 189 | const unsigned int ERR_STRING_MAX = 1024; 190 | wchar_t wstr[ERR_STRING_MAX]; 191 | memset(wstr, 0, sizeof(wstr)); 192 | MultiByteToWideChar(CP_ACP, 0, errstr, strlen(errstr), 193 | wstr, ERR_STRING_MAX - 1); 194 | LOG_ERROR() << errpfx << wstr << std::endl; 195 | } 196 | 197 | void TaskContext::ConfigureEnv() 198 | { 199 | if (!env_) 200 | BuroEnv = env_ = new DbEnv(0u); 201 | 202 | env_->set_cachesize(0, ENV_CACHE_SIZE, 1); 203 | env_->set_lg_bsize(LOG_BUF_SIZE); 204 | env_->set_lg_max(LOG_FILE_SIZE); 205 | env_->log_set_config(DB_LOG_AUTO_REMOVE | DB_LOG_DIRECT, 1); 206 | if (!DbPassword.empty()) 207 | { 208 | if (DbKey.empty()) 209 | DbKey = PasswordToKey(DbPassword); 210 | 211 | env_->set_encrypt(DbKey.c_str(), DB_ENCRYPT_AES); 212 | } 213 | env_->open(WstringToUTF8(GenerateEnvPath()).c_str(), DB_CREATE | DB_RECOVER | DB_INIT_LOG | DB_INIT_MPOOL | DB_INIT_TXN | (!DbKey.empty() ? DB_ENCRYPT : 0), 0); 214 | env_->set_errpfx("ENVIRONMENT:"); 215 | env_->set_errcall(EnvErrHandler); 216 | } 217 | 218 | void keeper::TaskContext::DisplayTaskConfig() const 219 | { 220 | LOG_VERBOSE() << "Compression Level = " << CompressionLevel << std::endl; 221 | LOG_VERBOSE() << "Encrypted = " << (!DbPassword.empty() ? "true" : "false") << std::endl; 222 | LOG_VERBOSE() << "File names encrypted = " << (isEncryptedFileNames ? "true" : "false") << std::endl; 223 | } 224 | 225 | //function to compare one file events. it makes them sorted by date. there is can't be two events with same timestamp. 226 | #pragma warning(suppress: 4100) 227 | int DbEventsCompare(DB *db_, const DBT *data1, const DBT *data2, size_t *locp) 228 | { 229 | if ((data1->size != sizeof(DbFileEvent)) || (data2->size != sizeof(DbFileEvent))) 230 | throw std::exception("DbEventsCompare: Corrupted database"); 231 | 232 | if (static_cast(data1->data)->EventTimeStamp >= static_cast(data2->data)->EventTimeStamp) 233 | return 1; 234 | else 235 | return -1; 236 | } 237 | 238 | //function to compare file names in the b-tree 239 | int DbPathsCompare(Db*, const Dbt *data1, const Dbt *data2, size_t *locp) 240 | { 241 | locp = nullptr; 242 | auto dataLen1 = data1->get_size(); 243 | auto dataLen2 = data2->get_size(); 244 | 245 | //check it containt's even count of bytes 246 | if ((dataLen1 & 1) != 0 || (dataLen2 & 1) != 0) 247 | throw std::exception("DbPathsCompare: Corrupted database"); 248 | dataLen1 /= sizeof(wchar_t); 249 | dataLen2 /= sizeof(wchar_t); 250 | wchar_t* path1 = static_cast(data1->get_data()); 251 | wchar_t* path2 = static_cast(data2->get_data()); 252 | 253 | auto result = _wcsnicmp(path1, path2, min(dataLen1, dataLen2)); 254 | if (result == 0) 255 | { 256 | if (dataLen1 < dataLen2) 257 | return -1; 258 | if (dataLen1 > dataLen2) 259 | return 1; 260 | } 261 | return result; 262 | } 263 | #if (1) 264 | size_t DbPathsPrefix(DB*, const DBT *data1, const DBT *data2) 265 | { 266 | auto dataLen1 = data1->size; 267 | auto dataLen2 = data2->size; 268 | 269 | //check it containt's even count of bytes 270 | if ((dataLen1 & 1) != 0 || (dataLen2 & 1) != 0) 271 | throw std::exception("DbPathsPrefix: Corrupted database"); 272 | dataLen1 /= sizeof(wchar_t); 273 | dataLen2 /= sizeof(wchar_t); 274 | 275 | wchar_t* path1 = static_cast(data1->data); 276 | wchar_t* path2 = static_cast(data2->data); 277 | uint32_t idx; 278 | for (idx = 0; idx < min(dataLen1, dataLen2); idx++) 279 | { 280 | if (*path1 != *path2) 281 | return (idx + 1) * sizeof(wchar_t); 282 | path1++; 283 | path2++; 284 | } 285 | //return (min(dataLen1, dataLen2)/* + 1*/) * sizeof(wchar_t); 286 | if (dataLen1 != dataLen2) 287 | return (min(dataLen1, dataLen2) + 1) * sizeof(wchar_t); 288 | else 289 | return dataLen1 * sizeof(wchar_t); 290 | } 291 | #else 292 | //prefix function copied from DB reference 293 | size_t DbPathsPrefix(DB *dbp, const DBT *a, const DBT *b) 294 | { 295 | byte *p1, *p2; 296 | 297 | size_t cnt = 1; 298 | size_t len = a->size > b->size ? b->size : a->size; 299 | for (p1 = (byte*)a->data, p2 = (byte*)b->data; len--; ++p1, ++p2, ++cnt) 300 | if (*p1 != *p2) 301 | return (cnt); 302 | /* 303 | * They match up to the smaller of the two sizes. 304 | * Collate the longer after the shorter. 305 | */ 306 | if (a->size < b->size) 307 | return (a->size + 1); 308 | if (b->size < a->size) 309 | return (b->size + 1); 310 | return (b->size); 311 | } 312 | #endif 313 | 314 | bool TaskContext::OpenDatabase(bool CreateFreshDB) 315 | { 316 | using namespace std; 317 | wstring dbFullPath = GenerateDbPath(); 318 | //wstring envFullPath = GenerateEnvPath(); 319 | 320 | if (!boost::filesystem::exists(dbFullPath)) 321 | { 322 | //no DB exists 323 | if (!CreateFreshDB) 324 | return false; 325 | } 326 | string dbNameUTF8 = keeper::WstringToUTF8(dbFullPath); 327 | 328 | 329 | try 330 | { 331 | ConfigureEnv(); 332 | 333 | if (!eventsDb_) 334 | eventsDb_ = new Db(env_, 0); 335 | eventsDb_->set_flags((!DbPassword.empty() ? DB_ENCRYPT : 0) | DB_DUPSORT); 336 | eventsDb_->set_dup_compare(DbEventsCompare); 337 | eventsDb_->set_bt_compare(DbPathsCompare); 338 | //screw it, its buggy 339 | //eventsDb_.set_bt_prefix(DbPathsPrefix); 340 | 341 | if (!configDb_) 342 | configDb_ = new Db(env_, 0); 343 | configDb_->set_flags(!DbPassword.empty() ? DB_ENCRYPT : 0); 344 | 345 | //open existing 346 | eventsDb_->open(nullptr, 347 | dbNameUTF8.c_str(), 348 | EVENTS_DB_TABLE, 349 | DB_BTREE, 350 | (CreateFreshDB ? DB_CREATE : 0) | DB_AUTO_COMMIT, 351 | 0); 352 | 353 | configDb_->open(nullptr, 354 | dbNameUTF8.c_str(), 355 | CONFIG_DB_TABLE, 356 | DB_BTREE, 357 | (CreateFreshDB ? DB_CREATE : 0) | DB_AUTO_COMMIT, 358 | 0); 359 | 360 | eventsDb_->set_errpfx("EVENTS DB:"); 361 | eventsDb_->set_errcall(EnvErrHandler); 362 | configDb_->set_errpfx("CONFIG DB:"); 363 | configDb_->set_errcall(EnvErrHandler); 364 | } 365 | catch (DbException) 366 | { 367 | LOG_FATAL() << "Database open failed" << endl; 368 | throw; 369 | } 370 | 371 | if (!CreateFreshDB) 372 | { 373 | //load config from DB 374 | DWORD CompressionLevelSaved; 375 | if (!GetConfigValueDWord(PARAM_COMPRESSION, CompressionLevelSaved)) 376 | throw /*std::runtime_error(string("Can't load parameter: ") + PARAM_COMPRESSION)*/; 377 | if (CompressionLevelSaved != CompressionLevel) 378 | { 379 | if ((CompressionLevelSaved != 0) && (CompressionLevel == 0)) 380 | { 381 | LOG_WARNING() << "Compression used in this repository" << endl; 382 | CompressionLevel = CompressionLevelSaved; 383 | } 384 | 385 | if ((CompressionLevelSaved == 0) && (CompressionLevel != 0)) 386 | { 387 | LOG_WARNING() << "Compression not used in this repository" << endl; 388 | CompressionLevel = CompressionLevelSaved; 389 | } 390 | 391 | if ((CompressionLevelSaved != 0) && (CompressionLevel != 0)) 392 | { 393 | LOG_INFO() << "Compression ratio changed to " << CompressionLevel << endl; 394 | SetConfigValueDWord(PARAM_COMPRESSION, CompressionLevel); 395 | } 396 | } 397 | 398 | if (!DbPassword.empty()) 399 | { 400 | if (!GetConfigValueBinaryArr(PARAM_FILES_ENCODE_KEY, FileEncodeKey_, crypto_stream_chacha20_KEYBYTES)) 401 | throw; 402 | } 403 | 404 | DWORD isEncodeFileNamesSaved = 0; 405 | GetConfigValueDWord(PARAM_NAMES_ENCODE, isEncodeFileNamesSaved); 406 | if (isEncodeFileNamesSaved != 0) 407 | { 408 | if (!isEncryptedFileNames) 409 | { 410 | LOG_WARNING() << "File names encryption used in this repository" << endl; 411 | isEncryptedFileNames = true; 412 | } 413 | 414 | if (isEncryptedFileNames) 415 | { 416 | if (!GetConfigValueBinaryArr(PARAM_NAMES_ENCODE_KEY, NamesEncodeKey_, crypto_stream_chacha20_KEYBYTES) || 417 | !GetConfigValueBinaryArr(PARAM_NAMES_ENCODE_NONCE, NamesEncodeNonce_, crypto_stream_chacha20_NONCEBYTES)) 418 | throw; 419 | } 420 | } 421 | DisplayTaskConfig(); 422 | } 423 | 424 | return true; 425 | } 426 | 427 | bool TaskContext::CreateDatabase() 428 | { 429 | using namespace std; 430 | 431 | wstring dbFullPath = GenerateDbPath(); 432 | if (boost::filesystem::exists(dbFullPath)) 433 | { 434 | //DB already exists 435 | return false; 436 | } 437 | 438 | boost::filesystem::create_directories(boost::filesystem::path(dbFullPath).parent_path()); 439 | 440 | string dbNameUTF8 = keeper::WstringToUTF8(dbFullPath); 441 | 442 | std::wstring envDirectory = GenerateEnvPath(); 443 | if (!boost::filesystem::exists(envDirectory)) 444 | { 445 | boost::filesystem::create_directories(envDirectory); 446 | SetFileAttributes(envDirectory.c_str(), GetFileAttributes(envDirectory.c_str()) | FILE_ATTRIBUTE_HIDDEN); 447 | } 448 | 449 | if (OpenDatabase(true)) 450 | { 451 | //save config to DB 452 | if (CompressionLevel > 9) 453 | CompressionLevel = 0; 454 | SetConfigValueDWord(PARAM_COMPRESSION, CompressionLevel); 455 | 456 | //generate key for files encryption 457 | if (!DbPassword.empty()) 458 | { 459 | randombytes_buf(FileEncodeKey_, crypto_stream_chacha20_KEYBYTES); 460 | SetConfigValueBinaryArr(PARAM_FILES_ENCODE_KEY, FileEncodeKey_, crypto_stream_chacha20_KEYBYTES); 461 | } 462 | 463 | if (isEncryptedFileNames) 464 | { 465 | randombytes_buf(NamesEncodeKey_, crypto_stream_chacha20_KEYBYTES); 466 | randombytes_buf(NamesEncodeNonce_, crypto_stream_chacha20_NONCEBYTES); 467 | SetConfigValueBinaryArr(PARAM_NAMES_ENCODE_KEY, NamesEncodeKey_, crypto_stream_chacha20_KEYBYTES); 468 | SetConfigValueBinaryArr(PARAM_NAMES_ENCODE_NONCE, NamesEncodeNonce_, crypto_stream_chacha20_NONCEBYTES); 469 | SetConfigValueDWord(PARAM_NAMES_ENCODE, 1); 470 | } 471 | else 472 | SetConfigValueDWord(PARAM_NAMES_ENCODE, 0); 473 | 474 | if (env_) 475 | env_->txn_checkpoint(0, 0, 0); 476 | 477 | DisplayTaskConfig(); 478 | 479 | return true; 480 | } 481 | else 482 | return false; 483 | } 484 | 485 | bool TaskContext::CompressDatabase() 486 | { 487 | DB_COMPACT cData = { 0 }; 488 | cData.compact_fillpercent = 80; 489 | if (eventsDb_) 490 | eventsDb_->compact(nullptr, nullptr, nullptr, &cData, DB_FREELIST_ONLY, nullptr); 491 | 492 | //removing DB log files 493 | if (env_) 494 | env_->log_archive(nullptr, DB_ARCH_REMOVE); 495 | return true; 496 | } 497 | 498 | Db & keeper::TaskContext::GetMainDB() 499 | { 500 | return *eventsDb_; 501 | } 502 | 503 | DbEnv & keeper::TaskContext::GetEnv() 504 | { 505 | return *env_; 506 | } 507 | 508 | bool TaskContext::SetConfigValueDWord(const std::string & name, DWORD val) 509 | { 510 | Dbt key = Dbt((void*)name.c_str(), name.length()); 511 | Dbt data = Dbt((void*)&val, sizeof(DWORD)); 512 | configDb_->put(nullptr, &key, &data, DB_OVERWRITE_DUP); 513 | 514 | return true; 515 | } 516 | 517 | bool TaskContext::GetConfigValueDWord(const std::string & name, DWORD& val) 518 | { 519 | Dbt key((void*)name.c_str(), name.length()); 520 | Dbt data; 521 | 522 | int result = configDb_->get(nullptr, &key, &data, 0); 523 | if (result != 0) 524 | { 525 | LOG_ERROR() << "Can't read value from DB: " << name << std::endl; 526 | return false; 527 | } 528 | if (data.get_size() != sizeof(DWORD)) 529 | return false; 530 | val = *(DWORD*)data.get_data(); 531 | return true; 532 | } 533 | 534 | bool TaskContext::SetConfigValueBinaryArr(const std::string & name, byte * val, unsigned int len) 535 | { 536 | Dbt key = Dbt((void*)name.c_str(), name.length()); 537 | Dbt data = Dbt(val, len); 538 | configDb_->put(nullptr, &key, &data, DB_OVERWRITE_DUP); 539 | 540 | return true; 541 | } 542 | bool TaskContext::GetConfigValueBinaryArr(const std::string & name, byte * val, unsigned int len) 543 | { 544 | Dbt key((void*)name.c_str(), name.length()); 545 | Dbt data; 546 | 547 | int result = configDb_->get(nullptr, &key, &data, 0); 548 | if (result != 0) 549 | { 550 | LOG_ERROR() << "Can't read value from DB: " << name << std::endl; 551 | return false; 552 | } 553 | if (data.get_size() != len) 554 | return false; 555 | memcpy(val, data.get_data(), len); 556 | 557 | return true; 558 | } 559 | } -------------------------------------------------------------------------------- /keeper/TaskContext.h: -------------------------------------------------------------------------------- 1 | #pragma once 2 | #include 3 | #include "WildCardNameChecker.h" 4 | 5 | namespace keeper 6 | { 7 | enum class Task 8 | { 9 | Undetected, //used in CLI parser 10 | Backup, 11 | Restore, 12 | Purge, 13 | DumpDB 14 | }; 15 | 16 | struct TaskContext 17 | { 18 | public: 19 | TaskContext(); 20 | ~TaskContext(); 21 | 22 | keeper::Task Task; 23 | //avoiding BOOST's bug with extended length filenames 24 | bool SetSourceDirectory(std::wstring InitialDirectory); 25 | const std::wstring& GetSourceDirectory() const; 26 | bool SetDestinationDirectory(std::wstring InitialDirectory); 27 | const std::wstring& GetDestinationDirectory() const; 28 | bool SetRestoreTimeStamp(std::string timestamp); 29 | const boost::posix_time::ptime& GetRestoreTimeStamp() const; 30 | const boost::posix_time::ptime& GetPurgeTimeStamp() const; 31 | bool SetPurgeTimestampFromArg(std::string timestamp); 32 | bool OpenDatabase(bool CreateFreshDB = false); 33 | bool CreateDatabase(); 34 | bool CompressDatabase(); 35 | Db& GetMainDB(); 36 | DbEnv& GetEnv(); 37 | 38 | DWORD CompressionLevel = 0; 39 | unsigned int Threads = 0; //number ot threads for boost::iostreams::lzma_compressor 40 | std::string DbPassword; 41 | byte FileEncodeKey_[crypto_stream_chacha20_KEYBYTES]; 42 | bool isEncryptedFileNames = false; 43 | byte NamesEncodeKey_[crypto_stream_chacha20_KEYBYTES]; 44 | byte NamesEncodeNonce_[crypto_stream_chacha20_NONCEBYTES]; 45 | 46 | //file names wildcards support 47 | WildCardNameChecker NamesChecker; 48 | 49 | private: 50 | bool SetPurgeTimeStampFromDate(std::string timestamp); 51 | bool SetPurgeTimeStampFromDuration(std::string timestamp); 52 | 53 | bool SetConfigValueDWord(const std::string& name, DWORD val); 54 | bool GetConfigValueDWord(const std::string & name, DWORD& val); 55 | bool SetConfigValueBinaryArr(const std::string& name, byte* val, unsigned int len); 56 | bool GetConfigValueBinaryArr(const std::string& name, byte* val, unsigned int len); 57 | void DisplayTaskConfig() const; 58 | 59 | std::wstring sourceDirectory_; 60 | std::wstring destinationDirectory_; 61 | //std::wstring envDirectory_; 62 | Db* eventsDb_ = nullptr; 63 | Db* configDb_ = nullptr; 64 | 65 | void ConfigureEnv(); 66 | DbEnv* env_ = nullptr; 67 | 68 | boost::posix_time::ptime restoreTimeStamp_ = boost::posix_time::not_a_date_time; 69 | boost::posix_time::ptime purgeTimeStamp_ = boost::posix_time::not_a_date_time; 70 | std::wstring GenerateDbPath(); 71 | std::wstring GenerateEnvPath(); 72 | std::string DbKey; //key derived from password 73 | }; 74 | } -------------------------------------------------------------------------------- /keeper/TaskDumpDB.cpp: -------------------------------------------------------------------------------- 1 | #include "stdafx.h" 2 | #include "TaskDumpDB.h" 3 | #include "GlobalUtils.h" 4 | #include 5 | 6 | using namespace keeper; 7 | using namespace ConsoleLogger; 8 | 9 | TaskDumpDB::TaskDumpDB(keeper::TaskContext & ctx) : 10 | ctx_(ctx) 11 | { 12 | } 13 | 14 | TaskDumpDB::~TaskDumpDB() 15 | { 16 | } 17 | 18 | bool TaskDumpDB::Run() 19 | { 20 | if (!ctx_.OpenDatabase()) 21 | { 22 | LOG_FATAL() << "Can't open database" << std::endl; 23 | return false; 24 | } 25 | 26 | Dbc* mainCursor = nullptr; 27 | Dbt keyMain, dataMain; 28 | ctx_.GetMainDB().cursor(NULL, &mainCursor, 0); 29 | 30 | BOOST_SCOPE_EXIT_ALL(&) 31 | { 32 | if (mainCursor != nullptr) 33 | mainCursor->close(); 34 | }; 35 | 36 | int result; 37 | std::wostringstream s; 38 | 39 | while (true) 40 | { 41 | result = mainCursor->get(&keyMain, &dataMain, DB_NEXT_NODUP); 42 | if (result) 43 | break; 44 | 45 | s << std::endl << DbtToWstring(keyMain); 46 | CONSOLE() << s.str() << std::endl; 47 | s.str(std::wstring()); 48 | s.clear(); 49 | 50 | while (true) 51 | { 52 | if (dataMain.get_size() != sizeof(DbFileEvent)) 53 | throw std::exception("Corrupted DB data"); 54 | 55 | //events 56 | DbFileEvent const* pEvent = static_cast(dataMain.get_data()); 57 | 58 | s << PTimeToWstring(ConvertMillisecToPtime(pEvent->EventTimeStamp)) << L" : "; 59 | switch (static_cast(pEvent->FileEvent)) 60 | { 61 | case FileEventType::Deleted: 62 | s << "DELETED"; 63 | break; 64 | case FileEventType::Added: 65 | s << "ADDED"; 66 | break; 67 | case FileEventType::Changed: 68 | s << "CHANGED"; 69 | break; 70 | default: 71 | s << "UNKNOWN"; 72 | break; 73 | } 74 | 75 | //size 76 | ULONGLONG fileSize = static_cast(pEvent->FileAttributes.nFileSizeHigh) * (static_cast(MAXDWORD) + 1) + pEvent->FileAttributes.nFileSizeLow; 77 | s << L"\nSize: "; 78 | s << std::dec << fileSize; 79 | 80 | //file attributes 81 | auto fileAttr = pEvent->FileAttributes.dwFileAttributes; 82 | s << L" Attributes: "; 83 | if (fileAttr & FILE_ATTRIBUTE_READONLY) 84 | s << L"READONLY "; 85 | if (fileAttr & FILE_ATTRIBUTE_HIDDEN) 86 | s << L"HIDDEN "; 87 | if (fileAttr & FILE_ATTRIBUTE_SYSTEM) 88 | s << L"SYSTEM "; 89 | if (fileAttr & FILE_ATTRIBUTE_DIRECTORY) 90 | s << L"DIRECTORY "; 91 | if (fileAttr & FILE_ATTRIBUTE_ARCHIVE) 92 | s << L"ARCHIVE "; 93 | if (fileAttr & FILE_ATTRIBUTE_DEVICE) 94 | s << L"DEVICE "; 95 | if (fileAttr & FILE_ATTRIBUTE_NORMAL) 96 | s << L"NORMAL "; 97 | if (fileAttr & FILE_ATTRIBUTE_TEMPORARY) 98 | s << L"TEMPORARY "; 99 | if (fileAttr & FILE_ATTRIBUTE_SPARSE_FILE) 100 | s << L"SPARSE_FILE "; 101 | if (fileAttr & FILE_ATTRIBUTE_REPARSE_POINT) 102 | s << L"REPARSE_POINT "; 103 | if (fileAttr & FILE_ATTRIBUTE_COMPRESSED) 104 | s << L"COMPRESSED "; 105 | if (fileAttr & FILE_ATTRIBUTE_OFFLINE) 106 | s << L"OFFLINE "; 107 | if (fileAttr & FILE_ATTRIBUTE_NOT_CONTENT_INDEXED) 108 | s << L"NOT_CONTENT_INDEXED "; 109 | if (fileAttr & FILE_ATTRIBUTE_ENCRYPTED) 110 | s << L"ENCRYPTED "; 111 | #ifndef _USING_V110_SDK71_ 112 | if (fileAttr & FILE_ATTRIBUTE_INTEGRITY_STREAM) 113 | s << L"INTEGRITY_STREAM "; 114 | if (fileAttr & FILE_ATTRIBUTE_NO_SCRUB_DATA) 115 | s << L"NO_SCRUB_DATA "; 116 | if (fileAttr & FILE_ATTRIBUTE_EA) 117 | s << L"EA "; 118 | #endif 119 | if (fileAttr & FILE_ATTRIBUTE_VIRTUAL) 120 | s << L"VIRTUAL"; 121 | CONSOLE() << s.str() << std::endl; 122 | s.str(std::wstring()); 123 | s.clear(); 124 | 125 | //dates 126 | SYSTEMTIME systemTime; 127 | FILETIME fileTime; 128 | 129 | fileTime = pEvent->FileAttributes.ftCreationTime; 130 | FileTimeToLocalFileTime(&fileTime, &fileTime); 131 | FileTimeToSystemTime(&fileTime, &systemTime); 132 | s << L"Created:" << SystemTimeToWstring(systemTime); 133 | 134 | fileTime = pEvent->FileAttributes.ftLastAccessTime; 135 | FileTimeToLocalFileTime(&fileTime, &fileTime); 136 | FileTimeToSystemTime(&fileTime, &systemTime); 137 | s << L" Last access:" << SystemTimeToWstring(systemTime); 138 | 139 | fileTime = pEvent->FileAttributes.ftLastWriteTime; 140 | FileTimeToLocalFileTime(&fileTime, &fileTime); 141 | FileTimeToSystemTime(&fileTime, &systemTime); 142 | s << L" Write time:" << SystemTimeToWstring(systemTime); 143 | CONSOLE() << s.str() << std::endl; 144 | s.str(std::wstring()); 145 | s.clear(); 146 | 147 | result = mainCursor->get(&keyMain, &dataMain, DB_NEXT_DUP); 148 | if (result) 149 | break; 150 | } 151 | } 152 | 153 | return true; 154 | } 155 | -------------------------------------------------------------------------------- /keeper/TaskDumpDB.h: -------------------------------------------------------------------------------- 1 | #pragma once 2 | #include "TaskContext.h" 3 | 4 | class TaskDumpDB final 5 | { 6 | public: 7 | TaskDumpDB(keeper::TaskContext& ctx); 8 | ~TaskDumpDB(); 9 | 10 | bool Run(); 11 | private: 12 | keeper::TaskContext& ctx_; 13 | }; 14 | 15 | -------------------------------------------------------------------------------- /keeper/TaskPurge.cpp: -------------------------------------------------------------------------------- 1 | #include "stdafx.h" 2 | #include "TaskPurge.h" 3 | #include "GlobalUtils.h" 4 | #include 5 | #include 6 | #include 7 | #include "FileIO.h" 8 | #include "FilesTransformer.h" 9 | #include "TnxGuard.h" 10 | 11 | using namespace keeper; 12 | using namespace ConsoleLogger; 13 | using namespace boost::posix_time; 14 | 15 | TaskPurge::TaskPurge(keeper::TaskContext & ctx) : 16 | ctx_(ctx) 17 | { 18 | } 19 | 20 | inline FileEventType GetEventType(const Dbt& dbt) 21 | { 22 | return static_cast(static_cast(dbt.get_data())->FileEvent); 23 | } 24 | 25 | bool TaskPurge::Run() 26 | { 27 | auto purgeTimeStamp = ctx_.GetPurgeTimeStamp(); 28 | if (purgeTimeStamp == not_a_date_time) 29 | { 30 | LOG_FATAL() << "Purge timestamp not defined" << std::endl; 31 | return false; 32 | } 33 | 34 | if (!ctx_.OpenDatabase()) 35 | { 36 | LOG_FATAL() << "Can't open database" << std::endl; 37 | return false; 38 | } 39 | 40 | LOG_INFO() << "Purging files older than " << PTimeToWstring(purgeTimeStamp) << std::endl; 41 | 42 | Dbc* mainCursor; 43 | Dbt keyMain, dataMain; 44 | TnxGuard tnxGuard(ctx_.GetEnv()); 45 | 46 | ctx_.GetMainDB().cursor(tnxGuard.Get(), &mainCursor, 0); 47 | BOOST_SCOPE_EXIT_ALL(&) 48 | { 49 | if (mainCursor != nullptr) 50 | mainCursor->close(); 51 | }; 52 | 53 | int64_t timestamp64 = keeper::ConvertPtimeToMillisec(purgeTimeStamp); 54 | std::set deletedDirsTimestamps64; 55 | std::wstring relativePath; 56 | int result; 57 | DbFileEvent* pEvent; 58 | while (true) 59 | { 60 | result = mainCursor->get(&keyMain, &dataMain, DB_NEXT_NODUP); 61 | if (result == DB_NOTFOUND) //end of DB 62 | break; 63 | 64 | while (true) 65 | { 66 | if (dataMain.get_size() != sizeof(DbFileEvent)) 67 | throw std::exception("Corrupted DB data"); 68 | 69 | pEvent = static_cast(dataMain.get_data()); 70 | auto eventTimestamp64 = pEvent->EventTimeStamp; 71 | if (eventTimestamp64 < timestamp64) 72 | { 73 | //check if next record present 74 | result = mainCursor->get(&keyMain, &dataMain, DB_NEXT_DUP); 75 | if (result == DB_NOTFOUND) 76 | { 77 | //it was the last event 78 | if (static_cast(pEvent->FileEvent) == FileEventType::Deleted) 79 | { 80 | mainCursor->del(0); 81 | deletedDirsTimestamps64.insert(eventTimestamp64); 82 | } 83 | break; 84 | } 85 | 86 | //step back and delete 87 | result = mainCursor->get(&keyMain, &dataMain, DB_PREV_DUP); 88 | //delete DB record 89 | mainCursor->del(0); 90 | 91 | deletedDirsTimestamps64.insert(eventTimestamp64); 92 | } 93 | 94 | result = mainCursor->get(&keyMain, &dataMain, DB_NEXT_DUP); 95 | if (result == DB_NOTFOUND) 96 | break; 97 | } 98 | } 99 | mainCursor->close(); 100 | tnxGuard.Commit(); 101 | 102 | if (!deletedDirsTimestamps64.empty()) 103 | { 104 | LOG_INFO() << "Purging backups at:" << std::endl; 105 | for (auto& ts : deletedDirsTimestamps64) 106 | { 107 | std::wstring storeOldDir = ctx_.GetSourceDirectory() + PTimeToWstringSafeSymbols(ConvertMillisecToPtime(ts)); 108 | if (boost::filesystem::exists(storeOldDir)) 109 | { 110 | LOG_INFO() << PTimeToWstring(ConvertMillisecToPtime(ts)) << std::endl; 111 | keeper::FileIO::RemoveDir(storeOldDir); 112 | } 113 | } 114 | 115 | //compress DB 116 | LOG_VERBOSE() << "Compressing database" << std::endl; 117 | ctx_.CompressDatabase(); 118 | } 119 | 120 | LOG_VERBOSE() << "Deleting orphaned files (without DB record)" << std::endl; 121 | 122 | keeper::FilesTransformer transformer(ctx_); 123 | std::wstring mirrorDir = ctx_.GetSourceDirectory() + MIRROR_SUB_DIR; 124 | auto skipPathCharsCount = mirrorDir.length(); 125 | boost::filesystem::recursive_directory_iterator dirIterator(mirrorDir); 126 | std::vector filesToDelete; 127 | std::wstring restoredRelativePath; 128 | 129 | Dbc* mainCursor2; 130 | TnxGuard tnxGuard2(ctx_.GetEnv()); 131 | ctx_.GetMainDB().cursor(tnxGuard2.Get(), &mainCursor2, 0); 132 | 133 | while (true) 134 | { 135 | if (dirIterator == boost::filesystem::recursive_directory_iterator()) 136 | break; 137 | const std::wstring& sourceFullPath = dirIterator->path().wstring(); 138 | relativePath = sourceFullPath.substr(skipPathCharsCount, skipPathCharsCount - sourceFullPath.length()); 139 | restoredRelativePath = transformer.GetOriginalName(relativePath, is_directory(dirIterator->path())); 140 | 141 | //get the last event 142 | Dbt key = keeper::WstringToDbt(restoredRelativePath); 143 | Dbt data; 144 | 145 | result = mainCursor2->get(&key, &data, DB_SET); 146 | if (result == 0) 147 | { 148 | while (true) 149 | { 150 | result = mainCursor2->get(&key, &data, DB_NEXT_DUP); 151 | if (result == DB_NOTFOUND) 152 | break; 153 | //TODO: result check 154 | } 155 | pEvent = static_cast(data.get_data()); 156 | if (static_cast(pEvent->FileEvent) == FileEventType::Deleted) 157 | filesToDelete.push_back(dirIterator->path().wstring()); 158 | } 159 | else 160 | if (result == DB_NOTFOUND) 161 | { 162 | filesToDelete.push_back(dirIterator->path().wstring()); 163 | } 164 | ++dirIterator; 165 | } 166 | mainCursor2->close(); 167 | tnxGuard2.Commit(); 168 | 169 | for (const auto& filename : filesToDelete) 170 | { 171 | if (boost::filesystem::exists(filename)) 172 | keeper::FileIO::RemoveDir(filename); 173 | } 174 | 175 | return true; 176 | } 177 | -------------------------------------------------------------------------------- /keeper/TaskPurge.h: -------------------------------------------------------------------------------- 1 | #pragma once 2 | #include "TaskContext.h" 3 | 4 | class TaskPurge final 5 | { 6 | public: 7 | TaskPurge(keeper::TaskContext& ctx); 8 | bool Run(); 9 | 10 | private: 11 | keeper::TaskContext& ctx_; 12 | }; -------------------------------------------------------------------------------- /keeper/TaskRestore.cpp: -------------------------------------------------------------------------------- 1 | #include "stdafx.h" 2 | #include "TaskRestore.h" 3 | #include 4 | #include 5 | #include "GlobalUtils.h" 6 | #include "FileIO.h" 7 | 8 | using namespace ConsoleLogger; 9 | using namespace boost::posix_time; 10 | 11 | TaskRestore::TaskRestore(keeper::TaskContext & ctx) : 12 | ctx_(ctx) 13 | { 14 | } 15 | 16 | TaskRestore::~TaskRestore() 17 | { 18 | } 19 | 20 | bool IsDirectory(const Dbt& rec) 21 | { 22 | if ((static_cast(rec.get_data()))->FileAttributes.dwFileAttributes & FILE_ATTRIBUTE_DIRECTORY) 23 | return true; 24 | else 25 | return false; 26 | } 27 | 28 | void RestoreFileTimestamps(const std::wstring& path, const WIN32_FILE_ATTRIBUTE_DATA& attribData) 29 | { 30 | HANDLE hFile = CreateFile(path.c_str(), FILE_WRITE_ATTRIBUTES, FILE_SHARE_READ | FILE_SHARE_WRITE, NULL, OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL, NULL); 31 | if (hFile != NULL) 32 | { 33 | SetFileTime(hFile, &attribData.ftCreationTime, &attribData.ftLastAccessTime, &attribData.ftLastWriteTime); 34 | CloseHandle(hFile); 35 | } 36 | else 37 | LOG_ERROR() << "Can't change file's timestamps: " << path << std::endl; 38 | } 39 | 40 | void TaskRestore::RestoreFromMirrorFolder(const Dbt& key, const DbFileEvent& data) 41 | { 42 | std::wstring relativePath = keeper::DbtToWstring(key); 43 | 44 | if (ctx_.NamesChecker.IsFilteringEnabled) 45 | { 46 | if (!ctx_.NamesChecker.IsFitPattern(relativePath)) 47 | return; 48 | } 49 | 50 | std::wstring destinationFullPath = ctx_.GetDestinationDirectory() + relativePath; 51 | bool isRestoreSucceed; 52 | bool isDirectory = (data.FileAttributes.dwFileAttributes & FILE_ATTRIBUTE_DIRECTORY) != 0; 53 | if (isDirectory) 54 | { 55 | if (boost::filesystem::exists(destinationFullPath)) 56 | isRestoreSucceed = true; 57 | else 58 | isRestoreSucceed = keeper::FileIO::CreateDir(destinationFullPath); 59 | } 60 | else 61 | { 62 | auto transformedPath_ = transformer->GetTransformedName(relativePath, isDirectory); 63 | isRestoreSucceed = transformer->RestoreFile(ctx_.GetSourceDirectory() + MIRROR_SUB_DIR + transformedPath_, destinationFullPath); 64 | } 65 | if (isRestoreSucceed) 66 | { 67 | SetFileAttributes(destinationFullPath.c_str(), data.FileAttributes.dwFileAttributes); 68 | RestoreFileTimestamps(destinationFullPath, data.FileAttributes); 69 | } 70 | } 71 | 72 | void TaskRestore::RestoreFromEventFolder(const Dbt & key, const DbFileEvent& data) 73 | { 74 | boost::posix_time::ptime storeOldTimestamp = keeper::ConvertMillisecToPtime(data.EventTimeStamp); 75 | bool isDirectory = (data.FileAttributes.dwFileAttributes & FILE_ATTRIBUTE_DIRECTORY) != 0; 76 | 77 | std::wstring relativePath = keeper::DbtToWstring(key); 78 | 79 | if (ctx_.NamesChecker.IsFilteringEnabled) 80 | { 81 | if (!ctx_.NamesChecker.IsFitPattern(relativePath)) 82 | return; 83 | } 84 | 85 | auto transformedPath_ = transformer->GetTransformedName(relativePath, isDirectory); 86 | std::wstring storeOldFullPath = ctx_.GetSourceDirectory() + keeper::PTimeToWstringSafeSymbols(storeOldTimestamp) + L'\\' + transformedPath_; 87 | std::wstring destinationFullPath = ctx_.GetDestinationDirectory() + relativePath; 88 | bool restoreResult; 89 | 90 | if (data.FileAttributes.dwFileAttributes & FILE_ATTRIBUTE_DIRECTORY) 91 | { 92 | if (boost::filesystem::exists(destinationFullPath)) 93 | restoreResult = true; 94 | else 95 | restoreResult = keeper::FileIO::CreateDir(destinationFullPath); 96 | } 97 | else 98 | { 99 | restoreResult = transformer->RestoreFile(storeOldFullPath, destinationFullPath); 100 | } 101 | if (restoreResult) 102 | { 103 | SetFileAttributes(destinationFullPath.c_str(), data.FileAttributes.dwFileAttributes); 104 | RestoreFileTimestamps(destinationFullPath, data.FileAttributes); 105 | } 106 | } 107 | 108 | inline FileEventType GetEventType(const Dbt& dbt) 109 | { 110 | return static_cast(static_cast(dbt.get_data())->FileEvent); 111 | } 112 | 113 | bool TaskRestore::Run() 114 | { 115 | if (!ctx_.OpenDatabase()) 116 | { 117 | LOG_FATAL() << "Can't open database" << std::endl; 118 | return false; 119 | } 120 | transformer = std::make_unique(ctx_); 121 | 122 | const ptime& timestamp = ctx_.GetRestoreTimeStamp(); 123 | int64_t userTimestamp64 = (timestamp != not_a_date_time) ? keeper::ConvertPtimeToMillisec(timestamp) : 0; 124 | 125 | Dbc* mainCursor = nullptr; 126 | ctx_.GetMainDB().cursor(nullptr, &mainCursor, 0); 127 | BOOST_SCOPE_EXIT_ALL(&) 128 | { 129 | if (mainCursor != nullptr) 130 | mainCursor->close(); 131 | }; 132 | 133 | Dbt keyMain, dataMain/*, dataMainPrev*/; 134 | DbFileEvent fileEvent = { 0 }, fileEventPrev = { 0 }; 135 | 136 | Dbt dataNext; 137 | int64_t eventTimestamp; 138 | int result; 139 | 140 | while (true) 141 | { 142 | result = mainCursor->get(&keyMain, &dataMain, DB_NEXT_NODUP); 143 | if (result == DB_NOTFOUND) //end of DB 144 | break; 145 | 146 | //got the file name, seek for bounded events 147 | bool isFirstEvent = true; 148 | bool isFileRestored = false; 149 | while (true) 150 | { 151 | eventTimestamp = static_cast(dataMain.get_data())->EventTimeStamp; 152 | fileEvent = *static_cast(dataMain.get_data()); 153 | 154 | if (timestamp != not_a_date_time) 155 | { 156 | //timestamp match exactly 157 | if (eventTimestamp == userTimestamp64) 158 | { 159 | if (GetEventType(dataMain) == FileEventType::Deleted) 160 | break; 161 | result = mainCursor->get(&keyMain, &dataNext, DB_NEXT_DUP); 162 | //is it the last event? 163 | if (result == DB_NOTFOUND) 164 | RestoreFromMirrorFolder(keyMain, fileEvent); 165 | else 166 | RestoreFromEventFolder(keyMain, fileEvent); 167 | 168 | isFileRestored = true; 169 | break; 170 | } 171 | 172 | //got event after timestamp 173 | if (eventTimestamp > userTimestamp64) 174 | { 175 | isFileRestored = true; //in any case 176 | if (isFirstEvent) 177 | { 178 | //first event occured AFTER timestamp 179 | 180 | //if first event !Added, then restore from the next event, cause Added event was purged 181 | if (GetEventType(dataMain) != FileEventType::Added) 182 | { 183 | //read next event 184 | result = mainCursor->get(&keyMain, &dataNext, DB_NEXT_DUP); 185 | if (result == DB_NOTFOUND) 186 | RestoreFromMirrorFolder(keyMain, fileEvent); 187 | else 188 | RestoreFromEventFolder(keyMain, fileEvent); 189 | 190 | break; 191 | } 192 | else 193 | break; //there was no such file at the given time 194 | } 195 | else 196 | { 197 | //previous event as the actual 198 | if (static_cast(fileEventPrev.FileEvent) == FileEventType::Deleted) //no such file at this time 199 | break; 200 | 201 | RestoreFromEventFolder(keyMain, fileEvent); 202 | } 203 | 204 | break; 205 | } 206 | }//if (timestamp != not_a_date_time) 207 | else 208 | { 209 | //seek till the end 210 | } 211 | //proceed to the next event 212 | fileEventPrev = fileEvent; 213 | isFirstEvent = false; 214 | 215 | result = mainCursor->get(&keyMain, &dataMain, DB_NEXT_DUP); 216 | if (result != 0) 217 | break; 218 | }//////////////////////////////////////////// 219 | 220 | if (isFileRestored) 221 | continue; 222 | 223 | if (result == DB_NOTFOUND) 224 | { 225 | //got the last event 226 | //if (timestamp == not_a_date_time) 227 | { 228 | //ReplayLastEvent(keyMain, dataMain); 229 | if (static_cast(fileEventPrev.FileEvent) != FileEventType::Deleted) 230 | RestoreFromMirrorFolder(keyMain, fileEventPrev); 231 | } 232 | } 233 | }//while 234 | 235 | return true; 236 | } 237 | -------------------------------------------------------------------------------- /keeper/TaskRestore.h: -------------------------------------------------------------------------------- 1 | #pragma once 2 | #include "TaskContext.h" 3 | #include "FilesTransformer.h" 4 | 5 | class TaskRestore final 6 | { 7 | public: 8 | TaskRestore(keeper::TaskContext& ctx); 9 | ~TaskRestore(); 10 | bool Run(); 11 | 12 | private: 13 | void RestoreFromEventFolder(const Dbt & key, const DbFileEvent& data); 14 | void RestoreFromMirrorFolder(const Dbt & key, const DbFileEvent& data); 15 | keeper::TaskContext& ctx_; 16 | std::unique_ptr transformer; 17 | }; 18 | 19 | -------------------------------------------------------------------------------- /keeper/TestMain.cpp: -------------------------------------------------------------------------------- 1 | #include "stdafx.h" 2 | #define DOCTEST_CONFIG_IMPLEMENT 3 | #include "doctest\doctest.h" 4 | 5 | int main(int argc, char** argv) 6 | { 7 | sodium_init(); 8 | 9 | doctest::Context context; 10 | context.applyCommandLine(argc, argv); 11 | 12 | return context.run(); 13 | } -------------------------------------------------------------------------------- /keeper/TnxGuard.cpp: -------------------------------------------------------------------------------- 1 | #include "stdafx.h" 2 | #include "TnxGuard.h" 3 | 4 | TnxGuard::TnxGuard(DbEnv& env) : 5 | env_(env) 6 | { 7 | env_.txn_begin(NULL, &txn_, 0); 8 | } 9 | 10 | TnxGuard::~TnxGuard() 11 | { 12 | if (!isAborted_ && !isCommited_ && (txn_ != nullptr)) 13 | txn_->commit(0); 14 | } 15 | 16 | DbTxn* TnxGuard::Get() 17 | { 18 | return txn_; 19 | } 20 | 21 | void TnxGuard::Commit() 22 | { 23 | if (!isAborted_ && !isCommited_ && (txn_ != nullptr)) 24 | txn_->commit(0); 25 | isCommited_ = true; 26 | } 27 | 28 | void TnxGuard::Abort() 29 | { 30 | if (!isAborted_ && !isCommited_ && (txn_ != nullptr)) 31 | txn_->abort(); 32 | isAborted_ = true; 33 | } 34 | -------------------------------------------------------------------------------- /keeper/TnxGuard.h: -------------------------------------------------------------------------------- 1 | #pragma once 2 | class TnxGuard 3 | { 4 | public: 5 | TnxGuard(DbEnv&); 6 | ~TnxGuard(); 7 | DbTxn* Get(); 8 | void Commit(); 9 | void Abort(); 10 | private: 11 | DbEnv& env_; 12 | DbTxn* txn_ = nullptr; 13 | bool isAborted_ = false; 14 | bool isCommited_ = false; 15 | }; 16 | -------------------------------------------------------------------------------- /keeper/WildCardNameChecker.cpp: -------------------------------------------------------------------------------- 1 | #include "stdafx.h" 2 | #include "WildCardNameChecker.h" 3 | #include 4 | #include 5 | 6 | using namespace std; 7 | 8 | keeper::WildCardNameChecker::WildCardNameChecker() 9 | { 10 | } 11 | 12 | void keeper::WildCardNameChecker::AddMasksToSet(const wstring & masks, std::vector& s) 13 | { 14 | vector masksVect; 15 | boost::split(masksVect, masks, [](wchar_t ch) -> bool { return ch == (wchar_t)MASKS_SEPARATOR; }); 16 | 17 | for (auto const& mask : masksVect) 18 | { 19 | s.push_back(wregex(MaskToPattern(mask), regex_constants::icase)); 20 | } 21 | } 22 | 23 | void keeper::WildCardNameChecker::AddIncludeMask(const std::wstring & masks) 24 | { 25 | AddMasksToSet(masks, IncludePatterns_); 26 | IsFilteringEnabled = IsFilteringEnabled || !IncludePatterns_.empty(); 27 | } 28 | 29 | void keeper::WildCardNameChecker::AddExcludeMask(const std::wstring & masks) 30 | { 31 | AddMasksToSet(masks, ExcludePatterns_); 32 | IsFilteringEnabled = IsFilteringEnabled || !ExcludePatterns_.empty(); 33 | } 34 | 35 | std::wstring keeper::WildCardNameChecker::MaskToPattern(const std::wstring & mask) 36 | { 37 | std::wstring pattern = mask; 38 | boost::replace_all(pattern, "\\", "\\\\"); 39 | boost::replace_all(pattern, "^", "\\^"); 40 | boost::replace_all(pattern, ".", "\\."); 41 | boost::replace_all(pattern, "$", "\\$"); 42 | boost::replace_all(pattern, "|", "\\|"); 43 | boost::replace_all(pattern, "(", "\\("); 44 | boost::replace_all(pattern, ")", "\\)"); 45 | boost::replace_all(pattern, "[", "\\["); 46 | boost::replace_all(pattern, "]", "\\]"); 47 | boost::replace_all(pattern, "*", "\\*"); 48 | boost::replace_all(pattern, "+", "\\+"); 49 | boost::replace_all(pattern, "?", "\\?"); 50 | boost::replace_all(pattern, "/", "\\/"); 51 | 52 | // Convert chars '*?' back to their regex equivalents 53 | boost::replace_all(pattern, "\\?", "."); 54 | boost::replace_all(pattern, "\\*", ".*"); 55 | 56 | return pattern; 57 | } 58 | 59 | bool keeper::WildCardNameChecker::IsFitPattern(const std::wstring & fileName) 60 | { 61 | if (!IsFilteringEnabled) 62 | return true; 63 | 64 | bool includeMatch = false; 65 | for (auto const& expr : IncludePatterns_) 66 | { 67 | if (regex_match(fileName, expr)) 68 | { 69 | includeMatch = true; 70 | break; 71 | } 72 | } 73 | if (!IncludePatterns_.empty() && (includeMatch == false)) 74 | return false; 75 | 76 | for (auto const& expr : ExcludePatterns_) 77 | { 78 | if (regex_match(fileName, expr)) 79 | return false; 80 | } 81 | 82 | return true; 83 | } -------------------------------------------------------------------------------- /keeper/WildCardNameChecker.h: -------------------------------------------------------------------------------- 1 | #pragma once 2 | #include 3 | 4 | namespace keeper 5 | { 6 | class WildCardNameChecker 7 | { 8 | public: 9 | WildCardNameChecker(); 10 | void AddIncludeMask(const std::wstring& mask); 11 | void AddExcludeMask(const std::wstring& mask); 12 | bool IsFitPattern(const std::wstring& fileName); 13 | 14 | bool IsFilteringEnabled = false; 15 | 16 | private: 17 | void AddMasksToSet(const std::wstring & masks, std::vector& v); 18 | std::vector IncludePatterns_; 19 | std::vector ExcludePatterns_; 20 | std::wstring MaskToPattern(const std::wstring& mask); 21 | }; 22 | } 23 | -------------------------------------------------------------------------------- /keeper/WildCardNamesCheckerTest.cpp: -------------------------------------------------------------------------------- 1 | #include "stdafx.h" 2 | #include "doctest\doctest.h" 3 | #include "WildCardNameChecker.h" 4 | 5 | TEST_CASE("WildCardNameChecker") 6 | { 7 | SUBCASE("excluding names") 8 | { 9 | keeper::WildCardNameChecker checker; 10 | checker.IsFilteringEnabled = true; 11 | checker.AddExcludeMask(L"*.mp3"); 12 | CHECK(checker.IsFitPattern(L"mydoc.xls")); 13 | CHECK(checker.IsFitPattern(L"mp3mp3mp3")); 14 | CHECK(!checker.IsFitPattern(L".mp3")); 15 | CHECK(checker.IsFitPattern(L"file.mp3333")); 16 | CHECK(!checker.IsFitPattern(L"song.mp3")); 17 | checker.AddExcludeMask(L"*.aac"); 18 | CHECK(checker.IsFitPattern(L"mydoc.doc")); 19 | CHECK(!checker.IsFitPattern(L"song.aac")); 20 | CHECK(!checker.IsFitPattern(L"song.mp3")); 21 | } 22 | SUBCASE("exclude masks set") 23 | { 24 | keeper::WildCardNameChecker checker; 25 | checker.IsFilteringEnabled = true; 26 | checker.AddExcludeMask(L"*.mp3;*.aac;*.wav"); 27 | CHECK(checker.IsFitPattern(L"mydoc.xls")); 28 | CHECK(checker.IsFitPattern(L"mydoc.doc")); 29 | CHECK(!checker.IsFitPattern(L"song.mp3")); 30 | CHECK(!checker.IsFitPattern(L"song.aac")); 31 | CHECK(!checker.IsFitPattern(L"song.wav")); 32 | } 33 | SUBCASE("including names") 34 | { 35 | keeper::WildCardNameChecker checker; 36 | checker.IsFilteringEnabled = true; 37 | checker.AddIncludeMask(L"*.mp3"); 38 | CHECK(!checker.IsFitPattern(L"mydoc.xls")); 39 | CHECK(checker.IsFitPattern(L"song.mp3")); 40 | checker.AddIncludeMask(L"*.aac"); 41 | CHECK(!checker.IsFitPattern(L"mydoc.doc")); 42 | CHECK(checker.IsFitPattern(L"song.aac")); 43 | CHECK(checker.IsFitPattern(L"song.mp3")); 44 | } 45 | SUBCASE("include masks set") 46 | { 47 | keeper::WildCardNameChecker checker; 48 | checker.IsFilteringEnabled = true; 49 | checker.AddIncludeMask(L"*.mp3;*.aac;*.wav"); 50 | CHECK(!checker.IsFitPattern(L"mydoc.xls")); 51 | CHECK(!checker.IsFitPattern(L"mydoc.doc")); 52 | CHECK(checker.IsFitPattern(L"song.mp3")); 53 | CHECK(checker.IsFitPattern(L"song.aac")); 54 | CHECK(checker.IsFitPattern(L"song.wav")); 55 | } 56 | } -------------------------------------------------------------------------------- /keeper/basen.hpp: -------------------------------------------------------------------------------- 1 | /** 2 | * base-n, 1.0 3 | * Copyright (C) 2012 Andrzej Zawadzki (azawadzki@gmail.com) 4 | * 5 | * Permission is hereby granted, free of charge, to any person obtaining a copy 6 | * of this software and associated documentation files (the "Software"), to deal 7 | * in the Software without restriction, including without limitation the rights 8 | * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | * copies of the Software, and to permit persons to whom the Software is 10 | * furnished to do so, subject to the following conditions: 11 | * 12 | * The above copyright notice and this permission notice shall be included in 13 | * all copies or substantial portions of the Software. 14 | * 15 | * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | * SOFTWARE. 22 | **/ 23 | #ifndef BASEN_HPP 24 | #define BASEN_HPP 25 | 26 | #include 27 | #include 28 | #include 29 | #include 30 | 31 | namespace bn 32 | { 33 | 34 | template 35 | void encode_b16(Iter1 start, Iter1 end, Iter2 out); 36 | 37 | template 38 | void encode_b32(Iter1 start, Iter1 end, Iter2 out); 39 | 40 | template 41 | void encode_b64(Iter1 start, Iter1 end, Iter2 out); 42 | 43 | template 44 | void decode_b16(Iter1 start, Iter1 end, Iter2 out); 45 | 46 | template 47 | void decode_b32(Iter1 start, Iter1 end, Iter2 out); 48 | 49 | template 50 | void decode_b64(Iter1 start, Iter1 end, Iter2 out); 51 | 52 | namespace impl 53 | { 54 | 55 | const int Error = -1; 56 | 57 | namespace { 58 | 59 | char extract_partial_bits(char value, size_t start_bit, size_t bits_count) 60 | { 61 | assert(start_bit + bits_count < 8); 62 | // shift extracted bits to the beginning of the byte 63 | char t1 = value >> (8 - bits_count - start_bit); 64 | // mask out bits on the left 65 | char t2 = t1 & ~(0xff << bits_count); 66 | return t2; 67 | } 68 | 69 | char extract_overlapping_bits(char previous, char next, size_t start_bit, size_t bits_count) 70 | { 71 | assert(start_bit + bits_count < 16); 72 | size_t bits_count_in_previous = 8 - start_bit; 73 | size_t bits_count_in_next = bits_count - bits_count_in_previous; 74 | char t1 = previous << bits_count_in_next; 75 | char t2 = next >> (8 - bits_count_in_next) & ~(0xff << bits_count_in_next) ; 76 | return (t1 | t2) & ~(0xff << bits_count); 77 | } 78 | 79 | } 80 | 81 | struct b16_conversion_traits 82 | { 83 | static size_t group_length() 84 | { 85 | return 4; 86 | } 87 | 88 | static char encode(unsigned int index) 89 | { 90 | static const char* const dictionary = "0123456789ABCDEF"; 91 | assert(index < strlen(dictionary)); 92 | return dictionary[index]; 93 | } 94 | 95 | static char decode(char c) 96 | { 97 | if (c >= '0' && c <= '9') { 98 | return c - '0'; 99 | } else if (c >= 'A' && c <= 'F') { 100 | return c - 'A' + 10; 101 | } 102 | return Error; 103 | } 104 | }; 105 | 106 | struct b32_conversion_traits 107 | { 108 | static size_t group_length() 109 | { 110 | return 5; 111 | } 112 | 113 | static char encode(unsigned int index) 114 | { 115 | static const char * dictionary = "ABCDEFGHIJKLMNOPQRSTUVWXYZ234567"; 116 | assert(index < strlen(dictionary)); 117 | return dictionary[index]; 118 | } 119 | 120 | static char decode(char c) 121 | { 122 | if (c >= 'A' && c <= 'Z') { 123 | return c - 'A'; 124 | } else if (c >= '2' && c <= '7') { 125 | return c - '2' + 26; 126 | } 127 | return Error; 128 | } 129 | }; 130 | 131 | struct b64_conversion_traits 132 | { 133 | static size_t group_length() 134 | { 135 | return 6; 136 | } 137 | 138 | static char encode(unsigned int index) 139 | { 140 | static const char* const dictionary = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/"; 141 | assert(index < strlen(dictionary)); 142 | return dictionary[index]; 143 | } 144 | 145 | static char decode(char c) 146 | { 147 | const int alph_len = 26; 148 | if (c >= 'A' && c <= 'Z') { 149 | return c - 'A'; 150 | } else if (c >= 'a' && c <= 'z') { 151 | return c - 'a' + alph_len * 1; 152 | } else if (c >= '0' && c <= '9') { 153 | return c - '0' + alph_len * 2; 154 | } else if (c == '+') { 155 | return c - '+' + alph_len * 2 + 10; 156 | } else if (c == '/') { 157 | return c - '/' + alph_len * 2 + 11; 158 | } 159 | return Error; 160 | } 161 | }; 162 | 163 | template 164 | void decode(Iter1 start, Iter1 end, Iter2 out) 165 | { 166 | Iter1 iter = start; 167 | size_t output_current_bit = 0; 168 | char buffer = 0; 169 | 170 | while (iter != end) { 171 | if (std::isspace(*iter)) { 172 | ++iter; 173 | continue; 174 | } 175 | char value = ConversionTraits::decode(*iter); 176 | if (value == Error) { 177 | // malformed data, but let's go on... 178 | ++iter; 179 | continue; 180 | } 181 | size_t bits_in_current_byte = std::min(output_current_bit + ConversionTraits::group_length(), 8) - output_current_bit; 182 | if (bits_in_current_byte == ConversionTraits::group_length()) { 183 | // the value fits within current byte, so we can extract it directly 184 | buffer |= value << (8 - output_current_bit - ConversionTraits::group_length()); 185 | output_current_bit += ConversionTraits::group_length(); 186 | // check if we filled up current byte completely; in such case we flush output and continue 187 | if (output_current_bit == 8) { 188 | *out++ = buffer; 189 | buffer = 0; 190 | output_current_bit = 0; 191 | } 192 | } else { 193 | // the value spans across the current and the next byte 194 | size_t bits_in_next_byte = ConversionTraits::group_length() - bits_in_current_byte; 195 | // fill the current byte and flush it to our output 196 | buffer |= value >> bits_in_next_byte; 197 | *out++ = buffer; 198 | buffer = 0; 199 | // save the remainder of our value in the buffer; it will be flushed 200 | // during next iterations 201 | buffer |= value << (8 - bits_in_next_byte); 202 | output_current_bit = bits_in_next_byte; 203 | } 204 | ++iter; 205 | } 206 | } 207 | 208 | template 209 | void encode(Iter1 start, Iter1 end, Iter2 out) 210 | { 211 | #if __cplusplus >= 201103 || (defined(_MSC_VER) && _MSC_VER > 1700) 212 | static_assert(sizeof(*start) == sizeof(char), "only char-size input supported"); 213 | #else // compatibility with older C++ without static assertions 214 | assert(sizeof(*start) == sizeof(char) && "only char-size input supported"); 215 | #endif 216 | 217 | Iter1 iter = start; 218 | size_t start_bit = 0; 219 | bool has_backlog = false; 220 | char backlog = 0; 221 | 222 | while (has_backlog || iter != end) { 223 | if (!has_backlog) { 224 | if (start_bit + ConversionTraits::group_length() < 8) { 225 | // the value fits within single byte, so we can extract it 226 | // directly 227 | char v = extract_partial_bits(*iter, start_bit, ConversionTraits::group_length()); 228 | *out++ = ConversionTraits::encode(v); 229 | // since we know that start_bit + ConversionTraits::group_length() < 8 we don't need to go 230 | // to the next byte 231 | start_bit += ConversionTraits::group_length(); 232 | } else { 233 | // our bits are spanning across byte border; we need to keep the 234 | // starting point and move over to next byte. 235 | backlog = *iter++; 236 | has_backlog = true; 237 | } 238 | } else { 239 | // encode value which is made from bits spanning across byte 240 | // boundary 241 | char v; 242 | if (iter == end) 243 | v = extract_overlapping_bits(backlog, 0, start_bit, ConversionTraits::group_length()); 244 | else 245 | v = extract_overlapping_bits(backlog, *iter, start_bit, ConversionTraits::group_length()); 246 | *out++ = ConversionTraits::encode(v); 247 | has_backlog = false; 248 | start_bit = (start_bit + ConversionTraits::group_length()) % 8; 249 | } 250 | } 251 | } 252 | 253 | } // impl 254 | 255 | using namespace bn::impl; 256 | 257 | template 258 | void encode_b16(Iter1 start, Iter1 end, Iter2 out) 259 | { 260 | encode(start, end, out); 261 | } 262 | 263 | template 264 | void encode_b32(Iter1 start, Iter1 end, Iter2 out) 265 | { 266 | encode(start, end, out); 267 | } 268 | 269 | template 270 | void encode_b64(Iter1 start, Iter1 end, Iter2 out) 271 | { 272 | encode(start, end, out); 273 | } 274 | 275 | template 276 | void decode_b16(Iter1 start, Iter1 end, Iter2 out) 277 | { 278 | decode(start, end, out); 279 | } 280 | 281 | template 282 | void decode_b32(Iter1 start, Iter1 end, Iter2 out) 283 | { 284 | decode(start, end, out); 285 | } 286 | 287 | template 288 | void decode_b64(Iter1 start, Iter1 end, Iter2 out) 289 | { 290 | decode(start, end, out); 291 | } 292 | 293 | } // bn 294 | 295 | #endif // BASEN_HPP 296 | -------------------------------------------------------------------------------- /keeper/bocu1.cpp: -------------------------------------------------------------------------------- 1 | /* 2 | ****************************************************************************** 3 | * 4 | * Copyright (C) 2002, International Business Machines 5 | * Corporation and others. All Rights Reserved. 6 | * 7 | * For licensing terms see the ICU X License: 8 | * http://oss.software.ibm.com/cvs/icu/~checkout~/icu/license.html 9 | * 10 | ****************************************************************************** 11 | * file name: bocu1.c 12 | * encoding: US-ASCII 13 | * tab size: 8 (not used) 14 | * indentation:4 15 | * 16 | * created on: 2002jan24 17 | * created by: Markus W. Scherer 18 | * 19 | * This is a sample implementation of encoder and decoder functions for BOCU-1, 20 | * a MIME-compatible Binary Ordered Compression for Unicode. 21 | */ 22 | #include "stdafx.h" 23 | #include 24 | #include 25 | 26 | /* 27 | * Standard ICU header. 28 | * - Includes inttypes.h or defines its types. 29 | * - Defines UChar for UTF-16 as an unsigned 16-bit type (wchar_t or uint16_t). 30 | * - Defines UTF* macros to handle reading and writing 31 | * of in-process UTF-8/16 strings. 32 | */ 33 | //#include "unicode/utypes.h" 34 | 35 | #include "bocu1.h" 36 | 37 | using namespace BOCU1; 38 | 39 | /* BOCU-1 implementation functions ------------------------------------------ */ 40 | 41 | /** 42 | * Compute the next "previous" value for differencing 43 | * from the current code point. 44 | * 45 | * @param c current code point, 0..0x10ffff 46 | * @return "previous code point" state value 47 | */ 48 | int32_t bocu1Prev(int32_t c) 49 | { 50 | /* compute new prev */ 51 | if(0x3040<=c && c<=0x309f) { 52 | /* Hiragana is not 128-aligned */ 53 | return 0x3070; 54 | } else if(0x4e00<=c && c<=0x9fa5) { 55 | /* CJK Unihan */ 56 | return 0x4e00-BOCU1_REACH_NEG_2; 57 | } else if(0xac00<=c && c<=0xd7a3) { 58 | /* Korean Hangul */ 59 | return (0xd7a3+0xac00)/2; 60 | } else { 61 | /* mostly small scripts */ 62 | return (c&~0x7f)+BOCU1_ASCII_PREV; 63 | } 64 | } 65 | 66 | /** 67 | * Encode a difference -0x10ffff..0x10ffff in 1..4 bytes 68 | * and return a packed integer with them. 69 | * 70 | * The encoding favors small absolut differences with short encodings 71 | * to compress runs of same-script characters. 72 | * 73 | * @param diff difference value -0x10ffff..0x10ffff 74 | * @return 75 | * 0x010000zz for 1-byte sequence zz 76 | * 0x0200yyzz for 2-byte sequence yy zz 77 | * 0x03xxyyzz for 3-byte sequence xx yy zz 78 | * 0xwwxxyyzz for 4-byte sequence ww xx yy zz (ww>0x03) 79 | */ 80 | /*U_CFUNC*/ int32_t BOCU1::packDiff(int32_t diff) 81 | { 82 | int32_t result, m, lead, count, shift; 83 | 84 | if(diff>=BOCU1_REACH_NEG_1) { 85 | /* mostly positive differences, and single-byte negative ones */ 86 | if(diff<=BOCU1_REACH_POS_1) { 87 | /* single byte */ 88 | return 0x01000000|(BOCU1_MIDDLE+diff); 89 | } else if(diff<=BOCU1_REACH_POS_2) { 90 | /* two bytes */ 91 | diff-=BOCU1_REACH_POS_1+1; 92 | lead=BOCU1_START_POS_2; 93 | count=1; 94 | } else if(diff<=BOCU1_REACH_POS_3) { 95 | /* three bytes */ 96 | diff-=BOCU1_REACH_POS_2+1; 97 | lead=BOCU1_START_POS_3; 98 | count=2; 99 | } else { 100 | /* four bytes */ 101 | diff-=BOCU1_REACH_POS_3+1; 102 | lead=BOCU1_START_POS_4; 103 | count=3; 104 | } 105 | } else { 106 | /* two- and four-byte negative differences */ 107 | if(diff>=BOCU1_REACH_NEG_2) { 108 | /* two bytes */ 109 | diff-=BOCU1_REACH_NEG_1; 110 | lead=BOCU1_START_NEG_2; 111 | count=1; 112 | } else if(diff>=BOCU1_REACH_NEG_3) { 113 | /* three bytes */ 114 | diff-=BOCU1_REACH_NEG_2; 115 | lead=BOCU1_START_NEG_3; 116 | count=2; 117 | } else { 118 | /* four bytes */ 119 | diff-=BOCU1_REACH_NEG_3; 120 | lead=BOCU1_START_NEG_4; 121 | count=3; 122 | } 123 | } 124 | 125 | /* encode the length of the packed result */ 126 | if(count<3) { 127 | result=(count+1)<<24; 128 | } else /* count==3, MSB used for the lead byte */ { 129 | result=0; 130 | } 131 | 132 | /* calculate trail bytes like digits in itoa() */ 133 | shift=0; 134 | do { 135 | NEGDIVMOD(diff, BOCU1_TRAIL_COUNT, m); 136 | result|=BOCU1_TRAIL_TO_BYTE(m)<0); 139 | 140 | /* add lead byte */ 141 | result|=(lead+diff)<0x10ffff) { 164 | /* illegal argument */ 165 | return 0; 166 | } 167 | 168 | prev=*pPrev; 169 | if(prev==0) { 170 | /* lenient handling of initial value 0 */ 171 | prev=*pPrev=BOCU1_ASCII_PREV; 172 | } 173 | 174 | if(c<=0x20) { 175 | /* 176 | * ISO C0 control & space: 177 | * Encode directly for MIME compatibility, 178 | * and reset state except for space, to not disrupt compression. 179 | */ 180 | if(c!=0x20) { 181 | *pPrev=BOCU1_ASCII_PREV; 182 | } 183 | return 0x01000000|c; 184 | } 185 | 186 | /* 187 | * all other Unicode code points c==U+0021..U+10ffff 188 | * are encoded with the difference c-prev 189 | * 190 | * a new prev is computed from c, 191 | * placed in the middle of a 0x80-block (for most small scripts) or 192 | * in the middle of the Unihan and Hangul blocks 193 | * to statistically minimize the following difference 194 | */ 195 | *pPrev=bocu1Prev(c); 196 | return packDiff(c-prev); 197 | } 198 | 199 | /** 200 | * Function for BOCU-1 decoder; handles multi-byte lead bytes. 201 | * 202 | * @param pRx pointer to the decoder state structure 203 | * @param b lead byte; 204 | * BOCU1_MIN<=b=BOCU1_START_NEG_2) { 213 | /* positive difference */ 214 | if(b=BOCU1_START_NEG_3) { 230 | /* two bytes */ 231 | c=((int32_t)b-BOCU1_START_NEG_2)*BOCU1_TRAIL_COUNT+BOCU1_REACH_NEG_1; 232 | count=1; 233 | } else if(b>BOCU1_MIN) { 234 | /* three bytes */ 235 | c=((int32_t)b-BOCU1_START_NEG_3)*BOCU1_TRAIL_COUNT*BOCU1_TRAIL_COUNT+BOCU1_REACH_NEG_2; 236 | count=2; 237 | } else { 238 | /* four bytes */ 239 | c=-BOCU1_TRAIL_COUNT*BOCU1_TRAIL_COUNT*BOCU1_TRAIL_COUNT+BOCU1_REACH_NEG_3; 240 | count=3; 241 | } 242 | } 243 | 244 | /* set the state for decoding the trail byte(s) */ 245 | pRx->diff=c; 246 | pRx->count=count; 247 | return -1; 248 | } 249 | 250 | /** 251 | * Function for BOCU-1 decoder; handles multi-byte trail bytes. 252 | * 253 | * @param pRx pointer to the decoder state structure 254 | * @param b trail byte 255 | * @return result value, same as decodeBocu1 256 | * 257 | * @see decodeBocu1 258 | */ 259 | int32_t decodeBocu1TrailByte(Bocu1Rx *pRx, uint8_t b) { 260 | int32_t t, c, count; 261 | 262 | if(b<=0x20) { 263 | /* skip some C0 controls and make the trail byte range contiguous */ 264 | t=bocu1ByteToTrail[b]; 265 | if(t<0) { 266 | /* illegal trail byte value */ 267 | pRx->prev=BOCU1_ASCII_PREV; 268 | pRx->count=0; 269 | return -99; 270 | } 271 | #if BOCU1_MAX_TRAIL<0xff 272 | } else if(b>BOCU1_MAX_TRAIL) { 273 | return -99; 274 | #endif 275 | } else { 276 | t=(int32_t)b-BOCU1_TRAIL_BYTE_OFFSET; 277 | } 278 | 279 | /* add trail byte into difference and decrement count */ 280 | c=pRx->diff; 281 | count=pRx->count; 282 | 283 | if(count==1) { 284 | /* final trail byte, deliver a code point */ 285 | c=pRx->prev+c+t; 286 | if(0<=c && c<=0x10ffff) { 287 | /* valid code point result */ 288 | pRx->prev=bocu1Prev(c); 289 | pRx->count=0; 290 | return c; 291 | } else { 292 | /* illegal code point result */ 293 | pRx->prev=BOCU1_ASCII_PREV; 294 | pRx->count=0; 295 | return -99; 296 | } 297 | } 298 | 299 | /* intermediate trail byte */ 300 | if(count==2) { 301 | pRx->diff=c+t*BOCU1_TRAIL_COUNT; 302 | } else /* count==3 */ { 303 | pRx->diff=c+t*BOCU1_TRAIL_COUNT*BOCU1_TRAIL_COUNT; 304 | } 305 | pRx->count=count-1; 306 | return -1; 307 | } 308 | 309 | /** 310 | * BOCU-1 decoder function. 311 | * 312 | * @param pRx pointer to the decoder state structure; 313 | * the initial values should be 0 which 314 | * decodeBocu1 will set to actual initial state values 315 | * @param b an input byte 316 | * @return 317 | * 0..0x10ffff for a result code point 318 | * -1 if only the state changed without code point output 319 | * <-1 if an error occurs 320 | */ 321 | int32_t BOCU1::decodeBocu1(Bocu1Rx *pRx, uint8_t b) 322 | { 323 | int32_t prev, c, count; 324 | 325 | if(pRx==NULL) { 326 | /* illegal argument */ 327 | return -99; 328 | } 329 | 330 | prev=pRx->prev; 331 | if(prev==0) { 332 | /* lenient handling of initial 0 values */ 333 | prev=pRx->prev=BOCU1_ASCII_PREV; 334 | count=pRx->count=0; 335 | } else { 336 | count=pRx->count; 337 | } 338 | 339 | if(count==0) { 340 | /* byte in lead position */ 341 | if(b<=0x20) { 342 | /* 343 | * Direct-encoded C0 control code or space. 344 | * Reset prev for C0 control codes but not for space. 345 | */ 346 | if(b!=0x20) { 347 | pRx->prev=BOCU1_ASCII_PREV; 348 | } 349 | return b; 350 | } 351 | 352 | /* 353 | * b is a difference lead byte. 354 | * 355 | * Return a code point directly from a single-byte difference. 356 | * 357 | * For multi-byte difference lead bytes, set the decoder state 358 | * with the partial difference value from the lead byte and 359 | * with the number of trail bytes. 360 | * 361 | * For four-byte differences, the signedness also affects the 362 | * first trail byte, which has special handling farther below. 363 | */ 364 | if(b>=BOCU1_START_NEG_2 && bprev=bocu1Prev(c); 368 | return c; 369 | } else if(b==BOCU1_RESET) { 370 | /* only reset the state, no code point */ 371 | pRx->prev=BOCU1_ASCII_PREV; 372 | return -1; 373 | } else { 374 | return decodeBocu1LeadByte(pRx, b); 375 | } 376 | } else { 377 | /* trail byte in any position */ 378 | return decodeBocu1TrailByte(pRx, b); 379 | } 380 | } 381 | 382 | /** 383 | * \def UPRV_BLOCK_MACRO_BEGIN 384 | * Defined as the "do" keyword by default. 385 | * @internal 386 | */ 387 | #ifndef UPRV_BLOCK_MACRO_BEGIN 388 | #define UPRV_BLOCK_MACRO_BEGIN do 389 | #endif 390 | 391 | /** 392 | * \def UPRV_BLOCK_MACRO_END 393 | * Defined as "while (FALSE)" by default. 394 | * @internal 395 | */ 396 | #ifndef UPRV_BLOCK_MACRO_END 397 | #define UPRV_BLOCK_MACRO_END while (FALSE) 398 | #endif 399 | 400 | #define U16_NEXT(s, i, length, c) UPRV_BLOCK_MACRO_BEGIN { \ 401 | (c)=(s)[(i)++]; \ 402 | if(U16_IS_LEAD(c)) { \ 403 | uint16_t __c2; \ 404 | if((i)!=(length) && U16_IS_TRAIL(__c2=(s)[(i)])) { \ 405 | ++(i); \ 406 | (c)=U16_GET_SUPPLEMENTARY((c), __c2); \ 407 | } \ 408 | } \ 409 | } UPRV_BLOCK_MACRO_END 410 | 411 | /** 412 | * Is this code unit a lead surrogate (U+d800..U+dbff)? 413 | * @param c 16-bit code unit 414 | * @return TRUE or FALSE 415 | * @stable ICU 2.4 416 | */ 417 | #define U16_IS_LEAD(c) (((c)&0xfffffc00)==0xd800) 418 | 419 | /** 420 | * Is this code unit a trail surrogate (U+dc00..U+dfff)? 421 | * @param c 16-bit code unit 422 | * @return TRUE or FALSE 423 | * @stable ICU 2.4 424 | */ 425 | #define U16_IS_TRAIL(c) (((c)&0xfffffc00)==0xdc00) 426 | 427 | typedef int32_t UChar32; 428 | 429 | /** 430 | * Helper constant for U16_GET_SUPPLEMENTARY. 431 | * @internal 432 | */ 433 | #define U16_SURROGATE_OFFSET ((0xd800<<10UL)+0xdc00-0x10000) 434 | 435 | /** 436 | * Get a supplementary code point value (U+10000..U+10ffff) 437 | * from its lead and trail surrogates. 438 | * The result is undefined if the input values are not 439 | * lead and trail surrogates. 440 | * 441 | * @param lead lead surrogate (U+d800..U+dbff) 442 | * @param trail trail surrogate (U+dc00..U+dfff) 443 | * @return supplementary code point (U+10000..U+10ffff) 444 | * @stable ICU 2.4 445 | */ 446 | #define U16_GET_SUPPLEMENTARY(lead, trail) \ 447 | (((UChar32)(lead)<<10UL)+(UChar32)(trail)-U16_SURROGATE_OFFSET) 448 | 449 | /** 450 | * Append a code point to a string, overwriting 1 or 2 code units. 451 | * The offset points to the current end of the string contents 452 | * and is advanced (post-increment). 453 | * "Unsafe" macro, assumes a valid code point and sufficient space in the string. 454 | * Otherwise, the result is undefined. 455 | * 456 | * @param s const UChar * string buffer 457 | * @param i string offset 458 | * @param c code point to append 459 | * @see U16_APPEND 460 | * @stable ICU 2.4 461 | */ 462 | #define U16_APPEND_UNSAFE(s, i, c) UPRV_BLOCK_MACRO_BEGIN { \ 463 | if((uint32_t)(c)<=0xffff) { \ 464 | (s)[(i)++]=(uint16_t)(c); \ 465 | } else { \ 466 | (s)[(i)++]=(uint16_t)(((c)>>10)+0xd7c0); \ 467 | (s)[(i)++]=(uint16_t)(((c)&0x3ff)|0xdc00); \ 468 | } \ 469 | } UPRV_BLOCK_MACRO_END 470 | 471 | /** 472 | * Write a packed BOCU-1 byte sequence into a byte array, 473 | * without overflow check. 474 | * Test function. 475 | * 476 | * @param packed packed BOCU-1 byte sequence, see packDiff() 477 | * @param p pointer to byte array 478 | * @return number of bytes 479 | * 480 | * @see packDiff 481 | */ 482 | int32_t writePacked(int32_t packed, uint8_t *p) 483 | { 484 | int32_t count = BOCU1_LENGTH_FROM_PACKED(packed); 485 | switch (count) { 486 | case 4: 487 | *p++ = (uint8_t)(packed >> 24); 488 | case 3: 489 | *p++ = (uint8_t)(packed >> 16); 490 | case 2: 491 | *p++ = (uint8_t)(packed >> 8); 492 | case 1: 493 | *p++ = (uint8_t)packed; 494 | default: 495 | break; 496 | } 497 | 498 | return count; 499 | } 500 | 501 | /** 502 | * Encode a UTF-16 string in BOCU-1. 503 | * Does not check for overflows, but otherwise useful function. 504 | * 505 | * @param s input UTF-16 string 506 | * @param length number of UChar code units in s 507 | * @param p pointer to output byte array 508 | * @return number of bytes output 509 | */ 510 | int32_t BOCU1::writeString(const wchar_t *s, int32_t length, uint8_t *p) { 511 | uint8_t *p0; 512 | int32_t c, prev, i; 513 | 514 | prev = 0; 515 | p0 = p; 516 | i = 0; 517 | while (i < length) { 518 | U16_NEXT(s, i, length, c); 519 | p += writePacked(encodeBocu1(&prev, c), p); 520 | } 521 | return (int32_t)(p - p0); 522 | } 523 | 524 | /** 525 | * Decode a BOCU-1 byte sequence to a UTF-16 string. 526 | * Does not check for overflows, but otherwise useful function. 527 | * 528 | * @param p pointer to input BOCU-1 bytes 529 | * @param length number of input bytes 530 | * @param s point to output UTF-16 string array 531 | * @return number of UChar code units output 532 | */ 533 | int32_t BOCU1::readString(const uint8_t *p, int32_t length, wchar_t *s) 534 | { 535 | Bocu1Rx rx = { 0, 0, 0 }; 536 | int32_t c, i, sLength; 537 | 538 | i = sLength = 0; 539 | while (i < length) { 540 | c = decodeBocu1(&rx, p[i++]); 541 | if (c < -1) { 542 | //log_err("error: readString detects encoding error at string index %ld\n", i); 543 | throw std::runtime_error("Error while decoding BOCU-1 string"); 544 | //return -1; 545 | } 546 | if (c >= 0) { 547 | U16_APPEND_UNSAFE(s, sLength, c); 548 | } 549 | } 550 | return sLength; 551 | } 552 | -------------------------------------------------------------------------------- /keeper/bocu1.h: -------------------------------------------------------------------------------- 1 | /* 2 | ****************************************************************************** 3 | * 4 | * Copyright (C) 2002, International Business Machines 5 | * Corporation and others. All Rights Reserved. 6 | * 7 | * For licensing terms see the ICU X License: 8 | * http://oss.software.ibm.com/cvs/icu/~checkout~/icu/license.html 9 | * 10 | ****************************************************************************** 11 | * file name: bocu1.h 12 | * encoding: US-ASCII 13 | * tab size: 8 (not used) 14 | * indentation:4 15 | * 16 | * created on: 2002jan24 17 | * created by: Markus W. Scherer 18 | * 19 | * This is the definition file for the sample implementation of BOCU-1, 20 | * a MIME-compatible Binary Ordered Compression for Unicode. 21 | */ 22 | 23 | #ifndef __BOCU1_H__ 24 | #define __BOCU1_H__ 25 | 26 | /* 27 | * Standard ICU header. 28 | * - Includes inttypes.h or defines its types. 29 | * - Defines UChar for UTF-16 as an unsigned 16-bit type (wchar_t or uint16_t). 30 | * - Defines UTF* macros to handle reading and writing 31 | * of in-process UTF-8/16 strings. 32 | */ 33 | //#include "unicode/utypes.h" 34 | #include 35 | 36 | namespace BOCU1 37 | { 38 | /* BOCU-1 constants and macros ---------------------------------------------- */ 39 | 40 | /* 41 | * BOCU-1 encodes the code points of a Unicode string as 42 | * a sequence of byte-encoded differences (slope detection), 43 | * preserving lexical order. 44 | * 45 | * Optimize the difference-taking for runs of Unicode text within 46 | * small scripts: 47 | * 48 | * Most small scripts are allocated within aligned 128-blocks of Unicode 49 | * code points. Lexical order is preserved if the "previous code point" state 50 | * is always moved into the middle of such a block. 51 | * 52 | * Additionally, "prev" is moved from anywhere in the Unihan and Hangul 53 | * areas into the middle of those areas. 54 | * 55 | * C0 control codes and space are encoded with their US-ASCII bytes. 56 | * "prev" is reset for C0 controls but not for space. 57 | */ 58 | 59 | /* initial value for "prev": middle of the ASCII range */ 60 | #define BOCU1_ASCII_PREV 0x40 61 | 62 | /* bounding byte values for differences */ 63 | #define BOCU1_MIN 0x21 64 | #define BOCU1_MIDDLE 0x90 65 | #define BOCU1_MAX_LEAD 0xfe 66 | #define BOCU1_MAX_TRAIL 0xff 67 | #define BOCU1_RESET 0xff 68 | 69 | /* number of lead bytes */ 70 | #define BOCU1_COUNT (BOCU1_MAX_LEAD-BOCU1_MIN+1) 71 | 72 | /* adjust trail byte counts for the use of some C0 control byte values */ 73 | #define BOCU1_TRAIL_CONTROLS_COUNT 20 74 | #define BOCU1_TRAIL_BYTE_OFFSET (BOCU1_MIN-BOCU1_TRAIL_CONTROLS_COUNT) 75 | 76 | /* number of trail bytes */ 77 | #define BOCU1_TRAIL_COUNT ((BOCU1_MAX_TRAIL-BOCU1_MIN+1)+BOCU1_TRAIL_CONTROLS_COUNT) 78 | 79 | /* 80 | * number of positive and negative single-byte codes 81 | * (counting 0==BOCU1_MIDDLE among the positive ones) 82 | */ 83 | #define BOCU1_SINGLE 64 84 | 85 | /* number of lead bytes for positive and negative 2/3/4-byte sequences */ 86 | #define BOCU1_LEAD_2 43 87 | #define BOCU1_LEAD_3 3 88 | #define BOCU1_LEAD_4 1 89 | 90 | /* The difference value range for single-byters. */ 91 | #define BOCU1_REACH_POS_1 (BOCU1_SINGLE-1) 92 | #define BOCU1_REACH_NEG_1 (-BOCU1_SINGLE) 93 | 94 | /* The difference value range for double-byters. */ 95 | #define BOCU1_REACH_POS_2 (BOCU1_REACH_POS_1+BOCU1_LEAD_2*BOCU1_TRAIL_COUNT) 96 | #define BOCU1_REACH_NEG_2 (BOCU1_REACH_NEG_1-BOCU1_LEAD_2*BOCU1_TRAIL_COUNT) 97 | 98 | /* The difference value range for 3-byters. */ 99 | #define BOCU1_REACH_POS_3 \ 100 | (BOCU1_REACH_POS_2+BOCU1_LEAD_3*BOCU1_TRAIL_COUNT*BOCU1_TRAIL_COUNT) 101 | 102 | #define BOCU1_REACH_NEG_3 (BOCU1_REACH_NEG_2-BOCU1_LEAD_3*BOCU1_TRAIL_COUNT*BOCU1_TRAIL_COUNT) 103 | 104 | /* The lead byte start values. */ 105 | #define BOCU1_START_POS_2 (BOCU1_MIDDLE+BOCU1_REACH_POS_1+1) 106 | #define BOCU1_START_POS_3 (BOCU1_START_POS_2+BOCU1_LEAD_2) 107 | #define BOCU1_START_POS_4 (BOCU1_START_POS_3+BOCU1_LEAD_3) 108 | /* ==BOCU1_MAX_LEAD */ 109 | 110 | #define BOCU1_START_NEG_2 (BOCU1_MIDDLE+BOCU1_REACH_NEG_1) 111 | #define BOCU1_START_NEG_3 (BOCU1_START_NEG_2-BOCU1_LEAD_2) 112 | #define BOCU1_START_NEG_4 (BOCU1_START_NEG_3-BOCU1_LEAD_3) 113 | /* ==BOCU1_MIN+1 */ 114 | 115 | /* The length of a byte sequence, according to the lead byte (!=BOCU1_RESET). */ 116 | #define BOCU1_LENGTH_FROM_LEAD(lead) \ 117 | ((BOCU1_START_NEG_2<=(lead) && (lead)>24 : 4) 124 | 125 | /* 126 | * 12 commonly used C0 control codes (and space) are only used to encode 127 | * themselves directly, 128 | * which makes BOCU-1 MIME-usable and reasonably safe for 129 | * ASCII-oriented software. 130 | * 131 | * These controls are 132 | * 0 NUL 133 | * 134 | * 7 BEL 135 | * 8 BS 136 | * 137 | * 9 TAB 138 | * a LF 139 | * b VT 140 | * c FF 141 | * d CR 142 | * 143 | * e SO 144 | * f SI 145 | * 146 | * 1a SUB 147 | * 1b ESC 148 | * 149 | * The other 20 C0 controls are also encoded directly (to preserve order) 150 | * but are also used as trail bytes in difference encoding 151 | * (for better compression). 152 | */ 153 | #define BOCU1_TRAIL_TO_BYTE(t) ((t)>=BOCU1_TRAIL_CONTROLS_COUNT ? (t)+BOCU1_TRAIL_BYTE_OFFSET : bocu1TrailToByte[t]) 154 | 155 | /* 156 | * Byte value map for control codes, 157 | * from external byte values 0x00..0x20 158 | * to trail byte values 0..19 (0..0x13) as used in the difference calculation. 159 | * External byte values that are illegal as trail bytes are mapped to -1. 160 | */ 161 | static int8_t 162 | bocu1ByteToTrail[BOCU1_MIN] = { 163 | /* 0 1 2 3 4 5 6 7 */ 164 | -1, 0x00, 0x01, 0x02, 0x03, 0x04, 0x05, -1, 165 | 166 | /* 8 9 a b c d e f */ 167 | -1, -1, -1, -1, -1, -1, -1, -1, 168 | 169 | /* 10 11 12 13 14 15 16 17 */ 170 | 0x06, 0x07, 0x08, 0x09, 0x0a, 0x0b, 0x0c, 0x0d, 171 | 172 | /* 18 19 1a 1b 1c 1d 1e 1f */ 173 | 0x0e, 0x0f, -1, -1, 0x10, 0x11, 0x12, 0x13, 174 | 175 | /* 20 */ 176 | -1 177 | }; 178 | 179 | /* 180 | * Byte value map for control codes, 181 | * from trail byte values 0..19 (0..0x13) as used in the difference calculation 182 | * to external byte values 0x00..0x20. 183 | */ 184 | static int8_t 185 | bocu1TrailToByte[BOCU1_TRAIL_CONTROLS_COUNT] = { 186 | /* 0 1 2 3 4 5 6 7 */ 187 | 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x10, 0x11, 188 | 189 | /* 8 9 a b c d e f */ 190 | 0x12, 0x13, 0x14, 0x15, 0x16, 0x17, 0x18, 0x19, 191 | 192 | /* 10 11 12 13 */ 193 | 0x1c, 0x1d, 0x1e, 0x1f 194 | }; 195 | 196 | /** 197 | * Integer division and modulo with negative numerators 198 | * yields negative modulo results and quotients that are one more than 199 | * what we need here. 200 | * This macro adjust the results so that the modulo-value m is always >=0. 201 | * 202 | * For positive n, the if() condition is always FALSE. 203 | * 204 | * @param n Number to be split into quotient and rest. 205 | * Will be modified to contain the quotient. 206 | * @param d Divisor. 207 | * @param m Output variable for the rest (modulo result). 208 | */ 209 | #define NEGDIVMOD(n, d, m) { \ 210 | (m)=(n)%(d); \ 211 | (n)/=(d); \ 212 | if((m)<0) { \ 213 | --(n); \ 214 | (m)+=(d); \ 215 | } \ 216 | } 217 | 218 | /* State for BOCU-1 decoder function. */ 219 | struct Bocu1Rx 220 | { 221 | int32_t prev, count, diff; 222 | }; 223 | 224 | //typedef struct Bocu1Rx Bocu1Rx; 225 | 226 | int32_t packDiff(int32_t diff); 227 | 228 | int32_t encodeBocu1(int32_t *pPrev, int32_t c); 229 | 230 | int32_t decodeBocu1(Bocu1Rx *pRx, uint8_t b); 231 | 232 | int32_t writeString(const wchar_t *s, int32_t length, uint8_t *p); 233 | 234 | int32_t readString(const uint8_t *p, int32_t length, wchar_t *s); 235 | } 236 | #endif 237 | -------------------------------------------------------------------------------- /keeper/fpaq.cpp: -------------------------------------------------------------------------------- 1 | #include "stdafx.h" 2 | #include "fpaq.h" 3 | 4 | using namespace fpaq; 5 | 6 | int StateMap::dt[256] = { 0 }; 7 | 8 | StateMap::StateMap(int n) : N(n), cxt(0) 9 | { 10 | //alloc(t, N); 11 | t.resize(N, 0); 12 | for (int i = 0; i < N; ++i) 13 | { 14 | // Count 1 bits to determine initial probability. 15 | U32 n2 = (i & 1) * 2 + (i & 2) + (i >> 2 & 1) + (i >> 3 & 1) + (i >> 4 & 1) + (i >> 5 & 1) + (i >> 6 & 1) + (i >> 7 & 1) + 3; 16 | t[i] = n2 << 28 | 6; 17 | } 18 | if (dt[0] == 0) 19 | for (int i = 0; i < 256; ++i) 20 | dt[i] = 32768 / (i + i + 3); 21 | } 22 | 23 | // Update the model with bit y (0..1). 24 | // limit (1..255) controls the rate of adaptation (higher = slower) 25 | void StateMap::update(int y, int limit) 26 | { 27 | assert(cxt >= 0 && cxt < N); 28 | assert(y == 0 || y == 1); 29 | assert(limit >= 0 && limit < 255); 30 | int n = t[cxt] & 255, p = t[cxt] >> 14; // count, prediction 31 | if (n < limit) ++t[cxt]; 32 | t[cxt] += ((y << 18) - p)*dt[n] & 0xffffff00; 33 | } 34 | 35 | int StateMap::p(int cx) 36 | { 37 | assert(cx >= 0 && cx < N); 38 | return t[cxt = cx] >> 16; 39 | } 40 | 41 | Predictor::Predictor() : cxt(0), sm(0x10000) 42 | { 43 | for (int i = 0; i < 0x100; ++i) 44 | state[i] = 0x66; 45 | } 46 | 47 | void Predictor::update(int y) 48 | { 49 | sm.update(y, 90); 50 | int& st = state[cxt]; 51 | (st += st + y) &= 255; 52 | if ((cxt += cxt + y) >= 256) 53 | cxt = 0; 54 | } 55 | int Predictor::p() 56 | { 57 | return sm.p(cxt << 8 | state[cxt]); 58 | } 59 | 60 | // Constructor 61 | template 62 | Encoder::Encoder(Mode m, Iter1& start, Iter1& end, Iter2& out) : predictor(), mode(m), /*archive(f),*/ x1(0), 63 | x2(0xffffffff), x(0), start_(start), end_(end), out_(out) 64 | { 65 | // In DECOMPRESS mode, initialize x to the first 4 bytes of the archive 66 | if (mode == Mode::DECOMPRESS) 67 | { 68 | int c = 0; 69 | for (int i = 0; i < 4; ++i) 70 | { 71 | if (start_ != end_) 72 | c = *start_++; 73 | else 74 | c = 0; 75 | x = (x << 8) + (c & 0xff); 76 | } 77 | } 78 | } 79 | 80 | // encode(y) -- Encode bit y by splitting the range [x1, x2] in proportion 81 | // to P(1) and P(0) as given by the predictor and narrowing to the appropriate 82 | // subrange. Output leading bytes of the range as they become known. 83 | template 84 | inline void Encoder::encode(int y) 85 | { 86 | // Update the range 87 | const U32 p = predictor.p(); 88 | assert(p <= 0xffff); 89 | assert(y == 0 || y == 1); 90 | const U32 xmid = x1 + (x2 - x1 >> 16)*p + ((x2 - x1 & 0xffff)*p >> 16); 91 | assert(xmid >= x1 && xmid < x2); 92 | if (y) 93 | x2 = xmid; 94 | else 95 | x1 = xmid + 1; 96 | predictor.update(y); 97 | 98 | // Shift equal MSB's out 99 | while (((x1^x2) & 0xff000000) == 0) 100 | { 101 | *out_++ = x2 >> 24; 102 | x1 <<= 8; 103 | x2 = (x2 << 8) + 255; 104 | } 105 | } 106 | 107 | // Decode one bit from the archive, splitting [x1, x2] as in the encoder 108 | // and returning 1 or 0 depending on which subrange the archive point x is in. 109 | template 110 | inline int Encoder::decode() 111 | { 112 | 113 | // Update the range 114 | const U32 p = predictor.p(); 115 | assert(p <= 0xffff); 116 | const U32 xmid = x1 + (x2 - x1 >> 16)*p + ((x2 - x1 & 0xffff)*p >> 16); 117 | assert(xmid >= x1 && xmid < x2); 118 | int y = 0; 119 | if (x <= xmid) 120 | { 121 | y = 1; 122 | x2 = xmid; 123 | } 124 | else 125 | x1 = xmid + 1; 126 | predictor.update(y); 127 | 128 | // Shift equal MSB's out 129 | while (((x1^x2) & 0xff000000) == 0) 130 | { 131 | x1 <<= 8; 132 | x2 = (x2 << 8) + 255; 133 | int c; 134 | if (start_ != end_) 135 | c = *start_++; 136 | else 137 | c = 0; 138 | x = (x << 8) + c; 139 | } 140 | return y; 141 | } 142 | 143 | // Should be called when there is no more to compress. 144 | template 145 | void Encoder::flush() { 146 | 147 | // In COMPRESS mode, write out the remaining bytes of x, x1 < x < x2 148 | if (mode == Mode::COMPRESS) { 149 | while (((x1^x2) & 0xff000000) == 0) 150 | { 151 | *out_++ = x2 >> 24; 152 | x1 <<= 8; 153 | x2 = (x2 << 8) + 255; 154 | } 155 | *out_++ = x2 >> 24; // First unequal byte 156 | } 157 | } 158 | 159 | size_t fpaq::fpaqCompress(byte* start, size_t count, byte* out) 160 | { 161 | byte* start_ = start; 162 | byte* end_ = start + count; 163 | byte* out_ = out; 164 | 165 | fpaq::Encoder encoder(fpaq::Mode::COMPRESS, start_, end_, out_); 166 | while (start_ != end_) 167 | { 168 | encoder.encode(1); 169 | byte c = *start_++; 170 | for (int i = 7; i >= 0; --i) 171 | encoder.encode((c >> i) & 1); 172 | } 173 | encoder.encode(0); 174 | encoder.flush(); 175 | 176 | return out_ - out; 177 | } 178 | 179 | size_t fpaq::fpaqDecompress(byte* start, size_t count, byte* out) 180 | { 181 | byte* start_ = start; 182 | byte* end_ = start + count; 183 | byte* out_ = out; 184 | 185 | fpaq::Encoder encoder(fpaq::Mode::DECOMPRESS, start_, end_, out_); 186 | 187 | while (encoder.decode()) 188 | { 189 | int c = 1; 190 | while (c < 256) 191 | c += c + encoder.decode(); 192 | *out_++ = c - 256; 193 | } 194 | 195 | return out_ - out; 196 | } 197 | -------------------------------------------------------------------------------- /keeper/fpaq.h: -------------------------------------------------------------------------------- 1 | #pragma once 2 | 3 | //fpaq0f2 - Adaptive order 0 compressor. 4 | //originally implemented by Matt Mahoney, 2008 5 | #include 6 | #include 7 | 8 | namespace fpaq 9 | { 10 | typedef unsigned char U8; 11 | typedef unsigned short U16; 12 | typedef unsigned int U32; 13 | 14 | // A StateMap maps a context to a probability. After a bit prediction, the 15 | // map is updated in the direction of the actual value to improve future 16 | // predictions in the same context. 17 | 18 | class StateMap 19 | { 20 | protected: 21 | const int N; // Number of contexts 22 | int cxt; // Context of last prediction 23 | //U32 *t; // cxt -> prediction in high 24 bits, count in low 8 bits 24 | std::vector t; 25 | static int dt[256]; // reciprocal table: i -> 16K/(i+1.5) 26 | public: 27 | StateMap(int n = 256); // create allowing n contexts 28 | 29 | // Predict next bit to be updated in context cx (0..n-1). 30 | // Return prediction as a 16 bit number (0..65535) that next bit is 1. 31 | int p(int cx); 32 | 33 | // Update the model with bit y (0..1). 34 | // limit (1..255) controls the rate of adaptation (higher = slower) 35 | void update(int y, int limit = 255); 36 | }; 37 | 38 | 39 | /* A Predictor estimates the probability that the next bit of 40 | uncompressed data is 1. Methods: 41 | p() returns P(1) as a 16 bit number (0..65535). 42 | update(y) trains the predictor with the actual bit (0 or 1). 43 | */ 44 | class Predictor 45 | { 46 | int cxt; // Context: 0=not EOF, 1..255=last 0-7 bits with a leading 1 47 | StateMap sm; 48 | int state[256]; 49 | public: 50 | Predictor(); 51 | // Assume order 0 stream of 9-bit symbols 52 | int p(); 53 | void update(int y); 54 | }; 55 | 56 | /* An Encoder does arithmetic encoding. Methods: 57 | Encoder(COMPRESS, f) creates encoder for compression to archive f, which 58 | must be open past any header for writing in binary mode 59 | Encoder(DECOMPRESS, f) creates encoder for decompression from archive f, 60 | which must be open past any header for reading in binary mode 61 | encode(bit) in COMPRESS mode compresses bit to file f. 62 | decode() in DECOMPRESS mode returns the next decompressed bit from file f. 63 | flush() should be called when there is no more to compress. 64 | */ 65 | 66 | enum class Mode 67 | { 68 | COMPRESS, 69 | DECOMPRESS 70 | }; 71 | 72 | template 73 | class Encoder 74 | { 75 | private: 76 | Predictor predictor; // Computes next bit probability (0..65535) 77 | const Mode mode; // Compress or decompress? 78 | U32 x1, x2; // Range, initially [0, 1), scaled by 2^32 79 | U32 x; // Last 4 input bytes of archive. 80 | //yeah, it is bad and i'm feel bad 81 | Iter1& start_; 82 | const Iter1& end_; 83 | Iter2& out_; 84 | public: 85 | Encoder(Mode m, Iter1& start, Iter1& end, Iter2& out); 86 | void encode(int y); // Compress bit y 87 | int decode(); // Uncompress and return bit y 88 | void flush(); // Call when done compressing 89 | }; 90 | 91 | size_t fpaqCompress(byte* start, size_t count, byte* out); 92 | size_t fpaqDecompress(byte* start, size_t count, byte* out); 93 | } 94 | -------------------------------------------------------------------------------- /keeper/keeper.cpp: -------------------------------------------------------------------------------- 1 | #include "stdafx.h" 2 | #include "TaskContext.h" 3 | #include "CLIparser.h" 4 | #include "TaskBackup.h" 5 | #include "TaskDumpDB.h" 6 | #include "TaskRestore.h" 7 | #include "TaskPurge.h" 8 | #include "GlobalUtils.h" 9 | 10 | using namespace std; 11 | using namespace ConsoleLogger; 12 | using namespace keeper; 13 | 14 | keeper::TaskContext taskCtx; 15 | 16 | //for the emergency DB flush 17 | DbEnv* BuroEnv = nullptr; 18 | BOOL WINAPI CtrlHandler(DWORD fdwCtrlType); 19 | #ifdef DOCTEST_CONFIG_DISABLE 20 | int wmain(int argc, wchar_t *argv[]) 21 | { 22 | SetLogLevel(ConsoleLogger::LogLevel::info); 23 | 24 | LOG_INFO() << MESSAGE_ABOUT << endl; 25 | LOG_INFO() << "This software comes with absolutely no warranty" << endl; 26 | 27 | if (!ParseCLITask(taskCtx, argc, argv)) 28 | exit(-1); 29 | 30 | sodium_init(); 31 | 32 | #ifdef _DEBUG 33 | SetLogLevel(ConsoleLogger::LogLevel::trace); 34 | SetAttrLogLevelVisible(true); 35 | SetAttrTimestampVisible(true); 36 | #endif 37 | auto normalExit = []() 38 | { 39 | LOG_INFO() << "Finished without errors" << endl; 40 | exit(0); 41 | }; 42 | auto abnormalExit = []() 43 | { 44 | LOG_INFO() << "Finished with errors" << endl; 45 | exit(-1); 46 | }; 47 | 48 | if (!SetConsoleCtrlHandler((PHANDLER_ROUTINE)CtrlHandler, TRUE)) 49 | LOG_WARNING() << "Could not set control handler" << std::endl; 50 | 51 | try 52 | { 53 | switch (taskCtx.Task) 54 | { 55 | case keeper::Task::Backup: 56 | { 57 | TaskBackup taskBackup(taskCtx); 58 | taskBackup.Run(); 59 | } 60 | break; 61 | 62 | case keeper::Task::DumpDB: 63 | { 64 | TaskDumpDB taskDump(taskCtx); 65 | taskDump.Run(); 66 | } 67 | break; 68 | 69 | case keeper::Task::Restore: 70 | { 71 | TaskRestore taskRestore(taskCtx); 72 | taskRestore.Run(); 73 | } 74 | break; 75 | 76 | case keeper::Task::Purge: 77 | { 78 | TaskPurge taskPurge(taskCtx); 79 | taskPurge.Run(); 80 | } 81 | break; 82 | 83 | default: 84 | break; 85 | } 86 | } 87 | catch (DbException& e) 88 | { 89 | LOG_FATAL() << "DB error: " << e.what() << endl; 90 | abnormalExit(); 91 | } 92 | catch (runtime_error& e) 93 | { 94 | LOG_FATAL() << StrAnsiToOEM(e.what()) << endl; 95 | abnormalExit(); 96 | } 97 | catch (exception& e) 98 | { 99 | LOG_FATAL() << StrAnsiToOEM(e.what()) << endl; 100 | abnormalExit(); 101 | } 102 | catch (...) 103 | { 104 | LOG_FATAL() << "Error occured" << endl; 105 | abnormalExit(); 106 | } 107 | 108 | if (GetMaxLogLevelPrinted() > LogLevel::warning) 109 | abnormalExit(); 110 | else 111 | normalExit(); 112 | } 113 | #endif //DOCTEST_CONFIG_DISABLE 114 | 115 | BOOL WINAPI CtrlHandler(DWORD fdwCtrlType) 116 | { 117 | using namespace std; 118 | switch (fdwCtrlType) 119 | { 120 | case CTRL_C_EVENT: 121 | case CTRL_CLOSE_EVENT: 122 | case CTRL_BREAK_EVENT: 123 | case CTRL_LOGOFF_EVENT: 124 | case CTRL_SHUTDOWN_EVENT: 125 | //praise the Void 126 | if (BuroEnv) 127 | BuroEnv->txn_checkpoint(0, 0, 0); 128 | return true; 129 | default: 130 | return FALSE; 131 | } 132 | } 133 | -------------------------------------------------------------------------------- /keeper/keeper.rc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sapozhnikov/keeper/b83c5c3bb4852ca4a52311854076a2015b112a13/keeper/keeper.rc -------------------------------------------------------------------------------- /keeper/keeper.vcxproj: -------------------------------------------------------------------------------- 1 |  2 | 3 | 4 | 5 | Debug 6 | Win32 7 | 8 | 9 | Doctest debug x86 10 | Win32 11 | 12 | 13 | Doctest debug x86 14 | x64 15 | 16 | 17 | Release 18 | Win32 19 | 20 | 21 | Debug 22 | x64 23 | 24 | 25 | Release 26 | x64 27 | 28 | 29 | 30 | {D42BBE5E-51FF-48A9-A338-2FB442AC63B7} 31 | Win32Proj 32 | keeper 33 | 10.0 34 | 35 | 36 | 37 | Application 38 | true 39 | v142 40 | Unicode 41 | 42 | 43 | Application 44 | true 45 | v142 46 | Unicode 47 | 48 | 49 | Application 50 | false 51 | v142 52 | true 53 | Unicode 54 | 55 | 56 | Application 57 | true 58 | v141 59 | Unicode 60 | 61 | 62 | Application 63 | true 64 | v141 65 | Unicode 66 | 67 | 68 | Application 69 | false 70 | v142 71 | true 72 | Unicode 73 | 74 | 75 | 76 | 77 | 78 | 79 | 80 | 81 | 82 | 83 | 84 | 85 | 86 | 87 | 88 | 89 | 90 | 91 | 92 | 93 | 94 | 95 | 96 | 97 | 98 | 99 | true 100 | D:\libs\doctest-2.3.5;D:\libs\libsodium\include;D:\libs\db-18.1.25\build_windows;D:\libs\boost_1_73_0;D:\libs\xz-5.2.5\src\liblzma\api;$(IncludePath) 101 | D:\libs\libsodium\Win32\Debug\v140\static;D:\libs\db-18.1.25\build_windows\Win32\Static Debug;D:\libs\boost_1_73_0\stage\lib;D:\libs\xz-5.2.5\windows\vs2019\ReleaseMT\Win32\liblzma;$(LibraryPath) 102 | buro 103 | 104 | 105 | true 106 | D:\libs\xz-5.2.5\src\liblzma\api;D:\libs\doctest-2.3.5;D:\libs\libsodium\include;D:\libs\db-18.1.25\build_windows;D:\libs\boost_1_73_0;$(IncludePath) 107 | D:\libs\libsodium\Win32\Debug\v140\static;D:\libs\db-18.1.25\build_windows\Win32\Static Debug;D:\libs\boost_1_73_0\stage\lib;D:\libs\xz-5.2.5\windows\vs2019\ReleaseMT\Win32\liblzma;$(LibraryPath) 108 | buro 109 | 110 | 111 | true 112 | buro 113 | 114 | 115 | true 116 | buro 117 | 118 | 119 | false 120 | D:\libs\libsodium\include;D:\libs\boost_1_73_0;D:\libs\db-18.1.25\build_windows;D:\libs\xz-5.2.5\src\liblzma\api;$(IncludePath) 121 | D:\libs\libsodium\Win32\Release\v140\static;D:\libs\db-18.1.25\build_windows\Win32\Static Release;D:\libs\boost_1_73_0\stage\lib;D:\libs\xz-5.2.5\windows\vs2019\ReleaseMT\Win32\liblzma;$(LibraryPath) 122 | buro 123 | 124 | 125 | false 126 | D:\libs\libsodium\include;D:\libs\boost_1_73_0;D:\libs\db-18.1.25\build_windows;D:\libs\xz-5.2.5\src\liblzma\api;$(IncludePath) 127 | D:\libs\libsodium\x64\Release\v140\static;D:\libs\db-18.1.25\build_windows\x64\Static Release;D:\libs\boost_1_73_0\stage\lib;D:\libs\xz-5.2.5\windows\vs2019\ReleaseMT\x64\liblzma;$(LibraryPath) 128 | buro 129 | 130 | 131 | 132 | Use 133 | Level4 134 | Disabled 135 | WIN32;_DEBUG;_CONSOLE;LZMA_API_STATIC;DOCTEST_CONFIG_DISABLE;%(PreprocessorDefinitions) 136 | true 137 | MultiThreadedDebug 138 | 139 | 140 | 141 | 142 | Console 143 | true 144 | libdb181sd.lib;libsodium.lib;liblzma.lib;Ws2_32.lib;%(AdditionalDependencies) 145 | 146 | 147 | 148 | 149 | Use 150 | Level4 151 | Disabled 152 | WIN32;_DEBUG;_CONSOLE;LZMA_API_STATIC;%(PreprocessorDefinitions) 153 | true 154 | MultiThreadedDebug 155 | 156 | 157 | 158 | 159 | Console 160 | true 161 | libdb181sd.lib;libsodium.lib;Ws2_32.lib;liblzma.lib;%(AdditionalDependencies) 162 | 163 | 164 | 165 | 166 | Use 167 | Level3 168 | Disabled 169 | _DEBUG;_CONSOLE;%(PreprocessorDefinitions) 170 | true 171 | 172 | 173 | Console 174 | true 175 | 176 | 177 | 178 | 179 | Use 180 | Level3 181 | Disabled 182 | _DEBUG;_CONSOLE;%(PreprocessorDefinitions) 183 | true 184 | 185 | 186 | Console 187 | true 188 | 189 | 190 | 191 | 192 | Level3 193 | Use 194 | MaxSpeed 195 | true 196 | true 197 | WIN32;NDEBUG;_CONSOLE;LZMA_API_STATIC;DOCTEST_CONFIG_DISABLE;%(PreprocessorDefinitions) 198 | true 199 | MultiThreaded 200 | 201 | 202 | Console 203 | true 204 | true 205 | false 206 | libdb181s.lib;libsodium.lib;liblzma.lib;%(AdditionalDependencies) 207 | 208 | 209 | 210 | 211 | Level3 212 | Use 213 | MaxSpeed 214 | true 215 | true 216 | NDEBUG;_CONSOLE;DOCTEST_CONFIG_DISABLE;LZMA_API_STATIC;%(PreprocessorDefinitions) 217 | true 218 | MultiThreaded 219 | 220 | 221 | Console 222 | true 223 | true 224 | false 225 | libdb181s.lib;libsodium.lib;liblzma.lib;%(AdditionalDependencies) 226 | 227 | 228 | 229 | 230 | 231 | 232 | 233 | 234 | 235 | 236 | 237 | 238 | 239 | 240 | 241 | 242 | 243 | 244 | 245 | 246 | 247 | 248 | 249 | 250 | 251 | 252 | 253 | NotUsing 254 | NotUsing 255 | NotUsing 256 | NotUsing 257 | 258 | 259 | false 260 | 261 | 262 | 263 | 264 | 265 | 266 | 267 | 268 | true 269 | true 270 | 271 | 272 | true 273 | true 274 | true 275 | 276 | 277 | 278 | 279 | 280 | 281 | Create 282 | Create 283 | Create 284 | Create 285 | Create 286 | Create 287 | 288 | 289 | 290 | 291 | 292 | 293 | 294 | false 295 | 296 | 297 | true 298 | true 299 | true 300 | 301 | 302 | 303 | 304 | 305 | 306 | 307 | 308 | -------------------------------------------------------------------------------- /keeper/keeper.vcxproj.filters: -------------------------------------------------------------------------------- 1 |  2 | 3 | 4 | 5 | {4FC737F1-C7A5-4376-A066-2A32D752A2FF} 6 | cpp;c;cc;cxx;def;odl;idl;hpj;bat;asm;asmx 7 | 8 | 9 | {93995380-89BD-4b04-88EB-625FBE52EBFB} 10 | h;hh;hpp;hxx;hm;inl;inc;xsd 11 | 12 | 13 | {67DA6AB6-F800-4c08-8B7A-83BB121AAD01} 14 | rc;ico;cur;bmp;dlg;rc2;rct;bin;rgs;gif;jpg;jpeg;jpe;resx;tiff;tif;png;wav;mfcribbon-ms 15 | 16 | 17 | {4d61b8e3-9f5a-43be-9908-24cecae37d0f} 18 | 19 | 20 | {e6ceca98-5fd5-4080-95a3-488536ebcbc5} 21 | 22 | 23 | {d0b7acdd-92f5-46c2-8107-5c578621bd60} 24 | 25 | 26 | {8957d651-2062-409c-922c-3495de3d6e42} 27 | 28 | 29 | {df74a085-a161-423f-9a6b-cf0b6871b483} 30 | 31 | 32 | {4aa67b19-88bb-4f7a-b285-cb1e9495f2e3} 33 | 34 | 35 | 36 | 37 | Header Files 38 | 39 | 40 | Header Files 41 | 42 | 43 | Header Files\tasks 44 | 45 | 46 | Header Files 47 | 48 | 49 | Header Files\tasks 50 | 51 | 52 | Header Files\tasks 53 | 54 | 55 | Header Files 56 | 57 | 58 | Header Files 59 | 60 | 61 | Header Files\tasks 62 | 63 | 64 | Header Files 65 | 66 | 67 | Header Files 68 | 69 | 70 | Header Files 71 | 72 | 73 | Header Files\FilesTransform 74 | 75 | 76 | Header Files\FilesTransform 77 | 78 | 79 | Header Files\FilesTransform 80 | 81 | 82 | Header Files 83 | 84 | 85 | Header Files 86 | 87 | 88 | Header Files 89 | 90 | 91 | Header Files 92 | 93 | 94 | Header Files 95 | 96 | 97 | Header Files 98 | 99 | 100 | 101 | 102 | Source Files 103 | 104 | 105 | Source Files 106 | 107 | 108 | Source Files 109 | 110 | 111 | Source Files\tasks 112 | 113 | 114 | Source Files 115 | 116 | 117 | Source Files\tasks 118 | 119 | 120 | Source Files\tasks 121 | 122 | 123 | Source Files 124 | 125 | 126 | Source Files\tasks 127 | 128 | 129 | Source Files 130 | 131 | 132 | Source Files 133 | 134 | 135 | Source Files\FilesTransform 136 | 137 | 138 | Source Files\FilesTransform 139 | 140 | 141 | Source Files 142 | 143 | 144 | Source Files 145 | 146 | 147 | Tests 148 | 149 | 150 | Tests 151 | 152 | 153 | Tests 154 | 155 | 156 | Source Files 157 | 158 | 159 | Source Files 160 | 161 | 162 | Source Files\boost 163 | 164 | 165 | Tests 166 | 167 | 168 | 169 | 170 | Resource Files 171 | 172 | 173 | -------------------------------------------------------------------------------- /keeper/keeper.vcxproj.user: -------------------------------------------------------------------------------- 1 |  2 | 3 | 4 | buro --purge --srcdir="y:\backup\myprojects" --older=P1M --verbose 5 | WindowsLocalDebugger 6 | 7 | 8 | --test-case="DbCompress" 9 | WindowsLocalDebugger 10 | 11 | -------------------------------------------------------------------------------- /keeper/resource.h: -------------------------------------------------------------------------------- 1 | //{{NO_DEPENDENCIES}} 2 | // Microsoft Visual C++ generated include file. 3 | // Used by keeper.rc 4 | 5 | // Next default values for new objects 6 | // 7 | #ifdef APSTUDIO_INVOKED 8 | #ifndef APSTUDIO_READONLY_SYMBOLS 9 | #define _APS_NEXT_RESOURCE_VALUE 101 10 | #define _APS_NEXT_COMMAND_VALUE 40001 11 | #define _APS_NEXT_CONTROL_VALUE 1001 12 | #define _APS_NEXT_SYMED_VALUE 101 13 | #endif 14 | #endif 15 | -------------------------------------------------------------------------------- /keeper/stdafx.cpp: -------------------------------------------------------------------------------- 1 | // stdafx.cpp : source file that includes just the standard includes 2 | // keeper.pch will be the pre-compiled header 3 | // stdafx.obj will contain the pre-compiled type information 4 | 5 | #include "stdafx.h" 6 | 7 | // TODO: reference any additional headers you need in STDAFX.H 8 | // and not in this file 9 | -------------------------------------------------------------------------------- /keeper/stdafx.h: -------------------------------------------------------------------------------- 1 | // stdafx.h : include file for standard system include files, 2 | // or project specific include files that are used frequently, but 3 | // are changed infrequently 4 | // 5 | 6 | #pragma once 7 | 8 | #include "targetver.h" 9 | 10 | #include 11 | #include 12 | 13 | 14 | 15 | // TODO: reference additional headers your program requires here 16 | #include 17 | #include //include it AFTER berkley stuff 18 | #include 19 | #include "CommonDefinitions.h" 20 | #include "ConsoleLogger.h" 21 | #include "TnxGuard.h" 22 | #include 23 | 24 | #define SODIUM_STATIC 1 25 | #define SODIUM_EXPORT 26 | #include 27 | -------------------------------------------------------------------------------- /keeper/targetver.h: -------------------------------------------------------------------------------- 1 | #pragma once 2 | 3 | // Including SDKDDKVer.h defines the highest available Windows platform. 4 | 5 | // If you wish to build your application for a previous Windows platform, include WinSDKVer.h and 6 | // set the _WIN32_WINNT macro to the platform you wish to support before including SDKDDKVer.h. 7 | 8 | #include 9 | -------------------------------------------------------------------------------- /logo_buro.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sapozhnikov/keeper/b83c5c3bb4852ca4a52311854076a2015b112a13/logo_buro.png --------------------------------------------------------------------------------