├── AUTHORS ├── COPYRIGHT ├── LICENSE.CC-BY-SA-4.0 ├── Makefile ├── TODO ├── _include └── unicode.tex ├── appendix ├── 01-test-cases.rst ├── 02-proofs.rst ├── 03-pseudocode.rst ├── 04-extras.rst ├── 05-hybrid.rst └── index.rst ├── apt-get-build-dep.sh ├── causal ├── 01-ordering.rst ├── 02-consistency.rst ├── 03-freshness.rst ├── 04-state.rst ├── 05-visibility.rst ├── 06-application.rst ├── 07-interface.rst └── index.rst ├── conf.py ├── greet.rst ├── hci.rst ├── index.rst ├── make.bat └── ratchet.rst /AUTHORS: -------------------------------------------------------------------------------- 1 | Ximin Luo 2 | Ximin Luo 3 | -------------------------------------------------------------------------------- /COPYRIGHT: -------------------------------------------------------------------------------- 1 | (c) 2014-2015 Mega Limited 2 | (c) 2016 Ximin Luo 3 | 4 | This work is licensed under the Creative Commons 5 | Attribution-ShareAlike 4.0 International License. 6 | 7 | To view a copy of this license, see the file LICENSE.CC-BY-SA-4.0 in this 8 | directory, or for other formats, visit: 9 | 10 | https://creativecommons.org/licenses/by-sa/4.0/ 11 | -------------------------------------------------------------------------------- /LICENSE.CC-BY-SA-4.0: -------------------------------------------------------------------------------- 1 | Attribution-ShareAlike 4.0 International 2 | 3 | ======================================================================= 4 | 5 | Creative Commons Corporation ("Creative Commons") is not a law firm and 6 | does not provide legal services or legal advice. Distribution of 7 | Creative Commons public licenses does not create a lawyer-client or 8 | other relationship. Creative Commons makes its licenses and related 9 | information available on an "as-is" basis. Creative Commons gives no 10 | warranties regarding its licenses, any material licensed under their 11 | terms and conditions, or any related information. Creative Commons 12 | disclaims all liability for damages resulting from their use to the 13 | fullest extent possible. 14 | 15 | Using Creative Commons Public Licenses 16 | 17 | Creative Commons public licenses provide a standard set of terms and 18 | conditions that creators and other rights holders may use to share 19 | original works of authorship and other material subject to copyright 20 | and certain other rights specified in the public license below. The 21 | following considerations are for informational purposes only, are not 22 | exhaustive, and do not form part of our licenses. 23 | 24 | Considerations for licensors: Our public licenses are 25 | intended for use by those authorized to give the public 26 | permission to use material in ways otherwise restricted by 27 | copyright and certain other rights. Our licenses are 28 | irrevocable. Licensors should read and understand the terms 29 | and conditions of the license they choose before applying it. 30 | Licensors should also secure all rights necessary before 31 | applying our licenses so that the public can reuse the 32 | material as expected. Licensors should clearly mark any 33 | material not subject to the license. This includes other CC- 34 | licensed material, or material used under an exception or 35 | limitation to copyright. More considerations for licensors: 36 | wiki.creativecommons.org/Considerations_for_licensors 37 | 38 | Considerations for the public: By using one of our public 39 | licenses, a licensor grants the public permission to use the 40 | licensed material under specified terms and conditions. If 41 | the licensor's permission is not necessary for any reason--for 42 | example, because of any applicable exception or limitation to 43 | copyright--then that use is not regulated by the license. Our 44 | licenses grant only permissions under copyright and certain 45 | other rights that a licensor has authority to grant. Use of 46 | the licensed material may still be restricted for other 47 | reasons, including because others have copyright or other 48 | rights in the material. A licensor may make special requests, 49 | such as asking that all changes be marked or described. 50 | Although not required by our licenses, you are encouraged to 51 | respect those requests where reasonable. More_considerations 52 | for the public: 53 | wiki.creativecommons.org/Considerations_for_licensees 54 | 55 | ======================================================================= 56 | 57 | Creative Commons Attribution-ShareAlike 4.0 International Public 58 | License 59 | 60 | By exercising the Licensed Rights (defined below), You accept and agree 61 | to be bound by the terms and conditions of this Creative Commons 62 | Attribution-ShareAlike 4.0 International Public License ("Public 63 | License"). To the extent this Public License may be interpreted as a 64 | contract, You are granted the Licensed Rights in consideration of Your 65 | acceptance of these terms and conditions, and the Licensor grants You 66 | such rights in consideration of benefits the Licensor receives from 67 | making the Licensed Material available under these terms and 68 | conditions. 69 | 70 | 71 | Section 1 -- Definitions. 72 | 73 | a. Adapted Material means material subject to Copyright and Similar 74 | Rights that is derived from or based upon the Licensed Material 75 | and in which the Licensed Material is translated, altered, 76 | arranged, transformed, or otherwise modified in a manner requiring 77 | permission under the Copyright and Similar Rights held by the 78 | Licensor. For purposes of this Public License, where the Licensed 79 | Material is a musical work, performance, or sound recording, 80 | Adapted Material is always produced where the Licensed Material is 81 | synched in timed relation with a moving image. 82 | 83 | b. Adapter's License means the license You apply to Your Copyright 84 | and Similar Rights in Your contributions to Adapted Material in 85 | accordance with the terms and conditions of this Public License. 86 | 87 | c. BY-SA Compatible License means a license listed at 88 | creativecommons.org/compatiblelicenses, approved by Creative 89 | Commons as essentially the equivalent of this Public License. 90 | 91 | d. Copyright and Similar Rights means copyright and/or similar rights 92 | closely related to copyright including, without limitation, 93 | performance, broadcast, sound recording, and Sui Generis Database 94 | Rights, without regard to how the rights are labeled or 95 | categorized. For purposes of this Public License, the rights 96 | specified in Section 2(b)(1)-(2) are not Copyright and Similar 97 | Rights. 98 | 99 | e. Effective Technological Measures means those measures that, in the 100 | absence of proper authority, may not be circumvented under laws 101 | fulfilling obligations under Article 11 of the WIPO Copyright 102 | Treaty adopted on December 20, 1996, and/or similar international 103 | agreements. 104 | 105 | f. Exceptions and Limitations means fair use, fair dealing, and/or 106 | any other exception or limitation to Copyright and Similar Rights 107 | that applies to Your use of the Licensed Material. 108 | 109 | g. License Elements means the license attributes listed in the name 110 | of a Creative Commons Public License. The License Elements of this 111 | Public License are Attribution and ShareAlike. 112 | 113 | h. Licensed Material means the artistic or literary work, database, 114 | or other material to which the Licensor applied this Public 115 | License. 116 | 117 | i. Licensed Rights means the rights granted to You subject to the 118 | terms and conditions of this Public License, which are limited to 119 | all Copyright and Similar Rights that apply to Your use of the 120 | Licensed Material and that the Licensor has authority to license. 121 | 122 | j. Licensor means the individual(s) or entity(ies) granting rights 123 | under this Public License. 124 | 125 | k. Share means to provide material to the public by any means or 126 | process that requires permission under the Licensed Rights, such 127 | as reproduction, public display, public performance, distribution, 128 | dissemination, communication, or importation, and to make material 129 | available to the public including in ways that members of the 130 | public may access the material from a place and at a time 131 | individually chosen by them. 132 | 133 | l. Sui Generis Database Rights means rights other than copyright 134 | resulting from Directive 96/9/EC of the European Parliament and of 135 | the Council of 11 March 1996 on the legal protection of databases, 136 | as amended and/or succeeded, as well as other essentially 137 | equivalent rights anywhere in the world. 138 | 139 | m. You means the individual or entity exercising the Licensed Rights 140 | under this Public License. Your has a corresponding meaning. 141 | 142 | 143 | Section 2 -- Scope. 144 | 145 | a. License grant. 146 | 147 | 1. Subject to the terms and conditions of this Public License, 148 | the Licensor hereby grants You a worldwide, royalty-free, 149 | non-sublicensable, non-exclusive, irrevocable license to 150 | exercise the Licensed Rights in the Licensed Material to: 151 | 152 | a. reproduce and Share the Licensed Material, in whole or 153 | in part; and 154 | 155 | b. produce, reproduce, and Share Adapted Material. 156 | 157 | 2. Exceptions and Limitations. For the avoidance of doubt, where 158 | Exceptions and Limitations apply to Your use, this Public 159 | License does not apply, and You do not need to comply with 160 | its terms and conditions. 161 | 162 | 3. Term. The term of this Public License is specified in Section 163 | 6(a). 164 | 165 | 4. Media and formats; technical modifications allowed. The 166 | Licensor authorizes You to exercise the Licensed Rights in 167 | all media and formats whether now known or hereafter created, 168 | and to make technical modifications necessary to do so. The 169 | Licensor waives and/or agrees not to assert any right or 170 | authority to forbid You from making technical modifications 171 | necessary to exercise the Licensed Rights, including 172 | technical modifications necessary to circumvent Effective 173 | Technological Measures. For purposes of this Public License, 174 | simply making modifications authorized by this Section 2(a) 175 | (4) never produces Adapted Material. 176 | 177 | 5. Downstream recipients. 178 | 179 | a. Offer from the Licensor -- Licensed Material. Every 180 | recipient of the Licensed Material automatically 181 | receives an offer from the Licensor to exercise the 182 | Licensed Rights under the terms and conditions of this 183 | Public License. 184 | 185 | b. Additional offer from the Licensor -- Adapted Material. 186 | Every recipient of Adapted Material from You 187 | automatically receives an offer from the Licensor to 188 | exercise the Licensed Rights in the Adapted Material 189 | under the conditions of the Adapter's License You apply. 190 | 191 | c. No downstream restrictions. You may not offer or impose 192 | any additional or different terms or conditions on, or 193 | apply any Effective Technological Measures to, the 194 | Licensed Material if doing so restricts exercise of the 195 | Licensed Rights by any recipient of the Licensed 196 | Material. 197 | 198 | 6. No endorsement. Nothing in this Public License constitutes or 199 | may be construed as permission to assert or imply that You 200 | are, or that Your use of the Licensed Material is, connected 201 | with, or sponsored, endorsed, or granted official status by, 202 | the Licensor or others designated to receive attribution as 203 | provided in Section 3(a)(1)(A)(i). 204 | 205 | b. Other rights. 206 | 207 | 1. Moral rights, such as the right of integrity, are not 208 | licensed under this Public License, nor are publicity, 209 | privacy, and/or other similar personality rights; however, to 210 | the extent possible, the Licensor waives and/or agrees not to 211 | assert any such rights held by the Licensor to the limited 212 | extent necessary to allow You to exercise the Licensed 213 | Rights, but not otherwise. 214 | 215 | 2. Patent and trademark rights are not licensed under this 216 | Public License. 217 | 218 | 3. To the extent possible, the Licensor waives any right to 219 | collect royalties from You for the exercise of the Licensed 220 | Rights, whether directly or through a collecting society 221 | under any voluntary or waivable statutory or compulsory 222 | licensing scheme. In all other cases the Licensor expressly 223 | reserves any right to collect such royalties. 224 | 225 | 226 | Section 3 -- License Conditions. 227 | 228 | Your exercise of the Licensed Rights is expressly made subject to the 229 | following conditions. 230 | 231 | a. Attribution. 232 | 233 | 1. If You Share the Licensed Material (including in modified 234 | form), You must: 235 | 236 | a. retain the following if it is supplied by the Licensor 237 | with the Licensed Material: 238 | 239 | i. identification of the creator(s) of the Licensed 240 | Material and any others designated to receive 241 | attribution, in any reasonable manner requested by 242 | the Licensor (including by pseudonym if 243 | designated); 244 | 245 | ii. a copyright notice; 246 | 247 | iii. a notice that refers to this Public License; 248 | 249 | iv. a notice that refers to the disclaimer of 250 | warranties; 251 | 252 | v. a URI or hyperlink to the Licensed Material to the 253 | extent reasonably practicable; 254 | 255 | b. indicate if You modified the Licensed Material and 256 | retain an indication of any previous modifications; and 257 | 258 | c. indicate the Licensed Material is licensed under this 259 | Public License, and include the text of, or the URI or 260 | hyperlink to, this Public License. 261 | 262 | 2. You may satisfy the conditions in Section 3(a)(1) in any 263 | reasonable manner based on the medium, means, and context in 264 | which You Share the Licensed Material. For example, it may be 265 | reasonable to satisfy the conditions by providing a URI or 266 | hyperlink to a resource that includes the required 267 | information. 268 | 269 | 3. If requested by the Licensor, You must remove any of the 270 | information required by Section 3(a)(1)(A) to the extent 271 | reasonably practicable. 272 | 273 | b. ShareAlike. 274 | 275 | In addition to the conditions in Section 3(a), if You Share 276 | Adapted Material You produce, the following conditions also apply. 277 | 278 | 1. The Adapter's License You apply must be a Creative Commons 279 | license with the same License Elements, this version or 280 | later, or a BY-SA Compatible License. 281 | 282 | 2. You must include the text of, or the URI or hyperlink to, the 283 | Adapter's License You apply. You may satisfy this condition 284 | in any reasonable manner based on the medium, means, and 285 | context in which You Share Adapted Material. 286 | 287 | 3. You may not offer or impose any additional or different terms 288 | or conditions on, or apply any Effective Technological 289 | Measures to, Adapted Material that restrict exercise of the 290 | rights granted under the Adapter's License You apply. 291 | 292 | 293 | Section 4 -- Sui Generis Database Rights. 294 | 295 | Where the Licensed Rights include Sui Generis Database Rights that 296 | apply to Your use of the Licensed Material: 297 | 298 | a. for the avoidance of doubt, Section 2(a)(1) grants You the right 299 | to extract, reuse, reproduce, and Share all or a substantial 300 | portion of the contents of the database; 301 | 302 | b. if You include all or a substantial portion of the database 303 | contents in a database in which You have Sui Generis Database 304 | Rights, then the database in which You have Sui Generis Database 305 | Rights (but not its individual contents) is Adapted Material, 306 | 307 | including for purposes of Section 3(b); and 308 | c. You must comply with the conditions in Section 3(a) if You Share 309 | all or a substantial portion of the contents of the database. 310 | 311 | For the avoidance of doubt, this Section 4 supplements and does not 312 | replace Your obligations under this Public License where the Licensed 313 | Rights include other Copyright and Similar Rights. 314 | 315 | 316 | Section 5 -- Disclaimer of Warranties and Limitation of Liability. 317 | 318 | a. UNLESS OTHERWISE SEPARATELY UNDERTAKEN BY THE LICENSOR, TO THE 319 | EXTENT POSSIBLE, THE LICENSOR OFFERS THE LICENSED MATERIAL AS-IS 320 | AND AS-AVAILABLE, AND MAKES NO REPRESENTATIONS OR WARRANTIES OF 321 | ANY KIND CONCERNING THE LICENSED MATERIAL, WHETHER EXPRESS, 322 | IMPLIED, STATUTORY, OR OTHER. THIS INCLUDES, WITHOUT LIMITATION, 323 | WARRANTIES OF TITLE, MERCHANTABILITY, FITNESS FOR A PARTICULAR 324 | PURPOSE, NON-INFRINGEMENT, ABSENCE OF LATENT OR OTHER DEFECTS, 325 | ACCURACY, OR THE PRESENCE OR ABSENCE OF ERRORS, WHETHER OR NOT 326 | KNOWN OR DISCOVERABLE. WHERE DISCLAIMERS OF WARRANTIES ARE NOT 327 | ALLOWED IN FULL OR IN PART, THIS DISCLAIMER MAY NOT APPLY TO YOU. 328 | 329 | b. TO THE EXTENT POSSIBLE, IN NO EVENT WILL THE LICENSOR BE LIABLE 330 | TO YOU ON ANY LEGAL THEORY (INCLUDING, WITHOUT LIMITATION, 331 | NEGLIGENCE) OR OTHERWISE FOR ANY DIRECT, SPECIAL, INDIRECT, 332 | INCIDENTAL, CONSEQUENTIAL, PUNITIVE, EXEMPLARY, OR OTHER LOSSES, 333 | COSTS, EXPENSES, OR DAMAGES ARISING OUT OF THIS PUBLIC LICENSE OR 334 | USE OF THE LICENSED MATERIAL, EVEN IF THE LICENSOR HAS BEEN 335 | ADVISED OF THE POSSIBILITY OF SUCH LOSSES, COSTS, EXPENSES, OR 336 | DAMAGES. WHERE A LIMITATION OF LIABILITY IS NOT ALLOWED IN FULL OR 337 | IN PART, THIS LIMITATION MAY NOT APPLY TO YOU. 338 | 339 | c. The disclaimer of warranties and limitation of liability provided 340 | above shall be interpreted in a manner that, to the extent 341 | possible, most closely approximates an absolute disclaimer and 342 | waiver of all liability. 343 | 344 | 345 | Section 6 -- Term and Termination. 346 | 347 | a. This Public License applies for the term of the Copyright and 348 | Similar Rights licensed here. However, if You fail to comply with 349 | this Public License, then Your rights under this Public License 350 | terminate automatically. 351 | 352 | b. Where Your right to use the Licensed Material has terminated under 353 | Section 6(a), it reinstates: 354 | 355 | 1. automatically as of the date the violation is cured, provided 356 | it is cured within 30 days of Your discovery of the 357 | violation; or 358 | 359 | 2. upon express reinstatement by the Licensor. 360 | 361 | For the avoidance of doubt, this Section 6(b) does not affect any 362 | right the Licensor may have to seek remedies for Your violations 363 | of this Public License. 364 | 365 | c. For the avoidance of doubt, the Licensor may also offer the 366 | Licensed Material under separate terms or conditions or stop 367 | distributing the Licensed Material at any time; however, doing so 368 | will not terminate this Public License. 369 | 370 | d. Sections 1, 5, 6, 7, and 8 survive termination of this Public 371 | License. 372 | 373 | 374 | Section 7 -- Other Terms and Conditions. 375 | 376 | a. The Licensor shall not be bound by any additional or different 377 | terms or conditions communicated by You unless expressly agreed. 378 | 379 | b. Any arrangements, understandings, or agreements regarding the 380 | Licensed Material not stated herein are separate from and 381 | independent of the terms and conditions of this Public License. 382 | 383 | 384 | Section 8 -- Interpretation. 385 | 386 | a. For the avoidance of doubt, this Public License does not, and 387 | shall not be interpreted to, reduce, limit, restrict, or impose 388 | conditions on any use of the Licensed Material that could lawfully 389 | be made without permission under this Public License. 390 | 391 | b. To the extent possible, if any provision of this Public License is 392 | deemed unenforceable, it shall be automatically reformed to the 393 | minimum extent necessary to make it enforceable. If the provision 394 | cannot be reformed, it shall be severed from this Public License 395 | without affecting the enforceability of the remaining terms and 396 | conditions. 397 | 398 | c. No term or condition of this Public License will be waived and no 399 | failure to comply consented to unless expressly agreed to by the 400 | Licensor. 401 | 402 | d. Nothing in this Public License constitutes or may be interpreted 403 | as a limitation upon, or waiver of, any privileges and immunities 404 | that apply to the Licensor or You, including from the legal 405 | processes of any jurisdiction or authority. 406 | 407 | 408 | ======================================================================= 409 | 410 | Creative Commons is not a party to its public licenses. 411 | Notwithstanding, Creative Commons may elect to apply one of its public 412 | licenses to material it publishes and in those instances will be 413 | considered the "Licensor." Except for the limited purpose of indicating 414 | that material is shared under a Creative Commons public license or as 415 | otherwise permitted by the Creative Commons policies published at 416 | creativecommons.org/policies, Creative Commons does not authorize the 417 | use of the trademark "Creative Commons" or any other trademark or logo 418 | of Creative Commons without its prior written consent including, 419 | without limitation, in connection with any unauthorized modifications 420 | to any of its public licenses or any other arrangements, 421 | understandings, or agreements concerning use of licensed material. For 422 | the avoidance of doubt, this paragraph does not form part of the public 423 | licenses. 424 | 425 | Creative Commons may be contacted at creativecommons.org. 426 | -------------------------------------------------------------------------------- /Makefile: -------------------------------------------------------------------------------- 1 | # Makefile for Sphinx documentation 2 | # 3 | 4 | # You can set these variables from the command line. 5 | SPHINXOPTS = 6 | SPHINXBUILD = sphinx-build 7 | PAPER = 8 | BUILDDIR = _build 9 | 10 | # Internal variables. 11 | PAPEROPT_a4 = -D latex_paper_size=a4 12 | PAPEROPT_letter = -D latex_paper_size=letter 13 | ALLSPHINXOPTS = -d $(BUILDDIR)/doctrees $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) . 14 | # the i18n builder cannot share the environment and doctrees with the others 15 | I18NSPHINXOPTS = $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) . 16 | 17 | .PHONY: help clean html dirhtml singlehtml pickle json htmlhelp qthelp devhelp epub latex latexpdf text man changes linkcheck doctest gettext 18 | 19 | help: 20 | @echo "Please use \`make ' where is one of" 21 | @echo " html to make standalone HTML files" 22 | @echo " dirhtml to make HTML files named index.html in directories" 23 | @echo " singlehtml to make a single large HTML file" 24 | @echo " pickle to make pickle files" 25 | @echo " json to make JSON files" 26 | @echo " htmlhelp to make HTML files and a HTML help project" 27 | @echo " qthelp to make HTML files and a qthelp project" 28 | @echo " devhelp to make HTML files and a Devhelp project" 29 | @echo " epub to make an epub" 30 | @echo " latex to make LaTeX files, you can set PAPER=a4 or PAPER=letter" 31 | @echo " latexpdf to make LaTeX files and run them through pdflatex" 32 | @echo " text to make text files" 33 | @echo " man to make manual pages" 34 | @echo " texinfo to make Texinfo files" 35 | @echo " info to make Texinfo files and run them through makeinfo" 36 | @echo " gettext to make PO message catalogs" 37 | @echo " changes to make an overview of all changed/added/deprecated items" 38 | @echo " linkcheck to check all external links for integrity" 39 | @echo " doctest to run all doctests embedded in the documentation (if enabled)" 40 | 41 | clean: 42 | -rm -rf $(BUILDDIR)/* 43 | 44 | html: 45 | $(SPHINXBUILD) -b html $(ALLSPHINXOPTS) $(BUILDDIR)/html 46 | @echo 47 | @echo "Build finished. The HTML pages are in $(BUILDDIR)/html." 48 | 49 | dirhtml: 50 | $(SPHINXBUILD) -b dirhtml $(ALLSPHINXOPTS) $(BUILDDIR)/dirhtml 51 | @echo 52 | @echo "Build finished. The HTML pages are in $(BUILDDIR)/dirhtml." 53 | 54 | singlehtml: 55 | $(SPHINXBUILD) -b singlehtml $(ALLSPHINXOPTS) $(BUILDDIR)/singlehtml 56 | @echo 57 | @echo "Build finished. The HTML page is in $(BUILDDIR)/singlehtml." 58 | 59 | pickle: 60 | $(SPHINXBUILD) -b pickle $(ALLSPHINXOPTS) $(BUILDDIR)/pickle 61 | @echo 62 | @echo "Build finished; now you can process the pickle files." 63 | 64 | json: 65 | $(SPHINXBUILD) -b json $(ALLSPHINXOPTS) $(BUILDDIR)/json 66 | @echo 67 | @echo "Build finished; now you can process the JSON files." 68 | 69 | htmlhelp: 70 | $(SPHINXBUILD) -b htmlhelp $(ALLSPHINXOPTS) $(BUILDDIR)/htmlhelp 71 | @echo 72 | @echo "Build finished; now you can run HTML Help Workshop with the" \ 73 | ".hhp project file in $(BUILDDIR)/htmlhelp." 74 | 75 | qthelp: 76 | $(SPHINXBUILD) -b qthelp $(ALLSPHINXOPTS) $(BUILDDIR)/qthelp 77 | @echo 78 | @echo "Build finished; now you can run "qcollectiongenerator" with the" \ 79 | ".qhcp project file in $(BUILDDIR)/qthelp, like this:" 80 | @echo "# qcollectiongenerator $(BUILDDIR)/qthelp/msg-notes.qhcp" 81 | @echo "To view the help file:" 82 | @echo "# assistant -collectionFile $(BUILDDIR)/qthelp/msg-notes.qhc" 83 | 84 | devhelp: 85 | $(SPHINXBUILD) -b devhelp $(ALLSPHINXOPTS) $(BUILDDIR)/devhelp 86 | @echo 87 | @echo "Build finished." 88 | @echo "To view the help file:" 89 | @echo "# mkdir -p $$HOME/.local/share/devhelp/msg-notes" 90 | @echo "# ln -s $(BUILDDIR)/devhelp $$HOME/.local/share/devhelp/msg-notes" 91 | @echo "# devhelp" 92 | 93 | epub: 94 | $(SPHINXBUILD) -b epub $(ALLSPHINXOPTS) $(BUILDDIR)/epub 95 | @echo 96 | @echo "Build finished. The epub file is in $(BUILDDIR)/epub." 97 | 98 | latex: 99 | $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex 100 | @echo 101 | @echo "Build finished; the LaTeX files are in $(BUILDDIR)/latex." 102 | @echo "Run \`make' in that directory to run these through (pdf)latex" \ 103 | "(use \`make latexpdf' here to do that automatically)." 104 | 105 | latexpdf: 106 | $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex 107 | @echo "Running LaTeX files through pdflatex..." 108 | $(MAKE) -C $(BUILDDIR)/latex all-pdf 109 | @echo "pdflatex finished; the PDF files are in $(BUILDDIR)/latex." 110 | 111 | text: 112 | $(SPHINXBUILD) -b text $(ALLSPHINXOPTS) $(BUILDDIR)/text 113 | @echo 114 | @echo "Build finished. The text files are in $(BUILDDIR)/text." 115 | 116 | man: 117 | $(SPHINXBUILD) -b man $(ALLSPHINXOPTS) $(BUILDDIR)/man 118 | @echo 119 | @echo "Build finished. The manual pages are in $(BUILDDIR)/man." 120 | 121 | texinfo: 122 | $(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo 123 | @echo 124 | @echo "Build finished. The Texinfo files are in $(BUILDDIR)/texinfo." 125 | @echo "Run \`make' in that directory to run these through makeinfo" \ 126 | "(use \`make info' here to do that automatically)." 127 | 128 | info: 129 | $(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo 130 | @echo "Running Texinfo files through makeinfo..." 131 | make -C $(BUILDDIR)/texinfo info 132 | @echo "makeinfo finished; the Info files are in $(BUILDDIR)/texinfo." 133 | 134 | gettext: 135 | $(SPHINXBUILD) -b gettext $(I18NSPHINXOPTS) $(BUILDDIR)/locale 136 | @echo 137 | @echo "Build finished. The message catalogs are in $(BUILDDIR)/locale." 138 | 139 | changes: 140 | $(SPHINXBUILD) -b changes $(ALLSPHINXOPTS) $(BUILDDIR)/changes 141 | @echo 142 | @echo "The overview file is in $(BUILDDIR)/changes." 143 | 144 | linkcheck: 145 | $(SPHINXBUILD) -b linkcheck $(ALLSPHINXOPTS) $(BUILDDIR)/linkcheck 146 | @echo 147 | @echo "Link check complete; look for any errors in the above output " \ 148 | "or in $(BUILDDIR)/linkcheck/output.txt." 149 | 150 | doctest: 151 | $(SPHINXBUILD) -b doctest $(ALLSPHINXOPTS) $(BUILDDIR)/doctest 152 | @echo "Testing of doctests in the sources finished, look at the " \ 153 | "results in $(BUILDDIR)/doctest/output.txt." 154 | -------------------------------------------------------------------------------- /TODO: -------------------------------------------------------------------------------- 1 | # Theory topics 2 | 3 | - distinguish between sender (of a packet) and author (of a message) 4 | - explain why we should hash ciphertext and/or (plaintext *and* a secret) 5 | - explain that we assume message ID that can be calculated as soon as it is 6 | decrypted (for incoming messages) or encrypted (for outgoing messages). this 7 | doesn't hold true for mpCAT's "parent hashes", which can only be calculated 8 | after the message is echoed back from the server. 9 | 10 | # Misc 11 | 12 | - "sender" -> "author" in other places too 13 | 14 | - move "tangents" (as pointed out by michael) to better places, maybe appendix 15 | 16 | - It'd be easier to read if some typographic conventions are 17 | honoured in the writing. I'd suggest to use :math:`` 18 | environments for all formulaic content as well as variable 19 | references. This would render variables (such as "u" and "a") 20 | usually in italics, and make reading much simpler. 21 | 22 | - factor out the security/false-positive argument somewhere common 23 | - made twice in causal/02-consistency, once in appendix/05-hybrid 24 | 25 | - categorise rules in 05-hybrid in a more systematic way 26 | -------------------------------------------------------------------------------- /_include/unicode.tex: -------------------------------------------------------------------------------- 1 | %% shim to add latex support for rst/sphinx unicode entities 2 | \usepackage{amssymb} 3 | \DeclareUnicodeCharacter{21A6}{$\mapsto$} 4 | \DeclareUnicodeCharacter{2190}{$\leftarrow$} 5 | \DeclareUnicodeCharacter{2194}{$\leftrightarrow$} 6 | \DeclareUnicodeCharacter{21D2}{$\Rightarrow$} 7 | \DeclareUnicodeCharacter{21D4}{$\Leftrightarrow$} 8 | \DeclareUnicodeCharacter{2200}{$\forall$} 9 | \DeclareUnicodeCharacter{2203}{$\exists$} 10 | \DeclareUnicodeCharacter{2204}{$\nexists$} 11 | \DeclareUnicodeCharacter{2205}{$\emptyset$} 12 | \DeclareUnicodeCharacter{2208}{$\in$} 13 | \DeclareUnicodeCharacter{2209}{$\notin$} 14 | \DeclareUnicodeCharacter{2227}{$\wedge$} 15 | \DeclareUnicodeCharacter{2228}{$\vee$} 16 | \DeclareUnicodeCharacter{2229}{$\cap$} 17 | \DeclareUnicodeCharacter{2260}{$\neq$} 18 | \DeclareUnicodeCharacter{2261}{$\equiv$} 19 | \DeclareUnicodeCharacter{2264}{$\leq$} 20 | \DeclareUnicodeCharacter{2265}{$\geq$} 21 | \DeclareUnicodeCharacter{2282}{$\subset$} 22 | \DeclareUnicodeCharacter{2286}{$\subseteq$} 23 | \DeclareUnicodeCharacter{2288}{$\nsubseteq$} 24 | \DeclareUnicodeCharacter{228F}{$\sqsubset$} 25 | \DeclareUnicodeCharacter{22A5}{$\perp$} 26 | -------------------------------------------------------------------------------- /appendix/01-test-cases.rst: -------------------------------------------------------------------------------- 1 | ========== 2 | Test cases 3 | ========== 4 | 5 | Causal ordering 6 | --------------- 7 | 8 | Greeting protocols 9 | ------------------ 10 | 11 | Key-rotation ratchets 12 | --------------------- 13 | -------------------------------------------------------------------------------- /appendix/02-proofs.rst: -------------------------------------------------------------------------------- 1 | ====== 2 | Proofs 3 | ====== 4 | 5 | .. include:: 6 | .. include:: 7 | 8 | Causal ordering 9 | =============== 10 | 11 | .. _diff-apply-model: 12 | 13 | Formal model for diff and apply 14 | ------------------------------- 15 | 16 | This is similar to, but not exactly the same as, darcs' `theory of patches`. 17 | Their theory is more developed with more "useful" results, but is also places 18 | more constraints on what valid patches are. 19 | 20 | We try to be more general (less constrained), to demonstrate that the rest of 21 | our work depends on fewer assumptions than their model. For example, we don't 22 | need a notion of "compose two patches into another patch", and we only have 23 | partial notions of "applying n patches is equivalent to as if network had much higher latency. 190 | mlst estimates might too generous, not such a big deal. it might even be 191 | unaffected since it's calculated as max() of information from multiple people. 192 | 193 | - dst too high 194 | - if no-one notices -> no problem, as if network had much lower latency - 195 | actually a good thing! 196 | - if someone notices, send a "bad timestamp" message, emit a UI warning "mlst 197 | appears to be from the future; mlst calculations of later messages may also 198 | be affected by this" 199 | - diagram of example situation 200 | - how to handle (might be innocent)? getting quite complex, leave for 201 | future research 202 | 203 | Could be integrated into freshness checks, but complexity probably not worth it. 204 | -------------------------------------------------------------------------------- /appendix/05-hybrid.rst: -------------------------------------------------------------------------------- 1 | =============== 2 | Hybrid ordering 3 | =============== 4 | 5 | .. include:: 6 | .. include:: 7 | 8 | Causal orders have some nice properties explained elsewhere, but are also hard 9 | to implement correctly when not everyone is allowed to see the full history. 10 | (At the time of writing, we don't have a full solution for this yet.) 11 | 12 | One alternative is to have a total order for membership operations, while 13 | leaving messages to be in a causal order. Here, we present a scheme for this 14 | that requires no extra client packets, but does require one extra round of 15 | latency - the minimum necessary for a total order that preserves context. 16 | 17 | The scheme only applies to certain membership operations packets - for example, 18 | messages in the causal order of a session transcript may instead be sent over a 19 | different transport that offers more protection against metadata analysis. Such 20 | hybrid strategies may be researched in further depth elsewhere. 21 | 22 | Models and assumptions 23 | ====================== 24 | 25 | Before we begin to describe our scheme, we'll clarify our assumptions about the 26 | context that we're working in. The models we define are not completely formal, 27 | but should be precise enough to understand the rest of the chapter with. 28 | 29 | Group transport channel 30 | ----------------------- 31 | 32 | We model the transport channel as a globally consistent sequence of events, 33 | where each event at index i is associated with a set of *channel members*, 34 | denoted by mem[i]. An event may be a packet [#Npket]_ with mem[i] = mem[i-1], 35 | or else it is a member-change, that explicitly encodes a change to the channel 36 | membership. A packet is visible to mem[i], and a member-change is visible to 37 | both mem[i-1] and mem[i]. (The indexes are only indicative, for the purposes of 38 | this document, and need not be visible to the members nor otherwise exposed.) 39 | 40 | From each member m's point of view, they see a member-change that adds them to 41 | the channel, some packets or member-changes *not* acting on themself, then a 42 | member-change that removes them from the channel. The next event, if any, would 43 | be a member-change that adds them to the channel again, and this cycle can 44 | repeat any number of times. 45 | 46 | These events need not be authenticated, at this layer, between the members. 47 | Further, this model only constrains events *received* by members, and is 48 | neutral to how events *sent* by members are handled. For example, a member may 49 | send a packet P immediately after he receives event i with mem[i] = M, i.e. 50 | intending to send P to M, but P may be added to the channel event sequence at a 51 | later index j > i+1 where mem[j] |ne| M. Our scheme may be viewed as a 52 | user-level (end-to-end) method of addressing these security deficiencies of the 53 | transport, which cannot be satisfied on the transport layer itself. 54 | 55 | It is not necessary to specify exactly the mechanism for achieving this model. 56 | However, a typical scenario would be a dumb server that accepts all events from 57 | all clients, and echoes them back reliably to everyone in the same total order. 58 | This is quite standard for existing instant messaging protocols such as XMPP, 59 | and requires no extra logic on the server side. Therefore, clients can 60 | transparently run our scheme on top of existing infrastructure. [#Nprot]_ 61 | 62 | What we *do* expect however, is that the mechanism takes responsibility for 63 | delivering, reliably and in a consistent order, events to its relevant visible 64 | users. We do not *assume* this to be the case; any deviation from it is 65 | detected by our scheme, and regarded as incorrect behaviour of the mechanism 66 | (e.g. the server). But in the case where they are met, the scheme works under 67 | these assumptions to be efficient and robust. 68 | 69 | For our hybrid order scheme, we only require that certain packets are sent over 70 | this channel, i.e. initial packets and final-candidate packets, defined later. 71 | 72 | .. [#Npket] We use the term "packet" to refer to transport-level packets; the 73 | term "message" refers specifically to verified-decrypted messages, that 74 | comprise the causal order history graph. 75 | 76 | .. [#Nprot] Note that some protocols may use different concrete sequences of 77 | protocol "event" packets, but logically these are equivalent to the above 78 | model. For example, in XMPP an "I entered the channel" event, in practise 79 | consists of many enter events for each member in the channel, from just 80 | before I enter it, then a single enter-event of me entering the channel. 81 | One can write software adapters to translate between these differences. 82 | 83 | Group membership operation 84 | -------------------------- 85 | 86 | We model the operation as a process G on the (logical) session membership. G is 87 | initialised from some previous state - either a null state, or the result of 88 | the previous operation - that encodes the previous session membership, M'; and 89 | some initial packet pI received from the channel, sent by a member of M', which 90 | defines the intended next session membership, M. [#Noini]_ 91 | 92 | Depending on the packets received by G, the operation may finish with success 93 | ("completes"), upon which the session membership atomically changes to M and 94 | the result state is stored for the next operation; or with failure ("fails"), 95 | upon which it remains at M' and all temporary state related to G is destroyed. 96 | 97 | We assume that G handles authentication and ordering of its own packets, that 98 | only fresh packets (relative to pI) are accepted, and that packets will not be 99 | misinterpreted to be part of another operation, that has a different initial 100 | packet. If the operation finishes, then this is authentic; if it completes, 101 | then any resulting state is also authentic. G defines the subject of these 102 | authentications - e.g. by all members in M, or only by the initiator. We 103 | assume that the people who choose which particular G to plug into our scheme, 104 | make a sensible decision that fits their security requirements. 105 | 106 | A corollary is that packets that belong to different operations have different 107 | content (i.e. are unique). This should not be difficult; indeed otherwise it 108 | would be susceptible to replay attacks. Duplicate packets should be ignored. 109 | 110 | For a general G, the packets required to finish the operation may be ordered 111 | differently across members. For example, in a final round where everyone must 112 | respond to previous values, it doesn't matter what order we receive these in; 113 | G can still complete in the same way for everyone. 114 | 115 | However, for our scheme to remain relatively simple, we place an additional 116 | restriction. Define a "final-candidate" packet as one which *may* finish G in 117 | some linearisation of the causal order of G's packets. Then, we require that 118 | these packets must be sent over the group channel, as opposed to out-of-band. 119 | We use this fact to define an implicit agreement mechanism, so that members can 120 | determine which particular packet finishes G. We call this final packet pF. 121 | Given a channel event sequence that "almost" finishes G, there may be several 122 | possible packets appended to this, that could each actually finish G; we call 123 | these "final packet proposals"; the agreement mechanism defines which one is 124 | actually accepted, and hence the result of the operation. 125 | 126 | As with everything in the channel, pF is defined only after the packet is 127 | *received*. For example, suppose we have a G that expects a final round of 3 128 | packets {F[A], F[B], F[C]}, and C receives F[A] and F[B]. Now, she has all 129 | information needed to generate F[C] and complete the operation, but she must 130 | not execute the completion yet, since there may be an fail packet A added to 131 | the channel event sequence *before* F[C]. In this case, everyone including C 132 | must fail G, and pF is A and not F[C]. 133 | 134 | In terms of who is able to decode and identify initial or final packets, we 135 | assume that: 136 | 137 | - | for all pI with a given prev_pF: only { all of M'\|M } can identify it, or 138 | | for all pI with a given prev_pF: everyone can identify it 139 | - | for all pF with a given prev_pI: 140 | | if pF success: only { all of M'\|M } or only { all of M } can identify it 141 | | if pF failure: only { all of M'\|M } can identify it 142 | 143 | No other partitions are allowed, i.e. a process G may not be such that e.g. for 144 | a pIa with a given prev_pF, M'\|M can identify it, but for a pIb on the same 145 | prev_pF, everyone can identify it; nor that e.g. some members of X := M'-M can 146 | identify a pF, but others cannot. 147 | 148 | For the pI case, the arrangement of the { or, for all } quantifiers simplify 149 | the cases we have to analyse. For the pF case, the arrangement is more complex, 150 | but the simpler form would be *too* restrictive, as explained later. Typical 151 | real-world instances of G already fit these constraints; we just specify them 152 | more precisely here to benefit further discussions. These constraints are 153 | justified further in :ref:`hybrid-partial-vis`. 154 | 155 | .. [#Noini] If the membership protocol supports outsider initiation (i.e. join 156 | as well as invite), it is the *reply* to the join request that is treated 157 | as the pI within our scheme here. This fits our requirement that pI be 158 | authored by member of the previous session (if any), who knows the previous 159 | state associated with it. 160 | 161 | Ordering and consistency 162 | ======================== 163 | 164 | .. _hybrid-context-preserve: 165 | 166 | Context preservation 167 | -------------------- 168 | 169 | By our assumptions of G, the context of a final packet is already preserved, 170 | since G will only interpret it within the scope of the initial packet. To 171 | preserve the context of an initial packet, we add to it explicit references to 172 | that context, i.e. the following information: 173 | 174 | prev_pF: 175 | packet-id (defined later) of last accepted pF 176 | pmId: 177 | latest message-ids seen, in sub-session derived from pF 178 | 179 | This lets us convey our context to R := M'&M, the members that remain in the 180 | session. It is unnecessary to convey it to excluded members, since they are no 181 | longer part of the session; nor included members, since the context is not 182 | visible to them. [#Npctx]_ If there was no previous sub-session, then we have 183 | neither context, nor anyone to convey it to (since R = |emptyset|), so we can 184 | set a new random value for pF (for uniqueness, which might be useful later), 185 | and set pmId = |emptyset| too. 186 | 187 | These references, of course, must be authenticated by the operation initiator. 188 | If G supports "additional authenticated data", we can simply use this feature. 189 | Otherwise, perhaps such a feature can be added. For example, some protocols 190 | retroactively authenticate the protocol version to prevent downgrade attacks; 191 | the same mechanism could be adapted to authenticate arbitrary data. Typically 192 | this is not private, but as argued elsewhere, our scheme does not try to 193 | protect the privacy of the existence of session membership changes, and these 194 | references are meant to be hashes that don't leak information about the content 195 | of the session. 196 | 197 | If it is not feasible to achieve in-operation authentication, a fallback is to 198 | resend this information as part of a message in the newly-created authenticated 199 | session. Others should expect this message, abort the session if it is not 200 | received within some timeout, or verify it against the older unauthenticated 201 | claims if it is received. This keeps the membership operation component more 202 | decoupled from the rest of the system. However, it would take longer to achieve 203 | our desired security property. 204 | 205 | .. [#Npctx] Another option is to refer to earlier context, from when they were 206 | last in the session. There are many issues to be researched here, e.g. what 207 | if they were previous in the session, but the inviter does not know this. 208 | 209 | Agreement 210 | --------- 211 | 212 | Now that we have context preserving pI and pF packets, we need a way select a 213 | subset of these that forms a total order. [#Ncord]_ Equivalently, we need an 214 | agreement algorithm, consistent across all members, to accept a single pI/pF 215 | that follows a pF/pI, and reject other proposals. 216 | 217 | .. digraph:: hybrid_agreement 218 | 219 | rankdir=RL; 220 | node [height=0.5, width=0.5, style=filled, label=""]; 221 | i1a [fillcolor="#66ff66", label="+a"]; 222 | f1a [fillcolor="#ff6666", label="s"]; 223 | f1b [fillcolor="#66ff66", label="f"]; 224 | f1c [fillcolor="#ff6666", label="f"]; 225 | i2a [fillcolor="#ff6666", label="+b"]; 226 | i2bf [fillcolor="#66ff66", label="-c,s"]; 227 | i3a [fillcolor="#66ff66", label="+b"]; 228 | f2a [fillcolor="#ff6666", label="f"]; 229 | f1a -> i1a; 230 | f1b -> i1a; 231 | f1c -> i1a; 232 | i2a -> f1b; 233 | i2bf -> f1b; 234 | f2a -> i2bf; 235 | i3a -> i2bf; 236 | 237 | The above diagram helps visualise this. Nodes labelled "-m" or "+m" are pI 238 | packets that include or exclude a member from the session; and those labelled 239 | "s" or "f" are success or failure pF packets linked to that operation. Green 240 | nodes are accepted proposals, and all other nodes are red, meaning a rejected 241 | proposal. For every accepted proposal, there is at most one accepted proposal 242 | that references it. In the above example, a proposal to include *a* is accepted 243 | but the operation fails, and is followed by a proposal to exclude *c* which is 244 | accepted and succeeds immediately without requiring more packets; further 245 | proposals to fail "-c" are rejected, and then a second proposal to include *b* 246 | is accepted after the first one was rejected earlier. 247 | 248 | Our agreement algorithm, ignoring a few special cases mentioned later (rule EIO 249 | and EII), is as follows: for a given parent pF/pI, we accept the first pI/pF 250 | proposal in the channel event sequence that references it, except that: 251 | 252 | Rule XP: 253 | We ignore proposals that are added to the channel when the channel membership 254 | does not include its target membership, i.e. if M |NotSubsetEqual| mem[i]. 255 | 256 | This rule exists because we want everyone in M to reach the same answer. This 257 | is not possible if some of them are not in the channel when the proposals are 258 | echoed back. Given this rule, it is therefore wise, *before* one sends out a 259 | proposal at index i, to check that M |subseteq| mem[i] - though of course this 260 | does not guarantee that this is actually added to the channel at an index j 261 | where M |subseteq| mem[j]. 262 | 263 | We don't worry about non-(initial or final-candidate) packets - if any are 264 | missing from the channel, then G itself decides whether to finish or not. 265 | Everyone should reach the same conclusion, as long as the transport works 266 | according to our expectations. 267 | 268 | This should also work in the degenerate case of a 1-packet operation (e.g. with 269 | a key dictator). Here, a packet is both a pI proposal and a pF that is accepted 270 | immediately, before futher channel events are processed. Implementors may need 271 | to handle this specifically; but this should not cause major difficulties. 272 | 273 | The advantage of this agreement algorithm is that it requires no further round 274 | trips between members, beyond what was going to be sent by G anyway. Members 275 | must wait one round-trip before they know their proposals are accepted, but 276 | this is necessary for any total order that preserves context. 277 | 278 | One might argue that, if agreement isn't reached, then our consistency checks 279 | below will emit warnings already - so this complexity is unnecessary. However, 280 | those checks are meant to detect *incorrect behaviour* of the transport; but 281 | concurrency conflicts can happen *even when* the transport behaves as expected. 282 | This agreement scheme aims to reduce *false negatives*, which is vital both for 283 | a good user experience, and to avoid training users to ignore warnings. 284 | 285 | One attack that a server can execute is to block operations without detection. 286 | When the victim sends a proposal, the server first passes it out-of-band to a 287 | co-operating insider who generates a conflicting proposal. Then, it echoes this 288 | proposal before the victim's, causing members to reject the latter. This is all 289 | within the bounds of normal behaviour of the group transport, so the attack is 290 | not detectable and therefore a problem. For now, we ignore this, since this 291 | power is inherent to the idea of a server-dictated total order, and the need to 292 | have a co-operating insider increases the cost of this attack. Nevertheless, 293 | this is not ideal, and we welcome suggestions for improvements. 294 | 295 | .. [#Ncord] That is, over the context-preserving ordering relation, based on 296 | explicit references, not the group channel ordering relation. 297 | 298 | Consistency 299 | ----------- 300 | 301 | As each member accepts proposals, they calculate a "chain hash" (CH) for it: 302 | 303 | pId: 304 | H(packet data||sender||recipients) \ 305 | CH(pId): 306 | H(CH(pId of previous accepted proposal) || pId || pId-type) 307 | 308 | The encoding should be unambiguous; the actual implementation may add extra 309 | padding characters beyond simple concatenation, if this is required. 310 | 311 | We include the (unauthenticated) channel sender and recipients in the hash for 312 | the packet-id, so that our consistency checks also cover this information. This 313 | is necessary, because our agreement algorithm uses that information. If we omit 314 | them, then the transport could claim different memberships to different 315 | members, which may cause them to reach different results when running the 316 | algorithm, but still calculate the same packet-id. 317 | 318 | The CH of an accepted proposal packet, as calculated by a member, attests to 319 | all proposals accepted by them up to and including it. "pId-type" attests to 320 | how they interpret the packet, in case this might be different across members 321 | depending on local state, environment, etc. Here, we use "1" for pI, "2" for 322 | pF, and "3" for pI+pF packets. 323 | 324 | As with other things, these may only be calculated when a packet is *received* 325 | from the channel. Further, since packet contents are unique, these packet-ids 326 | are also unique. [#Nhash]_ We ignore rejected packets, because we don't use 327 | information contained in them to make decisions; and this also makes things 328 | easier later when we deal with partial visibility issues. 329 | 330 | For incoming new members to be able to match up with everyone's CH values, 331 | every pI should also contain: 332 | 333 | prev_CH: 334 | chain hash of prev_pF = CH(prev_pF) 335 | 336 | When an incoming member accepts their first pI, they may read the previous CH 337 | from this record, trusting it opportunistically until the next consistency 338 | check (described below). When starting a new session from scratch with a random 339 | prev_pF (as defined in :ref:`hybrid-context-preserve`), we can use any value 340 | here; in practise we arbitrarily use prev_CH = H("" || prev_pF || 0xFF). 341 | 342 | Chain hashes would be unnecessary if we can be sure that every accepted packet 343 | is cryptographically bound to the previous accepted packet i.e. an attacker 344 | cannot trick anyone into interpreting it as referencing a different one. Most G 345 | do bind packet *contents* thus, but we must also hash in information about the 346 | *channel members* which is typically ignored. So chain hashes *are* necessary 347 | here. Another benefit is that we avoid relying on properties of G. 348 | 349 | When operation finishes, everyone in the resulting session (i.e. the previous 350 | members for a successful finish, or the next members for a failed finish) 351 | authenticates and sends the following information: 352 | 353 | ( last accepted pId, CH(pId) ) 354 | 355 | This could be done as part of a message in our (authenticated) causal order 356 | history graph. This essentially corresponds to an "ack" that we described 357 | previously, but for the total order of accepted proposals. Then, we may ensure 358 | consistency by checking that we receive full-acks for every accepted proposal, 359 | similar to as described in :doc:`../causal/02-consistency`. 360 | 361 | Why is this mechanism necessary? Surely G already authenticates its result 362 | identically across all members? Well, firstly, we don't assume that G does this 363 | by *all members*, only by some party that is satisfactory. Secondly, *even if* 364 | G does authenticate the result (of a particular pF) by all members, whether it 365 | is actually accepted by each member is outside of its control. For example, two 366 | members may be able to send two different pF proposals, that can each cause G 367 | to reach different but valid results. The transport could arrange itself so 368 | that some members receive one proposal and some the other, causing the session 369 | to split outside of G's control. We could specify that G should prevent this, 370 | but this is generally not a cryptographic problem. [#Ngcon]_ Instead, we do 371 | this consistency check that works regardless of the guarantees that G makes. 372 | 373 | .. [#Nhash] Our definition allows an outside observer to potentially calculate 374 | pIds and CHs. This is not a problem for anything we describe, but may cause 375 | a problem for applications that build on top of us. One may use a different 376 | definition, but it would need to be done based on how packets are encoded, 377 | which is specific to the membership operation. If it is necessary, one may 378 | adapt :ref:`encoding-message-identifiers` to apply instead to this context. 379 | 380 | .. [#Ngcon] We would have to make some statement about the second-to-final 381 | packet, and how it must depend on all members, or something like that. 382 | 383 | .. _hybrid-partial-vis: 384 | 385 | Partial visibility 386 | ================== 387 | 388 | Of initial proposals 389 | -------------------- 390 | 391 | Earlier, we required that pI proposals must be identifiable (decodable, 392 | interpretable) by all of M'\|M (i.e. using any secret information they have) or 393 | everyone (i.e. without any secret information). Let's look at this in more 394 | detail. We consider the following cases: 395 | 396 | 1. for all pI with a given prev_pF: only { all of M' } can identify that pI 397 | 2. for all pI with a given prev_pF: only { all of M } can identify that pI 398 | 3. for all pI with a given prev_pF: only { all of M'\|M } can identify that pI 399 | 4. for all pI with a given prev_pF: everyone can identify that pI 400 | 401 | M' is specific to the prev_pF and constant across all pI, and M is specific to 402 | each pI. Other cases are too complex to think about, or trivially useless. 403 | 404 | With (1), the included members, I := M-M' cannot run the agreement algorithm 405 | and therefore we require further packets to tell them that they are actually 406 | supposed to be participating in G. So we forbid this possibility, because we 407 | are aiming for something with no extra packets. Generally, existing instances 408 | of G don't hide pI from I anyway, so we don't lose much with this. 409 | 410 | With (2), different sets of pI (with the same prev_pF) are visible to different 411 | members, since each pI proposes different X := M'-M. It does not seem feasible 412 | to be able to tweak the agreement algorithm to work around this, since there is 413 | no set of members-who-can-see-a-pI that is constant across all these pI. For 414 | example, the auto-kick solution suggested in (3) fails because of the lack of a 415 | constant M' that is able to derive the same actions to execute. So we forbid 416 | this possibility as well. TODO: add the session-split example. 417 | 418 | With (3), different sets of pI (with the same prev_pF) are visible to different 419 | members, since each pI proposes different M. So (across all these pI) different 420 | members would reach different results for the agreement. To work around this, 421 | we specify that in this case, after a given pI is accepted by M', they must all 422 | try to kick extra members not in M, and these latter members should interpret 423 | this as "another proposal, with the same prev_pF, for a G that does not include 424 | us, was accepted". Unlike in (2), the M' are constant (being determined by the 425 | same pF), so this is consistent across all members. 426 | 427 | With (4), this is the simplest case, and we don't need to handle it specially, 428 | but it leaks information about session membership changes. However, arguably, 429 | in the context of our scheme, this does not cause *extra leakage* - the server 430 | has this metadata anyway, in the form of channel membership changes. If we 431 | require membership changes to be private, we will need to achieve this some 432 | other way, i.e. not using a hybrid ordering on top of a server transport. 433 | (TODO: what about group channels that fit our model, but not using a server?) 434 | 435 | Of final proposals 436 | ------------------ 437 | 438 | For the case of pF proposals, we have the similar cases as above, with slightly 439 | different arrangements of quantifiers. For all pF with a given prev_pI, one of 440 | the following might apply: 441 | 442 | 1. only { all of M' } can identify that pF 443 | 2. only { all of M } can identify that pF 444 | 3. only { all of M'\|M } can identify that pF 445 | 4. everyone can identify that pF 446 | 447 | We can immediately forbid (1) for the same reason as in the previous section, 448 | and forbid (4) as being unnecessary - nobody outside of M'\|M needs to identify 449 | pF packets, only pI ones, as per `hybrid-own-enter-channel` - it gives us no 450 | advantage to (3), and we leak less information that way. 451 | 452 | For a success pF, it is reasonable and common that G has the security property 453 | that only M may verify (and therefore identify) its final packets. In concrete 454 | terms, flipping a bit to turn a valid final packet into an invalid one, may be 455 | indistinguishable to anyone not in M. This is case (2), so we must support it. 456 | 457 | For a failure pF, M' *must* be able to identify pF, since they remain in the 458 | session and will need to refer to this in any subsequent pI proposals. Note 459 | that this implies that we forbid "implicit failures" - e.g. in the "bit flip" 460 | scenario mentioned above, an invalid final packet *must not* be accepted as a 461 | failure pF, because it is not identifiable by X := M'-M. Instead, it should be 462 | ignored, and a *separate* failure pF proposal be sent. 463 | 464 | Since different pF are identifiable by different members, we must extend our 465 | agreement algorithm to ensure everyone still reaches the same conclusion. The 466 | extension is similar to the one from (3) in the previous section. If a success 467 | pF is accepted by M, they must all try to kick X, and X should interpret this 468 | as "G was completed to exclude us". This deals with the case where X receives a 469 | failure pF after a success pF (that they can't identify), which may cause them 470 | to incorrectly conclude that G failed. 471 | 472 | Session and channel interactions 473 | ================================ 474 | 475 | Entering a channel 476 | ------------------ 477 | 478 | .. _hybrid-own-enter-channel: 479 | 480 | The agreement algorithm described so far (including extensions) for a pI 481 | assumes that everyone knows the full event sequence from its prev_pF up to it. 482 | However, this is not the case for new members that entered a channel after the 483 | prev_pF - they don't know if other proposals were sent before they entered. 484 | 485 | As a new member entering the channel, the next pI proposal we see, if it is not 486 | already ignored due to rule XP, may be divided into a few cases: 487 | 488 | Rule EIO: 489 | It isn't trying to include us. Then, we ignore it, but assume that others 490 | will accept it or have already accepted another packet with the same prev_pF. 491 | So we store prev_pF and reject any future proposals that reference it. 492 | 493 | Rule EII: 494 | It is trying to include us, and its prev_pF has not yet been blacklisted by 495 | rule EIO. Then, we accept it opportunistically: 496 | 497 | - If we have seen prev_pF, then we know this acceptance is correct. 498 | - Otherwise, this could be incorrect - another proposal that references the 499 | same prev_pF could have been accepted before we entered the channel. But in 500 | this case, the server-order consistency check would fail later, or (if G is 501 | contributive) we would not be able to complete the operation, or someone 502 | would kick us because we're in the channel at an inappropriate time. 503 | 504 | For others that enter the channel: 505 | 506 | Rule ES: 507 | If another member enters outside of a membership operation, notify the local 508 | user that they would like to join the session. 509 | 510 | Rule EO: 511 | If another member enters during a membership operation, then (assuming we've 512 | already applied rule XP) they are not part of the pending new sub-session, so 513 | kick them. Optionally, notify the local user that they probably want to join 514 | the channel, so that they can manually invite them. 515 | 516 | Leaving a channel 517 | ----------------- 518 | 519 | If we leave a channel for whatever reason, we can no longer be sure that we 520 | didn't miss any packets. Therefore: 521 | 522 | Rule LI: 523 | If we leave the channel, we must clear all state to do with the session. This 524 | is simpler to reason about than trying some complicated recovery logic, 525 | especially in terms general to *any* membership operation. 526 | 527 | For others that leave the channel: 528 | 529 | Rule LS: 530 | If another member leaves outside of a membership operation, propose an 531 | operation to exclude them. 532 | 533 | Rule LOI: 534 | If another member leaves during an operation that involves including or 535 | keeping them in the session, then this is interpreted as a pseudo pF packet 536 | which is immediately accepted as a *failure* of the operation - since we know 537 | that the leaver is unable to complete it, having cleared all state. 538 | 539 | The pId of this pseudo-packet is defined as: 540 | 541 | | pId = H(prev_pI||"leave"||remaining channel members) 542 | 543 | Rule LOX: 544 | If another member leaves during an operation that involves excluding or 545 | keeping them in the session, and it fails (for the latter case, this is 546 | inevitable due to rule LOI), then propose an operation to exclude them. 547 | 548 | The following rules are about entering, but the reasoning behind them demands 549 | that we discuss them *after* the above rules about leaving: 550 | 551 | Rule EAL: 552 | When trying to exclude a member from the cryptographic session, who already 553 | left the channel: if this member re-enters the channel before we exclude them 554 | from the session, we can just auto-kick them until we exclude them. 555 | 556 | The above auto-exclusion rules, together with rule XP, should ensure that we 557 | mostly don't get to a state where a member is in the channel and expects to be 558 | included, but no-one tries to do this. 559 | 560 | If everyone that wants to exclude them leaves the channel, then there are a few 561 | race conditions where they would never be kicked. [#Nrace]_ So, they should 562 | leave and re-enter the channel after a timeout. When they leave, existing 563 | members will notice it and later try to exclude them, which is the corrected 564 | behaviour that we want. 565 | 566 | .. [#Nrace] For example: 567 | 568 | 1. X leaves and re-enters. X is still part of others' sessions, but X has 569 | cleared that session (rule LI) and expects to be included into a new one. 570 | 2. G accepted to add Y, G completes, Y now part of session and thinks X is. 571 | 3. X doesn't get auto-kicked for whatever reason (e.g. slow transport) 572 | 4. Everyone that wants to exclude X leaves (e.g. disconnected), but X remains 573 | in the channel. 574 | 5. Y remains in session, X still in Y's old session, but (as in step 1) 575 | expects to be included into a new session. 576 | 577 | Excluded from a session 578 | ----------------------- 579 | 580 | When we exclude someone, as prescribed previously, they stay in the channel 581 | until they are kicked, since (in the general case, we must assume) they can't 582 | identify a success pF. Then, their state is inconsistent with other members, so 583 | we can't re-include them, until they've left the channel and reset it. In terms 584 | of a rule: 585 | 586 | Rule IAL: 587 | When someone tries to include a member into the session, who was previously 588 | excluded but not yet left the channel, auto-kick them from the channel. 589 | 590 | If everyone that wants to kick them leaves (e.g. disconnected), then they would 591 | never be kicked. So, they should auto-leave the channel after a timeout, in 592 | case this happens, ideally after consistency is reached or itself times out. 593 | -------------------------------------------------------------------------------- /appendix/index.rst: -------------------------------------------------------------------------------- 1 | ========== 2 | Appendices 3 | ========== 4 | 5 | Subtopics: 6 | 7 | .. toctree:: 8 | :glob: 9 | 10 | * 11 | -------------------------------------------------------------------------------- /apt-get-build-dep.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | sudo apt-get install python-sphinx graphviz 3 | -------------------------------------------------------------------------------- /causal/01-ordering.rst: -------------------------------------------------------------------------------- 1 | ======== 2 | Ordering 3 | ======== 4 | 5 | .. include:: 6 | .. include:: 7 | 8 | Data structures 9 | =============== 10 | 11 | Partial orders 12 | -------------- 13 | 14 | A **partial order** (`wikipedia 15 | `_) is a set of elements 16 | with a binary relation |le| that is reflexive, antisymmetric, and transitive. 17 | For convenience, we also define |ge| (`after`) such that a |ge| b |iff| b |le| 18 | a, and |perp| (`independent`) such that a |perp| b |iff| (¬ a |le| b |and| ¬ a 19 | |ge| b). Note that in a partial order, ¬ (a |le| b) does *not* imply a |ge| b; 20 | this is true however for `total orders` a.k.a. `linear orders`. 21 | 22 | This can be visualised as a `directed acyclic graph`_ (DAG), where nodes 23 | represent the elements, and paths represent the relationships: 24 | 25 | .. digraph:: relations 26 | 27 | rankdir=RL; 28 | node [height=0.5, width=0.5, style=filled, label=""]; 29 | m [fillcolor="#ffffff", label="m"]; 30 | p [fillcolor="#ff6666"]; 31 | q [fillcolor="#ff6666"]; 32 | s [fillcolor="#66ff66"]; 33 | t [fillcolor="#66ff66"]; 34 | x [fillcolor="#999999"]; 35 | y [fillcolor="#999999"]; 36 | z [fillcolor="#999999"]; 37 | s -> m -> p; 38 | t -> m -> q; 39 | s -> x -> p; 40 | z -> q; t -> y; 41 | 42 | In the diagram above, where an edge a |leftarrow| b means that a |le| b, 43 | 44 | - red nodes (and m [#lang]_) are `before` m; all others are `not-before` m. 45 | - green nodes (and m) are `after` m; all others are `not-after` m. 46 | - grey nodes are neither before nor after m, but `causally independent` of it. 47 | 48 | Using this, we begin to define our data model and its "real world" semantics. 49 | Each node m |in| G represents a message [#rxtx]_; a |le| b (a `before` b) means 50 | "the author of b knew the contents of a, when they wrote b". Transitivity also 51 | implies that they knew all the messages before a as well (i.e. anc(a), see 52 | below), when they wrote b. We'll talk about how to *enforce* these semantics 53 | soon; for now we continue with its *structural properties*: 54 | 55 | To reduce visual clutter later, let's further define: 56 | 57 | **anc[G](m)** = {a |in| G | a |le| m} -- m is a message, G is a graph or subgraph 58 | The set of ancestors of m, including m. This is the `possible transitive 59 | causes` of m, like the discrete equivalent of a `light cone`_. Sometimes it's 60 | obvious which G we are talking about, in which case we'll omit [G] and write 61 | simply anc(m). [#ancg]_ 62 | 63 | **max(G)** = {a |in| G | |forall| v |in| G: ¬ v > a} -- G is a graph or subgraph 64 | The latest messages in G. Note that unlike with total orders, this is not a 65 | single element, but a set of nodes that are all causally independent of each 66 | other, i.e. form an `anti-chain`_. 67 | 68 | In our scheme, each message m declares its `immediate predecessors` (`parents`) 69 | P = **pre[G](m)**. This is why we drew "|leftarrow|" for |le| earlier, rather 70 | than "|rightarrow|" - the references belong to the later message; one cannot 71 | predict in advance what messages will come after yours. `Predecessors` means 72 | that |forall| p |in| P: p < m, and `immediate` means that there is nothing 73 | between them, i.e. |NotExists| q: p < q < m. 74 | 75 | The reason we mandate `immediacy` is to eliminate redundant information and 76 | simplify security checks; this will be explained in more detail at the end of 77 | the next section. For now, note that one consequence of `immediacy` is that all 78 | parents form an anti-chain, and that pre(m) |equiv| max(anc(m) \\ {m}). Over 79 | the whole session, it means that messages (nodes) and their parent references 80 | (edges) form a `transitive reduction`_ over the |le| relationship. 81 | 82 | .. _Directed acyclic graph: https://en.wikipedia.org/wiki/Directed_acyclic_graph 83 | .. _Light cone: https://en.wikipedia.org/wiki/Light_cone 84 | .. _Anti-chain: https://en.wikipedia.org/wiki/Antichain 85 | .. _Transitive reduction: https://en.wikipedia.org/wiki/Transitive_reduction 86 | 87 | .. [#lang] In natural language, |le| is really `before or equal to`. We say 88 | `before` here because that's shorter. We use |le| instead of < because that 89 | makes most formal descriptions shorter. Unless otherwise specified, we'll 90 | use the terms `before` for |le| and `strictly before` for <, and likewise 91 | for `after` and |ge|. 92 | 93 | .. [#rxtx] Some other systems treat send vs deliver (of each message m) as two 94 | separate events, but we don't do this for simplicity. We mandate that the 95 | author delivers m to itself *immediately* after they send it to others, so 96 | nobody will never see an event that is after send[m] but not-after 97 | deliver[m], nor an event that is before deliver[m] but not-before send[m]. 98 | 99 | .. [#ancg] Specifically, either we'll talk about one graph at a time, or we'll 100 | talk about different versions of the same history graph G₀ |subseteq| G₁ 101 | where anc[G₀](m) |equiv| anc[G₁](m). This is broken when we finally get to 102 | :doc:`05-visibility` so we'll bring back the [G] notation. This also 103 | applies to the other terms we define here of the form xxx[G](m). 104 | 105 | Causal orders 106 | ------------- 107 | 108 | Now we formally introduce agents that can observe and generate events, that we 109 | mentioned informally when discussing the real world semantics of |le|. 110 | 111 | In a **causal order**, each event m has exactly one **author(m)** and a set 112 | **readers(m)**, both together called its **members(m)**. We also define: 113 | 114 | **by[G](u)** 115 | The set of all messages authored by u in G. This is totally-ordered on |le| 116 | i.e. every element is either before or after every other, so it can be 117 | represented as a list. This property constrains the number of branches that 118 | may exist in any G, and helps it be "more linear", and we can use this to 119 | prove performance bounds on various operations on G. 120 | 121 | Unlike with anc[*](m) and pre[*](m), this is generally not constant even if 122 | G₀ |subseteq| G₁, so notation-wise we'll omit [G] less often. 123 | 124 | **context[G](m)**: members(m) |rightarrow| anc(m) = u |mapsto| max1(by[G](u) |cap| anc[G](m)) 125 | This is a mapping (a.k.a dictionary, associative array) from each member of m 126 | (call this u), to: 127 | 128 | - m itself, if u is the author of m; or else 129 | - the latest message authored by u, that was seen by the author of m when 130 | they wrote m; or else 131 | - |bot| (`null`) if no such message exists. 132 | 133 | Above, max1() gives the (single) maximum element of a totally-ordered set, or 134 | |bot| if it is empty. 135 | 136 | As with anc[*](m) and pre[*](m), notation-wise we'll often omit [G] for the 137 | same reasons. [#ancg]_ 138 | 139 | Context is semantically equivalent to a `vector clock`_ - a mapping from each 140 | entity to some representation of the latest event that the clock's owner saw 141 | from them. As with vector clocks, this contains redundant information, and may 142 | be locally calculated recursively from pre(m) as follows: 143 | 144 | context(m)[u] = 145 | - if author(m) = u : m 146 | - else: max1({ context(p)[u] | p |in| pre(m) } \\ {|bot|}) 147 | 148 | TODO: probably move most of the rest of this section to the appendix, it 149 | doesn't add that much insight or new constructive models. 150 | 151 | But watch out! A minor "optimisation" will make this susceptible to re-ordering 152 | attacks: 153 | 154 | context(m)[u] = 155 | - if author(m) = u : m 156 | - else if (|exists| p |in| pre(m): author(p) = u) : p -- **insecure optimisation** 157 | - else: max1({ context(p)[u] | p |in| pre(m) } \\ {|bot|}) 158 | 159 | The attack allows malicious authors to `rewind` the context they are supposed 160 | to declare with each message; vector clocks can also suffer from this if the 161 | implementation makes too many naive assumptions. 162 | 163 | The problem stems from trusting pre(m) too much in the latter definition. For 164 | example, C should not be allowed to claim that their last-received-from-A is 9, 165 | if they've already claimed that their last-received-from-B is 12, but in 166 | message 12, B claimed that their last-received-from-A is 10. A more simple 167 | version of this with only 2 members is in the diagram below: 168 | 169 | .. digraph:: freshness_consistency 170 | 171 | rankdir=RL; 172 | node [height=0.5, width=0.5]; 173 | edge [weight=2]; 174 | 175 | subgraph clusterA { 176 | label="by(A)"; 177 | labeljust="r"; 178 | 2 -> 1; 179 | node [style=invis]; 180 | r -> s -> 2 [style=invis]; 181 | } 182 | 183 | subgraph clusterB { 184 | label="by(B)"; 185 | labeljust="r"; 186 | 4 -> 3; 187 | node [style=invis]; 188 | 3 -> p -> q [style=invis]; 189 | } 190 | 191 | edge [weight=1]; 192 | 3 -> 2; 193 | 4 -> 1 [color="#ff0000", headport=se, tailport=nw]; 194 | 195 | We must forbid the 1 |leftarrow| 4 reference. More generally, we want context 196 | to always advance, never rewind. We'll call this *context consistency*, and 197 | formally it means: 198 | 199 | | |forall| m', m |in| G: m' < m |implies| context(m') |sqsubset| context(m) where 200 | | c' |sqsubset| c (c' is `strictly-less-advanced` than c) is defined as: 201 | | c' |ne| c |and| |forall| u: c'(u) = |bot| |or| c'(u) |le| c(u) 202 | 203 | A related concept is `parent consistency` - we must never reference parents of 204 | strictly-earlier messages, since it is redundant and we should instead simply 205 | reference the strictly-earlier message itself. Formally: 206 | 207 | |forall| m', m |in| G: m' < m |implies| |forall| p' |in| pre(m'), p |in| 208 | pre(m): ¬ p |le| p' 209 | 210 | These properties are quite complex. But we don't have to check them directly: 211 | 212 | **Theorem**: `transitive reduction` entails `parent consistency`. Proof sketch: 213 | if m' < m then |exists| p |in| pre(m): m' |le| p < m. Since |le| is transitive, 214 | |forall| p' |in| pre(m'): p' < p |equiv| anc(p') |subset| anc(p). By transitive 215 | reduction, no other q |ne| p |in| pre(m) may belong to anc(p), i.e. |forall| q 216 | |ne| p |in| pre(m): q |notin| anc(p') |equiv| ¬ q |le| p'. Earlier we had p' < 217 | p so this holds even if q = p, i.e. |forall| q |in| pre(m): ¬ q |le| p'. ∎ 218 | 219 | **Theorem**: `parent consistency` entails `context consistency`, even if using 220 | the insecure optimisation above. (If using our original secure definitions, it 221 | holds regardless of `parent consistency`.) Proof sketch: this is long-winded 222 | and not interesting or insightful; do it later and put it in the appendix. 223 | 224 | Based on this result, we mandate that real encodings of m *must not* encode 225 | context(m) but only pre(m), and readers must check that it is an anti-chain 226 | (see below) before committing it to their local history graph. This way, we get 227 | the other two properties for free. If needed, we can calculate context(m) from 228 | pre(m) locally, and this remains secure regardless of naive optimisations. 229 | 230 | As a general principle, redundant information is a security liability: it must 231 | be checked for consistency, using the canonical sources of that information - 232 | but then we might as well use those sources to directly calculate it. In these 233 | specific case, we see that enforcing immediacy and anti-chain parents actually 234 | lets us achieve some stronger guarantees for free, i.e. protection against 235 | rewinding of vector clocks. 236 | 237 | .. _Vector clock: https://en.wikipedia.org/wiki/Vector_clock 238 | 239 | Invariants 240 | ---------- 241 | 242 | To summarise, here are invariants on our *causal order* data structure. 243 | 244 | |le| is reflexive 245 | |forall| a: a |le| a 246 | 247 | |le| is anti-symmetric: 248 | |forall| a, b: a |le| b |and| b |le| a |implies| a = b 249 | 250 | |le| is transitive: 251 | |forall| a, b, c: a |le| b |and| b |le| c |implies| a |le| c 252 | 253 | pre(m) is an anti-chain; the history G forms a transitive reduction of |le|: 254 | |forall| m: |forall| p, p' |in| pre(m): p = p' |or| p |perp| p' 255 | 256 | by(u) is a chain / total-order: 257 | |forall| u: |forall| m, m' |in| by(u): m |le| m' |or| m' |le| m 258 | 259 | Security 260 | ======== 261 | 262 | Security dictates that we have some way to enforce the *real-world semantics* 263 | that we chose earlier, under the invariants defined above. Let's proceed: 264 | 265 | Enforcing semantics of |le| 266 | --------------------------- 267 | 268 | a |le| b (real-world semantics) 269 | The author of b knew the contents of a, when they wrote b. 270 | 271 | Our mechanism to achieve this securely (in a sense that we'll define) is as 272 | follows. First, we deduce that parent references must at least be immutable and 273 | unforgeable. That is, for a given message m that references p using a string 274 | s[p], it must be the case that an attacker that sees s[p] and wishes to break 275 | our semantics, cannot create a new message p' that s[p] could also refer to. If 276 | they could, then our semantics is trivially broken. 277 | 278 | One simple option to achieve this `unforgeable` property, is to let s[p] be a 279 | hash of the ciphertext of message p. Those familiar with Git will see the 280 | similarities with its data model. :ref:`encoding-message-identifiers` explores 281 | this and alternatives in more detail. 282 | 283 | This leaves attackers with two ways to know a valid reference s[p]: (1) derive 284 | it from its ciphertext, or (2) see it elsewhere, e.g. referenced by a child of 285 | p, *without having seen* p itself. (1) does not break our semantics, but (2) 286 | does. That is, in message m, the attacker could make a false declaration that 287 | they've seen a real message p (breaking our semantics), if they see someone 288 | else refer to p in a third message n where p |in| pre(n) - and no-one can 289 | detect whether this declaration is false or true. (Note that this is a distinct 290 | case from falsely declaring that one has seen a fake i.e non-existent message, 291 | which *can* be detected as described further below.) 292 | 293 | We assume that people won't do this - there is no strategic benefit in claiming 294 | to know something that you are entitled to know but temporarily don't. The 295 | worst that may happen is that someone issues a challenge you cannot answer, but 296 | only your reputation would suffer. Also, if you haven't seen the message, you 297 | are unable to verify that the reference is indeed valid, so someone else might 298 | be trying to trick *you*. If this security assessment turns out to be wrong, 299 | there are :ref:`measures ` we can take to 300 | close this hole, but they are more complex so we ignore them for now. 301 | 302 | Enforcing invariants 303 | -------------------- 304 | 305 | First, note the distinction between enforcing the *structure* vs *semantics* of 306 | invariants. We enforce structure by writing correct code for our causal order 307 | data structure, and this can be statically checked e.g. that ``a <= a`` returns 308 | true for all ``a``. By contrast, enforcing semantic reflexivity means to 309 | enforce that: whenever our code does return true in response to executing ``a 310 | <= a``, this *also* has the real-world semantics that "the author of a knew the 311 | contents of a, when they wrote a". 312 | 313 | For **reflexivity**, this is true by definition of our chosen semantics for 314 | |le|, as we just stated. No run-time checks are required. 315 | 316 | For **anti-symmetry**, we gain this if our parent references are unforgeable as 317 | required by the previous section. Then, there is no way to generate distinct 318 | messages a, b such that both a |le| b (i.e. b references a, perhaps indirectly) 319 | and b |le| a (likewise). This protection is achieved when we specify the hash 320 | algorithm for the protocol; no run-time checks are required. 321 | 322 | For **transitivity**, we must show messages to the user in `topological 323 | order`_. For each incoming m, if any of its parents p |in| pre(m) have not yet 324 | been delivered [#dlvr]_, we place m in a buffer, and wait for all of pre(m) to 325 | be delivered first before delivering m. This allows us to verify that the 326 | parent references are actually of real messages. To protect against DoS, the 327 | buffer should be limited in size; see the next section for details. 328 | 329 | The result is that we deliver all of anc(m) before we deliver m, i.e. the user 330 | sees anc(m) before m. This applies for messages the user sends too, assuming we 331 | set the parents correctly. This achieves transitivity on the local side, but it 332 | is quite hard to detect whether *other* members are doing this. (One option is 333 | to require m to mention *all* of anc(m), but obviously this costs too much 334 | bandwidth.) But earlier, we argued that there is no incentive for users to 335 | cheat this; we will make this assumption going forward. 336 | 337 | For **transitive reduction**, we do this locally using only information from 338 | anc(m). Since messages are only added to the data structure in topological 339 | order, everyone has this information so they can enforce it themselves. 340 | Messages that break this may simply be dropped, with a local UI warning as to 341 | the source. The :ref:`merge algorithm for DAGs ` includes a 342 | check that the inputs form an anti-chain, and it may be run on the pre(m) of 343 | every message accepted, including for single-parent messages. 344 | 345 | For **author total ordering**, this is detectable only when someone accepts 346 | messages in both forks, which not everyone else might see. Users detecting this 347 | *must* refuse to participate any further in the session, which (via mechanisms 348 | described in the next chapter) hints to other members that something is wrong. 349 | 350 | To help others narrow down the cause of this non-participation, the detecting 351 | user *should* re-broadcast all messages within both forks to everyone else, 352 | then leave the conversation with a NACK error message that references both 353 | forks. To be clear, this is not security-critical since these messages might be 354 | dropped by an attacker; its purpose is to give a UX improvement in the case 355 | where the attacker does not drop packets. Note that one of the references may 356 | have to be indirect if we already replied to the other fork, to preserve 357 | transitive reduction. We should never need to do this for a triple fork; NACK 358 | should be immediate upon detecting a double fork. 359 | 360 | (TODO: enumerate all possible error reasons for NACK messages) 361 | 362 | .. _Topological order: https://en.wikipedia.org/wiki/Topological_sort 363 | 364 | .. [#dlvr] Here, `deliver` is standard distributed-systems terminology, and 365 | means to make the message available to higher layers for subsequent 366 | operations, such as displaying in a UI. However, in colloquial messaging 367 | contexts, "deliver" sometimes means "remote completion of a send", e.g. as 368 | in "delivery receipt"; so sometimes we'll use the term `accept` instead for 369 | the former concept. 370 | 371 | Detecting transport attacks 372 | --------------------------- 373 | 374 | Most of these are a straightforward consequences of the previous sections, but 375 | those are rather abstract so we'll re-describe them in terms of defences 376 | against "real-world" attacks. 377 | 378 | By `transport attacks`, we mean replays, re-orders, and drops of packets, by an 379 | attacker that can affect the physical communications infrastructure. We assume 380 | that external cryptographic authentication will allow us to detect packets 381 | injected by a non-member. 382 | 383 | Detecting replays is easy, assuming that our decryption scheme verify-decrypts 384 | duplicate packets so that they have the same parents (and contents and other 385 | metadata); then this will be deserialised into an already-existing node in our 386 | transcript graph. If we cache the ciphertext (and there is :ref:`reason to 387 | `), we don't even need to verify-decrypt it the second time. 388 | 389 | Enforcing transitivity is basically an exercise in correcting the order of 390 | messages, so we are already covered there. 391 | 392 | For drops, let's consider `causal drops` first - drops of messages that caused 393 | (are `before`) a message we *have* already received. Messages in the delivery 394 | buffer will have three types of parents: those already accepted, those also in 395 | the buffer, and those that haven't been seen yet. For the last case, this is 396 | either because the parent p doesn't exist (the author is lying), or because the 397 | transport is being unreliable or malicious. 398 | 399 | We don't assume the transport is reliable, so we should give it a grace period 400 | within which to expect p to arrive. After this expires, we should assume that 401 | we'll never receive p for whatever reason, emit a UI warning ("referenced 402 | message p didn't arrive on time"), and drop messages that transitively point to 403 | p from the buffer. This is safe even if p turns out to be real and we receive 404 | it later; others' :ref:`reliability ` schemes will detect this and 405 | resend us the dropped messages. If we do eventually receive p later, we should 406 | cancel the warning, or at least downgrade its severity, based on how timely we 407 | expect the transport to be. 408 | 409 | If we never receive p, we cannot know *for sure* whether the author was lying 410 | or the transport was malicious. In the basic case, there is no incentive for 411 | the author to lie, but if we just *assume* the transport is malicious (and take 412 | some action A in response to this) then ironically we *give* the author an 413 | incentive to lie - they can frame the transport to induce us to do A, which may 414 | have unintended consequences. So, we should be careful in how we communicate 415 | this fault to the user, and not imply blame on any party. 416 | 417 | Detecting *non-causal drops* - drops of messages not-before a message we've 418 | already received, and therefore we don't see any references to - is more 419 | complex, and we will cover this in :doc:`freshness <03-freshness>`. 420 | 421 | If the first message has replay protection (e.g. if it is fresh), we also 422 | inductively gain this for the entire session. This is because each message 423 | contains unforgeable references to previous parent messages, so everything is 424 | "anchored" to the first non-replayable message. In other words, this ultimately 425 | depends on the session membership control protocol, which we discuss elsewhere. 426 | TODO: link 427 | 428 | Other topics 429 | ============ 430 | 431 | The following sections talk about common "application-level" topics that are 432 | consequences of our scheme, including defences of some criticisms, that we have 433 | come across in various informal discussions. 434 | 435 | They do not add new *constructive* ideas on top of what we already discussed, 436 | so feel free to skip to the next chapter if you only want to read about what 437 | our proposals *are*. 438 | 439 | Buffering and asynchronous operation 440 | ------------------------------------ 441 | 442 | Our strong ordering (transitivity) approach requires a reliable transport. If a 443 | message is dropped, this prevents later messages from being displayed. Note 444 | that this is *not* a security concern, but a reliability one - if an attacker 445 | wants to prevent those messages from being displayed, and is able to drop the 446 | first message, they can just *drop the later messages* as well. 447 | 448 | We can improve reliability with end-to-end recovery (explained :ref:`next 449 | `), but this is less effective in an asynchronous scenario, i.e. 450 | where members may not online at the same time. We depend more heavily on a 451 | third-party service to store-and-forward messages in an intelligent way, e.g. 452 | to resend ciphertext even if the original author is offline. In the worst case 453 | where no two members are online simultaneously, this dependency becomes an 454 | absolute *requirement*. 455 | 456 | Such a service must provide the interface "send ciphertext T to recipient R". 457 | By itself, this should not reduce data security, since we assume there is an 458 | attacker that already stores all ciphertext. (Metadata security is a harder 459 | problem that is outside of our current scope - but at least this architecture 460 | is no worse than existing transports.) 461 | 462 | This adds extra development cost in a real application: at present there are 463 | architectural issues with some existing asynchronous delivery or "push" systems 464 | that make this sort of reliability awkward; email works, but leaks significant 465 | mounts of metadata. However, we believe that there is demand for a general 466 | reliable asynchronous transport as a piece of core internet infrastructure, 467 | which would be useful for many other services and applications too. So, at 468 | least within the scope of this document, we will work with this assumption, to 469 | clearly divide our problem between security and asynchronous reliability. 470 | 471 | We may also explore quasi-reliability schemes, that avoid buffering but still 472 | provide strong ordering. For example, if the transport can deliver large 473 | packets without splitting them, then whenever the user sends a message m, we 474 | can "piggyback" all messages from anc(m) that are not :ref:`fully-acked 475 | ` alongside m. So no-one will ever see messages out-of-order, thus 476 | avoiding buffering. This exists already as an ad-hoc practice in email, where 477 | often people leave a quoted parent in the body of their own message. 478 | 479 | This is not a full replacement for a reliable transport - if unacked, packets 480 | get larger and larger, and the ability to deliver them in one piece effectively 481 | becomes a reliability problem. However, it could greatly improve the buffer 482 | waiting period for the majority of unreliable cases, and does not require any 483 | complex behaviour on the part of the always-online store-and-forward server. 484 | 485 | Other approaches forfeit reliability, and display messages immediately upon 486 | receipt even if received out-of-order. Such strategies necessarily break our 487 | strong ordering property which we consider critical to security, so we won't 488 | discuss them further here - but see :ref:`consistency-without-reliability` for 489 | a more detailed exploration down this path. 490 | 491 | The importance of strong ordering 492 | --------------------------------- 493 | 494 | It is common for messages to depend on context for their precise meaning, in 495 | both natural and computer languages; delivering such messages out-of-order is 496 | not even *semantically correct*. We conjecture that these context-dependent 497 | messages are a majority, based on briefly scanning our own mailboxes and thread 498 | histories. One might argue that humans are pretty good at reconstructing the 499 | order ourselves, but this places an extra burden on the user. 500 | 501 | Some have argued that strong ordering is not important for asynchronous and/or 502 | high-latency use cases, and therefore buffering is not worth the supposed UX 503 | cost. But receiving messages out-of-order is *itself a UX cost*. Futher, strong 504 | ordering may not be important in some cases, but this is definitely not always 505 | true - we can imagine scenarios where critical meetings might take place over 506 | several days, where integrity is more important than timeliness. Therefore, we 507 | think it's prudent to prioritise achieving strong security then try to optimise 508 | the UX around this, rather than vice-versa. 509 | 510 | Generally, even asynchronous transports are mostly reliable - otherwise they 511 | would have no users. Now, let's consider this minority case where out-of-order 512 | delivery might actually occur. Suppose Alice sends message (1) then (2), and 513 | Bob's machine receives (2) after time *t*. The only case where Bob would prefer 514 | to be shown (2) before/without (1) being available, is if (2) does not require 515 | (1) to be understood correctly. As previously conjectured, we assume this is a 516 | minority (of an outer minority). Then, we have the following cases: 517 | 518 | a. Bob receives (1) after a further time *t'* < *t*. 519 | b. Bob receives (1) after a further time *t'* > *t*. 520 | c. Bob doesn't receive (1) at all, after some timeout *T*. 521 | 522 | We argue that most users won't notice (a), that (b) and (c) are more problems 523 | of the transport that incentivises users towards better transports regardless 524 | of whether buffering is done, and that (c) can further be effectively handled 525 | under our system via timeouts and by restarting the session, such as described 526 | in :ref:`hci-ordering`. 527 | 528 | One might argue that scenarios where strong ordering is security-critical is 529 | also a minority, but we feel it's better to achieve this much-needed security 530 | for this minority, than to achieve slightly more convenience for another 531 | less-needy minority. 532 | 533 | TODO: describe a hybrid scheme, using lossy ratchets to derive a "session key", 534 | and using this as the long-term key (instead of the real long-term key) in a 535 | strongly-ordered ratchet. this prevents "drop attacks" from forcing users to 536 | repeatedly use their long-term key for messaging material. 537 | -------------------------------------------------------------------------------- /causal/03-freshness.rst: -------------------------------------------------------------------------------- 1 | ====================== 2 | Freshness and presence 3 | ====================== 4 | 5 | Non-causal drops 6 | ================ 7 | 8 | As mentioned at the end of the previous section, we are so far unable to detect 9 | non-causal drops - drops of messages that are not an ancestor of messages we 10 | *have* seen and delivered. This does not affect transcript consistency, as 11 | already argued. However, sometimes we might still want this as a security 12 | property, especially in low-latency protocols. 13 | 14 | One way we can do this is via fresh nacks - messages that we can be sure are 15 | fresh (as of time t), that say "no messages sent by u since t". Our by(u) total 16 | ordering encodes the "no messages since" meaning implicitly, so we just need to 17 | prove freshness since time t. 18 | 19 | A message m implies its sender was present at time of context(m)(self), if it 20 | exists - we trust our own timestamps, and the partial order establishes the 21 | "after" relationship. If context(m)(self) does not exist, then we'll have to 22 | use something else written by ourselves, such as e.g. a key exchange message 23 | that helped to derive the secrets that m is auth-encrypted by - we don't have a 24 | mechanism to securely trust others' timestamps, and won't try to construct one 25 | here. Note that we just store our timestamps locally, we don't need to send 26 | them anywhere - and they wouldn't trust ours anyway. 27 | 28 | The above achieves *end-to-end* security for freshness. By contrast, XMPP 29 | presence is not end-to-end secure - it assumes a trusted server. Neither is 30 | looking at "local receive time of a packet" - it assumes a trusted network. 31 | 32 | .. _heartbeats: 33 | 34 | Heartbeats 35 | ---------- 36 | 37 | We detect non-causal drops by periodically [#Nreg]_ sending out heartbeats, and 38 | waiting for these to be fully-acked (i.e. setting an ack-monitor on them); this 39 | proves freshness as of the time of the heartbeat. To be efficient, we can omit 40 | a heartbeat if we've already sent another message inside the previous period, 41 | excluding explicit acks which don't need to be acked. We may refer to these 42 | messages as "implicit heartbeats", akin to implicit acks. 43 | 44 | If someone takes too long to ack a (implicit or explicit) heartbeat, we should 45 | issue a warning to the effect of "user not present". This is a distinct warning 46 | from message-not-fully-acked - it is triggered for *each member* that fails to 47 | respond, it might have a different timeout than the former, and it would be 48 | indicated on a non-message UI element (since heartbeats might not be displayed 49 | explicitly) - for example by dulling the respective user avatar. 50 | 51 | .. [#Nreg] If heartbeats are regular, they may be distinguished from actual 52 | messages even to an eavesdropper who can't read their contents. (The same 53 | may be said of regular explicit acks and identical-ciphertext resends, 54 | though these probably reveal less information.) In general, metadata 55 | security is a more complex topic, and it is unclear whether these things 56 | are a real security problem to be worth spending effort on, so we'll note 57 | the issue but won't pursue it here. 58 | 59 | Presence and expiry 60 | =================== 61 | 62 | Freshness is an indicator of presence, which we'll define roughly as "a good 63 | chance a user will ack messages sent to them". We adopt end-to-end definitions, 64 | so "presence" is considered from the local user's perspective, rather than the 65 | internet - if a user's connection goes down, from their POV everyone else is 66 | absent, rather than themselves. Typically, presence statuses must expire unless 67 | explicitly refreshed, to be able to reliably indicate absence. 68 | 69 | Our heartbeats are a rudimentary indicator of presence, one that is only partly 70 | useful since it works within an already-established session. There are many 71 | end-to-end secure presence mechanisms that are more general purpose, e.g. TODO. 72 | Yet other mechanisms are simple but still effective - e.g. detecting that the 73 | local network interfaces are disconnected, is a strong indicator of absence. 74 | 75 | Locally, we may integrate our heartbeats with these other mechanisms, for more 76 | accuracy and reliability. The exact logic will depend on the mechanism and what 77 | interfaces it provides for integration, but it should be fairly intuitive and 78 | straightforward. For example, the combined system should indicate presence when 79 | a member acks a heartbeat *or* any external mechanism indicates presence, and 80 | any of these should reset any expiry-based absence indicators. 81 | 82 | Intended absence 83 | ---------------- 84 | 85 | The only absence signal we have described so far is expiry of presence. This is 86 | a minimum measure to ensure safety - to detect the inability to communicate, 87 | either innocent (e.g. lack of signal) or malicious (e.g. censorship). 88 | 89 | In some applications however, members may want to declare intentional absence 90 | from an ongoing session. This is semantically distinct from parting the session 91 | - absent users still have the *right* to see messages, so these should still be 92 | encrypted to their key. This allows us to support a long-running session where 93 | users go online and offline, but still want to see what was said during their 94 | absence. This is common with some existing centralised hosted chat systems, but 95 | we can achieve this end-to-end too. [#Nsrv]_ If no external presence mechanism 96 | supports this declaration, we may implement intended-absence messages that set 97 | the intended-absence state for the sender, but not otherwise displayed. 98 | 99 | Tracking intended-absence as a distinct state from unintended-absence, allows 100 | us to downgrade the severity of warnings that will fire due to it, since now we 101 | know that this is not due to an attack. For example, when a not-fully-acked 102 | warning is about to be emitted, if the users who haven't acked that message are 103 | all intended-absent, then the severity of the warning should be reduced. The 104 | rate of resends may be reduced too, since they won't succeed anyway. (If we 105 | have no other external presence mechanism though, they should not be stopped 106 | completely - otherwise all parts of a partition will stop sending, and we'll 107 | never regain presence.) Later when the absent user returns, the rate of resends 108 | should be restored, or even reset to the maximum. 109 | 110 | Absence must be considered when choosing a greeting protocol. A GKE implicitly 111 | requires the presence of certain participants, yet any member may unilaterally 112 | become absent (either declared or faked) and block it from succeeding. This may 113 | be a concern if membership operations are supposed to be reserved only for 114 | special members. As a baseline, we recommended that systems choose a greeting 115 | protocol that may part a member *without their own input*, otherwise anyone may 116 | block themselves from being parted. This topic will be explored in more detail 117 | in later parts. 118 | 119 | .. [#Nsrv] A dumb server that just stores-and-forwards the ciphertext may be 120 | used as an optional component - this has no impact on security, but keeps 121 | the session history available even in the case that all members go offline. 122 | 123 | Timestamps 124 | ========== 125 | 126 | So far we have avoided communicating any timestamps in the protocol, nor to 127 | rely on specific timestamp values for others' messages. (Heartbeat timestamps 128 | are only used locally as a lower-bound.) 129 | 130 | However, specific values for timestamps are useful as a UI indication. How do 131 | we do this? 132 | 133 | Limitations 134 | ----------- 135 | 136 | TODO Clock reliability. 137 | 138 | - new device, time not set 139 | - when travelling, some users change system clock instead of time zone 140 | - change system clock for testing and forget about it 141 | 142 | Verifiability. 143 | 144 | If a member lies in the contents of a message, this is not something the 145 | messaging protocol can detect. We treat timestamps as a similar sort of data - 146 | no other part of the system or the security models depends on them being even 147 | vaguely correct. 148 | 149 | Simplest option is to use local delivery time of a message. Additionally, each 150 | message may contain the claimed sender local time, which may be displayed on 151 | the side. 152 | 153 | See appendix (TODO: link) for discussion of some more complex techniques, which 154 | may result in better accuracy when users have skewed clocks. However, they are 155 | quite complex and don't achieve very strong end-to-end security (as mentioned 156 | above), which is why they are only in the appendix. 157 | -------------------------------------------------------------------------------- /causal/04-state.rst: -------------------------------------------------------------------------------- 1 | ================= 2 | History and state 3 | ================= 4 | 5 | .. include:: 6 | .. include:: 7 | 8 | Now back to session semantics. If you're coming into this directly from an 9 | external resource, make sure you've read the first parts of :doc:`01-ordering`. 10 | 11 | Regarding membership, so far in our data model we've only defined that each 12 | message is associated with a set of members. Now, suppose we reach a fork in 13 | our session history G, i.e. \|max(G)\| > 1, and the messages in this set all 14 | have different members. What then, is the membership of our current session G? 15 | 16 | It's good to be lazy in computer science, so we first start with a rain check - 17 | "do we really need to answer this"? Unfortunately yes, and the result *must* be 18 | a single membership set, and not (e.g.) a set of "possible membership sets": if 19 | the user wanted to send a new message *right now* before the fork is resolved, 20 | we must send it to *some specific* set of members, and we must indicate this 21 | somewhere in the UI so they *know* who they are sending it to. 22 | 23 | Union of events vs dynamic state 24 | ================================ 25 | 26 | To recap: we want to develop a dynamic group session protocol, where "dynamic" 27 | means "members may change at any time". Our security constraints around context 28 | and ordering led us to represent the session history as an immutable graph of 29 | messages, whose edges represent a transitively-reduced |le| relationship. 30 | 31 | We make the high-level observation that, in models based on an immutable 32 | history of events, each event may be associated with two types of data: with 33 | type 1 the of the whole is just the union of the of every event; 34 | with type 2 the of the whole is an "aggregated form" of the of 35 | every event, and the former "changes" as we accept more events into the local 36 | history. Applying this to group sessions and version control, we have: 37 | 38 | +-------------------+-------------------+-------------------------------+ 39 | | Model | Type 1 "union" | Type 2 "dynamic" | 40 | +===================+===================+===============================+ 41 | | Group session | Message content | Session / message membership | 42 | +-------------------+-------------------+-------------------------------+ 43 | | Version control | Commit message | Commit tree content | 44 | +-------------------+-------------------+-------------------------------+ 45 | 46 | In some sense, the messages (union of events) is the "main focus" of our 47 | system, but membership (dynamic state) is also quite important - and where the 48 | hard problems lie. Luckily, in DVCSs it's the commit tree content (dynamic 49 | state) that is the "main focus" of those systems, and we can use solutions from 50 | there to inspire solutions for the analogous problems in ours. 51 | 52 | It's not immediately obvious that there is an "aggregated form" of the tree 53 | content of multiple commits. But in fact, that is what a merge algorithm does: 54 | assuming there are no conflicts, it gives us a single output tree state, for 55 | any input set of forked branch heads. 56 | 57 | For group sessions, we don't want to force users to "add a merge commit" (i.e. 58 | write a new message) every time there is a fork. What we *can* do now however, 59 | is to define "the" state of a forked history G, in line with our analogies: 60 | 61 | **members(G)** = merge[G](max(G)) 62 | Current session membership at G, i.e. after having received/accepted all 63 | messages in G, and no other messages. 64 | 65 | Now, we need an algorithm merge[G](X) to merge the state of a set of nodes, 66 | like the ones that exist in modern DVCSs. Furthermore, the algorithm must never 67 | require manual conflict resolution - we must have a result *before* the user 68 | writes any message. 69 | 70 | From here on, we'll ignore the `type 1` column and message content. When we 71 | refer to the "state" of a node, we mean the `type 2` state, and when we refer 72 | to "merged state", we mean the aggregated state. In our group session system, 73 | `state` represents a set of members, but for now to keep things simple, we'll 74 | ignore this and mostly talk about an opaque semantics-free `state`. To re-state 75 | our aim in these terms, we want to define a function: 76 | 77 | **merge[G](X)** -- where: 78 | - G is a :doc:`partial order <01-ordering>` history graph, where each node 79 | is associated with some type-2 state (e.g. "set of members") 80 | - X is a set of nodes |in| G 81 | - the output is a state 82 | 83 | 3-way-merge 84 | =========== 85 | 86 | To begin with, note that it's impossible to do correct merges without knowing 87 | previous relevant history. [#mhst]_ To demonstrate this, see the below cases. 88 | In both cases, we want to merge the same states - unordered sets {a, b} and 89 | {b, c}, but the two cases have different histories: 90 | 91 | .. digraph:: merge_history 92 | 93 | rankdir=RL; 94 | node [style="filled"]; 95 | 96 | x1 [fillcolor="#66ff66",label="b"]; 97 | z1 [label="abc"]; 98 | l1 [fillcolor="#6666ff",label="ab"]; 99 | r1 [fillcolor="#6666ff",label="bc"]; 100 | l1 -> z1 [label="-c"]; 101 | r1 -> z1 [label="-a"]; 102 | x1 -> l1 [color="#666666"]; 103 | x1 -> r1 [color="#666666"]; 104 | 105 | x0 [fillcolor="#66ff66",label="abc"]; 106 | z0 [label="b"]; 107 | l0 [fillcolor="#6666ff",label="ab"]; 108 | r0 [fillcolor="#6666ff",label="bc"]; 109 | l0 -> z0 [label="+a"]; 110 | r0 -> z0 [label="+c"]; 111 | x0 -> l0 [color="#666666"]; 112 | x0 -> r0 [color="#666666"]; 113 | 114 | The node label "bc" means the state at that node is {b, c}, and so on. Edges 115 | are labeled with the operation that the child node performs on the state at the 116 | parent node. Blue nodes are the nodes to be merged, green is the output (a 117 | candidate node with merged state), and gray edges indicate merge parents (i.e. 118 | nodes with >1 parent). 119 | 120 | So as a minimum requirement, our merge algorithm must use *some* information 121 | from the previous history. The most basic version of this is a 3-way merge: 122 | 123 | 3-way-merge(o, a, b) -- where 124 | - o, a, b are states 125 | - o is interpreted as some form of both a's past and b's past 126 | - the output is a state, or |bot| if there is a merge conflict 127 | 128 | Based on intuition, we suggest that this should satisfy some invariants. First, 129 | if nothing changed on one branch, then it should simply use the other one, i.e. 130 | |forall| o, a: 3-way-merge(o, o, a) = 3-way-merge(o, a, o) = a. Next, in a 131 | forked history neither side is favoured - so the order of the last arguments 132 | should not change the result, i.e. |forall| o, a, b: 3-way-merge(o, a, b) = 133 | 3-way-merge(o, b, a). We'll come back to these later. 134 | 135 | To get an intuitive feel on how it should work, we can define 3-way-merge in 136 | terms of more familiar `diff` and `apply` operations: 137 | 138 | 3-way-merge(o, a, b) = apply(b, diff(o, a)) = apply(a, diff(o, b)) 139 | 140 | This is easier to understand, but specifying invariants equivalent to the ones 141 | above is more complex; see :ref:`the appendix ` for details. 142 | In a real implementation it suffices to define only 3-way-merge, and it's also 143 | slightly more efficient. 144 | 145 | From our apply-diff definition, we see a potential source of conflicts. We 146 | apply a diff (o, a) on top of a context (b) that it was not intended for. How 147 | likely this is to happen, depends on the state data type. For example, this can 148 | quite easily happen with lists-of-lines (as DVCSs use), and happens less often 149 | with semantic diff/merge algorithms. 150 | 151 | Let's start with the simplest option to represent group session membership, an 152 | unordered set, and try to define 3-way-merge for this data type. 153 | 154 | | Set-3-way-merge(o, a, b) 155 | | = Set-apply(b, Set-diff(o, a)) 156 | | = Set-apply(b, [insert (a \\ o), delete (o \\ a)]) 157 | | = b |cup| (a \\ o) \\ (o \\ a) 158 | | = a |cup| b \\ o |cup| (a |cap| b |cap| o) -- equivalent to the previous; 159 | more "obviously" symmetric but probably slightly less efficient 160 | 161 | It's fairly straightforward to check that this satisfies our invariants above. 162 | To check symmetry with respect to {a, b}, it helps to draw a Venn diagram. 163 | 164 | We're in luck - this is well-defined for all arguments, and no conflicts can 165 | result from this algorithm. So let's use this for now as our state data type, 166 | and continue with the rest of the merge algorithm. 167 | 168 | .. [#mhst] If the `state` data type does not allow `undo` operations, then we 169 | can get by without knowing history. This is exactly how state-based CRDTs 170 | work, see :ref:`below `. In that literature, "no undo" 171 | is stated instead as "states must increase along some partial order". This 172 | is unrelated to our partial order on G, also explained below. 173 | 174 | Our system already has history however, so we wanted to start by exploring 175 | the minimal amount of additional complexity, i.e. to use a simple unordered 176 | set as our `state` data type for membership sets. This *does* allow `undo`, 177 | such as adding then removing a member. Similarly, DVCSs that use line-based 178 | diff algorithms also allow `undo` - you can remove one line from a file, 179 | commit it, then re-add it in a later commit. Neither unordered sets nor 180 | line-lists are suitable as a state-based CRDT. 181 | 182 | If these paragraphs confused you, just ignore them and ignore state-based 183 | CRDTs; it's not essential for understanding the rest of this document. 184 | 185 | General merge 186 | ============= 187 | 188 | What about more complex histories? It turns out, we can build a merge algorithm 189 | that works on arbitrary histories, using only 3-way-merge (which doesn't query 190 | the history) and queries on the history. There are several advantages to this: 191 | 192 | It is completely independent of the *state type* - i.e. merge does not need to 193 | manipulate state directly, it simply uses 3-way-merge (which does do that). So 194 | if we change our state type, we only need to provide a new 3-way-merge, and not 195 | a completely new merge algorithm. 196 | 197 | Another advantage is that, since it doesn't touch the state directly, it also 198 | does not generate merge conflicts beyond what 3-way-merge generates. We saw 199 | earlier that our 3-way-merge for unordered sets never results in conflicts, so 200 | things are looking pretty good for our original requirements. 201 | 202 | We didn't invent this algorithm ourselves: git did the hard work, via years of 203 | mailing list discussions and engineering experience. However, it's unclear if 204 | they know this is a *correct* solution or just an heuristic: even now, ``man 205 | git-merge`` says "[the default merge algorithm] has been reported to result in 206 | fewer merge conflicts without causing mismerges by tests done on actual merge 207 | commits taken from Linux 2.6 kernel development history." 208 | 209 | Our novel contribution here then, is a proof sketch that this merge algorithm 210 | is indeed *correct* and *unique*, derived from some fairly simple proposals on 211 | how "reasonable" merge algorithms should behave. We'll get to that later; first 212 | we go through a more "blind" derivation, that's hopefully more intuitive if you 213 | haven't seen how this works before. 214 | 215 | To restate our problem, we want to define:: 216 | 217 | merge(G: graph, X: {node}) -> state 218 | 219 | and we assume there is some 3-way-merge for our state data type:: 220 | 221 | 3-way-merge(o: state, a: state, b: state) -> state 222 | 223 | Notation wise: ``f(x: T0) -> T1`` means that input argument ``x`` has type 224 | ``T0`` and ``f`` returns a value of type ``T1``; ``{T}`` is the type of an 225 | immutable, unordered set whose inner values are of type ``T``. 226 | 227 | To simplify our pending explanation, we'll assume some things for now, then 228 | un-assume and handle them after we've explained the core of the algorithm. 229 | 230 | 1. X is an anti-chain, i.e. no two nodes in it are |le| each other. Later 231 | we'll describe how to detect and handle this case. 232 | 233 | 2. X is a list. Later we'll show that the merge result is the same regardless 234 | of the order of elements. (Duplicate elements are handled by (1)). 235 | 236 | Simple intuitive derivation 237 | --------------------------- 238 | 239 | Let's try a recursive definition. As with all recursive derivations, first 240 | let's pretend we already have what we want to define:: 241 | 242 | merge(G, X: [node]) -> state # omitting "G: graph" to reduce clutter 243 | 244 | To make our lives easier, let's derive a simpler form first:: 245 | 246 | def lca2(G, a: node, b: node) -> [node]: 247 | # lowest common ancestors of 2 nodes 248 | return max({ v in G | v <= a and v <= b }) 249 | 250 | def merge2(G, a: node, b: node) -> state: 251 | O = lca2(G, a, b) # calculate parent node(s), for 3-way-merge 252 | if size(O) == 1: # base case 253 | s_o = O[0].state 254 | if size(O) >= 2: # recurse! 255 | s_o = merge(G, O) 256 | return 3-way-merge(s_o, a.state, b.state) 257 | 258 | There, that wasn't so hard. But now how do we turn ``merge2`` into ``merge``? 259 | As in many other areas of computer science, if we have a binary operation, we 260 | can use `fold`_ (sometimes called `reduce`) to apply it to a non-empty list. We 261 | have to do some minor fiddling as well:: 262 | 263 | def merge-fold-recurse(G, X: [node]) -> state: 264 | def merge2'(a: node, b: node) -> node: 265 | s = merge2(G, a, b) 266 | return "new temp node" in G with state = s, parents = {a, b} 267 | return fold(merge2', X).state 268 | 269 | .. _fold: https://en.wikipedia.org/wiki/Fold_%28higher-order_function%29 270 | 271 | "new temp node" means to create a temporary node in G just for the duration of 272 | ``fold``, so that it has something to work with. We'll get rid of this soon 273 | below (in a real implementation G should be immutable anyway) but for now this 274 | gives a good intuition on how the algorithm works. 275 | 276 | But wait, there's another way of applying the ``fold``. Instead of applying it 277 | to ``merge2``, we apply it *inside* ``merge2``, to ``3-way-merge`` which is 278 | also a binary operation (if we fix the ``o`` argument):: 279 | 280 | def lca(G, X: [node]) -> [node]: 281 | # lowest common ancestors of n nodes 282 | return max({ v in G | v <= x for all x in X }) 283 | 284 | def merge-recurse-fold(G, X: [node]) -> state: 285 | O = lca(G, X) # calculate the parent node / nodes 286 | if size(O) == 1: # base case 287 | s_o = O[0].state 288 | if size(O) >= 2: # recurse! 289 | s_o = merge-recurse-fold(G, O) 290 | def 3-way-merge-with-o(a: node, b: node) -> node: 291 | s = 3-way-merge(s_o, a.state, b.state) 292 | return "new temp node" in G with state = s, parents = {a, b} 293 | return fold(3-way-merge-with-o, X).state 294 | 295 | (If you're confused by this, try to compare this with ``merge2``. The "general 296 | idea" is the same; we just turned binary operations on (a, b) into their n-ary 297 | counterparts.) 298 | 299 | So which version is correct? Or perhaps they are identical, and both correct? 300 | 301 | A more precise derivation 302 | ------------------------- 303 | 304 | It turns out, ``fold-recurse`` is correct (and what git recursive-merge does), 305 | and ``recurse-fold`` is wrong. To explain why, we'll justify our steps more 306 | with a semi-formal argument. Hopefully it's precise enough to be turned into a 307 | more rigorous proof later, if one wanted to do that. 308 | 309 | First, let's define an `operational edge`. This is an edge from a node to its 310 | parent, where the author of the node intended to introduce some state change, 311 | the "operation". So for example: (a) for a node with a single edge whose state 312 | differs from its parent's, that edge is an `operational edge`; (b) for a node 313 | with many parents where the author only performed the merge algorithm and did 314 | nothing else, none of these edges are `operational edges`; (c) for a node v 315 | with many parents where the author also performed extra operations *on top of* 316 | the merge algorithm, we may instead treat this as virtual nodes (v₀ |leftarrow| 317 | v₁) where v₀ points to the actual parents of v as in case (b) and the edge from 318 | v₁ to v₀ is the operational edge as in case (a). (These cases are not used 319 | in the below proof-sketch, but do become useful in an actual rigorous proof.) 320 | 321 | Our other definitions: 322 | 323 | - \\ is a binary infix operator denoting set subtraction. 324 | - |SquareUnion| is a binary infix operator denoting set *disjoint* union. That 325 | is, A |SquareUnion| B is equal to A |cup| B but with the extra assertion that 326 | A |cap| B = |emptyset|. 327 | - anc*(V) = |bigcup| {anc(v) |forall| v |in| V}, the union of ancestors of all 328 | nodes in set V. 329 | 330 | Our assumptions, which we think are reasonable starting points, but perhaps not 331 | fully rigourous or "maximally simple", are: 332 | 333 | 1. Given node a, its state ``a.state`` "applies exactly once" all operational 334 | edges in anc(a). We'll abbreviate this as ``a.state`` |cong| anc(a). 335 | 336 | 2. If a |le| b, then diff(``a.state``, ``b.state``) |cong| anc(b) \\ anc(a). 337 | 338 | a. More generally, for all subgraphs A, B and states s, t: if A |subseteq| B 339 | and s |cong| A and t |cong| B, then diff(s, t) |cong| B \\ A. 340 | 341 | b. Similarly, for all subgraphs A, D and states s, diffs d: if A and D are 342 | disjoint and s |cong| A and d |cong| D, then apply(s, d) |cong| A 343 | |SquareUnion| D. 344 | 345 | 3. For ``merge(X)``, we want to find some t |cong| anc*(X). 346 | 347 | Now follows our proof-sketch. Let A = anc(a), B = anc(b). Sometimes we'll also 348 | treat these node-sets as subgraphs (i.e. including the edges between them); 349 | hopefully it's clear from the context. Now, our goal for ``merge2(a, b)`` is to 350 | to find t |cong| A |cup| B, and show that this is the case. 351 | 352 | First, by elementary set theory let's note that A |cup| B 353 | = A |SquareUnion| (B \\ A) 354 | = B |SquareUnion| (A \\ B) 355 | = (A |cap| B) |SquareUnion| (B \\ A) |SquareUnion| (A \\ B) 356 | 357 | WLOG let's try to find some s |cong| A \\ B. Define o = lca2(a, b), O = anc*(o) 358 | = A |cap| B. (Note here that o is also a node-set.) Let p be some (yet unknown) 359 | state such that p |cong| O. By (1) we also have ``a.state`` |cong| A. Now O 360 | |subseteq| A so by (2.a) and the previous sentences, we have s = diff(p, 361 | ``a.state``) |cong| A \\ O = A \\ B. 362 | 363 | How do we find p? Well, if o is a singleton set, then by (1) p = ``o[0].state`` 364 | satisfies what we need. If not, then by (3) ``merge(o)`` also satisfies what we 365 | need - and o |subset| A, o |subset| B, due to our "anti-chain" assumption, so 366 | this induction step eventually reaches a base case, and is therefore a valid 367 | step. (Notice that this is exactly how we defined ``merge2`` earlier.) 368 | 369 | Now we can find t. By (1) we have ``b.state`` |cong| B, and we just found s 370 | |cong| A \\ B. So by (2.b) we have apply(``b.state``, s) |cong| B |SquareUnion| 371 | (A \\ B) = A |cup| B. In other words, t = 3-way-merge(``p.state``, ``a.state``, 372 | ``b.state``) |cong| A |cup| B. ∎ 373 | 374 | To extend this to three arguments, imagine that we redo the proof, but with (A 375 | |cup| B) and C in place of A and B, and with ``merge2(a, b)`` (which we just 376 | proved) in place of ``a.state``. We will find that we need some o such that O = 377 | anc*(o) = (A |cup| B) |cap| C. In our 2-arg proof, this was o = lca2(a, b). In 378 | our 3-arg proof, this must be instead be o = lcaU({a, b}, c), where: 379 | 380 | lcaU(A, b) 381 | | = max({ v |in| G | v |in| anc(b) |and| v |in| anc*(A)}) 382 | | = max({ v |in| G | v |le| b |and| v |le| a for some a |in| A }) 383 | 384 | Then, the rest of the proof follows exactly as for the 2-arg case. We can also 385 | see that the result is the same regardless of the order of arguments - in all 386 | cases, we are finding some t |cong| anc*({a, b, c}). We can deduce by further 387 | induction, that ``fold-recurse`` |equiv| ``merge`` is correct, and symmetric 388 | with respect to the order of its arguments. ∎ 389 | 390 | Similar lines of reasoning also tell us why ``recurse-fold`` is wrong, because 391 | when we execute ``O = lca(X)``, this is analogous to finding O = A |cap| B 392 | |cap| C (``O`` |ne| O, but related), then trying to calculate t |cong| A |cup| 393 | B |cup| C by finding diffs |cong| to each component of the expression O 394 | |SquareUnion| (A \\ O) |SquareUnion| (B \\ O) |SquareUnion| (C \\ O) - but this 395 | is clearly not equal to A |cup| B |cup| C. 396 | 397 | As a concrete example, see: 398 | 399 | .. digraph:: fold_recurse_vs_recurse_fold 400 | 401 | rankdir=RL; 402 | node [style="filled"]; 403 | 404 | o [label="ab"]; 405 | a [label="a"]; 406 | u [fillcolor="#6666ff",label="abu"]; 407 | b [fillcolor="#6666ff",label="ab"]; 408 | v [fillcolor="#6666ff",label="av"]; 409 | m [fillcolor="#66ff66",label=""]; 410 | 411 | a -> o [label="-b"]; 412 | u -> o [label="+u"]; 413 | b -> a [label="+b"]; 414 | v -> a [label="+v"]; 415 | m -> u [color="#666666"]; 416 | m -> b [color="#666666"]; 417 | m -> v [color="#666666"]; 418 | 419 | For the green node trying to merge the blue nodes, ``fold-recurse`` gives the 420 | correct answer {abuv}, but ``recurse-fold`` gives {auv}. 421 | 422 | .. _merge-algorithm: 423 | 424 | Complete algorithm 425 | ------------------ 426 | 427 | Using lcaU from our above proof, we can get rid of our earlier "new temp node" 428 | shenanigans. Sadly, we can no longer use fold and must iterate manually: 429 | 430 | .. code-block:: python 431 | 432 | def lcaU(G, A: {node}, b: node) -> {node}: # note 1 433 | # lowest common ancestors of a node-set and a node 434 | return "max({ v in G | v <= b and v <= a for some a in A })" 435 | 436 | def mergeU(G, A: {node}, s_a: state, b: node) -> state: 437 | if A empty: 438 | return b.state 439 | O = lcaU(G, A, b) # calculate parent node(s), for 3-way-merge 440 | CHECK( O & (A | {b}) is empty ) # anti-chain check, see below 441 | s_o = merge(G, O) # recurse upwards, to ancestors 442 | return 3-way-merge(s_o, s_a, b.state) # note 2 443 | 444 | def merge_(G, B: {node}, A: {node}, s_a: state) -> state: 445 | if B empty: 446 | return s_a 447 | b = choose_any_element(B) 448 | s = mergeU(G, A, s_a, b) 449 | return merge_(G, B - {b}, A | {b}, s) # tail recurse sideways, to remaining args 450 | 451 | def merge(G, X: {node}) -> state: 452 | return merge_(G, X, {}, default-state) # note 3 453 | 454 | 1. See appendix (TODO: link) for an optimised "real implementation" of this. 455 | 2. To reduce clutter, we assume 3-way-merge doesn't give merge conflicts. 456 | Extending the code to handle that is a straightforward engineering exercise. 457 | 3. We could short-cut ``merge`` to check if X is empty or a singleton, but 458 | these cases are in fact already covered in the child functions. 459 | 460 | Now we can check that each input set is an anti-chain. Consider lcaU(A, b) and 461 | suppose we know A is an anti-chain. Then, if a |le| b for any a |in| A, either 462 | a or b will be within the result of ``lcaU``. Equivalently, if lcaU(A, b) |cap| 463 | (A |cup| {b}) = |emptyset|, then we know that b is causally independent of all 464 | a |in| A. If we add this check to ``mergeU``, then ``merge_`` will start with a 465 | base case A = |emptyset| which is an anti-chain, and as it processes elements 466 | from B and moves them into A, it will by induction check all pairs {x, x'} from 467 | X for causal independence. 468 | 469 | For our group session, we reject such inputs because they break our protocol 470 | invariants, and CHECK simply throws an exception to be handled further up the 471 | stack. [#tred]_ During run-time, whenever we accept a message m, we must check 472 | that merge(pre(m)) does not throw an exception. This enforces the transitive 473 | reduction property, as promised in the first chapter. 474 | 475 | Certain extreme history graphs can cause our recursive ``merge`` to stack 476 | overflow. It is possible to implement ``merge`` iteratively, but the non-tail 477 | recursion from ``mergeU`` back to ``merge`` forces such an implementation to be 478 | very complex and hard to understand. A better solution is to simply add an LRU 479 | cache to ``merge``. This is sufficient to prevent overflow, as long as we run 480 | the aforementioned merge(pre(m)) check for every accepted message. 481 | 482 | To summarise, if we fulfill the following requirements: 483 | 484 | - We have a partial order G on state-update events (nodes). 485 | 486 | - We provide a ternary operator `3-way-merge` on states which obeys: [#idem]_ 487 | 488 | Identity under the fixed first argument 489 | |forall| o, a: 3-way-merge(o, a, o) = 3-way-merge(o, o, a) = a 490 | 491 | Commutative under a fixed first argument 492 | |forall| o, a, b: 3-way-merge(o, a, b) = 3-way-merge(o, b, a) 493 | 494 | Alternatively, see :ref:`the appendix ` for the equivalent 495 | model stated in terms of `diff` and `apply` operations. 496 | 497 | Then we can define `merge` as ``merge`` from the pseudo-code above. Its 498 | properties are: 499 | 500 | - commutative - the input is an unordered set. 501 | 502 | - associative, in some sense. That is, the following two are equivalent, for 503 | all node-sets A, B: 504 | 505 | - merge(A |cup| B) 506 | - merge({v} |cup| B) where v = new node { parents = A, state = merge(A) } 507 | 508 | - closed (i.e. free from merge-conflicts), if 3-way-merge is closed 509 | 510 | .. [#tred] For other systems, such as DVCSs, this is not such a crucial error; 511 | they can instead ignore the extraneous older nodes, and run the algorithm 512 | as if the inputs were max(anc*(X)) - which is definitely an anti-chain. 513 | Tweaking our algorithm to do this efficiently is left as an exercise, but 514 | note that it is not enough to let CHECK continue or e.g. ``return as``. 515 | 516 | .. [#idem] There is another nice property for 3-way-merge to have, but it's not 517 | strictly necessary for us: 518 | 519 | `Idempotent `_ under a fixed first argument 520 | |forall| o, a: 3-way-merge(o, a, a) = a 521 | 522 | If this holds, then `merge` is also idempotent (i.e. merge(X) = a, if 523 | |forall| x |in| X: ``x.state`` = a). We actually do have both of these "by 524 | accident" because 3-way-merge on unordered sets happens to be idempotent, 525 | but it's not a *necessary* property. It has no effect on our derivation of 526 | the general merge algorithm, nor any other results here. 527 | 528 | :ref:`State-based CRDTs ` *do* require this, because 529 | their systems can't deduplicate identical events received more than once. 530 | We *can* do this - it's necessary, to protect against replay attacks. 531 | 532 | Other topics 533 | ============ 534 | 535 | Avoiding merges 536 | --------------- 537 | 538 | Others have suggested instead to avoid the complexities with merge algorithms, 539 | and just *define* forked histories as completely different continuations of the 540 | session, never to be reconciled via a merge. 541 | 542 | However, this misunderstands the semantics of forks - and does not actually 543 | solve the original problem. Forks represent not a conscious user decision to 544 | "create a new thread of discussion", but express an inherent property of our 545 | universe that two events may happen at the same time. Even if we supported 546 | user-conscious forks, we would *still* have to support these "accidental" 547 | forks: two users that *don't want to consciously fork* (i.e. *want* the session 548 | to proceed as a *single branch*) could *still* create an accidental fork. This 549 | possibility also exists with DVCSs - for example, two users can end up with 550 | forked histories without ever having explicitly run ``git branch``. 551 | 552 | If we avoid merges, we in effect assert that different scenarios (accidental vs 553 | conscious forks) are equivalent. This prevents the user from expressing their 554 | intentions accurately, and we consider this an unacceptable design choice. 555 | 556 | .. _comparison-vs-crdts: 557 | 558 | Comparison vs CRDTs 559 | ------------------- 560 | 561 | Our history-based merge has a similar purpose to CRDTs [#crdt]_, but these are 562 | designed with different assumptions, and have different requirements on what 563 | they need. To compare: 564 | 565 | - Systems that use CRDTs do not generally transmit history, and apply operation 566 | and state-update events (for op-based and state-based CRDTs, respectively) 567 | outside of their intended context - i.e. the knowledge of the event's author, 568 | at the time that they generated that event. They are specifically designed 569 | around being able to ignore this context, i.e. to "move" these events around 570 | arbitrary places of recipients' histories. 571 | 572 | By contrast, history-merge specifically requires the opposite - that there is 573 | an immutable history that everyone has a strong eventual consistent view of, 574 | with each state-update being immutably associated with a particular event on 575 | this history. We do this for security (integrity) reasons, as do DVCSs. 576 | 577 | - To achieve the `conflict-free` property (and to be able to correctly re-apply 578 | events under arbitrarily different contexts), CRDTs place extra requirements 579 | on what valid events and operations/states are. History-merge only requires 580 | there be a 3-way-merge over the states. 581 | 582 | To be clear: as far as I know, nobody has studied the differences between 583 | these requirements, but I strongly suspect that the 3-way-merge requirement 584 | is weaker (easier). For example, history-merge can support arbitrary set 585 | remove/re-add operations, but there is no CRDT that does this. [#cset]_ 586 | 587 | - The term `partial order` is used in completely unrelated ways. To explain: 588 | 589 | - In a state-based CRDT, the term refers to a partial order on the *states*. 590 | It (the order itself) has no "real representation" in terms of a data 591 | structure, but rather is a mathematical concept inherent to the definition 592 | of that particular CRDT, chosen independently of (and constant with respect 593 | to) any run of a protocol that uses it. Conceptually and in general, there 594 | *could* exist other mathematical partial orders over the same states, that 595 | also satisfy the same laws as required by the CRDT. 596 | 597 | - In the context of history-merge, the term refers to the partial order on 598 | the *events*. It (the order itself) is a data structure with real bits 599 | representing it, constructed during each run of the protocol. There can be 600 | no other partial order on the same events. 601 | 602 | It's *possible* (we haven't proved or checked it) that the laws we defined 603 | for 3-way-merge necessarily induce a partial order on the states. However, 604 | this is not directly relevant to any of our other discussions about our 605 | history-merge - and that is why we haven't bothered to prove or check it. 606 | 607 | - With history-merge, state updates do not have to increase along the states' 608 | partial order, if one even exists. 609 | 610 | - The history-based merge algorithm is indeed "idempotent, commutative and 611 | associative" in the same way that state-based CRDTs merge algorithms are. To 612 | be clear: as described, history-merge is a pure function over a given history 613 | graph and its associated states; whereas the `merge` of state-based CRDTs is 614 | typically described as an impure procedure that updates the local current 615 | state. But fundamentally, they are nearly the same concept, the difference 616 | being that the CRDT `merge` works without a history. 617 | 618 | .. [#crdt] See `CRDTs Illustrated `_ for 619 | the definitions of both op-based and state-based CRDTs. The rest of the 620 | video is also a good introduction. See `Wikipedia 621 | `_ for 622 | more detailed information. 623 | 624 | .. [#cset] That is, without tweaking the representation of the unordered set, 625 | which has other implications such as privacy loss or extra storage cost. 626 | TODO: link to Riak's set implementation which uses "casual context". 627 | -------------------------------------------------------------------------------- /causal/05-visibility.rst: -------------------------------------------------------------------------------- 1 | ================== 2 | Partial visibility 3 | ================== 4 | 5 | Representation 6 | ============== 7 | 8 | Member-sets, diff and apply 9 | 10 | For any given message m, some parents might not be visible to some of the 11 | members. Do we also hide the parent references then? How to represent this. 12 | 13 | Security properties we need to keep in mind, and how to achieve these - e.g. 14 | can't allow *arbitrary* different parent pointers for different members. 15 | 16 | Operations on the causal order 17 | ============================== 18 | 19 | Define "cut" in the graph, generalisation of "before" and "after". 20 | 21 | Expand definition of "acked-by". 22 | 23 | Expand algorithms for FIN/ACK. 24 | 25 | Rejoin semantics 26 | ================ 27 | 28 | Unclear what "parent-pointer" semantics should be, e.g. when I rejoin, should I 29 | "point to" my last message from before I left? Probably not - unclear what the 30 | semantics are, not intuitive, etc.. but could explore further. 31 | 32 | If we do disallow the above, we then need to adjust "by(u) is a total order" 33 | checks. In both cases (allow vs disallow), other members should still be able 34 | to identify "this is the same person" across both sub-sessions. (This is for 35 | usability and convenience; if you must be anonymous just rotate public keys.) 36 | 37 | Merge algorithm 38 | =============== 39 | 40 | Under partial visibility, our merge algorithm from the previous section does 41 | not work consistently. Here are some examples that indicate the problem. 42 | 43 | (Taken literally, they ignore that by(u) must be a total order; but we can 44 | regain this property by rewriting "a" instead as multiple members {a1, a2, a3, 45 | ...}, however many is necessary. But for our current purposes, it's easier to 46 | follow if we shortcut this and just write "a" instead.) 47 | 48 | In this example, a has full visibility of the history, and can execute 49 | the merge as normal: 50 | 51 | .. digraph:: merge_full_visibility 52 | 53 | rankdir=BT; 54 | node [style="filled"]; 55 | label="visible to a, not b"; 56 | 57 | O [label="a"]; 58 | A [label="auv"]; 59 | A1 [label="au"]; 60 | A2 [label="av"]; 61 | A -> O [label="+uv"]; 62 | A1 -> A [label="-v"]; 63 | A2 -> A [label="-u"]; 64 | subgraph cluster_d { 65 | label="visible to a, b"; 66 | B [label="ab"]; 67 | C [label="abu",fillcolor="#6666ff"]; 68 | D [label="abv",fillcolor="#6666ff"]; 69 | X [label="ab",fillcolor="#66ff66"]; 70 | C -> B [color="#666666"]; 71 | D -> B [color="#666666"]; 72 | X -> C [color="#666666"]; 73 | X -> D [color="#666666"]; 74 | } 75 | B -> O [label="+b"]; 76 | C -> A1 [color="#666666"]; 77 | D -> A2 [color="#666666"]; 78 | 79 | But if b executes the merge, they have an incomplete view of history, and get a 80 | different result: 81 | 82 | .. digraph:: merge_partial_visibility 83 | 84 | rankdir=BT; 85 | node [style="filled"]; 86 | 87 | B0 [label="ab"]; 88 | C0 [label="abu",fillcolor="#6666ff"]; 89 | D0 [label="abv",fillcolor="#6666ff"]; 90 | X0 [label="abuv",fillcolor="#66ff66"]; 91 | C0 -> B0 [label="\"+u\""]; 92 | D0 -> B0 [label="\"+v\""]; 93 | X0 -> C0 [color="#666666"]; 94 | X0 -> D0 [color="#666666"]; 95 | 96 | Furthermore, b cannot distinguish between this history, vs an alternative 97 | history where a does actually get the same result as b: 98 | 99 | .. digraph:: merge_full_visibility_alternative 100 | 101 | rankdir=BT; 102 | node [style="filled"]; 103 | label="visible to a, not b"; 104 | 105 | O [label="a"]; 106 | A1 [label="au"]; 107 | A2 [label="av"]; 108 | A1 -> O [label="+u"]; 109 | A2 -> O [label="+v"]; 110 | subgraph cluster_d { 111 | label="visible to a, b"; 112 | B [label="ab"]; 113 | C [label="abu",fillcolor="#6666ff"]; 114 | D [label="abv",fillcolor="#6666ff"]; 115 | X [label="abuv",fillcolor="#66ff66"]; 116 | C -> B [color="#666666"]; 117 | D -> B [color="#666666"]; 118 | X -> C [color="#666666"]; 119 | X -> D [color="#666666"]; 120 | } 121 | B -> O [label="+b"]; 122 | C -> A1 [color="#666666"]; 123 | D -> A2 [color="#666666"]; 124 | 125 | Notice how the subgraph "visible to a, b" is identical in both cases, including 126 | even its edges to nodes outside the subgraph. We conclude that there is no way 127 | for "b" to properly execute the merge algorithm. 128 | 129 | Further options 130 | =============== 131 | 132 | At this point maybe we want to go back to exploring CRDTs, since they don't 133 | require history. 134 | 135 | Another option is to force the state representation to increase along a partial 136 | order, in the same way that state-based CRDTs do. For example, instead of 137 | representing ``{a, b, u}`` in the first graph above, we could represent it as 138 | ``{a: 1, b: 1, u: 1, v: -1}`` where: 139 | 140 | - e.g. ``1`` means they are a member of that message, and have been included 1 141 | times before 142 | - e.g. ``-1`` means they are not a member of that message, but previously was, 143 | and have been excluded 1 times before. 144 | 145 | This would allow b to execute the merge correctly. However it reveals to them 146 | that v was part of the previous sub-session, which is what we were trying to 147 | avoid [#mabu]_. Nevertheless, we do still retain message {contents, history} 148 | privacy against b. (TODO: there is probably a CRDT that is equivalent to this.) 149 | 150 | .. [#mabu] In our example there is also a message with members ``abv`` that b 151 | sees, so b will find out regardless - but the author of the message with 152 | members ``abu`` *cannot* rely on this fact in general. 153 | 154 | TODO: write up some reasons that we partially figured out, on why it's probably 155 | impossible to avoid this privacy leak to b, if we want everyone to keep a 156 | consistent view of the session. 157 | -------------------------------------------------------------------------------- /causal/06-application.rst: -------------------------------------------------------------------------------- 1 | Applications 2 | ============ 3 | 4 | Real-world multi-user chat 5 | -------------------------- 6 | 7 | Handling concurrency 8 | -------------------- 9 | -------------------------------------------------------------------------------- /causal/07-interface.rst: -------------------------------------------------------------------------------- 1 | ============================ 2 | User interface and workflows 3 | ============================ 4 | 5 | .. include:: 6 | .. include:: 7 | 8 | Displaying a causal order 9 | ========================= 10 | 11 | It's unclear the "best" way to draw a causal order - there are many layout 12 | algorithms for general DAGs, that optimise for different aims. One approach we 13 | believe is suited for group messaging applications, is to convert the causal 14 | order into a total order (a linear sequence). These are simple to display, and 15 | intuitive as a human interface for our scenario. To be clear, this artifical 16 | order is only used for display; the core system, responsible for security 17 | properties, works entirely and directly on the causal order. 18 | 19 | Our first conclusion is that any linear order must not be the *only* interface 20 | shown to a user. This is because *all* linear conversions are open to context 21 | rebinding. The simplest case of a non-total causal order is G = (a |rightarrow| 22 | o |leftarrow| b). WLOG suppose that our conversion turns G into (o, a, b) for 23 | some user. Given only this information in a display, they cannot distinguish 24 | (o, a) from (a, b) - yet a is not a real parent of b. This might lead to an 25 | attack - if one user sees (o) and has a reasonable expectation that someone 26 | will reply (b), then they may say (a) in the hope that (b) will be linearised 27 | after it, rebinding its context. 28 | 29 | To protect against this, implementions *must* distinguish messages whose pre(m) 30 | does not equal the singleton set containing the preceding message in the total 31 | order; and *should* provide a secondary interface that shows the underlying 32 | causal order. This may be implemented as parent reference annotations, hover 33 | behaviour that highlights the real parents, or some other UX innovation. For 34 | example: 35 | 36 | | Alice: innocent question? 37 | | Chuck: incriminating question? 38 | | Bob: innocent answer! [2] 39 | 40 | The [2] means "the last message(s) the sender saw was the 2nd message above 41 | this one". Messages without annotations are implicitly [1] - this has the nice 42 | advantage of looking identical to the unadorned messages of a naive system that 43 | does not guarantee causal order. In practise, only a minority of messages need 44 | to be annotated as above; we believe this is acceptable for usability. 45 | 46 | TODO 47 | - could minor-warn/notice for messages that are "badly-ordered", or omit annotations for low numbers 48 | 49 | There are several caveats when selecting a linear conversion, and we need to 50 | consider the trade-offs carefully. Let us introduce a few properties that would 51 | be nice to achieve: 52 | 53 | - globally-consistent / canonical: the conversion outputs the same total order 54 | for every user, given the same causal order input 55 | 56 | - append-only: it does not need to insert newly-delivered messages (from the 57 | causally-ordered input) into the middle of a previous totally-ordered output. 58 | In a real application the user only sees the last few messages in the 59 | sequence, and earlier ones are "beyond the scroll bar", so messages inserted 60 | there may never be seen. 61 | 62 | - responsive: it accepts all possible causal orders and may work on them 63 | immediately, without waiting for any other input 64 | 65 | Unfortunately, we cannot achieve all three at once. Suppose we can. WLOG 66 | suppose our conversion turns G (from above) into (o, a, b) for all users. If 67 | one user receives (o |leftarrow| b), and does not receive (a) for a long time, 68 | a responsive conversion must output (o, b). But then when they finally receive 69 | (a), they will need to insert this into the middle of the previous output. 70 | 71 | Delivery order 72 | -------------- 73 | 74 | We believe that global consistency is the least important out of the three 75 | properties above, and therefore sacrificing it is the best option. It's 76 | questionable that it gains us anything - false-but-apparent parents are not 77 | semantic, so it's not important if they are different across users. As noted 78 | above, we ought to have a secondary interface to indicate the real causal order 79 | *anyway*, to guard against context rebinding. 80 | 81 | A linear conversion follows quite naturally from topics already discussed: the 82 | order in which we deliver messages (for any given user) is a total order, being 83 | a topological sort of the causal order. It is practically free to implement - 84 | when the engine delivers a new message, we immediately append it to the current 85 | display, and record the index for later lookup. This conversion is responsive 86 | and append-only, but not globally consistent. It is also nearly identical to 87 | existing chat behaviours, save for the buffering of dangling messages. [#Nres]_ 88 | Given its simplicity and relatively good properties, we recommend this as the 89 | default for implementors. 90 | 91 | Another approach achieves global consistency, but sacrifices one of the other 92 | properties: bounce all messages via a central server, which dictates the 93 | ordering for everyone. This strategy is popular with many existing non-secure 94 | messaging systems. If we embed the causal order, we can re-gain end-to-end 95 | security, and protect against the server from violating causality or context. 96 | Then, we must either sacrifice responsiveness by waiting until the server has 97 | reflected our message back to us before displaying it, or sacrifice append-only 98 | and be prepared to insert others' messages before messages that we sent and 99 | displayed immediately. 100 | 101 | In either case, we still need *another mechanism* to ensure that the server is 102 | providing the same total ordering to everyone, the whole point of doing this in 103 | the first place. TODO: outline a mechanism for this. Or, we could omit this 104 | mechanism and fall back to "delivery order" as above; then "global consistency" 105 | would be achieved on a best-effort basis, relying on the server being friendly. 106 | 107 | .. [#Nres] One may argue that due to this buffering, we are sacrificing 108 | responsiveness; however there is no *additional* waiting beyond what is 109 | necessary in order to achieve causal consistency - as discussed before, 110 | sacrificing the latter results in less security and greater complexity. 111 | 112 | Server-dictated order 113 | --------------------- 114 | 115 | TODO 116 | 117 | See mpCAT. 118 | 119 | Others 120 | ====== 121 | 122 | How to display warnings etc. 123 | -------------------------------------------------------------------------------- /causal/index.rst: -------------------------------------------------------------------------------- 1 | =============== 2 | Causal ordering 3 | =============== 4 | 5 | In this chapter we build a solid semantic data model for our group session, so 6 | that we can build precise security properties on top of it later. This does 7 | *not* mean we ignore security; on the contrary, we solve distributed systems 8 | topics specifically *given* our security constraints, based on the principle 9 | that *if you don't define something, an attacker will define it for you*. 10 | 11 | We begin by describing some properties that we would like our system to have. 12 | 13 | In general, messages are not self-contained but implicitly refer to previous 14 | messages (their `context` or `causes`), and this is critical to their meaning. 15 | A `context rebinding` attack (including `replay attacks`) presents a message to 16 | the victim recipient under a different context, in the hope that the latter 17 | interprets it differently from what its author intended. We'd like to guarantee 18 | not only a message's contents, but its context as well. 19 | 20 | We'd also like members to be able to send messages without requiring approval 21 | from others (or the system as a whole, or any other entity) - or equivalently, 22 | potentially having messages rejected by them. Another nice property is to have 23 | a total (linear) order for our messages, that is globally-consistent. 24 | 25 | Our first obstacle is that we cannot achieve all three properties. [#ordp]_ So 26 | which one shall we abandon? If we abandon the no-approval property, then we can 27 | achieve the others, by using some mechanism (e.g. a consensus protocol) to 28 | approve one message at a time in sequence. However, this has several downsides. 29 | It requires interaction with other members of the session *for every single 30 | message*, which is not feasible in an asynchronous scenario. Rejections must be 31 | handled manually [#arej]_ by the user, but we consider this an unacceptable 32 | UX. Finally, the guarantees of consensus algorithms are much lower than the 33 | computational security guarantees we can achieve when preserving context. 34 | 35 | We consider it unacceptable to abandon the context-preserving property, since 36 | this is a security issue. To abandon the globally-consistent total order has 37 | much less severe consequences. In the messaging layer, we can execute all our 38 | session mechanics, and reason about all our security properties, in terms of 39 | the context-preserving partial (causal) order of messages. Then, in the UI, we 40 | can recover most of the UX advantage that total orders have over partial 41 | orders, by showing a subjective (non-global) linearisation of the "true" order. 42 | 43 | The rest of this chapter will build on top of these choices, and further 44 | explore their consequences. [#cseq]_ We'll also explore more complex mechanics: 45 | in the previous paragraphs, we talked about messages, but we'll also explore 46 | the semantics of membership operations and privacy across subsessions as well. 47 | 48 | (Side topic: some key-rotation ratchets implicitly preserve context, since the 49 | dependencies of which previous keys are used to protect each message matches 50 | the actual causal order of messages. Our explicit formal treatment here works 51 | outside of any ratchet, but we'll return to the relationships between ratchet 52 | message dependency and causal orders in :doc:`another chapter <../ratchet>`.) 53 | 54 | .. toctree:: 55 | :glob: 56 | 57 | * 58 | 59 | .. [#ordp] The basic distributed case is where two members each publish a 60 | message with the same context O, call these a, b, without having seen the 61 | other message first. Forcing these into a total order, WLOG say [O, a, b], 62 | would add a to the apparent context of b, potentially causing it to be 63 | interpreted incorrectly. ∎ Note that this holds regardless of how the total 64 | order is formed - including e.g. by reflecting all messages via a server or 65 | leader - as long as we don't require approvals or allow rejections. 66 | 67 | .. [#arej] Automatic retries are effectively a context rebinding attack upon 68 | yourself: suppose we try to send a message m with context O and it is 69 | rejected. This means some other message m' was accepted also with context 70 | O. If we automatically retry m now, it will have m' in its context, even 71 | though the author of m originally wrote it without such a context. ∎ 72 | 73 | .. [#cseq] Certain consequences we derive may not be appropriate for some 74 | applications. For example, we temporarily prevent some messages from being 75 | displayed in the UI, in some cases. Applications that can accept weaker 76 | ordering guarantees, such as streaming non-sensitive video, may prefer to 77 | avoid this and choose a different system. However, one should be careful 78 | when making this security tradeoff: too often people claim "tradeoff" but 79 | in reality they only considered the usability or performance gain, without 80 | quantitatively understanding the security loss. 81 | -------------------------------------------------------------------------------- /conf.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | # 3 | # msg-notes documentation build configuration file, created by 4 | # sphinx-quickstart on Thu May 8 13:11:30 2014. 5 | # 6 | # This file is execfile()d with the current directory set to its containing dir. 7 | # 8 | # Note that not all possible configuration values are present in this 9 | # autogenerated file. 10 | # 11 | # All configuration values have a default; values that are commented out 12 | # serve to show the default. 13 | 14 | import sys, os 15 | 16 | # If extensions (or modules to document with autodoc) are in another directory, 17 | # add these directories to sys.path here. If the directory is relative to the 18 | # documentation root, use os.path.abspath to make it absolute, like shown here. 19 | #sys.path.insert(0, os.path.abspath('.')) 20 | 21 | # -- General configuration ----------------------------------------------------- 22 | 23 | # If your documentation needs a minimal Sphinx version, state it here. 24 | #needs_sphinx = '1.0' 25 | 26 | # Add any Sphinx extension module names here, as strings. They can be extensions 27 | # coming with Sphinx (named 'sphinx.ext.*') or your custom ones. 28 | extensions = ['sphinx.ext.graphviz'] 29 | 30 | # Add any paths that contain templates here, relative to this directory. 31 | templates_path = ['_templates'] 32 | 33 | # The suffix of source filenames. 34 | source_suffix = '.rst' 35 | 36 | # The encoding of source files. 37 | #source_encoding = 'utf-8-sig' 38 | 39 | # The master toctree document. 40 | master_doc = 'index' 41 | 42 | # General information about the project. 43 | project = u'msg-notes' 44 | copyright = u'See AUTHORS and COPYING, licensed under CC-BY-SA-4.0' 45 | 46 | # The version info for the project you're documenting, acts as replacement for 47 | # |version| and |release|, also used in various other places throughout the 48 | # built documents. 49 | # 50 | # The short X.Y version. 51 | version = '0.0' 52 | # The full version, including alpha/beta/rc tags. 53 | release = '0.0.0' 54 | 55 | # The language for content autogenerated by Sphinx. Refer to documentation 56 | # for a list of supported languages. 57 | #language = None 58 | 59 | # There are two options for replacing |today|: either, you set today to some 60 | # non-false value, then it is used: 61 | #today = '' 62 | # Else, today_fmt is used as the format for a strftime call. 63 | #today_fmt = '%B %d, %Y' 64 | 65 | # List of patterns, relative to source directory, that match files and 66 | # directories to ignore when looking for source files. 67 | exclude_patterns = ['_build', '.git'] 68 | 69 | # The reST default role (used for this markup: `text`) to use for all documents. 70 | #default_role = None 71 | 72 | # If true, '()' will be appended to :func: etc. cross-reference text. 73 | #add_function_parentheses = True 74 | 75 | # If true, the current module name will be prepended to all description 76 | # unit titles (such as .. function::). 77 | #add_module_names = True 78 | 79 | # If true, sectionauthor and moduleauthor directives will be shown in the 80 | # output. They are ignored by default. 81 | #show_authors = False 82 | 83 | # The name of the Pygments (syntax highlighting) style to use. 84 | pygments_style = 'sphinx' 85 | 86 | # A list of ignored prefixes for module index sorting. 87 | #modindex_common_prefix = [] 88 | 89 | 90 | # -- Options for HTML output --------------------------------------------------- 91 | 92 | # The theme to use for HTML and HTML Help pages. See the documentation for 93 | # a list of builtin themes. 94 | html_theme = 'default' 95 | 96 | # Theme options are theme-specific and customize the look and feel of a theme 97 | # further. For a list of options available for each theme, see the 98 | # documentation. 99 | #html_theme_options = {} 100 | 101 | # Add any paths that contain custom themes here, relative to this directory. 102 | #html_theme_path = [] 103 | 104 | # The name for this set of Sphinx documents. If None, it defaults to 105 | # " v documentation". 106 | #html_title = None 107 | 108 | # A shorter title for the navigation bar. Default is the same as html_title. 109 | #html_short_title = None 110 | 111 | # The name of an image file (relative to this directory) to place at the top 112 | # of the sidebar. 113 | #html_logo = None 114 | 115 | # The name of an image file (within the static path) to use as favicon of the 116 | # docs. This file should be a Windows icon file (.ico) being 16x16 or 32x32 117 | # pixels large. 118 | #html_favicon = None 119 | 120 | # Add any paths that contain custom static files (such as style sheets) here, 121 | # relative to this directory. They are copied after the builtin static files, 122 | # so a file named "default.css" will overwrite the builtin "default.css". 123 | html_static_path = ['_static'] 124 | 125 | # If not '', a 'Last updated on:' timestamp is inserted at every page bottom, 126 | # using the given strftime format. 127 | #html_last_updated_fmt = '%b %d, %Y' 128 | 129 | # If true, SmartyPants will be used to convert quotes and dashes to 130 | # typographically correct entities. 131 | #html_use_smartypants = True 132 | 133 | # Custom sidebar templates, maps document names to template names. 134 | #html_sidebars = {} 135 | 136 | # Additional templates that should be rendered to pages, maps page names to 137 | # template names. 138 | #html_additional_pages = {} 139 | 140 | # If false, no module index is generated. 141 | #html_domain_indices = True 142 | 143 | # If false, no index is generated. 144 | #html_use_index = True 145 | 146 | # If true, the index is split into individual pages for each letter. 147 | #html_split_index = False 148 | 149 | # If true, links to the reST sources are added to the pages. 150 | #html_show_sourcelink = True 151 | 152 | # If true, "Created using Sphinx" is shown in the HTML footer. Default is True. 153 | #html_show_sphinx = True 154 | 155 | # If true, "(C) Copyright ..." is shown in the HTML footer. Default is True. 156 | #html_show_copyright = True 157 | 158 | # If true, an OpenSearch description file will be output, and all pages will 159 | # contain a tag referring to it. The value of this option must be the 160 | # base URL from which the finished HTML is served. 161 | #html_use_opensearch = '' 162 | 163 | # This is the file name suffix for HTML files (e.g. ".xhtml"). 164 | #html_file_suffix = None 165 | 166 | # Output file base name for HTML help builder. 167 | htmlhelp_basename = 'msg-notes' 168 | 169 | 170 | # -- Options for LaTeX output -------------------------------------------------- 171 | 172 | latex_elements = { 173 | # The paper size ('letterpaper' or 'a4paper'). 174 | 'papersize': 'a4paper', 175 | 176 | # The font size ('10pt', '11pt' or '12pt'). 177 | #'pointsize': '10pt', 178 | 179 | # Additional stuff for the LaTeX preamble. 180 | 'preamble': """ 181 | \\input{../../_include/unicode.tex} 182 | """, 183 | } 184 | 185 | # Grouping the document tree into LaTeX files. List of tuples 186 | # (source start file, target name, title, author, documentclass [howto/manual]). 187 | latex_documents = [ 188 | ('index', 'msg-notes.tex', u'Topics in secure messaging systems', 189 | u'Ximin Luo', 'manual'), 190 | ] 191 | 192 | # The name of an image file (relative to this directory) to place at the top of 193 | # the title page. 194 | #latex_logo = None 195 | 196 | # For "manual" documents, if this is true, then toplevel headings are parts, 197 | # not chapters. 198 | #latex_use_parts = False 199 | 200 | # If true, show page references after internal links. 201 | #latex_show_pagerefs = False 202 | 203 | # If true, show URL addresses after external links. 204 | #latex_show_urls = False 205 | 206 | # Documents to append as an appendix to all manuals. 207 | #latex_appendices = [] 208 | 209 | # If false, no module index is generated. 210 | #latex_domain_indices = True 211 | 212 | 213 | # -- Options for manual page output -------------------------------------------- 214 | 215 | # One entry per manual page. List of tuples 216 | # (source start file, name, description, authors, manual section). 217 | man_pages = [ 218 | ('index', 'msg-notes', u'Topics in secure messaging systems', 219 | [u'Ximin Luo'], 1) 220 | ] 221 | 222 | # If true, show URL addresses after external links. 223 | #man_show_urls = False 224 | 225 | 226 | # -- Options for Texinfo output ------------------------------------------------ 227 | 228 | # Grouping the document tree into Texinfo files. List of tuples 229 | # (source start file, target name, title, author, 230 | # dir menu entry, description, category) 231 | texinfo_documents = [ 232 | ('index', 'msg-notes', u'Topics in secure messaging systems', 233 | u'Ximin Luo', 'msg-notes', 'Topics in secure messaging systems', 234 | 'Miscellaneous'), 235 | ] 236 | 237 | # Documents to append as an appendix to all manuals. 238 | #texinfo_appendices = [] 239 | 240 | # If false, no module index is generated. 241 | #texinfo_domain_indices = True 242 | 243 | # How to display URL addresses: 'footnote', 'no', or 'inline'. 244 | #texinfo_show_urls = 'footnote' 245 | -------------------------------------------------------------------------------- /greet.rst: -------------------------------------------------------------------------------- 1 | ================== 2 | Greeting protocols 3 | ================== 4 | 5 | Subtopics: 6 | 7 | .. toctree:: 8 | :glob: 9 | 10 | greet-* 11 | 12 | TODO: pages for the following 13 | 14 | - Security definitions 15 | - Asynchronity, pre-keys 16 | - DH3 vs Schnorr ZKP 17 | - P2P vs GKA/BD, contributory 18 | - Notes from mpCAT summit #1 19 | -------------------------------------------------------------------------------- /hci.rst: -------------------------------------------------------------------------------- 1 | =============================== 2 | Indicating security information 3 | =============================== 4 | 5 | Here we describe, in abstract terms, concepts that must be communicated to the 6 | user during secure and insecure situations, so that they may distinguish them 7 | and be able to take appropriate action. We do not advocate specific concrete UX 8 | features here (though we give a few suggestions as examples), but we *do* 9 | intend for a designer to use these discussions to develop such proposals. 10 | 11 | Integrity and reliability 12 | ========================= 13 | 14 | Real parents of a message 15 | ------------------------- 16 | 17 | As explained elsewhere, we don't use a consensus mechanism for approving 18 | messages, and therefore the session transcript is necessarily a partial order 19 | and not a total (linear) order - even if there is (e.g.) a central server 20 | dictating a secondary linear order for display purposes. 21 | 22 | This means that some messages may not be displayed directly below the one(s) 23 | that they are directly replying to. Most of the time this isn't a problem, and 24 | human users can naturally resolve any ambiguity. However, for cases where this 25 | is not possible, or if the session is security-critical, then there should be a 26 | secondary interface to indicate the real parents/ancestors of a message. 27 | 28 | Some suggestions include: 29 | 30 | - annotations to indicate real parents, for example:: 31 | 32 | Alice: Who likes ice cream? 33 | Bob: Who likes SSLv3? 34 | Carol: Me! [2] # indicates replying to -2nd message relative to this one 35 | 36 | - highlight (gray out) a message's (non-)ancestors on hover / long-press 37 | 38 | Messages sent during a member change 39 | ------------------------------------ 40 | 41 | TODO(xl): generalise so these aren't specific to GKAs 42 | 43 | It takes *some* time to run the GKA (group key agreement) to change the session 44 | key to include/exclude a user. During this time, messages may be sent (by 45 | anyone) to the older set of members. When the GKA succeeds, there might be a 46 | race condition like this:: 47 | 48 | Bob sends: (1) # final message of final round of GKA 49 | Carol sends: (2) # message encrypted to old members 50 | Alice recvs: (1) from Bob # Alices finishes GKA, and updates her view of the membership 51 | Alice recvs: (2) from Carol 52 | 53 | In the above diagram, (1) and (2) are messages. Carol sends (2) before she has 54 | received (1) from Bob. When Alices receives (1), her UI will say something like 55 | "Dave has joined the room" and her next message will be encrypted to Dave - but 56 | later she will receive (2) from Carol that was *not* encrypted to Dave. 57 | 58 | The UI must be able to show that (2) was sent to the *old* members rather than 59 | the new members. We know this information in the underlying system; the problem 60 | is how to *indicate* it in a non-confusing way. 61 | 62 | Own messages sent after starting a member change 63 | ------------------------------------------------ 64 | 65 | Some membership operations may take some time to complete. If we exclude a 66 | member, and the greeting system does not immediately make this effective, then 67 | the UI for sending messages must *clearly indicate* that messages we send are 68 | still being encrypted to the old membership, that includes the member in the 69 | process of being excluded. 70 | 71 | (The issue also exists for joining, but is less serious since any message 72 | intended for the newer group implicitly has permission to be seen by the older 73 | group.) 74 | 75 | Some suggestions include: 76 | 77 | - Disabling the send button whilst the exclusion operation is in progress. 78 | This degrades the user experience, and should be avoided. 79 | 80 | - Have explicit notices that a member was included / excluded - in addition to 81 | the user list, which is typically not in the primary area of the screen. When 82 | the local user excludes a member, there must already have been an inclusion 83 | notice for that member, (hopefully) conveying to the local user that a 84 | temporary lack of an exclusion notice means they are not yet excluded. 85 | 86 | .. _hci-ordering: 87 | 88 | Messages received out-of-order 89 | ------------------------------ 90 | 91 | We guarantee ordering - roughly, that messages are seen by recipients in the 92 | order in which they were sent - by buffering messages received out-of-order. 93 | Say Alice sends messages (1) and (2) to Bob. If the transport is unreliable and 94 | Bob receives (2) first, then we won't display (2) until we receive (1), so 95 | that we can display both in the correct order. 96 | 97 | If we don't receive (1) after a reasonable time, we *might* try to communicate 98 | this to the user - e.g. a notice like "(2) received out-of-order; waiting for 99 | older (1)". Importantly, we must not display the *contents* of these messages 100 | in waiting, to avoid breaking our security assumptions. [#buf]_ We could also 101 | offer some advice on why this might be happening, but in a way that avoids 102 | giving a false impression on *the exact cause*: the symptom could either 103 | indicate an unreliable transport, or an attack by the transport, or even 104 | another member, but *we don't know which*. 105 | 106 | On the other hand, the above may add unnecessary complexity to the UX, and 107 | burden the user with information that is hard to take action on. So, we might 108 | just simply ignore when messages are being held in the buffer - i.e. to not 109 | reveal this to the user at all, and pretend that we haven't received (2), to 110 | avoid frustrating the user. This may seem unnatural at first, but it is quite 111 | common for transport schemes that provide ordering guarantees - e.g. in TCP, if 112 | higher-sequence packets arrive earlier than lower-sequence ones, they are 113 | buffered completely invisibly to higher-layer applications and to the user. 114 | 115 | As a failsafe to this simple option, we can trigger a warning if held messages 116 | remain in the buffer for too long. Optionally, we could automatically end the 117 | session and then display the contents of the buffered out-of-order messages - 118 | this is safe because there are no further messages to be sent, so there is no 119 | longer any chance of breaking the aforementioned security assumptions. 120 | 121 | .. [#buf] Whenever we send a message, we also implicitly acknowledge all 122 | previous messages, in an efficient way. If we send a message after showing 123 | some messages but not their parents, then this acknowledgement is a lie 124 | that propogates and corrupts other members' views of the session and its 125 | consistency. See :ref:`consistency-without-reliability` for a more detailed 126 | exploration of this. 127 | 128 | Messages not yet acknowledged 129 | ----------------------------- 130 | 131 | After a message is displayed (authored by either us or others), we wait for 132 | everyone else to acknowledge it. During this time, the message is "not 133 | fully-acked", and we also know exactly who has not yet acknowledged it. 134 | Ideally, this set of people will eventually become empty, at which point we 135 | know that everyone has seen it ("fully-acked"). The question is how to display 136 | the "not fully-acked" status during the interim period. For example: 137 | 138 | - We could gray out the message until it is fully-acked. 139 | 140 | - A variant of this is to gray it only after a grace timeout period, if it is 141 | not already fully-acked by that time. The reasoning behind this is that 142 | *every* message will take *some* time for everyone to ack it, but graying 143 | *every* message immediately when displayed might be annoying or weird. 144 | 145 | - We could add extra indicators to a message when it is fully-acked. 146 | 147 | For comparison: At the time of writing, Facebook chat with 4 people looks 148 | something like this:: 149 | 150 | Alice : (1) 151 | Dave : (2) 152 | Carol : (3) 153 | [seen by Bob] 154 | 155 | This means Bob has seen {1,2,3} - this is the equivalent of "acked" in our 156 | system. (Ours is more secure, but the *intended* semantics are the same.) With 157 | our strong ordering guarantees mentioned above, we can additionally deduce that 158 | Carol has seen {1,2,3}, and Dave has seen {1,2} - but what has Alice seen? She 159 | might have seen {1,2} or only {1}; it would be good to let the user know 160 | exactly which. Yes, this extra information is likely unnecessary in "most 161 | typical cases", but it is still good to make it accessible in a secondary 162 | interface, in case the user has critical needs for a particular case. 163 | 164 | As with drop detection, if we reach a good state (here, consistency) *after* 165 | the timeout, we should cancel the warning or at least downgrade its severity. 166 | Also, here the warning is associated with a message that is *already displayed* 167 | to the user. Therefore, it is important that the warning is shown not as a 168 | point-event for the user to dismiss and forget, but as a persistent state 169 | shown together with the message, in the same space and time. TODO(xl): reword 170 | 171 | Users not responding to heartbeats 172 | ---------------------------------- 173 | 174 | We optionally send heartbeats to check that other people are alive. The purpose 175 | of this is to detect that "no messages received from Dave" really means "Dave 176 | didn't send any messages", as opposed to "the attacker dropped all of Dave's 177 | messages". 178 | 179 | There could be some indication of this in the UI - for example, graying out a 180 | username when we don't see their heartbeat responses, etc. Extra information 181 | could be provided via a secondary interface, like "user not responsive since 19 182 | seconds ago". 183 | 184 | Confidentiality and authenticity 185 | ================================ 186 | 187 | TODO 188 | 189 | Other considerations 190 | ==================== 191 | 192 | Secondary interface 193 | ------------------- 194 | 195 | In most typical cases, the user may not care too much about the nuances of some 196 | security properties (e.g. group integrity and reliability), and therefore 197 | shouldn't be burdened by the extra security information that our system offers. 198 | However, in some cases, it may be a strong user concern, so we should provide a 199 | way for the user to access this information; it would be irresponsible not to. 200 | The existence of these features are also a good way of distinguishing our 201 | secure application from competing insecure applications; see next section. 202 | 203 | By *secondary interface*, we mean UI features that exist away from the main 204 | view, so as to not overload the user with too much information. This is meant 205 | to achieve a smooth default UX for most users, but also make detailed security 206 | information accessible for users with high security needs. One approach is to 207 | provide extra information in pop-ups that are only activated by long-press or 208 | double-click on a related component in the primary interface; designers may 209 | evaluate a wide range of other possibilities too. 210 | 211 | High-level summary indicators 212 | ----------------------------- 213 | 214 | The properties above are fine-grained information about error conditions of 215 | specific messages or users. However, the desired state is for there to be no 216 | errors, and hopefully this should hold most of the time (i.e. with a 217 | non-malicious and semi-reliable transport). Rather than scanning our eye 218 | through an entire list of messages or users to check that all of them are OK, 219 | it would be more efficient to have a small number of "summary" indicators. 220 | Users can then tell at-a-glance if everything is OK - if so, there is no more 221 | work to do; if not, *then* they can check a more time-consuming secondary 222 | interface for specific information about errors. 223 | 224 | For example, there could be one indicator for each of the properties mentioned 225 | above, that is quantified over *all* messages/users: 226 | 227 | - no out-of-order messages being queued for display = "not missing any older messages" 228 | - seen a heartbeat for all users recently = "not missing any newer messages" 229 | - all messages acked by everyone else = "everyone else has seen what we've seen" 230 | 231 | The first two could even be combined into the same indicator, to communicate 232 | "not missing any messages". However, it is probably good to separate the third 233 | item, since it takes longer than the other two to reach assurance about. 234 | 235 | "Indicator" is used loosely; of course it is OK and maybe preferable if they 236 | are invisible for good states, and only show themselves when something is 237 | actually wrong. 238 | 239 | Relationships with existing and insecure notions 240 | ------------------------------------------------ 241 | 242 | Many messaging applications already have notions analogue to the ones described 243 | above, but these are not end-to-end secure. For example, with XMPP presence, 244 | the server tells the client when a contact is online/offline, or when they 245 | join/part a channel. Other applications have an idea of delivery receipts, but 246 | these are authenticated by the server rather than by the actual recipient. 247 | 248 | Such information should be regarded as untrusted. We implement *more secure* 249 | (i.e. end-to-end) forms of these notions, so the UI should emphasise the 250 | properties from our system, and de-emphasise the less secure ones. 251 | 252 | In some cases, it could be good to *combine* the information from both sources. 253 | For example, if we haven't received an (authenticated) acknowledgement, we are 254 | *unsure* if a message has been delivered or not. But, if the local internet is 255 | offline, or if the server is unreachable, then we *know* that a message has 256 | *not* been delivered. These differences in meaning could be communicated via 257 | different states (e.g. colours) of the same UI indicator. 258 | 259 | In other cases, the untrusted source may be *replaced completely* by our more 260 | secure notions. For example, in the case of freshness, it is trivial for the 261 | server to report false presences, so our system (authenticated heartbeats) 262 | should be used where possible. For the case of joining/parting a room, this is 263 | probably more appropriate too - i.e. hiding XMPP join/part events, and only 264 | indicate GKA events in the UI. 265 | -------------------------------------------------------------------------------- /index.rst: -------------------------------------------------------------------------------- 1 | .. msg-notes documentation master file, created by 2 | sphinx-quickstart on Thu May 8 13:11:30 2014. 3 | You can adapt this file completely to your liking, but it should at least 4 | contain the root `toctree` directive. 5 | 6 | Topics in secure messaging systems 7 | ================================== 8 | 9 | Current models of secure communication sessions only deal with two parties. In 10 | this scenario, the session exists between single startup and shutdown phases. 11 | From the point of view of each participant, there is only one *other* partner 12 | to the session, and all other parties are untrusted outsiders. 13 | 14 | When we extend communication to multiple parties, we discover qualitatively new 15 | issues. There are more mechanics to the session - members may join and part 16 | within the lifetime of the session, outside of the initial startup or final 17 | shutdown. There are multiple *other* partners, each of which may try to act 18 | against each other. This demands additional security requirements, such as 19 | ensuring that everyone sees the same consistent view of the session, and that 20 | members can read the session only between the time(s) that they join and part. 21 | [#Npart]_ We also work under the premise that every member knows the session 22 | membership, but again only for the periods that they actually participate in 23 | the session. [#Npriv]_ 24 | 25 | In these documents, we discuss these theoretical issues and outline approaches 26 | for dealing with them. We assume an efficient packet/cell/stanza broadcast 27 | operation, that costs near-constant time with the number of participants. We 28 | aim for a single but modular protocol system that can support both synchronous 29 | and asynchronous operation (where not all members may be online simultaneously, 30 | perhaps not even any two), whilst maintaining strong security properties across 31 | all cases. [#Nasync]_ 32 | 33 | **Outline**. First, we deal with session group integrity. One can use existing 34 | two-party protocols to multicast messages to several individuals, but nothing 35 | guarantees the coherent existence of a *united group*. We discuss the topics 36 | within this area, and the high-level design choices available. We then propose 37 | a concrete scheme for solving these issues, namely a causal order of messages, 38 | encoded by unforgeable immutable pointers. [#Nhash]_ This protects against 39 | transport attacks (reorder, replay, drop of messages) and simplifies the 40 | problem of freshness. It enforces session transcript consistency, protecting 41 | against malicious senders. This section is primarily related to distributed 42 | systems, secondarily to security and cryptography. 43 | 44 | Next, we deal with session membership control - protocols that operate on who 45 | is part of the session. (This includes session establishment, which is the only 46 | case that needs to be considered in 2-party protocols.) We discuss the security 47 | properties common in this area, as well as their practical importance. We look 48 | at the tradeoffs between different schemes, in terms of description complexity, 49 | runtime efficiency and security. This section is primarily related to security 50 | and cryptography, and secondarily to distributed systems. 51 | 52 | Third, we look at how the causal ordering may be used as a framework to develop 53 | key-rotation ratchets based on more complex greeting protocols, as an extension 54 | of 2-party DH that current ratchets are based on, by analysing the dependency 55 | relationships between the messages of the underlying protocol. 56 | 57 | Throughout, we discuss UI issues that arise from our proposals. We also present 58 | a summary of generic group messaging UI issues, that all applications will need 59 | to resolve. 60 | 61 | .. [#Npart] Some group communication applications do not require this, but we 62 | think this more closely matches intuition for a "*private*" chat. Also, one 63 | may reverse this behaviour by building *on top of* hide-by-default, but one 64 | cannot reverse show-by-default - so it's more flexible to pick the former. 65 | (Similar to how one can reverse deniability but not non-repudiability.) 66 | 67 | .. [#Npriv] An even more private setting would be to make non-contact members 68 | pseudonymous (e.g. using an ephemeral identity key), but this would involve 69 | more complexity so we'll overlook it for now. However, we believe that our 70 | proposals do not prevent this from being added in the future. 71 | 72 | .. [#Nasync] It is probably true that asynchronous sessions *often* expect 73 | different security and timing/latency properties than synchronous sessions, 74 | but the former does not *necessarily* determine the latter. (TODO: give 75 | examples). Furthermore, these differences are *quantitative* and not 76 | *qualitative* differences. So, we aim for strong security properties *and* 77 | the flexibility of asynchronous operation, and propose all of our timing 78 | rules are proposed in terms of an abstract basic time unit, which can be 79 | chosen by the application based on what its timing/latency properties are. 80 | 81 | .. [#Nhash] These are implemented using cryptographic hashes, but the mentioned 82 | properties are the important interfaces to the overall scheme. 83 | 84 | .. toctree:: 85 | :glob: 86 | :maxdepth: 2 87 | 88 | causal/index 89 | greet 90 | ratchet 91 | hci 92 | appendix/index 93 | 94 | Indices and tables 95 | ================== 96 | 97 | * :ref:`genindex` 98 | * :ref:`modindex` 99 | * :ref:`search` 100 | 101 | -------------------------------------------------------------------------------- /make.bat: -------------------------------------------------------------------------------- 1 | @ECHO OFF 2 | 3 | REM Command file for Sphinx documentation 4 | 5 | if "%SPHINXBUILD%" == "" ( 6 | set SPHINXBUILD=sphinx-build 7 | ) 8 | set BUILDDIR=_build 9 | set ALLSPHINXOPTS=-d %BUILDDIR%/doctrees %SPHINXOPTS% . 10 | set I18NSPHINXOPTS=%SPHINXOPTS% . 11 | if NOT "%PAPER%" == "" ( 12 | set ALLSPHINXOPTS=-D latex_paper_size=%PAPER% %ALLSPHINXOPTS% 13 | set I18NSPHINXOPTS=-D latex_paper_size=%PAPER% %I18NSPHINXOPTS% 14 | ) 15 | 16 | if "%1" == "" goto help 17 | 18 | if "%1" == "help" ( 19 | :help 20 | echo.Please use `make ^` where ^ is one of 21 | echo. html to make standalone HTML files 22 | echo. dirhtml to make HTML files named index.html in directories 23 | echo. singlehtml to make a single large HTML file 24 | echo. pickle to make pickle files 25 | echo. json to make JSON files 26 | echo. htmlhelp to make HTML files and a HTML help project 27 | echo. qthelp to make HTML files and a qthelp project 28 | echo. devhelp to make HTML files and a Devhelp project 29 | echo. epub to make an epub 30 | echo. latex to make LaTeX files, you can set PAPER=a4 or PAPER=letter 31 | echo. text to make text files 32 | echo. man to make manual pages 33 | echo. texinfo to make Texinfo files 34 | echo. gettext to make PO message catalogs 35 | echo. changes to make an overview over all changed/added/deprecated items 36 | echo. linkcheck to check all external links for integrity 37 | echo. doctest to run all doctests embedded in the documentation if enabled 38 | goto end 39 | ) 40 | 41 | if "%1" == "clean" ( 42 | for /d %%i in (%BUILDDIR%\*) do rmdir /q /s %%i 43 | del /q /s %BUILDDIR%\* 44 | goto end 45 | ) 46 | 47 | if "%1" == "html" ( 48 | %SPHINXBUILD% -b html %ALLSPHINXOPTS% %BUILDDIR%/html 49 | if errorlevel 1 exit /b 1 50 | echo. 51 | echo.Build finished. The HTML pages are in %BUILDDIR%/html. 52 | goto end 53 | ) 54 | 55 | if "%1" == "dirhtml" ( 56 | %SPHINXBUILD% -b dirhtml %ALLSPHINXOPTS% %BUILDDIR%/dirhtml 57 | if errorlevel 1 exit /b 1 58 | echo. 59 | echo.Build finished. The HTML pages are in %BUILDDIR%/dirhtml. 60 | goto end 61 | ) 62 | 63 | if "%1" == "singlehtml" ( 64 | %SPHINXBUILD% -b singlehtml %ALLSPHINXOPTS% %BUILDDIR%/singlehtml 65 | if errorlevel 1 exit /b 1 66 | echo. 67 | echo.Build finished. The HTML pages are in %BUILDDIR%/singlehtml. 68 | goto end 69 | ) 70 | 71 | if "%1" == "pickle" ( 72 | %SPHINXBUILD% -b pickle %ALLSPHINXOPTS% %BUILDDIR%/pickle 73 | if errorlevel 1 exit /b 1 74 | echo. 75 | echo.Build finished; now you can process the pickle files. 76 | goto end 77 | ) 78 | 79 | if "%1" == "json" ( 80 | %SPHINXBUILD% -b json %ALLSPHINXOPTS% %BUILDDIR%/json 81 | if errorlevel 1 exit /b 1 82 | echo. 83 | echo.Build finished; now you can process the JSON files. 84 | goto end 85 | ) 86 | 87 | if "%1" == "htmlhelp" ( 88 | %SPHINXBUILD% -b htmlhelp %ALLSPHINXOPTS% %BUILDDIR%/htmlhelp 89 | if errorlevel 1 exit /b 1 90 | echo. 91 | echo.Build finished; now you can run HTML Help Workshop with the ^ 92 | .hhp project file in %BUILDDIR%/htmlhelp. 93 | goto end 94 | ) 95 | 96 | if "%1" == "qthelp" ( 97 | %SPHINXBUILD% -b qthelp %ALLSPHINXOPTS% %BUILDDIR%/qthelp 98 | if errorlevel 1 exit /b 1 99 | echo. 100 | echo.Build finished; now you can run "qcollectiongenerator" with the ^ 101 | .qhcp project file in %BUILDDIR%/qthelp, like this: 102 | echo.^> qcollectiongenerator %BUILDDIR%\qthelp\msg-notes.qhcp 103 | echo.To view the help file: 104 | echo.^> assistant -collectionFile %BUILDDIR%\qthelp\msg-notes.ghc 105 | goto end 106 | ) 107 | 108 | if "%1" == "devhelp" ( 109 | %SPHINXBUILD% -b devhelp %ALLSPHINXOPTS% %BUILDDIR%/devhelp 110 | if errorlevel 1 exit /b 1 111 | echo. 112 | echo.Build finished. 113 | goto end 114 | ) 115 | 116 | if "%1" == "epub" ( 117 | %SPHINXBUILD% -b epub %ALLSPHINXOPTS% %BUILDDIR%/epub 118 | if errorlevel 1 exit /b 1 119 | echo. 120 | echo.Build finished. The epub file is in %BUILDDIR%/epub. 121 | goto end 122 | ) 123 | 124 | if "%1" == "latex" ( 125 | %SPHINXBUILD% -b latex %ALLSPHINXOPTS% %BUILDDIR%/latex 126 | if errorlevel 1 exit /b 1 127 | echo. 128 | echo.Build finished; the LaTeX files are in %BUILDDIR%/latex. 129 | goto end 130 | ) 131 | 132 | if "%1" == "text" ( 133 | %SPHINXBUILD% -b text %ALLSPHINXOPTS% %BUILDDIR%/text 134 | if errorlevel 1 exit /b 1 135 | echo. 136 | echo.Build finished. The text files are in %BUILDDIR%/text. 137 | goto end 138 | ) 139 | 140 | if "%1" == "man" ( 141 | %SPHINXBUILD% -b man %ALLSPHINXOPTS% %BUILDDIR%/man 142 | if errorlevel 1 exit /b 1 143 | echo. 144 | echo.Build finished. The manual pages are in %BUILDDIR%/man. 145 | goto end 146 | ) 147 | 148 | if "%1" == "texinfo" ( 149 | %SPHINXBUILD% -b texinfo %ALLSPHINXOPTS% %BUILDDIR%/texinfo 150 | if errorlevel 1 exit /b 1 151 | echo. 152 | echo.Build finished. The Texinfo files are in %BUILDDIR%/texinfo. 153 | goto end 154 | ) 155 | 156 | if "%1" == "gettext" ( 157 | %SPHINXBUILD% -b gettext %I18NSPHINXOPTS% %BUILDDIR%/locale 158 | if errorlevel 1 exit /b 1 159 | echo. 160 | echo.Build finished. The message catalogs are in %BUILDDIR%/locale. 161 | goto end 162 | ) 163 | 164 | if "%1" == "changes" ( 165 | %SPHINXBUILD% -b changes %ALLSPHINXOPTS% %BUILDDIR%/changes 166 | if errorlevel 1 exit /b 1 167 | echo. 168 | echo.The overview file is in %BUILDDIR%/changes. 169 | goto end 170 | ) 171 | 172 | if "%1" == "linkcheck" ( 173 | %SPHINXBUILD% -b linkcheck %ALLSPHINXOPTS% %BUILDDIR%/linkcheck 174 | if errorlevel 1 exit /b 1 175 | echo. 176 | echo.Link check complete; look for any errors in the above output ^ 177 | or in %BUILDDIR%/linkcheck/output.txt. 178 | goto end 179 | ) 180 | 181 | if "%1" == "doctest" ( 182 | %SPHINXBUILD% -b doctest %ALLSPHINXOPTS% %BUILDDIR%/doctest 183 | if errorlevel 1 exit /b 1 184 | echo. 185 | echo.Testing of doctests in the sources finished, look at the ^ 186 | results in %BUILDDIR%/doctest/output.txt. 187 | goto end 188 | ) 189 | 190 | :end 191 | -------------------------------------------------------------------------------- /ratchet.rst: -------------------------------------------------------------------------------- 1 | ===================== 2 | Key-rotation ratchets 3 | ===================== 4 | 5 | Subtopics: 6 | 7 | .. toctree:: 8 | :glob: 9 | 10 | ratchet-* 11 | 12 | TODO: pages for the following 13 | 14 | - Security definitions: past-conf (fwd-sec) vs future-conf (self-heal) 15 | - Optimisation against a KEX dependency graph 16 | --------------------------------------------------------------------------------