├── .gitignore ├── LICENSE ├── README.md ├── bin ├── de-gfm ├── doilit ├── echars ├── kdrfc ├── kramdown-rfc ├── kramdown-rfc-autolink-iref-cleanup ├── kramdown-rfc-cache-i-d-bibxml ├── kramdown-rfc-cache-subseries-bibxml ├── kramdown-rfc-clean-svg-ids ├── kramdown-rfc-extract-figures-tables ├── kramdown-rfc-extract-markdown ├── kramdown-rfc-extract-sourcecode ├── kramdown-rfc-lsr └── kramdown-rfc2629 ├── data ├── encoding-fallbacks.txt ├── kramdown-rfc2629.erb ├── math.json ├── studly.rb └── v3.rnc ├── examples ├── Makefile ├── draft-ietf-core-block-xx.mkd ├── draft-ietf-core-block-xx.txt ├── draft-ietf-core-block-xx.xml ├── draft-rfcxml-general-template-bare-00.xml-edited.md ├── draft-rfcxml-general-template-standard-00.xml-edited.md ├── skel.mkd ├── skel.txt ├── stupid-s.mkd ├── stupid-s.txt ├── stupid-s.xml ├── stupid.mkd ├── stupid.txt └── stupid.xml ├── kramdown-rfc.gemspec ├── kramdown-rfc2629.gemspec └── lib ├── kramdown-rfc ├── autolink-iref-cleanup.rb ├── command.rb ├── doi.rb ├── erb.rb ├── gzip-clone.rb ├── kdrfc-processor.rb ├── parameterset.rb ├── refxml.rb ├── rexml-all-text.rb ├── rexml-formatters-conservative.rb ├── rfc8792.rb ├── svg-id-cleanup.rb └── yamlcheck.rb └── kramdown-rfc2629.rb /.gitignore: -------------------------------------------------------------------------------- 1 | .refcache 2 | *.gem 3 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Copyright (c) 2010-2015 Carsten Bormann 2 | 3 | Permission is hereby granted, free of charge, to any person obtaining a copy 4 | of this software and associated documentation files (the "Software"), to deal 5 | in the Software without restriction, including without limitation the rights 6 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 7 | copies of the Software, and to permit persons to whom the Software is 8 | furnished to do so, subject to the following conditions: 9 | 10 | The above copyright notice and this permission notice shall be included in 11 | all copies or substantial portions of the Software. 12 | 13 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 14 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 15 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 16 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 17 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 18 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN 19 | THE SOFTWARE. 20 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # kramdown-rfc 2 | 3 | [kramdown][] is a [markdown][] parser by Thomas Leitner, which has a 4 | number of backends for generating HTML, Latex, and markdown again. 5 | 6 | **kramdown-rfc** is an additional backend to that: It allows the 7 | generation of [XML2RFC][] XML markup (originally known as [RFC 2629][] 8 | compliant markup, with a newer version documented in [RFC 7749][]; 9 | version 3 now documented in [RFC 7991][] etc. and [v3][]). 10 | 11 | Who would care? Anybody who is writing Internet-Drafts and RFCs in 12 | the [IETF][] and prefers (or has co-authors who prefer) to do part of 13 | their work in markdown. 14 | 15 | kramdown-rfc is documented on this page, and also on 16 | [the wiki][]. 17 | 18 | [the wiki]: https://github.com/cabo/kramdown-rfc/wiki 19 | 20 | # Usage 21 | 22 | Start by installing the kramdown-rfc gem (this automatically 23 | installs appropriate versions of referenced gems such as kramdown as 24 | well): 25 | 26 | gem install kramdown-rfc 27 | 28 | (Add a `sudo` and a space in front of that command if you don't have 29 | all the permissions needed.) 30 | 31 | The guts of kramdown-rfc are in one Ruby file, 32 | `lib/kramdown-rfc2629.rb` --- this melds nicely into the extension 33 | structure provided by kramdown. `bin/kramdown-rfc` started out as 34 | a simple command-line program showing how to use this, but can now do 35 | much more (see below). 36 | 37 | To use kramdown-rfc, you'll need Ruby (at least version 2.3, but 38 | preferably a current version), and maybe 39 | [XML2RFC][] if you want to see the fruits of your work. 40 | 41 | kramdown-rfc mydraft.md >mydraft.xml 42 | xml2rfc mydraft.xml 43 | 44 | (The most popular file name extension that IETF people have for 45 | markdown is .md -- for those who tend to think about GNU machine 46 | descriptions here, any extension such as .mkd will do, too.) 47 | 48 | A more brief interface for both calling kramdown-rfc and XML2RFC 49 | is provided by `kdrfc`: 50 | 51 | kdrfc mydraft.md 52 | 53 | `kdrfc` can also use a remote installation of XML2RFC if needed: 54 | 55 | kdrfc -r mydraft.md 56 | 57 | # Versions of RFCXML 58 | 59 | Since RFC 8650, RFCs are using an updated grammar as defined in RFC 60 | 7991 to 7998 and further updated informally since, colloquially called "[v3][]". 61 | As RFC 2629 is no longer the governing standard, what was called kramdown-rfc2629 is 62 | now called kramdown-rfc. The latter command defaults to v3 processing 63 | rules; from 2022-02-22T22:02:22 on, the old kramdown-rfc2629 driver program does as well (1.6.1). 64 | (-3/--v3 and -2/--v2 select v3 and v2 explicitly; the latter should 65 | only be needed if there is a reason to to make a document look 66 | like it's 2016.) 67 | 68 | See also [v3 announcement mail][]. 69 | 70 | [v3 announcement mail]: https://mailarchive.ietf.org/arch/msg/rfc-markdown/JC__LDDGuUbSFqyaEntF9r4kwKw 71 | 72 | # Examples 73 | 74 | For historical interest 75 | `stupid.mkd` was an early markdown version of an actual Internet-Draft 76 | (for a protocol called [STuPiD][] \[sic!]). This demonstrated some, 77 | but not all features of kramdown-rfc. Since markdown/kramdown 78 | does not cater for all the structure of an RFC 7991 style document, 79 | some of the markup is in XML, and the example switches between XML and 80 | markdown using kramdown's `{::nomarkdown}` and `{:/nomarkdown}` (this 81 | is ugly, but works well enough). `stupid.xml` and `stupid.txt` show 82 | what kramdown-rfc and xml2rfc make out of this. 83 | 84 | `stupid-s.mkd` is the same document in the new sectionized format 85 | supported by kramdown-rfc. The document metadata are in a short 86 | piece of YAML at the start, and from there, `abstract`, `middle`, 87 | references (`normative` and `informative`) and `back` are sections 88 | delimited in the markdown file. See the example for how this works. 89 | The sections `normative` and `informative` can be populated right from 90 | the metadata, so there is never a need to write XML any more. 91 | Much less scary, and no `{:/nomarkdown}` etc. is needed any more. 92 | Similarly, `stupid-s.xml` and `stupid-s.txt` show what 93 | kramdown-rfc and xml2rfc make out of this. 94 | 95 | `draft-ietf-core-block-xx.mkd` is a real-world example of a current 96 | Internet-Draft done this way. For RFC and Internet-Draft references, 97 | it uses document prolog entities instead of caching the references in 98 | the XML (i.e., not standalone mode, this is easier to handle when 99 | collaborating with XML-only co-authors). See the `bibxml` metadata. 100 | 101 | # The YAML header 102 | 103 | Please consult the examples for the structure of the YAML header, this should be mostly 104 | obvious. The `stand_alone` attribute controls whether the RFC/I-D 105 | references are inserted into the document (yes) or entity-referenced 106 | (no), the latter leads to increased build time, but may be more 107 | palatable for a final XML conversion. 108 | The author entry can be a single hash or a list, as in: 109 | 110 | author: 111 | ins: C. Bormann 112 | name: Carsten Bormann 113 | org: Universität Bremen TZI 114 | abbrev: TZI 115 | street: Bibliothekstr. 1 116 | city: Bremen 117 | code: D-28359 118 | country: Germany 119 | phone: +49-421-218-63921 120 | email: cabo@tzi.org 121 | 122 | or 123 | 124 | author: 125 | - 126 | ins: C. Bormann 127 | name: Carsten Bormann 128 | org: Universität Bremen TZI 129 | email: cabo@tzi.org 130 | - 131 | ins: Z. Shelby 132 | name: Zach Shelby 133 | org: Sensinode 134 | role: editor 135 | street: Kidekuja 2 136 | city: Vuokatti 137 | code: 88600 138 | country: Finland 139 | phone: "+358407796297" 140 | email: zach@sensinode.com 141 | - 142 | role: editor 143 | ins: P. Thubert 144 | name: Pascal Thubert 145 | org: Cisco Systems 146 | abbrev: Cisco 147 | street: 148 | - Village d'Entreprises Green Side 149 | - 400, Avenue de Roumanille 150 | - Batiment T3 151 | city: Biot - Sophia Antipolis 152 | code: '06410' 153 | country: FRANCE 154 | phone: "+33 4 97 23 26 34" 155 | email: pthubert@cisco.com 156 | 157 | (the hash keys are the XML GIs from RFC 7749, with a flattened 158 | structure. As RFC 7749 requires giving both the full name and 159 | surname/initials, we use `ins` as an abbreviation for 160 | "initials/surname". Yes, the toolchain is Unicode-capable, even if 161 | the final RFC output is still in ASCII.) 162 | 163 | Note that the YAML header needs to be syntactically valid YAML. 164 | Where there is a potential for triggering some further YAML feature, a 165 | string should be put in quotes (like the "+358407796297" above, which 166 | might otherwise be interpreted as a number, losing the + sign). 167 | 168 | ## References 169 | 170 | The references section is built from the references listed in the YAML 171 | header and from references made inline to RFCs and I-Ds in the 172 | markdown text. Since kramdown-rfc cannot know whether a reference 173 | is normative or informative, no entry is generated by default in the 174 | references section. By indicating a normative reference as in 175 | `{{!RFC2119}}` or an informative one as in `{{?RFC1925}}`, you can 176 | completely automate the referencing, without the need to write 177 | anything in the header. Alternatively, you can write something like: 178 | 179 | informative: 180 | RFC1925: 181 | normative: 182 | RFC2119: 183 | 184 | and then just write `{{RFC2119}}` or `{{RFC1925}}`. (Yes, there is a 185 | colon in the YAML, because this is a hash that could provide other 186 | information.) 187 | 188 | Since version 1.1, references imported from the [BibXML][] databases 189 | can be supplied with a replacement label (anchor name). E.g., RFC 793 190 | could be referenced as `{{!TCP=RFC0793}}`, further references then just 191 | can say `{{TCP}}`; both will get `[TCP]` as the label. In the 192 | YAML, the same replacement can be expressed as in the first example: 193 | 194 | normative: 195 | TCP: RFC0793 196 | informative: 197 | SST: DOI.10.1145/1282427.1282421 198 | 199 | Notes about this feature: 200 | 201 | * Thank you, Martin Thomson, for supplying an implementation and 202 | insisting this be done. 203 | * While this feature is now available, you are not forced to use it 204 | for everything: readers of documents often benefit from not having 205 | to look up references, so continuing to use the draft names and RFC 206 | numbers as labels may be the preferable style in many cases. 207 | * As a final caveat, renaming anchors does not work in the 208 | `stand_alone: no` mode (except for IANA and DOI), as there is no 209 | such mechanism in XML entity referencing; exporting to XML while 210 | maintaining live references then may require some manual editing to 211 | get rid of the custom anchors. 212 | 213 | If your references are not in the [BibXML][] databases and do not 214 | have a DOI (that also happens to have correct data) either, you need 215 | to spell it out like in the examples below: 216 | 217 | informative: 218 | RFC1925: 219 | WEI: 220 | title: "6LoWPAN: the Wireless Embedded Internet" 221 | # see the quotes above? Needed because of the embedded colon. 222 | author: 223 | - 224 | ins: Z. Shelby 225 | name: Zach Shelby 226 | - 227 | ins: C. Bormann 228 | name: Carsten Bormann 229 | date: 2009 230 | seriesinfo: 231 | ISBN: 9780470747995 232 | ann: This is a really good reference on 6LoWPAN. 233 | ASN.1: 234 | title: > 235 | Information Technology — ASN.1 encoding rules: 236 | Specification of Basic Encoding Rules (BER), Canonical Encoding 237 | Rules (CER) and Distinguished Encoding Rules (DER) 238 | # YAML's ">" syntax used above is a good way to write longer titles 239 | author: 240 | org: International Telecommunications Union 241 | date: 1994 242 | seriesinfo: 243 | ITU-T: Recommendation X.690 244 | REST: 245 | target: http://www.ics.uci.edu/~fielding/pubs/dissertation/fielding_dissertation.pdf 246 | title: Architectural Styles and the Design of Network-based Software Architectures 247 | author: 248 | ins: R. Fielding 249 | name: Roy Thomas Fielding 250 | org: University of California, Irvine 251 | date: 2000 252 | seriesinfo: 253 | "Ph.D.": "Dissertation, University of California, Irvine" 254 | format: 255 | PDF: http://www.ics.uci.edu/~fielding/pubs/dissertation/fielding_dissertation.pdf 256 | COAP: 257 | title: "CoAP: An Application Protocol for Billions of Tiny Internet Nodes" 258 | seriesinfo: 259 | DOI: 10.1109/MIC.2012.29 260 | date: 2012 261 | author: 262 | - 263 | ins: C. Bormann 264 | name: Carsten Bormann 265 | - 266 | ins: A. P. Castellani 267 | name: Angelo P. Castellani 268 | - 269 | ins: Z. Shelby 270 | name: Zach Shelby 271 | IPSO: 272 | title: IP for Smart Objects (IPSO) 273 | author: 274 | - org: 275 | date: false 276 | seriesinfo: 277 | Web: http://ipso-alliance.github.io/pub/ 278 | normative: 279 | ECMA262: 280 | author: 281 | org: European Computer Manufacturers Association 282 | title: ECMAScript Language Specification 5.1 Edition 283 | date: 2011-06 284 | target: http://www.ecma-international.org/publications/files/ecma-st/ECMA-262.pdf 285 | seriesinfo: 286 | ECMA: Standard ECMA-262 287 | RFC2119: 288 | RFC6690: 289 | 290 | (as in the author list, `ins` is an abbreviation for 291 | "initials/surname"; note that the first title had to be put in double 292 | quotes as it contains a colon which is special syntax in YAML.) 293 | Then you can simply reference `{{ASN.1}}` and 294 | `{{ECMA262}}` in the text. (Make sure the reference keys are valid XML 295 | names, though.) 296 | 297 | # Experimental features 298 | 299 | Most of the [kramdown syntax][kdsyntax] is supported and does 300 | something useful; with the exception of the math syntax (math has no 301 | special support in XML2RFC), and HTML syntax of course. 302 | 303 | A number of more esoteric features have recently been added. 304 | (The minimum required version for each full feature is indicated.) 305 | 306 | (1.3.x) 307 | Slowly improving support for SVG generating tools for XML2RFCv3 (i.e., 308 | with `-3` flag). 309 | These tools must be installed and callable from the command line. 310 | 311 | The basic idea is to mark an input code block with one of the following 312 | labels (language types), yielding some plaintext form in the .TXT 313 | output and a graphical form in the .HTML output. The plaintext is the 314 | input in some cases (e.g., ASCII art, `mscgen`), or some plaintext 315 | output generated by the tool (e.g., `plantuml-utxt`). 316 | 317 | Currently supported labels as of 1.3.9: 318 | 319 | * [goat][], [ditaa][]: ASCII (plaintext) art to figure conversion 320 | * [mscgen][]: Message Sequence Charts 321 | * [plantuml][]: widely used multi-purpose diagram generator 322 | * plantuml-utxt: Like plantuml, except that a plantuml-generated 323 | plaintext form is used 324 | * [mermaid][]: Very experimental; the conversion to SVG is prone to 325 | generate black-on-black text in this version 326 | * math: display math using [tex2svg][] for HTML/PDF and [utftex][] 327 | for plaintext 328 | 329 | [goat]: https://github.com/blampe/goat 330 | [ditaa]: https://github.com/stathissideris/ditaa 331 | [mscgen]: http://www.mcternan.me.uk/mscgen/ 332 | [plantuml]: https://plantuml.com 333 | [mermaid]: https://github.com/mermaid-js/mermaid-cli 334 | [tex2svg]: https://github.com/mathjax/MathJax-demos-node/blob/master/direct/tex2svg 335 | [utftex]: https://github.com/bartp5/libtexprintf 336 | 337 | Note that this feature does not play well with the CI (continuous 338 | integration) support in Martin Thomson's [I-D Template][], as that may 339 | not have the tools installed in its docker instance. 340 | 341 | More details have been collected on the [wiki][svg]. 342 | 343 | [svg]: https://github.com/cabo/kramdown-rfc/wiki/SVG 344 | 345 | (1.2.9:) 346 | The YAML header now allows specifying [kramdown_options][]. 347 | 348 | [kramdown_options]: https://kramdown.gettalong.org/options.html 349 | 350 | This was added specifically to provide easier access to the kramdown 351 | `auto_id_prefix` feature, which prefixes by some distinguishing string 352 | the anchors that are auto-generated for sections, avoiding conflicts: 353 | 354 | ```yaml 355 | kramdown_options: 356 | auto_id_prefix: sec- 357 | ``` 358 | 359 | (1.2.8:) 360 | An experimental feature was added to include [BCP 14] boilerplate: 361 | 362 | ```markdown 363 | {::boilerplate bcp14} 364 | ``` 365 | 366 | which saves some typing. Saying "bcp14+" instead of "bcp14" adds some 367 | random clarifications at the end of the [standard boilerplate text][] that 368 | you may or may not want to have. (Do we need other boilerplate items 369 | beyond BCP14?) 370 | 371 | [BCP 14]: https://www.rfc-editor.org/info/bcp14 372 | 373 | [standard boilerplate text]: https://tools.ietf.org/html/rfc8174#page-3 374 | 375 | (1.0.35:) 376 | An experimental command `doilit` has been added. It can be used to 377 | convert DOIs given on the command line into references entries for 378 | kramdown-rfc YAML, saving a lot of typing. Note that the DOI database 379 | is not of very consistent quality, so you likely have to hand-edit the 380 | result before including it into the document (use `-v` to see raw JSON 381 | data from the DOI database, made somewhat readable by converting it 382 | into YAML). Use `-c` to enable caching (requires `open-uri-cached` 383 | gem). Use `-h=handle` in front of a DOI to set a handle different 384 | from the default `a`, `b`, etc. Similarly, use `-x=handle` to 385 | generate XML2RFCv2 XML instead of kramdown-rfc YAML. 386 | 387 | (1.0.31:) 388 | The kramdown `smart_quotes` feature can be controlled better. 389 | By default, it is on (with default kramdown settings), unless `coding: 390 | us-ascii` is in effect (1.3.14: or --v3 is given), in which case it is off by default. 391 | It also can be explicitly set on (`true`) or off (`false`) in the YAML 392 | header, or to a specific value (an array of four kramdown entity names 393 | or character numbers). E.g., for a German text (that is not intended 394 | to become an Internet-Draft), one might write: 395 | 396 | ```yaml 397 | smart_quotes: [sbquo, lsquo, bdquo, ldquo] 398 | pi: 399 | topblock: no 400 | private: yes 401 | ``` 402 | 403 | (1.0.30:) 404 | kramdown-rfc now uses kramdown 1.10, which leads to two notable updates: 405 | 406 | * Support for empty link texts in the standard markdown 407 | reference syntax, as in `[](#RFC7744)`. 408 | * Language names in fenced code blocks now support all characters 409 | except whitespace, so you can go wild with `asn.1` and `C#`. 410 | 411 | A heuristic generates missing initials/surname from the `name` entry 412 | in author information. This should save a lot of redundant typing. 413 | You'll need to continue using the `ins` entry as well if that 414 | heuristic fails (e.g., for Spanish names). 415 | 416 | Also, there is some rather experimental support for markdown display 417 | math (blocks between `$$` pairs) if the `tex2mail` tool is available. 418 | 419 | (1.0.23:) 420 | Move up to kramdown 1.6.0. This inherits a number of fixes and one 421 | nice feature: 422 | Markdown footnote definitions that turn into `cref`s can have their 423 | attributes in the footnote definition: 424 | 425 | ```markdown 426 | {:cabo: source="cabo"} 427 | 428 | (This section to be removed by the RFC editor.)[^1] 429 | 430 | [^1]: here is my editorial comment: warble warble. 431 | {:cabo} 432 | 433 | Another questionable paragraph.[^2] 434 | 435 | [^2]: so why not delete it? 436 | {: source="observer"} 437 | ``` 438 | 439 | (1.0.23:) 440 | As before, IAL attributes on a codeblock go to the figure element. 441 | Language attributes on the code block now become the artwork type, and any 442 | attribute with a name that starts "artwork-" is moved over to the artwork. 443 | So this snippet now does the obvious things: 444 | 445 | ```markdown 446 | ~~~ abnf 447 | a = b / %s"foo" / %x0D.0A 448 | ~~~ 449 | {: artwork-align="center" artwork-name="syntax"} 450 | ``` 451 | 452 | (1.0.22:) 453 | Index entries can be created with `(((item)))` or 454 | `(((item, subitem)))`; use quotes for weird entries: `(((",", comma)))`. 455 | If the index entry is to be marked "primary", prefix an (unquoted) `!` 456 | as in `(((!item)))`. 457 | 458 | In addition, auto-indexing is supported by hijacking the kramdown 459 | "abbrev" syntax: 460 | 461 | *[IANA]: 462 | *[MUST]: BCP14 463 | *[CBOR]: (((Object Representation, Concise Binary))) (((CBOR))) 464 | 465 | The word in square brackets (which must match exactly, 466 | case-sensitively) is entered into the index automatically for each 467 | place where it occurs. If no title is given, just the word is entered 468 | (first example). If one is given, that becomes the main item (the 469 | auto-indexed word becomes the subitem, second example). If full 470 | control is desired (e.g., for multiple entries per occurrence), just 471 | write down the full index entries instead (third example). 472 | 473 | (1.0.20:) 474 | As an alternative referencing syntax for references with text, 475 | `{{ref}}` can be expressed as `[text](#ref)`. As a special case, a 476 | simple `[ref]` is interpreted as `[](#ref)` (except that the latter 477 | syntax is not actually allowed by kramdown). This syntax does not 478 | allow for automatic entry of items as normative/informative. 479 | 480 | (1.0.16:) Markdown footnotes are converted into `cref`s (XML2RFC formal 481 | comments; note that these are only visible if the pi "comments" is set to yes). 482 | The anchor is taken from the markdown footnote name. The source, if 483 | needed, can be supplied by an [IAL][], as in (first example also uses an 484 | [ALD][]): 485 | 486 | ```markdown 487 | {:cabo: source="cabo"} 488 | 489 | (This section to be removed by the RFC editor.)[^1]{:cabo} 490 | 491 | [^1]: here is my editorial comment 492 | 493 | Another questionable paragraph.[^2]{: source="observer"} 494 | 495 | [^2]: so why not delete it 496 | ``` 497 | 498 | Note that XML2RFC v2 doesn't allow structure in crefs. If you put any, 499 | you get the escaped verbatim XML... 500 | 501 | (1.0.11:) Allow overriding "style" attribute (via IAL = 502 | [inline attribute list][IAL]) in lists and spans 503 | as in: 504 | 505 | ```markdown 506 | {:req: counter="bar" style="format R(%d)"} 507 | 508 | {: req} 509 | * Foo 510 | * Bar 511 | * Bax 512 | 513 | Text outside the list, so a new IAL is needed. 514 | 515 | * Foof 516 | * Barf 517 | * Barx 518 | {: req} 519 | ``` 520 | 521 | (1.0.5:) An IAL attribute "cols" can be added to tables to override 522 | the column layout. For example, `cols="* 20 30c r"` sets the width attributes to 523 | 20 and 30 for the middle columns and sets the right two columns to 524 | center and right alignment, respectively. The alignment from `cols` 525 | overrides that from the kramdown table, if present. 526 | 527 | (1.0.2:) An IAL attribute "vspace" can be added to a definition list 528 | to break after the definition term: 529 | 530 | ```markdown 531 | {: vspace="0"} 532 | word: 533 | : definition 534 | 535 | anotherword: 536 | : another definition 537 | ``` 538 | 539 | (0.x:) Files can be included with the syntax `{::include fn}` (needs 540 | to be in column 1 since 1.0.22; can be suppressed for use in servers 541 | by setting environment variable KRAMDOWN_SAFE since 1.0.22). A 542 | typical example from a recent RFC, where the contents of a figure was 543 | machine-generated: 544 | 545 | ```markdown 546 | ~~~~~~~~~~ 547 | {::include ../ghc/packets-new/p4.out} 548 | ~~~~~~~~~~ 549 | {: #example2 title="A longer RPL example"} 550 | ``` 551 | 552 | (0.x:) A page break can be forced by adding a horizontal rule (`----`, 553 | note that this creates ugly blank space in some HTML converters). 554 | 555 | # Risks and Side-Effects 556 | 557 | The code is not very polished, but now quite stable; it has been 558 | successfully used for hundreds of non-trivial Internet-Drafts and RFCs. 559 | You probably still need to 560 | skim [v3][] if you want to write an Internet-Draft, but you 561 | don't really need to understand XML very much. Knowing the basics of 562 | YAML helps with the metadata (but you'll understand it from the 563 | examples). 564 | 565 | Occasionally, you do need to reach through to the XML arcana, e.g. by 566 | setting attribute values using kramdown's ["IAL" syntax][IAL]. 567 | This can for instance be used to obtain unnumbered appendices: 568 | 569 | ```markdown 570 | Acknowledgements 571 | ================ 572 | {: numbered="false"} 573 | 574 | John Mattsson was nice enough to point out the need for this being documented. 575 | ``` 576 | 577 | 578 | Note that this specific example is covered by a predefined 579 | kramdown-rfc ["attribute list definition" (ALD)][ALD]: 580 | 581 | ```markdown 582 | {:unnumbered: numbered="false"} 583 | ``` 584 | 585 | so the conventional way to write this example would be the somewhat simpler: 586 | 587 | ```markdown 588 | Acknowledgements 589 | ================ 590 | {:unnumbered} 591 | 592 | John Mattsson was nice enough to point out the need for this being documented. 593 | ``` 594 | 595 | 596 | # Upconversion 597 | 598 | If you have an old RFC and want to convert it to markdown, try just 599 | using that RFC, it is 80 % there. It may be possible to automate the 600 | remaining 20 % some more, but that hasn't been done. 601 | 602 | If you have XML, there is an experimental upconverter that does 99 % 603 | of the work. Please [contact the 604 | author](mailto:cabo@tzi.org?subject=Markdown%20for%20RFCXML) if you want 605 | to try it. 606 | 607 | Actually, if the XML was generated by kramdown-rfc, you can simply 608 | extract the input markdown from that XML file (but will of course lose 609 | any edits that have been made to the XML file after generation): 610 | 611 | kramdown-rfc-extract-markdown myfile.xml >myfile.md 612 | 613 | 614 | # Tools 615 | 616 | Joe Hildebrand has a 617 | [grunt][] plugin for kramdown-rfc at: 618 | https://github.com/hildjj/grunt-kramdown-rfc2629 619 | . 620 | Get started with it at: 621 | https://github.com/hildjj/grunt-init-rfc 622 | . 623 | This provides a self-refreshing web page with the 624 | kramdown-rfc/xml2rfc rendition of the draft you are editing. 625 | 626 | [grunt]: http://gruntjs.com 627 | 628 | Martin Thomson has an [I-D Template][] for github repositories that enable 629 | collaboration on draft development. 630 | This supports kramdown-rfc out of the 631 | box. Just name your draft like `draft-ietf-unicorn-protocol.md` and 632 | follow the installation instructions. 633 | 634 | [I-D Template]: https://github.com/martinthomson/i-d-template 635 | 636 | # Related Work 637 | 638 | Moving from XML to Markdown for RFC writing apparently is a 639 | no-brainer, so I'm not the only one who has written code for this. 640 | 641 | [Miek Gieben][] has done a [similar thing][pandoc2rfc] employing 642 | pandoc, now documented in [RFC 7328][]. He uses multiple input files instead of 643 | kramdown-rfc's sectionized input format. He keeps the metadata in 644 | a separate XML file, similar to the way the previous version of 645 | kramdown-rfc stored (and still can store) the metadata in XML in 646 | the markdown document. He also uses a slightly different referencing 647 | syntax, which is closer to what markdown does elsewhere but more 648 | verbose (this syntax is now also supported in kramdown-rfc). 649 | (Miek now also has a new thing going on with mostly different syntax, 650 | see [mmark][] and its [github repository][mmark-git].) 651 | 652 | Other human-oriented markup input languages that are being used for authoring RFCXML include: 653 | 654 | * [asciidoc][], with the [asciidoctor-rfc][] tool, as documented in [draft-ribose-asciirfc][]. 655 | * [orgmode][] (please help supply a more specific link here). 656 | 657 | # License 658 | 659 | Since kramdown version 1.0, kramdown itself is MIT licensed, which 660 | made it possible to license kramdown-rfc under the same license. 661 | 662 | [kramdown]: https://kramdown.gettalong.org 663 | [kdsyntax]: http://kramdown.gettalong.org/syntax.html 664 | [IAL]: https://kramdown.gettalong.org/syntax.html#inline-attribute-lists 665 | [ALD]: https://kramdown.gettalong.org/syntax.html#attribute-list-definitions 666 | [stupid]: http://tools.ietf.org/id/draft-hartke-xmpp-stupid-00 667 | [RFC 2629]: https://www.rfc-editor.org/rfc/rfc2629.html 668 | [RFC 7749]: https://www.rfc-editor.org/rfc/rfc7749.html 669 | [RFC 7991]: https://www.rfc-editor.org/rfc/rfc7991.html 670 | [v3]: https://authors.ietf.org/rfcxml-vocabulary 671 | [markdown]: http://en.wikipedia.org/wiki/Markdown 672 | [IETF]: http://www.ietf.org 673 | [Miek Gieben]: http://www.miek.nl/ 674 | [pandoc2rfc]: https://github.com/miekg/pandoc2rfc/ 675 | [XML2RFC]: https://github.com/ietf-tools/xml2rfc 676 | [RFC 7328]: http://tools.ietf.org/html/rfc7328 677 | [mmark-git]: https://github.com/miekg/mmark 678 | [mmark]: https://mmark.nl 679 | [YAML]: http://www.yaml.org/spec/1.2/spec.html 680 | [draft-ribose-asciirfc]: https://tools.ietf.org/html/draft-ribose-asciirfc 681 | [asciidoctor-rfc]: https://github.com/metanorma/asciidoctor-rfc 682 | [asciidoc]: http://www.methods.co.nz/asciidoc/ 683 | [orgmode]: http://orgmode.org 684 | [BibXML]: https://bib.ietf.org/ 685 | -------------------------------------------------------------------------------- /bin/de-gfm: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env ruby -Ku 2 | 3 | Encoding.default_external = "UTF-8" # wake up, smell the coffee 4 | 5 | require 'kramdown' 6 | require 'kramdown-parser-gfm' 7 | 8 | options = '' 9 | while /\A-([4bck]+)\z/ === ARGV[0] 10 | ARGV.shift 11 | options << $1 12 | end 13 | 14 | if /k/ === options # kramdown 15 | MARKDOWN_BR = "\\\\\n" 16 | end 17 | 18 | if /c/ === options # commonmark 19 | MARKDOWN_BR = "\\\n" 20 | end 21 | 22 | if /b/ === options # universal HTML 23 | MARKDOWN_BR = "
\n" 24 | end 25 | 26 | MARKDOWN_BR ||= " \n" # original Gruber 27 | 28 | module Kramdown 29 | 30 | module Converter 31 | 32 | # Converts an element tree to the kramdown format. 33 | class Kramdown < Base 34 | 35 | # Argh 36 | def convert_br(_el, _opts) 37 | MARKDOWN_BR 38 | end 39 | end 40 | end 41 | end 42 | 43 | list_indent = 2 44 | list_indent = 4 if /4/ === options 45 | 46 | doc = Kramdown::Document.new(ARGF.read, input: 'GFM', gfm_quirks: 'paragraph_end', 47 | list_indent: list_indent) 48 | puts doc.to_kramdown 49 | -------------------------------------------------------------------------------- /bin/doilit: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env ruby 2 | require 'yaml' 3 | require 'kramdown-rfc2629' 4 | require 'kramdown-rfc/parameterset' 5 | require 'kramdown-rfc/refxml' 6 | require 'kramdown-rfc/doi' 7 | 8 | # doilit -c 10.6028/NIST.SP.800-183 10.1016/j.adhoc.2015.04.007 10.1109/MIC.2012.29 10.1145/2507924.2507954 9 | 10 | $verbose = false 11 | $fuzzy = false 12 | $handle = "a" 13 | $xml = false 14 | $site = "https://dx.doi.org" 15 | 16 | litent = {} 17 | ARGV.each do |doi| 18 | case doi 19 | when "-c" 20 | begin 21 | require 'open-uri/cached' 22 | rescue LoadError 23 | warn '*** please "gem install open-uri-cached" to enable caching' 24 | end 25 | next 26 | when "-f" 27 | $fuzzy = true 28 | next 29 | when "-v" 30 | $verbose = true 31 | next 32 | when /\A-s=(.*)/ 33 | $site = $1 34 | next 35 | when /\A-h=(.*)/ 36 | $handle = $1 37 | next 38 | when /\A-x=(.*)/ 39 | $handle = $1 40 | $xml = true 41 | next 42 | when /\A-/ 43 | warn "*** Usage: doilit [-c] [-f] [-v] [-h=handle|-x=xmlhandle] doi..." 44 | exit 1 45 | end 46 | 47 | lit = doi_fetch_and_convert(doi, fuzzy: $fuzzy, verbose: $verbose, site: $site) 48 | 49 | while litent[$handle] 50 | $handle.succ! 51 | end 52 | litent[$handle] = lit 53 | end 54 | if $xml 55 | litent.each do |k, v| 56 | puts KramdownRFC::ref_to_xml(k, v) 57 | end 58 | else 59 | # 1.9 compat: s/lines/each_line.to_a/ 60 | puts litent.to_yaml.gsub(/^/, " ").each_line.to_a[1..-1] 61 | end 62 | -------------------------------------------------------------------------------- /bin/echars: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env ruby 2 | require 'unicode/name' 3 | require 'unicode/scripts' 4 | require 'unicode/blocks' 5 | require 'json' 6 | require 'differ' 7 | module Differ 8 | module Format 9 | module Color 10 | class << self 11 | def as_change(change) # monkey patch 12 | as_insert(change) << "\n" << as_delete(change) 13 | end 14 | end 15 | end 16 | end 17 | end 18 | 19 | 20 | def readable(c) 21 | j = c.to_json 22 | if j.size == 3 23 | j[1...-1] 24 | else 25 | j 26 | end 27 | end 28 | 29 | def explain(s) 30 | ret = '' 31 | hist = Hash.new(0) 32 | s.each_char do |c| 33 | hist[c] += 1 unless c.ord.between?(32, 126) 34 | end 35 | hist.delete("\n") 36 | hist.keys.sort.group_by {|c| Unicode::Blocks.block(c)}.each do |block, l| 37 | scripts = Set[] 38 | l.each do |c| 39 | scripts << Unicode::Scripts.scripts(c) 40 | end 41 | ret << "*** #{block}" 42 | ret << " (#{scripts.join})" if scripts.size == 1 43 | ret << "\n" 44 | l.each do |c| 45 | ret << "#{readable(c)}: U+#{"%04X %4d" % [c.ord, hist[c]] 46 | } #{Unicode::Name.correct(c) || 47 | Unicode::Name.label(c) 48 | }" 49 | ret << " (#{Unicode::Scripts.scripts(c).join(", ")})" if scripts.size != 1 50 | ret << "\n" 51 | end 52 | end 53 | ret 54 | end 55 | 56 | s = ARGF.read 57 | es = explain(s) 58 | n = s.unicode_normalize 59 | en = explain(n) 60 | if es == en 61 | puts es 62 | else 63 | puts "*** Warning: some characters are not normalized and are shown in red." 64 | puts " ...showing a normalized variant (NFC) in green." 65 | puts " Lack of normalization may or may not be a problem." 66 | puts " (Characters may appear to be under wrong block heading.)" 67 | puts Differ.diff_by_line(en, es).format_as(:color) 68 | end 69 | -------------------------------------------------------------------------------- /bin/kdrfc: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env ruby -KU 2 | require 'kramdown-rfc/kdrfc-processor' 3 | require 'optparse' 4 | 5 | # try to get this from gemspec. 6 | KDRFC_VERSION=Gem.loaded_specs["kramdown-rfc2629"].version rescue "unknown-version" 7 | 8 | kdrfc = KramdownRFC::KDRFC.new 9 | kdrfc.options.txt = true # default 10 | 11 | op = OptionParser.new do |opts| 12 | opts.banner = <= 1645567342 # Time.parse("2022-02-22T22:02:22Z").to_i 74 | kdrfc.options.v3 = true # new default from the above date 75 | end 76 | end 77 | 78 | warn "*** v2 #{kdrfc.options.v2.inspect} v3 #{kdrfc.options.v3.inspect}" if kdrfc.options.verbose 79 | 80 | case ARGV.size 81 | when 1 82 | fn = ARGV[0] 83 | begin 84 | kdrfc.process(fn) 85 | rescue StandardError => e 86 | warn e.to_s 87 | exit 1 88 | end 89 | else 90 | puts op 91 | exit 1 92 | end 93 | -------------------------------------------------------------------------------- /bin/kramdown-rfc: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env ruby 2 | # -*- coding: utf-8 -*- 3 | require 'ostruct' 4 | 5 | $options ||= OpenStruct.new 6 | 7 | $options.v3 = true 8 | require 'kramdown-rfc/command' 9 | 10 | -------------------------------------------------------------------------------- /bin/kramdown-rfc-autolink-iref-cleanup: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env ruby 2 | # -*- coding: utf-8 -*- 3 | require 'rexml/document' 4 | require 'kramdown-rfc/autolink-iref-cleanup' 5 | 6 | d = REXML::Document.new(ARGF.read) 7 | autolink_iref_cleanup(d) 8 | puts d.to_s 9 | -------------------------------------------------------------------------------- /bin/kramdown-rfc-cache-i-d-bibxml: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env ruby 2 | # prerequisite: 3 | # gem install net-http-persistent 4 | # 5 | # dumps all bibxml for current, "Active" I-Ds in cache 6 | # reasonably efficient after initial call if the output is retained 7 | # 8 | # requires Ruby 2.4 or above because of "liberal_parsing" option 9 | # 10 | # uses ENV["KRAMDOWN_REFCACHEDIR"] for where you want to have your bibxml3 data 11 | # 12 | 13 | require 'csv' 14 | require 'fileutils' 15 | 16 | begin 17 | require 'net/http/persistent' 18 | rescue LoadError 19 | warn "*** please install net-http-persistent:" 20 | warn " gem install net-http-persistent" 21 | warn "(prefix by sudo only if required)." 22 | exit 72 # EX_OSFILE 23 | end 24 | 25 | 26 | TARGET_DIR = ENV["KRAMDOWN_REFCACHEDIR"] || ( 27 | path = File.expand_path("~/.cache/xml2rfc") 28 | warn "*** set environment variable KRAMDOWN_REFCACHEDIR to #{path} to actually use the cache" 29 | path 30 | ) 31 | 32 | FileUtils.mkdir_p(TARGET_DIR) 33 | FileUtils.chdir(TARGET_DIR) 34 | 35 | $http = Net::HTTP::Persistent.new name: 'allid' 36 | 37 | KRAMDOWN_PERSISTENT_VERBOSE = true 38 | 39 | def get_and_write_resource_persistently(url, fn, age_verbose=false) 40 | t1 = Time.now 41 | response = $http.request(URI(url)) 42 | if response.code != "200" 43 | raise "*** Status code #{response.code} while fetching #{url}" 44 | else 45 | File.write(fn, response.body) 46 | end 47 | t2 = Time.now 48 | warn "#{url} -> #{fn} (#{"%.3f" % (t2 - t1)} s)" if KRAMDOWN_PERSISTENT_VERBOSE 49 | if age_verbose 50 | if age = response.get_fields("age") 51 | warn "(working from a web cache, index is #{age.first} seconds stale)" 52 | end 53 | end 54 | end 55 | 56 | CLEAR_RET = "\e[K\r" # XXX all the world is ECMA-48 (ISO 6429), no? 57 | 58 | def noisy(name) 59 | print "#{name}...#{CLEAR_RET}" 60 | end 61 | def clear_noise 62 | print CLEAR_RET 63 | end 64 | 65 | ALL_ID2_SOURCE = "https://www.ietf.org/id/all_id2.txt" 66 | ALL_ID2_COPY = ".all_id2.txt" 67 | 68 | get_and_write_resource_persistently(ALL_ID2_SOURCE, ALL_ID2_COPY, true) unless ENV["KRAMDOWN_DONT_REFRESH_ALL_ID2"] 69 | ix = File.read(ALL_ID2_COPY).lines.grep_v(/^#/).join 70 | 71 | csv = CSV.new(ix, col_sep: "\t", liberal_parsing: true) 72 | 73 | drafts = csv.read 74 | active = drafts.select { |d| d[2] == "Active" } 75 | active_names = active.map { |a| a[0] } 76 | puts "#{active_names.size} active drafts" 77 | 78 | active_names.each do |name| 79 | if name =~ /\Adraft-(.*)-(\d\d)\z/ 80 | namepart = $1 81 | version = $2 82 | name0 = "reference.I-D.#{namepart}.xml" 83 | noisy(name0) if File.exist?(name0) 84 | name1 = "reference.I-D.draft-#{namepart}-#{version}.xml" 85 | if File.exist?(name1) 86 | noisy(name1) 87 | FileUtils.touch(name0) # because name1 already exists, we believe name0 is fresh 88 | else 89 | begin 90 | url0 = "https://datatracker.ietf.org/doc/bibxml3/draft-#{namepart}.xml" 91 | get_and_write_resource_persistently(url0, name0) # get name0 first 92 | url1 = "https://datatracker.ietf.org/doc/bibxml3/draft-#{namepart}-#{version}.xml" 93 | get_and_write_resource_persistently(url1, name1) # then name1 to mark this as updated 94 | rescue => e 95 | warn "*** #{name0}: #{e}" 96 | end 97 | end 98 | else 99 | warn "*** Malformed draft name: #{name}" 100 | end 101 | end 102 | clear_noise 103 | -------------------------------------------------------------------------------- /bin/kramdown-rfc-cache-subseries-bibxml: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env ruby 2 | # prerequisite: 3 | # gem install net-http-persistent 4 | # 5 | # generates referencegroup files for BCP and STD series (like bibxml9) 6 | # unfortunately, needs to re-fetch rfc-index.xml 7 | # uses pretty slow built-in XML parser, so it will take a while even on regen 8 | # 9 | # uses ENV["KRAMDOWN_REFCACHEDIR"] for where you want to have your bibxml9 data 10 | # 11 | 12 | require 'rexml/document' 13 | require 'fileutils' 14 | 15 | begin 16 | require 'net/http/persistent' 17 | rescue LoadError 18 | warn "*** please install net-http-persistent:" 19 | warn " gem install net-http-persistent" 20 | warn "(prefix by sudo only if required)." 21 | exit 72 # EX_OSFILE 22 | end 23 | 24 | 25 | TARGET_DIR = ENV["KRAMDOWN_REFCACHEDIR"] || ( 26 | path = File.expand_path("~/.cache/xml2rfc") 27 | warn "*** set environment variable KRAMDOWN_REFCACHEDIR to #{path} to actually use the cache" 28 | path 29 | ) 30 | 31 | FileUtils.mkdir_p(TARGET_DIR) 32 | FileUtils.chdir(TARGET_DIR) 33 | 34 | $http = Net::HTTP::Persistent.new name: 'subseries' 35 | 36 | KRAMDOWN_PERSISTENT_VERBOSE = true 37 | 38 | def get_and_write_resource_persistently(url, fn, age_verbose=false) 39 | t1 = Time.now 40 | response = $http.request(URI(url)) 41 | if response.code != "200" 42 | raise "*** Status code #{response.code} while fetching #{url}" 43 | else 44 | File.write(fn, response.body) 45 | end 46 | t2 = Time.now 47 | warn "#{url} -> #{fn} (#{"%.3f" % (t2 - t1)} s)" if KRAMDOWN_PERSISTENT_VERBOSE 48 | if age_verbose 49 | if age = response.get_fields("age") 50 | warn "(working from a web cache, index is #{age.first} seconds stale)" 51 | end 52 | end 53 | end 54 | 55 | CLEAR_RET = "\e[K\r" # XXX all the world is ECMA-48 (ISO 6429), no? 56 | 57 | def noisy(name) 58 | print "#{name}...#{CLEAR_RET}" 59 | end 60 | def clear_noise 61 | print CLEAR_RET 62 | end 63 | 64 | def normalize_name(n) 65 | n.sub(/([A-Z])0+/) {$1} 66 | end 67 | 68 | def regress_name(n) 69 | n.sub(/([A-Z])(\d+)/) {"#$1#{"%04d" % $2.to_i}"} 70 | end 71 | 72 | def regress_name_dot(n) 73 | n.sub(/([A-Z])(\d+)/) {"#$1.#{"%04d" % $2.to_i}"} 74 | end 75 | 76 | def get_ref(rfcname) 77 | name = "reference.#{regress_name_dot(rfcname)}.xml" 78 | begin 79 | file = File.read(name) # no age check 80 | rescue Errno::ENOENT 81 | get_and_write_resource_persistently("https://www.rfc-editor.org/refs/bibxml/" << name, name) 82 | file = File.read(name) 83 | end 84 | d = REXML::Document.new(file) 85 | d.xml_decl.nowrite 86 | "\n" << d.to_s.lstrip 87 | end 88 | 89 | def create_bib(series, subname, rfcnames) 90 | p [series, subname, rfcnames] 91 | subname_norm = normalize_name(subname) 92 | refs = %{\n} 93 | refs << %{\n} 94 | refs << rfcnames.map {|x| get_ref(x)}.join 95 | refs << "\n" 96 | File.write("reference.#{regress_name_dot(subname)}.xml", refs) 97 | end 98 | 99 | def handle_sub(series, entry) 100 | ids = entry.get_elements("doc-id") 101 | warn "** ids #{ids} #{entry}" unless ids.size == 1 102 | subname = ids.first.text 103 | isalso = entry.get_elements("is-also") 104 | if isalso.size == 1 105 | rfcs = isalso.first.get_elements("doc-id") 106 | if rfcs.size > 0 107 | rfcnames = rfcs.map {|r| r.text} 108 | create_bib(series, subname, rfcnames) 109 | end 110 | end 111 | end 112 | 113 | RFCINDEX_SOURCE = "https://www.rfc-editor.org/rfc/rfc-index.xml" 114 | RFCINDEX_COPY = File.basename(RFCINDEX_SOURCE) 115 | 116 | get_and_write_resource_persistently(RFCINDEX_SOURCE, RFCINDEX_COPY, true) unless ENV["KRAMDOWN_DONT_REFRESH_RFCINDEX"] 117 | 118 | doc = REXML::Document.new(File.read(RFCINDEX_COPY)) 119 | REXML::XPath.each(doc.root, "/rfc-index/bcp-entry") { |e| handle_sub("BCP", e) } 120 | REXML::XPath.each(doc.root, "/rfc-index/std-entry") { |e| handle_sub("STD", e) } 121 | -------------------------------------------------------------------------------- /bin/kramdown-rfc-clean-svg-ids: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env ruby 2 | # -*- coding: utf-8 -*- 3 | 4 | require 'rexml/document' 5 | require 'kramdown-rfc/rexml-formatters-conservative' 6 | require 'kramdown-rfc/svg-id-cleanup' 7 | 8 | def svg_clean_ids(s) 9 | d = REXML::Document.new(s) 10 | d.context[:attribute_quote] = :quote # Set double-quote as the attribute value delimiter 11 | 12 | svg_id_cleanup(d) 13 | 14 | tr = REXML::Formatters::Conservative.new 15 | o = '' 16 | tr.write(d, o) 17 | o 18 | rescue => detail 19 | warn "*** Can't clean SVG: #{detail}" 20 | d.to_s 21 | end 22 | 23 | puts svg_clean_ids(ARGF.read) 24 | -------------------------------------------------------------------------------- /bin/kramdown-rfc-extract-figures-tables: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env ruby -KU 2 | require 'rexml/document' 3 | require 'yaml' 4 | require 'optparse' 5 | 6 | PROGNAME = $0 # "kramdown-rfc-extract-figures-tables" 7 | 8 | target = :section 9 | targets = [:section, :note, :rfc] 10 | require 'optparse' 11 | begin 12 | op = OptionParser.new do |opts| 13 | opts.banner = "Usage: #{PROGNAME} [options] [-|document.xml]" 14 | opts.on("-tFMT", "--to=FMT", targets, "Target format #{targets.map(&:to_s)}") do |v| 15 | target = v 16 | end 17 | end 18 | op.parse! 19 | rescue Exception => e 20 | warn e 21 | exit 1 22 | end 23 | 24 | class String 25 | def spacify 26 | gsub(/\s+/, " ") 27 | end 28 | end 29 | 30 | lists = Hash.new { |h, k| h[k] = [] } 31 | d = REXML::Document.new(ARGF) 32 | unless d.root 33 | warn "** #{PROGNAME}: Cannot parse input" 34 | exit 1 35 | end 36 | REXML::XPath.each(d.root, %{//figure|//*[name()="table" or name()="texttable"]}) do |x| 37 | gi = x.name 38 | ref = x[:anchor] 39 | # p [gi, ref] 40 | out = [] 41 | REXML::XPath.each(x, "name") do |nm| 42 | out << nm.children.map{|ch| ch.to_s}.join 43 | # p [gi, out.last] 44 | end 45 | REXML::XPath.each(x, "@title") do |ttl| 46 | out << ttl.to_s.spacify 47 | # p [gi, out.last] 48 | end 49 | gi1 = if gi == "texttable"; "table" else gi end 50 | if out == [] # nameless 51 | # nameless, anchorless fig doesn't get a number; ignore 52 | next if gi == "figure" && !ref 53 | # Synthesize name (if not redundant) 54 | out = ["#{gi1.capitalize} #{lists[gi1].size + 1}"] if !ref 55 | end 56 | # p [gi, out] 57 | lists[gi1] << [ref, out.join(" ")] 58 | end 59 | 60 | lists.each do |k, v| 61 | item = k.capitalize 62 | title = "List of #{item}s" 63 | case target 64 | when :note 65 | puts 66 | puts "--- note_#{title.gsub(" ", "_")}" 67 | puts 68 | when :section, :rfc 69 | puts 70 | puts "# #{title}" 71 | puts "{:unnumbered}" 72 | puts 73 | puts "{:compact#{" hangindent=\"11\"" if target == :rfc}}" 74 | else 75 | fail 76 | end 77 | v.each_with_index do |(ref, ti), n| 78 | ti.sub!(/,?[\p{Zl}\p{Zp}\p{Cc}].*/, "") # first line of caption only 79 | if target == :rfc 80 | if ref 81 | puts "{{#{ref}}}:" 82 | puts ": {{<<#{ref}}}" 83 | else 84 | puts "#{item} #{n+1}:" 85 | puts ": #{ti}" 86 | warn "** No anchor on #{item} #{n+1} \"#{ti}\"" 87 | end 88 | puts 89 | else 90 | ti = "[#{ti}](##{ref})" if ref 91 | puts "#{n+1}. #{ti}" 92 | end 93 | end 94 | end 95 | -------------------------------------------------------------------------------- /bin/kramdown-rfc-extract-markdown: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env ruby -KU 2 | require 'kramdown-rfc/gzip-clone' 3 | require 'base64' 4 | 5 | EMBEDDED_RE = %r{} 6 | embedded = ARGF.read.scan(EMBEDDED_RE) 7 | unless embedded.empty? 8 | puts Gzip.decompress(Base64.decode64(embedded[0][0])) 9 | else 10 | warn "*** No embedded markdown source found!" 11 | end 12 | -------------------------------------------------------------------------------- /bin/kramdown-rfc-extract-sourcecode: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env ruby -KU 2 | require 'rexml/document' # for SVG and bibxml acrobatics 3 | require 'yaml' 4 | require 'shellwords' 5 | require 'fileutils' 6 | require_relative '../lib/kramdown-rfc/rfc8792' 7 | require_relative '../lib/kramdown-rfc/rexml-all-text.rb' 8 | 9 | def clean(s) 10 | s.gsub!(/\//, ":") 11 | s.gsub!(/\A([-.]+)/) {"_" * $1.size } 12 | s.gsub!(/[^-.:\w]+/, "-") 13 | s 14 | end 15 | 16 | $seen_slugs = {} 17 | 18 | def slugify_like_xml2rfc(s, delta = 0) 19 | s = s.unicode_normalize(:nfkd).force_encoding(Encoding::ASCII).scrub('').downcase 20 | s.gsub!(/[^-\w\s\/]/, '') 21 | s = s.strip 22 | s.gsub!(/[-\s\/]/, '-') 23 | n = 32 - delta 24 | nmax = [s.size, 40 - delta].min 25 | while $seen_slugs[slugex = s[0...n]] && n < nmax 26 | n += 1 27 | end 28 | if $seen_slugs[slugex] 29 | m = 2 30 | while $seen_slugs[slugex = "#{s[0...n]}-#{m}"] 31 | m += 1 32 | if m == 1000 33 | raise ArgumentError, "Can't build distinguishable slug for '#{s}'" 34 | end 35 | end 36 | end 37 | $seen_slugs[slugex] = s 38 | slugex 39 | end 40 | 41 | target = nil 42 | dir = nil 43 | out_type = out_name = nil 44 | unfold = true 45 | targets = [:list, :files, :zip, :yaml] 46 | require 'optparse' 47 | begin 48 | op = OptionParser.new do |opts| 49 | opts.banner = "Usage: kramdown-rfc-extract-sourcecode [options] document.xml" 50 | opts.on("-tFMT", "--to=FMT", targets, "Target format #{targets.map(&:to_s)}") do |v| 51 | target = v 52 | end 53 | opts.on("-dDIR", "--dir=DIR", "Target directory (default: sourcecode)") do |v| 54 | dir = v 55 | end 56 | opts.on("-xTYPE/NAME", "--extract=TYPE/NAME", "Extract single item to stdout") do |v| 57 | target = :stdout 58 | out_type, out_name = v.split("/", 2) 59 | end 60 | opts.on("-f", "--[no-]unfold", "RFC8792-unfold (default: yes)") do |v| 61 | unfold = v 62 | end 63 | end 64 | op.parse! 65 | rescue Exception => e 66 | warn e 67 | exit 1 68 | end 69 | if dir 70 | target ||= :files 71 | unless [:files, :zip].include? target 72 | warn "** Unused argument --dir=#{dir}" 73 | end 74 | end 75 | dir ||= "sourcecode" 76 | 77 | target ||= :list 78 | 79 | gensym = "unnamed-000" 80 | taken = Hash.new { |h, k| h[k] = Hash.new } 81 | warned = Hash.new { |h, k| h[k] = Hash.new } 82 | filenames = ARGV.inspect 83 | begin 84 | d = REXML::Document.new(ARGF) 85 | dr = d.root 86 | unless dr 87 | warn "*** Can't parse: #{filenames}" 88 | exit(1) 89 | end 90 | rescue Errno::ENOENT 91 | warn "*** Not found: #{filenames}" 92 | exit(1) 93 | rescue => e 94 | begin 95 | warn "*** Can't parse: #{filenames} (#{e.to_s[0..120]})" 96 | rescue => e 97 | warn "*** Can't parse: #{filenames} (#{e.to_s[0..120]})" 98 | end 99 | exit(1) 100 | end 101 | REXML::XPath.each(dr, "//sourcecode|//artwork") do |x| 102 | if ty = x[:type] 103 | is_svg = false 104 | REXML::XPath.each(x, "svg") do is_svg = true end 105 | ty = clean(ty) 106 | if is_svg && ty != "svg" 107 | warn "** replacing SVG type '#{ty}' by type 'svg'" 108 | ty = "svg" 109 | end 110 | ext = ty 111 | if ty.empty? 112 | ty = "=txt" 113 | ext = "txt" 114 | end 115 | name = x[:name] 116 | name_is_from_slug = false 117 | if !name || name.empty? 118 | REXML::XPath.each(x, "ancestor::*/name[position() = 1]") do |nameel| 119 | unless sname = nameel['slugifiedName'] 120 | alltext = nameel.all_text 121 | nameel.add_attribute('slugifiedName', 122 | sname = clean("name-" << slugify_like_xml2rfc(alltext, 5))) 123 | end 124 | name = clean(sname.to_s.sub(/^name-/, '')) 125 | name_is_from_slug = true 126 | end 127 | if !name || name.empty? 128 | name = gensym.succ!.dup 129 | end 130 | name = "#{name}.#{ext}" 131 | end 132 | name = clean(name) 133 | if taken[ty][name] 134 | if name_is_from_slug || is_svg 135 | # rename old entry as well if it was from slug? 136 | nameparts = name.split(".") 137 | if is_svg && nameparts[-1] != "svg" 138 | nameparts << "svg" 139 | end 140 | ext1 = nameparts.pop || "txt" 141 | suffix = "b" 142 | while taken[ty][name = [*nameparts[0...-1], "#{nameparts[-1]}-#{suffix}", ext1].join(".")] 143 | suffix.succ! 144 | end 145 | taken[ty][name] = '' 146 | else 147 | unless warned[ty][name] 148 | warn "Concatenating to #{ty}/#{name}." 149 | warned[ty][name] = true 150 | end 151 | end 152 | else 153 | taken[ty][name] = '' 154 | end 155 | extracted = false 156 | REXML::XPath.each(x, "svg") do |svg| 157 | # According to v3.rnc, there should be only one... 158 | taken[ty][name] << svg.to_s 159 | extracted = true 160 | end 161 | unless extracted 162 | taken[ty][name] << handle_artwork_sourcecode(x.all_text, unfold) 163 | end 164 | end 165 | end 166 | 167 | def make_directory_from(dir, taken) 168 | if File.exist?(dir) 169 | bak = "#{dir}.bak" 170 | begin 171 | FileUtils.mv(dir, bak) 172 | rescue Errno::EEXIST 173 | bak.succ! 174 | retry 175 | end 176 | end 177 | FileUtils.mkdir_p(dir) 178 | taken.each { |dir1, v| 179 | FileUtils.mkdir_p("#{dir}/#{dir1}") 180 | v.each { |fn, value| 181 | IO.write("#{dir}/#{dir1}/#{fn}", value) 182 | } 183 | } 184 | end 185 | 186 | case target 187 | when :stdout 188 | if dataset = taken[out_type] 189 | unless out_name 190 | case dataset.size 191 | when 0 192 | warn "No sourcecodes under #{out_type}" 193 | exit(1) 194 | when 1 195 | out_name ||= dataset.keys.first 196 | else 197 | warn "Multiple sourcecodes under #{out_type}: #{dataset.keys}" 198 | exit(1) 199 | end 200 | end 201 | if data = dataset[out_name] 202 | puts data 203 | exit 204 | end 205 | end 206 | warn "*** Cannot find sourcecode #{out_type}/#{out_name}" 207 | exit(1) 208 | when :yaml 209 | puts taken.to_yaml 210 | when :list 211 | puts Hash[ 212 | taken.map {|k, v| 213 | [k, v.keys] 214 | } 215 | ].to_yaml 216 | when :files 217 | make_directory_from(dir, taken) 218 | when :zip 219 | make_directory_from(dir, taken) 220 | zip = "#{dir}.zip" 221 | if File.exist?(zip) 222 | bak = "#{zip}.bak" 223 | begin 224 | FileUtils.mv(zip, bak) 225 | rescue Errno::EEXIST # This doesn't actually happen. XXX 226 | bak.succ! 227 | retry 228 | end 229 | end 230 | cmd = ["zip", "-mr", zip, dir].shelljoin 231 | warn cmd 232 | system(cmd) 233 | end 234 | -------------------------------------------------------------------------------- /bin/kramdown-rfc-lsr: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env ruby -KU 2 | # frozen_string_literal: true 3 | 4 | # List Section References from a RFCXML document 5 | # 6 | # (PoC, in urgent need of refactoring) 7 | # Requires xml2rfc and tidy commands in path 8 | # Use without open-uri-cached is untested 9 | 10 | require 'rexml/document' 11 | require 'yaml' 12 | require 'json' 13 | require 'shellwords' 14 | require 'fileutils' 15 | begin 16 | require 'open-uri/cached' 17 | rescue LoadError 18 | warn '*** please "gem install open-uri-cached" to enable caching' 19 | require 'open-uri' 20 | end 21 | require_relative '../lib/kramdown-rfc/rexml-all-text.rb' 22 | 23 | target = :shortname 24 | require 'optparse' 25 | begin 26 | op = OptionParser.new do |opts| 27 | opts.banner = "Usage: kramdown-rfc-lsr [options] xml-source" 28 | opts.on("-u", "--url", "Source is URL") do |v| 29 | target = :url 30 | end 31 | opts.on("-s", "--shortname", "Source is shortname (default)") do |v| 32 | target = :shortname 33 | end 34 | opts.on("-f", "--file", "Source is filename") do |v| 35 | target = :file 36 | end 37 | end 38 | op.parse! 39 | rescue Exception => e 40 | warn e 41 | exit 1 42 | end 43 | 44 | $exit_code = 0 45 | 46 | if target != :file && ARGV.size != 1 47 | puts op 48 | exit 1 49 | end 50 | xmlsource = ARGV[0] 51 | if target == :shortname 52 | xmlsource = case xmlsource 53 | when /^(?:rfc)?(\d+)$/i 54 | "https://www.rfc-editor.org/rfc/rfc#{$1.to_i.to_s}.xml" 55 | when /^(?:draft-|I-D.|)(.*-\d\d)$/ 56 | "https://www.ietf.org/archive/id/draft-#$1.xml" 57 | # XXX find xml source for most recent version! 58 | else 59 | warn "*** Can't parse shortname #{xmlsource.inspect}" 60 | puts op 61 | exit 1 62 | end 63 | target = :url 64 | end 65 | 66 | begin 67 | xml = case target 68 | when :file 69 | ARGF.read 70 | when :url 71 | URI(xmlsource).open.read 72 | else 73 | fail 74 | end 75 | rescue Exception => e 76 | warn "#{xmlsource.inspect}: #{e}" 77 | exit 1 78 | end 79 | 80 | doc = REXML::Document.new(xml) 81 | 82 | REXML::XPath.match(doc.root, '//xi:include').each do |el| 83 | begin 84 | refdoc = REXML::Document.new(URI(el[:href]).open.read) 85 | el.replace_with(refdoc.root) 86 | rescue => e 87 | warn "*** error getting xinclude #{el} resolved: #{e}" 88 | end 89 | end 90 | 91 | def series_info_to_URI(si) 92 | case si[:name] 93 | when "RFC" 94 | "https://www.rfc-editor.org/rfc/rfc#{si[:value]}.xml" 95 | when "Internet-Draft" 96 | "https://www.ietf.org/archive/id/#{si[:value]}.xml" 97 | end 98 | end 99 | 100 | 101 | def series_info_to_name(si) 102 | case si[:name] 103 | when "RFC" 104 | "RFC#{si[:value]}" 105 | when "Internet-Draft" 106 | si[:value] 107 | end 108 | end 109 | 110 | def section_number_to_pn_candidates(s) 111 | if /^[0-9]/ =~ s 112 | ["section-#{s}"] 113 | elsif /[.]/ =~ s 114 | ["section-#{s.downcase}", "section-appendix.#{s.downcase}"] 115 | else 116 | ["section-appendix.#{s.downcase}"] 117 | end 118 | end 119 | 120 | def section_number_to_htmlid(s) 121 | if /^[0-9]/ =~ s 122 | "section-#{s}" 123 | else 124 | "appendix-#{s.upcase}" 125 | end 126 | end 127 | 128 | references = Hash[REXML::XPath.match(doc.root, "//reference").map {|r| 129 | si = REXML::XPath.match(r, "seriesInfo").map {|s| 130 | series_info_to_URI(s)}.compact.first 131 | sn = REXML::XPath.match(r, "seriesInfo").map {|s| 132 | series_info_to_name(s)}.compact.first 133 | [r[:anchor], si ? [si, sn] : nil] 134 | }] # XXX duplicates? 135 | 136 | heading1 = "# #{xmlsource}" 137 | title = REXML::XPath.first(doc.root, "//title") 138 | heading1 << "\n(#{title.all_text})" if title 139 | puts heading1 140 | 141 | per_reference = Hash.new { |h, k| h[k] = Set[]} 142 | 143 | REXML::XPath.each(doc.root, "//xref[@section]") do |x| 144 | trg = x[:target] 145 | if x[:relative] 146 | puts "\n## #{x[:target]}#{x[:relative]}: #{x[:section]}" 147 | else 148 | # p x 149 | per_reference[trg] << x[:section] 150 | end 151 | end 152 | 153 | def error_out(s) 154 | warn "" 155 | warn s 156 | warn "" 157 | $exit_code = 1 158 | end 159 | 160 | def num_expand(s) 161 | s.gsub(/\d+/) {|n| "%09d" % n.to_i} 162 | end 163 | 164 | def want_one(secs, what) 165 | case secs.size 166 | when 0 167 | error_out "*** cannot match #{what}" 168 | "*** DOESN'T EXIST ***" 169 | when 1 170 | yield secs.first 171 | else 172 | error_out "*** multiple matches for #{what}" 173 | "*** MULTIPLE MATCHES ***" 174 | end 175 | end 176 | require 'open3' 177 | 178 | module OpenURI 179 | class << self 180 | def processed(uri, old, camo, *rest) 181 | newuri = uri.to_s.sub(old, camo) # camo name for processed data 182 | response = Cache.get(newuri) || ( 183 | unprocessed = open_uri(uri, *rest).read 184 | fn = [OpenURI::Cache.cache_path, uri.sub(/.*\//, '')].join('/') 185 | File.open(fn, 'wb'){|f| f.write unprocessed } 186 | new_fn = yield newuri, fn 187 | Cache.set(newuri, File.open(new_fn)) 188 | ) 189 | response 190 | end 191 | def prepped(uri) 192 | processed(uri, /\.xml$/, ".prepped.xml") do |newuri, fn| 193 | _prep_out, s = Open3.capture2("xml2rfc", "--prep", fn) 194 | fail s.inspect unless s.success? 195 | fn.sub(/\.xml$/, ".prepped.xml") # xml2rfc creates new file 196 | end 197 | end 198 | def tidied(uri) 199 | processed(uri, /\.html$/, ".tidied.html") do |newuri, fn| 200 | _prep_out, s = Open3.capture2("tidy", "-mq", "-asxml", "-f", "/dev/null", fn) 201 | fail s.inspect unless s.exited? # can't check success 202 | fn # -m makes in-place change 203 | end 204 | end 205 | end 206 | end 207 | 208 | # go through section-referenced documents in sequence 209 | per_reference.keys.sort_by {|x| num_expand(x)}.each do |trg| 210 | uri, sname = references[trg] 211 | add = +'' 212 | if sname != trg 213 | add << " [#{sname}]" 214 | end 215 | begin 216 | ref = URI(uri).open.read 217 | refdoc = REXML::Document.new(ref) 218 | if REXML::XPath.match(refdoc.root, "/rfc/front/abstract[@pn]").size == 0 219 | ref = OpenURI.prepped(uri).read 220 | refdoc = REXML::Document.new(ref) 221 | add << " [+prep]" 222 | end 223 | add << " (#{REXML::XPath.match(refdoc.root, "//title").first.all_text})" 224 | rescue OpenURI::HTTPError => e 225 | begin 226 | jsonuri = uri.sub(/\.xml$/, ".json") 227 | refjson = URI(jsonuri).open.read 228 | refdata = JSON.load(refjson) 229 | add << " (#{refdata["title"].strip})" 230 | rescue OpenURI::HTTPError => e 231 | add << " [No XML or JSON]" 232 | rescue Exception => e 233 | warn "*** error getting #{jsonuri.inspect}: #{e}" 234 | end 235 | rescue Exception => e 236 | warn "*** error getting #{uri.inspect}: #{e}" 237 | end 238 | puts "\n## #{trg}#{add}" 239 | unless refdoc 240 | begin 241 | htmluri = uri.sub(/\.xml$/, ".html") 242 | refhtml = OpenURI.tidied(htmluri).read 243 | refhtmldoc = REXML::Document.new(refhtml) 244 | rescue Exception => e 245 | warn "*** error tidying up HTML for #{htmluri.inspect}: #{e}" 246 | end 247 | end 248 | # go through individual section references in sequence 249 | per_reference[trg].to_a.sort_by {|x| num_expand(x)}.each do |s| 250 | add = +'' 251 | if refdoc # find section name in XML from anchor s 252 | secpn = section_number_to_pn_candidates(s) 253 | secs = secpn.flat_map{ |c| 254 | REXML::XPath.match(refdoc.root, "//section[@pn=$pn]", 255 | {}, {"pn" => c})} 256 | what = "#{secpn.join(" or ")} in #{trg}" 257 | add << " (#{want_one(secs, what) do |sec| 258 | sec[:title] || sec.elements["name"].all_text 259 | end})" 260 | elsif refhtmldoc # find section name in HTML from anchor s 261 | secpn = section_number_to_htmlid(s) 262 | secs = REXML::XPath.match(refhtmldoc.root, 263 | "//xmlns:a[@id=$pn]/ancestor::xmlns:span", 264 | {"xmlns" => "http://www.w3.org/1999/xhtml"}, 265 | {"pn" => secpn}) 266 | what = "#{secpn} in #{trg}" 267 | add << " (#{want_one(secs, what) do |sec| 268 | sec.text.sub(/^\.\s+/, '') 269 | end})" 270 | end 271 | puts "* #{/^[0-9]/ =~ s ? "Section" : "Appendix"} #{s}#{add}" 272 | end 273 | end 274 | 275 | exit $exit_code 276 | -------------------------------------------------------------------------------- /bin/kramdown-rfc2629: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env ruby 2 | # -*- coding: utf-8 -*- 3 | require 'kramdown-rfc/command' 4 | 5 | -------------------------------------------------------------------------------- /data/encoding-fallbacks.txt: -------------------------------------------------------------------------------- 1 | 00a0 2 | 00a1 ! 3 | 00a2 [cents] 4 | 00a3 GBP 5 | 00a4 [currency units] 6 | 00a5 JPY 7 | 00a6 | 8 | 00a7 S. 9 | 00a9 (C) 10 | 00aa a 11 | 00ab << 12 | 00ac [not] 13 | 00ae (R) 14 | 00af _ 15 | 00b0 o 16 | 00b1 +/- 17 | 00b2 ^2 18 | 00b3 ^3 19 | 00b4 ' 20 | 00b5 [micro] 21 | 00b6 P. 22 | 00b7 . 23 | 00b8 , 24 | 00b9 ^1 25 | 00ba o 26 | 00bb >> 27 | 00bc 1/4 28 | 00bd 1/2 29 | 00be 3/4 30 | 00bf ? 31 | 00c0 A 32 | 00c1 A 33 | 00c2 A 34 | 00c3 A 35 | 00c4 Ae 36 | 00c5 Ae 37 | 00c6 AE 38 | 00c7 C 39 | 00c8 E 40 | 00c9 E 41 | 00ca E 42 | 00cb E 43 | 00cc I 44 | 00cd I 45 | 00ce I 46 | 00cf I 47 | 00d0 [ETH] 48 | 00d1 N 49 | 00d2 O 50 | 00d3 O 51 | 00d4 O 52 | 00d5 O 53 | 00d6 Oe 54 | 00d7 x 55 | 00d8 Oe 56 | 00d9 U 57 | 00da U 58 | 00db U 59 | 00dc Ue 60 | 00dd Y 61 | 00de [THORN] 62 | 00df ss 63 | 00e0 a 64 | 00e1 a 65 | 00e2 a 66 | 00e3 a 67 | 00e4 ae 68 | 00e5 ae 69 | 00e6 ae 70 | 00e7 c 71 | 00e8 e 72 | 00e9 e 73 | 00ea e 74 | 00eb e 75 | 00ec i 76 | 00ed i 77 | 00ee i 78 | 00ef i 79 | 00f0 [eth] 80 | 00f1 n 81 | 00f2 o 82 | 00f3 o 83 | 00f4 o 84 | 00f5 o 85 | 00f6 oe 86 | 00f7 / 87 | 00f8 oe 88 | 00f9 u 89 | 00fa u 90 | 00fb u 91 | 00fc ue 92 | 00fd y 93 | 00fe [thorn] 94 | 00ff y 95 | 0152 OE 96 | 0153 oe 97 | 0161 s 98 | 0178 Y 99 | 0192 f 100 | 02dc ~ 101 | 2002 102 | 2003 103 | 2009 104 | 2013 -- 105 | 2014 --- 106 | 2018 ' 107 | 2019 ' 108 | 201a ' 109 | 201c " 110 | 201d " 111 | 201e " 112 | 2020 *!* 113 | 2021 *!!* 114 | 2022 o 115 | 2026 ... 116 | 2030 [/1000] 117 | 2032 ' 118 | 2039 < 119 | 203a > 120 | 2044 / 121 | 20ac EUR 122 | 2122 [TM] 123 | 2190 <-- 124 | 2192 --> 125 | 2194 <-> 126 | 21d0 <== 127 | 21d2 ==> 128 | 21d4 <=> 129 | 2212 - 130 | 2217 * 131 | 2264 <= 132 | 2265 >= 133 | 2329 < 134 | 232a > 135 | 0021 ! 136 | 0023 # 137 | 0024 $ 138 | 0025 % 139 | 0028 ( 140 | 0029 ) 141 | 002a * 142 | 002b + 143 | 002c , 144 | 002d - 145 | 002e . 146 | 002f / 147 | 003a : 148 | 003b ; 149 | 003d = 150 | 003f ? 151 | 0040 @ 152 | 005b [ 153 | 005d ] 154 | 005e ^ 155 | 005f _ 156 | 0060 ` 157 | 007b { 158 | 007c | 159 | 007d } 160 | 017d Z 161 | 017e z 162 | 2010 - 163 | -------------------------------------------------------------------------------- /data/kramdown-rfc2629.erb: -------------------------------------------------------------------------------- 1 | "?> 2 | 3 | 6 | 7 | <% if $options.v3 %> 8 | [ 9 | 10 | 11 | 12 | 13 | <% else %> 14 | 16 | <% ps.arr("bibxml") do |tag, sys| -%> 17 | SYSTEM "<%= sys %>"> 18 | <% end -%> 19 | <% ps.arr("entity", false) do |en, ev| -%> 20 | "<%=ev%>"> 21 | <% end -%> 22 | ]> 23 | 24 | <% 25 | ps.rest.fetch("consensus") do 26 | # consensus not given -- default intelligently 27 | cat = ps.has("category") || ps.has("cat") 28 | MUST_CONSENSUS = {"std" => true, "bcp" => true} 29 | ps["consensus"] = true if MUST_CONSENSUS[cat] 30 | end 31 | rfcattrs = ps.attrs("ipr", "docName=docname", "category=cat", 32 | "consensus", "submissionType=submissiontype=stream", "xml:lang=lang", 33 | "number", "obsoletes", "updates", "seriesNo=seriesno") 34 | TRUE_FALSE = {nil => "true", false => "false", true => "true", 35 | "yes" => "true", "no" => "false"} 36 | YES_NO = {"true" => "yes", "false" => "no"} 37 | pis = KramdownRFC::ParameterSet.new({}) 38 | ps.arr("pi", false) do |pi, val| 39 | pis[pi] = TRUE_FALSE[val] || val 40 | end 41 | if $options.v3 42 | piattrs = pis.attrs("tocDepth=tocdepth", "tocInclude=toc", 43 | "sortRefs=sortrefs", "symRefs=symrefs", "indexInclude=index") 44 | if piattrs != "" 45 | rfcattrs << " " << piattrs 46 | end 47 | end 48 | pis.rest.each do |pi, val| 49 | v = YES_NO[val] || pis.escattr(val) 50 | -%> 51 | ="<%=v%>"?> 52 | <% end -%> 53 | 54 | > 55 | <% ps.arr("v3xml2rfc", false) do |pi, val| 56 | Array(val).each do |el| 57 | -%> 58 | ="<%=ps.escattr(el)%>"?> 59 | <% end 60 | end -%> 61 | 62 | <%= ps.ele("title", ps.attr("abbrev=titleabbrev")) %> 63 | 64 | <% ps.arr("author") do |au| 65 | aups = KramdownRFC::authorps_from_hash(au) 66 | -%> 67 | <%= KramdownRFC::person_element_from_aups("author", aups) -%> 68 | <% aups.warn_if_leftovers -%> 69 | <% end -%> 70 | 71 | /> 72 | 73 | <%= ps.ele("area") %> 74 | <%= ps.ele("workgroup=wg") %> 75 | <%= ps.ele("keyword=kw") %> 76 | 77 | 78 | 79 | <%= sechash.delete("abstract") %> 80 | 81 | 82 | 83 | <% if $options.v3 -%> 84 | <% venue = ps[:venue] -%> 85 | <% if venue -%> 86 | <% venue = KramdownRFC::ParameterSet.new(venue) -%> 87 | 88 | <% if (dn = ps.av[:docName]) && 89 | (dt = dn.rpartition('-')[0]) != "" 90 | -%> 91 | 92 | <% if latest = venue[:latest] -%> 93 | The latest revision of this draft can be found at . 94 | <% end -%> 95 | Status information for this document may be found at . 96 | 97 | <% end -%> 98 | <% mail = venue[:mail] -%> 99 | <% homepage = venue[:home] -%> 100 | <% gtype = venue[:type] -%> 101 | <% if mail || homepage -%> 102 | 103 | <% end -%> 104 | <% if mail 105 | mail_local, mail_host = mail.split("@", 2) 106 | end 107 | if mail_host -%> 108 | <% default_links = { 109 | "iab.org" => true, 110 | "ietf.org" => true, 111 | "irtf.org" => true, 112 | }[mail_host] 113 | mail_subdomain, mail_domain = mail_host.split(".", 2) 114 | group = venue[:group] || mail_local # XXX 115 | arch = venue[:arch] || default_links && "https://mailarchive.ietf.org/arch/browse/#{mail_local}/" 116 | subscribe = venue[:subscribe] || default_links && "https://www.ietf.org/mailman/listinfo/#{mail_local}/" 117 | GROUPS = {"ietf" => "Working ", "irtf" => "Research "} 118 | gtype ||= "#{GROUPS[mail_subdomain]}Group" -%> 119 | Discussion of this document takes place on the 120 | <%=group%> <%=gtype%> mailing list ()<% if arch -%>, 121 | which is archived at <% end -%>. 122 | <% if subscribe -%> 123 | Subscribe at . 124 | <% end -%> 125 | <% end -%> 126 | <% if homepage -%> 127 | <%=gtype%> information can be found at . 128 | <% end -%> 129 | <% if mail || homepage -%> 130 | 131 | <% end -%> 132 | <% if repo = venue[:repo] || ((gh = venue[:github]) && "https://github.com/#{gh}") -%> 133 | Source for this draft and an issue tracker can be found at 134 | . 135 | <% end -%> 136 | <%= venue.ele("t=text", nil, nil, true) -%> 137 | 138 | <% venue.warn_if_leftovers -%> 139 | <% end -%> 140 | <% end -%> 141 | 142 | <% sechash.keys.each do |k| -%> 143 | <% if k =~ /\A(to_be_removed_)?note_(.*)/ -%> 144 | 145 | <% option = "" 146 | text = "" 147 | if $1 148 | if $options.v3 149 | option = " removeInRFC=\"true\"" 150 | else 151 | text = " [This note is to be removed before publishing as an RFC.]\n" 152 | end 153 | end 154 | -%> 155 | "<%= option %>> 156 | <%= text -%> 157 | 158 | <%= sechash.delete(k) -%> 159 | 160 | 161 | 162 | <% end -%> 163 | <% end -%> 164 | 165 | 166 | 167 | 168 | 169 | {:/nomarkdown} 170 | {:quote: gi="blockquote"} 171 | {:aside: gi="aside"} 172 | {:cref: gi="cref"} 173 | {:markers: sourcecode-markers="true"} 174 | {:unnumbered: numbered="false"} 175 | {:vspace: vspace="0"} 176 | {:removeinrfc: removeinrfc="true"} 177 | {:notoc: toc="exclude"} 178 | {:compact: spacing="compact"} 179 | {:noabbrev: noabbrev="true"} 180 | {::nomarkdown} 181 | 182 | <%= sechash.delete("middle") %> 183 | 184 | 185 | 186 | 187 | 188 | <% displayref.each do |k, v| -%> 189 | 190 | <% end -%> 191 | 192 | <% shn = sechash.delete("normative") 193 | shi = sechash.delete("informative") 194 | shc = shn && shi 195 | if shc 196 | -%> 197 | 198 | <% end -%> 199 | 200 | <% if shn -%> 201 | 202 | 203 | <%= shn %> 204 | 205 | 206 | <% end -%> 207 | 208 | <% if shi -%> 209 | 210 | 211 | <%= shi %> 212 | 213 | 214 | <% end -%> 215 | 216 | <% if shc -%> 217 | 218 | <% end -%> 219 | 220 | <%= sechash.delete("back") %> 221 | <% sh = sechash.delete("contributor") -%> 222 | <% consec = ps.has("contributor") -%> 223 | <% if sh || consec -%> 224 | <% if $options.v3 -%> 225 |
226 | Contributors 227 | <% else -%> 228 |
229 | <% warn "*** To use YAML contributors, use --v3 (kdrfc -3)" if consec -%> 230 | <% end -%> 231 | <%= sh -%> 232 | <% if $options.v3 && consec 233 | ps.arr("contributor") do |au| 234 | if Hash === au 235 | aups = KramdownRFC::authorps_from_hash(au) 236 | -%> 237 | <%= KramdownRFC::person_element_from_aups("contact", aups) -%> 238 | <%= if contrib = aups["contribution"] 239 | < 246 | <% aups.warn_if_leftovers -%> 247 | <% else -%> 248 | <%= < 256 | <% end -%> 257 | <% end -%> 258 | <% end -%> 259 |
260 | <% end -%> 261 | 262 | 263 | 264 | <% if $source -%> 265 | 268 | <% end -%> 269 | 270 | 271 | -------------------------------------------------------------------------------- /data/math.json: -------------------------------------------------------------------------------- 1 | {"replacements":[["\\textfractionsolidus","⁄"],["\\leftrightsquigarrow","↭"],["\\textpertenthousand","‱"],["\\blacktriangleright","▸"],["\\blacktriangledown","▾"],["\\blacktriangleleft","◂"],["\\twoheadrightarrow","↠"],["\\leftrightharpoons","⇋"],["\\rightleftharpoons","⇌"],["\\textreferencemark","※"],["\\circlearrowright","↻"],["\\rightrightarrows","⇉"],["\\vartriangleright","⊳"],["\\textordmasculine","º"],["\\textvisiblespace","␣"],["\\twoheadleftarrow","↞"],["\\downharpoonright","⇂"],["\\ntrianglerighteq","⋭"],["\\rightharpoondown","⇁"],["\\textperthousand","‰"],["\\leftrightarrows","⇆"],["\\textmusicalnote","♪"],["\\nleftrightarrow","↮"],["\\rightleftarrows","⇄"],["\\bigtriangledown","▽"],["\\textordfeminine","ª"],["\\ntrianglelefteq","⋬"],["\\rightthreetimes","⋌"],["\\trianglerighteq","⊵"],["\\vartriangleleft","⊲"],["\\rightsquigarrow","⇝"],["\\downharpoonleft","⇃"],["\\curvearrowright","↷"],["\\circlearrowleft","↺"],["\\leftharpoondown","↽"],["\\nLeftrightarrow","⇎"],["\\curvearrowleft","↶"],["\\guilsinglright","›"],["\\leftthreetimes","⋋"],["\\leftrightarrow","↔"],["\\rightharpoonup","⇀"],["\\guillemotright","»"],["\\downdownarrows","⇊"],["\\hookrightarrow","↪"],["\\hspace{0.25em}"," "],["\\dashrightarrow","⇢"],["\\leftleftarrows","⇇"],["\\trianglelefteq","⊴"],["\\ntriangleright","⋫"],["\\doublebarwedge","⌆"],["\\upharpoonright","↾"],["\\rightarrowtail","↣"],["\\looparrowright","↬"],["\\Leftrightarrow","⇔"],["\\sphericalangle","∢"],["\\divideontimes","⋇"],["\\measuredangle","∡"],["\\blacktriangle","▴"],["\\ntriangleleft","⋪"],["\\mathchar\"1356","⁁"],["\\texttrademark","™"],["\\mathchar\"2208","⌖"],["\\triangleright","▹"],["\\leftarrowtail","↢"],["\\guilsinglleft","‹"],["\\upharpoonleft","↿"],["\\mathbb{gamma}","ℽ"],["\\fallingdotseq","≒"],["\\looparrowleft","↫"],["\\textbrokenbar","¦"],["\\hookleftarrow","↩"],["\\smallsetminus","﹨"],["\\dashleftarrow","⇠"],["\\guillemotleft","«"],["\\leftharpoonup","↼"],["\\mathbb{Gamma}","ℾ"],["\\bigtriangleup","△"],["\\textcircledP","℗"],["\\risingdotseq","≓"],["\\triangleleft","◃"],["\\mathsterling","£"],["\\textcurrency","¤"],["\\triangledown","▿"],["\\blacklozenge",""],["\\sfrac{5}{6}","⅚"],["\\preccurlyeq","≼"],["\\Rrightarrow","⇛"],["\\circledcirc","⊚"],["\\nRightarrow","⇏"],["\\sfrac{3}{8}","⅜"],["\\sfrac{1}{3}","⅓"],["\\sfrac{2}{5}","⅖"],["\\vartriangle","▵"],["\\Updownarrow","⇕"],["\\nrightarrow","↛"],["\\sfrac{1}{2}","½"],["\\sfrac{3}{5}","⅗"],["\\succcurlyeq","≽"],["\\sfrac{4}{5}","⅘"],["\\diamondsuit","♦"],["\\hphantom{0}"," "],["\\sfrac{1}{6}","⅙"],["\\curlyeqsucc","⋟"],["\\blacksquare","▪"],["\\hphantom{,}"," "],["\\curlyeqprec","⋞"],["\\sfrac{1}{8}","⅛"],["\\sfrac{7}{8}","⅞"],["\\sfrac{1}{5}","⅕"],["\\sfrac{2}{3}","⅔"],["\\updownarrow","↕"],["\\backepsilon","∍"],["\\circleddash","⊝"],["\\eqslantless","⋜"],["\\sfrac{3}{4}","¾"],["\\sfrac{5}{8}","⅝"],["\\hspace{1pt}"," "],["\\sfrac{1}{4}","¼"],["\\mathbb{Pi}","ℿ"],["\\mathcal{M}","ℳ"],["\\mathcal{o}","ȓ4"],["\\mathcal{O}","ᵊA"],["\\nsupseteqq","⊉"],["\\mathcal{B}","ℬ"],["\\textrecipe","℞"],["\\nsubseteqq","⊈"],["\\subsetneqq","⊊"],["\\mathcal{I}","ℑ"],["\\upuparrows","⇈"],["\\mathcal{e}","ℯ"],["\\mathcal{L}","ℒ"],["\\nleftarrow","↚"],["\\mathcal{H}","ℋ"],["\\mathcal{E}","ℰ"],["\\eqslantgtr","⋝"],["\\curlywedge","⋏"],["\\varepsilon","ε"],["\\supsetneqq","⊋"],["\\rightarrow","→"],["\\mathcal{R}","ℛ"],["\\sqsubseteq","⊑"],["\\mathcal{g}","ℊ"],["\\sqsupseteq","⊒"],["\\complement","∁"],["\\Rightarrow","⇒"],["\\gtreqqless","⋛"],["\\lesseqqgtr","⋚"],["\\circledast","⊛"],["\\nLeftarrow","⇍"],["\\Lleftarrow","⇚"],["\\varnothing","∅"],["\\mathcal{N}","𝒩"],["\\Leftarrow","⇐"],["\\gvertneqq","≩"],["\\mathbb{C}","ℂ"],["\\supsetneq","⊋"],["\\leftarrow","←"],["\\nleqslant","≰"],["\\mathbb{Q}","ℚ"],["\\mathbb{Z}","ℤ"],["\\llbracket","〚"],["\\mathbb{H}","ℍ"],["\\spadesuit","♠"],["\\mathit{o}","ℴ"],["\\mathbb{P}","ℙ"],["\\rrbracket","〛"],["\\supseteqq","⊇"],["\\copyright","©"],["\\textsc{k}","ĸ"],["\\gtreqless","⋛"],["\\mathbb{j}","ⅉ"],["\\pitchfork","⋔"],["\\estimated","℮"],["\\ngeqslant","≱"],["\\mathbb{e}","ⅇ"],["\\therefore","∴"],["\\triangleq","≜"],["\\varpropto","∝"],["\\subsetneq","⊊"],["\\heartsuit","♥"],["\\mathbb{d}","ⅆ"],["\\lvertneqq","≨"],["\\checkmark","✓"],["\\nparallel","∦"],["\\mathbb{R}","ℝ"],["\\lesseqgtr","⋚"],["\\downarrow","↓"],["\\mathbb{D}","ⅅ"],["\\mathbb{i}","ⅈ"],["\\backsimeq","⋍"],["\\mathbb{N}","ℕ"],["\\Downarrow","⇓"],["\\subseteqq","⊆"],["\\setminus","∖"],["\\succnsim","⋩"],["\\doteqdot","≑"],["\\clubsuit","♣"],["\\emptyset","∅"],["\\sqsupset","⊐"],["\\fbox{~~}","▭"],["\\curlyvee","⋎"],["\\varkappa","ϰ"],["\\llcorner","⌞"],["\\varsigma","ς"],["\\approxeq","≊"],["\\backcong","≌"],["\\supseteq","⊇"],["\\circledS","Ⓢ"],["\\circledR","®"],["\\textcent","¢"],["\\urcorner","⌝"],["\\lrcorner","⌟"],["\\boxminus","⊟"],["\\texteuro","€"],["\\vartheta","ϑ"],["\\barwedge","⊼"],["\\ding{86}","✶"],["\\sqsubset","⊏"],["\\subseteq","⊆"],["\\intercal","⊺"],["\\ding{73}","☆"],["\\ulcorner","⌜"],["\\recorder","⌕"],["\\precnsim","⋨"],["\\parallel","∥"],["\\boxtimes","⊠"],["\\ding{55}","✗"],["\\multimap","⊸"],["\\maltese","✠"],["\\nearrow","↗"],["\\swarrow","↙"],["\\lozenge","◊"],["\\sqrt[3]","∛"],["\\succsim","≿"],["\\dotplus","∔"],["\\tilde{}","~"],["\\check{}","ˇ"],["\\lessgtr","≶"],["\\Upsilon","ϒ"],["\\Cdprime","Ъ"],["\\gtrless","≷"],["\\backsim","∽"],["\\nexists","∄"],["\\dotplus","∔"],["\\searrow","↘"],["\\lessdot","⋖"],["\\boxplus","⊞"],["\\upsilon","υ"],["\\epsilon","ε"],["\\diamond","⋄"],["\\bigstar","★"],["\\ddagger","‡"],["\\cdprime","ъ"],["\\Uparrow","⇑"],["\\sqrt[4]","∜"],["\\between","≬"],["\\sqangle","∟"],["\\digamma","Ϝ"],["\\uparrow","↑"],["\\nwarrow","↖"],["\\precsim","≾"],["\\breve{}","˘"],["\\because","∵"],["\\bigcirc","◯"],["\\acute{}","´"],["\\grave{}","`"],["\\check{}","ˇ"],["\\lesssim","≲"],["\\partial","∂"],["\\natural","♮"],["\\supset","⊃"],["\\hstrok","ħ"],["\\Tstrok","Ŧ"],["\\coprod","∐"],["\\models","⊧"],["\\otimes","⊗"],["\\degree","°"],["\\gtrdot","⋗"],["\\preceq","≼"],["\\Lambda","Λ"],["\\lambda","λ"],["\\cprime","ь"],["\\varrho","ϱ"],["\\Bumpeq","≎"],["\\hybull","⁃"],["\\lmidot","ŀ"],["\\nvdash","⊬"],["\\lbrace","{"],["\\bullet","•"],["\\varphi","φ"],["\\bumpeq","≏"],["\\ddot{}","¨"],["\\Lmidot","Ŀ"],["\\Cprime","Ь"],["\\female","♀"],["\\rtimes","⋊"],["\\gtrsim","≳"],["\\mapsto","↦"],["\\daleth","ℸ"],["\\square","■"],["\\nVDash","⊯"],["\\rangle","〉"],["\\tstrok","ŧ"],["\\oslash","⊘"],["\\ltimes","⋉"],["\\lfloor","⌊"],["\\marker","▮"],["\\Subset","⋐"],["\\Vvdash","⊪"],["\\propto","∝"],["\\Hstrok","Ħ"],["\\dlcrop","⌍"],["\\forall","∀"],["\\nVdash","⊮"],["\\Supset","⋑"],["\\langle","〈"],["\\ominus","⊖"],["\\rfloor","⌋"],["\\circeq","≗"],["\\eqcirc","≖"],["\\drcrop","⌌"],["\\veebar","⊻"],["\\ulcrop","⌏"],["\\nvDash","⊭"],["\\urcrop","⌎"],["\\exists","∃"],["\\approx","≈"],["\\dagger","†"],["\\boxdot","⊡"],["\\succeq","≽"],["\\bowtie","⋈"],["\\subset","⊂"],["\\Sigma","Σ"],["\\Omega","Ω"],["\\nabla","∇"],["\\colon",":"],["\\boxHu","╧"],["\\boxHd","╤"],["\\aleph","ℵ"],["\\gnsim","⋧"],["\\boxHU","╩"],["\\boxHD","╦"],["\\equiv","≡"],["\\lneqq","≨"],["\\alpha","α"],["\\amalg","∐"],["\\boxhU","╨"],["\\boxhD","╥"],["\\uplus","⊎"],["\\boxhu","┴"],["\\kappa","κ"],["\\sigma","σ"],["\\boxDL","╗"],["\\Theta","Θ"],["\\Vdash","⊩"],["\\boxDR","╔"],["\\boxDl","╖"],["\\sqcap","⊓"],["\\boxDr","╓"],["\\bar{}","¯"],["\\dashv","⊣"],["\\vDash","⊨"],["\\boxdl","┐"],["\\boxVl","╢"],["\\boxVh","╫"],["\\boxVr","╟"],["\\boxdr","┌"],["\\boxdL","╕"],["\\boxVL","╣"],["\\boxVH","╬"],["\\boxVR","╠"],["\\boxdR","╒"],["\\theta","θ"],["\\lhblk","▄"],["\\uhblk","▀"],["\\ldotp","."],["\\ldots","…"],["\\boxvL","╡"],["\\boxvH","╪"],["\\boxvR","╞"],["\\boxvl","┤"],["\\boxvh","┼"],["\\boxvr","├"],["\\Delta","Δ"],["\\boxUR","╚"],["\\boxUL","╝"],["\\oplus","⊕"],["\\boxUr","╙"],["\\boxUl","╜"],["\\doteq","≐"],["\\happy","㋡"],["\\varpi","ϖ"],["\\smile","☺"],["\\boxul","┘"],["\\simeq","≃"],["\\boxuR","╘"],["\\boxuL","╛"],["\\boxhd","┬"],["\\gimel","ℷ"],["\\Gamma","Γ"],["\\lnsim","⋦"],["\\sqcup","⊔"],["\\omega","ω"],["\\sharp","♯"],["\\times","×"],["\\block","█"],["\\hat{}","^"],["\\wedge","∧"],["\\vdash","⊢"],["\\angle","∠"],["\\infty","∞"],["\\gamma","γ"],["\\asymp","≍"],["\\rceil","⌉"],["\\dot{}","˙"],["\\lceil","⌈"],["\\delta","δ"],["\\gneqq","≩"],["\\frown","⌢"],["\\phone","☎"],["\\vdots","⋮"],["\\boxr","└"],["\\k{i}","į"],["\\`{I}","Ì"],["\\perp","⊥"],["\\\"{o}","ö"],["\\={I}","Ī"],["\\`{a}","à"],["\\v{T}","Ť"],["\\surd","√"],["\\H{O}","Ő"],["\\vert","|"],["\\k{I}","Į"],["\\\"{y}","ÿ"],["\\\"{O}","Ö"],["\\'{Y}","Ý"],["\\u{u}","ў"],["\\u{G}","Ğ"],["\\.{E}","Ė"],["\\.{z}","ż"],["\\v{t}","ť"],["\\prec","≺"],["\\H{o}","ő"],["\\mldr","…"],["\\'{y}","ý"],["\\cong","≅"],["\\.{e}","ė"],["\\'{L}","Ĺ"],["\\star","*"],["\\.{Z}","Ż"],["\\'{e}","é"],["\\geqq","≧"],["\\cdot","⋅"],["\\`{U}","Ù"],["\\'{l}","ĺ"],["\\v{L}","Ľ"],["\\c{s}","ş"],["\\'{s}","ś"],["\\~{A}","Ã"],["\\Vert","‖"],["\\k{e}","ę"],["\\lnot","¬"],["\\'{z}","ź"],["\\leqq","≦"],["\\beta","β"],["\\beth","ℶ"],["\\'{E}","É"],["\\~{n}","ñ"],["\\u{i}","й"],["\\c{S}","Ş"],["\\c{N}","Ņ"],["\\H{u}","ű"],["\\v{n}","ň"],["\\'{S}","Ś"],["\\={U}","Ū"],["\\~{O}","Õ"],["\\'{Z}","Ź"],["\\v{E}","Ě"],["\\'{R}","Ŕ"],["\\H{U}","Ű"],["\\v{N}","Ň"],["\\prod","∏"],["\\v{s}","š"],["\\\"{U}","Ü"],["\\c{n}","ņ"],["\\k{U}","Ų"],["\\c{R}","Ŗ"],["\\'{A}","Á"],["\\~{o}","õ"],["\\v{e}","ě"],["\\v{S}","Š"],["\\u{A}","Ă"],["\\circ","∘"],["\\\"{u}","ü"],["\\flat","♭"],["\\v{z}","ž"],["\\r{U}","Ů"],["\\`{O}","Ò"],["\\={u}","ū"],["\\oint","∮"],["\\c{K}","Ķ"],["\\k{u}","ų"],["\\not<","≮"],["\\not>","≯"],["\\`{o}","ò"],["\\\"{I}","Ï"],["\\v{D}","Ď"],["\\.{G}","Ġ"],["\\r{u}","ů"],["\\not=","≠"],["\\`{u}","ù"],["\\v{c}","č"],["\\c{k}","ķ"],["\\.{g}","ġ"],["\\'{N}","Ń"],["\\odot","⊙"],["\\`{e}","э"],["\\c{T}","Ţ"],["\\v{d}","ď"],["\\\"{e}","ё"],["\\'{I}","Í"],["\\v{R}","Ř"],["\\k{a}","ą"],["\\nldr","‥"],["\\`{A}","À"],["\\'{n}","ń"],["\\~{N}","Ñ"],["\\nmid","∤"],["\\.{C}","Ċ"],["\\zeta","ζ"],["\\~{u}","ũ"],["\\`{E}","Э"],["\\~{a}","ã"],["\\c{t}","ţ"],["\\={o}","ō"],["\\v{r}","ř"],["\\={A}","Ā"],["\\.{c}","ċ"],["\\~{U}","Ũ"],["\\k{A}","Ą"],["\\\"{a}","ä"],["\\u{U}","Ў"],["\\iota","ι"],["\\={O}","Ō"],["\\c{C}","Ç"],["\\gneq","≩"],["\\'{c}","ć"],["\\boxH","═"],["\\hbar","ℏ"],["\\\"{A}","Ä"],["\\boxv","│"],["\\boxh","─"],["\\male","♂"],["\\'{u}","ú"],["\\sqrt","√"],["\\succ","≻"],["\\c{c}","ç"],["\\'{C}","Ć"],["\\v{l}","ľ"],["\\u{a}","ă"],["\\v{Z}","Ž"],["\\'{o}","ó"],["\\c{G}","Ģ"],["\\v{C}","Č"],["\\lneq","≨"],["\\\"{E}","Ё"],["\\={a}","ā"],["\\c{l}","ļ"],["\\'{a}","á"],["\\={E}","Ē"],["\\boxV","║"],["\\u{g}","ğ"],["\\'{O}","Ó"],["\\'{g}","ǵ"],["\\u{I}","Й"],["\\c{L}","Ļ"],["\\k{E}","Ę"],["\\.{I}","İ"],["\\~{I}","Ĩ"],["\\quad"," "],["\\c{r}","ŗ"],["\\'{r}","ŕ"],["\\\"{Y}","Ÿ"],["\\={e}","ē"],["\\'{U}","Ú"],["\\leq","≤"],["\\Cup","⋓"],["\\Psi","Ψ"],["\\neq","≠"],["\\k{}","˛"],["\\={}","‾"],["\\H{}","˝"],["\\cup","∪"],["\\geq","≥"],["\\mho","℧"],["\\Dzh","Џ"],["\\cap","∩"],["\\bot","⊥"],["\\psi","ψ"],["\\chi","χ"],["\\c{}","¸"],["\\Phi","Φ"],["\\ast","*"],["\\ell","ℓ"],["\\top","⊤"],["\\lll","⋘"],["\\tau","τ"],["\\Cap","⋒"],["\\sad","☹"],["\\iff","⇔"],["\\eta","η"],["\\eth","ð"],["\\d{}","̣"],["\\rho","ρ"],["\\dzh","џ"],["\\div","÷"],["\\phi","ϕ"],["\\Rsh","↱"],["\\vee","∨"],["\\b{}","ˍ"],["\\t{}","͡"],["\\int","∫"],["\\sim","∼"],["\\r{}","˚"],["\\Lsh","↰"],["\\yen","¥"],["\\ggg","⋙"],["\\mid","∣"],["\\sum","∑"],["\\Dz","Ѕ"],["\\Re","ℜ"],["\\oe","œ"],["\\DH","Ð"],["\\ll","≪"],["\\ng","ŋ"],["\\'G","Ѓ"],["\\wr","≀"],["\\wp","℘"],["\\=I","І"],["\\:)","☺"],["\\:(","☹"],["\\AE","Æ"],["\\AA","Å"],["\\ss","ß"],["\\dz","ѕ"],["\\ae","æ"],["\\aa","å"],["\\th","þ"],["\\to","→"],["\\Pi","Π"],["\\mp","∓"],["\\Im","ℑ"],["\\pm","±"],["\\pi","π"],["\\\"I","Ї"],["\\'C","Ћ"],["\\in","∈"],["\\'K","Ќ"],["\\'k","ќ"],["\\'c","ћ"],["\\'g","ѓ"],["\\ni","∋"],["\\ne","≠"],["\\TH","Þ"],["\\Xi","Ξ"],["\\nu","ν"],["\\NG","Ŋ"],["\\:G","㋡"],["\\xi","ξ"],["\\OE","Œ"],["\\gg","≫"],["\\DJ","Đ"],["\\=e","є"],["\\=E","Є"],["\\mu","μ"],["\\dj","đ"],["\\:"," "],["\\;"," "],["\\&","&"],["\\$","$"],["\\%","%"],["\\#","#"],["\\,"," "],["\\-","­"],["\\S","§"],["\\P","¶"],["\\O","Ø"],["\\L","Ł"],["\\}","}"],["\\o","ø"],["\\l","ł"],["\\h","ℎ"],["\\i","ℹ"],["-","−"]],"combiningmarks":[["\\doubleunderline","̳"],["\\strikethrough","̵"],["\\underline","̲"],["\\overline","̅"],["\\tilde","̃"],["\\grave","̀"],["\\acute","́"],["\\slash","̸"],["\\breve","̆"],["\\ddot","̈"],["\\dot","̇"],["\\bar","̅"],["\\vec","⃗"],["\\hat","̂"]],"subsuperscripts":[["_x","ₓ"],["_v","ᵥ"],["_u","ᵤ"],["_t","ₜ"],["_s","ₛ"],["_r","ᵣ"],["_p","ₚ"],["_o","ₒ"],["_n","ₙ"],["_m","ₘ"],["_l","ₗ"],["_k","ₖ"],["_j","ⱼ"],["_i","ᵢ"],["_h","ₕ"],["_e","ₑ"],["_a","ₐ"],["^∫","ᶴ"],["_>","˲"],["_=","₌"],["_<","˱"],["_9","₉"],["_8","₈"],["_7","₇"],["_6","₆"],["_5","₅"],["_4","₄"],["_3","₃"],["_2","₂"],["_1","₁"],["_0","₀"],["_-","₋"],["_−","₋"],["_+","₊"],["_)","₎"],["_(","₍"],["_ρ","ᵨ"],["_χ","ᵪ"],["_φ","ᵩ"],["_β","ᵦ"],["_γ","ᵧ"],["^φ","ᵠ"],["^χ","ᵡ"],["^δ","ᵟ"],["^γ","ᵞ"],["^β","ᵝ"],["^8","⁸"],["^9","⁹"],["^<","˂"],["^=","⁼"],["^>","˃"],["^0","⁰"],["^1","¹"],["^2","²"],["^3","³"],["^4","⁴"],["^5","⁵"],["^6","⁶"],["^7","⁷"],["^(","⁽"],["^)","⁾"],["^*","*"],["^+","⁺"],["^-","⁻"],["^−","⁻"],["^P","ᴾ"],["^R","ᴿ"],["^T","ᵀ"],["^U","ᵁ"],["^V","ᄑ"],["^W","ᵂ"],["^H","ᴴ"],["^I","ᴵ"],["^J","ᴶ"],["^K","ᴷ"],["^L","ᴸ"],["^M","ᴹ"],["^N","ᴺ"],["^O","ᴼ"],["^A","ᴬ"],["^B","ᴮ"],["^D","ᴰ"],["^E","ᴱ"],["^G","ᴳ"],["^x","ˣ"],["^y","ʸ"],["^z","ᶻ"],["^p","ᵖ"],["^r","ʳ"],["^s","ˢ"],["^t","ᵗ"],["^u","ᵘ"],["^v","ᵛ"],["^w","ʷ"],["^h","ʰ"],["^i","ⁱ"],["^j","ʲ"],["^k","ᵏ"],["^l","ˡ"],["^m","ᵐ"],["^n","ⁿ"],["^o","ᵒ"],["^a","ᵃ"],["^b","ᵇ"],["^c","ᶜ"],["^d","ᵈ"],["^e","ᵉ"],["^f","ᶠ"],["^g","ᵍ"]]} 2 | -------------------------------------------------------------------------------- /data/studly.rb: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env ruby 2 | 3 | # xml2rfc/trunk/cli/xml2rfc/data/v3.rnc 4 | 5 | v3 = File.read("v3.rnc") 6 | studly = {} 7 | sc = v3.scan(/attribute [A-Za-z]*[A-Z][A-Za-z]*/).each do |s| 8 | _, an = s.split(" ") 9 | studly[an] = true 10 | end 11 | 12 | puts studly.keys.sort 13 | -------------------------------------------------------------------------------- /examples/Makefile: -------------------------------------------------------------------------------- 1 | OPEN=$(word 1, $(wildcard /usr/bin/xdg-open /usr/bin/open /bin/echo)) 2 | SOURCES?=${wildcard *.mkd} 3 | TEXT=${SOURCES:.mkd=.txt} 4 | HTML=${SOURCES:.mkd=.html} 5 | 6 | text: $(TEXT) 7 | html: $(HTML) 8 | 9 | %.xml: %.mkd 10 | kramdown-rfc2629 $< >$@.new 11 | mv $@.new $@ 12 | 13 | %.html: %.xml 14 | xml2rfc --html $< 15 | $(OPEN) $@ 16 | 17 | %.txt: %.xml 18 | xml2rfc $< 19 | -------------------------------------------------------------------------------- /examples/draft-rfcxml-general-template-bare-00.xml-edited.md: -------------------------------------------------------------------------------- 1 | --- 2 | stand_alone: true 3 | ipr: trust200902 4 | submissiontype: IETF 5 | 6 | cat: info 7 | title: '' 8 | author: 9 | - name: John (Insert-Author-Name-here;-bug-707) 10 | org: (Insert Author affiliation here) 11 | 12 | --- abstract 13 | 14 | \[REPLACE] 15 | 16 | --- middle 17 | 18 | # \[REPLACE] 19 | 20 | --- back 21 | -------------------------------------------------------------------------------- /examples/draft-rfcxml-general-template-standard-00.xml-edited.md: -------------------------------------------------------------------------------- 1 | --- 2 | stand_alone: true 3 | ipr: trust200902 4 | cat: info # Check 5 | submissiontype: IETF 6 | area: General [REPLACE] 7 | wg: Internet Engineering Task Force 8 | 9 | docname: draft-rfcxml-general-template-standard-00 10 | obsoletes: 4711, 4712 # Remove if not needed/Replace 11 | updates: 4710 # Remove if not needed/Replace 12 | 13 | title: Title [REPLACE] 14 | abbrev: Abbreviated Title [REPLACE] 15 | lang: en 16 | kw: 17 | - keyword1 18 | - keyword2 19 | # date: 2022-02-02 -- date is filled in automatically by xml2rfc if not given 20 | author: 21 | - role: editor # remove if not true 22 | ins: I. J-P. Surname [REPLACE] 23 | name: fullname [REPLACE] 24 | org: Organization [REPLACE/DELETE] 25 | street: Street [REPLACE/DELETE] 26 | city: City [REPLACE/DELETE] 27 | region: Region [REPLACE/DELETE] # not always available 28 | code: Postal code [REPLACE/DELETE] 29 | country: DE # use TLD (except UK) or country name 30 | phone: Phone [REPLACE/DELETE] 31 | email: Email [REPLACE/DELETE] 32 | uri: URI [REPLACE/DELETE] 33 | contributor: # Same structure as author list, but goes into contributors 34 | - name: Carsten Bormann 35 | org: Universität Bremen TZI 36 | email: cabo@tzi.org 37 | uri: https://rfc.space 38 | - name: Jay Daley 39 | org: IETF Administration LLC 40 | email: exec-director@ietf.org 41 | contribution: | 42 | Jay provided the **XML** version of the template. 43 | 44 | That was quite helpful. 45 | 46 | normative: 47 | RFC5234: # REPLACE 48 | informative: 49 | exampleRefMin: 50 | title: Title [REPLACE] 51 | author: 52 | - name: Givenname Surname[REPLACE] 53 | org: (ignored here anyway) 54 | - name: Givenname Surname1 Surname2 55 | surname: Surname1 Surname2 # needed for Spanish names etc. 56 | org: (ignored here anyway) 57 | date: 2006 58 | exampleRefOrg: 59 | target: http://www.example.com/ 60 | title: Title [REPLACE] 61 | author: 62 | - org: Organization [REPLACE] 63 | date: 1984-04 64 | 65 | --- abstract 66 | 67 | Abstract [REPLACE] 68 | 69 | --- middle 70 | 71 | # Introduction 72 | 73 | Introductory text [REPLACE] 74 | 75 | ## Requirements Language 76 | 77 | {::boilerplate bcp14-tagged} 78 | 79 | # Body [REPLACE] 80 | 81 | Some body text [REPLACE] 82 | 83 | This document normatively references {{RFC5234}} and has more 84 | information in {{exampleRefMin}} and {{exampleRefOrg}}. [REPLACE] 85 | 86 | 1. Ordered list item [REPLACE/DELETE] 87 | 2. Ordered list item [REPLACE/DELETE] 88 | 89 | * Bulleted list item [REPLACE/DELETE] 90 | * Bulleted list item [REPLACE/DELETE] 91 | 92 | 93 | {:vspace} 94 | First term: 95 | : Definition of the first term 96 | 97 | Second term: 98 | : Definition of the second term 99 | 101 | 102 | 103 | | Table head 1 [REPLACE] | Table head2 [REPLACE] | 104 | | Cell 11 [REPLACE] | Cell 12 [REPLACE] | 105 | | Cell 21 [REPLACE] | Cell 22 [REPLACE] | 106 | {: title="A nice table [REPLACE]"} 107 | 108 | ~~~~ language-REPLACE/DELETE 109 | source code goes here [REPLACE] 110 | ~~~~ 111 | {: title='Source [REPLACE]' sourcecode-markers="true"} 112 | 113 | 114 | # IANA Considerations {#IANA} 115 | 116 | This memo includes no request to IANA. [CHECK] 117 | 118 | 119 | # Security Considerations {#Security} 120 | 121 | This document should not affect the security of the Internet. [CHECK] 122 | 123 | 124 | --- back 125 | 126 | # Appendix 1 [REPLACE/DELETE] 127 | 128 | This becomes an Appendix [REPLACE] 129 | 130 | 131 | # Acknowledgements {#Acknowledgements} 132 | {: numbered="false"} 133 | 134 | This template uses extracts from templates written by 135 | {{{Pekka Savola}}}, {{{Elwyn Davies}}} and 136 | {{{Henrik Levkowetz}}}. [REPLACE] 137 | 138 | -------------------------------------------------------------------------------- /examples/skel.mkd: -------------------------------------------------------------------------------- 1 | --- 2 | coding: utf-8 3 | 4 | title: Many fine lunches and dinners 5 | abbrev: mfld 6 | docname: draft-mfld-00 7 | category: info 8 | 9 | stand_alone: yes 10 | pi: [toc, sortrefs, symrefs, comments] 11 | 12 | author: 13 | - 14 | ins: C. Bormann 15 | name: Carsten Bormann 16 | org: Universität Bremen TZI 17 | street: Postfach 330440 18 | city: Bremen 19 | code: D-28359 20 | country: Germany 21 | phone: +49-421-218-63921 22 | email: cabo@tzi.org 23 | 24 | --- abstract 25 | 26 | insert abstract here 27 | 28 | --- middle 29 | 30 | # Introduction 31 | 32 | This MAY {{?RFC2119}} be useful. 33 | -------------------------------------------------------------------------------- /examples/skel.txt: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | Network Working Group C. Bormann 6 | Internet-Draft Universitaet Bremen TZI 7 | Intended status: Informational March 20, 2015 8 | Expires: September 21, 2015 9 | 10 | 11 | Many fine lunches and dinners 12 | draft-mfld-00 13 | 14 | Abstract 15 | 16 | insert abstract here 17 | 18 | Status of This Memo 19 | 20 | This Internet-Draft is submitted in full conformance with the 21 | provisions of BCP 78 and BCP 79. 22 | 23 | Internet-Drafts are working documents of the Internet Engineering 24 | Task Force (IETF). Note that other groups may also distribute 25 | working documents as Internet-Drafts. The list of current Internet- 26 | Drafts is at http://datatracker.ietf.org/drafts/current/. 27 | 28 | Internet-Drafts are draft documents valid for a maximum of six months 29 | and may be updated, replaced, or obsoleted by other documents at any 30 | time. It is inappropriate to use Internet-Drafts as reference 31 | material or to cite them other than as "work in progress." 32 | 33 | This Internet-Draft will expire on September 21, 2015. 34 | 35 | Copyright Notice 36 | 37 | Copyright (c) 2015 IETF Trust and the persons identified as the 38 | document authors. All rights reserved. 39 | 40 | This document is subject to BCP 78 and the IETF Trust's Legal 41 | Provisions Relating to IETF Documents 42 | (http://trustee.ietf.org/license-info) in effect on the date of 43 | publication of this document. Please review these documents 44 | carefully, as they describe your rights and restrictions with respect 45 | to this document. Code Components extracted from this document must 46 | include Simplified BSD License text as described in Section 4.e of 47 | the Trust Legal Provisions and are provided without warranty as 48 | described in the Simplified BSD License. 49 | 50 | 51 | 52 | 53 | 54 | 55 | 56 | Bormann Expires September 21, 2015 [Page 1] 57 | 58 | Internet-Draft mfld March 2015 59 | 60 | 61 | Table of Contents 62 | 63 | 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 64 | 2. Informative References . . . . . . . . . . . . . . . . . . . 2 65 | Author's Address . . . . . . . . . . . . . . . . . . . . . . . . 2 66 | 67 | 1. Introduction 68 | 69 | This MAY [RFC2119] be useful. 70 | 71 | 2. Informative References 72 | 73 | [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 74 | Requirement Levels", BCP 14, RFC 2119, March 1997. 75 | 76 | Author's Address 77 | 78 | Carsten Bormann 79 | Universitaet Bremen TZI 80 | Postfach 330440 81 | Bremen D-28359 82 | Germany 83 | 84 | Phone: +49-421-218-63921 85 | Email: cabo@tzi.org 86 | 87 | 88 | 89 | 90 | 91 | 92 | 93 | 94 | 95 | 96 | 97 | 98 | 99 | 100 | 101 | 102 | 103 | 104 | 105 | 106 | 107 | 108 | 109 | 110 | 111 | 112 | Bormann Expires September 21, 2015 [Page 2] 113 | -------------------------------------------------------------------------------- /examples/stupid-s.mkd: -------------------------------------------------------------------------------- 1 | --- 2 | title: STUN/TURN using PHP in Despair 3 | abbrev: STuPiD 4 | docname: draft-hartke-xmpp-stupid-00 5 | date: 2009-07-05 6 | category: info 7 | 8 | ipr: trust200902 9 | area: General 10 | workgroup: XMPP Working Group 11 | keyword: Internet-Draft 12 | 13 | stand_alone: yes 14 | pi: [toc, sortrefs, symrefs] 15 | 16 | author: 17 | - 18 | ins: K. Hartke 19 | name: Klaus Hartke 20 | organization: Universität Bremen TZI 21 | email: hartke@tzi.org 22 | - 23 | ins: C. Bormann 24 | name: Carsten Bormann 25 | org: Universität Bremen TZI 26 | street: Postfach 330440 27 | city: Bremen 28 | code: D-28359 29 | country: Germany 30 | phone: +49-421-218-63921 31 | facsimile: +49-421-218-7000 32 | email: cabo@tzi.org 33 | 34 | normative: 35 | RFC2119: 36 | RFC3986: 37 | RFC4086: 38 | RFC4648: 39 | 40 | informative: 41 | RFC5389: 42 | I-D.ietf-behave-turn: 43 | STUNT: 44 | target: http://deusty.blogspot.com/2007/09/stunt-out-of-band-channels.html 45 | title: STUNT & out-of-band channels 46 | author: 47 | name: Robbie Hanson 48 | ins: R. Hanson 49 | date: 2007-09-17 50 | I-D.meyer-xmpp-e2e-encryption: 51 | I-D.ietf-xmpp-3920bis: 52 | 53 | 54 | 55 | --- abstract 56 | 57 | NAT (Network Address Translator) Traversal may require TURN 58 | (Traversal Using Relays around NAT) functionality in certain 59 | cases that are not unlikely to occur. There is little 60 | incentive to deploy TURN servers, except by those who need 61 | them — who may not be in a position to deploy a new protocol 62 | on an Internet-connected node, in particular not one with 63 | deployment requirements as high as those of TURN. 64 | 65 | "STUN/TURN using PHP in Despair" is a highly deployable 66 | protocol for obtaining TURN-like functionality, while also 67 | providing the most important function of STUN. 68 | 69 | --- middle 70 | 71 | Introduction {#problems} 72 | ============ 73 | 74 | NAT (Network Address Translator) Traversal may require TURN 75 | (Traversal Using Relays around NAT) 76 | {{I-D.ietf-behave-turn}} 77 | functionality in certain 78 | cases that are not unlikely to occur. There is little 79 | incentive to deploy TURN servers, except by those who need 80 | them — who may not be in a position to deploy a new protocol 81 | on an Internet-connected node, in particular not one with 82 | deployment requirements as high as those of TURN. 83 | 84 | "STUN/TURN using PHP in Despair" is a highly deployable 85 | protocol for obtaining TURN-like functionality, while also 86 | providing the most important function of STUN 87 | {{RFC5389}}. 88 | 89 | The high degree of deployability is achieved by making STuPiD 90 | a Web service, implementable in any Web application deployment 91 | scheme. As PHP appears to be the solution of choice for 92 | avoiding deployment problems in the Web world, a PHP-based 93 | sample implementation of STuPiD is presented in {{figimpl}} in {{impl}}. 94 | (This single-page script has been tested with a free-of-charge 95 | web hoster, so it should be deployable by literally everyone.) 96 | 97 | 98 | The Need for Standardization {#need} 99 | ---------------------------- 100 | 101 | If STuPiD is so easy to deploy, why standardize on it? 102 | First of all, STuPiD server implementations will be done by 103 | other people than the clients making use of the service. 104 | Clearly communicating between these communities is a good 105 | idea, in particular if there are security considerations. 106 | 107 | Having one standard form of STuPiD service instead of one 108 | specific to each kind of client also creates an incentive 109 | for optimized implementations. 110 | 111 | Finally, where STuPiD becomes part of a client standard 112 | (such as a potential extension to XMPP's in-band byte-stream 113 | protocol as hinted in {{xmpp}}), it is a good 114 | thing if STuPiD is already defined. 115 | 116 | Hence, this document focuses on the definition of the STuPiD 117 | service itself, tries to make this as general as possible 118 | without increasing complexity or cost and leaves the details 119 | of any client standards to future documents. 120 | 121 | 122 | Basic Protocol Operation {#ops} 123 | ======================== 124 | 125 | The STuPiD protocol will typically be used with application 126 | instances that first attempt to obtain connectivity using 127 | mechanisms similar to those described in the STUN 128 | specification {{RFC5389}}. However, with STuPiD, 129 | STUN is not really needed for TCP, as was demonstrated in 130 | previous STUN-like implementations {{STUNT}}. 131 | Instead, STuPiD (like {{STUNT}}) provides a 132 | simple Web service that 133 | echoes the remote address and port of an incoming HTTP 134 | request; in most cases, this is enough to get the job done. 135 | 136 | In case no connection can be established with this simple 137 | STUN(T)-like mechanism, a TURN-like relay is needed as a final 138 | fall-back. 139 | The STuPiD protocol supports this, but solely provides a way 140 | for the data to be 141 | relayed. STuPiD relies on an out-of-band channel to notify 142 | the peer whenever new data is available (synchronization signal). 143 | See {{xmpp}} for one likely example of such an 144 | out-of-band channel. 145 | (Note that the out-of-band channel may have a much lower 146 | throughput than the STuPiD relay channel — this is exactly 147 | the case in the example provided in {{xmpp}}, 148 | where the out-of-band channel is typically throughput-limited 149 | to on the order of a few kilobits per second.) 150 | 151 | By designing the STuPiD web service in such a way that it can 152 | be implemented by a simple PHP script such as that presented 153 | in {{impl}}, it is easy to deploy by those who 154 | need the STuPiD services. 155 | The combination of the low-throughput out-of-band channel for 156 | synchronization and the STuPiD web service for bulk data 157 | relaying is somewhat silly but gets the job done. 158 | 159 | The STuPiD data relay is implemented as follows (see {{figops}}): 160 | 161 | 1. Peer A, the source of the data to be relayed, stores a chunk of 162 | data at the STuPiD server using an opaque identifier, the "chunk 163 | identifier". How that chunk identifier is chosen is local to Peer 164 | A; it could be composed of a random session id and a sequence number. 165 | 166 | 2. Peer A notifies the receiver of the data, Peer 167 | B, that a new data chunk is available, specifying the URI needed 168 | for retrieval. 169 | This notification is provided through an out-out-band channel. 170 | (As an optimization for multiple consecutive transfers, A might 171 | inform B once of a constant prefix of that URI and only send a 172 | varying part such as a sequence number in each notification — 173 | this is something to be decided in the client-client notification 174 | protocol.) 175 | 176 | 3. Peer B retrieves the data from the STuPiD server using the URI 177 | provided by Peer A. 178 | 179 | Note that the data transfer mechanism is one-way, i.e. to send 180 | data in the other direction as well, Peer B needs to perform 181 | the same steps using a STuPiD server at the same or a 182 | different location. 183 | 184 | ~~~~~~~~~~ 185 | 186 | 187 | STuPiD ```````````````````````````````, 188 | Script <----------------------------. , 189 | | , 190 | ^ , | , 191 | | , | , 192 | (1) | , | , (3) 193 | POST | , | , GET 194 | | , | , 195 | | v | v 196 | 197 | Peer A -----------------------> Peer B 198 | (2) 199 | out-of-band 200 | Notification 201 | ~~~~~~~~~~ 202 | {: #figops title="STuPiD Protocol Operation"} 203 | 204 | 205 | Protocol Definition 206 | =================== 207 | 208 | Terminology {#Terminology} 209 | ----------- 210 | In this document, the key words "MUST", "MUST NOT", "REQUIRED", 211 | "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", 212 | and "OPTIONAL" are to be interpreted as described in BCP 14, RFC 2119 213 | {{RFC2119}} and indicate requirement levels for compliant STuPiD 214 | implementations. 215 | 216 | 217 | Discovering External IP Address and Port 218 | ---------------------------------------- 219 | 220 | A client may discover its external IP address and the port 221 | required for port prediction by performing a HTTP GET 222 | request to a STuPiD server. The STuPiD server MUST reply 223 | with the remote address and remote port in the following 224 | format: 225 | 226 | host ":" port 227 | 228 | where 'host' and 'port' are defined as in {{RFC3986}}. 229 | 230 | 231 | Storing Data 232 | ------------ 233 | 234 | Data chunks are stored using the POST request of HTTP. The 235 | STuPiD server MUST support one URI parameter which is passed 236 | as query-string: 237 | 238 | 'chid': A unique ID identifying the data chunk to be stored. 239 | The ID SHOULD be chosen from the characters of the base64url 240 | set {{RFC4648}}. 241 | 242 | The payload of the POST request MUST be the data to be 243 | stored. The 'Content-Type' SHOULD be 244 | 'application/octet-stream', although a STuPiD server 245 | implementation SHOULD simply ignore the 'Content-Type' as a 246 | client implementation may be restricted and may not able to 247 | specify a specific 'Content-Type'. (E.g., in certain cases, 248 | the peer may be limited to sending the data as 249 | multipart-form-encoded — still, the data is stored as a 250 | byte stream.) 251 | 252 | STuPiD servers may reject data chunks that are larger than 253 | some predefined limit. 254 | This maximum size in bytes of each data chunk is RECOMMENDED 255 | to be 65536 or more. 256 | 257 | As HTTP already provides data transparency, 258 | the data chunk SHOULD NOT be encoded using Base64 or any 259 | other data transparency mechanism; in any case, the STuPiD 260 | server will not attempt to decode the chunk. 261 | 262 | The sender MUST wait for the HTTP response before 263 | going on to notify the receiver. 264 | 265 | 266 | Notification 267 | ------------ 268 | 269 | The sender notifies the receiver of the data chunk by passing 270 | via an out-of-band channel (which is not part of the STuPiD 271 | protocol): 272 | 273 | The full URL from which the data chunk can be retrieved, 274 | i.e. the same URL that was used to store the data chunk, 275 | including the chunk ID parameter. 276 | 277 | The exact notification mechanism over the out-of-band channel 278 | and the definition of a session is dependent on the 279 | out-of-band channel. See {{xmpp}} for one 280 | example of such an out-of-band channel. 281 | 282 | 283 | Retrieving Data 284 | --------------- 285 | 286 | The notified peer retrieves the data chunk using a GET request 287 | with the URL supplied by the sender. The STuPiD server MUST 288 | set the 'Content-Type' of the returned body to 289 | 'application/octet-stream'. 290 | 291 | 292 | Implementation Notes 293 | ==================== 294 | 295 | A STuPiD server implementation SHOULD delete stored data some 296 | time after it was stored. It is RECOMMENDED not to delete the 297 | data before five minutes have elapsed after it was stored. 298 | Different client protocols will have different reactions to 299 | data that have been deleted prematurely and cannot be 300 | retrieved by the notified peer; this may be as trivial as 301 | packet loss or it may cause a reliable byte-stream to fail 302 | ({{impl}}). 303 | (TODO: It may be useful to provide some hints in the storing 304 | POST request.) 305 | 306 | STuPiD clients should aggregate data in order to minimize the 307 | number of requests to the STuPiD server per second. 308 | The specific aggregation method chosen depends on the data 309 | rate required (and the maximum chunk size), the latency 310 | requirements, and the application semantics. 311 | 312 | Clearly, it is up to the implementation to decide how the data 313 | chunks are actually stored. A sufficiently silly STuPiD server 314 | implementation might for instance use a MySQL database. 315 | 316 | 317 | Security Considerations 318 | ======================= 319 | 320 | The security objectives of STuPiD are to be as secure as if 321 | NAT traversal had succeeded, i.e., an on-path attacker can 322 | overhear and fake messages, but an off-path attacker cannot. 323 | If a higher level of security is desired, it should be 324 | provided on top of the data relayed by STuPiD, e.g. by using 325 | XTLS {{I-D.meyer-xmpp-e2e-encryption}}. 326 | 327 | Much of the security of STuPiD is based on the assumption that 328 | an off-path attacker cannot guess the chunk identifiers. A 329 | suitable source of randomness {{RFC4086}} should 330 | be used to generate at least a sufficiently large part of the 331 | chunk identifiers (e.g., the chunk identifier could be a hard 332 | to guess prefix followed by a serial number). 333 | 334 | To protect the STuPiD server against denial of service and 335 | possibly some forms of theft of service, it is RECOMMENDED 336 | that the POST side of the STuPiD server be protected by some 337 | form of authentication such as HTTP authentication. There is 338 | little need to protect the GET side. 339 | 340 | --- back 341 | 342 | 343 | Examples {#xmp} 344 | ======== 345 | 346 | This appendix provides some examples of the STuPiD protocol operation. 347 | 348 | ~~~~~~~~~~ 349 | Request: 350 | 351 | GET /stupid.php HTTP/1.0 352 | User-Agent: Example/1.11.4 353 | Accept: */* 354 | Host: example.org 355 | Connection: Keep-Alive 356 | 357 | Response: 358 | 359 | HTTP/1.1 200 OK 360 | Date: Sun, 05 Jul 2009 00:30:37 GMT 361 | Server: Apache/2.2 362 | Cache-Control: no-cache, must-revalidate 363 | Expires: Sat, 26 Jul 1997 05:00:00 GMT 364 | Vary: Accept-Encoding 365 | Content-Length: 17 366 | Keep-Alive: timeout=1, max=400 367 | Connection: Keep-Alive 368 | Content-Type: application/octet-stream 369 | 370 | 192.0.2.239:36654 371 | ~~~~~~~~~~ 372 | {: #figxmpdisco title="Discovering External IP Address and Port"} 373 | 374 | ~~~~~~~~~~ 375 | Request: 376 | 377 | POST /stupid.php?chid=i781hf64-0 HTTP/1.0 378 | User-Agent: Example/1.11.4 379 | Accept: */* 380 | Host: example.org 381 | Connection: Keep-Alive 382 | Content-Type: application/octet-stream 383 | Content-Length: 11 384 | 385 | Hello World 386 | 387 | Response: 388 | 389 | HTTP/1.1 200 OK 390 | Date: Sun, 05 Jul 2009 00:20:34 GMT 391 | Server: Apache/2.2 392 | Cache-Control: no-cache, must-revalidate 393 | Expires: Sat, 26 Jul 1997 05:00:00 GMT 394 | Vary: Accept-Encoding 395 | Content-Length: 0 396 | Keep-Alive: timeout=1, max=400 397 | Connection: Keep-Alive 398 | Content-Type: application/octet-stream 399 | ~~~~~~~~~~ 400 | {: #figxmpstore title="Storing Data"} 401 | 402 | ~~~~~~~~~~ 403 | Request: 404 | 405 | GET /stupid.php?chid=i781hf64-0 HTTP/1.0 406 | User-Agent: Example/1.11.4 407 | Accept: */* 408 | Host: example.org 409 | Connection: Keep-Alive 410 | 411 | Response: 412 | 413 | HTTP/1.1 200 OK 414 | Date: Sun, 05 Jul 2009 00:21:29 GMT 415 | Server: Apache/2.2 416 | Cache-Control: no-cache, must-revalidate 417 | Expires: Sat, 26 Jul 1997 05:00:00 GMT 418 | Vary: Accept-Encoding 419 | Content-Length: 11 420 | Keep-Alive: timeout=1, max=400 421 | Connection: Keep-Alive 422 | Content-Type: application/octet-stream 423 | 424 | Hello World 425 | ~~~~~~~~~~ 426 | {: #figxmpretr title="Retrieving Data"} 427 | 428 | 429 | Sample Implementation {#impl} 430 | ===================== 431 | 432 | ~~~~~~~~~~ 433 | 475 | ~~~~~~~~~~ 476 | {: #figimpl title="STuPiD Sample Implementation"} 477 | 478 | 479 | Using XMPP as Out-Of-Band Channel {#xmpp} 480 | ================================= 481 | 482 | XMPP {{I-D.ietf-xmpp-3920bis}} is a good choice for 483 | an out-of-band channel. 484 | 485 | The notification protocol is closely modeled after XMPP's 486 | In-Band Bytestreams (IBB, see 487 | http://xmpp.org/extensions/xep-0047.html). Just replace the 488 | namespace and insert the STuPiD Retrieval URI instead of the 489 | actual Base64 encoded data, see {{figxmpnots}}. 490 | (Note that the current proposal redundantly sends a sid and a 491 | seq as well as the chid composed of these two; it may be 492 | possible to optimize this, possibly sending the constant prefix 493 | of the URI once at bytestream creation time.) 494 | 495 | Notifications MUST be processed in the order they are 496 | received. If an out-of-sequence notification is received for a 497 | particular session (determined by checking the 'seq' attribute), 498 | then this indicates that a notification has been lost. The 499 | recipient MUST NOT process such an out-of-sequence notification, 500 | nor any that follow it within the same session; instead, the 501 | recipient MUST consider the session invalid. (Adapted from 502 | http://xmpp.org/extensions/xep-0047.html#send) 503 | 504 | Of course, other methods can be used for setup and teardown, such as Jingle 505 | (see http://xmpp.org/extensions/xep-0261.html). 506 | 507 | ~~~~~~~~~~ 508 | 512 | 516 | 517 | ~~~~~~~~~~ 518 | {: #figxmpcri title="Creating a Bytestream: Initiator requests session"} 519 | 520 | 521 | ~~~~~~~~~~ 522 | 526 | ~~~~~~~~~~ 527 | {: #figxmpcrr title="Creating a Bytestream: Responder accepts session"} 528 | 529 | 530 | 531 | ~~~~~~~~~~ 532 | 536 | 540 | 541 | ~~~~~~~~~~ 542 | {: #figxmpnots title="Sending Notifications: Notification in an IQ stanza"} 543 | 544 | ~~~~~~~~~~ 545 | 549 | ~~~~~~~~~~ 550 | {: #figxmpnota title="Sending Notifications: Acknowledging notification using IQ"} 551 | 552 | ~~~~~~~~~~ 553 | 557 | 559 | 560 | ~~~~~~~~~~ 561 | {: #figxmpclor title="Closing the Bytestream: Request"} 562 | 563 | ~~~~~~~~~~ 564 | 568 | ~~~~~~~~~~ 569 | {: #figxmpclos title="Closing the Bytestream: Success response"} 570 | -------------------------------------------------------------------------------- /examples/stupid-s.txt: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | XMPP Working Group K. Hartke 5 | Internet-Draft C. Bormann 6 | Intended status: Informational Universitaet Bremen TZI 7 | Expires: January 6, 2010 July 5, 2009 8 | 9 | 10 | STUN/TURN using PHP in Despair 11 | draft-hartke-xmpp-stupid-00 12 | 13 | Status of this Memo 14 | 15 | This Internet-Draft is submitted to IETF in full conformance with the 16 | provisions of BCP 78 and BCP 79. 17 | 18 | Internet-Drafts are working documents of the Internet Engineering 19 | Task Force (IETF), its areas, and its working groups. Note that 20 | other groups may also distribute working documents as Internet- 21 | Drafts. 22 | 23 | Internet-Drafts are draft documents valid for a maximum of six months 24 | and may be updated, replaced, or obsoleted by other documents at any 25 | time. It is inappropriate to use Internet-Drafts as reference 26 | material or to cite them other than as "work in progress." 27 | 28 | The list of current Internet-Drafts can be accessed at 29 | http://www.ietf.org/ietf/1id-abstracts.txt. 30 | 31 | The list of Internet-Draft Shadow Directories can be accessed at 32 | http://www.ietf.org/shadow.html. 33 | 34 | This Internet-Draft will expire on January 6, 2010. 35 | 36 | Copyright Notice 37 | 38 | Copyright (c) 2009 IETF Trust and the persons identified as the 39 | document authors. All rights reserved. 40 | 41 | This document is subject to BCP 78 and the IETF Trust's Legal 42 | Provisions Relating to IETF Documents in effect on the date of 43 | publication of this document (http://trustee.ietf.org/license-info). 44 | Please review these documents carefully, as they describe your rights 45 | and restrictions with respect to this document. 46 | 47 | 48 | 49 | 50 | 51 | 52 | 53 | 54 | 55 | Hartke & Bormann Expires January 6, 2010 [Page 1] 56 | 57 | Internet-Draft STuPiD July 2009 58 | 59 | 60 | Abstract 61 | 62 | NAT (Network Address Translator) Traversal may require TURN 63 | (Traversal Using Relays around NAT) functionality in certain cases 64 | that are not unlikely to occur. There is little incentive to deploy 65 | TURN servers, except by those who need them -- who may not be in a 66 | position to deploy a new protocol on an Internet-connected node, in 67 | particular not one with deployment requirements as high as those of 68 | TURN. 69 | 70 | "STUN/TURN using PHP in Despair" is a highly deployable protocol for 71 | obtaining TURN-like functionality, while also providing the most 72 | important function of STUN. 73 | 74 | 75 | Table of Contents 76 | 77 | 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3 78 | 1.1. The Need for Standardization . . . . . . . . . . . . . . . 3 79 | 2. Basic Protocol Operation . . . . . . . . . . . . . . . . . . . 4 80 | 3. Protocol Definition . . . . . . . . . . . . . . . . . . . . . 6 81 | 3.1. Terminology . . . . . . . . . . . . . . . . . . . . . . . 6 82 | 3.2. Discovering External IP Address and Port . . . . . . . . . 6 83 | 3.3. Storing Data . . . . . . . . . . . . . . . . . . . . . . . 6 84 | 3.4. Notification . . . . . . . . . . . . . . . . . . . . . . . 7 85 | 3.5. Retrieving Data . . . . . . . . . . . . . . . . . . . . . 7 86 | 4. Implementation Notes . . . . . . . . . . . . . . . . . . . . . 8 87 | 5. Security Considerations . . . . . . . . . . . . . . . . . . . 9 88 | 6. References . . . . . . . . . . . . . . . . . . . . . . . . . . 10 89 | 6.1. Normative References . . . . . . . . . . . . . . . . . . . 10 90 | 6.2. Informative References . . . . . . . . . . . . . . . . . . 10 91 | Appendix A. Examples . . . . . . . . . . . . . . . . . . . . . . 11 92 | Appendix B. Sample Implementation . . . . . . . . . . . . . . . . 14 93 | Appendix C. Using XMPP as Out-Of-Band Channel . . . . . . . . . . 15 94 | Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 17 95 | 96 | 97 | 98 | 99 | 100 | 101 | 102 | 103 | 104 | 105 | 106 | 107 | 108 | 109 | 110 | 111 | Hartke & Bormann Expires January 6, 2010 [Page 2] 112 | 113 | Internet-Draft STuPiD July 2009 114 | 115 | 116 | 1. Introduction 117 | 118 | NAT (Network Address Translator) Traversal may require TURN 119 | (Traversal Using Relays around NAT) [I-D.ietf-behave-turn] 120 | functionality in certain cases that are not unlikely to occur. There 121 | is little incentive to deploy TURN servers, except by those who need 122 | them -- who may not be in a position to deploy a new protocol on an 123 | Internet-connected node, in particular not one with deployment 124 | requirements as high as those of TURN. 125 | 126 | "STUN/TURN using PHP in Despair" is a highly deployable protocol for 127 | obtaining TURN-like functionality, while also providing the most 128 | important function of STUN [RFC5389]. 129 | 130 | The high degree of deployability is achieved by making STuPiD a Web 131 | service, implementable in any Web application deployment scheme. As 132 | PHP appears to be the solution of choice for avoiding deployment 133 | problems in the Web world, a PHP-based sample implementation of 134 | STuPiD is presented in Figure 5 in Appendix B. (This single-page 135 | script has been tested with a free-of-charge web hoster, so it should 136 | be deployable by literally everyone.) 137 | 138 | 1.1. The Need for Standardization 139 | 140 | If STuPiD is so easy to deploy, why standardize on it? First of all, 141 | STuPiD server implementations will be done by other people than the 142 | clients making use of the service. Clearly communicating between 143 | these communities is a good idea, in particular if there are security 144 | considerations. 145 | 146 | Having one standard form of STuPiD service instead of one specific to 147 | each kind of client also creates an incentive for optimized 148 | implementations. 149 | 150 | Finally, where STuPiD becomes part of a client standard (such as a 151 | potential extension to XMPP's in-band byte-stream protocol as hinted 152 | in Appendix C), it is a good thing if STuPiD is already defined. 153 | 154 | Hence, this document focuses on the definition of the STuPiD service 155 | itself, tries to make this as general as possible without increasing 156 | complexity or cost and leaves the details of any client standards to 157 | future documents. 158 | 159 | 160 | 161 | 162 | 163 | 164 | 165 | 166 | 167 | Hartke & Bormann Expires January 6, 2010 [Page 3] 168 | 169 | Internet-Draft STuPiD July 2009 170 | 171 | 172 | 2. Basic Protocol Operation 173 | 174 | The STuPiD protocol will typically be used with application instances 175 | that first attempt to obtain connectivity using mechanisms similar to 176 | those described in the STUN specification [RFC5389]. However, with 177 | STuPiD, STUN is not really needed for TCP, as was demonstrated in 178 | previous STUN-like implementations [STUNT]. Instead, STuPiD (like 179 | [STUNT]) provides a simple Web service that echoes the remote address 180 | and port of an incoming HTTP request; in most cases, this is enough 181 | to get the job done. 182 | 183 | In case no connection can be established with this simple STUN(T)- 184 | like mechanism, a TURN-like relay is needed as a final fall-back. 185 | The STuPiD protocol supports this, but solely provides a way for the 186 | data to be relayed. STuPiD relies on an out-of-band channel to 187 | notify the peer whenever new data is available (synchronization 188 | signal). See Appendix C for one likely example of such an out-of- 189 | band channel. (Note that the out-of-band channel may have a much 190 | lower throughput than the STuPiD relay channel -- this is exactly the 191 | case in the example provided in Appendix C, where the out-of-band 192 | channel is typically throughput-limited to on the order of a few 193 | kilobits per second.) 194 | 195 | By designing the STuPiD web service in such a way that it can be 196 | implemented by a simple PHP script such as that presented in 197 | Appendix B, it is easy to deploy by those who need the STuPiD 198 | services. The combination of the low-throughput out-of-band channel 199 | for synchronization and the STuPiD web service for bulk data relaying 200 | is somewhat silly but gets the job done. 201 | 202 | The STuPiD data relay is implemented as follows (see Figure 1): 203 | 204 | 1. Peer A, the source of the data to be relayed, stores a chunk of 205 | data at the STuPiD server using an opaque identifier, the "chunk 206 | identifier". How that chunk identifier is chosen is local to 207 | Peer A; it could be composed of a random session id and a 208 | sequence number. 209 | 210 | 2. Peer A notifies the receiver of the data, Peer B, that a new data 211 | chunk is available, specifying the URI needed for retrieval. 212 | This notification is provided through an out-out-band channel. 213 | (As an optimization for multiple consecutive transfers, A might 214 | inform B once of a constant prefix of that URI and only send a 215 | varying part such as a sequence number in each notification -- 216 | this is something to be decided in the client-client notification 217 | protocol.) 218 | 219 | 220 | 221 | 222 | 223 | Hartke & Bormann Expires January 6, 2010 [Page 4] 224 | 225 | Internet-Draft STuPiD July 2009 226 | 227 | 228 | 3. Peer B retrieves the data from the STuPiD server using the URI 229 | provided by Peer A. 230 | 231 | Note that the data transfer mechanism is one-way, i.e. to send data 232 | in the other direction as well, Peer B needs to perform the same 233 | steps using a STuPiD server at the same or a different location. 234 | 235 | 236 | STuPiD ```````````````````````````````, 237 | Script <----------------------------. , 238 | | , 239 | ^ , | , 240 | | , | , 241 | (1) | , | , (3) 242 | POST | , | , GET 243 | | , | , 244 | | v | v 245 | 246 | Peer A -----------------------> Peer B 247 | (2) 248 | out-of-band 249 | Notification 250 | 251 | Figure 1: STuPiD Protocol Operation 252 | 253 | 254 | 255 | 256 | 257 | 258 | 259 | 260 | 261 | 262 | 263 | 264 | 265 | 266 | 267 | 268 | 269 | 270 | 271 | 272 | 273 | 274 | 275 | 276 | 277 | 278 | 279 | Hartke & Bormann Expires January 6, 2010 [Page 5] 280 | 281 | Internet-Draft STuPiD July 2009 282 | 283 | 284 | 3. Protocol Definition 285 | 286 | 3.1. Terminology 287 | 288 | In this document, the key words "MUST", "MUST NOT", "REQUIRED", 289 | "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", 290 | and "OPTIONAL" are to be interpreted as described in BCP 14, RFC 2119 291 | [RFC2119] and indicate requirement levels for compliant STuPiD 292 | implementations. 293 | 294 | 3.2. Discovering External IP Address and Port 295 | 296 | A client may discover its external IP address and the port required 297 | for port prediction by performing a HTTP GET request to a STuPiD 298 | server. The STuPiD server MUST reply with the remote address and 299 | remote port in the following format: 300 | 301 | host ":" port 302 | 303 | where 'host' and 'port' are defined as in [RFC3986]. 304 | 305 | 3.3. Storing Data 306 | 307 | Data chunks are stored using the POST request of HTTP. The STuPiD 308 | server MUST support one URI parameter which is passed as query- 309 | string: 310 | 311 | 'chid': A unique ID identifying the data chunk to be stored. The ID 312 | SHOULD be chosen from the characters of the base64url set [RFC4648]. 313 | 314 | The payload of the POST request MUST be the data to be stored. The 315 | 'Content-Type' SHOULD be 'application/octet-stream', although a 316 | STuPiD server implementation SHOULD simply ignore the 'Content-Type' 317 | as a client implementation may be restricted and may not able to 318 | specify a specific 'Content-Type'. (E.g., in certain cases, the peer 319 | may be limited to sending the data as multipart-form-encoded -- 320 | still, the data is stored as a byte stream.) 321 | 322 | STuPiD servers may reject data chunks that are larger than some 323 | predefined limit. This maximum size in bytes of each data chunk is 324 | RECOMMENDED to be 65536 or more. 325 | 326 | As HTTP already provides data transparency, the data chunk SHOULD NOT 327 | be encoded using Base64 or any other data transparency mechanism; in 328 | any case, the STuPiD server will not attempt to decode the chunk. 329 | 330 | The sender MUST wait for the HTTP response before going on to notify 331 | the receiver. 332 | 333 | 334 | 335 | Hartke & Bormann Expires January 6, 2010 [Page 6] 336 | 337 | Internet-Draft STuPiD July 2009 338 | 339 | 340 | 3.4. Notification 341 | 342 | The sender notifies the receiver of the data chunk by passing via an 343 | out-of-band channel (which is not part of the STuPiD protocol): 344 | 345 | The full URL from which the data chunk can be retrieved, i.e. the 346 | same URL that was used to store the data chunk, including the chunk 347 | ID parameter. 348 | 349 | The exact notification mechanism over the out-of-band channel and the 350 | definition of a session is dependent on the out-of-band channel. See 351 | Appendix C for one example of such an out-of-band channel. 352 | 353 | 3.5. Retrieving Data 354 | 355 | The notified peer retrieves the data chunk using a GET request with 356 | the URL supplied by the sender. The STuPiD server MUST set the 357 | 'Content-Type' of the returned body to 'application/octet-stream'. 358 | 359 | 360 | 361 | 362 | 363 | 364 | 365 | 366 | 367 | 368 | 369 | 370 | 371 | 372 | 373 | 374 | 375 | 376 | 377 | 378 | 379 | 380 | 381 | 382 | 383 | 384 | 385 | 386 | 387 | 388 | 389 | 390 | 391 | Hartke & Bormann Expires January 6, 2010 [Page 7] 392 | 393 | Internet-Draft STuPiD July 2009 394 | 395 | 396 | 4. Implementation Notes 397 | 398 | A STuPiD server implementation SHOULD delete stored data some time 399 | after it was stored. It is RECOMMENDED not to delete the data before 400 | five minutes have elapsed after it was stored. Different client 401 | protocols will have different reactions to data that have been 402 | deleted prematurely and cannot be retrieved by the notified peer; 403 | this may be as trivial as packet loss or it may cause a reliable 404 | byte-stream to fail (Appendix B). (TODO: It may be useful to provide 405 | some hints in the storing POST request.) 406 | 407 | STuPiD clients should aggregate data in order to minimize the number 408 | of requests to the STuPiD server per second. The specific 409 | aggregation method chosen depends on the data rate required (and the 410 | maximum chunk size), the latency requirements, and the application 411 | semantics. 412 | 413 | Clearly, it is up to the implementation to decide how the data chunks 414 | are actually stored. A sufficiently silly STuPiD server 415 | implementation might for instance use a MySQL database. 416 | 417 | 418 | 419 | 420 | 421 | 422 | 423 | 424 | 425 | 426 | 427 | 428 | 429 | 430 | 431 | 432 | 433 | 434 | 435 | 436 | 437 | 438 | 439 | 440 | 441 | 442 | 443 | 444 | 445 | 446 | 447 | Hartke & Bormann Expires January 6, 2010 [Page 8] 448 | 449 | Internet-Draft STuPiD July 2009 450 | 451 | 452 | 5. Security Considerations 453 | 454 | The security objectives of STuPiD are to be as secure as if NAT 455 | traversal had succeeded, i.e., an on-path attacker can overhear and 456 | fake messages, but an off-path attacker cannot. If a higher level of 457 | security is desired, it should be provided on top of the data relayed 458 | by STuPiD, e.g. by using XTLS [I-D.meyer-xmpp-e2e-encryption]. 459 | 460 | Much of the security of STuPiD is based on the assumption that an 461 | off-path attacker cannot guess the chunk identifiers. A suitable 462 | source of randomness [RFC4086] should be used to generate at least a 463 | sufficiently large part of the chunk identifiers (e.g., the chunk 464 | identifier could be a hard to guess prefix followed by a serial 465 | number). 466 | 467 | To protect the STuPiD server against denial of service and possibly 468 | some forms of theft of service, it is RECOMMENDED that the POST side 469 | of the STuPiD server be protected by some form of authentication such 470 | as HTTP authentication. There is little need to protect the GET 471 | side. 472 | 473 | 474 | 475 | 476 | 477 | 478 | 479 | 480 | 481 | 482 | 483 | 484 | 485 | 486 | 487 | 488 | 489 | 490 | 491 | 492 | 493 | 494 | 495 | 496 | 497 | 498 | 499 | 500 | 501 | 502 | 503 | Hartke & Bormann Expires January 6, 2010 [Page 9] 504 | 505 | Internet-Draft STuPiD July 2009 506 | 507 | 508 | 6. References 509 | 510 | 6.1. Normative References 511 | 512 | [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 513 | Requirement Levels", BCP 14, RFC 2119, March 1997. 514 | 515 | [RFC3986] Berners-Lee, T., Fielding, R., and L. Masinter, "Uniform 516 | Resource Identifier (URI): Generic Syntax", STD 66, 517 | RFC 3986, January 2005. 518 | 519 | [RFC4086] Eastlake, D., Schiller, J., and S. Crocker, "Randomness 520 | Requirements for Security", BCP 106, RFC 4086, June 2005. 521 | 522 | [RFC4648] Josefsson, S., "The Base16, Base32, and Base64 Data 523 | Encodings", RFC 4648, October 2006. 524 | 525 | 6.2. Informative References 526 | 527 | [I-D.ietf-behave-turn] 528 | Rosenberg, J., Mahy, R., and P. Matthews, "Traversal Using 529 | Relays around NAT (TURN): Relay Extensions to Session 530 | Traversal Utilities for NAT (STUN)", 531 | draft-ietf-behave-turn-16 (work in progress), July 2009. 532 | 533 | [I-D.ietf-xmpp-3920bis] 534 | Saint-Andre, P., "Extensible Messaging and Presence 535 | Protocol (XMPP): Core", draft-ietf-xmpp-3920bis-22 (work 536 | in progress), December 2010. 537 | 538 | [I-D.meyer-xmpp-e2e-encryption] 539 | Meyer, D. and P. Saint-Andre, "XTLS: End-to-End Encryption 540 | for the Extensible Messaging and Presence Protocol (XMPP) 541 | Using Transport Layer Security (TLS)", 542 | draft-meyer-xmpp-e2e-encryption-02 (work in progress), 543 | June 2009. 544 | 545 | [RFC5389] Rosenberg, J., Mahy, R., Matthews, P., and D. Wing, 546 | "Session Traversal Utilities for NAT (STUN)", RFC 5389, 547 | October 2008. 548 | 549 | [STUNT] Hanson, R., "STUNT & out-of-band channels", 550 | September 2007, . 552 | 553 | 554 | 555 | 556 | 557 | 558 | 559 | Hartke & Bormann Expires January 6, 2010 [Page 10] 560 | 561 | Internet-Draft STuPiD July 2009 562 | 563 | 564 | Appendix A. Examples 565 | 566 | This appendix provides some examples of the STuPiD protocol 567 | operation. 568 | 569 | Request: 570 | 571 | GET /stupid.php HTTP/1.0 572 | User-Agent: Example/1.11.4 573 | Accept: */* 574 | Host: example.org 575 | Connection: Keep-Alive 576 | 577 | Response: 578 | 579 | HTTP/1.1 200 OK 580 | Date: Sun, 05 Jul 2009 00:30:37 GMT 581 | Server: Apache/2.2 582 | Cache-Control: no-cache, must-revalidate 583 | Expires: Sat, 26 Jul 1997 05:00:00 GMT 584 | Vary: Accept-Encoding 585 | Content-Length: 17 586 | Keep-Alive: timeout=1, max=400 587 | Connection: Keep-Alive 588 | Content-Type: application/octet-stream 589 | 590 | 192.0.2.239:36654 591 | 592 | Figure 2: Discovering External IP Address and Port 593 | 594 | 595 | 596 | 597 | 598 | 599 | 600 | 601 | 602 | 603 | 604 | 605 | 606 | 607 | 608 | 609 | 610 | 611 | 612 | 613 | 614 | 615 | Hartke & Bormann Expires January 6, 2010 [Page 11] 616 | 617 | Internet-Draft STuPiD July 2009 618 | 619 | 620 | Request: 621 | 622 | POST /stupid.php?chid=i781hf64-0 HTTP/1.0 623 | User-Agent: Example/1.11.4 624 | Accept: */* 625 | Host: example.org 626 | Connection: Keep-Alive 627 | Content-Type: application/octet-stream 628 | Content-Length: 11 629 | 630 | Hello World 631 | 632 | Response: 633 | 634 | HTTP/1.1 200 OK 635 | Date: Sun, 05 Jul 2009 00:20:34 GMT 636 | Server: Apache/2.2 637 | Cache-Control: no-cache, must-revalidate 638 | Expires: Sat, 26 Jul 1997 05:00:00 GMT 639 | Vary: Accept-Encoding 640 | Content-Length: 0 641 | Keep-Alive: timeout=1, max=400 642 | Connection: Keep-Alive 643 | Content-Type: application/octet-stream 644 | 645 | Figure 3: Storing Data 646 | 647 | 648 | 649 | 650 | 651 | 652 | 653 | 654 | 655 | 656 | 657 | 658 | 659 | 660 | 661 | 662 | 663 | 664 | 665 | 666 | 667 | 668 | 669 | 670 | 671 | Hartke & Bormann Expires January 6, 2010 [Page 12] 672 | 673 | Internet-Draft STuPiD July 2009 674 | 675 | 676 | Request: 677 | 678 | GET /stupid.php?chid=i781hf64-0 HTTP/1.0 679 | User-Agent: Example/1.11.4 680 | Accept: */* 681 | Host: example.org 682 | Connection: Keep-Alive 683 | 684 | Response: 685 | 686 | HTTP/1.1 200 OK 687 | Date: Sun, 05 Jul 2009 00:21:29 GMT 688 | Server: Apache/2.2 689 | Cache-Control: no-cache, must-revalidate 690 | Expires: Sat, 26 Jul 1997 05:00:00 GMT 691 | Vary: Accept-Encoding 692 | Content-Length: 11 693 | Keep-Alive: timeout=1, max=400 694 | Connection: Keep-Alive 695 | Content-Type: application/octet-stream 696 | 697 | Hello World 698 | 699 | Figure 4: Retrieving Data 700 | 701 | 702 | 703 | 704 | 705 | 706 | 707 | 708 | 709 | 710 | 711 | 712 | 713 | 714 | 715 | 716 | 717 | 718 | 719 | 720 | 721 | 722 | 723 | 724 | 725 | 726 | 727 | Hartke & Bormann Expires January 6, 2010 [Page 13] 728 | 729 | Internet-Draft STuPiD July 2009 730 | 731 | 732 | Appendix B. Sample Implementation 733 | 734 | 776 | 777 | Figure 5: STuPiD Sample Implementation 778 | 779 | 780 | 781 | 782 | 783 | Hartke & Bormann Expires January 6, 2010 [Page 14] 784 | 785 | Internet-Draft STuPiD July 2009 786 | 787 | 788 | Appendix C. Using XMPP as Out-Of-Band Channel 789 | 790 | XMPP [I-D.ietf-xmpp-3920bis] is a good choice for an out-of-band 791 | channel. 792 | 793 | The notification protocol is closely modeled after XMPP's In-Band 794 | Bytestreams (IBB, see http://xmpp.org/extensions/xep-0047.html). 795 | Just replace the namespace and insert the STuPiD Retrieval URI 796 | instead of the actual Base64 encoded data, see Figure 8. (Note that 797 | the current proposal redundantly sends a sid and a seq as well as the 798 | chid composed of these two; it may be possible to optimize this, 799 | possibly sending the constant prefix of the URI once at bytestream 800 | creation time.) 801 | 802 | Notifications MUST be processed in the order they are received. If 803 | an out-of-sequence notification is received for a particular session 804 | (determined by checking the 'seq' attribute), then this indicates 805 | that a notification has been lost. The recipient MUST NOT process 806 | such an out-of-sequence notification, nor any that follow it within 807 | the same session; instead, the recipient MUST consider the session 808 | invalid. (Adapted from 809 | http://xmpp.org/extensions/xep-0047.html#send) 810 | 811 | Of course, other methods can be used for setup and teardown, such as 812 | Jingle (see http://xmpp.org/extensions/xep-0261.html). 813 | 814 | 818 | 822 | 823 | 824 | Figure 6: Creating a Bytestream: Initiator requests session 825 | 826 | 827 | 831 | 832 | Figure 7: Creating a Bytestream: Responder accepts session 833 | 834 | 835 | 836 | 837 | 838 | 839 | Hartke & Bormann Expires January 6, 2010 [Page 15] 840 | 841 | Internet-Draft STuPiD July 2009 842 | 843 | 844 | 848 | 852 | 853 | 854 | Figure 8: Sending Notifications: Notification in an IQ stanza 855 | 856 | 857 | 861 | 862 | Figure 9: Sending Notifications: Acknowledging notification using IQ 863 | 864 | 865 | 869 | 871 | 872 | 873 | Figure 10: Closing the Bytestream: Request 874 | 875 | 876 | 880 | 881 | Figure 11: Closing the Bytestream: Success response 882 | 883 | 884 | 885 | 886 | 887 | 888 | 889 | 890 | 891 | 892 | 893 | 894 | 895 | Hartke & Bormann Expires January 6, 2010 [Page 16] 896 | 897 | Internet-Draft STuPiD July 2009 898 | 899 | 900 | Authors' Addresses 901 | 902 | Klaus Hartke 903 | Universitaet Bremen TZI 904 | 905 | Email: hartke@tzi.org 906 | 907 | 908 | Carsten Bormann 909 | Universitaet Bremen TZI 910 | Postfach 330440 911 | Bremen D-28359 912 | Germany 913 | 914 | Phone: +49-421-218-63921 915 | Fax: +49-421-218-7000 916 | Email: cabo@tzi.org 917 | 918 | 919 | 920 | 921 | 922 | 923 | 924 | 925 | 926 | 927 | 928 | 929 | 930 | 931 | 932 | 933 | 934 | 935 | 936 | 937 | 938 | 939 | 940 | 941 | 942 | 943 | 944 | 945 | 946 | 947 | 948 | 949 | 950 | 951 | Hartke & Bormann Expires January 6, 2010 [Page 17] 952 | 953 | -------------------------------------------------------------------------------- /examples/stupid.mkd: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 6 | 7 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | STUN/TURN using PHP in Despair 17 | 18 | 19 | Universität Bremen TZI 20 |
21 | hartke@tzi.org 22 |
23 | 24 | 25 | Universität Bremen TZI 26 |
27 | 28 | Postfach 330440 29 | Bremen 30 | D-28359 31 | Germany 32 | 33 | +49-421-218-63921 34 | +49-421-218-7000 35 | cabo@tzi.org 36 |
37 | 38 | 39 | 40 | 41 | General 42 | XMPP Working Group 43 | Internet-Draft 44 | 45 | 46 | NAT (Network Address Translator) Traversal may require TURN 47 | (Traversal Using Relays around NAT) functionality in certain 48 | cases that are not unlikely to occur. There is little 49 | incentive to deploy TURN servers, except by those who need 50 | them — who may not be in a position to deploy a new protocol 51 | on an Internet-connected node, in particular not one with 52 | deployment requirements as high as those of TURN. 53 | 54 | 55 | "STUN/TURN using PHP in Despair" is a highly deployable 56 | protocol for obtaining TURN-like functionality, while also 57 | providing the most important function of STUN. 58 | 59 | 60 | 61 | 62 | 63 | 64 | {:/nomarkdown} 65 | 66 | Introduction {#problems} 67 | ============ 68 | 69 | NAT (Network Address Translator) Traversal may require TURN 70 | (Traversal Using Relays around NAT) 71 | {{I-D.ietf-behave-turn}} 72 | functionality in certain 73 | cases that are not unlikely to occur. There is little 74 | incentive to deploy TURN servers, except by those who need 75 | them — who may not be in a position to deploy a new protocol 76 | on an Internet-connected node, in particular not one with 77 | deployment requirements as high as those of TURN. 78 | 79 | "STUN/TURN using PHP in Despair" is a highly deployable 80 | protocol for obtaining TURN-like functionality, while also 81 | providing the most important function of STUN 82 | {{RFC5389}}. 83 | 84 | The high degree of deployability is achieved by making STuPiD 85 | a Web service, implementable in any Web application deployment 86 | scheme. As PHP appears to be the solution of choice for 87 | avoiding deployment problems in the Web world, a PHP-based 88 | sample implementation of STuPiD is presented in {{figimpl}} in {{impl}}. 89 | (This single-page script has been tested with a free-of-charge 90 | web hoster, so it should be deployable by literally everyone.) 91 | 92 | 93 | The Need for Standardization {#need} 94 | ---------------------------- 95 | 96 | If STuPiD is so easy to deploy, why standardize on it? 97 | First of all, STuPiD server implementations will be done by 98 | other people than the clients making use of the service. 99 | Clearly communicating between these communities is a good 100 | idea, in particular if there are security considerations. 101 | 102 | Having one standard form of STuPiD service instead of one 103 | specific to each kind of client also creates an incentive 104 | for optimized implementations. 105 | 106 | Finally, where STuPiD becomes part of a client standard 107 | (such as a potential extension to XMPP's in-band byte-stream 108 | protocol as hinted in {{xmpp}}), it is a good 109 | thing if STuPiD is already defined. 110 | 111 | Hence, this document focuses on the definition of the STuPiD 112 | service itself, tries to make this as general as possible 113 | without increasing complexity or cost and leaves the details 114 | of any client standards to future documents. 115 | 116 | 117 | Basic Protocol Operation {#ops} 118 | ======================== 119 | 120 | The STuPiD protocol will typically be used with application 121 | instances that first attempt to obtain connectivity using 122 | mechanisms similar to those described in the STUN 123 | specification {{RFC5389}}. However, with STuPiD, 124 | STUN is not really needed for TCP, as was demonstrated in 125 | previous STUN-like implementations {{STUNT}}. 126 | Instead, STuPiD (like {{STUNT}}) provides a 127 | simple Web service that 128 | echoes the remote address and port of an incoming HTTP 129 | request; in most cases, this is enough to get the job done. 130 | 131 | In case no connection can be established with this simple 132 | STUN(T)-like mechanism, a TURN-like relay is needed as a final 133 | fall-back. 134 | The STuPiD protocol supports this, but solely provides a way 135 | for the data to be 136 | relayed. STuPiD relies on an out-of-band channel to notify 137 | the peer whenever new data is available (synchronization signal). 138 | See {{xmpp}} for one likely example of such an 139 | out-of-band channel. 140 | (Note that the out-of-band channel may have a much lower 141 | throughput than the STuPiD relay channel — this is exactly 142 | the case in the example provided in {{xmpp}}, 143 | where the out-of-band channel is typically throughput-limited 144 | to on the order of a few kilobits per second.) 145 | 146 | By designing the STuPiD web service in such a way that it can 147 | be implemented by a simple PHP script such as that presented 148 | in {{impl}}, it is easy to deploy by those who 149 | need the STuPiD services. 150 | The combination of the low-throughput out-of-band channel for 151 | synchronization and the STuPiD web service for bulk data 152 | relaying is somewhat silly but gets the job done. 153 | 154 | The STuPiD data relay is implemented as follows (see {{figops}}): 155 | 156 | 1. Peer A, the source of the data to be relayed, stores a chunk of 157 | data at the STuPiD server using an opaque identifier, the "chunk 158 | identifier". How that chunk identifier is chosen is local to Peer 159 | A; it could be composed of a random session id and a sequence number. 160 | 161 | 2. Peer A notifies the receiver of the data, Peer 162 | B, that a new data chunk is available, specifying the URI needed 163 | for retrieval. 164 | This notification is provided through an out-out-band channel. 165 | (As an optimization for multiple consecutive transfers, A might 166 | inform B once of a constant prefix of that URI and only send a 167 | varying part such as a sequence number in each notification — 168 | this is something to be decided in the client-client notification 169 | protocol.) 170 | 171 | 3. Peer B retrieves the data from the STuPiD server using the URI 172 | provided by Peer A. 173 | 174 | Note that the data transfer mechanism is one-way, i.e. to send 175 | data in the other direction as well, Peer B needs to perform 176 | the same steps using a STuPiD server at the same or a 177 | different location. 178 | 179 | ~~~~~~~~~~ 180 | 181 | 182 | STuPiD ```````````````````````````````, 183 | Script <----------------------------. , 184 | | , 185 | ^ , | , 186 | | , | , 187 | (1) | , | , (3) 188 | POST | , | , GET 189 | | , | , 190 | | v | v 191 | 192 | Peer A -----------------------> Peer B 193 | (2) 194 | out-of-band 195 | Notification 196 | ~~~~~~~~~~ 197 | {: #figops title="STuPiD Protocol Operation"} 198 | 199 | 200 | Protocol Definition 201 | =================== 202 | 203 | Terminology {#Terminology} 204 | ----------- 205 | In this document, the key words "MUST", "MUST NOT", "REQUIRED", 206 | "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", 207 | and "OPTIONAL" are to be interpreted as described in BCP 14, RFC 2119 208 | {{RFC2119}} and indicate requirement levels for compliant STuPiD 209 | implementations. 210 | 211 | 212 | Discovering External IP Address and Port 213 | ---------------------------------------- 214 | 215 | A client may discover its external IP address and the port 216 | required for port prediction by performing a HTTP GET 217 | request to a STuPiD server. The STuPiD server MUST reply 218 | with the remote address and remote port in the following 219 | format: 220 | 221 | host ":" port 222 | 223 | where 'host' and 'port' are defined as in {{RFC3986}}. 224 | 225 | 226 | Storing Data 227 | ------------ 228 | 229 | Data chunks are stored using the POST request of HTTP. The 230 | STuPiD server MUST support one URI parameter which is passed 231 | as query-string: 232 | 233 | 'chid': A unique ID identifying the data chunk to be stored. 234 | The ID SHOULD be chosen from the characters of the base64url 235 | set {{RFC4648}}. 236 | 237 | The payload of the POST request MUST be the data to be 238 | stored. The 'Content-Type' SHOULD be 239 | 'application/octet-stream', although a STuPiD server 240 | implementation SHOULD simply ignore the 'Content-Type' as a 241 | client implementation may be restricted and may not able to 242 | specify a specific 'Content-Type'. (E.g., in certain cases, 243 | the peer may be limited to sending the data as 244 | multipart-form-encoded — still, the data is stored as a 245 | byte stream.) 246 | 247 | STuPiD servers may reject data chunks that are larger than 248 | some predefined limit. 249 | This maximum size in bytes of each data chunk is RECOMMENDED 250 | to be 65536 or more. 251 | 252 | As HTTP already provides data transparency, 253 | the data chunk SHOULD NOT be encoded using Base64 or any 254 | other data transparency mechanism; in any case, the STuPiD 255 | server will not attempt to decode the chunk. 256 | 257 | The sender MUST wait for the HTTP response before 258 | going on to notify the receiver. 259 | 260 | 261 | Notification 262 | ------------ 263 | 264 | The sender notifies the receiver of the data chunk by passing 265 | via an out-of-band channel (which is not part of the STuPiD 266 | protocol): 267 | 268 | The full URL from which the data chunk can be retrieved, 269 | i.e. the same URL that was used to store the data chunk, 270 | including the chunk ID parameter. 271 | 272 | The exact notification mechanism over the out-of-band channel 273 | and the definition of a session is dependent on the 274 | out-of-band channel. See {{xmpp}} for one 275 | example of such an out-of-band channel. 276 | 277 | 278 | Retrieving Data 279 | --------------- 280 | 281 | The notified peer retrieves the data chunk using a GET request 282 | with the URL supplied by the sender. The STuPiD server MUST 283 | set the 'Content-Type' of the returned body to 284 | 'application/octet-stream'. 285 | 286 | 287 | Implementation Notes 288 | ==================== 289 | 290 | A STuPiD server implementation SHOULD delete stored data some 291 | time after it was stored. It is RECOMMENDED not to delete the 292 | data before five minutes have elapsed after it was stored. 293 | Different client protocols will have different reactions to 294 | data that have been deleted prematurely and cannot be 295 | retrieved by the notified peer; this may be as trivial as 296 | packet loss or it may cause a reliable byte-stream to fail 297 | ({{impl}}). 298 | (TODO: It may be useful to provide some hints in the storing 299 | POST request.) 300 | 301 | STuPiD clients should aggregate data in order to minimize the 302 | number of requests to the STuPiD server per second. 303 | The specific aggregation method chosen depends on the data 304 | rate required (and the maximum chunk size), the latency 305 | requirements, and the application semantics. 306 | 307 | Clearly, it is up to the implementation to decide how the data 308 | chunks are actually stored. A sufficiently silly STuPiD server 309 | implementation might for instance use a MySQL database. 310 | 311 | 312 | Security Considerations 313 | ======================= 314 | 315 | The security objectives of STuPiD are to be as secure as if 316 | NAT traversal had succeeded, i.e., an on-path attacker can 317 | overhear and fake messages, but an off-path attacker cannot. 318 | If a higher level of security is desired, it should be 319 | provided on top of the data relayed by STuPiD, e.g. by using 320 | XTLS {{I-D.meyer-xmpp-e2e-encryption}}. 321 | 322 | Much of the security of STuPiD is based on the assumption that 323 | an off-path attacker cannot guess the chunk identifiers. A 324 | suitable source of randomness {{RFC4086}} should 325 | be used to generate at least a sufficiently large part of the 326 | chunk identifiers (e.g., the chunk identifier could be a hard 327 | to guess prefix followed by a serial number). 328 | 329 | To protect the STuPiD server against denial of service and 330 | possibly some forms of theft of service, it is RECOMMENDED 331 | that the POST side of the STuPiD server be protected by some 332 | form of authentication such as HTTP authentication. There is 333 | little need to protect the GET side. 334 | 335 | {::nomarkdown} 336 | 337 | 338 | 339 | 340 | 341 | 342 | 343 | {:/nomarkdown} 344 | ![:include:](RFC2119) 345 | 346 | ![:include:](RFC3986) 347 | 348 | ![:include:](RFC4086) 349 | 350 | ![:include:](RFC4648) 351 | 352 | {::nomarkdown} 353 | 354 | 355 | 356 | 357 | 358 | {:/nomarkdown} 359 | ![:include:](RFC5389) 360 | 361 | ![:include:](I-D.ietf-behave-turn) 362 | 363 | 364 | 365 | STUNT & out-of-band channels 366 | 367 | 368 | 369 | 370 | 371 | 372 | 373 | 374 | ![:include:](I-D.meyer-xmpp-e2e-encryption) 375 | 376 | ![:include:](I-D.ietf-xmpp-3920bis) 377 | 378 | {::nomarkdown} 379 | 380 | {:/nomarkdown} 381 | 382 | Examples {#xmp} 383 | ======== 384 | 385 | This appendix provides some examples of the STuPiD protocol operation. 386 | 387 | ~~~~~~~~~~ 388 | Request: 389 | 390 | GET /stupid.php HTTP/1.0 391 | User-Agent: Example/1.11.4 392 | Accept: */* 393 | Host: example.org 394 | Connection: Keep-Alive 395 | 396 | Response: 397 | 398 | HTTP/1.1 200 OK 399 | Date: Sun, 05 Jul 2009 00:30:37 GMT 400 | Server: Apache/2.2 401 | Cache-Control: no-cache, must-revalidate 402 | Expires: Sat, 26 Jul 1997 05:00:00 GMT 403 | Vary: Accept-Encoding 404 | Content-Length: 17 405 | Keep-Alive: timeout=1, max=400 406 | Connection: Keep-Alive 407 | Content-Type: application/octet-stream 408 | 409 | 192.0.2.239:36654 410 | ~~~~~~~~~~ 411 | {: #figxmpdisco title="Discovering External IP Address and Port"} 412 | 413 | ~~~~~~~~~~ 414 | Request: 415 | 416 | POST /stupid.php?chid=i781hf64-0 HTTP/1.0 417 | User-Agent: Example/1.11.4 418 | Accept: */* 419 | Host: example.org 420 | Connection: Keep-Alive 421 | Content-Type: application/octet-stream 422 | Content-Length: 11 423 | 424 | Hello World 425 | 426 | Response: 427 | 428 | HTTP/1.1 200 OK 429 | Date: Sun, 05 Jul 2009 00:20:34 GMT 430 | Server: Apache/2.2 431 | Cache-Control: no-cache, must-revalidate 432 | Expires: Sat, 26 Jul 1997 05:00:00 GMT 433 | Vary: Accept-Encoding 434 | Content-Length: 0 435 | Keep-Alive: timeout=1, max=400 436 | Connection: Keep-Alive 437 | Content-Type: application/octet-stream 438 | ~~~~~~~~~~ 439 | {: #figxmpstore title="Storing Data"} 440 | 441 | ~~~~~~~~~~ 442 | Request: 443 | 444 | GET /stupid.php?chid=i781hf64-0 HTTP/1.0 445 | User-Agent: Example/1.11.4 446 | Accept: */* 447 | Host: example.org 448 | Connection: Keep-Alive 449 | 450 | Response: 451 | 452 | HTTP/1.1 200 OK 453 | Date: Sun, 05 Jul 2009 00:21:29 GMT 454 | Server: Apache/2.2 455 | Cache-Control: no-cache, must-revalidate 456 | Expires: Sat, 26 Jul 1997 05:00:00 GMT 457 | Vary: Accept-Encoding 458 | Content-Length: 11 459 | Keep-Alive: timeout=1, max=400 460 | Connection: Keep-Alive 461 | Content-Type: application/octet-stream 462 | 463 | Hello World 464 | ~~~~~~~~~~ 465 | {: #figxmpretr title="Retrieving Data"} 466 | 467 | 468 | Sample Implementation {#impl} 469 | ===================== 470 | 471 | ~~~~~~~~~~ 472 | 514 | ~~~~~~~~~~ 515 | {: #figimpl title="STuPiD Sample Implementation"} 516 | 517 | 518 | Using XMPP as Out-Of-Band Channel {#xmpp} 519 | ================================= 520 | 521 | XMPP {{I-D.ietf-xmpp-3920bis}} is a good choice for 522 | an out-of-band channel. 523 | 524 | The notification protocol is closely modeled after XMPP's 525 | In-Band Bytestreams (IBB, see 526 | http://xmpp.org/extensions/xep-0047.html). Just replace the 527 | namespace and insert the STuPiD Retrieval URI instead of the 528 | actual Base64 encoded data, see {{figxmpnots}}. 529 | (Note that the current proposal redundantly sends a sid and a 530 | seq as well as the chid composed of these two; it may be 531 | possible to optimize this, possibly sending the constant prefix 532 | of the URI once at bytestream creation time.) 533 | 534 | Notifications MUST be processed in the order they are 535 | received. If an out-of-sequence notification is received for a 536 | particular session (determined by checking the 'seq' attribute), 537 | then this indicates that a notification has been lost. The 538 | recipient MUST NOT process such an out-of-sequence notification, 539 | nor any that follow it within the same session; instead, the 540 | recipient MUST consider the session invalid. (Adapted from 541 | http://xmpp.org/extensions/xep-0047.html#send) 542 | 543 | Of course, other methods can be used for setup and teardown, such as Jingle 544 | (see http://xmpp.org/extensions/xep-0261.html). 545 | 546 | ~~~~~~~~~~ 547 | 551 | 555 | 556 | ~~~~~~~~~~ 557 | {: #figxmpcri title="Creating a Bytestream: Initiator requests session"} 558 | 559 | 560 | ~~~~~~~~~~ 561 | 565 | ~~~~~~~~~~ 566 | {: #figxmpcrr title="Creating a Bytestream: Responder accepts session"} 567 | 568 | 569 | 570 | ~~~~~~~~~~ 571 | 575 | 579 | 580 | ~~~~~~~~~~ 581 | {: #figxmpnots title="Sending Notifications: Notification in an IQ stanza"} 582 | 583 | ~~~~~~~~~~ 584 | 588 | ~~~~~~~~~~ 589 | {: #figxmpnota title="Sending Notifications: Acknowledging notification using IQ"} 590 | 591 | ~~~~~~~~~~ 592 | 596 | 598 | 599 | ~~~~~~~~~~ 600 | {: #figxmpclor title="Closing the Bytestream: Request"} 601 | 602 | ~~~~~~~~~~ 603 | 607 | ~~~~~~~~~~ 608 | {: #figxmpclos title="Closing the Bytestream: Success response"} 609 | 610 | {::nomarkdown} 611 | 612 | 613 | 614 | -------------------------------------------------------------------------------- /kramdown-rfc.gemspec: -------------------------------------------------------------------------------- 1 | spec = Gem::Specification.new do |s| 2 | s.name = 'kramdown-rfc' 3 | s.version = '1.7.29' 4 | s.summary = "Kramdown extension for generating RFCXML (RFC 799x)." 5 | s.description = %{An RFCXML (RFC 799x) generating backend for Thomas Leitner's 6 | "kramdown" markdown parser. Mostly useful for RFC writers.} 7 | s.add_dependency('kramdown-rfc2629', s.version) 8 | s.author = "Carsten Bormann" 9 | s.email = "cabo@tzi.org" 10 | s.homepage = "http://github.com/cabo/kramdown-rfc" 11 | s.license = 'MIT' 12 | end 13 | -------------------------------------------------------------------------------- /kramdown-rfc2629.gemspec: -------------------------------------------------------------------------------- 1 | spec = Gem::Specification.new do |s| 2 | s.name = 'kramdown-rfc2629' 3 | s.version = '1.7.29' 4 | s.summary = "Kramdown extension for generating RFCXML (RFC 799x)." 5 | s.description = %{An RFCXML (RFC 799x) generating backend for Thomas Leitner's 6 | "kramdown" markdown parser. Mostly useful for RFC writers.} 7 | s.add_dependency('kramdown', '~> 2.4.0') 8 | s.add_dependency('kramdown-parser-gfm', '~> 1.1') 9 | s.add_dependency('certified', '~> 1.0') 10 | s.add_dependency('json_pure', '~> 2.0') 11 | s.add_dependency('unicode-name', '~> 1.0') 12 | s.add_dependency('unicode-blocks', '~> 1.0') 13 | s.add_dependency('unicode-scripts', '~> 1.0') 14 | s.add_dependency('net-http-persistent', '~> 4.0') 15 | s.add_dependency('differ', '~> 0.1') 16 | s.add_dependency('base64', '>= 0.1') 17 | s.add_dependency('ostruct', '~> 0.6') 18 | s.files = Dir['lib/**/*.rb'] + 19 | %w(README.md LICENSE kramdown-rfc2629.gemspec 20 | bin/kdrfc bin/kramdown-rfc bin/kramdown-rfc2629 21 | bin/doilit bin/echars bin/kramdown-rfc-extract-markdown 22 | bin/kramdown-rfc-extract-sourcecode 23 | bin/kramdown-rfc-extract-figures-tables 24 | bin/kramdown-rfc-lsr data/kramdown-rfc2629.erb 25 | data/encoding-fallbacks.txt data/math.json 26 | bin/kramdown-rfc-cache-subseries-bibxml 27 | bin/kramdown-rfc-autolink-iref-cleanup 28 | bin/de-gfm 29 | bin/kramdown-rfc-clean-svg-ids) 30 | s.require_path = 'lib' 31 | s.executables = ['kramdown-rfc', 'kramdown-rfc2629', 'doilit', 'echars', 32 | 'kramdown-rfc-extract-markdown', 33 | 'kramdown-rfc-extract-sourcecode', 34 | 'kramdown-rfc-extract-figures-tables', 35 | 'kramdown-rfc-lsr', 36 | 'kdrfc', 'kramdown-rfc-cache-i-d-bibxml', 37 | 'kramdown-rfc-cache-subseries-bibxml', 38 | 'kramdown-rfc-autolink-iref-cleanup', 39 | 'de-gfm', 40 | 'kramdown-rfc-clean-svg-ids'] 41 | s.required_ruby_version = '>= 2.3.0' 42 | s.author = "Carsten Bormann" 43 | s.email = "cabo@tzi.org" 44 | s.homepage = "http://github.com/cabo/kramdown-rfc" 45 | s.license = 'MIT' 46 | end 47 | -------------------------------------------------------------------------------- /lib/kramdown-rfc/autolink-iref-cleanup.rb: -------------------------------------------------------------------------------- 1 | require 'rexml/document' 2 | 3 | def autolink_iref_cleanup(d) 4 | 5 | d.root.get_elements("//section[@anchor]").each do |sec| 6 | anchor = sec['anchor'] 7 | irefs = {} 8 | sec.get_elements(".//xref[@target='#{anchor}'][@format='none']").each do |xr| 9 | ne = xr.previous_element # 9c87e84 iref now before xref 10 | if ne && ne.name == "iref" && (item = ne['item']) 11 | irefs[item] = ne['subitem'] # XXX one subitem only 12 | ne.remove 13 | chi = xr.children 14 | chi[1..-1].reverse.each do |ch| 15 | xr.parent.insert_after(xr, ch) 16 | end 17 | xr.replace_with(chi[0]) 18 | end 19 | end 20 | irefs.each do |k, v| 21 | sec.insert_after(sec.get_elements("name").first, 22 | e = REXML::Element.new("iref", sec)) 23 | e.attributes["item"] = k 24 | e.attributes["subitem"] = v 25 | e.attributes["primary"] = 'true' 26 | end 27 | end 28 | 29 | end 30 | -------------------------------------------------------------------------------- /lib/kramdown-rfc/command.rb: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env ruby 2 | # -*- coding: utf-8 -*- 3 | require 'kramdown-rfc2629' 4 | require 'kramdown-rfc/parameterset' 5 | require 'kramdown-rfc/refxml' 6 | require 'kramdown-rfc/rfc8792' 7 | require 'yaml' 8 | require 'kramdown-rfc/erb' 9 | require 'date' 10 | 11 | # try to get this from gemspec. 12 | KDRFC_VERSION=Gem.loaded_specs["kramdown-rfc2629"].version rescue "unknown-version" 13 | 14 | Encoding.default_external = "UTF-8" # wake up, smell the coffee 15 | 16 | def add_quote(s) 17 | l = s.lines 18 | l.map {|li| "> #{li}"}.join 19 | end 20 | 21 | def process_chunk(s, nested, dedent, fold, quote) 22 | process_includes(s) if nested 23 | s = remove_indentation(s) if dedent 24 | s = fold8792_1(s, *fold) if fold 25 | s = add_quote(s) if quote 26 | s 27 | end 28 | 29 | def process_includes(input) 30 | input.gsub!(/^\{::include((?:-[a-z0-9]+)*)\s+(.*?)\}/) { 31 | include_flags = $1 32 | fn = [$2] 33 | chunks = false 34 | nested = false 35 | dedent = false 36 | fold = false 37 | quote = false 38 | include_flags.split("-") do |flag| 39 | case flag 40 | when "" 41 | when "nested" 42 | nested = true 43 | when "quote" 44 | quote = true 45 | when "dedent" 46 | dedent = true 47 | when /\Afold(\d*)(left(\d*))?(dry)?\z/ 48 | fold = [$1.to_i, # col 0 for '' 49 | ($3.to_i if $2), # left 0 for '', nil if no "left" 50 | $4] # dry 51 | when "all", "last" 52 | fn = fn.flat_map{|n| Dir[n]} 53 | fn = [fn.last] if flag == "last" 54 | chunks = fn.map{ |f| 55 | ret = process_chunk(File.read(f), nested, dedent, fold, quote) 56 | nested = false; dedent = false; fold = false; quote = false 57 | ret 58 | } 59 | else 60 | warn "** unknown include flag #{flag}" 61 | end 62 | end 63 | chunks = fn.map{|f| File.read(f)} unless chunks # no all/last 64 | chunks = chunks.map {|ch| process_chunk(ch, nested, dedent, fold, quote)} 65 | chunks.join.chomp 66 | } 67 | end 68 | 69 | 70 | def boilerplate(key) 71 | ret = '' 72 | case key.downcase 73 | when /\Abcp14(info)?(\+)?(-tagged)?(-bcp(14)?)?\z/i 74 | # $1 $2 $3 $4 $5 75 | if $1 76 | ret << < 100 | *[MUST NOT]: 101 | *[REQUIRED]: 102 | *[SHALL]: 103 | *[SHALL NOT]: 104 | *[SHOULD]: 105 | *[SHOULD NOT]: 106 | *[RECOMMENDED]: 107 | *[NOT RECOMMENDED]: 108 | *[MAY]: 109 | *[OPTIONAL]: 110 | TAGGED 111 | end 112 | if $4 # experimental; idnits complains: 113 | if $5 114 | # ** The document contains RFC2119-like boilerplate, but doesn't seem to 115 | # mention RFC 2119. The boilerplate contains a reference [BCP14], but that 116 | # reference does not seem to mention RFC 2119 either. 117 | ret.sub!("BCP 14 {{!RFC2119}} {{!RFC8174}}", 118 | "{{!BCP14}} ({{RFC2119}}) ({{RFC8174}})") 119 | else 120 | # ** The document seems to lack a both a reference to RFC 2119 and the 121 | # recommended RFC 2119 boilerplate, even if it appears to use RFC 2119 122 | # keywords -- however, there's a paragraph with a matching beginning. 123 | # Boilerplate error? 124 | ret.sub!("{{!RFC2119}} {{!RFC8174}}", "{{!BCP14}}") 125 | end 126 | end 127 | ret 128 | when /\Arfc\s*7942(info)?\z/i 129 | if $1 130 | ret << < :normative, "?" => :informative } 187 | 188 | def yaml_load(input, *args) 189 | begin 190 | if YAML.respond_to?(:safe_load) 191 | begin 192 | YAML.safe_load(input, *args) 193 | rescue ArgumentError 194 | YAML.safe_load(input, permitted_classes: args[0], permitted_symbols: args[1], aliases: args[2]) 195 | end 196 | else 197 | YAML.load(input) 198 | end 199 | rescue Psych::SyntaxError => e 200 | warn "*** YAML syntax error: #{e}" 201 | exit 65 # EX_DATAERR 202 | end 203 | end 204 | 205 | def process_kramdown_options(coding_override = nil, 206 | smart_quotes = nil, typographic_symbols = nil, 207 | header_kramdown_options = nil) 208 | 209 | ascii_target = coding_override && coding_override =~ /ascii/ 210 | suppress_typography = ascii_target || $options.v3 211 | entity_output = ascii_target ? :numeric : :as_char; 212 | 213 | options = {input: 'RFC2629Kramdown', entity_output: entity_output, link_defs: {}} 214 | 215 | if smart_quotes.nil? && suppress_typography 216 | smart_quotes = false 217 | end 218 | if smart_quotes == false 219 | smart_quotes = ["'".ord, "'".ord, '"'.ord, '"'.ord] 220 | end 221 | case smart_quotes 222 | when Array 223 | options[:smart_quotes] = smart_quotes 224 | when nil, true 225 | # nothin 226 | else 227 | warn "*** Can't deal with smart_quotes value #{smart_quotes.inspect}" 228 | end 229 | 230 | if typographic_symbols.nil? && suppress_typography 231 | typographic_symbols = false 232 | end 233 | if typographic_symbols == false 234 | typographic_symbols = Hash[::Kramdown::Parser::Kramdown::TYPOGRAPHIC_SYMS.map { |k, v| 235 | if Symbol === v 236 | [v.intern, k] 237 | end 238 | }.compact] 239 | end 240 | # warn [:TYPOGRAPHIC_SYMBOLS, typographic_symbols].to_yaml 241 | case typographic_symbols 242 | when Hash 243 | options[:typographic_symbols] = typographic_symbols 244 | when nil, true 245 | # nothin 246 | else 247 | warn "*** Can't deal with typographic_symbols value #{typographic_symbols.inspect}" 248 | end 249 | 250 | if header_kramdown_options 251 | options.merge! header_kramdown_options 252 | end 253 | 254 | $global_markdown_options = options # For nested calls in bibref annotation processing and xref text 255 | 256 | options 257 | end 258 | 259 | XREF_SECTIONS_RE = ::Kramdown::Parser::RFC2629Kramdown::SECTIONS_RE 260 | XSR_PREFIX = "#{XREF_SECTIONS_RE} of " 261 | XSR_SUFFIX = ", (#{XREF_SECTIONS_RE})| \\((#{XREF_SECTIONS_RE})\\)" 262 | XREF_TXT = ::Kramdown::Parser::RFC2629Kramdown::XREF_TXT 263 | XREF_TXT_SUFFIX = " \\(#{XREF_TXT}\\)" 264 | 265 | def spacify_re(s) 266 | s.gsub(' ', '[\u00A0\s]+') 267 | end 268 | 269 | include ::Kramdown::Utils::Html 270 | 271 | # Make this a method so there is a more speaking traceback if this fails 272 | def read_erbfile 273 | erbfilename = ENV["KRAMDOWN_ERB_FILE"] || 274 | File.expand_path('../../../data/kramdown-rfc2629.erb', __FILE__) 275 | File.read(erbfilename, coding: "UTF-8") 276 | end 277 | 278 | def xml_from_sections(input) 279 | 280 | unless ENV["KRAMDOWN_NO_SOURCE"] 281 | require 'kramdown-rfc/gzip-clone' 282 | require 'base64' 283 | compressed_input = Gzip.compress_m0(input) 284 | $source = Base64.encode64(compressed_input) 285 | end 286 | 287 | sections = input.scan(RE_SECTION) 288 | # resulting in an array; each section is [section-label, nomarkdown-flag, section-text] 289 | line = 1 # skip "---" 290 | sections.each do |section| 291 | section << line 292 | line += 1 + section[2].lines.count 293 | end 294 | # warn "#{line-1} lines" 295 | 296 | # the first section is a YAML with front matter parameters (don't put a label here) 297 | # We put back the "---" plus gratuitous blank lines to hack the line number in errors 298 | yaml_in = input[/---\s*/] << sections.shift[2] 299 | begin 300 | require 'kramdown-rfc/yamlcheck' 301 | KramdownRFC::YAMLcheck.check_dup_keys(yaml_in) 302 | rescue => e 303 | warn "** Cannot check for duplicate keys in YAML header (#{e})" 304 | end 305 | ps = KramdownRFC::ParameterSet.new(yaml_load(yaml_in, [Date], [], true)) 306 | 307 | if v = ps[:v] 308 | warn "*** unsupported RFCXML version #{v}" if v != 3 309 | if $options.v2 310 | warn "*** command line --v2 wins over document's 'v: #{v}'" 311 | else 312 | $options.v3 = true 313 | $options.v = 3 314 | ps.default!(:stand_alone, true) 315 | ps.default!(:ipr, "trust200902") 316 | ps.default!(:pi, {"toc" => true, "sortrefs" => true, "symrefs" => true}) 317 | end 318 | end 319 | 320 | if r = ENV["KRAMDOWN_RFC_DOCREV"] 321 | warn "** building document revision -#{r}" 322 | unless n = ps.has(:docname) and n.sub!(/-latest\z/, "-#{r}") 323 | warn "** -d#{r}: docname #{n.inspect} doesn't have a '-latest' suffix" 324 | end 325 | end 326 | 327 | if o = ps[:'autolink-iref-cleanup'] 328 | $options.autolink_iref_cleanup = o 329 | end 330 | if o = ps[:'svg-id-cleanup'] 331 | $options.svg_id_cleanup = o 332 | end 333 | 334 | coding_override = ps.has(:coding) 335 | smart_quotes = ps[:smart_quotes] || ps[:"smart-quotes"] 336 | typographic_symbols = ps[:typographic_symbols] 337 | header_kramdown_options = ps[:kramdown_options] 338 | 339 | kramdown_options = process_kramdown_options(coding_override, 340 | smart_quotes, typographic_symbols, 341 | header_kramdown_options) 342 | 343 | # all the other sections are put in a Hash, possibly concatenated from parts there 344 | sechash = Hash.new{ |h,k| h[k] = ""} 345 | snames = [] # a stack of section names 346 | sections.each do |sname, nmdflag, text, line| 347 | # warn [:SNAME, sname, nmdflag, text[0..10]].inspect 348 | nmdin, nmdout = { 349 | "-" => ["", ""], # stay in nomarkdown 350 | "" => NMDTAGS, # pop out temporarily 351 | }[nmdflag || ""] 352 | if sname 353 | snames << sname # "--- label" -> push label (now current) 354 | else 355 | snames.pop # just "---" -> pop label (previous now current) 356 | end 357 | sechash[snames.last] << "#{nmdin}\n#{text}#{nmdout}" 358 | end 359 | 360 | ref_replacements = { } 361 | anchor_to_bibref = { } 362 | 363 | displayref = {} 364 | 365 | [:ref, :normative, :informative].each do |sn| 366 | if refs = ps.has(sn) 367 | warn "*** bad section #{sn}: #{refs.inspect}" unless refs.respond_to? :each 368 | refs.each do |k, v| 369 | if v.respond_to? :to_str 370 | if bibtagsys(v) # enable "foo: RFC4711" as a custom anchor definition 371 | anchor_to_bibref[k] = v.to_str 372 | end 373 | ref_replacements[v.to_str] = k 374 | end 375 | if Hash === v 376 | if aliasname = v.delete("-") 377 | ref_replacements[aliasname] = k 378 | end 379 | if bibref = v.delete("=") 380 | anchor_to_bibref[k] = bibref 381 | end 382 | if dr = v.delete("display") 383 | displayref[k.gsub("/", "_")] = dr 384 | end 385 | end 386 | end 387 | end 388 | end 389 | open_refs = ps[:ref] || { } # consumed 390 | 391 | norm_ref = { } 392 | 393 | # convenience replacement of {{-coap}} with {{I-D.ietf-core-coap}} 394 | # collect normative/informative tagging {{!RFC2119}} {{?RFC4711}} 395 | sechash.each do |k, v| 396 | next if k == "fluff" 397 | v.gsub!(/{{(#{ 398 | spacify_re(XSR_PREFIX) 399 | })?([\w.\/_\-]+@)?(?:([?!])(-)?|(-))([\w._\-]+)(?:=([\w.\/_\-]+))?(#{ 400 | # 2 3 4 5 6 7 401 | XREF_TXT_SUFFIX 402 | })?(#{ 403 | spacify_re(XSR_SUFFIX) 404 | })?}}/) do |match| 405 | xsr_prefix = $1 406 | subref = $2 407 | norminform = $3 408 | replacing = $4 || $5 409 | word = $6 410 | bibref = $7 411 | xrt_suffix = $8 412 | xsr_suffix = $9 413 | if replacing 414 | if new = ref_replacements[word] 415 | word = new 416 | else 417 | warn "*** no alias replacement for {{-#{word}}}" 418 | word = "-#{word}" 419 | end 420 | end # now, word is the anchor 421 | if bibref 422 | if old = anchor_to_bibref[word] 423 | if bibref != old 424 | warn "*** conflicting definitions for xref #{word}: #{old} != #{bibref}" 425 | end 426 | else 427 | anchor_to_bibref[word] = bibref 428 | end 429 | end 430 | 431 | # things can be normative in one place and informative in another -> normative 432 | # collect norm/inform above and assign it by priority here 433 | if norminform 434 | norm_ref[word] ||= norminform == '!' # one normative ref is enough 435 | end 436 | "{{#{xsr_prefix}#{subref}#{word}#{xrt_suffix}#{xsr_suffix}}}" 437 | end 438 | end 439 | 440 | [:normative, :informative].each do |k| 441 | ps.rest[k.to_s] ||= { } 442 | end 443 | 444 | norm_ref.each do |k, v| 445 | # could check bibtagsys here: needed if open_refs is nil or string 446 | kind = v ? :normative : :informative 447 | target = ps.has(kind) 448 | warn "** (#{kind} reference #{k} is both inline and in YAML header)" if target.has_key?(k) 449 | target[k] = open_refs[k] # add reference to normative/informative 450 | end 451 | # note that unused items from ref are considered OK, therefore no check for that here 452 | 453 | # also should allow norm/inform check of other references 454 | # {{?coap}} vs. {{!coap}} vs. {{-coap}} (undecided) 455 | # or {{?-coap}} vs. {{!-coap}} vs. {{-coap}} (undecided) 456 | # could require all references to be decided by a global flag 457 | overlap = [:normative, :informative].map { |s| (ps.has(s) || { }).keys }.reduce(:&) 458 | unless overlap.empty? 459 | warn "*** #{overlap.join(', ')}: both normative and informative" 460 | end 461 | 462 | stand_alone = ps[:stand_alone] 463 | 464 | [:normative, :informative].each do |sn| 465 | if refs = ps[sn] 466 | refs.each do |k, v| 467 | href = ::Kramdown::Parser::RFC2629Kramdown.idref_cleanup(k) 468 | kramdown_options[:link_defs][k] = ["##{href}", nil] # allow [RFC2119] in addition to {{RFC2119}} 469 | 470 | bibref = anchor_to_bibref[k] || k 471 | bts, url = bibtagsys(bibref, k, stand_alone) 472 | ann = v.delete("annotation") || v.delete("ann") if Hash === v 473 | if bts && (!v || v == {} || v.respond_to?(:to_str)) 474 | if stand_alone 475 | a = %{{: anchor="#{k}"}} 476 | a[-1...-1] = %{ ann="#{escape_html(ann, :attribute)}"} if ann 477 | sechash[sn.to_s] << %{\n#{NMDTAGS[0]}\n![:include:](#{bts})#{a}\n#{NMDTAGS[1]}\n} 478 | else 479 | warn "*** please use standalone mode for adding annotations to references" if ann 480 | bts.gsub!('/', '_') 481 | (ps.rest["bibxml"] ||= []) << [bts, url] 482 | sechash[sn.to_s] << %{&#{bts};\n} # ??? 483 | end 484 | else 485 | if v.nil? && (bri = bibref.to_i) != 0 486 | v = bri 487 | end # hack in {{?Err6543=8610}} 488 | if v && Integer === v 489 | case href 490 | when /\AErr(.*)/ 491 | epno = $1 492 | rfcno = v.to_s 493 | v = { 494 | "target" => "https://www.rfc-editor.org/errata/eid#{epno}", 495 | "title" => "RFC Errata Report #{epno}", 496 | "quote-title" => false, 497 | "seriesinfo" => { "RFC" => rfcno }, 498 | "date" => false 499 | } 500 | else 501 | # superfluous -- would be caught by next "unless" 502 | warn "*** don't know how to expand numeric ref #{k}" 503 | next 504 | end 505 | end 506 | unless v && Hash === v 507 | warn "*** don't know how to expand ref #{k} #{v.inspect}" 508 | next 509 | end 510 | if bts && !v.delete("override") 511 | warn "*** warning: explicit settings completely override canned bibxml in reference #{k}" 512 | end 513 | v["ann"] = ann if ann 514 | sechash[sn.to_s] << KramdownRFC::ref_to_xml(href, v) 515 | end 516 | end 517 | end 518 | end 519 | 520 | erbfile = read_erbfile 521 | erb = ERB.trim_new(erbfile, '-') 522 | # remove redundant nomarkdown pop outs/pop ins as they confuse kramdown 523 | input = erb.result(binding).gsub(%r"{::nomarkdown}\s*{:/nomarkdown}"m, "") 524 | ps.warn_if_leftovers 525 | sechash.delete("fluff") # fluff is a "commented out" section 526 | if !sechash.empty? # any sections unused by the ERb file? 527 | warn "*** sections left #{sechash.keys.inspect}!" 528 | end 529 | 530 | [input, kramdown_options, coding_override] 531 | end 532 | 533 | XML_RESOURCE_ORG_PREFIX = Kramdown::Converter::Rfc2629::XML_RESOURCE_ORG_PREFIX 534 | 535 | # return XML entity name, url, rewrite_anchor flag 536 | def bibtagsys(bib, anchor=nil, stand_alone=true) 537 | if bib =~ /\Arfc(\d+)/i 538 | rfc4d = "%04d" % $1.to_i 539 | [bib.upcase, 540 | "#{XML_RESOURCE_ORG_PREFIX}/bibxml/reference.RFC.#{rfc4d}.xml"] 541 | elsif $options.v3 && bib =~ /\A(bcp|std)(\d+)/i 542 | n4d = "%04d" % $2.to_i 543 | [bib.upcase, 544 | "#{XML_RESOURCE_ORG_PREFIX}/bibxml-rfcsubseries-new/reference.#{$1.upcase}.#{n4d}.xml"] 545 | elsif bib =~ /\A([-A-Z0-9]+)\./ && 546 | (xro = Kramdown::Converter::Rfc2629::XML_RESOURCE_ORG_MAP[$1]) 547 | dir, _ttl, rewrite_anchor = xro 548 | bib1 = ::Kramdown::Parser::RFC2629Kramdown.idref_cleanup(bib) 549 | if anchor && bib1 != anchor 550 | if rewrite_anchor 551 | a = %{?anchor=#{anchor}} 552 | else 553 | if !stand_alone 554 | warn "*** selecting a custom anchor '#{anchor}' for '#{bib1}' requires stand_alone mode" 555 | warn " the output will need manual editing to correct this" 556 | end 557 | end 558 | end 559 | [bib1, 560 | "#{XML_RESOURCE_ORG_PREFIX}/#{dir}/reference.#{bib}.xml#{a}"] 561 | end 562 | end 563 | 564 | def read_encodings 565 | encfilename = File.expand_path '../../../data/encoding-fallbacks.txt', __FILE__ 566 | encfile = File.read(encfilename, coding: "UTF-8") 567 | Hash[encfile.lines.map{|l| 568 | l.chomp!; 569 | x, s = l.split(" ", 2) 570 | [x.hex.chr(Encoding::UTF_8), s || " "]}] 571 | end 572 | 573 | FALLBACK = read_encodings 574 | 575 | def expand_tabs(s, tab_stops = 8) 576 | s.gsub(/([^\t\n]*)\t/) do 577 | $1 + " " * (tab_stops - ($1.size % tab_stops)) 578 | end 579 | end 580 | 581 | 582 | require 'optparse' 583 | require 'ostruct' 584 | 585 | $options ||= OpenStruct.new 586 | op = OptionParser.new do |opts| 587 | opts.banner = < file.xml 589 | Version: #{KDRFC_VERSION} 590 | BANNER 591 | opts.on("-V", "--version", "Show version and exit") do |v| 592 | puts "kramdown-rfc #{KDRFC_VERSION}" 593 | exit 594 | end 595 | opts.on("-H", "--help", "Show option summary and exit") do |v| 596 | puts opts 597 | exit 598 | end 599 | opts.on("-v", "--[no-]verbose", "Run verbosely") do |v| 600 | $options.verbose = v 601 | end 602 | opts.on("-3", "--[no-]v3", "Use RFCXML v3 processing rules") do |v| 603 | $options.v3 = v 604 | end 605 | opts.on("-2", "--[no-]v2", "Use RFCXML v2 processing rules") do |v| 606 | $options.v2 = v 607 | end 608 | end 609 | op.parse! 610 | 611 | if $options.v2 && $options.v3 612 | warn "*** can't have v2 and eat v3 cake" 613 | $options.v2 = false 614 | end 615 | 616 | if $options.v3.nil? && !$options.v2 617 | if Time.now.to_i >= 1645567342 # Time.parse("2022-02-22T22:02:22Z").to_i 618 | $options.v3 = true # new default from the above date 619 | end 620 | end 621 | 622 | warn "*** v2 #{$options.v2.inspect} v3 #{$options.v3.inspect}" if $options.verbose 623 | 624 | input = ARGF.read 625 | input.scrub! do |c| 626 | warn "*** replaced invalid UTF-8 byte sequence #{c.inspect} by U+FFFD REPLACEMENT CHARACTER" 627 | 0xFFFD.chr(Encoding::UTF_8) 628 | end 629 | if input[0] == "\uFEFF" 630 | warn "*** There is a leading byte order mark. Ignored." 631 | input[0..0] = '' 632 | end 633 | if input[-1] != "\n" 634 | # warn "*** added missing newline at end" 635 | input << "\n" # fix #26 636 | end 637 | process_includes(input) unless ENV["KRAMDOWN_SAFE"] 638 | input.gsub!(/^\{::boilerplate\s+(.*?)\}/) { 639 | bp = boilerplate($1) 640 | delta = bp.lines.count 641 | bp + "\n" 642 | } 643 | if input =~ /[\t]/ 644 | warn "*** Input contains HT (\"tab\") characters. Undefined behavior will ensue." 645 | input = expand_tabs(input) 646 | end 647 | 648 | if input =~ /\A---/ # this is a sectionized file 649 | do_the_tls_dance unless ENV["KRAMDOWN_DONT_VERIFY_HTTPS"] 650 | input, options, coding_override = xml_from_sections(input) 651 | else 652 | options = process_kramdown_options # all default 653 | end 654 | if input =~ /\A<\?xml/ # if this is a whole XML file, protect it 655 | input = "{::nomarkdown}\n#{input}\n{:/nomarkdown}\n" 656 | end 657 | 658 | if $options.v3_used && !$options.v3 659 | warn $options.v3_used 660 | $options.v3_used = nil 661 | $options.v3 = true 662 | end 663 | 664 | if coding_override 665 | input = input.encode(Encoding.find(coding_override), fallback: FALLBACK) 666 | end 667 | 668 | # 1.4.17: because of UTF-8 bibxml files, kramdown always needs to see UTF-8 (!) 669 | if input.encoding != Encoding::UTF_8 670 | input = input.encode(Encoding::UTF_8) 671 | end 672 | 673 | # warn "options: #{options.inspect}" 674 | doc = Kramdown::Document.new(input, options) 675 | $stderr.puts doc.warnings.to_yaml unless doc.warnings.empty? 676 | output = doc.to_rfc2629 677 | 678 | if $options.v3_used && !$options.v3 679 | warn $options.v3_used 680 | $options.v3 = true 681 | end 682 | 683 | # only reparse output document if cleanup actions required 684 | if $options.autolink_iref_cleanup || $options.svg_id_cleanup 685 | require 'rexml/document' 686 | 687 | d = REXML::Document.new(output) 688 | d.context[:attribute_quote] = :quote # Set double-quote as the attribute value delimiter 689 | 690 | if $options.autolink_iref_cleanup 691 | require 'kramdown-rfc/autolink-iref-cleanup' 692 | autolink_iref_cleanup(d) 693 | end 694 | 695 | if $options.svg_id_cleanup 696 | require 'kramdown-rfc/svg-id-cleanup' 697 | svg_id_cleanup(d) 698 | end 699 | 700 | output = d.to_s 701 | end 702 | 703 | if coding_override 704 | output = output.encode(Encoding.find(coding_override), fallback: FALLBACK) 705 | end 706 | 707 | puts output 708 | -------------------------------------------------------------------------------- /lib/kramdown-rfc/doi.rb: -------------------------------------------------------------------------------- 1 | require 'open-uri' 2 | require 'json' 3 | require 'yaml' 4 | 5 | ACCEPT_CITE_JSON = {"Accept" => "application/citeproc+json"} 6 | 7 | def doi_fetch_and_convert(doi, fuzzy: false, verbose: false, site: "https://dx.doi.org") 8 | doipath = doi.sub(/^([0-9.]+)_/) {"#$1/"} # convert initial _ back to / 9 | # warn "** SUB #{doi} #{doipath}" if doi != doipath 10 | begin 11 | cite = JSON.parse(URI("#{site}/#{doipath}").open(ACCEPT_CITE_JSON).read) 12 | puts cite.to_yaml if verbose 13 | doi_citeproc_to_lit(cite, fuzzy) 14 | rescue OpenURI::HTTPError => e 15 | begin 16 | site = "https://dl.acm.org" 17 | percent_escaped = doipath.gsub("/", "%2F") 18 | path = "#{site}/action/exportCiteProcCitation?targetFile=custom-bibtex&format=bibTex&dois=#{percent_escaped}" 19 | op = URI(path).open # first get a cookie, ignore result 20 | # warn [:META, op.meta].inspect 21 | cook = op.meta['set-cookie'].split('; ', 2)[0] 22 | cite = JSON.parse(URI(path).open("Cookie" => cook).read) 23 | cite = cite["items"].first[doipath] 24 | puts cite.to_yaml if verbose 25 | doi_citeproc_to_lit(cite, fuzzy) 26 | rescue 27 | raise e 28 | end 29 | end 30 | end 31 | 32 | def doi_citeproc_to_lit(cite, fuzzy) 33 | lit = {} 34 | ser = lit["seriesinfo"] = {} 35 | refcontent = [] 36 | lit["title"] = cite["title"] 37 | if (st = cite["subtitle"]) && Array === st # defensive 38 | st.delete('') 39 | if st != [] 40 | lit["title"] << ": " << st.join("; ") 41 | end 42 | end 43 | if authors = cite["author"] 44 | lit["author"] = authors.map do |au| 45 | lau = {} 46 | if (f = au["family"]) 47 | if (g = au["given"]) 48 | lau["name"] = "#{g} #{f}" 49 | lau["ins"] = "#{g[0]}. #{f}" 50 | else 51 | lau["name"] = "#{f}" 52 | # lau["ins"] = "#{g[0]}. #{f}" 53 | end 54 | end 55 | if (f = au["affiliation"]) && Array === f 56 | names = f.map { |affn| 57 | if Hash === affn && (n = affn["name"]) && String === n 58 | n 59 | end 60 | }.compact 61 | if names.size > 0 62 | lau["org"] = names.join("; ") 63 | end 64 | end 65 | lau 66 | end 67 | end 68 | if iss = cite["issued"] 69 | if dp = iss["date-parts"] 70 | if Integer === (dp = dp[0])[0] 71 | lit["date"] = ["%04d" % dp[0], *dp[1..-1].map {|p| "%02d" % p}].join("-") 72 | end 73 | end 74 | end 75 | if !lit.key?("date") && fuzzy && (iss = cite["created"]) 76 | if dp = iss["date-parts"] 77 | if Integer === (dp = dp[0])[0] 78 | lit["date"] = ["%04d" % dp[0], *dp[1..-1].map {|p| "%02d" % p}].join("-") 79 | end 80 | end 81 | end 82 | if (ct = cite["container-title"]) && ct != [] 83 | info = [] 84 | if v = cite["volume"] 85 | vi = "vol. #{v}" 86 | if (v = cite["journal-issue"]) && (issue = v["issue"]) 87 | vi << ", no. #{issue}" 88 | end 89 | info << vi 90 | end 91 | if p = cite["page"] 92 | info << "pp. #{p}" 93 | end 94 | rhs = info.join(", ") 95 | if info != [] 96 | ser[ct] = rhs 97 | else 98 | spl = ct.split(" ") 99 | ser[spl[0..-2].join(" ")] = spl[-1] 100 | end 101 | end 102 | if pub = cite["publisher"] 103 | refcontent << pub 104 | # info = [] 105 | # if t = cite["type"] 106 | # info << t 107 | # end 108 | # rhs = info.join(", ") 109 | # if info != [] 110 | # ser[pub] = rhs 111 | # else 112 | # spl = pub.split(" ") 113 | # ser[spl[0..-2].join(" ")] = spl[-1] 114 | # end 115 | end 116 | ["DOI", "ISBN"].each do |st| 117 | if a = cite[st] 118 | ser[st] = a 119 | end 120 | end 121 | if refcontent != [] 122 | lit["refcontent"] = refcontent.join(", ") 123 | end 124 | lit 125 | end 126 | -------------------------------------------------------------------------------- /lib/kramdown-rfc/erb.rb: -------------------------------------------------------------------------------- 1 | require 'erb' 2 | 3 | class ERB 4 | 5 | case version.sub("erb.rb [", "") 6 | when /\A2.1/ # works back to 1.9.1 7 | def self.trim_new(s, trim) 8 | ERB.new(s, nil, trim) 9 | end 10 | else 11 | def self.trim_new(s, trim) 12 | ERB.new(s, trim_mode: trim) 13 | end 14 | end 15 | 16 | end 17 | -------------------------------------------------------------------------------- /lib/kramdown-rfc/gzip-clone.rb: -------------------------------------------------------------------------------- 1 | require 'zlib' 2 | require 'stringio' 3 | 4 | # cloned from module ActiveSupport 5 | # A convenient wrapper for the zlib standard library that allows 6 | # compression/decompression of strings with gzip. 7 | # 8 | # gzip = Gzip.compress('compress me!') 9 | # # => "\x1F\x8B\b\x00o\x8D\xCDO\x00\x03K\xCE\xCF-(J-.V\xC8MU\x04\x00R>n\x83\f\x00\x00\x00" 10 | # 11 | # Gzip.decompress(gzip) 12 | # # => "compress me!" 13 | module Gzip 14 | class Stream < StringIO 15 | def initialize(*) 16 | super 17 | set_encoding "BINARY" 18 | end 19 | def close; rewind; end 20 | end 21 | 22 | # Decompresses a gzipped string. 23 | def self.decompress(source) 24 | Zlib::GzipReader.new(StringIO.new(source)).read 25 | end 26 | 27 | # Compresses a string using gzip, setting mtime to 0 28 | def self.compress_m0(source, level=Zlib::DEFAULT_COMPRESSION, strategy=Zlib::DEFAULT_STRATEGY) 29 | output = Stream.new 30 | gz = Zlib::GzipWriter.new(output, level, strategy) 31 | gz.mtime = 0 32 | gz.write(source) 33 | gz.close 34 | output.string 35 | end 36 | end 37 | # end 38 | -------------------------------------------------------------------------------- /lib/kramdown-rfc/kdrfc-processor.rb: -------------------------------------------------------------------------------- 1 | require 'uri' 2 | require 'net/http' 3 | require 'net/http/persistent' 4 | require 'open3' 5 | require 'ostruct' 6 | require 'json' 7 | 8 | module KramdownRFC 9 | 10 | class KDRFC 11 | 12 | attr_reader :options 13 | 14 | def initialize 15 | @options = OpenStruct.new 16 | end 17 | 18 | # ))) 19 | 20 | KDRFC_PREPEND = [ENV["KDRFC_PREPEND"]].compact 21 | KDRFC_XML2RFC_FLAGS = Array(ENV["KDRFC_XML2RFC_FLAGS"]&.split(",")) 22 | 23 | def v3_flag? 24 | [*(@options.v3 ? ["--v3"] : []), 25 | *(@options.v2 ? ["--v2"] : [])] 26 | end 27 | 28 | def process_mkd(input, output) 29 | warn "* converting locally from markdown #{input} to xml #{output}" if @options.verbose 30 | o, s = Open3.capture2(*KDRFC_PREPEND, "kramdown-rfc2629", *v3_flag?, input) 31 | if s.success? 32 | File.open(output, "w") do |fo| 33 | fo.print(o) 34 | end 35 | warn "* #{output} written" if @options.verbose 36 | else 37 | raise IOError.new("*** kramdown-rfc failed, status #{s.exitstatus}") 38 | end 39 | end 40 | 41 | def filename_ct(fn, ext) 42 | bn = File.basename(fn, ".*") 43 | if r = ENV["KRAMDOWN_RFC_DOCREV"] 44 | bn << "-#{r}" 45 | end 46 | {filename: "#{bn}.#{ext}", 47 | content_type: "text/plain"} 48 | end 49 | 50 | def run_idnits(*args) 51 | if @options.remote 52 | run_idnits_remotely(*args) 53 | else 54 | run_idnits_locally(*args) 55 | end 56 | end 57 | 58 | def run_idnits_locally(txt_fn) 59 | warn "* running idnits locally in txt #{txt_fn}" if @options.verbose 60 | unless system("idnits", txt_fn) 61 | warn "*** problem #$? running idnits" if @options.verbose 62 | warn "*** problem running idnits -- falling back to remote idnits processing" 63 | run_idnits_remotely(txt_fn) 64 | end 65 | end 66 | 67 | # curl -s https://author-tools.ietf.org/api/idnits -X POST -F file=@draft-ietf-core-comi.txt -F hidetext=true 68 | IDNITS_WEBSERVICE = ENV["KRAMDOWN_IDNITS_WEBSERVICE"] || 69 | 'https://author-tools.ietf.org/api/idnits' 70 | 71 | def run_idnits_remotely(txt_fn) 72 | url = URI(IDNITS_WEBSERVICE) 73 | req = Net::HTTP::Post.new(url) 74 | form = [["file", File.open(txt_fn), 75 | filename_ct(txt_fn, "txt")], 76 | ["hidetext", "true"]] 77 | diag = ["url/form: ", url, form].inspect 78 | req.set_form(form, 'multipart/form-data') 79 | warn "* requesting idnits at #{url}" if @options.verbose 80 | t0 = Time.now 81 | res = persistent_http.request(url, req) 82 | warn "* elapsed time: #{Time.now - t0}" if @options.verbose 83 | case res 84 | when Net::HTTPBadRequest 85 | result = checked_json(res.body) 86 | raise IOError.new("*** Remote Error: #{result["error"]}") 87 | when Net::HTTPOK 88 | case res.content_type 89 | when 'text/plain' 90 | if res.body == '' 91 | raise IOError.new("*** HTTP response is empty with status #{res.code}, not written") 92 | end 93 | puts res.body 94 | else 95 | warning = "*** HTTP response has unexpected content_type #{res.content_type} with status #{res.code}, #{diag}" 96 | warning << "\n" 97 | warning << res.body 98 | raise IOError.new(warning) 99 | end 100 | else 101 | raise IOError.new("*** HTTP response: #{res.code}, #{diag}") 102 | end 103 | end 104 | 105 | 106 | def process_xml(*args) 107 | if @options.remote 108 | process_xml_remotely(*args) 109 | else 110 | process_xml_locally(*args) 111 | end 112 | end 113 | 114 | def process_xml_locally(input, output, *flags) 115 | warn "* converting locally from xml #{input} to txt #{output}" if @options.verbose 116 | begin 117 | o, s = Open3.capture2(*KDRFC_PREPEND, "xml2rfc", *v3_flag?, *flags, *KDRFC_XML2RFC_FLAGS, input) 118 | puts o 119 | if s.success? 120 | warn "* #{output} written" if @options.verbose 121 | else 122 | raise IOError.new("*** xml2rfc failed, status #{s.exitstatus} (possibly try with -r)") 123 | end 124 | rescue Errno::ENOENT 125 | warn "*** falling back to remote xml2rfc processing (web service)" # if @options.verbose 126 | process_xml_remotely(input, output, *flags) 127 | end 128 | end 129 | 130 | # curl https://author-tools.ietf.org/api/render/text -X POST -F "file=@..." 131 | XML2RFC_WEBSERVICE = ENV["KRAMDOWN_XML2RFC_WEBSERVICE"] || 132 | 'https://author-tools.ietf.org/api/render/' 133 | 134 | MODE_AS_FORMAT = { 135 | "--text" => "text", 136 | "--html" => "html", 137 | "--v2v3" => "xml", 138 | "--pdf" => "pdf", 139 | } 140 | 141 | def checked_json(t) 142 | begin 143 | JSON.load(t) 144 | rescue => e 145 | raise IOError.new("*** JSON result: #{e.detailed_message}, #{diag}") 146 | end 147 | end 148 | 149 | def persistent_http 150 | $http ||= Net::HTTP::Persistent.new name: 'kramdown-rfc' 151 | end 152 | 153 | def process_xml_remotely(input, output, *flags) 154 | 155 | format = flags[0] || "--text" 156 | warn "* converting remotely from xml #{input} to #{format} #{output}" if @options.verbose 157 | maf = MODE_AS_FORMAT[format] 158 | unless maf 159 | raise ArgumentError.new("*** don't know how to convert remotely from xml #{input} to #{format} #{output}") 160 | end 161 | url = URI(XML2RFC_WEBSERVICE + maf) 162 | req = Net::HTTP::Post.new(url) 163 | form = [["file", File.open(input), 164 | filename_ct(input, "xml")]] 165 | diag = ["url/form: ", url, form].inspect 166 | req.set_form(form, 'multipart/form-data') 167 | warn "* requesting at #{url}" if @options.verbose 168 | t0 = Time.now 169 | res = persistent_http.request(url, req) 170 | warn "* elapsed time: #{Time.now - t0}" if @options.verbose 171 | case res 172 | when Net::HTTPBadRequest 173 | result = checked_json(res.body) 174 | raise IOError.new("*** Remote Error: #{result["error"]}") 175 | when Net::HTTPOK 176 | case res.content_type 177 | when 'application/json' 178 | if res.body == '' 179 | raise IOError.new("*** HTTP response is empty with status #{res.code}, not written") 180 | end 181 | # warn "* res.body #{res.body}" if @options.verbose 182 | result = checked_json(res.body) 183 | if logs = result["logs"] 184 | if errors = logs["errors"] 185 | errors.each do |err| 186 | warn("*** Error: #{err}") 187 | end 188 | end 189 | if warnings = logs["warnings"] 190 | warnings.each do |w| 191 | warn("** Warning: #{w}") 192 | end 193 | end 194 | end 195 | raise IOError.new("*** No useful result from remote") unless result["url"] 196 | res = persistent_http.request(URI(result["url"])) 197 | warn "* result content type #{res.content_type}" if @options.verbose 198 | if res.body == '' 199 | raise IOError.new("*** Second HTTP response is empty with status #{res.code}, not written") 200 | end 201 | File.open(output, "w") do |fo| 202 | fo.print(res.body) 203 | end 204 | warn "* #{output} written" if @options.verbose 205 | else 206 | warning = "*** HTTP response has unexpected content_type #{res.content_type} with status #{res.code}, #{diag}" 207 | warning << "\n" 208 | warning << res.body 209 | raise IOError.new(warning) 210 | end 211 | else 212 | raise IOError.new("*** HTTP response: #{res.code}, #{diag}") 213 | end 214 | end 215 | 216 | def process_the_xml(fn, base) 217 | process_xml(fn, "#{base}.prepped.xml", "--preptool") if @options.prep 218 | process_xml(fn, "#{base}.v2v3.xml", "--v2v3") if @options.v2v3 219 | process_xml(fn, "#{base}.txt") if @options.txt || @options.idnits 220 | process_xml(fn, "#{base}.html", "--html") if @options.html 221 | process_xml(fn, "#{base}.pdf", "--pdf") if @options.pdf 222 | run_idnits("#{base}.txt") if @options.idnits 223 | end 224 | 225 | def process(fn) 226 | case fn 227 | when /(.*)\.xml\z/ 228 | if @options.xml_only 229 | warn "*** You already have XML" 230 | else # FIXME: copy/paste 231 | process_the_xml(fn, $1) 232 | end 233 | when /(.*)\.mk?d\z/ 234 | xml = "#$1.xml" 235 | process_mkd(fn, xml) 236 | process_the_xml(xml, $1) unless @options.xml_only 237 | when /(.*)\.txt\z/ 238 | run_idnits(fn) if @options.idnits 239 | else 240 | raise ArgumentError.new("Unknown file type: #{fn}") 241 | end 242 | end 243 | 244 | # ((( 245 | end 246 | 247 | end 248 | -------------------------------------------------------------------------------- /lib/kramdown-rfc/parameterset.rb: -------------------------------------------------------------------------------- 1 | module KramdownRFC 2 | 3 | class ParameterSet 4 | include Kramdown::Utils::Html 5 | 6 | attr_reader :f 7 | def initialize(y) 8 | raise "*** invalid parameter set #{y.inspect}" unless Hash === y 9 | @f = y 10 | @av = {} 11 | end 12 | attr :av 13 | def [](pn) 14 | @f.delete(pn.to_s) 15 | end 16 | def []=(pn, val) 17 | @f[pn.to_s] = val 18 | end 19 | def default(pn, &block) 20 | @f.fetch(pn.to_s, &block) 21 | end 22 | def default!(pn, value) 23 | default(pn) { 24 | @f[pn.to_s] = value 25 | } 26 | end 27 | def has(pn) 28 | @f[pn.to_s] 29 | end 30 | def has?(pn) 31 | @f.key?(pn.to_s) 32 | end 33 | def escattr(str) 34 | escape_html(str.to_s, :attribute) 35 | end 36 | def van(pn) # pn is a parameter name, possibly with =aliases 37 | names = pn.to_s.split("=") 38 | [self[names.reverse.find{|nm| has?(nm)}], names.first] 39 | end 40 | def attr(pn) 41 | val, an = van(pn) 42 | @av[an.intern] = val 43 | %{#{an}="#{escattr(val)}"} if val # see attrtf below 44 | end 45 | def attrs(*pns) 46 | pns.map{ |pn| attr(pn) if pn }.compact.join(" ") 47 | end 48 | def attrtf(pn) # can do an overriding false value 49 | val, an = van(pn) 50 | @av[an.intern] = val 51 | %{#{an}="#{escattr(val)}"} unless val.nil? 52 | end 53 | def attrstf(*pns) 54 | pns.map{ |pn| attrtf(pn) if pn }.compact.join(" ") 55 | end 56 | def ele(pn, attr=nil, defcontent=nil, markdown=false) 57 | val, an = van(pn) 58 | val ||= defcontent 59 | val = [val] if Hash === val 60 | Array(val).map do |val1| 61 | a = Array(attr).dup 62 | if Hash === val1 63 | val1.each do |k, v| 64 | if k == ":" 65 | val1 = v 66 | else 67 | k = Kramdown::Element.attrmangle(k) || k 68 | a.unshift(%{#{k}="#{escattr(v)}"}) 69 | end 70 | end 71 | end 72 | v = val1.to_s.strip 73 | contents = 74 | if markdown 75 | ::Kramdown::Converter::Rfc2629::process_markdown(v) 76 | else 77 | escape_html(v) 78 | end 79 | %{<#{[an, *a.map(&:to_s)].join(" ").strip}>#{contents}} 80 | end.join(" ") 81 | end 82 | def arr(an, converthash=true, must_have_one=false, &block) 83 | arr = self[an] || [] 84 | arr = [arr] if Hash === arr && converthash 85 | arr << { } if must_have_one && arr.empty? 86 | Array(arr).each(&block) 87 | end 88 | def rest 89 | @f 90 | end 91 | def warn_if_leftovers 92 | if !@f.empty? 93 | warn "*** attributes left #{@f.inspect}!" 94 | end 95 | end 96 | end 97 | 98 | end 99 | -------------------------------------------------------------------------------- /lib/kramdown-rfc/refxml.rb: -------------------------------------------------------------------------------- 1 | require 'kramdown-rfc/erb' 2 | 3 | module KramdownRFC 4 | 5 | extend Kramdown::Utils::Html 6 | 7 | def self.escattr(str) 8 | escape_html(str.to_s, :attribute) 9 | end 10 | 11 | AUTHOR_ATTRIBUTES = %w{ 12 | initials surname fullname 13 | asciiInitials asciiSurname asciiFullname 14 | role 15 | } 16 | 17 | def self.ref_to_xml(k, v) 18 | vps = KramdownRFC::ParameterSet.new(v) 19 | erb = ERB.trim_new <<-REFERB, '-' 20 | > 22 | 23 | <%= vps.ele("title") -%> 24 | 25 | <% vps.arr("author", true, true) do |au| 26 | aups = authorps_from_hash(au) 27 | -%> 28 | > 29 | <%= aups.ele("organization=org", aups.attr("abbrev=orgabbrev"), "") %> 30 | 31 | <% aups.warn_if_leftovers -%> 32 | <% end -%> 33 | /> 34 | 35 | <% vps.arr("seriesinfo", false) do |k, v| -%> 36 | 37 | <% end -%> 38 | <% vps.arr("format", false) do |k, v| -%> 39 | 40 | <% end -%> 41 | <%= vps.ele("annotation=ann", nil, nil, true) -%> 42 | <%= vps.ele("refcontent=rc", nil, nil, true) -%> 43 | 44 | REFERB 45 | ret = erb.result(binding) 46 | vps.warn_if_leftovers 47 | ret 48 | end 49 | 50 | def self.treat_multi_attribute_member(ps, an) 51 | value = ps.rest[an] 52 | if Hash === value 53 | value.each do |k, v| 54 | ps.rest[if k == ':' 55 | an 56 | else 57 | Kramdown::Element.attrmangle(k + an) || 58 | Kramdown::Element.attrmangle(k) || 59 | k 60 | end] = v 61 | end 62 | end 63 | end 64 | 65 | def self.initializify(s) # XXX Jean-Pierre 66 | w = '\p{Lu}\p{Lo}' 67 | if s =~ /\A[-.#{w}]+[.]/u 68 | $& 69 | elsif s =~ /\A([#{w}])[^-]*/u 70 | ret = "#$1." 71 | while (s = $') && s =~ /\A(-[\p{L}])[^-]*/u 72 | ret << "#$1." 73 | end 74 | ret 75 | else 76 | warn "*** Can't initializify #{s}" 77 | s 78 | end 79 | end 80 | 81 | def self.looks_like_initial(s) 82 | s =~ /\A[\p{Lu}\p{Lo}]([-.][\p{Lu}\p{Lo}]?)*\z/u 83 | end 84 | 85 | def self.initials_from_parts_and_surname(aups, parts, s) 86 | ssz = s.size 87 | nonsurname = parts[0...-ssz] 88 | if (ns = parts[-ssz..-1]) != s 89 | warn "*** inconsistent surnames #{ns} and #{s}" 90 | end 91 | nonsurname.map{|x| initializify(x)}.join(" ") 92 | end 93 | 94 | def self.handle_ins(aups, ins_k, initials_k, surname_k) 95 | if ins = aups[ins_k] 96 | parts = ins.split('.').map(&:strip) # split on dots first 97 | if parts == [] 98 | warn "*** an empty '#{ins_k}:' value is not useful, try leaving it out" 99 | return 100 | end 101 | # Coalesce H.-P. 102 | i = 1; while i < parts.size 103 | if parts[i][0] == "-" 104 | parts[i-1..i] = [parts[i-1] + "." + parts[i]] 105 | else 106 | i += 1 107 | end 108 | end 109 | # Multiple surnames in ins? 110 | parts[-1..-1] = parts[-1].split 111 | s = if surname = aups.rest[surname_k] 112 | surname.split 113 | else parts.reverse.take_while{|x| !looks_like_initial(x)}.reverse 114 | end 115 | aups.rest[initials_k] = initials_from_parts_and_surname(aups, parts, s) 116 | aups.rest[surname_k] = s.join(" ") 117 | end 118 | end 119 | 120 | def self.handle_name(aups, fn_k, initials_k, surname_k) 121 | if name = aups.rest[fn_k] 122 | names = name.split(/ *\| */, 2) # boundary for given/last name 123 | if names == [] 124 | warn "*** an empty '#{fn_k}:' value is not useful, try leaving it out" 125 | return 126 | end 127 | if names[1] 128 | aups.rest[fn_k] = name = names.join(" ") # remove boundary 129 | if surname = aups.rest[surname_k] 130 | if surname != names[1] 131 | warn "*** inconsistent embedded surname #{names[1]} and surname #{surname}" 132 | end 133 | end 134 | aups.rest[surname_k] = names[1] 135 | end 136 | parts = name.split 137 | if parts == [] 138 | warn "*** a blank '#{fn_k}:' value is not useful, try leaving it out" 139 | return 140 | end 141 | surname = aups.rest[surname_k] || parts[-1] 142 | s = surname.split 143 | aups.rest[initials_k] ||= initials_from_parts_and_surname(aups, parts, s) 144 | aups.rest[surname_k] = s.join(" ") 145 | end 146 | end 147 | 148 | def self.authorps_from_hash(au) 149 | aups = KramdownRFC::ParameterSet.new(au) 150 | if n = aups[:name] 151 | warn "** both name #{n} and fullname #{fn} are set on one author" if fn = aups.rest["fullname"] 152 | aups.rest["fullname"] = n 153 | usename = true 154 | end 155 | ["fullname", "ins", "initials", "surname"].each do |an| 156 | treat_multi_attribute_member(aups, an) 157 | end 158 | handle_ins(aups, :ins, "initials", "surname") 159 | handle_ins(aups, :asciiIns, "asciiInitials", "asciiSurname") 160 | # hack ("heuristic for") initials and surname from name 161 | # -- only works for people with exactly one last name and uncomplicated first names 162 | # -- add surname for people with more than one last name 163 | if usename 164 | handle_name(aups, "fullname", "initials", "surname") 165 | handle_name(aups, "asciiFullname", "asciiInitials", "asciiSurname") 166 | end 167 | aups 168 | end 169 | 170 | # The below anticipates the "postalLine" changes. 171 | # If a postalLine is used (abbreviated "postal" in YAML), 172 | # non-postalLine elements are appended as further postalLines. 173 | # This prepares for how "country" is expected to be handled 174 | # specially with the next schema update. 175 | # So an address is now best keyboarded as: 176 | # postal: 177 | # - Foo Street 178 | # - 28359 Bar 179 | # country: Germany 180 | 181 | PERSON_ERB = <<~ERB 182 | <<%= element_name%> <%=aups.attrs(*AUTHOR_ATTRIBUTES)%>> 183 | <%= aups.ele("organization=org", aups.attrs("abbrev=orgabbrev", 184 | *[$options.v3 && "ascii=orgascii"]), "") %> 185 |
186 | <% postal_elements = %w{extaddr pobox street cityarea city region code sortingcode country postal}.select{|gi| aups.has(gi)} 187 | if postal_elements != [] -%> 188 | 189 | <% if pl = postal_elements.delete("postal") -%> 190 | <%= aups.ele("postalLine=postal") %> 191 | <% postal_elements.each do |gi| -%> 192 | <%= aups.ele("postalLine=" << gi) %> 193 | <% end -%> 194 | <% else -%> 195 | <% postal_elements.each do |gi| -%> 196 | <%= aups.ele(gi) %> 197 | <% end -%> 198 | <% end -%> 199 | 200 | <% end -%> 201 | <% %w{phone facsimile email uri}.select{|gi| aups.has(gi)}.each do |gi| -%> 202 | <%= aups.ele(gi) %> 203 | <% end -%> 204 |
205 | > 206 | ERB 207 | 208 | def self.person_element_from_aups(element_name, aups) 209 | erb = ERB.trim_new(PERSON_ERB, '-') 210 | erb.result(binding) 211 | end 212 | 213 | def self.dateattrs(date) 214 | begin 215 | case date 216 | when /\A\d\d\d\d\z/ 217 | %{year="#{date}"} 218 | when Integer 219 | %{year="#{"%04d" % date}"} 220 | when String 221 | Date.parse("#{date}-01").strftime(%{year="%Y" month="%B"}) 222 | when Date 223 | date.strftime(%{year="%Y" month="%B" day="%d"}) 224 | when Array # this allows to explicitly give a string 225 | %{year="#{date.join(" ")}"} 226 | when nil 227 | %{year="n.d."} 228 | end 229 | 230 | rescue ArgumentError 231 | warn "*** Invalid date: #{date} -- use 2012, 2012-07, or 2012-07-28" 232 | end 233 | end 234 | end 235 | -------------------------------------------------------------------------------- /lib/kramdown-rfc/rexml-all-text.rb: -------------------------------------------------------------------------------- 1 | require 'rexml/document' 2 | 3 | module REXML 4 | # all_text: Get all text from descendants that are Text or CData 5 | class Element 6 | def all_text 7 | @children.map {|c| c.all_text}.join 8 | end 9 | end 10 | class Text # also: ancestor of CData 11 | def all_text 12 | value 13 | end 14 | end 15 | class Child 16 | def all_text 17 | '' 18 | end 19 | end 20 | end 21 | -------------------------------------------------------------------------------- /lib/kramdown-rfc/rexml-formatters-conservative.rb: -------------------------------------------------------------------------------- 1 | require 'rexml/document' 2 | 3 | module REXML 4 | module Formatters 5 | # The Conservative formatter writes an XML document that parses to an 6 | # identical document as the source document. This means that no extra 7 | # whitespace nodes are inserted, and whitespace within text nodes is 8 | # preserved. Attributes are not sorted. 9 | class Conservative < Default 10 | def initialize 11 | @indentation = 0 12 | @level = 0 13 | @ie_hack = false 14 | end 15 | 16 | protected 17 | def write_element( node, output ) 18 | output << "<#{node.expanded_name}" 19 | 20 | node.attributes.each_attribute do |attr| 21 | output << " " 22 | attr.write( output ) 23 | end unless node.attributes.empty? 24 | 25 | if node.children.empty? 26 | output << "/" 27 | else 28 | output << ">" 29 | node.children.each { |child| 30 | write( child, output ) 31 | } 32 | output << "" 35 | end 36 | end 37 | end 38 | end 39 | -------------------------------------------------------------------------------- /lib/kramdown-rfc/rfc8792.rb: -------------------------------------------------------------------------------- 1 | 2 | # Note that this doesn't attempt to handle HT characters 3 | def remove_indentation(s) 4 | l = s.lines 5 | indent = l.grep(/\S/).map {|l| l[/^\s*/].size}.min 6 | l.map {|li| li.sub(/^ {0,#{indent}}/, "")}.join 7 | end 8 | 9 | def trim_empty_lines_around(s) # this deletes the trailing newline, which may need to be reconstructed 10 | s.sub(/\A(\r?\n)*/, '').sub(/(\r?\n)*\z/, '') 11 | end 12 | 13 | def fix_unterminated_line(s) 14 | s.sub(/[^\n]\z/) { "#$&\n" } # XXX 15 | end 16 | 17 | def handle_artwork_sourcecode(s, unfold = true) 18 | s = trim_empty_lines_around(s) 19 | s = unfold8792(s) if unfold 20 | fix_unterminated_line(s) 21 | end 22 | 23 | FOLD_MSG = "NOTE: '\\' line wrapping per RFC 8792".freeze 24 | UNFOLD_RE = /\A.*#{FOLD_MSG.sub("\\", "(\\\\\\\\\\\\\\\\?)")}.*\n\r?\n/ 25 | 26 | def unfold8792(s) 27 | if s =~ UNFOLD_RE 28 | indicator = $1 29 | s = $' 30 | sub = case indicator 31 | when "\\" 32 | s.gsub!(/\\\n[ \t]*/, '') 33 | when "\\\\" 34 | s.gsub!(/\\\n[ \t]*\\/, '') 35 | else 36 | fail "indicator" # Cannot happen 37 | end 38 | warn "** encountered RFC 8792 header without folded lines" unless sub 39 | end 40 | s 41 | end 42 | 43 | MIN_FOLD_COLUMNS = FOLD_MSG.size 44 | FOLD_COLUMNS = 69 45 | RE_IDENT = /\A[A-Za-z0-9_]\z/ 46 | 47 | def fold8792_1(s, columns = FOLD_COLUMNS, left = false, dry = false) 48 | if s.index("\t") 49 | warn "*** HT (\"TAB\") in text to be folded. Giving up." 50 | return s 51 | end 52 | if columns < MIN_FOLD_COLUMNS 53 | columns = 54 | if columns == 0 55 | FOLD_COLUMNS 56 | else 57 | warn "*** folding to #{MIN_FOLD_COLUMNS}, not #{columns}" 58 | MIN_FOLD_COLUMNS 59 | end 60 | end 61 | 62 | lines = s.lines.map(&:chomp) 63 | did_fold = false 64 | ix = 0 65 | while li = lines[ix] 66 | col = columns 67 | if li[col].nil? 68 | if li[-1] == "\\" 69 | lines[ix..ix] = [li << "\\", ""] 70 | ix += 1 71 | end 72 | ix += 1 73 | else 74 | did_fold = true 75 | min_indent = left || 0 76 | col -= 1 # space for "\\" 77 | while li[col] == " " # can't start new line with " " 78 | col -= 1 79 | end 80 | if col <= min_indent 81 | warn "*** Cannot RFC8792-fold1 to #{columns} cols #{"with indent #{left}" if left} |#{li.inspect}|" 82 | else 83 | if RE_IDENT === li[col] # Don't split IDs 84 | col2 = col 85 | while col2 > min_indent && RE_IDENT === li[col2-1] 86 | col2 -= 1 87 | end 88 | if col2 > min_indent 89 | col = col2 90 | end 91 | end 92 | rest = li[col..-1] 93 | indent = left || columns - rest.size 94 | if !left && li[-1] == "\\" 95 | indent -= 1 # leave space for next round 96 | end 97 | if indent > 0 98 | rest = " " * indent + rest 99 | end 100 | lines[ix..ix] = [li[0...col] << "\\", rest] 101 | end 102 | ix += 1 103 | end 104 | end 105 | 106 | if did_fold 107 | msg = FOLD_MSG.dup 108 | if !dry && columns >= msg.size + 4 109 | delta = columns - msg.size - 2 # 2 spaces 110 | half = delta/2 111 | msg = "#{"=" * half} #{msg} #{"=" * (delta - half)}" 112 | end 113 | lines[0...0] = [msg, ""] 114 | lines.map{|x| x << "\n"}.join 115 | else 116 | s 117 | end 118 | end 119 | -------------------------------------------------------------------------------- /lib/kramdown-rfc/svg-id-cleanup.rb: -------------------------------------------------------------------------------- 1 | require 'rexml/document' 2 | 3 | SVG_NAMESPACES = {"svg"=>"http://www.w3.org/2000/svg", 4 | "xlink"=>"http://www.w3.org/1999/xlink"} 5 | 6 | def svg_id_cleanup(d) 7 | gensym = "gensym000" 8 | 9 | REXML::XPath.each(d.root, "//svg:svg", SVG_NAMESPACES) do |x| 10 | gensym = gensym.succ 11 | # warn "*** SVG" 12 | # warn "*** SVG: #{x.to_s.size}" 13 | found_as_id = Set[] 14 | found_as_href = Set[] 15 | REXML::XPath.each(x, ".//*[@id]", SVG_NAMESPACES) do |y| 16 | # warn "*** ID: #{y}" 17 | name = y.attributes["id"] 18 | if found_as_id === name 19 | warn "*** duplicate ID #{name}" 20 | end 21 | found_as_id.add(name) 22 | y.attributes["id"] = "#{name}-#{gensym}" 23 | end 24 | REXML::XPath.each(x, ".//*[@xlink:href]", SVG_NAMESPACES) do |y| 25 | # warn "*** HREF: #{y}" 26 | name = y.attributes["href"] 27 | name1 = name[1..-1] 28 | if !found_as_id === name1 29 | warn "*** unknown HREF #{name}" 30 | end 31 | found_as_href.add(name1) 32 | y.attributes["xlink:href"] = "#{name}-#{gensym}" 33 | end 34 | found_as_id -= found_as_href 35 | warn "*** warning: unused ID: #{found_as_id.to_a.join(", ")}" unless found_as_id.empty? 36 | end 37 | rescue => detail 38 | warn "*** Can't clean SVG: #{detail}" 39 | end 40 | -------------------------------------------------------------------------------- /lib/kramdown-rfc/yamlcheck.rb: -------------------------------------------------------------------------------- 1 | require 'yaml' 2 | 3 | module KramdownRFC 4 | module YAMLcheck 5 | 6 | def self.short_name(node) 7 | if node.scalar? 8 | node.value 9 | else 10 | node.children&.map {|nm| short_name(nm)}&.join("_") 11 | end 12 | end 13 | 14 | # Does not follow aliases. 15 | def self.check_dup_keys1(node, path) 16 | if YAML::Nodes::Mapping === node 17 | children = node.children.each_slice(2) 18 | duplicates = children.map { |key_node, _value_node| 19 | key_node }.group_by{|nm| short_name(nm)}.select { |_value, nodes| nodes.size > 1 } 20 | 21 | duplicates.each do |key, nodes| 22 | name = (path + [key]).join("/") 23 | lines = nodes.map { |occurrence| occurrence.start_line + 1 }.join(", ") 24 | warn "** duplicate map key >#{name}< in YAML, lines #{lines}" 25 | end 26 | 27 | children.each do |key_node, value_node| 28 | newname = short_name(key_node) 29 | check_dup_keys1(value_node, path + Array(newname)) 30 | end 31 | else 32 | node.children.to_a.each { |child| check_dup_keys1(child, path) } 33 | end 34 | end 35 | 36 | def self.check_dup_keys(data) 37 | ast = YAML.parse_stream(data) 38 | check_dup_keys1(ast, []) 39 | end 40 | 41 | # check_dup_keys(DATA) 42 | 43 | end 44 | end 45 | --------------------------------------------------------------------------------