├── .github └── workflows │ ├── generated-pr.yml │ └── stale.yml ├── LICENSE ├── README.md ├── img ├── xtp-notes.001.jpg ├── xtp-notes.002.jpg ├── xtp-notes.003.jpg ├── xtp-notes.004.jpg ├── xtp-notes.005.jpg ├── xtp-notes.006.jpg ├── xtp-notes.007.jpg └── xtp-notes.key └── xtp-ctl.md /.github/workflows/generated-pr.yml: -------------------------------------------------------------------------------- 1 | name: Close Generated PRs 2 | 3 | on: 4 | schedule: 5 | - cron: '0 0 * * *' 6 | workflow_dispatch: 7 | 8 | permissions: 9 | issues: write 10 | pull-requests: write 11 | 12 | jobs: 13 | stale: 14 | uses: ipdxco/unified-github-workflows/.github/workflows/reusable-generated-pr.yml@v1 15 | -------------------------------------------------------------------------------- /.github/workflows/stale.yml: -------------------------------------------------------------------------------- 1 | name: Close Stale Issues 2 | 3 | on: 4 | schedule: 5 | - cron: '0 0 * * *' 6 | workflow_dispatch: 7 | 8 | permissions: 9 | issues: write 10 | pull-requests: write 11 | 12 | jobs: 13 | stale: 14 | uses: ipdxco/unified-github-workflows/.github/workflows/reusable-stale-issue.yml@v1 15 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | The MIT License (MIT) 2 | 3 | Copyright (c) 2016 Protocol Labs, Inc. 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in 13 | all copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN 21 | THE SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # XTP - eXternal Transports Protocol 2 | 3 | > Let's put exotic network transports outside the main process. 4 | > That way we can use implementations in other languages easily. 5 | > And isolate, sandbox, and resource constrain unproven implementations. 6 | 7 | ## Motivation 8 | 9 | External Transports are [libp2p network Transport](https://libp2p.io/transports) implementations provided as separate processes. This makes sense because: 10 | 11 | - Solid implementations for most transport protocol are scarce, should be able to use the best. 12 | - The 1 or 2 battle-tested implementations are written in a language different than your client app. 13 | - Thus trickier to use in the same process (even C is annoying in Go). 14 | - You are [writing a network stack in many languages](https://libp2p.io) 15 | - Want to offer a good implementation of the transport as a baseline to all languages (eg. use libquic from Go until go-quic is made...) 16 | - The transport must run differently: 17 | - The app runtime cannot run the language (eg javascript in a browser). 18 | - The app is in a different machine than the transport network interface. 19 | - May want to isolate the transport implementation because: 20 | - Want to constrain its resources, separately from the main process. Can jail/containerize it. 21 | - Maybe it is buggy or "not so trusted". 22 | - Should be able to crash and restart without crashing your main process. 23 | - Can sandbox it to avoid exploits. 24 | 25 | XTP is a protocol and tool suite for working with External Transports. 26 | 27 | ## Illustrations 28 | 29 | ![](img/xtp-notes.001.jpg) 30 | --- 31 | ![](img/xtp-notes.002.jpg) 32 | --- 33 | ![](img/xtp-notes.003.jpg) 34 | --- 35 | ![](img/xtp-notes.004.jpg) 36 | --- 37 | ![](img/xtp-notes.005.jpg) 38 | --- 39 | ![](img/xtp-notes.006.jpg) 40 | 41 | 42 | ## How is it built? 43 | 44 | #### The XTP project combines: 45 | 46 | - an RPC and [multiformats](https://github.com/multiformats) duplex stream protocol, called [xtp-ctl](./xtp-ctl.md). xtp-ctl is used to "control" the external transport implementation. 47 | - xtp-lib is a library of components, iplemented in various languages, including: 48 | - XTP Service - external transport wrapper 49 | - XTP Client - XTP Service bindings 50 | 51 | #### Component Diagram 52 | 53 | ![](img/xtp-notes.007.jpg) 54 | 55 | 56 | #### XTP is built with: 57 | 58 | - [multiformats](https://github.com/multiformats) for self-describing protocols 59 | - [multistream](https://github.com/multiformats/multistream) for connection negotiation 60 | - [multiaddr](https://github.com/multiformats/multiaddr) for endpoint addressing 61 | - the [libp2p](https://github.com/libp2p) [transport interface and modules](https://libp2p.io/transports) 62 | 63 | 64 | ## Project 65 | 66 | ### About 67 | 68 | XTP spun out of the [IPFS Project](https://github.com/ipfs/ipfs). 69 | 70 | ## Roadmap 71 | 72 | Implementation Roadmap: 73 | 74 | - [x] Define Use Cases and Constraints 75 | - [x] Define Architecture 76 | - [x] Chart out components 77 | - [x] Write readme explaining design 78 | - [ ] Write Specs 79 | - [ ] Write xtp-ctl spec (pipe + protocol) 80 | - [ ] Write XTP Service spec 81 | - [ ] Write XTP Client spec 82 | - [ ] Implement XTP in common languages 83 | - [ ] Implement go-xtp 84 | - [ ] Implement xtl-ctl in Go 85 | - [ ] Implement XTP Service in Go 86 | - [ ] Implement XTP Client bindings in Go 87 | - [ ] Implement js-xtp 88 | - [ ] Implement xtl-ctl in JS 89 | - [ ] Implement XTP Service bindings in JS 90 | - [ ] Implement XTP Client bindings in JS 91 | - [ ] Implement rust-xtp 92 | - [ ] Implement xtl-ctl in Rust 93 | - [ ] Implement XTP Service bindings in Rust 94 | - [ ] Implement XTP Client bindings in Rust 95 | - [ ] XTPify some transports 96 | - [ ] xtp-utp, using C++ libutp 97 | - [ ] xtp-quic, using C++ libquic 98 | - [ ] xtp-udt, using C++ libudt 99 | - [ ] xtp-webrtc, using node-webrtc? 100 | - [ ] xtp-ble 101 | 102 | 103 | ## Contribute 104 | 105 | Please contribute! Pull requests very much accepted. 106 | 107 | Please read, and abide by: 108 | - The [IPFS Project Contribution Guidelines](https://github.com/ipfs/community/blob/master/contribution-guidelines.md) 109 | - The [IPFS Project Code of Conduct](https://github.com/ipfs/community/blob/master/code-of-conduct.md) 110 | 111 | ## FAQ 112 | 113 | #### Isn't this much slower than in the same process? 114 | 115 | The data may cross the kernel again to move from the External Transport to the main process and vice versa. This additional kernel boundary passes will slow down the transport, adding syscall latencies. 116 | 117 | For extremely low latency applications this will be unacceptable. But in practice, most applications will not suffer much because: 118 | 119 | - External transports enable us to get implementations working right away. This can be huge in terms of development and deployment, shaving weeks off development, or enabling a transport that wouldn't otherwise be available at all. 120 | - Can optimize by creating a language-specific implementation of the transport later, once you prove the external process is the bottleneck, and you're sure the implementation is sound. 121 | - For most simple apps, the bottlenecks are not in crossing kernel boundaries, they are in the network latencies, storage media I/O, or data processing. 122 | - Good memory sharing or IPC implementations can be leveraged to improve or completely solve this slow down. 123 | 124 | So, don't worry about this until you PROVE it's a problem. Then, go ahead, write an implementation of the transport in your preferred language. 125 | 126 | #### Why the name XTP? can we change it? 127 | 128 | Sure, find something better. Maybe 129 | 130 | - Exo-Transports 131 | - XTransport 132 | - External Transports 133 | - (ETP) External Transport Protocol 134 | -------------------------------------------------------------------------------- /img/xtp-notes.001.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/libp2p/xtp/833c3427f83929d9650764f90f1c316cd62871e9/img/xtp-notes.001.jpg -------------------------------------------------------------------------------- /img/xtp-notes.002.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/libp2p/xtp/833c3427f83929d9650764f90f1c316cd62871e9/img/xtp-notes.002.jpg -------------------------------------------------------------------------------- /img/xtp-notes.003.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/libp2p/xtp/833c3427f83929d9650764f90f1c316cd62871e9/img/xtp-notes.003.jpg -------------------------------------------------------------------------------- /img/xtp-notes.004.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/libp2p/xtp/833c3427f83929d9650764f90f1c316cd62871e9/img/xtp-notes.004.jpg -------------------------------------------------------------------------------- /img/xtp-notes.005.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/libp2p/xtp/833c3427f83929d9650764f90f1c316cd62871e9/img/xtp-notes.005.jpg -------------------------------------------------------------------------------- /img/xtp-notes.006.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/libp2p/xtp/833c3427f83929d9650764f90f1c316cd62871e9/img/xtp-notes.006.jpg -------------------------------------------------------------------------------- /img/xtp-notes.007.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/libp2p/xtp/833c3427f83929d9650764f90f1c316cd62871e9/img/xtp-notes.007.jpg -------------------------------------------------------------------------------- /img/xtp-notes.key: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/libp2p/xtp/833c3427f83929d9650764f90f1c316cd62871e9/img/xtp-notes.key -------------------------------------------------------------------------------- /xtp-ctl.md: -------------------------------------------------------------------------------- 1 | # xtp-ctl -- XTP Control Protocol Spec 2 | 3 | ## Motivation 4 | 5 | [XTP](../README.md) requires a duplex pipe between the External Protocol process and the client application. This duplex pipe follows the `xtp-ctl` protocol, defined in this document. 6 | 7 | ## Spec 8 | 9 | ### Composition of Protocols 10 | 11 | `xtp-ctl` uses the following other standards: 12 | 13 | - [multiaddr](https://github.com/jbenet/multiaddr) for describing both "external transport" and "xtp-ctl pipe" endpoint addresses. 14 | - [multistream](https://github.com/jbenet/multistream) for a self-describing wire protocol 15 | - [libp2p transports](https://github.caom/libp2p/libp2p) for the transport protocol interface and modules to use 16 | - [libp2p stream multiplexing](https://github.caom/libp2p/libp2p) to interleave many transport protocol flows in the same xtp-ctl pipe. 17 | 18 | ### `xtp-ctl` Pipe 19 | 20 | The `xtp-ctl` wire protocol requires a reliable duplex pipe. 21 | 22 | WIP 23 | 24 | ### `xtp-ctl` Protocol 25 | 26 | The `xtp-ctl` wire protocol requires a reliable duplex pipe. 27 | 28 | WIP 29 | 30 | #### High Level Description 31 | 32 | WIP 33 | 34 | #### Wire Framing 35 | 36 | Discuss stream multiplexing. 37 | 38 | (Outside or inside the protocol??) 39 | 40 | WIP 41 | 42 | #### RPC Messages 43 | 44 | WIP 45 | 46 | (use protobuf??) 47 | 48 | ## Implementation Notes 49 | 50 | ### Multistream 51 | 52 | The `xtp-ctl` wire protocol is meant to be mounted on top of [multistream](https://github.com/jbenet/multistream), but it is also possible to use `xtp-ctl` without multistream. Therefore, this spec defines the `xtp-ctl` protocol directly, leaving the multistream mounting to the parent. 53 | 54 | The `xtp-ctl` pipe we propose here includes multistream, and suggests a transport, but leaves it up to the user to decide ultimately, as the user may have transport constraints. 55 | 56 | ### Head of Line Blocking 57 | 58 | If the `xtp-ctl` pipe is being transferred over a lossy protocol (eg. UDP across the internet), we strongly recommend using a transport such as QUIC to prevent head-of-line blocking issues. 59 | 60 | ### Standard Protocol Libraries 61 | 62 | It is recommended that `xtp-ctl` implementations use libraries from the [multiformats](https://github.com/multiformats) and [libp2p](https://github.com/libp2p/libp2p) projects. 63 | --------------------------------------------------------------------------------