├── 001.md ├── 002.md ├── 002 ├── keys.js ├── meeting-notes-arj-keks-2020-11-24 ├── meeting-notes-arj-keks-justin-2021-05-24 ├── meeting-notes-arj-keks-justin-2021-06-15 ├── meeting-notes-arj-keks-justin-2021-06-29 ├── metafeed-example1.svg ├── metafeed-example1.svg.license ├── metafeed-example2.svg ├── metafeed-example2.svg.license ├── presentation.html ├── presentation.md └── sharding-math.md ├── 003.md ├── 003 ├── audit-trustnet.md └── feedless-identity.md ├── 004.md ├── 004 ├── examples │ ├── m0.bbmsg │ └── m0.bbmsg.license ├── testvector-metafeed-managment-schema.json ├── testvector-metafeed-managment-schema.json.license ├── testvector-metafeed-managment.json └── testvector-metafeed-managment.json.license ├── 005.md ├── 006.md ├── 007.md ├── 008.md ├── 008 ├── bfe.json ├── bfe.test.js └── package.json ├── 009.md ├── 009 └── tangle_origin.md ├── 010.md ├── 011.md ├── CONTRIBUTING.md ├── LICENSES ├── CC-BY-4.0.txt └── CC0-1.0.txt ├── README.md ├── REVIEWING.md ├── TEMPLATE.md └── UPDATING.md /001.md: -------------------------------------------------------------------------------- 1 | # SSB URIs 2 | 3 | Author: Andre 'Staltz' Medeiros 4 | 5 | Date: 2022-10-01 6 | 7 | License: CC0-1.0 8 | 9 | ## Abstract 10 | 11 | The `ssb` scheme is registered as [provisional under the IANA](https://www.iana.org/assignments/uri-schemes/prov/ssb), but its format is not yet specified accessibly and clearly. This documents aims at providing the first specification of SSB URIs, as they are used in existing applications and will be adopted increasingly in more use cases. 12 | 13 | ## Motivation 14 | 15 | There is a thread on SSB, `%g3hPVPDEO1Aj/uPl0+J2NlhFB2bbFLIHlty+YuqFZ3w=.sha256`, where several participants gave their thoughts on the design of the SSB URI format, with several proposals put into consideration. The format of Query-only URIs were inspired by [Magnet URIs](https://www.iana.org/assignments/uri-schemes/prov/magnet). 16 | 17 | [ssb-uri](https://github.com/fraction/ssb-uri) is one outcome of that discussion, and is referred to in the IANA page. The `ssb-uri` library is depended on in [Patchwork](https://github.com/ssbc/patchwork/). As an alternative library, [ssb-custom-uri](https://git.sr.ht/~soapdog/ssb-custom-uri) is used in [Patchfox](https://github.com/soapdog/patchfox/). [Patchfoo](https://git.scuttlebot.io/%25YAg1hicat%2B2GELjE2QJzDwlAWcx0ML%2B1sXEdsWwvdt8%3D.sha256) supports both `ssb-uri` and `ssb-custom-uri`. [Manyverse](https://manyver.se) supports the URIs in this specification via the library [ssb-uri2](https://github.com/staltz/ssb-uri2). 18 | 19 | The specification in this document is compatible with `ssb-uri` while adding support for expressing more resources. 20 | 21 | ## Terminology 22 | 23 | The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC 2119](https://tools.ietf.org/html/rfc2119). 24 | 25 | ## Specification 26 | 27 | - **Canonical SSB URIs** 28 | - `ssb:message/classic/` 29 | - `ssb:message/bendybutt-v1/` 30 | - `ssb:message/gabbygrove-v1/` 31 | - `ssb:message/buttwoo-v1/` 32 | - `ssb:feed/classic/` 33 | - `ssb:feed/bendybutt-v1/` 34 | - `ssb:feed/gabbygrove-v1/` 35 | - `ssb:feed/buttwoo-v1/` 36 | - `ssb:feed/buttwoo-v1//` 37 | - `ssb:blob/classic/` 38 | - `ssb:address/multiserver?multiserverAddress=` 39 | - `ssb:encryption-key/box2-dm-dh/` 40 | - `ssb:identity/po-box/` 41 | - `ssb:identity/fusion/` 42 | - **Experimental SSB URIs** 43 | - `ssb:experimental?action=claim-http-invite&invite=&multiserverAddress=` 44 | - `ssb:experimental?action=consume-alias&alias=&userId=&signature=&roomId=&multiserverAddress=` 45 | - `ssb:experimental?action=start-http-auth&sid=&sc=` 46 | - **Deprecated SSB URIs** 47 | - `ssb:message/sha256/` 48 | - `ssb:feed/ed25519/` 49 | - `ssb:blob/sha256/` 50 | 51 | ### Overview 52 | 53 | There are two main categories of SSB URIs: **Canonical** and **Experimental**. The conceptual difference between these two is that canonical SSB URIs make use of the **path component** in the URI to specify the resource, while experimental SSB URIs use *only* the **query component** to specify the resource. 54 | 55 | The third category, **Deprecated**, is used to denote canonical SSB URIs that MUST still be supported by application, but SHOULD be avoided whenever possible. 56 | 57 | ### Canonical SSB URIs 58 | 59 | These SSB URIs are to be considered stable and universally acceptable by all SSB applications that support SSB URIs. They utilize the path component to specify "refs", i.e. identifiers for resources. SSB messages, feeds, and blobs **SHOULD** follow the respective syntaxes: 60 | 61 | ``` 62 | ssb:message/classic/ 63 | ssb:message/bendybutt-v1/ 64 | ssb:message/gabbygrove-v1/ 65 | ssb:message/buttwoo-v1/ 66 | ssb:feed/classic/ 67 | ssb:feed/bendybutt-v1/ 68 | ssb:feed/gabbygrove-v1/ 69 | ssb:feed/buttwoo-v1/ 70 | ssb:feed/buttwoo-v1// 71 | ssb:blob/classic/ 72 | ``` 73 | 74 | Where ``, ``, `` are *URI-safe Base64 encoded* strings that identify those refs. URI-safe Base64 is equivalent to Base64 where `+` characters are replaced with `-`, and `/` characters are replaced with `_`. 75 | 76 | In the special case of `ssb:feed/buttwoo-v1//`, the `` is the URI-safe Base64 encoded data of a buttwoo-v1 message URI (i.e. `` from `ssb:message/buttwoo-v1/`). 77 | 78 | **Examples:** 79 | 80 | ``` 81 | ssb:message/classic/g3hPVPDEO1Aj_uPl0-J2NlhFB2bbFLIHlty-YuqFZ3w= 82 | ssb:message/bendybutt-v1/PR2-btDEO1AjXuPl0TJ2N_hFB2bbFLIHlty0VF1ncty= 83 | ssb:feed/classic/-oaWWDs8g73EZFUMfW37R_ULtFEjwKN_DczvdYihjbU= 84 | ssb:feed/bendybutt-v1/APaWWDs8g73EZFUMfW37RBULtFEjwKNbDczvdYiRXtA= 85 | ssb:feed/gabbygrove-v1/FY5OG311W4j_KPh8H9B2MZt4WSziy_p-ABkKERJdujQ= 86 | ssb:blob/classic/sbBmsB7XWvmIzkBzreYcuzPpLtpeCMDIs6n_OJGSC1U= 87 | ``` 88 | 89 | There are also new concepts related to [Private Groups](https://github.com/ssbc/private-group-spec) which need identifiers, such as: 90 | 91 | - `ssb:encryption-key/box2-dm-dh/` 92 | - `ssb:identity/po-box/` 93 | 94 | Where `` are *URI-safe Base64 encoded* strings, similar to the previous canonical SSB URIs. 95 | 96 | **Multiserver address:** 97 | 98 | ``` 99 | ssb:address/multiserver?multiserverAddress= 100 | ``` 101 | 102 | Where `` is a [multiserver address](https://github.com/ssbc/multiserver-address) string, but [percent-encoded according to RFC3986 2.1](https://tools.ietf.org/html/rfc3986#section-2.1), for example: 103 | 104 | ``` 105 | ssb:address/multiserver?multiserverAddress=net%3Awx.larpa.net%3A8008~shs%3ADTNmX%2B4SjsgZ7xyDh5xxmNtFqa6pWi5Qtw7cE8aR9TQ%3D 106 | ``` 107 | 108 | ### Colon-separated canonical SSB URIs 109 | 110 | Canonical SSB URIs **SHOULD** use `/` to separate the parts of the path component, but they **MAY** also use `:` to separate the parts. The following syntax is acceptable (and is currently what `ssb-uri` utilizes, exclusively): 111 | 112 | ``` 113 | ssb:message:classic: 114 | ssb:message:bendybutt-v1: 115 | ssb:feed:classic: 116 | ssb:feed:bendybutt-v1: 117 | ssb:blob:classic: 118 | ssb:address:multiserver?multiserverAddress= 119 | ``` 120 | 121 | ### Experimental SSB URIs 122 | 123 | These SSB URIs are free-form and allow developers to pass any parameters to SSB applications without requiring consensus among the SSB applications. These URIs have an *empty authority* component and path component always matching `experimental` (this is to support Firefox's handling of custom schemes) but have a *non-empty query* component. The query parameters are the only mechanism through which information is conveyed. They satisfy the syntax below (any number of parameters allowed, but we show two, for illustration): 124 | 125 | ``` 126 | ssb:experimental?= 127 | ssb:experimental?=&= 128 | ``` 129 | 130 | #### Known experimental SSB URIs 131 | 132 | Experimental SSB URIs do not need to be included in this specification before they can be used, but we encourage developers to submit all known experimental SSB URIs depended by applications. This facilitates the future canonicalization of new SSB URIs. Please feel free to submit new entries here, as long as you know they are used in applications: 133 | 134 | 135 | **Action to join a room:** 136 | 137 | Some experimental SSB URIs have the query `action` to describe intents to perform certain tasks in the SSB application. 138 | 139 | A `action=claim-http-invite` experimental URI is used to refer to the consumption of a [Rooms 2](https://github.com/ssb-ngi-pointer/rooms2) invite code, syntax as follows, which includes a `multiserverAddress` query like the canonical multiserver address URI has too: 140 | 141 | ``` 142 | ssb:experimental?action=claim-http-invite&invite=&multiserverAddress=` 143 | ``` 144 | 145 | **Action to consume an alias:** 146 | 147 | A URI with query `action=consume-alias` allows us to refer to the processing [consuming a room alias](https://github.com/ssb-ngi-pointer/rooms2/blob/573cc4b3afc08a4eccaea530104524aa7f60af9f/docs/Alias/Alias%20consumption.md), syntax as follows: 148 | 149 | ``` 150 | ssb:experimental?action=consume-alias&alias=&userId=&signature=&multiserverAddress=&roomId= 151 | ``` 152 | 153 | **Action to start HTTP authentication:** 154 | 155 | A URI with query `action=start-http-auth` is for initiating [Sign-in with SSB](https://github.com/ssb-ngi-pointer/rooms2/blob/573cc4b3afc08a4eccaea530104524aa7f60af9f/docs/Setup/Sign-in%20with%20SSB.md) (HTTP Authentication) with an SSB remote server. The syntax is as follows: 156 | 157 | ``` 158 | ssb:experimental?action=start-http-auth&sid=&sc= 159 | ``` 160 | 161 | ### Process to canonicalize SSB URIs 162 | 163 | When experimental SSB URIs are adopted by SSB applications, if they are deemed useful and long-term URIs, there should be discussion among stakeholders (SSB application developers, and protocol developers) to specify a canonical SSB URI format (using path component) to replace the experimental SSB URI. Then, the new canonical SSB URI should be specified in this document. 164 | 165 | ## References 166 | 167 | ### Normative 168 | 169 | - [SSB URI at IANA](https://www.iana.org/assignments/uri-schemes/prov/ssb) 170 | - [RFC3986 Section 2.1](https://tools.ietf.org/html/rfc3986#section-2.1) 171 | - [multiserver addresses](https://github.com/ssbc/multiserver-address) 172 | 173 | ### Informative 174 | 175 | - [Magnet URIs](https://www.iana.org/assignments/uri-schemes/prov/magnet) 176 | 177 | ### Implementation 178 | 179 | - [ssbc/ssb-uri2](https://github.com/ssbc/ssb-uri2) in JavaScript 180 | 181 | ## Appendix A. Grammar 182 | 183 | The following grammar specifies all SSB URIs (both canonical and any experimental) using [Nearley](https://nearley.js.org): 184 | 185 | ```nearley 186 | uri -> scheme delimiter body 187 | 188 | scheme -> "ssb" 189 | 190 | delimiter -> ":" 191 | 192 | separator -> [:/] 193 | 194 | body -> parts1 (queries):? | parts3 (queries):? | parts4 (queries):? 195 | 196 | parts1 -> type1 197 | parts3 -> type3 separator alg3 separator value 198 | parts4 -> type4 separator alg4 separator value separator value 199 | 200 | queries -> "?" (query):+ 201 | query -> queryKey "=" queryVal 202 | queryKey -> [a-zA-Z] ([^=]):* 203 | queryVal -> [a-zA-Z0-9] ([^&]):* 204 | 205 | type1 -> "experimental" 206 | type3 -> "message" | "feed" | "blob" | "address" | "encryption-key" | "identity" 207 | type4 -> "feed" 208 | 209 | alg3 -> "classic" | "multiserver" | "bendybutt-v1" | "gabbygrove-v1" | "buttwoo-v1" | "box2-dm-dh" | "po-box" | "fusion" 210 | alg4 -> "buttwoo-v1" 211 | 212 | value -> ([0-9a-zA-Z\-\_\=]):+ 213 | ``` 214 | -------------------------------------------------------------------------------- /002.md: -------------------------------------------------------------------------------- 1 | # Metafeeds 2 | 3 | Author: Anders Rune Jensen 4 | 5 | Date: 2021-10-11 6 | 7 | License: CC0-1.0 8 | 9 | ## Abstract 10 | 11 | In classical SSB an identity is tied to a single feed. All messages 12 | for different kinds of applications are posted to this single 13 | feed. While it is possible to create multiple feeds, there has been no 14 | formal specification for how these feeds relate and what their 15 | purposes are. 16 | 17 | Metafeeds aim to solve these problems by tying an identity to a metafeed 18 | instead. A metafeed references other feeds (or even metafeeds) and 19 | contains metadata about the feed including purpose and feed 20 | format. This allows for things like feed rotation to a new feed 21 | format, splitting data into separate (sub)feeds and to create special 22 | indexing feeds for partial replication. 23 | 24 | A metafeed is tied to a single identity and thus should only be used 25 | on a single device. There is a separate [fusion identity] protocol 26 | that only deals with how to relate multiple devices to a single 27 | identity. This spec here is not for that use-case. 28 | 29 | Metafeeds will use a specialized feed format known as [bendy butt] that aims 30 | to be very easy to implement. The aim is that this will make it easier for 31 | implementations which do not need or want to support the classical SSB format. 32 | 33 | ## Terminology 34 | 35 | The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", 36 | "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be 37 | interpreted as described in RFC 2119. 38 | 39 | We use bencode and [BFE] notations as defined in the [bendy butt] spec. 40 | 41 | ## Specification 42 | 43 | ### Usage of Bendy Butt feed format 44 | 45 | Metafeeds **MUST** use the [bendy butt] feed format with a few additional 46 | constraints. 47 | 48 | The `content` dictionary inside the `contentSection` of meta feed messages 49 | **MUST** conform to the following rules: 50 | 51 | - Has a `type` field mapping to a BFE string (i.e. `<06 00> + data`) which 52 | can assume only one the following possible values: 53 | - `metafeed/add/existing` 54 | - `metafeed/add/derived` 55 | - `metafeed/update` 56 | - `metafeed/tombstone` 57 | - Has a `subfeed` field mapping to a BFE "feed ID", i.e. `<00> + format + data` 58 | - Has a `metafeed` field mapping to a BFE "Bendy Butt feed ID", i.e. 59 | `<00 03> + data` 60 | - (Only if the `type` is `metafeed/add/derived`): a `nonce` field mapping 61 | to a BFE "arbitrary bytes" with size 32, i.e. `<06 03> + nonce32bytes` 62 | 63 | The `contentSignature` field inside a decrypted `contentSection` **MUST** use 64 | the `subfeed`'s cryptographic keypair. 65 | 66 | ### Example of a metafeed 67 | 68 | Here is an an example of a metafeed with 2 subfeeds: one for `main` 69 | social data and another one for `application-x` in a different format. 70 | 71 | ![Diagram](./metafeed-example1.svg) 72 |
73 | digraph metafeed { 74 | 75 | rankdir=RL 76 | node [shape=record]; 77 | 78 | edge [tailclip=false]; 79 | a [label="{ | main }"] 80 | b [label="{ | application-x }"]; 81 | b:ref:a -> a:data [arrowhead=vee, arrowtail=dot, dir=both]; 82 | } 83 |
84 | 85 | Contents of messages in the metafeed that acts as metadata for feeds: 86 | 87 | ``` 88 | { 89 | "type" => "metafeed/add/existing", 90 | "feedpurpose" => "main", 91 | "subfeed" => (BFE-encoded feed ID for the 'main' feed), 92 | "metafeed" => (BFE-encoded Bendy Butt feed ID for the metafeed), 93 | "tangles" => { 94 | "metafeed" => { 95 | "root" => null, 96 | "previous" => null 97 | } 98 | }, 99 | }, 100 | { 101 | "type" => "metafeed/add/existing", 102 | "feedpurpose" => "application-x", 103 | "subfeed" => (BFE-encoded Bamboo feed ID), 104 | "metafeed" => (BFE-encoded Bendy Butt feed ID for the metafeed), 105 | } 106 | ``` 107 | 108 | Initially the metafeed spec supports three operations: `add/existing` 109 | `add/derived`, and `tombstone`. **Note**, signatures (see key 110 | management section) are left out in the examples here. 111 | 112 | Tombstoning means that the feed is no longer part of the metafeed. 113 | Whether or not the subfeed itself is tombstoned is a separate 114 | concern. 115 | 116 | Example tombstone message: 117 | 118 | ``` 119 | { 120 | "type" => "metafeed/tombstone", 121 | "subfeed" => (BFE-encoded Bamboo feed ID), 122 | "metafeed" => (BFE-encoded Bendy Butt feed ID for the metafeed), 123 | "reason" => (some BFE string), 124 | "tangles" => { 125 | "metafeed" => { 126 | "root" => (BFE-encoded message ID of the "metafeed/add" message), 127 | "previous" => (BFE-encoded message ID of the "metafeed/add" message), 128 | } 129 | } 130 | } 131 | ``` 132 | 133 | Updating the metadata on a subfeed which is a member of a metafeed 134 | is currently not supported. 135 | 136 | **Note**: while the `metafeed: ...` field on the add and tombstone messages 137 | seems redundant, it is important to have it and check that the `metafeed` field 138 | equals the author of the metafeed itself to protect against replay attacks. 139 | 140 | ### Applications example 141 | 142 | An example of the applications metafeed with two different 143 | applications. 144 | 145 | ![Diagram2](./metafeed-example2.svg) 146 |
147 | digraph Applications { 148 | 149 | rankdir=RL 150 | nodesep=0.6 151 | node [shape=record]; 152 | 153 | edge [tailclip=false]; 154 | a [label="{ | App1 }"] 155 | b [label="{ | App2 }"]; 156 | 157 | b:ref:a -> a:data [arrowhead=vee, arrowtail=dot, dir=both]; 158 | } 159 |
160 | 161 | ``` 162 | { 163 | "type" => "metafeed/add/derived", 164 | "feedpurpose" => "gathering", 165 | "subfeed" => (BFE-encoded feed ID dedicated for the gathering app), 166 | (other fields...) 167 | }, 168 | { 169 | "type" => "metafeed/add/derived", 170 | "feedpurpose" => "chess" 171 | "subfeed" => (BFE-encoded feed ID dedicated for the chess app), 172 | (other fields...) 173 | } 174 | ``` 175 | 176 | ### Tree structure 177 | 178 | Since a subfeed can be a metafeed itself, this means that the relationships 179 | between subfeeds and metafeeds is a tree. We refer to the top-most metafeed as 180 | the "root" metafeed. 181 | 182 | While a metafeed **MAY** contain any arbitrary subfeed, we prescribe a 183 | **RECOMMENDED** structure for the tree. 184 | 185 | Under the root metafeed, there **SHOULD** be only one *type* of subfeed: 186 | *versioning subfeeds*. For now, there **SHOULD** be only subfeed `v1`, but in 187 | the future, this spec will be extended to describe subfeed `v2` once it is time 188 | to deprecate `v1`. 189 | 190 | The subfeeds at the leafs of the tree contain actual content. Once there is a 191 | new versioning subfeed, the leaf feeds can be transferred under the new 192 | versioning subfeed via `metafeed/add/existing` messages, without having to 193 | recreate the leaf feeds. Thus, the tree structure is only concerned with the 194 | *organization* of feeds in order to assist partial replication. For example, 195 | by grouping together feeds that are part of the same application under a common 196 | metafeed, we can skip replication of those application feeds if they are not 197 | relevant to the user. 198 | 199 | #### v1 200 | 201 | This section describes the specification of the organization of subfeeds under 202 | the `v1` versioning subfeed. 203 | 204 | To start with, the `v1` versioning subfeed **MUST** be created with the 205 | following `content` on the root metafeed: 206 | 207 | ``` 208 | { 209 | "type" => "metafeed/add/derived", 210 | "feedpurpose" => "v1", 211 | "subfeed" => (BFE-encoded feed ID dedicated for the versioning subfeed), 212 | } 213 | ``` 214 | 215 | The feed format for `v1` **MUST** be [bendy butt], because it is a metafeed. 216 | 217 | The *direct* subfeeds of `v1` are the so-called *shard feeds*. The actual 218 | application-specific subfeeds are under the shard feeds. Sharding is based on 219 | 4 bits of entropy extracted from the application-specific subfeed, and 220 | can be represented by 1 hexadecimal digit. We will call that digit the "nibble". 221 | The nibbles are: `0`, `1`, `2`, `3`, `4`, `5`, `6`, `7`, `8`, `9`, `a`, `b`, 222 | `c`, `d`, `e`, `f`. The number of shards is specifically set at 16 to allow for 223 | efficient partial replication in realistic scenarios. See 224 | [sharding math](./sharding-math.md) for mathematical details on the choice of 225 | number of shards. 226 | 227 | The purpose of the shard feeds is to allocate the set of application-specific 228 | subfeeds into 16 separate groupings of feeds, i.e. one for each nibble. This 229 | way, if you are only interested in replicating a subset of the 230 | application-specific subfeeds, you can deterministically calculate the nibble 231 | for those application-specific subfeeds, and then you know which shard feeds 232 | to replicate. 233 | 234 | When adding a new application-specific subfeed to the tree, we need to determine 235 | the parent shard based on a "name", which is any UTF-8 string that the 236 | application can choose freely, but it is **RECOMMENDED** that this string be 237 | unique to the use case. Then, the shard feed's nibble is calculated as the first 238 | hexadecimal digit of the following SHA256 hash: 239 | 240 | ``` 241 | sha256_hash(concat_bytes(root_metafeed_id, name)) 242 | ``` 243 | 244 | where `root_metafeed_id` is the BFE-encoded ID of the root metafeed, and 245 | `name` is a BFE-encoded UTF-8 string. 246 | 247 | The nibble is then used to create a new shard feed, unless there is already 248 | one. There **MUST** be at most *one* shard feed for every unique nibble. The 249 | `content` on the root's message for the shard feed **MUST** have the nibble 250 | encoded as hexadecimal in the `feedpurpose` field of the `metafeed/add/derived` 251 | message. The feed format for a shard feed **MUST** be [bendy butt], because 252 | they are metafeeds. 253 | 254 | Once the shard feed is created, the application-specific subfeeds can be added 255 | as subfeeds of that one, either as `metafeed/add/derived` or 256 | `metafeed/add/existing`. 257 | 258 | The following diagram is an example of the organization of subfeeds under the v1 259 | specification: 260 | 261 | ```mermaid 262 | graph TB; 263 | root --> v1 264 | v1 --> 8 & a & c & 3 265 | 8 --> post 266 | a --> gathering 267 | a --> chess 268 | c --> vote 269 | 3 --> contact 270 | ``` 271 | 272 | It is **RECOMMENDED** that the application-specific subfeeds are leafs in the 273 | tree, but they **MAY** be metafeeds that contain other application-specific 274 | subfeeds. 275 | 276 | ### Key management, identity and metadata 277 | 278 | As mentioned earlier, in classical SSB the feed identity is the same 279 | as the feed. Here instead we want to decouple identity and feeds. 280 | 281 | #### Existing SSB identity 282 | 283 | To generate a metafeed and link it to an existing `main` feed, first 284 | a seed is generated: 285 | 286 | ```js 287 | const seed = crypto.randomBytes(32) 288 | ``` 289 | 290 | From this seed, a metafeed can be generated using: 291 | 292 | ```js 293 | const salt = 'ssb' 294 | const prk = hkdf.extract(lhash, hash_len, seed, salt) 295 | const mf_info = "ssb-meta-feed-seed-v1:metafeed" 296 | const mf_seed = hkdf.expand(hash, hash_len, prk, length, mf_info) 297 | const mf_key = ssbKeys.generate("ed25519", mf_seed) 298 | ``` 299 | 300 | Note we use `metafeed` here in the info. As the top/genesis metafeed is 301 | special we use that string, for all other derived feeds a nonce is used, 302 | which is also published in the corresponding `metafeed/add/derived` 303 | message. 304 | 305 | We also encrypt the seed as a private message from `main` to `main` (so 306 | it's a private message to yourself; notice this is JSON, because it's 307 | published on the main): 308 | 309 | ``` 310 | { 311 | "type": "metafeed/seed", 312 | "metafeed": ssb:feed/bendybutt-v1/bendyButtFeedID, 313 | "seed": seedBytesEncodedAsHexString 314 | } 315 | ``` 316 | 317 | By doing so we allow the existing feed to reconstruct the metafeed and 318 | all subfeeds from this seed. 319 | 320 | Then the metafeed is linked with the existing `main` feed using a new 321 | message on the metafeed signed by both the `main` feed and the meta 322 | feed. For details this see [bendy butt]. 323 | 324 | ``` 325 | { 326 | "type" => "metafeed/add/existing", 327 | "feedpurpose" => "main", 328 | "subfeed" => (BFE-encoded feed ID for the 'main' feed), 329 | "metafeed" => (BFE-encoded Bendy Butt feed ID for the metafeed), 330 | "tangles" => { 331 | "metafeed" => { 332 | "root" => (BFE nil), 333 | "previous" => (BFE nil) 334 | } 335 | } 336 | } 337 | ``` 338 | 339 | In order for existing applications to know that the existing feed 340 | supports metafeeds, a special message of type `metafeed/announce` 341 | is created on the `main` feed (notice this is JSON, because the 342 | main feed is not in Bendy Butt): 343 | 344 | ```js 345 | { 346 | // ... other msg.value field ... 347 | content: { 348 | type: 'metafeed/announce', 349 | metafeed: 'ssb:feed/bendybutt-v1/-oaWWDs8g73EZFUMfW37R_ULtFEjwKN_DczvdYihjbU=', 350 | subfeed: MAIN_FEED_ID, 351 | tangles: { 352 | metafeed: { 353 | root: null, 354 | previous: null 355 | } 356 | }, 357 | signature: SIGNATURE_OF_THE_ABOVE 358 | } 359 | } 360 | ``` 361 | 362 | Note that MAIN_FEED_ID is the ID of the main feed, and that 363 | SIGNATURE_OF_THE_ABOVE is the signature (using the metafeed 364 | keys) of the stringified `content` *without* `content.signature` 365 | itself, in a similar manner to how the message signature 366 | `msg.value.signature` is constructed relative to `msg.value`. So 367 | `msg.value.signature` is signed with the `main` feed's keys, but 368 | `msg.value.content.signature` is signed with the *metafeed keys*. 369 | 370 | A feed can only have **one** metafeed. If for whatever reason an 371 | existing metafeed needs to be superseed, a new message is created 372 | pointing to the previous `metafeed/announce` message via the tangle. 373 | 374 | #### New SSB identity 375 | 376 | A new identity also starts by constructing a seed. From this seed both 377 | the metafeed keys and the main feed keys are generated. The main 378 | should use the info: `ssb-meta-feed-seed-v1:` 379 | and the `nonce` is also published as part of the `metafeed/add/derived` 380 | message on the metafeed. 381 | 382 | ``` 383 | { 384 | "type" => "metafeed/add/derived", 385 | "feedpurpose" => "main", 386 | "subfeed" => (BFE-encoded feed ID for the 'main' feed), 387 | "metafeed" => (BFE-encoded Bendy Butt feed ID for the metafeed), 388 | "nonce" => (bencode byte sequence with 32 random bytes), 389 | "tangles" => { 390 | "metafeed" => { 391 | "root" => null, 392 | "previous" => null 393 | } 394 | } 395 | } 396 | ``` 397 | 398 | The seed will also be encrypted to the main feed and the metafeed 399 | linked to the main feed just like for existing feeds. 400 | 401 | #### Identity backwards compatibility 402 | 403 | By building a layer on top of existing feeds we maintain backwards 404 | compatible with existing clients. The identity to be used by new 405 | applications should be that of the metafeed. For backwards 406 | compatibility contact messages forming the follow graph together with 407 | secret handshake will continue to use the key of the main feed. 408 | 409 | It is worth noting that even though the examples above specify ways to 410 | generate new feeds from a single seed, it is perfectly fine and in 411 | some cases a better idea to generate a feed not from this seed. Thus 412 | in the case the main key being broken or stolen, you don't loose 413 | everything. 414 | 415 | If a key is reused in another part of the tree it must include a 416 | reference to the original subfeed or metafeed it was defined in. The 417 | original place is the authorative place for its metadata. 418 | 419 | Using [BIP32-Ed25519] instead was considered but that method has a 420 | weaker security model in the case of a key compromised where keys are 421 | shared between devices. 422 | 423 | ### Use cases 424 | 425 | Let us see how we can use the above abstraction to solve several 426 | common examples: 427 | 428 | #### New feed format 429 | 430 | Changing to a new feed format could be implemented by adding a new 431 | feed to the metafeed state, and by adding a tombstone message to the 432 | old feed pointing and assigning the new feed as active in the meta 433 | feed. 434 | 435 | In case of backwards compability with clients that do not support a 436 | newer feed format or in the case of only wanting to support newer feed 437 | formats, maintaining muliple feeds with the same content would be an 438 | interesting avenue to explore. As the hash of the messages in the two 439 | feeds would be different, there could be a way to include the hash of 440 | the corresponding message in old feed in the newer feed. 441 | 442 | Lower end clients could offload this extra storage requirement to 443 | larger peers in the network. 444 | 445 | #### Claims or indexes 446 | 447 | For classical SSB feeds if one would like to replicate a specific part 448 | of a feed, such as the contact messages, one could request another 449 | peer to generate a feed that only references these messages. Then when 450 | exchanging data, the original messages could be included as auxiliary 451 | data. This would only act as a claim, never as a proof that some 452 | messages were not left out. Naturally this comes down to trust 453 | then. Using the friend graph would be natural, as would using trustnet 454 | together with audits of these claims. 455 | 456 | #### Subfeeds 457 | 458 | Similar to claims it would be possible to create subfeeds that would 459 | only contain certain messages. This might be useful for specific 460 | apps. Another use case for this would be curated content, where 461 | specific messages are picked out that might be of particular interest 462 | to a certain application or specific people, or say messages within 463 | the last year. 464 | 465 | #### Ephemeral feeds 466 | 467 | Using the metadata it would be possible to attach a lifetime to feeds, 468 | meaning honest peers would delete the feeds after a specific 469 | time. This would enable applications to generate a short lived feed 470 | only for the communication between two parties. 471 | 472 | #### Allow list 473 | 474 | Similar to ephemeral feeds it would be possible to attach an allow 475 | list to a feed and only distribute this feed to people on the allow 476 | list. As with ephemeral feeds, this cannot be enforced, but assuming 477 | honest peers would give piece of mind that the data is only stored on 478 | a certain subset of the whole network. This can naturally be combined 479 | with private groups to better ensure safety. 480 | 481 | ### Open questions 482 | 483 | - In the case of claims, how are bad actors handled? 484 | - What are the broader consequences of ephemeral feeds. Maybe they can 485 | only be used in limited circumstances, and if so which ones? 486 | - For subfeeds and feed rotation what is the best way to handle 487 | potentially overlapping messages 488 | 489 | ## References 490 | 491 | ### Normative 492 | 493 | - [SIP 4](./004.md) "Bendy Butt" 494 | 495 | ### Informative 496 | 497 | - [BIP32-Ed25519] 498 | 499 | ### Implementation 500 | 501 | - [ssbc/ssb-meta-feeds](https://github.com/ssbc/ssb-meta-feeds) in JavaScript 502 | 503 | ### Acknowledgments and prior work 504 | 505 | CFT suggested the use of metafeeds 506 | in [ssb-observables issue 1](https://github.com/arj03/ssb-observables/issues/1). 507 | 508 | [BIP32-Ed25519]: https://github.com/wallet-io/bip32-ed25519/blob/master/doc/Ed25519_BIP.pdf 509 | [ssb-secure-partial-replication]: https://github.com/ssb-ngi-pointer/ssb-secure-partial-replication 510 | [fusion identity]: https://github.com/ssb-ngi-pointer/fusion-identity-spec/ 511 | [bencode]: https://en.wikipedia.org/wiki/Bencode 512 | [BFE]: https://github.com/ssbc/ssb-bfe-spec 513 | [bendy butt]: https://github.com/ssb-ngi-pointer/bendy-butt-spec 514 | -------------------------------------------------------------------------------- /002/keys.js: -------------------------------------------------------------------------------- 1 | // SPDX-FileCopyrightText: 2021 Anders Rune Jensen 2 | // 3 | // SPDX-License-Identifier: CC0-1.0 4 | 5 | const crypto = require("crypto") 6 | const ssbKeys = require('ssb-keys') 7 | const derive = require('derive-key') 8 | 9 | /* 10 | const length = 32 11 | const ikm = crypto.randomBytes(length) // aka as seed 12 | */ 13 | 14 | // or from backup: 15 | const ikm_hex = '4e2ce5ca70cd12cc0cee0a5285b61fbc3b5f4042287858e613f9a8bf98a70d39' 16 | ikm = Buffer.from(ikm_hex, 'hex') 17 | 18 | console.log("seed", ikm.toString('hex')) 19 | 20 | const mf_seed = derive('ssb-meta-feed', ikm, 'ssb-meta-feed-seed-v1:metafeed') 21 | //console.log('the derived meta feed seed is:', mf_seed) 22 | const mf_key = ssbKeys.generate("ed25519", mf_seed) 23 | console.log("mf_key", mf_key) 24 | 25 | const sf_seed = derive('ssb-meta-feed', ikm, 'ssb-meta-feed-seed-v1:subfeed-1') 26 | //console.log('the derived sub feed seed is:', sf_seed) 27 | const sf_key = ssbKeys.generate("ed25519", sf_seed) 28 | console.log("sf_key", sf_key) 29 | -------------------------------------------------------------------------------- /002/meeting-notes-arj-keks-2020-11-24: -------------------------------------------------------------------------------- 1 | 6 | 7 | --- 8 | tags: ssb 9 | --- 10 | 11 | # Call Notes Meta Feeds 12 | 13 | - arj, keks 14 | 15 | ## Simpler Subfeed ID Generation 16 | 17 | Proof of Possession 18 | 19 | ``` 20 | MF: [ 21 | { content:{type: main, id: @sf0, author:@mf, nonce: , sig_sf: suf_sf.sig }, sig_mf:sig_mf.sig} 22 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 23 | signed by subkey 24 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 25 | signed by metakey 26 | ] 27 | ``` 28 | 29 | Domain Separation of subkey and metakey signatures: 30 | 31 | sig_mf = sign(sk_mf, 0x00 || msg) 32 | sig_sf = sign(sk_sf, 0x01 || msg) 33 | 34 | -> no need to encode the tree depth 35 | 36 | ## Non-Hierarchical Deterministic 37 | ``` 38 | seed = rand(32 bytes) // maybe make this even a bit bigger, e.g. 64, 128 bytes 39 | mf_seed = hkdf_expand(key=seed, info="ssb-meta-feed-seed-v1:metafeed") 40 | (mf_sk, mf_pk) = ed25519_gen_from_seed(mf_seed) 41 | 42 | sf_seed = hkdf_expand(key=seed, info="ssb-meta-feed-seed-v1:subfeed-$pos") 43 | (sf_sk, sf_pk) = ed25519_gen_from_seed(sf_seed) 44 | 45 | 46 | // if you want to be able to recover keys generated non-hierarchically 47 | sf_enc_k = hkdf_expand(key=seed, info="ssb-meta-feed-seed-v1:subfeed-keyenc-$pos") 48 | sf_sk_enc = enc(sf_enc_k, orig_sk) 49 | ``` 50 | 51 | then 52 | 53 | ``` 54 | MF: [ 55 | { content:{type: main, id: @sf0, author:@mf, nonce: , enc_sf_sk: , sig_sf: suf_sf.sig}, sig_mf:sig_mf.sig} 56 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 57 | signed by subkey 58 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 59 | signed by metakey 60 | ] 61 | ``` 62 | 63 | but! selective forgettability is a good thing, and we lose it if everything can be learned from the main seed 64 | -------------------------------------------------------------------------------- /002/meeting-notes-arj-keks-justin-2021-05-24: -------------------------------------------------------------------------------- 1 | 6 | 7 | handshake between feeds - misunderstanding about being single device, 8 | need to clarify further. Maybe remove some of the examples. 9 | 10 | domain separation for main signing messages, look at sign in with ssb 11 | 12 | maybe a new incredibly simple feed format for meta feeds? 13 | 14 | related, maybe a different identifier, like 15 | %sdfasldkrf;skjdf;laksjdf=.cloaked for private groups versus 16 | @sdfasldkrf;skjdf;laksjdf=.ed25519 for current classic feeds 17 | 18 | keks thought it doesn't make sense to make a distinction between index 19 | feeds and claims. They should all just be claims. What happens if you 20 | have a broken implementation that creates broken index feeds? 21 | 22 | instead of using timestamp for nonce use random bytes 23 | 24 | the query language in 25 | https://github.com/ssb-ngi-pointer/ssb-secure-partial-replication-spec 26 | needs a proper spec. Maybe also a bit more general. 27 | 28 | meta feeds spec mentions subfeed-1 at one point, we probably should 29 | use the nonce instead for generating keys as they are unique. 30 | 31 | potential date for next meeting 15 June 32 | 33 | arj will send updated specs at the end of this week -------------------------------------------------------------------------------- /002/meeting-notes-arj-keks-justin-2021-06-15: -------------------------------------------------------------------------------- 1 | 6 | 7 | # Meta feeds & bendy butt spec 8 | 9 | Q: should we use a salt value for key derivation? 10 | 11 | K: yes, would probably be a good idea. Goes on to explain that it 12 | would be nice that something like the hmac key could also be derived 13 | from this, but okay to just use any key and one for the whole network. 14 | 15 | Q: Number of random bytes for seed input 16 | 17 | K: 32 bytes should be enough 18 | 19 | Q: Number of random bytes for nonce 20 | 21 | K: As very few keys will be generated one doesn't really need very 22 | much, we just need them to be unique. 23 | 24 | We ended up deciding that 32 bytes would probably also be a good 25 | number. 26 | 27 | Q: Put content & content signature in a box2 encrypted binary field 28 | 29 | K: yes 30 | 31 | There was a bit confusion about what payload signature and content 32 | signature ment and which keys signed what. Might be good with common 33 | terminology here. 34 | 35 | # Partial replication spec 36 | 37 | https://github.com/ssb-ngi-pointer/ssb-secure-partial-replication-spec 38 | 39 | Notes from Justin: 40 | 41 | Claims and Audits 42 | "...it MIGHT make sense for others to create claims and audit that these claims are indeed valid indexes." 43 | what is meant by MIGHT? Is this part of the protocol or just an idea? Can this step be omitted? 44 | 45 | 46 | "Because feeds are immutable, once you have verified a feed up until sequence x the past can never change." 47 | Who is verifying the claim? If bob is verifying Alice's feed, who gave Bob authority. Could Bob not verify until to sequence X, which would prevent Alice's feed from immutability? Ie. Alice could re-write the feed with Bob's help? Or is validation distributed amongst peers? 48 | 49 | 50 | "In order not to create too many verification messages, a new message should only be posted if claim is no longer valid." 51 | who is creating verification messages? Do I ask verification of my own messages, or do I ask to verify someone else's. If the claim is invalid, does the refer to reopening a claim? Or does this refer to validating a claim? 52 | 53 | 54 | "How often claims should be verified is at the discretion of the auditor." 55 | Who is the auditor? Who gives them the authority? 56 | 57 | 58 | "It is worth noting that it is limited what a malicious peer could do. The messages in a claim still needs to be signed by the author, so at worst messages can be left out." 59 | This still amounts to DoS, as messages could be prevented from circulating 60 | 61 | 62 | 63 | Notes from talk 64 | 65 | There were some question around who could audit, also who could change 66 | an audit (only self). That probably needs a bit of clarification. 67 | 68 | There was a strong need for a more detailed examples of invalid 69 | claims. Like what was valid. How can others look and this and see if 70 | they agree. Should this be machine readable? 71 | 72 | For audits only includes latestseq, what feed is that on? The index or 73 | indexed feed? Probably be better with both. 74 | 75 | We talked a bit about these being more like observables, if you only 76 | write when things change, how do you know if audits are stale? And 77 | should you only trust the audit up until latestseq? 78 | 79 | We talked a bit on the trust model (trustnet) and how to handle 80 | conflicting audits. 81 | 82 | Also talked about incentives, like if most of the network is light 83 | clients, who will audit? Will that centralise things? Touched on 84 | hybrid models where you replicate and audit feeds in hops 1. -------------------------------------------------------------------------------- /002/meeting-notes-arj-keks-justin-2021-06-29: -------------------------------------------------------------------------------- 1 | 6 | 7 | # Bendy butt spec 8 | 9 | State more clearly what exactly is encrypted 10 | 11 | DM self: could be done by generating a random key from the seed. No 12 | need to generate a new asymetric key. Could even be used as a group 13 | key. 14 | 15 | # Fusion identity 16 | 17 | Who signs what needs to be a lot clearer in the document, and also 18 | what the fusion identity key is used for. 19 | 20 | Talk about doing a general domain separation doc, maybe include it in 21 | the BFE spec? 22 | 23 | One way to avoid using the key to both private messages and signing 24 | would be to include a key in the init message. 25 | 26 | Another idea (probably better) would be to instead of signing in the 27 | proof of key step, then do a pre-image instead. Then when you invite 28 | someone, you include a hash of the pre-image and in entrust message 29 | you share this pre-image. Proof of key is then posting the same 30 | hash. That way you don't need to sign anything. 31 | 32 | Justin mention an idea where you password protect a public tombstone 33 | "key" and than later if the identity is compromised and you need to 34 | tombstone it, you could include some password derived information that 35 | would basically tell the world that you knew what the password was as 36 | a proof of knowledge. This would be more information when deciding on 37 | what the attest. You wouldn't want to disclose the password directly 38 | as that would tell people what kind of password style you might have. 39 | 40 | The was a question around exactly how attestation works. That probably 41 | needs to be tidied a bit to make it clear in the different use cases, 42 | how that would work. -------------------------------------------------------------------------------- /002/metafeed-example1.svg: -------------------------------------------------------------------------------- 1 | 2 | 3 | metafeed 4 | 5 | 6 | 7 | a 8 | 9 | 10 | 11 | main 12 | 13 | 14 | 15 | b 16 | 17 | 18 | 19 | applications 20 | 21 | 22 | 23 | b:a->a:data 24 | 25 | 26 | 27 | 28 | 29 | 30 | -------------------------------------------------------------------------------- /002/metafeed-example1.svg.license: -------------------------------------------------------------------------------- 1 | SPDX-FileCopyrightText: 2021 Anders Rune Jensen 2 | 3 | SPDX-License-Identifier: CC-BY-4.0 -------------------------------------------------------------------------------- /002/metafeed-example2.svg: -------------------------------------------------------------------------------- 1 | 2 | 3 | Applications 4 | 5 | 6 | 7 | a 8 | 9 | 10 | 11 | App1 12 | 13 | 14 | 15 | b 16 | 17 | 18 | 19 | App2 20 | 21 | 22 | 23 | b:a->a:data 24 | 25 | 26 | 27 | 28 | 29 | 30 | -------------------------------------------------------------------------------- /002/metafeed-example2.svg.license: -------------------------------------------------------------------------------- 1 | SPDX-FileCopyrightText: 2021 Anders Rune Jensen 2 | 3 | SPDX-License-Identifier: CC-BY-4.0 -------------------------------------------------------------------------------- /002/presentation.md: -------------------------------------------------------------------------------- 1 | 6 | 7 | --- 8 | theme: gaia 9 | _class: lead 10 | paginate: true 11 | backgroundColor: #fff 12 | backgroundImage: url('https://marp.app/assets/hero-background.jpg') 13 | marp: true 14 | --- 15 | 16 | # Meta feeds :curling_stone: 17 | 18 | Presentation for SSB's NGI Pointer advisor meeting 19 | 20 | **Presenter:** Andre Staltz 21 | **Co-presenter:** Alexander Cobleigh 22 | **Author:** Anders Rune Jensen 23 | 24 | --- 25 | 26 | # Context 27 | 28 | Currently, all SSB feeds in production have: 29 | 30 | - An $Ed25519$ identity 31 | - Unbounded length 32 | - Messages of various types 33 | 34 | --- 35 | 36 | # Problem 37 | 38 | - An $Ed25519$ identity 39 | - Ties all of the content on this feed to this keypair 40 | - Unbounded length 41 | - As feeds grow, replication gets heavier 42 | - Messages of various types 43 | - Replicators have no choice in replicating only some types, it's either all or none 44 | 45 | --- 46 | 47 | # Problem 48 | 49 | No way to split content across separate feeds **while keeping the same identity**. 50 | 51 | --- 52 | 53 | # Solution 54 | 55 | - Introduce 2 *hierarchies* of feeds: 56 | - **Root feed** 57 | - **Subfeed** 58 | - Introduce 3 *kinds* of feeds: 59 | - **Meta feed** 60 | - **Content feed** 61 | - **Index feed** 62 | 63 | --- 64 | 65 | # Solution 66 | 67 | *Hierarchies* of feeds: 68 | 69 | - **Root feed** 70 | - Its *kind* is necessarily a **meta feed** 71 | - Functions as the **identity** of a person or user 72 | - **Subfeed** 73 | - Is either a **content feed**, an **index feed**, or another **meta feed** 74 | 75 | --- 76 | 77 | # Solution 78 | 79 | *Kinds* of feeds: 80 | 81 | - **Meta feed** 82 | - Publishes only messages with metadata on other feeds 83 | - **Content feed** 84 | - Publishes messages with actual content 85 | - **Index feed** 86 | - Publishes messages which point to (contain a cypherlink of) messages that belong to other feeds 87 | 88 | --- 89 | 90 | # Meta feed 91 | 92 | *Genesis* 93 | 94 | Uses [Hashed Key Derivation Function (HKDF)](https://en.wikipedia.org/wiki/HKDF) to generate a seed for $Ed25519$. 95 | 96 | ```js 97 | // JavaScript 98 | const seed = crypto.randomBytes(32) 99 | const prk = hkdf.extract(lhash, hash_len, seed, salt) 100 | const mf_info = "ssb-meta-feed-seed-v1:metafeed" 101 | const mf_seed = hkdf.expand(hash, hash_len, prk, length, mf_info) 102 | const mf_key = ssbKeys.generate("ed25519", mf_seed) 103 | ``` 104 | 105 | --- 106 | 107 | # Meta feed 108 | 109 | *Message schema* 110 | 111 | - Messages published on a meta feed have a schema that encodes a vocabulary to describe subfeeds 112 | - The schema covers: 113 | - **Identity** of the subfeed 114 | - **Lifetime** of the subfeed 115 | - **Purpose** of the subfeed 116 | - **Format** of the messages on the subfeed 117 | 118 | --- 119 | 120 | # Meta feed 121 | 122 | *Message schema* 123 | 124 | ```js 125 | { 126 | "type": "metafeed/operation", 127 | "id": "@1nf..A3U=.ed25519", // Identity 128 | "operation": "add", // Lifetime 129 | "purpose": "main", // Purpose 130 | "feedtype": "classic" // Type 131 | } 132 | ``` 133 | 134 | --- 135 | 136 | # Use case: partial replication 137 | 138 | - Given an existing old-style SSB feed, *main* 139 | - **Meta feed** publishes metadata on *main* as a subfeed 140 | - *Main* announces itself to belong to **meta feed** 141 | - **Meta feed** publishes metadata on a new **index feed** 142 | - The **index feed** is dedicated to a subset of messages on *main* 143 | - Replicators can replicate the **index feed** and fetch only that subset 144 | - E.g. an index feed for `about` messages 145 | 146 | --- 147 | 148 | # Use case: *same-as* 149 | 150 | - Given two old-style SSB feeds, *desktop* and *mobile* 151 | - **Meta feed** publishes metadata on *desktop* as a subfeed 152 | - **Meta feed** publishes metadata on *mobile* as a subfeed 153 | - *Desktop* announces itself to belong to **meta feed** 154 | - *Mobile* announces itself to belong to **meta feed** 155 | - Replicators following the **meta feed** are aware of both subfeeds 156 | 157 | --- 158 | 159 | # Use case: new feed formats 160 | 161 | - Backwards-compatible path to support new feed formats 162 | - Given an existing old-style SSB feed, *main* 163 | - Given a new SSB feed format, *neo* 164 | - **Meta feed** publishes metadata on *main* as a subfeed 165 | - **Meta feed** publishes metadata on *neo* as a subfeed 166 | - *Main* announces itself to belong to **meta feed** 167 | - *Neo* announces itself to belong to **meta feed** 168 | - Replicators following the **meta feed** are aware of both subfeeds 169 | 170 | --- 171 | 172 | # Open questions 173 | 174 | *Ephemeral feeds* 175 | 176 | The root (meta) feed is not partially replicated, and it's assumed to not support deletions (?), so the costs of created ephemeral just-in-time subfeeds is not negligible. How to efficiently support ephemeral feeds? 177 | 178 | --- 179 | 180 | # Open questions 181 | 182 | *Fragility of the root feed* 183 | 184 | What happens when $Ed25519$, the scheme used by the root feed, is broken? How do we guarantee the long-term survival of the specifications on the root feed? 185 | 186 | --- 187 | 188 | # Open questions 189 | 190 | *Jevons paradox* 191 | 192 | > Increasing the efficiency with which a resource is used, but the rate of consumption of that resource rises due to increasing demand. 193 | 194 | E.g. with meta feeds in SSB, now that it's "cheap" to create subfeeds for alternative feed formats etc, will we have *more* storage consumed (as opposed to *less*, which is the goal of partial replication)? 195 | 196 | --- 197 | 198 | # Thank you 199 | 200 | # :pray: 201 | 202 | Link: https://github.com/ssb-ngi-pointer/ssb-meta-feed 203 | 204 | -------------------------------------------------------------------------------- /002/sharding-math.md: -------------------------------------------------------------------------------- 1 | 6 | 7 | # Sharding math 8 | 9 | ## Abstract 10 | 11 | In the tree of metafeeds, sharding is a technique of using intermediate nodes in the tree to group the leaf nodes into clusters. Peers who replicate a portion of the metafeed tree are thus not forced to know about all leaf feeds. 12 | 13 | However, what is the optimal sharding ratio - one that minimizes unnecessary replicated messages and feeds? 14 | 15 | ## Definitions 16 | 17 | Let $Feeds$ be the total number of content feeds, and $Feeds'$ be the number of content feeds which the current peer is going to replicate. $Feeds' \leq Feeds$. 18 | 19 | Let $Shards$ be the number of shard feeds, and $Shards'$ the number of shard feeds the current peer is going to replicate. 20 | 21 | $Replicating$ refers to the number of feeds (either content feeds or metafeeds) in the tree that are replicated. 22 | 23 | $Overhead$ is the total number of metafeed messages replicated. It includes messages from the root metafeed and from the shard metafeeds. Does not include messages from content feeds. 24 | 25 | If we replicate a shard feed, we say that we become *aware* of a content feed via the message on the shard feed declaring the content feed. 26 | 27 | $Awareness$ is the total number of content feeds that we become aware of by replicating shard feeds. 28 | 29 | ## Theorem 1 30 | 31 | $Replicating = Feeds' + Shards' + 1$. 32 | 33 | **Proof:** the root metafeed is always replicated, thus it counts as $1$. By definition, the only other types of feeds are content feeds and shard feeds, of which the ones we are replicating are $Feeds'$ and $Shards'$, respectively. ∎ 34 | 35 | ## Theorem 2 36 | 37 | $Overhead = Shards + Awareness$. 38 | 39 | **Proof:** we always replicate the root metafeed, which means we fetch all of its messages. Since the root metafeed only announces the existence of shard feeds, the number of messages in the root metafeed is $Shards$. 40 | 41 | The other type of metafeed that we replicate are shard feeds. By definition, $Awareness$ is the number of content feeds under the shard feeds that we're replicating. Since shard feeds only announce the existence of content feeds, the number of messages in a shard correspond to the number of content feeds under that shard. 42 | 43 | Overhead are the total number of metafeed messages, thus metafeed messages from the root ( $Shards$ ) plus metafeed messages from all shards ( $Awareness$ ). ∎ 44 | 45 | ## Case studies 46 | 47 | ### Average sharding 48 | 49 | | $Feeds=6$ | $Feeds'=3$ | | $Shards=3$ | $Shards'=2$ 50 | |-|-|-|-|-| 51 | 52 | ```mermaid 53 | graph TB; 54 | root-->A-->1 & 2 55 | root-->B-->3 & 4 56 | root-->C-->5 & 6 57 | 58 | root:::repl 59 | A:::repl 60 | B:::repl 61 | 1:::repl 62 | 2:::repl 63 | 4:::repl 64 | 65 | classDef repl fill:#9f9,stroke:#090; 66 | classDef default fill:#eee,stroke:#999,color:#999; 67 | 68 | linkStyle 7 opacity:0.2; 69 | linkStyle 8 opacity:0.2; 70 | ``` 71 | 72 | - $Replicating = 6$ 73 | - $Awareness = 4$ 74 | - $Overhead = 7$ 75 | 76 | ## Min sharding 77 | 78 | | $Feeds=6$ | $Feeds'=3$ | | $Shards=1$ | $Shards'=1$ | 79 | |-|-|-|-|-| 80 | 81 | ```mermaid 82 | graph TB; 83 | root-->A-->1 & 2 & 3 & 4 & 5 & 6 84 | 85 | root:::repl 86 | A:::repl 87 | 1:::repl 88 | 2:::repl 89 | 4:::repl 90 | 91 | classDef repl fill:#9f9,stroke:#090; 92 | classDef default fill:#eee,stroke:#999,color:#999; 93 | ``` 94 | 95 | - $Awareness = 6$ 96 | - $Replicating = 3 + 1 + 1 = 5$ 97 | - $Overhead = 1 + 6 = 7$ 98 | 99 | In the general case, min sharding has: 100 | 101 | | Formula | Conclusion | 102 | |--|--| 103 | | $Replicating = Feeds' + 2$ | :slightly_smiling_face: | 104 | | $Awareness = Feeds$ | :neutral_face: | 105 | | $Overhead = Feeds + 1$ | :grimacing: | 106 | 107 | Min sharding minimizes $Shards$ but **maximizes** $Awareness$ which leaves us with an overall large $Overhead$. 108 | 109 | ## Max sharding 110 | 111 | | $Feeds=6$ | $Feeds'=3$ | 112 | |-|-| 113 | | $Shards=Feeds=6$ | $Shards'=Feeds'=3$ | 114 | 115 | ```mermaid 116 | graph TB; 117 | root-->A-->1 118 | root-->B-->2 119 | root-->C-->3 120 | root-->D-->4 121 | root-->E-->5 122 | root-->F-->6 123 | 124 | root:::repl 125 | A:::repl 126 | B:::repl 127 | D:::repl 128 | 1:::repl 129 | 2:::repl 130 | 4:::repl 131 | 132 | classDef repl fill:#9f9,stroke:#090; 133 | classDef default fill:#eee,stroke:#999,color:#999; 134 | 135 | linkStyle 5 opacity:0.2; 136 | linkStyle 9 opacity:0.2; 137 | linkStyle 11 opacity:0.2; 138 | ``` 139 | 140 | - $Awareness = 3$ 141 | - $Replicating = 3 + 3 + 1 = 7$ 142 | - $Overhead = 6 + 3 = 9$ 143 | 144 | In the general case, max sharding has: 145 | 146 | | Formula | Conclusion | 147 | |--|--| 148 | | $Replicating = 2×Feeds' + 1$ | :grimacing: | 149 | | $Awareness = Feeds'$ | :slightly_smiling_face: | 150 | | $Overhead = Feeds + Feeds'$ | :grimacing: | 151 | 152 | Max sharding minimizes $Awareness$ but **maximizes** $Shards$ which leaves us with an overall large $Overhead$. 153 | 154 | ## Scenario: $Shards < Feeds'$ 155 | 156 | Because $Shards' \leq Shards$, it follows that $Shards' \leq Feeds'$. 157 | 158 | So 159 | 160 | $$ 161 | Replicating = Feeds' + Shards' + 1 162 | $$ 163 | 164 | becomes 165 | 166 | $$ 167 | Replicating \leq 2×Feeds' + 1 168 | $$ 169 | 170 | which means that $Replicating = O(Feeds')$. 171 | 172 | Further, because $Shards < Feeds'$ 173 | 174 | $$ 175 | \frac{Feeds'}{Shards} > 1 176 | $$ 177 | 178 | This means an **even distribution of feeds across shards** is highly likely to cause $Awareness = Feeds$, which leads to a large $Overhead$. 179 | 180 | ## Realistic case with 16 shards (4-bit) 181 | 182 | | $Feeds=128$ | $Feeds'=32$ | $Shards=16$ | 183 | |-|-|-| 184 | 185 | It's reasonable to assume that one user has a dozens of apps, and up to a hundred private groups. Let's set $Feeds$ at 128, as a convenient power of two. Let's set $Feeds'$ at 32, which means a few apps and a few dozen groups. 186 | 187 | Because we have 16 shards and 128 feeds, there are on average 8 feeds in each shard (assuming random shard allocation). $\alpha = \frac{Feeds}{Shards} = 8$ 188 | 189 | Because we have 32 feeds-to-replicate and 16 shards, there are on average 2 chosen feeds in each shard. $\alpha' = \frac{Feeds'}{Shards} = 2$. Sometimes there are 0 chosen feeds in a shard. So let's assume $Shards' = 14$. 190 | 191 | This means: 192 | 193 | | Formula | | | Conclusion | 194 | |--|--|--|-| 195 | | $Replicating =$ | $Feeds' + 14 + 1 =$ | $47$ | :slightly_smiling_face: | 196 | | $Awareness =$ | $Shards'×\alpha =$ | $112$ | :neutral_face: | 197 | | $Overhead =$ | $Shards + 112 =$ | $128$ | :grimacing: | 198 | 199 | ## Realistic case with 64 shards (6-bit) 200 | 201 | | $Feeds=128$ | $Feeds'=32$ | $Shards=64$ | 202 | |-|-|-| 203 | 204 | $\alpha = 2$ 205 | 206 | $\alpha' = 0.5$ 207 | 208 | $Shards' \leq 0.5×Shards = 32$ 209 | 210 | This means: 211 | 212 | | Formula | | | Conclusion | 213 | |--|--|--|-| 214 | | $Replicating \leq$ | $32 + 32 + 1 =$ | $65$ | :slightly_smiling_face: | 215 | | $Awareness \leq$ | $Shards' × \alpha =$ | $64$ | :slightly_smiling_face: | 216 | | $Overhead \leq$ | $Shards + 64 =$ | $128$ | :grimacing: | 217 | 218 | ## Realistic case with 256 shards (8-bit) 219 | 220 | | $Feeds=128$ | $Feeds'=32$ | $Shards=256$ | 221 | |-|-|-| 222 | 223 | In this case there are more shards than content feeds, so this most likely means 224 | that max sharding occurs. 225 | 226 | $\alpha = 1$ 227 | 228 | $\alpha' = 0.25$ 229 | 230 | $Shards' = Feeds' = 32$ 231 | 232 | This means: 233 | 234 | | Formula | | | Conclusion | 235 | |--|--|--|-| 236 | | $Replicating =$ | $32 + 32 + 1 =$ | $65$ | :slightly_smiling_face: | 237 | | $Awareness =$ | $Feeds' =$ | $32$ | :slightly_smiling_face: | 238 | | $Overhead =$ | $Feeds + Feeds' =$ | $160$ | :grimacing: | 239 | 240 | ## Realistic Case: clumping 241 | 242 | | $Feeds=128$ | $Feeds'=32$ | $Shards=16$ | 243 | | --------- | --------- | --------- | 244 | 245 | Assume Staltz has dozens of apps, and up to a hundred private groups. So lets set his $Feeds$ at 128, as a convenient power of two. 246 | 247 | Mix wants to replicate some subset of those (he only uses a couple of apps, and isn't in the same groups as Staltz) - set $Feeds'$ at 32. 248 | 249 | Let us assume that each Application/ group consists of 3 feeds, and it' "clumps" these into the same shard. 250 | 251 | So we have 32 / 3 ~= 11 clumps 252 | 253 | Expected number of groups 11 clumps would randomly land on with 16 shards: 254 | ``` 255 | 1 256 | 2 257 | 3 258 | 4 259 | 5 ✓ 260 | 6 ✓✓✓✓✓✓✓ 261 | 7 ✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓ 262 | 8 ✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓ 263 | 9 ✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓ 264 | 10 ✓✓✓✓✓✓✓✓✓✓✓ 265 | 11 ✓✓ 266 | 12 267 | 13 268 | 14 269 | 15 270 | 16 271 | ``` 272 |
273 | 274 | (see code) 275 | 276 | 277 | 278 | ```javascript 279 | const shards = 16 280 | const clumps = 11 281 | 282 | const shardsUsedCount = {} 283 | 284 | for (let i = 0; i < 1000; i++) { 285 | const shardsPicked = new Set() 286 | for (let c = 0; c < clumps; c++) { 287 | const shardPick = Math.floor((Math.random() * shards)) + 1 288 | shardsPicked.add(shardPick) 289 | } 290 | 291 | if (!shardsUsedCount[shardsPicked.size]) shardsUsedCount[shardsPicked.size] = 0 292 | shardsUsedCount[shardsPicked.size]++ 293 | } 294 | 295 | for (let s = 1; s <= shards; s++) { 296 | const count = Math.round((shardsUsedCount[s] || 0) / 6) 297 | console.log( 298 | s.toString().padStart(2), 299 | new Array(count).fill('✓').join('') 300 | ) 301 | } 302 | ``` 303 |
304 | 305 | 306 | So, about 8 of our shards need replicating (half in this case). 307 | Which means our Total Awareness is at about 50%, which seems great! 308 | 309 | If each app is a clump of 2 feeds on average, we need to replicate 10/16 shards ~= 60% 310 | 311 | - $Shards'=8$ 312 | - $\alpha = 8$ 313 | - $\alpha' = 2$ 314 | - $Replicating = 32 + 8 + 1 = 41$ 315 | - $Awareness = Shards'×\alpha = 8×8 = 64$ 316 | - $Overhead = Shards + 64 = 80$ 317 | 318 | ## Conclusion 319 | 320 | It's important to minimize both $Replicating$ and $Overhead$. But $Overhead$ is directly proportional to $Awareness$, which means we must minimize $Awareness$. We know that $Awareness$ is at its lowest when $Shards$ is at its highest, but $Overhead$ is also directly proportional to $Shards$, so we must minimize $Shards$ too. 321 | 322 | Maybe we should aim for $Feeds' < Shards < Feeds$ as a general rule? 323 | 324 | $Shards = 1 < Feeds' < Feeds$ is **min sharding** where the overhead is O(M) which is pretty bad. The "realistic case $Shards=8$" is not min sharding, but it is quite close to min sharding because $Shards < Feeds' < Feeds$, and overhead is pretty bad. 325 | 326 | On the other hand, $Feeds' < Shards = Feeds$ is **max sharding**, and the result is even worse, we end up with $O(Feeds+Feeds')$ overhead and $O(2×Feeds')$ feeds to replicate. 327 | 328 | $Shards = Feeds' < Feeds$ is somewhat an optimal situation ("realistic case `S=64`") but it quickly becomes $Shards < Feeds'$ when $Feeds'$ increases over time (e.g. joining new groups). 329 | 330 | So we want $Shards$ to be greater than $Feeds'$, but *significantly* smaller than $Feeds$. Thus $Feeds' < Shards < Feeds$. 331 | 332 | However, if we have clumping and noticing that $Shards = Feeds'$ is an optimal solution, we might have low overhead after all if $Shards < Feeds'$, as long as we avoid min sharding. 333 | -------------------------------------------------------------------------------- /003.md: -------------------------------------------------------------------------------- 1 | # Index feeds 2 | 3 | Author: Anders Rune Jensen 4 | 5 | Date: 2021-10-11 6 | 7 | License: CC0-1.0 8 | 9 | ## Abstract 10 | 11 | This document outlines how index feeds are inserted into a meta feed 12 | along with the structure of index feed messages. To understand how 13 | index feeds are used to migrate from classic feeds to metafeeds see 14 | [migration spec]. 15 | 16 | ## Specification 17 | 18 | There are two versions of spec for "index feeds". Version 0 should be 19 | considered deprecated and does not need to be supported. Version 1 20 | should be implemented. 21 | 22 | ### Version 1 23 | 24 | An index feed is a feed format that in practice is essentially the classic 25 | feed format, but has different SSB URIs for feeds and messages, to carry 26 | different semantics and allow peers to easily distinguish them. 27 | 28 | An index feed MUST be referred by the SSB URI `ssb:feed/indexed-v1/__` and 29 | its messages MUST be referred by the SSB URI `ssb:message/indexed-v1/__`. 30 | 31 | Similarly, when a metafeed is pointing to a child index feed, it MUST use 32 | `indexed-v1` BFE identifiers. Example: 33 | 34 | ``` 35 | { 36 | "type" => "metafeed/add/derived", 37 | "feedpurpose" => "index", 38 | "subfeed" => (An indexed-v1 BFE feed ID), 39 | "querylang" => "ssb-ql-0", 40 | "query" => '{"author":"@main.ed25519","type":"contact"}' 41 | } 42 | ``` 43 | 44 | The query language employed SHOULD be [ssb-ql-0]. 45 | 46 | The message `content` published by an indexed-v1 feed MUST have the shape: 47 | 48 | ```js 49 | { 50 | type: 'metafeed/index', 51 | indexed: '%msgkey' 52 | } 53 | ``` 54 | 55 | ### Version 0 56 | 57 | Indexes is a meta feed of feeds linking to a subset of messages in the 58 | main feed. These can be used for partial replication of a subset of 59 | the main feed. 60 | 61 | The feeds inside this meta feed should only contain hashes of the 62 | original messages as their content. 63 | 64 | Applications using the main feed should create at least two index 65 | feeds: 66 | 67 | ``` 68 | { 69 | "type" => "metafeed/add", 70 | "feedpurpose" => "index", 71 | "subfeed" => (Some BFE feed ID), 72 | "querylang" => "ssb-ql-0", 73 | "query" => '{"author":"@main.ed25519","type":"contact"}' 74 | } 75 | 76 | { 77 | "type" => "metafeed/add", 78 | "feedpurpose" => "index", 79 | "subfeed" => (Another BFE feed ID), 80 | "querylang" => "ssb-ql-0", 81 | "query" => '{"author":"@main.ed25519","type":"about"}' 82 | } 83 | ``` 84 | 85 | For the definition of the query language see [ssb-ql-0]. 86 | 87 | Index message format in a classic SSB feed: 88 | 89 | ```js 90 | { 91 | type: 'metafeed/index', 92 | indexed: { 93 | key: '%msgkey', 94 | sequence: 42 95 | } 96 | } 97 | ``` 98 | 99 | The `indexed.sequence` might seem redundant but it might be much cheaper 100 | for an implementation to resolve `author@sequence` and checking the hash 101 | of the message then keeping a total `hash => message` index. 102 | 103 | ## References 104 | 105 | ### Normative 106 | 107 | - [ssb-ql-0] 108 | 109 | ### Informative 110 | 111 | - [migration spec] 112 | 113 | ### Implementation 114 | 115 | - [ssbc/ssb-index-feeds](https://github.com/ssbc/ssb-index-feeds/) in JavaScript 116 | 117 | [ssb-ql-0]: https://github.com/ssb-ngi-pointer/ssb-subset-replication-spec#ssb-ql-0 118 | [migration spec]: https://github.com/ssbc/ssb-meta-feeds-migration 119 | -------------------------------------------------------------------------------- /003/audit-trustnet.md: -------------------------------------------------------------------------------- 1 | 6 | 7 | ## Index audits 8 | 9 | Indexes can be seen as a *claim*, in that, these are the messages 10 | matching a query. It is important that these are accurate and thus 11 | other nodes in the network should audit these. Thus catching malicious 12 | nodes but also simple programming errors. 13 | 14 | An auditor verifies an index by being in possesion of the same mesages 15 | and verifies that no messages are left out and that the index is not 16 | growing stale. Anyone can be an auditor, trust is assigned to auditors 17 | as described in the next section. This should keep bad auditors out of 18 | the system. 19 | 20 | After verifying an index feed, a message is posted on the audit feed: 21 | 22 | ``` 23 | { 24 | type: 'index/audit', 25 | latestseq: x, 26 | subfeed: '@index1.ed25519', 27 | metafeed: '@mf.bbfeed-v1', 28 | status: 'verified' 29 | } 30 | ``` 31 | 32 | Because the messages on feeds are immutable, once a feed has been 33 | verified up until sequence x, the past can never change. This holds 34 | true because the network will only accept new messages on a feed that 35 | correctly extends the existing feed. 36 | 37 | In order not to create too many verification messages, a new message 38 | should only be posted if an index is no longer valid or it has grown 39 | stale. How often indexes should be verified is at the discretion of 40 | the auditor. 41 | 42 | An index feed is considered stale if that has not been updated a week 43 | after a message is posted on the indexed feed. These should be posted 44 | as: 45 | 46 | ``` 47 | { 48 | type: 'index/audit', 49 | latestseq: x, 50 | subfeeed: '@index2.ed25519', 51 | metafeed: '@mf.bbfeed-v1', 52 | status: 'stale', 53 | reason: 'not updated since 2021-06-25' 54 | } 55 | ``` 56 | 57 | In case an index is invalid, the following kind of message should be 58 | posted: 59 | 60 | ``` 61 | { 62 | type: 'index/audit', 63 | latestseq: x, 64 | subfeeed: '@index2.ed25519', 65 | metafeed: '@mf.bbfeed-v1', 66 | status: 'invalid', 67 | reason: 'Missing the message %hash.sha256 with sequence 100' 68 | } 69 | ``` 70 | 71 | In the case where an index is no longer valid, the index feed and all 72 | messages referenced from this feed should be removed from the local 73 | database. After this, another index needs to be found an used instead. 74 | 75 | It is worth noting that it is limited what a malicious peer could 76 | do. The messages referenced in an index still needs to be signed by 77 | the author, so at worst messages can be left out. 78 | 79 | ## Trust 80 | 81 | Trust is a meta feed that contains one feed for each trust area with 82 | ratings within that area as defined in [trustnet]. One area where this 83 | will be used is for delegating trust related to verification of 84 | indexes: 85 | 86 | ``` 87 | { 88 | type: 'metafeed/add', 89 | feedpurpose: 'trustassignments', 90 | subfeed: '@assignments.ed25519', 91 | area: 'indexaudits' 92 | } 93 | ``` 94 | 95 | A trust assignment: 96 | 97 | ``` 98 | { 99 | type: 'trustnet/assignment', 100 | src: '@mf.bbfeed-v1', 101 | dest: '@othermf.bbfeed-v1', 102 | weight: 1.0 103 | } 104 | ``` 105 | 106 | Notice we use the meta feed as the destination. The index subfeed of 107 | the destination is given by the area and layout of meta feeds. 108 | 109 | ## Replication 110 | 111 | If that is not found, one uses trusted index audits feeds to find an 112 | index that can be used. Trusted is defined as: 113 | 114 | A target feed is trusted if: 115 | - One has assigned any positive, non-zero amount of trust to the 116 | target feed, or 117 | - the trustnet calculation returns the feed as trusted 118 | 119 | A trustnet calculation is performed as: 120 | - All positive, non-zero first order (direct) trust assignments are always 121 | returned as trusted. 122 | - If there are no first-order trust assignments with a trust weight exceeding 123 | `TrustNet.trustThreshold`, computation is shortcircuited and only the first 124 | order trust assignments are returned. 125 | - If there is at least one first-order trust assignment exceeding the trust 126 | threshold, then [Appleseed] ranks are calculated for the given trust graph. 127 | - TrustNet's trusted feeds are then calculated by breaking the Appleseed 128 | rankings into 3 ckmeans clusters, and discarding the cluster of with the lowest ranks. 129 | - If any direct trust assignments are not included in the top 2 clusters, then 130 | they are added to the concatenation of the top 2 clusters before returning 131 | the result as the trustnet calculation. 132 | 133 | If no verified indexes are available one should fall back to full 134 | replication of that main feed. 135 | 136 | A random new user will at first not trust anyone and thus can't do 137 | partial replication. On the other hand if they are onboarded using 138 | someone they know and trust that would enable partial replication. 139 | 140 | Lets look at how onboarding could work for Alice that got invited by 141 | Bob. First alice downloads Bobs main feed. She then trusts Bob and 142 | downloads Bobs meta feed, all trust assignments, indexes and 143 | audits. By using the metafeed field on trust assignments, Alice is 144 | able to recursively download trust assignments, their audits and from 145 | that decide what indexes to be used. 146 | 147 | [Appleseed]: https://github.com/cblgh/appleseed-metric 148 | [trustnet]: https://github.com/cblgh/trustnet 149 | -------------------------------------------------------------------------------- /003/feedless-identity.md: -------------------------------------------------------------------------------- 1 | 6 | 7 | **Notice**: this concept/draft was merged into https://github.com/ssb-ngi-pointer/fusion-identity-spec. This is just still here for archival purposes. 8 | 9 | Feedless identity is a concept where multiple regular identities are 10 | linked together with a specific purpose. The identity will have keypair that is 11 | shared with the linked members. The reason it is called feedless is 12 | because, as the name implies, all messages related to the identity is 13 | posted on the feeds of the members of an feedless identity. 14 | 15 | Each identity should have a feed containing messages related to 16 | feedless identities it is linked to. Each feedless identity works by 17 | mutual consent, meaning as long as all the members of an identity are 18 | mutually linked, the identity is considered valid. Any member of the 19 | identity can revoke the validity of an identity by creating a 20 | tombstone message. 21 | 22 | To create a new feedless identity, a keypair is created and the 23 | identity is announced: 24 | 25 | ``` 26 | { type: 'feedless/create', identity: '@id', name: 'arj' } 27 | ``` 28 | 29 | The identity can then be linked between identities: 30 | 31 | ``` 32 | { type: 'feedless/link', identity: '@id', from: '@mf', to: '@othermf', tangles: { feedless: { root: '%abc', previous: '%abcd' } } } 33 | ``` 34 | 35 | Once @othermf posts a similar message in their feedlessidentity 36 | subfeed, the identity is linked and the creator of the identity should 37 | send the private key of the identity to the new member. Messages of a 38 | feedless identity are tangled using the initial message as the root. 39 | 40 | The identity can be extended with new members by having any currrent 41 | member linking the new member and the new new member linking back. The 42 | reason why we don't need full consensus from existing members is that 43 | it slows down the process of adding new members. One could always 44 | build an internal process within an identity and if someone sidesteps 45 | this process it is public knowledge. In any case it seems like a 46 | bigger issue that somewould would leak the secret key to a person 47 | outside the group. 48 | 49 | Any member can revoke the feedless identity by posting the following 50 | message: 51 | 52 | ``` 53 | { type: 'feedless/tombstone', identity: '@id', tangles: { feedless: { root: '%abc', previous: '%abcd' } } } 54 | ``` 55 | 56 | Once another member sees this message they should also post a 57 | tombstone message, this is to make it harder for an adversary to try 58 | and keep the identity alive by withholding new messages after one of 59 | the identities has been compromised. It is up to the members to create 60 | a new feedless identity as the existing is now considered void and 61 | should not be used. 62 | 63 | For the name to be changed, it must be done in a consensus fashion, 64 | meaning all members must post a message acking the name: 65 | 66 | ``` 67 | { type: 'feedless/name', identity: '@id', name: 'arj', tangles: { feedless: { root: '%abc', previous: '%abcd' } } } 68 | ``` 69 | 70 | It might also be possible to operate with identities where instead of 71 | full censensus only a quorum (such as 2/3) is needed for a name 72 | change. 73 | 74 | A new identity added to the feedless identity can merge these messages 75 | by including the name in the link message. 76 | 77 | These feedless identities thus act as public groups. They work best 78 | for smaller groups where the members should be publicly known. For 79 | larger groups, [private-groups] should be considered instead. 80 | 81 | If a feedless identity A is added as a member of another feedless 82 | identity B (group of groups) then just one member of A needs to link 83 | back. A new `representative` field of the link message from B should 84 | point to any member of A. It doesn't have to be the member linking 85 | back as the list of members can be deduced from that member. 86 | 87 | For a good starting point for existing discussions on SSB going back 5 88 | years (linked in the thread): 89 | %YaWEWHDWAY6p/g9zIwCJovsd1SUyHpwuGGz3Ug/jtW8=.sha256 90 | 91 | Mix & arj talk (also added to this repo): 92 | - https://hackmd.io/iQlUz8nBSeecrrb5mQtcUQ 93 | - https://hackmd.io/AHdW4Z6lTAqEfPxEkSuSvQ?both 94 | -------------------------------------------------------------------------------- /004.md: -------------------------------------------------------------------------------- 1 | # Bendy Butt 2 | 3 | Author: Anders Rune Jensen 4 | 5 | Date: 2021-06-15 6 | 7 | License: CC0-1.0 8 | 9 | ## Abstract 10 | 11 | What is Bendy Butt? 12 | 13 | > A new feed format designed for [meta feeds] 14 | 15 | Why a new feed format, when we already have [gabby grove] and [bamboo]? 16 | 17 | > Because we would want to offer a feed format that is simple and easy 18 | > to implement in any programming language by reusing an established 19 | > format ([bencode]) with existing encoder libraries. Furthermore, as 20 | > meta feeds makes it easier to have multiple feed formats, we wanted 21 | > to show that creating a new feed format does not have to be hard. 22 | 23 | Can't you just use the classic format? 24 | 25 | > That would mean clients would have to support reading and writing 26 | > the classic format forever, even for applications that only use 27 | > newer feed formats. 28 | 29 | What is wrong with the classic format? 30 | 31 | > There are multiple problems, most of them come from the fact that it 32 | > is very tied to the internals of the v8 engine for Javascript. JSON 33 | > is good as an exchange format for values of data but not well suited 34 | > to cryptographically sign messages where a canonical representation 35 | > of each value is paramount. 36 | 37 | Can I use Bendy Butt for purposes other than meta feeds? 38 | 39 | > Yes, but Bendy Butt was not designed as a general purpose feed 40 | > format. We do not intend to extend Bendy Butt to cover any other 41 | > use cases besides meta feeds, but if the feed format constraints 42 | > are sufficient for your use cases, nothing stops you from using it. 43 | > To preserve the simplicity of this spec, we recommend not sending 44 | > patches to the spec to generalize it. 45 | 46 | ## Terminology 47 | 48 | The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", 49 | "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be 50 | interpreted as described in RFC 2119. 51 | 52 | **Notations for bencode and BFE** 53 | 54 | To avoid confusion with JSON, we shall use the notation `{ "a" => 1, "b" => 2 }` 55 | to denote a [bencode] *dictionary* with two entries: key "a" mapping to value 56 | `1` and key "b" mapping to value `2`. We shall use the notation `[a, b, c]` to 57 | denote a [bencode] *list* of three values `a`, `b`, and `c`. 58 | 59 | For sequences of bytes, we use the notation `<03 02 01>` to represent the 60 | sequence of bytes `03` followed by `02` followed by `01`. We use the notation 61 | `a + b + c` to denote *byte concatenation* of the byte sequences `a` followed by 62 | `b` followed by `c`. In other words `<05 04> + <03 02 01>` equals 63 | `<05 04 03 02 01>`. 64 | 65 | For abstract substitutions, we use `(something)` to denote whatever `something` 66 | describes. In other words, anything wrapped in parentheses is just a placeholder 67 | for something else. 68 | 69 | ## Specification 70 | 71 | A *Bendy Butt message* is encoded in [bencode] as a list of message `payload` 72 | and `signature`, which we shall denote `[payload, signature]`. SSB Binary Field 73 | Encodings ([SSB-BFE]) is used to encode all non-list non-dictionary non-integer 74 | values contained in the `payload` and `signature`. BFE also helps to 75 | disambiguate binary data like feed IDs and message IDs from generic data types 76 | such as strings. 77 | 78 | The message `payload` is a bencode list with 5 elements in this specific order, 79 | in other words, `[author, sequence, previous, timestamp, contentSection]`: 80 | 81 | 1) `author`, a BFE-encoded "feed ID" 82 | 2) `sequence`, a [bencode integer] greater than or equal to 1 83 | 3) `previous`, a BFE-encoded "message ID" of the previous message 84 | on the feed. For the first message this must be the BFE "nil" generic data 85 | type. 86 | 4) `timestamp`, a [bencode integer] representing the UNIX epoch timestamp of message 87 | creation. 88 | 5) `contentSection` can be one of two possible shapes: 89 | 1. `[content, contentSignature]` such that `content` is any bencode 90 | dictionary representing the message contents, all of which are internally 91 | encoded with BFE; and `contentSignature` is a BFE-encoded "signature". The 92 | signature is applied on `prefix + content`, where `prefix` is the string 93 | "bendybutt" encoded as UTF8 bytes. The signature uses some cryptographic 94 | keypair which **MAY** be different from the `author` keypair (for example, 95 | under the [meta feeds] spec, this will be the subfeed keypair). 96 | 2. BFE-encoded "encrypted data". Upon decryption, it takes the shape 97 | `[content, contentSignature]` as described above. 98 | 99 | The message `signature` is applied on `payload` using the `author`'s 100 | cryptographic keypair. 101 | 102 | Both the `signature` and the `contentSignature` use the same [HMAC signing capability] 103 | (`sodium.crypto_auth`) and `sodium.crypto_sign_detached` as in the classic 104 | SSB format (ed25519). 105 | 106 | The key or ID of a Bendy Butt message is the SHA256 hash of the bencoded bytes 107 | for `[payload, signature]`, i.e. `key = SHA256([payload, signature])`. 108 | 109 | ### Example 110 | 111 | Following our `[a, b]` and `{ "a" => 1, "b" => 2}` notation, which may look 112 | similar to JSON, here is a Bendy Butt message with the `content` dictionary 113 | equal to `{ "type" => 'greet', "text" => 'Good morning!' }`. 114 | 115 | ``` 116 | [ 117 | [ 118 | <00 03 5c 27 ac 6e f0 cd fb d0 f8 9a 89 a1 b6 5a 36 04 77 a3 3e c7 9c b7 ab 14 cd 90 76 25 59 be e2 ff>, 119 | 1, 120 | <06 02>, 121 | 12345, 122 | [ 123 | { 124 | "type" => <06 00 67 72 65 65 74>, 125 | "text" => <06 00 47 6f 6f 64 20 6d 6f 72 6e 69 6e 67 21> 126 | } 127 | <04 00 51 a6 7a 43 6a 66 f6 6d e0 3d 77 73 c0 b7 ba 98 84 61 32 46 c6 ee 6c 74 1b 1d 9e 59 18 24 b3 c7 1d a3 ec 35 bf e0 32 cf 86 55 7c f8 72 30 e9 56 ... 16 more bytes> 128 | ] 129 | ], 130 | <04 00 6d 57 9f 55 14 d2 d8 69 09 ad 7b 31 f8 24 4f a7 fc 6a 0d c1 1e f4 1a 92 71 86 fb 8d 1b fc d5 17 b3 88 05 f0 a6 48 aa ba 24 f4 46 b0 9e 65 64 b6 ... 16 more bytes> 131 | ] 132 | ``` 133 | 134 | Notice that is has the shape: 135 | 136 | ``` 137 | [ 138 | [ 139 | author, 140 | sequence, 141 | previous, 142 | timestamp, 143 | [ 144 | content, 145 | contentSignature 146 | ] 147 | ], 148 | signature 149 | ] 150 | ``` 151 | 152 | Here is the same data as above, but displayed in raw binary (hexadecimal): 153 | 154 | ``` 155 | Offset 00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F 156 | 000000 6C 6C 33 34 3A 00 03 5C 27 AC 6E F0 CD FB D0 F8 ll34:..\'¬nðÍûÐø 157 | 000010 9A 89 A1 B6 5A 36 04 77 A3 3E C7 9C B7 AB 14 CD ..¡¶Z6.w£>Ç.·«.Í 158 | 000020 90 76 25 59 BE E2 FF 69 31 65 32 3A 06 02 69 31 .v%Y¾âÿi1e2:..i1 159 | 000030 32 33 34 35 65 6C 64 34 3A 74 65 78 74 31 35 3A 2345eld4:text15: 160 | 000040 06 00 47 6F 6F 64 20 6D 6F 72 6E 69 6E 67 21 34 ..Good morning!4 161 | 000050 3A 74 79 70 65 37 3A 06 00 67 72 65 65 74 65 36 :type7:..greete6 162 | 000060 36 3A 04 00 51 A6 7A 43 6A 66 F6 6D E0 3D 77 73 6:..Q¦zCjfömà=ws 163 | 000070 C0 B7 BA 98 84 61 32 46 C6 EE 6C 74 1B 1D 9E 59 À·º..a2FÆîlt...Y 164 | 000080 18 24 B3 C7 1D A3 EC 35 BF E0 32 CF 86 55 7C F8 .$³Ç.£ì5¿à2Ï.U|ø 165 | 000090 72 30 E9 56 8E D5 7B 25 F6 77 FE 58 3B 17 3D BD r0éV.Õ{%öwþX;.=½ 166 | 0000A0 E7 08 82 0F 65 65 36 36 3A 04 00 6D 57 9F 55 14 ç...ee66:..mW.U. 167 | 0000B0 D2 D8 69 09 AD 7B 31 F8 24 4F A7 FC 6A 0D C1 1E ÒØi.­{1ø$O§üj.Á. 168 | 0000C0 F4 1A 92 71 86 FB 8D 1B FC D5 17 B3 88 05 F0 A6 ô..q.û..üÕ.³..ð¦ 169 | 0000D0 48 AA BA 24 F4 46 B0 9E 65 64 B6 9A DE 97 F9 18 Hªº$ôF°.ed¶.Þ.ù. 170 | 0000E0 04 AF 5F 7A F3 5E 5D 4B FD 85 0B 65 .¯_zó^]Ký..e 171 | ``` 172 | 173 | 174 | ### Validation 175 | 176 | For validation we differentiate between *message validity* and *content 177 | validity*. Since the content can be encrypted, there is potentially no way to 178 | validate that. This means a message should only be rejected before insertion 179 | into the local database if it fails the message validation rules. A message 180 | should not be included in the bendy butt feed if its content is not valid. 181 | 182 | The message **MUST** conform to the following rules: 183 | 184 | - Has the shape of a bencode list `[payload, signature]` 185 | - The payload has the shape of bencode list `[author, sequence, previous, timestamp, contentSection]` 186 | - The `previous` field matches the previous message's ID 187 | - The `signature` field must be correctly signing the `payload` 188 | - The [SSB-BFE] *type-format* for `author` is `0x0003` (there is no 189 | upgradeability in the middle of a feed) 190 | - The [SSB-BFE] *type-format* for `previous` is `0x0104` 191 | - The `author` is the same as in previous messages on the feed 192 | - The maximum size of a `[payload, signature]` in bytes must not exceed 8192 bytes 193 | 194 | Under the Bendy Butt spec, the content is free-form, but under the [meta feeds] 195 | spec there are strict constraints applied to this bencode dictionary. 196 | 197 | ## References 198 | 199 | ### Normative 200 | 201 | - [SSB-BFE] 202 | - [bencode] 203 | - [bencode integer] 204 | 205 | ### Informative 206 | 207 | - [gabby grove] 208 | - [bamboo] 209 | 210 | ### Implementation 211 | 212 | - [ssbc/ssb-bendy-butt](https://github.com/ssbc/ssb-bendy-butt) in JavaScript 213 | 214 | [SSB]: https://github.com/ssbc/ 215 | [gabby grove]: https://github.com/ssbc/ssb-spec-drafts/tree/master/drafts/draft-ssb-core-gabbygrove/00 216 | [bamboo]: https://github.com/AljoschaMeyer/bamboo 217 | [meta feeds]: https://github.com/ssb-ngi-pointer/ssb-meta-feed-spec 218 | [SSB-BFE]: https://github.com/ssb-ngi-pointer/ssb-binary-field-encodings 219 | [HMAC signing capability]: https://github.com/ssb-js/ssb-keys#signobjkeys-hmac_key-obj 220 | [bencode]: https://en.wikipedia.org/wiki/Bencode 221 | [bencode integer]: https://wiki.theory.org/index.php/BitTorrentSpecification#Integers 222 | -------------------------------------------------------------------------------- /004/examples/m0.bbmsg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ssbc/sips/e4da60f055206e16861b908ee0343eea2fc1fbe0/004/examples/m0.bbmsg -------------------------------------------------------------------------------- /004/examples/m0.bbmsg.license: -------------------------------------------------------------------------------- 1 | SPDX-FileCopyrightText: 2021 Anders Rune Jensen 2 | 3 | SPDX-License-Identifier: CC0-1.0 -------------------------------------------------------------------------------- /004/testvector-metafeed-managment-schema.json: -------------------------------------------------------------------------------- 1 | { 2 | "$schema": "https://json-schema.org/draft/2019-09/schema", 3 | "$id": "https://github.com/ssb-ngi-pointer/ssb-meta-feed-spec#vectors", 4 | "type": "object", 5 | "properties": { 6 | "Description": { 7 | "description": "Freeform description of the contents of the vector file", 8 | "type": "string" 9 | }, 10 | "Metadata": { 11 | "description": "addtional metadata needed to create the Entries", 12 | "type": "array", 13 | "items": { 14 | "type": "object", 15 | "properties": { 16 | "Name": { 17 | "type": "string", 18 | "description": "what this piece of metadata contains" 19 | }, 20 | "HexString": { 21 | "type": "string", 22 | "description": "a string of hexadecimal characters, representing some binary data", 23 | "pattern": "^[0-9a-f]+$" 24 | }, 25 | "Feed": { 26 | "description": "addtional feed references/public key, as ssb-ref", 27 | "type": "string", 28 | "pattern": "^@[0-9a-zA-Z]+" 29 | } 30 | } 31 | } 32 | }, 33 | "Entries": { 34 | "description": "the entries on a the example feed", 35 | "type": "array", 36 | "items": { 37 | "type": "object", 38 | "properties": { 39 | "Author": { 40 | "description": "the authors public key, as an ssb-ref", 41 | "type": "string", 42 | "pattern": "^@[0-9a-zA-Z]+=.bbfeed-v1$" 43 | }, 44 | "EncodedData": { 45 | "type": "string", 46 | "description": "the bencoded entry as a string of hexadecimal characters, representing binary data", 47 | "pattern": "^[0-9a-f]+$" 48 | }, 49 | "Sequence": { 50 | "type": "integer", 51 | "description": "the sequence number of that entry" 52 | }, 53 | "Previous": { 54 | "description": "the previous message hash, as a ssb-ref (its encoded as TFK when bencoded)", 55 | "type": "string", 56 | "pattern": "^%[0-9a-zA-Z+/]+=.bbmsg-v1$" 57 | }, 58 | "Timestamp": { 59 | "type": "integer" 60 | }, 61 | "HighlevelContent": { 62 | "description": "this contains a JSON version of the bencoded data, for a highlevel comparision. They all follow the same schema tho. The 1st object in the array is the content portion. The 2nd object is containing the signature bytes, created with the subfeeds keypair" 63 | } 64 | } 65 | } 66 | } 67 | } 68 | } -------------------------------------------------------------------------------- /004/testvector-metafeed-managment-schema.json.license: -------------------------------------------------------------------------------- 1 | SPDX-FileCopyrightText: 2021 Anders Rune Jensen 2 | 3 | SPDX-License-Identifier: CC0-1.0 -------------------------------------------------------------------------------- /004/testvector-metafeed-managment.json: -------------------------------------------------------------------------------- 1 | { 2 | "Description": "A bunch of metafeed related messages with double signed posts. Two feeds get added and one of them gets revoked/tombstoned", 3 | "Metadata": [ 4 | { 5 | "Name": "Seed for Metafeed KeyPair", 6 | "HexString": "7365633073656330736563307365633073656330736563307365633073656330" 7 | }, 8 | { 9 | "Name": "subfeed1 nonce", 10 | "HexString": "2323232323232323232323232323232323232323232323232323232323232323" 11 | }, 12 | { 13 | "Name": "subfeed1 author", 14 | "Feed": "@XWQputsvDZ/ZLvNAHPRsTMG/01neaZ0c3twDMOT5p+A=.ed25519" 15 | }, 16 | { 17 | "Name": "subfeed2 nonce", 18 | "HexString": "4242424242424242424242424242424242424242424242424242424242424242" 19 | }, 20 | { 21 | "Name": "subfeed2 author", 22 | "Feed": "@xQrDhN1BG0ozoCeH+So6X6MMI5pTmr0hz8jHlU/BUXA=.ggfeed-v1" 23 | } 24 | ], 25 | "Entries": [ 26 | { 27 | "EncodedData": "6c6c33343a00034ee0fb86402b717a3bc4840fe11f7301b8df44a365987ac3a135ec0e0549f70469316533343a010400000000000000000000000000000000000000000000000000000000000000006930656c6431313a66656564707572706f736531343a06006d61696e2064656661756c74383a6d6574616665656433343a00034ee0fb86402b717a3bc4840fe11f7301b8df44a365987ac3a135ec0e0549f704353a6e6f6e636533323a2323232323232323232323232323232323232323232323232323232323232323373a7375626665656433343a00005d6429badb2f0d9fd92ef3401cf46c4cc1bfd359de699d1cdedc0330e4f9a7e0373a74616e676c657364383a6d6574616665656464383a70726576696f7573323a0602343a726f6f74323a06026565343a7479706531343a06006d657461666565642f6164646536363a0400d934193e07461cad45f4447f3aacac44cd02ff22781916c4527f7e98383538f8f19cae67f44165bf1d1f42ecc293528c2b1c666ff39ce993d39670044444b80b656536363a040081bc0f370b97189d7efe0da8018aac2ae076968470179051e1ff001adfb66ffcb0ecf3e23e005c2a989e9f84e9ecdfec4c7bdfa0afec4694768cc17318bf610b65", 28 | "Key": "%K9TPrTAwY4etDpOcfd1C3EVKRScfWp+FZDQp/9rwwNA=.bbmsg-v1", 29 | "Author": "@TuD7hkArcXo7xIQP4R9zAbjfRKNlmHrDoTXsDgVJ9wQ=.bbfeed-v1", 30 | "Sequence": 1, 31 | "Previous": "%AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=.bbmsg-v1", 32 | "Timestamp": 0, 33 | "HighlevelContent": [ 34 | { 35 | "type": "metafeed/add", 36 | "feedpurpose": "main default", 37 | "subfeed": "@XWQputsvDZ/ZLvNAHPRsTMG/01neaZ0c3twDMOT5p+A=.ed25519", 38 | "metafeed": "@TuD7hkArcXo7xIQP4R9zAbjfRKNlmHrDoTXsDgVJ9wQ=.bbfeed-v1", 39 | "nonce": "IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyM=", 40 | "tangles": { 41 | "metafeed": { 42 | "root": null, 43 | "previous": null 44 | } 45 | } 46 | }, 47 | { 48 | "Name": "subfeed signature", 49 | "HexString": "0400d934193e07461cad45f4447f3aacac44cd02ff22781916c4527f7e98383538f8f19cae67f44165bf1d1f42ecc293528c2b1c666ff39ce993d39670044444b80b" 50 | } 51 | ], 52 | "Signature": "040081bc0f370b97189d7efe0da8018aac2ae076968470179051e1ff001adfb66ffcb0ecf3e23e005c2a989e9f84e9ecdfec4c7bdfa0afec4694768cc17318bf610b" 53 | }, 54 | { 55 | "EncodedData": "6c6c33343a00034ee0fb86402b717a3bc4840fe11f7301b8df44a365987ac3a135ec0e0549f70469326533343a01042bd4cfad30306387ad0e939c7ddd42dc454a45271f5a9f85643429ffdaf0c0d06930656c6431313a66656564707572706f736531343a06006578706572696d656e74616c383a6d6574616665656433343a00034ee0fb86402b717a3bc4840fe11f7301b8df44a365987ac3a135ec0e0549f704353a6e6f6e636533323a4242424242424242424242424242424242424242424242424242424242424242373a7375626665656433343a0001c50ac384dd411b4a33a02787f92a3a5fa30c239a539abd21cfc8c7954fc15170373a74616e676c657364383a6d6574616665656464383a70726576696f7573323a0602343a726f6f74323a06026565343a7479706531343a06006d657461666565642f6164646536363a040004cfe9bdcbd10453b922ceeeae32a7aac0c4b1ace2c4d3f02c0f36fc991a25859243bcce90ddb40b216266e114f1fd3c68a7b8c62f6452ba9a03334135479109656536363a04008b182bc4d5bf990912de216f2236f26fc17f3e5fdafa27fc73c956871325d417ff3edd24ec442d20d4f59794cfc8fe0677875a7b8fe4dcb58ee431f07aaa1f0865", 56 | "Key": "%lfa1JyrJZMHJDm1MC9MHo3HHu+xopw0+YgoYLLCsMSE=.bbmsg-v1", 57 | "Author": "@TuD7hkArcXo7xIQP4R9zAbjfRKNlmHrDoTXsDgVJ9wQ=.bbfeed-v1", 58 | "Sequence": 2, 59 | "Previous": "%K9TPrTAwY4etDpOcfd1C3EVKRScfWp+FZDQp/9rwwNA=.bbmsg-v1", 60 | "Timestamp": 0, 61 | "HighlevelContent": [ 62 | { 63 | "type": "metafeed/add", 64 | "feedpurpose": "experimental", 65 | "subfeed": "@xQrDhN1BG0ozoCeH+So6X6MMI5pTmr0hz8jHlU/BUXA=.ggfeed-v1", 66 | "metafeed": "@TuD7hkArcXo7xIQP4R9zAbjfRKNlmHrDoTXsDgVJ9wQ=.bbfeed-v1", 67 | "nonce": "QkJCQkJCQkJCQkJCQkJCQkJCQkJCQkJCQkJCQkJCQkI=", 68 | "tangles": { 69 | "metafeed": { 70 | "root": null, 71 | "previous": null 72 | } 73 | } 74 | }, 75 | { 76 | "Name": "subfeed2 signature", 77 | "HexString": "040004cfe9bdcbd10453b922ceeeae32a7aac0c4b1ace2c4d3f02c0f36fc991a25859243bcce90ddb40b216266e114f1fd3c68a7b8c62f6452ba9a03334135479109" 78 | } 79 | ], 80 | "Signature": "04008b182bc4d5bf990912de216f2236f26fc17f3e5fdafa27fc73c956871325d417ff3edd24ec442d20d4f59794cfc8fe0677875a7b8fe4dcb58ee431f07aaa1f08" 81 | }, 82 | { 83 | "EncodedData": "6c6c33343a00034ee0fb86402b717a3bc4840fe11f7301b8df44a365987ac3a135ec0e0549f70469336533343a010495f6b5272ac964c1c90e6d4c0bd307a371c7bbec68a70d3e620a182cb0ac31216930656c64373a7375626665656433343a00005d6429badb2f0d9fd92ef3401cf46c4cc1bfd359de699d1cdedc0330e4f9a7e0373a74616e676c657364383a6d6574616665656464383a70726576696f75736c33343a01042bd4cfad30306387ad0e939c7ddd42dc454a45271f5a9f85643429ffdaf0c0d065343a726f6f7433343a01042bd4cfad30306387ad0e939c7ddd42dc454a45271f5a9f85643429ffdaf0c0d06565343a7479706532303a06006d657461666565642f746f6d6273746f6e656536363a04006d30bd087a9239b4d4c728358222c8369f70c5e12c70d17d8e1dbac2e6fe2541b94edf234370921e73537e9e53fac070b3a5c39bd1969b870ec06bd9e79adb01656536363a04000c74a447cc421054e4fb413db7c6f1ccb0abb9b178324a4d3171d32593a1142c02be28e445df8df25e2b6a56cfbc74e67f687c57497a5a9cbabbe1baa766c70b65", 84 | "Key": "%d9hWhAE4KOPK2aWGtVOL7oi3OqErPcs59IpOU2Q4CGE=.bbmsg-v1", 85 | "Author": "@TuD7hkArcXo7xIQP4R9zAbjfRKNlmHrDoTXsDgVJ9wQ=.bbfeed-v1", 86 | "Sequence": 3, 87 | "Previous": "%lfa1JyrJZMHJDm1MC9MHo3HHu+xopw0+YgoYLLCsMSE=.bbmsg-v1", 88 | "Timestamp": 0, 89 | "HighlevelContent": [ 90 | { 91 | "type": "metafeed/tombstone", 92 | "subfeed": "@XWQputsvDZ/ZLvNAHPRsTMG/01neaZ0c3twDMOT5p+A=.ed25519", 93 | "tangles": { 94 | "metafeed": { 95 | "root": "%K9TPrTAwY4etDpOcfd1C3EVKRScfWp+FZDQp/9rwwNA=.bbmsg-v1", 96 | "previous": [ 97 | "%K9TPrTAwY4etDpOcfd1C3EVKRScfWp+FZDQp/9rwwNA=.bbmsg-v1" 98 | ] 99 | } 100 | } 101 | }, 102 | { 103 | "Name": "subfeed1 signature", 104 | "HexString": "04006d30bd087a9239b4d4c728358222c8369f70c5e12c70d17d8e1dbac2e6fe2541b94edf234370921e73537e9e53fac070b3a5c39bd1969b870ec06bd9e79adb01" 105 | } 106 | ], 107 | "Signature": "04000c74a447cc421054e4fb413db7c6f1ccb0abb9b178324a4d3171d32593a1142c02be28e445df8df25e2b6a56cfbc74e67f687c57497a5a9cbabbe1baa766c70b" 108 | } 109 | ] 110 | } 111 | -------------------------------------------------------------------------------- /004/testvector-metafeed-managment.json.license: -------------------------------------------------------------------------------- 1 | SPDX-FileCopyrightText: 2021 Anders Rune Jensen 2 | 3 | SPDX-License-Identifier: CC0-1.0 -------------------------------------------------------------------------------- /005.md: -------------------------------------------------------------------------------- 1 | # HTTP Invites 2 | 3 | Author: Andre 'Staltz' Medeiros 4 | 5 | Date: 2021-04-26 6 | 7 | License: CC0-1.0 8 | 9 | ## Abstract 10 | 11 | As part of the process of onboarding to SSB, new users often need to connect to a "pub server" or "room server" where content can be retrieved from. These servers often employ an access control system based on invite tokens, to prevent access to undesired actors from the public internet. The invite system deployed by these servers has been a convoluted algorithm repurposing secret-handshake to create an ephemeral muxrpc connection only for the initial remote procedure call to claim the invite token. In this document, we describe a simpler HTTP-based invite-token system for pubs and rooms that applies before any secret-handshake connection is built. 12 | 13 | ## Terminology 14 | 15 | The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC 2119](https://tools.ietf.org/html/rfc2119). 16 | 17 | ## Conditions 18 | 19 | This specification makes clear assumptions about the setup involved peers authenticating. 20 | 21 | **Server:** an SSB peer, known as the "server", **MUST** have an internet-public host address, **MUST** be accessible for secret-handshake connections under a multiserver address, and **MUST** support HTTPS requests as well as it **MUST NOT** support plain HTTP. 22 | 23 | **Client:** another SSB peer, known as the "client", **SHOULD** be able to open a secret-handshake and muxrpc connection with the server. The user controlling this SSB peer also **MUST** control a web browser used to make requests to the server. The client's browser and operating system **SHOULD** support hyperlinks to [SSB URIs](https://github.com/ssb-ngi-pointer/ssb-uri-spec), redirecting them to SSB applications that recognize and parse SSB URIs. The client's SSB application employed during SSB HTTP Authentication **MUST** be able to recognize and parse SSB URIs. 24 | 25 | ## Specification 26 | 27 | 1. Suppose an SSB user (known as "the client") has the SSB ID `userId` and has an SSB app supporting parsing SSB URIs 28 | 1. Suppose the server is hosted at domain `serverHost` and has generated an invite `inviteCode` 29 | 1. The invite link corresponding to `inviteCode` **SHOULD** be a URL in the format `https://${serverHost}/join?invite=${inviteCode}` 30 | 1. When the client visits that URL in a browser, the server **MUST** respond with HTML such that: 31 | 1. If the `inviteCode` is already claimed or otherwise no longer valid, an error page **SHOULD** be rendered as response, and no further steps in this specification apply 32 | 1. Otherwise, the `inviteCode` is *unclaimed*, and the following SSB URI **MUST** be rendered on the response page: `ssb:experimental?action=claim-http-invite&invite=${inviteCode}&postTo=${submissionUrl}` where `${submissionUrl}` is another URL on the server 33 | 1. The client's SSB app **SHOULD** parse the SSB URI and subsequently **SHOULD** send an HTTPS POST request to the endpoint `submissionUrl` with the header `Content-Type` equal `application/json` and the following body: `{"id":"${userId}","invite":"${inviteCode}"}` 34 | 1. The server receives the POST request and: 35 | 1. If the `inviteCode` is already claimed, the response **SHOULD** be an error, and no further steps in this specification apply 36 | 1. Otherwise, the `inviteCode` is now considered *claimed* for `userId`, which means: 37 | 1. The server **SHOULD** store the client's `userId` and allow the client to access resources on the server, effectively making the client a recognized member 38 | 1. The server **MUST** respond with header `Content-Type` equal `application/json` and body `{"multiserverAddress":"${serverMsAddr}"}` where `${serverMsAddr}` consititutes the server's multiserver address 39 | 1. The client receives the `submissionUrl` response, parses `${serverMsAddr}` from the response body, and **MAY** use that multiserver address to create a muxrpc connection with the server 40 | 1. If the server receives a muxrpc connection from the client, it **MUST** authorize it and grant them internal access 41 | 1. The client is now authenticated 42 | 43 | The JSON schemas for which the response from the `submissionUrl` **MUST** conform to is shown below. 44 | 45 | **Successful responses** 46 | 47 | ```json 48 | { 49 | "$schema": "http://json-schema.org/draft-07/schema#", 50 | "$id": "https://github.com/ssb-ngi-pointer/ssb-http-invite#claimed-json-endpoint-success", 51 | "type": "object", 52 | "properties": { 53 | "status": { 54 | "title": "Response status tag", 55 | "description": "Indicates the completion status of this response", 56 | "type": "string", 57 | "pattern": "^(successful)$" 58 | }, 59 | "multiserverAddress": { 60 | "title": "Multiserver address of the server", 61 | "description": "Should conform to https://github.com/ssbc/multiserver-address", 62 | "type": "string" 63 | } 64 | }, 65 | "required": [ 66 | "status", 67 | "multiserverAddress" 68 | ] 69 | } 70 | ``` 71 | 72 | **Failed responses** 73 | 74 | ```json 75 | { 76 | "$schema": "http://json-schema.org/draft-07/schema#", 77 | "$id": "https://github.com/ssb-ngi-pointer/ssb-http-invite#claimed-json-endpoint-error", 78 | "type": "object", 79 | "properties": { 80 | "status": { 81 | "title": "Response status tag", 82 | "description": "Indicates the completion status of this response", 83 | "type": "string" 84 | }, 85 | "error": { 86 | "title": "Response error", 87 | "description": "Describes the specific error that occurred", 88 | "type": "string" 89 | } 90 | }, 91 | "required": [ 92 | "status", 93 | "error" 94 | ] 95 | } 96 | ``` 97 | 98 | ### Programmatic invite façade 99 | 100 | As an additional endpoint for programmatic purposes, if the query parameter `encoding=json` is added to the invite link (for illustration: `https://${serverHost}/join?invite=${inviteCode}&encoding=json`), then the server **SHOULD** return a JSON response. The JSON body **MUST** conform to the following schemas: 101 | 102 | **Successful responses** 103 | 104 | ```json 105 | { 106 | "$schema": "http://json-schema.org/draft-07/schema#", 107 | "$id": "https://github.com/ssb-ngi-pointer/ssb-http-invite#invite-json-endpoint-success", 108 | "type": "object", 109 | "properties": { 110 | "status": { 111 | "title": "Response status tag", 112 | "description": "Indicates the completion status of this response", 113 | "type": "string", 114 | "pattern": "^(successful)$" 115 | }, 116 | "invite": { 117 | "title": "Invite code", 118 | "description": "Sequence of bytes that acts as a token to accept the invite", 119 | "type": "string" 120 | }, 121 | "postTo": { 122 | "title": "Submission URL", 123 | "description": "URL where clients should submit POST requests with a JSON body", 124 | "type": "string" 125 | } 126 | }, 127 | "required": [ 128 | "status", 129 | "invite", 130 | "postTo" 131 | ] 132 | } 133 | ``` 134 | 135 | **Failed responses** 136 | 137 | ```json 138 | { 139 | "$schema": "http://json-schema.org/draft-07/schema#", 140 | "$id": "https://github.com/ssb-ngi-pointer/ssb-http-invite#invite-json-endpoint-error", 141 | "type": "object", 142 | "properties": { 143 | "status": { 144 | "title": "Response status tag", 145 | "description": "Indicates the completion status of this response", 146 | "type": "string" 147 | }, 148 | "error": { 149 | "title": "Response error", 150 | "description": "Describes the specific error that occurred", 151 | "type": "string" 152 | } 153 | }, 154 | "required": [ 155 | "status", 156 | "error" 157 | ] 158 | } 159 | ``` 160 | 161 | ### Example 162 | 163 | Suppose the client has the SSB ID `@FlieaFef19uJ6jhHwv2CSkFrDLYKJd/SuIS71A5Y2as=.ed25519` and the server is hosted at `scuttlebutt.eu`. Then the invite user journey is: 164 | 165 | 1. Invite code `39c0ac1850ec9af14f1bb73` was generated by the server 166 | 1. The corresponding invite link is `https://scuttlebutt.eu/join?invite=39c0ac1850ec9af14f1bb73` 167 | 1. When the client opens that link in a browser, it renders a link to the SSB URI [ssb:experimental?action=claim-http-invite&invite=39c0ac1850ec9af14f1bb73&postTo=https%3A%2F%2Fscuttlebutt.eu%2Fclaiminvite](ssb:experimental?action=claim-http-invite&invite=39c0ac1850ec9af14f1bb73&postTo=https%3A%2F%2Fscuttlebutt.eu%2Fclaiminvite) 168 | 1. The client's SSB app processes the SSB URI and makes a POST request to `https://scuttlebutt.eu/claiminvite` with body 169 | ``` 170 | { 171 | "id": "@FlieaFef19uJ6jhHwv2CSkFrDLYKJd/SuIS71A5Y2as=.ed25519", 172 | "invite": "39c0ac1850ec9af14f1bb73" 173 | } 174 | ``` 175 | 1. The server accepts the POST request, and responds with the JSON body 176 | ``` 177 | { 178 | "status": "successful", 179 | "multiserverAddress": "net:scuttlebutt.eu:8008~shs:zz+n7zuFc4wofIgKeEpXgB+/XQZB43Xj2rrWyD0QM2M=" 180 | } 181 | ``` 182 | 1. The server now recognizes the client as an authorized member for any subsequent secret-handshake and muxrpc connections at the multiserver address `net:scuttlebutt.eu:8008~shs:zz+n7zuFc4wofIgKeEpXgB+/XQZB43Xj2rrWyD0QM2M=` 183 | 184 | The JSON endpoint `https://scuttlebutt.eu/join?invite=39c0ac1850ec9af14f1bb73&encoding=json` is an alternative to the SSB URI, and would respond with the following JSON: 185 | 186 | ```json 187 | { 188 | "status": "successful", 189 | "invite": "39c0ac1850ec9af14f1bb73", 190 | "postTo": "https://scuttlebutt.eu/claiminvite" 191 | } 192 | ``` 193 | 194 | After that, the same steps 4, 5, and 6 apply. 195 | 196 | ## Implementation notes 197 | 198 | The rendering of the invite façade HTML is unspecified on purpose. Implementors can choose to present the SSB URI either as a link, or as a code to be copied and pasted, or as an automatic redirect. 199 | 200 | Furthermore, the invite page is a good place to render instructions on how to install an SSB app, in case the invitee is uninitiated in SSB and this is their entry point. 201 | 202 | Specifically, these instructions can also use mobile operating systems deep linking capabilities. For instance, suppose the page recommends installing Manyverse: the page could link to `join.manyver.se` (with additional query parameters to pass on the invite code), which in turn uses Android Deep Linking redirect (see [this technical possibility](https://stackoverflow.com/questions/28744167/android-deep-linking-use-the-same-link-for-the-app-and-the-play-store)) to open Manyverse (if it's installed) or open Google Play Store (to install the app). Same idea should apply for mobile apps, say "Imaginary App" using the fixed URL "join.imaginary.app". Desktop apps are different as they can be installed without an app store. This paragraph was informed by Wouter Moraal's [UX Research for Manyverse](https://www.manyver.se/ux-research/). 203 | 204 | ## Security considerations 205 | 206 | ### Malicious web visitor 207 | 208 | A web visitor, either human or bot, could attempt brute force visiting all possible invite URLs, in order to authenticate themselves. However, this could easily be mitigated by rate limiting requests by the same IP address. 209 | 210 | ## References 211 | 212 | ### Normative 213 | 214 | - [SIP 1](./001.md) "SSB URIs" 215 | 216 | ### Informative 217 | 218 | - [UX Research for Manyverse](https://www.manyver.se/ux-research/) 219 | 220 | ### Implementation 221 | 222 | - [ssbc/go-ssb-room](https://github.com/ssbc/go-ssb-room/) server in Go 223 | - [ssbc/ssb-http-invite-client](https://github.com/ssbc/ssb-http-invite-client) 224 | client library in JavaScript 225 | 226 | ## Appendix A. List of new SSB URIs 227 | 228 | - `ssb:experimental?action=claim-http-invite&invite=${inviteCode}&postTo=${submissionUrl}` 229 | -------------------------------------------------------------------------------- /006.md: -------------------------------------------------------------------------------- 1 | # HTTP Authentication 2 | 3 | Author: Andre 'Staltz' Medeiros 4 | 5 | Date: 2021-04-26 6 | 7 | License: CC0-1.0 8 | 9 | ## Abstract 10 | 11 | SSB is a peer-to-peer system where there should not be a client-server distinction, but there are exceptions to this nature. "Pub servers" and "room servers" are nodes with privileged internet-wide presence that can provide replication and tunneled-connection services to several peers. Since these servers often also host a public HTTP interface, concern has been raised on how authenticated clients are to interact with these services over HTTP when requesting access-restricted resources. SSB HTTP Authentication is a protocol that utilizes both SSB muxrpc and HTTPS to grant an HTTP client (such as a browser) access to resources authorized for the SSB client peer. 12 | 13 | ## Terminology 14 | 15 | The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC 2119](https://tools.ietf.org/html/rfc2119). 16 | 17 | ## Conditions 18 | 19 | This specification makes clear assumptions about the setup involved peers authenticating. 20 | 21 | **Server:** an SSB peer, known as the "server", **MUST** have an internet-public host address, **MUST** be accessible for secret-handshake connections under a multiserver address, and **MUST** support HTTPS requests as well as it **MUST NOT** support plain HTTP. 22 | 23 | **Client:** another SSB peer, known as the "client", **MUST** be able to open a secret-handshake and muxrpc connection with the server. The user controlling this SSB peer also **SHOULD** control a web browser used to make requests to the server. The client's browser and operating system **SHOULD** support hyperlinks to [SSB URIs](https://github.com/ssb-ngi-pointer/ssb-uri-spec), redirecting them to SSB applications that recognize and parse SSB URIs. The client's SSB application employed during SSB HTTP Authentication **MUST** be able to recognize and parse SSB URIs. 24 | 25 | **Connections:** the server and the client **SHOULD** recognize each other's SSB IDs as mutually trusted (such as a mutual follow relationship, or "membership" relationship where the client has claimed an invite token from the server). For the authentication protocol described in this document to succeed, the two SSB peers **MUST** be connected to each other via muxrpc and secret-handshake for the entire duration of the protocol. The purpose of SSB HTTP Authentication is to extend the trust that exists between these peers from the muxrpc context to the context of HTTP in the browser. 26 | 27 | ## Specification 28 | 29 | A client known by its SSB ID `cid` is connected via secret-handshake and muxrpc to a server known by its SSB ID `sid`. The server is hosted at `serverHost`. A browser controlled by the same person or agent as the `cid` peer wishes to request an access-restricted resource (such as an admin dashboard) at the server over HTTP. All HTTP requests **MUST** be done with HTTPS. 30 | 31 | The three sides (browser client, SSB client `cid`, and SSB server `sid`) perform a [challenge-response authentication](https://en.wikipedia.org/wiki/Challenge%E2%80%93response_authentication) protocol, specified as UML sequence diagrams. We use the shorthands `sc`, `cc`, and `sol` to mean: 32 | 33 | - `sc`: "server's challenge" 34 | - `cc`: "client's challenge" 35 | - `sol`: "solution" 36 | 37 | The challenges, `cc` and `sc`, are 256-bit [cryptographic nonces](https://en.wikipedia.org/wiki/Cryptographic_nonce) encoded in base64. The solution `sol` is a cryptographic signature using the cryptographic keypair `cid` that identifies the client, described below: 38 | 39 | - `sid` is the servers's identity from their cryptographic keypair 40 | - `cid` is the client's identity from their cryptographic keypair 41 | - `sc` is a 256-bit nonce created by the server, encoded in base64 42 | - `cc` is a 256-bit nonce created by the client, encoded in base64 43 | - `sol` is the client's cryptographic signature of the string `=http-auth-sign-in:${sid}:${cid}:${sc}:${cc}` where `${x}` means string interpolation of the value `x` 44 | 45 | Both sides generate the nonces, but there are use cases where one side should start first. In other words, the challenge-response protocol described here can be either **client-initiated** or **server-initiated**. 46 | 47 | The HTTPS endpoint `/login` with the query parameter `ssb-http-auth` **MUST** be employed as specified here, but URLs for logging out and for Server-Sent Events (SSE) are left unspecified and implementations are free to choose their routes. The muxrpc APIs `httpAuth.requestSolution(sc, cc)`, `httpAuth.sendSolution(sc, cc, cr)`, `httpAuth.invalidateAllSolutions()` and the SSB URI `ssb:experimental?action=start-http-auth&sid=${sid}&sc=${sc}` **MUST** be employed as specified here. 48 | 49 | ### Client-initiated protocol 50 | 51 | In the client-initiated variant of the challenge-response protocol, the first step is the client creating `cc` and opening a web page in the browser. Then, the server attending to that HTTP request will call `httpAuth.requestSolution(sc, cc)` on the client SSB peer. 52 | 53 | The UML sequence diagram for the whole client-initial protocol is shown below: 54 | 55 | ```mermaid 56 | sequenceDiagram 57 | participant Umux as SSB client `cid` 58 | participant Uweb as Browser client 59 | participant Serv as SSB server `sid` 60 | 61 | note over Umux: Generates
challenge `cc` 62 | Umux->>Uweb: `https://${serverHost}/login
?ssb-http-auth=1&cid=${cid}&cc=${cc}` 63 | Uweb->>+Serv: `https://${serverHost}/login
?ssb-http-auth=1&cid=${cid}&cc=${cc}` 64 | Note over Serv: Generates
challenge `sc` 65 | alt client is disconnected from the server 66 | Serv-->>Uweb: HTTP 403 67 | else client is connected to the server 68 | Serv->>+Umux: (muxrpc async) `httpAuth.requestSolution(sc, cc)` 69 | Note over Umux: Generates
signature `sol` 70 | Umux-->>-Serv: respond httpAuth.requestSolution with `sol` 71 | alt `sol` is incorrect 72 | Serv-->>Uweb: HTTP 403 73 | else `sol` is correct 74 | Serv-->>-Uweb: HTTP 200, auth token 75 | Note over Uweb: Stores auth token as a cookie 76 | end 77 | end 78 | ``` 79 | 80 | 81 | ### Server-initiated protocol 82 | 83 | In the server-initiated variant of the challenge-response protocol, the first step is the browser requesting a login from the server without any input data. The server answers the browser, which in turn displays an SSB URI which the SSB peer knows how to open. 84 | 85 | The primary difference between this variant and the previous one is that the muxrpc async RPC call direction is reversed. Previously, the server called `httpAuth.requestSolution` **on the client**. In this variant, the client calls `httpAuth.sendSolution` **on the server**. The response is also different. In the previous case, the client's response is expected to be the `sol`. In this variant, the `sol` argument is provided by the client and the server's response is expected to be `true`. 86 | 87 | The secondary difference with this variant is the addition of [Server-Sent Events](https://html.spec.whatwg.org/multipage/server-sent-events.html) (SSE) between the browser and the server, to update the browser when the muxrpc protocol succeeds. 88 | 89 | The UML sequence diagram for the whole server-initial protocol is shown below: 90 | 91 | ```mermaid 92 | sequenceDiagram 93 | participant Umux as SSB client `cid` 94 | participant Uweb as Browser client 95 | participant Serv as SSB server `sid` 96 | 97 | Uweb->>Serv: Some URL on the `serverHost` 98 | activate Serv 99 | Note over Serv: Generates
challenge `sc` 100 | Serv-->>Uweb: Displays `ssb:experimental?
action=start-http-auth&sid=${sid}&sc=${sc}` 101 | Uweb->>Serv: Subscribe to some SSE URL with `sc` as input 102 | Uweb->>Umux: Consumes SSB URI 103 | Note over Umux: Generates
challenge `cc` 104 | Note over Umux: Generates
signature `sol` 105 | Umux->>+Serv: (muxrpc async) `httpAuth.sendSolution(sc, cc, sol)` 106 | alt `sol` is incorrect, or other errors 107 | Serv-->>Umux: `false` 108 | Serv-->>Uweb: (SSE) "redirect to ${url}" 109 | Uweb->>+Serv: GET `${url}` 110 | Serv-->>-Uweb: HTTP 403 111 | else `sol` is correct 112 | Serv-->>-Umux: `true` 113 | Serv-->>Uweb: (SSE) "redirect to ${url}" 114 | Uweb->>+Serv: GET `${url}` 115 | Serv-->>-Uweb: HTTP 200, auth token 116 | deactivate Serv 117 | Note over Uweb: Stores auth token as a cookie 118 | end 119 | ``` 120 | 121 | The SSB URI **MAY** also contain the query parameter `multiserverAddress` with value `msaddr` matching the server's multiserver address, in case the client does not know how to map the server's `sid` to a multiserver address in order to call the muxrpc `http.sendSolution`: 122 | 123 | ``` 124 | ssb:experimental?action=start-http-auth&sid=${sid}&sc=${sc}&multiserverAddress=${msaddr} 125 | ``` 126 | 127 | ### Sign-out 128 | 129 | As a RECOMMENDED muxrpc API, `httpAuth.invalidateAllSolutions` on the server allows the SSB peer to invalidate *all* auth tokens associated with the `cid`. See UML sequence diagram: 130 | 131 | ```mermaid 132 | sequenceDiagram 133 | participant Umux as SSB client `cid` 134 | participant Uweb as Browser client 135 | participant Serv as SSB server `sid` 136 | 137 | Umux->>+Serv: (muxrpc async) `httpAuth.invalidateAllSolutions()` 138 | Note over Serv: Invalidates every `sc`
and `authtoken` associated
with `cid` 139 | Serv-->>-Umux: respond httpAuth.invalidateAllSolutions with `true` 140 | Note over Uweb,Serv: Potentially thereafter... 141 | Uweb->>+Serv: Authenticate using `authtoken` 142 | Serv-->>-Uweb: HTTP 401 143 | ``` 144 | 145 | The browser client also has the option of signing out with HTTP endpoints. This does not require a muxrpc call with the SSB peer. See UML sequence diagram: 146 | 147 | ```mermaid 148 | sequenceDiagram 149 | participant Uweb as Browser client 150 | participant Serv as SSB server `sid` 151 | 152 | Uweb->>+Serv: Some URL on the `serverHost` 153 | Note over Serv: Invalidates `authtoken` 154 | Serv-->>-Uweb: HTTP 200 155 | Note over Uweb,Serv: Potentially thereafter... 156 | Uweb->>+Serv: Authenticate using `authtoken` 157 | Serv-->>-Uweb: HTTP 401 158 | ``` 159 | 160 | ## Security considerations 161 | 162 | ### Client consent 163 | 164 | Because the sign-in process is seemless and happens in the background via muxrpc, without any input from the user who controls the client SSB peer, there is potential for vulnerabilities if the SSB URI is manipulated or the URL is manipulated. 165 | 166 | SSB applications that control the client SSB peer therefore MUST prompt user consent before `httpAuth.requestSolution` or `httpAuth.sendSolution` are sent. The prompt SHOULD display the server's SSB ID and ask the user to confirm their intent to sign-in. 167 | 168 | ## References 169 | 170 | ### Normative 171 | 172 | - [SIP 1](./001.md) "SSB URIs" 173 | - [Server-Sent Events](https://html.spec.whatwg.org/multipage/server-sent-events.html) 174 | 175 | ### Informative 176 | 177 | - [Wikipedia: challenge-response authentication](https://en.wikipedia.org/wiki/Challenge%E2%80%93response_authentication) 178 | - [Wikipedia: cryptographic nonces](https://en.wikipedia.org/wiki/Cryptographic_nonce) 179 | 180 | ### Implementation 181 | 182 | - [ssbc/go-ssb-room](https://github.com/ssbc/go-ssb-room/) server in Go 183 | - [ssbc/ssb-http-auth-client](https://github.com/ssbc/ssb-http-auth-client) 184 | client library in JavaScript 185 | 186 | 187 | ## Appendix A. List of required HTTP endpoints 188 | 189 | - `/login` 190 | 191 | ## Appendix B. List of new muxrpc APIs 192 | 193 | - async 194 | - `httpAuth.requestSolution(sc, cc)` 195 | - `httpAuth.sendSolution(sc, cc, cr)` 196 | - `httpAuth.invalidateAllSolutions()` 197 | 198 | ## Appendix C. List of new SSB URIs 199 | 200 | - `ssb:experimental?action=start-http-auth&sid=${sid}&sc=${sc}` 201 | -------------------------------------------------------------------------------- /008.md: -------------------------------------------------------------------------------- 1 | # Binary Field Encodings 2 | 3 | Authors: Anders Rune Jensen, Mix Irving 4 | 5 | Date: 2022-10-02 6 | 7 | License: CC0-1.0 8 | 9 | ## Abstract 10 | 11 | Addressable objects in SSB such as messages and feeds were originally referred 12 | to by their hashes in base64 encoding, with special prefixes to characterize the 13 | kind of the object. In some cases, there is a need to use those hashes in a 14 | binary context, for example in encryption, and tag them with their respective 15 | kind. This specification defines binary encodings for common objects such as 16 | message IDs and feed IDs to be used in binary feed formats and encryption 17 | formats. 18 | 19 | ## Specification 20 | 21 | The binary encoding is defined as the concatenation of three parts, often known 22 | as **T-F-D**: 23 | 24 | - The `type` of thing as a UInt8 byte 25 | - The `format` of the `type` as a UInt8 byte 26 | - The `data` as a sequence of UInt8 bytes 27 | 28 | ### Types 29 | 30 | | Type code | Referencing | In `bfe.json` | 31 | |:-----------:| ------------------ | ---------------- | 32 | | 0 | Feed ID | `feed` | 33 | | 1 | Message ID | `message` | 34 | | 2 | Blob ID | `blob` | 35 | | 3 | Encryption key | `encryption-key` | 36 | | 4 | Signature | `signature` | 37 | | 5 | Encrypted data | `encrypted` | 38 | | 6 | Generic data | `generic` | 39 | | 7 | Identity | `identity` | 40 | 41 | #### 0. Feed ID formats 42 | 43 | A feed ID TFD represents the public portion of a cryptographic keypair used to 44 | identify a feed, and verify message signatures. 45 | 46 | | Type code | Format code | Data length | Specification | In `bfe.json` | 47 | |:---------:|:-----------:|-------------|--------------------|-----------------| 48 | | 0 | 0 | 32 bytes | [Classic SSB Feed] | `classic` | 49 | | 0 | 1 | 32 bytes | [Gabby Grove] | `gabbygrove-v1` | 50 | | 0 | 2 | 32 bytes | [Bamboo] | `bamboo` | 51 | | 0 | 3 | 32 bytes | [Bendy Butt] | `bendybutt-v1` | 52 | | 0 | 4 | 32 bytes | [Buttwoo] | `buttwoo-v1` | 53 | | 0 | 5 | 32 bytes | [Index feed] | `indexed-v1` | 54 | 55 | ##### Example 56 | 57 | Given a sigil-based string encoding of a classic SSB feed: 58 | 59 | ``` 60 | @6CAxOI3f+LUOVrbAl0IemqiS7ATpQvr9Mdw9LC4+Uv0=.ed25519 61 | │└─────────────────────┬────────────────────┘└───┬──┘ 62 | sigil base64 encoded data suffix 63 | ``` 64 | 65 | Its BFE encoding is the following bytes displayed in hexadecimal: 66 | 67 | ``` 68 | 00 00 e8 20 31 38 8d df f8 b5 0e 56 b6 c0 97 42 1e 9a a8 92 ec 04 e9 42 fa fd 31 dc 3d 2c 2e 3e 52 fd 69 | │ │ └────────────────────┬────────────────────────────────────────────────────────────────────────┘ 70 | type │ data 71 | format 72 | ``` 73 | 74 | #### 1. Message ID formats 75 | 76 | A message ID TFD represents the hash that uniquely identifies a message 77 | published on a feed. Some message ID formats directly reference the hash 78 | algorithm utilized, while others leave it implicit in the specification. 79 | 80 | | Type code | Format code | Data length | Specification | In `bfe.json` | 81 | |:---------:|:-----------:|-------------|-------------------|-----------------| 82 | | 1 | 0 | 32 bytes | [Classic SSB Msg] | `classic` | 83 | | 1 | 1 | 32 bytes | [Gabby Grove] | `gabbygrove-v1` | 84 | | 1 | 2 | 32 bytes | [Private Group] | `cloaked` | 85 | | 1 | 3 | 64 bytes | [Bamboo] | `bamboo` | 86 | | 1 | 4 | 32 bytes | [Bendy Butt] | `bendybutt-v1` | 87 | | 1 | 5 | 32 bytes | [Buttwoo] | `buttwoo-v1` | 88 | | 1 | 6 | 32 bytes | [Index feed] | `indexed-v1` | 89 | 90 | ##### Example 91 | 92 | Given a sigil-based string encoding of a classic SSB message ID: 93 | 94 | ``` 95 | %R8heq/tQoxEIPkWf0Kxn1nCm/CsxG2CDpUYnAvdbXY8=.sha256 96 | │└─────────────────────┬────────────────────┘└──┬──┘ 97 | sigil base64 encoded data suffix 98 | ``` 99 | 100 | Its BFE encoding is the following bytes displayed in hexadecimal: 101 | 102 | ``` 103 | 01 00 47 c8 5e ab fb 50 a3 11 08 3e 45 9f d0 ac 67 d6 70 a6 fc 2b 31 1b 60 83 a5 46 27 02 f7 5b 5d 8f 104 | │ │ └────────────────────┬────────────────────────────────────────────────────────────────────────┘ 105 | type │ data 106 | format 107 | ``` 108 | 109 | #### 2. Blob ID formats 110 | 111 | A blob ID TFD represents the hash that uniquely identifies the blob. 112 | 113 | | Type code | Format code | Data length | Specification | In `bfe.json` | 114 | |:---------:|:-----------:|-------------|--------------------|---------------| 115 | | 2 | 0 | 32 bytes | [Classic SSB Blob] | `classic` | 116 | 117 | ##### Example 118 | 119 | Given a sigil-based string encoding of a classic SSB blob ID: 120 | 121 | ``` 122 | &S7+CwHM6dZ9si5Vn4ftpk/l/ldbRMqzzJos+spZbWf4=.sha256 123 | │└─────────────────────┬────────────────────┘└───┬─┘ 124 | sigil base64 encoded data suffix 125 | ``` 126 | 127 | Its BFE encoding is the following bytes displayed in hexadecimal: 128 | 129 | ``` 130 | 02 00 4b bf 82 c0 73 3a 75 9f 6c 8b 95 67 e1 fb 69 93 f9 7f 95 d6 d1 32 ac f3 26 8b 3e b2 96 5b 59 fe 131 | │ │ └────────────────────┬────────────────────────────────────────────────────────────────────────┘ 132 | type │ data 133 | format 134 | ``` 135 | 136 | #### 3. Encryption Key formats 137 | 138 | Keys used for encryption 139 | 140 | | Type code | Format code | Data length | Specification | In `bfe.json` | 141 | |:---------:|:-----------:|-------------|------------------------|-----------------| 142 | | 3 | 0 | 32 bytes | [Private Group DM] | `box2-dm-dh` | 143 | | 3 | 1 | 32 bytes | [Private Group PO box] | `box2-pobox-dh` | 144 | 145 | 146 | #### 4. Signature formats 147 | 148 | | Type code | Format code | Data length | Specification | In `bfe.json` | 149 | |:---------:|:-----------:|-------------|-------------------------|---------------| 150 | | 4 | 0 | 64 bytes | [Classic SSB Signature] | `msg-ed25519` | 151 | 152 | 153 | ##### Example 154 | 155 | Given a base64 string encoding of a Classic SSB ed25519 signature: 156 | 157 | ``` 158 | nkY4Wsn9feosxvX7bpLK7OxjdSrw6gSL8sun1n2TMLXKySYK9L5itVQnV2nQUctFsrUOa2istD2vDk1B0uAMBQ==.sig.ed25519 159 | └─────────────────────────────────────┬────────────────────────────────────────────────┘└────┬─────┘ 160 | base64 encoded signature suffix 161 | ``` 162 | 163 | Its BFE encoding is the following bytes displayed in hexadecimal: 164 | 165 | ``` 166 | 04 00 9e 46 38 5a c9 fd 7d ea 2c c6 f5 fb 6e 92 ca ec ec 63 75 2a f0 ea 04 8b f2 cb a7 d6 7d 93 30 b5 ca c9 26 0a f4 be 62 b5 54 27 57 69 d0 51 cb 45 b2 b5 0e 6b 68 ac b4 3d af 0e 4d 41 d2 e0 0c 05 167 | │ │ └────────────────────┬────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘ 168 | type │ data 169 | format 170 | ``` 171 | 172 | #### 5. Encrypted data formats 173 | 174 | When content is encrypted (in other words, "boxed") in SSB, it is provided as 175 | uninterpretable bytes, plus a tag that identifies which algorithm was used for 176 | encrypting it, such as `box` or `box2`. 177 | 178 | | Type code | Format code | Data length | Specification | In `bfe.json` | 179 | |:---------:|:-----------:|-------------|-----------------|---------------| 180 | | 5 | 0 | Arbitrary | [Private Box] | `box1` | 181 | | 5 | 1 | Arbitrary | [Private Group] | `box2` | 182 | 183 | #### 6. Generic data formats 184 | 185 | BFE supports encoding data types with no semantics attached to them. They are 186 | merely categorized into formats that represent their data type. 187 | 188 | | Type code | Format code | Data length | Specification | In `bfe.json` | 189 | |:---------:|:-----------:|-------------|-----------------------|---------------| 190 | | 6 | 0 | Arbitrary | [UTF8] String | `string-UTF8` | 191 | | 6 | 1 | 1 byte | Boolean: data byte is 0 for False, 1 for True | `boolean` | 192 | | 6 | 2 | 0 bytes | [Null pointer] | `nil` | 193 | | 6 | 3 | Arbitrary | Any sequence of bytes | `any-bytes` | 194 | 195 | 196 | #### 7. Identity formats 197 | 198 | Identities are distinct from feedIds in that they are not a key bound to a single feed/ device, 199 | and they are never used for signing of messages. 200 | 201 | | Type code | Format code | Data length | Specification | In `bfe.json` | 202 | |:---------:|:-----------:|-------------|------------------------|---------------| 203 | | 7 | 0 | 32 | [Private Group PO box] | `po-box` | 204 | | 7 | 1 | 32 | [Private Group] | `group` | 205 | 206 | ## References 207 | 208 | ### Normative 209 | 210 | - [Classic SSB Feed] 211 | - [Classic SSB Msg] 212 | - [Classic SSB Blob] 213 | - [Classic SSB Signature] 214 | - [Gabby Grove] 215 | - [Bamboo] 216 | - [Private Group] 217 | - [Private Group DM] 218 | - [Private Group PO box] 219 | - [Bendy Butt] 220 | - [Buttwoo] 221 | - [Private Box] 222 | - [Null pointer] 223 | - [UTF8] 224 | - [Index feed] 225 | 226 | ### Informative 227 | 228 | - [TFK] 229 | - [Fusion Identity] 230 | - [Envelope Spec] 231 | 232 | ### Implementation 233 | 234 | - [ssbc/ssb-bfe](https://github.com/ssbc/ssb-bfe) in JavaScript 235 | 236 | [TFK]: https://github.com/ssbc/envelope-spec/blob/master/encoding/tfk.md 237 | [Classic SSB Feed]: https://ssbc.github.io/scuttlebutt-protocol-guide/#keys-and-identities 238 | [Classic SSB Msg]: https://ssbc.github.io/scuttlebutt-protocol-guide/#message-format 239 | [Classic SSB Blob]: https://ssbc.github.io/scuttlebutt-protocol-guide/#blobs 240 | [Classic SSB Signature]: https://ssbc.github.io/scuttlebutt-protocol-guide/#signature 241 | [Gabby Grove]: https://github.com/ssbc/ssb-spec-drafts/tree/master/drafts/draft-ssb-core-gabbygrove/00 242 | [Bamboo]: https://github.com/AljoschaMeyer/bamboo 243 | [Private Group]: https://github.com/ssbc/private-group-spec/tree/master/encryption 244 | [Private Group DM]: https://github.com/ssbc/private-group-spec/tree/master/direct-messages 245 | [Private Group PO box]: https://github.com/ssbc/private-group-spec/tree/master/po-box 246 | [Bendy Butt]: https://github.com/ssb-ngi-pointer/bendy-butt-spec 247 | [Buttwoo]: https://github.com/ssbc/ssb-buttwoo-spec/ 248 | [Private Box]: https://ssbc.github.io/scuttlebutt-protocol-guide/#private-messages 249 | [Envelope Spec]: https://github.com/ssbc/envelope-spec 250 | [Null pointer]: https://en.wikipedia.org/wiki/Null_pointer 251 | [UTF8]: https://datatracker.ietf.org/doc/html/rfc3629 252 | [Fusion Identity]: https://github.com/ssb-ngi-pointer/fusion-identity-spec/ 253 | [Bencode]: https://en.wikipedia.org/wiki/Bencode 254 | [Index feed]: https://github.com/ssbc/ssb-secure-partial-replication-spec#version-1 255 | -------------------------------------------------------------------------------- /008/bfe.json: -------------------------------------------------------------------------------- 1 | [ 2 | { 3 | "code": 0, 4 | "type": "feed", 5 | "formats": [ 6 | { "code": 0, "format": "classic", "data_length": 32, "sigil": "@", "suffix": ".ed25519" }, 7 | { "code": 1, "format": "gabbygrove-v1", "data_length": 32 }, 8 | { "code": 2, "format": "bamboo", "data_length": 32 }, 9 | { "code": 3, "format": "bendybutt-v1", "data_length": 32 }, 10 | { "code": 4, "format": "buttwoo-v1", "data_length": 32 }, 11 | { "code": 5, "format": "indexed-v1", "data_length": 32 } 12 | ] 13 | }, 14 | { 15 | "code": 1, 16 | "type": "message", 17 | "formats": [ 18 | { "code": 0, "format": "classic", "data_length": 32, "sigil": "%", "suffix": ".sha256" }, 19 | { "code": 1, "format": "gabbygrove-v1","data_length": 32 }, 20 | { "code": 2, "format": "cloaked", "data_length": 32, "sigil": "%", "suffix": ".cloaked" }, 21 | { "code": 3, "format": "bamboo", "data_length": 64 }, 22 | { "code": 4, "format": "bendybutt-v1", "data_length": 32 }, 23 | { "code": 5, "format": "buttwoo-v1", "data_length": 32 }, 24 | { "code": 6, "format": "indexed-v1", "data_length": 32 } 25 | ] 26 | }, 27 | { 28 | "code": 2, 29 | "type": "blob", 30 | "formats": [ 31 | { "code": 0, "format": "classic", "data_length": 32, "sigil": "&", "suffix": ".sha256" } 32 | ] 33 | }, 34 | { 35 | "code": 3, 36 | "type": "encryption-key", 37 | "formats": [ 38 | { "code": 0, "format": "box2-dm-dh", "data_length": 32 }, 39 | { "code": 1, "format": "box2-pobox-dh", "data_length": 32 } 40 | ] 41 | }, 42 | { 43 | "code": 4, 44 | "type": "signature", 45 | "formats": [ 46 | { "code": 0, "format": "msg-ed25519", "data_length": 64, "signature_length": 64, "suffix": ".sig.ed25519" } 47 | ] 48 | }, 49 | { 50 | "code": 5, 51 | "type": "encrypted", 52 | "formats": [ 53 | { "code": 0, "format": "box1", "suffix": ".box" }, 54 | { "code": 1, "format": "box2", "suffix": ".box2" } 55 | ] 56 | }, 57 | { 58 | "code": 6, 59 | "type": "generic", 60 | "formats": [ 61 | { "code": 0, "format": "string-UTF8" }, 62 | { "code": 1, "format": "boolean" }, 63 | { "code": 2, "format": "nil" }, 64 | { "code": 3, "format": "any-bytes" } 65 | ] 66 | }, 67 | { 68 | "code": 7, 69 | "type": "identity", 70 | "formats": [ 71 | { "code": 0, "format": "po-box" , "data_length": 32 }, 72 | { "code": 1, "format": "group", "data_length": 32 } 73 | ] 74 | } 75 | ] 76 | -------------------------------------------------------------------------------- /008/bfe.test.js: -------------------------------------------------------------------------------- 1 | // SPDX-FileCopyrightText: 2021 Anders Rune Jensen 2 | // 3 | // SPDX-License-Identifier: CC0-1.0 4 | 5 | const tape = require('tape') 6 | const bfeTypes = require('./bfe.json') 7 | 8 | tape('bfe', function (t) { 9 | t.false( 10 | bfeTypes.find((type, i) => type.code !== i), 11 | 'type.code = Array index in bfe.json' 12 | ) 13 | 14 | t.false( 15 | bfeTypes.reduce((acc, type) => { 16 | if (acc) return acc // already found a problem! 17 | 18 | const problem = type.formats.find((format, i) => format.code !== i) 19 | return problem 20 | ? { type: type.type, format: problem } 21 | : acc 22 | }, undefined), 23 | 'format.code = Array index in bfe.json' 24 | ) 25 | 26 | const sigils = new Set() 27 | loopOverTypes: for (const type of bfeTypes) { 28 | for (const {sigil} of type.formats) { 29 | if (sigils.has(sigil)) t.fail('Sigil is not unique for type ' + type.type) 30 | else if (sigil) { 31 | sigils.add(sigil) 32 | continue loopOverTypes; 33 | } 34 | } 35 | } 36 | t.pass('each sigil is unique to a type') 37 | t.equal(sigils.size, 3, 'there are only 3 sigils') 38 | 39 | const sigilsAndSuffixes = new Set() 40 | for (const type of bfeTypes) { 41 | for (const {sigil, suffix, format} of type.formats) { 42 | if (sigil || suffix) { 43 | const sigilSuffix = `|${sigil}|${suffix}|` 44 | if (sigilsAndSuffixes.has(sigilSuffix)) { 45 | t.fail('Sigil & suffix is not unique for type/format '+ type.type + ' ' + format) 46 | } 47 | else sigilsAndSuffixes.add(sigilSuffix) 48 | } 49 | } 50 | } 51 | t.pass('each sigil & suffix combination is unique') 52 | 53 | const sigillessSuffixFormats = bfeTypes.reduce((acc, type) => 54 | [ 55 | ...acc, 56 | ...type.formats.filter((format) => !format.sigil && format.suffix) 57 | ] 58 | , []) 59 | t.equal( 60 | sigillessSuffixFormats.length, 61 | new Set(sigillessSuffixFormats.map((type) => type.suffix)).size, 62 | 'every suffix-only format is unique' 63 | ) 64 | 65 | t.end() 66 | }) 67 | -------------------------------------------------------------------------------- /008/package.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "ssb-bfe-spec", 3 | "description": "Binary Field Encodings (BFE) spec for Secure Scuttlebutt (SSB)", 4 | "version": "0.8.0", 5 | "homepage": "https://github.com/ssbc/ssb-bfe-spec", 6 | "repository": { 7 | "type": "git", 8 | "url": "git://github.com/ssbc/ssb-bfe-spec.git" 9 | }, 10 | "main": "bfe.json", 11 | "files": [ 12 | "bfe.json" 13 | ], 14 | "scripts": { 15 | "test": "tape *.test.js | tap-spec", 16 | "prepublishOnly": "npm run test" 17 | }, 18 | "author": "public-domain", 19 | "contributors": [ 20 | "Anders Rune Jensen ", 21 | "Andre Staltz ", 22 | "Mix Irving " 23 | ], 24 | "license": "CC0-1.0", 25 | "devDependencies": { 26 | "tap-spec": "^5.0.0", 27 | "tape": "^5.3.1" 28 | } 29 | } 30 | -------------------------------------------------------------------------------- /009.md: -------------------------------------------------------------------------------- 1 | # Tangles 2 | 3 | Authors: Mix Irving , Andre Staltz 4 | 5 | Date: 2023-03-23 6 | 7 | License: CC0-1.0 8 | 9 | 10 | ## Abstract 11 | 12 | In any given set of SSB messages, it is impossible to determine their partial or 13 | total order based on timestamps alone. However, including the hashes of 14 | previous messages in a message allows us to determine the order of messages. In 15 | this SIP, we specify a consistent way of declaring hashes of previous messages, 16 | such that they form a directed acyclic graph (DAG). This DAG is useful for many 17 | use cases, including replication, and multi-writer "records". 18 | 19 | 20 | ## Motivation 21 | 22 | As a permissionless, decentralized, and eventually consistent database, SSB 23 | cannot rely on timestamps to determine the order of messages. Instead, when a 24 | message refers to the hash of another message, we can infer that the message 25 | that was referred to must have been created first, because we cannot create that 26 | hash without knowing the original data. These referred hashes effectively form 27 | a proof that a message was created after another message. 28 | 29 | A "tangle" in SSB is a pattern of declaring previous message hashes, which will 30 | define a directed acyclic graph (DAG) of messages. A tangle is a useful way to 31 | determine a partial ordering which in turn is useful for everything from 32 | replication to building multi-writer "records". 33 | 34 | A topological sort of a tangle renders a linear ordering of messages, which is 35 | useful for displaying messages in user interfaces. 36 | 37 | 38 | ## Terminology 39 | 40 | The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", 41 | "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be 42 | interpreted as described in RFC 2119. 43 | 44 | 45 | ## Specification 46 | 47 | There are different types of tangle, but they all MUST specify: 48 | 49 | 1. **Candidates messages**: the set of messages which _could_ be part of the 50 | tangle 51 | 2. **Recipe**: how the tangle (the DAG) is constructed from these candidates 52 | 53 | It is RECOMMENDED for every recipe to start with some "root message" and extend 54 | out from that point, checking the validity of messages as they are added. Some 55 | candidate messages may not have connections to the graph, or may be "invalid" 56 | extensions, in which case they are excluded from the tangle. 57 | 58 | A given message MAY belong to zero, one, or many tangles. Each tangle 59 | is a separate DAG. A tangle is identified by a human-readable string. 60 | 61 | We define: 62 | - **the root message** as the earliest message in a tangle. Causally it is the 63 | "oldest", and is often used to identify all other candidate messages. 64 | - **the tip(s)** of the tangle are the message(s) at the leading or "newest" end 65 | of the tangle i.e. it can transitively connected to the root, and no other 66 | messages yet link back to it. 67 | 68 | There MUST only be one root message per tangle. 69 | 70 | 71 | ### Tangle fields in messages 72 | 73 | Each message `m` in a tangle SHOULD have a `tangles` field in the `content` 74 | field, i.e. `m.content.tangles`. If `m.content.tangles` exists, its value MUST 75 | be an object with one field for each tangle. The key of each field MUST be a 76 | human-readable identifier string (lets refer to it here as `x`) for that tangle, 77 | and the value MUST be an object (called the "tangle data") with the fields: 78 | 79 | - `root`: the ID of the root message of the tangle or `null` if `m` is the root 80 | - `previous`: an Array of message IDs of the known tip(s) of the DAG at the time 81 | `m` was published, or `null` if `m` is the root 82 | 83 | 84 | ### Classic feed: single-author tangle 85 | 86 | The most trivial tangle is the classic SSB feed. In this case, the tangle for 87 | feed `A` is defined as: 88 | 89 | 1. **Candidates messages**: 90 | - MUST have `author` field equal to `A` 91 | 2. **Recipe**: 92 | 1. If the candidate message has `sequence` field equal to `1`, then its field 93 | `previous` MUST be `null` 94 | 2. If the candidate message has `sequence` field equal to `i` where `i` is a 95 | number greater than `1`, then its field `previous` MUST be the message ID of 96 | the message that has `sequence` field equal to `i - 1` 97 | 98 | 99 | ```mermaid 100 | flowchart RL 101 | A(A):::root;B(B);C(C);D(D); 102 | 103 | subgraph "feed" 104 | direction RL 105 | D-->C-->B-->A 106 | end 107 | 108 | classDef default fill: #e953da, stroke-width: 0, color: white 109 | classDef root fill: #4c44cf, stroke-width: 0, color: white 110 | classDef cluster stroke:#e953da,fill:none; 111 | ``` 112 | 113 | The tangle data these messages carry looks like: 114 | ```javascript 115 | A => { sequence: 1, previous: null, ... } 116 | B => { sequence: 2, previous: A, ... } 117 | C => { sequence: 3, previous: B, ... } 118 | D => { sequence: 4, previous: C, ... } 119 | ``` 120 | 121 | (The fields `author`, `signature`, `timestamp`, `hash`, and `content` have been 122 | ommited here to make the backlinking pattern clearer) 123 | 124 | This is a special case of a tangle, for a few reasons: 125 | 126 | - This tangle does not have "tangle data" with `root` and `previous` 127 | - The DAG is linear, with no branches or merges 128 | 129 | 130 | ### Multi-author tangle 131 | 132 | While the classic feed is a special case of a tangle without branches, in the 133 | case of multiple authors, we need to support branches and merges. This is 134 | because different authors may contribute concurrently, or while offline. 135 | 136 | The following is not a description of a specific "tangle data" with candidates 137 | and recipes, but rather a template for tangle data when there are messages from 138 | multiple authors. 139 | 140 | As a general pattern for multi-author tangle data: when a message is published, 141 | its `previous` field SHOULD have the IDs of all known **tips** of the tangle, 142 | and it SHOULD NOT have any other message IDs in its `previous` field. 143 | 144 | As an example, suppose there are messages with tangle data that look like: 145 | 146 | ```javascript 147 | A => { root: null, previous: null } 148 | B => { root: A, previous: [A] } 149 | X => { root: A, previous: [B] } 150 | Y => { root: A, previous: [B] } 151 | M => { root: A, previous: [X, Y] } 152 | ``` 153 | 154 | Then the directed acyclic graph (DAG) looks like: 155 | 156 | ```mermaid 157 | flowchart RL 158 | A(A):::root;B(B);X(X);Y(Y);M(M); 159 | 160 | M-->X-->B-->A 161 | M-->Y--->B 162 | 163 | classDef default fill: #e953da, stroke-width: 0, color: white 164 | classDef root fill: #4c44cf, stroke-width: 0, color: white 165 | ``` 166 | _Diagram where messages X, Y were both published concurrently (so were unaware 167 | of one another). Message M is aware of both X and Y, and extends the tangle 168 | from them. M is now the new "tip" of the tangle._ 169 | 170 | Note that message `M` points to `X` and `Y` as its `previous` messages. This 171 | conforms to the general pattern for multi-author tangle data. 172 | 173 | Further, the root message MUST NOT include its own ID (not known until 174 | published), so it sets its `root` value as `null`, which means "I am the root". 175 | 176 | The "tangle data" `{ root, previous }` for e.g. a "chess game tangle" SHOULD be 177 | stored at `msg.value.content.tangles.chess`, note the choice of a human-friendly 178 | name `chess` for the domain of a chess game. 179 | 180 | Some tangles may have _candidates_ that define a strict subset of messages as 181 | valid, e.g. only messages authored by peers the original author follows. 182 | 183 | 184 | ## Considerations 185 | 186 | **Privacy Considerations** – if a tangle is encrypted (e.g. to a group) then all 187 | messages in it should be encrypted similarly, because message backlinks reveal 188 | the other authors who may have participated in the tangle, which may be a 189 | privacy risk. This means that you also want to be careful of revealing 190 | individual messages in a tangle (a feature box1, box2 support). 191 | 192 | **Implementation Considerations** – it's important that when defining a tangle, 193 | the candidate messages and recipe are very clearly defined. The recipe may also 194 | need to include how a tangle should be presented to users (e.g. whether a tangle 195 | is linearised, and how). 196 | 197 | ## References 198 | 199 | ### Informative 200 | 201 | - [ssbdrv's document about tangles](https://github.com/cn-uofbasel/ssbdrv/blob/master/doc/tangle.md) 202 | - [Tangle Origin](./009/tangle_origin.md) Story 203 | - [tangle-js](https://gitlab.com/tangle-js) - a collection of NodeJS modules 204 | used for building tangles data, and reducing data held in messages. 205 | Effectively turns tangles into CRDTs 206 | 207 | ### Implementation 208 | 209 | JavaScript: 210 | - [`ssb-sort`](https://github.com/ssbc/ssb-sort) – used for tangle sorting used 211 | by many JS clients. 212 | - [`ssb-tribes/lib/get-group-tangle.js`](https://github.com/ssbc/ssb-tribes/blob/master/lib/get-group-tangle.js) – uses `@tangle` modules to determine tangle tips 213 | 214 | -------------------------------------------------------------------------------- /009/tangle_origin.md: -------------------------------------------------------------------------------- 1 | # Tangle Origin 2 | 3 | The shape and pattern of tangles come from the observation that because message 4 | ids are derived from hashes (over content, timestamp, signature), it is only 5 | possible to reference messages that have been published _in the past_. Said 6 | differently - you can only point _backwards in time_ in scuttlebutt. 7 | 8 | Here's the how you reach the idea of a tangle: 9 | 10 | ## 1. Identify the messages in the thread 11 | 12 | We start by identifying which messages are related to a thread. A thread starts 13 | with a "root" message `A`, and we look for all messages which have referenced 14 | `A` as their "root" (i.e. check the `.tangles.root` field) 15 | 16 | ```mermaid 17 | flowchart RL 18 | A(A):::root;B(B);E(E);C(C);D(D); 19 | 20 | B & C & D & E-->A 21 | 22 | %% styling 23 | classDef default fill: #e953da, stroke-width: 0, color: white 24 | classDef root fill: #4c44cf, stroke-width: 0, color: white 25 | ``` 26 | _Diagram showing the **backlinks** - references pointing back to a particular 27 | message - from thread replies to the root message of the thread._ 28 | 29 | ## 2. Order the messages 30 | 31 | ### a) by timestamp 32 | The naive approach is to order by timestamp. We have two timestamps to work 33 | with - the **received timestamp** (when we first got the message) or the 34 | **asserted timestamp** (when the author said they published it). We can't rely 35 | on either because sometimes people's system clocks are wrong, or sometimes 36 | people lie and assert a wrong timestamp. 37 | 38 | ```mermaid 39 | flowchart RL 40 | A(A):::root;B(B);E(E);C(C);D(D); 41 | 42 | E-->D-->C-->A-->B 43 | 44 | %% styling 45 | classDef default fill: #e953da, stroke-width: 0, color: white 46 | classDef root fill: #4c44cf, stroke-width: 0, color: white 47 | ``` 48 | _Diagram showing how ordering by timestamp can result in "incorrect" ordering._ 49 | 50 | This problem doesn't exist in centralised systems because there is (generally) 51 | a globally consistent time - the clock of the server. This time can be "wrong", 52 | but will still conserve relative ordering. 53 | 54 | ### b) by listing all previous messages 55 | 56 | And improvement on this is if each message list all messages it was aware of 57 | **previous** to it 58 | 59 | So we might have something like: 60 | 61 | msg | previous messages | notes 62 | ----|----------------|-- 63 | `A` | `[]` | the root has no messages before it 64 | `B` | `[A]` | 65 | `C` | `[A, B]` | 66 | `D` | `[A, B]` | see that `D` did not know about `C` _(did not have msg yet)_ 67 | `E` | `[A, B, C, D]` | 68 | 69 | ```mermaid 70 | flowchart RL 71 | A(A):::root;B(B);E(E);C(C);D(D); 72 | 73 | B-->A 74 | 75 | C-->A 76 | C-->B 77 | 78 | D-->A 79 | D-->B 80 | 81 | E-->A 82 | E-->B 83 | E-->C 84 | E-->D 85 | 86 | %% styling 87 | classDef default fill: #e953da, stroke-width: 0, color: white 88 | classDef root fill: #4c44cf, stroke-width: 0, color: white 89 | ``` 90 | _Diagram showing backlinks from each message to each previous message_ 91 | 92 | This is starting to reveal an order! We can be confident in this ordering 93 | because each of these backlinks is pointing to a message key, which is a _hash_ 94 | of the message. These **cannot** be known ahead of time (because hashes are 95 | sensitive to message content, which includes the asserted publishing time, and 96 | this cannot be known), therefore we have a guarentee that each of these links 97 | points **backwards in time** 98 | 99 | BUT, you can already see this isn't going to be sustainable. As a thread gets 100 | long, the previous messages that need to be mentioned keeps increasing. This 101 | leads us to our final solution. 102 | 103 | ### c) by listing immediately previous messages 104 | 105 | In the above table, we can see that there's a lot of redundancy. Check out this 106 | alternative 107 | 108 | msg | immediately previous 109 | ----|---------------------- 110 | `A` | `[]` 111 | `B` | `[A]` 112 | `C` | `[B]` 113 | `D` | `[B]` 114 | `E` | `[C, D]` 115 | 116 | If we look at any message we can derive **all** previous messages previous to 117 | it by recurrsively following the trail of **immediately previous**, collecting 118 | the messages mentioned as we go. 119 | 120 |
121 | See code example (click) 122 | In code, something like: 123 | 124 | ```js 125 | function allPrevious (msg) { 126 | return ( 127 | immediatelyPrev(msg) + 128 | immediatelyPrev(msg).map(prevMsg => allPrevious(prevMsg)) // recursion! 129 | ) 130 | } 131 | ``` 132 |
133 | 134 | So for a message like `C` that means: 135 | ``` 136 | allPrevious(C) 137 | = [B] + allPrevious(B) 138 | = [B] + ([A] + allPrevious(A)) 139 | = [B] + [A] + [] 140 | = [B, A] 141 | ``` 142 | 143 | Similarly with `E` we can follow collect previous links recursively (following 144 | the immediately previous links `[C, D]`) to get `[C, D, B, A]` 145 | 146 | ``` 147 | All messages prior to E 148 | = [C, D] + allPrevious(C) + allPrevious(D) 149 | = ... 150 | = [C, D, B, A] 151 | ``` 152 | 153 | This is great because it shows that by listing ONLY the messages immediately 154 | previous to the message being published, we can still figure out ALL the 155 | messages which are previous to our message in the thread. 156 | 157 | If we render these backlinks in a graph we can see a much less chaotic graph 158 | emerging: 159 | 160 | ```mermaid 161 | flowchart RL 162 | A(A):::root;B(B);E(E);C(C);D(D); 163 | 164 | E-->C-->B-->A 165 | E-->D-->B 166 | 167 | %% styling 168 | classDef default fill: #e953da, stroke-width: 0, color: white 169 | classDef root fill: #4c44cf, stroke-width: 0, color: white 170 | ``` 171 | 172 | _Diagram of a tangle of messages showing backlinks only to messages that were 173 | the current most recent nodes on the graph at time of publishing_ 174 | 175 | 176 | ### d) by listing immediately previous messages + breaking ties! 177 | 178 | The last solution for ordering is awesome, but it also leaves some ambiguity 179 | about how to display the messages. (Personally, I would love to see a client 180 | which shows messages in a graph like this, because sometimes it's very relevant 181 | that the author of `D` did not know about `C`) 182 | 183 | Most scuttlebutt clients flatten these graphs into a linear history. To do this 184 | you need to identify when it's not clear about ordering, and provide a rule for 185 | breakaing ties. 186 | In our example above this means sorting `[C, D]`. 187 | 188 | The most common tie breaking algorithm is something like: 189 | > sort by min("asserted timestamp", "received timestamp") 190 | 191 |
192 | See code example (click) 193 | or in code: 194 | 195 | ```js 196 | function sortMsgs (msgs) { 197 | return msgs.sort((A, B) => ( 198 | Math.min(B.timestamp, B.value.timestamp) - 199 | Math.min(A.timestamp, A.value.timestamp) 200 | )) 201 | } 202 | ``` 203 | _We take the min of these two timestamps because we trust our own clock more 204 | than a strangers? It limits their ability to lie about being in the future._ 205 |
206 | 207 | 208 | ```mermaid 209 | flowchart RL 210 | A(A):::root;B(B);E(E);C(C);D(D); 211 | 212 | E-->C-->B-->A 213 | E-->D-->B 214 | C-.->D 215 | 216 | %% styling 217 | classDef default fill: #e953da, stroke-width: 0, color: white 218 | classDef root fill: #4c44cf, stroke-width: 0, color: white 219 | ``` 220 | _Diagram the same as \(c\) but with a **solid line** showing the effect of 221 | tie-breaking by timestamp._ 222 | 223 | 224 | ## 3. Publishing a new message to a thread 225 | 226 | Following the algorithm described in \(c\) / (d), if a new message `F` was to 227 | be published (that had all the messages shown above), it would only need to list 228 | the immediately previous message(s) i.e. `[E]`, because that _implicitly_ tells 229 | us that it follows `[A, B, C, D, E]`. 230 | 231 | -------------------------------------------------------------------------------- /010.md: -------------------------------------------------------------------------------- 1 | # Threads 2 | 3 | Authors: Mix Irving mix@protozoa.nz 4 | 5 | Date: 2023-03-27 6 | 7 | License: CC0-1.0 8 | 9 | ## Abstract 10 | 11 | The `post` message type is the foundational unit for talking with peers on 12 | the original scuttlebutt network. 13 | 14 | `post` messages can be used standalone, but are most commonly collected as 15 | sets of related messages and formed into a "thread". 16 | 17 | During rendering, threads of posts are also further _decorated_ with `vote` 18 | messages (for likes), and `about` messages (for displaying author details). 19 | 20 | This SIP defines the expected schema for `post` and `vote` messages, and 21 | how they form a directed acyclic graph known as "threads" in SSB. 22 | 23 | ## Terminology 24 | 25 | The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC 2119](https://tools.ietf.org/html/rfc2119). 26 | 27 | ## Specification 28 | 29 | ## 1. `post` schema 30 | 31 | The following schemas relate to the `content` field in SSB messages. If a field is commented with `// optional`, it means it MAY appear in the `content` field, otherwise it SHOULD appear. 32 | 33 | ### 1.1. Root message 34 | 35 | A "root message" is the first message that starts a particular thread 36 | 37 | ```javascript 38 | { 39 | type: 'post', 40 | text: String, 41 | channel: String, // optional 42 | contentWarning: String, // optional 43 | mentions: Array, // optional 44 | recps: Array // optional 45 | } 46 | ``` 47 | 48 | ### 1.2. Response message 49 | 50 | These messages are used to post responses into a thread. They are NOT 51 | used to reply to specific messages, rather to the currently held context 52 | of a thread (see Tangles below). 53 | 54 | ```javascript 55 | { 56 | type: 'post', 57 | text: String, 58 | root: MsgId, 59 | branch: Array, 60 | channel: Channel, // optional 61 | contentWarning: String // optional 62 | mentions: Array, // optional 63 | recps: Array // optional 64 | } 65 | ``` 66 | 67 | NOTE: sometimes `branch` MAY be a `MsgId` instead of `Array` 68 | 69 | ### 1.2. Channel 70 | 71 | `Channel` should be the a `String` which: 72 | - SHOULD NOT contain 73 | - hash symbol `#` 74 | - punctuation `,."!?` 75 | - brackets `()[]` 76 | - spaces ` `, 77 | - SHOULD NOT be longer than 30 characters long 78 | 79 | When parsing a channel, you MUST normalize the channel name by: 80 | 1. removing disallowed characters 81 | 2. truncating to 30 characters (if required) 82 | 3. lowering the case of all characters 83 | - i.e. `NewZealand` is to be treated the same as `newzealand`, `newZealand` etc. 84 | 85 | See [ssb-ref](https://github.com/ssbc/ssb-ref/blob/main/index.js#L95-L104) for a reference implementation of `normalizeChannel` 86 | 87 | ### 1.3. Mention 88 | 89 | May take one of the following forms: 90 | 91 | ```javascript 92 | { 93 | link: FeedId, 94 | name: String // optional 95 | } 96 | ``` 97 | ```javascript 98 | { 99 | link: MsgId 100 | } 101 | ``` 102 | ```javascript 103 | { 104 | link: BlobId, 105 | size: Int, // optional 106 | name: String // optional 107 | } 108 | ``` 109 | 110 | You will also find `Mention` in the wild which are just a plain string 111 | ``` 112 | FeedId | MsgId | BlobId 113 | ``` 114 | 115 | ## 2. `vote` Schema 116 | 117 | `vote` messages are used for responding to `post` messages. Sometimes they are 118 | called "likes" but they are used for expressing a range of emotional responses. 119 | 120 | ```javascript 121 | { 122 | type: "vote", 123 | vote: { 124 | link: MsgId 125 | expression: String, 126 | value: 1 127 | }, 128 | root: MsgId, // optional 129 | branch: Array // optional 130 | } 131 | ``` 132 | 133 | where: 134 | - `link` is id of a `post` message you are posting an emotional response to 135 | - `expression` is a string describing the emotional response 136 | - if this string can be mapped into an emoji e.g. `heart` => :heart: then 137 | then that is displayed 138 | - otherwise it's the interpretation falls back to "like" / :+1: 139 | - `value` is the weight of your `expression` 140 | - `0` means "make my response nothing again please" 141 | 142 | Some clients also add thread tangle data to `vote` messages with the `root` and 143 | `branch` fields. The rationale of this is that with more frequent messages 144 | contributing to the tangle, forks are less likely - i.e. the tangle trends 145 | towards being more linear. 146 | 147 | 148 | ## 3. Thread 149 | 150 | A "thread" is a collection of `post` (and sometimes `vote`) messages that 151 | are responses to a growing context. For the thread to make sense we need: 152 | - to gather all messages related to the thread 153 | - to know how to _order_ the messages 154 | 155 | Because we don't have a centralised server, we don't have a guarenteed 156 | "clock" to which we can use to determine a canonical `createdAt` time for 157 | each message. Our solution is to use build a Directed Acyclic Graph (DAG) 158 | out of the messages, by backlinking to messages earlier in history. 159 | 160 | ### 3.1. Thread tangle 161 | 162 | We define: 163 | - **thread tangle** - a Directed Acyclic Graph of messages 164 | - **root message** - the first message of a particular thread tangle 165 | particular rules 166 | - **tangle tip(s)** - the messages in our tangle which have no messages 167 | linking to them 168 | 169 | ```mermaid 170 | flowchart RL 171 | C(C)-->B(B)-->A(A):::root 172 | 173 | %% styling 174 | classDef default fill: #e953da, stroke-width: 0, color: white 175 | classDef root fill: #4c44cf, stroke-width: 0, color: white 176 | ``` 177 | _Figure 1. A thread of `post` causally ordered with backlinks. The "root" of 178 | this thread tangle is message `A`. The "tip" of this tangle is message `C`._ 179 | 180 | To build this tangle, we add two new fields to any `post` messages following 181 | the root message: 182 | - `root`: set to the id of the root message of the thread 183 | - `branch`: set to the id of the latest message(s) in the thread 184 | - NOTE: this field is used for causal ordering, NOT for indicating what you are replying to (though these concepts may converge at times) 185 | 186 | For each of the messages in _Fig. 1_ the root and branch are as follows: 187 | ```javascript 188 | A => {} 189 | B => { root: A, previous: [A] } 190 | C => { root: A, prevoius: [B] } 191 | ``` 192 | 193 |
194 | See full content of A,B,C (click) 195 | Examples of what the content of messages in _Fig 1._ would look like 196 | 197 | ```javascript 198 | // A 199 | { 200 | type: 'post', 201 | text: 'Lets have a shared dinner!' 202 | } 203 | ``` 204 | ```javascript 205 | // B 206 | { 207 | type: 'post', 208 | text: 'Love it, shall we make pizza?', 209 | root: A, 210 | branch: [A] 211 | } 212 | ``` 213 | ```javascript 214 | // C 215 | { 216 | type: 'post', 217 | text: 'I will bring pineapple!', 218 | root: A, 219 | branch: [B] 220 | } 221 | ``` 222 |
223 |
224 | 225 | 226 | Because new messages may be posted concurrently in scuttlebutt, it's possible 227 | for threads to become "forked". We also say the graph now has "multiple tips". 228 | 229 | ```mermaid 230 | flowchart RL 231 | A(A):::root;B(B);C(C);D(D);M(M);N(N); 232 | N---M---C & D-->B-->A 233 | 234 | classDef default fill: #e953da, stroke-width: 0, color: white 235 | classDef root fill: #4c44cf, stroke-width: 0, color: white 236 | classDef hidden fill:none,stroke:none,color:#ffffff00; 237 | class M,N hidden 238 | 239 | linkStyle 0,1,2 stroke-width:0,stroke:none; 240 | ``` 241 | 242 | When you are publishing a new message to the thread, and notice there are 243 | multiple tips, then those tips become your `previous` 244 | 245 | ```mermaid 246 | flowchart RL 247 | A(A):::root;B(B);C(C);D(D);M(M);N(N); 248 | N---M-->C & D-->B-->A 249 | 250 | classDef default fill: #e953da, stroke-width: 0, color: white 251 | classDef root fill: #4c44cf, stroke-width: 0, color: white 252 | classDef hidden fill:none,stroke:none,color:#ffffff00; 253 | class N hidden 254 | 255 | linkStyle 0 stroke-width:0,stroke:none; 256 | ``` 257 | 258 | ```mermaid 259 | flowchart RL 260 | A(A):::root;B(B);C(C);D(D);M(M);N(N); 261 | N-->M-->C & D-->B-->A 262 | 263 | classDef default fill: #e953da, stroke-width: 0, color: white 264 | classDef root fill: #4c44cf, stroke-width: 0, color: white 265 | ``` 266 | 267 | ```javascript 268 | D => { root: A, previous: [B] } 269 | M => { root: A, previous: [C, D] } 270 | N => { root: A, previous: [N] } 271 | ``` 272 | 273 |
274 | See full content of D, M, N (click) 275 | ```javascript 276 | // D 277 | { 278 | type: 'post', 279 | text: 'yes, so long as we avoid pineapple (I am alergic)', 280 | root: A, 281 | branch: [B] 282 | } 283 | ``` 284 | 285 | ```javascript 286 | // M 287 | { 288 | type: 'post', 289 | text: 'Oh all good, I can leave the pineapple.' 290 | root: A, 291 | branch: [C, D] 292 | } 293 | ``` 294 |
295 |
296 | 297 | ### 3.2. Disconnected messages 298 | 299 | We define a **disconnected message** as a message which asserts it is part 300 | of a thread, but does not have a continuous link back to the root message of 301 | our thread 302 | 303 | ```mermaid 304 | flowchart RL 305 | A(A):::root;B(B);C(C);D(D);M(M);N(N);Y(Y) 306 | N-->M-->C & D-->B-->A 307 | Y:::disconnected-.->X:::hidden 308 | 309 | classDef default fill: #e953da, stroke-width: 0, color: white 310 | classDef root fill: #4c44cf, stroke-width: 0, color: white 311 | classDef hidden stroke:none,fill:none,color:none; 312 | classDef disconnected stroke:none,fill:#e95353,color:none; 313 | ``` 314 | 315 | Here message `Y` says it's part of the thread, but it backlinks to some message 316 | that we do no have 317 | ```javascript 318 | y => { root: A, branch: [X] } 319 | ``` 320 | 321 | This message is not well connected, but it's common to render it in later steps 322 | regardless 323 | 324 | 325 | ## 4. Rendering a thread 326 | 327 | Each message is generally displayed showing: 328 | - author name + image (derived from `msg.value.author` and other messages outside 329 | the scope of this spec) 330 | - `timestamp` 331 | - `text` rendered as markdown 332 | - sometimes further data such as 333 | - a summary of `vote` responses 334 | - info about any `fork` that this `post` started 335 | - a list of other threads which have linked to this message (backlinks) 336 | 337 | ### 4.1. Linear rendering of a tangle 338 | 339 | A tangle with forks is hard to render as a nice linear output (the most familiar 340 | way humans like to engage with threads in). 341 | 342 | We build a tangle, to get partial ordering, then for **concurrent messages**, we 343 | break ties using the "timestamp" that the message was published at. 344 | (timestamp here is the minimum of the asserted timestamp (`msg.value.timestamp)` 345 | and the timestamp the message was received at (`msg.timestamp` in the JS 346 | implementation) 347 | 348 | 349 | ```mermaid 350 | flowchart RL 351 | A(A):::root;B(B);C(C);D(D);M(M);N(N);Y(Y):::disconnected 352 | N-->M-->C & D-->B-->A 353 | Y 354 | 355 | 356 | D-.->C 357 | N-.->Y-.->M 358 | linkStyle 6,7,8 stroke:#53e9d4,stroke-width:4; 359 | 360 | classDef default fill: #e953da, stroke-width: 0, color: white 361 | classDef root fill: #4c44cf, stroke-width: 0, color: white 362 | classDef disconnected stroke:none,fill:#e95353,color:none; 363 | ``` 364 | _Figure. showing the creation of a tangle, then tie-breaking between concurrent 365 | messages `C`,`D`, and best-guess insertion of disconnected message `Y`_ 366 | 367 | For **disconnected messages** we try to place them in the correct position by 368 | timestamp alone. It could be a good idea to indicate in UI that this message may 369 | be in the wrong place and is missing context. 370 | 371 | 372 | ```mermaid 373 | flowchart RL 374 | A(A):::root;B(B);C(C);D(D);M(M);N(N);Y(Y):::disconnected 375 | N-->Y-->M-->D-->C-->B-->A 376 | classDef default fill: #e953da, stroke-width: 0, color: white 377 | classDef root fill: #4c44cf, stroke-width: 0, color: white 378 | classDef disconnected stroke:none,fill:#e95353,color:none; 379 | ``` 380 | _Figure. showing a linearised thread tangle, along with a "best guess" placement 381 | of disconnected message `Y`._ 382 | 383 | 384 | ### 4.2. Scuttlebutt Markdown 385 | 386 | The `text` field of `post` messages is Scuttlebutt flavoured Markdown, which is 387 | basic Markdown with a couple of important privacy enhancing features. 388 | 389 | #### 4.2.1 Image rendering 390 | 391 | DO NOT render images that link to the web. This leaks data, which allows 392 | tracking of users. 393 | ```markdown 394 | ![unsafe image](https://upload.wikimedia.org/wikipedia/commons/1/11/Panopticon.jpg) 395 | ``` 396 | Such images SHOULD instead be rendered as a link with an alert/ warning 397 | (see 4.2.2). 398 | 399 | ONLY render images that link to blobs that are hosted locally: 400 | ```markdown 401 | ![snails](&7AUsuVcxUzIJZPBBLqR3O2KvIiuECgdYq9lzEh0jcS8=.sha256) 402 | ``` 403 | 404 | #### 4.2.2 Link rendering 405 | 406 | When a link to the web is present, the interface SHOULD render an alert to users 407 | on-click. Similar to 4.2.1, this is to prevent accidental tracking. 408 | 409 | ```markdown 410 | [click here](www.google.com) 411 | ``` 412 | 413 | #### 4.2.3 Old @mentions (deprecated) 414 | 415 | Older messages used `mentions` in a fancy way so that the text didn't have 416 | honking ugly markdown links to feedIds in it, but most clients have given 417 | this up 418 | 419 | ```javascript! 420 | { 421 | type: 'post' 422 | text: 'This was an innovation @pfraze came up with', 423 | mentions: [ 424 | { 425 | link: '@hxGxqPrplLjRG2vtjQL87abX4QKqeLgCwQpS730nNwE=.ed25519', 426 | name: '@pfraze' 427 | } 428 | ] 429 | } 430 | ``` 431 | The mentions are then used to map the `text` to this before rendering: 432 | ```markdown 433 | This was an innovation [@pfraze](@hxGxqPrplLjRG2vtjQL87abX4QKqeLgCwQpS730nNwE=.ed25519) 434 | came up with', 435 | ``` 436 | 437 | ### 4.3. Blocked authors 438 | 439 | If some message `B` is from a blocked author, we may still use its tangle data 440 | for causal ordering, but not render that message in the thread. 441 | 442 | The message MUST NOT be displayed to by deault in the UI, but it is up to the 443 | particular client whether this means 444 | - not rendering it at all 445 | - rendering an empty skeleton message saying "message from blocked author" 446 | - rendering a skeleton saying "message from blocked author, do you want to 447 | display?" 448 | 449 | ### 4.4. Content warnings 450 | 451 | If the post contains a `contentWarning` field, the UI SHOULD NOT display the 452 | raw content and instead display a message which: 453 | 1. alerts that this message has a content warning 454 | 2. displays the String in `contentWarning` 455 | 456 | If the user clicks "show content" (or similar) then the message should be 457 | rendered as usual. 458 | 459 | 460 | ## Reference implementation 461 | 462 | - [ssb-threads](https://github.com/ssbc/ssb-threads) 463 | 464 | 465 | ## Informative references 466 | 467 | - this spec first started in thread: 468 | - `ssb:/message/classic/dm4NuPPxA_Jyxpu2yFj7SzAeF1u4G_75D-bzJGDt504=` 469 | - (or `%dm4NuPPxA/Jyxpu2yFj7SzAeF1u4G/75D+bzJGDt504=.sha256`) 470 | -------------------------------------------------------------------------------- /011.md: -------------------------------------------------------------------------------- 1 | # BIPF.tinySSB, a Serialization Format for JSON-like Data Types 2 | 3 | Author: cft 4 | 5 | Date: 2023-07-16 6 | 7 | License: CC0-1.0 8 | 9 | URL: [```https://github.com/ssbc/sips/blob/master/XXX.md```](https://github.com/ssbc/sips/blob/master/XXX.md) 10 | 11 | ## Abstract 12 | 13 | BIPF (Binary In-Place Format) is a serialization format for JSON-like 14 | data types that is optimized for in-place memory access. It is used in 15 | some versions of Secure Scuttlebut (SSB), including tinySSB. This 16 | document describes BIPF as used in tinySSB. 17 | 18 | ## Motivation 19 | 20 | SSB.classic uses human-readable JSON as representation format: 21 | append-only log entries are stored as JSON-encoding of the respective 22 | SSB data structure. Using JSON, which has no support for binary data, 23 | mandates that byte arrays are specially encoded (as ASCII strings 24 | according to BASE64), which further bloats a log entry. By choosing a 25 | binary representation as in BIPF, considerable space savings can be 26 | achieved and unnecessary transformations of the representation can be 27 | avoided. CBOR encoding also provides these features but was not 28 | considered because of the possibility of "indefinite lists encoding" 29 | which prevents the skipping of a list without parsing its elements. A 30 | link to a comprehensive comparison of serialization approaches can be 31 | found in the References section; see also the entry for 32 | BIPF.rationale. 33 | 34 | In order to explain the difference between "JSON data types" and 35 | BIPF's "JSON-like data types" we provide two BNF grammers. The first 36 | shows a simplified grammar for JSON (that ommits low-level rules 37 | e.g. for integers or floating point numbers and strings): 38 | 39 | ``` 40 | JSON_VAL ::= JSON_ATOM / JSON_LIST / JSON_DICT 41 | JSON_ATOM ::= 'true' / 'false' / 'null' / INT_VAL / DOUBLE_VAL / STR_VAL 42 | JSON_LIST ::= '[' ']' / '[' JSON_VAL *(',' JSON_VAL) ']' 43 | JSON_DICT ::= '{' '}' / '{' STR_VAL ':' JSON_VAL *(',' STR_VAL ':' JSON_VAL) '}' 44 | ``` 45 | 46 | BIPF is more expressive in that it also supports byte arrays. 47 | Moreover, any atom can be used as a key for a dictionary, not only 48 | strings. The following grammer for BIPF permits a direct comparison 49 | with the previous grammer for JSON: 50 | 51 | ``` 52 | BIPF_VAL ::= BIPF_ATOM / BIPF_LIST / BIPF_DICT 53 | BIPF_ATOM ::= 'true' / 'false' / 'null' / INT_VAL / DOUBLE_VAL / STR_VAL / BYTES_VAL 54 | BIPF_LIST ::= '[' ']' / '[' BIPF_VAL *(',' BIPF_VAL) ']' 55 | BIPF_DICT ::= '{' '}' / '{' BIPF_ATOM ':' BIPF_VAL *(',' BIPF_ATOM ':' BIPF_VAL) '}' 56 | ``` 57 | 58 | This grammer can be used as a human-readable format of BIPF data items 59 | for documentation purposes. However, BIPF is a binary format wherefore 60 | integer values, for example, are not represented as digits drawn from 61 | the ASCII character set, but as a series of bytes that encode this 62 | value. The binary BIPF represenation of the value 123 will be the byte 63 | array #7b#, instead of "123" which as a string would translate to the 64 | bytes #313233#. 65 | 66 | The Specification section below provides the complete list of encoding 67 | rules for the BIPF-supported data types. 68 | 69 | 70 | ## Terminology 71 | ``` 72 | data type Some abstract data type that we wish to encode such 73 | that instanves of this data type can be transferred and 74 | reconstructed by a remote peer. These are integers, 75 | strings or lists, for example. 76 | 77 | encoding rules A set or reversible mappings for turning potentially 78 | structured in-memory data values into a flat series 79 | of bytes that can be stored and/or transferred. 80 | 81 | external representation, sometimes also called "transfer syntax" 82 | A series of bytes that represent the (internal) 83 | values of a data type. From that series of bytes, a 84 | remote peer is able to create its own internal 85 | value of the data. 86 | 87 | serialization Turning an internal instance of a BIPF-supported data 88 | type item into a series of bytes that can be transferred. 89 | 90 | deserialization Turning a received BIPF series of bytes into an 91 | internal replica of the original data item. 92 | 93 | human readable A format expressed in ASCII characters and, if supported 94 | by the rendering, of UTF8 unicode characters in strings. 95 | This is in contrast with BIPF's binary representation where 96 | encoded values cannot be easily parsed by humans. 97 | ``` 98 | 99 | ## Specification 100 | 101 | For each of the 7 atomic and 2 structured types supported by BIPF, a 102 | bit pattern is assigned. These type identifiers are later used in the 103 | encoding of a corresponding value: 104 | 105 | ``` 106 | STRING : 0 (000) // utf8 encoded string 107 | BYTES : 1 (001) // raw byte sequence 108 | INT : 2 (010) // little endian, two's complement, minimal number of bytes 109 | DOUBLE : 3 (011) // IEEE 754-encoded double precision floating point 110 | LIST : 4 (100) // sequence of bipf-encoded values 111 | DICT : 5 (101) // sequence of alternating bipf-encoded key and value 112 | BOOLNULL: 6 (110) // 1 = true, 0 = false, no value means null 113 | EXTENDED: 7 (111) // custom type. Specific type should be indicated by varint at start of buffer 114 | ``` 115 | 116 | Note that the ```BOOLNULL``` bit pattern is used for three different 117 | atomic types. 118 | 119 | BIPF values are serialized with a TYPE-LENGTH-VALUE (TLV) encoding. 120 | To this end, T and L are combined into a single integer value called 121 | _tag_ which is encoded with unsigned LEB128 (also called 122 | ```varint```). The encoded _tag_ is then prepended to the bytes of 123 | the encoding of the value V proper, as shown in the following pseudo 124 | code: 125 | 126 | ``` 127 | bipf_enc(V) ::= concat( tag(V.type,V.length), V.bytes ) 128 | tag(typ,len) ::= LEB128( len<<3 + typ ) 129 | ``` 130 | 131 | LEB128 encoding (used for encoding the tag) is done by producing one 132 | encoded byte for each 7 input bit group, starting with the least 133 | significant group of 7 bits. All _encoded_ bytes will have the most 134 | significant bit set, except the last byte. Zero is encoded as byte 135 | 0x00. 136 | 137 | Integer values are encoded with the minimally required number of bytes in 138 | little-endian order using two's complement representation. 139 | 140 | Lists are encoded by prepending to the concatenation of BIPF-encoded 141 | elements a tag with ```typ=4``` and a ```len``` value which is the sum 142 | of the lengths of the BIPF-encoded elements: 143 | 144 | ``` 145 | bipf([e1, e2]) = concat( tag(4,len(bipf(e1))+len(bipf(e2)))), bipf(e1), bipf(e2) ) 146 | ``` 147 | 148 | A dictionary is encoded as if one would encode an alternating list of 149 | key and value elements, except that the ```typ``` value is DICT (5). 150 | 151 | ``` 152 | bipf({e1: e2}) = concat(tag(5,len(bipf(e1))+len(bipf(e2)))), bipf(e1), bipf(e2)) 153 | ``` 154 | 155 | Following the JSON tradition, a BIPF library will typicall provide at least 156 | these two methods: 157 | ``` 158 | bipf_dumps(V) turns an in-memory data item V into a series of bytes 159 | bipf_loads(S) turns a series of bytes S into the corresponding in-memory data item 160 | ``` 161 | 162 | One can expect BIPF libraries to have additional convenience methods like 163 | ```bipf_predict_encoded_length(V)``` etc. 164 | 165 | ## Considerations 166 | 167 | (a) BIPF.original (see the References section) departs from this 168 | document in the way integer values are encoded. In 169 | [```https://github.com/ssbc/bipf-spec```](https://github.com/ssbc/bipf-spec), 170 | the authors use a fixed-length encoding for integer values (4 bytes, 171 | little endian, two's complement). With space concerns in mind, tinySSB 172 | formats integers as little endian, two's complement, and retaining only 173 | the minimum number of bytes needed. 174 | 175 | (b) As pointed out in the Motivation section, a straight-forward 176 | human-readable representation of BIPF exists that is very close to 177 | JSON. The only syntactic extension needed is a delimiter for byte 178 | arrays where we chose the '#' hash sign as used in S-expressions 179 | (which is another serialization format from 1997). 180 | 181 | (c) We provide the following test vectors where for each human-readable 182 | BIPF-encoding we give its binary representation. 183 | 184 | ``` 185 | data item as human- its BIPF serialization 186 | readable BIPF as byte array 187 | ------------------- ------------------------ 188 | 189 | null 06 190 | 191 | false 0e00 192 | 193 | true 0e01 194 | 195 | 123 0a7b 196 | 197 | -123 0a85 198 | 199 | "¥€$!" 39c2a5e282ac2421 200 | 201 | #ABCD# 11abcd 202 | 203 | [123,true] 240a7b0e01 204 | 205 | {123:false} 250a7b0e00 206 | 207 | {#ABCD#:[123,null]} 3d11abcd1c0a7b06 208 | ``` 209 | 210 | 211 | ## References 212 | 213 | ### Normative 214 | 215 | unsigned LEB128 (varint): [```https://en.wikipedia.org/wiki/LEB128```](https://en.wikipedia.org/wiki/LEB128), 216 | 217 | ### Informative 218 | 219 | BASE64: [```https://www.rfc-editor.org/rfc/rfc4648```](https://www.rfc-editor.org/rfc/rfc4648) 220 | 221 | BIPF.original: [```https://github.com/ssbc/bipf-spec```](https://github.com/ssbc/bipf-spec) 222 | 223 | BIPF.rationale: [```https://github.com/ssbc/bipf```](https://github.com/ssbc/bipf) 224 | 225 | BNF: Section 2 of [```https://www.rfc-editor.org/rfc/rfc822```](https://www.rfc-editor.org/rfc/rfc822) 226 | 227 | CBOR: [```https://www.rfc-editor.org/rfc/rfc8949```](https://www.rfc-editor.org/rfc/rfc8949```) 228 | 229 | Comparison: [```https://en.wikipedia.org/wiki/Comparison_of_data-serialization_formats```](https://en.wikipedia.org/wiki/Comparison_of_data-serialization_formats) 230 | 231 | JSON: [```https://www.rfc-editor.org/rfc/rfc8259```](https://www.rfc-editor.org/rfc/rfc8259) 232 | 233 | S-exp: [https://web.archive.org/web/20230223024606/http://people.csail.mit.edu/rivest/Sexp.txt](https://web.archive.org/web/20230223024606/http://people.csail.mit.edu/rivest/Sexp.txt) 234 | 235 | SSB.classic: [```https://ssbc.github.io/scuttlebutt-protocol-guide/```](https://ssbc.github.io/scuttlebutt-protocol-guide/) 236 | 237 | UTF8: [```https://www.rfc-editor.org/rfc/rfc3629```](https://www.rfc-editor.org/rfc/rfc3629) 238 | 239 | varint in ProtoBuf: [```https://protobuf.dev/programming-guides/encoding/#varints```](https://protobuf.dev/programming-guides/encoding/#varints) 240 | 241 | 242 | ### Implementations 243 | 244 | BIPF.tinySSB: 245 | - Python: https://pypi.org/project/bipf/ 246 | 247 | BIPF.classic: 248 | - Go: https://git.sr.ht/~cryptix/go-exp/tree/bipf/item/bipf 249 | - JS: https://github.com/ssbc/bipf/ 250 | - Nim: https://github.com/BundleFeed/nim_bipf (npm package with JS compiled version: https://www.npmjs.com/package/nim_bipf) 251 | - Rust: https://github.com/jerive/bipf-rs 252 | -------------------------------------------------------------------------------- /CONTRIBUTING.md: -------------------------------------------------------------------------------- 1 | # Contributing a new SIP 2 | 3 | There are generally two types of SIPs that can be submitted: 4 | 5 | - Specifications of existing SSB functionality 6 | - Proposals for new SSB functionality 7 | 8 | Specs for existing SSB functionality are generally straightforward to accept, and we highly encourage and appreciate you for doing so! 9 | 10 | For new proposals, the review process may be more involved. This is because we want to ensure that your proposal is (1) related to SSB, (2) has a reference implementation, (3) makes sense and could improve SSB. 11 | 12 | For an easy process, please follow these steps: 13 | 14 | ## 1. Fork this repo 15 | 16 | ## 2. Copy the file `TEMPLATE.md` into `000.md` 17 | 18 | ## 3. Edit `000.md` 19 | 20 | We recommend using TEMPLATE.md as a starting point, such that you can just fill 21 | in the blanks. Not all sections are mandatory. 22 | 23 | - Title, author, date, license: **mandatory** 24 | - Abstract: **mandatory** 25 | - Motivation: optional 26 | - Terminology: optional 27 | - Specification: **mandatory** 28 | - Considerations: optional 29 | - Normative references: optional 30 | - Informative references: optional 31 | - Reference implementation: **mandatory** 32 | 33 | Here are what those sections mean: 34 | 35 | **Title:** The title of your SIP. Typically a short description in maximum 8 words. Preferably should NOT contain the word "SSB" nor the word "specification". 36 | 37 | **License:** The license for your SIP **MUST** be [CC0-1.0](https://creativecommons.org/publicdomain/zero/1.0/), in other words, "public domain". We want the "knowledge" of SSB to be a public asset. 38 | 39 | **Abstract:** A single paragraph that describes the problem your SIP addresses and a summary of the solution. Use at most 250 words. 40 | 41 | **Motivation:** A few paragraphs that give the context for this SIP. Why is this important? What is the problem? Use at most 1000 words. 42 | 43 | **Terminology:** A short section defining the terminology used in your SIP. We recommend including the below in most cases: 44 | 45 | > The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC 2119](https://tools.ietf.org/html/rfc2119). 46 | 47 | **Specification:** A detailed description of the solution. You can use diagrams and examples to help explain your proposal, but prefer to use "SHOULD"/"MUST" textual descriptions as the official source of truth for the specification. There is no maximum length for this section, but we encourage to be brief and simple. After all, you want people to implement your proposal. This section is important because it is the ONLY part of your SIP that is normative. The rest of the document is informative. 48 | 49 | **Considerations:** A section that discusses the tradeoffs of your proposal. We recommend addressing *Security Considerations*, *Privacy Considerations*, and/or *Compatibility Considerations* (describes how your proposal is compatible or incompatible with other SIPS). 50 | 51 | **Normative references:** A list of SIPs and RFCs that your SIP depends on. 52 | 53 | **Informative references:** A list of other links that are relevant to your SIP. 54 | 55 | **Reference implementation:** A link to a reference implementation of your proposal. The reference implementation is not "normative", because the specification section should be sufficient. But it is very helpful for both reviewers and implementers. 56 | 57 | **Extras:** for more informative sections, you can "Appendix A.", "Appendix B.", etc to the end of your SIP. You can also add files (source code or test vectors) to a folder called `000` which will then be renamed with the appropriate number when your SIP is merged. But please keep the size of the folder small, and prefer to use text to describe your proposal. 58 | 59 | ## 4. Submit a pull request 60 | 61 | Open a pull request targetting this repo, and ask for review from other stakeholders such as implementers in the SSB community or other SIP authors. 62 | 63 | ## 5. Review process 64 | 65 | The review should be done according to the [REVIEWING.md](REVIEWING.md). 66 | 67 | Once the reviewer merges your PR, the file `000.md` will be renamed to an appropriate new number, and your SIP will be listed on the README.md table. 68 | 69 | Congratulations! 70 | -------------------------------------------------------------------------------- /LICENSES/CC-BY-4.0.txt: -------------------------------------------------------------------------------- 1 | Creative Commons Attribution 4.0 International 2 | 3 | Creative Commons Corporation (“Creative Commons”) is not a law firm and does not provide legal services or legal advice. Distribution of Creative Commons public licenses does not create a lawyer-client or other relationship. Creative Commons makes its licenses and related information available on an “as-is” basis. Creative Commons gives no warranties regarding its licenses, any material licensed under their terms and conditions, or any related information. Creative Commons disclaims all liability for damages resulting from their use to the fullest extent possible. 4 | 5 | Using Creative Commons Public Licenses 6 | 7 | Creative Commons public licenses provide a standard set of terms and conditions that creators and other rights holders may use to share original works of authorship and other material subject to copyright and certain other rights specified in the public license below. The following considerations are for informational purposes only, are not exhaustive, and do not form part of our licenses. 8 | 9 | Considerations for licensors: Our public licenses are intended for use by those authorized to give the public permission to use material in ways otherwise restricted by copyright and certain other rights. Our licenses are irrevocable. Licensors should read and understand the terms and conditions of the license they choose before applying it. Licensors should also secure all rights necessary before applying our licenses so that the public can reuse the material as expected. Licensors should clearly mark any material not subject to the license. This includes other CC-licensed material, or material used under an exception or limitation to copyright. More considerations for licensors. 10 | 11 | Considerations for the public: By using one of our public licenses, a licensor grants the public permission to use the licensed material under specified terms and conditions. If the licensor’s permission is not necessary for any reason–for example, because of any applicable exception or limitation to copyright–then that use is not regulated by the license. Our licenses grant only permissions under copyright and certain other rights that a licensor has authority to grant. Use of the licensed material may still be restricted for other reasons, including because others have copyright or other rights in the material. A licensor may make special requests, such as asking that all changes be marked or described. Although not required by our licenses, you are encouraged to respect those requests where reasonable. More considerations for the public. 12 | 13 | Creative Commons Attribution 4.0 International Public License 14 | 15 | By exercising the Licensed Rights (defined below), You accept and agree to be bound by the terms and conditions of this Creative Commons Attribution 4.0 International Public License ("Public License"). To the extent this Public License may be interpreted as a contract, You are granted the Licensed Rights in consideration of Your acceptance of these terms and conditions, and the Licensor grants You such rights in consideration of benefits the Licensor receives from making the Licensed Material available under these terms and conditions. 16 | 17 | Section 1 – Definitions. 18 | 19 | a. Adapted Material means material subject to Copyright and Similar Rights that is derived from or based upon the Licensed Material and in which the Licensed Material is translated, altered, arranged, transformed, or otherwise modified in a manner requiring permission under the Copyright and Similar Rights held by the Licensor. For purposes of this Public License, where the Licensed Material is a musical work, performance, or sound recording, Adapted Material is always produced where the Licensed Material is synched in timed relation with a moving image. 20 | 21 | b. Adapter's License means the license You apply to Your Copyright and Similar Rights in Your contributions to Adapted Material in accordance with the terms and conditions of this Public License. 22 | 23 | c. Copyright and Similar Rights means copyright and/or similar rights closely related to copyright including, without limitation, performance, broadcast, sound recording, and Sui Generis Database Rights, without regard to how the rights are labeled or categorized. For purposes of this Public License, the rights specified in Section 2(b)(1)-(2) are not Copyright and Similar Rights. 24 | 25 | d. Effective Technological Measures means those measures that, in the absence of proper authority, may not be circumvented under laws fulfilling obligations under Article 11 of the WIPO Copyright Treaty adopted on December 20, 1996, and/or similar international agreements. 26 | 27 | e. Exceptions and Limitations means fair use, fair dealing, and/or any other exception or limitation to Copyright and Similar Rights that applies to Your use of the Licensed Material. 28 | 29 | f. Licensed Material means the artistic or literary work, database, or other material to which the Licensor applied this Public License. 30 | 31 | g. Licensed Rights means the rights granted to You subject to the terms and conditions of this Public License, which are limited to all Copyright and Similar Rights that apply to Your use of the Licensed Material and that the Licensor has authority to license. 32 | 33 | h. Licensor means the individual(s) or entity(ies) granting rights under this Public License. 34 | 35 | i. Share means to provide material to the public by any means or process that requires permission under the Licensed Rights, such as reproduction, public display, public performance, distribution, dissemination, communication, or importation, and to make material available to the public including in ways that members of the public may access the material from a place and at a time individually chosen by them. 36 | 37 | j. Sui Generis Database Rights means rights other than copyright resulting from Directive 96/9/EC of the European Parliament and of the Council of 11 March 1996 on the legal protection of databases, as amended and/or succeeded, as well as other essentially equivalent rights anywhere in the world. 38 | 39 | k. You means the individual or entity exercising the Licensed Rights under this Public License. Your has a corresponding meaning. 40 | 41 | Section 2 – Scope. 42 | 43 | a. License grant. 44 | 45 | 1. Subject to the terms and conditions of this Public License, the Licensor hereby grants You a worldwide, royalty-free, non-sublicensable, non-exclusive, irrevocable license to exercise the Licensed Rights in the Licensed Material to: 46 | 47 | A. reproduce and Share the Licensed Material, in whole or in part; and 48 | 49 | B. produce, reproduce, and Share Adapted Material. 50 | 51 | 2. Exceptions and Limitations. For the avoidance of doubt, where Exceptions and Limitations apply to Your use, this Public License does not apply, and You do not need to comply with its terms and conditions. 52 | 53 | 3. Term. The term of this Public License is specified in Section 6(a). 54 | 55 | 4. Media and formats; technical modifications allowed. The Licensor authorizes You to exercise the Licensed Rights in all media and formats whether now known or hereafter created, and to make technical modifications necessary to do so. The Licensor waives and/or agrees not to assert any right or authority to forbid You from making technical modifications necessary to exercise the Licensed Rights, including technical modifications necessary to circumvent Effective Technological Measures. For purposes of this Public License, simply making modifications authorized by this Section 2(a)(4) never produces Adapted Material. 56 | 57 | 5. Downstream recipients. 58 | 59 | A. Offer from the Licensor – Licensed Material. Every recipient of the Licensed Material automatically receives an offer from the Licensor to exercise the Licensed Rights under the terms and conditions of this Public License. 60 | 61 | B. No downstream restrictions. You may not offer or impose any additional or different terms or conditions on, or apply any Effective Technological Measures to, the Licensed Material if doing so restricts exercise of the Licensed Rights by any recipient of the Licensed Material. 62 | 63 | 6. No endorsement. Nothing in this Public License constitutes or may be construed as permission to assert or imply that You are, or that Your use of the Licensed Material is, connected with, or sponsored, endorsed, or granted official status by, the Licensor or others designated to receive attribution as provided in Section 3(a)(1)(A)(i). 64 | 65 | b. Other rights. 66 | 67 | 1. Moral rights, such as the right of integrity, are not licensed under this Public License, nor are publicity, privacy, and/or other similar personality rights; however, to the extent possible, the Licensor waives and/or agrees not to assert any such rights held by the Licensor to the limited extent necessary to allow You to exercise the Licensed Rights, but not otherwise. 68 | 69 | 2. Patent and trademark rights are not licensed under this Public License. 70 | 71 | 3. To the extent possible, the Licensor waives any right to collect royalties from You for the exercise of the Licensed Rights, whether directly or through a collecting society under any voluntary or waivable statutory or compulsory licensing scheme. In all other cases the Licensor expressly reserves any right to collect such royalties. 72 | 73 | Section 3 – License Conditions. 74 | 75 | Your exercise of the Licensed Rights is expressly made subject to the following conditions. 76 | 77 | a. Attribution. 78 | 79 | 1. If You Share the Licensed Material (including in modified form), You must: 80 | 81 | A. retain the following if it is supplied by the Licensor with the Licensed Material: 82 | 83 | i. identification of the creator(s) of the Licensed Material and any others designated to receive attribution, in any reasonable manner requested by the Licensor (including by pseudonym if designated); 84 | 85 | ii. a copyright notice; 86 | 87 | iii. a notice that refers to this Public License; 88 | 89 | iv. a notice that refers to the disclaimer of warranties; 90 | 91 | v. a URI or hyperlink to the Licensed Material to the extent reasonably practicable; 92 | 93 | B. indicate if You modified the Licensed Material and retain an indication of any previous modifications; and 94 | 95 | C. indicate the Licensed Material is licensed under this Public License, and include the text of, or the URI or hyperlink to, this Public License. 96 | 97 | 2. You may satisfy the conditions in Section 3(a)(1) in any reasonable manner based on the medium, means, and context in which You Share the Licensed Material. For example, it may be reasonable to satisfy the conditions by providing a URI or hyperlink to a resource that includes the required information. 98 | 99 | 3. If requested by the Licensor, You must remove any of the information required by Section 3(a)(1)(A) to the extent reasonably practicable. 100 | 101 | 4. If You Share Adapted Material You produce, the Adapter's License You apply must not prevent recipients of the Adapted Material from complying with this Public License. 102 | 103 | Section 4 – Sui Generis Database Rights. 104 | 105 | Where the Licensed Rights include Sui Generis Database Rights that apply to Your use of the Licensed Material: 106 | 107 | a. for the avoidance of doubt, Section 2(a)(1) grants You the right to extract, reuse, reproduce, and Share all or a substantial portion of the contents of the database; 108 | 109 | b. if You include all or a substantial portion of the database contents in a database in which You have Sui Generis Database Rights, then the database in which You have Sui Generis Database Rights (but not its individual contents) is Adapted Material; and 110 | 111 | c. You must comply with the conditions in Section 3(a) if You Share all or a substantial portion of the contents of the database. 112 | For the avoidance of doubt, this Section 4 supplements and does not replace Your obligations under this Public License where the Licensed Rights include other Copyright and Similar Rights. 113 | 114 | Section 5 – Disclaimer of Warranties and Limitation of Liability. 115 | 116 | a. Unless otherwise separately undertaken by the Licensor, to the extent possible, the Licensor offers the Licensed Material as-is and as-available, and makes no representations or warranties of any kind concerning the Licensed Material, whether express, implied, statutory, or other. This includes, without limitation, warranties of title, merchantability, fitness for a particular purpose, non-infringement, absence of latent or other defects, accuracy, or the presence or absence of errors, whether or not known or discoverable. Where disclaimers of warranties are not allowed in full or in part, this disclaimer may not apply to You. 117 | 118 | b. To the extent possible, in no event will the Licensor be liable to You on any legal theory (including, without limitation, negligence) or otherwise for any direct, special, indirect, incidental, consequential, punitive, exemplary, or other losses, costs, expenses, or damages arising out of this Public License or use of the Licensed Material, even if the Licensor has been advised of the possibility of such losses, costs, expenses, or damages. Where a limitation of liability is not allowed in full or in part, this limitation may not apply to You. 119 | 120 | c. The disclaimer of warranties and limitation of liability provided above shall be interpreted in a manner that, to the extent possible, most closely approximates an absolute disclaimer and waiver of all liability. 121 | 122 | Section 6 – Term and Termination. 123 | 124 | a. This Public License applies for the term of the Copyright and Similar Rights licensed here. However, if You fail to comply with this Public License, then Your rights under this Public License terminate automatically. 125 | 126 | b. Where Your right to use the Licensed Material has terminated under Section 6(a), it reinstates: 127 | 128 | 1. automatically as of the date the violation is cured, provided it is cured within 30 days of Your discovery of the violation; or 129 | 130 | 2. upon express reinstatement by the Licensor. 131 | 132 | c. For the avoidance of doubt, this Section 6(b) does not affect any right the Licensor may have to seek remedies for Your violations of this Public License. 133 | 134 | d. For the avoidance of doubt, the Licensor may also offer the Licensed Material under separate terms or conditions or stop distributing the Licensed Material at any time; however, doing so will not terminate this Public License. 135 | 136 | e. Sections 1, 5, 6, 7, and 8 survive termination of this Public License. 137 | 138 | Section 7 – Other Terms and Conditions. 139 | 140 | a. The Licensor shall not be bound by any additional or different terms or conditions communicated by You unless expressly agreed. 141 | 142 | b. Any arrangements, understandings, or agreements regarding the Licensed Material not stated herein are separate from and independent of the terms and conditions of this Public License. 143 | 144 | Section 8 – Interpretation. 145 | 146 | a. For the avoidance of doubt, this Public License does not, and shall not be interpreted to, reduce, limit, restrict, or impose conditions on any use of the Licensed Material that could lawfully be made without permission under this Public License. 147 | 148 | b. To the extent possible, if any provision of this Public License is deemed unenforceable, it shall be automatically reformed to the minimum extent necessary to make it enforceable. If the provision cannot be reformed, it shall be severed from this Public License without affecting the enforceability of the remaining terms and conditions. 149 | 150 | c. No term or condition of this Public License will be waived and no failure to comply consented to unless expressly agreed to by the Licensor. 151 | 152 | d. Nothing in this Public License constitutes or may be interpreted as a limitation upon, or waiver of, any privileges and immunities that apply to the Licensor or You, including from the legal processes of any jurisdiction or authority. 153 | 154 | Creative Commons is not a party to its public licenses. Notwithstanding, Creative Commons may elect to apply one of its public licenses to material it publishes and in those instances will be considered the “Licensor.” Except for the limited purpose of indicating that material is shared under a Creative Commons public license or as otherwise permitted by the Creative Commons policies published at creativecommons.org/policies, Creative Commons does not authorize the use of the trademark “Creative Commons” or any other trademark or logo of Creative Commons without its prior written consent including, without limitation, in connection with any unauthorized modifications to any of its public licenses or any other arrangements, understandings, or agreements concerning use of licensed material. For the avoidance of doubt, this paragraph does not form part of the public licenses. 155 | 156 | Creative Commons may be contacted at creativecommons.org. 157 | -------------------------------------------------------------------------------- /LICENSES/CC0-1.0.txt: -------------------------------------------------------------------------------- 1 | Creative Commons Legal Code 2 | 3 | CC0 1.0 Universal 4 | 5 | CREATIVE COMMONS CORPORATION IS NOT A LAW FIRM AND DOES NOT PROVIDE 6 | LEGAL SERVICES. DISTRIBUTION OF THIS DOCUMENT DOES NOT CREATE AN 7 | ATTORNEY-CLIENT RELATIONSHIP. CREATIVE COMMONS PROVIDES THIS 8 | INFORMATION ON AN "AS-IS" BASIS. CREATIVE COMMONS MAKES NO WARRANTIES 9 | REGARDING THE USE OF THIS DOCUMENT OR THE INFORMATION OR WORKS 10 | PROVIDED HEREUNDER, AND DISCLAIMS LIABILITY FOR DAMAGES RESULTING FROM 11 | THE USE OF THIS DOCUMENT OR THE INFORMATION OR WORKS PROVIDED 12 | HEREUNDER. 13 | 14 | Statement of Purpose 15 | 16 | The laws of most jurisdictions throughout the world automatically confer 17 | exclusive Copyright and Related Rights (defined below) upon the creator 18 | and subsequent owner(s) (each and all, an "owner") of an original work of 19 | authorship and/or a database (each, a "Work"). 20 | 21 | Certain owners wish to permanently relinquish those rights to a Work for 22 | the purpose of contributing to a commons of creative, cultural and 23 | scientific works ("Commons") that the public can reliably and without fear 24 | of later claims of infringement build upon, modify, incorporate in other 25 | works, reuse and redistribute as freely as possible in any form whatsoever 26 | and for any purposes, including without limitation commercial purposes. 27 | These owners may contribute to the Commons to promote the ideal of a free 28 | culture and the further production of creative, cultural and scientific 29 | works, or to gain reputation or greater distribution for their Work in 30 | part through the use and efforts of others. 31 | 32 | For these and/or other purposes and motivations, and without any 33 | expectation of additional consideration or compensation, the person 34 | associating CC0 with a Work (the "Affirmer"), to the extent that he or she 35 | is an owner of Copyright and Related Rights in the Work, voluntarily 36 | elects to apply CC0 to the Work and publicly distribute the Work under its 37 | terms, with knowledge of his or her Copyright and Related Rights in the 38 | Work and the meaning and intended legal effect of CC0 on those rights. 39 | 40 | 1. Copyright and Related Rights. A Work made available under CC0 may be 41 | protected by copyright and related or neighboring rights ("Copyright and 42 | Related Rights"). Copyright and Related Rights include, but are not 43 | limited to, the following: 44 | 45 | i. the right to reproduce, adapt, distribute, perform, display, 46 | communicate, and translate a Work; 47 | ii. moral rights retained by the original author(s) and/or performer(s); 48 | iii. publicity and privacy rights pertaining to a person's image or 49 | likeness depicted in a Work; 50 | iv. rights protecting against unfair competition in regards to a Work, 51 | subject to the limitations in paragraph 4(a), below; 52 | v. rights protecting the extraction, dissemination, use and reuse of data 53 | in a Work; 54 | vi. database rights (such as those arising under Directive 96/9/EC of the 55 | European Parliament and of the Council of 11 March 1996 on the legal 56 | protection of databases, and under any national implementation 57 | thereof, including any amended or successor version of such 58 | directive); and 59 | vii. other similar, equivalent or corresponding rights throughout the 60 | world based on applicable law or treaty, and any national 61 | implementations thereof. 62 | 63 | 2. Waiver. To the greatest extent permitted by, but not in contravention 64 | of, applicable law, Affirmer hereby overtly, fully, permanently, 65 | irrevocably and unconditionally waives, abandons, and surrenders all of 66 | Affirmer's Copyright and Related Rights and associated claims and causes 67 | of action, whether now known or unknown (including existing as well as 68 | future claims and causes of action), in the Work (i) in all territories 69 | worldwide, (ii) for the maximum duration provided by applicable law or 70 | treaty (including future time extensions), (iii) in any current or future 71 | medium and for any number of copies, and (iv) for any purpose whatsoever, 72 | including without limitation commercial, advertising or promotional 73 | purposes (the "Waiver"). Affirmer makes the Waiver for the benefit of each 74 | member of the public at large and to the detriment of Affirmer's heirs and 75 | successors, fully intending that such Waiver shall not be subject to 76 | revocation, rescission, cancellation, termination, or any other legal or 77 | equitable action to disrupt the quiet enjoyment of the Work by the public 78 | as contemplated by Affirmer's express Statement of Purpose. 79 | 80 | 3. Public License Fallback. Should any part of the Waiver for any reason 81 | be judged legally invalid or ineffective under applicable law, then the 82 | Waiver shall be preserved to the maximum extent permitted taking into 83 | account Affirmer's express Statement of Purpose. In addition, to the 84 | extent the Waiver is so judged Affirmer hereby grants to each affected 85 | person a royalty-free, non transferable, non sublicensable, non exclusive, 86 | irrevocable and unconditional license to exercise Affirmer's Copyright and 87 | Related Rights in the Work (i) in all territories worldwide, (ii) for the 88 | maximum duration provided by applicable law or treaty (including future 89 | time extensions), (iii) in any current or future medium and for any number 90 | of copies, and (iv) for any purpose whatsoever, including without 91 | limitation commercial, advertising or promotional purposes (the 92 | "License"). The License shall be deemed effective as of the date CC0 was 93 | applied by Affirmer to the Work. Should any part of the License for any 94 | reason be judged legally invalid or ineffective under applicable law, such 95 | partial invalidity or ineffectiveness shall not invalidate the remainder 96 | of the License, and in such case Affirmer hereby affirms that he or she 97 | will not (i) exercise any of his or her remaining Copyright and Related 98 | Rights in the Work or (ii) assert any associated claims and causes of 99 | action with respect to the Work, in either case contrary to Affirmer's 100 | express Statement of Purpose. 101 | 102 | 4. Limitations and Disclaimers. 103 | 104 | a. No trademark or patent rights held by Affirmer are waived, abandoned, 105 | surrendered, licensed or otherwise affected by this document. 106 | b. Affirmer offers the Work as-is and makes no representations or 107 | warranties of any kind concerning the Work, express, implied, 108 | statutory or otherwise, including without limitation warranties of 109 | title, merchantability, fitness for a particular purpose, non 110 | infringement, or the absence of latent or other defects, accuracy, or 111 | the present or absence of errors, whether or not discoverable, all to 112 | the greatest extent permissible under applicable law. 113 | c. Affirmer disclaims responsibility for clearing rights of other persons 114 | that may apply to the Work or any use thereof, including without 115 | limitation any person's Copyright and Related Rights in the Work. 116 | Further, Affirmer disclaims responsibility for obtaining any necessary 117 | consents, permissions or other rights required for any use of the 118 | Work. 119 | d. Affirmer understands and acknowledges that Creative Commons is not a 120 | party to this document and has no duty or obligation with respect to 121 | this CC0 or use of the Work. 122 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | ## Secure Scuttlebutt Implementation Protocols (SIPs) 2 | 3 | This repository contains the specifications for the Secure Scuttlebutt (SSB) protocol. 4 | 5 | ## List 6 | 7 | | SIP | Title | Date | 8 | |-----|-------|-------| 9 | | 1 | [SSB URIs](001.md) | 2022-10-01 | 10 | | 2 | [Metafeeds](002.md) | 2021-10-11 | 11 | | 3 | [Index Feeds](003.md) | 2021-10-11 | 12 | | 4 | [Bendy Butt](004.md) | 2021-06-15 | 13 | | 5 | [HTTP Invites](005.md) | 2021-04-26 | 14 | | 6 | [HTTP Authentication](006.md) | 2021-04-26 | 15 | | 7 | [Rooms 2](007.md) | 2022-10-10 | 16 | | 8 | [Binary Field Encodings](008.md) | 2022-10-02 | 17 | | 9 | [Tangles](009.md) | 2022-03-23 | 18 | | 10 | [Threads](010.md) | 2023-03-27 | 19 | | 11 | [BIPF for tinySSB](011.md) | 2023-07-16 | 20 | 21 | This list is intentionally **not** categorizing SIPs by maturity, category, or status. We don't want to make judgement calls here, because that could cause centralization of decision power. Instead, look for curations and categorizations of SIPs among the SSB community. 22 | 23 | ## Maintenance 24 | 25 | - When adding a new SIP, please read the [Contributing guide](CONTRIBUTING.md) 26 | - When reviewing a SIP, please read the [Reviewing guide](REVIEWING.md) 27 | - When updating a SIP, please read the [Updating guide](UPDATING.md) 28 | -------------------------------------------------------------------------------- /REVIEWING.md: -------------------------------------------------------------------------------- 1 | # Reviewing a new SIP 2 | 3 | Thank you for volunteering to read and review a new SIP! This is a very important part of the process. Here are a couple criteria you should keep in mind when reviewing pull requests. 4 | 5 | ## 1. Is it related to SSB? 6 | 7 | Generally speaking, SIPs can be "anything" SSB related. Avoid judging whether a SIP is "good" or "bad" based on whether you think it's a good idea or not. Instead, focus on whether it's related to SSB, and whether the SIPs Specification section *makes sense* considering its Motivation section. 8 | 9 | Typically, all SIPs should have other SIPs as normative references. We don't want to accept SIPs that are out of scope, and this can be either low-level tools that are generic and apply to other systems, or too high-level such as protocols internal to a certain application. 10 | 11 | ## 2. Is it well written? 12 | 13 | - Does it have all the "required" sections (see [CONTRIBUTING.md](CONTRIBUTING.md))? 14 | - Is the abstract short and descriptive? 15 | - Is the specification section *clear and concise*, i.e. long enough to prescribe how implementations should behave? 16 | - Is the specification section *coherent*, i.e. internally consistent with itself? 17 | - Is the specification section *complete*, i.e. are all implementation corner cases taken into account? 18 | - Does it list a reference implementation? 19 | 20 | If you can say yes to all these questions, then the SIP is well written! 21 | 22 | ## 3. Is it implementable? 23 | 24 | - Is it possible to implement the SIP in a reasonable amount of time with current programming paradigms? 25 | - Is the specification useful enough to understand how an implementation should work? 26 | - Does the reference implementation match the specification? 27 | 28 | If you can say yes to all these questions, then the SIP is implementable! 29 | 30 | ## Approved 31 | 32 | Try not to make the review process too heavy! We want to encourage people to submit SIPs, and we want to make it easy for them to do so. 33 | 34 | ## Reserve a SIP number 35 | 36 | Carefully check what is the latest SIP number, and reserve the next number for the SIP you are reviewing. Ask the author to: 37 | 38 | - Rename the file to the reserved number 39 | - Add a new entry to the table on README.md 40 | 41 | ## Merge the pull request 42 | 43 | That's it! 44 | -------------------------------------------------------------------------------- /TEMPLATE.md: -------------------------------------------------------------------------------- 1 | # (Title) 2 | 3 | Author: (Author) 4 | 5 | Date: (Date) 6 | 7 | License: CC0-1.0 8 | 9 | ## Abstract 10 | 11 | 12 | ## Motivation 13 | 14 | 15 | ## Terminology 16 | 17 | 18 | ## Specification 19 | 20 | 21 | ## Considerations 22 | 23 | 24 | ## References 25 | 26 | ### Normative 27 | 28 | (List SIPs and RFCs that this document depends on) 29 | 30 | ### Informative 31 | 32 | (List any other links here) 33 | 34 | ### Implementation 35 | 36 | (Put a link to your reference implementation) 37 | -------------------------------------------------------------------------------- /UPDATING.md: -------------------------------------------------------------------------------- 1 | # Updating a SIP 2 | 3 | There are generally two types of "updates" to a SIP. 4 | 5 | ## 1. Minor tweaks 6 | 7 | These include changes such as: 8 | 9 | - Fixing typos and grammar 10 | - Adding more non-normative examples or diagrams 11 | - Adding more informative references 12 | - Improving the abstract or motivation sections 13 | - Adding a security consideration 14 | - Update contact information 15 | 16 | For those cases, just submit a pull request that changes the SIP document. 17 | 18 | ## 2. Breaking changes 19 | 20 | These include changes such as: 21 | 22 | - Changing the specification such that implementations will have to change 23 | - Changing the normative references 24 | - Changing the abstract or motivation sections in a way that changes the scope and applicability of the SIP 25 | 26 | For these kinds of changes **do not change the SIP document**. Instead, submit a new SIP proposal that supersedes the old one. The new SIP should reference the old one, and explain why the new one should be preferred over the old one. It is okay to copy-paste the old SIP into the new one, and then make the changes. 27 | 28 | The reason for this is that we want to allow SIPs to be immutable, such that implementers can say that they implement a certain SIP, and not have to have versioning. 29 | --------------------------------------------------------------------------------