├── .gitignore ├── gossip.md └── README.md /.gitignore: -------------------------------------------------------------------------------- 1 | ### https://raw.github.com/github/gitignore/7751c25c6662ce6f9dc50f014e37156298ccf065/Global/SublimeText.gitignore 2 | 3 | # cache files for sublime text 4 | *.tmlanguage.cache 5 | *.tmPreferences.cache 6 | *.stTheme.cache 7 | 8 | # workspace files are user-specific 9 | *.sublime-workspace 10 | 11 | # project files should be checked into the repository, unless a significant 12 | # proportion of contributors will probably not be using SublimeText 13 | # *.sublime-project 14 | 15 | # sftp configuration file 16 | sftp-config.json 17 | 18 | # Package control specific files 19 | Package Control.last-run 20 | Package Control.ca-list 21 | Package Control.ca-bundle 22 | Package Control.system-ca-bundle 23 | Package Control.cache/ 24 | Package Control.ca-certs/ 25 | bh_unicode_properties.cache 26 | 27 | # Sublime-github package stores a github token in this file 28 | # https://packagecontrol.io/packages/sublime-github 29 | GitHub.sublime-settings 30 | 31 | 32 | ### https://raw.github.com/github/gitignore/7751c25c6662ce6f9dc50f014e37156298ccf065/Global/OSX.gitignore 33 | 34 | *.DS_Store 35 | .AppleDouble 36 | .LSOverride 37 | 38 | # Icon must end with two \r 39 | Icon 40 | 41 | # Thumbnails 42 | ._* 43 | 44 | # Files that might appear in the root of a volume 45 | .DocumentRevisions-V100 46 | .fseventsd 47 | .Spotlight-V100 48 | .TemporaryItems 49 | .Trashes 50 | .VolumeIcon.icns 51 | .com.apple.timemachine.donotpresent 52 | 53 | # Directories potentially created on remote AFP share 54 | .AppleDB 55 | .AppleDesktop 56 | Network Trash Folder 57 | Temporary Items 58 | .apdisk 59 | 60 | 61 | ### https://raw.github.com/github/gitignore/7751c25c6662ce6f9dc50f014e37156298ccf065/Node.gitignore 62 | 63 | # Logs 64 | logs 65 | *.log 66 | npm-debug.log* 67 | 68 | # Runtime data 69 | pids 70 | *.pid 71 | *.seed 72 | 73 | # Directory for instrumented libs generated by jscoverage/JSCover 74 | lib-cov 75 | 76 | # Coverage directory used by tools like istanbul 77 | coverage 78 | 79 | # nyc test coverage 80 | .nyc_output 81 | 82 | # Grunt intermediate storage (http://gruntjs.com/creating-plugins#storing-task-files) 83 | .grunt 84 | 85 | # node-waf configuration 86 | .lock-wscript 87 | 88 | # Compiled binary addons (http://nodejs.org/api/addons.html) 89 | build/Release 90 | 91 | # Dependency directories 92 | node_modules 93 | jspm_packages 94 | 95 | # Optional npm cache directory 96 | .npm 97 | 98 | # Optional REPL history 99 | .node_repl_history 100 | 101 | 102 | ### https://raw.github.com/github/gitignore/7751c25c6662ce6f9dc50f014e37156298ccf065/Sass.gitignore 103 | 104 | .sass-cache/ 105 | *.css.map 106 | 107 | 108 | bundle.js 109 | -------------------------------------------------------------------------------- /gossip.md: -------------------------------------------------------------------------------- 1 | # gossip 2 | 3 | [This was written by dominic](https://viewer.scuttlebot.io/%25dRIUnLTQ4t8fBVwcPlH%2FCGbyCd0e1Da7wjCsc7PQWpg%3D.sha256) 4 | 5 | ## Epidemic Broadcast Trees (explain i like i'm 5) 6 | I want to broadcast a message to my friends around the world and I have a couple of requirements 7 | 8 | * make sure everyone definitely gets it 9 | * make sure everyone gets it soon 10 | * send as few messages as possible 11 | * it should work for everyone 12 | 13 | How might I approach this? 14 | 15 | ## send it to everyone 16 | the simplest way would be to make a connection to each friend and send it to them directly. If I have 100 friends, that is 100 messages - that is actually good! but the problem is if everyone wants to distribute messages this way everyone needs 100 connections! to connect everyone to everyone means thousands of connections! This is a lot of work for each computer. 17 | 18 | ## send it to a few people who send it on 19 | lets say, everyone has just 5 connections to some other random friends, so you send a message to 5 people, they each send it to another 5 (thats 1+25 now) who then send it to another 5 (that is 1+25+125) now everyone has the message and some people have probarly received (since 151 messages went to 100 people) that isn't efficient. but it was less work for my computer! I only needed to send it to 5 people! I'm okay with that! 20 | 21 | ## send it to few people and send it on 22 | okay so lets try the above, can we avoid sending it to anyone _again_? We could do that if we had a thing called a "spanning tree" a spanning tree is a network that connects everything but doesn't have any extra connections. That means there is no loops - there is always a shortest path between any two friends (that may go through other closer friends). 23 | 24 | If you think of a ordinary tree it's like this - because all the branches come off the trunk, and then the leaves come off those branches. If you where an ant and wanted to travel from one leaf to another, you'd start walking towards the trunk, then go onto the branch the other leaf came off etc. 25 | 26 | An ordinary tree grows in this shape, so it's easy to see what the paths are, but the problem we have is all the leaves are in a jumble and we don't actually know which ones are close to each other! 27 | 28 | ## centrally organize into a tree 29 | if we had some sorts of top down view of the network, we could look and see which people where closest to each other, and tell them to connect. The people in far flung places would connect to people in more populous places who would connect to each other and messages would travel along those lines. 30 | 31 | But this has a big important job that someone has to do, and who is gonna pay them to do it? We'd rather have a system where everyone can just do their thing without any one having special responsibility (because that isn't fun) 32 | 33 | ## decentrally organize into a tree 34 | what if there was a way that people could just figure out on their own who is the closest? well, actually there is! When you send someone a message, you can measure how long it takes them to send a message back! this is equivalent to how close you are together! So, lets say we connect to random people like in the 2nd method - but then if we receive another message that we already know, we deactivate that connection. (since if we already know that message, we must have a faster path to the source along another path) 35 | 36 | So, the first time we build the network, it will have lots of extra connections, but the ones we don't need will get switched off. 37 | If we sometimes add new connections to other random people We'll eventually end up with the best network, the same as the top down network, but without having to have anyone at the top! 38 | 39 | So, once the network has "warmed up" it only needs to send 100 messages to make sure everyone gets everything, but no one person needs to send more than a couple of messages. 40 | 41 | --- 42 | 43 | okay, this story really needs pictures. also glosses over some aspects, but it's the basic gist of it. 44 | 45 | 46 | -------------------------------------------- 47 | 48 | 49 | But I glossed over one thing, that I'm gonna explain now. 50 | There is a problem with tree shape networks - we wanted to make sure that each message travels by the best route, but if we have only one way to get anywhere, what happens when something goes wrong? for example - one computer that is on the main branch crashes, runs out of batteries, enters a tunnel, or just the network is disrupted because too many people are using it. In real networks all of these things happen at least occasionally so when designing a network system we need to design it to still work when something goes wrong. 51 | 52 | What does that mean? If our network is a pure tree - it's also very fragile, because if a branch breaks, half the tree falls off. 53 | That means you suddenly can't share cat photos or debate about feminism because there is no path for your message to travel. 54 | 55 | How can we fix this? well, what we need is "redundency", or more simply, _backups_. We need the network tree to have extra connections that are still there, but we don't use unless we need them. So in EBT when we "deactivate" a connection we do not disconnect it, we just put it into _backup mode_. When you receive a message, you broadcast the whole message on the _active connections_ but on your _backup connections_ you just send a very short _note_ saying that you have the new message. 56 | 57 | Normally, you expect to receive the whole message along an active connection, and then receive notes about it on the backup connections. The message should arrive first, because it travels on the shortest path, and the notes take longer paths so they should arrive afterwards. 58 | 59 | What happens if you see a note about a message you havn't seen yet? Well, that means the fastest path isn't the fastest anymore! when that happens you send a short message along the backup connection, reactivating it. Now you have a new tree, and the network is still connected. 60 | 61 | 62 | -------------------------------- 63 | 64 | 65 | ## save loads of bandwidth this weird old trick 66 | 67 | There is also actually one more thing we do that saves a bunch of replication bandwidth when replicating - credit to [@cel](@f/6sQ6d2CMxRUhLpspgGIulDxDCwYD7DzFzPNr7u5AU=.ed25519) for having this idea. 68 | 69 | So, in gossip, two friends meet, then they exchange the latest news. But to save time (and bandwidth) we don't want to tell someone something they already know. 70 | 71 | So, when Alice and Bob connect, they'll first send a message saying what is the latest they've heard about each of their friends. so Bob might say "{Alice: yesterday, Bob: today, Carol: tuesday}" and if Alice says "{Alice: today, bob: yesterday, Carol: tuesday}" (note, alice and bob last talked yesterday) so Alice then knows to tell Bob her news from today - and Bob tells Alice her news, but neither of them have new updates about Carol. 72 | 73 | When also talks to Bob again tomorrow, Alice remembers that Bob had heard from Carol on tuesday. Alice hasn't heard anything new about Carol since then, so she doesn't need to mention Carol at first. If Bob has heard new news from Carol, He'll say something, and then Alice will request anything newer than tuesday. If Bob does the same, then they won't waste time talking about people who arn't doing anything. 74 | 75 | In most applications - you'll have some users who are very active, and then many others who don't interact very frequently. If there are thousands of users, some users might post many times a day, some every day, more every few days, some once a week and others might post then come back months later. 76 | 77 | If you connect and then reconnect then even quite an active user can be one that hasn't updated since you last connected. If the network is interrupted you'll need to reconnect, and sometimes the network is really bad and this happens lots! 78 | 79 | Because of this request skipping trick, connecting and starting the gossip only needs to mention the number of ids that have new news, but without this method, we have to send the total number of ids we want to replicate. If you have thousands of friends, that can add up to a lot of data! 80 | 81 | > (hmm, this could be a bit more explain-like-im-5, but working on it) 82 | 83 | 84 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # ssb field guide 2 | A field guide to developing with ssb 3 | 4 | Further reading: 5 | * [Scuttlebutt Protocol Guide](https://ssbc.github.io/scuttlebutt-protocol-guide/) 6 | * [gossip](https://github.com/nichoth/ssb-field-guide/blob/master/gossip.md) 7 | 8 | ## ssb 9 | The database layer is [secure-scuttlebutt](https://github.com/ssbc/secure-scuttlebutt). 10 | 11 | > A database of unforgeable append-only feeds, optimized for efficient replication for peer to peer protocols 12 | 13 | The database is implemented with [flumedb](https://github.com/flumedb/flumedb). Your data is probably stored in a file at `~/.ssb/flume/log.offset`. This is an append only log that has your messages and probably your friends and foafs messages as well. 14 | 15 | Also in `~/.ssb/flume` you will see some JSON files. These are materialized views of the database log (flume views). `friends.json` is a view of people within two hops in your foaf graph. 16 | 17 | ### feeds 18 | 19 | A feed is a signed append-only sequence of messages. Each identity has exactly one feed. Your ssb log contains many feeds (one for each identity that you are gossiping), all mixed together. Each message in a feed has a field `sequence` that is an increasing number. That way you can tell peers the latest sequence that you have, and then request or send newer messages. 20 | 21 | 22 | ## sbot 23 | 24 | [scuttlebot](https://github.com/ssbc/scuttlebot) (or sbot) provides network services on top of secure-scuttlebutt (the database). 25 | 26 | > The gossip and replication server for Secure Scuttlebutt - a distributed social network 27 | 28 | See also: 29 | 30 | * [Epidemic Broadcast Trees (explained like i'm five)](gossip.md) 31 | 32 | 33 | ## client 34 | 35 | A client is an application that uses sbot with various plugins as a backend. Each peer on the network corresponds to one database and one sbot instance, and the same sbot may be shared by several clients (see [ssb-party](https://www.npmjs.com/package/ssb-party)). All of these processes are probably running on one machine. 36 | 37 | ### ssb-client 38 | 39 | [ssb-client](https://github.com/ssbc/ssb-client) can be used to connect to an sbot running in a separate process. There are some important configuration bits that your application needs, `caps.shs` and `caps.sign`. Your client needs `shs`, and sbot needs `sign`. These determine which network your app connects to. By setting these you can do tests and development on a separate network. SHS stands for [secret handshake](https://github.com/auditdrivencrypto/secret-handshake). There is a [whitepaper](http://dominictarr.github.io/secret-handshake-paper/shs.pdf) about it too. 40 | 41 | Setting `caps.shs` makes gossip connections not occur with peers that have a different shs key. 42 | 43 | Setting `caps.sign` makes messages to be considered invalid that were created with a different sign key. 44 | 45 | If you only set the shs key, messages could leak out of your network if someone in the network changes their shs key, or adds messages manually. Setting the sign key ensures that the messages will not be able to be published (or validated) on the main ssb network, only on a network where that same sign key is used. 46 | 47 | `shs` and `sign` should be base64 encoded random strings 48 | ```js 49 | var crypto = require('crypto') 50 | crypto.randomBytes(32).toString('base64') 51 | ``` 52 | 53 | ```js 54 | var Client = require('ssb-client') 55 | var path = require('path') 56 | var home = require('os-homedir') 57 | var getKeys = require('ssb-keys').loadOrCreateSync 58 | 59 | var SSB_PATH = path.join(home(), '.ssb') 60 | var keys = getKeys(path.join(SSB_PATH, 'secret')) 61 | 62 | Client(keys, { 63 | path: SSB_PATH, 64 | caps: { 65 | shs: 'abc' 66 | } 67 | }, (err, sbot, config) => { 68 | // `sbot` here is an rpc client for a local sbot server 69 | }) 70 | ``` 71 | 72 | ### start an sbot 73 | The client depends on an sbot using the same `caps.shs` as the server. You can pass it in as an argument 74 | 75 | $ sbot server -- --caps.shs="abc" --caps.sign="123" 76 | 77 | 78 | See also [ssb-minimal](https://github.com/av8ta/ssb-minimal) 79 | 80 | 81 | ## sbot plugins 82 | Plugins expose methods via rpc to interact with an sbot. A plugin will typically read data from the ssb log, then create a view of that data that is saved somewhere, so it doesn't need to be recalculated from the beginning. 83 | 84 | An example is [ssb-contacts](https://github.com/ssbc/ssb-contacts). This uses flumeview-reduce to persist it's state, but you could use any persistence. 85 | 86 | Plugins have an interface used by [secret-stack](https://github.com/ssbc/secret-stack). For plugin authors this means you need a few fields in your module. `exports.manifest` is familiar if you have used [muxrpc](https://github.com/ssbc/muxrpc). The manifest is used to tell remote clients what functions your service is exposing via rpc. 87 | 88 | ```js 89 | exports.name = 'contacts' 90 | exports.version = require('./package.json').version 91 | exports.manifest = { 92 | stream: 'source', 93 | get: 'async' 94 | } 95 | exports.init = function (ssb, config) { /* ... */ } 96 | ``` 97 | 98 | 99 | ## replication 100 | https://github.com/ssbc/ssb-invite/blob/master/index.js#L195 101 | The remote server info is embedded in the invite code. 102 | 103 | The addresses of peers are stored in `~/.ssb/gossip.json`. 104 | 105 | [The line in ssb-invite that writes the address](https://github.com/ssbc/ssb-invite/blob/master/index.js#L263) 106 | 107 | [The part in ssb-gossip that writes the file](https://github.com/ssbc/ssb-gossip/blob/04b17c781b983980b318d9a0701060d0f831f7a7/index.js#L420) 108 | 109 | 110 | see @cel's [msg in ssb](https://viewer.scuttlebot.io/%250KAk8CvE7hNeV4GAFyzYdW8Qy%2Bb47tH%2F5O3RdH4znu0%3D.sha256) 111 | 112 | --------------------------------------- 113 | 114 | ## make a plugin 115 | Need to pass an object with a certain API that gets called by `sbot.use` 116 | 117 | ```js 118 | var myPlugin = { 119 | name: 'aaaaa', 120 | version: 0, 121 | manifest: { 122 | }, 123 | init: init 124 | } 125 | 126 | var sbot = Sbot.use(myPlugin)(config) 127 | console.log('**sbot aaaaa**', sbot.aaaaa) 128 | // logs the return value of `init` 129 | 130 | function init (sbot) { 131 | // this return value is accessible at `sbot.aaaaa` 132 | return { foo: 'foo' } 133 | } 134 | ``` 135 | 136 | ## make a databse view 137 | Here we create a materialized database view (a flumeView) 138 | 139 | * [flumeDB](https://github.com/flumedb/flumedb) 140 | * [ssb-links](https://github.com/ssbc/ssb-links/blob/master/index.js) 141 | * [flumeview-reduce](https://github.com/flumedb/flumeview-reduce) 142 | * [flumeUse](https://github.com/ssbc/ssb-db#db_flumeuse-view) 143 | * [flumeview-links](https://github.com/flumedb/flumeview-links) 144 | ------------------- 145 | * [flumeview-hashtable](https://github.com/flumedb/flumeview-hashtable) 146 | ----------------------------- 147 | 148 | ```js 149 | var codec = require('flumecodec') 150 | var createReduce = require('flumeview-reduce/inject') 151 | // this way our view state is persisted to disk 152 | var Store = require('flumeview-reduce/store/fs') 153 | 154 | var myPlugin = { 155 | name: 'aaaaa', 156 | version: 0, 157 | manifest: { 158 | }, 159 | init: init 160 | } 161 | 162 | var sbot = Sbot.use(myPlugin)(config) 163 | console.log('**sbot aaaaa**', sbot.aaaaa) 164 | 165 | sbot.publish({ 166 | type: 'post', 167 | text: 'Hello, world!' 168 | }, function (err, msg) { 169 | // the API we returned from `init` 170 | // which here is the api return by flumeview-reduce 171 | sbot.aaaaa.get(function (err, data) { 172 | console.log('**get**', err, msg) 173 | }) 174 | }) 175 | 176 | function init (sbot) { 177 | function reducer (acc, val) { 178 | acc.foo = acc.foo + 1 179 | return acc 180 | } 181 | function mapper (msg) { 182 | return msg 183 | } 184 | 185 | var initState = { foo: 0 } 186 | 187 | // need to use this API if we want to store data 188 | var Reduce = createReduce(Store) 189 | 190 | // view state is saved at ~/app-name/ok.json 191 | // b/c that's what we named it in `_flumeUse` here 192 | // the path is based on the path to the flume log 193 | // https://github.com/flumedb/flumeview-reduce/blob/master/inject.js#L90 194 | var view = sbot._flumeUse('ok', 195 | Reduce(1, reducer, mapper, codec.json, initState)) 196 | 197 | // the thing returned from here is at `sbot.aaaaa` 198 | return view 199 | } 200 | ``` 201 | 202 | ## ssb-db-2, JITDB 203 | `ssb-db2` is just a plugin like any other. I’m already using it in Manyverse. 204 | 205 | These projects are primarily for better performance. Secondary goal is to reset some of the legacy choices that wear us down. Third goal is to have a nicer API for querying. 206 | 207 | ### JITDB 208 | JITDB is just a lower level component. The name is called JIT because the indexes are created just-in-time, automatically inferred from the query. These are only bitvector indexes and prefix indexes. 209 | 210 | ---------------------------------------- 211 | 212 | ## ssb browser 213 | See [ssb-browser-core](https://github.com/arj03/ssb-browser-core) 214 | 215 | ### storage 216 | `ssb-browser-core` uses partial replication so the storage requirement is not as big as a normal client. 217 | 218 | The private key is saved in localstorage. 219 | 220 | When you run out of space, new writes will fail. 221 | 222 | [this article](https://web.dev/storage-for-the-web/) states that chrome can use up to 80% of available disk 223 | 224 | All of this is built on top of async-append-only-log, this in turn uses polyraf, which is a wrapper around [random-access-storage](https://github.com/random-access-storage/) family of libraries so that it works both in the browser and in node with the same api. In the browser it will use random-access-web which for firefox will use indexeddb but for chrome will use a special faster backend. 225 | 226 | #### blobs storage 227 | blobs are different than the log, as it is in a normal ssb client as well. There is a [small module](https://github.com/arj03/ssb-browser-core/blob/master/simple-blobs.js) that uses the same blobs protocol but stores the data using the `polyraf` library. The core library has a parameter where you can say that you only want to download and store blobs under a certain size. It will then stream larger blobs so they don’t take up space. Another approach would be use something like the [blobs purge](https://github.com/arj03/ssb-browser-demo/issues/8) library. 228 | 229 | ------------------------------------------------- 230 | 231 | ## secret handshake 232 | The handshake is mostly about authentication / verification; can the public keys of both peers be verified and are they using the same network key? If the answer to both of those questions is yes, then a **shared secret** is created - which I think of as a proof of verification or proof of identity. The shared secret can then be used to encrypt and decrypt further communications. 233 | 234 | you can use the same stream for the handshake and the box stream. The handshake will return a set of keys to each peer and those keys are used to create a pair of boxstreams (one for reading and one for writing). 235 | 236 | 237 | ## ssb-browser-core 238 | 239 | ### tests 240 | The way I have been testing this in browser demo is to have 2 browsers, could be 2 incognito modes or a chrome and a firefox. Then you basically create a follow messages on one of the browsers for the other identity. Do the same with the other browser. After that if you connect to the same room. You can use the one in browser demo, then the two browsers should start exchanging messages. 241 | 242 | ----------------------------------- 243 | 244 | > a room is basically a server that multiple peers can connect to. The room will faciliate e2e encrypted tunnels between the peers. This means that it won't have any trouble with hole punching and all of that stuff. This is an address to a room that works in the browser: https://github.com/arj03/ssb-browser-demo/blob/e935636feaddd6d13868a104a2ea0fdb3c176bff/defaultprefs.json#L12 245 | 246 | So you basically connect both clients to that using something like: https://github.com/arj03/ssb-browser-demo/blob/master/ui/connections.js 247 | 248 | ### how to get a list of connections 249 | 250 | Use `SSB.net.conn` 251 | 252 | [a recipe](https://github.com/staltz/ssb-conn#recipes) 253 | 254 | [in the wild](https://github.com/nichoth/sphotos/blob/b5a9ff8cf6c5393d91a42ad2a1fbcc20ab2bebfb/src/index.js#L61) 255 | 256 | -------------------- 257 | 258 | *credits* 259 | A lot of these notes were provided by [@cel](https://github.com/clehner) 260 | 261 | --------------------------------------------------------------------------------