├── PLAN-API-Documentation.md ├── PLAN-Elements-and-Design-Principles.jpg ├── PLAN-NodeSpace.md ├── PLAN-Proof-of-Correctness.md ├── README.md ├── SETUP.md └── hash-collision-odds.py /PLAN-API-Documentation.md: -------------------------------------------------------------------------------- 1 | # PLAN API Documentation 2 | 3 | ``` 4 | P.rivacy 5 | L.ogistics 6 | A.ccessibility 7 | P L A N.etworks 8 | ``` 9 | 10 | A technology is only as interesting as how it can be harnessed and applied in our world. 11 | 12 | ## Primary Interfaces 13 | 14 | PLAN features 7 primary areas of extension and interoperability. Together, they reflect PLAN's engineering mission to be modular, future-proof, and adaptable. 15 | 16 | | Area of Interoperability | Purpose | 17 | |:-------------------------------:|---------------------------------------------------------------------------------------------------------------------------------------------| 18 | | [Interoperable Data Structures](#Interoperable-Data-Structures) | Flexible, portable, self-describing, performant data structures | 19 | | [Persistent Data Interface](#Persistent-Data-Interface) | Abstracts a community's permanent data store; designed to be compatible with distributed ledgers (e.g. blockchains) | 20 | | [Channel Protocols](#channel-protocols) | Separates content streams by purpose and interpretation, not just by content type or format | 21 | | [Channel GUI Adapters](#Channel-GUI-Adapters) | Provides an interchangeable front-end GUI experience for a given channel type | 22 | | [Cloud File Interface](#Cloud-File-Interface) | Abstracts expendable shared bulk storage; designed to be compatible with distributed content-addressable storage systems | 23 | | [Secure Key Interface](#Secure-Key-Interface) | Abstracts private key handling and crypto services; designed to integrate third-party encryption and authentication systems | 24 | | [Web Services](#Web-Services) | Serves public requests for explicitly shared community content via conventional internet protocols | 25 | 26 | ## Interoperable Data Structures 27 | 28 | - PLAN features a powerful but tiny set of flexible, portable, self-describing, and performant data structures. 29 | - Thanks to [Protobufs](https://developers.google.com/protocol-buffers) and [gRPC](https://grpc.io), developers can access content in PLAN using every major language and over a network connection with only a few lines of code. 30 | - PLAN's standard unit of information storage, structure, and transport is `plan.Block`: 31 | ```golang 32 | // A portable, compact, self-describing, nestable information container inspired from HTTP. 33 | type Block struct { 34 | 35 | // An optional, name/label for this Block (i.e. a field-name). 36 | // A Block's label conforms to the context/protocol it's being used with (as applicable). 37 | Label string 38 | 39 | // Like a MIME type, this descriptor self-describes the data format of Block.Content. 40 | // Anyone handed this Block uses this field to accurately process/deserialize its content. 41 | // This is a "multicodec path" -- see: https://github.com/multiformats/multistream 42 | Codec string 43 | 44 | // This is a reserved integer alternative to Block.Codec. 45 | // See: https://github.com/multiformats/multicodec/blob/master/table.csv 46 | CodecCode uint32 47 | 48 | // Payload data, serialized in accordance with the accompanying codec descriptors (above). 49 | Content []byte 50 | 51 | // A Block can optionally contain nested "sub" blocks. A Block's sub-blocks 52 | // can be interpreted or employed any way a client or protocol sees fit. 53 | Subs []*Block 54 | } 55 | ``` 56 | - Importantly, each `plan.Block` instance is [self-describing](https://multiformats.io/) and can contain sub-blocks. Because each `plan.Block` can be accompanied by a label, codec descriptor, or any number of sub-blocks, it has the _simplicity and flexibility_ of JSON but the _efficiency and compactness_ of binary serialization. This means any hierarchy of information or content can be structured dynamically, and each element contains enough meta information for it to be safely analyzed and processed further. 57 | - Like other foundational data structures in PLAN, `plan.Block` is specified using [Protobufs](https://developers.google.com/protocol-buffers). This means boilerplate serialization and network handling code can be [trivially generated](https://github.com/plan-systems/plan-protobufs) for most major languages and environments, including C, C++, Haskell, Objective-C, Swift, C#, Go, Java, JavaScript, Python, and Ruby. Not bad! 58 | - Protobufs are faster, simpler, safer, more compact, and more efficient than JSON and XML. 59 | - A Protobuf struct ("message") can be composed of primitive data types or user-defined messages. 60 | - The fields of a Protobuf message are explicitly and strongly typed. 61 | - Revisions to a Protobuf message are backward-compatible with earlier revisions. 62 | - Protobufs pair well with [gRPC](https://grpc.io), opening up broad multi-language and multi-platform network support. 63 | - An entire `plan.Block` hierarchy can be serialized or deserialized using a single line of code — _in every major language and environment_. 64 | - PLAN's Protobuf-based data structures: 65 | 66 | | Protobuf File | Purpose | 67 | |--------------------|-------------------------------------------------| 68 | | [plan.proto](http://github.com/plan-systems/plan-protobufs/blob/master/pkg/plan/plan.proto) | PLAN-wide constants and data structures | 69 | | [pdi.proto](http://github.com/plan-systems/plan-protobufs/blob/master/pkg/pdi/pdi.proto) | Persistent Data Interface (PDI) data structures | 70 | | [ski.proto](http://github.com/plan-systems/plan-protobufs/blob/master/pkg/ski/ski.proto) | Secure Key Interface (SKI) data structures | 71 | | [repo.proto](http://github.com/plan-systems/plan-protobufs/blob/master/pkg/repo/repo.proto) | Repo-centric messages data structures | 72 | | [client.proto](http://github.com/plan-systems/plan-protobufs/blob/master/pkg/client/client.proto) | Client support and service | 73 | | [ch.proto](http://github.com/plan-systems/plan-protobufs/blob/master/pkg/ch/ch.proto) | Channel protocol data structures | 74 | 75 | 76 | --- 77 | 78 | 79 | ## Persistent Data Interface 80 | - The **Persistent Data Interface** ("PDI") is PLAN's append-only storage abstraction, storing a community's essential content (often text-based in nature). By the time content arrives at this layer, it has already been encrypted by the community's [community keyring](https://github.com/plan-systems/design-docs/blob/master/PLAN-Proof-of-Correctness.md#System-Security) and is therefore _cryptographically inaccessible_ to the rest of the world. 81 | - PLAN's persistent storage abstraction is a container for content, enforces postage, manages its structure and transport, and validates authorship. The PDI does not manage high-level user security or permissions. The PDI's primary purpose is to: 82 | 1. Store, replicate, and serve well-formed transactions (aka "extrinsics"), 83 | 2. Disallow/drop transactions whose author does not have community postage, and optionally 84 | 3. Sign/Seal _when_ a newly received transaction becomes part of the community record. 85 | - The PDI embraces an append-only model so that a wide range of centralized databases, replicating data types, and distributed ledgers can be used off the shelf. 86 | - PDI transactions ("channel entries") are modeled as immutable and permanent (though content mutability is recreated virtually via PLAN's higher-level channel database layer). 87 | - A PDI storage provider features [portability](PLAN-Proof-of-Correctness.md#Proof-of-Storage-Portability), so a community could start with a central database for convenience, and migrate to a distributed ledger better for scale later on down the road. 88 | - In [plan-core](http://github.com/plan-systems/plan-core), `StorageProvider` is a [gRPC](https://grpc.io/) service defined in [pdi.proto](https://github.com/plan-systems/plan-protobufs/blob/master/pkg/pdi/pdi.proto). A PDI storage node makes this service available to the clients/members of a community. There are two categories of PDI implementations: 89 | 1. **Centralized** - a `StorageProvider` implementation that uses a conventional central server or cluster. 90 | - Pros: 91 | - Low latency & high performance 92 | - Easy setup & maintenance 93 | - Static and predictable availability 94 | - Cons: 95 | - Single data choke-point 96 | - Single point of failure 97 | - [pdi-local](https://github.com/plan-systems/plan-pdi-local) is PLAN's centralized `StorageProvider` implementation. It is implemented internally using [Badger](https://github.com/dgraph-io/badger), a modern high performance key-value database. 98 | 2. **Decentralized** - a `StorageProvider` implementation that internally maintains peer connections with other nodes of its kind. Peers collectively maintain distributed state and, in effect, provide replicated, redundant storage. [Liveness vs Safety](PLAN-Proof-of-Correctness.md#Liveness-vs-Safety) discusses how one particular distributed ledger could fit well for one community but a poor fit for another. 99 | - Pros: 100 | - Highly censorship and [denial-of-service](https://en.wikipedia.org/wiki/Denial-of-service_attack) resistant 101 | - Not subject to central breach or failure 102 | - High redundancy; each node is a replica 103 | - "Offline-first" (network partitions can sub-operate and auto-sync when reconnected) 104 | - Cons: 105 | - Potentially high latency 106 | - Larger aggregate footprint 107 | - Firewall complications 108 | - In `pdi-eth` and `pdi-holo`, a private Ethereum and Holochain chain are used as the persistent datastore, respectively. Transaction payload blobs committed to `StorageProvider` are encoded into native blockchain transactions and committed to a private blockchain. For more, see the "[Scenario](PLAN-Proof-of-Correctness.md#Scenario)" section in the _PLAN Data Model Proof of Correctness_. 109 | 110 | --- 111 | 112 | ## Channel Protocols 113 | - PLAN's general purpose channels are its workhorse and _raison d'être_. Like files in a conventional operating system, users and productivity workflows in PLAN create new channels and new channel types all the time. 114 | - As a PLAN client UI interacts with a given channel, it does not use filename extensions, content-embedded markers, or blindly assume that content is in a particular form. Both PLAN channel "epochs" and channel entries each embed a `plan.Block`, making each a flexible self-describing container for content and information. _This offers profound interoperability in the way that HTTP headers also self-describe content for a HTTP response._ 115 | 116 | | Example Channel Descriptor | Expected Channel Entry Content Codecs | Example Client UI Experience | 117 | |----------------------|:--------------------:|--------------------------------------| 118 | | `/plan/ch/chat` | `txt`\|`rtf`\|`image` | A familiar "vertical scroller" where new entries appear in colored ovals at the bottom and previous entries vertically scroll upward to make room. | 119 | | `/plan/ch/geoplot` | `cords+(txt`\|`image)` | A map displays text and image annotations at each given geo-coordinate entry. Clicking/Tapping on an annotation causes a box to appear displaying who made the entry and when. | 120 | | `/plan/ch/file/pdf` | `ipfs`\|`binary` | The client UI represents this file-revision channel as a single monolithic object. Tapping on it causes the most recent channel entry (interpreted as the latest revision) to be fetched and opened locally on the client using a PDF viewing application. Power users can learn to open previous revisions of this "file". | 121 | | `/plan/ch/file/audio`| `ipfs`\|`mpg`\|`aac`\|`ogg`\|`flac` | Like other PLAN "file" channels, this client UI displays this channel as a single object, where opening/activating it causes the most recent entry to be fetched and played using the default media player app or using PLAN's integrated AV player. | 122 | | `/plan/ch/feed/rss` | `xml` | This channel is used to publish a sequence of text, audio, or video items with accompanying meta elements (e.g. title, link, thumbnail, and description). This channel's epoch content `Block` houses [RSS](https://en.wikipedia.org/wiki/RSS) channel information while PLAN channel entries correspond to familiar RSS `item` elements in xml. | 123 | | `/plan/ch/feed/atom` | `xml` | Similar to `feed/rss`, but PLAN channel entries instead conform to [Atom](https://en.wikipedia.org/wiki/Atom_(Web_standard)) xml. | 124 | | `/plan/ch/calendar` | `text/ifb`\|`text/ics` | The client UI presents a familiar visual calendar idiom containing events (entries) that are graphically rendered on the appropriate days and times. The user interacts with channel UI in real-time, scrolling from week to week, to day to day as the user zooms in "closer". | 125 | 126 | 127 | ### Custom Channel Protocols 128 | - A community or organization may have a specific need and always has the option to design a custom channel content protocol that meets specialized needs. 129 | - A custom-designed channel protocol generally will need an accompanying custom channel GUI adapter so PLAN clients can interact with that channel type. 130 | - For example, a brewery uses sensor arrays to monitor temperatures all over a warehouse. These sensors periodically write JSON data to a channel with a given custom channel type. The brewery has its own channel GUI adapter "bound" to the custom channel channel type that displays the floor plan of the brewery and overlaid with color swaths visualizing the most recent temperature readings. 131 | 132 | --- 133 | 134 | ## Channel GUI Adapters 135 | - A channel's protocol identifier string corresponds to a matching channel GUI adapter or "driver" in the PLAN client. Like a traditional hardware driver, a PLAN channel adapter is designed specifically to interface with a data consumer and producer having an established format and flow. 136 | - When a user accesses/opens a channel in PLAN, the client starts a new instance of the channel module designed _for that specific type of channel_. If multiple matching channel adapters are available, the client can choose based on user settings or can prompt the user to select one. 137 | - A PLAN channel adapter is a C# class that lives in the Unity client. New adapter instances are passed a gRPC connection set up for a given channel `UUID`. 138 | - A channel with type `/plan/ch/calendar`, could invoke the client's default `calendar` channel adapter _or_ instead use another that: 139 | - displays scheduled events on a horizontal timeline that extends from the past to the future 140 | - displays scheduled events on a geographically relevant map 141 | - overlays appointments from an outside calendar service 142 | - Users can choose alternate channel adapters in the way a media player offers alternate skins/UIs 143 | - Developers that create and extend channel adapters can focus on the API or GUI, rather than infrastructure related tasks. 144 | 145 | --- 146 | 147 | ## Cloud File Interface 148 | - The **Cloud File Interface** ("CFI") is an abstraction for [content-addressable storage](https://en.wikipedia.org/wiki/Content-addressable_storage), where files and content are referenced by hashname and are available across a community's network. 149 | - Unlike the [Persistent Data Interface](#Persistent-Data-Interface), content written to the CFI isn't necessarily intended to persist indefinitely (though it can). The CFI is an abstraction for scalable and expendable storage, and it serves a community's temporary and bulk storage needs. 150 | - Distributed storage systems typically only replicate on-demand (e.g. BitTorrent), so CFI content tends to only consume local storage for users accessing that content. This is in contrast to how entries posted to the PDI replicate to _every_ community node. 151 | - "File" channels allow PLAN users to interact with files and trees in familiar ways, but under the hood each entry in the channel is a CFI pathname that points to a revision of the file or tree. Since each revision pathname is _only_ a short string, these channels don't materially consume the community's permanent shared storage. When a user opens/views this object, the PLAN client hands off the most recent channel entry (a CFI pathname) to the CFI layer for retrieval while the PLAN client graphically reports progress. 152 | - Conveniently, this allows the PLAN client manage CFI garbage collection. For a given file channel, once an entry containing a CFI item is superseded longer than some grace period (or any other expiration function), the referenced item can be safely and automatically "unpinned" (or more aggressively reclaimed). 153 | - At the PLAN client level, the user is not burdened or distracted with the details associated with managing the CFI. The user never sees hashnames and doesn't have to know anything about how the PDI and CFI are working together. 154 | - Consider a film production team using PLAN as a collaboration and file-sharing tool. Their workflow is to present fresh cuts to the team for feedback and review. Instead of using the community's _permanent_ shared storage for short-term video files, the team uses a file channel where posted videos _appear_ in the channel and can be conveniently played. However, under the hood, the PLAN client is posting the file to the CFI and then placing a CFI pathname into an entry in the channel. The channel is set so that files older than X days expire and are unpinned/deallocated. 155 | - Like the PDI, the Cloud File Interface is designed to be pluggable, offer flexibility, and preserve portability. Most organizations using PLAN will be happy with PLAN's reference CFI implementation using [IPFS](https://ipfs.io/), a capable peer-to-peer distributed storage system. However, others may want to choose from other possibilities, such as [Dat](https://datproject.org/). 156 | - The CFI and PDI are architecturally disjoint, but a single storage layer could be used to implement both, provided it has the requisite capabilities. 157 | 158 | --- 159 | 160 | ## Secure Key Interface 161 | - The SKI abstracts private key storage, private key handling, and offers support for third-party encryption and authentication systems. 162 | - The SKI guarantees compartmentalization in PLAN, ensuring that private keys remain secure and reside "outside" of PLAN. 163 | 164 | 165 | --- 166 | 167 | ## Web Services 168 | 169 | A community using PLAN will inevitably be interested in making some of its parts accessible to the global public. A PLAN node allows publicly accessible services to serve explicitly designated community content and scale (as a distributed system) alongside traditional web or internet services. For example: 170 | - A musical artist uses PLAN to serve show recordings and official releases. 171 | - A documentary production uses PLAN to serve the film's trailer and the full film to users bearing a token. 172 | - A PLAN daemon periodically renders out an image of a map with spatial annotations from a community geo-space channel, served as an html page. 173 | - A PLAN email gateway daemon bridges access to the members of a PLAN community and the outside world. Unlike email, however, each incoming email contains an access token that the recipient previously issued the sender, effectively eliminating unsolicited messages ("spam"). Further, a sender who abuses their privileges (or loses or resells their token to a spammer), can be blocked without any concern of messages from _other_ senders being inadvertently filtered/blocked. 174 | 175 | 176 | 177 | --- 178 | --- 179 | 180 | 181 | Back to [README](README.md) 182 | -------------------------------------------------------------------------------- /PLAN-Elements-and-Design-Principles.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/plan-systems/design-docs/ef0b6c8f49be794801756b9ae59a5edd7b663bb5/PLAN-Elements-and-Design-Principles.jpg -------------------------------------------------------------------------------- /PLAN-NodeSpace.md: -------------------------------------------------------------------------------- 1 | # PLAN NodeSpace Data Structure 2 | 3 | ``` 4 | P.rivacy 5 | L.ogistics 6 | A.ccessibility 7 | P L A N.etworks 8 | ``` 9 | 10 | ## What is a NodeSpace? 11 | 12 | A NodeSpace is a PLAN channel data schema that reflects a collection of abstract objects ("nodes") in relation to each other and nodes in other NodeSpaces. Each node has a set of associated fields/parameters, including spatial coordinates and references ("links") to other nodes. 13 | 14 | ## Channel Data Structure 15 | 16 | The following tree structure defines the key/value entry storage schema for a NodeSpace channel. Each `NodeID` and `LayerID` is unique from other nodes and layers in the channel they reside in. Other than what is required to adequately represent content structure, all key/value entries are optional. 17 | 18 | ``` 19 | l//name => user-specified layer name 20 | /cord_space => cord space type 21 | /cord_unit => cord unit type 22 | /time_unit => time unit type 23 | /color => layer color ("RRGGBB") 24 | /index => layer index value (0, 1, 2..) 25 | 26 | l//... 27 | /... 28 | 29 | ... 30 | 31 | n///name => user-specified node name 32 | /uri[.] => [[/n/]|.][/] 33 | /x[1..4] => positional cord value 34 | /t => [start] time value 35 | /t.end => end time value 36 | /. => user-specified value 37 | /... 38 | 39 | n///... 40 | /... 41 | 42 | ... 43 | 44 | ``` 45 | 46 | 47 | ## Node Linking 48 | 49 | Each `uri` field allows a node to link to any other node. The linked node can reside in the same NodeSpace or an external NodeSpace specified by its channel URI. A node may contain any number of links, allowing most network graphs to be represented. The PLAN user interface offers node automated relationship visualization, offering users the ability to discern important information that may not otherwise be visible. 50 | 51 | 52 | ## Layers & Linking 53 | 54 | A NodeSpace contains a set of node layers, each identified by a `LayerID`. The default ("primary") node layer has an `index` value of 0 and is the implied layer when a node links to another node and no layer is specified. By convention, the primary node layer defines each node's `name`, default target `uri`, and a coordinate position as applicable. Additional node layers typically represent other contexts or representations of nodes (or node subsets). 55 | 56 | ## Example NodeSpaces 57 | 58 | Consider a shared creative maker-space. A NodeSpace channel can be used for each equipment workstation. The primary node layer could contain top-level information and contain a uri to a NodeSpace that presents a rich scene of information and links. Another NodeSpace can be used to represent project that have been created, where each project can cite a link to the project and links to what equipment was used. 59 | 60 | **NodeSpace uri `shop101-eq`** 61 | ``` 62 | /l/11/name => "Shop 101 Work Stations" 63 | /cord_space => "cartesian/xy" 64 | /cord_unit => "length/meters" 65 | /index => 0 66 | 67 | /l/30/name => "Training Coordinators" 68 | /index => 1 69 | 70 | /l/55/name => "Power Distribution" 71 | /index => 2 72 | 73 | /l/60/name => "Find Inspiration!" 74 | /index => 3 75 | 76 | /n/73l6/11/name => "Lincoln 220 Cutter/Welder" 77 | /uri => "shop101/lincoln-220-welcome" 78 | /x0 => 33.0 79 | /x1 => 20.0 80 | /30/uri => "shrugs-faculty/n/424323" 81 | /55/uri => "shrugs-hall/n/NE-breaker-8-01" 82 | /60/uri => "shop101-gallery/n/2020-1" 83 | 84 | /n/7110/11/name => "Miller Plasma Cutter" 85 | /uri => "shop101/miller-cutter-welcome" 86 | /x0 => 10.0 87 | /x1 => 30.0 88 | /30/uri => "shrugs-faculty/n/12331" 89 | /55/uri => "./8801/." 90 | 91 | /n/8801/11/name => "CrossFire Plasma Table" 92 | /uri => "shop101/cf-plasma-welcome" 93 | /x0 => 10.0 94 | /x1 => 18.5 95 | /30/uri.0 => "shrugs-faculty/n/424323" 96 | /uri.1 => "shrugs-TAs/n/6456" 97 | /55/uri => "shrugs-hall/n/NE-breaker-8-02" 98 | /60/uri => "shop101-gallery/n/2020-2" 99 | ``` 100 | 101 | **NodeSpace uri `shop101-gallery`** 102 | ``` 103 | /l/10/name => "Shop 101 Project Gallery" 104 | /time_unit => "utc/seconds" 105 | /index => 0 106 | /.desc => "Experience famous projects have been created" 107 | 108 | /l/30/name => "Equipment Used" 109 | /index => 1 110 | /.desc => "Equipment used to craft each project" 111 | 112 | /l/40/name => "Project Exhibit Location" 113 | /index => 2 114 | /cord_space => "earth-geospace/latlong" 115 | /cord_unit => "angle/degrees" 116 | /.desc => "Experience the project installation site" 117 | 118 | /n/2020-1/10/name => "Waldo's Wall" 119 | /.desc => "Steel triangles cut and welded into a mosaic" 120 | /uri => "shop101/waldos-wall-project" 121 | /uri.author => "shrugs-univ/n/17145" 122 | /t => 1593266130 123 | /30/uri.0 => "shop101-eq/n/73l6" 124 | /uri.1 => "shop101-eq/n/7110" 125 | /40/x0 => 38.8931 126 | /x1 => 77.0458 127 | 128 | /n/2020-2/30/name => "Stencil Shrug" 129 | /.desc => "Metal sculpture with carefully cut and welded metal" 130 | /uri => "shop101/shrug-sculpture" 131 | /uri.author => "shrugs-TAs/n/6456" 132 | /t => 1592056593 133 | /30/uri.0 => "shop101-eq/n/8801" 134 | /uri.1 => "shop101-eq/n/73l6" 135 | /40/uri => "acme-virtual-tours-321 136 | /x0 => 31.7683 137 | /x1 => -35.2137 138 | /t => 1593216511 139 | /t.end => 1655128530 140 | 141 | ``` 142 | 143 | 144 | ## Official Specification 145 | 146 | A user experiencing a NodeSpace will typically be in a richly populated spatial environment, complete with labels, placed graphics, text, and visual assets. This means the user interface will be repeatedly loading linked nodes as the user navigates and explores the space. This process is more efficient and highly facilitated by use of a binary serialization schema. 147 | 148 | See [plan.proto](https://github.com/plan-systems/plan-proto/blob/master/plan.proto) for the most up to date revision of NodeSpace. 149 | -------------------------------------------------------------------------------- /PLAN-Proof-of-Correctness.md: -------------------------------------------------------------------------------- 1 | # PLAN Security Model & Proof of Correctness 2 | 3 | ``` 4 | P.rivacy 5 | L.ogistics 6 | A.ccessibility 7 | P L A N.etworks 8 | ``` 9 | 10 | ## What is this? 11 | 12 | This document is a proof of correctness for the PLAN security model, an IRC-inspired virtual channel system that serves as a canvas for community content organization, access controls, real-world security features. The purpose of this system is to host PLAN-compatible clients, playback requested channel entries, process and propagate newly authored entries, and maintain a consistent and secure state. 13 | 14 | In computer science, a "proof of [correctness](https://en.wikipedia.org/wiki/Correctness_(computer_science))" refers to a formal walk-through and demonstration that a proposed method and/or design rigorously satisfies a given set of specifications or claims. The intention is to remove _all doubt_ that there exists a set of conditions such that the proposed method would _not_ meet all the specifications. 15 | 16 | Below, we express a [scenario](#scenario), list a [set of specifications](#Specifications-&-Requirements), and propose [a system of operation](#Proposed-System-of-Operation) intended to address the scenario and specifications. We then proceed to demonstrate [correctness for each specification](#Proof-of-Specifications), citing how the system and its prescribed operation satisfies that specification. 17 | 18 | This document, although labeled "proof", is not perfect and has areas needing deeper analysis. It is intended to be a blueprint and serve as an ongoing open analysis of a pluggable, distributed, and extensible system. The data structures listed here are intended to convey understanding and model correctness more than they are intended to be performant. 19 | 20 | 21 | ## Table of Contents 22 | 23 | 24 | - [Scenario](#scenario) 25 | - [Specifications & Requirements](#specifications--requirements) 26 | - [Proposed System of Operation](#proposed-system-of-operation) 27 | - [Standard Procedures](#standard-procedures) 28 | - [Liveness vs Safety](#Liveness-vs-Safety) 29 | - [Proof of Specifications](#proof-of-Specifications) 30 | 31 | 32 | --- 33 | 34 | 35 | # Scenario 36 | 37 | A founding set of community organizers ("admins") wish to form **C**, a secure distributed storage network comprised of computers with varying capabilities, each running a common peer-to-peer software daemon ("node"). **C** is characterized by a set of individual members for any given point in time, with one or more members charged with administering member status, member permissions, and community-global rules/policies. 38 | 39 | On their nodes, the members of **C** agree to employ **𝓛**, an _append-only_ [CRDT](https://en.wikipedia.org/wiki/Conflict-free_replicated_data_type). Data entries appended to **𝓛** ("transactions") are characterized by an arbitrary payload buffer, a signing public key, and a signature authenticating the transaction. Transactions on any given **𝓛** are considered to be "in the clear" (i.e. neither "wire" privacy _nor_ storage privacy is assumed). 40 | 41 | Let **𝓛C** be a CRDT whose genesis is under exclusive control of the admins of **C**. **𝓛C** is assumed to either contain (or have access to) a verification system such that a transaction submitted to **𝓛C** is acceptable _only if_ the transaction's author (signer) has explicit _𝓛-append permission_ ("postage"). At first this may appear to be a strong requirement, but it reflects the _transference_ of security liability of the key(s) specified during the genesis of **𝓛C** to an _external_ set of authorities. 42 | 43 | For example, a customized "private distro" of the [Ethereum](https://en.wikipedia.org/wiki/Ethereum) blockchain ("**⧫**") could be used to implement **𝓛** since: 44 | - The admins of **C**, on creating **⧫C**, would issue themselves some large bulk amount _C-Ether_ (postage) 45 | - The admins of **C** would periodically distribute portions of _C-Ether_ to members of **C** (a postage quota). 46 | - On **C**'s nodes, **⧫C**: 47 | - Large payload buffers would be split into 32k segments (Ethereum's transaction size limit) and _then_ committed. 48 | - Any transaction that does not "burn" an amount of postage commensurate with the byte size of the payload would be dropped/rejected. 49 | - Any transaction that attempts to transfer postage from a non-designated identities would be dropped/rejected. 50 | 51 | For context, consider watching the distinguished [George Glider](https://en.wikipedia.org/wiki/George_Gilder) in this [video clip](https://www.youtube.com/watch?v=cidZRD3NzHg&t=1214s) speak on the empowering nature of distributed information technology. 52 | 53 | --- 54 | 55 | ## On Digital Security 56 | 57 | We acknowledge that even the most advanced and secure systems are vulnerable to private key loss or theft, socially engineered deception, and physical coercion. That is, an adversary in possession of another's private keys without their knowledge, or an adversary manipulating or coercing others is difficult (or impossible) to prevent. Biometric authentication systems can mitigate _some_ of these threats, but they also introduce additional surfaces that could be exploited (e.g. spoofing a biometric device or exploiting an engineering oversight). 58 | 59 | Security frameworks often don't analyze or provision for the loss of private keys since the implications are typically catastrophic, effectively making the issue someone else's intractable problem. Any system lacking such analysis and provisioning can only be considered incomplete for every-day use. The system of operation presented here features swift countermeasures and recovery _once it becomes known_ that private keys have been compromised (or suspect activity has been witnessed). 60 | 61 | --- 62 | 63 | 64 | # Specifications & Requirements 65 | 66 | The members of **C** wish to assert... 67 | 68 | #### Signal Opacity 69 | - For all actors _not_ in **C**, all transactions sent to, read from, and residing on **𝓛C** are informationally opaque to the maximum extent possible. 70 | 71 | #### Access Exclusivity 72 | - _Only_ members of **C** effectively have read and append access to **𝓛C**. 73 | - Alternatively, parts of **C** can be set up for "public access" where non-members of **C** have read access to select community content. 74 | 75 | #### Permissions Assurance 76 | - There is a hierarchy of member admin policies and permissions that asserts itself in order to arrive at successive states (and cannot be circumvented). 77 | - Even if multiple members are (or become) covert adversaries of **C** or are otherwise coerced, it must still be impossible to: 78 | - impersonate other members, 79 | - insert unauthorized permission or privilege changes, 80 | - gain access to others' private keys or information, or 81 | - alter **𝓛C** in any way that poisons or destroys community content. 82 | 83 | #### Accountability Assurance 84 | - The members of **C** are confident and can rest assured that every member is: 85 | - **accountable**, in that any exercise of their authority is community-public information and cannot be altered or concealed, _and_ 86 | - **bound**, in that they cannot circumvent **C**'s established rules and governance. 87 | - Even members in the _highest positions of authority_ within **C** are both **accountable** and **bound**. 88 | 89 | #### Membership Fluidity 90 | - New members can be invited to and join **C** at any time (given that **C** policies and permissions are met). 91 | - A member can be deactivated from **C** such that they become equivalent to an actor that has never been a member of **C** (aside that deactivated members can retain their copies of **𝓡** before the community entered this new security "epoch"). 92 | 93 | #### Strong Eventual Consistency 94 | - For each node **ni** in **C**, it's local replica state ("**𝓡i**"), converges to a stable/monotonic state as **𝓛C** message traffic eventually "catches up", for any set of network traffic delivery conditions (natural or adversarial). That is, **𝓡1**...**𝓡N** mutate such that strong eventual consistency ("SEC") is guaranteed. 95 | 96 | #### Practical Security Provisioning 97 | - If/When it is discovered that a member's private keys are known to be either lost or possibly comprised, a "[Member Halt](#member-halt)" can be immediately initiated such that any actor in possession of said keys will have no further read or write access to **C**. 98 | - Actors that can initiate a Member Halt include: 99 | - the afflicted member, _or_ 100 | - a member peer (depending on community-global settings), _or even_ 101 | - an automated watchdog system on **C** (responding to abnormal or malicious activity) 102 | - Following a Member Halt, the afflicted member's security state enters new "epoch" and incurs no additional security liability. That is, there are no potential "gotchas" sometime down the road if an adversary gains access to previously compromised keys. 103 | 104 | #### Independence Assurance 105 | - In the event that: 106 | - an adversary gains access to admin/root private keys and hijacks **C**, _or_ 107 | - one or more admins becomes adversarial towards **C**, _or_ 108 | - **𝓛C** is otherwise corrupted or vandalized, 109 | - ...then **C** can elect to "hard fork" **𝓛C** to an earlier time state, where specified members are struck from the member registry and others are granted admin permission. 110 | 111 | #### Storage Portability 112 | - **C**, led by a coordinated admin effort, retains the ability to switch-out 𝓛 technologies. 113 | - For example, suppose **𝓛C** has the safety feature such that it automatically halts under suspicious network conditions or insufficient peer connectivity (but requires central connectivity). However, earlier in **C**'s history, they used a CRDT that favored [liveness over safety](#liveness-vs-safety), allowing partitions of **C** to still be able to collaborate without central connectivity. 114 | - Or for example, suppose a member of **C** inadvertently adds data to the community record that is inappropriate or legally unacceptable (even if it is no longer accessible at the client level). In such case, the community would want to "rebase" the community's persistent store so that the offending content entries are not carried over. 115 | 116 | 117 | --- 118 | 119 | # Proposed System of Operation 120 | 121 | The members of **C** present the following system of operation... 122 | 123 | ## System Synopsis 124 | 125 | - This system embraces a multi-tier security model, where each community member possesses a community-common keys as well as private keys. In effect, this places the entire system's infrastructure and transaction traffic inside a cryptographic bubble. 126 | - The system's data model is IRC-inspired in that member interaction takes the form of data entries posted serially to channels within a virtual channel address space. However, instead of channel entries just being rebroadcast to other connected clients (as on an IRC server), entries _persist_ as transactions replicated across **𝓛C**. 127 | - When a channel is created, it is assigned a protocol descriptor string, specifying the _kind_ of entries that are expected to appear that channel and _how_ UI clients should interpret them (functionally comparable to MIME types). This, plus the ability for _any_ channel entry to include arbitrary HTTP-style headers, creates many possibilities for visually-oriented client interfaces. 128 | - Also inspired from IRC, each channel has its own permissions settings. Each channel designates an access control channel ("ACC") to be used as an oracle for channel permissions. An ACC is a special channel type that additionally conforms to a protocol designed to specify channel permissions. Like other channels, each ACC also designates a parent ACC, and so on, all the way up to **C**'s root-level ACC. 129 | - Member, channel, and community security and key distribution uses "epochs" to demarcate security events, in effect furnishing [permissions assurance](#permissions-assurance). 130 | - In a flow known as [Channel Entry Validation](#channel-entry-validation), each community node ("**ni**") iteratively mutates its local replica ("**𝓡i**") by attempting to merge newly arriving entries from **𝓛C** into **𝓡i**. During validation, if **𝓡i** is not yet in a state to fully validate an incoming entry **e**, then **e** is said to be "deferred" for later processing. 131 | - This system, in effect, forms a secure and operational core outside **C**'s channel data space, comparable to how a traditional OS maintains internal pipelines and hierarchies of operations and permissions, designed to serve and protect user processes. 132 | 133 | --- 134 | 135 | ## System Security 136 | 137 | - Let `UUID` represent a constant-length independently generated identifier that ensures no reasonable chance of peer collision. Although it is difficult to express [collision odds](http://preshing.com/20110504/hash-collision-probabilities/) in meaningful human terms (even for "modest" probability spaces such as 1 in 2160), 20 to 32 pseudo-randomly generated bytes is [more than sufficient](hash-collision-odds.py). 138 | - Each member **m** in **C** securely maintains custody of two "keyrings": 139 | 1. **[]km**: **m**'s _personal keyring_, which: 140 | - decrypts/encrypts information "sent" to/from **m**, _and_ 141 | - creates signatures that authenticate information authored by **m**. 142 | 2. **[]kC**: the _community keyring_, which ensures **C**'s "community-public" data is only readable by members of **C**. 143 | - Each entry authored by **m** is encrypted by **m**'s local client using the latest community key on **[]kC**. 144 | - That is, **[]kC** encrypts/decrypts `EntryCrypt` traffic to/from **𝓛C** and **𝓡i** 145 | - Newly [issued community keys](#Issuing-a-New-Community-Epoch) are securely distributed to members via the [Community Epoch Channel](#Community-Epoch-Channel). 146 | 147 | 148 | --- 149 | 150 | 151 | ## Channel Entries 152 | 153 | - Transactions residing in **𝓛C** are storage containers for `EntryCrypt`: 154 | ```golang 155 | type EntryCrypt struct { 156 | CommunityKeyID UUID // The community key used to encrypt .InfoCrypt 157 | InfoCrypt []byte // EntryInfo encrypted with .CommunityKeyID 158 | ContentCrypt []byte // Channel content, encrypted with EntryInfo.ContentKeyID 159 | Sig []byte // Authenticates this entry; signed by EntryInfo.AuthorMemberID 160 | } 161 | ``` 162 | - Given entry **e** arriving from **𝓛C** and access to **[]kC**, node **ni** decrypts **e**`.InfoCrypt` into an `EntryInfo` ("**einfo**"): 163 | ```golang 164 | type EntryInfo struct { 165 | EntryOp int32 // Entry opcode. Typically, POST_CONTENT 166 | TimeAuthored int64 // When this struct was sealed/signed 167 | ChannelID UUID // Channel that this entry is posted to (or operates on) 168 | ChannelEpochID UUID // Epoch of the channel in effect when entry was authored 169 | AuthorMemberID UUID // Creator of this entry (and signer of EntryCrypt.Sig) 170 | AuthorMemberEpoch UUID // Epoch of the author's identity when entry was authored 171 | ContentKeyID UUID // Identifies the key used to encrypt EntryCrypt.ContentCrypt 172 | } 173 | ``` 174 | - Every entry specifies a destination `ChannelID` within **C**'s channel space to be merged into. 175 | - During [Channel Entry Validation](#Channel-Entry-Validation), newly arriving entries from **𝓛C** are validated and merged into node **ni**'s locally stored `CommunityRepo` ("**𝓡i**"). 176 | - **𝓡i** consists of: 177 | - a datastore for each channel `UUID` that makes an appearance in **C** 178 | - bookkeeping needed to resume sessions with **𝓛C** 179 | - a queue of entries to be merged in accordance with [Channel Entry Validation](#Channel-Entry-Validation). 180 | - a mechanism for "deferred" entries to be retried periodically 181 | ```golang 182 | // CommunityRepo is a node's replica/repo/𝓡i 183 | type CommunityRepo struct { 184 | ChannelsByID map[UUID]ChannelStore 185 | } 186 | 187 | // ChannelStore stores entries for a given channel and provides rapid access to them 188 | type ChannelStore struct { 189 | ChannelID UUID 190 | ChannelProtocol string // "/chType/ACC" or "/chType/desc/" 191 | EpochHistory []ChannelEpoch // Record of all past channel epochs 192 | EntryTable []EntryIndex // Entry info indexed by TimeAuthored and hashname 193 | ContentTome ContentTome // Entry content store/db for channel 194 | } 195 | 196 | // EntryIndex packages the the essential parts of an entry, plus status information. 197 | type EntryIndex struct { 198 | EntryInfo EntryInfo 199 | EntryStatus EntryStatus // Status of entry (e.g. LIVE, DEFERRED) 200 | ContentPos uint64 // Byte offset into ..ContentTome 201 | ContentLen uint32 // Byte length at .ContentPos in ..ContentTome 202 | } 203 | ``` 204 | 205 | --- 206 | 207 | ## Channel Epochs 208 | 209 | Under an append-only storage model, the mechanism that gives rise to mutable permissions and access controls is centered around `ChannelEpoch`. 210 | - In a procedure known as [issuing a new channel epoch](#Issuing-a-New-Channel-Epoch), an owner of channel **𝘾𝒉** posts a new revision to **𝘾𝒉**'s current `ChannelEpoch` to: 211 | - edit properties specific to **𝘾𝒉**, _or_ 212 | - designate a different parent ACC for **𝘾𝒉**. 213 | - Naturally, part of [Channel Entry Validation](#channel-entry-validation) is to reject entries from members that lack the appropriate permissions to issue a new `ChannelEpoch` for a given channel. 214 | ```golang 215 | // Specifies general epoch parameters and info 216 | type EpochInfo struct { 217 | TimeStarted timestamp 218 | TimeClosed timestamp 219 | EpochID UUID 220 | ParentEpochID UUID // 0 if this is the first epoch. 221 | TransitionSecs int // How long before previous epochs "expire" 222 | ... // Other epoch transition parameters 223 | } 224 | 225 | // ChannelEpoch represents a "rev" to a channel's security properties. 226 | type ChannelEpoch struct { 227 | EpochInfo EpochInfo 228 | AccessChannelID UUID // This is channel's parent ACC; cannot form circuit 229 | MemberGrants AccessGrants // Permissions for explicitly specified members 230 | DefaultGrants AccessGrants // Permissions for members not otherwise specified 231 | } 232 | ``` 233 | 234 | --- 235 | 236 | ## Channels 237 | Channels are intended as general-purpose containers for [channel entries](#channel-entries) of all forms. This system uses channels internally for administration and permissions controls. 238 | 239 | 240 | 1. **General purpose channels**, alas, are this system's primary deliverable and _raison d'être_. 241 | - Most channels are this type and a general purpose channel is somewhat analogous to a file on a traditional operating system. 242 | - When a new channel is created, the creator (and hence owner) specifies a "protocol descriptor", a string directing client UIs to consistently interpret, handle, and present channel entries in accordance with the expectations associated with the channel's protocol. 243 | - Each channel also names a governing access control channel ("parent ACC"). A channel's parent ACC is charged with returning a permission level for any given member `UUID`, allowing nodes in **C** to independently carry out [Channel Entry Validation](#channel-entry-validation). 244 | - General purpose channels are either: 245 | - **community-public**, where channel entry content is encrypted with the latest community key, _or_ 246 | - **private**, where entry content is encrypted with the key identified by **einfo**`.ContentKeyID`. 247 | - Key mechanics for private channels are similar to [starting a new community epoch](#issuing-a-new-Community-Epoch), except the channel owner updating the `ChannelEpoch` performs key generation and distribution. 248 | - Only members that have at least read-access are "sent" the keys needed in order to decrypt private channel entries. 249 | - By default, community admins _do not_ have the authority/means to gain access to a private channel's key. 250 | - This ensures that _only the members that have been explicitly given channel access_ could possibly have access to the channel's key. 251 | 2. **Access Control Channels** (ACCs) are specialized channels used to express permissions for all other channels, including other ACCs. 252 | - An ACC serves as an access authority that specifies: 253 | - channel permissions for a given member `UUID`, _and_ 254 | - default permissions for members not otherwise specified. 255 | - Like general purpose channels, each ACC must designate a parent ACC, and so on, all the way up to the _reserved_ [root ACC](#Root-Access-Control-Channel). 256 | - Multiple channels can name the _same_ ACC as their parent ACC, allowing a single ACC to conveniently manage permissions for any number of channels. 257 | - ACCs also are the vehicle for key distribution, where a channel owner "sends" a newly issued private channel to each member with access, using their public key. 258 | 3. **Reserved channels** are specialized channels used by the system to internally carry out community governance and member administration. 259 | - Reserved channels specify root-level information and permissions, namely admin and member records. 260 | - Entries in these channels must meet additional security/signing requirements and serve prescribed purposes. 261 | - Because reserved channels have nuanced specifications, they do not solely rely on ACCs for access controls. 262 | - The number, purpose, and use of these channels can be expanded to meet future needs. 263 | 264 | 265 | ## Reserved Channels 266 | 267 | 268 | #### Root Access Control Channel 269 | - This is **C**'s root access channel, specifying which members are community authorities ("admins"). 270 | - All community-public channels, including ACCs, are implicitly under authority of this channel. 271 | - When a new community is formed ("community genesis"), the initial entries this channel are auto-generated in accordance with the parameters and policies provided by the founding members. 272 | - Automated machinery in **C** can optionally be geared to use smart contracts on **𝓛C** to manage, monitor, or validate entries in this channel. 273 | - E.g. a majority vote of existing admins could be required in order to add a new admin to the root access channel. 274 | 275 | 276 | #### Member Epoch Channel 277 | - This is a special channel where members post revisions to their currently published `MemberEpoch`, also known as the _Community Member Registry_ channel. 278 | - `MemberEpoch` contains essential information about a specific member, such as their most recently published public keys and their "home" channel `UUID` 279 | ```golang 280 | // MemberEpoch contains a member's community-public info 281 | type MemberEpoch struct { 282 | MemberID UUID 283 | EpochInfo EpochInfo 284 | PubSigningKey []byte 285 | PubEncryptKey []byte 286 | HomeChannel UUID 287 | } 288 | ``` 289 | - Each entry is this channel embeds a `MemberEpoch`, **𝓔**, and is only considered valid if: 290 | - the member who signed the entry matches the `MemberID` that appears in **𝓔** (or is an authorized member delegated to do so), _and_ 291 | - the predecessor ("parent") epoch of **𝓔** is eligible to be superseded. 292 | - `MemberEpoch` importantly publishes a member's public keys to the rest of the community, allowing each node in **C** to maintain a database used to: 293 | - authenticate signatures on each `EntryCrypt` 294 | - encrypt entry content exclusively for a given member (used for key distribution) 295 | - Only a community admin (or certified delegate) is permitted to post a `MemberEpoch` for members _other than themselves_. This provides the means for: 296 | - [adding new members](#Adding-A-New-Member) to **C**, 297 | - [deactivating members](#deactivating-A-Member) from **C**, _and_ 298 | - restoring a member's access to **C** following a [Member Halt](#member-halt). 299 | - When a [Member Halt](#member-halt) has been issued on **m**, an special `MemberEpoch` entry ("**𝓔halt**") is posted to this channel. 300 | - The authorizing signature of **𝓔halt** must be: 301 | - one of **m**'s personal signatures, _or_ 302 | - a member previously designated by **m**. 303 | - **𝓔halt** is permitted to be signed by **m**'s recently superseded keys, otherwise an adversary in possession of **[]km** could "lock out" **m** by [issuing a new member epoch](#Issuing-a-New-Member-Epoch). 304 | - Once **𝓔halt** goes live, [Channel Entry Validation](#Channel-Entry-Validation) will defer _all further entries_ bearing **m**'s signature. 305 | - If/When the cause for concern is addressed, a community authority would [issue a new member epoch](#Issuing-a-New-member-Epoch) for **m**, superseding **𝓔halt**. 306 | 307 | 308 | #### Community Epoch Channel 309 | - This channel is where a community admin (or authorized agent) posts an entry that, in effect, replaces the current community key with a newly issued key. 310 | - This channel contains a succession of entries that embed: 311 | - an `EpochInfo` containing parameters associated with the new community epoch, _and_ 312 | - a newly generated symmetric key _for each_ member **m** in **C**, encrypted using **m**'s latest public key published in **C**'s [Member Epoch Channel](#Member-Epoch-Channel): 313 | ```golang 314 | // KeyIssue is the vessel used to securely pass a ski.KeyEntry to another member 315 | type KeyIssue struct { 316 | MemberID UUID // Specifies the recipient 317 | MemberEpochID UUID // Implies which key was used to encrypt .KeyEntryCrypt 318 | KeyEntryCrypt []byte // Encrypted ski.KeyEntry 319 | } 320 | ``` 321 | 322 | --- 323 | 324 | 325 | ## Standard Procedures 326 | 327 | 328 | #### Issuing a New Member Epoch 329 | 330 | - Given: member **m** wishes to replace their currently published `MemberEpoch` with a new revision: 331 | - **m** generates new encryption and signing key pairs and places the private keys into their personal keyring, **[]km**. 332 | - **m** prepares a replacement `MemberEpoch`, **𝓔′**, placing the newly generated public keys into **𝓔′**. 333 | - **m** packages **𝓔′** into a new entry ("**e𝓔′**"), signs it, and posts it to **C**'s [Member Epoch Channel](#Member-Epoch-Channel). 334 | - As **e𝓔′** propagates across **𝓛C** (and goes live on **𝓡i**): 335 | - [Channel Entry Validation](#Channel-Entry-Validation) now requires that entries authored by **m** must use the newly published signing key. 336 | - Other member clients intending to securely pass keys or content to **m** would use **m**'s updated public encryption key. 337 | - If a [Member Halt](#member-halt) has been ordered on **m**, then admin (or community authority) intervention is required before **m** is permitted to post a new `MemberEpoch`. 338 | 339 | 340 | #### Issuing a New Community Epoch 341 | - Given: an admin, delegated member, or an automated agent wants to issue a new/replacement community key and deprecate the previously issued community key. 342 | - The purpose of starting this new "community epoch" is so that an actor in possession of the community keyring ("**[]kC**") will no longer be able to decrypt community-public data on **C** _unless they are currently a member of **C**_. This typically applies after a [Member Halt](#member-halt) or after [deactivating a member](#deactivating-A-member) since it's important that the currently active community key (and an actor in possession of it) is longer used to encrypted community traffic. 343 | - As the newly posted community key epoch goes live across the nodes of **C**, all new `EntryCrypt` on **𝓛C** are encrypted using it. 344 | - A community authority or agent does the following: 345 | 1. Generates a new symmetrical key to serve as the next community key ("**kC**"). 346 | 2. Prepares a new entry containing an `EpochInfo` ("**𝓔C**"). 347 | 3. For each open/active `MemberEpoch` ("**𝓔m**") in the [Member Epoch Channel](#Member-Epoch-Channel) (for each member **m** in **C**): 348 | - Creates a new `KeyIssue` (intended for **m**), encrypting **kC** using **𝓔m**`.PubEncryptKey`, 349 | - Appends the `KeyIssue` key to **𝓔C**'s content body. 350 | 4. Posts **𝓔C** to the [Community Epoch Channel](#Community-Epoch-Channel). 351 | - Member **m**'s client, upon seeing **𝓔C** go live: 352 | 1. Searches the content body of **𝓔C** for a `KeyIssue` matching **m**'s member ID. 353 | 2. Recovers **kC** from the `KeyIssue` using the member's private keyring. 354 | 3. Adds **kC** to **m**'s own community keyring. 355 | 4. Uses **kC** to encrypt all subsequent authored entries bound for **𝓛C** 356 | - Within [**ΔC**](#on-network-latency), newly authored transactions are _strictly only_ readable by the current members of **C**. 357 | - An actor in possession of a halted keyring or any keys past member epochs will not have any of the keys needed to extract **kC**. 358 | - If **𝓛C** favors safety, then **C** could additionally be configured to reject entries encrypted with a community key older than **ΔC** since offline nodes effectively remain in a halted state until they regain access to a critical threshold of central validators. 359 | 360 | #### Issuing a New Channel Epoch 361 | - Given: an owner of channel **𝘾𝒉** decides to alter permissions on **𝘾𝒉**, such as: 362 | - editing **𝘾𝒉**'s default access permissions to be more restrictive, _or_ 363 | - removing access permissions for explicitly named members, _or_ 364 | - making **𝘾𝒉** private and granting access to only specifically listed members of **C**. 365 | - Similar to [issuing a new member epoch](#Issuing-a-New-Member-Epoch), **𝘾𝒉**'s owner posts an entry to **𝘾𝒉** to revise the current `ChannelEpoch`. 366 | - If **𝘾𝒉** is private, a procedure similar to [issuing a new community epoch](#issuing-a-new-Community-Epoch) distributes **𝘾𝒉**'s latest access key to members that have at least read-only access to **𝘾𝒉**. 367 | 368 | #### Member Halt 369 | - Given: member **m** (or their private keyring) is potentially under the control or influence of an another. 370 | - A "Member Halt" refers to an automated sequence of actions performed on **m**'s behalf once it's believed that their personal keyring ("**[]km**") is under the influence of another. 371 | - The conditions/requisites needed in order to initiate a Member Halt on another's behalf can be arbitrarily based on security needs and situational circumstances. 372 | - A Member Halt on **m** could be initiated by: 373 | - **m**, upon discovering that another actor has gained access to **[]km**, _or_ 374 | - a peer of **m** (previously designated by **m**), upon receiving a message or signal of duress from **m**, _or_ 375 | - a community automated agent, noticing damning/malicious activity originating from a holder of **[]km**. 376 | - Once a Member Halt is initiated on **m**: 377 | 1. A special entry is posted to the [Member Epoch Channel](#member-epoch-channel), signaling to all nodes in **C** to defer all further entries signed by **[]km** 378 | - ⇒ this prevents any actor in possession of **[]km** from posting any entries as **m**. 379 | 2. A special transaction is submitted to **𝓛C**, immediately "burning" the ability of **m** (or any actor in possession of **[]km**) to post further transactions. 380 | - As this propagates across **𝓛C**, subsequent transactions signed by **[]km** will be rejected because post permission on **𝓛C** will no longer exist for **[]km**. 381 | - This removes an adversary's ability to vandalize **𝓛C** (e.g filling it with junk data). 382 | - For example, for **⧫C**, the transaction would send all **m**'s _C-Ether_ to address `x0`. 383 | 3. An admin, automated agent, or delegated member(s) would [issue a new community epoch](#issuing-a-new-Community-Epoch) for **C**. 384 | - Since newly issued community keys _aren't_ posted for halted members, any agent(s) in possession of **[]km** would lose all further read access to **𝓛C** since they would not have a key to extract the newly issued community key for **C**. 385 | 386 | 387 | 388 | #### Member Halt Recovery 389 | - Given: a [Member Halt](#member-halt) was issued on **m**. 390 | - Some time later, admin(s) or delegated members can review the situation: 391 | - When appropriate, **m**'s access is restored and new keys issued using a variant of [adding a new member](#Adding-A-New-Member). 392 | - In the case that an adversary in possession of **[]km** transfers their postage (their privileges on **𝓛C**) to another identity _before_ a Member Halt is issued on **m**, entries using postage from the illicit postage could be identified and rejected. 393 | - In the case that an a adversary in possession of **[]km** [issued a new member epoch](#issuing-a-new-Member-Epoch) (impersonating **m**), then an admin in communication with **m** would issue new entries that rescind the earlier entries as appropriate. As normal [Channel Entry Validation](#Channel-Entry-Validation) proceeds, this will automatically result in any dependent (adversary-authored) entries to be removed from "live" status. 394 | 395 | 396 | 397 | #### Adding A New Member 398 | - Given: the permissions/prerequisites on **C** are met to bestow member status to actor **α**: 399 | 1. A designated authority of **C** generates and posts a special `MemberEpoch`, **𝓔α0**, in the [Member Epoch Channel](#Member-Epoch-Channel), containing: 400 | - a newly generated `MemberID` for **α**. 401 | - newly generated public keys. 402 | - information on **α**'s invitation, such as: 403 | - documentation of which authorities in **C** were connected to **α**'s invitation. 404 | - proof that **α**'s invitation was granted by the collective authority of **C**. 405 | 2. Also created is token **τ**, containing: 406 | - a copy of **𝓔a0**, _and_ 407 | - a copy of the community keyring, **[]kC**, _and_ 408 | - the private half of public keys in **𝓔a0**, _and_ 409 | - network addresses and other bootstrapping information needed in order to connect to **𝓛C**, _and_ 410 | - a token that bestows its bearer postage on **𝓛C**. 411 | 3. **τ** is encrypted with a passphrase, and is passed to **α** via USB device, email, file sharing, etc. 412 | 4. Using face-to-face communication, direct contact, or other secure means, **α** is passed the passphrase to **τ**. 413 | 5. On a newly created "blank" node, **nα** (or an existing node of **C** in a logged-out state): 414 | 1. **α** passes **τ** to the client 415 | 2. the client prompts for the passphrase that decrypts **τ** 416 | 3. the client opens **τ** and now has what it needs to: 417 | - connect and write to **𝓛C** 418 | - decrypt entries from **C** and rebuild **𝓡α** normally 419 | 4. Once that **𝓡α** is up to date (i.e. once **𝓔α0** is live in the _Member Epoch Channel_), **α** posts a successor `MemberEpoch` as they would when [starting a new member epoch](#issuing-a-new-Member-Epoch). 420 | 421 | #### Deactivating A Member 422 | - Given: the admins and/or collective authority of **C** decide that member **m** is to be "deactivated", meaning that **m**'s access **C** is to be immediately rescinded. 423 | - The designated authority of **C** performs a procedure identical to the [Member Halt](#member-halt) procedure. In effect: 424 | 1. Any subsequent entries authored by **m** will be rejected, _and_ 425 | 2. **m** will be unable to post any transactions to **𝓛C**, _and_ 426 | 4. All new entries on **𝓛C** will be unreadable to **m** within [**ΔC**](#on-network-latency) 427 | 428 | 429 | #### Channel Entry Validation 430 | - Given node **ni** in **C**, let **𝓡i** denote the local replica state of **𝓛C** on node **ni** at a given time. 431 | - _Channel Entry Validation_ is the process of merging incoming entries arriving from **𝓛C** such that entries placed into "live" status (on **𝓡i**) comply and are in integrity with all relevant author and channel permissions and intent. 432 | - Since entries can arrive at **ni** in arbitrary order from **𝓛C**, entries will arrive whose validation will depend on other entries that have not yet been processed — _or entries yet to even arrive_. 433 | - As node **ni** processes an incoming entry **e** to be merged with **𝓡i**: 434 | - If **e** satisfies _all channel properties and parent ACC permissions_, then **e** is placed into "live" status. 435 | - Otherwise, if for whatever reason **e** cannot complete validation, then **e** is placed into "deferred" status. 436 | - Node **ni** periodically reattempts to validate **e** as other entries go live on **𝓡i**. 437 | - Unless **e** was crafted with malicious intent, it would be unexpected for **e** to remain indefinitely deferred. 438 | - For each new entry **e** arriving from **𝓛C** (or is locally authored and also submitted to **𝓛C**): 439 | 1. Authenticate **e**: 440 | 1. **edigest** ⇐ ComputeDigest(**e**`.CommunityKeyID`, **e**`.InfoCrypt`, **e**.`ContentCrypt`) 441 | 2. **einfo** ⇐ `EntryInfo` ⇐ Decrypt(**e**`.InfoCrypt`, **[]kC**.LookupKey(**e**`.CommunityKeyID`) 442 | - if the community key is not found, then **e** is deferred. 443 | 3. **𝓔author** ⇐ **𝓡i**.LookupMemberEpoch(**einfo**.`AuthorMemberID`, **einfo**`.AuthorMemberEpoch`) 444 | - if **𝓔author** = `nil`, then **e** is deferred. 445 | - if **einfo**`.TimeAuthored` falls outside the scope of **𝓔author**, then **e** is deferred. 446 | 3. **kpub** ⇐ **𝓡i**.LookupPublicKey(**einfo**.`AuthorMemberID`, **einfo**`.AuthorMemberEpoch`) 447 | - if **kpub** = `nil`, then **e** is deferred. 448 | 4. ValidateSig(**edigest**, **e**`.Sig`, **kpub**) 449 | - if **e**`.Sig` is invalid, then **e** is deferred. 450 | 2. Validate **e** in its destination channel: 451 | 1. **𝘾𝒉dst** ⇐ **𝓡i**.GetChannelStore(**einfo**.`ChannelID`) 452 | - if **𝘾𝒉dst** = `nil`, then **e** is deferred. 453 | 2. Check that **e** cites an agreeable `ChannelEpoch`: 454 | - **𝓔cited** ⇐ **𝘾𝒉dst**.LookupEpoch(**einfo**.`ChannelEpochID`) 455 | - if **𝓔cited** = `nil`, then **e** is deferred. 456 | - if **einfo**`.TimeAuthored` falls outside the scope of **𝓔cited**, then **e** is deferred. 457 | - if **𝓔cited**.CanAccept(**einfo**), then proceed, otherwise **e** is deferred. 458 | 3. Check that **𝘾𝒉dst**'s parent ACC permits **e**: 459 | - **𝘾𝒉AC** ⇐ **𝓡i**.GetChannelStore(**𝓔cited**.`AccessChannelID`) 460 | - **𝓅author** ⇐ **𝘾𝒉AC**.LookupPermissions(**einfo**.`AuthorMemberID`) 461 | - if **𝓅author** does not allow **einfo**`.EntryOp`, then **e** is deferred. 462 | 4. If **e** inserts a permissions change but does not [issue a new channel epoch](#issuing-a-New-Channel-Epoch) as required, then **e** is deferred. 463 | 3. Merge **e** into **𝓡i**: 464 | - **𝘾𝒉dst**.InsertEntry(**e**) 465 | 4. Propagate the mutation of **𝘾𝒉dst** as required ("revalidation"): 466 | - if **𝘾𝒉dst** is now _equally_ or _less_ restrictive, then `return` since dependencies that are live will be unaffected. 467 | - if **𝘾𝒉dst** is now _more_ restrictive, then revalidate dependent channels: 468 | - **trev** ⇐ **einfo**`.TimeAuthored` 469 | - **[]𝘾𝒉dep** ⇐ **𝘾𝒉dst**.GetDependentChannels(**trev**) 470 | - for each **𝘾𝒉j** in **[]𝘾𝒉dep**: 471 | - Scanning forward from **trev** in **𝘾𝒉j**, for each entry **ek**: 472 | - (re)validate **ek** (4 validate steps above) 473 | - if **ek** is now deferred: 474 | - **𝘾𝒉j**.RemoveEntry(**ek**) 475 | - Defer **ek** normally 476 | - Sub-propagate the mutation of **𝘾𝒉j** as applicable 477 | - Although there are edge cases where change propagation could result in a cascading workload, the amount of work is generally either n/a or negligible. This is because: 478 | - Most entries are content-related, not access-control related. 479 | - Revalidation only needs to proceed if an entry makes a channel _more_ restrictive. 480 | - For example, compare the number of ACL-related files stored on a conventional workstation to the _total_ number of files. 481 | - Only ACCs tend to have dependent channels. 482 | - Mutations to a channel occur close to the present time, so only O(1) of all entry history typically needs to be revalidated. 483 | - If **𝓛C** favors safety over liveness, then there a highly limiting trailing time boundary for how "late" an entry can arrive (e.g. 10 seconds). 484 | - Revalidation can also be strategically managed, where multiple ACC mutations are scheduled such that only a single revalidation pass is needed. 485 | 486 | 487 | 488 | --- 489 | 490 | ## Liveness vs Safety 491 | 492 | "Liveness versus safety" refers to canonical tradeoffs made during the design of a blockchain or distributed ledger. This discussion is variant of the [CAP theorem](https://en.wikipedia.org/wiki/CAP_theorem) applied to distributed ledgers, where "[liveness](https://en.wikipedia.org/wiki/Liveness)" corresponds to _availability_ and "[safety](https://en.wikipedia.org/wiki/Safety_property)" corresponds to _consistency_. Here, we weigh the tradeoffs made by a given **𝓛C** implementation: 493 | 494 | - If **𝓛C** favors _liveness over safety_ (such as [Ethereum](https://www.ethereum.org/) or [Holochain](https://holochain.org/)), then partitions of **𝓛C** will operate independently and will synchronize when rejoined. This implies: 495 | 1. [**ΔC**](#on-network-latency) will reflect network latency and topology. 496 | 2. [Channel Entry Validation](#Channel-Entry-Validation) could potentially encounter an important but late-arriving entry, triggering a cascade of entry revalidation. 497 | 3. The nodes of **C** are conveniently "offline-first" and will operate in independent cells if network connectivity is limited. 498 | - As partitions rejoin after some time and synchronize, each **𝓡i** will receive new batches of old transactions (from other partitions). 499 | - If **𝓛C** favors _safety over liveness_ (such as a central server, [EOS](https://eos.io/), [DFINITY](https://dfinity.org/), [Hashgraph](https://www.hedera.com/)), then **𝓛C** by definition integrates a consensus mechanism that enforces a trailing timestamp boundary ("**tb**") for transactions. This implies: 500 | 1. **ΔC** is helpfully fast (1-10 seconds) 501 | 2. [Channel Entry Validation](#Channel-Entry-Validation) has the luxury to finalize entries older than **tb** since later-arriving entries are not possible. 502 | 3. However, the nodes of **C** _require central network connectivity_ in order to operate. If nodes partition from the central network: 503 | - the **𝓛C** nodes will _**halt**, even though the nodes still have connectivity with each other_, and 504 | - the **𝓛C** nodes will _only resume_ if/when the partition regains central network connectivity. 505 | 506 | 507 | --- 508 | 509 | ## Proof of Specifications 510 | 511 | _Each item below corresponds to each item in the [Specifications & Requirements](#Specifications--Requirements) section_. 512 | 513 | 514 | #### Proof of Signal Opacity 515 | 516 | [Signal Opacity](#signal-opacity) asserts that outside actors can't infer material information by analyzing community wire traffic. 517 | 518 | - Given that (a) all community channel entries reside within transactions stored on **𝓛C**, _and_ (b) transactions are considered to be "in the clear": 519 | - What information is discernible to actors outside of **C**? 520 | - Let **α** be an actor that is _not_ a member of **C**, implying that **α** does not possess the latest community keys. 521 | - ⇒ the _only_ information directly available to **α** is the `UUID` of the encryption key used for each `EntryCrypt` on **𝓛C**. 522 | 1. ⇒ information opacity is maximized since all information resides within `InfoCrypt` and `ContentCrypt`. 523 | - Adversaries snooping **𝓛C** can only discern _when_ a new community security epoch began (by noting the appearance of a new community key `UUID`). However, this is weak information since such an event could correspond to any number of circumstances. 524 | 2. ⇒ _only_ members of **C** effectively have read-access to **C**'s content and member activity. 525 | - If **α** is a former member of **C**, then **α**'s access is limited to read-only up to when **α** was [deactivated](#deactivating-A-Member) (when the [new community epoch was issued](#issuing-a-new-Community-Epoch) as part of deactivating a member). 526 | 527 | 528 | #### Proof of Access Exclusivity 529 | 530 | [Access Exclusivity](#Access-Exclusivity) asserts that only members of **C** can read and write to the shared data store. 531 | 532 | - _Read-Access Exclusivity_ 533 | - Only an actor is possession of the community keyring ("**[]kC**") has the ability to read community-public content residing on **𝓛C**. See [Proof of Signal Opacity](#proof-of-signal-opacity). 534 | - _Write-Access Exclusivity_ 535 | - There are _three_ layers that prevent the unauthorized mutation of **𝓛C** or a community replica/repo ("**𝓡i**"): 536 | 1. **𝓛C Postage**, the mechanism ensuring that **𝓛C** will only accept storage transaction **txnj** if: 537 | - **txnj** bears an author that **𝓛C** recognizes as having permission to post a transaction of that size, _and_ 538 | - **txnj** bears a valid signature that proves the contents and author borne by **txnj** is authentic. 539 | 2. **Community Keyring Access**, referring to that _only_ community members are issued **[]kC**. 540 | - So even if actor **α** is able procure postage on **𝓛C**, they must also submit an `EntryCrypt` containing an `EntryInfo` encrypted using a recent community key — otherwise **𝓡i** will reject it. 541 | 3. **Channel Entry Validation**, referring to "deep" validation of entries arriving from **𝓛C**. Part of this flow is verifying that: 542 | - the signature contained in a given `EntryCrypt` is a valid signature from the member `UUID` borne by its `EntryInfo`, _and_ 543 | - the member is a valid/current member of **C** (based on **𝓡i**'s [Member Epoch Channel](#Member-Epoch-Channel)). 544 | - Given that the members of **C** maintain exclusive possession of their private keys, _only_ they can mutate **𝓛C** or **𝓡i**. 545 | - In the case where **m**'s private keys are possibly compromised, **m** would immediately initiate a [Member Halt](#member-halt), leaving any actor in possession of **m**'s keys challenged or unable to move past the above layers. 546 | - For in-depth security scenario analysis, see [Proof of Practical Security Provisioning](#Proof-of-Practical-Security-Provisioning). 547 | 548 | #### Proof of Permissions Assurance 549 | 550 | [Permissions Assurance](#Permissions-Assurance) asserts the access controls on **C** remain in effect and cannot be circumvented. 551 | 552 | - In order for a channel entry to be "live" in a node's repo ("**𝓡i**"), it must repeatedly survive [Channel Entry Validation](#Channel-Entry-Validation). 553 | - ⇒ each successive state of **𝓡i** is, exclusively, an authorized mutation from its previous state. 554 | - However, if an important entry is withheld from node **ni**, it is easy to imagine dependent entries piling up and **𝓡i** being at a standstill. 555 | - In addition to transactions arriving out of order from **𝓛C** naturally, we must consider if entries are altered, reordered, or withheld by adversaries manipulating communications signals or infrastructure. 556 | - We can rule out corruption or alteration of entries since each `EntryCrypt` bears a signature dependent on its contents, so altered entries would be immediately rejected. 557 | - The case where an adversary covertly has possession of a member's private keys is discussed in [Proof of Practical Security Provisioning](#Proof-of-Practical-Security-Provisioning). 558 | - So then, could the _reordering, blocking, or withholding_ ("withholding") of entries from **𝓛C** cause **𝓡i** to pass through a state such that access controls or grants established by members of **C** could be circumvented or exploited? We separate the possibilities into two categories: 559 | 1. **Unauthorized key, channel, or content access** 560 | - These scenarios are characterized by gaining unauthorized access to a privileged key that in turn allows read access to restricted content within **𝓡i**. 561 | - Since any withholding of entries can't result in the _additional_ generation/grant of permissions, the remaining possibility is that withholding entries somehow result in **𝓡i** not mutating such that privileged data somehow remains accessible. This is a legitimate concern, however, [Channel Entry Validation](#Channel-Entry-Validation) requires that any mutation that is access-restrictive in nature must also [issue a new channel security epoch](#issuing-a-new-Channel-Epoch). In this process flow, _only_ members with access privileges are (securely) issued the newly generated private channel key. Similarly, this is why a [new community epoch is issued](#Issuing-a-New-Community-Epoch) when a member is [deactivated](#deactivating-A-Member) or a [Member Halt](#Member-halt) is issued (but not when a [new member is added](#adding-a-new-member)). 562 | - ⇒ In the case where entries are _withheld_ from **ni**, there is no possibility that subsequent entries could reveal restricted material since new peer entries would be encrypted with a newly issued member, channel or community key (after **ΔC**) . 563 | 2. **Circumventing access controls** 564 | - These scenarios are characterized by posting maliciously crafted channel entries on **C** intended to circumvent access controls. 565 | - Given [Proof of Access Exclusivity](#Proof-of-Access-Exclusivity), we know that in order for **𝓡i** to be mutated (or maliciously manipulated), the perpetrating actor must possess _both_ an active member and community keyring. 566 | - ⇒ this would start with **m** (or an adversary covertly in possession of **m**'s keys) authoring and submitting said entries in an attempt to somehow sidestep an aspect of the system's channel permissions enforcement. How could an entry go live across **C** whose parent ACC denies access? [Channel Entry Validation](#Channel-Entry-Validation)'s primary purpose is to _ensure that **only** entries that validate under the parent channel's ACC are placed into "live" status_. 567 | - What if **m** (or an adversary in possession of **m**'s keys) "hot-wires" their node or manually crafts an entry to perform an operation not allowed by **m**? 568 | - Although **m** would be able to submit doctored entries to **𝓛C**, [Channel Entry Validation](#Channel-Entry-Validation) running on _every other_ community node would see that **m** does not have the required privileges and would indefinitely defer (reject) the entry. 569 | - This is analogous to submitting a transaction on the global Bitcoin or Ethereum blockchain that transfers coinage from an ID that does not have sufficient funds — the transaction will never validate. 570 | 571 | #### Proof of Accountability Assurance 572 | 573 | [Accountability Assurance](#Accountability-Assurance) asserts that members who exercise authority are accountable and remain bound to community policies and expectations. 574 | 575 | - Every member action (mutation) on **C** is manifested as a channel entry, whether that is a routine content entry or a specially-signed entry in the [Member Epoch Channel](#Member-Epoch-Channel). 576 | - ⇒ every member interaction (channel entry) is replicated across **𝓛C**, an append-only storage layer, and available for review _to all other members of **C**_. 577 | - ⇒ entries are immutable on **𝓛C** and any attempt to conceal or rescind an entry will not alter the original entry. 578 | - ⇒ all actions in community-public channels, which includes all community administrative [reserved channels](#reserved-channels), are always openly visible for peer review and scrutiny. 579 | - In private channels, entry content is encrypted and _only_ members granted access by the channel's owner can read the channel's content. 580 | - By default, _not even community admins or authorities_ can gain access to private channel content unless they have been granted access. 581 | - Through private channel content is cryptographically secure, some metadata is community-public information — namely, who is generating activity, with whom, and how often. 582 | - Although the actions of a member in private channel **𝘾𝒉p** cannot be witnessed or reviewed by members outside of that channel, _any participant_ of **𝘾𝒉p** could be pressured by community authorities to turn over the channel's keyring (or face [deactivation](#deactivating-a-member]) or social/legal repercussions). 583 | - Community authorities are free to make use of private channels (just like other community members), but any action affecting reserved or community-public channels would be publicly available information and would be part of the permanent record that is **𝓛C**. 584 | - Important channels, such as [reserved channels](#reserved-channels) or channels used to conduct community governance, could harness the dependable and predictable nature of "smart contracts" on **𝓛C**. For example: 585 | - **C** has a _special_ community admin account capable of executing community propositions. This virtual agent (and its private keys) are wired into a smart contract on **𝓛C** such that ⅔ of **C**'s "council" members are needed to sign a proposition before the contract's "threshold" signature triggers the proposition to be executed with admin permissions. 586 | - **C** makes use of a channel UI extension that in effect recreates a voting booth, allowing the members of **C** to vote on community propositions. Corresponding smart contracts on **𝓛C** would be wired into these channels such that community members would have full assurance that voted propositions are executed reliably and predictably. 587 | 588 | 589 | #### Proof of Membership Fluidity 590 | 591 | [Membership Fluidity](#Membership-Fluidity) asserts that members can be added and deactivated at any time. 592 | 593 | - Both [Adding a New Member](#adding-A-New-Member) and [Deactivating a Member](#deactivating-a-member) procedures are implemented using standard entries in **C**'s channel system. These entries, like all other channel entries, undergo [Channel Entry Validation](#Channel-Entry-Validation) plus _additional_ checks and restrictions. 594 | - ⇒ Membership Fluidity is just a specific form of [Permissions Assurance](#Permissions-Assurance) and is thus addressed in [Proof of Permissions Assurance](#Proof-of-Permissions-Assurance). 595 | - ⇒ all the properties, assurances, and protection afforded by Channel Entry Validation extends to the ability for community authorities to add and deactivate members. 596 | 597 | 598 | #### Proof of Strong Eventual Consistency 599 | 600 | [Strong Eventual Consistency](#Strong-Eventual-Consistency) asserts that community replicas eventually arrive at the same state and are independent of network delivery. 601 | 602 | - For each node **ni** in **C**, it's local replica state ("**𝓡i**") at a given time is dependent on: 603 | - The set and order of entries that have arrived from **𝓛C** and have been authored locally. 604 | - [Channel Entry Validation](#Channel-Entry-Validation) reviewing newly arrived entries and entries previously deferred. 605 | - Given that natural or adversarial network conditions could cause entries to and from **𝓛C** to be delayed or withheld: 606 | - Once _all_ entries replicated across **𝓛C** _eventually_ arrive at **n1**...**nN**, are **𝓡1**...**𝓡N** _each_ in an equivalent ("consistent") state? 607 | - This system's [Proof of Permissions Assurance](#proof-of-Permissions-Assurance) implies that the eventual state of **𝓡i** is _independent_ of the order of arrival of entries from **𝓛C**. 608 | 609 | 610 | #### Proof of Practical Security Provisioning 611 | 612 | [Practical Security Provisioning](#Practical-Security-Provisioning) refers a system's inherent ability to address real-world security incidents. Although this section is intended to convincingly analyze how this system provisions for severe security scenarios, we acknowledge that the more severe and multifaceted the scenario, the more complex and divergent a complete analysis becomes. Given member **m** in **C**, this system provisions for: 613 | 614 | 1. _A scenario where..._ **m** loses/erases all copies of their private keyring ("**[]klost**"). 615 | - A community admin (or delegated member) would use a procedure similar to [adding a new member](#Adding-A-New-Member), resulting in a successor `MemberEpoch` to be issued for **m**. 616 | - ⇒ **m** would fully retain and resume their identity in **C**, however **m** would lack the private keys needed to access their private channels. 617 | - Fortunately, for each private channel **𝘾𝒉p** that **m** had access to, **m** could regain access if _at least_ one other member has _at least_ read-access to **𝘾𝒉p** (allowing **m** to successfully petition for the channel's keyring). 618 | - This could be automated and implemented in various ways, but **m** would not regain keys to private channels if the members are no longer active (since their client would need to be active to package keys into an entry in response to **m**'s petition). 619 | - If **m** were to someday recover **[]klost**, access to past data would be fully restored at no disadvantage. 620 | - If an adversary were to somehow recover **[]klost** in an _unlocked state_, they would, at most, have read-access up to the time when **m**'s new `MemberEpoch` was published. 621 | - Entries signed by **[]klost** and processed by **𝓡i** would be rejected since the `MemberEpoch` associated with the latest private key in **[]klost** would be superseded. 622 | 2. _A scenario where..._ an adversary ("**O**") gains access to **m**'s private keys ("**[]km**") through deception, coercion, or covert access to **m**'s client device. 623 | - as expected, **O** would be able to: 624 | - read all community-public data on **C** 625 | - author entries impersonating **m** 626 | - read content intended to be private for **m** 627 | - When **m**, a peer of **m**, an admin, or an automated notices something is amiss, they would initiate a [Member Halt](#member-halt) on **m**. As part of the Member Halt: 628 | 1. **O** would lose append/post access to **C** (twofold: no postage _and_ **m**'s `MemberEpoch` suspended), _and_ 629 | 2. **O** would lose further read access to **𝓛C** once a [new community epoch is issued](#Issuing-a-New-Community-Epoch). 630 | - In the case that **O** [issues a new member epoch](#issuing-a-new-Member-Epoch) (impersonating **m**) _before_ a Member Halt is issued: 631 | - **m** would _still_ be able to issue a [Member Halt](#member-halt), _and_ 632 | - an authority within **C**, in communication with **m**, would later rescind any entries in the [Member Epoch Channel](#member-epoch-channel) authored by **O**. 633 | 3. _A scenario where..._ a dishonest community authority ("**O**") within **C** covertly wishes to snoop on **m**. 634 | - At no point would **O** have had access to **[]km** since **m**'s private keys never leave **m**'s client process space by design. 635 | - ⇒ **O** has _no_ ability to gain access to content encrypted for **m**, including content in **m**'s private channels. 636 | - In the event that **O** posts a new `MemberEpoch` for **m** without permission, **m** would immediately become aware once **m**'s client sees the epoch issued. 637 | - Hijacking another's identity as in this case would not allow the perpetrator to gain access to any of **m**'s private data since only **m** has possession of **[]km**. 638 | - ⇒ **O** would only gain the ability to impersonate **m** only as long as **m** remains offline. 639 | 4. _A scenario where..._ multiple adversaries covertly infiltrate **C**. 640 | - Plan A: _Surgical [Member Halt](#member-halt) Attack_ 641 | - Since a member halt only suspends a member's access to **C**, its utility as an attack vector is limited to a one-time DoS for the targeted member. 642 | - This attack can be quickly be undone by an admin, and the offending member's integrity would immediately move under a spotlight. 643 | - Limitations could be added that would guard against adversarial behavior. Examples: 644 | - a member could be limited to ordering one Member Halt per day 645 | - members could limit _whom_ could order a halt on them, limiting anonymous misuse 646 | - Plan B: _Vandalization of **C**_ 647 | - The authorities of **C** could: 648 | 1. manually rescind the offending entries. 649 | 2. retroactively deactivate the `MemberEpoch` of the offending members, resulting in a _revalidation cascade_ as [Channel Entry Validation](#Channel-Entry-Validation) propagates the change. 650 | - Each dependent entry would automatically be placed into rejected status on each community node during propagation. 651 | 3. hard fork **𝓛C** to an earlier state if the vandalism was pervasive (e.g. though postage guards against this, gigabytes of junk appended to **𝓛C**). 652 | - [Proof of Independence Assurance](#Proof-of-Independence-Assurance) describes variations of this recovery flow. 653 | 654 | 655 | #### Proof of Independence Assurance 656 | 657 | [Independence Assurance](#Independence-Assurance) asserts that any member subset of **C** can independently fork **C** with a replacement governance or leadership structure. 658 | 659 | - Suppose some members of **C** decide, for whatever reason, that they are better off in their own community ("**C′**"), with their own pact of governance or leadership. 660 | - They desire **C′** to be the equivalent of **C** but _only up to a given time_ ("**tC′**") — at which time the authority structure or membership is arbitrarily altered. 661 | - Let **g** be the genesis authority of **C** at time of its creation. 662 | - Let **[]a** be the admins of **C** at time **tC′** 663 | - Let **[]a′** be the founders (and admins-to-be) of **C′** 664 | - Given that **𝓛C** is append-only CRDT, each transaction is assumed to have a timestamp and identifying ID. 665 | - ⇒ **𝓛C** is characterized as a set of sealed data transactions and can be partitioned at any given time index. 666 | - The founders of **C′** do the following: 667 | 1. Instantiate a new CRDT ("**𝓛C′**") and allocate bulk postage identically to **𝓛C**'s genesis _in addition to_ allocating bulk postage to **[]a′**. 668 | 2. Copy the parameters from **C**'s genesis _in addition to_ grant admin status to **[]a′**. 669 | 3. Transfer entries from **𝓛C** to **𝓛C′** up to time **tC′** (omitting entries as desired). 670 | - Allocations from step (1) ensure that transactions copied from **𝓛C** and posted to **𝓛C′** will clear. 671 | 4. Once is **𝓛C′** updated up to desired time, **[]a′** can effectively assert control: 672 | 1. They demote or deactivate **[]a** (and any other desired members), _and_ 673 | 2. For each member demoted/deactivated, reduce/burn their postage allocation on **𝓛C′** as appropriate. 674 | - ⇒ **C′** is free to operate independently of **C** and under the authority of **[]a′**. 675 | - Although **C′** is now operating under altered governance or leadership, the privacy of all legacy encrypted entry content is preserved. Otherwise, members from **C** would have had to disclose their private keys. 676 | 677 | #### Proof of Storage Portability 678 | 679 | [Storage Portability](#Storage-Portability) asserts that **C** is not permanently bound to its storage layer implementation. 680 | 681 | - Suppose **C** wishes to switch to an alternative CRDT technology. 682 | - Using a simplified form of the steps listed in [Proof of Independence Assurance](#proof-of-independence-assurance), **C** can coordinate a switch to a new CRDT medium almost transparently. 683 | - This is to say that **𝓛** is to **C** as a hard drive is to an operating system. **C**'s "hard drive" can be replaced: 684 | - with the an updated model, _or_ 685 | - with a different "brand", _or_ 686 | - with a different storage technology all together. 687 | 688 | 689 | --- 690 | 691 | ## On Network Latency 692 | 693 | In any system, replicated data transactions and messages take non-trivial periods of time to traverse and propagate across the network. Also, any number of nodes could be offline for indefinite periods of time. No assumptions are made about network connectivity or reachability in this proof, and propagation times are expressed as "**ΔC**": 694 | 695 | - Let **ΔC** be the time period needed for there to be at least a 99.7% chance that all _reachable_ nodes in **C** have received a given replicated transaction across **𝓛C**. 696 | - **ΔC** serves as a "rule-of-thumb" for characterizing information latency within **C**. It allows us to more fully express aggregate performance, decision making calculus, and case-study analysis. 697 | - Although **ΔC** depends on how aggressively **𝓛C** trades off redundancy with bandwidth conservation, **ΔC** generally tracks with `log(N)`, given `N` "well-connected" nodes. 698 | - ⇒ Even a "dumb" peer-based network has attractive latency characteristics as N increases at human scales. For example: 699 | - Consider a collection of N=1000 well-connected nodes, having: 700 | 1. a standard distribution of node neighbor latency centered at .2 sec (with σ = .1 sec) 701 | 2. a standard distribution of "swarm diameter" centered at 10 hops (with σ = 5 hops). 702 | - ⇒ **ΔC** is in the neighborhood of (.2 + 6 * .1) * (10 + 6 * 5) secs (about 1 minute). 703 | - If N is _doubled_, then **ΔC** _only_ increases a couple seconds since the limiting swarm diameter increases by 2-3 hops. 704 | 705 | Like the way an operating system is _only_ as swift as its host storage system, the latency and "liveness" (availability) of the system presented below is solely dependent on **𝓛**. This means that the design tradeoffs that **𝓛C** makes will determine **C**'s overall network properties and behavior. [Liveness vs Safety](#liveness-vs-safety) discusses tradeoffs for various choices of **𝓛**. 706 | 707 | --- 708 | 709 | Back to [README](README.md) -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # PLAN Design Docs 2 | 3 | ## Welcome to PLAN! 4 | 5 | PLAN is a software platform for communications, group collaboration, and spatial planning. PLAN leverages modern feats of engineering in distributed crypto systems, 3D graphics, and "trustless" protocols. It features: 6 | - Built-in collaboration and visualization tools for teams managing projects, organizations, and relationships 7 | - Integrated messaging, file sharing, interactive maps, tasking, scheduling, and logistics operations 8 | - Storage decentralization for risk mitigation and offgrid operation 9 | - An intuitive and visual 3D user interface where channels, files, and links are arranged in virtual spaces 10 | - End-to-end security, total data ownership, total data privacy, and mission-critical reliability 11 | - A [pluggable](PLAN-API-Documentation.md#Primary-Interfaces) architecture designed for extension and flexibility 12 | 13 | ![alt text](https://github.com/plan-systems/design-docs/blob/master/PLAN-Elements-and-Design-Principles.jpg "PLAN Elements and Design Principles") 14 | 15 | ## Why PLAN? 16 | 17 | Distributed technologies offer astonishing potential, but they lack a consistent and accessible graphical user experience. Distributed ledgers and other peer-based technologies are ripe to be integrated into a _unified_ visual interface that prioritizes human relationships and usability. Such a system must be resistant to vandalism, outside interference, and malicious actors. As an information visualizer, PLAN allows teams to communicate and conduct logistics planning efficiently, reliability, and privately. PLAN offers assurance that the information we store and depend on will be available not only during times of prosperity, but also during natural disaster, political crisis, or economic drought. 18 | 19 | PLAN is ideal for: 20 | - Community organizers 21 | - Educators and researchers 22 | - Entrepenuers and businesses 23 | - Journalists and media productions 24 | - Civic institutions 25 | - Trust networks and governance structures 26 | - Teams, clubs and gaming groups 27 | - Human development and support groups 28 | 29 | 30 | ## What's in This Repo? 31 | 32 | This repo presents and discusses the layers, abstractions, and technologies that comprise PLAN. It is written for a technical audience ready to understand and vet PLAN's architecture and design. We recommend that visitors read this document and then explore our other supporting documents: 33 | 1. [FAQ](#FAQ) 34 | 2. [API Documentation](PLAN-API-Documentation.md) 35 | 3. [PLAN Data Model Proof of Correctness](PLAN-Proof-of-Correctness.md) (whitepaper) 36 | 37 | ## About PLAN Systems 38 | 39 | [PLAN Systems](http://plan-systems.org) is a non-profit public charity, developing and providing publicly available systems that foster robust digital self-reliance for low resource communities, while also reducing the burden on local, state, and federal agencies. 40 | 41 | May PLAN empower organizations and individuals, and may it be an instrument of productivity and self-organization. 42 | 43 | ## Architecture Objectives 44 | 45 | The primary objective of PLAN's architecture and user interface is to simplify the complex nature of digital privacy and distributed systems using interactive visual idioms that blend into the user experience as seamlessly as possible. In most cases, it’s not particularly important to an end-user where exactly data resides, how it's served, or how encryption works — _but that it's intuitive and visual_. 46 | 47 | Instead of a constrained and sandboxed web browser experience, a PLAN user experiences their organization's structure and content through the [Unity](https://unity3d.com) graphics engine as it renders information organized in "channels" into virtual space. The client is served by a "pnode" that uses a storage abstraction compatible with many existing [distributed ledgers](https://en.wikipedia.org/wiki/Distributed_ledger). 48 | 49 | _For a visual aid, see the [PLAN Network Configurations Diagram](https://i2.wp.com/www.plan-systems.org/wp-content/uploads/2019/07/PLAN_networkconfig_diagram.jpg?ssl=1)._ 50 | 51 | PLAN design goals: 52 | - Multiplatform: _Android, iOS, Linux, macOS, Windows_ 53 | - Space channels can be experienced in 2D/orthographic, first person, isometric, or AE/VR 54 | - Peer-to-peer [persistent storage abstraction](PLAN-API-Documentation.md#persistent-data-interface) compatible with distributed ledgers 55 | - Peer-to-peer [cloud storage abstraction](PLAN-API-Documentation.md#cloud-file-interface) compatible with distributed storage systems 56 | - Pluggable content handling and rendering via [channel GUI adapters](PLAN-API-Documentation.md#channel-gui-adapters) 57 | - Offgrid operation to allow work when only part of the network reachable 58 | - Access controls for groups as a whole or individual members 59 | - Allow a community to peer-serve content to visitors 60 | 61 | PLAN users get: 62 | - A responsive, engaging, and visual experience 63 | - A first-class input and display device experience 64 | - Full horsepower of the device/workstation 65 | - Total data ownership and privacy 66 | 67 | How is PLAN implemented? 68 | - User client written in [Unity](https://unity3d.com) 3D engine (C# and .NET Core) 69 | - Peer-to-peer backend daemon ("pnode") written in [Go](https://golang.org) 70 | - A sequence of channel entries that are interpreted and visualized based on the channel's protocol identifier 71 | - Access control implemented also using channels; each channel is a child of some other channel, up to a root. 72 | - Local file system caching and partial network resilience to enable offgrid work 73 | 74 | 75 | ## Architecture Synopsis 76 | 77 | PLAN has a 3D/visual frontend with a peer-to-peer backend that uses one or more secure distributed storage providers. The PLAN client, written in [Unity](https://unity3d.com), connects to a "community" peer-to-peer PLAN node, a daemon written in [Go](https://golang.org). These nodes serve PLAN clients while replicating shared community data across the community's federation of nodes. 78 | 79 | What defines a community? In PLAN, a community is designed to reflect the human relationships that make up the community, whether that's a household, neighborhood, first-responders unit, off-grid farm, city council, media production, veterans network, maker-space, artist collective, emotional support network, small business, or gaming group. The entire community's network traffic and infrastructure is inaccessible to all others, forming a cryptographic _city wall_ of privacy and security. 80 | 81 | A PLAN community is formed when one or more founders/organizers gather and together use the community genesis tool that codifies initial governance and administration. In that process, they also decide which of the available [StorageProviders](PLAN-API-Documentation.md#persistent-data-interface) implementations are best for their needs. After that, the founders and subsequent members of the community maintain a copy of the "community keyring", giving members the ability to access the community's shared data repository. Anyone _without_ this keyring — anyone _not_ in the community — is _outside_ the community's crypto city wall. 82 | 83 | Inside this outer layer of security, residing on each community node, lives an IRC-inspired channel infrastructure. Each PLAN channel entry contains content and metadata that works like HTTP headers, allowing content to be richly interpreted. But a channel's "protocol identifier" also contributes to describing how entry content should be interpreted. 84 | 85 | When a PLAN channel is created, it is assigned a protocol identifier string (e.g. `/plan/channel/chat`). This [channel protocol](PLAN-API-Documentation.md#channel-protocols) implies the _kind_ of entries that are expected to appear in a channel and _how_ they should be interpreted and presented within the PLAN client. In the client, each channel type string maps to a channel GUI adapter or "driver" that is invoked when the user opens/views a channel, like when an OS opens a specific application to handle a given file type. A power user can set their client to invoke alternative channel adapters that may be more appropriate for their situation, client device, or personal taste. This leaves exciting possibilities for custom channel design and development. As PLAN grows into its core mission and vision, primary development is going towards channels that: 86 | - facilitate group communication and workflows 87 | - track tasks and task status 88 | - build interactive maps, charts, floor plans, and virtual spaces 89 | - track inventory and supplies 90 | - integrate files and media sharing 91 | - tally and track community polls and voting 92 | - coordinate and visualize calendars and scheduling 93 | - accept and filter custom-designed forms 94 | 95 | In addition to the entry protocol a channel is assigned, a PLAN channel is _also_ assigned an owning access control channel (ACC) that specifies channel permissions, limits, and behaviors. A channel's controlling ACC, like all channels, also cites its own controlling ACC, and so on — up to the community's root ACC. A community's root ACC, is one of several "hard wired" channels that serve core community functions and can only be altered by community admins. Another such channel, for example, is the community registry channel, containing the member ID and public keys of each community member. Functions such as member key regeneration and key exchange are carried out through community channels explicitly reserved for these purposes. 96 | 97 | Akin to IRC, PLAN channels are either "community-public" (readable only to members in the community), or they are private where entry content is encrypted such that only channel members have access. Only community members that have explicitly been given a private channel's key have the ability to decrypt its content. And although channels are fundamentally append-only, channels can be set so that new entry content can replace past content, allowing past entries to be edited (though past entries will naturally remain). The flexibility of a channel's protocol identifier plus the rich and flexible nature of PLAN's [interoperable data structures](PLAN-API-Documentation.md#Interoperable-Data-Structures) make PLAN's channel system dynamic and extensible — designed to be represented and interacted with via a local, graphical, high-performance interface. 98 | 99 | At any given time, each community node: 100 | - Merges newly appearing community channel entries from the storage layer into the community repo layer 101 | - Hosts connected Unity clients, serving client channel queries and decrypting content on the fly 102 | - Serves as a public HTTP gateway for community content designated to be publicly served (e.g. a public-facing web page containing a peer-served promo video) 103 | 104 | The permissions and rules of merge conflict resolution are deterministic so that strong eventual consistency (SEC) is preserved. This means that although a given node's data state may not be equal to other community nodes state (due to network constraints), each node is guaranteed to converge to a monotonic state. 105 | 106 | PLAN has two persistent pluggable storage layers, one characterized by append-only operations and the other characterized by content-based addressing. The former, the Persistent Data Interface (PDI) is used to host a community's channel data and is intentionally designed to be compatible with the append-only nature of blockchain storage. The latter, the [Cloud File Interface](PLAN-API-Documentation.md#cloud-file-interface) (CFI), is used to support a community's bulk data storage needs, while PLAN's channel protocols and GUI wrap hashnames and other implementation details that no one wants to see or interact with. For example, a channel of type `/plan/channel/file/cfi/video` is used as a wrapper whose entries are CFI pathnames to each successive revision of that file. This schema affords: 107 | - PLAN's deterministic infrastructure to know which CFI items are in use ("pinned") and which can be unpinned/deallocated. 108 | - Seamless UI integration and interactivity. In the client UI, a channel's wrapper identifier causes it to be presented as a single opaque object (like a traditional file), where its activation causes the latest revision to be fetched and consumed. This allows users to easily access community content while not having to have any understanding about what's happening under the hood (or having to interact with hashnames). 109 | 110 | ## Summary 111 | 112 | PLAN is a peer-to-peer community-first tool, built on append-only and pluggable content-based addressing storage. Community content is accessible via a _real-time_ visually intuitive interface protected inside cryptographic layers of privacy. Its open-ended channel sub-systems form a level foundation that supports flexible, defensible, and first-class human access. 113 | 114 | Using PLAN, communities arise from organizers and members who value owning their own data, having a formidable cryptographic-city wall, and the ability to continue operating in the face of Internet disruptions. 115 | 116 | --- 117 | 118 | 119 | 120 | 121 | 122 | # FAQ 123 | 124 | 125 | #### Q: Why PLAN? Aren't there enough blockchain and distributed ledgers already? 126 | - Indeed, there are many advanced distributed ledger projects available and new ones on the way. However, _PLAN is not a distributed ledger_. PLAN is an information organization and access control system that _resides atop a storage system_. 127 | - Consider: _PLAN is to distributed ledger as operating system is to harddrive_. If a more capable or suitable storage technology appears, PLAN's [Proof of Storage Portability](PLAN-Proof-of-Correctness.md#Proof-of-Storage-Portability) demonstrates how a community can switch storage technologies. 128 | 129 | #### Q: What are the ways PLAN is pluggable or can be otherwise be extended? 130 | - PLAN features an architecture intended for modularity, flexibility, and specialization. PLAN's [Primary Interfaces](PLAN-API-Documentation.md#Primary-Interfaces) span from storage layers, to PLAN GUI plugins called Channel Adapters that allow a channel to be experienced differently. 131 | 132 | #### Q: Where is a community data stored and who ultimately controls it? 133 | - The [Persistent Data Interface](PLAN-API-Documentation.md#Persistent-Data-Interface) is intended for _permanent_ community storage and should be regarded at the community's channel repository. 134 | - The [Cloud File Interface](PLAN-API-Documentation.md#cloud-file-interface) is an abstraction that exposes a distributed storage system, used for bulk data storage. It pairs well with the PDI since a channel entry can reference a CFI item using only a short string. 135 | - In both cases, _the community stores and controls its own data_. By design, a PLAN community is not dependent on storage or infrastructure outside the community (other than whatever third-party telecommunications infrastructure is needed) 136 | 137 | #### Q: But PLAN doesn't do X, fulfill need Y, or address use case Z. How will PLAN address this? 138 | - PLAN is not meant to be _all_ things to _all_ people. PLAN is intended for micro-sized to medium-sized organizations and groups that have few or no options when it comes to a multi-platform, secure, and integrated operations platform. PLAN is all about offering a reliable and easy-to-use logistics and planning tool for ad-hoc groups or under-resourced organizations. 139 | 140 | #### Q: Does a PLAN community admin wield all the power and control? 141 | - Not unless you want it that way. The phrase "community admin" is used in these docs to refer to an agent acting in accordance with community policies and bylaws on behalf of the community's already-established leadership. This means that a community can operate as strictly or as loosely as the founding members want, but those agreements are visible to the entire community. 142 | - For example, community **C** could be founded such that a majority vote from a member-appointed set of "board" members are required in order to add a new member to the community. This would be enforced by a smart contract wired into **C**'s storage layer. See PLAN's [Proof of Accountability Assurance](PLAN-Proof-of-Correctness.md#Proof-of-Accountability-Assurance) for more. 143 | 144 | #### Q: Is PLAN is locked into Unity? 145 | - Although [PLAN Systems](http://plan-systems.org) is making the primary PLAN client with [Unity](https://unity3d.com/), we would fully support development of a client made with [Unreal](https://www.unrealengine.com), [CRYENGINE](https://www.cryengine.com/), [Godot](https://godotengine.org/) or any other established 3D framework. 146 | 147 | #### Q: Can multiple PLAN communities interact or federate? 148 | - There are many interesting areas of research and future development related to how independent PLAN communities can interact and be allies. Open research areas include: 149 | - Peer Federations — a community's leadership grants privileges to another community's members en masse. 150 | - Hierarchical Federations — a community originates as subordinate in some way to parent community. 151 | - Commerce Federations — multiple communities maintain shared identity or marketplace federation infrastructure, facilitating trade and commerce. 152 | 153 | #### Q: How can I try PLAN or support its development? 154 | - Check out the [PLAN website](http://plan-systems.org) and fill out our contact form. This will allow you hear about announcements and upcoming releases. 155 | 156 | --- 157 | --- 158 | -------------------------------------------------------------------------------- /SETUP.md: -------------------------------------------------------------------------------- 1 | # Setup 2 | 3 | This document describes how to set up a PLAN development environment. 4 | 5 | * [Contributing to PLAN](#contributing-to-plan) 6 | * [Contributing to a PLAN component](#contributing-to-a-plan-component) 7 | * [Developing a new PLAN component in Go](#developing-a-new-plan-component-in-go) 8 | * [Advanced](#advanced) 9 | * [Creating a Multi-Repo Workspace](#creating-a-multi-repo-workspace) 10 | * [Developing new server-side language bindings](#developing-new-server-side-language-bindings) 11 | 12 | 13 | ## Contributing to PLAN 14 | 15 | PLAN uses Go modules, so you must be using Go version 1.11 or above and you must not have a `GOPATH` set. Nor should you clone PLAN repos you're working on into `~/go/src`. Each of PLAN's repos contains a Makefile to test and build. 16 | 17 | 18 | ### Contributing to a PLAN component 19 | 20 | The easiest way to contribute to PLAN is to provide bugfixes or new features for an existing PLAN component. 21 | 22 | First, click the "Fork" button on GitHub for the project you want to work on. Then clone your fork locally to wherever you'd like to work. 23 | 24 | ``` 25 | cd ~/src/PLAN # or wherever you'd like to work 26 | git clone git@github.com:/plan-pdi-local.git 27 | ``` 28 | 29 | Once you've done so, you can use Makefile targets to test, build, or cross-compile for distribution. 30 | 31 | ``` 32 | make check # lint and test 33 | make build # build for your architecture 34 | make release # build for all supported architectures and create tarballs 35 | ``` 36 | 37 | When you're ready to contribute your changes back, open a Pull Request on the project. 38 | 39 | 40 | ### Developing a new PLAN component in Go 41 | 42 | If you want to create a brand new PLAN component written in Go (such as a new PDI provider), you'll want to use [`plan-go`](https://github.com/plan-systems/plan-go). 43 | 44 | Create your new repo and initialize the Go modules system: 45 | 46 | ``` 47 | git init my-component 48 | cd my-component 49 | go mod init 50 | ``` 51 | 52 | Then import the specific subpackages you'll need for your application: 53 | 54 | ```go 55 | import ( 56 | "github.com/plan-systems/plan-go/vault" 57 | "github.com/plan-systems/plan-go/repo" 58 | "github.com/plan-systems/plan-go/ctx" 59 | ) 60 | ``` 61 | 62 | 63 | ## Advanced 64 | 65 | 66 | ### Creating a Multi-Repo Workspace 67 | 68 | If you are working on a large feature that might have cross-repository concerns, you should set up a Multi-Repo Workspace. This is how the core PLAN team sets up their workspace. This workspace will have the PLAN repositories cloned side-by-side somewhere on your file system, and only use Go modules support for third-party dependencies. 69 | 70 | ``` 71 | cd ~/src/PLAN # or wherever you'd like to work 72 | 73 | # clone the set of repos you'd like to work with 74 | for repo in {plan-go,plan-protobufs,plan-client-unity}; 75 | do 76 | git clone git@github.com:plan-systems/${repo}.git 77 | done 78 | ``` 79 | 80 | Now we're ready to work on a specific project. Each project has a Makefile target `make hack`, which uncomments the `replace` directives in the `go.mod` file so that the cross-repo dependencies point to the repo you have on your file system instead of the one on GitHub. The Make target also tells git to ignore changes to `go.mod` and `go.sum`. 81 | 82 | ``` 83 | cd plan-pdi-local 84 | make hack 85 | ``` 86 | 87 | When we're done, we can `make unhack` to comment-out the `replace` directives and tell git to stop ignoring changes to `go.mod` and `go.sum`. 88 | 89 | 90 | ### Developing new server-side language bindings 91 | 92 | PLAN uses [protobufs](https://developers.google.com/protocol-buffers/) for communication between components. The protobufs are defined in the [`plan-protobufs`](https://github.com/plan-systems/plan-protobufs) repo. This allows you to write a PLAN component in any language. 93 | 94 | Currently the PLAN team is building all server components in [Go](https://golang.org/). The core functions for PLAN in Go are defined in the [`plan-go`](https://github.com/plan-systems/plan-go) repo. If you wanted to build a component in another language, you could use the protobufs in `plan-protobufs` to create your own language bindings and implement the functionality in `plan-go` as a community project. 95 | -------------------------------------------------------------------------------- /hash-collision-odds.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/python 2 | # 3 | # Inspired by http://preshing.com/20110504/hash-collision-probabilities/ 4 | # 5 | # The purpose of this script is to print out the hash collision odds of various hashspace sizes, 6 | # also known as the "birthday problem" in another form. See above link. 7 | # 8 | # For example, in a hashspace size of 20 bytes, the chances of a hash collision occurring if you 9 | # generate 10^17 random hashes is about the odds of a meteor hitting your home (1 in 10 trillion). 10 | # In software design, this question appears as: what hashspace size do I need in order to effectively 11 | # rule out a hash collision, given that I expect there to be N "draws" in the hashspace and I want 12 | # the odds to be no worse than P. "Draw" is a reference to playing cards, where drawing cards 13 | # to form a hand is an attempt at what is possible. 14 | # 15 | # Points of reference: b10 b2 b64 16 | # ---------------------------------------------------------- 17 | # match 6 lottery combos 1e7 2^23 18 | # seconds in a century 3e9 2^31 1/2 19 | # seconds in all recorded history 1e11 2^36 20 | # seconds in a billion years 3e16 2^55 21 | # seconds since Earth formed 2e17 2^57 22 | # seconds since Big Bang 4e17 2^58 23 | # millimeters in a light year 1e18 2^60 24 | # grains of Sand on earth 1e19 2^64 1 25 | # drops of water on earth 1e26 2^85 26 | # protons in your body 1e28 2^93 27 | # diameter of the universe in mm 1e30 2^100 28 | # planck units length of universe 1e38 2^128 2 29 | # possible BTC or ETH addresses 1e48 2^160 30 | # possible 3x8 byte addresses 1e57 2^192 3 31 | # possible amp.tag addresses 1e57 2^192 32 | # number of atoms in universe 1e80 2^266 33 | # possible 32 byte addresses 1e77 2^256 4 34 | # 35 | # The talented and well-spoken Michael at Vsauce delightfully shows how BIG these numbers really are: 36 | # https://www.youtube.com/watch?v=ObiqJzfyACM&t=888s 37 | # 38 | # ~ Drew O'Meara, Fall 2018, PLAN Systems 39 | # 40 | # 41 | # Output: 42 | """ 43 | Space Size 44 | ---------- 45 | bytes 4 bytes 8 bytes 16 bytes 20 bytes 24 bytes 32 bytes 64 bytes 46 | bits 32 bits 64 bits 128 bits 160 bits 192 bits 256 bits 512 bits 47 | combos 4.3e+09 1.8e+19 3.4e+38 1.5e+48 6.3e+57 1.2e+77 1.3e+154 48 | ------------ ------------ ------------ ------------ ------------ ------------ ------------ 49 | Collision Odds 50 | -------------- 51 | 1 in 100 9269 607 million 2.6e+18 1.7e+23 1.1e+28 4.8e+37 1.6e+76 52 | 1 in 1000 2931 192 million 8.2e+17 5.4e+22 3.5e+27 1.5e+37 5.2e+75 53 | 1 in 10000 927 60 million 2.6e+17 1.7e+22 1.1e+27 4.8e+36 1.6e+75 54 | 1 in 100000 294 19 million 8.2e+16 5.4e+21 3.5e+26 1.5e+36 5.2e+74 55 | 1 in 1 million 93 6 million 2.6e+16 1.7e+21 1.1e+26 4.8e+35 1.6e+74 56 | 1 in 10 million 30 1 million 8.2e+15 5.4e+20 3.5e+25 1.5e+35 5.2e+73 57 | 1 in 100 million 10 607401 2.6e+15 1.7e+20 1.1e+25 4.8e+34 1.6e+73 58 | 1 in 1 billion 3 192077 825 trillion 5.4e+19 3.5e+24 1.5e+34 5.2e+72 59 | 1 in 10 billion 1 60741 261 trillion 1.7e+19 1.1e+24 4.8e+33 1.6e+72 60 | 1 in 100 billion 1 19208 82 trillion 5.4e+18 3.5e+23 1.5e+33 5.2e+71 61 | 1 in 1 trillion 1 6075 26 trillion 1.7e+18 1.1e+23 4.8e+32 1.6e+71 62 | 1 in 10 trillion 1 1921 8 trillion 5.4e+17 3.5e+22 1.5e+32 5.2e+70 63 | 1 in 100 trillion 1 608 3 trillion 1.7e+17 1.1e+22 4.8e+31 1.6e+70 64 | 1 in 1e+15 1 193 825 billion 5.4e+16 3.5e+21 1.5e+31 5.2e+69 65 | 1 in 1e+16 1 61 261 billion 1.7e+16 1.1e+21 4.8e+30 1.6e+69 66 | 1 in 1e+17 1 20 82 billion 5.4e+15 3.5e+20 1.5e+30 5.2e+68 67 | 1 in 1e+18 1 7 26 billion 1.7e+15 1.1e+20 4.8e+29 1.6e+68 68 | 1 in 1e+19 1 2 8 billion 541 trillion 3.5e+19 1.5e+29 5.2e+67 69 | """ 70 | 71 | import math 72 | 73 | oddsList = [ 74 | 100, 75 | 1000, 76 | 10000, 77 | 100000, 78 | 1000000, 79 | 10000000, 80 | 100000000, 81 | 1000000000, # 1 billion 82 | 1e10, 83 | 1e11, 84 | 1e12, 85 | 1e13, 86 | 1e14, 87 | 1e15, 88 | 1e16, 89 | 1e17, 90 | 1e18, 91 | 1e19, 92 | ] 93 | 94 | bitList = [ 95 | 32, 96 | 64, 97 | 128, 98 | 160, 99 | 192, 100 | 256, 101 | 512, 102 | ] 103 | 104 | def decimalTo2exp(value): 105 | exp = math.log(value) / math.log(2.) 106 | return "2^%.1f" % exp 107 | 108 | def getPrettyInt(N): 109 | fN = float(N) 110 | if fN < 1000000.: 111 | s = "%d" % int(fN) 112 | elif fN < 1e9: 113 | s = "%.1f million" % math.floor(fN/1e6) 114 | elif fN < 1e12: 115 | s = "%.0f billion" % (fN/1e9) 116 | elif fN < 1e15: 117 | s = "%.0f trillion" % (fN/1e12) 118 | else: 119 | s = "%.1e" % fN 120 | 121 | s = s.replace( ".0", "", 1 ) 122 | s = s.replace( " 1 ", " a " ) 123 | 124 | return "%-17s" % s 125 | 126 | 127 | print "\n\n\n" 128 | 129 | if 1: 130 | print "Space Size\n----------" 131 | 132 | line = "%-22s" % "bytes" 133 | for numBits in bitList: 134 | line += "%-17s" % ("%d bytes" % (numBits/8)) 135 | print line 136 | 137 | line = "%-22s" % "bits" 138 | for numBits in bitList: 139 | line += "%-17s" % ("%d bits" % numBits) 140 | print line 141 | 142 | line = "%-22s" % "combos" 143 | for numBits in bitList: 144 | line += "%-17s" % ("%.1e" % float(pow(2., numBits))) 145 | print line 146 | 147 | line = "%-22s" % "" 148 | for numBits in bitList: 149 | line += "%-17s" % ("------------") 150 | print line 151 | 152 | print "Collision Odds\n--------------" 153 | 154 | 155 | for odds in oddsList: 156 | line = '1 in ' + getPrettyInt(odds) 157 | Pcollision = 1./float(odds) 158 | for numBits in bitList: 159 | N = float(2**numBits) 160 | numPicksNeeded = math.ceil( math.sqrt(2. * N * Pcollision) ) 161 | line += getPrettyInt(numPicksNeeded) 162 | print line 163 | 164 | 165 | 166 | print decimalTo2exp(3e16) 167 | print decimalTo2exp(2e17) 168 | print decimalTo2exp(4e17) 169 | --------------------------------------------------------------------------------