├── .vuepress ├── CNAME ├── styles │ └── index.styl ├── highlight.js ├── config.js └── public │ └── images │ └── logo.svg ├── docs ├── gtd │ ├── component-dependencies.png │ ├── component-dependencies.dot │ └── README.md ├── objectives.meta.md ├── media │ └── README.md ├── ecosystem.meta.md ├── ecosystem.md ├── advanced-layouts.md └── objectives.md ├── .gitignore ├── getting-started ├── rust.md ├── README.md ├── go.md ├── js.md └── synthesis │ └── how-ipfs-web-gateways-work.md ├── .github └── workflows │ └── main.yml ├── package.json ├── FAQ.md ├── tutorial.md └── README.md /.vuepress/CNAME: -------------------------------------------------------------------------------- 1 | docs.ipld.io 2 | -------------------------------------------------------------------------------- /.vuepress/styles/index.styl: -------------------------------------------------------------------------------- 1 | div[class~="language-ipldsch"]::before { 2 | content: "ipld schema"; 3 | } 4 | -------------------------------------------------------------------------------- /docs/gtd/component-dependencies.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ipld/docs/HEAD/docs/gtd/component-dependencies.png -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | .nyc_output 2 | build 3 | coverage 4 | package-lock.json 5 | node_modules 6 | .DS_Store 7 | yarn.lock 8 | .dagdb.car 9 | .vuepress/dist 10 | -------------------------------------------------------------------------------- /getting-started/rust.md: -------------------------------------------------------------------------------- 1 | 2 | !!! 3 | 4 | This document has **moved**. 5 | 6 | You'll now find information like this in the [ipld/ipld](https://github.com/ipld/ipld/) meta-repo, 7 | and published to the web at https://ipld.io/ . 8 | 9 | All documentation, fixtures, specifications, and web content is now gathered into that repo. 10 | Please update your links, and direct new contributions there. 11 | 12 | !!! 13 | 14 | ---- 15 | 16 | TODO 17 | -------------------------------------------------------------------------------- /.github/workflows/main.yml: -------------------------------------------------------------------------------- 1 | name: gh-pages 2 | 3 | on: 4 | push: 5 | branches: 6 | - master 7 | 8 | jobs: 9 | build: 10 | 11 | runs-on: ubuntu-latest 12 | 13 | steps: 14 | - name: Checkout 15 | uses: actions/checkout@v1 16 | with: 17 | fetch-depth: 1 18 | 19 | - name: Build with Node.js 20 | uses: actions/setup-node@v1 21 | with: 22 | node-version: '12.x' 23 | - run: npm install 24 | - run: | 25 | npm run build 26 | cp ./.vuepress/CNAME ./.vuepress/dist/ 27 | 28 | - name: GitHub Pages action 29 | uses: peaceiris/actions-gh-pages@v3.6.3 30 | with: 31 | github_token: ${{ secrets.GITHUB_TOKEN }} 32 | publish_dir: ./.vuepress/dist/ 33 | -------------------------------------------------------------------------------- /getting-started/README.md: -------------------------------------------------------------------------------- 1 | 2 | !!! 3 | 4 | This document has **moved**. 5 | 6 | You'll now find information like this in the [ipld/ipld](https://github.com/ipld/ipld/) meta-repo, 7 | and published to the web at https://ipld.io/ . 8 | 9 | All documentation, fixtures, specifications, and web content is now gathered into that repo. 10 | Please update your links, and direct new contributions there. 11 | 12 | !!! 13 | 14 | ---- 15 | 16 | Using IPLD can vary from language to language. 17 | 18 | * [Getting started in JavaScript](./js) 19 | * [Persisting IPLD data in `js-ipfs` and `js-ipfs-lite`](./js#storing-ipld-data-in-ipfs) 20 | * [Getting started in Go]() 21 | * [Persisting IPLD data in `go-ipfs`]() 22 | * [Getting started in Rust]() 23 | * [Persisting IPLD data in `rust-ipfs`]() 24 | -------------------------------------------------------------------------------- /package.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "docs", 3 | "version": "0.0.0", 4 | "description": "", 5 | "main": "index.js", 6 | "scripts": { 7 | "prepare:ipldsch": "find . -name \\*.ipldsch -exec sh -c \"echo '---\\neditLink: false\\n---\\n\\n\\`\\`\\`ipldsch' > {}.md && cat {} >> {}.md && echo '\\`\\`\\`' >> {}.md\" \\;", 8 | "prepare:json": "find html/ -name \\*.json -exec sh -c \"echo '---\\neditLink: false\\n---\\n\\n\\`\\`\\`json' > {}.md && cat {} >> {}.md && echo '\\`\\`\\`' >> {}.md\" \\;", 9 | "build": "vuepress build", 10 | "serve": "vuepress dev" 11 | }, 12 | "keywords": [], 13 | "author": "Mikeal Rogers (https://www.mikealrogers.com/)", 14 | "license": "(Apache-2.0 AND MIT)", 15 | "devDependencies": { 16 | "vuepress": "^1.5.2" 17 | }, 18 | "dependencies": { 19 | "vuepress-plugin-code-switcher": "^1.0.3" 20 | } 21 | } 22 | -------------------------------------------------------------------------------- /docs/objectives.meta.md: -------------------------------------------------------------------------------- 1 | 2 | !!! 3 | 4 | This document has **moved**. 5 | 6 | You'll now find information like this in the [ipld/ipld](https://github.com/ipld/ipld/) meta-repo, 7 | and published to the web at https://ipld.io/ . 8 | 9 | All documentation, fixtures, specifications, and web content is now gathered into that repo. 10 | Please update your links, and direct new contributions there. 11 | 12 | !!! 13 | 14 | ---- 15 | 16 | Meta: Objectives and Scope of IPLD 17 | ================================== 18 | 19 | This document is the first translation to concrete steps after the Vision declarations, 20 | and the root back to which we expect to tie most major features -- 21 | almost everything should reasonably trace back to some of the bulletpoints in this document. 22 | 23 | This document is **not** intended to be introductory material. 24 | For some audiences, it might still viably serve that purpose; 25 | but it's not the primary goal, and simplified wording is not used in this document. 26 | -------------------------------------------------------------------------------- /docs/gtd/component-dependencies.dot: -------------------------------------------------------------------------------- 1 | digraph { 2 | "Pathing" -> "Data Model" 3 | "Traversals" -> {"Pathing", "Data Model"} 4 | "ADLs" -> {"Data Model", "Linking"} 5 | "Data Model" -> {"Linking", "#root"} 6 | "Schemas" -> "Data Model" 7 | "Selectors" -> "Traversals" 8 | "Graphsync" -> "Selectors" 9 | "Unixfsv2" -> {"Pathing", "Data Model", "Traversals", "ADLs"} 10 | "Unixfsv2" -> "Schemas" [style=dashed] 11 | "Pathing" -> {"Schemas", "ADLs"} [color=grey] 12 | } 13 | 14 | // dotted line means "you will have an easier time implementing this if your IPLD libraries support this feature", but it's not strictly necessary, either. 15 | // grey line means this feature is primarily defined in terms of the black lines outbound from the source node, but can also be defined in terms of the target of the grey lines. 16 | 17 | // literally everything that points to schemas does so with a dotted line (or a grey line), 18 | // because you can *always* implement those things without a schema implementation in your langauge of choice or involved in your dev process... it'll just probably be more work. 19 | -------------------------------------------------------------------------------- /.vuepress/highlight.js: -------------------------------------------------------------------------------- 1 | // this require is awkward but technically the proper way to find the 'prismjs' 2 | // that's used by vuepress -> @vuepress/markdown -> prismjs 3 | const prism = require( 4 | require.resolve('prismjs', 5 | require.resolve('@vuepress/markdown', 6 | require.resolve('vuepress')))) 7 | 8 | prism.languages.ipldsch = { 9 | typedef: { 10 | pattern: /^[ \t]*(?:type|advanced)[ \t][A-Z](_?[A-Za-z0-9])*\b/m, 11 | inside: { 12 | keyword: /^[ \t]*(type|advanced)/m, 13 | 'class-name': /[\w]+$/ 14 | } 15 | }, 16 | keyword: /\b(?:bool|int|float|string|bytes|null|nullable|optional)\b/, 17 | builtin: /\b(struct|union|enum)(?=[ \t]*\{)\b/, 18 | representation: { 19 | pattern: /^}[ \t]representation\b/m, 20 | inside: { 21 | builtin: /representation/ 22 | } 23 | }, 24 | operator: /=/, 25 | number: /\b-?\d+\.?\d*(?:e[+-]?\d+)?\b/i, 26 | punctuation: /[(){}:[\]\|&]/, 27 | string: { 28 | pattern: /(")(?:\\[\s\S]|(?!\1)[^\\])*\1/, 29 | greedy: true 30 | }, 31 | comment: { 32 | pattern: /(^|[^"])#.*/, 33 | lookbehind: true, 34 | greedy: true 35 | } 36 | } 37 | -------------------------------------------------------------------------------- /docs/media/README.md: -------------------------------------------------------------------------------- 1 | 2 | !!! 3 | 4 | This document has **moved**. 5 | 6 | You'll now find information like this in the [ipld/ipld](https://github.com/ipld/ipld/) meta-repo, 7 | and published to the web at https://ipld.io/ . 8 | 9 | All documentation, fixtures, specifications, and web content is now gathered into that repo. 10 | Please update your links, and direct new contributions there. 11 | 12 | !!! 13 | 14 | ---- 15 | 16 | Talks, Videos and More 17 | ====================== 18 | 19 | Here is some other IPLD content around the web: 20 | 21 | --- 22 | 23 | ## Foundations for Decentralization: Data with IPLD @ GPN19 24 | 25 | 26 | 27 | This talk introduces IPLD and is targeted at giving newcomers a rapid overview, from the top to bottom of the system. 28 | Our dream of decentralized software is introduced, how we use content-addressing and immutable linking to get there, 29 | what the IPLD [Data Model](/#data-model) is about, how we use [Advanced Layouts](./docs/advanced-layouts) to handle big data, 30 | and even how we use IPLD [Schemas](https://specs.ipld.io/schemas/) as a decentralized-development-friendly way to describe protocol evolution. 31 | 32 | (duration: 55min) 33 | (slides: [link](https://slides.com/warpfork/ipld-gpn-2019/)) 34 | 35 | --- 36 | -------------------------------------------------------------------------------- /FAQ.md: -------------------------------------------------------------------------------- 1 | 2 | !!! 3 | 4 | This document has **moved**. 5 | 6 | You'll now find information like this in the [ipld/ipld](https://github.com/ipld/ipld/) meta-repo, 7 | and published to the web at https://ipld.io/ . 8 | 9 | All documentation, fixtures, specifications, and web content is now gathered into that repo. 10 | Please update your links, and direct new contributions there. 11 | 12 | !!! 13 | 14 | ---- 15 | 16 | Occasionally Asked Questions 17 | ============================ 18 | 19 | General 20 | ------- 21 | 22 | ### How does compression relate to IPLD? 23 | 24 | Compression in your data itself is frowned upon, 25 | as it breaks the "emergent convergence" story. 26 | 27 | Instead, make compression something that is transparently carried out by: 28 | - transport ( e.g. HTTP gzip ) 29 | - or storage ( e.g. compressed badger blocks ) 30 | 31 | As a general rule of thumb: IPLD should be operating on the *logical* form of your data. 32 | 33 | 34 | Schema-related 35 | -------------- 36 | 37 | ### What's the difference between a union and an enum? 38 | 39 | Unions are a recursive, and can contain any other type (with... some details and restrictions in IPLD Schemas, due to representations). Enums are a scalar with countable cardinality. 40 | 41 | Unions are also known as "sum types" in some literature. We latched onto the term "union" because A) it's one word; and B) due to the popularization of that term for it in Facebook's GraphQL. 42 | -------------------------------------------------------------------------------- /.vuepress/config.js: -------------------------------------------------------------------------------- 1 | require('./highlight') 2 | 3 | function linkfix (md) { 4 | const defaultRender = md.renderer.rules.link_open 5 | 6 | md.renderer.rules.link_open = function (tokens, idx, options, env, self) { 7 | const hrefIndex = tokens[idx].attrIndex('href'); 8 | if (hrefIndex > -1) { 9 | const href = tokens[idx].attrs[hrefIndex][1] 10 | if (/\.(ipldsch|json)$/.test(href)) { 11 | tokens[idx].attrs[hrefIndex][1] = `${href}.md` 12 | } 13 | } 14 | 15 | return defaultRender(tokens, idx, options, env, self) 16 | } 17 | } 18 | 19 | module.exports = { 20 | title: 'IPLD Documentation', 21 | description: 'Documentation for the InterPlanetary Linked Data (IPLD) project', 22 | plugins: [ 'code-switcher' ], 23 | themeConfig: { 24 | repo: 'ipld/docs', 25 | editLinks: true, 26 | editLinkText: 'Edit this page on GitHub', 27 | lastUpdated: 'Last Updated', 28 | smoothScroll: true, 29 | nav: [ 30 | { text: 'Home', link: '/' } 31 | ], 32 | logo: '/images/logo.svg', 33 | sidebar: { 34 | '/schemas/': [{ 35 | title: 'IPLD Schemas', 36 | collapsable: false, 37 | sidebarDepth: 2, 38 | children: [ 39 | ['goals', 'Goals'], 40 | ['feature-summary', 'Feature Summary'], 41 | ['introduction', 'Introduction'], 42 | ['authoring-guide', 'Authoring Guide'], 43 | 'links', 44 | 'schema-kinds', 45 | ['representations', 'Representations'], 46 | ['advanced-layouts', 'Advanced Layouts'] 47 | ] 48 | }], 49 | '/': 'auto' 50 | } 51 | }, 52 | extendMarkdown: md => { 53 | // use more markdown-it plugins! 54 | md.use(linkfix) 55 | } 56 | } 57 | -------------------------------------------------------------------------------- /docs/ecosystem.meta.md: -------------------------------------------------------------------------------- 1 | 2 | !!! 3 | 4 | This document has **moved**. 5 | 6 | You'll now find information like this in the [ipld/ipld](https://github.com/ipld/ipld/) meta-repo, 7 | and published to the web at https://ipld.io/ . 8 | 9 | All documentation, fixtures, specifications, and web content is now gathered into that repo. 10 | Please update your links, and direct new contributions there. 11 | 12 | !!! 13 | 14 | ---- 15 | 16 | Meta: IPLD in the Interplanetary Ecosystem 17 | ========================================== 18 | 19 | This document is meant to answer very basic, high-level questions about how the different Interplanetary projects are related. 20 | 21 | It should be readable in minutes (and skimmable in less). 22 | After reading it, someone should be able to clearly conceptualize the relationship between IPFS, IPLD, and libp2p. 23 | 24 | We might expect to link to this document as one of the first primers for anyone who's heard of IPFS, but isn't yet clear on what IPLD is. 25 | (This is a common path right now, and we need a document that handles this quickly and concisely.) 26 | 27 | 28 | Review wishlist 29 | --------------- 30 | 31 | ### detail check 32 | 33 | Currently, some details are included (like "what is bitswap") which may be too fine-grained; 34 | we could either consider moving those elsewhere, 35 | or simply accept that the bottom half of this document is not going to be at the appropriate level of detail for many readers coming to it for high-level introduction. 36 | 37 | ### more external references to content addressing 38 | 39 | I'm not super happy with the links to content addressing background information. Better material to link to would be welcome. 40 | 41 | In particular, I'd like to get a high-level semantic primer -- *just* focusing on content-addressing as a *concept* -- and not introducing CIDs concretely (yet). 42 | (It's a lot easier to explain CIDs, and the various forms of future-proofing they encapsulate, if the relevance of hashing is already thoroughly understood.) 43 | 44 | I guess this is central enough we should also just write our own documentation focusing on this. 45 | -------------------------------------------------------------------------------- /.vuepress/public/images/logo.svg: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | -------------------------------------------------------------------------------- /getting-started/go.md: -------------------------------------------------------------------------------- 1 | 2 | !!! 3 | 4 | This document has **moved**. 5 | 6 | You'll now find information like this in the [ipld/ipld](https://github.com/ipld/ipld/) meta-repo, 7 | and published to the web at https://ipld.io/ . 8 | 9 | All documentation, fixtures, specifications, and web content is now gathered into that repo. 10 | Please update your links, and direct new contributions there. 11 | 12 | !!! 13 | 14 | ---- 15 | 16 | # Getting Started with IPLD in Go 17 | 18 | In Go, the primary way to use IPLD is via the `github.com/ipld/go-ipld-prime` 19 | module. 20 | 21 | Its API is centered around `Node`s, which can be built from scratch and encoded 22 | into blocks, where we can then obtain `Link`s to them. 23 | 24 | First, we'll need to import some IPLD packages and define a define a link 25 | builder, which specifies all the codecs and hashing features we'll use: 26 | 27 | ```go 28 | package main 29 | 30 | import ( 31 | "github.com/ipfs/go-cid" 32 | "github.com/ipld/go-ipld-prime" 33 | _ "github.com/ipld/go-ipld-prime/codec/dagcbor" 34 | "github.com/ipld/go-ipld-prime/fluent" 35 | cidlink "github.com/ipld/go-ipld-prime/linking/cid" 36 | basicnode "github.com/ipld/go-ipld-prime/node/basic" 37 | ) 38 | 39 | var linkBuilder = cidlink.LinkBuilder{cid.Prefix{ 40 | Version: 1, // Usually '1'. 41 | Codec: 0x71, // dag-cbor as per multicodec 42 | MhType: 0x15, // sha3-384 as per multihash 43 | MhLength: 48, // sha3-384 hash has a 48-byte sum. 44 | }} 45 | ``` 46 | 47 | With that out of the way, we can build our first nodes. Beware that, for the 48 | sake of brevity, we aren't handling any errors. 49 | 50 | ```go 51 | func main() { 52 | ctx := context.Background() 53 | // For now, we're not storing the blocks anywhere. 54 | store := func(ipld.LinkContext) (io.Writer, ipld.StoreCommitter, error) { 55 | return ioutil.Discard, func(lnk ipld.Link) error { return nil }, nil 56 | } 57 | 58 | eric := fluent.MustBuildMap(basicnode.Prototype.Any, 1, func(na fluent.MapAssembler) { 59 | na.AssembleEntry("name").AssignString("Eric Myhre") 60 | }) 61 | ericLink, _ := linkBuilder.Build(ctx, ipld.LinkContext{}, eric, store) 62 | people := fluent.MustBuildList(basicnode.Prototype.Any, 1, func(na fluent.ListAssembler) { 63 | na.AssembleValue().AssignLink(ericLink) 64 | }) 65 | peopleLink, _ := linkBuilder.Build(ctx, ipld.LinkContext{}, people, store) 66 | fmt.Println(peopleLink) 67 | } 68 | ``` 69 | 70 | We encode both nodes into blocks with `dag-cbor`, which can encode all JSON 71 | types, binary data, and IPLD links (CIDs). Finally, the `people` node contains a 72 | link to Eric's block. 73 | 74 | You can also implement a storage layer to save and retrieve blocks. For example, 75 | here's an in-memory map store: 76 | 77 | ```go 78 | func main() { 79 | ctx := context.Background() 80 | storage := make(map[ipld.Link][]byte) 81 | store := func(ipld.LinkContext) (io.Writer, ipld.StoreCommitter, error) { 82 | buf := bytes.Buffer{} 83 | return &buf, func(lnk ipld.Link) error { 84 | storage[lnk] = buf.Bytes() 85 | return nil 86 | }, nil 87 | } 88 | loader := func(lnk ipld.Link, _ ipld.LinkContext) (io.Reader, error) { 89 | return bytes.NewReader(storage[lnk]), nil 90 | } 91 | 92 | eric := fluent.MustBuildMap(basicnode.Prototype.Any, 1, func(na fluent.MapAssembler) { 93 | na.AssembleEntry("name").AssignString("Eric Myhre") 94 | }) 95 | ericLink, _ := linkBuilder.Build(ctx, ipld.LinkContext{}, eric, store) 96 | daniel := fluent.MustBuildMap(basicnode.Prototype.Any, 1, func(na fluent.MapAssembler) { 97 | na.AssembleEntry("name").AssignString("Daniel Martí") 98 | }) 99 | danielLink, _ := linkBuilder.Build(ctx, ipld.LinkContext{}, daniel, store) 100 | people := fluent.MustBuildList(basicnode.Prototype.Any, 2, func(na fluent.ListAssembler) { 101 | na.AssembleValue().AssignLink(ericLink) 102 | na.AssembleValue().AssignLink(danielLink) 103 | }) 104 | peopleLink, _ := linkBuilder.Build(ctx, ipld.LinkContext{}, people, store) 105 | 106 | nb := basicnode.Prototype.Any.NewBuilder() 107 | _ = peopleLink.Load(ctx, ipld.LinkContext{}, nb, loader) 108 | people2 := nb.Build() 109 | for itr := people2.ListIterator(); !itr.Done(); { 110 | _, value, _ := itr.Next() 111 | personLink, _ := value.AsLink() 112 | 113 | nb := basicnode.Prototype.Any.NewBuilder() 114 | _ = personLink.Load(ctx, ipld.LinkContext{}, nb, loader) 115 | person := nb.Build() 116 | 117 | name, _ := person.LookupByString("name") 118 | nameStr, _ := name.AsString() 119 | fmt.Println(nameStr) 120 | } 121 | } 122 | ``` 123 | 124 | This gives you an idea of what it's like to use links to connect pieces of data 125 | together. 126 | 127 | TODO: show that if two links are identical, the data is too 128 | TODO: show how to store IPLD data on IPFS with go-ipfs 129 | -------------------------------------------------------------------------------- /getting-started/js.md: -------------------------------------------------------------------------------- 1 | 2 | !!! 3 | 4 | This document has **moved**. 5 | 6 | You'll now find information like this in the [ipld/ipld](https://github.com/ipld/ipld/) meta-repo, 7 | and published to the web at https://ipld.io/ . 8 | 9 | All documentation, fixtures, specifications, and web content is now gathered into that repo. 10 | Please update your links, and direct new contributions there. 11 | 12 | !!! 13 | 14 | ---- 15 | 16 | # Getting Started with IPLD in JavaScript 17 | 18 | In JS the primary interface for creating data in IPLD is 19 | the `Block` API. 20 | 21 | A block in IPLD is data paired with an address (CID). With the 22 | `@ipld/block` API you can encode data directly into a block. 23 | 24 | ```js 25 | import Block from '@ipld/block/defaults' 26 | 27 | const person = { name: 'Mikeal Rogers' } 28 | const childBlock = Block.encoder(person, 'dag-cbor') 29 | const people = [ await childBlock.cid() ] 30 | const rootBlock = Block.encoder(people, 'dag-cbor') 31 | ``` 32 | 33 | The `defaults.js` export gives us a Block interface with a few 34 | basic codecs and hashing functions. One of those codecs is 35 | `dag-cbor` which can encode all JSON types, binary, and IPLD 36 | links (CIDs). 37 | 38 | You can use blocks in your program and connect them to any 39 | kind of storage or network layer you wish. As an example, 40 | here's an in-memory block cache. 41 | 42 | ```js 43 | const cache = new Map() 44 | const store = { 45 | get: async cid => cache.get(cid.toString()), 46 | put: async block => cache.set((await block.cid()).toString(), block) 47 | } 48 | 49 | // childBlock and rootBlock from first example 50 | await Promise.all([childBlock, rootBlock].map(b => store.put(b))) 51 | 52 | const printPeople = async () => { 53 | const rootLink = await rootBlock.cid() 54 | const peopleBlock = await store.get(rootLink) 55 | const people = peopleBlock.decode() 56 | for (const link of people) { 57 | const block = await store.get(link) 58 | console.log(block.decode()) 59 | } 60 | } 61 | printPeople() 62 | ``` 63 | 64 | This gives you an idea of what it's like to use links to connect 65 | different pieces of data together. Links are also useful as a value 66 | type. For instance, you can tell if two pieces of data are identical 67 | by comparing their Links and never actually retreiving the data. 68 | 69 | ```js 70 | const save = async obj => { 71 | const block = Block.encoder(obj, 'dag-cbor') 72 | await store.put(block) 73 | return block.cid() 74 | } 75 | const skating = await save('skating') 76 | const rowing = await save('rowing') 77 | const running = await save('running') 78 | 79 | const mikeal = await save({ name: 'Mikeal', interests: [ skating ] }) 80 | const robert = await save({ name: 'Robert', interests: [ rowing, running ]}) 81 | const steve = await save({ name: 'Steve', interests: [ running, skating ] }) 82 | 83 | const compareInterests = async (person1, person2) => { 84 | const blocks = await Promise.all([person1, person2].map(cid => store.get(cid))) 85 | const [ decode1, decode2 ] = blocks.map(block => block.decode()) 86 | const seen = new Set() 87 | const commonInterests = [] 88 | decode1.interests.forEach(cid => seen.add(cid.toString())) 89 | for (const cid of decode2.interests) { 90 | if (seen.has(cid.toString())) { 91 | const block = await store.get(cid) 92 | commonInterests.push(block.decode()) 93 | } 94 | } 95 | const people = `${decode1.name} and ${decode2.name}` 96 | if (commonInterests.length === 0) { 97 | console.log(`${people} have nothing in common.`) 98 | } else { 99 | console.log(`${people} have ${commonInterests.join(' and ')} in common`) 100 | } 101 | } 102 | 103 | await compareInterests(mikeal, robert) 104 | await compareInterests(robert, steve) 105 | await compareInterests(steve, mikeal) 106 | ``` 107 | 108 | Prints 109 | 110 | ``` 111 | Mikeal and Robert have nothing in common. 112 | Robert and Steve have running in common. 113 | Steve and Mikeal have skating in common. 114 | ``` 115 | 116 | ## Storing IPLD data in IPFS 117 | 118 | The easiest way to persist IPLD data so that it's globally availible 119 | is to using the IPFS network. 120 | 121 | The `js-ipfs` project has an API for storing block data at `ipfs.block`. 122 | 123 | ```js 124 | import Block from '@ipld/block/defaults' 125 | import IPFS from 'ipfs' 126 | 127 | const ipfs = await IPFS.create() 128 | 129 | const save = async obj => { 130 | const block = Block.encoder(obj, 'dag-cbor') 131 | const data = block.encode() 132 | const cid = await block.cid() 133 | 134 | // js-ipfs uses an older CID value type so we must convert to string 135 | await ipfs.block.put(data, { cid: cid.toString() }) 136 | return cid 137 | } 138 | 139 | const mikeal = await save({ name: 'Mikeal', interests: [ skating ] }) 140 | const robert = await save({ name: 'Robert', interests: [ rowing, running ]}) 141 | const steve = await save({ name: 'Steve', interests: [ running, skating ] }) 142 | 143 | console.log('Seeding Mikeal as ', mikeal.toString()) 144 | console.log('Seeding Robert as ', robert.toString()) 145 | console.log('Seeding Steve as ', steve.toString()) 146 | ``` 147 | 148 | TODO: `js-ipfs-lite`. 149 | -------------------------------------------------------------------------------- /docs/gtd/README.md: -------------------------------------------------------------------------------- 1 | 2 | !!! 3 | 4 | This document has **moved**. 5 | 6 | You'll now find information like this in the [ipld/ipld](https://github.com/ipld/ipld/) meta-repo, 7 | and published to the web at https://ipld.io/ . 8 | 9 | All documentation, fixtures, specifications, and web content is now gathered into that repo. 10 | Please update your links, and direct new contributions there. 11 | 12 | !!! 13 | 14 | --- 15 | 16 | Getting Things Done with IPLD 17 | ============================= 18 | 19 | There are lots of ways to get things done in IPLD. 20 | 21 | When designing a new system and trying to decide what parts of IPLD 22 | you should use to accomplish your goals, try going down this chart -- 23 | in order: if you can get something done with the things at the top of this file, 24 | do that; if you can't, only then should you keep reading down the list to 25 | the more invasive and high-effort options that are available. 26 | 27 | In order: 28 | 29 | 1. Use the Data Model. 30 | 2. Use Schemas. 31 | 3. Use ADLs. 32 | 4. Write a new codec. 33 | 34 | Use the Data Model 35 | ------------------ 36 | 37 | You can get lots done with just the [Data Model](/#data-model) and core APIs. 38 | 39 | Whenever you want to get something done with IPLD, 40 | we suggest you start by just sketching out what you want to do by using the Data Model. 41 | This is similar to saying "just sketch examples of your desired API in JSON"; 42 | maybe you'll want more formality to your structure later, 43 | and maybe you'll want to write more auto-validation rules, etc; 44 | but those can come later. 45 | 46 | Use Schemas 47 | ----------- 48 | 49 | [Schemas](https://specs.ipld.io/schemas/) are for when you want to formalize structures. 50 | They can be useful for documentation; 51 | they do validation of data structures so you know what you're handling; 52 | they can be used for version detection and feature detection; 53 | and with some libraries they can even be used for code generation. 54 | 55 | We find that using Schemas is often a good idea. But you don't have to! 56 | If what you want to do is easy enough to accomplish at the Data Model level alone, 57 | then go for it. 58 | 59 | Schemas are totally compatible with the Data Model approaches to handling data. 60 | They just decorate more behaviors onto it. 61 | Any code that's written to traverse over the regular Data Model-specified interfaces 62 | will also be able to traverse over data that's been processed with Schemas. 63 | 64 | Schemas can be used to do some very basic transformations of data -- 65 | like transforming long-form names for fields in data into shorter names 66 | before passing them into or out of serialization -- but that's the extent 67 | of their powers; they can't do any more dynamic restructuring than that. 68 | 69 | Use ADLs 70 | -------- 71 | 72 | [ADLs](/advanced-layouts/) -- Advanced Data Layouts -- are for when you want to present data 73 | *as if* it is IPLD Data Model (so that all the rest of our tools can work over it), 74 | but the serialized structure of the data is significantly (topologically) different. 75 | 76 | Examples of things people have wanted to do in the ecosystem which have been 77 | done as ADLs include: 78 | 79 | - large collections, sharded across multiple blocks, but still presenting as a single map or list 80 | - large bytes, sharded across multiple blocks, but still presenting as one coherent bytes object 81 | - encrypted data, where there's a ciphertext format, and also (conditionally) a cleartext format 82 | 83 | ADLs are an abstraction layer where extremely powerful transformations like 84 | those listed above can be implemented... while still reusing existing 85 | serialization codecs, and while still presenting things as Data Model interfaces 86 | so that all the same traversal logic that works on either plain Data Model or 87 | on Schema-processed data *also* still works on data processed by an ADL. 88 | 89 | ADLs are also noteworthy in that (like Schemas), one can always choose to handle 90 | data _without them_, and just handle the raw Data Model contents directly. 91 | Often this will be less useful... but it can be a good debugging tool, 92 | and useful for comprehensible and reusable data structure design. 93 | 94 | Write a new Codec 95 | ----------------- 96 | 97 | This should be your absolute last resort. 98 | 99 | Writing a new [Codec](/#codecs) poses future portability questions. 100 | Every IPLD library in every programming language will need to implement it; 101 | many, practically speaking, won't -- unless you give them a very good reason. 102 | Therefore, it's best to stick to one of the already widely supported codecs. 103 | 104 | There should also simply be very little reason that you would need to create 105 | a new wire format and a new codec for it. IPLD Schemas and their mechanisms 106 | for customizing representations already make it possible to choose any position 107 | you like on the gradient between schemaless (redundant but easy to read) 108 | and schema-required (densely packed, but impossible to understand without a schema) 109 | data... and most new serialization formats are invented because someone wants 110 | to stake out a new position on that gradient. Now, you shouldn't have to. 111 | 112 | The most typical reason to introduce new codecs is to bridge the IPLD system 113 | with some other, existing serialization formats. That's great, and we 114 | welcome (and absolutely designed for) this kind of bridging. 115 | But if you're designing a new system that's IPLD from the start, 116 | we really recommend using existing codecs. 117 | It's just easier to hit the ground running, and there's little value 118 | and lots of costs to trying to forge your own way through a new codec. 119 | 120 | But Why? 121 | -------- 122 | 123 | Why are there so many different layers? 124 | 125 | Because for standards to be useful and allow people to build an ecosystem of reusable systems together, they have to unify on some things; 126 | and for standards to be flexible enough to allow people to build things suitable to them and their specific tasks, they have to be extensible. 127 | 128 | We have different degrees of unification versus flexibility in each of these layers. 129 | That means you can *choose* how close of an orbit you want to keep with the rest of the ecosystem. 130 | -------------------------------------------------------------------------------- /docs/ecosystem.md: -------------------------------------------------------------------------------- 1 | 2 | !!! 3 | 4 | This document has **moved**. 5 | 6 | You'll now find information like this in the [ipld/ipld](https://github.com/ipld/ipld/) meta-repo, 7 | and published to the web at https://ipld.io/ . 8 | 9 | All documentation, fixtures, specifications, and web content is now gathered into that repo. 10 | Please update your links, and direct new contributions there. 11 | 12 | !!! 13 | 14 | ---- 15 | 16 | IPLD in the InterPlanetary Ecosystem 17 | ==================================== 18 | 19 | IPLD is a part of the "InterPlanetary Ecosystem". 20 | You can use it without any of the other projects in that ecosystem, but it may be useful to understand this history; 21 | and if you come to IPLD via one of the other InterPlanetary projects (such as IPFS), 22 | this document can help you understand the relationship and the distinctions. 23 | 24 | IPLD 25 | ---- 26 | 27 | IPLD is short for "InterPlanetary Linked Data". 28 | 29 | As the name suggests, IPLD is all about _data_: 30 | defining data structures, defining data serialization, 31 | defining ways we can expect to traverse data, 32 | and defining ways we can link data. 33 | 34 | IPLD is primarily a set of specs and a series of libraries: 35 | it's meant to help others write decentralized applications, 36 | and is not an application itself. 37 | 38 | See IPFS for an application that uses IPLD. 39 | 40 | See libp2p for libraries that can be used to transport IPLD data over networks. 41 | 42 | 43 | IPFS 44 | ---- 45 | 46 | IPFS is best understood as an application that glues together: 47 | 48 | - IPLD, to describe data, 49 | - libp2p, to transmit data around, 50 | - some concrete implementations of data storage, 51 | - and some fun features that make a concept of "files" and "directories" readily accessible to users of IPFS. 52 | 53 | (There's also a bunch of neat features like a web interface that accesses the abstraction of "files", 54 | various APIs, etc -- but check out the IPFS documentation for details of this, not ours ;)) 55 | 56 | In short: IPFS depends on IPLD. 57 | 58 | IPLD does not depend on IPFS. 59 | 60 | ### history 61 | 62 | The IPFS project is the first project of the InterPlanetary ecosystem. 63 | Many early ideas emerged in this project... and were only extracted sometime later! 64 | 65 | IPLD is one such idea that incubated in the IPFS project, and was only extracted later. 66 | (libp2p is another project with a similar origin story!) 67 | 68 | You may see stretch marks around this for some time: 69 | comments in old code which, read today, seem ambiguous about the separations; etcetera. 70 | 71 | We're working to increase the clarity of roles and responsibilities of all of these projects over time. 72 | Newly written work in IPLD will generally not refer to IPFS at all; 73 | and you can definitely count on it to not have direct dependencies on IPFS code. 74 | We regard IPFS as just one consumer of the IPLD libraries. 75 | 76 | 77 | libp2p 78 | ------ 79 | 80 | libp2p is a project which handles networking and transport of data. 81 | 82 | The features of libp2p of the features range from 83 | the basic (simple streaming transport APIs) 84 | to the useful (NAT traversal, "hole punching", protocol negociations, etc) 85 | to the neat (pubkey-based persistent network peer identity toolkits) 86 | to the awesome (DHTs and other advanced network routing and distributed storage components). 87 | 88 | libp2p is developed by some of the same people as IPLD, IPFS, and the other InterPlanetary projects. 89 | Like IPLD, libp2p is a project that historically emerged from IPFS when it became apparent that it would be widely reusable; 90 | now, many other projects also use libp2p as a networking abstraction, or lean on its routing and storage components. 91 | 92 | libp2p doesn't have any explicit dependency on IPLD, nor vice-versa. 93 | We do design them to work well together, though! 94 | libp2p is a great thing to check out if you want to build a distributed application with IPLD, and you need to handle networking. 95 | 96 | libp2p uses several of the Multiformats specifications in describing their data and their protocol negotiation systems. 97 | (However, they're often different subsets of the Multiformats than those that IPLD is most concerned with.) 98 | 99 | 100 | CIDs 101 | ---- 102 | 103 | CIDs -- **C**ontent **ID**entifiers -- are a specification for identifying content based on cryptographic hashing. 104 | See wikipedia on [Content Addressing](https://en.wikipedia.org/wiki/Content-addressable_storage#Content-addressed_vs._location-addressed) for a general introduction to this concept. 105 | 106 | We use CIDs for "linking" in IPLD. "Linking" in IPLD just has the semantic requirement of using immutable identifiers; 107 | in practice, we satisfy this aim by using CIDs for links. 108 | 109 | See also [CIDs in the IPFS documentation](https://docs.ipfs.io/concepts/content-addressing/). 110 | 111 | 112 | Multiformats 113 | ------------ 114 | 115 | ### multihash 116 | 117 | Multihash indicators are a part of CIDs. Therefore, you may see multihash in IPLD through linking, which uses CIDs. 118 | 119 | ### multicodecs 120 | 121 | Multicodec indicators are a part of CIDs. Therefore, you may see multicodec in IPLD through linking, which uses CIDs. 122 | 123 | ### multiaddrs 124 | 125 | Multiaddrs are one of the multiformats that IPLD has the *least* to do with. 126 | 127 | Multiaddrs are used to describe networking locations in libp2p. 128 | 129 | IPLD is concerned with data formats, and not with networking; 130 | therefore, it does not have direct interactions with Multiaddrs. 131 | 132 | 133 | Other 134 | ----- 135 | 136 | ### graphsync 137 | 138 | Graphsync is the name of a protocol and set of libraries for exchanging blocks of data containing IPLD data 139 | (and it uses IPLD Selectors to allow a high degree of user power for saying what to exchange). 140 | 141 | IPLD doesn't depend on graphsync, but they play together very nicely. 142 | 143 | Graphsync has a comparable purpose to bitswap, although it is more recently created. 144 | 145 | ### bitswap 146 | 147 | Bitswap is the name of another protocol and set of libraries for exchanging blocks of data containing IPLD data. 148 | 149 | IPLD doesn't depend on bitswap, but bitswap is often used to exchange IPLD data. 150 | 151 | Bitswap is a long-standing part of IPFS. It's also comparable in purpose to Graphsync. 152 | -------------------------------------------------------------------------------- /docs/advanced-layouts.md: -------------------------------------------------------------------------------- 1 | 2 | !!! 3 | 4 | This document has **moved**. 5 | 6 | You'll now find information like this in the [ipld/ipld](https://github.com/ipld/ipld/) meta-repo, 7 | and published to the web at https://ipld.io/ . 8 | 9 | All documentation, fixtures, specifications, and web content is now gathered into that repo. 10 | Please update your links, and direct new contributions there. 11 | 12 | !!! 13 | 14 | ---- 15 | 16 | Advanced Data Layouts 17 | ===================== 18 | 19 | Advanced Data Layouts (or just Advanced Layouts, or ADLs for short) are a kind of plugin system for IPLD 20 | which is used when we want to present some data as if it were a [[Data Model]] [[Node]], 21 | while actually storing it as a different [[Node]] (or as several [[Node]]s!). 22 | 23 | This is easiest to understand by example: 24 | 25 | - // sharded map example 26 | 27 | - // sharded bytes example 28 | 29 | - // encryption example 30 | 31 | 32 | Where do ADLs fit in? 33 | --------------------- 34 | 35 | You should probably read the doc about the [Data Model](data-model.md) first, if you haven't already. 36 | ADLs build upon the concepts that are introduced standardized by the Data Model. 37 | 38 | - ADLs convert [Data Model nodes](data-model.md#node) into *another node* 39 | (or, when writing new data, provide an interface to go the other way: let the user act like they're creating a node, but in the background create several nodes, or a different structure, which stores that data). 40 | - [Codecs](codecs.md) and ADLs compose smoothly -- Codecs can deserialize and serialize the Data Model data that is the "raw" "interior" content of an ADL. 41 | - [Schemas](https://specs.ipld.io/schemas/) technically have nothing to do with ADLs... 42 | - but, Schemas can be useful for _signaling_ when ADLs should be used to handle data ([more on that later](#signaling-with-schemas)); 43 | - and in practice, ADL specifications often include a Schema which describes them, simply for clarity (and ADL implementations might choose use that Schema in their internal code, too). 44 | - [Traversals](traversal.md) and [pathing](pathing.md) work transparently over ADLs (which is part of why ADLs exist and what makes them awesome in the first place)! 45 | 46 | 47 | How ADLs Work 48 | ------------- 49 | 50 | ### ADLs make nodes look like another node 51 | 52 | // emphasis on *one* node as the result: whether it be map or list or bytes or etc, *one*. 53 | 54 | // n.b. [..concrete example of what kind of transformation you'd be better off doing with schemas..] 55 | 56 | ### ADL interior data is still Data Model 57 | 58 | // clarify that without ADL code activated, the raw data can still be read and even traversed... just differently. 59 | 60 | // clarify that codecs and ADLs compose, there's a clear layering there. 61 | 62 | ### ADLs use code 63 | 64 | ADLs use code, and some sort of plugin system is needed in IPLD libraries to support this. 65 | 66 | How exactly those plugin systems work, and what kind of format the code needs to be authored in, 67 | and exactly what interfaces need to be adhered to: 68 | these will all vary per IPLD library and the language the IPLD library is in. 69 | 70 | (Someday, a system for portable ADL code would be neat. 71 | However, we currently consider that a research problem: 72 | some notes can be found in [open-research/ADLs-we-can-autoexecute](open-research/ADLs-we-can-autoexecute.md).) 73 | 74 | ### How do I know when to use an ADL to read data? 75 | 76 | We call this "the signaling problem". 77 | 78 | In short: you don't. 79 | 80 | Since the data composing the "raw", interior data of an ADL is just regular IPLD Data Model 81 | (it must be, after all, since it's produced by some [Codec](codec.md), which by definition produces data structures describable by the Data Model), 82 | then it follows that there's absolutely no way for this data to unambiguously indicate that it needs an ADL in order to be understood. 83 | If there was, it would imply that there's some kind of "reserved words" in the Data Model, 84 | which would violate some of our other central goals in IPLD, because it would mean some perfectly normal maps and lists would be invalid IPLD or gain magical meaning that they shouldn't; 85 | we don't want any of that. 86 | 87 | So! Signaling must come from somewhere else. 88 | 89 | There are a variety of valid options: 90 | 91 | #### Signaling with Schemas 92 | 93 | One useful system we have which can provide an answer to the signaling question are IPLD Schemas. 94 | Since Schemas are already a declarative way to talk about the structure of data, 95 | it's quite reasonable that they should also be able to talk about where the structure of data uses an ADL. 96 | 97 | However, you don't have to use IPLD Schemas if you want to use ADLs. 98 | Keep reading the next couple of sections for more alternatives that you can use to answer the signaling question. 99 | 100 | // TODO a doc on this, complete with syntax examples, is necessary. 101 | 102 | // a remark should be present here on the interesting limitation about *non*-recursive descriptions being somewhat high-friction to reach with this mechanism. 103 | // maybe this belongs in a separate deeper-diving doc in another page, though. 104 | 105 | #### Direct action within libraries 106 | 107 | 108 | #### Other declarative signaling 109 | 110 | We have no currently active specifications for other forms of declarative signaling. 111 | 112 | However, you can imagine making such a system yourself fairly easily: 113 | all that's necessary is to decide what that declarative format is that you want, 114 | and write a system that binds it to the relevant programmatic APIs of the IPLD libraries you use, 115 | and everything should work out from there. 116 | 117 | Additional declarative signaling specifications may be something that is ratified into IPLD in the future. 118 | (If you'd like to drive this work, please feel free to get in touch!) 119 | 120 | (Some systems have already done this in their own ways: for example, 121 | parts of the Filecoin Lotus project expose "paths" in their CLI which have an extension 122 | that is used in that application to signal where to engage ADLs. 123 | You can do things like this in your own applications, too! 124 | It's worth noting, however, that what the Filecoin Lotus project does here is not considered a well-specified IPLD behavior, 125 | and in fact contains several caveats which constrains what is valid data for that application to process to a range that is far narrower than what the IPLD Data Model specifies.) 126 | 127 | 128 | Practical Considerations for ADLs and using them 129 | ------------------------------------------------- 130 | 131 | ### not every application that processes IPLD data will necessarily support your ADLs 132 | 133 | // "running foreign code on somebody else's budget" is not something that happens at unbounded scale on public services 134 | 135 | // availability in many languages/libraries, and authorship/maint effort implied -- it's better to use community-common things if you can 136 | // similar to codecs in this regard 137 | 138 | // reminder that schemas *are* usable on public infra (like e.g. on hosted IPLD Explorer tools), because they have predictable computation cost envelopes -- reminder to prefer doing things with a schema rather than an ADL if you can; don't reach for ADLs just because you want a funky fresh custom format 139 | -------------------------------------------------------------------------------- /docs/objectives.md: -------------------------------------------------------------------------------- 1 | 2 | !!! 3 | 4 | This document has **moved**. 5 | 6 | You'll now find information like this in the [ipld/ipld](https://github.com/ipld/ipld/) meta-repo, 7 | and published to the web at https://ipld.io/ . 8 | 9 | All documentation, fixtures, specifications, and web content is now gathered into that repo. 10 | Please update your links, and direct new contributions there. 11 | 12 | !!! 13 | 14 | ---- 15 | 16 | Objectives and Scope of IPLD 17 | ============================ 18 | 19 | We want to see more decentralized designs appearing in technology. 20 | 21 | We believe specifications for data structures are a key part of enabling that. 22 | (See [[Theory of Change]] for the even bigger picture, 23 | and more on our concept of enablement.) 24 | 25 | Things IPLD is concerned with 26 | ----------------------------- 27 | 28 | ## Databases 29 | 30 | We want IPLD to provide many of the useful attributes of an SQL-style database for purely local/offline applications: 31 | IPLD Schemas and IPLD Selectors provide features comparable to DDLs and SQL, 32 | and set the stage for building migration systems. 33 | 34 | At the same time, IPLD works on a concept of "blocks" which are much more comparable to messages: 35 | they're easy to send and receive over the network, and critically, can be regarded individually 36 | (there's no supposition of monolithicity, as SQL-style databases have). 37 | 38 | Our aim is that IPLD should straddle this divide and operate on either side of it gracefully. 39 | The same libraries and concepts and tooling should work well regardless of if you're building an offline app 40 | or a massively distributed system with remote actors coming and going and sending messages all the time. 41 | 42 | ## Additional concerns 43 | 44 | In order to make building on decentralized data structures be excellent, we see the following major goals: 45 | 46 | - Some minimal legibility of data is required. 47 | - (This is what our work on the Data Model focuses around.) 48 | - It is awesome if our concept of data can be used across various encoding systems with a minimum of effort. 49 | - (This is why most IPLD libraries have a concept of various codecs; and we do a lot of work to specify codecs in terms of the Data Model.) 50 | - Whenever data is serialized, some standard indications of what codecs are necessary to use to transform that serial data back into a minimally legible form must be available, 51 | and these indicators should always be found very tightly bound to any stored data. 52 | - (This is what multicodecs are about, and why multicodec indicators are in CIDs.) 53 | - For data to be usable in decentralized/distributed systems, it's important for small pieces of data to link to other pieces of data; 54 | and this must be based on immutable references, because a user must be able to freely choose between fetching data either immediately or deferred at some later time: 55 | it must be possible to choose this without changing the correctness and coherency of their data by making their fetch actions at a later date. 56 | The most effective way to do this is content-addressing with the use of cryptographic hashes. 57 | - (This is what IPLD Linking and CIDs are all about.) 58 | - Given that we have identified that hashing data is a useful thing to do in the pursuit of building decentralized data structures, 59 | it would be excellent to standardize the process of feeding data to hashes, so that this can become a simple thing that happens without great special effort; 60 | and since there are various hashing systems in the world, it is important to identify which hashing algorithm was used to compute any hash we might store or reference. 61 | - (This is what multihashes are about, and why multihash indicators are in CIDs; and why IPLD libraries connect dots all the way through from Data Model to Codec to hashing.) 62 | - It's important to be able to traverse our minimally legible data, programmatically. 63 | We should be able to understand how to do this even without understanding any semantic details of this particular data, 64 | and we need this so that data remains usefully legible even in "deep time". 65 | - (This is again what the Data Model, and IPLD Linking, focuses on. IPLD Selectors then go further, providing a declarative way to describe traversals.) 66 | - It's useful to be able to define some topological expectations of the structure of data in a declarative way. 67 | - (This is what IPLD Schemas focus on.) 68 | - It's important that these topological expectations about the structure of data should be able to apply to any data -- 69 | even data that predates (or was otherwise causally independent of) the writing of the topological declarations. 70 | This is important because this kind of causal independence is frequent in decentralized and distributed development, 71 | and thus to be true to our vision and theory of change, we should support this kind of relationship. 72 | - (This figures deeply in the design of IPLD Schemas.) 73 | - It's desirable that these topological expectations about the structure of data should be something we can use to describe "version" detection and feature detection. 74 | Most concepts of "version" are brittle in decentralized systems (and especially, in decentralized development practices); 75 | we need ways to reason about data that continue to work when there's no central authority on version numbering, 76 | and mechanisms for describing data without resorting to explicit inbuilt versioning at all, so that it is possible to evolve the way we handle data separately from the data itself. 77 | - (This figures deeply in the design of IPLD Schemas.) 78 | - It is important to be able to collect large amounts of data, and make it accessible 79 | and navigable even if we split it up ("shard" it). 80 | - (We introduce an idea called Advanced Data Layouts as a way to describe such collections, and as part of this, describe how to make the sharded content directly legible as the IPLD Data Model.) 81 | 82 | 83 | Things IPLD is not concerned with 84 | ---------------------------------- 85 | 86 | - Encryption. It's important, but it's not our battle. 87 | We hope that IPLD will be useful for structuring data, both in cleartext and ciphertext. 88 | We hope the the Advanced Data Layout extension system might be applicable to making systems simultaneously use encryption and are traversable as IPLD! 89 | But essentially, we're looking to _enable_ these developments -- it's not on our roadmap to drive them. 90 | - Identity systems. It's important, but it's not our battle. 91 | Decentralized identity is a very expansive topic; it's exceeding unlikely that there will even be "one" answer to the questions in this field: 92 | it blurs between cryptosystem design, reputation systems, statistical properties of graphs... there's truly a whole field here, if not _several_, and no one-size-fits-all answers. 93 | We will try to provide guidance on how IPLD relates to, and might be usefully composed with, other decentralized identity systems; 94 | but ultimately, how to do this will depend on the choices of application developers and the needs of their applications. 95 | - Networking. How to get data from one place to another is important, but IPLD should work with any mechanism you choose. 96 | IPLD should even work fine and provide value with *no* networking, or work fantastically with "sneakernet". 97 | (The libp2p project is also pursuing useful work in networking and focused on decentralized systems; 98 | it's not directly related to IPLD, but they should work well together, and you might want to check it out.) 99 | -------------------------------------------------------------------------------- /getting-started/synthesis/how-ipfs-web-gateways-work.md: -------------------------------------------------------------------------------- 1 | 2 | !!! 3 | 4 | This document has **moved**. 5 | 6 | You'll now find information like this in the [ipld/ipld](https://github.com/ipld/ipld/) meta-repo, 7 | and published to the web at https://ipld.io/ . 8 | 9 | All documentation, fixtures, specifications, and web content is now gathered into that repo. 10 | Please update your links, and direct new contributions there. 11 | 12 | !!! 13 | 14 | ---- 15 | 16 | How IPFS Web Gateways Work 17 | ========================== 18 | 19 | This document talks through how IPFS web gateways work; 20 | the various forms of abstraction used in accomplishing parts of that task; 21 | and how and why these designs are cool (and how to can reuse parts of them). 22 | 23 | > :tea: 24 | > Note! This is an _aspirational_ document -- IPFS web gateways are a feature 25 | > which has substantial history. They haven't always worked like this. 26 | 27 | What are the goals? 28 | ------------------- 29 | 30 | ### What are IPFS Web Gateways, anyway? 31 | 32 | The "web gateway" is a feature of IPFS that serves the contents of _files_ over _HTTP_. 33 | It selects what content to serve by taking the path from the URL requested, 34 | and splitting it into two parts: the _CID_ and the remainder of the _path_. 35 | 36 | The CID is used to pick some piece of data to start at. 37 | The remainder of the path will be used as instructions to traverse IPLD data, 38 | starting from that data which the CID identified. 39 | At the end of that traversal, we'll try to understand whatever is pointed to as 40 | a bunch of bytes (a "file" isn't much more than that)... and pour it out into an HTTP response. 41 | 42 | This is useful because it means we can store data in IPFS, 43 | and that data can become easily accessible as normal webpages. 44 | 45 | 46 | How do we do it? 47 | ---------------- 48 | 49 | There's actually a ton of moving parts to this. 50 | Let's break it down: 51 | 52 | ### files are just bytes 53 | 54 | (... in this context, anyway.) 55 | 56 | TODO expound 57 | 58 | ### large bytes are provided by ADLs (FBL, specifically) 59 | 60 | TODO expound 61 | 62 | ### directories are provided by ADLs (HAMT, specifically) 63 | 64 | TODO expound 65 | 66 | ### navigating the path is defined by the Data Model 67 | 68 | ... which is neat, because most of it is also going through ADLs, 69 | as we just described in the previous sections. 70 | 71 | Because defining an ADL requires defining how the ADL will make its 72 | content data legible to regular IPLD Data Model semantics, 73 | we can use standard library functions for pathing over this data. 74 | No special path handling logic is written for the Web Gateways 75 | (or other users of unixfsv2) at all! 76 | 77 | ### knowing where to use ADLs, and which ones, is defined by IPLD Schemas 78 | 79 | ... and more specifically, the Unixfsv2 schema, 80 | which is pretty common and used by much more than the IPFS web gateway system. 81 | 82 | The web gateways know about the schema for unixfsv2, 83 | and they assume that it applies to whatever data is being requested through the gateway system. 84 | 85 | If the data doesn't match, it won't be renderable through the web gateway 86 | (and we return errors about how and where the data ceased to match the 87 | schema that's expected for unixfsv2 data). 88 | 89 | If the data does match: then the schema also tells us where to expect ADLs 90 | to come into effect, and also tells us what Data Model semantics to expect 91 | from the data that is made available once the ADL logic is applied. 92 | 93 | ### if you don't have this data: we fetch it 94 | 95 | This part hits the limits of what is directly in the scope of IPLD, 96 | but in brief, what happens in IPFS is: 97 | 98 | - a request for info about anyone who might have the missing content will be made to the DHT (provided by libp2p) 99 | - contact with those other IPFS nodes will be made (again brokered by libp2p, which might be handling NAT traversal, etc) 100 | - a protocol like bitswap or graphsync will be used to ferry raw blocks of IPLD data from those other IPFS nodes 101 | - ... and then the data is local: the process picks up again normally from there. 102 | 103 | 104 | What if we want to peek behind the curtain? 105 | ------------------------------------------- 106 | 107 | All of the structures above are serialized in CBOR. 108 | All the magic comes from IPLD semantic features like ADLs and Schemas. 109 | 110 | What's neat about this is it's also very easy to drop all the magic 111 | and look directly at the raw data. 112 | 113 | Want to see the raw CBOR objects that make up all these structures? No problem. 114 | They're hidden just barely under the hood, and we try to make it easy to lift that hood 115 | up and take a peek. 116 | 117 | In fact, you can inspect all the raw CBOR objects making up these structures 118 | simply by using IPLD libraries to start at the same CID, 119 | and then use regular Data Model traversal mechanisms to stroll around. 120 | If you provide the Unixfsv2 schema (and thereby its hints about where to apply ADLs), 121 | you'll get the high level view of "directories" (maps) and "files" (bytes); 122 | if you don't, you'll get to stroll around the raw CBOR objects instead. 123 | 124 | 125 | What's reusable? 126 | ---------------- 127 | 128 | Gosh. Dang near everything. 129 | 130 | - The Unixfsv2 schema is reusable. If you want to describe filesystem-like data, it's probably got what you want. 131 | - The Flexible Byte Layout (FBL) Advanced Data Layout is reusable for any of your big-sharded-byte-sequence needs! 132 | It's not unique to Unixfsv2 nor the IPFS web gateways; you can drop it in anywhere you want in any IPLD system. 133 | - The HAMT ADL is similarly reusable for any of your big-sharded-map needs! 134 | It's not unique to Unixfsv2 nor the IPFS web gateways; you can drop it in anywhere you want in any IPLD system. 135 | - ... even if you don't want any of these things in particular, hopefully this is 136 | a good example of how you can use the various parts of the IPLD system to accomplish your own goals! 137 | 138 | The whole IPFS web gateway system was built in terms of standard IPLD components, assembled: 139 | schemas, ADLs, and plain basic pathing and traversal over the Data Model were all we needed to accomplish all this. 140 | 141 | 142 | What other limits does this have? 143 | --------------------------------- 144 | 145 | It's important to note that the web gateways do almost all of this work 146 | using IPLD libraries and not much else... but, ADLs also have some special caveats. 147 | 148 | Because ADLs need some logic to make their translations of data work, 149 | and because IPLD doesn't itself include an interpreter or scripting language runtime 150 | (and even if it did, we'd be worried about capping its resource budget)... 151 | IPLD libraries typically require that a list of allowed ADLs be defined explicitly. 152 | 153 | The IPFS web gateways specifically allow the FBL and HAMT ADLs. 154 | 155 | If you wanted to produce data that used other sharding algorithms, 156 | but was otherwise structurally matching the Unixfsv2 schema, 157 | and wanted those to be correctly displayed by some IPFS web gateway? 158 | You'd need to reconfigure that gateway to expand the list of allowed ADLs 159 | to include whatever sharding system it is your wanted to use. 160 | 161 | 162 | Historically: how did this previously work? 163 | ------------------------------------------- 164 | 165 | > :closed_book: 166 | > This section is historical notes -- it's here for readers who were familiar 167 | > with the old system, and want to be refreshed on how it compares to the new 168 | > designs documented above, or for readers who want to understand some of the 169 | > learning experiences gathered from previous bouts at this work. 170 | > However, you shouldn't really need to read it unless one of those two 171 | > sources of curiosity appeals to you. 172 | 173 | The earliest implementations of IPFS used a codec called "dag-pb" to represent files and directories. 174 | ("dag-pb" is still around as an IPLD codec -- but nowadays, we consider it one of the more limited codecs, and a very esoteric one at that.) 175 | 176 | The "dag-pb" codec included many responsibilities that we now handle separately: 177 | it included the concept of directories (and sharding them) in the codec itself (whereas we now split this out into map ADLs); 178 | similarly, it include the concept of files (and sharding their component bytes) in the codec itself (whereas we now split this out into bytes ADLs). 179 | 180 | By baking these things into the codec itself, there were several major disadvantages compared to the present IPLD system design: 181 | 182 | - by baking these sharding mechanisms into a single codec, it was not possible to reuse them with other codecs; 183 | - because the sharding mechanisms were baked in the codec (and at the time, 184 | there wasn't really any coherent definiton of a Data Model at all), 185 | there was no way to _switch_ between high level and raw views of the data -- 186 | as a result, these systems were much harder to debug; 187 | - because the sharding mechanisms were tightly bound into the codec, 188 | there was no way to introduce or even experiment with other sharding mechanisms! 189 | 190 | Looking at the other direction, "dag-pb", even with all its special features, was more limited than the modern IPLD system designs: 191 | 192 | - by extracting the concept of ADLs, we've made it possible to shard other datastructures as well -- 193 | for example, we can now easily support sharded lists, a concept which simply didn't exist in dag-pb. 194 | - dag-pb had very (very) rigid ideas about the topology of data. 195 | In modern terms, we would say it doesn't support the complete IPLD Data Model. 196 | The huge behavioral gaps and asymmetry in representability between dag-pb and 197 | other early IPLD codecs produced a great deal of confusion. 198 | -------------------------------------------------------------------------------- /tutorial.md: -------------------------------------------------------------------------------- 1 | 2 | !!! 3 | 4 | This document has **moved**. 5 | 6 | You'll now find information like this in the [ipld/ipld](https://github.com/ipld/ipld/) meta-repo, 7 | and published to the web at https://ipld.io/ . 8 | 9 | All documentation, fixtures, specifications, and web content is now gathered into that repo. 10 | Please update your links, and direct new contributions there. 11 | 12 | !!! 13 | 14 | ---- 15 | 16 | # Thinking in data structures 17 | 18 | You're already using many data structures in your programs. Here's 19 | an example. 20 | 21 | 22 | 33 | 40 | 47 | 48 | 49 | Let's think about what this really is. 50 | 51 | We've taken some strings and we've assigned them to properties. 52 | If we didn't assign them to properties, how would we know what they mean? 53 | We only know what the values mean because we've assigned them to 54 | properties that mean something. 55 | 56 | What would it mean if we were to assign a value of `null` to the "twitter" 57 | property? It would mean that person isn't on twitter! Both the value **and** where 58 | we put that value have meaning when applied **together**. 59 | 60 | If you pass this data structure to another function, it can tell who this person is 61 | because it contains references to the information about that person. In IPLD, 62 | we would call `person` a **node**, and the properties that person contains references 63 | to would also be **nodes**. If we're talking about everything this person references, 64 | and everything those references might reference, we would call that the person's **graph**. 65 | 66 | Now let's build a data structure that captures some relationships between 67 | nodes. 68 | 69 | 70 | 90 | 97 | 104 | 105 | 106 | Let's explore our graph of the galaxy. It has two properties, one for each planet 107 | we've decided to index. If I want to see who is on planet earth, I check that property. 108 | 109 | I know that Mikeal, Eric, and Volker are all on the same planet not because of a property 110 | *inside* their graph but because of the **structure** of the graph that refers to them. 111 | 112 | There's a huge world of data structures out there, some of which you may be familiar with 113 | and some that you may not be familiar with yet. Something you need to keep in mind 114 | as you learn IPLD is that **how you *arrive* at information is often as important, if not more 115 | important, than the information you arrive at**. 116 | 117 | ## Addressing 118 | 119 | Depending on what language you program in, you may be familiar with the concept of a pointer. 120 | If you aren't, it's just a way to reference data in one place while it lives in another place. 121 | URLs are like pointers for The Web, they let you put a reference on one web page to a web page 122 | that could be on the other side of the world. 123 | 124 | In IPLD we use a standard called [CID](https://proto.school/#/anatomy-of-a-cid/01) 125 | for linking data together. CID's act like pointers **and** they act like URLs. 126 | They let us safely reference data that might be on the other side of the 127 | internet, or locally in memory, or anywhere else. **This is how we decentralize data structures!** 128 | 129 | 130 | 149 | 156 | 163 | 164 | 165 | Prints 166 | ``` 167 | bagaaierakbgholyvqnp2kjpr5ep6yh6u4uhop7cv7wvabydbctksletsilkq 168 | ``` 169 | 170 | This is the **address** (CID) of this data. 171 | 172 | The first thing to notice about this code is that we didn't *assign* a key to this data. We 173 | encoded the data as `'json'` and then we got the data's address. 174 | 175 | A CID tells us two things, it tells us how the data was encoded (`json`) and it tells us the 176 | **hash** of the data. With this identifier, I can store this data anywhere on the internet. 177 | I can then retreive this data from anyone on the internet **without trusting them** 😱 178 | 179 | Let's look back at a few things we did. 180 | 181 | 182 | 189 | 196 | 203 | 204 | 205 | We encode this data using JSON. Now it has been serialized to a binary representation we 206 | can store and/or send to someone else. 207 | 208 | 209 | 216 | 223 | 230 | 231 | 232 | Then we're going to do a cryptographic computation on that serialized binary which gives us 233 | a **globally unique** identifier for that binary. The CID tells us "this is JSON data 234 | that matches this **hash**." 235 | 236 | We can now ask any random device on the internet "do you have this CID?" 237 | 238 | If they say "yes, here's that data" I can receive the data without needing to trust them 239 | because I'm going to compute the hash of that data after I receive it and if it doesn't 240 | match the CID's hash I know they're lying. 241 | 242 | Once I have binary data that matches the CID, I know that I need to decode this data with JSON 243 | because the CID told me it was encoded with JSON. When one person sends another person data 244 | this way, the in-memory representations of these data structures are identical on each machine. 245 | 246 | It's worth taking a break and considering what this means. 247 | 248 | Hashes let anyone exchange **data** around the internet. CID's let anyone exchange **data structures** 249 | around the internet. 250 | 251 | # Linking 252 | 253 | Here's where things get **really** interesting. 254 | 255 | 256 | 281 | 288 | 295 | 296 | 297 | Now we have a data structure made of **blocks**. In IPLD, we call each hashed piece of data a **block**. 298 | A block could have one or many nodes *inside that block* (like a large nested JSON object). 299 | Most importantly, it can **link** to other **blocks**. 300 | 301 | In the above example, we created a **block** for each person. We then use the CID for each **block** 302 | to reference every person in our index of the galaxy. Note that we can't use JSON to encode the index 303 | of the galaxy and we use `'dag-cbor'` instead. JSON has a strictly defined set of types and our Link 304 | data type (CID) isn't in that set, so we use a *"codec"* the IPLD team wrote called `dag-cbor` because 305 | it knows how to represent links from any CID. You can use `dag-cbor` to encode any data you can 306 | encode with JSON as well as Links (CIDs) to other blocks. 307 | 308 | Let's pause for a moment and talk about what this means. 309 | 310 | CIDs can be encoded for any existing serialization and **any future** serialization format. 311 | More importantly, a data structure can freely create links *between* these serializations. If you 312 | have JSON data you don't need to change it or re-encode it, you can still create new data structures 313 | that reference that data using links. In addition to `JSON` there are [IPLD codecs for `git`, `bitcoin`, 314 | `zcash`, `ETH`, and more](). 315 | 316 | Now, let's move to a new computer and ask the internet for this index of the galaxy we just created. 317 | 318 | Once we have the block data we can decode it with `dag-cbor` into an in-memory data structure. We can 319 | traverse it and see all the links to other data. 320 | 321 | Before we ask the internet for the data of all the people in the galaxy, what can we say with **only** 322 | this index data? What can we learn from just looking at these hash based links? 323 | 324 | ```js 325 | { americans: [ Hash(0003), Hash(0001), Hash(0004) ], 326 | inAmerica: [ Hash(0001) ], 327 | onEarth: [ Hash(0001), Hash(0002), Hash(0003) ], 328 | onMars: [ Hash(0004) ] 329 | } 330 | ``` 331 | 332 | Wow, quite a lot actually: 333 | 334 | * There is 1 person in America and they are an American. 335 | * There are 2 Americans that are not in America. 336 | * The person on Mars is an American. 337 | * I know how many people are indexed on each planet. 338 | * There are only 4 unique people in this index. 339 | 340 | This is a limited index, with a very limited set, and none of the links have been traversed, 341 | but we know quite a lot already! 342 | 343 | Let's pretend that the data about Mars is really on Mars, and the data about earth is on Earth. 344 | So if we want to know "who is on Mars" we'll have to ask for that data from Mars, which is going 345 | to take a while. But what if we want to know "who is **not** on Mars?" To learn that we only 346 | need to request the data we have on our own planet! 347 | 348 | Something interesting is happening here. 349 | 350 | Everyone who requests the data we just looked at will get the same thing. We're all reading the 351 | same *"state"* of this index of the galaxy. We learn what we want to learn from this index and 352 | we only need to request the parts of the data stucture that are required for us to answer the 353 | questions we have. If this were a typical database it would need **all** the data in order to 354 | provide answers to these questions. We wouldn't be able to distribute the parts of this data 355 | structure to who needs it. We wouldn't be able to build local offline caches, or caches in CDN's, 356 | because the database can't share *part* of its larger global state. 357 | 358 | With IPLD, if people on Mars only care about the data about Mars, and people on Earth only care 359 | about data on Earth, they can very efficiently localize the data they access the most. 360 | 361 | **This is how we're going to decentralize The Web**, by building data structures we can share 362 | and distribute across The Internet. 363 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | 2 | !!! 3 | 4 | This document has **moved**. 5 | 6 | You'll now find information like this in the [ipld/ipld](https://github.com/ipld/ipld/) meta-repo, 7 | and published to the web at https://ipld.io/ . 8 | 9 | All documentation, fixtures, specifications, and web content is now gathered into that repo. 10 | Please update your links, and direct new contributions there. 11 | 12 | !!! 13 | 14 | ---- 15 | 16 | # InterPlanetary Linked Data 17 | 18 | Welcome to the internet of data structures! 19 | 20 | ## What is IPLD? 21 | 22 | IPLD isn't a project or a library, it's an ecosystem. An 23 | ecosystem of formats and data structures for building 24 | applications that can be fully decentralized. 25 | 26 | This ecosystem is held together by a few concepts and 27 | standards that ensure compatibility between formats 28 | and programming languages. 29 | 30 | You'll find numerous libraries for working with IPLD. 31 | You may even create a few of your own. That library is 32 | part of the IPLD ecosystem and you're now a part of its 33 | community. You don't need to be a committer in a repository 34 | in the IPLD GitHub org to be a part of this community, 35 | you're already almost there just by reading this far. 36 | 37 | ## From Data to Data Structures 38 | 39 | Hashes are wonderful. 40 | 41 | If you hash something you can give that hash to anyone else 42 | in the world, anywhere in the galaxy (IP stands for InterPlanetary), 43 | they can tell if they have that data and when they send 44 | it to you, you're able to verify it was the right data even 45 | without trusting that person. 46 | 47 | If you think of hashes like URLs, you can pass them around and 48 | ask people for them without fixing the place where that data exists 49 | into the link the way you do with a URL. This is really powerful, 50 | but by itself it's a bit limited. 51 | 52 | For one thing, there are lots of hash algorithms, and they keep 53 | getting better. A hash doesn't say *what* sort of hash it is. We 54 | solved that with something called [`multihash`](https://github.com/multiformats/multihash) 55 | which is just a hash that says "I'm a hash of this particular 56 | algorithm." 57 | 58 | So now, with multihash, a single identifier can get us any set 59 | of binary data (what we call a "Block") from anywhere in the world. 60 | 61 | That's *data*, we've got that covered, but now we need to go from 62 | *data* to **data structures**. 63 | 64 | This means that we have to think about different formats (JSON, CBOR, 65 | Bitcoin, Ethereum, etc) and we also need to think about different 66 | programming languages (JavaScript, Go, Rust, etc) because these 67 | languages have different basic types that are used to represent data 68 | structures in-memory. 69 | 70 | That's IPLD. 71 | 72 | IPLD is how we go from *data* to **data structures**. 73 | 74 | **We do this with a new link type and a data model.** 75 | 76 | ### Links 77 | 78 | For links we use a [CID](https://specs.ipld.io/block-layer/CID.html). A CID is an extension of `multihash`, 79 | in fact a `multihash` is part of a `CID`. We simply add a *codec* to a `multihash` 80 | that tells us what format the data is in (JSON, CBOR, 81 | Bitcoin, Ethereum, etc). This way, we can actually link between 82 | data in different formats and any link to data anyone ever gives 83 | us can be decoded so that it can become more than just a series 84 | of bytes. 85 | 86 | CID is a [standard](https://github.com/multiformats/cid) that anyone can implement, even people that 87 | have no other interest in IPLD beyond the need for hash links to 88 | different data types can use it. 89 | 90 | ### The Data Model 91 | 92 | The IPLD Data Model is a bit more conceptual and can be applied 93 | to formats that have never considered IPLD. 94 | 95 | Consider that every programming language has a library for JSON encoding and decoding. These 96 | languages all adapt the JSON data model (types) into the 97 | best fit for that language. A "number" in JSON has many potential mappings in 98 | different languages, a JSON library has to decide how it is most appropriately represented. 99 | Our data model is very similar to the JSON data model, but we add a few more types. 100 | 101 | First, we add a **link** type using CID as our link format. We also 102 | add a **binary** type, which are missing from JSON. We support nothing 103 | beyond these and the other types that JSON has already defined quite well. 104 | 105 | We then apply this data model to more than just our own formats. We 106 | have specifications and libraries for interpreting many formats into 107 | this data model. For instance, Bitcoin blocks know nothing about IPLD 108 | since they were designed long before IPLD existed. We have a decoder 109 | for Bitcoin that presents the data found in a Bitcoin block as the IPLD 110 | Data Model. 111 | 112 | This provides a consistent interpretation of all formats across all programming 113 | languages in IPLD terms. For instance, Bitcoin data includes hashes to identify 114 | blocks and transactions to form a heavily interlinked blockchain. Bitcoin doesn't 115 | use CID's internally, but when you use an IPLD codec to read Bitcoin data, the 116 | hashes will be normalized to CIDs so we maintain a consistent "link" type that 117 | includes the content address (hash), the hash algorithm (double-SHA2-256) and the 118 | IPLD codec (e.g. `bitcoin-block` or `bitcoin-tx`). 119 | 120 | This lets you build data structures that freely link between formats 121 | that can be traversed through a normalized data model. Effectively, 122 | you can write format agnostic tools for reading, distributing, and 123 | persisting all content addressed data. Your custom IPLD data structure, encoded 124 | in your chosen IPLD codec, can link to anyone else's IPLD data, arbitrary 125 | Bitcoin blocks, Git commits and a universe of content addressed data. 126 | 127 | ## Getting Started 128 | 129 | ### Tutorial 130 | 131 | We highly recommend you read the following tutorial 132 | before trying to use IPLD. This tutorial is language 133 | agnostic and includes examples in JavaScript, Go and Rust. 134 | 135 | * [Thinking in Data Structures](./tutorial.md) 136 | * [Addressing](./tutorial.md#addressing) 137 | * [Linking](./tutorial.md#linking) 138 | 139 | ### Programming language guides 140 | 141 | When you have understood the basics of IPLD, you will need to familiarize 142 | yourself with the available IPLD libraries available for your programming 143 | language. Each language is different in the way they present and interact with 144 | IPLD data. 145 | 146 | * [Getting started in JavaScript](./getting-started/js) 147 | * [Persisting IPLD data in `js-ipfs` and `js-ipfs-lite`](./getting-started/js#storing-ipld-data-in-ipfs) 148 | * [Getting started in Go](./getting-started/go) 149 | * [Persisting IPLD data in `go-ipfs`](./getting-started/go#IPFS) 150 | 151 | ## Media 152 | 153 | [Talks, blog posts, and other media about IPLD.](./docs/media) 154 | 155 | ## Further Reading 156 | 157 | * **[Getting Things Done with IPLD](./docs/gtd)**: How you approach IPLD depends 158 | on the problem you are trying to solve. Read our guide to understand how IPLD 159 | can be best used to solve your data problems. 160 | * **[IPLD in the InterPlanetary Ecosystem](./docs/ecosystem.md)**: The array 161 | of new technologies that IPLD relates to can be confusing at first. Read our 162 | guide to understand how IPLD relates to **IPFS**, **libp2p**, **CIDs**, 163 | **multiformats** and more. 164 | * **[Objectives and Scope of IPLD](./docs/objectives)**: Why are we pursuing 165 | this project and where do we hope it to go? Read our guide to understand what 166 | guides our development efforts. 167 | * **[IPLD Schemas](https://specs.ipld.io/schemas/)**: Describing the shape and 168 | layout of data structures is important for documentation, communication, 169 | validation and much more. Read about our the Schema language designed around 170 | the IPLD Data Model. 171 | * **[Advanced Data Layouts](./docs/advanced-layouts)**: A large amount of the 172 | power afforded by IPLD is in creating complex and potentially very large data 173 | structures. Advanced Data Layouts (ADLs) are how we formally define them and 174 | the logic used to create and traverse linked data and create abstractions that 175 | are familiar to any programmer (e.g. a "map" that holds millions of records on 176 | the persistent web). 177 | 178 | ### FAQ 179 | 180 | [Frequently Asked Questions](./FAQ) 181 | 182 | ## Glossary 183 | 184 | ### Block 185 | 186 | The term "block" is used to refer to the pairing of a raw binary and a CID. 187 | 188 | Most users don't work directly with raw blocks. Instead, block binary data is 189 | encoded and decoded to and from the IPLD [Data Model](#data-model). It's only if 190 | writing a storage system such as a database, a filesystem or a replicator that 191 | you will work directly with blocks. 192 | 193 | The term "block" is sometimes loosely used to refer to the Data Model 194 | instantiation of decoded raw block data. While this is often useful when 195 | conceptualizing data and the way links are composed and traversed, the strict 196 | definition of "block" is intended to refer to the `Binary:CID` pairing. 197 | 198 | See the [Block specification](https://specs.ipld.io/block-layer/block.html) for 199 | further details. 200 | 201 | ### CID 202 | 203 | CID is a **Content IDentifier**. It's a self-describing data structure identifier. In other 204 | words, it's a hash that says what kind of hash it is and how to decode the binary 205 | data identified by the hash. 206 | 207 | See the [CID specification](https://specs.ipld.io/block-layer/CID.html) further 208 | details. In addition, the [full format specification](https://github.com/multiformats/cid) 209 | is part of the multiformats project and should be referred to when creating a 210 | CID implementation for a new language. 211 | 212 | ### Codec 213 | 214 | The term "codec" refers to a [mutlicodec]() which defines a specific binary data 215 | encoder and/or decoder (usually both). 216 | 217 | A codec is responsible for encoding [Data Model](#data-model) instantiations into 218 | binary and decoding binary into the [Data Model](#data-model). 219 | 220 | Examples: 221 | 222 | * [DAG-CBOR](#dag-cbor) 223 | * [DAG-JSON](#dag-json) 224 | * bitcoin-block, bitcoin-tx 225 | * Git 226 | * CBOR 227 | * JSON 228 | 229 | #### DAG-CBOR 230 | 231 | DAG-CBOR is a codec that implements the IPLD Data Model as a subset of CBOR, with additional constraints 232 | for hash consistent representations. DAG-CBOR adds a "link" type using a CBOR 233 | tag. 234 | 235 | See the full [DAG-CBOR specification](https://specs.ipld.io/block-layer/codecs/dag-cbor.html) 236 | for further information, including details about features and types of CBOR 237 | encoding that are excluded from DAG-CBOR in order to make it a suitable format 238 | for content addressed data. 239 | 240 | #### DAG-JSON 241 | 242 | **Not recommended for production use** 243 | 244 | DAG-JSON is a codec that implements the IPLD Data Model using JSON serialization. 245 | DAG-JSON adds "link" and "binary" types as well as additional constraints for 246 | hash consistent representations. 247 | 248 | See the full [DAG-JSON specification](https://specs.ipld.io/block-layer/codecs/dag-json.html) 249 | for further information. 250 | 251 | ### Data Model 252 | 253 | The IPLD Data Model describes common base types that we call **kinds** ("types" 254 | is a term that we prefer to reserve for data structures described by IPLD Schemas). 255 | 256 | These *kinds* allow IPLD to create data structures using simple types 257 | accessible across many programming languages and encoding formats. 258 | 259 | Using the Data Model we can implement file systems, databases, and custom 260 | application data structures in a format agnostic way and even link between 261 | these structures and formats using common toolchains. 262 | 263 | All JSON types are Data Model kinds: 264 | 265 | * `boolean` (true, false) 266 | * `null` 267 | * `string` (utf8) 268 | * numbers: 269 | - `integer` 270 | - `float` 271 | * `map` 272 | * `list` 273 | 274 | In addition, there is a `bytes` kind for raw binary data. 275 | 276 | Finally, there is a `link` kind represented by a [CID](#CID). 277 | 278 | Read the full [IPLD Data Model specification](https://specs.ipld.io/data-model-layer/data-model.html) 279 | for further details. 280 | 281 | ### Nodes 282 | 283 | A "node" in IPLD is a point in a graph—an element of the Data Model in an 284 | instantiated data structure. A "block" will typically contain many nodes. Graph 285 | traversal involves navigating nodes across block boundaries and identifying specific 286 | elements within a Data Model representation. 287 | 288 | It is useful to consider the Data Model as being formed of "scalar" and "recursive" 289 | kinds when considering nodes and possible traversal. "Scalar" kinds are terminal 290 | nodes in the Data Model: null, boolean, integer, float, string, bytes. "Recursive" kinds 291 | can contain other kinds within them and therefore allow deeper traversal: map and list. 292 | The link kind is scalar, but is typically treated as a transparent node for the purpose 293 | of traversal such that data spanning many blocks can be addressed as a single graph of 294 | nodes. 295 | 296 | ### HAMT 297 | 298 | HAMT stands for ["Hash Array Mapped Trie"](https://en.wikipedia.org/wiki/Hash_array_mapped_trie). 299 | 300 | HAMT's in IPLD differ from typical in-memory implementation as we require the use of "buckets" 301 | to manage performance and use links to connect blocks to form potentially very large collections of data. 302 | 303 | Read our [IPLD Hashmap (HAMT) specification](https://specs.ipld.io/data-structures/hashmap.html) 304 | for furthe details. 305 | 306 | ### Graphsync 307 | 308 | Graphsync is a replication protocol for IPLD. It's used heavily by [Filecoin](https://filecoin.io) and experimentally by 309 | IPFS. 310 | 311 | Graphsync uses [IPLD Selectors](#selectors) to replicate sections of a larger IPLD graph. 312 | 313 | Read the full [Graphsync specification](https://specs.ipld.io/block-layer/graphsync/graphsync.html) 314 | for further details. 315 | 316 | ### Selectors 317 | 318 | Selectors define selection criteria for IPLD graph queries. 319 | 320 | IPLD Selectors are a declarative format for specifying a walk over a [Data Model](#data-model) 321 | graph—what nodes to walk over (or not), recursively; and, some positions to "visit" (with a callback, typically, though library implementation details may vary). 322 | You can think of Selectors as being roughly like "regexps for graphs". 323 | 324 | Selectors are natively implemented in most IPLD libraries (for performance reasons), 325 | but the format itself is standardized. 326 | The format is described in IPLD (using [IPLD Schemas](https://specs.ipld.io/schemas/), 327 | so it's possible to serialize Selectors in any [Codec](#codecs) you want, 328 | and it's also possible to inspect (and transform!) Selector documents using standard [Data Model](#data-model) tools. 329 | 330 | Read the full [IPLD Selectors specification](https://specs.ipld.io/selectors/selectors.html) 331 | for further details. 332 | 333 | ## Specifications 334 | 335 | You can refer to the [specifications website](https://ipld.github.io/specs/) for a full list of content. 336 | 337 | There are numerous links to specifications throughout this documentation site. 338 | 339 | The following curated list of specification pages may be of interest to those 340 | seeking additional forms of documentation: 341 | 342 | * **[IPLD Exploratory Specifications](https://specs.ipld.io/design/)**: 343 | Various documents about the design, and history of design thought, in and around IPLD. 344 | * **[IPLD Foundational Principles](https://specs.ipld.io/FOUNDATIONS.html)**: 345 | The foundation principals of globally content addressed data structures. 346 | * **[Core IPLD team members](https://specs.ipld.io/#ipld-team)**. 347 | --------------------------------------------------------------------------------