├── .gitignore ├── src ├── headless-cms.md ├── usecases │ ├── crud.md │ ├── zettelkasten.md │ ├── self-integrating-systems.md │ ├── governmental-open-data.md │ ├── job-matching.md │ ├── product-lifecyles.md │ ├── standardization-bodies.md │ ├── vocabulary-management.md │ ├── medical.md │ ├── social.md │ ├── science.md │ ├── self-integrating-applications.md │ ├── intro.md │ ├── personal-data-store.md │ ├── react.md │ ├── knowledge-management.md │ ├── ai.md │ ├── education.md │ ├── food-labels.md │ ├── verifiable-credentials.md │ ├── data-catalog.md │ ├── headless-cms.md │ ├── surveys.md │ └── e-commerce.md ├── 404.md ├── interoperability │ ├── verifiable-credentials.md │ ├── git.md │ ├── privacy.md │ ├── intro.md │ ├── vc-old.md │ ├── upgrade.md │ ├── ipfs.md │ ├── graph-database.md │ ├── json.md │ └── sql.md ├── links.md ├── framework.md ├── newsletter.md ├── extended.md ├── get-involved.md ├── extended-table.md ├── commits │ ├── versioning.md │ ├── ownership.md │ ├── suggestions.md │ ├── intro.md │ └── concepts.md ├── plugins.md ├── runtime.md ├── acknowledgements.md ├── schema │ ├── translations.md │ ├── migrations.md │ ├── compare.md │ ├── collections.md │ ├── datatypes.md │ ├── intro.md │ ├── faq.md │ └── classes.md ├── atomizing.md ├── trust.md ├── when-to-use.md ├── invitations.md ├── agents.md ├── endpoints.md ├── websockets.md ├── core │ ├── querying.md │ ├── paths.md │ ├── serialization.md │ ├── json-ad.md │ └── validations.md ├── SUMMARY.md ├── files.md ├── get-started.md ├── hierarchy.md ├── atomic-data-overview.md ├── tooling.md ├── create-json-ad.md ├── experimental-serialization.md ├── assets │ ├── atomic_data_logo.svg │ └── atomic_data_logo_padded.svg └── roadmap.md ├── examples └── atomic-fs │ └── people │ ├── mary.afd │ └── john.afd ├── theme ├── favicon.png └── favicon.svg ├── README.md ├── open-in.css ├── .vscode └── tasks.json ├── book.toml ├── .github └── workflows │ └── deploy.yml └── LICENSE /.gitignore: -------------------------------------------------------------------------------- 1 | /book 2 | .DS_Store 3 | -------------------------------------------------------------------------------- /src/headless-cms.md: -------------------------------------------------------------------------------- 1 | # Using as a Headless CMS 2 | -------------------------------------------------------------------------------- /src/usecases/crud.md: -------------------------------------------------------------------------------- 1 | # Atomic-Server for CRUD applications 2 | -------------------------------------------------------------------------------- /examples/atomic-fs/people/mary.afd: -------------------------------------------------------------------------------- 1 | firstName: Mary 2 | lastName: Amberson 3 | birthDate: 1993-10-13 4 | -------------------------------------------------------------------------------- /theme/favicon.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/atomicdata-dev/atomic-data-docs/HEAD/theme/favicon.png -------------------------------------------------------------------------------- /src/usecases/zettelkasten.md: -------------------------------------------------------------------------------- 1 | {{#title Atomic Data for Zettelkasten}} 2 | # Atomic Data for Zettelkasten 3 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # MOVED TO [ATOMIC-SERVER REPO](https://github.com/atomicdata-dev/atomic-server/tree/develop/docs) 2 | -------------------------------------------------------------------------------- /src/404.md: -------------------------------------------------------------------------------- 1 | ## Page is not found. 2 | 3 | Try **searching in the top bar**, the document you're looking for may have been moved! 4 | -------------------------------------------------------------------------------- /src/usecases/self-integrating-systems.md: -------------------------------------------------------------------------------- 1 | See https://www.nitrd.gov/nitrdgroups/images/b/ba/Steven_ray_the_future_of_software.pdf 2 | -------------------------------------------------------------------------------- /open-in.css: -------------------------------------------------------------------------------- 1 | footer { 2 | font-size: 0.8em; 3 | text-align: center; 4 | border-top: 1px solid black; 5 | padding: 5px 0; 6 | } 7 | -------------------------------------------------------------------------------- /src/usecases/governmental-open-data.md: -------------------------------------------------------------------------------- 1 | # Publishing governmental Open Data as Atomic Data 2 | 3 | - More information is more better 4 | - 5 | -------------------------------------------------------------------------------- /src/interoperability/verifiable-credentials.md: -------------------------------------------------------------------------------- 1 | {{#title How does Atomic Data relate to Verifiable Crendentials?}} 2 | # Verifiable Credentials 3 | -------------------------------------------------------------------------------- /src/usecases/job-matching.md: -------------------------------------------------------------------------------- 1 | WIP 2 | 3 | # Atomic Data for job matching and vacancies 4 | 5 | - https://sfia-online.org/en/about-sfia/sfia-guiding-principles 6 | - 7 | -------------------------------------------------------------------------------- /.vscode/tasks.json: -------------------------------------------------------------------------------- 1 | { 2 | "version": "2.0.0", 3 | "tasks": [ 4 | { 5 | "type": "shell", 6 | "label": "docs atomic data (mdbook serve)", 7 | "command": "mdbook serve", 8 | } 9 | ] 10 | } 11 | -------------------------------------------------------------------------------- /src/usecases/product-lifecyles.md: -------------------------------------------------------------------------------- 1 | - Link real-world objects to Atomic IDs 2 | - Track and prove ownership to prevent buying something that was stolen 3 | - See where parts are coming from, monitor the supply chain 4 | -------------------------------------------------------------------------------- /src/links.md: -------------------------------------------------------------------------------- 1 | # List of links of Atomic Data on the web 2 | 3 | - [Awesome Knowledge Graph](https://github.com/totogo/awesome-knowledge-graph) 4 | - [Awesome Semantic Web](https://github.com/semantalytics/awesome-semantic-web) 5 | -------------------------------------------------------------------------------- /src/usecases/standardization-bodies.md: -------------------------------------------------------------------------------- 1 | {{#title Atomic Data for Standardization bodies}} 2 | # Atomic Data for Standardization bodies 3 | 4 | Most industry domains work on standardization. 5 | (health, pharmaseuyical studies 6 | -------------------------------------------------------------------------------- /src/usecases/vocabulary-management.md: -------------------------------------------------------------------------------- 1 | {{#title Atomic Data for Vocabulary & Taxonomy management}} 2 | # Atomic Data for Vocabulary & Taxonomy management 3 | 4 | Describing abstract concepts, data models, and terms in a consistent way helps organisations 5 | -------------------------------------------------------------------------------- /src/framework.md: -------------------------------------------------------------------------------- 1 | # Atomic Data as a Framework 2 | 3 | Atomic Data is designed as a _specification_, which simply sets the rules. 4 | Implementing these rules can be a costly endeavor, which is why we've made some tooling available that helps you make other apps. 5 | -------------------------------------------------------------------------------- /src/interoperability/git.md: -------------------------------------------------------------------------------- 1 | {{#title Atomic Data and Git}} 2 | # Atomic Data and Git 3 | 4 | ## How to manage Atomic Data using GIT 5 | 6 | Although Git started as in invention to manage changes to the 7 | 8 | ## Similarities / differences between Git and Atomic Data + Commits 9 | 10 | Similarities: 11 | 12 | - Atomic Commits and Git Commits both use hashes to identify previous state 13 | - Both allow for decentralized storage 14 | - Both work with 15 | -------------------------------------------------------------------------------- /src/usecases/medical.md: -------------------------------------------------------------------------------- 1 | # Atomic Data for personalized health data storage 2 | 3 | - Store personal health data on your atomic server 4 | - Improve interoperability between medical measurement devices (e.g. blood sugar monitors, fitness tracking data, heart rate, weight) by standardizing these concepts using Atomic Schema. 5 | - Use Atomic Runtimes to give researchers the option to query your pod and find answers to research questions, without sacrificing privacy 6 | -------------------------------------------------------------------------------- /src/usecases/social.md: -------------------------------------------------------------------------------- 1 | # A decentralized Social Network powered Atomic Data 2 | 3 | Instead of only storing posts on twitter, facebook, or some other platform with advertisement incentives, people would store and control their own social media posts, and determine who gets access to which. 4 | Users would get full control over the way they view posts, and how they are ranked. 5 | This means no more advertisements, and therefore no dark incentives to maximize engagement. 6 | -------------------------------------------------------------------------------- /src/interoperability/privacy.md: -------------------------------------------------------------------------------- 1 | # Atomic Data - privacy and encryption 2 | 3 | Atomic Data is great for sharing, and therefore great as a format for Open Data. 4 | However, Atomic Data can also be used to share secret and private stuff. 5 | 6 | ## Using Encryption 7 | 8 | Encryption turns readable text into gibberish using some specific Key. 9 | The counterpart of that key can be used to _decrypt_ that gibberish, and turn it into its usable form. 10 | 11 | TODO! 12 | -------------------------------------------------------------------------------- /src/newsletter.md: -------------------------------------------------------------------------------- 1 | {{#title Official Atomic Data newsletter}} 2 | # Subscribe to the Atomic Data newsletter 3 | 4 | We'll send you an update (max once per month) when there's something relevant to share, such as 5 | 6 | - Major changes to the specification 7 | - Major new releases (with new features) 8 | - Use-cases, implementations 9 | - Tutorials, blog posts 10 | - Organizational / funding news 11 | 12 | [Click here to sign up to the Atomic Data Newsletter](http://eepurl.com/hHcRA1) 13 | -------------------------------------------------------------------------------- /src/extended.md: -------------------------------------------------------------------------------- 1 | {{#title Atomic Data Extended specification}} 2 | # Atomic Data Extended 3 | 4 | Atomic Data is a _modular_ specification, which means that you can choose to implement parts of it. 5 | All parts of Extended are _optional_ to implement. 6 | The _Core_ of the specification (described in the previous chapter) is required for all of the Extended spec to work, but not the other way around. 7 | 8 | However, many of the parts of Extended do depend on _eachother_. 9 | 10 | {{#include extended-table.md}} 11 | -------------------------------------------------------------------------------- /book.toml: -------------------------------------------------------------------------------- 1 | [book] 2 | title = "Atomic Data Docs" 3 | authors = ["Joep Meindertsma"] 4 | description = "Documentation for the Atomic Data standard." 5 | language = "en" 6 | 7 | [output.html] 8 | git-repository-icon = "fa-github" 9 | git-repository-url = "https://github.com/atomicdata-dev/atomic-data-docs" 10 | additional-css = ["open-in.css"] 11 | 12 | [output.linkcheck] 13 | optional = true 14 | 15 | [preprocessor.open-on-gh] 16 | command = "mdbook-open-on-gh" 17 | renderer = ["html"] 18 | text = "mdbook-open-on-gh" 19 | open-on-text = "[Suggest edits for this page on GitHub.]" 20 | -------------------------------------------------------------------------------- /src/usecases/science.md: -------------------------------------------------------------------------------- 1 | # Scientific publications powered by Atomic Data 2 | 3 | - Share and use concepts as URLs, which means they are always explicitly defined. 4 | - Attribution is improved 5 | - Reason about causal relationships that cross article boundaries, explore correlation graphs to gain new insights 6 | 7 | ## Challenges and opportunities 8 | 9 | - Scientific publications are a slow moving field 10 | - Publications tend to favor PDF, which is hard to make machine readable 11 | - Extending a syntax like LaTeX might provide a short path to referring to atomic data in publications 12 | - Getting scientists to host the atomic data is going to be one of the most difficult aspect 13 | -------------------------------------------------------------------------------- /.github/workflows/deploy.yml: -------------------------------------------------------------------------------- 1 | name: github pages 2 | 3 | permissions: write-all 4 | 5 | on: 6 | push: 7 | branches: 8 | - main 9 | 10 | jobs: 11 | deploy: 12 | runs-on: ubuntu-latest 13 | steps: 14 | - uses: actions/checkout@v2 15 | 16 | - name: Setup mdBook 17 | uses: peaceiris/actions-mdbook@v1 18 | with: 19 | mdbook-version: 'latest' 20 | 21 | - run: cargo install mdbook-open-on-gh 22 | 23 | - run: mdbook build 24 | 25 | - name: Deploy 26 | uses: peaceiris/actions-gh-pages@v3 27 | with: 28 | github_token: ${{ secrets.GITHUB_TOKEN }} 29 | publish_dir: ./book/html 30 | cname: docs.atomicdata.dev 31 | -------------------------------------------------------------------------------- /examples/atomic-fs/people/john.afd: -------------------------------------------------------------------------------- 1 | # This format should allow for comments like these, since it's meant to be a human to machine language. 2 | # Should values require quotes? Or only if its multi-line? 3 | firstName: John 4 | lastName: McLovin 5 | # If a Property is not available in the Class, you can the URL of the property 6 | https://schema.org/birthDate: 1991-01-20 7 | # Perhaps support relative paths 8 | bestFriend: ./mary 9 | # This is a TranslationBox 10 | # It actually represents a new Resource with a unique URL 11 | # Its URL can be generated deterministically by using the Shortname of the Property 12 | lifeStory: 13 | en: "John was born some day, and he will die some day!" 14 | nl: "John was ooit geboren, en hij zal ooit sterven!" 15 | -------------------------------------------------------------------------------- /src/usecases/self-integrating-applications.md: -------------------------------------------------------------------------------- 1 | # Self-integrating applications with Atomic Data 2 | 3 | Our digital workspaces are increasingly dependent on integrations. 4 | Receive ticket updates in your group chat app, send an e-mail when a new lead is added, save uploaded files to a specific folder in your cloud storage. 5 | Many SAAS platforms offer these types of integrations as simple click-to-enable features. 6 | However, writing integrations as a developer pretty much always involves manual labor. 7 | Integrations are costly, as developers will need to read API specifications, interpret them correctly, 8 | map the properties, implement the integration and then add tests to make sure it doesn't break. 9 | 10 | Tools like the OpenAPI specification make it easier to render 11 | 12 | We call application _self-integrating_ if there is no labor involved with linking one app to the other. 13 | -------------------------------------------------------------------------------- /src/get-involved.md: -------------------------------------------------------------------------------- 1 | {{#title Atomic Data - Get Involved}} 2 | # Get involved 3 | 4 | Atomic Data is an open specification, and that means that you're very welcome to share your thoughts and help make this standard as good as possible. 5 | 6 | Things you can do: 7 | 8 | - Join the [Discord server](https://discord.gg/a72Rv2P) for voice / text chat 9 | - Start playing with / contributing to [the implementations](tooling.md) 10 | - Drop an [issue on Github](https://github.com/ontola/atomic-data-docs/issues) to share your suggestions or criticism of this book / spec 11 | - Subscribe to the [newsletter](newsletter.md) 12 | - Join our [W3C Community Group](https://www.w3.org/community/atomic-data/) 13 | 14 | 20 | 21 | -------------------------------------------------------------------------------- /src/usecases/intro.md: -------------------------------------------------------------------------------- 1 | {{#title Various Use Cases for Atomic Data}} 2 | # Various Use Cases for Atomic Data 3 | 4 | Most of this book is either abstract or technical, but this section aims to be different. 5 | In this section, we'll present concrete examples of things that can be built with Atomic Data. 6 | Although you could use Atomic Data for pretty much any type of application, it is especially valuable where **data re-use**, **standardization**, and **data ownership** are important. 7 | 8 | 9 | * [As a Headless CMS](headless-cms.md) 10 | * [In a React project](react.md) 11 | * [Personal Data Store](personal-data-store.md) 12 | * [Artificial Intelligence](ai.md) 13 | * [E-commerce & marketplaces](e-commerce.md) 14 | * [Surveys](surveys.md) 15 | * [Verifiable Credentials](verifiable-credentials.md) 16 | * [Data Catalog](data-catalog.md) 17 | * [Education](education.md) 18 | * [Food labels](food-labels.md) 19 | -------------------------------------------------------------------------------- /src/extended-table.md: -------------------------------------------------------------------------------- 1 | - [Commits](commits/intro.md) communicate state changes. These Commits are signed using cryptographic keys, which ensures that every change can be audited. Commits are also used to construct a history of versions. 2 | - [Agents](agents.md) are Users that enable [authentication](authentication.md). They are Resources with their own Public and Private keys, which they use to identify themselves. 3 | - [Collections](schema/collections.md): querying, filtering, sorting and pagination. 4 | - [Paths](core/paths.md): traverse graphs. 5 | - [Hierarchies](hierarchy.md) used for authorization and keeping data organized. Similar to folder structures on file-systems. 6 | - [Invites](invitations.md): create new users and provide them with rights. 7 | - [WebSockets](websockets.md): real-time updates. 8 | - [Endpoints](endpoints.md): provide machine-readable descriptions of web services. 9 | - [Files](files.md): upload, download and metadata for files. 10 | -------------------------------------------------------------------------------- /theme/favicon.svg: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | -------------------------------------------------------------------------------- /src/commits/versioning.md: -------------------------------------------------------------------------------- 1 | # Atomic Data Versioning 2 | 3 | When Atomic Commits are applied to some Resource, the resource will change. 4 | However, its identifier (the Subject) will often remain the same. 5 | 6 | - Versioned representations should provide a link to the authority that might update it, and a link to where the latest version can be found. 7 | - The latest version should have a link to its permanent version. 8 | - Should [IPFS](../interoperability/ipfs.md) content-hash URLs be used for Versioned resources? 9 | 10 | ## Versioned Resources 11 | 12 | Properties: 13 | 14 | 15 | - latest: (ResourceArray, optional) 16 | - versions: (ResourceArray, optional) 17 | - currentVersion: (ResourceURL, required) 18 | 19 | ## Static Resource 20 | 21 | A static resource has a _content addressable_ URL, which means that its URL will never change. 22 | 23 | ## Hashing 24 | 25 | - Serialize all Atoms of the Subject (the entire Resource) as Atomic-NDJSON 26 | - Sort all lines (every atom) alphabetically 27 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2022 Joep Meindertsma 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /src/plugins.md: -------------------------------------------------------------------------------- 1 | # Atomic Plugins 2 | 3 | Atomic Plugins are applications that can run inside of an Atomic Server. 4 | They enhance the functionality of an Atomic Server. 5 | For example, they can: 6 | 7 | - Extend existing resources (e.g. automatic translations) 8 | - Provide new endpoints (maybe even ports?) with custom functionality (e.g. full text search for pod data, an e-mail server) 9 | - Periodically execute some code (e.g. fetch new data from a source) 10 | - Add datatypes and validation 11 | 12 | ## The Plugin Resource 13 | 14 | A Plugin itself is a Resource: it is described using Atoms. 15 | The most important Atom for a Plugin, is the `wasm` property: this contains the actual code. 16 | Other properties include: 17 | 18 | - `name` 19 | - `description` 20 | - `author` 21 | 22 | ## Registering a plugin 23 | 24 | When a plugin is installed, the Server needs to be aware of when the functionality of the plugin needs to be called: 25 | 26 | - Periodically (if so, when?) 27 | - On a certain endpoint (which endpoint? One or multiple?) 28 | - As a middleware when (specific) resources are created / read / updated. 29 | 30 | ## Hooks 31 | 32 | ### BeforeCommit 33 | 34 | Is run before a Commit is applied. 35 | Useful for performing authorization or data shape checks. 36 | -------------------------------------------------------------------------------- /src/runtime.md: -------------------------------------------------------------------------------- 1 | # Atomic Runtime 2 | 3 | _Status: not even ready to be part of the public docs_ 4 | 5 | Atomic Runtime offers a standardized way to bring algorithms to the data. 6 | This enables privacy-friendly analysis, where the analyzed data stays at the source and only the results of some algoritm are sent back. 7 | 8 | This is especially useful in research, where some data owners do not want to share their data with researches. 9 | Sharing confidential / privacy / sensitive data is inherently risky, so it is understandable that data owners might not want to share this. 10 | From the data owner perspective, it is safer to run any analysis locally, and send back the result to the researcher. 11 | This is what the Atomic Runtime allows for. 12 | 13 | ## Atomic Runtime Interface 14 | 15 | This is standardized set of query options that allow for fetching data from an Atomic Store. 16 | It uses the WASI (WebAssmemly Systems Interface) (TODO) 17 | 18 | ## Creating an Analysis 19 | 20 | - The Researcher writes code that interfaces on ART. 21 | - The Researcher issues an AnalysisRequest on the server of the Data Owner. 22 | - in a sandboxed environment which gives access to the data. 23 | 24 | ## Inspiration 25 | 26 | - Freek Dijkstra / SurfSare (implemented a similar python based solution) 27 | -------------------------------------------------------------------------------- /src/acknowledgements.md: -------------------------------------------------------------------------------- 1 | # Acknowledgements 2 | 3 | ## Authors: 4 | 5 | - **Joep Meindertsma** ([joepio](https://github.com/joepio/) from [Ontola.io](https://ontola.io/)) 6 | 7 | ## Special thanks to: 8 | 9 | - **Thom van Kalkeren** (my colleague, friend and programming mentor who came up with many great ideas on how to work with RDF, such as [HexTuples](https://github.com/ontola/hextuples) and [linked-delta](https://github.com/ontola/linked-delta)) 10 | - **Tim Berners-Lee** (for everything he did for linked data and the web) 11 | - **Ruben Verborgh** (for doing great work with RDF, such as the TPF spec) 12 | - **Pat McBennett** (for lots of valuable feedback on initial Atomic Data docs) 13 | - **Manu Sporny** (for his work on JSON-LD, which was an important inspiration for JSON-AD) 14 | - **Jonas Smedegaard** (for the various interesting talks we had and the feedback he provided) 15 | - **Arthur Dingemans** (for sharing his thoughts, providing feedback and his valuable suggestions) 16 | - **Anja Koopman** (for all her support, even when this project ate away days and nights of our time together) 17 | - **Alex Mikhalev** (for sharing many inspiring project and ideas) 18 | - **Daniel Lutrha** (for inspiring me to be more ambitious and for providing lots of technical ideas) 19 | - All the other people who contributed to linked data related standards 20 | -------------------------------------------------------------------------------- /src/schema/translations.md: -------------------------------------------------------------------------------- 1 | {{#title Atomic Data Translations}} 2 | # Atomic Translations 3 | 4 | _Status: design / concept stage_ 5 | 6 | Dealing with translations can be hard. 7 | 8 | ([Discussion](https://github.com/ontola/atomic-data/issues/6)) 9 | 10 | ## TranslationBox 11 | 12 | _URL: `https://atomicdata.dev/classes/TranslationBox` (does not resolve yet)_ 13 | 14 | A TranslationBox is a collection of translated strings, used to provide multiple translations. 15 | It has a long list of optional properties, each corresponding to some language. 16 | Each possible language Property uses the following URL template: `https://atomicdata.dev/languages/{langguageTag}`. 17 | Use a [BCP 47](http://www.rfc-editor.org/rfc/bcp/bcp47.txt) language tag, e.g. `nl` or `en-US`. 18 | 19 | For example: 20 | 21 | ```json 22 | { 23 | "@id": "https://example.com/john", 24 | "https://example.com/properties/lifestory": { 25 | "https://atomicdata.dev/languages/en": "Well, John was born and later he died.", 26 | "https://atomicdata.dev/languages/nl": "Tsja, John werd geboren en stierf later." 27 | }, 28 | } 29 | ``` 30 | 31 | Every single value used for Translation strings is an instance of the Translation class. 32 | 33 | A translation string uses the [MDString](https://atomicdata.dev/datatypes/markdown) datatype, which means it allows Markdown syntax. 34 | -------------------------------------------------------------------------------- /src/atomizing.md: -------------------------------------------------------------------------------- 1 | # Atomizing: How to create and publish Atomic Data 2 | 3 | Now that we're familiar with the basics of Atomic Data Core and its Schema, it's time to create some Atomic Data! 4 | We call the process of turning data into Atomic Data _Atomizing_. 5 | During this process, we **upgrade the data quality**. 6 | Our information becomes more valuable. 7 | Let's summarize what the advantages are: 8 | 9 | - Your data becomes **available on the web** (publicly, if you want it to) 10 | - It can now **link to other data**, an become part of a bigger web of data 11 | - It becomes **strictly typed**, so developers can easily and safely re-use it in their software 12 | - It becomes **easier to understand**, because people can look at the Properties and see what they mean 13 | - It can be **easily converted** into many formats (JSON, Turtle, CSV, XML, more...) 14 | 15 | ## Three ways to Atomize data 16 | 17 | In general, there are three ways to create Atomic Data: 18 | 19 | - [Using the **Atomic-Server** app + GUI](./atomic-server.md) (easy, only for direct user input) 20 | - [Create an **importable JSON-AD file**](./create-json-ad.md) (medium, useful if you want to convert existing data) 21 | - [Make your existing service / app **host and serialize Atomic Data**](./interoperability/upgrade.md) (hard, if you want to make your entire app be part of the Atomic Web!) 22 | -------------------------------------------------------------------------------- /src/usecases/personal-data-store.md: -------------------------------------------------------------------------------- 1 | {{#title Atomic Data for personal data stores}} 2 | # Atomic Data for personal data stores 3 | 4 | A Personal Data Store (or personal data service) is a place where you store all sorts of personal information. 5 | For example a list of contacts, todo items, pictures, or your profile data. 6 | Not that long ago, the default for this was the `my Documents` folder on your hard drive. 7 | But as web applications became better, we started moving our data to the cloud. 8 | More and more of our personal information is stored by large corporations who use the information to build profiles to show us ads. 9 | And as cloud consumers, we often don't have the luxury of moving our personal data to a place to where we want it to be. 10 | Many services don't even provide export functionality, and even if they do, the exports often lack information or are not interoperable with other apps. 11 | 12 | Atomic Data could help to re-introduce data ownership. 13 | Because the specification helps to standardize information, it becomes easier to make data interoperable. 14 | And even more important: Apps don't need their own back-end - they can use the same personal data store: an Atomic Server (such as [this one](https://github.com/atomicdata-dev/atomic-serverob/master/server/README.md)). 15 | 16 | Realizing this goal requires quite a bit of work, though. 17 | This specification needs to mature, and we need reliable implementations. 18 | We also need proper tutorials, libraries and tools that convince developers to use atomic data to power their applications. 19 | -------------------------------------------------------------------------------- /src/trust.md: -------------------------------------------------------------------------------- 1 | # Atomic Trust 2 | 3 | _status: just an idea_ 4 | 5 | Not all information on the web can be trusted. 6 | Instead of relying on some centralized authority to say which content is to be trusted, we can leverage our existing trust networks to determine what we can trust or not. 7 | 8 | Atomic Trust is a specification to share which actors, domains and resources we trust on the web. 9 | It's a decentralized model defined with Atomic Schema to create explicit trust networks. 10 | It can be used to calculate a score about a resource (such as a webpage). 11 | 12 | ## How it works 13 | 14 | When you view some resource (e.g. a news article on some website), your client (e.g. a browser plugin) checks your _trusted peers_ to find whether they (or their peers) have a _rating_ about that _resource_. 15 | 16 | ## Concepts 17 | 18 | ### ResourceRating 19 | 20 | A ResourceRating describes how an Actor (e.g. a person) rates some Resource (e.g. a specific article). 21 | 22 | Properties: 23 | 24 | - about: The resource that is being rated 25 | - ratedBy: The actor (Person or Organization) rating the resource 26 | - score: Number between -1 and 1. 1 is "Very trustworthy", 0 is "neutral" and -1 is "very untrustworthy". 27 | - comment: A string that describes the _why_. 28 | 29 | ### DomainRating 30 | 31 | A DomainRating describes how an Actor (e.g. a person) rates some Resource (e.g. a specific article). 32 | 33 | Same props as ResourceRating. 34 | 35 | ### Actor 36 | 37 | The individual that creates the rating. 38 | 39 | Properties 40 | 41 | - ratingList: A Collection of ratings 42 | -------------------------------------------------------------------------------- /src/usecases/react.md: -------------------------------------------------------------------------------- 1 | # Using Atomic Data in a JS / TS React project 2 | 3 | Atomic Data has been designed with front-end development in mind. 4 | The open source [Atomic-Data-Browser](https://github.com/atomicdata-dev/atomic-data-browser), which is feature-packed with chatrooms, a real-time collaborative rich text editor, tables and more, is powered by two libraries: 5 | 6 | - `@tomic/lib` ([docs](https://atomicdata-dev.github.io/atomic-data-browser/docs/modules/_tomic_lib.html)) is the core library, containing logic for fetching and storing data, keeping things in sync using websockets, and signing [commits](../commits/intro.md). 7 | - `@tomic/react` ([docs](https://atomicdata-dev.github.io/atomic-data-browser/docs/modules/_tomic_react.html)) is the react library, featuring various useful hooks that mimic `useState`, giving you real-time updates through your app. 8 | 9 | Check out the [template on CodeSandbox](https://codesandbox.io/s/atomic-data-react-template-4y9qu?file=/src/MyResource.tsx:0-1223): 10 | 11 | 17 | 18 | Feeling stuck? [Post an issue](https://github.com/atomicdata-dev/atomic-data-browser/issues/new) or [join the discord](https://discord.gg/a72Rv2P). 19 | -------------------------------------------------------------------------------- /src/commits/ownership.md: -------------------------------------------------------------------------------- 1 | # Atomic Ownership 2 | 3 | When data changes, we need a way of thinking about _who is able to change data_. 4 | Of course, anybody can copy some piece of data and make changes to is, but for P2P data sharing to work, we also need to be able to _verify_ that the changes are _authentic_. 5 | That's why Atomic Data has the _Ownership_ concept. 6 | 7 | ## Owner 8 | 9 | The Owner of a Resource is the _authority_ of the content of the resource and its Commits. 10 | It can be either a person or some more abstract entity, such as an organization or some hardware running somewhere. 11 | You can think of an Owner as the one accepting Pull Requests in a Git repository. 12 | By _default_, the Owner is the Domain. 13 | 14 | The Owner identifies itself using a generated key-pair. 15 | It signs authored Commits with his private key, so others can verify the authenticity of the Commits using the corresponding public key. 16 | 17 | If the Owner is not the domain, the owner SHOULD be explicitly referred to using the `owner` Property: 18 | 19 | - `owner` (required, AtomicURL, Owner) - the Owner of the resource 20 | 21 | ## Using HTTP(S) 22 | 23 | Any Atomic Resource has a Subject, and that Subject will be a URL. 24 | Resolving that URL will link to the resource itself and show you the latest state. 25 | HTTP URLs already show you who's in control, because the domain name points you there. 26 | 27 | ## Using IPFS 28 | 29 | TODO! 30 | 31 | ## Using DID 32 | 33 | The DID (decentralized identifiers) paves the way for a more decentralized web. 34 | 35 | TODO! Something with DID controllers https://w3c.github.io/did-core/#dfn-did-controllers 36 | 37 | ## Using Hypercore (DAT) 38 | 39 | TODO! 40 | -------------------------------------------------------------------------------- /src/interoperability/intro.md: -------------------------------------------------------------------------------- 1 | {{#title Atomic Data Interoperability - Relationship and comparison to other technology}} 2 | # Interoperability: Relation to other technology 3 | 4 | Atomic data is designed to be easy to use in existing projects, and be interoperable with existing formats. 5 | This section will discuss how Atomic Data differs from or is similar to various data formats and paradigms, and how it can interoperate. 6 | 7 | ## Upgrade guide 8 | 9 | * [Upgrade](upgrade.md): How to make your existing (server-side) application serve Atomic Data. From easy, to hard. 10 | 11 | ## Data formats 12 | 13 | * [JSON](json.md): Atomic Data is designed to be easily serializable to clean, idiomatic JSON. However, if you want to turn JSON into Atomic Data, you'll have to make sure that all keys in the JSON object are URLs that link to Atomic Properties, and the data itself also has to be available at its Subject URL. 14 | * [RDF](rdf.md): Atomic Data is a strict subset of RDF, and can therefore be trivially serialized to all RDF formats (Turtle, N-triples, RDF/XML, JSON-LD, and others). The other way around is more difficult. Turning RDF into Atomic Data requires that all predicates are Atomic Properties, the values must match its properties datatype, the atoms must be available at the subject URL, and the subject-predicate combinations must be unique. 15 | 16 | ## Protocols 17 | 18 | * [Solid](solid.md): A set of specifications that has many similarities with Atomic Data 19 | * [IPFS](ipfs.md): Content-based addressing to prevent 404s and centralization 20 | 21 | ## Database paradigms 22 | 23 | * [SQL](sql.md): How Atomic Data differs from and could interact with SQL databases 24 | * [Graph](graph-database.md): How it differs from some labeled property graphs, such as Neo4j 25 | -------------------------------------------------------------------------------- /src/when-to-use.md: -------------------------------------------------------------------------------- 1 | {{#title When (not) to use Atomic Data}} 2 | # When (not) to use Atomic Data 3 | 4 | ## When should you use Atomic Data 5 | 6 | - **Flexible schemas**. When dealing with structured wikis or semantic data, various instances of things will have different attributes. Atomic Data allows _any_ kind of property on _any_ resource. 7 | - **Open data**. Atomic Data is a bit harder to create than plain JSON, for example, but it is easier to re-use and understand. It's use of URLs for properties makes data self-documenting. 8 | - **High interoperability requirements**. When multiple groups of people have to use the same schema, Atomic Data provides easy ways to constrain and validate the data and ensure type safety. 9 | - **Connected / decentralized data**. With Atomic Data, you use URLs to point to things on other computers. This makes it possible to connect datasets very explicitly, without creating copies. Very useful for decentralized social networks, for example. 10 | - **Auditability & Versioning**. Using Atomic Commits, we can store all changes to data as transactions that can be replayed. This creates a complete audit log and history. 11 | - **JSON or RDF as Output**. Atomic Data serializes to idiomatic, clean JSON as well as various RDF formats (Turtle / JSON-LD / n-triples / RDF/XML). 12 | 13 | ## When not to use Atomic Data 14 | 15 | - **Internal use only**. If you're not sharing structured data, Atomic Data will probably only make things harder for you. 16 | - **Big Data**. If you're dealing with TeraBytes of data, you probably don't want to use Atomic Data. The added cost of schema validation and the lack of distributed / large scale persistence tooling makes it not the right choice. 17 | - **Video / Audio / 3D**. These should have unique, optimized binary representations and have very strict, static schemas. The advantages of atomic / linked data do little to improve this, unless it's just for metadata. 18 | -------------------------------------------------------------------------------- /src/schema/migrations.md: -------------------------------------------------------------------------------- 1 | # Migrations and API versioning in Atomic Data 2 | 3 | _Status: design / concept stage_ 4 | 5 | Models are rarely static. 6 | As new insights and business requirements influence our internal understanding of some domain, we often make changes to a model. 7 | That's why Atomic Classes have the optional `deprecatedProperties` Property, which helps to communicate if properties are changed in some newer version. 8 | 9 | For example, we might start off with a `Person => employer => Organization` relationship. 10 | Later, we might need some way to describe the Role of that person, and when the employment will end. 11 | To solve this, the `employer` relation might be expanded into a separate Resource, with a set of props (e.g. `role` and `startedAt`). 12 | 13 | Now, in many RESTful APIs, versioning is done by adding an endpoint such as `/api/v2`. 14 | Every versioned endpoint has differences, and their own documentation deprecation. 15 | However, this means that links made to _API versioned_ resources (`v1/somePerson`) should be updated (`v2/somePerson)`. 16 | This is not a good approach for linked data. 17 | 18 | Instead of _replacing_ the relationship, we recommend to simply _add a new Property_, and keeping the old one for some time. 19 | We might add a `Employment` Class, and give it `role`, `employee`, `organization` and `startedAt` properties. 20 | The relationship might look like this: `Person => employment => Employment => organization => Organization`. 21 | We now have _two_ places where we can find what the employer of a person is - the `employer` property, but also `employment`. 22 | Maintaining both might make sense, but is probably not necessary and only confusing. 23 | The API should communicate that the `employment` Property is the one that will be maintained, and the `employer` will be removed. 24 | 25 | The `employer` relationship should be added to `deprecatedProperties` in the `Person` class. 26 | -------------------------------------------------------------------------------- /src/invitations.md: -------------------------------------------------------------------------------- 1 | {{#title Atomic Data Invitations - Sharing using Tokens }} 2 | # Invitations & Tokens 3 | 4 | ([Discussion](https://github.com/ontola/atomic-data/issues/23)) 5 | 6 | At some point on working on something in a web application, you're pretty likely to share that, often not with the entire world. 7 | In order to make this process of inviting others as simple as possible, we've come up with an Invitation standard. 8 | 9 | ## Design goals 10 | 11 | - **Edit without registration**. Be able to edit or view things without being required to complete a registration process. 12 | - **Share with a single URL**. A single URL should contain all the information needed. 13 | - **(Un)limited URL usage**. A URL might be re-usable, or maybe not. 14 | 15 | ## Flow 16 | 17 | 1. The Owner or a resource creates an [Invite](https://atomicdata.dev/classes/Invite). This Invite points to a `target` Resource, provides `read` rights by default but can additionally add `write` rights, contains a bunch of `usagesLeft`. 18 | 1. The Guest opens the Invite URL. This returns the Invite resource, which provides the client with the information needed to do the next request which adds the actual rights. 19 | 1. The browser client app might generate a set of keys, or use an existing one. It sends the Agent URL to the Invite in a query param. 20 | 1. The server will respond with a Redirect resource, which links to the newly granted `target` resource. 21 | 1. The Guest will now be able to access the Resource. 22 | 23 | Try it on [https://atomicdata.dev/invites/1](https://atomicdata.dev/invites/1) 24 | 25 | ## Limitations and gotcha's 26 | 27 | - The one creating the Invite has to take security in consideration. Some URLs can be easily guessed! When implementing Invitations, make sure to use a good amount of randomness when creating the Subject. 28 | - Make sure that the invite is not publicly discoverable (e.g. through a Collection), this can happen if you set the `parent` of the invite to a public resource. 29 | -------------------------------------------------------------------------------- /src/commits/suggestions.md: -------------------------------------------------------------------------------- 1 | # Atomic Suggestions 2 | 3 | Atomic Suggestions is a proposed standard that enables decentralized collaboration on resources. 4 | It's basically Git for linked data. 5 | Practically, it should enable right-clicking on any piece of Atomic Data on the web, and suggesting an edit to the owner. 6 | 7 | ## Design goals 8 | 9 | - **Asynchronous collaboration**: Various users can work on the same thing at the same time. 10 | - **Branching & merging**: Issues that result from async changes (merge conflicts) can be resolved. 11 | 12 | ## Concepts 13 | 14 | ### Fork 15 | 16 | Forking is the first step to making a suggestion. 17 | Forking refers to: 18 | 19 | 1. copying some resource 20 | 1. changing the subject URL to some URL that you control 21 | 1. adding a reference to the original URL using the `atomic:originalSubject` Property. 22 | 23 | The newly created copy with the different URL is a _Fork_. 24 | Since the Fork is a resource that you own (see [Ownership](ownership.md)), you can make changes to is. 25 | 26 | Whenever you make changes, the app making the changes _should_ keep track of them as Atomic Commits. 27 | These Commits make it easier to apply (small) changes to (large) resources, even when multiple people are working on the same thing at the same time. 28 | 29 | ### Suggestion 30 | 31 | When you've forked some resource and made some changes, you can Suggest these changes to the original owner. 32 | This is done by sending an HTTP POST request containing the Commits to the Owner URL. 33 | 34 | A Suggestion is a (set of?) Commit(s?) that is proposed to be appended to some Ledger. 35 | The important difference between a Suggestion and a Commit, is that a Commit has been verified, signed and approved by the Controller. 36 | 37 | ### Controller 38 | 39 | The actor (person / organization) that is in control of a specific Resource and its Commits. 40 | 41 | ### Inbox 42 | 43 | An Inbox represents a resource that contains incoming Suggestions. 44 | It's similar to an e-mail inbox. 45 | -------------------------------------------------------------------------------- /src/agents.md: -------------------------------------------------------------------------------- 1 | {{#title Atomic Data Agents - Users and identities }} 2 | # Atomic Agents 3 | 4 | Atomic Agents are used for [authentication](./authentication.md): to set an identity and prove who an actor actually is. 5 | Agents can represent both actual individuals, or machines that interact with data. 6 | Agents are the entities that can get write / read rights. 7 | Agents are used to sign Requests and [Commits](commits/intro.md) and to accept [Invites](invitations.md). 8 | 9 | ## Design goals 10 | 11 | - **Decentralized**: Atomic Agents can be created by anyone, at any domain 12 | - **Easy**: It should be easy to work with, code with, and use 13 | - **Privacy-friendly**: Agents should allow for privacy friendly workflows 14 | - **Verifiable**: Others should be able to verify who did what 15 | - **Secure**: Resistant to attacks by malicious others 16 | 17 | ## The Agent model 18 | 19 | _url: https://atomicdata.dev/classes/Agent_ 20 | 21 | An Agent is a Resource with its own URL. 22 | When it is created, the one creating the Agent will generate a cryptographic (Ed25519) keypair. 23 | It is _required_ to include the [`publicKey`](https://atomicdata.dev/properties/publicKey) in the Agent resource. 24 | The [`privateKey`](https://atomicdata.dev/properties/privateKey) should be kept secret, and should be safely stored by the creator. 25 | For convenience, a `secret` can be generated, which is a single long string of characters that encodes both the `privateKey` and the `subject` of the Agent. 26 | This `secret` can be used to instantly, easily log in using a single string. 27 | 28 | The `publicKey` is used to verify commit signatures by that Agent, to check if that Agent actually did create and sign that Commit. 29 | 30 | ## Creating an Agent 31 | 32 | Since an Agent is used for verification of commits, the Agent's `subject` should resolve and be publicly available. 33 | This means that the one creating the Agent has to deal with this. 34 | One way of doing this, is by hosting an [Atomic Server](https://crates.io/crates/atomic-server). 35 | An easier way of doing this, is by accepting an [Invite](invitations.md) that exists on someone else's server. 36 | -------------------------------------------------------------------------------- /src/usecases/knowledge-management.md: -------------------------------------------------------------------------------- 1 | # Atomic Data for (semantic) knowledge graph management 2 | 3 | Knowledge **management** is about making valuable knowledge easily findable, so everybody in an organization can be as effective as possible. 4 | Knowledge **graphs** are information structures that help organizations to organize their knowledge using a graph model. 5 | Graphs are especially useful for structuring knowledge, as they allow having links between resources which makes relationships understandable and makes data browsable. 6 | 7 | Atomic Data is a Graph structure, and [Atomic-Server](https://crates.io/crates/atomic-server/) is an open source Graph database / knowledge management system. 8 | 9 | ## Knowledge management systems 10 | 11 | How do organizations store and share knowledge? 12 | Some rely completely on their brains and social networks: if you want to know how the copier works, ask Sara. 13 | But most use digital documents - more often than not in the cloud. 14 | If your knowledge is digital and online available, people can retrieve it from anywhere at great speed. 15 | Being able to search and browse through information is essential to making it effortless to retrieve. 16 | 17 | But good knowledge management systems are not just static: they have lives of their own. 18 | Knowledge changes over time. 19 | People add documents, make changes, move things. 20 | 21 | ## Why use Atomic-Server as a knowledge management system 22 | 23 | ### The entire web as one knowledge graph 24 | 25 | Atomic Data uses URLs to identify resources. 26 | This means that it 27 | 28 | ### Type-safe, decentralized data structures 29 | 30 | Contrary to many other types of graph systems, Atomic Data ensures type-safety by having a built-in schema language ([Atomic Schema](../schema/intro.md)). 31 | This means that it is very easy to share and re-use data models, which helps you standardize the classes and properties that you use. 32 | 33 | ## Non-goals of Atomic-Server 34 | 35 | - Deep, specific query requirements 36 | - Time-series data / data visualization 37 | 38 | ## Alternatives 39 | 40 | - **LinkedDataHub** by Atomgraph (unrelated, don't mind the name similarities): knowledge graph management tool that also supports RDF. Open source. 41 | - ** 42 | -------------------------------------------------------------------------------- /src/usecases/ai.md: -------------------------------------------------------------------------------- 1 | # Atomic Data & Artificial Intelligence 2 | 3 | Recent developments in machine learning (and specifically deep neural networks) have shown how powerful and versatile AI can be. 4 | Both Atomic Data and AI can be used to store and query knowledge, but we think of these technologies as complementary due to their unique characteristics: 5 | 6 | - Artificial Intelligence can make sense of (unstructured) data, so you can feed it any type of data. However, AIs often produce unpredictable and sometimes incorrect results. 7 | - Atomic Data helps to make data interoperable, reliable and predictable. However, it requires very strict inputs. 8 | 9 | There are two ways in which Atomic Data and AI can help each other: 10 | 11 | - AI can help to make creating Atomic Data easier. 12 | - Atomic Data can help train AIs. 13 | - Atomic Data can provide AIs with reliable, machine readable data for answering questions. 14 | 15 | ## Make it easier to create Atomic Data using AI 16 | 17 | While writing text, an AI might help make suggestions to disambiguate whatever it is you're writing about. 18 | For example, you may mention `John` and your knowledge graph editor (like `atomic-server`) could suggest `John Wayne` or `John Cena`. 19 | When making your selection, a link will be created which helps to make your knowledge graph more easily browsable. 20 | AI could help make these suggestions through context-aware _entity recognition_. 21 | 22 | ## Train AIs with Atomic Data 23 | 24 | During training, you could feed Atomic Data to your AI to help it construct a reliable, consistent model of the knowledge relevant to your organization or domain. 25 | You could use `atomic-server` as the knowledge store, and iterate over your resources and let your AI parse them. 26 | 27 | ## Provide AI with query access to answer questions 28 | 29 | Instead of training your AI, you might provide your AI with an interface to perform queries. 30 | Note that at this moment, I'm not aware of any AIs that can actually construct and execute queries, but because of recent advancements (e.g. ChatGPT), we know that there now exist AIs that can create SQL queries based on human text. 31 | In the future, you might let your AI query your `atomic-server` to find reliable and up-to-date answers to your questions. 32 | -------------------------------------------------------------------------------- /src/usecases/education.md: -------------------------------------------------------------------------------- 1 | # Atomic Data for Education - standardized, modular e-learning 2 | 3 | The Atomic Data specification can help make online educational content more **modular**. This has two direct benefits: 4 | 5 | - **Separate learning goals from how they are achieved**. Some might prefer watching a video, others may want to read. Both can describe the same topic, and share the same test. 6 | - **Improve discoverability**. Create links between topics so students know which knowledge is needed to advance to the next topic. 7 | 8 | ## Modular educational content - a model 9 | 10 | We can think of **Knowledge** as being building blocks that we need to do certain things. 11 | And we can think of **Lessons** as _teaching_ certain pieces of knowledge, while at the same time _requiring_ other pieces of knowledge. 12 | For example, an algebra class might require that you already know how to multiply, add, etc. 13 | We can think of **Test** as _verifying_ if a piece of knowledge is properly understood. 14 | 15 | Now there's also a relationship between the **Student** and all of these things. 16 | A student is following a bunch Lessons in which they've made some progress, has done some Tests which resulted in Scores. 17 | 18 | Describing our educational content in this fashion has a bunch of advantages. 19 | For students, this means they can know in advance if they can get started with a course, or if they need to learn something else first. 20 | Conversely, they can also discover new topics that depend on their previous piece of knowledge. 21 | For teachers, this means they can re-use existing lessons for the courses. 22 | 23 | ## What makes Atomic-Server a great tool for creating online courseware 24 | 25 | - Powerful built-in document editor 26 | - Drag & drop file support 27 | - Versioning 28 | - Open source, so no vendor lock-in, and full customizability 29 | - Real-time updates, great for collaboration 30 | - Online by default, so no extra hassle with putting courses on the internet 31 | 32 | However, there is still a lot to do! 33 | 34 | - Turn the model described above into an actual Atomic Schema data model 35 | - Build the GUI for the application 36 | - Add plugins / extenders for things like doing tests (without giving the answer to students!) 37 | - Create educational content 38 | -------------------------------------------------------------------------------- /src/endpoints.md: -------------------------------------------------------------------------------- 1 | {{#title Atomic Data Endpoints - describe how RESTful HTTP APIs behave}} 2 | # Atomic Endpoints 3 | 4 | _URL: https://atomicdata.dev/classes/Endpoint_ 5 | 6 | An Endpoint is a resource that accepts parameters in order to generate a response. 7 | You can think of it like a function in a programming language, or a API endpoint in an OpenAPI spec. 8 | It can be used to perform calculations on the server side, such as filtering data, sorting data, selecting a page in a collection, or performing some calculation. 9 | Because Endpoints are resources, they can be defined and read programmatically. 10 | This means that it's possible to render Endpoints as forms. 11 | 12 | The most important property in an Endpoint is [`parameters`](https://atomicdata.dev/properties/endpoint/parameters), which is the list of Properties that can be filled in. 13 | 14 | You can find a list of Endpoints supported by Atomic-Server on [atomicdata.dev/endpoints](https://atomicdata.dev/endpoints). 15 | 16 | Endpoint Resources are _dynamic_, because their properties could be calculated server-side. 17 | When a Property tends to be calculated server-side, they will have a [`isDynamic` property](https://atomicdata.dev/properties/isDynamic) set to `true`, which tells the client that it's probably useless to try to overwrite it. 18 | 19 | ## Incomplete resources 20 | 21 | A Server can also send one or more partial Resources for an Endpoint to the client, which means that some properties may be missing. 22 | When this is the case, the Resource will have an [`incomplete`](https://atomicdata.dev/properties/incomplete) property set to `true`. 23 | This tells the client that it has to individually fetch the resource from the server to get the full body. 24 | 25 | One scenario where this happens, is when fetching Collections that have other Collections as members. 26 | If we would not have incomplete resources, the server would have to perform expensive computations even if the data is not needed by the client. 27 | 28 | ## Design Goals 29 | 30 | - **Familiar API**: should look like something that most developers already know 31 | - **Auto-generate forms**: a front-end app should present Endpoints as forms that non-developers can interact with 32 | 33 | ([Discussion](https://github.com/atomicdata-dev/atomic-data-docs/issues/15)) 34 | -------------------------------------------------------------------------------- /src/schema/compare.md: -------------------------------------------------------------------------------- 1 | # Atomic Schema compared to alternatives 2 | 3 | Introducing yet another Schema language might seem like a bad idea - why add another competing standard instead of using one that exists? 4 | In this section, we'll discuss some existing 5 | 6 | ## JSON-Schema 7 | 8 | Really well-documented. 9 | Various implementations exist. 10 | Does not support RDF. 11 | All information is scoped to a schema, which means properties are not re-usable. 12 | In Atomic Data, we have both Properties and Classes, both of which have their own responsibilities in the schema. 13 | The Properties dictate the `datatype`, `name` and `description`, and the Classes communicate which properties are `required` or `recommended`. 14 | Like Atomic Schema, it can be described in JSON. 15 | 16 | ## SHACL 17 | 18 | SHACL (Shape Constraint Language) is an RDF ontology that provides. 19 | Like Atomic Schema, it can be described in RDF. 20 | 21 | ## ShEx 22 | 23 | [ShEx](https://shex.io/) (Shape Expressions) is a standard for describing RDF graph constraints. 24 | It introduces its own serialization format, called [ShExC](https://github.com/shexSpec/grammar/blob/master/ShExDoc.g4) (Shape Expressions Compact Syntax), which looks like this: 25 | 26 | ```ShExC 27 | PREFIX school: 28 | PREFIX xsd: 29 | PREFIX ex: 30 | 31 | # Node constraint 32 | school:enrolleeAge xsd:integer MinInclusive 13 MaxInclusive 20 33 | 34 | 35 | school:Enrollee { 36 | # Triple constraint (including node constraint IRI) 37 | ex:hasGuardian IRI {1,2} 38 | } 39 | ``` 40 | 41 | ## OWL 42 | 43 | OWL (the Web Ontology Language) is an ontology for ontologies: it can be used to create descriptions of how concepts in the world relate to each other. 44 | OWL can also be used for _reasoning_, which deduces new information from existing information. 45 | This means: adding new triples, based on the existing ones. 46 | For example, if you know that `John` is a `Human`, and `Humans` are `Organisms`, you can deduce that `John` is an `Organism`. 47 | 48 | However, OWL is not used for _constraining_ or _validating_ data. 49 | Reasoners should not be used for checking if some piece is valid data, but only to create new data - it is always assumed that the input data is correct. 50 | -------------------------------------------------------------------------------- /src/interoperability/vc-old.md: -------------------------------------------------------------------------------- 1 | # Atomic Data and Verifiable Credentials 2 | 3 | Verifiable Credentials are pieces of information that have cryptographic proof by some reliable third party. 4 | For example, you could have a credential that proves your degree, signed by your education. 5 | These credentials an enable privacy-friendly transactions where a credential owner can prove being part of some group, without needing to actually identify themselves. 6 | For example, you could prove that you're over 18 by showing a credential issued by your government, without actually having to show your ID card with your birthdate. 7 | Verifiable Credentials are still not that widely used, but various projects exists that have had moderate success in implementing it. 8 | 9 | In Atomic Data, _all information created with Atomic Commits is verifiable_. 10 | Atomic Commits are signed by specific individuals, and these signatures can be verified with the Public Key from the Agent who signed the Commit. 11 | 12 | ## W3C Verifiable Credentials spec 13 | 14 | The W3C Verifiable Credentials (W3CVC) specification has helped to create a spec to describe credentials. 15 | However, the approach is fundamentally different from how Atomic Data works. 16 | In the W3CVC spec, every credential is a resource. 17 | In Atomic Data, having a new type of `Credential` class that maps to W3CVC Credentials is definitely possible, but it is also highly redundant, as Commits already provide the same information. 18 | That's why we've opted for only signing Commits. 19 | 20 | In Atomic Commits, the _change in information_ is signed, instead of the _state_ of the data. 21 | This is by design, as storing signed state changes allows for fully verifiable and reversible history / version control with audit logs. 22 | 23 | ## Verifying data with Atomic Commits 24 | 25 | If you want to know whether a specific value that you see is signed by a specific Agent, you need to find the Commit that created the value. 26 | 27 | This can be achieved by using a Collection. 28 | The easiest way to do this, is by using the [`/all-versions` Endpoint](https://atomicdata.dev/all-versions) and finding the Signer of the version that is relevant to your question. 29 | 30 | In the near future, we will introduce a `/verify` Endpoint that will allow you to verify a specific value. 31 | 32 | Visit the [issue on github](https://github.com/ontola/atomic-data-docs/issues/22) to join the discussion about this subject. 33 | -------------------------------------------------------------------------------- /src/websockets.md: -------------------------------------------------------------------------------- 1 | {{#title Atomic Data Websockets - live synchronization}} 2 | # WebSockets in Atomic Data 3 | 4 | WebSockets are a very fast and efficient way to have a client and server communicate in an asynchronous fashion. 5 | They are used in Atomic Data to allow real-time updates, which makes it possible to create things like collaborative applications and multiplayer games. 6 | These have been implemented in `atomic-server` and `atomic-data-browser` (powered by `@tomic/lib`). 7 | 8 | ## Initializing a WebSocket connection 9 | 10 | Send an HTTP `GET` request to the `/ws` endpoint of an `atomic-server`. The Server should update that request to a secure WebSocket (`wss`) connection. 11 | Use `x-atomic` [authentication headers (read more here)](./authentication.md) and use `ws` as a subject when signing. 12 | The `WebSocket-Protocol` is `AtomicData`. 13 | 14 | ## Client to server messages 15 | 16 | - `SUBSCRIBE ${subject}` tells the Server that you'd like to receive Commits about this Subject. 17 | - `UNSUBSCRIBE ${subject}` tells the Server that you'd like to stop receiving Commits about this Subject. 18 | - `GET ${subject}` fetch an individual resource. 19 | - `AUTHENTICATE ${authenticationResource}` to set a user session for this websocket and allow authorized messages. The `authenticationResource` is a JSON-AD resource containing the signature and more, see [Authentication](../src/authentication.md). 20 | 21 | ## Server to client messages 22 | 23 | - `COMMIT ${CommitBody}` an entire [Commit](../src/commits/concepts.md) for a resource that you're subscribed to. 24 | - `RESOURCE ${Resource}` a JSON-AD Resource as a response to a `GET` message. If there is something wrong with this request (e.g. 404), return a `Error` Resource with the requested subject, similar to how the HTTP protocol server does this.` 25 | - `ERROR ${ErrorBody}` an Error resource is sent whenever something goes wrong. The `ErrorBody` is a plaintext, typically English description of what went wrong. 26 | 27 | ## Considerations 28 | 29 | - For many messages, there is no response to give if things are processed correctly. If a message is unknown or there is a different problem, return an `ERROR`. 30 | 31 | ## Example implementations 32 | 33 | - [Example client implementation in Typescript (@tomic/lib).](https://github.com/atomicdata-dev/atomic-data-browser/blob/main/lib/src/websockets.ts) 34 | - [Example server implementation in Rust using Actix-Web](https://github.com/atomicdata-dev/atomic-server/blob/master/server/src/handlers/web_sockets.rs) 35 | -------------------------------------------------------------------------------- /src/core/querying.md: -------------------------------------------------------------------------------- 1 | {{#title Querying Atomic Data}} 2 | # Querying Atomic Data 3 | 4 | There are multiple ways of getting Atomic Data into some system: 5 | 6 | - [**Subject Fetching**](#subject-fetching-http) requests a single subject right from its source 7 | - [**Atomic Collections**](../schema/collections.md) can filter, sort and paginate resources 8 | - [**Atomic Paths**](paths.md) is a simple way to traverse Atomic Graphs and target specific values 9 | - [**Triple Pattern Fragments**](#triple-pattern-fragments) allows querying for specific (combinations of) Subject, Property and Value. 10 | - [**SPARQL**](#SPARQL) is a powerful Query language for traversing linked data graphs 11 | 12 | ## Subject fetching (HTTP) 13 | 14 | The simplest way of getting Atomic Data when the Subject is an HTTP URL, is by sending a GET request to the subject URL. 15 | Set the `Content-Type` header to an Atomic Data compatible mime type, such as `application/ad+json`. 16 | 17 | ```HTTP 18 | GET https://atomicdata.dev/test HTTP/1.1 19 | Content-Type: application/ad+json 20 | ``` 21 | 22 | The server SHOULD respond with all the Atoms of which the requested URL is the subject: 23 | 24 | ```HTTP 25 | HTTP/1.1 200 OK 26 | Content-Type: application/ad+json 27 | Connection: Closed 28 | 29 | { 30 | "@id": "https://atomicdata.dev/test", 31 | "https://atomicdata.dev/properties/shortname": "1611489928" 32 | } 33 | ``` 34 | 35 | The server MAY also include other resources, if they are deemed relevant. 36 | 37 | ## Atomic Collections 38 | 39 | Collections are Resources that provide simple query options, such as filtering by Property or Value, and sorting. 40 | They also paginate resources. 41 | Under the hood, Collections are powered by Triple Pattern Fragments. 42 | Use query parameters to traverse pages, filter, or sort. 43 | 44 | [Read more about Collections](../schema/collections.md) 45 | 46 | ## Atomic Paths 47 | 48 | An Atomic Path is a string that consist of one or more URLs, which when traversed point to an item. 49 | 50 | [Read more about Atomic Paths](paths.md) 51 | 52 | ## SPARQL 53 | 54 | [SPARQL](https://www.w3.org/TR/rdf-sparql-query/) is a powerful RDF query language. 55 | Since all Atomic Data is also valid RDF, it should be possible to query Atomic Data using SPARQL. 56 | None of the exsisting implementations support a SPARQL endpoint, though. 57 | 58 | - Convert / serialize Atomic Data to RDF (for example by using an `accept` header: `curl -i -H "Accept: text/turtle" "https://atomicdata.dev"`) 59 | - Load it into a SPARQL engine of your choice 60 | -------------------------------------------------------------------------------- /src/SUMMARY.md: -------------------------------------------------------------------------------- 1 | ## Table of contents 2 | 3 | * [Atomic Data Overview](atomic-data-overview.md) 4 | * [Motivation](motivation.md) 5 | * [Strategy, history and roadmap](roadmap.md) 6 | * [When (not) to use it](when-to-use.md) 7 | 8 | # Specification (core) 9 | 10 | * [What is Atomic Data?](core/concepts.md) 11 | * [Serialization](core/serialization.md) 12 | * [JSON-AD](core/json-ad.md) 13 | * [Querying](core/querying.md) 14 | * [Paths](core/paths.md) 15 | * [Schema](schema/intro.md) 16 | * [Classes](schema/classes.md) 17 | * [Datatypes](schema/datatypes.md) 18 | * [FAQ](schema/faq.md) 19 | 20 | # Specification (extended) 21 | 22 | * [Atomic Data Extended](extended.md) 23 | * [Agents](agents.md) 24 | * [Hierarchy and authorization](hierarchy.md) 25 | * [Authentication](authentication.md) 26 | * [Invitations and sharing](invitations.md) 27 | * [Commits (writing data)](commits/intro.md) 28 | * [Concepts](commits/concepts.md) 29 | * [Compared to](commits/compare.md) 30 | * [WebSockets](websockets.md) 31 | * [Endpoints](endpoints.md) 32 | * [Collections, filtering, sorting](schema/collections.md) 33 | * [Uploading and downloading files](files.md) 34 | 35 | # Create Atomic Data 36 | 37 | * [Atomizing](atomizing.md) 38 | * [Using Atomic-Server](atomic-server.md) 39 | * [Creating a JSON-AD file](create-json-ad.md) 40 | * [Upgrade your existing project](interoperability/upgrade.md) 41 | 42 | # Use Atomic Data 43 | 44 | * [Interoperability and comparisons](interoperability/intro.md) 45 | * [RDF](interoperability/rdf.md) 46 | * [Solid](interoperability/solid.md) 47 | * [JSON](interoperability/json.md) 48 | * [IPFS](interoperability/ipfs.md) 49 | * [SQL](interoperability/sql.md) 50 | * [Graph Databases](interoperability/graph-database.md) 51 | * [Potential use cases](usecases/intro.md) 52 | * [As a Headless CMS](usecases/headless-cms.md) 53 | * [In a React project](usecases/react.md) 54 | * [Personal Data Store](usecases/personal-data-store.md) 55 | * [Artificial Intelligence](usecases/ai.md) 56 | * [E-commerce & marketplaces](usecases/e-commerce.md) 57 | * [Surveys](usecases/surveys.md) 58 | * [Verifiable Credentials](usecases/verifiable-credentials.md) 59 | * [Data Catalog](usecases/data-catalog.md) 60 | * [Education](usecases/education.md) 61 | * [Food labels](usecases/food-labels.md) 62 | * [**Software and libraries**](tooling.md) 63 | 64 | ----------- 65 | 66 | [Acknowledgements](acknowledgements.md) | 67 | [Newsletter](newsletter.md) | 68 | [Get involved](get-involved.md) 69 | -------------------------------------------------------------------------------- /src/files.md: -------------------------------------------------------------------------------- 1 | {{#title Uploading, downloading and describing files with Atomic Data}} 2 | # Uploading, downloading and describing files with Atomic Data 3 | 4 | The Atomic Data model (Atomic Schema) is great for describing structured data, but for many types of existing data, we already have a different way to represent them: files. 5 | In Atomic Data, files have two URLs. 6 | One _describes_ the file and its metadata, and the other is a URL that downloads the file. 7 | This allows us to present a better view when a user wants to take a look at some file, and learn about its context before downloading it. 8 | 9 | ## The File class 10 | 11 | _url: [https://atomicdata.dev/classes/File](https://atomicdata.dev/classes/File)_ 12 | 13 | Files always have a downloadURL. 14 | They often also have a filename, a filesize, a checksum, a mimetype, and an internal ID (more on that later). 15 | They also often have a [`parent`](https://atomicdata.dev/properties/parent), which can be used to set permissions / rights. 16 | 17 | ## Uploading a file 18 | 19 | In `atomic-server`, a `/upload` endpoint exists for uploading a file. 20 | 21 | - Decide where you want to add the file in the [hierarchy](hierarchy.md) of your server. You can add a file to any resource - your file will refer to this resource as its [`parent`](https://atomicdata.dev/properties/parent). Make sure you have `write` rights on this parent. 22 | - Use that parent to add a query parameter to the server's `/upload` endpoint, e.g. `/upload?parent=https%3A%2F%2Fatomicdata.dev%2Ffiles`. 23 | - Send an HTTP `POST` request to the server's `/upload` endpoint containing [`multi-part-form-data`](https://developer.mozilla.org/en-US/docs/Web/API/FormData/Using_FormData_Objects). You can upload multiple files in one request. Add [authentication](authentication.md) headers, and sign the HTTP request with the 24 | - The server will check your authentication headers, your permissions, and will persist your uploaded file(s). It will now create File resources. 25 | - The server will reply with an array of created Atomic Data Files 26 | 27 | ## Downloading a file 28 | 29 | Simply send an HTTP GET request to the File's [`download-url`](https://atomicdata.dev/properties/downloadURL) (make sure to authenticate this request). 30 | 31 | - [Discussion on specification](https://github.com/ontola/atomic-data-docs/issues/57) 32 | - [Discussion on Rust server implementation](https://github.com/atomicdata-dev/atomic-server/issues/72) 33 | - [Discussion on Typescript client implementation](https://github.com/atomicdata-dev/atomic-data-browser/issues/121) 34 | -------------------------------------------------------------------------------- /src/usecases/food-labels.md: -------------------------------------------------------------------------------- 1 | # Atomic Data for food label standardization 2 | 3 | In most countries, food producers are required to provide nutritional information on the packages of products, which helps citizens to make informed decisions about what to eat. 4 | But how about we upgrade these labels to machine-readable, atomic data? 5 | We could describe products using Atomic Data, and put their identifiers (Subject URLs) as QR codes on packages. 6 | Imagine these scenarios: 7 | 8 | ## Scan labels to get detailed, reliable, interactive information 9 | 10 | You want to know more about some new cereal you've just bought. 11 | You scan the QR code on the package. 12 | A web app opens that shows detailed, yet highly visual information about its nutritional value. 13 | The screen is no longer limited to what realistically fits on a package. 14 | The elements are interactive, and provide explanations. 15 | Everything is translated to the user's language. 16 | If the food is (soon to be) expired, the app will clearly and visually alert you. 17 | Click on the question mark next to `granulated sugars`, and you get an explanation of what this means to your health. 18 | E-numbers are clickable, too, and help you instantly understand far more about what they represent. 19 | When AR glasses become technologically feasible, you could even help people make better decisions while doing grocery shopping. 20 | 21 | Using _links_ instead of _names_ helps to guide consumers to _trustworthy_ pages that communicate clearly. 22 | The alternative is that they use search engines, and maybe end up reading misinformation. 23 | 24 | ## Provide nutritional advice based on shopping behavior 25 | 26 | You order a bunch of products on your favorite groceries delivery app. 27 | When going to the payment screen, you are shown a nutritional overview of your order. 28 | You see that with this diet, you might have a deficit of the Lysene amino acid. 29 | The shopping cart suggest adding egg, dairy or soy to your diet. 30 | This can be done, because the groceries app can easily check detailed information about the food in your shopping cart, and reason about your dietary intake. 31 | 32 | ## How to achieve all this 33 | 34 | 1. The governing body (e.g. the European Commision) should set up an [Atomic Server](https://github.com/atomicdata-dev/atomic-server/) and host it on some recognizable domain. 35 | 1. Create the [Class](https://atomicdata.dev/classes/Class) for a food product, containing the same (or more) information that is shown on food packages. 36 | 1. Create the Class for Ingredient. 37 | 1. Create instances for various Ingredients. Start with the E-numbers, work your way up to all kinds of used ingredients. Add Translations. 38 | 1. Give instructions to Producers on how to describe their Products. Give them to option to host their own Server and control their own data, and give them the option to use some EU server. 39 | -------------------------------------------------------------------------------- /src/schema/collections.md: -------------------------------------------------------------------------------- 1 | {{#title Atomic Data Collections - Filtering, Sorting & Querying}} 2 | # Atomic Collections 3 | 4 | _URL: [https://atomicdata.dev/classes/Collection](https://atomicdata.dev/classes/Collection)_ 5 | 6 | Sooner or later, developers will have to deal with (long) lists of items. 7 | For example, a set of blog posts, activities or users. 8 | These lists often need to be paginated, sorted, and filtered. 9 | For dealing with these problems, we have Atomic Collections. 10 | 11 | An Atomic Collection is a Resource that links to a set of resources. 12 | Note that Collections are designed to be _dynamic resources_, often (partially) generated at runtime. 13 | Collections are [Endpoints](../endpoints.md), which means that part of their properties are calculated server-side. 14 | Collections have various filters (`subject`, `property`, `value`) that can help to build a useful query. 15 | 16 | - [`members`](https://atomicdata.dev/properties/collection/members): How many items (members) are visible per page. 17 | - [`property`](https://atomicdata.dev/properties/collection/property): Filter results by a property URL. 18 | - [`value`](https://atomicdata.dev/properties/collection/value): Filter results by a Value. Combined with `property`, you can create powerful queries. 19 | - [`sort_by`](https://atomicdata.dev/properties/collection/sortBy): A property URL by which to sort. Defaults to the `subject`. 20 | - [`sort_desc`](https://atomicdata.dev/properties/collection/sortDesc): Sort descending, instead of ascending. Defaults to `false`. 21 | - [`current_page`](https://atomicdata.dev/properties/collection/currentPage): The number of the current page. 22 | - [`page_size`](https://atomicdata.dev/properties/collection/pageSize): How many items (members) are visible per page. 23 | - [`total_pages`](https://atomicdata.dev/properties/collection/totalPages): How many pages there are for the current collection. 24 | - [`total_members`](https://atomicdata.dev/properties/collection/totalMembers): How many items (members) are visible per page. 25 | 26 | 27 | ## Persisting Properties vs Query Parameters 28 | 29 | Since Atomic Collections are dynamic resources, you can pass query parameters to it. 30 | The keys of the query params match the shortnames of the properties of the Collection. 31 | 32 | For example, let's take the [Properties Collection on atomicdata.dev](https://atomicdata.dev/collections/property). 33 | We could limit the page size to 2 by adding the `page_size=2` query parameter: `https://atomicdata.dev/collections/property?page_size=2`. 34 | Or we could sort the list by the description property: `https://atomicdata.dev/collections/property?sort_by=https%3A%2F%2Fatomicdata.dev%2Fproperties%2Fdescription`. 35 | Note that URLs need to be URL encoded. 36 | 37 | These properties of Collections can either be set by passing query parameters, or they can be _persisted_ by the Collection creator / editor. 38 | -------------------------------------------------------------------------------- /src/usecases/verifiable-credentials.md: -------------------------------------------------------------------------------- 1 | {{#title Atomic Data and Verifiable Credentials / SSI}} 2 | # Atomic Data and Verifiable Credentials / SSI 3 | 4 | ## What are Verifiable Credentials / Self-Sovereign Identity 5 | 6 | Verifiable Credentials are pieces of information that have cryptographic proof by some reliable third party. 7 | For example, you could have a credential that proves your degree, signed by your education. 8 | These credentials an enable privacy-friendly transactions where a credential owner can prove being part of some group, without needing to actually identify themselves. 9 | For example, you could prove that you're over 18 by showing a credential issued by your government, without actually having to show your ID card with your birthdate. 10 | Verifiable Credentials are still not that widely used, but various projects exists that have had moderate success in implementing it. 11 | 12 | ## What makes Atomic Data suitable for this 13 | 14 | Firstly, [Atomic Commit](../commits/intro.md) are already verifiable using signatures that contain all the needed information. 15 | Secondly, [Atomic Schema](../schema/intro.md) can be used for standardizing Credential Schemas. 16 | 17 | ## Every Atomic Commit is a Verifiable Credential 18 | 19 | Every time an Agent updates a Resource, an [Atomic Commit](../commits/intro.md) is made. 20 | This Commit is cryptographically signed by an Agent, just like how Verfifiable Credentials are signed. 21 | In essence, this means that _all atomic data created through commits is fully verifiable_. 22 | 23 | How could this verification work? 24 | 25 | - **Find the Commit** that has created / edited the value that you want to verify. This can be made easier with a specialized Endpoint that takes a `resource`, `property` and `signer` and returns the associated Commit(s). 26 | - **Check the signer of the Commit**. Is that an Agent that you trust? 27 | - **Verify the signature** of the Commit using the public key of the Agent. 28 | 29 | Sometimes, credentials need to be revoked. 30 | How could revocation work? 31 | 32 | - **Find the Commit** (see above) 33 | - **Get the signer** (see above) 34 | - **Find the `/isRevoked` Endpoint of that signer**, send a Request there to make sure the linked Commit is still valid and not revoked. 35 | 36 | ([Discussion](https://github.com/ontola/atomic-data-docs/issues/22)) 37 | 38 | ## Use Atomic Schema for standardizing Credentials 39 | 40 | If you are a Verifier who wants to check someone's _birthdate_, you'll probably expect a certain datatype in return, such as a [date](https://atomicdata.dev/datatypes/date) that is formatted in some specific way. 41 | [Atomic Schema](../schema/intro.md) makes it possible to express which _properties_ are [required](https://atomicdata.dev/properties/requires) in a certain [Class](https://atomicdata.dev/classes/Class), and it also makes it possible to describe which [datatype](https://atomicdata.dev/classes/Datatype) is linked to a specific [Property](https://atomicdata.dev/classes/Property). 42 | Combined, they allow for fine-grained descriptions of models / classes / schemas. 43 | -------------------------------------------------------------------------------- /src/get-started.md: -------------------------------------------------------------------------------- 1 | {{#title Get started with Atomic Data}} 2 | # Get started with Atomic Data 3 | 4 | There's a couple of levels at which you can start working with Atomic Data (from easy to hard): 5 | 6 | - **Play with the demo**: Create an Agent, edit a document. 7 | - **Host your own Atomic-Server**. 8 | - **Create a react app with the template** 9 | - **Set up the full dev environment**. 10 | - **Create a library for Atomic Data**. 11 | 12 | ## Play with the demo 13 | 14 | - Open [the Invite](https://atomicdata.dev/invites/1) on `atomicdata.dev` 15 | - Press `Accept`. Now, the front-end app will generate a Private-Public Key pair. The public key will be sent to the server, which creates an Agent for you. 16 | - You're now signed in! You can edit the document in your screen. 17 | - Edit your Agent by going to [user settings](https://atomicdata.dev/app/agent) 18 | - Copy your `secret`, and save it somewhere safe. You can use this to sign in on a different machine. 19 | - Press `edit user` to add your name and perhaps a bio. 20 | - When you're done, visit user settings again and press `sign out` to erase your credentials and end the session. 21 | 22 | ## Host your own Atomic-Sesrver (locally) 23 | 24 | - If you have docker running, you can use this one-liner: `docker run -p 80:80 -v atomic-storage:/atomic-storage joepmeneer/atomic-server` (or use `cargo install atomic-server`, or the [binaries](https://github.com/atomicdata-dev/atomic-server/releases/)) 25 | - Now, visit `localhost` in your browser to access your server. 26 | - It's now only available locally. If you want to get it on the _internet_, you need to set up a domain name, and make sure its traffic is routed to your computer (search `port forwarding`). 27 | 28 | ## Host your own Atomic-Server (on a VPS) 29 | 30 | - **Set up a domain name** by using one of the many services that do this for you. 31 | - **Get a virtual private server (VPS)** on which you can run `atomic-server`. We are running atomicdata.dev on the cheapest VPS we could find: $3.50 / month at [Vultr.com (use this link to give us $10 bucks of hosting credit)](https://www.vultr.com/?ref=8970814-8H). 32 | 33 | 34 | 35 | - Browser app [atomic-data-browser](https://github.com/atomicdata-dev/atomic-data-browser) ([demo on atomicdata.dev](https://atomicdata.dev)) 36 | - Build a react app using [typescript & react libraries](https://github.com/atomicdata-dev/atomic-data-browser). Start with the [react template on codesandbox](https://codesandbox.io/s/atomic-data-react-template-4y9qu?file=/src/MyResource.tsx) 37 | - Host your own [atomic-server](https://github.com/atomicdata-dev/atomic-data-browser) (powers [atomicdata.dev](https://atomicdata.dev), run with `docker run -p 80:80 -v atomic-storage:/atomic-storage joepmeneer/atomic-server`) 38 | - Discover the command line tool: [atomic-cli](https://github.com/atomicdata-dev/atomic-server) (`cargo install atomic-cli`) 39 | - Use the Rust library: [atomic-lib](https://github.com/atomicdata-dev/atomic-server) 40 | 41 | Make sure to [join our Discord](https://discord.gg/a72Rv2P) if you'd like to discuss Atomic Data with others. 42 | -------------------------------------------------------------------------------- /src/interoperability/upgrade.md: -------------------------------------------------------------------------------- 1 | {{#title Upgrade your existing application to serve Atomic Data}} 2 | # Upgrade your existing application to serve Atomic Data 3 | 4 | You don't have to use [Atomic-Server](https://crates.io/crates/atomic-server) and ditch your existing projects or apps, if you want to adhere to Atomic Data specs. 5 | 6 | As the Atomic Data spec is modular, you can start out simply and conform to more specs as needed: 7 | 8 | 1. Map your JSON keys to new or existing Atomic Data properties 9 | 2. Add `@id` fields to your resources, make sure these URLs resolve using HTTP 10 | 3. Implement parts of the [Extended spec](../extended.md) 11 | 12 | There's a couple of levels you can go to, when adhering to the Atomic Data spec. 13 | 14 | ## Easy: map your JSON keys to Atomic Data Properties 15 | 16 | If you want to make your existing project compatible with Atomic Data, you probably don't have to get rid of your existing storage / DB implementation. 17 | The only thing that matters, is how you make the data accessible to others: the _serialization_. 18 | You can keep your existing software and logic, but simply change the last little part of your API. 19 | 20 | In short, this is what you'll have to do: 21 | 22 | Map all properties of resources to Atomic Properties. 23 | Either use [existing ones](https://atomicdata.dev/properties), or [create new ones](https://atomicdata.dev/app/new?classSubject=https%3A%2F%2Fatomicdata.dev%2Fclasses%2FProperty&parent=https%3A%2F%2Fatomicdata.dev%2Fagents%2F8S2U%2FviqkaAQVzUisaolrpX6hx%2FG%2FL3e2MTjWA83Rxk%3D&newSubject=https%3A%2F%2Fatomicdata.dev%2Fproperty%2Fsu98ox6tvkh). 24 | This means: take your JSON objects, and change things like `name` to `https://atomicdata.dev/properties/name`. 25 | 26 | That's it, you've done the most important step! 27 | 28 | Now your data is already more interoperable: 29 | 30 | - Every field has a clear **semantic meaning** and **datatype** 31 | - Your data can now be **easily imported** by Atomic Data systems 32 | 33 | ## Medium: add `@id` URLs that properly resolve 34 | 35 | Make sure that when the user requests some URL, that you return that resource as a [JSON-AD](../core/json-ad.md) object (at the very least if the user requests it using an HTTP `Accept: application/ad+json` header). 36 | 37 | - Your data can now be **linked to** by external data sources, it can become part of a **web of data**! 38 | 39 | ## Hard: implement Atomic Data Extended protocols 40 | 41 | You can go all out, and implement Commits, Hierarchies, Authentication, Collections and [more](https://docs.atomicdata.dev/extended.html). 42 | I'd suggest starting with [Commits](../commits/intro.md), as these allow users to modify data whilst maintaining versioning and auditability. 43 | Check out the [Atomic-Server source code](https://github.com/atomicdata-dev/atomic-server/tree/master/server) to get inspired on how to do this. 44 | 45 | ## Reach out for help 46 | 47 | If you need any help, join our [Discord](https://discord.gg/a72Rv2P). 48 | 49 | Also, share your thoughts on creating Atomic Data in [this issue on github](https://github.com/ontola/atomic-data-docs/issues/95). 50 | -------------------------------------------------------------------------------- /src/interoperability/ipfs.md: -------------------------------------------------------------------------------- 1 | {{#title How does Atomic Data relate to IPFS?}} 2 | # Atomic Data and IPFS 3 | 4 | ## What is IPFS 5 | 6 | IPFS (the InterPlanetary File System) is a standard that enables decentralized file storage and retrieval using content-based identifiers. 7 | Instead of using an HTTP URL like `http://example.com/helloworld`, it uses the IPFS scheme, such as `ipfs:QmX6j9DHcPhgBcBtZsuRkfmk2v7G5mzb11vU9ve9i8vDsL`. 8 | IPFS identifies things based on their unique content hash (the long, seemingly random string) using a thing called a Merkle DAG ([this great article](https://medium.com/textileio/whats-really-happening-when-you-add-a-file-to-ipfs-ae3b8b5e4b0f#:~:text=In%20practice%2C%20content%20addressing%20systems,function%2C%20to%20produce%20a%20digest.&text=From%20raw%20image%20to%20cryptographic%20digest%20to%20content%20id%20(multihash).) explains it nicely). 9 | This is called a [CID](https://github.com/multiformats/cid), or Content ID. 10 | This simple idea (plus some not so simple network protocols) allows for decentralized, temper-proof storage of data. 11 | This fixes some issues with HTTP that are related to its centralized philosophy: **no more 404s**! 12 | 13 | ## Why is IPFS interesting for Atomic Data 14 | 15 | Atomic Data is highly dependent on the availability of Resources, especially Properties and Datatypes. 16 | These resources are meant to be re-used a lot, and when these go offline or change (for whatever reason), it could cause issues and confusion. 17 | IPFS guarantees that these resources are entirely static, which means that they cannot change. 18 | This is useful when dealing with Properties, as a change in datatype could break things. 19 | IPFS also allows for location-independent fetching, which means that resources can be retrieved from any location, as long as it's online. 20 | This Peer-to-peer functionality is a very fundamental advantage of IPFS over HTTP, especially when the resources are very likely to be re-use, which is _especially_ the case for Atomic Data Properties. 21 | 22 | ## Considerations using IPFS URLs 23 | 24 | IPFS URLs are **static**, which means that their contents can never change. 25 | This is great for some types of data, but not so much for others. 26 | If you're describing a time-dependent thing (such as a person's job), you'll probably want to know what the _current_ value is, and that is not possible when you only have an IPFS identifier. 27 | This can be fixed by including an HTTP URL in IPFS bodies. 28 | 29 | IPFS data is also **hard to remove**, as it tends to be replicated across machines. 30 | If you're describing personal, private information, it can therefore be a bad idea to use IPFS. 31 | 32 | And finally, its **performance** is typically not as good as HTTP. 33 | If you know the IPFS gateway that hosts the IPFS resource that you're looking for, things improve drastically. 34 | Luckily for Atomic Data, this is often the case, as we know the HTTP url of the server and could try whether that server has an IPFS gateway. 35 | 36 | ## Atomic Data and IPLD 37 | 38 | IPLD (not IPFS) stands for InterPlanetary Linked Data, but is not related to RDF. 39 | The scope seems fundamentally different from RDF, too, but I have to read more about this. 40 | 41 | ## Share your thoughts 42 | 43 | Discuss on [this issue](https://github.com/ontola/atomic-data-docs/issues/42). 44 | -------------------------------------------------------------------------------- /src/schema/datatypes.md: -------------------------------------------------------------------------------- 1 | {{#title Atomic Data: Datatypes}} 2 | # Atomic Schema: Datatypes 3 | 4 | The Atomic Datatypes consist of some of the most commonly used [Datatypes](classes.md#Datatype). 5 | 6 | _Note: Please visit for the latest list of official Datatypes._ 7 | 8 | ## Slug 9 | 10 | _URL: `https://atomicdata.dev/datatypes/slug`_ 11 | 12 | A string with a limited set of allowed characters, used in IDE / Text editor context. 13 | Only letters, numbers and dashes are allowed. 14 | 15 | Regex: `^[a-z0-9]+(?:-[a-z0-9]+)*$` 16 | 17 | ## Atomic URL 18 | 19 | _URL: `https://atomicdata.dev/datatypes/atomicURL`_ 20 | 21 | A URL that should resolve to an [Atomic Resource](../core/concepts.md#Resource). 22 | 23 | ## URI 24 | 25 | _URL: `https://atomicdata.dev/datatypes/URI`_ 26 | 27 | A Uniform Resource Identifier, preferably a URL (i.e. an URI that can be fetched). 28 | Could be HTTP, HTTPS, or any other type of schema. 29 | 30 | ## String 31 | 32 | _URL: `https://atomicdata.dev/datatypes/string`_ 33 | 34 | UTF-8 String, no max character count. 35 | Newlines use backslash escaped `\n` characters. 36 | 37 | e.g. `String time! \n Second line!` 38 | 39 | ## Markdown 40 | 41 | _URL: `https://https://atomicdata.dev/datatypes/markdown`_ 42 | 43 | A markdown string, using the [CommonMark syntax](https://commonmark.org/). 44 | UTF-8 formatted, no max character count, newlines are `\n`. 45 | 46 | e.g. 47 | 48 | ```md 49 | # Heading 50 | 51 | Paragraph with [link](https://example.com). 52 | ``` 53 | 54 | ## Integer 55 | 56 | _URL: `https://atomicdata.dev/datatypes/integer`_ 57 | 58 | Signed Integer, max 64 bit. 59 | Max value: [`9223372036854775807`](https://en.wikipedia.org/wiki/9,223,372,036,854,775,807) 60 | 61 | e.g. `-420` 62 | 63 | ## Float 64 | 65 | _URL: `https://atomicdata.dev/datatypes/float`_ 66 | 67 | Number with a comma. 68 | Max value: [`9223372036854775807`](https://en.wikipedia.org/wiki/9,223,372,036,854,775,807) 69 | 70 | e.g. `-420` 71 | 72 | ## Boolean 73 | 74 | _URL: `https://atomicdata.dev/datatypes/boolean`_ 75 | 76 | True or false, one or zero. 77 | 78 | **String serialization** 79 | 80 | `true` or `false`. 81 | 82 | **Binary serialization** 83 | 84 | Use a single bit one boolean. 85 | 86 | 1 for `true`, or 0 for `false`. 87 | 88 | ## Date 89 | 90 | ISO date _without time_. 91 | `YYYY-MM-DD`. 92 | 93 | e.g. `1991-01-20` 94 | 95 | ## Timestamp 96 | 97 | _URL: `https://atomicdata.dev/datatypes/timestamp`_ 98 | 99 | Similar to [Unix Timestamp](https://www.unixtimestamp.com/). 100 | Milliseconds since midnight UTC 1970 Jan 01 (aka the [Unix Epoch](https://en.wikipedia.org/wiki/Unix_time)). 101 | Use this for most DateTime fields. 102 | Signed 64 bit integer (instead of 32 bit in Unix systems). 103 | 104 | e.g. `1596798919` (= 07 Aug 2020 11:15:19) 105 | 106 | ## ResourceArray 107 | 108 | _URL: `https://atomicdata.dev/datatypes/resourceArray`_ 109 | 110 | Sequential, ordered list of Atomic URIs. 111 | Serialized as a JSON array with strings. 112 | Note that other types of arrays are not included in this spec, but can be perfectly valid. 113 | 114 | ([Discussion](https://github.com/atomicdata-dev/atomic-data-docs/issues/127)) 115 | 116 | - e.g. `["https://example.com/1", "https://example.com/1"]` 117 | -------------------------------------------------------------------------------- /src/schema/intro.md: -------------------------------------------------------------------------------- 1 | {{#title Atomic Data Schema - modelling Atomic Data}} 2 | # Atomic Schema 3 | 4 | Atomic Schema is the proposed standard for specifying classes, properties and datatypes in Atomic Data. 5 | You can compare it to UML diagrams, or what XSD is for XML. 6 | Atomic Schema deals with validating and constraining the shape of data. 7 | It is designed for checking if all the required properties are present, and whether the values conform to the datatype requirements (e.g. `datetime`, or `URL`). 8 | 9 | This section will define various Classes, Properties and Datatypes (discussed in [Atomic Core: Concepts](../core/concepts.md)). 10 | 11 | ## Design Goals 12 | 13 | - **Decentralized**: Classes and Properties can be defined in external systems, and are resolved using web protocols such as HTTP. 14 | - **Typed**: Every Atom of data has a clear datatype. Validated data should be highly predictable. 15 | - **IDE-friendly**: Although Atomic Schema uses many URLs, users / developers should not have to type full URLs. The schema uses shortnames as aliases. 16 | - **Self-documenting**: When seeing a piece of data, simply following links will explain you how the data model is to be understood. This removes the need for (most of) existing API documentation. 17 | - **Extensible**: Anybody can create their own Datatypes, Properties and Classes. 18 | - **Accessible**: Support for languages, easily translatable. Useful for humans and machines. 19 | - **Atomic**: All the design goals of Atomic Data itself also apply here. Atomic Schema is defined using Atomic Data. 20 | 21 | ## In short 22 | 23 | In short, Atomic Schema works like this: 24 | 25 | The Property _field_ in an Atom, or the _key_ in a JSON-AD object, links to a **Property _Resource_**. 26 | It is important that the URL to the Property Resource resolves, as others can re-use it and check its datatype. 27 | This Property does three things: 28 | 29 | 1. it links to a **Datatype** which indicates which Value is acceptable. 30 | 1. it has a **description** which tells you what the property means, what the relationship between the Subject and the Value means. 31 | 1. it provides a **Shortname**, which is sometimes used as an alternative to the full URL of the Property. 32 | 33 | **DataTypes** define the shape of the Value, e.g. a Number (`124`) or Boolean (`true`). 34 | 35 | **Classes** are a special kind of Resource that describe an abstract class of things (such as "Person" or "Blog"). 36 | Classes can _recommend_ or _require_ a set of Properties. 37 | They behave as Models, similar to `struts` in C or `interfaces` in Typescript. 38 | A Resource _could_ have one or more classes, which _could_ provide information about which Properties are expected or required. 39 | 40 | **example:** 41 | 42 | ```json 43 | { 44 | "@id": "https://atomicdata.dev/classes/Agent", 45 | "https://atomicdata.dev/properties/description": "An Agent is a user that can create or modify data. It has two keys: a private and a public one. The private key should be kept secret. The public key is used to verify signatures (on [Commits](https://atomicdata.dev/classes/Commit)) set by the of the Agent.", 46 | "https://atomicdata.dev/properties/isA": [ 47 | "https://atomicdata.dev/classes/Class" 48 | ], 49 | "https://atomicdata.dev/properties/recommends": [ 50 | "https://atomicdata.dev/properties/name", 51 | "https://atomicdata.dev/properties/description" 52 | ], 53 | "https://atomicdata.dev/properties/requires": [ 54 | "https://atomicdata.dev/properties/publicKey" 55 | ], 56 | "https://atomicdata.dev/properties/shortname": "agent" 57 | } 58 | ``` 59 | -------------------------------------------------------------------------------- /src/usecases/data-catalog.md: -------------------------------------------------------------------------------- 1 | {{#title Atomic Server as a Data Catalog}} 2 | # Using Atomic-Server as a Data Catalog 3 | 4 | A data catalog is a system that collects metadata - data about data. 5 | They are inventories of datasets. 6 | 7 | They are often used to: 8 | 9 | - **Increase data-reuse of (open) datasets**. By making descriptions of datasets, you increase their discoverability. 10 | - **Manage data quality**. The more datasets you have, the more you'll want to make sure they are usable. This could mean settings serialization requirements or schema compliance. 11 | - **Manage compliance with privacy laws**. If you have datasets that contain GDPR-relevant data (personal data), you're legally required to maintain a list of where that data is stored, what you need it for and what you're doing with it. 12 | 13 | ## Why Atomic Server could be great for Data Catalogs 14 | 15 | [Atomic-Server](https://docs.atomicdata.dev/atomic-server.html) is a powerful Database that can be used as a modern, powerful data catalog. It has a few advantages over others: 16 | 17 | - Free & **open source**. MIT licensed! 18 | - Many built-in features, like **full-text search**, **history**, **live synchronization** and **rights management**. 19 | - Great **performance**. Requests take nanoseconds to milliseconds. 20 | - Very **easy to setup**. One single binary, no weird runtime dependencies. 21 | - Everything is linked data. Not just datasets (which you might), but also everything around them (users, comments, implementations). 22 | - Powerful **CMS capabilities**. With built in support for Tables and Documents, you can easily create webpages with articles or other types of resources using Atomic Server. 23 | - [Atomic Schema](../schema/intro.md) can be used to describe the **shape of your datasets**: the properties you use, which fields are required - things like that. Because Atomic Schema uses URLs, we can easily re-use properties and class definitions. This helps to make your datasets highly interoperable. 24 | 25 | ## When Atomic-Server is used for hosting the data, too 26 | 27 | Most datacatalogs only have metadata. However, if you convert your existing CSV / JSON / XML / ... datasets to _Atomic Data_, you can host them on Atomic-Server as well. This has a few advantages: 28 | 29 | - **Data previews** in the browser, users can navigate through the data without leaving the catalog. 30 | - Data itself becomes **browseable**, too, which means you can traverse a graph by clicking on link values. 31 | - **Standardized Querying** means you can easily, from the data catalog, can filter and sort the data. 32 | - **Cross-dataset search**. Search queries can be performed over multiple Atomic Data servers at once, enabling searching over multiple datasets. This is also called _federated search_. 33 | 34 | ## Atomic Server compared to CKAN 35 | 36 | - Atomic-Server is MIT licensed - which is more permissive than CKAN's AGPL license. 37 | - Whereas CKAN needs an external database, a python runtime, solrd and a HTTPS server, Atomic-Server has all of these built-in! 38 | - CKAN uses plain RDF, which has some [very important drawbacks](../interoperability/rdf.md). 39 | - But... Atomic-Server still misses a few essentials right now: 40 | 41 | ## What we should add to Atomic-Server before it's a decent Data Catalog 42 | 43 | - Add a model for datasets. This is absolutely essential. It could be based on (and link to) DCAT, but needs to be described using Atomic Schema. This step means we can generate forms for Datasets and we can validate their fields. 44 | - Add views for datasets. Atomic-Server already renders decent views for unknown resources, but a specific view should be created for Datasets. [Add a PR](https://github.com/atomicdata-dev/atomic-data-browser) if you have a React view! 45 | -------------------------------------------------------------------------------- /src/hierarchy.md: -------------------------------------------------------------------------------- 1 | {{#title Atomic Data Hierarchy, rights and authorization }} 2 | # Hierarchy, rights and authorization 3 | 4 | Hierarchies help make information easier to find and understand. 5 | For example, most websites use breadcrumbs to show you where you are. 6 | Your computer probably has a bunch of _drives_ and deeply nested _folders_ that contain _files_. 7 | We generally use these hierarchical elements to keep data organized, and to keep a tighter grip on rights management. 8 | For example, sharing a specific folder with a team, but a different folder could be private. 9 | 10 | Although you are free to use Atomic Data with your own custom authorization system, we have a standardized model that is currently being used by Atomic-Server. 11 | 12 | ## Design goals 13 | 14 | - **Fast**. Authorization can sometimes be costly, but in this model we'll be considering performance. 15 | - **Simple**. Easy to understand, easy to implement. 16 | - **Handles most basic use-cases**. Should deal with basic read / write access control, calculating the size of a folder, rendering things in a tree. 17 | 18 | ## Atomic Hierarchy Model 19 | 20 | - Every Resource SHOULD have a [`parent`](https://atomicdata.dev/properties/parent). There are some exceptions to this, which are discussed below. 21 | - Any Resource can be a `parent` of some other Resource, as long as both Resources exists on the same Atomic Server. 22 | - Grants / rights given in a `parent` also apply to all children, and their children. 23 | - There are few Classes that do not require `parent`s: 24 | 25 | ## Authorization 26 | 27 | - Any Resource might have [`read`](https://atomicdata.dev/properties/read) and [`write`](https://atomicdata.dev/properties/write) Atoms. These both contain a list of Agents. These Agents will be granted the rights to edit (using Commits) or read / use the Resources. 28 | - Rights are _additive_, which means that the rights add up. If a Resource itself has no `write` Atom containing your Agent, but it's `parent` _does_ have one, you will still get the `write` right. 29 | - Rights cannot be removed by children or parents - they can only be added. 30 | - `Commits` can not be edited. They can be `read` if the Agent has rights to read the [`subject`](https://atomicdata.dev/properties/subject) of the `Commit`. 31 | 32 | ## Top-level resources 33 | 34 | Some resources are special, as they do not require a `parent`: 35 | 36 | - [`Drive`](https://atomicdata.dev/classes/Drive)s are top-level items in the hierarchy: they do not have a `parent`. 37 | - [`Agent`](https://atomicdata.dev/classes/Agent)s are top-level items because they are not `owned` by anything. They can always `read` and `write` themselves. 38 | - [`Commit`](https://atomicdata.dev/classes/Commit)s are immutable, so they should never be edited by anyone. That's why they don't have a place in the hierarchy. Their `read` rights are determined by their subject. 39 | 40 | ## Authentication 41 | 42 | Authentication is about proving _who you are_, which is often the first step for authorization. See [authentication](./authentication.md). 43 | 44 | ## Current limitations of the Authorization model 45 | 46 | The specification is growing (and please contribute in the [docs repo](https://github.com/atomicdata-dev/atomic-data-docs/issues)), but the current specification lacks some features: 47 | 48 | - Rights can only be added, but not removed in the hierarchy. This means that you cannot have a secret folder inside a public folder. 49 | - No model for representing groups of Agents, or other runtime checks for authorization. ([issue](https://github.com/atomicdata-dev/atomic-data-docs/issues/73)) 50 | - No way to limit delete access or invite rights separately from write rights ([issue](https://github.com/atomicdata-dev/atomic-data-docs/issues/82)) 51 | - No way to request a set of rights for a Resource 52 | -------------------------------------------------------------------------------- /src/commits/intro.md: -------------------------------------------------------------------------------- 1 | {{#title Atomic Commits - Event standard for Atomic Data}} 2 | # Atomic Commits 3 | 4 | _Disclaimer: Work in progress, prone to change._ 5 | 6 | Atomic Commits is a specification for communicating _state changes_ (events / transactions / patches / deltas / mutations) of [Atomic Data](../core/concepts.md). 7 | It is the part of Atomic Data that is concerned with writing, editing, removing and updating information. 8 | 9 | ## Design goals 10 | 11 | - **Event sourced**: Store and standardize _changes_, as well as the _current_ state. This enables versioning, history playback, undo, audit logs, and more. 12 | - **Traceable origin**: Every change should be traceable to an actor and a point in time. 13 | - **Verifiable**: Have cryptographic proof for every change. Know _when_, and _what_ was changed by _whom_. 14 | - **Identifiable**: A single commit has an identifier - it is a resource. 15 | - **Decentralized**: Commits can be shared in P2P networks from device to device, whilst maintaining verifiability. 16 | - **Extensible**: The methods inside a commit are not fixed. Use-case specific methods can be added by anyone. 17 | - **Streamable**: The commits could be used in streaming context. 18 | - **Familiar**: Introduces as little new stuff as possible (no new formats or language to learn) 19 | - **Pub/Sub**: Subscribe to changes and get notified on changes. 20 | - **ACID-compliant**: An Atomic commit will only occur if it results in a valid state. 21 | - **Atomic**: All the Atomic Data design goals also apply here. 22 | 23 | ## Motivation 24 | 25 | Although it's a good idea to keep data at the source as much as possible, we'll often need to synchronize two systems. 26 | For example when data has to be queried or indexed differently than its source can support. 27 | Doing this synchronization can be very difficult, since most of our software is designed to only maintain and share the _current state_ of a system. 28 | 29 | I noticed this mainly when working on OpenBesluitvorming.nl - an open data project where we aimed to fetch and standardize meeting data (votes, meeting minutes, documents) from 150+ local governments in the Netherlands. 30 | We wrote software that fetched data from various systems (who all had different models, serialization formats and APIs), transformed this data to a single standard and share it through an API and a fulltext search endpoint. 31 | One of the hard parts was keeping our data in sync with the sources. 32 | How could we now if something was changed upstream? 33 | We queried all these systems every night for _all meetings from the next and previous month_, and made deep comparisons to our own data. 34 | 35 | This approach has a couple of issues: 36 | 37 | - It costs a lot of resources, both for us and for the data suppliers. 38 | - It's not real-time - we can only run this once every 24 ours (because of how costly it is). 39 | - It's very prone to errors. We've had issues during all phases of Extraction, Transformation and Loading (ETL) processing. 40 | - It causes privacy issues. When some data at the source is removed (because it contained faulty or privacy sensitive data), how do we learn about that? 41 | 42 | Persisting and sharing state changes could solve these issues. 43 | In order for this to work, we need to standardize this for all data suppliers. 44 | We need a specification that is easy to understand for most developers. 45 | 46 | Keeping track of where data comes from is essential to knowing whether you can trust it - whether you consider it to be true. 47 | When you want to persist data, that quickly becomes bothersome. 48 | Atomic Data and Atomic Commits aim to make this easier by using cryptography for ensuring data comes from some particular source, and is therefore trustworthy. 49 | 50 | If you want to know how Atomic Commits differ from other specs, see the [compare section](compare.md) 51 | -------------------------------------------------------------------------------- /src/usecases/headless-cms.md: -------------------------------------------------------------------------------- 1 | # Using Atomic-Server as an open source headless CMS 2 | 3 | ## Why people are switching to Headless CMS 4 | 5 | Traditionally, content management systems were responsible for both managing the content as well as producing the actual HTML views that the user saw. 6 | This approach has some issues regarding performance and flexibility that headless CMS tools solve. 7 | 8 | - **Great performance**. We want pages to load in milliseconds, not seconds. Headless CMS tools + JAMSTACK style architectures are designed to give both performant initial page loads, as well as consecutive / dynamic loads. 9 | - **High flexibility**. Designs change, and front-end developers want to use the tools that they know and love to create these designs effectively. With a headless CMS, you can build the front-end with the tools that you want, and make it look exactly like you want. 10 | - **Easier content management**. Not every CMS is as fun and easy to use, as an admin, as others. Headless CMS tools focus on the admin side of things, so the front-end devs don't have to work on the back-end as well. 11 | 12 | ## Atomic Server 13 | 14 | The [Atomic-Server](https://github.com/atomicdata-dev/atomic-server/blob/master/server/README.md) project may be the right choice for you if you're looking for a Headless CMS: 15 | 16 | 17 | - **Free and open source**. MIT licensed, no strings attached. 18 | - **Easy to use API**. Atomic-Server is built using the [Atomic Data specification](../atomic-data-overview.md). It is well-documented, and uses conventions that most web developers are already familiar with. 19 | - **Typescript & React libraries**. Use the existing react hooks to make your own fully editable, live-reloaded web application. 20 | - **Fast**. 1ms responses on my laptop. It's written in Rust, so it squeezes out every cycle of your server. 21 | - **Lightweight**. It's a single 8MB binary, no external dependencies needed. 22 | - **Easy to setup**. Just run the binary and open the address. Even HTTPS support is built-in. 23 | - **Clean, powerful admin GUI**. The Atomic-Data-Browser front-end gives you a very easy interface to manage your content. 24 | - **Share your data models**. Atomic Data is designed to achieve a more decentralized web. You can easily re-use existing data models, or share the ones you built. 25 | - **Files / Attachments**. Upload and preview files. 26 | - **Pagination / sorting / filtering**. Query your data. 27 | - **Versioning**. Built-in history, where each transaction is saved. 28 | - **Websockets**. If you need live updates and highly interactive apps (collaborative documents and chatrooms), we've got your back. 29 | - **Full-text search**. No need for a big elasticsearch server - atomic-server has one built-in. 30 | 31 | ## Limitations 32 | 33 | - No support for image resizing, [as of now](https://github.com/atomicdata-dev/atomic-server/issues/257) 34 | - No GraphQL support [(see issue)](https://github.com/atomicdata-dev/atomic-server/issues/251) 35 | 36 | ## Setting up the server 37 | 38 | - One-liners: `cargo install atomic-server` or `docker run -p 80:80 -v atomic-storage:/atomic-storage joepmeneer/atomic-server` 39 | - Check out the [readme!](https://github.com/atomicdata-dev/atomic-server) 40 | 41 | ## Using the data in your (React / NextJS) app 42 | 43 | The `@tomic/lib` and `@tomic/react` typescript NPM libraries can be used in any JS project. 44 | 45 | In the next section, we'll discuss how to use Atomic-Server in your React project. 46 | 47 | ## Compared to alternative open source headless CMS software 48 | 49 | - **Strapi**: Atomic-Server doesn't need an external database, is easier to setup, has live synchronization support and is way faster. However, Strapi has a plugin system, is more polished, and has GraphQL support. 50 | - ** 51 | -------------------------------------------------------------------------------- /src/atomic-data-overview.md: -------------------------------------------------------------------------------- 1 | {{#title Atomic Data}} 2 | ![# Atomic Data Docs - Overview](assets/atomic_data_logo_stroke.svg) 3 | 4 | **Atomic Data is a modular specification for sharing, modifying and modeling graph data. It combines the ease of use of JSON, the connectivity of RDF (linked data) and the reliability of type-safety.** 5 | 6 | ![Venn diagram showing Atomic Data is the combination of JSON, RDF and Type-Safety](assets/venn.svg) 7 | 8 | Atomic Data uses links to connect pieces of data, and therefore makes it easier to connect datasets to each other - even when these datasets exist on separate machines. 9 | 10 | Atomic Data has been designed with [the following goals in mind](motivation.md): 11 | 12 | - Give people more control over their data 13 | - Make linked data easier to use 14 | - Make it easier for developers to build highly interoperable apps 15 | - Make standardization easier and cheaper 16 | 17 | Atomic Data is [Linked Data](https://ontola.io/what-is-linked-data/), as it is a [strict subset of RDF](interoperability/rdf.md). 18 | It is type-safe (you know if something is a `string`, `number`, `date`, `URL`, etc.) and extensible through [Atomic Schema](schema/intro.md), which means that you can re-use or define your own Classes, Properties and Datatypes. 19 | 20 | The default serialization format for Atomic Data is [JSON-AD](core/json-ad.md), which is simply JSON where each key is a URL of an Atomic Property. 21 | These Properties are responsible for setting the `datatype` (to ensure type-safety) and setting `shortnames` (which help to keep names short, for example in JSON serialization) and `descriptions` (which provide semantic explanations of what a property should be used for). 22 | 23 | [Read more about Atomic Data Core](core/concepts.md) 24 | 25 | ## Atomic Data Extended 26 | 27 | Atomic Data Extended is a set of extra modules (on top of Atomic Data Core) that deal with data that changes over time, authentication, and authorization. 28 | 29 | {{#include extended-table.md}} 30 | 31 | ## Atomizing: how to create, convert and host Atomic Data 32 | 33 | Atomic Data has been designed to be very easy to create and host. 34 | In the Atomizing section, we'll show you how you can create Atomic Data in three ways: 35 | 36 | - [Using Atomic Server, from your browser](atomic-server.md) 37 | - [By creating JSON-AD (and optionally importing it)](create-json-ad.md) 38 | - [By upgrading your existing application](interoperability/upgrade.md) 39 | 40 | ## Tools & libraries 41 | 42 | - Browser app [atomic-data-browser](https://github.com/atomicdata-dev/atomic-data-browser) ([demo on atomicdata.dev](https://atomicdata.dev)) 43 | - Build a react app using [typescript & react libraries](https://github.com/atomicdata-dev/atomic-data-browser). Start with the [react template on codesandbox](https://codesandbox.io/s/atomic-data-react-template-4y9qu?file=/src/MyResource.tsx) 44 | - Host your own [atomic-server](https://github.com/atomicdata-dev/atomic-server) (powers [atomicdata.dev](https://atomicdata.dev), run with `docker run -p 80:80 -v atomic-storage:/atomic-storage joepmeneer/atomic-server`) 45 | - Discover the command line tool: [atomic-cli](https://github.com/atomicdata-dev/atomic-server) (`cargo install atomic-cli`) 46 | - Use the Rust library: [atomic-lib](https://github.com/atomicdata-dev/atomic-server) 47 | 48 | ## Get involved 49 | 50 | Make sure to [join our Discord](https://discord.gg/a72Rv2P) if you'd like to discuss Atomic Data with others. 51 | 52 | ## Status 53 | 54 | Keep in mind that none of the Atomic Data projects has reached a v1, which means that breaking changes can happen. 55 | 56 | ## Reading these docs 57 | 58 | This is written mostly as a book, so reading it in the order of the Table of Contents will probably give you the best experience. 59 | That being said, feel free to jump around - links are often used to refer to earlier discussed concepts. 60 | If you encounter any issues while reading, please leave an [issue on Github](https://github.com/ontola/atomic-data/issues). 61 | Use the arrows on the side / bottom to go to the next page. 62 | 63 | {{#include SUMMARY.md}} 64 | -------------------------------------------------------------------------------- /src/core/paths.md: -------------------------------------------------------------------------------- 1 | {{#title Atomic Data Paths}} 2 | # Atomic Paths 3 | 4 | An Atomic Path is a string that consists of at least one URL, followed by one or more URLs or Shortnames. 5 | Every single value in an Atomic Resource can be targeted through such a Path. 6 | They can be used as identifiers for specific Values. 7 | 8 | The simplest path, is the URL of a resource, which represents the entire Resource with all its properties. 9 | If you want to target a specific atom, you can use an Atomic Path with a second URL. 10 | This second URL can be replaced by a Shortname, if the Resource is an instance of a class which has properties with that Shortname (sounds more complicated than it is). 11 | 12 | ## Example 13 | 14 | Let's start with this simple Resource: 15 | 16 | ```json 17 | { 18 | "@id": "https://example.com/john", 19 | "https://example.com/lastName": "McLovin", 20 | } 21 | ``` 22 | 23 | Then the following Path targets the `McLovin` value: 24 | 25 | `https://example.com/john https://example.com/lastName` => `McLovin` 26 | 27 | Instead of using the full URL of the `lastName` Property, we can use its [shortname](https://atomicdata.dev/properties/shortname): 28 | 29 | `https://example.com/john lastname` => `McLovin` 30 | 31 | We can also traverse relationships between resources: 32 | 33 | ```json 34 | [{ 35 | "@id": "https://example.com/john", 36 | "https://example.com/lastName": "McLovin", 37 | "https://example.com/employer": "https://example.com/XCorp", 38 | },{ 39 | "@id": "https://example.com/XCorp", 40 | "https://example.com/description": "The greatest company!", 41 | }] 42 | ``` 43 | 44 | `https://example.com/john employer description` => `The greatest company!` 45 | 46 | In the example above, the XCorp subject exists and is the source of the `The greatest company!` value. 47 | We can use this path as a unique identifier for the description of John's current employer. 48 | Note that the data for the description of that employer does not have to be in John's control for this path to work - it can live on a totally different server. 49 | However, in Atomic Data it's also possible to include this description in the resource of John as a _Nested Resource_. 50 | 51 | ## Nested Resources 52 | 53 | All Atomic Data Resources that we've discussed so far have an explicit URL as a subject. 54 | Unfortunately, creating unique and resolvable URLs can be a bother, and sometimes not necessary. 55 | If you've worked with RDF, this is what Blank Nodes are used for. 56 | In Atomic Data, we have something similar: _Nested Resources_. 57 | 58 | Let's use a Nested Resource in the example from the previous section: 59 | 60 | ```json 61 | { 62 | "@id": "https://example.com/john", 63 | "https://example.com/lastName": "McLovin", 64 | "https://example.com/employer": { 65 | "https://example.com/description": "The greatest company!", 66 | } 67 | } 68 | ``` 69 | 70 | Now the `employer` is simply a nested Object. 71 | Note that it no longer has its own `@id`. 72 | However, we can still identify this Nested Resource using its Path. 73 | 74 | The Subject of the nested resource is its path: `https://example.com/john https://example.com/employer`, including the spacebar. 75 | 76 | Note that the path from before still resolves: 77 | 78 | `https://example.com/john employer description` => `The greatest company!` 79 | 80 | ## Traversing Arrays 81 | 82 | We can also navigate Arrays using paths. 83 | 84 | For example: 85 | 86 | ```json 87 | { 88 | "@id": "https://example.com/john", 89 | "hasShoes": [ 90 | { 91 | "https://example.com/name": "Mr. Boot", 92 | }, 93 | { 94 | "https://example.com/name": "Sunny Sandals", 95 | } 96 | ] 97 | } 98 | ``` 99 | 100 | The Path of `Mr. Boot` is: 101 | 102 | ``` 103 | https://example.com/john hasShoes 0 name 104 | ``` 105 | 106 | You can target an item in an array by using a number to indicate its position, starting with 0. 107 | 108 | Notice how the Resource with the `name: Mr. Boot` does not have an explicit `@id`, but it _does_ have a Path. 109 | This means that we still have a unique, globally resolvable identifier - yay! 110 | 111 | ## Try for yourself 112 | 113 | Install the [`atomic-cli`](https://github.com/atomicdata-dev/atomic-server/blob/master/cli/README.md) software and run `atomic-cli get https://atomicdata.dev/classes/Class description`. 114 | -------------------------------------------------------------------------------- /src/core/serialization.md: -------------------------------------------------------------------------------- 1 | {{#title Serialization of Atomic Data}} 2 | # Serialization of Atomic Data 3 | 4 | Atomic Data is not necessarily bound to a single serialization format. 5 | It's fundamentally a data model, and that's an important distinction to make. 6 | It can be serialized in different ways, but there is only one required: `JSON-AD`. 7 | 8 | ## JSON-AD 9 | 10 | [`JSON-AD`](json-ad.md) (more about that on the next page) is specifically designed to be a simple, complete and performant format for Atomic Data. 11 | 12 | ```json 13 | { 14 | "@id": "https://atomicdata.dev/properties/description", 15 | "https://atomicdata.dev/properties/datatype": "https://atomicdata.dev/datatypes/markdown", 16 | "https://atomicdata.dev/properties/description": "A textual description of something. When making a description, make sure that the first few words tell the most important part. Give examples. Since the text supports markdown, you're free to use links and more.", 17 | "https://atomicdata.dev/properties/isA": [ 18 | "https://atomicdata.dev/classes/Property" 19 | ], 20 | "https://atomicdata.dev/properties/parent": "https://atomicdata.dev/properties", 21 | "https://atomicdata.dev/properties/shortname": "description" 22 | } 23 | ``` 24 | 25 | [Read more about JSON-AD](json-ad.md) 26 | 27 | ## JSON (simple) 28 | 29 | Atomic Data is designed to be serializable to clean, simple [JSON](../interoperability/json.md), for usage in (client) apps that don't need to know the full URLs of properties. 30 | 31 | ````json 32 | { 33 | "@id": "https://atomicdata.dev/properties/description", 34 | "datatype": "https://atomicdata.dev/datatypes/markdown", 35 | "description": "A textual description of something. When making a description, make sure that the first few words tell the most important part. Give examples. Since the text supports markdown, you're free to use links and more.", 36 | "is-a": [ 37 | "https://atomicdata.dev/classes/Property" 38 | ], 39 | "parent": "https://atomicdata.dev/properties", 40 | "shortname": "description" 41 | } 42 | ```` 43 | 44 | [Read more about JSON and Atomic Data](json-ad.md) 45 | 46 | 47 | ## RDF serialization formats 48 | 49 | Since Atomic Data is a strict subset of RDF, RDF serialization formats can be used to communicate and store Atomic Data, such as N-Triples, Turtle, HexTuples, JSON-LD and [other RDF serialization formats](https://ontola.io/blog/rdf-serialization-formats/). 50 | However, not all valid RDF is valid Atomic Data. 51 | Atomic Data is more strict. 52 | Read more about serializing Atomic Data to RDF in the [RDF interoperability section](../interoperability/rdf.md). 53 | 54 | JSON-LD: 55 | 56 | ```json 57 | { 58 | "@context": { 59 | "datatype": { 60 | "@id": "https://atomicdata.dev/properties/datatype", 61 | "@type": "@id" 62 | }, 63 | "description": "https://atomicdata.dev/properties/description", 64 | "is-a": { 65 | "@container": "@list", 66 | "@id": "https://atomicdata.dev/properties/isA" 67 | }, 68 | "parent": { 69 | "@id": "https://atomicdata.dev/properties/parent", 70 | "@type": "@id" 71 | }, 72 | "shortname": "https://atomicdata.dev/properties/shortname" 73 | }, 74 | "@id": "https://atomicdata.dev/properties/description", 75 | "datatype": "https://atomicdata.dev/datatypes/markdown", 76 | "description": "A textual description of something. When making a description, make sure that the first few words tell the most important part. Give examples. Since the text supports markdown, you're free to use links and more.", 77 | "is-a": [ 78 | "https://atomicdata.dev/classes/Property" 79 | ], 80 | "parent": "https://atomicdata.dev/properties", 81 | "shortname": "description" 82 | } 83 | ``` 84 | 85 | Turtle / N-Triples: 86 | 87 | ```turtle 88 | . 89 | . 90 | "description"^^ . 91 | "https://atomicdata.dev/classes/Property"^^ . 92 | "A textual description of something. When making a description, make sure that the first few words tell the most important part. Give examples. Since the text supports markdown, you're free to use links and more."^^ . 93 | ``` 94 | -------------------------------------------------------------------------------- /src/interoperability/graph-database.md: -------------------------------------------------------------------------------- 1 | {{#title How does Atomic Data relate to Graph Databases?}} 2 | # Atomic Data and Graph Databases 3 | 4 | Atomic Data is fundamentally a _graph data model_. 5 | We can think of Atomic Resources as _nodes_, and links to other resources through _properties_ as _edges_. 6 | 7 | In the first section, we'll take a look at Atomic-Server as a Graph Database. 8 | After that, we'll explore how Atomic Data relates to some graph technologies. 9 | 10 | ## Atomic-Server as a database 11 | 12 | - **Built-in REST**. Everything is done over HTTP, there's no new query language or serialization to learn. It's all JSON. 13 | - **All resources have HTTP URLs**. This means that every single thing is identified by where it can be be found. Makes it easy to share data, if you want to! 14 | - **Sharable and re-usable data models**. Atomic Schema helps you share and re-use data models by simply pointing to URLs. 15 | - **Authorization built-in**. Managing rights in a hierarchy (similar to how tools like Google Drive or filesystems work) enable you to have a high degree of control over read / write rights. 16 | - **Built-in easy to use GUI**. Managing content on Atomic-Server can be done by anyone, as its GUI is extremely easy to use and has a ton of features. 17 | - **Dynamic indexing**. Indexes are created by performing Queries, resulting in great performance - without needing to manually configure indexing. 18 | - **Synchronization over WebSockets**. All changes (called [Commits](../commits/intro.md)) can be synchronized over WebSockets, allowing you to build realtime collaborative tools. 19 | - **Event-sourced**. All changes are stored and reversible, giving you a full versioned history. 20 | - **Open source**. All code is MIT-licensed. 21 | 22 | ## Comparing Atomic Data to Neo4j 23 | 24 | Neo4j is a popular graph database that supports multiple query languages. 25 | The first difference is that Atomic Data is not a single piece of software but a _specification_. 26 | However, we can compare Neo4j as a _product_ with the open source [Atomic-Server](https://crates.io/crates/atomic-server). 27 | Atomic-Server is fully open source and free (MIT licensed), whereas Neo4j is partially open source and GPL licensed. 28 | 29 | ### Labeled Property Graph 30 | 31 | The data model of Neo4j features a _labeled property graph_, which means that edges (relationships between nodes) can have their own properties. 32 | This can be useful when adding data to relationship between nodes. 33 | For example: in the `john - (knows) -> mary` relationship, you might want to specify _for how long_ they have known each other. 34 | In Neo4j, we can add this data to the labeled property graph. 35 | 36 | In Atomic Data, we'd have to make a new resource to describe the relation between the two, if we wanted to add information about the relationship itself. 37 | This is called _reification_. 38 | This process can be time consuming, especially in Atomic Data, as this means that you'll have to specify the Class of this relationship and its properties. 39 | However, one benefit of this approach, is that the relationship itself becomes clearly defined and re-usable. 40 | Another benefit is that the simpler model of Atomic Data maps perfectly to datamodels like JSON, which makes things very convenient and familiar for developers. 41 | 42 | ### Query language vs REST 43 | 44 | Neo4j supports multiple query languages, but its mainly known for _Cypher_. 45 | It is used for doing practically everything: reading, writing, modelling, and more. 46 | 47 | Atomic Data on the other hand does not have a query language. 48 | It uses a RESTful HTTP + JSON-AD approach for everything. 49 | Atomic Data uses [Endpoints](../endpoints.md) for specific goals that you'd do in a query language: 50 | 51 | - [Collections](../schema/collections.md) (which can filter by Property or Value, and sort by any Property) to generate lists of resources 52 | - [Paths](../core/paths.md) for traversing graphs by property 53 | 54 | And finally, data is written using [Commits](../commits/intro.md). 55 | Commits are very strict, as each one describes modifications to individual resources, and every Commits has to be signed. 56 | This means that with Atomic Data, we get _versioning + audit trails_ for all data, but at the cost of more storage requirements and a bit more expensive write process. 57 | 58 | ### Schema language and type safety 59 | 60 | In Neo4j, constraints can be added to the database by 61 | Atomic Data uses [Atomic Schema](../schema/intro.md) for validating datatypes and required properties in [Classes](../schema/classes.md). 62 | 63 | ### Other differences 64 | 65 | - Atomic Data has an [Authentication model](../agents.md) and [Hierarchy model](../hierarchy.md) for authorization. Neo4j uses [roles](https://neo4j.com/docs/operations-manual/current/authentication-authorization/built-in-roles/#auth-built-in-roles). 66 | - Neo4j is actually used in production by many big organizations 67 | -------------------------------------------------------------------------------- /src/schema/faq.md: -------------------------------------------------------------------------------- 1 | {{#title Atomic Schema FAQ}} 2 | # Atomic Schema FAQ 3 | 4 | ## How do I create a Property that supports multiple Datatypes? 5 | 6 | A property only has one single Datatype. 7 | However, feel free to create a new kind of Datatype that, in turn, refers to other Datatypes. 8 | Perhaps Generics, or Option like types should be part of the Atomic Base Datatypes. 9 | 10 | ## Do you have an `enum` datatype? 11 | 12 | In Atomic Data, `enum` is not a datatype, but it's a constraint that can be added to properties that have. 13 | You can set [`allows-only`](https://atomicdata.dev/properties/allowsOnly) on a Property, and use that to limit which values are allowed. 14 | 15 | ## How should a client deal with Shortname collisions? 16 | 17 | Atomic Data guarantees Subject-Property uniqueness, which means that Valid Resources are guaranteed to have only one of each Property. 18 | Properties offer Shortnames, which are short strings. 19 | These strings should be unique inside Classes, but these are not guaranteed to be unique inside all Resources. 20 | Note that Resources can have multiple Classes, and through that, they can have colliding Shortnames. 21 | Resources are also free to include Properties from other Classes, and their Shortnames, too, might collide. 22 | 23 | For example: 24 | 25 | ```json 26 | { 27 | "@id": "https://example.com/people/123", 28 | "https://example.com/name": "John", 29 | "https://another.example.com/someOtherName": "Martin" 30 | } 31 | ``` 32 | 33 | Let's assume that `https://example.com/name` and `https://another.example.com/someOtherName` are Properties that have the Shortname: `name`. 34 | 35 | What if a client tries something such as `people123.name`? 36 | To consistently return a single value, we need some type of _precedence_: 37 | 38 | 1. The earlier Class mentioned in the [`isA`](https://atomicdata.dev/properties/isA) Property of the resource. Resources can have multiple classes, but they appear in an ordered ResourceArray. Classes, internally should have no key collisions in required and recommended properties, which means that they might have. If these exist internally, sort the properties by how they are ordered in the `isA` array - first item is preferred. 39 | 1. When the Properties are not part of any of the mentioned Classes, use Alphabetical sorting of the Property URL. 40 | 41 | When shortname collisions are possible, it's recommended to not use the shortname, but use the URL of the Property: 42 | 43 | ``` 44 | people123."https://example.com/name" 45 | ``` 46 | 47 | It is likely that using the URL for keys is also the most _performant_, since it probably more closely mimics the internal data model. 48 | 49 | ## Atomic Data uses a lot of links. How do you deal with links that don't work? 50 | 51 | Many features in Atomic Data apps depend on the availability of Resources on their subject URL. 52 | If that server is offline, or the URL has changed, the existing links will break. 53 | This is a fundamental problem to HTTP, and not unique to Atomic Data. 54 | Like with websites, hosts should make sure that their server stays available, and that URLs remain static. 55 | 56 | One possible solution to this problem, is using Content Addressing, such as the [IPFS](../interoperability/ipfs.md) protocol enables, which is why we're planning for using that in the near future. 57 | 58 | Another approach, is using [foreign keys (see issue)](https://github.com/ontola/atomic-data-docs/issues/43). 59 | 60 | ## How does Atomic Schema relate to RDF / SHACL / SheX / OWL / RDFS? 61 | 62 | Atomic Schema is _the_ schema language for Atomic Data, whereas RDF has a couple of competing ones, which all vary greatly. 63 | In short, OWL is not designed for schema validation, but SHACL and SheX can maybe be compared to Atomic Schema. 64 | An important difference is that SHACL and SheX have to deal with all the complexities of RDF, whereas Atomic Data is more constrained. 65 | 66 | For more information, see [RDF interoperability](../interoperability/rdf.md). 67 | 68 | ## What are the risks of using Schema data hosted somewhere else? 69 | 70 | Every time you use an external URL in your data, you kind of create a dependency. 71 | This is fundamental to linked data. 72 | In Atomic Data, not having access to the Property in some JSON-AD resource will lead to now knowing how to interpret the data itself. 73 | You will no longer know what the Datatype was (other than the native JSON datatype, of course), or what the semantic meaning was of the relationship. 74 | 75 | There are multiple ways we can deal with this: 76 | 77 | - **Cache dependencies**: Atomic Server already stores a copy of every class and property that it uses by default. The `/path` endpoint then allows clients to fetch these from servers that have cached it. If the source goes offline, the validations can still be performed by the server. However, it might be a good idea to migrate the data to a hosted ontology, e.g. by cloning the cached ontology. 78 | - **Content-addressing**: using non-HTTP identifiers, such as with [IPFS](../interoperability/ipfs.md). 79 | 80 | ([Discussion](https://github.com/ontola/atomic-data-docs/issues/99)) 81 | 82 | ## How do I deal with subclasses / inheritance? 83 | 84 | Atomic Data does not have a concept of inheritance. 85 | However, you can use the `isA` property to link to _multiple Classes_ from a single resource. 86 | This effectively 87 | -------------------------------------------------------------------------------- /src/usecases/surveys.md: -------------------------------------------------------------------------------- 1 | {{#title Atomic Data for Surveys}} 2 | # Atomic Data for Surveys 3 | 4 | Surveys and Questionnaires haven't been evolving that much over the past few years. 5 | However, Atomic Data has a couple of unique characteristics that would make it especially suitable for surveys. 6 | It could help make surveys easier to **fill in**, easier to **analyze**, easier to **create**, and more **privacy friendly**. 7 | 8 | - **Re-useable survey responses** which enable **pre-filled form fields** which can save the respondent a lot of time. They also make it possible for users to use their own responses to **gather insights**, for example into their own health. 9 | - **Question standardization** which helps researchers to re-use (validated) questions, which saves time for the researcher 10 | - **Privacy friendly, yet highly personalized invites** as a researcher, send profile descriptions to servers, and let the servers tell if the question is relevant. 11 | 12 | ## Re-useable survey responses 13 | 14 | Since many surveys describe personal information, it makes sense, as a respondent, to have a way of storing the information you filled in in a place that you control. 15 | Making this possible enables a few nice use cases. 16 | 17 | 1. **Auto-fill forms**. Previously entered response data could be usable while filling in new surveys. This could result in a UX similar to auto-filling forms, but far more powerful and rich than browsers currently support. 18 | 2. **Analyze your own personal data**. Standardized survey responses could also be used to gather insights into your own personal information. For example, filling in a survey about how your shortness of breath linked to air pollution has been today could be used in a different app to make a graph that visualizes how your shortness of breath has progressed over the months for personal insight. 19 | 20 | Achieving something like this requires a high degree of standardization in both the surveys and the responses. The survey and its questions should provide information about: 21 | 22 | - The **question**. This is required in all survey questions, of course. 23 | - The **required datatype** of the response, such as 'string', or 'datetime' or some 'enumeration'. 24 | - A (link to a) **semantic definition** of the property being described. This is a bit more obscure: all pieces of linked data use links, instead of keys, to describe the relation between some resource and its property. For example, a normal resource might have a 'birthdate', while in linked data, we'd use ''. This semantic definition makes things easier to share, because it prevents misinterpretation. Links remove ambiguity. 25 | - **A query description**. This is even more obscure, but perhaps the most interesting. A query description means describing how a piece of information can be retrieved. Perhaps a question in a survey will want to know what your payment pointer is. If a piece of software wants to auto-fill this field, it needs to know where it can find your payment pointer. 26 | 27 | ## Question Standardization 28 | 29 | We can think of Questions as Resources that have a URL, and can be shared. 30 | Sharing questions like that can make it easier to use the same questions across surveys, which in turn can make it easier to interpret data. 31 | Some fields (e.g. medical) have highly standardized questions, which have been validated by studies. 32 | These Question resources should contain information about: 33 | 34 | - The **question** itself and its translations 35 | - The **datatype** of the response (e.g. `date`, `string`, `enum`), denoted by the [Property](https://atomicdata.dev/classes/Property) of the response. 36 | - The **path of the data**, relative to the user. For example, a user's `birthdate` can be found by going to `/ profile birthdate` 37 | 38 | [Atomic Schema](../schema/intro.md) and [Atomic Paths](../core/paths.md) can be of value here. 39 | 40 | ## Privacy friendly invites with client-side filtering 41 | 42 | Currently, a researcher needs to either build their own panel, or use a service that has a lot of respondents. 43 | Sometimes, researchers will need a very specific target audience, like a specific age group, nationality, gender, or owners of specific types of devices. 44 | Targeting these individuals is generally done by having a large database of personal information from many individuals. 45 | But there is another way of doing this: **client-side filtering** 46 | Instead of asking for the users data, and storing it centralized, we could send queries to decentralized personal data stores. 47 | There queries basically contain the targeting information and an invitation. 48 | The query is executed on the personal data store, and if the user characteristics align with the desired participants profile, the user receives an invite. 49 | The user only sees invitations that are highly relevant, without sharing _any_ information with the researcher. 50 | 51 | The Atomic Data specification solves at least part of this problem. 52 | [Paths](../core/paths.md) are used to describe the queries that researchers make. 53 | [AtomicServer](https://github.com/atomicdata-dev/atomic--rust/blob/master/server/README.md) can be used as the personal online data store. 54 | 55 | However, we still need to specify the process of sending a request to an individual (probably by introducing an [inbox](https://github.com/ontola/atomic-data/issues/28)) 56 | -------------------------------------------------------------------------------- /src/tooling.md: -------------------------------------------------------------------------------- 1 | {{#title Software and libraries for Atomic Data}} 2 | # Software and libraries for Atomic Data 3 | 4 | Although Atomic Data is a specification, it also has reference implementations: 5 | 6 | Open source (MIT licenced) software for Atomic Data: 7 | 8 | - **Server + Database**: [atomic-server](https://github.com/atomicdata-dev/atomic-server) 9 | - **GUI**: [atomic-data-browser](https://github.com/atomicdata-dev/atomic-data-browser) 10 | - **CLI**: [atomic-cli](https://github.com/atomicdata-dev/atomic-server) 11 | 12 | Libraries (MIT licenced) to build apps with: 13 | 14 | - Typescript / javascript library: [@tomic/lib (npm)](https://www.npmjs.com/package/@tomic/lib) 15 | - React library: [@tomic/react (npm)](https://www.npmjs.com/package/@tomic/react) 16 | - Rust library: [atomic-lib (crates.io)](https://crates.io/crates/atomic-lib) 17 | 18 | ## Applications 19 | 20 | ### `atomic-server` 21 | 22 | Server for hosting Atomic Data. Uses `atomic-lib`. 23 | 24 | - Responds to requests for created Atomic Resources, makes atomic data available at their URL. 25 | - Embedded database 26 | - Authorization, authentication, versioning, collections, pagination 27 | - Browser-friendly HTML presentation, JSON serialization, RDF serialization. 28 | 29 | One liner: `$ docker run -p 80:80 -v atomic-storage:/atomic-storage joepmeneer/atomic-server` 30 | 31 | [demo](https://atomicdata.dev/) 32 | 33 | [repository + issue tracker](https://github.com/atomicdata-dev/atomic-data-browser). 34 | 35 | ### `atomic-data-browser` 36 | 37 | Data browser, powered by `@tomic/lib` and `@tomic/react`. 38 | 39 | - View & edit atomic data, using dynamic forms 40 | - Collections with pagination and sorting 41 | - Client-side full-text search 42 | 43 | [demo](https://atomicdata.dev/) (same as `atomic-server`) 44 | 45 | [repository + issue tracker](https://github.com/atomicdata-dev/atomic-data-browser). 46 | 47 | ### `atomic-cli` 48 | 49 | A tool for generating / querying Atomic Data from the command line. Install with `cargo install atomic-cli`. 50 | 51 | ``` 52 | atomic 0.20.0 53 | Joep Meindertsma 54 | Create, share, fetch and model linked atomic data! 55 | 56 | USAGE: 57 | atomic-cli [SUBCOMMAND] 58 | 59 | FLAGS: 60 | -h, --help Prints help information 61 | -V, --version Prints version information 62 | 63 | SUBCOMMANDS: 64 | destroy Permanently removes a Resource. Uses Commits. 65 | edit Edit a single Atom from a Resource using your text editor. Uses Commits. 66 | get Traverses a Path and prints the resulting Resource or Value. 67 | help Prints this message or the help of the given subcommand(s) 68 | list List all bookmarks 69 | new Create a Resource 70 | remove Remove a single Atom from a Resource. Uses Commits. 71 | set Update an Atom's value. Uses Commits. 72 | tpf Finds Atoms using Triple Pattern Fragments. 73 | 74 | Visit https://github.com/atomicdata-dev/atomic-data-browser for more info 75 | ``` 76 | 77 | [repository + issue tracker](https://github.com/atomicdata-dev/atomic-data-browser). 78 | 79 | 80 | ### Raycast extension: Full-text search from your desktop 81 | 82 | [Install here](https://www.raycast.com/atomicdata-dev/atomic-data-browser). 83 | 84 | ## Libraries 85 | 86 | ### `@tomic/lib` and `@tomic/react` 87 | 88 | Javascript / typescript libraries, especially useful for creating front-end apps. 89 | 90 | Fork the [atomic-data-react-template](https://codesandbox.io/s/atomic-data-react-template-4y9qu?file=/src/MyResource.tsx) on codesandbox to get started directly! 91 | 92 | ### `atomic-lib` (Rust) 93 | 94 | Library that powers `atomic-server` and `atomic-cli`. Features: 95 | 96 | - An in-memory store 97 | - Parsing (JSON-AD) / Serialization (JSON-AD, JSON-LD, TTL, N-Triples) 98 | - Commit validation and processing 99 | - Constructing Collections 100 | - Path traversal 101 | - Basic validation 102 | 103 | [repository + issue tracker](https://github.com/atomicdata-dev/atomic-data-browser). 104 | 105 | ## Want to add to this list? Some ideas for tooling 106 | 107 | This document contains a set of ideas that would help achieve that success. 108 | Open a PR and [edit this file](https://github.com/ontola/atomic-data-docs/edit/master/src/tooling.md) to add your project! 109 | 110 | ### Atomic Companion 111 | 112 | A mobile app for granting permissions to your data and signing things. See [github issue](https://github.com/ontola/atomic-data-docs/issues/45). 113 | 114 | - Show a notification when you try to log in somewhere with your agent 115 | - Notifications for mentions and other social items 116 | - Check uptime of your server 117 | 118 | ### Atomizer (data importer and conversion kit) 119 | 120 | - Import data from some data source (CSV / SQL / JSON / RDF), fill in the gaps (mapping / IRI creation / datatypes) an create new Atoms 121 | - Perhaps a CLI, library, GUI or a combination of all of these 122 | 123 | ### Atomic Preview 124 | 125 | - A simple (JS) widget that can be embedded anywhere, which converts an Atomic Graph into an HTML view. 126 | - Would be useful for documentation, and as a default view for Atomic Data. 127 | - Use `@tomic/react` and `@tomic/lib` to get started 128 | 129 | ### Atomic-Dart + Flutter 130 | 131 | Library + front-end app for browsing / manipulating Atomic Data on mobile devices. 132 | -------------------------------------------------------------------------------- /src/core/json-ad.md: -------------------------------------------------------------------------------- 1 | {{#title JSON-AD: The Atomic Data serialization format}} 2 | # JSON-AD: The Atomic Data serialization format 3 | 4 | Although you can use various serialization formats for Atomic Data, `JSON-AD` is the _default_ and _only required_ serialization format. 5 | It is what the current [Rust](https://github.com/atomicdata-dev/atomic-data-browser) and [Typescript / React](https://github.com/atomicdata-dev/atomic-data-browser) implementations use to communicate. 6 | It is designed to feel familiar to developers and to be easy and performant to parse and serialize. 7 | It is inspired by [JSON-LD](https://json-ld.org/). 8 | 9 | It uses [JSON](https://www.ecma-international.org/publications-and-standards/standards/ecma-404/), but has some additional constraints: 10 | 11 | - Every single Object is a `Resource`. 12 | - Every Key is a [`Property`](https://atomicdata.dev/classes/Property) URL. Other keys are invalid. Each Property URL must resolve to an online Atomic Data Property. 13 | - The `@id` field is special: it defines the `Subject` of the `Resource`. If you send an HTTP GET request there with an `content-type: application/ad+json` header, you should get the full JSON-AD resource. 14 | - JSON arrays are mapped to [Resource Arrays](https://atomicdata.dev/datatypes/resourceArray) 15 | - Numbers can be [Integers](https://atomicdata.dev/datatypes/integer), [Timestamps](https://atomicdata.dev/datatypes/timestamp) or [Floats](https://atomicdata.dev/datatypes/float). 16 | - JSON booleans map to [Booleans](https://atomicdata.dev/datatypes/boolean). 17 | - JSON strings can be many datatypes, including [String](https://atomicdata.dev/datatypes/string), [Markdown](https://atomicdata.dev/datatypes/markdown), [Date](https://atomicdata.dev/datatypes/date) or other. 18 | - Nested JSON Objects are Nested Resources. A Nested Resource can either be _Anonymous_ (without an `@id` subject) or a Named Nested Resource (with an `@id` subject). Everywhere a Subject URL can be used as a value (i.e. all properties with the datatype [atomicURL](https://atomicdata.dev/datatypes/atomicURL)), a Nested Resource can be used instead. This also means that an item in an `ResourceArray` can be a Nested Resource. 19 | - The root data structure must either be a Named Resource (with an `@id`), or an Array containing Named Resources. When you want to describe multiple Resources in one JSON-AD document, use an array as the root item. 20 | 21 | Let's look at an example JSON-AD Resource: 22 | 23 | ```json 24 | { 25 | "@id": "https://atomicdata.dev/properties/description", 26 | "https://atomicdata.dev/properties/datatype": "https://atomicdata.dev/datatypes/markdown", 27 | "https://atomicdata.dev/properties/description": "A textual description of something. When making a description, make sure that the first few words tell the most important part. Give examples. Since the text supports markdown, you're free to use links and more.", 28 | "https://atomicdata.dev/properties/isA": [ 29 | "https://atomicdata.dev/classes/Property" 30 | ], 31 | "https://atomicdata.dev/properties/shortname": "description" 32 | } 33 | ``` 34 | 35 | The mime type (for HTTP content negotiation) is `application/ad+json` ([registration ongoing](https://github.com/ontola/atomic-data-docs/issues/60)). 36 | 37 | ## Nested, Anonymous and Named resources 38 | 39 | In JSON-AD, a Resource can be respresented in multiple ways: 40 | 41 | - **Subject**: A URL string, such as `https://atomicdata.dev/classes/Class`. 42 | - **Named Resource**: A JSON Object with an `@id` field containing the Subject. 43 | - **Anonymous Nested Resource** A JSON Object without an `@id` field. This is only possible if it is a Nested Resource, which means that it has a parent Resource. 44 | 45 | Note that this is also valid for `ResourceArrays`, which usually only contain Subjects, but are allowed to contain Nested Resources. 46 | 47 | In the following JSON-AD example, the `address` is a nested resource: 48 | 49 | ```json 50 | { 51 | "@id": "https://example.com/arnold", 52 | "https://example.com/properties/address": { 53 | "https://example.com/properties/firstLine": "Longstreet 22", 54 | "https://example.com/properties/city": "Watertown", 55 | "https://example.com/properties/country": "the Netherlands", 56 | } 57 | } 58 | ``` 59 | 60 | Nested Resources can be _named_ or _anonymous_. An _Anonymous Nested Resource_ does not have it's own `@id` field. 61 | It _does_ have its own unique [path](./paths.md), which can be used as its identifier. 62 | The `path` of the anonymous resource in the example above is `https://example.com/arnold https://example.com/properties/address`. 63 | 64 | ## JSON-AD Parsers, serializers and other libraries 65 | 66 | - **Typescript / Javacript**: [@tomic/lib](https://www.npmjs.com/package/@tomic/lib) JSON-AD parser + in-memory store. Works with [@tomic/react](https://www.npmjs.com/package/@tomic/lib) for rendering Atomic Data in React. 67 | - **Rust**: [atomic_lib](https://crates.io/crates/atomic_lib) has a JSON-AD parser / serializer (and does a lot more). 68 | 69 | ## Canonicalized JSON-AD 70 | 71 | When you need deterministic serialization of Atomic Data (e.g. when calculating a cryptographic hash or signature, used in Atomic Commits), you can use the following procedure: 72 | 73 | 1. Serialize your Resource to JSON-AD 74 | 1. Do not include empty objects, empty arrays or null values. 75 | 1. All keys are sorted alphabetically (lexicographically) - both in the root object, as in any nested objects. 76 | 1. The JSON-AD is minified: no newlines, no spaces. 77 | 78 | The last two steps of this process is more formally defined by the JSON Canonicalization Scheme (JCS, [rfc8785](https://tools.ietf.org/html/rfc8785)). 79 | 80 | ## Interoperability with JSON and JSON-LD 81 | 82 | [Read more about this subject](../interoperability/json.md). 83 | -------------------------------------------------------------------------------- /src/core/validations.md: -------------------------------------------------------------------------------- 1 | # Atomic Graph Validations 2 | 3 | An Graph is a set of Atoms. 4 | Since Atomic Data is designed to facilitate decentralized data storage, Graphs will often lack information or contain invalid data. 5 | In this section, we define some of these concepts. 6 | 7 | - A **Valid Graph** contains no mismatches between Datatypes from Properties and their usage in Atoms 8 | - A **Closed Graph** contains no unfetched outgoing links 9 | - A **Verified Graph** contains only Atoms from verified Authors 10 | - A **Schema Complete Graph** contains all used linked Properties 11 | - A **Frozen Graph** contains content-addressing identifiers (e.g. IPFS), all the way down 12 | 13 | These concepts are important when creating an implementation of a Store. 14 | 15 | ## Valid Graphs 16 | 17 | We refer to a Graph as Valid, if the following constraints are met: 18 | 19 | - **The Datatypes are correctly used**. The Graph does not contain Atoms where the Datatype of the Value does not match the Datatype of the Property of the Atom. 20 | - **The links work**. All URLs used in the Graph (Subject, Property, Value) resolve correctly to the required Datatype. 21 | - **The Class Restrictions are met**. If a Class sets required properties, these must be present in Resources that are instances of that Class. 22 | 23 | Making sure Graphs are Valid is of great importance to anyone creating, sharing or using Atomic Data. 24 | Services should specify whether they check the validity of graphs. 25 | 26 | ## Closed Graphs 27 | 28 | A Graph is Closed, when the Resources of all URLs are present in the Graph. 29 | In other words, if you were to fetch and download every single URL in a Graph, you would not have any more Atoms than before. 30 | There are no more unfetched outgoing links. 31 | 32 | Closed Graphs are _rarely_ required in Atomic Data; it's often perfectly fine to have outgoing links that do not have been fetched. 33 | 34 | ## Verified Graphs 35 | 36 | When you are given some Atomic Graph by someone, you initially don't know for sure whether the Atoms themselves are actually created by the one controlling the subject URL. 37 | Someone may have tempered with the data, or fabricated it. 38 | 39 | The process of Verification can be done in two ways: 40 | 41 | 1. **Request the subjects, and check if the atoms match**. 42 | 1. **Verify the signatures of the Resources or Commits** 43 | 44 | When one of these steps is taken, we say that the Graph is Verified. 45 | 46 | ## Schema Complete Graphs 47 | 48 | When a Graph has a set of Atoms, it might not possess all the information that is required to determine the datatype of each Atom. 49 | When that is the case, we say the Graph is _Schema Complete_. 50 | 51 | Having a Schema Complete Graph is essential for determining what the Datatype is of a Value. 52 | Most implementations of Atomic Data will need Schema Completeness to create fitting views, or apply functional business logic. 53 | 54 | Imagine some application (perhaps an app running inside a web-browser) that has only the following data: 55 | 56 | ```ndjson 57 | ["https://example.com/john","https://example.com/birthDate","1991-01-20"] 58 | ``` 59 | 60 | Now, by looking at this single Atom, we might assume that the Value is an ISO date, 61 | but this type information is not known yet to the application. 62 | This type information should be specified in the `example:birthDate` Property. 63 | It is the responsibility of the application to make sure it possess the required Schema data. 64 | 65 | We say a Graph is _Schema Complete_ when it contains _at least_ all the Property Classes that are used in the Property fields. 66 | 67 | So let's add the missing Property: `https://example.com/birthDate` 68 | 69 | ```ndjson 70 | ["https://example.com/john","https://example.com/birthDate","1991-01-20"] 71 | ["https://example.com/birthDate","https://atomicdata.dev/datatypes/Datatype","https://atomicdata.dev/datatypes/dateTime"] 72 | ``` 73 | 74 | Now, since we've introduced yet another Property, we need to include that one as well: 75 | 76 | ```ndjson 77 | ["https://example.com/john","https://example.com/birthDate","1991-01-20"] 78 | ["https://example.com/birthDate","https://atomicdata.dev/datatypes/Datatype","https://atomicdata.dev/datatypes/dateTime"] 79 | ["https://atomicdata.dev/datatypes/Datatype","https://atomicdata.dev/datatypes/Datatype","https://atomicdata.dev/datatypes/atomicURI"] 80 | ``` 81 | 82 | Since all valid Atomic Data requires Property fields to resolve to Atomic Properties Classes, which are required to have an associated DataType... 83 | We can safely say that the last atom in the example above (the one describing `https://atomicdata.dev/datatypes/Datatype`) will have to be pre 84 | sent in all Schema Complete Atomic Graphs. 85 | 86 | ## Frozen Graphs 87 | 88 | A Frozen Graph consists only of resources with content-addressing identifiers as Subjects. 89 | A [content-addressable](https://flyingzumwalt.gitbooks.io/decentralized-web-primer/avenues-for-access/lessons/power-of-content-addressing.html) URL (such as an IPFS URL) refers to specific immutable content, that is absolutely certain not to change over time. 90 | Due to its static nature, we call it _Frozen_. 91 | As long as a graph contains links to HTTP Resources, it is not Frozen, since responses from that HTTP address might change over time. 92 | 93 | Freezing a Graph, therefore, entails converting all resources to IFPS (or another content-addressable schema) Resources, and using only IPFS URLs. 94 | 95 | Freezing a Graph has performance benefits for clients, since clients can easily verify if they already have (part of) the Graph locally, simply by comparing the URLs or Resources. 96 | It also helps to make sure the content can be shared peer to peer 97 | 98 | Note that **Graphs with cyclical relations cannot be frozen**, since every iteration that you'd try to freeze will change its references and therefore also its contents, and therefore also its content hash. 99 | -------------------------------------------------------------------------------- /src/interoperability/json.md: -------------------------------------------------------------------------------- 1 | {{#title How does Atomic Data relate to JSON?}} 2 | # How does Atomic Data relate to JSON? 3 | 4 | Because JSON is so popular, Atomic Data is designed with JSON in mind. 5 | 6 | Atomic Data is often (by default) serialized to [JSON-AD](../core/json-ad.md), which itself uses JSON. 7 | JSON-AD uses URLs as keys, which is what gives Atomic Data many of its perks, but using these long strings as keys is not very easy to use in many contexts. 8 | That's why you can serialize Atomic Data to simple, clean JSON. 9 | 10 | ## From Atomic Data to plain JSON 11 | 12 | The JSON keys are then derived from the `shortnames` of properties. 13 | For example, we could convert this JSON-AD: 14 | 15 | ```json 16 | { 17 | "@id": "https://atomicdata.dev/properties/description", 18 | "https://atomicdata.dev/properties/datatype": "https://atomicdata.dev/datatypes/markdown", 19 | "https://atomicdata.dev/properties/description": "A textual description of something. When making a description, make sure that the first few words tell the most important part. Give examples. Since the text supports markdown, you're free to use links and more.", 20 | "https://atomicdata.dev/properties/isA": [ 21 | "https://atomicdata.dev/classes/Property" 22 | ], 23 | "https://atomicdata.dev/properties/shortname": "description" 24 | } 25 | ``` 26 | 27 | ... into this plain JSON: 28 | 29 | ```json 30 | { 31 | "@id": "https://atomicdata.dev/properties/description", 32 | "datatype": "https://atomicdata.dev/datatypes/markdown", 33 | "description": "A textual description of something. When making a description, make sure that the first few words tell the most important part. Give examples. Since the text supports markdown, you're free to use links and more.", 34 | "is-a": [ 35 | "https://atomicdata.dev/classes/Property" 36 | ], 37 | "shortname": "description" 38 | } 39 | ``` 40 | 41 | Note that when you serialize Atomic Data to plain JSON, some information is lost: the URLs are no longer there. 42 | This means that it is no longer possible to find out what the datatype of a single value is - we now only know if it's a `string`, but not if it actually represents a markdown string or something else. 43 | Most Atomic Data systems will therefore _not_ use this plain JSON serialization, but for some clients (e.g. a front-end app), it might be easier to use the plain JSON, as the keys are easier to write than the long URLs that JSON-AD uses. 44 | 45 | ## From JSON to JSON-AD 46 | 47 | Atomic Data requires a bit more information about pieces of data than JSON tends to contain. Let's take a look at a regular JSON example: 48 | 49 | ```json 50 | { 51 | "name": "John", 52 | "birthDate": "1991-01-20" 53 | } 54 | ``` 55 | 56 | We need more information to convert this JSON into Atomic Data. 57 | The following things are missing: 58 | 59 | * What is the **Subject** URL of the resource being described? 60 | * What is the **Property** URL of the keys being used? (`name` and `birthDate`), and consequentially, how should the values be parsed? What are their DataTypes? 61 | 62 | In order to make this conversion work, we need to link to three URLs that _resolve to atomic data resources_. 63 | The `@id` subject should resolve to the Resource itself, returning the JSON-AD from below. 64 | The Property keys (e.g. "https://example.com/properties/name") need to resolve to Atomic Properties. 65 | 66 | ```json 67 | { 68 | "@id": "https://example.com/people/john", 69 | "https://example.com/properties/name": "John", 70 | "https://example.com/properties/birthDate": "1991-01-20" 71 | } 72 | ``` 73 | 74 | In practice, the easiest approach to make this conversion, is to create the data and host it using software like [Atomic Server](https://github.com/atomicdata-dev/atomic-server/blob/master/server/README.md). 75 | 76 | ## From Atomic Data to JSON-LD 77 | 78 | Atomic Data is a strict subset of RDF, and the most popular serialization of RDF for JSON data is [JSON-LD](https://json-ld.org/). 79 | 80 | Since Atomic Schema requires the presence of a `key` slug in Properties, converting Atomic Data to JSON results in dev-friendly objects with nice shorthands. 81 | 82 | ```json 83 | { 84 | "@id": "https://example.com/people/John", 85 | "https://example.com/properties/lastname": "John", 86 | "https://example.com/properties/bestFriend": "https://example.com/sarah", 87 | } 88 | ``` 89 | 90 | Can be automatically converted to: 91 | 92 | ```json 93 | { 94 | "@context": { 95 | "@id": "https://example.com/people/John", 96 | "name": "https://example.com/properties/lastname", 97 | "bestFriend": "https://example.com/properties/bestFriend", 98 | }, 99 | "name": "John", 100 | "bestFriend": { 101 | "@id": "https://example.com/sarah" 102 | }, 103 | } 104 | ``` 105 | 106 | The `@context` object provides a _mapping_ to the original URLs. 107 | 108 | JSON-AD and JSON-LD are very similar by design, but there are some important differences: 109 | 110 | - JSON-AD is designed just for atomic data, and is therefore easier and more performant to parse / serialize. 111 | - JSON-LD uses `@context` to map keys to URLs. Any type of mapping is valid. JSON-AD, on the other hand, doesn't map anything - all keys are URLs. 112 | - JSON-LD uses nested objects for links and sequences, such as `@list`. JSON-AD does not. 113 | - Arrays in JSON-LD do not indicate ordered data - they indicate that for some subject-predicate combination, multiple values exist. This is a result of how RDF works. 114 | 115 | ## JSON-LD Requirements for valid Atomic Data 116 | 117 | - Make sure the URLs used in the `@context` resolve to Atomic Properties. 118 | 119 | - Convert JSON-LD arrays into ResourceArrays 120 | - Creating nested JSON objects is possible (by resolving the identifiers from `@id` relations), but it is up to the serializer to decide how deep this object nesting should happen. 121 | 122 | Note that as of now, there are no JSON-LD parsers for Atomic Data. 123 | -------------------------------------------------------------------------------- /src/schema/classes.md: -------------------------------------------------------------------------------- 1 | {{#title Atomic Data Classes}} 2 | # Atomic Schema: Classes 3 | 4 | The following Classes are some of the most fundamental concepts in Atomic Data, as they make data validation possible. 5 | 6 | Click the URLs of the classes to read the most actual data, and discover their properties! 7 | 8 | ## Property 9 | 10 | _URL: [`https://atomicdata.dev/classes/Property`](https://atomicdata.dev/classes/Property)_ 11 | 12 | The Property class. 13 | The thing that the Property field should link to. 14 | A Property is an abstract type of Resource that describes the relation between a Subject and a Value. 15 | A Property provides some semantic information about the relationship (in its `description`), it provides a shorthand (the `shortname`) and it links to a Datatype. 16 | 17 | Properties of a Property instance: 18 | 19 | - [`shortname`](https://atomicdata.dev/properties/shortname) - (required, Slug) the shortname for the property, used in ORM-style dot syntax (`thing.property.anotherproperty`). 20 | - [`description`](https://atomicdata.dev/properties/description) - (optional, AtomicURL, TranslationBox) the semantic meaning of the. 21 | - [`datatype`](https://atomicdata.dev/properties/datatype) - (required, AtomicURL, Datatype) a URL to an Atomic Datatype, which defines what the datatype should be of the Value in an Atom where the Property is the 22 | - [`classtype`](https://atomicdata.dev/properties/classtype) - (optional, AtomicURL, Class) if the `datatype` is an Atomic URL, the `classtype` defines which class(es?) is (are?) acceptable. 23 | 24 | ```json 25 | { 26 | "@id": "https://atomicdata.dev/properties/description", 27 | "https://atomicdata.dev/properties/datatype": "https://atomicdata.dev/datatypes/markdown", 28 | "https://atomicdata.dev/properties/description": "A textual description of something. When making a description, make sure that the first few words tell the most important part. Give examples. Since the text supports markdown, you're free to use links and more.", 29 | "https://atomicdata.dev/properties/isA": [ 30 | "https://atomicdata.dev/classes/Property" 31 | ], 32 | "https://atomicdata.dev/properties/shortname": "description" 33 | } 34 | ``` 35 | 36 | Visit [https://atomicdata.dev/properties](https://atomicdata.dev/properties) for a list of example Properties. 37 | 38 | ## Datatype 39 | 40 | _URL: [`https://atomicdata.dev/classes/Datatype`](https://atomicdata.dev/classes/Datatype)_ 41 | 42 | A Datatype specifies how a `Value` value should be interpreted. 43 | Datatypes are concepts such as `boolean`, `string`, `integer`. 44 | Since DataTypes can be linked to, you dan define your own. 45 | However, using non-standard datatypes limits how many applications will know what to do with the data. 46 | 47 | Properties: 48 | 49 | - `description` - (required, AtomicURL, TranslationBox) how the datatype functions. 50 | - `stringSerialization` - (required, AtomicURL, TranslationBox) how the datatype should be parsed / serialized as an UTF-8 string 51 | - `stringExample` - (required, string) an example `stringSerialization` that should be parsed correctly 52 | - `binarySerialization` - (optional, AtomicURL, TranslationBox) how the datatype should be parsed / serialized as a byte array. 53 | - `binaryExample` - (optional, string) an example `binarySerialization` that should be parsed correctly. Should have the same contents as the stringExample. Required if binarySerialization is present on the DataType. 54 | 55 | Visit [https://atomicdata.dev/datatypes](https://atomicdata.dev/datatypes) for a list of example Datatypes. 56 | 57 | ## Class 58 | 59 | _URL: [`https://atomicdata.dev/classes/Class`](https://atomicdata.dev/classes/Class)_ 60 | 61 | A Class is an abstract type of Resource, such as `Person`. 62 | It is convention to use an Uppercase in its URI. 63 | Note that in Atomic Data, a Resource can have several Classes - not just a single one. 64 | If you need to set more complex constraints to your Classes (e.g. maximum string length, Properties that depend on each other), check out [SHACL](https://www.w3.org/TR/shacl/). 65 | 66 | Properties: 67 | 68 | - `shortname` - (required, Slug) a short string shorthand. 69 | - `description` - (required, AtomicURL, TranslationBox) human readable explanation of what the Class represents. 70 | - `requires` - (optional, ResourceArray, Property) a list of Properties that are required. If absent, none are required. These SHOULD have unique shortnames. 71 | - `recommends` - (optional, ResourceArray, Property) a list of Properties that are recommended. These SHOULD have unique shortnames. 72 | 73 | 74 | 75 | 76 | 77 | 78 | A resource indicates it is an _instance_ of that class by adding a `https://atomicdata.dev/properties/isA` Atom. 79 | 80 | Example: 81 | 82 | ```json 83 | { 84 | "@id": "https://atomicdata.dev/classes/Class", 85 | "https://atomicdata.dev/properties/description": "A Class describes an abstract concept, such as 'Person' or 'Blogpost'. It describes the data shape of data and explains what the thing represents. It is convention to use Uppercase in its URL. Note that in Atomic Data, a Resource can have several Classes - not just a single one.", 86 | "https://atomicdata.dev/properties/isA": [ 87 | "https://atomicdata.dev/classes/Class" 88 | ], 89 | "https://atomicdata.dev/properties/recommends": [ 90 | "https://atomicdata.dev/properties/recommends", 91 | "https://atomicdata.dev/properties/requires" 92 | ], 93 | "https://atomicdata.dev/properties/requires": [ 94 | "https://atomicdata.dev/properties/shortname", 95 | "https://atomicdata.dev/properties/description" 96 | ], 97 | "https://atomicdata.dev/properties/shortname": "class" 98 | } 99 | ``` 100 | 101 | Check out a [list of example Classes](https://atomicdata.dev/classes/). 102 | -------------------------------------------------------------------------------- /src/interoperability/sql.md: -------------------------------------------------------------------------------- 1 | {{#title How does Atomic Data relate to SQL?}} 2 | # Atomic Data and SQL 3 | 4 | Atomic Data has some characteristics that make it similar and different from SQL. 5 | 6 | - Atomic Data has a _dynamic_ schema. Any Resource could have different properties, so you can **add new properties** to your data without performing any migrations. However, the properties themselves are still validated (contrary to most NoSQL solutions) 7 | - Atomic Data uses **HTTP URLs** in its data, which means it's easy to **share and reuse**. 8 | - Atomic Data separates _reading_ and _writing_, whereas SQL has one language for both. 9 | - Atomic Data has a standardized way of **storing changes** ([Commits](../commits/intro.md)) 10 | 11 | ## Tables and Rows vs. Classes and Properties 12 | 13 | At its core, SQL is a query language based around _tables_ and _rows_. 14 | The _tables_ in SQL are similar to `Classes` in Atomic Data: they both define a set of `properties` which an item could have. 15 | Every single item in a table is called a _row_ in SQL, and a `Resource` in Atomic Data. 16 | One difference is that in Atomic Data, you can add new properties to resources, without making changes to any tables (migrations). 17 | 18 | ## Dynamic vs static schema 19 | 20 | In SQL, the schema of the database defines which shape the data can have, which properties are required, what datatypes they have. 21 | In Atomic Data, the schema exists as a Resource on the web, which means that they can be retrieved using HTTP. 22 | An Atomic Database (such as [Atomic-Server](https://crates.io/crates/atomic-server)) uses a _dynamic schema_, 23 | which means that any Resource can have different properties, and the properties themselves can be validated, even when the server is not aware of these properties beforehand. 24 | In SQL, you'd have to manually adjust the schema of your database to add a new property. 25 | Atomic Data is a decentralized, open system, which can read new schema data from other sources. 26 | SQL is a centralized, closed system, which relies on the DB manager to define the schema. 27 | 28 | ## Identifiers: numbers vs. URLs 29 | 30 | In SQL, rows have numbers as identifiers, whereas in Atomic Data, every resource has a resolvable HTTP URL as an identifier. 31 | URLs are great identifiers, because you can open them and get more information about something. 32 | This means that with Atomic Data, other systems can re-use your data by referencing to it, and you can re-use data from other systems, too. 33 | With Atomic Data, you're making your data part of a bigger _web of data_, which opens up a lot of possibilities. 34 | 35 | ## Atomic Server combines server and database 36 | 37 | If you're building an App with SQL, you will always need some server that connects to your database. 38 | If you're building an App with Atomic Server, the database can function as your server, too. It deals with authentication, authorization, and more. 39 | 40 | ## Querying 41 | 42 | The SQL query language is for both _reading_ and _writing_ data. 43 | In Atomic Data a distinction is made between Query and Command - getting and setting (Command Query Responsibility Segregation, [CQRS](https://martinfowler.com/bliki/CQRS.html)). 44 | The [Query side](../core/querying.md) is handled using Subject Fetching (sending a GET request to a URL, to get a single resource) and [Collections](../schema/collections.md) (filtering and sorting data). 45 | The Command side is typically done using [Atomic Commits](../commits/intro.md), although you're free not to use it. 46 | 47 | SQL is way more powerful, as a query language. 48 | In SQL, the one creating the query basically defines the shape of a table that is requested, and the database returns that shape. 49 | Atomic Data does not offer such functionality. 50 | So if you need to create custom tables at runtime, you might be better off using SQL, or move your Atomic Data to a query system. 51 | 52 | ## Convert an SQL database to Atomic Data 53 | 54 | If you want to make your existing SQL project serve Atomic Data, you can keep your existing SQL database, see [the upgrade guide](upgrade.md). 55 | It basically boils down to mapping the rows (properties) in your SQL tables to Atomic Data [Properties](https://atomicdata.dev/classes/Property). 56 | 57 | When you want to _import arbitrary Atomic Data_, though, it might be easier to use `atomic-server`. 58 | If you want to store arbitrary Atomic Data in a SQL database, you might be best off by creating a `Resources` table with a `subject` and a `propertyValues` column, or create both a `properties` table and a `resources` one. 59 | 60 | ## Limitations of Atomic Data 61 | 62 | - SQL is far more common, many people will know how to use it. 63 | - SQL databases are battle-tested and has been powering countless of products for tens of years, whereas Atomic Server is at this moment in beta. 64 | - SQL databases have a more powerful and expressive query language, where you can define tables in your query and combine resources. 65 | - Atomic Data doesn't have a [mutli-node / distributed option](https://github.com/atomicdata-dev/atomic-server/issues/213) 66 | 67 | ## FAQ 68 | 69 | ### Is Atomic Data NOSQL or SQL? 70 | 71 | Generally, Atomic Data apps do not use SQL - so they are NOSQL. 72 | Atomic-server, for example, internally uses a key-value store (sled) for persistence. 73 | 74 | Like most NOSQL systems, Atomic Data does not limit data entries to a specific table shape, so you can add any property that you like to a resource. 75 | However, unlike most NOSQL systems, Atomic Data _does_ perform validations on each value. 76 | So in a way, Atomic Data tries to combine best of both worlds: the extendibility and flexibility of NOSQL, with the type safety of SQL. 77 | 78 | ### Is Atomic Data transactional / ACID? 79 | 80 | Yes, if you use Atomic-Server, then you can only write to the server by using Atomic Commits, which are in fact transactions. 81 | This means that if part of the transaction fails, it is reverted - transactions are only applied when they are 100% OK. 82 | This prevents inconsistent DB states. 83 | 84 | ### How does Atomic Server build indexes for its resources if the schema is not known in advance 85 | 86 | It creates indexed collections when users perform queries. 87 | This means that the first time your perform some type of query (that sorts and filters by some properties), it will be slow, but the next time you perform a similar query, it will be fast. 88 | -------------------------------------------------------------------------------- /src/usecases/e-commerce.md: -------------------------------------------------------------------------------- 1 | {{#title Atomic Data for e-commerce & marketplaces}} 2 | # Atomic Data for e-commerce & marketplaces 3 | 4 | Buying good and services on the internet is currently responsible for about 15% of all commerce, and is steadily climbing. 5 | The internet makes it easier to find products, compare prices, get information and reviews, and finally order something. 6 | But the current e-commerce situation is far from perfect, as large corporations tend to monopolize, which means that we have less competition which ultimately harms prices and quality for consumers. 7 | Atomic Data can help empower smaller businesses, make searching for specific things way easier and ultimately make things cheaper for everyone. 8 | 9 | ## Decentralize platform / sharing economy service marketplaces 10 | 11 | Platforms like Uber, AirBNB and SnapCar are virtual marketplaces that help people share and find services. 12 | These platforms are responsible for: 13 | 14 | 1. providing an interface for **managing offers** (e.g. describe your car, add specifications and pricing) 15 | 2. **hosting** the data of the offers themselves (make the data available on the internet) 16 | 3. providing a **search interface** (which means indexing the data from all the existing offers) 17 | 4. facilitating the **transaction** / payments 18 | 5. provide **trust** through reviews and warranties (e.g. refunds if the seller fails to deliver) 19 | 20 | The fact that these responsibilities are almost always combined in a single platforms leads to vendor lock-in and an uncompetitive landscape, which ultimately harms consumers. 21 | Currently, if you want to manage your listing / offer on various platforms, you need to manually adjust it on all these various platforms. 22 | Some companies even prohibit offering on multiple platforms (which is a legal problem, not a technical one). 23 | This means that the biggest (most known) platforms have the most listings, so if you're looking for a house / car / rental / meal, you're likely to go for the biggest business - because that's the one that has the biggest assortment. 24 | 25 | Compare this to how the web works: every browser should support every type of webpage, and it does not matter where the webpage is hosted. 26 | I can browse a webpage written on a mac on my windows machine, and I can read a webpage hosted by amazon on an google device. 27 | It does not matter, because the web is _standardized_ and _open_, instead of being _centralized_ and managed by one single company as _proprietary_ data. 28 | This openness of the web means that we get search engines like Google and Bing that _scrape_ the web and add it to their index. 29 | This results in a dynamic where those who want to sell their stuff will need to share their stuff using an open standard (for webpages things like HTML and sometimes a bit of metadata), so crawlers can properly index the webpages. 30 | We could do the same thing for _structured data_ instead of _pages_, and that's what Atomic Data is all about. 31 | 32 | Let's discuss a more practical example of what this could mean. 33 | Consider a restaurant owner who currently uses UberEats as their delivery platform. 34 | Using Atomic Data, they could define their menu on their own website. 35 | The Atomic Schema specification makes it easy to standardize how the data of a menu item looks like (e.g. price, image, title, allergens, vegan...). 36 | Several platforms (potentially modern variants of platforms like JustEat / UberEats) could then crawl this standardized Atomic Data, index it, and make it easily searchable. 37 | The customer would use one (or multiple) of these platforms, that would probably have the _exact same_ offers. 38 | Where these platforms might differ, is in their own service offering, such as delivery speed or price. 39 | This would result in a more competitive and free market, where customers would be able to pick a platform based on their service price and quality, instead of their list of offerings. 40 | It would empower the small business owner to be far more flexible in which service they will do business with. 41 | 42 | ## Highly personalized and customizable search 43 | 44 | Searching for products on the internet is mostly limited to text search. 45 | If we want to buy a jacket, we see tonnes of jackets that are not even available in our own size. 46 | Every single website has their own way of searching and filtering. 47 | 48 | Imagine making a search query in _one_ application, and sending that to _multiple suppliers_, after you'll receive a fully personalized and optimized list of products. 49 | Browsing in an application that you like to use, not bound to any one specific store, that doesn't track you, and doesn't show advertisements. 50 | It is a tool that helps you to find what you need, and it is the job of producers to accurately describe their products in a format that your product browser can understand. 51 | 52 | How do we get there? 53 | 54 | Well, for starters, producers and suppliers will need to reach a consensus on _how to describe their articles_. 55 | This is not new; for many products, we already have a common language. 56 | Shoes have a shoe size, televisions have a screen size in diagonal inches, brightness is measured in nits, etc. 57 | Describing this in a machine-readable and predictable format as data is the next logical step. 58 | This is, of course, where Atomic Schema could help. 59 | Atomic-server could be the connected, open source database that suppliers use to describe their products as data. 60 | 61 | Then we'll also need to build a search interface that performs federated queries, and product-dependent filter options. 62 | 63 | ## Product lifecycle & supply chain insights 64 | 65 | Imagine buying a product, and being able to see where each part came from. 66 | The car that you buy might contain a list of all the maintenance moments, and every replaced part. 67 | The raw materials used could be traced back to their origins. 68 | 69 | This requires a high degree of coordination from each step in the supply chain. 70 | This is exactly where Atomic Data shines, though, as it provides a highly standardized way of structuring, querying, authenticating an authorizing data. 71 | 72 | Before we get to this point, we'll need to: 73 | 74 | - Describe domain-specific product Classes using Atomic Schema, and their Properties. 75 | 76 | ## Product specific updates after purchase 77 | 78 | Imagine buying an external battery pack with a production error. 79 | All units with a serial number between 1561168 and 1561468 have a serious error, where overcharging could lead to spontaneous combustion. 80 | This is something that you'd like to know. 81 | But how would the _manufacturer_ of that resource know where to find you? 82 | Well, if your Atomic Server would have a list of all the things that you've bought, it could _automatically_ subscribe to safety updates from all manufacturers. 83 | When any of these manufacturers would publish a safety warning about a product that you possess, you'll get an alert. 84 | 85 | Before we have this, well need to: 86 | 87 | - Build notifications support (see [issue](https://github.com/atomicdata-dev/atomic-server/issues/77)) 88 | -------------------------------------------------------------------------------- /src/create-json-ad.md: -------------------------------------------------------------------------------- 1 | # How to create and publish a JSON-AD file 2 | 3 | [JSON-AD](core/json-ad.md) is the default serialization format of Atomic Data. 4 | It's just JSON, but with some extra requirements. 5 | 6 | Most notably, all keys are links to [Atomic Properties](https://atomicdata.dev/classes/Property). 7 | These Properties must be actually hosted somewhere on the web, so other people can visit them to read more about them. 8 | 9 | Ideally, in JSON-AD, each Resource has its own `@id`. 10 | This is the URL of the resource. 11 | This means that if someone visits that `@id`, they should get the resource they are requesting. 12 | That's great for people re-using your data, but as a data provider, implementing this can be a bit of a hassle. 13 | That's why there is a different way that allows you to create Atomic Data _without manually hosting every resource_. 14 | 15 | ## Creating JSON-AD without hosting individual resources yourself 16 | 17 | In this section, we'll create a single JSON-AD file containing various resources. 18 | This file can then be published, shared and stored like any other. 19 | 20 | The goal of this preparation, is to ultimately import it somewhere else. 21 | We'll be importing it to Atomic-Server. 22 | Atomic-Server will create URLs for every single resource upon importing it. 23 | This way, we only deal with the JSON-AD and the data structure, and we let Atomic-Server take care of hosting the data. 24 | 25 | Let's create a BlogPost. 26 | We know the fields that we need: a `name` and some `body`. 27 | But we can't use these keys in Atomic Data, we should use URLs that point to Properties. 28 | We can either create new Properties (see the Atomic-Server tutorial), or we can use existing ones, for example by searching on [AtomicData.dev/properties](https://atomicdata.dev/properties). 29 | 30 | ## Setting the first values 31 | 32 | ```json 33 | { 34 | "https://atomicdata.dev/properties/name": "Writing my first blogpost", 35 | "https://atomicdata.dev/properties/description": "Hi! I'm a blogpost. I'm also machine readable!", 36 | } 37 | ``` 38 | 39 | ## Adding a Class 40 | 41 | Classes help others understanding what a Resource's type is, such as BlogPost or Person. 42 | In Atomic Data, Resources can have multiple classes, so we should use an Array, like so: 43 | 44 | ```json 45 | { 46 | "https://atomicdata.dev/properties/name": "Writing my first blogpost", 47 | "https://atomicdata.dev/properties/description": "Hi! I'm a blogpost. I'm also machine readable!", 48 | "https://atomicdata.dev/properties/isA": ["https://atomicdata.dev/classes/Article"], 49 | } 50 | ``` 51 | 52 | Adding a Class helps people to understand the data, and it can provide guarantees to the data users about the _shape_ of the data: they now know which fields are _required_ or _recommended_. 53 | We can also use Classes to render Forms, which can be useful when the data should be edited later. 54 | For example, the BlogPost item 55 | 56 | ## Using exsisting Ontologies, Classes and Ontologies 57 | 58 | Ontologies are groups of concepts that describe some domain. 59 | For example, we could have an Ontology for Blogs that links to a bunch of related _Classes_, such as BlogPost and Person. 60 | Or we could have a Recipy Ontology that describes Ingredients, Steps and more. 61 | 62 | At this moment, there are relatively few Classes created in Atomic Data. 63 | You can find most on [atomicdata.dev/classes](https://atomicdata.dev/classes). 64 | 65 | So possibly the best way forward for you, is to define a Class using the Atomic Data Browser's tools for making resources. 66 | 67 | ## Multiple items 68 | 69 | If we want to have _multiple_ items, we can simply use a JSON Array at the root, like so: 70 | 71 | ```json 72 | [{ 73 | "https://atomicdata.dev/properties/name": "Writing my first blogpost", 74 | "https://atomicdata.dev/properties/description": "Hi! I'm a blogpost. I'm also machine readable!", 75 | "https://atomicdata.dev/properties/isA": ["https://atomicdata.dev/classes/Article"], 76 | },{ 77 | "https://atomicdata.dev/properties/name": "Another blogpost", 78 | "https://atomicdata.dev/properties/description": "I'm writing so much my hands hurt.", 79 | "https://atomicdata.dev/properties/isA": ["https://atomicdata.dev/classes/Article"], 80 | }] 81 | ``` 82 | 83 | ## Preventing duplication with `localId` 84 | 85 | When we want to _publish_ Atomic Data, we also want someone else to be able to _import_ it. 86 | An important thing to prevent, is _data duplication_. 87 | If you're importing a list of Blog posts, for example, you'd want to only import every article _once_. 88 | 89 | The way to preventing duplication, is by adding a `localId`. 90 | This `localId` is used by the importer to find out if it has already imported it before. 91 | So we, as data producers, need to make sure that our `localId` is _unique_ and _does not change_! 92 | We can use any type of string that we like, as long as it conforms to these requirements. 93 | Let's use a unique _slug_, a short name that is often used in URLs. 94 | 95 | ```json 96 | { 97 | "https://atomicdata.dev/properties/name": "Writing my first blogpost", 98 | "https://atomicdata.dev/properties/description": "Hi! I'm a blogpost. I'm also machine readable!", 99 | "https://atomicdata.dev/properties/isA": ["https://atomicdata.dev/classes/Article"], 100 | "https://atomicdata.dev/properties/localId": "my-first-blogpost", 101 | } 102 | ``` 103 | 104 | ## Describing relationships between resources using `localId` 105 | 106 | Let's say we also want to describe the `author` of the BlogPost, and give them an e-mail, a profile picture and some biography. 107 | This means we need to create a new Resource for each Author, and again have to think about the properties relevant for Author. 108 | We'll also need to create a link from BlogPost to Author, and perhaps the other way around, too. 109 | 110 | Normally, when we link things in Atomic Data, we can only use full URLs. 111 | But, since we don't have URLs yet for our Resources, we'll need a different solution. 112 | Again, this is where we can use `localId`! 113 | We can simply refer to the `localId`, instead of some URL that does not exist yet. 114 | 115 | ```json 116 | [{ 117 | "https://atomicdata.dev/properties/name": "Writing my first blogpost", 118 | "https://atomicdata.dev/properties/description": "Hi! I'm a blogpost. I'm also machine readable!", 119 | "https://atomicdata.dev/properties/author": "jon", 120 | "https://atomicdata.dev/properties/isA": ["https://atomicdata.dev/classes/Article"], 121 | "https://atomicdata.dev/properties/localId": "my-first-blogpost", 122 | },{ 123 | "https://atomicdata.dev/properties/name": "Another blogpost", 124 | "https://atomicdata.dev/properties/description": "I'm writing so much my hands hurt.", 125 | "https://atomicdata.dev/properties/author": "jon", 126 | "https://atomicdata.dev/properties/isA": ["https://atomicdata.dev/classes/Article"], 127 | "https://atomicdata.dev/properties/localId": "another-blogpost", 128 | },{ 129 | "https://atomicdata.dev/properties/name": "Jon Author", 130 | "https://atomicdata.dev/properties/isA": ["https://atomicdata.dev/classes/Person"], 131 | "https://atomicdata.dev/properties/localId": "jon", 132 | }] 133 | ``` 134 | 135 | ## Importing data using Atomic Sever 136 | 137 | _currently [under development](https://github.com/atomicdata-dev/atomic-server/issues/390)_ 138 | -------------------------------------------------------------------------------- /src/experimental-serialization.md: -------------------------------------------------------------------------------- 1 | # Experimental Atomic Data serialization formats 2 | 3 | These experimental formats are not supported or maintained. This document serves as an archived reference. 4 | 5 | ## AD3 (Deprecated) 6 | 7 | _AD3 is now deprecated in favor of JSON-AD_ 8 | 9 | AD3 stands for _Atomic Data Triples_, and it's the simplest and fastest way to serialize / parse Atomic Data. 10 | 11 | AD3 represents a single Atom as a single line, containing a JSON array of three strings, respectively representing the Subject, Property and Value. 12 | 13 | It looks like this: 14 | 15 | ```ad3 16 | ["https://example.com/subject","https://example.com/property","some object"] 17 | ["https://example.com/subject","https://example.com/otherProperty","https://example.com/somethingelse"] 18 | ``` 19 | 20 | It uses Newline Delimited JSON ([NDJSON](http://ndjson.org/)) for serialization, which is just a large string with newlines between each JSON object. 21 | 22 | NDJSON has some important benefits: 23 | 24 | - **Streaming parsing**: An NDJSON document can be parsed before it's fully loaded / transmitted. That is not possible with regular JSON. 25 | - **High compatibility**: NDJSON parsers can use JSON parsers, and are therefore everywhere. 26 | - **Performance**: Modern browsers have highly performant JSON parsing, which means that it's _fast_ in one of the most important contexts: the browser. 27 | 28 | _Mime type (not registered yet!): `application/ad3-ndjson`_ 29 | 30 | _File name extension: `.ad3`_ 31 | 32 | Disclaimer: note that AD3 is useful for communicating _current state_, but not for _state changes_. 33 | 34 | You can validate AD3 at [atomicdata.dev/validate](https://atomicdata.dev/validate). 35 | 36 | Atomic Triples is heavily inspired by [HexTuples-NDJSON](https://github.com/ontola/hextuples). 37 | 38 | Example serialization implementation written in Rust, to show you how _easy_ it is to serialize this! 39 | 40 | ```rust 41 | pub fn serialize_atoms_to_ad3(atoms: Vec) -> AtomicResult { 42 | let mut string = String::new(); 43 | for atom in atoms { 44 | // Use an exsting JSON serialization library to take care of the hard work (escaping quotes, etc.) 45 | let mut ad3_atom = serde_json::to_string(&vec![&atom.subject, &atom.property, &atom.value])?; 46 | ad3_atom.push_str("\n"); 47 | &string.push_str(&*ad3_atom); 48 | } 49 | return Ok(string); 50 | } 51 | ``` 52 | 53 | And an example parser: 54 | 55 | ```rust 56 | pub fn parse_ad3<'a, 'b>(string: &'b String) -> AtomicResult> { 57 | let mut atoms: Vec = Vec::new(); 58 | for line in string.lines() { 59 | match line.chars().next() { 60 | // These are comments 61 | Some('#') => {} 62 | Some(' ') => {} 63 | // That's an array, let's do this! 64 | Some('[') => { 65 | let string_vec: Vec = 66 | parse_json_array(line).expect(&*format!("Parsing error in {:?}", line)); 67 | if string_vec.len() != 3 { 68 | return Err(format!( 69 | "Wrong length of array at line {:?}: wrong length of array, should be 3", 70 | line 71 | ) 72 | .into()); 73 | } 74 | let subject = &string_vec[0]; 75 | let property = &string_vec[1]; 76 | let value = &string_vec[2]; 77 | atoms.push(Atom::new(subject, property, value)); 78 | } 79 | Some(char) => { 80 | return Err(format!( 81 | "AD3 Parsing error at {:?}, cannot start with {}", 82 | line, char 83 | ) 84 | .into()) 85 | } 86 | None => {} 87 | }; 88 | } 89 | return Ok(atoms); 90 | } 91 | ``` 92 | 93 | ## AD2 94 | 95 | AD2 (Atomic Data Doubles) is similar to AtomicTriples, with one exception: the Subject is left out. 96 | For many use-cases, omitting the Subject is a _bad idea_ - you'll most often need AD2! 97 | having no subject means that you can't describe multiple resources in a single document, and that is useful in many contexts. 98 | 99 | However, omitting the subject can be useful in (at least) three scenarios: 100 | 101 | - The **Subject is not yet known when creating the data** (for example, because it still has to be determined by some server or hash function). 102 | - The **Subject is already known by the client**, and leaving it out saves bandwidth. This happens for example during Subject Fetching, where the request itself contains the Subject, because the fetched URL itself is the Subject of all returned triples. Note that in this scenario, the server is unable to include 103 | - The **Atoms are only valid coming from a specific source**. Since 104 | 105 | ```ndjson 106 | ["https://example.com/property","some object"] 107 | ["https://example.com/otherProperty","https://example.com/somethingelse"] 108 | ``` 109 | 110 | Keep in mind that this approach also has some downsides: 111 | 112 | - It becomes impossible to include other resources in a single serialized document / response. 113 | 114 | - _Mime type (not registered yet!): `application/ad2-ndjson`_ 115 | - _File name extension: `.ad2`_ 116 | 117 | ### AtomicData-FS 118 | 119 | Possible extension: `.adf` 120 | 121 | FS stands for FileSystem. 122 | It should be designed as a format that's easy to manipulate Atomic Data by hand, using plaintext editors and IDE software. 123 | It fits nicely in our line-based paradigm, where we us IDEs and Github to manage our information. 124 | It should use Shortnames wherever possible to make life easier for those who modify instances. 125 | It might use hierarchical path structures to shape URLs. 126 | It might use hierarchical path structures to shape data, and set constraints (e.g. all items directly in the `./person` directory should be Person instances). 127 | Folder structure should reflect the structure inside URLs. 128 | 129 | Note that this format is _not_ useful for sending arbitrary Atomic Data to some client. 130 | It is useful for managing Atomic Data from a filesystem. 131 | 132 | An example AtomicData-FS dir can be [found in the repo](https://github.com/ontola/atomic-data/tree/master/examples/atomic-fs/people). 133 | 134 | ``` 135 | # in ./projectDir/people/john.adf 136 | # serialization uses YAML syntax 137 | firstName: John 138 | lastName: McLovin 139 | # If a Property is not available in the Class, you can the URL of the property 140 | https://schema.org/birthDate: 1991-01-20 141 | # Perhaps support relative paths to other local resources 142 | bestFriend: ./mary 143 | ``` 144 | 145 | Perhaps YAML isn't the right pick for this, because it's kind of hard to parse. 146 | 147 | ### AtomicData-Binary 148 | 149 | Possible extension: `.adb` 150 | 151 | A binary serialization format, designed to be performant and highly compressed. 152 | Perhaps it works like this: 153 | 154 | - An `adb` file consists of a large sequence of Maps and Statements 155 | - A _Map_ is a combination of an internal identifiers (the _ID_, some short binary object) and a URL strings. These make sure that URLs can be used again cheaply, if they are used multiple times. 156 | - A _Statement_ is a set of two IDs and a value, which can be a String, a URL or some binary format. 157 | - Perhaps some extra compression is possible, because many URLs will have a common domain. 158 | -------------------------------------------------------------------------------- /src/assets/atomic_data_logo.svg: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | -------------------------------------------------------------------------------- /src/roadmap.md: -------------------------------------------------------------------------------- 1 | # Strategy, history and roadmap for Atomic Data 2 | 3 | We have the ambition to make the internet more interoperable. 4 | We want Atomic Data to be a commonly used specification, enabling a vast amount of applications to work together and share information. 5 | This means we need a lot of people to understand and contribute to Atomic Data. 6 | In this document, discuss the strategic principles we use, the steps we took, and the path forward. 7 | This should help you understand how and where you may be able to contribute. 8 | 9 | ## Strategy for adoption 10 | 11 | - **Work on both specification and implementations (both client and server side) simultaneously** to make sure all ideas are both easily explainable and properly implementable. Don't design a spec with a large committee over many months, only to learn that it has implementation issues later on. 12 | - **Create libraries whenever possible.** Enable other developers to re-use the technology in their own stacks. Keep the code as modular as possible. 13 | - **Document everything**. Not just your APIs - also your ideas, considerations and decisions. 14 | - **Do everything public**. All code is open source, all issues are publicly visible. Allow outsiders to learn everything and start contributing. 15 | - **Make an all-in-one workspace app that stand on its own**. Atomic Data may be an abstract, technical story, but we still need end-user friendly applications that solve actual problems if we want to get as much adoption as possible. 16 | - **Let realistic use cases guide API design**. Don't fall victim to spending too much time for extremely rare edge-cases, while ignoring more common issues and wishes. 17 | - **Familiarity first**. Make tools and specs that feel familiar, build libraries for popular frameworks, and stick to conventions whenever possible. 18 | 19 | ## History 20 | 21 | - **First draft of specification** (2020-06). Atomic Data started as an unnamed bundle of ideas and best practices to improve how we work with linked data, but quickly turned into a single (draft) specification. The idea was to start with a cohesive and easy to understand documentation, and use that as a stepping stone for writing the first code. After this, the code and specification should both be worked on simultaneously to make sure ideas are both easily explainable and properly implementable. Many of the earliest ideas were changed to make implementation easier. 22 | - **[atomic-cli](https://crates.io/crates/atomic-cli) + [atomic-lib](https://docs.rs/atomic_lib/0.32.1/atomic_lib/)** (2020-07). The CLI functioned as the first platform to explore some of the most core ideas of Atomic Data, such as Properties and fetching. `atomic_lib` is the place where most logic resides. Written in Rust. 23 | - **[AtomicServer](https://github.com/atomicdata-dev/atomic-server/)** (2020-08). The server (using the same `atomic_lib` as the CLI) should be a fast, lightweight server that must be easy to set-up. Functions as a graph database with no dependencies. 24 | - **[Collections](schema/collections.md)** (2020-10). Allows users to perform basic queries, filtering, sorting and pagination. 25 | - **[Commits](commits/intro.md)** (2020-11). Allow keeping track of an event-sourced log of all activities that mutate resources, which in turn allows for versioning and adding new types of indexes later on. 26 | - **[JSON-AD](core/json-ad.md)** (2021-02). Instead of the earlier proposed serialization format `.ad3`, we moved to the more familiar `json-ad`. 27 | - **[Atomic-Data-Browser](https://github.com/atomicdata-dev/atomic-data-browser)** (2021-02). We wanted typescript and react libraries, as well as a nice interactive GUI that works in the browser. It should implement all relevant parts of the specification. 28 | - **[Endpoints](endpoints.md)** (2021-03). Machine readable API endpoints (think Swagger / OpenAPI spec) for things like versioning, path traversal and more. 29 | - **Classes and Properties editable from the browser** (2021-04). The data-browser is now powerful enough to use for managing the core ontological data of the project. 30 | - **[Hierarchies](hierarchy.md) & [Invitations](invitations.md)** (2021-06). Users can set rights, structure Resources and invite new people to collaborate. 31 | - **[Websockets](websockets.md)** (2021-08). Live synchronization between client and server. 32 | - **Use case: Document Editor** (2021-09). Notion-like editor with real-time synchronization. 33 | - **Full-text search** (2021-11). Powered by Tantivy. 34 | - **Authentication for read access** (2021-11). Allows for private data. 35 | - **Desktop support** (2021-12). Run Atomic-Server on the desktop, powered by Tauri. Easier install UX, system tray icon. 36 | - **File management** (2021-12). Upload, download and view Files. 37 | - **Indexed queries** (2022-01). Huge performance increase for queries. Allows for far bigger datasets. 38 | - **Use case: ChatRoom** (2022-04). Group chat application. To make this possible, we had to extend the Commit model with a `push` action, and allow Plugins to create new Commits. 39 | - **[JSON-AD Publishing and Importing](create-json-ad.md)** (2022-08). Creating and consuming Atomic Data becomes a whole lot easier. 40 | - **[@tomic/svelte](https://github.com/atomicdata-dev/atomic-svelte)** (2022-12). Library for integrating Atomic Data with Svelte(Kit). 41 | 42 | ## Where we're at 43 | 44 | Most of the specification seems to become pretty stable. 45 | The implementations are working better every day, although 1.0 releases are still quite a bit far away. 46 | At this point, the most important thing is to get developers to try out Atomic Data and provide feedback. 47 | That means not only make it easy to install the tools, but also allow people to make Atomic Data _without_ using any of our own tools. 48 | That's why we're now working on the JSON-AD and Atomizer projects (see below). 49 | 50 | ## Roadmap 51 | 52 | - **Video(s) about Atomic Data** (2023 Q1). Explain what Atomic Data is, why we're doing this, and how to get started. 53 | - **[Atomic Tables](https://github.com/atomicdata-dev/atomic-data-browser/issues/25)** (2023 Q2). A powerful table editor with keyboard / copy / paste / sort support that makes it easier to model and edit data. 54 | - **[E-mail registration](https://github.com/atomicdata-dev/atomic-server/issues/276)** (2023 Q1). This makes it easier for users to get started, and de-emphasizes the importance of private key management, as user can register new Private Keys using their e-mail address. 55 | - **Headless CMS tooling** (2023). Use Atomic-Server to host and edit data that is being read by a front-end JAMSTACK type of tool, such as NextJS or SvelteKit. 56 | - **[Atomizer](https://github.com/atomicdata-dev/atomic-server/issues/434)** (2023). Import files and automatically turn these into Atomic Data. 57 | - **Model Marketplace** (2023 Q4). A place where user can easily find, compare and use Classes, Properties and Ontologies. 58 | - **[Atomic-server plugins](https://github.com/atomicdata-dev/atomic-server/issues/73)** (2024). Let developers design new features without having to make PRs in Atomic-Server, and let users install apps without re-compiling (or even restarting) anything. 59 | - **Atomic-browser plugins** (2024). Create new views for Classes. 60 | - **1.0 release** (2024). Mark the specification, the server [(tracking issue)](https://github.com/atomicdata-dev/atomic-server/milestone/5) and the browser as _stable_. It is possible that the Spec will become 1.0 before any implementation is stable. Read the [STATUS.md](https://github.com/atomicdata-dev/atomic-server/blob/master/server/STATUS.md) document for an up-to-date list of features that are already stable. 61 | -------------------------------------------------------------------------------- /src/commits/concepts.md: -------------------------------------------------------------------------------- 1 | {{#title Atomic Commits: Concepts}} 2 | # Atomic Commits: Concepts 3 | 4 | ## Commit 5 | 6 | _url: [https://atomicdata.dev/classes/Commit](https://atomicdata.dev/classes/Commit)_ 7 | 8 | A Commit is a Resource that describes how a Resource must be updated. 9 | It can be used for auditing, versioning and feeds. 10 | It is cryptographically signed by an [Agent](https://atomicdata.dev/classes/Agent). 11 | 12 | The **required fields** are: 13 | 14 | - `subject` - The thing being changed. A Resource Subject URL (HTTP identifier) that the Commit is changing about. A Commit Subject must not contain query parameters, as these are reserved for dynamic resources. 15 | - `signer` - Who's making the change. The Atomic URL of the Author's profile - which in turn must contain a `publicKey`. 16 | - `signature` - Cryptographic proof of the change. A hash of the JSON-AD serialized Commit (without the `signature` field), signed by the Agent's `private-key`. This proves that the author is indeed the one who created this exact commit. The signature of the Commit is also used as the identifier of the commit. 17 | - `created-at` - When the change was made. A UNIX timestamp number of when the commit was created. 18 | 19 | The **optional method fields** describe how the data must be changed: 20 | 21 | - `destroy` - If true, the existing Resource will be removed. 22 | - `remove` - an array of Properties that need to be removed (including their values). 23 | - `set` - a Nested Resource which contains all the new or edited fields. 24 | - `push` - a Nested Resource which contains all the fields that are _appended_ to. This means adding items to a new or existing ResourceArray. 25 | 26 | These commands are executed in the order above. 27 | This means that you can set `destroy` to `true` and include `set`, which empties the existing resource and sets new values. 28 | 29 | ### Posting commits using HTTP 30 | 31 | Since Commits contains cryptographic proof of authorship, they can be accepted at a public endpoint. 32 | There is no need for authentication. 33 | 34 | A commit should be sent (using an HTTPS POST request) to a `/commmit` endpoint of an Atomic Server. 35 | The server then checks the signature and the author rights, and responds with a `2xx` status code if it succeeded, or an `5xx` error if something went wrong. 36 | The error will be a JSON object. 37 | 38 | ### Serialization with JSON-AD 39 | 40 | Let's look at an example Commit: 41 | 42 | ```json 43 | { 44 | "@id": "https://atomicdata.dev/commits/3n+U/3OvymF86Ha6S9MQZtRVIQAAL0rv9ZQpjViht4emjnqKxj4wByiO9RhfL+qwoxTg0FMwKQsNg6d0QU7pAw==", 45 | "https://atomicdata.dev/properties/createdAt": 1611489929370, 46 | "https://atomicdata.dev/properties/isA": [ 47 | "https://atomicdata.dev/classes/Commit" 48 | ], 49 | "https://atomicdata.dev/properties/set": { 50 | "https://atomicdata.dev/properties/shortname": "1611489928" 51 | }, 52 | "https://atomicdata.dev/properties/signature": "3n+U/3OvymF86Ha6S9MQZtRVIQAAL0rv9ZQpjViht4emjnqKxj4wByiO9RhfL+qwoxTg0FMwKQsNg6d0QU7pAw==", 53 | "https://atomicdata.dev/properties/signer": "https://surfy.ddns.net/agents/9YCs7htDdF4yBAiA4HuHgjsafg+xZIrtZNELz4msCmc=", 54 | "https://atomicdata.dev/properties/previousCommit": "https://surfy.ddns.net/commits/9YCs7htDdF4yBAiA4HuHgjsafg+xZIrtZNELz4msCmc=", 55 | "https://atomicdata.dev/properties/subject": "https://atomicdata.dev/test" 56 | } 57 | ``` 58 | 59 | This Commit can be sent to any Atomic Server. 60 | This server, in turn, should verify the signature and the author's rights before the server applies the Commit. 61 | 62 | ### Calculating the signature 63 | 64 | The signature is a base64 encoded Ed25519 signature of the deterministically serialized Commit. 65 | Calculating the signature is a delicate process that should be followed to the letter - even a single character in the wrong place will result in an incorrect signature, which makes the Commit invalid. 66 | 67 | The first step is **serializing the commit deterministically**. 68 | This means that the process will always end in the exact same string. 69 | 70 | - Serialize the Commit as JSON-AD. 71 | - Do not serialize the signature field. 72 | - Do not include empty objects or arrays. 73 | - If `destroy` is false, do not include it. 74 | - All keys are sorted alphabetically - both in the root object, as in any nested objects. 75 | - The JSON-AD is minified: no newlines, no spaces. 76 | 77 | This will result in a string. 78 | The next step is to sign this string using the Ed25519 private key from the Author. 79 | This signature is a byte array, which should be encoded in base64 for serialization. 80 | Make sure that the Author's URL resolves to a Resource that contains the linked public key. 81 | 82 | Congratulations, you've just created a valid Commit! 83 | 84 | Here are currently working implementations of this process, including serialization and signing (links are permalinks). 85 | 86 | - [in Rust (atomic-lib)](https://github.com/atomicdata-dev/atomic-server/blob/ceb88c1ae58811f2a9e6bacb7eaa39a2a7aa1513/lib/src/commit.rs#L81). 87 | - [in Typescript / Javascript (atomic-data-browser)](https://github.com/atomicdata-dev/atomic-data-browser/blob/fc899bb2cf54bdff593ee6b4debf52e20a85619e/src/atomic-lib/commit.ts#L51). 88 | 89 | If you want validate your implementation, check out the tests for these two projects. 90 | 91 | ### Applying the Commit 92 | 93 | If you're on the receiving end of a Commit (e.g. if you're writing a server or a client who has to parse Commits), you will _apply_ the Commit to your Store. 94 | If you have to _persist_ the Commit, you must perform all of the checks. 95 | If you're writing a client, and you trust the source of the Commit, you can probably skip the validation steps. 96 | 97 | Here's how you apply a Commit: 98 | 99 | 1. Check if the Subject URL is valid 100 | 2. Validate the signature. This means serialize the Commit deterministically (see above), check the Agent's publickey (you might need to fetch this one), verify if the signature matches. 101 | 3. Check if the timestamp matches is OK. I think an acceptable window is 10 seconds. 102 | 4. If the Commit is for an existing resource, get it. 103 | 5. Validate the Rights of the one making the Commit. 104 | 6. Check if the `previousCommit` of the Commit matches with the `previousCommit` of the Resource. 105 | 7. Iterate over the `set` fields. Overwrite existing, or add the new Values. Make sure the Datatypes match with the respective Properties. 106 | 8. Iterate over the `remove` fields. Remove existing properties. 107 | 9. If the Resource has one or more classes, check if the required Properties are there. 108 | 10. You might want to perform some custom validations now (e.g. if you accept an Invite, you should make sure that the one creating the Invite has the correct rights to actually make it!) 109 | 11. Store the created Commit as a Resource, and store the modified Resource! 110 | 111 | ## Limitations 112 | 113 | - Commits adjust **only one Resource at a time**, which means that you cannot change multiple in one commit. ([issue](https://github.com/atomicdata-dev/atomic-data-docs/issues/130)) 114 | - The one creating the Commit will **need to sign it**, which may make clients that write data more complicated than you'd like. You can also let Servers write Commits, but this makes them less verifiable / decentralized. 115 | - Commits require signatures, which means **key management**. Doing this securely is no trivial matter. 116 | - The signatures **require JSON-AD** serialization 117 | - If your implementation persists all Commits, you might need to **store a lot of data**. 118 | -------------------------------------------------------------------------------- /src/assets/atomic_data_logo_padded.svg: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | --------------------------------------------------------------------------------