146 | You may edit your local copy, but changes will not be synchronized until you
147 | pass your "local key" to an owner of the document and they authorize you.
148 |
149 |
e.stopPropagation()}>
150 | Your local key is:
151 |
210 | ${authStatus}
211 | ${authForm}
212 |
213 | `
214 |
215 | function copyToClipboard () {
216 | copy(localKey).then(() => {
217 | customAlert.show('"Local Key" copied to clipboard')
218 | state.localKeyCopied = true
219 | emit('render')
220 | })
221 | }
222 |
223 | function submit (event) {
224 | const input = event.target.querySelector('input')
225 | const writerKey = input.value.trim()
226 | if (writerKey !== '') {
227 | emit('authorize', writerKey)
228 | input.value = ''
229 | }
230 | event.preventDefault()
231 | }
232 | }
233 |
234 | function keydown (event) {
235 | if (event.key === ' ' || event.key === 'Enter') {
236 | event.target.click()
237 | }
238 | }
239 |
--------------------------------------------------------------------------------
/EXTRA-NOTES.md:
--------------------------------------------------------------------------------
1 | # Extra Notes
2 |
3 | These were part of an early draft of the blog post, but I cut them out because the post was too long.
4 |
5 | ### Dat Keys
6 |
7 | In the demo, each shopping list is a Dat archive. When a Dat archive is created, it gets a "public key" (also know as just the "key") which consists of a long random string of hexadecimal numbers. It looks something like this:
8 |
9 | `621d7eb5478cabe2597141c40231893dfebd3490bb14b1a38012fdc3f25b9696`
10 |
11 | The key is essentially the "name" used to represent the Dat archive when it's time to share and replicate the data. The data inside the archive is also encrypted, so somebody needs the public key to be able to read what's inside. The key is long enough to be "unguessable" ... it's safe to assume that the only people who can sync and read the data are those who got their hands on the public key. If you want to keep your shopping list secret just to yourself, **don't share that key!** If you want to share a list between friends, just share the key with them only. If you want the whole world to see your shopping list, post the key publicly.
12 |
13 | ### The "gateway" service
14 |
15 | When you create a shopping list, and you are online, the data is automatically synced to a web service that is part of the demo. It is a simple two direction sync between your web browser and the Node.js program that acts as the "gateway" service. When you go offline, syncing stops, and when you go online again, syncing starts again. There is a small status indicator in the upper right to give you some feedback on whether or not you are connected, and if your data has been synced.
16 |
17 | ### Who do you trust?
18 |
19 | It is very important to make sure that you trust the gateway service, as it has a full copy of your data and the public key, so whoever is running the server can read any shopping list synchronized with it.
20 |
21 | If privacy is your concern, it is very easy to re-install the server used in this demo on some hardware that you control -- instructions are in the [README](https://github.com/jimpick/dat-shopping-list/blob/master/README.md) file.
22 |
23 | ### The "swarm"
24 |
25 | Once the data is synced, the gateway server will keep it's copy of the data and make it available for syncing to any other computers on the internet that want it. If somebody else has the "key", they can connect to the gateway server to download the data using the Dat project's "discovery" mechanisms.
26 |
27 | The peer-to-peer networking that goes on when many peers are connecting and sharing data to other peers is called "the swarm", because when there are many peers, the network activity is as busy as a beehive.
28 |
29 | The good news is that the web app doesn't have to drain your cell phone battery doing all that communication. Once the data is synced to the gateway server, the server does all that work on your behalf, even when you are no longer online.
30 |
31 | In order to prevent abuse, the gateway that is used in the demo clears itself out every 24 hours, and has a limit on the number of shopping lists that it will host in the swarm. Nobody will lose data when the gateway resets, as the master copy is in their web browser storage, and they can resync at any time. But don't expect to be able to sync with devices that have been offline for a long time. If you run your own gateway service, you can modify it to "pin" your data in the swarm for a longer period of time.
32 |
33 | A possible future enhancement would be to use a "pinning" service such as [Hashbase](https://hashbase.io/) (commercial) or [Homebase](https://github.com/beakerbrowser/homebase) (self-hosted) to keep synchronized shopping lists alive in the swarm forever.
34 |
35 | ### Replicating a shopping list
36 |
37 | Once you have created a shopping list in one place, and you want to sync it to somewhere else, you have to do a little "cut-and-paste" between devices (or people if you are sharing with somebody else).
38 |
39 | On the device where you created the shopping list, you need to get the "key" of the shopping list. In a web browser, you can simply copy the URL from the browser's location bar, eg:
40 |
41 | [https://dat-shopping-list.glitch.me/doc/621d7eb5478cabe2597141c40231893dfebd3490bb14b1a38012fdc3f25b9696](https://dat-shopping-list.glitch.me/doc/621d7eb5478cabe2597141c40231893dfebd3490bb14b1a38012fdc3f25b9696)
42 |
43 | As it's hard to copy the URL in a mobile browser, and impossible if you have saved the app to your home screen, there is a shortcut you can use. Just tap the "hex number" in the upper right (under the status display) and the URL will be copied to your clipboard.
44 |
45 | You can simple paste the URL you copied into a chat app or a wiki to transfer it privately to your other device, and then open it in a web browser on the other device.
46 |
47 | If you have already saved the app to your home screen, it is impossible to open the URL directly. In that case, you can use the "Have a link? Paste it here" button on the home screen.
48 |
49 | Once you have opened the link, the shopping list will be synced down to your browser. It is also registered in the list of all shopping lists in your browser.
50 |
51 | ## Multiple writers
52 |
53 | When you open a new link from another device or from somebody else, you will see a notice at the top that says "You are not currently authorized to write to this document" in red letters. In this demo, "document", "shopping list" and "multiwriter Dat archive" all mean the same thing.
54 |
55 | Every separate device, browser or user can write their own changes, so they are called "writers". To make things easier to explain, we're going to refer to each instance of the document as a separate writer, even though they might be controlled by the same person, be on the same device in a different browser, etc.
56 |
57 | As a new writer, you can make changes to your local copy of the document, but they won't be automatically synced back to the original writer.
58 |
59 | On the original device that created the document, it is possible to "authorize" new writers, and their changes will be replicated and merged into the "source" document. Once a local key is authorized, you could consider the new writer to be an "owner" of the document, as they can also authorize new writers.
60 |
61 | Each new writer has their own "local key" which represents their changes. In order to get authorized, they must copy this key from their local writable copy and send it back to any writer that is already authorized to write to the document.
62 |
63 | In a user-friendly system, this "key exchange" might be automated in some manner, but for this demo, we wanted to teach the basics.
64 |
65 | As a new writer, if you are unauthorized, you will see your local key on the screen, with a green button to "Copy to Clipboard." Simply send this key to an "owner", and they will paste it into the "Add a writer" input in their shopping list and click "Authorize". This will update the document, and the new writer should see that they are now authorized on their next sync. Any changes that were made by the new writer before they were authorized will be incorporated into the shopping list.
66 |
67 | Currently, there is no mechanism for de-authorizing already authorized writers.
68 |
69 | ## Things to try
70 |
71 | The demo has been tested on the following platforms:
72 |
73 | * Google Chrome
74 | * Firefox
75 | * Apple Safari
76 | * Microsoft Edge
77 | * Mobile Safari on iOS
78 | * Google Chrome on Android
79 |
80 | Some suggested experiments:
81 |
82 | 1. Try creating a shopping list on a web browser on your desktop or laptop, and then open it in a web browser on your phone.
83 |
84 | 2. Try exchanging keys so that both devices can write to the same shopping list.
85 |
86 | 3. On a phone, try the "Add to Home Screen" feature from the web browser. It works differently on Android than iOS. On Android, you will see the same shopping lists in the home screen app as you see in your browser, and you can only save one icon to the home screen. On iOS, you can make multiple icons on your home screen, and each icon will act like a different web browser with separate storage and have it's own independent list of shopping lists.
87 |
88 | 4. After syncing, try putting your phone into 'airplane' mode. The status display should display that the network is offline, and when you make changes to the shopping list, it should display the number of records to sync. If the status display displays "Worker Ready", then that means your platform supports service workers, so you should be able reload the page even when offline.
89 |
90 | 5. Try making some lists and share them with other people.
91 |
92 | 6. Try making a list and putting it on every device and web browser that you have.
93 |
94 | 7. Try using the experimental `dat-next` [command line](https://github.com/joehand/dat-next) tool to download the files from one of your shopping lists to your local file system.
95 |
96 | ## Development Notes
97 |
98 | The demo was primarily developed on [Glitch](https://glitch.com/edit/#!/dat-shopping-list). Glitch is really neat. It gives you a multi-user editing environment, as well as a backing virtual machine so you can run Node.js, as well as one click forking.
99 |
100 | The source code is published on [GitHub](https://github.com/jimpick/dat-shopping-list). The README has some additional information on how to deploy the code to platforms such as Heroku and Zeit, as well as Docker and running the demo using the `npx` tool from npm.
101 |
102 | Presently, multiwriter support hasn't been introduced into all of the Dat project tools, but it is being developed in the "hyperdb-backend" branch of [hyperdrive](https://github.com/mafintosh/hyperdrive/tree/hyperdb-backend). The core of multiwriter support is implemented in the [hyperdb](https://github.com/mafintosh/hyperdb) library.
103 |
104 | Hyperdb is useful standalone, without hyperdrive. Hyperdrive gives you a filesystem abstraction, ideal if you are dealing with large files, whereas hyperdb gives you a key/value store. The dat-shopping-list demo is quite simple and could have easily been implemented using only hyperdb, but I wanted to try out the filesystem capabilities.
105 |
106 | For the client side web framework, I used the [choo](https://github.com/choojs/choo) framework. To generate the service worker, I used [Workbox](https://developers.google.com/web/tools/workbox/) from Google. For bundling, it uses [budo](https://github.com/mattdesl/budo) which is a development server for [browserify](http://browserify.org/) projects. Unfortunately, Glitch has a few filesystem issues, so I couldn't use watch.json with budo, so it's necessary to do a full rebuild after every edit. There is no separate production build for the demo... the development server is the production server. Many other very useful npm modules were used ... you can find the list in the [package.json](https://github.com/jimpick/dat-shopping-list/blob/master/package.json) file.
107 |
108 | I dogfooded the project by keeping my development [task list](https://dat-shopping-list.glitch.me/doc/95fe65d1af31a38b22a31ab31bc7862e80071f8482e17c8aacd18e02842b3f55) as a shopping list - it was nice being able to synchronize between many devices and to be able to check off tasks as they were completed. I even managed to find and squash some corruption issues in hyperdb that popped. I have a list of possible [future features](https://dat-shopping-list.glitch.me/doc/bc14e0054876d561e4890c747ff9d38fe87bcc83a969e2bdb2ce5e4147defe11) in another list.
109 |
--------------------------------------------------------------------------------
/stores/shoppingList.js:
--------------------------------------------------------------------------------
1 | const rai = require('random-access-idb')
2 | const toBuffer = require('to-buffer')
3 | // const hyperdrive = require('hyperdrive')
4 | const hyperdrive = require('@jimpick/hyperdrive-next')
5 | const crypto = require('hypercore/lib/crypto')
6 | const newId = require('monotonic-timestamp-base36')
7 | const dumpWriters = require('../lib/dumpWriters')
8 | const downloadZip = require('../lib/downloadZip')
9 | const connectToGateway = require('../lib/websocketGateway')
10 | const customAlert = require('../components/customAlert')
11 |
12 | require('events').prototype._maxListeners = 100
13 |
14 | module.exports = store
15 |
16 | function store (state, emitter) {
17 | state.shoppingList = []
18 | state.localKeyCopied = false
19 | state.writeStatusCollapsed = window.localStorage.getItem(
20 | 'writeStatusCollapsed'
21 | )
22 |
23 | emitter.on('DOMContentLoaded', updateDoc)
24 | emitter.on('navigate', updateDoc)
25 |
26 | emitter.on('addLink', link => {
27 | const match = link.match(/([0-9a-fA-F]{64})\/?$/)
28 | if (match) {
29 | const key = match[1]
30 | emitter.emit('pushState', `/doc/${key}`)
31 | } else {
32 | customAlert.show('URL or key must contain a 64 character hex value', () => {
33 | const textInput = document.querySelector('.content input[type="text"]')
34 | textInput.removeAttribute('disabled')
35 | const submitButton = document.querySelector('.content input[type="submit"]')
36 | submitButton.removeAttribute('disabled')
37 | })
38 | }
39 | })
40 |
41 | function updateDoc () {
42 | emitter.once('render', () => {
43 | document.body.scrollIntoView(true)
44 | // Do it again for mobile Safari
45 | setTimeout(() => document.body.scrollIntoView(true), 200)
46 | })
47 | state.error = null
48 | state.authorized = null
49 | state.shoppingList = []
50 | state.localKeyCopied = false
51 | state.docTitle = ''
52 | if (!state.params || !state.params.key) {
53 | state.archive = null
54 | state.key = null
55 | state.loading = false
56 | emitter.emit('render')
57 | } else {
58 | const keyHex = state.params.key
59 | console.log(`Loading ${keyHex}`)
60 | state.localFeedLength = null
61 | emitter.emit('fetchDocLastSync', keyHex)
62 | const storage = rai(`doc-${keyHex}`)
63 | const archive = hyperdrive(storage, keyHex)
64 | state.loading = true
65 | emitter.emit('render')
66 | archive.ready(() => {
67 | console.log('hyperdrive ready')
68 | console.log('Local key:', archive.db.local.key.toString('hex'))
69 | dumpWriters(archive)
70 | state.archive = archive
71 | state.key = archive.key
72 | if (state.cancelGatewayReplication) state.cancelGatewayReplication()
73 | state.cancelGatewayReplication = connectToGateway(
74 | archive, updateSyncStatus, updateConnecting
75 | )
76 | readShoppingList()
77 | archive.db.watch(() => {
78 | console.log('Archive updated:', archive.key.toString('hex'))
79 | dumpWriters(archive)
80 | readShoppingList()
81 | })
82 | })
83 | }
84 | }
85 |
86 | emitter.on('createDoc', docName => {
87 | const {publicKey: key, secretKey} = crypto.keyPair()
88 | const keyHex = key.toString('hex')
89 | console.log('Create doc:', docName, keyHex)
90 | const storage = rai(`doc-${keyHex}`)
91 | const archive = hyperdrive(storage, key, {secretKey})
92 | archive.ready(() => {
93 | console.log('hyperdrive ready')
94 | state.key = key
95 | state.archive = archive
96 | let shoppingList = ['Rice', 'Bananas', 'Kale', 'Avocados', 'Bread', 'Quinoa', 'Beer']
97 | writeDatJson(() => {
98 | writeShoppingListItems(() => {
99 | console.log('Done')
100 | emitter.emit('writeNewDocumentRecord', keyHex, docName)
101 | })
102 | })
103 |
104 | function writeDatJson (cb) {
105 | const json = JSON.stringify({
106 | url: `dat://${keyHex}/`,
107 | title: docName,
108 | description: `Dat Shopping List demo - https://${state.glitchAppName}.glitch.me/`
109 | }, null, 2)
110 | archive.writeFile('dat.json', json, err => {
111 | if (err) throw err
112 | cb()
113 | })
114 | }
115 |
116 | function writeShoppingListItems (cb) {
117 | const item = shoppingList.shift()
118 | if (!item) return cb()
119 | const json = JSON.stringify({
120 | name: item,
121 | bought: false,
122 | dateAdded: Date.now()
123 | })
124 | archive.writeFile(`/shopping-list/${newId()}.json`, json, err => {
125 | if (err) throw err
126 | writeShoppingListItems(cb)
127 | })
128 | }
129 | })
130 | })
131 |
132 | function updateSyncStatus (message) {
133 | const {
134 | key,
135 | connectedPeers,
136 | localUploadLength,
137 | remoteUploadLength,
138 | localDownloadLength,
139 | remoteDownloadLength
140 | } = message
141 | if (state.key && key !== state.key.toString('hex')) return
142 | state.connected = !!connectedPeers
143 | state.localUploadLength = state.loading ? null : localUploadLength
144 | state.localDownloadLength = state.loading ? null : localDownloadLength
145 | if (state.key && connectedPeers) {
146 | state.connecting = false
147 | state.syncedUploadLength = remoteUploadLength
148 | state.syncedDownloadLength = remoteDownloadLength
149 | emitter.emit(
150 | 'updateDocLastSync',
151 | {
152 | key,
153 | syncedUploadLength: remoteUploadLength,
154 | syncedDownloadLength: remoteDownloadLength
155 | }
156 | )
157 | }
158 | emitter.emit('render')
159 | }
160 |
161 | function updateConnecting (connecting) {
162 | state.connecting = connecting
163 | }
164 |
165 | function readShoppingList () {
166 | const archive = state.archive
167 | const shoppingList = []
168 | archive.readdir('/shopping-list', (err, fileList) => {
169 | if (err) {
170 | console.log('Error', err)
171 | state.error = 'Error loading shopping list'
172 | emitter.emit('render')
173 | return
174 | }
175 | console.log('Shopping list files:', fileList.length)
176 | readTitleFromDatJson((err, title) => {
177 | if (err) {
178 | console.log('Error', err)
179 | state.error = 'Error loading shopping list'
180 | emitter.emit('render')
181 | return
182 | }
183 | readShoppingListFiles(err => {
184 | if (err) {
185 | console.log('Error', err)
186 | state.error = 'Error loading shopping list'
187 | emitter.emit('render')
188 | return
189 | }
190 | console.log('Done reading files.', title)
191 | updateAuthorized(err => {
192 | if (err) throw err
193 | state.loading = false
194 | state.docTitle = title
195 | state.shoppingList = shoppingList
196 | emitter.emit('writeNewDocumentRecord', state.params.key, title)
197 | emitter.emit('render')
198 | })
199 | })
200 | })
201 |
202 | function readTitleFromDatJson (cb) {
203 | archive.readFile('dat.json', 'utf8', (err, contents) => {
204 | if (err) {
205 | console.error('dat.json error', err)
206 | return cb(null, 'Unknown')
207 | }
208 | if (!contents) return cb(null, 'Unknown')
209 | try {
210 | const metadata = JSON.parse(contents)
211 | cb(null, metadata.title)
212 | } catch (e) {
213 | console.error('Parse error', e)
214 | cb(null, 'Unknown')
215 | }
216 | })
217 | }
218 |
219 | function readShoppingListFiles (cb) {
220 | const file = fileList.shift()
221 | if (!file) return cb()
222 | archive.readFile(`/shopping-list/${file}`, 'utf8', (err, contents) => {
223 | if (err) return cb(err)
224 | try {
225 | const item = JSON.parse(contents)
226 | item.file = file
227 | shoppingList.push(item)
228 | } catch (e) {
229 | console.error('Parse error', e)
230 | }
231 | readShoppingListFiles(cb)
232 | })
233 | }
234 | })
235 | }
236 |
237 | function updateAuthorized (cb) {
238 | if (state.authorized === true) return cb()
239 | const db = state.archive.db
240 | console.log('Checking if local key is authorized')
241 | db.authorized(db.local.key, (err, authorized) => {
242 | if (err) return cb(err)
243 | console.log('Authorized status:', authorized)
244 | if (
245 | state.authorized === false &&
246 | authorized === true &&
247 | !state.writeStatusCollapsed
248 | ) {
249 | emitter.emit('toggleWriteStatusCollapsed')
250 | }
251 | state.authorized = authorized
252 | cb()
253 | })
254 | }
255 |
256 | emitter.on('toggleBought', itemFile => {
257 | const item = state.shoppingList.find(item => item.file === itemFile)
258 | console.log('toggleBought', itemFile, item)
259 | // item.bought = !item.bought
260 | const archive = state.archive
261 | const json = JSON.stringify({
262 | name: item.name,
263 | bought: !item.bought,
264 | dateAdded: item.dateAdded
265 | })
266 | archive.writeFile(`/shopping-list/${item.file}`, json, err => {
267 | if (err) throw err
268 | console.log(`Rewrote: ${item.file}`)
269 | })
270 | })
271 |
272 | emitter.on('remove', itemFile => {
273 | const item = state.shoppingList.find(item => item.file === itemFile)
274 | console.log('remove', itemFile, item)
275 | // item.bought = !item.bought
276 | const archive = state.archive
277 | archive.unlink(`/shopping-list/${item.file}`, err => {
278 | if (err) throw err
279 | console.log(`Unlinked: ${item.file}`)
280 | })
281 | })
282 |
283 | emitter.on('addItem', name => {
284 | console.log('addItem', name)
285 | const archive = state.archive
286 | const json = JSON.stringify({
287 | name,
288 | bought: false,
289 | dateAdded: Date.now()
290 | })
291 | const file = newId() + '.json'
292 | archive.writeFile(`/shopping-list/${file}`, json, err => {
293 | if (err) throw err
294 | console.log(`Created: ${file}`)
295 | })
296 | })
297 |
298 | emitter.on('authorize', writerKey => {
299 | console.log('authorize', writerKey)
300 | if (!writerKey.match(/^[0-9a-f]{64}$/)) {
301 | customAlert.show('Key must be a 64 character hex value')
302 | return
303 | }
304 | const archive = state.archive
305 | archive.authorize(toBuffer(writerKey, 'hex'), err => {
306 | if (err) {
307 | customAlert.show('Error while authorizing: ' + err.message)
308 | } else {
309 | console.log(`Authorized.`)
310 | customAlert.show('Authorized new writer')
311 | }
312 | emitter.emit('render')
313 | })
314 | })
315 |
316 | emitter.on('toggleWriteStatusCollapsed', docName => {
317 | state.writeStatusCollapsed = !state.writeStatusCollapsed
318 | window.localStorage.setItem(
319 | 'writeStatusCollapsed',
320 | state.writeStatusCollapsed
321 | )
322 | emitter.emit('render')
323 | })
324 |
325 | emitter.on('downloadZip', () => {
326 | console.log('Download zip')
327 | downloadZip(state.archive)
328 | })
329 | }
330 |
--------------------------------------------------------------------------------
/static/img/bg-landing-page.svg:
--------------------------------------------------------------------------------
1 |
--------------------------------------------------------------------------------