├── README.md ├── consortium ├── README.md └── facilitation.md ├── grants ├── 2018-MOSS │ ├── 00-announcement.md │ ├── 01-report1.md │ ├── 02-report2.md │ ├── 03-final-blogpost.md │ ├── MOSS-Dat-SoW.pdf │ ├── README.md │ └── application.md └── 2018-SamsungNEXT │ └── readme.md └── meeting-notes ├── 07-22January2020.md ├── 08-05February2020.md ├── 09-19February2020.md ├── 10-05March2020.md └── README.md /README.md: -------------------------------------------------------------------------------- 1 | [![deprecated](http://badges.github.io/stability-badges/dist/deprecated.svg)](https://dat-ecosystem.org/) 2 | 3 | More info on active projects and modules at [dat-ecosystem.org](https://dat-ecosystem.org/) 4 | 5 | --- 6 | 7 | # Dat Project Organizational Administration 8 | 9 | *Dat Project aims to be transparent about funding and the process of fundraising. We believe digital tools we rely on for community organization & communication should be transparently funded and operated. This repository contains information about Dat Project funding and other organizational administration.* 10 | 11 | *For details on Dat project governance, see [this repository](https://github.com/datproject/governance).* 12 | 13 | * Dat Project started in **[June 2013](https://github.com/datproject/dat/tree/464679267049899eafa345125a0f2212f91be456)** and has been funded mostly by grants by US-based private foundations. 14 | * Dat has received over **$1.75 million in grants**, which account for the majority of revenue. Additionally, Dat has received approximately $75k in private donations. 15 | * Dat has been supported by groups including: Knight Foundation, Sloan Foundation, Moore Foundation, and Mozilla. 16 | * [Code for Science & Society](http://codeforscience.org/), a US nonprofit, was founded in September 2016 to support the Dat Project. 17 | 18 | If you see something missing from this repository, or have questions - please open an issue! 19 | 20 | ## Financial Information 21 | 22 | As we continue to improve our transparency around operations, we hope to post new and existing grant applications, awards, and other related documents when applicable. We also hope to provide insight into the grant funding process. For other examples of financial transparency, see [this repository](https://github.com/joehand/open-source-finances). 23 | 24 | ### Donations 25 | 26 | See [opencollective.com/dat](https://opencollective.com/dat) for recurring donations. 27 | 28 | ## Grants and Grant Applications 29 | 30 | ### 2020 31 | 32 | * [NLnet](https://nlnet.nl) for a Rust port 33 | * July 2020 34 | 35 | ### 2019 36 | 37 | * [Wireline donation](https://www.wireline.io) 38 | * June 2019: Received $50,000 39 | 40 | ### 2018 41 | 42 | * [Handshake Free & Open Source Software Community Grant](https://www.handshake.org/) 43 | * December 2018: Received $100,000 44 | * [Samsung NEXT Stack Zero Grant](https://github.com/datproject/organization/tree/master/grants/2018-SamsungNEXT) 45 | * November 30 2018: Applied 46 | * January 2019: Received $63,000 ($70k with 10% fee to OpenCollective) 47 | * [Mozilla Open Source Support](https://github.com/datproject/organization/tree/master/grants/2018-MOSS) 48 | * April 2018: Applied 49 | * August 2018: Received $34,000 50 | * [Mozilla Open Web Fellow](https://github.com/datproject/organization/tree/master/grants/2018-MOWF) 51 | * Feburary 2018: Applied 52 | * September 2018: OWF Starting, Fellow working with CS&S on decentralized web. 53 | 54 | ### Pre-2018 55 | 56 | See [this blog post](https://blog.datproject.org/2017/09/15/dat-funding-history/) for a history of Dat Project funding. 57 | 58 | -------------------------------------------------------------------------------- /consortium/README.md: -------------------------------------------------------------------------------- 1 | # Dat Consortium 2 | 3 | This is a group comprised of representatives of various open source groups using and working on Dat tech in the 4 | community. 5 | -------------------------------------------------------------------------------- /consortium/facilitation.md: -------------------------------------------------------------------------------- 1 | # Call Facilitation 2 | 3 | Here's a guide for facilitating Consortium calls. 4 | 5 | This is a rough description of what Mauve has been doing thus far, and probably has a lot of room for 6 | improvement. 7 | 8 | ## Mock Agenda 9 | - Share the notes and ask for help/assistance 10 | - Catch up / Personal Updates / Announcements 11 | - Icebreaker question! 12 | - Foundation Discussion 13 | - CS&S Report by Jess or Joe 14 | - Event organization 15 | - Funding strategies 16 | - New conspirators (does anyone want to invite someone to the next meeting? who are they?) 17 | - Technical Discussion 18 | - DEPs 19 | - hyper* 20 | - UX (cli, desktop, docs) 21 | - Wrap up (last ~10/5 min) 22 | - Next meeting: Any suggested changes? 23 | - Who will do facilitation? 24 | - Upcoming birthdays 25 | 26 | --- 27 | 28 | ## Before the meeting 29 | 30 | - Send an email on the Monday of the week the call is on or update the description of the calendar event. 31 | - Create a new HackMD file for the meeting notes and share 32 | - Send an email an hour or two before the meeting, include link to the HackMD 33 | - Get in the meeting room 5-10 mins early to wait for people to show up 34 | 35 | ## During the meeting 36 | 37 | - Start recording the zoom meeting to the cloud before starting to actually talk about stuff 38 | - Repost the HackMD link a few times as people are joining in the room 39 | - Start with introductions. 40 | - Use the "raise hand" feature of Zoom for when people want to speak 41 | - Try to prioritize based on who raised their hand first, or those that haven't had a chance to talk 42 | - Go through the agenda items and ask if more items should be added, or if any should be tabled for later 43 | - Go through agenda items one at a time. 44 | - Sometimes it's good to go through each person on the call and ask them if they want to comment about the 45 | agenda item. This can be frustrating for those that are comfortable with voicing their opinions, but it can help 46 | for those that would otherwise not get a chance to speak. 47 | - Sometimes it's good to ask the room if anyone has something to add. Give a pause of a good few seconds while 48 | people decide if they want to add something. Do this at the end of every agenda item / meeting 49 | - Try to keep track of potential agenda items that you hear in the notes 50 | 51 | ## At the end of the meeting 52 | 53 | - Review agenda items that might have been proposed. 54 | - Ask people to propose agenda items or +1 existing ones 55 | - Record the agreed upon agenda items in the notes 56 | - Confirm availability for the next meeting 57 | - Thank everyone for coming and hand off the meeting to Mathias and Paul 58 | - Find someone else to facilitate next meeting 59 | 60 | ## After the meeting 61 | 62 | - Download the reocrded zoom call and listen to it at 1.5x speed and transcribe the conversations into the hackmd 63 | - Try to do this before Friday 64 | - Send out a follow up email with the updated notes, a brief summary of interesting things, and the date of the 65 | next meeting 66 | 67 | 68 | -------------------------------------------------------------------------------- /grants/2018-MOSS/00-announcement.md: -------------------------------------------------------------------------------- 1 | 2 | # Dat Project Receives Mozilla Open Source Support Grant 3 | 4 | **Community Building, Protocol Documentation, and Tooling** 5 | 6 | ## Milestones 7 | 8 | 1. Community Experience Research 9 | 2. Dat.Land Community Prototype 10 | 3. Dat Documentation & Implementation Guide (9 weeks) 11 | 12 | -------------------------------------------------------------------------------- /grants/2018-MOSS/01-report1.md: -------------------------------------------------------------------------------- 1 | # Dat MOSS Report 1 2 | 3 | *Report for milestone #3, completed 28 Feb 2019.* 4 | 5 | ### Milestone 3: 6 | 7 | #### Dat Documentation & Implementation Guide (9 weeks) 8 | 9 | * DEPs updates - complete specifications for all existing implementations 10 | * Dat Protocol Book - complete implementation guide for Hypercore module 11 | * Dat Project documentation updates (docs.datproject.org) with support for internationalization 12 | * Identify high-priority documentation needs for dependent modules 13 | * Continued development & support for dat.land community building 14 | 15 | ## Outcomes 16 | 17 | *For a full list of outcomes, see the [MOSS reporting issue](https://github.com/datproject/organization/issues/1). 18 | 19 | ### DEP Updates: 20 | 21 | * Completed or Drafted all DEPs describing impemented Dat software. 22 | * Patched DEP 000 "Template" - datprotocol/DEPs#55 23 | * Patched DEP 001 "Process" - datprotocol/DEPs#47 24 | * Patched DEP 004 "HyperDB" - datprotocol/DEPs#48 25 | * Reviewed DEP 004 "HyperDB" patch - datprotocol/DEPs#50 26 | * Authored DEP xxx "SLEEP Headers" (approval pending) - datprotocol/DEPs#56 27 | 28 | ### Dat Protocol Book: 29 | 30 | * Published the [Dat Protocol Implementation Guide](https://datprotocol.github.io/book/), providing implementors of the Dat Protocol with details on the hard-to-understand parts of Dat Protocol implementations. 31 | 32 | ### Dat Documentation Site Updates: 33 | 34 | * Published a new version of docs.datproject.org, migrating from our old documentation site software to support: 35 | * Internationalization support 36 | * Easier contributions 37 | * Better mobile support 38 | * Well-supported documentation website software 39 | * We've also updated outdated documentation, improved some tricky terminology based on community feedback, and identified contribution opportunities. 40 | 41 | ### High Prioritiy Documentation: 42 | 43 | * Updated Dat Project site based on initial community feedback and onboarding of our Mozilla fellow 44 | * [How Dat Works](https://datprotocol.github.io/how-dat-works/) developer onboarding that covers all the basic protocol and modules underlying the Dat Protocol. We also have an accompaning [blog post](https://blog.datproject.org/2019/01/21/how-dat-works/). This blog post led to a 2.5x one-day increase over our average daily unique visitors to datproject.org websites. 45 | 46 | ### Continued development & support for dat.land: 47 | 48 | * Launched early version of dat.land site based on community feedback. 49 | -------------------------------------------------------------------------------- /grants/2018-MOSS/02-report2.md: -------------------------------------------------------------------------------- 1 | # Dat MOSS Report 2 2 | 3 | Report to Mozilla on Milestones 1 + 2: 4 | 5 | ### Community Experience Research (3 weeks) 6 | 7 | * This milestone was achieved through an [online survey](https://datproject.typeform.com/report/kGKeKd/yH2z5AheOz1CRBDv), a [Dat meetup](https://events.datproject.org/) tangential to Data Tera Nemo (DTN) conference, discussions with key Dat contributors in Berlin prior to DTN, and informal interviews of people new to Dat at [Dat Night](https://datnight.org/) events. 8 | * Through this process, we learned how entry into the Dat ecosystem is difficult for people with varying goals. Because the work of identifying and organizing with existing community members took more time than expected, we were not able to do a full research process of “conditions for community growth” as outlined in the Statement of Work. 9 | * As a result of this work, we refocused (and rebranded) our core community efforts from “Dat Project” to “Dat Protocol Foundation” to highlight the goal of empowering a consortium of projects to build on a common foundation as discussed in [this blog post](https://blog.datproject.org/2019/12/02/dat-project-is-now-dat-protocol-foundation/). The consortium has now had five meetings. 10 | 11 | 12 | ### Dat.Land Community Prototype (4 weeks) 13 | 14 | * We launched an early prototype of this website last fall, before doing the Community Experience Research. Based on the feedback from the community research process, we created a new [website structure](https://dat.discourse.group/t/rebuilding-the-dat-website-proposed-site-map/79) to fully encompass all of the touch points Dat community members interacted and learned about Dat. 15 | * While this process was really helpful, it also became a bit overwhelming as it required a complete website overhaul and consolidating several different websites and content structures into one. As a result, it took much longer than expected and involved many hours of volunteer work not covered under this MOSS award. 16 | * The [new website](https://dat.foundation/) has launched with the help of the community. The website includes a CMS, allowing easier contributions and content development for various skill types. Our docs include a new tutorial and we will continue to expand those. While we did not get as far as hoped with the MOSS award, we feel it has allowed us to create a solid foundation for continued improvements and community contributions. 17 | 18 | 19 | -------------------------------------------------------------------------------- /grants/2018-MOSS/03-final-blogpost.md: -------------------------------------------------------------------------------- 1 | 2 | In our application for the Mozilla Open Source Support award, we wrote: 3 | 4 | > Over the last four years, Dat has grown from a data sharing tool to a wider community supporting the Dat Protocol. As we work to broaden our community, we recognize the need to improve onboarding new community members, documentation, and supporting implementations in other languages. We’ve organized a Dat Protocol working group to document the existing specification and support other implementations, but we still need someone to help with wider community engagement. Unfortunately, with our constrained resources focusing on development and maintenance, these crucial needs get left behind. 5 | 6 | > While harder to quantify, we recognize that this work is vital to the future of the Dat Project and for collaborations with groups like Mozilla Common Voice. Investments in documentation and community building make it easy for people to use Dat, to suggest improvements, and get answers to questions. With this milestone, we will support someone to conduct user testing on existing documentation, create an improved user funnel, and identify additional community needs. 7 | 8 | A key goal was to understand the needs of the community around documentation and learning resources. Through this support from Mozilla, we’ve advanced the Dat ecosystem in two foundational ways: 9 | 10 | 1. Documenting the protocol and guiding developers interested in implementing the Dat protocol in other languages. 11 | 2. A complete overhaul and rebrand of “Dat Project” into “Dat Foundation” including updating and consolidating websites and documentation. 12 | 13 | In line with our goal to operate transparently, we've published our reports to Mozilla in [this Github Repo](https://github.com/datproject/organization/tree/master/grants/2018-MOSS) and you can also view the [reports from inidividual contributors](https://github.com/datproject/organization/issues/1). 14 | 15 | **Protocol Documentation** 16 | 17 | A key challenge we’ve faced over the five years of developing Dat is prioritizing implementation of the protocol versus standardization. With a small team of developers, much of the balance has been weighed towards implementation. This award was a great opportunity to get people outside of the core Dat developers to evaluate and document the protocol, allowing for people to begin implementations in other languages. 18 | 19 | The [How Dat Works](https://datprotocol.github.io/how-dat-works/) guide is a highlight of the work funded by this MOSS award. Not only does it guide new users through every byte and bit of Dat, we feel like it is a learning resource for developers interested in cryptography, decentralization, and even the internet more broadly and is a prime example of how protocol documentation should be. Read more about the process behind creating How Dat Works in [this blog post](https://blog.datproject.org/2019/01/21/how-dat-works/). 20 | 21 | We are excited to see new Dat implementations being work on in [Rust](https://github.com/datrs), [C++](https://github.com/datcxx/), [Python](https://github.com/datpy), [Kotlin](https://github.com/datkt), and more. 22 | 23 | **Dat Website Updates** 24 | 25 | A [new Dat website](https://dat.foundation/) has launched with the help of the community. Based on the feedback from the community research process, we created a [new website structure](https://dat.discourse.group/t/rebuilding-the-dat-website-proposed-site-map/79) to fully encompass all of the touch points Dat community members interacted and learned about Dat. The website includes a CMS, allowing easier contributions, and content development for various skill types. Our docs include a [new tutorial](https://docs.dat.foundation/docs/kappa) and we will continue to expand those. This website includes a solid foundation for continued improvements and community contributions. 26 | 27 | Read more in [this post](https://blog.datproject.org/2019/12/02/dat-project-is-now-dat-protocol-foundation/) about the move from “Dat Project” to “Dat Foundation”. 28 | 29 | --- 30 | 31 | This award was a great reminder of all the work required outside of software development to support a functioning and inclusive community project. We are grateful for Mozilla's support of this work and excited that we've been able to engage our community, improve documentation, and create educational resources for people interested in decentralization more broadly. 32 | -------------------------------------------------------------------------------- /grants/2018-MOSS/MOSS-Dat-SoW.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/dat-ecosystem-archive/organization/a741011cd667e0bc2c5181d38b13f26c3cc865d6/grants/2018-MOSS/MOSS-Dat-SoW.pdf -------------------------------------------------------------------------------- /grants/2018-MOSS/README.md: -------------------------------------------------------------------------------- 1 | 2 | # Dat Project - 2018 MOSS Grant 3 | 4 | In August 2018, Dat Project received a Mozilla Open Source Support grant to support community building, protocol documentation, and related tooling. Our grant application, reports, and all related documentation will be recorded here. 5 | 6 | * [Application](application.md) 7 | * [Approved Statement of Work](MOSS-Dat-SoW.pdf) 8 | 9 | ## Status 10 | 11 | * Completed milestone #3 and invoiced Mozilla on February 28th. 12 | 13 | ## Milestones 14 | 15 | ### 1. Community Experience Research (3 weeks) 16 | 17 | - [x] Conduct survey of existing community members 18 | - [x] Research & document examples of conditions for community growth 19 | - [x] Run user tests on existing on-boarding with select members from representative user groups. 20 | 21 | ### 2. Dat.Land Community Prototype (4 weeks) 22 | 23 | - [x] Tutorial/Cookbook section covering 5-10 Dat development use cases 24 | - [x] Area for showcasing Dat-built applications, encourage “dogfooding” 25 | - [x] New developer on-boarding funnel 26 | - [x] Process for adding community-driven pages with 2-3 examples 27 | 28 | ### 3. Dat Documentation & Implementation Guide (9 weeks) 29 | 30 | - [x] DEPs updates - complete specifications for all existing implementations 31 | - [x] Dat Protocol Book - complete implementation guide for Hypercore module 32 | - [x] Dat Project documentation updates (docs.datproject.org) with support for 33 | internationalization 34 | - [x] Identify high-priority documentation needs for dependent modules 35 | - [x] Continued development & support for dat.land community building 36 | 37 | 38 | ## Reporting 39 | 40 | We will publish reports here when they are completed. Reports can include: how work went relative to expectations, any technical tradeoffs required, any new issues discovered, explanations of any changes from original SoW. Reports are no longer than 1-2 pages of bullet points. 41 | 42 | 1. Report #1 (Due after completion of Milestone 1) - COMPLETED 43 | 2. Report #2 (Due after completion of Milestones 2 - 3) 44 | 3. Final Blog Post - Detail what we were able to accomplish with MOSS award. 45 | -------------------------------------------------------------------------------- /grants/2018-MOSS/application.md: -------------------------------------------------------------------------------- 1 | # Dat MOSS Grant Application 2 | 3 | *Submitted April 2018* 4 | 5 | Mozilla Open Source Support (MOSS) "Foundational Technology" Application 6 | Mozilla’s mission is big and ambitious, and we couldn't achieve it without relying on software built by others in our community. We are eager to support you in your work on an open source/free software project that Mozilla relies on. 7 | 8 | * Before applying please make sure you have read and met the criteria for the MOSS Program 'Foundational Technology' track (found at https://wiki.mozilla.org/MOSS/Foundational_Technology). In particular, in order to apply for a award, you must submit this application with the name of a Mozilla champion who has agreed to support it. 9 | * If your application is successful, we may publish any part of this application which is not Personally Identifying Information, to promote program transparency and/or to serve as examples for other applicants. 10 | * We recommend you join the public mailing list for MOSS at: https://lists.mozilla.org/listinfo/moss , to make sure you receive updates on the progress of the program. 11 | * Reports suggest that Google Forms have a global character limit of 18-19k characters, over and above the limit on individual fields. 12 | 13 | #### Selection Criteria 14 | 15 | All criteria are indicative rather than determinative - that is to say, they will make us more or less likely to make an award, but none will guarantee a particular outcome. 16 | 17 | * How reliant is Mozilla on the project's technology? 18 | * Do others also rely on it outside of Mozilla products? 19 | * Is the technology unique? Are they doing something different? 20 | * What role does the project play in the open source ecosystem? 21 | * What sort of reputation does the project have in general, if any? This includes reputation in technical, inclusion and other areas 22 | * Is the project known for something besides the code we are relying on? 23 | * Will this award make a significant impact on the project? 24 | * Is the level of funding appropriate for the task to be accomplished? 25 | * Does the person (or group) who will receive the money have a track record of delivering? 26 | * What does the champion have to say? Is the case compelling? 27 | 28 | The minimum award for the Foundational Technology track is $10,000, and the maximum is $250,000. 29 | 30 | --- 31 | 32 | ## Application 33 | 34 | ### Project description: 35 | 36 | Dat is a nonprofit-backed data sharing protocol for applications of the future. With software built for researchers and data management, Dat empowers people with decentralized data tools. 37 | 38 | ### Requested amount: 39 | 40 | $230,000 41 | 42 | ### What are the concrete, specific outputs and outcomes this award would produce? 43 | 44 | *Please describe what you would use the funds for - what you are going to build, hack or fix, and how that would be of benefit to the world. As long as the costs are well explained, we are happy to consider funding activities whose benefit is harder to quantify than adding features - for example, maintenance, standardization, metrics, usability, user support, community building, tooling improvements, performance or documentation. Please also explain how you arrived at the dollar figure requested, by breaking it down as necessary into smaller pieces. The level of detail given should reflect how much money you are requesting; a $250k award will need much more detail than a $20k one. (Max 8k chars)* 45 | 46 | Our community and ecosystem is rapidly expanding to new areas including arts & humanities, offline applications in the Global South, and libraries and archives. To see these decentralized communities flourish, we need support to continue improving the core and unique features of Dat. The five proposed milestones, below, will create the foundation for decentralized applications for communities across a wide range of industries from work like Mozilla's Common Voice to offline mapping in the Amazon done by Digitial Democracy. 47 | 48 | Through these milestones, we will improve the user experience of the core Dat features. The innovative features of Dat will create new potential for connection and openness on the web. 49 | 50 | 1. Multiwriter Integration for Dat Command line and Node Library ($51,000) 51 | 52 | Over the last 6 months, we’ve been working hard on the low-level technology to allow multiple people to collaborate with a distributed file system. With this milestone, we will integrate this into the Dat command line and Node.js library. 53 | 54 | Once released, a new class of decentralized file management and data processing pipelines will be open to a wide range of users. Cloud-based file management and backup helped to spur a new wave of online collaboration tools but these tools lock users into specific providers, without having control of their data or being able to collaborate offline. With this milestone, developers will be able to use Dat to create peer-to-peer applications with location and owner-agnostic data pipelines on top of a solid foundation of privacy and security. 55 | 56 | In the process of developing the low-level libraries for this, we’ve made Dat much faster for handling small files. While we have much of the underlying work completed. But we need to work on integrating these improvements into our user-facing applications. This work will also include best practice guide for key management and data backups, as this is a new area for many users. 57 | 58 | With the completion of this milestone, users will be able to collaborate on decentralized file sharing applications with the Dat command line interface and Node library. 59 | 60 | (4 months for 2 people) 61 | 62 | 2. Versioning Integration for Dat Command line and Node Library ($51,000) 63 | 64 | While often essential to data analysis, versioning of datasets continues to be a difficult task – especially when it comes to accessing different versions quickly. By using technology popular in other blockchain systems, append-only logs, Dat enables data creators to automatically version datasets. However, unlike most blockchain software we do not require a global public log or excessive computation to verify new additions, bringing Dat more into line with the use cases of git. While everything added to Dat is versioned, we have yet to expose versioning interfaces in our user-facing tools. 65 | 66 | Through this milestone, we will work with the Common Voice project at Mozilla to identify the needs around data versioning and user interfaces to make these accessible. We recognize that, if done poorly, these features could cause a lot of user confusion and frustration. By grounding this work in specific use cases, we hope to improve their implementations. Once complete, users will be able to access automatically versioned subsets of data alongside human-tagged versions. Additionally, to improve reproducibility, users can create permanent identifiers for specific versions of data for replicating research and computation results. 67 | 68 | All of these features are available under-the-hood of Dat already, and are exposed in some advanced API interfaces. However, we have not had the resources to expose these features in the Node library or the Dat command line interface. This work will add APIs to the Dat Node.js library for managing, storing, and accessing different versions of data. In the Dat command line, we will allow users to interact with these underlying APIs to manage and create different versions of data. 69 | 70 | (3 months for 2 people) 71 | 72 | 3. Selective File Downloads for Dat Command line and Node Library ($34,000) 73 | 74 | With data moving around to more places, and becoming larger and larger, it’s more important than ever to be able to easily subset parts of a large dataset and only download the needed parts. For instance, in the Mozilla Common Voice project, the entire dataset is downloaded as a 12GB zip file and expected to continue growing. But within the dataset, there are separate subsets for training or other uses. Rather than downloading everything, Dat will enable users to download these on an as-needed basis, saving disk space and bandwidth. 75 | 76 | This feature is available in the current command line but is not interactive. This work will add a command line interface for selecting and managing partial downloads for a given dataset. Paired with the versioning milestone, these milestones will allow users to download and manage different versions of the same dataset. 77 | 78 | By combining these two features, we envision users can run data analysis on different versions of data without managing the datasets separately. 79 | 80 | (2 months for 1 person) 81 | 82 | 4. Community Building, Protocol Documentation, and Tooling ($34,000) 83 | 84 | Over the last four years, Dat has grown from a data sharing tool to a wider community supporting the Dat Protocol. As we work to broaden our community, we recognize the need to improve onboarding new community members, documentation, and supporting implementations in other languages. We’ve organized a Dat Protocol working group to document the existing specification and support other implementations, but we still need someone to help with wider community engagement. Unfortunately, with our constrained resources focusing on development and maintenance, these crucial needs get left behind. 85 | 86 | While harder to quantify, we recognize that this work is vital to the future of the Dat Project and for collaborations with groups like Mozilla Common Voice. Investments in documentation and community building make it easy for people to use Dat, to suggest improvements, and get answers to questions. With this milestone, we will support someone to conduct user testing on existing documentation, create an improved user funnel, and identify additional community needs. 87 | 88 | (4 months for 1 person) 89 | 90 | 5. Improved Network Analytics and Tooling ($25,500) 91 | 92 | One of our recurring challenges in building decentralized applications is providing analytics and debugging network problems. For example, in our work with libraries, the institutions want a measure of how often an asset it used so they can prioritize preservation and improved availability. Because not all downloads go through a single server, we have to use alternate methods for providing needed information about use. 93 | 94 | In this milestone, we will create tooling for institution-focused analytics and debugging. These services will work with existing Dat discovery services and institutional software to provide improve analytics about data usage in Dat. 95 | 96 | (3 months for 1 person) 97 | 98 | Milestones Total: $195,500 99 | Overhead: $34,500 (15% for Code for Science & Society) 100 | Total: $230,000 101 | 102 | This award will make a significant impact on the project, allowing us to bring on paid contributors who have previously been volunteers or short-term contractors. As an entirely grant-funded project, our funding is often project-based, making it more difficult to improve our core tooling. This award would be vital support for areas we have been under-resourced. 103 | 104 | ### Please share an endorsement from your Mozillian champion: 105 | 106 | *Ask your champion to write a paragraph or two on why supporting your work via this award is a great idea. This is not just about whether the goal is important, but also about whether you are the right team to achieve it. All endorsements must include the sentence: "I have read the text of this application and I believe the applicants are technically capable of achieving the work outlined." (Max 4k chars)* 107 | 108 | *[Redacted]* 109 | 110 | ### Please tell us more about how your project is managed. Please describe your core team. 111 | 112 | The Dat Project is an open source project with independent governance that is fiscally sponsored by the nonprofit Code for Science & Society (CS&S). The Dat Project is managed by three working groups that cover maintenance and development of the protocol (Protocol Working Group), community engagement (Community Working Group), and the project’s strategy, management, and funding (Core Governance Team). 113 | 114 | The Dat Protocol Working Group (PWG) governs changes to the protocol and is tasked with documenting existing implementations. We convened this team to enable community members to create Dat implementations in other languages. This team meets biweekly, and more information on their work including membership and past meeting minutes can be found https://github.com/datprotocol/working-group. 115 | 116 | We are forming a Community Working Group (CWG) to manage Code of Conduct, review documentation designed to welcome folks to the project (Contributing Guide, etc), respond to issues within the community, and develop resources and guidelines to keep the community safe and inclusive. The CWG will meet monthly and new members are on-boarded annually, we expect this group to start with approximately 6 people. 117 | 118 | The Dat Core Governance Team team is currently made up of four core Dat contributors and CSS leadership. This team develops the project’s strategic goals, assesses the project’s progress, makes decisions on Dat’s participation in partnerships and projects, writes grants and pursues funding to support work on Dat. The Core Governance Team contains members of both the PWG and the CWG, who keep the strategy team informed of activities and issues in those groups. This team also writes blog posts (https://blog.datproject.org/) and other public communication about the direction and priorities of the project, often in collaboration with the PWG and CWG. Membership on this team is limited to people who have more than one year of sustained involvement in Dat and can bring other skills and expertise (grant writing, financial planning, strategic partnerships) to the group. 119 | 120 | ### Please tell us more about your community. 121 | 122 | The Dat Project community brings together people working across different domains to solve problems of decentralized networks and data transfer. Our community includes researchers and scientists, people building peer-to-peer web applications, and activists committed to open public data. Our community is unified in its belief that decentralized approaches are a critical part of a better web. Dat participates in the quaretely Code for Science & Society Community calls, a summary of the last call is here. 123 | 124 | 125 | Projects with existing code and an existing community are preferred but this is not decisive. 126 | If you have participation guidelines / code of conduct please include pointers to them. 127 | 128 | Code of Conduct - https://github.com/datproject/Code-of-Conduct/blob/master/CODE_OF_CONDUCT.md 129 | Contributing Guidelines - https://github.com/datproject/dat/blob/master/CONTRIBUTING.md 130 | 131 | 132 | 133 | -------------------------------------------------------------------------------- /grants/2018-SamsungNEXT/readme.md: -------------------------------------------------------------------------------- 1 | 2 | # Samsung NEXT Stack Zero Grant 3 | 4 | > Description: a non-equity grant program that aims to support early-stage teams building decentralized technologies with funding, expertise, and access to the Samsung NEXT network. 5 | 6 | https://samsungnext.com/whats-next/introducing-the-samsung-next-stack-zero-grant/ 7 | 8 | 9 | ## Application: Dat CLI and Evangelism 10 | 11 | > 1. First, tell us your project's name.** 12 | 13 | Distributed real-time file sharing, by The Dat Project 14 | 15 | > 2. Give us your high-level pitch and tell us what problem you're trying to solve 16 | 17 | Ever tried collaboratively editing large files and folders with multiple machines? Usually this involves one of a few strategies: being in the same location (usb), using a cloud service (Dropbox, email), or using old but reliable technical tools (rsync). None of these easily store, track, and share your data over time. Git is the closest tool in wide use today, but is designed for editing source code, not large datasets. Because Git was designed to track changes to relatively small text files, many datasets are too big for it. While Git has revolutionized the way that open source code is written for the software development world, workflows around sharing data remain primitive. Users are stuck choosing between security, speed, or ease of use. So, we've taken the best parts of Git, BitTorrent, and Dropbox to design Dat, a next-generation file sharing tool that has the potential to rival all three. 18 | 19 | Dat boasts a host of user-facing applications, including a command line tool, desktop application, web browser, and a wide variety of modules for storing and sharing data. When data is shared with Dat, other users with the link can copy, modify, and contribute back to that data if they are granted permission. Our approach is to give a practical strategy for avoiding privatized and siloed data storage services that dominate the space. Dat is simple: we believe that by keeping Dat focused on low-level data version control and replication tasks, we can keep the user interface simple and encourage collaboration across data silos, improving the interoperability, transparency, and openness of the data shared on the web. 20 | 21 | 22 | > 3. Which fields or areas does your technology impact the most? Feel free to use comma-separated tags. 23 | 24 | Distributed applications, Peer-to-peer web 25 | 26 | > 4. Do you have a website? A public code repository would do. 27 | 28 | datproject.org 29 | 30 | > 5. Where is the team based? If distributed, give us a general idea. 31 | 32 | We work with community members globally. The grant would be managed by Dat's core governance team based in Portland, OR, Oakland, and Denmark. 33 | 34 | > 6. If accepted, how will you utilize the grant? 35 | 36 | Currently, the command line tool has no core maintainers, while the list of feature requests has increased substantially. This grant would be used to support the equivalent of one core software developer at 8k per month. 37 | 38 | The following activities are planned within the course of the grant. Proposed timelines are if the full 70k are granted, and would be adjusted accordingly for smaller award sizes. 39 | 40 | Objective 1: Needfinding and Maintenance (3 months) 41 | 42 | * Full triage and prioritization of open and outstanding bugs on the datproject/dat GitHub repository. 43 | * Bug fixing for the most pressing issues with accompanying tests. 44 | * Facilitation of weekly Dat community sync meetings where developer-volunteers can learn about the next steps, blockers, and pressing issues for the CLI tool. 45 | 46 | Objective 2: Feature parity with similar tools (4 months) 47 | 48 | * Modify the history of an archive to create branches and perform merges, just like git. This is a long-standing feature request which exists in the Dat web browser called Beaker, but not yet in the command line tool. 49 | * Run a daemon that manages local Dat instances, exposing commands through a local tcp connection that can be queried from other applications. This is a feature that exists in similar tools such as ipfs. 50 | * Pass more detailed options for specifying how the client machine accesses the network, such as bandwidth caps and bootstrap servers for peer discovery. These options are already available in lower-level APIs, but need to be exposed through easy-to-use interface. 51 | 52 | Objective 3: Documentation (1 month) 53 | 54 | * A thorough documentation review of the command line tool, including interviews with developer-users. 55 | * Complete update of the documentation, including a review of the interactive 'Try Dat' user guide. 56 | 57 | > 7. Tell us your story: how long have you been working on this project, how is the community involved, or any other detail you want. 58 | 59 | The governance team currently consists of Mathias Buus, Karissa McKelvey, Joe Hand, and Danielle Robinson. Our team has long experience in open access and data problem solving, and each team member is committed to creating more open access to knowledge. The team comes directly from academia, and have been working on open source technology for most of their careers. Finally, our team builds code modularly, meaning that even "failure" of our implementation with a particular partner could still see the overall project succeed in unforeseen ways, by emitting components that can have impact on their own. 60 | 61 | The first code went into Dat on August 17, 2013. Throughout its history, Dat has had a primary focus on Dat sharing in civic data and research. The team grew to include 8 people at its height in 2016, with both Sloan foundation and Knight Foundation funding. Together, we created a community of open data advocates who use dat to publish data as decentralized public resources. The core command line tool has an important role to play by providing a solid foundation that can be used across organizations that need to share data seamlessly, securely, and efficiently. 62 | 63 | The command line tool's GitHub repository now has over 7,200 followers, and a wide variety of contributors. Dat is an open source project with community governance through a set of working groups. Over the next year, we aim to create a sustainable model for Dat through consulting contracts, grants, and donations. We've already begun this process, but need help with core maintenance support for the basic file sharing tool, and maintenance is difficult to fund. 64 | 65 | > 8. What is the most groundbreaking thing about your solution? 66 | 67 | Collaborative, versioned development and distribution is a useful model for many data-driven tasks — not just for source code. For people and organizations accustomed to a Git-based development model, it is natural to want to extend that model to data. For example, those who sit at the nexus of software development and data production — librarians, scientists, government agencies, etc.— have started using Git for the development and distribution of datasets. However, many data managers post their raw data online as read-only, never taking contributions from peers and downstream contributors, because there is no system in place for tracking changes to the data. Git enables code to get better as more people use it, because it enables collaboration between programmers. For lack of similar systems for data, improvements to data rarely make it back to the original dataset in this way. 68 | 69 | Dat is built with Node.js, which we use because of its excellent cross-platform support (Linux, Mac, and Windows) as well as its emphasis on data streaming. This means that Dat will never crash if you put too much data into it. The upper limit in terms of dataset size depends on available storage and bandwidth. To test Dat's capability to work with large datasets, we worked with *RESEARCHER*, of Ohio State University, and *RESEARCHER2*, a data analyst and programmer, to build a system for importing and indexing the 400 GB Wide-field Infrared Survey Explorer (WISE) dataset from NASA/Caltech, which includes data on over 600 million stars. We were able to effectively build a query engine on top of Dat that allowed users to get access to particular data on any cluster of stars in the sky. In this way, a scientist could use a Dat HTTP endpoint to refer to a particular set of data that they used in their analysis, getting data points immediately, despite the large size of the initial dataset. This versioned workflow and query engine would be time-consuming to build, and necessarily out of reach for most researchers without Dat. 70 | 71 | > 9. How does your solution further decentralization? 72 | 73 | Our primary audiences to date include the scientific research community, librarians, open government/civic data hackers, activists, and data scientists. Bringing decentralized technology to these hard-to-reach sectors is a crucial step towards the legitimization of decentralized technology in the public sphere. Over the years, we have chosen development priorities through conversations with data-driven researchers and open data experts at GovLab, Freeman Lab, FiveThirtyEight, California Civic Data Coalition, Wikimedia Community, California Digital Library, Indiana University, Trillian Project, The Wurm Lab, USGS, Ocean Health Index, International Monetary Fund, Data & Society, BetaNYC, Digital Ocean, and Jupyter Project. Through these personal relationships, we have identified significant pain points that span the open data landscape centered around the discovery and distribution of data. By offering a simple command line tool, we increase the interoperability of decentralized data storage and access between organizations like these. 74 | 75 | > 10. Where can we find you online (social media, Gitlab, Github, etc.)? 76 | 77 | @dat_project, github.com/datproject 78 | 79 | > 11. How should we contact you (e-mail or other)? 80 | 81 | hi@datproject.org 82 | 83 | 84 | ## Enabling Multi-User Collaboration with the Hyper* Ecosystem 85 | 86 | > 1. First, tell us your project's name.** 87 | 88 | Distributed real-time file sharing, by The Dat Project 89 | 90 | > 2. Give us your high-level pitch and tell us what problem you're trying to solve 91 | 92 | Decentralized software has potential to return control of digital information to the people. Yet, wide adoption of peer-to-peer technology is hindered by technical and ethical challenges. Dat Project is an open source, community driven, nonprofit-backed initiative that develops peer-to-peer tools to reimagine how information is owned, shared, and accessed. With Dat, the secure, fully-decentralized sharing of dynamic data is as easy as sharing a web link. Dat achieves this by building on a suite of modular components for data discovery, storage, and indexing called the Hyper* stack. 93 | 94 | The Hyper* stack occupies a unique spot in the world of decentralized tech: easy-to-use Lego blocks for building decentralized applications. At its core lies a cryptographically-secure append-only log, somewhat like a blockchain minus the strong consistency requirement, that we’ve named a “hypercore.” Using a hypercore, it’s dead-simple for a single writer to securely and efficiently distribute an ever-changing dataset to its readers, with no risk of tampering, no need for always-on connectivity, and no wasted bandwidth. For a huge class of P2P applications, this simple primitive has proven transformative. 95 | 96 | A worldwide ecosystem of developers is already using Hyper* modules as building blocks to solve novel problems for their communities, many far beyond Dat’s original scope. Over 200 public NPM projects currently depend on Hyper*, with major projects including a P2P web browser (Beaker Browser), a desktop science library (Science Fair) and a user-controlled alternative to YouTube (Hypervision). 97 | 98 | Our community has already built incredible things on the single-writer version of Hyper*, yet years of feedback reveal crucial missing pieces, the foremost being a lack of multi-user collaboration. We believe allowing multiple writers to modify a shared, distributed hypercore will lay the foundation for groundbreaking decentralized applications that put community ownership of digital information first. Even better, our existing users have laundry lists of radical improvements to their existing systems that depend on multi-user features. Beaker wants to enable new tools for collaborative publishing on the web. Science Fair wants to fundamentally change the peer-review process, where papers, annotations, and comments can be fluidly exchanged between collaborators. The list goes on. 99 | 100 | Our users are eagerly waiting; we’re very excited to deliver these features to them. 101 | 102 | > 3. Which fields or areas does your technology impact the most? Feel free to use comma-separated tags. 103 | 104 | Developer tools, Data Science, Distributed computing, Networking 105 | 106 | > 4. Do you have a website? A public code repository would do. 107 | 108 | datproject.org 109 | 110 | > 5. Where is the team based? If distributed, give us a general idea. 111 | 112 | We work with community members globally. The grant would be managed by Dat's core governance team based in Portland, OR, Oakland, and Denmark. 113 | 114 | > 6. If accepted, how will you utilize the grant? 115 | 116 | 117 | 118 | As of now, all development on the Hyper* stack is done by module consumers, application developers who depend on the stack but have primary obligations elsewhere. This grant will fund a dedicated developer ($8k/month for one year) to work on critical improvements to our low-level modules and protocol specifications that will enable collaborative peer-to-peer data sharing. 119 | 120 | To give clarity, here's a brief overview of the relevant modules in the Hyper* ecosystem, along with how they're currently used: 121 | 122 | 123 | 124 | * **_Hypercore_** is our bread and butter. It's an append-only log combined with a Merkle tree whose cryptographic properties make it easy to efficiently distribute among untrusted peers. Fundamentally, a hypercore only has a single writer, but allows for many readers. 125 | * **_Hyperdrive_** behaves more like a P2P Dropbox. It's a distributed filesystem that, under the hood, is just two hypercores. Like hypercore, the current version of hyperdrive only supports a single writer with many readers. 126 | * **_Hyperdb_** combines many hypercores, from different writers, into a single directory-structured view. This is the key step that enables multi-user collaboration! It's powered by a trie-like data structure, and uses vector clocks to ensure correct ordering. 127 | * **_Hypertrie_** is the core data structure currently used in hyperdb, but extract into its own module and with multiple performance improvements. It only allows for a single writer, but provides hyperdb's kv-store interface. 128 | * **_Hyperswarm_** is the next generation of our networking stack. It focused on provided easy to use APIs for things like UDP hole punching (NAT traversal), distributed discovery using modular DHTs, and authenticated connectivity using the NOISE framework. 129 | 130 | We've spent time implementing these foundational pieces for multi-user collaboration. The next step is to integrate these components into our stack as first-class citizens, then deploy them to users. Our developer will perform the following tasks to achieve this goal: 131 | 132 | ##### Test hypertrie performance in hyperdrive 133 | 134 | By deploying a hypertrie-backed version of hyperdrive to users, we'll be able to test the performance of the data structure while retaining our current single-writer API. With this change alone, we're anticipating a large performance boost for hyperdrive, which will be a boon to existing projects built on Hyper*. 135 | 136 | Since this step happens behind the scenes and does not alter our public APIs, it won't involve any additional documentation. We will, however, gather community feedback about performance and stability to ensure that no regressions have occured. 137 | 138 | #### Integrate hypertrie into hyperdb 139 | 140 | By separating core data structures (hypertrie) from coordination logic, we have the opportunity to create a more modular version of hyperdb which can combine each writer's trie in novel ways (to support union mounting and symlinking, for example, both of which are not possible in our current prototype). 141 | 142 | With this feature complete, we'll have an extensible, fast, multi-user database that will serve as the backbone both for the next iteration of Dat, and for apps which currently depend on our hyperdb prototype. 143 | 144 | #### Integrate hyperdb into hyperdrive 145 | 146 | Once we're confident that our hypertrie-based version of hyperdb performs correctly, we can then replace hypertrie with hyperdb in hyperdrive. Since hyperdb exposes a superset of the hypertrie API, this will not be a breaking change for any existing applications. 147 | 148 | Most importantly, this will give us access to hyperdb's multi-user collaboration API in hyperdrive, which is the top-level module in the Hyper* stack. Since most of the ecosystem, including the Dat CLI tool itself, is built on hyperdrive, we anticipate that this change will have the largest impact on the community. 149 | 150 | Deploying this step to users will be the final task in our push to deploy multi-user collaboration. 151 | 152 | #### Add noise-network to discovery modules 153 | 154 | Currently the Hyper* stack uses a capability system for authentication. This capability system works similar to something Github gists where you share a secret link that gives you access to read the underlying data. 155 | 156 | Based on community feedback we want to expand this into a fully featured authentication system that gives users the ability to whitelist exactly which peers they want to provide access to, and also a way to revoke that access. We've already implemented the cryptographic part of this in a module called noise-network and the next step will be to integrate this functionality into the actual stack. 157 | 158 | 159 | > 7. Tell us your story: how long have you been working on this project, how is the community involved, or any other detail you want. 160 | 161 | The first code went into Dat on August 17, 2013. Throughout its history, Dat has had a primary focus on data sharing in civic data and research, only recently with a wider focus on more general decentralized applications. We quickly realized that Dat had potential beyond than open data sharing, and began applying it to everything that uses a file system -- including websites and applications. 162 | 163 | We now have over 30 partners who are directly using our code, and a network of over 7,200 open source developers who are following us. Our adopters are diverse: from indigenous people protecting the Amazon rainforest (Digital Democracy) to librarians preserving scientific artifacts (California Digital Library). What these projects have in common is that they embed the values of decentralized ownership directly into the code. 164 | 165 | > 8. What is the most groundbreaking thing about your solution? 166 | 167 | The Hyper* ecosystem is unlike any other P2P project out there. While other decentralization projects build monolithic systems that require total buy-in from developers, we’ve instead chosen to embrace the Unix philosophy at all levels of our stack: build small component that do one thing well. Mixing and matching these components allows users to build powerful applications we never could have anticipated at the outset, and the possible combinations are endless. We believe the emergence of our sizable community reflects the value and utility of these principles. 168 | 169 | Adding more collaborative data structures in addition to upgrades our existing modules will make the stack even more powerful and we are super excited about the future projects this will allow users to build. 170 | 171 | > 9. How does your solution further decentralization? 172 | 173 | By creating a modular core of reusable, stable, well-documented modules, we aim to enable a lot with a little. Developers in our community often know how each module works, how it can be replaced if necessary, and how it fits together with other modules to accomplish a high-level task. 174 | 175 | Our “standard library” of reusable components spans a variety of features that developers often want: video streaming (hypervision), versioning (dat), file sharing (hyperdrive), distributed p2p database (hyperdb), graph structures (hypercore-graph), consensus (hyperquorum), distributed indexes (hypertrie), swarming data exchange on intermittent connections (hyperswarm), and encrypted p2p communication (noise-peer), to list just a handful. Importantly, most of these components are, under the hood, just different ways of arranging hypercores! 176 | 177 | Once developers understand our core principles, they are empowered to play with and rearrange our modules as they see fit for their own use-cases. The total is greater than the sum of its parts. 178 | 179 | > 10. Where can we find you online (social media, Gitlab, Github, etc.)? 180 | 181 | @dat_project, github.com/datproject 182 | 183 | > 11. How should we contact you (e-mail or other)? 184 | 185 | hi@datproject.org 186 | -------------------------------------------------------------------------------- /meeting-notes/07-22January2020.md: -------------------------------------------------------------------------------- 1 | # Dat Consortium Meeting #7 2 | 3 | 2020/01/22 4 | 5 | ## Attending 6 | 7 | - @RangerMauve 8 | - @dpaez 9 | - cblgh 10 | - paul 11 | - mafintosh 12 | - karissa 13 | - kevin faaborg 14 | - sam 15 | - andrewosh 16 | - frando 17 | 18 | ## Agenda 19 | 20 | - Catch up / Funding updates / Personal Updates 21 | - Summer event 22 | - thinking about requirements/ideas for the summer event 23 | - what you'd like to see in terms of structure & attendance & length 24 | - Next meeting: any suggested changes? Who will do facilitation? 25 | - Action Items 26 | 27 | ## Next Agenda 28 | - Summer event 29 | - get some possible dates 30 | - continue the conversation about the event 31 | 32 | ## Meeting Notes 33 | - **Tech meeting notes** 34 | - Daemon updates (from Andrew) 35 | - lot of internal testing (at hyperdivision) 36 | - private beta 37 | - internal testing has been going really well 38 | - share drive at the office like a simple use case, it has been going pretty well too 39 | - testing FUSE portion too 40 | - minor edge cases bugs, pretty much solved those 41 | - daemon is close to a new beta release after major docs pass 42 | - :rocket: 43 | - breaking change: publish command 44 | - added very simple telemetry (eg: active cores in memory, etc) 45 | - added debugging modes, like running only on memory or in the backoground 46 | - sparse by default 47 | - Beaker update (from Paul) 48 | - Been using the daemon 49 | - two apps running on https. 50 | - hyperdrive.network 51 | - want links to be sharable 52 | - when do we make something an application that has a url in front of it? 53 | - that domain followed by the key of the drive you're looking at 54 | - make hyperdrive's default serving like a website 55 | - if you want a file exploring experience then you go to hyperdrive.network 56 | 57 | - file explorer inside beaker 58 | - can use service workers to access offline (needs more research) 59 | - beaker.network 60 | - like a RSS/reddit mix 61 | - HTTPS deployment vs dat? 62 | - reduces risk to push it on https 63 | - safer to need to know it works reliably no matter what 64 | - opens up ability to create gateway behaviors 65 | 66 | - having the sites available offline 67 | - can be done using service workers 68 | - if someone wants to help, feel free - paul 69 | 70 | - Questions/General comments 71 | - Everything is sparse by default (daemon) 72 | - no download command yet, should be easy to add 73 | - Frando mentions a use case like a seeder on the cloud. Andrew mentions that it could be done via API. 74 | - Karissa mentions that they are after a grant that can be used for the CLI or a seeder. 75 | 76 | - Karissa shares some updates about mapeo/kappa-osm 77 | - Update to hypercore 8 recently, will deploy in production somtime next 6mos. 78 | 79 | - cabal is already on hypercore 8 & will move to hyperswarm at some point 80 | - sam shares: https://github.com/cliqz-oss/browser-f/pull/1418 81 | 82 | --- 83 | 84 | ### Personal updates 85 | 86 | - Diego: Working on p2p commons SDK. Doing an update to use the latest hyperdrive and corestore. Doing okay. So far it's been a good journey to the Dat 2 family of modules. 87 | - Andrew: Polishing daemon, getting ready for internal beta. 88 | - cblgh: Cabal-desktop having lots of progress. Nearing a release soon. Joining/leaving channels and notifications. Quit job and working on thesis - might be integrated into cabal - about local subjective moderation. Been quiet on cabal front otherwise. Gonna meet with cinnamon and nettle and talk about Cabal. Oh yeah there's now a cabal open collective for cabal: https://opencollective.com/cabal-club 89 | - Frando: Still working on Sonar. Progressing. Working on fast replication and indexing. Working to not have all feeds open at all times - can run into issues. Thinking about data model, being able to pull in cabal feeds into the search index - still being touched up. Thinking about future funding. 90 | - Karissa: Talked about getting someone to help with organization bylaws for the group. So someone doesn't leave for 2 years and come back saying they're running the show - making sure there's continuity for the group. Talked to Sumana from changeset nyc. Wants to charge 150/h which is a lot. We don't have much in the OpenCollective - trying to run an even so trying to see how much we want to spend. Getting support from CS&S. Talked to Joe - might want to give us one person to come to one meeting a month and help with project coordination and making sure that things get on track or finished and that people are following up with things. Might take that instead of paying for someone. As part of the fiscal sponsorship we're already paying for a person from the nonprofit (CS&S). Applied for MOSS grant (Mozilla Open Source), we did not get it. We had a small grant from them in 2019 that we didn't finish on time. It basically was all finished but we didn't wrap up the last bits. Because we didn't finish on time they didn't want to give us the grant this year. Feel a bit responsible because I was at my fellowship, Joe can speak to what happened as well. We were hoping to fund the CLI development. Hopefully we can still get a grant, just not this round. It was a lot of work for how much it is, it's a strict program and it's not a lot of money. Gonna look for other sources. Kinda upset because we did a lot of work to try to get that grant. Been researching how postgres does these meetings (foundation-related). Anyone that contributes to the core gets invited, there's a larger presense. Something to think this year is how we want to add folks to these meetings. One thing to talk about is a mailing list instead of 22 people on a email thread - if anyone has a VPS that they'd want to set up a mailing list software on (mailman) it'd be great. If anyone has ideas please speak up. 91 | - Kevin: Questions about the update - sorry we didn't get the grant, also general questions about the CLI. Is it correct that the CLI on the website is using old hyper modules? 92 | - Karissa: Yeah 93 | - Kevin: So the new initiative would be to make a new CLI that uses the new modules 94 | - Karissa: Yeah, ideally based on hyperdrive-daemon with a nice UX. 95 | - Kevin: If it's thin, it wouldn't be too hard to also do an electron app 96 | - Karissa: Yeah, it might not be that much work, but it'd be good to take it seriously. Could get it pretty far if we used that approach. 97 | - Kevin: What do we think is a priority or a plan for getting dat CLI / desktop without grant money. 98 | - Karissa: I'd be down to help make that happen. 99 | - Kevin: Me too for electron. 100 | - Frando: We're building a CLI for Sonar, the hyperdrive-daemon also has a CLI. Maybe closer to the process we'll be able to do something. 101 | - sidenote: also CoBox has a CLI with hyperdrive 10 support, but they aren't on the call today 102 | - Kevin: Working at Ara, ara.ne, making a software suite for creators and fans, can publish their content avoiding 30% platform task. Check out the electron client. Nothing happened in the past two weeks, working on a lot of electron things 103 | - Mauve: finishing up hyperswarm-web. Been working on a [demo app]() using hyperswarm-web. Going to start working on dat-sdk and release 1.0 version (using old dat -1.0-). Docs for the sdk and 1.0 release. 104 | - Paul: Trying to nail down the final mechanisms and UI/UX in Beaker 105 | - Sam: Dat release in Cliqz coming soon. Auto-publish of the dat foundation website. FOSSDEM beginning in Feb, gonna be talking about Dat stuff there. 106 | 107 | ### Summer event planning 108 | 109 | - Karissa: My interest in this helping is to help make it happen, talking to people, shuffling money around if we need to pay people, talking to people on the ground and keeping folks in the loop. Curious to see what folks want from the event. Assuming a lot of folks can come. One question: Do you want to be big and short - a public part of it, do you want more like a work retreat? Like a 7 or 5 day thing - a hack week. 110 | - Mauve: maximize serendipity and collaboration. unconference. show up and think about what we want to talk about and get into groups and collaborate together and hack away on stuff. i was at the redecentralize unconf and that was my first and it was amazing. t he unconference style: talking to people is the conference and how do you easily form groups 111 | - Karissa: Funny you mentioned redecentralized - we could reach out to them to have them run the conference in the way that they ran theirs. They were at the last event we had in Berlin. They would probably show up. 112 | - CBLGH: +1 to unconferences, I co-hosted 3 last year and it was well received 113 | - paul: when? Eary mid summer ish? It may affect where we are as a community based on that timing. It'd be a different story in the next month vs three to six months from now. We'd be trying to expand the community a lot more, expect more people than usual. It'd change the vibe. 114 | - Mauve: Can you elaborate on growing community 115 | - Paul: New versions are coming out, hopefully the toolset will come to a point where we can sustain growth and the product will stick. It'd be good to see people downstream from the tools working on it. 116 | - Mauve: Could you clarify "we" 117 | - Paul: I guess the beaker team, but everyone else as well. 118 | - Karissa: Yeah, I agree. It'd be nice to get more eyes on things and meet in greet folks like y'all. It'd be nice to have if we do it in six months. 119 | - Paul: How many people will we have introducing to the tech or more people that are already established. 120 | - Frando: I agree to many things you said - I'm not sure abotu numbers, I could do wild guesses but it's hard at the moment - how many peopl would come to a dat conferences? I also want an unconference where people hack together, kinda like DTN where people met together. Maybe having a combination of them - Have a weeklong of hacking / unconference and organizations and space, but not planned slots. Maybe a bit smaller for people that already know - and on the weekend having two days of more like talks and demos - more of a confernece. A combination of these two could work really well. People could dive in and the unconference part could be great. 121 | - Karissa +1 122 | - frando: weekend talks.. since you don't know how many people would come, then try something like a presale or rsvp so we can see what the tendancy is from how many people. setting the dates should be the first thing 123 | - Diego: We can do like a twitter poll or something similar that we can share and make some questions about a possible dat event this summer… to see some expectations and gather some feedback 124 | - cblgh: I think Frando and everyone's comments are really sensible. If we want something a bit smaller scale because it's hard to predict - maybe we can have a few unconferences at the same time. Some people in North Americas, some in South, some in Europe. Maybe a reasonable decentralization where there's one person in each area that's responsible for the unconference in their area. people responsible in their area basically inviting people and making sure they come, having food. Maybe a precursor to a DTN style event. Paulo wanted to keep DTN alive - maybe we can combine with that. 125 | - Paul: I wanted to ask - what do we realistically think our budget is like - that could determine the approach we can take. 126 | - Karissa: We'll have at least 6k EU - just from grants. I think that we can go pretty far to ask for some folks to pitch in - especially some of the bigger grants like the SHuttleworth funded group - you can ask for things here and there the way it works. We could ask to get them to pitch in 3-5k. 127 | - cblgh: Link? 128 | - karissa: https://shuttleworthfoundation.org/ 129 | - to clarify: they offer this only for groups that are already funded by shuttleworth 130 | - Paul: I haven't run a conference but I'm guessing it's pricy. If it's 6-8 range, I'm not sure how much space would cost or food. The safe way would be an unconf - setting expectations correctly. If you come don't expect it to be polished 131 | - Karissa: Depends on the size, if it's 50 or less we can probably get a space / food donated to us. Being a non-profit we get a lot of nice folks. 132 | - mauve: nice to have in person at least, but unconferences as well would be good. 133 | - Diego: Agree, doing an international event communicates to others, if we do something 6 months from now, we can be in a position where we can show dat as stable and reliable - it's a good opportunity to show to others. 134 | - cblgh: Organized 3 unconferences, looked at when we had free space. Paid out of pocket - made food ourselves, recouped the cost through donations. People love to attend, even if they don't ahve prior history. If we have both where people setup up in their area, it shouldn't be that hard. Then maybe do 200 to 300 bucks, it shouldn't be hard to budget. 135 | - Karissa: I think doing unconferences in areas is good. Having it too close to the international conference would result in peopel not wanting to go and going to the regional one. Regarding budget, if we could spend on people's flights and hotels, like getting one big airbnb, it'd be good to use budget for. One of the benefits of having a public facing portion - it being a bit more organized than an unconfernece - if you have speakers, you might be able to get their companies to pay for them to talk since it'll be for exposure / marketing. We could try to go for something like that. It might be more work, but we can probably do it in a budget-neutral way. Maybe get companies to sponsor, to get a bigger space. Once you get a bigger space you need to pay more. But there are benefits from having a larger space and meeting more people / seeing talks. I agree the most important part is getting some dates set and structure of how many days so people can start blocking out their calendars - right now peopel are scheduling vacations. 136 | - Sam: Agree with what Karissa is saying - this talk of unconferences sounds more like meetups - having distributed official meetups would be good - it could be free, companies putting them on for one day events. There's also hackathons - Mozilla Global Hackathon, a couple of years ago we made a couple of rooms in our office availbe for anyone that wanted to hack in our office. If we're going to spend on a proper event, then it'd make sense if it was a bit bigger. All these things aren't something people would travel for. It needs to be big enough and have headline speakers that'd make sense. It'd make sense to put it along other conferences, having citical mass around and cheaper to attach. The other Mozilla event in October (MozFest) has siderooms that are easy to get for small tech groups. 137 | - Diego: I was thinking that it'd be important to strategically arrange things closer to a similar event, also I was thinking it depends on how we name this, if we name it a Dat event only - it could make things more difficult for getting more money, but if we call it a decentralization event it could help us connect with others and get more help. 138 | - cblgh: Regarding what sam touched on, do we know of other conferences that we can touch on, like Mozzfest. Scuttlebutt has one, but it's in Brazil. It'd be closer for Diego at least. Do people know of any conferences on the map? 139 | - Diego: One thing related to events, in South America you can do more with less money. 140 | - Paul: Wanted to +1 what Sam and Alex said. I'd say that having one larger one would be nice. It's harder to put together, getting meetups / hackathons are one thing, but having an event would get more eggs in one basket instead of spreading out. I wouldn't be able to find anyone in Austin, I'd have to go to the west coast or EU. Wanted to +1 Alex regarding to attaching to an event. 141 | - Sam: Had a look at what JS conferences are this summer in EU, not sure if anyone attends these: There's a few of them happening, but they're all quite mainstream. Dunno if there's a more alternative set of conferences that are more hidden away. If there's DTN that'd be really good, it was great last year. If anyone has a good source it'd be good to get a shortlist 142 | - Karissa: Agree with Paul - if we're gonna spend 6-8 k largly on food and planes, then doing it and making it bigger would be worth it / attachng to another confernce. Not sure about DTN itself - it was a lot of energy and people get tired. Just to mention it. DTN was like 3 days and everyone wanted to be engaged. One thing to note - not a bad thing, but it could be a really long stretch. One other thing - dats roots are in science, so attaching to a science conferences or infiltrating something like Mozfest sounds exciting - potentially lots of folks that haven't heard about dat yet. 143 | - Mauve: We should have a GH issue where peopel can post conferences we can piggyback onto. 144 | - Mafintosh: Has there been discussion about the type of event we were aiming for? Is it gonna be a conference or bootcamp? 145 | - Karissa: People wanted to do both - have unstructured time, but it's important to also have structure - having a public event, at least a day or two. Having the weekend be a public thing, and having weekdays being work week. 146 | - Mafintosh: We do that a lot in node core, we have a conference and then we also have a collaborative conference where peopel meet at smaller venues and does workshops and stuff. Maybe we can do that. 147 | - Karissa: Speaking of nodeconf, it'd be cool to see if anyone there would wanna help or give resources 148 | - Mafintosh: I know people there, one thing I know is it's really really expensive, massive deficites every year. Might wanna keep it small - it's a lot of work to even do small ones - we should be realistic in terms of scale also. Get some dats down 149 | - Karissa: Anyone against having a weekend of public stuff and 4-5 days of collaborative stuff 150 | - Diego: To add one thing - for small conferences, if we say the public event is a two day thing, have the first day for workshops or soemthings, the other day for talks. Two days of talks would be ambitious. 151 | - Mafintosh: We could do one day talk, one day summit, could be good to fit in a weekend for people with jobs. 152 | - Karissa: yeah, and people for whom this is their job could cowork and collaborate. 153 | - Mafintosh: I'd prefer earlier this year than later 154 | - Karissa: Diego said, it'd be good to do a poll, have some date options,mauve wnated a GH issue 155 | - Mauve: Why earier than later? 156 | - Mafintosh: Sooner than later because there's a lot of new stuff coming out, it'd be good to use the momentum instead of waiting 7 months. Getting feedback and excitement, and doing another one next year 157 | - Karissa: Earliest would be late April, should have at least two months in advance, could consider doing it in March. The earlier you do it the more work in a small period. 158 | - Mafintosh: I was thinking may/april, just not September 159 | - Karissa: Could probably figure out dates async, and then finalize them. Have a deadline for when we want to figure out the date. 160 | - Frando: i'd vote for june or july to give planning etc more time and still be in summer. +1 karissa, issue plus settling in two weeks 161 | 162 | 163 | ### End of meeting 164 | 165 | - Diego: There's some action items, these are the results of this meeting. Frando mentioned setting up the mailing list. Mauve mentioned having a GH issue for tracking conferences. Finally, we should get some dates before the next meeting. We can also use this time to prepare the agenda for next meeting. Guessing it'll be related to the summer event. 166 | - Pfrazee: Will lead figuring out date ranges. Have we settled on strucutre? 167 | - Karissa: A day of talks, a day of not talks, and a few days of people that do Dat for a job collaborating. 168 | - Pfrazee: So like 1-3 days of collab, one of talks, one of hangout. If people ask about location do we have any vibe on that? 169 | - Karissa: Did a suggestion of doing in EU, maybe hanging off another conference, else people might want to come if it's in the summer in EU. Brazil would be good, expensive to get to, but easier to host. 170 | - Paul: I like EU more, better than America right now. 171 | - Mafintosh: JS Congress in Munich in April. Munich is quite nice for flights. 172 | - Karissa: Sam said he could offer thir space for one day. Localted in Munich. Lets put that on the list of potentials, should get at least two or three 173 | - Paul: How to timebox? 174 | - Karissa: 5th - two weeks 175 | - Karissa: Wanted add an agenda item, any feedback for the meeting. Diego did a great job. 176 | - Diego: I can facilitate next time. Thank you very much, we have info in the notes, will share it with you via email. We'll see you on Feb 5th. 177 | 178 | ## Action Items 179 | * frando sets up mailing list for foundation group at riseup.net 180 | * maybe two .. one for foundation matters (invite-only?) and one for comm-comm (general public discussion) 181 | * mauve: GH issue where people can post conferences we can piggyback onto 182 | * pfrazee: get some possible dates for the event, lock in dates for big event by feb 5 183 | 184 | # Next Meeting Wednesday, 5 Feb 185 | -------------------------------------------------------------------------------- /meeting-notes/08-05February2020.md: -------------------------------------------------------------------------------- 1 | # Dat Consortium Meeting #7 2 | 3 | 2020/02/05 4 | 5 | ## Attending 6 | 7 | - @dpaez 8 | - @pfrazee 9 | - @andrewosh 10 | - Kevin Faaborg 11 | - Kira 12 | - Jessica Hardwicke @CS&S 13 | - Mauve 14 | 15 | ## Actions Items 16 | 17 | ### Previous Action Items Review 18 | 19 | - Frando was going to set up the mailing list, mauve will follow up 20 | - piggyback conferences, had a [twitter thread](https://twitter.com/RangerMauve/status/1221555008187047942), not conclusive. 21 | - Paul made a [spreadsheet](https://docs.google.com/spreadsheets/d/1C05PNuP_IZy5k6p3IYRQ0_F65LjPTzli3MWTYwV2b08/edit#gid=0) to track the date: [github repo](https://github.com/datproject/organization/pull/13). Need to find conferences to piggyback off of. Should get more info. Sam from Cliqz said they could donate space if we're in Munich. 22 | - Frando: Should continue with these topics, get more feedback from community. We should keep these for the next meeting. There is some progress, which is nice. 23 | 24 | ## Agenda 25 | - Summer event 26 | - get some possible dates 27 | - continue the conversation about the event 28 | 29 | ## Next Agenda 30 | 31 | - Paul will facilitate the next meeting 32 | - Talk more about event (particularly the agenda and how to split up work) 33 | - Talk about CS&S integration. 34 | 35 | ## Meeting Notes 36 | 37 | ## **Foundation meeting notes** 38 | 39 | #### Catching up 40 | 41 | - Diego: Working at Geut, doing p2p stuff. Using the dat2 modules (hyperdrive, corestore, hyperswarm). https://github.com/p2pcommons/sdk-js 42 | - Jessica: Working at CS&S since December, hired as the sponsored project services manager, interfaces with fiscally sponsored projects at CS&S. Dat is in a special situation because it was formed first and then CS&S was made after. Dat is technically not part of thr FSP program. Looking at the future for dat and what services CS&S can provide. From Portland, Oregon, now in Seattle. Worked in academia studying microbial-something and bioinformatics. Worked on genomics. Working in a lab. Fun bringing order to a lab. Like bringing order to things and that's why I'm here. 43 | - Andrew: Working on Hyperdrive with Mathias and on Beaker with Paul. Did a beta release of hyperdrive-daemon. Making stability improvements, exposing more things in the CLI to do more testing with groups and get more info. Improving stats interfaces when daemon is running, the peers you're connected to and upload/download stats. Building tools and small modules around the daemon to make it easy for people to use hyperdrives hosted by it. Kinda like a one way version of the dat CLI tool where you can only get stuff out of the dat that you're hosting. Today working on other modules around getting notifications from drives in the FUSE interface to pop up on your desktop. Gearing up towards the beta launch. 44 | - links: https://github.com/andrewosh/hypercopy 45 | - https://github.com/andrewosh/hyperdrive-group-notifier 46 | - Kevin: Using hyper stack for delivery. Our electron app has etherium in it. People can sell their albums without needing a centralized platform. Website is ara.one, working on the electron portion of it. 47 | - Kira: Working at Digital Democracy and Cabal, building maps and chat respectively. Haven't been in a while want to catch up 48 | - Mauve: Finishing dat-sdk release. Going to work with Jessica (CS&S)improving dat foundation resources access for outsiders/collaborators. 49 | - Paul: In Austin Texas, working on beaker. trying to get keyboard shortcuts to work. Updates: Added versioning support to the browser (certificate info in the slide, can scroll through version). Using dog fooding to drive development. Added page commands, the terminal that can open in the side panel can now have commands defined by the page you're viewing. You can add stuff to the webterm and stuff like help will come with it so you can go through it more easily. Developers can provide commands to perform actions or just help development. In addition to the tooling for making hyperdrives, there's precreated scaffolds with websites that have some functionality. There's a wiki, and a codepen-style editor, and a module one, kind of like a github repo, baked in some tools: A build process, testing process and a demo page. The tests can execute kind of like an index.html. Can run all your tests in the browser. For the build tooling it can use the page commands. You can have scripts in the module and the web terminal can run those scripts. Trying to make that awesome for people. Got a dev diary where I'm going to demonstrate that. General usability stuff other than that. 50 | 51 | #### Summer event 52 | 53 | - Diego: We need to figure out the date, maybe we can get the event close to another event. We could talk a bit more around expectations around the event. Karissa isn't here, might have more information. Any ideas or comments regarding possible dates or a location? Last meeting we mentioned maybe april/may even June as possible months for the event. 54 | - Mauve: Looking at the survey, early may or early june could be good. A few people have april as no-go dates. 55 | - Pfrazee: June seems like the least conflicted. 56 | - Mauve: I like the idea of having more structured stuff for new folks, and less structure for hacking. We should start thinking about the agenda, think about who wants to talk. Follow up via email and maybe start a new IRC channel? 57 | - Frando: We can talk about the potential agenda stuff in this call. Any topics regarding the structure of the event? Last meeting we talked about having a few extra days leading up the event for hacking together more closely. 58 | 59 | #### CS&S 60 | 61 | - Jessica: We should start thinking about Dat becoming an FSP, thinking if people want to. Part of the requirements are to have formal governance structures. It'd be good if we had more docs from CS&S, but it's something to start thinking about. 62 | - Mauve: Wanna come to the next meeting? We could maybe have an agenda item to discuss this stuff and you and I can put something together describing it. 63 | 64 | ## **Tech meeting notes** 65 | 66 | - Andrew: Any questions about the daemon? 67 | - Pfrazee: Lets do a rundown about what got fixed lately? Could be interesting to share before Q&A 68 | - Andrew: First new thing is caching. Since we share one corestore across all hypercores in the daemon, we can use this to create a shared cache in the daemon. Right now it onyl cached tree nodes and not the data, saw perf hits in hypertrie. Thought it'd be good to do a data cache to speed up. Hard to do per hypercore when you want a max for all hypercores. Got a new powerful cache object to all hypercores which has a cache-like interface. Can set max values, large for the daemon. Can now cache hypertrie nodes, has good perf improvements. Can simplify other middle modules, can remove hyperdrive cache and rely more on hypercore. Gonna measure how good of a change this is. Other thing that popued up when testing was extremely large directory sizes. When doing a first sync you might need lots of roundtrips to do the initial lookup, views that might issue stat calls, you can get tonnes of roundtrips which leads to high latency. Larger dirs when you do an ls with thousands of files, you can get 5 second lag for a node that's within 200ms ping time. Can send hints for a hypertrie path, can then fetch them yourself and verify them so there's no security issue. Can massively reduce the number of roundtrips. One roundtrip to do the initial lookup, then a bunch of things sent over. Those are the biggest updates relating to performance. 69 | - Pfrazee: You were saying earlier that the codebase was getting pretty stable? 70 | - Andrewosh: We've had daemons running for 4-5 days with no issues in the FUSE interface 71 | - Pfrazee: You were saying there were N days before the last catastrophic data loss 72 | - Andrew: yeah one test was to have everyone have directories and mount on each other. One thing we did was do an LS. Mathias has a last incident file and we can look when that was. The time keeps getting larger every day. 73 | - Kevin: what platforms were you testing on? 74 | - Andrew: This is on Mac and Linux. We don't have windows machines to test on. We have a handful of issues recently. Particularly not related to FUSE. Not going to support FUSE on windows. The daemon without FUSE needs to work on Windows. 75 | - Kevin: I'm mostly on Windows. Our customers are mostly mobile or Windows. That platform is totally important to use. 76 | - Andrew: Same 77 | - Kevin: Lemme know how to help 78 | - Andrew: Gonna look at the recent issue on github to see if there are any blockers. Once Beaker comes around it'll be a really big test since that'll have to run on Windows. We don't have too many apps to test on Windows. Running the test suite and getting it all to pass on windows would be good. 79 | - Mauve: backwards compat? 80 | - Paul: has an strategy for this which has an option for updating to latest drive. 81 | - Andrew: Ideal beta launch date, kinda hard to come across, going by milestones. Once we get a few days without incidents we could start working on docs, figured out we need more debugging tools. Guessing debugging would take another week and then we'd shift to documentation and then looking at a release. 82 | 83 | -------------------------------------------------------------------------------- /meeting-notes/09-19February2020.md: -------------------------------------------------------------------------------- 1 | # Dat Consortium Meeting #9 2 | 3 | 2020/02/19 4 | 5 | ## Attending 6 | 7 | - Pfrazee (Facilitating) 8 | - cblgh 9 | - Sam Macbeth 10 | - Mauve 11 | - Frando 12 | - mafintosh 13 | - diego paez 14 | - kevin faaborg 15 | 16 | ## Actions Items 17 | 18 | ### Previous Action Items Review 19 | 20 | * **Mailing list** 21 | mailing lists were created (by Frando): dat-consortium@lists.riseup.net for everyone in the consortium meetings, dat-commcomm@lists.riseup.net as an open-for-everyone community mailing list. Frando can subscribe everyone in the current consortium mails to the consortium list and add mods. Signup page for the commcomm list is https://lists.riseup.net/www/subscribe/dat-commcomm 22 | 23 | ## Agenda 24 | 25 | - Talk more about event (particularly the agenda and how to split up work) 26 | - Talk about CS&S integration. 27 | 28 | ## Action Items 29 | 30 | - Move to mailing list 31 | - Look over the CS&S Dat document and leave comments https://hackmd.io/yn9I9NPHSayCN11jhG4sMA?view 32 | - Generate list of people interested in being a part of the advisory committee 33 | - cblgh is going to pick a date for the event based on the spreadsheet (https://docs.google.com/spreadsheets/d/1C05PNuP_IZy5k6p3IYRQ0_F65LjPTzli3MWTYwV2b08/edit#gid=0) 34 | 35 | ## Next Agenda 36 | 37 | - Frando facilitating 38 | - Discuss Dat Consortium membership 39 | - Lock down roles & responsibilities for the event 40 | 41 | ## Meeting Notes 42 | 43 | ## Foundation meeting notes 44 | 45 | #### Catching up 46 | 47 | - Pfrazee: Working on Beaker 48 | - Alexander: Working on Cabal, and thesis on moderation 49 | - Frando: Working on Sonar project. Building a distributed database for file metadata, based on hyperdrive storage and kappa-db for metadata db. Getting stack and interface ready, hoping to share a video soon-ish. Also involved in the dat rust implementation as a side project 50 | - Diego: From Geut, collaborating on the p2p commons project. Upgrading to dat 2 modules. 51 | - Mauve: Working on dat sdk, datdot and Natakanu. lots of dat-based projects! 52 | - Sam: From Cliqz, working on dat integration with browser. Playing with other dat related things, prototyped bookmark sync in browsers using kappa-core and discovery-swarm-webrtc 53 | - Mathias: Writing all the hyper stuff 54 | 55 | #### Mailing list 56 | 57 | - Pfrazee: Are we switching officially? 58 | - Mauve: I think we should switch, it's been hard to manage the CC'd list so far 59 | - Pfrazee: Agreed 60 | - Frando: I agree, I would propose to use the Dat Consortium list as the mailing forum for the group that's currently in these meetings. I propose we discuss who's in these meetings, for now I propose taking everyone in the list now and add them. Make everyone a moderator and discuss how membership works in these meetings. Comm comm list was created that was for everyone, people in the consortium and otherwise. 61 | - Pfrazee: +1 to using mailing list, should have agenda item for membership discussion 62 | - Mauve: +1 to agenda item, something that should be figured out. 63 | - Same: +1 to mailing list 64 | 65 | #### Summer event 66 | 67 | - Pfrazee: We should talk about the event and how to split up work. 68 | - cblgh: Is there a date set? 69 | - Pfrazee: No, but there's a document 70 | - Frando: We agree it should be soon. We should go and choose a date that makes sense and get it set for next meeting and have people raise last objections. 71 | - Pfrazee: We should designate a date picker 72 | - cblgh: I can put together a few proposals and send it on the mailing list 73 | - pfrazee: We should figure out how to split work 74 | - mauve: Who has done events before and what kind of work is there to split? 75 | - cblgh: What sort of size will we have? 200ish? 76 | - pfrazee: It's in the air 77 | - cblgh: Figure out if its unconforence style, figure out snacks, good to have it be vegan, cook the food or buy in food? It'd be good to have an ending gathering / dinner, should book a venue and consider how many people? This is outside of organizing travel, beaurcracy for flying people in and paying them, managing speakers, creating a poster / website / outward facing PR thing. Getting info out, food during the event, event stuff like who speeks when, schedule unconference or not, last dinner thing 78 | - sam: One thing missed is tech, if we want to film talks and livestream them, there's a lot of work to do there, figure out how to do the sound properly if we wanted to do that 79 | - cblgh: getting recordings out is so much harder, it usually sticks on someone's harddrive forever. Livestreaming is easier for shitty setups, but you need to have one person to actuallt get that done 80 | - mauve: WHo has time for this stuff? 81 | - cblgh: Yeah, depends on deciding on the format and when time would need to be. I'd be interested 82 | - diego: same as Alex, it'd be nice to know things before. I guess the location would be in EU, the help that I can offer would likely be remote 83 | - Frando: It depends on the format and timing, but I'm up to contributing some time. Depending on the location, as things are looking towards EU, in any case you'd need a few people that are at the place a few days earlier, and I can imagine being part of a local group, depends on the location 84 | - Kevin: THis is about the Dat event? I definately hope to attend, not sure how I could help, but would be intereested if I could be effective. 85 | - Martin: I'd love to help. I live in Japan so it might be hard if it's not in the summer like July/August, any online tasks or homepage stuff I can help 86 | - Mathias: Down to help, very busy, but I can help pick up topics and some practical stuff 87 | - Mauve: Down to put together a doc of stuff that needs to be done and coordinate people to commit to that stuff 88 | - Sam: If it's EU based, up for coming along before and helping with setup and teardown, can contribute to hosting, if it's in Munich, I can help looking at venues or catering and stuff like that as a local. 89 | - Pfrazee: Pretty busy right now, but up to show up early and help with on the ground setup 90 | 91 | #### CS&S integration 92 | 93 | - Ranger: last time JEss from CS&S was here to tell us we havent really been using CS&S this year, in part bc the dat foundation isnt a formal fiscally-sponsored project like the others. Karissa/ranger/jess have been discussing how to fix that. 94 | - Shared document giving overview of this: https://hackmd.io/yn9I9NPHSayCN11jhG4sMA?view 95 | - Ranger: What the formal relation is? Responsibilities, resources available. Wants to involve people in the consortium so they have direct access. Grants can be administered via CS&S, they take 15% in admin fees. All websites and branding and related IP is owned by CS&S and has to be retained by a non-profit. They are also interested in finding leads for grant funding. They also have folks who are good at event planning. 96 | - Ranger: it would also be good to give consortium members access to all digital resources. Basically looking to give decision-making and management powers to the members so that the project doesn't lose the ability to make progress. 97 | - Ranger: RE funding, not 100% clear what we have atm due to accounting software changes but it's somewhere around 10-20k. Need to decide how to use it: maybe apportion grants within the community? Maybe spend it on the dat event? 98 | 99 | ## Tech meeting notes 100 | 101 | - Mathias: There's been a lot of work going in. Performance, bug fixes. If you're using stuff update it sicne there's lots of updates. Working a tonne on this new flush API for all the network stacks. It's a problem we've had since day zero: Making it eaiser to program. For example where you have a hypercore and you want to query somthing on it, when do you know when it's queried. Like if you do an HTTP request and want to get thedata, on a p2p network it's harder. You have a DHT and peers, when do you consider yourself bootstrapped. There's been a lot of issues over time. Do you wait for peers to come, do you wait for handshakes. Been working on this in a new API pimitive on the swarm called flush. API hooks into discovery primitives. You can call a callback and when you call it all pending DHT queries have been flushed and all results have finished. This means you don't need timeouts. THis is powerful because if you want to set up a hyperdrive that if there's anyone on the network with new stuff you need, you can spin up a swarm, join the swarm, wait for the swarm to flush and after that you're guaranteed to be up to date, and you can do a readdir and know if there's anything new. That's something that will be good. 102 | - Pfrazee: This sounds like something I'd need for Beaker. You need this when you want to get the initial data on a page load 103 | - Mathias: You can do this more linearly. 104 | - Pfrazee: Is there a guarantee on the upper limit on how much time it will take? 105 | - Mathias: Worst case is waiting fro all the timeouts, worst case is 5sec for DHT, 10sec for connections. Worst case is 10 sec. We added a bunch of heuristics to work around it. For the DHT in practice worst case is 1 sec. If you're getting data you don't wait for long timeouts, at most wait like 1 sec. Added a bunch of advanced heuristics. Can bring it down to 5 sec in worst case, best case much much faster. Like 200 400 ms. It's really fast. It's a pretty big change that was added. Added an API to all methods, ifAvailable. If it's true it'll do flush. The call with return after a flush, or return with an error. If anyone on the network has the data, you'll get it. Been using this API to do a faster timeout. 106 | - Pfrazee: I assume if you have local data, it'll use that 107 | - Mathias: Yeah, if you have it locally it'll use that. THe tricky part is the update call. It'll check if anyone on the network has data. It's the only one that isn't idempotent. All the gets you either have it or not, but theupdate waits for the network 108 | - Frando: Sounds very interesting. How good will this operation scale for large sparse swarms? Say you have a core or drive that's popular, a few thousand people joining the sewarm. They all have the archive sparsely synched and you want to download some data. If I understand you'd need to connect to everyone and check. Is there somethign that would make that part easier. 109 | - Mathias: Flush doesn't affect that. All it does is wait for discovery work to finish. Doesn't solve the problem you're taling about. The swarm won't connect to more than 40 peers which are randomly distributed. That use case is the main one we're focusing on it. The main way we will make it better, using something called peer exhange form bittorrent. You can use an extension to ask a peer if they know anybody that has the block. But that's not really part of the flush. The flush is given the current setup, check if there are more peers comming right now. So if you have that gossip thing on top you can still wait for the swrm to flush, but it doesn't solve it right now. 110 | - Martin: QUestion: how can this be visaluzed, it'd be good if there was a constant or viarable to control the timeout. 111 | - Mathias: Yeah, it's a no brainer to expose the max waiting time. Paul mentioned the Beaker use case, you can load the current version and wait for a flush. There's still UX to figure out, before it was just setting a timer, now it's a bit better. On that note, another thing that's important for UX - yesterday we made an API for detecting networking health from local view. You can figure out if you have very good p2p capabilities, hole punchability. These APIs are bieng exposed in the swarm and are easier to use. If you want an app to see if p2p is gonna work or if oyu need a relay, you can use the hole punchable API, which is kinda cool. 112 | - Pfrazee: Another optimization was the trie hints extension. 113 | - Mathias: That's not released yet. We made a massive update for hypertrie with an extension. Helps with high latency connections. Once we have the release it'll reduce to one round trip for queries. It'll be huge if you're using folder listing. Potentially listing a big folder could use 10 sec, but with it it'd go down to 100ms. Basically no added cost, but withthe same security guarantees 114 | 115 | - Martin: Working on trying to get dat working in react-native, have resoruces to try to get help. Can get in touch with martin@consento.org. 116 | - Sam: What part do you want? 117 | - Martin: Networking and Hyperdrive. 118 | - Sam: Version? 119 | - Martin: I'd b happy with the new one or older one. Not sure how the general update plan for the while organization is. 120 | - Mathais: The complexity for the new and old one is the same 121 | - Sam: Main issue in RN is getting the bundler to understand all these node dependencies and to substitute the right ones when you don't have a native implementation. It'd be good to use nodejs-mobile, might need more for new Dat 122 | - Martin: Looking to sponsore that. Need to figure out what needs to be done. There's a nodejs-native thing we want to use, but not sure about it. 123 | -------------------------------------------------------------------------------- /meeting-notes/10-05March2020.md: -------------------------------------------------------------------------------- 1 | # Dat consortium meeting #10 2 | 3 | 2020/03/05 4 | 5 | ## Attending 6 | 7 | - Frando (facilitating) 8 | - Andrew Osheroff 9 | - Alex cblgh 10 | - Diego Paez 11 | - Jessica Hardwicke (CS&S) 12 | - Kevin Farboorg 13 | - Martin Heidegger 14 | - Mathias Buus 15 | - Ranger Mauve 16 | - Paul Frazee 17 | - Sam Macbeth 18 | 19 | ## Previous action items 20 | 21 | - Move to mailing list (done) 22 | - Look over the CS&S Dat document and leave comments https://hackmd.io/yn9I9NPHSayCN11jhG4sMA?view 23 | - Generate list of people interested in being a part of the advisory committee 24 | - cblgh is going to pick a date for the event based on the spreadsheet (https://docs.google.com/spreadsheets/d/1C05PNuP_IZy5k6p3IYRQ0_F65LjPTzli3MWTYwV2b08/edit#gid=0) 25 | 26 | ## Agenda 27 | 28 | - Lock down roles and responsibilities for the Dat event, discuss next steps in Event planning 29 | - Dates for Dat event, talk about thoughts regarding postponing due to COVID-19 30 | - Discuss how to move forward with a collective ownership of the relationship of Dat to CS&S 31 | - Discuss Dat Consortium membership 32 | 33 | 34 | ## Action items 35 | 36 | - cblgh: Put together poll for dates and event formats 37 | - pick which time intervals you can attend a physical event: https://www.strawpoll.me/19484840 38 | - pick what kind of event you would prefer: https://www.strawpoll.me/19484885 39 | 40 | ## Next agenda 41 | - 42 | - revisiting poll 43 | - decision on if there are any blockers to forming dat advisory committe under cs&s 44 | - talk about what is dat event 45 | 46 | ## Meeting notes 47 | 48 | ### Intros: 49 | 50 | - Franz: Working on Sonar, kappa-core 51 | - Andrew: Working on hyperdrive-daemon, working on documentation, working with paul on group software 52 | - cblgh: Involved with Cabal and Kappa-core. Released new build of cabal desktop which was broken. Fixed it and re-released. Recently merged more granular event emitters in the app. e.g. When a new message is received or peers added gets an event. Should enable more efficient clients. 53 | - Diego: Working on p2pcommons project 54 | - Jessica: Working with CS&S, sponsored project services manager. Here for questions about the advisory commitee and formalizing the relationship with CS&S in general 55 | - Kevin: Working on Ara.one, desktop app based on electron with Hyper modules and Etherium parts. Allows podcasters to get donations to their podcast without platforms defunding them or taking them down if they say something they didn't like, also artists don't have to pay 30% to spotify because it's more decentralized. Got on first exchange on Fridsay. LAtoken.com liquid assets token, trade etherium for Ara. Writing medium posts and tweeting. At SF, going to indieweb meeting at Mozilla. 56 | - Martin: Consento, private store. Encrypted that allows you to keep private data for yourself. Runs on React Native, doesn't connect to Dat at the moment. Working on that. 57 | - Mathias: Working on hyperdrive with Andrew. Working on same stuff Andrew mentioned. 58 | - Mauve: Finished contract with Dat FOundation, working on Natakanu - file transfer for Indiginous groups in Canada, also helping with Datdot which is kinda like Filecoin but without cryptocurrency and less central 59 | - Paul: Working on beaker browser, writing a lot of docs. Lots of bug fixing, aiming to bring somebody that's not on the team for groups thing 60 | - Sam: Put Dat integration into the CLiqz browser. Released another dat seeder library based on sams-dat-api, in addition to discovery-swarm, seeds to discovery-swarm-webrtc. Helps with better connections to networks. 61 | 62 | ### Event Planning 63 | 64 | - Frando: There's the TODO list we could go through. There's the question of the dates, and rules and responsabilities. Also talka bout coronavirus maybe postponing the event. 65 | - Frando: Start with Coronavirus question. 66 | - cblgh: Some background on it: Coronavirus has been on the news, lots of events have been cancelled, more countries are having it. It'd be good to raise the question. Do people have thoughts abou this. I'd be a question about outbreaks in Sweden or Germany or the US. Maybe we could go down the list. For me, I'm curious what people think. I'm thinking a later date would be better because we'd have more info as to how this progressed. A later date could be later in August, but no specific date has been proposed. 67 | - Paul: Maybe we should talk about doing a virtual conference instead if we're concerned about the virus? 68 | - Sam: Agree that this makes the later dates more appealing so we can see waht happens and has more time to cancel if things get worse. Reports seem to say that the summer will help with the spread. Later in the summer could help things with that. 69 | - Franz: Will go down the lsit 70 | - Andrew: I think a virtual conference would make sense. No other thoughts on that. 71 | - Diego: Thinking later in August could be reasonable to give us time. Saw somewhere that companies were thinking about starting again after June. Relates to the summer. 72 | - Jessica: Not sure why there was a reason to do earlier in the year. THink it'd be good to postpone not only for Coronavirus but also to do more fundraising and promotion. I organized an event and that took a lot of time in addition to locking the venue. More into in-person conferences, but also into the idea of people attending digitally 73 | - Kevin: Second later in person event or virtual event, both sound good. 74 | - Martin: Not sure if online is the best. I can watch webinars and there's youtube channels, it's not much different, not much different from browsing content. Getting additional value might be hard. Was going to go to a conference that got cancelled due to the coronavirus, was stressful. Regarding date: thinking that as horrible as it gets it'll get better. In favor of July/August because that's when I'll be in EU. Saying friends in Netherlands, if there's something in August they won't be able to participate because it's holiday time 75 | - Mathias: Doing it soon is out of the question, doing virtual is good. Summer isn't good for me because going on summer vacation, could go from there and see. 76 | - Mauve: More for in-person thing. Virtual conference would mean video calls and broadcast talsk - would miss out on a lot of in-person interaction that makes up a conf. or maybe VR? otherwise - august would look 77 | - Frando: Never did a virtual conference that's more than a virtual work meeting. Would miss the in person interaction a lot - makes most of the event for me. Still interested in having an actual event - if the virus prevents in spring, lets do it later. Not good at predicing public health, august is not a good time for EU folks, mid august people are usually out, july or september could be good times. In Germany only big big events are cancelled. Events with thousands are being cancelled, but smallers aren't. DOn't know much to do with public health though. 78 | - Frando: What should we do next? 79 | - Mauve: people that proposed could give details 80 | - Paul: Not much experience with virtual conf, but maybe something like: shared chatroom on irc as a shared lobby, then maybe twitch or youtube with a schedule of talks and workshops. plus website etc. kinda like a live code training session. you have a chat with an audicence so there's quite some interaction possible. makes it particularily nice for workshops. 81 | - Sam: Recently saw that there's several platforms offering this as a service (Virutal Conferences as a Service). Supports different formats: keynote talk where someone talks, chat channels that are like corridor tracks. It's a possability for a very lightweight conference that doesnt' need much organizing 82 | - cblgh: Has anyone attended a virtual conference that was good. Could see hw you'd combine live streaming with chat 83 | - Martin: Haven't attended one but will attend one on Friday, one person in the office I work at is building a platform for a paid online conference withe goal of getting paid entrance tickets for each session and only people that paid can access the streams giving it exclusivity 84 | - Frando: One thought I had regarding the dat event: Having not one conference but a bunch of small unconference style events. There could be actual meetups across the world where people get together for the virutal conference, not everyone is in their room, people that live together could have events. Peopel can get togheter but it's not just one big conference. 85 | - Mauve: Would be concerned about having just videos, best part is folks talking togehter. Like the idea of decentralied conference. Thinking about doing that in Ottawa and Montreal communities where they bridge together. VR is really nice because you can show slides, but also have people go off to the side and have little conversations. 86 | - Martin: I organized a NodeSchool internatioanl day in 2016. We tried to have one day where all nodeschools in the world would have an event, we had a video call session where all the differentl locations would invite people. large reach, in the end had like 2000 ppl attending in all locations together. was nice because it was all over the world. international connections. would open the possibility to have local people more activated. the local groups would get more visiblity. it lacked in good presentations though, also it was a lot of effort. if the concept were to be improved it would allow people from our community to have intro presentations. eg if mafintosh would give a 20min intro plus qa that might be something nice 87 | - Frando: To summarize: Lots of people still want physical meetups. There are positive experiences of virtual conferences. Maybe we should plan an event in Autumn and do a virtual event in June? 88 | - Sam: At the moment we have the 5 proposals that were sent, this discussion is saying maybe we shouldn't do the earlier ones. Two of the earlier ones did well for beginning and end of june. This discussion is saying June is maybe too early, there's also disadvantages in August - everyone in EU is on holiday 89 | - Mauve: Maybe we should have another poll? 90 | - Frando: WOuld someone want to put together a poll? 91 | - clbgh: Won't put as much in, made a poll for dates. Will make another one for conference types 92 | 93 | ### CS&S & Dat 94 | 95 | - Frando: The question about CS&S: The project would become a formal 96 | - Mauve: Some stuff: TODO get it from the transcript 97 | - Jessica: Regarding emails with Karissa and Mauve. A lot of projects make it pretty clear with a single leader, the teams are smaller usually too. Aprpeicate the challenge of the distributed aspect of Dat. Helped think about what CS&S expects from advisory comittees. Still working on laying out and giving clarity on why certain people need to be accountable for certain things. Reason we say you need a comitee chair is liability for spending excess funding or releasing funding. Related to liability and accountability. Establishing who to communicate with regarding questions about funding and bills for who to talk to. If there's a path forward we should amke it clear from the get go, make it possible to make changes and update for rotaing roles. If anyone has specific questions, ask me or email after the meeting. Our comittess at other projects are all different. SOme are small, some are large. SOme attend strategy meetings - all of the members. Some projects it's just me and the project manager. THere's a lot of different ways to do it for CS&S. 98 | - Sam: Chairman role question: Because they take liability is there requirement on having a US resident. 99 | - Jessica: No requirement for US resident. Regarding liability, less strict and legal sense of the word, more about who to talk to 100 | - Frando: Regardin mauve's question we should see who's interested in helping with the application, who wants to be a chair or member. First question who in this group is up to be a part of the advisory comittee. 101 | - Sam: One more question from the document. There's a stement that the advisory comittee has to have a portion that's less than a number of people being paid by dat project. So a 5050 split of people that aren't funded by Dat? 102 | - Jessica: The idea is that less than half can't be paid to be work on Dat. Wouldn't be too difficult since there aren't a lot of people being paid right now and not a lot of funds left. Just to make sure that the committee has some amount of oversight to the project. 103 | - Sam: happy to help with the advisory comittee, happy with chairman role or rotating chairman role. 104 | - Frando: Going to go through list and everyone can say if they're interested. 105 | - Martin: before we go through it, another question: RUnning the foundation and getting the connection to CS&S, I would like to hear from people who are most influential in the community mafintosh and paul, what the goal would be before peopel sign up 106 | - Paul: The question of the future of the foundation has a lot to do with how to organize the projects building on hypercore. Been starting to discuss. THinking of splitting so that hyperdrive is one project kappa core another and this is an umbrella. I think before we figure out what these are it'd be hard to know what the role would be and what the advisory comittee would do. 107 | - cblgh: Interested in where the conversation is, haven't sen in the dat channel 108 | - paul: Talked to Karissa, she was gonna talk to kappa-core folks. 109 | - Frando: Reguarding the question of the dat foundation, Karissa and Joe had the role of making the foundation a thing, but that doesn't reflect the situation. We talked about reforming around this consortium and using the foundation to get funding and manage funding and do conferences but not be involved in technical decisions and stuff like that. If I understand Paul correctly, we should figure this out before putting together the comittee. 110 | - Paul: Yes that's correct 111 | - Mauve: I'd like to discuss in the open instead of back channels 112 | - cblgh: Disagree and agree with Mauve. With people doing stuff, there's a question of "What is dat" and it'd be good to have a call of people that are interested. Summarize the results. That could be a way to go forward. 113 | - Frando: We could postpone the advisory comittee question for another week. Have a more public forum or public discussion where we discuss the question of the dat foundation as an umbrella for the hyper spec or something like that. 114 | - cblgh: also would be good to have a deadline for the advisory committe question so we dont punt it into infinity e.g. a decision will have been made by may 1 115 | - Sam: Not sure I see why this is a blocker for what's discussed in setting up the advisory comittee for the foundation. The foundation is mostly a funding vehicle for things in the ecosystem, it might get political for what gets funding and what does not. THis other decision is more what the consortium is for. 116 | - Martin: It's stressful if you don't know what you'd be mainting for. Worried about people taking responsability for something they don't quite understand. We should iron out relationships for projects. This is a position where dat is weak at. Stuff just happens, how things are built and what's prioritized grow naurally. FUnding is not a blocker in order to get the CS&S liason relationship. Do a poll get people and do a limited lifetime until we iron stuff out and go on 117 | - Mauve: I don't think the Foundation should be making decisions, more of a steward. For example if cabal wants 118 | - Frando: We should have another discussion. We should decide next facilitator and agenda items. 119 | - Mauve: We should talk about "What is Dat", revisit Advisory comittee, review results of polls 120 | - Frando: Who will facilitate next meting? 121 | - Sam: I can do it next week. 122 | 123 | --- 124 | 125 | ## Technical Meeting Notes 126 | 127 | - Pfraze: What's the high level latest of past two weeks 128 | - Mafintosh: Doing a push on performance. It's not been bad but want it to be better. Our requirements are - at the bottom, does the thing work, if yes, does it work okay, is it then fast? All of our time was going into "Does it work", now moving on to "make it faster". Andrew has been compiling benchmarks. Test cases for common patterns people have. "How long does it take to read x files" "what if you have x latency". In the new version of HYperdrive 10 - we changed the core data strucutre to be a HAMT, the old one was naive. This has the benefit of running at logN time. Even with log time if you have a lot of round trips it can be slow. Doing ticks with hypercore extesnions to do remote cache warmup. That means it's very fast now, found a bug in it, since it's an extesnion it doesn't matter, jsut a bit slower. Fixing that bug. Everything is looking really good. 129 | - Frando: Question: Is this extension directly in hyperdrive? 130 | - Mafintosh: It's in hypertrie, it's an extension for lookups. There's a filename called (??extension) that does warmups. Does cache stuff, auto enables. Feel free to shoot qustions my way, it's really cool stuff 131 | - Andrew: Mainly looking at performance the past week, stability has been good, haven't had miuch issues 132 | - Mafintosh: Going a few weeks without issues 133 | - Andrew: has been fun with Beaker - really useful for group applications. Colalborative group wiki, you can sync by pulling updates from members of the wiki. Surfaced issues with latency. Paul's in austin so the round trip is very noticiable. Channel open latency 134 | - Mafintosh: Spending too many round trips for the core connection to opening a hypercore channel. The number of round trips is 7 from channel to getting data. For subsequent it's only one. It's only initial latency. Speccing out ways to getting down to 2 or 3. If you have a roundtrip of 100 or 200 ms you can save a second. 135 | - Pfrazee: Initial load is 5 to 10 seconds. It'd be great to get down the time. Reliability has been great, haven't had to worry about connecting. Never get a flat out 404. 136 | - Mafintosh: Same, haven't had a connection fail yet, it works every time. 137 | - Andrewosh: The 5-10 latency is the first channel open for a peer since we're running with a peer across the globe, once we connect to paul and open a channel in the daemon any subsequent requests will be fast. 138 | - Mafintosh: For the latency work it's in hypercore. Main perf that's working quite well is amazingly fast ??? with the new updating things we've done. round tripping in a datacenter. General performance is really good. 139 | - Andrew: Killer thing that helped was the groups. 140 | - Pfrazee: Not on the call, but in the last dev diary 141 | - Andrew: That's really fun. Started informally - mounting each others drives - software that takes advantage of that and everybody mounting in a group drive makes it fun and useful. Hopefully will ahve a private beta soon to show that off. 142 | - Mafintosh: Looking in heavy handed garbage collection (deletes), looking into that a lot. Lots of data in the group drives, right now the strategy is after a while destroying from the hypercore without shutting down. Thinking of doing the GC inline, delete parts of the file. Since hypercore stores all the data in one virutal file, if you wanna unlink a file you need to unlink a portion of the file. We can eventually run on pretty low stortage reqs. 143 | - BUch of QA I didn't write down 144 | - Mafintosh: TL;DR, groups are cool. GOnna have a post soon. Best bug we've had where you would clone a drive and delete all your data and dump a copy of a book on how to make computers. Only deleted one of our coworkers data. 145 | 146 | 147 | 148 | 149 | 150 | ### Dat Event Dates 151 | Poll: https://www.strawpoll.me/19414155/r 152 | 153 | **Current top 3:** 154 | * proposal 5: a weekend in august. **8 votes (27%)** 155 | * proposal 3: in Berlin after jsconf EU (june 1-2 / mon-tue) so (thu) june 4 - (sun) jun 6. **7 votes (23%)** 156 | * proposal 2: last weekend of june, june 26 (fri) until (sun) june 28. **6 votes (20%)** 157 | 158 | TODO List 159 | https://hackmd.io/gcIN6U_XQm6ogmGoQ-EBeg 160 | 161 | -------------------------------------------------------------------------------- /meeting-notes/README.md: -------------------------------------------------------------------------------- 1 | # Dat Foundation Meeting Notes 2 | 3 | Here you'll find the notes for the Dat consortium meetings and some dev talks too. 4 | 5 | 6 | --------------------------------------------------------------------------------