├── README.md ├── SUMMARY.md ├── book.json ├── data.md ├── stacks.md ├── stacks ├── form-editors.md ├── frameworks.md ├── portals.md ├── sna.md └── stores.md └── ux.md /README.md: -------------------------------------------------------------------------------- 1 | # UF6 2 | 3 | Uranium Hexafluoride - Used for enrichment 4 | 5 | ![](http://upload.wikimedia.org/wikipedia/commons/thumb/9/94/Uranium_hexafluoride_crystals_sealed_in_an_ampoule.jpg/330px-Uranium_hexafluoride_crystals_sealed_in_an_ampoule.jpg) 6 | 7 | # What's inside? 8 | 9 | - [User Experience Patterns](ux.md) 10 | - [Data structures & Interoperability](data.md) 11 | - [Graph Stacks](stacks.md) 12 | 13 | # Problem 14 | 15 | Too many graphs, too many [frameworks](https://docs.google.com/a/iilab.org/spreadsheet/ccc?key=0AplklDf0nYxWdFhmTWZUc0o0SzAzMkRuMTZCUVBVeHc&usp=drive_web#gid=0), too many data structures, too many good intentions, not enough concrete technical interoperability. 16 | 17 | Developers need reusable components, data structures and practical advice on the way to Linked Open Data without getting lost in the complexity of it. (i.e. Linked Data for the shy developer) 18 | 19 | ## Problem solvers 20 | - [Pudo](http://granoproject.org/) / Grano [demo](http://beta.grano.cc/#/) - [source](https://github.com/granoproject) 21 | - [iilab](https://iilab.org) / Open Oil Corporate Network Navigator [demo](https://openoil.iilab.org) - [source](https://github.com/iilab/openoil) 22 | - ... (Submit a Pull Request or send a message to uf6@iilab.org to be added) 23 | 24 | ### Others to get on board 25 | - [Journalism++](http://www.jplusplus.org/en/) / Detective.io - [demo](http://www.detective.io) - [source](https://github.com/jplusplus/detective.io) @jplusplus @n-kb 26 | - [Shidash](https://github.com/shidash) / Transparency Toolkit - [demo](http://transparencytoolkit.org/) - [source](https://github.com/TransparencyToolkit/Transparency-Toolkit) @shidash 27 | - [Atchai](http://www.atchai.com/) / NewsVerify [demo](http://opennewsplatform.atchai.com/) - [documentation](https://github.com/atchai/newsverify/wiki) - [source](https://github.com/atchai/newsverify) / Dataseed - [demo](https://getdataseed.com/visualise/mortality/1) - [documentation](https://getdataseed.com/documentation) @atchai @johngriffin 28 | - Open Corporates (https://opencorporates.com @sebbacon) 29 | - Parltrak (@stef @tttp) 30 | - @elf-pavlik 31 | - Storyful (@storyful @paulmwatson) 32 | - MC 33 | - Data Incubators (@timrdf) 34 | - ... (Submit a Pull Request with your name) 35 | 36 | # Solutions 37 | 38 | Find solutions by openly designing interoperability approaches such as: 39 | - Data enrichment in practice (our own user centric because its good to try and share about the hurdles) 40 | - How do I import CSV from Open Oil into Detective.io ? ## Add issue 41 | - How do I use Transparency Toolkit scrapers inside Grano? 42 | - How do I use the natural query display of Dataseed into the Open Oil Corporate Structure Navigator? 43 | - How do I [add Industries to MEP's declarations of interest.](https://twitter.com/eucampaign/status/491276416605253632) 44 | - Experimenting with user experience patterns 45 | - How to make query building natural for investigators? 46 | - How can patterns be found in the graph? 47 | - When are tables best, when are graphs best, when are they both needed, in which proportion?... 48 | - Identify opportunities to use common data structures 49 | - Provenance 50 | - Reputation 51 | - ... 52 | - Keep track of computer science development that can be applied to real world applications 53 | - Distributed Stores / Federated Queries 54 | - UI description layers 55 | - ... 56 | -------------------------------------------------------------------------------- /SUMMARY.md: -------------------------------------------------------------------------------- 1 | # Summary 2 | 3 | * Introduction 4 | 5 | -------------------------------------------------------------------------------- /book.json: -------------------------------------------------------------------------------- 1 | {} -------------------------------------------------------------------------------- /data.md: -------------------------------------------------------------------------------- 1 | # Data structures 2 | 3 | We'd like standards on data that allow simple and meaningful reuse of data such as: 4 | - Dealing with provenance 5 | - Deals with key entities like organisations and people 6 | 7 | Ideally there would be different levels of implementatin which would allow: 8 | - Minimum: very quick fixes that can be made to allow data sets to interoperate. 9 | - Core: the solid way to make data enriching predictable and repeatable 10 | - Extended: the cherry on top. 11 | 12 | ### Quality 13 | 14 | - [OpenRefine](http://openrefine.org/) 15 | - Exversion's [Junky](https://github.com/exversion/junky) 16 | - [Data Cleaning Boilerplate](https://github.com/exversion/data-cleaning-boilerplate) 17 | 18 | ### Provenance 19 | - __Preferred:__ [Provenance Ontology]( 20 | 21 | 22 | ![](http://www.w3.org/TR/2013/NOTE-prov-primer-20130430/images/key-concepts.png) 23 | 24 | - __Deprecated:__ [Open Provenance (Ontology)](http://openprovenance.org/model/opmx#example) 25 | - ICFJ provenance 26 | 27 | ### Verification 28 | 29 | ### Reputation 30 | 31 | ### Organisations/Institutions 32 | 33 | * __Preferred:__ Organisations ([Organisation Ontology](http://www.w3.org/TR/vocab-org/)) 34 | 35 | ![](http://www.w3.org/TR/vocab-org/img/OrgOntology20130502.png) 36 | 37 | * [Popolo for People/Organizations/Membership](http://popoloproject.com/) 38 | * 39 | 40 | ### Organisation Types 41 | 42 | * [Schema.org Organisation subclasses](http://schema.org/Organization) 43 | * [OpenSecrets Industries](https://www.opensecrets.org/industries/slist.php), based off [NAICS](http://www.census.gov/eos/www/naics/), related to [NACE (Europe)](http://epp.eurostat.ec.europa.eu/cache/ITY_OFFPUB/KS-RA-07-015/EN/KS-RA-07-015-EN.PDF) 44 | 45 | ### Company Equity/Control 46 | 47 | * [equity/control/executives](http://www.omg.org/spec/EDMC-FIBO/BE/1.0/Beta1/), 48 | 49 | ### Contracts 50 | 51 | * [contracts](http://www.omg.org/spec/EDMC-FIBO/FND/1.0/Beta1/) 52 | * [public contracts](https://code.google.com/p/public-contracts-ontology/) 53 | * [Open Contracting Data Standard](http://standard.open-contracting.org/) 54 | - Open Spending? 55 | * There's also a lot of financial taxonomies that might become relevant, check the [XBRL Taxonomies](http://www.xbrl.org/Taxonomies/), which are XML schema for things including standard SEC and UK accounting charts. 56 | -------------------------------------------------------------------------------- /stacks.md: -------------------------------------------------------------------------------- 1 | # Stacks 2 | 3 | * [Stores](stacks/stores.md) 4 | * [Social Network Analysis](stacks/sna.md) 5 | * [Open Data Publishing Frameworks](stacks/frameworks.md) 6 | * [Open Data Portals](stacks/portals.md) 7 | 8 | 9 | -------------------------------------------------------------------------------- /stacks/form-editors.md: -------------------------------------------------------------------------------- 1 | # Form editors 2 | 3 | | Project name | Home page | Language/Frameworks | Schema Format | Representation Format | Documentation | Open Code | Repository | Examples | Form Builder | | | 4 | |--------------------|-------------------------------------------|--------------------------|---------------|------------------------------|--------------------------------------------|-----------|--------------------------------------------|---------------------------------------|--------------------------------------------------------------|---|---| 5 | | AlpacaJS | http://www.alpacajs.org/ | Javascript | JSON Schema | Proprietary options object | http://www.alpacajs.org/documentation.html | Yes | https://github.com/gitana/alpaca | http://www.alpacajs.org/examples.html | http://www.alpacajs.org/demos/form-builder/form-builder.html | | | 6 | | JSON-Editor | http://jeremydorn.com/json-editor/ | | JSON Schema | format (HTML5) + template | http://marmotta.apache.org/start.html | Yes | https://github.com/jdorn/json-editor | http://jeremydorn.com/json-editor/ | http://jeremydorn.com/json-editor/ | | | 7 | | JSON-Edit | http://marianoguerra.github.io/json-edit/ | | JSON Schema | | ? | Yes | https://github.com/marianoguerra/json-edit | | http://marianoguerra.github.io/json-edit/ | | | 8 | | JSON Form | https://github.com/joshfire/jsonform | Javascript / Bootstrap | ~ JSON Schema | [Proprietary form object][1] | ? | Yes | https://github.com/joshfire/jsonform | | | | | 9 | | Onde | http://exavolt.github.io/onde/ | Javascript | JSON Schema | | ? | Yes | https://github.com/exavolt/onde | | | | | 10 | | VIE Form Generator | http://viejs.org/widgets/forms/ | Javascript / Backbone.js | JSON-LD? | Framework | ? | Yes | https://github.com/marianoguerra/json-edit | | | | | 11 | | Semantic Forms | https://github.com/jmvanel/semantic_forms | | RDF? | Framework | ? | Yes | https://github.com/jmvanel/semantic_forms | | | | | 12 | 13 | [1]: https://github.com/joshfire/jsonform/wiki#controlling-the-layout-of-the-form "Documentation about JSON Form layout" 14 | 15 | # Sources 16 | 17 | http://stackoverflow.com/questions/998832/gui-based-or-web-based-json-editor-that-works-like-property-explorer 18 | https://github.com/dreftymac/stackoverflow.questions.998832 19 | -------------------------------------------------------------------------------- /stacks/frameworks.md: -------------------------------------------------------------------------------- 1 | # Frameworks 2 | 3 | | Project name | Type | Description | Home page | Documentation | Open Code | Repository | Framework | ORM | Store | 4 | |-----------------|-----------------|---------------------------------|-------------------------------------|---------------------------------------|-----------|-------------------------------------------|----------------|--------|--------------------------| 5 | | Publish my data | Platform / Tool | Linked Data Publishing software | http://www.swirrl.com/publishmydata | ? | Yes | https://github.com/swirrl/publish_my_data | Rails | Tripod | Fuseki | 6 | | Marmotta | Framework | Linked Data Publishing | http://marmotta.apache.org/ | http://marmotta.apache.org/start.html | Yes | https://code.google.com/p/lmf/ | Java | | PostgreSQL, MySQL and H2 | 7 | | FLOD | Tool | Framework | http://flod.info/home | http://flod.info/tutorial | Yes | https://github.com/alangrafu/flod | Python / Flask | | SPARQL | | | | | 8 | -------------------------------------------------------------------------------- /stacks/portals.md: -------------------------------------------------------------------------------- 1 | # Portals 2 | 3 | | Project name | Type | Description | Home page | Documentation | Open Code | Repository | Framework | ORM | Store | 4 | |--------------|--------|-------------------------------------------------------------------------------------------|--------------------------|--------------------------------------------|-----------|---------------------------------|-----------------|------------|------------| 5 | | CKAN | Tool | Open Data Catalog Publishing software | http://ckan.org/ | http://docs.ckan.org/en/latest/index.html | Yes | https://github.com/ckan/ckan | Python / Pylons | SQLAlchemy | Postgresql | 6 | | DKAN | Plugin | Open data platform with a full suite of cataloging, publishing and visualization features | http://nucivic.com/dkan/ | http://docs.getdkan.com/dkan-documentation | Yes | https://github.com/NuCivic/dkan | Drupal | N/A | SQL | 7 | 8 | -------------------------------------------------------------------------------- /stacks/sna.md: -------------------------------------------------------------------------------- 1 | # Social Network Analysis projects 2 | 3 | Source URL : https://docs.google.com/spreadsheet/ccc?key=0AplklDf0nYxWdFhmTWZUc0o0SzAzMkRuMTZCUVBVeHc&usp=drive_web#gid=0 4 | 5 | | Project name | Type | Description | Home page | Documentation | Open Content | Open Code | Platform | Repository | Tech | Team | Team URL | 6 | |--------------------------------------------------------------------------|---------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------|-------------------------------------------------------------------------------------------------|---------------------|-------------------------------------------|-----------------------|---------------------------------------------------------------------------------------|--------------------------------|------------------------------------------------|----------------------------------------| 7 | | Poderopedia | platform | Maps out politicians, business and others in Chile, and other countries. | http://poderopedia.org/ | https://github.com/poderopedia/plug-and-play-1.0-RC/tree/master/Documentation | Yes (CC-BY) | Yes (GPL) | web | https://github.com/poderopedia/plug-and-play-1.0-RC | Python, web2py, Postgres | Miguel Paz, Juan Eduardo Hernández | | 8 | | PoderVocab | data standard | Ontology for Poderopedia | | dev.poderopedia.com/documentation/index | n/a | Yes (GPL) | - | https://github.com/poderopedia/PoderVocabulary | RDF Ontology | Miguel Paz, Juan Eduardo Hernández | | 9 | | Connected China | news app | Leadership structure of PRC visualized | http://connectedchina.reuters.com/ | http://blogs.reuters.com/connected-china/ | No | No | ipad | | | Reuters | http://reuters.com | 10 | | The Proxy Platform | news app | Illegal holdings in Eastern Europe | https://www.reportingproject.net/proxy/en/ | | No | No | web (flash) | | | OCCRP | https://www.reportingproject.net/ | 11 | | Rise Project | platform | Who is who of organized crime and business in Romania | http://www.riseproject.ro/ | | No | No | web | | | OCCRP | https://www.reportingproject.net/ | 12 | | LittleSis | platform | People and organisations in US politics and business | http://littlesis.org | http://littlesis.org/about | Yes (CC-BY-SA) | Yes (GPL) | web | https://github.com/littlesis-org/littlesis | PHP, Symfony, MySQL | Public Accountability Initiative | http://public-accountability.org/ | 13 | | Sigma.js | library | JS library for rendering network diagrams | http://sigmajs.org/ | http://sigmajs.org/examples.html | Yes (MIT) | Yes (MIT) | web | https://github.com/jacomyal/sigma.js | JS | Alexis Jacomy / Sciences-Po MédiaLab | | 14 | | They Rule | platform | Visualization of LittleSis data | http://www.theyrule.net/ | http://www.theyrule.net/about | Yes (CC-BY-SA) | No | web (flash) | | | Josh On | | 15 | | Muckety | platform | Mapping connections of the rich, famous & influential | http://news.muckety.com/ | | No | No | web (flash) | | | Muckety LLC | | 16 | | The Popolo project | data standard | Data standard for people, organisations, roles | http://popoloproject.com/ | http://popoloproject.com/specs/ | Yes (CC-BY) | n/a | - | https://github.com/opennorth/popolo-spec | RDF, JSON | James McKinney et al. / OpenNorth | http://opennorth.ca/ | 17 | | PopIt | tool | Reference data server for people in PMO settings | http://popit.mysociety.org/ | http://popit.mysociety.org/docs | n/a | Yes (AGPL) | web | https://github.com/mysociety/popit | Node, Ruby, MongoDB | MySociety | http://www.mysociety.org/ | 18 | | Mapa76 / analice.me | tool | Document entity extraction for large datasets | http://mapa76.info/ | | n/a | Yes (MIT) | web | https://github.com/hhba/mapa76 | Ruby, MongoDB, Redis | Hacks/Hackers Buenos Aires | http://hhba.info/ | 19 | | i2 Analysts Notebook | tool | Fancy commercial network analysis application | http://www-03.ibm.com/software/products/us/en/analysts-notebook/ | http://public.dhe.ibm.com/common/ssi/ecm/en/zzd03127usen/ZZD03127USEN.PDF | No | No | desktop | - | | IBM | http://ibm.com | 20 | | Gephi | tool | Graph analysis and visualization desktop tool | https://gephi.org/ | http://wiki.gephi.org/index.php/Main_Page | n/a | Yes (GPL) | desktop | https://github.com/gephi/gephi | Java, NetBeans | Gephi Consortium | | 21 | | NodeXL | tool | Excel plugin for social network analysis | http://nodexl.codeplex.com/ | http://nodexl.codeplex.com/documentation | n/a | Yes (MS-Pl) | desktop | http://nodexl.codeplex.com/SourceControl/latest | C++, Excel | | | 22 | | Immersion | news app | Visualization of mailbox interactions graph | https://immersion.media.mit.edu/ | | No | No | web | | | MIT Media Lab | http://macroconnections.media.mit.edu/ | 23 | | Investigative Dashboard | platform | helps investigators expose illicit ties that cross borders | http://www.investigativedashboard.org/ | | No | No | web | | | OCCRP | https://www.reportingproject.net/ | 24 | | DueDil | platform | Network graphs of Company House data | https://www.duedil.com/ | | No | No | web | | | DueDil Ltd. | | 25 | | Timelines | library | GitHub-style collaboration graphs | http://timelines.hhba.info/ | | No | No | web | https://github.com/hhba/timelines | JS | Hacks/Hackers Buenos Aires | http://hhba.info/ | 26 | | OpenCorporates | platform | Company data with ownership network structures | http://opencorporates.com/ | http://blog.opencorporates.com/2013/07/11/open-corporate-network-data-not-just-good-but-better/ | Yes (ODbL, no bulk) | No | web | | Ruby, Postgres, Solr | Chris Taggart, Seb Bacon et al. | | 27 | | Warren | tool | Facebook for terrorists | | | n/a | No | web | | Ruby, Neo4J | Jacqui Maher / NYTimes | | 28 | | Grano | tool | Facebook-like activity feeds for entities | | | n/a | Yes (MIT) | web | http://github.com/pudo/grano | Python, MongoDB, ElasticSearch | Friedrich Lindenberg | http://pudo.org | 29 | | tx_people | library | Django module for Popolo support | | | n/a | Yes (Apache) | web | https://github.com/texas/tx_people | Python, Django | Travis Swicegood / Texas Tribune | http://texastribune.com | 30 | | detective.io | platform | I take the liberty to revive it to shamelessly announce our new product, Detective.io :) We were working last May on an investigation to map all innovative energy projects in developing countries and couldn't find a proper way to store the data. We tried Poderopedia Plug 'n Play but needed more flexibility in the ontology (we had energy projects and products, for instance, that didn't fit in the default one). We therefore went ahead and designed our own product using Django and Neo4j. The key feature is that any OWL ontology can be fed to the platform. It then creates a new "investigation"" | i.e. a | | | | | | | | | 31 | | contribution interface to input data as well as a dedicated front-end. | | | | | | | | | | | | 32 | | We don't have network visualizations yet but make use of the Neo4j magic | | | | | | | | | | | | 33 | | with a graph search feature" | http://detective.io | | n/a | Yes (GPL) | web | https://github.com/jplusplus/detective.io | Python, Django, Neo4J | Journalism++ | http://jplusplus.org | | | 34 | | Quien Manda | platform | People and organisations in Spanish politics and business | http://quienmanda.es/ | | Yes (CC-BY) | Yes (AGPL) | web | https://github.com/civio/quienmanda.es | Ruby, Postgres | David Cabo (Civio, Sunlight, Global Integrity) | http://www.civio.es/ | 35 | | UCINet | tool | Scientific desktop social network analysis tool | https://sites.google.com/site/ucinetsoftware/ | https://sites.google.com/site/ucinetsoftware/document | n/a | No | desktop | https://sites.google.com/site/ucinetsoftware/downloads | Win32 | Lin Freeman, Martin Everett and Steve Borgatti | | 36 | | Untangled | tool | tbd | http://knightlab.northwestern.edu/tag/untangled/ | | n/a | n/a | ? | | | Rich Gordon, Jessica Soberman (NWU Knight Lab) | http://knightlab.northwestern.edu/ | 37 | | VIS (Visual Investigative Scenarios) | platform | (VIS) is a data visualization platform designed to assist investigative journalists, activists and others in mapping complex business or crime networks. | http://vis.occrp.org/ | | no | no | web | | PHP | OCCRP | http://reportingproject.org | 38 | | Graph Commons | platform | Graph Commons is a collaborative 'network mapping' platform and a knowledge base of relationships. You can create, navigate, and distribute network maps, learn about the relationships between organizations, people, and concepts. | http://graphcommons.com/ | http://graphcommons.com/about | no | no | web | | | Burak Arıkan | | 39 | | Network Mapper | tool | Mapping out and visualizing the connections between people and groups can dramatically improve our understanding of the affiliations and key nodes of influence between them. | http://www.google.com/ideas/projects/network-mapper/ | | no | no | web | https://docs.google.com/forms/d/13xhh7Zr12944zkcEVKTaNrMDPBO7SGJXrxsmz8p9kzo/viewform | | | | 40 | | OpenDuka | platform | | | | yes | yes | web | | | | | 41 | | Kumu.io | platform | Great datavis / node editing tool | http://kumu.io/ | | no | no | web | | | | | 42 | | OpenOil | tool | Oil contracts explorer | https://data.openoil.net/ | | no | yes | web | | | | -------------------------------------------------------------------------------- /stacks/stores.md: -------------------------------------------------------------------------------- 1 | # Stores 2 | 3 | 4 | Store | Open Source | Triples | Quads | Distributed | Install | Performance | Irrational fear | MQL | Mature | Ecosystem | Stacks 5 | ---------------- | ---------------- | --------- | ---------| -------------- | -------- | ------------------| ------------------| ------- | --------- | --------------- | --------- 6 | **Virtuoso** | Yes | Yes | Yes | No | Not great | Very good | Yes | No? | Yes? | Yes | ? 7 | **Neo4j** | Yes | No | No | No | Ok | Good | A little | No | Kinda | Yes | Fair amounts including Structr 8 | **Cayley** | Yes | Yes | Yes | No | Easy | ? | No | Some | No | Some momentum | [Bindings](https://github.com/google/cayley/wiki/Client-APIs) 9 | **Titan** | Yes | Yes | Yes | Yes | In progress | Porbably high | No | No but Gremlin | No? | Tinkerpop | Tinkerpop 10 | **OrientDB**| Yes | Yes | ? | Yes | Easy... Sort of.. | Good on paper | No | No | Sort of? | Yes | [Lots](https://github.com/orientechnologies/orientdb/wiki/Programming-Language-Bindings) 11 | **Dydra** | No | Yes | Yes | No | Easy | High | Of closed source | No | Seems like it | Is there? | [Ruby](http://docs.dydra.com/sdk/ruby) 12 | **Jena** | Yes | Yes | Yes | No | Easy | High | No | No | Yes | Yes | SPARQL/HTTP 13 | -------------------------------------------------------------------------------- /ux.md: -------------------------------------------------------------------------------- 1 | # User experience patterns 2 | 3 | This is a relatively new area and a number of approaches should be tested by allowing several components to be easily reused/composed/sequenced/layed out in different ways. 4 | 5 | Example use cases: 6 | - Add table interface to the Open Oil corporate network navigator 7 | - Add a graph visualisation component to grano-ql 8 | - ... 9 | 10 | Links: 11 | - [Rough thoughts from iilab about navigating/querying/investigating graphs](https://www.penflip.com/jun/iilab-graph/blob/master/graph-interfaces.txt) 12 | - ... 13 | 14 | ## Components 15 | 16 | This is a first attempt, using [Web Components](http://webcomponents.org/) / [Polymer](http://polymer-project.org/), to design abstract components that could be using different implementions, interaction patterns, be displayed/hidden... 17 | 18 | Other implementation following the same patterns could co-exist (in Angular, Meteor,...) 19 | 20 | ``` 21 | 22 | 32 | 33 | 43 | 44 | 51 | 52 | 58 | 59 | 65 | 66 | 72 | 73 | 74 | 75 | 76 | 77 | 78 | 79 | 80 | ``` --------------------------------------------------------------------------------