├── LICENSE ├── README.md ├── miro.md └── supplementary ├── README.md ├── analysis ├── jbms_coded_free_form.csv ├── jbms_labels.csv ├── jbms_ontology_survey.Rmd ├── jbms_paper_results.csv ├── jbms_results.csv └── rating_v_compliance.csv ├── questionnaire_comment_coding.csv ├── questionnaire_email_lists.csv ├── questionnaire_responses.csv ├── review_abstracts.csv ├── review_coding_scheme.csv ├── review_extraction.csv ├── review_papers_in_timeframe.csv ├── review_selected_papers.csv └── review_titles.csv /LICENSE: -------------------------------------------------------------------------------- 1 | Apache License 2 | Version 2.0, January 2004 3 | http://www.apache.org/licenses/ 4 | 5 | TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 6 | 7 | 1. Definitions. 8 | 9 | "License" shall mean the terms and conditions for use, reproduction, 10 | and distribution as defined by Sections 1 through 9 of this document. 11 | 12 | "Licensor" shall mean the copyright owner or entity authorized by 13 | the copyright owner that is granting the License. 14 | 15 | "Legal Entity" shall mean the union of the acting entity and all 16 | other entities that control, are controlled by, or are under common 17 | control with that entity. For the purposes of this definition, 18 | "control" means (i) the power, direct or indirect, to cause the 19 | direction or management of such entity, whether by contract or 20 | otherwise, or (ii) ownership of fifty percent (50%) or more of the 21 | outstanding shares, or (iii) beneficial ownership of such entity. 22 | 23 | "You" (or "Your") shall mean an individual or Legal Entity 24 | exercising permissions granted by this License. 25 | 26 | "Source" form shall mean the preferred form for making modifications, 27 | including but not limited to software source code, documentation 28 | source, and configuration files. 29 | 30 | "Object" form shall mean any form resulting from mechanical 31 | transformation or translation of a Source form, including but 32 | not limited to compiled object code, generated documentation, 33 | and conversions to other media types. 34 | 35 | "Work" shall mean the work of authorship, whether in Source or 36 | Object form, made available under the License, as indicated by a 37 | copyright notice that is included in or attached to the work 38 | (an example is provided in the Appendix below). 39 | 40 | "Derivative Works" shall mean any work, whether in Source or Object 41 | form, that is based on (or derived from) the Work and for which the 42 | editorial revisions, annotations, elaborations, or other modifications 43 | represent, as a whole, an original work of authorship. For the purposes 44 | of this License, Derivative Works shall not include works that remain 45 | separable from, or merely link (or bind by name) to the interfaces of, 46 | the Work and Derivative Works thereof. 47 | 48 | "Contribution" shall mean any work of authorship, including 49 | the original version of the Work and any modifications or additions 50 | to that Work or Derivative Works thereof, that is intentionally 51 | submitted to Licensor for inclusion in the Work by the copyright owner 52 | or by an individual or Legal Entity authorized to submit on behalf of 53 | the copyright owner. For the purposes of this definition, "submitted" 54 | means any form of electronic, verbal, or written communication sent 55 | to the Licensor or its representatives, including but not limited to 56 | communication on electronic mailing lists, source code control systems, 57 | and issue tracking systems that are managed by, or on behalf of, the 58 | Licensor for the purpose of discussing and improving the Work, but 59 | excluding communication that is conspicuously marked or otherwise 60 | designated in writing by the copyright owner as "Not a Contribution." 61 | 62 | "Contributor" shall mean Licensor and any individual or Legal Entity 63 | on behalf of whom a Contribution has been received by Licensor and 64 | subsequently incorporated within the Work. 65 | 66 | 2. Grant of Copyright License. Subject to the terms and conditions of 67 | this License, each Contributor hereby grants to You a perpetual, 68 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 69 | copyright license to reproduce, prepare Derivative Works of, 70 | publicly display, publicly perform, sublicense, and distribute the 71 | Work and such Derivative Works in Source or Object form. 72 | 73 | 3. Grant of Patent License. Subject to the terms and conditions of 74 | this License, each Contributor hereby grants to You a perpetual, 75 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 76 | (except as stated in this section) patent license to make, have made, 77 | use, offer to sell, sell, import, and otherwise transfer the Work, 78 | where such license applies only to those patent claims licensable 79 | by such Contributor that are necessarily infringed by their 80 | Contribution(s) alone or by combination of their Contribution(s) 81 | with the Work to which such Contribution(s) was submitted. If You 82 | institute patent litigation against any entity (including a 83 | cross-claim or counterclaim in a lawsuit) alleging that the Work 84 | or a Contribution incorporated within the Work constitutes direct 85 | or contributory patent infringement, then any patent licenses 86 | granted to You under this License for that Work shall terminate 87 | as of the date such litigation is filed. 88 | 89 | 4. Redistribution. You may reproduce and distribute copies of the 90 | Work or Derivative Works thereof in any medium, with or without 91 | modifications, and in Source or Object form, provided that You 92 | meet the following conditions: 93 | 94 | (a) You must give any other recipients of the Work or 95 | Derivative Works a copy of this License; and 96 | 97 | (b) You must cause any modified files to carry prominent notices 98 | stating that You changed the files; and 99 | 100 | (c) You must retain, in the Source form of any Derivative Works 101 | that You distribute, all copyright, patent, trademark, and 102 | attribution notices from the Source form of the Work, 103 | excluding those notices that do not pertain to any part of 104 | the Derivative Works; and 105 | 106 | (d) If the Work includes a "NOTICE" text file as part of its 107 | distribution, then any Derivative Works that You distribute must 108 | include a readable copy of the attribution notices contained 109 | within such NOTICE file, excluding those notices that do not 110 | pertain to any part of the Derivative Works, in at least one 111 | of the following places: within a NOTICE text file distributed 112 | as part of the Derivative Works; within the Source form or 113 | documentation, if provided along with the Derivative Works; or, 114 | within a display generated by the Derivative Works, if and 115 | wherever such third-party notices normally appear. The contents 116 | of the NOTICE file are for informational purposes only and 117 | do not modify the License. You may add Your own attribution 118 | notices within Derivative Works that You distribute, alongside 119 | or as an addendum to the NOTICE text from the Work, provided 120 | that such additional attribution notices cannot be construed 121 | as modifying the License. 122 | 123 | You may add Your own copyright statement to Your modifications and 124 | may provide additional or different license terms and conditions 125 | for use, reproduction, or distribution of Your modifications, or 126 | for any such Derivative Works as a whole, provided Your use, 127 | reproduction, and distribution of the Work otherwise complies with 128 | the conditions stated in this License. 129 | 130 | 5. Submission of Contributions. Unless You explicitly state otherwise, 131 | any Contribution intentionally submitted for inclusion in the Work 132 | by You to the Licensor shall be under the terms and conditions of 133 | this License, without any additional terms or conditions. 134 | Notwithstanding the above, nothing herein shall supersede or modify 135 | the terms of any separate license agreement you may have executed 136 | with Licensor regarding such Contributions. 137 | 138 | 6. Trademarks. This License does not grant permission to use the trade 139 | names, trademarks, service marks, or product names of the Licensor, 140 | except as required for reasonable and customary use in describing the 141 | origin of the Work and reproducing the content of the NOTICE file. 142 | 143 | 7. Disclaimer of Warranty. Unless required by applicable law or 144 | agreed to in writing, Licensor provides the Work (and each 145 | Contributor provides its Contributions) on an "AS IS" BASIS, 146 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 147 | implied, including, without limitation, any warranties or conditions 148 | of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A 149 | PARTICULAR PURPOSE. You are solely responsible for determining the 150 | appropriateness of using or redistributing the Work and assume any 151 | risks associated with Your exercise of permissions under this License. 152 | 153 | 8. Limitation of Liability. In no event and under no legal theory, 154 | whether in tort (including negligence), contract, or otherwise, 155 | unless required by applicable law (such as deliberate and grossly 156 | negligent acts) or agreed to in writing, shall any Contributor be 157 | liable to You for damages, including any direct, indirect, special, 158 | incidental, or consequential damages of any character arising as a 159 | result of this License or out of the use or inability to use the 160 | Work (including but not limited to damages for loss of goodwill, 161 | work stoppage, computer failure or malfunction, or any and all 162 | other commercial damages or losses), even if such Contributor 163 | has been advised of the possibility of such damages. 164 | 165 | 9. Accepting Warranty or Additional Liability. While redistributing 166 | the Work or Derivative Works thereof, You may choose to offer, 167 | and charge a fee for, acceptance of support, warranty, indemnity, 168 | or other liability obligations and/or rights consistent with this 169 | License. However, in accepting such obligations, You may act only 170 | on Your own behalf and on Your sole responsibility, not on behalf 171 | of any other Contributor, and only if You agree to indemnify, 172 | defend, and hold each Contributor harmless for any liability 173 | incurred by, or claims asserted against, such Contributor by reason 174 | of your accepting any such warranty or additional liability. 175 | 176 | END OF TERMS AND CONDITIONS 177 | 178 | APPENDIX: How to apply the Apache License to your work. 179 | 180 | To apply the Apache License to your work, attach the following 181 | boilerplate notice, with the fields enclosed by brackets "[]" 182 | replaced with your own identifying information. (Don't include 183 | the brackets!) The text should be enclosed in the appropriate 184 | comment syntax for the file format. We also recommend that a 185 | file or class name and description of purpose be included on the 186 | same "printed page" as the copyright notice for easier 187 | identification within third-party archives. 188 | 189 | Copyright [yyyy] [name of copyright owner] 190 | 191 | Licensed under the Apache License, Version 2.0 (the "License"); 192 | you may not use this file except in compliance with the License. 193 | You may obtain a copy of the License at 194 | 195 | http://www.apache.org/licenses/LICENSE-2.0 196 | 197 | Unless required by applicable law or agreed to in writing, software 198 | distributed under the License is distributed on an "AS IS" BASIS, 199 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 200 | See the License for the specific language governing permissions and 201 | limitations under the License. 202 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # MIRO – Minimum Information for Reporting of an Ontology 2 | 3 | The MIRO guidelines are a community-validated set of recommendations on what should be reported about an ontology and its development, most importantly in the context of ontology description papers intended for publishing in scientific journals or conferences. 4 | 5 | The detailed, most up to date [MIRO guidelines](https://github.com/owlcs/miro/blob/master/miro.md "MIRO guidelines") are maintained on GitHub. 6 | 7 | @nicolevasilevsky is maintaining a [checklist on Google Docs](https://docs.google.com/spreadsheets/d/1_oWgZJMDYSaqR84z_MlSCv7xit0tb1hy20ri2ssIsuI/edit#gid=0) 8 | -------------------------------------------------------------------------------- /miro.md: -------------------------------------------------------------------------------- 1 | MIRO – Minimum Information for Reporting of an Ontology 2 | ============== 3 | 4 | ## Published 22 December 2017 5 | 6 | **Editors**: 7 | 8 | Robert Stevens, Manchester University 9 | 10 | James Malone, FactBio Limited 11 | 12 | Chris Mungall, National Evolutionary Synthesis Center, Lawrence Berkeley Labs, Gene Ontology Consortium 13 | 14 | Nicolas Matentzoglu, Manchester University 15 | 16 | **Edit history:** 17 | 18 | | Version | Date | Contributors | Changes | 19 | |---|---|---|---| 20 | | 0.1 | 16.05.2016 | Robert Stevens, James Malone, Chris Mungall, Nicolas Matentzoglu | First version | 21 | | 0.2 | 27.05.2016 | James Malone|Review first version for wording and examples | 22 | | 0.3 | 28.06.2016 | Nicolas Matentzoglu | Unified capitalisation in Information Item names | 23 | | 0.4 | 02.12.2016 | James Malone | Add elucidation to items | 24 | | 0.5 | 12.03.2017 | Robert Stevens, James Malone, Chris Mungall, Nicolas Matentzoglu | First public release | 25 | | 0.6 | 13.08.2017 | James Malone | Update to reflect reviews from submitted publication | 26 | ---- 27 | 28 | # Summary 29 | This document contains the Minimum Information for Reporting of an Ontology (MIRO) guidelines. 30 | 31 | # Guidelines 32 | 33 | ## A. Basics 34 | 35 | ### A.1 Ontology name 36 | 37 | | | Specification | 38 | |---|---| 39 | | **Description:** | The full name of the ontology, including the acronym and the version number referred to in the report. | 40 | | **Importance:** | MUST | 41 | | **Example:** | Gene Ontology (GO), v2.13; Foundational Model of Anatomy Ontology (FMA), v4.22; | 42 | ---- 43 | 44 | ### A.2 Ontology owner 45 | 46 | | | Specification | 47 | |---|---| 48 | | **Description:** | The names, affiliations (where appropriate) and contact details of the person, people or consortium that manage the development of the ontology. | 49 | | **Importance:** | MUST | 50 | | **Example:** | GO Administrators, go@geneontology.org | 51 | ---- 52 | 53 | ### A.3 Ontology license 54 | 55 | | | Specification | 56 | |---|---| 57 | | **Description:** | The licence which governs the permissions surrounding the ontology. | 58 | | **Importance:** | MUST | 59 | | **Example:** | Creative Commons Attribution 3.0 (CC BY 3.0) | 60 | ---- 61 | 62 | ### A.4 Ontology URL 63 | 64 | | | Specification | 65 | |---|---| 66 | | **Description:** | The web location where the ontology file is available. | 67 | | **Importance:** | MUST | 68 | | **Example:** | http://www.berkeleybop.org/ontologies/doid.owl | 69 | ---- 70 | 71 | ### A.5 Ontology repository 72 | 73 | | | Specification | 74 | |---|---| 75 | | **Description:** | The web location (URL) of the version control system where current and previous versions of the ontology can be found. | 76 | | **Importance:** | MUST | 77 | | **Example:** | https://github.com/geneontology | 78 | ---- 79 | 80 | ### A.6 Methodological framework 81 | 82 | | | Specification | 83 | |---|---| 84 | | **Description:** | A name or description of the steps taken to develop the ontology. This should describe the overall organisation of the ontology development process. | 85 | | **Importance:** | MUST | 86 | | **Example:** | Methontology, On-To-Knowledge, Neon methodology. If no bespoke methodological framework was used, describe how the core activities of the ontology engineering lifecycle (such as ontology management, development and support) were addressed (http://onlinelibrary.wiley.com/doi/10.1002/047003033X.ch9/pdf). | 87 | ---- 88 | 89 | ## B. Motivation 90 | 91 | ### B.1 Need 92 | 93 | | | Specification | 94 | |---|---| 95 | | **Description:** | Justification of why the ontology is required. | 96 | | **Importance:** | MUST | 97 | | **Example:** | For example, the 2015 JBMS article "Unification of multi-species vertebrate anatomy ontologies for comparative biology in Uberon" has the following: "Here we present the unification of anatomy ontologies into Uberon, a single ontology resource that enables interoperability among disparate data and research groups." http://jbiomedsem.biomedcentral.com/articles/10.1186/2041-1480-5-21 | 98 | ---- 99 | 100 | ### B.2 Competition 101 | 102 | | | Specification | 103 | |---|---| 104 | | **Description:** | The names and citations for other ontology or ontologies in the same general area as the one being reported upon, together with a description on why the one being reported is needed instead or in addition to the others. | 105 | | **Importance:** | MUST | 106 | | **Example:** | The background section of the paper "The Software Ontology (SWO): a resource for reproducibility in biomedical data analysis, curation and digital preservation" in JBMS 2014 describes other attempts to describe software in ontologies or vocabularies. See http://jbiomedsem.biomedcentral.com/articles/10.1186/2041-1480-5-25 | 107 | ---- 108 | 109 | ### B.3 Target audience 110 | 111 | | | Specification | 112 | |---|---| 113 | | **Description:** | The community or organisation performing some task or use for which the ontology was developed. | 114 | | **Importance:** | MUST | 115 | | **Example:** | The Gene Ontology is used by curators of gene products to describe the major functional attributes of those gene products to provide a common language across species for describing the molecular function, biological process and cellular location of gene products. | 116 | ---- 117 | 118 | ## C. Scope, requirements, development community 119 | 120 | ### C.1 Scope and coverage 121 | 122 | | | Specification | 123 | |---|---| 124 | | **Description:** | The domain or field of interest for the ontology and the boundaries, granularity of representation and coverage of the ontology. State the requirements of the ontology, such as the competency questions it should satisfy. A visualisation or tabular representation is optional, but often helpful to illustrate the scope. | 125 | | **Importance:** | MUST | 126 | | **Example:** | From the Human Phenotype Ontology NAR 2013 article "human phenotypic abnormalities." https://academic.oup.com/nar/article-lookup/doi/10.1093/nar/gkt1026 | 127 | ---- 128 | 129 | ### C.2 Development community 130 | 131 | | | Specification | 132 | |---|---| 133 | | **Description:** | The person, group of people or organisation that actually creates the content of the ontology. This is distinct from the Ontology Owner (above) that is concerned with the management of the ontology's development. | 134 | | **Importance:** | MUST | 135 | | **Example:** | The Gene Ontology is developed by the Gene Ontology Consortium. | 136 | ---- 137 | 138 | ### C.3 Communication 139 | 140 | | | Specification | 141 | |---|---| 142 | | **Description:** | Location, usually URL, of the email list and/or the issue tracking systems used for development and managing feature requests for the ontology. | 143 | | **Importance:** | MUST | 144 | | **Example:** | http://www.ebi.ac.uk/efo/submit.html | 145 | ---- 146 | 147 | ## D. Knowledge acquisition 148 | 149 | ### D.1 Knowledge acquisition methodology 150 | 151 | | | Specification | 152 | |---|---| 153 | | **Description:** | How the knowledge in the ontology was gathered, sorted, verified, etc. | 154 | | **Importance:** | MUST | 155 | | **Example:** | Description of the source knowledge in the materials and method section of the article for The Software Ontology (SWO): a resource for reproducibility in biomedical data analysis, curation and digital preservation in JBMS 2014 https://jbiomedsem.biomedcentral.com/articles/10.1186/2041-1480-5-25 | 156 | ---- 157 | 158 | ### D.2 Source knowledge location 159 | 160 | | | Specification | 161 | |---|---| 162 | | **Description:** | The location of the source whence the knowledge was gathered. | 163 | | **Importance:** | SHOULD | 164 | | **Example:** | Description of data gathered such as found in https://softwareontology.wordpress.com/2011/04/04/an-agile-ontology/. Other examples may include the outputs from workshops, curating the literature for a field, crowd sourcing input and so on. | 165 | ---- 166 | 167 | ### D.3 Content selection 168 | 169 | | | Specification | 170 | |---|---| 171 | | **Description:** | The prioritisation of entities to be represented in the ontology and how that prioritisation was achieved. Some knowledge is more important or of greater priority to be in the ontology to support the requirements of that ontology. | 172 | | **Importance:** | SHOULD | 173 | | **Example:** | Description of the prioritisation process for the Software Ontology in https://softwareontology.wordpress.com/2011/04/04/an-agile-ontology/ | 174 | ---- 175 | 176 | ## E. Ontology content 177 | 178 | ### E.1 Knowledge Representation language 179 | 180 | | | Specification | 181 | |---|---| 182 | | **Description:** | the knowledge representation language used and why it was used. For a language like OWL, indicate the OWL profile and expressivity. | 183 | | **Importance:** | MUST | 184 | | **Example:** | OWL version 2, EL profile. | 185 | ---- 186 | 187 | ### E.2 Development environment 188 | 189 | | | Specification | 190 | |---|---| 191 | | **Description:** | The tool(s) used in developing the ontology. | 192 | | **Importance:** | OPTIONAL | 193 | | **Example:** | Protégé, WebProtégé, TawnyOWL, TopBraid Composer, Swoop, etc. | 194 | ---- 195 | 196 | ### E.3 Ontology metrics 197 | 198 | | | Specification | 199 | |---|---| 200 | | **Description:** | Number of classes, properties, axioms and types of axioms, rules and individuals in the ontology. | 201 | | **Importance:** | SHOULD | 202 | | **Example:** | From the Human Phenotype Ontology NAR 2013 article "Human Phenotype Ontology provides a structured, comprehensive and well-defined set of 10,088 classes (terms) describing human phenotypic abnormalities and 13,326 subclass relations between the HPO classes." https://academic.oup.com/nar/article-lookup/doi/10.1093/nar/gkt1026 | 203 | ---- 204 | 205 | ### E.4 Incorporation of other ontologies 206 | 207 | | | Specification | 208 | |---|---| 209 | | **Description:** | The names, versions and citations of external ontologies imported into the ontology and where they are placed in the host ontology. | 210 | | **Importance:** | MUST | 211 | | **Example:** | Import of Uberon core – extended version, 2016-05-11 release under the 'anatomical entity' class. | 212 | ---- 213 | 214 | ### E.5 Entity naming convention 215 | 216 | | | Specification | 217 | |---|---| 218 | | **Description:** | The naming scheme for the entities in the ontology, capturing orthography, organisation rules, acronyms, and so on. | 219 | | **Importance:** | MUST | 220 | | **Example:** | Open Biomedical Ontologies (OBO) Foundry naming conventions; see http://www.obofoundry.org/principles/fp-012-naming-conventions.html. | 221 | ---- 222 | 223 | ### E.6 Identifier generation policy 224 | 225 | | | Specification | 226 | |---|---| 227 | | **Description:** | What is the scheme used for creating identifiers for entities in the ontology. State whether identifiers are semantic-free or meaningful. | 228 | | **Importance:** | MUST | 229 | | **Example:** | Incremental class number, using 10 digit number with ontology name as the prefix. | 230 | ---- 231 | 232 | ### E.7 Entity metadata policy 233 | 234 | | | Specification | 235 | |---|---| 236 | | **Description:** | What metadata for each entity is to be present. This could include, but not be limited to: A natural language definition, editor, edit history, examples, entity label and synonyms, etc. | 237 | | **Importance:** | MUST | 238 | | **Example:** | Each class minimally requires a textual definition, a label, a creator and an edit date. The Ontology for Biomedical Investigations (OBI) paper has a section entitled "Choice of metadata conventions". | 239 | | 240 | ---- 241 | 242 | ### E.8 Upper ontology 243 | 244 | | | Specification | 245 | |---|---| 246 | | **Description:** | If an upper ontology is used, which one is used and why is it used? If not used, then why. | 247 | | **Importance:** | MUST | 248 | | **Example:** | SUMO, BFO. The Ontology for Biomedical Investigations (OBI) paper has a sentence "While each had its strengths and weaknesses, BFO was chosen due to its association with the nascent OBO Foundry" in its discussion. | 249 | ---- 250 | 251 | ### E.9 Ontology relationships 252 | 253 | | | Specification | 254 | |---|---| 255 | | **Description:** | The relationships or properties used in the ontology, which were used and why? Were new relationships required? Why? | 256 | | **Importance:** | MUST | 257 | | **Example:** | The schema.org relationships were imported and used. We used the OBO Relations Ontology. | 258 | ---- 259 | 260 | ### E.10 Axiom patterns 261 | 262 | | | Specification | 263 | |---|---| 264 | | **Description:** | An axiom pattern is a regular design of axioms or a template for axioms used to represent a category of entities or common aspects of a variety of types of entities. An axiom pattern may comprise both asserted and inferred axioms. The aim of a pattern is to achieve a consistent style of representation. An important family of axiom patterns are Ontology Design pattern (ODP) which are commonly used solutions for issues in representation. | 265 | | **Importance:** | MUST | 266 | | **Example:** | The axiom patterns described in the Section "The Ontology" for the Software Ontology in its JBMS paper (http://jbiomedsem.biomedcentral.com/articles/10.1186/2041-1480-5-25). Patterns can be presented as examples or in the form of axiom templates, such as a reference to an ontology design pattern (http://ontologydesignpatterns.org/wiki/Main_Page). | 267 | ---- 268 | 269 | ### E.11 Dereferenceable IRIs 270 | 271 | | | Specification | 272 | |---|---| 273 | | **Description:** | State whether or not the IRI used are dereferencable to a Web resource. Provide any standard prefix (CURIE). | 274 | | **Importance:** | SHOULD | 275 | | **Example:** | For example, http://purl.obolibrary.org/obo/GO_0006915 | 276 | ---- 277 | 278 | ## F. Managing Change 279 | 280 | ### F.1 Sustainability plan 281 | 282 | | | Specification | 283 | |---|---| 284 | | **Description:** | State whether the ontology will be actively maintained and developed. Describe a plan for how the ontology will be kept up to date. | 285 | | **Importance:** | MUST | 286 | | **Example:** | Outline of sustainability plan, including method to sustain and who will be responsible. | 287 | ---- 288 | 289 | ### F.2 Entity deprecation strategy 290 | 291 | | | Specification | 292 | |---|---| 293 | | **Description:** | Describe the procedures for managing entities that become removed, split or redefined. | 294 | | **Importance:** | MUST | 295 | | **Example:** | The owl:DeprecatedClass; no class is deleted from the ontology, but deprecated classes are labelled as obsolete with an annotation property. | 296 | ---- 297 | 298 | ### F.3 Versioning policy 299 | 300 | | | Specification | 301 | |---|---| 302 | | **Description:** | State or make reference to the policy that governs when new versions of the ontology are created and released. | 303 | | **Importance:** | MUST | 304 | | **Example:** | The Open Biomedical Ontologies Consortium has a versioning policy, see http://www.obofoundry.org/id-policy.html. This page states "Versions are named by a date in the following format: YYYY-MM-DD. For a given version of an ontology, the ontology should be accessible at the following URL, where is replaced by the IDSPACE in lower case." and gives further elaboration. | 305 | ---- 306 | 307 | ## G. Quality Assurance 308 | 309 | ### G.1 Testing 310 | 311 | | | Specification | 312 | |---|---| 313 | | **Description:** | Description of the procedure used to judge whether the ontology achieves the claims made for the ontology. State, for example, whether the ontology is logically consistent, answers the queries it claims to answer, and whether it can answer them in a time that is reasonable for the projected use case scenario (benchmarking). | 314 | | **Importance:** | MUST | 315 | | **Example:** | Examples of competency questions that can be asked. Examples of inferences that can be made. Classification time using an appropriate reasoner: "The ontology was successfully classified by both Pellet 2.3.1 and HermiT 1.3.8 (i.e., is logically consistent) in less than 1 second. All classes in the ontology are satisfiable." | 316 | ---- 317 | 318 | ### G.2 Evaluation 319 | 320 | | | Specification | 321 | |---|---| 322 | | **Description:** | A determination of whether the ontology is of value and significance. An evaluation should show that the motivation is justified and that the objectives of the ontology's development are met effectively and satisfactorily. Describe whether or not the ontology meets its stated requirements, competency questions and goals. | 323 | | **Importance:** | MUST | 324 | | **Example:** | There is an evaluation described for the Emotion Ontology in the paper "Evaluating the Emotion Ontology through use in the self-reporting of emotional responses in an academic conference". http://jbiomedsem.biomedcentral.com/articles/10.1186/2041-1480-5-38 | 325 | ---- 326 | 327 | ### G.3 Example of use 328 | 329 | | | Specification | 330 | |---|---| 331 | | **Description:** | An illustration of the ontology in use in its an application setting or use case. | 332 | | **Importance:** | MUST | 333 | | **Example:** | to Table S1 in The Ontology for Biomedical Investigations article in PLoS One 2016 https://www.ncbi.nlm.nih.gov/pubmed/27128319 shows the projects using OBI. | 334 | ---- 335 | 336 | ### G.4 Institutional endorsement 337 | 338 | | | Specification | 339 | |---|---| 340 | | **Description:** | State whether the ontology is endorsed by the W3C, the OBO foundry or some organisation representing a community. | 341 | | **Importance:** | OPTIONAL | 342 | | **Example:** | SNOMED is mandated for use by many national health organisations. The Chemicals of Biological Interest Ontology is a member of the OBO Foundry. | 343 | ---- 344 | 345 | 346 | ### G.5 Evidence of use 347 | 348 | | | Specification | 349 | |---|---| 350 | | **Description:** | An illustration of active projects and applications that use the ontology. | 351 | | **Importance:** | MUST | 352 | | **Example:** | The Gene Ontology is used to annotate the gene products for their molecular function, biological process and cellular component in many species specific and cross-species databases. | 353 | ---- 354 | 355 | -------------------------------------------------------------------------------- /supplementary/README.md: -------------------------------------------------------------------------------- 1 | # The minimum information for the reporting of an ontology (MIRO) guidelines: Supplementary Materials 2 | 3 | ## Datasets: MIRO community feedback (questionnaire) 4 | 5 | | Filename | Description | 6 | |---|---| 7 | | **questionnaire_responses.csv** | Anonymised questionnaire responses (without names and emails). | 8 | | **questionnaire_email_lists.csv** | The email lists we used to advertise the questionnaire / survey. | 9 | | **questionnaire_comment_coding.csv** | The coding of the free-form comments of the survey responses. | 10 | ---- 11 | 12 | ## Datasets: Paper review of MIRO compliance 13 | 14 | | Filename | Description | 15 | |---|---| 16 | | **review_papers_in_timeframe.csv** | List of all papers by journal within the given timeframe. | 17 | | **review_titles.csv** | Paper inclusion decisions based on titles. | 18 | | **review_abstracts.csv** | Paper inclusion decisions based on abstracts. | 19 | | **review_selected_papers.csv** | Papers selected for in-depth review. | 20 | | **review_coding_scheme.csv** | Coding scheme for data extraction. | 21 | | **review_extraction.csv** | Data extracted from all papers according to coding scheme. | 22 | ---- 23 | 24 | ## Analysis: Executable analysis of data (R markdown) 25 | 26 | | Filename | Description | 27 | |---|---| 28 | | **jbms_ontology_survey.Rmd** | Executable R markdown anlysis of the MIRO questionnaire and the paper review of MIRO compliance. | 29 | | **jbms_coded_free_form.csv** | Copy of questionnaire_comment_coding.csv (see above). | 30 | | **jbms_labels.csv** | A mapping of MIRO survey questions to MIRO information items. | 31 | | **jbms_paper_results.csv** | Copy of review_extraction.csv (see above). | 32 | | **jbms_results.csv** | Copy of questionnaire_responses.csv (see above). | 33 | | **rating_v_compliance.csv** | A manually created document that has MIRO items, survey questions, average importance and compliance side-by-side. | 34 | ---- 35 | 36 | 37 | 38 | -------------------------------------------------------------------------------- /supplementary/analysis/jbms_coded_free_form.csv: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/owlcs/miro/8dfb8fe40449723d24805f61dd7fde76dbb8ec18/supplementary/analysis/jbms_coded_free_form.csv -------------------------------------------------------------------------------- /supplementary/analysis/jbms_labels.csv: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/owlcs/miro/8dfb8fe40449723d24805f61dd7fde76dbb8ec18/supplementary/analysis/jbms_labels.csv -------------------------------------------------------------------------------- /supplementary/analysis/jbms_ontology_survey.Rmd: -------------------------------------------------------------------------------- 1 | --- 2 | title: 'The minimum information for the reporting of an ontology (MIRO) guidelines: 3 | Analysis' 4 | author: "Nicolas Matentzoglu, James Malone, Chris Mungall and Robert Stevens" 5 | date: "7 May 2016" 6 | --- 7 | 8 | This document contains the analysis for the paper "The minimum information for the reporting of an ontology (MIRO) guidelines", submitted to the Journal of Biomedical Semantics (JBMS) in March 2017. 9 | 10 | ```{r dirsetup, include=FALSE} 11 | ## wd is the directory that contains the four raw data files: jbms_results.csv, jbms_paper_results.csv, rating_v_compliance.csv, jbms_labels.csv, jbms_coded_free_form.csv 12 | wd<-"D:/Dropbox (Personal)/jbms2017_miro/survey results/jbms_submission/analysis/" 13 | 14 | ## directory to export the charts into: 15 | chartdir<-"D:/Dropbox (Personal)/jbms2017_miro/survey results/jbms_submission/analysis/" 16 | ``` 17 | 18 | ```{r functions, include=FALSE} 19 | aggregate_by_role<-function(df_long) { 20 | x<-df_long 21 | fun<-function(x) c(mean =mean(x,na.rm=TRUE),median =median(x,na.rm=TRUE),sd =sd(x,na.rm=TRUE),sum=sum(x,na.rm=TRUE) ) 22 | 23 | df_long_agg<-as.data.frame(as.list(aggregate(x$value,by=list(x$label),FUN=fun))) 24 | df_long_agg$group="all" 25 | 26 | x<-df_long[df_long$r_user,] 27 | y<-as.data.frame(as.list(aggregate(x$value,by=list(x$label),FUN=fun))) 28 | y$group="user" 29 | df_long_agg<-rbind(df_long_agg,y) 30 | 31 | x<-df_long[df_long$r_author,] 32 | y<-as.data.frame(as.list(aggregate(x$value,by=list(x$label),FUN=fun))) 33 | y$group="author" 34 | df_long_agg<-rbind(df_long_agg,y) 35 | 36 | x<-df_long[df_long$r_review,] 37 | y<-as.data.frame(as.list(aggregate(x$value,by=list(x$label),FUN=fun))) 38 | y$group="reviewer" 39 | df_long_agg<-rbind(df_long_agg,y) 40 | 41 | x<-df_long[df_long$r_reader,] 42 | y<-as.data.frame(as.list(aggregate(x$value,by=list(x$label),FUN=fun))) 43 | y$group="reader" 44 | df_long_agg<-rbind(df_long_agg,y) 45 | 46 | x<-df_long[df_long$r_developer,] 47 | y<-as.data.frame(as.list(aggregate(x$value,by=list(x$label),FUN=fun))) 48 | y$group="developer" 49 | df_long_agg<-rbind(df_long_agg,y) 50 | 51 | names(df_long_agg)<-c("ontology_feature","mean","median","sd","sum","group") 52 | 53 | df_long_agg<-reshape(df_long_agg,timevar = "group",idvar="ontology_feature",direction = "wide") 54 | 55 | df_long_agg<-df_long_agg[order(-df_long_agg$mean.reviewer),] 56 | df_long_agg$Reviewer<-1:nrow(df_long_agg) 57 | 58 | df_long_agg<-df_long_agg[order(-df_long_agg$mean.author),] 59 | df_long_agg$Author<-1:nrow(df_long_agg) 60 | 61 | df_long_agg<-df_long_agg[order(-df_long_agg$mean.user),] 62 | df_long_agg$User<-1:nrow(df_long_agg) 63 | 64 | df_long_agg<-df_long_agg[order(-df_long_agg$mean.reader),] 65 | df_long_agg$Reader<-1:nrow(df_long_agg) 66 | 67 | df_long_agg<-df_long_agg[order(-df_long_agg$mean.developer),] 68 | df_long_agg$Developer<-1:nrow(df_long_agg) 69 | 70 | 71 | df_long_agg<-df_long_agg[order(-df_long_agg$mean.all),] 72 | df_long_agg$All<-1:nrow(df_long_agg) 73 | return(df_long_agg) 74 | } 75 | trim <- function (x) gsub("^\\s+|\\s+$", "", x) 76 | ``` 77 | 78 | ```{r prepare, include=FALSE} 79 | setwd(wd) 80 | 81 | library(reshape2) 82 | library(plyr) 83 | library(ggplot2) 84 | library(xtable) 85 | library(knitr) 86 | 87 | results_f<-"jbms_results.csv" 88 | paper_results_f<-"jbms_paper_results.csv" 89 | ratingvcompliance_f<-"rating_v_compliance.csv" 90 | labels_f<-"jbms_labels.csv" 91 | comments_f<-"jbms_coded_free_form.csv" 92 | 93 | df_results<-read.csv(file=results_f,head=TRUE,sep=",",stringsAsFactors = FALSE) 94 | df_role<-df_results[ , grepl( "^role_" , names( df_results ) ) ] 95 | 96 | df_role$role_developer<-ifelse(df_role$role_developer=="Ontology developer",1,0) 97 | df_role$role_user<-ifelse(df_role$role_user=="Ontology user",1,0) 98 | df_role$role_reader<-ifelse(df_role$role_reader=="Read papers on ontologies",1,0) 99 | df_role$role_author<-ifelse(df_role$role_author=="Author papers on ontologies",1,0) 100 | df_role$role_review<-ifelse(df_role$role_review=="Review papers on ontologies",1,0) 101 | 102 | df_labels<-read.csv(file=labels_f,head=FALSE,sep=",") 103 | dfl<-as.data.frame(t(df_labels)) 104 | names(dfl) <- c("deprecated","col","orig","label") 105 | 106 | df_results$job<-ifelse(df_results$job=="",ifelse(df_results$job_other=="","Non given",df_results$job_other),df_results$job) 107 | df_results$job<-ifelse(df_results$job=="Lecturer in the University","An academic employee",df_results$job) 108 | df_results$job<-ifelse(df_results$job=="contractor","Other",df_results$job) 109 | 110 | df_results$job<-ifelse(df_results$job=="A private sector employee","Private sector employee",df_results$job) 111 | df_results$job<-ifelse(df_results$job=="A public sector/not-for profit employee","Public sector/not-for profit employee",df_results$job) 112 | df_results$job<-ifelse(df_results$job=="A student","Student",df_results$job) 113 | df_results$job<-ifelse(df_results$job=="An academic employee","Academic employee",df_results$job) 114 | df_results$job<-ifelse(df_results$job=="Consultant","Other",df_results$job) 115 | df_results$job<-ifelse(df_results$job=="Ontology Consulting Company Owner","Other",df_results$job) 116 | df_results$job<-ifelse(df_results$job=="Contractor","Other",df_results$job) 117 | df_results$job<-ifelse(df_results$job=="Consultant for academic/public sector/non-profit","Other",df_results$job) 118 | 119 | 120 | df_results$role_developer<-ifelse(df_results$role_developer=="Ontology developer","D","") 121 | df_results$role_user<-ifelse(df_results$role_user=="Ontology user","U","") 122 | df_results$role_reader<-ifelse(df_results$role_reader=="Read papers on ontologies","R","") 123 | df_results$role_author<-ifelse(df_results$role_author=="Author papers on ontologies","A","") 124 | df_results$role_review<-ifelse(df_results$role_review=="Review papers on ontologies","W","") 125 | 126 | df_results$r_developer<-ifelse(df_results$role_developer=="D",TRUE,FALSE) 127 | df_results$r_user<-ifelse(df_results$role_user=="U",TRUE,FALSE) 128 | df_results$r_reader<-ifelse(df_results$role_reader=="R",TRUE,FALSE) 129 | df_results$r_author<-ifelse(df_results$role_author=="A",TRUE,FALSE) 130 | df_results$r_review<-ifelse(df_results$role_review=="W",TRUE,FALSE) 131 | 132 | df_results$role<-paste(df_results$role_developer,df_results$role_user,df_results$role_reader,df_results$role_author,df_results$role_review,sep="") 133 | df_results$email<-ifelse(grepl(".*[@].*[.]",df_results$email),trim(df_results$email),"NA") 134 | df_results$tdl<-sub('.*\\.', '', df_results$email) 135 | 136 | df_comments<-read.csv(file=comments_f,head=TRUE,stringsAsFactors = FALSE,sep=",") 137 | 138 | df_paper_results<-read.csv(file=paper_results_f,head=TRUE,stringsAsFactors = FALSE,sep=",") 139 | 140 | ``` 141 | 142 | # Demographics 143 | ```{r demographics} 144 | 145 | fontsize<-16 146 | 147 | ## Role Analysis 148 | (count(df_results$role_user)) 149 | (count(df_results$role_developer)) 150 | (count(df_results$role_review)) 151 | (count(df_results$role_reader)) 152 | (count(df_results$role_author)) 153 | 154 | colctrole<-colSums(df_results[c("r_user","r_developer","r_review","r_reader","r_author")]) 155 | names(colctrole)<-c("User","Developer","Reviewer","Reader","Author") 156 | colctrole<-melt(colctrole) 157 | colctrole$id<-rownames(colctrole) 158 | 159 | ggplot(colctrole,aes(id,value)) +geom_bar(stat = "identity") + xlab("") + ylab("") + theme(text = element_text(size=fontsize)) + coord_flip() + theme(plot.margin = unit(c(0,0,-0.5,-0.5), "cm")) 160 | ggsave(paste(chartdir,"role.pdf",sep = ""), width = 3, height = 2.5) 161 | 162 | ### Cross-correlation matrix for role 163 | names(df_role)<-c("Developer","User","Reader","Author","Reviewer") 164 | corMat <- cor(df_role) 165 | kable(corMat) 166 | melted_cormat <- melt(corMat) 167 | 168 | ggplot(data = melted_cormat, aes(x=Var1, y=Var2, fill=value)) + geom_tile()+ xlab("") + ylab("") + scale_fill_gradient(low="white", high="black") + theme(text = element_text(size=fontsize)) + theme(plot.margin = unit(c(0,0,-0.5,0), "cm")) 169 | ggsave(paste(chartdir,"role_cormat.pdf",sep = ""), width = 7, height = 2.5) 170 | 171 | ### Combinations of ontology roles 172 | ct_r<-count(df_results$role) 173 | ct_r$pc<-round(ct_r$freq/nrow(df_results)*100,2) 174 | kable(ct_r[order(-ct_r$freq),]) 175 | 176 | ##TDL Analysis 177 | ct_tdl<-count(df_results$tdl) 178 | ct_tdl<-ct_tdl[order(-ct_tdl$freq),] 179 | 180 | ggplot(ct_tdl,aes(reorder(x,freq),freq)) +geom_bar(stat = "identity") + xlab("") + ylab("") + theme(text = element_text(size=fontsize)) + theme(plot.margin = unit(c(0,0,-0.5,0), "cm")) 181 | ggsave(paste(chartdir,"tdl.pdf",sep = ""), width = 6, height = 2.5) 182 | 183 | ##Job analysis 184 | ct<-count(df_results$job) 185 | ggplot(ct,aes(x=reorder(x,freq),y=freq)) + geom_bar(stat="identity") + ylab("") + xlab("") + coord_flip() + theme(text = element_text(size=fontsize)) + theme(plot.margin = unit(c(0,0,-0.5,-0.5), "cm")) 186 | ggsave(paste(chartdir,"job.pdf",sep = ""), width = 4, height = 2.5) 187 | 188 | 189 | write.csv(corMat,file = "correlation_matrix_usergroups.csv") 190 | 191 | ``` 192 | 193 | #Overall ranking 194 | ```{r overall_ranking} 195 | ##Create ranking by mean 196 | df<-df_results[ , grepl( "^o_" , names( df_results )) ] 197 | df$id<-df_results$id 198 | 199 | df_long<-melt(df,id.vars="id") 200 | df_long<-merge(df_long,df_results[ , c("id","r_developer","r_user","r_author","r_review","r_reader","role") ],by="id") 201 | 202 | df_long<-merge(df_long,dfl[c("col","label")],by.x = "variable",by.y = "col") 203 | df_long$label<-as.factor(df_long$label) 204 | 205 | df_long_agg<-aggregate(df_long$value,by=list(df_long$label),FUN=function(x) mean(x, na.rm=TRUE)) 206 | df_long_agg<-df_long_agg[order(-df_long_agg$x),] 207 | df_long$label <- factor(df_long$label, levels = df_long_agg$Group.1) 208 | 209 | #head(df_long[c("variable","value")]) 210 | 211 | ###Ordered mean rating of information items across all participants 212 | ggplot(df_long,aes(x=label,y=value)) + stat_summary(fun.y = mean, geom = "bar") + geom_hline(yintercept = 5)+ ylab("Mean Rating") + xlab("")+ geom_hline(yintercept = 3.0) + geom_hline(yintercept = 3.5) + coord_flip(ylim=c(2.6,5)) + theme(text = element_text(size=10)) + theme(plot.margin = unit(c(0,0,-0.5,-0.5), "cm")) + geom_text(aes(x=29.81, y=3.1,label = "SHOULD"), hjust = 0, vjust = 0, size=2.8) + geom_text(aes(x=29.81, y=2.5,label = "OPTIONAL"), hjust = 0, vjust = 0, size=2.8,colour="white") + geom_text(aes(x=29.81, y=4.1,label = "MUST"), hjust = 0, vjust = 0, size=2.8) 213 | ggsave(paste(chartdir,"overall_mean_results.pdf",sep = ""), width = 6, height = 6) 214 | 215 | ## Rankings across roles 216 | df_long_agg<-aggregate_by_role(df_long) 217 | write.csv(df_long_agg,file = "priority_by_group.csv") 218 | options(scipen = 100) 219 | df_long_agg<-df_long_agg[order(df_long_agg$sd.all),] 220 | kable(df_long_agg[c("ontology_feature","All","mean.all","median.all","sd.all")]) 221 | 222 | #print(xtable(df_long_agg[c("ontology_feature","All","mean.all","median.all","sd.all")],digits=c(0,0,0,2,0,2)),include.rownames=FALSE) 223 | 224 | ### Correlation of standard deviation and mean 225 | kable(cor(df_long_agg[c("sd.all","mean.all")])) 226 | 227 | 228 | ### Ranking table 229 | dfl_i<-df_long_agg[c("ontology_feature","All","Author","Developer","Reviewer","User","Reader")] 230 | dfl_i<-dfl_i[order(dfl_i$All),] 231 | kable(dfl_i) 232 | 233 | thresh<-4 234 | dfl_i_dev<-dfl_i 235 | #dfl_i_dev$author_d<-ifelse((dfl_i_dev$All-dfl_i_dev$Author)>=thresh,"less",ifelse((dfl_i_dev$All-dfl_i_dev$Author)<=-thresh,"more","")) 236 | #dfl_i_dev$developer_d<-ifelse((dfl_i_dev$All-dfl_i_dev$Developer)>=thresh,"less",ifelse((dfl_i_dev$All-dfl_i_dev$Developer)<=-thresh,"more","")) 237 | #dfl_i_dev$reviewer_d<-ifelse((dfl_i_dev$All-dfl_i_dev$Reviewer)>=thresh,"less",ifelse((dfl_i_dev$All-dfl_i_dev$Reviewer)<=-thresh,"more","")) 238 | #dfl_i_dev$user_d<-ifelse((dfl_i_dev$All-dfl_i_dev$User)>=thresh,"less",ifelse((dfl_i_dev$All-dfl_i_dev$User)<=-thresh,"more","")) 239 | #dfl_i_dev$reader_d<-ifelse((dfl_i_dev$All-dfl_i_dev$Reader)>=thresh,"less",ifelse((dfl_i_dev$All-dfl_i_dev$Reader)<=-thresh,"more","")) 240 | dfl_i_dev$author_d<-dfl_i_dev$All-dfl_i_dev$Author 241 | dfl_i_dev$developer_d<-dfl_i_dev$All-dfl_i_dev$Developer 242 | dfl_i_dev$reviewer_d<-dfl_i_dev$All-dfl_i_dev$Reviewer 243 | dfl_i_dev$user_d<-dfl_i_dev$All-dfl_i_dev$User 244 | dfl_i_dev$reader_d<-dfl_i_dev$All-dfl_i_dev$Reader 245 | kable(dfl_i_dev) 246 | 247 | ### Ranking of information items broken down by roles 248 | df_long_agg_long<-melt(dfl_i,id.vars="ontology_feature") 249 | df_long_agg_long$ontology_feature <- factor(df_long_agg_long$ontology_feature, levels = dfl_i$ontology_feature) 250 | ggplot(df_long_agg_long,aes(x=ontology_feature,y=value,group=variable,fill=variable)) + geom_bar(stat = "identity",position = "dodge") + coord_flip() 251 | ggsave(paste(chartdir,"overall_ranking_by_role.pdf",sep = ""), width = 4, height = 2.5) 252 | ``` 253 | 254 | # Comment Analysis 255 | ```{r analyse_comments} 256 | df<-df_comments[ , grepl( "_code[2-6]?$" , names( df_comments )) ] 257 | #df$id<-df_comments$id 258 | 259 | kw<-c() 260 | 261 | for(i in 1:nrow(df)) { 262 | kw<-c(kw,unique(as.character(df[i,]))) 263 | } 264 | 265 | ct_key<-count(kw) 266 | kable(ct_key[order(-ct_key$freq),]) 267 | 268 | df_comments_important<-df_comments[ , grepl( "important_comments_code[2-6]?$" , names( df_comments )) ] 269 | 270 | keywords<-c() 271 | 272 | for(i in 1:nrow(df)) { 273 | keywords<-c(keywords,unique(as.character(df[i,]))) 274 | } 275 | 276 | keywords<-keywords[keywords != ""]; 277 | 278 | ct_comments_all_key<-count(keywords) 279 | kable(ct_comments_all_key[order(-ct_comments_all_key$freq),]) 280 | 281 | df_comments_important<-df_comments[ , grepl( "important_comments_code[2-6]?$" , names( df_comments )) ] 282 | 283 | keywords<-c() 284 | 285 | for(i in 1:nrow(df_comments_important)) { 286 | keywords<-c(keywords,unique(as.character(df_comments_important[i,]))) 287 | } 288 | 289 | keywords<-keywords[keywords != ""]; 290 | ct_comments_important_key<-count(keywords) 291 | kable(ct_comments_important_key[order(-ct_comments_important_key$freq),]) 292 | 293 | 294 | ct_comments_important_key$group<-"ungrouped" 295 | 296 | ct_comments_important_key$group<-ifelse(ct_comments_important_key$group=="ungrouped",ifelse(ct_comments_important_key$x=="coverage"|ct_comments_important_key$x=="scope"|ct_comments_important_key$x=="interdisciplinary"|ct_comments_important_key$x=="granularity","Scope and Coverage","ungrouped"),ct_comments_important_key$group) 297 | 298 | ct_comments_important_key$group<-ifelse(ct_comments_important_key$group=="ungrouped",ifelse(ct_comments_important_key$x=="compatibility_other_ontologies"|ct_comments_important_key$x=="interoperability","Interoperability","ungrouped"),ct_comments_important_key$group) 299 | 300 | ct_comments_important_key$group<-ifelse(ct_comments_important_key$group=="ungrouped",ifelse(ct_comments_important_key$x=="use_case_suitability"|ct_comments_important_key$x=="use_case_adjustability"|ct_comments_important_key$x=="purpose_explicit","Use Case","ungrouped"),ct_comments_important_key$group) 301 | 302 | ct_comments_important_key$group<-ifelse(ct_comments_important_key$group=="ungrouped",ifelse(ct_comments_important_key$x=="user_community"|ct_comments_important_key$x=="motivated_editors"|ct_comments_important_key$x=="active_development"|ct_comments_important_key$x=="change_on_demand"|ct_comments_important_key$x=="not_one_off"|ct_comments_important_key$x=="commit_count"|ct_comments_important_key$x=="institution_endorsement","Active Community","ungrouped"),ct_comments_important_key$group) 303 | 304 | 305 | ct_comments_important_key$group<-ifelse(ct_comments_important_key$group=="ungrouped",ifelse(ct_comments_important_key$x=="evidence_added_value"|ct_comments_important_key$x=="evidence_application"|ct_comments_important_key$x=="number_projects"|ct_comments_important_key$x=="evidence_use"|ct_comments_important_key$x=="large_database_use","Evidence for use","ungrouped"),ct_comments_important_key$group) 306 | 307 | ct_comments_important_key$group<-ifelse(ct_comments_important_key$group=="ungrouped",ifelse(ct_comments_important_key$x=="domain"|ct_comments_important_key$x=="content"|ct_comments_important_key$x=="representational_adequacy"|ct_comments_important_key$x=="represents_reality"|ct_comments_important_key$x=="realist_vs_application"|ct_comments_important_key$x=="consistent_content"|ct_comments_important_key$x=="coherency","Content","ungrouped"),ct_comments_important_key$group) 308 | 309 | ct_comments_important_key$group<-ifelse(ct_comments_important_key$group=="ungrouped",ifelse(ct_comments_important_key$x=="definitions"|ct_comments_important_key$x=="documentation_quality"|ct_comments_important_key$x=="documentation"|ct_comments_important_key$x=="ontology_visualisation"|ct_comments_important_key$x=="language","Metadata and Documentation","ungrouped"),ct_comments_important_key$group) 310 | 311 | ct_comments_important_key$group<-ifelse(ct_comments_important_key$group=="ungrouped",ifelse(ct_comments_important_key$x=="obo_principles"|ct_comments_important_key$x=="availability"|ct_comments_important_key$x=="open_source"|ct_comments_important_key$x=="standardisation"|ct_comments_important_key$x=="sustainability"|ct_comments_important_key$x=="dereferenceble_uris"|ct_comments_important_key$x=="update_strategy","Publishing and Life Cycle","ungrouped"),ct_comments_important_key$group) 312 | 313 | ct_comments_important_key$group<-ifelse(ct_comments_important_key$group=="ungrouped",ifelse(ct_comments_important_key$x=="competition"|ct_comments_important_key$x=="quality","Other","ungrouped"),ct_comments_important_key$group) 314 | 315 | ct_comments_important_key$group<-ifelse(ct_comments_important_key$group=="ungrouped",ifelse(ct_comments_important_key$x=="hierarchy"|ct_comments_important_key$x=="classification"|ct_comments_important_key$x=="defined_dependencies"|ct_comments_important_key$x=="minimal_dependencies"|ct_comments_important_key$x=="property_selection_strategy"|ct_comments_important_key$x=="logically_sound","Representation","ungrouped"),ct_comments_important_key$group) 316 | 317 | ct_comments_important_key$group<-ifelse(ct_comments_important_key$group=="ungrouped",ifelse(ct_comments_important_key$x=="user_friendly"|ct_comments_important_key$x=="comprehensibility"|ct_comments_important_key$x=="difficulty_use"|ct_comments_important_key$x=="data_capture_support"|ct_comments_important_key$x=="data_migration_support","Usability","ungrouped"),ct_comments_important_key$group) 318 | 319 | agg<-aggregate(ct_comments_important_key$freq,by=list(ct_comments_important_key$group),sum) 320 | agg<-agg[order(-agg$x),] 321 | #print(xtable(agg,digits=c(0,0,0)),include.rownames=FALSE) 322 | 323 | keywords_other<-kw[!(kw %in% keywords)] 324 | 325 | ct_comments_other_key<-count(keywords_other) 326 | kable(ct_comments_other_key[order(-ct_comments_other_key$freq),]) 327 | 328 | ``` 329 | 330 | # Systematic review of MIRO compliance 331 | ```{r papers1} 332 | paper_metadata<-c("Name","URL","Domain","Language","License","Repository") 333 | df_paper_metadata<-df_paper_results[paper_metadata] 334 | df_paper_coding<-df_paper_results[, !names(df_paper_results) %in% paper_metadata] 335 | latex_paper_metadata<-df_paper_metadata[c("Name","Domain","URL")] 336 | #print(xtable(latex_paper_metadata,digits=c(0,0,0,0)),include.rownames=FALSE) 337 | 338 | df_coding_melt<-melt(df_paper_coding,id.vars = "Code") 339 | df_coding_melt$value<-as.factor(df_coding_melt$value) 340 | ggplot(df_coding_melt[df_coding_melt$variable %in% c("Ontology.name","Ontology.owner","OBO.Principles"),],aes(x=value,fill=value)) + geom_bar()+facet_wrap("variable")+ theme_bw() + scale_fill_grey() 341 | ``` 342 | 343 | ```{r papers2, fig.height=15} 344 | ggplot(df_coding_melt[!(df_coding_melt$variable %in% c("Ontology.name","Ontology.owner","OBO.Principles")),],aes(x=value,fill=value)) + geom_bar()+facet_wrap("variable",ncol = 3) + theme_bw() + scale_fill_grey() 345 | ggsave(paste(chartdir,"results_coding.pdf",sep = ""), width = 8, height = 10) 346 | ``` 347 | 348 | ```{r papers3} 349 | df_coding_melt$value<-as.numeric(as.character(df_coding_melt$value)) 350 | agg_compliance<-aggregate(df_coding_melt$value,by=list(df_coding_melt$variable),FUN=mean) 351 | agg_compliance$compliance<-ifelse(agg_compliance$Group.1=="Ontology.name"|agg_compliance$Group.1=="Ontology.owner",agg_compliance$x/2,agg_compliance$x); 352 | agg_compliance$compliance<-round(agg_compliance$compliance*100,2) 353 | names(agg_compliance)<-c("MIRO item","x","Compliance") 354 | write.csv(file = "agg_compliance.csv",agg_compliance) 355 | 356 | # AT THIS POINT, A DOCUMENT WAS CREATED MANUALLY WITH THE VALUES FOR COMPLIANCE AND RATINGS, AS MANUALLY MATCHING THEM TURNED OUT IMPOSSIBLE. FOR THAT, agg_compliance and priority_by_group (the mean_all column only) where merged together. 357 | df_cvr<-read.csv(file=ratingvcompliance_f,head=TRUE,sep=",",stringsAsFactors = FALSE) 358 | 359 | ``` 360 | 361 | ## Compliance vs Ratings 362 | ```{r compliancevrating} 363 | df_cvr$rating_cat<-ifelse(df_cvr$rating<3,"Optional",ifelse(df_cvr$rating<3.5,"Should","Must")) 364 | df_cvr$compliance_cat<-ifelse(df_cvr$compliance<20,"Very Low",ifelse(df_cvr$compliance<50,"Low",ifelse(df_cvr$compliance<80,"Medium","High"))) 365 | df_cvr$cat<-paste(substring(df_cvr$rating_cat, 1, 1),substring(df_cvr$compliance_cat, 1, 1),sep="") 366 | df_cvr$compliance_cat<-factor(df_cvr$compliance_cat, levels = c("Very Low","Low","Medium","High")) 367 | df_cvr$rating_cat<-factor(df_cvr$rating_cat, levels = c("Optional","Should","Must")) 368 | 369 | d_paper<-df_cvr[!is.na(df_cvr$rating),][c("miro_item_rating","rating","compliance","cat")] 370 | d_paper<-d_paper[order(-d_paper$compliance,-d_paper$rating),] 371 | #print(xtable(d_paper,digits=c(0,2,2,2,0)),include.rownames=FALSE) 372 | 373 | ct_cat<-plyr::count(d_paper$cat) 374 | ct_cat$pc<-round((ct_cat$freq/nrow(d_paper))*100,2) 375 | names(df_cvr)<-c("miro_item_rating", "Rating", "miro_item_comp", "mean_compliance", "Compliance", "Rating category", "Compliance category") 376 | ggplot(df_cvr,aes(y=Rating,x=Compliance,label=miro_item_comp,colour=`Rating category`)) + geom_point() +geom_text(aes(label=miro_item_comp),hjust=0.1, vjust=-0.6, size=3) 377 | ggsave(paste(chartdir,"scatter_compvrate.pdf",sep = ""), width = 12, height = 4) 378 | 379 | ggplot(df_cvr[!is.na(df_cvr$Rating),],aes(x=`Compliance category`,fill=`Rating category`)) + geom_bar() 380 | ggsave(paste(chartdir,"bar_cat_compvrate.pdf",sep = ""), width = 8, height = 2.5) 381 | ``` -------------------------------------------------------------------------------- /supplementary/analysis/jbms_paper_results.csv: -------------------------------------------------------------------------------- 1 | Code,Ontology name,Ontology owner,Ontology licence,Ontology URL,Ontology repository,Methodological framework,Need,Competition,Target Audience,Scope and Coverage,Community,Email list and issue tracking,Source knowledge,Raw data location,Feature prioritisation,KR language,Development environment,Ontology metrics,Imports and interoperability,Entity naming convention,Identifier generation policy,Entity metadata policy,Upper ontology,Ontology relationships,Axiom patterns,Dereferencable IRIs,Sustainability plan,Entity Deprecation Strategy,Versioning Policy,Testing,Evaluation,Example use,Institution Endorsement,Logical consistency,Evidence use,OBO Principles,URL,Name,Language,Domain,License,Repository 2 | P1,2,1,0,1,1,1,1,1,1,1,1,0,1,1,1,1,0,0,1,0,0,1,0,1,1,0,0,0,0,0,1,1,0,1,0,0,http://loted.eu/ontology,LOTED2,OWL,Public Procurement,NA,https://code.google.com/archive/p/loted2/ 3 | P10,2,1,0,0,0,1,1,0,1,1,1,0,1,1,1,1,1,1,0,0,0,0,1,1,1,0,0,0,0,0,1,0,0,1,0,0,https://sbmi.uth.edu/ontology/project/viso.htm,Vaccine Information Statement Ontology (VISO),OWL,Vaccine,NA,NA 4 | P11,2,1,1,1,1,1,1,1,1,1,1,1,1,0,0,1,1,1,1,1,0,0,1,1,1,0,0,0,0,0,1,1,0,0,1,2,http://purl.bioontology.org/ontology/INO),Interaction Network Ontology (INO),OWL,Vaccine,CC,https://sourceforge.net/projects/ino/ 5 | P12,2,1,0,1,1,1,1,1,1,1,1,1,1,1,0,1,1,1,1,0,0,1,1,1,1,0,0,0,1,0,0,1,0,1,1,1,http://www.ebi.ac.uk/cmpo/,Cellular Microscopy Phenotype Ontology (CMPO),OWL,Imaging,NA,https://github.com/EBISPOT/CMPO 6 | P13,2,1,1,1,1,1,1,1,1,1,1,1,0,0,1,1,1,1,1,1,1,1,1,1,1,0,0,0,0,0,0,1,0,1,0,1,http://purl.obolibrary.org/obo/ncro.owl,Non-Coding RNA Ontology (NCRO),OWL,Biology,Open,https://github.com/OmniSearch/ncro 7 | P15,2,1,1,1,1,1,1,0,1,1,0,1,1,0,0,1,1,1,1,1,0,1,1,1,1,0,0,0,0,0,1,1,0,1,0,1,https://github.com/vicoontology/VICO,Vaccination Informed Consent Ontology (VICO),OWL,Vaccine,Creative Commons 3.0 License,https://github.com/VICOontology/VICO 8 | P16,1,1,1,1,1,1,1,1,1,1,1,1,1,0,0,1,1,1,1,1,0,1,1,1,1,0,0,0,0,0,0,1,0,1,0,2,http://purl.obolibrary.org/obo/MicrO.owl,Ontology of Prokaryotic Phenotypic and Metabolic Characters (MicrO),OWL,Microbiology,CC-BY,https://github.com/carrineblank/MicrO 9 | P18,1,1,0,1,0,1,1,1,1,1,1,0,1,0,1,1,1,0,1,0,0,0,0,0,1,0,0,0,0,0,0,1,0,0,0,0,http://purl.org/net/exergame/ns#,Exergame Ontology (EO),OWL,Serious Games,NA,NA 10 | P19,2,2,0,1,1,1,1,0,1,1,1,1,1,0,0,1,1,1,1,0,0,0,1,1,1,0,0,0,0,1,1,1,0,1,1,1,https://github.com/IEDB/MRO,MHC Restriction Ontology (MRO),OWL,Immunology,NA,https://github.com/IEDB/MRO 11 | P2,1,1,0,1,1,1,1,1,1,1,1,0,1,0,1,1,0,1,1,0,0,0,0,1,1,1,0,0,0,1,1,1,0,0,1,0,http://contsem.unizar.es/def/sector-publico/pproc.html,Public Procurement Ontology (PPROC),OWL,Public Procurement,CC Attribution ShareAlike 4.0 International (website only),https://github.com/pproc 12 | P3,2,1,0,1,0,1,1,1,1,1,1,0,1,0,1,1,0,1,1,0,0,0,1,1,1,0,0,0,0,0,1,1,1,0,1,0,http://vroddon.sdf-eu.org/mco/mco-core/,MPEG-21 Media Contract Ontology (MCO),OWL,Media Contracts,NA,NA 13 | P4,2,1,0,1,0,1,1,1,1,1,1,0,1,0,1,1,0,0,1,0,0,0,0,1,1,0,0,0,0,1,1,1,0,0,1,1,http://www.sparontologies.net/ontologies/doco/source.html,Document Components Ontology (DoCO),OWL,Semantic Publishing,NA,NA 14 | P6,2,1,0,1,0,1,1,1,1,1,1,0,0,0,1,1,1,1,1,0,0,0,1,1,1,0,0,0,0,1,1,1,0,1,1,0,http://www.dmo-foundry.org/,Data Mining OPtimization Ontology (DMOP),OWL,Semantic Meta Mining. Data Mining,,NA 15 | P8,2,1,0,1,0,1,1,1,1,1,1,0,0,0,1,1,1,1,0,0,0,0,0,1,1,0,0,0,0,0,1,1,0,1,1,0,https://mybody.inrialpes.fr/mycfembryo/,My Corporis Fabrica Embryo Ontology (MyEF),RDFS,"developmental anatomy, 3D",,NA 16 | P9,2,1,0,1,0,1,1,1,1,1,1,0,1,1,1,1,1,1,1,0,0,0,1,1,1,0,0,0,0,0,0,0,0,0,0,0,http://bio-omix.tmd.ac.jp/periodontitis/,Periodontitis-Ontology (PeriO),OBO,Disease,NA,NA 17 | -------------------------------------------------------------------------------- /supplementary/analysis/jbms_results.csv: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/owlcs/miro/8dfb8fe40449723d24805f61dd7fde76dbb8ec18/supplementary/analysis/jbms_results.csv -------------------------------------------------------------------------------- /supplementary/analysis/rating_v_compliance.csv: -------------------------------------------------------------------------------- 1 | miro_item_rating,rating,miro_item_comp,mean_compliance,compliance 2 | Content: Development environment,2.878504673,Development.environment,0.733333333,73.33 3 | KA: Source knowledge location,3.357798165,Raw.data.location,0.266666667,26.67 4 | KA: Content selection,3.37962963,Feature.prioritisation,0.666666667,66.67 5 | Content: Ontology metrics,3.422018349,Ontology.metrics,0.8,80 6 | Content: Entity naming conventions,3.743119266,Entity.naming.convention,0.266666667,26.67 7 | SRD: Development community,3.768518519,Community,0.933333333,93.33 8 | Change: Versioning policy,3.798165138,Versioning.Policy,0.066666667,6.67 9 | SRD: Communication,3.798165138,Email.list.and.issue.tracking,0.4,40 10 | Change: Entity deprecation strategy,3.834862385,Entity.Deprecation.Strategy,0,0 11 | Motivation: Need,3.851851852,Need,1,100 12 | Content: Identifier generation policy,3.862385321,Identifier.generation.policy,0.066666667,6.67 13 | QA: Testing,3.871559633,Testing,0.266666667,26.67 14 | Content: Upper ontology,3.880733945,Upper.ontology,0.666666667,66.67 15 | Content: Entity metadata policy,3.889908257,Entity.metadata.policy,0.333333333,33.33 16 | Change: Sustainability plan,3.889908257,Sustainability.plan,0,0 17 | KA: Source knowledge,3.926605505,Source.knowledge,0.8,80 18 | Motivation: Target audience,3.944954128,Target.Audience,1,100 19 | Motivation: Competition,3.963302752,Competition,0.8,80 20 | QA: Evaluation,3.990825688,Evaluation,0.666666667,66.67 21 | Basics: Ontology repository,4.009174312,Ontology.repository,0.533333333,53.33 22 | Content: Incorporation of other ontologies,4.091743119,Imports.and.interoperability,0.866666667,86.67 23 | Content: KR language,4.110091743,KR.language,1,100 24 | Content: Ontology relationships,4.128440367,Ontology.relationships,0.933333333,93.33 25 | SRD: Scope and coverage,4.148148148,Scope.and.Coverage,1,100 26 | QA: Examples of usage,4.19266055,Example.use,0.866666667,86.67 27 | Basics: Ontology license,4.504587156,Ontology.licence,0.266666667,26.67 28 | Basics: Ontology owner,4.527777778,Ontology.owner,1.066666667,53.33 29 | Basics: Ontology name,4.706422018,Ontology.name,1.8,90 30 | Basics: Ontology URL,4.71559633,Ontology.URL,0.933333333,93.33 31 | Content: Axiom patterns,3.796296,Axiom.patterns,1,100 32 | ,,Dereferencable.IRIs,0.066666667,6.67 33 | ,,Evidence.use,0.533333333,53.33 34 | ,,Institution.Endorsement,0.066666667,6.67 35 | ,,Logical.consistency,0.6,60 36 | ,,OBO.Principles,0.6,60 37 | ,,Methodological.framework,1,100 38 | -------------------------------------------------------------------------------- /supplementary/questionnaire_comment_coding.csv: -------------------------------------------------------------------------------- 1 | id,important_comments,important_comments_code,important_comments_code3,important_comments_code5,important_comments_code2,basics_comments,basics_comments_code,basics_comments_code2,motivation_comments,motivation_comments_code,motivation_comments_code2,scope_comments,scope_comments_code,scope_comments_code2,ka_comments,ka_comments_code2,ka_comments_code,content_comments,content_comments_code4,content_comments_code3,content_comments_code2,content_comments_code,change_comments,change_comments_code,qa_comments,qa_comments_code3,qa_comments_code2,qa_comments_code,other_comments,other_comments_code,,,,,, 2 | 20a309a1c35d12f6fb51be8299798c2f,Is it open. ,,,,open_source,How to cite the ontology,citation,,Example use cases for the ontology,,example_use,,,,,,,,,,,,,,,,,,,,,,,,, 3 | a355790aadbbbf3ea34140fc1668fd9a,"does it fit my requirements, and if not, what are the obstacles to overcome?",use_case_adjustability,,,use_case_suitability,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,, 4 | 7f011a78ff47ddebd9e73807f7967dd7,"A very defined and minimal set of dependencies, so that I do not end up importing the entire world.",defined_dependencies,,,minimal_dependencies,,,,,,,,,,,,,"You missed Tawny-OWL in editing environments, curse you.",,,,tawny_owl,,,"No point at all asking whether an ontology fulfils it's original requirements -- people will just say ""yes"" and change the requirements to fit.",requirements_dishonest,,,,,,,,,, 5 | dc2619f1079ab33c673baf8f3a3307cd,whether it is being constantly updated or not,update_strategy,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,, 6 | ee3bd5dfdabb13478d8cd27d1d4e5884,Coverage,coverage,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,, 7 | 43d89a6a28ecbfe07eb8c2a9bf2b4235,Support for migration of existing data and of data capture and analysis artefacts seems to be the sine qua non,data_migration_support,,,data_capture_support,"Should include the version number(s) for which the report holds valid. Also, if the ontology includes optional extensions and localisations such as term translations or optional classes, information on the configuration being reported on and which versions of optional components were used.",configuration_optionals,version_number,"If the ontology has multiple sponsors, and where those sponsors may differ in their motivation, a description of the processes by which their differing development priorities are managed.",development_priority_management,stakeholder_motivation,The scope and requirements an ontology should change over time. Whilst its useful to describe the current scope and requirements - e.g. point to a high level 5 year development plan - a report could also describe any governance processes that control how scope and requirements can change.,,governance_process_scope_requirements_change,Full disclosure of how knowledge was elicited and prioritised could be an onerous requirement for the larger ontologies to create and the reader to absorb. Quite likely also not very edifying for either due to sheer scale.,,ka_full_disclosure_toomuch,Large and complex ontologies can not realistically briefly document all metamodel patterns and rules that the ontology-proper must follow.,,,,metamodel_patterns_toomuch,"Change Management discussion should include extent to which there is support for users of other older or competing ontologies wishing to migrate to the one being reported (mappings, data migration tools, data cleansing etc).",ontology_migration_support,,,,,,,,,,,, 8 | 04dbf43ef51799c04d3500c73133dbfa,Extent of coverage,coverage,,,,,,,"I've put five stars for all of these, but really the ""competition"" one is the most important.",competition,,,,,,,,,,,,,,,,,,,,,,,,,, 9 | 049babf76d90c11510ae19862e4d5e5f,"Evidence that the ontology was not just a ""one-off"" type of project.",not_one_off,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,, 10 | df6678e0b5818eab7623e93fcdc50674,quality,quality,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,, 11 | 466d0cd94064a115abef38f73494edb8,Does the ontology's content and organization such that it adds significant value to my work.,evidence_added_value,,,,The competency questions that the ontology is designed to address,cqs,,,,,,,,,,,,,,,,,,,,,,,,,,,,, 12 | a33cf1e1a8ce85129ec9c0201fe597ae,Utility for one or more tasks at hand.,,,,use_case_suitability,,,,"It is nice to see plausible use cases in ontology papers, and if there is any ""elegant"" modelling involved, the assumptions and modelling decisions should be described.",example_use,justify_modelling_decisions,"There is still no good ""ontology website in a box"" software, and probably the majority of ontologies have sucky websites (I would not exclude a certain ontology I know well, but we are trying!). ",,out_of_box_ontology_website,,,,"Important. I have seen few papers that do a good job explaining much of this, e.g., where does an upper ontology really help and what this or the other one was chosen.",,,,justification_upper_ontology,,,,,,,,,,,,,, 13 | 908e582489e30e9fe477fd55da5aacdd,definition of terms,,,,definitions,Report issue tracker,report_issue_tracker,,,,,,,,,,,,,,,,,,,,,,,,,,,,, 14 | 2c2f327c91038e5e0989a87894a20cf0,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,, 15 | daaf9589695b7ce98ccad8fb13a15810,Logically sound modeling.,,,,logically_sound,"For an OWL ontology I would consider its profile (DL, EL, RL, QL) as a fundamental ""basic"" piece of information to be included here.",owl_profile,,,,,"Some ontologies are designed from scratch to stay within a specific profile (e.g., EL) for computational benefits. Perhaps an extra question for this section is: was the modeling compromised by computational requirements? As in, did one trade-off modeling completeness for performance.",,tradeoff_performance_representation,,,,"I'm not 100% sure what you mean by ""relationships"" in question 31. Do you mean the kinds of OWL restrictions used (e.g., existential, cardinality restrictions...)? I answered with that interpretation. I would like to know the expressivity of the ontology, as well as the (rough) frequency with which particular constructs were used.",,,construct_frequency,expressivity,I would like to know what tools were used for inspecting and managing changes.,tools_changemanagement,Frameworks/tools used for quality assurance.,qa_tools,,,,,,,,,, 16 | bf24699756d1ae7ad6ba724c6e0edc35,how well we can describe the knowledge using this ontology.,,,,representational_adequacy,,,,,,,,,,,,,,,,,,,,,,,,"Every new ontology being proposed or submitted, should be asked to give a flash-talk, poster and submit short(4-page) paper at the ICBO of that or the following year. I suggest creating a section for it. That's the best way to highlight the new submissions. It can start from ICBO 2016.",all_new_ontologies_at_icbo,,,,,, 17 | fe7a903d630cb0e87d2784553af3e862,Yikes. Very hard to decide just one thing. I'd have to say available under an open license like CC. Otherwise can't even begin to asses other stuff.,,,,open_source,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,, 18 | 8a82d0c0f84f93989e6ad5d5f86d86a7,Fitness for purpose,,,,use_case_suitability,"Version of the ontology used/described, e.g., data, version number, etc.",,version_number,,,,,,,,,,,,,,,,,,,,,,,,,,,, 19 | 87d5439ab2d651e89f08bb0422dab215,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,, 20 | d4b10cce66a70ebc6c8576a47be553e3,Applicability to the intendend use with properly and thoroughly thought terms.,,,,use_case_suitability,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,, 21 | 919b61376bf098c1e0198c51dd873e9d,coverage of concepts and relations required.,coverage,,,,publication that describes the ontology and its utility in the domain.,citation,use_case_suitability,,,,,,,,,,,,,,,,,,,,,,,,,,,, 22 | 30965fc0e481c6b1f12875535bc4ff96,Active ontology development,active_development,,,,,,,,,,,,,,,,,,,,,,,,,,,"- availability in various formats (e.g. OBO, OWL, RDF) is important 23 | - how should I cite the ontology I am interested in? 24 | - coverage: a ""shallow"" ontology is not interesting... 25 | - examples should be carefully chosen 26 | - interoperability with other ontologies 27 | - roadmap: where they want to go in the future? 28 | - I hate ontologies that don't reuse others ...",available_formats,coverage,example_use,compatibility_other_ontologies,roadmap,citation,justification_notreusing 29 | df10030c693f170231c784523245cd50,Exact purpose and real applications of the ontology.,evidence_application,,,purpose_explicit,,,,The application(s) for which this ontology has been primarily developed.,example_use,,"Typical application scenario, interoperability with other ontologies, is it meant to be a standard,list of known applications already existing, typical questions the ontology is able to answer to.",compatibility_ontologies,example_use,Which databases can the ontology connect to?,,connect_database,,,,,,,,,,,,,,,,,,, 30 | ffc5d5bcb841883c7b46410b143f39a5,"I would want it to be in OWL. If we move past this most basic of requirements, then the one thing I would choose was good integration with other appropriate ontologies.",language,,,compatibility_other_ontologies,"Primary format and ancillary formats, e.g. developed in OWL but available in OWL and OBO... ",language,,,,,,,,,,,,,,,,,,,,,,"As it is a minimal information document that you are trying to build, I assigned many important features as 3 star, or unimportant to a MI document. MI documents are about reproducibility and clear understanding for other groups making use of any given ontology. As much as I think entity identifier generation policy and deprecation policies are supremely useful, I think enforcing these in a MI document might be troublesome. You just need a very few items in a MI document - but perhaps you can have a secondary, not-so-minimal document that would be a ""best practices"". In any case, the implementation of the MI document - how easy it is for people to provide the information - will be the main determining factor in its success, IMHO. Good luck - sounds like lots of fun to write! :)",identfier_generation_policy_toomuch,deprecation_policy_toomuch,,,,, 31 | 812d51ea1a7f71fea68c4ad93fbb0ce7,Authors requirements match mine.,,,,use_case_suitability,,,,,,,"The requirements should be made explicit, and referred back to in the evaluation section.",requirements_explicit,,"If ontologies rely on expert knowledge rather than books and databases, these experts should be either named or at least described.",,name_experts,"One of the key reporting features is expressivity: RDFS, one of the OWL Profiles or perhaps even description logic (ALC, etc). That way, users get an immediate sense of the complexity of the ontology, and an idea of which reasoners, if any, they might need to employ.",,,owl_profile,expressivity,"I think every ontology should have a DOI, that links to a website providing information about its development and content.",doi,"This is the most important part I believe. There should be a clear relationship between the original requirements and the evaluation. If reasoning might be needed, obtain some classification times. If the target was to classify genetic profiles, show that it does and how the ontology is used to generate additional benefits. To be very honest, while I understand that ontology description papers are extremely common now, without evaluation, they should not be considered for a conference or a journal.",requirements,reasoning_time,evaluation_crucial,"While this survey software is beautiful, it is not the most usability proved one :P",surveytool_problem,,,,,, 32 | 58238aa9cede4db381709dcb2a131380,"Consistency, in both content and design.",consistent_content,,,logically_sound,Initial release date?,release_date,,Would be good to require that the main motivational statement should feature one or more use cases (might also be a good way to clarify need when there is 'competition').,example_use,competition,The first two scope questions seem redundant against the motivational statements required just before.,,,"This kind of stuff is interesting and useful and seems suited to a paper, but that's (to my mind) not the same as an MI list. In a notional RDF record accompanying the paper, would this sort of thing go in?",,unclassified,,,,,,,,,,,,"I'd like to see some kind of high-level diagram included amongst the requirements (maybe as a link rather than in the paper, but somewhere).",visualisation,high_level_diagram,,,,, 33 | 192efc10731b36c483566cbfbc9f6108,"Are there editors who are bright and responsive? A broken ontology with intelligent, responsive editors can be fixed. A well structured ontology that can't adapt to user needs can be problematic.",motivated_editors,,,use_case_adjustability,,,,,,,,,,,,,Interested to know if any use of formal design patterns (Q32 kind of covers this). Also important to know how any design patterns fit with intended patterns of inferred classification.,,,,,Is inference used to manage classification sustainably as the ontology grows?,use_of_inference_manage_classification,,,,,,,,,,,, 34 | 11a05f97dc66713ba863d0b5c60c479b,Content and scope,scope,,,content,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,, 35 | 298ddddd3b46663fa1b4b3102e5c4ac1,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,, 36 | 1d53af0a5af4895ed8e5c5deebd69d90,,,,,,,,,,,,,,,,,,,,,,,,,,,,,"Please bear in mind that I am completing this as someone researching representation languages and someone who reviews papers, rather than a user of ontologies.",unclassified,,,,,, 37 | 8dfd8269045699f93221cfcb2e8ad2dc,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,, 38 | cc985dd0193d9582d044646f11aff4f9,Coverage,,,,coverage,Frequency of updates/releases and if still actively developed,release_frequency,,if aligned to any particular dataset.,dataset_alignment,,,,,,,,,,,,,,,,,,,,,,,,,, 39 | 398a4eac3293571c3b28402a37d76d8a,Sustainability,,,,sustainability,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,, 40 | 399309d7494d3433a8b5303b1693405c,"The size, committment, and quality of community that is using it.",,,,user_community,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,, 41 | 6e9cdb532bb5956be4177a80501bbc9a,Coverage,,,,coverage,,,,"The intended target audience is almost always a bit off, sometimes by a long shot. Audience and type of usage is typically hard to guess, for all standards.",example_use,target_audience,"For scope, cf previous comment on audience.",target_audience,,,,,,,,,,,,,,,,,,,,,,, 42 | 8ccd689e03b50e21b281f3d194d5388d,,,,,,,,,"From an academic perspective, I would definitely want to know about existing competing ontologies but from a more general perspective, I would class this as slightly less absolutely barebones essential than the other aspects so far.",competition,,,,,"Depending on the size and scope of the ontology, this information might be more self-evident in some cases than in others. Should still be covered by level of detail will depend on ontology scope and subject.",,unclassified,,,,,,,,,,,,,,,,,,, 43 | 14ae1b0ef54c86404e277d608284603d,Responsiveness of the developers.,motivated_editors,,,,"I'm thinking that the basics cover information required to decide whether an ontology might be of use, so the maintainers and history are not quite so important as where to find it.]\",use_case_suitability,location,"Motivation for why an ontology is needed is usually a major part of an introduction/background section anyway and since most ontology papers include a certain amount of promotion, it isn't as critical in evaluating a paper. The question of this ontology vs. a competing ontology is more important, not so much in the details, but that the authors are aware of the 'competition.' Not knowing suggests a lack of thoroughness on the authors' part.",competition,,"Engagement is important if it is active, but I won't reject an ontology paper if engagement activities are in the 'ramping up' stage. Email list and tracking information are important, but should be easy to find once the ontology has been located and tried (e.g., associated with or documented on the home page of the ontology or its repository).",homepage,,"Details on construction important, but locating the raw data doesn't need to be in the publication as long as it can be found linked to the ontology home page or repository.",,raw_data_location_toomuch,"Tools, shouldn't be important, and language only if nonstandard (e.g., Protege Frames, Common Logic, etc.) Otherwise, content is important, but secondary to the question of where the ontology is and what is required to load/review it. Conventions are important, but mostly to demonstrate that they are within reasonable standards and were followed.",,,ontology_location,tool_explore_ontology,"Sustainability is nice, but may be optimistic. Ontologies shouldn't be tied to a specific project (might discourage potential reuse) - if they are, they should be described in the larger project context, rather than as publications on their own.",sustainability,I wish there was more of this.,unclassified,,,,,,,,,, 44 | a68f1498d57a24395ee51f610c9c8e0c,The selection of properties used to describe an ontology class,property_selection_strategy,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,, 45 | 344207ccd1ebe29d8e4a603680bd08df,"actively developed, can get terms added if needed",active_development,change_on_demand,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,, 46 | 92221d73fac45f40c26e5c9652874a76,Content of the ontology,,,,content,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,, 47 | 0612365893572648d41aebc3856c78a7,Availability in an ontology implementation language ,,,,language,For me a short description is quite basic as well. ,,,,,,,,,,,,,,,,,,,,,,,Great work! Looking forward to see the results and apply/reuse them.,,,,,,, 48 | c0463886b035c1a2b28caea94fa16bc4,documentation,,,,documentation,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,, 49 | eb081932c9e08cdf43b871d22b496f55,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,, 50 | 7876b42ec1aa801bb4d7c0196691b878,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,, 51 | a9c8812757606777f1080f4b1eb665cb,Specific to my domain with sufficient number of concepts and properties,,domain,,coverage,,,,,,,,,,"If there was prior version of ontology, it source may be specified. Plus, when it was updated, what concepts were added etc.",,version_history_plus_metadata,,,,,,,,,,,,,,,,,,, 52 | 4b7ef5ee978d882a20e0eaf4b4d58bcf,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,, 53 | 4c0a9dc755c3cd510cb3cd470d6810c9,The extent of the documentation.,,,,documentation_quality,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,, 54 | 1812d66420c2c43a0dd1d9fb223adb43,,,,,,"Full name is important, abbreviation is best left as a local decision depending on use.",name,,,,,,,,,,,If you are using OBO it is probably a vocabulary not an ontology ;) ,,,ontology_vs_vocabulary,,,,,,,,Can your ontology be used to infer any new and interesting 'facts',interesting_inferences,,,,,, 55 | 373ebb85b8bfee85233043a2fb97b082,"Derreferenceable with content negotiation, and well documented",dereferenceble_uris,,,documentation_quality,"A title should be made available. A description should be made available. And also, the namespace prefix and namespace URI should be declared with the vann vocabulary, if possible.",name,vann_uri,,,,,,,,,,,,,,,,,,,,,,,,,,,, 56 | d569c08cd4a2d1ad7438da39e17977b7,URIs that resolve,dereferenceble_uris,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,, 57 | 4cbce673378b0174ca4c1b58a5129bd6,Interoperabililty.,interoperability,,,,,,,Use-case examples.,example_use,,Email list and issue tracking should be included in the basics.,email_list,issue_tracking,,,,,,,,,,,,,,,,,,,,,, 58 | 5ebd176a6bd835205bfba8767f8bbbfb,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,, 59 | d176050900b0c650483f16cf3c34fb6e,,,,,,Ontology visualization on the page is important.,visualisation,,,,,,,,,,,,,,,,,,,,,,,,,,,,, 60 | c84ac4dd2c8c8d2e1beab783a50e4af6,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,, 61 | 607fc2592f7f0ca1bcce19f457677f2a,Level of details described,,,,granularity,"In my opinion, one of the most important basic ontologies features is its dereferenceability, so that machines can easily access and use the ontology automatically",dereferencibility,,,,,Support and maintenance of an ontology is of the utmost importance in order to implement this ontology in serious applications.,support,active_development,,,,The content should be as self-descriptive as possible so no external references are required,,,content_self_descriptive,,,,"There are prominent tools for assessing the quality of an ontology, e.g., OntoClean. It would be nice to satisfy all the requirements of the highest quality.",ontoclean,qa_tools,,,,,,,,, 62 | 03ce83a30f7d43d96165df99f0e9b137,How much overhead does using this ontology add to my workload. Will it solve more problems then it creates.,difficulty_use,,,,If the ontology is under current development or is static. ,active_development,,,,,,,,,,,,,,,,,,,,,,Choosing to use an ontology as a developer has a different set of requirements to using it as a consumer. As a developer I want an ontology I x-reference to be developed using best practices and be maintained. As a consumer I want the terms I need for annotation there quickly and to be simple to find.,features_depend_on_usergroup,,,,,, 63 | 1afa408b32daa5d55e6041b86f037241,coverage,,,,coverage,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,, 64 | 4f6fb0b6f72af890f9119b2064f84d7c,completeness (at least not gross ignorance),,,,coverage,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,, 65 | 29431f01d6464aa68deeaaa4fd796bb9,Does it cover things I need?,,,,coverage,Important to understand the process for requesting updates to an ontology.,update_request_process,,,,,,,,,,,,,,,,,,,,,,,,,,,,, 66 | a5f02207922bfd24579a6e47eca87e47,"Number of contributors/""commits"". This is a proxy to the by-in, and health of the ontology.",active_development,commit_count,,,,,,Orthogonality is nice but should not prevent ideas from being explored. Requiring proof of orthogonality will ultimately stifle ontology development. Encouraging orthogonality however is highly recommended.,orthogonality,,Ultimately one or a few individuals champion the day-to-day curation of an ontology. Specifics as to the editor/editorial panel might be useful.,editor_info,active_development,,,,,,,,,"Deprecation is hugely important. Understanding why we do this (deprecate) should be required for those developing ontologies, absence of information on this suggests the ontology authors may not fully appreciate its importance.",deprecation_management,"Quality, utility etc. may evolve over time, it's not as critical to demonstrate both the ontology and utility, and in fact at the outset impossible in many cases.",qa_toomuch_inbeginning,,,,,,,,,, 67 | e602297fbd2aa292309ebc7211821f68,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,, 68 | e678e49be295cd7a7dc8f912ff40b732,Interoperability,interoperability,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,, 69 | 9a4f1181ecf48933806ec4c0c9ff68a9,Degree of interdisciplinarity,interdisciplinary,domain,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,, 70 | a8e74ca1d435d597f5a13b8f743068d5,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,, 71 | dd7fc058f3465bb945a7f577c1550751,scope and quality,scope,,,quality,Version of imported external ontologies,versions_imports,,,,,,,,,,,,,,,,,,,,,,,,,,,,, 72 | adfaea85949fdd98faa163a9a51745b8,Suitability for use in my specific problem at-hand,,,,use_case_suitability,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,, 73 | 20372827e3579d74e601331823b38d74,Standardization,,,,standardisation,Some information on usage.,evidence_use,,,,,,,,,,,,,,,,,,,,,,,,,,,,, 74 | 138c6993d0c48ae1a51b11f073ea0ac5,"Consitent, Concise, and Complete (with definitions)",definitions,coverage,,logically_sound,Please include all W3C recommendations for annotating ontologies,w3c_recommendation_annotation,,Not as important as the use cases that need to be addressed first.,example_use,,Not all ontologies are developed by a community though it may be intended for a community.,,,"It appears that there is an assumption that knowledge must be elicited. Oftentimes, it is gathered from official documents.",,ka_document_source,"Need to add annotation properties to accommodate synonyms, abbreviations, acronyms, etc. ",,rich_entity_annotations,rich_entity_annotations,,Please refer to W3C recommendations even when OBO is used,w3c_recommendations,Use of OWL-DL reasoning for consistency checking and inferencing,consistency,inference,,Please send a summary report on what I have entered. liju.fan@ofr.treasury.gov,unclassified,,,,,, 75 | 2635150d94b6e516f69ae3162fe3245c,Uptake by other data providers.,,,,evidence_use,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,, 76 | b390e65b59466e1b3aaedcdd6585d2eb,compatibility with other major ontologies/vocabularies,,,,compatibility_other_ontologies,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,, 77 | 10f38c328ff48406631ee450f7a13c6c,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,, 78 | 9748f1b759532012ee91b000785ba1c3,applicability to my job,,,,use_case_suitability,,,,,,,,,,,,,how can every class in an ontology have a parent (showing my ignorance!)?,,unclassified,unclassified,,,,,,,,,,,,,,, 79 | 7f361876b1e6482cb29cf3f6f79f68bd,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,, 80 | f94e8bf7cd381cae48604244dfd92be8,Set of terms covered,,,,coverage,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,, 81 | edd3a816ffd3f15014eef8ad8573df9e,Completeness of the ontology for a particular application in a particular domain.,use_case_suitability,,,coverage,,,,,,,,,,,,,,,,,,,,,,,,"The web interface for this survey has problems with the multiple choice question (creator, user, writer, ...). On a flat screen monitor the difference between selected and un-selected is very difficult to see as the color change is subtle and sensitive to viewing angle. I am really not sure what answer was submitted (I intended checking them all).",surveytool_problem,,,,,, 82 | 4e90ba734f62965ccec75eaaf38dd129,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,, 83 | 0e7d1eb77b222de8b6f4b8261e941156,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,, 84 | 741d33242e1b42a5db3c03b365002671,"its quality in terms of completeness, comprehensibility and completeness",comprehensibility,coverage,,quality,,,,,,,,,,Not sure what you mean by 'features' in Q21.,,unclassified,,,,,,,,,,,,,,,,,,, 85 | aaa613da1faff528f3ff443c7c9dee27,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,, 86 | 894ff9bd242e2ea83ac9437d7f4b668b,If it has a backing by a serious communnity/institution.,motivated_editors,,,user_community,"Useful to know when was the last release, and if the development is still active.",active_development,version_number,Who is using it and for what purposes would more useful information.,user_community,purpose,"Not sure what ""motivation for the ontology"" means. Knowing which kind of tools are used for the development/application of the ontology would be useful (reasoner, query engines, etc.)",tools_development,tools_application,"From the perspective of an ontology user, I would not care so much how the knowledge elicitation was done. As a reviewer, I would. The aspects really depend on the actor.",,ka_interest_depends_on_user,,,,,,Having the public location for the diffs across versions would be useful.,public_diff,I don't think a regular ontology user will really care about the QA process. A reviewer certainly would.,unclassified,,,,,,,,,, 87 | eff043b01e8246ce966863333e1d8dd1,How did the ontology support the use case for which it was used? ,,,,use_case_suitability,Is the ontology under active development? When was it last updated? ,active_development,last_update_date,,,,How is it being used by the community? i.e. what kind of applications.,user_community,example_use,,,,How did the different parts of the ontology relate to the application in which it was used/developed for.,,,ontology_parts_utilisation_application,,,,"Does it classify? Seems like a no brainer, but there are too many 'ontologies' in the wild that don't. ",consistency,inference,,,,,,,,, 88 | 1edd14433e565569d90f847ae01c342f,,,,,,,,,,,,,,,,,,,,,,,,,,,,,"The questions above are all trying to ascertain the importance of a given aspect of an ""ontology report"". Unfortunately, I have no idea what an ontology report is, and, as such, am at a bit of a loss about how to answer the questions. ",unclassified,,,,,, 89 | 7dd7b2e153a39dec557e08de1015fed0,coherency,,,,coherency,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,, 90 | 05dbfb921796406fde4700cbd7d636c4,Does it follow OBO Foundry principles and use an upper ontology (like BFO) so that it can be interoperable with my ontologies and the large number of imported terms from other ontologies used in mine.,interoperability,obo_principles,,compatibility_other_ontologies,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,, 91 | c338c7771517d7761155bd142ad726ce,"I don't think an ontology can be evaluated based on one thing. It must have a sensible hierarchy, but I would not use it on that alone (SWEET ontologies have a sensible hierarchy, but I don't use them",,,,hierarchy,,,,,,,,,,,,,,,,,,,,,,,,"Everything you mentioned here is important and should be included in an ontology report. However, I gave some slightly lower scores, to give a sense of priority. Also, for a few of the characteristics, although they are generally important, I can imaging cases where they might not be needed.",everything_important,,,,,, 92 | 89e2bd3bbd97cc8c62bc92c0c6a62d7b,"Which organisation endorse the ontology (W3C, FAO, WHO, etc.)",,,,institution_endorsement,I have tons of other metadata I wish to see in an ontology. I am currently writting a paper about a review we have made about ontologies metadata. We need to talk ;),lots_of_metadata,,There are a few semantic web vocabularies available for this.,metadata_vocabulary_use,,Many.,,,Any provenance information is interested. We have isolated 6 properties from PROV in particular.,,prov_provenance_features,The relations to other ontologies are particulary important. We have isolated 21 possible types of relations in our study.,,,relation_to_other_ontologies,,"Some of these metdata cannot be plan upfront, but at least they could be ""computed""... e.g., we can say the frequency of update of an ontology after 2-3 years of existence. Thus, I do believe being able to store these metadata is important, however, I do not think this is something you can actually report on when you releae the ontology (the first time).",updatecycles_toomuch_if_new,"Giving exemples of use is according to me more important than QA measure. People will figure out if this is good or bad if they understand what is the ontology for. QA measures can be so different one from another, than it might just be useless to compare/count on them.",qa_toomuch_inbeginning,examples_more_important,,"We need to talk. We have identified several groups of interested community/organization ready to reopen the box of describing ontology metadata (e.g., Stanford, FAO, INRA, etc.)",we_need_to_talk,,,,,, 93 | 48d6a5841528f392811329c4a434a3e6,Ability to classify the data that I need classified along multiple axes.,,classification,,,"To me the most important thing is to point at the actual ontology artifact rather than a homepage; from the artifact, the contributors can be derived.",url_ontology,,"For new ontologies reported in the literature, high quality justification is needed. I read a lot of papers that I reject that lack this. Don't ontologize for the sake of ontologizing! ",justification_development,,"An ontology without a tracker is useless as a community resource. Most ontologies do not perform adequate requirements analysis, and fundamentally this is how we know it is meeting specific needs.",requirements_explicit,issue_tracking,I think competency questions and requirements analysis is often missing. Also often missing is data showing the ontology in action.,requirements_analysis,competency_questions,"The most important things to me content wise are: quality and consistent axiomatization, versioning, and human readable text descriptions/definitions. ",human_readable_descriptions,versioning,quality,consistent,We desperately need better versioning tools for complex imports.,imports_versioning,"I don't feel that most ontology evaluations that I am privy to are adequate. For example, the OBO Foundry inclusion requires evaluation, but some of the Foundry ontologies are way behind the times and don't meet current standards. Also, when I review ontology papers I am often shocked at how often the ontology is not available or doesn't meet current best practices, yet they are often published anyway, as if other reviewers are not reviewing the ontology itself?",most_evaluations_inadequate,ontology_unavailable,,"Thanks for doing the survey, I hope that it helps our community do better.",,,,,,, 94 | e4b3921efffff00880843e13a0022c51,Broad acceptance within the community so that results got from using it can be put in context and compared and contrasted with other people's work.,,competition,,user_community,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,, 95 | c766a92e05a3c05427f05fc1dbbbb370,Who is using it ,,,,user_community,,,,,,,,,,,,,,,,,,,,,,,,Tools. Are there any tools that exploit the ontology to do something useful,tools_using_ontology,,,,,, 96 | e3f9c3b099a9f74e88b771aaabe760b7,"Is it ""enumerating"" the concepts within its target portion of the reality, or is it defining an application-specific data model?",,,,realist_vs_application,"Plus ways to contribute, suggest, request changes...",issue_tracking,update_request_process,,,,,,,,,,,,,,,,,,,,,"I missed a question about how important an ""information standard"" about an ontology would be. One star. However, that doesn't mean that the proposed types of information wouldn't be useful. I just don't see a need of (yet again) trying to prescribe (and bias) them.",ontology_information_standard,,,,,, 97 | 1c898a5545e2d05437b49622eee426d8,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,, 98 | b1a9544a5010812adf54ef8e0783b2f9,Community acceptance,,,,user_community,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,, 99 | dcd8c8ca9ca4e24854777036c8d56707,Fit for purpose,,,,use_case_suitability,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,, 100 | 92159f73f859f7a3855ef7f5c08a43de,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,, 101 | 01767840fd06639d089cf8f2e4bc019f,Completeness of coverage for specified domain ,,,,coverage,,,,,,,,,,,,,,,,,,,,,,,,"Thanks for doing this. Being a relative outsider, I feel very strongly that this field desperately needs some ""rules of engagement"". Let me know if I can participate further...very interested in this topic. ",we_need_to_talk,,,,,, 102 | e48762a5b8a97a38bbdaf1c72adcd6cf,It is a true and accurate representation of reality. ,,,,represents_reality,"I like seeing ontologies in protégé desktop, but I have found some of them that rise errors like this 103 | File not found: http://www.scai.fraunhofer.de/NDDcommon 104 | 105 | I'm still not sure what that means, but I think it uses files that are not available where they should be. So I would like ontologies to be ready to be loaded to protégé, or maybe that It can be explored on the web page fully, i.e., that relations other than is_a and logical axioms can be seen on the web page.",available_imports,explore_full_ontology,,,,,,,,,,,,,,,,,,,,,,,,,,,, 106 | 2df187454391dccc601fd9669abd943d,Whether it complies with OBO Foundry principles and interoperates with other OBO Foundry ontologies.,,obo_principles,,compatibility_other_ontologies,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,, 107 | d07569ee4edb88f564297097d5863015,"one thing is hard to choose. these are all factors that are important: classes and relationships used, availability, scope in how it applies to what i need it for",coverage,availability,scope,use_case_suitability,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,, 108 | 5fde376f53ecb25e01cc890376694dfa,"Is it user-friendly? Do I feel intelligent using it or do I feel like everything needs to be explained to me? Also, a tie with, is it interoperable with other ontologies?",interoperability,user_friendly,,compatibility_other_ontologies,"If previous repositories are included, justifications on why something was changed should be included as well.",justifications_changes,,"A crucial component for ontology development is working with users during the creation process. While an ontology can be built by academics because there is a niche interest, many users don't even realize that an ontology is a missing component in their decision-making process. ",user_involvement_development,,"I think the ""why"" of the scope and requirement is important, if I am understanding the question correctly. Issue tracking is, of course, very important but not for a publication.",justification_scope,justification_requirements,"Again, the ""why"" for prioritization. Also, if raw data is included, notes on interoperability success or issues would be very helpful.",justification_prioritisation,evidence_interoperability,"I wish there was a ""depends on the audience option""! As someone who is more familiar with applications of ontologies (yet developing more understanding of its their inner workings), too much detail on content be a bit tedious. However, if the report/publication is for ontology developers, the above information is helpful.",content_maybe_toomuch,feature_scope_depends_audience,,,"The ""why"" again for ontology entities no longer dealt with. Why is there no more need? Has it been replaced or was the entity irrelevant, conceptually wrong, or dysfunctional?",justification_deprecation,"I want to give the ""who is using it and how"" 10 stars! Maybe a ""What was learned from using this ontology?"" ""What problems were solved and how?""",evidence_use,lessons_learnt,problems_solved,,,,,,,, 109 | 74a3e43cb48c3aa835f7d03a52c590da,if it fits my requirements,,,,use_case_suitability,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,, 110 | baa836191e7930986f71f7f725d9a392,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,, 111 | c19e9e08cc2015ebad49ae2b72f06b33,A look at the ontology itself.,,,,ontology_visualisation,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,, 112 | a73ddb06fd3daae569a45e965dae5443,the fit with my specific use case,,,,use_case_suitability,Some documentation including examples of how and when the ontology should be used,example_use,,,,,"The motivation for most new ontologies is that ""the ontology available in the public domain did not fit the specific needs"". I would find it interesting to include information about the outreach done to the other ontologies owners AND their response to the request to for adaption of the currently available ontology to the specific need. This would more clearly define the gaps in the environment.",outreach_competition,,,,,,,,,,,,,,,,,,,,,,, 113 | b9cbeada37965a3592faf973a6fc569a,fitness for purpose (that means does it fit my requirements),,,,use_case_suitability,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,, 114 | fbab9f4297cc2068f488bd67e2dd7fc1,The soundness of the ontology against the intended conceptualisation.,,,,representational_adequacy,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,, 115 | 1591b5d028c23fae7116aebef6fc6d7b,number of projects already using this ontology,,,number_projects,evidence_use,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,, 116 | 89cbcd1933d66b1aab02557912936a0f,"Apart from it being well maintained - the domain (region of life sciences) and whether all (or most) of the classes are actually in this defined scope. Sorry, that's two. :)",active_development,coverage,scope,domain,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,, 117 | f00b9647b182080742d6a1334bb4e71e,coverage of my application domain,,,,coverage,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,, 118 | 736606c31713718430494d994a123538,,,,,,,,,I believe that the most important information on the motivation section is the actual application of the ontology. Was it developed for data publication? For interoperability? For the development of a knowledge-based system?,example_use,,,,,,,,,,,,,,,,,,,,,,,,,, 119 | 99f1263c42d300a37cbc231548d77cc4,Large data base using the ontology,evidence_use,large_database_use,,,,,,,,,,,,,,,Didn't understand question 32,unclassified,,,,,,,,,,I found the questions hard to answer. It seems everything is relevant but requiring all of it in an article seems unfeasible and harsh for students in particular.,all_requirements_toomuch,,,,,, 120 | 245cdf9ef3f6893bd1e3b51a979f9f38,Examples of the ontology's usage.,,,,use_case_suitability,Use case (if any),,,,,,,,,,,,,,,,,,,,,,,,,,,,,, 121 | ,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,, 122 | -------------------------------------------------------------------------------- /supplementary/questionnaire_email_lists.csv: -------------------------------------------------------------------------------- 1 | protege-user@mailman.stanford.edu 2 | obo-discuss@lists.sourceforge.net 3 | public-semweb-lifesci@w3.org 4 | semantic-web@w3.org 5 | public-owled@w3.org 6 | ontology-uk@googlegroups.com 7 | euon@googlegroups.com 8 | ontology@ebi 9 | bioportal list 10 | bfo-discuss 11 | -------------------------------------------------------------------------------- /supplementary/review_abstracts.csv: -------------------------------------------------------------------------------- 1 | Title,Abstract,Reviewer 1,Reviewer 3,Reviewer 4,Verdict 2 | Semantic Model for Legal Resources: Annotation and Reasoning over Normative Provisions,"A Semantic Web approach for an advanced access to legislative documents is presented in terms of a model of normativeprovisions and related axioms. In particular, relations between provisions are identified and modeled by introducing patterns able to describe Hohfeldian legal fundamental relations. Moreover, a query-based approach able to deal with relations between provision specific instances is described. Examples of semantic annotation of legal textual resources using RDF/OWL standards, as well as advanced access and reasoning facilities over provisions using SPARQL, are shown. The main benefit of the approach is represented by the ability to keep the complexity of the problem within a description logic computational tractability.",0,0,0,E 3 | LOTED2: An Ontology of European Public Procurement Notices,"Copyright and Moral Rights for the articles on this site are retained by the individual authors and/or other copyright owners. For more information on Open Research Online's data policy on reuse of materials please consult the policies page. Abstract. This paper describes the construction of the LOTED2 ontology for the representation of European public procurement notices. LOTED2follows initiatives around the creation of linked data-compliant representations of information regarding tender notices in Europe, but focusing on placing such representations within their legal context. It is therefore considered a legal ontology, as it supports the identification of legal concepts and more generally, legal reasoning. Unlike many other legal ontologies however, LOTED2 is designed to support the creation of Semantic Web applications. The methodology applied for building LOTED2 therefore seeks to find a compromise between the accurate representation of legal concepts and the usability of the ontology as a knowledge model for Semantic Web applications, while creating connections to other relevant ontologies in the domain.",1,1,1,I 4 | "PPROC, an Ontology for Transparency in Public Procurement","Public procurement or tendering refers to the process followed by public authorities for the procurement of goods and services. In most developed countries, the law requires public authorities to provide online information to ensure competitive tendering as far as possible, for which the adequate announcement of tenders is an essential requirement. In addition, transparency laws being proposed in such countries are making the monitoring of public contracts by citizens a fundamental right. This paper describes the PPROC ontology, which has been developed to give support to both processes, publication and accountability, by semantically describing public procurement processes and contracts. The PPROC ontology is extensive, since it covers not only the usual data about the tender, its objectives, deadlines and awardees, but also details of the whole process, from the initial contract publication to its termination. This makes it possible to use the ontology for both open data publication purposes and for the overall management of the public contract procurement process.",1,1,1,I 5 | Overview of the MPEG-21 Media Contract Ontology,"The MPEG-21 Media Contract Ontology (MCO), a part of the standard ISO/IEC 21000, is an ontology to represent contracts dealing with rights on multimedia assets and intellectual property protected content in general. A core model provides the elements to describe the permissions, obligations and prohibitions exchanged in the clauses of a contract. Specific vocabulary is defined in a model extension to represent the most common rights and constraints in the audiovisual context. Design principles, a methodology and a comparative analysis are given, as well as the practical guidelines to use the standard. A thorough description of the contract creation workflow from an original contract is given, including a sample contract text, the RDF version, the detailed mapping of the most relevant clauses and the reconstructed version. A set of MCO-related tools is described, including (i) the reference software to create and edit MCO contracts; (ii) modules to identify, store, search, validate and deliver MCO contracts and (iii) a tool to convert between the akin Contract Expression Language (CEL) contracts and the MCO contracts and (iv) the actual use of MCO in the Rightsdraw family of services.",1,1,0,I 6 | The Document Components Ontology (DoCO),"The description of document layers, as well as of the document discourse (e.g. the scientific discourse in scholarly articles) in machine-readable forms is crucial in facilitating semantic publishing and overall comprehension of documents by both users and machines. In this paper we introduce DoCO, the Document Components Ontology, i.e., an OWL 2 DL ontology that provides a general-purpose structured vocabulary of document elements to describe document parts in RDF. In addition to the formal description of the ontology, its utility in practice is showcased through several in-house solutions and other works of the Semantic Publishing community that rely on DoCO to annotate and retrieve document components of scholarly articles. ",1,1,1,I 7 | Time Ontology Extended for Non-Gregorian Calendar Applications,"We have extended OWL-Time to support the encoding of temporal position in a range of reference systems, in addition to the Gregorian calendar and conventional clock. Two alternative implementations are provided: as a pure extension or OWL-Time, or as a replacement, both of which preserve the same representation for the cases originally supported by OWL-Time. The combination of the generalized temporal position encoding and the temporal interval topology from OWL-Time support a range of applications in a variety of cultural and technical settings. These are illustrated with examples involving non-Gregorian calendars, Unix-time, and geologic time using both chronometric and stratigraphic timescales.",0,0,0,E 8 | PAROLE/SIMPLE ‘lemon’ ontology and lexicons,"The PAROLE/SIMPLE 'Lemon "" Ontology and Lexicon are the OWL/RDF version of the PAROLE/SIMPLE lexicons (defined during the PAROLE (LE2-4017) and SIMPLE (LE4-8346) IV FP EU projects) once mapped onto Lemon model and LexInfo ontology. Original PAROLE/SIMPLE lexicons contain morphological, syntactic and semantic information , organized according to a common model and to common linguistic specifications for 12 European languages. The data set we describe includes the PAROLE/SIMPLEmodel mapped to Lemon and LexInfo ontology and the Spanish & Catalan lexicons. All data are published in the Data Hub and are distributed under CC Attribution 3.0 Unported license. The Spanish lexicon contains 199466 triples and 7572 lexical entries fully annotated with syntactic and semantic information. The Catalan lexicon contains 343714 triples and 20545 lexical entries annotated with syntactic information half of which are also annotated with semantic information. In this paper we describe the resulting data, the mapping process and the benefits obtained. We demonstrate that the Linked Open Data principles prove essential for datasets such as original PAROLE/SIMPLE lexicons where harmonization and interoperability was crucial. The resulting data is lighter and better suited for exploitation. In addition , it easies further extensions and links to external resources such as WordNet, lemonUby, DBpedia etc.",0,0,0,E 9 | "lemonUby - a large, interlinked, syntactically-rich lexical resource for ontologies","We introduce lemonUby, a new lexical resource integrated in the Semantic Web which is the result of converting data extracted from the existing large-scale linked lexical resource UBY to the lemon lexicon model. The following data from UBY were converted: WordNet, FrameNet, VerbNet, English and German Wiktionary, the English and German entries of Omega-Wiki, as well as links between pairs of these lexicons at the word sense level (links between VerbNet and FrameNet, VerbNet and WordNet, WordNet and FrameNet, WordNet and Wiktionary, WordNet and German OmegaWiki). We linked lemonUby to other lexical resources and linguistic terminology repositories in the Linguistic Linked Open Data cloud and outline possible applications of this new dataset.",0,0,0,E 10 | OLiA – Ontologies of Linguistic Annotation," This paper describes the Ontologies of Linguistic Annotation (OLiA) as one of the data sets currently available as part of Linguistic Linked Open Data (LLOD) cloud. Within the LLOD cloud, the OLiA ontologies serve as a reference hub for annotation terminology for linguistic phenomena on a great band-width of languages, they have been used to facilitate interoperability and information integration of linguistic annotations in corpora, NLP pipelines, and lexical-semantic resources and mediate their linking with multiple community-maintained terminology repositories.",1,1,0,I 11 | Using ontologies to model human navigation behavior in information networks: A study based on Wikipedia,"The need to examine the behavior of different user groups is a fundamental requirement when building information systems. In this paper, we present Ontology-based Decentralized Search (OBDS), a novel method to model the navigation behavior of users equipped with different types of background knowledge. Ontology-based Decentralized Search combines ontologies and decentralized search, an established method for navigation in social networks, to model navigation behavior in information networks. The method uses ontologies as an explicit representation of background knowledge to inform the navigation process and guide it towards navigation targets. By using different ontologies, users equipped with different types of background knowledge can be represented. We demonstrate our method using four biomedical ontologies and their associated Wikipedia articles. We compare our simulation results with base line approaches and with results obtained from a user study and find that our method produces click paths that have properties similar to those originating from human navigators. The results suggest that our method can be used to model human navigation behavior in systems that are based on information networks such as Wikipedia. This paper makes the following contributions: (i) To the best of our knowledge, this is the first work to demonstrate the utility of ontologies in modeling human navigation and (ii) it yields new insights and understanding about the mechanisms of human navigation in information networks.",0,0,0,E 12 | "DBpedia - A Large-scale, Multilingual Knowledge Base Extracted from Wikipedia","The DBpedia community project extracts structured, multilingual knowledge from Wikipedia and makes it freely available using Semantic Web and Linked Data standards. The extracted knowledge, comprising more than 1.8 billion facts, is structured according to an ontology maintained by the community. The knowledge is obtained from different Wikipedia language editions, thus covering more than 100 languages, and mapped to the community ontology. The resulting data sets are linked to more than 30 other data sets in the Linked Open Data (LOD) cloud. The DBpedia project was started in 2006 and has meanwhile attracted large interest in research and practice. Being a central part of the LOD cloud, it serves as a connection hub for other data sets. For the research community, DBpedia provides a testbed serving real world data spanning many domains and languages. Due to the continuous growth of Wikipedia, DBpedia also provides an increasing added value for data acquisition, re-use and integration tasks within organisations. In this system report, we give an overview over the DBpedia community project, including its architecture, technical implementation, maintenance, internationalisation, usage statistics and showcase some popular DBpedia applications.",0,0,0,E 13 | LC3: A spatio-temporal and semantic model for knowledge discovery from geospatial datasets,"There is a need for decision-makers to be provided with both an overview of existing knowledge, and information which is as complete and up-to-date as possible on changes in certain features of the biosphere. Another objective is to bring together all the many attempts which have been made over the years at various levels (international, Community, national and regional) to obtain more information on the environment and the way it is changing. As a result, remote sensing tools monitor large amount of land cover informationg enabling study of dynamic processes. However the size of the dataset requires new tools to identify pattern and extract knowledge. We propose a model to discover knowledge on parcel data allowing analysis of dynamic geospatial phenomena using time, spatial and thematic data. The model is called Land Cover Change Continuum (LC3) and is able to track the evolution of spatial entities along time. Based on semantic web technologies, the model allows users to specify and to query spatio-temporal information based on semantic definitions. The semantic of spatial relationships are of interest to qualify filiation relationships. The result of this process permit to identify evolutive patterns as a basis for studying the dynamics of the geospatial environment. To this end, we use CORINE datasets to study changes in a specific part of France. In our approach, we consider entities as having several representations during their lifecycle. Each representation includes identity, spatial and descriptive properties that evolve over time.",0,0,0,E 14 | Exposing INSPIRE on the Semantic Web,"The INSPIRE Directive by the European Commission sets the legal and technical foundations towards interoperable Spatial Data Infrastructures (SDIs) across Europe. EU member states are already providing such services for several geospatial data themes (e.g., transportation networks, administrative units). Unfortunately, the INSPIRE ecosystem has been largely disjoint from the Semantic Web, without any means to repurpose existing SDIs as high-quality data sources, and thus multiply their value through interlinking, reasoning and inferencing. In this paper, we introduce a methodology that can assist stakeholders in exposing INSPIRE-aligned SDIs on the Semantic Web according to the recent GeoSPARQL standard. We develop methods for discovering INSPIRE data through a virtual SPARQL endpoint over existing INSPIRE catalogue services. Further, we implement a suite of tools for automatically transforming INSPIRE data and metadata into RDF triples with geometries. The compiled geographic and thematic information can then be loaded into semantic repositories for querying or interlinked with other data. Our open-source solutions essentially repurpose existing INSPIRE SDIs, so as to promote uptake and facilitate their reuse in practice. Finally, as a case study, we report our experience in validating this approach on a real-world SDI with publicly available data for Greece in order to expose its contents through (Geo)SPARQL endpoints.",0,0,0,E 15 | Engineering ontology-based access to real-world data sources,"The preparation of existing real-world datasets for publication as high-quality semantic web data is a complex task that requires the concerted execution of a variety of processing steps using a range of different tools. Faced with both changing input data and evolving requirements on the produced output, we face a significant engineering task for schema and data transformation. We argue that to achieve a robust and flexible transformation process, a high-level declarative description is needed, that can be used to drive the entire tool chain. We have implemented this idea for the deployment of ontology-based data access (OBDA) solutions, where semantically annotated views that integrate multiple data sources on different formats are created, based on an ontology and a collection of mappings. For illustration, we show how a single declarative description helps to orchestrate a complete tool chain, beginning with the download of data sets, and through to the installation of the data sets for a variety of tools including data and query transformation processes and reasoning services. We base our case study on several publicly available tabular and relational datasets concerning the operations of the petroleum industry in Norway. We include a discussion of the relative performance of the used tools on our case study, and an overview of lessons learnt for practical deployment of OBDA on real-world datasets.",0,0,0,E 16 | The Data Mining OPtimization Ontology,"The Data Mining OPtimization Ontology (DMOP) has been developed to support informed decision-making at various choice points of the data mining process. The ontology can be used by data miners and deployed in ontology-driven information systems. The primary purpose for which DMOP has been developed is the automation of algorithm and model selection through semantic meta-mining that makes use of an ontology-based meta-analysis of complete data mining processes in view of extracting patterns associated with mining performance. To this end, DMOP contains detailed descriptions of data mining tasks (e.g., learning, feature selection), data, algorithms, hypotheses such as mined models or patterns, and workflows. A development methodology was used for DMOP, including items such as competency questions and foundational ontology reuse. Several non-trivial modeling problems were encountered and due to the complexity of the data mining details, the ontology requires the use of the OWL 2 DL profile. DMOP was successfully evaluated for semantic meta-mining and used in constructing the Intelligent Discovery Assistant, deployed at the popular data mining environment RapidMiner.",1,1,1,I 17 | Using a suite of ontologies for preserving workflow-centric research objects,"Scientific workflows are a popular mechanism for specifying and automating data-driven in silico experiments. A significant aspect of their value lies in their potential to be reused. Once shared, workflows become useful building blocks that can be combined or modified for developing new experiments. However, previous studies have shown that storing workflow specifications alone is not sufficient to ensure that they can be successfully reused, without being able to understand what the workflows aim to achieve or to re-enact them. To gain an understanding of the workflow, and how it may be used and repurposed for their needs, scientists require access to additional resources such as annotations describing the workflow, datasets used and produced by the workflow, and provenance traces recording workflow executions. In this article, we present a novel approach to the preservation of scientific workflows through the application of research objects-aggregations of data and metadata that enrich the workflow specifications. Our approach is realised as a suite of ontologies that support the creation of workflow-centric research objects. Their design was guided by requirements elicited from previous empirical analyses of workflow decay and repair. The ontologies developed make use of and extend existing well known ontologies, namely the Object Reuse and Exchange (ORE) vocabulary, the Annotation Ontology (AO) and the W3C PROV ontology (PROVO). We illustrate the application of the ontologies for building Workflow Research Objects with a case-study that investigates Huntington's disease, performed in collaboration with a team from the Leiden University Medial Centre (HG-LUMC). Finally we present a number of tools developed for creating and managing Workflow-Centric Research Objects.",0,0,0,E 18 | WSMO-Lite and hRESTS: Lightweight semantic annotations for Web services and RESTful APIs,"Available online xxxx Keywords: WSMO-Lite SAWSDL Web services RESTful services a b s t r a c t Service-oriented computing has brought special attention to service description, especially in connection with semantic technologies. The expected proliferation of publicly accessible services can benefit greatly from tool support and automation, both of which are the focus of Semantic Web Service (SWS) frameworks that especially address service discovery, composition and execution. As the first SWS standard, in 2007 the World Wide Web Consortium produced a lightweight bottom-up specification called SAWSDL for adding semantic annotations to WSDL service descriptions. Building on SAWSDL, this article presents WSMO-Lite, a lightweight ontology of Web service semantics that distinguishes four semantic aspects of services: function, behavior, information model, and nonfunctional properties, which together form a basis for semantic automation. With the WSMO-Lite ontology, SAWSDL descriptions enable semantic automation beyond simple input/output matchmaking that is supported by SAWSDL itself. Further, to broaden the reach of WSMO-Lite and SAWSDL tools to the increasingly common RESTful services, the article adds hRESTS and MicroWSMO, two HTML microformats that mirror WSDL and SAWSDL in the documentation of RESTful services, enabling combining RESTful services with WSDL-based ones in a single semantic framework. To demonstrate the feasibility and versatility of this approach, the article presents common algorithms for Web service discovery and composition adapted to WSMO-Lite.",1,1,0,I 19 | Mímir: An open-source semantic search framework for interactive information seeking and discovery,"Semantic search is gradually establishing itself as the next generation search paradigm, which meets better a wider range of information needs, as compared to traditional full-text search. At the same time, however, expanding search towards document structure and external, formal knowledge sources (e.g. LOD resources) remains challenging, especially with respect to efficiency, usability, and scalability. This paper introduces Mímir – an open-source framework for integrated semantic search over text, document structure , linguistic annotations, and formal semantic knowledge. Mímir supports complex structural queries, as well as basic keyword search. Exploratory search and sense-making are supported through information visualisation interfaces, such as co-occurrence matrices and term clouds. There is also an interactive retrieval interface, where users can save, refine, and analyse the results of a semantic search over time. The more well-studied precision-oriented information seeking searches are also well supported. The generic and extensible nature of the Mímir platform is demonstrated through three different, real-world applications , one of which required indexing and search over tens of millions of documents and fifty to hundred times as many semantic annotations. Scaling up to over 150 million documents was also accomplished, via index federation and cloud-based deployment.",0,0,0,E 20 | Ontology-based representation and analysis of host-Brucella interactions,"Background: Biomedical ontologies are representations of classes of entities in the biomedical domain and how these classes are related in computer-and human-interpretable formats. Ontologies support data standardization and exchange and provide a basis for computer-assisted automated reasoning. IDOBRU is an ontology in the domain of Brucella and brucellosis. Brucella is a Gram-negative intracellular bacterium that causes brucellosis, the most common zoonotic disease in the world. In this study, IDOBRU is used as a platform to model and analyze how the hosts, especially host macrophages, interact with virulent Brucella strains or live attenuated Brucella vaccine strains. Such a study allows us to better integrate and understand intricate Brucella pathogenesis and host immunity mechanisms.",0,0,0,E 21 | My Corporis Fabrica Embryo: An ontology-based 3D spatio-temporal modeling of human embryo development,"BACKGROUND: 22 | Embryology is a complex morphologic discipline involving a set of entangled mechanisms, sometime difficult to understand and to visualize. Recent computer based techniques ranging from geometrical to physically based modeling are used to assist the visualization and the simulation of virtual humans for numerous domains such as surgical simulation and learning. On the other side, the ontology-based approach applied to knowledge representation is more and more successfully adopted in the life-science domains to formalize biological entities and phenomena, thanks to a declarative approach for expressing and reasoning over symbolic information. 3D models and ontologies are two complementary ways to describe biological entities that remain largely separated. Indeed, while many ontologies providing a unified formalization of anatomy and embryology exist, they remain only descriptive and make the access to anatomical content of complex 3D embryology models and simulations difficult. 23 | RESULTS: 24 | In this work, we present a novel ontology describing the development of the human embryology deforming 3D models. Beyond describing how organs and structures are composed, our ontology integrates a procedural description of their 3D representations, temporal deformation and relations with respect to their developments. We also created inferences rules to express complex connections between entities. It results in a unified description of both the knowledge of the organs deformation and their 3D representations enabling to visualize dynamically the embryo deformation during the Carnegie stages. Through a simplified ontology, containing representative entities which are linked to spatial position and temporal process information, we illustrate the added-value of such a declarative approach for interactive simulation and visualization of 3D embryos. 25 | CONCLUSIONS: 26 | Combining ontologies and 3D models enables a declarative description of different embryological models that capture the complexity of human developmental anatomy. Visualizing embryos with 3D geometric models and their animated deformations perhaps paves the way towards some kind of hypothesis-driven application. These can also be used to assist the learning process of this complex knowledge.",1,1,1,I 27 | Improving the Sequence Ontology terminology for genomic variant annotation,"BACKGROUND: 28 | The Genome Variant Format (GVF) uses the Sequence Ontology (SO) to enable detailed annotation of sequence variation. The annotation includes SO terms for the type of sequence alteration, the genomic features that are changed and the effect of the alteration. The SO maintains and updates the specification and provides the underlying ontologicial structure. 29 | METHODS: 30 | A requirements analysis was undertaken to gather terms missing in the SO release at the time, but needed to adequately describe the effects of sequence alteration on a set of variant genomic annotations. We have extended and remodeled the SO to include and define all terms that describe the effect of variation upon reference genomic features in the Ensembl variation databases. 31 | RESULTS: 32 | The new terminology was used to annotate the human reference genome with a set of variants from both COSMIC and dbSNP. A GVF file containing 170,853 sequence alterations was generated using the SO terminology to annotate the kinds of alteration, the effect of the alteration and the reference feature changed. There are four kinds of alteration and 24 kinds of effect seen in this dataset. (Ensembl Variation annotates 34 different SO consequence terms: http://www.ensembl.org/info/docs/variation/predicted_data.html). 33 | CONCLUSIONS: 34 | We explain the updates to the Sequence Ontology to describe the effect of variation on existing reference features. We have provided a set of annotations using this terminology, and the well defined GVF specification. We have also provided a provisional exploration of this large annotation dataset.",0,0,0,E 35 | Development of an Ontology for Periodontitis,"In the clinical dentists and periodontal researchers' community, there is an obvious demand for a systems model capable of linking the clinical presentation of periodontitis to underlying molecular knowledge. A computer-readable representation of processes on disease development will give periodontal researchers opportunities to elucidate pathways and mechanisms of periodontitis. An ontology for periodontitis can be a model for integration of large variety of factors relating to a complex disease such as chronic inflammation in different organs accompanied by bone remodeling and immune system disorders, which has recently been referred to as osteoimmunology. 36 | METHODS: 37 | Terms characteristic of descriptions related to the onset and progression of periodontitis were manually extracted from 194 review articles and PubMed abstracts by experts in periodontology. We specified all the relations between the extracted terms and constructed them into an ontology for periodontitis. We also investigated matching between classes of our ontology and that of Gene Ontology Biological Process. 38 | RESULTS: 39 | We developed an ontology for periodontitis called Periodontitis-Ontology (PeriO). The pathological progression of periodontitis is caused by complex, multi-factor interrelationships. PeriO consists of all the required concepts to represent the pathological progression and clinical treatment of periodontitis. The pathological processes were formalized with reference to Basic Formal Ontology and Relation Ontology, which accounts for participants in the processes realized by biological objects such as molecules and cells. We investigated the peculiarity of biological processes observed in pathological progression and medical treatments for the disease in comparison with Gene Ontology Biological Process (GO-BP) annotations. The results indicated that peculiarities of Perio existed in 1) granularity and context dependency of both the conceptualizations, and 2) causality intrinsic to the pathological processes. PeriO defines more specific concepts than GO-BP, and thus can be added as descendants of GO-BP leaf nodes. PeriO defines causal relationships between the process concepts, which are not shown in GO-BP. The difference can be explained by the goal of conceptualization: PeriO focuses on mechanisms of the pathogenic progress, while GO-BP focuses on cataloguing all of the biological processes observed in experiments. The goal of conceptualization in PeriO may reflect the domain knowledge where a consequence in the causal relationships is a primary interest. We believe the peculiarities can be shared among other diseases when comparing processes in disease against GO-BP. 40 | CONCLUSIONS: 41 | This is the first open biomedical ontology of periodontitis capable of providing a foundation for an ontology-based model of aspects of molecular biology and pathological processes related to periodontitis, as well as its relations with systemic diseases. PeriO is available at http://bio-omix.tmd.ac.jp/periodontitis/.",1,1,1,I 42 | Developing VISO: Vaccine Information Statement Ontology for patient education,"OBJECTIVE: 43 | To construct a comprehensive vaccine information ontology that can support personal health information applications using patient-consumer lexicon, and lead to outcomes that can improve patient education. 44 | METHODS: 45 | The authors composed the Vaccine Information Statement Ontology (VISO) using the web ontology language (OWL). We started with 6 Vaccine Information Statement (VIS) documents collected from the Centers for Disease Control and Prevention (CDC) website. Important and relevant selections from the documents were recorded, and knowledge triples were derived. Based on the collection of knowledge triples, the meta-level formalization of the vaccine information domain was developed. Relevant instances and their relationships were created to represent vaccine domain knowledge. 46 | RESULTS: 47 | The initial iteration of the VISO was realized, based on the 6 Vaccine Information Statements and coded into OWL2 with Protégé. The ontology consisted of 132 concepts (classes and subclasses) with 33 types of relationships between the concepts. The total number of instances from classes totaled at 460, along with 429 knowledge triples in total. Semiotic-based metric scoring was applied to evaluate quality of the ontology.",1,1,1,I 48 | Development and validation of a classification approach for extracting severity automatically from electronic health records,"Electronic Health Records (EHRs) contain a wealth of information useful for studying clinical phenotype-genotype relationships. Severity is important for distinguishing among phenotypes; however other severity indices classify patient-level severity (e.g., mild vs. acute dermatitis) rather than phenotype-level severity (e.g., acne vs. myocardial infarction). Phenotype-level severity is independent of the individual patient's state and is relative to other phenotypes. Further, phenotype-level severity does not change based on the individual patient. For example, acne is mild at the phenotype-level and relative to other phenotypes. Therefore, a given patient may have a severe form of acne (this is the patient-level severity), but this does not effect its overall designation as a mild phenotype at the phenotype-level. 49 | METHODS: 50 | We present a method for classifying severity at the phenotype-level that uses the Systemized Nomenclature of Medicine - Clinical Terms. Our method is called the Classification Approach for Extracting Severity Automatically from Electronic Health Records (CAESAR). CAESAR combines multiple severity measures - number of comorbidities, medications, procedures, cost, treatment time, and a proportional index term. CAESAR employs a random forest algorithm and these severity measures to discriminate between severe and mild phenotypes. 51 | RESULTS: 52 | Using a random forest algorithm and these severity measures as input, CAESAR differentiates between severe and mild phenotypes (sensitivity = 91.67, specificity = 77.78) when compared to a manually evaluated reference standard (k = 0.716). 53 | CONCLUSIONS: 54 | CAESAR enables researchers to measure phenotype severity from EHRs to identify phenotypes that are important for comparative effectiveness research.",0,0,0,E 55 | Expanding the mammalian phenotype ontology to support automated exchange of high throughput mouse phenotyping data generated by large-scale mouse knockout screens,"Background A vast array of data is about to emerge from the large scale high-throughput mouse knockout phenotyping projects worldwide. It is critical that this information is captured in a standardized manner, made accessible, and is fully integrated with other phenotype data sets for comprehensive querying and analysis across all phenotype data types. The volume of data generated by the high-throughput phenotyping screens is expected to grow exponentially, thus, automated methods and standards to exchange phenotype data are required. Results The IMPC (International Mouse Phenotyping Consortium) is using the Mammalian Phenotype (MP) ontology in the automated annotation of phenodeviant data from high throughput phenotyping screens. 287 new term additions with additional hierarchy revisions were made in multiple branches of the MP ontology to accurately describe the results generated by these high throughput screens. Conclusions Because these large scale phenotyping data sets will be reported using the MP as the common data standard for annotation and data exchange, automated importation of these data to MGI (Mouse Genome Informatics) and other resources is possible without curatorial effort. Maximum biomedical value of these mutant mice will come from integrating primary high-throughput phenotyping data with secondary, comprehensive phenotypic analyses combined with published phenotype details on these and related mutants at MGI and other resources.",0,0,0,E 56 | Development and application of an interaction network ontology for literature mining of vaccine-associated gene-gene interactions,"Background 57 | Literature mining of gene-gene interactions has been enhanced by ontology-based name classifications. However, in biomedical literature mining, interaction keywords have not been carefully studied and used beyond a collection of keywords. 58 | 59 | Methods 60 | In this study, we report the development of a new Interaction Network Ontology (INO) that classifies >800 interaction keywords and incorporates interaction terms from the PSI Molecular Interactions (PSI-MI) and Gene Ontology (GO). Using INO-based literature mining results, a modified Fisher’s exact test was established to analyze significantly over- and under-represented enriched gene-gene interaction types within a specific area. Such a strategy was applied to study the vaccine-mediated gene-gene interactions using all PubMed abstracts. The Vaccine Ontology (VO) and INO were used to support the retrieval of vaccine terms and interaction keywords from the literature. 61 | 62 | Results 63 | INO is aligned with the Basic Formal Ontology (BFO) and imports terms from 10 other existing ontologies. Current INO includes 540 terms. In terms of interaction-related terms, INO imports and aligns PSI-MI and GO interaction terms and includes over 100 newly generated ontology terms with ‘INO_’ prefix. A new annotation property, ‘has literature mining keywords’, was generated to allow the listing of different keywords mapping to the interaction types in INO. Using all PubMed documents published as of 12/31/2013, approximately 266,000 vaccine-associated documents were identified, and a total of 6,116 gene-pairs were associated with at least one INO term. Out of 78 INO interaction terms associated with at least five gene-pairs of the vaccine-associated sub-network, 14 terms were significantly over-represented (i.e., more frequently used) and 17 under-represented based on our modified Fisher’s exact test. These over-represented and under-represented terms share some common top-level terms but are distinct at the bottom levels of the INO hierarchy. The analysis of these interaction types and their associated gene-gene pairs uncovered many scientific insights. 64 | 65 | Conclusions 66 | INO provides a novel approach for defining hierarchical interaction types and related keywords for literature mining. The ontology-based literature mining, in combination with an INO-based statistical interaction enrichment test, provides a new platform for efficient mining and analysis of topic-specific gene interaction networks.",1,1,1,I 67 | The cellular microscopy phenotype ontology,"Background 68 | Phenotypic data derived from high content screening is currently annotated using free-text, thus preventing the integration of independent datasets, including those generated in different biological domains, such as cell lines, mouse and human tissues. 69 | 70 | Description 71 | We present the Cellular Microscopy Phenotype Ontology (CMPO), a species neutral ontology for describing phenotypic observations relating to the whole cell, cellular components, cellular processes and cell populations. CMPO is compatible with related ontology efforts, allowing for future cross-species integration of phenotypic data. CMPO was developed following a curator-driven approach where phenotype data were annotated by expert biologists following the Entity-Quality (EQ) pattern. These EQs were subsequently transformed into new CMPO terms following an established post composition process. 72 | 73 | Conclusion 74 | CMPO is currently being utilized to annotate phenotypes associated with high content screening datasets stored in several image repositories including the Image Data Repository (IDR), MitoSys project database and the Cellular Phenotype Database to facilitate data browsing and discoverability.",1,1,1,I 75 | The Non-Coding RNA Ontology (NCRO): a comprehensive resource for the unification of non-coding RNA biology,"In recent years, sequencing technologies have enabled the identification of a wide range of non-coding RNAs (ncRNAs). Unfortunately, annotation and integration of ncRNA data has lagged behind their identification. Given the large quantity of information being obtained in this area, there emerges an urgent need to integrate what is being discovered by a broad range of relevant communities. To this end, the Non-Coding RNA Ontology (NCRO) is being developed to provide a systematically structured and precisely defined controlled vocabulary for the domain of ncRNAs, thereby facilitating the discovery, curation, analysis, exchange, and reasoning of data about structures of ncRNAs, their molecular and cellular functions, and their impacts upon phenotypes. The goal of NCRO is to serve as a common resource for annotations of diverse research in a way that will significantly enhance integrative and comparative analysis of the myriad resources currently housed in disparate sources. It is our belief that the NCRO ontology can perform an important role in the comprehensive unification of ncRNA biology and, indeed, fill a critical gap in both the Open Biological and Biomedical Ontologies (OBO) Library and the National Center for Biomedical Ontology (NCBO) BioPortal. Our initial focus is on the ontological representation of small regulatory ncRNAs, which we see as the first step in providing a resource for the annotation of data about all forms of ncRNAs. The NCRO ontology is free and open to all users, accessible at: http://purl.obolibrary.org/obo/ncro.owl.",1,1,1,I 76 | OBIB-a novel ontology for biobanking,"Background 77 | Biobanking necessitates extensive integration of data to allow data analysis and specimen sharing. Ontologies have been demonstrated to be a promising approach in fostering better semantic integration of biobank-related data. Hitherto no ontology provided the coverage needed to capture a broad spectrum of biobank user scenarios. 78 | 79 | Methods 80 | Based in the principles laid out by the Open Biological and Biomedical Ontologies Foundry two biobanking ontologies have been developed. These two ontologies were merged using a modular approach consistent with the initial development principles. The merging was facilitated by the fact that both ontologies use the same Upper Ontology and re-use classes from a similar set of pre-existing ontologies. 81 | 82 | Results 83 | Based on the two previous ontologies the Ontology for Biobanking (http://purl.obolibrary.org/obo/obib.owl) was created. Due to the fact that there was no overlap between the two source ontologies the coverage of the resulting ontology is significantly larger than of the two source ontologies. The ontology is successfully used in managing biobank information of the Penn Medicine BioBank. 84 | 85 | Conclusions 86 | Sharing development principles and Upper Ontologies facilitates subsequent merging of ontologies to achieve a broader coverage.",1,1,0,I 87 | VICO: Ontology-based representation and integrative analysis of Vaccination Informed Consent forms,"Background 88 | 89 | Although signing a vaccination (or immunization) informed consent form is not a federal requirement in the US and Canada, such a practice is required by many states and pharmacies. The content and structures of these informed consent forms vary, which makes it hard to compare and analyze without standardization. To facilitate vaccination informed consent data standardization and integration, it is important to examine various vaccination informed consent forms, patient answers, and consent results. In this study, we report a Vaccination Informed Consent Ontology (VICO) that extends the Informed Consent Ontology and integrates related OBO foundry ontologies, such as the Vaccine Ontology, with a focus on vaccination screening questionnaire in the vaccination informed consent domain. 90 | 91 | Results 92 | 93 | Current VICO contains 993 terms, including 248 VICO specific terms and 709 terms imported from 17 OBO Foundry ontologies. VICO ontologically represents and integrates 12 vaccination informed consent forms from the Walgreens, Costco pharmacies, Rite AID, University of Maryland College Park, and the government of Manitoba, Canada. VICO extends Informed Consent Ontology (ICO) with vaccination screening questionnaires and questions. Our use cases and examples demonstrate five usages of VICO. First, VICO provides standard, robust and consistent representation and organization of the knowledge in different vaccination informed consent forms, questionnaires, and questions. Second, VICO integrates prior knowledge, e.g., the knowledge of vaccine contraindications imported from the Vaccine Ontology (VO). Third, VICO helps manage the complexity of the domain knowledge using logically defined ontological hierarchies and axioms. VICO glues multiple schemas that represent complex vaccination informed consent contents defined in different organizations. Fourth, VICO supports efficient query and comparison, e.g., through the Description Language (DL)-Query and SPARQL. Fifth, VICO helps discover new knowledge. For instance, by integrating the prior knowledge imported from the VO with a user’s answer to informed consent questions (e.g., allergic reaction question) for a specific vaccination, we can infer whether or not the patient can be vaccinated with the vaccine. 94 | 95 | Conclusions 96 | 97 | The Vaccination Informed Consent Ontology (VICO) represents entities related to vaccination informed consents with a special focus on vaccination informed consent forms, and questionnaires and questions in the forms. Our use cases and examples demonstrated how VICO could support a platform for vaccination informed consent data standardization, data integration, and data queries.",1,0,1,I 98 | Extending gene ontology in the context of extracellular RNA and vesicle communication,"Background 99 | 100 | To address the lack of standard terminology to describe extracellular RNA (exRNA) data/metadata, we have launched an inter-community effort to extend the Gene Ontology (GO) with subcellular structure concepts relevant to the exRNA domain. By extending GO in this manner, the exRNA data/metadata will be more easily annotated and queried because it will be based on a shared set of terms and relationships relevant to extracellular research. 101 | 102 | Methods 103 | 104 | By following a consensus-building process, we have worked with several academic societies/consortia, including ERCC, ISEV, and ASEMV, to identify and approve a set of exRNA and extracellular vesicle-related terms and relationships that have been incorporated into GO. In addition, we have initiated an ongoing process of extractions of gene product annotations associated with these terms from Vesiclepedia and ExoCarta, conversion of the extracted annotations to Gene Association File (GAF) format for batch submission to GO, and curation of the submitted annotations by the GO Consortium. As a use case, we have incorporated some of the GO terms into annotations of samples from the exRNA Atlas and implemented a faceted search interface based on such annotations. 105 | 106 | Results 107 | 108 | We have added 7 new terms and modified 9 existing terms (along with their synonyms and relationships) to GO. Additionally, 18,695 unique coding gene products (mRNAs and proteins) and 963 unique non-coding gene products (ncRNAs) which are associated with the terms: “extracellular vesicle”, “extracellular exosome”, “apoptotic body”, and “microvesicle” were extracted from ExoCarta and Vesiclepedia. These annotations are currently being processed for submission to GO. 109 | 110 | Conclusions 111 | 112 | As an inter-community effort, we have made a substantial update to GO in the exRNA context. We have also demonstrated the utility of some of the new GO terms for sample annotation and metadata search.",0,0,0,E 113 | "MicrO: an ontology of phenotypic and metabolic characters, assays, and culture media found in prokaryotic taxonomic descriptions","BACKGROUND: 114 | MicrO is an ontology of microbiological terms, including prokaryotic qualities and processes, material entities (such as cell components), chemical entities (such as microbiological culture media and medium ingredients), and assays. The ontology was built to support the ongoing development of a natural language processing algorithm, MicroPIE (or, Microbial Phenomics Information Extractor). During the MicroPIE design process, we realized there was a need for a prokaryotic ontology which would capture the evolutionary diversity of phenotypes and metabolic processes across the tree of life, capture the diversity of synonyms and information contained in the taxonomic literature, and relate microbiological entities and processes to terms in a large number of other ontologies, most particularly the Gene Ontology (GO), the Phenotypic Quality Ontology (PATO), and the Chemical Entities of Biological Interest (ChEBI). We thus constructed MicrO to be rich in logical axioms and synonyms gathered from the taxonomic literature. 115 | RESULTS: 116 | MicrO currently has ~14550 classes (~2550 of which are new, the remainder being microbiologically-relevant classes imported from other ontologies), connected by ~24,130 logical axioms (5,446 of which are new), and is available at (http://purl.obolibrary.org/obo/MicrO.owl) and on the project website at https://github.com/carrineblank/MicrO. MicrO has been integrated into the OBO Foundry Library (http://www.obofoundry.org/ontology/micro.html), so that other ontologies can borrow and re-use classes. Term requests and user feedback can be made using MicrO's Issue Tracker in GitHub. We designed MicrO such that it can support the ongoing and future development of algorithms that can leverage the controlled vocabulary and logical inference power provided by the ontology. 117 | CONCLUSIONS: 118 | By connecting microbial classes with large numbers of chemical entities, material entities, biological processes, molecular functions, and qualities using a dense array of logical axioms, we intend MicrO to be a powerful new tool to increase the computing power of bioinformatics tools such as the automated text mining of prokaryotic taxonomic descriptions using natural language processing. We also intend MicrO to support the development of new bioinformatics tools that aim to develop new connections between microbial phenotypes and genotypes (i.e., the gene content in genomes). Future ontology development will include incorporation of pathogenic phenotypes and prokaryotic habitats.",1,1,1,I 119 | Representing vision and blindness,"BACKGROUND: 120 | There have been relatively few attempts to represent vision or blindness ontologically. This is unsurprising as the related phenomena of sight and blindness are difficult to represent ontologically for a variety of reasons. Blindness has escaped ontological capture at least in part because: blindness or the employment of the term 'blindness' seems to vary from context to context, blindness can present in a myriad of types and degrees, and there is no precedent for representing complex phenomena such as blindness. 121 | METHODS: 122 | We explore current attempts to represent vision or blindness, and show how these attempts fail at representing subtypes of blindness (viz., color blindness, flash blindness, and inattentional blindness). We examine the results found through a review of current attempts and identify where they have failed. 123 | RESULTS: 124 | By analyzing our test cases of different types of blindness along with the strengths and weaknesses of previous attempts, we have identified the general features of blindness and vision. We propose an ontological solution to represent vision and blindness, which capitalizes on resources afforded to one who utilizes the Basic Formal Ontology as an upper-level ontology. 125 | CONCLUSIONS: 126 | The solution we propose here involves specifying the trigger conditions of a disposition as well as the processes that realize that disposition. Once these are specified we can characterize vision as a function that is realized by certain (in this case) biological processes under a range of triggering conditions. When the range of conditions under which the processes can be realized are reduced beyond a certain threshold, we are able to say that blindness is present. We characterize vision as a function that is realized as a seeing process and blindness as a reduction in the conditions under which the sight function is realized. This solution is desirable because it leverages current features of a major upper-level ontology, accurately captures the phenomenon of blindness, and can be implemented in many domain-specific ontologies.",1,1,1,I 127 | An accurate and precise representation of drug ingredients,"BACKGROUND: 128 | In previous work, we built the Drug Ontology (DrOn) to support comparative effectiveness research use cases. Here, we have updated our representation of ingredients to include both active ingredients (and their strengths) and excipients. Our update had three primary lines of work: 1) analysing and extracting excipients, 2) analysing and extracting strength information for active ingredients, and 3) representing the binding of active ingredients to cytochrome P450 isoenzymes as substrates and inhibitors of those enzymes. 129 | METHODS: 130 | To properly differentiate between excipients and active ingredients, we conducted an ontological analysis of the roles that various ingredients, including excipients, have in drug products. We used the value specification model of the Ontology for Biomedical Investigations to represent strengths of active ingredients and then analyzed RxNorm to extract excipient and strength information and modeled them according to the results of our analysis. We also analyzed and defined dispositions of molecules used in aggregate as active ingredients to bind cytochrome P450 isoenzymes. 131 | RESULTS: 132 | Our analysis of excipients led to 17 new classes representing the various roles that excipients can bear. We then extracted excipients from RxNorm and added them to DrOn for branded drugs. We found excipients for 5,743 branded drugs, covering ~27% of the 21,191 branded drugs in DrOn. Our analysis of active ingredients resulted in another new class, active ingredient role. We also extracted strengths for all types of tablets, capsules, and caplets, resulting in strengths for 5,782 drug forms, covering ~41% of the 14,035 total drug forms and accounting for ~97 % of the 5,970 tablets, capsules, and caplets in DrOn. We represented binding-as-substrate and binding-as-inhibitor dispositions to two cytochrome P450 (CYP) isoenzymes (CYP2C19 and CYP2D6) and linked these dispositions to 65 compounds. It is now possible to query DrOn automatically for all drug products that contain active ingredients whose molecular grains inhibit or are metabolized by a particular CYP isoenzyme. DrOn is open source and is available at http://purl.obolibrary.org/obo/dron.owl.",0,0,0,E 133 | Towards exergaming commons: composing the exergame ontology for publishing open game data,"BACKGROUND: 134 | It has been shown that exergames have multiple benefits for physical, mental and cognitive health. Only recently, however, researchers have started considering them as health monitoring tools, through collection and analysis of game metrics data. In light of this and initiatives like the Quantified Self, there is an emerging need to open the data produced by health games and their associated metrics in order for them to be evaluated by the research community in an attempt to quantify their potential health, cognitive and physiological benefits. 135 | METHODS: 136 | We have developed an ontology that describes exergames using the Web Ontology Language (OWL); it is available at http://purl.org/net/exergame/ns#. After an investigation of key components of exergames, relevant ontologies were incorporated, while necessary classes and properties were defined to model these components. A JavaScript framework was also developed in order to apply the ontology to online exergames. Finally, a SPARQL Endpoint is provided to enable open data access to potential clients through the web. 137 | RESULTS: 138 | Exergame components include details for players, game sessions, as well as, data produced during these game-playing sessions. The description of the game includes elements such as goals, game controllers and presentation hardware used; what is more, concepts from already existing ontologies are reused/repurposed. Game sessions include information related to the player, the date and venue where the game was played, as well as, the results/scores that were produced/achieved. These games are subsequently played by 14 users in multiple game sessions and the results derived from these sessions are published in a triplestore as open data. 139 | CONCLUSIONS: 140 | We model concepts related to exergames by providing a standardized structure for reference and comparison. This is the first work that publishes data from actual exergame sessions on the web, facilitating the integration and analysis of the data, while allowing open data access through the web in an effort to enable the concept of Open Trials for Active and Healthy Ageing.",1,1,1,I 141 | An ontology for major histocompatibility restriction,"MHC molecules are a highly diverse family of proteins that play a key role in cellular immune recognition. Over time, different techniques and terminologies have been developed to identify the specific type(s) of MHC molecule involved in a specific immune recognition context. No consistent nomenclature exists across different vertebrate species. 142 | PURPOSE: 143 | To correctly represent MHC related data in The Immune Epitope Database (IEDB), we built upon a previously established MHC ontology and created an ontology to represent MHC molecules as they relate to immunological experiments. 144 | DESCRIPTION: 145 | This ontology models MHC protein chains from 16 species, deals with different approaches used to identify MHC, such as direct sequencing verses serotyping, relates engineered MHC molecules to naturally occurring ones, connects genetic loci, alleles, protein chains and multi-chain proteins, and establishes evidence codes for MHC restriction. Where available, this work is based on existing ontologies from the OBO foundry. 146 | CONCLUSIONS: 147 | Overall, representing MHC molecules provides a challenging and practically important test case for ontology building, and could serve as an example of how to integrate other ontology building efforts into web resources.",1,1,1,I 148 | -------------------------------------------------------------------------------- /supplementary/review_coding_scheme.csv: -------------------------------------------------------------------------------- 1 | Group,Code,Range 2 | Basic,Ontology name,0-2 3 | Basic,Ontology owner,0-2 4 | Basic,Ontology licence,0-1 5 | Basic,Ontology URL,0-1 6 | Basic,Ontology repository,0-1 7 | Basic,Methodological framework,0-1 8 | Motivation,Need,0-1 9 | Motivation,Competition,0-1 10 | Motivation,Target Audience,0-1 11 | "Scope, Requirements, Community",Scope and Coverage,0-1 12 | "Scope, Requirements, Community",Community,0-1 13 | "Scope, Requirements, Community",Email list and issue tracking,0-1 14 | Knowledge Elicitation,Source knowledge,0-1 15 | Knowledge Elicitation,Raw data location,0-1 16 | Knowledge Elicitation,Feature prioritisation,0-1 17 | Ontology Content,KR language,0-1 18 | Ontology Content,Development environment,0-1 19 | Ontology Content,Ontology metrics,0-1 20 | Ontology Content,Imports and interoperability,0-1 21 | Ontology Content,Entity naming convention,0-1 22 | Ontology Content,Identifier generation policy,0-1 23 | Ontology Content,Entity metadata policy,0-1 24 | Ontology Content,Upper ontology,0-1 25 | Ontology Content,Ontology relationships,0-1 26 | Ontology Content,Axiom patterns,0-1 27 | Ontology Content,Dereferencable IRIs,0-1 28 | Managing Change,Sustainability plan,0-1 29 | Managing Change,Entity Deprecation Strategy,0-1 30 | Managing Change,Versioning Policy,0-1 31 | Quality Assurance,Testing,0-1 32 | Quality Assurance,Evaluation,0-1 33 | Quality Assurance,Example use,0-1 34 | Quality Assurance,Institution Endorsement,0-1 35 | Quality Assurance,Logical consistency,0-1 36 | Quality Assurance,Evidence use,0-1 37 | -------------------------------------------------------------------------------- /supplementary/review_extraction.csv: -------------------------------------------------------------------------------- 1 | Code,Ontology name,Ontology owner,Ontology licence,Ontology URL,Ontology repository,Methodological framework,Need,Competition,Target Audience,Scope and Coverage,Community,Email list and issue tracking,Source knowledge,Raw data location,Feature prioritisation,KR language,Development environment,Ontology metrics,Imports and interoperability,Entity naming convention,Identifier generation policy,Entity metadata policy,Upper ontology,Ontology relationships,Axiom patterns,Dereferencable IRIs,Sustainability plan,Entity Deprecation Strategy,Versioning Policy,Testing,Evaluation,Example use,Institution Endorsement,Logical consistency,Evidence use,OBO Principles,URL,Name,Language,Domain,License,Repository 2 | P1,2,1,0,1,1,1,1,1,1,1,1,0,1,1,1,1,0,0,1,0,0,1,0,1,1,0,0,0,0,0,1,1,0,1,0,0,http://loted.eu/ontology,LOTED2,OWL,Public Procurement,NA,https://code.google.com/archive/p/loted2/ 3 | P10,2,1,0,0,0,1,1,0,1,1,1,0,1,1,1,1,1,1,0,0,0,0,1,1,1,0,0,0,0,0,1,0,0,1,0,0,https://sbmi.uth.edu/ontology/project/viso.htm,Vaccine Information Statement Ontology (VISO),OWL,Vaccine,NA,NA 4 | P11,2,1,1,1,1,1,1,1,1,1,1,1,1,0,0,1,1,1,1,1,0,0,1,1,1,0,0,0,0,0,1,1,0,0,1,2,http://purl.bioontology.org/ontology/INO),Interaction Network Ontology (INO),OWL,Vaccine,CC,https://sourceforge.net/projects/ino/ 5 | P12,2,1,0,1,1,1,1,1,1,1,1,1,1,1,0,1,1,1,1,0,0,1,1,1,1,0,0,0,1,0,0,1,0,1,1,1,http://www.ebi.ac.uk/cmpo/,Cellular Microscopy Phenotype Ontology (CMPO),OWL,Imaging,NA,https://github.com/EBISPOT/CMPO 6 | P13,2,1,1,1,1,1,1,1,1,1,1,1,0,0,1,1,1,1,1,1,1,1,1,1,1,0,0,0,0,0,0,1,0,1,0,1,http://purl.obolibrary.org/obo/ncro.owl,Non-Coding RNA Ontology (NCRO),OWL,Biology,Open,https://github.com/OmniSearch/ncro 7 | P15,2,1,1,1,1,1,1,0,1,1,0,1,1,0,0,1,1,1,1,1,0,1,1,1,1,0,0,0,0,0,1,1,0,1,0,1,https://github.com/vicoontology/VICO,Vaccination Informed Consent Ontology (VICO),OWL,Vaccine,Creative Commons 3.0 License,https://github.com/VICOontology/VICO 8 | P16,1,1,1,1,1,1,1,1,1,1,1,1,1,0,0,1,1,1,1,1,0,1,1,1,1,0,0,0,0,0,0,1,0,1,0,2,http://purl.obolibrary.org/obo/MicrO.owl,Ontology of Prokaryotic Phenotypic and Metabolic Characters (MicrO),OWL,Microbiology,CC-BY,https://github.com/carrineblank/MicrO 9 | P18,1,1,0,1,0,1,1,1,1,1,1,0,1,0,1,1,1,0,1,0,0,0,0,0,1,0,0,0,0,0,0,1,0,0,0,0,http://purl.org/net/exergame/ns#,Exergame Ontology (EO),OWL,Serious Games,NA,NA 10 | P19,2,2,0,1,1,1,1,0,1,1,1,1,1,0,0,1,1,1,1,0,0,0,1,1,1,0,0,0,0,1,1,1,0,1,1,1,https://github.com/IEDB/MRO,MHC Restriction Ontology (MRO),OWL,Immunology,NA,https://github.com/IEDB/MRO 11 | P2,1,1,0,1,1,1,1,1,1,1,1,0,1,0,1,1,0,1,1,0,0,0,0,1,1,1,0,0,0,1,1,1,0,0,1,0,http://contsem.unizar.es/def/sector-publico/pproc.html,Public Procurement Ontology (PPROC),OWL,Public Procurement,CC Attribution ShareAlike 4.0 International (website only),https://github.com/pproc 12 | P3,2,1,0,1,0,1,1,1,1,1,1,0,1,0,1,1,0,1,1,0,0,0,1,1,1,0,0,0,0,0,1,1,1,0,1,0,http://vroddon.sdf-eu.org/mco/mco-core/,MPEG-21 Media Contract Ontology (MCO),OWL,Media Contracts,NA,NA 13 | P4,2,1,0,1,0,1,1,1,1,1,1,0,1,0,1,1,0,0,1,0,0,0,0,1,1,0,0,0,0,1,1,1,0,0,1,1,http://www.sparontologies.net/ontologies/doco/source.html,Document Components Ontology (DoCO),OWL,Semantic Publishing,NA,NA 14 | P6,2,1,0,1,0,1,1,1,1,1,1,0,0,0,1,1,1,1,1,0,0,0,1,1,1,0,0,0,0,1,1,1,0,1,1,0,http://www.dmo-foundry.org/,Data Mining OPtimization Ontology (DMOP),OWL,Semantic Meta Mining. Data Mining,,NA 15 | P8,2,1,0,1,0,1,1,1,1,1,1,0,0,0,1,1,1,1,0,0,0,0,0,1,1,0,0,0,0,0,1,1,0,1,1,0,https://mybody.inrialpes.fr/mycfembryo/,My Corporis Fabrica Embryo Ontology (MyEF),RDFS,"developmental anatomy, 3D",,NA 16 | P9,2,1,0,1,0,1,1,1,1,1,1,0,1,1,1,1,1,1,1,0,0,0,1,1,1,0,0,0,0,0,0,0,0,0,0,0,http://bio-omix.tmd.ac.jp/periodontitis/,Periodontitis-Ontology (PeriO),OBO,Disease,NA,NA 17 | -------------------------------------------------------------------------------- /supplementary/review_papers_in_timeframe.csv: -------------------------------------------------------------------------------- 1 | Title,Journal 2 | Distantly Supervised Web Relation Extraction for Knowledge Base Population,SWJ 3 | Crowd-based Ontology Engineering with the uComp Protege Plugin,SWJ 4 | Inferring Recommendation Interactions in Clinical Guidelines ,SWJ 5 | DWRank: Learning Concept Ranking for Ontology Search,SWJ 6 | Quality-Based Model For Effective and Robust Multi-User Pay-As-You-Go Ontology Matching,SWJ 7 | Visualizing Ontologies with VOWL,SWJ 8 | From hyperlinks to Semantic Web properties using Open Knowledge Extraction,SWJ 9 | An OWL ontology library representing judicial interpretations,SWJ 10 | Semantic Model for Legal Resources: Annotation and Reasoning over Normative Provisions,SWJ 11 | LOTED2: An Ontology of European Public Procurement Notices,SWJ 12 | "PPROC, an Ontology for Transparency in Public Procurement",SWJ 13 | Overview of the MPEG-21 Media Contract Ontology,SWJ 14 | Scalable Long-term Preservation of Relational Data through SPARQL queries,SWJ 15 | Signal/Collect: Processing Large Graphs in Seconds,SWJ 16 | The Document Components Ontology (DoCO),SWJ 17 | The Open University Linked Data - data.open.ac.uk,SWJ 18 | EventMedia: a LOD Dataset of Events Illustrated with Media,SWJ 19 | Time Ontology Extended for Non-Gregorian Calendar Applications,SWJ 20 | SISSVoc: A Linked Data API for access to SKOS vocabularies,SWJ 21 | Query Answering over Contextualized RDF/OWL Knowledge with Forall-Existential Bridge Rules: Decidable Finite Extension Classes,SWJ 22 | Quality Assessment for Linked Data: A Survey,SWJ 23 | LinkedSpending: OpenSpending becomes Linked Open Data,SWJ 24 | Semantic Technologies for Historical Research: A Survey,SWJ 25 | A logical characterisation of SPARQL federation,SWJ 26 | Link Traversal Querying for a diverse Web of Data,SWJ 27 | Forest Logging: A Trace-Based Analysis of Large Rule-Based Computations,SWJ 28 | "Sound, Complete and Minimal UCQ-Rewriting for Existential Rules",SWJ 29 | Probabilistic Description Logics under the Distribution Semantics,SWJ 30 | Coupling conceptual modeling and rules for the annotation of dramatic media,SWJ 31 | Multilingual Linked Data Patterns,SWJ 32 | Semantic Quran: A Multilingual Resource for Natural-Language Processing,SWJ 33 | Countering language attrition with PanLex and the Web of Data,SWJ 34 | DBnary: Wiktionary as a Lemon-Based Multilingual Lexical Resource in RDF,SWJ 35 | PAROLE/SIMPLE ‘lemon’ ontology and lexicons,SWJ 36 | "lemonUby - a large, interlinked, syntactically-rich lexical resource for ontologies",SWJ 37 | OLiA – Ontologies of Linguistic Annotation,SWJ 38 | Converting the PAROLE SIMPLE CLIPS Lexicon into RDF with lemon,SWJ 39 | Lexvo.org: Language-Related Information for the Linguistic Linked Data Cloud,SWJ 40 | Using ontologies to model human navigation behavior in information networks: A study based on Wikipedia,SWJ 41 | "OntoWiki - An Authoring, Publication and Visualization Interface for the Data Web",SWJ 42 | Collaborative multilingual knowledge management based on controlled natural language,SWJ 43 | "WYSIWYM - Integrated Visualization, Exploration and Authoring of Semantically Enriched Un-structured Content",SWJ 44 | Affective Graphs: The Visual Appeal of Linked Data,SWJ 45 | Linked SDMX Data,SWJ 46 | Migrating Bibliographic Datasets to the Semantic Web: the AGRIS case,SWJ 47 | Geospatial Dataset Curation through a Location-based Game,SWJ 48 | The RÉPENER Linked Dataset,SWJ 49 | eagle-i: biomedical research resource datasets,SWJ 50 | Converting neXtProt into Linked Data and nanopublications,SWJ 51 | Public spending as LOD: the case of Greece,SWJ 52 | "DBpedia - A Large-scale, Multilingual Knowledge Base Extracted from Wikipedia",SWJ 53 | Solving Guesstimation Problems Using the Semantic Web: Four Lessons from an Application,SWJ 54 | The Humanitarian eXchange Language: Coordinating Disaster Response with Semantic Web Technologies,SWJ 55 | A Dialogue with Linked Data: Voice-based Access to Market Data in the Sahel,SWJ 56 | Resource-Constrained Reasoning Using a Reasoner Composition Approach,SWJ 57 | A Closer Look at the Semantic Relationship between Datalog and Description Logics,SWJ 58 | Defeasibility in Answer Set Programs with Defaults and Argumentation Rules,SWJ 59 | Building semantic trees from XML documents,JWS 60 | The Graph of Things: A step towards the Live Knowledge Graph of connected things,JWS 61 | A rule-based agent-oriented approach for supporting weakly-structured scientific workflows,JWS 62 | Faceted search over RDF-based knowledge graphs,JWS 63 | CohEEL: Coherent and efficient named entity linking through random walks,JWS 64 | RIQ: Fast processing of SPARQL queries on RDF quadruples,JWS 65 | Sar-graphs: A language resource connecting linguistic knowledge with semantic relations from knowledge graphs,JWS 66 | Building event-centric knowledge graphs from news,JWS 67 | Learning the semantics of structured data sources,JWS 68 | Contextualized ranking of entity types based on knowledge graphs,JWS 69 | Triple Pattern Fragments: A low-cost knowledge graph interface for the Web,JWS 70 | Semantic Web in data mining and knowledge discovery: A comprehensive survey,JWS 71 | Personalized concept-based search on the Linked Open Data,JWS 72 | A mobile query service for integrated access to large numbers of online semantic web data sources,JWS 73 | Semantic reasoning on mobile devices: Do Androids dream of efficient reasoners?,JWS 74 | Queries with negation and inequalities over lightweight ontologies,JWS 75 | The description logic SHIQ with a flexible meta-modelling hierarchy,JWS 76 | C3D+P: A summarization method for interactive entity resolution,JWS 77 | The rationale of PROV,JWS 78 | Mining the Web of Linked Data with RapidMiner,JWS 79 | DIVE into the event-based browsing of linked historical media,JWS 80 | The Mannheim Search Join Engine,JWS 81 | Using semantic data to improve cross-lingual linking of article clusters,JWS 82 | Substructure counting graph kernels for machine learning from RDF data,JWS 83 | DeFacto—Temporal and multilingual Deep Fact Validation,JWS 84 | An unsupervised instance matcher for schema-free RDF data,JWS 85 | Automatic acquisition of class disjointness,JWS 86 | LC3: A spatio-temporal and semantic model for knowledge discovery from geospatial datasets,JWS 87 | Querying and integrating spatial–temporal information on the Web of Data via time geography,JWS 88 | Sextant: Visualizing time-evolving linked geospatial data,JWS 89 | Exposing INSPIRE on the Semantic Web,JWS 90 | A bootstrapping approach to entity linkage on the Semantic Web,JWS 91 | Neighborhood-user profiling based on perception relationship in the micro-blog scenario,JWS 92 | SACI: Sentiment analysis by collective inspection on social media content,JWS 93 | A novel approach to generate MCQs from domain ontology: Considering DL semantics and open-world assumption,JWS 94 | Inconsistency-tolerant query answering in ontology-based data access,JWS 95 | Optimising resolution-based rewriting algorithms for OWL ontologies,JWS 96 | Temporalizing rewritable query languages over knowledge bases,JWS 97 | Temporal query entailment in the Description Logic SHQ,JWS 98 | Complexity of answering counting aggregate queries over DL-Lite,JWS 99 | Engineering ontology-based access to real-world data sources,JWS 100 | Efficient SPARQL-to-SQL with R2RML mappings,JWS 101 | The Data Mining OPtimization Ontology,JWS 102 | Using a suite of ontologies for preserving workflow-centric research objects,JWS 103 | Methodology for geospatial data source discovery in ontology-driven geo-information integration architectures,JWS 104 | On the formulation of performant SPARQL queries,JWS 105 | Mining various semantic relationships from unstructured user-generated web data,JWS 106 | WSMO-Lite and hRESTS: Lightweight semantic annotations for Web services and RESTful APIs,JWS 107 | Linked hypernyms: Enriching DBpedia with Targeted Hypernym Discovery,JWS 108 | Global machine learning for spatial ontology population,JWS 109 | Discovering and understanding word level user intent in Web search queries,JWS 110 | SINA: Semantic interpretation of user queries for question answering on interlinked data,JWS 111 | Mímir: An open-source semantic search framework for interactive information seeking and discovery,JWS 112 | Tailored semantic annotation for semantic search,JWS 113 | An overview of semantic search evaluation initiatives,JWS 114 | OntoStudyEdit: a new approach for ontology-based representation and management of metadata in clinical and epidemiological research,JBMS 115 | Ontology-based representation and analysis of host-Brucella interactions,JBMS 116 | My Corporis Fabrica Embryo: An ontology-based 3D spatio-temporal modeling of human embryo development,JBMS 117 | KneeTex: an ontology–driven system for information extraction from MRI reports,JBMS 118 | Identification of sex-associated network patterns in Vaccine-Adverse Event Association Network in VAERS,JBMS 119 | Improving the Sequence Ontology terminology for genomic variant annotation,JBMS 120 | A framework for ontology-based question answering with application to parasite immunology,JBMS 121 | Development of an Ontology for Periodontitis,JBMS 122 | Ambiguity and variability of database and software names in bioinformatics,JBMS 123 | Discovering relations between indirectly connected biomedical concepts,JBMS 124 | Sequential pattern mining for discovering gene interactions and their contextual information from biomedical texts,JBMS 125 | Region Evolution eXplorer – A tool for discovering evolution trends in ontology regions,JBMS 126 | Networks of neuroinjury semantic predications to identify biomarkers for mild traumatic brain injury,JBMS 127 | Concept selection for phenotypes and diseases using learn to rank,JBMS 128 | Developing VISO: Vaccine Information Statement Ontology for patient education,JBMS 129 | Formalizing biomedical concepts from textual definitions,JBMS 130 | Capturing domain knowledge from multiple sources: the rare bone disorders use case,JBMS 131 | Annotation-based feature extraction from sets of SBML models,JBMS 132 | Evaluating drug-drug interaction information in NDF-RT and DrugBank,JBMS 133 | Exploring adverse drug events at the class level,JBMS 134 | Linking gene expression to phenotypes via pathway information,JBMS 135 | Semantic enrichment of longitudinal clinical study data using the CDISC standards and the semantic statistics vocabularies,JBMS 136 | Automated concept and relationship extraction for the semi-automated ontology management (SEAM) system,JBMS 137 | Development and validation of a classification approach for extracting severity automatically from electronic health records,JBMS 138 | Using description logics to evaluate the consistency of drug-class membership relations in NDF-RT,JBMS 139 | Completing the is-a structure in light-weight ontologies,JBMS 140 | Expanding the mammalian phenotype ontology to support automated exchange of high throughput mouse phenotyping data generated by large-scale mouse knockout screens,JBMS 141 | eNanoMapper: harnessing ontologies to enable data integration for nanomaterial risk assessment,JBMS 142 | Evaluating a variety of text-mined features for automatic protein function prediction with GOstruct,JBMS 143 | Supporting the annotation of chronic obstructive pulmonary disease (COPD) phenotypes with text mining workflows,JBMS 144 | Nanopublications for exposing experimental data in the life-sciences: a Huntington’s Disease case study,JBMS 145 | Development and application of an interaction network ontology for literature mining of vaccine-associated gene-gene interactions,JBMS 146 | Database citation in supplementary data linked to Europe PubMed Central full text biomedical articles,JBMS 147 | The cellular microscopy phenotype ontology,JBMS 148 | Filtering large-scale event collections using a combination of supervised and unsupervised learning for event trigger classification,JBMS 149 | Extracting a stroke phenotype risk factor from Veteran Health Administration clinical reports: an information content analysis,JBMS 150 | OmniSearch: a semantic search system based on the Ontology for MIcroRNA Target (OMIT) for microRNA-target gene interaction data,JBMS 151 | The Non-Coding RNA Ontology (NCRO): a comprehensive resource for the unification of non-coding RNA biology,JBMS 152 | OBIB-a novel ontology for biobanking,JBMS 153 | Active learning for ontological event extraction incorporating named entity recognition and unknown word handling,JBMS 154 | VICO: Ontology-based representation and integrative analysis of Vaccination Informed Consent forms,JBMS 155 | Extending gene ontology in the context of extracellular RNA and vesicle communication,JBMS 156 | "MicrO: an ontology of phenotypic and metabolic characters, assays, and culture media found in prokaryotic taxonomic descriptions",JBMS 157 | Representing vision and blindness,JBMS 158 | Inferring gene-to-phenotype and gene-to-disease relationships at Mouse Genome Informatics: challenges and solutions,JBMS 159 | Trivalent influenza vaccine adverse symptoms analysis based on MedDRA terminology using VAERS data in 2011,JBMS 160 | Modularising ontology and designing inference patterns to personalise health condition assessment: the case of obesity,JBMS 161 | Extracting drug-enzyme relation from literature as evidence for drug drug interaction,JBMS 162 | Using Semantic Web technologies for the generation of domain-specific templates to support clinical study metadata standards,JBMS 163 | miRiaD: A Text Mining Tool for Detecting Associations of microRNAs with Diseases,JBMS 164 | Linking rare and common disease: mapping clinical disease-phenotypes to ontologies in therapeutic target validation,JBMS 165 | An accurate and precise representation of drug ingredients,JBMS 166 | Towards exergaming commons: composing the exergame ontology for publishing open game data,JBMS 167 | Interoperability between phenotypes in research and healthcare terminologies—Investigating partial mappings between HPO and SNOMED CT,JBMS 168 | Reporting phenotypes in mouse models when considering body size as a potential confounder,JBMS 169 | An ontology for major histocompatibility restriction,JBMS 170 | -------------------------------------------------------------------------------- /supplementary/review_selected_papers.csv: -------------------------------------------------------------------------------- 1 | ID,Title,Journal,Year,Reviewer 3,Reviewer 1 2 | P1,LOTED2: An Ontology of European Public Procurement Notices,SWJ,2016,X,X 3 | P2,"PPROC, an Ontology for Transparency in Public Procurement",SWJ,2016,X,X 4 | P3,Overview of the MPEG-21 Media Contract Ontology,SWJ,2016,X,X 5 | P4,The Document Components Ontology (DoCO),SWJ,2016,,X 6 | P5,OLiA – Ontologies of Linguistic Annotation,SWJ,2015,,X 7 | P6,The Data Mining OPtimization Ontology,JWS,2015,X,X 8 | P7,WSMO-Lite and hRESTS: Lightweight semantic annotations for Web services and RESTful APIs,JWS,2015,,X 9 | P8,My Corporis Fabrica Embryo: An ontology-based 3D spatio-temporal modeling of human embryo development,JBMS,2015,X,X 10 | P9,Development of an Ontology for Periodontitis,JBMS,2015,X,X 11 | P10,Developing VISO: Vaccine Information Statement Ontology for patient education,JBMS,2015,X,X 12 | P11,Development and application of an interaction network ontology for literature mining of vaccine-associated gene-gene interactions,JBMS,2015,X, 13 | P12,The cellular microscopy phenotype ontology,JBMS,2016,X, 14 | P13,The Non-Coding RNA Ontology (NCRO): a comprehensive resource for the unification of non-coding RNA biology,JBMS,2016,X, 15 | P14,OBIB-a novel ontology for biobanking,JBMS,2016,X, 16 | P15,VICO: Ontology-based representation and integrative analysis of Vaccination Informed Consent forms,JBMS,2016,X, 17 | P16,"MicrO: an ontology of phenotypic and metabolic characters, assays, and culture media found in prokaryotic taxonomic descriptions",JBMS,2016,X, 18 | P17,Representing vision and blindness,JBMS,2016,,X 19 | P18,Towards exergaming commons: composing the exergame ontology for publishing open game data,JBMS,2016,,X 20 | P19,An ontology for major histocompatibility restriction,JBMS,2016,,X 21 | -------------------------------------------------------------------------------- /supplementary/review_titles.csv: -------------------------------------------------------------------------------- 1 | Title,Reviewer 1,Reviewer 2,Reviewer 3,Sum,Verdict 2 | Semantic Model for Legal Resources: Annotation and Reasoning over Normative Provisions,1,0,2,3,D 3 | LOTED2: An Ontology of European Public Procurement Notices,2,2,2,6,D 4 | "PPROC, an Ontology for Transparency in Public Procurement",2,2,2,6,D 5 | Overview of the MPEG-21 Media Contract Ontology,2,2,2,6,D 6 | The Document Components Ontology (DoCO),2,2,2,6,D 7 | Time Ontology Extended for Non-Gregorian Calendar Applications,1,2,2,5,D 8 | PAROLE/SIMPLE ‘lemon’ ontology and lexicons,2,1,2,5,D 9 | "lemonUby - a large, interlinked, syntactically-rich lexical resource for ontologies",2,0,0,2,D 10 | OLiA – Ontologies of Linguistic Annotation,2,1,2,5,D 11 | Using ontologies to model human navigation behavior in information networks: A study based on Wikipedia,1,0,2,3,D 12 | "DBpedia - A Large-scale, Multilingual Knowledge Base Extracted from Wikipedia",2,1,1,4,D 13 | LC3: A spatio-temporal and semantic model for knowledge discovery from geospatial datasets,2,0,2,4,D 14 | Exposing INSPIRE on the Semantic Web,2,0,0,2,D 15 | Engineering ontology-based access to real-world data sources,2,0,1,3,D 16 | The Data Mining OPtimization Ontology,2,2,2,6,D 17 | Using a suite of ontologies for preserving workflow-centric research objects,2,0,2,4,D 18 | WSMO-Lite and hRESTS: Lightweight semantic annotations for Web services and RESTful APIs,2,0,2,4,D 19 | Mímir: An open-source semantic search framework for interactive information seeking and discovery,2,0,1,3,D 20 | Ontology-based representation and analysis of host-Brucella interactions,2,1,2,5,D 21 | My Corporis Fabrica Embryo: An ontology-based 3D spatio-temporal modeling of human embryo development,2,1,2,5,D 22 | Improving the Sequence Ontology terminology for genomic variant annotation,2,2,2,6,D 23 | Development of an Ontology for Periodontitis,2,2,2,6,D 24 | Developing VISO: Vaccine Information Statement Ontology for patient education,2,2,2,6,D 25 | Development and validation of a classification approach for extracting severity automatically from electronic health records,2,0,0,2,D 26 | Expanding the mammalian phenotype ontology to support automated exchange of high throughput mouse phenotyping data generated by large-scale mouse knockout screens,1,2,2,5,D 27 | Development and application of an interaction network ontology for literature mining of vaccine-associated gene-gene interactions,2,2,2,6,D 28 | The cellular microscopy phenotype ontology,2,2,2,6,D 29 | The Non-Coding RNA Ontology (NCRO): a comprehensive resource for the unification of non-coding RNA biology,2,2,2,6,D 30 | OBIB-a novel ontology for biobanking,2,2,2,6,D 31 | VICO: Ontology-based representation and integrative analysis of Vaccination Informed Consent forms,2,0,2,4,D 32 | Extending gene ontology in the context of extracellular RNA and vesicle communication,2,2,2,6,D 33 | "MicrO: an ontology of phenotypic and metabolic characters, assays, and culture media found in prokaryotic taxonomic descriptions",2,2,2,6,D 34 | Representing vision and blindness,0,0,2,2,D 35 | An accurate and precise representation of drug ingredients,2,0,2,4,D 36 | Towards exergaming commons: composing the exergame ontology for publishing open game data,2,2,2,6,D 37 | An ontology for major histocompatibility restriction,2,2,0,4,D 38 | Distantly Supervised Web Relation Extraction for Knowledge Base Population,0,0,0,0,E 39 | Crowd-based Ontology Engineering with the uComp Protege Plugin,0,0,0,0,E 40 | Inferring Recommendation Interactions in Clinical Guidelines ,0,0,0,0,E 41 | DWRank: Learning Concept Ranking for Ontology Search,0,0,0,0,E 42 | Quality-Based Model For Effective and Robust Multi-User Pay-As-You-Go Ontology Matching,0,0,0,0,E 43 | Visualizing Ontologies with VOWL,0,0,0,0,E 44 | From hyperlinks to Semantic Web properties using Open Knowledge Extraction,0,0,0,0,E 45 | An OWL ontology library representing judicial interpretations,1,0,1,2,E 46 | Scalable Long-term Preservation of Relational Data through SPARQL queries,0,0,0,0,E 47 | Signal/Collect: Processing Large Graphs in Seconds,0,0,0,0,E 48 | The Open University Linked Data - data.open.ac.uk,0,0,1,1,E 49 | EventMedia: a LOD Dataset of Events Illustrated with Media,1,0,1,2,E 50 | SISSVoc: A Linked Data API for access to SKOS vocabularies,0,0,0,0,E 51 | Query Answering over Contextualized RDF/OWL Knowledge with Forall-Existential Bridge Rules: Decidable Finite Extension Classes,0,0,0,0,E 52 | Quality Assessment for Linked Data: A Survey,0,0,0,0,E 53 | LinkedSpending: OpenSpending becomes Linked Open Data,0,0,0,0,E 54 | Semantic Technologies for Historical Research: A Survey,0,0,0,0,E 55 | A logical characterisation of SPARQL federation,0,0,0,0,E 56 | Link Traversal Querying for a diverse Web of Data,0,0,0,0,E 57 | Forest Logging: A Trace-Based Analysis of Large Rule-Based Computations,0,0,0,0,E 58 | "Sound, Complete and Minimal UCQ-Rewriting for Existential Rules",0,0,0,0,E 59 | Probabilistic Description Logics under the Distribution Semantics,0,0,0,0,E 60 | Coupling conceptual modeling and rules for the annotation of dramatic media,0,0,1,1,E 61 | Multilingual Linked Data Patterns,0,0,0,0,E 62 | Semantic Quran: A Multilingual Resource for Natural-Language Processing,1,0,1,2,E 63 | Countering language attrition with PanLex and the Web of Data,1,0,0,1,E 64 | DBnary: Wiktionary as a Lemon-Based Multilingual Lexical Resource in RDF,0,0,1,1,E 65 | Converting the PAROLE SIMPLE CLIPS Lexicon into RDF with lemon,0,0,0,0,E 66 | Lexvo.org: Language-Related Information for the Linguistic Linked Data Cloud,0,0,0,0,E 67 | "OntoWiki - An Authoring, Publication and Visualization Interface for the Data Web",0,0,1,1,E 68 | Collaborative multilingual knowledge management based on controlled natural language,0,0,1,1,E 69 | "WYSIWYM - Integrated Visualization, Exploration and Authoring of Semantically Enriched Un-structured Content",0,0,0,0,E 70 | Affective Graphs: The Visual Appeal of Linked Data,0,0,0,0,E 71 | Linked SDMX Data,0,0,0,0,E 72 | Migrating Bibliographic Datasets to the Semantic Web: the AGRIS case,0,0,0,0,E 73 | Geospatial Dataset Curation through a Location-based Game,0,0,0,0,E 74 | The RÉPENER Linked Dataset,1,0,0,1,E 75 | eagle-i: biomedical research resource datasets,1,0,0,1,E 76 | Converting neXtProt into Linked Data and nanopublications,0,0,0,0,E 77 | Public spending as LOD: the case of Greece,0,0,0,0,E 78 | Solving Guesstimation Problems Using the Semantic Web: Four Lessons from an Application,0,0,0,0,E 79 | The Humanitarian eXchange Language: Coordinating Disaster Response with Semantic Web Technologies,0,0,1,1,E 80 | A Dialogue with Linked Data: Voice-based Access to Market Data in the Sahel,0,0,0,0,E 81 | Resource-Constrained Reasoning Using a Reasoner Composition Approach,0,0,0,0,E 82 | A Closer Look at the Semantic Relationship between Datalog and Description Logics,0,0,0,0,E 83 | Defeasibility in Answer Set Programs with Defaults and Argumentation Rules,0,0,0,0,E 84 | Building semantic trees from XML documents,0,0,0,0,E 85 | The Graph of Things: A step towards the Live Knowledge Graph of connected things,0,0,0,0,E 86 | A rule-based agent-oriented approach for supporting weakly-structured scientific workflows,0,0,0,0,E 87 | Faceted search over RDF-based knowledge graphs,0,0,0,0,E 88 | CohEEL: Coherent and efficient named entity linking through random walks,0,0,0,0,E 89 | RIQ: Fast processing of SPARQL queries on RDF quadruples,0,0,0,0,E 90 | Sar-graphs: A language resource connecting linguistic knowledge with semantic relations from knowledge graphs,0,0,0,0,E 91 | Building event-centric knowledge graphs from news,0,0,0,0,E 92 | Learning the semantics of structured data sources,0,0,0,0,E 93 | Contextualized ranking of entity types based on knowledge graphs,0,0,0,0,E 94 | Triple Pattern Fragments: A low-cost knowledge graph interface for the Web,0,0,0,0,E 95 | Semantic Web in data mining and knowledge discovery: A comprehensive survey,0,0,0,0,E 96 | Personalized concept-based search on the Linked Open Data,0,0,0,0,E 97 | A mobile query service for integrated access to large numbers of online semantic web data sources,0,0,0,0,E 98 | Semantic reasoning on mobile devices: Do Androids dream of efficient reasoners?,0,0,0,0,E 99 | Queries with negation and inequalities over lightweight ontologies,0,0,0,0,E 100 | The description logic SHIQ with a flexible meta-modelling hierarchy,0,0,0,0,E 101 | C3D+P: A summarization method for interactive entity resolution,0,0,0,0,E 102 | The rationale of PROV,1,0,1,2,E 103 | Mining the Web of Linked Data with RapidMiner,0,0,0,0,E 104 | DIVE into the event-based browsing of linked historical media,1,0,0,1,E 105 | The Mannheim Search Join Engine,0,0,0,0,E 106 | Using semantic data to improve cross-lingual linking of article clusters,0,0,0,0,E 107 | Substructure counting graph kernels for machine learning from RDF data,0,0,0,0,E 108 | DeFacto—Temporal and multilingual Deep Fact Validation,1,0,0,1,E 109 | An unsupervised instance matcher for schema-free RDF data,0,0,0,0,E 110 | Automatic acquisition of class disjointness,0,0,0,0,E 111 | Querying and integrating spatial–temporal information on the Web of Data via time geography,0,0,1,1,E 112 | Sextant: Visualizing time-evolving linked geospatial data,0,0,0,0,E 113 | A bootstrapping approach to entity linkage on the Semantic Web,0,0,0,0,E 114 | Neighborhood-user profiling based on perception relationship in the micro-blog scenario,0,0,0,0,E 115 | SACI: Sentiment analysis by collective inspection on social media content,0,0,0,0,E 116 | A novel approach to generate MCQs from domain ontology: Considering DL semantics and open-world assumption,0,0,0,0,E 117 | Inconsistency-tolerant query answering in ontology-based data access,0,0,0,0,E 118 | Optimising resolution-based rewriting algorithms for OWL ontologies,0,0,0,0,E 119 | Temporalizing rewritable query languages over knowledge bases,0,0,0,0,E 120 | Temporal query entailment in the Description Logic SHQ,0,0,0,0,E 121 | Complexity of answering counting aggregate queries over DL-Lite,0,0,0,0,E 122 | Efficient SPARQL-to-SQL with R2RML mappings,0,0,0,0,E 123 | Methodology for geospatial data source discovery in ontology-driven geo-information integration architectures,0,0,1,1,E 124 | On the formulation of performant SPARQL queries,0,0,0,0,E 125 | Mining various semantic relationships from unstructured user-generated web data,0,0,0,0,E 126 | Linked hypernyms: Enriching DBpedia with Targeted Hypernym Discovery,0,0,0,0,E 127 | Global machine learning for spatial ontology population,0,0,0,0,E 128 | Discovering and understanding word level user intent in Web search queries,0,0,0,0,E 129 | SINA: Semantic interpretation of user queries for question answering on interlinked data,0,0,0,0,E 130 | Tailored semantic annotation for semantic search,0,0,0,0,E 131 | An overview of semantic search evaluation initiatives,0,0,0,0,E 132 | OntoStudyEdit: a new approach for ontology-based representation and management of metadata in clinical and epidemiological research,1,1,1,3,E 133 | KneeTex: an ontology–driven system for information extraction from MRI reports,1,1,1,3,E 134 | Identification of sex-associated network patterns in Vaccine-Adverse Event Association Network in VAERS,0,0,0,0,E 135 | A framework for ontology-based question answering with application to parasite immunology,1,0,1,2,E 136 | Ambiguity and variability of database and software names in bioinformatics,0,0,0,0,E 137 | Discovering relations between indirectly connected biomedical concepts,0,0,0,0,E 138 | Sequential pattern mining for discovering gene interactions and their contextual information from biomedical texts,0,0,0,0,E 139 | Region Evolution eXplorer – A tool for discovering evolution trends in ontology regions,0,0,0,0,E 140 | Networks of neuroinjury semantic predications to identify biomarkers for mild traumatic brain injury,0,0,0,0,E 141 | Concept selection for phenotypes and diseases using learn to rank,0,0,0,0,E 142 | Formalizing biomedical concepts from textual definitions,0,0,0,0,E 143 | Capturing domain knowledge from multiple sources: the rare bone disorders use case,1,1,1,3,E 144 | Annotation-based feature extraction from sets of SBML models,0,0,0,0,E 145 | Evaluating drug-drug interaction information in NDF-RT and DrugBank,0,0,0,0,E 146 | Exploring adverse drug events at the class level,1,0,0,1,E 147 | Linking gene expression to phenotypes via pathway information,0,0,0,0,E 148 | Semantic enrichment of longitudinal clinical study data using the CDISC standards and the semantic statistics vocabularies,0,0,0,0,E 149 | Automated concept and relationship extraction for the semi-automated ontology management (SEAM) system,0,0,0,0,E 150 | Using description logics to evaluate the consistency of drug-class membership relations in NDF-RT,0,0,0,0,E 151 | Completing the is-a structure in light-weight ontologies,0,0,0,0,E 152 | eNanoMapper: harnessing ontologies to enable data integration for nanomaterial risk assessment,0,0,1,1,E 153 | Evaluating a variety of text-mined features for automatic protein function prediction with GOstruct,0,0,0,0,E 154 | Supporting the annotation of chronic obstructive pulmonary disease (COPD) phenotypes with text mining workflows,0,0,0,0,E 155 | Nanopublications for exposing experimental data in the life-sciences: a Huntington’s Disease case study,0,0,0,0,E 156 | Database citation in supplementary data linked to Europe PubMed Central full text biomedical articles,0,0,0,0,E 157 | Filtering large-scale event collections using a combination of supervised and unsupervised learning for event trigger classification,0,0,0,0,E 158 | Extracting a stroke phenotype risk factor from Veteran Health Administration clinical reports: an information content analysis,0,0,1,1,E 159 | OmniSearch: a semantic search system based on the Ontology for MIcroRNA Target (OMIT) for microRNA-target gene interaction data,1,0,1,2,E 160 | Active learning for ontological event extraction incorporating named entity recognition and unknown word handling,0,0,0,0,E 161 | Inferring gene-to-phenotype and gene-to-disease relationships at Mouse Genome Informatics: challenges and solutions,0,0,0,0,E 162 | Trivalent influenza vaccine adverse symptoms analysis based on MedDRA terminology using VAERS data in 2011,0,0,1,1,E 163 | Modularising ontology and designing inference patterns to personalise health condition assessment: the case of obesity,0,1,0,1,E 164 | Extracting drug-enzyme relation from literature as evidence for drug drug interaction,0,0,0,0,E 165 | Using Semantic Web technologies for the generation of domain-specific templates to support clinical study metadata standards,0,0,0,0,E 166 | miRiaD: A Text Mining Tool for Detecting Associations of microRNAs with Diseases,0,0,0,0,E 167 | Linking rare and common disease: mapping clinical disease-phenotypes to ontologies in therapeutic target validation,0,0,0,0,E 168 | Interoperability between phenotypes in research and healthcare terminologies—Investigating partial mappings between HPO and SNOMED CT,0,0,0,0,E 169 | Reporting phenotypes in mouse models when considering body size as a potential confounder,0,0,0,0,E 170 | --------------------------------------------------------------------------------