├── LICENSE ├── README.md ├── phylografter-sync ├── .gitignore ├── .sync_status.json └── refresh_nexsons_from_phylografter.py ├── pull-studies.bash ├── shard-list.txt └── shards ├── .gitignore └── README.md /LICENSE: -------------------------------------------------------------------------------- 1 | Copyright (c) 2013, Open Tree Of Life Team 2 | 3 | All rights reserved. 4 | 5 | Redistribution and use in source and binary forms, with or without 6 | modification, are permitted provided that the following conditions are met: 7 | 8 | Redistributions of source code must retain the above copyright notice, this 9 | list of conditions and the following disclaimer. 10 | 11 | Redistributions in binary form must reproduce the above copyright notice, 12 | this list of conditions and the following disclaimer in the documentation 13 | and/or other materials provided with the distribution. 14 | 15 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND 16 | ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED 17 | WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE 18 | DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE 19 | FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL 20 | DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR 21 | SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER 22 | CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, 23 | OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 24 | OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 25 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # phylesystem 2 | 3 | This is the top-level repository in the Open Tree of Life 4 | phylogenetic study document store. 5 | The data are actually stored in different repositories referred 6 | to as "shards" in much of our documentation/email discussions. 7 | 8 | If you want to browse the data in phylesystem, you can 9 | look at the repository for the only shard. The repo is at https://github.com/OpenTreeOfLife/phylesystem-1 10 | If you want to find study `ot_WXYZ` you should look in `study/ot_YZ/ot_WXYZ/ot_WXYZ` (in other words, in the 11 | study subdirectory look in the directory named using the study's prefix ("ot_" or "pg_") followed by the last 2 digits of the ID. In that directory there will be a subdirectory that corresponds to the study ID. 12 | and the JSON file will be in that directory. 13 | 14 | Phylesystem serves as a crude database storage for the 15 | web services implemented using the code base 16 | [phylesystem-api](https://github.com/OpenTreeOfLife/phylesystem-api). 17 | This layer is the API for accessing the NexSON data store. 18 | 19 | Currently we do not support pull requests to the shards. 20 | The server running the NexSON services API is the only 21 | writer. We hope to add functionality to smoothly deal 22 | with pull requests soon. 23 | 24 | See [our publication on phylesystem](http://dx.doi.org/10.1093/bioinformatics/btv276) for details. See [the germinator repository's wiki](https://github.com/OpenTreeOfLife/germinator/wiki) for Open Tree of Life documentation. 25 | 26 | # Usage 27 | 28 | Most users will only want/need to run the 29 | 30 | $ bash pull-studies.bash 31 | 32 | script to fetch and pull the latest NexSON for all of the shards 33 | 34 | 35 | 36 | 37 | 38 | # History 39 | 40 | The repository formerly known as phylesystem was moved to: 41 | https://github.com/mtholder/old-phylesystem 42 | before the current architecture was decided. 43 | That repository has a set of files that agree with the intial commit to the 44 | phylesystem-1 repo; and it had scripts for updating from phylografter 45 | 46 | ## License 47 | 48 | The *code* in this repository is released under the 2-clause BSD license. See 49 | the LICENSE file for more details. 50 | 51 | The *data* in this repository is released under the [Create Commons Zero 1.0 Universal](https://creativecommons.org/publicdomain/zero/1.0/) license. 52 | 53 | -------------------------------------------------------------------------------- /phylografter-sync/.gitignore: -------------------------------------------------------------------------------- 1 | study 2 | -------------------------------------------------------------------------------- /phylografter-sync/.sync_status.json: -------------------------------------------------------------------------------- 1 | { 2 | "default_sha": "9e0896532fb64d26c40a4498622b3583a2447f28", 3 | "from": "2010-01-01T00:00:00", 4 | "studies": [], 5 | "study2sha": { 6 | "1087": "68bbbaf75d398c3aeb5474ccd0aba0fb8a135c6a", 7 | "1446": "cfc0bfa7bcdf5489f6d26e631085d0cdc0bd6573", 8 | "2421": "9b53b5a43d482323aa54a51126b7e143497ec1e4", 9 | "2422": "9b53b5a43d482323aa54a51126b7e143497ec1e4", 10 | "2428": "9b53b5a43d482323aa54a51126b7e143497ec1e4", 11 | "2484": "9b53b5a43d482323aa54a51126b7e143497ec1e4", 12 | "2586": "9b53b5a43d482323aa54a51126b7e143497ec1e4", 13 | "2618": "9b53b5a43d482323aa54a51126b7e143497ec1e4", 14 | "2719": "9b53b5a43d482323aa54a51126b7e143497ec1e4", 15 | "2737": "9b53b5a43d482323aa54a51126b7e143497ec1e4", 16 | "2738": "9b53b5a43d482323aa54a51126b7e143497ec1e4", 17 | "2739": "9b53b5a43d482323aa54a51126b7e143497ec1e4", 18 | "2744": "9b53b5a43d482323aa54a51126b7e143497ec1e4", 19 | "2760": "9b53b5a43d482323aa54a51126b7e143497ec1e4", 20 | "2770": "9b53b5a43d482323aa54a51126b7e143497ec1e4", 21 | "2772": "9b53b5a43d482323aa54a51126b7e143497ec1e4", 22 | "2787": "9b53b5a43d482323aa54a51126b7e143497ec1e4", 23 | "2793": "9b53b5a43d482323aa54a51126b7e143497ec1e4", 24 | "2846": "9b53b5a43d482323aa54a51126b7e143497ec1e4", 25 | "2847": "cfc0bfa7bcdf5489f6d26e631085d0cdc0bd6573", 26 | "2862": "b1001d8bad27478b84a70517cdf6dadbac1a0dfb", 27 | "2863": "8e7ca5e20a1f19fa7d086c5b24464dfb63a795ef", 28 | "2880": "0c525f54607b28b48d5818d62a358259028fc07c", 29 | "2882": "0c525f54607b28b48d5818d62a358259028fc07c", 30 | "2883": "0c525f54607b28b48d5818d62a358259028fc07c", 31 | "2884": "0c525f54607b28b48d5818d62a358259028fc07c", 32 | "2885": "0c525f54607b28b48d5818d62a358259028fc07c", 33 | "2886": "0c525f54607b28b48d5818d62a358259028fc07c", 34 | "2891": "0c525f54607b28b48d5818d62a358259028fc07c", 35 | "2892": "0c525f54607b28b48d5818d62a358259028fc07c", 36 | "2895": "0c525f54607b28b48d5818d62a358259028fc07c", 37 | "2914": "0c525f54607b28b48d5818d62a358259028fc07c", 38 | "2925": "0c525f54607b28b48d5818d62a358259028fc07c", 39 | "2927": "0c525f54607b28b48d5818d62a358259028fc07c", 40 | "2928": "0c525f54607b28b48d5818d62a358259028fc07c", 41 | "2929": "0c525f54607b28b48d5818d62a358259028fc07c", 42 | "2930": "0c525f54607b28b48d5818d62a358259028fc07c", 43 | "2931": "0c525f54607b28b48d5818d62a358259028fc07c", 44 | "36": "0c525f54607b28b48d5818d62a358259028fc07c", 45 | "516": "0c525f54607b28b48d5818d62a358259028fc07c", 46 | "865": "0c525f54607b28b48d5818d62a358259028fc07c" 47 | }, 48 | "to": "2014-06-10T23:17:22", 49 | "to_download_from_pg": [], 50 | "to_upload_to_phylesystem": [] 51 | } -------------------------------------------------------------------------------- /phylografter-sync/refresh_nexsons_from_phylografter.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | from peyotl.phylografter.sync import PhylografterNexsonDocStoreSync 3 | import sys 4 | import os 5 | 6 | def get_default_dir_dict(top_level=None): 7 | r = '.' if top_level is None else top_level 8 | t = os.path.abspath(r) 9 | d = {'nexson_dir': t, 10 | 'nexson_state_db': os.path.join(t, '.sync_status.json'), # stores the state of this repo. *very* hacky primitive db. 11 | } 12 | return d 13 | 14 | 15 | if __name__ == '__main__': 16 | if '-h' in sys.argv: 17 | sys.stderr.write('''sync.py is a short-term hack. 18 | 19 | It stores info about the last communication with phylografter in .to_download.json 20 | Based on this info, it tries to download as few studies as possible to make 21 | the NexSONs in treenexus/studies/#/... match the export from phylografter. 22 | 23 | -h gives this help message. 24 | 25 | If other arguments aree supplied, it should be the study #'s to be downloaded. 26 | ''') 27 | sys.exit(0) 28 | 29 | dd = get_default_dir_dict() 30 | if len(sys.argv) > 1: 31 | to_download = sys.argv[1:] 32 | else: 33 | to_download = None 34 | sync = PhylografterNexsonDocStoreSync(dd) 35 | sync.run(to_download=to_download) 36 | 37 | -------------------------------------------------------------------------------- /pull-studies.bash: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | pushd . >/dev/null 3 | scriptdir=$(dirname "$0") 4 | cd $scriptdir 5 | for giturl in $(cat shard-list.txt) 6 | do 7 | shardreponame=$(echo $giturl | sed -E 's/.*\///' | sed -E 's/\.git//') 8 | rd=$(echo "shards/$shardreponame") 9 | if test -d "$rd" 10 | then 11 | cd "$rd" 12 | echo "pulling to update $shardreponame" 13 | git pull origin master 14 | else 15 | cd shards 16 | echo "cloning $giturl" 17 | git clone "$giturl" 18 | fi 19 | cd - > /dev/null 20 | done 21 | 22 | popd >/dev/null 23 | -------------------------------------------------------------------------------- /shard-list.txt: -------------------------------------------------------------------------------- 1 | https://github.com/OpenTreeOfLife/phylesystem-1.git 2 | -------------------------------------------------------------------------------- /shards/.gitignore: -------------------------------------------------------------------------------- 1 | phylesystem-* 2 | -------------------------------------------------------------------------------- /shards/README.md: -------------------------------------------------------------------------------- 1 | This directory should be treated as read-only. --------------------------------------------------------------------------------