├── .gitignore ├── backup_db.sh ├── config.yml ├── docker-compose.yml ├── start.sql ├── start.sh ├── Dockerfile └── README.md /.gitignore: -------------------------------------------------------------------------------- 1 | data/ 2 | data/* 3 | AnnotationHubServer3.0/ 4 | AnnotationHubServer3.0/* -------------------------------------------------------------------------------- /backup_db.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | mysqldump -u root --password=$MYSQL_ROOT_PASSWORD -h db annotationhub | gzip > /data/annotationhub.sql.gz 4 | -------------------------------------------------------------------------------- /config.yml: -------------------------------------------------------------------------------- 1 | mysql_url: "mysql2://hubuser:MYSQL_REMOTE_PASSWORD@db/annotationhub" 2 | sqlite_filename: "annotationhub.sqlite3" 3 | ahs_database_type: "mysql" -------------------------------------------------------------------------------- /docker-compose.yml: -------------------------------------------------------------------------------- 1 | db: 2 | image: "mysql:8.0.19" 3 | environment: 4 | MYSQL_ROOT_PASSWORD: mysecretpassword 5 | annotationhub: 6 | environment: 7 | MYSQL_ROOT_PASSWORD: mysecretpassword 8 | MYSQL_REMOTE_PASSWORD: 9 | image: "bioconductor/annotationhub_docker" 10 | links: 11 | - "db" 12 | ports: 13 | - "3000:3000" 14 | volumes: 15 | - "./AnnotationHubServer3.0:/AnnotationHubServer3.0" 16 | - "./data:/data" 17 | command: "/bin/bash --login /tmp/start.sh" 18 | 19 | -------------------------------------------------------------------------------- /start.sql: -------------------------------------------------------------------------------- 1 | drop database if exists annotationhub; 2 | create database annotationhub; 3 | flush privileges; 4 | DROP USER IF EXISTS 'hubuser'@'%'; 5 | flush privileges; 6 | CREATE USER 'hubuser'@'%' IDENTIFIED BY 'MYSQL_REMOTE_PASSWORD'; 7 | GRANT ALL PRIVILEGES ON annotationhub.* TO 'hubuser'@'%' WITH GRANT OPTION; 8 | flush privileges; 9 | DROP USER IF EXISTS 'ahuser'@'%'; 10 | flush privileges; 11 | CREATE USER 'ahuser'@'%' IDENTIFIED BY 'MYSQL_REMOTE_PASSWORD'; 12 | GRANT ALL PRIVILEGES ON annotationhub.* TO 'ahuser'@'%' WITH GRANT OPTION; 13 | flush privileges; 14 | -------------------------------------------------------------------------------- /start.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | 3 | if [ -z "${MYSQL_REMOTE_PASSWORD}" ]; then 4 | echo "MYSQL_REMOTE_PASSWORD is not set; exiting!" 5 | echo "(see README.md)" 6 | exit 1 7 | fi 8 | 9 | cd /tmp 10 | 11 | echo get database from master 12 | mysqldump --lock-tables=false --password=$MYSQL_REMOTE_PASSWORD -u hubuser_readonly -h annotationhub.bioconductor.org annotationhub | gzip > /tmp/dump.sql.gz 13 | 14 | echo create local database and user 15 | cat /tmp/start.sql | sed "s/MYSQL_REMOTE_PASSWORD/$MYSQL_REMOTE_PASSWORD/" | mysql -u root --password=$MYSQL_ROOT_PASSWORD -h db 16 | 17 | echo populate local database 18 | zcat /tmp/dump.sql.gz | mysql -u root --password=$MYSQL_ROOT_PASSWORD -h db annotationhub 19 | 20 | curl https://annotationhub.bioconductor.org/metadata/annotationhub.sqlite3 > /AnnotationHubServer3.0/annotationhub.sqlite3 21 | 22 | cd /AnnotationHubServer3.0 23 | 24 | cat /tmp/config.yml | sed "s/MYSQL_REMOTE_PASSWORD/$MYSQL_REMOTE_PASSWORD/" > config.yml 25 | 26 | echo start app 27 | #ruby app.rb 28 | shotgun -o 0.0.0.0 -p 3000 app.rb 29 | 30 | sleep infinity 31 | 32 | -------------------------------------------------------------------------------- /Dockerfile: -------------------------------------------------------------------------------- 1 | # call me bioconductor/annotationhub_docker 2 | 3 | FROM ubuntu:18.04 4 | 5 | EXPOSE 3000 6 | 7 | RUN DEBIAN_FRONTEND=noninteractive \ 8 | apt-get update && apt-get dist-upgrade -y 9 | 10 | RUN DEBIAN_FRONTEND=noninteractive \ 11 | apt-get install -y \ 12 | curl g++ git libsqlite3-dev make patch rsync subversion \ 13 | build-essential zlib1g-dev libssl-dev libreadline-dev libyaml-dev \ 14 | libxml2-dev libxslt-dev libffi-dev mysql-client \ 15 | libmysqlclient-dev 16 | 17 | RUN git clone https://github.com/rbenv/rbenv.git ~/.rbenv 18 | 19 | RUN echo 'eval "$(rbenv init -)"' >> ~/.bash_profile 20 | 21 | RUN git clone https://github.com/rbenv/ruby-build.git ~/.rbenv/plugins/ruby-build 22 | 23 | RUN ./root/.rbenv/plugins/ruby-build/install.sh 24 | 25 | ENV PATH /root/.rbenv/bin:/root/.rbenv/shims:$PATH 26 | ENV CONFIGURE_OPTS --disable-install-doc 27 | 28 | #RUN source ~/.bash_profile 29 | 30 | RUN rbenv install -v 2.6.5 31 | RUN rbenv global 2.6.5 32 | 33 | RUN cd /tmp && curl -LO https://raw.githubusercontent.com/Bioconductor/AnnotationHubServer3.0/master/Gemfile && \ 34 | gem install bundler && bundle install 35 | 36 | ENV AHS_DATABASE_TYPE mysql 37 | 38 | RUN echo hi2 39 | 40 | ADD start.sh /tmp/start.sh 41 | ADD start.sql /tmp/start.sql 42 | ADD config.yml /tmp/config.yml 43 | ADD backup_db.sh /bin/backup_db.sh 44 | 45 | 46 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | 2 | # Docker Container for AnnotationHub Server 3 | 4 | ## Important Note 5 | 6 | The way this Docker Container is currently set up, 7 | it can only be used by Bioconductor core members 8 | who have access to the production server 9 | and know its credentials. 10 | 11 | It should probably change so it can be used by 12 | anyone who wants to test recipes. But that 13 | is not going to happen right away. 14 | 15 | ## Table of Contents 16 | 17 | - [Setting up the docker container](#setup) 18 | - [Prerequisites](#prerequisites) 19 | - [Run the continer](#run) 20 | - [Determine URL of server](#url) 21 | - [Using the container](#use) 22 | - [Testing the changes](#test) 23 | - [Pushing changes to production](#push) 24 | 25 | 26 | ## Setting up the docker container 27 | 28 | 29 | ### Prerequisites 30 | 31 | Set the environment variable `MYSQL_REMOTE_PASSWORD` to 32 | the correct value, found in the "AnnotationHub production 33 | server" section of the Google Doc "Credentials For 34 | Bioconductor Cloud resources". 35 | 36 | The container will not work properly unless this is set. 37 | 38 | You can set it by doing: 39 | 40 | export MYSQL_REMOTE_PASSWORD=XXX 41 | 42 | where XXX is replaced with the correct password. This password should also be 43 | added to 'MYSQL_REMOTE_PASSWORD:' in the docker-compose.yml file. 44 | 45 | If you are on Linux, install docker using 46 | [Docker Engine](https://docs.docker.com/installation/). Also install 47 | [Docker Compose](https://docs.docker.com/compose/install/). 48 | If you are on Mac or Windows, install Docker Desktop instead. You will not need 49 | to install Docker Compose because it is included in Docker Desktop. 50 | [Docker Desktop for Mac](https://docs.docker.com/docker-for-mac/install/), 51 | [Docker Desktop for Windows](https://docs.docker.com/docker-for-windows/install/). 52 | 53 | Clone the AnnotationHubServer3.0 code to the same directory as 54 | this README: 55 | 56 | git clone https://github.com/Bioconductor/AnnotationHubServer3.0.git 57 | 58 | Or if you already have this repository checked out elsewhere 59 | on your system, make a symbolic link to it in the current directory 60 | (the same directory as this README). 61 | 62 | Create a directory called "data" in the same directory as 63 | this README. 64 | 65 | 66 | ### Run The Container 67 | 68 | Open a terminal window (a Docker Quickstart Terminal if 69 | you are on Mac or windows) and change to the same 70 | directory where this README is. Issue the command: 71 | 72 | docker-compose up 73 | 74 | This command will *pull down the database contents 75 | from the production AnnotationHub server* and then 76 | start a local server. 77 | 78 | When you see a line like this: 79 | 80 | annotationhub_1 | == Shotgun/WEBrick on http://0.0.0.0:3000/ 81 | 82 | ...the AnnotationHub server is running in the container. 83 | 84 | To verify that it is running, you can determine its URL 85 | (in the next section). 86 | 87 | 88 | ### Determining URL of server 89 | 90 | If you are on linux, the URL of the server is likely 91 | [http://localhost:3000/resource](http://localhost:3000/resource). 92 | 93 | If you are in the cloud you need to use the public DNS name of your instance 94 | as your IP address. 95 | 96 | If you are using boot2docker (deprecated by Docker Toolbox), 97 | you can determine your Docker host's IP address with the command 98 | `boot2docker ip`. If this returns `1.2.3.4`, your URL 99 | would be `http://1.2.3.4:3000/resource`. 100 | 101 | If you are using Docker Toolbox, the command to determine 102 | your Docker host's IP address is 103 | 104 | docker-machine ip default 105 | 106 | If this returns `1.2.3.4`, your URL would be 107 | `http://1.2.3.4:3000/resource`. 108 | 109 | 110 | ## Using the container 111 | 112 | Start a new R session in a new terminal window. 113 | Assuming your server URL 114 | is `http://1.2.3.4:3000/resource`, enter the following 115 | at the R prompt: 116 | 117 | options(AH_SERVER_POST_URL="http://1.2.3.4:3000/resource") ## used by AnnotationHubData to insert metadata 118 | options(ANNOTATION_HUB_URL="http://1.2.3.4:3000") ## used by AnnotationHub to get metadata 119 | 120 | Replace the URL with your actual URL, of course. These options must be set before loading `AnnotationHub` and `AnnotationHubData`. 121 | 122 | library(AnnotationHub) 123 | library(AnnotationHubData) 124 | 125 | Now you can run recipes, etc. and the insertions will 126 | happen inside the docker container, not in the production 127 | database. 128 | 129 | 130 | ## Testing the changes 131 | 132 | You can interact with the docker db with R or mysql. 133 | 134 | #### R session 135 | 136 | To view the docker db from R you must first convert the mysql db to sqlite. From another terminal window do 137 | 138 | docker exec -ti annotationhub_annotationhub_1 bash 139 | 140 | That will log you in to the annotationhub server container. Then do the following: 141 | 142 | cd /AnnotationHubServer3.0/ 143 | ruby convert_db.rb 144 | 145 | That will convert the mysql database to sqlite. You can then type 146 | 147 | exit 148 | 149 | to exit the container, start R and set the variables as described in [Using the container](#use). AnnotationHub should recognize that there are changes and download the new sqlite database, but if not you can remove the old one from your cache to trigger a copy. 150 | 151 | #### mysql session 152 | 153 | mysql is not exposed to the host, so you need to do this from the mysql machine. Run 154 | 155 | docker ps 156 | 157 | to see a list of containers that are running. One will have the string 'db' in it. Let's say the full name is 'db1'. Connect to the container with: 158 | 159 | docker exec -ti db_1 bash 160 | 161 | From within the resulting prompt start mysql and query the db as usual: 162 | 163 | mysql -p -u ahuser 164 | 165 | When you are satisfied that the changes you have made are 166 | correct, you can update the production database (see next 167 | section). If you have messed up and you don't want to 168 | push your changes to production, you can just exit 169 | the container (press Control-C in the window where it is 170 | running) and start over again. 171 | 172 | 173 | ## Pushing changes to production 174 | 175 | You need to back up the database inside the docker 176 | container. You can do it like this: 177 | 178 | docker exec annotationhub_annotationhub_1 bash /bin/backup_db.sh 179 | 180 | Note that `annotationhub_annotationhub_1` is the name of the 181 | docker container that has the annotation hub server on it; this 182 | name may vary, the `docker ps` command will give you the 183 | accurate container name. 184 | 185 | Now in the `data` directory on your local machine, 186 | there is a file called `annotationhub.sql.gz`. Upload this to the production machine. 187 | 188 | Log into the production machine and make a backup of production db: 189 | 190 | mysqldump -p -u ahuser annotationhub | gzip > dbdump_YYYY_MM_DD_fromProd.sql.gz 191 | 192 | Drop the old db and create an empty one: 193 | 194 | mysql -p -u root 195 | drop database annotationhub; 196 | create database annotationhub; 197 | quit; 198 | 199 | Fill the empty with the modified db: 200 | 201 | zcat dbdump_YYYY_MM_DD_fromDocker.sql.gz | mysql -p -u root annotationhub 202 | --------------------------------------------------------------------------------