├── .dockerignore ├── .gitignore ├── Dockerfile ├── README.md ├── brwsr-stardog ├── Dockerfile ├── data │ └── placeholder ├── docker-compose.yml ├── readme.md ├── stardog.properties └── start-stardog-service.sh ├── docker-compose.yml ├── entrypoint.sh ├── load.sh ├── requirements.txt ├── src ├── app │ ├── __init__.py │ ├── client.py │ ├── config-template.py │ ├── static │ │ ├── bootstrap.min.css │ │ ├── brwsr.css │ │ ├── brwsr.js │ │ ├── fonts │ │ │ ├── glyphicons-halflings-regular.eot │ │ │ ├── glyphicons-halflings-regular.svg │ │ │ ├── glyphicons-halflings-regular.ttf │ │ │ └── glyphicons-halflings-regular.woff │ │ ├── js │ │ │ └── sunburst.js │ │ └── logo-no-text-150dpi.png │ ├── templates │ │ ├── base.html │ │ ├── graph.html │ │ ├── resource.html │ │ └── sparql.html │ └── views.py ├── gunicorn_config.py └── run.py └── update.sh /.dockerignore: -------------------------------------------------------------------------------- 1 | .git 2 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | lib/ 2 | bin/ 3 | include/ 4 | .Python 5 | pip-selfcheck.json 6 | *.pyc 7 | src/app/config.py 8 | .DS_Store 9 | -------------------------------------------------------------------------------- /Dockerfile: -------------------------------------------------------------------------------- 1 | FROM python:2.7.13-wheezy 2 | 3 | MAINTAINER laurens.rietveld@vu.nl 4 | 5 | ENV BRWSR_APP="/usr/local/brwsr" 6 | 7 | COPY ./requirements.txt /requirements.txt 8 | RUN pip install -r requirements.txt 9 | 10 | COPY ./src ${BRWSR_APP} 11 | ENV CONFIG_FILE=${BRWSR_APP}/app/config.py 12 | RUN cp ${BRWSR_APP}/app/config-template.py ${CONFIG_FILE} 13 | 14 | 15 | COPY entrypoint.sh /sbin/entrypoint.sh 16 | RUN chmod 755 /sbin/entrypoint.sh 17 | 18 | WORKDIR ${BRWSR_APP} 19 | ENTRYPOINT ["/sbin/entrypoint.sh"] 20 | CMD ["app:start"] 21 | EXPOSE 5000 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | brwsr 2 | ===== 3 | 4 | **Lightweight Linked Data Browser** 5 | 6 | ### Features 7 | * Browse all resources starting with a designated URI prefix in a SPARQL endpoint 8 | * Implements content negotiation to serve representations of these resources both as HTML, and as RDF/XML, Turtle and JSON-LD 9 | * HTML page asynchronously calls service to retrieve preferred labels for all resources. 10 | 11 | In short, it is a very lightweight Python-based alternative to Pubby (with a slightly more attractive interface design) 12 | 13 | 14 | 15 | ### Docker-based 16 | 17 | #### Prerequisites 18 | * Make sure you have [Docker](https://www.docker.com) and [docker-compose](https://docs.docker.com/compose/install/) installed for your platform 19 | 20 | #### Using 21 | * To use the base setup, copy this docker-compose file to a directory: . 22 | * Run `docker-compose up`. 23 | * This will run brwsr at with the DBPedia SPARQL endpoint. 24 | * To modify the configuration, change the environment variables in the `docker-compose.yml` file. 25 | * For configuration parameters, see below. 26 | 27 | ### As a Flask application 28 | 29 | #### Installation 30 | * Open up a terminal 31 | * In a directory of your choice, run `git clone https://github.com/Data2Semantics/brwsr.git` 32 | * (optional: setup a virtualenv and activate it) 33 | * run `pip install -r requirements.txt` 34 | * Rename `config-template.py` to `config.py` 35 | * Make the appropriate settings in the file (see below) 36 | 37 | #### Using 38 | 39 | * Start it with `python run.py` if you're playing around, otherwise 40 | * Adjust the `gunicorn_config.py` for your system, and start brwsr with `gunicorn -c gunicorn_config.py app:app` to run in daemon mode on port 5000 (behind e.g. an Apache or Nginx proxy) 41 | 42 | 43 | ### Configuration Parameters 44 | 45 | 46 | | Parameter | Allowed Values | Default | Description | `docker-compose.yml` or `config.py` | 47 | | :--------- | :-------------- | :------- | :----------- | :----------------------- | 48 | | `LOCAL_STORE` | `True`, `False` | `False` | Set `LOCAL_STORE` to `True` if you want brwsr to just load a (smallish) RDF file into server memory rather than operate on an external SPARQL store | both | 49 | | `LOCAL_FILE` | File path | `None` | Set `LOCAL_FILE` to the relative or absolute path of the file you want brwsr to load when `LOCAL_STORE` is True. The brwsr application will just use RDFLib to guess the file format based on the extension. You can use UNIX file masks such as * and ? to load multiple files. When using Docker, make sure the files are on a filesystem that is accessible to Docker | both | 50 | | `SPARQL_ENDPOINT` | URL | `http://your.sparql.endpoint.here/sparql` | Set this to the SPARQL endpoint uri of your triplestore e.g. `http://dbpedia.org/sparql` | both | 51 | | `SPARQL_ENDPOINT_MAPPING` | Dictionary | `None` | If brwsr is backed by multiple separate triple stores, use `SPARQL_ENDPOINT_MAPPING` to make sure that each URI for which the `LOCAL_NAME` (i.e. the URI with the `DEFAULT_BASE` removed, if present) starts with a key of the `SPARQL_ENDPOINT_MAPPING` dictionary, the proper SPARQL endpoint is used. You can also use Python-style regular expressions in the prefix description (the keys of this dictionary) Note that brwsr will allways *also* query the default `SPARQL_ENDPOINT` (see example in `config-template.py`). | only `config.py` | 52 | | `DRUID_STATEMENTS_URL` | URL | `None` | Set this to the statements URL of a Druid instance (), e.g. <"http://druid.instance.url/_api/datasets/Username/Dataset/statements.triply"> | both | 53 | | `LDF_STATEMENTS_URL` | URL | `None` | Set this to the statements URL of a Linked Data Fragments service (), e.g. | both | 54 | | `DEFAULT_BASE` | URI | `http://your.base.uri.here` | The DEFAULT_BASE is the prefix of the URI's in the triple store that can be browsed by brwsr. Requests to brwsr only include the local name (i.e. the the part after the third slash '/'), the `DEFAULT_BASE` is *always* prepended to this local name to make up the URI that's used to query the triple store: e.g. `http://dbpedia.org` (without the last slash!)| both | 55 | | `LOCAL_DOCUMENT_INFIX` | String | `doc` | The `LOCAL_DOCUMENT_INFIX` is the infix used between the `DEFAULT_BASE` and the local name of the URI to denote the HTML representation of the RDF resource (see the Cool URI's specification) | both | 56 | | `LOCAL_SERVER_NAME` | URI | `http://your.server.name.here` | The LOCAL_SERVER_NAME is the address brwsr listens to. It needs to know this to build proper requests when you click a URI in the brwsr page of a resource: e.g. "http://localhost:5000" if running Flask. | both | 57 | | `BEHIND_PROXY` | `True`, `False` | `False` | By default brwsr assumes it is running at the root of the server. If you want to run brwsr under a directory (e.g. http://example.com/brwsr rather than http://example.com), you need to do this via a reverse proxy, and tell brwsr about it (set `BEHIND_PROXY` to True, and configure the proxy, see below) | both | 58 | | `START_LOCAL_NAME` | String | `some/local/name` | The START_LOCAL_NAME is the local name of the first URI shown in brwsr if no URI is specified, e.g. "`resource/Amsterdam`" when using the DBPedia settings | both | 59 | | `START_URI` | URI | `DEFAULT_BASE + "/" + START_LOCAL_NAME` | The `START_URI` is the URI that is shown when browsing to the `SERVER_NAME` URL. It is simply the combination of the `DEFAULT_BASE` and the `START_LOCAL_NAME` (i.e. there is no need to change this, usually) e.g. this will become "`http://dbpedia.org/resource/Amsterdam`" | both | 60 | | `QUERY_RESULTS_LIMIT` | Integer | `5000` | Set query results limit because otherwise your browser might crash. | both | 61 | | `BROWSE_EXTERNAL_URIS` | `True`, `False` | `True` | Browse URIs that do not match the `DEFAULT_BASE`. This allows for browsing resources from different namespaces within the same endpoint store. | both | 62 | | `DEREFERENCE_EXTERNAL_URIS` | `True`, `False` | `False` | Dereference external URIs (i.e. retrieve RDF served at that location, and display the resource). This may be slow, depending on the responsiveness of the server at hand. Also, the resulting RDF is stored locally (in memory) which means that this is a potential memory hog for servers that are visited frequently. | both | 63 | | `PORT` | Integer | `5000` | The port via which to run brwsr | both (but needs care when using Docker) | 64 | | `DEBUG` | `True`, `False` | `False` | Switch on debug logging | both | 65 | | `SPARQL_METHOD` | `GET`, `POST` | `GET` | Set the HTTP method to use for communicating with SPARQL endpoint. | both 66 | | `CUSTOM_PARAMETERS` | Dictionary | `{'reasoning': True}` | Set any custom parameters to be sent to the SPARQL endpoint, e.g. `CUSTOM_PARAMETERS = {'reasoning': 'true'}` for Stardog | only `config.py` | 67 | | `CACHE_TIMEOUT` | Integer | `300` | Set the cache timeout in seconds | both | 68 | | `SUNBURST_DEPTH`| Integer | `1` | Depth of the Sunburst visualization (default = 1). **Warning**: setting this to a value > 1 will really make the visualization a *lot* slower, also depending on the number of endpoints, or services you are calling.| both | 69 | 70 | 71 | 72 | #### Example Nginx Reverse Proxy configuration for use with Gunicorn: 73 | 74 | ``` 75 | server { 76 | listen 80; 77 | server_name your.server.name.here; 78 | access_log /var/log/nginx/brwsr_access.log; 79 | error_log /var/log/nginx/brswr_error.log; 80 | 81 | 82 | location /socket.io { 83 | proxy_pass http://127.0.0.1:5000/socket.io; 84 | proxy_redirect off; 85 | proxy_buffering off; 86 | 87 | proxy_set_header Host $host; 88 | proxy_set_header X-Real-IP $remote_addr; 89 | proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; 90 | 91 | proxy_http_version 1.1; 92 | proxy_set_header Upgrade $http_upgrade; 93 | proxy_set_header Connection "Upgrade"; 94 | } 95 | 96 | location / { 97 | add_header Access-Control-Allow-Origin *; 98 | 99 | proxy_pass http://127.0.0.1:5000; 100 | proxy_redirect off; 101 | 102 | proxy_set_header Host $host; 103 | proxy_set_header X-Real-IP $remote_addr; 104 | proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; 105 | } 106 | } 107 | 108 | ``` 109 | 110 | #### Running `brwsr` at a relative location other than '/' 111 | If for some reason you want or need to run brwsr at a relative path other than '/', you should do two things: 112 | 113 | 1. Set the `BEHIND_PROXY` parameter in your `config.py` or `docker-compose.yaml` to `True` 114 | 2. Setup a Nginx proxy along the lines of the below example (adapted from ): 115 | 116 | ``` 117 | location /myprefix { 118 | proxy_pass http://localhost:5000; 119 | proxy_set_header Host $host; 120 | proxy_set_header Upgrade $http_upgrade; 121 | proxy_set_header Connection "upgrade"; 122 | proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; 123 | proxy_set_header X-Scheme $scheme; 124 | proxy_set_header X-Script-Name /myprefix; 125 | } 126 | ``` 127 | 128 | Where `myprefix` should be set to the location you want to be running brwsr under 129 | 130 | The `proxy_pass` setting should point to the address and port you are running brwsr at (default is localhost port 5000). 131 | 132 | 133 | ### Acknowledgements 134 | This work was funded by the Dutch national programme COMMIT/ and the NWO-funded CLARIAH project. 135 | -------------------------------------------------------------------------------- /brwsr-stardog/Dockerfile: -------------------------------------------------------------------------------- 1 | FROM java:8-jre-alpine 2 | 3 | ENV STARDOG_VERSION 4.1.2 4 | 5 | ENV STARDOG_HOME /data 6 | ENV STARDOG_INSTALL_DIR /opt/stardog 7 | 8 | ENV STARDOG_START_PARAMS "" 9 | ENV STARDOG_DB_NAME stardog 10 | ENV STARDOG_CREATE_PARAMS "-n ${STARDOG_DB_NAME} -v -o versioning.enabled=true preserve.bnode.ids=false strict.parsing=false --" 11 | ENV STARDOG_JAVA_ARGS "-Xms2g -Xmx2g -XX:MaxDirectMemorySize=2g" 12 | ENV PATH ${STARDOG_INSTALL_DIR}/bin:${PATH} 13 | 14 | RUN mkdir -p ${STARDOG_HOME} 15 | RUN mkdir -p ${STARDOG_INSTALL_DIR} 16 | 17 | WORKDIR ${STARDOG_INSTALL_DIR} 18 | 19 | RUN apk update 20 | RUN apk add unzip bash 21 | 22 | ADD stardog-*.zip /tmp 23 | RUN unzip -d /tmp /tmp/stardog-*.zip 24 | RUN rm -f /tmp/stardog-*.zip 25 | RUN cp -r /tmp/stardog-*/* ${STARDOG_INSTALL_DIR}/ 26 | RUN rm -rf /tmp/stardog-* 27 | 28 | ADD stardog-license-key.bin ${STARDOG_INSTALL_DIR} 29 | ADD stardog.properties ${STARDOG_INSTALL_DIR} 30 | ADD start-stardog-service.sh ${STARDOG_INSTALL_DIR} 31 | RUN chmod +x ${STARDOG_INSTALL_DIR}/start-stardog-service.sh 32 | 33 | WORKDIR ${STARDOG_HOME} 34 | 35 | CMD ${STARDOG_INSTALL_DIR}/start-stardog-service.sh 36 | 37 | EXPOSE 5820 38 | -------------------------------------------------------------------------------- /brwsr-stardog/data/placeholder: -------------------------------------------------------------------------------- 1 | Put your turtle/trig/nquads/ntriples/rdfxml files here 2 | -------------------------------------------------------------------------------- /brwsr-stardog/docker-compose.yml: -------------------------------------------------------------------------------- 1 | version: '3' 2 | services: 3 | brwsr: 4 | image: clariah/brwsr:latest 5 | ports: 6 | - "127.0.0.1:5000:5000" 7 | environment: 8 | - SPARQL_ENDPOINT=http://stardog:5820/stardog/query 9 | - LOCAL_SERVER_NAME=http://localhost:5000 10 | - DEFAULT_BASE=http://dbpedia.org 11 | - START_LOCAL_NAME=resource/Amsterdam 12 | stardog: 13 | build: 14 | context: . 15 | dockerfile: Dockerfile 16 | environment: 17 | - STARDOG_START_PARAMS=--disable-security 18 | volumes: 19 | - ./data/:/var/data 20 | ports: 21 | - "127.0.0.1:5820:5820" 22 | -------------------------------------------------------------------------------- /brwsr-stardog/readme.md: -------------------------------------------------------------------------------- 1 | # brwsr-stardog Docker container 2 | 3 | This will allow you to start a docker container where Stardog and brwsr work together (no need for a separate Stardog instance). 4 | 5 | How to use this: 6 | 7 | * Add the data you want to host to the `data/` folder (this will be a shared volume with the Stardog docker container) 8 | * Download the latest version of stardog and put the zipfile (e.g. `stardog-5.0-beta.zip`) and the `stardog-license-key.bin` in this directory. 9 | * Make necessary changes to the brwsr configuration in `docker-compose.yml`. 10 | * You are likely to want to change the `DEFAULT_BASE` and `START_LOCAL_NAME` parameters to suit your data. 11 | * Make necessary changes to the `stardog.properties` file to suit your need. 12 | * By default the `start-stardog-service.sh` script creates a `stardog` database and fills it with all data in the `data/` folder. 13 | * Run `docker-compose build` to build the `stardog` and `brwsr` images (if needed) 14 | * Run `docker-compose up` to start 15 | 16 | Stardog will be running at and brwser will be at . 17 | 18 | ### Acknowledgements 19 | 20 | The stardog Dockerfile and startup scripts were shamelessly copied and adapted from Rene Pietszch . 21 | -------------------------------------------------------------------------------- /brwsr-stardog/stardog.properties: -------------------------------------------------------------------------------- 1 | query.all.graphs = true 2 | database.archetypes=PROV 3 | reasoning.punning.enabled=true 4 | reasoning.type=SL 5 | reasoning.sameas=FULL 6 | search.enabled=true 7 | search.wildcard.search.enabled=true 8 | query.all.graphs=true 9 | strict.parsing=false 10 | -------------------------------------------------------------------------------- /brwsr-stardog/start-stardog-service.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | rm -f ${STARDOG_HOME}/*.lock 4 | rm -f ${STARDOG_HOME}/*.log 5 | 6 | # copy license from install dir if not already in STARDOG_HOME 7 | if [ ! -f ${STARDOG_HOME}/stardog-license-key.bin ]; then 8 | cp ${STARDOG_INSTALL_DIR}/stardog-license-key.bin ${STARDOG_HOME}/stardog-license-key.bin 9 | fi 10 | 11 | # copy server properties from install dir if not already in STARDOG_HOME 12 | if [ ! -f ${STARDOG_HOME}/stardog.properties ]; then 13 | cp ${STARDOG_INSTALL_DIR}/stardog.properties ${STARDOG_HOME}/stardog.properties 14 | fi 15 | 16 | echo "starting stardog with the following environment:" 17 | echo "STARDOG_START_PARAMS: ${STARDOG_START_PARAMS}" 18 | echo "STARDOG_CREATE_PARAMS: ${STARDOG_CREATE_PARAMS}" 19 | echo "STARDOG_DB_NAME: ${STARDOG_DB_NAME}" 20 | 21 | ${STARDOG_INSTALL_DIR}/bin/stardog-admin server start ${STARDOG_START_PARAMS} 22 | ${STARDOG_INSTALL_DIR}/bin/stardog-admin db create ${STARDOG_CREATE_PARAMS} 23 | ${STARDOG_INSTALL_DIR}/bin/stardog data add ${STARDOG_DB_NAME} /var/data/*.nq 24 | ${STARDOG_INSTALL_DIR}/bin/stardog data add ${STARDOG_DB_NAME} /var/data/*.ttl 25 | ${STARDOG_INSTALL_DIR}/bin/stardog data add ${STARDOG_DB_NAME} /var/data/*.rdf 26 | ${STARDOG_INSTALL_DIR}/bin/stardog data add ${STARDOG_DB_NAME} /var/data/*.trig 27 | ${STARDOG_INSTALL_DIR}/bin/stardog data add ${STARDOG_DB_NAME} /var/data/*.nt 28 | 29 | ${STARDOG_INSTALL_DIR}/bin/stardog-admin server stop 30 | ${STARDOG_INSTALL_DIR}/bin/stardog-admin server start --foreground --disable-security ${STARDOG_START_PARAMS} 31 | -------------------------------------------------------------------------------- /docker-compose.yml: -------------------------------------------------------------------------------- 1 | version: '3' 2 | services: 3 | qber: 4 | image: clariah/brwsr:latest 5 | ports: 6 | - "127.0.0.1:5000:5000" 7 | environment: 8 | - SPARQL_ENDPOINT=http://dbpedia.org/sparql 9 | - DEFAULT_BASE=http://dbpedia.org 10 | - LOCAL_SERVER_NAME=http://localhost:5000 11 | - START_LOCAL_NAME=resource/Amsterdam 12 | -------------------------------------------------------------------------------- /entrypoint.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | set -e #exit on errors 3 | 4 | #print statements as they are executed 5 | [[ -n $DEBUG_ENTRYPOINT ]] && set -x 6 | 7 | case ${1} in 8 | app:start) 9 | python run.py 10 | ;; 11 | app:help) 12 | echo "Available options:" 13 | echo " app:start - Starts brwsr (default)" 14 | echo " [command] - Execute the specified command, eg. /bin/bash." 15 | ;; 16 | *) 17 | exec "$@" 18 | ;; 19 | esac 20 | 21 | exit 0 22 | -------------------------------------------------------------------------------- /load.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | 4 | echo "load.sh" 5 | echo "Load files into brwsr from the command line" 6 | echo "" 7 | echo "Usage: load.sh '../path/to/files/*.nq'" 8 | echo " make sure that the path to files is within single quotes ('') and relative to the 'src' directory" 9 | echo "" 10 | 11 | export LOCAL_STORE=True 12 | export LOCAL_FILE=$@ 13 | 14 | cd src 15 | python run.py 16 | -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | Flask==0.10.1 2 | Flask-Bootstrap==3.3.0.1 3 | Flask-Cache==0.13.1 4 | gevent==1.0.1 5 | gevent-socketio==0.3.6 6 | gevent-websocket==0.9.3 7 | greenlet==0.4.5 8 | gunicorn==19.1.1 9 | html5lib==0.999999999 10 | isodate==0.5.4 11 | itsdangerous==0.24 12 | Jinja2==2.7.3 13 | keepalive==0.5 14 | MarkupSafe==0.23 15 | pyparsing==2.1.9 16 | rdfextras==0.4 17 | rdflib==4.2.1 18 | rdflib-jsonld==0.2 19 | requests==2.4.3 20 | six==1.10.0 21 | SPARQLWrapper==1.7.6 22 | webencodings==0.5 23 | Werkzeug==0.9.6 24 | -------------------------------------------------------------------------------- /src/app/__init__.py: -------------------------------------------------------------------------------- 1 | from flask import Flask 2 | from flask.ext.cache import Cache 3 | from flask_bootstrap import Bootstrap 4 | import os 5 | import config, client 6 | 7 | from datetime import datetime 8 | print "init", datetime.now().isoformat() 9 | 10 | class ReverseProxied(object): 11 | '''Wrap the application in this middleware and configure the 12 | front-end server to add these headers, to let you quietly bind 13 | this to a URL other than / and to an HTTP scheme that is 14 | different than what is used locally. 15 | 16 | In nginx: 17 | location /myprefix { 18 | proxy_pass http://192.168.0.1:5001; 19 | proxy_set_header Host $host; 20 | proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; 21 | proxy_set_header X-Scheme $scheme; 22 | proxy_set_header X-Script-Name /myprefix; 23 | } 24 | 25 | :param app: the WSGI application 26 | ''' 27 | def __init__(self, app): 28 | self.app = app 29 | 30 | def __call__(self, environ, start_response): 31 | script_name = environ.get('HTTP_X_SCRIPT_NAME', '') 32 | if script_name: 33 | environ['SCRIPT_NAME'] = script_name 34 | path_info = environ['PATH_INFO'] 35 | if path_info.startswith(script_name): 36 | environ['PATH_INFO'] = path_info[len(script_name):] 37 | 38 | scheme = environ.get('HTTP_X_SCHEME', '') 39 | if scheme: 40 | environ['wsgi.url_scheme'] = scheme 41 | return self.app(environ, start_response) 42 | 43 | 44 | app = Flask(__name__) 45 | Bootstrap(app) 46 | cache = Cache(app,config={'CACHE_TYPE': 'simple'}) 47 | 48 | # Do something special if the application root is other than the default ("/") 49 | try: 50 | if config.BEHIND_PROXY: 51 | app.wsgi_app = ReverseProxied(app.wsgi_app) 52 | except: 53 | print "WARNING: You are using an older version of the configuration file that does not set the 'BEHIND_PROXY' parameter, have a look at 'config-template.py' to see what to add." 54 | 55 | app.debug = True 56 | if (hasattr(config, 'DEBUG')): 57 | app.debug = config.DEBUG 58 | 59 | import views 60 | 61 | if not app.debug or os.environ.get("WERKZEUG_RUN_MAIN") == "true": 62 | # The app is not in debug mode or we are in the reloaded process 63 | client.init() 64 | 65 | def run(): 66 | app.run(host="0.0.0.0", port=(config.PORT if hasattr(config, 'PORT') else 5000)) 67 | -------------------------------------------------------------------------------- /src/app/client.py: -------------------------------------------------------------------------------- 1 | from SPARQLWrapper import SPARQLWrapper, JSON, XML 2 | from threading import Thread 3 | import logging 4 | import requests 5 | from config import * 6 | import rdfextras 7 | import traceback 8 | import glob 9 | import re 10 | rdfextras.registerplugins() 11 | import rdflib 12 | from rdflib import Dataset, URIRef, RDFS 13 | 14 | # from app import app 15 | 16 | from datetime import datetime 17 | print "client", datetime.now().isoformat() 18 | 19 | # log = app.logger 20 | log = logging.getLogger(__name__) 21 | log.setLevel(logging.DEBUG) 22 | 23 | # For backwards compatibility: some configurations do not specify the 24 | # SPARQL_METHOD parameter 25 | try: 26 | if SPARQL_METHOD is not None: 27 | pass 28 | except: 29 | SPARQL_METHOD = 'GET' 30 | 31 | 32 | # Properties that are typically used to give the label for a resource 33 | # Used to render the sunburst 34 | label_properties = ['http://www.w3.org/2004/02/skos/core#prefLabel', 35 | 'http://www.w3.org/2004/02/skos/core#altLabel', 36 | str(RDFS['label']), 37 | 'http://xmlns.com/foaf/0.1/name', 38 | 'http://purl.org/dc/elements/1.1/title', 39 | 'http://purl.org/dc/terms/title'] 40 | 41 | 42 | g = Dataset() 43 | 44 | 45 | def load_file(filename): 46 | log.info("Loading {}...".format(filename)) 47 | format = rdflib.util.guess_format(filename) 48 | g.load(filename, format=format) 49 | log.info("... done loading {}".format(filename)) 50 | 51 | 52 | def load_data(data, format="json-ld"): 53 | g.parse(data=data, format=format) 54 | 55 | 56 | def init(): 57 | if LOCAL_STORE: 58 | log.info("Loading local file(s): {}".format(LOCAL_FILE)) 59 | try: 60 | files = glob.glob(LOCAL_FILE) 61 | if len(files) == 0: 62 | log.error( 63 | "No files match the UNIX file pattern specified in {}, or the pattern is invalid.".format(LOCAL_FILE)) 64 | return 65 | for filename in files: 66 | log.info("Trying to load file: {}".format(filename)) 67 | t = Thread(target=load_file, args=(filename,)) 68 | t.start() 69 | except: 70 | log.error(traceback.format_exc()) 71 | raise Exception( 72 | "Cannot guess file format for {} or could not load file".format(LOCAL_FILE)) 73 | 74 | 75 | def get_predicates(sparqls, url): 76 | predicate_query = u""" 77 | SELECT DISTINCT ?p WHERE {{ 78 | {{ <{url}> ?p [] . }} 79 | UNION 80 | {{ [] ?p <{url}> . }} 81 | }} 82 | """.format(url=url) 83 | 84 | predicates = [] 85 | for s in sparqls: 86 | s.setQuery(predicate_query) 87 | log.debug(predicate_query) 88 | 89 | try: 90 | sparql_results = list(s.query().convert()["results"]["bindings"]) 91 | 92 | predicates.extend([r['p']['value'] for r in sparql_results]) 93 | 94 | # log.debug(predicates) 95 | except: 96 | log.warning( 97 | "Could not determine related predicates, because there were no triples where {} occurs als subject or object".format(url)) 98 | 99 | return predicates 100 | 101 | 102 | def visit(url, format='html', external=False, depth=1): 103 | log.debug("Starting query") 104 | 105 | # If this uri is not in our namespace, and DEREFERENCE_EXTERNAL_URIS is true 106 | # We go out, and add the retrieved RDF to our local store 107 | if external and DEREFERENCE_EXTERNAL_URIS: 108 | log.debug("Dereferencing external uri") 109 | dereference(url) 110 | 111 | if LOCAL_STORE: 112 | return visit_local(url, format=format, depth=depth) 113 | else: 114 | return visit_sparql(url, format=format, depth=depth) 115 | 116 | 117 | def get_sparql_endpoints(url): 118 | sparql = None 119 | sparqls = [] 120 | 121 | for prefix, endpoint in SPARQL_ENDPOINT_MAPPING.items(): 122 | # If the URL starts with the default base + the prefix, or 123 | # If the URL starts with the prefix itself, or 124 | # If the URL matches the regular expression in the prefix, 125 | # AND if the endpoint is not already in the list... 126 | if (url.startswith(DEFAULT_BASE + prefix + '/') or url.startswith(prefix) or re.match(prefix, url) is not None) and endpoint not in sparqls: 127 | s = SPARQLWrapper(endpoint) 128 | sparqls.append(s) 129 | 130 | # Also add the default endpoint 131 | if SPARQL_ENDPOINT is not None: 132 | sparql = SPARQLWrapper(SPARQL_ENDPOINT) 133 | sparqls.append(sparql) 134 | 135 | log.debug("Will be using the following endpoints: {}".format( 136 | [s.endpoint for s in sparqls])) 137 | 138 | for s in sparqls: 139 | s.setReturnFormat(JSON) 140 | s.setMethod(SPARQL_METHOD) 141 | for key, value in CUSTOM_PARAMETERS.items(): 142 | sparql.addParameter(key, value) 143 | 144 | return sparqls 145 | 146 | 147 | def druid_to_sparql_results(druid_results): 148 | sparql_results = [] 149 | for triple in druid_results: 150 | print triple 151 | ds, dp, do = tuple(triple) 152 | 153 | if ds['termType'] == u'NamedNode': 154 | s = {'value': ds['value'], 'type': 'uri'} 155 | elif ds['termType'] == u'BlankNode': 156 | s = {'value': ds['value'], 'type': 'bnode'} 157 | 158 | p = {'value': dp['value'], 'type': 'uri'} 159 | 160 | if do['termType'] == u'NamedNode': 161 | o = {'value': do['value'], 'type': 'uri'} 162 | elif do['termType'] == u'BlankNode': 163 | o = {'value': do['value'], 'type': 'bnode'} 164 | elif do['termType'] == u'Literal': 165 | o = {'value': do['value'], 'type': 'literal'} 166 | if 'datatype' in do: 167 | o['datatype'] = do['datatype'] 168 | if 'language' in do: 169 | o['lang'] = do['language'] 170 | 171 | sparql_results.append({'s': s, 'p': p, 'o': o}) 172 | 173 | return sparql_results 174 | 175 | 176 | def visit_druid(url, format='html'): 177 | log.debug("Visiting druid at {}".format(DRUID_STATEMENTS_URL)) 178 | po = requests.get(DRUID_STATEMENTS_URL, headers={ 179 | 'Accept': 'text/json'}, params={'subject': url}).json() 180 | sp = requests.get(DRUID_STATEMENTS_URL, headers={ 181 | 'Accept': 'text/json'}, params={'object': url}).json() 182 | so = requests.get(DRUID_STATEMENTS_URL, headers={ 183 | 'Accept': 'text/json'}, params={'predicate': url}).json() 184 | 185 | druid_results = [] 186 | druid_results.extend(po) 187 | druid_results.extend(sp) 188 | druid_results.extend(so) 189 | 190 | sparql_results = druid_to_sparql_results(druid_results) 191 | 192 | if format == 'html': 193 | return sparql_results 194 | else: 195 | return 'Not supported' 196 | 197 | 198 | def retrieve_ldf_results(url): 199 | log.debug("Visiting Linked Data Fragments at {}".format(LDF_STATEMENTS_URL)) 200 | po = requests.get(LDF_STATEMENTS_URL, headers={ 201 | 'Accept': 'application/json'}, params={'subject': url}).content 202 | sp = requests.get(LDF_STATEMENTS_URL, headers={ 203 | 'Accept': 'application/json'}, params={'object': url}).content 204 | so = requests.get(LDF_STATEMENTS_URL, headers={ 205 | 'Accept': 'application/json'}, params={'predicate': url}).content 206 | 207 | load_data(po) 208 | load_data(sp) 209 | load_data(so) 210 | 211 | 212 | def visit_sparql(url, format='html', depth=1): 213 | sparqls = get_sparql_endpoints(url) 214 | predicates = get_predicates(sparqls, url) 215 | 216 | if format == 'html': 217 | limit_fraction = QUERY_RESULTS_LIMIT / 3 218 | if len(predicates) > 1: 219 | predicate_query_limit_fraction = ( 220 | limit_fraction * 2) / len(predicates) 221 | else: 222 | predicate_query_limit_fraction = limit_fraction * 2 223 | 224 | results = [] 225 | 226 | def predicate_specific_sparql(sparql, query): 227 | log.debug(query) 228 | 229 | sparql.setQuery(query) 230 | res = sparql.query().convert() 231 | results.extend( 232 | list(res["results"]["bindings"])) 233 | 234 | threads = [] 235 | local_results = [] 236 | for p in predicates: 237 | q = u"""SELECT DISTINCT ?s ?p ?o ?g WHERE {{ 238 | {{ 239 | GRAPH ?g {{ 240 | {{ 241 | <{url}> <{predicate}> ?o . 242 | BIND(<{url}> as ?s) 243 | BIND(<{predicate}> as ?p) 244 | }} UNION {{ 245 | ?s <{predicate}> <{url}>. 246 | BIND(<{url}> as ?o) 247 | BIND(<{predicate}> as ?p) 248 | }} 249 | }} 250 | }} UNION {{ 251 | {{ 252 | <{url}> <{predicate}> ?o . 253 | BIND(<{url}> as ?s) 254 | BIND(<{predicate}> as ?p) 255 | }} UNION {{ 256 | ?s <{predicate}> <{url}>. 257 | BIND(<{url}> as ?o) 258 | BIND(<{predicate}> as ?p) 259 | }} 260 | }} 261 | }} LIMIT {limit}""".format(url=url, predicate=p, limit=predicate_query_limit_fraction) 262 | 263 | for s in sparqls: 264 | # Start processes for each endpoint, for each predicate query 265 | process = Thread(target=predicate_specific_sparql, args=[s, q]) 266 | process.start() 267 | threads.append(process) 268 | 269 | url_is_predicate_query = u"""SELECT DISTINCT ?s ?p ?o ?g WHERE {{ 270 | {{ 271 | GRAPH ?g {{ 272 | ?s <{url}> ?o. 273 | BIND(<{url}> as ?p) 274 | }} 275 | }} UNION {{ 276 | ?s <{url}> ?o. 277 | BIND(<{url}> as ?p) 278 | }} 279 | }} LIMIT {limit}""".format(url=url, limit=limit_fraction) 280 | 281 | for s in sparqls: 282 | process = Thread(target=predicate_specific_sparql, 283 | args=[s, url_is_predicate_query]) 284 | process.start() 285 | threads.append(process) 286 | 287 | # We now pause execution on the main thread by 'joining' all of our started threads. 288 | # This ensures that each has finished processing the urls. 289 | for process in threads: 290 | process.join() 291 | 292 | if LDF_STATEMENTS_URL is not None: 293 | retrieve_ldf_results(url) 294 | 295 | # We also add local results (result of dereferencing) 296 | local_results = list(visit_local(url, format)) 297 | 298 | results.extend(local_results) 299 | 300 | # If a Druid statements URL is specified, we'll try to receive it as 301 | # well 302 | if DRUID_STATEMENTS_URL is not None: 303 | results.extend(visit_druid(url, format)) 304 | 305 | if depth > 1: 306 | # If depth is larger than 1, we proceed to extend the results with the results of 307 | # visiting all object resources for every triple in the resultset. 308 | newresults = [] 309 | 310 | objects = set([r['o']['value'] for r in results if r['o']['value'] != url and r['o']['type']=='uri']) 311 | 312 | for o in objects: 313 | newresults.extend( 314 | visit(o, format=format, depth=depth - 1)) 315 | 316 | results.extend(newresults) 317 | 318 | else: 319 | q = u""" 320 | CONSTRUCT {{ 321 | ?s ?p ?o . 322 | }} WHERE {{ 323 | {{ 324 | GRAPH ?g {{ 325 | {{ 326 | <{url}> ?p ?o . 327 | BIND(<{url}> as ?s) 328 | }} UNION {{ 329 | ?s ?p <{url}>. 330 | BIND(<{url}> as ?o) 331 | }} UNION {{ 332 | ?s <{url}> ?o. 333 | BIND(<{url}> as ?p) 334 | }} 335 | }} 336 | }} UNION {{ 337 | {{ 338 | <{url}> ?p ?o . 339 | BIND(<{url}> as ?s) 340 | }} UNION {{ 341 | ?s ?p <{url}>. 342 | BIND(<{url}> as ?o) 343 | }} UNION {{ 344 | ?s <{url}> ?o. 345 | BIND(<{url}> as ?p) 346 | }} 347 | }} 348 | }} LIMIT {limit}""".format(url=url, limit=QUERY_RESULTS_LIMIT) 349 | 350 | result_dataset = Dataset() 351 | 352 | for s in sparqls: 353 | s.setQuery(q) 354 | s.setReturnFormat(XML) 355 | 356 | result_dataset += s.query().convert() 357 | 358 | if format == 'jsonld': 359 | results = result_dataset.serialize(format='json-ld') 360 | elif format == 'rdfxml': 361 | s.setReturnFormat(XML) 362 | results = result_dataset.serialize(format='pretty-xml') 363 | elif format == 'turtle': 364 | s.setReturnFormat(XML) 365 | results = result_dataset.serialize(format='turtle') 366 | else: 367 | results = 'Nothing' 368 | 369 | log.debug("Received results") 370 | 371 | return results 372 | 373 | 374 | def visit_local(url, format='html', depth=1): 375 | if format == 'html': 376 | q = u"""SELECT DISTINCT ?s ?p ?o ?g WHERE {{ 377 | {{ 378 | GRAPH ?g {{ 379 | {{ 380 | <{url}> ?p ?o . 381 | BIND(<{url}> as ?s) 382 | }} UNION {{ 383 | ?s ?p <{url}>. 384 | BIND(<{url}> as ?o) 385 | }} UNION {{ 386 | ?s <{url}> ?o. 387 | BIND(<{url}> as ?p) 388 | }} 389 | }} 390 | }} UNION {{ 391 | {{ 392 | <{url}> ?p ?o . 393 | BIND(<{url}> as ?s) 394 | }} UNION {{ 395 | ?s ?p <{url}>. 396 | BIND(<{url}> as ?o) 397 | }} UNION {{ 398 | ?s <{url}> ?o. 399 | BIND(<{url}> as ?p) 400 | }} 401 | }} 402 | }} LIMIT {limit} """.format(url=url, limit=QUERY_RESULTS_LIMIT) 403 | 404 | results = g.query(q) 405 | else: 406 | # q = u"DESCRIBE <{}>".format(url) 407 | 408 | q = u""" 409 | CONSTRUCT {{ 410 | ?s ?p ?o . 411 | }} WHERE {{ 412 | {{ 413 | GRAPH ?g {{ 414 | {{ 415 | <{url}> ?p ?o . 416 | BIND(<{url}> as ?s) 417 | }} UNION {{ 418 | ?s ?p <{url}>. 419 | BIND(<{url}> as ?o) 420 | }} UNION {{ 421 | ?s <{url}> ?o. 422 | BIND(<{url}> as ?p) 423 | }} 424 | }} 425 | }} UNION {{ 426 | {{ 427 | <{url}> ?p ?o . 428 | BIND(<{url}> as ?s) 429 | }} UNION {{ 430 | ?s ?p <{url}>. 431 | BIND(<{url}> as ?o) 432 | }} UNION {{ 433 | ?s <{url}> ?o. 434 | BIND(<{url}> as ?p) 435 | }} 436 | }} 437 | }} LIMIT {limit}""".format(url=url, limit=QUERY_RESULTS_LIMIT) 438 | 439 | if format == 'jsonld': 440 | results = g.query(q).serialize(format='json-ld') 441 | elif format == 'rdfxml': 442 | results = g.query(q).serialize(format='pretty-xml') 443 | elif format == 'turtle': 444 | results = g.query(q).serialize(format='turtle') 445 | else: 446 | results = 'Nothing' 447 | 448 | log.debug("Received results") 449 | 450 | return results 451 | 452 | 453 | def dereference(uri): 454 | uriref = URIRef(uri) 455 | 456 | if uriref not in g.graphs(): 457 | resource_graph = g.graph(uriref) 458 | 459 | headers = { 460 | 'Accept': 'text/turtle, application/x-turtle, application/rdf+xml, text/trig'} 461 | 462 | try: 463 | response = requests.get(uri, headers=headers, timeout=2) 464 | except: 465 | log.error(traceback.format_exc()) 466 | return 467 | 468 | if response.status_code == 200: 469 | content_type = response.headers['content-type'] 470 | 471 | if 'turtle' in content_type: 472 | f = 'turtle' 473 | elif 'rdf' in content_type: 474 | f = 'xml' 475 | elif 'n3' in content_type: 476 | f = 'n3' 477 | elif 'n-quads' in content_type: 478 | f = 'nquads' 479 | elif 'trig' in content_type: 480 | f = 'trig' 481 | elif 'json' in content_type: 482 | f = 'jsonld' 483 | else: 484 | f = None 485 | print "Format {} not recognised as valid RDF serialization format".format(content_type) 486 | 487 | if f is not None: 488 | # Parse the response into the graph with the given URI in our 489 | # local store Dataset 490 | resource_graph.parse(data=response.text, format=f) 491 | else: 492 | log.warning("URI did not return any recognisable result") 493 | 494 | 495 | def query(query): 496 | return g.query(query) 497 | 498 | 499 | def remote_query(query, accept=['application/sparql-results+json', 'text/json', 'application/json']): 500 | endpoints = SPARQL_ENDPOINT_MAPPING.values() 501 | if SPARQL_ENDPOINT is not None: 502 | endpoints.append(SPARQL_ENDPOINT) 503 | 504 | log.debug(endpoints) 505 | 506 | header = {'Accept': ', '.join(accept)} 507 | params = {'query': query} 508 | 509 | log.debug(header) 510 | 511 | results = None 512 | for endpoint in endpoints: 513 | response = requests.get(endpoint, headers=header, params=params) 514 | 515 | try: 516 | # The response is a JSON SPARQL result 517 | response_json = response.json() 518 | log.debug(response_json) 519 | if 'results' in response_json: 520 | # Add provenance information to the bindings 521 | bindings = [] 522 | for b in response_json['results']['bindings']: 523 | b['endpoint'] = {'value': endpoint, 'type': 'uri'} 524 | bindings.append(b) 525 | 526 | if results is None: 527 | results = response_json 528 | results['head']['vars'].append('endpoint') 529 | results['results']['bindings'] = bindings 530 | else: 531 | results['results']['bindings'].extend(bindings) 532 | else: 533 | raise Exception("JSON but not SPARQL results") 534 | except: 535 | # The response is the result of a CONSTRUCT or DESCRIBE or ASK 536 | # query 537 | if results is None: 538 | results = '\n# ===\n# Result from {}\n# ===\n'.format( 539 | endpoint) + response.content 540 | else: 541 | results += '\n# ===\n# Result from {}\n# ===\n'.format( 542 | endpoint) + response.content 543 | 544 | return results 545 | 546 | 547 | def traverse(uri, results, source='s', target='o', visited={}, depth=0, maxdepth=2): 548 | """Traverse the results and build a sunburst JSON graph in the direction indicated by source/target 549 | 550 | i.e. to build all outgoing edges, specify 's' and 'o', respectively 551 | otherwise, specify 'o' and 's'. 552 | """ 553 | 554 | log.info("{} Traversing from {} to {}".format(" "*depth*10, source, target)) 555 | # log.debug(visited) 556 | if uri in visited.keys(): 557 | log.debug(u"{} Already visited {}".format(" "*depth*10, uri)) 558 | return visited[uri], visited 559 | elif depth >= maxdepth: 560 | log.debug(u"Maximum depth exceeded") 561 | return [], visited 562 | 563 | log.debug(u"{} Visiting {}".format(" "*depth*10, uri)) 564 | 565 | 566 | edges = {} 567 | edge_array = [] 568 | 569 | for r in results: 570 | if r[source]['value'] != uri: 571 | # Continue to next result if this result does not apply to the current node 572 | continue 573 | 574 | children = [] 575 | if r[target]['type'] not in ['literal', 'typed-literal']: 576 | log.debug(u"{} Found child {}".format(" "*depth*10, r[target]['value'])) 577 | 578 | children, visited = traverse(r[target]['value'], results, source=source, target=target, visited=visited, depth=depth+1) 579 | 580 | node = { 581 | "name": r[target]['value'], 582 | "size": 1000, 583 | } 584 | 585 | if len(children) > 0: 586 | node["children"] = children 587 | 588 | edges.setdefault(r['p']['value'], {}).setdefault('children', {})[r[target]['value']] = node 589 | 590 | # Iterate over the edges, to rewrite to arrays of dictionaries 591 | log.debug(u"{} Rewriting children dictionary to array for {}".format(" "*depth*10, uri)) 592 | for pk, pv in edges.items(): 593 | child_array = [] 594 | for sk, sv in pv['children'].items(): 595 | child_array.append(sv) 596 | edge_array.append({ 597 | 'name': pk, 598 | 'children': child_array 599 | }) 600 | 601 | visited[uri] = edge_array 602 | return edge_array, visited 603 | 604 | 605 | def prepare_sunburst(uri, results, maxdepth=1): 606 | log.debug(u"Preparing sunburst for {}".format(uri)) 607 | 608 | # Traverse outgoing edges 609 | outgoing_array, v = traverse(uri, results, source='s', target='o', visited={}, maxdepth=maxdepth) 610 | # Traverse incoming edges 611 | incoming_array, v = traverse(uri, results, source='o', target='s', visited={}, maxdepth=maxdepth) 612 | 613 | labels = set() 614 | # Find the labels for this resource 615 | for r in results: 616 | if r['s']['value'] == uri and r['p']['value'] in label_properties: 617 | labels.add(r['o']['value']) 618 | 619 | return list(labels), {'name': uri, 'children': incoming_array}, {'name': uri, 'children': outgoing_array} 620 | 621 | 622 | -------------------------------------------------------------------------------- /src/app/config-template.py: -------------------------------------------------------------------------------- 1 | from urlparse import urljoin 2 | import os 3 | 4 | 5 | 6 | # Set LOCAL_STORE to True if you want brwsr to just load a (smallish) RDF file into server memory 7 | # rather than operate on an external SPARQL store 8 | LOCAL_STORE = os.getenv('LOCAL_STORE') or False 9 | 10 | # Set LOCAL_FILE to the relative or absolute path of the file you want brwsr to load when 11 | # LOCAL_STORE is True. The brwsr application will just use RDFLib to guess the file format based on the extension. 12 | # You can use UNIX file masks such as * and ? to load multiple files 13 | LOCAL_FILE = os.getenv('LOCAL_FILE') or 'justsomeexample.trig' 14 | 15 | # Set this to the SPARQL endpoint uri of your triplestore 16 | # e.g. "http://dbpedia.org/sparql" 17 | SPARQL_ENDPOINT = os.getenv('SPARQL_ENDPOINT') or "http://your.sparql.endpoint.here/sparql" 18 | 19 | # If brwsr is backed by multiple separate triple stores, use SPARQL_ENDPOINT_MAPPING to 20 | # make sure that each URI for which the LOCAL_NAME (i.e. the URI with the DEFAULT_BASE remove, if present) 21 | # starts with a key of the SPARQL_ENDPOINT_MAPPING file, the proper SPARQL endpoint is used. 22 | # 23 | # You can also use Python-style regular expressions in the prefix description (the keys of this dictionary) 24 | # 25 | # Note that brwsr will allways *also* query the default SPARQL_ENDPOINT 26 | # 27 | # Example: 28 | # SPARQL_ENDPOINT_MAPPING = { 29 | # "/example": "http://the.sparql.endpoint.for.uris.starting.with/example/sparql", 30 | # "http://dbpedia.org/\w+/": "http://dbpedia.org/sparql" 31 | # } 32 | # NB: This parameter cannot be set using environment variables 33 | SPARQL_ENDPOINT_MAPPING = {} 34 | 35 | # The statements Url of a Druid instance (http://triply.cc), 36 | # e.g. "http://druid.instance.url/_api/datasets/Username/Dataset/statements.triply" 37 | DRUID_STATEMENTS_URL = os.getenv('DRUID_STATEMENTS_URL') or None 38 | 39 | # The statements Url of a Linked Data Fragments service (http://linkeddatafragments.org), 40 | # e.g. "http://data.linkeddatafragments.org/dbpedia2014" 41 | LDF_STATEMENTS_URL = os.getenv('LDF_STATEMENTS_URL') or None 42 | 43 | # The DEFAULT_BASE is the prefix of the URI's in the triple store that can be browsed by brwsr 44 | # Requests to brwsr only include the local name (i.e. the the part after the third slash '/'), 45 | # the DEFAULT_BASE is *always* prepended to this local name to make up the URI that's used to 46 | # query the triple store 47 | # e.g. "http://dbpedia.org" (without the last slash!) 48 | DEFAULT_BASE = os.getenv('DEFAULT_BASE') or "http://your.base.uri.here" 49 | 50 | # The LOCAL_DOCUMENT_INFIX is the infix used between the DEFAULT_BASE and the local name of the URI 51 | # to denote the HTML representation of the RDF resource (see the Cool URI's specification) 52 | LOCAL_DOCUMENT_INFIX = os.getenv('LOCAL_DOCUMENT_INFIX') or 'doc' 53 | 54 | # The LOCAL_SERVER_NAME is the address brwsr listens to. It needs to know this to build proper 55 | # requests when you click a URI in the brwsr page of a resource. 56 | # e.g. "http://localhost:5000" if running flask. 57 | LOCAL_SERVER_NAME = os.getenv('LOCAL_SERVER_NAME') or "http://your.server.name.here" 58 | 59 | # By default brwsr assumes it is running at the root of the server, 60 | # If you want to run brwsr under a directory (e.g. http://example.com/brwsr rather than http://example.com), you need to do this 61 | # via a reverse proxy, and tell brwsr about it (set BEHIND_PROXY to True) 62 | # 63 | ######### 64 | # Example Nginx configuration (adapted from http://flask.pocoo.org/snippets/35/) 65 | ######### 66 | # 67 | # location /myprefix { 68 | # proxy_pass http://localhost:5000; 69 | # proxy_set_header Host $host; 70 | # proxy_set_header Upgrade $http_upgrade; 71 | # proxy_set_header Connection "upgrade"; 72 | # proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; 73 | # proxy_set_header X-Scheme $scheme; 74 | # proxy_set_header X-Script-Name /myprefix; 75 | # } 76 | # 77 | # Where 'myprefix' should be set to the location you want to be running brwsr under 78 | # The 'proxy_pass' setting should point to the address and port you are running brwsr at (default is localhost port 5000). 79 | # 80 | ######### 81 | BEHIND_PROXY = os.getenv('BEHIND_PROXY') or False 82 | 83 | 84 | # The START_LOCAL_NAME is the local name of the first URI shown in brwsr if no URI is specified 85 | # e.g. "resource/Amsterdam" when using the DBPedia settings 86 | START_LOCAL_NAME = os.getenv('START_LOCAL_NAME') or "some/local/name" 87 | 88 | # The START_URI is simply the combination of the DEFAULT_BASE and the START_LOCAL_NAME 89 | # (i.e. there is no need to change this, usually) 90 | # e.g. this will become "http://dbpedia.org/resource/Amsterdam" 91 | START_URI = os.getenv('START_URI') or urljoin(DEFAULT_BASE, START_LOCAL_NAME) 92 | 93 | # Set query results limit because otherwise your browser might crash. 94 | QUERY_RESULTS_LIMIT = os.getenv('QUERY_RESULTS_LIMIT') or 1000 95 | 96 | # The port via which to run brwsr 97 | PORT = os.getenv('PORT') or 5000 98 | 99 | # Debug logging 100 | DEBUG = os.getenv('DEBUG') or False 101 | 102 | # Browse URIs that do not match the DEFAULT_BASE 103 | BROWSE_EXTERNAL_URIS = os.getenv('BROWSE_EXTERNAL_URIS') or True 104 | 105 | # Dereference external URIs (i.e. retrieve RDF served at that location, and display the resource) 106 | # NB: This may be slow, depending on the responsiveness of the server at hand 107 | # NB: The resulting RDF is stored locally (in memory) which means that this is a potential memory hog for 108 | # servers that are visited frequently. TODO: store results in a triple store 109 | DEREFERENCE_EXTERNAL_URIS = os.getenv('DEREFERENCE_EXTERNAL_URIS') or False 110 | 111 | # Set the HTTP method to use for communicating with SPARQL endpoint. 'GET' is the default. 112 | SPARQL_METHOD = os.getenv('SPARQL_METHOD') or 'GET' 113 | 114 | # Set any custom parameters to be sent to the SPARQL endpoint 115 | # e.g. CUSTOM_PARAMETERS = {'reasoning': 'true'} for Stardog 116 | # NB: This parameter cannot be set using environment variables 117 | CUSTOM_PARAMETERS = {'reasoning': 'true'} 118 | 119 | # The cache timeout in seconds 120 | CACHE_TIMEOUT = os.getenv('CACHE_TIMEOUT') or 300 121 | 122 | # Depth of the Sunburst visualization (default = 1) 123 | # Warning: setting this to a value > 1 will really make the visualization a *lot* slower, 124 | # also depending on the number of endpoints, or services you are calling. 125 | SUNBURST_DEPTH = os.getenv('SUNBURST_DEPTH') or 1 126 | 127 | # Use the prefLabel.org service for rendering labels for known URIs 128 | PREFLABEL_SERVICE = os.getenv('PREFLABEL_SERVICE') or True 129 | -------------------------------------------------------------------------------- /src/app/static/brwsr.css: -------------------------------------------------------------------------------- 1 | .literal:hover { 2 | font-weight: bolder; 3 | } 4 | 5 | td div { 6 | max-height: 200px; 7 | overflow: auto; 8 | } 9 | 10 | td { 11 | width: 50%; 12 | } 13 | 14 | .img { 15 | height: 50px; 16 | float: left; 17 | margin-left: -50px; 18 | margin-top: 18px; 19 | } 20 | -------------------------------------------------------------------------------- /src/app/static/brwsr.js: -------------------------------------------------------------------------------- 1 | $( document ).ready(function() { 2 | $(".graph").hide(); 3 | 4 | $(".resource").each(function(index){ 5 | var element = this; 6 | var element_text = $(this).text(); 7 | 8 | if (element_text && (element_text.substr(0,4) == 'http' || element_text.substr(0,6) == 'nodeID')) { 9 | updateLabel(element_text,element); 10 | if (preflabel_service == true) { 11 | $.get('http://preflabel.org/api/v1/label/'+encodeURIComponent($(element).text())+"?callback=?", function(data){ 12 | updateLabel(data,element); 13 | }); 14 | } 15 | } 16 | }); 17 | 18 | $(".graphs").each(function(index){ 19 | var element = this; 20 | var anchor = $(""); 22 | var count = $(this).children(".graph").length; 23 | 24 | anchor.append(icon); 25 | if (count>1){ 26 | anchor.append(''+count+''); 27 | } 28 | 29 | 30 | $(element).append(anchor); 31 | }); 32 | 33 | $(".graphs").hover( 34 | function(e){ 35 | 36 | $(this).children(".gplaceholder").hide(); 37 | $(this).children(".graph").show(); 38 | }, 39 | function(e){ 40 | $(this).children(".graph").hide(); 41 | $(this).children(".gplaceholder").show(); 42 | } 43 | ); 44 | 45 | $(".graph").each(function(index){ 46 | var element = this; 47 | 48 | var anchor = $(""); 49 | anchor.attr('href',$(element).attr('local')); 50 | 51 | 52 | 53 | var icon = $(""); 54 | 55 | anchor.append(icon); 56 | 57 | $(element).html(anchor); 58 | }); 59 | 60 | $(".literal").each(function(index){ 61 | var element = this; 62 | var element_text = $(this).text(); 63 | if (element_text.length > 200 ) { 64 | 65 | var span = $(''); 66 | 67 | if ($(element).attr('lang') != '') { 68 | var lang = $(""); 69 | lang.text($(element).attr('lang')); 70 | span.append(lang); 71 | } 72 | 73 | var dotdot = $('...'); 74 | var head = $(''); 75 | head.append(element_text.substr(0,200)); 76 | 77 | var open = $(''); 78 | 79 | var tail = $(''); 80 | tail.append(element_text.substr(200)); 81 | 82 | var close = $(''); 83 | 84 | span.append(head); 85 | span.append(dotdot); 86 | span.append(open); 87 | span.append(tail); 88 | span.append(close); 89 | tail.hide(); 90 | close.hide(); 91 | 92 | span.on('click', function(e){ 93 | close.toggle(); 94 | tail.toggle(); 95 | dotdot.toggle(); 96 | open.toggle(); 97 | }); 98 | 99 | 100 | $(element).html(span); 101 | } 102 | }); 103 | 104 | function updateLabel(label,element){ 105 | var anchor = $(""); 106 | 107 | anchor.append(label); 108 | anchor.attr('href',$(element).attr('local')); 109 | 110 | var icon = $(""); 111 | var icon_anchor = $(''); 112 | icon_anchor.append(icon) 113 | icon_anchor.attr('href',$(element).text()); 114 | icon_anchor.css('margin-right','3px'); 115 | 116 | var view = $(""); 117 | var view_anchor = $(''); 118 | view_anchor.append(view) 119 | view_anchor.attr('href',"/graph?uri="+$(element).text()); 120 | view_anchor.css('margin-right', '3px'); 121 | 122 | 123 | var span = $(""); 124 | span.append(view_anchor); 125 | span.append(icon_anchor); 126 | 127 | 128 | span.append(anchor); 129 | 130 | $(element).html(span); 131 | } 132 | 133 | }); 134 | -------------------------------------------------------------------------------- /src/app/static/fonts/glyphicons-halflings-regular.eot: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Data2Semantics/brwsr/20d985d83ff1d697252c51a48d340013b65eea7d/src/app/static/fonts/glyphicons-halflings-regular.eot -------------------------------------------------------------------------------- /src/app/static/fonts/glyphicons-halflings-regular.svg: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | 45 | 46 | 47 | 48 | 49 | 50 | 51 | 52 | 53 | 54 | 55 | 56 | 57 | 58 | 59 | 60 | 61 | 62 | 63 | 64 | 65 | 66 | 67 | 68 | 69 | 70 | 71 | 72 | 73 | 74 | 75 | 76 | 77 | 78 | 79 | 80 | 81 | 82 | 83 | 84 | 85 | 86 | 87 | 88 | 89 | 90 | 91 | 92 | 93 | 94 | 95 | 96 | 97 | 98 | 99 | 100 | 101 | 102 | 103 | 104 | 105 | 106 | 107 | 108 | 109 | 110 | 111 | 112 | 113 | 114 | 115 | 116 | 117 | 118 | 119 | 120 | 121 | 122 | 123 | 124 | 125 | 126 | 127 | 128 | 129 | 130 | 131 | 132 | 133 | 134 | 135 | 136 | 137 | 138 | 139 | 140 | 141 | 142 | 143 | 144 | 145 | 146 | 147 | 148 | 149 | 150 | 151 | 152 | 153 | 154 | 155 | 156 | 157 | 158 | 159 | 160 | 161 | 162 | 163 | 164 | 165 | 166 | 167 | 168 | 169 | 170 | 171 | 172 | 173 | 174 | 175 | 176 | 177 | 178 | 179 | 180 | 181 | 182 | 183 | 184 | 185 | 186 | 187 | 188 | 189 | 190 | 191 | 192 | 193 | 194 | 195 | 196 | 197 | 198 | 199 | 200 | 201 | 202 | 203 | 204 | 205 | 206 | 207 | 208 | 209 | 210 | 211 | 212 | 213 | 214 | 215 | 216 | 217 | 218 | 219 | 220 | 221 | 222 | 223 | 224 | 225 | 226 | 227 | 228 | -------------------------------------------------------------------------------- /src/app/static/fonts/glyphicons-halflings-regular.ttf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Data2Semantics/brwsr/20d985d83ff1d697252c51a48d340013b65eea7d/src/app/static/fonts/glyphicons-halflings-regular.ttf -------------------------------------------------------------------------------- /src/app/static/fonts/glyphicons-halflings-regular.woff: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Data2Semantics/brwsr/20d985d83ff1d697252c51a48d340013b65eea7d/src/app/static/fonts/glyphicons-halflings-regular.woff -------------------------------------------------------------------------------- /src/app/static/js/sunburst.js: -------------------------------------------------------------------------------- 1 | function drawSunburstForConcept(service_url, uri) { 2 | $('#graphIn').empty(); 3 | $('#graphOut').empty(); 4 | $('#title').empty(); 5 | // $('#graphOut').empty() 6 | $("#loading").show(); 7 | 8 | $("#brws_button").attr('href','/browse?uri='+uri); 9 | 10 | console.log('loading...'); 11 | $.get(service_url, { 12 | 'uri': uri 13 | }, function(data) { 14 | $("#loading").hide(); 15 | console.log('done loading...'); 16 | console.log(data); 17 | 18 | labels = "" 19 | for(i in data.labels){ 20 | labels += data.labels[i] 21 | if (i" + labels + ""); 26 | 27 | if (data) { 28 | if (data.outgoing.hasOwnProperty('children') && data.outgoing.children.length > 0) { 29 | console.log('calling outgoing'); 30 | console.log(data.outgoing.children); 31 | var tree = sortChildren(data.outgoing); 32 | drawSunburst(service_url, tree, "#graphOut", "relations from"); 33 | } else { 34 | $("#graphOut").html("
No relations from this resource"); 35 | } 36 | if (data.incoming.hasOwnProperty('children') && data.incoming.children.length > 0) { 37 | console.log('calling incoming'); 38 | var tree = sortChildren(data.incoming); 39 | drawSunburst(service_url, tree, "#graphIn", "relations to"); 40 | } else { 41 | $("#graphIn").html("
No relations to this resource"); 42 | } 43 | 44 | } else { 45 | $("#noresponse").show(); 46 | } 47 | 48 | }); 49 | } 50 | 51 | function sortChildren(tree) { 52 | _.each(tree.children, function(item, key) { 53 | sortedChildren = _.sortBy(item.children, function(o) { 54 | return o.name.toLowerCase(); 55 | }); 56 | tree.children[key].children = sortedChildren; 57 | }); 58 | return tree; 59 | } 60 | 61 | 62 | function drawSunburst(service_url, root, target, inorout) { 63 | var width = 486, 64 | height = 486, 65 | radius = Math.min(width, height) / 2 - 10, 66 | color = d3.scale.category20c(); 67 | 68 | 69 | 70 | var svg = d3.select(target).append("svg") 71 | .attr("width", width) 72 | .attr("height", height) 73 | .append("g") 74 | .attr("transform", "translate(" + width / 2 + "," + height * 0.52 + ")"); 75 | 76 | var partition = d3.layout.partition() 77 | .sort(null) 78 | .size([2 * Math.PI, radius * radius]) 79 | .value(function(d) { 80 | return 1; 81 | }); 82 | 83 | 84 | 85 | 86 | var arc = d3.svg.arc() 87 | .startAngle(function(d) { 88 | return d.x; 89 | }) 90 | .endAngle(function(d) { 91 | return d.x + d.dx; 92 | }) 93 | .innerRadius(function(d) { 94 | return Math.sqrt(d.y); 95 | }) 96 | .outerRadius(function(d) { 97 | return Math.sqrt(d.y + d.dy); 98 | }); 99 | 100 | 101 | var path = svg.datum(root).selectAll("path") 102 | .data(partition.nodes) 103 | .enter().append("path") 104 | .attr("display", function(d) { 105 | return d.depth ? null : "none"; 106 | }) // hide inner ring 107 | .attr("d", arc) 108 | .style("stroke", "#fff") 109 | .style("fill", function(d) { 110 | return color((d.children ? d : d.parent).name); 111 | }) 112 | .style("fill-rule", "evenodd") 113 | .on("mouseover", update_legend) 114 | .on("mouseout", remove_legend) 115 | .on("click", function(d) { 116 | setTimeout(function() { 117 | handleClick(service_url, d); 118 | }, 10); 119 | }) 120 | .each(stash); 121 | 122 | 123 | var burst = d3.selectAll("input").on("change", function change() { 124 | var value = this.value === "count" ? 125 | function() { 126 | return 1; 127 | } : 128 | function(d) { 129 | return d.size; 130 | }; 131 | 132 | path 133 | .data(partition.value(value).nodes) 134 | .transition() 135 | .duration(1500) 136 | .attrTween("d", arcTween); 137 | }); 138 | 139 | 140 | svg.select("g").append("svg:text") 141 | .style("font-size", "4em") 142 | .style("font-weight", "bold") 143 | .text(function(d) { 144 | return root; 145 | }); 146 | 147 | 148 | var center = svg.append("g"); 149 | 150 | center.append("rect") 151 | .attr("rx", "10px") 152 | .attr("ry", "10px") 153 | .attr("x", -10) 154 | .attr("y", -14) 155 | .attr("fill", "#008cba") 156 | .attr("width", 10) 157 | .attr("height", 20); 158 | 159 | center.append("text") 160 | .text(inorout) 161 | .style("fill", "white") 162 | .classed("inorout", true); 163 | 164 | center.selectAll('rect') 165 | .attr("width", function(d) { 166 | return this.parentNode.getBBox().width + 10; 167 | }) 168 | 169 | // Stash the old values for transition. 170 | function stash(d) { 171 | d.x0 = d.x; 172 | d.dx0 = d.dx; 173 | } 174 | 175 | // Interpolate the arcs in data space. 176 | function arcTween(a) { 177 | var i = d3.interpolate({ 178 | x: a.x0, 179 | dx: a.dx0 180 | }, a); 181 | return function(t) { 182 | var b = i(t); 183 | a.x0 = b.x; 184 | a.dx0 = b.dx; 185 | return arc(b); 186 | }; 187 | 188 | 189 | } 190 | 191 | var legend = d3.select("#legend") 192 | 193 | function update_legend(d) { 194 | legend.html("

" + d.name + "

") 195 | legend.transition().duration(200).style("opacity", "1"); 196 | } 197 | 198 | function remove_legend(d) { 199 | legend.transition().duration(1000).style("opacity", "0"); 200 | } 201 | 202 | function handleClick(service_url, d) { 203 | console.log(d); 204 | console.log("Waiting 1 sec before calling the draw function..."); 205 | drawSunburstForConcept(service_url, d.name); 206 | } 207 | 208 | 209 | d3.select(self.frameElement).style("height", height + "px"); 210 | } 211 | -------------------------------------------------------------------------------- /src/app/static/logo-no-text-150dpi.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Data2Semantics/brwsr/20d985d83ff1d697252c51a48d340013b65eea7d/src/app/static/logo-no-text-150dpi.png -------------------------------------------------------------------------------- /src/app/templates/base.html: -------------------------------------------------------------------------------- 1 | {% extends "bootstrap/base.html" %} 2 | {% block title %}brwsr{% endblock %} 3 | 4 | {% block navbar %} 5 |
6 | 7 |
8 | {% endblock %} 9 | 10 | {% block content %} 11 |
12 |
13 |
14 |

brwsr brws, qry, vz{% if local == True %}, rld{% endif %}

15 |
16 |
17 |
18 |
19 | {% block brwsr_content %} 20 | {% endblock %} 21 |
22 |
23 |
24 |
25 |
26 |

2015/2016 - Rinke Hoekstra, Vrije Universiteit Amsterdam, work supported by Data2Semantics, a COMMIT/ project, and CLARIAH. See http://github.com/Data2Semantics/brwsr.

27 |
28 |
29 |
30 | {% endblock %} 31 | 32 | {% block scripts %} 33 | {{super()}} 34 | 35 | 38 | 39 | {% endblock %} 40 | 41 | {% block styles %} 42 | {{super()}} 43 | 44 | 49 | 51 | 53 | {% endblock %} 54 | 55 | {% block bootstrap_js_bottom %} 56 | 57 | {% endblock %} 58 | -------------------------------------------------------------------------------- /src/app/templates/graph.html: -------------------------------------------------------------------------------- 1 | {% extends "base.html" %} 2 | 3 | {% block brwsr_content %} 4 | 75 | 76 |
77 |
78 |

 

79 |
80 | Loading ... 81 |
82 |
83 | Did not receive anything ... 84 |
85 |
86 |   87 |
88 |
89 |
90 | 91 |
92 |
93 |
94 |
95 |
96 |
97 |
98 |
99 |
100 |
101 | 102 | 103 | 104 | 105 | 106 | {% endblock %} 107 | 108 | 109 | {% block scripts %} 110 | {{super()}} 111 | 112 | 113 | 114 | 122 | {% endblock %} 123 | -------------------------------------------------------------------------------- /src/app/templates/resource.html: -------------------------------------------------------------------------------- 1 | {% extends "base.html" %} 2 | 3 | {% block brwsr_content %} 4 | 5 | 6 | {% if results|length > 0 %} 7 | {% for group in results|groupby(attribute='s') %} 8 | 9 | 10 | {% if group.grouper.value == resource %} 11 |

{{ resource }} relations from

12 | 13 | 14 | 15 | 16 | 17 | {% for predicate in group.list|sort(attribute='p')|groupby(attribute='p') %} 18 | 19 | 22 | 42 | 43 | {% endfor %} 44 | 45 |
predicateobject
20 | {{ predicate.grouper.value }} 21 | 23 |
24 | {% for object in predicate.list|sort(attribute='o')|groupby(attribute='o') %} 25 | 26 | {% for graph in object.list %} 27 | {% if graph.g.local != "" %} 28 | 29 | {% endif %} 30 | {% endfor %} 31 | 32 | {% if object.grouper.type == 'uri' or object.grouper.type == 'bnode' %} 33 | {{ object.grouper.value }} 34 |
35 | {% else %} 36 | {{ object.grouper.value }} 37 |
38 | {% endif %} 39 | {% endfor %} 40 |
41 |
46 | {% endif %} 47 | {% endfor %} 48 | 49 | {% for group in results|groupby(attribute='o') %} 50 | {% if group.grouper.value == resource %} 51 |

{{ resource }} relations to

52 | 53 | 54 | 55 | 56 | 57 | {% for predicate in group.list|sort(attribute='p')|groupby(attribute='p') %} 58 | 59 | 76 | 79 | 80 | {% endfor %} 81 | 82 |
subjectpredicate
60 |
61 | {% for subject in predicate.list|sort(attribute='s')|groupby(attribute='s') %} 62 | {% for graph in subject.list %} 63 | {% if graph.g.local != "" %} 64 | 65 | {% endif %} 66 | {% endfor %} 67 | {% if subject.grouper.type == 'uri' or subject.grouper.type == 'bnode' %} 68 | {{ subject.grouper.value }} 69 |
70 | {% else %} 71 | {{ subject.grouper.value }}
72 | {% endif %} 73 | {% endfor %} 74 |
75 | 		77 | {{ predicate.grouper.value }} 78 |
83 | {% endif %} 84 | {% endfor %} 85 | 86 | {% for group in results|groupby(attribute='p') %} 87 | {% if group.grouper.value == resource %} 88 |

{{ resource }} in predicate position

89 | 90 | 91 | 92 | 93 | 94 | {% for subject in group.list|sort(attribute='s')|groupby(attribute='s') %} 95 | 96 | 113 | 132 | 133 | {% endfor %} 134 | 135 |
subjectobject
97 |
98 | {% for subject in subject.list|sort(attribute='s')|groupby(attribute='s') %} 99 | {% for graph in subject.list %} 100 | {% if graph.g.local != "" %} 101 | 102 | {% endif %} 103 | {% endfor %} 104 | {% if subject.grouper.type == 'uri' or subject.grouper.type == 'bnode' %} 105 | {{ subject.grouper.value }} 106 |
107 | {% else %} 108 | {{ subject.grouper.value }}
109 | {% endif %} 110 | {% endfor %} 111 |
112 | 		114 |
115 | {% for subject in subject.list|sort(attribute='s')|groupby(attribute='s') %} 116 | {% for object in subject.list|sort(attribute='o')|groupby(attribute='o') %} 117 | {% for graph in object.list %} 118 | {% if graph.g.local != "" %} 119 | 120 | {% endif %} 121 | {% endfor %} 122 | {% if object.grouper.type == 'uri' or object.grouper.type == 'bnode' %} 123 | {{ object.grouper.value }} 124 |
125 | {% else %} 126 | {{ object.grouper.value }}
127 | {% endif %} 128 | {% endfor %} 129 | {% endfor %} 130 |
131 |
136 | {% endif %} 137 | {% endfor %} 138 | 139 | {% else %} 140 |

{{ resource }}

141 | No results found 142 | 143 | {% endif %} 144 | 145 | 146 | 147 | {% endblock %} 148 | -------------------------------------------------------------------------------- /src/app/templates/sparql.html: -------------------------------------------------------------------------------- 1 | {% extends "base.html" %} 2 | 3 | {% block brwsr_content %} 4 |
5 | {% endblock %} 6 | {% block scripts %} 7 | {{super()}} 8 | 9 | 10 | 24 | {% endblock %} 25 | 26 | {% block styles %} 27 | {{super()}} 28 | 29 | 33 | {% endblock %} 34 | 35 | 36 | -------------------------------------------------------------------------------- /src/app/views.py: -------------------------------------------------------------------------------- 1 | from flask import render_template, request, jsonify, make_response, redirect, url_for, abort 2 | from werkzeug.http import parse_accept_header 3 | import logging 4 | from client import visit, query, init, prepare_sunburst, remote_query 5 | from config import * 6 | import traceback 7 | from rdflib import URIRef, Literal, BNode 8 | 9 | from app import app, cache 10 | from datetime import datetime 11 | print "views", datetime.now().isoformat() 12 | 13 | log = app.logger 14 | log.setLevel(logging.DEBUG) 15 | 16 | 17 | def make_cache_key(*args, **kwargs): 18 | path = request.path 19 | args = str(hash(frozenset(request.args.items()))) 20 | return (path + args).encode('utf-8') 21 | 22 | def localize_rdflib_result(resource): 23 | resource_result = {} 24 | 25 | resource_string = unicode(resource) 26 | 27 | resource_result['value'] = resource_string 28 | if isinstance(resource, URIRef) or isinstance(resource, BNode): 29 | resource_result['type'] = 'uri' 30 | 31 | if resource_string.startswith(DEFAULT_BASE) and '#' not in resource_string: 32 | resource_result['local'] = resource_string.replace(DEFAULT_BASE, LOCAL_SERVER_NAME) 33 | elif BROWSE_EXTERNAL_URIS or '#' in resource_string: 34 | resource_result['local'] = url_for('browse', uri=resource_string, _external=True) 35 | else: 36 | resource_result['local'] = resource_string 37 | elif isinstance(resource, Literal): 38 | resource_result['type'] = 'literal' 39 | 40 | return resource_result 41 | 42 | 43 | def localize_results(results): 44 | log.debug("Localizing results") 45 | local_results = [] 46 | 47 | if LOCAL_STORE: 48 | log.debug("Transforming RDFLib results") 49 | 50 | for (s, p, o, graph) in results: 51 | local_result = {} 52 | local_result['s'] = localize_rdflib_result(s) 53 | local_result['p'] = localize_rdflib_result(p) 54 | local_result['o'] = localize_rdflib_result(o) 55 | local_result['g'] = localize_rdflib_result(graph) 56 | local_results.append(local_result) 57 | else: 58 | for result in results: 59 | local_result = {} 60 | for v in ['s', 'p', 'o', 'g']: 61 | if v not in result: 62 | local_uri = "" 63 | local_result[v] = {} 64 | local_result[v]['type'] = 'uri' 65 | local_result[v]['value'] = local_uri 66 | local_result[v]['local'] = local_uri 67 | elif result[v]['type'] == 'uri' and result[v]['value'].startswith(DEFAULT_BASE) and '#' not in result[v]['value']: 68 | local_uri = result[v]['value'].replace(DEFAULT_BASE, LOCAL_SERVER_NAME) 69 | local_result[v] = result[v] 70 | local_result[v]['local'] = local_uri 71 | elif BROWSE_EXTERNAL_URIS or '#' in result[v]['value']: 72 | local_uri = url_for('browse', uri=result[v]['value'], _external=True) 73 | local_result[v] = result[v] 74 | local_result[v]['local'] = local_uri 75 | else: 76 | local_result[v] = result[v] 77 | local_result[v]['local'] = result[v]['value'] 78 | 79 | local_results.append(local_result) 80 | 81 | return local_results 82 | 83 | 84 | def document(resource_suffix=""): 85 | if resource_suffix: 86 | uri = u"{}/{}".format(DEFAULT_BASE, resource_suffix) 87 | else: 88 | uri = START_URI 89 | 90 | log.debug('The URI we will use is: ' + uri) 91 | 92 | if 'Accept' in request.headers: 93 | mimetype = parse_accept_header(request.headers['Accept']).best 94 | log.debug("Looking for mime type '{}'".format(mimetype)) 95 | else: 96 | log.debug("No accept header, using 'text/html'") 97 | mimetype = 'text/html' 98 | 99 | try: 100 | if mimetype in ['text/html', 'application/xhtml_xml','*/*']: 101 | local_resource_uri = u"{}/{}".format(LOCAL_SERVER_NAME,resource_suffix) 102 | results = visit(uri, format='html') 103 | local_results = localize_results(results) 104 | 105 | return render_template('resource.html', local_resource=local_resource_uri, resource=uri, results=local_results, local=LOCAL_STORE, preflabel=PREFLABEL_SERVICE) 106 | elif mimetype in ['application/json']: 107 | response = make_response(visit(uri,format='jsonld'),200) 108 | response.headers['Content-Type'] = 'application/json' 109 | return response 110 | elif mimetype in ['application/rdf+xml','application/xml']: 111 | response = make_response(visit(uri,format='rdfxml'),200) 112 | response.headers['Content-Type'] = 'application/rdf+xml' 113 | return response 114 | elif mimetype in ['application/x-turtle','text/turtle']: 115 | response = make_response(visit(uri,format='turtle'),200) 116 | response.headers['Content-Type'] = 'text/turtle' 117 | return response 118 | except Exception as e: 119 | log.error(e) 120 | log.error(traceback.format_exc()) 121 | return make_response("Incorrect mimetype or other error", 400) 122 | 123 | 124 | 125 | @app.route('/favicon.ico') 126 | @cache.cached(timeout=CACHE_TIMEOUT, key_prefix=make_cache_key) 127 | def icon(): 128 | return jsonify({'error': "no icon"}) 129 | 130 | 131 | @app.route('/graph') 132 | @cache.cached(timeout=CACHE_TIMEOUT, key_prefix=make_cache_key) 133 | def graph(): 134 | uri = request.args.get('uri', None) 135 | return render_template('graph.html', uri=uri, service_url="{}/graph/json".format(LOCAL_SERVER_NAME), local=LOCAL_STORE) 136 | 137 | 138 | @app.route('/graph/json') 139 | @cache.cached(timeout=CACHE_TIMEOUT, key_prefix=make_cache_key) 140 | def graph_json(): 141 | uri = request.args.get('uri', None) 142 | 143 | # Visit the selected URI up to a depth of 2 144 | results = visit(uri, format='html', depth=SUNBURST_DEPTH) 145 | local_results = localize_results(results) 146 | # graph = prepare_graph(local_results) 147 | labels, incoming, outgoing = prepare_sunburst(uri, local_results, SUNBURST_DEPTH) 148 | 149 | return jsonify({'labels': labels, 'incoming': incoming, 'outgoing': outgoing}) 150 | 151 | 152 | @app.route('/browse') 153 | @cache.cached(timeout=CACHE_TIMEOUT, key_prefix=make_cache_key) 154 | def browse(): 155 | uri = request.args.get('uri', None) 156 | 157 | if uri is None: 158 | return document() 159 | else: 160 | if 'Accept' in request.headers: 161 | mimetype = parse_accept_header(request.headers['Accept']).best 162 | else: 163 | log.debug("No accept header, using 'text/html'") 164 | mimetype = 'text/html' 165 | 166 | try: 167 | if mimetype in ['text/html', 'application/xhtml_xml', '*/*']: 168 | try: 169 | results = visit(uri, format='html', external=True) 170 | except: #when the uri is a javascript injection (or any other illegal string) this will fail and the uri shouldn't be send to the response 171 | return render_template('resource.html', local_resource='http://bla', resource="error in sparql query", results=[], local=LOCAL_STORE, preflabel=PREFLABEL_SERVICE) 172 | local_results = localize_results(results) 173 | if local_results == []: #when there are no results the uri might be a javascript injection and the uri shouldn't be send to the response 174 | uri = "unknown uri" 175 | return render_template('resource.html', local_resource='http://bla', resource=uri, results=local_results, local=LOCAL_STORE, preflabel=PREFLABEL_SERVICE) 176 | elif mimetype in ['application/json']: 177 | response = make_response(visit(uri, format='jsonld', external=True), 200) 178 | response.headers['Content-Type'] = 'application/json' 179 | return response 180 | elif mimetype in ['application/rdf+xml', 'application/xml']: 181 | response = make_response(visit(uri, format='rdfxml', external=True), 200) 182 | response.headers['Content-Type'] = 'application/rdf+xml' 183 | return response 184 | elif mimetype in ['application/x-turtle', 'text/turtle']: 185 | response = make_response(visit(uri, format='turtle', external=True), 200) 186 | response.headers['Content-Type'] = 'text/turtle' 187 | return response 188 | except Exception as e: 189 | log.error(e) 190 | log.error(traceback.format_exc()) 191 | return traceback.format_exc() 192 | 193 | 194 | @app.route('/') 195 | @cache.cached(timeout=CACHE_TIMEOUT, key_prefix=make_cache_key) 196 | def redirect(resource_suffix): 197 | log.debug("Retrieved resource_suffix " + resource_suffix) 198 | if resource_suffix.startswith('{}/'.format(LOCAL_DOCUMENT_INFIX)): 199 | log.debug("DOC Retrieved resource_suffix " + resource_suffix) 200 | return document(resource_suffix[(len(LOCAL_DOCUMENT_INFIX)+1):]) 201 | else: 202 | log.debug("ID Retrieved resource_suffix " + resource_suffix) 203 | if resource_suffix.startswith('http'): 204 | abort(500) 205 | 206 | resource_suffix = u"{}/{}".format(LOCAL_DOCUMENT_INFIX,resource_suffix) 207 | 208 | redirect_url = url_for('redirect',resource_suffix=resource_suffix,_external=True) 209 | 210 | response = make_response('Moved permanently',303) 211 | response.headers['Location'] = redirect_url 212 | response.headers['Accept'] = request.headers['Accept'] 213 | 214 | return response 215 | 216 | 217 | @app.route('/sparql') 218 | @cache.cached(timeout=CACHE_TIMEOUT, key_prefix=make_cache_key) 219 | def sparql(): 220 | if config.LOCAL_STORE: 221 | log.info('Querying local store') 222 | return render_template('sparql.html', endpoint=url_for('local_sparql')) 223 | else: 224 | log.info('Querying remote endpoints') 225 | return render_template('sparql.html', endpoint=url_for('remote_sparql')) 226 | 227 | 228 | @cache.cached(timeout=CACHE_TIMEOUT, key_prefix=make_cache_key) 229 | @app.route('/remote/sparql', methods=['POST', 'GET']) 230 | def remote_sparql(): 231 | """This is a wrapper around remote SPARQL endpoints to allow querying from JavaScript (YASQE/YASR)""" 232 | if request.method == 'POST': 233 | q = request.form['query'] 234 | else: 235 | q = request.args.get('query', '') 236 | 237 | log.debug(q) 238 | results = remote_query(q, accept=request.headers.getlist('accept')) 239 | log.debug(results) 240 | 241 | if isinstance(results, dict): 242 | return jsonify(results) 243 | else: 244 | return results 245 | 246 | 247 | @app.route('/local/sparql', methods=['POST']) 248 | @cache.cached(timeout=CACHE_TIMEOUT, key_prefix=make_cache_key) 249 | def local_sparql(): 250 | if config.LOCAL_STORE: 251 | log.debug("Querying local store") 252 | q = request.form['query'] 253 | log.debug(q) 254 | results = query(q).serialize(format='json') 255 | log.debug(results) 256 | return results 257 | else: 258 | log.warning("No local store configured") 259 | 260 | 261 | @app.route('/') 262 | @cache.cached(timeout=CACHE_TIMEOUT, key_prefix=make_cache_key) 263 | def index(): 264 | if len(START_LOCAL_NAME) > 0: 265 | 266 | redirect_url = url_for('redirect',resource_suffix=START_LOCAL_NAME,_external=True,_scheme="http") 267 | log.debug("ROOT Redirecting to "+redirect_url) 268 | 269 | response = make_response('Moved permanently',303) 270 | response.headers['Location'] = redirect_url 271 | response.headers['Accept'] = request.headers['Accept'] 272 | 273 | return response 274 | else: 275 | return document() 276 | 277 | 278 | @app.route('/reload') 279 | def reload(): 280 | # Refresh the local store by reloading the files 281 | init() 282 | 283 | # Browse the uri passed in the GET request (if none, just show the START_URI) 284 | return browse() 285 | -------------------------------------------------------------------------------- /src/gunicorn_config.py: -------------------------------------------------------------------------------- 1 | workers = 1 2 | worker_class = 'socketio.sgunicorn.GeventSocketIOWorker' 3 | bind = '0.0.0.0:5000' 4 | pidfile = '/tmp/gunicorn-brwsr.pid' 5 | debug = True 6 | loglevel = 'debug' 7 | errorlog = '/tmp/gunicorn_brwsr_error.log' 8 | accesslog = '/tmp/gunicorn_brwsr_access.log' 9 | daemon = True 10 | -------------------------------------------------------------------------------- /src/run.py: -------------------------------------------------------------------------------- 1 | from app import run 2 | 3 | from datetime import datetime 4 | print "run", datetime.now().isoformat() 5 | 6 | if __name__ == "__main__": 7 | run() 8 | -------------------------------------------------------------------------------- /update.sh: -------------------------------------------------------------------------------- 1 | echo "Updating from git" 2 | git pull 3 | echo "Killing existing gunicorn process" 4 | pkill -F /tmp/gunicorn-brwsr.pid 5 | echo "Waiting for a couple of seconds" 6 | sleep 3 7 | echo "Activating the virtual environment" 8 | source bin/activate 9 | echo "Changing directory to src" 10 | cd src 11 | echo "Starting the gunicorn process" 12 | gunicorn -c gunicorn_config.py app:app 13 | --------------------------------------------------------------------------------