├── .gitignore ├── Architecture.md ├── Components.md ├── Dataflows.md ├── Installation.md ├── Manual-steps-for-Ubuntu-14.04.md ├── README.md ├── Runtime-environment.md ├── Work-with-us.md └── images ├── architecture1.png ├── compose.png ├── compose_load_balancing.png ├── geocoder_dataflow.png ├── logo.png ├── map_dataflow.png ├── realtime_dataflow.png └── route_dataflow.png /.gitignore: -------------------------------------------------------------------------------- 1 | .DS_Store -------------------------------------------------------------------------------- /Architecture.md: -------------------------------------------------------------------------------- 1 | Digitransit architecture is based on microservices architecture. Microservices are small, autonomous services that work together that allow us to build larger applications on top of APIs that the services provide. 2 | 3 | ![Architecture](images/architecture1.png) 4 | 5 | Digitransit has three types of components: 6 | 7 | 1. UI 8 | 2. API Service components 9 | 3. Deployment 10 | 11 | ## UI 12 | 13 | User interface is a responsive application built with React. Application connects to backend APIs. 14 | 15 | ## API Service components 16 | 17 | Routing, Geocoding, Map tiles, and realtime data are provided through APIs. There are no dependencies between these logical entities. (e.g. Routing will not use geocoder, Map won't use routing). API service component consists of 1..n Docker containers. 18 | 19 | ## Deployment 20 | 21 | Digitransit-deploy project contains tools for DevOps teams. These tools can be used to build docker images, start containers, and link containers together. 22 | 23 | -------------------------------------------------------------------------------- /Components.md: -------------------------------------------------------------------------------- 1 | ## Digitransit-ui 2 | User interface 3 | * https://github.com/HSLdevcom/digitransit-ui 4 | 5 | ## Digitransit-otp 6 | Open Trip Planner routing API 7 | * https://github.com/HSLdevcom/OpenTripPlanner 8 | 9 | ## Digitransit-geocoder 10 | Geocoding service 11 | * https://github.com/HSLdevcom/digitransit-geocoder 12 | 13 | ## Digitransit-realtime-server 14 | Realtime information server 15 | * https://github.com/HSLdevcom/navigator-server 16 | 17 | ## Digitransit-map 18 | Map tile server 19 | * https://github.com/HSLdevcom/digitransit-map 20 | 21 | ## Digitransit-deploy 22 | Deployment script for platform 23 | * https://github.com/HSLdevcom/digitransit-deploy -------------------------------------------------------------------------------- /Dataflows.md: -------------------------------------------------------------------------------- 1 | ## Overview 2 | Goal of the data flow is to load raw data from multiple sources, convert that to different format and load it into to components. E.g. in case of Finland geocoding, raw data sources consist of Open Street Map, couple datasets form National Land Survey, and various other datasets. 3 | 4 | ## Supported data formats 5 | There isn’t one data format that would suit all needs for Journey planning. Basically, we want to convert various data formats into 4 different groups: 6 | 7 | 1. Transportation schedules and associated geographic information. 8 | 2. Geocoding data 9 | 3. Realtime data 10 | 4. Map tile data 11 | 12 | We start with raw data. Then, depending on component, we apply conversions in order to refine data into proper data format. After refinement, data gets loaded into component. Finally components provides loaded data through it’s API. 13 | 14 | ### Route data flow 15 | 16 | ![Route data flow](images/route_dataflow.png) 17 | 18 | Route data is built of the whole Finland area. First, we download raw data in various formats from Helsinki, Tampere, Oulu, etc. and this data is stored on disk. Also, we load Finnish national data from Finnish Transport Agency that contains national routes for e.g. trains. Some data is cleaned, some isn't. 19 | 20 | For Helsinki and Tampere data we apply GTFS shape to OSM map transformation. The purpose of this step is to better allign routes on OSM map. In different map datasources road locations can slightly vary and this step allows us to better draw routes on OSM map. Conversion is currently done using this tool: https://github.com/hannesj/gtfs_shape_mapfit.git 21 | 22 | Finnish national data is in Kalkati-format. Our goal is to get every data into GTFS so Kalkati-to-GTFS transformation is applied using this tool: https://github.com/HSLdevcom/kalkati2gtfs 23 | 24 | After conversion some GTFS transformations are done using OneBusAway transformer (http://developer.onebusaway.org/modules/onebusaway-gtfs-modules/1.3.4-SNAPSHOT/onebusaway-gtfs-transformer-cli.html) 25 | 26 | Route data flow can be studied more closely here: 27 | https://github.com/HSLdevcom/digitransit-deploy/blob/master/roles/route-server-docker-image/templates/build-routes.sh.j2 28 | 29 | Environment dependencies can be studied more closely here: 30 | https://github.com/HSLdevcom/digitransit-deploy/blob/master/roles/route-server-docker-image/templates/install-dependencies.sh.j2 31 | 32 | ### Geocoding data flow 33 | 34 | ![Geocoding data flow](images/geocoder_dataflow.png) 35 | 36 | Geocoder fetches data from multiple sources, converts them into JSON and loads data into ElasticSearch database. Our Geocoder API is a simple REST wrapper on ElasticSearch data. 37 | 38 | Geocoder API docs: 39 | http://digitransit.fi/geocoder/ 40 | 41 | Geocoding data flow can be studied more closely here: 42 | https://github.com/HSLdevcom/digitransit-geocoder/blob/master/scripts/import-data.sh 43 | 44 | Different conversions can be studied more closely here: 45 | https://github.com/HSLdevcom/digitransit-geocoder/tree/master/geocoder 46 | 47 | Geocoder API implementation: 48 | https://github.com/HSLdevcom/digitransit-geocoder/blob/master/geocoder/app.py 49 | 50 | ### Realtime data flow 51 | 52 | ![Realtime data flow](images/realtime_dataflow.png) 53 | 54 | Heavy lifting of realtime data is done in other systems. Digitransit user interface integrates to MQTT datasources in order to read realtime data from vehicles. In Helsinki city area realtime service, vehicle data is sent once every 30 seconds. Our realtime server works as a cache for MQTT datasource thus enabling user interface to query "old" realtime information and show that data immediately to user. 55 | 56 | You might notice that currently the name of realtime server is "Navigator-server". Realtime functionality will be changing in near future and implementation details are not yet clear. Navigator-server is a proof of concept that was developed by HSL earlier. 57 | 58 | Read more about MQTT: 59 | http://mqtt.org/ 60 | 61 | See Realtime-server code: 62 | https://github.com/HSLdevcom/navigator-server 63 | 64 | ### Map tile data flow 65 | 66 | ![Map tile data flow](images/map_dataflow.png) 67 | 68 | Map tile services provide Open Street Map data in two different format: 69 | 70 | 1. Styled Raster tiles 71 | 2. Styles Vector tiles 72 | 73 | First, OSM data is loaded into Postgis in vector format. Then two different Tessera map servers are used to serve that data as tiles. 74 | 75 | Postgis import is done using these files: 76 | https://github.com/HSLdevcom/digitransit-deploy/tree/master/roles/postgis-osm-docker-image/templates 77 | 78 | Vector map server is a Tessera map server started as Docker container. See Dockerfile: 79 | https://github.com/HSLdevcom/digitransit-deploy/blob/master/roles/vector-map-server-docker-image/templates/Dockerfile.j2 80 | 81 | Raster map server runs also on Tessera. See Dockerfile for it: 82 | https://github.com/HSLdevcom/digitransit-deploy/blob/master/roles/map-server-docker-image/templates/Dockerfile.j2 83 | 84 | Vector map style can be seen here: 85 | https://github.com/HSLdevcom/osm-bright.tm2source 86 | 87 | Raster map style (Mapbox studio project): 88 | https://github.com/HSLdevcom/digitransit-map 89 | 90 | -------------------------------------------------------------------------------- /Installation.md: -------------------------------------------------------------------------------- 1 | ## Requirements 2 | 3 | ## Initialize server 4 | 5 | ## Customize digitransit-deploy 6 | 7 | ## Build Open Trip Planner graph 8 | - Configure Open Street Map datasource 9 | - Configure GTFS datasources 10 | - Configure other datasources 11 | - Converting other formats into GTFS 12 | - Fitting GTFS shapes onto Open Street Map roads 13 | - Build OTP graph 14 | 15 | ## Setup digitransit-geocoder 16 | - Configure Open Street Map datasource 17 | - Configure custom datasources 18 | - Build Geocoder database 19 | 20 | ## Setup digitransit-map 21 | - Build Postgres 22 | - Build Map vector server 23 | - Build Map server 24 | 25 | ## Setup digitransit-realtime 26 | 27 | ## Setup digitransit-otp 28 | - configure OTP graph datasource 29 | - configure GTFS-RT datasources 30 | - configure Disruption info datasource 31 | 32 | ## Setup digitransit-ui 33 | - Configure OTP 34 | - Configure Map tiles 35 | - Configure Real time 36 | - Configure Geocoding 37 | - Customizing UI styles 38 | 39 | ## Setup load balancing 40 | 41 | ## Setup docker compose 42 | 43 | ## Start environment -------------------------------------------------------------------------------- /Manual-steps-for-Ubuntu-14.04.md: -------------------------------------------------------------------------------- 1 | 2 | - add custom apt repositories for openjdk 8 and nodejs 3 | for this run following commands as root. The adding of the repository keys are not added yet in the docs, so there could be a massage of not signed packages during the upate and during the installation 4 | ```bash 5 | echo "deb http://ppa.launchpad.net/openjdk-r/ppa/ubuntu trusty main" >> /etc/apt/sources.list 6 | echo "deb https://deb.nodesource.com/node_4.x precise main" >> /etc/apt/sources.list 7 | apt-get update 8 | ``` 9 | - install depencies of opentripplanner 10 | ```bash 11 | apt-get install openjdk-8-jdk-headless 12 | apt-get install nodejs 13 | apt-get install git 14 | apt-get install build-essential 15 | ``` 16 | - add ppa for maven3 (througth main repository only maven 2 available) install maven 3 (>= 3.1.1 for some dependencies during install of opentripplanner 17 | ``` 18 | add-apt-repository ppa:andrei-pozolotin/maven3 19 | apt-get update 20 | apt-get install maven3 21 | ``` 22 | - creating user for installing and running opentripplanner 23 | ```bash 24 | adduser digitransit 25 | ``` 26 | - change to user digitransit and it's homedirectory 27 | ```bash 28 | su digitransit 29 | cd 30 | ``` 31 | - clone opentripplanner 32 | ```bash 33 | git clone https://github.com/hsldevcom/opentripplanner 34 | ``` 35 | - build opentripplanner with maven 36 | ```bash 37 | cd opentripplanner 38 | mvn package 39 | ``` 40 | - create data direcory 41 | ```bash 42 | mkdir ~/data/ 43 | ``` 44 | - get openstreetmap graph for your region, for example download it from [bbbike.org Extract service of pbf-files](http://extract.bbbike.org/) and copy it into the data folder in the homefolder of digitransit 45 | - get gtfs plandata for your region (if not available, ask your public transport provider to hand it out to you in a very kindly way) and copy it into the data folder too 46 | - build graph (the first parameter -Xmx is defining the available maximum memory of this java instance. Be shure that this number fits into your machines memory. The default max memory is 256M, but for larger reagions it's reccomanded to put a decend size for not getting some heap errors) 47 | ```bash 48 | cd 49 | java -Xmx7500M -jar opentripplanner/target/otp-0.20.0-SNAPSHOT-shaded.jar --build ./data/ 50 | ``` 51 | - create directory for the running instance and copy data into it 52 | ```bash 53 | mkdir otp_running_instance 54 | cp opentripplanner/target/otp-0.20.0-SNAPSHOT-shaded.jar otp_running_instances/. 55 | mkdir otp_running_instance/data 56 | cp data/Graph.obj otp_running_instance/data/. 57 | ``` 58 | - run opentripplanner, give the server your preferred portnumber (here 1234, routed to 80 through a reverse proxy as described below 59 | ```bash 60 | java -Xmx2048M -Duser.timezone=Europe/Rome -jar otp-0.20.0-SNAPSHOT-shaded.jar --server --port 1234 --basePath . --graphs data --autoScan --verbose 61 | ``` 62 | - eventually create reverse proxy on apache (as root user) 63 | ```bash 64 | cd /etc/apache2/sites-available/ 65 | touch digitransit.conf 66 | ``` 67 | Add to the created file with your preferred editor the following lines: 68 | ``` 69 | 70 | 71 | ServerName digitransit.localhost 72 | ServerAlias digitransit.localhost 73 | 74 | 75 | ServerAdmin webmaster@localhost 76 | 77 | ProxyPreserveHost On 78 | ProxyRequests off 79 | ProxyPass / http://localhost:1234/ 80 | ProxyPassReverse / http://localhost:1234/ 81 | 82 | ErrorLog ${APACHE_LOG_DIR}/error.log 83 | CustomLog ${APACHE_LOG_DIR}/access.log combined 84 | 85 | 86 | 87 | ``` 88 | then activate the new site 89 | ```bash 90 | cd ../site-enabled/. 91 | ln -s ../sites-available/digitransit 92 | service apache2 restart 93 | ``` 94 | - go to your browser and try to access the server trough your URL set in the reverse proxy configuration 95 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | [![Stories in Ready](https://badge.waffle.io/HSLdevcom/digitransit.png?label=ready&title=Ready)](https://waffle.io/HSLdevcom/digitransit) 2 | 3 | ![Logo](https://raw.githubusercontent.com/wiki/HSLdevcom/digitransit/images/logo.png) 4 | 5 | Digitransit Platform is a open source journey planning solution that combines several open source components into a modern, highly available route planning service. Route planning algorithms and APIs are provided by Open Trip Planner (OTP). OTP is a great solution for general route planning but in order to provide top-notch journey planning other components such as Mobile friendly user interface, Map tile serving, Geocoding, and various data conversion tools are needed. Digitransit platform provides these tools. 6 | 7 | ## Licensing 8 | The source code of the platform is dual-licensed under the EUPL v1.2 and AGPLv3 licenses. 9 | 10 | ## Issues 11 | Our issue tracking and roadmap is handled in [https://digitransit.atlassian.net](https://digitransit.atlassian.net). 12 | 13 | ## Key features 14 | * Multimodal routing 15 | * Realtime information support 16 | * Mobile friendly user interface that supports themes 17 | * Configurable for any region 18 | 19 | ## Demos 20 | * [Finland - Helsinki city area demo](http://matka.hsl.fi/) 21 | * [Finland - National demo](http://digitransit.fi/digitransit-ui/) 22 | 23 | ## Digitransit platform 24 | * [Architecture](Architecture.md) 25 | * [Components](Components.md) 26 | * [Dataflows](Dataflows.md) 27 | * [Runtime environment](Runtime-environment.md) 28 | * [Installation](Installation.md) 29 | * [Work with us](Work-with-us.md) 30 | 31 | ## Digitransit project related documentation 32 | [Project documentation can be found in wiki](https://github.com/HSLdevcom/digitransit/wiki) 33 | 34 | 35 | -------------------------------------------------------------------------------- /Runtime-environment.md: -------------------------------------------------------------------------------- 1 | Digitransit platform components need to be weaved together to form a runtime that can be used for multimodal routing. Each component is first built as Docker image and then started as a Docker container. 2 | 3 | Since logical entities (e.g. Map and Routing) consist of multiple containers we need a way to form a Docker compose which configures networking between containers. To get an idea how containers are linked check this out: 4 | 5 | ![Compose](images/compose.png) 6 | 7 | and then read this Docker-compose file: 8 | https://github.com/HSLdevcom/digitransit-deploy/blob/master/roles/run/templates/docker-compose.yaml.j2 9 | 10 | Gray circle in the image above marks what lives inside compose. HAProxy works as a load balancer for the compose. 11 | 12 | Why do we need HAProxy in front of the compose? We actually run multiple composes in production. One called "Active" on "Passive". 13 | 14 | ![Compose load balancing](images/compose_load_balancing.png) 15 | 16 | This enables us to modify components, load new data, and launch new composes without affecting Active production compose. Once we feel that new compose is ready, we do a simple flip, so that Active and Passive are changed. 17 | 18 | At the moment we run all services in one compose but this will likely change in near future. 19 | -------------------------------------------------------------------------------- /Work-with-us.md: -------------------------------------------------------------------------------- 1 | Interested in developing with us? Great! Best way to get started is to read this documentation, see what's happening on our demo sites and familiarize yourself with component code. Should you live in Finland, you can also Visit us HSL premises (Opastinsilta 6A Helsinki) 2 | 3 | Fastest way to ask questions is probably through http://facebook.com/HSLdevcom -------------------------------------------------------------------------------- /images/architecture1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/HSLdevcom/digitransit/e2c39c46bd1552e5b52e6cccfdc1f3819981b392/images/architecture1.png -------------------------------------------------------------------------------- /images/compose.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/HSLdevcom/digitransit/e2c39c46bd1552e5b52e6cccfdc1f3819981b392/images/compose.png -------------------------------------------------------------------------------- /images/compose_load_balancing.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/HSLdevcom/digitransit/e2c39c46bd1552e5b52e6cccfdc1f3819981b392/images/compose_load_balancing.png -------------------------------------------------------------------------------- /images/geocoder_dataflow.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/HSLdevcom/digitransit/e2c39c46bd1552e5b52e6cccfdc1f3819981b392/images/geocoder_dataflow.png -------------------------------------------------------------------------------- /images/logo.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/HSLdevcom/digitransit/e2c39c46bd1552e5b52e6cccfdc1f3819981b392/images/logo.png -------------------------------------------------------------------------------- /images/map_dataflow.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/HSLdevcom/digitransit/e2c39c46bd1552e5b52e6cccfdc1f3819981b392/images/map_dataflow.png -------------------------------------------------------------------------------- /images/realtime_dataflow.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/HSLdevcom/digitransit/e2c39c46bd1552e5b52e6cccfdc1f3819981b392/images/realtime_dataflow.png -------------------------------------------------------------------------------- /images/route_dataflow.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/HSLdevcom/digitransit/e2c39c46bd1552e5b52e6cccfdc1f3819981b392/images/route_dataflow.png --------------------------------------------------------------------------------