├── .gitignore ├── LICENSE.txt ├── README.md ├── blank.png ├── data └── road-trip-wilderness.mbtiles ├── mbtiles.py ├── mbtiles2files.py └── serve_mbtiles.py /.gitignore: -------------------------------------------------------------------------------- 1 | *.pyc 2 | -------------------------------------------------------------------------------- /LICENSE.txt: -------------------------------------------------------------------------------- 1 | Copyright (c) 2013 Matthew Perry 2 | All rights reserved. 3 | 4 | Redistribution and use in source and binary forms, with or without modification, 5 | are permitted provided that the following conditions are met: 6 | 7 | * Redistributions of source code must retain the above copyright notice, this 8 | list of conditions and the following disclaimer. 9 | 10 | * Redistributions in binary form must reproduce the above copyright notice, this 11 | list of conditions and the following disclaimer in the documentation and/or 12 | other materials provided with the distribution. 13 | 14 | * Neither the name of the software nor the names of its 15 | contributors may be used to endorse or promote products derived from 16 | this software without specific prior written permission. 17 | 18 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND 19 | ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED 20 | WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE 21 | DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR 22 | ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES 23 | (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; 24 | LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON 25 | ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT 26 | (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS 27 | SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 28 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | ## python-mbtiles 2 | 3 | Some python tools for working with [mbtiles](http://mapbox.com/mbtiles-spec/): 4 | 5 | >MBTiles is a specification for storing tiled map data in SQLite databases 6 | >for immediate use and for transfer. The files are designed for portability 7 | >of thousands, hundreds of thousands, or even millions of standard map tile 8 | >images in a single file. 9 | 10 | ### Similar projects 11 | 12 | This is nothing more than an experiment at this point. For more full-featured libraries, you may also want to check out 13 | 14 | * [mbutil](https://github.com/mapbox/mbutil) 15 | * [landez](https://github.com/makinacorpus/landez) 16 | 17 | ### Project Goals 18 | 19 | #### Python classes 20 | 21 | Abstract the details of accessing utfgrid and image data from the sqlite datastore. See `mbtiles.py` 22 | 23 | ```python 24 | tileset = MbtileSet(mbtiles='./data/road-trip-wilderness.mbtiles') 25 | zoom, col, row = 6, 9, 40 26 | tile = tileset.get_tile(zoom, col, row) 27 | binary_png = tile.get_png() 28 | text_json = tile.get_json() 29 | ``` 30 | 31 | #### Tile web server 32 | 33 | Provide a fast, simple, non-blocking web server (using Tornado) to serve image and utfgrid data. You are able to pass a `callback` parameter on utfgrids for dynamic JSONP allowing easy cross-domain and framework-agnostic loading of utfgrid json tiles. See `serve_mbtiles.py` 34 | 35 | ```bash 36 | python serve_mbtiles.py # runs on 8988 37 | wget http://localhost:8988/test/6/9/40.png 38 | wget http://localhost:8988/test/6/9/40.json 39 | wget http://localhost:8988/test/6/9/40.json?callback=test 40 | wget http://localhost:8988/test/6/9/23.json?origin=top # invert y-axis for top-origin tile scheme like Google, etc. 41 | ``` 42 | 43 | #### Covert mbtiles to png/json files 44 | 45 | A script to convert mbtiles files into png/json files on the filesystem. This eliminates the single-file advantages of mbtiles but gains portability in that tiles can be served statically without a web server in front of it. See `mbtiles2files.py`. 46 | 47 | ```bash 48 | # Bottom-origin tiles (TMS) 49 | python mbtiles2files.py -f data/road-trip-wilderness.mbtiles -o /tmp/output 50 | ls /tmp/output/6/9/40.* 51 | 52 | # Invert to top-origin tiles (Google, OSM, etc.) 53 | python mbtiles2files.py -f data/road-trip-wilderness.mbtiles -o /tmp/output --invert 54 | ls /tmp/output/6/9/23.* 55 | ``` 56 | ### Example 57 | 58 | 59 | ### Roadmap 60 | 61 | * Make error handling more robust 62 | * Config file for the server (port, list of mbtiles to serve) 63 | * Handle jpg (coding was stupidly implemented assuming png) 64 | * Test cases, docs 65 | * setup.py file, cheesehop it, etc. 66 | -------------------------------------------------------------------------------- /blank.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/perrygeo/python-mbtiles/1cc63047be0254139e1079ba4bb036a6c6904337/blank.png -------------------------------------------------------------------------------- /data/road-trip-wilderness.mbtiles: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/perrygeo/python-mbtiles/1cc63047be0254139e1079ba4bb036a6c6904337/data/road-trip-wilderness.mbtiles -------------------------------------------------------------------------------- /mbtiles.py: -------------------------------------------------------------------------------- 1 | import sqlite3 2 | import sys 3 | import zlib 4 | import json 5 | import os 6 | import shutil 7 | 8 | 9 | class MbtileSet: 10 | 11 | def __init__(self, mbtiles, outdir=None, origin="bottom"): 12 | self.conn = sqlite3.connect(mbtiles) 13 | self.outdir = outdir 14 | self.origin = origin 15 | if self.origin not in ['bottom','top']: 16 | raise Exception("origin must be either `bottom` or `top`") 17 | 18 | 19 | def write_all(self): 20 | if not self.outdir: 21 | raise Exception("Must specify the outdir property to write_all") 22 | cur = self.conn.cursor() 23 | for row in cur.execute('select zoom_level, tile_column, tile_row from map'): 24 | z, x, y = row[:3] 25 | tile = Mbtile(z, x, y, self.conn, self.origin) 26 | tile.write_png(self.outdir) 27 | tile.write_json(self.outdir) 28 | 29 | def get_tile(self, z, x, y): 30 | return Mbtile(z, x, y, self.conn, self.origin) 31 | 32 | 33 | class Mbtile: 34 | 35 | def __init__(self, z, x, y, conn, origin): 36 | self.zoom = z 37 | self.col = x 38 | self.row = y 39 | self.conn = conn 40 | self.origin = origin 41 | self.blank_png_path = os.path.join(os.path.dirname(os.path.abspath(__file__)), 'blank.png') 42 | 43 | @property 44 | def output_row(self): 45 | ''' 46 | self.row will ALWAYS refer to the bottom-origin tile scheme since MBTiles uses it internally 47 | self.output_row CAN be set to top-origin scheme (like Google Maps etc.) my passing origin="bottom" 48 | 49 | code must account for this but making the front-facing Y coordinate use the self.output_row property 50 | ''' 51 | y = self.row 52 | if self.origin == 'top': 53 | # invert y axis to top origin 54 | ymax = 1 << self.zoom; 55 | y = ymax - self.row - 1; 56 | return y 57 | 58 | def get_png(self): 59 | c = self.conn.cursor() 60 | c.execute('''select tile_data from tiles 61 | where zoom_level = %s 62 | and tile_column = %s 63 | and tile_row = %s''' % (self.zoom,self.col,self.row)) 64 | row = c.fetchone() 65 | if not row: 66 | return None 67 | 68 | return bytes(row[0]) 69 | 70 | def get_json(self): 71 | c = self.conn.cursor() 72 | c2 = self.conn.cursor() 73 | c.execute('''select grid from grids 74 | where zoom_level = %s 75 | and tile_column = %s 76 | and tile_row = %s''' % (self.zoom,self.col,self.row)) 77 | row = c.fetchone() 78 | if not row: 79 | return None 80 | 81 | bt = bytes(row[0]) 82 | j = zlib.decompress(bt) 83 | tgd = json.loads(j) 84 | 85 | kq = ''' 86 | SELECT 87 | keymap.key_name AS key_name, 88 | keymap.key_json AS key_json 89 | FROM map 90 | JOIN grid_utfgrid ON grid_utfgrid.grid_id = map.grid_id 91 | JOIN grid_key ON grid_key.grid_id = map.grid_id 92 | JOIN keymap ON grid_key.key_name = keymap.key_name 93 | WHERE zoom_level = %s AND tile_column = %s AND tile_row = %s; 94 | ''' % (self.zoom, self.col, self.row) 95 | keys = [] 96 | for keyrow in c2.execute(kq): 97 | keyname, keydata = keyrow 98 | keys.append((keyname, eval(keydata))) 99 | datadict = dict(keys) 100 | tgd[u'data'] = datadict 101 | 102 | return json.dumps(tgd) 103 | 104 | def write_png(self, outdir): 105 | z, x, y = [str(i) for i in [self.zoom, self.col, self.output_row]] 106 | pngdir = os.path.join(outdir, z, x) 107 | try: 108 | os.makedirs(pngdir) 109 | except OSError as e: 110 | pass 111 | png = self.get_png() 112 | path = os.path.join(pngdir, y + ".png") 113 | if png: 114 | fh = open(path, 'wb') 115 | fh.write(png) 116 | fh.close() 117 | else: 118 | shutil.copyfile(self.blank_png_path, path) 119 | 120 | def write_json(self, outdir): 121 | z, x, y = [str(i) for i in [self.zoom, self.col, self.output_row]] 122 | jsondir = os.path.join(outdir, z, x) 123 | try: 124 | os.makedirs(jsondir) 125 | except OSError: 126 | pass 127 | path = os.path.join(jsondir, y + ".json") 128 | json = self.get_json() 129 | if json: 130 | fh = open(path , 'w') 131 | fh.write(self.get_json()) 132 | fh.close() 133 | -------------------------------------------------------------------------------- /mbtiles2files.py: -------------------------------------------------------------------------------- 1 | from mbtiles import MbtileSet 2 | import optparse 3 | 4 | if __name__ == '__main__': 5 | parser = optparse.OptionParser( 6 | usage="\nOutputs png/json files from an mbtiles file\nmbtiles2files.py [options] -f -o ") 7 | parser.add_option('-f', '--file', help='Path to .mbtiles file', action='store', 8 | dest='file', type='string') 9 | parser.add_option('-o', '--output', help='Output directory', action='store', 10 | dest='output', type='string') 11 | parser.add_option('-i', '--invert', help='Invert Y axis (True = Top Y origin ala google and bing)', action='store_true', 12 | dest='invert', default=False) 13 | (opts, args) = parser.parse_args() 14 | 15 | if not opts.file: 16 | parser.error("Please specify a valid .mbtiles file") 17 | if not opts.output: 18 | parser.error("Please specify and output directory") 19 | 20 | origin = "bottom" 21 | if opts.invert: 22 | origin = "top" 23 | 24 | tileset = MbtileSet(mbtiles=opts.file, outdir=opts.output, origin=origin) 25 | tileset.write_all() 26 | -------------------------------------------------------------------------------- /serve_mbtiles.py: -------------------------------------------------------------------------------- 1 | import tornado.ioloop 2 | import tornado.web 3 | import os 4 | from mbtiles import MbtileSet 5 | 6 | class MainHandler(tornado.web.RequestHandler): 7 | def get(self): 8 | self.write(''' 9 |

Sample URLs

10 | 15 | ''') 16 | 17 | class MbtilesHandler(tornado.web.RequestHandler): 18 | def initialize(self, ext, mbtiles): 19 | self.ext = ext 20 | self.mbtiles = mbtiles 21 | self.tileset = MbtileSet(mbtiles=mbtiles) 22 | 23 | def get(self, z, x, y): 24 | origin = self.get_arguments('origin') 25 | try: 26 | origin = origin[0] 27 | except IndexError: 28 | origin = 'bottom' 29 | 30 | if origin == 'top': 31 | # invert y axis to top origin 32 | ymax = 1 << int(z); 33 | y = ymax - int(y) - 1; 34 | 35 | tile = self.tileset.get_tile(z, x, y) 36 | if self.ext == 'png': 37 | self.set_header('Content-Type', 'image/png') 38 | self.write(tile.get_png()) 39 | elif self.ext == 'json': 40 | callback = self.get_arguments('callback') 41 | try: 42 | callback = callback[0] 43 | except IndexError: 44 | callback = None 45 | 46 | self.set_header('Content-Type', 'application/json') 47 | if callback: 48 | self.write("%s(%s)" % (callback, tile.get_json())) 49 | else: 50 | self.write(tile.get_json()) 51 | 52 | 53 | if __name__ == "__main__": 54 | urls = [(r"/", MainHandler),] 55 | 56 | thisdir = os.path.abspath(os.path.dirname(__file__)) 57 | tilesets = [ 58 | ('test', os.path.join(thisdir, 'data', 'road-trip-wilderness.mbtiles'), ['png','json'],), 59 | ] 60 | 61 | for t in tilesets: 62 | for ext in t[2]: 63 | urls.append( 64 | (r'/%s/([0-9]+)/([0-9]+)/([0-9]+).%s' % (t[0],ext), 65 | MbtilesHandler, 66 | {"ext": ext, "mbtiles": t[1]} 67 | ) 68 | ) 69 | 70 | application = tornado.web.Application(urls, debug=True) 71 | application.listen(8988) 72 | tornado.ioloop.IOLoop.instance().start() 73 | --------------------------------------------------------------------------------