├── .gitignore ├── AUTHORS ├── LICENSE ├── README.md ├── bin └── osm-tools └── lua └── osm-tools ├── network-profiles └── highway.lua ├── osm-to-db.lua ├── osmdb.lua ├── protocol-buffer-defs └── osm.proto ├── read-osmpbf.lua └── transform.lua /.gitignore: -------------------------------------------------------------------------------- 1 | .DS_Store 2 | -------------------------------------------------------------------------------- /AUTHORS: -------------------------------------------------------------------------------- 1 | Leyland, Geoff 2 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Copyright (c) 2015 Geoff Leyland 2 | 3 | Permission is hereby granted, free of charge, to any person obtaining a copy 4 | of this software and associated documentation files (the "Software"), to deal 5 | in the Software without restriction, including without limitation the rights 6 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 7 | copies of the Software, and to permit persons to whom the Software is 8 | furnished to do so, subject to the following conditions: 9 | 10 | The above copyright notice and this permission notice shall be included in 11 | all copies or substantial portions of the Software. 12 | 13 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 14 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 15 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 16 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 17 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 18 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN 19 | THE SOFTWARE. 20 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # lua-osm-tools - Tools for handling OSM data 2 | 3 | My set of tools for dealing with OSM data and converting it to routable 4 | road networks. The tools include: 5 | 6 | - osmdb: a format for storing OSM data pretty much verbatim in a database. 7 | xml and pbf are a awkward formats to work with so the first thing 8 | I do tends to be read it all into a sqlite db. 9 | 10 | - readosm-pbf: a library for reading osm pbf files (not xml). 11 | This is a bit of an experiment: 12 | because it gives LuaJIT a few more opportunities to compile traces it ends 13 | up about the same speed as [my binding](https://github.com/geoffleyland/lua-readosm) 14 | to [readosm](https://www.gaia-gis.it/fossil/readosm/index), 15 | even though readosm is *much* quicker if you're using it from C. 16 | The tools here will work with either readosm-pbf or lua-readosm. 17 | 18 | - osm-tools: a tool with subcommands to: 19 | 20 | - read-verbatim: dump a pbf file to a database. 21 | 22 | 23 | # osmdb 24 | 25 | osmdb is a simple db format for storing osm data. 26 | I'm sure this must already exist, right? 27 | 28 | In any case, it's not a geographic database 29 | - it's just a literal copy of the data in an OSM XML or PBF file. 30 | It's not really intended for you to add to the database once you've read it - 31 | it's more of a import-the-whole-map-once-then-read-it kind of database. 32 | 33 | Tables are: 34 | 35 | - elements - just an element id and type ("node", "way", or "relation") 36 | - info - element metadata: version, changeset, user, uid and timestamp. 37 | This is in a separate table from elements so it's easy to choose to not 38 | store metadata. 39 | - tags: tags for every element: a parent id, a key and a value. 40 | - locations: a latitude and longitude for node elements 41 | - nodes: ordered lists of node ids for way elements 42 | - members: ordered lists of members (role and id) for relations. 43 | 44 | `osmdb.create(dbname)` creates a new database and its tables. 45 | 46 | `osmdb.open(dbname)` opens and existing database. 47 | 48 | `db:begin()` begins a transaction. It's a good idea to `begin` before 49 | importing a lot of elements into the database. 50 | 51 | `db:commit()` commits a transaction, builds indexes and vaccuums the database. 52 | (This probably only works on SQLite). 53 | 54 | `db:insert_element(type, object)` inserts an element into the database. 55 | The type and object format are exactly what you get from either of the 56 | osm readers I've built, so you can just go: 57 | 58 | for type, element in f:lines() do 59 | db:insert_element(type, element) 60 | end 61 | 62 | to read a whole file into the database. This is what `osm-tools import` does. 63 | 64 | `db:elements(type, with_children)` is an iterator that returns all the 65 | elements in the database. 66 | You can use `type` to select a single type of element, 67 | for example to only read nodes. 68 | If `with_children` is true or the string `"with_children"` then: 69 | 70 | - a way will contain a `nodes` table containing a list of node objects. 71 | - a relation will contain a `members` table containing 72 | `{ role="role", member= }` 73 | elements, where the member is element objects. 74 | 75 | Otherwise: 76 | 77 | - a way will contain a `node_refs` table, with a list of node ids. 78 | - a relation will contain a `member_refs` table containing 79 | `{role="role", id=id }` elements. 80 | 81 | This format is intended to be exactly the same as what you get from the OSM 82 | readers. 83 | 84 | *Note* that the ids stored in the database are NOT THE SAME as OSM ids. 85 | OSM node, way and relation ids are *nearly*, but not actually, unique 86 | over the nodes, ways and relations. 87 | It's kind of a shame that they aren't. 88 | Since there are three types of OSM objects, 89 | the eids stored in this osmdb format are 4 * osm_id + type_code, where 90 | type_code is 0, 1 or 2 for nodes, ways and relations respectively. 91 | 92 | The objects returned from `db:elements` have both an id and an eid member. 93 | -------------------------------------------------------------------------------- /bin/osm-tools: -------------------------------------------------------------------------------- 1 | #! /usr/bin/env lua 2 | 3 | local unpack = table.unpack or unpack 4 | 5 | local HELP = 6 | [[ 7 | %s 8 | Do things to osm data. 9 | Possible commands are: 10 | ]] 11 | 12 | local function list_commands(C) 13 | for k in pairs(C) do io.stderr:write(" ", k, "\n") end 14 | end 15 | 16 | local COMMANDS 17 | 18 | COMMANDS = 19 | { 20 | help = 21 | { 22 | help = 23 | [[ 24 | %s %s [command] 25 | display all possible commands or help for a given command. 26 | ]], 27 | doit = function(command) 28 | if command then 29 | command = command:lower() 30 | if COMMANDS[command] then 31 | io.stderr:write(COMMANDS[command].help:format(arg[0], command)) 32 | else 33 | io.stderr:write("Unknown command, possible commands are:\n") 34 | list_commands(COMMANDS) 35 | end 36 | else 37 | io.stderr:write(HELP:format(arg[0])) 38 | list_commands(COMMANDS) 39 | end 40 | return true 41 | end 42 | }, 43 | 44 | import = 45 | { 46 | help = 47 | [[ 48 | %s %s 49 | Import an osm file to an osm database verbatim. The database is created. 50 | ]], 51 | doit = function(osmfile, dbfile) 52 | if not osmfile or not dbfile then 53 | return nil, "missing argument" 54 | end 55 | local osm_to_db = require"osm-tools.osm-to-db" 56 | local ok, message = pcall(osm_to_db.import, osmfile, dbfile) 57 | return ok, message 58 | end 59 | }, 60 | 61 | transform = 62 | { 63 | help = 64 | [[ 65 | %s %s 66 | Import an osm file to an osm database while filtering and transforming the data. 67 | ]], 68 | doit = function(osmfile, dbfile, transform_name) 69 | if not osmfile or not dbfile then 70 | return nil, "missing argument" 71 | end 72 | local transform = require"osm-tools.transform" 73 | 74 | local filter = (function() 75 | if transform_name:match("%.") then 76 | return require(transform_name) 77 | else 78 | return require("osm-tools.network-profiles."..transform_name) 79 | end 80 | end)() 81 | 82 | local ok, message = pcall(transform.transform, osmfile, dbfile, filter) 83 | return ok, message 84 | end 85 | }, 86 | 87 | } 88 | 89 | 90 | local command = COMMANDS[(arg[1] or "help"):lower()] 91 | if not command then 92 | io.stderr:write(("Unknown command '%s'.\n\n"):format(arg[1])) 93 | COMMANDS.help.doit() 94 | os.exit(1) 95 | end 96 | 97 | local ok, message = command.doit(unpack(arg, 2)) 98 | if not ok then 99 | io.stderr:write("Error: ", message, "\n\n") 100 | COMMANDS.help.doit(arg[1]) 101 | os.exit(1) 102 | else 103 | os.exit() 104 | end 105 | -------------------------------------------------------------------------------- /lua/osm-tools/network-profiles/highway.lua: -------------------------------------------------------------------------------- 1 | local scores = 2 | { 3 | highway = 4 | { 5 | abandoned = -1, 6 | bridleway = -1, 7 | construction = -1, 8 | crossing = -1, 9 | cycleway = -1, 10 | footway = -1, 11 | paper = -1, 12 | path = -1, 13 | ["path-disabled"] = -1, 14 | pedestrian = -1, 15 | platform = -1, 16 | proposed = -3, 17 | raceway = -1, 18 | rest_area = -1, 19 | service = -1, 20 | services = -1, 21 | steps = -1, 22 | subway = -1, 23 | tidal_path = -3, 24 | track = -3, 25 | traffic_signals = -1, 26 | ["unclassified;track"] = -1, 27 | undefined = -1, 28 | unknown = -1, 29 | unmarked_route = -1, 30 | }, 31 | vehicle = 32 | { 33 | no = -1, 34 | }, 35 | motorcar = 36 | { 37 | no = -1, 38 | }, 39 | motor_vehicle = 40 | { 41 | no = -1, 42 | }, 43 | psv = 44 | { 45 | yes = 2, 46 | }, 47 | type = 48 | { 49 | proposed = -3, 50 | }, 51 | service = 52 | { 53 | beach_access = -1, 54 | driveway = -1, 55 | ["Pylon road"] = -1, 56 | slipway = -1, 57 | yard = -1, 58 | }, 59 | building = 60 | { 61 | yes = -1, 62 | }, 63 | access = 64 | { 65 | no = -1, 66 | }, 67 | } 68 | 69 | 70 | local function reverse_in_place(t) 71 | local L = #t+1 72 | for i = 1, math.floor((L-1) / 2) do 73 | t[i], t[L-i] = t[L-i], t[i] 74 | end 75 | end 76 | 77 | 78 | local function add_name(name, list, map) 79 | if name and not map[name] then 80 | list[#list+1] = name 81 | map[name] = true 82 | end 83 | end 84 | 85 | 86 | local function filter_highways(type, o) 87 | if type == "way" then 88 | if not o.tags.highway then return end 89 | local s = 0 90 | for field, value in pairs(o.tags) do 91 | local f = scores[field] 92 | if f then 93 | s = s + (f[value] or 0) 94 | end 95 | end 96 | if s >= 0 then 97 | local new_tags = {} 98 | 99 | local oneway, reverse = o.tags.oneway 100 | if oneway and oneway == "-1" then 101 | reverse = true 102 | oneway = "yes" 103 | end 104 | oneway = (oneway and oneway:lower() == "yes") or 105 | (o.tags.junction and o.tags.junction:lower() == "roundabout") 106 | new_tags.oneway = oneway 107 | if reverse then 108 | reverse_in_place(o.node_refs) 109 | end 110 | 111 | local level 112 | if o.tags.bridge and o.tags.bridge:lower() ~= "no" then level = 1 end 113 | if o.tags.tunnel and o.tags.tunnel:lower() ~= "no" then level = -1 end 114 | new_tags.level = level 115 | 116 | local names, map = {}, {} 117 | add_name(o.tags.name, names, map) 118 | add_name(o.tags.alt_name, names, map) 119 | add_name(o.tags.old_name, names, map) 120 | add_name(o.tags.ref, names, map) 121 | add_name(o.tags.old_ref, names, map) 122 | local name = table.concat(names, ";") 123 | new_tags.name = name ~= "" and name or nil 124 | 125 | 126 | new_tags.speed = o.tags.maxspeed or "50" 127 | 128 | return { id = o.id, node_refs = o.node_refs, tags = new_tags } 129 | end 130 | elseif type == "node" then 131 | local new_tags = {} 132 | local hw = o.tags.highway 133 | if hw == "traffic_signals" then new_tags.lights = 1 end 134 | if hw == "give_way" then new_tags.give_way = 1 end 135 | if hw == "stop" then new_tags.stop = 1 end 136 | if o.tags.traffic_calming then new_tags.bump = 1 end 137 | return { id = o.id, latitude = o.latitude, longitude = o.longitude, tags = new_tags } 138 | end 139 | end 140 | 141 | return filter_highways 142 | -------------------------------------------------------------------------------- /lua/osm-tools/osm-to-db.lua: -------------------------------------------------------------------------------- 1 | local ok, lro = pcall(require, "osm-tools.read-osmpbf") 2 | lro = ok and lro or nil 3 | local ok, cro = pcall(require, "readosm") 4 | cro = ok and cro or nil 5 | local osmdb = require"osm-tools.osmdb" 6 | 7 | 8 | ------------------------------------------------------------------------------ 9 | 10 | local db, count, start 11 | 12 | local function count_one() 13 | count = count + 1 14 | if count % 10000 == 0 then 15 | local elapsed = os.clock() - start 16 | io.stderr:write(("Read %5dk elements in %4ds (%3dk elements/s, %3.1fMB)... \r"): 17 | format(count / 1000, elapsed, count / elapsed / 1000, collectgarbage('count') / 1024)) 18 | end 19 | end 20 | 21 | 22 | local function read_element(type, element) 23 | db:insert_element(type, element) 24 | count_one() 25 | end 26 | 27 | 28 | local function import(osmname, db_or_dbname) 29 | if type(db_or_dbname) == "string" then 30 | db = osmdb.create(db_or_dbname) 31 | else 32 | db = db_or_dbname 33 | end 34 | assert(db, "Couldn't open db.") 35 | db:begin() 36 | 37 | local ro = (function() 38 | if osmname:match(".pbf$") then 39 | return assert(lro or cro, "You need to install read-osmpbf or lua-readosm to read PBF OSM files") 40 | else 41 | return assert(cro, "You need to install lua-readosm to read XML-format OSM files") 42 | end 43 | end)() 44 | 45 | local f = ro.open(osmname) 46 | count = 0 47 | start = os.clock() 48 | 49 | if f.lines then 50 | for type, element in f:lines() do 51 | read_element(type, element) 52 | end 53 | else 54 | f:parse(read_element) 55 | end 56 | 57 | f:close() 58 | local elapsed = os.clock() - start 59 | io.stderr:write(("Read %5dk elements in %4ds (%3dk elements/s). \n"): 60 | format(count / 1000, elapsed, count / elapsed /1000)) 61 | 62 | io.stderr:write("Building indexes...\n") 63 | db:commit() 64 | io.stderr:write(("Finished in %4ds.\n"): 65 | format(os.clock() - start)) 66 | 67 | return db 68 | end 69 | 70 | 71 | ------------------------------------------------------------------------------ 72 | 73 | return { import = import } 74 | 75 | ------------------------------------------------------------------------------ 76 | -------------------------------------------------------------------------------- /lua/osm-tools/osmdb.lua: -------------------------------------------------------------------------------- 1 | local patengi = require"patengi" 2 | 3 | ------------------------------------------------------------------------------ 4 | -- IDs aren't unique across nodes, types and ways. 5 | -- It's tempting to wonder why on earth they did that, but let's not dwell on 6 | -- it. 7 | -- Let's just create a unique ID that's 4*osm_id + type_code. 8 | -- We'll call in an "eid" for "element id". My first idea "uid" (for 9 | -- "unique_id") is already taken by the user_id field 10 | 11 | local type_code = 12 | { 13 | node = 0, 14 | way = 1, 15 | relation = 2, 16 | } 17 | 18 | local NODE_CODE = type_code.node 19 | 20 | 21 | ------------------------------------------------------------------------------ 22 | 23 | local osmdb = {} 24 | osmdb.__index = osmdb 25 | 26 | 27 | local function build_queries(db) 28 | return 29 | { 30 | insert_element = db:prepare[[ 31 | INSERT INTO elements 32 | (eid, type) 33 | VALUES 34 | (:eid, :type); 35 | ]], 36 | insert_info = db:prepare[[ 37 | INSERT INTO info 38 | (eid, version, changeset, user, uid, timestamp) 39 | VALUES 40 | (:eid, :version, :changeset, :user, :uid, :timestamp); 41 | ]], 42 | insert_location = db:prepare[[ 43 | INSERT INTO locations 44 | (eid, latitude, longitude) 45 | VALUES 46 | (:eid, :latitude, :longitude); 47 | ]], 48 | insert_node = db:prepare[[ 49 | INSERT INTO nodes 50 | (parent_eid, ord, node_eid) 51 | VALUES 52 | (:parent_eid, :ord, :node_eid); 53 | ]], 54 | insert_member = db:prepare[[ 55 | INSERT INTO members 56 | (parent_eid, ord, role, member_eid) 57 | VALUES 58 | (:parent_eid, :ord, :role, :member_eid); 59 | ]], 60 | insert_tag = db:prepare[[ 61 | INSERT INTO tags 62 | (parent_eid, key, value) 63 | VALUES 64 | (:parent_eid, :key, :value); 65 | ]], 66 | 67 | select_all_elements = db:prepare[[ 68 | SELECT eid, type 69 | FROM elements; 70 | ]], 71 | select_some_elements = db:prepare[[ 72 | SELECT eid, type 73 | FROM elements 74 | WHERE type = :1; 75 | ]], 76 | select_element = db:prepare[[ 77 | SELECT eid, type 78 | FROM elements 79 | WHERE eid = :eid; 80 | ]], 81 | select_info = db:prepare[[ 82 | SELECT version, changeset, user, uid, timestamp 83 | FROM info 84 | WHERE eid = :eid; 85 | ]], 86 | select_location = db:prepare[[ 87 | SELECT latitude, longitude 88 | FROM locations 89 | WHERE eid = :eid; 90 | ]], 91 | select_tags = db:prepare[[ 92 | SELECT key, value 93 | FROM tags 94 | WHERE parent_eid = :parent_eid; 95 | ]], 96 | select_nodes = db:prepare[[ 97 | SELECT node_eid 98 | FROM nodes 99 | WHERE parent_eid = :parent_eid 100 | ORDER BY ord; 101 | ]], 102 | select_members = db:prepare[[ 103 | SELECT role, type, member_eid 104 | FROM members 105 | WHERE parent_eid = :parent_eid 106 | ORDER BY ord; 107 | ]], 108 | } 109 | end 110 | 111 | 112 | function osmdb:new(db) 113 | return setmetatable({ db = db, Q = build_queries(db), values = {} }, osmdb) 114 | end 115 | 116 | 117 | function osmdb:close() 118 | self:commit() 119 | self.db:close() 120 | end 121 | 122 | 123 | ------------------------------------------------------------------------------ 124 | 125 | function osmdb.create(filename) 126 | os.remove(filename) 127 | local db = patengi.open(filename) 128 | 129 | db:exec[[ 130 | CREATE TABLE elements 131 | ( 132 | eid INTEGER PRIMARY KEY, 133 | type VARCHAR 134 | );]] 135 | db:exec[[ 136 | CREATE TABLE info 137 | ( 138 | eid INTEGER PRIMARY KEY, 139 | version INTEGER, 140 | changeset INTEGER, 141 | user VARCHAR, 142 | uid INTEGER, 143 | timestamp DATETIME 144 | );]] 145 | db:exec[[ 146 | CREATE TABLE tags 147 | ( 148 | parent_eid INTEGER, 149 | key VARCHAR, 150 | value VARCHAR 151 | );]] 152 | db:exec[[ 153 | CREATE TABLE locations 154 | ( 155 | eid INTEGER PRIMARY KEY, 156 | latitude REAL, 157 | longitude REAL 158 | );]] 159 | db:exec[[ 160 | CREATE TABLE nodes 161 | ( 162 | parent_eid INTEGER, 163 | ord INTEGER, 164 | node_eid INTEGER 165 | );]] 166 | db:exec[[ 167 | CREATE TABLE members 168 | ( 169 | parent_eid INTEGER, 170 | ord INTEGER, 171 | role VARCHAR, 172 | type VARCHAR, 173 | member_eid INTEGER 174 | );]] 175 | 176 | return osmdb:new(db) 177 | end 178 | 179 | 180 | function osmdb.open(filename) 181 | return osmdb:new(patengi.open(filename)) 182 | end 183 | 184 | 185 | ------------------------------------------------------------------------------ 186 | 187 | 188 | function osmdb:begin() 189 | self.db:exec("BEGIN;") 190 | end 191 | 192 | 193 | function osmdb:commit() 194 | self.db:exec("COMMIT;") 195 | self.db:exec("CREATE INDEX ni ON nodes (parent_eid, ord);") 196 | self.db:exec("CREATE INDEX mi ON members (parent_eid, ord);") 197 | self.db:exec("CREATE INDEX ti ON tags (parent_eid);") 198 | self.db:exec("CREATE INDEX tki ON tags (key);") 199 | self.db:exec("VACUUM;") 200 | self.db:exec("ANALYZE;") 201 | end 202 | 203 | 204 | function osmdb:_insert_element(o, type) 205 | local values = self.values 206 | values.eid = tonumber(o.id) * 4 + type_code[type] 207 | values.type = type 208 | self.Q.insert_element:exec(values) 209 | 210 | values.parent_eid = values.eid 211 | for k, v in pairs(o.tags) do 212 | values.key = k 213 | values.value = v 214 | self.Q.insert_tag:exec(values) 215 | end 216 | 217 | if o.version or o.changeset or o.user or o.uid or o.timestamp then 218 | values.version = tonumber(o.version) 219 | values.changeset = tonumber(o.changeset) 220 | values.user = o.user 221 | values.uid = tonumber(o.uid) 222 | values.timestamp = o.timestamp and o.timestamp 223 | self.Q.insert_info:exec(values) 224 | end 225 | 226 | return values.eid 227 | end 228 | 229 | 230 | function osmdb:insert_node(o) 231 | self:_insert_element(o, "node") 232 | 233 | local values = self.values 234 | values.latitude = tonumber(o.latitude) 235 | values.longitude = tonumber(o.longitude) 236 | self.Q.insert_location:exec(values) 237 | end 238 | 239 | 240 | function osmdb:insert_way(o) 241 | local values = self.values 242 | values.parent_eid = self:_insert_element(o, "way") 243 | 244 | for i, n in ipairs(o.node_refs) do 245 | values.ord = i 246 | values.node_eid = tonumber(n) * 4 + NODE_CODE 247 | self.Q.insert_node:exec(values) 248 | end 249 | end 250 | 251 | 252 | function osmdb:insert_relation(o) 253 | local values = self.values 254 | values.parent_eid = self:_insert_element(o, "relation") 255 | 256 | for i, n in ipairs(o.member_refs) do 257 | values.ord = i 258 | values.role = n.role 259 | values.type = n.member_type 260 | values.member_eid = tonumber(n.id) * 4 + type_code[n.member_type] 261 | self.Q.insert_member:exec(values) 262 | end 263 | end 264 | 265 | 266 | local element_inserters = 267 | { 268 | node = osmdb.insert_node, 269 | way = osmdb.insert_way, 270 | relation = osmdb.insert_relation, 271 | } 272 | 273 | function osmdb:insert_element(type, o) 274 | local f = element_inserters[type] 275 | if f then 276 | f(self, o) 277 | else 278 | io.stderr:write(("Unknown data type '%s'\n"):format(type)) 279 | end 280 | end 281 | 282 | 283 | ------------------------------------------------------------------------------ 284 | 285 | function osmdb:read_element(element, with_children) 286 | with_children = with_children == "with_children" or with_children == true 287 | 288 | element.id = math.floor(element.eid / 4) 289 | 290 | local version, changeset, user, uid, timestamp = 291 | self.Q.select_info:uexec(element.eid) 292 | if version then 293 | element.version = version 294 | element.changeset = changeset 295 | element.user = user 296 | element.uid = uid 297 | element.timestamp = timestamp 298 | end 299 | 300 | element.tags = {} 301 | for key, value in self.Q.select_tags:urows(element.eid) do 302 | element.tags[key] = value 303 | end 304 | 305 | if element.type == "node" then 306 | element.latitude, element.longitude = 307 | self.Q.select_location:uexec(element.eid) 308 | elseif element.type == "way" then 309 | if with_children then 310 | element.nodes = {} 311 | for node_eid in self.Q.select_nodes:urows(element.eid) do 312 | local n = self.Q.select_element:nexec(node_eid, "element") 313 | element.nodes[#element.nodes+1] = self:read_element(n) 314 | end 315 | else 316 | element.node_refs = {} 317 | for node_eid in self.Q.select_nodes:urows(element.eid) do 318 | element.node_refs[#element.node_refs+1] = math.floor(node_eid / 4) 319 | end 320 | end 321 | elseif element.type == "relation" then 322 | if with_children then 323 | element.members = {} 324 | for role, type, member_eid in self.Q.select_members:urows(element.eid) do 325 | local m = self.Q.select_element:nexec(member_eid) 326 | if m then 327 | element.members[#element.members+1] = 328 | { role = role, type = type, member = self:read_element(m) } 329 | end 330 | end 331 | else 332 | element.member_refs = {} 333 | for role, type, member_eid in self.Q.select_members:urows(element.eid) do 334 | element.member_refs[#element.member_refs+1] = 335 | { role = role, type = type, id = math.floor(member_eid / 4), eid = member_eid } 336 | end 337 | end 338 | end 339 | return element 340 | end 341 | 342 | 343 | local function get_iterator(self, type) 344 | if not type or type == "" then 345 | return self.Q.select_all_elements:nrows() 346 | else 347 | return self.Q.select_some_elements:nrows(type) 348 | end 349 | end 350 | 351 | 352 | function osmdb:elements(type, with_children) 353 | return coroutine.wrap(function() 354 | for element in get_iterator(self, type) do 355 | self:read_element(element, with_children) 356 | coroutine.yield(element) 357 | end 358 | end) 359 | end 360 | 361 | 362 | ------------------------------------------------------------------------------ 363 | 364 | return osmdb 365 | 366 | ------------------------------------------------------------------------------ 367 | -------------------------------------------------------------------------------- /lua/osm-tools/protocol-buffer-defs/osm.proto: -------------------------------------------------------------------------------- 1 | message BlobHeader 2 | { 3 | required string type = 1; 4 | optional bytes indexdata = 2; 5 | required int32 datasize = 3; 6 | } 7 | 8 | message Blob 9 | { 10 | optional bytes raw = 1; // No compression 11 | optional int32 raw_size = 2; // Only set when compressed, to the uncompressed size 12 | optional bytes zlib_data = 3; 13 | // optional bytes lzma_data = 4; // PROPOSED. 14 | // optional bytes OBSOLETE_bzip2_data = 5; // Deprecated. 15 | } 16 | 17 | message HeaderBlock { 18 | optional HeaderBBox bbox = 1; 19 | /* Additional tags to aid in parsing this dataset */ 20 | repeated string required_features = 4; 21 | repeated string optional_features = 5; 22 | 23 | optional string writingprogram = 16; 24 | optional string source = 17; // From the bbox field. 25 | 26 | /* Tags that allow continuing an Osmosis replication */ 27 | 28 | // replication timestamp, expressed in seconds since the epoch, 29 | // otherwise the same value as in the "timestamp=..." field 30 | // in the state.txt file used by Osmosis 31 | optional int64 osmosis_replication_timestamp = 32; 32 | 33 | // replication sequence number (sequenceNumber in state.txt) 34 | optional int64 osmosis_replication_sequence_number = 33; 35 | 36 | // replication base URL (from Osmosis' configuration.txt file) 37 | optional string osmosis_replication_base_url = 34; 38 | } 39 | 40 | message HeaderBBox 41 | { 42 | required sint64 left = 1; 43 | required sint64 right = 2; 44 | required sint64 top = 3; 45 | required sint64 bottom = 4; 46 | } 47 | 48 | message StringTable 49 | { 50 | repeated bytes s = 1; 51 | } 52 | 53 | message PrimitiveBlock 54 | { 55 | required StringTable stringtable = 1; 56 | repeated PrimitiveGroup primitivegroup = 2; 57 | 58 | // Granularity, units of nanodegrees, used to store coordinates in this block 59 | optional int32 granularity = 17 [default=100]; 60 | 61 | // Offset value between the output coordinates coordinates and the granularity grid, in units of nanodegrees. 62 | optional int64 lat_offset = 19 [default=0]; 63 | optional int64 lon_offset = 20 [default=0]; 64 | 65 | // Granularity of dates, normally represented in units of milliseconds since the 1970 epoch. 66 | optional int32 date_granularity = 18 [default=1000]; 67 | 68 | 69 | // Proposed extension: 70 | //optional BBox bbox = XX; 71 | } 72 | 73 | message PrimitiveGroup 74 | { 75 | repeated Node nodes = 1; 76 | optional DenseNodes dense = 2; 77 | repeated Way ways = 3; 78 | repeated Relation relations = 4; 79 | // repeated ChangeSet changesets = 5; 80 | } 81 | 82 | message Info 83 | { 84 | optional int32 version = 1 [default = -1]; 85 | optional int32 timestamp = 2; 86 | optional int64 changeset = 3; 87 | optional int32 uid = 4; 88 | optional int32 user_sid = 5; // String IDs 89 | 90 | // The visible flag is used to store history information. It indicates that 91 | // the current object version has been created by a delete operation on the 92 | // OSM API. 93 | // When a writer sets this flag, it MUST add a required_features tag with 94 | // value "HistoricalInformation" to the HeaderBlock. 95 | // If this flag is not available for some object it MUST be assumed to be 96 | // true if the file has the required_features tag "HistoricalInformation" 97 | // set. 98 | optional bool visible = 6; 99 | } 100 | 101 | message Node 102 | { 103 | required sint64 id = 1; 104 | required sint64 lat = 7; 105 | required sint64 lon = 8; 106 | repeated uint32 keys = 9 [packed = true]; // Denote strings 107 | repeated uint32 vals = 10 [packed = true];// Denote strings 108 | optional Info info = 11; // Contains metadata 109 | } 110 | 111 | message DenseNodes 112 | { 113 | repeated sint64 id = 1 [packed = true]; // DELTA coded 114 | 115 | //repeated Info info = 4; 116 | optional DenseInfo denseinfo = 5; 117 | 118 | repeated sint64 lat = 8 [packed = true]; // DELTA coded 119 | repeated sint64 lon = 9 [packed = true]; // DELTA coded 120 | 121 | // Special packing of keys and vals into one array. May be empty if all nodes in this block are tagless. 122 | repeated int32 keys_vals = 10 [packed = true]; 123 | } 124 | 125 | message DenseInfo 126 | { 127 | repeated int32 version = 1 [packed = true]; 128 | repeated sint64 timestamp = 2 [packed = true]; // DELTA coded 129 | repeated sint64 changeset = 3 [packed = true]; // DELTA coded 130 | repeated sint32 uid = 4 [packed = true]; // DELTA coded 131 | repeated sint32 user_sid = 5 [packed = true]; // String IDs for usernames. DELTA coded 132 | 133 | // The visible flag is used to store history information. It indicates that 134 | // the current object version has been created by a delete operation on the 135 | // OSM API. 136 | // When a writer sets this flag, it MUST add a required_features tag with 137 | // value "HistoricalInformation" to the HeaderBlock. 138 | // If this flag is not available for some object it MUST be assumed to be 139 | // true if the file has the required_features tag "HistoricalInformation" 140 | // set. 141 | repeated bool visible = 6 [packed = true]; 142 | } 143 | 144 | message Way 145 | { 146 | required int64 id = 1; 147 | // Parallel arrays. 148 | repeated uint32 keys = 2 [packed = true]; 149 | repeated uint32 vals = 3 [packed = true]; 150 | 151 | optional Info info = 4; 152 | 153 | repeated sint64 refs = 8 [packed = true]; // DELTA coded 154 | } 155 | 156 | message Relation 157 | { 158 | enum MemberType 159 | { 160 | NODE = 0; 161 | WAY = 1; 162 | RELATION = 2; 163 | } 164 | required int64 id = 1; 165 | 166 | // Parallel arrays. 167 | repeated uint32 keys = 2 [packed = true]; 168 | repeated uint32 vals = 3 [packed = true]; 169 | 170 | optional Info info = 4; 171 | 172 | // Parallel arrays 173 | repeated int32 roles_sid = 8 [packed = true]; 174 | repeated sint64 memids = 9 [packed = true]; // DELTA encoded 175 | repeated MemberType types = 10 [packed = true]; 176 | } -------------------------------------------------------------------------------- /lua/osm-tools/read-osmpbf.lua: -------------------------------------------------------------------------------- 1 | local zlib = require"zlib" 2 | local pb = require"pb" 3 | local osmpbf = require"osm-tools.protocol-buffer-defs.osm" 4 | 5 | 6 | ------------------------------------------------------------------------------ 7 | -- Read a file header 8 | 9 | local acceptable_features = 10 | { 11 | ["OsmSchema-V0.6"] = true, 12 | ["DenseNodes"] = true, 13 | } 14 | 15 | local function read_header(data) 16 | local header = osmpbf.HeaderBlock() 17 | header:Parse(data) 18 | for _, v in ipairs(header.required_features) do 19 | if not acceptable_features[v] then 20 | error(("osmpbfread: unsupported feature '%s'"):format(v)) 21 | end 22 | end 23 | end 24 | 25 | 26 | ------------------------------------------------------------------------------ 27 | -- For what it's worth, we try to do as little as possible when reading a file 28 | -- so that we don't spend too much time pulling data the user doesn't want 29 | -- into lua-land. 30 | -- So instead of populating a table with data fields, we give a table a 31 | -- a metatable that will go and fetch the data when it's asked for. 32 | -- It actually seems to work (if you're not looking at all the data), 33 | -- and speeds things up a little without costing much in the worst case. 34 | 35 | local function id_field(msg) return msg.id end 36 | local function version_field(msg) return msg.info and msg.info.version or nil end 37 | local function changeset_field(msg) return msg.info and msg.info.changeset or nil end 38 | local function uid_field(msg) return msg.info and msg.info.uid or nil end 39 | local function user_field(msg, block) 40 | return msg.info and msg.info.user_sid and block.stringtable.s[msg.info.user_sid+1] or nil 41 | end 42 | -- We copy readosm and return a timestamp string. 43 | -- This avoids complications with UTC and local time. 44 | local function timestamp_field(msg) 45 | return msg.info and os.date("!%Y-%m-%dT%H:%M:%SZ", msg.info.timestamp) or nil 46 | end 47 | 48 | 49 | local NO_TAGS = setmetatable({}, { __newindex = function() error("read only table") end }) 50 | 51 | local function tag_field(msg, block) 52 | if not msg.keys then return NO_TAGS end 53 | 54 | local tags = {} 55 | for i, k in pairs(msg.keys) do 56 | tags[block.stringtable.s[k+1]] = block.stringtable.s[msg.vals[i]+1] 57 | end 58 | return tags 59 | end 60 | 61 | 62 | local node_fields = 63 | { 64 | id = id_field, 65 | version = version_field, 66 | timestamp = timestamp_field, 67 | changeset = changeset_field, 68 | uid = uid_field, 69 | user = user_field, 70 | latitude = function(msg, block) 71 | return (block.lat_offset + msg.lat * block.granularity) * 1e-9 72 | end, 73 | longitude = function(msg, block) 74 | return (block.lon_offset + msg.lon * block.granularity) * 1e-9 75 | end, 76 | } 77 | 78 | local way_fields = 79 | { 80 | id = id_field, 81 | tags = tag_field, 82 | version = version_field, 83 | timestamp = timestamp_field, 84 | changeset = changeset_field, 85 | uid = uid_field, 86 | user = user_field, 87 | node_refs = function(msg) 88 | local ids = {} 89 | local id = 0 90 | for i, v in ipairs(msg.refs) do 91 | id = id + v 92 | ids[i] = id 93 | end 94 | return ids 95 | end, 96 | } 97 | 98 | local relation_fields = 99 | { 100 | id = id_field, 101 | tags = tag_field, 102 | version = version_field, 103 | timestamp = timestamp_field, 104 | changeset = changeset_field, 105 | uid = uid_field, 106 | user = user_field, 107 | member_refs = function(msg, block) 108 | local m = {} 109 | local id = 0 110 | for i, v in ipairs(msg.memids) do 111 | id = id + v 112 | m[i] = 113 | { 114 | id = id, 115 | role = block.stringtable.s[msg.roles_sid[i]+1], 116 | member_type = msg.types[i]:lower() 117 | } 118 | end 119 | return m 120 | end, 121 | } 122 | 123 | local MESSAGE_KEY, BLOCK_KEY = {}, {} 124 | 125 | local function make_metatable(fields) 126 | return 127 | { 128 | __index = function (o, k) 129 | local f = fields[k] 130 | local v = f and f(o[MESSAGE_KEY], o[BLOCK_KEY], o) or nil 131 | o[k] = v 132 | return v 133 | end 134 | } 135 | end 136 | 137 | 138 | local node_mt = make_metatable(node_fields) 139 | local way_mt = make_metatable(way_fields) 140 | local relation_mt = make_metatable(relation_fields) 141 | 142 | 143 | ------------------------------------------------------------------------------ 144 | 145 | local function no_tags() return NO_TAGS end 146 | 147 | local function some_tags(t, _, o) 148 | local tags = {} 149 | while true do 150 | local si = t.indexes[t.string_index] 151 | if not si or si == 0 then break end 152 | tags[t.strings[si+1]] = t.strings[t.indexes[t.string_index+1]+1] 153 | t.string_index = t.string_index + 2 154 | end 155 | return tags 156 | end 157 | 158 | local dense_mt_no_tags = make_metatable{ tags = no_tags } 159 | local dense_mt_tags = make_metatable{ tags = some_tags } 160 | 161 | local function read_dense_nodes(block, elements, config) 162 | local id = 0 163 | local lat, lon = block.lat_offset * 1e-9, block.lon_offset * 1e-9 164 | local dense = elements.dense 165 | local granularity = block.granularity * 1e-9 166 | local date_granularity = block.date_granularity * 1e-3 167 | local keys_vals = dense.keys_vals 168 | local next_string_index = 1 169 | 170 | local tag_info = keys_vals and { indexes = keys_vals, strings = block.stringtable.s } or nil 171 | local dense_mt = keys_vals and dense_mt_tags or dense_mt_no_tags 172 | 173 | -- There are two copies of this code, one with, and one without node info. 174 | -- Handling info is slow for a number of reasons: 175 | -- - it seems like initialising a larger table takes longer (possibly 176 | -- because the table is more than 4 entries) 177 | -- - getting the user string takes a while 178 | -- - the delta encoding 179 | -- - info is optional anyway, so if we didn't have two loops, we'd have to 180 | -- check for it every time through the loop (and the JIT might not be able 181 | -- to remove the if?) 182 | -- So we ask the user if they want info, and we run a different loop 183 | -- if they do or if they don't or there's no info anyway. 184 | if not (config.info and dense.denseinfo) then 185 | for i, v in ipairs(dense.id) do 186 | id = id + v 187 | lat = lat + dense.lat[i] 188 | lon = lon + dense.lon[i] 189 | if keys_vals then 190 | tag_info.string_index = next_string_index 191 | while true do 192 | local si = keys_vals[next_string_index] 193 | if not si or si == 0 then break end 194 | next_string_index = next_string_index + 2 195 | end 196 | next_string_index = next_string_index + 1 197 | end 198 | local n = 199 | { 200 | id = id, 201 | latitude = lat * granularity, 202 | longitude = lon * granularity, 203 | [MESSAGE_KEY] = tag_info, 204 | } 205 | coroutine.yield("node", setmetatable(n, dense_mt)) 206 | end 207 | else 208 | local info = dense.denseinfo 209 | local timestamp, changeset, uid, user_sid = 0, 0, 0, 0 210 | for i, v in ipairs(dense.id) do 211 | id = id + v 212 | lat = lat + dense.lat[i] 213 | lon = lon + dense.lon[i] 214 | if keys_vals then 215 | tag_info.string_index = next_string_index 216 | while true do 217 | local si = keys_vals[next_string_index] 218 | if not si or si == 0 then break end 219 | next_string_index = next_string_index + 2 220 | end 221 | next_string_index = next_string_index + 1 222 | end 223 | timestamp = timestamp + info.timestamp[i] 224 | changeset = changeset + info.changeset[i] 225 | uid = uid + info.uid[i] 226 | user_sid = user_sid + info.user_sid[i] 227 | local n = 228 | { 229 | id = id, 230 | latitude = lat * granularity, 231 | longitude = lon * granularity, 232 | version = info and info.version[i], 233 | timestamp = os.date("!%Y-%m-%dT%H:%M:%SZ", timestamp * date_granularity), 234 | changeset = changeset, 235 | uid = uid, 236 | user = block.stringtable.s[user_sid+1], 237 | [MESSAGE_KEY] = tag_info, 238 | } 239 | coroutine.yield("node", setmetatable(n, dense_mt)) 240 | end 241 | end 242 | end 243 | 244 | 245 | local function make_element(mt, msg, block) 246 | return setmetatable({ [MESSAGE_KEY] = msg, [BLOCK_KEY] = block }, mt) 247 | end 248 | 249 | local function read_elements(data, config) 250 | local block = osmpbf.PrimitiveBlock() 251 | block:Parse(data) 252 | 253 | if block.primitivegroup then for _, elements in ipairs(block.primitivegroup) do 254 | if elements.nodes and config.nodes then for _, n in ipairs(elements.nodes) do 255 | coroutine.yield("node", make_element(node_mt, n, block)) 256 | end end 257 | if elements.ways and config.ways then for _, w in ipairs(elements.ways) do 258 | coroutine.yield("way", make_element(way_mt, w, block)) 259 | end end 260 | if elements.relations and config.relations then for _, r in ipairs(elements.relations) do 261 | coroutine.yield("relation", make_element(relation_mt, r, block)) 262 | end end 263 | 264 | if elements.dense and config.nodes then 265 | read_dense_nodes(block, elements, config) 266 | end 267 | 268 | end end 269 | end 270 | 271 | 272 | ------------------------------------------------------------------------------ 273 | 274 | local readers = 275 | { 276 | OSMHeader = read_header, 277 | OSMData = read_elements, 278 | } 279 | 280 | 281 | local function read_blob(fin, config) 282 | local length_data = fin:read(4) 283 | if not length_data then return end 284 | 285 | local blob_length = 0 286 | for i = 1, 4 do 287 | blob_length = blob_length * 2^8 + length_data:byte(i) 288 | end 289 | 290 | local header = osmpbf.BlobHeader() 291 | header:Parse(fin:read(blob_length)) 292 | 293 | local reader = readers[header.type] 294 | if not reader then 295 | error(("osmpbfread: unknown datatype '%s'"):format(header.type)) 296 | end 297 | 298 | local blob = osmpbf.Blob() 299 | blob:Parse(fin:read(header.datasize)) 300 | 301 | local blob_data = blob.raw and blob.raw or zlib.inflate()(blob.zlib_data) 302 | reader(blob_data, config) 303 | 304 | return true 305 | end 306 | 307 | 308 | ------------------------------------------------------------------------------ 309 | 310 | local osmpbf_file_mt = 311 | { 312 | __index = 313 | { 314 | lines = function(t) 315 | return coroutine.wrap(function() while read_blob(t.file, t.config) do end end) 316 | end, 317 | close = function(t) 318 | t.file:close() 319 | end 320 | } 321 | } 322 | 323 | 324 | local no_config = 325 | { 326 | nodes = true, 327 | ways = true, 328 | relations = true, 329 | info = true, 330 | } 331 | 332 | 333 | local function open(filename, what) 334 | what = what and what:lower() 335 | config = not what and no_config or 336 | { 337 | nodes = what:find("node"), 338 | ways = what:find("way"), 339 | relations = what:find("relation"), 340 | info = what:find("info"), 341 | } 342 | local file = assert(io.open(filename, "r")) 343 | return setmetatable({ file = file, config = config }, osmpbf_file_mt) 344 | end 345 | 346 | 347 | ------------------------------------------------------------------------------ 348 | 349 | return { open = open } 350 | 351 | ------------------------------------------------------------------------------ 352 | -------------------------------------------------------------------------------- /lua/osm-tools/transform.lua: -------------------------------------------------------------------------------- 1 | local ok, lro = pcall(require, "osm-tools.read-osmpbf") 2 | lro = ok and lro or nil 3 | local ok, cro = pcall(require, "readosm") 4 | cro = ok and cro or nil 5 | local osmdb = require"osm-tools.osmdb" 6 | 7 | 8 | ------------------------------------------------------------------------------ 9 | 10 | local db, count, start, transformer 11 | local wanted_node_ids 12 | 13 | local function count_one() 14 | count = count + 1 15 | if count % 100000 == 0 then 16 | local elapsed = os.clock() - start 17 | io.stderr:write(("Read %5dk elements in %4ds (%3dk elements/s, %3.1fMB)... \r"): 18 | format(count / 1000, elapsed, count / elapsed / 1000, collectgarbage('count') / 1024)) 19 | end 20 | end 21 | 22 | 23 | local function read_way(type, element) 24 | element = transformer(type, element) 25 | if element then 26 | for _, n in ipairs(element.node_refs) do 27 | wanted_node_ids[tonumber(n)] = true 28 | end 29 | db:insert_element(type, element) 30 | end 31 | count_one() 32 | end 33 | 34 | 35 | local function read_node(type, element) 36 | if wanted_node_ids[tonumber(element.id)] then 37 | element = transformer(type, element) or element 38 | db:insert_element(type, element) 39 | end 40 | count_one() 41 | end 42 | 43 | 44 | local function read_relation(type, element) 45 | local element = transformer(type, element) 46 | if element then 47 | db:insert_element(type, element) 48 | end 49 | count_one() 50 | end 51 | 52 | 53 | local function read_osm(osmname, db_or_dbname) 54 | start = os.clock() 55 | 56 | if type(db_or_dbname) == "string" then 57 | db = osmdb.create(db_or_dbname) 58 | else 59 | db = db_or_dbname 60 | end 61 | assert(db, "Couldn't open db.") 62 | db:begin() 63 | 64 | wanted_node_ids = {} 65 | count = 0 66 | 67 | local ro = (function() 68 | if osmname:match(".pbf$") then 69 | return assert(lro or cro, "You need to install read-osmpbf or lua-readosm to read PBF OSM files") 70 | else 71 | return assert(cro, "You need to install lua-readosm to read XML-format OSM files") 72 | end 73 | end)() 74 | 75 | local f = ro.open(osmname, "ways") 76 | 77 | if f.lines then 78 | for t, e in f:lines() do read_way(t, e) end 79 | f:close() 80 | 81 | f = ro.open(osmname, "nodes") 82 | for t, e in f:lines() do read_node(t, e) end 83 | f:close() 84 | 85 | f = ro.open(osmname, "relations") 86 | for t, e in f:lines() do read_relation(t, e) end 87 | f:close() 88 | 89 | else 90 | f:parse(read_way) 91 | f:close() 92 | 93 | f = ro.open(osmname, "nodes") 94 | f:parse(read_node) 95 | f:close() 96 | 97 | f = ro.open(osmname, "relations") 98 | f:parse(read_relation) 99 | f:close() 100 | end 101 | 102 | local elapsed = os.clock() - start 103 | io.stderr:write(("Read %5dk elements in %4ds (%3dk elements/s)... \n"): 104 | format(count / 1000, elapsed, count / elapsed / 1000)) 105 | 106 | db:commit() 107 | end 108 | 109 | 110 | local function transform(osm_file, db_file, transform_in) 111 | io.stderr:write(("Converting %s -> %s\n"):format(osm_file, db_file)) 112 | transformer = transform_in 113 | db = osmdb.create(db_file) 114 | read_osm(osm_file, db) 115 | return db 116 | end 117 | 118 | 119 | ------------------------------------------------------------------------------ 120 | 121 | return { transform = transform } 122 | 123 | ------------------------------------------------------------------------------ 124 | --------------------------------------------------------------------------------