├── .gitignore ├── LICENSE ├── MASSOSMTutorial2014-01-31.md ├── MASSOSMTutorial2014-01-31.txt ├── README.md ├── data ├── bike-paths-lds.RData ├── leeds-shops.qgs ├── map.osm ├── map.osm.db ├── osmo-cways.osm ├── osmo-cways.osm.db └── potter.osm ├── figure ├── All_OSM_data_in_and_directly_surround_Potternewton_Park.png ├── Bike_paths_of_Chapeltown.png ├── Bike_paths_of_Chapeltown1.png ├── Bike_paths_of_Chapeltown2.png ├── Line_data_of_Potternewton_park.png ├── Manual_selection_of_bounding_box.png ├── Preliminary_plot_of_Chapletown_with_osmar.png ├── Residential_streets_in_Chapeltown.png ├── import-osm.png ├── unnamed-chunk-1.png ├── unnamed-chunk-2.png ├── unnamed-chunk-5.png ├── unnamed-chunk-7.png ├── unnamed-chunk-8.png ├── unnamed-chunk-81.png ├── unnamed-chunk-82.png └── unnamed-chunk-9.png ├── md2pdf.R ├── osm-eg.RData ├── osm-tutorial.Rproj ├── osm.Rmd ├── osm.html ├── osm.log ├── osm.md ├── osm.pdf ├── osm.synctex.gz ├── osm.tex ├── osmfigs ├── Export-leeds.png ├── import-osm.png ├── import.png ├── load-style.png ├── loaded-line-data.png ├── manual-selection.png └── open-osmdb.png └── styles └── lines-style.qml /.gitignore: -------------------------------------------------------------------------------- 1 | .Rproj.user 2 | .Rhistory 3 | .RData 4 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | The MIT License (MIT) 2 | 3 | Copyright (c) 2014 Robin 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy of 6 | this software and associated documentation files (the "Software"), to deal in 7 | the Software without restriction, including without limitation the rights to 8 | use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of 9 | the Software, and to permit persons to whom the Software is furnished to do so, 10 | subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS 17 | FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR 18 | COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER 19 | IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN 20 | CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 21 | -------------------------------------------------------------------------------- /MASSOSMTutorial2014-01-31.md: -------------------------------------------------------------------------------- 1 | # MASS OSM Tutorial 2014-01-31 2 | ## 1. Introduction 3 | 4 | ### 1.1. Metadata 5 | * MASS OSM Tutorial 2014-01-31 6 | * https://docs.google.com/document/d/1hKBfZBQ80UR4k_jCMIbV_Sx5PEp5kr0xGy6fge9QREk/edit 7 | * This work is free of known copyright restrictions. 8 | * Notes from the MASS OpenStreetMap tutorial session on the 31st of January 2014 organised by Robin Lovelace 9 | 10 | ### 1.2. Contents 11 | [TOC] 12 | 13 | ## 2. Documentation 14 | * https://github.com/Robinlovelace/osm-tutorial 15 | 16 | ## 3. Preparation 17 | * BBC Radio 4 documentary “Mapping the Void” about OSM and its role in disaster response: 18 | * http://www.bbc.co.uk/programmes/b03s6mf0 19 | 20 | ## 4. Notes 21 | 22 | ### 4.1. The changing nature of OSM 23 | * OpenStreetMap (OSM) has grown in popularity and is now mainstream 24 | * It is no longer just about ways, it is an Open Map for many things 25 | * Buildings 26 | * Points of interest 27 | * Names 28 | * Incorporating open data on public transport stops and timetables 29 | * To contribute to OSM originally one needed GPS, but now there are other ways to contribute 30 | * National mapping agencies now using and contributing to OpenStreetMap 31 | * http://geospatial.blogs.com/geospatial/2013/07/un-ggim-on-trends-over-the-next-5-10-years-in-the-geospatial-sector.html 32 | 33 | ### 4.2. OSM for disaster response and mapping voids 34 | * Crisis Mappers 35 | * http://crisismappers.net/ 36 | * Baghdad 37 | * http://wiki.openstreetmap.org/wiki/Baghdad 38 | * Map Kibera 39 | * http://mapkibera.org/ 40 | * Started in 2009 when there was no map 41 | * Spawned a Citizen News Channel 42 | * Haiti 43 | * http://hot.openstreetmap.org/projects/haiti-2 44 | * http://wiki.openstreetmap.org/wiki/WikiProject_Haiti/Earthquake_map_resources 45 | * Humanitarian OpenStreetMap Team 46 | * http://hot.openstreetmap.org/ 47 | * Red Cross 48 | * http://www.wired.co.uk/news/archive/2013-11/14/red-cross-typhoon-philippines 49 | * Visualising information from email, text messages and blog posts 50 | 51 | ### 4.3. OSM utility for education 52 | * Students get to see their contribution to something that is preserved for the long term 53 | * This is empowering 54 | * Tandale Tanzania 55 | * http://developmentseed.org/blog/2012/aug/14/community-mapping-tandale-tanzania/ 56 | 57 | ### 4.4. Complimentary Open Mapping 58 | * Wikimapia 59 | * http://wikimapia.org/ 60 | * Wikipedia 61 | * http://www.wikipedia.org/ 62 | * Ushahidi 63 | * http://www.ushahidi.com/ 64 | 65 | ### 4.5. Exercises with OSM 66 | * Looking at changes over time 67 | * Exploring data formats and conversion 68 | * .osm is an XML format 69 | * .pbf .gzip are compressed versions 70 | * Practice handling large data sizes 71 | * Creating subsets of the data and releasing/serving it 72 | * Testing the routemap factor 73 | * Euclidean distance / road distance 74 | * The closer this is to 1 for the average journey in an area, the more developed the areas transport infrastructure is 75 | * Knowing the variation in this measure can be useful… 76 | * Find something to add to the map on campus or from the local area 77 | 78 | ### 4.6. OSM Frameworks Tools and Services 79 | * http://wiki.openstreetmap.org/wiki/Frameworks 80 | * OSMOSIS 81 | * http://wiki.openstreetmap.org/wiki/Osmosis 82 | * Java application for processing OSM data 83 | * Travelling Salesman 84 | * http://wiki.openstreetmap.org/wiki/Traveling_salesman 85 | * Java application for routing based on OSM data 86 | * Nominatim 87 | * http://wiki.openstreetmap.org/wiki/Nominatim 88 | * Tool to search OSM data by name and address and to generate synthetic addresses of OSM points (reverse geocoding) 89 | * Cloudmade 90 | * http://cloudmade.com/ 91 | * http://en.wikipedia.org/wiki/CloudMade 92 | * OSM data packager and re-distributer 93 | * Stamen 94 | * http://maps.stamen.com/ 95 | * Provides some rendered styles for OSM data 96 | * MapBox 97 | * https://www.mapbox.com/ 98 | * Allows for custom styling of OSM data 99 | * Tilemill 100 | * https://www.mapbox.com/tilemill/ 101 | * Design studio for creating interactive maps 102 | 103 | ### 4.7. Future 104 | * Will OSM ever be able to compete with the likes of Google for providing detail about real time live traffic? 105 | * Protecting people that do the mapping/provide information 106 | * Some people don’t want things mapping and are prepared to kill! 107 | 108 | ### 4.8. Miscellaneous 109 | * http://www.cyclestreets.net/ 110 | * http://openrouteservice.org/ 111 | * Kenya/Nairobi data 112 | * http://wiki.openstreetmap.org/wiki/Nairobi 113 | * http://downloads.cloudmade.com/africa/eastern_africa/kenya/nairobi 114 | * Map Kibera 115 | * http://mapkibera.org/ 116 | * http://www.geog.leeds.ac.uk/people/a.turner/src/andyt/java/projects/GENESIS/ 117 | 118 | ### 4.9. Next Steps 119 | * Andy to write up his notes and add these as markdown to the github repo 120 | * Do some coding and run through the tutorial in a subsequent meetings 121 | 122 | ## 5. References 123 | * https://github.com/Robinlovelace/osm-tutorial 124 | * GEOG5870M Web Based GIS 2013-03-07 125 | * https://docs.google.com/document/d/1QWT1pV4sBWRF6GR_hFfXO0uMkST0sXjLNQK6TjLdJEM/edit 126 | * GEOG5870M Web Based GIS 2013 to 2014 127 | * https://docs.google.com/document/d/1ozluS6B2jHNmkMigCyftxnkwBfNCDDQxK62FWcOgPCE/edit -------------------------------------------------------------------------------- /MASSOSMTutorial2014-01-31.txt: -------------------------------------------------------------------------------- 1 | MASS OSM Tutorial 2014-01-31 2 | 1. Introduction 3 | 1.1. Metadata 4 | * MASS OSM Tutorial 2014-01-31 5 | * https://docs.google.com/document/d/1hKBfZBQ80UR4k_jCMIbV_Sx5PEp5kr0xGy6fge9QREk/edit 6 | * This work is free of known copyright restrictions. 7 | * Notes from the MASS OpenStreetMap tutorial session on the 31st of January 2014 organised by Robin Lovelace 8 | 1.2. Contents 9 | 1. Introduction 10 | 1.1. Metadata 11 | 1.2. Contents 12 | 2. Documentation 13 | 3. Preparation 14 | 4. Notes 15 | 4.1. The changing nature of OSM 16 | 4.2. OSM for disaster response and mapping voids 17 | 4.3. OSM utility for education 18 | 4.4. Complimentary Open Mapping 19 | 4.5. Exercises with OSM 20 | 4.6. OSM Frameworks Tools and Services 21 | 4.7. OSM Future 22 | 4.8. Miscellaneous 23 | 4.9. Next Steps 24 | 5. References 25 | 2. Documentation 26 | * https://github.com/Robinlovelace/osm-tutorial 27 | 3. Preparation 28 | * BBC Radio 4 documentary “Mapping the Void” about OSM and its role in disaster response: 29 | * http://www.bbc.co.uk/programmes/b03s6mf0 30 | 4. Notes 31 | 4.1. The changing nature of OSM 32 | * OpenStreetMap (OSM) has grown in popularity and is now mainstream 33 | * It is no longer just about ways, it is an Open Map for many things 34 | * Buildings 35 | * Points of interest 36 | * Names 37 | * Incorporating open data on public transport stops and timetables 38 | * To contribute to OSM originally one needed GPS, but now there are other ways to contribute 39 | * National mapping agencies now using and contributing to OpenStreetMap 40 | * http://geospatial.blogs.com/geospatial/2013/07/un-ggim-on-trends-over-the-next-5-10-years-in-the-geospatial-sector.html 41 | 4.2. OSM for disaster response and mapping voids 42 | * Crisis Mappers 43 | * http://crisismappers.net/ 44 | * Baghdad 45 | * http://wiki.openstreetmap.org/wiki/Baghdad 46 | * Map Kibera 47 | * http://mapkibera.org/ 48 | * Started in 2009 when there was no map 49 | * Spawned a Citizen News Channel 50 | * Haiti 51 | * http://hot.openstreetmap.org/projects/haiti-2 52 | * http://wiki.openstreetmap.org/wiki/WikiProject_Haiti/Earthquake_map_resources 53 | * Humanitarian OpenStreetMap Team 54 | * http://hot.openstreetmap.org/ 55 | * Red Cross 56 | * http://www.wired.co.uk/news/archive/2013-11/14/red-cross-typhoon-philippines 57 | * Visualising information from email, text messages and blog posts 58 | 4.3. OSM utility for education 59 | * Students get to see their contribution to something that is preserved for the long term 60 | * This is empowering 61 | * Tandale Tanzania 62 | * http://developmentseed.org/blog/2012/aug/14/community-mapping-tandale-tanzania/ 63 | 4.4. Complimentary Open Mapping 64 | * Wikimapia 65 | * http://wikimapia.org/ 66 | * Wikipedia 67 | * http://www.wikipedia.org/ 68 | * Ushahidi 69 | * http://www.ushahidi.com/ 70 | 4.5. Exercises with OSM 71 | * Looking at changes over time 72 | * and Diffs over time 73 | * Exploring data formats and conversion 74 | * .osm is an XML format 75 | * .pbf .gzip are compressed versions 76 | * Practice handling large data sizes 77 | * Creating subsets of the data and releasing/serving it 78 | * Testing the routemap factor 79 | * Euclidean distance / road distance 80 | * The closer this is to 1 for the average journey in an area, the more developed the areas transport infrastructure is 81 | * Knowing the variation in this measure can be useful… 82 | * Find something to add to the map on campus or from the local area 83 | 4.6. OSM Frameworks Tools and Services 84 | * http://wiki.openstreetmap.org/wiki/Frameworks 85 | * OSMOSIS 86 | * http://wiki.openstreetmap.org/wiki/Osmosis 87 | * Java application for processing OSM data 88 | * Travelling Salesman 89 | * http://wiki.openstreetmap.org/wiki/Traveling_salesman 90 | * Java application for routing based on OSM data 91 | * Nominatim 92 | * http://wiki.openstreetmap.org/wiki/Nominatim 93 | * Tool to search OSM data by name and address and to generate synthetic addresses of OSM points (reverse geocoding) 94 | * Cloudmade 95 | * http://cloudmade.com/ 96 | * http://en.wikipedia.org/wiki/CloudMade 97 | * OSM data packager and re-distributer 98 | * Stamen 99 | * http://maps.stamen.com/ 100 | * Provides some rendered styles for OSM data 101 | * MapBox 102 | * https://www.mapbox.com/ 103 | * Allows for custom styling of OSM data 104 | * Tilemill 105 | * https://www.mapbox.com/tilemill/ 106 | * Design studio for creating interactive maps 107 | 4.7. Future 108 | * Will OSM ever be able to compete with the likes of Google for providing detail about real time live traffic? 109 | * Protecting people that do the mapping/provide information 110 | * Some people don’t want things mapping and are prepared to kill! 111 | 4.8. Miscellaneous 112 | * http://www.cyclestreets.net/ 113 | * http://openrouteservice.org/ 114 | * Kenya/Nairobi data 115 | * http://wiki.openstreetmap.org/wiki/Nairobi 116 | * http://downloads.cloudmade.com/africa/eastern_africa/kenya/nairobi 117 | * Map Kibera 118 | * http://mapkibera.org/ 119 | * http://www.geog.leeds.ac.uk/people/a.turner/src/andyt/java/projects/GENESIS/ 120 | 4.9. Next Steps 121 | * Andy to write up his notes and add these as markdown to the github repo 122 | * Do some coding and run through the tutorial in a subsequent meetings 123 | 5. References 124 | * https://github.com/Robinlovelace/osm-tutorial 125 | * GEOG5870M Web Based GIS 2013-03-07 126 | * https://docs.google.com/document/d/1QWT1pV4sBWRF6GR_hFfXO0uMkST0sXjLNQK6TjLdJEM/edit 127 | * GEOG5870M Web Based GIS 2013 to 2014 128 | * https://docs.google.com/document/d/1ozluS6B2jHNmkMigCyftxnkwBfNCDDQxK62FWcOgPCE/edit -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | osm-tutorial 2 | ============ 3 | 4 | Tutorial of importing, loading and analysing Open Street Map data with R and QGIS 5 | -------------------------------------------------------------------------------- /data/bike-paths-lds.RData: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Robinlovelace/osm-tutorial/8303ced4d8188c4163e8827d0fb40dfdcf53e7bc/data/bike-paths-lds.RData -------------------------------------------------------------------------------- /data/leeds-shops.qgs: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | degrees 6 | 7 | -1.81055623499999996 8 | 53.58992172087499739 9 | -1.2914935649999999 10 | 54.00949737912500126 11 | 12 | 0 13 | 14 | 15 | +proj=longlat +datum=WGS84 +no_defs 16 | 3452 17 | 4326 18 | EPSG:4326 19 | WGS 84 20 | longlat 21 | WGS84 22 | true 23 | 24 | 25 | 26 | 27 | 28 | 31 | 32 | 33 | 34 | 35 | points20140207115618950 36 | /media/SAMSUNG/geodata/shops-leeds.osm|layername=points 37 | 38 | 39 | 40 | 41 | 42 | points 43 | 44 | 45 | +proj=longlat +datum=WGS84 +no_defs 46 | 3452 47 | 4326 48 | EPSG:4326 49 | WGS 84 50 | longlat 51 | WGS84 52 | true 53 | 54 | 55 | ogr 56 | COALESCE( "name", '<NULL>' ) 57 | 58 | 59 | 60 | 61 | 62 | 63 | 64 | 65 | 66 | 67 | 68 | 69 | 70 | 71 | 72 | 73 | 74 | 75 | 76 | 77 | 78 | 79 | 80 | 0 81 | 0 82 | 0 83 | name 84 | 85 | 86 | 105 | 106 | 107 | 108 | generatedlayout 109 | 110 | 111 | 112 | 113 | 114 | 115 | 116 | EPSG:4326 117 | 118 | 119 | false 120 | 121 | 122 | 0 123 | 255 124 | 255 125 | 255 126 | 255 127 | 255 128 | 255 129 | 130 | 131 | 2 132 | true 133 | 134 | 135 | 136 | -------------------------------------------------------------------------------- /data/map.osm.db: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Robinlovelace/osm-tutorial/8303ced4d8188c4163e8827d0fb40dfdcf53e7bc/data/map.osm.db -------------------------------------------------------------------------------- /data/osmo-cways.osm.db: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Robinlovelace/osm-tutorial/8303ced4d8188c4163e8827d0fb40dfdcf53e7bc/data/osmo-cways.osm.db -------------------------------------------------------------------------------- /data/potter.osm: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | 45 | 46 | 47 | 48 | 49 | 50 | 51 | 52 | 53 | 54 | 55 | 56 | 57 | 58 | 59 | 60 | 61 | 62 | 63 | 64 | 65 | 66 | 67 | 68 | 69 | 70 | 71 | 72 | 73 | 74 | 75 | 76 | 77 | 78 | 79 | 80 | 81 | 82 | 83 | 84 | 85 | 86 | 87 | 88 | 89 | 90 | 91 | 92 | 93 | 94 | 95 | 96 | 97 | 98 | 99 | 100 | 101 | 102 | 103 | 104 | 105 | 106 | 107 | 108 | 109 | 110 | 111 | 112 | 113 | 114 | 115 | 116 | 117 | 118 | 119 | 120 | 121 | 122 | 123 | 124 | 125 | 126 | 127 | 128 | 129 | 130 | 131 | 132 | 133 | 134 | 135 | 136 | 137 | 138 | 139 | 140 | 141 | 142 | 143 | 144 | 145 | 146 | 147 | 148 | 149 | 150 | 151 | 152 | 153 | 154 | 155 | 156 | 157 | 158 | 159 | 160 | 161 | 162 | 163 | 164 | 165 | 166 | 167 | 168 | 169 | 170 | 171 | 172 | 173 | 174 | 175 | 176 | 177 | 178 | 179 | 180 | 181 | 182 | 183 | 184 | 185 | 186 | 187 | 188 | 189 | 190 | 191 | 192 | 193 | 194 | 195 | 196 | 197 | 198 | 199 | 200 | 201 | 202 | 203 | 204 | 205 | 206 | 207 | 208 | 209 | 210 | 211 | 212 | 213 | 214 | 215 | 216 | 217 | 218 | 219 | 220 | 221 | 222 | 223 | 224 | 225 | 226 | 227 | 228 | 229 | 230 | 231 | 232 | 233 | 234 | 235 | 236 | 237 | 238 | 239 | 240 | 241 | 242 | 243 | 244 | 245 | 246 | 247 | 248 | 249 | 250 | 251 | 252 | 253 | 254 | 255 | 256 | 257 | 258 | 259 | 260 | 261 | 262 | 263 | 264 | 265 | 266 | 267 | 268 | 269 | 270 | 271 | 272 | 273 | 274 | 275 | 276 | 277 | 278 | 279 | 280 | 281 | 282 | 283 | 284 | 285 | 286 | 287 | 288 | 289 | 290 | 291 | 292 | 293 | 294 | 295 | 296 | 297 | 298 | 299 | 300 | 301 | 302 | 303 | 304 | 305 | 306 | 307 | 308 | 309 | 310 | 311 | 312 | 313 | 314 | 315 | 316 | 317 | 318 | 319 | 320 | 321 | 322 | 323 | 324 | 325 | 326 | 327 | 328 | 329 | 330 | 331 | 332 | 333 | 334 | 335 | 336 | 337 | 338 | 339 | 340 | 341 | 342 | 343 | 344 | 345 | 346 | 347 | 348 | 349 | 350 | 351 | 352 | 353 | 354 | 355 | 356 | 357 | 358 | 359 | 360 | 361 | 362 | 363 | 364 | 365 | 366 | 367 | 368 | 369 | 370 | 371 | 372 | 373 | 374 | 375 | 376 | 377 | 378 | 379 | 380 | 381 | 382 | 383 | 384 | 385 | 386 | 387 | 388 | 389 | 390 | 391 | 392 | 393 | 394 | 395 | 396 | 397 | 398 | 399 | 400 | 401 | 402 | 403 | 404 | 405 | 406 | 407 | 408 | 409 | 410 | 411 | 412 | 413 | 414 | 415 | 416 | 417 | 418 | 419 | 420 | 421 | 422 | 423 | 424 | 425 | 426 | 427 | 428 | 429 | 430 | 431 | 432 | 433 | 434 | 435 | 436 | 437 | 438 | 439 | 440 | 441 | 442 | 443 | 444 | 445 | 446 | 447 | 448 | 449 | 450 | 451 | 452 | 453 | 454 | 455 | 456 | 457 | 458 | 459 | 460 | 461 | 462 | 463 | 464 | 465 | 466 | 467 | 468 | 469 | 470 | 471 | 472 | 473 | 474 | 475 | 476 | 477 | 478 | 479 | 480 | 481 | 482 | 483 | 484 | 485 | 486 | 487 | 488 | 489 | 490 | 491 | 492 | 493 | 494 | 495 | 496 | 497 | 498 | 499 | 500 | 501 | 502 | 503 | 504 | 505 | 506 | 507 | 508 | 509 | 510 | 511 | 512 | 513 | 514 | 515 | 516 | 517 | 518 | 519 | 520 | 521 | 522 | 523 | 524 | 525 | 526 | 527 | 528 | 529 | 530 | 531 | 532 | 533 | 534 | 535 | 536 | 537 | 538 | 539 | 540 | 541 | 542 | 543 | 544 | 545 | 546 | 547 | 548 | 549 | 550 | 551 | 552 | 553 | 554 | 555 | 556 | 557 | 558 | 559 | 560 | 561 | 562 | 563 | 564 | 565 | 566 | 567 | 568 | 569 | 570 | 571 | 572 | 573 | 574 | 575 | 576 | 577 | 578 | 579 | 580 | 581 | 582 | 583 | 584 | 585 | 586 | 587 | 588 | 589 | 590 | 591 | 592 | 593 | 594 | 595 | 596 | 597 | 598 | 599 | 600 | 601 | 602 | 603 | 604 | 605 | 606 | 607 | 608 | 609 | 610 | 611 | 612 | 613 | 614 | 615 | 616 | 617 | 618 | 619 | 620 | 621 | 622 | 623 | 624 | 625 | 626 | 627 | 628 | 629 | 630 | 631 | 632 | 633 | 634 | 635 | 636 | 637 | 638 | 639 | 640 | 641 | 642 | 643 | 644 | 645 | 646 | 647 | 648 | 649 | 650 | 651 | 652 | 653 | 654 | 655 | 656 | 657 | 658 | 659 | 660 | 661 | 662 | 663 | 664 | 665 | 666 | 667 | 668 | 669 | 670 | 671 | 672 | 673 | 674 | 675 | 676 | 677 | 678 | 679 | 680 | 681 | 682 | 683 | 684 | 685 | 686 | 687 | 688 | 689 | 690 | 691 | 692 | 693 | 694 | 695 | 696 | 697 | 698 | 699 | 700 | 701 | 702 | 703 | 704 | 705 | 706 | 707 | 708 | 709 | 710 | 711 | 712 | 713 | 714 | 715 | 716 | 717 | 718 | 719 | 720 | 721 | 722 | 723 | 724 | 725 | 726 | 727 | 728 | 729 | 730 | 731 | 732 | 733 | 734 | 735 | 736 | 737 | 738 | 739 | 740 | 741 | 742 | 743 | 744 | 745 | 746 | 747 | 748 | 749 | 750 | 751 | 752 | 753 | 754 | 755 | 756 | 757 | -------------------------------------------------------------------------------- /figure/All_OSM_data_in_and_directly_surround_Potternewton_Park.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Robinlovelace/osm-tutorial/8303ced4d8188c4163e8827d0fb40dfdcf53e7bc/figure/All_OSM_data_in_and_directly_surround_Potternewton_Park.png -------------------------------------------------------------------------------- /figure/Bike_paths_of_Chapeltown.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Robinlovelace/osm-tutorial/8303ced4d8188c4163e8827d0fb40dfdcf53e7bc/figure/Bike_paths_of_Chapeltown.png -------------------------------------------------------------------------------- /figure/Bike_paths_of_Chapeltown1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Robinlovelace/osm-tutorial/8303ced4d8188c4163e8827d0fb40dfdcf53e7bc/figure/Bike_paths_of_Chapeltown1.png -------------------------------------------------------------------------------- /figure/Bike_paths_of_Chapeltown2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Robinlovelace/osm-tutorial/8303ced4d8188c4163e8827d0fb40dfdcf53e7bc/figure/Bike_paths_of_Chapeltown2.png -------------------------------------------------------------------------------- /figure/Line_data_of_Potternewton_park.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Robinlovelace/osm-tutorial/8303ced4d8188c4163e8827d0fb40dfdcf53e7bc/figure/Line_data_of_Potternewton_park.png -------------------------------------------------------------------------------- /figure/Manual_selection_of_bounding_box.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Robinlovelace/osm-tutorial/8303ced4d8188c4163e8827d0fb40dfdcf53e7bc/figure/Manual_selection_of_bounding_box.png -------------------------------------------------------------------------------- /figure/Preliminary_plot_of_Chapletown_with_osmar.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Robinlovelace/osm-tutorial/8303ced4d8188c4163e8827d0fb40dfdcf53e7bc/figure/Preliminary_plot_of_Chapletown_with_osmar.png -------------------------------------------------------------------------------- /figure/Residential_streets_in_Chapeltown.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Robinlovelace/osm-tutorial/8303ced4d8188c4163e8827d0fb40dfdcf53e7bc/figure/Residential_streets_in_Chapeltown.png -------------------------------------------------------------------------------- /figure/import-osm.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Robinlovelace/osm-tutorial/8303ced4d8188c4163e8827d0fb40dfdcf53e7bc/figure/import-osm.png -------------------------------------------------------------------------------- /figure/unnamed-chunk-1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Robinlovelace/osm-tutorial/8303ced4d8188c4163e8827d0fb40dfdcf53e7bc/figure/unnamed-chunk-1.png -------------------------------------------------------------------------------- /figure/unnamed-chunk-2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Robinlovelace/osm-tutorial/8303ced4d8188c4163e8827d0fb40dfdcf53e7bc/figure/unnamed-chunk-2.png -------------------------------------------------------------------------------- /figure/unnamed-chunk-5.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Robinlovelace/osm-tutorial/8303ced4d8188c4163e8827d0fb40dfdcf53e7bc/figure/unnamed-chunk-5.png -------------------------------------------------------------------------------- /figure/unnamed-chunk-7.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Robinlovelace/osm-tutorial/8303ced4d8188c4163e8827d0fb40dfdcf53e7bc/figure/unnamed-chunk-7.png -------------------------------------------------------------------------------- /figure/unnamed-chunk-8.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Robinlovelace/osm-tutorial/8303ced4d8188c4163e8827d0fb40dfdcf53e7bc/figure/unnamed-chunk-8.png -------------------------------------------------------------------------------- /figure/unnamed-chunk-81.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Robinlovelace/osm-tutorial/8303ced4d8188c4163e8827d0fb40dfdcf53e7bc/figure/unnamed-chunk-81.png -------------------------------------------------------------------------------- /figure/unnamed-chunk-82.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Robinlovelace/osm-tutorial/8303ced4d8188c4163e8827d0fb40dfdcf53e7bc/figure/unnamed-chunk-82.png -------------------------------------------------------------------------------- /figure/unnamed-chunk-9.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Robinlovelace/osm-tutorial/8303ced4d8188c4163e8827d0fb40dfdcf53e7bc/figure/unnamed-chunk-9.png -------------------------------------------------------------------------------- /md2pdf.R: -------------------------------------------------------------------------------- 1 | mess <- paste('pandoc -f markdown -t latex -s -o', "osm.tex", 2 | "osm.md") 3 | system(mess) # create latex file 4 | 5 | mess <- paste("sed -i -e 's/plot of.chunk.//g' osm.tex") 6 | system(mess) # replace "plot of chunk " text with nowt 7 | 8 | mess <- paste("sed -i -e 's/\\\\section{Open Street Map: loading, analysing and visualising free maps//g' osm.tex") 9 | system(mess) # replace "plot of chunk " text with nowt 10 | 11 | mess <- paste("sed -i -e 's/with R and QGIS}//g' osm.tex") 12 | system(mess) # replace "plot of chunk " text with nowt 13 | 14 | mess <- paste("sed -i -e 's/width=\\\\maxwidth/width=10cm/g' osm.tex") 15 | system(mess) # reduce plot size 16 | 17 | # mess <- paste("sed -i -e 's/\\\\section{References}/\\\\newpage \\\\section{References}/g' osm.tex") 18 | # system(mess) # Put refs on new page 19 | 20 | mess <- "sed -i -e '64i\\\\\\maketitle' osm.tex" 21 | system(mess) # make title, after \begin{document} 22 | 23 | mess <- "sed -i -e '62i\\\\\\usepackage[margin=1.8cm]{geometry}' osm.tex" 24 | system(mess) # shrink margins 25 | 26 | idx <- 62 27 | # open the file and read in all the lines 28 | conn <- file("osm.tex") 29 | text <- readLines(conn) 30 | block <- "\\author{ 31 | Lovelace, Robin\\\\ 32 | \\texttt{r.lovelace@leeds.ac.uk} 33 | } 34 | \\title{Harnessing Open Street Map Data with R and QGIS}" 35 | text_block <- unlist(strsplit(block, split='\n')) 36 | # concatenate the old file with the new text 37 | mytext <- c(text[1:idx],text_block,text[(idx+1):length(text)]) 38 | writeLines(mytext, conn, sep="\n") 39 | close(conn) 40 | -------------------------------------------------------------------------------- /osm-eg.RData: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Robinlovelace/osm-tutorial/8303ced4d8188c4163e8827d0fb40dfdcf53e7bc/osm-eg.RData -------------------------------------------------------------------------------- /osm-tutorial.Rproj: -------------------------------------------------------------------------------- 1 | Version: 1.0 2 | 3 | RestoreWorkspace: Default 4 | SaveWorkspace: Default 5 | AlwaysSaveHistory: Default 6 | 7 | EnableCodeIndexing: Yes 8 | UseSpacesForTab: Yes 9 | NumSpacesForTab: 2 10 | Encoding: UTF-8 11 | 12 | RnwWeave: Sweave 13 | LaTeX: pdfLaTeX 14 | -------------------------------------------------------------------------------- /osm.Rmd: -------------------------------------------------------------------------------- 1 | Open Street Map: loading, analysing and visualising free maps with R and QGIS 2 | ======================================================== 3 | 4 | This tutorial shows how open source tools can be used to 5 | harness a huge and rapidly growing open source geodatabase: Open Street Map. 6 | It is targeted at people new to Open Street Map, but who 7 | already have an understanding of basic GIS concepts. 8 | Previous experience with the programs QGIS and R would be beneficial, 9 | but not essential, for completing the exercises. There are 10 | also a number of resources available on-line for more advanced functions, 11 | as described below. 12 | 13 | # Introduction 14 | 15 | Open Street Map (OSM) is a crowd-sourced map of the world, 16 | the archetype of 'volunteered geographical information' 17 | ([Goodchild 2007](http://www.ncgia.ucsb.edu/projects/vgi/docs/position/Goodchild_VGI2007.pdf)). 18 | The aim is simple: "to create a set of map data that’s free 19 | to use, editable, and licensed under new 20 | copyright schemes", as described in an excellent review 21 | article on the subject 22 | ([Haklay and Weber 2008](http://discovery.ucl.ac.uk/13849/1/13849.pdf)). 23 | Putting the public in charge of 24 | editing the world's surface may seem like a risky 25 | business, given that cartographers have specialist skills 26 | developed over centuries. This issue is described in 27 | [Mapping the Void](https://dl.dropboxusercontent.com/u/15008199/egs2stay/Mapping_the_Void_-_Mapping_the_Void_b03s6mf0_default.m4a), an excellent BBC Radio 4 Documentary 28 | on the subject (Salisbury and Jenkins 2014). 29 | Yet the emergence 30 | of high resolution aerial photography covering the entirety 31 | of the Earth's surface and the 32 | explosion in GPS ownership via smartphones has enabled citizens 33 | to become accurate sensors of the world. 34 | [Neis et al. (2012)](http://www.mdpi.com/1999-5903/4/1/1/pdf) 35 | believe this phenomenon is more than merely technological. 36 | In OSM, they see a "revolutionary paradigm shift on how map data is 37 | being collected". 38 | Citizen mappers have the added advantage that they may know their local 39 | area far better than any professional cartographer. 40 | 41 | This tutorial adds a small nugget of information to the growing literature 42 | on OSM, by demonstrating how the data can be accessed for teaching or research 43 | materials. It's a completely free and open dataset, so we may as well use it! 44 | There is already some good on-line material about OSM data, including: 45 | 46 | - a [paper](http://www.mdpi.com/1999-5903/5/2/282/pdf) comparing the quality and 47 | coverage of OSM map data in different parts of the world (Neis et al. 2013) 48 | - an [overview](http://www.library.carleton.ca/sites/default/files/help/gis/WorkingWithOpenStreetMap.pdf) 49 | of handling OSM data in ArcMap 50 | - a [tutorial](http://elogeo.nottingham.ac.uk/xmlui/bitstream/handle/url/289/osm-tutorial-final-2.pdf?sequence=1) illustrating 51 | its potential use for GIS education and store location planning (Lovelace 2013). 52 | 53 | Yet nothing has focused on simply loading the data using different tools. 54 | In this paper we use QGIS and R because these are (arguably) the most popular 55 | open source programs used by geographers, the discipline with most to gain 56 | from OSM data. Before delving into the method (which is actually quite straightforward), 57 | let's put OSM data in context. 58 | 59 | ## Why (and why not) use OSM data? 60 | 61 | Of course there are teething issues with any large-scale open source 62 | database, including variable data quality, 63 | patchy and incomplete coverage and inconsistencies from place to place (Haklay 2010). Yet all of these 64 | issues are gradually being ironed out. 65 | The advantages of Open Street Map outweigh these downsides for 66 | many applications *already*. These include: 67 | 68 | - Rapid updates of new projects 69 | - Greater range of attributes (e.g. shop names) 70 | - Ability to share data with anyone without breaching license 71 | 72 | In additions there are a number of ethical benefits of using OSM: it's community 73 | a map for the greater good ([Wroclawski 2014](http://www.theguardian.com/technology/2014/jan/14/why-the-world-needs-openstreetmap)). 74 | 75 | There is a strong community of OSM volunteers who use the service for humanitarian purposes 76 | (Salisbury and Jenkins 2014). 77 | Two excellent examples of this help underline the 78 | ethical side of OSM data. [Tindale](http://explore.ramanitanzania.org/), 79 | a settlement in Tanzania that has been mapped rapidly 80 | thanks to a single academic project, in collaboration with the authorities, enabling 81 | better policy making in the area (see [video](http://www.youtube.com/watch?v=KqrGyvNnWkA)). 82 | Second, 83 | the rapid response of the OSM community to the Typhoon Haiyan disaster. In a matter of days, an 84 | army of volunteers had helped to map out the affected zone, aiding relief efforts (e.g. see 85 | [MapBox's blog posts on the subject](https://www.mapbox.com/blog/typhoon-haiyan-openstreetmap/)). 86 | 87 | ## An overview of the tutorial 88 | 89 | All of the code and data used to create this tutorial is available 90 | on [GitHub](http://github.com). Feel free to download the project as a `.zip` file from the 91 | [project's repository](https://github.com/Robinlovelace/osm-tutorial) entitled osm-tutorial from 92 | github.com/Robinlovelace. This zip file contains all of the code and data 93 | used to generate the tutorial, which is entirely reproducible. If you would like to modify or improve it 94 | in any way, please fork a version for you own use, citing the original where appropriate. 95 | 96 | The focus is mainly on the technical challenge of extracting the 97 | data from servers 'in the cloud' and onto your desktop. We also cover some basic 98 | tasks in handling, subsetting and visualising the data, first in the 99 | graphical user interface (GUI) of QGIS, and then in the command line with R. 100 | The next stage talks about editing raw OSM data, essential if you 101 | would like to take subsets of very large OSM datasets, such as `planet.osm` 102 | which takes up more than 30 GB of memory. The next stage is 103 | to talk about OSM data in PostGIS, a geodatabase program ideal 104 | for querying very large spatial datasets. Finally, there is a brief 105 | section on further resources. None of the sections require any of the 106 | previous ones although the level of difficulty generally increases 107 | as you progress. 108 | 109 | # Getting the data 110 | 111 | OSM data of a specific area 112 | can be downloaded directly from the [main map page](http://www.openstreetmap.org), from the 113 | [Overpass API](http://overpass-api.de/) or, for the entire planet, from the huge (currently 32 GB) 114 | [planet.osm file](http://planet.openstreetmap.org/). A number of third parties also provide more manageable 115 | chunks of this dataset, such as the single country datasets provided by 116 | [GEOFABIK](http://download.geofabrik.de/). Command line programs 117 | [Osmosis](http://wiki.openstreetmap.org/wiki/Osmosis) and 118 | [Osm2pgsl](http://wiki.openstreetmap.org/wiki/Osm2pgsql) can be used to process raw OSM data 119 | in either `.osm` or `.osm.pbf` file formats. The former is essentially a `.xml` (Extensible Markup Language) 120 | text file (encoded with the popular UTF-8 characterset); the latter is a compressed version of the former. 121 | How we transfer these datasets into a useful form depends on the program you are using. 122 | In this tutorial we will explain how to do it in QGIS and R, as well describing the basics of 123 | getting it into a [PostGIS](http://postgis.net/) database. 124 | 125 | # OSM data in QGIS 126 | 127 | A `.osm` file can be downloaded from the openstreetmap.org with the bounding box selected by 128 | default depending on the current view, or via a manual selection, as shown below. 129 | 130 | ```{r Manual selection of bounding box, fig.width=12, fig.height=8, echo=FALSE} 131 | library(png) 132 | library(grid) 133 | img <- readPNG("osmfigs/manual-selection.png") 134 | grid.raster(img) 135 | ``` 136 | 137 | To load this file into QGIS, you can simply use the `Add Vector Layer` button on the 138 | left of the screen. However this does not generate satisfactory results. 139 | The *recommended* way to import the data is via the the OSM plugin. When this is 140 | installed in QGIS 2.0, use the menus `Vector > OpenStreetMap` to import the xml file 141 | and convert it into a SpatiaLite database. Then you can import it into the QGIS workspace. 142 | 143 | ![Import the osm data to SpatiaLite](osmfigs/import-osm.png) 144 | 145 | After this step the file has been saved as a `.osm.db` file. Use the 146 | `Export Topology to SpatiaLite` element of the same menu to 147 | load the file. Choose the type of spatial data you would like to load - 148 | Points, Lines or Polygons. At this stage one can also select which variables 149 | ("Tags") you would like to add to the attribute data. 150 | 151 | ![Select Polylines](osmfigs/open-osmdb.png) 152 | 153 | The data is now loaded into QGIS allowing standard methods of analysis. 154 | You will notice that the data are not styled at all, leading to very bland 155 | maps. To counter this, there have been custom styles developed for visualising OSM data in QGIS, 156 | e.g. [those by Anita Grazer](http://anitagraser.com/2012/02/25/light-styles-for-osm-layers-in-qgis/). 157 | Unfortunately these files do not seem to be working with the current version of QGIS so 158 | alternative ready-made styles must be created, as suggested by a 159 | [stackexchange question](http://gis.stackexchange.com/questions/42645/is-there-up-to-date-osm-sld-file-for-geoserver). 160 | 161 | Once the data is loaded into QGIS, it can be used as with any other spatial data. 162 | Next, let's see how R can handle OSM data, via the `osmar` package. 163 | 164 | # Using osmar 165 | 166 | `osmar` is an R package for downloading and interrogating OSM data that accesses 167 | the data directly from the internet via the R command line. 168 | There is an excellent 169 | [online tutorial](http://journal.r-project.org/archive/2013-1/eugster-schlesinger.pdf) 170 | which provides a detailed account of the package (Eugster & Schlesinger, 2012). 171 | Here we will focus on loading some basic data on bicycle paths in Leeds. 172 | First the package must be loaded: 173 | 174 | ```{r, results='hide'} 175 | library(osmar) # if the package is not already installed, use install.packages("osmar") 176 | ``` 177 | 178 | To download data directly, one first sets the source and a bounding box, 179 | and then use the `get_osm` function to download it. Selecting Chapeltown as the 180 | centrepoint of the map, we can download all the data in the square km surrounding it. 181 | 182 | ```{r, Preliminary plot of Chapletown with osmar} 183 | src <- osmsource_api() 184 | bb <- center_bbox(-1.53492, 53.81934, 1000, 1000) 185 | ctown <- get_osm(bb, source = src) 186 | plot(ctown) 187 | points(-1.53492, 53.81934, col="red", lwd = 5) 188 | ``` 189 | 190 | This shows that the data has successfully been loaded and saved as an 191 | object called `ctown`. Let's try analysing this object further. 192 | In fact, `ctown` is technically a list, composed of 3 objects: 193 | nodes (points), ways (lines) and relations (polygons composed of 194 | many lines). Such OSM data is thus provided a class of its own, 195 | and each sub-object can be called separately using the `$` symbol: 196 | 197 | ```{r} 198 | names(ctown) 199 | class(ctown) 200 | summary(ctown$ways) 201 | ``` 202 | 203 | Let's use the dataset we have loaded to investigate the cycle 204 | paths in the vicinity of my house. First we need to understand the data 205 | contained in the object. Let's look at the tags and the attributes of the `ways` object: 206 | 207 | ```{r} 208 | summary(ctown$ways$tags) # summary of the tag data 209 | ctown$ways$attrs[1:8, 1:6] # attributes of first 8 ways - see I'm in there! 210 | ``` 211 | 212 | From looking at the [OSM tagging system](http://wiki.openstreetmap.org/wiki/Tags), we can deduce that 213 | `id` is the element's id, 214 | `k` refers to the OSM key (the variables for which the element 215 | has values) and that `v` is the value assigned for each 216 | id - key combination. Because OSM data is not a simple data frame, 217 | we cannot use the usual R notation for subsetting. Instead we use the 218 | `find` function. Let us take a subset of bicycle paths in the area 219 | and plot them. 220 | 221 | ```{r Bike paths of Chapeltown} 222 | bikePaths <- find(ctown, way(tags(k == "bicycle" & v == "yes" )) ) 223 | bikePaths <- find_down(ctown, way(bikePaths)) 224 | bikePaths <- subset(ctown, ids= bikePaths) 225 | plot(ctown) 226 | plot_ways(bikePaths, add = T, col = "red", lwd = 3) 227 | save(bikePaths, file="data/bike-paths-lds.RData") 228 | 229 | # analysis of time of addition 230 | class(bikePaths$nodes$attrs$timestamp) 231 | tstamp <- as.POSIXct(bikePaths$nodes$attrs$timestamp, format=) 232 | hist(bikePaths$nodes$attrs$timestamp, breaks = "year") 233 | ``` 234 | 235 | The above code block is used to identify all ways in which cycling 236 | is permitted, "overriding default access", according OSM's excellent 237 | [wiki page on bicycle paths](http://wiki.openstreetmap.org/wiki/Bicycle). 238 | 239 | According to this source, the correct way to refer to an on-road cycle path 240 | is with the `cycleway` tag. However, none of these have been added to the 241 | roads that have on-road cycle lanes in this example dataset (as of January 2014). 242 | Perhaps someone will add these soon. 243 | 244 | ```{r} 245 | which(ctown$ways$tags$k == "cycleway") 246 | ``` 247 | 248 | There are, by contrast, a large number of ways classified as "residential". 249 | Let us use the same method to select them and add them to the map. 250 | 251 | ```{r Residential streets in Chapeltown} 252 | res <- find(ctown, way(tags(k == "highway" & v == "residential" )) ) 253 | res <- find_down(ctown, way(res)) 254 | res <- subset(ctown, ids= res) 255 | plot(ctown) 256 | plot_ways(res, add = T, col = "green", lwd = 3) 257 | ``` 258 | 259 | # Handling raw OSM data 260 | 261 | Although this section will be of most use for dealing with 262 | very large files, a small example is used here to showcase the methods 263 | The file `map.osm` downloaded directly from [openstreetmap.org](http://www.openstreetmap.org) 264 | is small, allowing available on-line, from 265 | [here](https://github.com/Robinlovelace/osm-tutorial/blob/master/data/map.osm?raw=true) 266 | (highlighting the advantages 267 | of open data for educational purposes - there are few license restrictions). 268 | 269 | The Java command line tool [Osmosis](http://wiki.openstreetmap.org/wiki/Osmosis) 270 | is used for this purpose. 271 | 272 | To show how Osmosis can subset raw OSM files, let us take a real world example from 273 | the dataset we imported with the `osmar` package. Imagine that we want to extract all 274 | osm data (nodes, ways and relations) from the area in and around Potternewton Park. 275 | We could do this manually by looking up the park on-line and selecting the associated 276 | elements in QGIS. But imagine we want to do it from the 277 | command line (e.g. for batch processing). 278 | First we need a bounding box of all items containing the 279 | text string "Potternewton Park". For this we use `osmar`: 280 | 281 | ```{r, Line data of Potternewton park} 282 | potter <- find(ctown, way(tags(grepl("Potternewton Park", ctown$ways$tags$v)))) 283 | potter <- find_down(ctown, way(potter)) 284 | potter <- subset(ctown, ids = potter) 285 | plot_ways(potter) # sanity check 286 | potter.l <- as_sp(potter, "lines") # convert data to sp class 287 | b <- bbox(potter.l) # save the bounding box 288 | b[1, ] <- (b[1, ] - mean(b[1, ])) * 1.05 + mean(b[1, ]) 289 | b[2, ] <- (b[2, ] - mean(b[2, ])) * 1.05 + mean(b[2, ]) 290 | # scale longitude and latitude (increase bb by 5% for plot) replace 1.05 291 | # with 1.xx for an xx% increase in the plot size 292 | b 293 | ``` 294 | 295 | Now that we know the bounding box, we can transfer to using osmosis from 296 | the command line. If the shell is open in the correct working directory 297 | (e.g. "osm-tutorial-master", if downloaded from GitHub), the following 298 | code will subset the data and output a new, smaller `.osm` file. 299 | 300 | ``` 301 | osmosis --read-xml data/map.osm --bounding-box top=53.822093 left=-1.528815 302 | bottom=53.817486 right=-1.521155 --write-xml potter.osm 303 | 304 | ``` 305 | 306 | To check whether or not this has worked, and demonstrate `osmar`'s ability to read in 307 | `.osm` files, let us try to load an plot the dataset just exported by osmosis. 308 | 309 | ```{r All OSM data in and directly surround Potternewton Park, warning=FALSE} 310 | src <- osmsource_osmosis(file="data/potter.osm") 311 | bp <- center_bbox(mean(b[1, ]), mean(b[2, ]), 1000, 1000) 312 | potter <- get_osm(bp, src) 313 | plot(potter) 314 | ``` 315 | 316 | Local knowledge or a quick look on an online will confirm that this is 317 | indeed Potternewton Park, with is rectangular tennis courts, play area 318 | and skate park just west of its centre. The file `potter.osm` can 319 | equally be loaded into QGIS or any other GIS. The point is to illustrate 320 | how Osmosis works. 321 | 322 | For further functionality, including clipping to polygons, extracting files from 323 | enormous and compressed `planet.osm.bz2` files and ways to extract only elements with 324 | certain attributes, please refer to the 325 | [osmosis page of the osm wiki](http://wiki.openstreetmap.org/wiki/Osmosis). 326 | The final section discusses (but does not implement) 327 | the most advanced method of harnessing OSM data, via a PostGIS database. 328 | 329 | # Creating a PostGIS database of OSM data 330 | 331 | There are a number of advantages of storing large datasets 332 | in a database: 333 | 334 | - The data can be accessed by a variety of 3rd party programs 335 | - Spatial queries can be made by a variety of clients accessing the db, widening the functionality 336 | - Huge datasets can be stored in a database, as it sits on the hard disk, only being 337 | transferred to RAM when a specific query is called 338 | 339 | Because of the size and complexity of the planet-wide OSM database, it must 340 | be stored in an spatial database to be used. For this one can 341 | use the command line tool 342 | [osm2pgsql](http://wiki.openstreetmap.org/wiki/Osm2pgsql). 343 | 344 | Once you have Postgres and PostGIS installed on your computer, 345 | and are logged in as the Postgres user the process is straightforward. 346 | The series of steps required to convert a `.osm` file into a 347 | PostGIS enabled databased are as follows. 348 | 349 | ```{bash} 350 | createdb osmTutdb # create the database called osmTutdb 351 | 352 | psql osmTutdb -c "create extension postgis" # add PostGIS functionality 353 | 354 | psql osmTutdb -c "create extension pgrouting" # add pgrouting functionality (not essential) 355 | 356 | osm2pgsql -d osmTutdb map.osm # dump the map.osm data into the new database 357 | ``` 358 | 359 | The problem with this method of doing things seems to be that osm2pgsql 360 | ignores the timestamp and user id of the elements, a serious drawback 361 | for some applications. According to a 362 | [bug report](https://trac.openstreetmap.org/ticket/4894) this is not 363 | solved by the addition of the `--extra-attributes` tag and this problem 364 | was witnessed to persist in osm2pgsql 0.83.0. 365 | 366 | To select all cycleways in an area using osmosis, this was the 367 | command used (only found to work on up-to-date versions): 368 | 369 | ```{bash} 370 | osmosis --read-xml map.osm --tf accept-ways highway=cycleway --used-node --write-xml osmo-cways.osm 371 | ``` 372 | 373 | 374 | 375 | 376 | Using a spatial database, it does not take a very large leap to realise that 377 | organisations can create a customised version of the OSM database for their own purposes. 378 | For example, an organisation interested in store location analysis could keep maintained 379 | a version of OSM with everything with elements labelled as shops removed. Alternatively, 380 | an organisation interested in woodlands could maintain an up-to-date version 381 | of OSM woodlands. Both organisations could supplement these custom databases with their 382 | own data, combining the best of crowd-sourced and centralised data collection methods. 383 | 384 | The [OSM-GB](http://wiki.openstreetmap.org/wiki/Osm2pgsql) project, hosted at the University 385 | of Nottingham does precisely this, with the aim of quality-checking community contributed data. 386 | Here is not the place to describe how to set-up a PostGIS database with OSM data, but is well 387 | worth flagging as it has great potential, especially when combined with rapidly 388 | evolving open source web mapping technologies such as 389 | [GeoServer](http://geoserver.org/display/GEOS/Welcome) 390 | (part of the 391 | [GeoNode](http://geonode.org/) stack), [Leaflet](http://leafletjs.com/) and 392 | [GeoDjango](https://www.djangoproject.com/). 393 | 394 | # Conclusion 395 | 396 | With the certainty of peak oil and possibility of 397 | effective climate change regulations, transport will become increasing 398 | expensive in future years. Thus, the tendency towards geographical 399 | homogenisation of economic activity may go into reverse 400 | ([Greer 2009](http://books.google.co.uk/books?hl=en&lr=&id=mkV_knlze0QC&oi=fnd&pg=PP2&dq=ecotechnic+future&ots=nATRuCVL31&sig=bwafIZ7kfmZMK1EscQcKyIGeYsU&redir_esc=y#v=onepage&q=ecotechnic%20future&f=false); 401 | [Curtis 2009](http://www.sciencedirect.com/science/article/pii/S0921800909003334)). 402 | This is bad news for many, but it is good news for people with 403 | a strong interest in regional diversity, local economies and geographic diversity. 404 | 405 | It is also potentially good news for geographers advocating for location-specific 406 | solutions. With increased concern over the 407 | highly centralised power structures of the internet following the 408 | revelations leaked by Edward Snowdon about massive online spying 409 | and infringement of digital privacy, there 410 | is a huge potential for community-based, problem-specific solutions. 411 | It is with this wider context in mind that this tutorial ends - 412 | think of the potential benefits if citizens were encouraged to be 413 | both producers and consumers of the maps on which we all now depend. 414 | Happy mapping! 415 | 416 | # References 417 | 418 | Curtis, F. (2009). Peak globalization: Climate change, oil depletion and global trade. Ecological Economics, 69(2), 427-434. 419 | 420 | Eugster, M. J., & Schlesinger, T. (2013). osmar: OpenStreetMap and R. The R Journal, 5(1), 53-63. 421 | 422 | Goodchild, M. F. (2007). Citizens as sensors: the world of volunteered 423 | geography. GeoJournal, 69(4), 211–221. 424 | 425 | Greer, J. M. (2009). The Ecotechnic Future: Envisioning a post-peak world. New Society Publishers. 426 | 427 | Haklay, M., & Weber, P. (2008). Openstreetmap: User-generated street maps. Pervasive Computing, IEEE, 7(4), 12-18. 428 | 429 | Lovelace, R. (2013). Open Source Data and Methods: A tutorial using crowd-sourced data for store location analysis. EloGeo repository. 430 | 431 | Neis, P., Zielstra, D., & Zipf, A. (2011). The street network evolution of crowdsourced maps: OpenStreetMap in Germany 2007–2011. Future Internet, 4(1), 1-21. 432 | 433 | Neis, P., Zielstra, D., & Zipf, A. (2013). Comparison of Volunteered Geographic Information Data Contributions and Community Development for Selected World Regions. Future Internet, 5(2), 282-300. 434 | 435 | Salisbury, C. & Jenkins, J. (2014). Mapping the Void. Broadcast on BBC Radio 4, 11:00AM Mon, 27 Jan 2014. Available on I-Player until 11:32AM Mon, 3 Feb 2014. Available on my 436 | [Dropbox account](https://dl.dropboxusercontent.com/u/15008199/egs2stay/Mapping_the_Void_-_Mapping_the_Void_b03s6mf0_default.m4a) for forseable future. 437 | 438 | Wroclawski, S. (2014). Why the world needs OpenStreetMap. The Guardian. Tuesday 14 January 2014 11.52 GMT. 439 | 440 | ```{r} 441 | source("md2pdf.R") # convert markdown document to LaTeX 442 | ``` 443 | 444 | 445 | 446 | 447 | -------------------------------------------------------------------------------- /osm.log: -------------------------------------------------------------------------------- 1 | This is pdfTeX, Version 3.1415926-1.40.10 (TeX Live 2009/Debian) (format=pdflatex 2013.10.5) 1 FEB 2014 18:26 2 | entering extended mode 3 | %&-line parsing enabled. 4 | **osm.tex 5 | (./osm.tex 6 | LaTeX2e <2009/09/24> 7 | Babel and hyphenation patterns for english, usenglishmax, dumylang, noh 8 | yphenation, loaded. 9 | (/usr/share/texmf-texlive/tex/latex/base/article.cls 10 | Document Class: article 2007/10/19 v1.4h Standard LaTeX document class 11 | (/usr/share/texmf-texlive/tex/latex/base/size10.clo 12 | File: size10.clo 2007/10/19 v1.4h Standard LaTeX file (size option) 13 | ) 14 | \c@part=\count79 15 | \c@section=\count80 16 | \c@subsection=\count81 17 | \c@subsubsection=\count82 18 | \c@paragraph=\count83 19 | \c@subparagraph=\count84 20 | \c@figure=\count85 21 | \c@table=\count86 22 | \abovecaptionskip=\skip41 23 | \belowcaptionskip=\skip42 24 | \bibindent=\dimen102 25 | ) 26 | (/usr/share/texmf-texlive/tex/latex/amsfonts/amssymb.sty 27 | Package: amssymb 2009/06/22 v3.00 28 | 29 | (/usr/share/texmf-texlive/tex/latex/amsfonts/amsfonts.sty 30 | Package: amsfonts 2009/06/22 v3.00 Basic AMSFonts support 31 | \@emptytoks=\toks14 32 | \symAMSa=\mathgroup4 33 | \symAMSb=\mathgroup5 34 | LaTeX Font Info: Overwriting math alphabet `\mathfrak' in version `bold' 35 | (Font) U/euf/m/n --> U/euf/b/n on input line 96. 36 | )) 37 | (/usr/share/texmf-texlive/tex/latex/amsmath/amsmath.sty 38 | Package: amsmath 2000/07/18 v2.13 AMS math features 39 | \@mathmargin=\skip43 40 | 41 | For additional information on amsmath, use the `?' option. 42 | (/usr/share/texmf-texlive/tex/latex/amsmath/amstext.sty 43 | Package: amstext 2000/06/29 v2.01 44 | 45 | (/usr/share/texmf-texlive/tex/latex/amsmath/amsgen.sty 46 | File: amsgen.sty 1999/11/30 v2.0 47 | \@emptytoks=\toks15 48 | \ex@=\dimen103 49 | )) 50 | (/usr/share/texmf-texlive/tex/latex/amsmath/amsbsy.sty 51 | Package: amsbsy 1999/11/29 v1.2d 52 | \pmbraise@=\dimen104 53 | ) 54 | (/usr/share/texmf-texlive/tex/latex/amsmath/amsopn.sty 55 | Package: amsopn 1999/12/14 v2.01 operator names 56 | ) 57 | \inf@bad=\count87 58 | LaTeX Info: Redefining \frac on input line 211. 59 | \uproot@=\count88 60 | \leftroot@=\count89 61 | LaTeX Info: Redefining \overline on input line 307. 62 | \classnum@=\count90 63 | \DOTSCASE@=\count91 64 | LaTeX Info: Redefining \ldots on input line 379. 65 | LaTeX Info: Redefining \dots on input line 382. 66 | LaTeX Info: Redefining \cdots on input line 467. 67 | \Mathstrutbox@=\box26 68 | \strutbox@=\box27 69 | \big@size=\dimen105 70 | LaTeX Font Info: Redeclaring font encoding OML on input line 567. 71 | LaTeX Font Info: Redeclaring font encoding OMS on input line 568. 72 | \macc@depth=\count92 73 | \c@MaxMatrixCols=\count93 74 | \dotsspace@=\muskip10 75 | \c@parentequation=\count94 76 | \dspbrk@lvl=\count95 77 | \tag@help=\toks16 78 | \row@=\count96 79 | \column@=\count97 80 | \maxfields@=\count98 81 | \andhelp@=\toks17 82 | \eqnshift@=\dimen106 83 | \alignsep@=\dimen107 84 | \tagshift@=\dimen108 85 | \tagwidth@=\dimen109 86 | \totwidth@=\dimen110 87 | \lineht@=\dimen111 88 | \@envbody=\toks18 89 | \multlinegap=\skip44 90 | \multlinetaggap=\skip45 91 | \mathdisplay@stack=\toks19 92 | LaTeX Info: Redefining \[ on input line 2666. 93 | LaTeX Info: Redefining \] on input line 2667. 94 | ) 95 | (/usr/share/texmf-texlive/tex/generic/ifxetex/ifxetex.sty 96 | Package: ifxetex 2009/01/23 v0.5 Provides ifxetex conditional 97 | ) 98 | (/usr/share/texmf-texlive/tex/generic/oberdiek/ifluatex.sty 99 | Package: ifluatex 2009/04/17 v1.2 Provides the ifluatex switch (HO) 100 | Package ifluatex Info: LuaTeX not detected. 101 | ) 102 | (/usr/share/texmf-texlive/tex/latex/base/inputenc.sty 103 | Package: inputenc 2008/03/30 v1.1d Input encoding file 104 | \inpenc@prehook=\toks20 105 | \inpenc@posthook=\toks21 106 | 107 | (/usr/share/texmf-texlive/tex/latex/base/utf8.def 108 | File: utf8.def 2008/04/05 v1.1m UTF-8 support for inputenc 109 | Now handling font encoding OML ... 110 | ... no UTF-8 mapping file for font encoding OML 111 | Now handling font encoding T1 ... 112 | ... processing UTF-8 mapping file for font encoding T1 113 | 114 | (/usr/share/texmf-texlive/tex/latex/base/t1enc.dfu 115 | File: t1enc.dfu 2008/04/05 v1.1m UTF-8 support for inputenc 116 | defining Unicode char U+00A1 (decimal 161) 117 | defining Unicode char U+00A3 (decimal 163) 118 | defining Unicode char U+00AB (decimal 171) 119 | defining Unicode char U+00BB (decimal 187) 120 | defining Unicode char U+00BF (decimal 191) 121 | defining Unicode char U+00C0 (decimal 192) 122 | defining Unicode char U+00C1 (decimal 193) 123 | defining Unicode char U+00C2 (decimal 194) 124 | defining Unicode char U+00C3 (decimal 195) 125 | defining Unicode char U+00C4 (decimal 196) 126 | defining Unicode char U+00C5 (decimal 197) 127 | defining Unicode char U+00C6 (decimal 198) 128 | defining Unicode char U+00C7 (decimal 199) 129 | defining Unicode char U+00C8 (decimal 200) 130 | defining Unicode char U+00C9 (decimal 201) 131 | defining Unicode char U+00CA (decimal 202) 132 | defining Unicode char U+00CB (decimal 203) 133 | defining Unicode char U+00CC (decimal 204) 134 | defining Unicode char U+00CD (decimal 205) 135 | defining Unicode char U+00CE (decimal 206) 136 | defining Unicode char U+00CF (decimal 207) 137 | defining Unicode char U+00D0 (decimal 208) 138 | defining Unicode char U+00D1 (decimal 209) 139 | defining Unicode char U+00D2 (decimal 210) 140 | defining Unicode char U+00D3 (decimal 211) 141 | defining Unicode char U+00D4 (decimal 212) 142 | defining Unicode char U+00D5 (decimal 213) 143 | defining Unicode char U+00D6 (decimal 214) 144 | defining Unicode char U+00D8 (decimal 216) 145 | defining Unicode char U+00D9 (decimal 217) 146 | defining Unicode char U+00DA (decimal 218) 147 | defining Unicode char U+00DB (decimal 219) 148 | defining Unicode char U+00DC (decimal 220) 149 | defining Unicode char U+00DD (decimal 221) 150 | defining Unicode char U+00DE (decimal 222) 151 | defining Unicode char U+00DF (decimal 223) 152 | defining Unicode char U+00E0 (decimal 224) 153 | defining Unicode char U+00E1 (decimal 225) 154 | defining Unicode char U+00E2 (decimal 226) 155 | defining Unicode char U+00E3 (decimal 227) 156 | defining Unicode char U+00E4 (decimal 228) 157 | defining Unicode char U+00E5 (decimal 229) 158 | defining Unicode char U+00E6 (decimal 230) 159 | defining Unicode char U+00E7 (decimal 231) 160 | defining Unicode char U+00E8 (decimal 232) 161 | defining Unicode char U+00E9 (decimal 233) 162 | defining Unicode char U+00EA (decimal 234) 163 | defining Unicode char U+00EB (decimal 235) 164 | defining Unicode char U+00EC (decimal 236) 165 | defining Unicode char U+00ED (decimal 237) 166 | defining Unicode char U+00EE (decimal 238) 167 | defining Unicode char U+00EF (decimal 239) 168 | defining Unicode char U+00F0 (decimal 240) 169 | defining Unicode char U+00F1 (decimal 241) 170 | defining Unicode char U+00F2 (decimal 242) 171 | defining Unicode char U+00F3 (decimal 243) 172 | defining Unicode char U+00F4 (decimal 244) 173 | defining Unicode char U+00F5 (decimal 245) 174 | defining Unicode char U+00F6 (decimal 246) 175 | defining Unicode char U+00F8 (decimal 248) 176 | defining Unicode char U+00F9 (decimal 249) 177 | defining Unicode char U+00FA (decimal 250) 178 | defining Unicode char U+00FB (decimal 251) 179 | defining Unicode char U+00FC (decimal 252) 180 | defining Unicode char U+00FD (decimal 253) 181 | defining Unicode char U+00FE (decimal 254) 182 | defining Unicode char U+00FF (decimal 255) 183 | defining Unicode char U+0102 (decimal 258) 184 | defining Unicode char U+0103 (decimal 259) 185 | defining Unicode char U+0104 (decimal 260) 186 | defining Unicode char U+0105 (decimal 261) 187 | defining Unicode char U+0106 (decimal 262) 188 | defining Unicode char U+0107 (decimal 263) 189 | defining Unicode char U+010C (decimal 268) 190 | defining Unicode char U+010D (decimal 269) 191 | defining Unicode char U+010E (decimal 270) 192 | defining Unicode char U+010F (decimal 271) 193 | defining Unicode char U+0110 (decimal 272) 194 | defining Unicode char U+0111 (decimal 273) 195 | defining Unicode char U+0118 (decimal 280) 196 | defining Unicode char U+0119 (decimal 281) 197 | defining Unicode char U+011A (decimal 282) 198 | defining Unicode char U+011B (decimal 283) 199 | defining Unicode char U+011E (decimal 286) 200 | defining Unicode char U+011F (decimal 287) 201 | defining Unicode char U+0130 (decimal 304) 202 | defining Unicode char U+0131 (decimal 305) 203 | defining Unicode char U+0132 (decimal 306) 204 | defining Unicode char U+0133 (decimal 307) 205 | defining Unicode char U+0139 (decimal 313) 206 | defining Unicode char U+013A (decimal 314) 207 | defining Unicode char U+013D (decimal 317) 208 | defining Unicode char U+013E (decimal 318) 209 | defining Unicode char U+0141 (decimal 321) 210 | defining Unicode char U+0142 (decimal 322) 211 | defining Unicode char U+0143 (decimal 323) 212 | defining Unicode char U+0144 (decimal 324) 213 | defining Unicode char U+0147 (decimal 327) 214 | defining Unicode char U+0148 (decimal 328) 215 | defining Unicode char U+014A (decimal 330) 216 | defining Unicode char U+014B (decimal 331) 217 | defining Unicode char U+0150 (decimal 336) 218 | defining Unicode char U+0151 (decimal 337) 219 | defining Unicode char U+0152 (decimal 338) 220 | defining Unicode char U+0153 (decimal 339) 221 | defining Unicode char U+0154 (decimal 340) 222 | defining Unicode char U+0155 (decimal 341) 223 | defining Unicode char U+0158 (decimal 344) 224 | defining Unicode char U+0159 (decimal 345) 225 | defining Unicode char U+015A (decimal 346) 226 | defining Unicode char U+015B (decimal 347) 227 | defining Unicode char U+015E (decimal 350) 228 | defining Unicode char U+015F (decimal 351) 229 | defining Unicode char U+0160 (decimal 352) 230 | defining Unicode char U+0161 (decimal 353) 231 | defining Unicode char U+0162 (decimal 354) 232 | defining Unicode char U+0163 (decimal 355) 233 | defining Unicode char U+0164 (decimal 356) 234 | defining Unicode char U+0165 (decimal 357) 235 | defining Unicode char U+016E (decimal 366) 236 | defining Unicode char U+016F (decimal 367) 237 | defining Unicode char U+0170 (decimal 368) 238 | defining Unicode char U+0171 (decimal 369) 239 | defining Unicode char U+0178 (decimal 376) 240 | defining Unicode char U+0179 (decimal 377) 241 | defining Unicode char U+017A (decimal 378) 242 | defining Unicode char U+017B (decimal 379) 243 | defining Unicode char U+017C (decimal 380) 244 | defining Unicode char U+017D (decimal 381) 245 | defining Unicode char U+017E (decimal 382) 246 | defining Unicode char U+200C (decimal 8204) 247 | defining Unicode char U+2013 (decimal 8211) 248 | defining Unicode char U+2014 (decimal 8212) 249 | defining Unicode char U+2018 (decimal 8216) 250 | defining Unicode char U+2019 (decimal 8217) 251 | defining Unicode char U+201A (decimal 8218) 252 | defining Unicode char U+201C (decimal 8220) 253 | defining Unicode char U+201D (decimal 8221) 254 | defining Unicode char U+201E (decimal 8222) 255 | defining Unicode char U+2030 (decimal 8240) 256 | defining Unicode char U+2031 (decimal 8241) 257 | defining Unicode char U+2039 (decimal 8249) 258 | defining Unicode char U+203A (decimal 8250) 259 | defining Unicode char U+2423 (decimal 9251) 260 | ) 261 | Now handling font encoding OT1 ... 262 | ... processing UTF-8 mapping file for font encoding OT1 263 | 264 | (/usr/share/texmf-texlive/tex/latex/base/ot1enc.dfu 265 | File: ot1enc.dfu 2008/04/05 v1.1m UTF-8 support for inputenc 266 | defining Unicode char U+00A1 (decimal 161) 267 | defining Unicode char U+00A3 (decimal 163) 268 | defining Unicode char U+00B8 (decimal 184) 269 | defining Unicode char U+00BF (decimal 191) 270 | defining Unicode char U+00C5 (decimal 197) 271 | defining Unicode char U+00C6 (decimal 198) 272 | defining Unicode char U+00D8 (decimal 216) 273 | defining Unicode char U+00DF (decimal 223) 274 | defining Unicode char U+00E6 (decimal 230) 275 | defining Unicode char U+00EC (decimal 236) 276 | defining Unicode char U+00ED (decimal 237) 277 | defining Unicode char U+00EE (decimal 238) 278 | defining Unicode char U+00EF (decimal 239) 279 | defining Unicode char U+00F8 (decimal 248) 280 | defining Unicode char U+0131 (decimal 305) 281 | defining Unicode char U+0141 (decimal 321) 282 | defining Unicode char U+0142 (decimal 322) 283 | defining Unicode char U+0152 (decimal 338) 284 | defining Unicode char U+0153 (decimal 339) 285 | defining Unicode char U+2013 (decimal 8211) 286 | defining Unicode char U+2014 (decimal 8212) 287 | defining Unicode char U+2018 (decimal 8216) 288 | defining Unicode char U+2019 (decimal 8217) 289 | defining Unicode char U+201C (decimal 8220) 290 | defining Unicode char U+201D (decimal 8221) 291 | ) 292 | Now handling font encoding OMS ... 293 | ... processing UTF-8 mapping file for font encoding OMS 294 | 295 | (/usr/share/texmf-texlive/tex/latex/base/omsenc.dfu 296 | File: omsenc.dfu 2008/04/05 v1.1m UTF-8 support for inputenc 297 | defining Unicode char U+00A7 (decimal 167) 298 | defining Unicode char U+00B6 (decimal 182) 299 | defining Unicode char U+00B7 (decimal 183) 300 | defining Unicode char U+2020 (decimal 8224) 301 | defining Unicode char U+2021 (decimal 8225) 302 | defining Unicode char U+2022 (decimal 8226) 303 | ) 304 | Now handling font encoding OMX ... 305 | ... no UTF-8 mapping file for font encoding OMX 306 | Now handling font encoding U ... 307 | ... no UTF-8 mapping file for font encoding U 308 | defining Unicode char U+00A9 (decimal 169) 309 | defining Unicode char U+00AA (decimal 170) 310 | defining Unicode char U+00AE (decimal 174) 311 | defining Unicode char U+00BA (decimal 186) 312 | defining Unicode char U+02C6 (decimal 710) 313 | defining Unicode char U+02DC (decimal 732) 314 | defining Unicode char U+200C (decimal 8204) 315 | defining Unicode char U+2026 (decimal 8230) 316 | defining Unicode char U+2122 (decimal 8482) 317 | defining Unicode char U+2423 (decimal 9251) 318 | )) 319 | (/usr/share/texmf-texlive/tex/latex/graphics/color.sty 320 | Package: color 2005/11/14 v1.0j Standard LaTeX Color (DPC) 321 | 322 | (/etc/texmf/tex/latex/config/color.cfg 323 | File: color.cfg 2007/01/18 v1.5 color configuration of teTeX/TeXLive 324 | ) 325 | Package color Info: Driver file: pdftex.def on input line 130. 326 | 327 | (/usr/share/texmf-texlive/tex/latex/pdftex-def/pdftex.def 328 | File: pdftex.def 2010/03/12 v0.04p Graphics/color for pdfTeX 329 | \Gread@gobject=\count99 330 | )) 331 | (/usr/share/texmf-texlive/tex/latex/fancyvrb/fancyvrb.sty 332 | Package: fancyvrb 2008/02/07 333 | 334 | Style option: `fancyvrb' v2.7a, with DG/SPQR fixes, and firstline=lastline fix 335 | <2008/02/07> (tvz) (/usr/share/texmf-texlive/tex/latex/graphics/keyval.sty 336 | Package: keyval 1999/03/16 v1.13 key=value parser (DPC) 337 | \KV@toks@=\toks22 338 | ) 339 | \FV@CodeLineNo=\count100 340 | \FV@InFile=\read1 341 | \FV@TabBox=\box28 342 | \c@FancyVerbLine=\count101 343 | \FV@StepNumber=\count102 344 | \FV@OutFile=\write3 345 | ) 346 | (/usr/share/texmf-texlive/tex/latex/graphics/graphicx.sty 347 | Package: graphicx 1999/02/16 v1.0f Enhanced LaTeX Graphics (DPC,SPQR) 348 | 349 | (/usr/share/texmf-texlive/tex/latex/graphics/graphics.sty 350 | Package: graphics 2009/02/05 v1.0o Standard LaTeX Graphics (DPC,SPQR) 351 | 352 | (/usr/share/texmf-texlive/tex/latex/graphics/trig.sty 353 | Package: trig 1999/03/16 v1.09 sin cos tan (DPC) 354 | ) 355 | (/etc/texmf/tex/latex/config/graphics.cfg 356 | File: graphics.cfg 2009/08/28 v1.8 graphics configuration of TeX Live 357 | ) 358 | Package graphics Info: Driver file: pdftex.def on input line 91. 359 | ) 360 | \Gin@req@height=\dimen112 361 | \Gin@req@width=\dimen113 362 | ) 363 | (/usr/share/texmf-texlive/tex/latex/hyperref/hyperref.sty 364 | Package: hyperref 2009/10/09 v6.79a Hypertext links for LaTeX 365 | 366 | (/usr/share/texmf-texlive/tex/generic/oberdiek/ifpdf.sty 367 | Package: ifpdf 2009/04/10 v2.0 Provides the ifpdf switch (HO) 368 | Package ifpdf Info: pdfTeX in pdf mode detected. 369 | ) 370 | (/usr/share/texmf-texlive/tex/generic/oberdiek/ifvtex.sty 371 | Package: ifvtex 2008/11/04 v1.4 Switches for detecting VTeX and its modes (HO) 372 | Package ifvtex Info: VTeX not detected. 373 | ) 374 | (/usr/share/texmf-texlive/tex/latex/oberdiek/hycolor.sty 375 | Package: hycolor 2009/10/02 v1.5 Code for color options of hyperref/bookmark (H 376 | O) 377 | 378 | (/usr/share/texmf-texlive/tex/latex/oberdiek/xcolor-patch.sty 379 | Package: xcolor-patch 2009/10/02 xcolor patch 380 | )) 381 | \@linkdim=\dimen114 382 | \Hy@linkcounter=\count103 383 | \Hy@pagecounter=\count104 384 | 385 | (/usr/share/texmf-texlive/tex/latex/hyperref/pd1enc.def 386 | File: pd1enc.def 2009/10/09 v6.79a Hyperref: PDFDocEncoding definition (HO) 387 | Now handling font encoding PD1 ... 388 | ... no UTF-8 mapping file for font encoding PD1 389 | ) 390 | (/usr/share/texmf-texlive/tex/generic/oberdiek/etexcmds.sty 391 | Package: etexcmds 2007/12/12 v1.2 Prefix for e-TeX command names (HO) 392 | 393 | (/usr/share/texmf-texlive/tex/generic/oberdiek/infwarerr.sty 394 | Package: infwarerr 2007/09/09 v1.2 Providing info/warning/message (HO) 395 | ) 396 | Package etexcmds Info: Could not find \expanded. 397 | (etexcmds) That can mean that you are not using pdfTeX 1.50 or 398 | (etexcmds) that some package has redefined \expanded. 399 | (etexcmds) In the latter case, load this package earlier. 400 | ) 401 | (/usr/share/texmf-texlive/tex/latex/latexconfig/hyperref.cfg 402 | File: hyperref.cfg 2002/06/06 v1.2 hyperref configuration of TeXLive 403 | ) 404 | (/usr/share/texmf-texlive/tex/latex/oberdiek/kvoptions.sty 405 | Package: kvoptions 2009/08/13 v3.4 Keyval support for LaTeX options (HO) 406 | 407 | (/usr/share/texmf-texlive/tex/generic/oberdiek/kvsetkeys.sty 408 | Package: kvsetkeys 2009/07/30 v1.5 Key value parser with default handler suppor 409 | t (HO) 410 | )) 411 | Package hyperref Info: Option `unicode' set `true' on input line 2864. 412 | 413 | (/usr/share/texmf-texlive/tex/latex/hyperref/puenc.def 414 | File: puenc.def 2009/10/09 v6.79a Hyperref: PDF Unicode definition (HO) 415 | Now handling font encoding PU ... 416 | ... no UTF-8 mapping file for font encoding PU 417 | ) 418 | Package hyperref Info: Option `colorlinks' set `true' on input line 2864. 419 | Package hyperref Info: Hyper figures OFF on input line 2975. 420 | Package hyperref Info: Link nesting OFF on input line 2980. 421 | Package hyperref Info: Hyper index ON on input line 2983. 422 | Package hyperref Info: Plain pages OFF on input line 2990. 423 | Package hyperref Info: Backreferencing OFF on input line 2995. 424 | 425 | Implicit mode ON; LaTeX internals redefined 426 | Package hyperref Info: Bookmarks ON on input line 3191. 427 | (/usr/share/texmf-texlive/tex/latex/ltxmisc/url.sty 428 | \Urlmuskip=\muskip11 429 | Package: url 2006/04/12 ver 3.3 Verb mode for urls, etc. 430 | ) 431 | LaTeX Info: Redefining \url on input line 3428. 432 | 433 | (/usr/share/texmf-texlive/tex/generic/oberdiek/bitset.sty 434 | Package: bitset 2007/09/28 v1.0 Data type bit set (HO) 435 | 436 | (/usr/share/texmf-texlive/tex/generic/oberdiek/intcalc.sty 437 | Package: intcalc 2007/09/27 v1.1 Expandable integer calculations (HO) 438 | ) 439 | (/usr/share/texmf-texlive/tex/generic/oberdiek/bigintcalc.sty 440 | Package: bigintcalc 2007/11/11 v1.1 Expandable big integer calculations (HO) 441 | 442 | (/usr/share/texmf-texlive/tex/generic/oberdiek/pdftexcmds.sty 443 | Package: pdftexcmds 2009/09/23 v0.6 LuaTeX support for pdfTeX utility functions 444 | (HO) 445 | 446 | (/usr/share/texmf-texlive/tex/generic/oberdiek/ltxcmds.sty 447 | Package: ltxcmds 2009/08/05 v1.0 Some LaTeX kernel commands for general use (HO 448 | ) 449 | ) 450 | Package pdftexcmds Info: LuaTeX not detected. 451 | Package pdftexcmds Info: \pdf@primitive is available. 452 | Package pdftexcmds Info: \pdf@ifprimitive is available. 453 | ))) 454 | \Fld@menulength=\count105 455 | \Field@Width=\dimen115 456 | \Fld@charsize=\dimen116 457 | \Field@toks=\toks23 458 | Package hyperref Info: Hyper figures OFF on input line 4377. 459 | Package hyperref Info: Link nesting OFF on input line 4382. 460 | Package hyperref Info: Hyper index ON on input line 4385. 461 | Package hyperref Info: backreferencing OFF on input line 4392. 462 | Package hyperref Info: Link coloring ON on input line 4395. 463 | Package hyperref Info: Link coloring with OCG OFF on input line 4402. 464 | Package hyperref Info: PDF/A mode OFF on input line 4407. 465 | 466 | (/usr/share/texmf-texlive/tex/generic/oberdiek/atbegshi.sty 467 | Package: atbegshi 2008/07/31 v1.9 At begin shipout hook (HO) 468 | ) 469 | \Hy@abspage=\count106 470 | \c@Item=\count107 471 | \c@Hfootnote=\count108 472 | ) 473 | *hyperref using default driver hpdftex* 474 | (/usr/share/texmf-texlive/tex/latex/hyperref/hpdftex.def 475 | File: hpdftex.def 2009/10/09 v6.79a Hyperref driver for pdfTeX 476 | \Fld@listcount=\count109 477 | ) 478 | Package hyperref Info: Option `breaklinks' set `true' on input line 56. 479 | 480 | (/usr/share/texmf-texlive/tex/latex/geometry/geometry.sty 481 | Package: geometry 2008/12/21 v4.2 Page Geometry 482 | \Gm@cnth=\count110 483 | \Gm@cntv=\count111 484 | \c@Gm@tempcnt=\count112 485 | \Gm@bindingoffset=\dimen117 486 | \Gm@wd@mp=\dimen118 487 | \Gm@odd@mp=\dimen119 488 | \Gm@even@mp=\dimen120 489 | \Gm@dimlist=\toks24 490 | ) (./osm.aux) 491 | \openout1 = `osm.aux'. 492 | 493 | LaTeX Font Info: Checking defaults for OML/cmm/m/it on input line 69. 494 | LaTeX Font Info: ... okay on input line 69. 495 | LaTeX Font Info: Checking defaults for T1/cmr/m/n on input line 69. 496 | LaTeX Font Info: ... okay on input line 69. 497 | LaTeX Font Info: Checking defaults for OT1/cmr/m/n on input line 69. 498 | LaTeX Font Info: ... okay on input line 69. 499 | LaTeX Font Info: Checking defaults for OMS/cmsy/m/n on input line 69. 500 | LaTeX Font Info: ... okay on input line 69. 501 | LaTeX Font Info: Checking defaults for OMX/cmex/m/n on input line 69. 502 | LaTeX Font Info: ... okay on input line 69. 503 | LaTeX Font Info: Checking defaults for U/cmr/m/n on input line 69. 504 | LaTeX Font Info: ... okay on input line 69. 505 | LaTeX Font Info: Checking defaults for PD1/pdf/m/n on input line 69. 506 | LaTeX Font Info: ... okay on input line 69. 507 | LaTeX Font Info: Checking defaults for PU/pdf/m/n on input line 69. 508 | LaTeX Font Info: ... okay on input line 69. 509 | 510 | (/usr/share/texmf-texlive/tex/context/base/supp-pdf.mkii 511 | [Loading MPS to PDF converter (version 2006.09.02).] 512 | \scratchcounter=\count113 513 | \scratchdimen=\dimen121 514 | \scratchbox=\box29 515 | \nofMPsegments=\count114 516 | \nofMParguments=\count115 517 | \everyMPshowfont=\toks25 518 | \MPscratchCnt=\count116 519 | \MPscratchDim=\dimen122 520 | \MPnumerator=\count117 521 | \everyMPtoPDFconversion=\toks26 522 | ) 523 | Package hyperref Info: Link coloring ON on input line 69. 524 | (/usr/share/texmf-texlive/tex/latex/hyperref/nameref.sty 525 | Package: nameref 2007/05/29 v2.31 Cross-referencing by name of section 526 | 527 | (/usr/share/texmf-texlive/tex/latex/oberdiek/refcount.sty 528 | Package: refcount 2008/08/11 v3.1 Data extraction from references (HO) 529 | ) 530 | \c@section@level=\count118 531 | ) 532 | LaTeX Info: Redefining \ref on input line 69. 533 | LaTeX Info: Redefining \pageref on input line 69. 534 | (./osm.out) 535 | (./osm.out) 536 | \@outlinefile=\write4 537 | \openout4 = `osm.out'. 538 | 539 | \AtBeginShipoutBox=\box30 540 | 541 | *geometry auto-detecting driver* 542 | *geometry detected driver: pdftex* 543 | -------------------- Geometry parameters 544 | paper: class default 545 | landscape: -- 546 | twocolumn: -- 547 | twoside: -- 548 | asymmetric: -- 549 | h-parts: 51.21504pt, 511.86491pt, 51.21504pt 550 | v-parts: 51.21504pt, 692.5399pt, 51.21504pt 551 | hmarginratio: -- 552 | vmarginratio: -- 553 | lines: -- 554 | heightrounded: -- 555 | bindingoffset: 0.0pt 556 | truedimen: -- 557 | includehead: -- 558 | includefoot: -- 559 | includemp: -- 560 | driver: pdftex 561 | -------------------- Page layout dimensions and switches 562 | \paperwidth 614.295pt 563 | \paperheight 794.96999pt 564 | \textwidth 511.86491pt 565 | \textheight 692.5399pt 566 | \oddsidemargin -21.05495pt 567 | \evensidemargin -21.05495pt 568 | \topmargin -58.05495pt 569 | \headheight 12.0pt 570 | \headsep 25.0pt 571 | \footskip 30.0pt 572 | \marginparwidth 65.0pt 573 | \marginparsep 11.0pt 574 | \columnsep 10.0pt 575 | \skip\footins 9.0pt plus 4.0pt minus 2.0pt 576 | \hoffset 0.0pt 577 | \voffset 0.0pt 578 | \mag 1000 579 | 580 | (1in=72.27pt, 1cm=28.45pt) 581 | ----------------------- 582 | LaTeX Font Info: Try loading font information for U+msa on input line 71. 583 | (/usr/share/texmf-texlive/tex/latex/amsfonts/umsa.fd 584 | File: umsa.fd 2009/06/22 v3.00 AMS symbols A 585 | ) 586 | LaTeX Font Info: Try loading font information for U+msb on input line 71. 587 | 588 | (/usr/share/texmf-texlive/tex/latex/amsfonts/umsb.fd 589 | File: umsb.fd 2009/06/22 v3.00 AMS symbols B 590 | ) 591 | LaTeX Font Info: Try loading font information for OMS+cmr on input line 115. 592 | 593 | 594 | (/usr/share/texmf-texlive/tex/latex/base/omscmr.fd 595 | File: omscmr.fd 1999/05/25 v2.5h Standard LaTeX font definitions 596 | ) 597 | LaTeX Font Info: Font shape `OMS/cmr/m/n' in size <10> not available 598 | (Font) Font shape `OMS/cmsy/m/n' tried instead on input line 115. 599 | [1 600 | 601 | {/var/lib/texmf/fonts/map/pdftex/updmap/pdftex.map}] 602 |
603 | File: figure/Manual_selection_of_bounding_box.png Graphic file (type png) 604 | 605 | 606 | LaTeX Font Info: Try loading font information for OML+cmtt on input line 232 607 | . 608 | LaTeX Font Info: No file OMLcmtt.fd. on input line 232. 609 | 610 | 611 | LaTeX Font Warning: Font shape `OML/cmtt/m/n' undefined 612 | (Font) using `OML/cmm/m/it' instead 613 | (Font) for symbol `textgreater' on input line 232. 614 | 615 | 616 | File: osmfigs/import-osm.png Graphic file (type png) 617 | 618 | 619 | 620 | File: osmfigs/open-osmdb.png Graphic file (type png) 621 | 622 | [2] [3 <./figure/Manual_selection_of_bounding_box. 623 | png (PNG copy)> <./osmfigs/import-osm.png> <./osmfigs/open-osmdb.png>] 624 | LaTeX Font Info: Font shape `OT1/cmtt/bx/n' in size <10> not available 625 | (Font) Font shape `OT1/cmtt/m/n' tried instead on input line 281. 626 | 627 |
629 | File: figure/Preliminary_plot_of_Chapletown_with_osmar.png Graphic file (type p 630 | ng) 631 | [4] [5 <./figure/Pr 632 | eliminary_plot_of_Chapletown_with_osmar.png>] 633 |
634 | File: figure/Bike_paths_of_Chapeltown.png Graphic file (type png) 635 | 636 | 637 |
638 | File: figure/Residential_streets_in_Chapeltown.png Graphic file (type png) 639 | 640 | [6] [7 <./figure/Bike_paths_ 641 | of_Chapeltown.png> <./figure/Residential_streets_in_Chapeltown.png>] 642 |
643 | File: figure/Line_data_of_Potternewton_park.png Graphic file (type png) 644 | 645 | [8 <./figure/Line_data_of_Potte 646 | rnewton_park.png>] 647 |
649 | File: figure/All_OSM_data_in_and_directly_surround_Potternewton_Park.png Graphi 650 | c file (type png) 651 | 652 | 653 | Underfull \hbox (badness 1990) in paragraph at lines 555--561 654 | []\OT1/cmr/m/n/10 For fur-ther func-tion-al-ity, in-clud-ing clip-ping to poly- 655 | gons, ex-tract-ing files from enor-mous and com-pressed 656 | [] 657 | 658 | [9 <./figure/All_OSM_data_in_and_directly_surround_Potternewton_Park.png>] 659 | [10] (./osm.aux) 660 | 661 | LaTeX Font Warning: Some font shapes were not available, defaults substituted. 662 | 663 | ) 664 | Here is how much of TeX's memory you used: 665 | 7362 strings out of 495061 666 | 98912 string characters out of 1182621 667 | 180574 words of memory out of 3000000 668 | 10398 multiletter control sequences out of 15000+50000 669 | 10347 words of font info for 40 fonts, out of 3000000 for 9000 670 | 28 hyphenation exceptions out of 8191 671 | 38i,6n,39p,293b,431s stack positions out of 5000i,500n,10000p,200000b,50000s 672 | 681 | Output written on osm.pdf (10 pages, 898623 bytes). 682 | PDF statistics: 683 | 199 PDF objects out of 1000 (max. 8388607) 684 | 29 named destinations out of 1000 (max. 500000) 685 | 121 words of extra memory for PDF output out of 10000 (max. 10000000) 686 | 687 | -------------------------------------------------------------------------------- /osm.md: -------------------------------------------------------------------------------- 1 | Open Street Map: loading, analysing and visualising free maps with R and QGIS 2 | ======================================================== 3 | 4 | This tutorial shows how open source tools can be used to 5 | harness a huge and rapidly growing open source geodatabase: Open Street Map. 6 | It is targeted at people new to Open Street Map, but who 7 | already have an understanding of basic GIS concepts. 8 | Previous experience with the programs QGIS and R would be beneficial, 9 | but not essential, for completing the exercises. There are 10 | also a number of resources available on-line for more advanced functions, 11 | as described below. 12 | 13 | # Introduction 14 | 15 | Open Street Map (OSM) is a crowd-sourced map of the world, 16 | the archetype of 'volunteered geographical information' 17 | ([Goodchild 2007](http://www.ncgia.ucsb.edu/projects/vgi/docs/position/Goodchild_VGI2007.pdf)). 18 | The aim is simple: "to create a set of map data that's free 19 | to use, editable, and licensed under new 20 | copyright schemes", as described in an excellent review 21 | article on the subject 22 | ([Haklay and Weber 2008](http://discovery.ucl.ac.uk/13849/1/13849.pdf)). 23 | Putting the public in charge of 24 | editing the world's surface may seem like a risky 25 | business, given that cartographers have specialist skills 26 | developed over centuries. This issue is described in 27 | [Mapping the Void](https://dl.dropboxusercontent.com/u/15008199/egs2stay/Mapping_the_Void_-_Mapping_the_Void_b03s6mf0_default.m4a), an excellent BBC Radio 4 Documentary 28 | on the subject (Salisbury and Jenkins 2014). 29 | Yet the emergence 30 | of high resolution aerial photography covering the entirety 31 | of the Earth's surface and the 32 | explosion in GPS ownership via smartphones has enabled citizens 33 | to become accurate sensors of the world. 34 | [Neis et al. (2012)](http://www.mdpi.com/1999-5903/4/1/1/pdf) 35 | believe this phenomenon is more than merely technological. 36 | In OSM, they see a "revolutionary paradigm shift on how map data is 37 | being collected". 38 | Citizen mappers have the added advantage that they may know their local 39 | area far better than any professional cartographer. 40 | 41 | This tutorial adds a small nugget of information to the growing literature 42 | on OSM, by demonstrating how the data can be accessed for teaching or research 43 | materials. It's a completely free and open dataset, so we may as well use it! 44 | There is already some good on-line material about OSM data, including: 45 | 46 | - a [paper](http://www.mdpi.com/1999-5903/5/2/282/pdf) comparing the quality and 47 | coverage of OSM map data in different parts of the world (Neis et al. 2013) 48 | - an [overview](http://www.library.carleton.ca/sites/default/files/help/gis/WorkingWithOpenStreetMap.pdf) 49 | of handling OSM data in ArcMap 50 | - a [tutorial](http://elogeo.nottingham.ac.uk/xmlui/bitstream/handle/url/289/osm-tutorial-final-2.pdf?sequence=1) illustrating 51 | its potential use for GIS education and store location planning (Lovelace 2013). 52 | 53 | Yet nothing has focused on simply loading the data using different tools. 54 | In this paper we use QGIS and R because these are (arguably) the most popular 55 | open source programs used by geographers, the discipline with most to gain 56 | from OSM data. Before delving into the method (which is actually quite straightforward), 57 | let's put OSM data in context. 58 | 59 | ## Why (and why not) use OSM data? 60 | 61 | Of course there are teething issues with any large-scale open source 62 | database, including variable data quality, 63 | patchy and incomplete coverage and inconsistencies from place to place (Haklay 2010). Yet all of these 64 | issues are gradually being ironed out. 65 | The advantages of Open Street Map outweigh these downsides for 66 | many applications *already*. These include: 67 | 68 | - Rapid updates of new projects 69 | - Greater range of attributes (e.g. shop names) 70 | - Ability to share data with anyone without breaching license 71 | 72 | In additions there are a number of ethical benefits of using OSM: it's community 73 | a map for the greater good ([Wroclawski 2014](http://www.theguardian.com/technology/2014/jan/14/why-the-world-needs-openstreetmap)). 74 | 75 | There is a strong community of OSM volunteers who use the service for humanitarian purposes 76 | (Salisbury and Jenkins 2014). 77 | Two excellent examples of this help underline the 78 | ethical side of OSM data. [Tindale](http://explore.ramanitanzania.org/), 79 | a settlement in Tanzania that has been mapped rapidly 80 | thanks to a single academic project, in collaboration with the authorities, enabling 81 | better policy making in the area (see [video](http://www.youtube.com/watch?v=KqrGyvNnWkA)). 82 | Second, 83 | the rapid response of the OSM community to the Typhoon Haiyan disaster. In a matter of days, an 84 | army of volunteers had helped to map out the affected zone, aiding relief efforts (e.g. see 85 | [MapBox's blog posts on the subject](https://www.mapbox.com/blog/typhoon-haiyan-openstreetmap/)). 86 | 87 | ## An overview of the tutorial 88 | 89 | All of the code and data used to create this tutorial is available 90 | on [GitHub](http://github.com). Feel free to download the project as a `.zip` file from the 91 | [project's repository](https://github.com/Robinlovelace/osm-tutorial) entitled osm-tutorial from 92 | github.com/Robinlovelace. This zip file contains all of the code and data 93 | used to generate the tutorial, which is entirely reproducible. If you would like to modify or improve it 94 | in any way, please fork a version for you own use, citing the original where appropriate. 95 | 96 | The focus is mainly on the technical challenge of extracting the 97 | data from servers 'in the cloud' and onto your desktop. We also cover some basic 98 | tasks in handling, subsetting and visualising the data, first in the 99 | graphical user interface (GUI) of QGIS, and then in the command line with R. 100 | The next stage talks about editing raw OSM data, essential if you 101 | would like to take subsets of very large OSM datasets, such as `planet.osm` 102 | which takes up more than 30 GB of memory. The next stage is 103 | to talk about OSM data in PostGIS, a geodatabase program ideal 104 | for querying very large spatial datasets. Finally, there is a brief 105 | section on further resources. None of the sections require any of the 106 | previous ones although the level of difficulty generally increases 107 | as you progress. 108 | 109 | # Getting the data 110 | 111 | OSM data of a specific area 112 | can be downloaded directly from the [main map page](http://www.openstreetmap.org), from the 113 | [Overpass API](http://overpass-api.de/) or, for the entire planet, from the huge (currently 32 GB) 114 | [planet.osm file](http://planet.openstreetmap.org/). A number of third parties also provide more manageable 115 | chunks of this dataset, such as the single country datasets provided by 116 | [GEOFABIK](http://download.geofabrik.de/). Command line programs 117 | [Osmosis](http://wiki.openstreetmap.org/wiki/Osmosis) and 118 | [Osm2pgsl](http://wiki.openstreetmap.org/wiki/Osm2pgsql) can be used to process raw OSM data 119 | in either `.osm` or `.osm.pbf` file formats. The former is essentially a `.xml` (Extensible Markup Language) 120 | text file (encoded with the popular UTF-8 characterset); the latter is a compressed version of the former. 121 | How we transfer these datasets into a useful form depends on the program you are using. 122 | In this tutorial we will explain how to do it in QGIS and R, as well describing the basics of 123 | getting it into a [PostGIS](http://postgis.net/) database. 124 | 125 | # OSM data in QGIS 126 | 127 | A `.osm` file can be downloaded from the openstreetmap.org with the bounding box selected by 128 | default depending on the current view, or via a manual selection, as shown below. 129 | 130 | ![plot of chunk Manual selection of bounding box](figure/Manual_selection_of_bounding_box.png) 131 | 132 | 133 | To load this file into QGIS, you can simply use the `Add Vector Layer` button on the 134 | left of the screen. However this does not generate satisfactory results. 135 | The *recommended* way to import the data is via the the OSM plugin. When this is 136 | installed in QGIS 2.0, use the menus `Vector > OpenStreetMap` to import the xml file 137 | and convert it into a SpatiaLite database. Then you can import it into the QGIS workspace. 138 | 139 | ![Import the osm data to SpatiaLite](osmfigs/import-osm.png) 140 | 141 | After this step the file has been saved as a `.osm.db` file. Use the 142 | `Export Topology to SpatiaLite` element of the same menu to 143 | load the file. Choose the type of spatial data you would like to load - 144 | Points, Lines or Polygons. At this stage one can also select which variables 145 | ("Tags") you would like to add to the attribute data. 146 | 147 | ![Select Polylines](osmfigs/open-osmdb.png) 148 | 149 | The data is now loaded into QGIS allowing standard methods of analysis. 150 | You will notice that the data are not styled at all, leading to very bland 151 | maps. To counter this, there have been custom styles developed for visualising OSM data in QGIS, 152 | e.g. [those by Anita Grazer](http://anitagraser.com/2012/02/25/light-styles-for-osm-layers-in-qgis/). 153 | Unfortunately these files do not seem to be working with the current version of QGIS so 154 | alternative ready-made styles must be created, as suggested by a 155 | [stackexchange question](http://gis.stackexchange.com/questions/42645/is-there-up-to-date-osm-sld-file-for-geoserver). 156 | 157 | Once the data is loaded into QGIS, it can be used as with any other spatial data. 158 | Next, let's see how R can handle OSM data, via the `osmar` package. 159 | 160 | # Using osmar 161 | 162 | `osmar` is an R package for downloading and interrogating OSM data that accesses 163 | the data directly from the internet via the R command line. 164 | There is an excellent 165 | [online tutorial](http://journal.r-project.org/archive/2013-1/eugster-schlesinger.pdf) 166 | which provides a detailed account of the package (Eugster & Schlesinger, 2012). 167 | Here we will focus on loading some basic data on bicycle paths in Leeds. 168 | First the package must be loaded: 169 | 170 | 171 | ```r 172 | library(osmar) # if the package is not already installed, use install.packages('osmar') 173 | ``` 174 | 175 | ``` 176 | ## Loading required package: XML 177 | ## Loading required package: RCurl 178 | ## Loading required package: bitops 179 | ## Loading required package: gtools 180 | ## Loading required package: geosphere 181 | ## Loading required package: sp 182 | ## 183 | ## Attaching package: 'osmar' 184 | ## 185 | ## The following object is masked from 'package:utils': 186 | ## 187 | ## find 188 | ``` 189 | 190 | 191 | To download data directly, one first sets the source and a bounding box, 192 | and then use the `get_osm` function to download it. Selecting Chapeltown as the 193 | centrepoint of the map, we can download all the data in the square km surrounding it. 194 | 195 | 196 | ```r 197 | src <- osmsource_api() 198 | bb <- center_bbox(-1.53492, 53.81934, 1000, 1000) 199 | ctown <- get_osm(bb, source = src) 200 | plot(ctown) 201 | points(-1.53492, 53.81934, col = "red", lwd = 5) 202 | ``` 203 | 204 | ![plot of chunk Preliminary plot of Chapletown with osmar](figure/Preliminary_plot_of_Chapletown_with_osmar.png) 205 | 206 | 207 | This shows that the data has successfully been loaded and saved as an 208 | object called `ctown`. Let's try analysing this object further. 209 | In fact, `ctown` is technically a list, composed of 3 objects: 210 | nodes (points), ways (lines) and relations (polygons composed of 211 | many lines). Such OSM data is thus provided a class of its own, 212 | and each sub-object can be called separately using the `$` symbol: 213 | 214 | 215 | ```r 216 | names(ctown) 217 | ``` 218 | 219 | ``` 220 | ## [1] "nodes" "ways" "relations" 221 | ``` 222 | 223 | ```r 224 | class(ctown) 225 | ``` 226 | 227 | ``` 228 | ## [1] "osmar" "list" 229 | ``` 230 | 231 | ```r 232 | summary(ctown$ways) 233 | ``` 234 | 235 | ``` 236 | ## osmar$ways object 237 | ## 180 ways, 509 tags, 1272 refs 238 | ## 239 | ## ..$attrs data.frame: 240 | ## id, changeset, timestamp, version, visible, user, uid 241 | ## ..$tags data.frame: 242 | ## id, k, v 243 | ## ..$refs data.frame: 244 | ## id, ref 245 | ## 246 | ## Key-Value contingency table: 247 | ## Key Value Freq 248 | ## 1 highway residential 72 249 | ## 2 source:name OS_OpenData_Locator 50 250 | ## 3 highway service 41 251 | ## 4 created_by JOSM 25 252 | ## 5 source Bing 23 253 | ## 6 highway unclassified 17 254 | ## 7 highway footway 15 255 | ## 8 building yes 11 256 | ## 9 lit yes 11 257 | ## 10 source npe 10 258 | ``` 259 | 260 | 261 | Let's use the dataset we have loaded to investigate the cycle 262 | paths in the vicinity of my house. First we need to understand the data 263 | contained in the object. Let's look at the tags and the attributes of the `ways` object: 264 | 265 | 266 | ```r 267 | summary(ctown$ways$tags) # summary of the tag data 268 | ``` 269 | 270 | ``` 271 | ## id k v 272 | ## Min. :5.09e+06 highway :156 residential : 72 273 | ## 1st Qu.:7.73e+06 name :129 OS_OpenData_Locator: 50 274 | ## Median :8.46e+07 source : 57 service : 41 275 | ## Mean :8.95e+07 source:name: 55 yes : 32 276 | ## 3rd Qu.:1.50e+08 created_by : 25 JOSM : 25 277 | ## Max. :2.60e+08 building : 12 Bing : 23 278 | ## (Other) : 75 (Other) :266 279 | ``` 280 | 281 | ```r 282 | ctown$ways$attrs[1:8, 1:6] # attributes of first 8 ways - see I'm in there! 283 | ``` 284 | 285 | ``` 286 | ## id changeset timestamp version visible user 287 | ## 1 5088536 15128484 2013-02-22 22:08:24 13 true CompactDstrxion 288 | ## 2 22818969 13039300 2012-09-08 23:06:53 20 true LeedsTracker 289 | ## 3 6273628 483846 2007-09-21 17:25:37 1 true SteveC 290 | ## 4 6273619 14114856 2012-12-01 17:57:45 2 true sc71 291 | ## 5 6273721 444107 2007-09-16 17:23:14 1 true noii 292 | ## 6 6273722 444107 2007-09-16 17:23:16 1 true noii 293 | ## 7 6273726 444107 2007-09-16 17:23:20 1 true noii 294 | ## 8 6273736 18672988 2013-11-02 10:56:24 5 true RobinLovelace 295 | ``` 296 | 297 | 298 | From looking at the [OSM tagging system](http://wiki.openstreetmap.org/wiki/Tags), we can deduce that 299 | `id` is the element's id, 300 | `k` refers to the OSM key (the variables for which the element 301 | has values) and that `v` is the value assigned for each 302 | id - key combination. Because OSM data is not a simple data frame, 303 | we cannot use the usual R notation for subsetting. Instead we use the 304 | `find` function. Let us take a subset of bicycle paths in the area 305 | and plot them. 306 | 307 | 308 | ```r 309 | bikePaths <- find(ctown, way(tags(k == "bicycle" & v == "yes"))) 310 | bikePaths <- find_down(ctown, way(bikePaths)) 311 | bikePaths <- subset(ctown, ids = bikePaths) 312 | plot(ctown) 313 | plot_ways(bikePaths, add = T, col = "red", lwd = 3) 314 | ``` 315 | 316 | ![plot of chunk Bike paths of Chapeltown](figure/Bike_paths_of_Chapeltown1.png) 317 | 318 | ```r 319 | save(bikePaths, file = "data/bike-paths-lds.RData") 320 | 321 | # analysis of time of addition 322 | class(bikePaths$nodes$attrs$timestamp) 323 | ``` 324 | 325 | ``` 326 | ## [1] "POSIXlt" "POSIXt" 327 | ``` 328 | 329 | ```r 330 | tstamp <- as.POSIXct(bikePaths$nodes$attrs$timestamp, format = ) 331 | hist(bikePaths$nodes$attrs$timestamp, breaks = "year") 332 | ``` 333 | 334 | ![plot of chunk Bike paths of Chapeltown](figure/Bike_paths_of_Chapeltown2.png) 335 | 336 | 337 | The above code block is used to identify all ways in which cycling 338 | is permitted, "overriding default access", according OSM's excellent 339 | [wiki page on bicycle paths](http://wiki.openstreetmap.org/wiki/Bicycle). 340 | 341 | According to this source, the correct way to refer to an on-road cycle path 342 | is with the `cycleway` tag. However, none of these have been added to the 343 | roads that have on-road cycle lanes in this example dataset (as of January 2014). 344 | Perhaps someone will add these soon. 345 | 346 | 347 | ```r 348 | which(ctown$ways$tags$k == "cycleway") 349 | ``` 350 | 351 | ``` 352 | ## integer(0) 353 | ``` 354 | 355 | 356 | There are, by contrast, a large number of ways classified as "residential". 357 | Let us use the same method to select them and add them to the map. 358 | 359 | 360 | ```r 361 | res <- find(ctown, way(tags(k == "highway" & v == "residential"))) 362 | res <- find_down(ctown, way(res)) 363 | res <- subset(ctown, ids = res) 364 | plot(ctown) 365 | plot_ways(res, add = T, col = "green", lwd = 3) 366 | ``` 367 | 368 | ![plot of chunk Residential streets in Chapeltown](figure/Residential_streets_in_Chapeltown.png) 369 | 370 | 371 | # Handling raw OSM data 372 | 373 | Although this section will be of most use for dealing with 374 | very large files, a small example is used here to showcase the methods 375 | The file `map.osm` downloaded directly from [openstreetmap.org](http://www.openstreetmap.org) 376 | is small, allowing available on-line, from 377 | [here](https://github.com/Robinlovelace/osm-tutorial/blob/master/data/map.osm?raw=true) 378 | (highlighting the advantages 379 | of open data for educational purposes - there are few license restrictions). 380 | 381 | The Java command line tool [Osmosis](http://wiki.openstreetmap.org/wiki/Osmosis) 382 | is used for this purpose. 383 | 384 | To show how Osmosis can subset raw OSM files, let us take a real world example from 385 | the dataset we imported with the `osmar` package. Imagine that we want to extract all 386 | osm data (nodes, ways and relations) from the area in and around Potternewton Park. 387 | We could do this manually by looking up the park on-line and selecting the associated 388 | elements in QGIS. But imagine we want to do it from the 389 | command line (e.g. for batch processing). 390 | First we need a bounding box of all items containing the 391 | text string "Potternewton Park". For this we use `osmar`: 392 | 393 | 394 | ```r 395 | potter <- find(ctown, way(tags(grepl("Potternewton Park", ctown$ways$tags$v)))) 396 | potter <- find_down(ctown, way(potter)) 397 | potter <- subset(ctown, ids = potter) 398 | plot_ways(potter) # sanity check 399 | ``` 400 | 401 | ![plot of chunk Line data of Potternewton park](figure/Line_data_of_Potternewton_park.png) 402 | 403 | ```r 404 | potter.l <- as_sp(potter, "lines") # convert data to sp class 405 | b <- bbox(potter.l) # save the bounding box 406 | b[1, ] <- (b[1, ] - mean(b[1, ])) * 1.05 + mean(b[1, ]) 407 | b[2, ] <- (b[2, ] - mean(b[2, ])) * 1.05 + mean(b[2, ]) 408 | # scale longitude and latitude (increase bb by 5% for plot) replace 1.05 409 | # with 1.xx for an xx% increase in the plot size 410 | b 411 | ``` 412 | 413 | ``` 414 | ## min max 415 | ## x -1.529 -1.521 416 | ## y 53.817 53.822 417 | ``` 418 | 419 | 420 | Now that we know the bounding box, we can transfer to using osmosis from 421 | the command line. If the shell is open in the correct working directory 422 | (e.g. "osm-tutorial-master", if downloaded from GitHub), the following 423 | code will subset the data and output a new, smaller `.osm` file. 424 | 425 | ``` 426 | osmosis --read-xml data/map.osm --bounding-box top=53.822093 left=-1.528815 427 | bottom=53.817486 right=-1.521155 --write-xml potter.osm 428 | 429 | ``` 430 | 431 | To check whether or not this has worked, and demonstrate `osmar`'s ability to read in 432 | `.osm` files, let us try to load an plot the dataset just exported by osmosis. 433 | 434 | 435 | ```r 436 | src <- osmsource_osmosis(file = "data/potter.osm") 437 | bp <- center_bbox(mean(b[1, ]), mean(b[2, ]), 1000, 1000) 438 | potter <- get_osm(bp, src) 439 | plot(potter) 440 | ``` 441 | 442 | ![plot of chunk All OSM data in and directly surround Potternewton Park](figure/All_OSM_data_in_and_directly_surround_Potternewton_Park.png) 443 | 444 | 445 | Local knowledge or a quick look on an online will confirm that this is 446 | indeed Potternewton Park, with is rectangular tennis courts, play area 447 | and skate park just west of its centre. The file `potter.osm` can 448 | equally be loaded into QGIS or any other GIS. The point is to illustrate 449 | how Osmosis works. 450 | 451 | For further functionality, including clipping to polygons, extracting files from 452 | enormous and compressed `planet.osm.bz2` files and ways to extract only elements with 453 | certain attributes, please refer to the 454 | [osmosis page of the osm wiki](http://wiki.openstreetmap.org/wiki/Osmosis). 455 | The final section discusses (but does not implement) 456 | the most advanced method of harnessing OSM data, via a PostGIS database. 457 | 458 | # Creating a PostGIS database of OSM data 459 | 460 | There are a number of advantages of storing large datasets 461 | in a database: 462 | 463 | - The data can be accessed by a variety of 3rd party programs 464 | - Spatial queries can be made by a variety of clients accessing the db, widening the functionality 465 | - Huge datasets can be stored in a database, as it sits on the hard disk, only being 466 | transferred to RAM when a specific query is called 467 | 468 | Because of the size and complexity of the planet-wide OSM database, it must 469 | be stored in an spatial database to be used. For this one can 470 | use the command line tool 471 | [osm2pgsql](http://wiki.openstreetmap.org/wiki/Osm2pgsql). 472 | 473 | Once you have Postgres and PostGIS installed on your computer, 474 | and are logged in as the Postgres user the process is straightforward. 475 | The series of steps required to convert a `.osm` file into a 476 | PostGIS enabled databased are as follows. 477 | 478 | ```{bash} 479 | createdb osmTutdb # create the database called osmTutdb 480 | 481 | psql osmTutdb -c "create extension postgis" # add PostGIS functionality 482 | 483 | psql osmTutdb -c "create extension pgrouting" # add pgrouting functionality (not essential) 484 | 485 | osm2pgsql -d osmTutdb map.osm # dump the map.osm data into the new database 486 | ``` 487 | 488 | The problem with this method of doing things seems to be that osm2pgsql 489 | ignores the timestamp and user id of the elements, a serious drawback 490 | for some applications. According to a 491 | [bug report](https://trac.openstreetmap.org/ticket/4894) this is not 492 | solved by the addition of the `--extra-attributes` tag and this problem 493 | was witnessed to persist in osm2pgsql 0.83.0. 494 | 495 | To select all cycleways in an area using osmosis, this was the 496 | command used (only found to work on up-to-date versions): 497 | 498 | ```{bash} 499 | osmosis --read-xml map.osm --tf accept-ways highway=cycleway --used-node --write-xml osmo-cways.osm 500 | ``` 501 | 502 | 503 | 504 | 505 | Using a spatial database, it does not take a very large leap to realise that 506 | organisations can create a customised version of the OSM database for their own purposes. 507 | For example, an organisation interested in store location analysis could keep maintained 508 | a version of OSM with everything with elements labelled as shops removed. Alternatively, 509 | an organisation interested in woodlands could maintain an up-to-date version 510 | of OSM woodlands. Both organisations could supplement these custom databases with their 511 | own data, combining the best of crowd-sourced and centralised data collection methods. 512 | 513 | The [OSM-GB](http://wiki.openstreetmap.org/wiki/Osm2pgsql) project, hosted at the University 514 | of Nottingham does precisely this, with the aim of quality-checking community contributed data. 515 | Here is not the place to describe how to set-up a PostGIS database with OSM data, but is well 516 | worth flagging as it has great potential, especially when combined with rapidly 517 | evolving open source web mapping technologies such as 518 | [GeoServer](http://geoserver.org/display/GEOS/Welcome) 519 | (part of the 520 | [GeoNode](http://geonode.org/) stack), [Leaflet](http://leafletjs.com/) and 521 | [GeoDjango](https://www.djangoproject.com/). 522 | 523 | # Conclusion 524 | 525 | With the certainty of peak oil and possibility of 526 | effective climate change regulations, transport will become increasing 527 | expensive in future years. Thus, the tendency towards geographical 528 | homogenisation of economic activity may go into reverse 529 | ([Greer 2009](http://books.google.co.uk/books?hl=en&lr=&id=mkV_knlze0QC&oi=fnd&pg=PP2&dq=ecotechnic+future&ots=nATRuCVL31&sig=bwafIZ7kfmZMK1EscQcKyIGeYsU&redir_esc=y#v=onepage&q=ecotechnic%20future&f=false); 530 | [Curtis 2009](http://www.sciencedirect.com/science/article/pii/S0921800909003334)). 531 | This is bad news for many, but it is good news for people with 532 | a strong interest in regional diversity, local economies and geographic diversity. 533 | 534 | It is also potentially good news for geographers advocating for location-specific 535 | solutions. With increased concern over the 536 | highly centralised power structures of the internet following the 537 | revelations leaked by Edward Snowdon about massive online spying 538 | and infringement of digital privacy, there 539 | is a huge potential for community-based, problem-specific solutions. 540 | It is with this wider context in mind that this tutorial ends - 541 | think of the potential benefits if citizens were encouraged to be 542 | both producers and consumers of the maps on which we all now depend. 543 | Happy mapping! 544 | 545 | # References 546 | 547 | Curtis, F. (2009). Peak globalization: Climate change, oil depletion and global trade. Ecological Economics, 69(2), 427-434. 548 | 549 | Eugster, M. J., & Schlesinger, T. (2013). osmar: OpenStreetMap and R. The R Journal, 5(1), 53-63. 550 | 551 | Goodchild, M. F. (2007). Citizens as sensors: the world of volunteered 552 | geography. GeoJournal, 69(4), 211–221. 553 | 554 | Greer, J. M. (2009). The Ecotechnic Future: Envisioning a post-peak world. New Society Publishers. 555 | 556 | Haklay, M., & Weber, P. (2008). Openstreetmap: User-generated street maps. Pervasive Computing, IEEE, 7(4), 12-18. 557 | 558 | Lovelace, R. (2013). Open Source Data and Methods: A tutorial using crowd-sourced data for store location analysis. EloGeo repository. 559 | 560 | Neis, P., Zielstra, D., & Zipf, A. (2011). The street network evolution of crowdsourced maps: OpenStreetMap in Germany 2007–2011. Future Internet, 4(1), 1-21. 561 | 562 | Neis, P., Zielstra, D., & Zipf, A. (2013). Comparison of Volunteered Geographic Information Data Contributions and Community Development for Selected World Regions. Future Internet, 5(2), 282-300. 563 | 564 | Salisbury, C. & Jenkins, J. (2014). Mapping the Void. Broadcast on BBC Radio 4, 11:00AM Mon, 27 Jan 2014. Available on I-Player until 11:32AM Mon, 3 Feb 2014. Available on my 565 | [Dropbox account](https://dl.dropboxusercontent.com/u/15008199/egs2stay/Mapping_the_Void_-_Mapping_the_Void_b03s6mf0_default.m4a) for forseable future. 566 | 567 | Wroclawski, S. (2014). Why the world needs OpenStreetMap. The Guardian. Tuesday 14 January 2014 11.52 GMT. 568 | 569 | 570 | ```r 571 | source("md2pdf.R") # convert markdown document to LaTeX 572 | ``` 573 | 574 | 575 | 576 | 577 | 578 | -------------------------------------------------------------------------------- /osm.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Robinlovelace/osm-tutorial/8303ced4d8188c4163e8827d0fb40dfdcf53e7bc/osm.pdf -------------------------------------------------------------------------------- /osm.synctex.gz: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Robinlovelace/osm-tutorial/8303ced4d8188c4163e8827d0fb40dfdcf53e7bc/osm.synctex.gz -------------------------------------------------------------------------------- /osm.tex: -------------------------------------------------------------------------------- 1 | \documentclass[]{article} 2 | \usepackage[T1]{fontenc} 3 | \usepackage{lmodern} 4 | \usepackage{amssymb,amsmath} 5 | \usepackage{ifxetex,ifluatex} 6 | \usepackage{fixltx2e} % provides \textsubscript 7 | % use upquote if available, for straight quotes in verbatim environments 8 | \IfFileExists{upquote.sty}{\usepackage{upquote}}{} 9 | \ifnum 0\ifxetex 1\fi\ifluatex 1\fi=0 % if pdftex 10 | \usepackage[utf8]{inputenc} 11 | \else % if luatex or xelatex 12 | \ifxetex 13 | \usepackage{mathspec} 14 | \usepackage{xltxtra,xunicode} 15 | \else 16 | \usepackage{fontspec} 17 | \fi 18 | \defaultfontfeatures{Mapping=tex-text,Scale=MatchLowercase} 19 | \newcommand{\euro}{€} 20 | \fi 21 | % use microtype if available 22 | \IfFileExists{microtype.sty}{\usepackage{microtype}}{} 23 | \usepackage{color} 24 | \usepackage{fancyvrb} 25 | \newcommand{\VerbBar}{|} 26 | \newcommand{\VERB}{\Verb[commandchars=\\\{\}]} 27 | \DefineVerbatimEnvironment{Highlighting}{Verbatim}{commandchars=\\\{\}} 28 | % Add ',fontsize=\small' for more characters per line 29 | \newenvironment{Shaded}{}{} 30 | \newcommand{\KeywordTok}[1]{\textcolor[rgb]{0.00,0.44,0.13}{\textbf{{#1}}}} 31 | \newcommand{\DataTypeTok}[1]{\textcolor[rgb]{0.56,0.13,0.00}{{#1}}} 32 | \newcommand{\DecValTok}[1]{\textcolor[rgb]{0.25,0.63,0.44}{{#1}}} 33 | \newcommand{\BaseNTok}[1]{\textcolor[rgb]{0.25,0.63,0.44}{{#1}}} 34 | \newcommand{\FloatTok}[1]{\textcolor[rgb]{0.25,0.63,0.44}{{#1}}} 35 | \newcommand{\CharTok}[1]{\textcolor[rgb]{0.25,0.44,0.63}{{#1}}} 36 | \newcommand{\StringTok}[1]{\textcolor[rgb]{0.25,0.44,0.63}{{#1}}} 37 | \newcommand{\CommentTok}[1]{\textcolor[rgb]{0.38,0.63,0.69}{\textit{{#1}}}} 38 | \newcommand{\OtherTok}[1]{\textcolor[rgb]{0.00,0.44,0.13}{{#1}}} 39 | \newcommand{\AlertTok}[1]{\textcolor[rgb]{1.00,0.00,0.00}{\textbf{{#1}}}} 40 | \newcommand{\FunctionTok}[1]{\textcolor[rgb]{0.02,0.16,0.49}{{#1}}} 41 | \newcommand{\RegionMarkerTok}[1]{{#1}} 42 | \newcommand{\ErrorTok}[1]{\textcolor[rgb]{1.00,0.00,0.00}{\textbf{{#1}}}} 43 | \newcommand{\NormalTok}[1]{{#1}} 44 | \usepackage{graphicx} 45 | % Redefine \includegraphics so that, unless explicit options are 46 | % given, the image width will not exceed the width of the page. 47 | % Images get their normal width if they fit onto the page, but 48 | % are scaled down if they would overflow the margins. 49 | \makeatletter 50 | \def\ScaleIfNeeded{% 51 | \ifdim\Gin@nat@width>\linewidth 52 | \linewidth 53 | \else 54 | \Gin@nat@width 55 | \fi 56 | } 57 | \makeatother 58 | \let\Oldincludegraphics\includegraphics 59 | {% 60 | \catcode`\@=11\relax% 61 | \gdef\includegraphics{\@ifnextchar[{\Oldincludegraphics}{\Oldincludegraphics[width=\ScaleIfNeeded]}}% 62 | \usepackage[margin=1.8cm]{geometry} 63 | \author{ 64 | Lovelace, Robin\\ 65 | \texttt{r.lovelace@leeds.ac.uk} 66 | } 67 | \title{Harnessing Open Street Map Data with R and QGIS} 68 | }% 69 | \ifxetex 70 | \maketitle 71 | \usepackage[setpagesize=false, % page size defined by xetex 72 | unicode=false, % unicode breaks when used with xetex 73 | xetex]{hyperref} 74 | \else 75 | \usepackage[unicode=true]{hyperref} 76 | \fi 77 | \hypersetup{breaklinks=true, 78 | bookmarks=true, 79 | pdfauthor={}, 80 | pdftitle={}, 81 | colorlinks=true, 82 | citecolor=blue, 83 | urlcolor=blue, 84 | linkcolor=magenta, 85 | pdfborder={0 0 0}} 86 | \urlstyle{same} % don't use monospace font for urls 87 | \setlength{\parindent}{0pt} 88 | \setlength{\parskip}{6pt plus 2pt minus 1pt} 89 | \setlength{\emergencystretch}{3em} % prevent overfull lines 90 | \setcounter{secnumdepth}{0} 91 | 92 | \author{} 93 | \date{} 94 | 95 | \begin{document} 96 | 97 | 98 | with R and 99 | QGIS}\label{open-street-map-loading-analysing-and-visualising-free-maps-with-r-and-qgis} 100 | 101 | This tutorial shows how open source tools can be used to harness a huge 102 | and rapidly growing open source geodatabase: Open Street Map. It is 103 | targeted at people new to Open Street Map, but who already have an 104 | understanding of basic GIS concepts. Previous experience with the 105 | programs QGIS and R would be beneficial, but not essential, for 106 | completing the exercises. There are also a number of resources available 107 | on-line for more advanced functions, as described below. 108 | 109 | \section{Introduction}\label{introduction} 110 | 111 | Open Street Map (OSM) is a crowd-sourced map of the world, the archetype 112 | of `volunteered geographical information' 113 | (\href{http://www.ncgia.ucsb.edu/projects/vgi/docs/position/Goodchild_VGI2007.pdf}{Goodchild 114 | 2007}). The aim is simple: ``to create a set of map data that's free to 115 | use, editable, and licensed under new copyright schemes'', as described 116 | in an excellent review article on the subject 117 | (\href{http://discovery.ucl.ac.uk/13849/1/13849.pdf}{Haklay and Weber 118 | 2008}). Putting the public in charge of editing the world's surface may 119 | seem like a risky business, given that cartographers have specialist 120 | skills developed over centuries. This issue is described in 121 | \href{https://dl.dropboxusercontent.com/u/15008199/egs2stay/Mapping_the_Void_-_Mapping_the_Void_b03s6mf0_default.m4a}{Mapping 122 | the Void}, an excellent BBC Radio 4 Documentary on the subject 123 | (Salisbury and Jenkins 2014). Yet the emergence of high resolution 124 | aerial photography covering the entirety of the Earth's surface and the 125 | explosion in GPS ownership via smartphones has enabled citizens to 126 | become accurate sensors of the world. 127 | \href{http://www.mdpi.com/1999-5903/4/1/1/pdf}{Neis et al. (2012)} 128 | believe this phenomenon is more than merely technological. In OSM, they 129 | see a ``revolutionary paradigm shift on how map data is being 130 | collected''. Citizen mappers have the added advantage that they may know 131 | their local area far better than any professional cartographer. 132 | 133 | This tutorial adds a small nugget of information to the growing 134 | literature on OSM, by demonstrating how the data can be accessed for 135 | teaching or research materials. It's a completely free and open dataset, 136 | so we may as well use it! There is already some good on-line material 137 | about OSM data, including: 138 | 139 | \begin{itemize} 140 | \itemsep1pt\parskip0pt\parsep0pt 141 | \item 142 | a \href{http://www.mdpi.com/1999-5903/5/2/282/pdf}{paper} comparing 143 | the quality and coverage of OSM map data in different parts of the 144 | world (Neis et al. 2013) 145 | \item 146 | an 147 | \href{http://www.library.carleton.ca/sites/default/files/help/gis/WorkingWithOpenStreetMap.pdf}{overview} 148 | of handling OSM data in ArcMap 149 | \item 150 | a 151 | \href{http://elogeo.nottingham.ac.uk/xmlui/bitstream/handle/url/289/osm-tutorial-final-2.pdf?sequence=1}{tutorial} 152 | illustrating its potential use for GIS education and store location 153 | planning (Lovelace 2013). 154 | \end{itemize} 155 | 156 | Yet nothing has focused on simply loading the data using different 157 | tools. In this paper we use QGIS and R because these are (arguably) the 158 | most popular open source programs used by geographers, the discipline 159 | with most to gain from OSM data. Before delving into the method (which 160 | is actually quite straightforward), let's put OSM data in context. 161 | 162 | \subsection{Why (and why not) use OSM 163 | data?}\label{why-and-why-not-use-osm-data} 164 | 165 | Of course there are teething issues with any large-scale open source 166 | database, including variable data quality, patchy and incomplete 167 | coverage and inconsistencies from place to place (Haklay 2010). Yet all 168 | of these issues are gradually being ironed out. The advantages of Open 169 | Street Map outweigh these downsides for many applications 170 | \emph{already}. These include: 171 | 172 | \begin{itemize} 173 | \itemsep1pt\parskip0pt\parsep0pt 174 | \item 175 | Rapid updates of new projects 176 | \item 177 | Greater range of attributes (e.g.~shop names) 178 | \item 179 | Ability to share data with anyone without breaching license 180 | \end{itemize} 181 | 182 | In additions there are a number of ethical benefits of using OSM: it's 183 | community a map for the greater good 184 | (\href{http://www.theguardian.com/technology/2014/jan/14/why-the-world-needs-openstreetmap}{Wroclawski 185 | 2014}). 186 | 187 | There is a strong community of OSM volunteers who use the service for 188 | humanitarian purposes (Salisbury and Jenkins 2014). Two excellent 189 | examples of this help underline the ethical side of OSM data. 190 | \href{http://explore.ramanitanzania.org/}{Tindale}, a settlement in 191 | Tanzania that has been mapped rapidly thanks to a single academic 192 | project, in collaboration with the authorities, enabling better policy 193 | making in the area (see 194 | \href{http://www.youtube.com/watch?v=KqrGyvNnWkA}{video}). Second, the 195 | rapid response of the OSM community to the Typhoon Haiyan disaster. In a 196 | matter of days, an army of volunteers had helped to map out the affected 197 | zone, aiding relief efforts (e.g.~see 198 | \href{https://www.mapbox.com/blog/typhoon-haiyan-openstreetmap/}{MapBox's 199 | blog posts on the subject}). 200 | 201 | \subsection{An overview of the 202 | tutorial}\label{an-overview-of-the-tutorial} 203 | 204 | All of the code and data used to create this tutorial is available on 205 | \href{http://github.com}{GitHub}. Feel free to download the project as a 206 | \texttt{.zip} file from the 207 | \href{https://github.com/Robinlovelace/osm-tutorial}{project's 208 | repository} entitled osm-tutorial from github.com/Robinlovelace. This 209 | zip file contains all of the code and data used to generate the 210 | tutorial, which is entirely reproducible. If you would like to modify or 211 | improve it in any way, please fork a version for you own use, citing the 212 | original where appropriate. 213 | 214 | The focus is mainly on the technical challenge of extracting the data 215 | from servers `in the cloud' and onto your desktop. We also cover some 216 | basic tasks in handling, subsetting and visualising the data, first in 217 | the graphical user interface (GUI) of QGIS, and then in the command line 218 | with R. The next stage talks about editing raw OSM data, essential if 219 | you would like to take subsets of very large OSM datasets, such as 220 | \texttt{planet.osm} which takes up more than 30 GB of memory. The next 221 | stage is to talk about OSM data in PostGIS, a geodatabase program ideal 222 | for querying very large spatial datasets. Finally, there is a brief 223 | section on further resources. None of the sections require any of the 224 | previous ones although the level of difficulty generally increases as 225 | you progress. 226 | 227 | \section{Getting the data}\label{getting-the-data} 228 | 229 | OSM data of a specific area can be downloaded directly from the 230 | \href{http://www.openstreetmap.org}{main map page}, from the 231 | \href{http://overpass-api.de/}{Overpass API} or, for the entire planet, 232 | from the huge (currently 32 GB) 233 | \href{http://planet.openstreetmap.org/}{planet.osm file}. A number of 234 | third parties also provide more manageable chunks of this dataset, such 235 | as the single country datasets provided by 236 | \href{http://download.geofabrik.de/}{GEOFABIK}. Command line programs 237 | \href{http://wiki.openstreetmap.org/wiki/Osmosis}{Osmosis} and 238 | \href{http://wiki.openstreetmap.org/wiki/Osm2pgsql}{Osm2pgsl} can be 239 | used to process raw OSM data in either \texttt{.osm} or 240 | \texttt{.osm.pbf} file formats. The former is essentially a 241 | \texttt{.xml} (Extensible Markup Language) text file (encoded with the 242 | popular UTF-8 characterset); the latter is a compressed version of the 243 | former. How we transfer these datasets into a useful form depends on the 244 | program you are using. In this tutorial we will explain how to do it in 245 | QGIS and R, as well describing the basics of getting it into a 246 | \href{http://postgis.net/}{PostGIS} database. 247 | 248 | \section{OSM data in QGIS}\label{osm-data-in-qgis} 249 | 250 | A \texttt{.osm} file can be downloaded from the openstreetmap.org with 251 | the bounding box selected by default depending on the current view, or 252 | via a manual selection, as shown below. 253 | 254 | \begin{figure}[htbp] 255 | \centering 256 | \includegraphics{figure/Manual_selection_of_bounding_box.png} 257 | \caption{Manual selection of bounding box} 258 | \end{figure} 259 | 260 | To load this file into QGIS, you can simply use the 261 | \texttt{Add Vector Layer} button on the left of the screen. However this 262 | does not generate satisfactory results. The \emph{recommended} way to 263 | import the data is via the the OSM plugin. When this is installed in 264 | QGIS 2.0, use the menus \texttt{Vector \textgreater{} OpenStreetMap} to 265 | import the xml file and convert it into a SpatiaLite database. Then you 266 | can import it into the QGIS workspace. 267 | 268 | \begin{figure}[htbp] 269 | \centering 270 | \includegraphics{osmfigs/import-osm.png} 271 | \caption{Import the osm data to SpatiaLite} 272 | \end{figure} 273 | 274 | After this step the file has been saved as a \texttt{.osm.db} file. Use 275 | the \texttt{Export Topology to SpatiaLite} element of the same menu to 276 | load the file. Choose the type of spatial data you would like to load - 277 | Points, Lines or Polygons. At this stage one can also select which 278 | variables (``Tags'') you would like to add to the attribute data. 279 | 280 | \begin{figure}[htbp] 281 | \centering 282 | \includegraphics{osmfigs/open-osmdb.png} 283 | \caption{Select Polylines} 284 | \end{figure} 285 | 286 | The data is now loaded into QGIS allowing standard methods of analysis. 287 | You will notice that the data are not styled at all, leading to very 288 | bland maps. To counter this, there have been custom styles developed for 289 | visualising OSM data in QGIS, e.g. 290 | \href{http://anitagraser.com/2012/02/25/light-styles-for-osm-layers-in-qgis/}{those 291 | by Anita Grazer}. Unfortunately these files do not seem to be working 292 | with the current version of QGIS so alternative ready-made styles must 293 | be created, as suggested by a 294 | \href{http://gis.stackexchange.com/questions/42645/is-there-up-to-date-osm-sld-file-for-geoserver}{stackexchange 295 | question}. 296 | 297 | Once the data is loaded into QGIS, it can be used as with any other 298 | spatial data. Next, let's see how R can handle OSM data, via the 299 | \texttt{osmar} package. 300 | 301 | \section{Using osmar}\label{using-osmar} 302 | 303 | \texttt{osmar} is an R package for downloading and interrogating OSM 304 | data that accesses the data directly from the internet via the R command 305 | line. There is an excellent 306 | \href{http://journal.r-project.org/archive/2013-1/eugster-schlesinger.pdf}{online 307 | tutorial} which provides a detailed account of the package (Eugster \& 308 | Schlesinger, 2012). Here we will focus on loading some basic data on 309 | bicycle paths in Leeds. First the package must be loaded: 310 | 311 | \begin{Shaded} 312 | \begin{Highlighting}[] 313 | \KeywordTok{library}\NormalTok{(osmar) }\CommentTok{# if the package is not already installed, use install.packages('osmar')} 314 | \end{Highlighting} 315 | \end{Shaded} 316 | 317 | \begin{verbatim} 318 | ## Loading required package: XML 319 | ## Loading required package: RCurl 320 | ## Loading required package: bitops 321 | ## Loading required package: gtools 322 | ## Loading required package: geosphere 323 | ## Loading required package: sp 324 | ## 325 | ## Attaching package: 'osmar' 326 | ## 327 | ## The following object is masked from 'package:utils': 328 | ## 329 | ## find 330 | \end{verbatim} 331 | 332 | To download data directly, one first sets the source and a bounding box, 333 | and then use the \texttt{get\_osm} function to download it. Selecting 334 | Chapeltown as the centrepoint of the map, we can download all the data 335 | in the square km surrounding it. 336 | 337 | \begin{Shaded} 338 | \begin{Highlighting}[] 339 | \NormalTok{src <-}\StringTok{ }\KeywordTok{osmsource_api}\NormalTok{()} 340 | \NormalTok{bb <-}\StringTok{ }\KeywordTok{center_bbox}\NormalTok{(-}\FloatTok{1.53492}\NormalTok{, }\FloatTok{53.81934}\NormalTok{, }\DecValTok{1000}\NormalTok{, }\DecValTok{1000}\NormalTok{)} 341 | \NormalTok{ctown <-}\StringTok{ }\KeywordTok{get_osm}\NormalTok{(bb, }\DataTypeTok{source =} \NormalTok{src)} 342 | \KeywordTok{plot}\NormalTok{(ctown)} 343 | \KeywordTok{points}\NormalTok{(-}\FloatTok{1.53492}\NormalTok{, }\FloatTok{53.81934}\NormalTok{, }\DataTypeTok{col =} \StringTok{"red"}\NormalTok{, }\DataTypeTok{lwd =} \DecValTok{5}\NormalTok{)} 344 | \end{Highlighting} 345 | \end{Shaded} 346 | 347 | \begin{figure}[htbp] 348 | \centering 349 | \includegraphics{figure/Preliminary_plot_of_Chapletown_with_osmar.png} 350 | \caption{Preliminary plot of Chapletown with osmar} 351 | \end{figure} 352 | 353 | This shows that the data has successfully been loaded and saved as an 354 | object called \texttt{ctown}. Let's try analysing this object further. 355 | In fact, \texttt{ctown} is technically a list, composed of 3 objects: 356 | nodes (points), ways (lines) and relations (polygons composed of many 357 | lines). Such OSM data is thus provided a class of its own, and each 358 | sub-object can be called separately using the \texttt{\$} symbol: 359 | 360 | \begin{Shaded} 361 | \begin{Highlighting}[] 362 | \KeywordTok{names}\NormalTok{(ctown)} 363 | \end{Highlighting} 364 | \end{Shaded} 365 | 366 | \begin{verbatim} 367 | ## [1] "nodes" "ways" "relations" 368 | \end{verbatim} 369 | 370 | \begin{Shaded} 371 | \begin{Highlighting}[] 372 | \KeywordTok{class}\NormalTok{(ctown)} 373 | \end{Highlighting} 374 | \end{Shaded} 375 | 376 | \begin{verbatim} 377 | ## [1] "osmar" "list" 378 | \end{verbatim} 379 | 380 | \begin{Shaded} 381 | \begin{Highlighting}[] 382 | \KeywordTok{summary}\NormalTok{(ctown$ways)} 383 | \end{Highlighting} 384 | \end{Shaded} 385 | 386 | \begin{verbatim} 387 | ## osmar$ways object 388 | ## 179 ways, 505 tags, 1270 refs 389 | ## 390 | ## ..$attrs data.frame: 391 | ## id, changeset, timestamp, version, visible, user, uid 392 | ## ..$tags data.frame: 393 | ## id, k, v 394 | ## ..$refs data.frame: 395 | ## id, ref 396 | ## 397 | ## Key-Value contingency table: 398 | ## Key Value Freq 399 | ## 1 highway residential 71 400 | ## 2 source:name OS_OpenData_Locator 49 401 | ## 3 highway service 41 402 | ## 4 created_by JOSM 25 403 | ## 5 source Bing 23 404 | ## 6 highway unclassified 17 405 | ## 7 highway footway 15 406 | ## 8 building yes 11 407 | ## 9 lit yes 11 408 | ## 10 source npe 10 409 | \end{verbatim} 410 | 411 | Let's use the dataset we have loaded to investigate the cycle paths in 412 | the vicinity of my house. First we need to understand the data contained 413 | in the object. Let's look at the tags and the attributes of the 414 | \texttt{ways} object: 415 | 416 | \begin{Shaded} 417 | \begin{Highlighting}[] 418 | \KeywordTok{summary}\NormalTok{(ctown$ways$tags) }\CommentTok{# summary of the tag data} 419 | \end{Highlighting} 420 | \end{Shaded} 421 | 422 | \begin{verbatim} 423 | ## id k v 424 | ## Min. :5.09e+06 highway :155 residential : 71 425 | ## 1st Qu.:7.73e+06 name :128 OS_OpenData_Locator: 49 426 | ## Median :8.46e+07 source : 56 service : 41 427 | ## Mean :8.71e+07 source:name: 54 yes : 32 428 | ## 3rd Qu.:1.50e+08 created_by : 25 JOSM : 25 429 | ## Max. :2.45e+08 building : 12 Bing : 23 430 | ## (Other) : 75 (Other) :264 431 | \end{verbatim} 432 | 433 | \begin{Shaded} 434 | \begin{Highlighting}[] 435 | \NormalTok{ctown$ways$attrs[}\DecValTok{1}\NormalTok{:}\DecValTok{8}\NormalTok{, }\DecValTok{1}\NormalTok{:}\DecValTok{6}\NormalTok{] }\CommentTok{# attributes of first 8 ways - see I'm in there!} 436 | \end{Highlighting} 437 | \end{Shaded} 438 | 439 | \begin{verbatim} 440 | ## id changeset timestamp version visible user 441 | ## 1 5088536 15128484 2013-02-22 22:08:24 13 true CompactDstrxion 442 | ## 2 22818969 13039300 2012-09-08 23:06:53 20 true LeedsTracker 443 | ## 3 6273628 483846 2007-09-21 17:25:37 1 true SteveC 444 | ## 4 6273619 14114856 2012-12-01 17:57:45 2 true sc71 445 | ## 5 6273721 444107 2007-09-16 17:23:14 1 true noii 446 | ## 6 6273722 444107 2007-09-16 17:23:16 1 true noii 447 | ## 7 6273726 444107 2007-09-16 17:23:20 1 true noii 448 | ## 8 6273736 18672988 2013-11-02 10:56:24 5 true RobinLovelace 449 | \end{verbatim} 450 | 451 | From looking at the \href{http://wiki.openstreetmap.org/wiki/Tags}{OSM 452 | tagging system}, we can deduce that \texttt{id} is the element's id, 453 | \texttt{k} refers to the OSM key (the variables for which the element 454 | has values) and that \texttt{v} is the value assigned for each id - key 455 | combination. Because OSM data is not a simple data frame, we cannot use 456 | the usual R notation for subsetting. Instead we use the \texttt{find} 457 | function. Let us take a subset of bicycle paths in the area and plot 458 | them. 459 | 460 | \begin{Shaded} 461 | \begin{Highlighting}[] 462 | \NormalTok{bikePaths <-}\StringTok{ }\KeywordTok{find}\NormalTok{(ctown, }\KeywordTok{way}\NormalTok{(}\KeywordTok{tags}\NormalTok{(k ==}\StringTok{ "bicycle"} \NormalTok{&}\StringTok{ }\NormalTok{v ==}\StringTok{ "yes"}\NormalTok{)))} 463 | \NormalTok{bikePaths <-}\StringTok{ }\KeywordTok{find_down}\NormalTok{(ctown, }\KeywordTok{way}\NormalTok{(bikePaths))} 464 | \NormalTok{bikePaths <-}\StringTok{ }\KeywordTok{subset}\NormalTok{(ctown, }\DataTypeTok{ids =} \NormalTok{bikePaths)} 465 | \KeywordTok{plot}\NormalTok{(ctown)} 466 | \KeywordTok{plot_ways}\NormalTok{(bikePaths, }\DataTypeTok{add =} \NormalTok{T, }\DataTypeTok{col =} \StringTok{"red"}\NormalTok{, }\DataTypeTok{lwd =} \DecValTok{3}\NormalTok{)} 467 | \end{Highlighting} 468 | \end{Shaded} 469 | 470 | \begin{figure}[htbp] 471 | \centering 472 | \includegraphics{figure/Bike_paths_of_Chapeltown.png} 473 | \caption{Bike paths of Chapeltown} 474 | \end{figure} 475 | 476 | The above code block is used to identify all ways in which cycling is 477 | permitted, ``overriding default access'', according OSM's excellent 478 | \href{http://wiki.openstreetmap.org/wiki/Bicycle}{wiki page on bicycle 479 | paths}. 480 | 481 | According to this source, the correct way to refer to an on-road cycle 482 | path is with the \texttt{cycleway} tag. However, none of these have been 483 | added to the roads that have on-road cycle lanes in this example dataset 484 | (as of January 2014). Perhaps someone will add these soon. 485 | 486 | \begin{Shaded} 487 | \begin{Highlighting}[] 488 | \KeywordTok{which}\NormalTok{(ctown$ways$tags$k ==}\StringTok{ "cycleway"}\NormalTok{)} 489 | \end{Highlighting} 490 | \end{Shaded} 491 | 492 | \begin{verbatim} 493 | ## integer(0) 494 | \end{verbatim} 495 | 496 | There are, by contrast, a large number of ways classified as 497 | ``residential''. Let us use the same method to select them and add them 498 | to the map. 499 | 500 | \begin{Shaded} 501 | \begin{Highlighting}[] 502 | \NormalTok{res <-}\StringTok{ }\KeywordTok{find}\NormalTok{(ctown, }\KeywordTok{way}\NormalTok{(}\KeywordTok{tags}\NormalTok{(k ==}\StringTok{ "highway"} \NormalTok{&}\StringTok{ }\NormalTok{v ==}\StringTok{ "residential"}\NormalTok{)))} 503 | \NormalTok{res <-}\StringTok{ }\KeywordTok{find_down}\NormalTok{(ctown, }\KeywordTok{way}\NormalTok{(res))} 504 | \NormalTok{res <-}\StringTok{ }\KeywordTok{subset}\NormalTok{(ctown, }\DataTypeTok{ids =} \NormalTok{res)} 505 | \KeywordTok{plot}\NormalTok{(ctown)} 506 | \KeywordTok{plot_ways}\NormalTok{(res, }\DataTypeTok{add =} \NormalTok{T, }\DataTypeTok{col =} \StringTok{"green"}\NormalTok{, }\DataTypeTok{lwd =} \DecValTok{3}\NormalTok{)} 507 | \end{Highlighting} 508 | \end{Shaded} 509 | 510 | \begin{figure}[htbp] 511 | \centering 512 | \includegraphics{figure/Residential_streets_in_Chapeltown.png} 513 | \caption{Residential streets in Chapeltown} 514 | \end{figure} 515 | 516 | \section{Handling raw OSM data}\label{handling-raw-osm-data} 517 | 518 | Although this section will be of most use for dealing with very large 519 | files, a small example is used here to showcase the methods The file 520 | \texttt{map.osm} downloaded directly from 521 | \href{http://www.openstreetmap.org}{openstreetmap.org} is small, 522 | allowing available on-line, from 523 | \href{https://github.com/Robinlovelace/osm-tutorial/blob/master/data/map.osm?raw=true}{here} 524 | (highlighting the advantages of open data for educational purposes - 525 | there are few license restrictions). 526 | 527 | The Java command line tool 528 | \href{http://wiki.openstreetmap.org/wiki/Osmosis}{Osmosis} is used for 529 | this purpose. 530 | 531 | To show how Osmosis can subset raw OSM files, let us take a real world 532 | example from the dataset we imported with the \texttt{osmar} package. 533 | Imagine that we want to extract all osm data (nodes, ways and relations) 534 | from the area in and around Potternewton Park. We could do this manually 535 | by looking up the park on-line and selecting the associated elements in 536 | QGIS. But imagine we want to do it from the command line (e.g.~for batch 537 | processing). First we need a bounding box of all items containing the 538 | text string ``Potternewton Park''. For this we use \texttt{osmar}: 539 | 540 | \begin{Shaded} 541 | \begin{Highlighting}[] 542 | \NormalTok{potter <-}\StringTok{ }\KeywordTok{find}\NormalTok{(ctown, }\KeywordTok{way}\NormalTok{(}\KeywordTok{tags}\NormalTok{(}\KeywordTok{grepl}\NormalTok{(}\StringTok{"Potternewton Park"}\NormalTok{, ctown$ways$tags$v))))} 543 | \NormalTok{potter <-}\StringTok{ }\KeywordTok{find_down}\NormalTok{(ctown, }\KeywordTok{way}\NormalTok{(potter))} 544 | \NormalTok{potter <-}\StringTok{ }\KeywordTok{subset}\NormalTok{(ctown, }\DataTypeTok{ids =} \NormalTok{potter)} 545 | \KeywordTok{plot_ways}\NormalTok{(potter) }\CommentTok{# sanity check} 546 | \end{Highlighting} 547 | \end{Shaded} 548 | 549 | \begin{figure}[htbp] 550 | \centering 551 | \includegraphics{figure/Line_data_of_Potternewton_park.png} 552 | \caption{Line data of Potternewton park} 553 | \end{figure} 554 | 555 | \begin{Shaded} 556 | \begin{Highlighting}[] 557 | \NormalTok{potter.l <-}\StringTok{ }\KeywordTok{as_sp}\NormalTok{(potter, }\StringTok{"lines"}\NormalTok{) }\CommentTok{# convert data to sp class} 558 | \NormalTok{b <-}\StringTok{ }\KeywordTok{bbox}\NormalTok{(potter.l) }\CommentTok{# save the bounding box} 559 | \NormalTok{b[}\DecValTok{1}\NormalTok{, ] <-}\StringTok{ }\NormalTok{(b[}\DecValTok{1}\NormalTok{, ] -}\StringTok{ }\KeywordTok{mean}\NormalTok{(b[}\DecValTok{1}\NormalTok{, ])) *}\StringTok{ }\FloatTok{1.05} \NormalTok{+}\StringTok{ }\KeywordTok{mean}\NormalTok{(b[}\DecValTok{1}\NormalTok{, ])} 560 | \NormalTok{b[}\DecValTok{2}\NormalTok{, ] <-}\StringTok{ }\NormalTok{(b[}\DecValTok{2}\NormalTok{, ] -}\StringTok{ }\KeywordTok{mean}\NormalTok{(b[}\DecValTok{2}\NormalTok{, ])) *}\StringTok{ }\FloatTok{1.05} \NormalTok{+}\StringTok{ }\KeywordTok{mean}\NormalTok{(b[}\DecValTok{2}\NormalTok{, ])} 561 | \CommentTok{# scale longitude and latitude (increase bb by 5% for plot) replace 1.05} 562 | \CommentTok{# with 1.xx for an xx% increase in the plot size} 563 | \NormalTok{b} 564 | \end{Highlighting} 565 | \end{Shaded} 566 | 567 | \begin{verbatim} 568 | ## min max 569 | ## x -1.529 -1.521 570 | ## y 53.817 53.822 571 | \end{verbatim} 572 | 573 | Now that we know the bounding box, we can transfer to using osmosis from 574 | the command line. If the shell is open in the correct working directory 575 | (e.g. ``osm-tutorial-master'', if downloaded from GitHub), the following 576 | code will subset the data and output a new, smaller \texttt{.osm} file. 577 | 578 | \begin{verbatim} 579 | osmosis --read-xml data/map.osm --bounding-box top=53.822093 left=-1.528815 580 | bottom=53.817486 right=-1.521155 --write-xml potter.osm 581 | \end{verbatim} 582 | 583 | To check whether or not this has worked, and demonstrate 584 | \texttt{osmar}'s ability to read in \texttt{.osm} files, let us try to 585 | load an plot the dataset just exported by osmosis. 586 | 587 | \begin{Shaded} 588 | \begin{Highlighting}[] 589 | \NormalTok{src <-}\StringTok{ }\KeywordTok{osmsource_osmosis}\NormalTok{(}\DataTypeTok{file =} \StringTok{"data/potter.osm"}\NormalTok{)} 590 | \NormalTok{bp <-}\StringTok{ }\KeywordTok{center_bbox}\NormalTok{(}\KeywordTok{mean}\NormalTok{(b[}\DecValTok{1}\NormalTok{, ]), }\KeywordTok{mean}\NormalTok{(b[}\DecValTok{2}\NormalTok{, ]), }\DecValTok{1000}\NormalTok{, }\DecValTok{1000}\NormalTok{)} 591 | \NormalTok{potter <-}\StringTok{ }\KeywordTok{get_osm}\NormalTok{(bp, src)} 592 | \KeywordTok{plot}\NormalTok{(potter)} 593 | \end{Highlighting} 594 | \end{Shaded} 595 | 596 | \begin{figure}[htbp] 597 | \centering 598 | \includegraphics{figure/All_OSM_data_in_and_directly_surround_Potternewton_Park.png} 599 | \caption{All OSM data in and directly surround 600 | Potternewton Park} 601 | \end{figure} 602 | 603 | Local knowledge or a quick look on an online will confirm that this is 604 | indeed Potternewton Park, with is rectangular tennis courts, play area 605 | and skate park just west of its centre. The file \texttt{potter.osm} can 606 | equally be loaded into QGIS or any other GIS. The point is to illustrate 607 | how Osmosis works. 608 | 609 | For further functionality, including clipping to polygons, extracting 610 | files from enormous and compressed \texttt{planet.osm.bz2} files and 611 | ways to extract only elements with certain attributes, please refer to 612 | the \href{http://wiki.openstreetmap.org/wiki/Osmosis}{osmosis page of 613 | the osm wiki}. The final section discusses (but does not implement) the 614 | most advanced method of harnessing OSM data, via a PostGIS database. 615 | 616 | \section{Creating a PostGIS database of OSM 617 | data}\label{creating-a-postgis-database-of-osm-data} 618 | 619 | There are a number of advantages of storing large datasets in a 620 | database: 621 | 622 | \begin{itemize} 623 | \itemsep1pt\parskip0pt\parsep0pt 624 | \item 625 | The data can be accessed by a variety of 3rd party programs 626 | \item 627 | Spatial queries can be made by a variety of clients accessing the db, 628 | widening the functionality 629 | \item 630 | Huge datasets can be stored in a database, as it sits on the hard 631 | disk, only being transferred to RAM when a specific query is called 632 | \end{itemize} 633 | 634 | Because of the size and complexity of the planet-wide OSM database, it 635 | must be stored in an spatial database to be used. For this use the 636 | command line tool 637 | \href{http://wiki.openstreetmap.org/wiki/Osm2pgsql}{osm2pgsql}. The 638 | basics of installation and usage can be found online. 639 | 640 | Using a spatial database, it does not take a very large leap to realise 641 | that organisations can create a customised version of the OSM database 642 | for their own purposes. For example, an organisation interested in store 643 | location analysis could keep maintained a version of OSM with everything 644 | with elements labelled as shops removed. Alternatively, an organisation 645 | interested in woodlands could maintain an up-to-date version of OSM 646 | woodlands. Both organisations could supplement these custom databases 647 | with their own data, combining the best of crowd-sourced and centralised 648 | data collection methods. 649 | 650 | The \href{http://wiki.openstreetmap.org/wiki/Osm2pgsql}{OSM-GB} project, 651 | hosted at the University of Nottingham does precisely this, with the aim 652 | of quality-checking community contributed data. Here is not the place to 653 | describe how to set-up a PostGIS database with OSM data, but is well 654 | worth flagging as it has great potential, especially when combined with 655 | rapidly evolving open source web mapping technologies such as 656 | \href{http://geoserver.org/display/GEOS/Welcome}{GeoServer} (part of the 657 | \href{http://geonode.org/}{GeoNode} stack), 658 | \href{http://leafletjs.com/}{Leaflet} and 659 | \href{https://www.djangoproject.com/}{GeoDjango}. 660 | 661 | \section{Conclusion}\label{conclusion} 662 | 663 | With the certainty of peak oil and possibility of effective climate 664 | change regulations, transport will become increasing expensive in future 665 | years. Thus, the tendency towards geographical homogenisation of 666 | economic activity may go into reverse 667 | (\href{http://books.google.co.uk/books?hl=en\&lr=\&id=mkV_knlze0QC\&oi=fnd\&pg=PP2\&dq=ecotechnic+future\&ots=nATRuCVL31\&sig=bwafIZ7kfmZMK1EscQcKyIGeYsU\&redir_esc=y\#v=onepage\&q=ecotechnic\%20future\&f=false}{Greer 668 | 2009}; 669 | \href{http://www.sciencedirect.com/science/article/pii/S0921800909003334}{Curtis 670 | 2009}). This is bad news for many, but it is good news for people with a 671 | strong interest in regional diversity, local economies and geographic 672 | diversity. 673 | 674 | It is also potentially good news for geographers advocating for 675 | location-specific solutions. With increased concern over the highly 676 | centralised power structures of the internet following the revelations 677 | leaked by Edward Snowdon about massive online spying and infringement of 678 | digital privacy, there is a huge potential for community-based, 679 | problem-specific solutions. It is with this wider context in mind that 680 | this tutorial ends - think of the potential benefits if citizens were 681 | encouraged to be both producers and consumers of the maps on which we 682 | all now depend. Happy mapping! 683 | 684 | \section{References}\label{references} 685 | 686 | Curtis, F. (2009). Peak globalization: Climate change, oil depletion and 687 | global trade. Ecological Economics, 69(2), 427-434. 688 | 689 | Eugster, M. J., \& Schlesinger, T. (2013). osmar: OpenStreetMap and R. 690 | The R Journal, 5(1), 53-63. 691 | 692 | Goodchild, M. F. (2007). Citizens as sensors: the world of volunteered 693 | geography. GeoJournal, 69(4), 211--221. 694 | 695 | Greer, J. M. (2009). The Ecotechnic Future: Envisioning a post-peak 696 | world. New Society Publishers. 697 | 698 | Haklay, M., \& Weber, P. (2008). Openstreetmap: User-generated street 699 | maps. Pervasive Computing, IEEE, 7(4), 12-18. 700 | 701 | Lovelace, R. (2013). Open Source Data and Methods: A tutorial using 702 | crowd-sourced data for store location analysis. EloGeo repository. 703 | 704 | Neis, P., Zielstra, D., \& Zipf, A. (2011). The street network evolution 705 | of crowdsourced maps: OpenStreetMap in Germany 2007--2011. Future 706 | Internet, 4(1), 1-21. 707 | 708 | Neis, P., Zielstra, D., \& Zipf, A. (2013). Comparison of Volunteered 709 | Geographic Information Data Contributions and Community Development for 710 | Selected World Regions. Future Internet, 5(2), 282-300. 711 | 712 | Salisbury, C. \& Jenkins, J. (2014). Mapping the Void. Broadcast on BBC 713 | Radio 4, 11:00AM Mon, 27 Jan 2014. Available on I-Player until 11:32AM 714 | Mon, 3 Feb 2014. Available on my 715 | \href{https://dl.dropboxusercontent.com/u/15008199/egs2stay/Mapping_the_Void_-_Mapping_the_Void_b03s6mf0_default.m4a}{Dropbox 716 | account} for forseable future. 717 | 718 | Wroclawski, S. (2014). Why the world needs OpenStreetMap. The Guardian. 719 | Tuesday 14 January 2014 11.52 GMT. 720 | 721 | \begin{Shaded} 722 | \begin{Highlighting}[] 723 | \KeywordTok{source}\NormalTok{(}\StringTok{"md2pdf.R"}\NormalTok{) }\CommentTok{# convert markdown document to LaTeX} 724 | \end{Highlighting} 725 | \end{Shaded} 726 | 727 | \end{document} 728 | -------------------------------------------------------------------------------- /osmfigs/Export-leeds.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Robinlovelace/osm-tutorial/8303ced4d8188c4163e8827d0fb40dfdcf53e7bc/osmfigs/Export-leeds.png -------------------------------------------------------------------------------- /osmfigs/import-osm.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Robinlovelace/osm-tutorial/8303ced4d8188c4163e8827d0fb40dfdcf53e7bc/osmfigs/import-osm.png -------------------------------------------------------------------------------- /osmfigs/import.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Robinlovelace/osm-tutorial/8303ced4d8188c4163e8827d0fb40dfdcf53e7bc/osmfigs/import.png -------------------------------------------------------------------------------- /osmfigs/load-style.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Robinlovelace/osm-tutorial/8303ced4d8188c4163e8827d0fb40dfdcf53e7bc/osmfigs/load-style.png -------------------------------------------------------------------------------- /osmfigs/loaded-line-data.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Robinlovelace/osm-tutorial/8303ced4d8188c4163e8827d0fb40dfdcf53e7bc/osmfigs/loaded-line-data.png -------------------------------------------------------------------------------- /osmfigs/manual-selection.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Robinlovelace/osm-tutorial/8303ced4d8188c4163e8827d0fb40dfdcf53e7bc/osmfigs/manual-selection.png -------------------------------------------------------------------------------- /osmfigs/open-osmdb.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Robinlovelace/osm-tutorial/8303ced4d8188c4163e8827d0fb40dfdcf53e7bc/osmfigs/open-osmdb.png --------------------------------------------------------------------------------