├── ArcGis ├── v1.00 │ └── GISF2E_ArcGIS_v1.00.tbx ├── GISF2E_ArcGIS.tbx ├── v1.10 │ ├── GISF2E_ArcGIS_v1.10.tbx │ └── README ├── v2.00 │ ├── GISF2E_ArcGIS_v2.00.tbx │ ├── README │ └── GISF2E_ArcGIS_v2.00.py └── v2.10 │ ├── GISF2E_ArcGIS_v2.10.tbx │ ├── README │ └── GISF2E_ArcGIS_v2.10.py ├── Python ├── v1.00 │ ├── README │ └── GISF2E_python_v1.00.py ├── v1.10 │ ├── README │ └── GISF2E_python_v1.10.py └── v1.20 │ ├── README │ └── GISF2E_python_v1.20.py ├── QGIS ├── v1.00 │ ├── README │ ├── GISF2E_qgis_non-OSM_v1.00.py │ └── GISF2E_qgis_OSM_v1.00.py ├── v1.10 │ ├── README │ ├── GISF2E_qgis_non-OSM_v1.10.py │ └── GISF2E_qgis_OSM_v1.10.py └── v1.20 │ ├── README │ ├── GISF2E_qgis_non-OSM_v1.20.py │ └── GISF2E_qgis_OSM_v1.20.py └── README.md /ArcGis/v1.00/GISF2E_ArcGIS_v1.00.tbx: -------------------------------------------------------------------------------- 1 | GISF2E ArcGIS - v1.00 2 | 3 | To download the tool, please visit http://csun.uic.edu/codes/GISF2E.html 4 | 5 | To cite, plese use: Karduni, A., Kermanshah, A., & Derrible, S., 2016 “A protocol to convert spatial polyline data to network formats and applications to world urban road networks”, Scientific Data, 3:160046 6 | The article is available at: http://www.nature.com/articles/sdata201646 7 | -------------------------------------------------------------------------------- /ArcGis/GISF2E_ArcGIS.tbx: -------------------------------------------------------------------------------- 1 | """ 2 | GISF2E ArcGIS - v1.00 3 | 4 | For the ArcGIS version of the tool, please visit http://csun.uic.edu/codes.html 5 | 6 | To cite, plese use: Karduni, A., Kermanshah, A., & Derrible, S., 2016 “A protocol to convert spatial polyline data to network formats and applications to world urban road networks”, Scientific Data, 3:160046 7 | The article is available at: http://www.nature.com/articles/sdata201646 8 | 9 | """ 10 | -------------------------------------------------------------------------------- /ArcGis/v1.10/GISF2E_ArcGIS_v1.10.tbx: -------------------------------------------------------------------------------- 1 | GISF2E ArcGIS - v1.10 2 | 3 | To download the tool, please visit http://csun.uic.edu/codes/GISF2E.html 4 | 5 | To cite, plese use: Karduni, A., Kermanshah, A., & Derrible, S., 2016 “A protocol to convert spatial polyline data to network formats and applications to world urban road networks”, Scientific Data, 3:160046 6 | The article is available at: http://www.nature.com/articles/sdata201646 7 | 8 | For changes with previous versions, see README file. 9 | -------------------------------------------------------------------------------- /ArcGis/v2.00/GISF2E_ArcGIS_v2.00.tbx: -------------------------------------------------------------------------------- 1 | GISF2E ArcGIS - v2.00 2 | 3 | To download the tool as a toolbox, please visit http://csun.uic.edu/codes/GISF2E.html. 4 | 5 | To cite, please use: Karduni, A., Kermanshah, A., & Derrible, S., 2016 “A protocol to convert spatial polyline data to network formats and applications to world urban road networks”, Scientific Data, 3:160046 6 | The article is available at: http://www.nature.com/articles/sdata201646 7 | 8 | For changes with previous versions, see README file. 9 | -------------------------------------------------------------------------------- /ArcGis/v2.10/GISF2E_ArcGIS_v2.10.tbx: -------------------------------------------------------------------------------- 1 | GISF2E ArcGIS - v2.10 2 | 3 | To download the tool as a toolbox, please visit http://csun.uic.edu/codes/GISF2E.html. 4 | 5 | To cite, please use: Karduni, A., Kermanshah, A., & Derrible, S., 2016 “A protocol to convert spatial polyline data to network formats and applications to world urban road networks”, Scientific Data, 3:160046 6 | The article is available at: http://www.nature.com/articles/sdata201646 7 | 8 | For changes with previous versions, see README file. 9 | -------------------------------------------------------------------------------- /Python/v1.00/README: -------------------------------------------------------------------------------- 1 | README - GIS F2E Python – GIS Features to Edge List Tool (2D and 3D OSM version) 2 | 3 | This python code was made specially to be used in python. See tutorial for specifications at: http://csun.uic.edu/codes/GISF2E/Python/GISF2E_python_tutorial.zip 4 | 5 | To cite, please use: Karduni, A., Kermanshah, A., & Derrible, S., 2016 “A protocol to convert spatial polyline data to network formats and applications to world urban road networks”, Scientific Data, 3:160046 6 | 7 | The article is available at: http://www.nature.com/articles/sdata201646 8 | 9 | version 1.00 - tested for pandas version 0.14.1 10 | -------------------------------------------------------------------------------- /QGIS/v1.00/README: -------------------------------------------------------------------------------- 1 | README - GIS F2E QGIS – GIS Features to Edge List Tool (2D and 3D OSM version) 2 | 3 | These two python codes were made specially to work with QGIS (http://www.qgis.org). See tutorial for specifications at: http://csun.uic.edu/codes/GISF2E/QGIS/GISF2E_qgis_tutorial.zip 4 | 5 | To cite, please use: Karduni, A., Kermanshah, A., & Derrible, S., 2016 “A protocol to convert spatial polyline data to network formats and applications to world urban road networks”, Scientific Data, 3:160046 6 | 7 | The article is available at: http://www.nature.com/articles/sdata201646 8 | 9 | version 1.00 - tested for pandas version 0.14.1 10 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # FeaturestoEdgelist 2 | 3 | The ArcGIS Features to Edgelist tool allows anyone who uses ArcGIS to export a feature (e.g., polylines) as a network; whether you are working with streets, pipelines, electric wires or just about any interconnected linear features. The tool creates two new GIS layers, one with all nodes and one with all edges, and the edgelist is in .csv format. An edgelist is a list of all links/edges in the network with start node ID, end node ID, and edge ID). The edgelist can then be imported by any program, such as R or python scripts (e.g., using igraph or NetworkX), Gephi, or NodeXL (Excel Add-On) to name a few. 4 | -------------------------------------------------------------------------------- /Python/v1.10/README: -------------------------------------------------------------------------------- 1 | README - GIS F2E Python – GIS Features to Edge List Tool (2D and 3D OSM version) 2 | 3 | This python code was made specially to be used in python. See tutorial for specifications at: http://csun.uic.edu/codes/GISF2E/Python/GISF2E_python_tutorial.zip 4 | 5 | To cite, please use: Karduni, A., Kermanshah, A., & Derrible, S., 2016 “A protocol to convert spatial polyline data to network formats and applications to world urban road networks”, Scientific Data, 3:160046 6 | 7 | The article is available at: http://www.nature.com/articles/sdata201646 8 | 9 | version 1.10 - tested for pandas version 0.18.1 10 | version 1.00 - tested for pandas version 0.14.1 11 | 12 | 13 | What’s new in version 1.10 14 | 15 | - minor fix to work well with all versions of pandas. 16 | -------------------------------------------------------------------------------- /QGIS/v1.10/README: -------------------------------------------------------------------------------- 1 | README - GIS F2E QGIS – GIS Features to Edge List Tool (2D and 3D OSM version) 2 | 3 | These two python codes were made specially to work with QGIS (http://www.qgis.org). See tutorial for specifications at: http://csun.uic.edu/codes/GISF2E/QGIS/GISF2E_qgis_tutorial.zip 4 | 5 | To cite, please use: Karduni, A., Kermanshah, A., & Derrible, S., 2016 “A protocol to convert spatial polyline data to network formats and applications to world urban road networks”, Scientific Data, 3:160046 6 | 7 | The article is available at: http://www.nature.com/articles/sdata201646 8 | 9 | version 1.10 - tested for pandas version 0.18.1 10 | version 1.00 - tested for pandas version 0.14.1 11 | 12 | 13 | What’s new in version 1.10 14 | 15 | - minor fix to work well with all versions of pandas. 16 | -------------------------------------------------------------------------------- /Python/v1.20/README: -------------------------------------------------------------------------------- 1 | README - GIS F2E Python – GIS Features to Edge List Tool (2D and 3D OSM version) 2 | 3 | This python code was made specially to be used in python. See tutorial for specifications at: http://csun.uic.edu/codes/GISF2E/Python/GISF2E_python_tutorial.zip 4 | 5 | To cite, please use: Karduni, A., Kermanshah, A., & Derrible, S., 2016 “A protocol to convert spatial polyline data to network formats and applications to world urban road networks”, Scientific Data, 3:160046 6 | 7 | The article is available at: http://www.nature.com/articles/sdata201646 8 | 9 | version 1.20 - tested for pandas version 0.19.2 10 | version 1.10 - tested for pandas version 0.18.1 11 | version 1.00 - tested for pandas version 0.14.1 12 | 13 | 14 | What’s new in version 1.20 15 | 16 | - minor fix to work well with the new version of pandas 0.19.2 17 | -------------------------------------------------------------------------------- /QGIS/v1.20/README: -------------------------------------------------------------------------------- 1 | README - GIS F2E QGIS – GIS Features to Edge List Tool (2D and 3D OSM version) 2 | 3 | These two python codes were made specially to work with QGIS (http://www.qgis.org). See tutorial for specifications at: http://csun.uic.edu/codes/GISF2E/QGIS/GISF2E_qgis_tutorial.zip 4 | 5 | To cite, please use: Karduni, A., Kermanshah, A., & Derrible, S., 2016 “A protocol to convert spatial polyline data to network formats and applications to world urban road networks”, Scientific Data, 3:160046 6 | 7 | The article is available at: http://www.nature.com/articles/sdata201646 8 | 9 | version 1.20 - tested for pandas version 0.19.2 10 | version 1.10 - tested for pandas version 0.18.1 11 | version 1.00 - tested for pandas version 0.14.1 12 | 13 | 14 | What’s new in version 1.20 15 | 16 | - minor fix to work well with the new version of pandas 0.19.2 17 | -------------------------------------------------------------------------------- /ArcGis/v1.10/README: -------------------------------------------------------------------------------- 1 | README - GIS F2E ArcGIS Desktop - GIS Features to Edge List Tool (2D and 3D OSM version) 2 | 3 | GISF2E allows users to convert line features to networks edgelists. It works with OpenStreetMap data to create non-planar graphs and any other data source to create planar graphs. 4 | 5 | To cite, please use: Karduni, A., Kermanshah, A., & Derrible, S., 2016 “A protocol to convert spatial polyline data to network formats and applications to world urban road networks”, Scientific Data, 3:160046 6 | 7 | The article is available at: http://www.nature.com/articles/sdata201646 8 | 9 | version 1.10 - tested with version ArcGIS 10.3 , 10.4 10 | version 1.00 - tested with version ArcGIS 10.3 , 10.4 11 | 12 | 13 | Changelog: 14 | 15 | version 1.10: In this version we fixed some bugs regarding the file path names and outputs. Based on many feedback from our users, We also changed the tool to output a non-directed edgelist with only one row for each edge as our edgelists do not include information related to directed graphs. Users who need to include such information can modify the model to adapt to their specific needs. 16 | 17 | version 1.00: First version of GISF2E that creates an edgelist from openStreetMaps data. This version outputs a directed edgelist with two rows for each edge. This would allow for inclusion of edge information such as oneway or lane speeds. 18 | -------------------------------------------------------------------------------- /ArcGis/v2.00/README: -------------------------------------------------------------------------------- 1 | README - GIS F2E ArcGIS Desktop - GIS Features to Edge List Tool (2D and 3D OSM version) 2 | 3 | GISF2E allows users to convert line features to networks edgelists. It works with OpenStreetMap data to create non-planar graphs and any other data source to create planar graphs. 4 | 5 | 6 | *********NOTE: Only the file GISF2E_ArcGIS_v2.00.tbx is needed to work in ArcGIS, but the code is included for your information********* 7 | 8 | 9 | To cite, please use: Karduni, A., Kermanshah, A., & Derrible, S., 2016 “A protocol to convert spatial polyline data to network formats and applications to world urban road networks”, Scientific Data, 3:160046 10 | 11 | The article is available at: http://www.nature.com/articles/sdata201646 12 | 13 | version 2.00 - tested with version ArcGIS 10.4 , 10.6 14 | version 1.10 - tested with version ArcGIS 10.3 , 10.4 15 | version 1.00 - tested with version ArcGIS 10.3 , 10.4 16 | 17 | 18 | Changelog: 19 | 20 | version 2.00: In this version, we updated GISF2E to work with arcpy (code supplied but not needed to work in ArcGIS). We also updated the inputs needed for the tool to work to only Data Workspace and input street file. If the input streets file includes tunnel AND bridge fields with either 0,1 or True False values, the produced graph will be non-planar, if those fields do not exists, the tool will now produce a planar graph. The output edges, nodes, and edgelist will be produced in the data workspace folder. The tool now removes nodes with the degree of two that are basically artifacts not really needed for analysis. As a result, the tool now produce a cleaner graph. 21 | 22 | version 1.10: In this version we fixed some bugs regarding the file path names and outputs. Based on many feedback from our users, We also changed the tool to output a non-directed edgelist with only one row for each edge as our edgelists do not include information related to directed graphs. Users who need to include such information can modify the model to adapt to their specific needs. 23 | 24 | version 1.00: First version of GISF2E that creates an edgelist from openStreetMaps data. This version outputs a directed edgelist with two rows for each edge. This would allow for inclusion of edge information such as oneway or lane speeds. 25 | -------------------------------------------------------------------------------- /ArcGis/v2.10/README: -------------------------------------------------------------------------------- 1 | README - GIS F2E ArcGIS Desktop - GIS Features to Edge List Tool (2D and 3D OSM version) 2 | 3 | GISF2E allows users to convert line features to networks edgelists. It works with OpenStreetMap data to create non-planar graphs and any other data source to create planar graphs. 4 | 5 | 6 | *********NOTE: Only the file GISF2E_ArcGIS_v2.10.tbx is needed to work in ArcGIS, but the code is included for your information********* 7 | 8 | 9 | To cite, please use: Karduni, A., Kermanshah, A., & Derrible, S., 2016 “A protocol to convert spatial polyline data to network formats and applications to world urban road networks”, Scientific Data, 3:160046 10 | 11 | The article is available at: http://www.nature.com/articles/sdata201646 12 | 13 | version 2.10 - tested with version ArcGIS 10.7.1 14 | version 2.00 - tested with version ArcGIS 10.4 , 10.6 15 | version 1.10 - tested with version ArcGIS 10.3 , 10.4 16 | version 1.00 - tested with version ArcGIS 10.3 , 10.4 17 | 18 | 19 | Changelog: 20 | 21 | version 2.10: In this version, GISF2E was updated to work with ArcGIS version 10.7.1. 22 | 23 | version 2.00: In this version, we updated GISF2E to work with arcpy (code supplied but not needed to work in ArcGIS). We also updated the inputs needed for the tool to work to only Data Workspace and input street file. If the input streets file includes tunnel AND bridge fields with either 0,1 or True False values, the produced graph will be non-planar, if those fields do not exists, the tool will now produce a planar graph. The output edges, nodes, and edgelist will be produced in the data workspace folder. The tool now removes nodes with the degree of two that are basically artifacts not really needed for analysis. As a result, the tool now produce a cleaner graph. 24 | 25 | version 1.10: In this version we fixed some bugs regarding the file path names and outputs. Based on many feedback from our users, We also changed the tool to output a non-directed edgelist with only one row for each edge as our edgelists do not include information related to directed graphs. Users who need to include such information can modify the model to adapt to their specific needs. 26 | 27 | version 1.00: First version of GISF2E that creates an edgelist from openStreetMaps data. This version outputs a directed edgelist with two rows for each edge. This would allow for inclusion of edge information such as oneway or lane speeds. 28 | -------------------------------------------------------------------------------- /ArcGis/v2.10/GISF2E_ArcGIS_v2.10.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | ''' 3 | GISF2E ArcGIS - v2.10 4 | 5 | To cite, please use: Karduni, A., Kermanshah, A., & Derrible, S., 2016 “A protocol to convert spatial polyline data to network formats and applications to world urban road networks”, Scientific Data, 3:160046 6 | The article is available at: http://www.nature.com/articles/sdata201646 7 | 8 | Learn more at https://csun.uic.edu/codes/GISF2E.html 9 | ''' 10 | 11 | #import all libraries required 12 | import arcpy 13 | import csv 14 | import datetime 15 | 16 | data_workspace = arcpy.GetParameterAsText(0) 17 | input_streets = arcpy.GetParameterAsText(1) 18 | print (data_workspace) 19 | print (input_streets) 20 | arcpy.AddMessage("starting GISF2E!") 21 | arcpy.AddMessage(datetime.datetime.now()) 22 | 23 | 24 | test = False 25 | if not data_workspace and not input_streets: 26 | data_workspace= raw_input("Please enter path output folder: ") 27 | input_streets = raw_input("Please enter path for input line shapefile: ") 28 | path = data_workspace + "\\" 29 | 30 | arcpy.AddMessage("creating nodes, edges, and shapefiles at:") 31 | arcpy.AddMessage(path) 32 | arcpy.AddMessage("using centerline firle:") 33 | arcpy.AddMessage(input_streets) 34 | arcpy.env.overwriteOutput = True 35 | 36 | elevFields = ["bridge","tunnel"] 37 | 38 | #function to get a line class attribute from tunnel and bridge fields 39 | def class_calc(bridge, tunnel): 40 | if (bridge == 1 or bridge == "T") and (tunnel==0 or tunnel == "F"): 41 | return 1 42 | elif (tunnel==1 or tunnel == "T") and (bridge == 0 or bridge == "F"): 43 | return 2 44 | elif (bridge == 0 or bridge == "F") and (tunnel==0 or tunnel == "F"): 45 | return 0 46 | elif (bridge == 1 or bridge == "T") and (tunnel==1 or tunnel == "T") : 47 | return 3 48 | 49 | #function to get unique values from a field 50 | def unique_values(table, field): 51 | with arcpy.da.SearchCursor(table, [field]) as cursor: 52 | return sorted({row[0] for row in cursor}) 53 | 54 | 55 | #getting a list of field names from shape files 56 | fieldnames = [field.name for field in arcpy.ListFields(input_streets)] 57 | 58 | 59 | # adding class attribute to line shapefile 60 | arcpy.AddField_management(input_streets,"class","String") 61 | input_streets_cursor = arcpy.UpdateCursor(input_streets) 62 | for row in input_streets_cursor: 63 | if "tunnel" in fieldnames and "bridge" in fieldnames: 64 | bridge = row.getValue(elevFields[0]) 65 | tunnel = row.getValue(elevFields[1]) 66 | roadClass = class_calc(bridge,tunnel) 67 | row.setValue("class",roadClass) 68 | input_streets_cursor.updateRow(row) 69 | else: 70 | row.setValue("class",0) 71 | input_streets_cursor.updateRow(row) 72 | 73 | 74 | classValues = unique_values(input_streets,"class") 75 | streetClasses = [] 76 | nodeClasses = [] 77 | #seperating line by class, disolving the lines and then seperating the edges based on the intersections 78 | for classValue in classValues: 79 | streetClass = streetsLayer = arcpy.MakeFeatureLayer_management (input_streets, "streetsLayer", where_clause="class = '"+ str(classValue)+"'") 80 | streetClass = arcpy.Dissolve_management(streetClass) 81 | streetClassToLine = arcpy.FeatureToLine_management(streetClass,path+classValue+"_lines") 82 | streetClassNodes = arcpy.FeatureVerticesToPoints_management(streetClassToLine,path+classValue+"_nodes","BOTH_ENDS") 83 | nodeClasses.append(streetClassNodes) 84 | streetClasses.append(streetClassToLine) 85 | arcpy.Delete_management(streetClass) 86 | 87 | arcpy.AddMessage("Different elevation layers seperated") 88 | 89 | #merging the results, deleting the seperate line files 90 | mergedEdges = arcpy.Merge_management(streetClasses,path+"edges.shp") 91 | arcpy.AddGeometryAttributes_management(mergedEdges,"LENGTH_GEODESIC","METERS") 92 | arcpy.AddField_management(mergedEdges,"LENGTH") 93 | arcpy.CalculateField_management(mergedEdges,"LENGTH","[LENGTH_GEO]") 94 | mergedNodes = arcpy.Merge_management(nodeClasses,path+"nodes.shp") 95 | for edge in streetClasses: 96 | arcpy.Delete_management(edge) 97 | for node in nodeClasses: 98 | arcpy.Delete_management(node) 99 | 100 | mergedNodes = arcpy.DeleteIdentical_management(mergedNodes,"Shape") 101 | 102 | arcpy.AddMessage("Created nodes and edges shapefiles") 103 | 104 | # spatial join between edges file and nodes file to find start nodes and end nodes for each edge 105 | EdgeNodeSpatialJoin = arcpy.SpatialJoin_analysis(mergedEdges,mergedNodes,path+"edgeNodeSpatialJoin.shp","JOIN_ONE_TO_MANY","KEEP_ALL",match_option="INTERSECT",) 106 | 107 | arcpy.AddMessage("edge node spatial join completed") 108 | 109 | # spatial join nodes to the previous spatial join to get start node, end node and edges 110 | NodeEdgeSpatialJoin = arcpy.SpatialJoin_analysis(mergedNodes,EdgeNodeSpatialJoin,path+"NodeEdgeSpatialJoin.shp","JOIN_ONE_TO_MANY","KEEP_ALL",match_option="INTERSECT") 111 | 112 | arcpy.AddMessage("node edge spatial join completed") 113 | 114 | 115 | #deleting duplocate edges 116 | outputEdgelist = [] 117 | arcpy.DeleteIdentical_management(NodeEdgeSpatialJoin,"TARGET_F_1") 118 | outCursor = arcpy.SearchCursor(NodeEdgeSpatialJoin) 119 | arcpy.AddMessage("creating Edge list") 120 | 121 | #creating edgelist 122 | for row in outCursor: 123 | startNode = row.getValue("TARGET_F_1") 124 | endNode = row.getValue("JOIN_FID") 125 | edge = row.getValue("TARGET_FID") 126 | length = row.getValue("LENGTH") 127 | if startNode != endNode: 128 | edgelist = (startNode,endNode,edge,length) 129 | outputEdgelist.append(edgelist) 130 | 131 | arcpy.AddMessage("created Edge list") 132 | arcpy.AddMessage("Writing edge list to file") 133 | 134 | #writing results to edgelist csv 135 | with open(path+"edgelist.csv","w") as csvFile: 136 | csvWriter = csv.writer(csvFile, lineterminator="\n") 137 | csvWriter.writerow(["Start_Node","End_Node","Edge","Length"]) 138 | csvWriter.writerows(outputEdgelist) 139 | 140 | arcpy.AddMessage("Deleting extra files") 141 | arcpy.Delete_management(EdgeNodeSpatialJoin) 142 | arcpy.Delete_management(NodeEdgeSpatialJoin) 143 | arcpy.AddMessage(datetime.datetime.now()) 144 | arcpy.AddMessage("Done!") 145 | -------------------------------------------------------------------------------- /ArcGis/v2.00/GISF2E_ArcGIS_v2.00.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | ''' 3 | GISF2E ArcGIS - v2.00 4 | 5 | To cite, please use: Karduni, A., Kermanshah, A., & Derrible, S., 2016 “A protocol to convert spatial polyline data to network formats and applications to world urban road networks”, Scientific Data, 3:160046 6 | The article is available at: http://www.nature.com/articles/sdata201646 7 | 8 | Learn more at https://csun.uic.edu/codes/GISF2E.html 9 | ''' 10 | 11 | #import all libraries required 12 | import arcpy 13 | import csv 14 | import datetime 15 | 16 | data_workspace = arcpy.GetParameterAsText(0) 17 | input_streets = arcpy.GetParameterAsText(1) 18 | print (data_workspace) 19 | print (input_streets) 20 | arcpy.AddMessage("starting GISF2E!") 21 | arcpy.AddMessage(datetime.datetime.now()) 22 | 23 | 24 | test = False 25 | if not data_workspace and not input_streets: 26 | data_workspace= raw_input("Please enter path output folder: ") 27 | input_streets = raw_input("Please enter path for input line shapefile: ") 28 | path = data_workspace + "\\" 29 | 30 | arcpy.AddMessage("creating nodes, edges, and shapefiles at:") 31 | arcpy.AddMessage(path) 32 | arcpy.AddMessage("using centerline firle:") 33 | arcpy.AddMessage(input_streets) 34 | arcpy.env.overwriteOutput = True 35 | 36 | elevFields = ["bridge","tunnel"] 37 | 38 | #function to get a line class attribute from tunnel and bridge fields 39 | def class_calc(bridge, tunnel): 40 | if (bridge == 1 or bridge == "T") and (tunnel==0 or tunnel == "F"): 41 | return 1 42 | elif (tunnel==1 or tunnel == "T") and (bridge == 0 or bridge == "F"): 43 | return 2 44 | elif (bridge == 0 or bridge == "F") and (tunnel==0 or tunnel == "F"): 45 | return 0 46 | elif (bridge == 1 or bridge == "T") and (tunnel==1 or tunnel == "T") : 47 | return 3 48 | 49 | #function to get unique values from a field 50 | def unique_values(table, field): 51 | with arcpy.da.SearchCursor(table, [field]) as cursor: 52 | return sorted({row[0] for row in cursor}) 53 | 54 | 55 | #getting a list of field names from shape files 56 | fieldnames = [field.name for field in arcpy.ListFields(input_streets)] 57 | 58 | 59 | # adding class attribute to line shapefile 60 | arcpy.AddField_management(input_streets,"class","String") 61 | input_streets_cursor = arcpy.UpdateCursor(input_streets) 62 | for row in input_streets_cursor: 63 | if "tunnel" in fieldnames and "bridge" in fieldnames: 64 | bridge = row.getValue(elevFields[0]) 65 | tunnel = row.getValue(elevFields[1]) 66 | roadClass = class_calc(bridge,tunnel) 67 | row.setValue("class",roadClass) 68 | input_streets_cursor.updateRow(row) 69 | else: 70 | row.setValue("class",0) 71 | input_streets_cursor.updateRow(row) 72 | 73 | 74 | classValues = unique_values(input_streets,"class") 75 | streetClasses = [] 76 | nodeClasses = [] 77 | #seperating line by class, disolving the lines and then seperating the edges based on the intersections 78 | for classValue in classValues: 79 | streetClass = streetsLayer = arcpy.MakeFeatureLayer_management (input_streets, "streetsLayer", where_clause="class = '"+ str(classValue)+"'") 80 | streetClass = arcpy.Dissolve_management(streetClass) 81 | streetClassToLine = arcpy.FeatureToLine_management(streetClass,path+classValue+"_lines") 82 | streetClassNodes = arcpy.FeatureVerticesToPoints_management(streetClassToLine,path+classValue+"_nodes","BOTH_ENDS") 83 | nodeClasses.append(streetClassNodes) 84 | streetClasses.append(streetClassToLine) 85 | arcpy.Delete_management(streetClass) 86 | 87 | arcpy.AddMessage("Different elevation layers seperated") 88 | 89 | #merging the results, deleting the seperate line files 90 | mergedEdges = arcpy.Merge_management(streetClasses,path+"edges.shp") 91 | arcpy.AddGeometryAttributes_management(mergedEdges,"LENGTH_GEODESIC","METERS") 92 | arcpy.AddField_management(mergedEdges,"LENGTH") 93 | arcpy.CalculateField_management(mergedEdges,"LENGTH","[LENGTH_GEO]") 94 | mergedNodes = arcpy.Merge_management(nodeClasses,path+"nodes.shp") 95 | for edge in streetClasses: 96 | arcpy.Delete_management(edge) 97 | for node in nodeClasses: 98 | arcpy.Delete_management(node) 99 | 100 | mergedNodes = arcpy.DeleteIdentical_management(mergedNodes,"Shape") 101 | 102 | arcpy.AddMessage("Created nodes and edges shapefiles") 103 | 104 | # spatial join between edges file and nodes file to find start nodes and end nodes for each edge 105 | EdgeNodeSpatialJoin = arcpy.SpatialJoin_analysis(mergedEdges,mergedNodes,path+"edgeNodeSpatialJoin.shp","JOIN_ONE_TO_MANY","KEEP_ALL",match_option="INTERSECT",) 106 | 107 | arcpy.AddMessage("edge node spatial join completed") 108 | 109 | # spatial join nodes to the previous spatial join to get start node, end node and edges 110 | NodeEdgeSpatialJoin = arcpy.SpatialJoin_analysis(mergedNodes,EdgeNodeSpatialJoin,path+"NodeEdgeSpatialJoin.shp","JOIN_ONE_TO_MANY","KEEP_ALL",match_option="INTERSECT") 111 | 112 | arcpy.AddMessage("node edge spatial join completed") 113 | 114 | 115 | #deleting duplocate edges 116 | outputEdgelist = [] 117 | arcpy.DeleteIdentical_management(NodeEdgeSpatialJoin,"TARGET_F_1") 118 | outCursor = arcpy.SearchCursor(NodeEdgeSpatialJoin) 119 | arcpy.AddMessage("creating Edge list") 120 | 121 | #creating edgelist 122 | for row in outCursor: 123 | startNode = row.getValue("TARGET_F_1") 124 | endNode = row.getValue("JOIN_FID") 125 | edge = row.getValue("TARGET_FID") 126 | length = row.getValue("LENGTH") 127 | if startNode != endNode: 128 | edgelist = (startNode,endNode,edge,length) 129 | outputEdgelist.append(edgelist) 130 | 131 | arcpy.AddMessage("created Edge list") 132 | arcpy.AddMessage("Writing edge list to file") 133 | 134 | #writing results to edgelist csv 135 | with open(path+"edgelist.csv","w") as csvFile: 136 | csvWriter = csv.writer(csvFile, lineterminator="\n") 137 | csvWriter.writerow(["Start_Node","End_Node","Edge","Length"]) 138 | csvWriter.writerows(outputEdgelist) 139 | 140 | arcpy.AddMessage("Deleting extra files") 141 | arcpy.Delete_management(EdgeNodeSpatialJoin) 142 | arcpy.Delete_management(NodeEdgeSpatialJoin) 143 | arcpy.AddMessage(datetime.datetime.now()) 144 | arcpy.AddMessage("Done!") 145 | -------------------------------------------------------------------------------- /QGIS/v1.10/GISF2E_qgis_non-OSM_v1.10.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | ''' 3 | GISF2E QGIS non-OSM - v1.10 4 | 5 | To cite, please use: Karduni, A., Kermanshah, A., & Derrible, S., 2016 “A protocol to convert spatial polyline data to network formats and applications to world urban road networks”, Scientific Data, 3:160046 6 | The article is available at: http://www.nature.com/articles/sdata201646 7 | 8 | Main authors: Felipe Macena, Rodrigo Marinho 9 | ''' 10 | 11 | #import all libraries required 12 | import shapefile 13 | import pandas as pd 14 | from pyproj import Proj 15 | import osgeo.ogr, osgeo.osr 16 | from osgeo import ogr 17 | import sys 18 | import os 19 | import time 20 | from math import pi, sin, cos, tan, sqrt 21 | from qgis.core import * 22 | def LLtoUTM(ReferenceEllipsoid, Lat, Long, zone = None): 23 | """converts lat/long to UTM coords. Equations from USGS Bulletin 1532 24 | East Longitudes are positive, West longitudes are negative. 25 | North latitudes are positive, South latitudes are negative 26 | Lat and Long are in decimal degrees 27 | Written by Chuck Gantz- chuck.gantz@globalstar.com""" 28 | 29 | a = _ellipsoid[ReferenceEllipsoid][_EquatorialRadius] 30 | eccSquared = _ellipsoid[ReferenceEllipsoid][_eccentricitySquared] 31 | k0 = 0.9996 32 | 33 | #Make sure the longitude is between -180.00 .. 179.9 34 | LongTemp = (Long+180)-int((Long+180)/360)*360-180 # -180.00 .. 179.9 35 | 36 | LatRad = Lat*_deg2rad 37 | LongRad = LongTemp*_deg2rad 38 | 39 | if zone is None: 40 | ZoneNumber = int((LongTemp + 180)/6) + 1 41 | else: 42 | ZoneNumber = zone 43 | 44 | if Lat >= 56.0 and Lat < 64.0 and LongTemp >= 3.0 and LongTemp < 12.0: 45 | ZoneNumber = 32 46 | 47 | # Special zones for Svalbard 48 | if Lat >= 72.0 and Lat < 84.0: 49 | if LongTemp >= 0.0 and LongTemp < 9.0:ZoneNumber = 31 50 | elif LongTemp >= 9.0 and LongTemp < 21.0: ZoneNumber = 33 51 | elif LongTemp >= 21.0 and LongTemp < 33.0: ZoneNumber = 35 52 | elif LongTemp >= 33.0 and LongTemp < 42.0: ZoneNumber = 37 53 | 54 | LongOrigin = (ZoneNumber - 1)*6 - 180 + 3 #+3 puts origin in middle of zone 55 | LongOriginRad = LongOrigin * _deg2rad 56 | 57 | #compute the UTM Zone from the latitude and longitude 58 | UTMZone = "%d%c" % (ZoneNumber, _UTMLetterDesignator(Lat)) 59 | 60 | eccPrimeSquared = (eccSquared)/(1-eccSquared) 61 | N = a/sqrt(1-eccSquared*sin(LatRad)*sin(LatRad)) 62 | T = tan(LatRad)*tan(LatRad) 63 | C = eccPrimeSquared*cos(LatRad)*cos(LatRad) 64 | A = cos(LatRad)*(LongRad-LongOriginRad) 65 | 66 | M = a*((1 67 | - eccSquared/4 68 | - 3*eccSquared*eccSquared/64 69 | - 5*eccSquared*eccSquared*eccSquared/256)*LatRad 70 | - (3*eccSquared/8 71 | + 3*eccSquared*eccSquared/32 72 | + 45*eccSquared*eccSquared*eccSquared/1024)*sin(2*LatRad) 73 | + (15*eccSquared*eccSquared/256 + 45*eccSquared*eccSquared*eccSquared/1024)*sin(4*LatRad) 74 | - (35*eccSquared*eccSquared*eccSquared/3072)*sin(6*LatRad)) 75 | 76 | UTMEasting = (k0*N*(A+(1-T+C)*A*A*A/6 77 | + (5-18*T+T*T+72*C-58*eccPrimeSquared)*A*A*A*A*A/120) 78 | + 500000.0) 79 | 80 | UTMNorthing = (k0*(M+N*tan(LatRad)*(A*A/2+(5-T+9*C+4*C*C)*A*A*A*A/24 81 | + (61 82 | -58*T 83 | +T*T 84 | +600*C 85 | -330*eccPrimeSquared)*A*A*A*A*A*A/720))) 86 | if Lat < 0: 87 | UTMNorthing = UTMNorthing + 10000000.0; #10000000 meter offset for southern hemisphere 88 | return (UTMZone, UTMEasting, UTMNorthing) 89 | 90 | 91 | def _UTMLetterDesignator(Lat): 92 | """This routine determines the correct UTM letter designator for the given 93 | latitude returns 'Z' if latitude is outside the UTM limits of 84N to 80S 94 | Written by Chuck Gantz- chuck.gantz@globalstar.com""" 95 | if Lat >= 0: return 'N' 96 | else: return 'S' 97 | 98 | def add_link(start,end,cont): 99 | pontos = [] 100 | pontos.append(osgeo.ogr.Geometry(osgeo.ogr.wkbLineString)) 101 | lista = list([testando['coordinates_utm'][start:end+1]][0]) 102 | for a in lista: 103 | pontos[-1].AddPoint(a[0],a[1]) 104 | featureIndex = cont 105 | feature = osgeo.ogr.Feature(layer_defn) 106 | feature.SetGeometry(pontos[-1]) 107 | feature.SetFID(featureIndex) 108 | layer.CreateFeature(feature) 109 | for name in range(len(field_names)): 110 | if field_names[name] != 'Edge' and field_names[name] != 'Length': 111 | feature.SetField(name, str(testando[field_names[name]][start_node])) 112 | elif field_names[name] == 'Edge': 113 | feature.SetField(name, cont+1) 114 | elif field_names[name] == 'Length': 115 | feature.SetField(name, distance(start,end)) 116 | layer.SetFeature(feature) 117 | def add_field(field_names,featureIndex): 118 | for name in range(len(field_names)): 119 | if field_names[name] == 'coord_x' or field_names[name] == 'coord_y': 120 | name_aux = 'coordinates_utm' 121 | else: 122 | name_aux = field_names[name] 123 | feature = layer.GetFeature(featureIndex) #lets get the first feature (FID=='0') 124 | if field_names[name] == 'coord_x': 125 | feature.SetField(name,testando_nodes[name_aux][featureIndex][0]) 126 | elif field_names[name] == 'coord_y': 127 | feature.SetField(name,testando_nodes[name_aux][featureIndex][1]) 128 | else: 129 | feature.SetField(name,int(testando_nodes[name_aux][featureIndex])) 130 | layer.SetFeature(feature) #now make the change permanent 131 | def distance(start,end): 132 | dist = 0 133 | for u in range(start,end): 134 | dist = dist + ((testando['coordinates_utm'][u][0] - testando['coordinates_utm'][u+1][0])**2 + (testando['coordinates_utm'][u][1] - testando['coordinates_utm'][u+1][1])**2)**0.5 135 | return dist 136 | def Edgelist(start,end,edge,length): 137 | global testando, columns 138 | datasheet = pd.DataFrame([[testando['coordinates_utm'][start][0],testando['coordinates_utm'][start][1],testando['coordinates_utm'][end][0],testando['coordinates_utm'][end][1],testando['Node'][start],testando['Node'][end],edge,length]],columns = columns, index = None) 139 | return datasheet 140 | 141 | _deg2rad, _rad2deg, _EquatorialRadius, _eccentricitySquared = pi / 180.0, 180.0 / pi, 2, 3 142 | 143 | _ellipsoid = [ 144 | # id, Ellipsoid name, Equatorial Radius, square of eccentricity 145 | # first once is a placeholder only, To allow array indices to match id numbers 146 | [ -1, "Placeholder", 0, 0], 147 | [ 1, "Airy", 6377563, 0.00667054], 148 | [ 2, "Australian National", 6378160, 0.006694542], 149 | [ 3, "Bessel 1841", 6377397, 0.006674372], 150 | [ 4, "Bessel 1841 (Nambia] ", 6377484, 0.006674372], 151 | [ 5, "Clarke 1866", 6378206, 0.006768658], 152 | [ 6, "Clarke 1880", 6378249, 0.006803511], 153 | [ 7, "Everest", 6377276, 0.006637847], 154 | [ 8, "Fischer 1960 (Mercury] ", 6378166, 0.006693422], 155 | [ 9, "Fischer 1968", 6378150, 0.006693422], 156 | [ 10, "GRS 1967", 6378160, 0.006694605], 157 | [ 11, "GRS 1980", 6378137, 0.00669438], 158 | [ 12, "Helmert 1906", 6378200, 0.006693422], 159 | [ 13, "Hough", 6378270, 0.00672267], 160 | [ 14, "International", 6378388, 0.00672267], 161 | [ 15, "Krassovsky", 6378245, 0.006693422], 162 | [ 16, "Modified Airy", 6377340, 0.00667054], 163 | [ 17, "Modified Everest", 6377304, 0.006637847], 164 | [ 18, "Modified Fischer 1960", 6378155, 0.006693422], 165 | [ 19, "South American 1969", 6378160, 0.006694542], 166 | [ 20, "WGS 60", 6378165, 0.006693422], 167 | [ 21, "WGS 66", 6378145, 0.006694542], 168 | [ 22, "WGS-72", 6378135, 0.006694318], 169 | [ 23, "WGS-84", 6378137, 0.00669438] 170 | ] 171 | 172 | 173 | 174 | #dividing each link by their osm_id information (osm_maps) or id information (non-osm_maps) 175 | 176 | sf = shapefile.Reader("C:\YOUR_PATH\input\Shape3.shp") 177 | shapes = sf.shapes() 178 | fields=sf.fields 179 | fields = [field[0] for field in sf.fields[1:]] 180 | start = time.time() 181 | print '\n'+str(time.time() - start)+'s - Reading and arranging the root file information...' 182 | sys.stdout.flush() 183 | #here you can change the utm zone depending on where the map is from 184 | #p = Proj(proj='utm',zone=int(utm_zone[:-1]),ellps='WGS84') 185 | 186 | if 'TYPE' in fields or 'Type' in fields: 187 | try: 188 | fields[fields.index('TYPE')] = 'type' 189 | except: 190 | fields[fields.index('Type')] = 'type' 191 | 192 | heads = [x.upper() if x in ['Group','coordinates','primeiro','coordinates_utm','Node'] else x for x in [sf.shapeRecords()[0].shape.__geo_interface__.keys()[0]]+fields] + ['Group','coordinates','primeiro','coordinates_utm','Node'] 193 | type_ = False 194 | if 'type' in map(lambda x: x.lower(),fields): 195 | heads.remove('type') 196 | heads.insert(3,'type 1') 197 | type_ = True 198 | 199 | dicti = {} 200 | for i in range(len(heads)-1): 201 | dicti[heads[i]] = [] 202 | 203 | testando = pd.DataFrame() 204 | 205 | n = 0 206 | for feature in sf.shapeRecords(): 207 | geom = feature.shape.__geo_interface__ 208 | atr = dict(zip(fields, feature.record)) 209 | if type_: 210 | atr['type 1'] = atr.pop('type') 211 | geom.update(atr) 212 | geom['Group'] = n 213 | j = geom['coordinates'] 214 | for i in range(len(heads)-4): 215 | u = [geom[heads[i]]] 216 | for y in range(len(j)): 217 | dicti[heads[i]].append(u[0]) 218 | dicti['coordinates'] = list(dicti['coordinates']) + list(j) 219 | dicti['primeiro'] = dicti['primeiro'] + [1] + [0]*(len(j) - 2) + [1] 220 | n += 1 221 | if ((n-1) % 800) == 0: 222 | dicti['coordinates_utm'] = None 223 | dicti['Node'] = None 224 | testando = pd.concat([testando,pd.DataFrame.from_dict(dicti)]) 225 | for i in range(len(heads)-1): 226 | dicti[heads[i]] = [] 227 | if ((n-1) % 800) != 0: 228 | dicti['coordinates_utm'] = None 229 | dicti['Node'] = None 230 | testando = pd.concat([testando,pd.DataFrame.from_dict(dicti)],ignore_index=True) 231 | 232 | utm_zone = LLtoUTM(23, testando['coordinates'][0][1], testando['coordinates'][0][0])[0] 233 | p = Proj(proj='utm',zone=int(utm_zone[:-1]),ellps='WGS84') 234 | 235 | testando['coordinates_utm'] = [p(testando['coordinates'][k][0],testando['coordinates'][k][1]) for k in range(len(testando['coordinates']))] 236 | testando['Node'] = None 237 | 238 | #testando = pd.DataFrame.from_dict(dicti) 239 | dicti = None 240 | geom = None 241 | atr = None 242 | #test before creating nodes for OSM maps 243 | a= testando.groupby('coordinates_utm').groups 244 | testando['continuidade'] = 1 245 | testando.ix[testando.primeiro == 1,'continuidade'] = 0; testando 246 | b1 = map(lambda x: (testando['coordinates_utm'][x],[x]),testando.groupby('continuidade').groups[0]) 247 | 248 | b = filter(lambda x: len(x[1]) >= 2, zip(a.keys(),a.values())) 249 | 250 | for h in (b): 251 | for k in range(len(h[1])): 252 | testando.loc[[h[1][k]], 'continuidade'] = 2 253 | b = b + b1 254 | points = sorted(testando.groupby('continuidade').groups[0] + testando.groupby('continuidade').groups[2]) 255 | testando.loc[points, 'Node'] = range(1,len(points)+1) 256 | 257 | print '\n'+str(time.time() - start)+'s - Reading and arranging are done!' 258 | sys.stdout.flush() 259 | 260 | #creating nodes shapefile 261 | path='C:\YOUR_PATH\output\Shape3_nodes.shp' 262 | spatialReference = osgeo.osr.SpatialReference() #will create a spatial reference locally to tell the system what the reference will be 263 | spatialReference.ImportFromProj4('+proj=utm +zone='+utm_zone+' ellps=WGS84 +datum=WGS84 +units=m') #here we define this reference to be utm Zone 16N with wgs84 264 | driver = osgeo.ogr.GetDriverByName('ESRI Shapefile') # will select the driver for our shp-file creation. 265 | shapeData = driver.CreateDataSource(path) #so there we will store our data 266 | layer = shapeData.CreateLayer('Nodes', spatialReference, osgeo.ogr.wkbPoint) #this will create a corresponding layer for our data with given spatial information 267 | layer_defn = layer.GetLayerDefn() # gets parameters of the current shapefile 268 | point = osgeo.ogr.Geometry(osgeo.ogr.wkbPoint) 269 | 270 | new_field = ogr.FieldDefn('Node', ogr.OFTInteger) #and a forth field 'oneway' stored as integer 271 | layer.CreateField(new_field) #self explaining 272 | new_field = ogr.FieldDefn('coord_x', ogr.OFTReal) #and a forth field 'oneway' stored as integer 273 | layer.CreateField(new_field) #self explaining 274 | new_field = ogr.FieldDefn('coord_y', ogr.OFTReal) #and a forth field 'oneway' stored as integer 275 | layer.CreateField(new_field) #self explaining 276 | 277 | 278 | field_names = [layer_defn.GetFieldDefn(i).GetName() for i in range(layer_defn.GetFieldCount())] 279 | aux_b, featureIndex = list(zip(*b)[0]), 0 280 | 281 | testando_nodes = testando[testando.continuidade != 1].drop_duplicates(['coordinates_utm']) 282 | testando_nodes.index = range(len(testando_nodes)) 283 | 284 | t, aux_b, a, b, b1, points = None, None, None, None, None, None 285 | 286 | print '\n'+str(time.time() - start)+'s - Adding nodes...' 287 | sys.stdout.flush() 288 | 289 | for featureIndex in range(len(testando_nodes)): 290 | point.AddPoint(testando_nodes['coordinates_utm'][featureIndex][0],testando_nodes['coordinates_utm'][featureIndex][1]) #create a new point at given coordinates 291 | feature = osgeo.ogr.Feature(layer_defn) 292 | feature.SetGeometry(point) 293 | feature.SetFID(featureIndex) 294 | layer.CreateFeature(feature) 295 | add_field(field_names,featureIndex) 296 | shapeData = None 297 | 298 | new_b = None 299 | testando_nodes = None 300 | 301 | print '\n'+str(time.time() - start)+'s - Nodes are done!' 302 | sys.stdout.flush() 303 | 304 | #Cut the raw links and Creat the edge shapefile 305 | columns = ['X_Coord_Start','Y_Coord_Start','X_Coord_End','Y_Coord_End','Start_Node','End_Node','Edge','Length'] 306 | edgelist = pd.DataFrame(columns = columns, index=None) 307 | 308 | 309 | 310 | 311 | #creating edges shapefile 312 | path='C:\YOUR_PATH\output\Shape3_edges.shp' 313 | spatialReference = osgeo.osr.SpatialReference() #will create a spatial reference locally to tell the system what the reference will be 314 | #here you can change the utm zone depending on where the map is from 315 | spatialReference.ImportFromProj4('+proj=utm +zone='+utm_zone+' ellps=WGS84 +datum=WGS84 +units=m') #here we define this reference to be utm Zone 16N with wgs84 316 | driver = osgeo.ogr.GetDriverByName('ESRI Shapefile') # will select the driver for our shp-file creation. 317 | shapeData = driver.CreateDataSource(path) #so there we will store our data 318 | layer = shapeData.CreateLayer('linestrings', spatialReference, osgeo.ogr.wkbLineString) #this will create a corresponding layer for our data with given spatial information. 319 | layer_defn = layer.GetLayerDefn() # gets parameters of the current shapefile 320 | 321 | for f in ['Edge']+heads[:-5]+['Length']: 322 | fieldDefn = ogr.FieldDefn(f, ogr.OFTString) 323 | layer.CreateField(fieldDefn) 324 | field_names = [layer_defn.GetFieldDefn(i).GetName() for i in range(layer_defn.GetFieldCount())] 325 | 326 | print '\n'+str(time.time() - start)+'s - Adding links and creating the Edgelist...' 327 | sys.stdout.flush() 328 | 329 | links = (testando.groupby(['Group']).groups) 330 | links1 = (testando.groupby(['Group','continuidade']).groups) 331 | cont = 0 332 | for link in range(len(links)): 333 | try: 334 | length_0 = len(links1[link,0]) 335 | except: 336 | length_0 = -1 337 | try: 338 | length_1 = len(links1[link,1]) 339 | except: 340 | length_1 = -1 341 | try: 342 | length_2 = len(links1[link,2]) 343 | except: 344 | length_2 = -1 345 | if length_0 == 1 and length_2 == 1: 346 | start_node, end_node = min(links1[link,0][0],links1[link,2][0]), max(links1[link,0][0],links1[link,2][0]) 347 | add_link(start_node,end_node,cont) 348 | edgelist = edgelist.append(Edgelist(start_node,end_node,cont+1,distance(start_node,end_node)),ignore_index=True) 349 | cont += 1 350 | elif length_0 == -1: 351 | for i in range(length_2-1): 352 | start_node, end_node = links1[link,2][i], links1[link,2][i+1] 353 | add_link(start_node,end_node,cont) 354 | edgelist = edgelist.append(Edgelist(start_node,end_node,cont+1,distance(start_node,end_node)),ignore_index=True) 355 | cont += 1 356 | elif length_2 == -1: 357 | start_node, end_node = links1[link,0][0], links1[link,0][1] 358 | add_link(start_node,end_node,cont) 359 | edgelist = edgelist.append(Edgelist(start_node,end_node,cont+1,distance(start_node,end_node)),ignore_index=True) 360 | cont += 1 361 | else: 362 | if length_0 > 1: 363 | start_node, end_node = links1[link,0][0], links1[link,2][0] 364 | add_link(start_node,end_node,cont) 365 | edgelist = edgelist.append(Edgelist(start_node,end_node,cont+1,distance(start_node,end_node)),ignore_index=True) 366 | cont += 1 367 | for i in range(length_2-1): 368 | start_node, end_node = links1[link,2][i], links1[link,2][i+1] 369 | add_link(start_node,end_node,cont) 370 | edgelist = edgelist.append(Edgelist(start_node,end_node,cont+1,distance(start_node,end_node)),ignore_index=True) 371 | cont += 1 372 | start_node, end_node = links1[link,2][-1], links1[link,0][1] 373 | add_link(start_node,end_node,cont) 374 | edgelist = edgelist.append(Edgelist(start_node,end_node,cont+1,distance(start_node,end_node)),ignore_index=True) 375 | cont += 1 376 | else: 377 | if links1[link,0][0] < links1[link,2][0]: 378 | start_node, end_node = links1[link,0][0], links1[link,2][0] 379 | add_link(start_node,end_node,cont) 380 | edgelist = edgelist.append(Edgelist(start_node,end_node,cont+1,distance(start_node,end_node)),ignore_index=True) 381 | cont += 1 382 | for i in range(length_2 - 1): 383 | start_node, end_node = links1[link,2][i], links1[link,2][i+1] 384 | add_link(start_node,end_node,cont) 385 | edgelist = edgelist.append(Edgelist(start_node,end_node,cont+1,distance(start_node,end_node)),ignore_index=True) 386 | cont += 1 387 | else: 388 | for i in range(length_2 - 1): 389 | start_node, end_node = links1[link,2][i], links1[link,2][i+1] 390 | add_link(start_node,end_node,cont) 391 | edgelist = edgelist.append(Edgelist(start_node,end_node,cont+1,distance(start_node,end_node)),ignore_index=True) 392 | cont += 1 393 | start_node, end_node = links1[link,2][-1], links1[link,0][0] 394 | add_link(start_node,end_node,cont) 395 | edgelist = edgelist.append(Edgelist(start_node,end_node,cont+1,distance(start_node,end_node)),ignore_index=True) 396 | cont += 1 397 | shapeData = None 398 | 399 | print '\n'+str(time.time() - start)+'s - Links and Edgelist are done!' 400 | sys.stdout.flush() 401 | 402 | print '\n'+str(time.time() - start)+'s - Saving Edgelist...' 403 | sys.stdout.flush() 404 | 405 | edgelist.to_csv('C:\YOUR_PATH\output\Shape3_edgelist.csv',index=False) 406 | 407 | wb = QgsVectorLayer('C:\YOUR_PATH\output\Shape3_edges.shp', 'Shape3_edges', 'ogr') 408 | QgsMapLayerRegistry.instance().addMapLayer(wb) 409 | wb = QgsVectorLayer('C:\YOUR_PATH\output\Shape3_nodes.shp', 'Shape3_nodes', 'ogr') 410 | QgsMapLayerRegistry.instance().addMapLayer(wb) 411 | print '\n'+str(time.time() - start)+'s - You are all set.' 412 | sys.stdout.flush() 413 | -------------------------------------------------------------------------------- /QGIS/v1.20/GISF2E_qgis_non-OSM_v1.20.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | ''' 3 | GISF2E QGIS non-OSM - v1.20 4 | 5 | To cite, please use: Karduni, A., Kermanshah, A., & Derrible, S., 2016 “A protocol to convert spatial polyline data to network formats and applications to world urban road networks”, Scientific Data, 3:160046 6 | The article is available at: http://www.nature.com/articles/sdata201646 7 | 8 | Main authors: Felipe Macena, Rodrigo Marinho 9 | ''' 10 | 11 | #import all libraries required 12 | import shapefile 13 | import pandas as pd 14 | from pyproj import Proj 15 | import osgeo.ogr, osgeo.osr 16 | from osgeo import ogr 17 | import sys 18 | import os 19 | import time 20 | from math import pi, sin, cos, tan, sqrt 21 | from qgis.core import * 22 | def LLtoUTM(ReferenceEllipsoid, Lat, Long, zone = None): 23 | """converts lat/long to UTM coords. Equations from USGS Bulletin 1532 24 | East Longitudes are positive, West longitudes are negative. 25 | North latitudes are positive, South latitudes are negative 26 | Lat and Long are in decimal degrees 27 | Written by Chuck Gantz- chuck.gantz@globalstar.com""" 28 | 29 | a = _ellipsoid[ReferenceEllipsoid][_EquatorialRadius] 30 | eccSquared = _ellipsoid[ReferenceEllipsoid][_eccentricitySquared] 31 | k0 = 0.9996 32 | 33 | #Make sure the longitude is between -180.00 .. 179.9 34 | LongTemp = (Long+180)-int((Long+180)/360)*360-180 # -180.00 .. 179.9 35 | 36 | LatRad = Lat*_deg2rad 37 | LongRad = LongTemp*_deg2rad 38 | 39 | if zone is None: 40 | ZoneNumber = int((LongTemp + 180)/6) + 1 41 | else: 42 | ZoneNumber = zone 43 | 44 | if Lat >= 56.0 and Lat < 64.0 and LongTemp >= 3.0 and LongTemp < 12.0: 45 | ZoneNumber = 32 46 | 47 | # Special zones for Svalbard 48 | if Lat >= 72.0 and Lat < 84.0: 49 | if LongTemp >= 0.0 and LongTemp < 9.0:ZoneNumber = 31 50 | elif LongTemp >= 9.0 and LongTemp < 21.0: ZoneNumber = 33 51 | elif LongTemp >= 21.0 and LongTemp < 33.0: ZoneNumber = 35 52 | elif LongTemp >= 33.0 and LongTemp < 42.0: ZoneNumber = 37 53 | 54 | LongOrigin = (ZoneNumber - 1)*6 - 180 + 3 #+3 puts origin in middle of zone 55 | LongOriginRad = LongOrigin * _deg2rad 56 | 57 | #compute the UTM Zone from the latitude and longitude 58 | UTMZone = "%d%c" % (ZoneNumber, _UTMLetterDesignator(Lat)) 59 | 60 | eccPrimeSquared = (eccSquared)/(1-eccSquared) 61 | N = a/sqrt(1-eccSquared*sin(LatRad)*sin(LatRad)) 62 | T = tan(LatRad)*tan(LatRad) 63 | C = eccPrimeSquared*cos(LatRad)*cos(LatRad) 64 | A = cos(LatRad)*(LongRad-LongOriginRad) 65 | 66 | M = a*((1 67 | - eccSquared/4 68 | - 3*eccSquared*eccSquared/64 69 | - 5*eccSquared*eccSquared*eccSquared/256)*LatRad 70 | - (3*eccSquared/8 71 | + 3*eccSquared*eccSquared/32 72 | + 45*eccSquared*eccSquared*eccSquared/1024)*sin(2*LatRad) 73 | + (15*eccSquared*eccSquared/256 + 45*eccSquared*eccSquared*eccSquared/1024)*sin(4*LatRad) 74 | - (35*eccSquared*eccSquared*eccSquared/3072)*sin(6*LatRad)) 75 | 76 | UTMEasting = (k0*N*(A+(1-T+C)*A*A*A/6 77 | + (5-18*T+T*T+72*C-58*eccPrimeSquared)*A*A*A*A*A/120) 78 | + 500000.0) 79 | 80 | UTMNorthing = (k0*(M+N*tan(LatRad)*(A*A/2+(5-T+9*C+4*C*C)*A*A*A*A/24 81 | + (61 82 | -58*T 83 | +T*T 84 | +600*C 85 | -330*eccPrimeSquared)*A*A*A*A*A*A/720))) 86 | if Lat < 0: 87 | UTMNorthing = UTMNorthing + 10000000.0; #10000000 meter offset for southern hemisphere 88 | return (UTMZone, UTMEasting, UTMNorthing) 89 | 90 | 91 | def _UTMLetterDesignator(Lat): 92 | """This routine determines the correct UTM letter designator for the given 93 | latitude returns 'Z' if latitude is outside the UTM limits of 84N to 80S 94 | Written by Chuck Gantz- chuck.gantz@globalstar.com""" 95 | if Lat >= 0: return 'N' 96 | else: return 'S' 97 | 98 | def add_link(start,end,cont): 99 | pontos = [] 100 | pontos.append(osgeo.ogr.Geometry(osgeo.ogr.wkbLineString)) 101 | lista = list([testando['coordinates_utm'][start:end+1]][0]) 102 | for a in lista: 103 | pontos[-1].AddPoint(a[0],a[1]) 104 | featureIndex = cont 105 | feature = osgeo.ogr.Feature(layer_defn) 106 | feature.SetGeometry(pontos[-1]) 107 | feature.SetFID(featureIndex) 108 | layer.CreateFeature(feature) 109 | for name in range(len(field_names)): 110 | if field_names[name] != 'Edge' and field_names[name] != 'Length': 111 | feature.SetField(name, str(testando[field_names[name]][start_node])) 112 | elif field_names[name] == 'Edge': 113 | feature.SetField(name, cont+1) 114 | elif field_names[name] == 'Length': 115 | feature.SetField(name, distance(start,end)) 116 | layer.SetFeature(feature) 117 | def add_field(field_names,featureIndex): 118 | for name in range(len(field_names)): 119 | if field_names[name] == 'coord_x' or field_names[name] == 'coord_y': 120 | name_aux = 'coordinates_utm' 121 | else: 122 | name_aux = field_names[name] 123 | feature = layer.GetFeature(featureIndex) #lets get the first feature (FID=='0') 124 | if field_names[name] == 'coord_x': 125 | feature.SetField(name,testando_nodes[name_aux][featureIndex][0]) 126 | elif field_names[name] == 'coord_y': 127 | feature.SetField(name,testando_nodes[name_aux][featureIndex][1]) 128 | else: 129 | feature.SetField(name,int(testando_nodes[name_aux][featureIndex])) 130 | layer.SetFeature(feature) #now make the change permanent 131 | def distance(start,end): 132 | dist = 0 133 | for u in range(start,end): 134 | dist = dist + ((testando['coordinates_utm'][u][0] - testando['coordinates_utm'][u+1][0])**2 + (testando['coordinates_utm'][u][1] - testando['coordinates_utm'][u+1][1])**2)**0.5 135 | return dist 136 | def Edgelist(start,end,edge,length): 137 | global testando, columns 138 | datasheet = pd.DataFrame([[testando['coordinates_utm'][start][0],testando['coordinates_utm'][start][1],testando['coordinates_utm'][end][0],testando['coordinates_utm'][end][1],testando['Node'][start],testando['Node'][end],edge,length]],columns = columns, index = None) 139 | return datasheet 140 | 141 | _deg2rad, _rad2deg, _EquatorialRadius, _eccentricitySquared = pi / 180.0, 180.0 / pi, 2, 3 142 | 143 | _ellipsoid = [ 144 | # id, Ellipsoid name, Equatorial Radius, square of eccentricity 145 | # first once is a placeholder only, To allow array indices to match id numbers 146 | [ -1, "Placeholder", 0, 0], 147 | [ 1, "Airy", 6377563, 0.00667054], 148 | [ 2, "Australian National", 6378160, 0.006694542], 149 | [ 3, "Bessel 1841", 6377397, 0.006674372], 150 | [ 4, "Bessel 1841 (Nambia] ", 6377484, 0.006674372], 151 | [ 5, "Clarke 1866", 6378206, 0.006768658], 152 | [ 6, "Clarke 1880", 6378249, 0.006803511], 153 | [ 7, "Everest", 6377276, 0.006637847], 154 | [ 8, "Fischer 1960 (Mercury] ", 6378166, 0.006693422], 155 | [ 9, "Fischer 1968", 6378150, 0.006693422], 156 | [ 10, "GRS 1967", 6378160, 0.006694605], 157 | [ 11, "GRS 1980", 6378137, 0.00669438], 158 | [ 12, "Helmert 1906", 6378200, 0.006693422], 159 | [ 13, "Hough", 6378270, 0.00672267], 160 | [ 14, "International", 6378388, 0.00672267], 161 | [ 15, "Krassovsky", 6378245, 0.006693422], 162 | [ 16, "Modified Airy", 6377340, 0.00667054], 163 | [ 17, "Modified Everest", 6377304, 0.006637847], 164 | [ 18, "Modified Fischer 1960", 6378155, 0.006693422], 165 | [ 19, "South American 1969", 6378160, 0.006694542], 166 | [ 20, "WGS 60", 6378165, 0.006693422], 167 | [ 21, "WGS 66", 6378145, 0.006694542], 168 | [ 22, "WGS-72", 6378135, 0.006694318], 169 | [ 23, "WGS-84", 6378137, 0.00669438] 170 | ] 171 | 172 | 173 | 174 | #dividing each link by their osm_id information (osm_maps) or id information (non-osm_maps) 175 | 176 | sf = shapefile.Reader("C:\YOUR_PATH\input\Shape3.shp") 177 | shapes = sf.shapes() 178 | fields=sf.fields 179 | fields = [field[0] for field in sf.fields[1:]] 180 | start = time.time() 181 | print '\n'+str(time.time() - start)+'s - Reading and arranging the root file information...' 182 | sys.stdout.flush() 183 | #here you can change the utm zone depending on where the map is from 184 | #p = Proj(proj='utm',zone=int(utm_zone[:-1]),ellps='WGS84') 185 | 186 | if 'TYPE' in fields or 'Type' in fields: 187 | try: 188 | fields[fields.index('TYPE')] = 'type' 189 | except: 190 | fields[fields.index('Type')] = 'type' 191 | 192 | heads = [x.upper() if x in ['Group','coordinates','primeiro','coordinates_utm','Node'] else x for x in [sf.shapeRecords()[0].shape.__geo_interface__.keys()[0]]+fields] + ['Group','coordinates','primeiro','coordinates_utm','Node'] 193 | type_ = False 194 | if 'type' in map(lambda x: x.lower(),fields): 195 | heads.remove('type') 196 | heads.insert(3,'type 1') 197 | type_ = True 198 | 199 | dicti = {} 200 | for i in range(len(heads)-1): 201 | dicti[heads[i]] = [] 202 | 203 | testando = pd.DataFrame() 204 | 205 | n = 0 206 | for feature in sf.shapeRecords(): 207 | geom = feature.shape.__geo_interface__ 208 | atr = dict(zip(fields, feature.record)) 209 | if type_: 210 | atr['type 1'] = atr.pop('type') 211 | geom.update(atr) 212 | geom['Group'] = n 213 | j = geom['coordinates'] 214 | for i in range(len(heads)-4): 215 | u = [geom[heads[i]]] 216 | for y in range(len(j)): 217 | dicti[heads[i]].append(u[0]) 218 | dicti['coordinates'] = list(dicti['coordinates']) + list(j) 219 | dicti['primeiro'] = dicti['primeiro'] + [1] + [0]*(len(j) - 2) + [1] 220 | n += 1 221 | if ((n-1) % 800) == 0: 222 | dicti['coordinates_utm'] = None 223 | dicti['Node'] = None 224 | testando = pd.concat([testando,pd.DataFrame.from_dict(dicti)]) 225 | for i in range(len(heads)-1): 226 | dicti[heads[i]] = [] 227 | if ((n-1) % 800) != 0: 228 | dicti['coordinates_utm'] = None 229 | dicti['Node'] = None 230 | testando = pd.concat([testando,pd.DataFrame.from_dict(dicti)],ignore_index=True) 231 | 232 | utm_zone = LLtoUTM(23, testando['coordinates'][0][1], testando['coordinates'][0][0])[0] 233 | if utm_zone[-1] == "S": 234 | p = Proj("+proj=utm +zone="+str(int(utm_zone[:-1]))+", +south +ellps=WGS84") 235 | else: 236 | p = Proj("+proj=utm +zone="+str(int(utm_zone[:-1]))+", +ellps=WGS84") 237 | 238 | testando['coordinates_utm'] = [p(testando['coordinates'][k][0],testando['coordinates'][k][1]) for k in range(len(testando['coordinates']))] 239 | testando['Node'] = None 240 | 241 | #testando = pd.DataFrame.from_dict(dicti) 242 | dicti = None 243 | geom = None 244 | atr = None 245 | #test before creating nodes for OSM maps 246 | a= testando.groupby('coordinates_utm').groups 247 | testando['continuidade'] = 1 248 | testando.ix[testando.primeiro == 1,'continuidade'] = 0; testando 249 | b1 = map(lambda x: (testando['coordinates_utm'][x],[x]),testando.groupby('continuidade').groups[0]) 250 | 251 | b = filter(lambda x: len(x[1]) >= 2, zip(a.keys(),a.values())) 252 | 253 | for h in (b): 254 | for k in range(len(h[1])): 255 | testando.loc[[h[1][k]], 'continuidade'] = 2 256 | b = b + b1 257 | points = sorted(list(testando.groupby('continuidade').groups[0]) + list(testando.groupby('continuidade').groups[2])) 258 | testando.loc[points, 'Node'] = range(1,len(points)+1) 259 | 260 | print '\n'+str(time.time() - start)+'s - Reading and arranging are done!' 261 | sys.stdout.flush() 262 | 263 | #creating nodes shapefile 264 | path='C:\YOUR_PATH\output\Shape3_nodes.shp' 265 | spatialReference = osgeo.osr.SpatialReference() #will create a spatial reference locally to tell the system what the reference will be 266 | spatialReference.ImportFromProj4('+proj=utm +zone='+utm_zone+' ellps=WGS84 +datum=WGS84 +units=m') #here we define this reference to be utm Zone 16N with wgs84 267 | driver = osgeo.ogr.GetDriverByName('ESRI Shapefile') # will select the driver for our shp-file creation. 268 | shapeData = driver.CreateDataSource(path) #so there we will store our data 269 | layer = shapeData.CreateLayer('Nodes', spatialReference, osgeo.ogr.wkbPoint) #this will create a corresponding layer for our data with given spatial information 270 | layer_defn = layer.GetLayerDefn() # gets parameters of the current shapefile 271 | point = osgeo.ogr.Geometry(osgeo.ogr.wkbPoint) 272 | 273 | new_field = ogr.FieldDefn('Node', ogr.OFTInteger) #and a forth field 'oneway' stored as integer 274 | layer.CreateField(new_field) #self explaining 275 | new_field = ogr.FieldDefn('coord_x', ogr.OFTReal) #and a forth field 'oneway' stored as integer 276 | layer.CreateField(new_field) #self explaining 277 | new_field = ogr.FieldDefn('coord_y', ogr.OFTReal) #and a forth field 'oneway' stored as integer 278 | layer.CreateField(new_field) #self explaining 279 | 280 | 281 | field_names = [layer_defn.GetFieldDefn(i).GetName() for i in range(layer_defn.GetFieldCount())] 282 | aux_b, featureIndex = list(zip(*b)[0]), 0 283 | 284 | testando_nodes = testando[testando.continuidade != 1].drop_duplicates(['coordinates_utm']) 285 | testando_nodes.index = range(len(testando_nodes)) 286 | 287 | t, aux_b, a, b, b1, points = None, None, None, None, None, None 288 | 289 | print '\n'+str(time.time() - start)+'s - Adding nodes...' 290 | sys.stdout.flush() 291 | 292 | for featureIndex in range(len(testando_nodes)): 293 | point.AddPoint(testando_nodes['coordinates_utm'][featureIndex][0],testando_nodes['coordinates_utm'][featureIndex][1]) #create a new point at given coordinates 294 | feature = osgeo.ogr.Feature(layer_defn) 295 | feature.SetGeometry(point) 296 | feature.SetFID(featureIndex) 297 | layer.CreateFeature(feature) 298 | add_field(field_names,featureIndex) 299 | shapeData = None 300 | 301 | new_b = None 302 | testando_nodes = None 303 | 304 | print '\n'+str(time.time() - start)+'s - Nodes are done!' 305 | sys.stdout.flush() 306 | 307 | #Cut the raw links and Creat the edge shapefile 308 | columns = ['X_Coord_Start','Y_Coord_Start','X_Coord_End','Y_Coord_End','Start_Node','End_Node','Edge','Length'] 309 | edgelist = pd.DataFrame(columns = columns, index=None) 310 | 311 | 312 | 313 | 314 | #creating edges shapefile 315 | path='C:\YOUR_PATH\output\Shape3_edges.shp' 316 | spatialReference = osgeo.osr.SpatialReference() #will create a spatial reference locally to tell the system what the reference will be 317 | #here you can change the utm zone depending on where the map is from 318 | spatialReference.ImportFromProj4('+proj=utm +zone='+utm_zone+' ellps=WGS84 +datum=WGS84 +units=m') #here we define this reference to be utm Zone 16N with wgs84 319 | driver = osgeo.ogr.GetDriverByName('ESRI Shapefile') # will select the driver for our shp-file creation. 320 | shapeData = driver.CreateDataSource(path) #so there we will store our data 321 | layer = shapeData.CreateLayer('linestrings', spatialReference, osgeo.ogr.wkbLineString) #this will create a corresponding layer for our data with given spatial information. 322 | layer_defn = layer.GetLayerDefn() # gets parameters of the current shapefile 323 | 324 | for f in ['Edge']+heads[:-5]+['Length']: 325 | fieldDefn = ogr.FieldDefn(f, ogr.OFTString) 326 | layer.CreateField(fieldDefn) 327 | field_names = [layer_defn.GetFieldDefn(i).GetName() for i in range(layer_defn.GetFieldCount())] 328 | 329 | print '\n'+str(time.time() - start)+'s - Adding links and creating the Edgelist...' 330 | sys.stdout.flush() 331 | 332 | links = (testando.groupby(['Group']).groups) 333 | links1 = (testando.groupby(['Group','continuidade']).groups) 334 | cont = 0 335 | for link in range(len(links)): 336 | try: 337 | length_0 = len(links1[link,0]) 338 | except: 339 | length_0 = -1 340 | try: 341 | length_1 = len(links1[link,1]) 342 | except: 343 | length_1 = -1 344 | try: 345 | length_2 = len(links1[link,2]) 346 | except: 347 | length_2 = -1 348 | if length_0 == 1 and length_2 == 1: 349 | start_node, end_node = min(links1[link,0][0],links1[link,2][0]), max(links1[link,0][0],links1[link,2][0]) 350 | add_link(start_node,end_node,cont) 351 | edgelist = edgelist.append(Edgelist(start_node,end_node,cont+1,distance(start_node,end_node)),ignore_index=True) 352 | cont += 1 353 | elif length_0 == -1: 354 | for i in range(length_2-1): 355 | start_node, end_node = links1[link,2][i], links1[link,2][i+1] 356 | add_link(start_node,end_node,cont) 357 | edgelist = edgelist.append(Edgelist(start_node,end_node,cont+1,distance(start_node,end_node)),ignore_index=True) 358 | cont += 1 359 | elif length_2 == -1: 360 | start_node, end_node = links1[link,0][0], links1[link,0][1] 361 | add_link(start_node,end_node,cont) 362 | edgelist = edgelist.append(Edgelist(start_node,end_node,cont+1,distance(start_node,end_node)),ignore_index=True) 363 | cont += 1 364 | else: 365 | if length_0 > 1: 366 | start_node, end_node = links1[link,0][0], links1[link,2][0] 367 | add_link(start_node,end_node,cont) 368 | edgelist = edgelist.append(Edgelist(start_node,end_node,cont+1,distance(start_node,end_node)),ignore_index=True) 369 | cont += 1 370 | for i in range(length_2-1): 371 | start_node, end_node = links1[link,2][i], links1[link,2][i+1] 372 | add_link(start_node,end_node,cont) 373 | edgelist = edgelist.append(Edgelist(start_node,end_node,cont+1,distance(start_node,end_node)),ignore_index=True) 374 | cont += 1 375 | start_node, end_node = links1[link,2][-1], links1[link,0][1] 376 | add_link(start_node,end_node,cont) 377 | edgelist = edgelist.append(Edgelist(start_node,end_node,cont+1,distance(start_node,end_node)),ignore_index=True) 378 | cont += 1 379 | else: 380 | if links1[link,0][0] < links1[link,2][0]: 381 | start_node, end_node = links1[link,0][0], links1[link,2][0] 382 | add_link(start_node,end_node,cont) 383 | edgelist = edgelist.append(Edgelist(start_node,end_node,cont+1,distance(start_node,end_node)),ignore_index=True) 384 | cont += 1 385 | for i in range(length_2 - 1): 386 | start_node, end_node = links1[link,2][i], links1[link,2][i+1] 387 | add_link(start_node,end_node,cont) 388 | edgelist = edgelist.append(Edgelist(start_node,end_node,cont+1,distance(start_node,end_node)),ignore_index=True) 389 | cont += 1 390 | else: 391 | for i in range(length_2 - 1): 392 | start_node, end_node = links1[link,2][i], links1[link,2][i+1] 393 | add_link(start_node,end_node,cont) 394 | edgelist = edgelist.append(Edgelist(start_node,end_node,cont+1,distance(start_node,end_node)),ignore_index=True) 395 | cont += 1 396 | start_node, end_node = links1[link,2][-1], links1[link,0][0] 397 | add_link(start_node,end_node,cont) 398 | edgelist = edgelist.append(Edgelist(start_node,end_node,cont+1,distance(start_node,end_node)),ignore_index=True) 399 | cont += 1 400 | shapeData = None 401 | 402 | print '\n'+str(time.time() - start)+'s - Links and Edgelist are done!' 403 | sys.stdout.flush() 404 | 405 | print '\n'+str(time.time() - start)+'s - Saving Edgelist...' 406 | sys.stdout.flush() 407 | 408 | edgelist.to_csv('C:\YOUR_PATH\output\Shape3_edgelist.csv',index=False) 409 | 410 | wb = QgsVectorLayer('C:\YOUR_PATH\output\Shape3_edges.shp', 'Shape3_edges', 'ogr') 411 | QgsMapLayerRegistry.instance().addMapLayer(wb) 412 | wb = QgsVectorLayer('C:\YOUR_PATH\output\Shape3_nodes.shp', 'Shape3_nodes', 'ogr') 413 | QgsMapLayerRegistry.instance().addMapLayer(wb) 414 | print '\n'+str(time.time() - start)+'s - You are all set.' 415 | sys.stdout.flush() 416 | -------------------------------------------------------------------------------- /QGIS/v1.10/GISF2E_qgis_OSM_v1.10.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | ''' 3 | GISF2E QGIS OSM - v1.10 4 | 5 | To cite, please use: Karduni, A., Kermanshah, A., & Derrible, S., 2016 “A protocol to convert spatial polyline data to network formats and applications to world urban road networks”, Scientific Data, 3:160046 6 | The article is available at: http://www.nature.com/articles/sdata201646 7 | 8 | Main authors: Felipe Macena, Rodrigo Marinho 9 | ''' 10 | 11 | #import all libraries required 12 | import shapefile 13 | import pandas as pd 14 | from pyproj import Proj 15 | import osgeo.ogr, osgeo.osr 16 | from osgeo import ogr 17 | import sys 18 | import os 19 | import time 20 | from math import pi, sin, cos, tan, sqrt 21 | from qgis.core import * 22 | def LLtoUTM(ReferenceEllipsoid, Lat, Long, zone = None): 23 | """converts lat/long to UTM coords. Equations from USGS Bulletin 1532 24 | East Longitudes are positive, West longitudes are negative. 25 | North latitudes are positive, South latitudes are negative 26 | Lat and Long are in decimal degrees 27 | Written by Chuck Gantz- chuck.gantz@globalstar.com""" 28 | 29 | a = _ellipsoid[ReferenceEllipsoid][_EquatorialRadius] 30 | eccSquared = _ellipsoid[ReferenceEllipsoid][_eccentricitySquared] 31 | k0 = 0.9996 32 | 33 | #Make sure the longitude is between -180.00 .. 179.9 34 | LongTemp = (Long+180)-int((Long+180)/360)*360-180 # -180.00 .. 179.9 35 | 36 | LatRad = Lat*_deg2rad 37 | LongRad = LongTemp*_deg2rad 38 | 39 | if zone is None: 40 | ZoneNumber = int((LongTemp + 180)/6) + 1 41 | else: 42 | ZoneNumber = zone 43 | 44 | if Lat >= 56.0 and Lat < 64.0 and LongTemp >= 3.0 and LongTemp < 12.0: 45 | ZoneNumber = 32 46 | 47 | # Special zones for Svalbard 48 | if Lat >= 72.0 and Lat < 84.0: 49 | if LongTemp >= 0.0 and LongTemp < 9.0:ZoneNumber = 31 50 | elif LongTemp >= 9.0 and LongTemp < 21.0: ZoneNumber = 33 51 | elif LongTemp >= 21.0 and LongTemp < 33.0: ZoneNumber = 35 52 | elif LongTemp >= 33.0 and LongTemp < 42.0: ZoneNumber = 37 53 | 54 | LongOrigin = (ZoneNumber - 1)*6 - 180 + 3 #+3 puts origin in middle of zone 55 | LongOriginRad = LongOrigin * _deg2rad 56 | 57 | #compute the UTM Zone from the latitude and longitude 58 | UTMZone = "%d%c" % (ZoneNumber, _UTMLetterDesignator(Lat)) 59 | 60 | eccPrimeSquared = (eccSquared)/(1-eccSquared) 61 | N = a/sqrt(1-eccSquared*sin(LatRad)*sin(LatRad)) 62 | T = tan(LatRad)*tan(LatRad) 63 | C = eccPrimeSquared*cos(LatRad)*cos(LatRad) 64 | A = cos(LatRad)*(LongRad-LongOriginRad) 65 | 66 | M = a*((1 67 | - eccSquared/4 68 | - 3*eccSquared*eccSquared/64 69 | - 5*eccSquared*eccSquared*eccSquared/256)*LatRad 70 | - (3*eccSquared/8 71 | + 3*eccSquared*eccSquared/32 72 | + 45*eccSquared*eccSquared*eccSquared/1024)*sin(2*LatRad) 73 | + (15*eccSquared*eccSquared/256 + 45*eccSquared*eccSquared*eccSquared/1024)*sin(4*LatRad) 74 | - (35*eccSquared*eccSquared*eccSquared/3072)*sin(6*LatRad)) 75 | 76 | UTMEasting = (k0*N*(A+(1-T+C)*A*A*A/6 77 | + (5-18*T+T*T+72*C-58*eccPrimeSquared)*A*A*A*A*A/120) 78 | + 500000.0) 79 | 80 | UTMNorthing = (k0*(M+N*tan(LatRad)*(A*A/2+(5-T+9*C+4*C*C)*A*A*A*A/24 81 | + (61 82 | -58*T 83 | +T*T 84 | +600*C 85 | -330*eccPrimeSquared)*A*A*A*A*A*A/720))) 86 | if Lat < 0: 87 | UTMNorthing = UTMNorthing + 10000000.0; #10000000 meter offset for southern hemisphere 88 | return (UTMZone, UTMEasting, UTMNorthing) 89 | 90 | 91 | def _UTMLetterDesignator(Lat): 92 | """This routine determines the correct UTM letter designator for the given 93 | latitude returns 'Z' if latitude is outside the UTM limits of 84N to 80S 94 | Written by Chuck Gantz- chuck.gantz@globalstar.com""" 95 | if Lat >= 0: return 'N' 96 | else: return 'S' 97 | 98 | def add_link(start,end,cont): 99 | pontos = [] 100 | pontos.append(osgeo.ogr.Geometry(osgeo.ogr.wkbLineString)) 101 | lista = list([testando['coordinates_utm'][start:end+1]][0]) 102 | for a in lista: 103 | pontos[-1].AddPoint(a[0],a[1]) 104 | featureIndex = cont 105 | feature = osgeo.ogr.Feature(layer_defn) 106 | feature.SetGeometry(pontos[-1]) 107 | feature.SetFID(featureIndex) 108 | layer.CreateFeature(feature) 109 | for name in range(len(field_names)): 110 | if field_names[name] != 'Edge' and field_names[name] != 'Length': 111 | feature.SetField(name, str(testando[field_names[name]][start_node])) 112 | elif field_names[name] == 'Edge': 113 | feature.SetField(name, cont+1) 114 | elif field_names[name] == 'Length': 115 | feature.SetField(name, distance(start,end)) 116 | layer.SetFeature(feature) 117 | def add_field(field_names,featureIndex): 118 | for name in range(len(field_names)): 119 | if field_names[name] == 'coord_x' or field_names[name] == 'coord_y': 120 | name_aux = 'coordinates_utm' 121 | else: 122 | name_aux = field_names[name] 123 | feature = layer.GetFeature(featureIndex) #lets get the first feature (FID=='0') 124 | if field_names[name] == 'coord_x': 125 | feature.SetField(name,testando_nodes[name_aux][featureIndex][0]) 126 | elif field_names[name] == 'coord_y': 127 | feature.SetField(name,testando_nodes[name_aux][featureIndex][1]) 128 | else: 129 | feature.SetField(name,int(testando_nodes[name_aux][featureIndex])) 130 | layer.SetFeature(feature) #now make the change permanent 131 | def distance(start,end): 132 | dist = 0 133 | for u in range(start,end): 134 | dist = dist + ((testando['coordinates_utm'][u][0] - testando['coordinates_utm'][u+1][0])**2 + (testando['coordinates_utm'][u][1] - testando['coordinates_utm'][u+1][1])**2)**0.5 135 | return dist 136 | def Edgelist(start,end,edge,length): 137 | global testando, columns 138 | datasheet = pd.DataFrame([[testando['coordinates_utm'][start][0],testando['coordinates_utm'][start][1],testando['coordinates_utm'][end][0],testando['coordinates_utm'][end][1],testando['Node'][start],testando['Node'][end],edge,length]],columns = columns, index = None) 139 | return datasheet 140 | 141 | _deg2rad, _rad2deg, _EquatorialRadius, _eccentricitySquared = pi / 180.0, 180.0 / pi, 2, 3 142 | 143 | _ellipsoid = [ 144 | # id, Ellipsoid name, Equatorial Radius, square of eccentricity 145 | # first once is a placeholder only, To allow array indices to match id numbers 146 | [ -1, "Placeholder", 0, 0], 147 | [ 1, "Airy", 6377563, 0.00667054], 148 | [ 2, "Australian National", 6378160, 0.006694542], 149 | [ 3, "Bessel 1841", 6377397, 0.006674372], 150 | [ 4, "Bessel 1841 (Nambia] ", 6377484, 0.006674372], 151 | [ 5, "Clarke 1866", 6378206, 0.006768658], 152 | [ 6, "Clarke 1880", 6378249, 0.006803511], 153 | [ 7, "Everest", 6377276, 0.006637847], 154 | [ 8, "Fischer 1960 (Mercury] ", 6378166, 0.006693422], 155 | [ 9, "Fischer 1968", 6378150, 0.006693422], 156 | [ 10, "GRS 1967", 6378160, 0.006694605], 157 | [ 11, "GRS 1980", 6378137, 0.00669438], 158 | [ 12, "Helmert 1906", 6378200, 0.006693422], 159 | [ 13, "Hough", 6378270, 0.00672267], 160 | [ 14, "International", 6378388, 0.00672267], 161 | [ 15, "Krassovsky", 6378245, 0.006693422], 162 | [ 16, "Modified Airy", 6377340, 0.00667054], 163 | [ 17, "Modified Everest", 6377304, 0.006637847], 164 | [ 18, "Modified Fischer 1960", 6378155, 0.006693422], 165 | [ 19, "South American 1969", 6378160, 0.006694542], 166 | [ 20, "WGS 60", 6378165, 0.006693422], 167 | [ 21, "WGS 66", 6378145, 0.006694542], 168 | [ 22, "WGS-72", 6378135, 0.006694318], 169 | [ 23, "WGS-84", 6378137, 0.00669438] 170 | ] 171 | 172 | 173 | 174 | #dividing each link by their osm_id information (osm_maps) or id information (non-osm_maps) 175 | 176 | sf = shapefile.Reader("C:\YOUR_PATH\input\Chicago_cut.shp") 177 | shapes = sf.shapes() 178 | fields=sf.fields 179 | fields = [field[0] for field in sf.fields[1:]] 180 | start = time.time() 181 | print '\n'+str(time.time() - start)+'s - Reading and arranging the root file information...' 182 | sys.stdout.flush() 183 | #here you can change the utm zone depending on where the map is from 184 | #p = Proj(proj='utm',zone=int(utm_zone[:-1]),ellps='WGS84') 185 | 186 | if 'TYPE' in fields or 'Type' in fields: 187 | try: 188 | fields[fields.index('TYPE')] = 'type' 189 | except: 190 | fields[fields.index('Type')] = 'type' 191 | 192 | heads = [x.upper() if x in ['Group','coordinates','primeiro','coordinates_utm','Node'] else x for x in [sf.shapeRecords()[0].shape.__geo_interface__.keys()[0]]+fields] + ['Group','coordinates','primeiro','coordinates_utm','Node'] 193 | type_ = False 194 | if 'type' in map(lambda x: x.lower(),fields): 195 | heads.remove('type') 196 | heads.insert(3,'type 1') 197 | type_ = True 198 | 199 | dicti = {} 200 | for i in range(len(heads)-1): 201 | dicti[heads[i]] = [] 202 | 203 | testando = pd.DataFrame() 204 | 205 | n = 0 206 | for feature in sf.shapeRecords(): 207 | geom = feature.shape.__geo_interface__ 208 | atr = dict(zip(fields, feature.record)) 209 | if type_: 210 | atr['type 1'] = atr.pop('type') 211 | geom.update(atr) 212 | geom['Group'] = n 213 | j = geom['coordinates'] 214 | for i in range(len(heads)-4): 215 | u = [geom[heads[i]]] 216 | for y in range(len(j)): 217 | dicti[heads[i]].append(u[0]) 218 | dicti['coordinates'] = list(dicti['coordinates']) + list(j) 219 | dicti['primeiro'] = dicti['primeiro'] + [1] + [0]*(len(j) - 2) + [1] 220 | n += 1 221 | if ((n-1) % 800) == 0: 222 | dicti['coordinates_utm'] = None 223 | dicti['Node'] = None 224 | testando = pd.concat([testando,pd.DataFrame.from_dict(dicti)]) 225 | for i in range(len(heads)-1): 226 | dicti[heads[i]] = [] 227 | if ((n-1) % 800) != 0: 228 | dicti['coordinates_utm'] = None 229 | dicti['Node'] = None 230 | testando = pd.concat([testando,pd.DataFrame.from_dict(dicti)],ignore_index=True) 231 | 232 | utm_zone = LLtoUTM(23, testando['coordinates'][0][1], testando['coordinates'][0][0])[0] 233 | p = Proj(proj='utm',zone=int(utm_zone[:-1]),ellps='WGS84') 234 | 235 | testando['coordinates_utm'] = [p(testando['coordinates'][k][0],testando['coordinates'][k][1]) for k in range(len(testando['coordinates']))] 236 | testando['Node'] = None 237 | 238 | #testando = pd.DataFrame.from_dict(dicti) 239 | dicti = None 240 | geom = None 241 | atr = None 242 | #test before creating nodes for OSM maps 243 | a= testando.groupby('coordinates_utm').groups 244 | testando['continuidade'] = 1 245 | testando.ix[testando.primeiro == 1,'continuidade'] = 0; testando 246 | b1 = map(lambda x: (testando['coordinates_utm'][x],[x]),testando.groupby('continuidade').groups[0]) 247 | 248 | b = filter(lambda x: (testando['bridge'][x[1][0]] == testando['bridge'][int(x[1][1])]) or (testando['tunnel'][x[1][0]] == testando['tunnel'][int(x[1][1])]), filter(lambda x: len(x[1]) >= 2, zip(a.keys(),a.values()))) 249 | 250 | for h in (b): 251 | for k in range(len(h[1])): 252 | testando.loc[[h[1][k]], 'continuidade'] = 2 253 | b = b + b1 254 | points = sorted(testando.groupby('continuidade').groups[0] + testando.groupby('continuidade').groups[2]) 255 | testando[points, 'Node'] = range(1,len(points)+1) 256 | 257 | print '\n'+str(time.time() - start)+'s - Reading and arranging are done!' 258 | sys.stdout.flush() 259 | 260 | #creating nodes shapefile 261 | path='C:\YOUR_PATH\output\Chicago_cut_nodes.shp' 262 | spatialReference = osgeo.osr.SpatialReference() #will create a spatial reference locally to tell the system what the reference will be 263 | spatialReference.ImportFromProj4('+proj=utm +zone='+utm_zone+' ellps=WGS84 +datum=WGS84 +units=m') #here we define this reference to be utm Zone 16N with wgs84 264 | driver = osgeo.ogr.GetDriverByName('ESRI Shapefile') # will select the driver for our shp-file creation. 265 | shapeData = driver.CreateDataSource(path) #so there we will store our data 266 | layer = shapeData.CreateLayer('Nodes', spatialReference, osgeo.ogr.wkbPoint) #this will create a corresponding layer for our data with given spatial information 267 | layer_defn = layer.GetLayerDefn() # gets parameters of the current shapefile 268 | point = osgeo.ogr.Geometry(osgeo.ogr.wkbPoint) 269 | 270 | new_field = ogr.FieldDefn('Node', ogr.OFTInteger) #and a forth field 'oneway' stored as integer 271 | layer.CreateField(new_field) #self explaining 272 | new_field = ogr.FieldDefn('coord_x', ogr.OFTReal) #and a forth field 'oneway' stored as integer 273 | layer.CreateField(new_field) #self explaining 274 | new_field = ogr.FieldDefn('coord_y', ogr.OFTReal) #and a forth field 'oneway' stored as integer 275 | layer.CreateField(new_field) #self explaining 276 | 277 | new_field = ogr.FieldDefn('bridge', ogr.OFTInteger) #we will create a new field called bridge as integer 278 | layer.CreateField(new_field) #self explaining 279 | new_field = ogr.FieldDefn('tunnel', ogr.OFTInteger) #and a sixth field 'tunnel' stored as integer 280 | layer.CreateField(new_field) #self explaining 281 | field_names = [layer_defn.GetFieldDefn(i).GetName() for i in range(layer_defn.GetFieldCount())] 282 | aux_b, featureIndex = list(zip(*b)[0]), 0 283 | 284 | testando_nodes = testando[testando.continuidade != 1].drop_duplicates(['coordinates_utm']) 285 | testando_nodes.index = range(len(testando_nodes)) 286 | 287 | t, aux_b, a, b, b1, points = None, None, None, None, None, None 288 | 289 | print '\n'+str(time.time() - start)+'s - Adding nodes...' 290 | sys.stdout.flush() 291 | 292 | for featureIndex in range(len(testando_nodes)): 293 | point.AddPoint(testando_nodes['coordinates_utm'][featureIndex][0],testando_nodes['coordinates_utm'][featureIndex][1]) #create a new point at given coordinates 294 | feature = osgeo.ogr.Feature(layer_defn) 295 | feature.SetGeometry(point) 296 | feature.SetFID(featureIndex) 297 | layer.CreateFeature(feature) 298 | add_field(field_names,featureIndex) 299 | shapeData = None 300 | 301 | new_b = None 302 | testando_nodes = None 303 | 304 | print '\n'+str(time.time() - start)+'s - Nodes are done!' 305 | sys.stdout.flush() 306 | 307 | #Cut the raw links and Creat the edge shapefile 308 | columns = ['X_Coord_Start','Y_Coord_Start','X_Coord_End','Y_Coord_End','Start_Node','End_Node','Edge','Length'] 309 | edgelist = pd.DataFrame(columns = columns, index=None) 310 | 311 | #creating edges shapefile 312 | path='C:\YOUR_PATH\output\Chicago_cut_edges.shp' 313 | spatialReference = osgeo.osr.SpatialReference() #will create a spatial reference locally to tell the system what the reference will be 314 | #here you can change the utm zone depending on where the map is from 315 | spatialReference.ImportFromProj4('+proj=utm +zone='+utm_zone+' ellps=WGS84 +datum=WGS84 +units=m') #here we define this reference to be utm Zone 16N with wgs84 316 | driver = osgeo.ogr.GetDriverByName('ESRI Shapefile') # will select the driver for our shp-file creation. 317 | shapeData = driver.CreateDataSource(path) #so there we will store our data 318 | layer = shapeData.CreateLayer('linestrings', spatialReference, osgeo.ogr.wkbLineString) #this will create a corresponding layer for our data with given spatial information. 319 | layer_defn = layer.GetLayerDefn() # gets parameters of the current shapefile 320 | 321 | for f in ['Edge']+heads[:-5]+['Length']: 322 | fieldDefn = ogr.FieldDefn(f, ogr.OFTString) 323 | layer.CreateField(fieldDefn) 324 | field_names = [layer_defn.GetFieldDefn(i).GetName() for i in range(layer_defn.GetFieldCount())] 325 | 326 | print '\n'+str(time.time() - start)+'s - Adding links and creating the Edgelist...' 327 | sys.stdout.flush() 328 | 329 | links = (testando.groupby(['Group']).groups) 330 | links1 = (testando.groupby(['Group','continuidade']).groups) 331 | cont = 0 332 | for link in range(len(links)): 333 | try: 334 | length_0 = len(links1[link,0]) 335 | except: 336 | length_0 = -1 337 | try: 338 | length_1 = len(links1[link,1]) 339 | except: 340 | length_1 = -1 341 | try: 342 | length_2 = len(links1[link,2]) 343 | except: 344 | length_2 = -1 345 | if length_0 == 1 and length_2 == 1: 346 | start_node, end_node = min(links1[link,0][0],links1[link,2][0]), max(links1[link,0][0],links1[link,2][0]) 347 | add_link(start_node,end_node,cont) 348 | edgelist = edgelist.append(Edgelist(start_node,end_node,cont+1,distance(start_node,end_node)),ignore_index=True) 349 | cont += 1 350 | elif length_0 == -1: 351 | for i in range(length_2-1): 352 | start_node, end_node = links1[link,2][i], links1[link,2][i+1] 353 | add_link(start_node,end_node,cont) 354 | edgelist = edgelist.append(Edgelist(start_node,end_node,cont+1,distance(start_node,end_node)),ignore_index=True) 355 | cont += 1 356 | elif length_2 == -1: 357 | start_node, end_node = links1[link,0][0], links1[link,0][1] 358 | add_link(start_node,end_node,cont) 359 | edgelist = edgelist.append(Edgelist(start_node,end_node,cont+1,distance(start_node,end_node)),ignore_index=True) 360 | cont += 1 361 | else: 362 | if length_0 > 1: 363 | start_node, end_node = links1[link,0][0], links1[link,2][0] 364 | add_link(start_node,end_node,cont) 365 | edgelist = edgelist.append(Edgelist(start_node,end_node,cont+1,distance(start_node,end_node)),ignore_index=True) 366 | cont += 1 367 | for i in range(length_2-1): 368 | start_node, end_node = links1[link,2][i], links1[link,2][i+1] 369 | add_link(start_node,end_node,cont) 370 | edgelist = edgelist.append(Edgelist(start_node,end_node,cont+1,distance(start_node,end_node)),ignore_index=True) 371 | cont += 1 372 | start_node, end_node = links1[link,2][-1], links1[link,0][1] 373 | add_link(start_node,end_node,cont) 374 | edgelist = edgelist.append(Edgelist(start_node,end_node,cont+1,distance(start_node,end_node)),ignore_index=True) 375 | cont += 1 376 | else: 377 | if links1[link,0][0] < links1[link,2][0]: 378 | start_node, end_node = links1[link,0][0], links1[link,2][0] 379 | add_link(start_node,end_node,cont) 380 | edgelist = edgelist.append(Edgelist(start_node,end_node,cont+1,distance(start_node,end_node)),ignore_index=True) 381 | cont += 1 382 | for i in range(length_2 - 1): 383 | start_node, end_node = links1[link,2][i], links1[link,2][i+1] 384 | add_link(start_node,end_node,cont) 385 | edgelist = edgelist.append(Edgelist(start_node,end_node,cont+1,distance(start_node,end_node)),ignore_index=True) 386 | cont += 1 387 | else: 388 | for i in range(length_2 - 1): 389 | start_node, end_node = links1[link,2][i], links1[link,2][i+1] 390 | add_link(start_node,end_node,cont) 391 | edgelist = edgelist.append(Edgelist(start_node,end_node,cont+1,distance(start_node,end_node)),ignore_index=True) 392 | cont += 1 393 | start_node, end_node = links1[link,2][-1], links1[link,0][0] 394 | add_link(start_node,end_node,cont) 395 | edgelist = edgelist.append(Edgelist(start_node,end_node,cont+1,distance(start_node,end_node)),ignore_index=True) 396 | cont += 1 397 | shapeData = None 398 | 399 | print '\n'+str(time.time() - start)+'s - Links and Edgelist are done!' 400 | sys.stdout.flush() 401 | 402 | print '\n'+str(time.time() - start)+'s - Saving Edgelist...' 403 | sys.stdout.flush() 404 | 405 | edgelist.to_csv('C:\YOUR_PATH\output\Chicago_cut_edgelist.csv',index=False) 406 | 407 | wb = QgsVectorLayer('C:\YOUR_PATH\output\Chicago_cut_edges.shp', 'Chicago_cut_edges', 'ogr') 408 | QgsMapLayerRegistry.instance().addMapLayer(wb) 409 | wb = QgsVectorLayer('C:\YOUR_PATH\output\Chicago_cut_nodes.shp', 'Chicago_cut_nodes', 'ogr') 410 | QgsMapLayerRegistry.instance().addMapLayer(wb) 411 | print '\n'+str(time.time() - start)+'s - You are all set.' 412 | sys.stdout.flush() 413 | -------------------------------------------------------------------------------- /QGIS/v1.00/GISF2E_qgis_non-OSM_v1.00.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | ''' 3 | GISF2E QGIS non-OSM - v1.00 - Note that the code was developed with pandas version 0.14.1, see version 1.10 that works with all pandas versions. 4 | 5 | To cite, please use: Karduni, A., Kermanshah, A., & Derrible, S., 2016 “A protocol to convert spatial polyline data to network formats and applications to world urban road networks”, Scientific Data, 3:160046 6 | The article is available at: http://www.nature.com/articles/sdata201646 7 | 8 | Main authors: Felipe Macena, Rodrigo Marinho 9 | ''' 10 | 11 | #import all libraries required 12 | import shapefile 13 | import pandas as pd 14 | from pyproj import Proj 15 | import osgeo.ogr, osgeo.osr 16 | from osgeo import ogr 17 | import sys 18 | import os 19 | import time 20 | from math import pi, sin, cos, tan, sqrt 21 | from qgis.core import * 22 | def LLtoUTM(ReferenceEllipsoid, Lat, Long, zone = None): 23 | """converts lat/long to UTM coords. Equations from USGS Bulletin 1532 24 | East Longitudes are positive, West longitudes are negative. 25 | North latitudes are positive, South latitudes are negative 26 | Lat and Long are in decimal degrees 27 | Written by Chuck Gantz- chuck.gantz@globalstar.com""" 28 | 29 | a = _ellipsoid[ReferenceEllipsoid][_EquatorialRadius] 30 | eccSquared = _ellipsoid[ReferenceEllipsoid][_eccentricitySquared] 31 | k0 = 0.9996 32 | 33 | #Make sure the longitude is between -180.00 .. 179.9 34 | LongTemp = (Long+180)-int((Long+180)/360)*360-180 # -180.00 .. 179.9 35 | 36 | LatRad = Lat*_deg2rad 37 | LongRad = LongTemp*_deg2rad 38 | 39 | if zone is None: 40 | ZoneNumber = int((LongTemp + 180)/6) + 1 41 | else: 42 | ZoneNumber = zone 43 | 44 | if Lat >= 56.0 and Lat < 64.0 and LongTemp >= 3.0 and LongTemp < 12.0: 45 | ZoneNumber = 32 46 | 47 | # Special zones for Svalbard 48 | if Lat >= 72.0 and Lat < 84.0: 49 | if LongTemp >= 0.0 and LongTemp < 9.0:ZoneNumber = 31 50 | elif LongTemp >= 9.0 and LongTemp < 21.0: ZoneNumber = 33 51 | elif LongTemp >= 21.0 and LongTemp < 33.0: ZoneNumber = 35 52 | elif LongTemp >= 33.0 and LongTemp < 42.0: ZoneNumber = 37 53 | 54 | LongOrigin = (ZoneNumber - 1)*6 - 180 + 3 #+3 puts origin in middle of zone 55 | LongOriginRad = LongOrigin * _deg2rad 56 | 57 | #compute the UTM Zone from the latitude and longitude 58 | UTMZone = "%d%c" % (ZoneNumber, _UTMLetterDesignator(Lat)) 59 | 60 | eccPrimeSquared = (eccSquared)/(1-eccSquared) 61 | N = a/sqrt(1-eccSquared*sin(LatRad)*sin(LatRad)) 62 | T = tan(LatRad)*tan(LatRad) 63 | C = eccPrimeSquared*cos(LatRad)*cos(LatRad) 64 | A = cos(LatRad)*(LongRad-LongOriginRad) 65 | 66 | M = a*((1 67 | - eccSquared/4 68 | - 3*eccSquared*eccSquared/64 69 | - 5*eccSquared*eccSquared*eccSquared/256)*LatRad 70 | - (3*eccSquared/8 71 | + 3*eccSquared*eccSquared/32 72 | + 45*eccSquared*eccSquared*eccSquared/1024)*sin(2*LatRad) 73 | + (15*eccSquared*eccSquared/256 + 45*eccSquared*eccSquared*eccSquared/1024)*sin(4*LatRad) 74 | - (35*eccSquared*eccSquared*eccSquared/3072)*sin(6*LatRad)) 75 | 76 | UTMEasting = (k0*N*(A+(1-T+C)*A*A*A/6 77 | + (5-18*T+T*T+72*C-58*eccPrimeSquared)*A*A*A*A*A/120) 78 | + 500000.0) 79 | 80 | UTMNorthing = (k0*(M+N*tan(LatRad)*(A*A/2+(5-T+9*C+4*C*C)*A*A*A*A/24 81 | + (61 82 | -58*T 83 | +T*T 84 | +600*C 85 | -330*eccPrimeSquared)*A*A*A*A*A*A/720))) 86 | if Lat < 0: 87 | UTMNorthing = UTMNorthing + 10000000.0; #10000000 meter offset for southern hemisphere 88 | return (UTMZone, UTMEasting, UTMNorthing) 89 | 90 | 91 | def _UTMLetterDesignator(Lat): 92 | """This routine determines the correct UTM letter designator for the given 93 | latitude returns 'Z' if latitude is outside the UTM limits of 84N to 80S 94 | Written by Chuck Gantz- chuck.gantz@globalstar.com""" 95 | if Lat >= 0: return 'N' 96 | else: return 'S' 97 | 98 | def add_link(start,end,cont): 99 | pontos = [] 100 | pontos.append(osgeo.ogr.Geometry(osgeo.ogr.wkbLineString)) 101 | lista = list([testando['coordinates_utm'][start:end+1]][0]) 102 | for a in lista: 103 | pontos[-1].AddPoint(a[0],a[1]) 104 | featureIndex = cont 105 | feature = osgeo.ogr.Feature(layer_defn) 106 | feature.SetGeometry(pontos[-1]) 107 | feature.SetFID(featureIndex) 108 | layer.CreateFeature(feature) 109 | for name in range(len(field_names)): 110 | if field_names[name] != 'Edge' and field_names[name] != 'Length': 111 | feature.SetField(name, str(testando[field_names[name]][start_node])) 112 | elif field_names[name] == 'Edge': 113 | feature.SetField(name, cont+1) 114 | elif field_names[name] == 'Length': 115 | feature.SetField(name, distance(start,end)) 116 | layer.SetFeature(feature) 117 | def add_field(field_names,featureIndex): 118 | for name in range(len(field_names)): 119 | if field_names[name] == 'coord_x' or field_names[name] == 'coord_y': 120 | name_aux = 'coordinates_utm' 121 | else: 122 | name_aux = field_names[name] 123 | feature = layer.GetFeature(featureIndex) #lets get the first feature (FID=='0') 124 | if field_names[name] == 'coord_x': 125 | feature.SetField(name,testando_nodes[name_aux][featureIndex][0]) 126 | elif field_names[name] == 'coord_y': 127 | feature.SetField(name,testando_nodes[name_aux][featureIndex][1]) 128 | else: 129 | feature.SetField(name,int(testando_nodes[name_aux][featureIndex])) 130 | layer.SetFeature(feature) #now make the change permanent 131 | def distance(start,end): 132 | dist = 0 133 | for u in range(start,end): 134 | dist = dist + ((testando['coordinates_utm'][u][0] - testando['coordinates_utm'][u+1][0])**2 + (testando['coordinates_utm'][u][1] - testando['coordinates_utm'][u+1][1])**2)**0.5 135 | return dist 136 | def Edgelist(start,end,edge,length): 137 | global testando, columns 138 | datasheet = pd.DataFrame([[testando['coordinates_utm'][start][0],testando['coordinates_utm'][start][1],testando['coordinates_utm'][end][0],testando['coordinates_utm'][end][1],testando['Node'][start],testando['Node'][end],edge,length]],columns = columns, index = None) 139 | return datasheet 140 | 141 | _deg2rad, _rad2deg, _EquatorialRadius, _eccentricitySquared = pi / 180.0, 180.0 / pi, 2, 3 142 | 143 | _ellipsoid = [ 144 | # id, Ellipsoid name, Equatorial Radius, square of eccentricity 145 | # first once is a placeholder only, To allow array indices to match id numbers 146 | [ -1, "Placeholder", 0, 0], 147 | [ 1, "Airy", 6377563, 0.00667054], 148 | [ 2, "Australian National", 6378160, 0.006694542], 149 | [ 3, "Bessel 1841", 6377397, 0.006674372], 150 | [ 4, "Bessel 1841 (Nambia] ", 6377484, 0.006674372], 151 | [ 5, "Clarke 1866", 6378206, 0.006768658], 152 | [ 6, "Clarke 1880", 6378249, 0.006803511], 153 | [ 7, "Everest", 6377276, 0.006637847], 154 | [ 8, "Fischer 1960 (Mercury] ", 6378166, 0.006693422], 155 | [ 9, "Fischer 1968", 6378150, 0.006693422], 156 | [ 10, "GRS 1967", 6378160, 0.006694605], 157 | [ 11, "GRS 1980", 6378137, 0.00669438], 158 | [ 12, "Helmert 1906", 6378200, 0.006693422], 159 | [ 13, "Hough", 6378270, 0.00672267], 160 | [ 14, "International", 6378388, 0.00672267], 161 | [ 15, "Krassovsky", 6378245, 0.006693422], 162 | [ 16, "Modified Airy", 6377340, 0.00667054], 163 | [ 17, "Modified Everest", 6377304, 0.006637847], 164 | [ 18, "Modified Fischer 1960", 6378155, 0.006693422], 165 | [ 19, "South American 1969", 6378160, 0.006694542], 166 | [ 20, "WGS 60", 6378165, 0.006693422], 167 | [ 21, "WGS 66", 6378145, 0.006694542], 168 | [ 22, "WGS-72", 6378135, 0.006694318], 169 | [ 23, "WGS-84", 6378137, 0.00669438] 170 | ] 171 | 172 | 173 | 174 | #dividing each link by their osm_id information (osm_maps) or id information (non-osm_maps) 175 | 176 | sf = shapefile.Reader("C:\Users\Rodrigo\Desktop\GIStool\input\Shape3.shp") 177 | shapes = sf.shapes() 178 | fields=sf.fields 179 | fields = [field[0] for field in sf.fields[1:]] 180 | start = time.time() 181 | print '\n'+str(time.time() - start)+'s - Reading and arranging the root file information...' 182 | sys.stdout.flush() 183 | #here you can change the utm zone depending on where the map is from 184 | #p = Proj(proj='utm',zone=int(utm_zone[:-1]),ellps='WGS84') 185 | 186 | if 'TYPE' in fields or 'Type' in fields: 187 | try: 188 | fields[fields.index('TYPE')] = 'type' 189 | except: 190 | fields[fields.index('Type')] = 'type' 191 | 192 | heads = [x.upper() if x in ['Group','coordinates','primeiro','coordinates_utm','Node'] else x for x in [sf.shapeRecords()[0].shape.__geo_interface__.keys()[0]]+fields] + ['Group','coordinates','primeiro','coordinates_utm','Node'] 193 | type_ = False 194 | if 'type' in map(lambda x: x.lower(),fields): 195 | heads.remove('type') 196 | heads.insert(3,'type 1') 197 | type_ = True 198 | 199 | dicti = {} 200 | for i in range(len(heads)-1): 201 | dicti[heads[i]] = [] 202 | 203 | testando = pd.DataFrame() 204 | 205 | n = 0 206 | for feature in sf.shapeRecords(): 207 | geom = feature.shape.__geo_interface__ 208 | atr = dict(zip(fields, feature.record)) 209 | if type_: 210 | atr['type 1'] = atr.pop('type') 211 | geom.update(atr) 212 | geom['Group'] = n 213 | j = geom['coordinates'] 214 | for i in range(len(heads)-4): 215 | u = [geom[heads[i]]] 216 | for y in range(len(j)): 217 | dicti[heads[i]].append(u[0]) 218 | dicti['coordinates'] = list(dicti['coordinates']) + list(j) 219 | dicti['primeiro'] = dicti['primeiro'] + [1] + [0]*(len(j) - 2) + [1] 220 | n += 1 221 | if ((n-1) % 800) == 0: 222 | dicti['coordinates_utm'] = None 223 | dicti['Node'] = None 224 | testando = pd.concat([testando,pd.DataFrame.from_dict(dicti)]) 225 | for i in range(len(heads)-1): 226 | dicti[heads[i]] = [] 227 | if ((n-1) % 800) != 0: 228 | dicti['coordinates_utm'] = None 229 | dicti['Node'] = None 230 | testando = pd.concat([testando,pd.DataFrame.from_dict(dicti)],ignore_index=True) 231 | 232 | utm_zone = LLtoUTM(23, testando['coordinates'][0][1], testando['coordinates'][0][0])[0] 233 | p = Proj(proj='utm',zone=int(utm_zone[:-1]),ellps='WGS84') 234 | 235 | testando['coordinates_utm'] = [p(testando['coordinates'][k][0],testando['coordinates'][k][1]) for k in range(len(testando['coordinates']))] 236 | testando['Node'] = None 237 | 238 | #testando = pd.DataFrame.from_dict(dicti) 239 | dicti = None 240 | geom = None 241 | atr = None 242 | #test before creating nodes for OSM maps 243 | a= testando.groupby('coordinates_utm').groups 244 | testando['continuidade'] = 1 245 | testando.ix[testando.primeiro == 1,'continuidade'] = 0; testando 246 | b1 = map(lambda x: (testando['coordinates_utm'][x],[x]),testando.groupby('continuidade').groups[0]) 247 | 248 | b = filter(lambda x: len(x[1]) >= 2, zip(a.keys(),a.values())) 249 | 250 | for h in (b): 251 | for k in range(len(h[1])): 252 | testando['continuidade'][h[1][k]] = 2 253 | b = b + b1 254 | points = sorted(testando.groupby('continuidade').groups[0] + testando.groupby('continuidade').groups[2]) 255 | testando['Node'][points] = range(1,len(points)+1) 256 | 257 | print '\n'+str(time.time() - start)+'s - Reading and arranging are done!' 258 | sys.stdout.flush() 259 | 260 | #creating nodes shapefile 261 | path='C:\Users\Rodrigo\Desktop\GIStool\output\Shape3_nodes.shp' 262 | spatialReference = osgeo.osr.SpatialReference() #will create a spatial reference locally to tell the system what the reference will be 263 | spatialReference.ImportFromProj4('+proj=utm +zone='+utm_zone+' ellps=WGS84 +datum=WGS84 +units=m') #here we define this reference to be utm Zone 16N with wgs84 264 | driver = osgeo.ogr.GetDriverByName('ESRI Shapefile') # will select the driver for our shp-file creation. 265 | shapeData = driver.CreateDataSource(path) #so there we will store our data 266 | layer = shapeData.CreateLayer('Nodes', spatialReference, osgeo.ogr.wkbPoint) #this will create a corresponding layer for our data with given spatial information 267 | layer_defn = layer.GetLayerDefn() # gets parameters of the current shapefile 268 | point = osgeo.ogr.Geometry(osgeo.ogr.wkbPoint) 269 | 270 | new_field = ogr.FieldDefn('Node', ogr.OFTInteger) #and a forth field 'oneway' stored as integer 271 | layer.CreateField(new_field) #self explaining 272 | new_field = ogr.FieldDefn('coord_x', ogr.OFTReal) #and a forth field 'oneway' stored as integer 273 | layer.CreateField(new_field) #self explaining 274 | new_field = ogr.FieldDefn('coord_y', ogr.OFTReal) #and a forth field 'oneway' stored as integer 275 | layer.CreateField(new_field) #self explaining 276 | 277 | new_field = ogr.FieldDefn('bridge', ogr.OFTInteger) #we will create a new field called bridge as integer 278 | layer.CreateField(new_field) #self explaining 279 | new_field = ogr.FieldDefn('tunnel', ogr.OFTInteger) #and a sixth field 'tunnel' stored as integer 280 | layer.CreateField(new_field) #self explaining 281 | field_names = [layer_defn.GetFieldDefn(i).GetName() for i in range(layer_defn.GetFieldCount())] 282 | aux_b, featureIndex = list(zip(*b)[0]), 0 283 | 284 | testando_nodes = testando[testando.continuidade != 1].drop_duplicates(['coordinates_utm']) 285 | testando_nodes.index = range(len(testando_nodes)) 286 | 287 | t, aux_b, a, b, b1, points = None, None, None, None, None, None 288 | 289 | print '\n'+str(time.time() - start)+'s - Adding nodes...' 290 | sys.stdout.flush() 291 | 292 | for featureIndex in range(len(testando_nodes)): 293 | point.AddPoint(testando_nodes['coordinates_utm'][featureIndex][0],testando_nodes['coordinates_utm'][featureIndex][1]) #create a new point at given coordinates 294 | feature = osgeo.ogr.Feature(layer_defn) 295 | feature.SetGeometry(point) 296 | feature.SetFID(featureIndex) 297 | layer.CreateFeature(feature) 298 | add_field(field_names,featureIndex) 299 | shapeData = None 300 | 301 | new_b = None 302 | testando_nodes = None 303 | 304 | print '\n'+str(time.time() - start)+'s - Nodes are done!' 305 | sys.stdout.flush() 306 | 307 | #Cut the raw links and Creat the edge shapefile 308 | columns = ['X_Coord_Start','Y_Coord_Start','X_Coord_End','Y_Coord_End','Start_Node','End_Node','Edge','Length'] 309 | edgelist = pd.DataFrame(columns = columns, index=None) 310 | 311 | #creating edges shapefile 312 | path='C:\Users\Rodrigo\Desktop\GIStool\output\Shape3_edges.shp' 313 | spatialReference = osgeo.osr.SpatialReference() #will create a spatial reference locally to tell the system what the reference will be 314 | #here you can change the utm zone depending on where the map is from 315 | spatialReference.ImportFromProj4('+proj=utm +zone='+utm_zone+' ellps=WGS84 +datum=WGS84 +units=m') #here we define this reference to be utm Zone 16N with wgs84 316 | driver = osgeo.ogr.GetDriverByName('ESRI Shapefile') # will select the driver for our shp-file creation. 317 | shapeData = driver.CreateDataSource(path) #so there we will store our data 318 | layer = shapeData.CreateLayer('linestrings', spatialReference, osgeo.ogr.wkbLineString) #this will create a corresponding layer for our data with given spatial information. 319 | layer_defn = layer.GetLayerDefn() # gets parameters of the current shapefile 320 | 321 | for f in ['Edge']+heads[:-5]+['Length']: 322 | fieldDefn = ogr.FieldDefn(f, ogr.OFTString) 323 | layer.CreateField(fieldDefn) 324 | field_names = [layer_defn.GetFieldDefn(i).GetName() for i in range(layer_defn.GetFieldCount())] 325 | 326 | print '\n'+str(time.time() - start)+'s - Adding links and creating the Edgelist...' 327 | sys.stdout.flush() 328 | 329 | links = (testando.groupby(['Group']).groups) 330 | links1 = (testando.groupby(['Group','continuidade']).groups) 331 | cont = 0 332 | for link in range(len(links)): 333 | try: 334 | length_0 = len(links1[link,0]) 335 | except: 336 | length_0 = -1 337 | try: 338 | length_1 = len(links1[link,1]) 339 | except: 340 | length_1 = -1 341 | try: 342 | length_2 = len(links1[link,2]) 343 | except: 344 | length_2 = -1 345 | if length_0 == 1 and length_2 == 1: 346 | start_node, end_node = min(links1[link,0][0],links1[link,2][0]), max(links1[link,0][0],links1[link,2][0]) 347 | add_link(start_node,end_node,cont) 348 | edgelist = edgelist.append(Edgelist(start_node,end_node,cont+1,distance(start_node,end_node)),ignore_index=True) 349 | cont += 1 350 | elif length_0 == -1: 351 | for i in range(length_2-1): 352 | start_node, end_node = links1[link,2][i], links1[link,2][i+1] 353 | add_link(start_node,end_node,cont) 354 | edgelist = edgelist.append(Edgelist(start_node,end_node,cont+1,distance(start_node,end_node)),ignore_index=True) 355 | cont += 1 356 | elif length_2 == -1: 357 | start_node, end_node = links1[link,0][0], links1[link,0][1] 358 | add_link(start_node,end_node,cont) 359 | edgelist = edgelist.append(Edgelist(start_node,end_node,cont+1,distance(start_node,end_node)),ignore_index=True) 360 | cont += 1 361 | else: 362 | if length_0 > 1: 363 | start_node, end_node = links1[link,0][0], links1[link,2][0] 364 | add_link(start_node,end_node,cont) 365 | edgelist = edgelist.append(Edgelist(start_node,end_node,cont+1,distance(start_node,end_node)),ignore_index=True) 366 | cont += 1 367 | for i in range(length_2-1): 368 | start_node, end_node = links1[link,2][i], links1[link,2][i+1] 369 | add_link(start_node,end_node,cont) 370 | edgelist = edgelist.append(Edgelist(start_node,end_node,cont+1,distance(start_node,end_node)),ignore_index=True) 371 | cont += 1 372 | start_node, end_node = links1[link,2][-1], links1[link,0][1] 373 | add_link(start_node,end_node,cont) 374 | edgelist = edgelist.append(Edgelist(start_node,end_node,cont+1,distance(start_node,end_node)),ignore_index=True) 375 | cont += 1 376 | else: 377 | if links1[link,0][0] < links1[link,2][0]: 378 | start_node, end_node = links1[link,0][0], links1[link,2][0] 379 | add_link(start_node,end_node,cont) 380 | edgelist = edgelist.append(Edgelist(start_node,end_node,cont+1,distance(start_node,end_node)),ignore_index=True) 381 | cont += 1 382 | for i in range(length_2 - 1): 383 | start_node, end_node = links1[link,2][i], links1[link,2][i+1] 384 | add_link(start_node,end_node,cont) 385 | edgelist = edgelist.append(Edgelist(start_node,end_node,cont+1,distance(start_node,end_node)),ignore_index=True) 386 | cont += 1 387 | else: 388 | for i in range(length_2 - 1): 389 | start_node, end_node = links1[link,2][i], links1[link,2][i+1] 390 | add_link(start_node,end_node,cont) 391 | edgelist = edgelist.append(Edgelist(start_node,end_node,cont+1,distance(start_node,end_node)),ignore_index=True) 392 | cont += 1 393 | start_node, end_node = links1[link,2][-1], links1[link,0][0] 394 | add_link(start_node,end_node,cont) 395 | edgelist = edgelist.append(Edgelist(start_node,end_node,cont+1,distance(start_node,end_node)),ignore_index=True) 396 | cont += 1 397 | shapeData = None 398 | 399 | print '\n'+str(time.time() - start)+'s - Links and Edgelist are done!' 400 | sys.stdout.flush() 401 | 402 | print '\n'+str(time.time() - start)+'s - Saving Edgelist...' 403 | sys.stdout.flush() 404 | 405 | edgelist.to_csv('C:\Users\Rodrigo\Desktop\GIStool\output\Shape3_edgelist.csv',index=False) 406 | 407 | wb = QgsVectorLayer('C:\Users\Rodrigo\Desktop\GIStool\output\Shape3_edges.shp', 'Shape3_edges', 'ogr') 408 | QgsMapLayerRegistry.instance().addMapLayer(wb) 409 | wb = QgsVectorLayer('C:\Users\Rodrigo\Desktop\GIStool\output\Shape3_nodes.shp', 'Shape3_nodes', 'ogr') 410 | QgsMapLayerRegistry.instance().addMapLayer(wb) 411 | print '\n'+str(time.time() - start)+'s - You are all set.' 412 | sys.stdout.flush() 413 | -------------------------------------------------------------------------------- /QGIS/v1.20/GISF2E_qgis_OSM_v1.20.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | ''' 3 | GISF2E QGIS OSM - v1.20 4 | 5 | To cite, please use: Karduni, A., Kermanshah, A., & Derrible, S., 2016 “A protocol to convert spatial polyline data to network formats and applications to world urban road networks”, Scientific Data, 3:160046 6 | The article is available at: http://www.nature.com/articles/sdata201646 7 | 8 | Main authors: Felipe Macena, Rodrigo Marinho 9 | ''' 10 | 11 | #import all libraries required 12 | import shapefile 13 | import pandas as pd 14 | from pyproj import Proj 15 | import osgeo.ogr, osgeo.osr 16 | from osgeo import ogr 17 | import sys 18 | import os 19 | import time 20 | from math import pi, sin, cos, tan, sqrt 21 | from qgis.core import * 22 | def LLtoUTM(ReferenceEllipsoid, Lat, Long, zone = None): 23 | """converts lat/long to UTM coords. Equations from USGS Bulletin 1532 24 | East Longitudes are positive, West longitudes are negative. 25 | North latitudes are positive, South latitudes are negative 26 | Lat and Long are in decimal degrees 27 | Written by Chuck Gantz- chuck.gantz@globalstar.com""" 28 | 29 | a = _ellipsoid[ReferenceEllipsoid][_EquatorialRadius] 30 | eccSquared = _ellipsoid[ReferenceEllipsoid][_eccentricitySquared] 31 | k0 = 0.9996 32 | 33 | #Make sure the longitude is between -180.00 .. 179.9 34 | LongTemp = (Long+180)-int((Long+180)/360)*360-180 # -180.00 .. 179.9 35 | 36 | LatRad = Lat*_deg2rad 37 | LongRad = LongTemp*_deg2rad 38 | 39 | if zone is None: 40 | ZoneNumber = int((LongTemp + 180)/6) + 1 41 | else: 42 | ZoneNumber = zone 43 | 44 | if Lat >= 56.0 and Lat < 64.0 and LongTemp >= 3.0 and LongTemp < 12.0: 45 | ZoneNumber = 32 46 | 47 | # Special zones for Svalbard 48 | if Lat >= 72.0 and Lat < 84.0: 49 | if LongTemp >= 0.0 and LongTemp < 9.0:ZoneNumber = 31 50 | elif LongTemp >= 9.0 and LongTemp < 21.0: ZoneNumber = 33 51 | elif LongTemp >= 21.0 and LongTemp < 33.0: ZoneNumber = 35 52 | elif LongTemp >= 33.0 and LongTemp < 42.0: ZoneNumber = 37 53 | 54 | LongOrigin = (ZoneNumber - 1)*6 - 180 + 3 #+3 puts origin in middle of zone 55 | LongOriginRad = LongOrigin * _deg2rad 56 | 57 | #compute the UTM Zone from the latitude and longitude 58 | UTMZone = "%d%c" % (ZoneNumber, _UTMLetterDesignator(Lat)) 59 | 60 | eccPrimeSquared = (eccSquared)/(1-eccSquared) 61 | N = a/sqrt(1-eccSquared*sin(LatRad)*sin(LatRad)) 62 | T = tan(LatRad)*tan(LatRad) 63 | C = eccPrimeSquared*cos(LatRad)*cos(LatRad) 64 | A = cos(LatRad)*(LongRad-LongOriginRad) 65 | 66 | M = a*((1 67 | - eccSquared/4 68 | - 3*eccSquared*eccSquared/64 69 | - 5*eccSquared*eccSquared*eccSquared/256)*LatRad 70 | - (3*eccSquared/8 71 | + 3*eccSquared*eccSquared/32 72 | + 45*eccSquared*eccSquared*eccSquared/1024)*sin(2*LatRad) 73 | + (15*eccSquared*eccSquared/256 + 45*eccSquared*eccSquared*eccSquared/1024)*sin(4*LatRad) 74 | - (35*eccSquared*eccSquared*eccSquared/3072)*sin(6*LatRad)) 75 | 76 | UTMEasting = (k0*N*(A+(1-T+C)*A*A*A/6 77 | + (5-18*T+T*T+72*C-58*eccPrimeSquared)*A*A*A*A*A/120) 78 | + 500000.0) 79 | 80 | UTMNorthing = (k0*(M+N*tan(LatRad)*(A*A/2+(5-T+9*C+4*C*C)*A*A*A*A/24 81 | + (61 82 | -58*T 83 | +T*T 84 | +600*C 85 | -330*eccPrimeSquared)*A*A*A*A*A*A/720))) 86 | if Lat < 0: 87 | UTMNorthing = UTMNorthing + 10000000.0; #10000000 meter offset for southern hemisphere 88 | return (UTMZone, UTMEasting, UTMNorthing) 89 | 90 | 91 | def _UTMLetterDesignator(Lat): 92 | """This routine determines the correct UTM letter designator for the given 93 | latitude returns 'Z' if latitude is outside the UTM limits of 84N to 80S 94 | Written by Chuck Gantz- chuck.gantz@globalstar.com""" 95 | if Lat >= 0: return 'N' 96 | else: return 'S' 97 | 98 | def add_link(start,end,cont): 99 | pontos = [] 100 | pontos.append(osgeo.ogr.Geometry(osgeo.ogr.wkbLineString)) 101 | lista = list([testando['coordinates_utm'][start:end+1]][0]) 102 | for a in lista: 103 | pontos[-1].AddPoint(a[0],a[1]) 104 | featureIndex = cont 105 | feature = osgeo.ogr.Feature(layer_defn) 106 | feature.SetGeometry(pontos[-1]) 107 | feature.SetFID(featureIndex) 108 | layer.CreateFeature(feature) 109 | for name in range(len(field_names)): 110 | if field_names[name] != 'Edge' and field_names[name] != 'Length': 111 | feature.SetField(name, str(testando[field_names[name]][start_node])) 112 | elif field_names[name] == 'Edge': 113 | feature.SetField(name, cont+1) 114 | elif field_names[name] == 'Length': 115 | feature.SetField(name, distance(start,end)) 116 | layer.SetFeature(feature) 117 | def add_field(field_names,featureIndex): 118 | for name in range(len(field_names)): 119 | if field_names[name] == 'coord_x' or field_names[name] == 'coord_y': 120 | name_aux = 'coordinates_utm' 121 | else: 122 | name_aux = field_names[name] 123 | feature = layer.GetFeature(featureIndex) #lets get the first feature (FID=='0') 124 | if field_names[name] == 'coord_x': 125 | feature.SetField(name,testando_nodes[name_aux][featureIndex][0]) 126 | elif field_names[name] == 'coord_y': 127 | feature.SetField(name,testando_nodes[name_aux][featureIndex][1]) 128 | else: 129 | feature.SetField(name,int(testando_nodes[name_aux][featureIndex])) 130 | layer.SetFeature(feature) #now make the change permanent 131 | def distance(start,end): 132 | dist = 0 133 | for u in range(start,end): 134 | dist = dist + ((testando['coordinates_utm'][u][0] - testando['coordinates_utm'][u+1][0])**2 + (testando['coordinates_utm'][u][1] - testando['coordinates_utm'][u+1][1])**2)**0.5 135 | return dist 136 | def Edgelist(start,end,edge,length): 137 | global testando, columns 138 | datasheet = pd.DataFrame([[testando['coordinates_utm'][start][0],testando['coordinates_utm'][start][1],testando['coordinates_utm'][end][0],testando['coordinates_utm'][end][1],testando['Node'][start],testando['Node'][end],edge,length]],columns = columns, index = None) 139 | return datasheet 140 | 141 | _deg2rad, _rad2deg, _EquatorialRadius, _eccentricitySquared = pi / 180.0, 180.0 / pi, 2, 3 142 | 143 | _ellipsoid = [ 144 | # id, Ellipsoid name, Equatorial Radius, square of eccentricity 145 | # first once is a placeholder only, To allow array indices to match id numbers 146 | [ -1, "Placeholder", 0, 0], 147 | [ 1, "Airy", 6377563, 0.00667054], 148 | [ 2, "Australian National", 6378160, 0.006694542], 149 | [ 3, "Bessel 1841", 6377397, 0.006674372], 150 | [ 4, "Bessel 1841 (Nambia] ", 6377484, 0.006674372], 151 | [ 5, "Clarke 1866", 6378206, 0.006768658], 152 | [ 6, "Clarke 1880", 6378249, 0.006803511], 153 | [ 7, "Everest", 6377276, 0.006637847], 154 | [ 8, "Fischer 1960 (Mercury] ", 6378166, 0.006693422], 155 | [ 9, "Fischer 1968", 6378150, 0.006693422], 156 | [ 10, "GRS 1967", 6378160, 0.006694605], 157 | [ 11, "GRS 1980", 6378137, 0.00669438], 158 | [ 12, "Helmert 1906", 6378200, 0.006693422], 159 | [ 13, "Hough", 6378270, 0.00672267], 160 | [ 14, "International", 6378388, 0.00672267], 161 | [ 15, "Krassovsky", 6378245, 0.006693422], 162 | [ 16, "Modified Airy", 6377340, 0.00667054], 163 | [ 17, "Modified Everest", 6377304, 0.006637847], 164 | [ 18, "Modified Fischer 1960", 6378155, 0.006693422], 165 | [ 19, "South American 1969", 6378160, 0.006694542], 166 | [ 20, "WGS 60", 6378165, 0.006693422], 167 | [ 21, "WGS 66", 6378145, 0.006694542], 168 | [ 22, "WGS-72", 6378135, 0.006694318], 169 | [ 23, "WGS-84", 6378137, 0.00669438] 170 | ] 171 | 172 | 173 | 174 | #dividing each link by their osm_id information (osm_maps) or id information (non-osm_maps) 175 | 176 | sf = shapefile.Reader("C:\YOUR_PATH\input\Chicago_cut.shp") 177 | shapes = sf.shapes() 178 | fields=sf.fields 179 | fields = [field[0] for field in sf.fields[1:]] 180 | start = time.time() 181 | print '\n'+str(time.time() - start)+'s - Reading and arranging the root file information...' 182 | sys.stdout.flush() 183 | #here you can change the utm zone depending on where the map is from 184 | #p = Proj(proj='utm',zone=int(utm_zone[:-1]),ellps='WGS84') 185 | 186 | if 'TYPE' in fields or 'Type' in fields: 187 | try: 188 | fields[fields.index('TYPE')] = 'type' 189 | except: 190 | fields[fields.index('Type')] = 'type' 191 | 192 | heads = [x.upper() if x in ['Group','coordinates','primeiro','coordinates_utm','Node'] else x for x in [sf.shapeRecords()[0].shape.__geo_interface__.keys()[0]]+fields] + ['Group','coordinates','primeiro','coordinates_utm','Node'] 193 | type_ = False 194 | if 'type' in map(lambda x: x.lower(),fields): 195 | heads.remove('type') 196 | heads.insert(3,'type 1') 197 | type_ = True 198 | 199 | dicti = {} 200 | for i in range(len(heads)-1): 201 | dicti[heads[i]] = [] 202 | 203 | testando = pd.DataFrame() 204 | 205 | n = 0 206 | for feature in sf.shapeRecords(): 207 | geom = feature.shape.__geo_interface__ 208 | atr = dict(zip(fields, feature.record)) 209 | if type_: 210 | atr['type 1'] = atr.pop('type') 211 | geom.update(atr) 212 | geom['Group'] = n 213 | j = geom['coordinates'] 214 | for i in range(len(heads)-4): 215 | u = [geom[heads[i]]] 216 | for y in range(len(j)): 217 | dicti[heads[i]].append(u[0]) 218 | dicti['coordinates'] = list(dicti['coordinates']) + list(j) 219 | dicti['primeiro'] = dicti['primeiro'] + [1] + [0]*(len(j) - 2) + [1] 220 | n += 1 221 | if ((n-1) % 800) == 0: 222 | dicti['coordinates_utm'] = None 223 | dicti['Node'] = None 224 | testando = pd.concat([testando,pd.DataFrame.from_dict(dicti)]) 225 | for i in range(len(heads)-1): 226 | dicti[heads[i]] = [] 227 | if ((n-1) % 800) != 0: 228 | dicti['coordinates_utm'] = None 229 | dicti['Node'] = None 230 | testando = pd.concat([testando,pd.DataFrame.from_dict(dicti)],ignore_index=True) 231 | 232 | utm_zone = LLtoUTM(23, testando['coordinates'][0][1], testando['coordinates'][0][0])[0] 233 | if utm_zone[-1] == "S": 234 | p = Proj("+proj=utm +zone="+str(int(utm_zone[:-1]))+", +south +ellps=WGS84") 235 | else: 236 | p = Proj("+proj=utm +zone="+str(int(utm_zone[:-1]))+", +ellps=WGS84") 237 | 238 | testando['coordinates_utm'] = [p(testando['coordinates'][k][0],testando['coordinates'][k][1]) for k in range(len(testando['coordinates']))] 239 | testando['Node'] = None 240 | 241 | #testando = pd.DataFrame.from_dict(dicti) 242 | dicti = None 243 | geom = None 244 | atr = None 245 | #test before creating nodes for OSM maps 246 | a= testando.groupby('coordinates_utm').groups 247 | testando['continuidade'] = 1 248 | testando.ix[testando.primeiro == 1,'continuidade'] = 0; testando 249 | b1 = map(lambda x: (testando['coordinates_utm'][x],[x]),testando.groupby('continuidade').groups[0]) 250 | 251 | b = filter(lambda x: (testando['bridge'][x[1][0]] == testando['bridge'][int(x[1][1])]) or (testando['tunnel'][x[1][0]] == testando['tunnel'][int(x[1][1])]), filter(lambda x: len(x[1]) >= 2, zip(a.keys(),a.values()))) 252 | 253 | for h in (b): 254 | for k in range(len(h[1])): 255 | testando.loc[[h[1][k]], 'continuidade'] = 2 256 | b = b + b1 257 | points = sorted(list(testando.groupby('continuidade').groups[0]) + list(testando.groupby('continuidade').groups[2])) 258 | testando[points, 'Node'] = range(1,len(points)+1) 259 | 260 | print '\n'+str(time.time() - start)+'s - Reading and arranging are done!' 261 | sys.stdout.flush() 262 | 263 | #creating nodes shapefile 264 | path='C:\YOUR_PATH\output\Chicago_cut_nodes.shp' 265 | spatialReference = osgeo.osr.SpatialReference() #will create a spatial reference locally to tell the system what the reference will be 266 | spatialReference.ImportFromProj4('+proj=utm +zone='+utm_zone+' ellps=WGS84 +datum=WGS84 +units=m') #here we define this reference to be utm Zone 16N with wgs84 267 | driver = osgeo.ogr.GetDriverByName('ESRI Shapefile') # will select the driver for our shp-file creation. 268 | shapeData = driver.CreateDataSource(path) #so there we will store our data 269 | layer = shapeData.CreateLayer('Nodes', spatialReference, osgeo.ogr.wkbPoint) #this will create a corresponding layer for our data with given spatial information 270 | layer_defn = layer.GetLayerDefn() # gets parameters of the current shapefile 271 | point = osgeo.ogr.Geometry(osgeo.ogr.wkbPoint) 272 | 273 | new_field = ogr.FieldDefn('Node', ogr.OFTInteger) #and a forth field 'oneway' stored as integer 274 | layer.CreateField(new_field) #self explaining 275 | new_field = ogr.FieldDefn('coord_x', ogr.OFTReal) #and a forth field 'oneway' stored as integer 276 | layer.CreateField(new_field) #self explaining 277 | new_field = ogr.FieldDefn('coord_y', ogr.OFTReal) #and a forth field 'oneway' stored as integer 278 | layer.CreateField(new_field) #self explaining 279 | 280 | new_field = ogr.FieldDefn('bridge', ogr.OFTInteger) #we will create a new field called bridge as integer 281 | layer.CreateField(new_field) #self explaining 282 | new_field = ogr.FieldDefn('tunnel', ogr.OFTInteger) #and a sixth field 'tunnel' stored as integer 283 | layer.CreateField(new_field) #self explaining 284 | field_names = [layer_defn.GetFieldDefn(i).GetName() for i in range(layer_defn.GetFieldCount())] 285 | aux_b, featureIndex = list(zip(*b)[0]), 0 286 | 287 | testando_nodes = testando[testando.continuidade != 1].drop_duplicates(['coordinates_utm']) 288 | testando_nodes.index = range(len(testando_nodes)) 289 | 290 | t, aux_b, a, b, b1, points = None, None, None, None, None, None 291 | 292 | print '\n'+str(time.time() - start)+'s - Adding nodes...' 293 | sys.stdout.flush() 294 | 295 | for featureIndex in range(len(testando_nodes)): 296 | point.AddPoint(testando_nodes['coordinates_utm'][featureIndex][0],testando_nodes['coordinates_utm'][featureIndex][1]) #create a new point at given coordinates 297 | feature = osgeo.ogr.Feature(layer_defn) 298 | feature.SetGeometry(point) 299 | feature.SetFID(featureIndex) 300 | layer.CreateFeature(feature) 301 | add_field(field_names,featureIndex) 302 | shapeData = None 303 | 304 | new_b = None 305 | testando_nodes = None 306 | 307 | print '\n'+str(time.time() - start)+'s - Nodes are done!' 308 | sys.stdout.flush() 309 | 310 | #Cut the raw links and Creat the edge shapefile 311 | columns = ['X_Coord_Start','Y_Coord_Start','X_Coord_End','Y_Coord_End','Start_Node','End_Node','Edge','Length'] 312 | edgelist = pd.DataFrame(columns = columns, index=None) 313 | 314 | #creating edges shapefile 315 | path='C:\YOUR_PATH\output\Chicago_cut_edges.shp' 316 | spatialReference = osgeo.osr.SpatialReference() #will create a spatial reference locally to tell the system what the reference will be 317 | #here you can change the utm zone depending on where the map is from 318 | spatialReference.ImportFromProj4('+proj=utm +zone='+utm_zone+' ellps=WGS84 +datum=WGS84 +units=m') #here we define this reference to be utm Zone 16N with wgs84 319 | driver = osgeo.ogr.GetDriverByName('ESRI Shapefile') # will select the driver for our shp-file creation. 320 | shapeData = driver.CreateDataSource(path) #so there we will store our data 321 | layer = shapeData.CreateLayer('linestrings', spatialReference, osgeo.ogr.wkbLineString) #this will create a corresponding layer for our data with given spatial information. 322 | layer_defn = layer.GetLayerDefn() # gets parameters of the current shapefile 323 | 324 | for f in ['Edge']+heads[:-5]+['Length']: 325 | fieldDefn = ogr.FieldDefn(f, ogr.OFTString) 326 | layer.CreateField(fieldDefn) 327 | field_names = [layer_defn.GetFieldDefn(i).GetName() for i in range(layer_defn.GetFieldCount())] 328 | 329 | print '\n'+str(time.time() - start)+'s - Adding links and creating the Edgelist...' 330 | sys.stdout.flush() 331 | 332 | links = (testando.groupby(['Group']).groups) 333 | links1 = (testando.groupby(['Group','continuidade']).groups) 334 | cont = 0 335 | for link in range(len(links)): 336 | try: 337 | length_0 = len(links1[link,0]) 338 | except: 339 | length_0 = -1 340 | try: 341 | length_1 = len(links1[link,1]) 342 | except: 343 | length_1 = -1 344 | try: 345 | length_2 = len(links1[link,2]) 346 | except: 347 | length_2 = -1 348 | if length_0 == 1 and length_2 == 1: 349 | start_node, end_node = min(links1[link,0][0],links1[link,2][0]), max(links1[link,0][0],links1[link,2][0]) 350 | add_link(start_node,end_node,cont) 351 | edgelist = edgelist.append(Edgelist(start_node,end_node,cont+1,distance(start_node,end_node)),ignore_index=True) 352 | cont += 1 353 | elif length_0 == -1: 354 | for i in range(length_2-1): 355 | start_node, end_node = links1[link,2][i], links1[link,2][i+1] 356 | add_link(start_node,end_node,cont) 357 | edgelist = edgelist.append(Edgelist(start_node,end_node,cont+1,distance(start_node,end_node)),ignore_index=True) 358 | cont += 1 359 | elif length_2 == -1: 360 | start_node, end_node = links1[link,0][0], links1[link,0][1] 361 | add_link(start_node,end_node,cont) 362 | edgelist = edgelist.append(Edgelist(start_node,end_node,cont+1,distance(start_node,end_node)),ignore_index=True) 363 | cont += 1 364 | else: 365 | if length_0 > 1: 366 | start_node, end_node = links1[link,0][0], links1[link,2][0] 367 | add_link(start_node,end_node,cont) 368 | edgelist = edgelist.append(Edgelist(start_node,end_node,cont+1,distance(start_node,end_node)),ignore_index=True) 369 | cont += 1 370 | for i in range(length_2-1): 371 | start_node, end_node = links1[link,2][i], links1[link,2][i+1] 372 | add_link(start_node,end_node,cont) 373 | edgelist = edgelist.append(Edgelist(start_node,end_node,cont+1,distance(start_node,end_node)),ignore_index=True) 374 | cont += 1 375 | start_node, end_node = links1[link,2][-1], links1[link,0][1] 376 | add_link(start_node,end_node,cont) 377 | edgelist = edgelist.append(Edgelist(start_node,end_node,cont+1,distance(start_node,end_node)),ignore_index=True) 378 | cont += 1 379 | else: 380 | if links1[link,0][0] < links1[link,2][0]: 381 | start_node, end_node = links1[link,0][0], links1[link,2][0] 382 | add_link(start_node,end_node,cont) 383 | edgelist = edgelist.append(Edgelist(start_node,end_node,cont+1,distance(start_node,end_node)),ignore_index=True) 384 | cont += 1 385 | for i in range(length_2 - 1): 386 | start_node, end_node = links1[link,2][i], links1[link,2][i+1] 387 | add_link(start_node,end_node,cont) 388 | edgelist = edgelist.append(Edgelist(start_node,end_node,cont+1,distance(start_node,end_node)),ignore_index=True) 389 | cont += 1 390 | else: 391 | for i in range(length_2 - 1): 392 | start_node, end_node = links1[link,2][i], links1[link,2][i+1] 393 | add_link(start_node,end_node,cont) 394 | edgelist = edgelist.append(Edgelist(start_node,end_node,cont+1,distance(start_node,end_node)),ignore_index=True) 395 | cont += 1 396 | start_node, end_node = links1[link,2][-1], links1[link,0][0] 397 | add_link(start_node,end_node,cont) 398 | edgelist = edgelist.append(Edgelist(start_node,end_node,cont+1,distance(start_node,end_node)),ignore_index=True) 399 | cont += 1 400 | shapeData = None 401 | 402 | print '\n'+str(time.time() - start)+'s - Links and Edgelist are done!' 403 | sys.stdout.flush() 404 | 405 | print '\n'+str(time.time() - start)+'s - Saving Edgelist...' 406 | sys.stdout.flush() 407 | 408 | edgelist.to_csv('C:\YOUR_PATH\output\Chicago_cut_edgelist.csv',index=False) 409 | 410 | wb = QgsVectorLayer('C:\YOUR_PATH\output\Chicago_cut_edges.shp', 'Chicago_cut_edges', 'ogr') 411 | QgsMapLayerRegistry.instance().addMapLayer(wb) 412 | wb = QgsVectorLayer('C:\YOUR_PATH\output\Chicago_cut_nodes.shp', 'Chicago_cut_nodes', 'ogr') 413 | QgsMapLayerRegistry.instance().addMapLayer(wb) 414 | print '\n'+str(time.time() - start)+'s - You are all set.' 415 | sys.stdout.flush() 416 | -------------------------------------------------------------------------------- /QGIS/v1.00/GISF2E_qgis_OSM_v1.00.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | ''' 3 | GISF2E QGIS OSM - v1.00 - Note that the code was developed with pandas version 0.14.1, see version 1.10 that works with all pandas versions. 4 | 5 | To cite, please use: Karduni, A., Kermanshah, A., & Derrible, S., 2016 “A protocol to convert spatial polyline data to network formats and applications to world urban road networks”, Scientific Data, 3:160046 6 | The article is available at: http://www.nature.com/articles/sdata201646 7 | 8 | Main authors: Felipe Macena, Rodrigo Marinho 9 | ''' 10 | 11 | #import all libraries required 12 | import shapefile 13 | import pandas as pd 14 | from pyproj import Proj 15 | import osgeo.ogr, osgeo.osr 16 | from osgeo import ogr 17 | import sys 18 | import os 19 | import time 20 | from math import pi, sin, cos, tan, sqrt 21 | from qgis.core import * 22 | def LLtoUTM(ReferenceEllipsoid, Lat, Long, zone = None): 23 | """converts lat/long to UTM coords. Equations from USGS Bulletin 1532 24 | East Longitudes are positive, West longitudes are negative. 25 | North latitudes are positive, South latitudes are negative 26 | Lat and Long are in decimal degrees 27 | Written by Chuck Gantz- chuck.gantz@globalstar.com""" 28 | 29 | a = _ellipsoid[ReferenceEllipsoid][_EquatorialRadius] 30 | eccSquared = _ellipsoid[ReferenceEllipsoid][_eccentricitySquared] 31 | k0 = 0.9996 32 | 33 | #Make sure the longitude is between -180.00 .. 179.9 34 | LongTemp = (Long+180)-int((Long+180)/360)*360-180 # -180.00 .. 179.9 35 | 36 | LatRad = Lat*_deg2rad 37 | LongRad = LongTemp*_deg2rad 38 | 39 | if zone is None: 40 | ZoneNumber = int((LongTemp + 180)/6) + 1 41 | else: 42 | ZoneNumber = zone 43 | 44 | if Lat >= 56.0 and Lat < 64.0 and LongTemp >= 3.0 and LongTemp < 12.0: 45 | ZoneNumber = 32 46 | 47 | # Special zones for Svalbard 48 | if Lat >= 72.0 and Lat < 84.0: 49 | if LongTemp >= 0.0 and LongTemp < 9.0:ZoneNumber = 31 50 | elif LongTemp >= 9.0 and LongTemp < 21.0: ZoneNumber = 33 51 | elif LongTemp >= 21.0 and LongTemp < 33.0: ZoneNumber = 35 52 | elif LongTemp >= 33.0 and LongTemp < 42.0: ZoneNumber = 37 53 | 54 | LongOrigin = (ZoneNumber - 1)*6 - 180 + 3 #+3 puts origin in middle of zone 55 | LongOriginRad = LongOrigin * _deg2rad 56 | 57 | #compute the UTM Zone from the latitude and longitude 58 | UTMZone = "%d%c" % (ZoneNumber, _UTMLetterDesignator(Lat)) 59 | 60 | eccPrimeSquared = (eccSquared)/(1-eccSquared) 61 | N = a/sqrt(1-eccSquared*sin(LatRad)*sin(LatRad)) 62 | T = tan(LatRad)*tan(LatRad) 63 | C = eccPrimeSquared*cos(LatRad)*cos(LatRad) 64 | A = cos(LatRad)*(LongRad-LongOriginRad) 65 | 66 | M = a*((1 67 | - eccSquared/4 68 | - 3*eccSquared*eccSquared/64 69 | - 5*eccSquared*eccSquared*eccSquared/256)*LatRad 70 | - (3*eccSquared/8 71 | + 3*eccSquared*eccSquared/32 72 | + 45*eccSquared*eccSquared*eccSquared/1024)*sin(2*LatRad) 73 | + (15*eccSquared*eccSquared/256 + 45*eccSquared*eccSquared*eccSquared/1024)*sin(4*LatRad) 74 | - (35*eccSquared*eccSquared*eccSquared/3072)*sin(6*LatRad)) 75 | 76 | UTMEasting = (k0*N*(A+(1-T+C)*A*A*A/6 77 | + (5-18*T+T*T+72*C-58*eccPrimeSquared)*A*A*A*A*A/120) 78 | + 500000.0) 79 | 80 | UTMNorthing = (k0*(M+N*tan(LatRad)*(A*A/2+(5-T+9*C+4*C*C)*A*A*A*A/24 81 | + (61 82 | -58*T 83 | +T*T 84 | +600*C 85 | -330*eccPrimeSquared)*A*A*A*A*A*A/720))) 86 | if Lat < 0: 87 | UTMNorthing = UTMNorthing + 10000000.0; #10000000 meter offset for southern hemisphere 88 | return (UTMZone, UTMEasting, UTMNorthing) 89 | 90 | 91 | def _UTMLetterDesignator(Lat): 92 | """This routine determines the correct UTM letter designator for the given 93 | latitude returns 'Z' if latitude is outside the UTM limits of 84N to 80S 94 | Written by Chuck Gantz- chuck.gantz@globalstar.com""" 95 | if Lat >= 0: return 'N' 96 | else: return 'S' 97 | 98 | def add_link(start,end,cont): 99 | pontos = [] 100 | pontos.append(osgeo.ogr.Geometry(osgeo.ogr.wkbLineString)) 101 | lista = list([testando['coordinates_utm'][start:end+1]][0]) 102 | for a in lista: 103 | pontos[-1].AddPoint(a[0],a[1]) 104 | featureIndex = cont 105 | feature = osgeo.ogr.Feature(layer_defn) 106 | feature.SetGeometry(pontos[-1]) 107 | feature.SetFID(featureIndex) 108 | layer.CreateFeature(feature) 109 | for name in range(len(field_names)): 110 | if field_names[name] != 'Edge' and field_names[name] != 'Length': 111 | feature.SetField(name, str(testando[field_names[name]][start_node])) 112 | elif field_names[name] == 'Edge': 113 | feature.SetField(name, cont+1) 114 | elif field_names[name] == 'Length': 115 | feature.SetField(name, distance(start,end)) 116 | layer.SetFeature(feature) 117 | def add_field(field_names,featureIndex): 118 | for name in range(len(field_names)): 119 | if field_names[name] == 'coord_x' or field_names[name] == 'coord_y': 120 | name_aux = 'coordinates_utm' 121 | else: 122 | name_aux = field_names[name] 123 | feature = layer.GetFeature(featureIndex) #lets get the first feature (FID=='0') 124 | if field_names[name] == 'coord_x': 125 | feature.SetField(name,testando_nodes[name_aux][featureIndex][0]) 126 | elif field_names[name] == 'coord_y': 127 | feature.SetField(name,testando_nodes[name_aux][featureIndex][1]) 128 | else: 129 | feature.SetField(name,int(testando_nodes[name_aux][featureIndex])) 130 | layer.SetFeature(feature) #now make the change permanent 131 | def distance(start,end): 132 | dist = 0 133 | for u in range(start,end): 134 | dist = dist + ((testando['coordinates_utm'][u][0] - testando['coordinates_utm'][u+1][0])**2 + (testando['coordinates_utm'][u][1] - testando['coordinates_utm'][u+1][1])**2)**0.5 135 | return dist 136 | def Edgelist(start,end,edge,length): 137 | global testando, columns 138 | datasheet = pd.DataFrame([[testando['coordinates_utm'][start][0],testando['coordinates_utm'][start][1],testando['coordinates_utm'][end][0],testando['coordinates_utm'][end][1],testando['Node'][start],testando['Node'][end],edge,length]],columns = columns, index = None) 139 | return datasheet 140 | 141 | _deg2rad, _rad2deg, _EquatorialRadius, _eccentricitySquared = pi / 180.0, 180.0 / pi, 2, 3 142 | 143 | _ellipsoid = [ 144 | # id, Ellipsoid name, Equatorial Radius, square of eccentricity 145 | # first once is a placeholder only, To allow array indices to match id numbers 146 | [ -1, "Placeholder", 0, 0], 147 | [ 1, "Airy", 6377563, 0.00667054], 148 | [ 2, "Australian National", 6378160, 0.006694542], 149 | [ 3, "Bessel 1841", 6377397, 0.006674372], 150 | [ 4, "Bessel 1841 (Nambia] ", 6377484, 0.006674372], 151 | [ 5, "Clarke 1866", 6378206, 0.006768658], 152 | [ 6, "Clarke 1880", 6378249, 0.006803511], 153 | [ 7, "Everest", 6377276, 0.006637847], 154 | [ 8, "Fischer 1960 (Mercury] ", 6378166, 0.006693422], 155 | [ 9, "Fischer 1968", 6378150, 0.006693422], 156 | [ 10, "GRS 1967", 6378160, 0.006694605], 157 | [ 11, "GRS 1980", 6378137, 0.00669438], 158 | [ 12, "Helmert 1906", 6378200, 0.006693422], 159 | [ 13, "Hough", 6378270, 0.00672267], 160 | [ 14, "International", 6378388, 0.00672267], 161 | [ 15, "Krassovsky", 6378245, 0.006693422], 162 | [ 16, "Modified Airy", 6377340, 0.00667054], 163 | [ 17, "Modified Everest", 6377304, 0.006637847], 164 | [ 18, "Modified Fischer 1960", 6378155, 0.006693422], 165 | [ 19, "South American 1969", 6378160, 0.006694542], 166 | [ 20, "WGS 60", 6378165, 0.006693422], 167 | [ 21, "WGS 66", 6378145, 0.006694542], 168 | [ 22, "WGS-72", 6378135, 0.006694318], 169 | [ 23, "WGS-84", 6378137, 0.00669438] 170 | ] 171 | 172 | 173 | 174 | #dividing each link by their osm_id information (osm_maps) or id information (non-osm_maps) 175 | 176 | sf = shapefile.Reader("C:\Users\Rodrigo\Desktop\GIStool\input\Chicago_cut.shp") 177 | shapes = sf.shapes() 178 | fields=sf.fields 179 | fields = [field[0] for field in sf.fields[1:]] 180 | start = time.time() 181 | print '\n'+str(time.time() - start)+'s - Reading and arranging the root file information...' 182 | sys.stdout.flush() 183 | #here you can change the utm zone depending on where the map is from 184 | #p = Proj(proj='utm',zone=int(utm_zone[:-1]),ellps='WGS84') 185 | 186 | if 'TYPE' in fields or 'Type' in fields: 187 | try: 188 | fields[fields.index('TYPE')] = 'type' 189 | except: 190 | fields[fields.index('Type')] = 'type' 191 | 192 | heads = [x.upper() if x in ['Group','coordinates','primeiro','coordinates_utm','Node'] else x for x in [sf.shapeRecords()[0].shape.__geo_interface__.keys()[0]]+fields] + ['Group','coordinates','primeiro','coordinates_utm','Node'] 193 | type_ = False 194 | if 'type' in map(lambda x: x.lower(),fields): 195 | heads.remove('type') 196 | heads.insert(3,'type 1') 197 | type_ = True 198 | 199 | dicti = {} 200 | for i in range(len(heads)-1): 201 | dicti[heads[i]] = [] 202 | 203 | testando = pd.DataFrame() 204 | 205 | n = 0 206 | for feature in sf.shapeRecords(): 207 | geom = feature.shape.__geo_interface__ 208 | atr = dict(zip(fields, feature.record)) 209 | if type_: 210 | atr['type 1'] = atr.pop('type') 211 | geom.update(atr) 212 | geom['Group'] = n 213 | j = geom['coordinates'] 214 | for i in range(len(heads)-4): 215 | u = [geom[heads[i]]] 216 | for y in range(len(j)): 217 | dicti[heads[i]].append(u[0]) 218 | dicti['coordinates'] = list(dicti['coordinates']) + list(j) 219 | dicti['primeiro'] = dicti['primeiro'] + [1] + [0]*(len(j) - 2) + [1] 220 | n += 1 221 | if ((n-1) % 800) == 0: 222 | dicti['coordinates_utm'] = None 223 | dicti['Node'] = None 224 | testando = pd.concat([testando,pd.DataFrame.from_dict(dicti)]) 225 | for i in range(len(heads)-1): 226 | dicti[heads[i]] = [] 227 | if ((n-1) % 800) != 0: 228 | dicti['coordinates_utm'] = None 229 | dicti['Node'] = None 230 | testando = pd.concat([testando,pd.DataFrame.from_dict(dicti)],ignore_index=True) 231 | 232 | utm_zone = LLtoUTM(23, testando['coordinates'][0][1], testando['coordinates'][0][0])[0] 233 | p = Proj(proj='utm',zone=int(utm_zone[:-1]),ellps='WGS84') 234 | 235 | testando['coordinates_utm'] = [p(testando['coordinates'][k][0],testando['coordinates'][k][1]) for k in range(len(testando['coordinates']))] 236 | testando['Node'] = None 237 | 238 | #testando = pd.DataFrame.from_dict(dicti) 239 | dicti = None 240 | geom = None 241 | atr = None 242 | #test before creating nodes for OSM maps 243 | a= testando.groupby('coordinates_utm').groups 244 | testando['continuidade'] = 1 245 | testando.ix[testando.primeiro == 1,'continuidade'] = 0; testando 246 | b1 = map(lambda x: (testando['coordinates_utm'][x],[x]),testando.groupby('continuidade').groups[0]) 247 | 248 | b = filter(lambda x: (testando['bridge'][x[1][0]] == testando['bridge'][int(x[1][1])]) or (testando['tunnel'][x[1][0]] == testando['tunnel'][int(x[1][1])]), filter(lambda x: len(x[1]) >= 2, zip(a.keys(),a.values()))) 249 | 250 | for h in (b): 251 | for k in range(len(h[1])): 252 | testando['continuidade'][h[1][k]] = 2 253 | b = b + b1 254 | points = sorted(testando.groupby('continuidade').groups[0] + testando.groupby('continuidade').groups[2]) 255 | testando['Node'][points] = range(1,len(points)+1) 256 | 257 | print '\n'+str(time.time() - start)+'s - Reading and arranging are done!' 258 | sys.stdout.flush() 259 | 260 | #creating nodes shapefile 261 | path='C:\Users\Rodrigo\Desktop\GIStool\output\Chicago_cut_nodes.shp' 262 | spatialReference = osgeo.osr.SpatialReference() #will create a spatial reference locally to tell the system what the reference will be 263 | spatialReference.ImportFromProj4('+proj=utm +zone='+utm_zone+' ellps=WGS84 +datum=WGS84 +units=m') #here we define this reference to be utm Zone 16N with wgs84 264 | driver = osgeo.ogr.GetDriverByName('ESRI Shapefile') # will select the driver for our shp-file creation. 265 | shapeData = driver.CreateDataSource(path) #so there we will store our data 266 | layer = shapeData.CreateLayer('Nodes', spatialReference, osgeo.ogr.wkbPoint) #this will create a corresponding layer for our data with given spatial information 267 | layer_defn = layer.GetLayerDefn() # gets parameters of the current shapefile 268 | point = osgeo.ogr.Geometry(osgeo.ogr.wkbPoint) 269 | 270 | new_field = ogr.FieldDefn('Node', ogr.OFTInteger) #and a forth field 'oneway' stored as integer 271 | layer.CreateField(new_field) #self explaining 272 | new_field = ogr.FieldDefn('coord_x', ogr.OFTReal) #and a forth field 'oneway' stored as integer 273 | layer.CreateField(new_field) #self explaining 274 | new_field = ogr.FieldDefn('coord_y', ogr.OFTReal) #and a forth field 'oneway' stored as integer 275 | layer.CreateField(new_field) #self explaining 276 | 277 | new_field = ogr.FieldDefn('bridge', ogr.OFTInteger) #we will create a new field called bridge as integer 278 | layer.CreateField(new_field) #self explaining 279 | new_field = ogr.FieldDefn('tunnel', ogr.OFTInteger) #and a sixth field 'tunnel' stored as integer 280 | layer.CreateField(new_field) #self explaining 281 | field_names = [layer_defn.GetFieldDefn(i).GetName() for i in range(layer_defn.GetFieldCount())] 282 | aux_b, featureIndex = list(zip(*b)[0]), 0 283 | 284 | testando_nodes = testando[testando.continuidade != 1].drop_duplicates(['coordinates_utm']) 285 | testando_nodes.index = range(len(testando_nodes)) 286 | 287 | t, aux_b, a, b, b1, points = None, None, None, None, None, None 288 | 289 | print '\n'+str(time.time() - start)+'s - Adding nodes...' 290 | sys.stdout.flush() 291 | 292 | for featureIndex in range(len(testando_nodes)): 293 | point.AddPoint(testando_nodes['coordinates_utm'][featureIndex][0],testando_nodes['coordinates_utm'][featureIndex][1]) #create a new point at given coordinates 294 | feature = osgeo.ogr.Feature(layer_defn) 295 | feature.SetGeometry(point) 296 | feature.SetFID(featureIndex) 297 | layer.CreateFeature(feature) 298 | add_field(field_names,featureIndex) 299 | shapeData = None 300 | 301 | new_b = None 302 | testando_nodes = None 303 | 304 | print '\n'+str(time.time() - start)+'s - Nodes are done!' 305 | sys.stdout.flush() 306 | 307 | #Cut the raw links and Creat the edge shapefile 308 | columns = ['X_Coord_Start','Y_Coord_Start','X_Coord_End','Y_Coord_End','Start_Node','End_Node','Edge','Length'] 309 | edgelist = pd.DataFrame(columns = columns, index=None) 310 | 311 | #creating edges shapefile 312 | path='C:\Users\Rodrigo\Desktop\GIStool\output\Chicago_cut_edges.shp' 313 | spatialReference = osgeo.osr.SpatialReference() #will create a spatial reference locally to tell the system what the reference will be 314 | #here you can change the utm zone depending on where the map is from 315 | spatialReference.ImportFromProj4('+proj=utm +zone='+utm_zone+' ellps=WGS84 +datum=WGS84 +units=m') #here we define this reference to be utm Zone 16N with wgs84 316 | driver = osgeo.ogr.GetDriverByName('ESRI Shapefile') # will select the driver for our shp-file creation. 317 | shapeData = driver.CreateDataSource(path) #so there we will store our data 318 | layer = shapeData.CreateLayer('linestrings', spatialReference, osgeo.ogr.wkbLineString) #this will create a corresponding layer for our data with given spatial information. 319 | layer_defn = layer.GetLayerDefn() # gets parameters of the current shapefile 320 | 321 | for f in ['Edge']+heads[:-5]+['Length']: 322 | fieldDefn = ogr.FieldDefn(f, ogr.OFTString) 323 | layer.CreateField(fieldDefn) 324 | field_names = [layer_defn.GetFieldDefn(i).GetName() for i in range(layer_defn.GetFieldCount())] 325 | 326 | print '\n'+str(time.time() - start)+'s - Adding links and creating the Edgelist...' 327 | sys.stdout.flush() 328 | 329 | links = (testando.groupby(['Group']).groups) 330 | links1 = (testando.groupby(['Group','continuidade']).groups) 331 | cont = 0 332 | for link in range(len(links)): 333 | try: 334 | length_0 = len(links1[link,0]) 335 | except: 336 | length_0 = -1 337 | try: 338 | length_1 = len(links1[link,1]) 339 | except: 340 | length_1 = -1 341 | try: 342 | length_2 = len(links1[link,2]) 343 | except: 344 | length_2 = -1 345 | if length_0 == 1 and length_2 == 1: 346 | start_node, end_node = min(links1[link,0][0],links1[link,2][0]), max(links1[link,0][0],links1[link,2][0]) 347 | add_link(start_node,end_node,cont) 348 | edgelist = edgelist.append(Edgelist(start_node,end_node,cont+1,distance(start_node,end_node)),ignore_index=True) 349 | cont += 1 350 | elif length_0 == -1: 351 | for i in range(length_2-1): 352 | start_node, end_node = links1[link,2][i], links1[link,2][i+1] 353 | add_link(start_node,end_node,cont) 354 | edgelist = edgelist.append(Edgelist(start_node,end_node,cont+1,distance(start_node,end_node)),ignore_index=True) 355 | cont += 1 356 | elif length_2 == -1: 357 | start_node, end_node = links1[link,0][0], links1[link,0][1] 358 | add_link(start_node,end_node,cont) 359 | edgelist = edgelist.append(Edgelist(start_node,end_node,cont+1,distance(start_node,end_node)),ignore_index=True) 360 | cont += 1 361 | else: 362 | if length_0 > 1: 363 | start_node, end_node = links1[link,0][0], links1[link,2][0] 364 | add_link(start_node,end_node,cont) 365 | edgelist = edgelist.append(Edgelist(start_node,end_node,cont+1,distance(start_node,end_node)),ignore_index=True) 366 | cont += 1 367 | for i in range(length_2-1): 368 | start_node, end_node = links1[link,2][i], links1[link,2][i+1] 369 | add_link(start_node,end_node,cont) 370 | edgelist = edgelist.append(Edgelist(start_node,end_node,cont+1,distance(start_node,end_node)),ignore_index=True) 371 | cont += 1 372 | start_node, end_node = links1[link,2][-1], links1[link,0][1] 373 | add_link(start_node,end_node,cont) 374 | edgelist = edgelist.append(Edgelist(start_node,end_node,cont+1,distance(start_node,end_node)),ignore_index=True) 375 | cont += 1 376 | else: 377 | if links1[link,0][0] < links1[link,2][0]: 378 | start_node, end_node = links1[link,0][0], links1[link,2][0] 379 | add_link(start_node,end_node,cont) 380 | edgelist = edgelist.append(Edgelist(start_node,end_node,cont+1,distance(start_node,end_node)),ignore_index=True) 381 | cont += 1 382 | for i in range(length_2 - 1): 383 | start_node, end_node = links1[link,2][i], links1[link,2][i+1] 384 | add_link(start_node,end_node,cont) 385 | edgelist = edgelist.append(Edgelist(start_node,end_node,cont+1,distance(start_node,end_node)),ignore_index=True) 386 | cont += 1 387 | else: 388 | for i in range(length_2 - 1): 389 | start_node, end_node = links1[link,2][i], links1[link,2][i+1] 390 | add_link(start_node,end_node,cont) 391 | edgelist = edgelist.append(Edgelist(start_node,end_node,cont+1,distance(start_node,end_node)),ignore_index=True) 392 | cont += 1 393 | start_node, end_node = links1[link,2][-1], links1[link,0][0] 394 | add_link(start_node,end_node,cont) 395 | edgelist = edgelist.append(Edgelist(start_node,end_node,cont+1,distance(start_node,end_node)),ignore_index=True) 396 | cont += 1 397 | shapeData = None 398 | 399 | print '\n'+str(time.time() - start)+'s - Links and Edgelist are done!' 400 | sys.stdout.flush() 401 | 402 | print '\n'+str(time.time() - start)+'s - Saving Edgelist...' 403 | sys.stdout.flush() 404 | 405 | edgelist.to_csv('C:\Users\Rodrigo\Desktop\GIStool\output\Chicago_cut_edgelist.csv',index=False) 406 | 407 | wb = QgsVectorLayer('C:\Users\Rodrigo\Desktop\GIStool\output\Chicago_cut_edges.shp', 'Chicago_cut_edges', 'ogr') 408 | QgsMapLayerRegistry.instance().addMapLayer(wb) 409 | wb = QgsVectorLayer('C:\Users\Rodrigo\Desktop\GIStool\output\Chicago_cut_nodes.shp', 'Chicago_cut_nodes', 'ogr') 410 | QgsMapLayerRegistry.instance().addMapLayer(wb) 411 | print '\n'+str(time.time() - start)+'s - You are all set.' 412 | sys.stdout.flush() 413 | -------------------------------------------------------------------------------- /Python/v1.10/GISF2E_python_v1.10.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | ''' 3 | GISF2E python - v1.10 4 | 5 | To cite, please use: Karduni, A., Kermanshah, A., & Derrible, S., 2016 “A protocol to convert spatial polyline data to network formats and applications to world urban road networks”, Scientific Data, 3:160046 6 | The article is available at: http://www.nature.com/articles/sdata201646 7 | 8 | Main authors: Felipe Macena, Rodrigo Marinho 9 | ''' 10 | 11 | 12 | #import all libraries required 13 | import shapefile 14 | import pandas as pd 15 | from pyproj import Proj 16 | import osgeo.ogr, osgeo.osr 17 | from osgeo import ogr 18 | import sys 19 | import os 20 | import time 21 | from math import pi, sin, cos, tan, sqrt 22 | 23 | def LLtoUTM(ReferenceEllipsoid, Lat, Long, zone = None): 24 | """converts lat/long to UTM coords. Equations from USGS Bulletin 1532 25 | East Longitudes are positive, West longitudes are negative. 26 | North latitudes are positive, South latitudes are negative 27 | Lat and Long are in decimal degrees 28 | Written by Chuck Gantz- chuck.gantz@globalstar.com""" 29 | 30 | a = _ellipsoid[ReferenceEllipsoid][_EquatorialRadius] 31 | eccSquared = _ellipsoid[ReferenceEllipsoid][_eccentricitySquared] 32 | k0 = 0.9996 33 | 34 | #Make sure the longitude is between -180.00 .. 179.9 35 | LongTemp = (Long+180)-int((Long+180)/360)*360-180 # -180.00 .. 179.9 36 | 37 | LatRad = Lat*_deg2rad 38 | LongRad = LongTemp*_deg2rad 39 | 40 | if zone is None: 41 | ZoneNumber = int((LongTemp + 180)/6) + 1 42 | else: 43 | ZoneNumber = zone 44 | 45 | if Lat >= 56.0 and Lat < 64.0 and LongTemp >= 3.0 and LongTemp < 12.0: 46 | ZoneNumber = 32 47 | 48 | # Special zones for Svalbard 49 | if Lat >= 72.0 and Lat < 84.0: 50 | if LongTemp >= 0.0 and LongTemp < 9.0:ZoneNumber = 31 51 | elif LongTemp >= 9.0 and LongTemp < 21.0: ZoneNumber = 33 52 | elif LongTemp >= 21.0 and LongTemp < 33.0: ZoneNumber = 35 53 | elif LongTemp >= 33.0 and LongTemp < 42.0: ZoneNumber = 37 54 | 55 | LongOrigin = (ZoneNumber - 1)*6 - 180 + 3 #+3 puts origin in middle of zone 56 | LongOriginRad = LongOrigin * _deg2rad 57 | 58 | #compute the UTM Zone from the latitude and longitude 59 | UTMZone = "%d%c" % (ZoneNumber, _UTMLetterDesignator(Lat)) 60 | 61 | eccPrimeSquared = (eccSquared)/(1-eccSquared) 62 | N = a/sqrt(1-eccSquared*sin(LatRad)*sin(LatRad)) 63 | T = tan(LatRad)*tan(LatRad) 64 | C = eccPrimeSquared*cos(LatRad)*cos(LatRad) 65 | A = cos(LatRad)*(LongRad-LongOriginRad) 66 | 67 | M = a*((1 68 | - eccSquared/4 69 | - 3*eccSquared*eccSquared/64 70 | - 5*eccSquared*eccSquared*eccSquared/256)*LatRad 71 | - (3*eccSquared/8 72 | + 3*eccSquared*eccSquared/32 73 | + 45*eccSquared*eccSquared*eccSquared/1024)*sin(2*LatRad) 74 | + (15*eccSquared*eccSquared/256 + 45*eccSquared*eccSquared*eccSquared/1024)*sin(4*LatRad) 75 | - (35*eccSquared*eccSquared*eccSquared/3072)*sin(6*LatRad)) 76 | 77 | UTMEasting = (k0*N*(A+(1-T+C)*A*A*A/6 78 | + (5-18*T+T*T+72*C-58*eccPrimeSquared)*A*A*A*A*A/120) 79 | + 500000.0) 80 | 81 | UTMNorthing = (k0*(M+N*tan(LatRad)*(A*A/2+(5-T+9*C+4*C*C)*A*A*A*A/24 82 | + (61 83 | -58*T 84 | +T*T 85 | +600*C 86 | -330*eccPrimeSquared)*A*A*A*A*A*A/720))) 87 | if Lat < 0: 88 | UTMNorthing = UTMNorthing + 10000000.0; #10000000 meter offset for southern hemisphere 89 | return (UTMZone, UTMEasting, UTMNorthing) 90 | 91 | 92 | def _UTMLetterDesignator(Lat): 93 | """This routine determines the correct UTM letter designator for the given 94 | latitude """ 95 | if Lat >= 0: return 'N' 96 | else: return 'S' 97 | 98 | def add_link(start,end,cont): 99 | pontos = [] 100 | pontos.append(osgeo.ogr.Geometry(osgeo.ogr.wkbLineString)) 101 | lista = list([testando['coordinates_utm'][start:end+1]][0]) 102 | for a in lista: 103 | pontos[-1].AddPoint(a[0],a[1]) 104 | featureIndex = cont 105 | feature = osgeo.ogr.Feature(layer_defn) 106 | feature.SetGeometry(pontos[-1]) 107 | feature.SetFID(featureIndex) 108 | layer.CreateFeature(feature) 109 | for name in range(len(field_names)): 110 | if field_names[name] != 'Edge' and field_names[name] != 'Length': 111 | feature.SetField(name, str(testando[field_names[name]][start_node])) 112 | elif field_names[name] == 'Edge': 113 | feature.SetField(name, cont+1) 114 | elif field_names[name] == 'Length': 115 | feature.SetField(name, distance(start,end)) 116 | layer.SetFeature(feature) 117 | def add_field(field_names,featureIndex): 118 | for name in range(len(field_names)): 119 | if field_names[name] == 'coord_x' or field_names[name] == 'coord_y': 120 | name_aux = 'coordinates_utm' 121 | else: 122 | name_aux = field_names[name] 123 | feature = layer.GetFeature(featureIndex) #lets get the first feature 124 | if field_names[name] == 'coord_x': 125 | feature.SetField(name,testando_nodes[name_aux][featureIndex][0]) 126 | elif field_names[name] == 'coord_y': 127 | feature.SetField(name,testando_nodes[name_aux][featureIndex][1]) 128 | else: 129 | feature.SetField(name,int(testando_nodes[name_aux][featureIndex])) 130 | layer.SetFeature(feature) #now make the change permanent 131 | def distance(start,end): 132 | dist = 0 133 | for u in range(start,end): 134 | dist = dist + ((testando['coordinates_utm'][u][0] - testando['coordinates_utm'][u+1][0])**2 + (testando['coordinates_utm'][u][1] - testando['coordinates_utm'][u+1][1])**2)**0.5 135 | return dist 136 | def Edgelist(start,end,edge,length): 137 | global testando, columns 138 | datasheet = pd.DataFrame([[testando['coordinates_utm'][start][0],testando['coordinates_utm'][start][1],testando['coordinates_utm'][end][0],testando['coordinates_utm'][end][1],testando['Node'][start],testando['Node'][end],edge,length]],columns = columns, index = None) 139 | return datasheet 140 | 141 | _deg2rad, _rad2deg, _EquatorialRadius, _eccentricitySquared = pi / 180.0, 180.0 / pi, 2, 3 142 | 143 | _ellipsoid = [ 144 | # id, Ellipsoid name, Equatorial Radius, square of eccentricity 145 | # first once is a placeholder only, To allow array indices to match id numbers 146 | [ -1, "Placeholder", 0, 0], 147 | [ 1, "Airy", 6377563, 0.00667054], 148 | [ 2, "Australian National", 6378160, 0.006694542], 149 | [ 3, "Bessel 1841", 6377397, 0.006674372], 150 | [ 4, "Bessel 1841 (Nambia] ", 6377484, 0.006674372], 151 | [ 5, "Clarke 1866", 6378206, 0.006768658], 152 | [ 6, "Clarke 1880", 6378249, 0.006803511], 153 | [ 7, "Everest", 6377276, 0.006637847], 154 | [ 8, "Fischer 1960 (Mercury] ", 6378166, 0.006693422], 155 | [ 9, "Fischer 1968", 6378150, 0.006693422], 156 | [ 10, "GRS 1967", 6378160, 0.006694605], 157 | [ 11, "GRS 1980", 6378137, 0.00669438], 158 | [ 12, "Helmert 1906", 6378200, 0.006693422], 159 | [ 13, "Hough", 6378270, 0.00672267], 160 | [ 14, "International", 6378388, 0.00672267], 161 | [ 15, "Krassovsky", 6378245, 0.006693422], 162 | [ 16, "Modified Airy", 6377340, 0.00667054], 163 | [ 17, "Modified Everest", 6377304, 0.006637847], 164 | [ 18, "Modified Fischer 1960", 6378155, 0.006693422], 165 | [ 19, "South American 1969", 6378160, 0.006694542], 166 | [ 20, "WGS 60", 6378165, 0.006693422], 167 | [ 21, "WGS 66", 6378145, 0.006694542], 168 | [ 22, "WGS-72", 6378135, 0.006694318], 169 | [ 23, "WGS-84", 6378137, 0.00669438] 170 | ] 171 | 172 | path = os.getcwd() 173 | 174 | while True: 175 | option = raw_input('Do you want to use input/output folders (1) or type your files addresses (2)? Option 1 or 2: ') 176 | if option == '1': 177 | sf = shapefile.Reader(path+'/input/'+os.listdir(path+'/input/')[0]) 178 | path_nodes = path+ '/output/Nodes_'+os.listdir(path+'/input/')[0][:-4]+'.shp' 179 | path_links = path+ '/output/Edges_'+os.listdir(path+'/input/')[0][:-4]+'.shp' 180 | path_edgelist = path+ '/output/Edgelist_'+os.listdir(path+'/input/')[0][:-4]+'.csv' 181 | break 182 | elif option == '2': 183 | sf = shapefile.Reader(raw_input("Please type where your file is located (Example:C:\Users\UIC\Documents\Shape_File.shp): ")) 184 | path_nodes = raw_input("Please type where you want the nodes shapefile to be located(Example:C:\Users\UIC\Documents\Nodes_File.shp): ") 185 | path_links = raw_input("Please type where you want the links shapefile to be located(Example:C:\Users\UIC\Documents\Links_File.shp): ") 186 | path_edgelist = raw_input("Please type where you want the Edgelist to be located(Example:C:\Users\UIC\Documents\Edgelist.csv): ") 187 | break 188 | 189 | osm = raw_input("Is your root file from Open Street Maps (Y/N)? ") 190 | osm = osm.lower() 191 | start = time.time() 192 | print '\n'+str(time.time() - start)+'s - Reading and arranging the root file information...' 193 | sys.stdout.flush() 194 | 195 | #dividing each link by their osm_id information (osm_maps) or id information (non-osm_maps) 196 | shapes = sf.shapes() 197 | fields=sf.fields 198 | fields = [field[0] for field in sf.fields[1:]] 199 | 200 | 201 | if 'TYPE' in fields or 'Type' in fields: 202 | try: 203 | fields[fields.index('TYPE')] = 'type' 204 | except: 205 | fields[fields.index('Type')] = 'type' 206 | 207 | heads = [x.upper() if x in ['Group','coordinates','primeiro','coordinates_utm','Node'] else x for x in [sf.shapeRecords()[0].shape.__geo_interface__.keys()[0]]+fields] + ['Group','coordinates','primeiro','coordinates_utm','Node'] 208 | type_ = False 209 | if 'type' in map(lambda x: x.lower(),fields): 210 | heads.remove('type') 211 | heads.insert(3,'type 1') 212 | type_ = True 213 | 214 | dicti = {} 215 | for i in range(len(heads)-1): 216 | dicti[heads[i]] = [] 217 | 218 | testando = pd.DataFrame() 219 | 220 | n = 0 221 | for feature in sf.shapeRecords(): 222 | geom = feature.shape.__geo_interface__ 223 | atr = dict(zip(fields, feature.record)) 224 | if type_: 225 | atr['type 1'] = atr.pop('type') 226 | geom.update(atr) 227 | geom['Group'] = n 228 | j = geom['coordinates'] 229 | for i in range(len(heads)-4): 230 | u = [geom[heads[i]]] 231 | for y in range(len(j)): 232 | dicti[heads[i]].append(u[0]) 233 | dicti['coordinates'] = list(dicti['coordinates']) + list(j) 234 | dicti['primeiro'] = dicti['primeiro'] + [1] + [0]*(len(j) - 2) + [1] 235 | n += 1 236 | if ((n-1) % 800) == 0: #The number 800 was chosen because it's shown the fastest results in a range of numbers from 100 to 1000 237 | dicti['coordinates_utm'] = None 238 | dicti['Node'] = None 239 | testando = pd.concat([testando,pd.DataFrame.from_dict(dicti)]) 240 | for i in range(len(heads)-1): 241 | dicti[heads[i]] = [] 242 | if ((n-1) % 800) != 0: 243 | dicti['coordinates_utm'] = None 244 | dicti['Node'] = None 245 | testando = pd.concat([testando,pd.DataFrame.from_dict(dicti)],ignore_index=True) 246 | 247 | utm_zone = LLtoUTM(23, testando['coordinates'][0][1], testando['coordinates'][0][0])[0] 248 | p = Proj(proj='utm',zone=int(utm_zone[:-1]),ellps='WGS84') 249 | 250 | testando['coordinates_utm'] = [p(testando['coordinates'][k][0],testando['coordinates'][k][1]) for k in range(len(testando['coordinates']))] 251 | testando['Node'] = None 252 | dicti = None 253 | geom = None 254 | atr = None 255 | 256 | #test before creating nodes for OSM maps 257 | a= testando.groupby('coordinates_utm').groups 258 | testando['continuidade'] = 1 259 | testando.ix[testando.primeiro == 1,'continuidade'] = 0; testando 260 | b1 = map(lambda x: (testando['coordinates_utm'][x],[x]),testando.groupby('continuidade').groups[0]) 261 | if osm == 'y'or osm == 'yes' or osm == 'y': 262 | b = filter(lambda x: (testando['bridge'][x[1][0]] == testando['bridge'][int(x[1][1])]) or (testando['tunnel'][x[1][0]] == testando['tunnel'][int(x[1][1])]), filter(lambda x: len(x[1]) >= 2, zip(a.keys(),a.values()))) 263 | else: # no test for non-OSM maps 264 | b = filter(lambda x: len(x[1]) >= 2, zip(a.keys(),a.values())) 265 | for h in (b): 266 | for k in range(len(h[1])): 267 | # testando['continuidade'][h[1][k]] = 2 268 | testando.loc[[h[1][k]],'continuidade'] = 2 269 | b = b + b1 270 | points = sorted(testando.groupby('continuidade').groups[0] + testando.groupby('continuidade').groups[2]) 271 | testando.loc[points,'Node'] = range(1,len(points)+1) 272 | 273 | print '\n'+str(time.time() - start)+'s - Reading and arranging are done!' 274 | sys.stdout.flush() 275 | 276 | #creating nodes shapefile 277 | spatialReference = osgeo.osr.SpatialReference() #will create a spatial reference locally to tell the system what the reference will be stored 278 | spatialReference.ImportFromProj4('+proj=utm +zone='+utm_zone+' ellps=WGS84 +datum=WGS84 +units=m') #here we define the utm zone as well as the world geodetic system(wgs) 279 | driver = osgeo.ogr.GetDriverByName('ESRI Shapefile') # will select the driver for our shp-file creation. 280 | shapeData = driver.CreateDataSource(path_nodes) #so there we will store our data 281 | layer = shapeData.CreateLayer('Nodes', spatialReference, osgeo.ogr.wkbPoint) #this will create a corresponding layer for our data with given spatial information 282 | layer_defn = layer.GetLayerDefn() # gets parameters of the current shapefile 283 | point = osgeo.ogr.Geometry(osgeo.ogr.wkbPoint) 284 | 285 | new_field = ogr.FieldDefn('Node', ogr.OFTInteger) #add a field 'Node' stored as integer 286 | layer.CreateField(new_field) #self explaining 287 | new_field = ogr.FieldDefn('coord_x', ogr.OFTReal) #add a field 'coord_x' stored as real 288 | layer.CreateField(new_field) #self explaining 289 | new_field = ogr.FieldDefn('coord_y', ogr.OFTReal) #add a field 'coord_y' stored as real 290 | layer.CreateField(new_field) #self explaining 291 | if osm == 'y'or osm == 'yes' or osm == 'y': #for osm maps 292 | new_field = ogr.FieldDefn('bridge', ogr.OFTInteger) #we will create a new field called bridge as integer 293 | layer.CreateField(new_field) #self explaining 294 | new_field = ogr.FieldDefn('tunnel', ogr.OFTInteger) #and another field called 'tunnel' stored as integer 295 | layer.CreateField(new_field) #self explaining 296 | field_names = [layer_defn.GetFieldDefn(i).GetName() for i in range(layer_defn.GetFieldCount())] 297 | aux_b, featureIndex = list(zip(*b)[0]), 0 298 | 299 | testando_nodes = testando[testando.continuidade != 1].drop_duplicates(['coordinates_utm']) 300 | testando_nodes.index = range(len(testando_nodes)) 301 | 302 | t, aux_b, a, b, b1, points = None, None, None, None, None, None 303 | 304 | print '\n'+str(time.time() - start)+'s - Adding nodes...' 305 | sys.stdout.flush() 306 | 307 | for featureIndex in range(len(testando_nodes)): 308 | point.AddPoint(testando_nodes['coordinates_utm'][featureIndex][0],testando_nodes['coordinates_utm'][featureIndex][1]) #create a new point at given coordinates 309 | feature = osgeo.ogr.Feature(layer_defn) 310 | feature.SetGeometry(point) 311 | feature.SetFID(featureIndex) 312 | layer.CreateFeature(feature) 313 | add_field(field_names,featureIndex) 314 | shapeData = None 315 | 316 | new_b = None 317 | testando_nodes = None 318 | 319 | print '\n'+str(time.time() - start)+'s - Nodes are done!' 320 | sys.stdout.flush() 321 | 322 | #Cut the raw links and Creat the edges shapefile 323 | columns = ['X_Coord_Start','Y_Coord_Start','X_Coord_End','Y_Coord_End','Start_Node','End_Node','Edge','Length'] 324 | edgelist = pd.DataFrame(columns = columns, index=None) 325 | 326 | #creating edges shapefile 327 | spatialReference = osgeo.osr.SpatialReference() #will create a spatial reference locally to tell the system what the reference will be stored 328 | #here you can change the utm zone depending on where the map is from 329 | spatialReference.ImportFromProj4('+proj=utm +zone='+utm_zone+' ellps=WGS84 +datum=WGS84 +units=m') #here we define the utm zone as well as the world geodetic system(wgs) 330 | driver = osgeo.ogr.GetDriverByName('ESRI Shapefile') # will select the driver for our shp-file creation. 331 | shapeData = driver.CreateDataSource(path_links) #so there we will store our data 332 | layer = shapeData.CreateLayer('linestrings', spatialReference, osgeo.ogr.wkbLineString) #this will create a corresponding layer for our data with given spatial information. 333 | layer_defn = layer.GetLayerDefn() # gets parameters of the current shapefile 334 | 335 | for f in ['Edge']+heads[:-5]+['Length']: 336 | fieldDefn = ogr.FieldDefn(f, ogr.OFTString) 337 | layer.CreateField(fieldDefn) 338 | field_names = [layer_defn.GetFieldDefn(i).GetName() for i in range(layer_defn.GetFieldCount())] 339 | 340 | print '\n'+str(time.time() - start)+'s - Adding links and creating the Edgelist...' 341 | sys.stdout.flush() 342 | 343 | #several tests in order to create the edges shapefile correctly 344 | links = (testando.groupby(['Group']).groups) 345 | links1 = (testando.groupby(['Group','continuidade']).groups) 346 | cont = 0 347 | for link in range(len(links)): 348 | try: 349 | length_0 = len(links1[link,0]) 350 | except: 351 | length_0 = -1 352 | try: 353 | length_1 = len(links1[link,1]) 354 | except: 355 | length_1 = -1 356 | try: 357 | length_2 = len(links1[link,2]) 358 | except: 359 | length_2 = -1 360 | if length_0 == 1 and length_2 == 1: 361 | start_node, end_node = min(links1[link,0][0],links1[link,2][0]), max(links1[link,0][0],links1[link,2][0]) 362 | add_link(start_node,end_node,cont) 363 | edgelist = edgelist.append(Edgelist(start_node,end_node,cont+1,distance(start_node,end_node)),ignore_index=True) 364 | cont += 1 365 | elif length_0 == -1: 366 | for i in range(length_2-1): 367 | start_node, end_node = links1[link,2][i], links1[link,2][i+1] 368 | add_link(start_node,end_node,cont) 369 | edgelist = edgelist.append(Edgelist(start_node,end_node,cont+1,distance(start_node,end_node)),ignore_index=True) 370 | cont += 1 371 | elif length_2 == -1: 372 | start_node, end_node = links1[link,0][0], links1[link,0][1] 373 | add_link(start_node,end_node,cont) 374 | edgelist = edgelist.append(Edgelist(start_node,end_node,cont+1,distance(start_node,end_node)),ignore_index=True) 375 | cont += 1 376 | else: 377 | if length_0 > 1: 378 | start_node, end_node = links1[link,0][0], links1[link,2][0] 379 | add_link(start_node,end_node,cont) 380 | edgelist = edgelist.append(Edgelist(start_node,end_node,cont+1,distance(start_node,end_node)),ignore_index=True) 381 | cont += 1 382 | for i in range(length_2-1): 383 | start_node, end_node = links1[link,2][i], links1[link,2][i+1] 384 | add_link(start_node,end_node,cont) 385 | edgelist = edgelist.append(Edgelist(start_node,end_node,cont+1,distance(start_node,end_node)),ignore_index=True) 386 | cont += 1 387 | start_node, end_node = links1[link,2][-1], links1[link,0][1] 388 | add_link(start_node,end_node,cont) 389 | edgelist = edgelist.append(Edgelist(start_node,end_node,cont+1,distance(start_node,end_node)),ignore_index=True) 390 | cont += 1 391 | else: 392 | if links1[link,0][0] < links1[link,2][0]: 393 | start_node, end_node = links1[link,0][0], links1[link,2][0] 394 | add_link(start_node,end_node,cont) 395 | edgelist = edgelist.append(Edgelist(start_node,end_node,cont+1,distance(start_node,end_node)),ignore_index=True) 396 | cont += 1 397 | for i in range(length_2 - 1): 398 | start_node, end_node = links1[link,2][i], links1[link,2][i+1] 399 | add_link(start_node,end_node,cont) 400 | edgelist = edgelist.append(Edgelist(start_node,end_node,cont+1,distance(start_node,end_node)),ignore_index=True) 401 | cont += 1 402 | else: 403 | for i in range(length_2 - 1): 404 | start_node, end_node = links1[link,2][i], links1[link,2][i+1] 405 | add_link(start_node,end_node,cont) 406 | edgelist = edgelist.append(Edgelist(start_node,end_node,cont+1,distance(start_node,end_node)),ignore_index=True) 407 | cont += 1 408 | start_node, end_node = links1[link,2][-1], links1[link,0][0] 409 | add_link(start_node,end_node,cont) 410 | edgelist = edgelist.append(Edgelist(start_node,end_node,cont+1,distance(start_node,end_node)),ignore_index=True) 411 | cont += 1 412 | shapeData = None 413 | 414 | print '\n'+str(time.time() - start)+'s - Links and Edgelist are done!' 415 | sys.stdout.flush() 416 | 417 | print '\n'+str(time.time() - start)+'s - Saving Edgelist...' 418 | sys.stdout.flush() 419 | 420 | #saving the edgelist to a 'csv. file' 421 | edgelist.to_csv(path_edgelist,index=False) 422 | 423 | print '\n'+str(time.time() - start)+'s - You are all set.' 424 | sys.stdout.flush() 425 | -------------------------------------------------------------------------------- /Python/v1.00/GISF2E_python_v1.00.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | ''' 3 | GISF2E python - v1.00 - Note that the code was developed with pandas version 0.14.1 and it runs much faster for this version than with later versions. 4 | 5 | To cite, please use: Karduni, A., Kermanshah, A., & Derrible, S., 2016 “A protocol to convert spatial polyline data to network formats and applications to world urban road networks”, Scientific Data, 3:160046 6 | The article is available at: http://www.nature.com/articles/sdata201646 7 | 8 | Main authors: Felipe Macena, Rodrigo Marinho 9 | ''' 10 | 11 | #import all libraries required 12 | import shapefile 13 | import pandas as pd 14 | from pyproj import Proj 15 | import osgeo.ogr, osgeo.osr 16 | from osgeo import ogr 17 | import sys 18 | import os 19 | import time 20 | from math import pi, sin, cos, tan, sqrt 21 | 22 | def LLtoUTM(ReferenceEllipsoid, Lat, Long, zone = None): 23 | """converts lat/long to UTM coords. Equations from USGS Bulletin 1532 24 | East Longitudes are positive, West longitudes are negative. 25 | North latitudes are positive, South latitudes are negative 26 | Lat and Long are in decimal degrees 27 | Written by Chuck Gantz- chuck.gantz@globalstar.com""" 28 | 29 | a = _ellipsoid[ReferenceEllipsoid][_EquatorialRadius] 30 | eccSquared = _ellipsoid[ReferenceEllipsoid][_eccentricitySquared] 31 | k0 = 0.9996 32 | 33 | #Make sure the longitude is between -180.00 .. 179.9 34 | LongTemp = (Long+180)-int((Long+180)/360)*360-180 # -180.00 .. 179.9 35 | 36 | LatRad = Lat*_deg2rad 37 | LongRad = LongTemp*_deg2rad 38 | 39 | if zone is None: 40 | ZoneNumber = int((LongTemp + 180)/6) + 1 41 | else: 42 | ZoneNumber = zone 43 | 44 | if Lat >= 56.0 and Lat < 64.0 and LongTemp >= 3.0 and LongTemp < 12.0: 45 | ZoneNumber = 32 46 | 47 | # Special zones for Svalbard 48 | if Lat >= 72.0 and Lat < 84.0: 49 | if LongTemp >= 0.0 and LongTemp < 9.0:ZoneNumber = 31 50 | elif LongTemp >= 9.0 and LongTemp < 21.0: ZoneNumber = 33 51 | elif LongTemp >= 21.0 and LongTemp < 33.0: ZoneNumber = 35 52 | elif LongTemp >= 33.0 and LongTemp < 42.0: ZoneNumber = 37 53 | 54 | LongOrigin = (ZoneNumber - 1)*6 - 180 + 3 #+3 puts origin in middle of zone 55 | LongOriginRad = LongOrigin * _deg2rad 56 | 57 | #compute the UTM Zone from the latitude and longitude 58 | UTMZone = "%d%c" % (ZoneNumber, _UTMLetterDesignator(Lat)) 59 | 60 | eccPrimeSquared = (eccSquared)/(1-eccSquared) 61 | N = a/sqrt(1-eccSquared*sin(LatRad)*sin(LatRad)) 62 | T = tan(LatRad)*tan(LatRad) 63 | C = eccPrimeSquared*cos(LatRad)*cos(LatRad) 64 | A = cos(LatRad)*(LongRad-LongOriginRad) 65 | 66 | M = a*((1 67 | - eccSquared/4 68 | - 3*eccSquared*eccSquared/64 69 | - 5*eccSquared*eccSquared*eccSquared/256)*LatRad 70 | - (3*eccSquared/8 71 | + 3*eccSquared*eccSquared/32 72 | + 45*eccSquared*eccSquared*eccSquared/1024)*sin(2*LatRad) 73 | + (15*eccSquared*eccSquared/256 + 45*eccSquared*eccSquared*eccSquared/1024)*sin(4*LatRad) 74 | - (35*eccSquared*eccSquared*eccSquared/3072)*sin(6*LatRad)) 75 | 76 | UTMEasting = (k0*N*(A+(1-T+C)*A*A*A/6 77 | + (5-18*T+T*T+72*C-58*eccPrimeSquared)*A*A*A*A*A/120) 78 | + 500000.0) 79 | 80 | UTMNorthing = (k0*(M+N*tan(LatRad)*(A*A/2+(5-T+9*C+4*C*C)*A*A*A*A/24 81 | + (61 82 | -58*T 83 | +T*T 84 | +600*C 85 | -330*eccPrimeSquared)*A*A*A*A*A*A/720))) 86 | if Lat < 0: 87 | UTMNorthing = UTMNorthing + 10000000.0; #10000000 meter offset for southern hemisphere 88 | return (UTMZone, UTMEasting, UTMNorthing) 89 | 90 | 91 | def _UTMLetterDesignator(Lat): 92 | """This routine determines the correct UTM letter designator for the given 93 | latitude """ 94 | if Lat >= 0: return 'N' 95 | else: return 'S' 96 | 97 | def add_link(start,end,cont): 98 | pontos = [] 99 | pontos.append(osgeo.ogr.Geometry(osgeo.ogr.wkbLineString)) 100 | lista = list([testando['coordinates_utm'][start:end+1]][0]) 101 | for a in lista: 102 | pontos[-1].AddPoint(a[0],a[1]) 103 | featureIndex = cont 104 | feature = osgeo.ogr.Feature(layer_defn) 105 | feature.SetGeometry(pontos[-1]) 106 | feature.SetFID(featureIndex) 107 | layer.CreateFeature(feature) 108 | for name in range(len(field_names)): 109 | if field_names[name] != 'Edge' and field_names[name] != 'Length': 110 | feature.SetField(name, str(testando[field_names[name]][start_node])) 111 | elif field_names[name] == 'Edge': 112 | feature.SetField(name, cont+1) 113 | elif field_names[name] == 'Length': 114 | feature.SetField(name, distance(start,end)) 115 | layer.SetFeature(feature) 116 | def add_field(field_names,featureIndex): 117 | for name in range(len(field_names)): 118 | if field_names[name] == 'coord_x' or field_names[name] == 'coord_y': 119 | name_aux = 'coordinates_utm' 120 | else: 121 | name_aux = field_names[name] 122 | feature = layer.GetFeature(featureIndex) #lets get the first feature 123 | if field_names[name] == 'coord_x': 124 | feature.SetField(name,testando_nodes[name_aux][featureIndex][0]) 125 | elif field_names[name] == 'coord_y': 126 | feature.SetField(name,testando_nodes[name_aux][featureIndex][1]) 127 | else: 128 | feature.SetField(name,int(testando_nodes[name_aux][featureIndex])) 129 | layer.SetFeature(feature) #now make the change permanent 130 | def distance(start,end): 131 | dist = 0 132 | for u in range(start,end): 133 | dist = dist + ((testando['coordinates_utm'][u][0] - testando['coordinates_utm'][u+1][0])**2 + (testando['coordinates_utm'][u][1] - testando['coordinates_utm'][u+1][1])**2)**0.5 134 | return dist 135 | def Edgelist(start,end,edge,length): 136 | global testando, columns 137 | datasheet = pd.DataFrame([[testando['coordinates_utm'][start][0],testando['coordinates_utm'][start][1],testando['coordinates_utm'][end][0],testando['coordinates_utm'][end][1],testando['Node'][start],testando['Node'][end],edge,length]],columns = columns, index = None) 138 | return datasheet 139 | 140 | _deg2rad, _rad2deg, _EquatorialRadius, _eccentricitySquared = pi / 180.0, 180.0 / pi, 2, 3 141 | 142 | _ellipsoid = [ 143 | # id, Ellipsoid name, Equatorial Radius, square of eccentricity 144 | # first once is a placeholder only, To allow array indices to match id numbers 145 | [ -1, "Placeholder", 0, 0], 146 | [ 1, "Airy", 6377563, 0.00667054], 147 | [ 2, "Australian National", 6378160, 0.006694542], 148 | [ 3, "Bessel 1841", 6377397, 0.006674372], 149 | [ 4, "Bessel 1841 (Nambia] ", 6377484, 0.006674372], 150 | [ 5, "Clarke 1866", 6378206, 0.006768658], 151 | [ 6, "Clarke 1880", 6378249, 0.006803511], 152 | [ 7, "Everest", 6377276, 0.006637847], 153 | [ 8, "Fischer 1960 (Mercury] ", 6378166, 0.006693422], 154 | [ 9, "Fischer 1968", 6378150, 0.006693422], 155 | [ 10, "GRS 1967", 6378160, 0.006694605], 156 | [ 11, "GRS 1980", 6378137, 0.00669438], 157 | [ 12, "Helmert 1906", 6378200, 0.006693422], 158 | [ 13, "Hough", 6378270, 0.00672267], 159 | [ 14, "International", 6378388, 0.00672267], 160 | [ 15, "Krassovsky", 6378245, 0.006693422], 161 | [ 16, "Modified Airy", 6377340, 0.00667054], 162 | [ 17, "Modified Everest", 6377304, 0.006637847], 163 | [ 18, "Modified Fischer 1960", 6378155, 0.006693422], 164 | [ 19, "South American 1969", 6378160, 0.006694542], 165 | [ 20, "WGS 60", 6378165, 0.006693422], 166 | [ 21, "WGS 66", 6378145, 0.006694542], 167 | [ 22, "WGS-72", 6378135, 0.006694318], 168 | [ 23, "WGS-84", 6378137, 0.00669438] 169 | ] 170 | 171 | path = os.getcwd() 172 | 173 | while True: 174 | option = raw_input('Do you want to use input/output folders (1) or type your files addresses (2)? Option 1 or 2: ') 175 | if option == '1': 176 | sf = shapefile.Reader(path+'/input/'+os.listdir(path+'/input/')[0]) 177 | path_nodes = path+ '/output/Nodes_'+os.listdir(path+'/input/')[0][:-4]+'.shp' 178 | path_links = path+ '/output/Edges_'+os.listdir(path+'/input/')[0][:-4]+'.shp' 179 | path_edgelist = path+ '/output/Edgelist_'+os.listdir(path+'/input/')[0][:-4]+'.csv' 180 | break 181 | elif option == '2': 182 | sf = shapefile.Reader(raw_input("Please type where your file is located (Example:C:\Users\UIC\Documents\Shape_File.shp): ")) 183 | path_nodes = raw_input("Please type where you want the nodes shapefile to be located(Example:C:\Users\UIC\Documents\Nodes_File.shp): ") 184 | path_links = raw_input("Please type where you want the links shapefile to be located(Example:C:\Users\UIC\Documents\Links_File.shp): ") 185 | path_edgelist = raw_input("Please type where you want the Edgelist to be located(Example:C:\Users\UIC\Documents\Edgelist.csv): ") 186 | break 187 | 188 | osm = raw_input("Is your root file from Open Street Maps (Y/N)? ") 189 | osm = osm.lower() 190 | start = time.time() 191 | print '\n'+str(time.time() - start)+'s - Reading and arranging the root file information...' 192 | sys.stdout.flush() 193 | 194 | #dividing each link by their osm_id information (osm_maps) or id information (non-osm_maps) 195 | shapes = sf.shapes() 196 | fields=sf.fields 197 | fields = [field[0] for field in sf.fields[1:]] 198 | 199 | 200 | if 'TYPE' in fields or 'Type' in fields: 201 | try: 202 | fields[fields.index('TYPE')] = 'type' 203 | except: 204 | fields[fields.index('Type')] = 'type' 205 | 206 | heads = [x.upper() if x in ['Group','coordinates','primeiro','coordinates_utm','Node'] else x for x in [sf.shapeRecords()[0].shape.__geo_interface__.keys()[0]]+fields] + ['Group','coordinates','primeiro','coordinates_utm','Node'] 207 | type_ = False 208 | if 'type' in map(lambda x: x.lower(),fields): 209 | heads.remove('type') 210 | heads.insert(3,'type 1') 211 | type_ = True 212 | 213 | dicti = {} 214 | for i in range(len(heads)-1): 215 | dicti[heads[i]] = [] 216 | 217 | testando = pd.DataFrame() 218 | 219 | n = 0 220 | for feature in sf.shapeRecords(): 221 | geom = feature.shape.__geo_interface__ 222 | atr = dict(zip(fields, feature.record)) 223 | if type_: 224 | atr['type 1'] = atr.pop('type') 225 | geom.update(atr) 226 | geom['Group'] = n 227 | j = geom['coordinates'] 228 | for i in range(len(heads)-4): 229 | u = [geom[heads[i]]] 230 | for y in range(len(j)): 231 | dicti[heads[i]].append(u[0]) 232 | dicti['coordinates'] = list(dicti['coordinates']) + list(j) 233 | dicti['primeiro'] = dicti['primeiro'] + [1] + [0]*(len(j) - 2) + [1] 234 | n += 1 235 | if ((n-1) % 800) == 0: #The number 800 was chosen because it's shown the fastest results in a range of numbers from 100 to 1000 236 | dicti['coordinates_utm'] = None 237 | dicti['Node'] = None 238 | testando = pd.concat([testando,pd.DataFrame.from_dict(dicti)]) 239 | for i in range(len(heads)-1): 240 | dicti[heads[i]] = [] 241 | if ((n-1) % 800) != 0: 242 | dicti['coordinates_utm'] = None 243 | dicti['Node'] = None 244 | testando = pd.concat([testando,pd.DataFrame.from_dict(dicti)],ignore_index=True) 245 | 246 | utm_zone = LLtoUTM(23, testando['coordinates'][0][1], testando['coordinates'][0][0])[0] 247 | p = Proj(proj='utm',zone=int(utm_zone[:-1]),ellps='WGS84') 248 | 249 | testando['coordinates_utm'] = [p(testando['coordinates'][k][0],testando['coordinates'][k][1]) for k in range(len(testando['coordinates']))] 250 | testando['Node'] = None 251 | dicti = None 252 | geom = None 253 | atr = None 254 | 255 | #test before creating nodes for OSM maps 256 | a= testando.groupby('coordinates_utm').groups 257 | testando['continuidade'] = 1 258 | testando.ix[testando.primeiro == 1,'continuidade'] = 0; testando 259 | b1 = map(lambda x: (testando['coordinates_utm'][x],[x]),testando.groupby('continuidade').groups[0]) 260 | if osm == 'y'or osm == 'yes' or osm == 'y': 261 | b = filter(lambda x: (testando['bridge'][x[1][0]] == testando['bridge'][int(x[1][1])]) or (testando['tunnel'][x[1][0]] == testando['tunnel'][int(x[1][1])]), filter(lambda x: len(x[1]) >= 2, zip(a.keys(),a.values()))) 262 | else: # no test for non-OSM maps 263 | b = filter(lambda x: len(x[1]) >= 2, zip(a.keys(),a.values())) 264 | for h in (b): 265 | for k in range(len(h[1])): 266 | testando['continuidade'][h[1][k]] = 2 267 | b = b + b1 268 | points = sorted(testando.groupby('continuidade').groups[0] + testando.groupby('continuidade').groups[2]) 269 | testando['Node'][points] = range(1,len(points)+1) 270 | 271 | print '\n'+str(time.time() - start)+'s - Reading and arranging are done!' 272 | sys.stdout.flush() 273 | 274 | #creating nodes shapefile 275 | spatialReference = osgeo.osr.SpatialReference() #will create a spatial reference locally to tell the system what the reference will be stored 276 | spatialReference.ImportFromProj4('+proj=utm +zone='+utm_zone+' ellps=WGS84 +datum=WGS84 +units=m') #here we define the utm zone as well as the world geodetic system(wgs) 277 | driver = osgeo.ogr.GetDriverByName('ESRI Shapefile') # will select the driver for our shp-file creation. 278 | shapeData = driver.CreateDataSource(path_nodes) #so there we will store our data 279 | layer = shapeData.CreateLayer('Nodes', spatialReference, osgeo.ogr.wkbPoint) #this will create a corresponding layer for our data with given spatial information 280 | layer_defn = layer.GetLayerDefn() # gets parameters of the current shapefile 281 | point = osgeo.ogr.Geometry(osgeo.ogr.wkbPoint) 282 | 283 | new_field = ogr.FieldDefn('Node', ogr.OFTInteger) #add a field 'Node' stored as integer 284 | layer.CreateField(new_field) #self explaining 285 | new_field = ogr.FieldDefn('coord_x', ogr.OFTReal) #add a field 'coord_x' stored as real 286 | layer.CreateField(new_field) #self explaining 287 | new_field = ogr.FieldDefn('coord_y', ogr.OFTReal) #add a field 'coord_y' stored as real 288 | layer.CreateField(new_field) #self explaining 289 | if osm == 'y'or osm == 'yes' or osm == 'y': #for osm maps 290 | new_field = ogr.FieldDefn('bridge', ogr.OFTInteger) #we will create a new field called bridge as integer 291 | layer.CreateField(new_field) #self explaining 292 | new_field = ogr.FieldDefn('tunnel', ogr.OFTInteger) #and another field called 'tunnel' stored as integer 293 | layer.CreateField(new_field) #self explaining 294 | field_names = [layer_defn.GetFieldDefn(i).GetName() for i in range(layer_defn.GetFieldCount())] 295 | aux_b, featureIndex = list(zip(*b)[0]), 0 296 | 297 | testando_nodes = testando[testando.continuidade != 1].drop_duplicates(['coordinates_utm']) 298 | testando_nodes.index = range(len(testando_nodes)) 299 | 300 | t, aux_b, a, b, b1, points = None, None, None, None, None, None 301 | 302 | print '\n'+str(time.time() - start)+'s - Adding nodes...' 303 | sys.stdout.flush() 304 | 305 | for featureIndex in range(len(testando_nodes)): 306 | point.AddPoint(testando_nodes['coordinates_utm'][featureIndex][0],testando_nodes['coordinates_utm'][featureIndex][1]) #create a new point at given coordinates 307 | feature = osgeo.ogr.Feature(layer_defn) 308 | feature.SetGeometry(point) 309 | feature.SetFID(featureIndex) 310 | layer.CreateFeature(feature) 311 | add_field(field_names,featureIndex) 312 | shapeData = None 313 | 314 | new_b = None 315 | testando_nodes = None 316 | 317 | print '\n'+str(time.time() - start)+'s - Nodes are done!' 318 | sys.stdout.flush() 319 | 320 | #Cut the raw links and Creat the edges shapefile 321 | columns = ['X_Coord_Start','Y_Coord_Start','X_Coord_End','Y_Coord_End','Start_Node','End_Node','Edge','Length'] 322 | edgelist = pd.DataFrame(columns = columns, index=None) 323 | 324 | #creating edges shapefile 325 | spatialReference = osgeo.osr.SpatialReference() #will create a spatial reference locally to tell the system what the reference will be stored 326 | #here you can change the utm zone depending on where the map is from 327 | spatialReference.ImportFromProj4('+proj=utm +zone='+utm_zone+' ellps=WGS84 +datum=WGS84 +units=m') #here we define the utm zone as well as the world geodetic system(wgs) 328 | driver = osgeo.ogr.GetDriverByName('ESRI Shapefile') # will select the driver for our shp-file creation. 329 | shapeData = driver.CreateDataSource(path_links) #so there we will store our data 330 | layer = shapeData.CreateLayer('linestrings', spatialReference, osgeo.ogr.wkbLineString) #this will create a corresponding layer for our data with given spatial information. 331 | layer_defn = layer.GetLayerDefn() # gets parameters of the current shapefile 332 | 333 | for f in ['Edge']+heads[:-5]+['Length']: 334 | fieldDefn = ogr.FieldDefn(f, ogr.OFTString) 335 | layer.CreateField(fieldDefn) 336 | field_names = [layer_defn.GetFieldDefn(i).GetName() for i in range(layer_defn.GetFieldCount())] 337 | 338 | print '\n'+str(time.time() - start)+'s - Adding links and creating the Edgelist...' 339 | sys.stdout.flush() 340 | 341 | #several tests in order to create the edges shapefile correctly 342 | links = (testando.groupby(['Group']).groups) 343 | links1 = (testando.groupby(['Group','continuidade']).groups) 344 | cont = 0 345 | for link in range(len(links)): 346 | try: 347 | length_0 = len(links1[link,0]) 348 | except: 349 | length_0 = -1 350 | try: 351 | length_1 = len(links1[link,1]) 352 | except: 353 | length_1 = -1 354 | try: 355 | length_2 = len(links1[link,2]) 356 | except: 357 | length_2 = -1 358 | if length_0 == 1 and length_2 == 1: 359 | start_node, end_node = min(links1[link,0][0],links1[link,2][0]), max(links1[link,0][0],links1[link,2][0]) 360 | add_link(start_node,end_node,cont) 361 | edgelist = edgelist.append(Edgelist(start_node,end_node,cont+1,distance(start_node,end_node)),ignore_index=True) 362 | cont += 1 363 | elif length_0 == -1: 364 | for i in range(length_2-1): 365 | start_node, end_node = links1[link,2][i], links1[link,2][i+1] 366 | add_link(start_node,end_node,cont) 367 | edgelist = edgelist.append(Edgelist(start_node,end_node,cont+1,distance(start_node,end_node)),ignore_index=True) 368 | cont += 1 369 | elif length_2 == -1: 370 | start_node, end_node = links1[link,0][0], links1[link,0][1] 371 | add_link(start_node,end_node,cont) 372 | edgelist = edgelist.append(Edgelist(start_node,end_node,cont+1,distance(start_node,end_node)),ignore_index=True) 373 | cont += 1 374 | else: 375 | if length_0 > 1: 376 | start_node, end_node = links1[link,0][0], links1[link,2][0] 377 | add_link(start_node,end_node,cont) 378 | edgelist = edgelist.append(Edgelist(start_node,end_node,cont+1,distance(start_node,end_node)),ignore_index=True) 379 | cont += 1 380 | for i in range(length_2-1): 381 | start_node, end_node = links1[link,2][i], links1[link,2][i+1] 382 | add_link(start_node,end_node,cont) 383 | edgelist = edgelist.append(Edgelist(start_node,end_node,cont+1,distance(start_node,end_node)),ignore_index=True) 384 | cont += 1 385 | start_node, end_node = links1[link,2][-1], links1[link,0][1] 386 | add_link(start_node,end_node,cont) 387 | edgelist = edgelist.append(Edgelist(start_node,end_node,cont+1,distance(start_node,end_node)),ignore_index=True) 388 | cont += 1 389 | else: 390 | if links1[link,0][0] < links1[link,2][0]: 391 | start_node, end_node = links1[link,0][0], links1[link,2][0] 392 | add_link(start_node,end_node,cont) 393 | edgelist = edgelist.append(Edgelist(start_node,end_node,cont+1,distance(start_node,end_node)),ignore_index=True) 394 | cont += 1 395 | for i in range(length_2 - 1): 396 | start_node, end_node = links1[link,2][i], links1[link,2][i+1] 397 | add_link(start_node,end_node,cont) 398 | edgelist = edgelist.append(Edgelist(start_node,end_node,cont+1,distance(start_node,end_node)),ignore_index=True) 399 | cont += 1 400 | else: 401 | for i in range(length_2 - 1): 402 | start_node, end_node = links1[link,2][i], links1[link,2][i+1] 403 | add_link(start_node,end_node,cont) 404 | edgelist = edgelist.append(Edgelist(start_node,end_node,cont+1,distance(start_node,end_node)),ignore_index=True) 405 | cont += 1 406 | start_node, end_node = links1[link,2][-1], links1[link,0][0] 407 | add_link(start_node,end_node,cont) 408 | edgelist = edgelist.append(Edgelist(start_node,end_node,cont+1,distance(start_node,end_node)),ignore_index=True) 409 | cont += 1 410 | shapeData = None 411 | 412 | print '\n'+str(time.time() - start)+'s - Links and Edgelist are done!' 413 | sys.stdout.flush() 414 | 415 | print '\n'+str(time.time() - start)+'s - Saving Edgelist...' 416 | sys.stdout.flush() 417 | 418 | #saving the edgelist to a 'csv. file' 419 | edgelist.to_csv(path_edgelist,index=False) 420 | 421 | print '\n'+str(time.time() - start)+'s - You are all set.' 422 | sys.stdout.flush() 423 | -------------------------------------------------------------------------------- /Python/v1.20/GISF2E_python_v1.20.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | ''' 3 | GISF2E python - v1.20 4 | 5 | To cite, please use: Karduni, A., Kermanshah, A., & Derrible, S., 2016 “A protocol to convert spatial polyline data to network formats and applications to world urban road networks”, Scientific Data, 3:160046 6 | The article is available at: http://www.nature.com/articles/sdata201646 7 | 8 | Main authors: Felipe Macena, Rodrigo Marinho 9 | ''' 10 | 11 | 12 | #import all libraries required 13 | import shapefile 14 | import pandas as pd 15 | from pyproj import Proj 16 | import osgeo.ogr, osgeo.osr 17 | from osgeo import ogr 18 | import sys 19 | import os 20 | import time 21 | from math import pi, sin, cos, tan, sqrt 22 | 23 | def LLtoUTM(ReferenceEllipsoid, Lat, Long, zone = None): 24 | """converts lat/long to UTM coords. Equations from USGS Bulletin 1532 25 | East Longitudes are positive, West longitudes are negative. 26 | North latitudes are positive, South latitudes are negative 27 | Lat and Long are in decimal degrees 28 | Written by Chuck Gantz- chuck.gantz@globalstar.com""" 29 | 30 | a = _ellipsoid[ReferenceEllipsoid][_EquatorialRadius] 31 | eccSquared = _ellipsoid[ReferenceEllipsoid][_eccentricitySquared] 32 | k0 = 0.9996 33 | 34 | #Make sure the longitude is between -180.00 .. 179.9 35 | LongTemp = (Long+180)-int((Long+180)/360)*360-180 # -180.00 .. 179.9 36 | 37 | LatRad = Lat*_deg2rad 38 | LongRad = LongTemp*_deg2rad 39 | 40 | if zone is None: 41 | ZoneNumber = int((LongTemp + 180)/6) + 1 42 | else: 43 | ZoneNumber = zone 44 | 45 | if Lat >= 56.0 and Lat < 64.0 and LongTemp >= 3.0 and LongTemp < 12.0: 46 | ZoneNumber = 32 47 | 48 | # Special zones for Svalbard 49 | if Lat >= 72.0 and Lat < 84.0: 50 | if LongTemp >= 0.0 and LongTemp < 9.0:ZoneNumber = 31 51 | elif LongTemp >= 9.0 and LongTemp < 21.0: ZoneNumber = 33 52 | elif LongTemp >= 21.0 and LongTemp < 33.0: ZoneNumber = 35 53 | elif LongTemp >= 33.0 and LongTemp < 42.0: ZoneNumber = 37 54 | 55 | LongOrigin = (ZoneNumber - 1)*6 - 180 + 3 #+3 puts origin in middle of zone 56 | LongOriginRad = LongOrigin * _deg2rad 57 | 58 | #compute the UTM Zone from the latitude and longitude 59 | UTMZone = "%d%c" % (ZoneNumber, _UTMLetterDesignator(Lat)) 60 | 61 | eccPrimeSquared = (eccSquared)/(1-eccSquared) 62 | N = a/sqrt(1-eccSquared*sin(LatRad)*sin(LatRad)) 63 | T = tan(LatRad)*tan(LatRad) 64 | C = eccPrimeSquared*cos(LatRad)*cos(LatRad) 65 | A = cos(LatRad)*(LongRad-LongOriginRad) 66 | 67 | M = a*((1 68 | - eccSquared/4 69 | - 3*eccSquared*eccSquared/64 70 | - 5*eccSquared*eccSquared*eccSquared/256)*LatRad 71 | - (3*eccSquared/8 72 | + 3*eccSquared*eccSquared/32 73 | + 45*eccSquared*eccSquared*eccSquared/1024)*sin(2*LatRad) 74 | + (15*eccSquared*eccSquared/256 + 45*eccSquared*eccSquared*eccSquared/1024)*sin(4*LatRad) 75 | - (35*eccSquared*eccSquared*eccSquared/3072)*sin(6*LatRad)) 76 | 77 | UTMEasting = (k0*N*(A+(1-T+C)*A*A*A/6 78 | + (5-18*T+T*T+72*C-58*eccPrimeSquared)*A*A*A*A*A/120) 79 | + 500000.0) 80 | 81 | UTMNorthing = (k0*(M+N*tan(LatRad)*(A*A/2+(5-T+9*C+4*C*C)*A*A*A*A/24 82 | + (61 83 | -58*T 84 | +T*T 85 | +600*C 86 | -330*eccPrimeSquared)*A*A*A*A*A*A/720))) 87 | if Lat < 0: 88 | UTMNorthing = UTMNorthing + 10000000.0; #10000000 meter offset for southern hemisphere 89 | return (UTMZone, UTMEasting, UTMNorthing) 90 | 91 | 92 | def _UTMLetterDesignator(Lat): 93 | """This routine determines the correct UTM letter designator for the given 94 | latitude """ 95 | if Lat >= 0: return 'N' 96 | else: return 'S' 97 | 98 | def add_link(start,end,cont): 99 | pontos = [] 100 | pontos.append(osgeo.ogr.Geometry(osgeo.ogr.wkbLineString)) 101 | lista = list([testando['coordinates_utm'][start:end+1]][0]) 102 | for a in lista: 103 | pontos[-1].AddPoint(a[0],a[1]) 104 | featureIndex = cont 105 | feature = osgeo.ogr.Feature(layer_defn) 106 | feature.SetGeometry(pontos[-1]) 107 | feature.SetFID(featureIndex) 108 | layer.CreateFeature(feature) 109 | for name in range(len(field_names)): 110 | if field_names[name] != 'Edge' and field_names[name] != 'Length': 111 | feature.SetField(name, str(testando[field_names[name]][start_node])) 112 | elif field_names[name] == 'Edge': 113 | feature.SetField(name, cont+1) 114 | elif field_names[name] == 'Length': 115 | feature.SetField(name, distance(start,end)) 116 | layer.SetFeature(feature) 117 | def add_field(field_names,featureIndex): 118 | for name in range(len(field_names)): 119 | if field_names[name] == 'coord_x' or field_names[name] == 'coord_y': 120 | name_aux = 'coordinates_utm' 121 | else: 122 | name_aux = field_names[name] 123 | feature = layer.GetFeature(featureIndex) #lets get the first feature 124 | if field_names[name] == 'coord_x': 125 | feature.SetField(name,testando_nodes[name_aux][featureIndex][0]) 126 | elif field_names[name] == 'coord_y': 127 | feature.SetField(name,testando_nodes[name_aux][featureIndex][1]) 128 | else: 129 | feature.SetField(name,int(testando_nodes[name_aux][featureIndex])) 130 | layer.SetFeature(feature) #now make the change permanent 131 | def distance(start,end): 132 | dist = 0 133 | for u in range(start,end): 134 | dist = dist + ((testando['coordinates_utm'][u][0] - testando['coordinates_utm'][u+1][0])**2 + (testando['coordinates_utm'][u][1] - testando['coordinates_utm'][u+1][1])**2)**0.5 135 | return dist 136 | def Edgelist(start,end,edge,length): 137 | global testando, columns 138 | datasheet = pd.DataFrame([[testando['coordinates_utm'][start][0],testando['coordinates_utm'][start][1],testando['coordinates_utm'][end][0],testando['coordinates_utm'][end][1],testando['Node'][start],testando['Node'][end],edge,length]],columns = columns, index = None) 139 | return datasheet 140 | 141 | _deg2rad, _rad2deg, _EquatorialRadius, _eccentricitySquared = pi / 180.0, 180.0 / pi, 2, 3 142 | 143 | _ellipsoid = [ 144 | # id, Ellipsoid name, Equatorial Radius, square of eccentricity 145 | # first once is a placeholder only, To allow array indices to match id numbers 146 | [ -1, "Placeholder", 0, 0], 147 | [ 1, "Airy", 6377563, 0.00667054], 148 | [ 2, "Australian National", 6378160, 0.006694542], 149 | [ 3, "Bessel 1841", 6377397, 0.006674372], 150 | [ 4, "Bessel 1841 (Nambia] ", 6377484, 0.006674372], 151 | [ 5, "Clarke 1866", 6378206, 0.006768658], 152 | [ 6, "Clarke 1880", 6378249, 0.006803511], 153 | [ 7, "Everest", 6377276, 0.006637847], 154 | [ 8, "Fischer 1960 (Mercury] ", 6378166, 0.006693422], 155 | [ 9, "Fischer 1968", 6378150, 0.006693422], 156 | [ 10, "GRS 1967", 6378160, 0.006694605], 157 | [ 11, "GRS 1980", 6378137, 0.00669438], 158 | [ 12, "Helmert 1906", 6378200, 0.006693422], 159 | [ 13, "Hough", 6378270, 0.00672267], 160 | [ 14, "International", 6378388, 0.00672267], 161 | [ 15, "Krassovsky", 6378245, 0.006693422], 162 | [ 16, "Modified Airy", 6377340, 0.00667054], 163 | [ 17, "Modified Everest", 6377304, 0.006637847], 164 | [ 18, "Modified Fischer 1960", 6378155, 0.006693422], 165 | [ 19, "South American 1969", 6378160, 0.006694542], 166 | [ 20, "WGS 60", 6378165, 0.006693422], 167 | [ 21, "WGS 66", 6378145, 0.006694542], 168 | [ 22, "WGS-72", 6378135, 0.006694318], 169 | [ 23, "WGS-84", 6378137, 0.00669438] 170 | ] 171 | 172 | path = os.getcwd() 173 | 174 | while True: 175 | option = raw_input('Do you want to use input/output folders (1) or type your files addresses (2)? Option 1 or 2: ') 176 | if option == '1': 177 | sf = shapefile.Reader(path+'/input/'+os.listdir(path+'/input/')[0]) 178 | path_nodes = path+ '/output/Nodes_'+os.listdir(path+'/input/')[0][:-4]+'.shp' 179 | path_links = path+ '/output/Edges_'+os.listdir(path+'/input/')[0][:-4]+'.shp' 180 | path_edgelist = path+ '/output/Edgelist_'+os.listdir(path+'/input/')[0][:-4]+'.csv' 181 | break 182 | elif option == '2': 183 | sf = shapefile.Reader(raw_input("Please type where your file is located (Example:C:\Users\UIC\Documents\Shape_File.shp): ")) 184 | path_nodes = raw_input("Please type where you want the nodes shapefile to be located(Example:C:\Users\UIC\Documents\Nodes_File.shp): ") 185 | path_links = raw_input("Please type where you want the links shapefile to be located(Example:C:\Users\UIC\Documents\Links_File.shp): ") 186 | path_edgelist = raw_input("Please type where you want the Edgelist to be located(Example:C:\Users\UIC\Documents\Edgelist.csv): ") 187 | break 188 | 189 | osm = raw_input("Is your root file from Open Street Maps (Y/N)? ") 190 | osm = osm.lower() 191 | start = time.time() 192 | print '\n'+str(time.time() - start)+'s - Reading and arranging the root file information...' 193 | sys.stdout.flush() 194 | 195 | #dividing each link by their osm_id information (osm_maps) or id information (non-osm_maps) 196 | shapes = sf.shapes() 197 | fields=sf.fields 198 | fields = [field[0] for field in sf.fields[1:]] 199 | 200 | 201 | if 'TYPE' in fields or 'Type' in fields: 202 | try: 203 | fields[fields.index('TYPE')] = 'type' 204 | except: 205 | fields[fields.index('Type')] = 'type' 206 | 207 | heads = [x.upper() if x in ['Group','coordinates','primeiro','coordinates_utm','Node'] else x for x in [sf.shapeRecords()[0].shape.__geo_interface__.keys()[0]]+fields] + ['Group','coordinates','primeiro','coordinates_utm','Node'] 208 | type_ = False 209 | if 'type' in map(lambda x: x.lower(),fields): 210 | heads.remove('type') 211 | heads.insert(3,'type 1') 212 | type_ = True 213 | 214 | dicti = {} 215 | for i in range(len(heads)-1): 216 | dicti[heads[i]] = [] 217 | 218 | testando = pd.DataFrame() 219 | 220 | n = 0 221 | for feature in sf.shapeRecords(): 222 | geom = feature.shape.__geo_interface__ 223 | atr = dict(zip(fields, feature.record)) 224 | if type_: 225 | atr['type 1'] = atr.pop('type') 226 | geom.update(atr) 227 | geom['Group'] = n 228 | j = geom['coordinates'] 229 | for i in range(len(heads)-4): 230 | u = [geom[heads[i]]] 231 | for y in range(len(j)): 232 | dicti[heads[i]].append(u[0]) 233 | dicti['coordinates'] = list(dicti['coordinates']) + list(j) 234 | dicti['primeiro'] = dicti['primeiro'] + [1] + [0]*(len(j) - 2) + [1] 235 | n += 1 236 | if ((n-1) % 800) == 0: #The number 800 was chosen because it's shown the fastest results in a range of numbers from 100 to 1000 237 | dicti['coordinates_utm'] = None 238 | dicti['Node'] = None 239 | testando = pd.concat([testando,pd.DataFrame.from_dict(dicti)]) 240 | for i in range(len(heads)-1): 241 | dicti[heads[i]] = [] 242 | if ((n-1) % 800) != 0: 243 | dicti['coordinates_utm'] = None 244 | dicti['Node'] = None 245 | testando = pd.concat([testando,pd.DataFrame.from_dict(dicti)],ignore_index=True) 246 | 247 | utm_zone = LLtoUTM(23, testando['coordinates'][0][1], testando['coordinates'][0][0])[0] 248 | if utm_zone[-1] == "S": 249 | p = Proj("+proj=utm +zone="+str(int(utm_zone[:-1]))+", +south +ellps=WGS84") 250 | else: 251 | p = Proj("+proj=utm +zone="+str(int(utm_zone[:-1]))+", +ellps=WGS84") 252 | 253 | testando['coordinates_utm'] = [p(testando['coordinates'][k][0],testando['coordinates'][k][1]) for k in range(len(testando['coordinates']))] 254 | testando['Node'] = None 255 | dicti = None 256 | geom = None 257 | atr = None 258 | 259 | #test before creating nodes for OSM maps 260 | a= testando.groupby('coordinates_utm').groups 261 | testando['continuidade'] = 1 262 | testando.ix[testando.primeiro == 1,'continuidade'] = 0; testando 263 | b1 = map(lambda x: (testando['coordinates_utm'][x],[x]),testando.groupby('continuidade').groups[0]) 264 | if osm == 'y'or osm == 'yes' or osm == 'y': 265 | b = filter(lambda x: (testando['bridge'][x[1][0]] == testando['bridge'][int(x[1][1])]) or (testando['tunnel'][x[1][0]] == testando['tunnel'][int(x[1][1])]), filter(lambda x: len(x[1]) >= 2, zip(a.keys(),a.values()))) 266 | else: # no test for non-OSM maps 267 | b = filter(lambda x: len(x[1]) >= 2, zip(a.keys(),a.values())) 268 | for h in (b): 269 | for k in range(len(h[1])): 270 | # testando['continuidade'][h[1][k]] = 2 271 | testando.loc[[h[1][k]],'continuidade'] = 2 272 | b = b + b1 273 | points = sorted(list(testando.groupby('continuidade').groups[0]) + list(testando.groupby('continuidade').groups[2])) 274 | testando.loc[points,'Node'] = range(1,len(points)+1) 275 | 276 | print '\n'+str(time.time() - start)+'s - Reading and arranging are done!' 277 | sys.stdout.flush() 278 | 279 | #creating nodes shapefile 280 | spatialReference = osgeo.osr.SpatialReference() #will create a spatial reference locally to tell the system what the reference will be stored 281 | spatialReference.ImportFromProj4('+proj=utm +zone='+utm_zone+' ellps=WGS84 +datum=WGS84 +units=m') #here we define the utm zone as well as the world geodetic system(wgs) 282 | driver = osgeo.ogr.GetDriverByName('ESRI Shapefile') # will select the driver for our shp-file creation. 283 | shapeData = driver.CreateDataSource(path_nodes) #so there we will store our data 284 | layer = shapeData.CreateLayer('Nodes', spatialReference, osgeo.ogr.wkbPoint) #this will create a corresponding layer for our data with given spatial information 285 | layer_defn = layer.GetLayerDefn() # gets parameters of the current shapefile 286 | point = osgeo.ogr.Geometry(osgeo.ogr.wkbPoint) 287 | 288 | new_field = ogr.FieldDefn('Node', ogr.OFTInteger) #add a field 'Node' stored as integer 289 | layer.CreateField(new_field) #self explaining 290 | new_field = ogr.FieldDefn('coord_x', ogr.OFTReal) #add a field 'coord_x' stored as real 291 | layer.CreateField(new_field) #self explaining 292 | new_field = ogr.FieldDefn('coord_y', ogr.OFTReal) #add a field 'coord_y' stored as real 293 | layer.CreateField(new_field) #self explaining 294 | if osm == 'y'or osm == 'yes' or osm == 'y': #for osm maps 295 | new_field = ogr.FieldDefn('bridge', ogr.OFTInteger) #we will create a new field called bridge as integer 296 | layer.CreateField(new_field) #self explaining 297 | new_field = ogr.FieldDefn('tunnel', ogr.OFTInteger) #and another field called 'tunnel' stored as integer 298 | layer.CreateField(new_field) #self explaining 299 | field_names = [layer_defn.GetFieldDefn(i).GetName() for i in range(layer_defn.GetFieldCount())] 300 | aux_b, featureIndex = list(zip(*b)[0]), 0 301 | 302 | testando_nodes = testando[testando.continuidade != 1].drop_duplicates(['coordinates_utm']) 303 | testando_nodes.index = range(len(testando_nodes)) 304 | 305 | t, aux_b, a, b, b1, points = None, None, None, None, None, None 306 | 307 | print '\n'+str(time.time() - start)+'s - Adding nodes...' 308 | sys.stdout.flush() 309 | 310 | for featureIndex in range(len(testando_nodes)): 311 | point.AddPoint(testando_nodes['coordinates_utm'][featureIndex][0],testando_nodes['coordinates_utm'][featureIndex][1]) #create a new point at given coordinates 312 | feature = osgeo.ogr.Feature(layer_defn) 313 | feature.SetGeometry(point) 314 | feature.SetFID(featureIndex) 315 | layer.CreateFeature(feature) 316 | add_field(field_names,featureIndex) 317 | shapeData = None 318 | 319 | new_b = None 320 | testando_nodes = None 321 | 322 | print '\n'+str(time.time() - start)+'s - Nodes are done!' 323 | sys.stdout.flush() 324 | 325 | #Cut the raw links and Creat the edges shapefile 326 | columns = ['X_Coord_Start','Y_Coord_Start','X_Coord_End','Y_Coord_End','Start_Node','End_Node','Edge','Length'] 327 | edgelist = pd.DataFrame(columns = columns, index=None) 328 | 329 | #creating edges shapefile 330 | spatialReference = osgeo.osr.SpatialReference() #will create a spatial reference locally to tell the system what the reference will be stored 331 | #here you can change the utm zone depending on where the map is from 332 | spatialReference.ImportFromProj4('+proj=utm +zone='+utm_zone+' ellps=WGS84 +datum=WGS84 +units=m') #here we define the utm zone as well as the world geodetic system(wgs) 333 | driver = osgeo.ogr.GetDriverByName('ESRI Shapefile') # will select the driver for our shp-file creation. 334 | shapeData = driver.CreateDataSource(path_links) #so there we will store our data 335 | layer = shapeData.CreateLayer('linestrings', spatialReference, osgeo.ogr.wkbLineString) #this will create a corresponding layer for our data with given spatial information. 336 | layer_defn = layer.GetLayerDefn() # gets parameters of the current shapefile 337 | 338 | for f in ['Edge']+heads[:-5]+['Length']: 339 | fieldDefn = ogr.FieldDefn(f, ogr.OFTString) 340 | layer.CreateField(fieldDefn) 341 | field_names = [layer_defn.GetFieldDefn(i).GetName() for i in range(layer_defn.GetFieldCount())] 342 | 343 | print '\n'+str(time.time() - start)+'s - Adding links and creating the Edgelist...' 344 | sys.stdout.flush() 345 | 346 | #several tests in order to create the edges shapefile correctly 347 | links = (testando.groupby(['Group']).groups) 348 | links1 = (testando.groupby(['Group','continuidade']).groups) 349 | cont = 0 350 | for link in range(len(links)): 351 | try: 352 | length_0 = len(links1[link,0]) 353 | except: 354 | length_0 = -1 355 | try: 356 | length_1 = len(links1[link,1]) 357 | except: 358 | length_1 = -1 359 | try: 360 | length_2 = len(links1[link,2]) 361 | except: 362 | length_2 = -1 363 | if length_0 == 1 and length_2 == 1: 364 | start_node, end_node = min(links1[link,0][0],links1[link,2][0]), max(links1[link,0][0],links1[link,2][0]) 365 | add_link(start_node,end_node,cont) 366 | edgelist = edgelist.append(Edgelist(start_node,end_node,cont+1,distance(start_node,end_node)),ignore_index=True) 367 | cont += 1 368 | elif length_0 == -1: 369 | for i in range(length_2-1): 370 | start_node, end_node = links1[link,2][i], links1[link,2][i+1] 371 | add_link(start_node,end_node,cont) 372 | edgelist = edgelist.append(Edgelist(start_node,end_node,cont+1,distance(start_node,end_node)),ignore_index=True) 373 | cont += 1 374 | elif length_2 == -1: 375 | start_node, end_node = links1[link,0][0], links1[link,0][1] 376 | add_link(start_node,end_node,cont) 377 | edgelist = edgelist.append(Edgelist(start_node,end_node,cont+1,distance(start_node,end_node)),ignore_index=True) 378 | cont += 1 379 | else: 380 | if length_0 > 1: 381 | start_node, end_node = links1[link,0][0], links1[link,2][0] 382 | add_link(start_node,end_node,cont) 383 | edgelist = edgelist.append(Edgelist(start_node,end_node,cont+1,distance(start_node,end_node)),ignore_index=True) 384 | cont += 1 385 | for i in range(length_2-1): 386 | start_node, end_node = links1[link,2][i], links1[link,2][i+1] 387 | add_link(start_node,end_node,cont) 388 | edgelist = edgelist.append(Edgelist(start_node,end_node,cont+1,distance(start_node,end_node)),ignore_index=True) 389 | cont += 1 390 | start_node, end_node = links1[link,2][-1], links1[link,0][1] 391 | add_link(start_node,end_node,cont) 392 | edgelist = edgelist.append(Edgelist(start_node,end_node,cont+1,distance(start_node,end_node)),ignore_index=True) 393 | cont += 1 394 | else: 395 | if links1[link,0][0] < links1[link,2][0]: 396 | start_node, end_node = links1[link,0][0], links1[link,2][0] 397 | add_link(start_node,end_node,cont) 398 | edgelist = edgelist.append(Edgelist(start_node,end_node,cont+1,distance(start_node,end_node)),ignore_index=True) 399 | cont += 1 400 | for i in range(length_2 - 1): 401 | start_node, end_node = links1[link,2][i], links1[link,2][i+1] 402 | add_link(start_node,end_node,cont) 403 | edgelist = edgelist.append(Edgelist(start_node,end_node,cont+1,distance(start_node,end_node)),ignore_index=True) 404 | cont += 1 405 | else: 406 | for i in range(length_2 - 1): 407 | start_node, end_node = links1[link,2][i], links1[link,2][i+1] 408 | add_link(start_node,end_node,cont) 409 | edgelist = edgelist.append(Edgelist(start_node,end_node,cont+1,distance(start_node,end_node)),ignore_index=True) 410 | cont += 1 411 | start_node, end_node = links1[link,2][-1], links1[link,0][0] 412 | add_link(start_node,end_node,cont) 413 | edgelist = edgelist.append(Edgelist(start_node,end_node,cont+1,distance(start_node,end_node)),ignore_index=True) 414 | cont += 1 415 | shapeData = None 416 | 417 | print '\n'+str(time.time() - start)+'s - Links and Edgelist are done!' 418 | sys.stdout.flush() 419 | 420 | print '\n'+str(time.time() - start)+'s - Saving Edgelist...' 421 | sys.stdout.flush() 422 | 423 | #saving the edgelist to a 'csv. file' 424 | edgelist.to_csv(path_edgelist,index=False) 425 | 426 | print '\n'+str(time.time() - start)+'s - You are all set.' 427 | sys.stdout.flush() 428 | --------------------------------------------------------------------------------