47 | 22 | Photo by Abigail Keenan on Unsplash 23 |
24 | 25 | 26 | ### The main contributions of OSMsc: 27 | * Construct semantic city objects based on the public dataset (OpenStreetMap), and apply geometric operations to build more complete city objects; 28 | * Fuse 3D and tag information from multiple data sources through the spatial analysis between OSMsc layers and other non-OSM data layers; 29 | * Propose the semantic connector(UrbanTile), and supplement the spatial semantics; 30 | * Output the CityJSON-formatted semantic city models. 31 | 32 | 33 | 34 | ### OSMsc的主要功能： 35 | * 内部集成了OSM数据的自动化下载，仅需几行简单的代码，即可完成城市对象的构建； 36 | * 轻松融合外部3D或者文本数据，丰富OSM城市对象的信息； 37 | * 城市模型内部的对象可以添加空间语义，彼此之间的空间关系可以查询或者推测出来； 38 | * 可以输出CityJSON或者html的可视化文件，CityJSON格式文件可以由https://ninja.cityjson.org 查看. 39 | 40 | 41 | 42 |  43 |OSMsc workflow
44 | 45 |
47 | Semantic city model generated by OSMsc
49 | 50 | ### Installation 51 | 52 | 53 | Install from [Github](https://github.com/ruirzma/osmsc) 54 | 55 | `git clone https://github.com/ruirzma/osmsc.git` 56 | 57 | `cd osmsc/` 58 | 59 | `pip install .` or `python setup.py install` 60 | 61 | 62 | Install from [PyPi](https://pypi.org/project/osmsc/) 63 | 64 | `pip install osmsc` 65 | 66 | 67 | Note: 68 | 69 | * OSMnx should be installed before OSMsc, installation errors of OSMnx could be resolved in the latest OSMnx [documentation](https://osmnx.readthedocs.io/en/stable/index.html). 70 | 71 | * If installing OSMnx manually, pls download the Python extension packages (Rtree, GDAL, Fiona, rasterio, etc.) from [here](https://www.lfd.uci.edu/~gohlke/pythonlibs/) for Windows and [Homebrew🍺](https://brew.sh/) for MacOS. 72 | 73 | 74 | ### Examples 75 | 76 | 77 | [OSMsc demonstration notebooks](https://github.com/ruirzma/osmsc/tree/main/examples) 78 | 79 | 80 | ## Citation 81 | If you use OSMsc in scientific work, I kindly ask you to cite it: 82 | 83 | ```bibtex 84 | @article{doi:10.1080/13658816.2023.2266824, 85 | author = {Rui Ma, Jiayu Chen, Chendi Yang and Xin Li}, 86 | title = {OSMsc: a framework for semantic 3D city modeling using OpenStreetMap}, 87 | journal = {International Journal of Geographical Information Science}, 88 | volume = {0}, 89 | number = {0}, 90 | pages = {1-26}, 91 | year = {2023}, 92 | publisher = {Taylor & Francis}, 93 | doi = {10.1080/13658816.2023.2266824}, 94 | URL = {https://doi.org/10.1080/13658816.2023.2266824}, 95 | eprint = {https://doi.org/10.1080/13658816.2023.2266824}} 96 | ``` 97 | 98 | 99 | ### Reference 100 | 1. Geoff Boeing, OSMnx, https://github.com/gboeing/osmnx 101 | 2. Nick Bristow, OSMuf, https://github.com/AtelierLibre/osmuf 102 | 3. Joris Van den Bossche, GeoPandas, https://github.com/geopandas/geopandas 103 | 104 | -------------------------------------------------------------------------------- /osmsc/plot.py: -------------------------------------------------------------------------------- 1 | """OSMsc plot module""" 2 | 3 | import geopandas as gpd 4 | import pydeck as pdk 5 | from contextily import add_basemap 6 | import matplotlib.pyplot as plt 7 | 8 | from .utils import bbox_from_gdf 9 | 10 | 11 | def viz_buildings(buildings_gdf, zoom=11, max_zoom=16, pitch=45, bearing=0, 12 | html_name = "3D_buildings"): 13 | """ 14 | Building DataFrame 3D Visualization. 15 | NOTE buildings_gdf should have "Building_height" column. 16 | 17 | Parameters 18 | ---------- 19 | buildings_gdf : GeoDataFrame 20 | zoom : int 21 | max_zoom : int 22 | pitch : int 23 | bearing : int 24 | html_name : string 25 | filename to be saved in the current project location. 26 | 27 | Returns 28 | ------- 29 | HTML 30 | """ 31 | bbox = bbox_from_gdf(buildings_gdf,buffer_value=0.002) 32 | (minlat, minlon, maxlat, maxlon) = bbox 33 | land_cover = [[minlat,minlon], [minlat,maxlon], [maxlat,minlon], [maxlat,maxlon]] 34 | 35 | max_Height = max(buildings_gdf["Building_height"]) 36 | 37 | INITIAL_VIEW_STATE = pdk.ViewState(latitude= (minlat + maxlat)/2, longitude=(minlon + maxlon)/2, 38 | zoom = zoom, max_zoom=max_zoom, pitch=pitch, bearing=bearing) 39 | 40 | polygon = pdk.Layer( 41 | "PolygonLayer", 42 | land_cover, 43 | stroked=False, 44 | # processes the data as a flat longitude-latitude pair 45 | get_polygon="-", 46 | get_fill_color=[0, 0, 0, 20], 47 | ) 48 | 49 | geojson = pdk.Layer( 50 | "GeoJsonLayer", 51 | buildings_gdf, 52 | opacity=0.8, 53 | stroked=False, 54 | filled=True, 55 | extruded=True, 56 | wireframe=True, 57 | get_elevation="Building_height", 58 | 59 | get_fill_color="[ Building_height/" + str(max_Height) + "*255, 250, 85]", 60 | get_line_color=[255, 255, 255] 61 | ) 62 | 63 | r = pdk.Deck(layers=[polygon, geojson], initial_view_state=INITIAL_VIEW_STATE) 64 | 65 | return r.to_html(html_name + ".html") 66 | 67 | 68 | def gdf_with_basemap(gdf, figsize = (12, 6), column = None, color = None,markersize = None, 69 | cmap = None ,legend=False, categorical = False, edgecolor = None, linewidth = None, 70 | facecolor= None ,alpha = None, zoom = "auto", tile_source = None): 71 | """ 72 | Plot gdf with basemap 73 | 74 | column : str, np.array, pd.Series (default None) 75 | The name of the dataframe column, np.array, or pd.Series to be plotted. 76 | If np.array or pd.Series are used then it must have same length as 77 | dataframe. Values are used to color the plot. Ignored if `color` is 78 | also set. 79 | 80 | cmap : str (default None) 81 | The name of a colormap recognized by matplotlib. 82 | 83 | color : str (default None) 84 | If specified, all objects will be colored uniformly. 85 | 86 | categorical : bool (default False) 87 | If False, cmap will reflect numerical values of the 88 | column being plotted. For non-numerical columns, this 89 | will be set to True. 90 | 91 | legend : bool (default False) 92 | Plot a legend. Ignored if no `column` is given, or if `color` is given. 93 | 94 | markersize : str or float or sequence (default None) 95 | Only applies to point geometries within a frame. 96 | If a str, will use the values in the column of the frame specified 97 | by markersize to set the size of markers. Otherwise can be a value 98 | to apply to all points, or a sequence of the same length as the 99 | number of points. 100 | 101 | figsize : tuple of integers (default None) 102 | Size of the resulting matplotlib.figure.Figure. If the argument 103 | axes is given explicitly, figsize is ignored. 104 | 105 | 106 | edgecolor : str (default None) 107 | 108 | facecolor ： str (default None) 109 | 110 | linewidth ： str (default None) 111 | 112 | alpha ： float (default None) 113 | 114 | zoom : int or 'auto' 115 | [Optional. Default='auto'] Level of detail for the basemap. If 'auto', 116 | it is calculated automatically. Ignored if `source` is a local file. 117 | 118 | tile_source : contextily.providers object or str 119 | [Optional. Default: Stamen Terrain web tiles] 120 | The tile source: web tile provider or path to local file. The web tile 121 | provider can be in the form of a `contextily.providers` object or a 122 | URL. The placeholders for the XYZ in the URL need to be `{x}`, `{y}`, 123 | `{z}`, respectively. For local file paths, the file is read with 124 | `rasterio` and all bands are loaded into the basemap. 125 | IMPORTANT: tiles are assumed to be in the Spherical Mercator 126 | projection (EPSG:3857), unless the `crs` keyword is specified. 127 | 128 | Returns 129 | ------- 130 | fig, ax 131 | """ 132 | fig, ax = plt.subplots(figsize = figsize) 133 | gdf.plot( ax=ax, column = column, color = color,markersize = markersize, 134 | cmap = cmap ,legend =legend, categorical = categorical, edgecolor = edgecolor, 135 | linewidth = linewidth, facecolor= facecolor ,alpha = alpha) 136 | 137 | add_basemap(ax, crs = gdf.crs, zoom = "auto", source= tile_source) 138 | 139 | return fig, ax 140 | 141 | 142 | 143 | 144 | -------------------------------------------------------------------------------- /osmsc/grid.py: -------------------------------------------------------------------------------- 1 | """Create grid inside Polygon objects.""" 2 | 3 | import geopandas as gpd 4 | import pandas as pd 5 | import numpy as np 6 | import math 7 | from shapely.geometry import Point 8 | 9 | from sklearn.neighbors.kde import KernelDensity 10 | from scipy.spatial.distance import cdist 11 | 12 | 13 | class Grid(object): 14 | """ 15 | Construction process of a Grid object 16 | """ 17 | 18 | def __init__(self, data_type, step, bandwidth): 19 | 20 | # Polygon or Point 21 | self.data_type = data_type 22 | 23 | # the smaller the step, the denser the grid, the higher the accuracy 24 | self.step = step 25 | 26 | # Bandwidth used for kernel density estimation 27 | self.bandwidth = bandwidth 28 | 29 | # Editted from https://github.com/lgervasoni/urbansprawl 30 | def create_grid_from_gdf(self, object_gdf): 31 | """ 32 | Create a regular grid with the input geodataframe 33 | Editted from https://github.com/lgervasoni/urbansprawl 34 | 35 | Returns 36 | ---------- 37 | geopandas.GeoDataFrame 38 | """ 39 | # Get bounding box 40 | west, south, east, north = object_gdf.total_bounds 41 | # Create indices 42 | grid_gdf = gpd.GeoDataFrame( [ Point(i,j) for i in np.arange(west, east, self.step) for j in np.arange(south, north, self.step) ], columns=["geometry"] ) 43 | # Set projection 44 | grid_gdf.crs = object_gdf.crs 45 | 46 | return grid_gdf 47 | 48 | 49 | def WeightedKernelDensityEstimation(self, X, Weights, Y, max_mb_per_chunk = 1000): 50 | """ 51 | Computes a Weighted Kernel Density Estimation 52 | Editted from https://github.com/lgervasoni/urbansprawl 53 | 54 | Returns 55 | ---------- 56 | pd.Series 57 | returns an array of the estimated densities rescaled between [0;1] 58 | """ 59 | def get_megabytes_pairwise_distances_allocation(X, Y): 60 | # Calculate MB needed to allocate pairwise distances 61 | return len(X) * len(Y) * 1e-6 62 | 63 | # During this procedure, pairwise euclidean distances are computed between inputs points X and points to estimate Y 64 | # For this reason, Y is divided in chunks to avoid big memory allocations. At most, X megabytes per chunk are allocated for pairwise distances 65 | Y_split = np.array_split( Y, math.ceil( get_megabytes_pairwise_distances_allocation(X,Y) / max_mb_per_chunk ) ) 66 | 67 | """ 68 | ### Step by step 69 | # Weighed KDE: Sum{ Weight_i * K( (X-Xi) / h) } 70 | W_norm = np.array( Weights / np.sum(Weights) ) 71 | cdist_values = cdist( Y, X, 'euclidean') / bandwidth 72 | Ks = np.exp( -.5 * ( cdist_values ) ** 2 ) 73 | PDF = np.sum( Ks * W_norm, axis=1) 74 | """ 75 | """ 76 | ### Complete version. Memory consuming 77 | PDF = np.sum( np.exp( -.5 * ( cdist( Y, X, 'euclidean') / bandwidth ) ** 2 ) * ( np.array( Weights / np.sum(Weights) ) ), axis=1) 78 | """ 79 | 80 | ### Divide Y in chunks to avoid big memory allocations 81 | PDF = np.concatenate( [ np.sum( np.exp( -.5 * ( cdist( Y_i, X, 'euclidean') / self.bandwidth ) ** 2 ) * ( np.array( Weights / np.sum(Weights) ) ), axis=1) for Y_i in Y_split ] ) 82 | 83 | # np.concatenate 数组拼接 84 | # np.concatenate((a, b), axis=None) array([1, 2, 3, 4, 5, 6]) 85 | 86 | # Rescale 87 | return pd.Series( PDF / PDF.sum() ) 88 | 89 | 90 | def calculate_kde(self, object_gdf, X_weights= None): 91 | """ 92 | Computes a Kernel Density Estimation 93 | Editted from https://github.com/lgervasoni/urbansprawl 94 | 95 | Returns 96 | ---------- 97 | pandas.Series 98 | """ 99 | 100 | self.grid_gdf = self.create_grid_from_gdf( object_gdf = object_gdf) 101 | 102 | if self.data_type == "Polygon": 103 | X_b = [ [p.x,p.y] for p in object_gdf.geometry.centroid.values] 104 | 105 | if self.data_type == "Point": 106 | X_b = [ [p.x,p.y] for p in object_gdf.geometry.values] 107 | 108 | X = np.array(X_b) 109 | # Points where the probability density function will be evaluated 110 | Y = np.array( [ [p.x,p.y] for p in self.grid_gdf.geometry.values ] ) 111 | 112 | if X_weights is None: 113 | 114 | kde = KernelDensity(kernel='gaussian', bandwidth = self.bandwidth).fit(X) 115 | 116 | # Sklearn returns the results in the form log(density) 117 | p = np.exp(kde.score_samples(Y)) 118 | 119 | else: 120 | # Weighted Kernel Density Estimation 121 | p = self.WeightedKernelDensityEstimation(X = X, Weights = X_weights, Y = Y,max_mb_per_chunk = 1000) 122 | 123 | return pd.Series( p / p.max() ) 124 | 125 | # Returns the spatial distribution density 126 | return pd.Series( p / p.max() ) 127 | 128 | 129 | def computed_grid(self, object_gdf, X_weights = None): 130 | """ 131 | Calculate land use mix indices on input grid 132 | Editted from https://github.com/lgervasoni/urbansprawl 133 | 134 | Returns 135 | ---------- 136 | pandas.DataFrame 137 | """ 138 | self.grid_gdf = self.create_grid_from_gdf( object_gdf) 139 | 140 | if X_weights is None: 141 | self.grid_gdf["no_weighted"] = self.calculate_kde(object_gdf = object_gdf, 142 | X_weights = None) 143 | else: 144 | self.grid_gdf["weighted"] = self.calculate_kde(object_gdf = object_gdf, 145 | X_weights = X_weights) 146 | 147 | return self.grid_gdf 148 | 149 | 150 | -------------------------------------------------------------------------------- /osmsc/fusion.py: -------------------------------------------------------------------------------- 1 | """ Data fusion for OSMsc objects""" 2 | 3 | 4 | import numpy as np 5 | import pandas as pd 6 | import geopandas as gpd 7 | import osmnx as ox 8 | 9 | from .feature import add_gdf_elevation_features 10 | from .grid import Grid 11 | 12 | 13 | 14 | def add_building_height(OSMscBuilding_gdf, externalBuilding_gdf, extColName = None ): 15 | """ 16 | Add building height features to OSM or building gdf. 17 | Please keep both gdf in the same crs 18 | 19 | Parameters 20 | ---------- 21 | OSMscBuilding_gdf : GeoDataFrame 22 | OSM or local building dataset 23 | externalBuilding_gdf : GeoDataFrame 24 | external building dataset, including height info 25 | extColName : string 26 | Height attr column 27 | 28 | Returns 29 | ------- 30 | GeoDataFrame 31 | """ 32 | print("Please wait a few minutes") 33 | 34 | # Spatial Join 35 | temp_gdf = gpd.sjoin(OSMscBuilding_gdf, externalBuilding_gdf, how="left", op="intersects") 36 | # For later quering 37 | uni_geom = temp_gdf['geometry'].unique() 38 | 39 | temp_gdf["Building_height"] = temp_gdf[extColName] 40 | 41 | building_height = [] 42 | for index in range(len(uni_geom)): 43 | heights_list = list(temp_gdf[temp_gdf["geometry"] == uni_geom[index]].Building_height) 44 | # average height of all intersected buildings 45 | building_height.append(np.mean(heights_list)) 46 | 47 | # Add attrs 48 | OSMscBuilding_gdf["Building_height"] = building_height 49 | 50 | return OSMscBuilding_gdf 51 | 52 | 53 | def add_building_tags(OSMscBuilding_gdf, tagColName ,externalBuilding_gdf, extColName): 54 | 55 | """ 56 | Add building tags to OSM or building gdf. 57 | Please keep both gdf in the same crs 58 | 59 | Parameters 60 | ---------- 61 | OSMscBuilding_gdf : GeoDataFrame 62 | OSM or local building dataset 63 | tagColName : string 64 | Column name to be added in the building dataframe 65 | externalBuilding_gdf : GeoDataFrame 66 | external building dataset, including height info 67 | extColName : string 68 | Original tag attr column 69 | 70 | Returns 71 | ------- 72 | GeoDataFrame 73 | """ 74 | 75 | intersection_area_gdf = gpd.overlay(OSMscBuilding_gdf, externalBuilding_gdf, how ='intersection') 76 | intersection_area_gdf["intersection_area"] = intersection_area_gdf["geometry"].area 77 | intersected_osmscID = list(intersection_area_gdf["osmscID"].unique()) 78 | 79 | for ID in intersected_osmscID: 80 | # for one OSMscID object 81 | area_list = list(intersection_area_gdf[intersection_area_gdf["osmscID"] == ID].intersection_area) 82 | index_list = list(intersection_area_gdf[intersection_area_gdf["osmscID"] == ID].index) 83 | 84 | # Sometimes, one OSM building intersects with several external buildings. 85 | if len(index_list) != 1: 86 | # Only keep one row with biggest intersection area 87 | # index_list for later delete indexing 88 | index_of_max_area = area_list.index(max(area_list)) 89 | del index_list[index_of_max_area] 90 | 91 | for i in index_list: 92 | intersection_area_gdf = intersection_area_gdf.drop(index=i) 93 | 94 | OSMscBuilding_gdf[tagColName] = ["Unknown" for i in range(len(OSMscBuilding_gdf))] 95 | 96 | for osmscID in intersected_osmscID: 97 | value = intersection_area_gdf[intersection_area_gdf["osmscID"] == osmscID][extColName].values 98 | index = OSMscBuilding_gdf[OSMscBuilding_gdf["osmscID"] == osmscID].index 99 | 100 | OSMscBuilding_gdf.loc[index, tagColName] = value 101 | 102 | for osmscID in OSMscBuilding_gdf["osmscID"]: 103 | if osmscID not in intersected_osmscID: 104 | index = OSMscBuilding_gdf[OSMscBuilding_gdf["osmscID"] == osmscID].index 105 | OSMscBuilding_gdf.loc[index, tagColName] = "Unknown" 106 | 107 | # replace None with "Unknown" for CityJSON output function 108 | for i in range(len(OSMscBuilding_gdf[tagColName].values)): 109 | col_list = OSMscBuilding_gdf[tagColName].values 110 | if col_list[i] == None: 111 | col_list[i] = "Unknown" 112 | 113 | OSMscBuilding_gdf[tagColName] = col_list 114 | 115 | return OSMscBuilding_gdf 116 | 117 | 118 | def add_elevation(cityobject_gdf, elevation_dataset = "aster30m", step = 5): 119 | """ 120 | Mostly, add elevation to UrbanTile and Transportation GeoDataFrame 121 | 122 | Parameters 123 | ---------- 124 | cityobject_gdf : GeoDataFrame 125 | UrbanTile or Transportation GeoDataFrame 126 | elevation_dataset : string 127 | Online elevation can be chose from https://www.opentopodata.org 128 | step : int 129 | Grid size 130 | 131 | Returns 132 | ------- 133 | GeoDataFrame 134 | """ 135 | 136 | # Create the grid GeoDataFrame (with elevation) 137 | grid_gdf = get_intersected_grid_gdf(cityobject_gdf, elevation_dataset = elevation_dataset, step = 5) 138 | grid_gdf["y_lat"] = grid_gdf["y_lat"].replace(np.nan, "None") 139 | grid_gdf["x_lng"] = grid_gdf["x_lng"].replace(np.nan, "None") 140 | grid_gdf["ground_elevation"] = grid_gdf["ground_elevation"].replace(np.nan, "None") 141 | 142 | elevation_list = [] 143 | cityobject_gdf["elevation"] = ["Unknown" for i in range(len(cityobject_gdf))] 144 | 145 | for i in range(len(cityobject_gdf)): 146 | _osmscid = list(cityobject_gdf["osmscID"])[i] 147 | 148 | # Extract all location in the grid. 149 | lat_list = list(grid_gdf[grid_gdf.osmscID == _osmscid].y_lat) 150 | lon_list = list(grid_gdf[grid_gdf.osmscID == _osmscid].x_lng) 151 | ele_list = list(grid_gdf[grid_gdf.osmscID == _osmscid].ground_elevation) 152 | 153 | # Make the mass point list 154 | massPoints_list = [] 155 | for j in range(len(lat_list)): 156 | massPoints_list.append({'lat': lat_list[j], 'lon': lon_list[j],'ele': ele_list[j]}) 157 | 158 | elevation_list.append(massPoints_list) 159 | 160 | cityobject_gdf["elevation"] = elevation_list 161 | 162 | return cityobject_gdf 163 | 164 | 165 | def get_intersected_grid_gdf(cityobject_gdf, elevation_dataset = "aster30m", step = 5): 166 | """ 167 | Mostly, add elevation to a grid 168 | 169 | Parameters 170 | ---------- 171 | cityobject_gdf : GeoDataFrame 172 | UrbanTile or Transportation GeoDataFrame 173 | elevation_dataset : string 174 | Online elevation can be chose from https://www.opentopodata.org 175 | step : int 176 | Grid size 177 | 178 | Returns 179 | ------- 180 | GeoDataFrame 181 | """ 182 | 183 | # Uniform the crs to epsg:4326 and then project it. 184 | if cityobject_gdf.crs != "epsg:4326": 185 | cityobject_gdf = cityobject_gdf.to_crs("epsg:4326") 186 | cityobject_gdf_prj = ox.project_gdf(cityobject_gdf) 187 | 188 | # Make the gird dataframe 189 | _Grid = Grid(data_type= "Polygon", step = 5, bandwidth = 10) 190 | Grid_gdf_prj = _Grid.computed_grid(cityobject_gdf_prj, X_weights = None) 191 | Grid_gdf_prj.crs = cityobject_gdf_prj.crs 192 | 193 | # Keep Point geometry 194 | Grid_gdf_prj["Point"] = Grid_gdf_prj["geometry"] 195 | 196 | # In the projection crs, only keep grid points inside cityobject_gdf_prj 197 | intersected_grid_gdf_prj = gpd.sjoin(cityobject_gdf_prj, Grid_gdf_prj,how="left", op="intersects") 198 | intersected_grid_gdf_prj["geometry"] = intersected_grid_gdf_prj["Point"] 199 | 200 | # For elevaion query, change the crs into epsg:4326 201 | intersected_grid_gdf = intersected_grid_gdf_prj.to_crs("epsg:4326") 202 | intersected_grid_gdf_add_ele = add_gdf_elevation_features(gdf= intersected_grid_gdf, data_type="Point", 203 | elevation_dataset = elevation_dataset) 204 | intersected_grid_gdf_add_ele.crs = "epsg:4326" 205 | 206 | return intersected_grid_gdf_add_ele 207 | 208 | 209 | 210 | 211 | 212 | -------------------------------------------------------------------------------- /osmsc/feature.py: -------------------------------------------------------------------------------- 1 | """Add Polygon-based features.""" 2 | 3 | import math 4 | import requests 5 | import networkx as nx 6 | import numpy as np 7 | import pandas as pd 8 | import geopandas as gpd 9 | import osmnx as ox 10 | 11 | from .utils import create_regularity_gdf 12 | 13 | def add_spatial_semantics_attr(left_gdf,right_gdf,semColName, how="left", op="intersects"): 14 | """ 15 | Explore the spatial semantic relationship between two GeoDataFrame 16 | 17 | Parameters 18 | ---------- 19 | left_gdf : GeoDataFrame 20 | right_gdf : GeoDataFrame 21 | # As for both gdf, more detail cna be found in 22 | # https://automating-gis-processes.github.io/CSC18/lessons/L4/spatial-join.html 23 | 24 | semColName : string e.g. containsBuilding or within Tile 25 | New column name 26 | 27 | Returns 28 | ------- 29 | GeoDataFrame 30 | """ 31 | 32 | # Obtain the osmscID and geometry info 33 | temp_left_gdf = left_gdf[["osmscID","geometry"]] 34 | temp_right_gdf = right_gdf[["osmscID","geometry"]] 35 | # relationship stored in GeoDataFrame 36 | temp_relationship = gpd.sjoin(temp_left_gdf, temp_right_gdf, how = how, op = op) 37 | 38 | relation_list = [] 39 | for left_ID in temp_left_gdf["osmscID"]: 40 | 41 | # right_ID_list for each left objects 42 | try: 43 | right_ID_list = list(temp_relationship[temp_relationship["osmscID_left"] == left_ID].osmscID_right) 44 | except: 45 | right_ID_list = None 46 | 47 | relation_list.append(right_ID_list) 48 | 49 | # Save the semantic info in the left_gdf 50 | left_gdf[semColName] = relation_list 51 | 52 | return left_gdf 53 | 54 | #################### intra layer attrs ###################### 55 | ################## For polygon cityobjects ################## 56 | 57 | def add_minimum_rotated_rectangle_attr(polygon_gdf): 58 | """ 59 | Obtain the minimum rotated rectangle and corresponding attributes for a GeoDataFrame. 60 | 61 | Parameters 62 | ---------- 63 | polygon_gdf : GeoDataFrame 64 | Add minimum_rotated_rectangle_attr into this GeoDataFrame. 65 | Before the operation, the GeoDataFrame need to be projected. 66 | 67 | Returns 68 | ------- 69 | GeoDataFrame 70 | """ 71 | 72 | temp_mrr_gdf = gpd.GeoDataFrame() 73 | temp_mrr_gdf["geometry"] = polygon_gdf["geometry"] 74 | temp_mrr_gdf.crs = polygon_gdf.crs 75 | 76 | mrr_geom = [geom.minimum_rotated_rectangle for geom in list(temp_mrr_gdf["geometry"])] 77 | temp_mrr_gdf["geometry"] = mrr_geom 78 | temp_mrr_gdf["mrr_area"] = temp_mrr_gdf["geometry"].area 79 | 80 | polygon_gdf["mrr_geometry"] = temp_mrr_gdf["geometry"] 81 | polygon_gdf["mrr_area"] = temp_mrr_gdf["mrr_area"] 82 | 83 | return polygon_gdf 84 | 85 | 86 | def add_minimum_circumscribed_circle_attr(polygon_gdf): 87 | """ 88 | Obtain the minimum circumscribed circle and corresponding attributes for a GeoDataFrame. 89 | 90 | Parameters 91 | ---------- 92 | polygon_gdf : GeoDataFrame 93 | Add minimum_circumscribed_circle_attr into this GeoDataFrame. 94 | Before the operation, the GeoDataFrame need to be projected. 95 | 96 | Returns 97 | ------- 98 | GeoDataFrame 99 | """ 100 | temp_mcc_gdf = create_regularity_gdf(polygon_gdf) 101 | # Add attr columns 102 | polygon_gdf["mcc_geometry"] = temp_mcc_gdf["geometry"] 103 | polygon_gdf["mcc_area"] = temp_mcc_gdf["geometry"].area 104 | 105 | return polygon_gdf 106 | 107 | 108 | def add_shape_factor_attr(polygon_gdf): 109 | """ 110 | Obtain the shape factor attribute for a GeoDataFrame. 111 | 112 | Parameters 113 | ---------- 114 | polygon_gdf : GeoDataFrame 115 | Add shape factor attribute into this GeoDataFrame. 116 | Before the operation, the GeoDataFrame need to be projected. 117 | 118 | Returns 119 | ------- 120 | GeoDataFrame 121 | """ 122 | 123 | temp_gdf = polygon_gdf 124 | temp_gdf["Polygon_area"] = temp_gdf["geometry"].area 125 | 126 | # shape factor calculation is based on minimum circumscribed circle geometry 127 | if "mcc_area" not in temp_gdf.columns: 128 | temp_gdf = add_minimum_circumscribed_circle_attr(temp_gdf) 129 | 130 | # Add attr columns 131 | polygon_gdf["shape_factor"] = temp_gdf["Polygon_area"]/temp_gdf["mcc_area"] 132 | # delete unnecessary column 133 | polygon_gdf = polygon_gdf.drop("Polygon_area",axis =1) 134 | 135 | return polygon_gdf 136 | 137 | 138 | def add_polygon_bearing_attr(polygon_gdf_prj, use_mrr = True): 139 | """ 140 | Obtain the polygon bearing attribute for a GeoDataFrame. 141 | 142 | Parameters 143 | ---------- 144 | polygon_gdf_prj : GeoDataFrame 145 | Add polygon bearing attribute into this GeoDataFrame. 146 | use_mrr : bool 147 | True, use the minimum rotated rectangle to represent the current polygon 148 | False, use the current polygon to calculate bearing attributes. 149 | 150 | Returns 151 | ------- 152 | GeoDataFrame 153 | """ 154 | # re-project the gdf_prj 155 | if polygon_gdf_prj.crs != "EPSG:4326": 156 | polygon_gdf = polygon_gdf_prj.to_crs("EPSG:4326") 157 | 158 | # Whether using minimum rotated rectangle 159 | if use_mrr: 160 | mbr_gdf = create_minimum_bounding_rectangle_gdf(polygon_gdf_prj) 161 | else: 162 | mbr_gdf = polygon_gdf 163 | 164 | polygon_brngs = [] 165 | for i in range(len(mbr_gdf)): 166 | # Traverse each geometry 167 | polygon = mbr_gdf.geometry[i] 168 | # Obtain the direction of each side (azimuth angle +90) 169 | brngs_list = get_brngList_for_polygon(polygon) 170 | polygon_brngs.append(brngs_list) 171 | 172 | polygon_gdf_prj["bearings"] = polygon_brngs 173 | 174 | return polygon_gdf_prj 175 | 176 | 177 | def create_minimum_bounding_rectangle_gdf(polygon_gdf): 178 | """ 179 | Obtain the minimum bounding rectangle for a GeoDataFrame. 180 | More detail can be found in 181 | https://gis.stackexchange.com/questions/22895/finding-minimum-area-rectangle-for-given-points 182 | 183 | Parameters 184 | ---------- 185 | polygon_gdf : GeoDataFrame 186 | Add minimum bounding rectangle into this GeoDataFrame. 187 | 188 | Returns 189 | ------- 190 | GeoDataFrame 191 | """ 192 | 193 | mbr_gdf = gpd.GeoDataFrame() 194 | mbr_gdf["geometry"] = [geo.minimum_rotated_rectangle for geo in polygon_gdf["geometry"]] 195 | mbr_gdf.crs = polygon_gdf.crs 196 | 197 | return mbr_gdf 198 | 199 | 200 | def get_brng_btw_points(pointA, pointB): 201 | """ 202 | Calculate the azimuth angle between two points 203 | More detail can be found in 204 | https://blog.csdn.net/qiannianlaoyao2010/article/details/102807883?ops_request_misc=%257B%2522request%255Fid%2522%253A%2522162157829716780366528793%2522%252C%2522scm%2522%253A%252220140713.1 205 | 206 | Parameters 207 | ---------- 208 | pointA : shapely.geometry.Point 209 | pointA : shapely.geometry.Point 210 | 211 | Returns 212 | ------- 213 | Float 214 | """ 215 | 216 | # Point (lon, lat) of GeoDataFrame 217 | lonA, latA = pointA 218 | lonB, latB = pointB 219 | 220 | radLatA = math.radians(latA) 221 | radLonA = math.radians(lonA) 222 | radLatB = math.radians(latB) 223 | radLonB = math.radians(lonB) 224 | dLon = radLonB - radLonA 225 | y = math.sin(dLon) * math.cos(radLatB) 226 | x = math.cos(radLatA) * math.sin(radLatB) - math.sin(radLatA) * math.cos(radLatB) * math.cos(dLon) 227 | brng = math.degrees(math.atan2(y, x)) 228 | brng = round((brng + 360) % 360, 4) 229 | brng = int(brng) 230 | 231 | return brng 232 | 233 | 234 | def get_brngList_for_polygon(polygon): 235 | """ 236 | Calculate the azimuth angle for a certain polygon 237 | 238 | Parameters 239 | ---------- 240 | polygon : shapely.geometry.Polygon 241 | 242 | Returns 243 | ------- 244 | List 245 | """ 246 | 247 | brngs_list = [] 248 | 249 | coords = list(polygon.boundary.coords) 250 | for i in range(len(coords)): 251 | if i != len(coords)-1: 252 | pointA = coords[i] 253 | pointB = coords[i+1] 254 | 255 | # Because the polygon nodes are connected counterclockwise, 256 | # the normal of the breath degree is rotated by 90 257 | brng = (get_brng_btw_points(pointA, pointB) + 90 ) % 360 258 | 259 | brngs_list.append(brng) 260 | 261 | return brngs_list 262 | 263 | 264 | 265 | ################## inter layer attrs ################## 266 | def add_interlayer_building_attr(UrbanTile_gdf, Building_gdf): 267 | """ 268 | Add interlayer building attrs to gdf, mostly to UrbanTile_gdf. 269 | The prerequisite is the spatial semantic relationship between two layer is known, namely, 270 | add_spatial_semantics_attr() has been run. 271 | 272 | Parameters 273 | ---------- 274 | UrbanTile_gdf : GeoDataFrame 275 | Add interlayer building attrs into this GeoDataFrame. 276 | Building_gdf : GeoDataFrame 277 | 278 | 279 | Returns 280 | ------- 281 | GeoDataFrame 282 | """ 283 | 284 | buildingDensity_list = [] 285 | avg_buildingHeight_list = [] 286 | avg_buildingArea_list = [] 287 | avg_buildingPerimeter_list = [] 288 | 289 | for row_num in range(len(UrbanTile_gdf)): 290 | 291 | # For one UrbanTile object 292 | ########## Avergae height/area/perimeter of buildings ########### 293 | bldg_height_list = [] 294 | bldg_area_list = [] 295 | bldg_perimeter_list = [] 296 | 297 | for bldg_ID in UrbanTile_gdf["containsBuilding"][row_num]: 298 | 299 | try: 300 | # Building_height 301 | a_height = float(Building_gdf[Building_gdf["osmscID"] == bldg_ID].Building_height) 302 | bldg_height_list.append(a_height) 303 | except: 304 | # some buildings do not have height 305 | bldg_height_list.append(0) 306 | 307 | try: 308 | a_area = float(Building_gdf[Building_gdf["osmscID"] == bldg_ID].Building_area) 309 | bldg_area_list.append(a_area) 310 | 311 | a_perimeter = float(Building_gdf[Building_gdf["osmscID"] == bldg_ID].Building_perimeter) 312 | bldg_perimeter_list.append(a_perimeter) 313 | 314 | except: 315 | # Some buildings need to add more area and perimeter attrs. 316 | bldg_area_list.append(0) 317 | bldg_perimeter_list.append(0) 318 | 319 | avg_buildingHeight_list.append(np.mean(bldg_height_list)) 320 | avg_buildingArea_list.append(np.mean(bldg_area_list)) 321 | avg_buildingPerimeter_list.append(np.mean(bldg_perimeter_list)) 322 | 323 | UrbanTile_area = float(UrbanTile_gdf["UrbanTile_area"].loc[row_num]) 324 | buildingDensity_list.append(np.sum(bldg_area_list)/UrbanTile_area) 325 | 326 | # Add attrs columns 327 | UrbanTile_gdf["buildingDensity"] = buildingDensity_list 328 | UrbanTile_gdf["avg_buildingHeight"] = avg_buildingHeight_list 329 | UrbanTile_gdf["avg_buildingArea"] = avg_buildingArea_list 330 | UrbanTile_gdf["avg_buildingPerimeter"] = avg_buildingPerimeter_list 331 | 332 | return UrbanTile_gdf 333 | 334 | 335 | def add_interlayer_vegetation_attr(UrbanTile_gdf, Vegetation_gdf): 336 | """ 337 | Add interlayer vegetation attrs to gdf, mostly to UrbanTile_gdf. 338 | The prerequisite is the spatial semantic relationship between two layer is known, namely, 339 | add_spatial_semantics_attr() has been run. 340 | 341 | Parameters 342 | ---------- 343 | UrbanTile_gdf : GeoDataFrame 344 | Add interlayer vegetation attrs into this GeoDataFrame. 345 | Building_gdf : GeoDataFrame 346 | 347 | 348 | Returns 349 | ------- 350 | GeoDataFrame 351 | """ 352 | vegetationDensity_list = [] 353 | avg_vegetationArea_list = [] 354 | avg_vegetationPerimeter_list = [] 355 | 356 | for row_num in range(len(UrbanTile_gdf)): 357 | 358 | # For one UrbanTile object 359 | ########## Avergae area/perimeter of vegetation objects ########### 360 | 361 | veg_area_list = [] 362 | veg_perimeter_list = [] 363 | 364 | for veg_ID in UrbanTile_gdf["containsVegetation"][row_num]: 365 | 366 | try: 367 | 368 | a_area = float(Vegetation_gdf[Vegetation_gdf["osmscID"] == veg_ID].Vegetation_area) 369 | veg_area_list.append(a_area) 370 | 371 | a_perimeter = float(Vegetation_gdf[Vegetation_gdf["osmscID"] == veg_ID].Vegetation_perimeter) 372 | veg_perimeter_list.append(a_perimeter) 373 | 374 | except: 375 | veg_area_list.append(0) 376 | veg_perimeter_list.append(0) 377 | 378 | avg_vegetationArea_list.append(np.mean(veg_area_list)) 379 | avg_vegetationPerimeter_list.append(np.mean(veg_perimeter_list)) 380 | 381 | UrbanTile_area = float(UrbanTile_gdf["UrbanTile_area"].loc[row_num]) 382 | vegetationDensity_list.append(np.sum(veg_area_list)/UrbanTile_area) 383 | 384 | # Add attrs columns 385 | UrbanTile_gdf["vegetationDensity"] = vegetationDensity_list 386 | UrbanTile_gdf["avg_vegetationArea"] = avg_vegetationArea_list 387 | UrbanTile_gdf["avg_vegetationPerimeter"] = avg_vegetationPerimeter_list 388 | 389 | return UrbanTile_gdf 390 | 391 | 392 | def add_interlayer_waterbody_attr(UrbanTile_gdf, Waterbody_gdf): 393 | """ 394 | Add interlayer waterbody attrs to gdf, mostly to UrbanTile_gdf. 395 | The prerequisite is the spatial semantic relationship between two layer is known, namely, 396 | add_spatial_semantics_attr() has been run. 397 | 398 | Parameters 399 | ---------- 400 | UrbanTile_gdf : GeoDataFrame 401 | Add interlayer waterbody attrs into this GeoDataFrame. 402 | Building_gdf : GeoDataFrame 403 | 404 | 405 | Returns 406 | ------- 407 | GeoDataFrame 408 | """ 409 | waterbodyDensity_list = [] 410 | avg_waterbodyArea_list = [] 411 | avg_waterbodyPerimeter_list = [] 412 | 413 | for row_num in range(len(UrbanTile_gdf)): 414 | 415 | # For one UrbanTile object 416 | ########## Avergae area/perimeter of waterbody objects ########### 417 | 418 | wat_area_list = [] 419 | wat_perimeter_list = [] 420 | 421 | for wat_ID in UrbanTile_gdf["containsWaterbody"][row_num]: 422 | 423 | try: 424 | 425 | a_area = float(Waterbody_gdf[Waterbody_gdf["osmscID"] == wat_ID].Waterbody_area) 426 | wat_area_list.append(a_area) 427 | 428 | a_perimeter = float(Waterbody_gdf[Waterbody_gdf["osmscID"] == wat_ID].Waterbody_perimeter) 429 | wat_perimeter_list.append(a_perimeter) 430 | 431 | except: 432 | wat_area_list.append(0) 433 | wat_perimeter_list.append(0) 434 | 435 | avg_waterbodyArea_list.append(np.mean(wat_area_list)) 436 | avg_waterbodyPerimeter_list.append(np.mean(wat_perimeter_list)) 437 | 438 | UrbanTile_area = float(UrbanTile_gdf["UrbanTile_area"].loc[row_num]) 439 | waterbodyDensity_list.append(np.sum(wat_area_list)/UrbanTile_area) 440 | 441 | # Add attrs columns 442 | UrbanTile_gdf["waterbodyDensity"] = waterbodyDensity_list 443 | UrbanTile_gdf["avg_waterbodyArea"] = avg_waterbodyArea_list 444 | UrbanTile_gdf["avg_waterbodyPerimeter"] = avg_waterbodyPerimeter_list 445 | 446 | return UrbanTile_gdf 447 | 448 | ################## Elevation ################## 449 | 450 | def add_graph_elevation_features(G, elevation_dataset= None, max_locations_per_batch=100): 451 | """ 452 | Add graph elevation features to Graph object. 453 | 454 | Editted from osmnx https://github.com/gboeing/osmnx 455 | Open Topo Data Dataset: https://www.opentopodata.org/#public-api 456 | 457 | Parameters 458 | ---------- 459 | G : Graph 460 | elevation_dataset : string 461 | Online elevation can be chose from https://www.opentopodata.org 462 | max_locations_per_batch : int 463 | 464 | Returns 465 | ------- 466 | Graph 467 | """ 468 | 469 | # editted from osmnx 470 | # Open Topo Data Dataset: https://www.opentopodata.org/#public-api 471 | 472 | if elevation_dataset is None: 473 | url_template = 'https://api.opentopodata.org/v1/aster30m?locations={}' 474 | else: 475 | url_template = 'https://api.opentopodata.org/v1/{}?locations={}'.format(elevation_dataset,{}) 476 | 477 | 478 | node_points = pd.Series({node:'{:.5f},{:.5f}'.format(data['y'], data['x']) for node, data in G.nodes(data=True)}) 479 | 480 | # API format is locations=lat,lng|lat,lng|lat,lng|lat,lng... 481 | 482 | results = [] 483 | for i in range(0, len(node_points), max_locations_per_batch): 484 | chunk = node_points.iloc[i : i + max_locations_per_batch] 485 | locations = '|'.join(chunk) 486 | url = url_template.format(locations) 487 | 488 | try: 489 | # request the elevations from the API 490 | response = requests.get(url) 491 | response_json = response.json() 492 | 493 | except: 494 | print('Server responded with {}: {}'.format(response.status_code, response.reason)) 495 | 496 | # append these elevation results to the list of all results 497 | results.extend(response_json['results']) 498 | 499 | # sanity check that all our vectors have the same number of elements 500 | if not (len(results) == len(G.nodes()) == len(node_points)): 501 | raise Exception('Graph has {} nodes but we received {} results from the elevation API.'.format(len(G.nodes()), len(results))) 502 | else: 503 | print('Graph has {} nodes and we received {} results from the elevation API.'.format(len(G.nodes()), len(results))) 504 | 505 | # add elevation as an attribute to the nodes 506 | df = pd.DataFrame(node_points, columns=['node_points']) 507 | df['elevation'] = [result['elevation'] for result in results] 508 | df['elevation'] = df['elevation'].round(3) # round to millimeter 509 | nx.set_node_attributes(G, name='elevation', values=df['elevation'].to_dict()) 510 | 511 | # print('Added elevation data to all nodes.') 512 | 513 | return G 514 | 515 | 516 | def add_gdf_elevation_features(gdf, elevation_dataset = None, data_type = None, max_locations_per_batch=100): 517 | """ 518 | Add gdf elevation features to GeoDataFrame. 519 | Open Topo Data Dataset: https://www.opentopodata.org/#public-api 520 | 521 | Parameters 522 | ---------- 523 | gdf : GeoDataFrame 524 | elevation_dataset : string 525 | Online elevation can be chose from https://www.opentopodata.org 526 | data_type : string 527 | "Polygon" or "Point" from shape.geometry 528 | max_locations_per_batch : int 529 | 530 | Returns 531 | ------- 532 | GeoDataFrame 533 | """ 534 | 535 | if elevation_dataset is None: 536 | url_template = 'https://api.opentopodata.org/v1/aster30m?locations={}' 537 | else: 538 | url_template = 'https://api.opentopodata.org/v1/{}?locations={}'.format(elevation_dataset,{}) 539 | 540 | # For polygon object 541 | # Make a pandas series of all the nodes' coordinates as 'lat,lng' 542 | if data_type == "Polygon": 543 | 544 | gdf_prj = ox.project_gdf(gdf) 545 | centroid_prj = gdf_prj["geometry"].centroid 546 | gdf["centroid"] = centroid_prj.to_crs("EPSG:4326") 547 | gdf["centroid_y_lat"] = gdf["centroid"].y 548 | gdf["centroid_x_lng"] = gdf["centroid"].x 549 | 550 | points = pd.Series('{:.5f},{:.5f}'.format(gdf.iloc[i].centroid_y_lat, gdf.iloc[i].centroid_x_lng) for i in range(len(gdf))) 551 | 552 | elif data_type == "Point": 553 | gdf["y_lat"] = gdf.geometry.y 554 | gdf["x_lng"] = gdf.geometry.x 555 | points = pd.Series('{:.5f},{:.5f}'.format(gdf.iloc[i].y_lat, gdf.iloc[i].x_lng) for i in range(len(gdf))) 556 | 557 | elif data_type is None: 558 | print("Please input the data_type of geometry column") 559 | 560 | # Store the elevation results. 561 | results = [] 562 | for i in range(0, len(points), max_locations_per_batch): 563 | chunk = points.iloc[i : i + max_locations_per_batch] 564 | locations = '|'.join(chunk) 565 | url = url_template.format(locations) 566 | response = requests.get(url) 567 | response_json = response.json() 568 | 569 | results.extend(response_json['results']) 570 | 571 | elevation = [results[i]['elevation'] for i in range(len(results))] 572 | 573 | gdf["ground_elevation"] = elevation 574 | 575 | return gdf 576 | 577 | 578 | 579 | 580 | 581 | 582 | 583 | 584 | -------------------------------------------------------------------------------- /osmsc/cityjson.py: -------------------------------------------------------------------------------- 1 | """ Construct and output CityJSON object """ 2 | 3 | import json 4 | from cjio import cityjson 5 | from cjio.models import CityObject, Geometry 6 | from .utils import fill_nan_list_position 7 | 8 | 9 | class city_json(object): 10 | """ 11 | Create CityJSON-schema objects, including building, vegetation, waterbody 12 | transportation and urban Tile objects. 13 | 14 | More info about CityJSON can be found in 15 | https://www.cityjson.org/specs/1.0.1/ 16 | """ 17 | 18 | def __init__(self, building_gdf = None , vegetation_gdf = None, 19 | waterbody_gdf = None, transportation_gdf = None, urban_Tile_gdf = None): 20 | 21 | self.building_gdf = building_gdf 22 | self.vegetation_gdf = vegetation_gdf 23 | self.waterbody_gdf = waterbody_gdf 24 | self.transportation_gdf = transportation_gdf 25 | self.urban_Tile_gdf = urban_Tile_gdf 26 | 27 | # create an empty CityModel 28 | self.cm = cityjson.CityJSON() 29 | 30 | 31 | def create_building_object(self, lod=1): 32 | """ 33 | Create CityJSON objects for all buildings 34 | Main class variable: self.building_gdf, self.cm 35 | 36 | Returns 37 | ------- 38 | cm : cityjson.CityJSON 39 | """ 40 | print("Building") 41 | # Create a CityJSON object for each building 42 | for b_index in range(len(self.building_gdf)): 43 | 44 | # Create empty building CityJSON object 45 | buildingObject = CityObject(id = str(self.building_gdf.iloc[b_index].osmscID)) 46 | 47 | temp_gdf = self.building_gdf.set_index(self.building_gdf["osmscID"]) 48 | attr_dict= json.loads(temp_gdf.to_json()) 49 | 50 | # Add OSM tags 51 | if "tags" in list(self.building_gdf.columns): 52 | original_attr = attr_dict["features"][b_index]["properties"] 53 | osm_tag = {"osm" + str(key): val for key, val in original_attr["tags"].items()} 54 | del original_attr["tags"] 55 | original_attr.update(osm_tag) 56 | else: 57 | original_attr = attr_dict['features'][b_index]['properties'] 58 | 59 | buildingObject.attributes = original_attr 60 | 61 | #######################################Geometry############################ 62 | b_geom = Geometry(type='Solid', lod=lod) 63 | 64 | building_height = float(self.building_gdf.iloc[b_index].Building_height) 65 | building_poly = self.building_gdf.iloc[b_index].geometry 66 | 67 | # Checking if vertices of polygon are in counter-clockwise 是否是逆时针 68 | building_poly_ccw = building_poly.exterior.is_ccw 69 | 70 | # Extract the point values that define the perimeter of the polygon 71 | x, y = building_poly.exterior.coords.xy 72 | x_list = list(x) 73 | y_list = list(y) 74 | 75 | # bottom surface 76 | bottom_sur = [] 77 | # top surface 78 | top_sur = [] 79 | # side surfaces 80 | side_sur = [] 81 | 82 | for i in range(len(x_list)-1): 83 | 84 | # The coordinates of the polygon are counterclockwise 85 | if building_poly_ccw: 86 | top_sur.append((x_list[i],y_list[i],building_height)) 87 | bottom_sur.append((x_list[-i-2],y_list[-i-2],0)) 88 | 89 | side_coor = [(x_list[i],y_list[i],0), (x_list[i+1],y_list[i+1],0), 90 | (x_list[i+1],y_list[i+1],building_height), (x_list[i],y_list[i],building_height)] 91 | 92 | side_sur.append([side_coor]) 93 | 94 | # The coordinates of polygon are clockwise 95 | else: 96 | top_sur.append((x_list[-i-2],y_list[-i-2],building_height)) 97 | bottom_sur.append((x_list[i],y_list[i],0)) 98 | 99 | side_coor = [(x_list[i],y_list[i],building_height),(x_list[i+1],y_list[i+1],building_height), 100 | (x_list[i+1],y_list[i+1],0),(x_list[i],y_list[i],0)] 101 | 102 | side_sur.append([side_coor]) 103 | 104 | # Building boundary and geometry 105 | b_bdry = [[top_sur],[bottom_sur]] + side_sur 106 | b_geom.boundaries.append(b_bdry) 107 | 108 | if lod > 1: 109 | # Revised-2023-4-30 110 | # Only for LoD1 building surface semantics 111 | side_index_list = side_index_list = [[0,i+2] for i in range(len(b_bdry)-2)] # [[0,2], [0,3], [0,4], [0,5]] 112 | 113 | b_geom.surfaces[0] = {'surface_idx': [[0,0]], 'type': 'RoofSurface'} 114 | b_geom.surfaces[1] = {'surface_idx': [[0,1]], 'type': 'GroundSurface'} 115 | b_geom.surfaces[2] = {'surface_idx': side_index_list, 'type': 'WallSurface'} 116 | 117 | # Add propertries to CityJSON objects 118 | buildingObject.geometry.append(b_geom) 119 | buildingObject.type = "Building" 120 | 121 | self.cm.cityobjects[buildingObject.id] = buildingObject 122 | 123 | return self.cm 124 | 125 | def create_vegetation_object(self): 126 | """ 127 | Create CityJSON objects for all vegetation objects 128 | Main class variable: self.vegetation_gdf, self.cm 129 | 130 | Returns 131 | ------- 132 | cm : cityjson.CityJSON 133 | """ 134 | print("Vegetation") 135 | # # Create a CityJSON object for each vegetation object 136 | for v_index in range(len(self.vegetation_gdf)): 137 | 138 | vegetationObject = CityObject(id = str(self.vegetation_gdf.iloc[v_index].osmscID)) 139 | 140 | temp_gdf = self.vegetation_gdf.set_index(self.vegetation_gdf["osmscID"]) 141 | attr_dict= json.loads(temp_gdf.to_json()) 142 | 143 | # Add OSM tags 144 | if "tags" in list(self.vegetation_gdf.columns): 145 | original_attr = attr_dict["features"][v_index]["properties"] 146 | osm_tag = {"osm" + str(key): val for key, val in original_attr["tags"].items()} 147 | del original_attr["tags"] 148 | original_attr.update(osm_tag) 149 | else: 150 | original_attr = attr_dict['features'][v_index]['properties'] 151 | 152 | vegetationObject.attributes = original_attr 153 | 154 | #######################################Geometry############################ 155 | v_geom = Geometry(type='MultiSurface', lod=0) 156 | 157 | vegetation_poly = self.vegetation_gdf.iloc[v_index].geometry 158 | 159 | try: 160 | # Don't consider the content of MultiPolygon at present, 161 | # and do it separately later 162 | 163 | # Checking if vertices of polygon are in counter-clockwise 是否是逆时针 164 | vegetation_poly_ccw = vegetation_poly.exterior.is_ccw 165 | 166 | # Extract the point values that define the perimeter of the polygon 167 | x, y = vegetation_poly.exterior.coords.xy 168 | x_list = list(x) 169 | y_list = list(y) 170 | top_sur = [] 171 | 172 | for i in range(len(x_list)-1): 173 | # The coordinates of the polygon are counterclockwise 174 | if vegetation_poly_ccw: 175 | top_sur.append((x_list[i],y_list[i],0)) 176 | # The coordinates of polygon are clockwise 177 | else: 178 | top_sur.append((x_list[-i-2],y_list[-i-2],0)) 179 | 180 | # Building boundary and geometry 181 | v_bdry = [top_sur] 182 | v_geom.boundaries.append(v_bdry) 183 | 184 | # Add propertries to CityJSON objects 185 | vegetationObject.geometry.append(v_geom) 186 | vegetationObject.type = "PlantCover" 187 | 188 | self.cm.cityobjects[vegetationObject.id] = vegetationObject 189 | 190 | except: 191 | pass 192 | 193 | return self.cm 194 | 195 | def create_waterbody_object(self): 196 | """ 197 | Create CityJSON objects for all waterbody objects 198 | Main class variable: self.waterbody_gdf, self.cm 199 | 200 | Returns 201 | ------- 202 | cm : cityjson.CityJSON 203 | """ 204 | print("Waterbody") 205 | # Create a CityJSON object for each waterbody 206 | for w_index in range(len(self.waterbody_gdf)): 207 | 208 | # Create empty waterbody CityJSON object 209 | waterbodyObject = CityObject(id= str(self.waterbody_gdf.iloc[w_index].osmscID)) 210 | 211 | temp_gdf = self.waterbody_gdf.set_index(self.waterbody_gdf["osmscID"]) 212 | attr_dict= json.loads(temp_gdf.to_json()) 213 | 214 | # Add OSM tags 215 | if "tags" in list(self.waterbody_gdf.columns): 216 | original_attr = attr_dict["features"][w_index]["properties"] 217 | osm_tag = {"osm" + str(key): val for key, val in original_attr["tags"].items()} 218 | del original_attr["tags"] 219 | original_attr.update(osm_tag) 220 | else: 221 | original_attr = attr_dict['features'][w_index]['properties'] 222 | 223 | waterbodyObject.attributes = original_attr 224 | 225 | #######################################Geometry############################ 226 | w_geom = Geometry(type='CompositeSurface', lod=0) 227 | 228 | waterbody_poly = self.waterbody_gdf.iloc[w_index].geometry 229 | 230 | # Checking if vertices of polygon are in counter-clockwise 是否是逆时针 231 | waterbody_poly_ccw = waterbody_poly.exterior.is_ccw 232 | 233 | # Extract the point values that define the perimeter of the polygon 234 | x, y = waterbody_poly.exterior.coords.xy 235 | x_list = list(x) 236 | y_list = list(y) 237 | 238 | top_sur = [] 239 | 240 | for i in range(len(x_list)-1): 241 | # The coordinates of the polygon are counterclockwise 242 | if waterbody_poly_ccw: 243 | top_sur.append((x_list[i],y_list[i],0)) 244 | # The coordinates of polygon are clockwise 245 | else: 246 | top_sur.append((x_list[-i-2],y_list[-i-2],0)) 247 | 248 | # Building boundary and geometry 249 | w_bdry = [top_sur] 250 | w_geom.boundaries.append(w_bdry) 251 | 252 | # Add propertries to CityJSON objects 253 | waterbodyObject.geometry.append(w_geom) 254 | waterbodyObject.type = "WaterBody" 255 | 256 | self.cm.cityobjects[waterbodyObject.id] = waterbodyObject 257 | return self.cm 258 | 259 | def create_transportation_object(self): 260 | """ 261 | Create CityJSON objects for all transportation objects 262 | Main class variable: self.transportation_gdf, self.cm 263 | 264 | Returns 265 | ------- 266 | cm : cityjson.CityJSON 267 | """ 268 | print("Transportation") 269 | # Create a CityJSON object for each transportation object 270 | # transportation_object refers to the street currently 271 | for r_index in range(len(self.transportation_gdf)): 272 | # Create empty transportation CityJSON object 273 | roadObject = CityObject(id = str(self.transportation_gdf.iloc[r_index].osmscID)) 274 | 275 | temp_gdf = self.transportation_gdf.set_index(self.transportation_gdf["osmscID"]) 276 | attr_dict= json.loads(temp_gdf.to_json()) 277 | 278 | # Add OSM tags 279 | # if "tags" in list(self.transportation_gdf.columns): 280 | # original_attr = attr_dict["features"][r_index]["properties"] 281 | # osm_tag = {"osm" + str(key): val for key, val in original_attr["tags"].items()} 282 | # del original_attr["tags"] 283 | # original_attr.update(osm_tag) 284 | # else: 285 | # original_attr = attr_dict['features'][r_index]['properties'] 286 | 287 | original_attr = attr_dict['features'][r_index]['properties'] 288 | roadObject.attributes = original_attr 289 | 290 | #######################################Geometry############################ 291 | r_geom = Geometry(type='CompositeSurface', lod=0) 292 | 293 | road_poly = self.transportation_gdf.iloc[r_index].geometry 294 | 295 | try: 296 | # Don't consider the content of MultiPolygon at present, 297 | # and do it separately later 298 | 299 | # Checking if vertices of polygon are in counter-clockwise 是否是逆时针 300 | road_poly_ccw = road_poly.exterior.is_ccw 301 | 302 | # Extract the point values that define the perimeter of the polygon 303 | x, y = road_poly.exterior.coords.xy 304 | x_list = list(x) 305 | y_list = list(y) 306 | 307 | top_sur = [] 308 | 309 | for i in range(len(x_list)-1): 310 | # The coordinates of the polygon are counterclockwise 311 | if road_poly_ccw: 312 | top_sur.append((x_list[i],y_list[i],0)) 313 | # The coordinates of polygon are clockwise 314 | else: 315 | top_sur.append((x_list[-i-2],y_list[-i-2],0)) 316 | 317 | # Building boundary and geometry 318 | r_bdry = [top_sur] 319 | r_geom.boundaries.append(r_bdry) 320 | 321 | # Add propertries to CityJSON objects 322 | roadObject.geometry.append(r_geom) 323 | roadObject.type = "Road" 324 | 325 | self.cm.cityobjects[roadObject.id] = roadObject 326 | 327 | except: 328 | pass 329 | return self.cm 330 | 331 | def create_urban_Tile_object(self): 332 | """ 333 | Create CityJSON objects for all urban Tile objects 334 | Main class variable: self.urban_Tile_gdf, self.cm 335 | 336 | Returns 337 | ------- 338 | cm : cityjson.CityJSON 339 | """ 340 | 341 | print("UrbanTile") 342 | 343 | # Create a CityJSON object for each urban Tile object 344 | for u_index in range(len(self.urban_Tile_gdf)): 345 | # Create empty urban Tile CityJSON object 346 | urbanTileObject = CityObject(id=str(self.urban_Tile_gdf.iloc[u_index].osmscID)) 347 | 348 | temp_gdf = self.urban_Tile_gdf.set_index(self.urban_Tile_gdf["osmscID"]) 349 | attr_dict= json.loads(temp_gdf.to_json()) 350 | 351 | # There is no OSM tag. 352 | u_attrs = attr_dict['features'][u_index]['properties'] 353 | urbanTileObject.attributes = u_attrs 354 | 355 | #######################################Geometry############################ 356 | u_geom = Geometry(type='CompositeSurface', lod=0) 357 | 358 | urbanTile_poly = self.urban_Tile_gdf.iloc[u_index].geometry 359 | 360 | try: 361 | # Don't consider the content of MultiPolygon at present, 362 | # and do it separately later 363 | 364 | # Checking if vertices of polygon are in counter-clockwise 是否是逆时针 365 | urbanTile_poly_ccw = urbanTile_poly.exterior.is_ccw 366 | 367 | # Extract the point values that define the perimeter of the polygon 368 | x, y = urbanTile_poly.exterior.coords.xy 369 | 370 | x_list = list(x) 371 | y_list = list(y) 372 | 373 | top_sur = [] 374 | 375 | for i in range(len(x_list)-1): 376 | # The coordinates of the polygon are counterclockwise 377 | if urbanTile_poly_ccw: 378 | top_sur.append((x_list[i],y_list[i],0)) 379 | # The coordinates of polygon are clockwise 380 | else: 381 | top_sur.append((x_list[-i-2],y_list[-i-2],0)) 382 | 383 | # Building boundary and geometry 384 | u_bdry = [top_sur] 385 | u_geom.boundaries.append(u_bdry) 386 | 387 | # Add propertries to CityJSON objects 388 | urbanTileObject.geometry.append(u_geom) 389 | urbanTileObject.type = "GenericCityObject" 390 | 391 | self.cm.cityobjects[urbanTileObject.id] = urbanTileObject 392 | 393 | except: 394 | pass 395 | 396 | return self.cm 397 | 398 | def drop_extra_geom(self, gdf): 399 | """ 400 | Drop extra geometry columns in GeoDataframe 401 | As the GeoDataframe.to_json() function only works for the GeoDataframe that have 402 | one geometry column 403 | 404 | Parameters 405 | ---------- 406 | gdf : geopandas.GeoDataFrame 407 | mostly building_gdf and urban_Tile_gdf 408 | 409 | Returns 410 | ------- 411 | gdf : geopandas.GeoDataFrame 412 | """ 413 | 414 | if 'mrr_geometry' in gdf.columns: 415 | gdf = gdf.drop("mrr_geometry",axis =1) 416 | if 'mcc_geometry' in gdf.columns: 417 | gdf = gdf.drop("mcc_geometry",axis =1) 418 | 419 | return gdf 420 | 421 | def output_json(self,filename = "cityName",building_lod = 1): 422 | """ 423 | Output CityJSON object into a JSON file. 424 | Main class variable: self.building_gdf, self.vegetation_gdf, self.waterbody_gdf 425 | self.transportation_gdf, self.urban_Tile_gdf, self.cm 426 | 427 | Parameters 428 | ---------- 429 | filename : str 430 | the filename to be saved in the current project location 431 | building_lod : int 432 | CityGML/CityJSON LoD of building objects 433 | """ 434 | 435 | 436 | print("Please wait a few minutes") 437 | 438 | # pre cleaning work 439 | # fill spatial semantic attrs 440 | # make each gdf has only one shapely column 441 | # Build city objects 442 | 443 | ################# Building ################# 444 | if self.building_gdf is not None: 445 | self.building_gdf = fill_nan_list_position(self.building_gdf,"withinTile") 446 | self.building_gdf = fill_nan_list_position(self.building_gdf,"Building_height") 447 | 448 | self.building_gdf = self.drop_extra_geom(self.building_gdf) 449 | self.cm = self.create_building_object(lod = building_lod) 450 | 451 | ################# Vegetation ################# 452 | if self.vegetation_gdf is not None: 453 | self.vegetation_gdf = fill_nan_list_position(self.vegetation_gdf,"withinTile") 454 | self.cm = self.create_vegetation_object() 455 | 456 | ################# Waterbody ################# 457 | if self.vegetation_gdf is not None: 458 | self.waterbody_gdf = fill_nan_list_position(self.waterbody_gdf,"withinTile") 459 | self.cm = self.create_waterbody_object() 460 | 461 | 462 | ################# Transportation ################# 463 | if self.transportation_gdf is not None: 464 | self.transportation_gdf = fill_nan_list_position(self.transportation_gdf,"AdjacentTile") 465 | self.cm = self.create_transportation_object() 466 | 467 | ################# UrbanTile ################# 468 | if self.urban_Tile_gdf is not None: 469 | self.urban_Tile_gdf = fill_nan_list_position(self.urban_Tile_gdf, "containsVegetation") 470 | self.urban_Tile_gdf = fill_nan_list_position(self.urban_Tile_gdf, "containsBuilding") 471 | self.urban_Tile_gdf = fill_nan_list_position(self.urban_Tile_gdf, "containsWaterbody") 472 | self.urban_Tile_gdf = fill_nan_list_position(self.urban_Tile_gdf, "AdjacentTransportation") 473 | 474 | self.urban_Tile_gdf = self.drop_extra_geom(self.urban_Tile_gdf) 475 | 476 | self.cm = self.create_urban_Tile_object() 477 | 478 | 479 | # reference_geometry 480 | print("reference_geometry") 481 | cityobjects, vertex_lookup = self.cm.reference_geometry() 482 | # multipoint multilinestring multisurface compositesurface 483 | # solid multisolid compositesolid 484 | print("add to json") 485 | self.cm.add_to_j(cityobjects,vertex_lookup) 486 | # cm.j["CityObjects"] 487 | 488 | print("update bbox") 489 | self.cm.update_bbox() 490 | 491 | # https://www.cityjson.org/tutorials/validation/ 492 | self.cm.validate() 493 | 494 | # save the CityJSON file 495 | cityjson.save(self.cm, filename + '.json') 496 | 497 | 498 | 499 | 500 | -------------------------------------------------------------------------------- /osmsc/cityobject.py: -------------------------------------------------------------------------------- 1 | """Construct OSMsc objects""" 2 | 3 | import numpy as np 4 | import pandas as pd 5 | import geopandas as gpd 6 | import osmnx as ox 7 | from shapely.ops import polygonize 8 | import time 9 | 10 | from .utils import download, street_graph_from_gdf, graph_from_gdfs, json_to_gdf, streets_from_street_graph 11 | from .geogroup import point_group, polygon_group, line_string_group 12 | 13 | #TODO logging 14 | 15 | 16 | ################################################################################# 17 | 18 | class building_group(polygon_group): 19 | """ 20 | Construct OSMsc building objects 21 | """ 22 | def query(self): 23 | """ 24 | Query objects from OpenStreetMap via Overpass API 25 | 26 | Returns 27 | ------- 28 | JSON 29 | """ 30 | # Default query 31 | # query buildings with all kinds of tags 32 | # key is "building" 33 | if not self.overpass_query: 34 | 35 | # print("The default setting is to query all buildings, if necessary, please enter your BuildingGroup overpass api query!!") 36 | # not including underground structure 37 | 38 | # if self.bbox: 39 | self.overpass_query = self.get_overpass_query() 40 | 41 | 42 | # return data in JSON 43 | return download(self.overpass_query) 44 | 45 | def get_overpass_query(self, lat = None, lon = None ): 46 | """ 47 | Get overpass query content from bbox or polygon boundary 48 | 49 | Parameters 50 | ---------- 51 | lat : list 52 | latitude list of polygon boundary nodes 53 | 54 | lon : list 55 | longitude list of polygon boundary nodes 56 | 57 | Returns 58 | ------- 59 | String 60 | """ 61 | 62 | if self.bbox: 63 | # Default query 64 | # query buildings with all kinds of tags 65 | # key is "building" 66 | self.overpass_query = """ 67 | [out:json][timeout:5000]; 68 | ( way["building"][!"building:levels:underground"]""" + str(self.bbox) + """; 69 | ); 70 | out geom; 71 | """ 72 | return self.overpass_query 73 | 74 | if self.place_name: 75 | overpass_poly = "" 76 | for i in range(len(lon)): 77 | overpass_poly = overpass_poly + str(lat[i]) + " " + str(lon[i]) + " " 78 | 79 | self.overpass_query = """[out:json][timeout:5000]; 80 | ( way['building'][!"building:levels:underground"] 81 | (poly:'""" + str(overpass_poly.rstrip()) + """'); 82 | ); 83 | out geom;""" 84 | 85 | return self.overpass_query 86 | 87 | def read_file(self): 88 | """ 89 | Read the local building datasets (if any) 90 | 91 | Returns 92 | ------- 93 | GeoDataFrame 94 | """ 95 | 96 | return gpd.GeoDataFrame.from_file(self.file_path) 97 | 98 | def get_gdf(self, tags = True, building_levels = False, height = False, sim_factor = 0.0005): 99 | """ 100 | Obtain OSM data and save as GeoDataFrame. 101 | 102 | Parameters 103 | ---------- 104 | tags : bool 105 | if True, need to extract tag info into current GeoDataFrame 106 | if False, the GeoDataFrame won't add OSM "tags" column. 107 | 108 | building_levels : bool 109 | if True, need to download building level into current GeoDataFrame 110 | if False, the GeoDataFrame won't add OSM "building_levels" column. 111 | 112 | height : bool 113 | if True, need to download building height into current GeoDataFrame, 114 | if OSM lacks such info, the default height is 3m 115 | if False, the GeoDataFrame won't add OSM "Building_height" column. 116 | 117 | sim_factor : float 118 | Factor of shapely geometry simplify function 119 | If there are still downloading errors (returns nothing), pls choose a larger sim_factor. 120 | 121 | Returns 122 | ------- 123 | GeoDataFrame 124 | """ 125 | # given place name exists 126 | if self.place_name: 127 | poly_gdf = ox.geocoder.geocode_to_gdf(self.place_name) 128 | 129 | temp_gdf_1 = gpd.GeoDataFrame() 130 | 131 | for i in range(len(poly_gdf.geometry.iloc[0])): 132 | 133 | lon, lat = poly_gdf.geometry.iloc[0][i].exterior.coords.xy 134 | time.sleep( 5 ) 135 | 136 | # Too many nodes would result in downloading error. 137 | if len(lon) < 200: 138 | #osm_json = get_osm_json_with_latlon(lat, lon) 139 | self.overpass_query = self.get_overpass_query( lat = lat, lon = lon ) 140 | osm_json = download(self.overpass_query) 141 | 142 | # Need to be simplified 143 | else: 144 | lon, lat = poly_gdf.geometry.iloc[0][i].simplify(sim_factor).exterior.coords.xy 145 | self.overpass_query = self.get_overpass_query( lat = lat, lon = lon ) 146 | osm_json = download(self.overpass_query) 147 | 148 | try: 149 | temp_gdf_2 = json_to_gdf(osm_json= osm_json, data_type= "Polygon", 150 | tags = True, building_levels = False, height = True) 151 | temp_gdf_2["Building_height"] = temp_gdf_2["Building_height"].fillna(3) # 假设一层 层高3m 152 | 153 | # osm_json is None 154 | except: 155 | temp_gdf_2 = gpd.GeoDataFrame() 156 | 157 | temp_gdf_1 = pd.concat([temp_gdf_1, temp_gdf_2], ignore_index=True) 158 | 159 | temp_gdf = temp_gdf_1 160 | # set crs 161 | temp_gdf = gpd.GeoDataFrame(temp_gdf,crs='epsg:4326') 162 | 163 | else: 164 | temp_gdf = json_to_gdf(osm_json= self.query(), data_type= self.data_type, 165 | tags= tags, building_levels = building_levels, 166 | height = height) 167 | 168 | # projection 169 | temp_gdf_prj = ox.project_gdf(temp_gdf) 170 | 171 | # Add GeoDataFrame columns 172 | temp_gdf["osmscID"] = ["Building_"+ str(i) for i in temp_gdf["osmid"]] 173 | temp_gdf["Building_area"] = temp_gdf_prj["geometry"].area 174 | temp_gdf["Building_perimeter"] = temp_gdf_prj["geometry"].length 175 | 176 | # # If download building level from OSM 177 | # building_level True and height False 178 | # if height is true, Building_height has already filled 179 | if building_levels and not height: 180 | # if building_levels tags is unavailable, assume it is 1 181 | temp_gdf["building_levels"] = temp_gdf["building_levels"].fillna(1) 182 | # assume level height is 3m 183 | temp_gdf["Building_height"] = temp_gdf["building_levels"] * 3 184 | 185 | 186 | return temp_gdf 187 | 188 | class vegetation_group(polygon_group): 189 | """ 190 | Construct OSMsc vegetation objects 191 | """ 192 | def query(self): 193 | """ 194 | Query objects from OpenStreetMap via Overpass API 195 | 196 | Returns 197 | ------- 198 | JSON 199 | """ 200 | self.overpass_query = self.get_overpass_query() 201 | 202 | # return data in JSON 203 | return download(self.overpass_query) 204 | 205 | def get_overpass_query(self, lat = None, lon = None ): 206 | """ 207 | Get overpass query content from bbox or polygon boundary 208 | 209 | Parameters 210 | ---------- 211 | lat : list 212 | latitude list of polygon boundary nodes 213 | 214 | lon : list 215 | longitude list of polygon boundary nodes 216 | 217 | Returns 218 | ------- 219 | String 220 | """ 221 | 222 | if self.bbox: 223 | # Default query 224 | self.overpass_query = """ 225 | [out:json][timeout:5000]; 226 | ( 227 | way["leisure"~"park"]""" + str(self.bbox) + """; 228 | way["landuse"~"grass"]""" + str(self.bbox) + """; 229 | way["leisure"~"pitch"]""" + str(self.bbox) + """; 230 | way["leisure"~"garden"]""" + str(self.bbox) + """; 231 | way["natural"~"scrub"]""" + str(self.bbox) + """; 232 | way["landuse"~"farmyard"]""" +str(self.bbox) + """; 233 | way["landuse"~"recreation_ground"]""" + str(self.bbox) + """; 234 | way["leisure"~"playground"]""" +str(self.bbox) + """; 235 | way["natural"~"wetland"]""" +str(self.bbox) + """; 236 | way["natural"~"wood"]""" +str(self.bbox) + """; 237 | way["landuse"~"meadow"]""" +str(self.bbox) + """; 238 | way["landuse"~"cemetery"]""" +str(self.bbox) + """; 239 | 240 | ); 241 | out geom; 242 | """ 243 | return self.overpass_query 244 | 245 | if self.place_name: 246 | overpass_poly = "" 247 | for i in range(len(lon)): 248 | overpass_poly = overpass_poly + str(lat[i]) + " " + str(lon[i]) + " " 249 | 250 | # Overpass API Docs 251 | # https://dev.overpass-api.de/overpass-doc/en/criteria/per_tag.html 252 | self.overpass_query = """ 253 | [out:json][timeout:5000]; 254 | ( way["leisure"~"^(park|garden|pitch|playground)"](poly:'""" + str(overpass_poly.rstrip()) + """'); 255 | way["landuse"~"^(grass|farmyard|recreation_ground|meadow|cemetery)"](poly:'""" + str(overpass_poly.rstrip()) + """'); 256 | way["natural"~"^(wetland|wood|scrub)"](poly:'""" + str(overpass_poly.rstrip()) + """'); 257 | 258 | ); 259 | out geom; 260 | """ 261 | 262 | return self.overpass_query 263 | 264 | def get_gdf(self, tags = True, sim_factor = 0.0005): 265 | """ 266 | Obtain OSM data and save as GeoDataFrame. 267 | 268 | Parameters 269 | ---------- 270 | tags : bool 271 | if False, the GeoDataFrame won't add OSM "tags" column. 272 | if True, need to extract tag info into current GeoDataFrame 273 | 274 | sim_factor : float 275 | Factor of shapely geometry simplify function 276 | If there are still downloading errors (returns nothing), pls choose a larger sim_factor. 277 | 278 | Returns 279 | ------- 280 | GeoDataFrame 281 | """ 282 | # given place name exists 283 | if self.place_name: 284 | poly_gdf = ox.geocoder.geocode_to_gdf(self.place_name) 285 | 286 | temp_gdf_1 = gpd.GeoDataFrame() 287 | for i in range(len(poly_gdf.geometry.iloc[0])): 288 | 289 | time.sleep( 5 ) 290 | lon, lat = poly_gdf.geometry.iloc[0][i].exterior.coords.xy 291 | 292 | # Too many nodes would result in downloading error. 293 | if len(lon) < 50: 294 | #osm_json = get_osm_json_with_latlon(lat, lon) 295 | self.overpass_query = self.get_overpass_query(lat, lon) 296 | osm_json = download(self.overpass_query) 297 | 298 | # Need to be simplified 299 | else: 300 | lon, lat = poly_gdf.geometry.iloc[0][i].simplify(sim_factor).exterior.coords.xy 301 | self.overpass_query = self.get_overpass_query(lat, lon) 302 | osm_json = download(self.overpass_query) 303 | 304 | try: 305 | temp_gdf_2 = json_to_gdf(osm_json= osm_json, data_type= "Polygon", 306 | tags = True, building_levels = False, height = False) 307 | 308 | # osm_json is None 309 | except: 310 | temp_gdf_2 = gpd.GeoDataFrame() 311 | temp_gdf_1 = pd.concat([temp_gdf_1, temp_gdf_2], ignore_index=True) 312 | 313 | temp_gdf = temp_gdf_1 314 | # set crs 315 | temp_gdf = gpd.GeoDataFrame(temp_gdf,crs='epsg:4326') 316 | 317 | else: 318 | temp_gdf = json_to_gdf(osm_json= self.query(), data_type= self.data_type, 319 | tags= tags) 320 | 321 | temp_gdf_prj = ox.project_gdf(temp_gdf) 322 | 323 | temp_gdf["osmscID"] = ["Vegetation_"+ str(i) for i in temp_gdf["osmid"]] 324 | temp_gdf["Vegetation_area"] = temp_gdf_prj["geometry"].area 325 | temp_gdf["Vegetation_perimeter"] = temp_gdf_prj["geometry"].length 326 | 327 | return temp_gdf 328 | 329 | class waterbody_group(polygon_group): 330 | """ 331 | Construct OSMsc waterbody objects 332 | """ 333 | def query(self): 334 | """ 335 | Query objects from OpenStreetMap via Overpass API 336 | 337 | Returns 338 | ------- 339 | JSON 340 | """ 341 | self.overpass_query = self.get_overpass_query() 342 | 343 | # return data in JSON 344 | return download(self.overpass_query) 345 | 346 | def get_overpass_query(self, lat = None, lon = None ): 347 | """ 348 | Get overpass query content from bbox or polygon boundary 349 | 350 | Parameters 351 | ---------- 352 | lat : list 353 | latitude list of polygon boundary nodes 354 | 355 | lon : list 356 | longitude list of polygon boundary nodes 357 | 358 | Returns 359 | ------- 360 | String 361 | """ 362 | 363 | if self.bbox: 364 | # Default query 365 | self.overpass_query = """ 366 | [out:json][timeout:5000]; 367 | ( way["leisure"~"ice_rink"]""" + str(self.bbox) + """; 368 | way["landuse"~"swimming_pool"]""" + str(self.bbox) + """; 369 | way["man_made"~"water_tower"]""" + str(self.bbox) + """; 370 | way["natural"~"water"]""" +str(self.bbox) + """; 371 | way["reservoir"~"water_storage"]""" +str(self.bbox) + """; 372 | 373 | ); 374 | out geom; 375 | """ 376 | return self.overpass_query 377 | 378 | if self.place_name: 379 | overpass_poly = "" 380 | for i in range(len(lon)): 381 | overpass_poly = overpass_poly + str(lat[i]) + " " + str(lon[i]) + " " 382 | 383 | self.overpass_query = """ 384 | [out:json][timeout:5000]; 385 | ( way["leisure"~"ice_rink"](poly:'""" + str(overpass_poly.rstrip()) + """'); 386 | way["landuse"~"swimming_pool"](poly:'""" + str(overpass_poly.rstrip()) + """'); 387 | way["man_made"~"water_tower"](poly:'""" + str(overpass_poly.rstrip()) + """'); 388 | way["natural"~"water"](poly:'""" + str(overpass_poly.rstrip()) + """'); 389 | way["reservoir"~"water_storage"](poly:'""" + str(overpass_poly.rstrip()) + """'); 390 | 391 | ); 392 | out geom; 393 | """ 394 | 395 | return self.overpass_query 396 | 397 | def get_gdf(self, tags = True, sim_factor = 0.0005): 398 | """ 399 | Obtain OSM data and save as GeoDataFrame. 400 | 401 | Parameters 402 | ---------- 403 | tags : bool 404 | if False, the GeoDataFrame won't add OSM "tags" column. 405 | if True, need to extract tag info into current GeoDataFrame 406 | 407 | sim_factor : float 408 | Factor of shapely geometry simplify function 409 | If there are still downloading errors (returns nothing), pls choose a larger sim_factor. 410 | 411 | Returns 412 | ------- 413 | GeoDataFrame 414 | """ 415 | # given place name exists 416 | if self.place_name: 417 | poly_gdf = ox.geocoder.geocode_to_gdf(self.place_name) 418 | 419 | temp_gdf_1 = gpd.GeoDataFrame() 420 | for i in range(len(poly_gdf.geometry.iloc[0])): 421 | lon, lat = poly_gdf.geometry.iloc[0][i].exterior.coords.xy 422 | time.sleep( 5 ) 423 | 424 | # Too many nodes would result in downloading error. 425 | if len(lon) < 50: 426 | #osm_json = get_osm_json_with_latlon(lat, lon) 427 | self.overpass_query = self.get_overpass_query(lat, lon) 428 | osm_json = download(self.overpass_query) 429 | 430 | # Need to be simplified 431 | else: 432 | lon, lat = poly_gdf.geometry.iloc[0][i].simplify(sim_factor).exterior.coords.xy 433 | self.overpass_query = self.get_overpass_query(lat, lon) 434 | osm_json = download(self.overpass_query) 435 | 436 | try: 437 | temp_gdf_2 = json_to_gdf(osm_json= osm_json, data_type= "Polygon", 438 | tags = True, building_levels = False, height = False) 439 | 440 | # osm_json is None 441 | except: 442 | temp_gdf_2 = gpd.GeoDataFrame() 443 | temp_gdf_1 = pd.concat([temp_gdf_1, temp_gdf_2], ignore_index=True) 444 | 445 | temp_gdf = temp_gdf_1 446 | # set crs 447 | temp_gdf = gpd.GeoDataFrame(temp_gdf,crs='epsg:4326') 448 | 449 | else: 450 | temp_gdf = json_to_gdf(osm_json= self.query(), data_type= self.data_type, 451 | tags= tags) 452 | 453 | temp_gdf_prj = ox.project_gdf(temp_gdf) 454 | 455 | temp_gdf["osmscID"] = ["WaterBody_"+ str(i) for i in temp_gdf["osmid"]] 456 | temp_gdf["Waterbody_area"] = temp_gdf_prj["geometry"].area 457 | temp_gdf["Waterbody_perimeter"] = temp_gdf_prj["geometry"].length 458 | 459 | return temp_gdf 460 | 461 | class transportation_group(polygon_group): 462 | """ 463 | Construct OSMsc transportation objects 464 | """ 465 | 466 | def get_gdf_prj(self, street_width = 3): 467 | """ 468 | Obtain OSM data and save as GeoDataFrame. 469 | To define the street width, this function has to use the 470 | projected coordinate system. 471 | 472 | Parameters 473 | ---------- 474 | street_width : float or int 475 | Assumed street width for OSM street 476 | 477 | Returns 478 | ------- 479 | GeoDataFrame 480 | """ 481 | if self.bbox: 482 | building_gdf = building_group(bbox= self.bbox).get_gdf() 483 | if self.place_name: 484 | building_gdf = building_group(place_name = self.place_name).get_gdf() 485 | # 486 | street_graph = street_graph_from_gdf(building_gdf) 487 | # Street geometry is LineString 488 | street_temp_gdf = streets_from_street_graph(street_graph) 489 | street_temp_gdf.crs = 'epsg:4326' 490 | 491 | # only keep user-defined trans_type 492 | if self.trans_type is not None: 493 | street_type_1_gdf = street_temp_gdf[ street_temp_gdf["highway"]== list(self.trans_type)[0] ] 494 | 495 | concat_gdf = street_type_1_gdf 496 | for i in range(len(self.trans_type)-1): 497 | street_type = list(self.trans_type)[i+1] 498 | street_type_2_gdf = street_temp_gdf[ street_temp_gdf["highway"]== street_type ] 499 | 500 | concat_gdf = pd.concat([concat_gdf,street_type_2_gdf]) 501 | # filtered street_temp_gdf 502 | street_temp_gdf = concat_gdf 503 | # set crs 504 | street_temp_gdf = gpd.GeoDataFrame(street_temp_gdf,crs='epsg:4326') 505 | 506 | 507 | # Empty street objects 508 | street_gdf_prj = gpd.GeoDataFrame() 509 | # re-project geometries to a projected CRS before buffer function 510 | street_temp_gdf_prj = ox.project_gdf(street_temp_gdf) 511 | # Assume street width is 3m and construct street outlines 512 | street_temp_gdf_prj["geometry"] = street_temp_gdf_prj["geometry"].buffer(street_width, cap_style = 3).exterior 513 | # Polygon street objects 514 | street_gdf_prj["geometry"] = list(polygonize(street_temp_gdf_prj["geometry"])) 515 | # set crs 516 | street_gdf_prj.crs = street_temp_gdf_prj.crs 517 | # Add osmscID column 518 | street_gdf_prj["osmscID"] = ["Transportation_"+str(i) for i in range(len(street_gdf_prj))] 519 | 520 | return street_gdf_prj 521 | 522 | class urban_Tile_group(polygon_group): 523 | """ 524 | Construct OSMsc urban Tile objects 525 | """ 526 | def get_gdf_prj(self): 527 | """ 528 | Construct urban Tile objects 529 | To define the street width, this function has to use the 530 | projected coordinate system. 531 | 532 | Returns 533 | ------- 534 | GeoDataFrame 535 | """ 536 | # Download the street dataframe 537 | street_gdf_prj = transportation_group(bbox= self.bbox, place_name = self.place_name, trans_type= self.trans_type).get_gdf_prj() 538 | # Merged streets are the basis for generating blocks 539 | # Revised 2022-8-15 540 | street_gdf_prj["geometry"] = street_gdf_prj["geometry"].buffer(0.001) 541 | street_dis_geom = street_gdf_prj.dissolve().iloc[0].geometry 542 | 543 | # Create urban_Tile_gdf 544 | urban_Tile_gdf_prj = gpd.GeoDataFrame() 545 | # All streets are merged into a polygon 546 | urban_Tile_gdf_prj["geometry"] = list(polygonize(street_dis_geom.boundary)) 547 | # delete the whole street polygon 548 | urban_Tile_gdf_prj = urban_Tile_gdf_prj.drop(index=0) 549 | # re-index 550 | urban_Tile_gdf_prj.index = urban_Tile_gdf_prj.index - 1 551 | # set crs 552 | urban_Tile_gdf_prj.crs = street_gdf_prj.crs 553 | 554 | # Add osmscID column 555 | urban_Tile_gdf_prj["osmscID"] = ["UrbanTile_"+str(i) for i in range(len(urban_Tile_gdf_prj))] 556 | # Add other attr columns 557 | urban_Tile_gdf_prj["UrbanTile_area"] = urban_Tile_gdf_prj["geometry"].area 558 | urban_Tile_gdf_prj["UrbanTile_perimeter"] = urban_Tile_gdf_prj["geometry"].length 559 | 560 | return urban_Tile_gdf_prj 561 | 562 | 563 | ####### TODO More city objects in OSMsc, namely, networks and points ######## 564 | class street_network(line_string_group): 565 | """ 566 | Construct street networks, rather than street polygons 567 | """ 568 | def query(self): 569 | """ 570 | Query objects from OpenStreetMap via Overpass API 571 | 572 | Returns 573 | ------- 574 | JSON 575 | """ 576 | # Default query 577 | if not self.overpass_query: 578 | 579 | print("The default setting is to query all street, if necessary, please enter your StreetNetwork overpass api query!!") 580 | 581 | self.overpass_query = """ 582 | [out:json][timeout:5000]; 583 | ( way["highway"]["area"!~"yes"]""" + str(self.bbox) + """; 584 | ); 585 | out geom; 586 | """ 587 | 588 | # return data in JSON format 589 | return download(self.overpass_query) 590 | 591 | def query_with_network_type(self, network_type): 592 | """ 593 | Query streets of a specific street type from OpenStreetMap via OSMnx 594 | More info about OSMnx: https://github.com/gboeing/osmnx 595 | 596 | Parameters 597 | ---------- 598 | network_type : string 599 | which type of street network to get, chose from 600 | {"all_private", "all", "bike", "drive", "drive_service", "walk"} 601 | 602 | Returns 603 | ------- 604 | Graph 605 | """ 606 | 607 | if self.bbox is None: 608 | print("Need to input bbox for StreetNetwork") 609 | return None 610 | else: 611 | (south, west, north, east) = self.bbox 612 | self.street_graph = ox.graph_from_bbox(south= south, west=west, north=north, 613 | east=east, network_type = network_type) 614 | return self.street_graph 615 | 616 | def query_all_streets_from_buildings(self): 617 | """ 618 | Query all_streets from OpenStreetMap via OSMnx 619 | More info about OSMnx: https://github.com/gboeing/osmnx 620 | 621 | Returns 622 | ------- 623 | Graph 624 | """ 625 | 626 | if self.scope is None: 627 | print("Need to input BuildingGroup_gdf for StreetNetwork") 628 | return None 629 | else: 630 | self.street_graph = street_graph_from_gdf(self.scope) 631 | return self.street_graph 632 | 633 | def query_streets_with_tags_from_buildings(self, primary = True, secondary = True, tertiary = False, 634 | residential = False, service = False): 635 | # TODO Needed to be modified later 636 | """ 637 | Query streets of a specific tag from OpenStreetMap via OSMnx 638 | More info about OSMnx: https://github.com/gboeing/osmnx 639 | 640 | Parameters 641 | ---------- 642 | primary : bool 643 | secondary : bool 644 | tertiary : bool 645 | residential : bool 646 | service : bool 647 | Specific tags needed to be kept 648 | 649 | Returns 650 | ------- 651 | Graph 652 | """ 653 | if self.scope is None: 654 | print("Need to input BuildingGroup_gdf for StreetNetwork") 655 | return None 656 | else: 657 | # Only based on the query result from OSMnx and "highway" tags 658 | self.street_graph = street_graph_from_gdf(self.scope) 659 | street_edges = ox.graph_to_gdfs(self.street_graph)[1] 660 | street_nodes = ox.graph_to_gdfs(self.street_graph)[0] 661 | 662 | if primary: 663 | temp_gdf_1 = street_edges[street_edges["highway"] == 'primary'] 664 | temp_gdf_edges = temp_gdf_1 665 | if secondary: 666 | temp_gdf_2 = street_edges[street_edges["highway"] == 'secondary'] 667 | temp_gdf_edges = pd.concat([temp_gdf_edges, temp_gdf_2]) 668 | if tertiary: 669 | temp_gdf_3 = street_edges[street_edges["highway"] == 'tertiary'] 670 | temp_gdf_edges = pd.concat([temp_gdf_edges, temp_gdf_3]) 671 | if residential: 672 | temp_gdf_4 = street_edges[street_edges["highway"] == 'residential'] 673 | temp_gdf_edges = pd.concat([temp_gdf_edges, temp_gdf_4]) 674 | if service: 675 | temp_gdf_5 = street_edges[street_edges["highway"] == 'service'] 676 | temp_gdf_edges = pd.concat([temp_gdf_edges, temp_gdf_5]) 677 | 678 | # u_list = list(temp_gdf_edges["u"]) 679 | # v_list = list(temp_gdf_edges["v"]) 680 | u_list = list(temp_gdf_edges["from"]) 681 | v_list = list(temp_gdf_edges["to"]) 682 | nodes_within_edges = u_list + v_list 683 | 684 | # Only keep the nodes located in the specific edges 685 | temp_gdf_nodes = gpd.GeoDataFrame() 686 | for i in range(len(list(street_nodes.index))): 687 | node_index = list(street_nodes.index)[i] 688 | if node_index in nodes_within_edges: 689 | temp_gdf_nodes = pd.concat([temp_gdf_nodes,street_nodes.loc[[node_index]]]) 690 | 691 | temp_gdf_nodes.gdf_name = "'unnamed_nodes'" 692 | 693 | # Generate graph from nodes and edges 694 | self.street_graph = graph_from_gdfs(temp_gdf_nodes,temp_gdf_edges,) 695 | 696 | return self.street_graph 697 | 698 | 699 | class tagged_point_group(point_group): 700 | """ 701 | Construct OSMsc tagged points. 702 | """ 703 | def query(self): 704 | """ 705 | Query objects from OpenStreetMap via Overpass API. 706 | Please input your query condition, if any. 707 | 708 | Returns 709 | ------- 710 | JSON 711 | """ 712 | if not self.overpass_query: 713 | 714 | print("The default setting is to query tagged points, if necessary, please enter your TaggedPointGroup overpass api query!!") 715 | 716 | self.overpass_query = """ 717 | [out:json][timeout:5000]; 718 | ( 719 | node""" + str(self.bbox) + """[~"."~"."]; 720 | ); 721 | out geom; 722 | """ 723 | 724 | return download(self.overpass_query) # json 725 | 726 | 727 | 728 | -------------------------------------------------------------------------------- /osmsc/utils.py: -------------------------------------------------------------------------------- 1 | """General utility functions.""" 2 | 3 | import osmnx as ox 4 | import numpy as np 5 | import pandas as pd 6 | import geopandas as gpd 7 | import networkx as nx 8 | import math 9 | import requests 10 | import random 11 | 12 | from shapely.geometry import Point, LineString, Polygon, MultiPolygon 13 | from shapely.ops import polygonize 14 | 15 | 16 | def download(overpass_query): 17 | """ 18 | Download OSM via Overpass API 19 | 20 | Parameters 21 | ---------- 22 | overpass_query : string 23 | Customized overpass API 24 | 25 | Returns 26 | ------- 27 | JSON 28 | """ 29 | overpass_url = "http://overpass-api.de/api/interpreter" 30 | response = requests.get(overpass_url, params={'data': overpass_query}) 31 | osm_json = response.json() 32 | 33 | return osm_json 34 | 35 | 36 | def json_to_gdf(osm_json, data_type, tags = True, building_levels = False, height = False): 37 | """ 38 | Convert the json data into GeoDataFrame. 39 | 40 | Parameters 41 | ---------- 42 | osm_json : dict 43 | Query result from OSM 44 | data_type : string 45 | Adjust the GeoPandas geometry object with OSMsc object datatype 46 | tags : bool 47 | The final GeoDataFrame has "tags" column, that are downloaded from OSM 48 | building_levels : bool 49 | The final GeoDataFrame has "building_levels" column, that are downloaded from OSM 50 | height : bool 51 | The final GeoDataFrame has "height" column, that are downloaded from OSM 52 | 53 | Returns 54 | ------- 55 | GeoDataFrame 56 | """ 57 | 58 | # when osm_json is not empty 59 | if osm_json['elements']: 60 | 61 | if data_type == "Polygon": 62 | # Collect the Polygons in osm_json 63 | coords_list = [] 64 | osm_id = [] 65 | tags_list = [] 66 | building_levels_list = [] 67 | building_height_list = [] 68 | 69 | for element in osm_json['elements']: 70 | 71 | # Polygon id (from osm "way" id) 72 | # osm_id.append(element["id"]) 73 | 74 | # Polygon nodes 75 | coords_one = [] 76 | for node in element["geometry"]: 77 | lon = node['lon'] 78 | lat = node['lat'] 79 | 80 | # in gpd geometry column, the order is lon, lat, 81 | coords_one.append((lon, lat)) 82 | # A Polygon has 3 points at least. 83 | if len(coords_one) > 2: 84 | coords_list.append(Polygon(coords_one)) 85 | 86 | # ensure that the two columns have the same number of columns 87 | osm_id.append(element["id"]) 88 | 89 | # Polygon tags 90 | if tags: 91 | 92 | # element["tags"] --> dict 93 | tags_list.append(element["tags"]) 94 | 95 | # Polygon building:levels tag 96 | try:# unknown tags 97 | if building_levels: 98 | if "building:levels" in element["tags"].keys(): 99 | building_levels_list.append(int(float(element["tags"]["building:levels"]))) 100 | else: 101 | building_levels_list.append(1) 102 | except: 103 | building_levels_list.append(1) 104 | try:# unknown tags 105 | if height: 106 | if "height" in element["tags"].keys(): 107 | # aviod "7 m" "7.5 m" 108 | height_string = element["tags"]["height"] 109 | building_height_list.append([float(i) for i in height_string.split() if i.isdigit() or i.replace(".", '', 1).isdigit()][0]) 110 | else: 111 | building_height_list.append(3) 112 | except: 113 | building_height_list.append(3) 114 | 115 | # # if building level>1, height = building level * 3 116 | if building_levels: 117 | if building_levels_list[-1] !=1 and building_height_list[-1] == 3: 118 | building_height_list[-1] = building_levels_list[-1] *3 119 | if building_height_list[-1] == 0: 120 | building_height_list[-1] = building_levels_list[-1] *3 121 | 122 | 123 | 124 | if data_type == "Point": 125 | coords_list = [] 126 | osm_id = [] 127 | tags_list = [] 128 | 129 | for element in osm_json['elements']: 130 | 131 | # Polygon id (from osm "way" id) 132 | osm_id.append(element["id"]) 133 | 134 | # Polygon nodes 135 | coords_one = [] 136 | 137 | lon = element['lon'] 138 | lat = element['lat'] 139 | 140 | # in gpd, the order is lon, lat, 141 | 142 | coords_one.append((lon, lat)) 143 | 144 | coords_list.append(Point(coords_one)) 145 | 146 | # Polygon tags 147 | if tags: 148 | # element["tags"] --> dict 149 | tags_list.append(element["tags"]) 150 | 151 | if data_type == "LineString": 152 | 153 | coords_list = [] 154 | osm_id = [] 155 | tags_list = [] 156 | 157 | for element in osm_json['elements']: 158 | 159 | # Polygon id (from osm "way" id) 160 | osm_id.append(element["id"]) 161 | 162 | # Polygon nodes 163 | coords_one = [] 164 | for node in element["geometry"]: 165 | lon = node['lon'] 166 | lat = node['lat'] 167 | 168 | # in gpd, the order is lon, lat, 169 | coords_one.append([lon, lat]) 170 | 171 | coords_list.append(LineString(coords_one)) 172 | 173 | if tags: 174 | # element["tags"] --> dict 175 | tags_list.append(element["tags"]) 176 | 177 | # create a new GeoDataFrame to store the Polygons 178 | temp_gdf = gpd.GeoDataFrame() 179 | temp_gdf["osmid"] = osm_id 180 | temp_gdf['geometry'] = coords_list 181 | # set crs 182 | temp_gdf = temp_gdf.set_crs(crs='epsg:4326') 183 | #gpd.GeoDataFrame(temp_gdf,crs='epsg:4326') 184 | 185 | # Add "tags" column 186 | if tags: 187 | temp_gdf["tags"] = tags_list 188 | 189 | # Add "building_levels" column 190 | if building_levels: 191 | temp_gdf["building_levels"] = building_levels_list 192 | if height: 193 | temp_gdf["Building_height"] = building_height_list 194 | 195 | return temp_gdf 196 | 197 | else: 198 | print("Your query returns nothing!") 199 | 200 | 201 | def fill_nan_list_position(gdf,colName): 202 | """ 203 | Fill the nan value of GeoDataFrame with "None" 204 | Designed for CityJSON object construction 205 | 206 | Parameters 207 | ---------- 208 | gdf : GeoDataFrame 209 | colName : string 210 | Which column needs to be applied with this funciton. 211 | 212 | Returns 213 | ------- 214 | GeoDataFrame 215 | """ 216 | item_list = [] 217 | for item in gdf[colName]: 218 | 219 | if item == [np.nan]: 220 | item_list.append("None") 221 | else: 222 | item_list.append(item) 223 | 224 | gdf[colName] = item_list 225 | 226 | return gdf 227 | 228 | 229 | def tailor_gdf_with_bbox(target_gdf_prj, bbox): 230 | """ 231 | Tailor the target GeoDataFrame, usually for vegetation or waterbody objects 232 | 233 | Parameters 234 | ---------- 235 | target_gdf : GeoDataFrame 236 | vegetation or waterbody GeoDataFrames 237 | bbox : tuple 238 | bbox --> (South, West, North, East) or (# minlat, minlon, maxlat, maxlon) 239 | 240 | 241 | Returns 242 | ------- 243 | GeoDataFrame 244 | """ 245 | # create bbox GeoDataFrame 246 | bbox_gdf = get_bbox_gdf(bbox) 247 | # convert the crs 248 | bbox_gdf_prj = ox.project_gdf(bbox_gdf) 249 | 250 | # tailor the target_gdf 251 | tailored_gdf = gpd.overlay(target_gdf_prj, bbox_gdf_prj, how="intersection") 252 | 253 | return tailored_gdf 254 | 255 | 256 | ################# for bbox ############################# 257 | def bbox_from_gdf(gdf, buffer_value = 0.001): 258 | """ 259 | Obtain bbox from the whole GeoDataFrame 260 | bbox --> (South, West, North, East) or (# minlat, minlon, maxlat, maxlon) 261 | 262 | Parameters 263 | ---------- 264 | gdf : GeoDataFrame 265 | buffer_value : float 266 | buffer variable, see more in GeoDataFrame.Geometry.buffer() 267 | 268 | Returns 269 | ------- 270 | Tuple 271 | """ 272 | 273 | # the whole geometry 274 | gdf_boundary =gdf.cascaded_union.convex_hull.buffer(buffer_value) 275 | # get the bounds 276 | (West, South, East, North) = gdf_boundary.bounds 277 | # create the bbox 278 | bbox = (South, West, North, East) 279 | 280 | return bbox 281 | 282 | def bbox_from_place_name(place_name): 283 | """ 284 | Obtain bbox from place_name 285 | bbox --> (South, West, North, East) or (# minlat, minlon, maxlat, maxlon) 286 | Place name can be checked in https://nominatim.openstreetmap.org/ui/search.html 287 | 288 | Parameters 289 | ---------- 290 | place_name : str 291 | 292 | Returns 293 | ------- 294 | Tuple 295 | """ 296 | # get place ploygon 297 | poly_gdf = ox.geocoder.geocode_to_gdf(place_name) 298 | # get the bounds of this ploygon 299 | bounds = poly_gdf.geometry.iloc[0].bounds 300 | min_lon, min_lat, max_lon, max_lat = list(bounds) 301 | # create the bbox 302 | bbox = (min_lat, min_lon, max_lat, max_lon) 303 | 304 | return bbox 305 | 306 | def get_bbox_gdf(bbox): 307 | """ 308 | Create bbox GeoDataFrame 309 | 310 | Parameters 311 | ---------- 312 | bbox : tuple 313 | bbox --> (South, West, North, East) or (# minlat, minlon, maxlat, maxlon) 314 | 315 | 316 | Returns 317 | ------- 318 | GeoDataFrame 319 | """ 320 | 321 | # create bbox geometry 322 | min_lat, min_lon, max_lat, max_lon = bbox 323 | bbox_geometry = Polygon([(min_lon,min_lat),(min_lon,max_lat ),(max_lon, max_lat), (max_lon,min_lat)]) 324 | 325 | # create bbox GeoDataFrame 326 | bbox_gdf = gpd.GeoDataFrame() 327 | bbox_gdf["geometry"] = [bbox_geometry] 328 | bbox_gdf.crs = "epsg:4326" 329 | 330 | return bbox_gdf 331 | 332 | ################# for osm_json ############################# 333 | def get_keys_from_osm_json(osm_json): 334 | """ 335 | Obtain all keys from queried OSM results, and do not include repeated elements. 336 | 337 | Parameters 338 | ---------- 339 | osm_json : JSON 340 | 341 | Returns 342 | ------- 343 | List 344 | """ 345 | 346 | osm_keys = [] 347 | for element in osm_json['elements']: 348 | keys = list(element["tags"].keys()) 349 | 350 | osm_keys = list(set(osm_keys + keys)) 351 | 352 | return osm_keys 353 | 354 | def get_values_from_osm_json(osm_json, osm_key): 355 | """ 356 | Obtain all values for a specific key from queried OSM results 357 | 358 | Parameters 359 | ---------- 360 | osm_json : JSON 361 | osm_key : string 362 | A specific key, for instance, "building" 363 | 364 | Returns 365 | ------- 366 | List 367 | """ 368 | 369 | osm_values = [] 370 | for element in osm_json['elements']: 371 | 372 | # Not every element has this key 373 | try: 374 | values = element["tags"][str(osm_key)] 375 | if values not in osm_values: 376 | osm_values.append(values) 377 | except: 378 | pass 379 | 380 | return osm_values 381 | 382 | 383 | ################# for overpass_api ############################# 384 | def customize_query_content(osm_dict_k_v, bbox, union_set, intersection_set, Polygon, Graph, Point): 385 | """ 386 | Customize a query statement 387 | 388 | Parameters 389 | ---------- 390 | osm_dict_k_v : dict 391 | For example, osm_dict_k_v = {osm_key_1:osm_values_1, osm_key_2: osm_values_2} 392 | osm_key -- str 393 | osm_values -- list [str,str,str, ...] 394 | bbox : tuple 395 | bbox --> (South, West, North, East) or (# minlat, minlon, maxlat, maxlon) 396 | union_set : bool 397 | True, return the union set of all query results. 398 | intersection_set : bool 399 | True, return the intersection set of all query results 400 | Polygon : bool 401 | True, Polygon geometry saved in GeoPandas geometry colunmn 402 | Graph : bool 403 | True, Graph geometry saved in GeoPandas geometry colunmn 404 | Point : bool 405 | True, Point geometry saved in GeoPandas geometry colunmn 406 | 407 | Returns 408 | ------- 409 | String 410 | """ 411 | # overpass content is one row 412 | if intersection_set: 413 | conditions = "" 414 | for osm_key, osm_values in osm_dict_k_v.items(): 415 | 416 | # Meet the conditions of different conditions at the same time. 417 | # And the'|' of a condition is an "or" relationship 418 | conditions = conditions + "[" + '"' + str(osm_key) + '"' + "~" +'"' + '|'.join(osm_values) +'"' + "]" 419 | 420 | if Polygon or Graph: 421 | query_content = """way""" + conditions + str(bbox) + """;""" 422 | elif Point: 423 | query_content = """node""" + conditions + str(bbox) + """[~"."~"."];""" 424 | 425 | # overpass content is multiple rows 426 | if union_set: 427 | 428 | if Polygon or Graph: 429 | query_content = "" 430 | for osm_key, osm_values in osm_dict_k_v.items(): 431 | query_content = query_content + "way[" + '"' + str(osm_key) + '"' + "~" +'"' + '|'.join(osm_values) +'"' + "]"+ str(bbox) + ";" 432 | 433 | elif Point: 434 | query_content = "" 435 | for osm_key, osm_values in osm_dict_k_v.items(): 436 | query_content = query_content + "node[" + '"' + str(osm_key) + '"' + "~" +'"' + '|'.join(osm_values) +'"' + "]"+ str(bbox) + """[~"."~"."];""" 437 | 438 | return query_content 439 | 440 | # TODO more overpass api settings 441 | def create_overpass_query(osm_dict_k_v, bbox, union_set = True, intersection_set = False, 442 | Polygon = True, Graph = False, Point = False): 443 | 444 | """ 445 | Create a overpass query according to k_v pairs 446 | 447 | Parameters 448 | ---------- 449 | osm_dict_k_v : dict 450 | For example, osm_dict_k_v = {osm_key_1:osm_values_1, osm_key_2: osm_values_2} 451 | osm_key -- str 452 | osm_values -- list [str,str,str, ...] 453 | bbox : tuple 454 | bbox --> (South, West, North, East) or (# minlat, minlon, maxlat, maxlon) 455 | union_set : bool 456 | True, return the union set of all query results. 457 | intersection_set : bool 458 | True, return the intersection set of all query results 459 | Polygon : bool 460 | True, Polygon geometry saved in GeoPandas geometry colunmn 461 | Graph : bool 462 | True, Graph geometry saved in GeoPandas geometry colunmn 463 | Point : bool 464 | True, Point geometry saved in GeoPandas geometry colunmn 465 | 466 | Returns 467 | ------- 468 | String 469 | """ 470 | 471 | # Build main query content 472 | query_content = customize_query_content(osm_dict_k_v, bbox, union_set , intersection_set ,Polygon , Graph , Point ) 473 | 474 | if query_content: 475 | # Complete overpass query 476 | overpass_query = """ 477 | [out:json][timeout:50]; 478 | ( """ + query_content + """ ); 479 | out geom; 480 | """ 481 | else: 482 | print("please enter your target data_type!") 483 | return None 484 | 485 | return overpass_query 486 | 487 | 488 | ###################### osmnx utils ################################ 489 | # TODO rewrite osmnx package parts for osmsc 490 | def graph_from_gdfs(gdf_nodes, gdf_edges, graph_attrs=None): 491 | """ 492 | Updated from osmnx https://github.com/gboeing/osmnx 493 | Convert node and edge GeoDataFrames to a MultiDiGraph. 494 | 495 | This function is the inverse of `graph_to_gdfs` and is designed to work in 496 | conjunction with it. However, you can convert arbitrary node and edge 497 | GeoDataFrames as long as gdf_nodes is uniquely indexed by `osmid` and 498 | gdf_edges is uniquely multi-indexed by `u`, `v`, `key` (following normal 499 | MultiDiGraph structure). This allows you to load any node/edge shapefiles 500 | or GeoPackage layers as GeoDataFrames then convert them to a MultiDiGraph 501 | for graph analysis. 502 | 503 | Parameters 504 | ---------- 505 | gdf_nodes : geopandas.GeoDataFrame 506 | GeoDataFrame of graph nodes uniquely indexed by osmid 507 | gdf_edges : geopandas.GeoDataFrame 508 | GeoDataFrame of graph edges uniquely multi-indexed by u, v, key 509 | graph_attrs : dict 510 | the new G.graph attribute dict. if None, use crs from gdf_edges as the 511 | only graph-level attribute (gdf_edges must have crs attribute set) 512 | 513 | Returns 514 | ------- 515 | G : networkx.MultiDiGraph 516 | """ 517 | if graph_attrs is None: 518 | graph_attrs = {"crs": gdf_edges.crs} 519 | G = nx.MultiDiGraph(**graph_attrs) 520 | 521 | # add edges and their attributes to graph, but filter out null attribute 522 | # values so that edges only get attributes with non-null values 523 | 524 | # attr_names = gdf_edges.columns.to_list() 525 | # for (u, v, k), attr_vals in zip(gdf_edges.index, gdf_edges.values): 526 | 527 | for (u, v, k) in gdf_edges.index: 528 | # data_all = zip(attr_names, attr_vals) 529 | # data = {name: val for name, val in data_all if isinstance(val, list) or pd.notnull(val)} 530 | # G.add_edge(u, v, key=k, **data) 531 | G.add_edge(u, v, key=k) 532 | 533 | # add nodes' attributes to graph 534 | for col in gdf_nodes.columns: 535 | nx.set_node_attributes(G, name=col, values=gdf_nodes[col].dropna()) 536 | 537 | return G 538 | 539 | 540 | 541 | ##################### osmuf utils ################################ 542 | # TODO rewrite osmuf package parts for osmsc 543 | def street_graph_from_gdf(gdf, network_type='drive'): 544 | """ 545 | Updated from osmuf https://github.com/AtelierLibre/osmuf 546 | Download streets within a convex hull around a GeoDataFrame. 547 | 548 | Used to ensure that all streets around city blocks are downloaded, not just 549 | those inside an arbitrary bounding box. 550 | 551 | Parameters 552 | ---------- 553 | gdf : geodataframe 554 | currently accepts a projected gdf 555 | 556 | network_type : string 557 | network_type as defined in osmnx 558 | 559 | Returns 560 | ------- 561 | networkx multidigraph 562 | """ 563 | # generate convex hull around the gdf 564 | boundary = gdf_convex_hull(gdf) 565 | 566 | # download the highway network within this boundary 567 | street_graph = ox.graph_from_polygon(boundary, network_type, 568 | simplify=True, retain_all=True, 569 | truncate_by_edge=True, clean_periphery=False) 570 | # remove duplicates which make polygonization fail 571 | street_graph = ox.get_undirected(street_graph) 572 | 573 | return street_graph 574 | 575 | def streets_from_street_graph(street_graph): 576 | """ 577 | Updated from osmuf https://github.com/AtelierLibre/osmuf 578 | Convert a networkx multidigraph to a GeoDataFrame. 579 | 580 | Primarily here to allow future filtering of streets data for osmuf purposes 581 | 582 | Parameters 583 | ---------- 584 | street_graph : networkx multidigraph 585 | 586 | Returns 587 | ------- 588 | GeoDataFrame 589 | """ 590 | 591 | # convert to gdf 592 | streets = ox.graph_to_gdfs(street_graph, nodes=False) 593 | # write index into a column 594 | streets['street_id'] = streets.index 595 | 596 | # insert filtering/processing here for OSMuf purposes 597 | 598 | return streets 599 | 600 | def gdf_convex_hull(gdf): 601 | 602 | """ 603 | Updated from osmuf https://github.com/AtelierLibre/osmuf 604 | Creates a convex hull around the total extent of a GeoDataFrame. 605 | 606 | Used to define a polygon for retrieving geometries within. When calculating 607 | densities for urban blocks we need to retrieve the full extent of e.g. 608 | buildings within the blocks, not crop them to an arbitrary bounding box. 609 | 610 | Parameters 611 | ---------- 612 | gdf : geodataframe 613 | currently accepts a projected gdf 614 | 615 | Returns 616 | ------- 617 | shapely polygon 618 | """ 619 | ### INSERT CHECK FOR CRS HERE? 620 | 621 | # project gdf back to geographic coordinates as footprints_from_polygon 622 | # requires it 623 | gdf_temp = ox.projection.project_gdf(gdf, to_latlong=True) 624 | # determine the boundary polygon to fetch buildings within 625 | # buffer originally 0.000225, buffer actually needs to go whole block away 626 | # to get complete highways therefor trying 0.001 627 | boundary=gdf_temp.cascaded_union.convex_hull.buffer(0.001) 628 | # NOTE - maybe more efficient to generate boundary first then reproject second? 629 | 630 | return boundary 631 | 632 | def graph_to_polygons(G, node_geometry=True, fill_edge_geometry=True): 633 | """ 634 | Updated from osmuf https://github.com/AtelierLibre/osmuf 635 | Convert the edges of a graph into a GeoDataFrame of polygons. 636 | 637 | Parameters 638 | ---------- 639 | G : networkx multidigraph 640 | 641 | node_geometry : bool 642 | if True, create a geometry column from node x and y data 643 | 644 | fill_edge_geometry : bool 645 | if True, fill in missing edge geometry fields using origin and 646 | destination nodes 647 | Returns 648 | ------- 649 | GeoDataFrame 650 | gdf_polygons 651 | """ 652 | 653 | # create a list to hold edges 654 | edges = [] 655 | # loop through the edges in the graph 656 | for u, v, key, data in G.edges(keys=True, data=True): 657 | 658 | # for each edge, add key and all attributes in data dict to the 659 | # edge_details 660 | edge_details = {'u':u, 'v':v, 'key':key} 661 | for attr_key in data: 662 | edge_details[attr_key] = data[attr_key] 663 | 664 | # if edge doesn't already have a geometry attribute, create one now 665 | # if fill_edge_geometry==True 666 | if 'geometry' not in data: 667 | if fill_edge_geometry: 668 | point_u = Point((G.nodes[u]['x'], G.nodes[u]['y'])) 669 | point_v = Point((G.nodes[v]['x'], G.nodes[v]['y'])) 670 | edge_details['geometry'] = LineString([point_u, point_v]) 671 | else: 672 | edge_details['geometry'] = np.nan 673 | 674 | edges.append(edge_details) 675 | 676 | # extract the edge geometries from the list of edge dictionaries 677 | edge_geometry = [] 678 | 679 | for edge in edges: 680 | edge_geometry.append(edge['geometry']) 681 | 682 | # create a list to hold polygons 683 | polygons = [] 684 | 685 | polygons = list(polygonize(edge_geometry)) 686 | 687 | # Create a GeoDataFrame from the list of polygons and set the CRS 688 | 689 | # an option here is to feed it a list of dictionaries with a 'geometry' key 690 | # this would be one step e.g. 691 | # gdf_polys = gpd.GeoDataFrame(polygons) 692 | 693 | # Greate an empty GeoDataFrame 694 | gdf_polygons = gpd.GeoDataFrame() 695 | 696 | # Create a new column called 'geometry' to the GeoDataFrame 697 | gdf_polygons['geometry'] = None 698 | 699 | # Assign the list of polygons to the geometry column 700 | gdf_polygons.geometry = polygons 701 | 702 | # Set the crs 703 | gdf_polygons.crs = G.graph['crs'] 704 | gdf_polygons.gdf_name = '{}_polygons'.format('name') #G.graph['name']) 705 | 706 | return gdf_polygons 707 | 708 | def gen_regularity(gdf): 709 | """ 710 | Updated from osmuf https://github.com/AtelierLibre/osmuf 711 | Returns a geodataframe of smallest enclosing 'circles' (shapely polygons) 712 | generated from the input geodataframe of polygons. 713 | 714 | It includes columns that contain the area of the original polygon and the 715 | circles and the 'form factor' ratio of the area of the polygon to the area 716 | of the enclosing circle. 717 | 718 | Parameters 719 | ---------- 720 | poly_gdf : geodataframe 721 | a geodataframe containing polygons. 722 | 723 | Returns 724 | ------- 725 | GeoDataFrame 726 | """ 727 | #gdf_regularity = gdf[['morpho_Tile_id', 'geometry']].copy() 728 | gdf_regularity = gdf[['geometry']].copy() 729 | 730 | # write the area of each polygon into a column 731 | gdf_regularity['poly_area_m2'] = gdf.area.round(decimals=1) 732 | 733 | # replace the polygon geometry with the smallest enclosing circle 734 | gdf_regularity['geometry'] = gdf_regularity['geometry'].apply(circlizer) 735 | 736 | # calculate the area of the smallest enclosing circles 737 | gdf_regularity['SEC_area_m2'] = gdf_regularity.area.round(decimals=1) 738 | 739 | # calculate 'regularity' as "the ratio between the area of the polygon and 740 | # the area of the circumscribed circle C" Barthelemy M. and Louf R., (2014) 741 | gdf_regularity['regularity'] = gdf_regularity['poly_area_m2']/gdf_regularity['SEC_area_m2'] 742 | 743 | return gdf_regularity 744 | 745 | def circlizer(x): 746 | """Updated from osmuf https://github.com/AtelierLibre/osmuf""" 747 | # takes a shapely polygon or multipolygon and returns a shapely circle polygon 748 | SEC = sec() 749 | (cx, cy, buff) = SEC.make_circle(extract_poly_coords(x)) 750 | donut = Point(cx, cy).buffer(buff) 751 | 752 | return donut 753 | 754 | def extract_poly_coords(geom): 755 | """Updated from osmuf https://github.com/AtelierLibre/osmuf""" 756 | # extract the coordinates of shapely polygons and multipolygons 757 | # as a list of tuples 758 | if geom.type == 'Polygon': 759 | exterior_coords = geom.exterior.coords[:] 760 | interior_coords = [] 761 | for interior in geom.interiors: 762 | interior_coords += interior.coords[:] 763 | elif geom.type == 'MultiPolygon': 764 | exterior_coords = [] 765 | interior_coords = [] 766 | for part in geom: 767 | epc = extract_poly_coords(part) # Recursive call 768 | exterior_coords += epc['exterior_coords'] 769 | interior_coords += epc['interior_coords'] 770 | else: 771 | raise ValueError('Unhandled geometry type: ' + repr(geom.type)) 772 | 773 | return exterior_coords + interior_coords 774 | 775 | def create_regularity_gdf(morpho_Tiles_gdf): 776 | """Updated from osmuf https://github.com/AtelierLibre/osmuf""" 777 | 778 | # poly_area_m2 --> block area 779 | # SEC_area_m2 --> smallest enclosing circle 780 | # geometry --> smallest enclosing circle geometry 781 | # calculate "regularity" as "the ratio between the area of the polygon and 782 | # the area of the circumscribed circle C" Barthelemy M. and Louf R., (2014) 783 | # regularity --> poly_area_m2 / SEC_area_m2 784 | regularity_gdf = gen_regularity(morpho_Tiles_gdf) 785 | 786 | # Add the radius of SEC_R_m circumscribed circle 787 | regularity_gdf['SEC_area_m2'] = regularity_gdf['SEC_area_m2'].astype('float') 788 | regularity_gdf['SEC_R_m'] = np.sqrt(regularity_gdf['SEC_area_m2']/math.pi) 789 | 790 | # Add the position of the center of the block circumscribed circle 791 | regularity_gdf['centroid'] = regularity_gdf['geometry'].centroid 792 | 793 | return regularity_gdf 794 | 795 | ##################### Smallest enclosing circle ########################### 796 | 797 | class sec(object): 798 | """ 799 | Smallest enclosing circle - Library (Python) 800 | https://www.nayuki.io/page/smallest-enclosing-circle 801 | 802 | Copyright (c) 2018 Project Nayuki 803 | https://www.nayuki.io/page/smallest-enclosing-circle 804 | 805 | This program is free software: you can redistribute it and/or modify 806 | it under the terms of the GNU Lesser General Public License as published by 807 | the Free Software Foundation, either version 3 of the License, or 808 | (at your option) any later version. 809 | 810 | This program is distributed in the hope that it will be useful, 811 | but WITHOUT ANY WARRANTY; without even the implied warranty of 812 | MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 813 | GNU Lesser General Public License for more details. 814 | 815 | You should have received a copy of the GNU Lesser General Public License 816 | along with this program (see COPYING.txt and COPYING.LESSER.txt). 817 | If not, see