32 | 
33 | API for local information on COVID-19
(hotlines, websites, test sites, health departments)
6 | 7 | Search dashboard • 8 | Swagger docs • 9 | Contact 10 | 11 |
12 | 13 | 18 | 19 | 20 | ## What is this good for? 21 | 22 | This API provides local information and addresses on COVID-19 for a given location (e.g. local hotlines & websites, nearby test sites, relevant health departments). It can be easily integrated into existing websites and apps, giving the user relevant information for their location. E.g., a tracing app could use this API to refer the user to their nearest test site in case of an infection risk. Features: 23 | 24 | - **Local information** (hotlines, websites, test sites, health departments) for major German cities (more coming soon) 25 | - **Integration in websites & apps** via REST API and Python/JavaScript clients 26 | - **Built-in location search** for cities, neighborhoods, states, ... 27 | 28 | Check out our [search dashboard](http://ec2-3-90-67-33.compute-1.amazonaws.com:8600) to get an idea of which data the API offers! 29 | 30 |
31 |
32 |
33 |
Our API in practice – web and mobile
36 | 37 | 38 | ## Client libraries 39 | 40 | We now have client libraries for Python ([covid-local-py](https://github.com/cotect/covid-local-py)) and JavaScript ([covid-local-js](https://github.com/cotect/covid-local-js)), so you can access the API directly from code. Both repos show some examples in their READMEs. Please make sure to still read the usage guide below to get an idea of how the API works in general. (Need a client for another language? [Generate it yourself](#generating-client-libraries) or [reach out](mailto:johannes.rieke@gmail.com)) 41 | 42 | 43 | ## Usage 44 | 45 | You can try out the API using our live deployment at 46 | http://ec2-3-90-67-33.compute-1.amazonaws.com (note that the main page does not return information!). 47 | 48 | For example, to get all local information for Berlin Mitte, go to: 49 | 50 | http://ec2-3-90-67-33.compute-1.amazonaws.com/all?place_name=Berlin&20Mitte 51 | 52 | The data is returned as JSON. Note that information for hierachically higher areas 53 | (e.g. country-wide hotlines) are automatically returned as well. 54 | 55 | ### Endpoints 56 | 57 | Above, we used the `/all` endpoint to request all information from the database. You can 58 | also use the more specific endpoints `/hotlines`, `/websites`, `/test_sites` and 59 | `/health_departments`, which will only a return a subset of the data, e.g.: 60 | 61 | http://ec2-3-90-67-33.compute-1.amazonaws.com/hotlines?place_name=Berlin&20Mitte 62 | 63 | ### Place search 64 | 65 | To specify the location of the query, we support two options: You can either use the 66 | `place_name` parameter like above (with a city, neighborhood, state, ...). Under the 67 | hood, this searches on geonames.org and simply uses the first result to search our 68 | database. To get more control over the place selection (e.g. if the place name is 69 | ambiguous), you can use the `/places` endpoint: 70 | 71 | http://ec2-3-90-67-33.compute-1.amazonaws.com/places?q=Berlin&20Mitte 72 | 73 | This returns a list of places for your query. It uses the 74 | [geonames.org location search](http://www.geonames.org/export/geonames-search.html) 75 | with sensible defaults (e.g. search only for cities and districts) and clean 76 | formatting. If you found the correct place among these results, you can extract its 77 | `geonames_id` and pass it to the other endpoints like this: 78 | 79 | http://ec2-3-90-67-33.compute-1.amazonaws.com/all?geonames_id=2950159 80 | 81 | ### Docs 82 | 83 | For more details on endpoints, query parameters, and output formats, please have a 84 | look at the [Swagger docs](http://ec2-3-90-67-33.compute-1.amazonaws.com/docs). 85 | 86 | 87 | ## Running the API locally 88 | 89 | To run the API locally, clone this repo and run the following command: 90 | 91 | cd ./covid-local-api/app/covid_local_api 92 | uvicorn local_test:app --reload 93 | 94 | The API should now be accessible at 127.0.0.1:8000. You can also deploy the API with 95 | docker, using the dockerfile in the repo (note that this will serve the API at port 80 instead of 8000). 96 | 97 | To start the [search dashboard](http://ec2-3-90-67-33.compute-1.amazonaws.com:8600), 98 | run: 99 | 100 | streamlit run search-dashboard.py 101 | 102 | This will start the dashboard on port 8501. Note that the dockerfile automatically 103 | starts the dashboard along with the API (using the `prestart.sh` file; docker deployment uses port 8600 instead of 8501). 104 | 105 | 106 | ## Data 107 | 108 | Help us collect new data with our Google Form: 109 | [https://bit.ly/covid-local-form](https://bit.ly/covid-local-form) 110 | 111 | The data for this project is stored in a 112 | [Google Sheet](https://docs.google.com/spreadsheets/d/1AXadba5Si7WbJkfqQ4bN67cbP93oniR-J6uN0_Av958/edit?usp=sharing) 113 | (note that there is one worksheet for each data type). If you think that any of the 114 | data is wrong, please add a comment directly to the document or write to 115 | johannes.rieke@gmail.com. You can also use our 116 | [dashboard](http://ec2-3-90-67-33.compute-1.amazonaws.com:8600) to search through the 117 | data. 118 | 119 | 120 | ## Requirements 121 | 122 | Python 3.7 and all packages in requirements.txt 123 | 124 | 125 | ## Generating client libraries 126 | 127 | Client libraries can be generated automatically with the [OpenAPI Generator](https://openapi-generator.tech). If you update an existing client, please make sure to copy its `README.md` file before, as it was probably adapted manually. 128 | 129 | For Python, run: 130 | 131 | ```shell 132 | openapi-generator generate -i http://ec2-3-90-67-33.compute-1.amazonaws.com/openapi.json -g python -o covid-local-py --additional-properties packageName=covid_local,projectName=covid-local-py,packageUrl=https://github.com/cotect/covid-local-py,packageVersion=0.1.0 --git-host github.com --git-user-id cotect --git-repo-id covid-local-py 133 | ``` 134 | 135 | For JavaScript, run: 136 | 137 | ```shell 138 | openapi-generator generate -i http://ec2-3-90-67-33.compute-1.amazonaws.com/openapi.json -g javascript -o covid-local-js --additional-properties moduleName=CovidLocal,projectName=covid-local-js,projectVersion=0.1.0 --git-host github.com --git-user-id cotect --git-repo-id covid-local-js 139 | ``` 140 | 141 | Both commands will use the API definition from the live deployment and write the client to `covid-local-py` or `covid-local-js`. After creating the client, you need to manually replace all occurences (in all files) of `http://localhost` with `http://ec2-3-90-67-33.compute-1.amazonaws.com`. This way, the client will pull data from the live deployment of the API (instead of a local deployment). 142 | -------------------------------------------------------------------------------- /app/covid_local_api/place_handler.py: -------------------------------------------------------------------------------- 1 | import logging 2 | import csv 3 | import json 4 | from typing import List, Optional 5 | 6 | from covid_local_api.utils.place_request_utils import ( 7 | GEONAMES_ID_PREFIX, 8 | OSM_ID_PREFIX, 9 | map_geonames_to_wikidata, 10 | map_osm_to_wikidata, 11 | map_wikidata_to_geonames, 12 | map_wikidata_to_osm, 13 | request_geonames_hierarchy, 14 | request_osm_hierarchy, 15 | search_geonames, 16 | search_osm, 17 | ) 18 | 19 | 20 | def load_place_hierarchy(hierarchy_csv_path: str): 21 | place_hierarchy = {} 22 | 23 | with open(hierarchy_csv_path, "r") as f: 24 | csv_reader = csv.reader(f, delimiter=",") 25 | for row in csv_reader: 26 | if row and len(row) == 2: 27 | place_hierarchy[row[0]] = row[1] 28 | return place_hierarchy 29 | 30 | 31 | def load_place_mapping(mapping_json_path: str): 32 | place_mapping = {} 33 | 34 | with open(mapping_json_path, "r") as f: 35 | place_mapping = json.load(f) 36 | 37 | return place_mapping 38 | 39 | 40 | def create_inverse_mapping( 41 | input_mapping: dict, filter_prefix: Optional[str] = None 42 | ) -> dict: 43 | new_mapping = {} 44 | for key, value_set in input_mapping.items(): 45 | if filter_prefix and not key.startswith(filter_prefix): 46 | continue 47 | 48 | for item in value_set: 49 | new_mapping.setdefault(item, set()).add(key) 50 | return new_mapping 51 | 52 | 53 | class PlaceHandler: 54 | def __init__( 55 | self, 56 | place_wikidata_mapping: dict, 57 | place_hierarchy: dict, 58 | country_codes: List[str] = None, 59 | resolve_unknown: bool = False, 60 | ): 61 | 62 | self._log = logging.getLogger(__name__) 63 | 64 | self._place_wikidata_mapping = place_wikidata_mapping 65 | self._place_hierarchy = place_hierarchy 66 | self._country_codes = country_codes 67 | 68 | self._place_inverse_mapping = create_inverse_mapping( 69 | self._place_wikidata_mapping 70 | ) 71 | 72 | def __getitem__(self, key): 73 | key = key.strip().upper() 74 | 75 | if key in self._place_wikidata_mapping: 76 | return list(self._place_wikidata_mapping[key]) 77 | 78 | if key in self._place_inverse_mapping: 79 | return list(self._place_inverse_mapping[key]) 80 | 81 | return [] 82 | 83 | def __contains__(self, key): 84 | key = key.strip().upper() 85 | 86 | if key in self._place_wikidata_mapping or key in self._place_inverse_mapping: 87 | return True 88 | 89 | def search_places(self, query: str, limit: int = 5): 90 | search_result = [] 91 | added_wikidata_ids = set() 92 | geonames_results = search_geonames(query, limit, self._country_codes) 93 | for result in geonames_results: 94 | wikidata_id = self.map_geonames_to_wikidata(result[0]) 95 | if wikidata_id and wikidata_id not in added_wikidata_ids: 96 | search_result.append((wikidata_id, result[1])) 97 | added_wikidata_ids.add(wikidata_id) 98 | 99 | if len(search_result) < limit: 100 | osm_results = search_osm(query, limit, self._country_codes) 101 | for result in osm_results: 102 | wikidata_id = self.map_osm_to_wikidata(result[0]) 103 | if wikidata_id and wikidata_id not in added_wikidata_ids: 104 | search_result.append((wikidata_id, result[1])) 105 | added_wikidata_ids.add(wikidata_id) 106 | if len(search_result) == limit: 107 | break 108 | 109 | return search_result 110 | 111 | def resolve_hierarchies(self, key: str) -> list: 112 | key = key.strip().upper() 113 | 114 | place_hierarchies = [] 115 | 116 | if key in self._place_wikidata_mapping: 117 | for wikidata_id in self._place_wikidata_mapping[key]: 118 | place_hierarchies.append(self.resolve_wikidata_hierarchy(wikidata_id)) 119 | return place_hierarchies 120 | 121 | key_to_add = None 122 | if key in self._place_inverse_mapping: 123 | key_to_add = key 124 | elif key.startswith(GEONAMES_ID_PREFIX): 125 | key_to_add = self.map_geonames_to_wikidata(key) 126 | print(key_to_add) 127 | elif key.startswith(OSM_ID_PREFIX): 128 | key_to_add = self.map_osm_to_wikidata()(key) 129 | 130 | if key_to_add: 131 | # Key is wikidata id 132 | place_hierarchies.append(self.resolve_wikidata_hierarchy(key_to_add)) 133 | return place_hierarchies 134 | 135 | def map_geonames_to_wikidata(self, geonames_id: str) -> str: 136 | geonames_id = str(geonames_id).strip().upper() 137 | if not geonames_id.startswith(GEONAMES_ID_PREFIX): 138 | geonames_id = GEONAMES_ID_PREFIX + geonames_id 139 | 140 | # TODO only return one result? 141 | if geonames_id in self and self[geonames_id]: 142 | return self[geonames_id][0] 143 | else: 144 | return map_geonames_to_wikidata(geonames_id) 145 | 146 | def map_wikidata_to_geonames(self, wikidata_id: str) -> str: 147 | if wikidata_id in self and self[wikidata_id]: 148 | for result in self[wikidata_id]: 149 | if result.startswith(GEONAMES_ID_PREFIX): 150 | return result 151 | return map_wikidata_to_geonames(wikidata_id) 152 | 153 | def map_osm_to_wikidata(self, osm_id: str) -> list: 154 | osm_id = str(osm_id).strip().upper() 155 | if not osm_id.startswith(OSM_ID_PREFIX): 156 | osm_id = OSM_ID_PREFIX + osm_id 157 | 158 | # TODO only return one result? 159 | if osm_id in self and self[osm_id]: 160 | return self[osm_id][0] 161 | else: 162 | return map_osm_to_wikidata(osm_id) 163 | 164 | def map_wikidata_to_osm(self, wikidata_id: str): 165 | if wikidata_id in self and self[wikidata_id]: 166 | for result in self[wikidata_id]: 167 | if result.startswith(OSM_ID_PREFIX): 168 | return result 169 | 170 | return map_wikidata_to_osm(wikidata_id) 171 | 172 | def request_wikidata_hierarchy_with_geonames(self, wikidata_id: str): 173 | wikidata_id = wikidata_id.strip().upper() 174 | wikidata_hierarchy = [] 175 | try: 176 | geonames_id = self.map_wikidata_to_geonames(wikidata_id) 177 | if not geonames_id: 178 | return [] 179 | geonames_hierarchy = request_geonames_hierarchy(geonames_id) 180 | for geonames_id in geonames_hierarchy: 181 | wikidata_id = self.map_geonames_to_wikidata(geonames_id) 182 | if wikidata_id and wikidata_id not in wikidata_hierarchy: 183 | wikidata_hierarchy.append(wikidata_id) 184 | return wikidata_hierarchy 185 | except Exception: 186 | self._log.info("Failed to request geonames hierarchy.", exc_info=True) 187 | return [] 188 | 189 | def request_wikidata_hierarchy_with_osm(self, wikidata_id: str): 190 | wikidata_id = wikidata_id.strip().upper() 191 | wikidata_hierarchy = [] 192 | try: 193 | osm_id = self.map_wikidata_to_osm(wikidata_id) 194 | if not osm_id: 195 | return [] 196 | osm_hierarchy = request_osm_hierarchy(osm_id) 197 | for osm_id in osm_hierarchy: 198 | wikidata_id = self.map_osm_to_wikidata(osm_id) 199 | if wikidata_id and wikidata_id not in wikidata_hierarchy: 200 | wikidata_hierarchy.append(wikidata_id) 201 | return wikidata_hierarchy 202 | except Exception: 203 | self._log.info("Failed to request osm hierarchy.", exc_info=True) 204 | return [] 205 | 206 | def resolve_wikidata_hierarchy( 207 | self, wikidata_id: str, prefer_geonames: bool = True, prefer_osm: bool = False 208 | ) -> List[str]: 209 | 210 | if wikidata_id in self._place_hierarchy: 211 | wikidata_hierarchy = [] 212 | current_item = wikidata_id 213 | wikidata_hierarchy.append(current_item) 214 | while current_item in self._place_hierarchy: 215 | current_item = self._place_hierarchy[current_item] 216 | wikidata_hierarchy.append(current_item) 217 | return list(reversed(wikidata_hierarchy)) 218 | 219 | geonames_wkdt_hierarchy = [] 220 | osm_wkdt_hierarchy = [] 221 | 222 | if prefer_geonames or not prefer_osm: 223 | geonames_wkdt_hierarchy = self.request_wikidata_hierarchy_with_geonames( 224 | wikidata_id 225 | ) 226 | if not geonames_wkdt_hierarchy: 227 | # Use osm as fallback 228 | self._log.info( 229 | "Fallback to using osm to resolve wikidata hierachy: " + wikidata_id 230 | ) 231 | return self.request_wikidata_hierarchy_with_osm(wikidata_id) 232 | 233 | if prefer_osm or not prefer_geonames: 234 | osm_wkdt_hierarchy = self.request_wikidata_hierarchy_with_osm(wikidata_id) 235 | if not osm_wkdt_hierarchy and not geonames_wkdt_hierarchy: 236 | # Use geonames as fallback 237 | self._log.info( 238 | "Fallback to using geonames to resolve wikidata hierachy: " 239 | + wikidata_id 240 | ) 241 | return self.request_wikidata_hierarchy_with_geonames(wikidata_id) 242 | 243 | if not geonames_wkdt_hierarchy and not osm_wkdt_hierarchy: 244 | self._log.info("Failed to get wikidata hiearchy for " + wikidata_id) 245 | 246 | if len(geonames_wkdt_hierarchy) > len(osm_wkdt_hierarchy): 247 | return geonames_wkdt_hierarchy 248 | else: 249 | return osm_wkdt_hierarchy 250 | -------------------------------------------------------------------------------- /app/covid_local_api/endpoints.py: -------------------------------------------------------------------------------- 1 | import geocoder 2 | import uvicorn 3 | from starlette.responses import RedirectResponse 4 | from fastapi import FastAPI, Query, HTTPException 5 | from typing import List 6 | from enum import Enum 7 | from timeloop import Timeloop 8 | from datetime import timedelta 9 | 10 | from covid_local_api.__version__ import __version__ 11 | from covid_local_api.db_handler import DatabaseHandler 12 | from covid_local_api.schema import ( 13 | ResultsList, 14 | Place, 15 | ) 16 | from covid_local_api.utils import endpoint_utils, place_request_utils 17 | 18 | 19 | # TODO: Implement place handler code at some point in the future like below. 20 | # from covid_local_api.place_handler import ( 21 | # PlaceHandler, 22 | # load_place_hierarchy, 23 | # load_place_mapping, 24 | # ) 25 | # data_path = os.path.join(os.path.dirname(os.path.realpath(__file__)), "data") 26 | # place_handler = PlaceHandler( 27 | # load_place_mapping(os.path.join(data_path, "DE_placeid-to-wikidata.json")), 28 | # load_place_hierarchy(os.path.join(data_path, "DE_place-hierarchy.csv")), 29 | # country_codes=["DE"], 30 | # ) 31 | # @app.get("/test_place_handler") 32 | # def test_place_handler(place_id: str = Query(..., description="Place ID to filter.")): 33 | # return place_handler.resolve_hierarchies(place_id) 34 | 35 | 36 | # Initialize database and schedule daily update 37 | db = DatabaseHandler() 38 | tl = Timeloop() 39 | 40 | 41 | @tl.job(interval=timedelta(seconds=86400)) # once per day 42 | def update_database(): 43 | db.update_database() 44 | 45 | 46 | tl.start() 47 | 48 | 49 | # Initialize API 50 | app = FastAPI( 51 | title="COVID-19 Local API", 52 | description="API to get local help information about COVID-19 (hotlines, websites, " 53 | "test sites, health departments)", 54 | version=__version__, 55 | ) 56 | 57 | 58 | # ---------------------------------- Helper functions ---------------------------------- 59 | place_name_query = Query( 60 | None, 61 | description="The name of the place, e.g. a city, neighborhood, state (either " 62 | "place_name or geonames_id must be provided)", 63 | ) 64 | 65 | 66 | geonames_id_query = Query( 67 | None, 68 | description="The geonames.org id of the place (either place_name or " 69 | "geonames_id must be provided)", 70 | ) 71 | 72 | 73 | class SearchProvider(str, Enum): 74 | """Enum of the available search providers for the places endpoint""" 75 | 76 | geonames = "geonames" 77 | 78 | 79 | def geocoder_to_place(result): 80 | """Convert a result object from geocoder to a Place object""" 81 | return Place( 82 | name=result.address, 83 | country=result.country, 84 | country_code=result.country_code, 85 | state=result.state, 86 | description=result.description + " - " + result.class_description, 87 | geonames_id=result.geonames_id, 88 | lat=result.lat, 89 | lon=result.lng, 90 | search_provider=SearchProvider.geonames, 91 | ) 92 | 93 | 94 | def find_place(place_name=None, geonames_id=None): 95 | """Finds and returns the place for the given query parameters. 96 | 97 | If geonames_id is given, simply get some more information about it. If 98 | place_name is given, search the /places endpoint and return the first result. If 99 | neither is given, raise an error. 100 | 101 | Args: 102 | place_name (str, optional): The name of the place to search for (used as query 103 | parameter for the places endpoint) 104 | geonames_id (int, optional): The geonames.org id of the place 105 | 106 | Returns: 107 | Place: The found place 108 | """ 109 | if geonames_id is None and place_name is None: 110 | raise HTTPException(400, "Either place_name or geonames_id must be provided") 111 | elif geonames_id is None: 112 | # Search by place_name and use first search result. 113 | places = search_places(q=place_name, limit=1, search_provider="geonames") 114 | if len(places) == 0: 115 | raise HTTPException( 116 | 400, f"Could not find any match for place_name: {place_name}" 117 | ) 118 | else: 119 | return places[0] 120 | else: 121 | # Get details for this geonames_id and return as Place object. 122 | search_result = geocoder.geonames( 123 | geonames_id, key=place_request_utils.get_geonames_user(), method="details" 124 | )[0] 125 | place = geocoder_to_place(search_result) 126 | return place 127 | 128 | 129 | def get_hierarchy(geonames_id): 130 | """Returns geonames ids of hierarchical parents (e.g. country for a city)""" 131 | hierarchy = geocoder.geonames( 132 | geonames_id, key=place_request_utils.get_geonames_user(), method="hierarchy", 133 | ) 134 | hierarchy = hierarchy[::-1] # reverse, so that more local areas come first 135 | geonames_ids_hierarchy = [item.geonames_id for item in hierarchy] 136 | return geonames_ids_hierarchy 137 | 138 | 139 | # ---------------------------------- Endpoints ----------------------------------------- 140 | @app.get( 141 | "/places", 142 | summary="Search for places via free-form query", 143 | response_model=List[Place], 144 | ) 145 | def search_places( 146 | q: str = Query( 147 | ..., 148 | description="Free-form query string (e.g. a city, neighborhood, state, ...)", 149 | ), 150 | limit: int = Query(5, description="Maximum number of entries to return"), 151 | search_provider: SearchProvider = Query( 152 | SearchProvider.geonames, 153 | description="The search provider (only geonames supported so far)", 154 | ), 155 | ): 156 | if search_provider == SearchProvider.geonames: 157 | # Search geonames API. 158 | search_results = geocoder.geonames( 159 | q, 160 | key=place_request_utils.get_geonames_user(), 161 | maxRows=limit, 162 | featureClass=["A", "P"], 163 | ) 164 | 165 | # Format the search results to Place objects and return them. 166 | places = [geocoder_to_place(result) for result in search_results] 167 | return places 168 | else: 169 | raise HTTPException(400, f"Search provider not supported: {search_provider}") 170 | 171 | 172 | @app.get( 173 | "/all", summary="Get all items for a place", response_model=ResultsList, 174 | ) 175 | def get_all( 176 | place_name: str = place_name_query, 177 | geonames_id: int = geonames_id_query, 178 | max_distance: float = Query( 179 | 0.5, description="Maximum distance in degrees lon/lat for test sites" 180 | ), 181 | limit: int = Query(5, description="Maximum number of test sites to return"), 182 | ): 183 | place = find_place(place_name, geonames_id) 184 | geonames_ids_hierarchy = get_hierarchy(place.geonames_id) 185 | return { 186 | "place": place, 187 | "hotlines": db.get("hotlines", geonames_ids_hierarchy), 188 | "websites": db.get("websites", geonames_ids_hierarchy), 189 | "test_sites": db.get_nearby( 190 | "test_sites", place.lat, place.lon, max_distance=max_distance, limit=limit 191 | ), 192 | "health_departments": db.get("health_departments", geonames_ids_hierarchy), 193 | } 194 | 195 | 196 | @app.get( 197 | "/hotlines", summary=f"Get hotlines for a place", response_model=ResultsList, 198 | ) 199 | def get_hotlines( 200 | place_name: str = place_name_query, geonames_id: int = geonames_id_query, 201 | ): 202 | place = find_place(place_name, geonames_id) 203 | geonames_ids_hierarchy = get_hierarchy(place.geonames_id) 204 | return { 205 | "place": place, 206 | "hotlines": db.get("hotlines", geonames_ids_hierarchy), 207 | } 208 | 209 | 210 | @app.get( 211 | "/websites", summary=f"Get websites for a place", response_model=ResultsList, 212 | ) 213 | def get_websites( 214 | place_name: str = place_name_query, geonames_id: int = geonames_id_query, 215 | ): 216 | place = find_place(place_name, geonames_id) 217 | geonames_ids_hierarchy = get_hierarchy(place.geonames_id) 218 | return { 219 | "place": place, 220 | "websites": db.get("websites", geonames_ids_hierarchy), 221 | } 222 | 223 | 224 | @app.get( 225 | "/test_sites", 226 | summary=f"Get nearby test sites for a place (sorted by distance to place)", 227 | response_model=ResultsList, 228 | ) 229 | def get_test_sites( 230 | place_name: str = place_name_query, 231 | geonames_id: int = geonames_id_query, 232 | max_distance: float = Query( 233 | 0.5, description="Maximum distance in degrees lon/lat for test sites" 234 | ), 235 | limit: int = Query(5, description="Maximum number of test sites to return"), 236 | ): 237 | place = find_place(place_name, geonames_id) 238 | # lat, lon = get_lat_lon(geonames_id) 239 | return { 240 | "geonames_id": geonames_id, 241 | "test_sites": db.get_nearby( 242 | "test_sites", place.lat, place.lon, max_distance=max_distance, limit=limit 243 | ), 244 | } 245 | 246 | 247 | @app.get( 248 | "/health_departments", 249 | summary=f"Get responsible health departments for a place", 250 | response_model=ResultsList, 251 | ) 252 | # TODO: This doesn't return results if e.g. Berlin is selected but the health department 253 | # is in Berlin Mitte. Maybe also search for the direct children of the geonames id 254 | # (but is direct children enough)? 255 | def get_health_departments( 256 | place_name: str = place_name_query, geonames_id: int = geonames_id_query, 257 | ): 258 | place = find_place(place_name, geonames_id) 259 | geonames_ids_hierarchy = get_hierarchy(place.geonames_id) 260 | return { 261 | "place": place, 262 | "health_departments": db.get("health_departments", geonames_ids_hierarchy), 263 | } 264 | 265 | 266 | @app.get( 267 | "/test", summary="Shows all entries for Berlin Mitte (redirects to /all endpoint)", 268 | ) 269 | def test(): 270 | response = RedirectResponse(url="/all?geonames_id=6545310") 271 | return response 272 | 273 | 274 | # Use function names as operation IDs 275 | endpoint_utils.use_route_names_as_operation_ids(app) 276 | 277 | 278 | # Run uvicorn server directly in here for debugging 279 | if __name__ == "__main__": 280 | uvicorn.run(app, debug=True) 281 | -------------------------------------------------------------------------------- /app/covid_local_api/utils/place_request_utils.py: -------------------------------------------------------------------------------- 1 | import logging 2 | import os 3 | import random 4 | from typing import List 5 | 6 | import requests 7 | from qwikidata.linked_data_interface import get_entity_dict_from_api 8 | from qwikidata.sparql import return_sparql_query_results 9 | 10 | log = logging.getLogger(__name__) 11 | 12 | GEONAMES_ENDPOINT = os.getenv("GEONAMES_ENDPOINT", "http://api.geonames.org") 13 | GEONAMES_ENDPOINT_V3 = os.getenv("GEONAMES_ENDPOINT_V3", "http://www.geonames.org") 14 | GEONAMES_USERS = os.getenv("GEONAMES_USERS", "sap_ekg").replace(" ", "").split(",") 15 | 16 | # TODO: Refactor this. 17 | def get_geonames_user(): 18 | """Return a random geonames username""" 19 | return random.choice(GEONAMES_USERS) 20 | 21 | OSM_NOMATIM_ENDPOINT = os.getenv( 22 | "OSM_NOMATIM_ENDPOINT", "https://nominatim.openstreetmap.org" 23 | ) 24 | 25 | IGNORED_GEONAMES_ID = ["6295630", "6255148"] 26 | 27 | OSM_TYPE_MAPPING = {"relation": "R", "way": "W", "node": "N"} 28 | OSM_ID_PREFIX = "OSM:" 29 | GEONAMES_ID_PREFIX = "GN:" 30 | 31 | 32 | def request_geonames_hierarchy(geonames_id: str, fast: bool = True) -> List[str]: 33 | geonames_id = str(geonames_id).strip().upper().lstrip(GEONAMES_ID_PREFIX) 34 | if fast: 35 | # Only request a single JSON instead of the full hierarchy 36 | try: 37 | # 38 | request_url = ( 39 | GEONAMES_ENDPOINT_V3 40 | + "/getJSON?geonameId={geonames_id}&style=full&username={geonames_user}" 41 | ) 42 | response_json = requests.get( 43 | request_url.format( 44 | geonames_id=geonames_id, geonames_user=random.choice(GEONAMES_USERS) 45 | ) 46 | ).json() 47 | sorted_geonames_hierarchy = [] 48 | 49 | if "countryId" in response_json and response_json["countryId"]: 50 | sorted_geonames_hierarchy.append( 51 | GEONAMES_ID_PREFIX + str(response_json["countryId"]) 52 | ) 53 | 54 | if "adminId1" in response_json and response_json["adminId1"]: 55 | sorted_geonames_hierarchy.append( 56 | GEONAMES_ID_PREFIX + str(response_json["adminId1"]) 57 | ) 58 | 59 | if "adminId2" in response_json and response_json["adminId2"]: 60 | sorted_geonames_hierarchy.append( 61 | GEONAMES_ID_PREFIX + str(response_json["adminId2"]) 62 | ) 63 | 64 | if "adminId3" in response_json and response_json["adminId3"]: 65 | sorted_geonames_hierarchy.append( 66 | GEONAMES_ID_PREFIX + str(response_json["adminId3"]) 67 | ) 68 | 69 | if "adminId4" in response_json and response_json["adminId4"]: 70 | sorted_geonames_hierarchy.append( 71 | GEONAMES_ID_PREFIX + str(response_json["adminId4"]) 72 | ) 73 | 74 | if "adminId5" in response_json and response_json["adminId5"]: 75 | sorted_geonames_hierarchy.append( 76 | GEONAMES_ID_PREFIX + str(response_json["adminId5"]) 77 | ) 78 | 79 | if "geonameId" in response_json and response_json["geonameId"]: 80 | sorted_geonames_hierarchy.append( 81 | GEONAMES_ID_PREFIX + str(response_json["geonameId"]) 82 | ) 83 | 84 | return sorted_geonames_hierarchy 85 | except Exception: 86 | log.info("Failed to get geonames hierarchy.", exc_info=True) 87 | return None 88 | else: 89 | try: 90 | request_url = ( 91 | GEONAMES_ENDPOINT_V3 92 | + "/hierarchyJSON?style=full&geonameId={geonames_id}&username={geonames_user}" 93 | ) 94 | response = requests.get( 95 | request_url.format( 96 | geonames_id=geonames_id, geonames_user=random.choice(GEONAMES_USERS) 97 | ) 98 | ) 99 | sorted_geonames_hierarchy = [] 100 | for area in response.json()["geonames"]: 101 | area_id = str(area["geonameId"]) 102 | if area_id and area_id not in IGNORED_GEONAMES_ID: 103 | if "adminId5" in area and area["adminId5"]: 104 | # admin id 5 does not seem to be fully supported in hierarchy 105 | # add it manually 106 | area_admin_id_5 = area["adminId5"] 107 | if ( 108 | area_admin_id_5 not in sorted_geonames_hierarchy 109 | and area_id != area_admin_id_5 110 | ): 111 | sorted_geonames_hierarchy.append( 112 | GEONAMES_ID_PREFIX + str(area_admin_id_5) 113 | ) 114 | sorted_geonames_hierarchy.append(GEONAMES_ID_PREFIX + str(area_id)) 115 | return sorted_geonames_hierarchy 116 | except Exception: 117 | log.info("Failed to get geonames hierarchy.", exc_info=True) 118 | return None 119 | 120 | 121 | def request_osm_hierarchy(osm_id: str) -> List[str]: 122 | osm_id = str(osm_id).strip().upper().lstrip(OSM_ID_PREFIX) 123 | try: 124 | # Assume R as base type 125 | osm_type = "R" 126 | if osm_id[0] in OSM_TYPE_MAPPING.values(): 127 | osm_type = osm_id[0] 128 | osm_id = osm_id[1:] 129 | 130 | request_url = ( 131 | OSM_NOMATIM_ENDPOINT 132 | + "/details.php?osmtype={osm_type}&osmid={osm_id}&format=json&addressdetails=1&hierarchy=0&linkedplaces=0&polygon_geojson=0&keywords=0&extratags=0" 133 | ) 134 | response = requests.get(request_url.format(osm_type=osm_type, osm_id=osm_id)) 135 | 136 | country_code = None 137 | osm_id_to_level = [] 138 | for area in response.json()["address"]: 139 | if "osm_id" in area and area["osm_id"]: 140 | osm_id = str(area["osm_type"]) + str(area["osm_id"]) 141 | osm_id_to_level.append((osm_id, int(area["admin_level"]))) 142 | elif "type" in area and area["type"] == "country_code": 143 | # extract country code 144 | country_code = area["localname"] 145 | 146 | sorted_osm_hierarchy = [] 147 | if country_code: 148 | country_id = map_countrycode_to_osm(country_code) 149 | if country_id: 150 | sorted_osm_hierarchy.append(country_id) 151 | 152 | # sort by admin level 153 | osm_id_to_level.sort(key=lambda x: x[1]) 154 | for osm_id in list(zip(*osm_id_to_level))[0]: 155 | sorted_osm_hierarchy.append(OSM_ID_PREFIX + osm_id) 156 | 157 | return sorted_osm_hierarchy 158 | except Exception: 159 | log.debug("Failed to get osm geonames hierarchy: " + osm_id, exc_info=True) 160 | return None 161 | 162 | 163 | def map_countrycode_to_osm(country_code: str) -> str: 164 | try: 165 | request_url = ( 166 | OSM_NOMATIM_ENDPOINT + "/search?country={country_code}&format=json" 167 | ) 168 | response = requests.get(request_url.format(country_code=country_code.upper())) 169 | osm_obj = response.json()[0] 170 | osm_type = OSM_TYPE_MAPPING[osm_obj["osm_type"]] 171 | osm_id = osm_type + str(osm_obj["osm_id"]) 172 | return OSM_ID_PREFIX + osm_id 173 | except Exception: 174 | log.debug("Failed to map country code to osm id.", exc_info=True) 175 | return None 176 | 177 | 178 | def map_osm_to_wikidata(osm_id: str) -> str: 179 | osm_id = str(osm_id).strip().upper().lstrip(OSM_ID_PREFIX) 180 | try: 181 | # Try to get wikidata id from nominatim API 182 | if osm_id[0] not in OSM_TYPE_MAPPING.values(): 183 | raise Exception("Only ids with N,W,R prefix are allowed.") 184 | 185 | nominatim_lookup_url = ( 186 | OSM_NOMATIM_ENDPOINT 187 | + "/lookup?osm_ids={osm_id}&format=json&extratags=1&addressdetails=0&namedetails=0" 188 | ) 189 | response = requests.get(nominatim_lookup_url.format(osm_id=osm_id)) 190 | if len(response.json()) > 1: 191 | log.info("Found more than one wikidata id for osm id: " + osm_id) 192 | if "wikidata" not in response.json()[0]["extratags"]: 193 | raise Exception("No wikidata information found with OSM item.") 194 | return response.json()[0]["extratags"]["wikidata"] 195 | except Exception: 196 | try: 197 | if osm_id[0] != "R": 198 | raise Exception( 199 | "Only osm relations (prefix R) are supported by wikidata." 200 | ) 201 | 202 | osm_id = osm_id[1:] 203 | 204 | # Fallback: search for id in wikidata 205 | sparql_query = """ 206 | SELECT ?id WHERE {{ 207 | ?id wdt:{id_type} "{id}". 208 | }} 209 | """ 210 | 211 | res = return_sparql_query_results( 212 | sparql_query.format(id_type="P402", id=osm_id) 213 | ) 214 | if len(res["results"]["bindings"]) > 1: 215 | log.info("Found more than one wikidata id for osm id: " + osm_id) 216 | return os.path.basename(res["results"]["bindings"][0]["id"]["value"]) 217 | except Exception: 218 | log.info("Failed to map osm id to wikidata id.", exc_info=True) 219 | return None 220 | 221 | 222 | def map_geonames_to_wikidata(geonames_id: str) -> str: 223 | geonames_id = str(geonames_id).strip().upper().lstrip(GEONAMES_ID_PREFIX) 224 | try: 225 | request_url = ( 226 | GEONAMES_ENDPOINT_V3 227 | + "/getJSON?geonameId={geonames_id}&style=full&username={geonames_user}" 228 | ) 229 | response_json = requests.get( 230 | request_url.format( 231 | geonames_id=geonames_id, geonames_user=random.choice(GEONAMES_USERS) 232 | ) 233 | ).json() 234 | 235 | if "alternateNames" in response_json and response_json["alternateNames"]: 236 | for tag in response_json["alternateNames"]: 237 | if "lang" in tag and tag["lang"] == "wkdt": 238 | if tag["name"]: 239 | return tag["name"] 240 | raise Exception() 241 | except Exception: 242 | try: 243 | # Fallback: search for id in wikidata 244 | sparql_query = """ 245 | SELECT ?id WHERE {{ 246 | ?id wdt:{id_type} "{id}". 247 | }} 248 | """ 249 | 250 | res = return_sparql_query_results( 251 | sparql_query.format(id_type="P1566", id=geonames_id) 252 | ) 253 | 254 | if len(res["results"]["bindings"]) > 1: 255 | log.info( 256 | "Found more than one wikidata id for geonames id: " 257 | + str(geonames_id) 258 | ) 259 | return os.path.basename(res["results"]["bindings"][0]["id"]["value"]) 260 | except Exception: 261 | log.debug("Failed to map geonames id to wikidata id.", exc_info=True) 262 | return None 263 | 264 | 265 | def map_wikidata_to_osm(wikidata_id: str) -> str: 266 | wikidata_id = wikidata_id.upper() 267 | try: 268 | wikidata_result = get_entity_dict_from_api(wikidata_id) 269 | if len(wikidata_result["claims"]["P402"]) > 1: 270 | log.info("Found more than one osm id for wikidata id: " + wikidata_id) 271 | return ( 272 | OSM_ID_PREFIX 273 | + "R" 274 | + wikidata_result["claims"]["P402"][0]["mainsnak"]["datavalue"]["value"] 275 | ) 276 | except Exception: 277 | log.debug("Failed to map wikidata id to osm id.", exc_info=True) 278 | return None 279 | 280 | 281 | def map_wikidata_to_geonames(wikidata_id: str) -> str: 282 | wikidata_id = wikidata_id.upper() 283 | try: 284 | wikidata_result = get_entity_dict_from_api(wikidata_id) 285 | if len(wikidata_result["claims"]["P1566"]) > 1: 286 | log.info("Found more than one geonames id for wikidata id: " + wikidata_id) 287 | return ( 288 | GEONAMES_ID_PREFIX 289 | + wikidata_result["claims"]["P1566"][0]["mainsnak"]["datavalue"]["value"] 290 | ) 291 | except Exception: 292 | log.debug("Failed to map wikidata id to geonames id.", exc_info=True) 293 | return None 294 | 295 | 296 | def search_osm(query: str, limit: int = 5, country_codes: List[str] = None) -> str: 297 | try: 298 | country_code_filter = "" 299 | if country_codes: 300 | country_code_filter = "&countrycodes=" + ",".join(country_codes) 301 | 302 | request_url = ( 303 | OSM_NOMATIM_ENDPOINT 304 | + "/search?q={query}&limit={limit}&format=json" 305 | + country_code_filter 306 | ) 307 | response = requests.get(request_url.format(query=query, limit=limit)) 308 | results = [] 309 | for place in response.json(): 310 | if ( 311 | "osm_type" in place 312 | and place["osm_type"] 313 | and "osm_id" in place 314 | and place["osm_id"] 315 | ): 316 | name = place["display_name"] 317 | results.append( 318 | ( 319 | name, 320 | OSM_ID_PREFIX 321 | + OSM_TYPE_MAPPING[place["osm_type"]] 322 | + str(place["osm_id"]), 323 | ) 324 | ) 325 | return results 326 | except Exception: 327 | log.info("Failed to execute search for: " + query, exc_info=True) 328 | return [] 329 | 330 | 331 | def search_geonames(query: str, limit: int = 5, country_codes: List[str] = None) -> str: 332 | try: 333 | country_code_filter = "" 334 | if country_codes: 335 | for country_code in country_codes: 336 | country_code_filter += "&country=" + country_code 337 | 338 | request_url = ( 339 | GEONAMES_ENDPOINT 340 | + "/searchJSON?q={query}&maxRows={max_rows}&username={username}&orderby=relevance&featureClass=P&featureClass=A" 341 | + country_code_filter 342 | ) 343 | response = requests.get( 344 | request_url.format( 345 | query=query, max_rows=limit, username=random.choice(GEONAMES_USERS) 346 | ) 347 | ) 348 | results = [] 349 | for place in response.json()["geonames"]: 350 | name = place["toponymName"] 351 | results.append((GEONAMES_ID_PREFIX + str(place["geonameId"]), name)) 352 | return results 353 | except Exception: 354 | log.info("Failed to execute search for: " + query, exc_info=True) 355 | return [] 356 | --------------------------------------------------------------------------------