├── .env ├── .gitignore ├── Dockerfile ├── README.md ├── docker-compose.yaml ├── requirements.txt └── teslamate_fix_addrs.py /.env: -------------------------------------------------------------------------------- 1 | DB_USER=teslamate 2 | DB_PASSWD=123456 3 | DB_HOST=database 4 | DB_PORT=5432 5 | DB_NAME=teslamate 6 | BATCH=10 7 | HTTP_TIMEOUT=5 8 | HTTP_RETRY=5 9 | INTERVAL=5 10 | MODE=0 11 | KEY= 12 | USER_AGENT= -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | .vscode 2 | -------------------------------------------------------------------------------- /Dockerfile: -------------------------------------------------------------------------------- 1 | FROM ubuntu:22.04 2 | 3 | RUN mkdir /root/app 4 | 5 | RUN apt update && apt install -y python3 python3-pip 6 | 7 | COPY * /root/app 8 | 9 | RUN pip install -r /root/app/requirements.txt 10 | 11 | ENTRYPOINT ["python3", "/root/app/teslamate_fix_addrs.py"] 12 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # teslamate fix addrs 2 | 3 | Fix empty addresses in teslamate. 4 | 5 | **Thanks [@WayneJz](https://github.com/WayneJz) for the inspiration. See [teslamate-addr-fix](https://github.com/WayneJz/teslamate-addr-fix), address fixer written by go.** 6 | 7 | 8 | 9 | 10 | ## Notice 11 | 12 | **Must create a [backup](https://docs.teslamate.org/docs/maintenance/backup_restore) before doing this.** 13 | 14 | 15 | 16 | 17 | ## Pre-requisite 18 | 19 | - You have teslamate [broken address issue](https://github.com/adriankumpf/teslamate/issues/2956) 20 | 21 | - You have access to openstreetmap.org **via your HTTP proxy** 22 | 23 | 24 | ## Guides 25 | ### How it works 26 | 27 | Teslamate fix addrs has two main functions: 28 | 29 | **Fix empty addresses** 30 | 31 | Fix empty addressed by [open street map nominatim api](https://nominatim.openstreetmap.org/ui/reverse.html). All drives or charging processes will record a position with latitude and longitude infomations, use open street map api to resolve addresses. 32 | 33 | teslamate_fix_addrs will search all drives or charging processes, get records which address id are not set, add these addresses and link these records and addresses. 34 | 35 | 36 | 37 | **Update address details** 38 | 39 | Update address by [amap api](https://lbs.amap.com/api/webservice/summary). Some of address is not correct resoved by open street map, use amap to get addresses is much better (and faster) in China, update address if amap has more details. 40 | 41 | when the program run in first round, all addresses with comma (which means this address is added by open street map) in display_name column will be updated. In subsequent rounds, it will only check new added records by compare updated_at column. 42 | 43 | 44 | 45 | ### Run Mode 46 | 47 | `-m` `--mode` or environment `MODE` is used to configure teslamate_fix_addrs' running mode. 48 | 49 | * 0: fix empty address only. 50 | * 1: use amap to update address only. 51 | * 2: do both. 52 | 53 | If you what to update addresses by amap, remember to [apply a key](https://lbs.amap.com/api/webservice/guide/create-project/get-key) first, and pass the key value by `-k` `--key` or environment `KEY` 54 | 55 | 56 | 57 | ### Infinity mode 58 | 59 | `-i` `--interval` or environment `INTERVAL` is used to configure execution intervals. if `INTERVAL` equals 0, this program only run once, otherwise it will continuously run at interval seconds. 60 | 61 | 62 | 63 | ### Low memory support 64 | 65 | User `-b` `--batch` or environment `BATCH` to limit the number of records for one loop which can save memory use. 66 | 67 | All added or modified records will be commited at the end of each loop. 68 | 69 | 70 | 71 | ### Parameter priority 72 | 73 | All parameters can be passed to teslamate_fix_addrs by command line parameters or set environment values, the parameter priority is: 74 | 75 | >command line parameter > environment values > default values 76 | 77 | 78 | 79 | ### Proxy 80 | 81 | If open street map is baned in your region, you need a proxy to get access to it. Set proxy settings by environment values: 82 | 83 | * HTTP_PROXY=http://proxy.ip:port 84 | * HTTPS_PROXY=http://proxy.ip:port 85 | 86 | If you use the socks protocol proxy, set environment variable: 87 | * HTTP_PROXY=socks5://proxy.ip:port 88 | * HTTPS_PROXY=socks5://proxy.ip:port 89 | 90 | 91 | 92 | ### Get access to DB 93 | 94 | If you installed teslamate by docker, you can choose alternative solutions. 95 | 96 | 1. Expose DB port by add `ports` in your docker-compose.yaml 97 | 98 | ``` 99 | services: 100 | database: 101 | image: postgres:15 102 | restart: always 103 | environment: 104 | - POSTGRES_USER=teslamate 105 | - POSTGRES_PASSWORD=123456 106 | - POSTGRES_DB=teslamate 107 | ports: 108 | - 5432:5432 109 | ``` 110 | 111 | 112 | 113 | 2. Add teslamate_fix_addrs in your docker-compose.yaml, so they are in the same network. 114 | 115 | ``` 116 | services: 117 | database: 118 | image: postgres:15 119 | restart: always 120 | environment: 121 | - POSTGRES_USER=teslamate 122 | - POSTGRES_PASSWORD=123456 123 | - POSTGRES_DB=teslamate 124 | 125 | teslamate_fix_addrs: 126 | image: hipudding/teslamate_fix_addrs:latest 127 | container_name: teslmate_fix_addrs 128 | restart: unless-stopped 129 | environment: 130 | - DB_USER=teslamate 131 | - DB_PASSWD=123456 132 | - DB_HOST=database 133 | - DB_PORT=5432 134 | - DB_NAME=teslamate 135 | - BATCH=10 136 | - HTTP_TIMEOUT=5 137 | - HTTP_RETRY=5 138 | - INTERVAL=10 139 | - MODE=0 140 | - SINCE=2024-01-24 141 | - KEY= 142 | - USER_AGENT=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36 143 | ``` 144 | 145 | 146 | 147 | 148 | 149 | ### Usage examples 150 | 151 | **Docker compose** 152 | 153 | ``` 154 | teslamate_fix_addrs: 155 | image: hipudding/teslamate_fix_addrs:latest 156 | container_name: teslmate_fix_addrs 157 | restart: unless-stopped 158 | environment: 159 | - DB_USER=teslamate 160 | - DB_PASSWD=123456 161 | - DB_HOST=database 162 | - DB_PORT=5432 163 | - DB_NAME=teslamate 164 | - BATCH=10 165 | - HTTP_TIMEOUT=5 166 | - HTTP_RETRY=5 167 | - INTERVAL=5 168 | - MODE=0 169 | - SINCE=2024-01-24 170 | - KEY= 171 | - USER_AGENT=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36 172 | ``` 173 | 174 | 175 | 176 | **Run python script** 177 | 178 | ``` 179 | usage: teslamate_fix_addrs.py [-h] -u USER -p PASSWORD -H HOST -P PORT -d DBNAME [-b BATCH] [-t TIMEOUT] [-r RETRY] [-i INTERVAL] [-ua USER_AGENT] 180 | 181 | Usage of address fixer. 182 | 183 | options: 184 | -h, --help show this help message and exit 185 | -u USER, --user USER db user name(DB_USER). 186 | -p PASSWORD, --password PASSWORD db password(DB_PASSWD). 187 | -H HOST, --host HOST db host name or ip address(DB_HOST). 188 | -P PORT, --port PORT db port(DB_PORT). 189 | -d DBNAME, --dbname DBNAME db name(DB_NAME). 190 | -b BATCH, --batch BATCH batch size for one loop(BATCH). 191 | -t TIMEOUT, --timeout TIMEOUT http request timeout(s)(HTTP_TIMEOUT). 192 | -r RETRY, --retry RETRY http request max retries(HTTP_RETRY). 193 | -i INTERVAL, --interval INTERVAL if value not 0, run in infinity mode, fix record in every interval seconds(INTERVAL). 194 | -m MODE, --mode MODE run mode: 0 -> fix empty record; 1 -> update address by amap; 2 -> do both(MODE). 195 | -k KEY, --key KEY API key for calling amap(KEY). 196 | -s SINCE, --since SINCE Update from specified date(YYYY-mm-dd). 197 | -ua USER_AGENT, --user_agent USER_AGENT Custom User-Agent for HTTP requests(USER_AGENT). 198 | ``` 199 | 200 | 201 | 202 | ### Run in sandbox 203 | 204 | Worry about damaging existing data？ You can have a try in sandbox. 205 | 206 | 1. Prepare a different machine than the one where your teslamate is located, or different docker containers. 207 | 208 | 2. [Backup](https://docs.teslamate.org/docs/maintenance/backup_restore/) your data from teslamate database. 209 | 210 | 3. Launch a simple demo (sandbox) in the machine or container in step 1. 211 | 212 | ``` 213 | version: "3" 214 | 215 | services: 216 | database: 217 | image: postgres:15 218 | restart: always 219 | environment: 220 | - POSTGRES_USER=teslamate 221 | - POSTGRES_PASSWORD=123456 222 | - POSTGRES_DB=teslamate 223 | 224 | grafana: 225 | image: teslamate/grafana:latest 226 | restart: always 227 | environment: 228 | - DATABASE_USER=teslamate 229 | - DATABASE_PASS=123456 230 | - DATABASE_NAME=teslamate 231 | - DATABASE_HOST=database 232 | ports: 233 | - 3000:3000 234 | volumes: 235 | - teslamate-grafana-data:/var/lib/grafana 236 | 237 | teslamate_fix_addrs: 238 | image: hipudding/teslamate_fix_addrs:latest 239 | container_name: teslmate_fix_addrs 240 | restart: unless-stopped 241 | environment: 242 | - DB_USER=teslamate 243 | - DB_PASSWD=123456 244 | - DB_HOST=database 245 | - DB_PORT=5432 246 | - DB_NAME=teslamate 247 | - BATCH=10 248 | - HTTP_TIMEOUT=5 249 | - HTTP_RETRY=5 250 | - INTERVAL=10 251 | - MODE=0 252 | - SINCE=2024-01-24 253 | - KEY= 254 | - USER_AGENT=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36 255 | 256 | volumes: 257 | teslamate-grafana-data: 258 | ``` 259 | 260 | 4. [Restore](https://docs.teslamate.org/docs/maintenance/backup_restore/) your backup file to this demo. 261 | 5. Wait a moment and enjoy. (default username and password for grafana is admin/admin) 262 | 263 | 264 | 265 | ## Disclaimer 266 | 267 | Only use this program after properly created backups, I am **not** responsible for any data loss or software failure related to this. 268 | 269 | This project is only for study purpose, and **no web proxy (or its download link) provided**. If the network proxy is used in violation of local laws and regulations, the user is responsible for the consequences. 270 | 271 | When you download, copy, compile or execute the source code or binary program of this project, it means that you have accepted the disclaimer as mentioned. 272 | 273 | 274 | 275 | ## Contributing and Issue 276 | 277 | Welcome to contribute code or submit an issue. 278 | -------------------------------------------------------------------------------- /docker-compose.yaml: -------------------------------------------------------------------------------- 1 | services: 2 | teslamate_fix_addrs: 3 | image: huafengchun/teslamate_fix_addrs:latest 4 | container_name: teslmate_fix_addrs 5 | restart: unless-stopped 6 | environment: 7 | - DB_USER=teslamate 8 | - DB_PASSWD=123456 9 | - DB_HOST=database 10 | - DB_PORT=5432 11 | - DB_NAME=teslamate 12 | - BATCH=10 13 | - HTTP_TIMEOUT=5 14 | - HTTP_RETRY=5 15 | - INTERVAL=5 16 | - MODE=0 17 | - KEY= 18 | - USER_AGENT= 19 | -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | Requests 2 | SQLAlchemy 3 | psycopg2-binary 4 | pysocks 5 | -------------------------------------------------------------------------------- /teslamate_fix_addrs.py: -------------------------------------------------------------------------------- 1 | from sqlalchemy.ext.automap import automap_base 2 | from sqlalchemy.orm import Session 3 | from sqlalchemy import create_engine, or_, func 4 | from sqlalchemy.engine.url import URL 5 | import requests 6 | from requests.adapters import HTTPAdapter 7 | import json 8 | from datetime import datetime 9 | import logging 10 | import argparse 11 | import os 12 | import signal 13 | from threading import Timer 14 | import time 15 | 16 | logging.basicConfig( 17 | level=logging.INFO, 18 | format='%(asctime)s - %(levelname)s - %(message)s', 19 | datefmt='%Y-%m-%d %H:%M:%S' # 设置时间格式 20 | ) 21 | 22 | 23 | 24 | def handler(signum, frame): 25 | '''Contrl-C handler.''' 26 | logging.info("Ctrl-C pressed, exit.") 27 | os._exit(0) 28 | 29 | 30 | signal.signal(signal.SIGINT, handler) 31 | 32 | 33 | class EnvDefault(argparse.Action): 34 | '''args priority: cli args -> ENV -> default.''' 35 | 36 | def __init__(self, envvar, required=True, default=None, **kwargs): 37 | if envvar in os.environ: 38 | default = os.environ[envvar] 39 | if required and default: 40 | required = False 41 | super(EnvDefault, self).__init__(default=default, 42 | required=required, 43 | **kwargs) 44 | 45 | def __call__(self, parser, namespace, values, option_string=None): 46 | setattr(namespace, self.dest, values) 47 | 48 | 49 | parser = argparse.ArgumentParser(description='Usage of address fixer.') 50 | parser.add_argument("-u", 51 | "--user", 52 | required=True, 53 | type=str, 54 | action=EnvDefault, 55 | envvar="DB_USER", 56 | help="db user name(DB_USER).") 57 | parser.add_argument("-p", 58 | "--password", 59 | required=True, 60 | type=str, 61 | action=EnvDefault, 62 | envvar="DB_PASSWD", 63 | help="db password(DB_PASSWD).") 64 | parser.add_argument("-H", 65 | "--host", 66 | required=True, 67 | type=str, 68 | action=EnvDefault, 69 | envvar="DB_HOST", 70 | help="db host name or ip address(DB_HOST).") 71 | parser.add_argument("-P", 72 | "--port", 73 | required=True, 74 | type=str, 75 | action=EnvDefault, 76 | envvar="DB_PORT", 77 | help="db port(DB_PORT).") 78 | parser.add_argument("-d", 79 | "--dbname", 80 | required=True, 81 | type=str, 82 | action=EnvDefault, 83 | envvar="DB_NAME", 84 | help="db name(DB_NAME).") 85 | parser.add_argument("-b", 86 | "--batch", 87 | required=False, 88 | type=int, 89 | default=10, 90 | action=EnvDefault, 91 | envvar="BATCH", 92 | help="batch size for one loop(BATCH).") 93 | parser.add_argument("-t", 94 | "--timeout", 95 | required=False, 96 | type=int, 97 | default=5, 98 | action=EnvDefault, 99 | envvar="HTTP_TIMEOUT", 100 | help="http request timeout(s)(HTTP_TIMEOUT).") 101 | parser.add_argument("-r", 102 | "--retry", 103 | required=False, 104 | type=int, 105 | default=5, 106 | action=EnvDefault, 107 | envvar="HTTP_RETRY", 108 | help="http request max retries(HTTP_RETRY).") 109 | parser.add_argument( 110 | "-i", 111 | "--interval", 112 | required=False, 113 | type=int, 114 | default=0, 115 | action=EnvDefault, 116 | envvar="INTERVAL", 117 | help= 118 | "if value not 0, run in infinity mode, fix record in every interval seconds(INTERVAL)." 119 | ) 120 | parser.add_argument( 121 | "-m", 122 | "--mode", 123 | required=False, 124 | type=int, 125 | default=0, 126 | action=EnvDefault, 127 | envvar="MODE", 128 | help= 129 | "run mode: 0 -> fix empty record; 1 -> update address by amap; 2 -> do both(MODE)." 130 | ) 131 | parser.add_argument("-k", 132 | "--key", 133 | required=False, 134 | type=str, 135 | default='', 136 | action=EnvDefault, 137 | envvar="KEY", 138 | help="API key for calling amap(KEY).") 139 | 140 | parser.add_argument("-s", 141 | "--since", 142 | required=False, 143 | type=lambda d: datetime.strptime(d, '%Y-%m-%d'), 144 | default=datetime.min, 145 | action=EnvDefault, 146 | envvar="SINCE", 147 | help="Update from specified date(YYYY-mm-dd).") 148 | parser.add_argument( 149 | "-ua", 150 | "--user_agent", 151 | required=False, 152 | type=str, 153 | default='teslamate/#v1.29.2', 154 | action=EnvDefault, 155 | envvar="USER_AGENT", 156 | help="Custom User-Agent for HTTP requests(USER_AGENT)." 157 | ) 158 | args = parser.parse_args() 159 | 160 | 161 | def custom_json_dumps(d): 162 | '''do not add backslash in json.''' 163 | return d 164 | 165 | conn_url = URL.create( 166 | drivername="postgresql", 167 | username=args.user, 168 | password=args.password, 169 | host=args.host, 170 | port=args.port, 171 | database=args.dbname 172 | ) 173 | 174 | engine = create_engine(conn_url, json_serializer=custom_json_dumps, echo=False) 175 | 176 | # open street map api. 177 | osm_resolve_url = "https://nominatim.openstreetmap.org/reverse?lat=%.6f&lon=%.6f&format=jsonv2&addressdetails=1&extratags=1&namedetails=1&zoom=18" 178 | 179 | # amap api. 180 | amap_coordinate_transformation_url = "https://restapi.amap.com/v3/assistant/coordinate/convert?key=%s&coordsys=gps&output=json&locations=%s,%s" 181 | amap_resolve_url = "https://restapi.amap.com/v3/geocode/regeo?key=%s&output=json&location=%s,%s&poitype=all&extensions=all" 182 | 183 | # last updated record id. 184 | last_update_id = 0 185 | 186 | # reflact Objects from db tables. 187 | Base = automap_base() 188 | Base.prepare(autoload_with=engine) 189 | Drives = Base.classes.drives 190 | ChargingProcesses = Base.classes.charging_processes 191 | Positions = Base.classes.positions 192 | Addresses = Base.classes.addresses 193 | 194 | # reference to teslamate's source code, get address value from multiple keys. 195 | house_number_aliases = ['house_number', 'street_number'] 196 | 197 | road_aliases = [ 198 | "road", "footway", "street", "street_name", "residential", "path", 199 | "pedestrian", "road_reference", "road_reference_intl", "square", "place" 200 | ] 201 | 202 | neighborhood_aliases = [ 203 | "neighbourhood", "suburb", "city_district", "district", "quarter", 204 | "borough", "city_block", "residential", "commercial", "houses", 205 | "subdistrict", "subdivision", "ward" 206 | ] 207 | 208 | municipality_aliases = [ 209 | "municipality", "local_administrative_area", "subcounty" 210 | ] 211 | 212 | village_aliases = ["village", "municipality", "hamlet", "locality", "croft"] 213 | 214 | city_aliases = ["city", "town", "township"] 215 | 216 | city_aliases.extend(village_aliases) 217 | city_aliases.extend(municipality_aliases) 218 | 219 | county_aliases = ["county", "county_code", "department"] 220 | 221 | state_aliases = ['state', 'province', 'state_code'] 222 | 223 | country_aliases = ['country', 'country_name'] 224 | 225 | 226 | def get_address_str(address, addr_keys): 227 | '''get address value from multiple keys.''' 228 | for addr_key in addr_keys: 229 | if addr_key in address: 230 | return address[addr_key] 231 | return None 232 | 233 | 234 | def get_address_name(address): 235 | ''' 236 | address names comes from multiple places. 237 | 1. address.name. 238 | 2. address.namedetails.name. 239 | 3. address.namedetails.alt_name. 240 | 4. first element in address.display_name. 241 | ''' 242 | name = '' 243 | if 'name' in address.keys() and len(address['name']): 244 | name = address['name'] 245 | if 'namedetails' in address.keys() and address['namedetails'] is not None: 246 | if 'name' in address['namedetails'].keys(): 247 | name = address['namedetails']['name'] 248 | if 'alt_name' in address['namedetails'].keys(): 249 | name = address['namedetails']['alt_name'] 250 | if len(name) == 0: 251 | name = address['display_name'].split(',')[0] 252 | return name 253 | 254 | 255 | def get_position(session, position_id): 256 | '''get position id from table positions by position_ids.''' 257 | position = session.query(Positions).filter( 258 | Positions.id == position_id).first() 259 | # position_id is foreign key to table positions. position will never be None. 260 | if position == None: 261 | # fatal error, exit now. 262 | logging.fatal("Position with ID %s is not found." % position_id) 263 | assert (False) 264 | return position 265 | 266 | 267 | def http_request(url): 268 | '''get response by calling map api.''' 269 | http_session = requests.Session() 270 | http_session.mount('http://', HTTPAdapter(max_retries=args.retry)) 271 | http_session.mount('https://', HTTPAdapter(max_retries=args.retry)) 272 | headers = { 273 | 'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7', 274 | 'accept-language': 'zh-CN,zh;q=0.9,en-US;q=0.8,en;q=0.7', 275 | 'cache-control': 'max-age=0', 276 | 'dnt': '1', 277 | 'priority': 'u=0, i', 278 | 'sec-ch-ua': '"Chromium";v="128", "Not;A=Brand";v="24", "Google Chrome";v="128"', 279 | 'sec-ch-ua-mobile': '?0', 280 | 'sec-ch-ua-platform': '"macOS"', 281 | 'sec-fetch-dest': 'document', 282 | 'sec-fetch-mode': 'navigate', 283 | 'sec-fetch-site': 'none', 284 | 'sec-fetch-user': '?1', 285 | 'upgrade-insecure-requests': '1', 286 | 'User-Agent': args.user_agent 287 | } 288 | 289 | try: 290 | response = http_session.get(url=url, timeout=args.timeout, headers=headers) 291 | if response.status_code != requests.codes.ok: 292 | logging.error( 293 | "Http request failed by url: %s, code: %d, body: %s" % 294 | (url, response.status_code, response.text)) 295 | return None 296 | raw = response.text 297 | return raw 298 | except: 299 | logging.error("Http request exception by url: %s" % (url)) 300 | return None 301 | 302 | 303 | def get_address_in_db(session, osm_id): 304 | '''select address from db, get address id which just added.''' 305 | return session.query(Addresses).filter(Addresses.osm_id == osm_id).first() 306 | 307 | 308 | def add_osm_address(session, osm_address, raw): 309 | '''add osm address to db.''' 310 | exist_address = get_address_in_db(session, osm_address['osm_id']) 311 | if exist_address is None: 312 | logging.info("oms id = %d is not exist!" % osm_address['osm_id']) 313 | if exist_address is None: 314 | address = Addresses( 315 | display_name=osm_address['display_name'], 316 | latitude=osm_address['lat'], 317 | longitude=osm_address['lon'], 318 | name=get_address_name(osm_address), 319 | house_number=get_address_str(osm_address['address'], 320 | house_number_aliases), 321 | road=get_address_str(osm_address['address'], road_aliases), 322 | neighbourhood=get_address_str(osm_address['address'], 323 | neighborhood_aliases), 324 | city=get_address_str(osm_address['address'], city_aliases), 325 | county=get_address_str(osm_address['address'], county_aliases), 326 | postcode=get_address_str(osm_address['address'], ['postcode']), 327 | state=get_address_str(osm_address['address'], state_aliases), 328 | state_district=get_address_str(osm_address['address'], 329 | ['state_district']), 330 | country=get_address_str(osm_address['address'], country_aliases), 331 | raw=raw, 332 | inserted_at=datetime.now().replace(microsecond=0), 333 | updated_at=datetime.now().replace(microsecond=0), 334 | osm_id=osm_address['osm_id'], 335 | osm_type=osm_address['osm_type']) 336 | session.add(address) 337 | logging.info("address added: %s." % osm_address['display_name']) 338 | else: 339 | logging.info("address is already exist: %d, %s." % 340 | (osm_address['osm_id'], osm_address['display_name'])) 341 | 342 | 343 | def get_address(session, position): 344 | ''' 345 | return address id and display_name by position id. 346 | Address will add into db if not exists. 347 | ''' 348 | url = osm_resolve_url % (position.latitude, position.longitude) 349 | raw = http_request(url) 350 | if raw is None: 351 | return None, None 352 | 353 | osm_address = json.loads(raw) 354 | if osm_address == None: 355 | return None, None 356 | 357 | add_osm_address(session, osm_address, raw) 358 | added_address = get_address_in_db(session, osm_address['osm_id']) 359 | return added_address.id, added_address.display_name 360 | 361 | 362 | def fix_address(session, batch_size, empty_count): 363 | processed_count = 0 364 | # get empty records in drives. 365 | empty_drive_addresses = session\ 366 | .query(Drives)\ 367 | .filter(or_(Drives.start_address_id.is_(None), Drives.end_address_id.is_(None)))\ 368 | .filter(Drives.start_position_id.is_not(None))\ 369 | .filter(Drives.end_position_id.is_not(None))\ 370 | .limit(batch_size)\ 371 | .all() 372 | 373 | # get empty records in charging_processes, all records are LE batch_size. 374 | empty_charging_addresses = [] 375 | if len(empty_drive_addresses) < batch_size: 376 | empty_charging_addresses = session\ 377 | .query(ChargingProcesses)\ 378 | .filter(ChargingProcesses.address_id.is_(None))\ 379 | .filter(ChargingProcesses.position_id.is_not(None))\ 380 | .limit(batch_size - len(empty_drive_addresses))\ 381 | .all() 382 | 383 | # processing drives. 384 | for empty_drive_address in empty_drive_addresses: 385 | logging.info("processing drive address (%d left)" % (empty_count - processed_count)) 386 | 387 | # get positions. 388 | start_position_id = empty_drive_address.start_position_id 389 | end_position_id = empty_drive_address.end_position_id 390 | start_position = get_position(session, start_position_id) 391 | end_position = get_position(session, end_position_id) 392 | 393 | # get addresses. 394 | start_address_id, start_address = get_address(session, start_position) 395 | end_address_id, end_address = get_address(session, end_position) 396 | if start_address_id is None or end_address_id is None: 397 | continue 398 | 399 | # update address ids. 400 | empty_drive_address.start_address_id = start_address_id 401 | empty_drive_address.end_address_id = end_address_id 402 | logging.info("Changing drives(id = %d) start address to %s" % 403 | (empty_drive_address.id, start_address)) 404 | logging.info("Changing drives(id = %d) end address to %s" % 405 | (empty_drive_address.id, end_address)) 406 | processed_count += 1 407 | 408 | # processing charging. 409 | for empty_charging_address in empty_charging_addresses: 410 | logging.info("processing charging address (%d left)" % (empty_count - processed_count)) 411 | 412 | # get position. 413 | position_id = empty_charging_address.position_id 414 | position = get_position(session, position_id) 415 | 416 | # get address. 417 | address_id, address = get_address(session, position) 418 | if address_id is None: 419 | continue 420 | 421 | # update address id. 422 | empty_charging_address.address_id = address_id 423 | logging.info("Changing charging(id = %d) to %s" % 424 | (empty_charging_address.id, address)) 425 | processed_count += 1 426 | 427 | # records processed. 428 | return processed_count 429 | 430 | 431 | def get_empty_record_count(session): 432 | '''get all empty records count.''' 433 | empty_count = session\ 434 | .query(Drives.id)\ 435 | .filter(or_(Drives.start_address_id.is_(None), Drives.end_address_id.is_(None)))\ 436 | .filter(Drives.start_position_id.is_not(None))\ 437 | .filter(Drives.end_position_id.is_not(None))\ 438 | .count() 439 | 440 | empty_count += session\ 441 | .query(ChargingProcesses.id)\ 442 | .filter(ChargingProcesses.address_id.is_(None))\ 443 | .filter(ChargingProcesses.position_id.is_not(None))\ 444 | .count() 445 | return empty_count 446 | 447 | 448 | def fix_empty_records(): 449 | # for low memory devices. 450 | while True: 451 | with Session(engine) as session: 452 | logging.info("checking empty records...") 453 | empty_count = get_empty_record_count(session) 454 | if fix_address(session, args.batch, empty_count) == 0: 455 | # all recoreds are fixed. 456 | break 457 | else: 458 | # commit at end of each batch. 459 | logging.info("saving...") 460 | session.commit() 461 | 462 | 463 | def get_field(find, keys): 464 | '''get field from a dict object''' 465 | item = find 466 | for key in keys: 467 | if isinstance(key, str): 468 | if key in item: 469 | item = item[key] 470 | else: 471 | return '' 472 | elif isinstance(key, int): 473 | if len(item) > 0: 474 | item = item[key] 475 | else: 476 | return '' 477 | # we should have find address str here. 478 | if not isinstance(item, str): 479 | # some address will be empty list. 480 | if len(item) == 0: 481 | return '' 482 | logging.fatal("key error when parse amap response.") 483 | assert (False) 484 | return item 485 | 486 | 487 | def update_address_in_db(need_update_address, address_details): 488 | # parse response. 489 | country = get_field(address_details, 490 | ['regeocode', 'addressComponent', 'country']) 491 | province = get_field(address_details, 492 | ['regeocode', 'addressComponent', 'province']) 493 | 494 | municipality = province in ['北京市', '天津市', '上海市', '重庆市'] 495 | if municipality: 496 | city = province + get_field( 497 | address_details, ['regeocode', 'addressComponent', 'district']) 498 | else: 499 | city = get_field(address_details, 500 | ['regeocode', 'addressComponent', 'city']) 501 | 502 | township = get_field(address_details, 503 | ['regeocode', 'addressComponent', 'township']) 504 | display_name = get_field(address_details, 505 | ['regeocode', 'formatted_address']) 506 | neighborhood = get_field( 507 | address_details, 508 | ['regeocode', 'addressComponent', 'neighborhood', 'name']) 509 | street_number = get_field( 510 | address_details, 511 | ['regeocode', 'addressComponent', 'streetNumber', 'number']) 512 | road = get_field(address_details, ['regeocode', 'roads', 0, 'name']) 513 | name = get_field(address_details, ['regeocode', 'aois', 0, 'name']) 514 | if len(name) == 0: 515 | name = get_field(address_details, ['regeocode', 'pois', 0, 'name']) 516 | if len(name) == 0: 517 | name = get_field(address_details, ['regeocode', 'roads', 0, 'name']) 518 | 519 | # update db record. 520 | logging.info("update address from %s to %s" % 521 | (need_update_address.display_name, display_name)) 522 | need_update_address.state = province 523 | need_update_address.county = township 524 | need_update_address.city = city 525 | need_update_address.house_number = street_number 526 | need_update_address.display_name = display_name 527 | need_update_address.country = country 528 | need_update_address.updated_at = datetime.now().replace(microsecond=0) 529 | 530 | # record last processed record id, skip records which id less than this. 531 | # assume that address record will not updated. 532 | # if language changed, teslamate will update all addressed, remember to 533 | # restart me to re-process all records. 534 | global last_update_id 535 | last_update_id = need_update_address.id 536 | 537 | # if some address is empty, do not update them. 538 | if len(road) > 0: 539 | need_update_address.road = road 540 | 541 | if len(name) > 0: 542 | need_update_address.name = name 543 | 544 | if len(neighborhood) > 0: 545 | need_update_address.neighbourhood = neighborhood 546 | 547 | 548 | def request_amap_api(url): 549 | '''request from amap api and loads as dict''' 550 | response = http_request(url) 551 | if response is None: 552 | return None 553 | else: 554 | # amap limits access frequency 555 | time.sleep(0.3) 556 | 557 | response_dict = json.loads(response) 558 | if response_dict is None or response_dict['status'] != '1': 559 | logging.error("request amap api error: %s" % response) 560 | return None 561 | return response_dict 562 | 563 | 564 | def get_update_record_count(session): 565 | # record last updated id to save cpu time. 566 | return session\ 567 | .query(Addresses)\ 568 | .filter(Addresses.updated_at >= args.since)\ 569 | .filter(Addresses.id > last_update_id)\ 570 | .count() 571 | 572 | 573 | def get_need_update_addresses(session, batch_size): 574 | # record last update id to save cpu time. 575 | return session\ 576 | .query(Addresses)\ 577 | .filter(Addresses.updated_at >= args.since)\ 578 | .filter(Addresses.id > last_update_id)\ 579 | .order_by(Addresses.id)\ 580 | .limit(batch_size)\ 581 | .all() 582 | 583 | 584 | def update_address(session, batch_size, need_update_count): 585 | '''update address str by amap api.''' 586 | processed_count = 0 587 | if len(args.key) == 0: 588 | logging.error("Amap key is not set.") 589 | return 0 590 | 591 | need_update_addresses = get_need_update_addresses(session, batch_size) 592 | 593 | for need_update_address in need_update_addresses: 594 | logging.info("processing update address (%d left)" % 595 | (need_update_count - processed_count)) 596 | gps_lat = need_update_address.latitude 597 | gps_lon = need_update_address.longitude 598 | 599 | # transform coordinate 600 | url = amap_coordinate_transformation_url % (args.key, gps_lon, gps_lat) 601 | transformed_coordinate = request_amap_api(url) 602 | if transformed_coordinate is None: 603 | continue 604 | 605 | locations = transformed_coordinate['locations'] 606 | amap_lon = round(float(locations.split(',')[0]), 6) 607 | amap_lat = round(float(locations.split(',')[1]), 6) 608 | 609 | # get address details 610 | url = amap_resolve_url % (args.key, amap_lon, amap_lat) 611 | address_details = request_amap_api(url) 612 | if address_details is None: 613 | continue 614 | 615 | # update db 616 | update_address_in_db(need_update_address, address_details) 617 | 618 | processed_count += 1 619 | 620 | return processed_count 621 | 622 | 623 | def update_address_by_amap(): 624 | while True: 625 | with Session(engine) as session: 626 | logging.info("updating address by amap...") 627 | need_update_count = get_update_record_count(session) 628 | if update_address(session, args.batch, need_update_count) == 0: 629 | # all recoreds are updated. 630 | break 631 | else: 632 | # commit at end of each batch. 633 | logging.info("saving...") 634 | session.commit() 635 | 636 | 637 | def main(): 638 | if args.mode == 0 or args.mode == 2: 639 | fix_empty_records() 640 | if args.mode == 1 or args.mode == 2: 641 | update_address_by_amap() 642 | 643 | # wrong mode, do nothing and exit. 644 | if args.mode < 0 or args.mode > 2: 645 | logging.info("nothing to do, bye.") 646 | # if interval is set, run in infinity mode. 647 | elif args.interval != 0: 648 | loop_timer = Timer(args.interval, main) 649 | loop_timer.start() 650 | 651 | 652 | if __name__ == '__main__': 653 | main() 654 | --------------------------------------------------------------------------------