├── berlin.webm ├── requirements.txt ├── locations_in_berlin_all_time_weighted.png ├── LICENSE ├── README.md ├── location_to_geojson.py └── process.py /berlin.webm: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jacopofar/location-data-to-heatmap/HEAD/berlin.webm -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | numpy==1.22.3 2 | scipy==1.8.0 3 | matplotlib==3.5.1 4 | scikit-image==0.19.2 5 | moviepy==1.0.3 6 | -------------------------------------------------------------------------------- /locations_in_berlin_all_time_weighted.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jacopofar/location-data-to-heatmap/HEAD/locations_in_berlin_all_time_weighted.png -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2019 Jacopo Farina 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Location to heatmap 2 | 3 | # Archived 4 | This project is archived, Google Takeout now has a better format and there are better ways to generate this representation. 5 | 6 | Have a look at `location_to_geojson.py` (no dependencies needed) to generate GeoJSON files from your activities that can be loaded in QGIS or [geojson.io](https://geojson.io/#map=2/20.0/0.0) and many other tools. 7 | 8 | To generate raster representations like this I strongly recommend [Datashader](https://datashader.org/), it's powerful, easy to use and fast. 9 | 10 | This repository is now kept only for reference. 11 | ## Old readme for reference 12 | 13 | This tool creates static and animated heatmaps of a given zone based on the data from Google location service. 14 | 15 | It can be useful to find parts of a city that you never visited and visualize your movement and habits. 16 | 17 | ![Heatmap of locations in Berlin](locations_in_berlin_all_time_weighted.png) 18 | 19 | Click here to see the ![animated version](berlin.webm) 20 | 21 | 22 | ## Usage 23 | You need to have the Google location service active for some time to collect the data. Use [Google Takeout](https://takeout.google.com/) to export the location history. 24 | 25 | Then, choose a city or region you are interested in and produce a background map, for example taking a screenshot of Open Street Map. 26 | 27 | Then, create a virtualenv, install the dependencies and run the script like this (example coordinates for Berlin, Germany): 28 | 29 | python3 -m venv .venv 30 | python3 -m pip install -r requirements.txt 31 | python3 process.py region_name 132700000 135500000 524300000 526000000 1500 /path/to/location/history/export.json /path/to/background/map/image.png 32 | 33 | You can use `python3 process.py --help` to get a description, but in short the numbers you see are the decimal coordinates multiplied by 10^7, the zoom level (1000 = 1 pixel per 10 meters). 34 | 35 | It will create a global heatmap and one for every 15 minutes span after midnight. These images are then merged in an animated GIF and a webm video using the amazing [MoviePy](http://zulko.github.io/moviepy/) library. 36 | 37 | ## License 38 | MIT licensed, use as you wish. 39 | 40 | Made with ❤️ with Python and open source libraries. 41 | -------------------------------------------------------------------------------- /location_to_geojson.py: -------------------------------------------------------------------------------- 1 | from dataclasses import dataclass 2 | import json 3 | from pathlib import Path 4 | from sys import argv 5 | 6 | INTERMEDIATE_POINTS = 3 7 | 8 | 9 | @dataclass 10 | class Point: 11 | lat: float 12 | lng: float 13 | accuracyMeters: int | None = None 14 | 15 | def __hash__(self) -> int: 16 | return hash(self.lat) ^ hash(self.lng) ^ hash(self.accuracyMeters) 17 | 18 | 19 | @dataclass 20 | class Activity: 21 | type: str 22 | points: list[Point] 23 | 24 | 25 | def read_file(fname) -> dict[str, list[Activity]]: 26 | ret = {} 27 | with open(fname) as f: 28 | raw_data = json.load(f) 29 | for event in raw_data["timelineObjects"]: 30 | if "activitySegment" not in event: 31 | continue 32 | if "activityType" not in event["activitySegment"]: 33 | # old files usually 34 | print(f"Ignoring a segment without an activity type") 35 | continue 36 | activity_type = event["activitySegment"]["activityType"] 37 | if "simplifiedRawPath" in event["activitySegment"]: 38 | waypoints_raw = event["activitySegment"]["simplifiedRawPath"]["points"] 39 | elif "waypointPath" in event["activitySegment"]: 40 | waypoints_raw = event["activitySegment"]["waypointPath"]["waypoints"] 41 | else: 42 | print(f"No waypoints found for {activity_type}, skipping...") 43 | continue 44 | points: list[Point] = [] 45 | for wr in waypoints_raw: 46 | if "latE7" in wr: 47 | points.append( 48 | Point( 49 | wr["latE7"] / 1e7, 50 | wr["lngE7"] / 1e7, 51 | accuracyMeters=wr.get("accuracyMeters"), 52 | ) 53 | ) 54 | elif "latitudeE7" in wr: 55 | points.append( 56 | Point( 57 | wr["latitudeE7"] / 1e7, 58 | wr["longitudeE7"] / 1e7, 59 | accuracyMeters=wr.get("accuracyMeters"), 60 | ) 61 | ) 62 | else: 63 | raise Exception(f"Unknown waypoint format: {wr}") 64 | print(f"Activity {activity_type} had {len(points)} waypoints") 65 | 66 | if activity_type not in ret: 67 | ret[activity_type] = [] 68 | 69 | ret[activity_type].append(Activity(activity_type, points)) 70 | return ret 71 | 72 | 73 | def activity_grid( 74 | activities: dict[str, list[Activity]], rounding: int 75 | ) -> dict[str, list[Point]]: 76 | """Aggregates waypoints to the given rounding. 77 | 78 | Rounding is here the number of decimals after the decimal separator. 79 | So rounding = 3 means that the first 3 decimals are used 80 | """ 81 | ret: dict[str, dict[Point, int]] = {} 82 | for activity_type, activities in activities.items(): 83 | ret[activity_type] = {} 84 | for activity in activities: 85 | # assume only some activities actually let you explore the world 86 | if activity_type not in ("WALKING", "CYCLING", "RUNNING"): 87 | for point in activity.points: 88 | rounded_point = Point( 89 | round(point.lat * (10**rounding)) / (10**rounding), 90 | round(point.lng * (10**rounding)) / (10**rounding), 91 | ) 92 | if rounded_point not in ret[activity_type]: 93 | ret[activity_type][rounded_point] = 1 94 | ret[activity_type][rounded_point] += 1 95 | else: 96 | visited_points = set() 97 | for p1, p2 in zip(activity.points, activity.points[1:]): 98 | # brutal interpolation 99 | for s in range(INTERMEDIATE_POINTS): 100 | lat_i = p1.lat + (p2.lat - p1.lat) * s / INTERMEDIATE_POINTS 101 | lng_i = p1.lng + (p2.lng - p1.lng) * s / INTERMEDIATE_POINTS 102 | rounded_point = Point( 103 | round(lat_i * (10**rounding)) / (10**rounding), 104 | round(lng_i * (10**rounding)) / (10**rounding), 105 | ) 106 | visited_points.add(rounded_point) 107 | for rounded_point in visited_points: 108 | if rounded_point not in ret[activity_type]: 109 | ret[activity_type][rounded_point] = 1 110 | ret[activity_type][rounded_point] += 1 111 | 112 | return ret 113 | 114 | 115 | if __name__ == "__main__": 116 | if len(argv) == 1: 117 | print("Usage: python3 location_to_geojson.py /path/to/google/takeout/Semantic Location History") 118 | exit(1) 119 | PRECISION = 3 120 | total_grid = {"ALL": {}} 121 | for fname in Path(argv[1]).glob("**/*.json"): 122 | print(f"Processing {fname}") 123 | data = read_file(fname) 124 | grid = activity_grid(data, PRECISION) 125 | for activity_type, points in grid.items(): 126 | if activity_type not in total_grid: 127 | total_grid[activity_type] = {} 128 | for point, count in points.items(): 129 | if point not in total_grid[activity_type]: 130 | total_grid[activity_type][point] = count 131 | total_grid[activity_type][point] += count 132 | 133 | if point not in total_grid["ALL"]: 134 | total_grid["ALL"][point] = count 135 | total_grid["ALL"][point] += count 136 | 137 | for activity_type, point_count in total_grid.items(): 138 | features = [] 139 | for tile, count in point_count.items(): 140 | coords = [ 141 | [tile.lng, tile.lat], 142 | [tile.lng, tile.lat + 1 / 10**PRECISION], 143 | [tile.lng + 1 / 10**PRECISION, tile.lat + 1 / 10**PRECISION], 144 | [tile.lng + 1 / 10**PRECISION, tile.lat], 145 | [tile.lng, tile.lat], 146 | [tile.lng + 1 / 10**PRECISION, tile.lat + 1 / 10**PRECISION], 147 | ] 148 | feature = { 149 | "type": "Feature", 150 | "properties": {"type": activity_type, "count": count}, 151 | "geometry": {"type": "LineString", "coordinates": coords}, 152 | } 153 | features.append(feature) 154 | 155 | geojson = {"type": "FeatureCollection", "features": features} 156 | with open(f"history_{activity_type}.geojson", "w") as fw: 157 | json.dump(geojson, fw, indent=2) 158 | -------------------------------------------------------------------------------- /process.py: -------------------------------------------------------------------------------- 1 | import json 2 | from datetime import datetime 3 | import argparse 4 | 5 | import imageio 6 | import numpy as np 7 | from scipy import ndimage 8 | from matplotlib import pyplot as plt 9 | import matplotlib.image as img 10 | from skimage.transform import resize 11 | from moviepy.video.io.ImageSequenceClip import ImageSequenceClip 12 | 13 | 14 | def iso8601_to_epoch(iso_date: str): 15 | return datetime.fromisoformat(iso_date[:19]).strftime('%s') 16 | 17 | def get_locations( 18 | location_data, x0, x1, y0, y1, scaling_factor, 19 | minutes_since_last_midnight_filter=None, 20 | ): 21 | """Produce an heatmap matrix of the given bounding box and scaling. 22 | 23 | Coordinates are in E7 format (decimal degrees multiplied by 10^7, 24 | and rounded to be integers). 25 | 26 | Optionally a range of minutes after midnight can be given. 27 | 28 | Parameters 29 | ---------- 30 | location_data : dict 31 | the 'location' key of from Google location data export 32 | x0 : int 33 | longitude min, in E7 format 34 | x1 : int 35 | longitude max, in E7 format 36 | y0 : int 37 | latitude min, in E7 format 38 | y1 : int 39 | latitude max, in E7 format 40 | scaling_factor : int 41 | scaling factor, the higher the bigger the matrix 42 | 1000 means about 1 cell per 10 meters 43 | 1 px = 10 meters = ~0.00009 lat/long degrees 44 | minutes_since_last_midnight_filter : Tuple[int, int], optional 45 | the number of minutes, if specified will consider only the points 46 | with a timestamp that is N minutes after UTC midnight, where N is 47 | between the two given values 48 | 49 | Returns 50 | ------- 51 | Tuple[ndarray, int, int] 52 | The resulting heatmap, and the number of processed and skipped entries 53 | """ 54 | 55 | height_in_pixels = int((y1 - y0) / scaling_factor) 56 | width_in_pixels = int((x1 - x0) / scaling_factor) 57 | map_size = (height_in_pixels, width_in_pixels) 58 | 59 | place_map = np.zeros(map_size) 60 | 61 | processed, skipped = 0, 0 62 | for index_location, loc in enumerate(location_data): 63 | processed += 1 64 | if minutes_since_last_midnight_filter is not None: 65 | dt = datetime.fromtimestamp(int(iso8601_to_epoch(loc['timestamp']))/1000) 66 | sample_minutes = dt.hour * 60 + dt.minute 67 | if (sample_minutes < minutes_since_last_midnight_filter[0] or 68 | sample_minutes > minutes_since_last_midnight_filter[1]): 69 | skipped += 1 70 | continue 71 | x = round((int(loc['longitudeE7'] - x0)) / scaling_factor) 72 | y = round((int(loc['latitudeE7'] - y0)) / scaling_factor) 73 | if (x >= place_map.shape[1] or 74 | y >= place_map.shape[0] or x < 0 or y < 0): 75 | skipped += 1 76 | else: 77 | if index_location + 1 < len(location_data): 78 | place_map[y][x] += ( 79 | (int(iso8601_to_epoch(loc['timestamp'])) - 80 | int(iso8601_to_epoch(location_data[index_location + 1]['timestamp']))) 81 | / (1000 * 60)) 82 | else: 83 | place_map[y][x] += 1 84 | print('dots processed:', processed, 'dots outside the rectangle:', skipped) 85 | return place_map, processed, skipped 86 | 87 | 88 | def main( 89 | input_file: str, 90 | base_file: str, 91 | place_name: str, 92 | x0: int, 93 | x1: int, 94 | y0: int, 95 | y1: int, 96 | scaling_factor: int, 97 | ): 98 | print('Reading location data JSON...') 99 | location_data = json.loads(open(input_file).read())['locations'] 100 | print('Data imported. Processing...') 101 | 102 | bins = list(range(1, 100, 1)) 103 | minutes_step = 15 104 | # weight of previous frames over new one. The inverse of the decay factor 105 | frame_persistence_factor = 4 106 | 107 | all_minutes_starts = list(range(0, 24*60, minutes_step)) 108 | base_map = np.mean(img.imread(base_file), axis=-1) 109 | base_map = np.stack([base_map, base_map, base_map], axis=-1) 110 | moving_average_frame = None 111 | quintiles = None 112 | filenames = [] 113 | fig = None 114 | for frame_idx, selected_minute in enumerate( 115 | [None] + all_minutes_starts): 116 | print(f'frame {frame_idx} of {len(all_minutes_starts)}') 117 | place_map, processed, skipped = get_locations( 118 | location_data, 119 | x0, 120 | x1, 121 | y0, 122 | y1, 123 | scaling_factor, 124 | minutes_since_last_midnight_filter=(( 125 | selected_minute, selected_minute + minutes_step) 126 | if selected_minute is not None else None)) 127 | 128 | place_map_draw = None 129 | 130 | if processed == skipped: 131 | print('no points for this map, generating an empty one') 132 | place_map_draw = place_map 133 | else: 134 | place_map_blurred = ndimage.filters.gaussian_filter( 135 | place_map, 1) 136 | flattened = place_map_blurred.flatten() 137 | if selected_minute is None: 138 | # the first iteration is over non-time filtered point 139 | # and is used to generate the bin once for all 140 | quintiles = np.percentile( 141 | flattened[np.nonzero(flattened)], bins) 142 | place_map_draw = np.searchsorted( 143 | quintiles, place_map_blurred) / len(bins) 144 | 145 | if base_map.shape != place_map_draw.shape: 146 | base_map = resize( 147 | base_map, place_map_draw.shape, anti_aliasing=True) 148 | 149 | if moving_average_frame is None: 150 | moving_average_frame = place_map_draw 151 | else: 152 | moving_average_frame = ( 153 | (moving_average_frame + 154 | place_map_draw * frame_persistence_factor) 155 | / (1 + frame_persistence_factor)) 156 | print('min/avg/max of original matrix:' 157 | f'{np.min(place_map_draw,axis=(0,1))}/' 158 | f'{np.average(place_map_draw,axis=(0,1))}/' 159 | f'{np.max(place_map_draw,axis=(0,1))}') 160 | my_dpi = 90 161 | if fig is None: 162 | fig = plt.figure( 163 | figsize=( 164 | place_map_draw.shape[1]/my_dpi, 165 | place_map_draw.shape[0]/my_dpi), 166 | dpi=my_dpi) 167 | if selected_minute is not None: 168 | plt.title(f'Location history for zone: {place_name} and' 169 | f' hour {int(selected_minute / 60)}:' 170 | f'{(selected_minute % 60):02}' 171 | f' + {minutes_step} minutes (UTC)') 172 | else: 173 | plt.title(f'Location history for zone: {place_name}' 174 | ' at any moment of the day') 175 | 176 | plt.xlabel('Longitude') 177 | plt.ylabel('Latitude') 178 | # extent is used to show the coordinates in the axis 179 | plt.imshow( 180 | base_map, 181 | extent=[v/10000000 for v in [x0, x1, y0, y1]], 182 | alpha=0.48) 183 | plt.imshow( 184 | moving_average_frame, 185 | cmap=plt.cm.Spectral, 186 | extent=[v/10000000 for v in [x0, x1, y0, y1]], 187 | origin='lower', 188 | alpha=0.5) 189 | if selected_minute is not None: 190 | # note the :04 to add the trailing 0s 191 | # so the lexicographic order is numeric as well 192 | # and the subsequent command line command follows it 193 | frame_file = (f'locations_in_{place_name}_time_' 194 | f'{frame_idx:04}.png') 195 | plt.savefig(frame_file) 196 | filenames.append(frame_file) 197 | else: 198 | plt.savefig(f'locations_in_{place_name}' 199 | '_all_time_weighted.png') 200 | if selected_minute is None: 201 | # for simplicity, everything is drawn on the same matrix 202 | # it has to be "cleared" to avoid a "flash" on the first frame 203 | moving_average_frame = None 204 | # then, also move the quintiles so that the expected 205 | # distribution of values in a frame in normalized 206 | # for the total time, otherwise, calculating the 207 | # quintiles over the whole day, every frame would be dark 208 | quintiles = quintiles * len(all_minutes_starts) 209 | # clear the figure, faster than deleting and recreating a new one 210 | plt.clf() 211 | print('generating the GIF...') 212 | with imageio.get_writer( 213 | f'{place_name}.gif', 214 | mode='I', 215 | duration=0.3, 216 | subrectangles=True, 217 | ) as writer: 218 | for filename in filenames: 219 | print(f'Appending frame {filename} to the GIF') 220 | image = imageio.imread(filename) 221 | # GIF is quite space hungry 222 | if image.shape[0] > 500: 223 | factor = image.shape[0] / 500 224 | image = resize( 225 | image, 226 | (round(image.shape[0] / factor), 227 | round(image.shape[1] / factor), 228 | image.shape[2]), 229 | anti_aliasing=True) 230 | writer.append_data(image) 231 | 232 | print('generating the video...') 233 | isc = ImageSequenceClip(filenames, fps=4) 234 | isc.write_videofile(f'{place_name}.webm') 235 | 236 | 237 | if __name__ == '__main__': 238 | parser = argparse.ArgumentParser() 239 | parser.add_argument('place_name', help='Name for the title') 240 | parser.add_argument('x0', type=int, help='min longitude, in E7 format') 241 | parser.add_argument('x1', type=int, help='max longitude, in E7 format') 242 | parser.add_argument('y0', type=int, help='min latitude, in E7 format') 243 | parser.add_argument('y1', type=int, help='max latituden, in E7 format') 244 | parser.add_argument( 245 | 'scaling_factor', 246 | type=int, 247 | help=''' 248 | scaling factor between pixels and coordinates. 249 | 1000 means about 1 cell per 10 meters 250 | 1 px = 10 meters = ~0.00009 lat/long degrees 251 | ''') 252 | parser.add_argument('input_file', help='input Takeout JSON') 253 | parser.add_argument( 254 | 'base_file', 255 | help='the map background for the given coordinates') 256 | 257 | args = parser.parse_args() 258 | main( 259 | args.input_file, 260 | args.base_file, 261 | args.place_name, 262 | args.x0, 263 | args.x1, 264 | args.y0, 265 | args.y1, 266 | args.scaling_factor, 267 | ) 268 | --------------------------------------------------------------------------------