├── .gitignore
├── LICENSE
├── README.md
├── config
    └── map_sources.json
├── documentation
    └── documentation.md
├── requirements.txt
├── setup.py
├── src
    └── TileDL.py
├── templates
    └── index.html
└── utils
    └── dependency_installer.py


/.gitignore:
--------------------------------------------------------------------------------
 1 | ## MacOS
 2 | **/.DS_Store
 3 | 
 4 | ## Virtual Environment
 5 | **/.venv
 6 | .venv
 7 | 
 8 | ## UV Virtual Environment
 9 | .python-version
10 | pyproject.toml
11 | main.py
12 | uv.lock
13 | 
14 | **/src/TileDL_002.py
15 | **/src/TileDL_002.ipynb
16 | **/templates/styles.css
17 | **/templates/index_copy.html


--------------------------------------------------------------------------------
/LICENSE:
--------------------------------------------------------------------------------
 1 | MIT License
 2 | 
 3 | Copyright (c) 2025 Matthew Drummonds
 4 | 
 5 | Permission is hereby granted, free of charge, to any person obtaining a copy
 6 | of this software and associated documentation files (the "Software"), to deal
 7 | in the Software without restriction, including without limitation the rights
 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
 9 | copies of the Software, and to permit persons to whom the Software is
10 | furnished to do so, subject to the following conditions:
11 | 
12 | The above copyright notice and this permission notice shall be included in all
13 | copies or substantial portions of the Software.
14 | 
15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21 | SOFTWARE.
22 | 


--------------------------------------------------------------------------------
/README.md:
--------------------------------------------------------------------------------
  1 | # Map Tile Downloader
  2 | 
  3 | The Map Tile Downloader is a Flask-based web application designed to download map tiles from various sources. It allows users to select specific areas on a map, choose zoom levels, and download tiles for offline use. The application supports converting tiles to 8-bit color depth for compatibility with Meshtastic® UI Maps and provides options to view and manage cached tiles.
  4 | 
  5 | <img width="964" alt="image" src="https://github.com/user-attachments/assets/57b041c9-2be6-4bf4-9ed4-98df24472a47" />
  6 | 
  7 | ## Features
  8 | 
  9 | - **Custom Area Downloads**: Draw polygons on the map to select specific areas for tile downloading.
 10 | - **World Basemap Downloads**: Download tiles for the entire world at zoom levels 0-7.
 11 | - **8-bit Conversion**: Option to convert downloaded tiles to 8-bit color depth for Meshtastic® UI Maps.
 12 | - **Cache Management**: View and delete cached tiles for different map styles.
 13 | - **Progress Tracking**: Real-time progress bar showing downloaded, skipped, and failed tiles.
 14 | - **Configurable Map Sources**: Easily add or modify map sources via a JSON configuration file.
 15 | 
 16 | ## Prerequisites
 17 | 
 18 | - Python 3.8 or higher (For Windows, make sure that Python in installed with the ADD TO PATH option selected)
 19 | - A modern web browser (Chrome, Firefox, Edge, etc.)
 20 | - Git (for cloning the repository)
 21 | 
 22 | ## Installation
 23 | 
 24 | 1. Clone the Repository (or download the zip file and extract to the location of your choice):
 25 |    
 26 | 		git clone https://github.com/mattdrum/map-tile-downloader.git
 27 | 		cd map-tile-downloader
 28 |    
 29 | 2. Install Dependencies (Optional) :
 30 |     	The application will automatically install required dependencies from requirements.txt on startup. However, you can manually install them using:
 31 | 
 32 | 		pip install -r requirements.txt
 33 | 
 34 | 3. Set Up Configuration (Optional, default sources are included) :
 35 | 
 36 |    Ensure the config/map_sources.json file is present and correctly formatted. See the Configuration section below for an example.
 37 |    
 38 |    
 39 | ## Configuration
 40 | The application uses a JSON configuration file (config/map_sources.json) to define available map sources. Each entry consists of a name and a URL template for the tiles.
 41 | 
 42 | Example map_sources.json:
 43 | 	
 44 | 		"OpenStreetMap": "https://{s}.tile.openstreetmap.org/{z}/{x}/{y}.png",
 45 | 		"OpenTopoMap": "https://{s}.tile.opentopomap.org/{z}/{x}/{y}.png",
 46 | 		"Stamen Terrain": "http://{s}.tile.stamen.com/terrain/{z}/{x}/{y}.png",
 47 | 		"CartoDB Positron": "https://{s}.basemaps.cartocdn.com/light_all/{z}/{x}/{y}.png",
 48 | 		"CartoDB Dark Matter": "https://{s}.basemaps.cartocdn.com/dark_all/{z}/{x}/{y}.png"
 49 | 	
 50 | 
 51 | Adding a New Map Source: Simply add a new key-value pair to the JSON file with the map name and its tile URL template.
 52 | 
 53 | 
 54 | ## Usage
 55 | 1.	Navigate to the application directory and Run the Application:
 56 | 
 57 | 		python src/TileDL.py
 58 | 	
 59 | 	The application will start a local server at http://localhost:5000.
 60 | 	- Alternatively you may create a Batch file "StartMap.bat" to launch from windows:
 61 |  - 		@echo off
 62 | 		cd /d C:\(extractlocation)\map-tile-downloader
 63 | 		python scr/TileDL.py
 64 | 		pause
 65 | 
 66 | 3. 	Access the Web Interface:
 67 | 
 68 | 	Open your web browser and navigate to http://localhost:5000.
 69 | 		
 70 | 4. 	Select Map Style:
 71 | 
 72 | 	Choose a map style from the dropdown menu. The available options are loaded from map_sources.json.
 73 | 5. 	Draw Polygons:
 74 | 	
 75 | 	Use the drawing tools to select areas on the map for which you want to download tiles.
 76 | 		
 77 | 6.	Set Zoom Levels:
 78 | 	
 79 | 	Specify the minimum and maximum zoom levels for the tiles you wish to download.
 80 | 
 81 | 7.	Download Tiles:
 82 | 
 83 | 	Click "Download Tiles" to start downloading tiles for the selected areas and zoom levels.
 84 | 	Alternatively, click "Download World Basemap" to download tiles for the entire world at zoom levels 0-7.
 85 | 
 86 | 8.	Monitor Progress:
 87 | 
 88 | 	The progress bar will display the number of downloaded, skipped, and failed tiles.
 89 | 
 90 | 9.	Manage Cache:
 91 | 
 92 | 	Check "View cached tiles" to see outlines of cached tiles on the map.
 93 | 
 94 | 	Use "Delete Cache" to remove cached tiles for the selected map style.
 95 | 
 96 | 
 97 | ## Contributing
 98 | 
 99 | We welcome contributions to improve the Map Tile Downloader! To contribute:
100 | 
101 | - Fork the Repository: Create your own fork of the project.
102 | - Create a Feature Branch: Work on your feature or bug fix in a separate branch.
103 | - Submit a Pull Request: Once your changes are ready, submit a pull request to the main repository.
104 | - Coding Standards: Follow PEP 8 for Python code and ensure your code is well-documented.
105 | - Testing: Test your changes locally before submitting a pull request.
106 | 
107 | ## License
108 | 
109 | This project is licensed under the MIT License. See the  file for details.
110 | 
111 | Contact Information
112 | For questions, suggestions, or support, please open an issue on the GitHub repository or contact k4mbd.ham@gmail.com
113 | 
114 | ## Acknowledgements
115 | 
116 | - Leaflet: For the interactive map interface.
117 | - Flask: For the web framework.
118 | - SocketIO: For real-time communication.
119 | - Mercantile: For tile calculations.
120 | - Shapely: For geometric operations.
121 | - Pillow: For image processing.
122 | 
123 | Special thanks to all contributors and the open-source community!
124 | 


--------------------------------------------------------------------------------
/config/map_sources.json:
--------------------------------------------------------------------------------
1 | {
2 |   "Standard OSM": "https://{s}.tile.openstreetmap.org/{z}/{x}/{y}.png",
3 |   "OpenTopoMap Outdoors": "https://{s}.tile.opentopomap.org/{z}/{x}/{y}.png",
4 |   "CartoDB Positron": "https://{s}.basemaps.cartocdn.com/light_all/{z}/{x}/{y}.png",
5 |   "CartoDB Dark Matter": "https://{s}.basemaps.cartocdn.com/dark_all/{z}/{x}/{y}.png",
6 |   "Esri World Imagery Satellite": "https://server.arcgisonline.com/ArcGIS/rest/services/World_Imagery/MapServer/tile/{z}/{y}/{x}",
7 |   "Google Satellite": "https://mt1.google.com/vt/lyrs=s&x={x}&y={y}&z={z}"
8 | }
9 | 


--------------------------------------------------------------------------------
/documentation/documentation.md:
--------------------------------------------------------------------------------
  1 | # Map Tile Downloader - Documentation
  2 | 
  3 | Documentation for Map Tile Downloader
  4 | 
  5 | ## Table of Contents
  6 | * [About](#about)
  7 | * [Features](#features)
  8 | * [Prerequisites](#prerequisites)
  9 | * [Installation](#installation)
 10 | * [Configuration](#configuration)
 11 | * [Usage](#usage)
 12 | * [Contributing](#contributing)
 13 | * [License](#license)
 14 | * [Acknowledgements](#acknowledgements)
 15 | 
 16 | 
 17 | ## About
 18 | The Map Tile Downloader is a Flask-based web application designed to download map tiles from various sources. It allows users to select specific areas on a map, choose zoom levels, and download tiles for offline use. The application supports converting tiles to 8-bit color depth for compatibility with Meshtastic® UI Maps and provides options to view and manage cached tiles.
 19 | 
 20 | <img width="964" alt="image" src="https://github.com/user-attachments/assets/57b041c9-2be6-4bf4-9ed4-98df24472a47" />
 21 | 
 22 | 
 23 | ## Features
 24 | 
 25 | - **Custom Area Downloads**: Draw polygons, or rectangles on the map to select specific areas for tile downloading.
 26 | - **World Basemap Downloads**: Download tiles for the entire world at zoom levels 0-7.
 27 | - **8-bit Conversion**: Option to convert downloaded tiles to 8-bit color depth for Meshtastic® UI Maps.
 28 | - **Cache Management**: View and delete cached tiles for different map styles.
 29 | - **Progress Tracking**: Real-time progress bar showing downloaded, skipped, and failed tiles.
 30 | - **Configurable Map Sources**: Easily add or modify map sources via a JSON configuration file.
 31 | 
 32 | ## Prerequisites
 33 | 
 34 | - Python 3.8 or higher.
 35 |     - (For Windows, make sure that Python in installed with the ADD TO PATH option selected).
 36 |     - Note: While not immediately required, installing a current and supported Python version is recommended, such as Python 3.11, is recommended. Older versions may become deprecated in the future.
 37 | - A Virtual Environment to install the required libraries and run the code.
 38 | - A modern Web Browser (Chrome, Firefox, Edge, etc.).
 39 | - Git (for cloning the repository).
 40 | 
 41 | ## Virtual Environments
 42 | 2. ### Create and Activate a Virtual Environment
 43 | * Select your preferred way to create a virtual environment. There are many tools and methods to create them.
 44 | 
 45 | 
 46 | ## Installation
 47 | 0. Select the path on where the project will be created.
 48 | 
 49 | 1. Clone the Repository (or download the zip file and extract to the location of your choice):
 50 |    
 51 |     git clone https://github.com/mattdrum/map-tile-downloader.git
 52 |     cd map-tile-downloader`
 53 |    
 54 | 2. Install Dependencies
 55 | 2.1) Activate virtual environment.
 56 | 
 57 | 2.2.1) Install Dependencies using the dependency installer (Optional):
 58 | 
 59 | The application `utils/dependency_installer.py` will automatically install required dependencies from requirements.txt after running the code.
 60 | 
 61 | - Run the following code inside the terminal:
 62 |         `utils/dependency_installer.py`
 63 | 
 64 |     Note:
 65 |     Trying to run the code above might cause some errors when using a virtual environment created with UV from Astral.
 66 |     To solve this issue, make sure that pip is installed as a package inside the activated environment.
 67 | 
 68 |     **Only for virtual environments created with UV Astral** 
 69 |     Run the following command to install pip inside the activated environment:
 70 |     `uv add pip`
 71 | 
 72 | 2.2.2) Install all the libraries manually:
 73 |         
 74 | `pip install -r requirements.txt`
 75 | 
 76 | 2.2.3) Manually install libraries one by one:
 77 | 
 78 | `pip install LibraryName`
 79 | 
 80 | 
 81 | 3. Set Up Configuration (Optional, default sources are included) :
 82 | 
 83 |    Ensure the config/map_sources.json file is present and correctly formatted. See the Configuration section below for an example.
 84 |    
 85 |    
 86 | ## Configuration
 87 | The application uses a JSON configuration file (config/map_sources.json) to define available map sources. Each entry consists of a name and a URL template for the tiles.
 88 | 
 89 | Example map_sources.json:
 90 |     
 91 |         "OpenStreetMap": "https://{s}.tile.openstreetmap.org/{z}/{x}/{y}.png",
 92 |         "OpenTopoMap": "https://{s}.tile.opentopomap.org/{z}/{x}/{y}.png",
 93 |         "Stamen Terrain": "http://{s}.tile.stamen.com/terrain/{z}/{x}/{y}.png",
 94 |         "CartoDB Positron": "https://{s}.basemaps.cartocdn.com/light_all/{z}/{x}/{y}.png",
 95 |         "CartoDB Dark Matter": "https://{s}.basemaps.cartocdn.com/dark_all/{z}/{x}/{y}.png"
 96 |     
 97 | 
 98 | Adding a New Map Source: Simply add a new key-value pair to the JSON file with the map name and its tile URL template.
 99 | 
100 | 
101 | ## Usage
102 | 0. Activate the virtual environment
103 | 
104 | 1.	Navigate to the application directory and Run the Application:
105 | 
106 |         python src/TileDL.py
107 |     
108 |     The application will start a local server at http://localhost:5000.
109 |     - Alternatively you may create a Batch file "StartMap.bat" to launch from windows:
110 |  - 		@echo off
111 |         cd /d C:\(extractlocation)\map-tile-downloader
112 |         python scr/TileDL.py
113 |         pause
114 | 
115 | 3. 	Access the Web Interface:
116 | 
117 |     Open your web browser and navigate to http://localhost:5000.
118 |         
119 | 4. 	Select Map Style:
120 | 
121 |     Choose a map style from the dropdown menu. The available options are loaded from map_sources.json.
122 | 5. 	Draw Polygons:
123 |     
124 |     Use the drawing tools to select areas on the map for which you want to download tiles.
125 |         
126 | 6.	Set Zoom Levels:
127 |     
128 |     Specify the minimum and maximum zoom levels for the tiles you wish to download.
129 | 
130 | 7.	Download Tiles:
131 | 
132 |     Click "Download Tiles" to start downloading tiles for the selected areas and zoom levels.
133 |     Alternatively, click "Download World Basemap" to download tiles for the entire world at zoom levels 0-7.
134 | 
135 | 8.	Monitor Progress:
136 | 
137 |     The progress bar will display the number of downloaded, skipped, and failed tiles.
138 | 
139 | 9.	Manage Cache:
140 | 
141 |     Check "View cached tiles" to see outlines of cached tiles on the map.
142 | 
143 |     Use "Delete Cache" to remove cached tiles for the selected map style.
144 | 
145 | 
146 | ## Contributing
147 | 
148 | We welcome contributions to improve the Map Tile Downloader! To contribute:
149 | 
150 | - Fork the Repository: Create your own fork of the project.
151 | - Create a Feature Branch: Work on your feature or bug fix in a separate branch.
152 | - Submit a Pull Request: Once your changes are ready, submit a pull request to the main repository.
153 | - Coding Standards: Follow PEP 8 for Python code and ensure your code is well-documented.
154 | - Testing: Test your changes locally before submitting a pull request.
155 | 
156 | ## License
157 | 
158 | This project is licensed under the MIT License. See the  file for details.
159 | 
160 | Contact Information
161 | For questions, suggestions, or support, please open an issue on the GitHub repository or contact k4mbd.ham@gmail.com
162 | 
163 | ## Acknowledgements
164 | 
165 | - Leaflet: For the interactive map interface.
166 | - Flask: For the web framework.
167 | - SocketIO: For real-time communication.
168 | - Mercantile: For tile calculations.
169 | - Shapely: For geometric operations.
170 | - Pillow: For image processing.
171 | 
172 | Special thanks to all contributors and the open-source community!
173 | 


--------------------------------------------------------------------------------
/requirements.txt:
--------------------------------------------------------------------------------
1 | flask
2 | flask-socketio
3 | requests
4 | mercantile
5 | shapely
6 | pillow


--------------------------------------------------------------------------------
/setup.py:
--------------------------------------------------------------------------------
 1 | from setuptools import setup, find_packages
 2 | 
 3 | setup(
 4 |     name="map-tile-downloader",
 5 |     version="0.1",
 6 |     packages=find_packages(where="src"),
 7 |     package_dir={"": "src"},
 8 |     install_requires=[
 9 |         "flask",
10 |         "flask-socketio",
11 |         "requests",
12 |         "mercantile",
13 |         "shapely",
14 |         "pillow"
15 |     ],
16 |     entry_points={
17 |         "console_scripts": [
18 |             "map-tile-downloader = tileDL:main"
19 |         ]
20 |     }
21 | )


--------------------------------------------------------------------------------
/src/TileDL.py:
--------------------------------------------------------------------------------
  1 | import subprocess
  2 | import sys
  3 | import os
  4 | from flask import Flask, render_template, request, send_file, jsonify
  5 | from flask_socketio import SocketIO, emit
  6 | import mercantile
  7 | import requests
  8 | from concurrent.futures import ThreadPoolExecutor, as_completed
  9 | from pathlib import Path
 10 | import zipfile
 11 | import random
 12 | import shutil
 13 | import re
 14 | import time
 15 | import json
 16 | from shapely.geometry import Polygon, box
 17 | from shapely.ops import unary_union
 18 | import threading
 19 | from PIL import Image
 20 | from pathlib import Path
 21 | 
 22 | # Base directory for caching tiles, absolute path relative to script location
 23 | BASE_DIR = Path(__file__).parent.parent  # Root of map-tile-downloader
 24 | CACHE_DIR = BASE_DIR / 'tile-cache' ## Note: Fixed referece location for cached tiles
 25 | DOWNLOADS_DIR = BASE_DIR / 'downloads'
 26 | CACHE_DIR.mkdir(exist_ok=True)
 27 | DOWNLOADS_DIR.mkdir(exist_ok=True)
 28 | 
 29 | ## Note: Moved to 'utils/dependency_installer.py'
 30 | # Ensure dependencies are installed
 31 | #def install_dependencies():
 32 | #    try:
 33 | #        with open('requirements.txt', 'r') as f:
 34 | #            requirements = f.read().splitlines()
 35 | #        subprocess.check_call([sys.executable, '-m', 'pip', 'install', *requirements])
 36 | #    except subprocess.CalledProcessError as e:
 37 | #        print(f"Failed to install dependencies: {e}")
 38 | #        sys.exit(1)
 39 | #
 40 | ## Install dependencies on startup if not already installed
 41 | #install_dependencies()
 42 | 
 43 | app = Flask(__name__, template_folder='../templates')
 44 | socketio = SocketIO(app)
 45 | 
 46 | 
 47 | # Load map sources from config file
 48 | CONFIG_DIR = Path('config')
 49 | MAP_SOURCES_FILE = CONFIG_DIR / 'map_sources.json'
 50 | MAP_SOURCES = {}
 51 | if MAP_SOURCES_FILE.exists():
 52 |     with open(MAP_SOURCES_FILE, 'r') as f:
 53 |         MAP_SOURCES = json.load(f)
 54 | else:
 55 |     print("Warning: map_sources.json not found. No map sources available.")
 56 |     sys.exit(1)
 57 | 
 58 | # Global event for cancellation
 59 | download_event = threading.Event()
 60 | 
 61 | def sanitize_style_name(style_name):
 62 |     """Convert map style name to a filesystem-safe directory name."""
 63 |     style_name = re.sub(r'\s+', '-', style_name)  # Replace spaces with hyphens
 64 |     style_name = re.sub(r'[^a-zA-Z0-9-_]', '', style_name)  # Remove non-alphanumeric except hyphens and underscores
 65 |     return style_name
 66 | 
 67 | def get_style_cache_dir(style_name):
 68 |     """Get the cache directory path for a given map style name."""
 69 |     sanitized_name = sanitize_style_name(style_name)
 70 |     return CACHE_DIR / sanitized_name
 71 | 
 72 | def download_tile(tile, map_style, style_cache_dir, convert_to_8bit, max_retries=3):
 73 |     """Download a single tile with retries if not cancelled and not in cache, converting to 8-bit if specified."""
 74 |     if not download_event.is_set():
 75 |         return None
 76 |     tile_dir = style_cache_dir / str(tile.z) / str(tile.x)
 77 |     tile_path = tile_dir / f"{tile.y}.png"
 78 |     if tile_path.exists():
 79 |         bounds = mercantile.bounds(tile)
 80 |         socketio.emit('tile_skipped', {
 81 |             'west': bounds.west,
 82 |             'south': bounds.south,
 83 |             'east': bounds.east,
 84 |             'north': bounds.north
 85 |         })
 86 |         return tile_path
 87 |     subdomain = random.choice(['a', 'b', 'c']) if '{s}' in map_style else ''
 88 |     url = map_style.replace('{s}', subdomain).replace('{z}', str(tile.z)).replace('{x}', str(tile.x)).replace('{y}', str(tile.y))
 89 |     headers = {'User-Agent': 'MapTileDownloader/1.0'}
 90 |     for attempt in range(max_retries):
 91 |         try:
 92 |             response = requests.get(url, headers=headers, timeout=10)
 93 |             if response.status_code == 200:
 94 |                 tile_dir.mkdir(parents=True, exist_ok=True)
 95 |                 with open(tile_path, 'wb') as f:
 96 |                     f.write(response.content)
 97 |                 if convert_to_8bit:
 98 |                     with Image.open(tile_path) as img:
 99 |                         if img.mode != 'P':  # Only convert if not already 8-bit palette
100 |                             img = img.quantize(colors=256)
101 |                             img.save(tile_path)
102 |                 bounds = mercantile.bounds(tile)
103 |                 socketio.emit('tile_downloaded', {
104 |                     'west': bounds.west,
105 |                     'south': bounds.south,
106 |                     'east': bounds.east,
107 |                     'north': bounds.north
108 |                 })
109 |                 return tile_path
110 |             else:
111 |                 time.sleep(2 ** attempt)  # Exponential backoff
112 |         except requests.RequestException:
113 |             time.sleep(2 ** attempt)  # Exponential backoff
114 |     socketio.emit('tile_failed', {
115 |         'tile': f"{tile.z}/{tile.x}/{tile.y}"
116 |     })
117 |     return None
118 | 
119 | def get_world_tiles():
120 |     """Generate list of tiles for zoom levels 0 to 7 for the entire world."""
121 |     tiles = []
122 |     for z in range(8):  # 0 to 7 inclusive
123 |         for x in range(2**z):
124 |             for y in range(2**z):
125 |                 tiles.append(mercantile.Tile(x, y, z))
126 |     return tiles
127 | 
128 | def get_tiles_for_polygons(polygons_data, min_zoom, max_zoom):
129 |     """Generate list of tiles that intersect with the given polygons for the specified zoom range."""
130 |     polygons = [Polygon([(lng, lat) for lat, lng in poly]) for poly in polygons_data]
131 |     overall_polygon = unary_union(polygons)
132 |     west, south, east, north = overall_polygon.bounds
133 |     all_tiles = []
134 |     for z in range(min_zoom, max_zoom + 1):
135 |         tiles = mercantile.tiles(west, south, east, north, zooms=[z])
136 |         for tile in tiles:
137 |             tile_bbox = mercantile.bounds(tile)
138 |             tile_box = box(tile_bbox.west, tile_bbox.south, tile_bbox.east, tile_bbox.north)
139 |             if any(tile_box.intersects(poly) for poly in polygons):
140 |                 all_tiles.append(tile)
141 |     all_tiles.sort(key=lambda tile: (tile.z, -tile.x, tile.y))
142 |     return all_tiles
143 | 
144 | def download_tiles_with_retries(tiles, map_style, style_cache_dir, convert_to_8bit):
145 |     """Download tiles with efficient retries using parallelism and adaptive backoff."""
146 |     socketio.emit('download_started', {'total_tiles': len(tiles)})
147 |     retry_queue = []
148 |     max_workers = 5
149 |     batch_size = 10
150 |     
151 |     def process_batch(batch):
152 |         with ThreadPoolExecutor(max_workers=max_workers) as executor:
153 |             futures = {executor.submit(download_tile, tile, map_style, style_cache_dir, convert_to_8bit): tile for tile in batch}
154 |             for future in as_completed(futures):
155 |                 if future.result() is None and download_event.is_set():
156 |                     retry_queue.append(futures[future])
157 |     
158 |     while tiles and download_event.is_set():
159 |         for i in range(0, len(tiles), batch_size):
160 |             if not download_event.is_set():
161 |                 break
162 |             batch = tiles[i:i + batch_size]
163 |             process_batch(batch)
164 |         tiles = retry_queue if retry_queue else []
165 |         retry_queue = []
166 |         if tiles:
167 |             delay = min(2 ** len(retry_queue), 8)
168 |             time.sleep(delay)
169 |     
170 |     if download_event.is_set():
171 |         socketio.emit('tiles_downloaded')
172 | 
173 | def create_zip(style_cache_dir, style_name):
174 |     """Create a zip file from the style-specific cache directory in the downloads folder."""
175 |     sanitized_name = sanitize_style_name(style_name)
176 |     zip_path = DOWNLOADS_DIR / f'{sanitized_name}.zip'  # Absolute path
177 |     with zipfile.ZipFile(zip_path, 'w', compression=zipfile.ZIP_DEFLATED) as zipf:
178 |         for root, _, files in os.walk(style_cache_dir):
179 |             for file in files:
180 |                 file_path = Path(root) / file
181 |                 arcname = file_path.relative_to(style_cache_dir)
182 |                 zipf.write(file_path, arcname)
183 |     return str(zip_path)  # Return as string for send_file
184 | 
185 | @app.route('/')
186 | def index():
187 |     """Render the main page."""
188 |     return render_template('index.html')
189 | 
190 | @app.route('/get_map_sources')
191 | def get_map_sources():
192 |     """Return the list of map sources from the config file."""
193 |     return jsonify(MAP_SOURCES)
194 | 
195 | @socketio.on('start_download')
196 | def handle_start_download(data):
197 |     """Handle download request for tiles within polygons."""
198 |     try:
199 |         polygons_data = data['polygons']
200 |         min_zoom = data['min_zoom']
201 |         max_zoom = data['max_zoom']
202 |         map_style_url = data['map_style']
203 |         convert_to_8bit = data.get('convert_to_8bit', False)
204 |         style_name = next(name for name, url in MAP_SOURCES.items() if url == map_style_url)
205 |         style_cache_dir = get_style_cache_dir(style_name)
206 |         if min_zoom < 0 or max_zoom > 19 or min_zoom > max_zoom:
207 |             emit('error', {'message': 'Invalid zoom range (must be 0-19, min <= max)'})
208 |             return
209 |         if not polygons_data:
210 |             emit('error', {'message': 'No polygons provided'})
211 |             return
212 |         tiles = get_tiles_for_polygons(polygons_data, min_zoom, max_zoom)
213 |         download_event.set()
214 |         download_tiles_with_retries(tiles, map_style_url, style_cache_dir, convert_to_8bit)
215 |         if download_event.is_set():
216 |             zip_path = create_zip(style_cache_dir, style_name)
217 |             emit('download_complete', {'zip_url': f'/download_zip?path={zip_path}'})
218 |     except Exception as e:
219 |         print(f"Error processing download: {e}")
220 |         emit('error', {'message': 'An error occurred while processing your request'})
221 | 
222 | @socketio.on('start_world_download')
223 | def handle_start_world_download(data):
224 |     """Handle download request for world basemap tiles (zoom 0-7)."""
225 |     try:
226 |         map_style_url = data['map_style']
227 |         convert_to_8bit = data.get('convert_to_8bit', False)
228 |         style_name = next(name for name, url in MAP_SOURCES.items() if url == map_style_url)
229 |         style_cache_dir = get_style_cache_dir(style_name)
230 |         tiles = get_world_tiles()
231 |         download_event.set()
232 |         download_tiles_with_retries(tiles, map_style_url, style_cache_dir, convert_to_8bit)
233 |         if download_event.is_set():
234 |             zip_path = create_zip(style_cache_dir, style_name)
235 |             emit('download_complete', {'zip_url': f'/download_zip?path={zip_path}'})
236 |     except Exception as e:
237 |         print(f"Error processing world download: {e}")
238 |         emit('error', {'message': 'An error occurred while processing your request'})
239 | 
240 | @socketio.on('cancel_download')
241 | def handle_cancel_download():
242 |     """Handle cancellation of the download."""
243 |     download_event.clear()
244 |     emit('download_cancelled')
245 | 
246 | @app.route('/download_zip')
247 | def download_zip():
248 |     """Send the zip file to the user."""
249 |     zip_path = request.args.get('path')
250 |     while not Path(zip_path).exists():  # Wait until the file is created
251 |         time.sleep(0.5)
252 |     return send_file(zip_path, as_attachment=True, download_name=Path(zip_path).name)
253 | 
254 | @app.route('/tiles/<style_name>/<int:z>/<int:x>/<int:y>.png')
255 | def serve_tile(style_name, z, x, y):
256 |     """Serve a cached tile if it exists."""
257 |     style_cache_dir = get_style_cache_dir(style_name)
258 |     tile_path = style_cache_dir / str(z) / str(x) / f"{y}.png"
259 |     if tile_path.exists():
260 |         return send_file(tile_path)
261 |     return '', 404
262 | 
263 | @app.route('/delete_cache/<style_name>', methods=['DELETE'])
264 | def delete_cache(style_name):
265 |     """Delete the cache directory for a specific style."""
266 |     cache_dir = get_style_cache_dir(style_name)
267 |     if cache_dir.exists():
268 |         shutil.rmtree(cache_dir)
269 |         return '', 204
270 |     return 'Cache not found', 404
271 | 
272 | @app.route('/get_cached_tiles/<style_name>')
273 | def get_cached_tiles_route(style_name):
274 |     """Return a list of [z, x, y] for cached tiles of the given style."""
275 |     style_cache_dir = get_style_cache_dir(style_name)
276 |     if not style_cache_dir.exists():
277 |         return jsonify([])
278 |     cached_tiles = []
279 |     for z_dir in style_cache_dir.iterdir():
280 |         if z_dir.is_dir():
281 |             try:
282 |                 z = int(z_dir.name)
283 |                 for x_dir in z_dir.iterdir():
284 |                     if x_dir.is_dir():
285 |                         try:
286 |                             x = int(x_dir.name)
287 |                             for y_file in x_dir.glob('*.png'):
288 |                                 try:
289 |                                     y = int(y_file.stem)
290 |                                     cached_tiles.append([z, x, y])
291 |                                 except ValueError:
292 |                                     pass
293 |                         except ValueError:
294 |                             pass
295 |             except ValueError:
296 |                 pass
297 |     return jsonify(cached_tiles)
298 | 
299 | if __name__ == '__main__':
300 |     CACHE_DIR.mkdir(exist_ok=True)
301 |     CONFIG_DIR.mkdir(exist_ok=True)
302 |     socketio.run(app, debug=True)
303 | 


--------------------------------------------------------------------------------
/templates/index.html:
--------------------------------------------------------------------------------
  1 | <!DOCTYPE html>
  2 | <html lang="en">
  3 | <head>
  4 |     <meta charset="UTF-8">
  5 |     <title>Map Tile Downloader</title>
  6 |     <link rel="stylesheet" href="https://unpkg.com/leaflet@1.9.4/dist/leaflet.css" />
  7 |     <link rel="stylesheet" href="https://unpkg.com/leaflet-draw@1.0.4/dist/leaflet.draw.css" />
  8 |     <script src="https://unpkg.com/leaflet@1.9.4/dist/leaflet.js"></script>
  9 |     <script src="https://unpkg.com/leaflet-draw@1.0.4/dist/leaflet.draw.js"></script>
 10 |     <script src="https://cdnjs.cloudflare.com/ajax/libs/socket.io/4.5.1/socket.io.js"></script>
 11 |     <style>
 12 |         body, html {
 13 |             margin: 0;
 14 |             padding: 0;
 15 |             height: 100%;
 16 |         }
 17 |         #map {
 18 |             height: 100vh;
 19 |             width: 100%;
 20 |         }
 21 |         .form-container {
 22 |             position: absolute;
 23 |             top: 8px;
 24 |             right: 10px;
 25 |             z-index: 1000;
 26 |             background: rgba(234, 233, 233, 0.822);
 27 |             padding: 10px;
 28 |             border-radius: 8px;
 29 |             box-shadow: 0 0 10px rgba(0,0,0,0.5);
 30 |         }
 31 |         #progress {
 32 |             position: absolute;
 33 |             top: 188px;
 34 |             right: 11px;
 35 |             z-index: 1000;
 36 |             background: rgba(241, 242, 243, 0.736);
 37 |             padding: 10px;
 38 |             border-radius: 5px;
 39 |             box-shadow: 2px 4px 8px rgba(0,0.2,0,0.5);
 40 |         }
 41 |     </style>
 42 | </head>
 43 | <body>
 44 |     <div style="position: relative;">
 45 |         <div id="map"></div>
 46 |         <div class="form-container">
 47 |             <form id="downloadForm">
 48 |                 <label for="map_style">Map Style:</label>
 49 |                 <select id="map_style" name="map_style"></select><br>
 50 |                 <label for="min_zoom">Min Zoom (0-19):</label>
 51 |                 <input type="number" id="min_zoom" name="min_zoom" min="0" max="19" value="10" required><br>
 52 |                 <label for="max_zoom">Max Zoom (0-19):</label>
 53 |                 <input type="number" id="max_zoom" name="max_zoom" min="0" max="19" value="12" required><br>
 54 |                 <input type="checkbox" id="use_cache" name="use_cache"> Use cached tiles<br>
 55 |                 <input type="checkbox" id="convert_to_8bit" checked> Convert to 8 bit for Meshtastic UI Maps<br>
 56 |                 <input type="checkbox" id="view_cached_tiles"> View cached tiles<br>
 57 |                 <button type="button" id="downloadBtn">Download Tiles</button>
 58 |                 <button type="button" id="downloadWorldBtn">Download World Basemap</button>
 59 |                 <button type="button" id="cancelBtn" disabled>Cancel Download</button>
 60 |                 <button type="button" id="deleteCacheBtn">Delete Cache</button>
 61 |             </form>
 62 |         </div>
 63 |         <div id="progress">Ready</div>
 64 |     </div>
 65 | 
 66 |     <script>
 67 |         // Initialize Socket.IO and Leaflet map
 68 |         var socket = io();
 69 |         var map = L.map('map');
 70 |         var tileLayer = L.tileLayer('https://{s}.tile.openstreetmap.org/{z}/{x}/{y}.png', {
 71 |             attribution: '© <a href="https://www.openstreetmap.org/copyright">OpenStreetMap</a> contributors',
 72 |             maxZoom: 19
 73 |         }).addTo(map);
 74 | 
 75 |         // Set initial view to user's location if possible
 76 |         if (navigator.geolocation) {
 77 |             navigator.geolocation.getCurrentPosition(function(position) {
 78 |                 var lat = position.coords.latitude;
 79 |                 var lng = position.coords.longitude;
 80 |                 map.setView([lat, lng], 13);
 81 |             }, function() {
 82 |                 map.setView([51.505, -0.09], 13);
 83 |             });
 84 |         } else {
 85 |             map.setView([51.505, -0.09], 13);
 86 |         }
 87 | 
 88 |         // Set up drawing controls
 89 |         var drawnItems = new L.FeatureGroup();
 90 |         map.addLayer(drawnItems);
 91 |         var drawControl = new L.Control.Draw({
 92 |             draw: { polygon: true, marker: false, circle: false, circlemarker: false, polyline: false, rectangle: true },
 93 |             edit: { featureGroup: drawnItems }
 94 |         });
 95 |         map.addControl(drawControl);
 96 | 
 97 |         map.on('draw:created', function(e) {
 98 |             drawnItems.addLayer(e.layer);
 99 |             document.getElementById('downloadBtn').disabled = drawnItems.getLayers().length === 0;
100 |         });
101 | 
102 |         map.on('draw:deleted', function() {
103 |             document.getElementById('downloadBtn').disabled = drawnItems.getLayers().length === 0;
104 |         });
105 | 
106 |         // Create layer groups
107 |         var missingTilesLayer = L.layerGroup().addTo(map);
108 |         var cachedTilesLayer = L.layerGroup().addTo(map);
109 |         var downloadProgressLayer = L.layerGroup().addTo(map);
110 | 
111 |         // Handle tile loading errors to draw missing tile grid
112 |         function onTileError(e) {
113 |             var coords = e.coords;
114 |             var z = coords.z;
115 |             var x = coords.x;
116 |             var y = coords.y;
117 |             var topLeft = map.unproject([x * 256, y * 256], z);
118 |             var bottomRight = map.unproject([(x + 1) * 256, (y + 1) * 256], z);
119 |             var bounds = L.latLngBounds(topLeft, bottomRight);
120 |             L.rectangle(bounds, { color: "red", weight: 1, fill: false }).addTo(missingTilesLayer);
121 |         }
122 | 
123 |         // Sanitize map style for cache URL
124 |         function sanitizeStyleName(name) {
125 |             name = name.replace(/\s+/g, '-');  // Replace spaces with hyphens
126 |             name = name.replace(/[^a-zA-Z0-9-_]/g, '');  // Remove non-alphanumeric except hyphens and underscores
127 |             return name;
128 |         }
129 | 
130 |         // Update tile layer based on map style and cache settings
131 |         function updateTileLayer() {
132 |             var mapStyleSelect = document.getElementById('map_style');
133 |             var mapStyleUrl = mapStyleSelect.value;
134 |             var styleName = sanitizeStyleName(mapStyleSelect.options[mapStyleSelect.selectedIndex].text);
135 |             var useCache = document.getElementById('use_cache').checked;
136 |             var tileUrl = useCache ? `/tiles/${styleName}/{z}/{x}/{y}.png` : mapStyleUrl;
137 |             tileLayer.setUrl(tileUrl);
138 |             tileLayer.off('loading');
139 |             tileLayer.off('tileerror', onTileError);
140 |             missingTilesLayer.clearLayers();
141 |             if (useCache) {
142 |                 tileLayer.on('loading', function() {
143 |                     missingTilesLayer.clearLayers();
144 |                 });
145 |                 tileLayer.on('tileerror', onTileError);
146 |             }
147 |         }
148 | 
149 |         // Populate map style dropdown from server
150 |         fetch('/get_map_sources')
151 |             .then(response => response.json())
152 |             .then(data => {
153 |                 var select = document.getElementById('map_style');
154 |                 for (var name in data) {
155 |                     var option = document.createElement('option');
156 |                     option.value = data[name];
157 |                     option.text = name;
158 |                     select.appendChild(option);
159 |                 }
160 |                 updateTileLayer(); // Set initial tile layer
161 |             });
162 | 
163 |         // Bind updateTileLayer to input changes
164 |         document.getElementById('map_style').addEventListener('change', function() {
165 |             updateTileLayer();
166 |             if (document.getElementById('view_cached_tiles').checked) {
167 |                 showCachedTiles();
168 |             }
169 |         });
170 |         document.getElementById('use_cache').addEventListener('change', updateTileLayer);
171 | 
172 |         // Handle tile download for polygons
173 |         document.getElementById('downloadBtn').addEventListener('click', function() {
174 |             var polygons = [];
175 |             drawnItems.eachLayer(function(layer) {
176 |                 if (layer instanceof L.Polygon) {
177 |                     var latlngs = layer.getLatLngs()[0];
178 |                     polygons.push(latlngs.map(function(latlng) { return [latlng.lat, latlng.lng]; }));
179 |                 }
180 |             });
181 |             if (polygons.length === 0) {
182 |                 alert('Please draw at least one shape.');
183 |                 return;
184 |             }
185 |             var data = {
186 |                 polygons: polygons,
187 |                 min_zoom: parseInt(document.getElementById('min_zoom').value),
188 |                 max_zoom: parseInt(document.getElementById('max_zoom').value),
189 |                 map_style: document.getElementById('map_style').value,
190 |                 convert_to_8bit: document.getElementById('convert_to_8bit').checked
191 |             };
192 |             socket.emit('start_download', data);
193 |         });
194 | 
195 |         // Handle world basemap download
196 |         document.getElementById('downloadWorldBtn').addEventListener('click', function() {
197 |             var mapStyleUrl = document.getElementById('map_style').value;
198 |             var convertTo8bit = document.getElementById('convert_to_8bit').checked;
199 |             socket.emit('start_world_download', {map_style: mapStyleUrl, convert_to_8bit: convertTo8bit});
200 |         });
201 | 
202 |         // Handle cache deletion
203 |         document.getElementById('deleteCacheBtn').addEventListener('click', function() {
204 |             if (confirm('Are you sure you want to delete the cached tiles for this style?')) {
205 |                 var mapStyleSelect = document.getElementById('map_style');
206 |                 var styleName = sanitizeStyleName(mapStyleSelect.options[mapStyleSelect.selectedIndex].text);
207 |                 fetch(`/delete_cache/${styleName}`, { method: 'DELETE' })
208 |                     .then(response => {
209 |                         if (response.ok) {
210 |                             alert('Cache deleted successfully');
211 |                             if (document.getElementById('view_cached_tiles').checked) {
212 |                                 showCachedTiles();
213 |                             }
214 |                         } else {
215 |                             alert('Failed to delete cache');
216 |                         }
217 |                     });
218 |             }
219 |         });
220 | 
221 |         // Handle cancel download
222 |         document.getElementById('cancelBtn').addEventListener('click', function() {
223 |             socket.emit('cancel_download');
224 |         });
225 | 
226 |         // Handle view cached tiles checkbox
227 |         document.getElementById('view_cached_tiles').addEventListener('change', function() {
228 |             if (this.checked) {
229 |                 showCachedTiles();
230 |             } else {
231 |                 cachedTilesLayer.clearLayers();
232 |             }
233 |         });
234 | 
235 |         // Function to show cached tiles
236 |         function showCachedTiles() {
237 |             cachedTilesLayer.clearLayers();
238 |             var mapStyleSelect = document.getElementById('map_style');
239 |             var styleName = sanitizeStyleName(mapStyleSelect.options[mapStyleSelect.selectedIndex].text);
240 |             fetch(`/get_cached_tiles/${styleName}`)
241 |                 .then(response => response.json())
242 |                 .then(data => {
243 |                     data.forEach(function(tile) {
244 |                         var z = tile[0], x = tile[1], y = tile[2];
245 |                         var topLeft = map.unproject([x * 256, y * 256], z);
246 |                         var bottomRight = map.unproject([(x + 1) * 256, (y + 1) * 256], z);
247 |                         var bounds = L.latLngBounds(topLeft, bottomRight);
248 |                         L.rectangle(bounds, { color: "#0000ff", weight: 1, fill: false }).addTo(cachedTilesLayer);
249 |                     });
250 |                 });
251 |         }
252 | 
253 |         // Progress tracking
254 |         var totalTiles = 0;
255 |         var downloadedTiles = 0;
256 |         var skippedTiles = 0;
257 |         var failedTiles = 0;
258 | 
259 |         socket.on('download_started', function(data) {
260 |             totalTiles = data.total_tiles;
261 |             downloadedTiles = 0;
262 |             skippedTiles = 0;
263 |             failedTiles = 0;
264 |             downloadProgressLayer.clearLayers();
265 |             document.getElementById('cancelBtn').disabled = false;
266 |             updateProgress();
267 |         });
268 | 
269 |         socket.on('tile_downloaded', function(data) {
270 |             downloadedTiles++;
271 |             updateProgress();
272 |             var bounds = [[data.south, data.west], [data.north, data.east]];
273 |             L.rectangle(bounds, { color: "#ff7800", weight: 1, fill: false }).addTo(downloadProgressLayer);
274 |         });
275 | 
276 |         socket.on('tile_skipped', function(data) {
277 |             skippedTiles++;
278 |             updateProgress();
279 |             var bounds = [[data.south, data.west], [data.north, data.east]];
280 |             L.rectangle(bounds, { color: "#00ff00", weight: 1, fill: false }).addTo(downloadProgressLayer);
281 |         });
282 | 
283 |         socket.on('tile_failed', function(data) {
284 |             failedTiles++;
285 |             updateProgress();
286 |         });
287 | 
288 |         socket.on('tiles_downloaded', function() {
289 |             document.getElementById('progress').innerHTML += '<br>Tiles downloaded, preparing zip...';
290 |         });
291 | 
292 |         socket.on('zipping_started', function() {
293 |             document.getElementById('progress').innerHTML += '<br>Zipping tiles...';
294 |         });
295 | 
296 |         socket.on('download_complete', function(data) {
297 |             document.getElementById('cancelBtn').disabled = true;
298 |             downloadProgressLayer.clearLayers();
299 |             document.getElementById('progress').innerHTML = 'Ready';
300 |             window.location.href = data.zip_url;
301 |         });
302 | 
303 |         socket.on('download_cancelled', function() {
304 |             document.getElementById('cancelBtn').disabled = true;
305 |             downloadProgressLayer.clearLayers();
306 |             document.getElementById('progress').innerHTML = 'Ready';
307 |             alert('Download cancelled');
308 |         });
309 | 
310 |         socket.on('error', function(data) {
311 |             alert(data.message);
312 |         });
313 | 
314 |         function updateProgress() {
315 |             var progress = ((downloadedTiles + skippedTiles + failedTiles) / totalTiles * 100).toFixed(2);
316 |             var progressText = `Progress: ${progress}% (${downloadedTiles} downloaded, ${skippedTiles} skipped, ${failedTiles} failed / ${totalTiles})`;
317 |             document.getElementById('progress').innerHTML = progressText;
318 |         }
319 |     </script>
320 | </body>
321 | </html>
322 | 


--------------------------------------------------------------------------------
/utils/dependency_installer.py:
--------------------------------------------------------------------------------
 1 | import subprocess
 2 | import sys
 3 | 
 4 | # Ensure dependencies are installed
 5 | def install_dependencies():
 6 |     """
 7 |     Installs all libraries located inside the requirements.txt
 8 | 
 9 |     Args
10 |         - The path is automatic no inputs for the user are required.
11 |         - Path location for requirements.txt
12 |     
13 |     Returns
14 |         - sys.executable for pip install requirements in requirements.txt
15 |         - print statement for Successul installation
16 |     """
17 |     try:
18 |         with open('requirements.txt', 'r') as f:
19 |             requirements = f.read().splitlines()
20 |         subprocess.check_call([sys.executable, '-m', 'pip', 'install', *requirements])
21 |         print(f"\nDependencies Installed Successfully.")
22 |     except subprocess.CalledProcessError as e:
23 |         print(f"Failed to install dependencies: {e}")
24 |         sys.exit(1)
25 | 
26 | # Install dependencies on startup if not already installed
27 | install_dependencies()


--------------------------------------------------------------------------------