├── .gitignore ├── README.md ├── images ├── enable_plugin.JPG ├── plugin_main_window.JPG ├── selected_product.JPG └── tool_bar_location.JPG ├── options.cfg ├── requirements-dev.txt ├── requirements.txt └── wpgpDatasets ├── CHANGELOG ├── __init__.py ├── config.ini ├── lib ├── __init__.py ├── about_window.py ├── csv_parser.py ├── downloader.py ├── main_window.py ├── utils.py └── wpftp.py ├── media ├── logo.png ├── wp.ico └── wpgpDatasets.csv.gz ├── metadata.txt ├── pb_tool.cfg ├── plugin_upload.py ├── ui ├── about_window.py ├── about_window.ui └── main_window.ui ├── wp_datasets.py └── wp_datasets_dialog.py /.gitignore: -------------------------------------------------------------------------------- 1 | .idea/ 2 | .test/*.log -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # wpDatasets 2 | 3 | About 4 | ----- 5 | ##### A plugin for QGIS 3 that helps users download raster products from the WorldPop Global Project 6 | 7 | The WorldPop Global High Resolution Population Denominators Project, 8 | funded by the Bill and Melinda Gates Foundation (OPP1134076), has 9 | produced an open-access archive of 3-arc seconds (approximately 100m 10 | at the equator) gridded population datasets, also structured by gender 11 | and age groups, for 249 countries, dependencies, and territories for 12 | 21-years (2000-2020), using the methods described by [Stevens et al., 13 | 2015](https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0107042), 14 | [Gaughan et al., 2016](https://www.nature.com/articles/sdata20165), and [Pezzulo et al., 15 | 2017](https://www.nature.com/articles/sdata201789). In addition, the project has also 16 | made available the covariate datasets used as inputs to produce the gridded population 17 | datasets (Lloyd et al., under review). These datasets are available for download from 18 | the WorldPop website and FTP server using a range of methods and tools. 19 | 20 | Installation Instructions 21 | ----- 22 | 23 | 1. Download the latest release [zip](https://github.com/wpgp/wpgpDataQPD/releases/download/v1.1/wpgpDatasets.zip) file. 24 | 2. Use it with 'Install from Zip file' (found at QGIS3 -> Plugins-> Manage and install Plugins ...) 25 | 26 | 27 | Usage 28 | ----- 29 | 30 | 1. After you put the folder into the appropriate directory you should enable it from 31 | __Plugins ->Manage and install Plugins ...__ 32 | 33 | ![enable plugin image](/images/enable_plugin.JPG) 34 | 35 | 2. A new entry should appear at the plugins menu: 36 | 37 | ![](/images/tool_bar_location.JPG) 38 | 39 | Clicking the button will open the plugin's main window: 40 | 41 | ![](/images/plugin_main_window.JPG) 42 | 43 | From this point the plugin is very straight forward: 44 | 45 | ![](/images/selected_product.JPG) 46 | 47 | - Each country, is a parent which contains a number of datasets. 48 | 49 | - After you locate the desired dataset from the list you can then click `download` and the addon will 50 | fetch that raster from the WorldPop FTP and store it your desired folder. 51 | - If the `Add downloaded file into Layer List` is checked, the downloaded file will be added automatically 52 | at the running QGIS layer list 53 | 54 | 55 | ISSUES 56 | ----- 57 | 58 | 59 | For any issues that may arise please open a ticket in our [GitHub repository](https://github.com/wpgp/wpgpDataQPD/issues) 60 | -------------------------------------------------------------------------------- /images/enable_plugin.JPG: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wpgp/wpgpDataQPD/5efadbd90a3800d1e190e61b969a93e7b8474f73/images/enable_plugin.JPG -------------------------------------------------------------------------------- /images/plugin_main_window.JPG: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wpgp/wpgpDataQPD/5efadbd90a3800d1e190e61b969a93e7b8474f73/images/plugin_main_window.JPG -------------------------------------------------------------------------------- /images/selected_product.JPG: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wpgp/wpgpDataQPD/5efadbd90a3800d1e190e61b969a93e7b8474f73/images/selected_product.JPG -------------------------------------------------------------------------------- /images/tool_bar_location.JPG: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wpgp/wpgpDataQPD/5efadbd90a3800d1e190e61b969a93e7b8474f73/images/tool_bar_location.JPG -------------------------------------------------------------------------------- /options.cfg: -------------------------------------------------------------------------------- 1 | [bumpversion] 2 | current_version = 0.1.0 3 | commit = No 4 | tag = True 5 | 6 | [bumpversion:file:wpgpDownload/__init__.py] 7 | search = __version__ = '{current_version}' 8 | replace = __version__ = '{new_version}' -------------------------------------------------------------------------------- /requirements-dev.txt: -------------------------------------------------------------------------------- 1 | bumpversion 2 | pb_tool -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | PyQt5 2 | numpy 3 | -------------------------------------------------------------------------------- /wpgpDatasets/CHANGELOG: -------------------------------------------------------------------------------- 1 | 1.2.1 2 | Fixed a bug in the download progress bar that caused an error during download. 3 | 4 | -------------------------------------------------------------------------------- /wpgpDatasets/__init__.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | __author__ = 'Nikolaos Ves' 3 | __email__ = 'vesnikos@gmail.com' 4 | __version__ = '0.1.0' 5 | 6 | from qgis.gui import QgisInterface 7 | 8 | 9 | # noinspection PyPep8Naming 10 | def classFactory(iface: QgisInterface): # pylint: disable=invalid-name 11 | """Load WpDatasets class from file WpDatasets.""" 12 | 13 | # type iface: QgsInterface 14 | from .wp_datasets import WpDatasets 15 | return WpDatasets(iface) 16 | 17 | -------------------------------------------------------------------------------- /wpgpDatasets/config.ini: -------------------------------------------------------------------------------- 1 | [ftp] 2 | server=ftp.worldpop.org.uk 3 | manifest=/assets/wpgpDatasets.csv 4 | sig=/assets/wpgpDatasets.md5 5 | 6 | [app] 7 | last_download_folder= 8 | about_text=wpgpDatasets-v1.2 9 | -------------------------------------------------------------------------------- /wpgpDatasets/lib/__init__.py: -------------------------------------------------------------------------------- 1 | from .csv_parser import WpCsvParser 2 | from .wpftp import wpFtp 3 | 4 | __all__ = ['wpFtp', 'WpCsvParser'] 5 | -------------------------------------------------------------------------------- /wpgpDatasets/lib/about_window.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | 3 | # Form implementation generated from reading ui file 'ui/about_window.ui' 4 | # 5 | # Created by: PyQt5 UI code generator 5.9 6 | # 7 | # WARNING! All changes made in this file will be lost! 8 | 9 | from PyQt5 import QtCore, QtGui, QtWidgets 10 | 11 | class Ui_AboutDialog(QtWidgets.QDialog): 12 | def setupUi(self, AboutDialog): 13 | AboutDialog.setObjectName("AboutDialog") 14 | AboutDialog.resize(258, 81) 15 | sizePolicy = QtWidgets.QSizePolicy(QtWidgets.QSizePolicy.Minimum, QtWidgets.QSizePolicy.Minimum) 16 | sizePolicy.setHorizontalStretch(0) 17 | sizePolicy.setVerticalStretch(0) 18 | sizePolicy.setHeightForWidth(AboutDialog.sizePolicy().hasHeightForWidth()) 19 | AboutDialog.setSizePolicy(sizePolicy) 20 | self.gridLayout_2 = QtWidgets.QGridLayout(AboutDialog) 21 | self.gridLayout_2.setObjectName("gridLayout_2") 22 | self.gridLayout = QtWidgets.QGridLayout() 23 | self.gridLayout.setObjectName("gridLayout") 24 | self.lbl_text = QtWidgets.QLabel(AboutDialog) 25 | sizePolicy = QtWidgets.QSizePolicy(QtWidgets.QSizePolicy.Fixed, QtWidgets.QSizePolicy.Fixed) 26 | sizePolicy.setHorizontalStretch(0) 27 | sizePolicy.setVerticalStretch(0) 28 | sizePolicy.setHeightForWidth(self.lbl_text.sizePolicy().hasHeightForWidth()) 29 | self.lbl_text.setSizePolicy(sizePolicy) 30 | self.lbl_text.setFrameShape(QtWidgets.QFrame.NoFrame) 31 | self.lbl_text.setFrameShadow(QtWidgets.QFrame.Raised) 32 | self.lbl_text.setAlignment(QtCore.Qt.AlignLeading|QtCore.Qt.AlignLeft|QtCore.Qt.AlignTop) 33 | self.lbl_text.setObjectName("lbl_text") 34 | self.gridLayout.addWidget(self.lbl_text, 0, 0, 1, 2) 35 | self.lbl_png = QtWidgets.QLabel(AboutDialog) 36 | sizePolicy = QtWidgets.QSizePolicy(QtWidgets.QSizePolicy.Fixed, QtWidgets.QSizePolicy.Fixed) 37 | sizePolicy.setHorizontalStretch(0) 38 | sizePolicy.setVerticalStretch(0) 39 | sizePolicy.setHeightForWidth(self.lbl_png.sizePolicy().hasHeightForWidth()) 40 | self.lbl_png.setSizePolicy(sizePolicy) 41 | self.lbl_png.setObjectName("lbl_png") 42 | self.gridLayout.addWidget(self.lbl_png, 2, 0, 1, 2) 43 | self.btn_ok = QtWidgets.QPushButton(AboutDialog) 44 | self.btn_ok.setObjectName("btn_ok") 45 | self.gridLayout.addWidget(self.btn_ok, 3, 1, 1, 1) 46 | self.gridLayout_2.addLayout(self.gridLayout, 0, 0, 1, 1) 47 | 48 | self.retranslateUi(AboutDialog) 49 | QtCore.QMetaObject.connectSlotsByName(AboutDialog) 50 | 51 | def retranslateUi(self, AboutDialog): 52 | _translate = QtCore.QCoreApplication.translate 53 | AboutDialog.setWindowTitle(_translate("AboutDialog", "Dialog")) 54 | self.lbl_text.setText(_translate("AboutDialog", "TextLabel")) 55 | self.lbl_png.setText(_translate("AboutDialog", "PNG_HOLDER")) 56 | self.btn_ok.setText(_translate("AboutDialog", "Ok")) 57 | 58 | -------------------------------------------------------------------------------- /wpgpDatasets/lib/csv_parser.py: -------------------------------------------------------------------------------- 1 | import platform 2 | import warnings 3 | import gzip 4 | from itertools import chain 5 | from pathlib import Path 6 | from typing import Any, Dict, List, Tuple, Union 7 | import csv 8 | 9 | import numpy as np 10 | 11 | # Numpy version numbers 12 | np_major, np_minor, np_micro = map(int, np.version.version.split('.')) 13 | 14 | 15 | class WpCsvParser(object): 16 | 17 | def __init__(self, csv_file=Union[str, Path], **kwargs): 18 | 19 | self._csv_path = Path(csv_file) 20 | self.delimiter = kwargs.get('delimiter') or ',' 21 | 22 | # Encoding 23 | # If provided use that, otherwise use the following heuristics 24 | self.encoding = kwargs.get('encoding') 25 | if self.encoding is None: 26 | if self.os == 'Windows': 27 | self.encoding = 'oem' # only available from python 3.6 onwards 28 | 29 | # if still None default to 'utf-8' 30 | if self.encoding is None: 31 | self.encoding = 'unicode_escape' 32 | 33 | # Private 34 | self._df = None 35 | self._isos = None 36 | self._indexes = None 37 | 38 | @property 39 | def csv_path(self) -> str: 40 | """ The _obsolete_ path to the CSV file including the file extension. Posix compliant """ 41 | res = self._csv_path 42 | res = Path(res) 43 | if not res.is_file(): 44 | raise AttributeError('CSV file not found. Folder = %s' % res.parent) 45 | res = Path(res).as_posix() 46 | return res 47 | 48 | @property 49 | def df(self) -> np.ndarray: 50 | """ 51 | Returns a numpy.ndarray that holds the data. 52 | Lazy loading. 53 | """ 54 | if self._df is None: 55 | delimiter = self.delimiter 56 | encoding = self.encoding 57 | lines = [] 58 | with gzip.open(self.csv_path, mode='rt', encoding=encoding) as fh: 59 | 60 | reader = csv.reader(fh, dialect='excel') 61 | for row_id, row in enumerate(reader): 62 | lines.append(row) 63 | df = np.array(lines, dtype=object) 64 | 65 | self._df = df 66 | return self._df 67 | 68 | @property 69 | def isos(self) -> List[Tuple[str, str]]: 70 | """ 71 | Returns a list of unique iso/name values. Filters out ISO3 that are more than 3 chars long. 72 | (e.g. IND_correct) 73 | :return: List(List(iso,english_name)) 74 | """ 75 | 76 | if self._isos is None: 77 | indexes = self.indexes 78 | idx_iso3 = indexes['idx_iso3'] 79 | idx_name_english = indexes['idx_name_english'] 80 | df = self.df[1:] # remove header, work only with data 81 | isos = df[:, [idx_name_english, idx_iso3]].astype(str) 82 | 83 | # filter the uniques per row 84 | # axis parameter was introduced from numpy 1.13 onwards. Unfortunately QGIS 3.0 - Standalone - ships with 85 | # numpy 1.12. 86 | if int(np_minor) >= 13: 87 | isos = np.unique(isos, axis=0) 88 | else: 89 | # iterates through the isos and filters the uniques keeping a records on how many rows under 90 | # that iso exist. 91 | from collections import Counter 92 | c = Counter() 93 | for k, v in isos: 94 | c.update(('%s+%s' % (k, v),)) # Damn you python for making me do this 95 | isos = np.array(list(map(lambda x: x.split('+'), list(c.keys())))) 96 | 97 | # remove double quotes from strings if any 98 | # ['"ARM"', '"Armenia"'] -> ['ARM', 'Armenia'] 99 | isos = list(map(lambda row: [str(x).replace('"', '') for x in row], isos)) 100 | 101 | # remove the ISOs that are more than 3 letters long. 102 | # eg. IND_correct 103 | condition = np.char.str_len(np.array(isos)[:, 1]) == 3 # the 2nd column of each row should be 3 chars long 104 | isos = np.array(isos)[condition] 105 | 106 | self._isos = isos.copy().tolist() 107 | 108 | return self._isos 109 | 110 | @property 111 | def indexes(self) -> Dict[str, int]: 112 | """ Build the row Indexess. Each index represent the column order of the said attribute.""" 113 | 114 | if self._indexes is None: 115 | indexes = dict() 116 | header = self.df[0] 117 | idx_id = np.where(header == 'ID')[0][0] 118 | idx_iso3 = np.where(header == 'ISO3')[0][0] 119 | idx_name_english = np.where(header == 'Country')[0][0] 120 | idx_cvt_name = np.where(header == 'Covariate')[0][0] 121 | idx_path_to_raster = np.where(header == 'PathToRaster')[0][0] 122 | idx_description = np.where(header == 'Description')[0][0] 123 | 124 | # column position in the csv 125 | indexes['idx_iso3'] = idx_iso3 126 | indexes['idx_name_english'] = idx_name_english 127 | indexes['idx_cvt_name'] = idx_cvt_name 128 | indexes['idx_path_to_raster'] = idx_path_to_raster 129 | indexes['idx_description'] = idx_description 130 | indexes['idx_id'] = idx_id 131 | 132 | # for k, v in indexes.items(): 133 | # print(k, v) 134 | 135 | self._indexes = indexes.copy() 136 | 137 | return self._indexes 138 | 139 | def products_per_iso(self, iso: str) -> Union[Tuple[list, list, list], List[Tuple[Any, Any, Any]]]: 140 | # Name, Description, FTP_PATH 141 | 142 | _df = self.df[1:, :] 143 | idx_iso = self.indexes['idx_iso3'] 144 | idx = np.where(_df[:, idx_iso] == iso) 145 | if len(idx[0]) == 0: 146 | iso = '"' + iso + '"' 147 | idx = np.where(_df[:, idx_iso] == iso) 148 | if len(idx[0]) == 0: 149 | warnings.warn('Warning, no products where found for this ISO. Check spelling') 150 | return [], [], [] 151 | 152 | idx_name, idx_description, path_to_raster_idx, = self.indexes['idx_cvt_name'], \ 153 | self.indexes['idx_description'], \ 154 | self.indexes['idx_path_to_raster'] 155 | 156 | per_iso_entries = _df[idx][:, [idx_name, idx_description, path_to_raster_idx]] 157 | name, description, path_to_raster = np.split(per_iso_entries, 3, axis=1) 158 | 159 | # clean double quotes if any 160 | name = np.core.defchararray.replace(name.astype(str), '"', '') 161 | path_to_raster = np.core.defchararray.replace(path_to_raster.astype(str), '"', '') 162 | description = np.core.defchararray.replace(description.astype(str), '"', '') 163 | 164 | name, description, path = list(chain.from_iterable(name.tolist())), \ 165 | list(chain.from_iterable(description.tolist())), \ 166 | list(chain.from_iterable(path_to_raster.tolist())), 167 | 168 | return list(zip(name, description, path)) 169 | 170 | @property 171 | def os(self): 172 | return platform.system() 173 | -------------------------------------------------------------------------------- /wpgpDatasets/lib/downloader.py: -------------------------------------------------------------------------------- 1 | from pathlib import Path 2 | from typing import Union 3 | from PyQt5.QtCore import pyqtSignal, QObject, QCoreApplication 4 | from PyQt5.QtWidgets import QProgressBar 5 | # from qgis.core import QgsApplication 6 | 7 | from .wpftp import wpFtp 8 | 9 | 10 | class DownloadThread(QObject): 11 | finished = pyqtSignal(str) # emits the LOCAL path (str) of the downloaded file 12 | 13 | def __init__(self, progress_bar: QProgressBar = None, 14 | parent=None, **kwargs): 15 | 16 | super().__init__(parent=parent) 17 | 18 | self.ftp = wpFtp(server=kwargs['server']) 19 | self.urls = None 20 | self.download_folder = None 21 | self.total_filesize = 0 22 | self.progress_bar = progress_bar 23 | self.progress = 0 24 | 25 | def __pbar(self, x): 26 | # sizeof = sys.getsizeof(x) 27 | sizeof = len(x) 28 | # # self.progress_bar 29 | self.progress += sizeof / self.total_filesize * 100 30 | if self.progress_bar is None: 31 | print(int(self.progress)) 32 | else: 33 | if self.progress_bar.value() != self.progress: 34 | self.progress_bar.setValue(int(self.progress)) 35 | try: 36 | QgsApplication.processEvents() 37 | except NameError: 38 | QCoreApplication.instance().processEvents() 39 | finally: 40 | pass 41 | 42 | def run(self, urls: Union[str, Path], download_folder: Union[Path, str],): 43 | self.urls = list(map(Path, urls)) # remote paths 44 | self.download_folder = Path(download_folder) 45 | 46 | for url in self.urls: 47 | self.total_filesize = self.ftp.get_total_filesize(self.urls) 48 | local_file_name = self.download_folder / url.name 49 | self.ftp.download(url, local_file_name, callback=lambda x: self.__pbar(x)) 50 | 51 | # HACK: Fill the pbar to full if it is not full at this stage. 52 | if self.progress_bar and self.progress_bar.value() < self.progress_bar.maximum(): 53 | self.progress_bar.setValue(self.progress_bar.maximum()) 54 | 55 | posix_path = local_file_name.as_posix() 56 | self.finished.emit(posix_path) 57 | -------------------------------------------------------------------------------- /wpgpDatasets/lib/main_window.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | 3 | # Form implementation generated from reading ui file 'ui/main_window.ui' 4 | # 5 | # Created by: PyQt5 UI code generator 5.10.1 6 | # 7 | # WARNING! All changes made in this file will be lost! 8 | 9 | from PyQt5 import QtCore, QtGui, QtWidgets 10 | 11 | class Ui_wpMainWindow(object): 12 | def setupUi(self, wpMainWindow): 13 | wpMainWindow.setObjectName("wpMainWindow") 14 | wpMainWindow.resize(469, 330) 15 | wpMainWindow.setSizeGripEnabled(False) 16 | wpMainWindow.setModal(False) 17 | self.horizontalLayout = QtWidgets.QHBoxLayout(wpMainWindow) 18 | self.horizontalLayout.setObjectName("horizontalLayout") 19 | self.MainGrid = QtWidgets.QGridLayout() 20 | self.MainGrid.setContentsMargins(0, -1, -1, -1) 21 | self.MainGrid.setHorizontalSpacing(6) 22 | self.MainGrid.setVerticalSpacing(8) 23 | self.MainGrid.setObjectName("MainGrid") 24 | self.le_directory = QtWidgets.QLineEdit(wpMainWindow) 25 | self.le_directory.setObjectName("le_directory") 26 | self.MainGrid.addWidget(self.le_directory, 2, 0, 1, 3) 27 | self.wp_header = QtWidgets.QLabel(wpMainWindow) 28 | self.wp_header.setScaledContents(True) 29 | self.wp_header.setObjectName("wp_header") 30 | self.MainGrid.addWidget(self.wp_header, 0, 0, 1, 1) 31 | self.btn_about = QtWidgets.QPushButton(wpMainWindow) 32 | self.btn_about.setObjectName("btn_about") 33 | self.MainGrid.addWidget(self.btn_about, 4, 0, 1, 1) 34 | self.btn_browse = QtWidgets.QPushButton(wpMainWindow) 35 | self.btn_browse.setObjectName("btn_browse") 36 | self.MainGrid.addWidget(self.btn_browse, 2, 3, 1, 1) 37 | self.pb_progressBar = QtWidgets.QProgressBar(wpMainWindow) 38 | self.pb_progressBar.setProperty("value", 24) 39 | self.pb_progressBar.setObjectName("pb_progressBar") 40 | self.MainGrid.addWidget(self.pb_progressBar, 3, 2, 1, 2) 41 | self.tree_widget = QtWidgets.QTreeWidget(wpMainWindow) 42 | self.tree_widget.setSelectionMode(QtWidgets.QAbstractItemView.ExtendedSelection) 43 | self.tree_widget.setObjectName("tree_widget") 44 | self.tree_widget.headerItem().setText(0, "1") 45 | self.MainGrid.addWidget(self.tree_widget, 1, 0, 1, 4) 46 | self.cbox_add_to_layer = QtWidgets.QCheckBox(wpMainWindow) 47 | self.cbox_add_to_layer.setText("") 48 | self.cbox_add_to_layer.setChecked(False) 49 | self.cbox_add_to_layer.setTristate(False) 50 | self.cbox_add_to_layer.setObjectName("cbox_add_to_layer") 51 | self.MainGrid.addWidget(self.cbox_add_to_layer, 3, 1, 1, 1) 52 | self.btn_close = QtWidgets.QPushButton(wpMainWindow) 53 | self.btn_close.setObjectName("btn_close") 54 | self.MainGrid.addWidget(self.btn_close, 4, 3, 1, 1) 55 | self.btn_download = QtWidgets.QPushButton(wpMainWindow) 56 | self.btn_download.setObjectName("btn_download") 57 | self.MainGrid.addWidget(self.btn_download, 4, 2, 1, 1) 58 | self.label = QtWidgets.QLabel(wpMainWindow) 59 | self.label.setObjectName("label") 60 | self.MainGrid.addWidget(self.label, 3, 0, 1, 1) 61 | self.horizontalLayout.addLayout(self.MainGrid) 62 | 63 | self.retranslateUi(wpMainWindow) 64 | QtCore.QMetaObject.connectSlotsByName(wpMainWindow) 65 | 66 | def retranslateUi(self, wpMainWindow): 67 | _translate = QtCore.QCoreApplication.translate 68 | wpMainWindow.setWindowTitle(_translate("wpMainWindow", "Dialog")) 69 | self.le_directory.setPlaceholderText(_translate("wpMainWindow", "Folder/to/Download")) 70 | self.wp_header.setText(_translate("wpMainWindow", "

WorldPop Dataset

")) 71 | self.btn_about.setText(_translate("wpMainWindow", "About")) 72 | self.btn_browse.setText(_translate("wpMainWindow", "Browse")) 73 | self.btn_close.setText(_translate("wpMainWindow", "Close")) 74 | self.btn_download.setText(_translate("wpMainWindow", "Download")) 75 | self.label.setText(_translate("wpMainWindow", "Add downloaded file(s) into Layer List")) 76 | 77 | 78 | if __name__ == "__main__": 79 | import sys 80 | app = QtWidgets.QApplication(sys.argv) 81 | wpMainWindow = QtWidgets.QDialog() 82 | ui = Ui_wpMainWindow() 83 | ui.setupUi(wpMainWindow) 84 | wpMainWindow.show() 85 | sys.exit(app.exec_()) 86 | 87 | -------------------------------------------------------------------------------- /wpgpDatasets/lib/utils.py: -------------------------------------------------------------------------------- 1 | import os 2 | import sys 3 | import gzip 4 | import platform 5 | from hashlib import md5 6 | from typing import Union 7 | from pathlib import Path 8 | 9 | from PyQt5.QtCore import QFileInfo 10 | from qgis.core import QgsRasterLayer 11 | 12 | 13 | def md5_digest(file: Union[Path, str], gz=False)->str: 14 | """ 15 | Returns the MD5 signature of the file. If the file is gz'ed, extract it contains before calculating. 16 | :param file: path to the file to generate the md5 signature. 17 | :param gz: If the file is compressed by the gz library. 18 | :return: MD5 hash string 19 | 20 | """ 21 | file = Path(file) 22 | # if the file doesn't exist, return 0 23 | if not file.is_file(): 24 | return '0' 25 | 26 | m = md5() 27 | if file.suffixes[-1].lower() == '.gz': 28 | gz = True 29 | 30 | # open/read the file with gzip module. 31 | if gz: 32 | m.update(gzip.open(file.as_posix()).read()) 33 | else: 34 | m.update(file.open(mode='rb').read()).hex() 35 | 36 | return m.digest().hex() 37 | 38 | 39 | def qgis3_add_raster_to_project(iface, raster_path: Path) -> bool: 40 | """ 41 | Adds the Raster(raster_path) at the current open qgis project 42 | 43 | :type iface: qgis.gui.QgisInterface 44 | :type raster_path: Path 45 | """ 46 | 47 | # TODO: Refactor: NV: Feels like incorrect usage of the APIs 48 | q_file_info = QFileInfo(raster_path.as_posix()) 49 | q_base_name = q_file_info.baseName() 50 | q_raster_layer = QgsRasterLayer(q_file_info.filePath(), q_base_name) 51 | if q_raster_layer.isValid(): 52 | iface.addRasterLayer(raster_path.as_posix(), raster_path.name) 53 | 54 | return True 55 | 56 | 57 | def get_default_download_directory() -> str: 58 | system = platform.system() 59 | if system == 'Windows': 60 | if sys.version_info.major == 3: 61 | import winreg 62 | else: 63 | import __winreg as winreg 64 | 65 | key = winreg.OpenKey(winreg.HKEY_CURRENT_USER, 66 | r'Software\Microsoft\Windows\CurrentVersion\Explorer\User Shell Folders') 67 | path, _ = winreg.QueryValueEx(key, 'desktop') 68 | elif system == 'Linux': 69 | path = os.path.expanduser('~') 70 | 71 | # for mac? others? a safe default 72 | else: 73 | path = os.path.expanduser('~') 74 | 75 | path = Path(path).as_posix() # make the path posix compliant 76 | return path 77 | 78 | 79 | def are_same(this: Path, other: Path) -> bool: 80 | """Compare if two files are the same or not, using MD5 hash. True, are the same. False are different""" 81 | 82 | # this function reads the whole file into memory. 83 | # For big tiffs that would slow things considerably 84 | 85 | this_md5 = md5(this.read_bytes()).hexdigest() 86 | other_md5 = md5(other.read_bytes()).hexdigest() 87 | 88 | return this_md5 == other_md5 89 | 90 | 91 | def resolve(name) -> Path: 92 | """Provided a name, returns /name Path object.""" 93 | dirname = Path(__file__).parent.parent 94 | return dirname.joinpath(name) 95 | 96 | 97 | def has_internet() -> bool: 98 | import urllib.request as req 99 | from urllib.error import URLError 100 | target = 'http://www.google.com/' 101 | try: 102 | payload = req.urlopen(target).read(100) 103 | except URLError: 104 | return False 105 | 106 | # empty payload, means we can't go through 107 | if not len(payload): 108 | return False 109 | 110 | # Finally 111 | return True 112 | 113 | 114 | BASE_ROOT = resolve('.') 115 | CSV_SIGNATURE = md5_digest(BASE_ROOT / 'media/wpgpDatasets.csv.gz') 116 | -------------------------------------------------------------------------------- /wpgpDatasets/lib/wpftp.py: -------------------------------------------------------------------------------- 1 | # functions 2 | import gzip 3 | import os 4 | import ftplib 5 | import logging 6 | import shutil 7 | 8 | from io import BytesIO 9 | from typing import Union 10 | from pathlib import Path 11 | from socket import gaierror, error 12 | from configparser import ConfigParser 13 | from tempfile import TemporaryDirectory 14 | 15 | 16 | from .utils import md5_digest 17 | 18 | 19 | class wpFtp(object): 20 | """ 21 | Convenience Class for ftp operations. 22 | ftp = wpFtp(server,username,password) 23 | """ 24 | # password = ftp_details.password 25 | # username = ftp_details.username 26 | # server = ftp_details.server 27 | timeout = 10 # ftp timeout, in seconds 28 | logger = logging.getLogger('library::wpFtp') 29 | logger.setLevel(logging.INFO) 30 | 31 | def __new__(cls, server='', username='anonymous', password='', config: ConfigParser = None): 32 | # if something goes wrong, return None 33 | try: 34 | ftp = ftplib.FTP(server, username, password, timeout=cls.timeout) 35 | cls.logger.info('FTP connection ok. Adress: %s, username: %s, password: %s' % (server, username, password)) 36 | except gaierror: 37 | cls.logger.error('getaddrinfo failed. Target Server was: %s' % server) 38 | return None 39 | except ftplib.error_perm as e: 40 | cls.logger.error('Permision Error: %s' % str(e)) 41 | return None 42 | except TimeoutError as e: 43 | cls.logger.error('Timeout Error. Is an FTP running at %s ?' % server) 44 | return None 45 | # socket.error 46 | except error as e: 47 | cls.logger.error('socket error. error was %s\n' % str(e)) 48 | return None 49 | instance = super().__new__(cls) 50 | instance.ftp = ftp 51 | instance.username = username 52 | instance.password = password 53 | instance.server = server 54 | return instance 55 | 56 | def __init__(self, server='', username='anonymous', password='', config: ConfigParser = None): 57 | self.config = config 58 | self.ftp.sendcmd("TYPE i") # switch to binary mode 59 | 60 | @property 61 | def csv_signature(self) -> str: 62 | bio = BytesIO() 63 | ftp_sig_file = self.config['ftp']['sig'] 64 | p = Path(ftp_sig_file) 65 | self.ftp.retrbinary('RETR ' + p.as_posix(), bio.write) 66 | bio.seek(0) 67 | result = bio.read().decode('utf-8') 68 | result = result.strip() 69 | result = result.split(' ')[0] 70 | return result 71 | 72 | def __repr__(self): 73 | return type(self).__name__ + '@' + self.server 74 | 75 | def __del__(self): 76 | self.ftp.close() 77 | 78 | @property 79 | def newer_version_exists(self) -> bool: 80 | """ 81 | Checks if a new version exists in the ftp server 82 | :return: bool 83 | """ 84 | local_csv_file_gz = self.config['app']['csv_file_gz'] 85 | local_csv_file_gz = Path(local_csv_file_gz) 86 | if not local_csv_file_gz.is_file(): 87 | return True 88 | 89 | return self.csv_signature != md5_digest(local_csv_file_gz) 90 | 91 | def get_filesilze(self, ftp_absolute_path: Path) -> int or None: 92 | """ Returns the filesize from the ftp. Returns None if the file is not in the ftp """ 93 | filesize = self.ftp.size(ftp_absolute_path.as_posix()) 94 | # if response_code != '213': 95 | # raise wpException("Not ok return code (%s), when tried to retrieve filesize" % response_code) 96 | if filesize >= 0: 97 | return filesize 98 | return None 99 | 100 | def get_total_filesize(self, ftp_absolute_paths: list) -> int or None: 101 | """ Returns the total size of files from the ftp. Returns None if the file is not in the ftp """ 102 | filesize = 0 103 | for ftp_path in ftp_absolute_paths: 104 | filesize += self.ftp.size(ftp_path.as_posix()) 105 | # if response_code != '213': 106 | # raise wpException("Not ok return code (%s), when tried to retrieve filesize" % response_code) 107 | if filesize >= 0: 108 | return filesize 109 | return None 110 | 111 | def download(self, from_ftp_absolute_path: Union[str, Path], 112 | to_local_absolute_path: Union[str, Path], callback=None) -> Path: 113 | """ Download a file from the remote ftp, stores is locally. 114 | If file exists locally, it is removed beforehand. 115 | :return Path object to file. 116 | :rtype Path 117 | """ 118 | from_ftp_absolute_path = Path(from_ftp_absolute_path) 119 | to_local_absolute_path = Path(to_local_absolute_path) 120 | 121 | if self.get_filesilze(from_ftp_absolute_path) is None: 122 | raise TypeError 123 | if to_local_absolute_path.is_file(): 124 | os.remove(to_local_absolute_path.as_posix()) 125 | with to_local_absolute_path.open('wb') as fp: 126 | def __callback(data): 127 | fp.write(data) 128 | if callback: 129 | callback(data) 130 | 131 | self.ftp.retrbinary('RETR ' + from_ftp_absolute_path.as_posix(), __callback) 132 | 133 | return to_local_absolute_path 134 | 135 | def dl_wpgpDatasets(self)->Path: 136 | """ 137 | Get the manifest file dl_wpgpDatasets from the ftp 138 | Compress it and save it in the path defined in the config.ini 139 | """ 140 | 141 | remote_file = Path(self.config['ftp']['manifest']) 142 | local_file = Path(self.config['app']['csv_file_gz']) 143 | 144 | with TemporaryDirectory() as t_dir: 145 | output_t_file = Path(t_dir) / remote_file.name 146 | csv_file = self.download(remote_file, output_t_file) 147 | 148 | with csv_file.open(mode='rb') as f_in: 149 | with gzip.open(local_file, 'wb') as f_out: 150 | shutil.copyfileobj(f_in, f_out) 151 | 152 | return local_file 153 | -------------------------------------------------------------------------------- /wpgpDatasets/media/logo.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wpgp/wpgpDataQPD/5efadbd90a3800d1e190e61b969a93e7b8474f73/wpgpDatasets/media/logo.png -------------------------------------------------------------------------------- /wpgpDatasets/media/wp.ico: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wpgp/wpgpDataQPD/5efadbd90a3800d1e190e61b969a93e7b8474f73/wpgpDatasets/media/wp.ico -------------------------------------------------------------------------------- /wpgpDatasets/media/wpgpDatasets.csv.gz: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wpgp/wpgpDataQPD/5efadbd90a3800d1e190e61b969a93e7b8474f73/wpgpDatasets/media/wpgpDatasets.csv.gz -------------------------------------------------------------------------------- /wpgpDatasets/metadata.txt: -------------------------------------------------------------------------------- 1 | # This file should be included when you package your plugin. 2 | # Mandatory items: 3 | 4 | [general] 5 | name=wpDatasets 6 | qgisMinimumVersion=3.0 7 | description=A simple qgis 3 addon to downlaod wp datasets from WorldPop ftp server. 8 | version=1.2.1 9 | 10 | # Author contact information 11 | author=Nikolaos Ves,Maksym Bondarenko,Aron Gergely,David Kerr and Alessandro Sorichetta 12 | email=info@worldpop.org 13 | 14 | about=A Qgis (version > 3) for discovering and downloading WorldPop (GlobalProject) datasets. 15 | 16 | homepage=https://sdi.worldpop.org/services/gis 17 | tracker=https://github.com/wpgp/wpgpQGIS/issues 18 | repository=https://github.com/wpgp/wpgpQGIS 19 | # End of mandatory metadata 20 | 21 | # Recommended items: 22 | 23 | # Uncomment the following line and add your changelog: 24 | changelog= 25 | 26 | # Tags are comma separated with spaces allowed 27 | tags=worldpop, datasets, raster 28 | 29 | category=Web 30 | icon=media/wp.ico 31 | # experimental flag 32 | experimental=True 33 | 34 | # deprecated flag (applies to the whole plugin, not just a single version) 35 | deprecated=False 36 | 37 | -------------------------------------------------------------------------------- /wpgpDatasets/pb_tool.cfg: -------------------------------------------------------------------------------- 1 | #/*************************************************************************** 2 | # WpDatasets 3 | # 4 | # Configuration file for plugin builder tool (pb_tool) 5 | # Generated by Plugin Builder: http://g-sherman.github.io/Qgis-Plugin-Builder/ 6 | # ------------------- 7 | # begin : 2018-06-01 8 | # copyright : (C) 2018 by Nikolaos Ves 9 | # email : vesnikos@gmail.com 10 | # ***************************************************************************/ 11 | # 12 | #/*************************************************************************** 13 | # * * 14 | # * This program is free software; you can redistribute it and/or modify * 15 | # * it under the terms of the GNU General Public License as published by * 16 | # * the Free Software Foundation; either version 2 of the License, or * 17 | # * (at your option) any later version. * 18 | # * * 19 | # ***************************************************************************/ 20 | # 21 | # 22 | # You can install pb_tool using: 23 | # pip install http://geoapt.net/files/pb_tool.zip 24 | # 25 | # Consider doing your development (and install of pb_tool) in a virtualenv. 26 | # 27 | # For details on setting up and using pb_tool, see: 28 | # http://g-sherman.github.io/plugin_build_tool/ 29 | # 30 | # Issues and pull requests here: 31 | # https://github.com/g-sherman/plugin_build_tool: 32 | # 33 | # Sane defaults for your plugin generated by the Plugin Builder are 34 | # already set below. 35 | # 36 | # As you add Python source files and UI files to your plugin, add 37 | # them to the appropriate [files] section below. 38 | 39 | [plugin] 40 | # Name of the plugin. This is the name of the directory that will 41 | # be created in .qgis2/python/plugins 42 | name: wppgdatasets 43 | 44 | # Full path to where you want your plugin directory copied. If empty, 45 | # the QGIS default path will be used. Don't include the plugin name in 46 | # the path. 47 | plugin_path: 48 | 49 | [files] 50 | # Python files that should be deployed with the plugin 51 | python_files : __init__.py wp_datasets.py 52 | 53 | # The main dialog file that is loaded (not compiled) 54 | main_dialog: wp_datasets_dialog.py 55 | 56 | # Other ui files for dialogs you create (these will be compiled) 57 | compiled_ui_files: 58 | 59 | # Resource file(s) that will be compiled 60 | resource_files: 61 | 62 | # Other files required for the plugin 63 | extras : metadata.txt config.ini 64 | 65 | # Other directories to be deployed with the plugin. 66 | # These must be subdirectories under the plugin directory 67 | extra_dirs: media lib ui 68 | 69 | # ISO code(s) for any locales (translations), separated by spaces. 70 | # Corresponding .ts files must exist in the i18n directory 71 | locales: 72 | 73 | [help] 74 | # the built help directory that should be deployed with the plugin 75 | dir: help/build/html 76 | # the name of the directory to target in the deployed plugin 77 | target: help 78 | 79 | 80 | 81 | -------------------------------------------------------------------------------- /wpgpDatasets/plugin_upload.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # coding=utf-8 3 | """This script uploads a plugin package on the server. 4 | Authors: A. Pasotti, V. Picavet 5 | git sha : $TemplateVCSFormat 6 | """ 7 | 8 | import sys 9 | import getpass 10 | import xmlrpc.client 11 | from optparse import OptionParser 12 | 13 | # Configuration 14 | PROTOCOL = 'http' 15 | SERVER = 'plugins.qgis.org' 16 | PORT = '80' 17 | ENDPOINT = '/plugins/RPC2/' 18 | VERBOSE = False 19 | 20 | 21 | def main(parameters, arguments): 22 | """Main entry point. 23 | 24 | :param parameters: Command line parameters. 25 | :param arguments: Command line arguments. 26 | """ 27 | address = "%s://%s:%s@%s:%s%s" % ( 28 | PROTOCOL, 29 | parameters.username, 30 | parameters.password, 31 | parameters.server, 32 | parameters.port, 33 | ENDPOINT) 34 | print("Connecting to: %s" % hide_password(address)) 35 | 36 | server = xmlrpc.client.ServerProxy(address, verbose=VERBOSE) 37 | 38 | try: 39 | plugin_id, version_id = server.plugin.upload( 40 | xmlrpc.client.Binary(open(arguments[0]).read())) 41 | print("Plugin ID: %s" % plugin_id) 42 | print("Version ID: %s" % version_id) 43 | except xmlrpc.client.ProtocolError as err: 44 | print("A protocol error occurred") 45 | print("URL: %s" % hide_password(err.url, 0)) 46 | print("HTTP/HTTPS headers: %s" % err.headers) 47 | print("Error code: %d" % err.errcode) 48 | print("Error message: %s" % err.errmsg) 49 | except xmlrpc.client.Fault as err: 50 | print("A fault occurred") 51 | print("Fault code: %d" % err.faultCode) 52 | print("Fault string: %s" % err.faultString) 53 | 54 | 55 | def hide_password(url, start=6): 56 | """Returns the http url with password part replaced with '*'. 57 | 58 | :param url: URL to upload the plugin to. 59 | :type url: str 60 | 61 | :param start: Position of start of password. 62 | :type start: int 63 | """ 64 | start_position = url.find(':', start) + 1 65 | end_position = url.find('@') 66 | return "%s%s%s" % ( 67 | url[:start_position], 68 | '*' * (end_position - start_position), 69 | url[end_position:]) 70 | 71 | 72 | if __name__ == "__main__": 73 | parser = OptionParser(usage="%prog [options] plugin.zip") 74 | parser.add_option( 75 | "-w", "--password", dest="password", 76 | help="Password for plugin site", metavar="******") 77 | parser.add_option( 78 | "-u", "--username", dest="username", 79 | help="Username of plugin site", metavar="user") 80 | parser.add_option( 81 | "-p", "--port", dest="port", 82 | help="Server port to connect to", metavar="80") 83 | parser.add_option( 84 | "-s", "--server", dest="server", 85 | help="Specify server name", metavar="plugins.qgis.org") 86 | options, args = parser.parse_args() 87 | if len(args) != 1: 88 | print("Please specify zip file.\n") 89 | parser.print_help() 90 | sys.exit(1) 91 | if not options.server: 92 | options.server = SERVER 93 | if not options.port: 94 | options.port = PORT 95 | if not options.username: 96 | # interactive mode 97 | username = getpass.getuser() 98 | print("Please enter user name [%s] :" % username, end=' ') 99 | res = input() 100 | if res != "": 101 | options.username = res 102 | else: 103 | options.username = username 104 | if not options.password: 105 | # interactive mode 106 | options.password = getpass.getpass() 107 | main(options, args) 108 | -------------------------------------------------------------------------------- /wpgpDatasets/ui/about_window.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | 3 | # Form implementation generated from reading ui file 'ui/about_window.ui' 4 | # 5 | # Created by: PyQt5 UI code generator 5.9 6 | # 7 | # WARNING! All changes made in this file will be lost! 8 | 9 | from PyQt5 import QtCore, QtGui, QtWidgets 10 | 11 | class Ui_AboutDialog(object): 12 | def setupUi(self, AboutDialog): 13 | AboutDialog.setObjectName("AboutDialog") 14 | AboutDialog.resize(258, 81) 15 | sizePolicy = QtWidgets.QSizePolicy(QtWidgets.QSizePolicy.Minimum, QtWidgets.QSizePolicy.Minimum) 16 | sizePolicy.setHorizontalStretch(0) 17 | sizePolicy.setVerticalStretch(0) 18 | sizePolicy.setHeightForWidth(AboutDialog.sizePolicy().hasHeightForWidth()) 19 | AboutDialog.setSizePolicy(sizePolicy) 20 | self.gridLayout_2 = QtWidgets.QGridLayout(AboutDialog) 21 | self.gridLayout_2.setObjectName("gridLayout_2") 22 | self.gridLayout = QtWidgets.QGridLayout() 23 | self.gridLayout.setObjectName("gridLayout") 24 | self.lbl_text = QtWidgets.QLabel(AboutDialog) 25 | sizePolicy = QtWidgets.QSizePolicy(QtWidgets.QSizePolicy.Fixed, QtWidgets.QSizePolicy.Fixed) 26 | sizePolicy.setHorizontalStretch(0) 27 | sizePolicy.setVerticalStretch(0) 28 | sizePolicy.setHeightForWidth(self.lbl_text.sizePolicy().hasHeightForWidth()) 29 | self.lbl_text.setSizePolicy(sizePolicy) 30 | self.lbl_text.setFrameShape(QtWidgets.QFrame.NoFrame) 31 | self.lbl_text.setFrameShadow(QtWidgets.QFrame.Raised) 32 | self.lbl_text.setAlignment(QtCore.Qt.AlignLeading|QtCore.Qt.AlignLeft|QtCore.Qt.AlignTop) 33 | self.lbl_text.setObjectName("lbl_text") 34 | self.gridLayout.addWidget(self.lbl_text, 0, 0, 1, 2) 35 | self.lbl_png = QtWidgets.QLabel(AboutDialog) 36 | sizePolicy = QtWidgets.QSizePolicy(QtWidgets.QSizePolicy.Fixed, QtWidgets.QSizePolicy.Fixed) 37 | sizePolicy.setHorizontalStretch(0) 38 | sizePolicy.setVerticalStretch(0) 39 | sizePolicy.setHeightForWidth(self.lbl_png.sizePolicy().hasHeightForWidth()) 40 | self.lbl_png.setSizePolicy(sizePolicy) 41 | self.lbl_png.setObjectName("lbl_png") 42 | self.gridLayout.addWidget(self.lbl_png, 2, 0, 1, 2) 43 | self.btn_ok = QtWidgets.QPushButton(AboutDialog) 44 | self.btn_ok.setObjectName("btn_ok") 45 | self.gridLayout.addWidget(self.btn_ok, 3, 1, 1, 1) 46 | self.gridLayout_2.addLayout(self.gridLayout, 0, 0, 1, 1) 47 | 48 | self.retranslateUi(AboutDialog) 49 | QtCore.QMetaObject.connectSlotsByName(AboutDialog) 50 | 51 | def retranslateUi(self, AboutDialog): 52 | _translate = QtCore.QCoreApplication.translate 53 | AboutDialog.setWindowTitle(_translate("AboutDialog", "Dialog")) 54 | self.lbl_text.setText(_translate("AboutDialog", "TextLabel")) 55 | self.lbl_png.setText(_translate("AboutDialog", "PNG_HOLDER")) 56 | self.btn_ok.setText(_translate("AboutDialog", "Ok")) 57 | 58 | -------------------------------------------------------------------------------- /wpgpDatasets/ui/about_window.ui: -------------------------------------------------------------------------------- 1 | 2 | 3 | AboutDialog 4 | 5 | 6 | 7 | 0 8 | 0 9 | 258 10 | 81 11 | 12 | 13 | 14 | 15 | 0 16 | 0 17 | 18 | 19 | 20 | Dialog 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 0 30 | 0 31 | 32 | 33 | 34 | QFrame::NoFrame 35 | 36 | 37 | QFrame::Raised 38 | 39 | 40 | TextLabel 41 | 42 | 43 | Qt::AlignLeading|Qt::AlignLeft|Qt::AlignTop 44 | 45 | 46 | 47 | 48 | 49 | 50 | 51 | 0 52 | 0 53 | 54 | 55 | 56 | PNG_HOLDER 57 | 58 | 59 | 60 | 61 | 62 | 63 | Ok 64 | 65 | 66 | 67 | 68 | 69 | 70 | 71 | 72 | 73 | 74 | -------------------------------------------------------------------------------- /wpgpDatasets/ui/main_window.ui: -------------------------------------------------------------------------------- 1 | 2 | 3 | wpMainWindow 4 | 5 | 6 | 7 | 0 8 | 0 9 | 469 10 | 330 11 | 12 | 13 | 14 | Dialog 15 | 16 | 17 | false 18 | 19 | 20 | false 21 | 22 | 23 | 24 | 25 | 26 | 0 27 | 28 | 29 | 6 30 | 31 | 32 | 8 33 | 34 | 35 | 36 | 37 | Folder/to/Download 38 | 39 | 40 | 41 | 42 | 43 | 44 | <html><head/><body><p><span style=" font-size:16pt; font-weight:600;">WorldPop Dataset</span></p></body></html> 45 | 46 | 47 | true 48 | 49 | 50 | 51 | 52 | 53 | 54 | About 55 | 56 | 57 | 58 | 59 | 60 | 61 | Browse 62 | 63 | 64 | 65 | 66 | 67 | 68 | 24 69 | 70 | 71 | 72 | 73 | 74 | 75 | QAbstractItemView::ExtendedSelection 76 | 77 | 78 | 79 | 1 80 | 81 | 82 | 83 | 84 | 85 | 86 | 87 | 88 | 89 | 90 | false 91 | 92 | 93 | false 94 | 95 | 96 | 97 | 98 | 99 | 100 | Close 101 | 102 | 103 | 104 | 105 | 106 | 107 | Download 108 | 109 | 110 | 111 | 112 | 113 | 114 | Add downloaded file(s) into Layer List 115 | 116 | 117 | 118 | 119 | 120 | 121 | 122 | 123 | 124 | 125 | -------------------------------------------------------------------------------- /wpgpDatasets/wp_datasets.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | import configparser 3 | from pathlib import Path 4 | 5 | from PyQt5.QtGui import QIcon 6 | from PyQt5.QtWidgets import QAction 7 | 8 | from .wp_datasets_dialog import WpMainWindow, wpFactory 9 | 10 | # Global Variables 11 | BASE_DIR = Path(__file__).parent # Path object 12 | CSV_FILE_GZ = Path(BASE_DIR / 'media' / 'wpgpDatasets.csv.gz') 13 | INI_FILE = Path(BASE_DIR / 'config.ini') 14 | 15 | 16 | class WpDatasets: 17 | """QGIS Plugin Implementation.""" 18 | 19 | def __init__(self, iface): 20 | 21 | self.iface = iface 22 | 23 | self.canvas = iface.mapCanvas() 24 | # initialize plugin directory 25 | self.plugin_dir = BASE_DIR 26 | 27 | 28 | # Class methods 29 | self.dialog = None 30 | 31 | # Declare instance attributes 32 | self.actions = [] 33 | self.menu = u'&wpgpDatasets' 34 | self.toolbar = self.iface.addToolBar(u'wpgpDatasets') 35 | self.toolbar.setObjectName(u'wpgpDatasets') 36 | 37 | def add_action(self, icon_path: str, text: str, callback, enabled_flag: bool = True, 38 | add_to_menu: str = 'plugins', add_to_toolbar: bool = True, 39 | status_tip: str = None, whats_this: str = None, parent=None): 40 | 41 | """ 42 | Add a toolbar icon to the toolbar. 43 | 44 | :param icon_path: Path to the icon for this action. Can be a resource 45 | path (e.g. ':/plugins/foo/bar.png') or a normal file system path. 46 | :type icon_path: str 47 | 48 | :param text: Text that should be shown in menu items for this action. 49 | :type text: str 50 | 51 | :param callback: Function to be called when the action is triggered. 52 | :type callback: function 53 | 54 | :param enabled_flag: A flag indicating if the action should be enabled 55 | by default. Defaults to True. 56 | :type enabled_flag: bool 57 | 58 | :param add_to_menu: Flag indicating whether the action should also 59 | be added to the menu. Defaults to True. 60 | :type add_to_menu: bool 61 | 62 | :param add_to_toolbar: Flag indicating whether the action should also 63 | be added to the toolbar. Defaults to True. 64 | :type add_to_toolbar: bool 65 | 66 | :param status_tip: Optional text to show in a popup when mouse pointer 67 | hovers over the action. 68 | :type status_tip: str 69 | 70 | :param parent: Parent widget for the new action. Defaults None. 71 | :type parent: QWidget 72 | 73 | :param whats_this: Optional text to show in the status bar when the 74 | mouse pointer hovers over the action. 75 | 76 | :returns: The action that was created. Note that the action is also 77 | added to self.actions list. 78 | :rtype: QAction 79 | """ 80 | 81 | icon = QIcon(icon_path) 82 | action = QAction(icon, text, parent) 83 | action.triggered.connect(callback) 84 | action.setEnabled(enabled_flag) 85 | 86 | if status_tip is not None: 87 | action.setStatusTip(status_tip) 88 | 89 | if whats_this is not None: 90 | action.setWhatsThis(whats_this) 91 | 92 | if add_to_toolbar: 93 | self.toolbar.addAction(action) 94 | 95 | if add_to_menu: 96 | menu_action = None 97 | if add_to_menu == 'plugins': 98 | menu_action = self.iface.addPluginToMenu 99 | elif add_to_menu == 'rasters': 100 | menu_action = self.iface.addPluginToRasterMenu 101 | elif add_to_menu == 'vectors': 102 | menu_action = self.iface.addPluginToVectorMenu 103 | elif add_to_menu == 'web': 104 | menu_action = self.iface.addPluginToWebMenu 105 | if menu_action is None: 106 | raise AttributeError("add_to_menu should be a choice between plugins, rasters, vectors, web") 107 | 108 | menu_action(self.menu, action) 109 | 110 | self.actions.append(action) 111 | 112 | return action 113 | 114 | # noinspection PyPep8Naming 115 | def initGui(self): 116 | """Create the menu entries and toolbar icons inside the QGIS GUI.""" 117 | # Required 118 | 119 | icon_path = BASE_DIR / 'media' / 'wp.ico' 120 | self.add_action( 121 | icon_path.as_posix(), 122 | text=u'Download WorldPop Global Dataset', 123 | callback=self.run, 124 | parent=self.iface.mainWindow(), 125 | status_tip='Download WorldPop Global Dataset' 126 | ) 127 | 128 | def unload(self): 129 | """Removes the plugin menu item and icon from QGIS GUI.""" 130 | # Required 131 | for action in self.actions: 132 | self.iface.removePluginMenu(u'&wpgpDatasets', action) 133 | self.iface.removeToolBarIcon(action) 134 | # remove the toolbar 135 | del self.toolbar 136 | 137 | def run(self): 138 | """Run method that performs all the real work""" 139 | 140 | config = configparser.ConfigParser() 141 | config.read(INI_FILE.as_posix()) 142 | config['app']['csv_file_gz'] = CSV_FILE_GZ.as_posix() 143 | 144 | self.dialog = wpFactory(iface=self.iface, config=config) 145 | 146 | if not self.dialog: 147 | return 148 | 149 | self.dialog.show() 150 | -------------------------------------------------------------------------------- /wpgpDatasets/wp_datasets_dialog.py: -------------------------------------------------------------------------------- 1 | import configparser 2 | import platform 3 | from pathlib import Path 4 | from typing import Union 5 | 6 | from PyQt5 import QtCore, QtWidgets 7 | from PyQt5.QtGui import QIcon, QPixmap 8 | from PyQt5.QtWidgets import QFileDialog, QHeaderView, QMessageBox, QTreeWidgetItem 9 | 10 | from qgis.gui import QgisInterface 11 | 12 | from .lib import WpCsvParser 13 | from .lib.utils import qgis3_add_raster_to_project, get_default_download_directory, has_internet 14 | from .lib.about_window import Ui_AboutDialog 15 | from .lib.downloader import DownloadThread 16 | from .lib.main_window import Ui_wpMainWindow 17 | 18 | BASE_DIR = Path(__file__).parent 19 | 20 | UI_FILE = BASE_DIR / 'ui' / 'main_window.ui' 21 | assert UI_FILE.is_file() 22 | 23 | 24 | def wpFactory(config: configparser.ConfigParser, iface: QgisInterface, parent=None): 25 | if not has_internet(): 26 | QMessageBox().information(parent, 'No internet :(', 'This plugin requires internet to function.', 27 | QMessageBox.Ok) 28 | return 0 29 | 30 | from .lib.wpftp import wpFtp 31 | server = config['ftp']['server'] 32 | ftp = wpFtp(server=server, config=config) 33 | if ftp.newer_version_exists: 34 | resp = QMessageBox().question(parent, 'Newer CSV is available', 35 | 'A newer manifest file exists in the the FTP server.\n' 36 | 'This newer version contains information to list any new WorldPop productsn\n' 37 | 'It is recommend to update it for normal functionality.\n\n' 38 | 'Do this now?', 39 | buttons=QMessageBox.Yes | QMessageBox.No | QMessageBox.Cancel, 40 | defaultButton=QMessageBox.Yes 41 | ) 42 | if resp == QMessageBox.Yes: 43 | ftp.dl_wpgpDatasets() 44 | if resp == QMessageBox.No: 45 | pass 46 | if resp == QMessageBox.Cancel: 47 | return 0 48 | 49 | return WpMainWindow(config, iface, parent=None) 50 | 51 | 52 | class WpMainWindow(QtWidgets.QDialog, Ui_wpMainWindow): 53 | 54 | def __init__(self, config: configparser.ConfigParser, iface: QgisInterface, parent=None): 55 | super(WpMainWindow, self).__init__(parent=parent) 56 | self.iface = iface 57 | self.config = config 58 | csv_file_gz = config['app']['csv_file_gz'] 59 | self.csv_dataset = WpCsvParser(csv_file=csv_file_gz) 60 | 61 | self.setupUi(self) 62 | 63 | # Init stage 64 | self.setWindowTitle('World Pop Downloader') 65 | self._download_folder = None 66 | self.le_directory.setText(self._download_folder) 67 | self.le_directory.setPlaceholderText('/path/to/download/folder') 68 | 69 | self.pb_progressBar.setValue(0) 70 | icon = BASE_DIR / 'media' / 'wp.ico' 71 | self.setWindowIcon(QIcon(QPixmap(icon.as_posix()))) 72 | 73 | # TREE WIDGET 74 | self.tree_widget.setColumnCount(2) 75 | self.tree_widget.setHeaderLabels(['Name', 'Description']) 76 | self.tree_widget.setSortingEnabled(True) 77 | self.tree_widget.header().setResizeContentsPrecision(500) 78 | self.tree_widget.header().setSectionResizeMode(0, QHeaderView.ResizeToContents) 79 | if platform.system == 'Windows': 80 | self.tree_widget.header().resizeSections() 81 | 82 | # Adding data 83 | self._add_items() 84 | # and sort 85 | self.tree_widget.sortItems(0, QtCore.Qt.AscendingOrder) 86 | 87 | # Connections 88 | self.btn_close.clicked.connect(self.close) 89 | self.btn_about.clicked.connect(self._about_dialog) 90 | self.btn_browse.clicked.connect(self._file_dialog) 91 | self.btn_download.clicked.connect(self._download) 92 | 93 | @property 94 | def isos(self): 95 | isos = self.csv_dataset.isos 96 | return isos 97 | 98 | def _after_download(self, local_file: str): 99 | self.btn_download.setEnabled(True) 100 | self.btn_close.setEnabled(True) 101 | if self.cbox_add_to_layer.isChecked(): 102 | raster_path = Path(local_file) 103 | qgis3_add_raster_to_project(self.iface, raster_path=raster_path) 104 | 105 | def _download(self) -> Union[str, None]: 106 | 107 | # check if download folder is correct 108 | if self._download_folder is None: 109 | self._file_dialog() 110 | # user has selected 'Cancel' when instructed to select a folder 111 | if self._download_folder is None: 112 | return 113 | 114 | if not Path(self._download_folder).is_dir(): 115 | self._file_dialog() 116 | if not Path(self._download_folder).is_dir(): 117 | return 118 | 119 | # Grab and check it the URL exists from the TreeWidget, then pass it at the download function 120 | 121 | items = self.tree_widget.selectedItems() 122 | if len(items) == 0: 123 | return 124 | 125 | urls = [] 126 | for item in items: 127 | 128 | # 3d index of the item contains the ftp path of the object to download 129 | url = item.data(2, QtCore.Qt.DisplayRole) 130 | 131 | # Show warning that the user has not selected a valid selection. 132 | if url is None: 133 | QMessageBox.information(self, 'Invalid Selection', 'Please select any of the child products to download.', 134 | QMessageBox.Ok) 135 | return 136 | 137 | urls.append(url) 138 | 139 | self._do_download(urls) 140 | 141 | def _do_download(self, urls: Union[str, Path]): 142 | 143 | self.btn_download.setEnabled(False) 144 | self.btn_close.setEnabled(False) 145 | 146 | dl = DownloadThread(progress_bar=self.pb_progressBar, 147 | parent=self, server=self.config['ftp']['server']) 148 | dl.finished.connect(self._after_download) 149 | dl.run(urls=urls, download_folder=self._download_folder) 150 | 151 | def _add_items(self): 152 | for iso_idx, (iso_name, iso_iso3) in enumerate(self.isos): 153 | top_item = QTreeWidgetItem((iso_name, iso_iso3)) 154 | products_per_iso = self.csv_dataset.products_per_iso(iso_iso3) 155 | _ = [QTreeWidgetItem(x) for x in products_per_iso] 156 | top_item.addChildren(_) 157 | self.tree_widget.addTopLevelItem(top_item) 158 | 159 | def _about_dialog(self): 160 | text = self.config['app']['about_text'] or 'Text not found!' 161 | png = BASE_DIR / 'media' / 'logo.png' # BASE_DIR is Path object 162 | logo = QPixmap(png.as_posix(), "PNG") 163 | logo = logo.scaled(324, 186) # 10% of the original size 164 | 165 | class About(Ui_AboutDialog): 166 | def __init__(self, parent): 167 | super().__init__(parent) 168 | self.setupUi(self) 169 | self.gridLayout.setSizeConstraint(3) # set fixed size 170 | self.lbl_text.setWordWrap(True) 171 | 172 | about_dialog = About(self) 173 | about_dialog.lbl_text.setText(text) 174 | 175 | about_dialog.lbl_png.setText('') 176 | about_dialog.lbl_png.setPixmap(logo) 177 | 178 | # disable resizing 179 | about_dialog.setFixedSize(about_dialog.sizeHint()) 180 | 181 | about_dialog.btn_ok.clicked.connect(lambda x: about_dialog.close()) # close on click 182 | about_dialog.exec() 183 | 184 | def _file_dialog(self): 185 | caption = 'Please select a folder to download the product' 186 | default_root = self._download_folder 187 | if default_root is None: 188 | default_root = get_default_download_directory() 189 | 190 | if not Path(default_root).is_dir(): 191 | default_root = get_default_download_directory() 192 | 193 | # if user press cancel, it returns an Empty string 194 | dirname = QFileDialog().getExistingDirectory( 195 | self, caption=caption, directory=default_root, 196 | options=QFileDialog.ShowDirsOnly | QFileDialog.DontResolveSymlinks | QFileDialog.ReadOnly) 197 | 198 | if not dirname == '': 199 | self._download_folder = dirname 200 | 201 | self.le_directory.setText(dirname) 202 | --------------------------------------------------------------------------------