├── .gitignore ├── README.md ├── REST ├── AddFeaturesOnTimer.py └── README.md ├── build_org ├── README.md ├── clone_groups.ipynb ├── configure_org.ipynb ├── create_share_group.ipynb └── register_application.ipynb ├── common_workflows ├── README.md ├── csv_geocode.ipynb ├── distribute_items.ipynb ├── standard_geography.ipynb ├── update_webmaps.ipynb └── vector_data_products.ipynb ├── feature_layers ├── README.md ├── create_views.ipynb ├── csv_upload.ipynb ├── enable_time.ipynb ├── geojson_upload.ipynb ├── manage_fields.ipynb ├── manage_indexes.ipynb ├── shapefile_upload.ipynb └── update_data.ipynb ├── partnerutils ├── README.md ├── __init__.py ├── clone_utils.py ├── cool_utils.py ├── etl_utils.py ├── feature_utils.py ├── processing_utils.py └── user_utils.py ├── sample_data ├── NYC_Restaurant_Inspections.csv ├── NYC_Restaurant_Inspections.geojson └── sample_census_tract_geoid.csv └── setup.py /.gitignore: -------------------------------------------------------------------------------- 1 | # local 2 | client_server 3 | *DS_STORE 4 | *test.csv 5 | test/ 6 | test_data/ 7 | stage/ 8 | .vscode/ 9 | !.vscode/settings.json 10 | !.vscode/tasks.json 11 | !.vscode/launch.json 12 | !.vscode/extensions.json 13 | 14 | # Byte-compiled / optimized / DLL files 15 | __pycache__/ 16 | *.py[cod] 17 | *$py.class 18 | 19 | # C extensions 20 | *.so 21 | 22 | # Distribution / packaging 23 | .Python 24 | env/ 25 | build/ 26 | develop-eggs/ 27 | dist/ 28 | downloads/ 29 | eggs/ 30 | .eggs/ 31 | lib/ 32 | lib64/ 33 | parts/ 34 | sdist/ 35 | var/ 36 | wheels/ 37 | *.egg-info/ 38 | .installed.cfg 39 | *.egg 40 | 41 | # PyInstaller 42 | # Usually these files are written by a python script from a template 43 | # before PyInstaller builds the exe, so as to inject date/other infos into it. 44 | *.manifest 45 | *.spec 46 | 47 | # Installer logs 48 | pip-log.txt 49 | pip-delete-this-directory.txt 50 | 51 | # Unit test / coverage reports 52 | htmlcov/ 53 | .tox/ 54 | .coverage 55 | .coverage.* 56 | .cache 57 | nosetests.xml 58 | coverage.xml 59 | *.cover 60 | .hypothesis/ 61 | 62 | # Translations 63 | *.mo 64 | *.pot 65 | 66 | # Django stuff: 67 | *.log 68 | local_settings.py 69 | 70 | # Flask stuff: 71 | instance/ 72 | .webassets-cache 73 | 74 | # Scrapy stuff: 75 | .scrapy 76 | 77 | # Sphinx documentation 78 | docs/_build/ 79 | 80 | # PyBuilder 81 | target/ 82 | 83 | # Jupyter Notebook 84 | .ipynb_checkpoints 85 | 86 | # pyenv 87 | .python-version 88 | 89 | # celery beat schedule file 90 | celerybeat-schedule 91 | 92 | # SageMath parsed files 93 | *.sage.py 94 | 95 | # dotenv 96 | .env 97 | 98 | # virtualenv 99 | .venv 100 | venv/ 101 | ENV/ 102 | 103 | # Spyder project settings 104 | .spyderproject 105 | .spyproject 106 | 107 | # Rope project settings 108 | .ropeproject 109 | 110 | # mkdocs documentation 111 | /site 112 | 113 | # mypy 114 | .mypy_cache/ 115 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Esri Partner Tools 2 | 3 | > Useful tools for Esri Partners built with the [ArcGIS API for Python](https://developers.arcgis.com/python/) 4 | 5 | 6 | Contents 7 | 8 | 9 | * [About](#about) 10 | * [Prerequisites](#prerequisites) 11 | * [Contents](#contents) 12 | * [Getting Started](#getting-started) 13 | * [Sample Data](#sample-data) 14 | * [Issues and Contributing](#issues-and-contributing) 15 | 16 | 17 | 18 | ## About 19 | 20 | Partners working with Esri and ArcGIS implement many common workflows. The [ArcGIS API for Python](https://developers.arcgis.com/python/) is an awesome automation library. This repo is meant to be a collection of POC scripts to automate some of these workflows. 21 | 22 | While much of the code is in Jupyter Notebooks, it can easily be ported to pure python to run on the server or as headless apps. [`partnerutils/`](/partnerutils) can also be installed as a local package: 23 | > `$ pip install -q -U git+https://github.com/mpayson/esri-partner-tools` 24 | 25 | ## Prerequisites 26 | 27 | * Install the [ArcGIS API for Python](https://developers.arcgis.com/python/) ([instructions](https://developers.arcgis.com/python/guide/install-and-set-up/)) 28 | * Access to [Jupyter Notebooks](http://jupyter.org/) 29 | 30 | ## Contents 31 | 32 | * **[`partnerutils/`](/partnerutils) - Functions that I've found helpful** 33 | * [`cool_utils.py`](/partnerutils/cool_utils.py) - functions I want to remember and hopefully you will too! 34 | * [`etl_utils.py`](/partnerutils/etl_utils.py) - assist with common ETL logic 35 | * [`user_utils.py`](/partnerutils/user_utils.py) - assist with adding users 36 | * [`clone_utils.py`](/partnerutils/clone_utils.py) - assist with cloning groups & items 37 | * [`feature_utils.py`](partnerutils/feature_utils.py) - assist with features and feature data types 38 | * **[`common_workflows/`](/common_workflows) - Common workflows with the Python API** 39 | * [`csv_geocode.ipynb`](/common_workflows/csv_geocode.ipynb) - [geocode](https://developers.arcgis.com/features/geocoding/) rows in `csvs` and `dataframes` 40 | * [`vector_data_products.ipynb`](/common_workflows/vector_data_products.ipynb) - end-to-end workflows for managing vector content and derivative information products 41 | * [`distribute_items.ipynb`](/common_workflows/distribute_items.ipynb) - common patterns for distributing items to another organization 42 | * [`standard_geography.ipynb`](/common_workflows/standard_geography.ipynb) - enrich [standard geography](https://developers.arcgis.com/rest/geoenrichment/api-reference/standard-geography-query.htm) ids, such as `census blocks`, with geometries 43 | * **[`feature_layers/`](/feature_layers) - Common operations with [hosted feature layers](https://doc.arcgis.com/en/arcgis-online/share-maps/hosted-web-layers.htm)** 44 | * [`csv_upload.ipynb`](/feature_layers/csv_upload.ipynb) - upload a folder of `csvs` & `dataframes` 45 | * [`shapefile_upload.ipynb`](/feature_layers/shapefile_upload.ipynb) - upload a folder of `Shapefiles` 46 | * [`geojson_upload.ipynb`](/feature_layers/geojson_upload.ipynb) - upload a geojson file 47 | * [`update_data.ipynb`](/feature_layers/update_data.ipynb) - a couple different workflows for updating uploaded / hosted data 48 | * [`create_views.ipynb`](/feature_layers/create_views.ipynb) - create database views with separate permissions against one authoritative layer 49 | * [`manage_fields.ipynb`](/feature_layers/manage_fields.ipynb) - view and edit fields 50 | * [`manage_indexes.ipynb`](/feature_layers/manage_indexes.ipynb) - view, edit, and refresh indexes 51 | * [`enable_time.ipynb`](/feature_layers/enable_time.ipynb) - add time metadata that will be reflected in ArcGIS app UIs 52 | * **[`build_org/`](/build_org) - Automate new ArcGIS Online deployments** 53 | * [`clone_groups.ipynb`](/build_org/clone_groups.ipynb) - clone groups and their items 54 | * [`configure_org.ipynb`](/build_org/configure_org.ipynb) - customize org UI, create groups, & add users 55 | * [`create_share_group.ipynb`](/build_org/create_share_group.ipynb) - create a [group](https://doc.arcgis.com/en/arcgis-online/share-maps/groups.htm) and invite members to share content with your users 56 | * [`register_application.ipynb`](/build_org/register_application.ipynb) automatically create and [register an app](https://developers.arcgis.com/documentation/core-concepts/security-and-authentication/signing-in-arcgis-online-users/) 57 | 58 | ## Getting Started 59 | 60 | Many samples use [`partnerutils`](/partnerutils). To use this package, either copy & paste the functions as specified in each notebook OR: 61 | 62 | `$ pip install -q -U git+https://github.com/mpayson/esri-partner-tools` 63 | 64 | This will install the `partnerutils` as a local package in your active environment. The utilities can then be used as follows 65 | 66 | ```python 67 | from arcgis.gis import GIS 68 | from partnerutils.processing_utils import batch_geocode_memo 69 | 70 | gis = GIS(username="username", password="password") 71 | addresses = ['El Burrito Redlands CA', '380 New York St Redlands CA'] 72 | results = batch_geocode_memo(addresses) 73 | 74 | print(results) 75 | ``` 76 | 77 | Shout out to Ryan @ SafeGraph for showing me this is [a thing](https://github.com/SafeGraphInc/safegraph_py). Otherwise, the notebooks should give enough detail to get started. If not, **[holler](https://github.com/mpayson/esri-partner-tools/issues)**! 78 | 79 | ## Sample Data 80 | 81 | I included some sample data for testing and trialing: 82 | * [`NYC_Restaurant_Inspections.csv`](/sample_data/NYC_Restaurant_Inspections.csv) - a slice of DOHMH New York City Restaurant Inspection Results. [Source](https://data.cityofnewyork.us/Health/DOHMH-New-York-City-Restaurant-Inspection-Results/43nn-pn8j]) 83 | * [`sample_census_tract_geoid.csv`](/sample_data/sample_census_tract_geoid.csv) - a couple census tract geoids. Copied from [here](https://geo.nyu.edu/catalog/nyu-2451-34513) 84 | 85 | ## Issues and Contributing 86 | 87 | Want to request a new sample? Have a question? Would [__love__](https://github.com/mpayson/esri-partner-tools/issues) to hear from you. 88 | 89 | And PRs always welcome! 90 | -------------------------------------------------------------------------------- /REST/AddFeaturesOnTimer.py: -------------------------------------------------------------------------------- 1 | import threading 2 | import sys 3 | import requests 4 | import json 5 | from datetime import datetime 6 | 7 | 8 | # Disable warnings 9 | requests.packages.urllib3.disable_warnings() 10 | 11 | # Get Parameters 12 | addfsURL = str(sys.argv[1]) 13 | interval = int(sys.argv[2]) 14 | deviceID = int(sys.argv[3]) 15 | quantity = int(sys.argv[4]) 16 | username = str(sys.argv[5]) 17 | password = str(sys.argv[6]) 18 | 19 | # Generate Token 20 | tokenURL = 'https://www.arcgis.com/sharing/rest/generateToken' 21 | params = {'f': 'pjson', 'username': username, 'password': password, 'referer': 'http://www.arcgis.com'} 22 | r = requests.post(tokenURL, data = params, verify=False) 23 | response = json.loads(r.content) 24 | token = response['token'] 25 | 26 | # define timer 27 | def fireTimer(): 28 | threading.Timer(interval, fireTimer).start() 29 | 30 | #get timestamp 31 | noww = datetime.now() 32 | dateTimeNow = str(noww) 33 | 34 | attr = [{"attributes":{"pk":deviceID, "amount":quantity, "datetime1":dateTimeNow}}] 35 | params = {"features": json.dumps(attr), 'token': token, 'f': 'json'} 36 | r = requests.post(addfsURL, data = params, verify=False) 37 | 38 | print(r.json) 39 | 40 | fireTimer() 41 | -------------------------------------------------------------------------------- /REST/README.md: -------------------------------------------------------------------------------- 1 | # ArcGISAddFeatures 2 | Headless ArcGIS python script to push attribute data into an ArcGIS Online hosted data table. The script runs on a timer at a user defined interval. 3 | 4 | Prerequisites: 5 | ArcGIS Online account 6 | Hosted data table with fields 'pk', 'amount', 'datetime1' 7 | 8 | Usage: 9 | * AddFeaturesOnTimer.py 'URL of the hosted table' 'interval (sec)' 'ID' 'Value' 'My User Name' 'My Password' 10 | -------------------------------------------------------------------------------- /build_org/README.md: -------------------------------------------------------------------------------- 1 | # Build Org 2 | 3 | > Automate new ArcGIS Online deployments 4 | 5 | * [`clone_groups.ipynb`](/build_org/clone_groups.ipynb) - clone groups and their items 6 | * [`configure_org.ipynb`](/build_org/configure_org.ipynb) - customize org UI, create groups, & add users 7 | * [`create_share_group.ipynb`](/build_org/create_share_group.ipynb) - create a [group](https://doc.arcgis.com/en/arcgis-online/share-maps/groups.htm) and invite members to share content with your users 8 | * [`register_application.ipynb`](/build_org/register_application.ipynb) automatically create and [register an app](https://developers.arcgis.com/documentation/core-concepts/security-and-authentication/signing-in-arcgis-online-users/) -------------------------------------------------------------------------------- /build_org/clone_groups.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Clone Groups\n", 8 | "*Jupyter Notebook to clone groups and their items. Configurable notebook over this awesome [`clone_items`](https://esri.github.io/arcgis-python-api/apidoc/html/arcgis.gis.toc.html#arcgis.gis.ContentManager.clone_items) function*" 9 | ] 10 | }, 11 | { 12 | "cell_type": "code", 13 | "execution_count": null, 14 | "metadata": { 15 | "collapsed": true, 16 | "jupyter": { 17 | "outputs_hidden": true 18 | } 19 | }, 20 | "outputs": [], 21 | "source": [ 22 | "# common imports\n", 23 | "from arcgis.gis import GIS" 24 | ] 25 | }, 26 | { 27 | "cell_type": "markdown", 28 | "metadata": {}, 29 | "source": [ 30 | "***Note**, if you are unable to import local `partnerutils`, **copy the following functions** from [`clone_utils`](https://github.com/mpayson/esri-partner-tools/blob/master/partnerutils/clone_utils.py)" 31 | ] 32 | }, 33 | { 34 | "cell_type": "code", 35 | "execution_count": null, 36 | "metadata": { 37 | "collapsed": true, 38 | "jupyter": { 39 | "outputs_hidden": true 40 | } 41 | }, 42 | "outputs": [], 43 | "source": [ 44 | "from partnerutils.clone_utils import search_group_title, search_item_title, clone_items_modify" 45 | ] 46 | }, 47 | { 48 | "cell_type": "markdown", 49 | "metadata": {}, 50 | "source": [ 51 | "## User Input" 52 | ] 53 | }, 54 | { 55 | "cell_type": "markdown", 56 | "metadata": {}, 57 | "source": [ 58 | "### GIS Configuration\n", 59 | "Parameter information [here](https://developers.arcgis.com/python/guide/using-the-gis/).\n", 60 | "* **source** - the ArcGIS Online organization that contains the default groups & items\n", 61 | "* **target** - the new organization" 62 | ] 63 | }, 64 | { 65 | "cell_type": "code", 66 | "execution_count": null, 67 | "metadata": { 68 | "collapsed": true, 69 | "jupyter": { 70 | "outputs_hidden": true 71 | } 72 | }, 73 | "outputs": [], 74 | "source": [ 75 | "# log in to respective portals\n", 76 | "source = GIS(\"\", \"\", \"\")\n", 77 | "target = GIS(\"\", \"\", \"\")" 78 | ] 79 | }, 80 | { 81 | "cell_type": "markdown", 82 | "metadata": {}, 83 | "source": [ 84 | "### Clone Configuration\n", 85 | "Defines the groups that will be cloned and the naming structures of cloned items & groups" 86 | ] 87 | }, 88 | { 89 | "cell_type": "code", 90 | "execution_count": null, 91 | "metadata": { 92 | "collapsed": true, 93 | "jupyter": { 94 | "outputs_hidden": true 95 | } 96 | }, 97 | "outputs": [], 98 | "source": [ 99 | "# groups to be copied from source to target organization\n", 100 | "GROUPS = [\"\", \"\"]\n", 101 | "\n", 102 | "# (optional) new item name structure, {} replaced by source item name\n", 103 | "NAME_TEMPLATE = \" {}\"\n", 104 | "# (optional) user folder for new items\n", 105 | "FOLDER = \" Content\" " 106 | ] 107 | }, 108 | { 109 | "cell_type": "markdown", 110 | "metadata": {}, 111 | "source": [ 112 | "## Execution" 113 | ] 114 | }, 115 | { 116 | "cell_type": "markdown", 117 | "metadata": {}, 118 | "source": [ 119 | "### Modify Functions\n", 120 | "Sometimes you don't want a 1:1 copy, these functions will modify the cloned item properties. A few notes:\n", 121 | "* `modify_item_callback` receives the cloned item and its gis. It should return a flattened dict of properties [here](https://esri.github.io/arcgis-python-api/apidoc/html/arcgis.gis.toc.html?highlight=clone_items#arcgis.gis.Item.update)\n", 122 | "* `modify_group_callback` receives the cloned group, its expected title, and its gis. It should return a dict of properties [here](https://esri.github.io/arcgis-python-api/apidoc/html/arcgis.gis.toc.html?highlight=clone_items#arcgis.gis.Group.update)\n", 123 | "* The default behavior is to update the item to match the NAME_TEMPLATE, if it's not defined these will not be called" 124 | ] 125 | }, 126 | { 127 | "cell_type": "code", 128 | "execution_count": null, 129 | "metadata": { 130 | "collapsed": true, 131 | "jupyter": { 132 | "outputs_hidden": true 133 | } 134 | }, 135 | "outputs": [], 136 | "source": [ 137 | "# update title functions\n", 138 | "def modify_item_callback(item, target_gis):\n", 139 | " title = NAME_TEMPLATE.format(item.title)\n", 140 | " while search_item_title(target_gis, title):\n", 141 | " title = input(\"Title `{0}` for ITEM `{1}` already exists \\n new title: \"\n", 142 | " .format(title, item.title))\n", 143 | " return {\"title\": title}\n", 144 | "def modify_group_callback(group, expected_title, target_gis):\n", 145 | " title = NAME_TEMPLATE.format(expected_title)\n", 146 | " while search_group_title(target_gis, title):\n", 147 | " title = input(\"Title `{0}` for GROUP `{1}` already exists \\n new title: \"\n", 148 | " .format(title, expected_title))\n", 149 | " return {\"title\": title}\n", 150 | "\n", 151 | "# these functions provide little utility if NAME_TEMPLATE is None\n", 152 | "# so don't use if this is the case\n", 153 | "modify_item = modify_item_callback if NAME_TEMPLATE else None\n", 154 | "modify_group = modify_group_callback if NAME_TEMPLATE else None" 155 | ] 156 | }, 157 | { 158 | "cell_type": "markdown", 159 | "metadata": {}, 160 | "source": [ 161 | "### Clone" 162 | ] 163 | }, 164 | { 165 | "cell_type": "code", 166 | "execution_count": null, 167 | "metadata": { 168 | "collapsed": true, 169 | "jupyter": { 170 | "outputs_hidden": true 171 | } 172 | }, 173 | "outputs": [], 174 | "source": [ 175 | "# get groups\n", 176 | "source_groups = [search_group_title(source, title) for title in GROUPS]\n", 177 | "\n", 178 | "# copy the groups\n", 179 | "results = clone_items_modify(source_groups, target,\n", 180 | " modify_item_callback=modify_item,\n", 181 | " modify_group_callback=modify_group,\n", 182 | " copy_data=False, search_existing_items=False, folder=FOLDER)" 183 | ] 184 | } 185 | ], 186 | "metadata": { 187 | "kernelspec": { 188 | "display_name": "Python 3", 189 | "language": "python", 190 | "name": "python3" 191 | }, 192 | "language_info": { 193 | "codemirror_mode": { 194 | "name": "ipython", 195 | "version": 3 196 | }, 197 | "file_extension": ".py", 198 | "mimetype": "text/x-python", 199 | "name": "python", 200 | "nbconvert_exporter": "python", 201 | "pygments_lexer": "ipython3", 202 | "version": "3.7.4" 203 | } 204 | }, 205 | "nbformat": 4, 206 | "nbformat_minor": 4 207 | } 208 | -------------------------------------------------------------------------------- /build_org/configure_org.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Configure Org\n", 8 | "*Jupyter Notebook to customize an ArcGIS Online organization UI & add users*" 9 | ] 10 | }, 11 | { 12 | "cell_type": "code", 13 | "execution_count": null, 14 | "metadata": { 15 | "collapsed": true 16 | }, 17 | "outputs": [], 18 | "source": [ 19 | "# common imports\n", 20 | "from arcgis.gis import GIS" 21 | ] 22 | }, 23 | { 24 | "cell_type": "markdown", 25 | "metadata": {}, 26 | "source": [ 27 | "***Note**, if you are unable to import local `partnerutils`, **copy the following functions** from [`user_utils`](https://github.com/mpayson/esri-partner-tools/blob/master/partnerutils/user_utils.py) and [`clone_utils`](https://github.com/mpayson/esri-partner-tools/blob/master/partnerutils/clone_utils.py)" 28 | ] 29 | }, 30 | { 31 | "cell_type": "code", 32 | "execution_count": 1, 33 | "metadata": { 34 | "collapsed": true 35 | }, 36 | "outputs": [], 37 | "source": [ 38 | "from partnerutils.user_utils import add_users_csv\n", 39 | "from partnerutils.clone_utils import search_group_title" 40 | ] 41 | }, 42 | { 43 | "cell_type": "markdown", 44 | "metadata": {}, 45 | "source": [ 46 | "## User Input" 47 | ] 48 | }, 49 | { 50 | "cell_type": "markdown", 51 | "metadata": {}, 52 | "source": [ 53 | "### GIS Configuration\n", 54 | "Organization to be built. Parameter information [here](https://developers.arcgis.com/python/guide/using-the-gis/)." 55 | ] 56 | }, 57 | { 58 | "cell_type": "code", 59 | "execution_count": null, 60 | "metadata": { 61 | "collapsed": true 62 | }, 63 | "outputs": [], 64 | "source": [ 65 | "# org to be customized\n", 66 | "gis = GIS(\"\", \"\", \"\")" 67 | ] 68 | }, 69 | { 70 | "cell_type": "markdown", 71 | "metadata": {}, 72 | "source": [ 73 | "### UI Configuration\n", 74 | "Constants to define organization users and UI building blocks. This script will also search for groups and create them if they don't exist, as defined by the [schema](https://esri.github.io/arcgis-python-api/apidoc/html/arcgis.gis.toc.html#arcgis.gis.GroupManager.create)." 75 | ] 76 | }, 77 | { 78 | "cell_type": "code", 79 | "execution_count": null, 80 | "metadata": { 81 | "collapsed": true 82 | }, 83 | "outputs": [], 84 | "source": [ 85 | "# path to csv with users to be added\n", 86 | "USER_CSV = \"\"\n", 87 | "\n", 88 | "# (optional) groups to share with users\n", 89 | "GROUPS = [\"\", \"\"]\n", 90 | "\n", 91 | "# (optional) group schema to create new groups if GROUPS don't exist\n", 92 | "GROUPSCHEMA = {\n", 93 | " \"tags\": \"test, group, poc, scripts\",\n", 94 | " \"description\": \"Test group for partner python scripts\",\n", 95 | " \"access\": 'private',\n", 96 | " \"is_invitation_only\": True,\n", 97 | " \"users_update_items\": False\n", 98 | "}\n", 99 | "\n", 100 | "# (optional) organization UX component locations\n", 101 | "THUMBNAIL_PATH = \"\"\n", 102 | "FOOTER_PATH = \"