├── .gitignore ├── README.md ├── REST ├── AddFeaturesOnTimer.py └── README.md ├── build_org ├── README.md ├── clone_groups.ipynb ├── configure_org.ipynb ├── create_share_group.ipynb └── register_application.ipynb ├── common_workflows ├── README.md ├── csv_geocode.ipynb ├── distribute_items.ipynb ├── standard_geography.ipynb ├── update_webmaps.ipynb └── vector_data_products.ipynb ├── feature_layers ├── README.md ├── create_views.ipynb ├── csv_upload.ipynb ├── enable_time.ipynb ├── geojson_upload.ipynb ├── manage_fields.ipynb ├── manage_indexes.ipynb ├── shapefile_upload.ipynb └── update_data.ipynb ├── partnerutils ├── README.md ├── __init__.py ├── clone_utils.py ├── cool_utils.py ├── etl_utils.py ├── feature_utils.py ├── processing_utils.py └── user_utils.py ├── sample_data ├── NYC_Restaurant_Inspections.csv ├── NYC_Restaurant_Inspections.geojson └── sample_census_tract_geoid.csv └── setup.py /.gitignore: -------------------------------------------------------------------------------- 1 | # local 2 | client_server 3 | *DS_STORE 4 | *test.csv 5 | test/ 6 | test_data/ 7 | stage/ 8 | .vscode/ 9 | !.vscode/settings.json 10 | !.vscode/tasks.json 11 | !.vscode/launch.json 12 | !.vscode/extensions.json 13 | 14 | # Byte-compiled / optimized / DLL files 15 | __pycache__/ 16 | *.py[cod] 17 | *$py.class 18 | 19 | # C extensions 20 | *.so 21 | 22 | # Distribution / packaging 23 | .Python 24 | env/ 25 | build/ 26 | develop-eggs/ 27 | dist/ 28 | downloads/ 29 | eggs/ 30 | .eggs/ 31 | lib/ 32 | lib64/ 33 | parts/ 34 | sdist/ 35 | var/ 36 | wheels/ 37 | *.egg-info/ 38 | .installed.cfg 39 | *.egg 40 | 41 | # PyInstaller 42 | # Usually these files are written by a python script from a template 43 | # before PyInstaller builds the exe, so as to inject date/other infos into it. 44 | *.manifest 45 | *.spec 46 | 47 | # Installer logs 48 | pip-log.txt 49 | pip-delete-this-directory.txt 50 | 51 | # Unit test / coverage reports 52 | htmlcov/ 53 | .tox/ 54 | .coverage 55 | .coverage.* 56 | .cache 57 | nosetests.xml 58 | coverage.xml 59 | *.cover 60 | .hypothesis/ 61 | 62 | # Translations 63 | *.mo 64 | *.pot 65 | 66 | # Django stuff: 67 | *.log 68 | local_settings.py 69 | 70 | # Flask stuff: 71 | instance/ 72 | .webassets-cache 73 | 74 | # Scrapy stuff: 75 | .scrapy 76 | 77 | # Sphinx documentation 78 | docs/_build/ 79 | 80 | # PyBuilder 81 | target/ 82 | 83 | # Jupyter Notebook 84 | .ipynb_checkpoints 85 | 86 | # pyenv 87 | .python-version 88 | 89 | # celery beat schedule file 90 | celerybeat-schedule 91 | 92 | # SageMath parsed files 93 | *.sage.py 94 | 95 | # dotenv 96 | .env 97 | 98 | # virtualenv 99 | .venv 100 | venv/ 101 | ENV/ 102 | 103 | # Spyder project settings 104 | .spyderproject 105 | .spyproject 106 | 107 | # Rope project settings 108 | .ropeproject 109 | 110 | # mkdocs documentation 111 | /site 112 | 113 | # mypy 114 | .mypy_cache/ 115 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Esri Partner Tools 2 | 3 | > Useful tools for Esri Partners built with the [ArcGIS API for Python](https://developers.arcgis.com/python/) 4 | 5 |
6 | Contents 7 | 8 | 9 | * [About](#about) 10 | * [Prerequisites](#prerequisites) 11 | * [Contents](#contents) 12 | * [Getting Started](#getting-started) 13 | * [Sample Data](#sample-data) 14 | * [Issues and Contributing](#issues-and-contributing) 15 | 16 |
17 | 18 | ## About 19 | 20 | Partners working with Esri and ArcGIS implement many common workflows. The [ArcGIS API for Python](https://developers.arcgis.com/python/) is an awesome automation library. This repo is meant to be a collection of POC scripts to automate some of these workflows. 21 | 22 | While much of the code is in Jupyter Notebooks, it can easily be ported to pure python to run on the server or as headless apps. [`partnerutils/`](/partnerutils) can also be installed as a local package: 23 | > `$ pip install -q -U git+https://github.com/mpayson/esri-partner-tools` 24 | 25 | ## Prerequisites 26 | 27 | * Install the [ArcGIS API for Python](https://developers.arcgis.com/python/) ([instructions](https://developers.arcgis.com/python/guide/install-and-set-up/)) 28 | * Access to [Jupyter Notebooks](http://jupyter.org/) 29 | 30 | ## Contents 31 | 32 | * **[`partnerutils/`](/partnerutils) - Functions that I've found helpful** 33 | * [`cool_utils.py`](/partnerutils/cool_utils.py) - functions I want to remember and hopefully you will too! 34 | * [`etl_utils.py`](/partnerutils/etl_utils.py) - assist with common ETL logic 35 | * [`user_utils.py`](/partnerutils/user_utils.py) - assist with adding users 36 | * [`clone_utils.py`](/partnerutils/clone_utils.py) - assist with cloning groups & items 37 | * [`feature_utils.py`](partnerutils/feature_utils.py) - assist with features and feature data types 38 | * **[`common_workflows/`](/common_workflows) - Common workflows with the Python API** 39 | * [`csv_geocode.ipynb`](/common_workflows/csv_geocode.ipynb) - [geocode](https://developers.arcgis.com/features/geocoding/) rows in `csvs` and `dataframes` 40 | * [`vector_data_products.ipynb`](/common_workflows/vector_data_products.ipynb) - end-to-end workflows for managing vector content and derivative information products 41 | * [`distribute_items.ipynb`](/common_workflows/distribute_items.ipynb) - common patterns for distributing items to another organization 42 | * [`standard_geography.ipynb`](/common_workflows/standard_geography.ipynb) - enrich [standard geography](https://developers.arcgis.com/rest/geoenrichment/api-reference/standard-geography-query.htm) ids, such as `census blocks`, with geometries 43 | * **[`feature_layers/`](/feature_layers) - Common operations with [hosted feature layers](https://doc.arcgis.com/en/arcgis-online/share-maps/hosted-web-layers.htm)** 44 | * [`csv_upload.ipynb`](/feature_layers/csv_upload.ipynb) - upload a folder of `csvs` & `dataframes` 45 | * [`shapefile_upload.ipynb`](/feature_layers/shapefile_upload.ipynb) - upload a folder of `Shapefiles` 46 | * [`geojson_upload.ipynb`](/feature_layers/geojson_upload.ipynb) - upload a geojson file 47 | * [`update_data.ipynb`](/feature_layers/update_data.ipynb) - a couple different workflows for updating uploaded / hosted data 48 | * [`create_views.ipynb`](/feature_layers/create_views.ipynb) - create database views with separate permissions against one authoritative layer 49 | * [`manage_fields.ipynb`](/feature_layers/manage_fields.ipynb) - view and edit fields 50 | * [`manage_indexes.ipynb`](/feature_layers/manage_indexes.ipynb) - view, edit, and refresh indexes 51 | * [`enable_time.ipynb`](/feature_layers/enable_time.ipynb) - add time metadata that will be reflected in ArcGIS app UIs 52 | * **[`build_org/`](/build_org) - Automate new ArcGIS Online deployments** 53 | * [`clone_groups.ipynb`](/build_org/clone_groups.ipynb) - clone groups and their items 54 | * [`configure_org.ipynb`](/build_org/configure_org.ipynb) - customize org UI, create groups, & add users 55 | * [`create_share_group.ipynb`](/build_org/create_share_group.ipynb) - create a [group](https://doc.arcgis.com/en/arcgis-online/share-maps/groups.htm) and invite members to share content with your users 56 | * [`register_application.ipynb`](/build_org/register_application.ipynb) automatically create and [register an app](https://developers.arcgis.com/documentation/core-concepts/security-and-authentication/signing-in-arcgis-online-users/) 57 | 58 | ## Getting Started 59 | 60 | Many samples use [`partnerutils`](/partnerutils). To use this package, either copy & paste the functions as specified in each notebook OR: 61 | 62 | `$ pip install -q -U git+https://github.com/mpayson/esri-partner-tools` 63 | 64 | This will install the `partnerutils` as a local package in your active environment. The utilities can then be used as follows 65 | 66 | ```python 67 | from arcgis.gis import GIS 68 | from partnerutils.processing_utils import batch_geocode_memo 69 | 70 | gis = GIS(username="username", password="password") 71 | addresses = ['El Burrito Redlands CA', '380 New York St Redlands CA'] 72 | results = batch_geocode_memo(addresses) 73 | 74 | print(results) 75 | ``` 76 | 77 | Shout out to Ryan @ SafeGraph for showing me this is [a thing](https://github.com/SafeGraphInc/safegraph_py). Otherwise, the notebooks should give enough detail to get started. If not, **[holler](https://github.com/mpayson/esri-partner-tools/issues)**! 78 | 79 | ## Sample Data 80 | 81 | I included some sample data for testing and trialing: 82 | * [`NYC_Restaurant_Inspections.csv`](/sample_data/NYC_Restaurant_Inspections.csv) - a slice of DOHMH New York City Restaurant Inspection Results. [Source](https://data.cityofnewyork.us/Health/DOHMH-New-York-City-Restaurant-Inspection-Results/43nn-pn8j]) 83 | * [`sample_census_tract_geoid.csv`](/sample_data/sample_census_tract_geoid.csv) - a couple census tract geoids. Copied from [here](https://geo.nyu.edu/catalog/nyu-2451-34513) 84 | 85 | ## Issues and Contributing 86 | 87 | Want to request a new sample? Have a question? Would [__love__](https://github.com/mpayson/esri-partner-tools/issues) to hear from you. 88 | 89 | And PRs always welcome! 90 | -------------------------------------------------------------------------------- /REST/AddFeaturesOnTimer.py: -------------------------------------------------------------------------------- 1 | import threading 2 | import sys 3 | import requests 4 | import json 5 | from datetime import datetime 6 | 7 | 8 | # Disable warnings 9 | requests.packages.urllib3.disable_warnings() 10 | 11 | # Get Parameters 12 | addfsURL = str(sys.argv[1]) 13 | interval = int(sys.argv[2]) 14 | deviceID = int(sys.argv[3]) 15 | quantity = int(sys.argv[4]) 16 | username = str(sys.argv[5]) 17 | password = str(sys.argv[6]) 18 | 19 | # Generate Token 20 | tokenURL = 'https://www.arcgis.com/sharing/rest/generateToken' 21 | params = {'f': 'pjson', 'username': username, 'password': password, 'referer': 'http://www.arcgis.com'} 22 | r = requests.post(tokenURL, data = params, verify=False) 23 | response = json.loads(r.content) 24 | token = response['token'] 25 | 26 | # define timer 27 | def fireTimer(): 28 | threading.Timer(interval, fireTimer).start() 29 | 30 | #get timestamp 31 | noww = datetime.now() 32 | dateTimeNow = str(noww) 33 | 34 | attr = [{"attributes":{"pk":deviceID, "amount":quantity, "datetime1":dateTimeNow}}] 35 | params = {"features": json.dumps(attr), 'token': token, 'f': 'json'} 36 | r = requests.post(addfsURL, data = params, verify=False) 37 | 38 | print(r.json) 39 | 40 | fireTimer() 41 | -------------------------------------------------------------------------------- /REST/README.md: -------------------------------------------------------------------------------- 1 | # ArcGISAddFeatures 2 | Headless ArcGIS python script to push attribute data into an ArcGIS Online hosted data table. The script runs on a timer at a user defined interval. 3 | 4 | Prerequisites: 5 | ArcGIS Online account 6 | Hosted data table with fields 'pk', 'amount', 'datetime1' 7 | 8 | Usage: 9 | * AddFeaturesOnTimer.py 'URL of the hosted table' 'interval (sec)' 'ID' 'Value' 'My User Name' 'My Password' 10 | -------------------------------------------------------------------------------- /build_org/README.md: -------------------------------------------------------------------------------- 1 | # Build Org 2 | 3 | > Automate new ArcGIS Online deployments 4 | 5 | * [`clone_groups.ipynb`](/build_org/clone_groups.ipynb) - clone groups and their items 6 | * [`configure_org.ipynb`](/build_org/configure_org.ipynb) - customize org UI, create groups, & add users 7 | * [`create_share_group.ipynb`](/build_org/create_share_group.ipynb) - create a [group](https://doc.arcgis.com/en/arcgis-online/share-maps/groups.htm) and invite members to share content with your users 8 | * [`register_application.ipynb`](/build_org/register_application.ipynb) automatically create and [register an app](https://developers.arcgis.com/documentation/core-concepts/security-and-authentication/signing-in-arcgis-online-users/) -------------------------------------------------------------------------------- /build_org/clone_groups.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Clone Groups\n", 8 | "*Jupyter Notebook to clone groups and their items. Configurable notebook over this awesome [`clone_items`](https://esri.github.io/arcgis-python-api/apidoc/html/arcgis.gis.toc.html#arcgis.gis.ContentManager.clone_items) function*" 9 | ] 10 | }, 11 | { 12 | "cell_type": "code", 13 | "execution_count": null, 14 | "metadata": { 15 | "collapsed": true, 16 | "jupyter": { 17 | "outputs_hidden": true 18 | } 19 | }, 20 | "outputs": [], 21 | "source": [ 22 | "# common imports\n", 23 | "from arcgis.gis import GIS" 24 | ] 25 | }, 26 | { 27 | "cell_type": "markdown", 28 | "metadata": {}, 29 | "source": [ 30 | "***Note**, if you are unable to import local `partnerutils`, **copy the following functions** from [`clone_utils`](https://github.com/mpayson/esri-partner-tools/blob/master/partnerutils/clone_utils.py)" 31 | ] 32 | }, 33 | { 34 | "cell_type": "code", 35 | "execution_count": null, 36 | "metadata": { 37 | "collapsed": true, 38 | "jupyter": { 39 | "outputs_hidden": true 40 | } 41 | }, 42 | "outputs": [], 43 | "source": [ 44 | "from partnerutils.clone_utils import search_group_title, search_item_title, clone_items_modify" 45 | ] 46 | }, 47 | { 48 | "cell_type": "markdown", 49 | "metadata": {}, 50 | "source": [ 51 | "## User Input" 52 | ] 53 | }, 54 | { 55 | "cell_type": "markdown", 56 | "metadata": {}, 57 | "source": [ 58 | "### GIS Configuration\n", 59 | "Parameter information [here](https://developers.arcgis.com/python/guide/using-the-gis/).\n", 60 | "* **source** - the ArcGIS Online organization that contains the default groups & items\n", 61 | "* **target** - the new organization" 62 | ] 63 | }, 64 | { 65 | "cell_type": "code", 66 | "execution_count": null, 67 | "metadata": { 68 | "collapsed": true, 69 | "jupyter": { 70 | "outputs_hidden": true 71 | } 72 | }, 73 | "outputs": [], 74 | "source": [ 75 | "# log in to respective portals\n", 76 | "source = GIS(\"