├── .gitignore ├── .DS_Store ├── open_datasets ├── dob_permits │ ├── copy.sql │ ├── after.sql │ ├── data.json │ └── create.sql ├── .DS_Store ├── dcp_censustracts │ ├── .DS_Store │ ├── nyct2010wi_16d.zip │ └── nyct2010wi_16d │ │ ├── nyct2010wi.dbf │ │ ├── nyct2010wi.shp │ │ ├── nyct2010wi.shx │ │ ├── nyct2010wi.prj │ │ └── nyct2010wi.shp.xml ├── dohmh_facilities_daycare │ ├── copy.sql │ ├── data.json │ └── create.sql ├── hhc_facilities_hospitals │ ├── copy.sql │ ├── create.sql │ └── data.json ├── bic_facilities_tradewaste │ ├── copy.sql │ ├── create.sql │ └── data.json ├── dfta_facilities_contracts │ ├── copy.sql │ ├── create.sql │ └── data.json ├── nysparks_facilities_parks │ ├── copy.sql │ ├── data.json │ └── create.sql ├── dcp_councildistricts │ ├── nyccwi_16d.zip │ └── nyccwi_16d │ │ ├── nyccwi.dbf │ │ ├── nyccwi.shp │ │ ├── nyccwi.shx │ │ ├── nyccwi.prj │ │ └── nyccwi.shp.xml ├── fdny_facilities_firehouses │ ├── copy.sql │ ├── create.sql │ └── data.json ├── doe_facilities_universalprek │ ├── copy.sql │ ├── data.json │ └── create.sql ├── nysdec_facilities_solidwaste │ ├── copy.sql │ ├── data.json │ └── create.sql ├── nysdoh_nursinghomebedcensus │ ├── copy.sql │ ├── data.json │ └── create.sql ├── nysopwdd_facilities_providers │ ├── copy.sql │ ├── data.json │ └── create.sql ├── dsny_facilities_specialdropoff │ ├── copy.sql │ ├── create.sql │ └── data.json ├── nysomh_facilities_mentalhealth │ ├── copy.sql │ ├── data.json │ └── create.sql ├── doe_facilities_busroutesgarages │ ├── copy.sql │ ├── data.json │ └── create.sql ├── dca_facilities_operatingbusinesses │ ├── copy.sql │ ├── data.json │ └── create.sql ├── nysdoh_facilities_healthfacilities │ ├── copy.sql │ ├── data.json │ └── create.sql ├── nysparks_facilities_historicplaces │ ├── copy.sql │ ├── create.sql │ └── data.json ├── dcla_facilities_culturalinstitutions │ ├── copy.sql │ ├── create.sql │ └── data.json ├── dcp_cdboundaries │ └── data.json ├── dcp_boroboundaries │ └── data.json ├── dcp_ntaboundaries │ └── data.json ├── dcas_facilities_colp │ └── data.json ├── doitt_zipcodes │ └── data.json ├── dot_streetratings │ └── data.json ├── usnps_facilities_parks │ └── data.json ├── nysdec_facilities_lands │ └── data.json ├── dot_facilities_benches │ └── data.json ├── dpr_streettrees │ └── data.json ├── usdot_facilities_ports │ └── data.json ├── usdot_facilities_airports │ └── data.json ├── dcp_mappluto │ ├── afterold.sql │ ├── after.sql │ └── data.json ├── dcp_facilities_sfpsd │ └── data.json ├── dpr_parksproperties │ └── data.json ├── doitt_subwaylines │ └── data.json ├── doitt_buildingfootprints │ └── data.json ├── doitt_facilities_libraries │ └── data.json ├── doitt_subwaystations │ └── data.json ├── dep_greeninfrastructure │ └── data.json ├── nycha_facilities_policeservice │ └── data.json ├── doitt_facilities_collegesandunis │ └── data.json ├── doe_facilities_publicschools │ └── data.json └── mta_busstops │ ├── after.sql │ └── data.json ├── package.json ├── LICENSE ├── loader.js ├── lib ├── get.js └── push.js └── readme.MD /.gitignore: -------------------------------------------------------------------------------- 1 | /node_modules 2 | /other_datasets/* 3 | /temp -------------------------------------------------------------------------------- /.DS_Store: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/NYCPlanning/db-civic-data-loader/HEAD/.DS_Store -------------------------------------------------------------------------------- /open_datasets/dob_permits/copy.sql: -------------------------------------------------------------------------------- 1 | \COPY dob_permits FROM './temp/dob_permits/dob_permits.csv' CSV HEADER; -------------------------------------------------------------------------------- /open_datasets/.DS_Store: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/NYCPlanning/db-civic-data-loader/HEAD/open_datasets/.DS_Store -------------------------------------------------------------------------------- /open_datasets/dcp_censustracts/.DS_Store: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/NYCPlanning/db-civic-data-loader/HEAD/open_datasets/dcp_censustracts/.DS_Store -------------------------------------------------------------------------------- /open_datasets/dohmh_facilities_daycare/copy.sql: -------------------------------------------------------------------------------- 1 | \COPY dohmh_facilities_daycare FROM './temp/dohmh_facilities_daycare/dohmh_facilities_daycare.csv' CSV HEADER; -------------------------------------------------------------------------------- /open_datasets/hhc_facilities_hospitals/copy.sql: -------------------------------------------------------------------------------- 1 | \COPY hhc_facilities_hospitals FROM './temp/hhc_facilities_hospitals/hhc_facilities_hospitals.csv' CSV HEADER; -------------------------------------------------------------------------------- /open_datasets/bic_facilities_tradewaste/copy.sql: -------------------------------------------------------------------------------- 1 | \COPY bic_facilities_tradewaste FROM './temp/bic_facilities_tradewaste/bic_facilities_tradewaste.csv' CSV HEADER; -------------------------------------------------------------------------------- /open_datasets/dfta_facilities_contracts/copy.sql: -------------------------------------------------------------------------------- 1 | \COPY dfta_facilities_contracts FROM './temp/dfta_facilities_contracts/dfta_facilities_contracts.csv' CSV HEADER; -------------------------------------------------------------------------------- /open_datasets/nysparks_facilities_parks/copy.sql: -------------------------------------------------------------------------------- 1 | \COPY nysparks_facilities_parks FROM './temp/nysparks_facilities_parks/nysparks_facilities_parks.csv' CSV HEADER; -------------------------------------------------------------------------------- /open_datasets/dcp_censustracts/nyct2010wi_16d.zip: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/NYCPlanning/db-civic-data-loader/HEAD/open_datasets/dcp_censustracts/nyct2010wi_16d.zip -------------------------------------------------------------------------------- /open_datasets/dcp_councildistricts/nyccwi_16d.zip: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/NYCPlanning/db-civic-data-loader/HEAD/open_datasets/dcp_councildistricts/nyccwi_16d.zip -------------------------------------------------------------------------------- /open_datasets/fdny_facilities_firehouses/copy.sql: -------------------------------------------------------------------------------- 1 | \COPY fdny_facilities_firehouses FROM './temp/fdny_facilities_firehouses/fdny_facilities_firehouses.csv' CSV HEADER; -------------------------------------------------------------------------------- /open_datasets/doe_facilities_universalprek/copy.sql: -------------------------------------------------------------------------------- 1 | \COPY doe_facilities_universalprek FROM './temp/doe_facilities_universalprek/doe_facilities_universalprek.csv' CSV HEADER; -------------------------------------------------------------------------------- /open_datasets/nysdec_facilities_solidwaste/copy.sql: -------------------------------------------------------------------------------- 1 | \COPY nysdec_facilities_solidwaste FROM './temp/nysdec_facilities_solidwaste/nysdec_facilities_solidwaste.csv' CSV HEADER; -------------------------------------------------------------------------------- /open_datasets/nysdoh_nursinghomebedcensus/copy.sql: -------------------------------------------------------------------------------- 1 | \COPY nysdoh_nursinghomebedcensus FROM './temp/nysdoh_nursinghomebedcensus/nysdoh_nursinghomebedcensus.csv' CSV HEADER; -------------------------------------------------------------------------------- /open_datasets/nysopwdd_facilities_providers/copy.sql: -------------------------------------------------------------------------------- 1 | \COPY nysopwdd_facilities_providers FROM './temp/nysopwdd_facilities_providers/nysopwdd_facilities_providers.csv' CSV HEADER; -------------------------------------------------------------------------------- /open_datasets/dcp_councildistricts/nyccwi_16d/nyccwi.dbf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/NYCPlanning/db-civic-data-loader/HEAD/open_datasets/dcp_councildistricts/nyccwi_16d/nyccwi.dbf -------------------------------------------------------------------------------- /open_datasets/dcp_councildistricts/nyccwi_16d/nyccwi.shp: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/NYCPlanning/db-civic-data-loader/HEAD/open_datasets/dcp_councildistricts/nyccwi_16d/nyccwi.shp -------------------------------------------------------------------------------- /open_datasets/dcp_councildistricts/nyccwi_16d/nyccwi.shx: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/NYCPlanning/db-civic-data-loader/HEAD/open_datasets/dcp_councildistricts/nyccwi_16d/nyccwi.shx -------------------------------------------------------------------------------- /open_datasets/dsny_facilities_specialdropoff/copy.sql: -------------------------------------------------------------------------------- 1 | \COPY dsny_facilities_specialdropoff FROM './temp/dsny_facilities_specialdropoff/dsny_facilities_specialdropoff.csv' CSV HEADER; -------------------------------------------------------------------------------- /open_datasets/nysomh_facilities_mentalhealth/copy.sql: -------------------------------------------------------------------------------- 1 | \COPY nysomh_facilities_mentalhealth FROM './temp/nysomh_facilities_mentalhealth/nysomh_facilities_mentalhealth.csv' CSV HEADER; -------------------------------------------------------------------------------- /open_datasets/dcp_censustracts/nyct2010wi_16d/nyct2010wi.dbf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/NYCPlanning/db-civic-data-loader/HEAD/open_datasets/dcp_censustracts/nyct2010wi_16d/nyct2010wi.dbf -------------------------------------------------------------------------------- /open_datasets/dcp_censustracts/nyct2010wi_16d/nyct2010wi.shp: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/NYCPlanning/db-civic-data-loader/HEAD/open_datasets/dcp_censustracts/nyct2010wi_16d/nyct2010wi.shp -------------------------------------------------------------------------------- /open_datasets/dcp_censustracts/nyct2010wi_16d/nyct2010wi.shx: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/NYCPlanning/db-civic-data-loader/HEAD/open_datasets/dcp_censustracts/nyct2010wi_16d/nyct2010wi.shx -------------------------------------------------------------------------------- /open_datasets/doe_facilities_busroutesgarages/copy.sql: -------------------------------------------------------------------------------- 1 | \COPY doe_facilities_busroutesgarages FROM './temp/doe_facilities_busroutesgarages/doe_facilities_busroutesgarages.csv' CSV HEADER; -------------------------------------------------------------------------------- /open_datasets/dca_facilities_operatingbusinesses/copy.sql: -------------------------------------------------------------------------------- 1 | \COPY dca_facilities_operatingbusinesses FROM './temp/dca_facilities_operatingbusinesses/dca_facilities_operatingbusinesses.csv' CSV HEADER; -------------------------------------------------------------------------------- /open_datasets/nysdoh_facilities_healthfacilities/copy.sql: -------------------------------------------------------------------------------- 1 | \COPY nysdoh_facilities_healthfacilities FROM './temp/nysdoh_facilities_healthfacilities/nysdoh_facilities_healthfacilities.csv' CSV HEADER; -------------------------------------------------------------------------------- /open_datasets/nysparks_facilities_historicplaces/copy.sql: -------------------------------------------------------------------------------- 1 | \COPY nysparks_facilities_historicplaces FROM './temp/nysparks_facilities_historicplaces/nysparks_facilities_historicplaces.csv' CSV HEADER; -------------------------------------------------------------------------------- /open_datasets/dcla_facilities_culturalinstitutions/copy.sql: -------------------------------------------------------------------------------- 1 | \COPY dcla_facilities_culturalinstitutions FROM './temp/dcla_facilities_culturalinstitutions/dcla_facilities_culturalinstitutions.csv' CSV HEADER; -------------------------------------------------------------------------------- /open_datasets/fdny_facilities_firehouses/create.sql: -------------------------------------------------------------------------------- 1 | -- create table to load csv from the nyc open data portal 2 | DROP TABLE IF EXISTS fdny_facilities_firehouses; 3 | CREATE TABLE fdny_facilities_firehouses ( 4 | Fire_House_Name text, 5 | Street_Address text, 6 | Borough text 7 | ) -------------------------------------------------------------------------------- /open_datasets/dsny_facilities_specialdropoff/create.sql: -------------------------------------------------------------------------------- 1 | -- create table to load csv from the nyc open data portal 2 | DROP TABLE IF EXISTS dsny_facilities_specialdropoff; 3 | CREATE TABLE dsny_facilities_specialdropoff ( 4 | Borough text, 5 | DropOff_Site_Location text, 6 | X_Coordinate double precision, 7 | Y_Coordinate double precision 8 | ) -------------------------------------------------------------------------------- /open_datasets/hhc_facilities_hospitals/create.sql: -------------------------------------------------------------------------------- 1 | -- create table to load csv from the nyc open data portal 2 | DROP TABLE IF EXISTS hhc_facilities_hospitals; 3 | CREATE TABLE hhc_facilities_hospitals ( 4 | Facility_Type text, 5 | Borough text, 6 | Facility_Name text, 7 | Cross_Streets text, 8 | Phone text, 9 | Location_1 text 10 | ) -------------------------------------------------------------------------------- /open_datasets/bic_facilities_tradewaste/create.sql: -------------------------------------------------------------------------------- 1 | -- create table to load csv from the nyc open data portal 2 | DROP TABLE IF EXISTS bic_facilities_tradewaste; 3 | CREATE TABLE bic_facilities_tradewaste ( 4 | TYPE text, 5 | BUS_NAME text, 6 | MAILING_OFFICE text, 7 | MAIL_CITY text, 8 | MSTAT text, 9 | COMP_PHONE text, 10 | Location_1 text 11 | ) -------------------------------------------------------------------------------- /open_datasets/dcp_cdboundaries/data.json: -------------------------------------------------------------------------------- 1 | { 2 | "url": "http://www1.nyc.gov/assets/planning/download/zip/data-maps/open-data/nycd16b.zip", 3 | "load": "shp2pgsql", 4 | "loadFiles":[ 5 | { 6 | "file":"nycd_16b/nycd.shp", 7 | "table": "dcp_cdboundaries" 8 | } 9 | ], 10 | "shp2pgsql": [ 11 | "-d", 12 | "-s 2263:4326" 13 | ] 14 | } -------------------------------------------------------------------------------- /open_datasets/dcp_boroboundaries/data.json: -------------------------------------------------------------------------------- 1 | { 2 | "url": "http://www1.nyc.gov/assets/planning/download/zip/data-maps/open-data/nybb16b.zip", 3 | 4 | "load": "shp2pgsql", 5 | "loadFiles": [ 6 | { 7 | "file":"nybb_16b/nybb.shp", 8 | "table": "dcp_boroboundaries" 9 | } 10 | ], 11 | "shp2pgsql": [ 12 | "-d", 13 | "-s 2263:4326" 14 | ] 15 | } -------------------------------------------------------------------------------- /open_datasets/dcp_ntaboundaries/data.json: -------------------------------------------------------------------------------- 1 | { 2 | "url": "http://www1.nyc.gov/assets/planning/download/zip/data-maps/open-data/nynta16b.zip", 3 | 4 | "load": "shp2pgsql", 5 | "loadFiles": [ 6 | { 7 | "file":"nynta_16b/nynta.shp", 8 | "table": "dcp_ntaboundaries" 9 | } 10 | ], 11 | "shp2pgsql": [ 12 | "-d", 13 | "-s 2263:4326" 14 | ] 15 | } -------------------------------------------------------------------------------- /open_datasets/dob_permits/after.sql: -------------------------------------------------------------------------------- 1 | ALTER TABLE dob_permits 2 | ADD COLUMN "bbl" text; 3 | 4 | UPDATE dob_permits SET bbl = CASE 5 | WHEN borough='MANHATTAN' THEN 1 6 | WHEN borough='BRONX' THEN 2 7 | WHEN borough='BROOKLYN' THEN 3 8 | WHEN borough='QUEENS' THEN 4 9 | WHEN borough='STATEN ISLAND' THEN 5 10 | ELSE 0 11 | END || block || RIGHT(lot,4) -------------------------------------------------------------------------------- /open_datasets/dob_permits/data.json: -------------------------------------------------------------------------------- 1 | { 2 | "url": "https://nycopendata.socrata.com/api/views/ipu4-2q9a/rows.csv?accessType=DOWNLOAD", 3 | "saveFile": "dob_permits.csv", 4 | "load": "csv", 5 | "loadFiles": [ 6 | { 7 | "file":"dob_permits.csv", 8 | "table": "dob_permits" 9 | } 10 | ], 11 | "csv": [ 12 | "create", 13 | "copy" 14 | ] 15 | } -------------------------------------------------------------------------------- /open_datasets/dcas_facilities_colp/data.json: -------------------------------------------------------------------------------- 1 | { 2 | "url": "http://www1.nyc.gov/assets/planning/download/zip/data-maps/open-data/nyc-colp-shp-2016.zip", 3 | 4 | "load": "shp2pgsql", 5 | "loadFiles": [ 6 | { 7 | "file":"shp/nycCOLP_201602.shp", 8 | "table": "dcas_facilities_colp" 9 | } 10 | ], 11 | "shp2pgsql": [ 12 | "-d", 13 | "-s 2263:4326" 14 | ] 15 | } -------------------------------------------------------------------------------- /open_datasets/doitt_zipcodes/data.json: -------------------------------------------------------------------------------- 1 | { 2 | "url": "https://data.cityofnewyork.us/api/file_data/YObIR0MbpUVA0EpQzZSq5x55FzKGM2ejSeahdvjqR20?filename=ZIP_CODE_040114.zip", 3 | "load": "shp2pgsql", 4 | "loadFiles":[ 5 | { 6 | "file":"ZIP_CODE_040114.shp", 7 | "table": "doitt_zipcodes" 8 | } 9 | ], 10 | "shp2pgsql": [ 11 | "-d", 12 | "-s 2263:4326" 13 | ] 14 | } -------------------------------------------------------------------------------- /open_datasets/dot_streetratings/data.json: -------------------------------------------------------------------------------- 1 | { 2 | "url": "http://www.nyc.gov/html/dot/downloads/misc/street-assessment-rating.zip", 3 | 4 | "load": "shp2pgsql", 5 | "loadFiles":[ 6 | { 7 | "file":"StreetAssessmentRating/StreetAssessmentRating.shp", 8 | "table": "dot_streetratings" 9 | } 10 | ], 11 | "shp2pgsql": [ 12 | "-d", 13 | "-s 2263:4326" 14 | ] 15 | } -------------------------------------------------------------------------------- /open_datasets/usnps_facilities_parks/data.json: -------------------------------------------------------------------------------- 1 | { 2 | "url": "https://irma.nps.gov/DataStore/DownloadFile/558608", 3 | "saveFile": "usnps_facilities_parks.zip", 4 | "load": "shp2pgsql", 5 | "loadFiles": [ 6 | { 7 | "file":"nps_boundary.shp", 8 | "table": "usnps_facilities_parks" 9 | } 10 | ], 11 | "shp2pgsql": [ 12 | "-d", 13 | "-s 4326:4326" 14 | ] 15 | } -------------------------------------------------------------------------------- /open_datasets/nysdec_facilities_lands/data.json: -------------------------------------------------------------------------------- 1 | { 2 | "url": "http://gis.ny.gov/gisdata/data/ds_1114/DEC_lands.zip", 3 | 4 | "load": "shp2pgsql", 5 | "loadFiles": [ 6 | { 7 | "file":"DEC_lands.shp", 8 | "table": "nysdec_facilities_lands" 9 | } 10 | ], 11 | "saveFile":"nysdec_facilities_lands.zip", 12 | "shp2pgsql": [ 13 | "-d", 14 | "-s 26918:4326" 15 | ] 16 | } -------------------------------------------------------------------------------- /open_datasets/dfta_facilities_contracts/create.sql: -------------------------------------------------------------------------------- 1 | -- create table to load csv from the nyc open data portal 2 | DROP TABLE IF EXISTS dfta_facilities_contracts; 3 | CREATE TABLE dfta_facilities_contracts ( 4 | Contract_Type text, 5 | Provider_ID text, 6 | Program_Name text, 7 | Sponsor_Name text, 8 | Program_Address text, 9 | Program_Borough text, 10 | Program_Zipcode text, 11 | Program_Phone text 12 | ) -------------------------------------------------------------------------------- /open_datasets/dot_facilities_benches/data.json: -------------------------------------------------------------------------------- 1 | { 2 | "url": "http://www.nyc.gov/html/dot/downloads/misc/2015_citybench.zip", 3 | 4 | "load": "shp2pgsql", 5 | "loadFiles": [ 6 | { 7 | "file":"NYC_CityBench.shp", 8 | "table": "dot_facilities_benches" 9 | } 10 | ], 11 | "saveFile":"dot_facilities_benches.zip", 12 | "shp2pgsql": [ 13 | "-d", 14 | "-s 2263:4326" 15 | ] 16 | } -------------------------------------------------------------------------------- /open_datasets/dpr_streettrees/data.json: -------------------------------------------------------------------------------- 1 | { 2 | "url": "https://data.cityofnewyork.us/api/geospatial/pi5s-9p35?method=export&format=Original", 3 | "saveFile": "dpr_streettrees.zip", 4 | "load": "shp2pgsql", 5 | "loadFiles": [ 6 | { 7 | "file":"TreesCount2015Trees.shp", 8 | "table": "dpr_streettrees" 9 | } 10 | ], 11 | "shp2pgsql": [ 12 | "-d", 13 | "-s 2263:4326" 14 | ] 15 | } 16 | -------------------------------------------------------------------------------- /open_datasets/bic_facilities_tradewaste/data.json: -------------------------------------------------------------------------------- 1 | { 2 | "url": "https://data.cityofnewyork.us/api/views/7atx-5a3s/rows.csv?accessType=DOWNLOAD", 3 | "saveFile": "bic_facilities_tradewaste.csv", 4 | "load": "csv", 5 | "loadFiles": [ 6 | { 7 | "file":"bic_facilities_tradewaste.csv", 8 | "table": "bic_facilities_tradewaste" 9 | } 10 | ], 11 | "csv": [ 12 | "create", 13 | "copy" 14 | ] 15 | } -------------------------------------------------------------------------------- /open_datasets/dfta_facilities_contracts/data.json: -------------------------------------------------------------------------------- 1 | { 2 | "url": "https://data.cityofnewyork.us/api/views/6j6t-3ixh/rows.csv?accessType=DOWNLOAD", 3 | "saveFile": "dfta_facilities_contracts.csv", 4 | "load": "csv", 5 | "loadFiles": [ 6 | { 7 | "file":"dfta_facilities_contracts.csv", 8 | "table": "dfta_facilities_contracts" 9 | } 10 | ], 11 | "csv": [ 12 | "create", 13 | "copy" 14 | ] 15 | } -------------------------------------------------------------------------------- /open_datasets/dohmh_facilities_daycare/data.json: -------------------------------------------------------------------------------- 1 | { 2 | "url": "https://data.cityofnewyork.us/api/views/dsg6-ifza/rows.csv?accessType=DOWNLOAD", 3 | "saveFile": "dohmh_facilities_daycare.csv", 4 | "load": "csv", 5 | "loadFiles": [ 6 | { 7 | "file":"dohmh_facilities_daycare.csv", 8 | "table": "dohmh_facilities_daycare" 9 | } 10 | ], 11 | "csv": [ 12 | "create", 13 | "copy" 14 | ] 15 | } -------------------------------------------------------------------------------- /open_datasets/hhc_facilities_hospitals/data.json: -------------------------------------------------------------------------------- 1 | { 2 | "url": "https://data.cityofnewyork.us/api/views/f7b6-v6v3/rows.csv?accessType=DOWNLOAD", 3 | "saveFile": "hhc_facilities_hospitals.csv", 4 | "load": "csv", 5 | "loadFiles": [ 6 | { 7 | "file":"hhc_facilities_hospitals.csv", 8 | "table": "hhc_facilities_hospitals" 9 | } 10 | ], 11 | "csv": [ 12 | "create", 13 | "copy" 14 | ] 15 | } -------------------------------------------------------------------------------- /open_datasets/nysparks_facilities_parks/data.json: -------------------------------------------------------------------------------- 1 | { 2 | "url": "https://data.ny.gov/api/views/9uuk-x7vh/rows.csv?accessType=DOWNLOAD", 3 | "saveFile": "nysparks_facilities_parks.csv", 4 | "load": "csv", 5 | "loadFiles": [ 6 | { 7 | "file":"nysparks_facilities_parks.csv", 8 | "table": "nysparks_facilities_parks" 9 | } 10 | ], 11 | "csv": [ 12 | "create", 13 | "copy" 14 | ] 15 | } 16 | -------------------------------------------------------------------------------- /open_datasets/fdny_facilities_firehouses/data.json: -------------------------------------------------------------------------------- 1 | { 2 | "url": "https://data.cityofnewyork.us/api/views/pg99-2ihy/rows.csv?accessType=DOWNLOAD", 3 | "saveFile": "fdny_facilities_firehouses.csv", 4 | "load": "csv", 5 | "loadFiles": [ 6 | { 7 | "file":"fdny_facilities_firehouses.csv", 8 | "table": "fdny_facilities_firehouses" 9 | } 10 | ], 11 | "csv": [ 12 | "create", 13 | "copy" 14 | ] 15 | } -------------------------------------------------------------------------------- /open_datasets/nysdec_facilities_solidwaste/data.json: -------------------------------------------------------------------------------- 1 | { 2 | "url": "https://data.ny.gov/api/views/2fni-raj8/rows.csv?accessType=DOWNLOAD", 3 | "saveFile": "nysdec_facilities_solidwaste.csv", 4 | "load": "csv", 5 | "loadFiles": [ 6 | { 7 | "file":"nysdec_facilities_solidwaste.csv", 8 | "table": "nysdec_facilities_solidwaste" 9 | } 10 | ], 11 | "csv": [ 12 | "create", 13 | "copy" 14 | ] 15 | } -------------------------------------------------------------------------------- /open_datasets/nysdoh_nursinghomebedcensus/data.json: -------------------------------------------------------------------------------- 1 | { 2 | "url": "https://health.data.ny.gov/api/views/izta-vnpq/rows.csv?accessType=DOWNLOAD", 3 | "saveFile": "nysdoh_nursinghomebedcensus.csv", 4 | "load": "csv", 5 | "loadFiles": [ 6 | { 7 | "file":"nysdoh_nursinghomebedcensus.csv", 8 | "table": "nysdoh_nursinghomebedcensus" 9 | } 10 | ], 11 | "csv": [ 12 | "create", 13 | "copy" 14 | ] 15 | } -------------------------------------------------------------------------------- /open_datasets/usdot_facilities_ports/data.json: -------------------------------------------------------------------------------- 1 | { 2 | "url": "http://www.rita.dot.gov/bts/sites/rita.dot.gov.bts/files/AdditionalAttachmentFiles/ports.zip", 3 | "saveFile": "usdot_facilities_ports.zip", 4 | "load": "shp2pgsql", 5 | "loadFiles": [ 6 | { 7 | "file":"ports.shp", 8 | "table": "usdot_facilities_ports" 9 | } 10 | ], 11 | "shp2pgsql": [ 12 | "-d", 13 | "-s 4326:4326" 14 | ] 15 | } -------------------------------------------------------------------------------- /open_datasets/nysomh_facilities_mentalhealth/data.json: -------------------------------------------------------------------------------- 1 | { 2 | "url": "https://data.ny.gov/api/views/6nvr-tbv8/rows.csv?accessType=DOWNLOAD", 3 | "saveFile": "nysomh_facilities_mentalhealth.csv", 4 | "load": "csv", 5 | "loadFiles": [ 6 | { 7 | "file":"nysomh_facilities_mentalhealth.csv", 8 | "table": "nysomh_facilities_mentalhealth" 9 | } 10 | ], 11 | "csv": [ 12 | "create", 13 | "copy" 14 | ] 15 | } -------------------------------------------------------------------------------- /open_datasets/nysopwdd_facilities_providers/data.json: -------------------------------------------------------------------------------- 1 | { 2 | "url": "https://data.ny.gov/api/views/ieqx-cqyk/rows.csv?accessType=DOWNLOAD", 3 | "saveFile": "nysopwdd_facilities_providers.csv", 4 | "load": "csv", 5 | "loadFiles": [ 6 | { 7 | "file":"nysopwdd_facilities_providers.csv", 8 | "table": "nysopwdd_facilities_providers" 9 | } 10 | ], 11 | "csv": [ 12 | "create", 13 | "copy" 14 | ] 15 | } -------------------------------------------------------------------------------- /open_datasets/nysparks_facilities_historicplaces/create.sql: -------------------------------------------------------------------------------- 1 | -- create table to load csv from the nyc open data portal 2 | DROP TABLE IF EXISTS nysparks_facilities_historicplaces; 3 | CREATE TABLE nysparks_facilities_historicplaces ( 4 | Resource_Name text, 5 | County text, 6 | National_Register_Date date, 7 | National_Register_Number text, 8 | Longitude double precision, 9 | Latitude double precision, 10 | Location text 11 | ) -------------------------------------------------------------------------------- /open_datasets/doe_facilities_universalprek/data.json: -------------------------------------------------------------------------------- 1 | { 2 | "url": "https://data.cityofnewyork.us/api/views/kiyv-ks3f/rows.csv?accessType=DOWNLOAD", 3 | "saveFile": "doe_facilities_universalprek.csv", 4 | "load": "csv", 5 | "loadFiles": [ 6 | { 7 | "file":"doe_facilities_universalprek.csv", 8 | "table": "doe_facilities_universalprek" 9 | } 10 | ], 11 | "csv": [ 12 | "create", 13 | "copy" 14 | ] 15 | } -------------------------------------------------------------------------------- /open_datasets/dsny_facilities_specialdropoff/data.json: -------------------------------------------------------------------------------- 1 | { 2 | "url": "https://data.cityofnewyork.us/api/views/a34j-ihvy/rows.csv?accessType=DOWNLOAD", 3 | "saveFile": "dsny_facilities_specialdropoff.csv", 4 | "load": "csv", 5 | "loadFiles": [ 6 | { 7 | "file":"dsny_facilities_specialdropoff.csv", 8 | "table": "dsny_facilities_specialdropoff" 9 | } 10 | ], 11 | "csv": [ 12 | "create", 13 | "copy" 14 | ] 15 | } -------------------------------------------------------------------------------- /open_datasets/usdot_facilities_airports/data.json: -------------------------------------------------------------------------------- 1 | { 2 | "url": "http://www.rita.dot.gov/bts/sites/rita.dot.gov.bts/files/AdditionalAttachmentFiles/airports.zip", 3 | "saveFile": "usdot_facilities_airports.zip", 4 | "load": "shp2pgsql", 5 | "loadFiles": [ 6 | { 7 | "file":"airports.shp", 8 | "table": "usdot_facilities_airports" 9 | } 10 | ], 11 | "shp2pgsql": [ 12 | "-d", 13 | "-s 4326:4326" 14 | ] 15 | } -------------------------------------------------------------------------------- /open_datasets/dcp_mappluto/afterold.sql: -------------------------------------------------------------------------------- 1 | DROP TABLE IF EXISTS dcp_mappluto; 2 | CREATE TABLE dcp_mappluto AS 3 | SELECT * FROM mnmappluto 4 | UNION ALL 5 | SELECT * FROM bxmappluto 6 | UNION ALL 7 | SELECT * FROM qnmappluto 8 | UNION ALL 9 | SELECT * FROM bkmappluto 10 | UNION ALL 11 | SELECT * FROM simappluto; 12 | 13 | DROP TABLE mnmappluto; 14 | DROP TABLE bxmappluto; 15 | DROP TABLE qnmappluto; 16 | DROP TABLE bkmappluto; 17 | DROP TABLE simappluto; -------------------------------------------------------------------------------- /open_datasets/doe_facilities_busroutesgarages/data.json: -------------------------------------------------------------------------------- 1 | { 2 | "url": "https://data.cityofnewyork.us/api/views/8yac-vygm/rows.csv?accessType=DOWNLOAD", 3 | "saveFile": "doe_facilities_busroutesgarages.csv", 4 | "load": "csv", 5 | "loadFiles": [ 6 | { 7 | "file":"doe_facilities_busroutesgarages.csv", 8 | "table": "doe_facilities_busroutesgarages" 9 | } 10 | ], 11 | "csv": [ 12 | "create", 13 | "copy" 14 | ] 15 | } -------------------------------------------------------------------------------- /open_datasets/dcp_facilities_sfpsd/data.json: -------------------------------------------------------------------------------- 1 | { 2 | "url": "http://www1.nyc.gov/assets/planning/download/zip/data-maps/open-data/nyc_facilities2015_shp.zip", 3 | "saveFile":"dcp_facilities_sfpsd.zip", 4 | "load": "shp2pgsql", 5 | "loadFiles": [ 6 | { 7 | "file":"Facilities_2015_shp/Facilities.shp", 8 | "table": "dcp_facilities_sfpsd" 9 | } 10 | ], 11 | 12 | "shp2pgsql": [ 13 | "-d", 14 | "-s 2263:4326" 15 | ] 16 | } -------------------------------------------------------------------------------- /open_datasets/dpr_parksproperties/data.json: -------------------------------------------------------------------------------- 1 | { 2 | "url": "https://data.cityofnewyork.us/api/geospatial/rjaj-zgq7?method=export&format=Original", 3 | "saveFile": "dpr_parksproperties.zip", 4 | "load": "shp2pgsql", 5 | "loadFiles": [ 6 | { 7 | "file":"DPR_ParksProperties_001/DPR_ParksProperties_001.shp", 8 | "table": "dpr_parksproperties" 9 | } 10 | ], 11 | "shp2pgsql": [ 12 | "-d", 13 | "-s 2263:4326" 14 | ] 15 | } -------------------------------------------------------------------------------- /open_datasets/dcla_facilities_culturalinstitutions/create.sql: -------------------------------------------------------------------------------- 1 | -- create table to load csv from the nyc open data portal 2 | DROP TABLE IF EXISTS dcla_facilities_culturalinstitutions; 3 | CREATE TABLE dcla_facilities_culturalinstitutions ( 4 | Organization_Name text, 5 | Address text, 6 | City text, 7 | State text, 8 | Zip_Code text, 9 | Main_Phone text, 10 | Discipline text, 11 | Council_District text, 12 | Community_Board text, 13 | Borough text 14 | ) -------------------------------------------------------------------------------- /open_datasets/doitt_subwaylines/data.json: -------------------------------------------------------------------------------- 1 | { 2 | "url": "https://nycpublicdata.carto.com/api/v2/sql?filename=doitt_subwaylines.zip&q=SELECT%20*%20FROM%20nyc_subway_line&format=shp", 3 | "saveFile": "doitt_subwaylines.zip", 4 | "load": "shp2pgsql", 5 | "loadFiles": [ 6 | { 7 | "file":"doitt_subwaylines.shp", 8 | "table": "doitt_subwaylines" 9 | } 10 | ], 11 | "shp2pgsql": [ 12 | "-d", 13 | "-s 4326:4326" 14 | ] 15 | } 16 | -------------------------------------------------------------------------------- /open_datasets/nysdoh_facilities_healthfacilities/data.json: -------------------------------------------------------------------------------- 1 | { 2 | "url": "https://health.data.ny.gov/api/views/vn5v-hh5r/rows.csv?accessType=DOWNLOAD", 3 | "saveFile": "nysdoh_facilities_healthfacilities.csv", 4 | "load": "csv", 5 | "loadFiles": [ 6 | { 7 | "file":"nysdoh_facilities_healthfacilities.csv", 8 | "table": "nysdoh_facilities_healthfacilities" 9 | } 10 | ], 11 | "csv": [ 12 | "create", 13 | "copy" 14 | ] 15 | } -------------------------------------------------------------------------------- /open_datasets/nysparks_facilities_historicplaces/data.json: -------------------------------------------------------------------------------- 1 | { 2 | "url": "https://data.ny.gov/api/views/y36f-mkpp/rows.csv?accessType=DOWNLOAD", 3 | "saveFile": "nysparks_facilities_historicplaces.csv", 4 | "load": "csv", 5 | "loadFiles": [ 6 | { 7 | "file":"nysparks_facilities_historicplaces.csv", 8 | "table": "nysparks_facilities_historicplaces" 9 | } 10 | ], 11 | "csv": [ 12 | "create", 13 | "copy" 14 | ] 15 | } 16 | -------------------------------------------------------------------------------- /open_datasets/dca_facilities_operatingbusinesses/data.json: -------------------------------------------------------------------------------- 1 | { 2 | "url": "https://data.cityofnewyork.us/api/views/w7w3-xahh/rows.csv?accessType=DOWNLOAD", 3 | "saveFile": "dca_facilities_operatingbusinesses.csv", 4 | "load": "csv", 5 | "loadFiles": [ 6 | { 7 | "file":"dca_facilities_operatingbusinesses.csv", 8 | "table": "dca_facilities_operatingbusinesses" 9 | } 10 | ], 11 | "csv": [ 12 | "create", 13 | "copy" 14 | ] 15 | } -------------------------------------------------------------------------------- /open_datasets/dcla_facilities_culturalinstitutions/data.json: -------------------------------------------------------------------------------- 1 | { 2 | "url": "https://data.cityofnewyork.us/api/views/u35m-9t32/rows.csv?accessType=DOWNLOAD", 3 | "saveFile": "dcla_facilities_culturalinstitutions.csv", 4 | "load": "csv", 5 | "loadFiles": [ 6 | { 7 | "file":"dcla_facilities_culturalinstitutions.csv", 8 | "table": "dcla_facilities_culturalinstitutions" 9 | } 10 | ], 11 | "csv": [ 12 | "create", 13 | "copy" 14 | ] 15 | } -------------------------------------------------------------------------------- /open_datasets/doitt_buildingfootprints/data.json: -------------------------------------------------------------------------------- 1 | { 2 | "url": "https://data.cityofnewyork.us/api/geospatial/nqwf-w8eh?method=export&format=Original", 3 | 4 | "load": "shp2pgsql", 5 | "loadFiles":[ 6 | { 7 | "file":"building_0716.shp", 8 | "table": "doitt_buildingfootprints" 9 | } 10 | ], 11 | "saveFile" : "doitt_buildingfootprints.zip", 12 | "shp2pgsql": [ 13 | "-d", 14 | "-s 2263:4326", 15 | "-W \"latin1\"" 16 | ] 17 | } 18 | -------------------------------------------------------------------------------- /open_datasets/doitt_facilities_libraries/data.json: -------------------------------------------------------------------------------- 1 | { 2 | "url": "https://data.cityofnewyork.us/api/geospatial/p4pf-fyc4?method=export&format=Shapefile", 3 | "saveFile": "doitt_facilities_libraries.zip", 4 | "load": "shp2pgsql", 5 | "loadFiles": [ 6 | { 7 | "file":"geo_export_9c2fc8d8-3af8-4f04-a4eb-c31f34d68ac7.shp", 8 | "table": "doitt_facilities_libraries" 9 | } 10 | ], 11 | "shp2pgsql": [ 12 | "-d", 13 | "-s 4326:4326" 14 | ] 15 | } -------------------------------------------------------------------------------- /open_datasets/doitt_subwaystations/data.json: -------------------------------------------------------------------------------- 1 | { 2 | "url": "https://nycpublicdata.carto.com/api/v2/sql?filename=doitt_subwaystations.zip&q=SELECT%20*%20FROM%20nyc_subway_station&format=shp", 3 | "saveFile": "doitt_subwaystations.zip", 4 | "load": "shp2pgsql", 5 | "loadFiles": [ 6 | { 7 | "file":"doitt_subwaystations.shp", 8 | "table": "doitt_subwaystations" 9 | } 10 | ], 11 | "shp2pgsql": [ 12 | "-d", 13 | "-s 4326:4326" 14 | ] 15 | } 16 | -------------------------------------------------------------------------------- /open_datasets/dep_greeninfrastructure/data.json: -------------------------------------------------------------------------------- 1 | { 2 | "url": "https://data.cityofnewyork.us/api/file_data/wpYHtRhvWqCGKQ9N91VgCogV5dYDIv643DFR-Nte8ss?filename=DEP_GREEN_INFRASTRUCTURE.shp.zip", 3 | "saveFile": "dep_greeninfrastructure.zip", 4 | "load": "shp2pgsql", 5 | "loadFiles": [ 6 | { 7 | "file":"DEP_GREEN_INFRASTRUCTURE.shp", 8 | "table": "dep_greeninfrastructure" 9 | } 10 | ], 11 | "shp2pgsql": [ 12 | "-d", 13 | "-s 2263:4326" 14 | ] 15 | } -------------------------------------------------------------------------------- /open_datasets/nycha_facilities_policeservice/data.json: -------------------------------------------------------------------------------- 1 | { 2 | "url": "https://data.cityofnewyork.us/api/geospatial/72wx-vdjr?method=export&format=Shapefile", 3 | "saveFile": "nycha_facilities_policeservice.zip", 4 | "load": "shp2pgsql", 5 | "loadFiles": [ 6 | { 7 | "file":"geo_export_39f7575b-a769-4570-9228-cd56ad3628a0.shp", 8 | "table": "nycha_facilities_policeservice" 9 | } 10 | ], 11 | "shp2pgsql": [ 12 | "-d", 13 | "-s 4326:4326" 14 | ] 15 | } -------------------------------------------------------------------------------- /open_datasets/doitt_facilities_collegesandunis/data.json: -------------------------------------------------------------------------------- 1 | { 2 | "url": "https://data.cityofnewyork.us/api/geospatial/4kym-4xw5?method=export&format=Shapefile", 3 | "saveFile": "doitt_facilities_collegesandunis.zip", 4 | "load": "shp2pgsql", 5 | "loadFiles": [ 6 | { 7 | "file":"geo_export_39f7575b-a769-4570-9228-cd56ad3628a0.shp", 8 | "table": "doitt_facilities_collegesandunis" 9 | } 10 | ], 11 | "shp2pgsql": [ 12 | "-d", 13 | "-s 2263:4326" 14 | ] 15 | } -------------------------------------------------------------------------------- /open_datasets/doe_facilities_publicschools/data.json: -------------------------------------------------------------------------------- 1 | { 2 | "url": "https://data.cityofnewyork.us/api/file_data/IjLvNcAZDM01muXoNu_-a2ggw13oNeeV4xH-rPS_Dig?filename=Public_School_Locations.zip ", 3 | 4 | "load": "shp2pgsql", 5 | "loadFiles": [ 6 | { 7 | "file":"Public_Schools_Points_2011-2012A.shp", 8 | "table": "doe_facilities_publicschools" 9 | } 10 | ], 11 | "saveFile":"doe_facilities_publicschools.zip", 12 | "shp2pgsql": [ 13 | "-d", 14 | "-s 2263:4326" 15 | ] 16 | } -------------------------------------------------------------------------------- /package.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "civic-data-loader", 3 | "version": "1.0.0", 4 | "description": "Quickly ETL file-based data into PostgreSQL/PostGIS", 5 | "main": "loader.js", 6 | "scripts": { 7 | "test": "echo \"Error: no test specified\" && exit 1" 8 | }, 9 | "author": "", 10 | "license": "ISC", 11 | "dependencies": { 12 | "dotenv": "^2.0.0", 13 | "download-file": "^0.1.5", 14 | "fs-extra": "^0.30.0", 15 | "ftp-get": "^0.3.1", 16 | "mustache": "^2.2.1", 17 | "unzip": "^0.1.11" 18 | } 19 | } 20 | -------------------------------------------------------------------------------- /open_datasets/nysparks_facilities_parks/create.sql: -------------------------------------------------------------------------------- 1 | -- create table to load csv from the nyc open data portal 2 | DROP TABLE IF EXISTS nysparks_facilities_parks; 3 | CREATE TABLE nysparks_facilities_parks ( 4 | Name text, 5 | Category text, 6 | Region text, 7 | County text, 8 | Golf text, 9 | Camp text, 10 | Playground text, 11 | Nature_Center text, 12 | Beach text, 13 | Facility_URL text, 14 | Golf_URL text, 15 | Nature_Center_URL text, 16 | Longitude double precision, 17 | Latitude double precision, 18 | Location text 19 | ) -------------------------------------------------------------------------------- /open_datasets/mta_busstops/after.sql: -------------------------------------------------------------------------------- 1 | DROP TABLE IF EXISTS mta_busstops; 2 | CREATE TABLE mta_busstops AS 3 | SELECT * FROM manhattanstops 4 | UNION ALL 5 | SELECT * FROM bronxstops 6 | UNION ALL 7 | SELECT * FROM queensstops 8 | UNION ALL 9 | SELECT * FROM brooklynstops 10 | UNION ALL 11 | SELECT * FROM statenislandstops 12 | UNION ALL 13 | SELECT * FROM buscostops; 14 | 15 | DROP TABLE manhattanstops; 16 | DROP TABLE bronxstops; 17 | DROP TABLE brooklynstops; 18 | DROP TABLE statenislandstops; 19 | DROP TABLE buscostops; 20 | 21 | ALTER TABLE mta_busstops RENAME COLUMN wkb_geometry TO geom; -------------------------------------------------------------------------------- /open_datasets/dcp_councildistricts/nyccwi_16d/nyccwi.prj: -------------------------------------------------------------------------------- 1 | PROJCS["NAD_1983_StatePlane_New_York_Long_Island_FIPS_3104_Feet",GEOGCS["GCS_North_American_1983",DATUM["D_North_American_1983",SPHEROID["GRS_1980",6378137.0,298.257222101]],PRIMEM["Greenwich",0.0],UNIT["Degree",0.0174532925199433]],PROJECTION["Lambert_Conformal_Conic"],PARAMETER["False_Easting",984250.0],PARAMETER["False_Northing",0.0],PARAMETER["Central_Meridian",-74.0],PARAMETER["Standard_Parallel_1",40.66666666666666],PARAMETER["Standard_Parallel_2",41.03333333333333],PARAMETER["Latitude_Of_Origin",40.16666666666666],UNIT["Foot_US",0.3048006096012192]] -------------------------------------------------------------------------------- /open_datasets/dcp_censustracts/nyct2010wi_16d/nyct2010wi.prj: -------------------------------------------------------------------------------- 1 | PROJCS["NAD_1983_StatePlane_New_York_Long_Island_FIPS_3104_Feet",GEOGCS["GCS_North_American_1983",DATUM["D_North_American_1983",SPHEROID["GRS_1980",6378137.0,298.257222101]],PRIMEM["Greenwich",0.0],UNIT["Degree",0.0174532925199433]],PROJECTION["Lambert_Conformal_Conic"],PARAMETER["False_Easting",984250.0],PARAMETER["False_Northing",0.0],PARAMETER["Central_Meridian",-74.0],PARAMETER["Standard_Parallel_1",40.66666666666666],PARAMETER["Standard_Parallel_2",41.03333333333333],PARAMETER["Latitude_Of_Origin",40.16666666666666],UNIT["Foot_US",0.3048006096012192]] -------------------------------------------------------------------------------- /open_datasets/doe_facilities_universalprek/create.sql: -------------------------------------------------------------------------------- 1 | -- create table to load csv from the nyc open data portal 2 | DROP TABLE IF EXISTS doe_facilities_universalprek; 3 | CREATE TABLE doe_facilities_universalprek ( 4 | LOCCODE text, 5 | PreK_Type text, 6 | Borough text, 7 | LocName text, 8 | NOTE text, 9 | phone text, 10 | address text, 11 | zip text, 12 | Day_Length text, 13 | Seats double precision, 14 | X double precision, 15 | Y double precision, 16 | Email text, 17 | Website text, 18 | MEALS text, 19 | INDOOR_OUTDOOR text, 20 | EXTENDED_DAY text, 21 | SEMS_CODE text 22 | ) -------------------------------------------------------------------------------- /open_datasets/doe_facilities_busroutesgarages/create.sql: -------------------------------------------------------------------------------- 1 | -- create table to load csv from the nyc open data portal 2 | DROP TABLE IF EXISTS doe_facilities_busroutesgarages; 3 | CREATE TABLE doe_facilities_busroutesgarages ( 4 | School_Year text, 5 | Route_Number text, 6 | Service_Type text, 7 | Vehicle_TypeDescription text, 8 | Route_Start_Date text, 9 | Vendor_Code text, 10 | Vendor_Name text, 11 | Vendor_Affiliation text, 12 | Garage_Street_Address text, 13 | Garage_City text, 14 | Garage_State text, 15 | Garage_Zip text, 16 | XCoordinates double precision, 17 | YCoordinates double precision 18 | ) -------------------------------------------------------------------------------- /open_datasets/dcp_mappluto/after.sql: -------------------------------------------------------------------------------- 1 | DROP TABLE IF EXISTS dcp_mappluto; 2 | CREATE TABLE dcp_mappluto AS 3 | SELECT * FROM mnmappluto 4 | UNION ALL 5 | SELECT * FROM bxmappluto 6 | UNION ALL 7 | SELECT * FROM qnmappluto 8 | UNION ALL 9 | SELECT * FROM bkmappluto 10 | UNION ALL 11 | SELECT * FROM simappluto 12 | ; 13 | 14 | -- -- DROP TABLE mnmappluto; 15 | -- -- DROP TABLE bxmappluto; 16 | -- -- DROP TABLE qnmappluto; 17 | -- -- DROP TABLE bkmappluto; 18 | -- -- DROP TABLE simappluto; 19 | 20 | -- UPDATE dcp_mappluto SET geom = ST_GeometryN(st_makevalid(geom),1) WHERE ST_GeometryType(ST_makevalid(geom)) = 'ST_GeometryCollection'; 21 | -------------------------------------------------------------------------------- /open_datasets/dca_facilities_operatingbusinesses/create.sql: -------------------------------------------------------------------------------- 1 | -- create table to load csv from the nyc open data portal 2 | DROP TABLE IF EXISTS dca_facilities_operatingbusinesses; 3 | CREATE TABLE dca_facilities_operatingbusinesses ( 4 | DCA_License_Number text, 5 | License_Type text, 6 | License_Expiration_Date text, 7 | License_Category text, 8 | Business_Name text, 9 | Business_Name_2 text, 10 | Address_Building text, 11 | Address_Street_Name text, 12 | Secondary_Address_Street_Name text, 13 | Address_City text, 14 | Address_State text, 15 | Address_ZIP text, 16 | Contact_Phone_Number text, 17 | Address_Borough text, 18 | Detail text, 19 | Latitude text, 20 | Longitude text 21 | ) -------------------------------------------------------------------------------- /open_datasets/nysdoh_nursinghomebedcensus/create.sql: -------------------------------------------------------------------------------- 1 | -- create table to load csv from the nyc open data portal 2 | DROP TABLE IF EXISTS nysdoh_nursinghomebedcensus; 3 | CREATE TABLE nysdoh_nursinghomebedcensus ( 4 | Facility_ID text, 5 | Facility_Name text, 6 | Certification_Number text, 7 | Street_Address text, 8 | City text, 9 | State text, 10 | Zip_Code text, 11 | County text, 12 | Area_Office text, 13 | Phone_Number text, 14 | Website text, 15 | Bed_Census_Date text, 16 | Weeks_Since_Census text, 17 | Data_Recency_Category text, 18 | Bed_Type_Service_Category text, 19 | Total_Capacity integer, 20 | Total_Available integer, 21 | Total_Available_Category text, 22 | Bed_Availability_Notes text, 23 | Location text 24 | ) -------------------------------------------------------------------------------- /open_datasets/dcp_mappluto/data.json: -------------------------------------------------------------------------------- 1 | { 2 | "url": "http://www1.nyc.gov/assets/planning/download/zip/data-maps/open-data/mappluto_16v1.zip", 3 | "table": "dcp_mappluto", 4 | "load": "shp2pgsql", 5 | "loadFiles": [ 6 | { 7 | "file": "Manhattan/MNMapPLUTO.shp", 8 | "table": "mnmappluto" 9 | }, 10 | { 11 | "file": "Bronx/BXMapPLUTO.shp", 12 | "table": "bxmappluto" 13 | }, 14 | { 15 | "file": "Brooklyn/BKMapPLUTO.shp", 16 | "table": "bkmappluto" 17 | }, 18 | { 19 | "file": "Queens/QNMapPLUTO.shp", 20 | "table": "qnmappluto" 21 | }, 22 | { 23 | "file": "Staten_Island/SIMapPLUTO.shp", 24 | "table": "simappluto" 25 | } 26 | ], 27 | "shp2pgsql": [ 28 | "-d", 29 | "-s 2263:4326" 30 | ] 31 | } -------------------------------------------------------------------------------- /open_datasets/nysdec_facilities_solidwaste/create.sql: -------------------------------------------------------------------------------- 1 | -- create table to load csv from the nyc open data portal 2 | DROP TABLE IF EXISTS nysdec_facilities_solidwaste; 3 | CREATE TABLE nysdec_facilities_solidwaste ( 4 | Facility_Name text, 5 | Location_Address text, 6 | Location_Address2 text, 7 | City text, 8 | State text, 9 | Zip_Code text, 10 | County text, 11 | Region text, 12 | Phone_Number text, 13 | Owner_Name text, 14 | Owner_Type text, 15 | Activity_Desc text, 16 | Activity_Number text, 17 | Active text, 18 | East_Coordinate double precision, 19 | North_Coordinate double precision, 20 | Accuracy_Code text, 21 | Waste_Types text, 22 | Regulatory_Status text, 23 | Authorization_Number text, 24 | Authorization_Issue_Date date, 25 | Expiration_Date date, 26 | Location text 27 | ) -------------------------------------------------------------------------------- /open_datasets/mta_busstops/data.json: -------------------------------------------------------------------------------- 1 | { 2 | "url": "https://raw.githubusercontent.com/NYCPlanning/nyc-gtfs2geojson/master/geojson.zip", 3 | "table": "mta_busstops", 4 | "load": "ogr2ogr", 5 | "loadFiles": [ 6 | { 7 | "file": "bronx/stops.geojson", 8 | "table": "bronxstops" 9 | }, 10 | { 11 | "file": "manhattan/stops.geojson", 12 | "table": "manhattanstops" 13 | }, 14 | { 15 | "file": "queens/stops.geojson", 16 | "table": "queensstops" 17 | }, 18 | { 19 | "file": "statenisland/stops.geojson", 20 | "table": "statenislandstops" 21 | }, 22 | { 23 | "file": "brooklyn/stops.geojson", 24 | "table": "brooklynstops" 25 | }, 26 | { 27 | "file": "busco/stops.geojson", 28 | "table": "buscostops" 29 | } 30 | ], 31 | "ogr2ogr": [ 32 | "-f" 33 | ] 34 | } -------------------------------------------------------------------------------- /open_datasets/nysomh_facilities_mentalhealth/create.sql: -------------------------------------------------------------------------------- 1 | -- create table to load csv from the nyc open data portal 2 | DROP TABLE IF EXISTS nysomh_facilities_mentalhealth; 3 | CREATE TABLE nysomh_facilities_mentalhealth ( 4 | Row_Created_Date_Time text, 5 | Sponsor_Name text, 6 | Sponsor_Code text, 7 | Agency_Name text, 8 | Agency_Code text, 9 | Facility_Name text, 10 | Facility_Code text, 11 | Program_Name text, 12 | Program_Code text, 13 | Populations_Served text, 14 | Agency_Phone text, 15 | Program_Phone text, 16 | Program_Address_1 text, 17 | Program_Address_2 text, 18 | Program_City text, 19 | Program_State text, 20 | Program_Zip text, 21 | Operating_Certificate_Required text, 22 | Program_Tier text, 23 | Operating_Certificate_Duration text, 24 | Program_County text, 25 | Program_Region text, 26 | Program_Type_Description text, 27 | Program_Category_Description text, 28 | Program_Subcategory_Description text, 29 | Location text 30 | ) -------------------------------------------------------------------------------- /open_datasets/nysopwdd_facilities_providers/create.sql: -------------------------------------------------------------------------------- 1 | -- create table to load csv from the nyc open data portal 2 | DROP TABLE IF EXISTS nysopwdd_facilities_providers; 3 | CREATE TABLE nysopwdd_facilities_providers ( 4 | Developmental_Disability_Services_Office text, 5 | Service_Provider_Agency text, 6 | Street_Address_ text, 7 | Street_Address_Line_2 text, 8 | City text, 9 | State text, 10 | Zip_Code text, 11 | Phone text, 12 | County text, 13 | Website_Url text, 14 | Intermediate_Care_Facilities_ICFs text, 15 | Individual_Residential_Alternative_IRA text, 16 | Family_Care text, 17 | Consolidated_Supports_And_Services text, 18 | Individual_Support_Services_ISSs text, 19 | Day_Training text, 20 | Day_Treatment text, 21 | Senior_Geriatric_Services text, 22 | Day_Habilitation text, 23 | Work_Shop text, 24 | Prevocational text, 25 | Supported_Employment_Enrollments text, 26 | Community_Habilitation text, 27 | Family_Support_Services text, 28 | Care_at_Home_Waiver_Services text, 29 | Developmental_Centers_And_Special_Population_Services text, 30 | Location_1 text 31 | ) -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | The MIT License (MIT) 2 | 3 | Copyright (c) 2017 NYC Department of City Planning 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /open_datasets/dohmh_facilities_daycare/create.sql: -------------------------------------------------------------------------------- 1 | -- create table to load csv from the nyc open data portal 2 | DROP TABLE IF EXISTS dohmh_facilities_daycare; 3 | CREATE TABLE dohmh_facilities_daycare ( 4 | Center_Name text, 5 | Legal_Name text, 6 | Building text, 7 | Street text, 8 | Borough text, 9 | ZipCode text, 10 | Phone text, 11 | Permit_Number text, 12 | Permit_Expiration text, 13 | Status text, 14 | Age_Range text, 15 | Maximum_Capacity text, 16 | Day_Care_ID text, 17 | Program_Type text, 18 | Facility_Type text, 19 | Child_Care_Type text, 20 | Building_Identification_Number text, 21 | URL text, 22 | Date_Permitted text, 23 | Actual text, 24 | Violation_Rate_Percent text, 25 | Average_Violation_Rate_Percent text, 26 | Total_Educational_Workers text, 27 | Average_Total_Educational_Workers text, 28 | Staff_Turnover_Rate text, 29 | Average_Staff_Turn_Over_Rate text, 30 | Public_Health_Hazard_Violation_Rate text, 31 | Average_Public_Health_Hazard_Violation_Rate text, 32 | Critical_Violation_Rate text, 33 | Average_Critical_Violation_Rate text, 34 | Inspection_Date text, 35 | Regulation_Summary text, 36 | Violation_Category text, 37 | Health_Code_Sub_Section text, 38 | Violation_Status text, 39 | Inspection_Summary_Result text 40 | ) -------------------------------------------------------------------------------- /open_datasets/nysdoh_facilities_healthfacilities/create.sql: -------------------------------------------------------------------------------- 1 | -- create table to load csv from the nyc open data portal 2 | DROP TABLE IF EXISTS nysdoh_facilities_healthfacilities; 3 | CREATE TABLE nysdoh_facilities_healthfacilities ( 4 | Facility_ID text, 5 | Facility_Name text, 6 | Short_Description text, 7 | Description text, 8 | Facility_Open_Date date, 9 | Facility_Address_1 text, 10 | Facility_Address_2 text, 11 | Facility_City text, 12 | Facility_State text, 13 | Facility_Zip_Code text, 14 | Facility_Phone_Number text, 15 | Facility_Fax_Number text, 16 | Facility_Website text, 17 | Facility_County_Code text, 18 | Facility_County text, 19 | Regional_Office_ID text, 20 | Regional_Office text, 21 | Main_Site_Name text, 22 | Main_Site_Facility_ID text, 23 | Operating_Certificate_Number text, 24 | Operator_Name text, 25 | Operator_Address_1 text, 26 | Operator_Address_2 text, 27 | Operator_City text, 28 | Operator_State text, 29 | Operator_ZIp_Code text, 30 | Cooperator_Name text, 31 | Cooperator_Address text, 32 | Cooperator_Address_2 text, 33 | Cooperator_City text, 34 | Cooperator_State text, 35 | Cooperator_Zip_Code text, 36 | Ownership_Type text, 37 | Facility_Latitude double precision, 38 | Facility_Longitude double precision, 39 | Facility_Location text 40 | ) -------------------------------------------------------------------------------- /open_datasets/dob_permits/create.sql: -------------------------------------------------------------------------------- 1 | -- create table to load csv from the nyc open data portal 2 | DROP TABLE IF EXISTS dob_permits; 3 | CREATE TABLE dob_permits ( 4 | borough text, 5 | binnumber text, 6 | housenumber text, 7 | streetname text, 8 | jobnumber text, 9 | jobdocnumber text, 10 | jobtype text, 11 | selfcert text, 12 | block text, 13 | lot text, 14 | communityboard text, 15 | zipcode text, 16 | buildingtype text, 17 | residential text, 18 | specialdistrict1 text, 19 | specialdistrict2 text, 20 | worktype text, 21 | permitstatus text, 22 | filingstatus text, 23 | permittype text, 24 | permitsequencenumber text, 25 | permitsubtype text, 26 | oilgas text, 27 | sitefill text, 28 | filingdate date, 29 | issuancedate date, 30 | expirationdate date, 31 | jobstartdate text, 32 | permitteefirstname text, 33 | permitteelastname text, 34 | permitteebusinessname text, 35 | permitteephone text, 36 | permiteelicensetype text, 37 | permiteelicensenum text, 38 | actassuperintendent text, 39 | permitteeothertitle text, 40 | hiclicense text, 41 | sitesafetymgrfirstname text, 42 | sitesafetymgrlastname text, 43 | sitesafetymgrbusinessname text, 44 | superintendentname text, 45 | superintendentbusinessname text, 46 | ownerbusinesstype text, 47 | nonprofit text, 48 | ownerbusinessname text, 49 | ownerfirstname text, 50 | ownerlastname text, 51 | ownerhousenum text, 52 | ownerhousestreetname text, 53 | ownerhousecity text, 54 | ownerhousestate text, 55 | ownerhousezip text, 56 | ownerphonenumber text, 57 | dobrundate date 58 | ) -------------------------------------------------------------------------------- /loader.js: -------------------------------------------------------------------------------- 1 | var fs=require('fs'); 2 | 3 | var Get = require('./lib/get.js'); 4 | var Push = require('./lib/push.js'); 5 | 6 | //get the command that the user passed in 7 | var command = process.argv[2]; 8 | 9 | //get the name of the dataset that the user passed in 10 | var dataset = process.argv[3]; 11 | 12 | //make sure the dataset config exists 13 | var open_datasets = fs.readdirSync('./open_datasets'); 14 | var other_datasets = fs.readdirSync('./other_datasets'); 15 | 16 | var dataset_directory = (open_datasets.indexOf(dataset) > -1) ? 'open_datasets' : 17 | (other_datasets.indexOf(dataset) > -1) ? 'other_datasets' : null 18 | 19 | 20 | if(dataset_directory) { 21 | 22 | var dirPath = './' + dataset_directory + '/' + dataset; 23 | 24 | //get the configuration for this dataset from its data.json file 25 | var config = require(dirPath + '/data.json') 26 | 27 | //use defined saveFile name or use the URL 28 | config.saveFile = config.saveFile || getFilename(config.url); 29 | config.writePath = './temp/' + dataset; 30 | config.dataset = dataset; 31 | config.dataset_directory = dataset_directory; 32 | 33 | //install == get + push 34 | if (command=='install') { 35 | Get(config) 36 | .then(function(){ 37 | Push(config); 38 | }); 39 | } 40 | 41 | //get == just download file 42 | if (command=='get') { 43 | Get(config); 44 | } 45 | 46 | //push == just load into db 47 | if (command=='push') { 48 | Push(config) 49 | .then(function(){ 50 | console.log('Done Pushing Dataset') 51 | }) 52 | } 53 | } else { 54 | console.log('Error: Could not find a dataset config with that name') 55 | } 56 | 57 | // get everything to the right of the last /... this should be the filename. But maybe not. 58 | function getFilename(url) { 59 | return url.split('/').pop(); 60 | } 61 | -------------------------------------------------------------------------------- /lib/get.js: -------------------------------------------------------------------------------- 1 | module.exports = function(config) { 2 | //load dependencies 3 | var download = require('download-file'), 4 | unzip = require('unzip'), 5 | fs = require('fs-extra'), 6 | Mustache = require('mustache') 7 | FTP = require('ftp-get') 8 | 9 | //get FTP credentials 10 | var ftpAuth = { 11 | username: process.env.FTPUSERNAME, 12 | password: process.env.FTPPASSWORD 13 | } 14 | 15 | 16 | console.log('Downloading dataset from ' + config.url) 17 | 18 | //check if HTTP or FTP 19 | if(config.url.indexOf('http') > -1) { 20 | return getHTTP(config); 21 | } else if (config.url.indexOf('ftp') > -1) { 22 | return getFTP(config); 23 | } 24 | 25 | 26 | 27 | function getHTTP(config) { 28 | return new Promise( 29 | function(resolve, reject) { 30 | 31 | 32 | var options = { 33 | directory: config.writePath, 34 | filename: config.saveFile 35 | } 36 | download(config.url, options, function(err){ 37 | if (err) throw err 38 | 39 | console.log('Saved file to ' + config.writePath + '/' + config.saveFile) 40 | var ext = getExtension(config.saveFile); 41 | if (ext=='zip') { 42 | 43 | console.log('Unzipping ' + config.writePath + '/' + config.saveFile) 44 | var stream = fs.createReadStream(config.writePath + '/' + config.saveFile) 45 | .pipe(unzip.Extract({ path: config.writePath })); 46 | 47 | stream.on('close', function() { 48 | resolve(); 49 | }) 50 | } else { 51 | resolve(); 52 | } 53 | }) 54 | } 55 | ) 56 | } 57 | 58 | 59 | function getFTP(config) { 60 | return new Promise( 61 | function(resolve, reject) { 62 | 63 | config.url=Mustache.render(config.url, ftpAuth) 64 | fs.emptyDirSync(config.writePath); 65 | 66 | FTP.get(config.url, config.writePath + '/' + config.saveFile, function (err, res) { 67 | if(!err) { 68 | console.log('Saved file to ' + config.writePath + '/' + config.saveFile) 69 | var ext = getExtension(config.saveFile); 70 | if (ext=='zip') { 71 | console.log('Unzipping ' + config.writePath + '/' + config.saveFile) 72 | fs.createReadStream(config.writePath + '/' + config.saveFile) 73 | .pipe(unzip.Extract({ path: config.writePath })); 74 | resolve(); 75 | } else { 76 | resolve(); 77 | } 78 | } 79 | }) 80 | } 81 | ); 82 | } 83 | 84 | // get everything to the right of the last /... this should be the filename. But maybe not. 85 | function getFilename(url) { 86 | return url.split('/').pop(); 87 | } 88 | 89 | //get file extension from filename 90 | function getExtension(filename) { 91 | return filename.split('.').pop(); 92 | } 93 | } 94 | -------------------------------------------------------------------------------- /lib/push.js: -------------------------------------------------------------------------------- 1 | require('dotenv').config(); 2 | Mustache = require('mustache'); 3 | 4 | var child = require('child_process') 5 | 6 | function execute(command) { 7 | return child.execSync(command, {stdio:[0,1,2]}) 8 | } 9 | 10 | 11 | module.exports = function(config) { 12 | console.log('Pushing into database using ' + config.load + ' method...'); 13 | 14 | return new Promise( 15 | function(resolve, reject) { 16 | 17 | //shp2pgsql method 18 | if(config.load == 'shp2pgsql') { 19 | 20 | 21 | //recursive function to run through the loadfiles one by one 22 | var loadFiles = config.loadFiles; 23 | 24 | var i=0; 25 | (function push(i) { 26 | if(i < loadFiles.length) { 27 | 28 | var command = Mustache.render('shp2pgsql {{{options}}} {{{filePath}}} {{table}} | psql {{{DATABASE_URL}}}', { 29 | options: config.shp2pgsql.join(' '), 30 | filePath: './temp/' + config.dataset + '/' + loadFiles[i].file, 31 | DATABASE_URL: process.env.DATABASE_URL, 32 | table: loadFiles[i].table 33 | }) 34 | 35 | execute(command) 36 | 37 | i++; 38 | (i==loadFiles.length) ? resolve() : push(i); 39 | } 40 | })(0); 41 | } 42 | 43 | //csv method 44 | if(config.load == 'csv') { 45 | 46 | //recursive function to run through the loadfiles one by one 47 | var loadFiles = config.loadFiles; 48 | 49 | var i=0; 50 | (function push(i) { 51 | if(i < loadFiles.length) { 52 | var filePath = './temp/' + config.dataset + '/' + loadFiles[i].file 53 | 54 | //execute create.sql and copy.sql 55 | config.csv.forEach(function(file) { 56 | console.log('Executing ' + file + '.sql'); 57 | if(file == 'create') { 58 | var command = Mustache.render('psql {{{DATABASE_URL}}} -f {{{path}}}{{file}}.sql', { 59 | DATABASE_URL: process.env.DATABASE_URL, 60 | path: config.dataset_directory + '/' + config.dataset + '/', 61 | file: file 62 | }); 63 | 64 | var response = execute(command) 65 | console.log('Done') 66 | } 67 | 68 | if(file == 'copy') { 69 | var command = Mustache.render('psql {{{DATABASE_URL}}} -c "\\COPY {{dataset}} FROM \'{{{filePath}}}\' CSV HEADER;"', { 70 | DATABASE_URL: process.env.DATABASE_URL, 71 | path: 'datasets/' + config.dataset + '/', 72 | filePath: filePath, 73 | dataset: config.dataset 74 | }); 75 | 76 | var response = execute(command) 77 | console.log('Done') 78 | } 79 | }); 80 | i++; 81 | (i==loadFiles.length) ? resolve() : push(i); 82 | } 83 | })(0); 84 | } 85 | 86 | //ogr2ogr method 87 | 88 | if(config.load == 'ogr2ogr') { 89 | 90 | //recursive function to run through the loadfiles one by one 91 | var loadFiles = config.loadFiles; 92 | 93 | var i=0; 94 | (function push(i) { 95 | console.log(i); 96 | if(i < loadFiles.length) { 97 | var filePath = './temp/' + config.dataset + '/' + loadFiles[i].file 98 | 99 | var command = Mustache.render('ogr2ogr {{options}} "PostgreSQL" PG:"{{{DATABASE_URL}}}" {{{filePath}}} -nln {{table}}',{ 100 | options: config.ogr2ogr.join(' '), 101 | filePath: filePath, 102 | DATABASE_URL: process.env.DATABASE_URL, 103 | table: loadFiles[i].table 104 | }); 105 | 106 | execute(command) 107 | i++; 108 | (i==loadFiles.length) ? resolve() : push(i); 109 | 110 | } 111 | })(0); 112 | } 113 | } 114 | ) 115 | } 116 | -------------------------------------------------------------------------------- /readme.MD: -------------------------------------------------------------------------------- 1 | #Civic Data Loader 2 | 3 | A node.js CLI tool for automated dataset loading into the PostgreSQL/PostGIS, making the full refresh of a dataset as simple as `node loader install {datasetName}` Inspired by John Krauss' [docker4data](https://github.com/talos/docker4data) 4 | 5 | ##Structure 6 | 7 | `datasetName` should be unique, and takes the form of {agency}_{dataset}, e.g. `dcp_mappluto`. Each dataset should have a folder with its datasetName in the `open_datasets` directory. When executing the CLI tool, the datasetName is passed in as an argument, and the script searches for `{datasetName}/data.json` and then runs accordingly. 8 | 9 | ##data.json 10 | 11 | `data.json` includes everything the script needs to download, process, and load the dataset. An example `data.json` for NYC borough boundaries looks like: 12 | 13 | ``` 14 | { 15 | "url": "http://www1.nyc.gov/assets/planning/download/zip/data-maps/open-data/nybb16b.zip", 16 | 17 | "load": "shp2pgsql", 18 | "loadFiles": [ 19 | { 20 | "file":"nybb_16b/nybb.shp", 21 | "table": "dcp_boroughboundaries" 22 | } 23 | ], 24 | "shp2pgsql": [ 25 | "-d", 26 | "-s 2263:4326" 27 | ] 28 | } 29 | ``` 30 | - `url` - the download link, grabbed when using the `Get` method. 31 | - `saveFile` - optional - the name of the file to save after a Get. (useful when the downloaded file has a generic name or does not have the correct file extension) 32 | - `load` - which loading type to use. `shp2pgsql`, `csv`, and `ogr2ogr` are the available options, see "Loading Types" below. 33 | - `loadFiles` - an array of files to be loaded when using the `Push` method. Most datasets will just have one, but some, like pluto, have a single file to download but many files to upload. 34 | - `shp2pgsql` - an array of arguments for the shp2pgsql command. Only used if `load` is set to 'shp2pgsql' 35 | 36 | ##How to Use 37 | 38 | - Clone this repo 39 | - Install dependencies `npm install` 40 | - Rename `.env.sample` to `.env` and enter the postgres connection string as `DATABASE_URL`: 41 | `DATABASE_URL=postgres://user:password@host:port/database` 42 | - Enter `node loader {command} {datasetName}` in the terminal, like `node loader install dcp_boroboundaries` 43 | - Watch the magic happen! 44 | 45 | ##Commands 46 | 47 | The workflows are divided into 2 parts: `get`, `push`. 48 | 49 | - `get` downloads the file to the temp directory and unzips if necessary 50 | - `push` grabs the file(s) from the temp directory and loads it(them) into the database 51 | - `install` does both `get` and `push` 52 | 53 | 54 | ##Loading Types: 55 | 56 | `shp2pgsql` - For Shapefiles. In the case of much of NYC's open data, a conversion from NY state plane to WGS84 is required. (this is why the `-s 2263:4326` flag is set in the example above) 57 | 58 | `csv` - For CSVs. If this type is chosen, you must also provide `create.sql` and `copy.sql` in the dataset's directory. 59 | 60 | `ogr2ogr` - Useful for loading geojson files. File should be a geojson FeatureCollection. 61 | 62 | ##create.sql 63 | If loading csvs, this is simply a CREATE TABLE statement that is run before loading the data. In the future we could use ogr2ogr to do column type guessing if people want it. 64 | ``` 65 | -- create table to load csv from the nyc open data portal 66 | DROP TABLE IF EXISTS nysopwdd_facilities_providers; 67 | CREATE TABLE nysopwdd_facilities_providers ( 68 | Developmental_Disability_Services_Office text, 69 | Service_Provider_Agency text, 70 | Street_Address_ text, 71 | Street_Address_Line_2 text, 72 | City text, 73 | State text, 74 | Zip_Code text, 75 | Phone text, 76 | County text, 77 | Website_Url text, 78 | Intermediate_Care_Facilities_ICFs text, 79 | Individual_Residential_Alternative_IRA text, 80 | Family_Care text, 81 | Consolidated_Supports_And_Services text, 82 | Individual_Support_Services_ISSs text, 83 | Day_Training text, 84 | Day_Treatment text, 85 | Senior_Geriatric_Services text, 86 | Day_Habilitation text, 87 | Work_Shop text, 88 | Prevocational text, 89 | Supported_Employment_Enrollments text, 90 | Community_Habilitation text, 91 | Family_Support_Services text, 92 | Care_at_Home_Waiver_Services text, 93 | Developmental_Centers_And_Special_Population_Services text, 94 | Location_1 text 95 | ) 96 | ``` 97 | ##copy.sql 98 | If loading csvs, this is the copy statement with the source file and the destination table. 99 | `\COPY nysopwdd_facilities_providers FROM './temp/nysopwdd_facilities_providers/nysopwdd_facilities_providers.csv' CSV HEADER;` 100 | 101 | 102 | ##Non-public data 103 | 104 | The loader will search for a matching dataset in both the `open_datasets` and `other_datasets` directories. `open_datasets` is included in this repo, but the contents of `other_datasets` is ignored by git, so you can use it to create loading configurations for your non-publicly accessible data. 105 | 106 | 107 | TODO: 108 | - lots of error handling 109 | - add all the datas! 110 | - add an "after" handler that can execute SQL after the push is successful the data have been loaded. 111 | -------------------------------------------------------------------------------- /open_datasets/dcp_councildistricts/nyccwi_16d/nyccwi.shp.xml: -------------------------------------------------------------------------------- 1 | 20161109111918001.0ISO 19139 Metadata Implementation SpecificationFALSEnyccwiwithheldLocal Area Network002912287.0687871067382.508606116411.371399273617.84320110.000ProjectedGCS_North_American_1983Linear Unit: Foot_US (0.304801)NAD_1983_StatePlane_New_York_Long_Island_FIPS_3104_Feet<ProjectedCoordinateSystem xsi:type='typens:ProjectedCoordinateSystem' xmlns:xsi='http://www.w3.org/2001/XMLSchema-instance' xmlns:xs='http://www.w3.org/2001/XMLSchema' xmlns:typens='http://www.esri.com/schemas/ArcGIS/10.3'><WKT>PROJCS[&quot;NAD_1983_StatePlane_New_York_Long_Island_FIPS_3104_Feet&quot;,GEOGCS[&quot;GCS_North_American_1983&quot;,DATUM[&quot;D_North_American_1983&quot;,SPHEROID[&quot;GRS_1980&quot;,6378137.0,298.257222101]],PRIMEM[&quot;Greenwich&quot;,0.0],UNIT[&quot;Degree&quot;,0.0174532925199433]],PROJECTION[&quot;Lambert_Conformal_Conic&quot;],PARAMETER[&quot;False_Easting&quot;,984250.0],PARAMETER[&quot;False_Northing&quot;,0.0],PARAMETER[&quot;Central_Meridian&quot;,-74.0],PARAMETER[&quot;Standard_Parallel_1&quot;,40.66666666666666],PARAMETER[&quot;Standard_Parallel_2&quot;,41.03333333333333],PARAMETER[&quot;Latitude_Of_Origin&quot;,40.16666666666666],UNIT[&quot;Foot_US&quot;,0.3048006096012192],AUTHORITY[&quot;EPSG&quot;,2263]]</WKT><XOrigin>-120039300</XOrigin><YOrigin>-96540300</YOrigin><XYScale>37212589.015695661</XYScale><ZOrigin>-100000</ZOrigin><ZScale>10000</ZScale><MOrigin>-100000</MOrigin><MScale>10000</MScale><XYTolerance>0.0032808333333333331</XYTolerance><ZTolerance>0.001</ZTolerance><MTolerance>0.001</MTolerance><HighPrecision>true</HighPrecision><WKID>102718</WKID><LatestWKID>2263</LatestWKID></ProjectedCoordinateSystem>Select c:\temp\BYTES_GP\Districts.gdb\nyccwi C:\temp\16D\GIS_OUTPUT\Districts\shp\nyccwi__Iteration.shp "ObjectID>=0 AND ObjectID<3000"Select C:\temp\16D\GIS_OUTPUT\Districts\shp\nyccwi__Iteration.shp C:\temp\16D\GIS_OUTPUT\Districts\shp\nyccwi.shp #201611091127130020161109112713001500000005000NYC Department of City PlanningBYTES of the BIG APPLE Coordinator212.720.3505120 Broadway, 31st FloorNew YorkNew York10271New York City Department of City Planning120 Broadway, 31st FloorNew YorkNew York10271USAvailable at the following website: http://www.nyc.gov/html/dcp/html/bytes/applbyte.shtmlFreehttp://www.nyc.gov/html/dcp/html/bytes/applbyte.shtml0.000ESRI ShapefileOpen SpecificationNew York City Council Districts Water Included16DNew York City Department of City PlanningNew York City Department of City PlanningBYTES of the BIG APPLE16DNo major changes since previous version.10/27/201610/27/2016<DIV STYLE="text-align:Left;"><DIV><P><SPAN>The City Council Districts are the result of the recent redistricting process, which takes place every ten years to reflect population changes reported in the 2000 Census. These geographies were redrawn by the New York City Council Redistricting Commission.</SPAN></P></DIV></DIV>These districts were created by the Department of City Planning to aid city agencies in administering public services.New York City Department of City PlanningBYTES of the BIG APPLE Coordinator212.720.3505120 Broadway, 31st FloorNew YorkNY10271USQueensNew York CityRichmondKingsBrooklynBronxNew YorkStaten IslandManhattanCity CouncilboundariesPolitical BoundariesQueensNew York CityCity CouncilRichmondboundariesKingsBrooklynBronxNew YorkPolitical BoundariesStaten IslandManhattanThe data is freely available to all New York City agencies and the public.<DIV STYLE="text-align:Left;"><P><SPAN>This dataset is being provided by the Department of City Planning (DCP) on DCP’s website for informational purposes only. DCP does not warranty the completeness, accuracy, content, or fitness for any particular purpose or use of the dataset, nor are any such warranties to be implied or inferred with respect to the dataset as furnished on the website. DCP and the City are not liable for any deficiencies in the completeness, accuracy, content, or fitness for any particular purpose or use the dataset, or applications utilizing the dataset, provided by any third party.</SPAN></P></DIV>ground conditionDepartment of City Planning.Microsoft Windows 7 Version 6.1 (Build 7601) Service Pack 1; Esri ArcGIS 10.3.1.49591-74.260380-73.69920640.91769140.485808The District files are created from the Department of City Planning LION file of the same release. The LION file is spatially aligned with the NYCMap aerial photography.nyccwiFeature Class0FIDFIDOID400Internal feature number.EsriSequential unique whole numbers that are automatically generated.COUNDISTCounDistSmallInteger550City Council District NumberShapeShapeGeometry000Feature geometry.ESRICoordinates defining the features.Shape_LengShape_LengDouble1900Shape_AreaShape_AreaDouble1900Area of feature in internal units squared.EsriPositive real numbers that are automatically generated.datasetEPSG8.6.20SimpleFALSE0FALSEFALSE20161109 2 | -------------------------------------------------------------------------------- /open_datasets/dcp_censustracts/nyct2010wi_16d/nyct2010wi.shp.xml: -------------------------------------------------------------------------------- 1 | 20161109111907001.0ISO 19139 Metadata Implementation SpecificationFALSEnyct2010wiwithheldLocal Area Network002912287.0687871067382.508423116411.371399273617.84320110.000ProjectedGCS_North_American_1983Linear Unit: Foot_US (0.304801)NAD_1983_StatePlane_New_York_Long_Island_FIPS_3104_Feet<ProjectedCoordinateSystem xsi:type='typens:ProjectedCoordinateSystem' xmlns:xsi='http://www.w3.org/2001/XMLSchema-instance' xmlns:xs='http://www.w3.org/2001/XMLSchema' xmlns:typens='http://www.esri.com/schemas/ArcGIS/10.3'><WKT>PROJCS[&quot;NAD_1983_StatePlane_New_York_Long_Island_FIPS_3104_Feet&quot;,GEOGCS[&quot;GCS_North_American_1983&quot;,DATUM[&quot;D_North_American_1983&quot;,SPHEROID[&quot;GRS_1980&quot;,6378137.0,298.257222101]],PRIMEM[&quot;Greenwich&quot;,0.0],UNIT[&quot;Degree&quot;,0.0174532925199433]],PROJECTION[&quot;Lambert_Conformal_Conic&quot;],PARAMETER[&quot;False_Easting&quot;,984250.0],PARAMETER[&quot;False_Northing&quot;,0.0],PARAMETER[&quot;Central_Meridian&quot;,-74.0],PARAMETER[&quot;Standard_Parallel_1&quot;,40.66666666666666],PARAMETER[&quot;Standard_Parallel_2&quot;,41.03333333333333],PARAMETER[&quot;Latitude_Of_Origin&quot;,40.16666666666666],UNIT[&quot;Foot_US&quot;,0.3048006096012192],AUTHORITY[&quot;EPSG&quot;,2263]]</WKT><XOrigin>-120039300</XOrigin><YOrigin>-96540300</YOrigin><XYScale>37212589.015695661</XYScale><ZOrigin>-100000</ZOrigin><ZScale>10000</ZScale><MOrigin>-100000</MOrigin><MScale>10000</MScale><XYTolerance>0.0032808333333333331</XYTolerance><ZTolerance>0.001</ZTolerance><MTolerance>0.001</MTolerance><HighPrecision>true</HighPrecision><WKID>102718</WKID><LatestWKID>2263</LatestWKID></ProjectedCoordinateSystem>Select c:\temp\BYTES_GP\Districts.gdb\nyct2010wi C:\temp\16D\GIS_OUTPUT\Districts\shp\nyct2010wi__Iteration.shp "ObjectID>=0 AND ObjectID<3000"Select C:\temp\16D\GIS_OUTPUT\Districts\shp\nyct2010wi__Iteration.shp C:\temp\16D\GIS_OUTPUT\Districts\shp\nyct2010wi.shp #201611091127370020161109112737001500000005000New York City Department of City PlanningBYTES of the BIG APPLE Coordinator212.720.3505120 Broadway, 31st FloorNew YorkNew York10271USNew York City Department of City Planning120 Broadway, 31st FloorNew YorkNew York10271USAvailable at the following website: http://www.nyc.gov/html/dcp/html/bytes/applbyte.shtmlFreehttp://www.nyc.gov/html/dcp/html/bytes/applbyte.shtml0.000ESRI ShapefileOpen SpecificationNew York City Census Tracts for 2010 US Census Water Included16DNew York City Department of City PlanningNew York City Department of City PlanningBYTES of the BIG APPLE16D10/27/201610/27/2016<DIV STYLE="text-align:Left;"><DIV><P><SPAN>The Census Tracts for the 2010 US Census. These boundary files are derived from the US Census Bureau's TIGER project and have been geographically modified to fit the New York City base map.</SPAN></P></DIV></DIV>These districts were created by the Department of City Planning to aid city agencies in administering public services.New York City Department of City PlanningBYTES of the BIG APPLE Coordinator212.720.3505120 Broadway, 31st FloorNew YorkNY10271USNew YorkManhattanQueensBrooklynRichmondBronxKingsStaten IslandNew York CityboundariesboundaryboroughNew YorkManhattanQueensBrooklynRichmondboundariesBronxKingsboundaryboroughStaten IslandNew York CityThe data is freely available to all New York City agencies and the public.<DIV STYLE="text-align:Left;"><P><SPAN>This dataset is being provided by the Department of City Planning (DCP) on DCP’s website for informational purposes only. DCP does not warranty the completeness, accuracy, content, or fitness for any particular purpose or use of the dataset, nor are any such warranties to be implied or inferred with respect to the dataset as furnished on the website. DCP and the City are not liable for any deficiencies in the completeness, accuracy, content, or fitness for any particular purpose or use the dataset, or applications utilizing the dataset, provided by any third party.</SPAN></P></DIV>ground conditionDepartment of City Planning.Microsoft Windows 7 Version 6.1 (Build 7601) Service Pack 1; Esri ArcGIS 10.3.1.49591-74.260380-73.69920640.91769140.485808These data are accurate as of US Census 2010.The District files are created from the same release version of the Department of City Planning LION file. The LION file is spatially aligned with NYCMap aerial photography.nyct2010wiFeature Class0FIDFIDOID400Internal feature number.EsriSequential unique whole numbers that are automatically generated.CTLabelCTLabelString7The census tract identifier for the polygon. Each census tract number is unique to its borough.00BoroCodeBoroCodeString1Borough of census tract.1Manhattan2Bronx3Brooklyn4Queens5Staten Island00BoroNameBoroNameString32Borough Name.00CDEligibilCDEligibilString1Community Development Block Grant Eligibility.00CT2010CT2010String6String value of the census tract number.00BoroCT2010BoroCT2010String7Merged string of borough code and census tract number.00PUMAPUMAString4Public Use Microdata Areas00ShapeShapeGeometry000Feature geometry.ESRICoordinates defining the features.SHAPE_AreaShape_AreaDouble19Area of feature in internal units squared.ESRIPositive real numbers that are automatically generated.00NTACodeNTACodeString400Neighborhood Tabulation Area CodeShape_LengShape_LengDouble1900NTANameNTANameString7500Neighborhood Tabulation Area NamedatasetEPSG8.6.20SimpleFALSE0FALSEFALSE20161109 2 | --------------------------------------------------------------------------------