├── .github └── workflows │ └── ci.yml ├── .gitignore ├── .markdownlint.yml ├── .readthedocs.yml ├── CHANGELOG.md ├── LICENSE ├── README.md ├── bin ├── ci.sh ├── clean.sh └── html.sh ├── build └── build.sh ├── docs ├── admin.rst ├── app-config │ ├── configure-custom-pivots.md │ ├── configure-dashboards.md │ ├── configure-investigation.md │ ├── configure-ontology.md │ ├── configure-pygraphistry.md │ ├── configure-python.md │ ├── configure.md │ ├── connector-management.md │ ├── email.md │ ├── index.rst │ └── templates.md ├── commands.md ├── conf.py ├── config-files │ ├── config.json │ └── daemon.json ├── debugging │ ├── debug-container-networking.md │ ├── debug-faq.md │ ├── debug-logs.md │ ├── index.rst │ ├── performance-tuning.md │ └── system-metrics-to-csv.sh ├── faq.md ├── index.rst ├── install │ ├── cloud │ │ ├── aws.md │ │ ├── aws_marketplace.md │ │ ├── azure.md │ │ ├── azure_marketplace.md │ │ ├── azure_marketplace_private_offer.md │ │ └── index.rst │ ├── cluster │ │ ├── docker-compose-mode.md │ │ ├── index.rst │ │ └── kubernetes-mode.md │ ├── on-prem │ │ ├── index.rst │ │ ├── manual.md │ │ ├── rhel8_prereqs_install.sh │ │ ├── rhel_7_6_setup.md │ │ ├── ubuntu_18_04_lts_setup.md │ │ ├── ubuntu_20_04_setup.sh │ │ ├── ubuntu_22_04_setup.sh │ │ └── vGPU.md │ └── testing-an-install.md ├── planning │ ├── architecture.md │ ├── benchmarking.md │ ├── browser.md │ ├── deployment-planning.md │ ├── hardware-software.md │ └── index.rst ├── security │ ├── SSO.md │ ├── authentication.md │ ├── configure-security.md │ ├── configure-tls-caddy-manual-lets-encrypt-handshake.md │ ├── index.rst │ └── threatmodel.md ├── static │ ├── favicon.ico │ ├── graphistry.css │ ├── graphistry_banner_transparent_colored.png │ └── img │ │ ├── Louie │ │ └── Databricks_SSO_App_Connection.png │ │ ├── OIDC_setup │ │ ├── oidc_setup_auth0_1_1.png │ │ ├── oidc_setup_auth0_1_2.png │ │ ├── oidc_setup_auth0_1_3.png │ │ ├── oidc_setup_auth0_1_4.png │ │ ├── oidc_setup_auth0_1_5.png │ │ ├── oidc_setup_auth0_1_6.png │ │ ├── oidc_setup_auth0_1_7.png │ │ ├── oidc_setup_auth0_1_8.png │ │ ├── oidc_setup_auth0_2_1.png │ │ ├── oidc_setup_auth0_2_2.png │ │ ├── oidc_setup_auth0_3_1.png │ │ ├── oidc_setup_auth0_3_2.png │ │ ├── oidc_setup_auth0_3_3.png │ │ ├── oidc_setup_auth0_3_4.png │ │ ├── oidc_setup_auth0_3_5.png │ │ ├── oidc_setup_auth0_3_6.png │ │ ├── oidc_setup_auth0_3_7.png │ │ ├── oidc_setup_auth0_3_8.png │ │ ├── oidc_setup_graphistry_1_1.png │ │ ├── oidc_setup_graphistry_1_2.png │ │ ├── oidc_setup_graphistry_1_3.png │ │ ├── oidc_setup_graphistry_1_4.png │ │ ├── oidc_setup_graphistry_1_5.png │ │ ├── oidc_setup_graphistry_1_6.png │ │ ├── oidc_setup_graphistry_1_7.png │ │ ├── oidc_setup_graphistry_1_8.png │ │ ├── oidc_setup_keycloak_1_1.png │ │ ├── oidc_setup_keycloak_1_2.png │ │ ├── oidc_setup_keycloak_1_3.png │ │ ├── oidc_setup_keycloak_1_4.png │ │ ├── oidc_setup_keycloak_1_5.png │ │ ├── oidc_setup_keycloak_1_6.png │ │ ├── oidc_setup_keycloak_1_7.png │ │ ├── oidc_setup_keycloak_2_1.png │ │ ├── oidc_setup_keycloak_2_2.png │ │ ├── oidc_setup_keycloak_2_3.png │ │ ├── oidc_setup_okta_1_1.png │ │ ├── oidc_setup_okta_1_10.png │ │ ├── oidc_setup_okta_1_11.png │ │ ├── oidc_setup_okta_1_2.png │ │ ├── oidc_setup_okta_1_3.png │ │ ├── oidc_setup_okta_1_4.png │ │ ├── oidc_setup_okta_1_5.png │ │ ├── oidc_setup_okta_1_6.png │ │ ├── oidc_setup_okta_1_7.png │ │ ├── oidc_setup_okta_1_8.png │ │ ├── oidc_setup_okta_1_9.png │ │ ├── oidc_setup_okta_2_1.png │ │ ├── oidc_setup_okta_2_2.png │ │ ├── oidc_setup_okta_2_3.png │ │ ├── oidc_setup_okta_3_1.png │ │ ├── oidc_setup_okta_3_2.png │ │ ├── oidc_setup_okta_3_3.png │ │ ├── oidc_setup_okta_3_4.png │ │ └── oidc_setup_okta_3_5.png │ │ ├── add_user.png │ │ ├── admin.png │ │ ├── cfg_users.png │ │ ├── grafana-import-dcgm-dashboard-6.png │ │ ├── jaeger-inspecting-trace-with-error.png │ │ ├── jaeger-trace-list-including-trace-with-error.png │ │ ├── prometheus-forge-etl-python-metric-example.png │ │ ├── set_creds.png │ │ ├── set_roles.png │ │ └── signup.png ├── support.rst ├── telemetry │ ├── docker-compose.md │ ├── index.rst │ └── kubernetes.md └── tools │ ├── backup-and-restore.md │ ├── bridge.md │ ├── developer.md │ ├── index.rst │ ├── nvidia-docker-in-docker.md │ ├── update-backup-migrate.md │ └── user-creation.md └── infra ├── Dockerfile ├── build-docs.sh ├── docker-compose.yml ├── requirements-python.txt └── requirements-system.txt /.github/workflows/ci.yml: -------------------------------------------------------------------------------- 1 | name: CI Tests 2 | 3 | on: 4 | 5 | #NOTE: All jobs gated by auth job 6 | 7 | #Regular dev 8 | push: 9 | pull_request: 10 | 11 | #Enable UI-driven branch testing 12 | workflow_dispatch: 13 | 14 | #Test main bidaily @ 1a 15 | schedule: 16 | - cron: '0 1 1-31/2 * *' 17 | 18 | jobs: 19 | 20 | test-docs: 21 | 22 | runs-on: ubuntu-latest 23 | 24 | steps: 25 | - uses: actions/checkout@v3 26 | 27 | - name: Test building docs 28 | run: | 29 | ./bin/ci.sh 30 | 31 | test-readme: 32 | 33 | runs-on: ubuntu-latest 34 | 35 | steps: 36 | - uses: actions/checkout@v3 37 | 38 | - name: Set up Python 3.8 39 | uses: actions/setup-python@v4 40 | with: 41 | python-version: 3.8 42 | 43 | - name: Test building docs 44 | continue-on-error: true 45 | run: | 46 | docker run --rm -v "$(pwd)/README.md:/workdir/README.md:ro" -v "$(pwd)/.markdownlint.yaml:/workdir/.markdownlint.yaml:ro" ghcr.io/igorshubovych/markdownlint-cli:v0.37.0 README.md 47 | 48 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | _build/ 2 | doctrees/ 3 | -------------------------------------------------------------------------------- /.readthedocs.yml: -------------------------------------------------------------------------------- 1 | # .readthedocs.yml 2 | # Read the Docs configuration file 3 | # See https://docs.readthedocs.io/en/stable/config-file/v2.html for details 4 | 5 | # Required 6 | version: 2 7 | 8 | build: 9 | os: ubuntu-22.04 10 | tools: 11 | python: "3.12" 12 | apt_packages: 13 | # More closely mirror https://github.com/sphinx-doc/sphinx-docker-images 14 | - graphviz 15 | - imagemagick 16 | - make 17 | - pandoc 18 | - texlive-latex-base 19 | - texlive-latex-recommended 20 | - texlive-latex-extra 21 | - texlive-fonts-recommended 22 | commands: 23 | 24 | # setup 25 | - pip install "graphistry[docs]" 26 | - pip install -r infra/requirements-python.txt 27 | #- cp -r demos docs/source/demos 28 | - cp README.md docs/README.md 29 | 30 | # build html 31 | - sphinx-build -b html -d doctrees docs $READTHEDOCS_OUTPUT/html/ 32 | 33 | # build epub 34 | - sphinx-build -b epub -d doctrees docs _build/latexpdf 35 | - mkdir -p $READTHEDOCS_OUTPUT/epub 36 | - ls _build/latexpdf 37 | - ls $READTHEDOCS_OUTPUT/epub 38 | - cp _build/latexpdf/GraphistryCLI.epub $READTHEDOCS_OUTPUT/epub/GraphistryCLI.epub 39 | 40 | # build pdf 41 | - sphinx-build -b latex -d doctrees docs _build/latexpdf 42 | - ls _build/latexpdf 43 | - cd _build/latexpdf && (pdflatex -file-line-error -interaction=nonstopmode Graphistry.tex || test -f _build/latexpdf/Graphistry.pdf) && (pdflatex -file-line-error -interaction=nonstopmode Graphistry.tex || test -f _build/latexpdf/Graphistry.pdf) && echo ok || { echo fail && exit 1 ; } 44 | - mkdir -p $READTHEDOCS_OUTPUT/pdf 45 | - cp _build/latexpdf/Graphistry.pdf $READTHEDOCS_OUTPUT/pdf/Graphistry.pdf 46 | 47 | 48 | #for nav links? 49 | formats: 50 | - pdf 51 | - epub 52 | - htmlzip 53 | 54 | python: 55 | install: 56 | - method: pip 57 | path: . 58 | extra_requirements: 59 | - docs 60 | -------------------------------------------------------------------------------- /CHANGELOG.md: -------------------------------------------------------------------------------- 1 | # Changelog 2 | 3 | All notable changes to this repo are documented in this file. The Graphistry server is tracked in the main [Graphistry major release history documentation](https://graphistry.zendesk.com/hc/en-us/articles/360033184174-Enterprise-Release-List-Downloads). 4 | 5 | The changelog format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/). 6 | This project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html) and all PyGraphistry-specific breaking changes are explictly noted here. 7 | 8 | ## [Development] 9 | 10 | ## [v2.41.8 - 2024-10-27] 11 | 12 | ### Added 13 | 14 | * ReadTheDocs site: [https://graphistry-admin-docs.readthedocs.io/](https://graphistry-admin-docs.readthedocs.io/) 15 | * Python endpoint 16 | 17 | ### Changed 18 | 19 | * Reorganized documentation into thematic areas 20 | * Update RAPIDS base image 21 | * Update RAPIDS Python env name to `base` from `rapids` 22 | * Updated most docker compose command references to `docker compose` from `docker-compose` 23 | 24 | ### Infra 25 | 26 | * Sphinx port 27 | * CI 28 | 29 | ### Fixed 30 | 31 | * Telemetry images 32 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Copyright (c) 2019, Graphistry, Inc. 2 | All rights reserved. 3 | 4 | Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: 5 | 6 | 1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. 7 | 8 | 2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. 9 | 10 | 3. Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. 11 | 12 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 13 | 14 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Welcome to Graphistry: Admin Guide 2 | 3 | Graphistry is the most scalable graph-based visual analysis and investigation automation platform. It supports both cloud and on-prem deployment options. Big graphs are tons of fun! 4 | 5 | 6 | ## Quick administration links 7 | 8 | * [Top commands](https://graphistry-admin-docs.readthedocs.io/en/latest/commands.html) 9 | * [Plan deployments](https://graphistry-admin-docs.readthedocs.io/en/latest/planning/hardware-software.html) 10 | * Install: [Cloud](https://graphistry-admin-docs.readthedocs.io/en/latest/install/cloud/index.html) & [On-prem](https://graphistry-admin-docs.readthedocs.io/en/latest/install/on-prem/index.html) 11 | * [Configure](https://graphistry-admin-docs.readthedocs.io/en/latest/app-config/index.html) 12 | * [Debugging & performance](https://graphistry-admin-docs.readthedocs.io/en/latest/debugging/index.html) 13 | * [Security](https://graphistry-admin-docs.readthedocs.io/en/latest/security/index.html) 14 | * [Operations & tools](https://graphistry-admin-docs.readthedocs.io/en/latest/tools/index.html) 15 | * [FAQ](https://graphistry-admin-docs.readthedocs.io/en/latest/faq/index.html) & [support options](https://graphistry-admin-docs.readthedocs.io/en/latest/support.html) 16 | 17 | ## Further reading 18 | 19 | * [Main Graphistry documentation](https://hub.graphistry.com/docs) and same path on your local server 20 | * [Release portal](https://graphistry.zendesk.com/hc/en-us/articles/360033184174) for enterprise admins to download the latest 21 | * [Release notes](https://graphistry.zendesk.com/hc/en-us/articles/360033184174) 22 | * [Graphistry Hub](https://hub.graphistry.com): Graphistry-managed GPU servers, including free and team tiers 23 | * Docker (self-hosted): See [enterprise release portal](https://graphistry.zendesk.com/hc/en-us/articles/360033184174) 24 | * [Kubernetes Helm charts](https://github.com/graphistry/graphistry-helm) - Experimental 25 | 26 | 27 | ## Quick GPU Docker environment test 28 | 29 | You can test your GPU environment via Graphistry's [base RAPIDS Docker image on DockerHub](https://hub.docker.com/r/graphistry/graphistry-forge-base): 30 | 31 | ```bash 32 | docker run --rm -it --entrypoint=/bin/bash graphistry/graphistry-forge-base:latest -c "source activate base && python3 -c \"import cudf; print(cudf.DataFrame({'x': [0,1,2]})['x'].sum())\"" 33 | ``` 34 | 35 | => 36 | ``` 37 | 3 38 | ``` 39 | 40 | See the installation and debugging sections for additional scenarios such as ensuring Docker Compose is correctly defaulting to a GPU runtime. 41 | 42 | 43 | -------------------------------------------------------------------------------- /bin/ci.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | set -ex 3 | 4 | DOCS_FORMAT=${DOCS_FORMAT:-all} # Default to building all formats if not specified 5 | 6 | ( 7 | cd infra \ 8 | && docker compose build \ 9 | && docker compose run --rm -e DOCS_FORMAT=$DOCS_FORMAT sphinx 10 | ) 11 | -------------------------------------------------------------------------------- /bin/clean.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | set -ex 3 | 4 | SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" 5 | 6 | # check if _build/ and doctrees exist 7 | if [ ! -d "${SCRIPT_DIR}/../_build/" ]; then 8 | echo "Directory _build/ does not exist, nothing to clean" 9 | else 10 | rm -rf "${SCRIPT_DIR}/../_build/" 11 | fi 12 | 13 | if [ ! -d "${SCRIPT_DIR}/../doctrees/" ]; then 14 | echo "Directory doctrees/ does not exist, nothing to clean" 15 | else 16 | rm -rf "${SCRIPT_DIR}/../doctrees/" 17 | fi 18 | 19 | echo "Cleaned up build artifacts." 20 | -------------------------------------------------------------------------------- /bin/html.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | set -ex 3 | 4 | SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" 5 | 6 | ci_file="$script_dir/ci.html" 7 | 8 | DOCS_FORMAT=html "${SCRIPT_DIR}/ci.sh" 9 | -------------------------------------------------------------------------------- /build/build.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | ### Generates ~/readme.pdf and ~/hardware-software.pdf 4 | ### ~/build $ ./build.sh 5 | 6 | echo "==== BUILDING readme.pdf ====" 7 | DOCS=$(ls docs/*.md) 8 | docker run --rm -it -v $PWD/..:/source jagregory/pandoc --toc -V documentclass=report -s \ 9 | README.md \ 10 | $DOCS \ 11 | -o readme.pdf 12 | 13 | echo "==== BUILDING hardware-software.pdf ====" 14 | docker run --rm -it -v $PWD/..:/source jagregory/pandoc --toc -V documentclass=report -s \ 15 | docs/hardware-software.md \ 16 | -o hardware-software.pdf 17 | 18 | ls -al ../*.pdf 19 | -------------------------------------------------------------------------------- /docs/admin.rst: -------------------------------------------------------------------------------- 1 | .. toctree:: 2 | :maxdepth: 2 3 | :hidden: 4 | 5 | README 6 | 7 | 8 | .. toctree:: 9 | :maxdepth: 2 10 | 11 | commands 12 | planning/index 13 | install/cloud/index 14 | install/on-prem/index 15 | install/cluster/index 16 | install/testing-an-install 17 | app-config/index 18 | debugging/index 19 | security/index 20 | telemetry/index 21 | tools/index 22 | faq 23 | support 24 | -------------------------------------------------------------------------------- /docs/app-config/configure-dashboards.md: -------------------------------------------------------------------------------- 1 | # Configure Dashboards 2 | 3 | Graphistry maintains [graph-app-kit](http://github.com/graphistry/graph-app-kit), which extends [Streamlit](https://streamlit.io/) with Docker, multi-app, graph, and other practical extensions. 4 | 5 | ## Default configuration 6 | 7 | ### Public dashboards 8 | 9 | * Publicly available at `public/dash/` (note the trailing slash) 10 | * Files stored at `./data/public_views` 11 | * Web editing: Users with Jupyter notebook access can edit views in the Jupyter folder `graph-app-kit-public` 12 | 13 | ### Private dashboards 14 | 15 | * Available to site staff at `private/dash/` (note the trailing slash) 16 | * Files stored at `./data/private_views` 17 | * Web editing: Users with Jupyter notebook access can edit views in the Jupyter folder `graph-app-kit-private` 18 | 19 | ## Disable dashboards 20 | 21 | 1. Remove one or both dashboards from the menus: 22 | * Go to the admin dashboard: `django-waffle` -> `flags` -> `flag_show_{public,private}_dashboard` 23 | * Set `Everyone` to `No` 24 | * `Save` 25 | 26 | 2. Disable the dashboard services: 27 | * Edit your `docker-compose.override.yml` 28 | * Services: `graph-app-kit-public` and `graph-app-kit-private` 29 | 30 | ## Add or modify python libraries 31 | 32 | Python libraries can be added or modified as desired. See [graph-app-kit documentation](https://github.com/graphistry/graph-app-kit/blob/master/docs/additional-packages.md) for more information on how to use this feature. 33 | -------------------------------------------------------------------------------- /docs/app-config/configure-ontology.md: -------------------------------------------------------------------------------- 1 | # Configure Custom Graphistry Ontology 2 | 3 | The Graphistry ontology is a set of mappings that guide automatic visualization and simplify data worklows. This document overviews the ontology and how to extend. For the formal format, see Graphistry's [convict-format specification](https://github.com/graphistry/graphistry-cli/blob/master/docs/configure-investigation.md#schema). 4 | 5 | ## Out-of-the-box ontologies 6 | 7 | Graphistry supports out-of-the-box ontologies of common systems: 8 | 9 | * SIEM Models: Much of Splunk CIM, ArcSight CEF, and ElasticSearch Common Schema 10 | * Vendor-specific models, such as FireEye HX/NX/iSIGHT, AWS CloudWatch, and WinLog 11 | * Classifying IPs as internal vs. external by RFC 1918 12 | 13 | See below for the list of built-in types they map to. 14 | 15 | ## Define custom ontologies 16 | 17 | 1. Edit `data/investigations/config/config.json` as per below 18 | 2. Restart docker service `pivot`: `docker compose restart pivot` 19 | 20 | Generally, you can limit the amount of work by mapping custom column names to built-in types, and thereby reuse their preconfigured settings. 21 | 22 | ## Ontology types 23 | 24 | ### Primary 25 | 26 | Key `ontology` defines: 27 | 28 | * For each type, such as `user`: 29 | * Default icon: string name supported by [Font Awesome 4](https://fontawesome.com/v4.7.0/icons/), such as [user-o](https://fontawesome.com/v4.7.0/icon/user-o) 30 | * Default color: string hex value, such as `#F00` for red 31 | * Default size: number, typically between 10 and 200 32 | * Displayed title: prioritized cascade based on entity type and available column names 33 | * New types: 34 | For the automatic table -> graph transform (aka hypergraph transform), the mapping from _table column names_ to _node entity types_. 35 | 36 | ### Secondary 37 | 38 | Additional settings exist such specific to individual layouts and connectors 39 | 40 | 41 | ## How to extend the ontology 42 | 43 | ### Easiest: Ask Graphistry to do it for you! 44 | 45 | Ideally, you can provide representative sample data that has the columns and values of interest, and if a data schema is available, that too. 46 | 47 | Ex: For Splunk users wanting support for a new product, provide the output of 48 | * `search index=some_product | fields * | dedup 20 event_type | head 1000` 49 | * select all columns in the Field Selector 50 | * download the CSV 51 | 52 | ### Add new types 53 | 54 | For example, to create a new node type `ip`, 55 | 56 | 1. Extend `data/investigations/config/config.json`: 57 | 58 | ```json 59 | { 60 | "ontology": { 61 | "icons": { 62 | "ip": "device", 63 | }, 64 | "colors": { 65 | "ip": "#F00", 66 | }, 67 | "sizes": { 68 | "ip": 100 69 | } 70 | } 71 | } 72 | ``` 73 | 74 | 2. Restart the pivot service: 75 | 76 | ```user@server.com:/var/graphistry $ docker compose stop pivot nginx && docker compose up -d``` 77 | 78 | ### Override default node/edge titles 79 | 80 | Graphistry picks the displayed title for each node and edge through the first match on the following cascade: 81 | 82 | * By type match: Does the element's `type` value have a corresponding `byType` binding? 83 | * By field match: Does the element contain a column name in `byField`? 84 | * By `pointTitle` , `edgeTitle`, if available 85 | * Use an element ID provided with the graph 86 | * Use an element ID generated by the system 87 | 88 | Ex: 89 | 90 | ```json 91 | { 92 | "ontology": { 93 | "titles": { 94 | "byType": { 95 | "geo": "address", 96 | "user": "name" 97 | }, 98 | "byField": ["src_ip", "dest_ip"] 99 | } 100 | } 101 | } 102 | ``` 103 | 104 | 105 | ### Configure new columns / new hypergraph transforms 106 | 107 | The existing ontology may already have all the types you want, but a new data source may have columns that need to be mapped into it. 108 | 109 | For example, to recognize `src_ip` and `dest_ip` columns as both generating `ip`-type nodes: 110 | 111 | 1. Extend `data/investigations/config/config.json`: 112 | 113 | ```json 114 | { 115 | "ontology": { 116 | "products": [ 117 | { 118 | "name": "my_extension_1", 119 | "colTypes": { 120 | "src_ip": "ip", 121 | "dest_ip": "ip" 122 | } 123 | } 124 | ] 125 | } 126 | } 127 | ``` 128 | 129 | 2. Restart the pivot service: 130 | ``` 131 | user@server.com:/var/graphistry $ docker compose stop pivot nginx && docker compose up -d 132 | ``` 133 | 134 | ## Built-in types 135 | The current set of built-in types is below. Upon system start, Graphistry emits the list of `Ontology types` for your installed version. You can also add your own (see above). 136 | 137 | We recommend using built-in types when possible. Each type comes with a built-in color, icon, size, and mappings from common data sources to it. This saves you work now, and as more connectors become supported and new features are added, you will automatically benefit from them in the future as well. 138 | 139 | ```json 140 | [ 141 | "actor", "agent", "alert", "amazon", "amex", "arn", "asn", "availabilityzone", 142 | "baidu", "bucket", 143 | "cidr", "city", "cloud", "cny", "code", "container", "continent", "cookie", "count", "country", 144 | "direction", "discover", "domain", "domainReputation", 145 | "email", "error", "eur", "event", "extension", 146 | "facebook", "file", "filePath", "filepath", "flag", "flickr", 147 | "gateway", "gbp", "geo", "github", "google", "googleplus", "group", 148 | "hash", "hashReputation", "host", "httpMethod", "httpmethod", 149 | "id", "ils", "image", "inr", "instagram", "instance", "ip", "ipReputation", 150 | "jcb", "jpy", 151 | "key", "krw", 152 | "language", "linkedin", "log", 153 | "mac", "machine_type", "machinetype", "mastercard", "medium", "message", "money", 154 | "name", "netbios", "networkinterface", "number", 155 | "organization", "os", 156 | "packer", "path", "payload", "paypal", "phone", "pinterest", "pod", "port", "process", "program", "protocol", 157 | "qq", "quora", 158 | "reddit", "role", "rub", 159 | "score", "size", "skype", "slack", "snapchat", "state", "stripe", "subnet", 160 | "tag", "telegram", "time", "timezone", "toolkit", "try", "tumblr", "twitch", "twitter", 161 | "uri", "urifragment", "uripath", "uriquery", "url", "urlReputation", "usd", "user", "useragent", 162 | "vendor", "version", "vine", "visa", "volume", "vpc", 163 | "wechat", "weibo", "whatsapp", 164 | "xbt", 165 | "yahoo", "youtube" 166 | ] 167 | ``` 168 | 169 | ## Layouts and IPs 170 | 171 | Layouts have additional options. The most common to modify is to flag values for being "inside" in the network map layout. 172 | 173 | You can put any regular expression here: 174 | 175 | ```json 176 | 177 | "layouts": { 178 | "network": { 179 | "ipInternalAcceptList": ["/10\.*/", "/127.0.0.1/"] 180 | } 181 | }, 182 | 183 | ``` 184 | 185 | 186 | ## Testing your ontology 187 | 188 | 1. Syntax errors: 189 | 190 | Graphistry tries to detect syntax error, and upon one, logs the error and stops. To see what is going on: 191 | 192 | `docker ps` <- see if `pivot` is unhealthy or in a restart loop 193 | `docker compose logs pivot` <- see the precise error message 194 | 195 | 2. Satisfactory configuration 196 | 197 | We recommend creating a Manual Data pivot. For example, to test various `ip` columns, use the following: 198 | 199 | * Query: `[ {"src_ip": "10.10.0.0", "dest_ip": "10.10.0.1", "ip": "10.10.0.2"} ]` 200 | * JQ: `.` 201 | * Nodes: `src_ip`, `dest_ip`, `ip` 202 | -------------------------------------------------------------------------------- /docs/app-config/configure-pygraphistry.md: -------------------------------------------------------------------------------- 1 | # Configure PyGraphistry 2 | 3 | PyGraphistry can be configured for tasks such as eliminating analyst boilerplate, customizing security handling, and supporting advanced server configurations. 4 | 5 | Configuration is for two basic APIs: the upload API for how PyGraphistry sends data to a Graphistry server, and the client API for how a URL is loaded into a user's browser. 6 | 7 | For server configuration, see [main configuration docs](configure.md). For REST API settings for uploading and viewing visualizations, see main developer documentation. 8 | 9 | ## The Cascade 10 | 11 | PyGraphistry configuration settings resolve through the following cascade, with the top items overriding the further ones: 12 | 13 | | Priority | Name | Primary Use | 14 | |:---:|:---|---| 15 | | 5 | `graph.settings(url_params={...})` | Analyst or developer fine-tuning an individual visualization's style via | 16 | | 4 | `graphistry.register(...)` | Analyst or developer | 17 | | 3 | Environment variables | Developer or sysadmin | 18 | | 2 | `graphistry.config` | Sysadmin | 19 | | 1 | Default | | 20 | 21 | Graphistry's built-in Jupyter server comes with a predefined `graphistry.config`. 22 | 23 | ## Settings 24 | 25 | ### General 26 | | Setting | Default | Type | Description 27 | |---|---|---|---| 28 | | `api` | 1 | `1` _JSON_
`2` _protobuf_
` 3` (recommended: JWT+Arrow) | Upload format and wire protocol | 29 | | `certificate_validation` | `True` | boolean | Unsafe: Disable to allow ignore TLS failures such as for known-faulty CAs | 30 | | `client_protocol_hostname` | `None` | FQDN, including protocol, for overriding `protocol` and `hostname` for what URL is used in browsers from displaying visualizations | 31 | | `hostname` | `"hub.graphistry.com"` | string | Domain (and optional path) for where to upload data and load visualizations 32 | | `protocol` | `"https"` | `"https"` or `"http"` | | 33 | 34 | ### 1.0 API (DEPRECATED) 35 | 36 | Deprecated 1.0 API option (api=1, api=2) 37 | 38 | | Setting | Default | Type | Description 39 | | `api_key` | `None` | string | *deprecated* | 40 | | `dataset_prefix` | `"PyGraphistry/"` | string | *deprecated* Prefix on upload location | 41 | 42 | 43 | ## Usage Modes 44 | 45 | ### graph.settings(url_params={...}) 46 | 47 | Override and add query parameters to the loaded visualization iframe by adding key/value strings to `url_params`. This does not control the protocol, domain, nor path, so is primarily for styling and debugging purposes. 48 | 49 | #### Example 50 | 51 | ```python 52 | my_graph.settings(url_params={ 53 | 'play': '0', 54 | 'my_correlation_id': 'session-123' 55 | }).plot() 56 | ``` 57 | 58 | 59 | ### graphistry.register() 60 | 61 | Note: Setup of the environment lets developers and analysts skip manually configuring `register()`, which may be preferrable 62 | 63 | Global module settings can be defined via `register()`: 64 | 65 | ``` 66 | register(api=3, username='...', password='...', server=None, protocol=None, api=None, certificate_validation=None, ...) 67 | ``` 68 | 69 | See PyGraphistry docs for individual connectors such as `.bolt(...)` and `.tigergraph(...)`. 70 | 71 | #### Example: Neo4j (bolt/cypher) 72 | 73 | ```python 74 | import graphistry 75 | graphistry.register(api=3, username='...', password='...', protocol='http', server='my.server.com') 76 | 77 | g = graphistry.bolt({'server': 'bolt://...', 'auth': ('my_user', 'my_pwd')}) 78 | 79 | g.cypher("MATCH (a)-[b]->(c) RETURN a,b,c LIMIT 10").plot() 80 | g.cypher("MATCH (a)-[b]->(c) RETURN a,b,c LIMIT 100000").plot() 81 | ... 82 | ``` 83 | 84 | ### Environment variables 85 | 86 | #### General 87 | 88 | | Graphistry Setting | Environment Variable | Description | 89 | |:---|:---|:---| 90 | | `api_version` | `GRAPHISTRY_API_VERSION` || 91 | | `certificate_validation` | `GRAPHISTRY_CERTIFICATE_VALIDATION` || 92 | | `client_protocol_hostname` | `GRAPHISTRY_CLIENT_PROTOCOL_HOSTNAME` || 93 | | `hostname` | `GRAPHISTRY_HOSTNAME` || 94 | | `protocol` | `GRAPHISTRY_PROTOCOL` || 95 | | | `PYGRAPHISTRY_CONFIG` | Absolute path of `graphistry.config` 96 | 97 | #### 1.0 API (DEPRECATED) 98 | 99 | | Graphistry Setting | Environment Variable | Description | 100 | |:---|:---|:---| 101 | | `api_key` | `GRAPHISTRY_API_KEY` || 102 | | `dataset_prefix` | `GRAPHISTRY_DATASET_PREFIX` || 103 | 104 | There are multiple common ways to set environment variables: 105 | 106 | * OS user login, e.g., `.bashrc` file 107 | * In an invocation script, `MY_FLD=MY_VAL python myscript.py` 108 | * The `environment:` section of Graphistry's `~/docker-compose.yml` for `notebook` or the `~/.env` file. 109 | WARNING: Editing `.env` is preferred over editing `.yml` in order to simplify upgrading 110 | * In Python via `os.environ['MY_FLD'] = 'MY_VAL'` 111 | 112 | 113 | ### graphistry.config 114 | 115 | Specify a `json` file using key/values from the Settings table. 116 | 117 | PyGraphistry automatically checks for `graphistry.config` as follows: 118 | 119 | ```python 120 | config_paths = [ 121 | os.path.join('/etc/graphistry', '.pygraphistry'), 122 | os.path.join(os.path.expanduser('~'), '.pygraphistry'), 123 | os.environ.get('PYGRAPHISTRY_CONFIG', '') 124 | ] 125 | ``` 126 | 127 | ## Graphistry Enterprise: Install packages into built-in Jupyter notebook 128 | 129 | If you are using Graphistry's built-in Jupyter server, it autoconfigures `PYGRAPHISTRY_CONFIG`, `graphistry.config`, and `PYTHONPATH`. 130 | 131 | The `PYTHONPATH` is automatically set to correspond to your host's `data/py_envs/*` folders, so custom package installs will persist across container restarts and rebuilds. 132 | 133 | ### Install new packages 134 | 135 | You can likely just `pip install you_package` and will work 136 | 137 | Safety tip: Use `pip install --no-deps your_package` . This avoids risks of breaking existing GPU packages with unintended dependency upgrades. 138 | 139 | You typically need to restart your Jupyter notebook's Python kernel after installing new packages. 140 | 141 | ### List custom package installs 142 | 143 | Check on your host environment, `ls ./data/py_envs/*` 144 | 145 | You may also be able to check via your Jupyter notebook environment. See `env` to find where the custom packages are mounted, and check that folder. 146 | 147 | ### Uninstall packages 148 | 149 | Perform the usual `pip uninstall your_package` command. 150 | 151 | If there are lingering file issues, check your `data/py_envs` folder for any unintended files and folders. 152 | 153 | You may need to restart your Jupyter notebook's Python kernel after uninstalling packages to have the intended effect. 154 | 155 | ## Bundled installs 156 | 157 | Graphistry Enterprise servers come with dependencies built-in, so you can skip this section 158 | 159 | For custom environments, you may want to add some that PyGraphistry prebundles. Run `pip install graphistry[bundle_name]`, with the following bundle names as common ones: 160 | 161 | * None: (`pip install graphistry`) - no extras 162 | * `umap_learn`: For CPU UMAP support 163 | * `ai`: For AI/ML support, including 1GB+ PyTorch install 164 | * RAPIDS.AI: You can also get far by installing the RAPIDS.ai ecosystem, especially cudf, cuml, and cugraph 165 | * For more options, see the `setup.py` file in the PyGraphistry Github repository 166 | 167 | 168 | ## Examples 169 | 170 | ### Speed up some uploads 171 | 172 | ```python 173 | import graphistry 174 | graphistry.register(api=3, username='..', password='...') 175 | ``` 176 | 177 | ### Preset a 1.0 API key for all system users 178 | 179 | Create Python-readable `/etc/graphistry/.pygraphistry`: 180 | 181 | ```json 182 | { 183 | "api_key": "SHARED_USERS_KEY", 184 | "protocol": "https", 185 | "hostname":"my.server.com" 186 | } 187 | ``` 188 | 189 | For Jupyter notebooks, you may want to create per-user login `.pygraphistry` files. Please contact staff for further options and requests. 190 | 191 | ### Different URLs for internal upload vs external viewing 192 | 193 | In scenarios like Graphistry running on a notebook server, you may prefer to send uploads to a local host (`http://localhost`, `http://nginx`, ...), and tell browser client viewers to use a separate, public host and subpath (`http://graphistry.site.ngo/graphistry`): 194 | 195 | ```python 196 | import os 197 | import graphistry 198 | 199 | ### Internal URL: PyGraphistry app -> Graphistry Server 200 | GRAPHISTRY_HOSTNAME_PROTOCOL='https' 201 | os.environ['GRAPHISTRY_HOSTNAME'] = "localhost" 202 | 203 | ### External URL: Webpage iframe URL -> Graphistry Server 204 | os.environ['GRAPHISTRY_CLIENT_PROTOCOL_HOSTNAME'] = "https://graph.site.ngo/graphistry" 205 | 206 | graphistry.register( 207 | key='MY_API_KEY', 208 | protocol=GRAPHISTRY_HOSTNAME_PROTOCOL 209 | ) 210 | ``` 211 | 212 | 213 | -------------------------------------------------------------------------------- /docs/app-config/configure-python.md: -------------------------------------------------------------------------------- 1 | # Configure the Python Endpoint 2 | 3 | The Python endpoint allows any user granted access a way to retrieve datasets stored within Graphistry and process it using unrestricted arbitrary user provided Python code. This Python code can include a limited set of external libraries such as `numpy` and `cudf`, in addition to `graphistry`, and can access all computational resources available to the forge-etl-python server including GPU compute. The result is returned to the user as a string or as JSON. 4 | 5 | ## Safe defaults 6 | 7 | * Graphistry Hub: The Python endpoint is not available to Graphistry Hub users at this time 8 | 9 | * Graphistry Enterprise: The Python endpoint must be explicitly turned on for regular Graphistry Enterprise users 10 | 11 | The more restricted GFQL endpoint is default-on for both Graphistry Hub and Graphistry Enterprise 12 | 13 | ## Toggling 14 | 15 | The endpoint must be both on in general, and individual user types explicitly allowed: 16 | 17 | 1. Enable access to individual users via the Graphistry admin panel's feature flag area 18 | 19 | 1. The flag must also be enabled at the system-level via the `ENABLE_PYTHON_ENDPOINT` environment variable in `data/config/custom.env` 20 | 21 | We recommend checking individual user access before enabling the endpoint. 22 | 23 | ## Further reading 24 | 25 | See also: 26 | 27 | * The [Graphistry REST API for the Python endpoint](https://hub.graphistry.com/docs/Python/python-api/) 28 | * The [Graphistry REST API for the GFQL endpoint](https://hub.graphistry.com/docs/GFQL/gfql-api/) 29 | -------------------------------------------------------------------------------- /docs/app-config/connector-management.md: -------------------------------------------------------------------------------- 1 | # Managed Connector CLI 2 | 3 | Graphistry-managed connectors simplify more secure secret management and automation patterns than storing credentionals as environment variables. This document describes REST-based remote management using a bash script. It walks through the needed secret management and configuration. The examples are for Databricks. 4 | 5 | You can manage Louie connectors stored in Graphistry using the local console CLI from a Graphistry server or remotely via the REST API. The REST API is easier to start with. For convenience, we include bash scripts for using the remote REST API. Functionalities include creating, updating, deleting, getting, and listing connectors. 6 | 7 | ## Prerequisites 8 | 9 | ### Remote REST API bash scripts: Python cryptography library 10 | 11 | The REST curl bash scripts invoke the Python cryptography library. 12 | 13 | The script uses the Python cryptography library to decrypt keyjson. Install the library using one of the following commands: 14 | ```bash 15 | pip install cryptography 16 | ``` 17 | or 18 | ```bash 19 | pip3 install cryptography 20 | ``` 21 | 22 | ### Environment variables 23 | 24 | #### Needed keys: Remote API client configuration 25 | 26 | To use the relevant management APIs, ensure you have the following environment variables: 27 | 28 | ```bash 29 | # For administration scripts 30 | export GRAPHISTRY_USERNAME=username 31 | export GRAPHISTRY_PASSWORD=pass 32 | export GRAPHISTRY_BASE_PATH=http://localhost 33 | 34 | # For signing 35 | export GRAPHISTRY_ENCRYPTION_KEYS=encryption_key 36 | export GRAPHISTRY_NEXUS_SIGNING_KEY=signing_key 37 | export GRAPHISTRY_NEXUS_SIGNING_SALT=salt_key 38 | ``` 39 | 40 | Additionally, use or get a copy of `$GRAPHISTRY_HOME/etc/scripts/connector_management.sh` 41 | 42 | #### Needed keys: Graphistry server configuration 43 | 44 | Your Graphistry server uses similarly named keys with the same values: 45 | 46 | ```bash 47 | # For administration scripts 48 | export GRAPHISTRY_USERNAME=username 49 | export GRAPHISTRY_PASSWORD=pass 50 | export GRAPHISTRY_BASE_PATH=http://localhost 51 | 52 | # For signing 53 | export GRAPHISTRY_NEXUS_ENCRYPTION_KEYS=encryption_key 54 | export GRAPHISTRY_NEXUS_SIGNING_KEY=signing_key 55 | export GRAPHISTRY_NEXUS_SIGNING_SALT=salt_key 56 | ``` 57 | 58 | Note the difference of `GRAPHISTRY_NEXUS_ENCRYPTION_KEYS` vs `GRAPHISTRY_ENCRYPTION_KEYS`. Future upgrades will normalize on dropping the term `_NEXUS_`. 59 | 60 | #### Find or generate signing keys 61 | 62 | For self-hosted Graphistry server users, you need to generate encryption and signing keys. If you are using Graphistry Hub, please contact Graphistry staff to obtain your keys. 63 | 64 | The keys are: `GRAPHISTRY_NEXUS_ENCRYPTION_KEYS`, `GRAPHISTRY_NEXUS_SIGNING_KEY`, and `GRAPHISTRY_NEXUS_SIGNING_SALT`. 65 | 66 | Check if you already have them set in your `./data/config/custom.env`, and read on to replace them if desired: 67 | 68 | ```bash 69 | cat ./data/config/custom.env | grep -E "GRAPHISTRY_NEXUS_ENCRYPTION_KEYS|GRAPHISTRY_NEXUS_SIGNING_KEY|GRAPHISTRY_NEXUS_SIGNING_SALT" 70 | ``` 71 | 72 | This should list your keys. To rotate them, delete them from the file and continue. 73 | 74 | To generate and append the necessary environmental variables to your `./data/custom.env`, run: 75 | 76 | ```bash 77 | { echo ""; bash -c "$(sed '1 a\set +x' ./etc/scripts/cred-gen.sh)" | grep -E "GRAPHISTRY_NEXUS_ENCRYPTION_KEYS|GRAPHISTRY_NEXUS_SIGNING_KEY|GRAPHISTRY_NEXUS_SIGNING_SALT"; } 78 | >> ./data/custom.env 79 | ``` 80 | 81 | Running the immediately prior checking script should find and list your keys. 82 | 83 | Once confirmed, restart the Nexus service to switch to the new signing keys: 84 | 85 | ```bash 86 | ./release up -d --force-recreate --no-deps nexus 87 | ``` 88 | 89 | ## How to Use: Remote bash client over the REST API 90 | 91 | The bash script calls the underlying remote REST APIs on your behalf, including Python callouts for encrypt. Use it with various combinations of environment variables and actions to perform specific operations on connectors. 92 | 93 | ### Location 94 | 95 | You may find a copy of the standalone bash REST API client at `$GRAPHISTRY_HOME/etc/scripts/connector_management.sh` 96 | 97 | The following examples assume you are calling it from `$GRAPHISTRY_HOME`. You can call it from anywhere, including on other computers with network access to Graphistry. 98 | 99 | ### Settings as environment variables 100 | 101 | Control settings and configuration by defining the following environment variables during the script invocation. 102 | 103 | Client configuration environment variables detailed above: 104 | 105 | 106 | - `GRAPHISTRY_BASE_PATH=http://localhost` 107 | - `GRAPHISTRY_USERNAME` 108 | - `GRAPHISTRY_NEXUS_SIGNING_SALT` 109 | - `GRAPHISTRY_PASSWORD` 110 | - `GRAPHISTRY_ENCRYPTION_KEYS` (note no `NEXUS`) 111 | - `GRAPHISTRY_NEXUS_SIGNING_KEY` 112 | 113 | Command environment variables: 114 | 115 | 116 | - `ACTION`: Action to perform (`create`, `update`, `delete`, `get`, `list`, `upsert_pat`, `delete_pat`, `delete_all_pats`, `update_all_pats`). The default action is list. 117 | - `KEYJSON`: JSON data containing connector configuration value. The default is an empty JSON object. 118 | - `CONNECTOR_TYPE`: Type of the connector. The default value is Databricks. 119 | - `CONNECTOR_ID`: The ID of the connector. This can be left empty for certain actions. 120 | - `CONNECTOR_NAME`: The name of the connector. 121 | - `PAT_KEY`: The key(username) for the Personal Access Token (PAT) to be managed. 122 | - `PAT_VALUE`: The value for the PAT to be managed. 123 | - `PATS_JSON`: JSON string containing multiple PATs for bulk updates. 124 | - `PATS_CSV`: Path to a CSV file containing multiple PATs for bulk updates. 125 | 126 | 127 | ### Additional Help 128 | 129 | For further assistance and detailed usage instructions, refer to the following command: 130 | 131 | ```bash 132 | ACTION=help ./etc/scripts/connector_management.sh 133 | ``` 134 | 135 | ### Top commands 136 | 137 | #### List connectors 138 | 139 | To list all existing connectors, use the following example command: 140 | 141 | ```bash 142 | ACTION=list ./etc/scripts/connector_management.sh 143 | ``` 144 | 145 | #### Get connector 146 | 147 | To retrieve details of a specific connector, use the following example command: 148 | 149 | ```bash 150 | ACTION=get CONNECTOR_ID="connector_id" ./etc/scripts/connector_management.sh 151 | ``` 152 | 153 | #### Create connector 154 | 155 | To create a new connector, use the following example command for Databricks 156 | 157 | ```bash 158 | ACTION=create ORG_NAME=my_org_123 CONNECTOR_NAME="MyDatabricksConnector" CONNECTOR_DETAIL='{"host": "abc123.cloud.databricks.com", "workspace_id": "xyz456"}' KEYJSON='{"pats": {"user1": "aa11", "user2": "bb22"}}' ./etc/scripts/connector_management.sh 159 | ``` 160 | 161 | #### Update connector 162 | 163 | To update an existing connector, use the following example command: 164 | 165 | ```bash 166 | ACTION=update CONNECTOR_ID="connector_id" CONNECTOR_NAME="UpdatedConnectorName" CONNECTOR_DETAIL='{"host": "abc123-updated.cloud.databricks.com", "workspace_id": "xyz456-updated"}' KEYJSON='{"user1": "updated_pat1", "user2": "updated_pat2"}' ./etc/scripts/connector_management.sh 167 | ``` 168 | 169 | #### Delete connector 170 | 171 | To delete a connector, use the following example command: 172 | 173 | ```bash 174 | ACTION=delete CONNECTOR_ID="connector_id" ./etc/scripts/connector_management.sh 175 | ``` 176 | 177 | ### Databricks 178 | 179 | Databricks connector management is largely via the above generic commands 180 | 181 | Additional per-user PAT management is supported through the following Databricks-specific connector commands. When SSO is not an option, this allows storing per-user PAT entries that get automatically applied upon regular Graphistry authentication. 182 | 183 | #### Manage one or more Personal Access Tokens (PATs) 184 | 185 | To upsert, delete, delete all, or update all PATs associated with a connector, use the following example commands: 186 | 187 | ```bash 188 | # Upsert a PAT 189 | ACTION=upsert_pat CONNECTOR_ID="sample_connector_id" PAT_KEY="sample_key" PAT_VALUE="sample_value" ./etc/scripts/connector_management.sh 190 | 191 | # Delete a PAT 192 | ACTION=delete_pat CONNECTOR_ID="sample_connector_id" PAT_KEY="sample_key" ./etc/scripts/connector_management.sh 193 | 194 | # Delete all PATs 195 | ACTION=delete_all_pats CONNECTOR_ID="sample_connector_id" ./etc/scripts/connector_management.sh 196 | 197 | # Update all PATs 198 | ACTION=update_all_pats CONNECTOR_ID="sample_connector_id" PATS_JSON='{"user10":"pat_value"}' ./etc/scripts/connector_management.sh 199 | 200 | # Update all PATs using csv file 201 | ACTION=update_all_pats CONNECTOR_ID="sample_connector_id" PATS_CSV='path/to/pats.csv' ./etc/scripts/connector_management.sh 202 | ``` 203 | 204 | #### CSV File Example 205 | 206 | ``` 207 | username,PAT 208 | user123,abc123pat 209 | user456,def456pat 210 | user789,ghi789pat 211 | test1,jkl012pat 212 | ``` 213 | 214 | #### Keyjson with PATs Example 215 | 216 | ```json 217 | { 218 | "host": "updated_host_url", 219 | "pats": { 220 | "user1": "updated_pat1", 221 | "user2": "updated_pat2" 222 | }, 223 | "token": "updated_service_token", 224 | "workspace_id": "updated_workspace_id" 225 | } 226 | ``` 227 | -------------------------------------------------------------------------------- /docs/app-config/email.md: -------------------------------------------------------------------------------- 1 | # Graphistry email 2 | 3 | Graphistry optionally uses email for easy and secure account creation, invitations, and notifications. Many workflows switch to a more secure opt-in mode when email is enabled. 4 | 5 | ## Enable email 6 | 7 | In `data/config/custom.env`, set: 8 | 9 | ```bash 10 | DISABLE_EMAIL=false 11 | ``` 12 | ## Configure email 13 | 14 | Several options are possible, contact staff for host (smtp), mailgun, and debug 15 | -------------------------------------------------------------------------------- /docs/app-config/index.rst: -------------------------------------------------------------------------------- 1 | 2 | App configuration 3 | ================= 4 | 5 | 6 | Visual Playbooks 7 | ---------------- 8 | 9 | .. toctree:: 10 | :maxdepth: 1 11 | :titlesonly: 12 | 13 | configure 14 | configure-custom-pivots 15 | configure-investigation 16 | configure-ontology 17 | connector-management 18 | templates 19 | 20 | Streamlit 21 | --------- 22 | 23 | .. toctree:: 24 | :maxdepth: 1 25 | :titlesonly: 26 | 27 | configure-dashboards 28 | 29 | System 30 | ------ 31 | 32 | .. toctree:: 33 | :maxdepth: 1 34 | :titlesonly: 35 | 36 | configure-python 37 | configure-pygraphistry 38 | email 39 | -------------------------------------------------------------------------------- /docs/app-config/templates.md: -------------------------------------------------------------------------------- 1 | # Investigation Templates 2 | 3 | Investigation templates bring a lightweight form of automation to investigations. They work just like regular investigations, except they add a few key features that, combined with existing investigation features, unlock useful workflows. 4 | 5 | For even friendlier templates that analysts are more comfortable tweaking, consider replacing individual pivots [custom pivots](configure-custom-pivots.md). 6 | 7 | **Contents** 8 | 1. Sample workflows 9 | 1. Create a template 10 | 1. Manual: Instantiate a template 11 | 1. URL API: Linking a template 12 | 1. Splunk integration 13 | 1. Best practices 14 | * Manual data for first step 15 | * Multiple entry points 16 | * Set time range and provide instructions 17 | * Naming 18 | * Cross-linking 19 | 20 | 21 | ## 1. Sample workflows 22 | 23 | * **In-tool**: Create a base template such as for looking at an account, and instantiate whenever you are investigating a new account 24 | * **From an alert email or dashboard**: Include a link to a 360 view for that alert or involved entities, and center it on the time range of the incident 25 | * **Splunk UI**: Teach Splunk to include 360 views whenever it mentions an account, IP, or alert 26 | 27 | 28 | ## 2. Create a template 29 | 30 | Any investigation can be reused as a template. From an investigation (or `save-a-copy` of one), in the investigation details, check `Template`. When you save and return to the content home, it should have moved into the top `Templates` section. 31 | 32 | ## 3. Manual: Instantiate a template 33 | 34 | From the content home, navigate to your template, and press the `new` button. This will create a new investigation that is based off of the most recent version of the template, similar to how `clone` works on an investigation. Editing a template keeps past investigations safe and untouched. 35 | 36 | ## 4. URL API: Linking a template 37 | 38 | The magic happens when the URI API is used to enable users of web applications to jump into prebuilt investigations with just one click. 39 | 40 | Consider the following URL for triggering a phone history check: 41 | 42 | `/pivot/template?investigation=453d190914cf9fa0&pivot[0][events][0][phone]=1.800.555.5555&time=1504401120.000&before=-1d&after=+1d&name=Phone-History-555-5555` 43 | 44 | This URL: Instantiates template `453d190914cf9fa0`, names it `Phone-History-555-5555`, overrides the global time range to center at `1504401120` (epoch time) and runs searches +/- 1 day from then. The first pivot will be populated with one record, and that record will have field `phone` mapped to the string `"1.800.555.5555"`. 45 | 46 | 47 | | FIELD | OPTIONAL | DEFAULT | FORMAT | NOTES | 48 | |--------------------:|------- |------- |-------- |------- | 49 | | **investigation** | required | | ID | Get template ID from its URL. Ex: 453d190914cf9fa0 | 50 | | **name** | optional | "Copy of [template name]" | String | Recommend using a short standard pattern to group together ("[Phone History] ...") | 51 | | **time** | optional | now | Number or string | Epoch time (number) or best-effort if not a number. Ex: 1504401120 | 52 | | **before** | optional | -7d | [+/-][number][ms/s/min/h/d/w/mon/y] | Ex: -1d | 53 | | **after** | optional | +0d | [+/-][number][ms/s/min/h/d/w/mon/y] | Ex: +3min | 54 | | **pivot** | optional | | see below | see below | 55 | 56 | URL parameter `pivot` follows one of the two following formats: 57 | * `[step][field]`, e.g., `pivot[0][index]=index%3Dalerts`, the URI-encoded form of string `"index=alerts"` 58 | * `[step][field][list_index][record_field]`, e.g., `pivot[0][events][0][phone]=1-800-555-5555` sets the first step's events to JSON list `[ {"phone": "1-800-555-5555"} ]` 59 | 60 | You can therefore set or override most investigation step values, not just the first one. Likewise, if you want to trigger an investigation over multiple values, you can provide a list of them. 61 | 62 | ## 5. Splunk integration 63 | 64 | Splunk users can easily jump into Graphistry investigations without much thinking from any Splunk search result or dashboard, even if they don't know which ones are available ahead of time. To do so, you simply register Graphistry templates as Splunk workflow actions. 65 | 66 | To make a template appear as a Workflow Action on a specific kind of event: 67 | 68 | 1. Settings -> Event Types -> new: 69 | 70 | * Search string: The events you want the template to appear on (if you don't hav event types already known). Ex: "index=calls phone=*". 71 | * Tag(s): An identifier to associate with these events 72 | 73 | 2. Settings -> Fields -> Workflow actions -> new 74 | 75 | * Label: What appears in Splunk's action menu. Ex: `Check Graphistry for Phone 360: $phone$` 76 | * Apply only to fields, tags: the search result column and/or tag from Step 1 77 | * Show action in: Both 78 | * Action type: Link 79 | * Link configuration: Template URL, using `$fld$` to populate values. Ex: `https://my_graphistry.com/pivot/template?investigation=453d190914cf9fa0&pivot[0][events][0][phone]=$phone$&pivot[0][events][0][time]=$time_epoch$` 80 | * Open link in: New window 81 | * Link method: get 82 | 83 | ## 6. Best practices 84 | 85 | ### Manual data for first step 86 | 87 | By making the first step an `Enter data` one, most of the parameters can be set on it. The URL generates an initial graph, and subsequent steps expand on them. 88 | 89 | ### Multiple entry points 90 | 91 | You can likely combine multiple templates into one. For example, in IT scenarios, 360 views for IP's, MAC addresses, and host names likely look the same. Make the first step create a graph for one or more of these, the next ones derive one value type from the other (or a canonical ID), and the remaining steps look the same. 92 | 93 | ### Set time range and provide instructions 94 | 95 | Analysts unfamiliar with your template would strongly benefit from instructions telling them what to modify (if anything) and how to use the investigation. Many options likely have sane defaults on a per-template basis, such as the time range, so we recommend including them in your URLs. 96 | 97 | ### Naming 98 | 99 | Content management can become an issue. Use a custom short description name, such as `name=%5BPhone%20360%5D%20555-5555` (=> `[Phone 360] 555-5555`. The generated investigations can now be easily searched and sorted. 100 | 101 | ### Cross-linking 102 | 103 | You can include templates as links within templates! For example, whenever a phone number node is generated, you can include attribute `link` with value `/pivot/template?investigation=...` . 104 | -------------------------------------------------------------------------------- /docs/commands.md: -------------------------------------------------------------------------------- 1 | # Top commands 2 | 3 | Graphistry supports advanced command-line administration via standard `docker compose`, `.yml` / `.env` files, and `caddy` reverse-proxy configuration. 4 | 5 | ## Login to server 6 | 7 | | Image | Command | 8 | |--: |:-- | 9 | | **AWS** | `ssh -i [your_key].pem ubuntu@[your_public_ip]` | 10 | | **Azure** | `ssh -i [your_key].pem [your_user]@[your_public_ip]`
`ssh [your_user]@[your_public_ip]` (pwd-based) | 11 | | **Google** | `gcloud compute [your_instance] ssh` | 12 | | **On-prem / BYOL** | Contact your admin | 13 | 14 | ## CLI commands 15 | 16 | All likely require `sudo`. Run from where your `docker-compose.yml` file is located: `/home/ubuntu/graphistry` (AWS), `/var/graphistry` (Azure), or `/var/graphistry//` (recommended on-prem). 17 | 18 | | TASK | COMMAND | NOTES | 19 | |--: |:--- |:--- | 20 | | **Install** | `docker load -i containers.tar.gz` | Install the `containers.tar.gz` Graphistry release from the current folder. You may need to first run `tar -xvvf my-graphistry-release.tar.gz`. | 21 | | **Start
interactive** | `docker compose up` | Starts Graphistry, close with ctrl-c | 22 | | **Start
daemon** | `docker compose up -d` | Starts Graphistry as background process | 23 | | **Start
namespaced (concurrent)** | `docker compose -p my_unique_namespace up` | Starts Graphistry in a specific namespace. Enables running multiple independent instances of Graphistry. NOTE: Must modify Caddy service in `docker-compose.yml` to use non-conflicting public ports, and likewise change global volumes to be independent. | 24 | | **Stop** | `docker compose stop` | Stops Graphistry | 25 | | **Restart (soft)** | `docker restart ` | Soft restart. May also need to restart service `nginx`. | 26 | | **Restart (hard)** | `docker up -d --force-recreate --no-deps ` | Restart with fresh state. May also need to restart service `nginx`. | 27 | | **Reset** | `docker compose down -v && docker compose up -d` | Stop Graphistry, remove all internal state (including the user account database!), and start fresh . | 28 | | **Status** | `docker compose ps`, `docker ps`, and `docker status` | Status: Uptime, healthchecks, ... | 29 | | **GPU Status** | `nvidia-smi` | See GPU processes, compute/memory consumption, and driver. Ex: `watch -n 1.5 nvidia-smi`. Also, `docker run --rm -it nvidia/cuda:11.5.0-base-ubuntu20.04 nvidia-smi` for in-container test. | 30 | | **1.0 API Key** | docker compose exec streamgl-vgraph-etl curl "http://0.0.0.0:8080/api/internal/provision?text=MYUSERNAME" | Generates API key for a developer or notebook user (1.0 API is deprecated)| 31 | | **Logs** | `docker compose logs ` | Ex: Watch all logs, starting with the 20 most recent lines: `docker compose logs -f -t --tail=20 forge-etl-python` . You likely need to switch Docker to use the local json logging driver by deleting the two default managed Splunk log driver options in `/etc/docker/daemon.json` and then restarting the `docker` daemon (see below). | 32 | | **Create Users** | Use Admin Panel (see [Create Users](tools/user-creation.md)) or `etc/scripts/rest` | 33 | | **Restart Docker Daemon** | `sudo service docker restart` | Use when changing `/etc/docker/daemon.json`, ... | 34 | | **Jupyter shell**| `docker exec -it -u root graphistry_notebook_1 bash` then `source activate rapids` | Use for admin tasks like global package installs | -------------------------------------------------------------------------------- /docs/config-files/config.json: -------------------------------------------------------------------------------- 1 | { 2 | "log": { 3 | "level": "debug" 4 | }, 5 | "graphistry": { 6 | 7 | "host": "http://MY.ADDRESS.COM" 8 | }, 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | "authentication": { 18 | "passwordHash": "$2a$10$s/bwo97icnlAvenqnByau.rwoA2mmntSzzEVdjfzhBXYRZxzhVVN.", 19 | "username": "admin" 20 | }, 21 | 22 | 23 | "pivots": { 24 | "show": [ 25 | 26 | 27 | "http-search", 28 | "http-expand", 29 | "manual-data" 30 | ] 31 | } 32 | } 33 | -------------------------------------------------------------------------------- /docs/config-files/daemon.json: -------------------------------------------------------------------------------- 1 | { 2 | "default-runtime": "nvidia", 3 | "runtimes": { 4 | "nvidia": { 5 | "path": "nvidia-container-runtime", 6 | "runtimeArgs": [] 7 | } 8 | } 9 | } 10 | -------------------------------------------------------------------------------- /docs/debugging/debug-container-networking.md: -------------------------------------------------------------------------------- 1 | 2 | # Debugging Container Networking 3 | 4 | The following tests may help pinpoint loading failures. 5 | 6 | ## Prerequisites 7 | 8 | Check the main tests (https://github.com/graphistry/graphistry-cli) 9 | 10 | * All containers are running 11 | * Healthchecks passes 12 | 13 | ## Mongo container 14 | 15 | ### A. Host is running Mongo 16 | 17 | *Note*: Database, collection initializated by `launch` (e.g., during `init`) and does not persist between runs. 18 | 19 | ``` 20 | docker exec monolith-network-mongo /bin/bash -c "echo 'db.stats().ok' | mongo localhost/cluster -quiet" 21 | ``` 22 | => 23 | ``` 24 | 1 25 | ``` 26 | 27 | ### B. Mongo has registered workers 28 | 29 | *Note*: Populated by `monolith-network-viz` on node process start 30 | 31 | ``` 32 | docker exec monolith-network-mongo /bin/bash -c "echo 'db.node_monitor.find()' | mongo localhost/cluster -quiet" 33 | ``` 34 | => 10+ workers 35 | ``` 36 | { "_id" : ObjectId("5b4d7049f160a28b5001a6bf"), "ip" : "localhost", "pid" : 9867, "port" : 10011, "active" : true, "updated" : ISODate("2018-07-30T16:18:31.835Z") } 37 | { "_id" : ObjectId("5b4d727ff160a28b5001a6c3"), "ip" : "localhost", "pid" : 13325, "port" : 10002, "active" : true, "updated" : ISODate("2018-07-30T16:18:30.697Z") } 38 | { "_id" : ObjectId("5b4d729af160a28b5001a6c4"), "ip" : "localhost", "pid" : 13392, "port" : 10010, "active" : true, "updated" : ISODate("2018-07-30T16:18:32.108Z") } 39 | { "_id" : ObjectId("5b4d72e0f160a28b5001a6ca"), "ip" : "localhost", "pid" : 13966, "port" : 10001, "active" : true, "updated" : ISODate("2018-07-30T16:18:31.205Z") } 40 | { "_id" : ObjectId("5b4d7306f160a28b5001a6cc"), "ip" : "localhost", "pid" : 14156, "port" : 10009, "active" : true, "updated" : ISODate("2018-07-30T16:18:31.717Z") } 41 | { "_id" : ObjectId("5b4d75fff160a28b5001a6d0"), "ip" : "localhost", "pid" : 17872, "port" : 10000, "active" : true, "updated" : ISODate("2018-07-30T16:18:30.618Z") } 42 | ... 43 | ``` 44 | 45 | 46 | ## Browser 47 | 48 | ### A. Can access site: 49 | 50 | Browse to 51 | ``` 52 | curl http://MY_GRAPHISTRY_SERVER.com/central/healthcheck 53 | ``` 54 | => 55 | ``` 56 | {"success":true,"lookup_id":"","uptime_ms":,"interval_ms":} 57 | ``` 58 | 59 | ### B. Browser has web sockets enabled 60 | 61 | Passes test at https://www.websocket.org/echo.html 62 | 63 | ### C. Can follow central redirect: 64 | 65 | 66 | Open browser developer network analysis panel and visit 67 | 68 | ``` 69 | http://MY_GRAPHISTRY_SERVER.com/graph/graph.html?dataset=Twitter 70 | ``` 71 | => 72 | ``` 73 | 302 on `/graph/graph.html?dataset=Twitter 74 | 200 on `/graph.graph.html?dataset=Twitter&workbook=` 75 | Page UI loads (`vendor..css`, ...) 76 | Socket connects (`/worker//socket.io/?dataset=Twitter&...`) 77 | Dataset positions stream in (`/worker//vbo?id=&buffer=curPoints`) 78 | ``` 79 | 80 | This call sequence stress a lot of the pipeline. 81 | 82 | ## NGINX 83 | 84 | *Note* Assumes underlying containers are fulfilling these requests (see other tests) 85 | 86 | ### A. Can server central routes 87 | 88 | ``` 89 | curl -s -I localhost/central/healthcheck | grep HTTP 90 | ``` 91 | => 92 | ``` 93 | HTTP/1.1 200 OK 94 | ``` 95 | 96 | 97 | ### B. Can receive central redirect: 98 | 99 | ``` 100 | curl -s -I localhost/graph/graph.html?dataset=Twitter | grep "HTTP\|Location" 101 | ``` 102 | => 103 | ``` 104 | HTTP/1.1 302 Found 105 | Location: /graph/graph.html?dataset=Twitter&workbook= 106 | ``` 107 | 108 | and 109 | 110 | ``` 111 | curl -s -I localhost/graph/graph.html?dataset=Twitter | grep "HTTP\|Location" 112 | ``` 113 | => 114 | ``` 115 | HTTP/1.1 302 Found 116 | Location: /graph/graph.html?dataset=Twitter&workbook= 117 | ``` 118 | 119 | ### C. Can serve worker routes 120 | 121 | ``` 122 | curl -s -I localhost/worker/10000/healthcheck | grep HTTP 123 | ``` 124 | => 125 | ``` 126 | HTTP/1.1 200 OK 127 | ``` 128 | 129 | 130 | ## Viz container 131 | 132 | ### A. Container has a running central server 133 | 134 | 135 | ``` 136 | docker exec monolith-network-viz curl -s -I localhost:3000/central/healthcheck | grep HTTP 137 | ``` 138 | => 139 | ``` 140 | HTTP/1.1 200 OK 141 | ``` 142 | 143 | and 144 | 145 | ``` 146 | docker exec monolith-network-viz curl -s -I localhost:3000/graph/graph.html?dataset=Twitter | grep "HTTP\|Location" 147 | ``` 148 | => 149 | ``` 150 | HTTP/1.1 302 Found 151 | Location: /graph/graph.html?dataset=Twitter&workbook= 152 | ``` 153 | 154 | 155 | ### C. Can communicate with Mongo 156 | 157 | First find mongo configuration for MONGO_USERNAME and MONGO_PASSWORD: 158 | `docker exec monolith-network-viz cat central-cloud-options.json` or `docker exec monolith-network-viz ps -eafww | grep central` 159 | 160 | Plug those into `` and `` below: 161 | 162 | ``` 163 | docker exec -w /var/graphistry/packages/central monolith-network-viz node -e "x = require('mongodb').MongoClient.connect('mongodb://@mongo/cluster').then(function (db) { return db.collection('node_monitor').find() }).then(function (cursor) { return cursor.toArray() }).then(console.log.bind(null, 'ok'), console.error.bind(null, 'error'))" 164 | ``` 165 | 166 | => 167 | ``` 168 | ok [ { _id: , 169 | ip: 'localhost', 170 | pid: , 171 | port: , 172 | active: true, 173 | updated: