├── .all-contributorsrc ├── .gitignore ├── README.md ├── clients └── turbo-geth.md ├── ethereum-node ├── .template.env ├── readme.md └── rinkeby.yml ├── graph-node ├── advanced │ ├── README.md │ ├── docker-compose.png │ └── docker-compose.yml ├── basic │ ├── README.md │ ├── docker-compose.png │ └── docker-compose.yml └── insane │ ├── README.md │ ├── docker-compose.yml │ ├── grafana │ ├── grafana.ini │ └── provisioning │ │ ├── dashboards │ │ ├── default.yaml │ │ ├── indexing.json │ │ ├── metrics.json │ │ └── postgres.json │ │ └── datasources │ │ ├── postgres.yaml │ │ └── prometheus.yaml │ ├── nginx │ └── nginx.conf │ └── prometheus │ └── prometheus.yml ├── indexer-agent └── README.md ├── indexer-install ├── .template.env ├── README.md ├── grafana │ ├── grafana.ini │ └── provisioning │ │ ├── dashboards │ │ ├── default.yaml │ │ ├── indexing.json │ │ ├── metrics.json │ │ └── postgres.json │ │ └── datasources │ │ ├── postgres.yaml │ │ └── prometheus.yaml ├── graph.yml ├── non-swarm │ ├── .template.env │ ├── docker-compose.yml │ ├── grafana │ │ ├── grafana.ini │ │ └── provisioning │ │ │ ├── dashboards │ │ │ ├── default.yaml │ │ │ ├── indexing.json │ │ │ ├── metrics.json │ │ │ └── postgres.json │ │ │ └── datasources │ │ │ ├── postgres.yaml │ │ │ └── prometheus.yaml │ ├── nginx │ │ ├── .htpasswd │ │ └── nginx.conf │ └── prometheus │ │ └── prometheus.yml ├── prometheus │ └── prometheus.yml ├── screenshots │ ├── status.png │ ├── swarmpit.png │ └── traefik.png ├── scripts │ ├── 0.0-run-docker-compose-viz.sh │ ├── 1.1-subgraph-create-jannis-gravity.sh │ ├── 1.1.out │ ├── 1.2-subgraph-deploy-jannis-gravity.sh │ ├── 1.2.out │ ├── 2.1-subgraph-create-molochventures-moloch.sh │ ├── 2.1.out │ ├── 2.2-subgraph-deploy-molochventures-moloch.sh │ ├── 2.2.out │ ├── 3.1-subgraph-create-uniswap-uniswap-v2.sh │ ├── 3.1.out │ ├── 3.2-subgraph-deploy-uniswap-uniswap-v2.sh │ ├── 3.2.out │ ├── 4.1-subgraph-create-synthetixio-team-synthetix.sh │ ├── 4.1.out │ ├── 4.2-subgraph-deploy-synthetixio-team-synthetix.sh │ ├── 4.2.out │ ├── 5.1-health-check-indexer-node.sh │ ├── 5.1.out │ ├── 6.1-docker-build-indexer-service.sh │ ├── 7.1-docker-build-indexer-agent.sh │ └── readme.md ├── swarmpit.yml └── traefik.yml ├── monitoring ├── README.md ├── dashboards │ ├── indexing.json │ ├── metrics.json │ └── postgres.json ├── docker-compose.png ├── docker-compose.yml └── prometheus.yml ├── nginx ├── README.md ├── indexer.conf └── monitoring.conf ├── notes.md ├── notes └── levers-to-pull.md ├── performance └── readme.md └── vps-introduction └── readme.md /.all-contributorsrc: -------------------------------------------------------------------------------- 1 | { 2 | "files": [ 3 | "README.md" 4 | ], 5 | "imageSize": 100, 6 | "commit": false, 7 | "contributors": [ 8 | { 9 | "login": "pi0neerpat", 10 | "name": "Patrick Gallagher", 11 | "avatar_url": "https://avatars1.githubusercontent.com/u/35622595?v=4", 12 | "profile": "http://oneclickdapp.com", 13 | "contributions": [ 14 | "doc" 15 | ] 16 | }, 17 | { 18 | "login": "yasiryagi", 19 | "name": "yasiryagi", 20 | "avatar_url": "https://avatars2.githubusercontent.com/u/4862448?v=4", 21 | "profile": "https://github.com/yasiryagi", 22 | "contributions": [ 23 | "doc" 24 | ] 25 | }, 26 | { 27 | "login": "pkrasam", 28 | "name": "pk", 29 | "avatar_url": "https://avatars1.githubusercontent.com/u/4514654?v=4", 30 | "profile": "https://github.com/pkrasam", 31 | "contributions": [ 32 | "doc", 33 | "code" 34 | ] 35 | }, 36 | { 37 | "login": "amxx", 38 | "name": "Hadrien Croubois", 39 | "avatar_url": "https://avatars0.githubusercontent.com/u/2432299?v=4", 40 | "profile": "http://hadriencroubois.com", 41 | "contributions": [ 42 | "code" 43 | ] 44 | } 45 | ], 46 | "contributorsPerLine": 7, 47 | "projectName": "indexer-docker-compose", 48 | "projectOwner": "pi0neerpat", 49 | "repoType": "github", 50 | "repoHost": "https://github.com", 51 | "skipCi": true 52 | } 53 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | .cache 2 | .env 3 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 |

Welcome to indexer-docker-compose 👋

2 |

3 | 4 | Twitter: pi0neerpat 5 | 6 |

7 | 8 | 9 | 10 | [![All Contributors](https://img.shields.io/badge/all_contributors-4-orange.svg?style=flat-square)](#contributors-) 11 | 12 | 13 | 14 | > A guide to set up The Graph infrastructure with Docker Compose / Nginx 15 | 16 | ## Guides 17 | 18 | > ❓ Are you a self-hosted noob? There's a useful guide to setting up a new server in [/vps-introduction](./vps-introduction). I wouldn't recommend starting with this project without some basic experience hosting your own services. 19 | 20 | ### 👶 Phase 0 - Deploy Graph Node 21 | 22 | Complete these in this following order: 23 | 24 | 1. [Graph Node docker-compose - BASIC](./graph-node/basic) 25 | 2. [Graph Node docker-compose - ADVANCED](./graph-node/advanced) 26 | 3. [Monitoring Infra](./monitoring) 27 | 28 | _Extra Resources_: 29 | 30 | - See `/graph-node/insane` for a full phase 0 docker-compose from @amxx 31 | 32 | ### ⛏️ Phase 1 - Staking, Basic Actions & Customization 33 | 34 | Getting started with "Indexing-as-a-service" 35 | 36 | #### Mission 1 37 | 38 | 1. Install the Indexer using Swarm - [indexer-install](./indexer-install) 39 | 2. (Soon) Using the Indexer-Agent CLI - [indexer-agent](./indexer-agent) 40 | 41 | #### Mission 2 42 | 43 | Use the new Dashboard to signal + stake. (Follow instructions provided by TheGraph) 44 | 45 | _Extra Resources_: 46 | 47 | - See `/indexer-install/non-swarm` for a vanilla docker-compose from @amxx 48 | - See `/indexer-install/scripts` for installation and testing scripts from @pkrasam 49 | 50 | ### 🤑 Phase 2 - Honest Profit Maximization 51 | 52 | There is only 1 mission here. No new components were added. 53 | 54 | ---- 55 | 56 | (not yet started) Phase 3 - Network Stress Test 57 | 58 | (not yet started) Phase 4 - Rational Profit Maximization 59 | 60 | --- 61 | ## Author 62 | 63 | 👤 **Patrick Gallagher** 64 | 65 | - Website: https://patrickgallagher.dev 66 | - Twitter: [@pi0neerpat](https://twitter.com/pi0neerpat) 67 | - Github: [@pi0neerpat](https://github.com/pi0neerpat) 68 | 69 | ## 🤝 Contributing 70 | 71 | Contributions are very welcome. Please follow the rToken project [code of conduct](https://github.com/rtoken-project/rtoken-monorepo/blob/master/code-of-conduct.md). 72 | 73 | ## Show your support 74 | 75 | Don't forget to mention `@pi0neerpat` in the feedback surveys :wink: 76 | 77 | Give a ⭐️ if this project helped you! 78 | 79 | ## Contributors ✨ 80 | 81 | Thanks goes to these wonderful people ([emoji key](https://allcontributors.org/docs/en/emoji-key)): 82 | 83 | 84 | 85 | 86 | 87 | 88 | 89 | 90 | 91 | 92 |

Patrick Gallagher
📖
yasiryagi
📖
pk
📖 💻
93 | 94 | 95 | 96 | 97 | 98 | 99 | This project follows the [all-contributors](https://github.com/all-contributors/all-contributors) specification. Contributions of any kind welcome! 100 | 101 | --- 102 | 103 | _This README was generated with ❤️ by [readme-md-generator](https://github.com/kefranabg/readme-md-generator)_ 104 | -------------------------------------------------------------------------------- /clients/turbo-geth.md: -------------------------------------------------------------------------------- 1 |

Turbo Geth Support for Indexers

2 | 3 | TODO 4 | -------------------------------------------------------------------------------- /ethereum-node/.template.env: -------------------------------------------------------------------------------- 1 | HOSTNAME=https://rinkeby.* 2 | EMAIL= 3 | USERNAME= 4 | PASSWORD= 5 | -------------------------------------------------------------------------------- /ethereum-node/readme.md: -------------------------------------------------------------------------------- 1 | ### Rinkeby GETH node 2 | 3 | ```bash 4 | export $(grep -v '^#' .env | xargs -0) 5 | docker stack deploy -c rinkeby.yml rinkeby 6 | ``` 7 | -------------------------------------------------------------------------------- /ethereum-node/rinkeby.yml: -------------------------------------------------------------------------------- 1 | version: "3" 2 | 3 | # run this rinkeby node with 4 | # HOSTNAME=my-rinkeby.domain EMAIL=any@email.address docker-compose up -d 5 | # it will be open to the world. easiest wayy to secure it would be putting some basic auth in the nginx config 6 | # For this look at https://github.com/nginx-proxy/nginx-proxy#basic-authentication-support 7 | 8 | services: 9 | rinkeby-rpc: 10 | image: ethereum/client-go 11 | restart: unless-stopped 12 | ports: 13 | - "30303:30303" 14 | - "30303:30303/udp" 15 | # - "8545:8545" 16 | volumes: 17 | - ~/data/rinkeby/chaindata:/chaindata 18 | stop_signal: SIGINT 19 | stop_grace_period: 2m 20 | command: 21 | - --nousb 22 | - --http 23 | - --http.api 24 | - "eth,net,web3" 25 | - --http.addr=0.0.0.0 26 | - --http.vhosts=* 27 | - --http.corsdomain=* 28 | - --ws 29 | - --ws.origins=* 30 | - --ws.addr=0.0.0.0 31 | - --ws.api 32 | - "eth,net,web3" 33 | - --graphql 34 | - --rinkeby 35 | - --graphql.corsdomain=* 36 | - --graphql.vhosts=* 37 | - --datadir 38 | - "/chaindata" 39 | - --syncmode=fast 40 | networks: 41 | - default 42 | - traefik-public 43 | deploy: 44 | mode: replicated 45 | replicas: 1 46 | placement: 47 | constraints: 48 | - node.role == manager 49 | # resources: 50 | # limits: 51 | # memory: 128M 52 | # reservations: 53 | # memory: 64M 54 | labels: 55 | - traefik.enable=true 56 | - traefik.docker.network=traefik-public 57 | - traefik.constraint-label=traefik-public 58 | - "traefik.http.routers.rinkeby-http.rule=Host(`${DOMAIN}`)" 59 | - traefik.http.routers.rinkeby-http.entrypoints=http 60 | - traefik.http.routers.rinkeby-http.middlewares=https-redirect 61 | - "traefik.http.routers.rinkeby-https.rule=Host(`${DOMAIN}`)" 62 | - traefik.http.routers.rinkeby-https.entrypoints=https 63 | - traefik.http.routers.rinkeby-https.tls=true 64 | - traefik.http.routers.rinkeby-https.tls.certresolver=le 65 | - traefik.http.services.rinkeby.loadbalancer.server.port=8545 66 | - traefik.http.middlewares.rinkeby-auth.basicauth.users=${USERNAME?Variable not set}:${PASSWORD?Variable not set} 67 | - traefik.http.routers.rinkeby-https.middlewares=rinkeby-auth 68 | 69 | networks: 70 | traefik-public: 71 | external: true 72 | -------------------------------------------------------------------------------- /graph-node/advanced/README.md: -------------------------------------------------------------------------------- 1 |

Graph Node docker-compose - ADVANCED

2 | 3 | # Table of Contents 4 | 5 | 6 | 7 | - [Table of Contents](#table-of-contents) 8 | - [Outline](#outline) 9 | - [Run separate Indexer and Query nodes](#run-separate-indexer-and-query-nodes) 10 | - [Add Websockets & Health Monitoring](#add-websockets--health-monitoring) 11 | - [Update Nginx config](#update-nginx-config) 12 | - [Check Indexer health](#check-indexer-health) 13 | - [Harden the Postgres Database](#harden-the-postgres-database) 14 | - [Bonus: Deploy your own subgraph](#bonus-deploy-your-own-subgraph) 15 | - [Extras](#extras) 16 | - [Next Steps](#next-steps) 17 | 18 | 19 | # Outline 20 | 21 | :exclamation: This guide assumes you've completed [Graph Node docker-compose - BASIC](../basic). 22 | 23 | Now we need to improve our setup to meet "production" demands 24 | 25 | # Run separate Indexer and Query nodes 26 | 27 | So far we have been running a `graph-node` in "combined-node" mode. This means it performs both indexing and serving queries. 28 | 29 | We need to separate these functions since the demand for each may scale up/down depending on how many subgraphs are syncing, and how many requests we are serving, 30 | 31 | Stop your existing graph-nodes 32 | 33 | ```bash 34 | cd ~/indexer-docker-compose/graph-node/basic && docker-compose down 35 | ``` 36 | 37 | Navigate to this folder, update `docker-compose.yml` with your Postgres login and Web3 Provider info, and then start the new docker compose. 38 | 39 | ```bash 40 | cd ~/indexer-docker-compose/graph-node/advanced 41 | nano docker-compose.yml 42 | docker-compose up -d 43 | ``` 44 | 45 | In the new `docker-compose.yml` we create two separate instances of graph-nodes. One in "query-node" mode, and the other in "index-node" mode. 46 | 47 | ```yaml 48 | services: 49 | graph-node-query: 50 | # ... 51 | environment: 52 | node_role: "query-node" 53 | node_id: "query-node" 54 | graph-node-indexer: 55 | # ... 56 | ports: 57 | - "8100:8000" # http 58 | - "8120:8020" # json-rpc 59 | - "8140:8040" # metrics 60 | environment: 61 | node_role: "index-node" 62 | node_id: "index-node" 63 | BLOCK_INGESTOR: "index-node" 64 | ``` 65 | 66 | Things to pay attention to: 67 | 68 | - Deploying subgraphs is now handled by the "index-node", while queries are handled by the "query-node". 69 | - We are binding the "index-node" to a different set of ports eg. `81XX`. Your create/deploy commands will need to reflect this port change. 70 | - In the BASIC guide we set Postgres to store data in `~/subgraph-data/postgres`. Since we do the same here, we won't lose any existing subgraph sync data (unless you changed your authentication) 71 | 72 | # Add Websockets & Health Monitoring 73 | 74 | ## Update Nginx config 75 | 76 | In order to support websockets and health monitoring, we must change our Nginx config. Replace the `indexer.conf` you created in the BASIC guide to the final one in the `/nginx` folder of this repo. You will need to update the `server_name`. 77 | 78 | ```bash 79 | sudo cp ~/indexer-docker-compose/nginx/indexer.conf /etc/nginx/sites-enabled 80 | 81 | nano /etc/nginx/sites-enabled/indexer.conf 82 | # Update server_name 83 | ``` 84 | 85 | Next update `/etc/nginx/nginx.conf` to add support for [Nginx connection upgrades](http://nginx.org/en/docs/http/websocket.html) as follows: 86 | 87 | ```bash 88 | sudo sudo nano /etc/nginx/nginx.conf 89 | ``` 90 | 91 | ``` 92 | http { 93 | # Add this code block within "http" 94 | map $http_upgrade $connection_upgrade { 95 | default upgrade; 96 | '' close; 97 | } 98 | } 99 | ``` 100 | 101 | Excellent! Now we have support for websockets, and we can perform health checks against the query node. Let's put the changes into effect: 102 | 103 | ```bash 104 | sudo nginx -t 105 | sudo systemctl reload nginx 106 | ``` 107 | 108 | We also need to generate SSL certs to acces `https://indexer.mysite.com/index-node/` from the browser 109 | 110 | Now we need Certbot to issue a certificate. 111 | 112 | ```bash 113 | # Install 114 | sudo apt-get install software-properties-common 115 | sudo add-apt-repository ppa:certbot/certbot 116 | sudo apt-get update 117 | sudo apt-get install python-certbot-nginx 118 | 119 | # Run 120 | sudo certbot --nginx certonly 121 | 122 | # If successful, restart Nginx 123 | sudo systemctl reload nginx 124 | ``` 125 | 126 | Things to pay attention to: 127 | 128 | - Health checks are performed against `index-node/` 129 | - This config only exposes the "query-node" on ports `80XX`. If you want external access to your "index-node", you will need to add that. 130 | 131 | ## Check Indexer health 132 | 133 | To check the health and status of a particular subgraph, we use the `index-node/graphql` endpoint. Here is an example of how to check the subgraph `jannis/gravity`: 134 | 135 | | Property | value | 136 | | ------------ | ---------------------------------------------- | 137 | | URL | `http://indexer.mysite.com/index-node/graphql` | 138 | | Request type | POST | 139 | 140 | ```graphql 141 | # Query body 142 | { 143 | indexingStatusForCurrentVersion(subgraphName: "jannis/gravity") { 144 | synced 145 | health 146 | fatalError { 147 | message 148 | block { 149 | number 150 | hash 151 | } 152 | handler 153 | } 154 | chains { 155 | chainHeadBlock { 156 | number 157 | } 158 | latestBlock { 159 | number 160 | } 161 | } 162 | } 163 | } 164 | ``` 165 | 166 | You should get a response like this: 167 | 168 | ```json 169 | { 170 | "data": { 171 | "indexingStatusForCurrentVersion": { 172 | "chains": [ 173 | { 174 | "chainHeadBlock": { 175 | "number": "10637299" 176 | }, 177 | "latestBlock": { 178 | "number": "10637299" 179 | } 180 | } 181 | ], 182 | "fatalError": null, 183 | "health": "healthy", 184 | "synced": true 185 | } 186 | } 187 | } 188 | ``` 189 | 190 | > :100: PRO-TIP: Learn more about health checks [here](https://thegraph.com/docs/deploy-a-subgraph#checking-subgraph-health) 191 | 192 | # Harden the Postgres Database 193 | 194 | The Graph team does not recommend running postgres using `docker-compose`, since it needs to be very stable. I have not needed to do this yet, however I will list your options just in case: 195 | 196 | - Run Postgres as systemd service on your server 197 | - Use a 3rd-party provider 198 | 199 | # Bonus: Deploy your own subgraph 200 | 201 | In this example we will use the subgraph `jannis/gravity` to demonstrate how you would deploy your own subgraph to your indexer. 202 | 203 | First install `graph-cli`, download the desired subgraph repo, and make any changes to the `subgraph.yaml` 204 | 205 | ```bash 206 | # Install the graph-cli 207 | npm i -g @graphprotocol/graph-cli 208 | 209 | # Use any existing subgraph 210 | git clone https://github.com/Jannis/gravity-subgraph.git && cd gravity-subgraph 211 | ``` 212 | 213 | Now we are ready to push our subgraph to our indexer. 214 | 215 | In `package.json`, add the following scripts (don't forget to add a comma in the line above) 216 | 217 | ```json 218 | "create-indexer": "graph create jannis/gravity --node http://127.0.0.1:8020", 219 | "deploy-indexer": "graph deploy jannis/gravity --debug --ipfs https://testnet.thegraph.com/ipfs/ --node http://127.0.0.1:8020" 220 | ``` 221 | 222 | Now generate the files, and deploy 223 | 224 | ```bash 225 | yarn 226 | yarn codegen 227 | yarn create-indexer 228 | yarn deploy-indexer 229 | ``` 230 | 231 | If successful, you will see `Deployed to http://127.0.0.1:8000/subgraphs/name/jannis/gravity/graphql`. Check that your subgraph is syncing using docker logs, as mentioned above, and happy querying! 232 | 233 | # Extras 234 | 235 | ### Useful Commands 236 | 237 | You may find these useful to check how you server is performing 238 | 239 | ```bash 240 | # Check Memory 241 | free -m 242 | ps -o pid,user,%mem,command ax | sort -b -k3 -r 243 | 244 | # Check Storage 245 | ncdu 246 | 247 | # Hardware monitoring dashboard in a terminal 248 | # https://github.com/bcicen/ctop 249 | docker run --rm -ti \ 250 | --name=ctop \ 251 | --volume /var/run/docker.sock:/var/run/docker.sock:ro \ 252 | quay.io/vektorlab/ctop:latest 253 | ``` 254 | 255 | # Next Steps 256 | 257 | You are absolutely crushing it! 💪 258 | 259 | Continue to [Monitoring Infra](../../monitoring) 260 | -------------------------------------------------------------------------------- /graph-node/advanced/docker-compose.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/pi0neerpat/indexer-docker-compose/7aaaa53170c2e20e49a6f2a4ac5982cc0a6f7b9b/graph-node/advanced/docker-compose.png -------------------------------------------------------------------------------- /graph-node/advanced/docker-compose.yml: -------------------------------------------------------------------------------- 1 | version: "3" 2 | services: 3 | # Query 4 | # adapted from https://github.com/graphprotocol/mission-control-indexer/blob/8cc08c4c72ed7a7fcab2bfeace172626c9d08ee3/k8s/base/query-node/deployment.yaml 5 | graph-node-query: 6 | image: graphprotocol/graph-node:latest 7 | ports: 8 | - "8000:8000" # http 9 | - "8030:8030" # index-node 10 | - "8001:8001" # ws 11 | - "8040:8040" # metrics 12 | depends_on: 13 | - postgres 14 | environment: 15 | postgres_host: postgres:5432 16 | postgres_user: graph-node 17 | postgres_pass: let-me-in 18 | postgres_db: graph-node 19 | ipfs: "https://testnet.thegraph.com/ipfs/" 20 | # CHANGE ME 21 | ethereum: "mainnet:https://mainnet.infura.io/v3/" 22 | node_role: "query-node" 23 | node_id: "query-node" 24 | GRAPH_KILL_IF_UNRESPONSIVE: "true" 25 | EXPERIMENTAL_SUBGRAPH_VERSION_SWITCHING_MODE: "synced" # ? 26 | RUST_LOG: info 27 | restart: always 28 | 29 | # Index 30 | # adapted from https://github.com/graphprotocol/mission-control-indexer/blob/8cc08c4c72ed7a7fcab2bfeace172626c9d08ee3/k8s/base/index-node/stateful_set.yaml 31 | graph-node-indexer: 32 | image: graphprotocol/graph-node:latest 33 | ports: 34 | - "8100:8000" # http 35 | - "8120:8020" # json-rpc 36 | - "8140:8040" # metrics 37 | depends_on: 38 | - postgres 39 | environment: 40 | postgres_host: postgres:5432 41 | postgres_user: graph-node 42 | postgres_pass: let-me-in 43 | postgres_db: graph-node 44 | ipfs: "https://testnet.thegraph.com/ipfs/" 45 | # CHANGE ME 46 | ethereum: "mainnet:https://mainnet.infura.io/v3/" 47 | node_role: "index-node" 48 | node_id: "index-node" 49 | BLOCK_INGESTOR: "index-node" # Only need to specify one block ingestor 50 | GRAPH_KILL_IF_UNRESPONSIVE: "true" 51 | RUST_LOG: info 52 | restart: always 53 | 54 | postgres: 55 | image: postgres 56 | ports: 57 | - "5432:5432" 58 | command: ["postgres", "-cshared_preload_libraries=pg_stat_statements"] 59 | environment: 60 | POSTGRES_USER: graph-node 61 | # CHANGE ME 62 | POSTGRES_PASSWORD: let-me-in 63 | POSTGRES_DB: graph-node 64 | volumes: 65 | - ~/subgraph-data/postgres:/var/lib/postgresql/data 66 | restart: always 67 | -------------------------------------------------------------------------------- /graph-node/basic/README.md: -------------------------------------------------------------------------------- 1 |

Graph Node docker-compose - BASIC

2 | 3 | # Table of Contents 4 | 5 | 6 | 7 | - [Table of Contents](#table-of-contents) 8 | - [Overview](#overview) 9 | - [Prerequisites](#prerequisites) 10 | - [Route requests with Nginx](#route-requests-with-nginx) 11 | - [Indexer Infrastructure](#indexer-infrastructure) 12 | - [Start docker-compose](#start-docker-compose) 13 | - [Deploy your first Subgraph](#deploy-your-first-subgraph) 14 | - [Deploy the required subgraphs](#deploy-the-required-subgraphs) 15 | - [Test the Indexer](#test-the-indexer) 16 | - [Next steps](#next-steps) 17 | 18 | 19 | # Overview 20 | 21 | 1. Provision a server and set up reverse-proxy routing using Nginx 22 | 2. Start `graph-node` using `docker-compose` 23 | 3. Deploy the "gravity" subgraph to the `graph-node` instance using `httpie` 24 | 4. After syncing, query the indexer from your local machine to test that it works 25 | 26 | ## Prerequisites 27 | 28 | ### Software 29 | 30 | You need the following installed on your server: 31 | 32 | - docker 33 | - docker-compose 34 | - nginx 35 | - httpie 36 | 37 | If you are unfamiliar with using Nginx, I've added a basic introduction in the [Nginx folder](../../nginx). 38 | 39 | ### Clone this repo 40 | 41 | Clone this repo to `~/indexer-docker-compose` with: 42 | 43 | ```bash 44 | git clone https://github.com/pi0neerpat/indexer-docker-compose.git ~/indexer-docker-compose 45 | ``` 46 | 47 | ### Web3 Provider 48 | 49 | You will also need a Web3 Provider. For this tutorial you can use the free Infura tier, however it will not work for any subgraphs except `jannis/gravity`. See the wiki [Setup: Ethereum Nodes and Providers](https://github.com/graphprotocol/mission-control-indexer/wiki/Setup:-Ethereum-Nodes-and-Providers) for more info. 50 | 51 | # Route requests with Nginx 52 | 53 | Let's expose port `8000` on our domain using Nginx. 54 | 55 | ```bash 56 | sudo nano /etc/nginx/sites-enabled/indexer.conf 57 | ``` 58 | 59 | Paste the following, and update your `server_name`. Here we are using the subdomain "indexer". 60 | 61 | ```js 62 | server { 63 | # Update your domain here 64 | server_name indexer.mysite.com; 65 | 66 | location / { 67 | proxy_pass http://localhost:8000; 68 | proxy_http_version 1.1; 69 | proxy_set_header Upgrade $http_upgrade; 70 | proxy_set_header Connection 'upgrade'; 71 | proxy_set_header Host $host; 72 | proxy_cache_bypass $http_upgrade; 73 | } 74 | 75 | error_page 404 404.html; 76 | 77 | access_log /var/log/nginx/access.log; 78 | error_log /var/log/nginx/error.log; 79 | } 80 | ``` 81 | 82 | Next, check your config is correct and restart the Nginx service 83 | 84 | ```bash 85 | sudo nginx -t 86 | # nginx: the configuration file /etc/nginx/nginx.conf syntax is ok 87 | # nginx: configuration file /etc/nginx/nginx.conf test is successful 88 | 89 | sudo systemctl reload nginx 90 | ``` 91 | 92 | # Indexer Infrastructure 93 | 94 | ## Start docker-compose 95 | 96 | Navigate to this folder and update `docker-compose.yml` with your Infura key: 97 | 98 | ```yaml 99 | version: "3" 100 | services: 101 | graph-node: 102 | image: graphprotocol/graph-node:latest 103 | # ... 104 | # UPDATE HERE 105 | ethereum: "mainnet:https://mainnet.infura.io/v3/" 106 | postgres: 107 | #... 108 | environment: 109 | POSTGRES_USER: graph-node 110 | POSTGRES_PASSWORD: let-me-in # CHANGE ME 111 | ``` 112 | 113 | Create a folder to hold your Postgres data 114 | 115 | ``` 116 | mkdir ~/subgraph-data/postgres 117 | ``` 118 | 119 | Start the containers using these commands: 120 | 121 | ```bash 122 | docker-compose up -d 123 | 124 | # Then you can inspect logs with 125 | docker-compose logs -f 126 | 127 | # If you're having trouble getting it started try restarting with this command 128 | # WARNING!! - may delete all your data and require complete re-syncing 129 | sudo rm -rf data && docker-compose up -d 130 | ``` 131 | 132 | ## Deploy your first Subgraph 133 | 134 | First, allocate the name of your subgraph on your indexer. 135 | 136 | ```bash 137 | http post localhost:8020 \ 138 | jsonrpc="2.0" \ 139 | id="1" \ 140 | method="subgraph_create" \ 141 | params:='{"name": "jannis/gravity"}' 142 | ``` 143 | 144 | Next deploy the subgraph located at the ipfs hash 145 | 146 | ```bash 147 | http post localhost:8020 \ 148 | jsonrpc="2.0" \ 149 | id="1" \ 150 | method="subgraph_deploy" \ 151 | params:='{"name": "jannis/gravity", "ipfs_hash": "QmbeDC4G8iPAUJ6tRBu99vwyYkaSiFwtXWKwwYkoNphV4X"}' 152 | ``` 153 | 154 | If successful, you should see a response like this: 155 | 156 | ```bash 157 | HTTP/1.1 200 OK 158 | content-length: 199 159 | content-type: application/json; charset=utf-8 160 | date: Tue, 11 Aug 2020 04:01:06 GMT 161 | { 162 | "id": "1", 163 | "jsonrpc": "2.0", 164 | "result": { 165 | "playground": ":8000/subgraphs/name/jannis/gravity/graphql", 166 | "queries": ":8000/subgraphs/name/jannis/gravity", 167 | "subscriptions": ":8001/subgraphs/name/jannis/gravity" 168 | } 169 | } 170 | ``` 171 | 172 | > :100: PRO-TIP: This approach can be used to deploy any existing subgraph. However, the subgraph data must be actively pinned to the IPFS endpoint specified in `docker-compose.yaml`. If the IPFS endpoint cannot find your subgraph, you will need to use `graph-cli deploy` to pin it to your IPFS endpoint. 173 | 174 | Let's check the docker logs to see if our indexer is syncing properly. 175 | 176 | ```bash 177 | # Get the docker container ID 178 | docker ps 179 | > CONTAINER ID IMAGE COMMAND 180 | > c4b58b9800d1 graphprotocol/graph-node:latest "/bin/sh -c start" 181 | 182 | # Open logs for the container 183 | docker logs c4b5 -f 184 | > Aug 11 03:50:55.775 INFO Scanning blocks [2804111, 2805110], range_size: 1000, subgraph_id: QmbeDC4G8iPAUJ6tRBu99vwyY.... 185 | ``` 186 | 187 | ## Deploy the required subgraphs 188 | 189 | To make things easier, here are the commands to deploy all three subgraphs required for the indexer testnet 190 | 191 | ```bash 192 | # Moloch 193 | http post localhost:8020 jsonrpc="2.0" id="1" method="subgraph_create" params:='{"name": "molochventures/moloch"}' 194 | http post localhost:8020 jsonrpc="2.0" id="1" method="subgraph_deploy" params:='{"name": "molochventures/moloch", "ipfs_hash": "QmTXzATwNfgGVukV1fX2T6xw9f6LAYRVWpsdXyRWzUR2H9"}' 195 | 196 | # Uniswap 197 | http post localhost:8020 jsonrpc="2.0" id="1" method="subgraph_create" params:='{"name": "uniswap/uniswap-v2"}' 198 | http post localhost:8020 jsonrpc="2.0" id="1" method="subgraph_deploy" params:='{"name": "uniswap/uniswap-v2", "ipfs_hash": "QmXKwSEMirgWVn41nRzkT3hpUBw29cp619Gx58XW6mPhZP"}' 199 | 200 | # Synthetix 201 | http post localhost:8020 jsonrpc="2.0" id="1" method="subgraph_create" params:='{"name": "synthetixio-team/synthetix"}' 202 | http post localhost:8020 jsonrpc="2.0" id="1" method="subgraph_deploy" params:='{"name": "synthetixio-team/synthetix", "ipfs_hash": "Qme2hDXrkBpuXAYEuwGPAjr6zwiMZV4FHLLBa3BHzatBWx"}' 203 | ``` 204 | 205 | To remove a subgraph, use this command: 206 | 207 | ```bash 208 | http post localhost:8020 jsonrpc="2.0" id="1" method="subgraph_remove" params:='{"name": "jannis/gravity"}' 209 | ``` 210 | 211 | To reassign a subgraph, use this command (effectively changing the node ID): 212 | ```bash 213 | http post localhost:8120 jsonrpc="2.0" id="1" method="subgraph_reassign" params:='{"name": "uniswap/uniswap-v2", "ipfs_hash": "QmXKwSEMirgWVn41nRzkT3hpUBw29cp619Gx58XW6mPhZP", "node_id": "something"}' 214 | ``` 215 | If you did not assign a node id at deployment it is assigned "default". 216 | To restart the subgraph, assign it to a different node id and then back to the orginal. 217 | 218 | ## Test the Indexer 219 | 220 | Great job! Now that you've deployed your subgraph, let's make a call to our indexer to see if it works! 221 | 222 | Using your favorite API testing tool, such as [Postman](https://www.postman.com/downloads/), create a query as follows: 223 | 224 | | Property | value | 225 | | ------------ | --------------------------------------------------------- | 226 | | URL | `http://indexer.mysite.com/subgraphs/name/jannis/gravity` | 227 | | Request type | POST | 228 | 229 | ```graphql 230 | # Query body 231 | query { 232 | gravatars(first: 5) { 233 | id 234 | owner 235 | displayName 236 | imageUrl 237 | } 238 | } 239 | ``` 240 | 241 | You should get a response like this: 242 | 243 | > NOTE: It may take up to 20 minutes to start to see any data (32 GB ram, 8 vCPU, Infura). Until then you will probably get a response like `"gravatars": []` 244 | 245 | ```json 246 | { 247 | "data": { 248 | "gravatars": [ 249 | { 250 | "displayName": "w1m3l", 251 | "id": "0x10", 252 | "imageUrl": "https://ucarecdn.com/98c4659f-70e4-4ad3-b7fd-c92ab570344c/-/crop/1114x1115/0,63/-/preview/", 253 | "owner": "0x2664984287ed631529747ae0c76935e7c9b6e603" 254 | } 255 | // ... 256 | ] 257 | } 258 | } 259 | ``` 260 | 261 | # Next steps 262 | 263 | Superb job! :+1: You've started indexing subgraphs, and can query it using your server's public GraphQL endpoint. 264 | 265 | Continue to [Graph Node docker-compose - ADVANCED](../advanced) 266 | -------------------------------------------------------------------------------- /graph-node/basic/docker-compose.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/pi0neerpat/indexer-docker-compose/7aaaa53170c2e20e49a6f2a4ac5982cc0a6f7b9b/graph-node/basic/docker-compose.png -------------------------------------------------------------------------------- /graph-node/basic/docker-compose.yml: -------------------------------------------------------------------------------- 1 | version: "3" 2 | services: 3 | graph-node: 4 | image: graphprotocol/graph-node:latest 5 | ports: 6 | - "8000:8000" 7 | - "8001:8001" 8 | - "8020:8020" 9 | - "8030:8030" 10 | - "8040:8040" 11 | depends_on: 12 | - postgres 13 | environment: 14 | postgres_host: postgres:5432 15 | postgres_user: graph-node 16 | postgres_pass: let-me-in 17 | postgres_db: graph-node 18 | ipfs: "https://testnet.thegraph.com/ipfs/" 19 | # CHANGE ME 20 | ethereum: "mainnet:https://mainnet.infura.io/v3/" 21 | RUST_LOG: info 22 | node_role: "combined-node" 23 | node_id: "default" 24 | postgres: 25 | image: postgres 26 | ports: 27 | - "5432:5432" 28 | command: ["postgres", "-cshared_preload_libraries=pg_stat_statements"] 29 | environment: 30 | POSTGRES_USER: graph-node 31 | # CHANGE ME 32 | POSTGRES_PASSWORD: let-me-in 33 | POSTGRES_DB: graph-node 34 | volumes: 35 | - ~/subgraph-data/postgres:/var/lib/postgresql/data 36 | -------------------------------------------------------------------------------- /graph-node/insane/README.md: -------------------------------------------------------------------------------- 1 |

Graph Node docker-compose - INSANE

2 | 3 | Thanks to [@amxx](https://github.com/amxx/) for providing this option! 4 | 5 | Docs will probably not be added here. You know what you're doing right? ;) 6 | -------------------------------------------------------------------------------- /graph-node/insane/docker-compose.yml: -------------------------------------------------------------------------------- 1 | version: '3' 2 | services: 3 | 4 | graphnode-query-1: 5 | image: graphprotocol/graph-node:latest 6 | restart: always 7 | depends_on: 8 | - postgres 9 | ports: 10 | - '8000:8000' 11 | - '8001:8001' 12 | - '8030:8030' 13 | environment: 14 | postgres_host: postgres:5432 15 | postgres_user: graph-node 16 | postgres_pass: 17 | postgres_db: graph-node 18 | ipfs: 'https://testnet.thegraph.com/ipfs/' 19 | ethereum: 'mainnet:https://mainnet.infura.io/v3/' 20 | GRAPH_NODE_ID: 'missioncontrol_query_1' 21 | DISABLE_BLOCK_INGESTOR: 'true' 22 | logging: 23 | driver: "json-file" 24 | options: 25 | max-size: "200k" 26 | max-file: "10" 27 | 28 | graphnode-indexer-1: 29 | image: graphprotocol/graph-node:latest 30 | restart: always 31 | depends_on: 32 | - postgres 33 | ports: 34 | - '8020:8020' 35 | - '8040:8040' 36 | environment: 37 | postgres_host: postgres:5432 38 | postgres_user: graph-node 39 | postgres_pass: 40 | postgres_db: graph-node 41 | ipfs: 'https://testnet.thegraph.com/ipfs/' 42 | ethereum: ' 43 | mainnet:https://mainnet.infura.io/v3/ 44 | ropsten:https://ropsten.infura.io/v3/ 45 | rinkeby:https://rinkeby.infura.io/v3/ 46 | goerli:https://goerli.infura.io/v3/ 47 | kovan:https://kovan.infura.io/v3/ 48 | ' 49 | GRAPH_NODE_ID: 'missioncontrol_indexer_1' 50 | #GRAPH_ETHEREUM_CLEANUP_BLOCKS: 'true' 51 | logging: 52 | driver: "json-file" 53 | options: 54 | max-size: "200k" 55 | max-file: "10" 56 | 57 | postgres: 58 | image: postgres 59 | user: '1000:1000' 60 | restart: always 61 | command: 62 | - "postgres" 63 | - "-cshared_preload_libraries=pg_stat_statements" 64 | ports: 65 | - '5432:5432' 66 | environment: 67 | POSTGRES_USER: graph-node 68 | POSTGRES_PASSWORD: 20b3b5fc7430d88ad8f7a9d5ed4dc6d45805754e212a85bef63668d82bc0dfe4 69 | POSTGRES_DB: graph-node 70 | volumes: 71 | - /data/postgres:/var/lib/postgresql/data 72 | 73 | prometheus: 74 | image: prom/prometheus 75 | user: '1000:1000' 76 | restart: always 77 | depends_on: 78 | - graphnode-indexer-1 79 | command: 80 | - "--log.level=warn" 81 | - "--config.file=/etc/prometheus/prometheus.yml" 82 | - "--storage.tsdb.path=/prometheus" 83 | ports: 84 | - 9090:9090 85 | volumes: 86 | - ./prometheus:/etc/prometheus:ro 87 | - /data/prometheus:/prometheus 88 | 89 | grafana: 90 | image: grafana/grafana 91 | user: '1000:1000' 92 | restart: always 93 | depends_on: 94 | - postgres 95 | - prometheus 96 | ports: 97 | - 3000:3000 98 | environment: 99 | postgres_host: postgres:5432 100 | postgres_user: graph-node 101 | postgres_pass: 102 | postgres_db: graph-node 103 | volumes: 104 | - ./grafana:/etc/grafana/ 105 | - /data/grafana:/var/lib/grafana 106 | 107 | nginx: 108 | image: nginx 109 | restart: always 110 | ports: 111 | - '80:80' 112 | - '443:443' 113 | volumes: 114 | - ./nginx:/etc/nginx:ro 115 | - /etc/letsencrypt:/etc/letsencrypt 116 | -------------------------------------------------------------------------------- /graph-node/insane/grafana/grafana.ini: -------------------------------------------------------------------------------- 1 | [server] 2 | domain = 54.215.29.38 3 | root_url = %(protocol)s://%(domain)s:%(http_port)s/grafana/ 4 | serve_from_sub_path = true 5 | 6 | -------------------------------------------------------------------------------- /graph-node/insane/grafana/provisioning/dashboards/default.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: 1 2 | providers: 3 | - name: 'default' 4 | orgId: 1 5 | folder: '' 6 | folderUid: '' 7 | type: file 8 | allowUiUpdates: true 9 | updateIntervalSeconds: 31536000 10 | options: 11 | path: /etc/grafana/provisioning/dashboards 12 | -------------------------------------------------------------------------------- /graph-node/insane/grafana/provisioning/dashboards/indexing.json: -------------------------------------------------------------------------------- 1 | { 2 | "annotations": { 3 | "list": [ 4 | { 5 | "builtIn": 1, 6 | "datasource": "-- Grafana --", 7 | "enable": true, 8 | "hide": true, 9 | "iconColor": "rgba(0, 211, 255, 1)", 10 | "name": "Annotations & Alerts", 11 | "type": "dashboard" 12 | } 13 | ] 14 | }, 15 | "editable": true, 16 | "gnetId": null, 17 | "graphTooltip": 0, 18 | "id": 4, 19 | "iteration": 1590595997503, 20 | "links": [], 21 | "panels": [ 22 | { 23 | "columns": [], 24 | "datasource": "postgres", 25 | "fieldConfig": { 26 | "defaults": { 27 | "custom": {} 28 | }, 29 | "overrides": [] 30 | }, 31 | "fontSize": "90%", 32 | "gridPos": { 33 | "h": 10, 34 | "w": 24, 35 | "x": 0, 36 | "y": 0 37 | }, 38 | "id": 10, 39 | "links": [], 40 | "pageSize": null, 41 | "scroll": true, 42 | "showHeader": true, 43 | "sort": { 44 | "col": 6, 45 | "desc": true 46 | }, 47 | "styles": [ 48 | { 49 | "alias": "blocks behind", 50 | "align": "auto", 51 | "colorMode": null, 52 | "colors": [ 53 | "rgba(50, 172, 45, 0.97)", 54 | "rgba(237, 129, 40, 0.89)", 55 | "rgba(245, 54, 54, 0.9)" 56 | ], 57 | "dateFormat": "YYYY-MM-DD HH:mm:ss", 58 | "decimals": 0, 59 | "mappingType": 1, 60 | "pattern": "blocks_behind_network", 61 | "thresholds": [ 62 | "" 63 | ], 64 | "type": "number", 65 | "unit": "none" 66 | }, 67 | { 68 | "alias": "", 69 | "align": "auto", 70 | "colorMode": null, 71 | "colors": [ 72 | "rgba(245, 54, 54, 0.9)", 73 | "rgba(237, 129, 40, 0.89)", 74 | "rgba(50, 172, 45, 0.97)" 75 | ], 76 | "decimals": 1, 77 | "pattern": "lag", 78 | "thresholds": [], 79 | "type": "number", 80 | "unit": "m" 81 | }, 82 | { 83 | "alias": "syn", 84 | "align": "auto", 85 | "colorMode": null, 86 | "colors": [ 87 | "rgba(245, 54, 54, 0.9)", 88 | "rgba(237, 129, 40, 0.89)", 89 | "rgba(50, 172, 45, 0.97)" 90 | ], 91 | "dateFormat": "YYYY-MM-DD HH:mm:ss", 92 | "decimals": 2, 93 | "mappingType": 1, 94 | "pattern": "synced", 95 | "thresholds": [], 96 | "type": "string", 97 | "unit": "short", 98 | "valueMaps": [ 99 | { 100 | "text": "✓", 101 | "value": "true" 102 | }, 103 | { 104 | "text": "—", 105 | "value": "false" 106 | } 107 | ] 108 | } 109 | ], 110 | "targets": [ 111 | { 112 | "format": "table", 113 | "group": [], 114 | "metricColumn": "none", 115 | "rawQuery": true, 116 | "rawSql": "-- grafana ignore\nselect\n g.name || (case\n when g.pending_version = v.id then ' (P)'\n when g.current_version = v.id then ' (C)'\n else ' (U)' end) as subgraph_name,\n d.id as deployment,\n s.id as schema,\n replace(a.node_id, 'index_node_','') as nodeId,\n d.synced::text,\n n.name as network,\n (n.head_block_number - d.latest_ethereum_block_number) as blocks_behind_network,\n (case when n.name = 'mainnet' then ((n.head_block_number - d.latest_ethereum_block_number)/4)::text\n when n.name = 'rinkeby' then ((n.head_block_number - d.latest_ethereum_block_number)/4)::text\n when n.name = 'kovan' then ((n.head_block_number - d.latest_ethereum_block_number)/15)::text\n when n.name = 'poa-core' then ((n.head_block_number - d.latest_ethereum_block_number)/12)::text\n else 'ø' end) as lag\nfrom subgraphs.subgraph_deployment as d,\n subgraphs.subgraph_deployment_assignment as a,\n subgraphs.subgraph_version as v,\n subgraphs.subgraph as g,\n subgraphs.ethereum_contract_data_source as ds,\n subgraphs.subgraph_manifest m,\n ethereum_networks as n,\n deployment_schemas as s\nwhere g.id = v.subgraph\n and v.id in (g.pending_version, g.current_version)\n and a.id = d.id\n and m.id = d.manifest \n and ds.id = m.data_sources[1]\n and s.subgraph = d.id\n and v.deployment = d.id\n and not d.failed\n and n.name = ds.network\norder by blocks_behind_network desc, subgraph_name;", 117 | "refId": "A", 118 | "select": [ 119 | [ 120 | { 121 | "params": [ 122 | "value" 123 | ], 124 | "type": "column" 125 | } 126 | ] 127 | ], 128 | "timeColumn": "time", 129 | "where": [ 130 | { 131 | "name": "$__timeFilter", 132 | "params": [], 133 | "type": "macro" 134 | } 135 | ] 136 | } 137 | ], 138 | "timeFrom": null, 139 | "timeShift": null, 140 | "title": "Subgraph Block vs. Network Block", 141 | "transform": "table", 142 | "type": "table-old" 143 | }, 144 | { 145 | "columns": [], 146 | "datasource": "postgres", 147 | "fieldConfig": { 148 | "defaults": { 149 | "custom": {} 150 | }, 151 | "overrides": [] 152 | }, 153 | "fontSize": "100%", 154 | "gridPos": { 155 | "h": 8, 156 | "w": 24, 157 | "x": 0, 158 | "y": 10 159 | }, 160 | "id": 33, 161 | "pageSize": null, 162 | "showHeader": true, 163 | "sort": { 164 | "col": 0, 165 | "desc": true 166 | }, 167 | "styles": [ 168 | { 169 | "alias": "", 170 | "align": "auto", 171 | "dateFormat": "YYYY-MM-DD HH:mm:ss", 172 | "decimals": null, 173 | "pattern": "/vid|block_number/", 174 | "type": "number", 175 | "unit": "none" 176 | }, 177 | { 178 | "alias": "", 179 | "align": "right", 180 | "colorMode": null, 181 | "colors": [ 182 | "rgba(245, 54, 54, 0.9)", 183 | "rgba(237, 129, 40, 0.89)", 184 | "rgba(50, 172, 45, 0.97)" 185 | ], 186 | "decimals": 2, 187 | "pattern": "/.*/", 188 | "thresholds": [], 189 | "type": "number", 190 | "unit": "none" 191 | } 192 | ], 193 | "targets": [ 194 | { 195 | "format": "table", 196 | "group": [], 197 | "metricColumn": "none", 198 | "rawQuery": true, 199 | "rawSql": "SELECT\n vid, block_number, subgraph_id, message, handler\nFROM subgraphs.subgraph_error\nORDER BY vid DESC", 200 | "refId": "A", 201 | "select": [ 202 | [ 203 | { 204 | "params": [ 205 | "value" 206 | ], 207 | "type": "column" 208 | } 209 | ] 210 | ], 211 | "table": "subgraphs.subgraph_error", 212 | "timeColumn": "time", 213 | "where": [] 214 | } 215 | ], 216 | "timeFrom": null, 217 | "timeShift": null, 218 | "title": "Fatal errors", 219 | "transform": "table", 220 | "type": "table-old" 221 | }, 222 | { 223 | "columns": [], 224 | "datasource": "postgres", 225 | "fieldConfig": { 226 | "defaults": { 227 | "custom": {} 228 | }, 229 | "overrides": [] 230 | }, 231 | "fontSize": "100%", 232 | "gridPos": { 233 | "h": 7, 234 | "w": 24, 235 | "x": 0, 236 | "y": 18 237 | }, 238 | "id": 6, 239 | "links": [], 240 | "pageSize": null, 241 | "scroll": true, 242 | "showHeader": true, 243 | "sort": { 244 | "col": 1, 245 | "desc": true 246 | }, 247 | "styles": [ 248 | { 249 | "alias": "", 250 | "align": "auto", 251 | "colorMode": null, 252 | "colors": [ 253 | "rgba(245, 54, 54, 0.9)", 254 | "rgba(237, 129, 40, 0.89)", 255 | "rgba(50, 172, 45, 0.97)" 256 | ], 257 | "dateFormat": "YYYY-MM-DD HH:mm:ss", 258 | "decimals": 0, 259 | "mappingType": 1, 260 | "pattern": "num_assigned", 261 | "thresholds": [], 262 | "type": "number", 263 | "unit": "short" 264 | } 265 | ], 266 | "targets": [ 267 | { 268 | "format": "table", 269 | "group": [], 270 | "metricColumn": "none", 271 | "rawQuery": true, 272 | "rawSql": "select replace(node_id, 'index_node_', '') as nodeId,\n count(*) as num_assigned,\n array_agg(id) as deployments\n from subgraphs.subgraph_deployment_assignment\n group by node_id\n order by node_id asc;\n", 273 | "refId": "A", 274 | "select": [ 275 | [ 276 | { 277 | "params": [ 278 | "value" 279 | ], 280 | "type": "column" 281 | } 282 | ] 283 | ], 284 | "timeColumn": "time", 285 | "where": [ 286 | { 287 | "name": "$__timeFilter", 288 | "params": [], 289 | "type": "macro" 290 | } 291 | ] 292 | } 293 | ], 294 | "timeFrom": null, 295 | "timeShift": null, 296 | "title": "Subgraph Node Assignments", 297 | "transform": "table", 298 | "type": "table-old" 299 | }, 300 | { 301 | "cacheTimeout": null, 302 | "datasource": "postgres", 303 | "fieldConfig": { 304 | "defaults": { 305 | "custom": { 306 | "align": null 307 | }, 308 | "mappings": [], 309 | "nullValueMode": "connected", 310 | "thresholds": { 311 | "mode": "absolute", 312 | "steps": [ 313 | { 314 | "color": "green", 315 | "value": null 316 | }, 317 | { 318 | "color": "red", 319 | "value": 80 320 | } 321 | ] 322 | }, 323 | "unit": "none" 324 | }, 325 | "overrides": [ 326 | { 327 | "matcher": { 328 | "id": "byName", 329 | "options": "name" 330 | }, 331 | "properties": [ 332 | { 333 | "id": "displayName", 334 | "value": "Network name" 335 | } 336 | ] 337 | }, 338 | { 339 | "matcher": { 340 | "id": "byName", 341 | "options": "block_number" 342 | }, 343 | "properties": [ 344 | { 345 | "id": "displayName", 346 | "value": "Block number" 347 | } 348 | ] 349 | }, 350 | { 351 | "matcher": { 352 | "id": "byName", 353 | "options": "block_hash" 354 | }, 355 | "properties": [ 356 | { 357 | "id": "displayName", 358 | "value": "Block Hash" 359 | } 360 | ] 361 | }, 362 | { 363 | "matcher": { 364 | "id": "byName", 365 | "options": "Block number" 366 | }, 367 | "properties": [ 368 | { 369 | "id": "custom.width", 370 | "value": 101 371 | } 372 | ] 373 | }, 374 | { 375 | "matcher": { 376 | "id": "byName", 377 | "options": "Network name" 378 | }, 379 | "properties": [ 380 | { 381 | "id": "custom.width", 382 | "value": 116 383 | } 384 | ] 385 | }, 386 | { 387 | "matcher": { 388 | "id": "byName", 389 | "options": "Block Hash" 390 | }, 391 | "properties": [ 392 | { 393 | "id": "custom.width", 394 | "value": 396 395 | } 396 | ] 397 | } 398 | ] 399 | }, 400 | "gridPos": { 401 | "h": 8, 402 | "w": 12, 403 | "x": 0, 404 | "y": 25 405 | }, 406 | "id": 2, 407 | "interval": "10s", 408 | "links": [], 409 | "maxDataPoints": 100, 410 | "options": { 411 | "showHeader": true, 412 | "sortBy": [] 413 | }, 414 | "pluginVersion": "7.0.1", 415 | "targets": [ 416 | { 417 | "format": "table", 418 | "group": [], 419 | "metricColumn": "none", 420 | "rawQuery": true, 421 | "rawSql": "SELECT\n name,\n head_block_number AS \"block_number\",\n head_block_hash AS \"block_hash\"\nFROM ethereum_networks\nORDER BY name;", 422 | "refId": "A", 423 | "select": [ 424 | [ 425 | { 426 | "params": [ 427 | "value" 428 | ], 429 | "type": "column" 430 | } 431 | ] 432 | ], 433 | "table": "ethereum_networks", 434 | "timeColumn": "time", 435 | "where": [] 436 | } 437 | ], 438 | "timeFrom": null, 439 | "timeShift": null, 440 | "title": "Network Head Blocks", 441 | "type": "table" 442 | }, 443 | { 444 | "alert": { 445 | "alertRuleTags": {}, 446 | "conditions": [ 447 | { 448 | "evaluator": { 449 | "params": [ 450 | 0 451 | ], 452 | "type": "gt" 453 | }, 454 | "operator": { 455 | "type": "and" 456 | }, 457 | "query": { 458 | "params": [ 459 | "A", 460 | "3m", 461 | "now" 462 | ] 463 | }, 464 | "reducer": { 465 | "params": [], 466 | "type": "avg" 467 | }, 468 | "type": "query" 469 | } 470 | ], 471 | "executionErrorState": "alerting", 472 | "for": "5m", 473 | "frequency": "1m", 474 | "handler": 1, 475 | "name": "Deployed subgraphs that have not started alert", 476 | "noDataState": "no_data", 477 | "notifications": [ 478 | { 479 | "uid": "okc8ZyRZz" 480 | } 481 | ] 482 | }, 483 | "aliasColors": {}, 484 | "bars": false, 485 | "dashLength": 10, 486 | "dashes": false, 487 | "datasource": "postgres", 488 | "fieldConfig": { 489 | "defaults": { 490 | "custom": {} 491 | }, 492 | "overrides": [] 493 | }, 494 | "fill": 1, 495 | "fillGradient": 0, 496 | "gridPos": { 497 | "h": 8, 498 | "w": 12, 499 | "x": 12, 500 | "y": 25 501 | }, 502 | "hiddenSeries": false, 503 | "id": 29, 504 | "legend": { 505 | "avg": false, 506 | "current": false, 507 | "max": false, 508 | "min": false, 509 | "show": true, 510 | "total": false, 511 | "values": false 512 | }, 513 | "lines": true, 514 | "linewidth": 1, 515 | "nullPointMode": "null", 516 | "options": { 517 | "dataLinks": [] 518 | }, 519 | "percentage": false, 520 | "pointradius": 2, 521 | "points": false, 522 | "renderer": "flot", 523 | "seriesOverrides": [], 524 | "spaceLength": 10, 525 | "stack": false, 526 | "steppedLine": false, 527 | "targets": [ 528 | { 529 | "format": "time_series", 530 | "group": [], 531 | "metricColumn": "none", 532 | "rawQuery": true, 533 | "rawSql": "select now() as time, count(*) as num_subgraphs\n from (\n select d.id, d.latest_ethereum_block_number block\n from subgraphs.subgraph_deployment d,\n subgraphs.subgraph_deployment_assignment a\n where d.id = a.id\n and not d.failed) a\n where a.block < 100;\n", 534 | "refId": "A", 535 | "select": [ 536 | [ 537 | { 538 | "params": [ 539 | "value" 540 | ], 541 | "type": "column" 542 | } 543 | ] 544 | ], 545 | "timeColumn": "time", 546 | "where": [ 547 | { 548 | "name": "$__timeFilter", 549 | "params": [], 550 | "type": "macro" 551 | } 552 | ] 553 | } 554 | ], 555 | "thresholds": [ 556 | { 557 | "colorMode": "critical", 558 | "fill": true, 559 | "line": true, 560 | "op": "gt", 561 | "value": 0 562 | } 563 | ], 564 | "timeFrom": null, 565 | "timeRegions": [], 566 | "timeShift": null, 567 | "title": "Deployed subgraphs that have not started", 568 | "tooltip": { 569 | "shared": true, 570 | "sort": 0, 571 | "value_type": "individual" 572 | }, 573 | "type": "graph", 574 | "xaxis": { 575 | "buckets": null, 576 | "mode": "time", 577 | "name": null, 578 | "show": true, 579 | "values": [] 580 | }, 581 | "yaxes": [ 582 | { 583 | "format": "short", 584 | "label": null, 585 | "logBase": 1, 586 | "max": null, 587 | "min": null, 588 | "show": true 589 | }, 590 | { 591 | "format": "short", 592 | "label": null, 593 | "logBase": 1, 594 | "max": null, 595 | "min": null, 596 | "show": true 597 | } 598 | ], 599 | "yaxis": { 600 | "align": false, 601 | "alignLevel": null 602 | } 603 | } 604 | ], 605 | "refresh": "", 606 | "schemaVersion": 25, 607 | "style": "dark", 608 | "tags": [], 609 | "templating": { 610 | "list": [ 611 | { 612 | "datasource": "Elasticsearch", 613 | "filters": [], 614 | "hide": 0, 615 | "label": "", 616 | "name": "Filters", 617 | "skipUrlSync": false, 618 | "type": "adhoc" 619 | } 620 | ] 621 | }, 622 | "time": { 623 | "from": "now-6h", 624 | "to": "now" 625 | }, 626 | "timepicker": { 627 | "refresh_intervals": [ 628 | "10s", 629 | "30s", 630 | "1m", 631 | "5m", 632 | "15m", 633 | "30m", 634 | "1h", 635 | "2h", 636 | "1d" 637 | ], 638 | "time_options": [ 639 | "5m", 640 | "15m", 641 | "1h", 642 | "6h", 643 | "12h", 644 | "24h", 645 | "2d", 646 | "7d", 647 | "30d" 648 | ] 649 | }, 650 | "timezone": "", 651 | "title": "Indexing Status", 652 | "uid": "7rcuDImZk", 653 | "version": 4 654 | } 655 | -------------------------------------------------------------------------------- /graph-node/insane/grafana/provisioning/dashboards/postgres.json: -------------------------------------------------------------------------------- 1 | { 2 | "annotations": { 3 | "list": [ 4 | { 5 | "builtIn": 1, 6 | "datasource": "-- Grafana --", 7 | "enable": true, 8 | "hide": true, 9 | "iconColor": "rgba(0, 211, 255, 1)", 10 | "name": "Annotations & Alerts", 11 | "type": "dashboard" 12 | } 13 | ] 14 | }, 15 | "editable": true, 16 | "gnetId": null, 17 | "graphTooltip": 0, 18 | "id": 3, 19 | "links": [], 20 | "panels": [ 21 | { 22 | "columns": [], 23 | "datasource": "postgres", 24 | "fontSize": "100%", 25 | "gridPos": { 26 | "h": 8, 27 | "w": 24, 28 | "x": 0, 29 | "y": 0 30 | }, 31 | "id": 4, 32 | "links": [], 33 | "pageSize": null, 34 | "scroll": true, 35 | "showHeader": true, 36 | "sort": { 37 | "col": 1, 38 | "desc": true 39 | }, 40 | "styles": [ 41 | { 42 | "alias": "Age", 43 | "align": "auto", 44 | "dateFormat": "YYYY-MM-DD HH:mm:ss", 45 | "pattern": "age", 46 | "type": "number", 47 | "unit": "s" 48 | }, 49 | { 50 | "alias": "", 51 | "align": "auto", 52 | "colorMode": null, 53 | "colors": [ 54 | "rgba(245, 54, 54, 0.9)", 55 | "rgba(237, 129, 40, 0.89)", 56 | "rgba(50, 172, 45, 0.97)" 57 | ], 58 | "decimals": 2, 59 | "pattern": "/.*/", 60 | "thresholds": [], 61 | "type": "number", 62 | "unit": "short" 63 | } 64 | ], 65 | "targets": [ 66 | { 67 | "format": "table", 68 | "group": [], 69 | "metricColumn": "none", 70 | "rawQuery": true, 71 | "rawSql": "-- grafana ignore\nselect application_name,\n extract(epoch from age(now(), xact_start)) as age,\n query from pg_stat_activity\n where query not like '%grafana ignore%'\n and state='active'\norder by query_start desc", 72 | "refId": "A", 73 | "select": [ 74 | [ 75 | { 76 | "params": [ 77 | "value" 78 | ], 79 | "type": "column" 80 | } 81 | ] 82 | ], 83 | "timeColumn": "time", 84 | "where": [ 85 | { 86 | "name": "$__timeFilter", 87 | "params": [], 88 | "type": "macro" 89 | } 90 | ] 91 | } 92 | ], 93 | "timeFrom": null, 94 | "timeShift": null, 95 | "title": "Active Queries", 96 | "transform": "table", 97 | "type": "table" 98 | }, 99 | { 100 | "columns": [], 101 | "datasource": "postgres", 102 | "fontSize": "100%", 103 | "gridPos": { 104 | "h": 5, 105 | "w": 13, 106 | "x": 0, 107 | "y": 8 108 | }, 109 | "id": 9, 110 | "links": [], 111 | "pageSize": null, 112 | "pluginVersion": "6.2.1", 113 | "scroll": true, 114 | "showHeader": true, 115 | "sort": { 116 | "col": 1, 117 | "desc": false 118 | }, 119 | "styles": [ 120 | { 121 | "alias": "", 122 | "align": "auto", 123 | "dateFormat": "YYYY-MM-DD", 124 | "pattern": "last_autovacuum", 125 | "type": "date" 126 | }, 127 | { 128 | "alias": "tx left", 129 | "align": "auto", 130 | "colorMode": "row", 131 | "colors": [ 132 | "rgba(245, 54, 54, 0.9)", 133 | "rgba(237, 129, 40, 0.89)", 134 | "rgba(50, 172, 45, 0.97)" 135 | ], 136 | "decimals": null, 137 | "pattern": "tx_before_wraparound_vacuum", 138 | "thresholds": [ 139 | "20000000", 140 | "40000000" 141 | ], 142 | "type": "number", 143 | "unit": "short" 144 | }, 145 | { 146 | "alias": "", 147 | "align": "auto", 148 | "colorMode": null, 149 | "colors": [ 150 | "rgba(245, 54, 54, 0.9)", 151 | "rgba(237, 129, 40, 0.89)", 152 | "rgba(50, 172, 45, 0.97)" 153 | ], 154 | "dateFormat": "YYYY-MM-DD HH:mm:ss", 155 | "decimals": 2, 156 | "mappingType": 1, 157 | "pattern": "xid_age", 158 | "thresholds": [], 159 | "type": "number", 160 | "unit": "short" 161 | } 162 | ], 163 | "targets": [ 164 | { 165 | "format": "table", 166 | "group": [], 167 | "metricColumn": "none", 168 | "rawQuery": true, 169 | "rawSql": "-- grafana ignore\nSELECT\n relname as table,\n least((SELECT setting::int FROM pg_settings WHERE name = 'autovacuum_freeze_max_age') - age(relfrozenxid), \n (SELECT setting::int FROM pg_settings WHERE name = 'autovacuum_multixact_freeze_max_age') - mxid_age(relminmxid))\n tx_before_wraparound_vacuum,\n pg_stat_get_last_autovacuum_time(c.oid) AS last_autovacuum,\n age(relfrozenxid) AS xid_age,\n mxid_age(relminmxid) AS mxid_age\nFROM\n pg_class c\nWHERE\n c.relname in ('ethereum_blocks', 'eth_call_cache','subgraph_deployment')\n and c.relfrozenxid != 0\n", 170 | "refId": "A", 171 | "select": [ 172 | [ 173 | { 174 | "params": [ 175 | "value" 176 | ], 177 | "type": "column" 178 | } 179 | ] 180 | ], 181 | "timeColumn": "time", 182 | "where": [ 183 | { 184 | "name": "$__timeFilter", 185 | "params": [], 186 | "type": "macro" 187 | } 188 | ] 189 | } 190 | ], 191 | "timeFrom": null, 192 | "timeShift": null, 193 | "title": "Transactions until wraparound vacuum", 194 | "transform": "table", 195 | "type": "table" 196 | }, 197 | { 198 | "columns": [], 199 | "datasource": "postgres", 200 | "fontSize": "100%", 201 | "gridPos": { 202 | "h": 3, 203 | "w": 11, 204 | "x": 13, 205 | "y": 8 206 | }, 207 | "id": 11, 208 | "pageSize": null, 209 | "showHeader": true, 210 | "sort": { 211 | "col": 0, 212 | "desc": true 213 | }, 214 | "styles": [ 215 | { 216 | "alias": "Tables needing vacuum", 217 | "align": "auto", 218 | "colorMode": null, 219 | "colors": [ 220 | "rgba(245, 54, 54, 0.9)", 221 | "rgba(237, 129, 40, 0.89)", 222 | "rgba(50, 172, 45, 0.97)" 223 | ], 224 | "decimals": 0, 225 | "pattern": "tables_needing_vacuum", 226 | "thresholds": [], 227 | "type": "number", 228 | "unit": "none" 229 | }, 230 | { 231 | "alias": "Txns past", 232 | "align": "auto", 233 | "colorMode": null, 234 | "colors": [ 235 | "rgba(245, 54, 54, 0.9)", 236 | "rgba(237, 129, 40, 0.89)", 237 | "rgba(50, 172, 45, 0.97)" 238 | ], 239 | "dateFormat": "YYYY-MM-DD HH:mm:ss", 240 | "decimals": 0, 241 | "mappingType": 1, 242 | "pattern": "txns_past", 243 | "thresholds": [], 244 | "type": "number", 245 | "unit": "locale" 246 | }, 247 | { 248 | "alias": "Last autovacuum", 249 | "align": "auto", 250 | "colorMode": null, 251 | "colors": [ 252 | "rgba(245, 54, 54, 0.9)", 253 | "rgba(237, 129, 40, 0.89)", 254 | "rgba(50, 172, 45, 0.97)" 255 | ], 256 | "dateFormat": "YYYY-MM-DD HH:mm:ss", 257 | "decimals": 2, 258 | "mappingType": 1, 259 | "pattern": "last_autovacuum", 260 | "thresholds": [], 261 | "type": "date", 262 | "unit": "short" 263 | } 264 | ], 265 | "targets": [ 266 | { 267 | "format": "table", 268 | "group": [], 269 | "metricColumn": "none", 270 | "rawQuery": true, 271 | "rawSql": "-- grafana ignore\nselect count(*) as tables_needing_vacuum,\n -min(tx_before_wraparound_vacuum) as txns_past,\n min(last_autovacuum) as last_autovacuum\n from (\n select oid::regclass::text AS table,\n least(\n (select setting::int\n from pg_settings\n where name = 'autovacuum_freeze_max_age') - age(relfrozenxid),\n (select setting::int\n from pg_settings\n where name = 'autovacuum_multixact_freeze_max_age')\n - mxid_age(relminmxid)) as tx_before_wraparound_vacuum,\n pg_stat_get_last_autovacuum_time(oid) AS last_autovacuum,\n age(relfrozenxid) AS xid_age,\n mxid_age(relminmxid) AS mxid_age\n from pg_class\n where relfrozenxid != 0\n and oid > 16384\n and relkind='r') a where a.tx_before_wraparound_vacuum < 0;\n", 272 | "refId": "A", 273 | "select": [ 274 | [ 275 | { 276 | "params": [ 277 | "value" 278 | ], 279 | "type": "column" 280 | } 281 | ] 282 | ], 283 | "timeColumn": "time", 284 | "where": [ 285 | { 286 | "name": "$__timeFilter", 287 | "params": [], 288 | "type": "macro" 289 | } 290 | ] 291 | } 292 | ], 293 | "timeFrom": null, 294 | "timeShift": null, 295 | "title": "Autovacuum pressure", 296 | "transform": "table", 297 | "type": "table" 298 | }, 299 | { 300 | "columns": [], 301 | "datasource": "postgres", 302 | "fontSize": "100%", 303 | "gridPos": { 304 | "h": 9, 305 | "w": 24, 306 | "x": 0, 307 | "y": 13 308 | }, 309 | "id": 7, 310 | "links": [], 311 | "pageSize": null, 312 | "scroll": true, 313 | "showHeader": true, 314 | "sort": { 315 | "col": 4, 316 | "desc": false 317 | }, 318 | "styles": [ 319 | { 320 | "alias": "Granted", 321 | "align": "auto", 322 | "dateFormat": "YYYY-MM-DD HH:mm:ss", 323 | "decimals": 0, 324 | "mappingType": 1, 325 | "pattern": "granted", 326 | "preserveFormat": false, 327 | "sanitize": false, 328 | "thresholds": [ 329 | "" 330 | ], 331 | "type": "string", 332 | "unit": "none", 333 | "valueMaps": [ 334 | { 335 | "text": "✓", 336 | "value": "true" 337 | }, 338 | { 339 | "text": "—", 340 | "value": "false" 341 | } 342 | ] 343 | }, 344 | { 345 | "alias": "Age", 346 | "align": "auto", 347 | "colorMode": null, 348 | "colors": [ 349 | "rgba(245, 54, 54, 0.9)", 350 | "rgba(237, 129, 40, 0.89)", 351 | "rgba(50, 172, 45, 0.97)" 352 | ], 353 | "decimals": 2, 354 | "pattern": "age", 355 | "thresholds": [], 356 | "type": "number", 357 | "unit": "s" 358 | } 359 | ], 360 | "targets": [ 361 | { 362 | "format": "table", 363 | "group": [], 364 | "metricColumn": "none", 365 | "rawQuery": true, 366 | "rawSql": "-- grafana ignore\nSELECT a.application_name,\n coalesce(extract(epoch from age(now(), a.xact_start)), 0) as age,\n l.relation::regclass,\n l.mode,\n l.GRANTED::varchar,\n l.locktype \"Target\"\nFROM pg_stat_activity a\nJOIN pg_locks l ON l.pid = a.pid\nwhere pg_backend_pid() != a.pid\norder by granted asc, age desc;", 367 | "refId": "A", 368 | "select": [ 369 | [ 370 | { 371 | "params": [ 372 | "value" 373 | ], 374 | "type": "column" 375 | } 376 | ] 377 | ], 378 | "timeColumn": "time", 379 | "where": [ 380 | { 381 | "name": "$__timeFilter", 382 | "params": [], 383 | "type": "macro" 384 | } 385 | ] 386 | } 387 | ], 388 | "timeFrom": null, 389 | "timeShift": null, 390 | "title": "Active locks", 391 | "transform": "table", 392 | "type": "table" 393 | }, 394 | { 395 | "columns": [], 396 | "datasource": "postgres", 397 | "fontSize": "100%", 398 | "gridPos": { 399 | "h": 8, 400 | "w": 24, 401 | "x": 0, 402 | "y": 22 403 | }, 404 | "id": 5, 405 | "links": [], 406 | "pageSize": null, 407 | "scroll": true, 408 | "showHeader": true, 409 | "sort": { 410 | "col": 4, 411 | "desc": false 412 | }, 413 | "styles": [ 414 | { 415 | "alias": "Age", 416 | "align": "auto", 417 | "dateFormat": "YYYY-MM-DD HH:mm:ss", 418 | "pattern": "age", 419 | "type": "number", 420 | "unit": "s" 421 | }, 422 | { 423 | "alias": "", 424 | "align": "auto", 425 | "colorMode": null, 426 | "colors": [ 427 | "rgba(245, 54, 54, 0.9)", 428 | "rgba(237, 129, 40, 0.89)", 429 | "rgba(50, 172, 45, 0.97)" 430 | ], 431 | "decimals": 2, 432 | "pattern": "/.*/", 433 | "thresholds": [], 434 | "type": "number", 435 | "unit": "short" 436 | } 437 | ], 438 | "targets": [ 439 | { 440 | "format": "table", 441 | "group": [], 442 | "metricColumn": "none", 443 | "rawQuery": true, 444 | "rawSql": "-- grafana ignore\nselect client_addr,\n application_name,\n usename,\n state,\n extract(epoch from age(now(), xact_start)) as age\n from pg_stat_activity\n where query not like '%grafana ignore%'\n and state like '%idle in transaction%'\norder by query_start desc", 445 | "refId": "A", 446 | "select": [ 447 | [ 448 | { 449 | "params": [ 450 | "value" 451 | ], 452 | "type": "column" 453 | } 454 | ] 455 | ], 456 | "timeColumn": "time", 457 | "where": [ 458 | { 459 | "name": "$__timeFilter", 460 | "params": [], 461 | "type": "macro" 462 | } 463 | ] 464 | } 465 | ], 466 | "timeFrom": null, 467 | "timeShift": null, 468 | "title": "Idle transactions", 469 | "transform": "table", 470 | "type": "table" 471 | } 472 | ], 473 | "schemaVersion": 22, 474 | "style": "dark", 475 | "tags": [], 476 | "templating": { 477 | "list": [] 478 | }, 479 | "time": { 480 | "from": "now-6h", 481 | "to": "now" 482 | }, 483 | "timepicker": { 484 | "refresh_intervals": [ 485 | "5s", 486 | "10s", 487 | "30s", 488 | "1m", 489 | "5m", 490 | "15m", 491 | "30m", 492 | "1h", 493 | "2h", 494 | "1d" 495 | ], 496 | "time_options": [ 497 | "5m", 498 | "15m", 499 | "1h", 500 | "6h", 501 | "12h", 502 | "24h", 503 | "2d", 504 | "7d", 505 | "30d" 506 | ] 507 | }, 508 | "timezone": "", 509 | "title": "Postgres Statistics", 510 | "uid": "Mo6FxoiWz", 511 | "variables": { 512 | "list": [] 513 | }, 514 | "version": 4 515 | } 516 | -------------------------------------------------------------------------------- /graph-node/insane/grafana/provisioning/datasources/postgres.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: 1 2 | datasources: 3 | - access: proxy 4 | editable: true 5 | name: postgres 6 | orgId: 1 7 | type: postgres 8 | url: $postgres_host 9 | user: $postgres_user 10 | database: $postgres_db 11 | secureJsonData: 12 | password: $postgres_pass 13 | jsonData: 14 | sslmode: "disable" 15 | postgresVersion: 906 16 | -------------------------------------------------------------------------------- /graph-node/insane/grafana/provisioning/datasources/prometheus.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: 1 2 | datasources: 3 | - access: proxy 4 | editable: true 5 | name: prometheus 6 | orgId: 1 7 | type: prometheus 8 | url: http://prometheus:9090 9 | version: 1 10 | -------------------------------------------------------------------------------- /graph-node/insane/nginx/nginx.conf: -------------------------------------------------------------------------------- 1 | events {} 2 | http { 3 | # See http://nginx.org/en/docs/http/websocket.html 4 | map $http_upgrade $connection_upgrade { 5 | default upgrade; 6 | '' close; 7 | } 8 | 9 | server { 10 | listen 80; 11 | listen [::]:80; 12 | 13 | # Uncomment to enable SSL 14 | #listen 443 ssl; 15 | #listen [::]:443 ssl; 16 | #ssl_certificate /etc/letsencrypt/live//fullchain.pem; 17 | #ssl_certificate_key /etc/letsencrypt/live//privkey.pem; 18 | 19 | location /.well-known { 20 | root /etc/letsencrypt/webroot; 21 | } 22 | 23 | location /prometheus/ { 24 | rewrite ^/prometheus/(.*)$ /$1 break; 25 | 26 | # Proxy configuration. 27 | proxy_pass http://prometheus:9090; 28 | proxy_http_version 1.1; 29 | proxy_set_header Connection $connection_upgrade; 30 | proxy_set_header Host $host; 31 | proxy_set_header Upgrade $http_upgrade; 32 | proxy_cache_bypass $http_upgrade; 33 | 34 | # Gateway timeout. 35 | proxy_read_timeout 30s; 36 | proxy_send_timeout 30s; 37 | 38 | } 39 | 40 | location /grafana/ { 41 | # rewrite ^/grafana/(.*)$ /$1 break; # NOT NEEDED, grafana configured as a subdomain 42 | 43 | # Proxy configuration. 44 | proxy_pass http://grafana:3000; 45 | proxy_http_version 1.1; 46 | proxy_set_header Connection $connection_upgrade; 47 | proxy_set_header Host $host; 48 | proxy_set_header Upgrade $http_upgrade; 49 | proxy_cache_bypass $http_upgrade; 50 | 51 | # Gateway timeout. 52 | proxy_read_timeout 30s; 53 | proxy_send_timeout 30s; 54 | 55 | } 56 | 57 | location /monitoring/ { 58 | rewrite ^/monitoring/(.*)$ /$1 break; 59 | 60 | # Proxy configuration. 61 | proxy_pass http://graphnode-query-1:8030; 62 | proxy_http_version 1.1; 63 | proxy_set_header Connection $connection_upgrade; 64 | proxy_set_header Host $host; 65 | proxy_set_header Upgrade $http_upgrade; 66 | proxy_cache_bypass $http_upgrade; 67 | 68 | # Gateway timeout. 69 | proxy_read_timeout 30s; 70 | proxy_send_timeout 30s; 71 | } 72 | 73 | location /metrics/ { 74 | rewrite ^/metrics/(.*)$ /$1 break; 75 | 76 | # Proxy configuration. 77 | proxy_pass http://graphnode-indexer-1:8040; 78 | proxy_http_version 1.1; 79 | proxy_set_header Connection $connection_upgrade; 80 | proxy_set_header Host $host; 81 | proxy_set_header Upgrade $http_upgrade; 82 | proxy_cache_bypass $http_upgrade; 83 | 84 | # Gateway timeout. 85 | proxy_read_timeout 30s; 86 | proxy_send_timeout 30s; 87 | } 88 | 89 | 90 | location / { 91 | # Move WebSocket and HTTP requests into /ws/ and /http/ prefixes; 92 | # this allows us to forward both types of requests to different 93 | # query node ports 94 | if ( $connection_upgrade = "upgrade" ) { rewrite ^(.*)$ /ws/$1; } 95 | if ( $connection_upgrade != "upgrade" ) { rewrite ^(.*)$ /http/$1; } 96 | 97 | location /http { 98 | # Remove the /http/ again 99 | rewrite ^/http/(.*)$ $1 break; 100 | 101 | # Proxy configuration. 102 | proxy_pass http://graphnode-query-1:8000; 103 | proxy_http_version 1.1; 104 | proxy_set_header Connection $connection_upgrade; 105 | proxy_set_header Host $host; 106 | proxy_set_header Upgrade $http_upgrade; 107 | proxy_cache_bypass $http_upgrade; 108 | 109 | # Gateway timeout. 110 | proxy_read_timeout 30s; 111 | proxy_send_timeout 30s; 112 | } 113 | 114 | location /ws { 115 | # Remove the /ws/ again 116 | rewrite ^/ws/(.*)$ $1 break; 117 | 118 | # Proxy configuration. 119 | proxy_pass http://graphnode-query-1:8001; 120 | proxy_http_version 1.1; 121 | proxy_set_header Connection $connection_upgrade; 122 | proxy_set_header Host $host; 123 | proxy_set_header Upgrade $http_upgrade; 124 | proxy_cache_bypass $http_upgrade; 125 | 126 | # Gateway timeout. 127 | proxy_read_timeout 3600s; 128 | proxy_send_timeout 3600s; 129 | } 130 | } 131 | } 132 | } 133 | 134 | -------------------------------------------------------------------------------- /graph-node/insane/prometheus/prometheus.yml: -------------------------------------------------------------------------------- 1 | global: 2 | scrape_interval: 15s 3 | evaluation_interval: 15s 4 | 5 | rule_files: 6 | 7 | alerting: 8 | 9 | scrape_configs: 10 | - job_name: 'graphnode-indexer-1-metrics' 11 | metrics_path: /metrics 12 | static_configs: 13 | - targets: ['graphnode-indexer-1:8040'] 14 | 15 | -------------------------------------------------------------------------------- /indexer-agent/README.md: -------------------------------------------------------------------------------- 1 | ## Install Indexer Service & Agent 2 | 3 | Set npm to use The Graph's private package registry. I recommend not using yarn, as it's caused me authentication issues in the past with [Verdaccio](https://verdaccio.org/). 4 | 5 | ```bash 6 | # Don't use yarn 7 | npm set registry https://testnet.thegraph.com/npm-registry 8 | npm login 9 | ``` 10 | 11 | Install the CLI tools and revert your registry back to the original. 12 | 13 | ```bash 14 | # Install the latest 15 | npm install -g @graphprotocol/indexer-cli 16 | npm install -g @graphprotocol/indexer-agent 17 | npm install -g @graphprotocol/indexer-service # optional 18 | 19 | npm set registry https://registry.npmjs.com/ 20 | ``` 21 | 22 | - Postgres for Agent 23 | - Connection to Rinkeby node: Contract interactions-only, no syncing 24 | 25 | ## Set up Agent configs 26 | 27 | TODO 28 | 29 | rinkeby node can be frree Infura node! 30 | 31 | public indexer url http://localhost:7600 32 | 33 | > :warning: this will be made public on-chain! 34 | 35 | geo-coordinates useful for customer to decide which indxer to use, based on location 36 | 37 | ## Running the agent 38 | 39 | Pass Rinkeby account mnemonic 40 | Netwoirk subgraph endpoint 41 | 42 | ## Run the Indexer serive 43 | 44 | ```bash 45 | ./run-indexer-service.sh 46 | ``` 47 | 48 | Provides metrics you can scrape with Prometheus, 49 | Queries can be sent to port `7600`, where to 50 | 51 | You should receive Status code `402: Payment required`. 52 | 53 | ### Set up payments to test a real the query 54 | 55 | ## Run the Agent 56 | 57 | -------------------------------------------------------------------------------- /indexer-install/.template.env: -------------------------------------------------------------------------------- 1 | MNEMONIC='apple ...' 2 | INDEXER_ADDRESS=0x... 3 | 4 | IPFS_ENDPOINT=https://api.thegraph.com/ipfs/ 5 | 6 | GRAPH_QUERY_SERVER_DOMAIN=graph.mysite.com 7 | ETH_ENDPOINT_GRAPH_NODE_MAINNET=https:// 8 | 9 | ETH_ENDPOINT_INDEXER=https://rinkeby.infura.io/v3/.... 10 | 11 | GEO_COORDINATES='0.12345 0.12345' 12 | 13 | DB_USER=graph-node 14 | DB_PASSWORD= 15 | DB_NAME_GRAPH_NODE=graph-node 16 | DB_NAME_INDEXER=indexer 17 | PUBLIC_INDEXER_SERVER_DOMAIN=indexer.mysite.com 18 | PRIVATE_INDEXER_AGENT_DOMAIN=agent.mysite.com 19 | INDEXER_SERVICE_INDEXER_ADDRESS= 20 | 21 | GF_SERVER_ROOT_URL=https://grafana.mysite.com 22 | GF_SERVER_DOMAIN=grafana.mysite.com 23 | 24 | PROMETHEUS_SERVER_DOMAIN=prometheus.mysite.com 25 | PROMETHEUS_ADMIN_USER= 26 | PROMETHEUS_HASHED_PASSWORD= 27 | -------------------------------------------------------------------------------- /indexer-install/README.md: -------------------------------------------------------------------------------- 1 |

Phase 1 Mission 1 - Setting up the Indexer Stack

2 | 3 | This tutorial is also available via my blog: https://patrickgallagher.dev/blog/2020/09/06/graph-indexer-phase1 4 | 5 | Hello Indexers, congrats on making it to Phase 1! Now its time to add some query handling & monetization tooling with two new additions- the indexer agent and indexer service. 6 | 7 | During Phase 0 it seemed like the hardware requirements were over-stated. Rather than continue to overpay for one large under-utilized server, I decided to downsize significantly ($20/mo compared to $160/mo), and set up Docker Swarm so I'm ready to scale up as needed. I compared performance before/after this transition, if you're interested in [seeing the results](../performance). Just be mindful that the phase 0 test harness is not good at approximating actual usage, since queries are duplicated, and thus the responses are cached. 8 | 9 | 🛸🛸🛸🛸 Lets migrate our infra to Docker Swarm! 🛸🛸🛸🛸 10 | 11 | > Salty about having to learn something new? See [non-swarm](./non-swarm) for the vanilla docker-compose provided by @amxx (thanks!). I haven't tested it myself, so you're on your own with that one. 12 | 13 | # Set up Swarm + Traefik 14 | 15 | ![](./screenshots/traefik.png) 16 | 17 | If you're like me, and never used Swarm before, then start with this tutorial https://dockerswarm.rocks/ which was adapted here for convenience. If you'd rather dive in, you can try following theses commands. 18 | 19 | Once you're done setting up swarm, jump to the next section. 20 | 21 | ```bash 22 | # Connect as root 23 | ssh root@ 24 | 25 | export USE_HOSTNAME=traefik.mysite.com 26 | echo $USE_HOSTNAME > /etc/hostname 27 | hostname -F /etc/hostname 28 | ``` 29 | 30 | Start a new docker swarm 31 | 32 | ```bash 33 | docker swarm init 34 | ``` 35 | 36 | If you already have additional machines ready, you can add them to the swarm now. Otherwise skip this step. 37 | it. 38 | 39 | ```bash 40 | docker swarm join-token worker 41 | ``` 42 | 43 | Check the cluster is running 44 | 45 | ```bash 46 | docker node ls 47 | ``` 48 | 49 | ## Add Traefik 50 | 51 | Create an overlay network 52 | 53 | ```bash 54 | docker network create --driver=overlay traefik-public 55 | ``` 56 | 57 | > The overlay network sits on top of (overlays) the host-specific networks and allows containers connected to it to communicate securely when encryption is enabled 58 | 59 | Now set environment variables 60 | 61 | ```bash 62 | export NODE_ID=$(docker info -f '{{.Swarm.NodeID}}') 63 | docker node update --label-add traefik-public.traefik-public-certificates=true $NODE_ID 64 | export EMAIL=patrick@somemailserver.com 65 | export DOMAIN=traefik.mysite.com 66 | export USERNAME=admin 67 | export PASSWORD=changethis 68 | # Use openssl to generate the password hash 69 | export HASHED_PASSWORD=$(openssl passwd -apr1 $PASSWORD) 70 | echo $HASHED_PASSWORD 71 | # > $apr1$89eqM5Ro$CxaFELthUKV21DpI3UTQO. 72 | ``` 73 | 74 | Download a template docker-compose file for Traefik 75 | 76 | ```bash 77 | curl -L dockerswarm.rocks/traefik.yml -o traefik.yml 78 | # Check the contents traefik.yml 79 | cat traefik.yml 80 | # Deploy 81 | docker stack deploy -c traefik.yml traefik 82 | # Check it 83 | docker stack ps traefik 84 | docker service logs traefik_traefik 85 | ``` 86 | 87 | ### Using Traefik 88 | 89 | Log in to traefik.mysite.com to see the dashboard 90 | 91 | > If you need to read the client IP in your applications/stacks using the X-Forwarded-For or X-Real-IP headers provided by Traefik, you need to make Traefik listen directly, not through Docker Swarm mode. See "Getting the client IP" 92 | 93 | ## Add Swarmpit for monitoring 94 | 95 | I highly recommend following these instructions to set up a Swarmpit dashboard https://dockerswarm.rocks/swarmpit/ 96 | 97 | ![](./screenshots/swarmpit.png) 98 | 99 | # Deploy the Indexer Stack 100 | 101 | First pull down the repo 102 | 103 | ```bash 104 | git clone https://github.com/pi0neerpat/indexer-docker-compose 105 | cd indexer-docker-compose/indexer-install 106 | ``` 107 | 108 | To allow Traefik to "see" our docker containers, we need to update our docker-compose file with the labels. Docker labels don’t do anything by themselves, but Traefik reads these so it knows how to treat containers. Note that this need to be service-level labels rather than container-level (i.e. under the deploy tag). 109 | 110 | 💪 I've already done the heavy lifting for you and created `graph.yml` in this folder. 111 | 112 | ```yml 113 | networks: 114 | # Network between all containers in the stack 115 | - default 116 | # Networks used to communicate with Traefik 117 | - traefik-public 118 | deploy: 119 | labels: 120 | - traefik.enable=true 121 | - traefik.docker.network=traefik-public 122 | - traefik.constraint-label=traefik-public 123 | # A router called "graph-query-http" should accept requests from eg. graph.mysite.com 124 | - "traefik.http.routers.graph-query-http.rule=Host(`${GRAPH_QUERY_SERVER_DOMAIN}`)" 125 | # The router should accept requests via http / port 80 126 | - traefik.http.routers.graph-query-http.entrypoints=http 127 | # The router should use the middleware "https-redirect" 128 | - traefik.http.routers.graph-query-http.middlewares=https-redirect 129 | # A router called "graph-query-https" should accept requests from eg. graph.mysite.com 130 | - "traefik.http.routers.graph-query-https.rule=Host(`${GRAPH_QUERY_SERVER_DOMAIN}`)" 131 | # The router should accept requests via https / port 443 132 | - traefik.http.routers.graph-query-https.entrypoints=https 133 | # SSL cert stuff 134 | - traefik.http.routers.graph-query-https.tls=true 135 | - traefik.http.routers.graph-query-https.tls.certresolver=le 136 | # A service called "graph-query" should be created using port 8000 of this docker container 137 | - traefik.http.services.graph-query.loadbalancer.server.port=8000 138 | ``` 139 | 140 | If you're continuing from the [phase 0 tutorial](https://github.com/pi0neerpat/indexer-docker-compose#phase-0---deploy-graph-node), you will need to update the location of the postgres volume in `graph.yml`. 141 | 142 | ```yml 143 | postgres: 144 | volumes: 145 | - /data/postgres:/var/lib/postgresql/data 146 | # Change to 147 | - ~/subgraph-data/postgres:/var/lib/postgresql/data 148 | ``` 149 | 150 | Now specify the variables for your set up by copying `.template.env` to `.env` and editing accordingly. 151 | 152 | Next, create a label in this node, so that the Postgres database is always deployed to the same node and uses the existing volume: 153 | 154 | ```bash 155 | export NODE_ID=$(docker info -f '{{.Swarm.NodeID}}') 156 | docker node update --label-add postgres.postgres-data=true $NODE_ID 157 | ``` 158 | 159 | Create some necessary directories. See [docker docs](https://docs.docker.com/engine/install/linux-postinstall/) for more help with managing permissions. 160 | 161 | ```bash 162 | mkdir -p /data/grafana /data/prometheus /data/postgres /data/rinkeby/chaindata 163 | sudo chown "$USER":"$USER" /data/ -R 164 | ``` 165 | 166 | Log in to docker using your credentials and pull the private images. 167 | 168 | ```bash 169 | docker login 170 | docker pull graphprotocol/indexer-service:latest 171 | docker pull graphprotocol/indexer-agent:latest 172 | ``` 173 | 174 | Load the `.env` file (this command is a bit wonky, so feel free to suggest a better approach), and then deploy the indexer stack. 175 | 176 | ```bash 177 | source .env 178 | # OR 179 | export $(grep -v '^#' .env | xargs -0) 180 | 181 | docker stack deploy -c graph.yml graph 182 | ``` 183 | 184 | If you ever need to change `graph.yml` or `.env` configs, simply run these commands again. There is no need to stop the stack or services from running. 185 | 186 | Now you should be able to access your grafana at the url you provided for `GF_SERVER_DOMAIN`. The Dashboards should already be created for you. 187 | 188 | If you need to create fresh subgraphs, use the commands from the basic tutorial [here](../graph-node/basic). 189 | 190 | If you're migrating from an existing set up, your subgraphs might not being syncing right away. This is likely due to a change in `node_id` the subgraphs are assigned to. Change the `node_id` using `subgraph_reassign`. 191 | 192 | ```bash 193 | http post :8020 \ 194 | jsonrpc="2.0" id="1" \ 195 | method="subgraph_reassign" \ 196 | params:='{"name": "molochventures/moloch", "ipfs_hash": \ 197 | "QmTXzATwNfgGVukV1fX2T6xw9f6LAYRVWpsdXyRWzUR2H9", "node_id": "missioncontrol_indexer_1"}' 198 | ``` 199 | 200 | ## Indexer Agent 201 | 202 | If you check the indexer-service status, you will see that both the indexer-agent/service fail to start due to a missing database. To add it, open a terminal inside the postgres container. 203 | 204 | ```bash 205 | docker exec -ti bash 206 | 207 | psql -U graph-node 208 | 209 | CREATE DATABASE indexer; 210 | ``` 211 | 212 | Now indexer service and agent should be running. If you don't have Rinkeby ETH and GRT in your wallet, then the agent may fail to start. 213 | 214 | # Completing Mission 1 215 | 216 | To wrap up the Phase 1 Mission 1 challenge, we need to provide the output from our indexer-service status, as described [here](https://github.com/graphprotocol/mission-control-indexer#successful-completion). 217 | 218 | On your local machine, set npm to use The Graph's private package registry Verdaccio (another awesome [self-hosted tool](https://verdaccio.org/)). I recommend against yarn, as it can be troublesome to set up with Verdaccio. 219 | 220 | ```bash 221 | # On your local machine 222 | npm set registry https://testnet.thegraph.com/npm-registry 223 | npm login 224 | # Use the credentials provided in the email from Eva 225 | ``` 226 | 227 | Install the CLI tools, then reset your registry back to normal. 228 | 229 | ```bash 230 | # Install the latest 231 | npm install -g \ 232 | @graphprotocol/graph-cli \ 233 | @graphprotocol/indexer-cli 234 | 235 | npm set registry https://registry.npmjs.com/ 236 | ``` 237 | 238 | Now, on your server, uncomment the network and traefik tags for the indexer-agent and deploy the stack. This will temporarily expose our indexer-agent to the world, which you should not normally do. Instead you should use ssh tunneling, since the agent is like the admin portal for your indexer. I plan to find a better solution for accessing it, I just haven't figured out a good method yet. 239 | 240 | ```yml 241 | # Swarm 242 | networks: 243 | - default 244 | - traefik-public 245 | deploy: 246 | # NOTE: Debug use only! Agent should not be exposed publicly. Instead use the CLI via ssh tunnel 247 | labels: 248 | - traefik.enable=true 249 | - traefik.docker.network=traefik-public 250 | - traefik.constraint-label=traefik-public 251 | - traefik.http.routers.indexer-agent-http.service=indexer-agent 252 | - traefik.http.routers.indexer-agent-https.service=indexer-agent 253 | #... 254 | ``` 255 | 256 | Connect the indexer CLI on your local machine to the indexer-agent. Then check the status. 257 | 258 | ```bash 259 | graph indexer connect http://agent.mysite.com 260 | graph indexer status 261 | ``` 262 | 263 | ![](./screenshots/status.png) 264 | 265 | ## Update the indexer rules 266 | 267 | This is very important, since the agent will STOP SYNCING your subgraphs unless you stake GRT. I personally would like the ability to turn off this feature using a “always sync” flag. For now though, we need to update our rules table. This will set the minimum stake needed to sync at 50 GRT, and set Moloch and Uniswap to 100 GRT stake (I guess we don’t need to worry about Synthetix for now). 268 | 269 | ```bash 270 | graph indexer rules set global minStake 50 271 | graph indexer rules set QmXKwSEMirgWVn41nRzkT3hpUBw29cp619Gx58XW6mPhZP allocationAmount 100 272 | graph indexer rules set QmTXzATwNfgGVukV1fX2T6xw9f6LAYRVWpsdXyRWzUR2H9 allocationAmount 100 273 | 274 | graph indexer rules get all --merged 275 | ``` 276 | 277 | Now that we are done, DON’T FORGET to remove the traefik tags on the index-agent and redeploy the stack. We don’t want to start a bad habit. 278 | 279 | # Whats next? 280 | 281 | 👏👏👏 Awesome work! Hopefully you learned some good stuff today. Head back to the [indexer-docker-compose repo](https://github.com/pi0neerpat/indexer-docker-compose) to start Phase 1 Mission 2 on interacting with the Agent. 282 | 283 | #### Additional Resources 284 | 285 | - See an issue or have a comment? Feel free to leave an issue [here](https://github.com/pi0neerpat/indexer-docker-compose/issues). 286 | - [Non-swarm](./non-swarm) docker-compose provided by @amxx 287 | - [Convenience scripts](./scripts) for the entire set up provided by @pkrasam 288 | 289 | #### Definitions 290 | 291 | - **Indexer Agent**: A small component that comes with a small database with maybe 200 rows. It doesn't require a lot of CPU, therefore it can be run on the same machine as the graph-nodes. 292 | - **Indexer Service**: TODO 293 | -------------------------------------------------------------------------------- /indexer-install/grafana/grafana.ini: -------------------------------------------------------------------------------- 1 | [server] 2 | domain = 54.215.29.38 3 | root_url = %(protocol)s://%(domain)s:%(http_port)s/grafana/ 4 | serve_from_sub_path = true 5 | 6 | -------------------------------------------------------------------------------- /indexer-install/grafana/provisioning/dashboards/default.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: 1 2 | providers: 3 | - name: 'default' 4 | orgId: 1 5 | folder: '' 6 | folderUid: '' 7 | type: file 8 | allowUiUpdates: true 9 | updateIntervalSeconds: 31536000 10 | options: 11 | path: /etc/grafana/provisioning/dashboards 12 | -------------------------------------------------------------------------------- /indexer-install/grafana/provisioning/dashboards/postgres.json: -------------------------------------------------------------------------------- 1 | { 2 | "annotations": { 3 | "list": [ 4 | { 5 | "builtIn": 1, 6 | "datasource": "-- Grafana --", 7 | "enable": true, 8 | "hide": true, 9 | "iconColor": "rgba(0, 211, 255, 1)", 10 | "name": "Annotations & Alerts", 11 | "type": "dashboard" 12 | } 13 | ] 14 | }, 15 | "editable": true, 16 | "gnetId": null, 17 | "graphTooltip": 0, 18 | "id": 3, 19 | "links": [], 20 | "panels": [ 21 | { 22 | "columns": [], 23 | "datasource": "postgres", 24 | "fontSize": "100%", 25 | "gridPos": { 26 | "h": 8, 27 | "w": 24, 28 | "x": 0, 29 | "y": 0 30 | }, 31 | "id": 4, 32 | "links": [], 33 | "pageSize": null, 34 | "scroll": true, 35 | "showHeader": true, 36 | "sort": { 37 | "col": 1, 38 | "desc": true 39 | }, 40 | "styles": [ 41 | { 42 | "alias": "Age", 43 | "align": "auto", 44 | "dateFormat": "YYYY-MM-DD HH:mm:ss", 45 | "pattern": "age", 46 | "type": "number", 47 | "unit": "s" 48 | }, 49 | { 50 | "alias": "", 51 | "align": "auto", 52 | "colorMode": null, 53 | "colors": [ 54 | "rgba(245, 54, 54, 0.9)", 55 | "rgba(237, 129, 40, 0.89)", 56 | "rgba(50, 172, 45, 0.97)" 57 | ], 58 | "decimals": 2, 59 | "pattern": "/.*/", 60 | "thresholds": [], 61 | "type": "number", 62 | "unit": "short" 63 | } 64 | ], 65 | "targets": [ 66 | { 67 | "format": "table", 68 | "group": [], 69 | "metricColumn": "none", 70 | "rawQuery": true, 71 | "rawSql": "-- grafana ignore\nselect application_name,\n extract(epoch from age(now(), xact_start)) as age,\n query from pg_stat_activity\n where query not like '%grafana ignore%'\n and state='active'\norder by query_start desc", 72 | "refId": "A", 73 | "select": [ 74 | [ 75 | { 76 | "params": [ 77 | "value" 78 | ], 79 | "type": "column" 80 | } 81 | ] 82 | ], 83 | "timeColumn": "time", 84 | "where": [ 85 | { 86 | "name": "$__timeFilter", 87 | "params": [], 88 | "type": "macro" 89 | } 90 | ] 91 | } 92 | ], 93 | "timeFrom": null, 94 | "timeShift": null, 95 | "title": "Active Queries", 96 | "transform": "table", 97 | "type": "table" 98 | }, 99 | { 100 | "columns": [], 101 | "datasource": "postgres", 102 | "fontSize": "100%", 103 | "gridPos": { 104 | "h": 5, 105 | "w": 13, 106 | "x": 0, 107 | "y": 8 108 | }, 109 | "id": 9, 110 | "links": [], 111 | "pageSize": null, 112 | "pluginVersion": "6.2.1", 113 | "scroll": true, 114 | "showHeader": true, 115 | "sort": { 116 | "col": 1, 117 | "desc": false 118 | }, 119 | "styles": [ 120 | { 121 | "alias": "", 122 | "align": "auto", 123 | "dateFormat": "YYYY-MM-DD", 124 | "pattern": "last_autovacuum", 125 | "type": "date" 126 | }, 127 | { 128 | "alias": "tx left", 129 | "align": "auto", 130 | "colorMode": "row", 131 | "colors": [ 132 | "rgba(245, 54, 54, 0.9)", 133 | "rgba(237, 129, 40, 0.89)", 134 | "rgba(50, 172, 45, 0.97)" 135 | ], 136 | "decimals": null, 137 | "pattern": "tx_before_wraparound_vacuum", 138 | "thresholds": [ 139 | "20000000", 140 | "40000000" 141 | ], 142 | "type": "number", 143 | "unit": "short" 144 | }, 145 | { 146 | "alias": "", 147 | "align": "auto", 148 | "colorMode": null, 149 | "colors": [ 150 | "rgba(245, 54, 54, 0.9)", 151 | "rgba(237, 129, 40, 0.89)", 152 | "rgba(50, 172, 45, 0.97)" 153 | ], 154 | "dateFormat": "YYYY-MM-DD HH:mm:ss", 155 | "decimals": 2, 156 | "mappingType": 1, 157 | "pattern": "xid_age", 158 | "thresholds": [], 159 | "type": "number", 160 | "unit": "short" 161 | } 162 | ], 163 | "targets": [ 164 | { 165 | "format": "table", 166 | "group": [], 167 | "metricColumn": "none", 168 | "rawQuery": true, 169 | "rawSql": "-- grafana ignore\nSELECT\n relname as table,\n least((SELECT setting::int FROM pg_settings WHERE name = 'autovacuum_freeze_max_age') - age(relfrozenxid), \n (SELECT setting::int FROM pg_settings WHERE name = 'autovacuum_multixact_freeze_max_age') - mxid_age(relminmxid))\n tx_before_wraparound_vacuum,\n pg_stat_get_last_autovacuum_time(c.oid) AS last_autovacuum,\n age(relfrozenxid) AS xid_age,\n mxid_age(relminmxid) AS mxid_age\nFROM\n pg_class c\nWHERE\n c.relname in ('ethereum_blocks', 'eth_call_cache','subgraph_deployment')\n and c.relfrozenxid != 0\n", 170 | "refId": "A", 171 | "select": [ 172 | [ 173 | { 174 | "params": [ 175 | "value" 176 | ], 177 | "type": "column" 178 | } 179 | ] 180 | ], 181 | "timeColumn": "time", 182 | "where": [ 183 | { 184 | "name": "$__timeFilter", 185 | "params": [], 186 | "type": "macro" 187 | } 188 | ] 189 | } 190 | ], 191 | "timeFrom": null, 192 | "timeShift": null, 193 | "title": "Transactions until wraparound vacuum", 194 | "transform": "table", 195 | "type": "table" 196 | }, 197 | { 198 | "columns": [], 199 | "datasource": "postgres", 200 | "fontSize": "100%", 201 | "gridPos": { 202 | "h": 3, 203 | "w": 11, 204 | "x": 13, 205 | "y": 8 206 | }, 207 | "id": 11, 208 | "pageSize": null, 209 | "showHeader": true, 210 | "sort": { 211 | "col": 0, 212 | "desc": true 213 | }, 214 | "styles": [ 215 | { 216 | "alias": "Tables needing vacuum", 217 | "align": "auto", 218 | "colorMode": null, 219 | "colors": [ 220 | "rgba(245, 54, 54, 0.9)", 221 | "rgba(237, 129, 40, 0.89)", 222 | "rgba(50, 172, 45, 0.97)" 223 | ], 224 | "decimals": 0, 225 | "pattern": "tables_needing_vacuum", 226 | "thresholds": [], 227 | "type": "number", 228 | "unit": "none" 229 | }, 230 | { 231 | "alias": "Txns past", 232 | "align": "auto", 233 | "colorMode": null, 234 | "colors": [ 235 | "rgba(245, 54, 54, 0.9)", 236 | "rgba(237, 129, 40, 0.89)", 237 | "rgba(50, 172, 45, 0.97)" 238 | ], 239 | "dateFormat": "YYYY-MM-DD HH:mm:ss", 240 | "decimals": 0, 241 | "mappingType": 1, 242 | "pattern": "txns_past", 243 | "thresholds": [], 244 | "type": "number", 245 | "unit": "locale" 246 | }, 247 | { 248 | "alias": "Last autovacuum", 249 | "align": "auto", 250 | "colorMode": null, 251 | "colors": [ 252 | "rgba(245, 54, 54, 0.9)", 253 | "rgba(237, 129, 40, 0.89)", 254 | "rgba(50, 172, 45, 0.97)" 255 | ], 256 | "dateFormat": "YYYY-MM-DD HH:mm:ss", 257 | "decimals": 2, 258 | "mappingType": 1, 259 | "pattern": "last_autovacuum", 260 | "thresholds": [], 261 | "type": "date", 262 | "unit": "short" 263 | } 264 | ], 265 | "targets": [ 266 | { 267 | "format": "table", 268 | "group": [], 269 | "metricColumn": "none", 270 | "rawQuery": true, 271 | "rawSql": "-- grafana ignore\nselect count(*) as tables_needing_vacuum,\n -min(tx_before_wraparound_vacuum) as txns_past,\n min(last_autovacuum) as last_autovacuum\n from (\n select oid::regclass::text AS table,\n least(\n (select setting::int\n from pg_settings\n where name = 'autovacuum_freeze_max_age') - age(relfrozenxid),\n (select setting::int\n from pg_settings\n where name = 'autovacuum_multixact_freeze_max_age')\n - mxid_age(relminmxid)) as tx_before_wraparound_vacuum,\n pg_stat_get_last_autovacuum_time(oid) AS last_autovacuum,\n age(relfrozenxid) AS xid_age,\n mxid_age(relminmxid) AS mxid_age\n from pg_class\n where relfrozenxid != 0\n and oid > 16384\n and relkind='r') a where a.tx_before_wraparound_vacuum < 0;\n", 272 | "refId": "A", 273 | "select": [ 274 | [ 275 | { 276 | "params": [ 277 | "value" 278 | ], 279 | "type": "column" 280 | } 281 | ] 282 | ], 283 | "timeColumn": "time", 284 | "where": [ 285 | { 286 | "name": "$__timeFilter", 287 | "params": [], 288 | "type": "macro" 289 | } 290 | ] 291 | } 292 | ], 293 | "timeFrom": null, 294 | "timeShift": null, 295 | "title": "Autovacuum pressure", 296 | "transform": "table", 297 | "type": "table" 298 | }, 299 | { 300 | "columns": [], 301 | "datasource": "postgres", 302 | "fontSize": "100%", 303 | "gridPos": { 304 | "h": 9, 305 | "w": 24, 306 | "x": 0, 307 | "y": 13 308 | }, 309 | "id": 7, 310 | "links": [], 311 | "pageSize": null, 312 | "scroll": true, 313 | "showHeader": true, 314 | "sort": { 315 | "col": 4, 316 | "desc": false 317 | }, 318 | "styles": [ 319 | { 320 | "alias": "Granted", 321 | "align": "auto", 322 | "dateFormat": "YYYY-MM-DD HH:mm:ss", 323 | "decimals": 0, 324 | "mappingType": 1, 325 | "pattern": "granted", 326 | "preserveFormat": false, 327 | "sanitize": false, 328 | "thresholds": [ 329 | "" 330 | ], 331 | "type": "string", 332 | "unit": "none", 333 | "valueMaps": [ 334 | { 335 | "text": "✓", 336 | "value": "true" 337 | }, 338 | { 339 | "text": "—", 340 | "value": "false" 341 | } 342 | ] 343 | }, 344 | { 345 | "alias": "Age", 346 | "align": "auto", 347 | "colorMode": null, 348 | "colors": [ 349 | "rgba(245, 54, 54, 0.9)", 350 | "rgba(237, 129, 40, 0.89)", 351 | "rgba(50, 172, 45, 0.97)" 352 | ], 353 | "decimals": 2, 354 | "pattern": "age", 355 | "thresholds": [], 356 | "type": "number", 357 | "unit": "s" 358 | } 359 | ], 360 | "targets": [ 361 | { 362 | "format": "table", 363 | "group": [], 364 | "metricColumn": "none", 365 | "rawQuery": true, 366 | "rawSql": "-- grafana ignore\nSELECT a.application_name,\n coalesce(extract(epoch from age(now(), a.xact_start)), 0) as age,\n l.relation::regclass,\n l.mode,\n l.GRANTED::varchar,\n l.locktype \"Target\"\nFROM pg_stat_activity a\nJOIN pg_locks l ON l.pid = a.pid\nwhere pg_backend_pid() != a.pid\norder by granted asc, age desc;", 367 | "refId": "A", 368 | "select": [ 369 | [ 370 | { 371 | "params": [ 372 | "value" 373 | ], 374 | "type": "column" 375 | } 376 | ] 377 | ], 378 | "timeColumn": "time", 379 | "where": [ 380 | { 381 | "name": "$__timeFilter", 382 | "params": [], 383 | "type": "macro" 384 | } 385 | ] 386 | } 387 | ], 388 | "timeFrom": null, 389 | "timeShift": null, 390 | "title": "Active locks", 391 | "transform": "table", 392 | "type": "table" 393 | }, 394 | { 395 | "columns": [], 396 | "datasource": "postgres", 397 | "fontSize": "100%", 398 | "gridPos": { 399 | "h": 8, 400 | "w": 24, 401 | "x": 0, 402 | "y": 22 403 | }, 404 | "id": 5, 405 | "links": [], 406 | "pageSize": null, 407 | "scroll": true, 408 | "showHeader": true, 409 | "sort": { 410 | "col": 4, 411 | "desc": false 412 | }, 413 | "styles": [ 414 | { 415 | "alias": "Age", 416 | "align": "auto", 417 | "dateFormat": "YYYY-MM-DD HH:mm:ss", 418 | "pattern": "age", 419 | "type": "number", 420 | "unit": "s" 421 | }, 422 | { 423 | "alias": "", 424 | "align": "auto", 425 | "colorMode": null, 426 | "colors": [ 427 | "rgba(245, 54, 54, 0.9)", 428 | "rgba(237, 129, 40, 0.89)", 429 | "rgba(50, 172, 45, 0.97)" 430 | ], 431 | "decimals": 2, 432 | "pattern": "/.*/", 433 | "thresholds": [], 434 | "type": "number", 435 | "unit": "short" 436 | } 437 | ], 438 | "targets": [ 439 | { 440 | "format": "table", 441 | "group": [], 442 | "metricColumn": "none", 443 | "rawQuery": true, 444 | "rawSql": "-- grafana ignore\nselect client_addr,\n application_name,\n usename,\n state,\n extract(epoch from age(now(), xact_start)) as age\n from pg_stat_activity\n where query not like '%grafana ignore%'\n and state like '%idle in transaction%'\norder by query_start desc", 445 | "refId": "A", 446 | "select": [ 447 | [ 448 | { 449 | "params": [ 450 | "value" 451 | ], 452 | "type": "column" 453 | } 454 | ] 455 | ], 456 | "timeColumn": "time", 457 | "where": [ 458 | { 459 | "name": "$__timeFilter", 460 | "params": [], 461 | "type": "macro" 462 | } 463 | ] 464 | } 465 | ], 466 | "timeFrom": null, 467 | "timeShift": null, 468 | "title": "Idle transactions", 469 | "transform": "table", 470 | "type": "table" 471 | } 472 | ], 473 | "schemaVersion": 22, 474 | "style": "dark", 475 | "tags": [], 476 | "templating": { 477 | "list": [] 478 | }, 479 | "time": { 480 | "from": "now-6h", 481 | "to": "now" 482 | }, 483 | "timepicker": { 484 | "refresh_intervals": [ 485 | "5s", 486 | "10s", 487 | "30s", 488 | "1m", 489 | "5m", 490 | "15m", 491 | "30m", 492 | "1h", 493 | "2h", 494 | "1d" 495 | ], 496 | "time_options": [ 497 | "5m", 498 | "15m", 499 | "1h", 500 | "6h", 501 | "12h", 502 | "24h", 503 | "2d", 504 | "7d", 505 | "30d" 506 | ] 507 | }, 508 | "timezone": "", 509 | "title": "Postgres Statistics", 510 | "uid": "Mo6FxoiWz", 511 | "variables": { 512 | "list": [] 513 | }, 514 | "version": 4 515 | } 516 | -------------------------------------------------------------------------------- /indexer-install/grafana/provisioning/datasources/postgres.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: 1 2 | datasources: 3 | - access: proxy 4 | editable: true 5 | name: postgres 6 | orgId: 1 7 | type: postgres 8 | url: $postgres_host 9 | user: $postgres_user 10 | database: $postgres_db 11 | secureJsonData: 12 | password: $postgres_pass 13 | jsonData: 14 | sslmode: "disable" 15 | postgresVersion: 906 16 | -------------------------------------------------------------------------------- /indexer-install/grafana/provisioning/datasources/prometheus.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: 1 2 | datasources: 3 | - access: proxy 4 | editable: true 5 | name: prometheus 6 | orgId: 1 7 | type: prometheus 8 | url: http://prometheus:9090 9 | version: 1 10 | -------------------------------------------------------------------------------- /indexer-install/non-swarm/.template.env: -------------------------------------------------------------------------------- 1 | MNEMONIC= 2 | INFURA_API= 3 | ALCHEMY_API= 4 | DB_USER= 5 | DB_PASSWORD= 6 | DB_NAME_GRAPHNODES=graphnode 7 | DB_NAME_INDEXER=indexer 8 | -------------------------------------------------------------------------------- /indexer-install/non-swarm/docker-compose.yml: -------------------------------------------------------------------------------- 1 | version: '3' 2 | services: 3 | 4 | ############################################################################ 5 | # TURBO-GETH - Archive Node # 6 | ############################################################################ 7 | 8 | # turbo-geth: 9 | # image: turbo-geth:latest 10 | # build: ../turbo-geth 11 | # restart: always 12 | # command: 13 | # - 'tg' 14 | # - '--nousb' 15 | # - '--metrics' 16 | # - '--pprof.addr=0.0.0.0' 17 | # - '--pprof.port=6060' 18 | # - '--private.api.addr=0.0.0.0:9090' 19 | # - '--ipcdisable' 20 | # #ports: 21 | # # - 30303:30303 22 | # volumes: 23 | # - /data/turbogeth:/root/.local/share/turbogeth 24 | # 25 | # rpcdaemon: 26 | # image: turbo-geth:latest 27 | # restart: always 28 | # depends_on: 29 | # - turbo-geth 30 | # command: 31 | # - 'rpcdaemon' 32 | # - '--private.api.addr=turbo-geth:9090' 33 | # - '--http.addr=0.0.0.0' 34 | # - '--http.vhosts=*' 35 | # - '--http.corsdomain=*' 36 | # - '--http.api=eth,debug,net' 37 | # ports: 38 | # - 8545:8545 39 | 40 | ############################################################################ 41 | # POSTGRES - Database # 42 | ############################################################################ 43 | 44 | postgres: 45 | image: postgres 46 | user: '1000:1000' 47 | restart: always 48 | command: 49 | - 'postgres' 50 | - '-cshared_preload_libraries=pg_stat_statements' 51 | ports: 52 | - 5432:5432 53 | environment: 54 | POSTGRES_USER: '${DB_USER}' 55 | POSTGRES_PASSWORD: '${DB_PASSWORD}' 56 | volumes: 57 | - /data/postgres:/var/lib/postgresql/data 58 | 59 | ############################################################################ 60 | # GRAPHNODES - indexering and query # 61 | ############################################################################ 62 | 63 | graphnode-query-1: 64 | image: graphprotocol/graph-node:latest 65 | restart: always 66 | depends_on: 67 | - postgres 68 | #ports: 69 | # - '8000:8000' 70 | # - '8001:8001' 71 | # - '8020:8020' 72 | # - '8030:8030' 73 | # - '8040:8040' 74 | environment: 75 | postgres_host: 'postgres:5432' 76 | postgres_db: 'graphnode' 77 | postgres_user: '${DB_USER}' 78 | postgres_pass: '${DB_PASSWORD}' 79 | ipfs: 'https://testnet.thegraph.com/ipfs/' 80 | ethereum: 'mainnet:https://eth-mainnet.alchemyapi.io/jsonrpc/${ALCHEMY_API}' 81 | GRAPH_NODE_ID: 'missioncontrol_query_1' 82 | DISABLE_BLOCK_INGESTOR: 'true' 83 | logging: 84 | driver: 'json-file' 85 | options: 86 | max-size: '200k' 87 | max-file: '10' 88 | 89 | graphnode-indexer-1: 90 | image: graphprotocol/graph-node:latest 91 | restart: always 92 | depends_on: 93 | - postgres 94 | environment: 95 | postgres_host: 'postgres:5432' 96 | postgres_db: 'graphnode' 97 | postgres_user: '${DB_USER}' 98 | postgres_pass: '${DB_PASSWORD}' 99 | ipfs: 'https://testnet.thegraph.com/ipfs/' 100 | # mainnet:http://rpcdaemon:8545 101 | ethereum: ' 102 | mainnet:https://eth-mainnet.alchemyapi.io/jsonrpc/${ALCHEMY_API} 103 | ropsten:https://eth-ropsten.alchemyapi.io/jsonrpc/${ALCHEMY_API} 104 | rinkeby:https://eth-rinkeby.alchemyapi.io/jsonrpc/${ALCHEMY_API} 105 | goerli:https://eth-goerli.alchemyapi.io/jsonrpc/${ALCHEMY_API} 106 | kovan:https://eth-kovan.alchemyapi.io/jsonrpc/${ALCHEMY_API} 107 | ' 108 | GRAPH_NODE_ID: 'missioncontrol_indexer_1' 109 | logging: 110 | driver: 'json-file' 111 | options: 112 | max-size: '200k' 113 | max-file: '10' 114 | 115 | ############################################################################ 116 | # INDEXER - mission-control network # 117 | ############################################################################ 118 | 119 | indexer-agent: 120 | image: graphprotocol/indexer-agent 121 | depends_on: 122 | - graphnode-query-1 123 | #ports: 124 | # - 18000:18000 125 | environment: 126 | INDEXER_AGENT_INDEXER_MANAGEMENT_PORT: 18000 127 | INDEXER_AGENT_MNEMONIC: '${MNEMONIC}' 128 | INDEXER_AGENT_INDEX_NODE_IDS: 'missioncontrol_indexer_1' 129 | INDEXER_AGENT_PUBLIC_INDEXER_URL: 'http://mission-control.thegraph.iex.ec/' 130 | INDEXER_AGENT_INDEXER_GEO_COORDINATES: '"37.630768,-119.032631"' 131 | INDEXER_AGENT_ETHEREUM: 'https://eth-rinkeby.alchemyapi.io/jsonrpc/${ALCHEMY_API}' 132 | INDEXER_AGENT_GRAPH_NODE_QUERY_ENDPOINT: 'http://graphnode-query-1:8000/' 133 | INDEXER_AGENT_GRAPH_NODE_ADMIN_ENDPOINT: 'http://graphnode-query-1:8020/' 134 | INDEXER_AGENT_GRAPH_NODE_STATUS_ENDPOINT: 'http://graphnode-query-1:8030/graphql' 135 | INDEXER_AGENT_NETWORK_SUBGRAPH_ENDPOINT: 'https://api.thegraph.com/subgraphs/name/graphprotocol/graph-network-rinkeby' 136 | INDEXER_AGENT_POSTGRES_HOST: 'postgres' 137 | INDEXER_AGENT_POSTGRES_PORT: 5432 138 | INDEXER_AGENT_POSTGRES_DATABASE: 'indexer' 139 | INDEXER_AGENT_POSTGRES_USERNAME: '${DB_USER}' 140 | INDEXER_AGENT_POSTGRES_PASSWORD: '${DB_PASSWORD}' 141 | logging: 142 | driver: 'json-file' 143 | options: 144 | max-size: '200k' 145 | max-file: '10' 146 | 147 | indexer-service: 148 | image: graphprotocol/indexer-service 149 | depends_on: 150 | - graphnode-query-1 151 | #ports: 152 | # - 7600:7600 153 | environment: 154 | INDEXER_SERVICE_PORT: 7600 155 | INDEXER_SERVICE_MNEMONIC: '${MNEMONIC}' 156 | INDEXER_SERVICE_ETHEREUM: 'https://eth-rinkeby.alchemyapi.io/jsonrpc/${ALCHEMY_API}' 157 | INDEXER_SERVICE_GRAPH_NODE_QUERY_ENDPOINT: 'http://graphnode-query-1:8000/' 158 | INDEXER_SERVICE_GRAPH_NODE_STATUS_ENDPOINT: 'http://graphnode-query-1:8030/graphql' 159 | INDEXER_SERVICE_NETWORK_SUBGRAPH_ENDPOINT: 'https://api.thegraph.com/subgraphs/name/graphprotocol/graph-network-rinkeby' 160 | SERVER_HOST: 'postgres' 161 | SERVER_PORT: 5432 162 | SERVER_DB_NAME: 'indexer' 163 | SERVER_DB_USER: '${DB_USER}' 164 | SERVER_DB_PASSWORD: '${DB_PASSWORD}' 165 | logging: 166 | driver: 'json-file' 167 | options: 168 | max-size: '200k' 169 | max-file: '10' 170 | 171 | ############################################################################ 172 | # MONITORING # 173 | ############################################################################ 174 | 175 | prometheus: 176 | image: prom/prometheus 177 | user: '1000:1000' 178 | restart: always 179 | depends_on: 180 | - graphnode-indexer-1 181 | command: 182 | - '--log.level=warn' 183 | - '--config.file=/etc/prometheus/prometheus.yml' 184 | - '--storage.tsdb.path=/prometheus' 185 | #ports: 186 | # - 9090:9090 187 | volumes: 188 | - ./prometheus:/etc/prometheus:ro 189 | - /data/prometheus:/prometheus 190 | 191 | grafana: 192 | image: grafana/grafana 193 | user: '1000:1000' 194 | restart: always 195 | depends_on: 196 | - postgres 197 | - prometheus 198 | #ports: 199 | # - 3000:3000 200 | environment: 201 | postgres_host: 'postgres:5432' 202 | postgres_db: 'graphnode' 203 | postgres_user: '${DB_USER}' 204 | postgres_pass: '${DB_PASSWORD}' 205 | volumes: 206 | - ./grafana:/etc/grafana/ 207 | - /data/grafana:/var/lib/grafana 208 | 209 | ############################################################################ 210 | # REVERSE PROXY # 211 | ############################################################################ 212 | 213 | nginx: 214 | image: nginx 215 | restart: always 216 | ports: 217 | - 80:80 218 | - 443:443 219 | volumes: 220 | - ./nginx:/etc/nginx:ro 221 | - /etc/letsencrypt:/etc/letsencrypt 222 | -------------------------------------------------------------------------------- /indexer-install/non-swarm/grafana/grafana.ini: -------------------------------------------------------------------------------- 1 | [server] 2 | domain = 54.215.29.38 3 | root_url = %(protocol)s://%(domain)s:%(http_port)s/grafana/ 4 | serve_from_sub_path = true 5 | 6 | -------------------------------------------------------------------------------- /indexer-install/non-swarm/grafana/provisioning/dashboards/default.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: 1 2 | providers: 3 | - name: 'default' 4 | orgId: 1 5 | folder: '' 6 | folderUid: '' 7 | type: file 8 | allowUiUpdates: true 9 | updateIntervalSeconds: 31536000 10 | options: 11 | path: /etc/grafana/provisioning/dashboards 12 | -------------------------------------------------------------------------------- /indexer-install/non-swarm/grafana/provisioning/dashboards/postgres.json: -------------------------------------------------------------------------------- 1 | { 2 | "annotations": { 3 | "list": [ 4 | { 5 | "builtIn": 1, 6 | "datasource": "-- Grafana --", 7 | "enable": true, 8 | "hide": true, 9 | "iconColor": "rgba(0, 211, 255, 1)", 10 | "name": "Annotations & Alerts", 11 | "type": "dashboard" 12 | } 13 | ] 14 | }, 15 | "editable": true, 16 | "gnetId": null, 17 | "graphTooltip": 0, 18 | "id": 3, 19 | "links": [], 20 | "panels": [ 21 | { 22 | "columns": [], 23 | "datasource": "postgres", 24 | "fontSize": "100%", 25 | "gridPos": { 26 | "h": 8, 27 | "w": 24, 28 | "x": 0, 29 | "y": 0 30 | }, 31 | "id": 4, 32 | "links": [], 33 | "pageSize": null, 34 | "scroll": true, 35 | "showHeader": true, 36 | "sort": { 37 | "col": 1, 38 | "desc": true 39 | }, 40 | "styles": [ 41 | { 42 | "alias": "Age", 43 | "align": "auto", 44 | "dateFormat": "YYYY-MM-DD HH:mm:ss", 45 | "pattern": "age", 46 | "type": "number", 47 | "unit": "s" 48 | }, 49 | { 50 | "alias": "", 51 | "align": "auto", 52 | "colorMode": null, 53 | "colors": [ 54 | "rgba(245, 54, 54, 0.9)", 55 | "rgba(237, 129, 40, 0.89)", 56 | "rgba(50, 172, 45, 0.97)" 57 | ], 58 | "decimals": 2, 59 | "pattern": "/.*/", 60 | "thresholds": [], 61 | "type": "number", 62 | "unit": "short" 63 | } 64 | ], 65 | "targets": [ 66 | { 67 | "format": "table", 68 | "group": [], 69 | "metricColumn": "none", 70 | "rawQuery": true, 71 | "rawSql": "-- grafana ignore\nselect application_name,\n extract(epoch from age(now(), xact_start)) as age,\n query from pg_stat_activity\n where query not like '%grafana ignore%'\n and state='active'\norder by query_start desc", 72 | "refId": "A", 73 | "select": [ 74 | [ 75 | { 76 | "params": [ 77 | "value" 78 | ], 79 | "type": "column" 80 | } 81 | ] 82 | ], 83 | "timeColumn": "time", 84 | "where": [ 85 | { 86 | "name": "$__timeFilter", 87 | "params": [], 88 | "type": "macro" 89 | } 90 | ] 91 | } 92 | ], 93 | "timeFrom": null, 94 | "timeShift": null, 95 | "title": "Active Queries", 96 | "transform": "table", 97 | "type": "table" 98 | }, 99 | { 100 | "columns": [], 101 | "datasource": "postgres", 102 | "fontSize": "100%", 103 | "gridPos": { 104 | "h": 5, 105 | "w": 13, 106 | "x": 0, 107 | "y": 8 108 | }, 109 | "id": 9, 110 | "links": [], 111 | "pageSize": null, 112 | "pluginVersion": "6.2.1", 113 | "scroll": true, 114 | "showHeader": true, 115 | "sort": { 116 | "col": 1, 117 | "desc": false 118 | }, 119 | "styles": [ 120 | { 121 | "alias": "", 122 | "align": "auto", 123 | "dateFormat": "YYYY-MM-DD", 124 | "pattern": "last_autovacuum", 125 | "type": "date" 126 | }, 127 | { 128 | "alias": "tx left", 129 | "align": "auto", 130 | "colorMode": "row", 131 | "colors": [ 132 | "rgba(245, 54, 54, 0.9)", 133 | "rgba(237, 129, 40, 0.89)", 134 | "rgba(50, 172, 45, 0.97)" 135 | ], 136 | "decimals": null, 137 | "pattern": "tx_before_wraparound_vacuum", 138 | "thresholds": [ 139 | "20000000", 140 | "40000000" 141 | ], 142 | "type": "number", 143 | "unit": "short" 144 | }, 145 | { 146 | "alias": "", 147 | "align": "auto", 148 | "colorMode": null, 149 | "colors": [ 150 | "rgba(245, 54, 54, 0.9)", 151 | "rgba(237, 129, 40, 0.89)", 152 | "rgba(50, 172, 45, 0.97)" 153 | ], 154 | "dateFormat": "YYYY-MM-DD HH:mm:ss", 155 | "decimals": 2, 156 | "mappingType": 1, 157 | "pattern": "xid_age", 158 | "thresholds": [], 159 | "type": "number", 160 | "unit": "short" 161 | } 162 | ], 163 | "targets": [ 164 | { 165 | "format": "table", 166 | "group": [], 167 | "metricColumn": "none", 168 | "rawQuery": true, 169 | "rawSql": "-- grafana ignore\nSELECT\n relname as table,\n least((SELECT setting::int FROM pg_settings WHERE name = 'autovacuum_freeze_max_age') - age(relfrozenxid), \n (SELECT setting::int FROM pg_settings WHERE name = 'autovacuum_multixact_freeze_max_age') - mxid_age(relminmxid))\n tx_before_wraparound_vacuum,\n pg_stat_get_last_autovacuum_time(c.oid) AS last_autovacuum,\n age(relfrozenxid) AS xid_age,\n mxid_age(relminmxid) AS mxid_age\nFROM\n pg_class c\nWHERE\n c.relname in ('ethereum_blocks', 'eth_call_cache','subgraph_deployment')\n and c.relfrozenxid != 0\n", 170 | "refId": "A", 171 | "select": [ 172 | [ 173 | { 174 | "params": [ 175 | "value" 176 | ], 177 | "type": "column" 178 | } 179 | ] 180 | ], 181 | "timeColumn": "time", 182 | "where": [ 183 | { 184 | "name": "$__timeFilter", 185 | "params": [], 186 | "type": "macro" 187 | } 188 | ] 189 | } 190 | ], 191 | "timeFrom": null, 192 | "timeShift": null, 193 | "title": "Transactions until wraparound vacuum", 194 | "transform": "table", 195 | "type": "table" 196 | }, 197 | { 198 | "columns": [], 199 | "datasource": "postgres", 200 | "fontSize": "100%", 201 | "gridPos": { 202 | "h": 3, 203 | "w": 11, 204 | "x": 13, 205 | "y": 8 206 | }, 207 | "id": 11, 208 | "pageSize": null, 209 | "showHeader": true, 210 | "sort": { 211 | "col": 0, 212 | "desc": true 213 | }, 214 | "styles": [ 215 | { 216 | "alias": "Tables needing vacuum", 217 | "align": "auto", 218 | "colorMode": null, 219 | "colors": [ 220 | "rgba(245, 54, 54, 0.9)", 221 | "rgba(237, 129, 40, 0.89)", 222 | "rgba(50, 172, 45, 0.97)" 223 | ], 224 | "decimals": 0, 225 | "pattern": "tables_needing_vacuum", 226 | "thresholds": [], 227 | "type": "number", 228 | "unit": "none" 229 | }, 230 | { 231 | "alias": "Txns past", 232 | "align": "auto", 233 | "colorMode": null, 234 | "colors": [ 235 | "rgba(245, 54, 54, 0.9)", 236 | "rgba(237, 129, 40, 0.89)", 237 | "rgba(50, 172, 45, 0.97)" 238 | ], 239 | "dateFormat": "YYYY-MM-DD HH:mm:ss", 240 | "decimals": 0, 241 | "mappingType": 1, 242 | "pattern": "txns_past", 243 | "thresholds": [], 244 | "type": "number", 245 | "unit": "locale" 246 | }, 247 | { 248 | "alias": "Last autovacuum", 249 | "align": "auto", 250 | "colorMode": null, 251 | "colors": [ 252 | "rgba(245, 54, 54, 0.9)", 253 | "rgba(237, 129, 40, 0.89)", 254 | "rgba(50, 172, 45, 0.97)" 255 | ], 256 | "dateFormat": "YYYY-MM-DD HH:mm:ss", 257 | "decimals": 2, 258 | "mappingType": 1, 259 | "pattern": "last_autovacuum", 260 | "thresholds": [], 261 | "type": "date", 262 | "unit": "short" 263 | } 264 | ], 265 | "targets": [ 266 | { 267 | "format": "table", 268 | "group": [], 269 | "metricColumn": "none", 270 | "rawQuery": true, 271 | "rawSql": "-- grafana ignore\nselect count(*) as tables_needing_vacuum,\n -min(tx_before_wraparound_vacuum) as txns_past,\n min(last_autovacuum) as last_autovacuum\n from (\n select oid::regclass::text AS table,\n least(\n (select setting::int\n from pg_settings\n where name = 'autovacuum_freeze_max_age') - age(relfrozenxid),\n (select setting::int\n from pg_settings\n where name = 'autovacuum_multixact_freeze_max_age')\n - mxid_age(relminmxid)) as tx_before_wraparound_vacuum,\n pg_stat_get_last_autovacuum_time(oid) AS last_autovacuum,\n age(relfrozenxid) AS xid_age,\n mxid_age(relminmxid) AS mxid_age\n from pg_class\n where relfrozenxid != 0\n and oid > 16384\n and relkind='r') a where a.tx_before_wraparound_vacuum < 0;\n", 272 | "refId": "A", 273 | "select": [ 274 | [ 275 | { 276 | "params": [ 277 | "value" 278 | ], 279 | "type": "column" 280 | } 281 | ] 282 | ], 283 | "timeColumn": "time", 284 | "where": [ 285 | { 286 | "name": "$__timeFilter", 287 | "params": [], 288 | "type": "macro" 289 | } 290 | ] 291 | } 292 | ], 293 | "timeFrom": null, 294 | "timeShift": null, 295 | "title": "Autovacuum pressure", 296 | "transform": "table", 297 | "type": "table" 298 | }, 299 | { 300 | "columns": [], 301 | "datasource": "postgres", 302 | "fontSize": "100%", 303 | "gridPos": { 304 | "h": 9, 305 | "w": 24, 306 | "x": 0, 307 | "y": 13 308 | }, 309 | "id": 7, 310 | "links": [], 311 | "pageSize": null, 312 | "scroll": true, 313 | "showHeader": true, 314 | "sort": { 315 | "col": 4, 316 | "desc": false 317 | }, 318 | "styles": [ 319 | { 320 | "alias": "Granted", 321 | "align": "auto", 322 | "dateFormat": "YYYY-MM-DD HH:mm:ss", 323 | "decimals": 0, 324 | "mappingType": 1, 325 | "pattern": "granted", 326 | "preserveFormat": false, 327 | "sanitize": false, 328 | "thresholds": [ 329 | "" 330 | ], 331 | "type": "string", 332 | "unit": "none", 333 | "valueMaps": [ 334 | { 335 | "text": "✓", 336 | "value": "true" 337 | }, 338 | { 339 | "text": "—", 340 | "value": "false" 341 | } 342 | ] 343 | }, 344 | { 345 | "alias": "Age", 346 | "align": "auto", 347 | "colorMode": null, 348 | "colors": [ 349 | "rgba(245, 54, 54, 0.9)", 350 | "rgba(237, 129, 40, 0.89)", 351 | "rgba(50, 172, 45, 0.97)" 352 | ], 353 | "decimals": 2, 354 | "pattern": "age", 355 | "thresholds": [], 356 | "type": "number", 357 | "unit": "s" 358 | } 359 | ], 360 | "targets": [ 361 | { 362 | "format": "table", 363 | "group": [], 364 | "metricColumn": "none", 365 | "rawQuery": true, 366 | "rawSql": "-- grafana ignore\nSELECT a.application_name,\n coalesce(extract(epoch from age(now(), a.xact_start)), 0) as age,\n l.relation::regclass,\n l.mode,\n l.GRANTED::varchar,\n l.locktype \"Target\"\nFROM pg_stat_activity a\nJOIN pg_locks l ON l.pid = a.pid\nwhere pg_backend_pid() != a.pid\norder by granted asc, age desc;", 367 | "refId": "A", 368 | "select": [ 369 | [ 370 | { 371 | "params": [ 372 | "value" 373 | ], 374 | "type": "column" 375 | } 376 | ] 377 | ], 378 | "timeColumn": "time", 379 | "where": [ 380 | { 381 | "name": "$__timeFilter", 382 | "params": [], 383 | "type": "macro" 384 | } 385 | ] 386 | } 387 | ], 388 | "timeFrom": null, 389 | "timeShift": null, 390 | "title": "Active locks", 391 | "transform": "table", 392 | "type": "table" 393 | }, 394 | { 395 | "columns": [], 396 | "datasource": "postgres", 397 | "fontSize": "100%", 398 | "gridPos": { 399 | "h": 8, 400 | "w": 24, 401 | "x": 0, 402 | "y": 22 403 | }, 404 | "id": 5, 405 | "links": [], 406 | "pageSize": null, 407 | "scroll": true, 408 | "showHeader": true, 409 | "sort": { 410 | "col": 4, 411 | "desc": false 412 | }, 413 | "styles": [ 414 | { 415 | "alias": "Age", 416 | "align": "auto", 417 | "dateFormat": "YYYY-MM-DD HH:mm:ss", 418 | "pattern": "age", 419 | "type": "number", 420 | "unit": "s" 421 | }, 422 | { 423 | "alias": "", 424 | "align": "auto", 425 | "colorMode": null, 426 | "colors": [ 427 | "rgba(245, 54, 54, 0.9)", 428 | "rgba(237, 129, 40, 0.89)", 429 | "rgba(50, 172, 45, 0.97)" 430 | ], 431 | "decimals": 2, 432 | "pattern": "/.*/", 433 | "thresholds": [], 434 | "type": "number", 435 | "unit": "short" 436 | } 437 | ], 438 | "targets": [ 439 | { 440 | "format": "table", 441 | "group": [], 442 | "metricColumn": "none", 443 | "rawQuery": true, 444 | "rawSql": "-- grafana ignore\nselect client_addr,\n application_name,\n usename,\n state,\n extract(epoch from age(now(), xact_start)) as age\n from pg_stat_activity\n where query not like '%grafana ignore%'\n and state like '%idle in transaction%'\norder by query_start desc", 445 | "refId": "A", 446 | "select": [ 447 | [ 448 | { 449 | "params": [ 450 | "value" 451 | ], 452 | "type": "column" 453 | } 454 | ] 455 | ], 456 | "timeColumn": "time", 457 | "where": [ 458 | { 459 | "name": "$__timeFilter", 460 | "params": [], 461 | "type": "macro" 462 | } 463 | ] 464 | } 465 | ], 466 | "timeFrom": null, 467 | "timeShift": null, 468 | "title": "Idle transactions", 469 | "transform": "table", 470 | "type": "table" 471 | } 472 | ], 473 | "schemaVersion": 22, 474 | "style": "dark", 475 | "tags": [], 476 | "templating": { 477 | "list": [] 478 | }, 479 | "time": { 480 | "from": "now-6h", 481 | "to": "now" 482 | }, 483 | "timepicker": { 484 | "refresh_intervals": [ 485 | "5s", 486 | "10s", 487 | "30s", 488 | "1m", 489 | "5m", 490 | "15m", 491 | "30m", 492 | "1h", 493 | "2h", 494 | "1d" 495 | ], 496 | "time_options": [ 497 | "5m", 498 | "15m", 499 | "1h", 500 | "6h", 501 | "12h", 502 | "24h", 503 | "2d", 504 | "7d", 505 | "30d" 506 | ] 507 | }, 508 | "timezone": "", 509 | "title": "Postgres Statistics", 510 | "uid": "Mo6FxoiWz", 511 | "variables": { 512 | "list": [] 513 | }, 514 | "version": 4 515 | } 516 | -------------------------------------------------------------------------------- /indexer-install/non-swarm/grafana/provisioning/datasources/postgres.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: 1 2 | datasources: 3 | - access: proxy 4 | editable: true 5 | name: postgres 6 | orgId: 1 7 | type: postgres 8 | url: $postgres_host 9 | user: $postgres_user 10 | database: $postgres_db 11 | secureJsonData: 12 | password: $postgres_pass 13 | jsonData: 14 | sslmode: "disable" 15 | postgresVersion: 906 16 | -------------------------------------------------------------------------------- /indexer-install/non-swarm/grafana/provisioning/datasources/prometheus.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: 1 2 | datasources: 3 | - access: proxy 4 | editable: true 5 | name: prometheus 6 | orgId: 1 7 | type: prometheus 8 | url: http://prometheus:9090 9 | version: 1 10 | -------------------------------------------------------------------------------- /indexer-install/non-swarm/nginx/.htpasswd: -------------------------------------------------------------------------------- 1 | admin:{SHA}aaaaaaaaaa= 2 | -------------------------------------------------------------------------------- /indexer-install/non-swarm/nginx/nginx.conf: -------------------------------------------------------------------------------- 1 | events {} 2 | http { 3 | # See http://nginx.org/en/docs/http/websocket.html 4 | map $http_upgrade $connection_upgrade { 5 | default upgrade; 6 | '' close; 7 | } 8 | 9 | server { 10 | listen 80; 11 | listen [::]:80; 12 | 13 | # Uncomment to enable SSL 14 | listen 443 ssl; 15 | listen [::]:443 ssl; 16 | ssl_certificate /etc/letsencrypt/live/mission-control.thegraph.iex.ec/fullchain.pem; 17 | ssl_certificate_key /etc/letsencrypt/live/mission-control.thegraph.iex.ec/privkey.pem; 18 | 19 | location /.well-known { 20 | root /etc/letsencrypt/webroot; 21 | } 22 | 23 | location /prometheus/ { 24 | rewrite ^/prometheus/(.*)$ /$1 break; 25 | 26 | # Proxy configuration. 27 | proxy_pass http://prometheus:9090; 28 | proxy_http_version 1.1; 29 | proxy_set_header Connection $connection_upgrade; 30 | proxy_set_header Host $host; 31 | proxy_set_header Upgrade $http_upgrade; 32 | proxy_cache_bypass $http_upgrade; 33 | 34 | # Gateway timeout. 35 | proxy_read_timeout 30s; 36 | proxy_send_timeout 30s; 37 | } 38 | 39 | location /grafana/ { 40 | # rewrite ^/grafana/(.*)$ /$1 break; # NOT NEEDED, grafana configured as a subdomain 41 | 42 | # Proxy configuration. 43 | proxy_pass http://grafana:3000; 44 | proxy_http_version 1.1; 45 | proxy_set_header Connection $connection_upgrade; 46 | proxy_set_header Host $host; 47 | proxy_set_header Upgrade $http_upgrade; 48 | proxy_cache_bypass $http_upgrade; 49 | 50 | # Gateway timeout. 51 | proxy_read_timeout 30s; 52 | proxy_send_timeout 30s; 53 | } 54 | 55 | location /indexer/ { 56 | rewrite ^/indexer/(.*)$ /$1 break; 57 | 58 | auth_basic "Restricted"; 59 | auth_basic_user_file /etc/nginx/.htpasswd; 60 | 61 | # Proxy configuration. 62 | proxy_pass http://indexer-agent:18000; 63 | proxy_http_version 1.1; 64 | proxy_set_header Connection $connection_upgrade; 65 | proxy_set_header Host $host; 66 | proxy_set_header Upgrade $http_upgrade; 67 | proxy_cache_bypass $http_upgrade; 68 | 69 | # Gateway timeout. 70 | proxy_read_timeout 30s; 71 | proxy_send_timeout 30s; 72 | } 73 | 74 | location / { 75 | # Proxy configuration. 76 | proxy_pass http://indexer-service:7600; 77 | proxy_http_version 1.1; 78 | proxy_set_header Connection $connection_upgrade; 79 | proxy_set_header Host $host; 80 | proxy_set_header Upgrade $http_upgrade; 81 | proxy_cache_bypass $http_upgrade; 82 | 83 | # Gateway timeout. 84 | proxy_read_timeout 30s; 85 | proxy_send_timeout 30s; 86 | } 87 | 88 | # location / { 89 | # # Move WebSocket and HTTP requests into /ws/ and /http/ prefixes; 90 | # # this allows us to forward both types of requests to different 91 | # # query node ports 92 | # if ( $connection_upgrade = "upgrade" ) { rewrite ^(.*)$ /ws/$1; } 93 | # if ( $connection_upgrade != "upgrade" ) { rewrite ^(.*)$ /http/$1; } 94 | # 95 | # location /http { 96 | # # Remove the /http/ again 97 | # rewrite ^/http/(.*)$ $1 break; 98 | # 99 | # # Proxy configuration. 100 | # proxy_pass http://graphnode-query-1:8000; 101 | # proxy_http_version 1.1; 102 | # proxy_set_header Connection $connection_upgrade; 103 | # proxy_set_header Host $host; 104 | # proxy_set_header Upgrade $http_upgrade; 105 | # proxy_cache_bypass $http_upgrade; 106 | # 107 | # # Gateway timeout. 108 | # proxy_read_timeout 30s; 109 | # proxy_send_timeout 30s; 110 | # } 111 | # 112 | # location /ws { 113 | # # Remove the /ws/ again 114 | # rewrite ^/ws/(.*)$ $1 break; 115 | # 116 | # # Proxy configuration. 117 | # proxy_pass http://graphnode-query-1:8001; 118 | # proxy_http_version 1.1; 119 | # proxy_set_header Connection $connection_upgrade; 120 | # proxy_set_header Host $host; 121 | # proxy_set_header Upgrade $http_upgrade; 122 | # proxy_cache_bypass $http_upgrade; 123 | # 124 | # # Gateway timeout. 125 | # proxy_read_timeout 3600s; 126 | # proxy_send_timeout 3600s; 127 | # } 128 | # } 129 | } 130 | } 131 | -------------------------------------------------------------------------------- /indexer-install/non-swarm/prometheus/prometheus.yml: -------------------------------------------------------------------------------- 1 | global: 2 | scrape_interval: 15s 3 | evaluation_interval: 15s 4 | 5 | rule_files: 6 | 7 | alerting: 8 | 9 | scrape_configs: 10 | - job_name: 'graphnode-metrics' 11 | metrics_path: /metrics 12 | static_configs: 13 | - targets: ['graphnode-indexer-1:8040', 'graphnode-query-1:8040'] 14 | - job_name: 'indexer-service-metrics' 15 | metrics_path: /metrics 16 | static_configs: 17 | - targets: ['indexer-service:7300'] 18 | - job_name: 'indexer-agent-metrics' 19 | metrics_path: /metrics 20 | static_configs: 21 | - targets: ['indexer-service:7300'] 22 | 23 | -------------------------------------------------------------------------------- /indexer-install/prometheus/prometheus.yml: -------------------------------------------------------------------------------- 1 | global: 2 | scrape_interval: 15s 3 | evaluation_interval: 15s 4 | 5 | rule_files: 6 | 7 | alerting: 8 | 9 | scrape_configs: 10 | - job_name: "graphnode-metrics" 11 | static_configs: 12 | - targets: 13 | [ 14 | "graphnode-indexer-1:8040", 15 | "graphnode-query-1:8040", 16 | "graphnode-indexer-2:8040", 17 | ] 18 | - job_name: "indexer-service-metrics" 19 | static_configs: 20 | - targets: ["indexer-service:7300"] 21 | - job_name: "indexer-agent-metrics" 22 | static_configs: 23 | - targets: ["indexer-agent:7300"] 24 | -------------------------------------------------------------------------------- /indexer-install/screenshots/status.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/pi0neerpat/indexer-docker-compose/7aaaa53170c2e20e49a6f2a4ac5982cc0a6f7b9b/indexer-install/screenshots/status.png -------------------------------------------------------------------------------- /indexer-install/screenshots/swarmpit.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/pi0neerpat/indexer-docker-compose/7aaaa53170c2e20e49a6f2a4ac5982cc0a6f7b9b/indexer-install/screenshots/swarmpit.png -------------------------------------------------------------------------------- /indexer-install/screenshots/traefik.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/pi0neerpat/indexer-docker-compose/7aaaa53170c2e20e49a6f2a4ac5982cc0a6f7b9b/indexer-install/screenshots/traefik.png -------------------------------------------------------------------------------- /indexer-install/scripts/0.0-run-docker-compose-viz.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | docker run --rm -it --name dcv -v $(pwd):/input pmsipilot/docker-compose-viz render -m image ./docker-compose.yml 4 | -------------------------------------------------------------------------------- /indexer-install/scripts/1.1-subgraph-create-jannis-gravity.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | http post localhost:8020 \ 4 | jsonrpc="2.0" \ 5 | id="1" \ 6 | method="subgraph_create" \ 7 | params:='{"name": "jannis/gravity"}' -------------------------------------------------------------------------------- /indexer-install/scripts/1.1.out: -------------------------------------------------------------------------------- 1 | HTTP/1.1 200 OK 2 | content-length: 78 3 | content-type: application/json; charset=utf-8 4 | date: Fri, 21 Aug 2020 09:31:40 GMT 5 | 6 | { 7 | "id": "1", 8 | "jsonrpc": "2.0", 9 | "result": { 10 | "id": "3b9cf00d5945a6afcd7604ad79298c7f" 11 | } 12 | } -------------------------------------------------------------------------------- /indexer-install/scripts/1.2-subgraph-deploy-jannis-gravity.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | http post localhost:8020 \ 4 | jsonrpc="2.0" \ 5 | id="1" \ 6 | method="subgraph_deploy" \ 7 | params:='{"name": "jannis/gravity", "ipfs_hash": "QmbeDC4G8iPAUJ6tRBu99vwyYkaSiFwtXWKwwYkoNphV4X"}' -------------------------------------------------------------------------------- /indexer-install/scripts/1.2.out: -------------------------------------------------------------------------------- 1 | HTTP/1.1 200 OK 2 | content-length: 199 3 | content-type: application/json; charset=utf-8 4 | date: Fri, 21 Aug 2020 09:32:24 GMT 5 | 6 | { 7 | "id": "1", 8 | "jsonrpc": "2.0", 9 | "result": { 10 | "playground": ":8000/subgraphs/name/jannis/gravity/graphql", 11 | "queries": ":8000/subgraphs/name/jannis/gravity", 12 | "subscriptions": ":8001/subgraphs/name/jannis/gravity" 13 | } 14 | } -------------------------------------------------------------------------------- /indexer-install/scripts/2.1-subgraph-create-molochventures-moloch.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | http post localhost:8120 \ 4 | jsonrpc="2.0" \ 5 | id="1" \ 6 | method="subgraph_create" \ 7 | params:='{"name": "molochventures/moloch"}' -------------------------------------------------------------------------------- /indexer-install/scripts/2.1.out: -------------------------------------------------------------------------------- 1 | HTTP/1.1 200 OK 2 | content-length: 78 3 | content-type: application/json; charset=utf-8 4 | date: Fri, 21 Aug 2020 12:32:24 GMT 5 | 6 | { 7 | "id": "1", 8 | "jsonrpc": "2.0", 9 | "result": { 10 | "id": "a060b9917b0c41e13c12ca49e99632be" 11 | } 12 | } -------------------------------------------------------------------------------- /indexer-install/scripts/2.2-subgraph-deploy-molochventures-moloch.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | http post localhost:8120 \ 4 | jsonrpc="2.0" \ 5 | id="1" \ 6 | method="subgraph_deploy" \ 7 | params:='{"name": "molochventures/moloch", "ipfs_hash": "QmTXzATwNfgGVukV1fX2T6xw9f6LAYRVWpsdXyRWzUR2H9"}' -------------------------------------------------------------------------------- /indexer-install/scripts/2.2.out: -------------------------------------------------------------------------------- 1 | HTTP/1.1 200 OK 2 | content-length: 78 3 | content-type: application/json; charset=utf-8 4 | date: Fri, 21 Aug 2020 12:32:24 GMT 5 | 6 | { 7 | "id": "1", 8 | "jsonrpc": "2.0", 9 | "result": { 10 | "id": "a060b9917b0c41e13c12ca49e99632be" 11 | } 12 | } -------------------------------------------------------------------------------- /indexer-install/scripts/3.1-subgraph-create-uniswap-uniswap-v2.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | http post localhost:8120 \ 4 | jsonrpc="2.0" \ 5 | id="1" \ 6 | method="subgraph_create" \ 7 | params:='{"name": "uniswap/uniswap-v2"}' -------------------------------------------------------------------------------- /indexer-install/scripts/3.1.out: -------------------------------------------------------------------------------- 1 | HTTP/1.1 200 OK 2 | content-length: 78 3 | content-type: application/json; charset=utf-8 4 | date: Fri, 21 Aug 2020 12:38:04 GMT 5 | 6 | { 7 | "id": "1", 8 | "jsonrpc": "2.0", 9 | "result": { 10 | "id": "ac4ca588d6461aef3fdb509f0925abc3" 11 | } 12 | } -------------------------------------------------------------------------------- /indexer-install/scripts/3.2-subgraph-deploy-uniswap-uniswap-v2.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | http post localhost:8120 \ 4 | jsonrpc="2.0" \ 5 | id="1" \ 6 | method="subgraph_deploy" \ 7 | params:='{"name": "uniswap/uniswap-v2", "ipfs_hash": "QmXKwSEMirgWVn41nRzkT3hpUBw29cp619Gx58XW6mPhZP"}' -------------------------------------------------------------------------------- /indexer-install/scripts/3.2.out: -------------------------------------------------------------------------------- 1 | HTTP/1.1 200 OK 2 | content-length: 211 3 | content-type: application/json; charset=utf-8 4 | date: Fri, 21 Aug 2020 12:38:46 GMT 5 | 6 | { 7 | "id": "1", 8 | "jsonrpc": "2.0", 9 | "result": { 10 | "playground": ":8000/subgraphs/name/uniswap/uniswap-v2/graphql", 11 | "queries": ":8000/subgraphs/name/uniswap/uniswap-v2", 12 | "subscriptions": ":8001/subgraphs/name/uniswap/uniswap-v2" 13 | } 14 | } -------------------------------------------------------------------------------- /indexer-install/scripts/4.1-subgraph-create-synthetixio-team-synthetix.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | http post localhost:8120 \ 4 | jsonrpc="2.0" \ 5 | id="1" \ 6 | method="subgraph_create" \ 7 | params:='{"name": "synthetixio-team/synthetix"}' -------------------------------------------------------------------------------- /indexer-install/scripts/4.1.out: -------------------------------------------------------------------------------- 1 | HTTP/1.1 200 OK 2 | content-length: 78 3 | content-type: application/json; charset=utf-8 4 | date: Fri, 21 Aug 2020 12:43:17 GMT 5 | 6 | { 7 | "id": "1", 8 | "jsonrpc": "2.0", 9 | "result": { 10 | "id": "eb7939554e975b27971d8a721200f9ee" 11 | } 12 | } -------------------------------------------------------------------------------- /indexer-install/scripts/4.2-subgraph-deploy-synthetixio-team-synthetix.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | http post localhost:8120 \ 4 | jsonrpc="2.0" \ 5 | id="1" \ 6 | method="subgraph_deploy" \ 7 | params:='{"name": "synthetixio-team/synthetix", "ipfs_hash": "Qme2hDXrkBpuXAYEuwGPAjr6zwiMZV4FHLLBa3BHzatBWx"}' -------------------------------------------------------------------------------- /indexer-install/scripts/4.2.out: -------------------------------------------------------------------------------- 1 | HTTP/1.1 200 OK 2 | content-length: 235 3 | content-type: application/json; charset=utf-8 4 | date: Fri, 21 Aug 2020 12:43:44 GMT 5 | 6 | { 7 | "id": "1", 8 | "jsonrpc": "2.0", 9 | "result": { 10 | "playground": ":8000/subgraphs/name/synthetixio-team/synthetix/graphql", 11 | "queries": ":8000/subgraphs/name/synthetixio-team/synthetix", 12 | "subscriptions": ":8001/subgraphs/name/synthetixio-team/synthetix" 13 | } 14 | } -------------------------------------------------------------------------------- /indexer-install/scripts/5.1-health-check-indexer-node.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | http post :8030/graphql query='{ indexingStatuses { subgraph node synced fatalError { message } health chains { $ 4 | -------------------------------------------------------------------------------- /indexer-install/scripts/5.1.out: -------------------------------------------------------------------------------- 1 | HTTP/1.1 200 OK 2 | access-control-allow-headers: Content-Type, User-Agent 3 | access-control-allow-methods: GET, OPTIONS, POST 4 | access-control-allow-origin: * 5 | content-length: 757 6 | content-type: application/json 7 | date: Thu, 27 Aug 2020 04:24:11 GMT 8 | 9 | { 10 | "data": { 11 | "indexingStatuses": [ 12 | { 13 | "chains": [ 14 | { 15 | "latestBlock": { 16 | "number": "10703663" 17 | } 18 | } 19 | ], 20 | "fatalError": null, 21 | "health": "healthy", 22 | "node": "default", 23 | "subgraph": "QmbeDC4G8iPAUJ6tRBu99vwyYkaSiFwtXWKwwYkoNphV4X", 24 | "synced": true 25 | }, 26 | { 27 | "chains": [ 28 | { 29 | "latestBlock": { 30 | "number": "7539663" 31 | } 32 | } 33 | ], 34 | "fatalError": null, 35 | "health": "healthy", 36 | "node": "index_node", 37 | "subgraph": "Qme2hDXrkBpuXAYEuwGPAjr6zwiMZV4FHLLBa3BHzatBWx", 38 | "synced": false 39 | }, 40 | { 41 | "chains": [ 42 | { 43 | "latestBlock": { 44 | "number": "10703663" 45 | } 46 | } 47 | ], 48 | "fatalError": null, 49 | "health": "healthy", 50 | "node": "index_node", 51 | "subgraph": "QmTXzATwNfgGVukV1fX2T6xw9f6LAYRVWpsdXyRWzUR2H9", 52 | "synced": true 53 | }, 54 | { 55 | "chains": [ 56 | { 57 | "latestBlock": { 58 | "number": "10488047" 59 | } 60 | } 61 | ], 62 | "fatalError": null, 63 | "health": "healthy", 64 | "node": "index_node", 65 | "subgraph": "QmXKwSEMirgWVn41nRzkT3hpUBw29cp619Gx58XW6mPhZP", 66 | "synced": false 67 | } 68 | ] 69 | } 70 | } -------------------------------------------------------------------------------- /indexer-install/scripts/6.1-docker-build-indexer-service.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | echo "NPM_TOKEN: $NPM_TOKEN" 4 | 5 | # Indexer service 6 | docker build \ 7 | --build-arg NPM_TOKEN=$NPM_TOKEN \ 8 | -f Dockerfile.indexer-service \ 9 | -t indexer-service:latest \ 10 | . -------------------------------------------------------------------------------- /indexer-install/scripts/7.1-docker-build-indexer-agent.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | echo "NPM_TOKEN: $NPM_TOKEN" 4 | 5 | # Indexer agent 6 | docker build \ 7 | --build-arg NPM_TOKEN=$NPM_TOKEN \ 8 | -f Dockerfile.indexer-agent \ 9 | -t indexer-agent:latest \ 10 | . -------------------------------------------------------------------------------- /indexer-install/scripts/readme.md: -------------------------------------------------------------------------------- 1 | The convenience scripts and documentation here was contributed by @pkrasam 2 | 3 | ## Indexer Management GF_SERVER_ROOT_URL 4 | 5 | port `10000` 6 | 7 | - Checks that you have 1,000 Graph tokens 8 | 9 | - Continuous checks what subgraphs are deployed, and checks whether they are worth indexing. It will allocate a portion of your stake towards the "interesting" ones. 10 | - Every allocation to a certain subgraph has a limited lifetime. Eventually the feels will go on-chain and be distributed. 11 | 12 | ## Commands 13 | 14 | connect to local indexer mgmt api. can install CLI on server 15 | 16 | - install on machine, with port forwarding (ssh -L) 17 | 18 | Check that endpoints 19 | 20 | ```bash 21 | graph indexer status 22 | # Report may not be correct 23 | ``` 24 | 25 | Check that your endpoints are correct: 26 | 27 | > main "service": For performing queries 28 | > Block processing "status":Checks health of subgraphs 29 | > State "channels": For payments 30 | 31 | ## Indexing Rules 32 | 33 | ```bash 34 | graph indexer --help 35 | ``` 36 | 37 | -`indexer rules set` : set and change rules -`indexer rules start (always)` : always index a subgraph, regardless of parameters 38 | 39 | Check current rules with 40 | 41 | ```bash 42 | indexer rules get [global/all] 43 | ``` 44 | 45 | Set a basic rule 46 | 47 | ```bash 48 | graph indexer set global minAverageQueryFees 10000 49 | ``` 50 | 51 | ```bash 52 | yarn add b258 53 | ``` 54 | -------------------------------------------------------------------------------- /indexer-install/swarmpit.yml: -------------------------------------------------------------------------------- 1 | version: "3.3" 2 | 3 | services: 4 | app: 5 | image: swarmpit/swarmpit:latest 6 | environment: 7 | - SWARMPIT_DB=http://db:5984 8 | - SWARMPIT_INFLUXDB=http://influxdb:8086 9 | volumes: 10 | - /var/run/docker.sock:/var/run/docker.sock:ro 11 | networks: 12 | - net 13 | - traefik-public 14 | deploy: 15 | resources: 16 | limits: 17 | cpus: "0.50" 18 | memory: 1024M 19 | reservations: 20 | cpus: "0.25" 21 | memory: 512M 22 | placement: 23 | constraints: 24 | - node.role == manager 25 | labels: 26 | - traefik.enable=true 27 | - traefik.docker.network=traefik-public 28 | - traefik.constraint-label=traefik-public 29 | - traefik.http.routers.swarmpit-http.rule=Host(`${DOMAIN?Variable not set}`) 30 | - traefik.http.routers.swarmpit-http.entrypoints=http 31 | - traefik.http.routers.swarmpit-http.middlewares=https-redirect 32 | - traefik.http.routers.swarmpit-https.rule=Host(`${DOMAIN?Variable not set}`) 33 | - traefik.http.routers.swarmpit-https.entrypoints=https 34 | - traefik.http.routers.swarmpit-https.tls=true 35 | - traefik.http.routers.swarmpit-https.tls.certresolver=le 36 | - traefik.http.services.swarmpit.loadbalancer.server.port=8080 37 | 38 | db: 39 | image: couchdb:2.3.0 40 | volumes: 41 | - /data/admin/swarmpit/db-data:/opt/couchdb/data 42 | networks: 43 | - net 44 | deploy: 45 | resources: 46 | limits: 47 | cpus: "0.30" 48 | memory: 512M 49 | reservations: 50 | cpus: "0.15" 51 | memory: 256M 52 | placement: 53 | constraints: 54 | - node.labels.swarmpit.db-data == true 55 | influxdb: 56 | image: influxdb:1.7 57 | volumes: 58 | - /data/admin/swarmpit/influx-data:/var/lib/influxdb 59 | networks: 60 | - net 61 | deploy: 62 | resources: 63 | reservations: 64 | cpus: "0.3" 65 | memory: 128M 66 | limits: 67 | cpus: "0.6" 68 | memory: 512M 69 | placement: 70 | constraints: 71 | - node.labels.swarmpit.influx-data == true 72 | agent: 73 | image: swarmpit/agent:latest 74 | environment: 75 | - DOCKER_API_VERSION=1.35 76 | volumes: 77 | - /var/run/docker.sock:/var/run/docker.sock:ro 78 | networks: 79 | - net 80 | deploy: 81 | mode: global 82 | resources: 83 | limits: 84 | cpus: "0.10" 85 | memory: 64M 86 | reservations: 87 | cpus: "0.05" 88 | memory: 32M 89 | 90 | networks: 91 | net: 92 | driver: overlay 93 | attachable: true 94 | traefik-public: 95 | external: true 96 | 97 | volumes: 98 | db-data: 99 | driver: local 100 | influx-data: 101 | driver: local 102 | -------------------------------------------------------------------------------- /indexer-install/traefik.yml: -------------------------------------------------------------------------------- 1 | version: '3.3' 2 | 3 | services: 4 | 5 | traefik: 6 | # Use the latest v2.2.x Traefik image available 7 | image: traefik:v2.2 8 | ports: 9 | # Listen on port 80, default for HTTP, necessary to redirect to HTTPS 10 | - 80:80 11 | # Listen on port 443, default for HTTPS 12 | - 443:443 13 | deploy: 14 | placement: 15 | constraints: 16 | # Make the traefik service run only on the node with this label 17 | # as the node with it has the volume for the certificates 18 | - node.labels.traefik-public.traefik-public-certificates == true 19 | labels: 20 | # Enable Traefik for this service, to make it available in the public network 21 | - traefik.enable=true 22 | # Use the traefik-public network (declared below) 23 | - traefik.docker.network=traefik-public 24 | # Use the custom label "traefik.constraint-label=traefik-public" 25 | # This public Traefik will only use services with this label 26 | # That way you can add other internal Traefik instances per stack if needed 27 | - traefik.constraint-label=traefik-public 28 | # admin-auth middleware with HTTP Basic auth 29 | # Using the environment variables USERNAME and HASHED_PASSWORD 30 | - traefik.http.middlewares.admin-auth.basicauth.users=${USERNAME?Variable not set}:${HASHED_PASSWORD?Variable not set} 31 | # https-redirect middleware to redirect HTTP to HTTPS 32 | # It can be re-used by other stacks in other Docker Compose files 33 | - traefik.http.middlewares.https-redirect.redirectscheme.scheme=https 34 | - traefik.http.middlewares.https-redirect.redirectscheme.permanent=true 35 | # traefik-http set up only to use the middleware to redirect to https 36 | # Uses the environment variable DOMAIN 37 | - traefik.http.routers.traefik-public-http.rule=Host(`${DOMAIN?Variable not set}`) 38 | - traefik.http.routers.traefik-public-http.entrypoints=http 39 | - traefik.http.routers.traefik-public-http.middlewares=https-redirect 40 | # traefik-https the actual router using HTTPS 41 | # Uses the environment variable DOMAIN 42 | - traefik.http.routers.traefik-public-https.rule=Host(`${DOMAIN?Variable not set}`) 43 | - traefik.http.routers.traefik-public-https.entrypoints=https 44 | - traefik.http.routers.traefik-public-https.tls=true 45 | # Use the special Traefik service api@internal with the web UI/Dashboard 46 | - traefik.http.routers.traefik-public-https.service=api@internal 47 | # Use the "le" (Let's Encrypt) resolver created below 48 | - traefik.http.routers.traefik-public-https.tls.certresolver=le 49 | # Enable HTTP Basic auth, using the middleware created above 50 | - traefik.http.routers.traefik-public-https.middlewares=admin-auth 51 | # Define the port inside of the Docker service to use 52 | - traefik.http.services.traefik-public.loadbalancer.server.port=8080 53 | volumes: 54 | # Add Docker as a mounted volume, so that Traefik can read the labels of other services 55 | - /var/run/docker.sock:/var/run/docker.sock:ro 56 | # Mount the volume to store the certificates 57 | - traefik-public-certificates:/certificates 58 | command: 59 | # Enable Docker in Traefik, so that it reads labels from Docker services 60 | - --providers.docker 61 | # Add a constraint to only use services with the label "traefik.constraint-label=traefik-public" 62 | - --providers.docker.constraints=Label(`traefik.constraint-label`, `traefik-public`) 63 | # Do not expose all Docker services, only the ones explicitly exposed 64 | - --providers.docker.exposedbydefault=false 65 | # Enable Docker Swarm mode 66 | - --providers.docker.swarmmode 67 | # Create an entrypoint "http" listening on port 80 68 | - --entrypoints.http.address=:80 69 | # Create an entrypoint "https" listening on port 443 70 | - --entrypoints.https.address=:443 71 | # Create the certificate resolver "le" for Let's Encrypt, uses the environment variable EMAIL 72 | - --certificatesresolvers.le.acme.email=${EMAIL?Variable not set} 73 | # Store the Let's Encrypt certificates in the mounted volume 74 | - --certificatesresolvers.le.acme.storage=/certificates/acme.json 75 | # Use the TLS Challenge for Let's Encrypt 76 | - --certificatesresolvers.le.acme.tlschallenge=true 77 | # Enable the access log, with HTTP requests 78 | - --accesslog 79 | # Enable the Traefik log, for configurations and errors 80 | - --log 81 | # Enable the Dashboard and API 82 | - --api 83 | networks: 84 | # Use the public network created to be shared between Traefik and 85 | # any other service that needs to be publicly available with HTTPS 86 | - traefik-public 87 | 88 | volumes: 89 | # Create a volume to store the certificates, there is a constraint to make sure 90 | # Traefik is always deployed to the same Docker node with the same volume containing 91 | # the HTTPS certificates 92 | traefik-public-certificates: 93 | 94 | networks: 95 | # Use the previously created public network "traefik-public", shared with other 96 | # services that need to be publicly available via this Traefik 97 | traefik-public: 98 | external: true 99 | -------------------------------------------------------------------------------- /monitoring/README.md: -------------------------------------------------------------------------------- 1 |

Monitoring Infra

2 | 3 | # Table of Contents 4 | 5 | 6 | - [Table of Contents](#table-of-contents) 7 | - [Set up Grafana and Prometheus](#set-up-grafana-and-prometheus) 8 | - [Configure Prometheus.yml](#configure-prometheusyml) 9 | - [Update docker-compose.yml](#update-docker-composeyml) 10 | - [Test it out](#test-it-out) 11 | - [Configure Nginx](#configure-nginx) 12 | - [Create the Grafana dashboards](#create-the-grafana-dashboards) 13 | - [Add Data Sources](#add-data-sources) 14 | - [Add Dashboards](#add-dashboards) 15 | - [Final testing](#final-testing) 16 | - [Next Steps](#next-steps) 17 | 18 | 19 | 20 | # Set up Grafana and Prometheus 21 | 22 | ## Configure Prometheus.yml 23 | 24 | For Prometheus to know where to scrape data from, we need to know the Docker bridge IP. First determine the ID for any `graph-node` container, and then use this command to get the bridge IP. 25 | 26 | ```bash 27 | docker ps 28 | # My Container ID is "5bd...", yours will be different 29 | 30 | docker inspect -f '{{range .NetworkSettings.Networks}}{{.Gateway}}{{end}}' 31 | > 172.17.0.1 32 | ``` 33 | 34 | Now we will update our `prometheus.yml` file which tells Prometheus where to scrape data from. 35 | 36 | ```bash 37 | cd ~/indexer-docker-compose/monitoring 38 | nano prometheus.yml 39 | ``` 40 | 41 | In our case our bridge IP is `172.17.0.1`, so replace this with yours 42 | 43 | ```yaml 44 | # ... 45 | scrape_configs: 46 | - job_name: "thegraph" 47 | static_configs: 48 | - targets: 49 | - 172.17.0.1:8140 # CHANGE ME 50 | - 172.17.0.1:8040 # CHANGE ME 51 | ``` 52 | 53 | Things to pay attention to: 54 | 55 | - Each graph-node exposes metrics for only its own operation. Thus, we need to scrape both the "query-node" `8040` and "index-node" `8140` 56 | 57 | ## Update docker-compose.yml 58 | 59 | Now we need to update the `docker-compose.yml` with our Grafana server name and password. 60 | 61 | ```bash 62 | nano docker-compose.yml 63 | ``` 64 | 65 | ```yaml 66 | services: 67 | grafana: 68 | image: grafana/grafana:6.4.4 69 | # ... 70 | environment: 71 | GF_SECURITY_ADMIN_PASSWORD: admin # CHANGE ME 72 | GF_SERVER_DOMAIN: "grafana.mysite.com" # CHANGE ME 73 | GF_SERVER_ROOT_URL: "https://grafana.mysite.com" # CHANGE ME 74 | ``` 75 | 76 | Finally we create the folder to store Prometheus data, and set the approriate permissions 77 | 78 | ```bash 79 | # Prometheus 80 | mkdir ~/monitoring-data 81 | mkdir ~/monitoring-data/prometheus 82 | sudo chown 65534:65534 $HOME/monitoring-data/prometheus 83 | 84 | # Grafana 85 | sudo chown 472:472 -R /docker/volumes 86 | ``` 87 | 88 | Now we can start running our monitoring services. 89 | 90 | ```bash 91 | docker-compose up -d 92 | ``` 93 | 94 | ## Test it out 95 | 96 | From your local machine, you can try to create a tunnel to port `3000` to see the Grafana dashboard 97 | 98 | ```bash 99 | ssh -L 127.0.0.1:8000:127.0.0.1:3000 user@mysite.com 100 | ``` 101 | 102 | Test that you can see the Prometheus UI as well. 103 | 104 | ```bash 105 | ssh -L 127.0.0.1:8000:127.0.0.1:9090 user@mysite.com 106 | ``` 107 | 108 | # Configure Nginx 109 | 110 | Next we need to permanently expose port `3000` to the world, so let's add a new server file `monitoring.conf` from the `/nginx` folder of this repo. You will need to update the `server_name`. 111 | 112 | ```bash 113 | sudo cp ~/indexer-docker-compose/nginx/monitoring.conf /etc/nginx/sites-enabled 114 | 115 | nano /etc/nginx/sites-enabled/monitoring.conf 116 | # Update server_name 117 | ``` 118 | 119 | Let's put the changes into effect: 120 | 121 | ```bash 122 | sudo nginx -t 123 | sudo systemctl reload nginx 124 | ``` 125 | 126 | Now we need Certbot to issue a certificate. All subdomains (grafana, prometheus, indexer) should get a certificate. 127 | 128 | ```bash 129 | sudo certbot --nginx certonly 130 | sudo systemctl reload nginx 131 | ``` 132 | 133 | You should now be able to access your Grafana at `grafana.mysite.com` 134 | 135 | # Create the Grafana dashboards 136 | 137 | ## Add Data Sources 138 | 139 | We will use the Docker bridge IP again for this step, so that Grafana can access the Prometheus and Postgres containers. 140 | 141 | Navigate to `grafana.mysite.com/datasources`, and use the green button "Add data source" to add a new source. 142 | 143 | ### Postgres 144 | 145 | 1. Optional, but recommended: Create a new postgres user with limited permissions. (You may need to stop your graph nodes first) 146 | 147 | ```bash 148 | # Enter the docker container 149 | docker exec -it bash 150 | 151 | # Switch users 152 | su postgres 153 | 154 | # open postgres cli and create a new user 155 | postgres 156 | CREATE USER grafana WITH PASSWORD 'grafana'; 157 | ``` 158 | 159 | 2. Add the user in the postgres data source 160 | 161 | Should be named (lowercase) "postgres". URL will be your Docker bridge IP on port `5432`, eg. `172.17.0.1:5432` 162 | 163 | ### Prometheus 164 | 165 | Should be named (lowercase) "prometheus". URL will be your Docker bridge IP on port `9090`, eg. `172.17.0.1:9090` 166 | 167 | ## Add Dashboards 168 | 169 | Navigate to `grafana.mysite.com/dashboards`, and use the grey "Import" button to create a new dashboard. 170 | 171 | Create a new dashboard using each of the following JSON files: 172 | 173 | - [indexing.json](./dashboards/indexing.json) 174 | - [metrics.json](./dashboards/metrics.json) 175 | - [postgres.json](./dashboards/postgres.json) 176 | 177 | # Final testing 178 | 179 | Let's check that our Prometheus endpoint is properly exposed. Update with your domain and paste into your browser: 180 | 181 | prometheus.mysite.com/federate?match[]=subgraph_query_execution_time_count&match[]=subgraph_count&match[]=QmXKwSEMirgWVn41nRzkT3hpUBw29cp619Gx58XW6mPhZP_sync_total_secs&match[]=Qme2hDXrkBpuXAYEuwGPAjr6zwiMZV4FHLLBa3BHzatBWx_sync_total_secs&match[]=QmTXzATwNfgGVukV1fX2T6xw9f6LAYRVWpsdXyRWzUR2H9_sync_total_secs 182 | 183 | You should get a response like 184 | 185 | ``` 186 | # TYPE QmTXzATwNfgGVukV1fX2T6xw9f6LAYRVWpsdXyRWzUR2H9_sync_total_secs untyped 187 | QmTXzATwNfgGVukV1fX2T6xw9f6LAYRVWpsdXyRWzUR2H9_sync_total_secs{instance="172.17.0.1:8040",job="thegraph-indexer"} 0 1597782701051 188 | ... 189 | ``` 190 | 191 | # Next Steps 192 | 193 | 🥳 Wow, look how far you made it! You tackled some very impressive problems today. You are an incredible person, and deserve a special treat! Congrats on completing Phase 0. Here we come Phase 1 194 | -------------------------------------------------------------------------------- /monitoring/dashboards/indexing.json: -------------------------------------------------------------------------------- 1 | { 2 | "annotations": { 3 | "list": [ 4 | { 5 | "builtIn": 1, 6 | "datasource": "-- Grafana --", 7 | "enable": true, 8 | "hide": true, 9 | "iconColor": "rgba(0, 211, 255, 1)", 10 | "name": "Annotations & Alerts", 11 | "type": "dashboard" 12 | } 13 | ] 14 | }, 15 | "editable": true, 16 | "gnetId": null, 17 | "graphTooltip": 0, 18 | "id": 4, 19 | "iteration": 1590595997503, 20 | "links": [], 21 | "panels": [ 22 | { 23 | "columns": [], 24 | "datasource": "postgres", 25 | "fieldConfig": { 26 | "defaults": { 27 | "custom": {} 28 | }, 29 | "overrides": [] 30 | }, 31 | "fontSize": "90%", 32 | "gridPos": { 33 | "h": 10, 34 | "w": 24, 35 | "x": 0, 36 | "y": 0 37 | }, 38 | "id": 10, 39 | "links": [], 40 | "pageSize": null, 41 | "scroll": true, 42 | "showHeader": true, 43 | "sort": { 44 | "col": 6, 45 | "desc": true 46 | }, 47 | "styles": [ 48 | { 49 | "alias": "blocks behind", 50 | "align": "auto", 51 | "colorMode": null, 52 | "colors": [ 53 | "rgba(50, 172, 45, 0.97)", 54 | "rgba(237, 129, 40, 0.89)", 55 | "rgba(245, 54, 54, 0.9)" 56 | ], 57 | "dateFormat": "YYYY-MM-DD HH:mm:ss", 58 | "decimals": 0, 59 | "mappingType": 1, 60 | "pattern": "blocks_behind_network", 61 | "thresholds": [ 62 | "" 63 | ], 64 | "type": "number", 65 | "unit": "none" 66 | }, 67 | { 68 | "alias": "", 69 | "align": "auto", 70 | "colorMode": null, 71 | "colors": [ 72 | "rgba(245, 54, 54, 0.9)", 73 | "rgba(237, 129, 40, 0.89)", 74 | "rgba(50, 172, 45, 0.97)" 75 | ], 76 | "decimals": 1, 77 | "pattern": "lag", 78 | "thresholds": [], 79 | "type": "number", 80 | "unit": "m" 81 | }, 82 | { 83 | "alias": "syn", 84 | "align": "auto", 85 | "colorMode": null, 86 | "colors": [ 87 | "rgba(245, 54, 54, 0.9)", 88 | "rgba(237, 129, 40, 0.89)", 89 | "rgba(50, 172, 45, 0.97)" 90 | ], 91 | "dateFormat": "YYYY-MM-DD HH:mm:ss", 92 | "decimals": 2, 93 | "mappingType": 1, 94 | "pattern": "synced", 95 | "thresholds": [], 96 | "type": "string", 97 | "unit": "short", 98 | "valueMaps": [ 99 | { 100 | "text": "✓", 101 | "value": "true" 102 | }, 103 | { 104 | "text": "—", 105 | "value": "false" 106 | } 107 | ] 108 | } 109 | ], 110 | "targets": [ 111 | { 112 | "format": "table", 113 | "group": [], 114 | "metricColumn": "none", 115 | "rawQuery": true, 116 | "rawSql": "-- grafana ignore\nselect\n g.name || (case\n when g.pending_version = v.id then ' (P)'\n when g.current_version = v.id then ' (C)'\n else ' (U)' end) as subgraph_name,\n d.id as deployment,\n s.id as schema,\n replace(a.node_id, 'index_node_','') as nodeId,\n d.synced::text,\n n.name as network,\n (n.head_block_number - d.latest_ethereum_block_number) as blocks_behind_network,\n (case when n.name = 'mainnet' then ((n.head_block_number - d.latest_ethereum_block_number)/4)::text\n when n.name = 'rinkeby' then ((n.head_block_number - d.latest_ethereum_block_number)/4)::text\n when n.name = 'kovan' then ((n.head_block_number - d.latest_ethereum_block_number)/15)::text\n when n.name = 'poa-core' then ((n.head_block_number - d.latest_ethereum_block_number)/12)::text\n else 'ø' end) as lag\nfrom subgraphs.subgraph_deployment as d,\n subgraphs.subgraph_deployment_assignment as a,\n subgraphs.subgraph_version as v,\n subgraphs.subgraph as g,\n subgraphs.ethereum_contract_data_source as ds,\n subgraphs.subgraph_manifest m,\n ethereum_networks as n,\n deployment_schemas as s\nwhere g.id = v.subgraph\n and v.id in (g.pending_version, g.current_version)\n and a.id = d.id\n and m.id = d.manifest \n and ds.id = m.data_sources[1]\n and s.subgraph = d.id\n and v.deployment = d.id\n and not d.failed\n and n.name = ds.network\norder by blocks_behind_network desc, subgraph_name;", 117 | "refId": "A", 118 | "select": [ 119 | [ 120 | { 121 | "params": [ 122 | "value" 123 | ], 124 | "type": "column" 125 | } 126 | ] 127 | ], 128 | "timeColumn": "time", 129 | "where": [ 130 | { 131 | "name": "$__timeFilter", 132 | "params": [], 133 | "type": "macro" 134 | } 135 | ] 136 | } 137 | ], 138 | "timeFrom": null, 139 | "timeShift": null, 140 | "title": "Subgraph Block vs. Network Block", 141 | "transform": "table", 142 | "type": "table" 143 | }, 144 | { 145 | "columns": [], 146 | "datasource": "postgres", 147 | "fieldConfig": { 148 | "defaults": { 149 | "custom": {} 150 | }, 151 | "overrides": [] 152 | }, 153 | "fontSize": "100%", 154 | "gridPos": { 155 | "h": 8, 156 | "w": 24, 157 | "x": 0, 158 | "y": 10 159 | }, 160 | "id": 33, 161 | "pageSize": null, 162 | "showHeader": true, 163 | "sort": { 164 | "col": 0, 165 | "desc": true 166 | }, 167 | "styles": [ 168 | { 169 | "alias": "", 170 | "align": "auto", 171 | "dateFormat": "YYYY-MM-DD HH:mm:ss", 172 | "decimals": null, 173 | "pattern": "/vid|block_number/", 174 | "type": "number", 175 | "unit": "none" 176 | }, 177 | { 178 | "alias": "", 179 | "align": "right", 180 | "colorMode": null, 181 | "colors": [ 182 | "rgba(245, 54, 54, 0.9)", 183 | "rgba(237, 129, 40, 0.89)", 184 | "rgba(50, 172, 45, 0.97)" 185 | ], 186 | "decimals": 2, 187 | "pattern": "/.*/", 188 | "thresholds": [], 189 | "type": "number", 190 | "unit": "none" 191 | } 192 | ], 193 | "targets": [ 194 | { 195 | "format": "table", 196 | "group": [], 197 | "metricColumn": "none", 198 | "rawQuery": true, 199 | "rawSql": "SELECT\n vid, block_number, subgraph_id, message, handler\nFROM subgraphs.subgraph_error\nORDER BY vid DESC", 200 | "refId": "A", 201 | "select": [ 202 | [ 203 | { 204 | "params": [ 205 | "value" 206 | ], 207 | "type": "column" 208 | } 209 | ] 210 | ], 211 | "table": "subgraphs.subgraph_error", 212 | "timeColumn": "time", 213 | "where": [] 214 | } 215 | ], 216 | "timeFrom": null, 217 | "timeShift": null, 218 | "title": "Fatal errors", 219 | "transform": "table", 220 | "type": "table" 221 | }, 222 | { 223 | "columns": [], 224 | "datasource": "postgres", 225 | "fieldConfig": { 226 | "defaults": { 227 | "custom": {} 228 | }, 229 | "overrides": [] 230 | }, 231 | "fontSize": "100%", 232 | "gridPos": { 233 | "h": 7, 234 | "w": 24, 235 | "x": 0, 236 | "y": 18 237 | }, 238 | "id": 6, 239 | "links": [], 240 | "pageSize": null, 241 | "scroll": true, 242 | "showHeader": true, 243 | "sort": { 244 | "col": 1, 245 | "desc": true 246 | }, 247 | "styles": [ 248 | { 249 | "alias": "", 250 | "align": "auto", 251 | "colorMode": null, 252 | "colors": [ 253 | "rgba(245, 54, 54, 0.9)", 254 | "rgba(237, 129, 40, 0.89)", 255 | "rgba(50, 172, 45, 0.97)" 256 | ], 257 | "dateFormat": "YYYY-MM-DD HH:mm:ss", 258 | "decimals": 0, 259 | "mappingType": 1, 260 | "pattern": "num_assigned", 261 | "thresholds": [], 262 | "type": "number", 263 | "unit": "short" 264 | } 265 | ], 266 | "targets": [ 267 | { 268 | "format": "table", 269 | "group": [], 270 | "metricColumn": "none", 271 | "rawQuery": true, 272 | "rawSql": "select replace(node_id, 'index_node_', '') as nodeId,\n count(*) as num_assigned,\n array_agg(id) as deployments\n from subgraphs.subgraph_deployment_assignment\n group by node_id\n order by node_id asc;\n", 273 | "refId": "A", 274 | "select": [ 275 | [ 276 | { 277 | "params": [ 278 | "value" 279 | ], 280 | "type": "column" 281 | } 282 | ] 283 | ], 284 | "timeColumn": "time", 285 | "where": [ 286 | { 287 | "name": "$__timeFilter", 288 | "params": [], 289 | "type": "macro" 290 | } 291 | ] 292 | } 293 | ], 294 | "timeFrom": null, 295 | "timeShift": null, 296 | "title": "Subgraph Node Assignments", 297 | "transform": "table", 298 | "type": "table" 299 | }, 300 | { 301 | "cacheTimeout": null, 302 | "datasource": "postgres", 303 | "fieldConfig": { 304 | "defaults": { 305 | "custom": { 306 | "align": null 307 | }, 308 | "mappings": [], 309 | "nullValueMode": "connected", 310 | "thresholds": { 311 | "mode": "absolute", 312 | "steps": [ 313 | { 314 | "color": "green", 315 | "value": null 316 | }, 317 | { 318 | "color": "red", 319 | "value": 80 320 | } 321 | ] 322 | }, 323 | "unit": "none" 324 | }, 325 | "overrides": [ 326 | { 327 | "matcher": { 328 | "id": "byName", 329 | "options": "name" 330 | }, 331 | "properties": [ 332 | { 333 | "id": "displayName", 334 | "value": "Network name" 335 | } 336 | ] 337 | }, 338 | { 339 | "matcher": { 340 | "id": "byName", 341 | "options": "block_number" 342 | }, 343 | "properties": [ 344 | { 345 | "id": "displayName", 346 | "value": "Block number" 347 | } 348 | ] 349 | }, 350 | { 351 | "matcher": { 352 | "id": "byName", 353 | "options": "block_hash" 354 | }, 355 | "properties": [ 356 | { 357 | "id": "displayName", 358 | "value": "Block Hash" 359 | } 360 | ] 361 | }, 362 | { 363 | "matcher": { 364 | "id": "byName", 365 | "options": "Block number" 366 | }, 367 | "properties": [ 368 | { 369 | "id": "custom.width", 370 | "value": 101 371 | } 372 | ] 373 | }, 374 | { 375 | "matcher": { 376 | "id": "byName", 377 | "options": "Network name" 378 | }, 379 | "properties": [ 380 | { 381 | "id": "custom.width", 382 | "value": 116 383 | } 384 | ] 385 | }, 386 | { 387 | "matcher": { 388 | "id": "byName", 389 | "options": "Block Hash" 390 | }, 391 | "properties": [ 392 | { 393 | "id": "custom.width", 394 | "value": 396 395 | } 396 | ] 397 | } 398 | ] 399 | }, 400 | "gridPos": { 401 | "h": 8, 402 | "w": 12, 403 | "x": 0, 404 | "y": 25 405 | }, 406 | "id": 2, 407 | "interval": "10s", 408 | "links": [], 409 | "maxDataPoints": 100, 410 | "options": { 411 | "showHeader": true, 412 | "sortBy": [] 413 | }, 414 | "pluginVersion": "7.0.1", 415 | "targets": [ 416 | { 417 | "format": "table", 418 | "group": [], 419 | "metricColumn": "none", 420 | "rawQuery": true, 421 | "rawSql": "SELECT\n name,\n head_block_number AS \"block_number\",\n head_block_hash AS \"block_hash\"\nFROM ethereum_networks\nORDER BY name;", 422 | "refId": "A", 423 | "select": [ 424 | [ 425 | { 426 | "params": [ 427 | "value" 428 | ], 429 | "type": "column" 430 | } 431 | ] 432 | ], 433 | "table": "ethereum_networks", 434 | "timeColumn": "time", 435 | "where": [] 436 | } 437 | ], 438 | "timeFrom": null, 439 | "timeShift": null, 440 | "title": "Network Head Blocks", 441 | "type": "table" 442 | }, 443 | { 444 | "alert": { 445 | "alertRuleTags": {}, 446 | "conditions": [ 447 | { 448 | "evaluator": { 449 | "params": [ 450 | 0 451 | ], 452 | "type": "gt" 453 | }, 454 | "operator": { 455 | "type": "and" 456 | }, 457 | "query": { 458 | "params": [ 459 | "A", 460 | "3m", 461 | "now" 462 | ] 463 | }, 464 | "reducer": { 465 | "params": [], 466 | "type": "avg" 467 | }, 468 | "type": "query" 469 | } 470 | ], 471 | "executionErrorState": "alerting", 472 | "for": "5m", 473 | "frequency": "1m", 474 | "handler": 1, 475 | "name": "Deployed subgraphs that have not started alert", 476 | "noDataState": "no_data", 477 | "notifications": [ 478 | { 479 | "uid": "okc8ZyRZz" 480 | } 481 | ] 482 | }, 483 | "aliasColors": {}, 484 | "bars": false, 485 | "dashLength": 10, 486 | "dashes": false, 487 | "datasource": "postgres", 488 | "fieldConfig": { 489 | "defaults": { 490 | "custom": {} 491 | }, 492 | "overrides": [] 493 | }, 494 | "fill": 1, 495 | "fillGradient": 0, 496 | "gridPos": { 497 | "h": 8, 498 | "w": 12, 499 | "x": 12, 500 | "y": 25 501 | }, 502 | "hiddenSeries": false, 503 | "id": 29, 504 | "legend": { 505 | "avg": false, 506 | "current": false, 507 | "max": false, 508 | "min": false, 509 | "show": true, 510 | "total": false, 511 | "values": false 512 | }, 513 | "lines": true, 514 | "linewidth": 1, 515 | "nullPointMode": "null", 516 | "options": { 517 | "dataLinks": [] 518 | }, 519 | "percentage": false, 520 | "pointradius": 2, 521 | "points": false, 522 | "renderer": "flot", 523 | "seriesOverrides": [], 524 | "spaceLength": 10, 525 | "stack": false, 526 | "steppedLine": false, 527 | "targets": [ 528 | { 529 | "format": "time_series", 530 | "group": [], 531 | "metricColumn": "none", 532 | "rawQuery": true, 533 | "rawSql": "select now() as time, count(*) as num_subgraphs\n from (\n select d.id, d.latest_ethereum_block_number block\n from subgraphs.subgraph_deployment d,\n subgraphs.subgraph_deployment_assignment a\n where d.id = a.id\n and not d.failed) a\n where a.block < 100;\n", 534 | "refId": "A", 535 | "select": [ 536 | [ 537 | { 538 | "params": [ 539 | "value" 540 | ], 541 | "type": "column" 542 | } 543 | ] 544 | ], 545 | "timeColumn": "time", 546 | "where": [ 547 | { 548 | "name": "$__timeFilter", 549 | "params": [], 550 | "type": "macro" 551 | } 552 | ] 553 | } 554 | ], 555 | "thresholds": [ 556 | { 557 | "colorMode": "critical", 558 | "fill": true, 559 | "line": true, 560 | "op": "gt", 561 | "value": 0 562 | } 563 | ], 564 | "timeFrom": null, 565 | "timeRegions": [], 566 | "timeShift": null, 567 | "title": "Deployed subgraphs that have not started", 568 | "tooltip": { 569 | "shared": true, 570 | "sort": 0, 571 | "value_type": "individual" 572 | }, 573 | "type": "graph", 574 | "xaxis": { 575 | "buckets": null, 576 | "mode": "time", 577 | "name": null, 578 | "show": true, 579 | "values": [] 580 | }, 581 | "yaxes": [ 582 | { 583 | "format": "short", 584 | "label": null, 585 | "logBase": 1, 586 | "max": null, 587 | "min": null, 588 | "show": true 589 | }, 590 | { 591 | "format": "short", 592 | "label": null, 593 | "logBase": 1, 594 | "max": null, 595 | "min": null, 596 | "show": true 597 | } 598 | ], 599 | "yaxis": { 600 | "align": false, 601 | "alignLevel": null 602 | } 603 | } 604 | ], 605 | "refresh": "", 606 | "schemaVersion": 25, 607 | "style": "dark", 608 | "tags": [], 609 | "templating": { 610 | "list": [ 611 | { 612 | "datasource": "Elasticsearch", 613 | "filters": [], 614 | "hide": 0, 615 | "label": "", 616 | "name": "Filters", 617 | "skipUrlSync": false, 618 | "type": "adhoc" 619 | } 620 | ] 621 | }, 622 | "time": { 623 | "from": "now-6h", 624 | "to": "now" 625 | }, 626 | "timepicker": { 627 | "refresh_intervals": [ 628 | "10s", 629 | "30s", 630 | "1m", 631 | "5m", 632 | "15m", 633 | "30m", 634 | "1h", 635 | "2h", 636 | "1d" 637 | ], 638 | "time_options": [ 639 | "5m", 640 | "15m", 641 | "1h", 642 | "6h", 643 | "12h", 644 | "24h", 645 | "2d", 646 | "7d", 647 | "30d" 648 | ] 649 | }, 650 | "timezone": "", 651 | "title": "Indexing Status", 652 | "uid": "7rcuDImZk", 653 | "version": 4 654 | } 655 | -------------------------------------------------------------------------------- /monitoring/dashboards/postgres.json: -------------------------------------------------------------------------------- 1 | 2 | { 3 | "annotations": { 4 | "list": [ 5 | { 6 | "builtIn": 1, 7 | "datasource": "-- Grafana --", 8 | "enable": true, 9 | "hide": true, 10 | "iconColor": "rgba(0, 211, 255, 1)", 11 | "name": "Annotations & Alerts", 12 | "type": "dashboard" 13 | } 14 | ] 15 | }, 16 | "editable": true, 17 | "gnetId": null, 18 | "graphTooltip": 0, 19 | "id": 3, 20 | "links": [], 21 | "panels": [ 22 | { 23 | "columns": [], 24 | "datasource": "postgres", 25 | "fontSize": "100%", 26 | "gridPos": { 27 | "h": 8, 28 | "w": 24, 29 | "x": 0, 30 | "y": 0 31 | }, 32 | "id": 4, 33 | "links": [], 34 | "pageSize": null, 35 | "scroll": true, 36 | "showHeader": true, 37 | "sort": { 38 | "col": 1, 39 | "desc": true 40 | }, 41 | "styles": [ 42 | { 43 | "alias": "Age", 44 | "align": "auto", 45 | "dateFormat": "YYYY-MM-DD HH:mm:ss", 46 | "pattern": "age", 47 | "type": "number", 48 | "unit": "s" 49 | }, 50 | { 51 | "alias": "", 52 | "align": "auto", 53 | "colorMode": null, 54 | "colors": [ 55 | "rgba(245, 54, 54, 0.9)", 56 | "rgba(237, 129, 40, 0.89)", 57 | "rgba(50, 172, 45, 0.97)" 58 | ], 59 | "decimals": 2, 60 | "pattern": "/.*/", 61 | "thresholds": [], 62 | "type": "number", 63 | "unit": "short" 64 | } 65 | ], 66 | "targets": [ 67 | { 68 | "format": "table", 69 | "group": [], 70 | "metricColumn": "none", 71 | "rawQuery": true, 72 | "rawSql": "-- grafana ignore\nselect application_name,\n extract(epoch from age(now(), xact_start)) as age,\n query from pg_stat_activity\n where query not like '%grafana ignore%'\n and state='active'\norder by query_start desc", 73 | "refId": "A", 74 | "select": [ 75 | [ 76 | { 77 | "params": [ 78 | "value" 79 | ], 80 | "type": "column" 81 | } 82 | ] 83 | ], 84 | "timeColumn": "time", 85 | "where": [ 86 | { 87 | "name": "$__timeFilter", 88 | "params": [], 89 | "type": "macro" 90 | } 91 | ] 92 | } 93 | ], 94 | "timeFrom": null, 95 | "timeShift": null, 96 | "title": "Active Queries", 97 | "transform": "table", 98 | "type": "table" 99 | }, 100 | { 101 | "columns": [], 102 | "datasource": "postgres", 103 | "fontSize": "100%", 104 | "gridPos": { 105 | "h": 5, 106 | "w": 13, 107 | "x": 0, 108 | "y": 8 109 | }, 110 | "id": 9, 111 | "links": [], 112 | "pageSize": null, 113 | "pluginVersion": "6.2.1", 114 | "scroll": true, 115 | "showHeader": true, 116 | "sort": { 117 | "col": 1, 118 | "desc": false 119 | }, 120 | "styles": [ 121 | { 122 | "alias": "", 123 | "align": "auto", 124 | "dateFormat": "YYYY-MM-DD", 125 | "pattern": "last_autovacuum", 126 | "type": "date" 127 | }, 128 | { 129 | "alias": "tx left", 130 | "align": "auto", 131 | "colorMode": "row", 132 | "colors": [ 133 | "rgba(245, 54, 54, 0.9)", 134 | "rgba(237, 129, 40, 0.89)", 135 | "rgba(50, 172, 45, 0.97)" 136 | ], 137 | "decimals": null, 138 | "pattern": "tx_before_wraparound_vacuum", 139 | "thresholds": [ 140 | "20000000", 141 | "40000000" 142 | ], 143 | "type": "number", 144 | "unit": "short" 145 | }, 146 | { 147 | "alias": "", 148 | "align": "auto", 149 | "colorMode": null, 150 | "colors": [ 151 | "rgba(245, 54, 54, 0.9)", 152 | "rgba(237, 129, 40, 0.89)", 153 | "rgba(50, 172, 45, 0.97)" 154 | ], 155 | "dateFormat": "YYYY-MM-DD HH:mm:ss", 156 | "decimals": 2, 157 | "mappingType": 1, 158 | "pattern": "xid_age", 159 | "thresholds": [], 160 | "type": "number", 161 | "unit": "short" 162 | } 163 | ], 164 | "targets": [ 165 | { 166 | "format": "table", 167 | "group": [], 168 | "metricColumn": "none", 169 | "rawQuery": true, 170 | "rawSql": "-- grafana ignore\nSELECT\n relname as table,\n least((SELECT setting::int FROM pg_settings WHERE name = 'autovacuum_freeze_max_age') - age(relfrozenxid), \n (SELECT setting::int FROM pg_settings WHERE name = 'autovacuum_multixact_freeze_max_age') - mxid_age(relminmxid))\n tx_before_wraparound_vacuum,\n pg_stat_get_last_autovacuum_time(c.oid) AS last_autovacuum,\n age(relfrozenxid) AS xid_age,\n mxid_age(relminmxid) AS mxid_age\nFROM\n pg_class c\nWHERE\n c.relname in ('ethereum_blocks', 'eth_call_cache','subgraph_deployment')\n and c.relfrozenxid != 0\n", 171 | "refId": "A", 172 | "select": [ 173 | [ 174 | { 175 | "params": [ 176 | "value" 177 | ], 178 | "type": "column" 179 | } 180 | ] 181 | ], 182 | "timeColumn": "time", 183 | "where": [ 184 | { 185 | "name": "$__timeFilter", 186 | "params": [], 187 | "type": "macro" 188 | } 189 | ] 190 | } 191 | ], 192 | "timeFrom": null, 193 | "timeShift": null, 194 | "title": "Transactions until wraparound vacuum", 195 | "transform": "table", 196 | "type": "table" 197 | }, 198 | { 199 | "columns": [], 200 | "datasource": "postgres", 201 | "fontSize": "100%", 202 | "gridPos": { 203 | "h": 3, 204 | "w": 11, 205 | "x": 13, 206 | "y": 8 207 | }, 208 | "id": 11, 209 | "pageSize": null, 210 | "showHeader": true, 211 | "sort": { 212 | "col": 0, 213 | "desc": true 214 | }, 215 | "styles": [ 216 | { 217 | "alias": "Tables needing vacuum", 218 | "align": "auto", 219 | "colorMode": null, 220 | "colors": [ 221 | "rgba(245, 54, 54, 0.9)", 222 | "rgba(237, 129, 40, 0.89)", 223 | "rgba(50, 172, 45, 0.97)" 224 | ], 225 | "decimals": 0, 226 | "pattern": "tables_needing_vacuum", 227 | "thresholds": [], 228 | "type": "number", 229 | "unit": "none" 230 | }, 231 | { 232 | "alias": "Txns past", 233 | "align": "auto", 234 | "colorMode": null, 235 | "colors": [ 236 | "rgba(245, 54, 54, 0.9)", 237 | "rgba(237, 129, 40, 0.89)", 238 | "rgba(50, 172, 45, 0.97)" 239 | ], 240 | "dateFormat": "YYYY-MM-DD HH:mm:ss", 241 | "decimals": 0, 242 | "mappingType": 1, 243 | "pattern": "txns_past", 244 | "thresholds": [], 245 | "type": "number", 246 | "unit": "locale" 247 | }, 248 | { 249 | "alias": "Last autovacuum", 250 | "align": "auto", 251 | "colorMode": null, 252 | "colors": [ 253 | "rgba(245, 54, 54, 0.9)", 254 | "rgba(237, 129, 40, 0.89)", 255 | "rgba(50, 172, 45, 0.97)" 256 | ], 257 | "dateFormat": "YYYY-MM-DD HH:mm:ss", 258 | "decimals": 2, 259 | "mappingType": 1, 260 | "pattern": "last_autovacuum", 261 | "thresholds": [], 262 | "type": "date", 263 | "unit": "short" 264 | } 265 | ], 266 | "targets": [ 267 | { 268 | "format": "table", 269 | "group": [], 270 | "metricColumn": "none", 271 | "rawQuery": true, 272 | "rawSql": "-- grafana ignore\nselect count(*) as tables_needing_vacuum,\n -min(tx_before_wraparound_vacuum) as txns_past,\n min(last_autovacuum) as last_autovacuum\n from (\n select oid::regclass::text AS table,\n least(\n (select setting::int\n from pg_settings\n where name = 'autovacuum_freeze_max_age') - age(relfrozenxid),\n (select setting::int\n from pg_settings\n where name = 'autovacuum_multixact_freeze_max_age')\n - mxid_age(relminmxid)) as tx_before_wraparound_vacuum,\n pg_stat_get_last_autovacuum_time(oid) AS last_autovacuum,\n age(relfrozenxid) AS xid_age,\n mxid_age(relminmxid) AS mxid_age\n from pg_class\n where relfrozenxid != 0\n and oid > 16384\n and relkind='r') a where a.tx_before_wraparound_vacuum < 0;\n", 273 | "refId": "A", 274 | "select": [ 275 | [ 276 | { 277 | "params": [ 278 | "value" 279 | ], 280 | "type": "column" 281 | } 282 | ] 283 | ], 284 | "timeColumn": "time", 285 | "where": [ 286 | { 287 | "name": "$__timeFilter", 288 | "params": [], 289 | "type": "macro" 290 | } 291 | ] 292 | } 293 | ], 294 | "timeFrom": null, 295 | "timeShift": null, 296 | "title": "Autovacuum pressure", 297 | "transform": "table", 298 | "type": "table" 299 | }, 300 | { 301 | "columns": [], 302 | "datasource": "postgres", 303 | "fontSize": "100%", 304 | "gridPos": { 305 | "h": 9, 306 | "w": 24, 307 | "x": 0, 308 | "y": 13 309 | }, 310 | "id": 7, 311 | "links": [], 312 | "pageSize": null, 313 | "scroll": true, 314 | "showHeader": true, 315 | "sort": { 316 | "col": 4, 317 | "desc": false 318 | }, 319 | "styles": [ 320 | { 321 | "alias": "Granted", 322 | "align": "auto", 323 | "dateFormat": "YYYY-MM-DD HH:mm:ss", 324 | "decimals": 0, 325 | "mappingType": 1, 326 | "pattern": "granted", 327 | "preserveFormat": false, 328 | "sanitize": false, 329 | "thresholds": [ 330 | "" 331 | ], 332 | "type": "string", 333 | "unit": "none", 334 | "valueMaps": [ 335 | { 336 | "text": "✓", 337 | "value": "true" 338 | }, 339 | { 340 | "text": "—", 341 | "value": "false" 342 | } 343 | ] 344 | }, 345 | { 346 | "alias": "Age", 347 | "align": "auto", 348 | "colorMode": null, 349 | "colors": [ 350 | "rgba(245, 54, 54, 0.9)", 351 | "rgba(237, 129, 40, 0.89)", 352 | "rgba(50, 172, 45, 0.97)" 353 | ], 354 | "decimals": 2, 355 | "pattern": "age", 356 | "thresholds": [], 357 | "type": "number", 358 | "unit": "s" 359 | } 360 | ], 361 | "targets": [ 362 | { 363 | "format": "table", 364 | "group": [], 365 | "metricColumn": "none", 366 | "rawQuery": true, 367 | "rawSql": "-- grafana ignore\nSELECT a.application_name,\n coalesce(extract(epoch from age(now(), a.xact_start)), 0) as age,\n l.relation::regclass,\n l.mode,\n l.GRANTED::varchar,\n l.locktype \"Target\"\nFROM pg_stat_activity a\nJOIN pg_locks l ON l.pid = a.pid\nwhere pg_backend_pid() != a.pid\norder by granted asc, age desc;", 368 | "refId": "A", 369 | "select": [ 370 | [ 371 | { 372 | "params": [ 373 | "value" 374 | ], 375 | "type": "column" 376 | } 377 | ] 378 | ], 379 | "timeColumn": "time", 380 | "where": [ 381 | { 382 | "name": "$__timeFilter", 383 | "params": [], 384 | "type": "macro" 385 | } 386 | ] 387 | } 388 | ], 389 | "timeFrom": null, 390 | "timeShift": null, 391 | "title": "Active locks", 392 | "transform": "table", 393 | "type": "table" 394 | }, 395 | { 396 | "columns": [], 397 | "datasource": "postgres", 398 | "fontSize": "100%", 399 | "gridPos": { 400 | "h": 8, 401 | "w": 24, 402 | "x": 0, 403 | "y": 22 404 | }, 405 | "id": 5, 406 | "links": [], 407 | "pageSize": null, 408 | "scroll": true, 409 | "showHeader": true, 410 | "sort": { 411 | "col": 4, 412 | "desc": false 413 | }, 414 | "styles": [ 415 | { 416 | "alias": "Age", 417 | "align": "auto", 418 | "dateFormat": "YYYY-MM-DD HH:mm:ss", 419 | "pattern": "age", 420 | "type": "number", 421 | "unit": "s" 422 | }, 423 | { 424 | "alias": "", 425 | "align": "auto", 426 | "colorMode": null, 427 | "colors": [ 428 | "rgba(245, 54, 54, 0.9)", 429 | "rgba(237, 129, 40, 0.89)", 430 | "rgba(50, 172, 45, 0.97)" 431 | ], 432 | "decimals": 2, 433 | "pattern": "/.*/", 434 | "thresholds": [], 435 | "type": "number", 436 | "unit": "short" 437 | } 438 | ], 439 | "targets": [ 440 | { 441 | "format": "table", 442 | "group": [], 443 | "metricColumn": "none", 444 | "rawQuery": true, 445 | "rawSql": "-- grafana ignore\nselect client_addr,\n application_name,\n usename,\n state,\n extract(epoch from age(now(), xact_start)) as age\n from pg_stat_activity\n where query not like '%grafana ignore%'\n and state like '%idle in transaction%'\norder by query_start desc", 446 | "refId": "A", 447 | "select": [ 448 | [ 449 | { 450 | "params": [ 451 | "value" 452 | ], 453 | "type": "column" 454 | } 455 | ] 456 | ], 457 | "timeColumn": "time", 458 | "where": [ 459 | { 460 | "name": "$__timeFilter", 461 | "params": [], 462 | "type": "macro" 463 | } 464 | ] 465 | } 466 | ], 467 | "timeFrom": null, 468 | "timeShift": null, 469 | "title": "Idle transactions", 470 | "transform": "table", 471 | "type": "table" 472 | } 473 | ], 474 | "schemaVersion": 22, 475 | "style": "dark", 476 | "tags": [], 477 | "templating": { 478 | "list": [] 479 | }, 480 | "time": { 481 | "from": "now-6h", 482 | "to": "now" 483 | }, 484 | "timepicker": { 485 | "refresh_intervals": [ 486 | "5s", 487 | "10s", 488 | "30s", 489 | "1m", 490 | "5m", 491 | "15m", 492 | "30m", 493 | "1h", 494 | "2h", 495 | "1d" 496 | ], 497 | "time_options": [ 498 | "5m", 499 | "15m", 500 | "1h", 501 | "6h", 502 | "12h", 503 | "24h", 504 | "2d", 505 | "7d", 506 | "30d" 507 | ] 508 | }, 509 | "timezone": "", 510 | "title": "Postgres Statistics", 511 | "uid": "Mo6FxoiWz", 512 | "variables": { 513 | "list": [] 514 | }, 515 | "version": 4 516 | } 517 | -------------------------------------------------------------------------------- /monitoring/docker-compose.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/pi0neerpat/indexer-docker-compose/7aaaa53170c2e20e49a6f2a4ac5982cc0a6f7b9b/monitoring/docker-compose.png -------------------------------------------------------------------------------- /monitoring/docker-compose.yml: -------------------------------------------------------------------------------- 1 | version: "3.2" 2 | services: 3 | grafana: 4 | image: grafana/grafana:6.4.4 5 | ports: 6 | - "3000:3000" 7 | volumes: 8 | - type: bind 9 | source: /docker/volumes/grafana/data/ 10 | target: /var/lib/grafana 11 | - type: bind 12 | source: /docker/volumes/grafana/log/ 13 | target: /var/log/grafana 14 | environment: 15 | GF_SECURITY_ADMIN_PASSWORD: admin 16 | GF_ANALYTICS_REPORTING_ENABLED: "false" 17 | GF_SERVER_DOMAIN: "grafana.mysite.com" 18 | GF_SERVER_ROOT_URL: "https://grafana.mysite.com" 19 | network_mode: bridge 20 | restart: unless-stopped 21 | # mem_limit: 4G 22 | # mem_reservation: 16M 23 | prometheus: 24 | image: prom/prometheus:v2.20.1 25 | command: 26 | - "--config.file=/etc/prometheus/prometheus.yml" 27 | - "--storage.tsdb.path=/prometheus" 28 | - "--storage.tsdb.retention.time=120d" 29 | - "--web.console.libraries=/usr/share/prometheus/console_libraries" 30 | - "--web.console.templates=/usr/share/prometheus/consoles" 31 | volumes: 32 | - type: "bind" 33 | source: $HOME/monitoring-data/prometheus 34 | target: /prometheus 35 | - type: "bind" 36 | source: $HOME/indexer-docker-compose/monitoring/prometheus.yml 37 | target: /etc/prometheus/prometheus.yml 38 | ports: 39 | - "9090:9090" 40 | network_mode: bridge 41 | restart: unless-stopped 42 | -------------------------------------------------------------------------------- /monitoring/prometheus.yml: -------------------------------------------------------------------------------- 1 | global: 2 | scrape_interval: 15s 3 | evaluation_interval: 15s 4 | 5 | rule_files: 6 | 7 | alerting: 8 | 9 | scrape_configs: 10 | - job_name: "thegraph" 11 | static_configs: 12 | - targets: 13 | - 172.17.0.1:8140 # CHANGE ME 14 | - 172.17.0.1:8040 # CHANGE ME 15 | -------------------------------------------------------------------------------- /nginx/README.md: -------------------------------------------------------------------------------- 1 | # Indexer Nginx Configurations 2 | 3 | After placing these in `etc/nginx/sites-enabled`, you should use Certbot to create certificates for **all sites**. 4 | 5 | # Help I've never used Nginx! 6 | 7 | ## Installation 8 | 9 | Adapted from [Digital Ocean - Installation on Ubuntu 18.04](https://www.digitalocean.com/community/tutorials/how-to-install-nginx-on-ubuntu-18-04#step-2-%E2%80%93-adjusting-the-firewall) 10 | 11 | ```bash 12 | sudo apt update 13 | sudo apt install nginx 14 | 15 | sudo ufw app list 16 | # Nginx Full 17 | # Nginx HTTP <- Choose this one while SSL is not be enabled 18 | # Nginx HTTPS <- Switch to this one later 19 | sudo ufw allow 'Nginx HTTP' 20 | 21 | # Confirm things are running 22 | sudo ufw status 23 | systemctl status nginx 24 | 25 | # Enable restart on reboot 26 | sudo systemctl enable nginx 27 | ``` 28 | 29 | ## Configuration 30 | 31 | To configure Nginx a `.conf` file is added to `/etc/nginx/sites-enabled` which tells Nginx how to handle incoming requests. 32 | 33 | Since we will be serving existing ports on our machine (instead of static sites), we will be using the [reverse-proxy](https://docs.nginx.com/nginx/admin-guide/web-server/reverse-proxy/) feature. Here is an example `.conf` file which allows anyone to access port `8000` on our server at `http://indexer.mysite.com/` 34 | 35 | ```js 36 | server { 37 | server_name indexer.mysite.com; 38 | 39 | location / { 40 | proxy_pass http://localhost:8000; 41 | # ... 42 | } 43 | } 44 | ``` 45 | 46 | After you make a change to a `.conf` file, you can test it and apply it using these commands: 47 | 48 | ```bash 49 | # Check the config files are OK 50 | sudo nginx -t 51 | 52 | # Restart the service the the new configs 53 | sudo systemctl reload nginx 54 | ``` 55 | 56 | That's pretty much it. You can do it! 57 | -------------------------------------------------------------------------------- /nginx/indexer.conf: -------------------------------------------------------------------------------- 1 | server { 2 | # CHANGE ME 3 | server_name indexer.mysite.com; 4 | 5 | location ^~ /index-node/ { 6 | # Remove the /index-node/ again 7 | rewrite ^/index-node/(.*)$ /$1 break; 8 | 9 | # Proxy configuration. 10 | proxy_pass http://127.0.0.1:8030; 11 | proxy_http_version 1.1; 12 | proxy_set_header Connection $connection_upgrade; 13 | proxy_set_header Host $host; 14 | proxy_set_header Upgrade $http_upgrade; 15 | proxy_cache_bypass $http_upgrade; 16 | # Gateway timeout. 17 | proxy_read_timeout 30s; 18 | proxy_send_timeout 30s; 19 | } 20 | 21 | location /nginx_status { 22 | stub_status; 23 | } 24 | 25 | location / { 26 | location = / { 27 | return 200; 28 | } 29 | 30 | # Move WebSocket and HTTP requests into /ws/ and /http/ prefixes; 31 | # this allows us to forward both types of requests to different 32 | # query node ports 33 | if ( $connection_upgrade = "upgrade" ) { 34 | rewrite ^(.*)$ /ws/$1; 35 | } 36 | if ( $connection_upgrade != "upgrade" ) { 37 | rewrite ^(.*)$ /http/$1; 38 | } 39 | 40 | location /http/ { 41 | # Remove the /http/ again 42 | rewrite ^/http/(.*)$ $1 break; 43 | 44 | # Proxy configuration. 45 | proxy_pass http://127.0.0.1:8000; 46 | proxy_http_version 1.1; 47 | proxy_set_header Connection $connection_upgrade; 48 | proxy_set_header Host $host; 49 | proxy_set_header Upgrade $http_upgrade; 50 | proxy_set_header X-Real-IP $remote_addr; 51 | proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; 52 | proxy_set_header Host $http_host; 53 | proxy_cache_bypass $http_upgrade; 54 | proxy_next_upstream error invalid_header http_502; 55 | 56 | # Gateway timeout. 57 | proxy_read_timeout 30s; 58 | proxy_send_timeout 30s; 59 | } 60 | 61 | location /ws { 62 | # Remove the /ws/ again 63 | rewrite ^/ws/(.*)$ $1 break; 64 | 65 | # Proxy configuration. 66 | proxy_pass http://127.0.0.1:8001; 67 | proxy_http_version 1.1; 68 | proxy_set_header Connection $connection_upgrade; 69 | proxy_set_header Host $host; 70 | proxy_set_header Upgrade $http_upgrade; 71 | proxy_set_header X-Real-IP $remote_addr; 72 | proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; 73 | proxy_set_header Host $http_host; 74 | proxy_cache_bypass $http_upgrade; 75 | 76 | # Gateway timeout. 77 | proxy_read_timeout 1800s; 78 | proxy_send_timeout 1800s; 79 | } 80 | } 81 | 82 | error_page 404 404.html; 83 | 84 | access_log /var/log/nginx/access.log; 85 | error_log /var/log/nginx/error.log; 86 | } 87 | -------------------------------------------------------------------------------- /nginx/monitoring.conf: -------------------------------------------------------------------------------- 1 | server { 2 | # CHANGE ME 3 | server_name prometheus.mysite.com; 4 | 5 | location / { 6 | proxy_pass http://127.0.0.1:9090; 7 | proxy_http_version 1.1; 8 | proxy_set_header Connection $connection_upgrade; 9 | proxy_set_header Host $host; 10 | proxy_set_header Upgrade $http_upgrade; 11 | proxy_cache_bypass $http_upgrade; 12 | } 13 | } 14 | 15 | server { 16 | # CHANGE ME 17 | server_name grafana.mysite.com; 18 | 19 | location / { 20 | proxy_pass http://0.0.0.0:3000; 21 | proxy_http_version 1.1; 22 | proxy_set_header Upgrade $http_upgrade; 23 | proxy_set_header Connection 'upgrade'; 24 | proxy_set_header Host $host; 25 | proxy_cache_bypass $http_upgrade; 26 | } 27 | } 28 | -------------------------------------------------------------------------------- /notes.md: -------------------------------------------------------------------------------- 1 | ## Notes 2 | 3 | > These are my personal notes, please ignore 4 | 5 | TODO: Optimize graph-node for less-powerful hardware by utilizing flags [see here](https://github.com/graphprotocol/graph-node/blob/master/docs/environment-variables.md) 6 | 7 | ```bash 8 | docker exec -ti bash 9 | psql -U graph-node 10 | 11 | # list databases 12 | \l 13 | # list tables 14 | \dt 15 | 16 | ``` 17 | 18 | Deleting indexer database scripts 19 | 20 | - https://gist.github.com/lutter/b74ee0ae20d1ca2614b9a060ff6df2d5 21 | - https://ptb.discordapp.com/channels/438038660412342282/737341252835737641/753748878146273350 22 | 23 | ## Adding new Dashboards 24 | 25 | https://github.com/graphprotocol/mission-control-indexer/wiki/Grafana-Dashboard-Enhancements 26 | -------------------------------------------------------------------------------- /notes/levers-to-pull.md: -------------------------------------------------------------------------------- 1 | It will likely become important to manage/throttle your query service in order to stay competitive. 2 | 3 | Keep in mind that the service you provide must match developer expectations. Otherwise they will build a buautiful query, but be frustrated when your service will not execute it. 4 | 5 | ### Throttling with query node Env Variables 6 | 7 | - SUBSCRIPTION_THROTTLE_INTERVAL: while a subgraph is syncing, subscriptions to that subgraph get updated at most this often, in ms. Default is 1000ms. 8 | 9 | - No one has built a dapp that needs this responsiveness. Increase it! 10 | 11 | - GRAPH_GRAPHQL_MAX_COMPLEXITY: maximum complexity for a graphql query. See here for what that means. Default is unlimited. Typical introspection queries have a complexity of just over 1 million, so setting a value below that may interfere with introspection done by graphql clients. 12 | 13 | - Reduce me! 14 | 15 | - GRAPH_GRAPHQL_MAX_DEPTH: maximum depth of a graphql query. Default (and maximum) is 255. 16 | 17 | - Definitely want to reduce this one 18 | 19 | - GRAPH_GRAPHQL_MAX_FIRST: maximum value that can be used for the first argument in GraphQL queries. If not provided, first defaults to 100. The default value for GRAPH_GRAPHQL_MAX_FIRST is 1000. 20 | 21 | - Yeah... you're not gonna show 1000 of anything on a screen. Reduce this! 22 | 23 | ## Throttling with 24 | -------------------------------------------------------------------------------- /performance/readme.md: -------------------------------------------------------------------------------- 1 |

Performance Testing

2 | 3 | Eventually this is where I'll discuss performance enhanecements. For now it is just notes from my own testing at specific hardware sizes. 4 | 5 | --- 6 | 7 | > I haven't completed testing yet! Stay tuned 8 | 9 | # Methods 10 | 11 | Using phase 0 test harness https://github.com/graphprotocol/mission-control-indexer/tree/master/testing/phase0. Note that queries are duplicated, so this does not really approximate actual usage. 12 | 13 | ```bash 14 | npm install -g typescript ts-node 15 | ``` 16 | 17 | Check a test passes with `cat report.csv` 18 | 19 | All tested with: 20 | 21 | - single indexer node, single query node 22 | - active indexer with fully synced moloch and uniswap 23 | 24 | # Results 25 | 26 | ## Standard phase 0 harness 27 | 28 | 300s test: `./cli test --output report.csv indexers.csv queries.csv | tee report.md` 29 | 30 | | Hardware | RAM Baseline | RAM Load | System Baseline | System Load | req/min | 31 | | :--------- | :----------- | :------- | --------------- | ----------- | ------- | 32 | | 32GB 8vCPU | 1.81GB | 1.79GB | 5% | 47% | 99 | 33 | | 8GB 4vCPU | 1.38GB | GB | 6% | % | | 34 | 35 | ## Increase connections 36 | 37 | ```bash 38 | ./cli test --output report.csv --duration 120 \ 39 | --max-error-rate .0005 \ 40 | --min-request-rate 10 \ 41 | --connections-per-indexer 100 \ 42 | indexers.csv queries.csv | tee report.md 43 | ``` 44 | 45 | | Hardware | RAM Baseline | RAM Load | System Baseline | System Load | req/min | 46 | | :--------- | :----------- | :------- | --------------- | ----------- | ------- | 47 | | 32GB 8vCPU | 1.81GB | 1.9GB | 4% | 45% | 190 | 48 | | 8GB 4vCPU | 2.15GB | 2.17GB | 6% | 24% | FAILED | 49 | 50 | - Increasing connections per indexer by 10x only doubled req/min 51 | 52 | ## Lower error rate 0.005% 53 | 54 | ```bash 55 | ./cli test --output report.csv --duration 120 \ 56 | --max-error-rate .000005 \ 57 | --min-request-rate 10 \ 58 | --connections-per-indexer 100 \ 59 | indexers.csv queries.csv | tee report.md 60 | ``` 61 | 62 | | Hardware | RAM Baseline | RAM Load | System Baseline | System Load | req/min | 63 | | :--------- | :----------- | :------- | --------------- | ----------- | ------- | 64 | | 32GB 8vCPU | 1.81GB | 1.96GB | 4% | 40% | 198 | 65 | 66 | ## Increase req rate 67 | 68 | ```bash 69 | ./cli test --output report.csv --duration 120 \ 70 | --max-error-rate .000005 \ 71 | --min-request-rate 100 \ 72 | --connections-per-indexer 10 \ 73 | indexers.csv queries.csv | tee report.md 74 | ``` 75 | 76 | | Hardware | RAM Baseline | RAM Load | System Baseline | System Load | req/min | 77 | | :--------- | :----------- | :------- | --------------- | ----------- | ------- | 78 | | 32GB 8vCPU | 1.81GB | 1.92 | 4% | 47% | 124 | 79 | 80 | ## High req rate and connections 81 | 82 | ```bash 83 | ./cli test --output report.csv --duration 120 \ 84 | --max-error-rate .000005 \ 85 | --min-request-rate 100 \ 86 | --connections-per-indexer 100 \ 87 | indexers.csv queries.csv | tee report.md 88 | ``` 89 | 90 | | Hardware | RAM Baseline | RAM Load | System Baseline | System Load | req/min | 91 | | :--------- | :----------- | :------- | --------------- | ----------- | ------- | 92 | | 32GB 8vCPU | 1.81GB | 1.94 | 4% | 42 | 203 | 93 | 94 | ## 5 min - 100 connections 95 | 96 | ```bash 97 | ./cli test --output report.csv --duration 300 \ 98 | --max-error-rate .000005 \ 99 | --min-request-rate 10 \ 100 | --connections-per-indexer 100 \ 101 | indexers.csv queries.csv | tee report.md 102 | ``` 103 | 104 | | Hardware | RAM Baseline | RAM Load | System Baseline | System Load | req/min | 105 | | :--------- | :----------- | :------- | --------------- | ----------- | ------- | 106 | | 32GB 8vCPU | 1.81GB | 1.95 | 4% | 48 | 220 | 107 | 108 | ## 5 min - 2 machines with 200 connections 109 | 110 | ```bash 111 | ./cli test --output report.csv --duration 300 \ 112 | --max-error-rate .000005 \ 113 | --min-request-rate 10 \ 114 | --connections-per-indexer 100 \ 115 | indexers.csv queries.csv | tee report.md 116 | ``` 117 | 118 | | Hardware | RAM Baseline | RAM Load | System Baseline | System Load | req/min | 119 | | :--------- | :----------- | :------- | --------------- | ----------- | ------- | 120 | | 32GB 8vCPU | 1.9GB | 2.0 | 3% | 78% | 1,000 | 121 | 122 | ## 5 min - 3 machines with 300 connections 123 | 124 | ```bash 125 | ./cli test --output report.csv --duration 300 \ 126 | --max-error-rate .000005 \ 127 | --min-request-rate 10 \ 128 | --connections-per-indexer 100 \ 129 | indexers.csv queries.csv | tee report.md 130 | ``` 131 | 132 | | Hardware | RAM Baseline | RAM Load | System Baseline | System Load | req/min | 133 | | :--------- | :----------- | :------- | --------------- | ----------- | ------- | 134 | | 32GB 8vCPU | 1.9GB | 2.01 | 3% | 81 | 1,100 | 135 | 136 | ## 5 min - 3 machines with 900 connections 137 | 138 | ```bash 139 | ./cli test --output report.csv --duration 300 \ 140 | --max-error-rate .000005 \ 141 | --min-request-rate 10 \ 142 | --connections-per-indexer 300 \ 143 | indexers.csv queries.csv | tee report.md 144 | ``` 145 | 146 | | Hardware | RAM Baseline | RAM Load | System Baseline | System Load | req/min | 147 | | :--------- | :----------- | :------- | --------------- | ----------- | ------- | 148 | | 32GB 8vCPU | 1.89GB | 2.0 | 3% | 82% | 850 | 149 | 150 | - This is 900 connections for a single indexer. Not sure that is even supported? 151 | - Internal server error on grafana - "Annotation query failed" 152 | 153 | ## 5 min - 2 machines with 300 connections 154 | 155 | ```bash 156 | ./cli test --output report.csv --duration 300 \ 157 | --max-error-rate .000005 \ 158 | --min-request-rate 10 \ 159 | --connections-per-indexer 150 \ 160 | indexers.csv queries.csv | tee report.md 161 | ``` 162 | 163 | | Hardware | RAM Baseline | RAM Load | System Baseline | System Load | req/min | 164 | | :--------- | :----------- | :------- | --------------- | ----------- | ------- | 165 | | 32GB 8vCPU | 1.89GB | 1.93 | 3% | 80% | 950 | 166 | 167 | ## 5 min - 4 machines with 400 connections 168 | 169 | ```bash 170 | ./cli test --output report.csv --duration 300 \ 171 | --max-error-rate .000005 \ 172 | --min-request-rate 10 \ 173 | --connections-per-indexer 100 \ 174 | indexers.csv queries.csv | tee report.md 175 | ``` 176 | 177 | | Hardware | RAM Baseline | RAM Load | System Baseline | System Load | req/min | 178 | | :--------- | :----------- | :------- | --------------- | ----------- | ------- | 179 | | 32GB 8vCPU | 1.8GB | 2.03 | 3% | 78% | 1,000 | 180 | 181 | ## Add a second query node? 182 | 183 | TODO: 184 | 185 | - use Traefik to set up multiple index nodes and try to increase req/min beyond 1k 186 | - Change postgres config to allow more connections 187 | - Does query node have some setting for connections. Couldn't find it in graph-node docs. Check my other notes 188 | 189 | # Picking a smaller server 190 | 191 | - Total db is 98GB for the phase 0 subgraphs, so likely will need 1TB. Unless I don't keep blocks. Will have to check whether this indeed does degrade performance if the archive node is nearby (!) 192 | - Choosing the \$40/mo DO droplet has 4vCPU, 8GB RAM, 160GB SSD, which seems like enough 193 | - Similar server on Hetzner Cloud is 12 EUR (\$14) 194 | - For comparable price of 34 EUR, can get a Hetzner Dedicated AX41-NVMe has 64 GB RAM, 6 CPU, 2x512GB SSD. 195 | -------------------------------------------------------------------------------- /vps-introduction/readme.md: -------------------------------------------------------------------------------- 1 | # Provisioning new servers 2 | 3 | ## Basic setup 4 | 5 | login as root 6 | 7 | ```bash 8 | # reset the root password 9 | sudo passwd root 10 | 11 | adduser myself 12 | usermod -aG sudo myself 13 | 14 | ufw app list 15 | ufw allow OpenSSH 16 | ufw enable 17 | 18 | dpkg-reconfigure -plow unattended-upgrades 19 | 20 | exit 21 | ``` 22 | 23 | Login with your new user and install npm using nvm 24 | 25 | ```bash 26 | curl -sL https://raw.githubusercontent.com/creationix/nvm/v0.33.11/install.sh -o install_nvm.sh 27 | bash install_nvm.sh 28 | source ~/.profile 29 | nvm install --lts 30 | nvm use --lts 31 | ``` 32 | 33 | Install docker using [these instructions](https://docs.docker.com/engine/install/ubuntu/). Alternatively use the official convenience scripts. 34 | 35 | ```bash 36 | # Download Docker 37 | curl -fsSL get.docker.com -o get-docker.sh 38 | # Install Docker using the stable channel (instead of the default "edge") 39 | CHANNEL=stable sh get-docker.sh 40 | # Remove Docker install script 41 | rm get-docker.sh 42 | 43 | # Remove need for sudo 44 | sudo groupadd docker 45 | sudo usermod -aG docker $USER 46 | # logout or restart the VM to take effect 47 | ``` 48 | 49 | If you're migrating from an existing server, don't forget the `-P` option on `rsync` 50 | 51 | ```bash 52 | rsync -P A host:B 53 | ``` 54 | 55 | See [Copy PostgreSQL database between computers](https://wiki-bsse.ethz.ch/display/ITDOC/Copy+PostgreSQL+database+between+computers) 56 | --------------------------------------------------------------------------------