├── roles ├── configure-docker │ ├── vars │ │ └── main.yml │ ├── defaults │ │ └── main.yml │ ├── handlers │ │ └── main.yml │ ├── README.md │ ├── meta │ │ └── main.yml │ └── tasks │ │ └── main.yml ├── configure-grafana │ ├── vars │ │ └── main.yml │ ├── defaults │ │ └── main.yml │ ├── handlers │ │ └── main.yml │ ├── README.md │ ├── templates │ │ └── grafana │ │ │ ├── dashboard.yml.j2 │ │ │ ├── datasource.yml.j2 │ │ │ └── dashboard-small.json.j2 │ ├── meta │ │ └── main.yml │ └── tasks │ │ └── main.yml ├── configure-psql │ ├── vars │ │ └── main.yml │ ├── defaults │ │ └── main.yml │ ├── handlers │ │ └── main.yml │ ├── README.md │ ├── tasks │ │ └── main.yml │ └── meta │ │ └── main.yml ├── prepare-steampipe │ ├── vars │ │ └── main.yml │ ├── defaults │ │ └── main.yml │ ├── handlers │ │ └── main.yml │ ├── README.md │ ├── tasks │ │ └── main.yml │ └── meta │ │ └── main.yml ├── run-docker-compose │ ├── vars │ │ └── main.yml │ ├── defaults │ │ └── main.yml │ ├── handlers │ │ └── main.yml │ ├── README.md │ ├── tasks │ │ └── main.yml │ ├── meta │ │ └── main.yml │ └── templates │ │ └── docker-compose │ │ └── docker-compose.yml.j2 ├── configure-prometheus │ ├── vars │ │ └── main.yml │ ├── defaults │ │ └── main.yml │ ├── handlers │ │ └── main.yml │ ├── README.md │ ├── templates │ │ └── prometheus │ │ │ └── etc-prometheus-yml.j2 │ ├── meta │ │ └── main.yml │ └── tasks │ │ └── main.yml ├── configure-pushgateway │ ├── vars │ │ └── main.yml │ ├── defaults │ │ └── main.yml │ ├── handlers │ │ └── main.yml │ ├── README.md │ ├── templates │ │ └── pushgateway │ │ │ └── pushgateway.yml.j2 │ ├── tasks │ │ └── main.yml │ └── meta │ │ └── main.yml ├── configure-steampipe │ ├── vars │ │ └── main.yml │ ├── defaults │ │ └── main.yml │ ├── handlers │ │ └── main.yml │ ├── README.md │ ├── files │ │ └── steampipe │ │ │ └── config │ ├── templates │ │ ├── steampipe │ │ │ └── etc-oci.spc.j2 │ │ └── postgresql │ │ │ ├── psql-example-compute-instance-running.py.j2 │ │ │ └── psql-example-block-volume-summary.py.j2 │ ├── meta │ │ └── main.yml │ └── tasks │ │ └── main.yml └── requirements.yml ├── ansible.cfg ├── hosts ├── group_vars └── all.yml ├── images ├── architecture.png ├── oci_api_key.jpg ├── grafana_login.jpg ├── oci_grafana_01.png ├── oci_grafana_02.png ├── oci_grafana_03.png ├── oci_group_user.jpg ├── oci_tenancy_ocid.jpg ├── oci_user_readonly.jpg ├── oci_user_readonly.png ├── oci_compute_instance.jpg ├── oci_group_readonly.jpg ├── oci_group_readonly.png ├── oci_policy_readonly.jpg └── oci_pushgateway_01.png ├── .gitignore ├── install.yml └── README.md /roles/configure-docker/vars/main.yml: -------------------------------------------------------------------------------- 1 | --- 2 | # vars file for docker -------------------------------------------------------------------------------- /roles/configure-grafana/vars/main.yml: -------------------------------------------------------------------------------- 1 | --- 2 | # vars file for docker -------------------------------------------------------------------------------- /roles/configure-psql/vars/main.yml: -------------------------------------------------------------------------------- 1 | --- 2 | # vars file for docker -------------------------------------------------------------------------------- /roles/prepare-steampipe/vars/main.yml: -------------------------------------------------------------------------------- 1 | --- 2 | # vars file for docker -------------------------------------------------------------------------------- /roles/run-docker-compose/vars/main.yml: -------------------------------------------------------------------------------- 1 | --- 2 | # vars file for docker -------------------------------------------------------------------------------- /ansible.cfg: -------------------------------------------------------------------------------- 1 | [defaults] 2 | inventory = hosts 3 | host_key_checking = False -------------------------------------------------------------------------------- /hosts: -------------------------------------------------------------------------------- 1 | [monitoring] 2 | localhost ansible_user=opc ansible_connection=local -------------------------------------------------------------------------------- /roles/configure-prometheus/vars/main.yml: -------------------------------------------------------------------------------- 1 | --- 2 | # vars file for docker -------------------------------------------------------------------------------- /roles/configure-psql/defaults/main.yml: -------------------------------------------------------------------------------- 1 | --- 2 | # defaults file for docker -------------------------------------------------------------------------------- /roles/configure-psql/handlers/main.yml: -------------------------------------------------------------------------------- 1 | --- 2 | # handlers file for docker -------------------------------------------------------------------------------- /roles/configure-pushgateway/vars/main.yml: -------------------------------------------------------------------------------- 1 | --- 2 | # vars file for docker -------------------------------------------------------------------------------- /roles/configure-steampipe/vars/main.yml: -------------------------------------------------------------------------------- 1 | --- 2 | # vars file for docker -------------------------------------------------------------------------------- /roles/configure-docker/defaults/main.yml: -------------------------------------------------------------------------------- 1 | --- 2 | # defaults file for docker -------------------------------------------------------------------------------- /roles/configure-docker/handlers/main.yml: -------------------------------------------------------------------------------- 1 | --- 2 | # handlers file for docker -------------------------------------------------------------------------------- /roles/configure-grafana/defaults/main.yml: -------------------------------------------------------------------------------- 1 | --- 2 | # defaults file for docker -------------------------------------------------------------------------------- /roles/configure-grafana/handlers/main.yml: -------------------------------------------------------------------------------- 1 | --- 2 | # handlers file for docker -------------------------------------------------------------------------------- /roles/configure-prometheus/defaults/main.yml: -------------------------------------------------------------------------------- 1 | --- 2 | # defaults file for docker -------------------------------------------------------------------------------- /roles/configure-prometheus/handlers/main.yml: -------------------------------------------------------------------------------- 1 | --- 2 | # handlers file for docker -------------------------------------------------------------------------------- /roles/configure-steampipe/defaults/main.yml: -------------------------------------------------------------------------------- 1 | --- 2 | # defaults file for docker -------------------------------------------------------------------------------- /roles/configure-steampipe/handlers/main.yml: -------------------------------------------------------------------------------- 1 | --- 2 | # handlers file for docker -------------------------------------------------------------------------------- /roles/prepare-steampipe/defaults/main.yml: -------------------------------------------------------------------------------- 1 | --- 2 | # defaults file for docker -------------------------------------------------------------------------------- /roles/prepare-steampipe/handlers/main.yml: -------------------------------------------------------------------------------- 1 | --- 2 | # handlers file for docker -------------------------------------------------------------------------------- /roles/run-docker-compose/defaults/main.yml: -------------------------------------------------------------------------------- 1 | --- 2 | # defaults file for docker -------------------------------------------------------------------------------- /roles/run-docker-compose/handlers/main.yml: -------------------------------------------------------------------------------- 1 | --- 2 | # handlers file for docker -------------------------------------------------------------------------------- /roles/configure-pushgateway/defaults/main.yml: -------------------------------------------------------------------------------- 1 | --- 2 | # defaults file for docker -------------------------------------------------------------------------------- /roles/configure-pushgateway/handlers/main.yml: -------------------------------------------------------------------------------- 1 | --- 2 | # handlers file for docker -------------------------------------------------------------------------------- /group_vars/all.yml: -------------------------------------------------------------------------------- 1 | # general vars 2 | docker_grafana_image: "grafana/grafana-oss:7.5.6" -------------------------------------------------------------------------------- /images/architecture.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/martinberger-ch/oci-monitoring/HEAD/images/architecture.png -------------------------------------------------------------------------------- /images/oci_api_key.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/martinberger-ch/oci-monitoring/HEAD/images/oci_api_key.jpg -------------------------------------------------------------------------------- /roles/requirements.yml: -------------------------------------------------------------------------------- 1 | --- 2 | collections: 3 | # With just the collection name 4 | - community.docker 5 | -------------------------------------------------------------------------------- /images/grafana_login.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/martinberger-ch/oci-monitoring/HEAD/images/grafana_login.jpg -------------------------------------------------------------------------------- /images/oci_grafana_01.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/martinberger-ch/oci-monitoring/HEAD/images/oci_grafana_01.png -------------------------------------------------------------------------------- /images/oci_grafana_02.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/martinberger-ch/oci-monitoring/HEAD/images/oci_grafana_02.png -------------------------------------------------------------------------------- /images/oci_grafana_03.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/martinberger-ch/oci-monitoring/HEAD/images/oci_grafana_03.png -------------------------------------------------------------------------------- /images/oci_group_user.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/martinberger-ch/oci-monitoring/HEAD/images/oci_group_user.jpg -------------------------------------------------------------------------------- /images/oci_tenancy_ocid.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/martinberger-ch/oci-monitoring/HEAD/images/oci_tenancy_ocid.jpg -------------------------------------------------------------------------------- /images/oci_user_readonly.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/martinberger-ch/oci-monitoring/HEAD/images/oci_user_readonly.jpg -------------------------------------------------------------------------------- /images/oci_user_readonly.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/martinberger-ch/oci-monitoring/HEAD/images/oci_user_readonly.png -------------------------------------------------------------------------------- /images/oci_compute_instance.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/martinberger-ch/oci-monitoring/HEAD/images/oci_compute_instance.jpg -------------------------------------------------------------------------------- /images/oci_group_readonly.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/martinberger-ch/oci-monitoring/HEAD/images/oci_group_readonly.jpg -------------------------------------------------------------------------------- /images/oci_group_readonly.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/martinberger-ch/oci-monitoring/HEAD/images/oci_group_readonly.png -------------------------------------------------------------------------------- /images/oci_policy_readonly.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/martinberger-ch/oci-monitoring/HEAD/images/oci_policy_readonly.jpg -------------------------------------------------------------------------------- /images/oci_pushgateway_01.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/martinberger-ch/oci-monitoring/HEAD/images/oci_pushgateway_01.png -------------------------------------------------------------------------------- /roles/run-docker-compose/README.md: -------------------------------------------------------------------------------- 1 | run-docker-compose 2 | ==================== 3 | 4 | This role executes docker-compose. 5 | -------------------------------------------------------------------------------- /roles/configure-psql/README.md: -------------------------------------------------------------------------------- 1 | configure-psql 2 | ============== 3 | 4 | This role installs PostgreSQL client files which are uses in Python. 5 | -------------------------------------------------------------------------------- /roles/configure-docker/README.md: -------------------------------------------------------------------------------- 1 | configure-docker 2 | ================ 3 | 4 | This role installs the Docker software and configures docker-compose. 5 | -------------------------------------------------------------------------------- /roles/configure-grafana/README.md: -------------------------------------------------------------------------------- 1 | configure-grafana 2 | ================= 3 | 4 | This role prepares Grafana related directories and configuration files for later used when running docker-compose. 5 | -------------------------------------------------------------------------------- /roles/prepare-steampipe/README.md: -------------------------------------------------------------------------------- 1 | prepare-steampipe 2 | ================= 3 | 4 | This role prepares Steampipe related directories and configuration files for later used when running docker-compose. 5 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | .vscode/* 2 | !.vscode/settings.json 3 | !.vscode/tasks.json 4 | !.vscode/launch.json 5 | !.vscode/extensions.json 6 | *.code-workspace 7 | 8 | # Local History for Visual Studio Code 9 | .history/ -------------------------------------------------------------------------------- /roles/configure-prometheus/README.md: -------------------------------------------------------------------------------- 1 | configure-prometheus 2 | ==================== 3 | 4 | This role prepares Prometheus related directories and configuration files for later used when running docker-compose. 5 | -------------------------------------------------------------------------------- /roles/configure-steampipe/README.md: -------------------------------------------------------------------------------- 1 | configure-steampipe 2 | ==================== 3 | 4 | This role prepares Prometheus related directories and configuration files for later used when running docker-compose. 5 | -------------------------------------------------------------------------------- /roles/configure-pushgateway/README.md: -------------------------------------------------------------------------------- 1 | configure-pushgateway 2 | ===================== 3 | 4 | This role prepares Pushgateway related directories and configuration files for later used when running docker-compose. 5 | -------------------------------------------------------------------------------- /roles/configure-grafana/templates/grafana/dashboard.yml.j2: -------------------------------------------------------------------------------- 1 | apiVersion: 1 2 | 3 | providers: 4 | - name: 'default' 5 | orgId: 1 6 | folder: '' 7 | type: file 8 | disableDeletion: false 9 | options: 10 | path: /etc/grafana/provisioning/dashboards -------------------------------------------------------------------------------- /roles/configure-steampipe/files/steampipe/config: -------------------------------------------------------------------------------- 1 | [DEFAULT] 2 | user= 3 | fingerprint= 4 | tenancy= 5 | region= 6 | key_file=~/.oci/ -------------------------------------------------------------------------------- /roles/configure-prometheus/templates/prometheus/etc-prometheus-yml.j2: -------------------------------------------------------------------------------- 1 | # A scrape configuration scraping a Node Exporter and the Prometheus server 2 | # itself. 3 | scrape_configs: 4 | - job_name: 'prometheus' 5 | static_configs: 6 | - targets: ['localhost:9090'] -------------------------------------------------------------------------------- /roles/configure-pushgateway/templates/pushgateway/pushgateway.yml.j2: -------------------------------------------------------------------------------- 1 | # The Push Gateway Configuration 2 | - job_name: pushgateway 3 | # prometheus push gateway pull 4 | honor_labels: false 5 | static_configs: 6 | - targets: ['{{ ansible_default_ipv4.address }}:9091'] -------------------------------------------------------------------------------- /roles/configure-steampipe/templates/steampipe/etc-oci.spc.j2: -------------------------------------------------------------------------------- 1 | connection "oci" { 2 | plugin = "oci" 3 | config_file_profile = "DEFAULT" # Name of the profile 4 | config_path = "~/.oci/config" # Path to config file 5 | regions = ["eu-frankfurt-1" , "eu-zurich-1"] # List of regions 6 | } -------------------------------------------------------------------------------- /install.yml: -------------------------------------------------------------------------------- 1 | # --------------------------------------------------------------------------- 2 | # Name.......: install.yml 3 | # Author.....: Martin Berger (mbg) martin.x.berger@accenture.com 4 | # Editor.....: Martin Berger 5 | # Date.......: 2022.11.17 6 | # Revision...: 7 | # Purpose....: Playbook to install all required components 8 | # Notes......: Role names = Tag names 9 | # Reference..: 10 | # License....: Apache License Version 2.0, January 2004 as shown 11 | # at http://www.apache.org/licenses/ 12 | # --------------------------------------------------------------------------- 13 | 14 | --- 15 | 16 | - name: Install OCI Monitoring 17 | hosts: monitoring 18 | become: yes 19 | roles: 20 | - configure-grafana 21 | - configure-prometheus 22 | - configure-pushgateway 23 | - configure-psql 24 | - configure-docker 25 | - prepare-steampipe 26 | - run-docker-compose 27 | - configure-steampipe 28 | 29 | -------------------------------------------------------------------------------- /roles/configure-steampipe/templates/postgresql/psql-example-compute-instance-running.py.j2: -------------------------------------------------------------------------------- 1 | import psycopg2 2 | from prometheus_client import CollectorRegistry, Gauge, push_to_gateway 3 | from prometheus_client import Summary 4 | 5 | # set postgresql connect string 6 | uri = f'{{ steampipe_connect_string }}' 7 | 8 | try: 9 | # verify connection 10 | con = psycopg2.connect(uri) 11 | print(f'Connected to DB.') 12 | 13 | try: 14 | # execute sql query 15 | cur = con.cursor() 16 | cur.execute('select count(*) from oci_core_instance where lifecycle_state=\'RUNNING\';') 17 | print('Query ran') 18 | except: 19 | print('Query failed') 20 | raise 21 | else: 22 | # set variable with query return value 23 | ci_summary = cur.fetchone()[0] 24 | print(ci_summary) 25 | 26 | if ci_summary is None: 27 | ci_summary = 0 28 | 29 | # prepare pushgateway 30 | registry = CollectorRegistry() 31 | g = Gauge('oci_compute_running_summary', 'OCI Compute Running Summary', registry=registry) 32 | g.set(int(ci_summary)) 33 | 34 | # push data to pushgateway 35 | push_to_gateway('{{ ansible_default_ipv4.address }}:9091', job='oci_compute', registry=registry) 36 | 37 | finally: 38 | con.close() 39 | print(f'Connection closed.') 40 | except Exception as e: 41 | print('Something went wrong:', e) -------------------------------------------------------------------------------- /roles/configure-steampipe/templates/postgresql/psql-example-block-volume-summary.py.j2: -------------------------------------------------------------------------------- 1 | import psycopg2 2 | from prometheus_client import CollectorRegistry, Gauge, push_to_gateway 3 | from prometheus_client import Summary 4 | 5 | # set postgresql connect string 6 | uri = f'{{ steampipe_connect_string }}' 7 | 8 | try: 9 | # verify connection 10 | con = psycopg2.connect(uri) 11 | print(f'Connected to DB.') 12 | 13 | try: 14 | # execute sql query 15 | cur = con.cursor() 16 | cur.execute('SELECT sum(size_in_gbs) from oci_core_volume where lifecycle_state=\'AVAILABLE\';') 17 | print('Query ran') 18 | except: 19 | print('Query failed') 20 | raise 21 | else: 22 | # set variable with query return value 23 | bv_summary = cur.fetchone()[0] 24 | print(bv_summary) 25 | 26 | if bv_summary is None: 27 | bv_summary = 0 28 | 29 | # prepare pushgateway 30 | registry = CollectorRegistry() 31 | g = Gauge('oci_compute_blockvolumes_summary', 'OCI Compute Block Volumes Summary', registry=registry) 32 | g.set(int(bv_summary)) 33 | 34 | # push data to pushgateway 35 | push_to_gateway('{{ ansible_default_ipv4.address }}:9091', job='oci_blockvolume', registry=registry) 36 | 37 | finally: 38 | con.close() 39 | print(f'Connection closed.') 40 | except Exception as e: 41 | print('Something went wrong:', e) 42 | -------------------------------------------------------------------------------- /roles/configure-pushgateway/tasks/main.yml: -------------------------------------------------------------------------------- 1 | # --------------------------------------------------------------------------- 2 | # Name.......: configure-pushgateway/tasks/main.yml 3 | # Author.....: Martin Berger (mbg) martin.x.berger@accenture.com 4 | # Editor.....: Martin Berger 5 | # Date.......: 2022.11.17 6 | # Revision...: 7 | # Purpose....: Prepare directories and files for Pushgateway 8 | # Notes......: 9 | # Reference..: 10 | # License....: Apache License Version 2.0, January 2004 as shown 11 | # at http://www.apache.org/licenses/ 12 | # --------------------------------------------------------------------------- 13 | 14 | --- 15 | 16 | # - Configure Pushgateway --------------------------------------------------- 17 | # - Verify user and group 18 | # - Copy scrape configuration 19 | # - Enable port 9091 in firewall 20 | # --------------------------------------------------------------------------- 21 | 22 | - name: Ensure group 'prometheus' exists 23 | group: 24 | name: prometheus 25 | state: present 26 | tags: configure-pushgateway 27 | 28 | - name: Add Gateway Scrape Config 29 | lineinfile: 30 | insertafter: EOF 31 | path: /etc/prometheus/prometheus.yml 32 | line: "{{ lookup('template', 'pushgateway/pushgateway.yml.j2') }}" 33 | mode: 0644 34 | owner: prometheus 35 | group: prometheus 36 | tags: configure-pushgateway 37 | 38 | - name: Enable port 9091 in firewall rules 39 | firewalld: 40 | port: 9091/tcp 41 | state: enabled 42 | tags: configure-pushgateway -------------------------------------------------------------------------------- /roles/run-docker-compose/tasks/main.yml: -------------------------------------------------------------------------------- 1 | # --------------------------------------------------------------------------- 2 | # Name.......: run-docker-compose/tasks/main.yml 3 | # Author.....: Martin Berger (mbg) martin.x.berger@accenture.com 4 | # Editor.....: Martin Berger 5 | # Date.......: 2022.11.17 6 | # Revision...: 7 | # Purpose....: Run Docker Compose File 8 | # Notes......: 9 | # Reference..: 10 | # License....: Apache License Version 2.0, January 2004 as shown 11 | # at http://www.apache.org/licenses/ 12 | # --------------------------------------------------------------------------- 13 | 14 | --- 15 | 16 | # - Run Docker-Compose ------------------------------------------------------ 17 | # - Copy docker-compose file 18 | # - Run task 19 | # --------------------------------------------------------------------------- 20 | - name: Create directory if they don't exist 21 | file: 22 | path: "{{ item }}" 23 | state: directory 24 | owner: root 25 | group: root 26 | mode: 0775 27 | loop: 28 | - /etc/docker 29 | tags: run-docker-compose 30 | 31 | - name: Copy docker-compose.yml file 32 | template: 33 | dest: /etc/docker/docker-compose.yml 34 | force: true 35 | src: docker-compose/docker-compose.yml.j2 36 | mode: 0644 37 | tags: run-docker-compose 38 | 39 | - name: Run docker-compose 40 | shell: 41 | cmd: "docker-compose -f /etc/docker/docker-compose.yml up -d" 42 | tags: run-docker-compose 43 | 44 | 45 | - name: Change file permission for docker.sock 46 | file: 47 | path: /var/run/docker.sock 48 | mode: '0666' 49 | tags: run-docker-compose -------------------------------------------------------------------------------- /roles/prepare-steampipe/tasks/main.yml: -------------------------------------------------------------------------------- 1 | # --------------------------------------------------------------------------- 2 | # Name.......: prepare-steampipe/tasks/main.yml 3 | # Author.....: Martin Berger (mbg) martin.x.berger@accenture.com 4 | # Editor.....: Martin Berger 5 | # Date.......: 2022.11.17 6 | # Revision...: 7 | # Purpose....: Prepare directories and files for Prometheus 8 | # Notes......: 9 | # Reference..: 10 | # License....: Apache License Version 2.0, January 2004 as shown 11 | # at http://www.apache.org/licenses/ 12 | # --------------------------------------------------------------------------- 13 | 14 | --- 15 | 16 | # - Prepare Steampipe -------------------------------------------------------- 17 | # - Verify user and group 18 | # - Create directories 19 | # --------------------------------------------------------------------------- 20 | 21 | - name: Ensure group 'steampipe' exists 22 | group: 23 | name: steampipe 24 | state: present 25 | tags: prepare-steampipe 26 | 27 | - name: Add the user steampipe with a group of 'steampipe' 28 | user: 29 | name: steampipe 30 | comment: Steampipe User 31 | group: steampipe 32 | uid: 9193 33 | tags: prepare-steampipe 34 | 35 | - name: create directory if they don't exist 36 | file: 37 | path: "{{ item }}" 38 | state: directory 39 | mode: 0775 40 | owner: steampipe 41 | group: steampipe 42 | loop: 43 | - /home/steampipe/sp 44 | - /home/steampipe/config 45 | - /home/steampipe/sql 46 | - /home/steampipe/py 47 | - /home/steampipe/tmp 48 | - /home/steampipe/sp/config 49 | - /home/steampipe/.oci 50 | - /home/steampipe/.steampipe 51 | tags: prepare-steampipe -------------------------------------------------------------------------------- /roles/configure-psql/tasks/main.yml: -------------------------------------------------------------------------------- 1 | # --------------------------------------------------------------------------- 2 | # Name.......: configure-psql/tasks/main.yml 3 | # Author.....: Martin Berger (mbg) martin.x.berger@accenture.com 4 | # Editor.....: Martin Berger 5 | # Date.......: 2022.11.17 6 | # Revision...: 7 | # Purpose....: Prepare directories and files for PSQL 8 | # Notes......: 9 | # Reference..: 10 | # License....: Apache License Version 2.0, January 2004 as shown 11 | # at http://www.apache.org/licenses/ 12 | # --------------------------------------------------------------------------- 13 | 14 | --- 15 | 16 | # - Configure PSQL ------------------------------------------------------- 17 | # - Install PostgreSQL related files 18 | # - Install Prometheus client 19 | # --------------------------------------------------------------------------- 20 | 21 | - name: Install the latest version of pgsql 22 | dnf: 23 | name: postgresql 24 | state: present 25 | tags: configure-psql 26 | 27 | - name: Install the latest version of pgsql-devel 28 | dnf: 29 | name: postgresql-devel 30 | state: present 31 | tags: configure-psql 32 | 33 | - name: Install the latest version of gcc 34 | dnf: 35 | name: gcc 36 | state: present 37 | tags: configure-psql 38 | 39 | - name: Install the latest version of python3-devel 40 | dnf: 41 | name: python3-devel 42 | state: present 43 | tags: configure-psql 44 | 45 | - name: Install psql for Python 3 specifically, using the 'pip3' executable 46 | pip: 47 | name: psycopg2 48 | executable: pip3 49 | tags: configure-psql 50 | 51 | - name: Install client for Python 3 specifically, using the 'pip3' executable 52 | pip: 53 | name: prometheus_client 54 | executable: pip3 55 | tags: configure-psql -------------------------------------------------------------------------------- /roles/configure-psql/meta/main.yml: -------------------------------------------------------------------------------- 1 | galaxy_info: 2 | author: your name 3 | description: your role description 4 | company: your company (optional) 5 | 6 | # If the issue tracker for your role is not on github, uncomment the 7 | # next line and provide a value 8 | # issue_tracker_url: http://example.com/issue/tracker 9 | 10 | # Choose a valid license ID from https://spdx.org - some suggested licenses: 11 | # - BSD-3-Clause (default) 12 | # - MIT 13 | # - GPL-2.0-or-later 14 | # - GPL-3.0-only 15 | # - Apache-2.0 16 | # - CC-BY-4.0 17 | license: license (GPL-2.0-or-later, MIT, etc) 18 | 19 | min_ansible_version: 2.9 20 | 21 | # If this a Container Enabled role, provide the minimum Ansible Container version. 22 | # min_ansible_container_version: 23 | 24 | # 25 | # Provide a list of supported platforms, and for each platform a list of versions. 26 | # If you don't wish to enumerate all versions for a particular platform, use 'all'. 27 | # To view available platforms and versions (or releases), visit: 28 | # https://galaxy.ansible.com/api/v1/platforms/ 29 | # 30 | # platforms: 31 | # - name: Fedora 32 | # versions: 33 | # - all 34 | # - 25 35 | # - name: SomePlatform 36 | # versions: 37 | # - all 38 | # - 1.0 39 | # - 7 40 | # - 99.99 41 | 42 | galaxy_tags: [] 43 | # List tags for your role here, one per line. A tag is a keyword that describes 44 | # and categorizes the role. Users find roles by searching for tags. Be sure to 45 | # remove the '[]' above, if you add tags to this list. 46 | # 47 | # NOTE: A tag is limited to a single word comprised of alphanumeric characters. 48 | # Maximum 20 tags per role. 49 | 50 | dependencies: [] 51 | # List your role dependencies here, one per line. Be sure to remove the '[]' above, 52 | # if you add dependencies to this list. 53 | -------------------------------------------------------------------------------- /roles/configure-docker/meta/main.yml: -------------------------------------------------------------------------------- 1 | galaxy_info: 2 | author: your name 3 | description: your role description 4 | company: your company (optional) 5 | 6 | # If the issue tracker for your role is not on github, uncomment the 7 | # next line and provide a value 8 | # issue_tracker_url: http://example.com/issue/tracker 9 | 10 | # Choose a valid license ID from https://spdx.org - some suggested licenses: 11 | # - BSD-3-Clause (default) 12 | # - MIT 13 | # - GPL-2.0-or-later 14 | # - GPL-3.0-only 15 | # - Apache-2.0 16 | # - CC-BY-4.0 17 | license: license (GPL-2.0-or-later, MIT, etc) 18 | 19 | min_ansible_version: 2.9 20 | 21 | # If this a Container Enabled role, provide the minimum Ansible Container version. 22 | # min_ansible_container_version: 23 | 24 | # 25 | # Provide a list of supported platforms, and for each platform a list of versions. 26 | # If you don't wish to enumerate all versions for a particular platform, use 'all'. 27 | # To view available platforms and versions (or releases), visit: 28 | # https://galaxy.ansible.com/api/v1/platforms/ 29 | # 30 | # platforms: 31 | # - name: Fedora 32 | # versions: 33 | # - all 34 | # - 25 35 | # - name: SomePlatform 36 | # versions: 37 | # - all 38 | # - 1.0 39 | # - 7 40 | # - 99.99 41 | 42 | galaxy_tags: [] 43 | # List tags for your role here, one per line. A tag is a keyword that describes 44 | # and categorizes the role. Users find roles by searching for tags. Be sure to 45 | # remove the '[]' above, if you add tags to this list. 46 | # 47 | # NOTE: A tag is limited to a single word comprised of alphanumeric characters. 48 | # Maximum 20 tags per role. 49 | 50 | dependencies: [] 51 | # List your role dependencies here, one per line. Be sure to remove the '[]' above, 52 | # if you add dependencies to this list. 53 | -------------------------------------------------------------------------------- /roles/configure-grafana/meta/main.yml: -------------------------------------------------------------------------------- 1 | galaxy_info: 2 | author: your name 3 | description: your role description 4 | company: your company (optional) 5 | 6 | # If the issue tracker for your role is not on github, uncomment the 7 | # next line and provide a value 8 | # issue_tracker_url: http://example.com/issue/tracker 9 | 10 | # Choose a valid license ID from https://spdx.org - some suggested licenses: 11 | # - BSD-3-Clause (default) 12 | # - MIT 13 | # - GPL-2.0-or-later 14 | # - GPL-3.0-only 15 | # - Apache-2.0 16 | # - CC-BY-4.0 17 | license: license (GPL-2.0-or-later, MIT, etc) 18 | 19 | min_ansible_version: 2.9 20 | 21 | # If this a Container Enabled role, provide the minimum Ansible Container version. 22 | # min_ansible_container_version: 23 | 24 | # 25 | # Provide a list of supported platforms, and for each platform a list of versions. 26 | # If you don't wish to enumerate all versions for a particular platform, use 'all'. 27 | # To view available platforms and versions (or releases), visit: 28 | # https://galaxy.ansible.com/api/v1/platforms/ 29 | # 30 | # platforms: 31 | # - name: Fedora 32 | # versions: 33 | # - all 34 | # - 25 35 | # - name: SomePlatform 36 | # versions: 37 | # - all 38 | # - 1.0 39 | # - 7 40 | # - 99.99 41 | 42 | galaxy_tags: [] 43 | # List tags for your role here, one per line. A tag is a keyword that describes 44 | # and categorizes the role. Users find roles by searching for tags. Be sure to 45 | # remove the '[]' above, if you add tags to this list. 46 | # 47 | # NOTE: A tag is limited to a single word comprised of alphanumeric characters. 48 | # Maximum 20 tags per role. 49 | 50 | dependencies: [] 51 | # List your role dependencies here, one per line. Be sure to remove the '[]' above, 52 | # if you add dependencies to this list. 53 | -------------------------------------------------------------------------------- /roles/configure-prometheus/meta/main.yml: -------------------------------------------------------------------------------- 1 | galaxy_info: 2 | author: your name 3 | description: your role description 4 | company: your company (optional) 5 | 6 | # If the issue tracker for your role is not on github, uncomment the 7 | # next line and provide a value 8 | # issue_tracker_url: http://example.com/issue/tracker 9 | 10 | # Choose a valid license ID from https://spdx.org - some suggested licenses: 11 | # - BSD-3-Clause (default) 12 | # - MIT 13 | # - GPL-2.0-or-later 14 | # - GPL-3.0-only 15 | # - Apache-2.0 16 | # - CC-BY-4.0 17 | license: license (GPL-2.0-or-later, MIT, etc) 18 | 19 | min_ansible_version: 2.9 20 | 21 | # If this a Container Enabled role, provide the minimum Ansible Container version. 22 | # min_ansible_container_version: 23 | 24 | # 25 | # Provide a list of supported platforms, and for each platform a list of versions. 26 | # If you don't wish to enumerate all versions for a particular platform, use 'all'. 27 | # To view available platforms and versions (or releases), visit: 28 | # https://galaxy.ansible.com/api/v1/platforms/ 29 | # 30 | # platforms: 31 | # - name: Fedora 32 | # versions: 33 | # - all 34 | # - 25 35 | # - name: SomePlatform 36 | # versions: 37 | # - all 38 | # - 1.0 39 | # - 7 40 | # - 99.99 41 | 42 | galaxy_tags: [] 43 | # List tags for your role here, one per line. A tag is a keyword that describes 44 | # and categorizes the role. Users find roles by searching for tags. Be sure to 45 | # remove the '[]' above, if you add tags to this list. 46 | # 47 | # NOTE: A tag is limited to a single word comprised of alphanumeric characters. 48 | # Maximum 20 tags per role. 49 | 50 | dependencies: [] 51 | # List your role dependencies here, one per line. Be sure to remove the '[]' above, 52 | # if you add dependencies to this list. 53 | -------------------------------------------------------------------------------- /roles/configure-steampipe/meta/main.yml: -------------------------------------------------------------------------------- 1 | galaxy_info: 2 | author: your name 3 | description: your role description 4 | company: your company (optional) 5 | 6 | # If the issue tracker for your role is not on github, uncomment the 7 | # next line and provide a value 8 | # issue_tracker_url: http://example.com/issue/tracker 9 | 10 | # Choose a valid license ID from https://spdx.org - some suggested licenses: 11 | # - BSD-3-Clause (default) 12 | # - MIT 13 | # - GPL-2.0-or-later 14 | # - GPL-3.0-only 15 | # - Apache-2.0 16 | # - CC-BY-4.0 17 | license: license (GPL-2.0-or-later, MIT, etc) 18 | 19 | min_ansible_version: 2.9 20 | 21 | # If this a Container Enabled role, provide the minimum Ansible Container version. 22 | # min_ansible_container_version: 23 | 24 | # 25 | # Provide a list of supported platforms, and for each platform a list of versions. 26 | # If you don't wish to enumerate all versions for a particular platform, use 'all'. 27 | # To view available platforms and versions (or releases), visit: 28 | # https://galaxy.ansible.com/api/v1/platforms/ 29 | # 30 | # platforms: 31 | # - name: Fedora 32 | # versions: 33 | # - all 34 | # - 25 35 | # - name: SomePlatform 36 | # versions: 37 | # - all 38 | # - 1.0 39 | # - 7 40 | # - 99.99 41 | 42 | galaxy_tags: [] 43 | # List tags for your role here, one per line. A tag is a keyword that describes 44 | # and categorizes the role. Users find roles by searching for tags. Be sure to 45 | # remove the '[]' above, if you add tags to this list. 46 | # 47 | # NOTE: A tag is limited to a single word comprised of alphanumeric characters. 48 | # Maximum 20 tags per role. 49 | 50 | dependencies: [] 51 | # List your role dependencies here, one per line. Be sure to remove the '[]' above, 52 | # if you add dependencies to this list. 53 | -------------------------------------------------------------------------------- /roles/prepare-steampipe/meta/main.yml: -------------------------------------------------------------------------------- 1 | galaxy_info: 2 | author: your name 3 | description: your role description 4 | company: your company (optional) 5 | 6 | # If the issue tracker for your role is not on github, uncomment the 7 | # next line and provide a value 8 | # issue_tracker_url: http://example.com/issue/tracker 9 | 10 | # Choose a valid license ID from https://spdx.org - some suggested licenses: 11 | # - BSD-3-Clause (default) 12 | # - MIT 13 | # - GPL-2.0-or-later 14 | # - GPL-3.0-only 15 | # - Apache-2.0 16 | # - CC-BY-4.0 17 | license: license (GPL-2.0-or-later, MIT, etc) 18 | 19 | min_ansible_version: 2.9 20 | 21 | # If this a Container Enabled role, provide the minimum Ansible Container version. 22 | # min_ansible_container_version: 23 | 24 | # 25 | # Provide a list of supported platforms, and for each platform a list of versions. 26 | # If you don't wish to enumerate all versions for a particular platform, use 'all'. 27 | # To view available platforms and versions (or releases), visit: 28 | # https://galaxy.ansible.com/api/v1/platforms/ 29 | # 30 | # platforms: 31 | # - name: Fedora 32 | # versions: 33 | # - all 34 | # - 25 35 | # - name: SomePlatform 36 | # versions: 37 | # - all 38 | # - 1.0 39 | # - 7 40 | # - 99.99 41 | 42 | galaxy_tags: [] 43 | # List tags for your role here, one per line. A tag is a keyword that describes 44 | # and categorizes the role. Users find roles by searching for tags. Be sure to 45 | # remove the '[]' above, if you add tags to this list. 46 | # 47 | # NOTE: A tag is limited to a single word comprised of alphanumeric characters. 48 | # Maximum 20 tags per role. 49 | 50 | dependencies: [] 51 | # List your role dependencies here, one per line. Be sure to remove the '[]' above, 52 | # if you add dependencies to this list. 53 | -------------------------------------------------------------------------------- /roles/run-docker-compose/meta/main.yml: -------------------------------------------------------------------------------- 1 | galaxy_info: 2 | author: your name 3 | description: your role description 4 | company: your company (optional) 5 | 6 | # If the issue tracker for your role is not on github, uncomment the 7 | # next line and provide a value 8 | # issue_tracker_url: http://example.com/issue/tracker 9 | 10 | # Choose a valid license ID from https://spdx.org - some suggested licenses: 11 | # - BSD-3-Clause (default) 12 | # - MIT 13 | # - GPL-2.0-or-later 14 | # - GPL-3.0-only 15 | # - Apache-2.0 16 | # - CC-BY-4.0 17 | license: license (GPL-2.0-or-later, MIT, etc) 18 | 19 | min_ansible_version: 2.9 20 | 21 | # If this a Container Enabled role, provide the minimum Ansible Container version. 22 | # min_ansible_container_version: 23 | 24 | # 25 | # Provide a list of supported platforms, and for each platform a list of versions. 26 | # If you don't wish to enumerate all versions for a particular platform, use 'all'. 27 | # To view available platforms and versions (or releases), visit: 28 | # https://galaxy.ansible.com/api/v1/platforms/ 29 | # 30 | # platforms: 31 | # - name: Fedora 32 | # versions: 33 | # - all 34 | # - 25 35 | # - name: SomePlatform 36 | # versions: 37 | # - all 38 | # - 1.0 39 | # - 7 40 | # - 99.99 41 | 42 | galaxy_tags: [] 43 | # List tags for your role here, one per line. A tag is a keyword that describes 44 | # and categorizes the role. Users find roles by searching for tags. Be sure to 45 | # remove the '[]' above, if you add tags to this list. 46 | # 47 | # NOTE: A tag is limited to a single word comprised of alphanumeric characters. 48 | # Maximum 20 tags per role. 49 | 50 | dependencies: [] 51 | # List your role dependencies here, one per line. Be sure to remove the '[]' above, 52 | # if you add dependencies to this list. 53 | -------------------------------------------------------------------------------- /roles/configure-pushgateway/meta/main.yml: -------------------------------------------------------------------------------- 1 | galaxy_info: 2 | author: your name 3 | description: your role description 4 | company: your company (optional) 5 | 6 | # If the issue tracker for your role is not on github, uncomment the 7 | # next line and provide a value 8 | # issue_tracker_url: http://example.com/issue/tracker 9 | 10 | # Choose a valid license ID from https://spdx.org - some suggested licenses: 11 | # - BSD-3-Clause (default) 12 | # - MIT 13 | # - GPL-2.0-or-later 14 | # - GPL-3.0-only 15 | # - Apache-2.0 16 | # - CC-BY-4.0 17 | license: license (GPL-2.0-or-later, MIT, etc) 18 | 19 | min_ansible_version: 2.9 20 | 21 | # If this a Container Enabled role, provide the minimum Ansible Container version. 22 | # min_ansible_container_version: 23 | 24 | # 25 | # Provide a list of supported platforms, and for each platform a list of versions. 26 | # If you don't wish to enumerate all versions for a particular platform, use 'all'. 27 | # To view available platforms and versions (or releases), visit: 28 | # https://galaxy.ansible.com/api/v1/platforms/ 29 | # 30 | # platforms: 31 | # - name: Fedora 32 | # versions: 33 | # - all 34 | # - 25 35 | # - name: SomePlatform 36 | # versions: 37 | # - all 38 | # - 1.0 39 | # - 7 40 | # - 99.99 41 | 42 | galaxy_tags: [] 43 | # List tags for your role here, one per line. A tag is a keyword that describes 44 | # and categorizes the role. Users find roles by searching for tags. Be sure to 45 | # remove the '[]' above, if you add tags to this list. 46 | # 47 | # NOTE: A tag is limited to a single word comprised of alphanumeric characters. 48 | # Maximum 20 tags per role. 49 | 50 | dependencies: [] 51 | # List your role dependencies here, one per line. Be sure to remove the '[]' above, 52 | # if you add dependencies to this list. 53 | -------------------------------------------------------------------------------- /roles/configure-prometheus/tasks/main.yml: -------------------------------------------------------------------------------- 1 | # --------------------------------------------------------------------------- 2 | # Name.......: configure-prometheus/tasks/main.yml 3 | # Author.....: Martin Berger (mbg) martin.x.berger@accenture.com 4 | # Editor.....: Martin Berger 5 | # Date.......: 2022.11.17 6 | # Revision...: 7 | # Purpose....: Prepare directories and files for Prometheus 8 | # Notes......: 9 | # Reference..: 10 | # License....: Apache License Version 2.0, January 2004 as shown 11 | # at http://www.apache.org/licenses/ 12 | # --------------------------------------------------------------------------- 13 | 14 | --- 15 | 16 | # - Configure Prometheus ------------------------------------------------------- 17 | # - Verify user and group 18 | # - Create directories 19 | # - Copy configuration file 20 | # - Enable port 9090 in firewall 21 | # --------------------------------------------------------------------------- 22 | 23 | - name: Ensure group 'prometheus' exists 24 | group: 25 | name: prometheus 26 | state: present 27 | tags: configure-prometheus 28 | 29 | - name: Add the user prometheus with a group of 'prometheus' 30 | user: 31 | name: prometheus 32 | comment: Prometheus User 33 | group: prometheus 34 | tags: configure-prometheus 35 | 36 | - name: Create directories if they don't exist 37 | file: 38 | path: "{{ item }}" 39 | state: directory 40 | owner: prometheus 41 | group: prometheus 42 | mode: 0775 43 | loop: 44 | - /etc/prometheus 45 | - /home/prometheus/data 46 | tags: configure-prometheus 47 | 48 | - name: Copy Prometheus configuration file 49 | template: 50 | dest: /etc/prometheus/prometheus.yml 51 | force: true 52 | src: prometheus/etc-prometheus-yml.j2 53 | mode: 0644 54 | owner: prometheus 55 | group: prometheus 56 | tags: configure-prometheus 57 | 58 | - name: Enable port 9090 in firewall rules 59 | firewalld: 60 | port: 9090/tcp 61 | state: enabled 62 | tags: configure-prometheus 63 | -------------------------------------------------------------------------------- /roles/configure-grafana/templates/grafana/datasource.yml.j2: -------------------------------------------------------------------------------- 1 | # config file version 2 | apiVersion: 1 3 | 4 | # list of datasources that should be deleted from the database 5 | deleteDatasources: 6 | - name: Prometheus 7 | orgId: 1 8 | 9 | # list of datasources to insert/update depending 10 | # what's available in the database 11 | datasources: 12 | # name of the datasource. Required 13 | - name: Prometheus 14 | # datasource type. Required 15 | type: prometheus 16 | # access mode. proxy or direct (Server or Browser in the UI). Required 17 | access: proxy 18 | # org id. will default to orgId 1 if not specified 19 | orgId: 1 20 | # custom UID which can be used to reference this datasource in other parts of the configuration, if not specified will be generated automatically 21 | uid: my_unique_uid 22 | # url 23 | url: 'http://{{ ansible_default_ipv4.address }}:9090' 24 | # Deprecated, use secureJsonData.password 25 | password: 26 | # database user, if used 27 | user: 28 | # database name, if used 29 | database: 30 | # enable/disable basic auth 31 | basicAuth: 32 | # basic auth username 33 | basicAuthUser: 34 | # Deprecated, use secureJsonData.basicAuthPassword 35 | basicAuthPassword: 36 | # enable/disable with credentials headers 37 | withCredentials: 38 | # mark as default datasource. Max one per org 39 | isDefault: 40 | # fields that will be converted to json and stored in jsonData 41 | jsonData: 42 | graphiteVersion: '1.1' 43 | tlsAuth: false 44 | tlsAuthWithCACert: false 45 | # json object of data that will be encrypted. 46 | secureJsonData: 47 | tlsCACert: '...' 48 | tlsClientCert: '...' 49 | tlsClientKey: '...' 50 | # database password, if used 51 | password: 52 | # basic auth password 53 | basicAuthPassword: 54 | version: 1 55 | # allow users to edit datasources from the UI. 56 | editable: false -------------------------------------------------------------------------------- /roles/configure-grafana/tasks/main.yml: -------------------------------------------------------------------------------- 1 | # --------------------------------------------------------------------------- 2 | # Name.......: configure-grafana/tasks/main.yml 3 | # Author.....: Martin Berger (mbg) martin.x.berger@accenture.com 4 | # Editor.....: Martin Berger 5 | # Date.......: 2022.11.17 6 | # Revision...: 7 | # Purpose....: Prepare directories and files for Grafana 8 | # Notes......: 9 | # Reference..: 10 | # License....: Apache License Version 2.0, January 2004 as shown 11 | # at http://www.apache.org/licenses/ 12 | # --------------------------------------------------------------------------- 13 | 14 | --- 15 | 16 | # - Configure Grafana ------------------------------------------------------- 17 | # - Create directories 18 | # - Configure Prometheus as Data Source 19 | # - Configure example dashboard 20 | # - Enable port 3000 in firewall 21 | # --------------------------------------------------------------------------- 22 | 23 | - name: Create directories if they don't exist 24 | file: 25 | path: "{{ item }}" 26 | state: directory 27 | owner: root 28 | group: root 29 | mode: 0775 30 | loop: 31 | - /etc/grafana/provisioning/datasources 32 | - /etc/grafana/provisioning/notifiers 33 | - /etc/grafana/provisioning/plugins 34 | - /etc/grafana/provisioning/dashboards 35 | - /etc/grafana/provisioning/alerting 36 | tags: configure-grafana 37 | 38 | - name: Copy Grafana Data Source Configuration 39 | template: 40 | dest: /etc/grafana/provisioning/datasources/datasource.yml 41 | force: true 42 | src: grafana/datasource.yml.j2 43 | mode: 0644 44 | tags: configure-grafana 45 | 46 | - name: Copy Grafana Data Source Dashboard General Configuration 47 | template: 48 | dest: /etc/grafana/provisioning/dashboards/dashboard.yml 49 | force: true 50 | src: grafana/dashboard.yml.j2 51 | mode: 0644 52 | tags: configure-grafana 53 | 54 | - name: Copy Grafana Data Source Example Dashboard 55 | template: 56 | dest: /etc/grafana/provisioning/dashboards/dashboard-small.json 57 | force: true 58 | src: grafana/dashboard-small.json.j2 59 | mode: 0644 60 | tags: configure-grafana 61 | 62 | - name: Enable port 3000 in firewall rules 63 | firewalld: 64 | port: 3000/tcp 65 | state: enabled 66 | tags: configure-grafana -------------------------------------------------------------------------------- /roles/configure-docker/tasks/main.yml: -------------------------------------------------------------------------------- 1 | # --------------------------------------------------------------------------- 2 | # Name.......: configure-docker/tasks/main.yml 3 | # Author.....: Martin Berger (mbg) martin.x.berger@accenture.com 4 | # Editor.....: Martin Berger 5 | # Date.......: 2022.11.17 6 | # Revision...: 7 | # Purpose....: YAML file to Configure Docker 8 | # Notes......: 9 | # Reference..: 10 | # License....: Apache License Version 2.0, January 2004 as shown 11 | # at http://www.apache.org/licenses/ 12 | # --------------------------------------------------------------------------- 13 | 14 | --- 15 | 16 | # - Install Docker ----------------------------------------------------------- 17 | # - Install lates version 18 | # - Enable service 19 | # - Configure docker-compose 20 | # ---------------------------------------------------------------------------- 21 | 22 | - name: Add Docker repository 23 | get_url: 24 | url: https://download.docker.com/linux/centos/docker-ce.repo 25 | dest: /etc/yum.repos.d/docker-ce.repo 26 | tags: configure-docker 27 | 28 | - name: Remove unused RunC Lightweight Container 29 | dnf: 30 | name: 31 | - runc 32 | state: absent 33 | tags: configure-docker 34 | 35 | - name: Install the latest version of Docker and Co. 36 | dnf: 37 | name: 38 | - containerd.io 39 | - docker-ce 40 | - docker-ce-cli 41 | state: latest 42 | tags: configure-docker 43 | 44 | - name: Enable service for Docker 45 | ansible.builtin.systemd: 46 | name: docker 47 | enabled: yes 48 | tags: configure-docker 49 | 50 | - name: Make sure a service unit for Docker is running 51 | ansible.builtin.systemd: 52 | state: started 53 | name: docker 54 | tags: configure-docker 55 | 56 | - name: Install docker for Python 3 specifically, using the 'pip3' executable 57 | pip: 58 | name: docker 59 | executable: pip3 60 | tags: configure-docker 61 | 62 | - name: Install Docker-Compose, set permissions and relink 63 | shell: | 64 | curl -L "https://github.com/docker/compose/releases/download/1.29.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose 65 | chmod +x /usr/local/bin/docker-compose 66 | ln -s /usr/local/bin/docker-compose /usr/bin/docker-compose 67 | ignore_errors: yes 68 | tags: configure-docker 69 | -------------------------------------------------------------------------------- /roles/run-docker-compose/templates/docker-compose/docker-compose.yml.j2: -------------------------------------------------------------------------------- 1 | version: '3.1' 2 | 3 | networks: 4 | monitoring: 5 | driver: bridge 6 | 7 | services: 8 | prometheus: 9 | image: prom/prometheus 10 | container_name: prometheus 11 | volumes: 12 | - /etc/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:Z 13 | command: 14 | - '--config.file=/etc/prometheus/prometheus.yml' 15 | - '--storage.tsdb.path=/home/prometheus/data' 16 | - '--storage.tsdb.retention.time=21d' 17 | ports: 18 | - "9090:9090" 19 | restart: unless-stopped 20 | networks: 21 | - monitoring 22 | 23 | pushgateway: 24 | image: prom/pushgateway 25 | container_name: pushgateway 26 | volumes: 27 | - /etc/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:Z 28 | ports: 29 | - "9091:9091" 30 | restart: unless-stopped 31 | 32 | grafana: 33 | image: grafana/grafana 34 | container_name: grafana 35 | volumes: 36 | - /etc/grafana/provisioning:/etc/grafana/provisioning:Z 37 | environment: 38 | GF_SECURITY_ADMIN_USER: "admin" 39 | GF_SECURITY_ADMIN_PASSWORD: "Welcome1" 40 | GF_USERS_ALLOW_SIGN_UP: "false" 41 | ports: 42 | - "3000:3000" 43 | restart: unless-stopped 44 | networks: 45 | - monitoring 46 | 47 | steampipe: 48 | image: turbot/steampipe 49 | container_name: steampipe 50 | command: "service start --foreground" 51 | volumes: 52 | - type: bind 53 | source: /home/steampipe/.oci 54 | target: /home/steampipe/.oci 55 | - type: bind 56 | source: /home/steampipe/sp/config 57 | target: /home/steampipe/.steampipe/config 58 | - type: volume 59 | source: steampipe_data 60 | target: /home/steampipe/.steampipe/db/14.2.0/data 61 | - type: volume 62 | source: steampipe_internal 63 | target: /home/steampipe/.steampipe/internal 64 | - type: volume 65 | source: steampipe_logs 66 | target: /home/steampipe/.steampipe/logs 67 | - type: volume 68 | source: steampipe_plugins 69 | target: /home/steampipe/.steampipe/plugins 70 | ports: 71 | - "9193:9193" 72 | restart: unless-stopped 73 | networks: 74 | - monitoring 75 | 76 | volumes: 77 | steampipe_data: 78 | steampipe_internal: 79 | steampipe_logs: 80 | steampipe_plugins: -------------------------------------------------------------------------------- /roles/configure-steampipe/tasks/main.yml: -------------------------------------------------------------------------------- 1 | # --------------------------------------------------------------------------- 2 | # Name.......: configure-steampipe/tasks/main.yml 3 | # Author.....: Martin Berger (mbg) martin.x.berger@accenture.com 4 | # Editor.....: Martin Berger 5 | # Date.......: 2022.11.17 6 | # Revision...: 7 | # Purpose....: Prepare directories and files for Prometheus 8 | # Notes......: 9 | # Reference..: 10 | # License....: Apache License Version 2.0, January 2004 as shown 11 | # at http://www.apache.org/licenses/ 12 | # --------------------------------------------------------------------------- 13 | 14 | --- 15 | 16 | # - Configure Steampipe ------------------------------------------------------- 17 | # - Install OCI Plugin 18 | # - Get PG Connct String vom Steampipe Configuration 19 | # - Copy oci.spc configuration file 20 | # - Copy Pythion example scripts 21 | # --------------------------------------------------------------------------- 22 | - name: Install OCI Plugin 23 | community.docker.docker_container_exec: 24 | container: steampipe 25 | command: /bin/bash -c "steampipe plugin install oci" 26 | tags: configure-steampipe 27 | 28 | - name: Get PG Connect String 29 | shell: docker exec --tty steampipe steampipe service status --show-password | grep postgres 30 | args: 31 | executable: /bin/bash 32 | register: steampipe_connect_full 33 | tags: configure-steampipe 34 | 35 | - name: Setting facts so that they will be persisted in the fact cache 36 | set_fact: 37 | steampipe_connect_string: "{{ steampipe_connect_full.stdout[22:90] }}" 38 | tags: configure-steampipe 39 | 40 | - debug: msg="{{steampipe_connect_string}}" 41 | tags: configure-steampipe 42 | 43 | - name: Register dummy host with variable to use it in next playbook 44 | add_host: 45 | name: "dummy_host" 46 | transfer_var: "{{ steampipe_connect_string }}" 47 | tags: configure-steampipe 48 | 49 | - name: Remove default file (delete file) 50 | ansible.builtin.file: 51 | path: /home/steampipe/config/oci.spc 52 | state: absent 53 | tags: configure-steampipe 54 | 55 | - name: Copy OCI Steampipe template 56 | template: 57 | dest: /home/steampipe/config/oci.spc 58 | force: true 59 | src: steampipe/etc-oci.spc.j2 60 | mode: 0644 61 | owner: steampipe 62 | group: steampipe 63 | tags: configure-steampipe 64 | 65 | - debug: msg="{{ hostvars['dummy_host']['transfer_var'] }}" 66 | tags: configure-steampipe 67 | 68 | - name: Set Fact for Python Template processing based on Dummy Host variable 69 | set_fact: 70 | steampipe_connect_string: "{{ hostvars['dummy_host']['transfer_var'] }}" 71 | cacheable: no 72 | tags: configure-steampipe 73 | 74 | - name: Copy OCI BV Python Template eu-zurich-1 75 | template: 76 | dest: /home/steampipe/py/psql-example-block-volume-summary.py 77 | force: true 78 | src: postgresql/psql-example-block-volume-summary.py.j2 79 | mode: 0644 80 | owner: steampipe 81 | group: steampipe 82 | tags: configure-steampipe 83 | 84 | - name: Copy OCI CI Python Template eu-zurich-1 85 | template: 86 | dest: /home/steampipe/py/psql-example-compute-instance-running.py 87 | force: true 88 | src: postgresql/psql-example-compute-instance-running.py.j2 89 | mode: 0644 90 | owner: steampipe 91 | group: steampipe 92 | tags: configure-steampipe 93 | 94 | - name: Adding user steampipe to docker group 95 | user: name=steampipe 96 | groups=docker 97 | append=yes 98 | tags: configure-steampipe -------------------------------------------------------------------------------- /roles/configure-grafana/templates/grafana/dashboard-small.json.j2: -------------------------------------------------------------------------------- 1 | { 2 | "annotations": { 3 | "list": [ 4 | { 5 | "builtIn": 1, 6 | "datasource": { 7 | "type": "datasource", 8 | "uid": "grafana" 9 | }, 10 | "enable": true, 11 | "hide": true, 12 | "iconColor": "rgba(0, 211, 255, 1)", 13 | "name": "Annotations & Alerts", 14 | "target": { 15 | "limit": 100, 16 | "matchAny": false, 17 | "tags": [], 18 | "type": "dashboard" 19 | }, 20 | "type": "dashboard" 21 | } 22 | ] 23 | }, 24 | "editable": true, 25 | "fiscalYearStartMonth": 0, 26 | "graphTooltip": 0, 27 | "id": 1, 28 | "links": [], 29 | "liveNow": false, 30 | "panels": [ 31 | { 32 | "datasource": { 33 | "type": "prometheus", 34 | "uid": "my_unique_uid" 35 | }, 36 | "fieldConfig": { 37 | "defaults": { 38 | "color": { 39 | "mode": "palette-classic" 40 | }, 41 | "custom": { 42 | "axisCenteredZero": false, 43 | "axisColorMode": "text", 44 | "axisLabel": "", 45 | "axisPlacement": "auto", 46 | "barAlignment": 0, 47 | "drawStyle": "line", 48 | "fillOpacity": 0, 49 | "gradientMode": "none", 50 | "hideFrom": { 51 | "legend": false, 52 | "tooltip": false, 53 | "viz": false 54 | }, 55 | "lineInterpolation": "linear", 56 | "lineWidth": 1, 57 | "pointSize": 5, 58 | "scaleDistribution": { 59 | "type": "linear" 60 | }, 61 | "showPoints": "auto", 62 | "spanNulls": false, 63 | "stacking": { 64 | "group": "A", 65 | "mode": "none" 66 | }, 67 | "thresholdsStyle": { 68 | "mode": "off" 69 | } 70 | }, 71 | "mappings": [], 72 | "thresholds": { 73 | "mode": "absolute", 74 | "steps": [ 75 | { 76 | "color": "green", 77 | "value": null 78 | }, 79 | { 80 | "color": "red", 81 | "value": 80 82 | } 83 | ] 84 | } 85 | }, 86 | "overrides": [] 87 | }, 88 | "gridPos": { 89 | "h": 9, 90 | "w": 12, 91 | "x": 0, 92 | "y": 0 93 | }, 94 | "id": 2, 95 | "options": { 96 | "legend": { 97 | "calcs": [], 98 | "displayMode": "list", 99 | "placement": "bottom", 100 | "showLegend": true 101 | }, 102 | "tooltip": { 103 | "mode": "single", 104 | "sort": "none" 105 | } 106 | }, 107 | "targets": [ 108 | { 109 | "datasource": { 110 | "type": "prometheus", 111 | "uid": "my_unique_uid" 112 | }, 113 | "editorMode": "code", 114 | "exemplar": true, 115 | "expr": "oci_compute_blockvolumes_summary", 116 | "interval": "", 117 | "legendFormat": "", 118 | "queryType": "randomWalk", 119 | "range": true, 120 | "refId": "A" 121 | } 122 | ], 123 | "title": "OCI Block Volumes eu-zurich-1", 124 | "type": "timeseries" 125 | }, 126 | { 127 | "datasource": { 128 | "type": "prometheus", 129 | "uid": "my_unique_uid" 130 | }, 131 | "fieldConfig": { 132 | "defaults": { 133 | "color": { 134 | "mode": "palette-classic" 135 | }, 136 | "custom": { 137 | "axisCenteredZero": false, 138 | "axisColorMode": "text", 139 | "axisLabel": "", 140 | "axisPlacement": "auto", 141 | "barAlignment": 0, 142 | "drawStyle": "line", 143 | "fillOpacity": 0, 144 | "gradientMode": "none", 145 | "hideFrom": { 146 | "legend": false, 147 | "tooltip": false, 148 | "viz": false 149 | }, 150 | "lineInterpolation": "linear", 151 | "lineWidth": 1, 152 | "pointSize": 5, 153 | "scaleDistribution": { 154 | "type": "linear" 155 | }, 156 | "showPoints": "auto", 157 | "spanNulls": false, 158 | "stacking": { 159 | "group": "A", 160 | "mode": "none" 161 | }, 162 | "thresholdsStyle": { 163 | "mode": "off" 164 | } 165 | }, 166 | "mappings": [], 167 | "thresholds": { 168 | "mode": "absolute", 169 | "steps": [ 170 | { 171 | "color": "green", 172 | "value": null 173 | }, 174 | { 175 | "color": "red", 176 | "value": 80 177 | } 178 | ] 179 | } 180 | }, 181 | "overrides": [] 182 | }, 183 | "gridPos": { 184 | "h": 9, 185 | "w": 12, 186 | "x": 12, 187 | "y": 0 188 | }, 189 | "id": 3, 190 | "options": { 191 | "legend": { 192 | "calcs": [], 193 | "displayMode": "list", 194 | "placement": "bottom", 195 | "showLegend": true 196 | }, 197 | "tooltip": { 198 | "mode": "single", 199 | "sort": "none" 200 | } 201 | }, 202 | "targets": [ 203 | { 204 | "datasource": { 205 | "type": "prometheus", 206 | "uid": "my_unique_uid" 207 | }, 208 | "editorMode": "code", 209 | "exemplar": true, 210 | "expr": "oci_compute_running_summary", 211 | "interval": "", 212 | "legendFormat": "", 213 | "queryType": "randomWalk", 214 | "range": true, 215 | "refId": "A" 216 | } 217 | ], 218 | "title": "OCI Compute Instances eu-zurich-1", 219 | "type": "timeseries" 220 | } 221 | ], 222 | "schemaVersion": 37, 223 | "style": "dark", 224 | "tags": [], 225 | "templating": { 226 | "list": [] 227 | }, 228 | "time": { 229 | "from": "now-6h", 230 | "to": "now" 231 | }, 232 | "timepicker": {}, 233 | "timezone": "", 234 | "title": "OCI Demo - eu-zurich-1", 235 | "uid": "fxzXh8M7k", 236 | "version": 1, 237 | "weekStart": "" 238 | } -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Installation Guide for OCI Monitoring with Steampipe.io 2 | 3 | This guide shows you how to install and setup a nice monitoring solution based on Steampipe.io, Docker and Co. 4 | 5 | Steampipe - select * from cloud - is a powerful tool where you can interact to Cloud providers like Oracle Cloud Infrastructure, Azure, AWS and many more with SQL statements. Steampipe is an open source project and uses plugins to communicate with the providers. In the background, there is a PostgreSQL server running with the Steampipe Postgres Foreign Data Wrapper. The server provides an interface where you can run query against with other languages like Python. In this guide, we install the infrastructure as docker containers, configure the OCI access and gather information by Python scripts to monitor the result in a Grafana dashboard. 6 | 7 | This guide is tested in OL 8 running on Oracle Cloud Infrastructure. 8 | 9 | ## How it works 10 | 11 | ![Architecture](images/architecture.png) 12 | 13 | 1. Execute Python script against steampipe.io by SQL syntax 14 | 2. Steampipe gathers the information from Oracle Cloud Infrastructure 15 | 3. The return value is pushed by the Python script to Prometheus Pushgateway 16 | 4. Prometheus scrapes the metric from the Pushgateway 17 | 5. Grafana reads the metric from Prometheus data source 18 | 19 | ## Installed components by Ansible roles 20 | 21 | - Docker 22 | - Steampipe 23 | - Grafana 24 | - Prometheus 25 | - Pushgateway 26 | - PostgreSQL 27 | 28 | The Docker containers are started by docker-compose. 29 | 30 | ## New OS User Steampipe added 31 | 32 | During the Ansible playbook execution, a new OS user called _steampipe_ is created automatically. This user is used for the OCI CLI and Steampipe.io configuration. 33 | 34 | ## Links 35 | 36 | - [Steampipe](https://steampipe.io/) 37 | - [Steampipe OCI Plugin](https://hub.steampipe.io/plugins/turbot/oci) 38 | - [Prometheus](https://prometheus.io/) 39 | - [Grafana](https://grafana.com/) 40 | - [OCI CLI](https://docs.oracle.com/en-us/iaas/Content/API/Concepts/cliconcepts.htm) 41 | 42 | ## Compute Node Setup 43 | 44 | - OL 8 Compute Instance up and running with Internet access 45 | - SSH keys for user _opc_ 46 | - /etc/hosts configured (done by OCI) 47 | - Ansible and Git packages installed 48 | 49 | ![OCI Compute Image](images/oci_compute_instance.jpg) 50 | 51 | ## Oracle Cloud Infrastructure IAM Requirements 52 | 53 | - An OCI User and Group with _read all-resources_ and _request.operation='GetConfiguration'_ privileges is required to run steampipe.io - see section below. 54 | 55 | ## Oracle Cloud Infrastructure - Create the user for OCI API access - based on OCI CLI 56 | 57 | First we need an OCI group, user and policy for monitoring. If you have admin privileges and an 58 | already configured OCI CLI, you can do it by CLI commands. 59 | 60 | - OCID of created user 61 | - OCID of the tenancy 62 | 63 | User, group and policy can be created in web interface too. 64 | 65 | ### Create Group 66 | 67 | ![OCI Group](images/oci_group_readonly.jpg) 68 | 69 | ```bash 70 | oci iam group create --name oci_group_readonly --description "OCI Group with read all-resources privileges." 71 | ``` 72 | 73 | ### Create IAM User 74 | 75 | ![OCI User](images/oci_user_readonly.jpg) 76 | 77 | ```bash 78 | oci iam user create --name oci_user_readonly --description "OCI User with read all-resources." 79 | ``` 80 | 81 | ### Add User to Group 82 | 83 | ![OCI Group](images/oci_group_user.jpg) 84 | 85 | ```bash 86 | oci iam group add-user \ 87 | --user-id \ 88 | --group-id 89 | ``` 90 | 91 | ### Create Policy 92 | 93 | According Steampipe.io: 94 | 95 | ![OCI Policy](images/oci_policy_readonly.jpg) 96 | 97 | ```bash 98 | oci iam policy create \ 99 | --compartment-id \ 100 | --name oci_policy_readonly \ 101 | --description "OCI Policy with read all-resources." \ 102 | --statements '[ "allow group oci_group_readonly to read all-resources on tenancy","allow group oci_group_readonly to manage all-resources in tenancy where request.operation='GetConfiguration'" ]' \ 103 | ``` 104 | 105 | ### Gather Tenancy OCID Information 106 | 107 | The tenancy OCID will be used later for the OCI CLI configuration. 108 | 109 | Menu -> Governance & Administration -> Tenancy Details. 110 | 111 | ![OCI Policy](images/oci_tenancy_ocid.jpg) 112 | 113 | ## OS Packages - root 114 | 115 | ### Update the OS and install YUM Packages for Ansible and Git 116 | 117 | ```bash 118 | sudo su - 119 | dnf upgrade 120 | dnf install -y ansible git 121 | ``` 122 | 123 | ## GitHub Clone and Ansible Playbook Execution - opc 124 | 125 | ### Clone the GitHub repository to a local folder and change to subdirectory 126 | 127 | As user _opc_, clone the repository and proceed the further steps. 128 | 129 | ```bash 130 | mkdir git 131 | cd git 132 | git clone https://github.com/martinberger-ch/oci-monitoring.git 133 | 134 | cd oci-monitoring 135 | ``` 136 | 137 | ### Install Docker module from the Ansible Galaxy Collection 138 | 139 | Installs the community docker module for Ansible. 140 | 141 | ```bash 142 | ansible-galaxy collection install -r roles/requirements.yml 143 | ``` 144 | 145 | ### Run the Ansible Playbook 146 | 147 | Creates users and directories, installs required software and configures Docker containers. User is _opc_. 148 | 149 | ```bash 150 | ansible-playbook install.yml 151 | ``` 152 | 153 | ## Verification 154 | 155 | Verify all Docker containers are running: 156 | 157 | ```bash 158 | $ sudo docker ps 159 | CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 160 | f7f2e137f4a1 prom/pushgateway "/bin/pushgateway" About an hour ago Up About an hour 0.0.0.0:9091->9091/tcp pushgateway 161 | c6ecc72065c9 prom/prometheus "/bin/prometheus --c…" About an hour ago Up About an hour 0.0.0.0:9090->9090/tcp prometheus 162 | 3485de8cc1f9 grafana/grafana "/run.sh" About an hour ago Up About an hour 0.0.0.0:3000->3000/tcp grafana 163 | 8e821aa0044b turbot/steampipe "docker-entrypoint.s…" About an hour ago Up 30 minutes 0.0.0.0:9193->9193/tcp steampipe 164 | ``` 165 | 166 | ### Network Security 167 | 168 | The Ansible playbooks opens these ports inside the VM for external access. Take care: you need 169 | to open these ports in the OCI VCN Security List too to get web access too. 170 | 171 | - 3000 - Grafana 172 | - 9090 - Prometheus 173 | - 9091 - Prometheus Push Gateway 174 | - 9093 - Steampipe Service 175 | 176 | ### Reachability Verification 177 | 178 | Verify if Grafana is reachable by your workstation - IP: 179 | 180 | ![Grafana Login](images/grafana_login.jpg) 181 | 182 | ## OCI CLI - steampipe 183 | 184 | As OS user _steampipe_, install the OCI CLI and configure it. 185 | 186 | ### Install OCI CLI 187 | 188 | Install and configure the OCI CLI. Press _Enter_ when asked for directory, scripts, modify profile etc. Do not change the settings. 189 | 190 | ```bash 191 | sudo su - steampipe 192 | bash -c "$(curl -L https://raw.githubusercontent.com/oracle/oci-cli/master/scripts/install/install.sh)" 193 | ``` 194 | 195 | ### Configure OCI CLI 196 | 197 | Execute the setup with your user and tenant OCID, create a new API Signing Key Pair without password. This key is later used in OCI web interface. Do not change other settings and let the default values. 198 | 199 | Use these parameters: 200 | 201 | - OCID of created user _oci_user_readonly_ from above 202 | - OCID of the tenancy from above 203 | - Your preferred region - e.g. _eu-zurich-1_. 204 | - Config location: /home/steampipe/.oci/config 205 | 206 | Important: Press Y=yes when asked for a new API Signing RSA key pair. 207 | 208 | ```bash 209 | oci setup config 210 | This command provides a walkthrough of creating a valid CLI config file. 211 | 212 | The following links explain where to find the information required by this 213 | script: 214 | 215 | User API Signing Key, OCID and Tenancy OCID: 216 | 217 | https://docs.cloud.oracle.com/Content/API/Concepts/apisigningkey.htm#Other 218 | 219 | Region: 220 | 221 | https://docs.cloud.oracle.com/Content/General/Concepts/regions.htm 222 | 223 | General config documentation: 224 | 225 | https://docs.cloud.oracle.com/Content/API/Concepts/sdkconfig.htm 226 | 227 | 228 | Enter a location for your config [/home/steampipe/.oci/config]: 229 | Enter a user OCID: 230 | Enter a tenancy OCID: 231 | Enter a region by index or name(e.g. 232 | 1: af-johannesburg-1, 2: ap-chiyoda-1, 3: ap-chuncheon-1, 4: ap-dcc-canberra-1, 5: ap-hyderabad-1, 233 | 6: ap-ibaraki-1, 7: ap-melbourne-1, 8: ap-mumbai-1, 9: ap-osaka-1, 10: ap-seoul-1, 234 | 11: ap-singapore-1, 12: ap-sydney-1, 13: ap-tokyo-1, 14: ca-montreal-1, 15: ca-toronto-1, 235 | 16: eu-amsterdam-1, 17: eu-dcc-milan-1, 18: eu-frankfurt-1, 19: eu-madrid-1, 20: eu-marseille-1, 236 | 21: eu-milan-1, 22: eu-paris-1, 23: eu-stockholm-1, 24: eu-zurich-1, 25: il-jerusalem-1, 237 | 26: me-abudhabi-1, 27: me-dcc-muscat-1, 28: me-dubai-1, 29: me-jeddah-1, 30: mx-queretaro-1, 238 | 31: sa-santiago-1, 32: sa-saopaulo-1, 33: sa-vinhedo-1, 34: uk-cardiff-1, 35: uk-gov-cardiff-1, 239 | 36: uk-gov-london-1, 37: uk-london-1, 38: us-ashburn-1, 39: us-gov-ashburn-1, 40: us-gov-chicago-1, 240 | 41: us-gov-phoenix-1, 42: us-langley-1, 43: us-luke-1, 44: us-phoenix-1, 45: us-sanjose-1): 241 | Do you want to generate a new API Signing RSA key pair? (If you decline you will be asked to supply the path to an existing key.) [Y/n]: Y 242 | Enter a directory for your keys to be created [/home/steampipe/.oci]: 243 | Enter a name for your key [oci_api_key]: 244 | Public key written to: /home/steampipe/.oci/oci_api_key_public.pem 245 | Enter a passphrase for your private key (empty for no passphrase): 246 | Private key written to: /home/steampipe/.oci/oci_api_key.pem 247 | Fingerprint: 72:ef:ef:ad:32:17:23:ac:4d:3c:04:08:ce:e5:ab:aa 248 | Config written to /home/steampipe/.oci/config 249 | 250 | 251 | If you haven't already uploaded your API Signing public key through the 252 | console, follow the instructions on the page linked below in the section 253 | 'How to upload the public key': 254 | 255 | https://docs.cloud.oracle.com/Content/API/Concepts/apisigningkey.htm#How2 256 | 257 | 258 | ``` 259 | 260 | ### Upload API Key 261 | 262 | Copy the content of the public key file created by OCI CLI and add it to the user's API 263 | configuration. 264 | 265 | ```bash 266 | cat /home/steampipe/.oci/oci_api_key_public.pem 267 | ``` 268 | 269 | ![API Key](images/oci_api_key.jpg) 270 | 271 | Verify the functionality of the OCI CLI - get a list of subscribed OCI regions: 272 | 273 | ```bash 274 | oci iam region-subscription list 275 | 276 | { 277 | "data": [ 278 | { 279 | "is-home-region": true, 280 | "region-key": "ZRH", 281 | "region-name": "eu-zurich-1", 282 | "status": "READY" 283 | } 284 | ] 285 | 286 | } 287 | ``` 288 | 289 | ### File /home/steampipe/config/oci.spc - Steampipe Region Filter - steampipe 290 | 291 | The configuration is provided by Ansible and corresponds with the files created during OCI CLI setup. You can rename the connection and filter for your regions. Just edit the file _/home/steampipe/config/oci.spc_ and restart the Steampipe container - example: 292 | 293 | ```bash 294 | connection "oci" { 295 | plugin = "oci" 296 | config_file_profile = "DEFAULT" # Name of the profile 297 | config_path = "~/.oci/config" # Path to config file 298 | regions = ["eu-zurich-1"] # List of regions 299 | } 300 | ``` 301 | 302 | How to restart the Docker container for Steampipe.io as OS user _root_: 303 | 304 | ```bash 305 | sudo su - 306 | # docker stop steampipe 307 | # docker start steampipe 308 | ``` 309 | 310 | ### Steampipe Verification - steampipe 311 | 312 | Verify if Steampipe.io is working properly and the OCI plugin is is installed as expected. Execute as OS user _root_: 313 | 314 | ```bash 315 | # docker exec -it steampipe steampipe plugin list 316 | 317 | +--------------------------------------------+---------+-------------+ 318 | | Installed Plugin | Version | Connections | 319 | +--------------------------------------------+---------+-------------+ 320 | | hub.steampipe.io/plugins/turbot/oci@latest | 0.17.2 | oci | 321 | +--------------------------------------------+---------+-------------+ 322 | 323 | 324 | ``` 325 | 326 | Note: If the _Connections_ columns is empty, restart as user root the steampipe container again and wait a couple of 327 | seconds before re-execute the statement. 328 | 329 | ```bash 330 | # docker stop steampipe 331 | # docker start steampipe 332 | ``` 333 | 334 | Verify the services are is up and running. 335 | 336 | ```bash 337 | # docker exec -it steampipe steampipe service status 338 | Steampipe service is running: 339 | 340 | Database: 341 | 342 | Host(s): localhost, 127.0.0.1, 172.18.0.4 343 | Port: 9193 344 | Database: steampipe 345 | User: steampipe 346 | Password: ********* [use --show-password to reveal] 347 | Connection string: postgres://steampipe@localhost:9193/steampipe 348 | 349 | Managing the Steampipe service: 350 | 351 | # Get status of the service 352 | steampipe service status 353 | 354 | # View database password for connecting from another machine 355 | steampipe service status --show-password 356 | 357 | # Restart the service 358 | steampipe service restart 359 | 360 | # Stop the service 361 | steampipe service stop 362 | 363 | ``` 364 | 365 | Example query for any running Compute Instances in your defined region. 366 | 367 | ```bash 368 | # docker exec -it steampipe steampipe query "select display_name,shape,region from oci_core_instance where lifecycle_state='RUNNING';" 369 | +-----------------------+---------------------+-------------+ 370 | | display_name | shape | region | 371 | +-----------------------+---------------------+-------------+ 372 | | openvpn_access_server | VM.Standard.E2.1 | eu-zurich-1 | 373 | | ci-automation-manager | VM.Standard.E4.Flex | eu-zurich-1 | 374 | | ci-steampipe-v14 | VM.Standard.E4.Flex | eu-zurich-1 | 375 | +-----------------------+---------------------+-------------+ 376 | ``` 377 | 378 | Example query for your home region: 379 | 380 | ```bash 381 | # docker exec -it steampipe steampipe query "select key,title,status from oci_region where is_home_region=true;" 382 | +-----+-------------+--------+ 383 | | key | title | status | 384 | +-----+-------------+--------+ 385 | | ZRH | eu-zurich-1 | READY | 386 | +-----+-------------+--------+ 387 | ``` 388 | 389 | Example query for MFA verification: 390 | 391 | ```bash 392 | # docker exec -it steampipe steampipe query "select name, id, is_mfa_activated from oci_identity_user;" 393 | +-----------------+------------------------+------------------+ 394 | | name | id | is_mfa_activated | 395 | +-----------------+------------------------+------------------+ 396 | | homer_simpson | ocid1.user.oc1.aaaa... | false | 397 | | lisa_simpson | ocid1.user.oc1.aaaa... | true | 398 | | ned_flanders | ocid1.user.oc1.aaaa... | false | 399 | | nelson_muntz | ocid1.user.oc1.aaaa... | false | 400 | +-----------------+------------------------+------------------+ 401 | ``` 402 | 403 | Steampipe is now ready to gather data from the Oracle Cloud Infrastructure Account. 404 | 405 | ## Python Example Scripts 406 | 407 | In subdirectory of new add OS user steampipe _/home/steampipe/py_ there are two basic examples with pre-configured PostgreSQL connect string. There you can see how to get the data from Steampipe PostgreSQL service in Python3 and push them to the Prometheus Pushgateway. Feel free to adapt the queries and files. You can verify the pushed data in browser by URL "http://your-public-ip:9091". If the port is not reachable, check your OCI Security List Ingress settings. 408 | 409 | | Script | Purpose | 410 | |-------------------------------------------|------------------------------------------------------| 411 | | pgsql-example-block-volume-summary.py | Summary of Block Volume in OCI Region Zurich | 412 | | pgsql-example-compute-instance-running.py | Summary of available Instances in OCI Region Zurich | 413 | 414 | Run the script as OS user _steampipe_, example: 415 | 416 | ```bash 417 | $ cd /home/steampipe/py 418 | $ python3 psql-example-compute-instance-running.py 419 | Connected to DB. 420 | Query ran 421 | 3 422 | Connection closed. 423 | ``` 424 | 425 | Behind the Python script - variables like _steamipe_connect_string are replaced during the Ansible deployment. 426 | 427 | ```bash 428 | import psycopg2 429 | from prometheus_client import CollectorRegistry, Gauge, push_to_gateway 430 | from prometheus_client import Summary 431 | 432 | # set postgresql connect string 433 | uri = f'{{ steampipe_connect_string }}' 434 | 435 | try: 436 | # verify connection 437 | con = psycopg2.connect(uri) 438 | print(f'Connected to DB.') 439 | 440 | try: 441 | # execute sql query 442 | cur = con.cursor() 443 | cur.execute('SELECT sum(size_in_gbs) from oci_core_volume where lifecycle_state=\'AVAILABLE\';') 444 | print('Query ran') 445 | except: 446 | print('Query failed') 447 | raise 448 | else: 449 | # set variable with query return value 450 | bv_summary = cur.fetchone()[0] 451 | print(bv_summary) 452 | 453 | if bv_summary is None: 454 | bv_summary = 0 455 | 456 | # prepare pushgateway 457 | registry = CollectorRegistry() 458 | g = Gauge('oci_compute_blockvolumes_summary', 'OCI Compute Block Volumes Summary', registry=registry) 459 | g.set(int(bv_summary)) 460 | 461 | # push data to pushgateway 462 | push_to_gateway('{{ ansible_default_ipv4.address }}:9091', job='oci_blockvolume', registry=registry) 463 | 464 | finally: 465 | con.close() 466 | print(f'Connection closed.') 467 | except Exception as e: 468 | print('Something went wrong:', e) 469 | ``` 470 | 471 | The result is pushed as a metric, this can be verified on the Pushgateway homepage. 472 | 473 | ## Prometheus Push Gateway 474 | 475 | According the Python script, new data is loaded in Prometheus Push Gateway to port 9091 and scraped by Prometheus port 9090. Example for Protheus Gateway where data is loaded by job _oci_compute_. 476 | 477 | ![OCI Prometheus Push Gateway 01](images/oci_pushgateway_01.png) 478 | 479 | ## Grafana 480 | 481 | Grafana is reachable by "http://your-public-ip:3000". 482 | 483 | - Username: admin 484 | - Password: Welcome1 485 | 486 | The Prometheus data source and a basic dashboard are configured during the Grafana Docker setup process. Example for dashboard _OCI Demo - eu-zurich-1_: 487 | 488 | Prometheus data source: 489 | ![OCI Grafana 01](images/oci_grafana_01.png) 490 | 491 | Sample dashboard OCI Demo: 492 | ![OCI Grafana 02](images/oci_grafana_02.png) 493 | 494 | Here you can see the pushed metric from the Python script by name: 495 | ![OCI Grafana 03](images/oci_grafana_03.png) 496 | 497 | ## Troubleshooting 498 | 499 | ### Docker Logs 500 | 501 | To verify if Steampipe is running properly: 502 | 503 | ```bash 504 | # docker logs steampipe 505 | ``` 506 | 507 | User steampipe is not able to run docker commands: 508 | 509 | ```bash 510 | Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get "http://%2Fvar%2Frun%2Fdocker.sock/v1.24/containers/steampipe/json": dial unix /var/run/docker.sock: connect: permission denied 511 | 512 | ``` 513 | 514 | Verify if the docker.sock file has permissions 0666 set. 515 | 516 | ### Steampipe Access Logs 517 | 518 | The foreign data wrapper logs are stored locally on the Docker volume: 519 | 520 | ```bash 521 | [root@ci-steampipe-v14 _data]# pwd 522 | /var/lib/docker/volumes/docker_steampipe_logs/_data 523 | 524 | [root@ci-steampipe-v14 _data]# ls -latr 525 | total 24 526 | -rw-r--r--. 1 steampipe root 0 Nov 10 22:39 plugin-2022-11-10.log 527 | -rw-------. 1 steampipe root 6114 Nov 10 22:39 database-2022-11-10.log 528 | drwx-----x. 3 root root 19 Nov 16 20:18 .. 529 | -rw-r--r--. 1 steampipe root 0 Nov 16 20:19 plugin-2022-11-16.log 530 | drwxr-xr-x. 2 steampipe root 126 Nov 16 20:19 . 531 | -rw-------. 1 steampipe root 16177 Nov 16 20:34 database-2022-11-16.log 532 | 533 | ``` 534 | 535 | ### Steampipe Restart 536 | 537 | ```bash 538 | Something went wrong: no connection config loaded for connection 'oci' 539 | ``` 540 | 541 | Restarting Steampipe as OS user root: 542 | 543 | ```bash 544 | # docker stop steampipe 545 | # docker start steampipe 546 | ``` 547 | 548 | Verify OCI CLI functionality first. 549 | --------------------------------------------------------------------------------