├── .github
├── ISSUE_TEMPLATE
│ └── docs-request.md
└── pull_request_template.md
├── LICENSE
├── README.md
├── docs
├── Reporting_RabbitMQ_For_Kubernetes_Issues.md
└── Reporting_RabbitMQ_Issues.md
└── scripts
├── .gitignore
├── .gitkeep
├── apply-policy-to-all-vhosts
├── rabbitmq-collect-env
├── rabbitmq-collect-env.ps1
└── rabbitmq-k8s-runbook
├── README.md
└── rabbitmq-k8s-diagnostic-runbook
/.github/ISSUE_TEMPLATE/docs-request.md:
--------------------------------------------------------------------------------
1 | ---
2 | name: Docs request
3 | about: For requesting documentation changes.
4 | title: ''
5 | labels: ''
6 | assignees: pstack2021
7 |
8 | ---
9 |
10 | ## Overview of changes requested
11 |
12 | ## Assessment of priority
13 |
14 | Please delete the answers to the questions below as appropriate.
15 |
16 | For example, for the question: **How blocking is this documentation?:** The acceptable answers are: **Very/Somewhat/Not at all** if your answer is **Very** please delete the remaining answers **Somewhat** and **Not at all**.
17 |
18 | **How blocking is this documentation?:** Very/Somewhat/Not at all
19 |
20 | **Is there a user waiting on this change?:** Y/N
21 |
22 | **Roughly how urgently is this documentation needed:** ASAP/This Week/This Month/At some point
23 |
24 | ## Links to related issues / PRs / context
25 |
--------------------------------------------------------------------------------
/.github/pull_request_template.md:
--------------------------------------------------------------------------------
1 | ## Overview of changes
2 |
3 | ## Assessment of priority
4 |
5 | Please delete the answers to the questions below as appropriate.
6 |
7 | For example, for the question: **How blocking is this documentation?:** The acceptable answers are: **Very/Somewhat/Not at all** if your answer is **Very** please delete the remaining answers **Somewhat** and **Not at all**.
8 |
9 | **How blocking is this documentation?:** Very/Somewhat/Not at all
10 |
11 | **Is there a user waiting on this change?:** Y/N
12 |
13 | **Roughly how urgently is this documentation needed:** ASAP/This Week/This Month/At some point
14 |
15 | ## Links to related issues / PRs / context
16 |
--------------------------------------------------------------------------------
/LICENSE:
--------------------------------------------------------------------------------
1 | Apache License
2 | Version 2.0, January 2004
3 | https://www.apache.org/licenses/
4 |
5 | TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
6 |
7 | 1. Definitions.
8 |
9 | "License" shall mean the terms and conditions for use, reproduction,
10 | and distribution as defined by Sections 1 through 9 of this document.
11 |
12 | "Licensor" shall mean the copyright owner or entity authorized by
13 | the copyright owner that is granting the License.
14 |
15 | "Legal Entity" shall mean the union of the acting entity and all
16 | other entities that control, are controlled by, or are under common
17 | control with that entity. For the purposes of this definition,
18 | "control" means (i) the power, direct or indirect, to cause the
19 | direction or management of such entity, whether by contract or
20 | otherwise, or (ii) ownership of fifty percent (50%) or more of the
21 | outstanding shares, or (iii) beneficial ownership of such entity.
22 |
23 | "You" (or "Your") shall mean an individual or Legal Entity
24 | exercising permissions granted by this License.
25 |
26 | "Source" form shall mean the preferred form for making modifications,
27 | including but not limited to software source code, documentation
28 | source, and configuration files.
29 |
30 | "Object" form shall mean any form resulting from mechanical
31 | transformation or translation of a Source form, including but
32 | not limited to compiled object code, generated documentation,
33 | and conversions to other media types.
34 |
35 | "Work" shall mean the work of authorship, whether in Source or
36 | Object form, made available under the License, as indicated by a
37 | copyright notice that is included in or attached to the work
38 | (an example is provided in the Appendix below).
39 |
40 | "Derivative Works" shall mean any work, whether in Source or Object
41 | form, that is based on (or derived from) the Work and for which the
42 | editorial revisions, annotations, elaborations, or other modifications
43 | represent, as a whole, an original work of authorship. For the purposes
44 | of this License, Derivative Works shall not include works that remain
45 | separable from, or merely link (or bind by name) to the interfaces of,
46 | the Work and Derivative Works thereof.
47 |
48 | "Contribution" shall mean any work of authorship, including
49 | the original version of the Work and any modifications or additions
50 | to that Work or Derivative Works thereof, that is intentionally
51 | submitted to Licensor for inclusion in the Work by the copyright owner
52 | or by an individual or Legal Entity authorized to submit on behalf of
53 | the copyright owner. For the purposes of this definition, "submitted"
54 | means any form of electronic, verbal, or written communication sent
55 | to the Licensor or its representatives, including but not limited to
56 | communication on electronic mailing lists, source code control systems,
57 | and issue tracking systems that are managed by, or on behalf of, the
58 | Licensor for the purpose of discussing and improving the Work, but
59 | excluding communication that is conspicuously marked or otherwise
60 | designated in writing by the copyright owner as "Not a Contribution."
61 |
62 | "Contributor" shall mean Licensor and any individual or Legal Entity
63 | on behalf of whom a Contribution has been received by Licensor and
64 | subsequently incorporated within the Work.
65 |
66 | 2. Grant of Copyright License. Subject to the terms and conditions of
67 | this License, each Contributor hereby grants to You a perpetual,
68 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable
69 | copyright license to reproduce, prepare Derivative Works of,
70 | publicly display, publicly perform, sublicense, and distribute the
71 | Work and such Derivative Works in Source or Object form.
72 |
73 | 3. Grant of Patent License. Subject to the terms and conditions of
74 | this License, each Contributor hereby grants to You a perpetual,
75 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable
76 | (except as stated in this section) patent license to make, have made,
77 | use, offer to sell, sell, import, and otherwise transfer the Work,
78 | where such license applies only to those patent claims licensable
79 | by such Contributor that are necessarily infringed by their
80 | Contribution(s) alone or by combination of their Contribution(s)
81 | with the Work to which such Contribution(s) was submitted. If You
82 | institute patent litigation against any entity (including a
83 | cross-claim or counterclaim in a lawsuit) alleging that the Work
84 | or a Contribution incorporated within the Work constitutes direct
85 | or contributory patent infringement, then any patent licenses
86 | granted to You under this License for that Work shall terminate
87 | as of the date such litigation is filed.
88 |
89 | 4. Redistribution. You may reproduce and distribute copies of the
90 | Work or Derivative Works thereof in any medium, with or without
91 | modifications, and in Source or Object form, provided that You
92 | meet the following conditions:
93 |
94 | (a) You must give any other recipients of the Work or
95 | Derivative Works a copy of this License; and
96 |
97 | (b) You must cause any modified files to carry prominent notices
98 | stating that You changed the files; and
99 |
100 | (c) You must retain, in the Source form of any Derivative Works
101 | that You distribute, all copyright, patent, trademark, and
102 | attribution notices from the Source form of the Work,
103 | excluding those notices that do not pertain to any part of
104 | the Derivative Works; and
105 |
106 | (d) If the Work includes a "NOTICE" text file as part of its
107 | distribution, then any Derivative Works that You distribute must
108 | include a readable copy of the attribution notices contained
109 | within such NOTICE file, excluding those notices that do not
110 | pertain to any part of the Derivative Works, in at least one
111 | of the following places: within a NOTICE text file distributed
112 | as part of the Derivative Works; within the Source form or
113 | documentation, if provided along with the Derivative Works; or,
114 | within a display generated by the Derivative Works, if and
115 | wherever such third-party notices normally appear. The contents
116 | of the NOTICE file are for informational purposes only and
117 | do not modify the License. You may add Your own attribution
118 | notices within Derivative Works that You distribute, alongside
119 | or as an addendum to the NOTICE text from the Work, provided
120 | that such additional attribution notices cannot be construed
121 | as modifying the License.
122 |
123 | You may add Your own copyright statement to Your modifications and
124 | may provide additional or different license terms and conditions
125 | for use, reproduction, or distribution of Your modifications, or
126 | for any such Derivative Works as a whole, provided Your use,
127 | reproduction, and distribution of the Work otherwise complies with
128 | the conditions stated in this License.
129 |
130 | 5. Submission of Contributions. Unless You explicitly state otherwise,
131 | any Contribution intentionally submitted for inclusion in the Work
132 | by You to the Licensor shall be under the terms and conditions of
133 | this License, without any additional terms or conditions.
134 | Notwithstanding the above, nothing herein shall supersede or modify
135 | the terms of any separate license agreement you may have executed
136 | with Licensor regarding such Contributions.
137 |
138 | 6. Trademarks. This License does not grant permission to use the trade
139 | names, trademarks, service marks, or product names of the Licensor,
140 | except as required for reasonable and customary use in describing the
141 | origin of the Work and reproducing the content of the NOTICE file.
142 |
143 | 7. Disclaimer of Warranty. Unless required by applicable law or
144 | agreed to in writing, Licensor provides the Work (and each
145 | Contributor provides its Contributions) on an "AS IS" BASIS,
146 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
147 | implied, including, without limitation, any warranties or conditions
148 | of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
149 | PARTICULAR PURPOSE. You are solely responsible for determining the
150 | appropriateness of using or redistributing the Work and assume any
151 | risks associated with Your exercise of permissions under this License.
152 |
153 | 8. Limitation of Liability. In no event and under no legal theory,
154 | whether in tort (including negligence), contract, or otherwise,
155 | unless required by applicable law (such as deliberate and grossly
156 | negligent acts) or agreed to in writing, shall any Contributor be
157 | liable to You for damages, including any direct, indirect, special,
158 | incidental, or consequential damages of any character arising as a
159 | result of this License or out of the use or inability to use the
160 | Work (including but not limited to damages for loss of goodwill,
161 | work stoppage, computer failure or malfunction, or any and all
162 | other commercial damages or losses), even if such Contributor
163 | has been advised of the possibility of such damages.
164 |
165 | 9. Accepting Warranty or Additional Liability. While redistributing
166 | the Work or Derivative Works thereof, You may choose to offer,
167 | and charge a fee for, acceptance of support, warranty, indemnity,
168 | or other liability obligations and/or rights consistent with this
169 | License. However, in accepting such obligations, You may act only
170 | on Your own behalf and on Your sole responsibility, not on behalf
171 | of any other Contributor, and only if You agree to indemnify,
172 | defend, and hold each Contributor harmless for any liability
173 | incurred by, or claims asserted against, such Contributor by reason
174 | of your accepting any such warranty or additional liability.
175 |
176 | END OF TERMS AND CONDITIONS
177 |
178 | APPENDIX: How to apply the Apache License to your work.
179 |
180 | To apply the Apache License to your work, attach the following
181 | boilerplate notice, with the fields enclosed by brackets "{}"
182 | replaced with your own identifying information. (Don't include
183 | the brackets!) The text should be enclosed in the appropriate
184 | comment syntax for the file format. We also recommend that a
185 | file or class name and description of purpose be included on the
186 | same "printed page" as the copyright notice for easier
187 | identification within third-party archives.
188 |
189 | Copyright 2007-2017 Pivotal Software, Inc.
190 |
191 | Licensed under the Apache License, Version 2.0 (the "License");
192 | you may not use this file except in compliance with the License.
193 | You may obtain a copy of the License at
194 |
195 | https://www.apache.org/licenses/LICENSE-2.0
196 |
197 | Unless required by applicable law or agreed to in writing, software
198 | distributed under the License is distributed on an "AS IS" BASIS,
199 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
200 | See the License for the specific language governing permissions and
201 | limitations under the License.
202 |
--------------------------------------------------------------------------------
/README.md:
--------------------------------------------------------------------------------
1 | # RabbitMQ Support and Troubleshooting Extras
2 |
3 | This repository contains tools that are not yet ready to
4 | be included into RabbitMQ distribution, cannot be a plugin
5 | or cannot ship with RabbitMQ for any other reason:
6 |
7 | * [`rabbitmq-collect-env` - Linux/POSIX](./scripts/rabbitmq-collect-env):
8 | collects RabbitMQ and selected OS logs, system-level metrics and other
9 | environment information (iostat, kernel limits and similar) that is not always
10 | directly related to RabbitMQ but can provide additional insights about the
11 | overall state of the node
12 |
13 | * [`rabbitmq-collect-env.ps1` - Windows](./scripts/rabbitmq-collect-env.ps1):
14 | collects RabbitMQ logs, system-level information and other environment
15 | information that can provide additional insights about the overall state of the
16 | node
17 |
18 | ## Copyright & License
19 |
20 | (c) VMware, Inc or its affiliates. 2007-2023
21 |
22 | See LICENSE.
23 |
--------------------------------------------------------------------------------
/docs/Reporting_RabbitMQ_For_Kubernetes_Issues.md:
--------------------------------------------------------------------------------
1 | # Data Collection Form for VMware RabbitMQ for Kubernetes Issues that Require Further Investigation/Escalation
2 |
3 | If a VMware RabbitMQ for Kubernetes issue needs to be assigned to the VMware RabbitMQ for Kubernetes Research and Development team for further investigation and resolution, please answer the questions in this form and submit these answers in the ticket that you are assigning to the research and development to get the issue resolved.
4 |
5 | ## The Research and Development Team Need the Following Information to Work on the Issue
6 |
7 | ### What is the product type?
8 |
9 | * Commercial
10 | * Open source
11 |
12 | ### What is the product version?
13 | #### Commercial: what product version is used?
14 |
15 | The product versions currently in use are: 1.0, 1.1, 1.2, and 1.3. There will be future versions.
16 |
17 | Run the following command to get the product version:
18 |
19 | ``` shell
20 | kubectl get PackageInstall tanzu-rabbitmq -o yaml -n rabbitmq-system | grep "version:"
21 | ```
22 |
23 | #### Open source: What are the operator versions and RabbitMQ images being used?
24 |
25 | Run the following commands to get the operator versions and RabbitMQ images being used:
26 |
27 | ``` shell
28 | kubectl get deployment rabbitmq-cluster-operator -n rabbitmq-system -o yaml | grep " image:"
29 | kubectl get deployment messaging-topology-operator -n rabbitmq-system -o yaml | grep " image:"
30 | kubectl get rabbitmqcluster.rabbitmq.com hello-world -n rabbit-cluster -o yaml | grep " image:"
31 | ```
32 |
33 | ### What is the The Kubernetes distribution?
34 |
35 | Possible options are: TKG, GKE, AKS, EKS, Openshift, Anthos, etc.
36 |
37 | ### What is the Kubernetes version?
38 |
39 | Run the following command to get the Kubernetes version:
40 |
41 | ``` shell
42 | kubectl version
43 | ```
44 |
45 | ### Is the issue related to the installation or upgrade of the product?
46 |
47 | To investigate and provide further information, see [Specific Issues during the Installation of the Commercial Tanzu RabbitMQ for Kubernetes Product](#issues-during-install-upgrade).
48 |
49 | ### Is the issue related to the deployed operators?
50 |
51 | To investigate and provide further information (status, description and logs) about the deployed operators, see [Retrieving Information about the Deployed Operators](#retrieve-operator-details).
52 |
53 | ### Is the issue related to the RabbitMQ cluster object(s)?
54 |
55 | To investigate and provide further information (status, description and logs) about the RabbitMQ cluster object(s), see [Retrieving Information about the RabbitMQ Cluster](#retrieve-cluster-information).
56 |
57 |
58 | ### Provide a detailed description of the scenario causing the issue
59 |
60 | by answering the following questions
61 |
62 | * Is an issue happening during installation or upgrade of the product or running a scenario on a correctly deployed RabbitMQ cluster?
63 | * Is the issue happening with a specific scenario such as standby replication, MQTT, STOMP? If the answer is yes, please see the next question otherwise skip the next question.
64 | * Can the issue still be reproduced with this specific scenario? If the answer is yes, provide as much detail as possible about this scenario including any logs.
65 | * Is the issue impacting a specific Kubernetes Operator (Cluster Operator, Messaging Topology Operator, and Standby Replication Operator) or is the issue impacting RabbitMQ core functionality?
66 |
67 | To retrieve useful informations about the Kubernetes cluster that is deployed such as the number of nodes, the total number of cores,
68 | cpu and memory usage, the network (Calico, CNI, or others), and the storage classes that are in use, run the following commands:
69 |
70 | ``` shell
71 | kubectl get nodes
72 | kubectl describe nodes
73 | kubectl get storageclasses
74 | kubectl describe storageclasses
75 | kubectl top node
76 | kubectl top pod --all-namespaces
77 | ```
78 |
79 | ## Specific issues during the Installation of the Commercial Tanzu RabbitMQ for Kubernetes Version
80 |
81 | If the issue is related to the [installation of the Tanzu RabbitMQ for Kubernetes commercial product using the Carvel toolchain](https://docs.vmware.com/en/VMware-Tanzu-RabbitMQ-for-Kubernetes/1.3/tanzu-rmq/GUID-installation.html), check the following:
82 |
83 | ### Check the Prerequites are Complete and cert-manager is Installed
84 |
85 | **Important:** Some example commands are used in this section. Before running the commands, edit the required parameters and values in these commands based on your own specific environment.
86 |
87 | Verify that the Tanzu Cluster Essentials package is installed on the Kubernetes cluster and the `kapp-controller` and `secretgen-controller` Kubernetes controllers are installed and running. Also, verify that `cert-manager` is installed on the Kubernetes cluster.
88 |
89 | You can to get deployment information for the `kapp-controller` and `secretgen-controller` controllers and `cert-manager` by running the following commands:
90 |
91 | ``` shell
92 | kubectl describe deployment kapp-controller -n kapp-controller | grep "kapp-controller.carvel.dev/version"
93 | kubectl describe deploy secretgen-controller -n secretgen-controller | grep "secretgen-controller.carvel.dev/version"
94 | kubectl describe deployment cert-manager -n cert-manager | grep version
95 | ```
96 |
97 | To get status information and a description of the replicaset and the pods running inside the `kapp-controller` controller as well as review the logs, run the following commands.
98 |
99 | ``` shell
100 | kubectl get all -n kapp-controller
101 | kubectl describe replicaset kapp-controller-54fdd6557d -n kapp-controller
102 | kubectl describe pod kapp-controller-54fdd6557d-782cw -n kapp-controller
103 | kubectl logs kapp-controller-54fdd6557d-782cw -c kapp-controller -n kapp-controller
104 | ```
105 |
106 | To get status information and a description of the replicaset and the pods running inside the `secretgen-controller` controller as well as review the logs, run the following commands:
107 |
108 | ``` shell
109 | kubectl get all -n secretgen-controller
110 | kubectl describe replicaset secretgen-controller-7995bcbd87 -n secretgen-controller
111 | kubect logs secretgen-controller-7995bcbd87-kqctv -n secretgen-controller
112 | ```
113 |
114 | To get the status, description of the `PackageRepository` and `PackageInstall` objects, and review the Yaml definition files of these objects, run the following commands:
115 |
116 | ``` shell
117 | kubectl describe PackageRepository tanzu-rabbitmq-repo
118 | kubectl get PackageRepository tanzu-rabbitmq-repo -o yaml > tanzu-rabbitmq-repo.yml
119 | kubectl describe PackageInstall tanzu-rabbitmq-install
120 | kubectl get PackageInstall tanzu-rabbitmq-install -o yaml > tanzu-rabbitmq.yml
121 | ```
122 |
123 | If the prerequisites are ok, next, check the Operator and RabbitMQ server information.
124 |
125 | ## Retrieving Information about the Deployed Operators
126 |
127 | **Important:** Example commands are used in this section. Before running the commands, edit the required parameters and values in these commands based on your own specific environment.
128 |
129 | If the prerequisites are ok, next, complete checks on the Operators. By default, Operators are installed by the `PackageInstall` object in the `rabbitmq-system` namespace.
130 |
131 | To get the status, description, and definition file for the deployed Cluster Operator, information about the replicaset and the pods running inside
132 | the `rabbitmq-system` namespace, and to review the logs for the Cluster Operator pods, run the following commands.
133 |
134 | ``` shell
135 | kubectl get all -n rabbitmq-system
136 | kubectl describe rs rabbitmq-cluster-operator-767c4c7575 -n rabbitmq-system
137 | kubectl describe deployment rabbitmq-cluster-operator -n rabbitmq-system
138 | kubectl get deployment rabbitmq-cluster-operator -o yaml > rabbitmq-cluster-operator-deploy.yml
139 | kubectl describe pod rabbitmq-cluster-operator-767c4c7575-6bvmp -n rabbitmq-system
140 | kubectl describe pod rabbitmq-cluster-operator-767c4c7575-6bvmp -n rabbitmq-system
141 | kubectl logs rabbitmq-cluster-operator-767c4c7575-6bvmp -n rabbitmq-system > rabbitmq-cluster-operator.log
142 | ```
143 |
144 | If the issue reported is related to a Messaging topology operator or Standby Replication Operator scenario we need informations for these operators too.
145 | To get the status, description, and definition file for the deployed Messaging Topology Operator or Standby Replication Operator, information about the replicaset and the pods running inside the `rabbitmq-system` namespace, and to review the logs for the Messaging Topology Operator or Standby Replication Operator pods, run the following commands.
146 |
147 | ``` shell
148 | kubectl describe rs messaging-topology-operator-678ff579dd -n rabbitmq-system
149 | kubectl describe deployment messaging-topology-operator -n rabbitmq-system
150 | kubectl get deployment messaging-topology-operator -o yaml > rabbitmq-messaging-topology-operator-deploy.yml
151 | kubectl describe pod messaging-topology-operator-678ff579dd-7597g -n rabbitmq-system
152 | kubectl logs messaging-topology-operator-678ff579dd-7597g -n rabbitmq-system > messaging-topology-operator.log
153 | kubectl describe rs standby-replication-operator-545c66cb66 -n rabbitmq-system
154 | kubectl describe deployment standby-replication-operator -n rabbitmq-system
155 | kubectl get deployment standby-replication-operator -o yaml > rabbitmq-standby-replication-operator-deploy.yml
156 | kubectl describe pod standby-replication-operator-545c66cb66-cskph -n rabbitmq-system
157 | kubectl logs standby-replication-operator-545c66cb66-cskph -n rabbitmq-system > rabbitmq-standby-operator.log
158 | ```
159 |
160 | If you are using the messaging topology operator with Cert Manager to manage certificates and you are having issues with TLS connections or certificates it may be a good idea to check the status of the cert-manager namespace objects too with the usual kubectl get/describe command on the objects: deployments, pods and check the logs of the pods too to see if there are issues.
161 |
162 | ``` shell
163 | kubectl get all -n cert-manager
164 | kubectl get logs -l app.kubernetes.io/instance=cert-manager -n cert-manager
165 | ```
166 |
167 | ## Retrieving Information about the RabbitMQ Cluster
168 |
169 | **Important:** Example commands are used in this section. Before running the commands, edit the required parameters and values in these commands based on your own specific environment.
170 |
171 | This section contains specific checks on the RabbitMQ deployed cluster.
172 |
173 | To get the status, description of the statefulset and pods running inside the namespace where RabbitMQ is installed, and the logs for the RabbitMQ cluster, run the following commands:
174 |
175 | ``` shell
176 | kubectl get all -n rabbitmq-cluster
177 | kubectl describe statefulset hello-world-server -n rabbitmq-cluster
178 | kubectl describe pod hello-world-server-0 -n rabbitmq-cluster
179 | kubectl describe pod hello-world-server-1 -n rabbitmq-cluster
180 | kubectl describe pod hello-world-server-2 -n rabbitmq-cluster
181 | kubectl logs hello-world-server-0 -n rabbitmq-cluster > rabbit-server-0.log
182 | kubectl logs hello-world-server-1 -n rabbitmq-cluster > rabbit-server-1.log
183 | kubectl logs hello-world-server-2 -n rabbitmq-cluster > rabbit-server-2.log
184 | ```
185 |
186 | Some informations about the status of the persistent volume/persistent volume claims can be useful too.
187 | Command to use as example (if the broker is deployed in the rabbitmq-system namespace):
188 |
189 | ``` shell
190 | kubectl get pv -n rabbitmq-system
191 | kubectl get pvc -n rabbitmq-system
192 | ```
193 |
194 |
195 | To get the yaml definition for the RabbitMQ cluster, run the following command:
196 | Command to use as example:
197 |
198 | ``` shell
199 | kubectl get rabbitmqcluster.rabbitmq.com rabbit-cluster-name -o yaml > rabbitmq_cluster.yml
200 | ```
201 |
202 | ## Diagnostic tool
203 | We developed a (still experimental) tool in order to collect all these informations based on the commands of this documentation which can be found here:
204 |
205 | https://github.com/rabbitmq/support-tools/tree/main/scripts/rabbitmq-k8s-runbook
206 |
207 |
208 | Note: If the issue is specific to the RabbitMQ core deployment, you must follow the [Best Practices for reporting RabbitMQ issues](https://github.com/rabbitmq/support-tools/blob/master/docs/Reporting_RabbitMQ_Issues.md) guidelines.
209 |
210 |
211 |
--------------------------------------------------------------------------------
/docs/Reporting_RabbitMQ_Issues.md:
--------------------------------------------------------------------------------
1 | # Best Practices for reporting RabbitMQ issues
2 |
3 | When reporting an issue with RabbitMQ, providing as much information as possible will assist the RabbitMQ Core team in determining the issue's root cause.
4 |
5 | ## Gather Information
6 |
7 | The [`rabbitmq-collect-env`](https://raw.githubusercontent.com/rabbitmq/support-tools/master/scripts/rabbitmq-collect-env)
8 | script will run commands to gather system information and RabbitMQ logs. At the
9 | root of the archive created by this script will be a file called `overview`
10 | which contains some important information.
11 |
12 | *Note:* use [`rabbitmq-collect-env.ps1`](https://github.com/rabbitmq/support-tools/blob/main/scripts/rabbitmq-collect-env.ps1)
13 | on Windows systems.
14 |
15 | If the script can't be run, use `rabbitmqctl` commands to get information like
16 | policies, environment and report.
17 |
18 | Other _absolutely necessary_ information includes the following:
19 |
20 | * RabbitMQ version
21 | * Erlang version
22 | * Client library (or libraries) used and versions
23 | * System topology - for every server running RabbitMQ, provide the following information:
24 | * Hostname
25 | * IP address
26 | * Cluster membership
27 | * Node type (disc, ram)
28 |
29 | ## General Questions
30 |
31 | ### Essential Information
32 |
33 | Answering these questions usually pinpoints the root cause of an issue.
34 |
35 | * How long did the RabbitMQ environment work correctly before the issue started?
36 | * When did the issue first occur (including time zone)?
37 | * What changed on or about the same time as when the issue started? _ANY_ changes must be noted:
38 | * Software versions (including operating system patches)
39 | * Environment changes - networking, firewalls
40 | * Client application changes - AMQP library version, client application updates
41 | * Client application workload - did it increase or decrease outside of the normal range?
42 | * Is the issue recurring? If so, is recurrence regular or intermittent?
43 |
44 | ### Other Information
45 |
46 | * Which protocol(s) are being used? e.g. AMQP / MQTT / STOMP
47 | * What exceptions are being returned via the RabbitMQ client library?
48 | * Which connection / channel / queue / exchange are the exceptions for?
49 | * Is a specific RabbitMQ node failing?
50 | * Are multiple RabbitMQ nodes failing at the same time?
51 | * Is the Erlang VM PID changing? (`beam.smp` process)
52 | * Did any `rabbitmqctl` commands fail during the incident? In which way?
53 | * What actions were taken to remedy the failures?
54 | * In which way did the observed behaviour change after these remedies?
55 | * Were all RabbitMQ nodes restarted at once or only the affected node(s)?
56 | * Did any RabbitMQ node remain running during restarts?
57 | * Is the issue limited to certain queues?
58 | * Is the issue limited to certain exchanges?
59 | * Does the issue involve Federated queues or exchanges?
60 | * If Federation is involved, what is the topology?
61 | * If queue Federation is involved, where are messages published and consumed?
62 | * How many connections & channels does the cluster handle?
63 | * How many queues does the RabbitMQ cluster have?
64 | * How many messages are published / consumed on average and during peaks?
65 | * How large are message payloads?
66 | * Are the messages published as persistent & queues durable?
67 |
68 | ### Deployment
69 |
70 | * How is RabbitMQ deployed?
71 | * On which IaaS is the RabbitMQ cluster deployed? AWS / GCP / vSphere / bare-metal
72 | * What network throughput do the RabbitMQ nodes have?
73 | * What network throughput do the client nodes have?
74 | * If applicable, provide details about the load balancer or proxy that sits between RabbitMQ and clients.
75 | * What network throughput do load balancer/proxy nodes have?
76 | * What kind of disks are the RabbitMQ using, and, if known, expected throughput?
77 |
--------------------------------------------------------------------------------
/scripts/.gitignore:
--------------------------------------------------------------------------------
1 | *.txt
2 |
--------------------------------------------------------------------------------
/scripts/.gitkeep:
--------------------------------------------------------------------------------
1 | # RabbitMQ Support and Troubleshooting Extras
2 |
3 | This repository contains tools that are not yet ready to
4 | be included into RabbitMQ distribution, cannot be a plugin
5 | or cannot ship with RabbitMQ for any other reason.
6 |
7 |
8 | ## Scripts
9 |
10 | See `scripts`.
11 |
12 |
13 | ## Copyright & License
14 |
15 | (c) Pivotal Software, Inc. 2007-2017.
16 |
17 | See LICENSE.
18 |
--------------------------------------------------------------------------------
/scripts/apply-policy-to-all-vhosts:
--------------------------------------------------------------------------------
1 | #!/usr/bin/env bash
2 |
3 | # Apply a single policy to all vhosts
4 | #
5 | # https://www.rabbitmq.com/parameters.html
6 | #
7 | # Example:
8 | #
9 | # export POLICY_NAME='limit_hystrix_queues'
10 | # export POLICY_PATTERN='spring.cloud.stream.hystrix.*'
11 | # export POLICY_DEFINITION='{"max-length-bytes":100000}'
12 | # export POLICY_PRIORITY='51'
13 | # export POLICY_APPLY_TO='queues'
14 | #
15 | # apply-policy-to-all-vhosts
16 |
17 | set -e -o pipefail
18 |
19 | rabbitmqctl list_vhosts |
20 | tail -n +2 |
21 | while read -r _vhost
22 | do
23 | rabbitmqctl set_policy -p "$_vhost" \
24 | --priority "${POLICY_PRIORITY:?must be set}" \
25 | --apply-to "${POLICY_APPLY_TO:?must be set}" \
26 | "${POLICY_NAME:?must be set}" \
27 | "${POLICY_PATTERN:?must be set}" \
28 | "${POLICY_DEFINITION:?must be set}"
29 | done
30 |
--------------------------------------------------------------------------------
/scripts/rabbitmq-collect-env:
--------------------------------------------------------------------------------
1 | #!/bin/sh
2 |
3 | ## -------------------------------------------------------------------
4 | ##
5 | ## rabbitmq-collect-env: Collect info for RabbitMQ troubleshooting
6 | ##
7 | ## Based on `riak-debug' as developed by Basho Technologies, Inc
8 | ##
9 | ## Copyright (c) 2017 Basho Technologies, Inc. All Rights Reserved.
10 | ## Copyright (c) 2017 Pivotal Software, Inc. All rights reserved.
11 | ##
12 | ## This file is provided to you under the Apache License,
13 | ## Version 2.0 (the "License"); you may not use this file
14 | ## except in compliance with the License. You may obtain
15 | ## a copy of the License at
16 | ##
17 | ## https://www.apache.org/licenses/LICENSE-2.0
18 | ##
19 | ## Unless required by applicable law or agreed to in writing,
20 | ## software distributed under the License is distributed on an
21 | ## "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
22 | ## KIND, either express or implied. See the License for the
23 | ## specific language governing permissions and limitations
24 | ## under the License.
25 | ##
26 | ## -------------------------------------------------------------------
27 |
28 | set +e
29 |
30 | # value greater than 2 is debug verbosity
31 | verbose=0
32 |
33 | hostname="$(hostname)"
34 | readonly hostname
35 |
36 | output_dir='output_dir_init'
37 | system_output_dir='system_output_dir_init'
38 | rabbitmq_output_dir='rabbitmq_output_dir_init'
39 |
40 | cp_args='-f'
41 | rm_args='-rf'
42 | mv_args='-f'
43 |
44 | onexit()
45 | {
46 | if [ -d "$output_dir" ]
47 | then
48 | if [ $verbose -gt 2 ]
49 | then
50 | printf "[DEBUG] preserving output directory '%s'\\n" "$output_dir"
51 | else
52 | rm "$rm_args" "$output_dir"
53 | fi
54 | fi
55 | }
56 |
57 | trap onexit EXIT
58 |
59 | mkdir_or_die()
60 | {
61 | if [ -d "$1" ]
62 | then
63 | return
64 | fi
65 |
66 | if mkdir -p "$1"
67 | then
68 | if [ $verbose -gt 2 ]
69 | then
70 | printf "[DEBUG] created directory: '%s'\\n" "$1"
71 | fi
72 | else
73 | printf "[ERROR] could not create directory '%s'\\n" "$1" 1>&2
74 | exit 1
75 | fi
76 | }
77 |
78 | command_exists()
79 | {
80 | command -v "$1" > /dev/null 2>&1
81 | }
82 |
83 | user_check()
84 | {
85 | name="$(id -un)"
86 | if [ "$name" = 'root' ]
87 | then
88 | if [ $verbose -gt 2 ]
89 | then
90 | printf "[DEBUG] running %s as root\\n" "$0"
91 | fi
92 | elif [ "$name" = 'rabbitmq' ]
93 | then
94 | printf "[INFO] running as 'rabbitmq' user, some commands may fail. Consider running as 'root' user.\\n"
95 | else
96 | printf "[WARN] script is not running as the 'root' or 'rabbitmq' user, many commands may fail.\\n"
97 | fi
98 | }
99 |
100 | usage()
101 | {
102 | cat << 'EOF'
103 | rabbitmq-collect-env: Collect info from a RabbitMQ node for troubleshooting
104 |
105 | Usage: rabbitmq-collect-env [-h] [-v [-v [-v]]]
106 |
107 | -h, --help Print this usage statement.
108 | -v, --verbose Print verbose messages to stdout. More instances of
109 | this option will result in more verbose output.
110 | EOF
111 | }
112 |
113 | parse_options()
114 | {
115 | while [ -n "$1" ]
116 | do
117 | case "$1" in
118 | -h|--help)
119 | usage
120 | exit 0
121 | ;;
122 | -v|--verbose)
123 | verbose="$((verbose + 1))"
124 | ;;
125 | esac
126 | shift
127 | done
128 | }
129 |
130 | create_output_directories()
131 | {
132 | output_dir="$(mktemp -d)"
133 | top_output_dir="$output_dir/$hostname"
134 | system_output_dir="$top_output_dir/system"
135 | rabbitmq_output_dir="$top_output_dir/rabbitmq"
136 |
137 | mkdir_or_die "$system_output_dir"
138 | mkdir_or_die "$rabbitmq_output_dir"
139 |
140 | if [ $verbose -gt 2 ]
141 | then
142 | printf "[DEBUG] top-level output dir: '%s'\\n" "$output_dir"
143 | fi
144 | }
145 |
146 | build_command_args()
147 | {
148 | if [ $verbose -gt 2 ]
149 | then
150 | cp_args='-vf'
151 | rm_args='-vrf'
152 | mv_args='-vf'
153 | fi
154 | }
155 |
156 | init_environment()
157 | {
158 | # Start PCF / BOSH init
159 |
160 | if [ -s /var/vcap/jobs/rabbitmq-server/env ]
161 | then
162 | # shellcheck source=/dev/null
163 | . /var/vcap/jobs/rabbitmq-server/env
164 | fi
165 |
166 | if [ -d /var/vcap/packages/rabbitmq-server/bin ]
167 | then
168 | export PATH="/var/vcap/packages/rabbitmq-server/bin:$PATH"
169 | fi
170 |
171 | if [ -d /var/vcap/packages/rabbitmq-server/sbin ]
172 | then
173 | export PATH="/var/vcap/packages/rabbitmq-server/sbin:$PATH"
174 | fi
175 |
176 | if [ -d /var/vcap/packages/erlang/bin ]
177 | then
178 | export PATH="/var/vcap/packages/erlang/bin:$PATH"
179 | fi
180 |
181 | if [ -z "$RABBITMQ_ENV" ] && \
182 | [ -f /var/vcap/jobs/rabbitmq-server/packages/rabbitmq-server/privbin/rabbitmq-env ]
183 | then
184 | RABBITMQ_ENV='/var/vcap/jobs/rabbitmq-server/packages/rabbitmq-server/privbin/rabbitmq-env'
185 | fi
186 |
187 | if [ -z "$RABBITMQ_ENV" ] && \
188 | [ -f /var/vcap/jobs/rabbitmq-server/packages/rabbitmq-server/sbin/rabbitmq-env ]
189 | then
190 | RABBITMQ_ENV='/var/vcap/jobs/rabbitmq-server/packages/rabbitmq-server/sbin/rabbitmq-env'
191 | fi
192 |
193 | if [ -d /var/vcap/jobs/rabbitmq-server/packages/rabbitmq-server/etc ]
194 | then
195 | rabbitmq_etc_dir='/var/vcap/jobs/rabbitmq-server/packages/rabbitmq-server/etc'
196 | fi
197 |
198 | if [ -d /var/vcap/jobs/rabbitmq-server/etc ]
199 | then
200 | rabbitmq_etc_dir='/var/vcap/jobs/rabbitmq-server/etc'
201 | fi
202 |
203 | # End PCF / BOSH init
204 |
205 | if [ -z "$RABBITMQ_ENV" ]
206 | then
207 | RABBITMQ_ENV="$(which rabbitmq-env)"
208 | fi
209 |
210 | if [ -z "$RABBITMQ_ENV" ]
211 | then
212 | RABBITMQ_ENV='/usr/lib/rabbitmq/bin/rabbitmq-env'
213 | fi
214 | if [ -z "$RABBITMQ_SCRIPTS_DIR" ]
215 | then
216 | # shellcheck disable=SC2155
217 | RABBITMQ_SCRIPTS_DIR="$(dirname "$RABBITMQ_ENV")"
218 | fi
219 | if [ -s "$RABBITMQ_ENV" ]
220 | then
221 | # shellcheck source=/dev/null
222 | . "$RABBITMQ_ENV"
223 | else
224 | printf "[WARN] expected to find rabbitmq-env at '%s', but file does not exist.\\n" "$RABBITMQ_ENV"
225 | fi
226 |
227 | # NB: it is very important to *not* export these as they will screw up
228 | # execution of rabbitmqctl later on
229 | unset RABBITMQ_ENV
230 | unset RABBITMQ_SCRIPTS_DIR
231 |
232 | if [ -z "$rabbitmq_etc_dir" ]
233 | then
234 | rabbitmq_etc_dir='/etc/rabbitmq'
235 | fi
236 | if [ ! -d "$rabbitmq_etc_dir" ]
237 | then
238 | printf "[WARN] expected to find RabbitMQ 'etc' directory at '%s', but directory does not exist.\\n" "$rabbitmq_etc_dir"
239 | fi
240 |
241 | if [ -z "$RABBITMQ_LOG_BASE" ]
242 | then
243 | if [ -d /var/vcap/sys/log/rabbitmq-server ]
244 | then
245 | RABBITMQ_LOG_BASE=/var/vcap/sys/log/rabbitmq-server
246 | elif [ -d /var/log/rabbitmq ]
247 | then
248 | RABBITMQ_LOG_BASE=/var/log/rabbitmq
249 | fi
250 | fi
251 | if [ ! -d "$RABBITMQ_LOG_BASE" ]
252 | then
253 | printf "[WARN] expected to find log directory at '%s', but directory does not exist.\\n" "$RABBITMQ_LOG_BASE"
254 | fi
255 | }
256 |
257 | ### collect output_file command [cmd_args...]
258 | # Prints the output of a command to the given file.
259 | # Relies on the ${verbose} global.
260 | # When ${verbose} is 0, this script will output
261 | # - `.` for a successfully captured command
262 | # - `_` for a command that could not be found
263 | # - `E` for a command that returned a non-zero retval
264 | # When $verbose is 1, failed commands will be logged.
265 | # When $verbose is 2, missing commands will logged as well.
266 | # When $verbose is 3, successful commands will be logged as well.
267 | collect()
268 | {
269 | # Capture and shift away the output directory.
270 | collect_output_dir="$1"
271 | shift
272 |
273 | if [ ! -d "$collect_output_dir" ]
274 | then
275 | printf "[ERROR] collect - expected directory '%s' to exist!\\n" "$collect_output_dir" 1>&2
276 | exit 1
277 | fi
278 |
279 | collect_output_info_dir="$collect_output_dir/.info"
280 | mkdir_or_die "$collect_output_info_dir"
281 |
282 | # Capture and shift away the output file name so we can handle target
283 | # command and arguments as a single variable.
284 | collect_outfile="$1"
285 | shift
286 |
287 | # If the executable can't be found, log the miss and return.
288 | if ! command_exists "$1"
289 | then
290 | if [ $verbose -gt 1 ]
291 | then
292 | printf "[WARN] command '%s' not found.\\n" "$1"
293 | else
294 | printf '_'
295 | fi
296 | return
297 | fi
298 |
299 | if [ $verbose -gt 2 ]
300 | then
301 | printf "[DEBUG] running '%s'\\n" "$*"
302 | fi
303 | # shellcheck disable=SC2048
304 | $* < /dev/null >> "$collect_output_dir/$collect_outfile" 2>&1
305 | collect_rv=$?
306 |
307 | # Record the text of the command and the returned value in the .info/$out
308 | # file to aid automation.
309 | # Note: this will miss some escaping for, e.g., find, but it's not critical.
310 | # The command that was run can be fetched with `head -1 .info/$out`.
311 | # The returned value can be fetched with `tail -1 .info/$out`.
312 | echo "$*" > "$collect_output_info_dir/$collect_outfile"
313 | echo $collect_rv >> "$collect_output_info_dir/$collect_outfile"
314 |
315 | if [ $collect_rv -eq 0 ]
316 | then
317 | if [ $verbose -gt 2 ]
318 | then
319 | printf "[DEBUG] '%s' exit code: %d\\n" "$*" "$collect_rv"
320 | else
321 | printf '.'
322 | fi
323 | else
324 | if [ $verbose -gt 0 ]
325 | then
326 | printf "[ERROR] command '%s' failed, exit code: %d\\n" "$*" "$collect_rv" 1>&2
327 | else
328 | printf 'E'
329 | fi
330 | fi
331 |
332 | return $collect_rv
333 | }
334 |
335 | collect_system()
336 | {
337 | collect "$system_output_dir" "$@"
338 | }
339 |
340 | collect_system_info()
341 | {
342 | collect_system blkid blkid
343 | collect_system date date
344 | collect_system df_h df -h
345 | collect_system df_i df -i
346 | collect_system dmesg dmesg
347 | collect_system dmesg_t dmesg -T # NB: This will fail on most systems.
348 | collect_system dmidecode dmidecode
349 | collect_system dpkg dpkg -l
350 | collect_system printenv printenv
351 | collect_system fdisk fdisk -l
352 | collect_system free free -h
353 | collect_system hostname hostname
354 | collect_system ifconfig ifconfig -a
355 | collect_system java_version java -version
356 | collect_system last last
357 | collect_system proc_uptime cat /proc/uptime
358 | collect_system lsb_release lsb_release -a
359 | collect_system lsblk lsblk -a
360 | collect_system lscpu lscpu
361 | collect_system lsof_tcp lsof -nPi TCP
362 | collect_system mount mount
363 | collect_system netstat_an netstat -an
364 | collect_system netstat_i netstat -i
365 | collect_system netstat_rn netstat -rn
366 | collect_system netstat_s netstat -s
367 | collect_system pfctl_nat pfctl -s nat
368 | collect_system pfctl_rules pfctl -s rules
369 | collect_system pkg_info pkg_info
370 | collect_system ps ps aux
371 | collect_system pstree pstree -panl
372 | collect_system rpm rpm -qa
373 | collect_system sestatus sestatus -v
374 | collect_system sysctl sysctl -a
375 | collect_system uname uname -a
376 | collect_system uptime uptime
377 | collect_system vmstat vmstat -S M 1 10
378 | collect_system w w
379 | collect_system zfs_list zfs list
380 | collect_system zpool_list zpool list
381 |
382 | # If swapctl exists, prefer it over swapon
383 | if command_exists swapctl
384 | then
385 | collect_system swapctl swapctl -s
386 | else
387 | collect_system swapon swapon -s
388 | fi
389 |
390 | # Get device readahead
391 | if command_exists blockdev
392 | then
393 | for mount_point in $(mount | awk '/^\// { print $1 }')
394 | do
395 | flat_point="$(echo "$mount_point" | tr '/' '_')"
396 | collect_system "blockdev$flat_point" blockdev --getra "$mount_point"
397 | done
398 | else
399 | collect_system blockdev_not_available echo 'blockdev command is not available'
400 | fi
401 |
402 | if command_exists systemctl
403 | then
404 | collect_system systemctl_show_rabbitmq systemctl show rabbitmq-server
405 | collect_system systemctl_status_rabbitmq systemctl status rabbitmq-server
406 | else
407 | collect_system systemctl_not_available echo 'systemctl command is not available'
408 | fi
409 |
410 | # Running iptables commands if the module is not loaded can automatically
411 | # load them. This is rarely desired and can even cause connectivity
412 | # problems if, e.g., nf_conntrack gets autoloaded and autoenabled.
413 | if [ -f /proc/modules ]
414 | then
415 | if grep '^iptable_filter' /proc/modules > /dev/null 2>&1
416 | then
417 | collect_system iptables_rules iptables -n -L
418 | else
419 | collect_system iptables_rules echo "iptables module not loaded"
420 | fi
421 |
422 | if grep '^nf_conntrack' /proc/modules > /dev/null 2>&1
423 | then
424 | collect_system iptables_nat iptables -t nat -n -L
425 | else
426 | collect_system iptables_nat echo "nf_conntrack module not loaded"
427 | fi
428 | fi
429 |
430 | # Capture iostat, based on (a guess of) which distribution is running.
431 | case "$(uname)" in
432 | Linux)
433 | collect_system iostat_linux iostat -txm 1 10;;
434 | *)
435 | collect_system iostat_bsd iostat -dIw 1 -c 5;;
436 | esac
437 |
438 | ## Collect Files
439 | [ -f /etc/release ] && collect_system release cat /etc/release
440 | [ -f /etc/redhat-release ] && collect_system redhat_release cat /etc/redhat-release
441 | [ -f /etc/debian_version ] && collect_system debian_version cat /etc/debian_version
442 | [ -f /etc/security/limits.conf ] && collect_system limits_conf cat /etc/security/limits.conf
443 |
444 | [ -f /var/log/messages ] && collect_system messages cat /var/log/messages
445 | [ -f /var/log/syslog ] && collect_system messages cat /var/log/syslog
446 | [ -f /var/log/kern.log ] && collect_system messages cat /var/log/kern.log
447 |
448 | [ -f /proc/version ] && collect_system proc_version cat /proc/version
449 | [ -f /proc/cpuinfo ] && collect_system proc_cpuinfo cat /proc/cpuinfo
450 | [ -f /proc/meminfo ] && collect_system proc_meminfo cat /proc/meminfo
451 | [ -f /proc/diskstats ] && collect_system proc_diskstats cat /proc/diskstats
452 | [ -f /proc/vmstat ] && collect_system proc_vmstat cat /proc/vmstat
453 |
454 | ## Collect Directories and Finds
455 | [ -d /dev/disk/by-id ] && collect_system disk_by_id \
456 | ls -l /dev/disk/by-id
457 | [ -d /sys/block ] && collect_system schedulers \
458 | find /sys/block/ -type l -print \
459 | -exec cat {}/queue/scheduler \;
460 | [ -d /proc/net/bonding ] && collect_system bonding \
461 | find /proc/net/bonding/ -type f -print \
462 | -exec cat {} \;
463 | [ -d /sys/class/net ] && collect_system rx_crc_errors \
464 | find /sys/class/net/ -type l -print \
465 | -exec cat {}/statistics/rx_crc_errors \;
466 |
467 | if [ -d /etc/security/limits.d ] && ls -1 /etc/security/limits.d/*.conf > /dev/null 2>&1
468 | then
469 | collect_limits_dir="$system_output_dir/limits.d"
470 | mkdir_or_die "$collect_limits_dir"
471 | cp "$cp_args" /etc/security/limits.d/*.conf "$collect_limits_dir"
472 | fi
473 | }
474 |
475 | collect_rabbitmq()
476 | {
477 | collect "$rabbitmq_output_dir" "$@"
478 | }
479 |
480 | collect_rabbitmq_info()
481 | {
482 | set > "$rabbitmq_output_dir/shell_variables"
483 |
484 | if [ -d "$rabbitmq_etc_dir" ]
485 | then
486 | readonly rabbitmq_output_etc_dir="$rabbitmq_output_dir/etc"
487 | mkdir_or_die "$rabbitmq_output_etc_dir"
488 | # gh-31
489 | # Rather than copying everything in /etc/rabbitmq, only copy *.conf* files and enabled_plugins
490 | if [ -d "$rabbitmq_etc_dir" ] && ls -1 "$rabbitmq_etc_dir"/*.conf* > /dev/null 2>&1
491 | then
492 | if [ $verbose -gt 2 ]
493 | then
494 | echo "[DEBUG] running command 'cp $cp_args \"$rabbitmq_etc_dir/*.conf*\" \"$rabbitmq_output_etc_dir\""
495 | else
496 | printf '.'
497 | fi
498 | cp "$cp_args" "$rabbitmq_etc_dir"/*.conf* "$rabbitmq_output_etc_dir"
499 | fi
500 |
501 | readonly rabbitmq_enabled_plugins_file="$rabbitmq_etc_dir/enabled_plugins"
502 | if [ -f "$rabbitmq_enabled_plugins_file" ]
503 | then
504 | if [ $verbose -gt 2 ]
505 | then
506 | echo "[DEBUG] running command 'cp $cp_args \"$rabbitmq_enabled_plugins_file\" \"$rabbitmq_output_etc_dir\""
507 | else
508 | printf '.'
509 | fi
510 | cp "$cp_args" "$rabbitmq_enabled_plugins_file" "$rabbitmq_output_etc_dir"
511 | fi
512 |
513 | readonly rabbitmq_etc_confd_dir="$rabbitmq_etc_dir/conf.d"
514 | if [ -d "$rabbitmq_etc_confd_dir" ] && ls -1 "$rabbitmq_etc_confd_dir"/* > /dev/null 2>&1
515 | then
516 | readonly rabbitmq_output_etc_confd_dir="$rabbitmq_output_etc_dir/conf.d"
517 | mkdir_or_die "$rabbitmq_output_etc_confd_dir"
518 | if [ $verbose -gt 2 ]
519 | then
520 | echo "[DEBUG] running command 'cp $cp_args \"$rabbitmq_etc_confd_dir/*\" \"$rabbitmq_output_etc_confd_dir\""
521 | else
522 | printf '.'
523 | fi
524 | cp "$cp_args" "$rabbitmq_etc_confd_dir"/* "$rabbitmq_output_etc_confd_dir"
525 | fi
526 | else
527 | printf "[WARN] '%s' directory not present.\\n" "$rabbitmq_etc_dir"
528 | fi
529 |
530 | if [ -d "$RABBITMQ_LOG_BASE" ]
531 | then
532 | rabbitmq_output_logs_dir="$rabbitmq_output_dir/logs"
533 | mkdir_or_die "$rabbitmq_output_logs_dir"
534 | if [ $verbose -gt 2 ]
535 | then
536 | echo "[DEBUG] running command 'tar -C \"$RABBITMQ_LOG_BASE\" -cf - | tar -C \"$rabbitmq_output_logs_dir\" -xf -'"
537 | else
538 | printf '.'
539 | fi
540 | tar -C "$RABBITMQ_LOG_BASE" -cf - . | tar -C "$rabbitmq_output_logs_dir" -xf -
541 | else
542 | printf "[WARN] '%s' directory not present.\\n" "$RABBITMQ_LOG_BASE"
543 | fi
544 |
545 | readonly with_timeout="timeout --kill-after 30 120"
546 |
547 | # shellcheck disable=SC2086
548 | collect_rabbitmq rabbitmqctl_vhosts $with_timeout rabbitmqctl list_vhosts name default_queue_type cluster_state
549 |
550 | # shellcheck disable=SC2086
551 | collect_rabbitmq rabbitmqctl_status $with_timeout rabbitmqctl status
552 |
553 | # shellcheck disable=SC2086
554 | collect_rabbitmq rabbitmqctl_cluster_status $with_timeout rabbitmqctl cluster_status
555 |
556 | # shellcheck disable=SC2086
557 | collect_rabbitmq rabbitmqctl_maybe_stuck $with_timeout rabbitmqctl eval 'rabbit_diagnostics:maybe_stuck().'
558 |
559 | pid=$($with_timeout rabbitmqctl eval 'os:getpid().' | tr -d '""')
560 | [ -f "/proc/$pid/limits" ] && collect_rabbitmq rabbitmq_pid_limits cat "/proc/$pid/limits"
561 |
562 | # shellcheck disable=SC2086
563 | collect_rabbitmq rabbitmqctl_process_limit $with_timeout rabbitmqctl eval 'erlang:system_info(process_limit).'
564 |
565 | # shellcheck disable=SC2086
566 | collect_rabbitmq rabbitmqctl_report $with_timeout rabbitmqctl report
567 |
568 | # shellcheck disable=SC2086
569 | collect_rabbitmq rabbitmqctl_inet_i $with_timeout rabbitmqctl eval 'inet:i().'
570 | }
571 |
572 | collect_overview_info()
573 | {
574 | overview_file="$top_output_dir/overview"
575 | ip_addresses="$(ifconfig | grep 'inet addr:' | cut -d: -f2 | awk '{ print $1 }')"
576 | rabbitmq_version="$(rabbitmqctl eval '{_,_,V}=proplists:lookup(rabbit,application:which_applications()),V.' | tr -d '"')"
577 | erlang_version="$(rabbitmqctl eval 'erlang:system_info(otp_release).' | tr -d '"')"
578 | {
579 | echo "Hostname: $hostname"
580 | echo "IP Addresses:"
581 | echo "$ip_addresses"
582 | echo "RabbitMQ: $rabbitmq_version"
583 | echo "Erlang: $erlang_version"
584 | } > "$overview_file" 2>&1
585 | }
586 |
587 |
588 | build_output_archive()
589 | {
590 | if [ $verbose -gt 2 ]
591 | then
592 | echo "[DEBUG] running command 'tar -C \"$output_dir\" -zcf \"$RABBITMQ_LOG_BASE/rabbitmq-env-$hostname.tgz\"'"
593 | else
594 | printf '.'
595 | fi
596 |
597 | old_output_archive_dir="$(mktemp -d)"
598 | readonly old_output_archive_dir
599 |
600 | output_archive_dir="$(mktemp -d)"
601 | readonly output_archive_dir
602 |
603 | output_archive="$output_archive_dir/rabbitmq-env-$hostname-$(date '+%Y%m%d-%H%M%S').tgz"
604 | readonly output_archive
605 |
606 | printf "[INFO] moving old tgz archives (if any) to : '%s'\\n" "$old_output_archive_dir"
607 | find "$output_dir" -type f -name 'rabbitmq-env-*.tgz' -exec mv "$mv_args" {} "$old_output_archive_dir" \;
608 |
609 | tar -C "$output_dir" -zcf "$output_archive" .
610 | printf "[INFO] output archive: '%s'\\n" "$output_archive"
611 | }
612 |
613 | init()
614 | {
615 | parse_options "$@"
616 | user_check
617 | create_output_directories
618 | build_command_args
619 | init_environment
620 | }
621 |
622 | main()
623 | {
624 | init "$@"
625 | collect_system_info
626 | collect_rabbitmq_info
627 | collect_overview_info
628 | build_output_archive
629 | }
630 |
631 | main "$@"
632 |
--------------------------------------------------------------------------------
/scripts/rabbitmq-collect-env.ps1:
--------------------------------------------------------------------------------
1 | param(
2 | [switch]$Debug = $false,
3 | [switch]$Verbose = $false,
4 | [int]$DebugLevel = 0,
5 | [int]$VerboseLevel = 0,
6 | [int]$OutputWidth = 8192
7 | )
8 |
9 | ## -------------------------------------------------------------------
10 | ##
11 | ## rabbitmq-collect-env.ps1:
12 | ## Collect artifacts for RabbitMQ troubleshooting on Windows systems
13 | ##
14 | ## Based on `riak-debug' as developed by Basho Technologies, Inc
15 | ##
16 | ## Copyright (c) 2017 Basho Technologies, Inc. All Rights Reserved.
17 | ## Copyright (c) 2007-2023 VMware, Inc. or its affiliates. All rights reserved.
18 | ##
19 | ## This file is provided to you under the Apache License,
20 | ## Version 2.0 (the "License"); you may not use this file
21 | ## except in compliance with the License. You may obtain
22 | ## a copy of the License at
23 | ##
24 | ## https://www.apache.org/licenses/LICENSE-2.0
25 | ##
26 | ## Unless required by applicable law or agreed to in writing,
27 | ## software distributed under the License is distributed on an
28 | ## "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
29 | ## KIND, either express or implied. See the License for the
30 | ## specific language governing permissions and limitations
31 | ## under the License.
32 | ##
33 | ## -------------------------------------------------------------------
34 |
35 | $ErrorActionPreference = 'Stop'
36 |
37 | if ($Debug)
38 | {
39 | $DebugPreference = 'Continue'
40 | Write-Debug -Message 'Enabling debug output'
41 | if ($DebugLevel -gt 0)
42 | {
43 | Set-PSDebug -Strict -Trace 1
44 | }
45 | }
46 | else
47 | {
48 | $DebugPreference = 'SilentlyContinue'
49 | Set-PSDebug -Off
50 | }
51 |
52 | if ($Verbose)
53 | {
54 | $VerbosePreference = 'Continue'
55 | Write-Verbose -Message 'Enabling verbose output'
56 | }
57 | else
58 | {
59 | $VerbosePreference = 'SilentlyContinue'
60 | }
61 |
62 | Set-StrictMode -Version Latest -ErrorAction 'Stop'
63 |
64 | function New-TemporaryDirectory
65 | {
66 | New-Variable -Name parent -Option Constant -Value ([System.IO.Path]::GetTempPath())
67 | Do
68 | {
69 | $tmpDir = New-Item -ItemType Directory -Path (Join-Path -Path $parent -ChildPath (New-Guid))
70 | }
71 | While (-Not $tmpDir)
72 |
73 | return $tmpDir
74 | }
75 |
76 | function Get-Times
77 | {
78 | New-Variable -Name now -Option Constant -Value (Get-Date)
79 |
80 | # Local Time
81 | New-Variable -Name nowLocalStr -Option Constant -Value ($now.ToString("yyyy-MM-ddTHH:mm:ss.fffK"))
82 |
83 | # UTC
84 | New-Variable -Name nowUTC -Option Constant -Value ($now.ToUniversalTime())
85 | New-Variable -Name nowUTCStr -Option Constant -Value ($nowUTC.ToString("yyyyMMddTHHmmssZ"))
86 | New-Variable -Name nowUTCFileName -Option Constant -Value ($nowUTC.ToString("yyyyMMddTHHmmssZ"))
87 |
88 | $rv = New-Object PsObject -Property @{
89 | Now = $now;
90 | NowLocal = $nowLocalStr;
91 | NowUTC = $nowUTCStr;
92 | NowFileName = $nowUTCFileName;
93 | }
94 |
95 | return $rv
96 | }
97 |
98 | function Run-Init
99 | {
100 | New-Variable -Name t -Option Constant -Value (Get-Times)
101 |
102 | $startMsg = "START TIME UTC: $($t.NowUTC) LOCAL: $($t.NowLocal)"
103 | Write-Verbose -Message $startMsg
104 |
105 | New-Variable -Name curdir -Scope Script -Option Constant -Value $PSScriptRoot
106 | Write-Verbose -Message "curdir: $curdir"
107 |
108 | New-Variable -Name outputFile -Scope Script -Option Constant -Value `
109 | $(Join-Path -Path $curdir -ChildPath "rabbitmq-collect-env-output-$($t.NowFileName).txt")
110 | Write-Verbose -Message "outputFile: $outputFile"
111 |
112 | New-Variable -Name logsArchiveFile -Scope Script -Option Constant -Value `
113 | $(Join-Path -Path $curdir -ChildPath "rabbitmq-collect-env-logs-$($t.NowFileName).zip")
114 | Write-Verbose -Message "logsArchiveFile: $logsArchiveFile"
115 |
116 | $initOutFileArgs = @{
117 | Append = $true
118 | FilePath = $outputFile
119 | Width = $OutputWidth
120 | Encoding = 'UTF8'
121 | }
122 | if ($VerboseLevel -gt 0) {
123 | $initOutFileArgs += @{
124 | Verbose = $true
125 | }
126 | } else {
127 | $initOutFileArgs += @{
128 | Verbose = $false
129 | }
130 | }
131 | New-Variable -Name outFileArgs -Scope Script -Option Constant -Value $initOutFileArgs
132 |
133 | $initAddContentArgs = @{
134 | Path = $outputFile
135 | }
136 | if ($VerboseLevel -gt 0) {
137 | $initAddContentArgs += @{
138 | Verbose = $true
139 | }
140 | } else {
141 | $initAddContentArgs += @{
142 | Verbose = $false
143 | }
144 | }
145 | New-Variable -Name addContentArgs -Scope Script -Option Constant -Value $initAddContentArgs
146 |
147 | New-Variable -Name sep -Scope Script -Option Constant -Value '------------------------------------------------------------------------------------------------------------------------------------'
148 |
149 | Remove-Item -ErrorAction 'SilentlyContinue' -Force -LiteralPath $outputFile
150 |
151 | Add-Content @addContentArgs -Value "$sep$([Environment]::NewLine)$startMsg"
152 | }
153 |
154 | function Get-ErlangInfo
155 | {
156 | $msg = 'Erlang information'
157 | Write-Verbose -Message $msg
158 | Add-Content @addContentArgs -Value "$sep$([Environment]::NewLine)$msg$([Environment]::NewLine)"
159 |
160 | New-Variable -Name erts_version -Option Constant `
161 | -Value (Get-ChildItem -LiteralPath HKLM:\SOFTWARE\WOW6432Node\Ericsson\Erlang | Select-Object -Last 1).PSChildName
162 |
163 | New-Variable -Name erlangProgramFilesPath -Option Constant `
164 | -Value ((Get-ItemProperty -LiteralPath HKLM:\SOFTWARE\WOW6432Node\Ericsson\Erlang\$erts_version).'(default)')
165 |
166 | New-Variable -Name erl_exe -Option Constant `
167 | -Value (Join-Path -Path $erlangProgramFilesPath -ChildPath 'bin' | Join-Path -ChildPath 'erl.exe')
168 |
169 | New-Variable -Name otp_version -Option Constant `
170 | -Value $(& $erl_exe -boot no_dot_erlang -noshell -eval "{ok,Version}=file:read_file(filename:join([code:root_dir(),'releases',erlang:system_info(otp_release),'OTP_VERSION'])),io:fwrite(Version),halt().")
171 |
172 | $msg = "otp_version: $otp_version"
173 | Write-Verbose -Message $msg
174 | Add-Content @addContentArgs -Value $msg
175 |
176 | $msg = "erts_version: $erts_version"
177 | Write-Verbose -Message $msg
178 | Add-Content @addContentArgs -Value $msg
179 |
180 | $msg = "erlangProgramFilesPath: $erlangProgramFilesPath"
181 | Write-Verbose -Message $msg
182 | Add-Content @addContentArgs -Value $msg
183 |
184 | $msg = "erl_exe: $erl_exe"
185 | Write-Verbose -Message $msg
186 | Add-Content @addContentArgs -Value $msg
187 | }
188 |
189 | function Get-RabbitMQInfo
190 | {
191 | $msg = 'RabbitMQ information'
192 | Write-Verbose -Message $msg
193 | Add-Content @addContentArgs -Value "$sep$([Environment]::NewLine)$msg$([Environment]::NewLine)"
194 |
195 | New-Variable -Name rmqInstallDir -Option Constant `
196 | -Value (Resolve-Path -LiteralPath (Get-ItemProperty -Name Install_Dir -LiteralPath 'HKLM:\SOFTWARE\WOW6432Node\VMware, Inc.\RabbitMQ Server').Install_Dir)
197 |
198 | $msg = "RabbitMQ installation directory: $rmqInstallDir"
199 | Write-Verbose -Message $msg
200 | Add-Content @addContentArgs -Value $msg
201 |
202 | New-Variable -Name rmqServerDir -Option Constant `
203 | -Value (Resolve-Path -LiteralPath (Get-ChildItem -LiteralPath $rmqInstallDir -Filter 'rabbitmq_server-*' | Select-Object -First 1).FullName)
204 | $msg = "RabbitMQ server installation directory: $rmqServerDir"
205 | Write-Verbose -Message $msg
206 | Add-Content @addContentArgs -Value $msg
207 |
208 | New-Variable -Name rmqServerSbinDir -Option Constant `
209 | -Value (Resolve-Path -LiteralPath (Join-Path -Path $rmqServerDir.Path -ChildPath 'sbin'))
210 | $msg = "RabbitMQ server sbin directory: $rmqServerSbinDir"
211 | Write-Verbose -Message $msg
212 | Add-Content @addContentArgs -Value $msg
213 |
214 | New-Variable -Name rmqEnvBat -Option Constant `
215 | -Value (Resolve-Path -LiteralPath (Join-Path -Path $rmqServerSbinDir.Path -ChildPath 'rabbitmq-env.bat'))
216 | $msg = "RabbitMQ environment batch file: $rmqEnvBat"
217 | Add-Content @addContentArgs -Value $msg
218 |
219 | New-Variable -Name rabbitmqctl -Option Constant `
220 | -Value (Resolve-Path -LiteralPath (Join-Path -Path $rmqServerSbinDir.Path -ChildPath 'rabbitmqctl.bat'))
221 | $msg = "RabbitMQ rabbitmqctl.bat command: $($rabbitmqctl.Path)"
222 | Write-Verbose -Message $msg
223 | Add-Content @addContentArgs -Value $msg
224 |
225 | ########################################################################
226 | # Get RabbitMQ configuration from rabbit_prelaunch
227 | #
228 | $tmpFile = New-TemporaryFile
229 | $tmpFileName = $tmpFile.FullName -replace '\\','/'
230 | try
231 | {
232 | $msg = 'COMMAND / CONFIG: rabbitmqctl.bat eval rabbit_prelaunch:get_context/0'
233 | Write-Verbose -Message $msg
234 | Add-Content @addContentArgs -Value "$sep$([Environment]::NewLine)$msg"
235 |
236 | $evalString = '"C=rabbit_prelaunch:get_context(),Keys=[advanced_config_file,conf_env_file,config_base_dir,data_base_dir,data_dir,enabled_plugins_file,home_dir,log_base_dir,main_config_file,main_log_file,quorum_queue_dir,rabbitmq_base,rabbitmq_home,stream_queue_dir],file:write_file(""' + $tmpFileName + '"",rabbit_json:encode([{K,unicode:characters_to_binary(maps:get(K,C))} || K <- Keys]))."'
237 | & $rabbitmqctl eval $evalString | Out-Null
238 |
239 | New-Variable -Name rmqConfigJson -Option Constant -Value (Get-Content -LiteralPath $tmpFile | ConvertFrom-Json)
240 | $rmqConfigJson | Out-File @outFileArgs
241 | Write-Verbose -Message $rmqConfigJson
242 | }
243 | finally
244 | {
245 | Remove-Item -ErrorAction 'SilentlyContinue' -Force $tmpFile
246 | }
247 |
248 | ########################################################################
249 | # Get RabbitMQ log files
250 | #
251 | $msg = 'COMMAND: getting RabbitMQ log files'
252 | Write-Verbose -Message $msg
253 | Add-Content @addContentArgs -Value "$sep$([Environment]::NewLine)$msg"
254 | New-Variable -Name rmqLogBaseDir -Option Constant -Value $rmqConfigJson.log_base_dir
255 | New-Variable -Name rmqMainLogFileFullName -Option Constant -Value $rmqConfigJson.main_log_file
256 | New-Variable -Name rmqMainLogFileShortName -Option Constant -Value (Split-Path -Leaf -Resolve $rmqMainLogFileFullName)
257 | New-Variable -Name rmqMainLogFileFullNameTempParent -Option Constant -Value (New-TemporaryDirectory)
258 | New-Variable -Name rmqMainLogFileFullNameTemp -Option Constant -Value (Join-Path -Path $rmqMainLogFileFullNameTempParent -ChildPath $rmqMainLogFileShortName)
259 | if (Test-Path -Path $rmqLogBaseDir)
260 | {
261 | try
262 | {
263 | # Note:
264 | # The combination of -Exclude and -LiteralPath does NOT work correctly using Powershell 5.1
265 | Get-ChildItem -Recurse -Path $rmqLogBaseDir -Exclude $rmqMainLogFileShortName | Compress-Archive -Force -DestinationPath $logsArchiveFile
266 | Copy-Item -Force -LiteralPath $rmqMainLogFileFullName -Destination $rmqMainLogFileFullNameTemp
267 | Compress-Archive -LiteralPath $rmqMainLogFileFullNameTemp -Update -DestinationPath $logsArchiveFile
268 | }
269 | finally
270 | {
271 | Remove-Item -Recurse -Force -ErrorAction 'SilentlyContinue' -LiteralPath $rmqMainLogFileFullNameTempParent
272 | }
273 | }
274 | else
275 | {
276 | $msg = "[ERROR] RabbitMQ log base dir does not exist: $rmqLogBaseDir"
277 | Write-Error -ErrorAction 'Continue' -Message $msg
278 | Add-Content @addContentArgs -Value $msg
279 | }
280 |
281 | ########################################################################
282 | # rabbitmq-env.bat (runs rabbitmq-env-conf.bat if it exists)
283 | #
284 | if (Test-Path -Path $rmqEnvBat)
285 | {
286 | New-Variable -Name rabbitmqEnvRunnerTmpl -Option Constant -Value @"
287 | @echo off
288 | setlocal
289 | setlocal enabledelayedexpansion
290 | setlocal enableextensions
291 | if ERRORLEVEL 1 (
292 | echo "Failed to enable command extensions!"
293 | exit /B 1
294 | )
295 | call "@@PATH@@"
296 | set
297 | "@
298 | $tmpFile = New-TemporaryFile
299 | $rabbitmqEnvRunner = "$tmpFile.bat"
300 | Move-Item -Force -LiteralPath $tmpFile -Destination $rabbitmqEnvRunner
301 | try
302 | {
303 | Set-Content -LiteralPath $rabbitmqEnvRunner -Value $($rabbitmqEnvRunnerTmpl -replace '@@PATH@@', $rmqEnvBat.Path)
304 | $msg = "COMMAND: rabbitmq-env.bat"
305 | Write-Verbose -Message $msg
306 | Add-Content @addContentArgs -Value "$sep$([Environment]::NewLine)$msg$([Environment]::NewLine)"
307 | & $rabbitmqEnvRunner | Out-File @outFileArgs
308 | }
309 | finally
310 | {
311 | Remove-Item -ErrorAction 'SilentlyContinue' -Force $tmpFile
312 | Remove-Item -ErrorAction 'SilentlyContinue' -Force $rabbitmqEnvRunner
313 | }
314 | }
315 | else
316 | {
317 | $msg = "[ERROR] could not find expected RabbitMQ environment batch file: $rmqEnvBatFullName"
318 | Write-Error -ErrorAction 'Continue' -Message $msg
319 | Add-Content @addContentArgs -Value $msg
320 | }
321 |
322 | $msg = "COMMAND: rabbitmqctl.bat report"
323 | Write-Verbose -Message $msg
324 | Add-Content @addContentArgs -Value "$sep$([Environment]::NewLine)$msg$([Environment]::NewLine)"
325 | & $rabbitmqctl report | ForEach-Object { $_ -replace '\x1b\[[0-9;]*m','' } | Out-File @outFileArgs
326 | }
327 |
328 | function Get-Win32_OperatingSystem
329 | {
330 | $msg = 'COMMAND: "Get-CimInstance Win32_OperatingSystem"'
331 | Write-Verbose -Message $msg
332 | Add-Content @addContentArgs -Value "$sep$([Environment]::NewLine)$msg"
333 | Get-CIMInstance Win32_OperatingSystem | Format-List -Property '*' | Out-File @outFileArgs
334 | }
335 |
336 | function Get-Win32_Process
337 | {
338 | $msg = 'COMMAND: "Get-CimInstance Win32_Process | Select-Object -Property Name,WorkingSetSize,Path,CommandLine"'
339 | Write-Verbose -Message $msg
340 | Add-Content @addContentArgs -Value "$sep$([Environment]::NewLine)$msg"
341 | Get-CimInstance Win32_Process | Select-Object -Property Name,WorkingSetSize,CommandLine | Sort-Object -Property Name | Format-List | Out-File @outFileArgs
342 | }
343 |
344 | function Get-Win32_Services
345 | {
346 | $msg = 'COMMAND: "Get-Service -ErrorAction ''SilentlyContinue'' | Where-Object { $_.Status -eq ''Running'' } | Select-Object -Property Name,Description,BinaryPathName"'
347 | Write-Verbose -Message $msg
348 | Add-Content @addContentArgs -Value "$sep$([Environment]::NewLine)$msg"
349 | Get-Service -ErrorAction 'SilentlyContinue' | Where-Object { $_.Status -eq 'Running' } | Select-Object -Property Name,Description,BinaryPathName | Format-List | Out-File @outFileArgs
350 | }
351 |
352 | function Run-Main
353 | {
354 | Get-ErlangInfo
355 | Get-RabbitMQInfo
356 | Get-Win32_OperatingSystem
357 | Get-Win32_Process
358 | Get-Win32_Services
359 | }
360 |
361 | function Run-End
362 | {
363 | New-Variable -Name t -Option Constant -Value (Get-Times)
364 | $msg = "STOP TIME UTC: $($t.NowUTC) LOCAL: $($t.NowLocal)"
365 | Write-Verbose -Message $msg
366 | Add-Content @addContentArgs -Value "$sep$([Environment]::NewLine)$msg"
367 | }
368 |
369 | try
370 | {
371 | Run-Init
372 | Run-Main
373 | Run-End
374 | }
375 | finally
376 | {
377 | Set-PSDebug -Off
378 | }
379 |
--------------------------------------------------------------------------------
/scripts/rabbitmq-k8s-runbook/README.md:
--------------------------------------------------------------------------------
1 | # RabbitMQ for Kubernetes diagnostic script
2 |
3 | This tool is intended to be a simple script for collecting diagnostic informations: status, object yaml definitions, logs for the RabbitMQ operators and RabbitMQ clusters deployed in Kubernetes.
4 |
5 | It is based on this guideline and set of commands: https://github.com/rabbitmq/support-tools/blob/main/docs/Reporting_RabbitMQ_For_Kubernetes_Issues.md
6 |
7 | It can be used with both Tanzu RabbitMQ commercial and open source versions.
8 |
9 | It can run with 4 options
10 | * get_k8s_info
11 | * get_carvel_components_info
12 | * get_operators_info
13 | * get_rabbitmq_cluster_info
14 |
15 | ### Get K8s cluster info:
16 |
17 | ```
18 | ./rabbitmq-k8s-diagnostic-runbook get_k8s_cluster_info ./diagnostics
19 | ```
20 |
21 | This command returns useful informations on the Kubernetes cluster (version, status of the nodes, status of the storageclasses ecc...)
22 |
23 | ### Get specific commercial Carvel components diagnostic info:
24 |
25 | ```
26 | ./rabbitmq-k8s-diagnostic-runbook get_carvel_components_info -n rabbitmq-system ./diagnostics
27 | ```
28 |
29 | This command which should be used just if you are running the commercial version of RabbitMQ based on this doc: https://docs.vmware.com/en/VMware-Tanzu-RabbitMQ-for-Kubernetes/1.3/tanzu-rmq/GUID-installation.html
30 | returns informations and logs on the components of the Carvel Suite (kapp, secret-gen controller as well as the PackageRepository and PackageInstall objects used).
31 |
32 | It takes in input the namespace where you deployed the PackageRepository and PackageInstall objects. If none is provided then it will search in the default namespace.
33 |
34 | Note: kapp, and secretgen controller are supposed to be deployed in the default installation namespaces for these components: kapp and secret-gen namespaces
35 |
36 | ### Get Operators diagnostic info:
37 |
38 | ```
39 | ./rabbitmq-k8s-diagnostic-runbook get_operators_info -n rabbitmq-system ./diagnostics
40 | ```
41 |
42 | This command returns informations and logs on the three RabbitMQ operators: cluster-operator, messaging-topology-operator and standby replication operator (this last one just available in the commercial version).
43 | It takes in input the namespace where the operators are installed, if none is provided will search in the rabbitmq-system.
44 |
45 | ### Get RabbitMQ cluster diagnostic info:
46 |
47 | ```
48 | ./rabbitmq-k8s-diagnostic-runbook get_rabbitmq_cluster_info -n rabbitmq-cluster ./diagnostics
49 | ```
50 |
51 | Returns informations and logs on the RabbitMQ cluster deployed on the namespace passed (If none specified will search on the default namespace).
52 | You can run it on different namespaces where a RabbitMQ cluster is deployed
53 |
54 | ### Generated report informations:
55 |
56 | the output of the scripts is a set of files containing informations and logs.
57 | For every of the four commands run a subfolder will be created: k8s-cluster, carvel, operators, rabbitmq_cluster
58 | So you will have:
59 |
60 | * k8s-cluster/info: Some informations about the Kubernetes cluster (version, status of the nodes, storage classes)
61 | * carvel/info: informations about kapp and secret-gen controller objects as well as .yaml definitions of the PackageRepository and PackageInstall objects
62 | * carvel/logs: logs file for kapp and secretgen-controller pods
63 | * operators/info: Status of the RabbitMQ operator objects as well asyaml definition files of the statefulset
64 | * operators/logs: Logs of operators pods as well as of the secretgen pods.
65 | * rabbitmq_cluster/info: Status of the RabbitMQ cluster objects as well as some .yaml definition files of the statefulset
66 | * rabbitmq_cluster/info: Logs of the RabbitMQ cluster pods
67 |
68 | Note: the folder you pass in input will be overwritten. So run the script with an empty or non-existing folder.
69 | You can then zip and send the info to R&D for investigation.
70 | For commercial version you need also to run the option get_carvel_components_info, for OSS is not necessary
71 |
--------------------------------------------------------------------------------
/scripts/rabbitmq-k8s-runbook/rabbitmq-k8s-diagnostic-runbook:
--------------------------------------------------------------------------------
1 | #!/bin/bash
2 | #
3 | # RabbitMQ Diagnostic tool for cluster operator
4 | #
5 | # Copyright 2022 VMware, Inc. All Rights Reserved.
6 | #
7 | # This product is licensed to you under the Mozilla Public license, Version 2.0 (the "License"). You may not use this product except in compliance with the Mozilla Public License.
8 | #
9 | # This product may include a number of subcomponents with separate copyright notices and license terms. Your use of these subcomponents is subject to the terms and conditions of the subcomponent's license, as noted in the LICENSE file.
10 |
11 | set -euo pipefail
12 |
13 | RABBITMQ_OPERATORS_NAMESPACE=""
14 | RABBITMQ_CLUSTER_NAMESPACE=""
15 | KAPP_NAMESPACE="kapp-controller"
16 | SECRETGEN_NAMESPACE="secretgen-controller"
17 | # namesapce where PackageRepo and PackageInstall type are deployed (if not specified default namespace - commercial version only)
18 | CARVEL_PACKAGES_NAMESPACE=""
19 | OUTPUT_DIRECTORY=""
20 | YEL=$'\e[1;33m'
21 | RED=$'\e[1;31m'
22 | END=$'\e[0m'
23 |
24 | usage() {
25 | usage=$(
26 | cat <<-END
27 | USAGE:
28 | Get info about the Kubernetes cluster (last parameter is the output directory)
29 | rabbitmq-k8s-diagnostic-runbook get_k8s_cluster_info output_directory
30 |
31 | Get info about specific carvel components (commercial version only): tanzu-eseential, kapp, secretgen-controller, PackageRepo and PackageInstall (last parameter is the output directory).
32 | To specify the namespace where the PackageRepo and PackageInstall are deployed (if none we will search on default namespace)
33 | rabbitmq-k8s-diagnostic-runbook get_carvel_components_info -n namespace
34 |
35 | Get info about operators: cluster-operator, messaging-topology-operator (if deployed) and standby-replication-operator (if deployed in commercial version only) last parameter is the output directory
36 | To specify the namespace where the operators are deployed (if none we will search on rabbitmq-system namespace)
37 | rabbitmq-k8s-diagnostic-runbook get_operators_info -n namespace
38 |
39 | Get info about the rabbitmq cluster deployed (last parameter is the output directory)
40 | To specify the namespace where the cluster is deployed (if none we will search on default namespace)
41 | rabbitmq-k8s-diagnostic-runbook get_rabbitmq_cluster_info -n namespace
42 |
43 | END
44 | )
45 | echo "$usage"
46 | }
47 |
48 |
49 | data_diagnostic_cluster_info() {
50 |
51 | OUTPUT_DIRECTORY=$1
52 |
53 | printf "\\n[INFO] Getting some Kubernetes cluster info and storing them in $OUTPUT_DIRECTORY/k8s-cluster/info.\\n"
54 | mkdir -p $OUTPUT_DIRECTORY/k8s-cluster/info
55 |
56 | # Get K8s cluster info
57 | kubectl version > $OUTPUT_DIRECTORY/k8s-cluster/info/k8s_version.info
58 | kubectl get nodes > $OUTPUT_DIRECTORY/k8s-cluster/info/k8s_nodes.info
59 | kubectl describe nodes >> $OUTPUT_DIRECTORY/k8s-cluster/info/k8s_nodes.info
60 | kubectl get storageclass > $OUTPUT_DIRECTORY/k8s-cluster/info/k8s_storage.info
61 | kubectl describe storageclass >> $OUTPUT_DIRECTORY/k8s-cluster/info/k8s_storage.info
62 | kubectl top nodes > $OUTPUT_DIRECTORY/k8s-cluster/info/top.info
63 | kubectl top pod --all-namespaces >> $OUTPUT_DIRECTORY/k8s-cluster/info/top.info
64 |
65 | printf "[INFO] Completed storing K8s info in $OUTPUT_DIRECTORY/k8s-cluster/info\\n"
66 | }
67 |
68 | data_diagnostic_carvel_components_info() {
69 |
70 | CARVEL_PACKAGES_NAMESPACE=$1
71 | OUTPUT_DIRECTORY=$2
72 |
73 | printf "\\n[INFO] Getting info about Tanzu commercial carvel components: tanzu-essential, kapp, secret-gen components and the deployed PackageRepo and PackageInstall\\n"
74 | mkdir -p $OUTPUT_DIRECTORY/carvel/info
75 | mkdir -p $OUTPUT_DIRECTORY/carvel/logs
76 |
77 | data_diagnostic_carvel_versions
78 | data_diagnostic_carvel_status
79 | data_diagnostic_carvel_logs
80 |
81 | printf "\\n[INFO]Ending get_carvel_components_info\\n"
82 |
83 | }
84 |
85 | data_diagnostic_carvel_versions() {
86 |
87 | set +e
88 |
89 | printf "\\n[INFO] Looking for kapp-controller and secret-gen controller versions and saving it in a file.\\n"
90 | kapp_version=$(kubectl describe deployment kapp-controller -n kapp-controller | grep "kapp-controller.carvel.dev/version:")
91 | if [[ -z "${kapp_version}" ]]; then
92 | printf "[ERROR] ${RED}kapp is not installed in the kapp-controller namespace (this is if you are using Tanzu RabbitMQ commercial version only) ${END}\\n"
93 | exit -1
94 | fi
95 |
96 | secret_gen_controller_version=$(kubectl describe deploy secretgen-controller -n secretgen-controller | grep "secretgen-controller.carvel.dev/version")
97 | if [[ -z "${secret_gen_controller_version}" ]]; then
98 | printf "[ERROR] ${RED}secret-gen controller is not installed in the secretgen-controller namespace (this is if you are using Tanzu RabbitMQ commercial version only) ${END}\\n"
99 | exit -1
100 | fi
101 |
102 | echo "kapp version: $kapp_version" > $OUTPUT_DIRECTORY/carvel/info/carvel_versions.info
103 | echo "secretgen-controller version: $secret_gen_controller_version" >> $OUTPUT_DIRECTORY/carvel/info/carvel_versions.info
104 |
105 | printf "\\n[INFO] kapp-controller and secret-gen controller versions saved in $OUTPUT_DIRECTORY/carvel/info/carvel_versions.log\\n"
106 | }
107 |
108 | data_diagnostic_carvel_status() {
109 |
110 | set -e
111 |
112 | # kapp status namespace
113 | printf "\\n[INFO] Looking for kapp objects status and definitions in kapp-controller namespace and saving it to a file.\\n"
114 | kubectl get all -n $KAPP_NAMESPACE > $OUTPUT_DIRECTORY/carvel/info/kapp_get_all_objects_status.info
115 | kubectl describe replicaset -n $KAPP_NAMESPACE >> $OUTPUT_DIRECTORY/carvel/info/kapp_get_all_objects_status.info
116 | kubectl describe deploy -n $KAPP_NAMESPACE >> $OUTPUT_DIRECTORY/carvel/info/kapp_get_all_objects_status.info
117 | kubectl get deploy -n $KAPP_NAMESPACE -o yaml > $OUTPUT_DIRECTORY/carvel/info/kapp-controller-definition.yaml
118 |
119 |
120 | # secretgen-controller status namespace
121 | printf "[INFO] Looking for secretgen-controller objects status and definitions in secretgen-controller namespace and saving it to files in $OUTPUT_DIRECTORY/carvel/info.\\n"
122 | kubectl get all -n $SECRETGEN_NAMESPACE > $OUTPUT_DIRECTORY/carvel/info/sec_gen_get_all_objects_status.info
123 | kubectl describe replicaset -n $SECRETGEN_NAMESPACE >> $OUTPUT_DIRECTORY/carvel/info/sec_gen_get_all_objects_status.info
124 | kubectl describe deploy -n $SECRETGEN_NAMESPACE >> $OUTPUT_DIRECTORY/carvel/info/sec_gen_get_all_objects_status.info
125 | kubectl get deploy -n $SECRETGEN_NAMESPACE -o yaml > $OUTPUT_DIRECTORY/carvel/info/secretgen-controller-definition.yaml
126 |
127 | # PackageRepository and PackageInstall namespaces
128 | printf "[INFO] Looking for PackageRepo and PackageInstall objects status in secretgen-controller namespace and saving it to files in $OUTPUT_DIRECTORY/carvel/info.\\n"
129 | packagerepo=$(kubectl get PackageRepository -n $CARVEL_PACKAGES_NAMESPACE 2>/dev/null )
130 | if [[ -z "${packagerepo}" ]]; then
131 | printf "\\n${RED}[ERROR] PackageRepository not found in $CARVEL_PACKAGES_NAMESPACE namespace exiting from the script ${END}\\n"
132 | exit -1
133 | fi
134 | echo $packagerepo > $OUTPUT_DIRECTORY/carvel/info/packagerepo.info
135 | kubectl describe PackageRepository -n $CARVEL_PACKAGES_NAMESPACE >> $OUTPUT_DIRECTORY/carvel/info/packagerepo.info
136 | kubectl get PackageRepository -n $CARVEL_PACKAGES_NAMESPACE -o yaml > $OUTPUT_DIRECTORY/carvel/info/packagerepo_definition.yaml
137 |
138 | packageinstall=$(kubectl get PackageInstall -n $CARVEL_PACKAGES_NAMESPACE 2>/dev/null)
139 | if [[ -z "${packageinstall}" ]]; then
140 | printf "\\n${RED}[ERROR] PackageInstall not found in $CARVEL_PACKAGES_NAMESPACE namespace exiting from the script ${END}\\n"
141 | exit -1
142 | fi
143 | echo $packageinstall > $OUTPUT_DIRECTORY/carvel/info/packageinstall.info
144 |
145 | kubectl describe PackageInstall -n $CARVEL_PACKAGES_NAMESPACE >> $OUTPUT_DIRECTORY/carvel/info/packageinstall.info
146 | kubectl get PackageInstall -n $CARVEL_PACKAGES_NAMESPACE -o yaml > $OUTPUT_DIRECTORY/carvel/info/packageinstall_definition.yaml
147 |
148 | printf "[INFO] data_diagnostic_carvel_status completed info saved to files\\n"
149 |
150 | }
151 |
152 | data_diagnostic_carvel_logs() {
153 |
154 | printf "\\n[INFO] Saving pod logs of kapp-controller and secretgen-controller to files in $OUTPUT_DIRECTORY/carvel/logs\\n"
155 |
156 | kubectl logs -l app=kapp-controller -n $KAPP_NAMESPACE -c kapp-controller --tail -1 > $OUTPUT_DIRECTORY/carvel/logs/kapp-controller.log
157 | kubectl logs -l app=secretgen-controller -n $SECRETGEN_NAMESPACE --tail -1 > $OUTPUT_DIRECTORY/carvel/logs/secretgen-controller.log
158 |
159 | printf "[INFO] data_diagnostic_carvel_logs completed logs of kapp-controller and secretgen-controller saved to files\\n"
160 |
161 | }
162 |
163 | data_diagnostic_rabbitmq_cluster_info() {
164 |
165 | printf "\\n[INFO] Getting info on the RabbitMQ cluster deployed"
166 |
167 | RABBITMQ_CLUSTER_NAMESPACE=$1
168 | OUTPUT_DIRECTORY=$2
169 |
170 | rabbitmq_cluster_output_folder=$OUTPUT_DIRECTORY/rabbitmqcluster_$RABBITMQ_CLUSTER_NAMESPACE
171 | rabbitmq_cluster_output_info_folder=$rabbitmq_cluster_output_folder/info
172 | rabbitmq_cluster_output_logs_folder=$rabbitmq_cluster_output_folder/logs
173 |
174 | mkdir -p $rabbitmq_cluster_output_info_folder
175 | mkdir -p $rabbitmq_cluster_output_logs_folder
176 |
177 | data_diagnostic_rabbitmq_cluster_status
178 | data_diagnostic_rabbitmq_cluster_definitions
179 | data_diagnostic_rabbitmq_cluster_logs
180 |
181 | printf "[INFO] Ending get_rabbitmq_cluster_info\\n"
182 |
183 | }
184 |
185 | data_diagnostic_rabbitmq_cluster_status() {
186 |
187 | set +e
188 |
189 | printf "\\n[INFO] Checking if a rabbitmq cluster is deployed on namespace $RABBITMQ_CLUSTER_NAMESPACE\\n"
190 |
191 | rabbitmq_cluster=$(kubectl get rabbitmqcluster.rabbitmq.com -n $RABBITMQ_CLUSTER_NAMESPACE 2>/dev/null)
192 | if [[ -z "${rabbitmq_cluster}" ]]; then
193 | printf "\\n[ERROR] ${RED} rabbitmqcluster.rabbitmq.com is not deployed in the specified $RABBITMQ_CLUSTER_NAMESPACE namespace ${END}\\n"
194 | exit -1
195 | fi
196 |
197 | printf "[INFO] Get all objects and description: rabbitmqcluster, statefulset, pods, pvc deployed in the $RABBITMQ_CLUSTER_NAMESPACE namespace and saving in $OUTPUT_DIRECTORY/rabbitmqcluster/info\\n"
198 | kubectl get all -l app.kubernetes.io/component=rabbitmq -n $RABBITMQ_CLUSTER_NAMESPACE > $rabbitmq_cluster_output_info_folder/get_all_objects.info
199 | kubectl describe rabbitmqcluster.rabbitmq.com -n $RABBITMQ_CLUSTER_NAMESPACE > $rabbitmq_cluster_output_info_folder/rabbitmqcluster.info
200 |
201 | rabbitmq_stateful_set=$(kubectl get statefulset -l app.kubernetes.io/component=rabbitmq -n $RABBITMQ_CLUSTER_NAMESPACE 2>/dev/null)
202 | if [[ -z "${rabbitmq_cluster}" ]]; then
203 | printf "\\n[ERROR] ${RED} rabbitmqcluster.rabbitmq.com statefulset not present in the specified $RABBITMQ_CLUSTER_NAMESPACE namespace ${END}"
204 | exit -1
205 | fi
206 |
207 | kubectl describe statefulset -l app.kubernetes.io/component=rabbitmq -n $RABBITMQ_CLUSTER_NAMESPACE >> $rabbitmq_cluster_output_info_folder/statefulset.info
208 |
209 | rabbitmq_pods=$(kubectl get pod -l app.kubernetes.io/component=rabbitmq -n $RABBITMQ_CLUSTER_NAMESPACE 2>/dev/null)
210 | if [[ -z "${rabbitmq_pods}" ]]; then
211 | printf "\\n[ERROR] ${RED} rabbitmqcluster.rabbitmq.com pods not present in the specified $RABBITMQ_CLUSTER_NAMESPACE namespace ${END}\\n"
212 | exit -1
213 | fi
214 |
215 | kubectl describe pod -l app.kubernetes.io/component=rabbitmq -n $RABBITMQ_CLUSTER_NAMESPACE > $rabbitmq_cluster_output_info_folder/pods.info
216 |
217 | kubectl get pv -n $RABBITMQ_CLUSTER_NAMESPACE > $rabbitmq_cluster_output_info_folder/storage.info
218 | kubectl describe pv -n $RABBITMQ_CLUSTER_NAMESPACE >> $rabbitmq_cluster_output_info_folder/storage.info
219 | kubectl get pvc -n $RABBITMQ_CLUSTER_NAMESPACE >> $rabbitmq_cluster_output_info_folder/storage.info
220 | kubectl describe pvc -n $RABBITMQ_CLUSTER_NAMESPACE >> $rabbitmq_cluster_output_info_folder/storage.info
221 |
222 | printf "[INFO] Checking RabbitMQ version and storing it in $OUTPUT_DIRECTORY/rabbitmqcluster/info"
223 |
224 | pod_name=$(head -1 $rabbitmq_cluster_output_info_folder/pods.info | sed s/"Name: "//g)
225 | rabbitmq_version=$(kubectl exec -it $pod_name -n $RABBITMQ_CLUSTER_NAMESPACE -- rabbitmqctl version)
226 | echo "RabbitMQ version: $rabbitmq_version" > $rabbitmq_cluster_output_info_folder/rabbitmq_version.info
227 |
228 | printf "\\n[INFO] Get/Description of all objects completed and stored in a file in $OUTPUT_DIRECTORY/rabbitmqcluster/info\\n"
229 |
230 | }
231 |
232 | data_diagnostic_rabbitmq_cluster_definitions() {
233 |
234 | set -e
235 |
236 | printf "[INFO] Get yaml defitions of rabbitmqcluster.rabbitmq.com, statefulset and pod and saving them in $OUTPUT_DIRECTORY/rabbitmqcluster/info\\n"
237 |
238 | kubectl get rabbitmqcluster.rabbitmq.com -n $RABBITMQ_CLUSTER_NAMESPACE -o yaml > $rabbitmq_cluster_output_info_folder/rabbitmqcluster-definition.yaml
239 | kubectl get statefulset -l app.kubernetes.io/component=rabbitmq -n $RABBITMQ_CLUSTER_NAMESPACE -o yaml > $rabbitmq_cluster_output_info_folder/statefulset-definition.yaml
240 | kubectl get pod -l app.kubernetes.io/component=rabbitmq -n $RABBITMQ_CLUSTER_NAMESPACE -o yaml >> $rabbitmq_cluster_output_info_folder/pods-definition.yaml
241 |
242 | printf "[INFO] Yaml definition export completed \\n"
243 | }
244 |
245 | data_diagnostic_rabbitmq_cluster_logs() {
246 |
247 |
248 | printf "[INFO] Get pod logs of rabbitmqcluster in $RABBITMQ_CLUSTER_NAMESPACE and saving in a file $OUTPUT_DIRECTORY/rabbitmqcluster/logs/pods.log\\n"
249 |
250 | kubectl logs -l app.kubernetes.io/component=rabbitmq -n $RABBITMQ_CLUSTER_NAMESPACE --tail -1 >> $rabbitmq_cluster_output_logs_folder/pods.log
251 |
252 | printf "[INFO] Get pod logs completed and stored in $OUTPUT_DIRECTORY/rabbitmqcluster/logs/pods.log\\n"
253 |
254 | }
255 |
256 | data_diagnostic_operators_info() {
257 |
258 |
259 | printf "\\n[INFO] Getting info about the RabbitMQ operators deployed\\n"
260 |
261 | RABBITMQ_OPERATORS_NAMESPACE=$1
262 | OUTPUT_DIRECTORY=$2
263 |
264 | mkdir -p $OUTPUT_DIRECTORY/operators/info
265 | mkdir -p $OUTPUT_DIRECTORY/operators/logs
266 |
267 | data_diagnostic_operators_versions
268 | data_diagnostic_operators_status
269 | data_diagnostic_operators_definition
270 | data_diagnostic_operators_logs
271 |
272 | printf "\\n[INFO] info about the cluster operator function ended\\n"
273 | }
274 |
275 |
276 | data_diagnostic_operators_versions() {
277 |
278 | set +e
279 |
280 | printf "[INFO] Checking if operators are deployed in $RABBITMQ_OPERATORS_NAMESPACE namespace if yes saving versions in $OUTPUT_DIRECTORY/operators/info/operators_versions.log\\n"
281 | # Get versions:
282 | cluster_operator_image=$(kubectl get deployment rabbitmq-cluster-operator -n ${RABBITMQ_OPERATORS_NAMESPACE} -o yaml 2>/dev/null | grep " image:")
283 | if [[ -z "$cluster_operator_image" ]]; then
284 | printf "\\n[ERROR] ${RED} cluster operator is not deployed in ${RABBITMQ_OPERATORS_NAMESPACE} namespace ${END}\\n"
285 | exit -1
286 | fi
287 |
288 | cluster_operator_version=${cluster_operator_image#*/}
289 | echo "cluster operator version: $cluster_operator_version" > $OUTPUT_DIRECTORY/operators/info/operators_versions.info
290 | messaging_operator_image=$(kubectl get deployment messaging-topology-operator -n ${RABBITMQ_OPERATORS_NAMESPACE} -o yaml 2>/dev/null | grep " image:")
291 | if [[ -z "${messaging_operator_image}" ]]; then
292 | printf "\\n${YEL}[WARN] messaging operator is not deployed in rabbitmq-system namespace${END}\\n"
293 | else
294 | messaging_operator_version=${messaging_operator_image#*/}
295 | echo "messaging operator version: $messaging_operator_version" >> $OUTPUT_DIRECTORY/operators/info/operators_versions.info
296 | fi
297 |
298 | standby_operator_image=$(kubectl get deployment standby-replication-operator -n ${RABBITMQ_OPERATORS_NAMESPACE} -o yaml 2>/dev/null | grep " image:")
299 | if [[ -z "${standby_operator_image}" ]]; then
300 | printf "\\n${YEL}[WARN] standby-replication operator is not deployed in rabbitmq-system namespace. This operator is available just in the Tanzu RabbitMQ commercial version${END}\\n"
301 | else
302 | standby_operator_version=${standby_operator_image#*/}
303 | echo "standby operator version: $standby_operator_version" >> $OUTPUT_DIRECTORY/operators/info/operators_versions.info
304 | fi
305 |
306 | printf "[INFO] Checking if cert-manager is deployed in cert-manager namespace and if yes savin version in $OUTPUT_DIRECTORY/operators/info/operators_versions.log\\n"
307 | cert_manager_image=$(kubectl get deploy -l app.kubernetes.io/name=cert-manager -n cert-manager -o yaml 2>/dev/null | grep " image:")
308 | cert_manager_version=${cert_manager_image#*/}
309 | if [[ -n "${cert_manager_version}" ]]; then
310 | echo "cert manager version: $cert_manager_version" >> $OUTPUT_DIRECTORY/operators/info/operators_versions.info
311 | else
312 | printf "\\n${YEL}[WARN] cert manager is not deployed in cert-manager namespace${END}\\n"
313 | fi
314 |
315 |
316 | printf "[INFO] ending saving operator versions\\n\\n"
317 |
318 | }
319 |
320 | data_diagnostic_operators_status() {
321 |
322 | set -e
323 |
324 | printf "\\n[INFO] Getting info about operator objects: deployment, replicasets, pods and saving them in $OUTPUT_DIRECTORY/operators/info/operator_report.info"
325 |
326 | # Get status (deployment, rs, pods):
327 | kubectl get all -n ${RABBITMQ_OPERATORS_NAMESPACE} > $OUTPUT_DIRECTORY/operators/info/operator_report.info
328 | printf "\\nkubectl describe deployment output:\\n" >> $OUTPUT_DIRECTORY/operators/info/operator_report.info
329 | kubectl describe deploy -l app.kubernetes.io/component=rabbitmq-operator -n ${RABBITMQ_OPERATORS_NAMESPACE} >> $OUTPUT_DIRECTORY/operators/info/operator_report.info
330 | printf "\\kubectl describe replicaset output:\\n" >> $OUTPUT_DIRECTORY/operators/info/operator_report.info
331 | kubectl describe rs -l app.kubernetes.io/component=rabbitmq-operator -n ${RABBITMQ_OPERATORS_NAMESPACE} >> $OUTPUT_DIRECTORY/operators/info/operator_report.info
332 | printf "\\nkubectl describe pod output:\\n" >> $OUTPUT_DIRECTORY/operators/info/operator_report.info
333 | kubectl describe pod -l app.kubernetes.io/component=rabbitmq-operator -n ${RABBITMQ_OPERATORS_NAMESPACE} >> $OUTPUT_DIRECTORY/operators/info/operator_report.info
334 |
335 | printf "\\n[INFO] Getting info about cert-manager objects if installed: deployment, pods saving them in $OUTPUT_DIRECTORY/operators/info/certmanager_report.info"
336 | if [[ -n "${cert_manager_version}" ]]; then
337 | kubectl get all -l app.kubernetes.io/instance=cert-manager -n cert-manager > $OUTPUT_DIRECTORY/operators/info/certmanager_report.info
338 | kubectl describe deployment cert-manager -n cert-manager >> $OUTPUT_DIRECTORY/operators/info/certmanager_report.info
339 | kubectl describe deployment cert-manager-cainjector -n cert-manager >> $OUTPUT_DIRECTORY/operators/info/certmanager_report.info
340 | kubectl describe deployment cert-manager-webhook -n cert-manager >> $OUTPUT_DIRECTORY/operators/info/certmanager_report.info
341 | fi
342 |
343 | printf "\\n[INFO] ending storing info about operator objects: deployment, replicasets, pods\\n"
344 | }
345 |
346 | data_diagnostic_operators_definition() {
347 |
348 | printf "\\n[INFO] Getting the yaml definition of the operators deployment and saving them in $OUTPUT_DIRECTORY/operators/info"
349 | kubectl get deployment rabbitmq-cluster-operator -n ${RABBITMQ_OPERATORS_NAMESPACE} -o yaml > $OUTPUT_DIRECTORY/operators/info/rabbitmq-cluster-operator-deploy.yml
350 |
351 | if [[ -n "${messaging_operator_image}" ]]; then
352 | kubectl get deployment messaging-topology-operator -n ${RABBITMQ_OPERATORS_NAMESPACE} -o yaml > $OUTPUT_DIRECTORY/operators/info/rabbitmq-messaging-topology-operator-deploy.yml
353 | fi
354 |
355 | if [[ -n "${standby_operator_image}" ]]; then
356 | kubectl get deployment standby-replication-operator -n ${RABBITMQ_OPERATORS_NAMESPACE} -o yaml > $OUTPUT_DIRECTORY/operators/info/rabbitmq-standby-replication-operator-deploy.yml
357 | fi
358 |
359 | printf "\\n[INFO] ending getting yaml definition of the operator deployments\\n"
360 |
361 | }
362 |
363 | data_diagnostic_operators_logs() {
364 |
365 | printf "\\n[INFO] Getting operator pod logs saving them in $OUTPUT_DIRECTORY/operators/logs"
366 |
367 | # Get logs for cluster operator
368 | kubectl logs -l app.kubernetes.io/name=rabbitmq-cluster-operator --tail -1 -n ${RABBITMQ_OPERATORS_NAMESPACE} > $OUTPUT_DIRECTORY/operators/logs/cluster-operator.log
369 | # Get logs for messaging topology operator
370 | if [[ -n "${messaging_operator_image}" ]]; then
371 | kubectl logs -l app.kubernetes.io/name=messaging-topology-operator --tail -1 -n ${RABBITMQ_OPERATORS_NAMESPACE} > $OUTPUT_DIRECTORY/operators/logs/messaging-operator.log
372 |
373 | fi
374 |
375 | if [[ -n "${standby_operator_image}" ]]; then
376 | kubectl logs -l app.kubernetes.io/name=standby-replication-operator --tail -1 -n ${RABBITMQ_OPERATORS_NAMESPACE} > $OUTPUT_DIRECTORY/operators/logs/standby-replication-operator.log
377 |
378 | fi
379 |
380 | if [[ -n "${cert_manager_version}" ]]; then
381 | kubectl logs -l app.kubernetes.io/instance=cert-manager --tail -1 -n cert-manager > $OUTPUT_DIRECTORY/operators/logs/cert-manager.log
382 | fi
383 |
384 | printf "\\n[INFO] Ending getting and storing operator pod logs\\n"
385 |
386 | }
387 |
388 | main() {
389 | if [[ "$1" == "--help" ]]; then
390 | usage
391 | exit 0
392 | fi
393 |
394 | case "$1" in
395 | # Get useful info and logs about the K8s cluster
396 | "get_k8s_cluster_info")
397 | shift 1
398 | if [[ "$#" != "1" ]]; then
399 | usage
400 | exit 1
401 | fi
402 |
403 | data_diagnostic_cluster_info $1
404 | ;;
405 | # Get info and logs on specific carvel commercial components: tanzu-essential, kapp, secret-gen controller, PackageRepository and PackageInstall
406 | "get_carvel_components_info")
407 | shift 1
408 | if [[ "$#" -gt 3 || "$#" == 2 ]]; then
409 | usage
410 | exit 1
411 | fi
412 | if [[ "$#" -lt 3 ]]; then
413 | namespace="default"
414 | output_dir=$1
415 | else
416 | namespace=$2
417 | output_dir=$3
418 | fi
419 |
420 | data_diagnostic_carvel_components_info $namespace $output_dir
421 | ;;
422 | # Get info and logs on the deployed operators in the given namespace
423 | "get_operators_info")
424 | shift 1
425 | if [[ "$#" -gt 3 || "$#" == 2 ]]; then
426 | usage
427 | exit 1
428 | fi
429 | if [[ "$#" -lt 3 ]]; then
430 | namespace="rabbitmq-system"
431 | output_dir=$1
432 | else
433 | namespace=$2
434 | output_dir=$3
435 | fi
436 | data_diagnostic_operators_info $namespace $output_dir
437 | ;;
438 | # Get info and logs of the deployed RabbitMQ cluster deployed in the specified namespace
439 | "get_rabbitmq_cluster_info")
440 | shift 1
441 | if [[ "$#" -gt 3 || "$#" == 2 ]]; then
442 | usage
443 | exit 1
444 | fi
445 | if [[ "$#" -lt 3 ]]; then
446 | namespace="default"
447 | output_dir=$1
448 | else
449 | namespace=$2
450 | output_dir=$3
451 | fi
452 | data_diagnostic_rabbitmq_cluster_info $namespace $output_dir
453 | ;;
454 | "help")
455 | usage
456 | ;;
457 | *)
458 | usage
459 | exit 1
460 | ;;
461 | esac
462 | }
463 |
464 | if [[ "$#" -ge 2 ]]; then
465 | main "$@"
466 | else
467 | usage
468 | fi
469 |
--------------------------------------------------------------------------------