├── LICENSE ├── README.md ├── config.csv └── sysinfo-snapshot.py /LICENSE: -------------------------------------------------------------------------------- 1 | BSD 3-Clause License 2 | 3 | Copyright (c) 2021, Mellanox Technologies Ltd. 4 | All rights reserved. 5 | 6 | Redistribution and use in source and binary forms, with or without 7 | modification, are permitted provided that the following conditions are met: 8 | 9 | 1. Redistributions of source code must retain the above copyright notice, this 10 | list of conditions and the following disclaimer. 11 | 12 | 2. Redistributions in binary form must reproduce the above copyright notice, 13 | this list of conditions and the following disclaimer in the documentation 14 | and/or other materials provided with the distribution. 15 | 16 | 3. Neither the name of the copyright holder nor the names of its 17 | contributors may be used to endorse or promote products derived from 18 | this software without specific prior written permission. 19 | 20 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" 21 | AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 22 | IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE 23 | DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE 24 | FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL 25 | DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR 26 | SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER 27 | CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, 28 | OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 29 | OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 30 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # linux-sysinfo-snapshot 2 | Linux Sysinfo Snapshot 3 | 4 | 1. Objective 5 | 6 | Automated sysinfo-snapshot tool is designed to take a snapshot of all the configuration and relevant information on the server and Mellanox's adapters. 7 | 8 | 2. Description 9 | The Sysinfo Snapshot is a python script that gathers system information and places it into a tar file. 10 | 11 | 3. Script Expected Output: 12 | By default, the followings are the expected file output when running the script: 13 | * "Performance tuning analyze" html file: this file dumps the performance status. 14 | * "Sysinfo snapshot" html file: this file dumps the server info and status. 15 | * "SR-IOV" html file: this file dumps all sr-iov related commands/internal files. 16 | * "commands_txt_output" folder which contains output of each command in a separate file. 17 | * "Status-log-sysinfo-snapshot" A log which contains each command invoked on the host, if it passed / failed and time taken. 18 | * commands_txt_output directory - a collection of commands output saved into files 19 | * "err_messages" folder which contains error message logs. 20 | 21 | 4 Specific Requirements:\ 22 | The followings are the server/system requirements to run the script: 23 | * Python installed- minimum required Python version 2.6 24 | * Linux based OS 25 | * Root (super user) privileges 26 | NOTE: The tool doesn't change any module status during runtime, except for the 'mst module'. If the 'mst module' is stopped, the tool loads it via 'mst start' and revert \ 27 | it to it's initial status via 'mst stop'. 28 | 29 | 5.1 Running the tool without a configuration file: 30 | Running Sysinfo Snapshot without a configuration file will gather the relevant information and \ 31 | configuration on the server and Nvidia's adapters, By adding additional flags, the user will have more \ 32 | control on the generated output, for more detailed information regarding the flags, please refer to section 5.3 - Script flags. 33 | 34 | To run the script without a configuration: 35 | 1. Untar the file by invoking - tar -zxvf sysinfo-snapshot-.tgz 36 | 2. Run the following command (as admin): 37 | #./sysinfo-snapshot.py 38 | 3. Extract the tar file from the default directory: /tmp/ 39 | 4. Open the relevant html/text files 40 | 41 | 5.2 Running the tool with a configuration file 42 | Running Sysinfo Snapshot with a configuration file will provide a more advanced control on the 43 | gathered information and configuration on the server and Nvidia's adapters, the configuration is 44 | intended to list all commands that are gathered by the Sysinfo Snapshot tool. 45 | Modifying the configuration file will allow the user to control which commands are allowed to run on 46 | the host running the tool. 47 | 48 | To run the script with a configuration file: 49 | 1. Untar the file by invoking - tar -zxvf sysinfo-snapshot-.tgz 50 | 2. Run the following command (as admin): 51 | #./sysinfo-snapshot.py --generate_config 52 | 3. Review the generated config.csv file and modify the Approved column based on the requirements 53 | 4. Run the following command (as admin): 54 | #./sysinfo-snapshot.py –config ./config.csv 55 | 5. Extract the tar file from the default directory: /tmp/ 56 | 6. Open the relevant html/text files 57 | 58 | 5.3 Script Flags \ 59 | There are many flags (options) for the user to refer to and add. Please see the list of flags below (5.1.1 to 5.1.23). \ 60 | 5.3.1 d|--dir \ 61 | By default, the destination directory is /tmp. The destination directory can be changed using -d|--dir \ 62 | flags/options. If the destination directory does not exist, the script automatically creates it. \ 63 | 5.3.2 v|--version \ 64 | show the tool's version information and exit. \ 65 | 5.3.3 -p|--perf \ 66 | By default, the "Performance tuning analyze" html file is added to the output. This html file dumps the \ 67 | performance status according to the Performance Tuning for Mellanox Adapters. In addition, it \ 68 | contains the output of the tool 'mlnx_tune'. If you add the '-p|' or the '--perf' flag, the output of the \ 69 | tests below is added to the html output: \ 70 | ib_write_bw \ 71 | ib_write_lat \ 72 | 5.3.4 --ufm \ 73 | Add ufm logs to the output. \ 74 | 5.3.5 --no_fw \ 75 | do not add firmware commands to the output. \ 76 | 5.3.6 --fsdump \ 77 | add fsdump firmware command to the output. \ 78 | 5.3.7 --mtusb \ 79 | By default, I2C firmware related server commands/functions are not added to the output. They can be \ 80 | added to the output by providing the '--mtusb' flag. \ 81 | 5.3.8 --with_inband \ 82 | add in-band cable info to the output. \ 83 | 5.3.9 --ibdiagnet_ext \ 84 | Add ibdiagnet ext command to the output. \ 85 | 5.3.10 --ibdiagnet \ 86 | By default, the 'ibdiagnet' command is not added to the output. It can be added by providing the '-- ibdiagnet' flag. \ 87 | 5.3.11 --no_ib \ 88 | By default, IB commands are added to the output. These commands can be removed from the output \ 89 | by providing the flag '--no_ib'. \ 90 | 5.3.12 --openstack \ 91 | gather openstack relevant conf and log files. \ 92 | 5.3.13 --asap \ 93 | gather asap relevant commands output. \ 94 | 5.3.14 --asap_tc \ 95 | gather asap tc filter commands output. \ 96 | 5.3.15 --rdma_debug \ 97 | gather rdma tool that comes with iproute2 commands output. \ 98 | 5.3.16 --gpu \ 99 | gather Nvidia GPU commands. \ 100 | 5.3.17 --json \ 101 | By default, the JSON output file is not generated. It can be added to the output by providing the \ 102 | '--json' flag. To have the JSON output file, make sure the json python module is installed. \ 103 | 5.3.18 --pcie \ 104 | By default, the PCIE commands/functions are not added to the output. They can be added by adding \ 105 | the '--pcie' flag, e.g. 'lspci -vvvxxxxx'. \ 106 | 5.3.19 --pcie_debug \ 107 | Generate only PCIE debug information. \ 108 | 5.3.20 --config \ 109 | set the customized configuration file path including the filename, to choose which commands are \ 110 | approved to run. In case a path is not provided, the default file name(config.csv) and it path are set \ 111 | for the same directory. \ 112 | 5.3.21 --generate_config \ 113 | Generates configuration file under provided path, Path must be full path, including file name. \ 114 | Generated config file will include all the commands available in the script listed. By default, all the \ 115 | commands that run will be marked as yes for execution, unless additional flag is required for them. \ 116 | In case a path is not provided, a default path is assumed, which is current directory with config.csv file \ 117 | name. \ 118 | 5.3.22 --check_fw \ 119 | This flag checks if the current adapter firmware is the latest version released. The expected output is \ 120 | in the performance html file (Internet access is required). \ 121 | 5.3.23 --verbose \ 122 | first verbosity level, available if option is provided only once, lists sections in process.second verbosity \ 123 | level, vailable if option is provided twice, lists sections and commands in process. \ 124 | 5.3.24 -t | --non_root \ 125 | Allow the tool to run as non-root user, commands/files that require root permissions are missing.\ 126 | 5.3.25 -t | --nvsm_dump \ 127 | Collect nvsm dump health.\ 128 | 5.3.26 --no_fw_regdumps.\ 129 | disable regdumps firmware command. 130 | 5.3.27 -- no_mstconfig.\ 131 | disable mstconfig firmware command. 132 | 5.3.28 --no_cables \ 133 | disable mlxlink, mget_temp, mlxmcg command that is related to cables. 134 | 5.3.29 --all_var_log \ 135 | collect all logs in /var/log/ dir. 136 | 6. Generate Config – Guidelines 137 | 138 | 6.1 Usage \ 139 | Adding --generate_config will generate a csv configuration file which includes all the commands \ 140 | available in the script listed. \ 141 | By default, all the commands that run will be marked as yes for execution, unless additional flag is \ 142 | required for them. \ 143 | Modifying the configuration file is done by only changing the values under "approved" column. \ 144 | The allowed values - "yes" or "no". \ 145 | To run the sysinfo-snapshot using the modified configuration file, please add the --config flag. \ 146 | Note: Adding additional flags while generating the configuration file will not change the \ 147 | defualt allowed values. 148 | 149 | 6.2 Generated config file format \ 150 | • First line has the sysinfo-snapshot generated version, \ 151 | e.g sysinfo-snapshot version 3.7.0. 152 | 153 | • csv configuration file format. \ 154 | 155 | Commands: generated automatically by the system \ 156 | Approved: "is" or "no" \ 157 | related flag: By default all the commands that run will be marked as yes for execution \ 158 | generated automatically by the system. please refer to Script Flags for more information. \ 159 | Files and directories are annotated with "file: " prefix, to help identifying them. 160 | 161 | 6.3 Functions \ 162 | Generated functions invoke multiple related queries that are gathered from the customer server. 163 | 164 | ibdev2pcidev: Map each IB device in /sys/class/infiniband/ to it PCI device \ 165 | ethtool_all_interfaces: Gather the output of ethtool command on all relevant ethernet interfaces \ 166 | with each of the following flags:"-i", "-g", "-a", "-k", "-c", "-T", "--show-priv-flags", "-n", "-l", "-x", "-S" \ 167 | ib_write_bw_test: Invoke relevant ib_write_bw test \ 168 | Installed_packages: Invoke relevant command: rpm -qa --last / dpkg --list / pkglist \ 169 | mst_commands_query_output: From each relevant mst device: \ 170 | • Gather mstregdump / mstdump command three consecutive runs \ 171 | • Gather mlxconfig/mstconfig with -e flag \ 172 | • Gather flint/mstflint with q and dc flags \ 173 | • Gather mlxdump with pcie_uc --all flag \ 174 | ethtool_version: Gets the ethtool version installed /usr/sbin/ethtool or /sbin/ethtool \ 175 | asap_parameters: Run the following: \ 176 | • ovs-dpctl dump-flows -m \ 177 | • tc qdisc show \ 178 | • ovs-vsctl get Open_vSwitch . other_config \ 179 | • ovs-vsctl show \ 180 | • ovs-ofctl dump-flows \ 181 | asap_tc_information: Run the following: \ 182 | tc -s filter show dev \ 183 | show_irq_affinity_all: From each relevant mst device, gather show_irq_affinity.sh output \ 184 | yy_MLX_modules_parameters: Gather relevant parameters from: /sys/module/mlx*/parameters/ \ 185 | yy_ib_modules_parameters_handler: Gather relevant IB parameters from: /sys/module/ib_*/parameters/ \ 186 | proc_net_bonding_files: Gather files from /proc/net/bonding/ \ 187 | sys_class_net_files: Gather files from /sys/class/net/ \ 188 | 189 | teamdctl_state / teamdctl_state_view / eamdctl_config_dump / teamdctl_config_dump_actual \ 190 | teamdctl_config_dump_noports: ther relevant teamdctl output 191 | 192 | show_pretty_gids: Gather relevant gids and gid_attrs information from: /sys/class/infiniband \ 193 | devlink_handler: Gather relevant output from each relevant devlink reporter \ 194 | ufm_logs: Gather relevant information from: /opt/ufm/scripts/vsysinfo output \ 195 | sys_class_net_ecn_ib: Gather relevant ecn files from /sys/class/net \ 196 | performance_lspci: Gather relevant information from: lspci, mstflint/flint for providing \ 197 | performance advise in the performance html file \ 198 | hyper_threading / core_frequency: Gather relevant information from: /proc/cpuinfo \ 199 | ib_mc_info_show / Multicast_Information: Gather relevant information from saquery output \ 200 | sm_version: Gather opensm version installed: rpm -qa | grep opensm / dpkg -l | grep opensm \ 201 | perfquery_cards_ports: Gather relevant information from perfquery command output 202 | ib_find_bad_ports / ib_find_disabled_ports_handler: Gather relevant information from iblinkinfo command output \ 203 | ib_switches_FW_scan_handler: Gather relevant information from ibswitches and ibdiagnet commands output \ 204 | ib_topology_viewer: Gather relevant information from ibnetdiscover command output \ 205 | get_numa_node_sys_files_exclude_uevent_files: Invoke the following command: \ 206 | "find /sys | grep numa_node | grep -v uevent" \ 207 | se_linux_status: Gather SELinux configuration from command getenforce output \ 208 | ip_forwarding: Gather relevant information from: \ 209 | /proc/sys/net/ipv4 \ 210 | /proc/sys/net/ipv6/ \ 211 | perf_samples: Gather relevant perf samples information from: \ 212 | ethtool -s and /sys/class/infiniband/device counters \ 213 | mget_temp_query: Gather information from mget_temp command output \ 214 | rdma_tool: Gather the output of: /opt/mellanox/iproute2/sbin/rdma \ 215 | command with each of the following flags: \ 216 | "resource show","resource show cm_id","resource show qp","res show cq" \ 217 | roce counters: Gather relevant counter and hardware counter information from Infiniband \ 218 | devices listed under - /sys/class/infiniband \ 219 | /sys/class/infiniband//ports/ \ 220 | USER: Gather relevant information from logname command output \ 221 | congestion_control_parameters: Gather relevant counter and hardware counter information from Infiniband devices listed under: \ 222 | /sys/kernel/debug/mlx5 \ 223 | networkManager_system_connections: Gather NetworkManager information: \ 224 | systemctl status NetworkManager \ 225 | get relevant connection files from: \ 226 | /etc/NetworkManager/system-connections/ 227 | 228 | Note, Commands that are not listed as function in the above list are invoked directly as a shell command. 229 | -------------------------------------------------------------------------------- /config.csv: -------------------------------------------------------------------------------- 1 | Generated by sysinfo-snapshot version 3.7.0 2 | Command,Approved,related flag 3 | arp -an,no, 4 | perf_samples,no,no_ib/perf 5 | flint -v,yes, 6 | /opt/mellanox/ethtool/sbin/ethtool --version,yes, 7 | file: performance_tuning_analyze.html,yes, 8 | file: /etc/host.conf,yes, 9 | file: /etc/systemd/system/network-online.target.wants/NetworkManager-wait-online.service,yes, 10 | file: /etc/*release*,yes, 11 | rdma_tool,no,rdma_debug 12 | lstopo-no-graphics -v -c,yes, 13 | file: /sys/class/net/,yes, 14 | uname,yes, 15 | lspci -vv,yes, 16 | ibnetdiscover,yes,no_ib 17 | ethtool_all_interfaces,yes, 18 | ib_write_bw_test,no,no_ib/perf 19 | teamdctl_state_view,yes, 20 | file: /etc/opensm/partitions.conf,yes, 21 | /etc/init.d/nv_peer_mem status,no,gpu 22 | file: /etc/ntp.conf,yes, 23 | snap_rpc.py controller_list,yes, 24 | file: /proc/devices,yes, 25 | ip link,no, 26 | file: sr_iov.html,yes, 27 | iscsiadm -m iface,yes, 28 | file: /proc/diskstats,yes, 29 | Installed_packages,yes, 30 | ip -6 addr show,no, 31 | get_numa_node_sys_files_exclude_uevent_files,yes, 32 | mlxconfig_query,yes, 33 | NetworkManager --print-config,yes, 34 | file: /proc/cmdline,yes, 35 | lspci,yes, 36 | ofed_info -s,yes, 37 | file: /etc/bluefield_version,yes, 38 | file: /etc/hosts,no, 39 | file: var/log/syslog,yes, 40 | lspci -tv |grep 'NVIDIA' -A7,no,gpu 41 | file: /boot/grub/grub.cfg,yes, 42 | show_irq_affinity_all,yes, 43 | iscsiadm --version,yes, 44 | se_linux_status,yes, 45 | roce counters,yes, 46 | file: /var/log/containers/neutron/openvswitch-agent.log,no,openstack 47 | ucx_info -v,yes, 48 | file: /proc/uptime,yes, 49 | file: /etc/hosts.deny,no, 50 | lscpu,yes, 51 | biosdecode,yes, 52 | show_pretty_gids,yes, 53 | file: /etc/hosts.allow,no, 54 | ib_find_bad_ports,yes,no_ib 55 | file: /proc/meminfo,yes, 56 | file: /etc/default/grub,yes, 57 | file: /etc/udev/udev.conf,yes, 58 | mget_temp_query,yes, 59 | file: /sys/class/infiniband/*/device/,yes, 60 | dkms status,yes, 61 | file: /etc/dhcp/dhclient.conf,yes, 62 | virsh version,yes, 63 | ucx_info -c,yes, 64 | perfquery_cards_ports,yes,no_ib 65 | dmidecode -s processor-manufacturer,yes, 66 | blkid -c /dev/null | sort,yes, 67 | file: /etc/sysconfig/network-scripts/ifcfg*,no, 68 | file :/etc/libvma.conf,yes, 69 | core_frequency,yes, 70 | sm_status,yes,no_ib 71 | ibdev2pcidev,yes, 72 | file: /etc/tuned.conf,yes, 73 | USER,no, 74 | virsh vcpupin,yes, 75 | ompi_info,yes, 76 | file: /proc/net/softnet_stat,yes, 77 | file: /etc/security/limits.conf,yes, 78 | file : /proc/config.gz,yes, 79 | file :/proc/zoneinfo,yes, 80 | file: /boot/grub/grub.conf,yes, 81 | get_ib_sys_files_exclude_uevent_files,yes,no_ib 82 | file: /etc/infiniband/openib.conf.rpmsave,yes, 83 | file: /etc/modprobe.d/vxlan.conf,yes, 84 | ip_forwarding,yes, 85 | numastat -n,yes, 86 | ibstat,yes, 87 | netstat -anp,yes, 88 | /etc/mlnx_snap,yes, 89 | sys_class_net_ecn_ib,yes, 90 | teamdctl_config_dump,yes, 91 | irqbalance,yes, 92 | file: /etc/rdma/rdma.conf,yes, 93 | file :/boot/config-$(uname -r),yes, 94 | teamdctl_state,yes, 95 | ovs-vsctl --version,yes, 96 | file: /etc/opensm/opensm.conf,yes, 97 | free,yes, 98 | file: /proc/sys/net/ipv4/igmp_max_memberships,yes, 99 | file: /boot/grub/menu.lst,yes, 100 | sys_class_net_files,yes, 101 | ib_find_disabled_ports,yes,no_ib 102 | virsh list --all,yes, 103 | mstflint -v,yes, 104 | route -n,no, 105 | file: /etc/sysctl.conf,yes, 106 | mlxcables --DDM/--read_all_regs,yes, 107 | file: /etc/sysconfig/network-scripts/,no, 108 | file: var/log/boot.log,yes, 109 | python_used_version,yes, 110 | file: /boot/grub2/menu.lst,yes, 111 | dmidecode,yes, 112 | ib_write_bw -h | grep -i cuda,no,gpu 113 | /usr/local/cuda/extras/demo_suite/bandwidthTest --memory=pinned --mode=range --start=65536 --end=65011712 --increment=4194304 --device=all --htod,no,gpu 114 | ip -s -s link show,no, 115 | file: /etc/systemd/system/sysinit.target.wants/openibd.service,yes, 116 | file: /proc/buddyinfo,yes, 117 | asap_parameters,no,asap 118 | ibdev2netdev,yes, 119 | file: /proc/net/dev_mcast,yes, 120 | netstat -nlp,yes, 121 | iscsiadm -m host,yes, 122 | sm_master_is,yes,no_ib 123 | file: /proc/modules,yes, 124 | lsblk,yes, 125 | ucx_info -f,yes, 126 | cma_roce_mode,yes, 127 | file: /etc/systemd/system/mlnx_interface_mgr@.service,yes, 128 | sys_class_infiniband_ib_paameters,yes, 129 | file: /etc/netplan/,no, 130 | iscsiadm -m node,yes, 131 | mlxdump,no,no_fw/fsdump 132 | tgtadm --version,yes, 133 | numactl --hardware,yes, 134 | mst_commands_query_output,yes,no_fw 135 | file: /sys/bus/pci/drivers/,yes, 136 | ip_link_show_devices,yes, 137 | initctl list,yes, 138 | file: /sys/bus/pci/devices/,yes, 139 | virtnet query --all,yes, 140 | performance_lspci,yes, 141 | ufm_logs,no,ufm 142 | dmesg -T,yes, 143 | ib_topology_viewer,yes,no_ib 144 | modinfo,yes, 145 | sysclass_IB_modules_parameters,yes, 146 | journalctl -u virtio-net-controller,yes, 147 | mst status -v,yes, 148 | service iptables status,yes, 149 | ib_mc_info_show,yes,no_ib 150 | file: /proc/net/igmp,yes, 151 | service irqbalance status,yes, 152 | ip -s -s addr show,no, 153 | file: /etc/issue,yes, 154 | file: /var/lib/config-data/puppet-generated/neutron/,no, openstack 155 | journalctl -u mlnx_snap,yes, 156 | file: /etc/modprobe.d/mlnx.conf,yes, 157 | tuned-adm active,yes, 158 | file: /sys/class/infiniband/*/iov,yes, 159 | lsmod,yes, 160 | sminfo,yes,no_ib 161 | nvidia-smi -q -d clock,no,gpu 162 | Multicast_Information,yes,no_ib 163 | file: /proc/pagetypeinfo,yes, 164 | ip m s,no, 165 | file: /etc/default/mlnx_snap,yes, 166 | ip route show table all,no, 167 | ip -6 route show,no, 168 | mst status,yes, 169 | lspci -xxxvvv,yes, 170 | sm_version,yes,no_ib 171 | ibv_devinfo -v,yes, 172 | file: /proc/dma,yes, 173 | teamdctl_config_dump_noports,yes, 174 | flint -d xx q,yes, 175 | ib_switches_FW_scan,yes,no_ib 176 | yy_MLX_modules_parameters,yes, 177 | ibstatus,yes, 178 | mount,yes, 179 | file :/proc/interrupts,yes, 180 | file: /proc/net/vlan/,yes, 181 | file: var/log/messages,yes, 182 | ibswitches,yes,no_ib 183 | file: /etc/resolv.conf,yes, 184 | file: /var/lib/config-data/puppet-generated/nova_libvirt,no, openstack 185 | congestion_control_parameters,yes, 186 | /usr/local/cuda/extras/demo_suite/deviceQuery,no,gpu 187 | mlxreg -d --reg_name ROCE_ACCL --get,yes, 188 | nvidia-smi,no,gpu 189 | iscsiadm -m session,yes, 190 | bridge fdb show dev p3p1,yes, 191 | snap_rpc.py emulation_functions_list,yes, 192 | service cpuspeed status,yes, 193 | mlnx_qos_handler,yes, 194 | service --status-all,yes, 195 | file: /proc/slabinfo,yes, 196 | file: /etc/modprobe.conf,yes, 197 | lspci -vvvxxxxx,no,pcie 198 | gcc --version,yes, 199 | mlnx_tune,yes,no_ib 200 | hyper_threading,yes, 201 | netstat -nr,no, 202 | brctl --version,yes, 203 | ovs-dpctl show,yes, 204 | lspci_vf,yes, 205 | lspci -tv,yes, 206 | file: /etc/network/,no, 207 | memlock,yes, 208 | ibhosts,yes,no_ib 209 | devlink_handler,yes, 210 | file: /sys/fs/cgroup/net_prio/net_prio.ifpriomap,yes, 211 | latency,no,no_ib/perf 212 | file: /proc/mounts,yes, 213 | "nvidia-smi --format=csv --query-supported-clocks=gr,mem",no,gpu 214 | netstat -i,yes, 215 | hostname,no, 216 | file: /etc/sysconfig/network/ifcfg*,yes, 217 | netstat -s,yes, 218 | sysctl -a,yes, 219 | mlxlink / mstlink,yes, 220 | ecn_configuration,yes, 221 | proc_net_bonding_files,yes, 222 | file: /etc/modprobe.d/,yes, 223 | ip n s,no, 224 | file: /etc/udev/rules.d/,yes, 225 | file: /var/log/containers/nova/nova-compute.log,no,openstack 226 | file: /sys/devices/system/clocksource/clocksource0/current_clocksource,yes, 227 | service firewalld status,yes, 228 | networkManager_system_connections,yes, 229 | time,yes, 230 | file: /proc/version,yes, 231 | lstopo-no-graphics,yes, 232 | file: /sys/devices/system/node/,yes, 233 | df -lh,yes, 234 | cma_roce_tos,yes, 235 | file: /boot/grub2/grub.cfg,yes, 236 | date,yes, 237 | file: /proc/partitions,yes, 238 | file: /proc/sys/net/ipv4/igmp_max_msf,yes, 239 | file: /proc/cpuinfo,yes, 240 | fdisk -l,yes, 241 | ofed_info,yes, 242 | /usr/local/cuda/extras/demo_suite/bandwidthTest --memory=pinned --mode=range --start=65536 --end=65011712 --increment=4194304 --device=all --dtoh,no,gpu 243 | file: /etc/yum.conf,yes, 244 | file: /lib/udev/rules.d/,yes, 245 | modinfo nv_peer_mem,no,gpu 246 | ps -eLo,yes, 247 | file: /proc/stat,yes, 248 | mlxmcg -d,yes,no_fw 249 | ulimit -a,yes, 250 | mlxfwmanager --online-query-psid,no,check_fw 251 | ifconfig -a,no, 252 | file: /etc/infiniband/,yes, 253 | brctl show,yes, 254 | ibdiagnet,no,no_ib/ibdiagnet 255 | asap_tc_information,no,asap_tc 256 | ovs-vsctl show,yes, 257 | ethtool_version,yes, 258 | tgtadm --mode target --op show,yes, 259 | mlxcables,yes, 260 | teamdctl_config_dump_actual,yes, 261 | file: /boot/grub2/grub.conf,yes, 262 | nvidia-smi topo -m,no,gpu 263 | --------------------------------------------------------------------------------