├── LICENSE.md ├── README.md ├── ioprof.pl └── ioprof.py /LICENSE.md: -------------------------------------------------------------------------------- 1 | GNU GENERAL PUBLIC LICENSE 2 | Version 2, June 1991 3 | 4 | Copyright (C) 1989, 1991 Free Software Foundation, Inc., 5 | 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA 6 | Everyone is permitted to copy and distribute verbatim copies 7 | of this license document, but changing it is not allowed. 8 | 9 | Preamble 10 | 11 | The licenses for most software are designed to take away your 12 | freedom to share and change it. By contrast, the GNU General Public 13 | License is intended to guarantee your freedom to share and change free 14 | software--to make sure the software is free for all its users. This 15 | General Public License applies to most of the Free Software 16 | Foundation's software and to any other program whose authors commit to 17 | using it. (Some other Free Software Foundation software is covered by 18 | the GNU Lesser General Public License instead.) You can apply it to 19 | your programs, too. 20 | 21 | When we speak of free software, we are referring to freedom, not 22 | price. Our General Public Licenses are designed to make sure that you 23 | have the freedom to distribute copies of free software (and charge for 24 | this service if you wish), that you receive source code or can get it 25 | if you want it, that you can change the software or use pieces of it 26 | in new free programs; and that you know you can do these things. 27 | 28 | To protect your rights, we need to make restrictions that forbid 29 | anyone to deny you these rights or to ask you to surrender the rights. 30 | These restrictions translate to certain responsibilities for you if you 31 | distribute copies of the software, or if you modify it. 32 | 33 | For example, if you distribute copies of such a program, whether 34 | gratis or for a fee, you must give the recipients all the rights that 35 | you have. You must make sure that they, too, receive or can get the 36 | source code. And you must show them these terms so they know their 37 | rights. 38 | 39 | We protect your rights with two steps: (1) copyright the software, and 40 | (2) offer you this license which gives you legal permission to copy, 41 | distribute and/or modify the software. 42 | 43 | Also, for each author's protection and ours, we want to make certain 44 | that everyone understands that there is no warranty for this free 45 | software. If the software is modified by someone else and passed on, we 46 | want its recipients to know that what they have is not the original, so 47 | that any problems introduced by others will not reflect on the original 48 | authors' reputations. 49 | 50 | Finally, any free program is threatened constantly by software 51 | patents. We wish to avoid the danger that redistributors of a free 52 | program will individually obtain patent licenses, in effect making the 53 | program proprietary. To prevent this, we have made it clear that any 54 | patent must be licensed for everyone's free use or not licensed at all. 55 | 56 | The precise terms and conditions for copying, distribution and 57 | modification follow. 58 | 59 | GNU GENERAL PUBLIC LICENSE 60 | TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION 61 | 62 | 0. This License applies to any program or other work which contains 63 | a notice placed by the copyright holder saying it may be distributed 64 | under the terms of this General Public License. The "Program", below, 65 | refers to any such program or work, and a "work based on the Program" 66 | means either the Program or any derivative work under copyright law: 67 | that is to say, a work containing the Program or a portion of it, 68 | either verbatim or with modifications and/or translated into another 69 | language. (Hereinafter, translation is included without limitation in 70 | the term "modification".) Each licensee is addressed as "you". 71 | 72 | Activities other than copying, distribution and modification are not 73 | covered by this License; they are outside its scope. The act of 74 | running the Program is not restricted, and the output from the Program 75 | is covered only if its contents constitute a work based on the 76 | Program (independent of having been made by running the Program). 77 | Whether that is true depends on what the Program does. 78 | 79 | 1. You may copy and distribute verbatim copies of the Program's 80 | source code as you receive it, in any medium, provided that you 81 | conspicuously and appropriately publish on each copy an appropriate 82 | copyright notice and disclaimer of warranty; keep intact all the 83 | notices that refer to this License and to the absence of any warranty; 84 | and give any other recipients of the Program a copy of this License 85 | along with the Program. 86 | 87 | You may charge a fee for the physical act of transferring a copy, and 88 | you may at your option offer warranty protection in exchange for a fee. 89 | 90 | 2. You may modify your copy or copies of the Program or any portion 91 | of it, thus forming a work based on the Program, and copy and 92 | distribute such modifications or work under the terms of Section 1 93 | above, provided that you also meet all of these conditions: 94 | 95 | a) You must cause the modified files to carry prominent notices 96 | stating that you changed the files and the date of any change. 97 | 98 | b) You must cause any work that you distribute or publish, that in 99 | whole or in part contains or is derived from the Program or any 100 | part thereof, to be licensed as a whole at no charge to all third 101 | parties under the terms of this License. 102 | 103 | c) If the modified program normally reads commands interactively 104 | when run, you must cause it, when started running for such 105 | interactive use in the most ordinary way, to print or display an 106 | announcement including an appropriate copyright notice and a 107 | notice that there is no warranty (or else, saying that you provide 108 | a warranty) and that users may redistribute the program under 109 | these conditions, and telling the user how to view a copy of this 110 | License. (Exception: if the Program itself is interactive but 111 | does not normally print such an announcement, your work based on 112 | the Program is not required to print an announcement.) 113 | 114 | These requirements apply to the modified work as a whole. If 115 | identifiable sections of that work are not derived from the Program, 116 | and can be reasonably considered independent and separate works in 117 | themselves, then this License, and its terms, do not apply to those 118 | sections when you distribute them as separate works. But when you 119 | distribute the same sections as part of a whole which is a work based 120 | on the Program, the distribution of the whole must be on the terms of 121 | this License, whose permissions for other licensees extend to the 122 | entire whole, and thus to each and every part regardless of who wrote it. 123 | 124 | Thus, it is not the intent of this section to claim rights or contest 125 | your rights to work written entirely by you; rather, the intent is to 126 | exercise the right to control the distribution of derivative or 127 | collective works based on the Program. 128 | 129 | In addition, mere aggregation of another work not based on the Program 130 | with the Program (or with a work based on the Program) on a volume of 131 | a storage or distribution medium does not bring the other work under 132 | the scope of this License. 133 | 134 | 3. You may copy and distribute the Program (or a work based on it, 135 | under Section 2) in object code or executable form under the terms of 136 | Sections 1 and 2 above provided that you also do one of the following: 137 | 138 | a) Accompany it with the complete corresponding machine-readable 139 | source code, which must be distributed under the terms of Sections 140 | 1 and 2 above on a medium customarily used for software interchange; or, 141 | 142 | b) Accompany it with a written offer, valid for at least three 143 | years, to give any third party, for a charge no more than your 144 | cost of physically performing source distribution, a complete 145 | machine-readable copy of the corresponding source code, to be 146 | distributed under the terms of Sections 1 and 2 above on a medium 147 | customarily used for software interchange; or, 148 | 149 | c) Accompany it with the information you received as to the offer 150 | to distribute corresponding source code. (This alternative is 151 | allowed only for noncommercial distribution and only if you 152 | received the program in object code or executable form with such 153 | an offer, in accord with Subsection b above.) 154 | 155 | The source code for a work means the preferred form of the work for 156 | making modifications to it. For an executable work, complete source 157 | code means all the source code for all modules it contains, plus any 158 | associated interface definition files, plus the scripts used to 159 | control compilation and installation of the executable. However, as a 160 | special exception, the source code distributed need not include 161 | anything that is normally distributed (in either source or binary 162 | form) with the major components (compiler, kernel, and so on) of the 163 | operating system on which the executable runs, unless that component 164 | itself accompanies the executable. 165 | 166 | If distribution of executable or object code is made by offering 167 | access to copy from a designated place, then offering equivalent 168 | access to copy the source code from the same place counts as 169 | distribution of the source code, even though third parties are not 170 | compelled to copy the source along with the object code. 171 | 172 | 4. You may not copy, modify, sublicense, or distribute the Program 173 | except as expressly provided under this License. Any attempt 174 | otherwise to copy, modify, sublicense or distribute the Program is 175 | void, and will automatically terminate your rights under this License. 176 | However, parties who have received copies, or rights, from you under 177 | this License will not have their licenses terminated so long as such 178 | parties remain in full compliance. 179 | 180 | 5. You are not required to accept this License, since you have not 181 | signed it. However, nothing else grants you permission to modify or 182 | distribute the Program or its derivative works. These actions are 183 | prohibited by law if you do not accept this License. Therefore, by 184 | modifying or distributing the Program (or any work based on the 185 | Program), you indicate your acceptance of this License to do so, and 186 | all its terms and conditions for copying, distributing or modifying 187 | the Program or works based on it. 188 | 189 | 6. Each time you redistribute the Program (or any work based on the 190 | Program), the recipient automatically receives a license from the 191 | original licensor to copy, distribute or modify the Program subject to 192 | these terms and conditions. You may not impose any further 193 | restrictions on the recipients' exercise of the rights granted herein. 194 | You are not responsible for enforcing compliance by third parties to 195 | this License. 196 | 197 | 7. If, as a consequence of a court judgment or allegation of patent 198 | infringement or for any other reason (not limited to patent issues), 199 | conditions are imposed on you (whether by court order, agreement or 200 | otherwise) that contradict the conditions of this License, they do not 201 | excuse you from the conditions of this License. If you cannot 202 | distribute so as to satisfy simultaneously your obligations under this 203 | License and any other pertinent obligations, then as a consequence you 204 | may not distribute the Program at all. For example, if a patent 205 | license would not permit royalty-free redistribution of the Program by 206 | all those who receive copies directly or indirectly through you, then 207 | the only way you could satisfy both it and this License would be to 208 | refrain entirely from distribution of the Program. 209 | 210 | If any portion of this section is held invalid or unenforceable under 211 | any particular circumstance, the balance of the section is intended to 212 | apply and the section as a whole is intended to apply in other 213 | circumstances. 214 | 215 | It is not the purpose of this section to induce you to infringe any 216 | patents or other property right claims or to contest validity of any 217 | such claims; this section has the sole purpose of protecting the 218 | integrity of the free software distribution system, which is 219 | implemented by public license practices. Many people have made 220 | generous contributions to the wide range of software distributed 221 | through that system in reliance on consistent application of that 222 | system; it is up to the author/donor to decide if he or she is willing 223 | to distribute software through any other system and a licensee cannot 224 | impose that choice. 225 | 226 | This section is intended to make thoroughly clear what is believed to 227 | be a consequence of the rest of this License. 228 | 229 | 8. If the distribution and/or use of the Program is restricted in 230 | certain countries either by patents or by copyrighted interfaces, the 231 | original copyright holder who places the Program under this License 232 | may add an explicit geographical distribution limitation excluding 233 | those countries, so that distribution is permitted only in or among 234 | countries not thus excluded. In such case, this License incorporates 235 | the limitation as if written in the body of this License. 236 | 237 | 9. The Free Software Foundation may publish revised and/or new versions 238 | of the General Public License from time to time. Such new versions will 239 | be similar in spirit to the present version, but may differ in detail to 240 | address new problems or concerns. 241 | 242 | Each version is given a distinguishing version number. If the Program 243 | specifies a version number of this License which applies to it and "any 244 | later version", you have the option of following the terms and conditions 245 | either of that version or of any later version published by the Free 246 | Software Foundation. If the Program does not specify a version number of 247 | this License, you may choose any version ever published by the Free Software 248 | Foundation. 249 | 250 | 10. If you wish to incorporate parts of the Program into other free 251 | programs whose distribution conditions are different, write to the author 252 | to ask for permission. For software which is copyrighted by the Free 253 | Software Foundation, write to the Free Software Foundation; we sometimes 254 | make exceptions for this. Our decision will be guided by the two goals 255 | of preserving the free status of all derivatives of our free software and 256 | of promoting the sharing and reuse of software generally. 257 | 258 | NO WARRANTY 259 | 260 | 11. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY 261 | FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN 262 | OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES 263 | PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED 264 | OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF 265 | MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS 266 | TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE 267 | PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, 268 | REPAIR OR CORRECTION. 269 | 270 | 12. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING 271 | WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR 272 | REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, 273 | INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING 274 | OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED 275 | TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY 276 | YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER 277 | PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE 278 | POSSIBILITY OF SUCH DAMAGES. 279 | 280 | END OF TERMS AND CONDITIONS 281 | 282 | How to Apply These Terms to Your New Programs 283 | 284 | If you develop a new program, and you want it to be of the greatest 285 | possible use to the public, the best way to achieve this is to make it 286 | free software which everyone can redistribute and change under these terms. 287 | 288 | To do so, attach the following notices to the program. It is safest 289 | to attach them to the start of each source file to most effectively 290 | convey the exclusion of warranty; and each file should have at least 291 | the "copyright" line and a pointer to where the full notice is found. 292 | 293 | 294 | Copyright (C) 295 | 296 | This program is free software; you can redistribute it and/or modify 297 | it under the terms of the GNU General Public License as published by 298 | the Free Software Foundation; either version 2 of the License, or 299 | (at your option) any later version. 300 | 301 | This program is distributed in the hope that it will be useful, 302 | but WITHOUT ANY WARRANTY; without even the implied warranty of 303 | MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 304 | GNU General Public License for more details. 305 | 306 | You should have received a copy of the GNU General Public License along 307 | with this program; if not, write to the Free Software Foundation, Inc., 308 | 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. 309 | 310 | Also add information on how to contact you by electronic and paper mail. 311 | 312 | If the program is interactive, make it output a short notice like this 313 | when it starts in an interactive mode: 314 | 315 | Gnomovision version 69, Copyright (C) year name of author 316 | Gnomovision comes with ABSOLUTELY NO WARRANTY; for details type `show w'. 317 | This is free software, and you are welcome to redistribute it 318 | under certain conditions; type `show c' for details. 319 | 320 | The hypothetical commands `show w' and `show c' should show the appropriate 321 | parts of the General Public License. Of course, the commands you use may 322 | be called something other than `show w' and `show c'; they could even be 323 | mouse-clicks or menu items--whatever suits your program. 324 | 325 | You should also get your employer (if you work as a programmer) or your 326 | school, if any, to sign a "copyright disclaimer" for the program, if 327 | necessary. Here is a sample; alter the names: 328 | 329 | Yoyodyne, Inc., hereby disclaims all copyright interest in the program 330 | `Gnomovision' (which makes passes at compilers) written by James Hacker. 331 | 332 | , 1 April 1989 333 | Ty Coon, President of Vice 334 | 335 | This General Public License does not permit incorporating your program into 336 | proprietary programs. If your program is a subroutine library, you may 337 | consider it more useful to permit linking proprietary applications with the 338 | library. If this is what you want to do, use the GNU Lesser General 339 | Public License instead of this License. 340 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | DISCONTINUATION OF PROJECT 2 | 3 | This project will no longer be maintained by Intel. 4 | 5 | Intel has ceased development and contributions including, but not limited to, maintenance, bug fixes, new releases, or updates, to this project. 6 | 7 | Intel no longer accepts patches to this project. 8 | 9 | If you have an ongoing need to use this project, are interested in independently developing it, or would like to maintain patches for the open source software community, please create your own fork of this project. 10 | 11 | Contact: webadmin@linux.intel.com 12 | Linux I/O Profiler (ioprof) 13 | =========================== 14 | 15 | The Linux I/O profiler (ioprof) is a tool that provides significant insight into I/O workloads 16 | while remaining easy to use. It reports the following information: 17 | 18 | * I/O Histogram - Great for determining size of hot data for SSD caching 19 | * I/O Heatmap - Useful visualization to "see" where the hot data resides 20 | * I/O Size Stats - IOPS and bandwidth stats, which is useful for mixed workloads 21 | * Top Files (opt) - Can ID top accessed files in EXT3/EXT4 filesystems 22 | * Zipf Theta - An estimate of Zipfian distribution theta 23 | 24 | The tool is recommended to be used to further analyze I/O intensive workloads after running tools like iostat, since blktrace/blkparse can affect performance. 25 | 26 | It is intended to be stable enough to use to profile production systems and makes every 27 | attempt to minimize resource utilization. In additon, the trace file is self-contained 28 | and can be offloaded for analysis on a separate system. 29 | 30 | Contents 31 | ======== 32 | 33 | * README - This file 34 | * LICENSE - GPLv2 license 35 | * ioprof.pl - The script 36 | 37 | Dependencies 38 | ============ 39 | Perl v5.x and Perl Core Library 40 | 41 | Requires the following tools: 42 | * fdisk 43 | * blktrace 44 | * blkparse 45 | 46 | Optional PDF report requires: 47 | * gnuplot 48 | * pdf2latex 49 | * terminal png 50 | 51 | Design 52 | ====== 53 | The tool currently groups statistics into 1MB "buckets" to provide relatively 54 | accurate results, while minimizing system resources. 55 | 56 | TODO: 57 | ===== 58 | * Confirm XFS filesystem tracing 59 | * Add option to specifiy output file name 60 | * Add option to specify temp directory 61 | * Improve file mapping performance 62 | 63 | Maintainers 64 | =========== 65 | Benjamin Donie 66 | -------------------------------------------------------------------------------- /ioprof.pl: -------------------------------------------------------------------------------- 1 | #!/usr/bin/perl -w 2 | # I/O Profiler for Linux 3 | # Copyright (c) 2017, Intel Corporation. 4 | # 5 | # This program is free software; you can redistribute it and/or modify it 6 | # under the terms and conditions of the GNU General Public License, 7 | # version 2, as published by the Free Software Foundation. 8 | # 9 | # This program is distributed in the hope it will be useful, but WITHOUT 10 | # ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or 11 | # FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for 12 | # more details. 13 | # 14 | # You should have received a copy of the GNU General Public License 15 | # along with this program; if not, write to the Free Software 16 | # Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. 17 | 18 | ### I/O profing tool 19 | my $VERSION = '2.0.5'; 20 | print "$0 ($VERSION)\n"; 21 | 22 | my $VERBOSE = 0; 23 | my $DEBUG = 0; 24 | my $FASTFILE = 1; 25 | my $SINGLE_THREADED = 0; 26 | my $READAHEAD = 1; 27 | my $THREAD_POOL = 0; 28 | 29 | use strict; 30 | use POSIX ":sys_wait_h"; 31 | use Getopt::Std; 32 | use threads; 33 | use threads::shared; 34 | use Thread::Semaphore; 35 | use Thread::Queue; 36 | $| = 1; # Force flush after every write/print 37 | 38 | ### Shared Variables: Locking requird when multi-threaded 39 | my $io_total :shared; # Number of total I/O's 40 | my $read_total :shared; # Number of buckets read (1 I/O can touch many buckets) 41 | my $write_total :shared; # Number of buckets written (1 I/O can touch many buckets) 42 | my @reads :shared; # Array of read hits by bucket ID 43 | my @writes :shared; # Array of write hits by bucket ID 44 | my %r_totals :shared; # Hash of read I/O's with I/O size as key 45 | my %w_totals :shared; # Hash of write I/O's with I/O size as key 46 | my $bucket_hits_total :shared; # Total number of bucket hits (not the total buckets) 47 | my $total_blocks :shared; # Total number of LBA's accessed during profiling 48 | my %files_to_lbas :shared; # Files and the lba ranges associated with them 49 | my $max_bucket_hits :shared; # The hottest bucket 50 | my @bucket_to_files :shared; # List of files that reside on each bucket 51 | my $TERM :shared = 0; # Thread pool done with work 52 | my $trace_files :shared = 0; # Map filesystem files to block LBAs 53 | 54 | ### Semaphores: These are the locks for the shared variables 55 | my $read_semaphore; # Lock for the global read hit array 56 | my $write_semaphore; # Lock for the global write hit array 57 | my $read_totals_semaphore; # Lock for the global read totals 58 | my $write_totals_semaphore; # Lock for the global write totals 59 | my $total_semaphore; # Lock for the global I/O totals 60 | my $total_blocks_semaphore; # Lock for the global total LBA's accessed 61 | my $files_to_lbas_semaphore; # Lock for the global file->lba mapping hash 62 | my $max_bucket_hits_semaphore; # Lock for the global maximum hits per bucket 63 | my $bucket_to_files_semaphore; # Lock for the global bucket_to_files; 64 | my $term_semaphore; # Lock for the global TERM; 65 | my $trace_files_semaphore; # Lock for the global trace_files; 66 | 67 | ### Thread Queue 68 | my $IDLE_QUEUE = Thread::Queue->new(); 69 | 70 | ### Thread-local Variables: We use these to avoid locking constantly 71 | my $thread_io_total=0; # Thread-local total I/O count (I/O ops) 72 | my (%thread_r_totals, %thread_w_totals); # Thread-local I/O size counts (ops) 73 | my $thread_bucket_hits_total=0; # Thread-local total bucket hits (buckets) 74 | my $thread_read_total=0; # Thread-local total read count (I/O ops) 75 | my $thread_write_total=0; # Thread-local total write count (I/O ops) 76 | my %thread_reads; # Thread-local read count hash (buckets) 77 | my %thread_writes; # Thread-local write count hash (buckets) 78 | my $thread_total_blocks=0; # Thread-local total blocks accessed (lbas) 79 | my $thread_max_bucket_hits=0; # Thread-local maximum bucket hits (bucket hits) 80 | 81 | # Globals (Single-threaded, so no locking required) 82 | my %file_hit_count; # Count of I/O's to each file 83 | my @files_to_cleanup; # Files to delete after running this script 84 | my $CMD; # Command to run 85 | my $total_lbas; # Total logical blocks, regardless of sector size 86 | my $tar_file; # .tar file outputted from 'trace' mode 87 | my $fdisk_file; # File capture of fdisk tool output 88 | my @pidlist; # Multi-process PID array, to keep track 89 | 90 | 91 | my $top_files = ""; # Top files list 92 | my $dev = ""; # Device (e.g. /dev/sda1) 93 | my $dev_str = ""; # Device string (e.g. sda1 for /dev/sda1) 94 | 95 | ### Unit Scales 96 | my $KiB = 1024; 97 | my $MiB = 1048576; 98 | my $GiB = 1073741824; 99 | 100 | ### Config Settings 101 | my $bucket_size = 1 * $MiB; # Size of the bucket for totaling I/O counts (e.g. 1MB buckets) 102 | my $num_buckets = 1; # Number of total buckets for this device 103 | my $timeout = 3; # Seconds between each print 104 | my $runtime = 0; # Runtime for 'live' and 'trace' modes 105 | my $live_iterations = 0; # How many iterations for live mode. Each iteration is $timeout seconds long 106 | my $sector_size = 512; # Sector size (usually obtained with fdisk) 107 | my $live = 0; # Live mode 0=disabled, 1=enabled (command line arg) 108 | my $percent = 0.020; # Histogram threshold for each level (e.g. 0.02% of total drive size) 109 | my $total_capacity_GiB = 0; # Total drive capacity 110 | my $mode = "unknown"; # Processing mode (live, trace, post) 111 | my $pdf_report = 0; # Generate a PDF report instead of a text report 112 | my $top_count_limit = 10; # How many files to list in Top Files list (e.g. Top 10 files) 113 | my $thread_count = 0; # Thread count 114 | my $cpu_affinity = 0; # Tie each thread to a CPU for load balancing 115 | my $thread_max = 128; # Maximum thread count 116 | my $buffer_size = 1024; # blktrace buffer size 117 | my $buffer_count = 8; # blktrace buffer count 118 | 119 | ### Gnuplot Settings 120 | my $xwidth = 800; # gnuplot x-width 121 | my $yheight = 600; # gnuplot y-height 122 | 123 | ### Analysis strings 124 | my $analysis_histogram_iops = "Histogram analysis coming soon.\n"; 125 | my $analysis_heatmap = "Heatmap analysis coming soon.\n"; 126 | my $analysis_stats_iops = "IOPS analysis coming soon.\n"; 127 | my $analysis_stats_bw = "Bandwidth analysis coming soon.\n"; 128 | 129 | ### Heatmap Globals 130 | my $SCALEX=5; # Scale heatmap to term width - $SCALEX chars 131 | my $SCALEY=20; # Scale heatmap to term height - $SCALEY chars 132 | my $min_x=5; # Minium terminal width in chars 133 | my $min_y=5; # Minimum terminal height in chars 134 | 135 | ### ANSI COLORS 136 | my $black = "\e[40m"; 137 | my $red = "\e[41m"; 138 | my $green = "\e[42m"; 139 | my $yellow = "\e[43m"; 140 | my $blue = "\e[44m"; 141 | my $magenta = "\e[45m"; 142 | my $cyan = "\e[46m"; 143 | my $white = "\e[47m"; 144 | my $none = "\e[0m"; 145 | 146 | ### Heatmap Key 147 | my @colors=($white,$blue,$cyan,$green,$yellow,$magenta,$red); 148 | 149 | ### Heatmap Globals 150 | my $color_index; # Index in heatmap array 151 | my $choices=scalar(@colors); # Number of color choices 152 | my $vpc; # VPC=Values Per Choice. IOPS Range for each color 153 | my $cap=0; # Maximum IOPS per heatmap block 154 | my $rate; # How densely we pack buckets into heatmap blocks 155 | my @block_map; # Heatmap to print 156 | 157 | ### File Mapping Globals 158 | my $fibmap = 1; # FIBMAP ioctl number 159 | my $extents; # EXT extents 160 | my $mounttype; # Filesystem type (e.g. ext4) 161 | my $mountpoint; # Mountpoint 162 | 163 | ### Print usage 164 | sub usage 165 | { 166 | print "Invalid command\n\n"; 167 | print @ARGV; 168 | print "Usage:\n"; 169 | print "$0 -m trace -d -r [-v] [-f] # run trace for post-processing later\n"; 170 | print "$0 -m post -t [-v] [-p] # post-process mode\n"; 171 | print "$0 -m live -d -r [-v] # live mode\n"; 172 | print "\nCommand Line Arguments:\n"; 173 | print "-d : The device to trace (e.g. /dev/sdb). You can run traces to multiple devices (e.g. /dev/sda and /dev/sdb)\n"; 174 | print " at the same time, but please only run 1 trace to a single device (e.g. /dev/sdb) at a time\n"; 175 | print "-r : Runtime (seconds) for tracing\n"; 176 | print "-t : A .tar file is created during the 'trace' phase. Please use this file for the 'post' phase\n"; 177 | print " You can offload this file and run the 'post' phase on another system.\n"; 178 | print "-v : (OPTIONAL) Print verbose messages.\n"; 179 | print "-f : (OPTIONAL) Map all files on the device specified by -d during 'trace' phase to their LBA ranges.\n"; 180 | print " This is useful for determining the most fequently accessed files, but may take a while on really large filesystems\n"; 181 | print "-p : (OPTIONAL) Generate a .pdf output file in addition to STDOUT. This requires 'pdflatex', 'gnuplot' and 'terminal png'\n"; 182 | print " to be installed.\n"; 183 | exit; 184 | } # sub usage 185 | 186 | ### Check arguments 187 | sub check_args 188 | { 189 | my %opt; 190 | getopts('m:d:t:fr:vpx', \%opt) or usage(); 191 | print "check args\n" if ($VERBOSE); 192 | 193 | if(defined($opt{'v'})) 194 | { 195 | $VERBOSE=1; # Enable verbose messaging, may slow things down 196 | print "VERBOSE enabled\n"; 197 | } 198 | 199 | # Hidden DEBUG switch 200 | if(defined($opt{'x'})) 201 | { 202 | $VERBOSE=1; # Enable verbose messaging, may slow things down 203 | $DEBUG=1; # Enable debug messaging, will slow things down 204 | print "VERBOSE and DEBUG enabled\n"; 205 | } 206 | 207 | if(!$opt{'m'}) { usage(); } 208 | $mode = $opt{'m'}; 209 | print "Mode: $mode\n" if ($VERBOSE); 210 | 211 | if($mode eq 'live') 212 | { 213 | # Check for invalid args 214 | if(!$opt{'d'} || !$opt{'r'}) { usage(); } 215 | 216 | $dev = $opt{'d'}; 217 | $runtime = $opt{'r'}; 218 | print "Dev: $dev Runtime: $runtime\n" if ($DEBUG); 219 | 220 | $live=1; 221 | if ($dev =~ /\/dev\/(\S+)/) { $dev_str = $1; } 222 | $dev_str =~ s/\//_/g; 223 | print "str: $dev_str\n" if($DEBUG); 224 | die("ERROR: dev_str=$dev_str invalid") if ($dev_str eq ""); 225 | 226 | # Check if $dev is not a block special file 227 | die("ERROR: Cannot find $dev or $dev is not a block special file") if (! -b $dev); 228 | } 229 | elsif($mode eq 'post') 230 | { 231 | # Check Args 232 | if (!$opt{'t'}) { usage(); } 233 | $tar_file = $opt{'t'}; 234 | 235 | if($tar_file =~ /(\S+).tar/) 236 | { 237 | $dev_str = $1; 238 | } 239 | die("ERROR: dev_str=$dev_str invalid") if ($dev_str eq ""); 240 | 241 | if ($opt{'p'}) 242 | { 243 | check_pdf_prereqs(); 244 | $pdf_report = 1; 245 | } 246 | $fdisk_file = "fdisk.$dev_str"; 247 | print "fdisk_file: $fdisk_file\n" if ($DEBUG); 248 | print "Option -t $tar_file\n" if($DEBUG); 249 | push(@files_to_cleanup, $fdisk_file); 250 | 251 | die("ERROR: Cannot find $tar_file or $tar_file is not a plain file") if (! -f $tar_file); 252 | } 253 | elsif($mode eq 'trace') 254 | { 255 | # Check for invalid args 256 | if(!$opt{'d'} || !$opt{'r'}) { usage(); } 257 | 258 | if($opt{'f'}) { $trace_files = 1 ;} 259 | print "Trace files = $trace_files\n" if($DEBUG); 260 | 261 | $dev = $opt{'d'}; 262 | $runtime = $opt{'r'}; 263 | if ($runtime < 3) 264 | { 265 | $runtime = 3; # Min Runtime 266 | } 267 | print "Runtime: $runtime\n" if ($DEBUG); 268 | 269 | if ($dev =~ /\/dev\/(\S+)/) { $dev_str = $1; } 270 | $dev_str =~ s/\//_/g; 271 | print "Option -d $dev\n" if($DEBUG); 272 | print "str: $dev_str\n" if($DEBUG); 273 | die("ERROR: dev_str=$dev_str invalid") if ($dev_str eq ""); 274 | 275 | # Check if $dev is not a block special file 276 | die("ERROR: Cannot find $dev or $dev is not a block special file") if (! -b $dev); 277 | } 278 | else 279 | { 280 | usage(); 281 | } 282 | 283 | return; 284 | } 285 | 286 | ### Check prereqs for gnuplot and latex 287 | sub check_pdf_prereqs 288 | { 289 | `which gnuplot`; 290 | if ($? != 0) { die("ERROR: gnuplot not installed. Please offload the trace file for processing."); } 291 | `which pdflatex`; 292 | if ($? != 0) { die("ERROR: pdflatex not installed. Please offload the trace file for processing."); } 293 | `echo 'set terminal png' > pngtest.txt; gnuplot pngtest.txt >/dev/null 2>&1`; 294 | if ($? != 0) { `rm -f pngtest.txt`; die("ERROR: gnuplot PNG terminal not installed. Please offload the trace file for processing."); } 295 | `rm -f pngtest.txt`; 296 | } 297 | 298 | ### Check prereqs for blktrace 299 | sub check_trace_prereqs 300 | { 301 | `which blktrace`; 302 | if ($? != 0) { die("ERROR: blktrace not installed. Please install blktrace"); } 303 | `which blkparse`; 304 | if($? != 0) { die("ERROR: blkparse not installed. Please install blkparse"); } 305 | } 306 | 307 | ### Check if debugfs is mounted 308 | sub mount_debugfs 309 | { 310 | `mount | grep debugfs >/dev/null`; 311 | if($?) 312 | { 313 | print "Need to mount debugfs\n" if ($VERBOSE); 314 | `mount -t debugfs debugfs /sys/kernel/debug`; 315 | if($? !=0 ) { die("ERROR: Failed to mount debugfs"); } 316 | print "mounted debugfs successfully\n" if ($VERBOSE); 317 | } 318 | return; 319 | } 320 | 321 | sub lba_to_bucket 322 | { 323 | my $lba = shift; 324 | return int($lba * $sector_size / $bucket_size); 325 | } 326 | 327 | sub bucket_to_lba 328 | { 329 | my $bucket = shift; 330 | return int($bucket * $bucket_size / $sector_size); 331 | } 332 | 333 | # debufs method 334 | # This method can only be used on ext2/ext3/ext4 filesystems 335 | # I don't plan on using this method right now 336 | # In testing the debugfs method, I found it to be approximately 30% slower than the ioctl method 337 | sub debugfs_method 338 | { 339 | my $file = shift; 340 | $extents = ""; 341 | $file =~ s/$mountpoint//; 342 | print "file: $file\n" if ($DEBUG); 343 | my @extent_out = `debugfs -R "dump_extents $file" $dev 2>/dev/null`; 344 | 345 | # Pattern 346 | #0/ 0 1/ 1 0 - 3701 7440384 - 7444085 3702 347 | foreach my $line (@extent_out) 348 | { 349 | print $line if ($DEBUG); 350 | if($line =~ /\s+\d+\/\s+\d+\s+\d+\/\s+\d+\s+\d+\s+-\s+\d+\s+(\d+)\s+-\s+(\d+)/) 351 | { 352 | $extents .= "$1:$2 "; 353 | print "$extents\n" if ($DEBUG); 354 | } 355 | } 356 | } 357 | 358 | # Filesystem cluter (io_block_size) to LBA 359 | sub fs_cluster_to_lba 360 | { 361 | my $fs_cluster_size = shift; 362 | my $sector_size = shift; 363 | my $io_cluster = shift; 364 | 365 | return $io_cluster * ($fs_cluster_size / $sector_size); 366 | } 367 | 368 | # ioctl method 369 | # This method "should" be usable regardless of filesystem 370 | # There is some risk that FIBMAP!=1. Need to address this later 371 | # I plan to use the ioctl method because it is 30% faster than the debugfs method 372 | sub ioctl_method 373 | { 374 | my $file = shift; 375 | my $FH; 376 | my $output = ""; 377 | my $start=-1; 378 | my $end=-1; 379 | my $prev=-1; 380 | my $cluster_id; 381 | #my $contig = "Contig"; 382 | 383 | 384 | print "#$file#\n" if ($DEBUG); 385 | #my $fs_cluster_size = `stat '$file' | grep "IO Block"| awk '{ print \$7 }'`; 386 | #my $file_blocks = `stat '$file' | grep Blocks | awk '{ print \$4 }'`; 387 | 388 | my ($dev,$ino,$mode,$nlink,$uid,$gid,$rdev,$size, $atime,$mtime,$ctime,$blksize,$blocks) = stat($file); 389 | my $fs_cluster_size = $blksize; 390 | my $file_blocks = $blocks; 391 | 392 | my $file_size = -s $file; 393 | my $io_blocks = ($file_blocks * $sector_size) / $fs_cluster_size; 394 | if (!defined($fs_cluster_size) || $fs_cluster_size == 0) { return; } 395 | my $file_clusters = ($file_blocks * $sector_size) / $fs_cluster_size; 396 | 397 | open($FH, "<", $file) or die("ERORR: Failed to open file $file"); 398 | 399 | my $log_id_a = 0; 400 | my $log_id_b = int($file_clusters); 401 | my $arg_a = pack "I", $log_id_a; 402 | my $arg_b = pack "I", $log_id_b; 403 | ioctl($FH, $fibmap, $arg_a) or die("ERROR: ioctl failed $!"); 404 | ioctl($FH, $fibmap, $arg_b) or die("ERROR: ioctl failed $!"); 405 | my $cluster_id_a = unpack "I", $arg_a; 406 | my $cluster_id_b = unpack "I", $arg_b; 407 | if($file_clusters == ($cluster_id_b - $cluster_id_a)) 408 | { 409 | close($FH); 410 | my $start_lba = fs_cluster_to_lba($fs_cluster_size, $sector_size, $cluster_id_a); 411 | my $end_lba = fs_cluster_to_lba($fs_cluster_size, $sector_size, $cluster_id_b); 412 | $output = "$start_lba:$end_lba"; 413 | return; 414 | } 415 | 416 | if ($READAHEAD) 417 | { 418 | my $log_id = 0; 419 | my $skip = 1; 420 | my $prev_log_id = -1; 421 | while ($log_id <= int($file_clusters)) 422 | { 423 | my $arg = pack "I", $log_id; 424 | ioctl($FH, $fibmap, $arg) or die("ERROR: ioctl failed $!"); 425 | if($? != 0) { next; } 426 | $cluster_id = unpack "I", $arg; 427 | print "$file : log=$log_id prev=$prev_log_id : cid=$cluster_id previd=$prev: skip=$skip\n" if ($DEBUG); 428 | 429 | 430 | if ($start == -1) 431 | { 432 | $start = fs_cluster_to_lba($fs_cluster_size, $sector_size, $cluster_id); 433 | $output .= "$start:"; 434 | } 435 | 436 | if ($cluster_id == ($prev + $skip)) 437 | { 438 | print "Contig!\n" if ($DEBUG); 439 | $prev_log_id = $log_id; 440 | $prev = $cluster_id; 441 | 442 | # Keep doubling readahead value (aka skip) 443 | $skip = ($skip == (1<<16)) ? 32 : $skip << 1; 444 | 445 | $log_id += $skip; 446 | next; 447 | } 448 | elsif(($skip != 1) && ($prev != -1)) 449 | { 450 | print "Not Contig, skip miss, skip=$skip\n" if ($DEBUG); 451 | $log_id = $prev_log_id; 452 | $skip = 1; 453 | } 454 | elsif($prev != -1) 455 | { 456 | my $prev_lba = fs_cluster_to_lba($fs_cluster_size, $sector_size, $prev); 457 | my $curr_lba = fs_cluster_to_lba($fs_cluster_size, $sector_size, $cluster_id); 458 | 459 | if($prev_lba != 0) 460 | { 461 | $output .= "$prev_lba "; 462 | } 463 | if($cluster_id != 0) 464 | { 465 | $output .= "$curr_lba:"; 466 | } 467 | print "Not Contig, skip=$skip\n" if ($DEBUG); 468 | #$contig = "Not Contig"; 469 | } 470 | 471 | $prev = $cluster_id; 472 | $log_id += $skip; 473 | } 474 | 475 | $end = fs_cluster_to_lba($fs_cluster_size, $sector_size, $cluster_id); 476 | if ($end != 0) 477 | { 478 | $output .= "$end"; 479 | } 480 | } 481 | else 482 | { 483 | for (0 .. int($file_clusters)) 484 | { 485 | my $log_id = $_; 486 | my $arg = pack "I", $log_id; 487 | ioctl($FH, $fibmap, $arg) or die("ERROR: ioctl failed $!"); 488 | if($? != 0) { next; } 489 | $cluster_id = unpack "I", $arg; 490 | print "$file : $log_id : $cluster_id\n" if ($DEBUG); 491 | if ($start == -1) 492 | { 493 | $start = fs_cluster_to_lba($fs_cluster_size, $sector_size, $cluster_id); 494 | $output .= "$start:"; 495 | } 496 | 497 | if ($cluster_id == ($prev + 1)) 498 | { 499 | print "Contig!\n" if ($DEBUG); 500 | } 501 | elsif($prev != -1) 502 | { 503 | my $prev_lba = fs_cluster_to_lba($fs_cluster_size, $sector_size, $prev); 504 | my $curr_lba = fs_cluster_to_lba($fs_cluster_size, $sector_size, $cluster_id); 505 | 506 | if($prev_lba != 0) 507 | { 508 | $output .= "$prev_lba "; 509 | } 510 | if($cluster_id != 0) 511 | { 512 | $output .= "$curr_lba:"; 513 | } 514 | print "Not Contig\n" if ($DEBUG); 515 | #$contig = "Not Contig"; 516 | } 517 | $prev = $cluster_id; 518 | } 519 | $end = fs_cluster_to_lba($fs_cluster_size, $sector_size, $cluster_id); 520 | if ($end != 0) 521 | { 522 | $output .= "$end"; 523 | } 524 | } 525 | 526 | close($FH); 527 | print "$output\n" if ($DEBUG); 528 | $extents = $output; 529 | 530 | } 531 | 532 | ### Print filetrace files 533 | sub printout 534 | { 535 | my $file = shift; 536 | my $OUT; 537 | print "printout: $file\n" if($DEBUG); 538 | open ($OUT,">>", "filetrace.$dev_str.$cpu_affinity.txt") || die("ERROR: Failed to open filetrace.$dev_str.$cpu_affinity.txt"); 539 | print $OUT "$file :: $extents\n"; 540 | close($OUT); 541 | } 542 | 543 | ### Find block ranges of each file 544 | sub block_ranges 545 | { 546 | my $file = shift; 547 | print "block_ranges: $file\n" if($DEBUG); 548 | 549 | if ($file =~ /^\/proc/) { print "/proc file: $file\n" if($DEBUG); return; } 550 | if ($file =~ /^\/sys/) { print "/sys file: $file\n" if ($DEBUG); return; } 551 | if ((-l $file) || (!-f $file) || (-z $file)) { print "Disqualified file: $file\n" if ($DEBUG); return; } 552 | 553 | 554 | if ($mounttype eq "ext4") 555 | { 556 | #debugfs_method($file); 557 | ioctl_method($file); 558 | } 559 | elsif ($mounttype eq "ext3") 560 | { 561 | ioctl_method($file); 562 | } 563 | else 564 | { 565 | die("ERROR: $mounttype is not supported yet"); 566 | } 567 | 568 | printout($file); 569 | } 570 | 571 | ### Find all files on device 572 | sub find_all_files 573 | { 574 | my @FILES; 575 | `rm -f filetrace.*`; 576 | my $cpu_count = `cat /proc/cpuinfo | grep processor | wc -l`; 577 | 578 | $mountpoint = `mount | grep $dev | awk '{ print \$3 }'`; 579 | if ($? != 0 ) 580 | { 581 | print "$dev not mounted\n"; 582 | return; 583 | } 584 | chomp($mountpoint); 585 | print "\nmountpoint: $mountpoint\n" if ($VERBOSE); 586 | $mounttype = `mount | grep $dev | awk '{ print \$5 }'`; 587 | chomp($mounttype); 588 | print "mounttype: $mounttype\n" if ($VERBOSE); 589 | 590 | if($FASTFILE) 591 | { 592 | my $FH; 593 | `find $mountpoint -xdev -name "*" > ioprof_files.$dev_str.txt`; 594 | open($FH, "<", "ioprof_files.$dev_str.txt") or die("ERROR: Failed to open file ioprof_files.$dev_str.txt"); 595 | while (<$FH>) 596 | { 597 | push(@FILES, $_); 598 | } 599 | close($FH); 600 | } 601 | else 602 | { 603 | @FILES = `find $mountpoint -mount -name "*"`; 604 | } 605 | 606 | my $file_count = scalar(@FILES); 607 | print "filecount=$file_count\n" if($VERBOSE); 608 | print @FILES if($DEBUG); 609 | 610 | my $set_count = int($file_count / $cpu_count) + 1; 611 | print "set_count=$set_count\n" if ($DEBUG); 612 | 613 | if($SINGLE_THREADED) 614 | { 615 | my $k = 0; 616 | for (0 .. $file_count) 617 | { 618 | if ($k > 100) 619 | { 620 | my $progress = $_ / $file_count * 100; 621 | printf "\r%05.2f%% COMPLETE", $progress; 622 | $k =0; 623 | } 624 | $k++; 625 | my $file = $FILES[$_]; 626 | if (defined $file) 627 | { 628 | chomp($file); 629 | print "$file\n" if($DEBUG); 630 | 631 | block_ranges($file); 632 | } 633 | } 634 | } 635 | else 636 | { 637 | 638 | for(0 .. ($cpu_count-1)) 639 | { 640 | $cpu_affinity = $_ % $cpu_count; 641 | print "$cpu_affinity\n" if ($DEBUG); 642 | my $pid = fork(); 643 | if($pid < 0) 644 | { 645 | # Failed to fork 646 | die("ERROR: Failed to fork process\n"); 647 | } 648 | elsif($pid == 0) 649 | { 650 | # Child process 651 | my $start = 0 + ($set_count * $cpu_affinity); 652 | if($start > $file_count) {exit; } 653 | my $end = ($set_count * ($cpu_affinity + 1)) - 1; 654 | $end = ($end >= $file_count) ? $file_count : $end; 655 | print "$cpu_affinity: start=$start end=$end\n" if ($VERBOSE); 656 | my $range = $end - $start; 657 | 658 | #if ($range == 0) { exit; } 659 | my $k = 0; 660 | 661 | for ($start .. $end) 662 | { 663 | if ($range >0) 664 | { 665 | my $progress = ($_ - $start) / $range * 100; 666 | if ($k > 100) 667 | { 668 | printf "\r%05.2f%% COMPLETE", $progress; 669 | $k =0; 670 | } 671 | $k++; 672 | } 673 | my $file = $FILES[$_]; 674 | if (defined $file) 675 | { 676 | chomp($file); 677 | print "$cpu_affinity: $file\n" if($DEBUG); 678 | 679 | block_ranges($file); 680 | } 681 | } 682 | exit; 683 | } 684 | else 685 | { 686 | # Parent process 687 | print "pid=$pid CPU=$cpu_affinity\n" if ($DEBUG); 688 | push(@pidlist, $pid); 689 | my $hexcpu = sprintf("0x%08x", (1 << $cpu_affinity)); 690 | print "hexcpu=$hexcpu\n" if ($DEBUG); 691 | `taskset -p $hexcpu $pid`; 692 | print "running $cpu_affinity\n" if ($DEBUG); 693 | } 694 | } 695 | 696 | print "\n\nWaiting on threads to finish\n\n"; 697 | foreach my $pid (@pidlist) 698 | { 699 | do 700 | { 701 | print("\rWaiting on $pid "); 702 | } while (!waitpid($pid, 0)); 703 | } 704 | print "\nDONE!\n"; 705 | print "Compressing files\n"; 706 | } 707 | `gzip --fast filetrace.*`; 708 | } 709 | 710 | ### Translate a bucket ID to a list of files 711 | sub bucket_to_file_list 712 | { 713 | my $bucket_id = shift; 714 | my @list; 715 | if(defined($bucket_to_files[$bucket_id])) 716 | { 717 | @list = split(" ", $bucket_to_files[$bucket_id]); 718 | 719 | } 720 | if (scalar(@list) == 1 && $list[0] eq ' ') 721 | { 722 | undef(@list); 723 | } 724 | 725 | return @list; 726 | } 727 | 728 | 729 | ### Translate a file to a list of buckets 730 | sub file_to_buckets 731 | { 732 | my $k=0; 733 | my $size = scalar(keys %files_to_lbas); 734 | 735 | foreach my $file (keys %files_to_lbas) 736 | { 737 | $k++; 738 | if($k % 100 == 0) { printf( "\rfile_to_buckets: %d %% (%d of %d)", ($k*100 / $size), $k, $size); } 739 | 740 | $file_hit_count{$file}=0; # Initialize file hit count 741 | 742 | foreach my $range (split(" ", $files_to_lbas{$file})) 743 | { 744 | my ($start, $finish) = split(':', $range); 745 | print "$file start=$start, finish=$finish\n" if ($DEBUG); 746 | next if ($start eq '' || $finish eq ''); 747 | my $start_bucket = lba_to_bucket($start); 748 | my $finish_bucket = lba_to_bucket($finish); 749 | 750 | print "$file s_lba=$start f_lba=$finish s_buc=$start_bucket f_buc=$finish_bucket\n" if ($DEBUG); 751 | 752 | for(my $i=$start_bucket; $i<=$finish_bucket; $i++) 753 | { 754 | if (defined($bucket_to_files[$i])) 755 | { 756 | # Wrap $file in \Q \E to quote meta chars, trust me 757 | if(!$bucket_to_files[$i] =~ /\Q$file\E/) 758 | { 759 | $bucket_to_files[$i] .= "$file "; 760 | } 761 | } 762 | else 763 | { 764 | $bucket_to_files[$i] .= "$file "; 765 | } 766 | } 767 | } 768 | } 769 | 770 | 771 | print "\rDone correlating files to buckets. Now time to count bucket hits.\n"; 772 | 773 | # Print all files 774 | if($trace_files && $DEBUG) 775 | { 776 | print "--------------------------------------\n"; 777 | print "Buckets to file list:\n"; 778 | for(my $i=0; $i<$num_buckets; $i++) 779 | { 780 | my @list = bucket_to_file_list($i); 781 | my $length = @list; 782 | if ($length > 0) 783 | { 784 | print "$i($length): "; 785 | print @list; 786 | print "\n"; 787 | } 788 | } 789 | print "--------------------------------------\n"; 790 | } 791 | return; 792 | } 793 | 794 | ## Add up I/O hits to each file touched by a bucket 795 | sub add_file_hits 796 | { 797 | my $bucket_id = shift; 798 | my $io_count = shift; 799 | my @list = bucket_to_file_list($bucket_id); 800 | if(scalar(@list) == 0 && $io_count != 0) { print "No file hit. bucket=$bucket_id, io_cnt=$io_count\n" if ($DEBUG);} 801 | foreach my $file (@list) 802 | { 803 | $file_hit_count{$file} += $io_count; 804 | } 805 | } 806 | 807 | ### Get logrithmic theta for Zipfian distribution 808 | sub theta_log 809 | { 810 | my $base = shift; 811 | my $value = shift; 812 | print "base=$base, value=$value\n" if($DEBUG); 813 | return log($value)/log($base); 814 | } 815 | 816 | ### Print results 817 | sub print_results 818 | { 819 | my $num = defined($1) ? $1 : 0; 820 | my $sum; 821 | my $k=0; 822 | my ($DATA, $DATA2, $HEADER, $STATS); 823 | my $buffer = ""; 824 | my $i; 825 | my ($read_sum, $write_sum); 826 | my %counts; 827 | $read_sum = $write_sum = 0; 828 | my ($row, $column); 829 | $row=$column=0; 830 | my $bw_total=0; 831 | my @histogram_iops; 832 | my @histogram_bw; 833 | 834 | print "num_buckets=$num_buckets bucket_size=$bucket_size\n" if($VERBOSE); 835 | print `date` if ($DEBUG); 836 | 837 | if($pdf_report) 838 | { 839 | open($DATA, ">data.$dev_str.$num") || die ("ERROR: Could not open: data"); 840 | } 841 | 842 | my $x=0; 843 | my $threshold = $num_buckets / 100; 844 | for($i=0; $i<$num_buckets; $i++) 845 | { 846 | $x++; 847 | if ($x > $threshold) 848 | { 849 | printf( "\rBucket Percent: %d %%", ($i / $num_buckets * 100)); 850 | $x=0; 851 | } 852 | if ($i!=0 && ($i % $xwidth) == 0) 853 | { 854 | if($pdf_report) 855 | { 856 | print $DATA "$buffer\n"; 857 | } 858 | $buffer=""; 859 | $row++; 860 | $column=0; 861 | } 862 | my $r = defined($reads[$i]) ? $reads[$i] : 0; 863 | my $w = defined($writes[$i]) ? $writes[$i] : 0; 864 | my $bucket_total = $r + $w; 865 | $bw_total += $bucket_total * $bucket_size; 866 | print "$i: bt=$bucket_total\n" if ($DEBUG); 867 | add_file_hits($i, $bucket_total) if ($trace_files); 868 | $counts{$bucket_total}++; 869 | $read_sum += $r; 870 | $write_sum += $w; 871 | 872 | $buffer .= sprintf("%d ", $bucket_total); 873 | #push(@map,$bucket_total); 874 | 875 | #$reads[$i]=0; 876 | #$writes[$i]=0; 877 | #undef($reads[$i]); 878 | #undef($writes[$i]); 879 | 880 | $column++; 881 | } 882 | print "\r \n"; 883 | while (($i % $xwidth) != 0) { $i++; $buffer .= "0 "; } 884 | if($pdf_report) 885 | { 886 | print $DATA "$buffer\n"; 887 | close($DATA); 888 | push(@files_to_cleanup, "data.$dev_str.$num"); 889 | } 890 | 891 | print "num_buckets=$num_buckets pfgp iot=$io_total bucket_hits_total=$bucket_hits_total r_sum=$read_sum w_sum=$write_sum yheight=$yheight\n" if($VERBOSE); 892 | 893 | print `date` if ($DEBUG); 894 | my $t=0; 895 | my $j=0; 896 | my $section_count=0; 897 | my $b_count=0; 898 | my $GB_tot=0; 899 | my $bw_tot=0; 900 | my $bw_count=0; 901 | my $io_sum=0; 902 | my $tot=0; 903 | if($pdf_report) 904 | { 905 | open($DATA, ">histogram_iops.$dev_str.$num") || die("ERROR: Could not open: histogram_iops.$dev_str.$num"); 906 | open($DATA2, ">histogram_bw.$dev_str.$num") || die("ERROR: Could not open: histogram_bw.$dev_str.$num"); 907 | } 908 | 909 | # %counts is a hash 910 | # each key "bucket_total" represents a particular I/O count for a bucket 911 | # each value represents the number of buckets that had this I/O count 912 | # This allows me to quickly tally up a histogram and is pretty 913 | # space efficient since most buckets tend to have zero I/O that 914 | # key tends to have the largest number of buckets 915 | # 916 | # Iterate through each key in decending order 917 | my $max_set=0; 918 | my $max=0; 919 | my $theta_count=1; 920 | my $theta_total=0; 921 | my $min=1; 922 | my $max_theta=0; 923 | my $min_theta=999; 924 | for my $total ( sort {$b<=>$a} keys %counts) 925 | { 926 | if ($total) 927 | { 928 | print "$total: $counts{$total}\n" if ($DEBUG); 929 | $tot += $total * $counts{$total}; 930 | print "tot=$tot\n" if($DEBUG); 931 | 932 | if (!$max_set) 933 | { 934 | $max_set = 1; 935 | $max = $total; 936 | } 937 | else 938 | { 939 | $theta_count++; 940 | $min = $total; 941 | my $cur_theta = theta_log($theta_count, $max) - theta_log($theta_count, $total); 942 | $max_theta = ($cur_theta > $max_theta) ? $cur_theta : $max_theta; 943 | $min_theta = ($cur_theta < $min_theta) ? $cur_theta : $min_theta; 944 | print "cur_theta: $cur_theta\n" if ($DEBUG); 945 | $theta_total += $cur_theta; 946 | } 947 | 948 | for(my $i=0; $i< $counts{$total}; $i++) 949 | { 950 | $section_count += $total; 951 | $b_count++; 952 | $bw_count += $total * $bucket_size; 953 | if(($b_count*$bucket_size/$GiB) > ($percent * $total_capacity_GiB)) 954 | { 955 | print "b_count: $b_count\n" if ($DEBUG); 956 | $bw_tot += $bw_count; 957 | $GB_tot += ($b_count * $bucket_size); 958 | $io_sum += $section_count; 959 | 960 | my $GB = sprintf("%.1f", $GB_tot / $GiB); 961 | my $io_perc = sprintf("%.1f", ($section_count / $bucket_hits_total)*100); 962 | my $bw_perc = sprintf("%.1f", ($bw_count / $bw_total)*100); 963 | my $io_sum_perc = sprintf("%.1f", ($io_sum/($bucket_hits_total))*100); 964 | 965 | if($pdf_report) 966 | { 967 | print $DATA "\"$GB GB\" $io_perc $section_count $io_sum_perc $io_sum\n"; 968 | print $DATA2 "\"$GB GB\" $bw_perc $bw_count\n"; 969 | } 970 | push(@histogram_iops, "$GB GB $io_perc% ($io_sum_perc% cumulative)"); 971 | push(@histogram_bw, "$GB GB $bw_perc%"); 972 | 973 | $b_count=0; 974 | $section_count=0; 975 | $bw_count=0; 976 | } 977 | } 978 | } 979 | } 980 | if ($b_count) 981 | { 982 | print "b_count: $b_count\n" if ($DEBUG); 983 | $bw_tot += $bw_count; 984 | $GB_tot += ($b_count * $bucket_size); 985 | $io_sum += $section_count; 986 | 987 | my $GB = sprintf("%.1f", $GB_tot / $GiB); 988 | my $io_perc = sprintf("%.1f", ($section_count / $bucket_hits_total)*100); 989 | my $bw_perc = sprintf("%.1f", ($bw_count / $bw_total)*100); 990 | my $io_sum_perc = sprintf("%.1f", ($io_sum/($bucket_hits_total))*100); 991 | 992 | if($pdf_report) 993 | { 994 | print $DATA "\"$GB GB\" $io_perc $section_count $io_sum_perc $io_sum\n"; 995 | print $DATA2 "\"$GB GB\" $bw_perc $bw_tot\n"; 996 | } 997 | push(@histogram_iops, "$GB GB $io_perc% ($io_sum_perc% cumulative)"); 998 | push(@histogram_bw, "$GB GB $bw_perc%"); 999 | 1000 | $b_count=0; 1001 | $section_count=0; 1002 | $bw_count=0; 1003 | } 1004 | if($pdf_report) 1005 | { 1006 | print $DATA "\" \" 0 0 0 0\n"; 1007 | close($DATA); 1008 | } 1009 | print "t=$t\n" if($DEBUG); 1010 | print `date` if($DEBUG); 1011 | 1012 | print "--------------------------------------------\n"; 1013 | print "Histogram IOPS:\n"; 1014 | foreach my $entry (@histogram_iops) 1015 | { 1016 | print "$entry\n"; 1017 | } 1018 | print "--------------------------------------------\n"; 1019 | 1020 | #print "Histogram BW:\n"; 1021 | #print "--------------------------------------------\n"; 1022 | #foreach my $entry (@histogram_bw) 1023 | #{ 1024 | #print "$entry\n"; 1025 | #} 1026 | #print "--------------------------------------------\n"; 1027 | 1028 | if ($theta_count) 1029 | { 1030 | #my $cur_theta = theta_log($theta_count, $max) - theta_log($theta_count, $min); 1031 | my $avg_theta = $theta_total / $theta_count; 1032 | my $med_theta = (($max_theta - $min_theta) / 2) + $min_theta; 1033 | my $approx_theta = ($avg_theta + $med_theta) / 2; 1034 | print "avg_t=$avg_theta med_t=$med_theta approx_t=$approx_theta min_t=$min_theta max_t=$max_theta\n" if ($VERBOSE); 1035 | $analysis_histogram_iops = sprintf("Approximate Zipfian Theta Range: %0.4f-%0.4f (est. %0.4f).\n", $min_theta, $max_theta, $approx_theta); 1036 | print "$analysis_histogram_iops"; 1037 | } 1038 | 1039 | # Print top hit files by sorting hash in numerical order by value 1040 | print "Trace_files: $trace_files\n" if ($DEBUG); 1041 | if($trace_files) 1042 | { 1043 | my $top_count=0; 1044 | print "--------------------------------------------\n"; 1045 | print "Top files by IOPS:\n"; 1046 | print "Total Hits: $bucket_hits_total\n"; 1047 | foreach my $filename (sort {$file_hit_count{$b} <=> $file_hit_count{$a}} keys %file_hit_count) 1048 | { 1049 | my $hits = $file_hit_count{$filename}; 1050 | #if ($hits > ($bucket_hits_total/ 100)) 1051 | if ($hits > 0) 1052 | { 1053 | my $hit_rate = $hits / $bucket_hits_total * 100; 1054 | printf "%0.2f%% (%d) %s\n", $hit_rate, $hits, $filename; 1055 | $top_files .= sprintf "%0.2f%%: (%d) %s\n", $hit_rate, $hits, $filename; 1056 | } 1057 | $top_count++; 1058 | if($top_count > $top_count_limit) { last; } 1059 | } 1060 | print "--------------------------------------------\n"; 1061 | } 1062 | if($pdf_report) 1063 | { 1064 | push(@files_to_cleanup, "histogram_iops.$dev_str.$num", "histogram_bw.$dev_str.$num"); 1065 | } 1066 | 1067 | } # print_results 1068 | 1069 | ### Print heatmap header for PDF 1070 | sub print_header_heatmap 1071 | { 1072 | my ($DATA, $HEADER, $STATS); 1073 | my $num = defined($1) ? $1 : 0; 1074 | 1075 | open($HEADER, ">header_heatmap.$dev_str.$num"); 1076 | print $HEADER "set terminal pngcairo enhanced font \"arial,10\" fontscale 1.0 size 800, 600\n"; 1077 | #print $HEADER "set terminal x11 size 800, 600\n"; 1078 | print $HEADER "set output 'heatmap_${dev_str}_${num}.png'\n"; 1079 | print $HEADER "unset key\n"; 1080 | print $HEADER "set view map\n"; 1081 | print $HEADER "set format x ''\n"; 1082 | print $HEADER "set format y ''\n"; 1083 | #print $HEADER "set xtics border in scale 0,0 mirror norotate offset character 0, 0, 0 autojustify\n"; 1084 | #print $HEADER "set ytics border in scale 0,0 mirror norotate offset character 0, 0, 0 autojustify\n"; 1085 | print $HEADER "set ztics border in scale 0,0 nomirror norotate offset character 0, 0, 0 autojustify\n"; 1086 | #print $HEADER "set nocbtics\n"; 1087 | print $HEADER "set rtics axis in scale 0,0 nomirror norotate offset character 0, 0, 0 autojustify\n"; 1088 | print $HEADER "set title \"Heat Map of IOPS over $dev\"\n"; 1089 | print $HEADER "set xrange [ 0 : $xwidth] noreverse nowriteback\n"; 1090 | print $HEADER "set yrange [ 0 : $yheight ] noreverse nowriteback\n"; 1091 | print $HEADER "set cblabel \"Total I/O's during test\"\n"; 1092 | print $HEADER "set palette defined (1 'white', 10 'blue', 20 'green', 60 'yellow', 70 'orange', 100 'red')\n"; 1093 | print $HEADER "plot 'data.$dev_str.$num' matrix with image\n"; 1094 | #print $HEADER "pause 1\n"; 1095 | close($HEADER); 1096 | push(@files_to_cleanup, "header_heatmap.$dev_str.$num", "heatmap_${dev_str}_${num}.png"); 1097 | 1098 | `gnuplot header_heatmap.$dev_str.$num`; 1099 | if($? != 0 ) { die("ERROR: Failed to run gnuplot header_heatmap.$dev_str.$num Error: $!"); } 1100 | 1101 | print "ph iot=$io_total\n" if ($DEBUG); 1102 | } # print_header 1103 | 1104 | ### Print histogram header for PDF 1105 | sub print_header_histogram_iops 1106 | { 1107 | my $HEADER; 1108 | my $num = defined($1) ? $1 : 0; 1109 | print "HEADER TIME!\n" if ($DEBUG); 1110 | 1111 | open($HEADER, ">header_histogram_iops.$dev_str.$num"); 1112 | print $HEADER "set terminal pngcairo enhanced font 'arial,10' fontscale 1.0 size 800, 600\n"; 1113 | print $HEADER "set output \'histogram_iops_${dev_str}_${num}.png\'\n"; 1114 | print $HEADER "set auto x\n"; 1115 | print $HEADER "set xlabel 'GB of capacity accessed by I/O'\n"; 1116 | print $HEADER "set ylabel '% of Total I/O'\n"; 1117 | print $HEADER "set title 'I/O distribution throughout total disk space'\n"; 1118 | print $HEADER "set style histogram clustered\n"; 1119 | print $HEADER "set boxwidth 0.95 relative\n"; 1120 | print $HEADER "set style fill transparent solid 0.5 noborder\n"; 1121 | #print $HEADER "set style fill transparent solid 0.5 noborder\n"; 1122 | print $HEADER "set xtics rotate by -90\n"; 1123 | #print $HEADER "set auto y\n"; 1124 | #print $HEADER "plot \'histogram_iops.$dev_str.$num\' using 2:xticlabels(1) with boxes lc rgb'blue90'\n"; 1125 | #print $HEADER "set y2range [0:100]\n"; 1126 | #print $HEADER "set yrange [0:9.0]\n"; 1127 | #print $HEADER "set y2tics 10 nomirror tc lt 2\n"; 1128 | #print $HEADER "set ytics 5 nomirror tc lt 1\n"; 1129 | #print $HEADER "set y2label 'stuff' tc lt 2\n"; 1130 | #print $HEADER "plot \'histogram_iops.$dev_str.$num\' using 2:xticlabels(1) with boxes lc rgb'blue90' linetype 1, '' u 0:4 with lines linetype 2\n"; 1131 | print $HEADER "plot \'histogram_iops.$dev_str.$num\' using 2:xticlabels(1) with boxes lc rgb'blue90'\n"; 1132 | close($HEADER); 1133 | push(@files_to_cleanup, "header_histogram_iops.$dev_str.$num", "histogram_iops_${dev_str}_${num}.png"); 1134 | 1135 | `gnuplot header_histogram_iops.$dev_str.$num`; 1136 | if($? != 0 ) { die("ERROR: Failed to run gnuplot header_histogram_iops.$dev_str.$num Error: $!"); } 1137 | 1138 | } # print_header_histogram_iops 1139 | 1140 | ### Print stats header for PDF 1141 | sub print_header_stats_iops 1142 | { 1143 | my $HEADER; 1144 | my $num = defined($1) ? $1 : 0; 1145 | print "HEADER TIME!\n" if ($DEBUG); 1146 | 1147 | open($HEADER, ">header_stats_iops.$dev_str.$num"); 1148 | print $HEADER "set terminal pngcairo enhanced font 'arial,10' fontscale 1.0 size 800, 600\n"; 1149 | print $HEADER "set output \'stats_iops_${dev_str}_${num}.png\'\n"; 1150 | print $HEADER "set auto x\n"; 1151 | print $HEADER "set yrange [0:100]\n"; 1152 | print $HEADER "set ylabel '% of Total I/O'\n"; 1153 | print $HEADER "set xlabel 'I/O Size'\n"; 1154 | print $HEADER "set title 'I/O Distribution by I/O Size'\n"; 1155 | print $HEADER "set style histogram clustered\n"; 1156 | print $HEADER "set boxwidth 0.95 relative\n"; 1157 | print $HEADER "set style fill transparent solid 0.5 noborder\n"; 1158 | print $HEADER "set xtics rotate by -90\n"; 1159 | print $HEADER "set grid\n"; 1160 | print $HEADER "set ytics 5\n"; 1161 | #print $HEADER "plot \'histogram_iops.$dev_str.$num\' using 2:xticlabels(1) with boxes lc rgb'blue90', "" notitle\n"; 1162 | print $HEADER "plot \'stats_iops.$dev_str.$num\' using 2:xticlabels(1) with boxes lc rgb'blue90' notitle\n"; 1163 | #print $HEADER "plot \'stats_iops.$dev_str.$num\' using 2:xticlabels(1) with labels lc rgb'blue90'\n"; 1164 | close($HEADER); 1165 | push(@files_to_cleanup, "header_stats_iops.$dev_str.$num", "stats_iops_${dev_str}_${num}.png"); 1166 | 1167 | `gnuplot header_stats_iops.$dev_str.$num`; 1168 | if($? != 0 ) { die("ERROR: Failed to run gnuplot header_histogram_iops.$dev_str.$num Error: $!"); } 1169 | 1170 | } # print_header_histogram_iops 1171 | 1172 | ### Create PDF report 1173 | sub create_report 1174 | { 1175 | my $LATEX; 1176 | my $num = defined($1) ? $1 : 0; 1177 | 1178 | print "REPORT TIME!\n" if ($DEBUG); 1179 | 1180 | open($LATEX, ">report.$dev_str.tex"); 1181 | print $LATEX "\\documentclass{article}\n"; 1182 | print $LATEX "\\usepackage{graphicx}\n"; 1183 | print $LATEX "\n"; 1184 | print $LATEX "\\begin{document}\n"; 1185 | print $LATEX "\\section{IO Profiling Report}\n"; 1186 | print $LATEX "\\subsection{Summary}\n"; 1187 | print $LATEX "$analysis_stats_iops\n"; 1188 | print $LATEX "\n\nPlease make sure to review the important notes in the section \"Caveat Emptor\" at the bottom of this document.\n"; 1189 | print $LATEX "\\subsection{Top Files}\n"; 1190 | if ($trace_files) 1191 | { 1192 | print $LATEX "The following files were the top $top_count_limit most accessed files by IOPS\n"; 1193 | print $LATEX "\\begin{verbatim}\n"; 1194 | print $LATEX "$top_files\n"; 1195 | print $LATEX "\\end{verbatim}\n"; 1196 | } 1197 | else 1198 | { 1199 | print $LATEX "Tracing was done without 'File tracing' enabled. To enable file tracing, please use the -f flag. Please note that it may take 3-5 minutes to gather information about file placement on large filesystems.\n"; 1200 | } 1201 | print $LATEX "\\subsection{IOPS Histogram}\n"; 1202 | print $LATEX "$analysis_histogram_iops\n"; 1203 | print $LATEX "\\includegraphics[width=\\textwidth]{histogram_iops_${dev_str}_${num}}\n"; 1204 | print $LATEX "\\subsection{IOPS Heatmap}\n"; 1205 | print $LATEX "$analysis_heatmap\n"; 1206 | print $LATEX "\\includegraphics[width=\\textwidth]{heatmap_${dev_str}_${num}}\n"; 1207 | print $LATEX "\\subsection{IOPS Statistics}\n"; 1208 | print $LATEX "\\includegraphics[width=\\textwidth]{stats_iops_${dev_str}_${num}}\n"; 1209 | print $LATEX "\\subsection{Bandwidth Statistics}\n"; 1210 | print $LATEX "$analysis_stats_bw\n"; 1211 | #print $LATEX "\\includegraphics[width=\\textwidth]{heatmaps_input}\n"; 1212 | #print $LATEX "\\includegraphics[width=\\textwidth]{heatmaps_output}\n"; 1213 | #print $LATEX "\\includegraphics[width=\\textwidth]{heatmaps_saswork}\n"; 1214 | print $LATEX "\\subsection{Caveat Emptor}\n"; 1215 | print $LATEX "This tool is intended to provide the user with additional insight into their own workload. This tool has the following limitations:\n"; 1216 | print $LATEX "\\begin{itemize}\n"; 1217 | print $LATEX "\\item In order to limit the trace impact on performance, blktrace is run as a collection of 3 second traces. Thus, there may be I/O's missed between runs.\n"; 1218 | print $LATEX "\\item Larger $bucket_size byte buckets are used to count logical sector hits instead of $sector_size bytes. This improves post processing times, but produces less granular results.\n"; 1219 | print $LATEX "\\item Translating files into logical block ranges may rely on FIBMAP ioctl() calls.\n"; 1220 | print $LATEX "\\item Translating file logical block ranges into $bucket_size byte buckets may result in some bucket overlap. Cross-correlating this with bucket I/O hits may result in some files being identified as being top hits without any file access. Please keep this in mind when making important migration or caching decisions.\n"; 1221 | print $LATEX "\\end{itemize}\n"; 1222 | print $LATEX "\\end{document}\n"; 1223 | close($LATEX); 1224 | 1225 | `pdflatex -interaction batchmode report.$dev_str.tex`; 1226 | if($? != 0 ) { die("ERROR: Failed to run report Error: $!"); } 1227 | 1228 | `rm -f report.$dev_str.tex`; 1229 | `rm -f report.$dev_str.aux`; 1230 | `rm -f report.$dev_str.log`; 1231 | 1232 | print "Your I/O profiling report is now available: report.$dev_str.pdf\n"; 1233 | 1234 | } # create_report 1235 | 1236 | ### Print I/O statistics 1237 | sub print_stats 1238 | { 1239 | my ($STATS_IOPS, $STATS_BW); 1240 | my $total_r_bw = 0; 1241 | my $total_w_bw = 0; 1242 | my $rtot=0; 1243 | my $wtot=0; 1244 | my $tot=0; 1245 | my $num = defined($1) ? $1 : 0; 1246 | my $stats_iops = ""; 1247 | my $stats_bw = ""; 1248 | 1249 | if($pdf_report) 1250 | { 1251 | open($STATS_IOPS, ">stats_iops.$dev_str.$num"); 1252 | open($STATS_BW, ">stats_bw.$dev_str.$num"); 1253 | } 1254 | 1255 | foreach my $j (sort {$a<=>$b} keys %r_totals) 1256 | { 1257 | $rtot += $r_totals{$j}; 1258 | #my $stuff = $r_totals{$j}; 1259 | my $r_perc = ($io_total) ? $r_totals{$j} / $io_total * 100 : 0; 1260 | print "r_perc=$r_perc\n" if ($DEBUG); 1261 | if ($r_perc > 0.5) 1262 | { 1263 | if($pdf_report) 1264 | { 1265 | printf($STATS_IOPS "\"%dK READ\" %d %d\n", ($j * $sector_size / $KiB), 1266 | (($io_total) ? $r_totals{$j}/$io_total*100 : 0), 1267 | (defined($r_totals{$j}) ? $r_totals{$j} : 0)); 1268 | printf($STATS_BW "\"%dK READ\" %d %d\n", ($j * $sector_size / $KiB), 1269 | ($total_blocks) ? ($j*$r_totals{$j}/$total_blocks*100):0, ($j*$KiB * $r_totals{$j}) ); 1270 | } 1271 | else 1272 | { 1273 | $stats_iops .= sprintf("\"%dK READ\" %0.2f%% (%d IO's)\n", ($j * $sector_size / $KiB), 1274 | (($io_total) ? $r_totals{$j}/$io_total*100 : 0), 1275 | (defined($r_totals{$j}) ? $r_totals{$j} : 0)); 1276 | $stats_bw .= sprintf("\"%dK READ\" %0.2f%% (%0.2f GiB)\n", ($j * $sector_size / $KiB), 1277 | ($total_blocks) ? ($j*$r_totals{$j}/$total_blocks*100):0, (($j*$sector_size) * $r_totals{$j} / $GiB) ); 1278 | 1279 | } 1280 | } 1281 | # BEN $r_totals{$j}=0; 1282 | $total_r_bw += $j * $sector_size * $r_totals{$j}/ $GiB; 1283 | } 1284 | foreach my $j (sort {$a<=>$b} keys %w_totals) 1285 | { 1286 | $wtot += $w_totals{$j}; 1287 | my $w_perc = ($io_total) ? $w_totals{$j} / $io_total * 100 : 0; 1288 | print "w_perc=$w_perc\n" if ($DEBUG); 1289 | if ($w_perc > 0.5) 1290 | { 1291 | if($pdf_report) 1292 | { 1293 | printf($STATS_IOPS "\"%dK WRITE\" %d %d\n", ($j * $sector_size / $KiB), 1294 | (($io_total) ? $w_totals{$j}/$io_total*100 : 0), 1295 | (defined($w_totals{$j}) ? $w_totals{$j} : 0)); 1296 | printf($STATS_BW "\"%dK WRITE\" %d %d\n", ($j * $sector_size / $KiB), 1297 | ($j*$w_totals{$j}/$total_blocks*100), ($j*$KiB * $w_totals{$j}) ); 1298 | } 1299 | else 1300 | { 1301 | 1302 | $stats_iops .= sprintf("\"%dK WRITE\" %0.2f%% (%d IO's)\n", ($j * $sector_size / $KiB), 1303 | (($io_total) ? $w_totals{$j}/$io_total*100 : 0), 1304 | (defined($w_totals{$j}) ? $w_totals{$j} : 0)); 1305 | $stats_bw .= sprintf("\"%dK WRITE\" %0.2f%% (%0.2f GiB)\n", ($j * $sector_size / $KiB), 1306 | ($j*$w_totals{$j}/$total_blocks*100), (($j*$sector_size) * $w_totals{$j} / $GiB) ); 1307 | } 1308 | 1309 | } 1310 | # BEN $w_totals{$j}=0; 1311 | $total_w_bw += $j * $sector_size * $w_totals{$j}/ $GiB; 1312 | } 1313 | 1314 | if($pdf_report) 1315 | { 1316 | print $STATS_IOPS "\"\" 0 0\n"; 1317 | print $STATS_BW "\"\" 0 0\n"; 1318 | close($STATS_IOPS); 1319 | close($STATS_BW); 1320 | 1321 | push(@files_to_cleanup, "stats_iops.$dev_str.$num"); 1322 | push(@files_to_cleanup, "stats_bw.$dev_str.$num"); 1323 | } 1324 | else 1325 | { 1326 | print "Stats IOPS:\n"; 1327 | print $stats_iops; 1328 | 1329 | print "Stats BW:\n"; 1330 | print $stats_bw; 1331 | } 1332 | 1333 | $tot = $rtot + $wtot; 1334 | print "rtot=$rtot wtot=$wtot tot=$tot ps iot=$io_total\n" if ($DEBUG); 1335 | my $caching_score=0; 1336 | my $ssd_score=0; 1337 | 1338 | # Analyze read percentage statistics 1339 | my $read_percent = sprintf("%0.2f", ($io_total) ? $read_total / $io_total * 100 : 0); 1340 | my $write_percent = sprintf("%0.2f", ($io_total) ? $write_total / $io_total * 100 : 0); 1341 | $analysis_stats_iops = "Your workload was approximately $read_percent\\% reads and $write_percent\\% writes. "; 1342 | if ($read_percent > 95) 1343 | { 1344 | #$analysis_stats_iops .= "Your workload was very read intensive. This indicates that write-through caching could improve your workload up to 20X.\n"; 1345 | $analysis_stats_iops .= "Your workload was very read intensive.\n"; 1346 | $caching_score += 20; 1347 | $ssd_score += 100; 1348 | } 1349 | elsif($read_percent > 70) 1350 | { 1351 | $analysis_stats_iops .= "Your workload was moderately read intensive.\n"; 1352 | $caching_score += 3; 1353 | $ssd_score += 70; 1354 | } 1355 | elsif($read_percent > 50) 1356 | { 1357 | $analysis_stats_iops .= "Your workload was evenly split between reads and writes.\n"; 1358 | $caching_score += 2; 1359 | $ssd_score += 50; 1360 | } 1361 | else 1362 | { 1363 | $analysis_stats_iops .= "Your workload was write intensive.\n"; 1364 | $caching_score += 1; 1365 | $ssd_score += 20; 1366 | } 1367 | 1368 | # Analyze IOPS distribution 1369 | my $sectors_per_4k = 4096 / $sector_size; 1370 | my $read_4k = (($io_total) ? (defined($r_totals{$sectors_per_4k}) ? $r_totals{$sectors_per_4k}: 0)/$io_total*100 : 0); 1371 | my $read_8k = (($io_total) ? (defined($r_totals{$sectors_per_4k*2}) ? $r_totals{$sectors_per_4k*2}: 0)/$io_total*100 : 0); 1372 | my $read_16k = (($io_total) ? (defined($r_totals{$sectors_per_4k*4}) ? $r_totals{$sectors_per_4k*2}: 0)/$io_total*100 : 0); 1373 | my $read_32k = (($io_total) ? (defined($r_totals{$sectors_per_4k*8}) ? $r_totals{$sectors_per_4k*2}: 0)/$io_total*100 : 0); 1374 | my $read_4k_total = (defined($r_totals{$sectors_per_4k})) ? $r_totals{$sectors_per_4k} : 0; 1375 | print "4k read: $read_4k 4k_total: $read_4k_total io_total: $io_total sectors_per_4k: $sectors_per_4k \n" if ($VERBOSE); 1376 | for my $bs ( sort {$a<=>$b} keys %r_totals) 1377 | { 1378 | print "bs: $bs count: $r_totals{$bs}\n" if($DEBUG); 1379 | } 1380 | 1381 | 1382 | if ($read_4k > 90) 1383 | { 1384 | $analysis_stats_iops .= "You workload was $read_4k\\% 4k reads.\n"; 1385 | $caching_score *= 1; 1386 | $ssd_score *= 1; 1387 | } 1388 | elsif(($read_4k + $read_8k ) > 90) 1389 | { 1390 | $analysis_stats_iops .= "You workload was greater than 90\\% 4k+8k reads.\n"; 1391 | $caching_score *= 0.7; 1392 | $ssd_score *= 0.7; 1393 | } 1394 | elsif(($read_4k + $read_8k + $read_16k ) > 90) 1395 | { 1396 | $analysis_stats_iops .= "You workload was greater than 90\\% 4k+8k+16k reads.\n"; 1397 | $caching_score *= 0.5; 1398 | $ssd_score *= 0.5; 1399 | } 1400 | elsif(($read_4k + $read_8k + $read_16k ) > 50) 1401 | { 1402 | $analysis_stats_iops .= "You workload was greater than 50\\% 4k+8k+16k reads.\n"; 1403 | $caching_score *= 0.3; 1404 | $ssd_score *= 0.3; 1405 | } 1406 | else 1407 | { 1408 | #$analysis_stats_iops .= "Your workload consists of primarily larger packet sizes.\n"; 1409 | $caching_score *= 0.1; 1410 | $ssd_score *= 0.1; 1411 | } 1412 | #$analysis_stats_iops .= "This workload is likely to see a $caching_score X benefit from caching or $ssd_score X benefit from SSDs\n"; 1413 | 1414 | 1415 | return; 1416 | 1417 | } # print_stats 1418 | 1419 | ### Combine thread-local counts into global counts 1420 | sub total_thread_counts 1421 | { 1422 | my $num = shift; 1423 | $max_bucket_hits_semaphore->down(); 1424 | print "Thread $num has a max_bucket_hits lock\n" if ($DEBUG); 1425 | if ($thread_max_bucket_hits > $max_bucket_hits) { $max_bucket_hits = $thread_max_bucket_hits; } 1426 | print "Thread $num releasing max_bucket_hits lock" if ($DEBUG); 1427 | $max_bucket_hits_semaphore->up(); 1428 | 1429 | $total_blocks_semaphore->down(); 1430 | print "Thread $num has total_blocks lock\n" if ($DEBUG); 1431 | $total_blocks += $thread_total_blocks; 1432 | print "Thread $num releasing total_blocks lock" if ($DEBUG); 1433 | $total_blocks_semaphore->up(); 1434 | 1435 | $total_semaphore->down(); 1436 | print "Thread $num has total lock\n" if($DEBUG); 1437 | $io_total += defined($thread_io_total) ? $thread_io_total : 0; 1438 | print "Thread $num releasing total lock\n" if ($DEBUG); 1439 | $total_semaphore->up(); 1440 | 1441 | $read_totals_semaphore->down(); 1442 | print "Thread $num has read_totals lock\n" if ($DEBUG); 1443 | $read_total += defined($thread_read_total) ? $thread_read_total : 0; 1444 | foreach my $j (keys %thread_r_totals) 1445 | { 1446 | $r_totals{$j} = defined($r_totals{$j}) ? $thread_r_totals{$j} + $r_totals{$j} : $thread_r_totals{$j}; 1447 | } 1448 | print "Thread $num releasing read_totals lock\n" if($DEBUG); 1449 | $read_totals_semaphore->up(); 1450 | 1451 | $write_totals_semaphore->down(); 1452 | print "Thread $num has write_totals lock\n" if ($DEBUG); 1453 | $write_total += defined($thread_write_total) ? $thread_write_total : 0; 1454 | foreach my $j (keys %thread_w_totals) 1455 | { 1456 | $w_totals{$j} = defined($w_totals{$j}) ? $thread_w_totals{$j} + $w_totals{$j} : $thread_w_totals{$j}; 1457 | } 1458 | print "Thread $num releasing write_totals lock\n" if ($DEBUG); 1459 | $write_totals_semaphore->up(); 1460 | 1461 | $total_semaphore->down(); 1462 | print "Thread $num has bucket_hits_total lock\n" if ($DEBUG); 1463 | $bucket_hits_total += defined($thread_bucket_hits_total) ? $thread_bucket_hits_total : 0; 1464 | print "Thread $num releasing bucket_hits_total lock\n" if ($DEBUG); 1465 | $total_semaphore->up(); 1466 | 1467 | $read_semaphore->down(); 1468 | print "Thread $num has read lock\n" if ($DEBUG); 1469 | for my $b( keys %thread_reads) 1470 | { 1471 | $reads[$b] = defined($reads[$b]) ? $reads[$b] + $thread_reads{$b} : $thread_reads{$b}; 1472 | } 1473 | print "Thread $num releasing read lock\n" if ($DEBUG); 1474 | $read_semaphore->up(); 1475 | 1476 | $write_semaphore->down(); 1477 | print "Thread $num has write lock\n" if ($DEBUG); 1478 | for my $b(keys %thread_writes) 1479 | { 1480 | $writes[$b] = defined($writes[$b]) ? $writes[$b] + $thread_writes{$b} : $thread_writes{$b}; 1481 | } 1482 | print "Thread $num releasing write lock\n" if ($DEBUG); 1483 | $write_semaphore->up(); 1484 | 1485 | return; 1486 | 1487 | 1488 | } 1489 | 1490 | ### Thread parse routine for blktrace output 1491 | sub thread_parse 1492 | { 1493 | my $file = shift; 1494 | my $num = shift; 1495 | my $linecount=0; 1496 | my ($a, $b, $c, $d); 1497 | my $re = qr/Q/; 1498 | chomp($file); 1499 | `gunzip $file.gz`; 1500 | 1501 | print "\nSTART: $file $num\n" if ($DEBUG); 1502 | open($CMD, "<$file"); 1503 | while(<$CMD>) 1504 | { 1505 | if($DEBUG) 1506 | { 1507 | print "\r$file: $linecount"; 1508 | $linecount++; 1509 | } 1510 | if ($_ =~ /(\S+)\s+Q\s+(\S+)\s+(\S+)$/) 1511 | { 1512 | parse_me($1, $2, $3); 1513 | } 1514 | } 1515 | close($CMD); 1516 | 1517 | total_thread_counts($num); 1518 | print "\nFINISH $file: $linecount lines\n" if ($DEBUG); 1519 | print "Remove $file\n" if ($DEBUG); 1520 | `rm -f $file`; 1521 | return; 1522 | } # thread_parse 1523 | 1524 | ### Parse blktrace output 1525 | sub parse_me 1526 | { 1527 | my $rw = shift; 1528 | my $lba = shift; 1529 | my $size = shift; 1530 | 1531 | print "rw=$rw lba=$lba size=$size\n" if($DEBUG); 1532 | 1533 | if ($rw eq "R" || $rw eq "RW") 1534 | { 1535 | $thread_total_blocks += $size; 1536 | $thread_io_total++; 1537 | $thread_read_total++; 1538 | $thread_r_totals{$size}++; 1539 | 1540 | my $bucket_hits = ($size * $sector_size) / $bucket_size; 1541 | if (($size * $sector_size) % $bucket_size != 0) { $bucket_hits++; } 1542 | 1543 | for(my $i=0; $i<$bucket_hits; $i++) 1544 | { 1545 | my $bucket = int(($lba * $sector_size)/ $bucket_size) + $i; 1546 | $thread_reads{$bucket} = defined($thread_reads{$bucket}) ? $thread_reads{$bucket} + 1 : 1; 1547 | if($thread_reads{$bucket} > $thread_max_bucket_hits) { $thread_max_bucket_hits = $thread_reads{$bucket}; } 1548 | 1549 | $thread_bucket_hits_total++; 1550 | } 1551 | } 1552 | if ($rw eq "W" || $rw eq "WS") 1553 | { 1554 | $thread_total_blocks += $size; 1555 | $thread_io_total++; 1556 | $thread_write_total++; 1557 | $thread_w_totals{$size}++; 1558 | 1559 | my $bucket_hits = ($size * 512) / $bucket_size; 1560 | if (($size * 512) % $bucket_size != 0) { $bucket_hits++; } 1561 | 1562 | for(my $i=0; $i<$bucket_hits; $i++) 1563 | { 1564 | my $bucket = int(($lba * $sector_size)/ $bucket_size) + $i; 1565 | 1566 | $thread_writes{$bucket} = defined($thread_writes{$bucket}) ? $thread_writes{$bucket} + 1 : 1; 1567 | if($thread_writes{$bucket} > $thread_max_bucket_hits) { $thread_max_bucket_hits = $thread_writes{$bucket}; } 1568 | 1569 | $thread_bucket_hits_total++; 1570 | } 1571 | } 1572 | return; 1573 | } 1574 | 1575 | ### File trace routine 1576 | sub parse_filetrace 1577 | { 1578 | my $file = shift; 1579 | my $num = shift; 1580 | my %thread_files_to_lbas; 1581 | `gunzip $file.gz`; 1582 | print "tracefile = $file $num\n" if($DEBUG); 1583 | my $FH; 1584 | open ($FH,"<", $file); 1585 | while (<$FH>) 1586 | { 1587 | print "$file: $_\n" if($DEBUG);; 1588 | if ($_ =~ /(\S+)\s+::\s+(.+)/) 1589 | { 1590 | my $object = $1; 1591 | my $ranges = $2; 1592 | $thread_files_to_lbas{$object} = $ranges; 1593 | print "$file: obj=$object ranges:$ranges\n" if($DEBUG); 1594 | } 1595 | } 1596 | close($FH); 1597 | print "Thread $num wants file_to_lba lock\n" if ($DEBUG); 1598 | $files_to_lbas_semaphore->down(); 1599 | print "Thread $num has file_to_lba lock\n" if ($DEBUG); 1600 | foreach my $object (keys %thread_files_to_lbas) 1601 | { 1602 | my $ranges = $thread_files_to_lbas{$object}; 1603 | $files_to_lbas{$object} = $ranges; 1604 | } 1605 | $files_to_lbas_semaphore->up(); 1606 | print "Thread $num freed file_to_lba lock\n" if ($DEBUG); 1607 | 1608 | return; 1609 | } 1610 | 1611 | ### Cleanup temp files 1612 | sub cleanup_files 1613 | { 1614 | print "Cleaning up temp files\n" if ($VERBOSE); 1615 | foreach my $file (@files_to_cleanup) 1616 | { 1617 | print "$file\n" if ($DEBUG); 1618 | `rm -f $file`; 1619 | } 1620 | `rm -f filetrace.*.txt`; 1621 | } 1622 | 1623 | ### Choose color for heatmap block 1624 | sub choose_color 1625 | { 1626 | my $num = shift; 1627 | if(!defined($num)|| $num == -1 || $num == 0) 1628 | { 1629 | $color_index=" "; 1630 | return $black; 1631 | } 1632 | $num = int($num); 1633 | $color_index = int($num / $vpc ); 1634 | 1635 | if($color_index > ($choices-1)) 1636 | { 1637 | print "num=$num\n" if($DEBUG); 1638 | print "HIT!\n" if ($DEBUG); 1639 | $color_index=7; 1640 | return $red; 1641 | } 1642 | #print "num=$num "; 1643 | my $color =$colors[$color_index]; 1644 | print "cap=$cap num=$num color_index=$color_index\n" if($DEBUG); 1645 | if(!defined($color)) 1646 | { 1647 | print "ch=$choices num=$num ci=$color_index vpc=$vpc cap=$cap\n"; 1648 | return $black; 1649 | } 1650 | return $color; 1651 | 1652 | } 1653 | 1654 | ### Clear screen for heatmap (UNUSED) 1655 | sub clear_screen 1656 | { 1657 | print "\033[2J"; 1658 | print "\[\033[0;0f\]\r"; 1659 | } 1660 | 1661 | ### Get block value by combining buckets into larger heatmap blocks for term 1662 | sub get_value 1663 | { 1664 | my $offset = int(shift); 1665 | my $rate = int(shift); 1666 | my $start = $offset * $rate; 1667 | my $end = $start+$rate; 1668 | my $sum = 0; 1669 | print "s=$start e=$end\n" if($DEBUG); 1670 | for($start..$end) 1671 | { 1672 | my $index = $_; 1673 | if(defined($reads[$index]) || defined($writes[$index])) 1674 | { 1675 | my $r = defined($reads[$index]) ? $reads[$index] : 0; 1676 | my $w = defined($writes[$index]) ? $writes[$index] : 0; 1677 | $sum = $sum + $r + $w; 1678 | } 1679 | #my $i=$map[$_]; 1680 | } 1681 | print "s=$sum " if($DEBUG); 1682 | #my $value = ($sum) ? $sum / $rate : -1; 1683 | return int($sum); 1684 | } 1685 | 1686 | ### Draw heatmap on color terminal 1687 | sub draw_heatmap 1688 | { 1689 | print "max_bucket_hits=$max_bucket_hits\n" if($DEBUG); 1690 | print_map() if ($DEBUG); 1691 | 1692 | #my $pigeons = scalar(@map); 1693 | my $pigeons = $num_buckets; 1694 | my $term_x = `tput cols` -$SCALEX; 1695 | my $term_y = `tput lines` -$SCALEY; 1696 | if($term_x < $min_x) 1697 | { 1698 | print "Make the terminal wider please\n"; 1699 | return; 1700 | } 1701 | elsif($term_y < $min_y) 1702 | { 1703 | print "Make the terminal taller please\n"; 1704 | return; 1705 | } 1706 | 1707 | my $holes = int($term_x * $term_y); 1708 | $rate = int($pigeons / $holes); 1709 | #$cap = int($choices * $vpc * $rate); 1710 | 1711 | my $i=0; 1712 | my $square_size = ($rate) ? (($rate * $bucket_size) / $MiB ): ($bucket_size / $MiB); 1713 | #clear_screen(); 1714 | #print "Heatmap: x:$term_x,y:$term_y \n"; 1715 | printf "This heatmap can help you 'see' hot spots. It is adjusted to terminal size, so each square = %0.2f MiB\n", $square_size; 1716 | print "The PDF report may be more precise with each pixel=1MB\n"; 1717 | print "Heatmap Key: Black (No I/O), white(Coldest),blue(Cold),cyan(Warm),green(Warmer),yellow(Very Warm),magenta(Hot),red(Hottest)\n"; 1718 | $cap=0; 1719 | 1720 | for(my $y=0; $y<$term_y; $y++) 1721 | { 1722 | #print "|"; 1723 | for(my $x=0; $x<$term_x; $x++) 1724 | { 1725 | my $index = $x + ($y * $term_x); 1726 | my $value=0; 1727 | print "y=$y, x=$x, i=$index\n" if ($DEBUG); 1728 | if($rate > 1) 1729 | { 1730 | $value = get_value($index, $rate); 1731 | } 1732 | else 1733 | { 1734 | #if(defined($map[$index])) 1735 | if(defined($reads[$index]) || defined($writes[$index])) 1736 | { 1737 | my $r = defined($reads[$index]) ? $reads[$index] : 0; 1738 | my $w = defined($writes[$index]) ? $writes[$index] : 0; 1739 | $value = $r + $w; 1740 | } 1741 | else 1742 | { 1743 | $value=-1; 1744 | } 1745 | } 1746 | $cap = ($value > $cap) ? $value : $cap; 1747 | $block_map[$index]=$value; 1748 | 1749 | print "v=$value " if ($DEBUG); 1750 | if ($x % 20 == 0) { print "\n" if($DEBUG); } 1751 | #my $color = choose_color($value); 1752 | #print "${color}$color_index"; 1753 | #print "v=$value $color_index"; 1754 | } 1755 | #print "${none}|\n"; 1756 | 1757 | } 1758 | 1759 | print "cap=$cap vpc=$vpc pigeons=$pigeons holes=$holes rate=$rate max_bucket_hits=$max_bucket_hits\n" if($VERBOSE); 1760 | $vpc = int($cap / $choices) ? int($cap / $choices) : 1; # values per choice 1761 | 1762 | print "+" . "-" x $term_x . "-+\n"; 1763 | for(my $y=0; $y<$term_y; $y++) 1764 | { 1765 | print "|"; 1766 | for(my $x=0; $x<$term_x; $x++) 1767 | { 1768 | my $index = $x + ($y * $term_x); 1769 | my $value = $block_map[$index]; 1770 | my $color = choose_color($value); 1771 | print "${color}$color_index"; 1772 | } 1773 | print "${none}|\n"; 1774 | } 1775 | print "${none}+" . "-" x $term_x . "-+\n"; 1776 | } 1777 | 1778 | sub worker 1779 | { 1780 | my ($work_q) = @_; 1781 | my $tid = threads->tid(); # Thread ID 1782 | print "tid $tid started\n" if ($DEBUG); 1783 | 1784 | do 1785 | { 1786 | # Non-blocking check for work 1787 | my $file = $work_q->dequeue_nb(); 1788 | 1789 | # If no more work, exit thread 1790 | if (!defined($file )) 1791 | { 1792 | $term_semaphore->down(); 1793 | $TERM=1; 1794 | $term_semaphore->up(); 1795 | print "Finished with work queue\n" if($DEBUG); 1796 | return; 1797 | } 1798 | 1799 | print "started file $file\n" if ($DEBUG); 1800 | 1801 | if ($file =~ /(blk.out.\S+).gz/) 1802 | { 1803 | my $new_file = $1; 1804 | thread_parse($new_file, $tid); 1805 | } 1806 | elsif ($file =~ /(filetrace.\S+.\S+.txt).gz/) 1807 | { 1808 | my $new_file = $1; 1809 | $trace_files_semaphore->down(); 1810 | $trace_files = 1; 1811 | $trace_files_semaphore->up(); 1812 | print "\nFound some filetrace files $trace_files\n" if ($DEBUG); 1813 | parse_filetrace($new_file, $tid); 1814 | } 1815 | } while (!$TERM); 1816 | 1817 | print "tid: $tid finished\n" if ($DEBUG); 1818 | return; 1819 | } 1820 | 1821 | 1822 | ############ 1823 | ### MAIN ### 1824 | ############ 1825 | 1826 | ### Check blktrace prereqs 1827 | check_trace_prereqs(); 1828 | 1829 | ### Check command line arguments 1830 | check_args(); 1831 | 1832 | ### Check if debugfs is mounted 1833 | if ($mode eq "live" || $mode eq "trace") 1834 | { 1835 | mount_debugfs(); 1836 | } 1837 | 1838 | ### Get block count and logical block size 1839 | if($mode eq 'trace') 1840 | { 1841 | 1842 | my $thread_count=0; 1843 | my $pid=-1; 1844 | #my $cpu_affinity=0; 1845 | 1846 | ### Check if sudo permissions 1847 | `sudo -v`; 1848 | if($? != 0 ) { die("ERROR: You need to have sudo permissions to collect all necessary data. Please run from a privilaged account."); } 1849 | 1850 | ### Save fdisk info 1851 | print "Running fdisk\n" if ($DEBUG); 1852 | my $fdisk_version = `fdisk -v`; 1853 | if($fdisk_version =~ /util-linux-ng/) 1854 | { 1855 | `fdisk -ul $dev > fdisk.$dev_str`; 1856 | } 1857 | else 1858 | { 1859 | `fdisk -l -u=sectors $dev > fdisk.$dev_str`; 1860 | } 1861 | 1862 | ### Cleanup previous mess 1863 | `rm -f blk.out*`; 1864 | 1865 | my $cpu_count = `cat /proc/cpuinfo | grep processor | wc -l`; 1866 | my $runcount=$runtime/$timeout; 1867 | print "\n"; 1868 | while ($runcount > 0) 1869 | { 1870 | my $percent = $thread_count * $timeout * 100 / $runtime; 1871 | my $time_left = $runcount * $timeout; 1872 | print "\r$percent % done ($time_left seconds left) "; 1873 | if ($cpu_affinity >= $cpu_count) { $cpu_affinity=0; } 1874 | if ($cpu_affinity == 0) { $cpu_affinity++; } 1875 | 1876 | `taskset -c $cpu_affinity blktrace -b $buffer_size -n $buffer_count -a queue -d $dev -o blk.out.$dev_str.$thread_count -w $timeout`; 1877 | if($? != 0) 1878 | { 1879 | print "$0 was not able to run the 'blktrace' tool required to trace all of your I/O\n\n"; 1880 | print "If you are using SLES 11 SP1, then it is likely that your default kernel is missing CONFIG_BLK_DEV_IO_TRACE\n"; 1881 | print "which is required to run blktrace. This is only available in the kernel-trace version of the kernel.\n"; 1882 | print "kernel-trace is available on the SLES11 SP1 DVD and you simply need to install this and boot to this\n"; 1883 | print "kernel version in order to get this working.\n\n"; 1884 | print "If you are using a differnt distro or custom kernel, you may need to rebuild your kernel with the 'CONFIG_BLK 1f40 _DEV_IO_TRACE'\n"; 1885 | print "option enabled. This should allow blktrace to function\n"; 1886 | die("ERROR: Could not run blktrace"); 1887 | } 1888 | 1889 | $pid = fork(); 1890 | if($pid < 0) 1891 | { 1892 | # Failed to fork 1893 | print "Failed to fork process... Exiting"; 1894 | exit(-1); 1895 | } 1896 | elsif ($pid ==0) 1897 | { 1898 | #child process 1899 | `nice -n 19 taskset -c $cpu_affinity blkparse -i blk.out.$dev_str.$thread_count -q -f \"%d %a %S %n\n\" | nice -n 19 taskset -c $cpu_affinity grep -v cfq| nice -n 19 taskset -c $cpu_affinity gzip --fast > blk.out.$dev_str.$thread_count.blkparse.gz; nice -n 19 taskset -c $cpu_affinity rm -f blk.out.$dev_str.$thread_count.blktrace.*; `; 1900 | exit(1); 1901 | } 1902 | else 1903 | { 1904 | print "PID=$pid CPU=$cpu_affinity\n" if ($DEBUG); 1905 | push(@pidlist, $pid); 1906 | } 1907 | 1908 | $thread_count++; 1909 | $cpu_affinity++; 1910 | $runcount--; 1911 | } 1912 | print "\rWaiting on threads to finish "; 1913 | foreach $pid (@pidlist) 1914 | { 1915 | do 1916 | { 1917 | print("\rWaiting on $pid "); 1918 | } while (!waitpid($pid, 0)); 1919 | } 1920 | undef(@pidlist); 1921 | print "\rMapping files to block locations "; 1922 | find_all_files() if($trace_files); 1923 | my $tarball_name = "$dev_str.tar"; 1924 | print "\rCreating tarball: $tarball_name "; 1925 | my $filetrace = ""; 1926 | if ($trace_files) { $filetrace = "filetrace.$dev_str.*.txt.gz"; } 1927 | `tar -cf $tarball_name blk.out.$dev_str.*.gz fdisk.$dev_str $filetrace`; 1928 | if ($? != 0) { die ("ERROR: failed to tarball $tarball_name"); } 1929 | print "\rCleaning up leftover files "; 1930 | `rm -f blk.out.$dev_str.*.gz`; 1931 | `rm -f fdisk.$dev_str`; 1932 | `rm -f filetrace.$dev_str.*.gz`; 1933 | print "\rFINISHED tracing: $tarball_name \n"; 1934 | print "Please use this file with $0 -m post -t $tarball_name to create a report\n"; 1935 | exit(); 1936 | } 1937 | 1938 | # Initialize Thread Semaphores 1939 | $read_semaphore = Thread::Semaphore->new(); 1940 | $write_semaphore = Thread::Semaphore->new(); 1941 | $read_totals_semaphore = Thread::Semaphore->new(); 1942 | $write_totals_semaphore = Thread::Semaphore->new(); 1943 | $total_semaphore = Thread::Semaphore->new(); 1944 | $total_blocks_semaphore = Thread::Semaphore->new(); 1945 | $files_to_lbas_semaphore = Thread::Semaphore->new(); 1946 | $max_bucket_hits_semaphore = Thread::Semaphore->new(); 1947 | $bucket_to_files_semaphore = Thread::Semaphore->new(); 1948 | $term_semaphore = Thread::Semaphore->new(); 1949 | $trace_files_semaphore = Thread::Semaphore->new(); 1950 | 1951 | if ($mode eq "post") 1952 | { 1953 | $read_total=0; 1954 | $write_total=0; 1955 | $max_bucket_hits=$io_total=0; 1956 | $bucket_hits_total=0; 1957 | $total_blocks=0; 1958 | $thread_total_blocks=0; 1959 | $thread_max_bucket_hits=0; 1960 | #my $cpu_affinity=1; 1961 | my $pid=-1; 1962 | my $cpu_count = `cat /proc/cpuinfo | grep processor | wc -l`; 1963 | $thread_max = $cpu_count; 1964 | my $file_count=0; 1965 | my @threads; 1966 | 1967 | my @file_list = `tar -tf $tar_file`; 1968 | 1969 | print "Unpacking $tar_file. This may take a minute.\n"; 1970 | `tar -xvf $tar_file`; 1971 | if($?) 1972 | { 1973 | die("ERROR: Failed to unpack input file: $tar_file"); 1974 | } 1975 | if(! -e $fdisk_file) 1976 | { 1977 | die("ERROR: Could not find fdisk file $fdisk_file. Please verify the .tar wasn't renamed"); 1978 | } 1979 | $sector_size=`cat $fdisk_file | grep 'Units'| awk '{ print \$9 }'`; 1980 | chomp($sector_size); 1981 | if($sector_size eq "bytes") 1982 | { 1983 | $sector_size=`cat $fdisk_file | grep 'Units'| awk '{ print \$8 }'`; 1984 | } 1985 | my $lba_line = `cat $fdisk_file | grep "sectors\$"`; 1986 | print "$lba_line\n" if ($DEBUG); 1987 | if ($lba_line =~ /(\d+) sectors$/) { $total_lbas=$1; } 1988 | $dev = `cat $fdisk_file | grep Disk | awk '{ print \$2 }'| head -1 | tr ':' ' '`; 1989 | chomp($dev); 1990 | chomp($total_lbas); 1991 | chomp($sector_size); 1992 | print "dev=$dev lbas=$total_lbas sec_size=$sector_size\n" if ($VERBOSE); 1993 | $total_capacity_GiB = $total_lbas * $sector_size / $GiB; 1994 | #print "lbas: $total_lbas sec_size: $sector_size total: $total_capacity_GiB GB\n"; 1995 | printf("lbas: %d sec_size: %d total: %0.2f GiB\n", $total_lbas, $sector_size, $total_capacity_GiB); 1996 | 1997 | $num_buckets = $total_lbas * $sector_size / $bucket_size; 1998 | 1999 | # Make the plot a matrix to keep gnuplot happy 2000 | $yheight=$xwidth = int(sqrt($num_buckets)); 2001 | print "x=$xwidth y=$yheight\n" if ($DEBUG); 2002 | 2003 | print "num_buckets=$num_buckets sector_size=$sector_size total_lbas=$total_lbas bucket_size=$bucket_size\n" if ($VERBOSE); 2004 | 2005 | `rm -f filetrace.$dev_str.*.txt`; 2006 | `rm -f blk.out.$dev_str.*.blkparse`; 2007 | print "Time to parse. Please wait...\n"; 2008 | my $size = scalar(@file_list); 2009 | 2010 | my $q; 2011 | if($THREAD_POOL && !$SINGLE_THREADED) 2012 | { 2013 | my %work_queues; 2014 | $q = Thread::Queue->new(); 2015 | $q->enqueue(@file_list); 2016 | 2017 | for(my $i=0; $i<$cpu_count; $i++) 2018 | { 2019 | my $t = threads->create('worker', $q); 2020 | push(@threads, $t); 2021 | } 2022 | } 2023 | else 2024 | { 2025 | 2026 | foreach my $file (@file_list) 2027 | { 2028 | printf "\rInput Percent: %d %% (File %d of %d)", ($file_count*100 / $size), $file_count, $size; 2029 | #print "\r$file_count of $size\n"; 2030 | if ($file =~ /(blk.out.\S+).gz/) 2031 | { 2032 | my $new_file = $1; 2033 | $file_count++; 2034 | 2035 | if($SINGLE_THREADED) 2036 | { 2037 | thread_parse($new_file, $file_count); 2038 | } 2039 | else 2040 | { 2041 | my $t=threads->new(\&thread_parse, $new_file, $file_count); 2042 | push(@threads, $t); 2043 | } 2044 | #thread_parse($new_file, $file_count); 2045 | } 2046 | elsif ($file =~ /(filetrace.\S+.\S+.txt).gz/) 2047 | { 2048 | my $new_file = $1; 2049 | $file_count++; 2050 | #`gunzip $file`; 2051 | $trace_files = 1; 2052 | print "\nFound some filetrace files\n" if ($DEBUG); 2053 | if($SINGLE_THREADED) 2054 | { 2055 | parse_filetrace($new_file, $file_count); 2056 | } 2057 | else 2058 | { 2059 | my $t=threads->new(\&parse_filetrace, $new_file, $file_count); 2060 | push(@threads, $t); 2061 | } 2062 | } 2063 | if(!$SINGLE_THREADED) 2064 | { 2065 | if(scalar @threads > $thread_max) 2066 | { 2067 | printf "\rInput Percent: %d %% (File %d of %d)", ($file_count*100 / $size), $file_count, $size; 2068 | foreach my $x (0 .. $#threads) 2069 | { 2070 | if(defined($threads[$x])) 2071 | { 2072 | if($threads[$x]->is_joinable()) 2073 | { 2074 | my $name = $threads[$x]->join(); 2075 | delete ($threads[$x]); 2076 | } 2077 | } 2078 | } 2079 | } 2080 | } 2081 | } 2082 | } 2083 | 2084 | 2085 | if($SINGLE_THREADED) 2086 | { 2087 | total_thread_counts(0); 2088 | } 2089 | else 2090 | { 2091 | while(scalar(@threads) && !$SINGLE_THREADED) 2092 | { 2093 | 2094 | if($THREAD_POOL) 2095 | { 2096 | my $tasks = $q->pending(); 2097 | printf "\rInput Percent: %d %% (Thread Count: %d)", (($size-$tasks)*100/$size), scalar(@threads); 2098 | sleep(1); 2099 | } 2100 | else 2101 | { 2102 | printf "\rInput Percent: %d %% (Thread Count: %d)", ($file_count*100 / $size), scalar(@threads); 2103 | } 2104 | foreach my $x (0 .. $#threads) 2105 | { 2106 | if(defined($threads[$x])) 2107 | { 2108 | if($threads[$x]->is_joinable()) 2109 | { 2110 | my $name = $threads[$x]->join(); 2111 | delete ($threads[$x]); 2112 | } 2113 | } 2114 | } 2115 | } 2116 | print "\rFinished parsing files. Now to analyze\n"; 2117 | undef(@threads); 2118 | } 2119 | 2120 | file_to_buckets(); 2121 | 2122 | print_results(); 2123 | print_stats(); 2124 | draw_heatmap(); 2125 | if($pdf_report) 2126 | { 2127 | print_header_heatmap(); 2128 | print_header_histogram_iops(); 2129 | print_header_stats_iops(); 2130 | create_report(); 2131 | } 2132 | cleanup_files(); 2133 | 2134 | exit; 2135 | 2136 | } 2137 | 2138 | if ($mode eq 'live') 2139 | { 2140 | $timeout=1; 2141 | my $fdisk_version = `fdisk -v`; 2142 | if($fdisk_version =~ /util-linux-ng/) 2143 | { 2144 | $sector_size = `fdisk -ul $dev 2>/dev/null| grep 'Units' | awk '{ print \$9 }'`; 2145 | $total_lbas = `fdisk -ul $dev | grep total | awk '{ print \$8 }'`; 2146 | } 2147 | else 2148 | { 2149 | $sector_size = `fdisk -l -u=sectors $dev 2>/dev/null| grep 'Units' | awk '{ print \$9 }'`; 2150 | my $lba_line = `fdisk -l -u=sectors $dev 2>/dev/null| grep "sectors\$"`; 2151 | if ($lba_line =~ /(\d+) sectors$/) { $total_lbas=$1; } 2152 | print "$lba_line\n" if ($DEBUG); 2153 | } 2154 | 2155 | my $term_x = `tput cols` -$SCALEX; 2156 | my $term_y = `tput lines` -$SCALEY; 2157 | 2158 | 2159 | chomp($total_lbas); 2160 | chomp($sector_size); 2161 | $total_capacity_GiB = $total_lbas * $sector_size / $GiB; 2162 | print "lbas: $total_lbas sec_size: $sector_size total: $total_capacity_GiB GB\n"; 2163 | 2164 | $num_buckets = `tput cols` * `tput lines`; 2165 | $bucket_size = $total_lbas * $sector_size / $num_buckets; 2166 | 2167 | # Make the plot a matrix to keep gnuplot happy 2168 | $yheight=$xwidth = int(sqrt($num_buckets)); 2169 | print "x=$xwidth y=$yheight\n" if ($DEBUG); 2170 | 2171 | print "num_buckets=$num_buckets sector_size=$sector_size total_lbas=$total_lbas bucket_size=$bucket_size\n" if ($VERBOSE); 2172 | 2173 | while(1) 2174 | { 2175 | $max_bucket_hits=$total_blocks=$read_total=$write_total=$io_total=$bucket_hits_total=0; 2176 | $thread_max_bucket_hits=$thread_io_total=$thread_bucket_hits_total=$thread_read_total=$thread_write_total=$thread_total_blocks=0; 2177 | 2178 | $live_iterations++; 2179 | if (($runtime!= 0) && (($live_iterations*$timeout) > $runtime)) 2180 | { 2181 | print "Exceeded $runtime runtime. Exiting.\n" if ($VERBOSE); 2182 | exit(); 2183 | } 2184 | 2185 | open ($CMD, "blktrace -b $buffer_size -n $buffer_count -a queue -w $timeout -d $dev -o - | blkparse -q -i - -f \"%d %a %S %n\n\" | grep -v cfq |"); 2186 | 2187 | while (<$CMD>) 2188 | { 2189 | if ($_ =~ /(\S+)\s+Q\s+(\S+)\s+(\S+)$/) 2190 | { 2191 | parse_me($1, $2, $3); 2192 | } 2193 | } 2194 | close($CMD); 2195 | total_thread_counts(0); 2196 | 2197 | print_results(); 2198 | print_stats(); 2199 | draw_heatmap(); 2200 | 2201 | undef(%thread_r_totals); 2202 | undef(%thread_w_totals); 2203 | undef(%thread_reads); 2204 | undef(%thread_writes); 2205 | undef($thread_max_bucket_hits); 2206 | undef(@reads); 2207 | undef(@writes); 2208 | undef(%r_totals); 2209 | undef(%w_totals); 2210 | } 2211 | } 2212 | -------------------------------------------------------------------------------- /ioprof.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/python -tt 2 | # I/O Profiler for Linux 3 | # Copyright (c) 2017, Intel Corporation. 4 | # 5 | # This program is free software; you can redistribute it and/or modify it 6 | # under the terms and conditions of the GNU General Public License, 7 | # version 2, as published by the Free Software Foundation. 8 | # 9 | # This program is distributed in the hope it will be useful, but WITHOUT 10 | # ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or 11 | # FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for 12 | # more details. 13 | # 14 | # You should have received a copy of the GNU General Public License 15 | # along with this program; if not, write to the Free Software 16 | # Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. 17 | 18 | import sys, getopt, os, re, string, stat, subprocess, math, shlex, time 19 | from multiprocessing import Process, Lock, Manager, Value, Array 20 | import multiprocessing 21 | 22 | # Global Variables 23 | 24 | class global_variables: 25 | #VERBOSE = False 26 | def __init__(self): 27 | self.version = "1.0.0.1" # Version string 28 | self.verbose = False # Verbose logging (-v flag) 29 | self.debug = False # Debug log level (-x flag) 30 | self.single_threaded = False # Single threaded for debug/profiling 31 | self.manager = Manager() # Multiprocess sync object 32 | 33 | self.io_total = Value('L', 0) # Number of total I/O's 34 | self.read_total = Value('L', 0) # Number of buckets read (1 I/O can touch many buckets) 35 | self.write_total = Value('L', 0) # Number of buckets written (1 I/O can touch many buckets) 36 | self.reads = self.manager.dict() # Array of read hits by bucket ID 37 | self.writes = self.manager.dict() # Array of write hits by bucket ID 38 | self.r_totals = self.manager.dict() # Hash of read I/O's with I/O size as key 39 | self.w_totals = self.manager.dict() # Hash of write I/O's with I/O size as key 40 | self.bucket_hits_total = Value('L', 0) # Total number of bucket hits (not the total buckets) 41 | self.total_blocks = Value('L', 0) # Total number of LBA's accessed during profiling 42 | self.files_to_lbas = self.manager.dict() # Files and the lba ranges associated with them 43 | self.max_bucket_hits = Value('L', 0) # The hottest bucket 44 | self.bucket_to_files = self.manager.dict() # List of files that reside on each bucket 45 | self.term = Value('L', 0) # Thread pool done with work 46 | self.trace_files = False # Map filesystem files to block LBAs 47 | 48 | ### Semaphores: These are the locks for the shared variables 49 | self.read_semaphore = self.manager.Lock() # Lock for the global read hit array 50 | self.write_semaphore = self.manager.Lock() # Lock for the global write hit array 51 | self.read_totals_semaphore = self.manager.Lock() # Lock for the global read totals 52 | self.write_totals_semaphore = self.manager.Lock() # Lock for the global write totals 53 | self.total_semaphore = self.manager.Lock() # Lock for the global I/O totals 54 | self.total_blocks_semaphore = self.manager.Lock() # Lock for the global total LBA's accessed 55 | self.files_to_lbas_semaphore = self.manager.Lock() # Lock for the global file->lba mapping hash 56 | self.max_bucket_hits_semaphore = self.manager.Lock() # Lock for the global maximum hits per bucket 57 | self.bucket_to_files_semaphore1 = self.manager.Lock() # Lock for the global bucket_to_files 58 | self.bucket_to_files_semaphore2 = self.manager.Lock() # Lock for the global bucket_to_files 59 | self.term_semaphore = self.manager.Lock() # Lock for the global TERM 60 | self.trace_files_semaphore = self.manager.Lock() # Lock for the global trace_files 61 | self.file_hit_count_semaphore = self.manager.Lock() # Lock for the global file_hit_count 62 | 63 | # Thread-local variables. Use these to avoid locking constantly 64 | self.thread_io_total = 0 # Thread-local total I/O count (I/O ops) 65 | self.thread_r_totals = {} # Thread-local read I/O size counts (ops) 66 | self.thread_w_totals = {} # Thread-local write I/O size counts (ops) 67 | self.thread_bucket_hits_total = 0 # Thread-local total bucket hits (buckets) 68 | self.thread_read_total = 0 # Thread-local total read count (I/O ops) 69 | self.thread_write_total = 0 # Thread-local total write count (I/O ops) 70 | self.thread_reads = {} # Thread-local read count hash (buckets) 71 | self.thread_writes = {} # Thread-local write count hash (buckets) 72 | self.thread_total_blocks = 0 # Thread-local total blocks accessed (lbas) 73 | self.thread_max_bucket_hits = 0 # Thread-local maximum bucket hits (bucket hits) 74 | 75 | # Globals 76 | self.file_hit_count = {} # Count of I/O's to each file 77 | self.cleanup = [] # Files to delete after running this script 78 | self.total_lbas = 0 # Total logical blocks, regardless of sector size 79 | self.tarfile = '' # .tar file outputted from 'trace' mode 80 | self.fdisk_file = "" # File capture of fdisk tool output 81 | 82 | self.top_files = [] # Top files list 83 | self.device = '' # Device (e.g. /dev/sdb) 84 | self.device_str = '' # Device string (e.g. sdb for /dev/sdb) 85 | 86 | # Unit Scales 87 | self.KiB = 1024 # 2^10 88 | self.MiB = 1048576 # 2^20 89 | self.GiB = 1073741824 # 2^30 90 | 91 | # Config settings 92 | self.bucket_size = 1 * self.MiB # Size of the bucket for totaling I/O counts (e.g. 1MB buckets) 93 | self.num_buckets = 1 # Number of total buckets for this device 94 | self.timeout = 3 # Seconds between each print 95 | self.runtime = 0 # Runtime for 'live' and 'trace' modes 96 | self.live_itterations = 0 # How many iterations for live mode. Each iteration is 'timeout' seconds long 97 | self.sector_size = 0 # Sector size (usually obtained with fdisk) 98 | self.percent = 0.020 # Histogram threshold for each level (e.g. 0.02% of total drive size) 99 | self.total_capacity_gib = 0 # Total drive capacity 100 | self.mode = '' # Processing mode (live, trace, post) 101 | self.pdf = False # Generate a PDF report instead of a text report 102 | self.top_count_limit = 10 # How many files to list in Top Files list (e.g. Top 10 files) 103 | self.thread_count = 0 # Thread Count 104 | self.cpu_affinity = 0 # Tie each thread to a CPU for load balancing 105 | self.thread_max = 32 # Max thread cout 106 | self.buffer_size = 1024 # blktrace buffer size 107 | self.buffer_count = 8 # blktrace buffer count 108 | 109 | # Gnuplot settings 110 | self.x_width = 800 # gnuplot x-width 111 | self.y_height = 600 # gnuplot y-height 112 | 113 | ### ANSI COLORS 114 | self.black = "\e[40m" 115 | self.red = "\e[41m" 116 | self.green = "\e[42m" 117 | self.yellow = "\e[43m" 118 | self.blue = "\e[44m" 119 | self.magenta = "\e[45m" 120 | self.cyan = "\e[46m" 121 | self.white = "\e[47m" 122 | self.none = "\e[0m" 123 | 124 | ### Heatmap Key 125 | self.colors = [self.black, self.red, self.green, self.yellow, self.blue, self.magenta, self.cyan, self.white, self.none] 126 | 127 | ### Heatmap Globals (TODO) 128 | self.color_index = 0 129 | self.choices = len(self.colors) 130 | self.vpc = 1 131 | self.cap = 0 132 | self.rate = 0 133 | 134 | self.mount_point = "" 135 | self.extents = [] 136 | self.files = [] 137 | # global_variables 138 | 139 | ### Print usage 140 | def usage(g, argv): 141 | name = os.path.basename(__file__) 142 | #name = argv[0] 143 | print "Invalid command\n" 144 | #print name + " " + str(argv) 145 | print name, 146 | for opt in argv: 147 | print opt, 148 | print "\n\nUsage:" 149 | print name + " -m trace -d -r [-v] [-f] # run trace for post-processing later" 150 | print name + " -m post -t [-v] [-p] # post-process mode" 151 | print name + " -m live -d -r [-v] # live mode" 152 | print "\nCommand Line Arguments:" 153 | print "-d : The device to trace (e.g. /dev/sdb). You can run traces to multiple devices (e.g. /dev/sda and /dev/sdb)" 154 | print " at the same time, but please only run 1 trace to a single device (e.g. /dev/sdb) at a time" 155 | print "-r : Runtime (seconds) for tracing" 156 | print "-t : A .tar file is created during the 'trace' phase. Please use this file for the 'post' phase" 157 | print " You can offload this file and run the 'post' phase on another system." 158 | print "-v : (OPTIONAL) Print verbose messages." 159 | print "-f : (OPTIONAL) Map all files on the device specified by -d during 'trace' phase to their LBA ranges." 160 | print " This is useful for determining the most fequently accessed files, but may take a while on really large filesystems" 161 | print "-p : (OPTIONAL) Generate a .pdf output file in addition to STDOUT. This requires 'pdflatex', 'gnuplot' and 'terminal png'" 162 | print " to be installed." 163 | sys.exit(-1) 164 | # usage (DONE) 165 | 166 | ### Check arguments 167 | def check_args(g, argv): 168 | 169 | # Gather command line arguments 170 | try: 171 | opts, args = getopt.getopt(argv,"m:d:t:fr:vpx") 172 | except getopt.GetoptError as err: 173 | print str(err) 174 | usage(g,argv) 175 | 176 | # Parse arguments 177 | for opt, arg in opts: 178 | if opt == '-m': 179 | g.mode = arg 180 | elif opt == '-d': 181 | g.device = arg 182 | elif opt == '-t': 183 | g.tarfile = arg 184 | elif opt == '-f': 185 | g.trace_files= True 186 | elif opt == '-r': 187 | g.runtime = int(arg) 188 | if g.runtime < 3: 189 | g.runtime = 3 # Min runtime 190 | elif opt == '-v': 191 | g.verbose = True 192 | elif opt == '-p': 193 | g.pdf = True 194 | elif opt == '-x': 195 | g.verbose = True 196 | g.debug = True 197 | else: 198 | usage(g,argv) 199 | 200 | # Check arguments 201 | if g.verbose == True or g.debug == True: 202 | verbose_print(g, "verbose: " + str(g.verbose) + " debug: " + str(g.debug)) 203 | 204 | if g.mode == 'live': 205 | verbose_print(g, "LIVE") 206 | if g.device == '' or g.runtime == '': 207 | usage(g,argv) 208 | debug_print(g, "Dev: " + g.device + " Runtime: " + g.runtime) 209 | match = re.search("\/dev\/(\S+)", g.device) 210 | try: 211 | debug_print(g,match.group(1)) 212 | g.device_str = string.replace(match.group(1), "/", "_") 213 | except: 214 | print "Invalid Device Type" 215 | usage(g, argv) 216 | statinfo = os.stat(g.device) 217 | if not stat.S_ISBLK(statinfo.st_mode): 218 | print "Device " + g.device + " is not a block device" 219 | usage(g,argv) 220 | elif g.mode == 'post': 221 | verbose_print(g, "POST") 222 | if g.tarfile == '': 223 | usage(g,argv) 224 | match = re.search("(\S+).tar", g.tarfile) 225 | try: 226 | debug_print(g,match.group(1)) 227 | g.device_str = match.group(1) 228 | except: 229 | print "ERROR: invalid tar file" + g.tarfile 230 | if g.pdf == True: 231 | verbose_print(g, "PDF Report Output") 232 | check_pdf_prereqs(g) 233 | 234 | print "PDF Report Output - COMING SOON..." # COMING SOON 235 | sys.exit(-1) # COMING SOON 236 | g.fdisk_file = "fdisk." + g.device_str 237 | debug_print(g, "fdisk_file: " + g.fdisk_file) 238 | g.cleanup.append(g.fdisk_file) 239 | elif g.mode == 'trace': 240 | verbose_print(g, "TRACE") 241 | check_trace_prereqs(g) 242 | if g.device == '' or g.runtime == '': 243 | usage(g,argv) 244 | debug_print(g, "Dev: " + g.device + " Runtime: " + str(g.runtime)) 245 | match = re.search("\/dev\/(\S+)", g.device) 246 | try: 247 | debug_print(g,match.group(1)) 248 | g.device_str = string.replace(match.group(1), "/", "_") 249 | except: 250 | print "Invalid Device Type" 251 | usage(g, argv) 252 | statinfo = os.stat(g.device) 253 | if not stat.S_ISBLK(statinfo.st_mode): 254 | print "Device " + g.device + " is not a block device" 255 | usage(g,argv) 256 | else: 257 | usage(g,argv) 258 | return 259 | # check_args (DONE) 260 | 261 | def debug_print(g, message): 262 | if g.debug == True: 263 | print message 264 | # debug_print (DONE) 265 | 266 | def verbose_print(g, message): 267 | if g.verbose == True: 268 | print message 269 | # verbose_print (DONE) 270 | 271 | ### Check prereqs for gnuplot and latex 272 | def check_pdf_prereqs(g): 273 | debug_print(g, "check_pdf_prereqs") 274 | 275 | rc = os.system("which gnuplot &> /dev/null") 276 | if rc != 0: 277 | print "ERROR: gnuplot not installed. Please offload the trace file for processing." 278 | sys.exit(1) 279 | else: 280 | debug_print(g, "which gnuplot: rc=" + str(rc)) 281 | rc = os.system("which pdflatex &> /dev/null") 282 | if rc != 0: 283 | print "ERROR: pdflatex not installed. Please offload the trace file for processing." 284 | sys.exit(1) 285 | else: 286 | debug_print(g, "which pdflatex: rc=" + str(rc)) 287 | rc = os.system("echo 'set terminal png' > pngtest.txt; gnuplot pngtest.txt >/dev/null 2>&1") 288 | if rc != 0: 289 | print "ERROR: gnuplot PNG terminal not installed. Please offload the trace file for processing." 290 | sys.exit(1) 291 | else: 292 | debug_print(g, "gnuplot pngtest.txt: rc=" + str(rc)) 293 | return 294 | # check_pdf_prereqs (DONE) 295 | 296 | ### Check prereqs for blktrace 297 | def check_trace_prereqs(g): 298 | debug_print(g, "check_trace_prereqs") 299 | rc = os.system("which blktrace &> /dev/null") 300 | if rc != 0: 301 | print "ERROR: blktrace not installed. Please install blktrace" 302 | sys.exit(1) 303 | else: 304 | debug_print(g, "which blktrace: rc=" + str(rc)) 305 | rc = os.system("which blkparse &> /dev/null") 306 | if rc != 0: 307 | print "ERROR: blkparse not installed. Please install blkparse" 308 | sys.exit(1) 309 | else: 310 | debug_print(g, "which blkparse: rc=" + str(rc)) 311 | # check_trace_prereqs (DONE) 312 | 313 | ### Check if debugfs is mounted 314 | def mount_debugfs(g): 315 | rc = os.system("mount | grep debugfs &> /dev/null") 316 | if rc != 0: 317 | debug_print(g, "Need to mount debugfs") 318 | rc = os.system("mount -t debugfs debugfs /sys/kernel/debug") 319 | if rc != 0: 320 | print "ERROR: Failed to mount debugfs" 321 | sys.exit(2) 322 | else: 323 | verbose_print(g, "Mounted debugfs successfully") 324 | return 325 | # mount_debugfs (DONE) 326 | 327 | ### Translate LBA to Bucket 328 | def lba_to_bucket(g, lba): 329 | bucket = (int(lba) * int(g.sector_size)) / int(g.bucket_size) 330 | if bucket > g.num_buckets: 331 | #printf("ERROR: lba=%d bucket=%d greater than num_buckets=%d\n", int(lba), bucket, g.num_buckets) 332 | bucket = g.num_buckets - 1 333 | return bucket 334 | # lba_to_bucket (DONE) 335 | 336 | ### Translate Bucket to LBA 337 | def bucket_to_lba(g, bucket): 338 | lba = (bucket * g.bucket_size) / g.sector_size 339 | return lba 340 | # bucket_to_lba (DONE) 341 | 342 | ### debugfs method 343 | ### # This method can only be used on ext2/ext3/ext4 filesystems 344 | ### I don't plan on using this method, long term 345 | ### In testing the debugfs method, I found it to be approximately 30% slower than the ioctl method in perl 346 | def debugfs_method(g, file): 347 | extents = [] 348 | file = string.replace(file, g.mountpoint, "") 349 | debug_print(g, "file: " + file) 350 | cmd = 'debugfs -R "dump_extents ' + file + '" ' + g.device + ' 2>/dev/null' 351 | debug_print(g, cmd) 352 | (rc, extent_out) = run_cmd(g, cmd) 353 | for line in extent_out: 354 | debug_print(g, line) 355 | match = re.search("\s+\d+\/\s+\d+\s+\d+\/\s+\d+\s+\d+\s+-\s+\d+\s+(\d+)\s+-\s+(\d+)", line) 356 | try: 357 | g.extents.append(match.group(1) + ":" + match.group(1)) 358 | except: 359 | debug_print(g, "no match") 360 | return 361 | # debugfs_method (DONE) 362 | 363 | ### Translate FS cluster to LBA 364 | def fs_cluster_to_lba(g, fs_cluster_size, sector_size, io_cluster): 365 | lba = io_cluster * (fs_cluster_size / sector_size) 366 | return lba 367 | # fs_cluster_to_lba (DONE) 368 | 369 | ### ioctl method (COMING SOON...) 370 | ### # This method "should" be usable regardless of filesystem 371 | ### There is some risk that FIBMAP!=1. Need to address this later 372 | ### I plan to use the ioctl method because it is 30% faster than the debugfs method 373 | def ioctl_method(g, file): 374 | print "ext3 will be supported once ioctl_method() is supported" 375 | sys.exit(-1) 376 | return 377 | # ioctl_method (COMING SOON...) 378 | 379 | ### Print filetrace files 380 | def printout(g, file): 381 | debug_print(g, "printout: " + file) 382 | cpu_affinity = 0 383 | filetrace = "filetrace." + g.device_str + "." + str(cpu_affinity) + ".txt" 384 | fo = open(filetrace, "a") 385 | try: 386 | fo.write(file + " :: " + g.extents) 387 | fo.close() 388 | except: 389 | print "ERROR: Failed to open " + filetrace 390 | sys.exit(3) 391 | # printout (DONE) 392 | 393 | def block_ranges(g, file): 394 | debug_print(g, "block_ranges: " + file) 395 | # TODO: exclude /proc and /sys mountpoints 396 | statinfo = os.stat(file) 397 | mode = statinfo.st_mode 398 | if stat.S_ISLNK(mode) or stat.ST_SIZE == 0 or not stat.ST_ISREG(mode): 399 | debug_print(g, "Disqualified file: " + file) 400 | return 401 | mounttype = os.system("mount | grep " + g.device + " | awk '{ print \$5 }'") 402 | if mounttype == "ext4": 403 | debugfs_method(g, file) 404 | elif mounttype == "ext3": 405 | ioctl_method(g, file) 406 | else: 407 | print "ERROR: " + mounttype + " is not supported yet" 408 | sys.exit(4) 409 | printout(file) 410 | return 411 | # block_ranges (DONE) 412 | 413 | def find_all_files(g): 414 | files = [] 415 | print "FIND ALL FILES" 416 | os.system("rm -f filetrace.* &>/dev/null") 417 | cpu_affinity = 0 418 | filetrace = "filetrace." + g.device_str + "." + str(cpu_affinity) + ".txt" 419 | os.system("touch " + filetrace) 420 | cmd = "cat /proc/cpuinfo | grep processor | wc -l" 421 | (rc, cpu_count) = run_cmd(g, cmd) 422 | 423 | cmd = "mount | grep " + g.device + "| awk '{ print \$3 }'" 424 | (rc, mountpoint) = run_cmd(g, cmd) 425 | if rc != 0: 426 | print g.device + " not mounted" 427 | os.system("gzip --fast filetrace.* &>/dev/null") 428 | return 429 | verbose_print(g, "mountpoint: " + mountpoint) 430 | 431 | cmd = 'mount | grep ' + g.device + " | awk '{ print \$5 }'" 432 | (rc, mounttype) = run_cmd(g, cmd) 433 | verbose_print(g, "mounttype: " + mounttype) 434 | 435 | ioprof_file = 'ioprof_files.' + g.device_str + '.txt' 436 | cmd = 'find ' + mountpoint + ' -xdev -name "*" > ' + ioprof_file 437 | rc = os.system(cmd) 438 | with open(ioprof_file, "r") as fo: 439 | for line in fo: 440 | g.files.append(line) 441 | debug_print(g, line) 442 | fo.close() 443 | 444 | file_count = len(g.files) 445 | debug_print(g, "filecount: " + file_count) 446 | 447 | # Single Threaded Method (TODO: Make Multi-threaded) 448 | k=0 449 | for i in range(0..file_count): 450 | if (k > 100): 451 | progress = (i / file_count) * 100 452 | printf("\r%05.2f%% COMPLETE", progress) 453 | k=0 454 | k = k + 1 455 | file = g.files[i] 456 | debug_print("file: " + file) 457 | block_ranges(file) 458 | 459 | os.system("gzip --fast filetrace.* &>/dev/null") 460 | return 461 | # find_all_files (DONE) 462 | 463 | ### Translate a bucket ID to a list of files 464 | def bucket_to_file_list(g, bucket_id): 465 | list="" 466 | try: 467 | list = g.bucket_to_files[bucket_id] 468 | except: 469 | pass 470 | return list 471 | # bucket_to_file_list (DONE) 472 | 473 | def file_to_bucket_helper(g, f): 474 | for file, r in f.iteritems(): 475 | #g.file_hit_count_semaphore.acquire() 476 | #g.file_hit_count[file]=0 # Initialize file hit count 477 | #g.file_hit_count_semaphore.release() 478 | tempstr = f[file] 479 | debug_print(g, "f=" + file + " r=" + r) 480 | x=0 481 | for range in tempstr.split(' '): 482 | if range == ' ' or range == '': 483 | continue # TODO 484 | debug_print(g, 'r(' + str(x) + ')=' + range) 485 | x+=1 486 | try: 487 | (start, finish) = range.split(':') 488 | except: 489 | continue 490 | debug_print(g, file + " start=" + start + ", finish=" + finish) 491 | if start == '' or finish == '': 492 | continue 493 | start_bucket = lba_to_bucket(g, start) 494 | finish_bucket = lba_to_bucket(g, finish) 495 | 496 | debug_print(g, file + " s_lba=" + start + " f_lba=" + finish + " s_buc=" + str(start_bucket) + "f_buc=" + str(finish_bucket )) 497 | #print "WAITING ON LOCK" 498 | i=start_bucket 499 | #print "GOT LOCK!" 500 | while i<= finish_bucket: 501 | if (i < (g.num_buckets/2)): 502 | g.bucket_to_files_semaphore1.acquire() 503 | else: 504 | g.bucket_to_files_semaphore2.acquire() 505 | try: 506 | g.bucket_to_files[i] = g.bucket_to_files[i] + file + " " 507 | except: 508 | g.bucket_to_files[i] = file + " " 509 | if (i < (g.num_buckets/2)): 510 | g.bucket_to_files_semaphore1.release() 511 | else: 512 | g.bucket_to_files_semaphore2.release() 513 | i+=1 514 | continue 515 | 516 | debug_print(g, "i=" + str(i)) 517 | if i in g.bucket_to_files: 518 | pattern = re.escape(file) 519 | match = re.search(pattern, g.bucket_to_files[i]) 520 | try: 521 | match.group(0) 522 | except: 523 | debug_print(g, "No Match! FILE=" + pattern + " PATTERN=" + g.bucket_to_files[i]) 524 | g.bucket_to_files[i] = g.bucket_to_files[i] + file + " " 525 | else: 526 | g.bucket_to_files[i] = file + " " 527 | debug_print(g, "i=" + str(i) + "file_to_buckets: " + g.bucket_to_files[i]) 528 | i+=1 529 | return 530 | 531 | ### Tranlate a file to a list of buckets 532 | def file_to_buckets(g): 533 | k=0 534 | size = len(g.files_to_lbas) 535 | print "Moving some memory around. This will take a few seconds..." 536 | f = dict(g.files_to_lbas) 537 | temp = {} 538 | plist = [] 539 | g.thread_max = 1024 540 | 541 | #if g.single_threaded == False: 542 | if False: 543 | for file, r in f.iteritems(): 544 | g.file_hit_count[file]=0 # Initialize file hit count 545 | temp[file] = r 546 | k+=1 547 | if k % 10 == 0: 548 | p = Process(target=file_to_bucket_helper, args=(g, temp)) 549 | plist.append(p) 550 | p.start() 551 | printf("\rfile_to_buckets: %d %% (%d of %d)", (k*100 / size), k, size) 552 | sys.stdout.flush() 553 | #if False: 554 | while len(plist) > g.thread_max: 555 | for p in plist: 556 | try: 557 | p.join(0) 558 | except: 559 | pass 560 | else: 561 | if not p.is_alive(): 562 | plist.remove(p) 563 | time.sleep(0.10) 564 | x=1 565 | while len(plist) > 0: 566 | dots="" 567 | for i in xrange(x): 568 | dots = dots + "." 569 | x+=1 570 | if x>3: 571 | x=1 572 | printf("\rWaiting on %3d threads to complete processing%-3s", len(plist), dots) 573 | printf(" ") 574 | sys.stdout.flush() 575 | for p in plist: 576 | try: 577 | p.join(0) 578 | except: 579 | pass 580 | else: 581 | if not p.is_alive(): 582 | plist.remove(p) 583 | time.sleep(0.10) 584 | 585 | print "\rDone correlating files to buckets. Now time to count bucket hits" 586 | return 587 | else: 588 | 589 | for file, r in f.iteritems(): 590 | k+=1 591 | if k % 100 == 0: 592 | printf("\rfile_to_buckets: %d %% (%d of %d)", (k*100 / size), k, size) 593 | sys.stdout.flush() 594 | g.file_hit_count[file]=0 # Initialize file hit count 595 | tempstr = f[file] 596 | debug_print(g, "f=" + file + " r=" + r) 597 | x=0 598 | for range in tempstr.split(' '): 599 | if range == ' ' or range == '': 600 | continue # TODO 601 | debug_print(g, 'r(' + str(x) + ')=' + range) 602 | x+=1 603 | try: 604 | (start, finish) = range.split(':') 605 | except: 606 | continue 607 | debug_print(g, file + " start=" + start + ", finish=" + finish) 608 | if start == '' or finish == '': 609 | continue 610 | start_bucket = lba_to_bucket(g, start) 611 | finish_bucket = lba_to_bucket(g, finish) 612 | 613 | debug_print(g, file + " s_lba=" + start + " f_lba=" + finish + " s_buc=" + str(start_bucket) + "f_buc=" + str(finish_bucket )) 614 | i=start_bucket 615 | while i<= finish_bucket: 616 | debug_print(g, "i=" + str(i)) 617 | if i in g.bucket_to_files: 618 | pattern = re.escape(file) 619 | match = re.search(pattern, g.bucket_to_files[i]) 620 | try: 621 | match.group(0) 622 | except: 623 | debug_print(g, "No Match! FILE=" + pattern + " PATTERN=" + g.bucket_to_files[i]) 624 | g.bucket_to_files[i] = g.bucket_to_files[i] + file + " " 625 | else: 626 | g.bucket_to_files[i] = file + " " 627 | debug_print(g, "i=" + str(i) + "file_to_buckets: " + g.bucket_to_files[i]) 628 | i+=1 629 | print "\rDone correlating files to buckets. Now time to count bucket hits" 630 | return 631 | # file_to_buckets (DONE) 632 | 633 | ### Add up I/O hits to each file touched by a bucket 634 | def add_file_hits(g, bucket_id, io_count): 635 | list = bucket_to_file_list(g, bucket_id) 636 | size = len(list) 637 | #print list 638 | if size == 0 and io_count != 0: 639 | debug_print(g, "No file hit. bucket=" + str(bucket_id) + ", io_cnt=" + str(io_count)) 640 | 641 | for file in list.split(' '): 642 | if file != '': 643 | debug_print(g, "file=" + file) 644 | try: 645 | g.file_hit_count[file] += io_count 646 | except: 647 | g.file_hit_count[file] = io_count 648 | return 649 | # add_file_hits (DONE) 650 | 651 | ### Get logrithmic theta for Zipfian distribution 652 | def theta_log(g, base, value): 653 | debug_print(g, "base=" + str(base) + ", value=" + str(value)) 654 | if value == 0 or base == 0: 655 | return 0 656 | else: 657 | result = math.log(value) / math.log(base) 658 | return result 659 | # theta_log (DONE) 660 | 661 | ### Print Results 662 | def print_results(g): 663 | num=0 664 | sum=0 665 | k=0 666 | buffer='' 667 | row=0 668 | column=0 669 | bw_total=0 670 | counts={} 671 | read_sum=0 672 | write_sum=0 673 | row=column=0 674 | bw_total=0 675 | histogram_iops=[] 676 | histogram_bw=[] 677 | 678 | 679 | g.verbose=True 680 | verbose_print(g, "num_buckets=" + str(g.num_buckets) + " bucket_size=" + str(g.bucket_size)) 681 | g.verbose=False 682 | if g.pdf == True: 683 | # TODO 684 | pass 685 | x=0 686 | threshold = g.num_buckets / 100 687 | i=0 688 | while i threshold: 691 | printf("\rBucket Percent: %d %% (%d of %d)", ((i * 100)/ g.num_buckets), i, g.num_buckets) 692 | sys.stdout.flush() 693 | x=0 694 | if i != 0 and (i % g.x_width) == 0: 695 | if g.pdf == True: 696 | # TODO 697 | pass 698 | buffer='' 699 | row += 1 700 | column=0 701 | 702 | 703 | r=0 704 | if i in g.reads: 705 | r=g.reads[i] 706 | w=0 707 | if i in g.writes: 708 | w=g.writes[i] 709 | 710 | bucket_total = r + w 711 | bw_total += bucket_total * g.bucket_size 712 | if g.trace_files: 713 | add_file_hits(g, i, bucket_total) 714 | if bucket_total in counts: 715 | counts[bucket_total] += 1 716 | else: 717 | counts[bucket_total] = 1 718 | debug_print(g, "bucket_total=" + str(bucket_total) + " counts[b_bucket_total]=" + str(counts[bucket_total])) 719 | read_sum += r 720 | write_sum += w 721 | buffer = buffer + ("%d " % bucket_total) 722 | column += 1 723 | i+=1 724 | 725 | print "\r " 726 | while (i % g.x_width) != 0: 727 | i+=1 728 | buffer = buffer + "0 " 729 | 730 | if g.pdf: 731 | # TODO 732 | pass 733 | 734 | verbose_print(g, "num_buckets=%s pfgp iot=%s bht=%s r_sum=%s w_sum=%s yheight=%s" % (g.num_buckets, g.io_total.value, g.bucket_hits_total.value, read_sum, write_sum, g.y_height)) 735 | 736 | t=0 737 | j=0 738 | section_count=0 739 | b_count=0 740 | gb_tot = 0 741 | bw_tot = 0 742 | bw_count = 0 743 | io_sum = 0 744 | tot = 0 745 | if g.pdf: 746 | # TODO 747 | pass 748 | 749 | max_set = 0 750 | max = 0 751 | theta_count = 1 752 | theta_total = 0 753 | min = 1 754 | max_theta = 0 755 | min_theta = 999 756 | 757 | 758 | # %counts is a hash 759 | # each key "bucket_total" represents a particular I/O count for a bucket 760 | # each value represents the number of buckets that had this I/O count 761 | # This allows me to quickly tally up a histogram and is pretty 762 | # space efficient since most buckets tend to have zero I/O that 763 | # key tends to have the largest number of buckets 764 | # 765 | # Iterate through each key in decending order 766 | for total in sorted(counts, reverse=True): 767 | debug_print(g, "total=" + str(total) + " counts=" + str(counts[total])) 768 | if total > 0: 769 | tot += total * counts[total] 770 | if max_set == 0: 771 | max_set=1 772 | max = total 773 | else: 774 | theta_count += 1 775 | min = total 776 | cur_theta = theta_log(g, theta_count, max) - theta_log(g, theta_count, total) 777 | if cur_theta > max_theta: 778 | max_theta = cur_theta 779 | if cur_theta < min_theta: 780 | min_theta = cur_theta 781 | debug_print(g, "cur_theta=" + str(cur_theta)) 782 | theta_total += cur_theta 783 | i=0 784 | while i (g.percent * g.total_capacity_gib): 789 | debug_print(g, "b_count:" + str(b_count)) 790 | bw_tot += bw_count 791 | gb_tot += (b_count * g.bucket_size) 792 | io_sum += section_count 793 | 794 | gb = "%.1f" % (gb_tot / g.GiB) 795 | if g.bucket_hits_total.value == 0: 796 | io_perc = "NA" 797 | io_sum_perc = "NA" 798 | bw_perc = "NA" 799 | else: 800 | debug_print(g, "b_count=" + str(b_count) + " s=" + str(section_count) + " ios=" + str(io_sum) + " bwc=" + str(bw_count)) 801 | io_perc = "%.1f" % ((float(section_count) / float(g.bucket_hits_total.value)) * 100.0) 802 | io_sum_perc = "%.1f" % ((float(io_sum) / float(g.bucket_hits_total.value)) * 100.0) 803 | if bw_total == 0: 804 | bw_perc = "%.1f" % (0) 805 | else: 806 | bw_perc = "%.1f" % ((bw_count / bw_total) * 100) 807 | 808 | if g.pdf: 809 | # TODO 810 | pass 811 | 812 | histogram_iops.append(str(gb) + " GB " + str(io_perc) + "% (" + io_sum_perc + "% cumulative)") 813 | histogram_bw.append(str(gb) + " GB " + str(bw_perc) + "% ") 814 | 815 | b_count=0 816 | section_count=0 817 | bw_count=0 818 | 819 | i += 1 820 | if b_count: 821 | debug_print(g, "b_count: " + str(b_count)) 822 | bw_tot += bw_count 823 | gb_tot += b_count * g.bucket_size 824 | io_sum += section_count 825 | 826 | gb = "%.1f" % (gb_tot / g.GiB) 827 | if g.bucket_hits_total.value == 0: 828 | io_perc = "NA" 829 | io_sum_perc = "NA" 830 | bw_perc = "NA" 831 | else: 832 | io_perc = "%.1f" % ((section_count / g.bucket_hits_total.value) * 100) 833 | io_sum_perc = "%.1f" % ((io_sum / g.bucket_hits_total.value) * 100) 834 | if bw_total == 0: 835 | bw_perc = "%.1f" % (0) 836 | else: 837 | bw_perc = "%.1f" % ((bw_count / bw_total) * 100) 838 | 839 | if g.pdf: 840 | # TODO 841 | pass 842 | 843 | histogram_iops.append(str(gb) + " GB " + str(io_perc) + "% (" + str(io_sum_perc) + "% cumulative)") 844 | histogram_bw.append(str(gb) + " GB " + str(bw_perc) + "% ") 845 | 846 | b_count = 0 847 | section_count = 0 848 | bw_count = 0 849 | 850 | if g.pdf: 851 | # TODO 852 | pass 853 | 854 | debug_print(g, "t=" + str(t)) 855 | 856 | print "--------------------------------------------" 857 | print "Histogram IOPS:" 858 | for entry in histogram_iops: 859 | print entry 860 | print "--------------------------------------------" 861 | 862 | # TODO: Check that this is consistent with Perl version 863 | if (theta_count): 864 | avg_theta = theta_total / theta_count 865 | med_theta = ((max_theta - min_theta) / 2 ) + min_theta 866 | approx_theta = (avg_theta + med_theta) / 2 867 | #string = "avg_t=%s med_t=%s approx_t=%s min_t=%s max_t=%s\n" % (avg_theta, med_theta, approx_theta, min_theta, max_theta) 868 | verbose_print(g, "avg_t=%s med_t=%s approx_t=%s min_t=%s max_t=%s\n" % (avg_theta, med_theta, approx_theta, min_theta, max_theta)) 869 | analysis_histogram_iops = "Approximate Zipfian Theta Range: %0.4f-%0.4f (est. %0.4f).\n" % (min_theta, max_theta, approx_theta) 870 | print analysis_histogram_iops 871 | 872 | debug_print(g, "Trace_files: " + str(g.trace_files)) 873 | if g.trace_files: 874 | top_count=0 875 | print "--------------------------------------------" 876 | print "Top files by IOPS:" 877 | print "Total I/O's: " + str(g.bucket_hits_total.value) 878 | if g.bucket_hits_total.value == 0: 879 | print "No Bucket Hits" 880 | else: 881 | for filename in sorted(g.file_hit_count, reverse=True, key=g.file_hit_count.get): 882 | hits = g.file_hit_count[filename] 883 | if hits > 0: 884 | hit_rate = (float(hits) / float(g.bucket_hits_total.value)) * 100.0 885 | print "%0.2f%% (%d) %s" % (hit_rate, hits, filename) 886 | if g.pdf: 887 | g.top_files = append("%0.2f%%: (%d) %s\n" % (hit_rate, hits, filename)) 888 | top_count += 1 889 | if top_count > g.top_count_limit: 890 | break 891 | print "--------------------------------------------" 892 | 893 | return 894 | # print_results (IN PROGRESS) 895 | 896 | ### Print heatmap header for PDF 897 | def print_header_heatmap(g): 898 | return 899 | # print_header_heatmap (TODO) 900 | 901 | ### Print histogram header for PDF 902 | def print_header_histogram_iops(g): 903 | return 904 | # print_header_histogram_iops (TODO) 905 | 906 | ### Print stats header for PDF 907 | def print_header_stats_iops(g): 908 | return 909 | # print_header_stats_iops (TODO) 910 | 911 | ### Create PDF Report 912 | def create_report(g): 913 | return 914 | # create_report (TODO) 915 | 916 | ### Print I/O statistics 917 | def print_stats(g): 918 | return 919 | # print_stats (TODO) 920 | 921 | 922 | ### Combine thread-local counts into global counts 923 | def total_thread_counts (g, num): 924 | 925 | g.max_bucket_hits_semaphore.acquire() 926 | debug_print(g, "Thread " + str(num) + " has max_bucket_hits lock t=" + str(g.thread_max_bucket_hits) + " g=" + str(g.max_bucket_hits.value)) 927 | if(g.thread_max_bucket_hits > g.max_bucket_hits.value): 928 | g.max_bucket_hits.value = g.thread_max_bucket_hits 929 | debug_print(g, "Thread " + str(num) + " releasing max_bucket_hits lock t=" + str(g.thread_max_bucket_hits) + " g=" + str(g.max_bucket_hits.value)) 930 | g.max_bucket_hits_semaphore.release() 931 | 932 | g.total_blocks_semaphore.acquire() 933 | debug_print(g, "Thread " + str(num) + " has total_blocks lock t=" + str(g.thread_total_blocks) + " g=" + str(g.total_blocks.value)) 934 | g.total_blocks.value += g.thread_total_blocks 935 | debug_print(g, "Thread " + str(num) + " releasing total_blocks lock t=" + str(g.thread_total_blocks) + " g=" + str(g.total_blocks.value)) 936 | g.total_blocks_semaphore.release() 937 | 938 | g.total_semaphore.acquire() 939 | debug_print(g, "Thread " + str(num) + " has total lock t=" + str(g.thread_io_total) + " g=" + str(g.io_total.value)) 940 | g.io_total.value += g.thread_io_total 941 | debug_print(g, "Thread " + str(num) + " releasing total lock t=" + str(g.thread_io_total) + " g=" + str(g.io_total.value)) 942 | g.total_semaphore.release() 943 | 944 | g.read_totals_semaphore.acquire() 945 | debug_print(g, "Thread " + str(num) + " has read_totals lock t=" + str(g.thread_read_total) + " g=" + str(g.read_total.value)) 946 | g.read_total.value += g.thread_read_total 947 | for io_size, hits in g.thread_r_totals.iteritems(): 948 | if io_size in g.r_totals: 949 | g.r_totals[io_size] += hits 950 | else: 951 | g.r_totals[io_size] = hits 952 | debug_print(g, "Thread " + str(num) + " releasing read_totals lock t=" + str(g.thread_read_total) + " g=" + str(g.read_total.value)) 953 | g.read_totals_semaphore.release() 954 | 955 | g.write_totals_semaphore.acquire() 956 | debug_print(g, "Thread " + str(num) + " has write_totals lock t=" + str(g.thread_write_total) + " g=" + str(g.write_total.value)) 957 | g.write_total.value += g.thread_write_total 958 | for io_size, hits in g.thread_w_totals.iteritems(): 959 | if io_size in g.w_totals: 960 | g.w_totals[io_size] += hits 961 | else: 962 | g.w_totals[io_size] = hits 963 | debug_print(g, "Thread " + str(num) + " releasing write_totals lock t=" + str(g.thread_write_total) + " g=" + str(g.write_total.value)) 964 | g.write_totals_semaphore.release() 965 | 966 | g.read_semaphore.acquire() 967 | debug_print(g, "Thread " + str(num) + " has read lock.") 968 | for bucket,value in g.thread_reads.iteritems(): 969 | try: 970 | g.reads[bucket] += value 971 | except: 972 | g.reads[bucket] = value 973 | debug_print(g, "Thread " + str(num) + " has read lock. Bucket=" + str(bucket) + " Value=" + str(value) + " g.reads[bucket]=" + str(g.reads[bucket])) 974 | g.read_semaphore.release() 975 | 976 | g.write_semaphore.acquire() 977 | debug_print(g, "Thread " + str(num) + " has write lock.") 978 | for bucket,value in g.thread_writes.iteritems(): 979 | try: 980 | g.writes[bucket] += value 981 | except: 982 | g.writes[bucket] = value 983 | debug_print(g, "Thread " + str(num) + " has write lock. Bucket=" + str(bucket) + " Value=" + str(value) + " g.writes[bucket]=" + str(g.writes[bucket])) 984 | g.write_semaphore.release() 985 | 986 | g.total_semaphore.acquire() 987 | debug_print(g, "Thread " + str(num) + " has total lock t=" + str(g.thread_bucket_hits_total) + " g=" + str(g.bucket_hits_total.value)) 988 | g.bucket_hits_total.value += g.thread_bucket_hits_total 989 | debug_print(g, "Thread " + str(num) + " releasing total lock t=" + str(g.thread_bucket_hits_total) + " g=" + str(g.bucket_hits_total.value)) 990 | g.total_semaphore.release() 991 | 992 | return 993 | # total_thread_counts (DONE) 994 | 995 | ### Thread parse routine for blktrace output 996 | def thread_parse(g, file, num): 997 | #print "thread_parse\n" 998 | linecount = 0 999 | os.system("gunzip " + file + ".gz") 1000 | debug_print(g, "\nSTART: " + file + " " + str(num) + "\n") 1001 | try: 1002 | fo = open(file, "r") 1003 | except: 1004 | print "ERROR: Failed to open " + file 1005 | sys.exit(3) 1006 | else: 1007 | count=0 1008 | hit_count = 0 1009 | for line in fo: 1010 | count += 1 1011 | result_set = regex_find(g, '(\S+)\s+Q\s+(\S+)\s+(\S+)$', line) 1012 | if result_set != False: 1013 | #(a, b, c) = regex_find('.*Q.*', line) 1014 | hit_count += 1 1015 | #print len(set) 1016 | #print "a=" + a + " b=" + b + " c=" + c + "\n" 1017 | #print set 1018 | #sys.stdout.flush() 1019 | try: 1020 | parse_me(g, result_set[0], int(result_set[1]), int(result_set[2])) 1021 | except: 1022 | pass 1023 | #sys.stdout.flush() 1024 | fo.close() 1025 | total_thread_counts(g, num) 1026 | debug_print(g, "\n FINISH" + file + " (" + str(count) + " lines) [hit_count=" + str(hit_count) + "]" + str(g.thread_io_total) + "\n") 1027 | rc = os.system("rm -f " + file) 1028 | return 1029 | 1030 | # thread_parse (DONE) 1031 | 1032 | ### Parse blktrace output 1033 | def parse_me(g, rw, lba, size): 1034 | debug_print(g, "rw=" + rw + " lba=" + str(lba) + " size=" + str(size)) 1035 | if (rw == 'R') or (rw == 'RW'): 1036 | # Read 1037 | g.thread_total_blocks += int(size) 1038 | g.thread_io_total += 1 1039 | g.thread_read_total += 1 1040 | if size in g.thread_r_totals: 1041 | g.thread_r_totals[size] += 1 1042 | else: 1043 | g.thread_r_totals[size] = 1 1044 | bucket_hits = (size * g.sector_size) / g.bucket_size 1045 | if ((size * g.sector_size) % g.bucket_size) != 0: 1046 | bucket_hits += 1 1047 | for i in xrange(0, bucket_hits): 1048 | bucket = int((lba * g.sector_size) / g.bucket_size) + i 1049 | if bucket > g.num_buckets: 1050 | # Not sure why, but we occassionally get buckets beyond our max LBA range 1051 | bucket = g.num_buckets - 1 1052 | if True: 1053 | if bucket in g.thread_reads: 1054 | g.thread_reads[bucket] += 1 1055 | else: 1056 | g.thread_reads[bucket] = 1 1057 | else: 1058 | try: 1059 | g.thread_reads[bucket] += 1 1060 | except: 1061 | g.thread_reads[bucket] = 1 1062 | if(g.thread_reads[bucket] > g.thread_max_bucket_hits): 1063 | g.thread_max_bucket_hits = g.thread_reads[bucket] 1064 | g.thread_bucket_hits_total += 1 1065 | elif (rw == 'W') or (rw == 'WS'): 1066 | # Write 1067 | g.thread_total_blocks += int(size) 1068 | g.thread_io_total += 1 1069 | g.thread_write_total += 1 1070 | if size in g.thread_w_totals: 1071 | g.thread_w_totals[size] += 1 1072 | else: 1073 | g.thread_w_totals[size] = 1 1074 | bucket_hits = (size * g.sector_size) / g.bucket_size 1075 | if ((size * g.sector_size) % g.bucket_size) != 0: 1076 | bucket_hits += 1 1077 | for i in xrange(0, bucket_hits): 1078 | bucket = int((lba * g.sector_size) / g.bucket_size) + i 1079 | if bucket > g.num_buckets: 1080 | # Not sure why, but we occassionally get buckets beyond our max LBA range 1081 | bucket = g.num_buckets - 1 1082 | if True: 1083 | if bucket in g.thread_writes: 1084 | g.thread_writes[bucket] += 1 1085 | else: 1086 | g.thread_writes[bucket] = 1 1087 | else: 1088 | try: 1089 | g.thread_writes[bucket] += 1 1090 | except: 1091 | g.thread_writes[bucket] = 1 1092 | if(g.thread_writes[bucket] > g.thread_max_bucket_hits): 1093 | g.thread_max_bucket_hits = g.thread_writes[bucket] 1094 | g.thread_bucket_hits_total += 1 1095 | return 1096 | # parse_me (DONE) 1097 | 1098 | ## File trace routine 1099 | def parse_filetrace(g, filename, num): 1100 | thread_files_to_lbas = {} 1101 | os.system("gunzip " + filename + ".gz") 1102 | debug_print(g, "tracefile = " + filename + " " + str(num) + "\n") 1103 | try: 1104 | fo = open(filename, "r") 1105 | except Exception as e: 1106 | print "ERROR: Failed to open " + filename + " Err: ", e 1107 | sys.exit(3) 1108 | else: 1109 | for line in fo: 1110 | result_set = regex_find(g, '(\S+)\s+::\s+(.+)', line) 1111 | if result_set != False: 1112 | object = result_set[0] 1113 | ranges = result_set[1] 1114 | thread_files_to_lbas[object] = ranges 1115 | debug_print(g, filename + ": obj=" + object + " ranges:" + ranges + "\n") 1116 | fo.close() 1117 | 1118 | debug_print(g, "Thread " + str(num) + "wants file_to_lba lock for " + filename + "\n") 1119 | g.files_to_lbas_semaphore.acquire() 1120 | for key,value in thread_files_to_lbas.iteritems(): 1121 | g.files_to_lbas[key] = value 1122 | debug_print(g, "k=" + str(key) + " value=" + str(g.files_to_lbas[key])) 1123 | g.files_to_lbas_semaphore.release() 1124 | debug_print(g, "Thread " + str(num) + "freed file_to_lba lock for " + filename + "\n") 1125 | 1126 | return 1127 | # parse_filetrace (DONE) 1128 | 1129 | ### Choose color for heatmap block 1130 | def choose_color(g, num): 1131 | if num == -1 or num == 0: 1132 | return g.black 1133 | g.color_index = num / g.vpc 1134 | if (g.color_index > (g.choices - 1)): 1135 | g.debug = True 1136 | debug_print(g, "HIT! num=" + num) 1137 | g.debug = False 1138 | g.color_index=7 1139 | return g.red 1140 | color = g.colors[g.color_index] 1141 | debug_print(g, "cap=" + g.cap + " num=" + num + " ci=" + g.color_index + " vpc=" + g.vpc + " cap=" + g.cap) 1142 | return color 1143 | # choose_color (DONE) 1144 | 1145 | ### Clear Screen for heatmap (UNUSED) 1146 | def clear_screen(g): 1147 | print "\033[2J" 1148 | print "\[\033[0;0f\]\r" 1149 | return 1150 | # clear_screen (DONE) 1151 | 1152 | ### Get block value by combining buckets into larger heatmap blocks for term 1153 | def get_value(g, offset, rate): 1154 | start = offset * rate 1155 | end = start + rate 1156 | sum = 0 1157 | 1158 | g.debug = True 1159 | debug_print(g, "start=" + start + " end=" + end) 1160 | g.debug = False 1161 | 1162 | index=start 1163 | while(index <= end): 1164 | index+=1 1165 | r = 0 1166 | w = 0 1167 | if index in g.reads: 1168 | r = g.reads[index] 1169 | if index in g.writes: 1170 | w = g.writes[index] 1171 | sum = sum + r + w 1172 | 1173 | g.debug = True 1174 | debug_print(g, "s=" + sum) 1175 | g.debug = False 1176 | 1177 | return sum 1178 | # get_value (DONE) 1179 | 1180 | ### Draw heatmap on color terminal 1181 | def draw_heatmap(g): 1182 | return 1183 | # draw_heatmap (TODO) 1184 | 1185 | ### Cleanup temp files 1186 | def cleanup_files(g): 1187 | verbose_print(g, "Cleaning up temp files\n") 1188 | for file in g.cleanup: 1189 | debug_print(g, file) 1190 | os.system("rm -f " + file) 1191 | os.system("rm -f filetrace.*.txt") 1192 | return 1193 | # cleanup_files (DONE) 1194 | 1195 | ### run_cmd 1196 | def run_cmd(g, cmd): 1197 | rc = 0 1198 | out = "" 1199 | debug_print(g, "cmd: " + cmd) 1200 | args = shlex.split(cmd) 1201 | 1202 | try: 1203 | p = subprocess.Popen(args, stdout=subprocess.PIPE, stderr=subprocess.PIPE) 1204 | except: 1205 | print "ERROR: problem with Popen" 1206 | sys.exit(1) 1207 | try: 1208 | out, error = p.communicate() 1209 | except: 1210 | print "ERROR: problem with Popen.communicate()" 1211 | rc = p.returncode 1212 | if rc == 0: 1213 | debug_print(g, "rc=" + str(p.returncode)) 1214 | return (rc, out) 1215 | 1216 | # run_cmd 1217 | 1218 | ### regex_find 1219 | def regex_find(g, pattern, input): 1220 | output = False 1221 | if g.verbose == True or g.debug == True: 1222 | print "PATTERN: ", pattern 1223 | for line in input.split("\n"): 1224 | if g.verbose == True or g.debug == True: 1225 | print "LINE: ", line 1226 | match = re.search(pattern, line) 1227 | if match != None: 1228 | output = match.groups() 1229 | if g.verbose == True or g.debug == True: 1230 | print 'MATCHED pattern' 1231 | break 1232 | return output 1233 | # regex_find 1234 | 1235 | def printf(format, *args): 1236 | sys.stdout.write(format % args) 1237 | # printf (DONE) 1238 | 1239 | ### MAIN 1240 | def main(argv): 1241 | g = global_variables() 1242 | print "VERSION: ", g.version 1243 | 1244 | check_args(g, argv) 1245 | 1246 | if g.mode == 'live' or g.mode == 'trace': 1247 | mount_debugfs(g) 1248 | 1249 | if g.mode == 'trace': 1250 | # Trace 1251 | 1252 | # Check sudo permissions 1253 | rc = os.system("sudo -v &>/dev/null") 1254 | if rc != 0: 1255 | print "ERROR: You need to have sudo permissions to collect all necessary data. Please run from a privilaged account." 1256 | sys.exit(6) 1257 | # Save fdisk info 1258 | debug_print(g, "Running fdisk") 1259 | fdisk_version = "" 1260 | (rc, fdisk_version) = run_cmd(g, "fdisk -v") 1261 | match = re.search("util-linux-ng", fdisk_version) 1262 | if match: 1263 | # RHEL 6.x 1264 | os.system("fdisk -ul "+g.device+" > fdisk."+g.device_str) 1265 | else: 1266 | # RHEL 7.x 1267 | os.system("fdisk -l -u=sectors "+g.device+" > fdisk."+g.device_str) 1268 | 1269 | os.system("rm -f blk.out.* &>/dev/null") # Cleanup previous mess 1270 | runcount = g.runtime / g.timeout 1271 | while runcount > 0: 1272 | time_left = runcount * g.timeout 1273 | percent_prog = (g.runtime - time_left) * 100 / g.runtime 1274 | printf( "\r%d %% done (%d seconds left)", percent_prog, time_left) 1275 | # BEN 1276 | sys.stdout.flush() 1277 | cmd = "blktrace -b " + str(g.buffer_size) + " -n " + str(g.buffer_count) + " -a queue -d " + str(g.device) + " -o blk.out." + str(g.device_str) + ".0 -w " + str(g.timeout) + " &> /dev/null" 1278 | rc = os.system(cmd) 1279 | if rc != 0: 1280 | print "Unable to run the 'blktrace' tool required to trace all of your I/O" 1281 | print "If you are using SLES 11 SP1, then it is likely that your default kernel is missing CONFIG_BLK_DEV_IO_TRACE" 1282 | print "which is required to run blktrace. This is only available in the kernel-trace version of the kernel." 1283 | print "kernel-trace is available on the SLES11 SP1 DVD and you simply need to install this and boot to this" 1284 | print "kernel version in order to get this working." 1285 | print "If you are using a differnt distro or custom kernel, you may need to rebuild your kernel with the 'CONFIG_BLK 1f40 _DEV_IO_TRACE'" 1286 | print "option enabled. This should allow blktrace to function\n" 1287 | print "ERROR: Could not run blktrace" 1288 | sys.exit(7) 1289 | cmd = "blkparse -i blk.out." + g.device_str + ".0 -q -f " + '" %d %a %S %n\n" | grep -v cfq | gzip --fast > blk.out.' + g.device_str + ".0.blkparse.gz;" 1290 | rc = os.system(cmd) 1291 | runcount -= 1 1292 | print "\rMapping files to block locations " 1293 | if g.trace_files: 1294 | find_all_files(g) 1295 | tarball_name = g.device_str + ".tar" 1296 | print "\rCreating tarball " + tarball_name 1297 | filetrace = "" 1298 | if g.trace_files: 1299 | filetrace = "filetrace." + g.device_str + ".*.txt.gz" 1300 | cmd = "tar -cf " + tarball_name + " blk.out." + g.device_str + ".*.gz fdisk." + g.device_str + " " + filetrace + " &> /dev/null" 1301 | print cmd 1302 | rc = os.system(cmd) 1303 | if rc != 0: 1304 | print "ERROR: failed to tarball " + tarball_name 1305 | sys.exit(8) 1306 | cmd = "rm -f blk.out." + g.device_str + ".*.gz; rm -f fdisk." + g.device_str + "; rm -f filetrace." + g.device_str + ".*.gz" 1307 | rc = os.system(cmd) 1308 | print "\rFINISHED tracing: " + tarball_name 1309 | name = os.path.basename(__file__) 1310 | print "Please use this file with " + name + " -m post -t " + tarball_name + " to create a report" 1311 | 1312 | elif g.mode == 'post': 1313 | # Post 1314 | g.THREAD_MAX = multiprocessing.cpu_count() * 4 1315 | cmd = 'tar -tf ' + g.tarfile 1316 | print g.tarfile 1317 | (rc, file_text) = run_cmd(g, cmd) 1318 | debug_print(g, file_text) 1319 | file_list = [] 1320 | for i in file_text.split("\n"): 1321 | debug_print(g, "i=" + i) 1322 | if i != "": 1323 | file_list.append(i) 1324 | if rc != 0: 1325 | print "ERROR: Failed to test input file: " + g.tarfile 1326 | sys.exit(9) 1327 | print "Unpacking " + g.tarfile + ". This may take a minute" 1328 | cmd = 'tar -xvf ' + g.tarfile 1329 | (rc, out) = run_cmd(g, cmd) 1330 | if rc != 0: 1331 | print "ERROR: Failed to unpack input file: " + g.tarfile 1332 | sys.exit(9) 1333 | 1334 | rc=0 1335 | out="" 1336 | (rc, out) = run_cmd(g, 'cat '+ g.fdisk_file ) 1337 | result = regex_find(g, "Units = sectors of \d+ \S \d+ = (\d+) bytes", out) 1338 | if result == False: 1339 | #Units: sectors of 1 * 512 = 512 bytes 1340 | result = regex_find(g, "Units: sectors of \d+ \* \d+ = (\d+) bytes", out) 1341 | if result == False: 1342 | print "ERROR: Sector Size Invalid" 1343 | sys.exit() 1344 | g.sector_size = int(result[0]) 1345 | verbose_print(g, "sector size="+ str(g.sector_size)) 1346 | result = regex_find(g, ".+ total (\d+) sectors", out) 1347 | if result == False: 1348 | #Disk /dev/sdb: 111.8 GiB, 120034123776 bytes, 234441648 sectors 1349 | result = regex_find(g, "Disk /dev/\w+: \d+.\d+ GiB, \d+ bytes, (\d+) sectors", out) 1350 | if result == False: 1351 | print "ERROR: Total LBAs is Invalid" 1352 | sys.exit() 1353 | g.total_lbas = int(result[0]) 1354 | verbose_print(g, "sector count ="+ str(g.total_lbas)) 1355 | 1356 | result = regex_find(g, "Disk (\S+): \S+ GB, \d+ bytes", out) 1357 | if result == False: 1358 | # LINE: Disk /dev/sdb: 111.8 GiB, 120034123776 bytes, 234441648 sectors 1359 | result = regex_find(g, "Disk (\S+):", out) 1360 | if result == False: 1361 | print "ERROR: Device Name is Invalid" 1362 | sys.exit() 1363 | g.device = result[0] 1364 | verbose_print(g, "dev="+ g.device + " lbas=" + str(g.total_lbas) + " sec_size=" + str(g.sector_size)) 1365 | 1366 | 1367 | g.total_capacity_gib = g.total_lbas * g.sector_size / g.GiB 1368 | printf("lbas: %d sec_size: %d total: %0.2f GiB\n", g.total_lbas, g.sector_size, g.total_capacity_gib) 1369 | 1370 | g.num_buckets = g.total_lbas * g.sector_size / g.bucket_size 1371 | 1372 | # Make the PDF plot a square matrix to keep gnuplot happy 1373 | g.y_height = g.x_width = int(math.sqrt(g.num_buckets)) 1374 | debug_print(g, "x=" + str(g.x_width) + " y=" + str(g.y_height)) 1375 | 1376 | g.debug=True 1377 | debug_print(g, "num_buckets=" + str(g.num_buckets) + " sector_size=" + str(g.sector_size) + " total_lbas=" + str(g.total_lbas) + " bucket_size=" + str(g.bucket_size)) 1378 | g.debug=False 1379 | rc = os.system("rm -f filetrace." + g.device_str + ".*.txt") 1380 | rc = os.system("rm -f blk.out." + g.device_str + ".*.blkparse") 1381 | print "Time to parse. Please wait...\n" 1382 | 1383 | size = len(file_list) 1384 | file_count = 0 1385 | 1386 | plist = [] 1387 | for filename in file_list: 1388 | file_count += 1 1389 | perc = file_count * 100 / size 1390 | printf("\rInput Percent: %d %% (File %d of %d) threads=%d", (file_count*100 / size), file_count, size, len(plist)) 1391 | sys.stdout.flush() 1392 | result = regex_find(g, "(blk.out.\S+).gz", filename) 1393 | if result != False: 1394 | new_file = result[0] 1395 | if g.single_threaded: 1396 | thread_parse(g, new_file, file_count) 1397 | debug_print(g, "blk.out hit = " + filename + "\n") 1398 | else: 1399 | p = Process(target=thread_parse, args=(g, new_file, file_count)) 1400 | plist.append(p) 1401 | p.start() 1402 | result = regex_find(g, "(filetrace.\S+.\S+.txt).gz", filename) 1403 | if result != False: 1404 | new_file = result[0] 1405 | g.trace_files=True 1406 | debug_print(g, "filetrace hit = " + filename+ "\n") 1407 | if g.single_threaded: 1408 | parse_filetrace(g, new_file, file_count) 1409 | debug_print(g, "blk.out hit = " + filename + "\n") 1410 | else: 1411 | p = Process(target=parse_filetrace, args=(g, new_file, file_count)) 1412 | plist.append(p) 1413 | p.start() 1414 | while len(plist) > g.thread_max: 1415 | for p in plist: 1416 | try: 1417 | p.join(0) 1418 | except: 1419 | pass 1420 | else: 1421 | if not p.is_alive(): 1422 | plist.remove(p) 1423 | time.sleep(0.10) 1424 | if g.single_threaded == False: 1425 | x=1 1426 | while len(plist) > 0: 1427 | dots="" 1428 | for i in xrange(x): 1429 | dots = dots + "." 1430 | x+=1 1431 | if x>3: 1432 | x=1 1433 | printf("\rWaiting on %3d threads to complete processing%-3s", len(plist), dots) 1434 | printf(" ") 1435 | sys.stdout.flush() 1436 | for p in plist: 1437 | try: 1438 | p.join(0) 1439 | except: 1440 | pass 1441 | else: 1442 | if not p.is_alive(): 1443 | plist.remove(p) 1444 | time.sleep(0.10) 1445 | print "\rFinished parsing files. Now to analyze \n" 1446 | file_to_buckets(g) 1447 | print_results(g) 1448 | print_stats(g) 1449 | draw_heatmap(g) 1450 | if g.pdf == True: 1451 | print_header_heatmap(g) 1452 | print_header_histogram_iops(g) 1453 | print_header_stats_iops(g) 1454 | create_report(g) 1455 | cleanup_files(g) 1456 | 1457 | elif g.mode == 'live': 1458 | # Live 1459 | print "Live Mode - Coming Soon ..." 1460 | 1461 | sys.exit() 1462 | # main (IN PROGRESS) 1463 | 1464 | ### Start MAIN 1465 | if __name__ == "__main__": 1466 | main(sys.argv[1:]) 1467 | --------------------------------------------------------------------------------