├── .gitignore ├── .gitmodules ├── LICENSE ├── Makefile ├── README.md ├── cross_compile.mk ├── parse_argp.c ├── parse_argp.h ├── parse_kallsyms.c ├── parse_kallsyms.h ├── perfetto_wrapper.cc ├── perfetto_wrapper.h ├── sched-analyzer-events.h ├── sched-analyzer-pp ├── README.md ├── freq.py ├── idle.py ├── requirements.txt ├── sa_track.py ├── sched-analyzer-pp ├── sched.py ├── settings.py ├── trace_processor.py └── utils.py ├── sched-analyzer.bpf.c ├── sched-analyzer.c ├── screenshots ├── sched-analyzer-screenshot-ipi.png ├── sched-analyzer-screenshot-pelt-filtered.png └── sched-analyzer-screenshot.png └── vmlinux.h /.gitignore: -------------------------------------------------------------------------------- 1 | sched-analyzer 2 | *.o 3 | *.a 4 | *.skel.h 5 | bpf/ 6 | perfetto_sdk/ 7 | cscope.* 8 | tags 9 | sched-analyzer-pp/__pycache__ 10 | -------------------------------------------------------------------------------- /.gitmodules: -------------------------------------------------------------------------------- 1 | [submodule "libbpf"] 2 | path = libbpf 3 | url = https://github.com/libbpf/libbpf.git 4 | [submodule "perfetto"] 5 | path = perfetto 6 | url = https://android.googlesource.com/platform/external/perfetto 7 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | GNU GENERAL PUBLIC LICENSE 2 | Version 2, June 1991 3 | 4 | Copyright (C) 1989, 1991 Free Software Foundation, Inc., 5 | 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA 6 | Everyone is permitted to copy and distribute verbatim copies 7 | of this license document, but changing it is not allowed. 8 | 9 | Preamble 10 | 11 | The licenses for most software are designed to take away your 12 | freedom to share and change it. By contrast, the GNU General Public 13 | License is intended to guarantee your freedom to share and change free 14 | software--to make sure the software is free for all its users. This 15 | General Public License applies to most of the Free Software 16 | Foundation's software and to any other program whose authors commit to 17 | using it. (Some other Free Software Foundation software is covered by 18 | the GNU Lesser General Public License instead.) You can apply it to 19 | your programs, too. 20 | 21 | When we speak of free software, we are referring to freedom, not 22 | price. Our General Public Licenses are designed to make sure that you 23 | have the freedom to distribute copies of free software (and charge for 24 | this service if you wish), that you receive source code or can get it 25 | if you want it, that you can change the software or use pieces of it 26 | in new free programs; and that you know you can do these things. 27 | 28 | To protect your rights, we need to make restrictions that forbid 29 | anyone to deny you these rights or to ask you to surrender the rights. 30 | These restrictions translate to certain responsibilities for you if you 31 | distribute copies of the software, or if you modify it. 32 | 33 | For example, if you distribute copies of such a program, whether 34 | gratis or for a fee, you must give the recipients all the rights that 35 | you have. You must make sure that they, too, receive or can get the 36 | source code. And you must show them these terms so they know their 37 | rights. 38 | 39 | We protect your rights with two steps: (1) copyright the software, and 40 | (2) offer you this license which gives you legal permission to copy, 41 | distribute and/or modify the software. 42 | 43 | Also, for each author's protection and ours, we want to make certain 44 | that everyone understands that there is no warranty for this free 45 | software. If the software is modified by someone else and passed on, we 46 | want its recipients to know that what they have is not the original, so 47 | that any problems introduced by others will not reflect on the original 48 | authors' reputations. 49 | 50 | Finally, any free program is threatened constantly by software 51 | patents. We wish to avoid the danger that redistributors of a free 52 | program will individually obtain patent licenses, in effect making the 53 | program proprietary. To prevent this, we have made it clear that any 54 | patent must be licensed for everyone's free use or not licensed at all. 55 | 56 | The precise terms and conditions for copying, distribution and 57 | modification follow. 58 | 59 | GNU GENERAL PUBLIC LICENSE 60 | TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION 61 | 62 | 0. This License applies to any program or other work which contains 63 | a notice placed by the copyright holder saying it may be distributed 64 | under the terms of this General Public License. The "Program", below, 65 | refers to any such program or work, and a "work based on the Program" 66 | means either the Program or any derivative work under copyright law: 67 | that is to say, a work containing the Program or a portion of it, 68 | either verbatim or with modifications and/or translated into another 69 | language. (Hereinafter, translation is included without limitation in 70 | the term "modification".) Each licensee is addressed as "you". 71 | 72 | Activities other than copying, distribution and modification are not 73 | covered by this License; they are outside its scope. The act of 74 | running the Program is not restricted, and the output from the Program 75 | is covered only if its contents constitute a work based on the 76 | Program (independent of having been made by running the Program). 77 | Whether that is true depends on what the Program does. 78 | 79 | 1. You may copy and distribute verbatim copies of the Program's 80 | source code as you receive it, in any medium, provided that you 81 | conspicuously and appropriately publish on each copy an appropriate 82 | copyright notice and disclaimer of warranty; keep intact all the 83 | notices that refer to this License and to the absence of any warranty; 84 | and give any other recipients of the Program a copy of this License 85 | along with the Program. 86 | 87 | You may charge a fee for the physical act of transferring a copy, and 88 | you may at your option offer warranty protection in exchange for a fee. 89 | 90 | 2. You may modify your copy or copies of the Program or any portion 91 | of it, thus forming a work based on the Program, and copy and 92 | distribute such modifications or work under the terms of Section 1 93 | above, provided that you also meet all of these conditions: 94 | 95 | a) You must cause the modified files to carry prominent notices 96 | stating that you changed the files and the date of any change. 97 | 98 | b) You must cause any work that you distribute or publish, that in 99 | whole or in part contains or is derived from the Program or any 100 | part thereof, to be licensed as a whole at no charge to all third 101 | parties under the terms of this License. 102 | 103 | c) If the modified program normally reads commands interactively 104 | when run, you must cause it, when started running for such 105 | interactive use in the most ordinary way, to print or display an 106 | announcement including an appropriate copyright notice and a 107 | notice that there is no warranty (or else, saying that you provide 108 | a warranty) and that users may redistribute the program under 109 | these conditions, and telling the user how to view a copy of this 110 | License. (Exception: if the Program itself is interactive but 111 | does not normally print such an announcement, your work based on 112 | the Program is not required to print an announcement.) 113 | 114 | These requirements apply to the modified work as a whole. If 115 | identifiable sections of that work are not derived from the Program, 116 | and can be reasonably considered independent and separate works in 117 | themselves, then this License, and its terms, do not apply to those 118 | sections when you distribute them as separate works. But when you 119 | distribute the same sections as part of a whole which is a work based 120 | on the Program, the distribution of the whole must be on the terms of 121 | this License, whose permissions for other licensees extend to the 122 | entire whole, and thus to each and every part regardless of who wrote it. 123 | 124 | Thus, it is not the intent of this section to claim rights or contest 125 | your rights to work written entirely by you; rather, the intent is to 126 | exercise the right to control the distribution of derivative or 127 | collective works based on the Program. 128 | 129 | In addition, mere aggregation of another work not based on the Program 130 | with the Program (or with a work based on the Program) on a volume of 131 | a storage or distribution medium does not bring the other work under 132 | the scope of this License. 133 | 134 | 3. You may copy and distribute the Program (or a work based on it, 135 | under Section 2) in object code or executable form under the terms of 136 | Sections 1 and 2 above provided that you also do one of the following: 137 | 138 | a) Accompany it with the complete corresponding machine-readable 139 | source code, which must be distributed under the terms of Sections 140 | 1 and 2 above on a medium customarily used for software interchange; or, 141 | 142 | b) Accompany it with a written offer, valid for at least three 143 | years, to give any third party, for a charge no more than your 144 | cost of physically performing source distribution, a complete 145 | machine-readable copy of the corresponding source code, to be 146 | distributed under the terms of Sections 1 and 2 above on a medium 147 | customarily used for software interchange; or, 148 | 149 | c) Accompany it with the information you received as to the offer 150 | to distribute corresponding source code. (This alternative is 151 | allowed only for noncommercial distribution and only if you 152 | received the program in object code or executable form with such 153 | an offer, in accord with Subsection b above.) 154 | 155 | The source code for a work means the preferred form of the work for 156 | making modifications to it. For an executable work, complete source 157 | code means all the source code for all modules it contains, plus any 158 | associated interface definition files, plus the scripts used to 159 | control compilation and installation of the executable. However, as a 160 | special exception, the source code distributed need not include 161 | anything that is normally distributed (in either source or binary 162 | form) with the major components (compiler, kernel, and so on) of the 163 | operating system on which the executable runs, unless that component 164 | itself accompanies the executable. 165 | 166 | If distribution of executable or object code is made by offering 167 | access to copy from a designated place, then offering equivalent 168 | access to copy the source code from the same place counts as 169 | distribution of the source code, even though third parties are not 170 | compelled to copy the source along with the object code. 171 | 172 | 4. You may not copy, modify, sublicense, or distribute the Program 173 | except as expressly provided under this License. Any attempt 174 | otherwise to copy, modify, sublicense or distribute the Program is 175 | void, and will automatically terminate your rights under this License. 176 | However, parties who have received copies, or rights, from you under 177 | this License will not have their licenses terminated so long as such 178 | parties remain in full compliance. 179 | 180 | 5. You are not required to accept this License, since you have not 181 | signed it. However, nothing else grants you permission to modify or 182 | distribute the Program or its derivative works. These actions are 183 | prohibited by law if you do not accept this License. Therefore, by 184 | modifying or distributing the Program (or any work based on the 185 | Program), you indicate your acceptance of this License to do so, and 186 | all its terms and conditions for copying, distributing or modifying 187 | the Program or works based on it. 188 | 189 | 6. Each time you redistribute the Program (or any work based on the 190 | Program), the recipient automatically receives a license from the 191 | original licensor to copy, distribute or modify the Program subject to 192 | these terms and conditions. You may not impose any further 193 | restrictions on the recipients' exercise of the rights granted herein. 194 | You are not responsible for enforcing compliance by third parties to 195 | this License. 196 | 197 | 7. If, as a consequence of a court judgment or allegation of patent 198 | infringement or for any other reason (not limited to patent issues), 199 | conditions are imposed on you (whether by court order, agreement or 200 | otherwise) that contradict the conditions of this License, they do not 201 | excuse you from the conditions of this License. If you cannot 202 | distribute so as to satisfy simultaneously your obligations under this 203 | License and any other pertinent obligations, then as a consequence you 204 | may not distribute the Program at all. For example, if a patent 205 | license would not permit royalty-free redistribution of the Program by 206 | all those who receive copies directly or indirectly through you, then 207 | the only way you could satisfy both it and this License would be to 208 | refrain entirely from distribution of the Program. 209 | 210 | If any portion of this section is held invalid or unenforceable under 211 | any particular circumstance, the balance of the section is intended to 212 | apply and the section as a whole is intended to apply in other 213 | circumstances. 214 | 215 | It is not the purpose of this section to induce you to infringe any 216 | patents or other property right claims or to contest validity of any 217 | such claims; this section has the sole purpose of protecting the 218 | integrity of the free software distribution system, which is 219 | implemented by public license practices. Many people have made 220 | generous contributions to the wide range of software distributed 221 | through that system in reliance on consistent application of that 222 | system; it is up to the author/donor to decide if he or she is willing 223 | to distribute software through any other system and a licensee cannot 224 | impose that choice. 225 | 226 | This section is intended to make thoroughly clear what is believed to 227 | be a consequence of the rest of this License. 228 | 229 | 8. If the distribution and/or use of the Program is restricted in 230 | certain countries either by patents or by copyrighted interfaces, the 231 | original copyright holder who places the Program under this License 232 | may add an explicit geographical distribution limitation excluding 233 | those countries, so that distribution is permitted only in or among 234 | countries not thus excluded. In such case, this License incorporates 235 | the limitation as if written in the body of this License. 236 | 237 | 9. The Free Software Foundation may publish revised and/or new versions 238 | of the General Public License from time to time. Such new versions will 239 | be similar in spirit to the present version, but may differ in detail to 240 | address new problems or concerns. 241 | 242 | Each version is given a distinguishing version number. If the Program 243 | specifies a version number of this License which applies to it and "any 244 | later version", you have the option of following the terms and conditions 245 | either of that version or of any later version published by the Free 246 | Software Foundation. If the Program does not specify a version number of 247 | this License, you may choose any version ever published by the Free Software 248 | Foundation. 249 | 250 | 10. If you wish to incorporate parts of the Program into other free 251 | programs whose distribution conditions are different, write to the author 252 | to ask for permission. For software which is copyrighted by the Free 253 | Software Foundation, write to the Free Software Foundation; we sometimes 254 | make exceptions for this. Our decision will be guided by the two goals 255 | of preserving the free status of all derivatives of our free software and 256 | of promoting the sharing and reuse of software generally. 257 | 258 | NO WARRANTY 259 | 260 | 11. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY 261 | FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN 262 | OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES 263 | PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED 264 | OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF 265 | MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS 266 | TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE 267 | PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, 268 | REPAIR OR CORRECTION. 269 | 270 | 12. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING 271 | WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR 272 | REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, 273 | INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING 274 | OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED 275 | TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY 276 | YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER 277 | PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE 278 | POSSIBILITY OF SUCH DAMAGES. 279 | 280 | END OF TERMS AND CONDITIONS 281 | 282 | How to Apply These Terms to Your New Programs 283 | 284 | If you develop a new program, and you want it to be of the greatest 285 | possible use to the public, the best way to achieve this is to make it 286 | free software which everyone can redistribute and change under these terms. 287 | 288 | To do so, attach the following notices to the program. It is safest 289 | to attach them to the start of each source file to most effectively 290 | convey the exclusion of warranty; and each file should have at least 291 | the "copyright" line and a pointer to where the full notice is found. 292 | 293 | 294 | Copyright (C) 295 | 296 | This program is free software; you can redistribute it and/or modify 297 | it under the terms of the GNU General Public License as published by 298 | the Free Software Foundation; either version 2 of the License, or 299 | (at your option) any later version. 300 | 301 | This program is distributed in the hope that it will be useful, 302 | but WITHOUT ANY WARRANTY; without even the implied warranty of 303 | MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 304 | GNU General Public License for more details. 305 | 306 | You should have received a copy of the GNU General Public License along 307 | with this program; if not, write to the Free Software Foundation, Inc., 308 | 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. 309 | 310 | Also add information on how to contact you by electronic and paper mail. 311 | 312 | If the program is interactive, make it output a short notice like this 313 | when it starts in an interactive mode: 314 | 315 | Gnomovision version 69, Copyright (C) year name of author 316 | Gnomovision comes with ABSOLUTELY NO WARRANTY; for details type `show w'. 317 | This is free software, and you are welcome to redistribute it 318 | under certain conditions; type `show c' for details. 319 | 320 | The hypothetical commands `show w' and `show c' should show the appropriate 321 | parts of the General Public License. Of course, the commands you use may 322 | be called something other than `show w' and `show c'; they could even be 323 | mouse-clicks or menu items--whatever suits your program. 324 | 325 | You should also get your employer (if you work as a programmer) or your 326 | school, if any, to sign a "copyright disclaimer" for the program, if 327 | necessary. Here is a sample; alter the names: 328 | 329 | Yoyodyne, Inc., hereby disclaims all copyright interest in the program 330 | `Gnomovision' (which makes passes at compilers) written by James Hacker. 331 | 332 | , 1 April 1989 333 | Ty Coon, President of Vice 334 | 335 | This General Public License does not permit incorporating your program into 336 | proprietary programs. If your program is a subroutine library, you may 337 | consider it more useful to permit linking proprietary applications with the 338 | library. If this is what you want to do, use the GNU Lesser General 339 | Public License instead of this License. 340 | -------------------------------------------------------------------------------- /Makefile: -------------------------------------------------------------------------------- 1 | SHELL=/bin/bash 2 | CROSS_COMPILE ?= 3 | CLANG ?= clang 4 | STRIP ?= llvm-strip 5 | BPFTOOL ?= bpftool 6 | 7 | VERSION=$(shell git describe --tags) 8 | 9 | include cross_compile.mk 10 | 11 | LIBBPF_SRC ?= $(abspath libbpf/src) 12 | PERFETTO_SRC ?= $(abspath perfetto/sdk) 13 | 14 | CFLAGS := -g -O2 -Wall -DSA_VERSION=$(VERSION) 15 | CFLAGS_BPF := $(CFLAGS) -target bpf -D__TARGET_ARCH_$(ARCH) -D__SA_BPF_BUILD 16 | LDFLAGS := -lelf -lz -lpthread 17 | 18 | SCHED_ANALYZER := sched-analyzer 19 | 20 | VMLINUX_H := vmlinux.h 21 | VMLINUX ?= /sys/kernel/btf/vmlinux 22 | 23 | LIBBPF_DIR := $(abspath bpf) 24 | LIBBPF_OBJ := $(LIBBPF_DIR)/usr/lib64/libbpf.a 25 | LIBBPF_INCLUDE := -I$(abspath bpf/usr/include) 26 | 27 | PERFETTO_DIR := $(abspath perfetto_sdk) 28 | PERFETTO_OBJ := $(PERFETTO_DIR)/libperfetto.a 29 | PERFETTO_INCLUDE := -I$(abspath $(PERFETTO_SRC)) 30 | 31 | SRC := sched-analyzer.c parse_argp.c parse_kallsyms.c 32 | OBJS :=$(subst .c,.o,$(SRC)) 33 | 34 | SRC_BPF := $(wildcard *.bpf.c) 35 | OBJS_BPF := $(subst .bpf.c,.bpf.o,$(SRC_BPF)) 36 | SKEL_BPF := $(subst .bpf.c,.skel.h,$(SRC_BPF)) 37 | 38 | SRC_PERFETTO := $(PERFETTO_SRC)/perfetto.cc 39 | OBJS_PERFETETO := $(PERFETTO_DIR)/$(subst .cc,.o,$(notdir $(SRC_PERFETTO))) 40 | 41 | SRC_PERFETTO_WRAPPER := perfetto_wrapper.cc 42 | OBJS_PERFETETO_WRAPPER := $(subst .cc,.o,$(notdir $(SRC_PERFETTO_WRAPPER))) 43 | 44 | INCLUDES := $(LIBBPF_INCLUDE) $(PERFETTO_INCLUDE) 45 | LDFLAGS := $(LIBBPF_OBJ) $(PERFETTO_OBJ) $(LDFLAGS) 46 | 47 | ifneq ($(STATIC),) 48 | LDFLAGS := $(LDFLAGS) $(shell [ $$(find /usr/lib -name libzstd.a | grep .) ] && echo -lzstd) 49 | LDFLAGS := $(LDFLAGS) -static 50 | endif 51 | 52 | ifneq ($(DEBUG),) 53 | CFLAGS := $(CFLAGS) -DDEBUG 54 | CFLAGS_BPF := $(CFLAGS_BPF) -DDEBUG 55 | endif 56 | 57 | all: $(SCHED_ANALYZER) 58 | 59 | $(OBJS_PERFETETO): $(SRC_PERFETTO) 60 | git submodule init 61 | git submodule update 62 | mkdir -p $(PERFETTO_DIR) 63 | $(CXX) $(CFLAGS) -c $^ -std=c++17 -o $@ 64 | 65 | $(OBJS_PERFETETO_WRAPPER): $(SRC_PERFETTO_WRAPPER) 66 | $(CXX) $(PERFETTO_INCLUDE) $(CFLAGS) -c $^ -std=c++17 -o $@ 67 | 68 | $(PERFETTO_OBJ): $(OBJS_PERFETETO) $(OBJS_PERFETETO_WRAPPER) 69 | $(AR) crf $@ $^ 70 | 71 | $(VMLINUX_H): 72 | ($(BPFTOOL) btf dump file $(VMLINUX) format c > $@) || (rm $@ && exit 1) 73 | 74 | $(LIBBPF_OBJ): 75 | git submodule init 76 | git submodule update 77 | $(MAKE) -C $(LIBBPF_SRC) CC=$(CC) BUILD_STATIC_ONLY=1 DESTDIR=$(LIBBPF_DIR) install 78 | 79 | %.bpf.o: %.bpf.c $(VMLINUX_H) $(LIBBPF_OBJ) 80 | $(CLANG) $(CFLAGS_BPF) $(INCLUDES) -c $< -o $@ 81 | $(STRIP) -g $@ 82 | 83 | %.skel.h: %.bpf.o 84 | $(BPFTOOL) gen skeleton $< > $@ 85 | 86 | %.o: %.c 87 | $(CC) $(CFLAGS) $(INCLUDES) -c $< -o $@ 88 | 89 | $(OBJS): $(OBJS_BPF) $(SKEL_BPF) $(PERFETTO_OBJ) 90 | 91 | $(SCHED_ANALYZER): $(OBJS) 92 | $(CXX) $(CFLAGS) $(INCLUDES) $(filter %.o,$^) $(LDFLAGS) -o $@ 93 | 94 | package: $(SCHED_ANALYZER) 95 | tar cfz $(SCHED_ANALYZER)-$(ARCH)-$(VERSION)$(shell [ "$(STATIC)x" != "x" ] && echo "-static").tar.gz $(SCHED_ANALYZER) 96 | 97 | package-pp: 98 | git archive -o sched-analyzer-pp-$(VERSION).tar.gz $(VERSION) sched-analyzer-pp/ 99 | 100 | release: 101 | [ "$(shell ls | grep $(SCHED_ANALYZER).*.tar.gz)x" == "x" ] || (echo "Release file found, clean then try again" && exit 1) 102 | $(MAKE) clobber 103 | $(MAKE) ARCH=x86 package 104 | $(MAKE) clean 105 | $(MAKE) ARCH=x86 STATIC=1 package 106 | $(MAKE) clobber 107 | $(MAKE) ARCH=arm64 package 108 | $(MAKE) clean 109 | $(MAKE) ARCH=arm64 STATIC=1 package 110 | $(MAKE) package-pp 111 | 112 | static: 113 | $(MAKE) STATIC=1 114 | 115 | debug: 116 | $(MAKE) DEBUG=1 117 | 118 | clean: 119 | rm -rf $(SCHED_ANALYZER) *.o *.skel.h 120 | 121 | clobber: clean 122 | $(MAKE) -C $(LIBBPF_SRC) clean 123 | rm -rf $(LIBBPF_DIR) $(PERFETTO_DIR) 124 | 125 | help: 126 | @echo "Following build targets are available:" 127 | @echo "" 128 | @echo " static: Create statically linked binary" 129 | @echo " debug: Create a debug build which contains verbose debug prints" 130 | @echo " clean: Clean sched-analyzer, but not dependent libraries" 131 | @echo " clobber: Clean everything" 132 | @echo "" 133 | @echo "Cross compile:" 134 | @echo "" 135 | @echo " make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu-" 136 | @echo "" 137 | @echo " You can only specifiy the ARCH and we will try to guess the correct gcc CROSS_COMPILE to use" 138 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # sched-analyzer 2 | 3 | BPF CO-RE based sched-analyzer 4 | 5 | This is a personal pet project and not affiliated with any employer 6 | or organization. 7 | 8 | sched-analyzer connects to various points in the kernel to extract internal 9 | info and produce perfetto trace with additional collected events/tracks. 10 | Perfetto has integration with python for usage with libraries like pandas for 11 | more sophisticated post-processing option. 12 | 13 | Each event recorded by sched-analyzer is processed in its own thread to ensure 14 | each BPF ringbuffer is emptied in parallel and reduce the chance of overflowing 15 | any of them and potentially lose data. 16 | 17 | Since we peek inside kernel internals which are not ABI, there's no guarantee 18 | this will work on every kernel. Or won't silently fail if for instance some 19 | arguments to the one of the tracepoints we attach to changes. 20 | 21 | ## Goal 22 | 23 | The BPF backend, `sched-analyzer`, will collect data and generate perfetto 24 | events for analysis and optionally additional post processing via python. 25 | 26 | ## Data collected 27 | 28 | * load_avg and runnable_avg of FAIR at runqueue level 29 | * util_avg of FAIR, RT, DL, IRQ and thermal pressure at runqueue level 30 | * load_avg, runnable_avg and util_avg of tasks running 31 | * uclamped util_avg of CPUs and tasks: clamp(util_avg, uclamp_min, uclamp_max) 32 | * util_est at runqueue level and of tasks 33 | * Number of tasks running for every runqueue 34 | * Track cpu_idle and cpu_idle_miss events 35 | * Track load balance entry/exit and some related info (Experimental) 36 | * Track IPI related info (Experimental) 37 | * Collect hard and soft irq entry/exit data (perfetto builtin functionality) 38 | * Filter tasks per pid or comm 39 | 40 | ## Planned work 41 | 42 | * Trace wake up path and why a task placement decision was made 43 | * Better tracing of load balancer to understand when it kicks and what it 44 | performs when it runs 45 | * Trace schedutil and its decision to select a frequency and when it's rate 46 | limited 47 | * Trace TEO idle governor and its decision making process 48 | * Add more python post processing tools to summarize task placement histogram 49 | for a sepcifc task(s) and residency of various PELT signals 50 | * Add more python post processing tools to summarize softirq residencies and CPU 51 | histogram 52 | * Track PELT at cgroup level (cfs_rq) 53 | 54 | 55 | # Requirements 56 | 57 | ``` 58 | sudo apt install linux-tools-$(uname -r) git clang llvm libelf1 zlib1g libelf-dev bpftool 59 | ``` 60 | 61 | Download latest release of perfetto from [github](https://github.com/google/perfetto/releases/) 62 | 63 | ### BTF 64 | 65 | You need a kernel compiled with BTF to Compile and Run. 66 | 67 | Required kernel config to get BTF: 68 | 69 | - CONFIG_DEBUG_INFO_BTF=y 70 | 71 | # Build 72 | 73 | ``` 74 | make 75 | 76 | make help // for generic help and how to static build and cross compile 77 | ``` 78 | 79 | g++-9 and g++-10 fail to create a working static build - see this [issue](https://github.com/google/perfetto/issues/549). 80 | 81 | # Usage 82 | 83 | ## sched-analyzer 84 | 85 | ### perfetto mode 86 | 87 | First make sure perfetto is [downloaded](https://github.com/google/perfetto/releases/) and in your PATH. 88 | You need to run the following commands once after every reboot: 89 | 90 | ``` 91 | sudo tracebox traced --background 92 | sudo tracebox traced_probes --background 93 | 94 | ``` 95 | 96 | You can also create a systemd unit file. Ensure perfetto binaries are in a PATH 97 | accessible by root. 98 | 99 | ``` 100 | cat << EOF | sudo tee /usr/lib/systemd/system//perfetto_traced.service 101 | [Unit] 102 | Description=Perfetto traced 103 | 104 | [Service] 105 | ExecStart=tracebox traced 106 | 107 | [Install] 108 | WantedBy=default.target 109 | 110 | EOF 111 | 112 | cat << EOF | sudo tee /usr/lib/systemd/system//perfetto_traced_probes.service 113 | [Unit] 114 | Description=Perfetto traced_probes 115 | 116 | [Service] 117 | ExecStart=tracebox traced_probes 118 | 119 | [Install] 120 | WantedBy=default.target 121 | 122 | EOF 123 | 124 | systemctl daemon-reload 125 | systemctl enable perfetto_traced.service 126 | systemctl enable perfetto_traced_probes.service 127 | systemctl start perfetto_traced.service 128 | systemctl start perfetto_traced_probes.service 129 | ``` 130 | 131 | To collect data run: 132 | 133 | ``` 134 | sudo ./sched-analyzer --cpu_nr_running --util_avg 135 | ``` 136 | 137 | Press `CTRL+c` to stop. `sched-analyzer.perfetto-trace` will be in PWD that you 138 | can open in [ui.perfetto.dev](https://ui.perfetto.dev) 139 | 140 | The captured data are under sched-analyzer tab in perfetto, expand it to see 141 | them 142 | 143 | ![perfetto-screenshot](screenshots/sched-analyzer-screenshot.png?raw=true) 144 | 145 | ### csv mode 146 | 147 | csv mode was deprecated and is no longer supported on main branch. You can 148 | still find it in [csv-mode branch](https://github.com/qais-yousef/sched-analyzer/tree/csv-mode). 149 | 150 | ### Examples 151 | 152 | #### Collect PELT data filtering for tasks with specific name 153 | 154 | ``` 155 | sudo ./sched-analyzer --util_avg --util_est --comm firefox 156 | ``` 157 | 158 | The filtering is actually globbing for a task which contains the string. It 159 | doens't look for exact match. 160 | 161 | ![perfetto-screenshot](screenshots/sched-analyzer-screenshot-pelt-filtered.png?raw=true) 162 | 163 | #### Collect when an IPI happen with info about who triggered it 164 | 165 | ``` 166 | sudo ./sched-analyzer --ipi 167 | ``` 168 | 169 | When clicking on the IPI slice in perfetto, you'd be able to see extra info 170 | about who send the IPI from the new trace events. 171 | 172 | 0x1 is for the callsite function. 173 | 0x2 is for the callback function. 174 | 175 | ![perfetto-screenshot](screenshots/sched-analyzer-screenshot-ipi.png?raw=true) 176 | 177 | ## sched-analyzer-pp 178 | 179 | Post process the produced sched-analyzer.perfetto-trace to detect potential 180 | problems or summarizes some metrics. 181 | 182 | See [REAMDE.md](sched-analyzer-pp) for more details on setup and usage. 183 | 184 | ### Example 185 | 186 | 187 | ``` 188 | sched-analyzer-pp --freq --freq-residency --idle-residency --util-avg CPU0 --util-avg CPU4 sched-analyzer.perfetto-trace 189 | 190 | CPU0 Frequency 191 | ┌──────────────────────────────────────────────────────────────────────────────────────────────┐ 192 | 2.06┤ ▐▌ ▐▀▌ │ 193 | 1.70┤ ▐▙ ▐ ▌ │ 194 | 1.33┤ ▐█▖ ▖ ▟ ▌ │ 195 | │ █▌ ▌ ▌ ▌ │ 196 | 0.97┤ ▜█ █ ▌ ▌ ▐ │ 197 | 0.60┤ ▐█▄▄▄█▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▌ ▙▄▄▄▟▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄│ 198 | └┬─────────┬──────────┬─────────┬─────────┬──────────┬─────────┬─────────┬──────────┬─────────┬┘ 199 | 0.00 0.36 0.72 1.07 1.43 1.79 2.15 2.51 2.87 3.22 200 | CPU4 Frequency 201 | ┌──────────────────────────────────────────────────────────────────────────────────────────────┐ 202 | 3.20┤ ▝▌ ▟▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀│ 203 | 2.55┤ ▌ ▗▄▐▘ │ 204 | 1.90┤ ▌ ▌ ▐▐▛ │ 205 | │ ▌ ▙▖ ▐▐▌ │ 206 | 1.25┤ ▌ █▌ ▐▐▌ │ 207 | 0.60┤ ▜▄▄▄▄█▙▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▟▐▌ │ 208 | └┬─────────┬──────────┬─────────┬─────────┬──────────┬─────────┬─────────┬──────────┬─────────┬┘ 209 | 0.00 0.36 0.72 1.07 1.43 1.79 2.15 2.51 2.87 3.22 210 | 211 | ──────────────────────────────────── CPU0 Frequency residency % ──────────────────────────────────── 212 | 0.6 ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 94.35000000000001 213 | 0.972 ▇ 1.47 214 | 1.332 ▇ 1.31 215 | 1.704 ▇ 0.68 216 | 2.064 ▇▇ 2.19 217 | 218 | ──────────────────────────────────── CPU4 Frequency residency % ──────────────────────────────────── 219 | 0.6 ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 40.18 220 | 0.828 ▇ 0.44 221 | 1.5 ▇▇ 1.1400000000000001 222 | 1.956 ▇ 0.42 223 | 2.184 ▇ 0.63 224 | 2.388 ▇ 0.42 225 | 2.592 ▇ 0.8200000000000001 226 | 2.772 ▇ 0.42 227 | 2.988 ▇ 0.42 228 | 3.096 ▇ 0.42 229 | 3.204 ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 54.72 230 | 231 | ─────────────────────────────────────── CPU Idle Residency % ─────────────────────────────────────── 232 | ▇▇▇▇▇▇ 6.67 233 | 0 ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 56.21 234 | ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 37.12 235 | 236 | ▇ 1.1 237 | 1 ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 68.69 238 | ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 30.21 239 | 240 | 0.32 241 | 2 ▇▇▇▇▇▇▇▇▇▇ 10.93 242 | ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 88.76 243 | 244 | ▇ 1.1 245 | 3 ▇▇▇▇▇▇▇▇▇ 9.82 246 | ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 89.08 247 | 248 | 0.05 249 | 4 0.08 250 | ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 99.87 251 | 252 | 0.01 253 | 5 ▇ 1.19 254 | ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 98.8 255 | 256 | 0.01 257 | 6 ▇ 1.34 258 | ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 98.65 259 | 260 | 0.01 261 | 7 ▇ 1.31 262 | ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 98.68 263 | ───────────────────────────────────── ▇▇▇ -1.0 ▇▇▇ 0.0 ▇▇▇ 1.0 ───────────────────────────────────── 264 | CPU0 util_avg 265 | ┌─────────────────────────────────────────────────────────────────────────────────────────────┐ 266 | 426.0┤ ▗█ │ 267 | 319.5┤ ▐▐ │ 268 | 213.0┤ ▟▐ │ 269 | │ ▐▖ ▙ ▌▐ │ 270 | 106.5┤ ▐▜▖ ▛▙ ▟▘▐ │ 271 | 0.0┤▄▟ ▀▜▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▌▝▀▙▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▟▘ ▐▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄│ 272 | └┬─────────┬─────────┬──────────┬─────────┬─────────┬─────────┬──────────┬─────────┬─────────┬┘ 273 | 0.00 0.36 0.72 1.07 1.43 1.79 2.15 2.51 2.87 3.22 274 | CPU4 util_avg 275 | ┌──────────────────────────────────────────────────────────────────────────────────────────────┐ 276 | 1024┤ ▄▄▛▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▌│ 277 | 768┤▙ ▟▘ ▌│ 278 | 512┤▐▖ ▟▘ ▙│ 279 | │ ▙ ▌ │ 280 | 256┤ ▝▙▄ ▌ │ 281 | 0┤ ▝▜▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▌ │ 282 | └┬─────────┬──────────┬─────────┬─────────┬──────────┬─────────┬─────────┬──────────┬─────────┬┘ 283 | 0.00 0.36 0.72 1.07 1.43 1.79 2.15 2.51 2.87 3.22 284 | ``` 285 | -------------------------------------------------------------------------------- /cross_compile.mk: -------------------------------------------------------------------------------- 1 | guess_arch := $(shell uname -m | \ 2 | sed 's/x86_64/x86/' | \ 3 | sed 's/aarch64/arm64/' | \ 4 | sed 's/ppc64le/powerpc/' | \ 5 | sed 's/mips.*/mips/' \ 6 | ) 7 | 8 | guess_cross_compile = $(shell echo $(1) | \ 9 | sed 's/x86/x86_64-linux-gnu-/' | \ 10 | sed 's/arm64/aarch64-linux-gnu-/' | \ 11 | sed 's/powerpc/powerpc64le-linux-gnu-/' | \ 12 | sed 's/mips/mips-linux-gnu-/' \ 13 | ) 14 | 15 | ARCH ?= $(call guess_arch) 16 | 17 | # If no CROSS_COMPILE was given and target ARCH is different from machinen's 18 | # ARCH, guess CROSS_COMPILE for the user 19 | ifeq ($(CROSS_COMPILE),) 20 | ifneq ($(ARCH), $(call guess_arch)) 21 | CROSS_COMPILE := $(call guess_cross_compile, $(ARCH)) 22 | CC := $(CROSS_COMPILE)gcc 23 | CXX := $(CROSS_COMPILE)$(CXX) 24 | AR := $(CROSS_COMPILE)$(AR) 25 | endif 26 | endif 27 | -------------------------------------------------------------------------------- /parse_argp.c: -------------------------------------------------------------------------------- 1 | /* SPDX-License-Identifier: GPL-2.0 */ 2 | /* Copyright (C) 2023 Qais Yousef */ 3 | #include 4 | #include 5 | 6 | #include "parse_argp.h" 7 | 8 | #define XSTR(x) STR(x) 9 | #define STR(x) #x 10 | 11 | const char *argp_program_version = "sched-analyzer " XSTR(SA_VERSION); 12 | const char *argp_program_bug_address = ""; 13 | 14 | static char doc[] = 15 | "Extract scheduer data using BPF and emit them into perfetto as track events"; 16 | 17 | struct sa_opts sa_opts = { 18 | /* perfetto opts */ 19 | .system = true, 20 | .app = false, 21 | /* controls */ 22 | .output = "sched-analyzer.perfetto-trace", 23 | .output_path = NULL, 24 | .max_size = 250 * 1024 * 1024, /* 250MiB */ 25 | .num_ftrace_event = 0, 26 | .num_atrace_cat = 0, 27 | .num_function_graph = 0, 28 | .num_function_filter = 0, 29 | .ftrace_event = { 0 }, 30 | .atrace_cat = { 0 }, 31 | .function_graph = { 0 }, 32 | .function_filter = { 0 }, 33 | /* events */ 34 | .load_avg_cpu = false, 35 | .runnable_avg_cpu = false, 36 | .util_avg_cpu = false, 37 | .load_avg_task = false, 38 | .runnable_avg_task = false, 39 | .util_avg_cpu = false, 40 | .util_avg_task = false, 41 | .util_avg_rt = false, 42 | .util_avg_dl = false, 43 | .util_avg_irq = false, 44 | .load_avg_thermal = false, 45 | .util_est_cpu = false, 46 | .util_est_task = false, 47 | .cpu_nr_running = false, 48 | .cpu_freq = false, 49 | .cpu_idle = false, 50 | .softirq = false, 51 | .sched_switch = false, 52 | .load_balance = false, 53 | .ipi = false, 54 | .irq = false, 55 | /* filters */ 56 | .num_pids = 0, 57 | .num_comms = 0, 58 | .pid = { 0 }, 59 | .comm = { { 0 } }, 60 | }; 61 | 62 | enum sa_opts_flags { 63 | OPT_DUMMY_START = 0x80, 64 | 65 | /* perfetto opts */ 66 | OPT_SYSTEM, 67 | OPT_APP, 68 | 69 | /* controls */ 70 | OPT_OUTPUT, 71 | OPT_OUTPUT_PATH, 72 | OPT_MAX_SIZE, 73 | OPT_FTRACE_EVENT, 74 | OPT_ATRACE_CAT, 75 | OPT_FUNCTION_GRAPH, 76 | OPT_FUNCTION_FILTER, 77 | 78 | /* events */ 79 | OPT_LOAD_AVG, 80 | OPT_RUNNABLE_AVG, 81 | OPT_UTIL_AVG, 82 | OPT_LOAD_AVG_CPU, 83 | OPT_RUNNABLE_AVG_CPU, 84 | OPT_UTIL_AVG_CPU, 85 | OPT_LOAD_AVG_TASK, 86 | OPT_RUNNABLE_AVG_TASK, 87 | OPT_UTIL_AVG_TASK, 88 | OPT_UTIL_AVG_RT, 89 | OPT_UTIL_AVG_DL, 90 | OPT_UTIL_AVG_IRQ, 91 | OPT_LOAD_AVG_THERMAL, 92 | OPT_UTIL_EST, 93 | OPT_UTIL_EST_CPU, 94 | OPT_UTIL_EST_TASK, 95 | OPT_CPU_NR_RUNNING, 96 | OPT_CPU_IDLE, 97 | OPT_LOAD_BALANCE, 98 | OPT_IPI, 99 | OPT_IRQ, 100 | 101 | /* filters */ 102 | OPT_FILTER_PID, 103 | OPT_FILTER_COMM, 104 | }; 105 | 106 | static const struct argp_option options[] = { 107 | /* perfetto opts */ 108 | { "system", OPT_SYSTEM, 0, 0, "Collect system wide data, requires traced and traced_probes to be running (default)." }, 109 | { "app", OPT_APP, 0, 0, "Collect only data generated by this app. Runs standalone without external dependencies on traced." }, 110 | /* controls */ 111 | { "output", OPT_OUTPUT, "FILE", 0, "Filename of the perfetto-trace file to produce." }, 112 | { "output_path", OPT_OUTPUT_PATH, "PATH", 0, "Path to store perfetto-trace. PWD by default for perfetto." }, 113 | { "max_size", OPT_MAX_SIZE, "SIZE(MiB)", 0, "Maximum size of perfetto file to produce, 250MiB by default." }, 114 | { "ftrace_event", OPT_FTRACE_EVENT, "FTRACE_EVENT", 0, "Add ftrace event to the captured data. Repeat for each event to add." }, 115 | /* events */ 116 | { "atrace_cat", OPT_ATRACE_CAT, "ATRACE_CATEGORY", 0, "Perfetto atrace category to add to perfetto config. Repeat for each category to add." }, 117 | { "function_graph", OPT_FUNCTION_GRAPH, "FUNCTION", 0, "Trace function call graph for a kernel FUNCTION. Based on ftrace function graph functionality. Repeat for each function to graph." }, 118 | { "function_filter", OPT_FUNCTION_FILTER, "FUNCTION", 0, "Filter the function call for a kernel FUNCTION. Based on ftrace function filter functionality. Repeat for each function to filter." }, 119 | /* events */ 120 | { "load_avg", OPT_LOAD_AVG, 0, 0, "Collect load_avg for CPU, tasks and thermal." }, 121 | { "runnable_avg", OPT_RUNNABLE_AVG, 0, 0, "Collect runnable_avg for CPU and tasks." }, 122 | { "util_avg", OPT_UTIL_AVG, 0, 0, "Collect util_avg for CPU, tasks, irq, dl and rt." }, 123 | { "load_avg_cpu", OPT_LOAD_AVG_CPU, 0, 0, "Collect load_avg for CPU." }, 124 | { "runnable_avg_cpu", OPT_RUNNABLE_AVG_CPU, 0, 0, "Collect runnable_avg for CPU." }, 125 | { "util_avg_cpu", OPT_UTIL_AVG_CPU, 0, 0, "Collect util_avg for CPU." }, 126 | { "load_avg_task", OPT_LOAD_AVG_TASK, 0, 0, "Collect load_avg for tasks." }, 127 | { "runnable_avg_task", OPT_RUNNABLE_AVG_TASK, 0, 0, "Collect runnable_avg for tasks." }, 128 | { "util_avg_task", OPT_UTIL_AVG_TASK, 0, 0, "Collect util_avg for tasks." }, 129 | { "util_avg_rt", OPT_UTIL_AVG_RT, 0, 0, "Collect util_avg for rt." }, 130 | { "util_avg_dl", OPT_UTIL_AVG_DL, 0, 0, "Collect util_avg for dl." }, 131 | { "util_avg_irq", OPT_UTIL_AVG_IRQ, 0, 0, "Collect util_avg for irq." }, 132 | { "load_avg_thermal", OPT_LOAD_AVG_THERMAL, 0, 0, "Collect load_avg for thermal pressure." }, 133 | { "util_est", OPT_UTIL_EST, 0, 0, "Collect util_est for CPU and tasks." }, 134 | { "util_est_cpu", OPT_UTIL_EST_CPU, 0, 0, "Collect util_est for CPU." }, 135 | { "util_est_task", OPT_UTIL_EST_TASK, 0, 0, "Collect util_est for tasks." }, 136 | { "cpu_nr_running", OPT_CPU_NR_RUNNING, 0, 0, "Collect nr_running tasks for each CPU." }, 137 | { "cpu_idle", OPT_CPU_IDLE, 0, 0, "Collect info about cpu idle states for each CPU." }, 138 | { "load_balance", OPT_LOAD_BALANCE, 0, 0, "Collect load balance related info." }, 139 | { "ipi", OPT_IPI, 0, 0, "Collect ipi related info." }, 140 | { "irq", OPT_IRQ, 0, 0, "Enable perfetto irq atrace category." }, 141 | /* filters */ 142 | { "pid", OPT_FILTER_PID, "PID", 0, "Collect data for task match pid only. Can be provided multiple times." }, 143 | { "comm", OPT_FILTER_COMM, "COMM", 0, "Collect data for tasks that contain comm only. Can be provided multiple times." }, 144 | { 0 }, 145 | }; 146 | 147 | static error_t parse_arg(int key, char *arg, struct argp_state *state) 148 | { 149 | char *end_ptr; 150 | 151 | switch (key) { 152 | /* perfetto opts */ 153 | case OPT_SYSTEM: 154 | sa_opts.system = true; 155 | sa_opts.app = false; 156 | break; 157 | case OPT_APP: 158 | sa_opts.system = false; 159 | sa_opts.app = true; 160 | break; 161 | /* controls */ 162 | case OPT_OUTPUT: 163 | sa_opts.output = arg; 164 | break; 165 | case OPT_OUTPUT_PATH: 166 | sa_opts.output_path = arg; 167 | break; 168 | case OPT_MAX_SIZE: 169 | errno = 0; 170 | sa_opts.max_size = strtol(arg, &end_ptr, 0) * 1024 * 1024; 171 | if (errno != 0) { 172 | perror("Unsupported max_size value\n"); 173 | return errno; 174 | } 175 | if (end_ptr == arg) { 176 | fprintf(stderr, "max_size: no digits were found\n"); 177 | argp_usage(state); 178 | return -EINVAL; 179 | } 180 | break; 181 | case OPT_FTRACE_EVENT: 182 | sa_opts.ftrace_event[sa_opts.num_ftrace_event] = arg; 183 | sa_opts.num_ftrace_event++; 184 | break; 185 | case OPT_ATRACE_CAT: 186 | sa_opts.atrace_cat[sa_opts.num_atrace_cat] = arg; 187 | sa_opts.num_atrace_cat++; 188 | break; 189 | case OPT_FUNCTION_GRAPH: 190 | sa_opts.function_graph[sa_opts.num_function_graph] = arg; 191 | sa_opts.num_function_graph++; 192 | break; 193 | case OPT_FUNCTION_FILTER: 194 | sa_opts.function_filter[sa_opts.num_function_filter] = arg; 195 | sa_opts.num_function_filter++; 196 | break; 197 | /* events */ 198 | case OPT_LOAD_AVG: 199 | sa_opts.load_avg_cpu = true; 200 | sa_opts.load_avg_task = true; 201 | sa_opts.load_avg_thermal = true; 202 | break; 203 | case OPT_RUNNABLE_AVG: 204 | sa_opts.runnable_avg_cpu = true; 205 | sa_opts.runnable_avg_task = true; 206 | break; 207 | case OPT_UTIL_AVG: 208 | sa_opts.util_avg_cpu = true; 209 | sa_opts.util_avg_task = true; 210 | sa_opts.util_avg_rt = true; 211 | sa_opts.util_avg_dl = true; 212 | sa_opts.util_avg_irq = true; 213 | break; 214 | case OPT_LOAD_AVG_CPU: 215 | sa_opts.load_avg_cpu = true; 216 | break; 217 | case OPT_RUNNABLE_AVG_CPU: 218 | sa_opts.runnable_avg_cpu = true; 219 | break; 220 | case OPT_UTIL_AVG_CPU: 221 | sa_opts.util_avg_cpu = true; 222 | break; 223 | case OPT_LOAD_AVG_TASK: 224 | sa_opts.load_avg_task = true; 225 | break; 226 | case OPT_RUNNABLE_AVG_TASK: 227 | sa_opts.runnable_avg_task = true; 228 | break; 229 | case OPT_UTIL_AVG_TASK: 230 | sa_opts.util_avg_task = true; 231 | break; 232 | case OPT_UTIL_AVG_RT: 233 | sa_opts.util_avg_rt = true; 234 | break; 235 | case OPT_UTIL_AVG_DL: 236 | sa_opts.util_avg_dl = true; 237 | break; 238 | case OPT_UTIL_AVG_IRQ: 239 | sa_opts.util_avg_irq = true; 240 | break; 241 | case OPT_LOAD_AVG_THERMAL: 242 | sa_opts.load_avg_thermal = true; 243 | break; 244 | case OPT_UTIL_EST: 245 | sa_opts.util_est_cpu = true; 246 | sa_opts.util_est_task = true; 247 | break; 248 | case OPT_UTIL_EST_CPU: 249 | sa_opts.util_est_cpu = true; 250 | break; 251 | case OPT_UTIL_EST_TASK: 252 | sa_opts.util_est_task = true; 253 | break; 254 | case OPT_CPU_NR_RUNNING: 255 | sa_opts.cpu_nr_running = true; 256 | break; 257 | case OPT_CPU_IDLE: 258 | sa_opts.cpu_idle = true; 259 | break; 260 | case OPT_LOAD_BALANCE: 261 | sa_opts.load_balance = true; 262 | break; 263 | case OPT_IPI: 264 | sa_opts.ipi = true; 265 | break; 266 | case OPT_IRQ: 267 | sa_opts.irq = true; 268 | break; 269 | case OPT_FILTER_PID: 270 | if (sa_opts.num_pids >= MAX_FILTERS_NUM) { 271 | fprintf(stderr, "Can't accept more --pid, dropping %s\n", arg); 272 | break; 273 | } 274 | errno = 0; 275 | sa_opts.pid[sa_opts.num_pids] = strtol(arg, &end_ptr, 0); 276 | if (errno != 0) { 277 | perror("Unsupported pid value\n"); 278 | return errno; 279 | } 280 | if (end_ptr == arg) { 281 | fprintf(stderr, "pid: no digits were found\n"); 282 | argp_usage(state); 283 | return -EINVAL; 284 | } 285 | sa_opts.num_pids++; 286 | break; 287 | case OPT_FILTER_COMM: 288 | if (sa_opts.num_comms >= MAX_FILTERS_NUM) { 289 | fprintf(stderr, "Can't accept more --comm, dropping %s\n", arg); 290 | break; 291 | } 292 | strncpy(sa_opts.comm[sa_opts.num_comms], arg, TASK_COMM_LEN-1); 293 | sa_opts.comm[sa_opts.num_comms][TASK_COMM_LEN-1] = 0; 294 | sa_opts.num_comms++; 295 | break; 296 | case ARGP_KEY_ARG: 297 | argp_usage(state); 298 | break; 299 | case ARGP_KEY_END: 300 | break; 301 | default: 302 | return ARGP_ERR_UNKNOWN; 303 | } 304 | 305 | return 0; 306 | } 307 | 308 | const struct argp argp = { 309 | .options = options, 310 | .parser = parse_arg, 311 | .doc = doc, 312 | }; 313 | -------------------------------------------------------------------------------- /parse_argp.h: -------------------------------------------------------------------------------- 1 | /* SPDX-License-Identifier: GPL-2.0 */ 2 | /* Copyright (C) 2023 Qais Yousef */ 3 | #ifndef __PARSE_ARGS_H__ 4 | #define __PARSE_ARGS_H__ 5 | #ifndef __SA_BPF_BUILD 6 | #include 7 | #include 8 | #endif 9 | 10 | #define TASK_COMM_LEN 16 11 | #define MAX_FILTERS_NUM 128 12 | 13 | struct sa_opts { 14 | /* perfetto opts */ 15 | bool system; 16 | bool app; 17 | /* controls */ 18 | char *output; 19 | const char *output_path; 20 | long max_size; 21 | unsigned int num_ftrace_event; 22 | unsigned int num_atrace_cat; 23 | unsigned int num_function_graph; 24 | unsigned int num_function_filter; 25 | char *ftrace_event[MAX_FILTERS_NUM]; 26 | char *atrace_cat[MAX_FILTERS_NUM]; 27 | char *function_graph[MAX_FILTERS_NUM]; 28 | char *function_filter[MAX_FILTERS_NUM]; 29 | /* events */ 30 | bool load_avg_cpu; 31 | bool runnable_avg_cpu; 32 | bool util_avg_cpu; 33 | bool load_avg_task; 34 | bool runnable_avg_task; 35 | bool util_avg_task; 36 | bool util_avg_rt; 37 | bool util_avg_dl; 38 | bool util_avg_irq; 39 | bool load_avg_thermal; 40 | bool util_est_cpu; 41 | bool util_est_task; 42 | bool cpu_nr_running; 43 | bool cpu_freq; 44 | bool cpu_idle; 45 | bool softirq; 46 | bool sched_switch; 47 | bool load_balance; 48 | bool ipi; 49 | bool irq; 50 | /* filters */ 51 | unsigned int num_pids; 52 | unsigned int num_comms; 53 | pid_t pid[MAX_FILTERS_NUM]; 54 | char comm[MAX_FILTERS_NUM][TASK_COMM_LEN]; 55 | }; 56 | 57 | extern struct sa_opts sa_opts; 58 | extern const struct argp argp; 59 | 60 | #endif /* __PARSE_ARGS_H__ */ 61 | -------------------------------------------------------------------------------- /parse_kallsyms.c: -------------------------------------------------------------------------------- 1 | /* SPDX-License-Identifier: GPL-2.0 */ 2 | /* Copyright (C) 2024 Qais Yousef */ 3 | #include 4 | #include 5 | #include 6 | #include 7 | #include 8 | 9 | #define MAX_NUM_SYMBOLS 500000 10 | #define LINE_SIZE 256 11 | #define SYMBOL_LEN 128 12 | 13 | struct symbol { 14 | void *address; 15 | char symbol[SYMBOL_LEN]; 16 | }; 17 | 18 | static struct symbol symbols[MAX_NUM_SYMBOLS]; 19 | static bool ready; 20 | 21 | 22 | static int cmp(const void *a, const void *b) 23 | { 24 | struct symbol *i = (struct symbol *)a; 25 | struct symbol *j = (struct symbol *)b; 26 | 27 | return i->address - j->address; 28 | } 29 | 30 | void parse_kallsyms(void) 31 | { 32 | char line[LINE_SIZE] = { 0 }; 33 | unsigned int i = 0; 34 | FILE *fp; 35 | 36 | fp = fopen("/proc/sys/kernel/kptr_restrict", "r"); 37 | if (fp) { 38 | if (fgets(line, LINE_SIZE, fp)) { 39 | char *end_ptr; 40 | int val = strtol(line, &end_ptr, 0); 41 | if (val >= 2) { 42 | fprintf(stderr, "/proc/sys/kernel/kptr_restrict is %d, won't be able to parse kallsyms\n", val); 43 | fclose(fp); 44 | return; 45 | } 46 | } 47 | fclose(fp); 48 | } else { 49 | fprintf(stderr, "Failed to open /proc/sys/kernel/kptr_restrict, might fail to parse kallsyms\n"); 50 | } 51 | 52 | fp = fopen("/proc/kallsyms", "r"); 53 | if (!fp) { 54 | fprintf(stderr, "Failed to open /proc/kallsyms\n"); 55 | return; 56 | } 57 | 58 | while (fgets(line, LINE_SIZE, fp) && i < MAX_NUM_SYMBOLS) { 59 | char *end_ptr; 60 | char *token; 61 | 62 | /* Get address first */ 63 | token = strtok(line, " "); 64 | if (!token) { 65 | fprintf(stderr, "Error parsing kallsyms\n"); 66 | break; 67 | } 68 | 69 | errno = 0; 70 | symbols[i].address = (void *)strtoul(line, &end_ptr, 16); 71 | if (errno != 0) { 72 | perror("Unsupported address value\n"); 73 | break; 74 | } 75 | if (end_ptr == line) { 76 | fprintf(stderr, "address: no digits were found\n"); 77 | break; 78 | } 79 | 80 | /* Skip 'type' field */ 81 | token = strtok(NULL, " "); 82 | if (!token) { 83 | fprintf(stderr, "Error parsing kallsyms\n"); 84 | break; 85 | } 86 | 87 | /* Get symbol name */ 88 | token = strtok(NULL, " "); 89 | if (!token) { 90 | fprintf(stderr, "Error parsing kallsyms\n"); 91 | break; 92 | } 93 | 94 | strncpy(symbols[i].symbol, token, SYMBOL_LEN-1); 95 | symbols[i].symbol[SYMBOL_LEN-1] = '0'; 96 | i++; 97 | } 98 | 99 | fclose(fp); 100 | 101 | qsort(symbols, MAX_NUM_SYMBOLS, sizeof(struct symbol), cmp); 102 | 103 | ready = true; 104 | } 105 | 106 | static char *____find_kallsyms(unsigned int start, unsigned int end, void *address) 107 | { 108 | while (start <= end-1) { 109 | if (symbols[start].address <= address && 110 | symbols[start+1].address >= address) 111 | return symbols[start].symbol; 112 | start++; 113 | } 114 | 115 | fprintf(stderr, "Couldn't find_kallsyms() for address: %p\n", address); 116 | 117 | return NULL; 118 | } 119 | 120 | static char *__find_kallsyms(unsigned int start, unsigned int pivot, 121 | unsigned int end, void *address) 122 | { 123 | if (end - start <= 100) 124 | return ____find_kallsyms(start, end, address); 125 | 126 | if (address == symbols[pivot].address) 127 | return symbols[pivot].symbol; 128 | 129 | if (address > symbols[pivot].address) 130 | return __find_kallsyms(pivot, pivot+(end-pivot)/2, end, address); 131 | 132 | if (address < symbols[pivot].address) 133 | return __find_kallsyms(start, start+(pivot-start)/2, pivot, address); 134 | 135 | fprintf(stderr, "Reached unexpeted path while searching for address: %p\n", address); 136 | 137 | return NULL; 138 | } 139 | 140 | char *find_kallsyms(void *address) 141 | { 142 | if (!address || !ready) 143 | return NULL; 144 | 145 | return __find_kallsyms(0, MAX_NUM_SYMBOLS/2, MAX_NUM_SYMBOLS-1, address); 146 | } 147 | -------------------------------------------------------------------------------- /parse_kallsyms.h: -------------------------------------------------------------------------------- 1 | /* SPDX-License-Identifier: GPL-2.0 */ 2 | /* Copyright (C) 2024 Qais Yousef */ 3 | #ifndef __PARSE_KALLSYMS_H__ 4 | #define __PARSE_KALLSYMS_H__ 5 | 6 | void parse_kallsyms(void); 7 | char *find_kallsyms(void *address); 8 | 9 | #endif /* __PARSE_KALLSYMS_H__ */ 10 | -------------------------------------------------------------------------------- /perfetto_wrapper.cc: -------------------------------------------------------------------------------- 1 | /* SPDX-License-Identifier: GPL-2.0 */ 2 | /* Copyright (C) 2023 Qais Yousef */ 3 | #include 4 | #include 5 | #include 6 | #include 7 | #include 8 | #include 9 | 10 | #include "parse_argp.h" 11 | #include "sched-analyzer-events.h" 12 | 13 | PERFETTO_DEFINE_CATEGORIES( 14 | perfetto::Category("pelt-cpu").SetDescription("Track PELT at CPU level"), 15 | perfetto::Category("pelt-task").SetDescription("Track PELT at task level"), 16 | perfetto::Category("nr-running-cpu").SetDescription("Track number of tasks running on each CPU"), 17 | perfetto::Category("cpu-idle").SetDescription("Track cpu idle info for each CPU"), 18 | perfetto::Category("load-balance").SetDescription("Track load balance internals"), 19 | perfetto::Category("ipi").SetDescription("Track inter-processor interrupts"), 20 | ); 21 | 22 | PERFETTO_TRACK_EVENT_STATIC_STORAGE(); 23 | 24 | enum sched_analyzer_track_ids { 25 | SA_TRACK_ID_CPU_IDLE_MISS = 1, /* must start from none 0 */ 26 | SA_TRACK_ID_LOAD_BALANCE, 27 | SA_TRACK_ID_IPI, 28 | }; 29 | 30 | #define TRACK_SPACING 1000 31 | #define TRACK_ID(ID) (SA_TRACK_ID_##ID * TRACK_SPACING) 32 | 33 | #define FAKE_DURATION 10000 /* 10us */ 34 | 35 | 36 | extern "C" void init_perfetto(void) 37 | { 38 | const char *android_traced_prodcuer = "/dev/socket/traced_producer"; 39 | const char *android_traced_consumer = "/dev/socket/traced_consumer"; 40 | 41 | if (!access(android_traced_prodcuer, F_OK)) 42 | setenv("PERFETTO_PRODUCER_SOCK_NAME", "/dev/socket/traced_producer", 0); 43 | if (!access(android_traced_consumer, F_OK)) 44 | setenv("PERFETTO_CONSUMER_SOCK_NAME", "/dev/socket/traced_consumer", 0); 45 | 46 | perfetto::TracingInitArgs args; 47 | 48 | // The backends determine where trace events are recorded. You may select one 49 | // or more of: 50 | 51 | // 1) The in-process backend only records within the app itself. 52 | if (sa_opts.app) 53 | args.backends |= perfetto::kInProcessBackend; 54 | 55 | // 2) The system backend writes events into a system Perfetto daemon, 56 | // allowing merging app and system events (e.g., ftrace) on the same 57 | // timeline. Requires the Perfetto `traced` daemon to be running (e.g., 58 | // on Android Pie and newer). 59 | if (sa_opts.system) 60 | args.backends |= perfetto::kSystemBackend; 61 | 62 | args.shmem_size_hint_kb = 1024*100; // 100 MiB 63 | 64 | perfetto::Tracing::Initialize(args); 65 | perfetto::TrackEvent::Register(); 66 | } 67 | 68 | extern "C" void flush_perfetto(void) 69 | { 70 | perfetto::TrackEvent::Flush(); 71 | } 72 | 73 | static std::unique_ptr tracing_session; 74 | static int fd; 75 | 76 | extern "C" void start_perfetto_trace(void) 77 | { 78 | char buffer[256]; 79 | 80 | perfetto::TraceConfig cfg; 81 | perfetto::TraceConfig::BufferConfig* buf; 82 | buf = cfg.add_buffers(); 83 | buf->set_size_kb(1024*100); // Record up to 100 MiB. 84 | buf->set_fill_policy(perfetto::TraceConfig::BufferConfig::RING_BUFFER); 85 | buf = cfg.add_buffers(); 86 | buf->set_size_kb(1024*100); // Record up to 100 MiB. 87 | buf->set_fill_policy(perfetto::TraceConfig::BufferConfig::RING_BUFFER); 88 | cfg.set_duration_ms(3600000); 89 | cfg.set_max_file_size_bytes(sa_opts.max_size); 90 | cfg.set_unique_session_name("sched-analyzer"); 91 | cfg.set_write_into_file(true); 92 | cfg.set_file_write_period_ms(1000); 93 | cfg.set_flush_period_ms(30000); 94 | cfg.set_enable_extra_guardrails(false); 95 | cfg.set_notify_traceur(true); 96 | cfg.mutable_incremental_state_config()->set_clear_period_ms(15000); 97 | 98 | /* Track Events Data Source */ 99 | perfetto::protos::gen::TrackEventConfig track_event_cfg; 100 | track_event_cfg.add_enabled_categories("sched-analyzer"); 101 | 102 | auto *te_ds_cfg = cfg.add_data_sources()->mutable_config(); 103 | te_ds_cfg->set_name("track_event"); 104 | te_ds_cfg->set_track_event_config_raw(track_event_cfg.SerializeAsString()); 105 | 106 | /* Android frametimeline */ 107 | auto *frametl_ds_cfg = cfg.add_data_sources()->mutable_config(); 108 | frametl_ds_cfg->set_name("android.surfaceflinger.frametimeline"); 109 | 110 | /* Ftrace Data Source */ 111 | perfetto::protos::gen::FtraceConfig ftrace_cfg; 112 | ftrace_cfg.add_ftrace_events("sched/sched_switch"); 113 | ftrace_cfg.add_ftrace_events("sched/sched_wakeup_new"); 114 | ftrace_cfg.add_ftrace_events("sched/sched_waking"); 115 | ftrace_cfg.add_ftrace_events("sched/sched_process_exit"); 116 | ftrace_cfg.add_ftrace_events("sched/sched_process_free"); 117 | ftrace_cfg.add_ftrace_events("power/suspend_resume"); 118 | ftrace_cfg.add_ftrace_events("power/cpu_frequency"); 119 | ftrace_cfg.add_ftrace_events("power/cpu_idle"); 120 | ftrace_cfg.add_ftrace_events("task/task_newtask"); 121 | ftrace_cfg.add_ftrace_events("task/task_rename"); 122 | ftrace_cfg.add_ftrace_events("ftrace/print"); 123 | if (sa_opts.irq) 124 | ftrace_cfg.add_atrace_categories("irq"); 125 | 126 | for (unsigned int i = 0; i < sa_opts.num_ftrace_event; i++) 127 | ftrace_cfg.add_ftrace_events(sa_opts.ftrace_event[i]); 128 | 129 | for (unsigned int i = 0; i < sa_opts.num_atrace_cat; i++) 130 | ftrace_cfg.add_atrace_categories(sa_opts.atrace_cat[i]); 131 | 132 | if (sa_opts.num_function_graph || sa_opts.num_function_filter) { 133 | ftrace_cfg.set_symbolize_ksyms(true); 134 | ftrace_cfg.set_ksyms_mem_policy(perfetto::protos::gen::FtraceConfig::KSYMS_RETAIN); 135 | ftrace_cfg.set_enable_function_graph(true); 136 | for (unsigned int i = 0; i < sa_opts.num_function_graph; i++) 137 | ftrace_cfg.add_function_graph_roots(sa_opts.function_graph[i]); 138 | for (unsigned int i = 0; i < sa_opts.num_function_filter; i++) 139 | ftrace_cfg.add_function_filters(sa_opts.function_filter[i]); 140 | } 141 | 142 | ftrace_cfg.set_drain_period_ms(1000); 143 | 144 | auto *ft_ds_cfg = cfg.add_data_sources()->mutable_config(); 145 | ft_ds_cfg->set_name("linux.ftrace"); 146 | ft_ds_cfg->set_ftrace_config_raw(ftrace_cfg.SerializeAsString()); 147 | 148 | /* Process Stats Data Source */ 149 | perfetto::protos::gen::ProcessStatsConfig ps_cfg; 150 | ps_cfg.set_proc_stats_poll_ms(10000); 151 | ps_cfg.set_record_thread_names(true); 152 | 153 | auto *ps_ds_cfg = cfg.add_data_sources()->mutable_config(); 154 | ps_ds_cfg->set_name("linux.process_stats"); 155 | ps_ds_cfg->set_process_stats_config_raw(ps_cfg.SerializeAsString()); 156 | 157 | /* On Android traces can be saved on specific path only */ 158 | const char *android_traces_path = "/data/misc/perfetto-traces"; 159 | DIR *dir = opendir(android_traces_path); 160 | if (!sa_opts.output_path) { 161 | sa_opts.output_path = "."; 162 | if (dir) { 163 | sa_opts.output_path = android_traces_path; 164 | closedir(dir); 165 | } 166 | } 167 | 168 | snprintf(buffer, 256, "%s/%s", sa_opts.output_path, sa_opts.output); 169 | fd = open(buffer, O_RDWR | O_CREAT | O_TRUNC, 0644); 170 | if (fd < 0) { 171 | snprintf(buffer, 256, "Failed to create %s/%s", sa_opts.output_path, sa_opts.output); 172 | perror(buffer); 173 | return; 174 | } 175 | 176 | tracing_session = perfetto::Tracing::NewTrace(); 177 | tracing_session->Setup(cfg, fd); 178 | tracing_session->StartBlocking(); 179 | } 180 | 181 | extern "C" void stop_perfetto_trace(void) 182 | { 183 | if (fd < 0) 184 | return; 185 | 186 | tracing_session->StopBlocking(); 187 | close(fd); 188 | } 189 | 190 | extern "C" void trace_cpu_load_avg(uint64_t ts, int cpu, int value) 191 | { 192 | char track_name[32]; 193 | snprintf(track_name, sizeof(track_name), "CPU%d load_avg", cpu); 194 | 195 | TRACE_COUNTER("pelt-cpu", track_name, ts, value); 196 | } 197 | 198 | extern "C" void trace_cpu_runnable_avg(uint64_t ts, int cpu, int value) 199 | { 200 | char track_name[32]; 201 | snprintf(track_name, sizeof(track_name), "CPU%d runnable_avg", cpu); 202 | 203 | TRACE_COUNTER("pelt-cpu", track_name, ts, value); 204 | } 205 | 206 | extern "C" void trace_cpu_util_avg(uint64_t ts, int cpu, int value) 207 | { 208 | char track_name[32]; 209 | snprintf(track_name, sizeof(track_name), "CPU%d util_avg", cpu); 210 | 211 | TRACE_COUNTER("pelt-cpu", track_name, ts, value); 212 | } 213 | 214 | extern "C" void trace_cpu_uclamped_avg(uint64_t ts, int cpu, int value) 215 | { 216 | char track_name[32]; 217 | snprintf(track_name, sizeof(track_name), "CPU%d uclamped_avg", cpu); 218 | 219 | TRACE_COUNTER("pelt-cpu", track_name, ts, value); 220 | } 221 | 222 | extern "C" void trace_cpu_util_est_enqueued(uint64_t ts, int cpu, int value) 223 | { 224 | char track_name[32]; 225 | snprintf(track_name, sizeof(track_name), "CPU%d util_est.enqueued", cpu); 226 | 227 | TRACE_COUNTER("pelt-cpu", track_name, ts, value); 228 | } 229 | 230 | extern "C" void trace_cpu_util_avg_rt(uint64_t ts, int cpu, int value) 231 | { 232 | char track_name[32]; 233 | snprintf(track_name, sizeof(track_name), "CPU%d util_avg_rt", cpu); 234 | 235 | TRACE_COUNTER("pelt-cpu", track_name, ts, value); 236 | } 237 | 238 | extern "C" void trace_cpu_util_avg_dl(uint64_t ts, int cpu, int value) 239 | { 240 | char track_name[32]; 241 | snprintf(track_name, sizeof(track_name), "CPU%d util_avg_rt", cpu); 242 | 243 | TRACE_COUNTER("pelt-cpu", track_name, ts, value); 244 | } 245 | 246 | extern "C" void trace_cpu_util_avg_irq(uint64_t ts, int cpu, int value) 247 | { 248 | char track_name[32]; 249 | snprintf(track_name, sizeof(track_name), "CPU%d util_avg_rt", cpu); 250 | 251 | TRACE_COUNTER("pelt-cpu", track_name, ts, value); 252 | } 253 | 254 | extern "C" void trace_cpu_load_avg_thermal(uint64_t ts, int cpu, int value) 255 | { 256 | char track_name[32]; 257 | snprintf(track_name, sizeof(track_name), "CPU%d load_avg_thermal", cpu); 258 | 259 | TRACE_COUNTER("pelt-cpu", track_name, ts, value); 260 | } 261 | 262 | extern "C" void trace_task_load_avg(uint64_t ts, const char *name, int pid, int value) 263 | { 264 | char track_name[32]; 265 | snprintf(track_name, sizeof(track_name), "%s-%d load_avg", name, pid); 266 | 267 | TRACE_COUNTER("pelt-task", track_name, ts, value); 268 | } 269 | 270 | extern "C" void trace_task_runnable_avg(uint64_t ts, const char *name, int pid, int value) 271 | { 272 | char track_name[32]; 273 | snprintf(track_name, sizeof(track_name), "%s-%d runnable_avg", name, pid); 274 | 275 | TRACE_COUNTER("pelt-task", track_name, ts, value); 276 | } 277 | 278 | extern "C" void trace_task_util_avg(uint64_t ts, const char *name, int pid, int value) 279 | { 280 | char track_name[32]; 281 | snprintf(track_name, sizeof(track_name), "%s-%d util_avg", name, pid); 282 | 283 | TRACE_COUNTER("pelt-task", track_name, ts, value); 284 | } 285 | 286 | extern "C" void trace_task_uclamped_avg(uint64_t ts, const char *name, int pid, int value) 287 | { 288 | char track_name[32]; 289 | snprintf(track_name, sizeof(track_name), "%s-%d uclamped_avg", name, pid); 290 | 291 | TRACE_COUNTER("pelt-task", track_name, ts, value); 292 | } 293 | 294 | extern "C" void trace_task_util_est_enqueued(uint64_t ts, const char *name, int pid, int value) 295 | { 296 | char track_name[32]; 297 | snprintf(track_name, sizeof(track_name), "%s-%d util_est.enqueued", name, pid); 298 | 299 | TRACE_COUNTER("pelt-task", track_name, ts, value); 300 | } 301 | 302 | extern "C" void trace_task_util_est_ewma(uint64_t ts, const char *name, int pid, int value) 303 | { 304 | char track_name[32]; 305 | snprintf(track_name, sizeof(track_name), "%s-%d util_est.ewma", name, pid); 306 | 307 | TRACE_COUNTER("pelt-task", track_name, ts, value); 308 | } 309 | 310 | extern "C" void trace_cpu_nr_running(uint64_t ts, int cpu, int value) 311 | { 312 | char track_name[32]; 313 | snprintf(track_name, sizeof(track_name), "CPU%d nr_running", cpu); 314 | 315 | TRACE_COUNTER("nr-running-cpu", track_name, ts, value); 316 | } 317 | 318 | extern "C" void trace_cpu_idle(uint64_t ts, int cpu, int state) 319 | { 320 | char track_name[32]; 321 | snprintf(track_name, sizeof(track_name), "CPU%d idle_state", cpu); 322 | 323 | TRACE_COUNTER("cpu-idle", track_name, ts, state); 324 | } 325 | 326 | extern "C" void trace_cpu_idle_miss(uint64_t ts, int cpu, int state, int miss) 327 | { 328 | TRACE_EVENT("cpu-idle", "cpu_idle_miss", 329 | perfetto::Track(TRACK_ID(CPU_IDLE_MISS) + cpu), ts, 330 | "CPU", cpu, 331 | "STATE", state, "MISS", miss < 0 ? "below" : "above"); 332 | 333 | TRACE_EVENT_END("cpu-idle", 334 | perfetto::Track(TRACK_ID(CPU_IDLE_MISS) + cpu), 335 | ts + FAKE_DURATION); 336 | } 337 | 338 | extern "C" void trace_lb_entry(uint64_t ts, int this_cpu, int lb_cpu, char *phase) 339 | { 340 | TRACE_EVENT_BEGIN("load-balance", phase, 341 | perfetto::Track(TRACK_ID(LOAD_BALANCE) + this_cpu), 342 | ts, "CPU", lb_cpu); 343 | } 344 | 345 | extern "C" void trace_lb_exit(uint64_t ts, int this_cpu, int lb_cpu) 346 | { 347 | TRACE_EVENT_END("load-balance", 348 | perfetto::Track(TRACK_ID(LOAD_BALANCE) + this_cpu), ts); 349 | } 350 | 351 | extern "C" void trace_lb_sd_stats(uint64_t ts, struct lb_sd_stats *sd_stats) 352 | { 353 | char track_name[64]; 354 | int i; 355 | 356 | for (i = 0; i < MAX_SD_LEVELS; i++) { 357 | if (sd_stats->level[i] == -1 && !sd_stats->balance_interval[i]) 358 | break; 359 | 360 | snprintf(track_name, sizeof(track_name), "CPU%d.level%d.balance_interval", 361 | sd_stats->cpu, sd_stats->level[i]); 362 | 363 | TRACE_COUNTER("load-balance", track_name, ts, sd_stats->balance_interval[i]); 364 | } 365 | } 366 | 367 | extern "C" void trace_lb_overloaded(uint64_t ts, unsigned int value) 368 | { 369 | char track_name[32]; 370 | snprintf(track_name, sizeof(track_name), "rd.overloaded"); 371 | 372 | TRACE_COUNTER("load-balance", track_name, ts, value); 373 | } 374 | 375 | extern "C" void trace_lb_overutilized(uint64_t ts, unsigned int value) 376 | { 377 | char track_name[32]; 378 | snprintf(track_name, sizeof(track_name), "rd.overutilized"); 379 | 380 | TRACE_COUNTER("load-balance", track_name, ts, value); 381 | } 382 | 383 | extern "C" void trace_lb_misfit(uint64_t ts, int cpu, unsigned long misfit_task_load) 384 | { 385 | char track_name[32]; 386 | snprintf(track_name, sizeof(track_name), "CPU%d misfit_task_load", cpu); 387 | 388 | TRACE_COUNTER("load-balance", track_name, ts, misfit_task_load); 389 | } 390 | 391 | extern "C" void trace_ipi_send_cpu(uint64_t ts, int from_cpu, int target_cpu, 392 | char *callsite, void *callsitep, 393 | char *callback, void *callbackp) 394 | { 395 | TRACE_EVENT("ipi", "ipi_send_cpu", 396 | perfetto::Track(TRACK_ID(IPI) + from_cpu), ts, 397 | "FROM_CPU", from_cpu, 398 | "TARGET_CPU", target_cpu, 399 | callsite ? callsite : "CALLSITE", 400 | callsite ? (void *)1 : callsitep, 401 | callback ? callback : "CALLBACK", 402 | callback ? (void *)2 : callbackp); 403 | 404 | TRACE_EVENT_END("ipi", perfetto::Track(TRACK_ID(IPI) + from_cpu), 405 | ts + FAKE_DURATION); 406 | } 407 | 408 | #if 0 409 | extern "C" int main(int argc, char **argv) 410 | { 411 | init_perfetto(); 412 | /* wait_for_perfetto(); */ 413 | 414 | start_perfetto_trace(); 415 | 416 | for (int i = 0; i < 2000; i++) { 417 | trace_cpu_pelt(0, i % 300); 418 | usleep(1000); 419 | } 420 | 421 | stop_perfetto_trace(); 422 | 423 | /* flush_perfetto(); */ 424 | 425 | return 0; 426 | } 427 | #endif 428 | -------------------------------------------------------------------------------- /perfetto_wrapper.h: -------------------------------------------------------------------------------- 1 | /* SPDX-License-Identifier: GPL-2.0 */ 2 | /* Copyright (C) 2023 Qais Yousef */ 3 | struct lb_sd_stats; 4 | 5 | void init_perfetto(void); 6 | void flush_perfetto(void); 7 | void start_perfetto_trace(void); 8 | void stop_perfetto_trace(void); 9 | void trace_cpu_load_avg(uint64_t ts, int cpu, int value); 10 | void trace_cpu_runnable_avg(uint64_t ts, int cpu, int value); 11 | void trace_cpu_util_avg(uint64_t ts, int cpu, int value); 12 | void trace_cpu_uclamped_avg(uint64_t ts, int cpu, int value); 13 | void trace_cpu_util_est_enqueued(uint64_t ts, int cpu, int value); 14 | void trace_cpu_util_avg_rt(uint64_t ts, int cpu, int value); 15 | void trace_cpu_util_avg_dl(uint64_t ts, int cpu, int value); 16 | void trace_cpu_util_avg_irq(uint64_t ts, int cpu, int value); 17 | void trace_cpu_load_avg_thermal(uint64_t ts, int cpu, int value); 18 | void trace_task_load_avg(uint64_t ts, const char *name, int pid, int value); 19 | void trace_task_runnable_avg(uint64_t ts, const char *name, int pid, int value); 20 | void trace_task_util_avg(uint64_t ts, const char *name, int pid, int value); 21 | void trace_task_uclamped_avg(uint64_t ts, const char *name, int pid, int value); 22 | void trace_task_util_est_enqueued(uint64_t ts, const char *name, int pid, int value); 23 | void trace_task_util_est_ewma(uint64_t ts, const char *name, int pid, int value); 24 | void trace_cpu_nr_running(uint64_t ts, int cpu, int value); 25 | void trace_cpu_idle(uint64_t ts, int cpu, int state); 26 | void trace_cpu_idle_miss(uint64_t ts, int cpu, int state, int miss); 27 | void trace_lb_entry(uint64_t ts, int this_cpu, int lb_cpu, char *phase); 28 | void trace_lb_exit(uint64_t ts, int this_cpu, int lb_cpu); 29 | void trace_lb_sd_stats(uint64_t ts, struct lb_sd_stats *sd_stats); 30 | void trace_lb_overloaded(uint64_t ts, unsigned int value); 31 | void trace_lb_overutilized(uint64_t ts, unsigned int value); 32 | void trace_lb_misfit(uint64_t ts, int cpu, unsigned long misfit_task_load); 33 | void trace_ipi_send_cpu(uint64_t ts, int from_cpu, int target_cpu, 34 | char *callsite, void *callsitep, 35 | char *callback, void *callbackp); 36 | -------------------------------------------------------------------------------- /sched-analyzer-events.h: -------------------------------------------------------------------------------- 1 | /* SPDX-License-Identifier: GPL-2.0 */ 2 | /* Copyright (C) 2022 Qais Yousef */ 3 | #ifndef __SCHED_ANALYZER_EVENTS_H__ 4 | #define __SCHED_ANALYZER_EVENTS_H__ 5 | 6 | #define TASK_COMM_LEN 16 7 | #define PELT_TYPE_LEN 4 8 | 9 | static inline void copy_str(char *dst, char *src, unsigned int len) 10 | { 11 | unsigned int i; 12 | for (i = 0; i < len; i++) 13 | dst[i] = src[i]; 14 | } 15 | 16 | enum pelt_type { 17 | PELT_TYPE_CFS, 18 | PELT_TYPE_RT, 19 | PELT_TYPE_DL, 20 | PELT_TYPE_IRQ, 21 | PELT_TYPE_THERMAL, 22 | }; 23 | 24 | struct rq_pelt_event { 25 | unsigned long long ts; 26 | int cpu; 27 | enum pelt_type type; 28 | unsigned long load_avg; 29 | unsigned long runnable_avg; 30 | unsigned long util_avg; 31 | unsigned long util_est_enqueued; 32 | unsigned long util_est_ewma; 33 | unsigned long uclamp_min; 34 | unsigned long uclamp_max; 35 | }; 36 | 37 | struct task_pelt_event { 38 | unsigned long long ts; 39 | int cpu; 40 | pid_t pid; 41 | char comm[TASK_COMM_LEN]; 42 | unsigned long load_avg; 43 | unsigned long runnable_avg; 44 | unsigned long util_avg; 45 | unsigned long util_est_enqueued; 46 | unsigned long util_est_ewma; 47 | unsigned long uclamp_min; 48 | unsigned long uclamp_max; 49 | int running; 50 | }; 51 | 52 | 53 | struct rq_nr_running_event { 54 | unsigned long long ts; 55 | int cpu; 56 | int nr_running; 57 | int change; 58 | }; 59 | 60 | struct sched_switch_event { 61 | unsigned long long ts; 62 | int cpu; 63 | pid_t pid; 64 | char comm[TASK_COMM_LEN]; 65 | int running; 66 | }; 67 | 68 | struct freq_idle_event { 69 | unsigned long long ts; 70 | int cpu; 71 | unsigned int frequency; 72 | int idle_state; 73 | int idle_miss; 74 | }; 75 | 76 | struct softirq_event { 77 | unsigned long long ts; 78 | int cpu; 79 | char softirq[TASK_COMM_LEN]; 80 | unsigned long duration; 81 | }; 82 | 83 | enum lb_phases { 84 | LB_NOHZ_IDLE_BALANCE, 85 | LB_RUN_REBALANCE_DOMAINS, 86 | LB_REBALANCE_DOMAINS, 87 | LB_BALANCE_FAIR, 88 | LB_PICK_NEXT_TASK_FAIR, 89 | LB_NEWIDLE_BALANCE, 90 | LB_LOAD_BALANCE, 91 | }; 92 | 93 | #define MAX_SD_LEVELS 10 94 | 95 | struct lb_sd_stats { 96 | int cpu; 97 | int level[MAX_SD_LEVELS]; 98 | unsigned int balance_interval[MAX_SD_LEVELS]; 99 | }; 100 | 101 | struct lb_event { 102 | unsigned long long ts; 103 | int this_cpu; 104 | int lb_cpu; 105 | enum lb_phases phase; 106 | bool entry; 107 | unsigned int overloaded; 108 | unsigned int overutilized; 109 | unsigned long misfit_task_load; 110 | struct lb_sd_stats sd_stats; 111 | }; 112 | 113 | struct ipi_event { 114 | unsigned long long ts; 115 | int from_cpu; 116 | int target_cpu; 117 | void *callsite; 118 | void *callback; 119 | }; 120 | 121 | 122 | #ifdef __VMLINUX_H__ 123 | char hi_softirq[TASK_COMM_LEN] = "hi"; 124 | char timer_softirq[TASK_COMM_LEN] = "timer"; 125 | char net_tx_softirq[TASK_COMM_LEN] = "net_tx"; 126 | char net_rx_softirq[TASK_COMM_LEN] = "net_rx"; 127 | char block_softirq[TASK_COMM_LEN] = "block"; 128 | char irq_poll_softirq[TASK_COMM_LEN] = "irq_poll"; 129 | char tasklet_softirq[TASK_COMM_LEN] = "tasklet"; 130 | char sched_softirq[TASK_COMM_LEN] = "sched"; 131 | char hrtimer_softirq[TASK_COMM_LEN] = "hrtimer"; 132 | char rcu_softirq[TASK_COMM_LEN] = "rcu"; 133 | char unknown_softirq[TASK_COMM_LEN] = "unknown"; 134 | 135 | static inline void copy_softirq(char *dst, unsigned int vec_nr) 136 | { 137 | switch (vec_nr) { 138 | case HI_SOFTIRQ: 139 | copy_str(dst, hi_softirq, TASK_COMM_LEN); 140 | break; 141 | case TIMER_SOFTIRQ: 142 | copy_str(dst, timer_softirq, TASK_COMM_LEN); 143 | break; 144 | case NET_TX_SOFTIRQ: 145 | copy_str(dst, net_tx_softirq, TASK_COMM_LEN); 146 | break; 147 | case NET_RX_SOFTIRQ: 148 | copy_str(dst, net_rx_softirq, TASK_COMM_LEN); 149 | break; 150 | case BLOCK_SOFTIRQ: 151 | copy_str(dst, block_softirq, TASK_COMM_LEN); 152 | break; 153 | case IRQ_POLL_SOFTIRQ: 154 | copy_str(dst, irq_poll_softirq, TASK_COMM_LEN); 155 | break; 156 | case TASKLET_SOFTIRQ: 157 | copy_str(dst, tasklet_softirq, TASK_COMM_LEN); 158 | break; 159 | case SCHED_SOFTIRQ: 160 | copy_str(dst, sched_softirq, TASK_COMM_LEN); 161 | break; 162 | case HRTIMER_SOFTIRQ: 163 | copy_str(dst, hrtimer_softirq, TASK_COMM_LEN); 164 | break; 165 | case RCU_SOFTIRQ: 166 | copy_str(dst, rcu_softirq, TASK_COMM_LEN); 167 | break; 168 | default: 169 | copy_str(dst, unknown_softirq, TASK_COMM_LEN); 170 | break; 171 | } 172 | } 173 | #endif /* __VMLINUX_H__ */ 174 | 175 | #endif /* __SCHED_ANALYZER_EVENTS_H__ */ 176 | -------------------------------------------------------------------------------- /sched-analyzer-pp/README.md: -------------------------------------------------------------------------------- 1 | # sched-analyzer-pp 2 | 3 | Post process the produced sched-analyzer.perfetto-trace to detect potential 4 | problems or summarizes some metrics. 5 | 6 | ## Requirements 7 | 8 | ``` 9 | pip install -r requirements.txt 10 | ``` 11 | 12 | ### Setup autocomplete 13 | 14 | ``` 15 | activate-global-python-argcomplete --user 16 | complete -r sched-analyzer-pp 17 | ``` 18 | 19 | You might need to add the following to your ~/.bash_completion to get it to 20 | work: 21 | 22 | ``` 23 | for file in ~/.bash_completion.d/* 24 | do 25 | . $file 26 | done 27 | ``` 28 | 29 | ### Add to PATH 30 | 31 | Add following to your .bashrc or equivalent 32 | 33 | ``` 34 | export PATH=~/path/to/sched-analyzer-pp:$PATH 35 | ``` 36 | 37 | ## Available analysis 38 | 39 | See `sched-analyzer-pp --help` for list of all available 40 | options 41 | 42 | * Plot frequency selection and residency for each policy on TUI 43 | * Plot frequency and residency seen by a task when it is RUNNING 44 | * Plot idle residencies on TUI 45 | * Plot various PELT data on TUI for CPU and Tasks 46 | * Plot various PELT data on TUI for a task when it is RUNNING 47 | * Plot nr-cpu-running on TUI 48 | * Save produced TUI output as png image (not available for all outputs) 49 | * Save raw data as csv 50 | 51 | ## Usage 52 | 53 | ### Plot freq and freq residency for each CPU 54 | 55 | Can be used on any perfetto trace captured with cpufreq data. 56 | 57 | ``` 58 | sched-analyzer-pp --freq --freq-residency sched-analyzer.perfetto-trace 59 | 60 | CPU0 Frequency 61 | ┌──────────────────────────────────────────────────────────────────────────────────────────────┐ 62 | 2.06┤ ▐▌ ▐▀▌ │ 63 | 1.70┤ ▐▙ ▐ ▌ │ 64 | 1.33┤ ▐█▖ ▖ ▟ ▌ │ 65 | │ █▌ ▌ ▌ ▌ │ 66 | 0.97┤ ▜█ █ ▌ ▌ ▐ │ 67 | 0.60┤ ▐█▄▄▄█▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▌ ▙▄▄▄▟▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄│ 68 | └┬─────────┬──────────┬─────────┬─────────┬──────────┬─────────┬─────────┬──────────┬─────────┬┘ 69 | 0.00 0.36 0.72 1.07 1.43 1.79 2.15 2.51 2.87 3.22 70 | CPU4 Frequency 71 | ┌──────────────────────────────────────────────────────────────────────────────────────────────┐ 72 | 3.20┤ ▝▌ ▟▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀│ 73 | 2.55┤ ▌ ▗▄▐▘ │ 74 | 1.90┤ ▌ ▌ ▐▐▛ │ 75 | │ ▌ ▙▖ ▐▐▌ │ 76 | 1.25┤ ▌ █▌ ▐▐▌ │ 77 | 0.60┤ ▜▄▄▄▄█▙▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▟▐▌ │ 78 | └┬─────────┬──────────┬─────────┬─────────┬──────────┬─────────┬─────────┬──────────┬─────────┬┘ 79 | 0.00 0.36 0.72 1.07 1.43 1.79 2.15 2.51 2.87 3.22 80 | 81 | ──────────────────────────────────── CPU0 Frequency residency % ──────────────────────────────────── 82 | 0.6 ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 94.35000000000001 83 | 0.972 ▇ 1.47 84 | 1.332 ▇ 1.31 85 | 1.704 ▇ 0.68 86 | 2.064 ▇▇ 2.19 87 | 88 | ──────────────────────────────────── CPU4 Frequency residency % ──────────────────────────────────── 89 | 0.6 ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 40.18 90 | 0.828 ▇ 0.44 91 | 1.5 ▇▇ 1.1400000000000001 92 | 1.956 ▇ 0.42 93 | 2.184 ▇ 0.63 94 | 2.388 ▇ 0.42 95 | 2.592 ▇ 0.8200000000000001 96 | 2.772 ▇ 0.42 97 | 2.988 ▇ 0.42 98 | 3.096 ▇ 0.42 99 | 3.204 ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 54.72 100 | ``` 101 | 102 | ### Plot freq and freq residency for a task 103 | 104 | Can be used on any perfetto trace captured with cpufreq and sched data. 105 | 106 | ``` 107 | sched-analyzer-pp --freq-task rampup --freq-residency-task rampup sched-analyzer.perfetto-trace 108 | 109 | 38673 rampup CPU0.0 Frequency 110 | ┌──────────────────────────────────────────────────────────────────────────────────────────────┐ 111 | 2.06┤ ▐▀▘ │ 112 | 1.70┤ ▐ │ 113 | 1.33┤ ▗ ▟ │ 114 | │ ▌ │ 115 | 0.97┤ ▌ │ 116 | 0.60┤ ▖ ▗ ▗▗ ▗▗ ▗ ▖ ▄▌ │ 117 | └┬─────────┬──────────┬─────────┬─────────┬──────────┬─────────┬─────────┬──────────┬─────────┬┘ 118 | 0.00 0.36 0.72 1.07 1.43 1.79 2.15 2.51 2.87 3.22 119 | 38673 rampup CPU4.0 Frequency 120 | ┌──────────────────────────────────────────────────────────────────────────────────────────────┐ 121 | 3.20┤ ▐▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀ │ 122 | 2.89┤ ▛ │ 123 | 2.58┤ ▐▘ │ 124 | │ ▐ │ 125 | 2.27┤ ▟ │ 126 | 1.96┤ ▌ │ 127 | └┬─────────┬──────────┬─────────┬─────────┬──────────┬─────────┬─────────┬──────────┬─────────┬┘ 128 | 0.00 0.36 0.72 1.07 1.43 1.79 2.15 2.51 2.87 3.22 129 | 130 | ──────────────────────────── 38673 rampup CPU0.0 Frequency residency % ───────────────────────────── 131 | 0.6 ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 87.97 132 | 0.972 ▇ 1.06 133 | 1.332 ▇▇▇▇▇▇ 6.0 134 | 1.704 ▇ 0.99 135 | 2.064 ▇▇▇▇ 3.97 136 | 137 | ──────────────────────────── 38673 rampup CPU4.0 Frequency residency % ───────────────────────────── 138 | 1.956 ▇▇ 1.56 139 | 2.184 ▇▇ 1.56 140 | 2.388 ▇▇ 1.56 141 | 2.772 ▇▇ 1.56 142 | 2.988 ▇▇ 1.56 143 | 3.096 ▇▇ 1.56 144 | 3.204 ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 90.62 145 | ``` 146 | 147 | ### Plot idle residency for each CPU 148 | 149 | Can be used on any perfetto trace captured with cpuidle data. 150 | 151 | Output will be colored on terminal. 152 | 153 | ``` 154 | sched-analyzer-pp --idle-residency sched-analyzer.perfetto-trace 155 | 156 | ─────────────────────────────────────── CPU Idle Residency % ─────────────────────────────────────── 157 | ▇▇▇▇▇▇ 6.67 158 | 0 ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 56.21 159 | ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 37.12 160 | 161 | ▇ 1.1 162 | 1 ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 68.69 163 | ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 30.21 164 | 165 | 0.32 166 | 2 ▇▇▇▇▇▇▇▇▇▇ 10.93 167 | ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 88.76 168 | 169 | ▇ 1.1 170 | 3 ▇▇▇▇▇▇▇▇▇ 9.82 171 | ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 89.08 172 | 173 | 0.05 174 | 4 0.08 175 | ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 99.87 176 | 177 | 0.01 178 | 5 ▇ 1.19 179 | ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 98.8 180 | 181 | 0.01 182 | 6 ▇ 1.34 183 | ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 98.65 184 | 185 | 0.01 186 | 7 ▇ 1.31 187 | ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 98.68 188 | ───────────────────────────────────── ▇▇▇ -1.0 ▇▇▇ 0.0 ▇▇▇ 1.0 ───────────────────────────────────── 189 | ``` 190 | 191 | ### Plot PELT data for a CPU or a Task 192 | 193 | Requires perfetto trace captured with sched-analyzer with one of the PELT flag 194 | set, ie: `sched-analyzer --util-avg --load-avg --runnable-avg`. Only specify 195 | the signal you're interested in, you don't have to specify them all. 196 | 197 | #### Plot PELT signal for all CPUs 198 | 199 | You can replace util-avg with runnable-avg etc for any of the examples below. 200 | 201 | ``` 202 | sched-analyzer-pp --util-avg CPU sched-analyzer.perfetto-trace 203 | 204 | CPU0 util_avg 205 | ┌─────────────────────────────────────────────────────────────────────────────────────────────┐ 206 | 426.0┤ ▗█ │ 207 | 319.5┤ ▐▐ │ 208 | 213.0┤ ▟▐ │ 209 | │ ▐▖ ▙ ▌▐ │ 210 | 106.5┤ ▐▜▖ ▛▙ ▟▘▐ │ 211 | 0.0┤▄▟ ▀▜▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▌▝▀▙▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▟▘ ▐▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄│ 212 | └┬─────────┬─────────┬──────────┬─────────┬─────────┬─────────┬──────────┬─────────┬─────────┬┘ 213 | 0.00 0.36 0.72 1.07 1.43 1.79 2.15 2.51 2.87 3.22 214 | CPU1 util_avg 215 | ┌─────────────────────────────────────────────────────────────────────────────────────────────┐ 216 | 401.0┤▙ │ 217 | 300.8┤▜ │ 218 | 200.5┤▝▌ │ 219 | │ ▜ │ 220 | 100.2┤ ▝▙▖ │ 221 | 0.0┤ ▀▜▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄│ 222 | └┬─────────┬─────────┬──────────┬─────────┬─────────┬─────────┬──────────┬─────────┬─────────┬┘ 223 | 0.00 0.36 0.72 1.07 1.43 1.79 2.15 2.51 2.87 3.22 224 | CPU2 util_avg 225 | ┌─────────────────────────────────────────────────────────────────────────────────────────────┐ 226 | 345.0┤▙ │ 227 | 258.8┤▐▖ │ 228 | 172.5┤ ▌ │ 229 | │ ▜▖ │ 230 | 86.2┤ ▙▖ │ 231 | 0.0┤ ▀▜▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄│ 232 | └┬─────────┬─────────┬──────────┬─────────┬─────────┬─────────┬──────────┬─────────┬─────────┬┘ 233 | 0.00 0.36 0.72 1.07 1.43 1.79 2.15 2.51 2.87 3.22 234 | CPU3 util_avg 235 | ┌──────────────────────────────────────────────────────────────────────────────────────────────┐ 236 | 90.0┤ ▐▌ │ 237 | 67.5┤ ▐▙ │ 238 | 45.0┤ ▐▐ │ 239 | │ ▐▝▙ │ 240 | 22.5┤ ▐ ▐█▄ ▗▖ │ 241 | 0.0┤▟▄▄▄▄▄▄▄▄▄▄▄▄▄▟▙▄▄▄▄▄▄▟▟ ▝▜▄▄▄▄▄▄▄▄▄▄▄▄▄▄▟▀▀▙▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▟▜▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄│ 242 | └┬─────────┬──────────┬─────────┬─────────┬──────────┬─────────┬─────────┬──────────┬─────────┬┘ 243 | 0.00 0.36 0.72 1.07 1.43 1.79 2.15 2.51 2.87 3.22 244 | CPU4 util_avg 245 | ┌──────────────────────────────────────────────────────────────────────────────────────────────┐ 246 | 1024┤ ▄▄▛▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▌│ 247 | 768┤▙ ▟▘ ▌│ 248 | 512┤▐▖ ▟▘ ▙│ 249 | │ ▙ ▌ │ 250 | 256┤ ▝▙▄ ▌ │ 251 | 0┤ ▝▜▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▌ │ 252 | └┬─────────┬──────────┬─────────┬─────────┬──────────┬─────────┬─────────┬──────────┬─────────┬┘ 253 | 0.00 0.36 0.72 1.07 1.43 1.79 2.15 2.51 2.87 3.22 254 | CPU5 util_avg 255 | ┌─────────────────────────────────────────────────────────────────────────────────────────────┐ 256 | 513.0┤▙ ▐│ 257 | 384.8┤▐ ▐│ 258 | 256.5┤▝▌ ▐│ 259 | │ ▜▖ ▐│ 260 | 128.2┤ ▌ ▐│ 261 | 0.0┤ ▙▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▟│ 262 | └┬─────────┬─────────┬──────────┬─────────┬─────────┬─────────┬──────────┬─────────┬─────────┬┘ 263 | 0.00 0.36 0.72 1.07 1.43 1.79 2.15 2.51 2.87 3.22 264 | CPU6 util_avg 265 | ┌─────────────────────────────────────────────────────────────────────────────────────────────┐ 266 | 513.0┤▙▖ ▐▖ │ 267 | 384.8┤ ▌ ▐▌ │ 268 | 256.5┤ ▌ ▐▜ │ 269 | │ ▜ ▐▝▙▖ │ 270 | 128.2┤ ▝▙▄ ▐ ▌ │ 271 | 0.0┤ ▝▜▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▟ ▙▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄│ 272 | └┬─────────┬─────────┬──────────┬─────────┬─────────┬─────────┬──────────┬─────────┬─────────┬┘ 273 | 0.00 0.36 0.72 1.07 1.43 1.79 2.15 2.51 2.87 3.22 274 | CPU7 util_avg 275 | ┌─────────────────────────────────────────────────────────────────────────────────────────────┐ 276 | 525.0┤▌ █│ 277 | 393.8┤▀▌ █│ 278 | 262.5┤ ▌ █│ 279 | │ ▜▖ █│ 280 | 131.2┤ ▙▄ █│ 281 | 0.0┤ ▝▜▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄█│ 282 | └┬─────────┬─────────┬──────────┬─────────┬─────────┬─────────┬──────────┬─────────┬─────────┬┘ 283 | 0.00 0.36 0.72 1.07 1.43 1.79 2.15 2.51 2.87 3.22 284 | ``` 285 | 286 | #### Plot PELT signal for specific CPU 287 | 288 | ``` 289 | sched-analyzer-pp --util-avg CPU4 sched-analyzer.perfetto-trace 290 | 291 | CPU4 util_avg 292 | ┌──────────────────────────────────────────────────────────────────────────────────────────────┐ 293 | 1024┤ ▄▄▛▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▌│ 294 | 768┤▙ ▟▘ ▌│ 295 | 512┤▐▖ ▟▘ ▙│ 296 | │ ▙ ▌ │ 297 | 256┤ ▝▙▄ ▌ │ 298 | 0┤ ▝▜▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▌ │ 299 | └┬─────────┬──────────┬─────────┬─────────┬──────────┬─────────┬─────────┬──────────┬─────────┬┘ 300 | 0.00 0.36 0.72 1.07 1.43 1.79 2.15 2.51 2.87 3.22 301 | ``` 302 | 303 | #### Plot PELT signal for a task 304 | 305 | ``` 306 | sched-analyzer-pp --util-avg rampup sched-analyzer.perfetto-trace 307 | 308 | rampup-38673 util_avg 309 | ┌──────────────────────────────────────────────────────────────────────────────────────────────┐ 310 | 1024┤ ▄▄▛▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀│ 311 | 768┤ ▟▘ │ 312 | 512┤ ▟▘ │ 313 | │ ▄ ▐▘ │ 314 | 256┤ ▝▙▖ ▐▀ │ 315 | 0┤ ▀▙▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▟▀ │ 316 | └┬─────────┬──────────┬─────────┬─────────┬──────────┬─────────┬─────────┬──────────┬─────────┬┘ 317 | 0.00 0.36 0.72 1.07 1.43 1.79 2.15 2.51 2.87 3.22 318 | ``` 319 | 320 | #### Plot PELT signal for a task showing only values when it is RUNNING 321 | 322 | ``` 323 | sched-analyzer-pp --util-avg-running rampup sched-analyzer.perfetto-trace 324 | 325 | rampup-38673 util_avg running 326 | ┌────────────────────────────────────────────────────────────────────────────────────────────┐ 327 | 1023.0┤ ▄▟▀▀▀▀▀▀▀▀▀▀▀▀▀▀▘ │ 328 | 767.2┤ ▟▘ │ 329 | 511.5┤ ▟▘ │ 330 | │ ▟▘ │ 331 | 255.8┤ ▐▘ │ 332 | 0.0┤ ▘▗ ▖ ▗▗ ▗ ▖ ▗ ▖ ▟▀ │ 333 | └┬─────────┬─────────┬─────────┬─────────┬──────────┬─────────┬─────────┬─────────┬─────────┬┘ 334 | 0.00 0.36 0.72 1.07 1.43 1.79 2.15 2.51 2.87 3.22 335 | ``` 336 | 337 | #### Plot histogram/residency for a PELT signal 338 | 339 | Each signal has a `-hist` and `-residency` option to show a histogram and 340 | residency % information. The residency shows the amount of % time spent at that 341 | specific value. 342 | 343 | ``` 344 | sched-analyzer-pp --util-avg CPU4 --util-avg-hist CPU4 --util-avg-residency CPU4 \ 345 | --util-avg-running rampup --util-avg-running-hist rampup \ 346 | --util-avg-running-residency rampup sched-analyzer.perfetto-trace 347 | 348 | CPU4 util_avg 349 | ┌──────────────────────────────────────────────────────────────────────────────────────────────┐ 350 | 1024┤ ▄▄▛▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▌│ 351 | 768┤▙ ▟▘ ▌│ 352 | 512┤▐▖ ▟▘ ▙│ 353 | │ ▙ ▌ │ 354 | 256┤ ▝▙▄ ▌ │ 355 | 0┤ ▝▜▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▌ │ 356 | └┬─────────┬──────────┬─────────┬─────────┬──────────┬─────────┬─────────┬──────────┬─────────┬┘ 357 | 0.00 0.36 0.72 1.07 1.43 1.79 2.15 2.51 2.87 3.22 358 | CPU4 util_avg Histogram 359 | ┌─────────────────────────────────────────────────────────────────────────────────────────────┐ 360 | 15000┤█ │ 361 | 11250┤█ │ 362 | 7500┤█ █│ 363 | │█ █│ 364 | 3750┤█ █│ 365 | 0┤████ ██ █ █ █ ██ ███ ██ ██ ██ ███ ██ ███ █ █ █ █████████│ 366 | └─┬─────┬───────┬────────────┬─────┬────────┬───┬────┬───────┬────┬──────┬───────┬──────────┬─┘ 367 | 13.0 72.0 166.0 309.0 380.0 478.0 525.0 575.0 665.0 724.0 801.0 889.0 1014.0 368 | 369 | ──────────────────────────────────── CPU4 util_avg residency % ───────────────────────────────────── 370 | 0.0 ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 46.54 371 | 1.0 ▇▇▇ 1.86 372 | 2.0 ▇ 0.62 373 | 6.0 ▇▇ 1.24 374 | 13.0 ▇▇ 1.24 375 | 17.0 ▇ 0.31 376 | 31.0 ▇▇ 0.93 377 | 72.0 ▇▇ 1.24 378 | 166.0 ▇▇ 1.24 379 | 309.0 ▇▇ 0.93 380 | 380.0 ▇ 0.31 381 | 468.0 ▇ 0.31 382 | 478.0 ▇ 0.31 383 | 525.0 0.06 384 | 527.0 0.21 385 | 533.0 0.01 386 | 542.0 ▇ 0.31 387 | 575.0 ▇ 0.31 388 | 605.0 ▇ 0.31 389 | 665.0 ▇ 0.31 390 | 708.0 ▇ 0.31 391 | 724.0 ▇ 0.31 392 | 776.0 ▇ 0.31 393 | 800.0 0.01 394 | 801.0 0.19 395 | 821.0 ▇ 0.31 396 | 859.0 ▇ 0.31 397 | 889.0 ▇ 0.31 398 | 914.0 ▇ 0.31 399 | 935.0 ▇ 0.31 400 | 951.0 ▇ 0.31 401 | 965.0 ▇ 0.31 402 | 976.0 ▇ 0.31 403 | 985.0 ▇ 0.31 404 | 992.0 ▇ 0.31 405 | 998.0 ▇ 0.31 406 | 1003.0 ▇ 0.31 407 | 1007.0 ▇ 0.31 408 | 1010.0 ▇ 0.31 409 | 1012.0 ▇ 0.31 410 | 1014.0 ▇ 0.31 411 | 1016.0 ▇ 0.31 412 | 1017.0 ▇ 0.31 413 | 1019.0 ▇ 0.62 414 | 1020.0 ▇ 0.31 415 | 1021.0 ▇ 0.62 416 | 1022.0 ▇▇ 0.93 417 | 1023.0 ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 28.48 418 | 1024.0 ▇▇▇▇▇▇▇ 3.72 419 | rampup-38673 util_avg running 420 | ┌────────────────────────────────────────────────────────────────────────────────────────────┐ 421 | 1023.0┤ ▄▟▀▀▀▀▀▀▀▀▀▀▀▀▀▀▘ │ 422 | 767.2┤ ▟▘ │ 423 | 511.5┤ ▟▘ │ 424 | │ ▟▘ │ 425 | 255.8┤ ▐▘ │ 426 | 0.0┤ ▘▗ ▖ ▗▗ ▗ ▖ ▗ ▖ ▟▀ │ 427 | └┬───────────────────────────┬───────────────────────────┬───────────────────────────────────┘ 428 | 0.0 1.0 2.0 429 | rampup-38673 util_avg running Histogram 430 | ┌──────────────────────────────────────────────────────────────────────────────────────────────┐ 431 | 3200┤ █│ 432 | 2400┤ █│ 433 | 1600┤ █│ 434 | │ █│ 435 | 800┤ █│ 436 | 0┤███ ████ █ ██ ██ ██ ██ ██ ██ █ █ ██ ██ ██ ██ █ ██ █ █████████│ 437 | └──┬───┬────┬─────────┬──────┬────┬─────────┬─────┬──────────┬─────┬───┬──────────┬──────────┬─┘ 438 | 22.0 64.0 118.0 233.0 303.0 366.0 478.0 542.0 665.0 724.0 776.0 889.0 1014.0 439 | 440 | ──────────────────────────── rampup-38673 util_avg running residency % ───────────────────────────── 441 | 0.0 ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 39.7 442 | 3.0 ▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 7.34 443 | 22.0 ▇▇▇▇▇▇▇▇▇▇▇ 6.1000000000000005 444 | 48.0 ▇ 0.67 445 | 64.0 ▇▇▇▇▇▇ 3.06 446 | 77.0 ▇ 0.61 447 | 118.0 ▇ 0.61 448 | 170.0 ▇ 0.61 449 | 233.0 ▇ 0.61 450 | 303.0 ▇ 0.61 451 | 366.0 ▇ 0.61 452 | 425.0 ▇ 0.61 453 | 478.0 ▇ 0.61 454 | 542.0 ▇ 0.61 455 | 605.0 ▇ 0.61 456 | 665.0 ▇ 0.61 457 | 724.0 ▇ 0.61 458 | 776.0 ▇ 0.61 459 | 821.0 ▇ 0.61 460 | 859.0 ▇ 0.61 461 | 889.0 ▇ 0.61 462 | 914.0 ▇ 0.61 463 | 935.0 ▇ 0.61 464 | 951.0 ▇ 0.61 465 | 965.0 ▇ 0.61 466 | 976.0 ▇ 0.61 467 | 985.0 ▇ 0.61 468 | 992.0 ▇ 0.61 469 | 998.0 ▇ 0.61 470 | 1003.0 ▇ 0.61 471 | 1007.0 ▇ 0.61 472 | 1010.0 ▇ 0.61 473 | 1012.0 ▇ 0.61 474 | 1014.0 ▇ 0.61 475 | 1016.0 ▇ 0.61 476 | 1017.0 ▇ 0.61 477 | 1019.0 ▇▇ 1.21 478 | 1020.0 ▇ 0.61 479 | 1021.0 ▇▇ 1.21 480 | 1022.0 ▇▇▇ 1.82 481 | 1023.0 ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 19.43 482 | ``` 483 | 484 | #### Manipulate plots 485 | 486 | You can control the start and end timestamp to zoom into a specific window of 487 | time. Histogram and residency info then will only be relative to this selected 488 | window then. 489 | 490 | You can also control the width and height of the TUI figures and frequency of 491 | ticks on x and y axis. And show a grid for both x and y axis or only one of 492 | them. 493 | 494 | ``` 495 | sched-analyzer-pp --util-avg CPU4 --util-avg-hist CPU4 --util-avg-residency CPU4 \ 496 | --util-avg rampup --util-avg-hist rampup --util-avg-residency rampup \ 497 | --ts-start 1.6 --ts-end 2.1 --fig-width-tui 130 --fig-height-tui 20 \ 498 | --grid-y-tui sched-analyzer.perfetto-trace 499 | 500 | CPU4 util_avg 501 | ┌┬────────────┬─────────────┬────────────┬─────────────┬────────────┬─────────────┬────────────┬─────────────┬────────────┬┐ 502 | 1019.0┤│ │ │ │ │ │ │ ▄▄▄▄▄▄▄▄▄▟▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀│ 503 | ││ │ │ │ │ │ │ ▄▄▄▄▄▛▀▀▀▀▘ │ ││ 504 | ││ │ │ │ │ │ ▗▄▟▀▀▘ │ │ ││ 505 | ││ │ │ │ │ │ ▗▄▄▛▀▀ │ │ │ ││ 506 | 764.2┤│ │ │ │ │ │ ▄▄▟ │ │ │ ││ 507 | ││ │ │ │ │ │ ▗▄▄▌ │ │ │ ││ 508 | ││ │ │ │ │ ▄▄▟ │ │ │ ││ 509 | 509.5┤│ │ │ │ │ ▗▄▄▌ │ │ │ ││ 510 | ││ │ │ │ │ ▐▀▀ │ │ │ │ ││ 511 | ││ │ │ │ │ ▐ │ │ │ │ ││ 512 | ││ │ │ │ │ ▐ │ │ │ │ ││ 513 | 254.8┤│ │ │ │ │ ▐ │ │ │ │ ││ 514 | ││ │ │ │ │ ▐ │ │ │ │ ││ 515 | ││ │ │ │ │ ▐ │ │ │ │ ││ 516 | ││ │ │ │ │ ▐ │ │ │ │ ││ 517 | 0.0┤▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▟ │ │ │ │ ││ 518 | └┼────────────┼─────────────┼────────────┼─────────────┼────────────┼─────────────┼────────────┼─────────────┼────────────┼┘ 519 | 1.600 1.656 1.711 1.767 1.822 1.878 1.933 1.989 2.044 2.100 520 | rampup-38673 util_avg 521 | ┌┬────────────┬─────────────┬────────────┬─────────────┬────────────┬─────────────┬────────────┬─────────────┬────────────┬┐ 522 | 1019.0┤│ │ │ │ │ │ │ ▄▄▄▄▄▄▄▄▄▟▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀│ 523 | ││ │ │ │ │ │ │ ▄▄▄▄▄▛▀▀▀▀▘ │ ││ 524 | ││ │ │ │ │ │ ▗▄▟▀▀▘ │ │ ││ 525 | ││ │ │ │ │ │ ▗▄▄▛▀▀ │ │ │ ││ 526 | 764.2┤│ │ │ │ │ │ ▄▄▟ │ │ │ ││ 527 | ││ │ │ │ │ │ ▗▄▄▌ │ │ │ ││ 528 | ││ │ │ │ │ ▄▄▟ │ │ │ ││ 529 | 509.5┤│ │ │ │ │ ▗▄▄▌ │ │ │ ││ 530 | ││ │ │ │ │ ▐▀▀ │ │ │ │ ││ 531 | ││ │ │ │ │ ▛▀▀ │ │ │ │ ││ 532 | ││ │ │ │ │ ▐▀▀▘ │ │ │ │ ││ 533 | 254.8┤│ │ │ │ │▛▀▀ │ │ │ │ ││ 534 | ││ │ │ │ ▐▀▀▘ │ │ │ │ ││ 535 | ││ │ │ │ ▗▄▄▛▀▀ │ │ │ │ │ ││ 536 | ││ │ │ │ ▗▄▟ │ │ │ │ │ ││ 537 | 0.0┤▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▛▀▀▀▀▘ │ │ │ │ │ ││ 538 | └┼────────────┼─────────────┼────────────┼─────────────┼────────────┼─────────────┼────────────┼─────────────┼────────────┼┘ 539 | 1.600 1.656 1.711 1.767 1.822 1.878 1.933 1.989 2.044 2.100 540 | CPU4 util_avg Histogram 541 | ┌┬────────────────────────────────────────────────────────┬──────┬───────┬──────┬──────┬─────┬─────────┬────────┬────────┬─┐ 542 | 2563.0┤██ │ │ │ │ │ │ │ │ │ │ 543 | │██ │ │ │ │ │ │ │ │ │ │ 544 | │██ │ │ │ │ │ │ │ │ │ │ 545 | │██ │ │ │ │ │ │ │ │ │ │ 546 | 1922.2┤██ │ │ │ │ │ │ │ │ │ │ 547 | │██ │ │ │ │ │ │ │ │ │ │ 548 | │██ │ │ │ │ │ │ │ │ │ │ 549 | 1281.5┤██ │ │ │ │ │ │ │ │ │ │ 550 | │██ │ │ │ │ │ │ │ │ │ │ 551 | │██ │ │ │ │ │ │ │ │ │ │ 552 | │██ │ │ │ │ │ │ │ │ │ │ 553 | 640.8┤██ │ │ │ │ │ │ │ │ │ │ 554 | │██ │ │ │ │ │ │ │ │ │ │ 555 | │██ │ │ │ │ │ │ │ │ │ │ 556 | │██ ██ ██ ██ ██ ██ ██ ██ ██ ██ ██████████████│ 557 | 0.0┤██ ██ ██ ██ ██ ██ ██ ██ ██ ██ ██████████████│ 558 | └┼────────────────────────────────────────────────────────┼──────┼───────┼──────┼──────┼─────┼─────────┼────────┼────────┼─┘ 559 | 0.0 478.0 542.0 605.0 665.0 724.0 776.0 859.0 935.0 1012.0 560 | rampup-38673 util_avg Histogram 561 | ┌───┬──────┬────┬─────┬──────┬────────┬──────┬──────┬──────┬──────┬───────┬──────┬──────┬─────┬─────────┬─────────┬────────┬─┐ 562 | 1652┤██ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ 563 | │██ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ 564 | │██ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ 565 | │██ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ 566 | 1239┤██ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ 567 | │██ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ 568 | │██ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ 569 | 826┤██ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ 570 | │██ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ 571 | │██ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ 572 | │██ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ 573 | 413┤██ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ 574 | │██ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ 575 | │██ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ 576 | │██ █ █ ██ ██ ██ ██ ██ ██ ██ ██ ██ █ ██ ██ ██ █ ██ █ █ ████████████│ 577 | 0┤██ █ █ ██ ██ ██ ██ ██ ██ ██ ██ ██ █ ██ ██ ██ █ ██ █ █ ████████████│ 578 | └───┼──────┼────┼─────┼──────┼────────┼──────┼──────┼──────┼──────┼───────┼──────┼──────┼─────┼─────────┼─────────┼────────┼─┘ 579 | 22.0 77.0 118.0 170.0 233.0 303.0 366.0 425.0 478.0 542.0 605.0 665.0 724.0 776.0 859.0 935.0 1012.0 580 | 581 | ─────────────────────────────────────────────────── CPU4 util_avg residency % ──────────────────────────────────────────────────── 582 | 0.0 ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 51.26 583 | 478.0 ▇▇▇▇▇ 2.0 584 | 542.0 ▇▇▇▇▇ 2.0 585 | 605.0 ▇▇▇▇▇ 2.0 586 | 665.0 ▇▇▇▇▇ 2.0 587 | 724.0 ▇▇▇▇▇ 2.0 588 | 776.0 ▇▇▇▇▇ 2.0 589 | 821.0 ▇▇▇▇▇ 2.0 590 | 859.0 ▇▇▇▇▇ 2.0 591 | 889.0 ▇▇▇▇▇ 2.0 592 | 914.0 ▇▇▇▇▇ 2.0 593 | 935.0 ▇▇▇▇▇ 2.0 594 | 951.0 ▇▇▇▇▇ 2.0 595 | 965.0 ▇▇▇▇▇ 2.0 596 | 976.0 ▇▇▇▇▇ 2.0 597 | 985.0 ▇▇▇▇▇ 2.0 598 | 992.0 ▇▇▇▇▇ 2.0 599 | 998.0 ▇▇▇▇▇ 2.0 600 | 1003.0 ▇▇▇▇▇ 2.0 601 | 1007.0 ▇▇▇▇▇ 2.0 602 | 1010.0 ▇▇▇▇▇ 2.0 603 | 1012.0 ▇▇▇▇▇ 2.0 604 | 1014.0 ▇▇▇▇▇ 2.0 605 | 1016.0 ▇▇▇▇▇ 2.0 606 | 1017.0 ▇▇▇▇▇ 2.0 607 | 1019.0 ▇▇ 0.74 608 | 609 | ─────────────────────────────────────────────── rampup-38673 util_avg residency % ──────────────────────────────────────────────── 610 | 0.0 ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 33.04 611 | 22.0 ▇▇▇▇▇▇▇ 1.98 612 | 48.0 ▇▇▇▇▇▇▇▇ 2.24 613 | 77.0 ▇▇▇▇▇▇▇ 2.0 614 | 118.0 ▇▇▇▇▇▇▇ 2.0 615 | 170.0 ▇▇▇▇▇▇▇ 2.0 616 | 233.0 ▇▇▇▇▇▇▇ 2.0 617 | 303.0 ▇▇▇▇▇▇▇ 2.0 618 | 366.0 ▇▇▇▇▇▇▇ 2.0 619 | 425.0 ▇▇▇▇▇▇▇ 2.0 620 | 478.0 ▇▇▇▇▇▇▇ 2.0 621 | 542.0 ▇▇▇▇▇▇▇ 2.0 622 | 605.0 ▇▇▇▇▇▇▇ 2.0 623 | 665.0 ▇▇▇▇▇▇▇ 2.0 624 | 724.0 ▇▇▇▇▇▇▇ 2.0 625 | 776.0 ▇▇▇▇▇▇▇ 2.0 626 | 821.0 ▇▇▇▇▇▇▇ 2.0 627 | 859.0 ▇▇▇▇▇▇▇ 2.0 628 | 889.0 ▇▇▇▇▇▇▇ 2.0 629 | 914.0 ▇▇▇▇▇▇▇ 2.0 630 | 935.0 ▇▇▇▇▇▇▇ 2.0 631 | 951.0 ▇▇▇▇▇▇▇ 2.0 632 | 965.0 ▇▇▇▇▇▇▇ 2.0 633 | 976.0 ▇▇▇▇▇▇▇ 2.0 634 | 985.0 ▇▇▇▇▇▇▇ 2.0 635 | 992.0 ▇▇▇▇▇▇▇ 2.0 636 | 998.0 ▇▇▇▇▇▇▇ 2.0 637 | 1003.0 ▇▇▇▇▇▇▇ 2.0 638 | 1007.0 ▇▇▇▇▇▇▇ 2.0 639 | 1010.0 ▇▇▇▇▇▇▇ 2.0 640 | 1012.0 ▇▇▇▇▇▇▇ 2.0 641 | 1014.0 ▇▇▇▇▇▇▇ 2.0 642 | 1016.0 ▇▇▇▇▇▇▇ 2.0 643 | 1017.0 ▇▇▇▇▇▇▇ 2.0 644 | 1019.0 ▇▇▇ 0.74 645 | ``` 646 | 647 | #### Show a summary of task sched states and time RUNNING on each CPU 648 | 649 | Show information about task's execution states including total time spent in 650 | each state as absolute sum and percentage. It also shows stats on the time 651 | spent in each state ie: min, max, P50, P90 etc. Runnable info are particularly 652 | useful to reason about the wake up latency of a specific task. 653 | 654 | This option can work on any perfetto trace captured with sched info. 655 | 656 | ``` 657 | sched-analyzer-pp --sched-states rampup sched-analyzer.perfetto-trace 658 | 659 | ==================================== 660 | :: 38673 | rampup | ['./rampup'] :: 661 | ==================================================================================================== 662 | ---------------------------------------------------------------------------------------------------- 663 | ───────────────────────────── Sum Time in State Exclude Sleeping (ms) ────────────────────────────── 664 | R 2.19 665 | Running ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 739.85 666 | 667 | ────────────────────────────── % Time in State Exclude Sleeping (ms) ─────────────────────────────── 668 | R 0.3 669 | Running ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 99.7 670 | 671 | ─────────────────────────────────── Sum Time Running on CPU (ms) ─────────────────────────────────── 672 | CPU0.0 ▇▇▇▇▇▇▇▇▇▇▇▇▇ 99.89 673 | CPU4.0 ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 639.96 674 | 675 | ──────────────────────────────────── % Time Running on CPU (ms) ──────────────────────────────────── 676 | CPU0.0 ▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 13.5 677 | CPU4.0 ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 86.5 678 | 679 | Time in State (ms): 680 | ---------------------------------------------------------------------------------------------------- 681 | count mean std min 50% 75% 90% 95% 99% max 682 | state 683 | R 107.0 0.02 0.02 0.00 0.02 0.02 0.02 0.02 0.12 0.20 684 | Running 107.0 6.91 53.70 -0.00 0.01 0.01 0.16 10.05 76.96 550.00 685 | S 94.0 10.05 0.00 10.04 10.05 10.05 10.05 10.05 10.05 10.05 686 | 687 | Time Running on CPU (ms): 688 | ---------------------------------------------------------------------------------------------------- 689 | count mean std min 50% 75% 90% 95% 99% max 690 | cpu 691 | 0.0 103.0 0.97 4.07 0.0 0.01 0.01 0.01 9.50 20.53 29.81 692 | 4.0 4.0 159.99 262.43 -0.0 44.98 197.47 408.99 479.49 535.90 550.00 693 | 694 | ``` 695 | 696 | #### Show a table summary for Top 100 tasks that are runnable, running and blocked 697 | 698 | Partial output is shown as it is quite long. Two tables are shown for each 699 | category. One is sorted by max value and the other is sorted by P90. The former 700 | should help identify worst case sceneraios and the latter should help identify 701 | tasks that has frequently seen high value. Runnable info are particularly 702 | important for to reason about latencies the tasks has experienced in the trace. 703 | 704 | This option can work on any perfetto trace captured with sched info. 705 | 706 | ``` 707 | sched-analyzer-pp --sched-report sched-analyzer.perfetto-trace 708 | 709 | States Summary (ms): 710 | ---------------------------------------------------------------------------------------------------- 711 | count mean std min 50% 75% 90% 95% 99% max 712 | state 713 | I 37.0 206.48 268.23 -0.0 79.95 319.94 553.99 857.98 960.37 999.98 714 | R 697.0 0.10 0.81 0.0 0.02 0.02 0.03 0.04 0.31 10.70 715 | Running 699.0 1.11 21.07 -0.0 0.02 0.04 0.06 0.08 10.00 550.00 716 | S 646.0 39.02 133.29 -0.0 10.05 10.05 70.50 115.10 1000.36 1023.82 717 | 718 | Top 100 Runnable Tasks (ms) - sorted-by max: 719 | ---------------------------------------------------------------------------------------------------- 720 | count mean std min 50% 75% 90% 95% 99% max 721 | name tid 722 | sched-analyzer 38666 167.0 0.08 0.83 0.00 0.02 0.02 0.02 0.02 0.13 10.70 723 | kworker/4:0 2060 4.0 2.51 4.99 0.00 0.02 2.51 7.01 8.50 9.70 10.00 724 | tracebox 3524 28.0 0.36 1.76 0.01 0.03 0.04 0.04 0.04 6.83 9.34 725 | kworker/0:0 21208 2.0 4.46 6.29 0.01 4.46 6.69 8.02 8.47 8.82 8.91 726 | . 727 | . 728 | . 729 | 730 | Top 100 Runnable Tasks (ms) - sorted-by 90%: 731 | ---------------------------------------------------------------------------------------------------- 732 | count mean std min 50% 75% 90% 95% 99% max 733 | name tid 734 | kworker/0:0 21208 2.0 4.46 6.29 0.01 4.46 6.69 8.02 8.47 8.82 8.91 735 | kworker/4:0 2060 4.0 2.51 4.99 0.00 0.02 2.51 7.01 8.50 9.70 10.00 736 | tracebox 3529 3.0 1.06 1.74 0.04 0.08 1.57 2.47 2.77 3.01 3.07 737 | . 738 | . 739 | . 740 | 741 | Top 100 Running Tasks (ms) - sorted-by max: 742 | ---------------------------------------------------------------------------------------------------- 743 | count mean std min 50% 75% 90% 95% 99% max 744 | name tid 745 | rampup 38673 107.0 6.91 53.70 -0.00 0.01 0.01 0.16 10.05 76.96 550.00 746 | tracebox 3524 28.0 0.20 0.71 -0.00 0.06 0.08 0.09 0.09 2.81 3.82 747 | . 748 | . 749 | . 750 | 751 | Top 100 Running Tasks (ms) - sorted-by 90%: 752 | ---------------------------------------------------------------------------------------------------- 753 | count mean std min 50% 75% 90% 95% 99% max 754 | name tid 755 | tracebox 3529 3.0 0.99 1.67 -0.00 0.05 1.48 2.35 2.63 2.86 2.92 756 | snapd 1133 2.0 0.95 1.31 0.02 0.95 1.41 1.69 1.78 1.85 1.87 757 | . 758 | . 759 | . 760 | ``` 761 | -------------------------------------------------------------------------------- /sched-analyzer-pp/freq.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | # 3 | # SPDX-License-Identifier: GPL-2.0 4 | # Copyright (C) 2024 Qais Yousef 5 | 6 | import numpy as np 7 | import pandas as pd 8 | import settings 9 | import utils 10 | 11 | query = "select ts, cpu, value as freq from counter as c left join cpu_counter_track as t on c.track_id = t.id where t.name = 'cpufreq'" 12 | 13 | def __find_clusters(): 14 | 15 | global clusters 16 | if clusters: 17 | return 18 | 19 | nr_cpus = len(df_freq.cpu.unique()) 20 | 21 | clusters = [0] 22 | df_freq_cpu = df_freq[df_freq.cpu == 0] 23 | for cpu in range(nr_cpus): 24 | df_freq_next_cpu = df_freq[df_freq.cpu == cpu] 25 | 26 | if np.array_equal(df_freq_cpu.freq.values, df_freq_next_cpu.freq.values): 27 | continue 28 | 29 | df_freq_cpu = df_freq[df_freq.cpu == cpu] 30 | clusters.append(cpu) 31 | 32 | def __init(): 33 | 34 | global df_freq 35 | if df_freq is None: 36 | df_freq = trace_freq.as_pandas_dataframe() 37 | if df_freq.empty: 38 | return 39 | 40 | df_freq.freq = df_freq.freq / 1000000 41 | 42 | __find_clusters() 43 | 44 | def init(trace): 45 | 46 | global trace_freq 47 | global df_freq 48 | global clusters 49 | global df_states 50 | 51 | trace_freq = trace.query(query) 52 | df_freq = None 53 | clusters = None 54 | 55 | __init() 56 | df_states = utils.get_df_states() 57 | 58 | def save_csv(prefix): 59 | 60 | df_freq.to_csv(prefix + '_freq.csv') 61 | 62 | def plot_matplotlib(plt, prefix): 63 | 64 | color = ['b', 'y', 'r'] 65 | i = 0 66 | 67 | num_rows = len(clusters) 68 | row_pos = 1 69 | 70 | if df_freq.empty: 71 | return 72 | 73 | plt.figure(figsize=(8,4*num_rows)) 74 | 75 | for cpu in clusters: 76 | df_freq_cpu = df_freq[df_freq.cpu == cpu].copy() 77 | df_freq_cpu = utils.convert_ts(df_freq_cpu, True) 78 | 79 | if df_freq_cpu.empty: 80 | continue 81 | 82 | plt.subplot(num_rows, 1, row_pos) 83 | row_pos += 1 84 | df_freq_cpu.freq.plot(title='CPU' + str(cpu) + ' Frequency', alpha=0.75, 85 | drawstyle='steps-post', style='-', color=color[i], xlim=(settings.ts_start, settings.ts_end)) 86 | plt.grid() 87 | 88 | i += 1 89 | if i == 3: 90 | i = 0 91 | 92 | plt.tight_layout() 93 | plt.savefig(prefix + '_frequency.png') 94 | 95 | def plot_residency_matplotlib(plt, prefix, abs=False): 96 | 97 | color = ['b', 'y', 'r'] 98 | i = 0 99 | 100 | num_rows = len(clusters) 101 | row_pos = 1 102 | 103 | if df_freq.empty: 104 | return 105 | 106 | plt.figure(figsize=(8,4*num_rows)) 107 | 108 | for cpu in clusters: 109 | df_freq_cpu = df_freq[df_freq.cpu == cpu].copy() 110 | df_freq_cpu = utils.convert_ts(df_freq_cpu, True) 111 | 112 | df_duration = utils.gen_df_duration_groupby(df_freq_cpu, 'freq', abs) 113 | 114 | plt.subplot(num_rows, 1, row_pos) 115 | row_pos += 1 116 | if not df_duration.empty: 117 | ax = df_duration.plot.bar(title='CPU' + str(cpu) + ' Frequency residency {}'.format('(ms)' if abs else '%'), alpha=0.75, color=color[i]) 118 | ax.bar_label(ax.containers[0]) 119 | plt.grid() 120 | 121 | i += 1 122 | if i == 3: 123 | i = 0 124 | 125 | plt.tight_layout() 126 | plt.savefig(prefix + '_frequency_residency.png') 127 | 128 | def plot_residency_abs_matplotlib(plt, prefix): 129 | 130 | plot_residency_matplotlib(plt, prefix, True) 131 | 132 | def plot_tui(plt): 133 | 134 | if df_freq.empty: 135 | return 136 | 137 | for cpu in clusters: 138 | df_freq_cpu = df_freq[df_freq.cpu == cpu].copy() 139 | df_freq_cpu = utils.convert_ts(df_freq_cpu, True) 140 | 141 | if not df_freq_cpu.empty: 142 | print() 143 | plt.cld() 144 | plt.plot_size(settings.fig_width_tui, settings.fig_height_tui) 145 | plt.plot(df_freq_cpu.index.values, df_freq_cpu.freq.values) 146 | plt.title('CPU' + str(cpu) + ' Frequency') 147 | plt.show() 148 | 149 | def plot_residency_tui(plt, abs=False): 150 | 151 | if df_freq.empty: 152 | return 153 | 154 | for cpu in clusters: 155 | df_freq_cpu = df_freq[df_freq.cpu == cpu].copy() 156 | df_freq_cpu = utils.convert_ts(df_freq_cpu, True) 157 | 158 | df_duration = utils.gen_df_duration_groupby(df_freq_cpu, 'freq', abs) 159 | if not df_duration.empty: 160 | print() 161 | plt.cld() 162 | plt.plot_size(settings.fig_width_tui, settings.fig_height_tui) 163 | plt.simple_bar(df_duration.index.values, df_duration.values, width=settings.fig_width_tui, title='CPU' + str(cpu) + ' Frequency residency {}'.format('(ms)' if abs else '%')) 164 | plt.show() 165 | 166 | def plot_residency_abs_tui(plt): 167 | 168 | plot_residency_tui(plt, True) 169 | 170 | def plot_task_tui(plt, threads=[]): 171 | 172 | if df_freq.empty: 173 | return 174 | 175 | if df_states.empty: 176 | return 177 | 178 | for thread in threads: 179 | df = df_states[df_states.name.str.contains(thread).fillna(False)] 180 | 181 | for thread in sorted(df.name.unique()): 182 | df_thread = df[df.name == thread] 183 | 184 | for tid in sorted(df_thread.tid.unique()): 185 | df_tid = df_thread[df_thread.tid == tid] 186 | df_tid_running = df_tid[df_tid.state == 'Running'] 187 | 188 | for cpu in sorted(df_tid_running.cpu.unique()): 189 | df_freq_cpu = df_freq[df_freq.cpu == cpu].copy() 190 | df_freq_cpu = utils.convert_ts(df_freq_cpu, True) 191 | 192 | df_tid_cpu = df_tid[(df_tid.cpu == cpu) | (df_tid.cpu.isna())].copy() 193 | 194 | df_freq_cpu = utils.multiply_df_tid_running(df_freq_cpu, 'freq', df_tid_cpu) 195 | 196 | if not df_freq_cpu.empty: 197 | print() 198 | plt.cld() 199 | plt.plot_size(settings.fig_width_tui, settings.fig_height_tui) 200 | plt.plot(df_freq_cpu.index.values, df_freq_cpu.freq.values) 201 | plt.title('{} {}'.format(tid, thread) + ' CPU' + str(cpu) + ' Frequency') 202 | plt.show() 203 | 204 | def plot_task_residency_tui(plt, threads=[], abs=False): 205 | 206 | if df_freq.empty: 207 | return 208 | 209 | if df_states.empty: 210 | return 211 | 212 | for thread in threads: 213 | df = df_states[df_states.name.str.contains(thread).fillna(False)] 214 | 215 | for thread in sorted(df.name.unique()): 216 | df_thread = df[df.name == thread] 217 | 218 | for tid in sorted(df_thread.tid.unique()): 219 | df_tid = df_thread[df_thread.tid == tid] 220 | df_tid_running = df_tid[df_tid.state == 'Running'] 221 | 222 | for cpu in sorted(df_tid_running.cpu.unique()): 223 | df_freq_cpu = df_freq[df_freq.cpu == cpu].copy() 224 | df_freq_cpu = utils.convert_ts(df_freq_cpu, True) 225 | 226 | df_tid_cpu = df_tid[(df_tid.cpu == cpu) | (df_tid.cpu.isna())].copy() 227 | 228 | df_freq_cpu = utils.multiply_df_tid_running(df_freq_cpu, 'freq', df_tid_cpu) 229 | 230 | df_duration = utils.gen_df_duration_groupby(df_freq_cpu, 'freq', abs) 231 | if not df_duration.empty: 232 | print() 233 | plt.cld() 234 | plt.plot_size(settings.fig_width_tui, settings.fig_height_tui) 235 | plt.simple_bar(df_duration.index.values, df_duration.values, width=settings.fig_width_tui, 236 | title='{} {}'.format(tid, thread) + ' CPU' + str(cpu) + ' Frequency residency {}'.format('(ms)' if abs else '%')) 237 | plt.show() 238 | 239 | def plot_task_residency_abs_tui(plt, threads=[]): 240 | 241 | plot_task_residency_tui(plt, threads, True) 242 | -------------------------------------------------------------------------------- /sched-analyzer-pp/idle.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | # 3 | # SPDX-License-Identifier: GPL-2.0 4 | # Copyright (C) 2024 Qais Yousef 5 | 6 | import numpy as np 7 | import pandas as pd 8 | import settings 9 | import utils 10 | 11 | query = "select ts, cpu, value as idle from counter as c left join cpu_counter_track as t on c.track_id = t.id where t.name = 'cpuidle'" 12 | 13 | 14 | def __init(): 15 | 16 | global df_idle 17 | if df_idle is None: 18 | df_idle = trace_idle.as_pandas_dataframe() 19 | if df_idle.empty: 20 | return 21 | 22 | df_idle = utils.convert_ts(df_idle) 23 | 24 | # This magic value is exit from idle. Values 0 and above are idle 25 | # states 26 | df_idle.idle = df_idle.infer_objects(copy=False).idle.replace(4294967295, -1) 27 | 28 | def init(trace): 29 | 30 | global trace_idle 31 | trace_idle = trace.query(query) 32 | 33 | global df_idle 34 | df_idle = None 35 | 36 | __init() 37 | 38 | def num_rows(): 39 | 40 | return int((len(df_idle.cpu.unique()) + 3) / 4) 41 | 42 | def save_csv(prefix): 43 | 44 | df_idle.to_csv(prefix + '_idle.csv') 45 | 46 | def plot_residency_matplotlib(plt, prefix, abs=False): 47 | 48 | func = globals()['num_rows'] 49 | num_rows = func() 50 | row_pos = 1 51 | 52 | if df_idle.empty: 53 | return 54 | 55 | nr_cpus = len(df_idle.cpu.unique()) 56 | 57 | plt.figure(figsize=(8*4,4*num_rows)) 58 | 59 | col = 0 60 | df_idle_cpu = df_idle[df_idle.cpu == 0] 61 | for cpu in range(nr_cpus): 62 | df_idle_cpu = df_idle[df_idle.cpu == cpu].copy() 63 | df_duration = utils.gen_df_duration_groupby(df_idle_cpu, 'idle', abs) 64 | 65 | if col == 0: 66 | plt.subplot(num_rows, 4, row_pos * 4 - 3) 67 | col = 1 68 | elif col == 1: 69 | plt.subplot(num_rows, 4, row_pos * 4 - 2) 70 | col = 2 71 | elif col == 2: 72 | plt.subplot(num_rows, 4, row_pos * 4 - 1) 73 | col = 3 74 | else: 75 | plt.subplot(num_rows, 4, row_pos * 4 - 0) 76 | col = 0 77 | row_pos += 1 78 | 79 | if not df_duration.empty: 80 | ax = df_duration.plot.bar(title='CPU{}'.format(cpu) + ' Idle residency {}'.format('(ms)' if abs else '%'), alpha=0.75, color='grey') 81 | ax.bar_label(ax.containers[0]) 82 | ax.set_xlabel('Idle State') 83 | plt.grid() 84 | 85 | plt.tight_layout() 86 | plt.savefig(prefix + '_idle.png') 87 | 88 | def plot_residency_abs_matplotlib(plt, prefix): 89 | 90 | plot_residency_matplotlib(plt, prefix, True) 91 | 92 | def plot_residency_tui(plt, abs=False): 93 | 94 | func = globals()['num_rows'] 95 | num_rows = func() 96 | row_pos = 1 97 | 98 | if df_idle.empty: 99 | return 100 | 101 | cpus = sorted(df_idle.cpu.unique()) 102 | nr_cpus = len(cpus) 103 | 104 | idle_states = sorted(df_idle.idle.unique()) 105 | idle_pct = [] 106 | labels = [] 107 | 108 | for state in idle_states: 109 | state_pct = [] 110 | labels.append("{}".format(state)) 111 | for cpu in range(nr_cpus): 112 | df_idle_cpu = df_idle[df_idle.cpu == cpu].copy() 113 | df_duration = utils.gen_df_duration_filtered(df_idle_cpu, df_idle_cpu.idle == state, abs) 114 | state_pct.append(df_duration) 115 | 116 | idle_pct.append(state_pct) 117 | 118 | print() 119 | plt.cld() 120 | plt.simple_multiple_bar(cpus, idle_pct, labels=labels, width=settings.fig_width_tui, title='CPU Idle Residency {}'.format('(ms)' if abs else '%')) 121 | plt.show() 122 | 123 | def plot_residency_abs_tui(plt): 124 | 125 | plot_residency_tui(plt, True) 126 | -------------------------------------------------------------------------------- /sched-analyzer-pp/requirements.txt: -------------------------------------------------------------------------------- 1 | argcomplete 2 | matplotlib 3 | pandas 4 | perfetto 5 | plotext 6 | -------------------------------------------------------------------------------- /sched-analyzer-pp/sa_track.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | # 3 | # SPDX-License-Identifier: GPL-2.0 4 | # Copyright (C) 2024 Qais Yousef 5 | 6 | import pandas as pd 7 | import settings 8 | import utils 9 | 10 | query = "select c.ts as ts, c.value as value, t.name as counter_name \ 11 | from counter as c left join process_counter_track as t on c.track_id = t.id \ 12 | left join process as p using (upid) \ 13 | where p.name like '%sched-analyzer' and counter_name like '{}'" 14 | 15 | def __multiply_df_tid_running(df, track): 16 | 17 | try: 18 | thread = track.split()[0].rsplit('-', 1)[0] 19 | tid = track.split()[0].rsplit('-', 1)[1] 20 | df_tid = utils.get_df_tid(thread, tid) 21 | if df_tid.empty: 22 | return df 23 | df = utils.multiply_df_tid_running(df, 'value', df_tid) 24 | return df 25 | except: 26 | return df 27 | 28 | 29 | def init(trace, signal): 30 | 31 | global sa_track_signal 32 | sa_track_signal = signal 33 | 34 | global trace_sa_track 35 | if "util_est" in signal: 36 | signal = "util_est.enq%" 37 | trace_sa_track = trace.query(query.format('% {}'.format(signal))) 38 | 39 | global df_sa_track 40 | df_sa_track = trace_sa_track.as_pandas_dataframe() 41 | if df_sa_track.empty: 42 | return 43 | 44 | def sa_track_save_csv(prefix): 45 | 46 | df_sa_track.to_csv(prefix + '_' + sa_track_signal + '.csv') 47 | 48 | def plot_sa_track_matplotlib(plt, prefix, tracks=[], multiply_running=False): 49 | 50 | if not any(tracks): 51 | tracks = sorted(df_sa_track.counter_name.unique()) 52 | 53 | num_rows = 0 54 | for track in tracks: 55 | subtracks = df_sa_track[df_sa_track.counter_name.str.contains(track)].counter_name.unique() 56 | num_rows += len(subtracks) 57 | row_pos = 1 58 | 59 | if not num_rows: 60 | return 61 | 62 | plt.figure(figsize=(8,4*num_rows)) 63 | 64 | for track in tracks: 65 | subtracks = df_sa_track[df_sa_track.counter_name.str.contains(track)].counter_name.unique() 66 | subtracks = sorted(subtracks) 67 | for track in subtracks: 68 | df = df_sa_track[df_sa_track.counter_name == track] 69 | df = utils.convert_ts(df, True) 70 | 71 | running_str = '' 72 | if multiply_running: 73 | df = __multiply_df_tid_running(df, track) 74 | running_str = ' running' 75 | 76 | plt.subplot(num_rows, 1, row_pos) 77 | row_pos += 1 78 | df.value.plot(title=track + running_str, 79 | drawstyle='steps-post', alpha=0.75, legend=False, 80 | xlim=(settings.ts_start, settings.ts_end)) 81 | plt.grid() 82 | 83 | plt.tight_layout() 84 | plt.savefig(prefix + '_' + sa_track_signal + running_str.replace(' ', '_') + '.png') 85 | 86 | def plot_sa_track_hist_matplotlib(plt, prefix, tracks=[], multiply_running=False): 87 | 88 | if not any(tracks): 89 | tracks = sorted(df_sa_track.counter_name.unique()) 90 | 91 | num_rows = 0 92 | for track in tracks: 93 | subtracks = df_sa_track[df_sa_track.counter_name.str.contains(track)].counter_name.unique() 94 | num_rows += len(subtracks) 95 | row_pos = 1 96 | 97 | if not num_rows: 98 | return 99 | 100 | plt.figure(figsize=(8,4*num_rows)) 101 | 102 | for track in tracks: 103 | subtracks = df_sa_track[df_sa_track.counter_name.str.contains(track)].counter_name.unique() 104 | subtracks = sorted(subtracks) 105 | for track in subtracks: 106 | df = df_sa_track[df_sa_track.counter_name == track] 107 | df = utils.convert_ts(df, True) 108 | 109 | running_str = '' 110 | if multiply_running: 111 | df = __multiply_df_tid_running(df, track) 112 | running_str = ' running' 113 | 114 | plt.subplot(num_rows, 1, row_pos) 115 | row_pos += 1 116 | plt.title(track + running_str + ' Histogram') 117 | df.value.hist(bins=100, density=False, grid=True, alpha=0.5, legend=True) 118 | 119 | plt.tight_layout() 120 | plt.savefig(prefix + '_' + sa_track_signal + running_str.replace(' ', '_') + '_hist.png') 121 | 122 | def plot_sa_track_tui(plt, tracks=[], multiply_running=False): 123 | 124 | if not any(tracks): 125 | tracks = sorted(df_sa_track.counter_name.unique()) 126 | 127 | for track in tracks: 128 | subtracks = df_sa_track[df_sa_track.counter_name.str.contains(track)].counter_name.unique() 129 | subtracks = sorted(subtracks) 130 | for track in subtracks: 131 | df = df_sa_track[df_sa_track.counter_name == track] 132 | df = utils.convert_ts(df, True) 133 | 134 | running_str = '' 135 | if multiply_running: 136 | df = __multiply_df_tid_running(df, track) 137 | running_str = ' running' 138 | 139 | if not df.empty: 140 | print() 141 | plt.cld() 142 | plt.plot_size(settings.fig_width_tui, settings.fig_height_tui) 143 | plt.plot(df.index.values, df.value.values) 144 | plt.title(track + running_str) 145 | plt.show() 146 | 147 | def plot_sa_track_hist_tui(plt, tracks=[], multiply_running=False): 148 | 149 | if not any(tracks): 150 | tracks = sorted(df_sa_track.counter_name.unique()) 151 | 152 | for track in tracks: 153 | subtracks = df_sa_track[df_sa_track.counter_name.str.contains(track)].counter_name.unique() 154 | subtracks = sorted(subtracks) 155 | for track in subtracks: 156 | df = df_sa_track[df_sa_track.counter_name == track] 157 | df = utils.convert_ts(df, True) 158 | 159 | running_str = '' 160 | if multiply_running: 161 | df = __multiply_df_tid_running(df, track) 162 | running_str = ' running' 163 | 164 | 165 | if not df.empty: 166 | df_hist = pd.Series(df.value.value_counts(ascending=True)) 167 | 168 | print() 169 | plt.cld() 170 | plt.plot_size(settings.fig_width_tui, settings.fig_height_tui) 171 | plt.bar(df_hist.index, df_hist.values, width=1/5) 172 | plt.title(track + running_str + ' Histogram') 173 | plt.show() 174 | 175 | def plot_sa_track_residency_tui(plt, tracks=[], multiply_running=False, abs=False): 176 | 177 | if not any(tracks): 178 | tracks = sorted(df_sa_track.counter_name.unique()) 179 | 180 | for track in tracks: 181 | subtracks = df_sa_track[df_sa_track.counter_name.str.contains(track)].counter_name.unique() 182 | subtracks = sorted(subtracks) 183 | for track in subtracks: 184 | df = df_sa_track[df_sa_track.counter_name == track] 185 | df = utils.convert_ts(df, True) 186 | 187 | running_str = '' 188 | if multiply_running: 189 | df = __multiply_df_tid_running(df, track) 190 | running_str = ' running' 191 | 192 | df_duration = utils.gen_df_duration_groupby(df, 'value', abs) 193 | if not df_duration.empty: 194 | print() 195 | plt.cld() 196 | plt.plot_size(settings.fig_width_tui, settings.fig_height_tui) 197 | plt.simple_bar(df_duration.index.values, df_duration.values, width=settings.fig_width_tui, 198 | title=track + running_str + ' residency {}'.format('(ms)' if abs else '%')) 199 | plt.show() 200 | 201 | def plot_sa_track_residency_abs_tui(plt, tracks=[], multiply_running=False): 202 | 203 | plot_sa_track_residency_tui(plt, tracks, multiply_running, True) 204 | -------------------------------------------------------------------------------- /sched-analyzer-pp/sched-analyzer-pp: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # PYTHON_ARGCOMPLETE_OK 3 | # 4 | # SPDX-License-Identifier: GPL-2.0 5 | # Copyright (C) 2024 Qais Yousef 6 | 7 | import argparse 8 | import contextlib 9 | import freq 10 | import idle 11 | import io 12 | import matplotlib 13 | import matplotlib.pyplot as pltlib 14 | import os 15 | import plotext as pltext 16 | import pydoc 17 | import sa_track 18 | import sched 19 | import settings 20 | import trace_processor as tp 21 | import utils 22 | 23 | try: 24 | import argcomplete 25 | except: 26 | pass 27 | 28 | args = None 29 | 30 | def add_sa_track_option(parser, name, help, running_options=False): 31 | parser.add_argument('--' + name, action='append', metavar='PATTERN', 32 | help=help) 33 | parser.add_argument('--' + name + '-hist', action='append', metavar='PATTERN') 34 | parser.add_argument('--' + name + '-residency', action='append', metavar='PATTERN') 35 | parser.add_argument('--' + name + '-residency-abs', action='append', metavar='PATTERN') 36 | 37 | if running_options: 38 | add_sa_track_option(parser, name + '-running', 'Same as ' + name + ' but filtered for when task is RUNNING, if the track is for a task.') 39 | 40 | def process_sa_track_option(args, trace, signal, running_options=False, postfix=''): 41 | vargs = vars(args) 42 | 43 | name = signal + postfix 44 | name_hist = name + '_hist' 45 | name_residency = name + '_residency' 46 | name_residency_abs = name + '_residency_abs' 47 | 48 | if vargs[name] or vargs[name_hist] or vargs[name_residency]: 49 | sa_track.init(trace, signal) 50 | show_tui = True 51 | 52 | if args.save_csv: 53 | sa_track.sa_track_save_csv(prefix) 54 | show_tui = False 55 | 56 | if vargs[name]: 57 | if args.save_png: 58 | sa_track.plot_sa_track_matplotlib(pltlib, prefix, vargs[name], 'running' in name) 59 | show_tui = False 60 | 61 | if show_tui: 62 | sa_track.plot_sa_track_tui(pltext, vargs[name], 'running' in name) 63 | 64 | if vargs[name_hist]: 65 | if args.save_png: 66 | sa_track.plot_sa_track_hist_matplotlib(pltlib, prefix, vargs[name_hist], 'running' in name) 67 | show_tui = False 68 | 69 | if show_tui: 70 | sa_track.plot_sa_track_hist_tui(pltext, vargs[name_hist], 'running' in name) 71 | 72 | if vargs[name_residency]: 73 | if args.save_png: 74 | print(name_residency + " --save-png: Not implemented yet.") 75 | show_tui = False 76 | 77 | if show_tui: 78 | sa_track.plot_sa_track_residency_tui(pltext, vargs[name_residency], 'running' in name) 79 | 80 | if vargs[name_residency_abs]: 81 | if args.save_png: 82 | print(name_residency_abs + " --save-png: Not implemented yet.") 83 | show_tui = False 84 | 85 | if show_tui: 86 | sa_track.plot_sa_track_residency_abs_tui(pltext, vargs[name_residency_abs], 'running' in name) 87 | 88 | if running_options: 89 | process_sa_track_option(args, trace, signal, postfix='_running') 90 | 91 | def parse_cmdline(): 92 | parser = argparse.ArgumentParser(description=''' 93 | Post Process sched-analyzer.perfetto-trace captured with sched-analyzer tool 94 | ''', formatter_class=argparse.RawTextHelpFormatter) 95 | 96 | # Global options 97 | parser.add_argument('filename', 98 | help='perfetto-trace file to post process.') 99 | 100 | parser.add_argument('--save-png', action='store_true', 101 | help='Save graphs as png images.') 102 | parser.add_argument('--save-csv', action='store_true', 103 | help='Save raw data as csv file.') 104 | 105 | # freq/idle modules options 106 | parser.add_argument('--freq', action='store_true', 107 | help='Show frequency for all cpus.') 108 | parser.add_argument('--freq-residency', action='store_true', 109 | help='Show percentage of time spent in each frequency for all cpus.') 110 | parser.add_argument('--freq-residency-abs', action='store_true', 111 | help='Show time in ms spent in each frequency for all cpus.') 112 | parser.add_argument('--freq-task', action='append', metavar='PATTERN', 113 | help='Show frequency for a given task') 114 | parser.add_argument('--freq-residency-task', action='append', metavar='PATTERN', 115 | help='Show frequency residency percentage for a given task') 116 | parser.add_argument('--freq-residency-abs-task', action='append', metavar='PATTERN', 117 | help='Show frequency residency in ms for a given task') 118 | parser.add_argument('--idle-residency', action='store_true', 119 | help='Show percentage of time spent in each idle state for all cpus.') 120 | parser.add_argument('--idle-residency-abs', action='store_true', 121 | help='Show time in ms spent in each idle state for all cpus.') 122 | 123 | # sa_track module options 124 | add_sa_track_option(parser, 'util-avg', 125 | 'Show util_avg for CPUs/tasks. Can be provided multiple times. \'CPU\' to show all cpus or specific task name to show all task that match the str', 126 | True) 127 | add_sa_track_option(parser, 'util-avg-rt', 128 | 'Show util_avg_rt for CPUs. Can be provided multiple times. \'CPU\' to show all cpus', 129 | False) 130 | add_sa_track_option(parser, 'util-avg-dl', 131 | 'Show util_avg_dl for CPUs. Can be provided multiple times. \'CPU\' to show all cpus', 132 | False) 133 | add_sa_track_option(parser, 'util-avg-irq', 134 | 'Show util_avg_irq for CPUs. Can be provided multiple times. \'CPU\' to show all cpus', 135 | False) 136 | add_sa_track_option(parser, 'util-est', 137 | 'Show util_est for CPUs/tasks. Can be provided multiple times. \'CPU\' to show all cpus or specific task name to show all task that match the str', 138 | True) 139 | add_sa_track_option(parser, 'load-avg-thermal', 140 | 'Show load_avg_thermal for CPUs. Can be provided multiple times. \'CPU\' to show all cpus', 141 | False) 142 | add_sa_track_option(parser, 'load-avg', 143 | 'Show load_avg for CPUs/tasks. Can be provided multiple times. \'CPU\' to show all cpus or specific task name to show all task that match the str', 144 | True) 145 | add_sa_track_option(parser, 'runnable-avg', 146 | 'Show runnable_avg for CPUs/tasks. Can be provided multiple times. \'CPU\' to show all cpus or specific task name to show all task that match the str', 147 | True) 148 | add_sa_track_option(parser, 'uclamped-avg', 149 | 'Show uclamped_avg for CPUs/tasks. Can be provided multiple times. \'CPU\' to show all cpus or specific task name to show all task that match the str', 150 | True) 151 | add_sa_track_option(parser, 'cpu-nr-running', 152 | 'Show nr of tasks running for CPUs. Can be provided multiple times. \'CPU\' to show all cpus.', 153 | False) 154 | 155 | # sched module options 156 | parser.add_argument('--sched-report', action='store_true', 157 | help='Print a general summary of sched info for all tasks in the trace.') 158 | parser.add_argument('--sched-states', action='append', metavar='PATTERN', 159 | help='Print a summary of sched-states for a task in the trace. Can be provided multiple times.') 160 | parser.add_argument('--sched-states-parent', action='append', metavar='PATTERN', 161 | help='Print a summary of sched-states for a all tasks belonging to parent in the trace. Can be provided multiple times.') 162 | 163 | 164 | # settings module options 165 | parser.add_argument('--ts-start', type=float, 166 | help='Plot starting from this timestamp') 167 | parser.add_argument('--ts-end', type=float, 168 | help='Plot end with this timestamp') 169 | parser.add_argument('--fig-width-tui', type=int, 170 | help='Width of TUI text and figures') 171 | parser.add_argument('--fig-height-tui', type=int, 172 | help='Height of TUI figures') 173 | parser.add_argument('--xfrequency-tui', type=int, 174 | help='Frequency of x-axis of time based TUI figures') 175 | parser.add_argument('--yfrequency-tui', type=int, 176 | help='Frequency of y-axis of time based TUI figures') 177 | parser.add_argument('--grid-tui', action='store_true', 178 | help='Show grid on both axis in TUI figures') 179 | parser.add_argument('--grid-x-tui', action='store_true', 180 | help='Show grid on x-axis in TUI figures') 181 | parser.add_argument('--grid-y-tui', action='store_true', 182 | help='Show grid on y-axis in TUI figures') 183 | 184 | parser.add_argument('--theme', type=str, default='clear', 185 | choices=['default', 'clear', 'pro', 'matrix', 186 | 'windows', 'girly', 'dark', 'metro', 187 | 'elegant', 'grandpa', 'salad', 'serious', 188 | 'scream', 'dreamland', 'sand', 'mature'], 189 | help='TUI Color theme.') 190 | 191 | try: 192 | argcomplete.autocomplete(parser) 193 | except: 194 | pass 195 | 196 | return parser.parse_args() 197 | 198 | def main(): 199 | args = parse_cmdline() 200 | 201 | pltext.clf() 202 | pltext.cld() 203 | pltext.theme(args.theme) 204 | matplotlib.use('Agg') 205 | 206 | prefix = args.filename.replace('.perfetto-trace', '') 207 | 208 | trace = tp.get_trace(args.filename) 209 | 210 | utils.init(trace) 211 | 212 | # Handle settings 213 | settings.init() 214 | if args.ts_start: 215 | settings.set_ts_start(args.ts_start) 216 | if args.ts_end: 217 | settings.set_ts_end(args.ts_end) 218 | if args.fig_width_tui: 219 | settings.set_fig_width_tui(args.fig_width_tui) 220 | if args.fig_height_tui: 221 | settings.set_fig_height_tui(args.fig_height_tui) 222 | if args.xfrequency_tui: 223 | settings.set_xfrequncy_tui(args.xfrequency_tui) 224 | if args.yfrequency_tui: 225 | settings.set_yfrequncy_tui(args.yfrequency_tui) 226 | 227 | pltext.xfrequency(settings.xfrequency_tui) 228 | pltext.yfrequency(settings.yfrequency_tui) 229 | 230 | if (args.grid_tui): 231 | pltext.grid(True, True) 232 | if (args.grid_x_tui): 233 | pltext.grid(True, False) 234 | if (args.grid_y_tui): 235 | pltext.grid(False, True) 236 | 237 | if args.freq or args.freq_residency or args.freq_residency_abs or args.freq_task or args.freq_residency_task or args.freq_residency_abs_task: 238 | freq.init(trace) 239 | show_tui = True 240 | 241 | if args.save_csv: 242 | freq.save_csv(prefix) 243 | show_tui = False 244 | 245 | if args.freq: 246 | if (args.save_png): 247 | freq.plot_matplotlib(pltlib, prefix) 248 | show_tui = False 249 | 250 | if show_tui: 251 | freq.plot_tui(pltext) 252 | 253 | if args.freq_residency: 254 | if (args.save_png): 255 | freq.plot_residency_matplotlib(pltlib, prefix) 256 | show_tui = False 257 | 258 | if show_tui: 259 | freq.plot_residency_tui(pltext) 260 | 261 | if args.freq_residency_abs: 262 | if (args.save_png): 263 | freq.plot_residency_abs_matplotlib(pltlib, prefix) 264 | show_tui = False 265 | 266 | if show_tui: 267 | freq.plot_residency_abs_tui(pltext) 268 | 269 | if args.freq_task: 270 | if (args.save_png): 271 | print("freq-task --save-png: Not implemented yet.") 272 | show_tui = False 273 | 274 | if show_tui: 275 | freq.plot_task_tui(pltext, args.freq_task) 276 | 277 | if args.freq_residency_task: 278 | if (args.save_png): 279 | print("freq-residency-task --save-png: Not implemented yet.") 280 | show_tui = False 281 | 282 | if show_tui: 283 | freq.plot_task_residency_tui(pltext, args.freq_residency_task) 284 | 285 | if args.freq_residency_abs_task: 286 | if (args.save_png): 287 | print("freq-residency-abs-task --save-png: Not implemented yet.") 288 | show_tui = False 289 | 290 | if show_tui: 291 | freq.plot_task_residency_abs_tui(pltext, args.freq_residency_abs_task) 292 | 293 | if args.idle_residency or args.idle_residency_abs: 294 | idle.init(trace) 295 | show_tui = True 296 | 297 | if args.save_csv: 298 | idle.save_csv(prefix) 299 | show_tui = False 300 | 301 | if args.idle_residency: 302 | if (args.save_png): 303 | idle.plot_residency_matplotlib(pltlib, prefix) 304 | show_tui = False 305 | 306 | if show_tui: 307 | idle.plot_residency_tui(pltext) 308 | 309 | if args.idle_residency_abs: 310 | if (args.save_png): 311 | idle.plot_residency_abs_matplotlib(pltlib, prefix) 312 | show_tui = False 313 | 314 | if show_tui: 315 | idle.plot_residency_abs_tui(pltext) 316 | 317 | process_sa_track_option(args, trace, 'util_avg', True) 318 | process_sa_track_option(args, trace, 'util_avg_rt', False) 319 | process_sa_track_option(args, trace, 'util_avg_dl', False) 320 | process_sa_track_option(args, trace, 'util_avg_irq', False) 321 | process_sa_track_option(args, trace, 'util_est', True) 322 | process_sa_track_option(args, trace, 'load_avg_thermal', False) 323 | process_sa_track_option(args, trace, 'load_avg', True) 324 | process_sa_track_option(args, trace, 'runnable_avg', True) 325 | process_sa_track_option(args, trace, 'uclamped_avg', True) 326 | process_sa_track_option(args, trace, 'cpu_nr_running', False) 327 | 328 | if args.sched_report or args.sched_states or args.sched_states_parent: 329 | sched.init(trace) 330 | show_tui = True 331 | 332 | if args.save_csv: 333 | sched.states_save_csv(prefix) 334 | show_tui = False 335 | 336 | if args.save_png: 337 | print("sched-* --save-png: Not implemented yet.") 338 | show_tui = False 339 | 340 | if show_tui: 341 | if args.sched_report: 342 | sched.sched_report(pltext) 343 | if args.sched_states: 344 | sched.states_summary(pltext, args.sched_states) 345 | if args.sched_states_parent: 346 | sched.states_summary_parent(pltext, args.sched_states_parent) 347 | 348 | trace.close() 349 | 350 | if __name__ == '__main__': 351 | f = io.StringIO() 352 | with contextlib.redirect_stdout(f), contextlib.redirect_stderr(f): 353 | try: 354 | main() 355 | except: 356 | pass 357 | pydoc.pipepager(f.getvalue(), cmd='less -RF') 358 | -------------------------------------------------------------------------------- /sched-analyzer-pp/sched.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | # 3 | # SPDX-License-Identifier: GPL-2.0 4 | # Copyright (C) 2024 Qais Yousef 5 | 6 | import pandas as pd 7 | import settings 8 | import utils 9 | 10 | 11 | def init_states(trace): 12 | 13 | global df_states 14 | df_states = utils.get_df_states() 15 | if df_states.empty: 16 | return 17 | 18 | df_states.dur = df_states.dur.astype(float) / 1000000 19 | df_states = utils.convert_ts(df_states) 20 | 21 | def init(trace): 22 | 23 | pd.set_option('display.max_columns', None) 24 | pd.set_option('display.max_rows', None) 25 | pd.set_option('display.width', settings.fig_width_tui) 26 | 27 | init_states(trace) 28 | 29 | def states_summary(plt, threads=[], parent=None): 30 | 31 | if df_states.empty: 32 | return 33 | 34 | for thread in threads: 35 | df = df_states[df_states.name.str.contains(thread).fillna(False)] 36 | 37 | for thread in sorted(df.name.unique()): 38 | if parent: 39 | df_thread = df[(df.name == thread) & (df.parent == parent)] 40 | else: 41 | df_thread = df[df.name == thread] 42 | 43 | for tid in sorted(df_thread.tid.unique()): 44 | df_tid = df_thread[df_thread.tid == tid] 45 | df_tid_running = df_tid[df_tid.state == 'Running'] 46 | 47 | print() 48 | fmt = ":: {} | {} | {} ::".format(tid, thread, df_tid.parent.unique()) 49 | print("=" * len(fmt)) 50 | print(fmt) 51 | print("="*settings.fig_width_tui) 52 | print("-"*settings.fig_width_tui) 53 | states = sorted(df_tid.state.unique()) 54 | if 'S' in states: 55 | states.remove('S') 56 | data = [] 57 | total = 0 58 | for state in states: 59 | sum = df_tid[df_tid.state == state].dur.sum().round(2) 60 | total += sum 61 | data.append(sum) 62 | 63 | plt.clf() 64 | plt.cld() 65 | plt.simple_bar(states, data, width=settings.fig_width_tui, title="Sum Time in State Exclude Sleeping (ms)") 66 | plt.show() 67 | 68 | print() 69 | data = [d * settings.fig_width_tui / total for d in data] 70 | 71 | plt.clf() 72 | plt.cld() 73 | plt.simple_bar(states, data, width=settings.fig_width_tui, title="% Time in State Exclude Sleeping") 74 | plt.show() 75 | 76 | print() 77 | cpus = sorted(df_tid_running.cpu.unique()) 78 | labels = ['CPU{}'.format(cpu) for cpu in cpus] 79 | data = [] 80 | total = 0 81 | for cpu in cpus: 82 | df_cpu = df_tid_running[df_tid_running.cpu == cpu] 83 | sum = df_cpu.dur.sum().round(2) 84 | total += sum 85 | data.append(sum) 86 | 87 | plt.clf() 88 | plt.cld() 89 | plt.simple_bar(labels, data, width=settings.fig_width_tui, title="Sum Time Running on CPU (ms)") 90 | plt.show() 91 | 92 | print() 93 | data = [d * settings.fig_width_tui / total for d in data] 94 | 95 | plt.clf() 96 | plt.cld() 97 | plt.simple_bar(labels, data, width=settings.fig_width_tui, title="% Time Running on CPU") 98 | plt.show() 99 | 100 | print() 101 | print("Time in State (ms):") 102 | print("-"*settings.fig_width_tui) 103 | print(df_tid.groupby(['state']) \ 104 | .dur.describe(percentiles=[.75, .90, .95, .99]).round(2)) 105 | 106 | print() 107 | print("Time Running on CPU (ms):") 108 | print("-"*settings.fig_width_tui) 109 | print(df_tid_running.groupby(['cpu']) \ 110 | .dur.describe(percentiles=[.75, .90, .95, .99]).round(2)) 111 | 112 | def states_save_csv(prefix): 113 | 114 | df_states.to_csv(prefix + '_sched_states.csv') 115 | 116 | def sched_report(plt): 117 | 118 | nr_top = settings.fig_width_tui 119 | 120 | if df_states.empty: 121 | return 122 | 123 | print() 124 | print("States Summary (ms):") 125 | print("-"*settings.fig_width_tui) 126 | print(df_states.groupby('state').dur.describe(percentiles=[.75, .90, .95, .99]).round(2)) 127 | 128 | df_runnable = df_states[(df_states.state == 'R') | (df_states.state == 'R+')] 129 | df_running = df_states[df_states.state == 'Running'] 130 | df_usleep = df_states[df_states.state == 'D'] 131 | 132 | if not df_runnable.empty: 133 | print() 134 | print("Top {} Runnable Tasks (ms) - sorted-by max:".format(nr_top)) 135 | print("-"*settings.fig_width_tui) 136 | print(df_runnable.sort_values(['dur'], ascending=False) \ 137 | .groupby(['name', 'tid']) \ 138 | .dur.describe(percentiles=[.75, .90, .95, .99]) \ 139 | .round(2).sort_values(['max'], ascending=False) \ 140 | .head(nr_top)) 141 | print() 142 | print("Top {} Runnable Tasks (ms) - sorted-by 90%:".format(nr_top)) 143 | print("-"*settings.fig_width_tui) 144 | print(df_runnable.sort_values(['dur'], ascending=False) \ 145 | .groupby(['name', 'tid']) \ 146 | .dur.describe(percentiles=[.75, .90, .95, .99]) \ 147 | .round(2).sort_values(['90%'], ascending=False) \ 148 | .head(nr_top)) 149 | 150 | if not df_running.empty: 151 | print() 152 | print("Top {} Running Tasks (ms) - sorted-by max:".format(nr_top)) 153 | print("-"*settings.fig_width_tui) 154 | print(df_running.sort_values(['dur'], ascending=False) \ 155 | .groupby(['name', 'tid']) \ 156 | .dur.describe(percentiles=[.75, .90, .95, .99]) \ 157 | .round(2).sort_values(['max'], ascending=False) \ 158 | .head(nr_top)) 159 | print() 160 | print("Top {} Running Tasks (ms) - sorted-by 90%:".format(nr_top)) 161 | print("-"*settings.fig_width_tui) 162 | print(df_running.sort_values(['dur'], ascending=False) \ 163 | .groupby(['name', 'tid']) \ 164 | .dur.describe(percentiles=[.75, .90, .95, .99]) \ 165 | .round(2).sort_values(['90%'], ascending=False) \ 166 | .head(nr_top)) 167 | 168 | if not df_usleep.empty: 169 | print() 170 | print("Top {} Uninterruptible Sleep Tasks (ms) - sorted-by max:".format(nr_top)) 171 | print("-"*settings.fig_width_tui) 172 | print(df_usleep.sort_values(['dur'], ascending=False) \ 173 | .groupby(['name', 'tid']) \ 174 | .dur.describe(percentiles=[.75, .90, .95, .99]) \ 175 | .round(2).sort_values(['max'], ascending=False) \ 176 | .head(nr_top)) 177 | print() 178 | print("Top {} Uninterruptible Sleep Tasks (ms) - sorted-by 90%:".format(nr_top)) 179 | print("-"*settings.fig_width_tui) 180 | print(df_usleep.sort_values(['dur'], ascending=False) \ 181 | .groupby(['name', 'tid']) \ 182 | .dur.describe(percentiles=[.75, .90, .95, .99]) \ 183 | .round(2).sort_values(['90%'], ascending=False) \ 184 | .head(nr_top)) 185 | 186 | def states_summary_parent(plt, parents=[]): 187 | 188 | if df_states.empty: 189 | return 190 | 191 | for parent in parents: 192 | df = df_states[df_states.parent.str.contains(parent).fillna(False)] 193 | 194 | for parent in sorted(df.parent.unique()): 195 | df_parent = df[df.parent == parent] 196 | 197 | states_summary(plt, df_parent.name.unique(), parent) 198 | -------------------------------------------------------------------------------- /sched-analyzer-pp/settings.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | # 3 | # SPDX-License-Identifier: GPL-2.0 4 | # Copyright (C) 2024 Qais Yousef 5 | 6 | import os 7 | import pandas as pd 8 | import utils 9 | 10 | def init(): 11 | global ts_start 12 | global ts_end 13 | 14 | global fig_width_tui 15 | global fig_height_tui 16 | 17 | global xfrequency_tui 18 | global yfrequency_tui 19 | 20 | ts_start = 0 21 | ts_end = (utils.trace_end_ts - utils.trace_start_ts)/1000000000 22 | 23 | fig_width_tui = min(100, os.get_terminal_size()[0]) 24 | fig_height_tui = 10 25 | 26 | xfrequency_tui = 10 27 | yfrequency_tui = 5 28 | 29 | def set_ts_start(ts): 30 | global ts_start 31 | 32 | ts_start = ts 33 | 34 | def set_ts_end(ts): 35 | global ts_end 36 | 37 | ts_end = ts 38 | 39 | def filter_ts(df): 40 | global ts_start 41 | global ts_end 42 | 43 | return df[(df.index >= ts_start) & (df.index <= ts_end)] 44 | 45 | def set_fig_width_tui(width): 46 | global fig_width_tui 47 | 48 | fig_width_tui = width 49 | 50 | def set_fig_height_tui(height): 51 | global fig_height_tui 52 | 53 | fig_height_tui = height 54 | 55 | def set_xfrequncy_tui(frequency): 56 | global xfrequency_tui 57 | 58 | xfrequency_tui = frequency 59 | 60 | def set_yfrequncy_tui(frequency): 61 | global yfrequency_tui 62 | 63 | yfrequency_tui = frequency 64 | -------------------------------------------------------------------------------- /sched-analyzer-pp/trace_processor.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | # 3 | # SPDX-License-Identifier: GPL-2.0 4 | # Copyright (C) 2024 Qais Yousef 5 | 6 | from perfetto.trace_processor import TraceProcessor as tp 7 | from perfetto.trace_processor import TraceProcessorConfig as tpc 8 | 9 | def which(program): 10 | import os 11 | def is_exe(fpath): 12 | return os.path.isfile(fpath) and os.access(fpath, os.X_OK) 13 | 14 | fpath, fname = os.path.split(program) 15 | if fpath: 16 | if is_exe(program): 17 | return program 18 | else: 19 | for path in os.environ["PATH"].split(os.pathsep): 20 | exe_file = os.path.join(path, program) 21 | if is_exe(exe_file): 22 | return exe_file 23 | 24 | return None 25 | 26 | def get_trace(file): 27 | bin_path=which('trace_processor_shell') 28 | if bin_path: 29 | config = tpc(bin_path=bin_path) 30 | trace = tp(file_path=file, config=config) 31 | else: 32 | trace = tp(file_path=file) 33 | 34 | return trace 35 | -------------------------------------------------------------------------------- /sched-analyzer-pp/utils.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | # 3 | # SPDX-License-Identifier: GPL-2.0 4 | # Copyright (C) 2024 Qais Yousef 5 | 6 | import numpy as np 7 | import settings 8 | 9 | start_ts_query = "SELECT start_ts FROM trace_bounds" 10 | end_ts_query = "SELECT end_ts FROM trace_bounds" 11 | 12 | def init_states(trace): 13 | 14 | query = "select ts, cpu, state, dur, tid, t.name, p.name as parent \ 15 | from thread_state \ 16 | left join thread as t using(utid) \ 17 | left join process as p using(upid)" 18 | 19 | global trace_states 20 | trace_states = trace.query(query) 21 | 22 | global df_states 23 | df_states = trace_states.as_pandas_dataframe() 24 | if df_states.empty: 25 | return 26 | 27 | def init(trace): 28 | global trace_start_ts 29 | global trace_end_ts 30 | 31 | trace_start_ts = trace.query(start_ts_query).as_pandas_dataframe().values.item() 32 | trace_end_ts = trace.query(end_ts_query).as_pandas_dataframe().values.item() 33 | 34 | init_states(trace) 35 | 36 | def convert_ts(df, reindex=False, method='ffill'): 37 | global trace_start_ts 38 | global trace_end_ts 39 | 40 | if df.empty: 41 | return df 42 | 43 | # Act on a copy, don't modify the original dataframe which could be a slice 44 | df = df.copy() 45 | 46 | # Upsample to insert more accurate intermediate data and match 47 | # trace_start/end timestamps 48 | if reindex: 49 | df['index'] = df.ts 50 | df.set_index('index', inplace=True) 51 | # Sample every 100us 52 | new_index = np.arange(trace_start_ts, trace_end_ts, 100 * 1000) 53 | df = df[~df.index.duplicated()].reindex(new_index, method=method) 54 | df.ts = df.index 55 | 56 | # Convert to time in seconds starting from 0 57 | df.ts = df.ts - trace_start_ts 58 | df.ts = df.ts / 1000000000 59 | df['_ts'] = df.ts 60 | df.set_index('ts', inplace=True) 61 | 62 | # Filter timestamps based on user requested range 63 | df = settings.filter_ts(df) 64 | 65 | if df.dropna().empty: 66 | return df.dropna() 67 | 68 | return df 69 | 70 | def get_df_states(): 71 | global df_states 72 | 73 | return df_states 74 | 75 | def get_df_tid(thread, tid): 76 | global df_states 77 | 78 | df = df_states[df_states.name.str.contains(thread).fillna(False)] 79 | df_thread = df[df.name == thread] 80 | df_tid = df_thread[df_thread.tid == int(tid)] 81 | 82 | return df_tid.copy() 83 | 84 | # 85 | # Returns df[col] * df_tid['Running'] to obtain a new df that is values of 86 | # df[col] when the tid is RUNNING 87 | # 88 | # df must have been converted with utils.convert_ts() 89 | # 90 | def multiply_df_tid_running(df, col, df_tid): 91 | global trace_start_ts 92 | 93 | if df.empty or df_tid.empty: 94 | return df 95 | 96 | # Save last index to chop off after reindexing with ffill 97 | last_ts = (df_tid.ts.iloc[-1] - trace_start_ts)/1000000000 98 | df_tid = convert_ts(df_tid, True) 99 | # Chop of ffilled values after last_ts 100 | df_tid.loc[df_tid.index > last_ts, 'dur'] = None 101 | # Convert all positive values to 1 for multiply with freq 102 | df_tid.loc[df_tid.dur > 0, 'dur'] = 1 103 | # Drop duration for !Running so we account for freq during 104 | # Running time only 105 | df_tid.loc[df_tid.state != 'Running', 'dur'] = None 106 | 107 | # Now we'll get the df[col] seen by the task on @cpu 108 | df[col] = df[col] * df_tid.dur 109 | 110 | if df.dropna().empty: 111 | return df.dropna() 112 | 113 | return df 114 | 115 | # 116 | # Helper function to create 'residency' of @df grouped by @col 117 | # 118 | def gen_df_duration_groupby(df, col, abs=False): 119 | if df.empty or df.dropna().empty: 120 | return df 121 | 122 | df['duration'] = -1 * df._ts.diff(periods=-1) 123 | # dropna to get accurate total_duration 124 | total_duration = df.dropna().duration.sum() 125 | if not total_duration: 126 | total_duration = 1 127 | if abs: 128 | total_duration = 0.1 # Convert to absolute value in ms 129 | df_duration = df.groupby(col).duration.sum() * 100 / total_duration 130 | 131 | return df_duration 132 | 133 | # 134 | # Helper function to create 'residency' of @df filtered by @filter 135 | # 136 | def gen_df_duration_filtered(df, filter, abs=False): 137 | if df.empty or df.dropna().empty: 138 | return df 139 | 140 | df['duration'] = -1 * df._ts.diff(periods=-1) 141 | # dropna to get accurate total_duration 142 | total_duration = df.dropna().duration.sum() 143 | if not total_duration: 144 | total_duration = 1 145 | if abs: 146 | total_duration = 0.1 # Convert to absolute value in ms 147 | df_duration = df[filter].duration.sum() * 100 / total_duration 148 | 149 | return df_duration 150 | -------------------------------------------------------------------------------- /sched-analyzer.bpf.c: -------------------------------------------------------------------------------- 1 | /* SPDX-License-Identifier: GPL-2.0 */ 2 | /* Copyright (C) 2022 Qais Yousef */ 3 | #include "vmlinux.h" 4 | #include 5 | #include 6 | #include 7 | 8 | #include "parse_argp.h" 9 | #include "sched-analyzer-events.h" 10 | 11 | extern int LINUX_KERNEL_VERSION __kconfig; 12 | 13 | #define UTIL_AVG_UNCHANGED 0x80000000 14 | 15 | /* 16 | * Global variables shared with userspace counterpart. 17 | */ 18 | struct sa_opts sa_opts; 19 | 20 | char LICENSE[] SEC("license") = "GPL"; 21 | 22 | //#define DEBUG 23 | #ifndef DEBUG 24 | #undef bpf_printk 25 | #define bpf_printk(...) 26 | #endif 27 | 28 | /* 29 | * Compatibility defs - for when kernel changes struct fields. 30 | */ 31 | struct task_struct__old { 32 | int cpu; 33 | } __attribute__((preserve_access_index)); 34 | 35 | struct util_est { 36 | unsigned int enqueued; 37 | unsigned int ewma; 38 | }; 39 | 40 | struct sched_avg__pre68 { 41 | u64 last_update_time; 42 | u64 load_sum; 43 | u64 runnable_sum; 44 | u32 util_sum; 45 | u32 period_contrib; 46 | long unsigned int load_avg; 47 | long unsigned int runnable_avg; 48 | long unsigned int util_avg; 49 | struct util_est util_est; 50 | }; 51 | 52 | 53 | #define RB_SIZE (256 * 1024) 54 | 55 | struct { 56 | __uint(type, BPF_MAP_TYPE_HASH); 57 | __uint(max_entries, 8192); 58 | __type(key, pid_t); 59 | __type(value, int); 60 | } sched_switch SEC(".maps"); 61 | 62 | struct { 63 | __uint(type, BPF_MAP_TYPE_HASH); 64 | __uint(max_entries, 8192); 65 | __type(key, int); 66 | __type(value, u64); 67 | } softirq_entry SEC(".maps"); 68 | 69 | struct { 70 | __uint(type, BPF_MAP_TYPE_HASH); 71 | __uint(max_entries, 8192); 72 | __type(key, int); 73 | __type(value, int); 74 | } lb_map SEC(".maps"); 75 | 76 | /* 77 | * We define multiple ring buffers, one per event. 78 | */ 79 | struct { 80 | __uint(type, BPF_MAP_TYPE_RINGBUF); 81 | __uint(max_entries, RB_SIZE); 82 | } rq_pelt_rb SEC(".maps"); 83 | 84 | struct { 85 | __uint(type, BPF_MAP_TYPE_RINGBUF); 86 | __uint(max_entries, RB_SIZE); 87 | } task_pelt_rb SEC(".maps"); 88 | 89 | struct { 90 | __uint(type, BPF_MAP_TYPE_RINGBUF); 91 | __uint(max_entries, RB_SIZE); 92 | } rq_nr_running_rb SEC(".maps"); 93 | 94 | struct { 95 | __uint(type, BPF_MAP_TYPE_RINGBUF); 96 | __uint(max_entries, RB_SIZE); 97 | } sched_switch_rb SEC(".maps"); 98 | 99 | struct { 100 | __uint(type, BPF_MAP_TYPE_RINGBUF); 101 | __uint(max_entries, RB_SIZE); 102 | } freq_idle_rb SEC(".maps"); 103 | 104 | struct { 105 | __uint(type, BPF_MAP_TYPE_RINGBUF); 106 | __uint(max_entries, RB_SIZE); 107 | } softirq_rb SEC(".maps"); 108 | 109 | struct { 110 | __uint(type, BPF_MAP_TYPE_RINGBUF); 111 | __uint(max_entries, RB_SIZE); 112 | } lb_rb SEC(".maps"); 113 | 114 | struct { 115 | __uint(type, BPF_MAP_TYPE_RINGBUF); 116 | __uint(max_entries, RB_SIZE); 117 | } ipi_rb SEC(".maps"); 118 | 119 | static inline bool entity_is_task(struct sched_entity *se) 120 | { 121 | if (bpf_core_field_exists(se->my_q)) 122 | return !BPF_CORE_READ(se, my_q); 123 | else 124 | return true; 125 | } 126 | 127 | static inline struct rq *rq_of(struct cfs_rq *cfs_rq) 128 | { 129 | if (bpf_core_field_exists(cfs_rq->rq)) 130 | return BPF_CORE_READ(cfs_rq, rq); 131 | else 132 | return container_of(cfs_rq, struct rq, cfs); 133 | } 134 | 135 | static inline bool cfs_rq_is_root(struct cfs_rq *cfs_rq) 136 | { 137 | struct rq *rq = rq_of(cfs_rq); 138 | 139 | if (rq) 140 | return &rq->cfs == cfs_rq; 141 | else 142 | return false; 143 | } 144 | 145 | SEC("raw_tp/pelt_se_tp") 146 | int BPF_PROG(handle_pelt_se, struct sched_entity *se) 147 | { 148 | if (entity_is_task(se)) { 149 | struct task_struct *p = container_of(se, struct task_struct, se); 150 | unsigned long uclamp_min, uclamp_max; 151 | struct task_pelt_event *e; 152 | char comm[TASK_COMM_LEN]; 153 | int *running, cpu; 154 | pid_t pid; 155 | 156 | if (bpf_core_field_exists(p->wake_cpu)) { 157 | cpu = BPF_CORE_READ(p, wake_cpu); 158 | } else { 159 | struct task_struct__old *p_old = (void *)p; 160 | cpu = BPF_CORE_READ(p_old, cpu); 161 | } 162 | pid = BPF_CORE_READ(p, pid); 163 | BPF_CORE_READ_STR_INTO(&comm, p, comm); 164 | 165 | running = bpf_map_lookup_elem(&sched_switch, &pid); 166 | 167 | uclamp_min = -1; 168 | uclamp_max = -1; 169 | 170 | if (bpf_core_field_exists(p->uclamp_req[UCLAMP_MIN].value)) 171 | uclamp_min = BPF_CORE_READ_BITFIELD_PROBED(p, uclamp_req[UCLAMP_MIN].value); 172 | if (bpf_core_field_exists(p->uclamp_req[UCLAMP_MAX].value)) 173 | uclamp_max = BPF_CORE_READ_BITFIELD_PROBED(p, uclamp_req[UCLAMP_MAX].value); 174 | 175 | bpf_printk("[%s] Req: uclamp_min = %lu uclamp_max = %lu", 176 | comm, uclamp_min, uclamp_max); 177 | 178 | if (bpf_core_field_exists(p->uclamp[UCLAMP_MIN].value)) { 179 | bool active = BPF_CORE_READ_BITFIELD_PROBED(p, uclamp[UCLAMP_MIN].active); 180 | if (active) 181 | uclamp_min = BPF_CORE_READ_BITFIELD_PROBED(p, uclamp[UCLAMP_MIN].value); 182 | } 183 | if (bpf_core_field_exists(p->uclamp[UCLAMP_MAX].value)) { 184 | bool active = BPF_CORE_READ_BITFIELD_PROBED(p, uclamp[UCLAMP_MAX].active); 185 | if (active) 186 | uclamp_max = BPF_CORE_READ_BITFIELD_PROBED(p, uclamp[UCLAMP_MAX].value); 187 | } 188 | 189 | bpf_printk("[%s] Eff: uclamp_min = %lu uclamp_max = %lu", 190 | comm, uclamp_min, uclamp_max); 191 | 192 | e = bpf_ringbuf_reserve(&task_pelt_rb, sizeof(*e), 0); 193 | if (e) { 194 | e->ts = bpf_ktime_get_boot_ns(); 195 | e->cpu = cpu; 196 | e->pid = pid; 197 | BPF_CORE_READ_STR_INTO(&e->comm, p, comm); 198 | e->load_avg = BPF_CORE_READ(se, avg.load_avg); 199 | e->runnable_avg = BPF_CORE_READ(se, avg.runnable_avg); 200 | e->util_avg = BPF_CORE_READ(se, avg.util_avg); 201 | e->util_est_enqueued = -1; 202 | e->util_est_ewma = -1; 203 | e->uclamp_min = uclamp_min; 204 | e->uclamp_max = uclamp_max; 205 | if (running) 206 | e->running = 1; 207 | else 208 | e->running = 0; 209 | bpf_ringbuf_submit(e, 0); 210 | } 211 | } 212 | 213 | return 0; 214 | } 215 | 216 | SEC("raw_tp/sched_util_est_se_tp") 217 | int BPF_PROG(handle_util_est_se, struct sched_entity *se) 218 | { 219 | if (entity_is_task(se)) { 220 | struct task_struct *p = container_of(se, struct task_struct, se); 221 | unsigned long util_est_enqueued, util_est_ewma; 222 | struct task_pelt_event *e; 223 | char comm[TASK_COMM_LEN]; 224 | int *running, cpu; 225 | pid_t pid; 226 | 227 | if (bpf_core_field_exists(p->wake_cpu)) { 228 | cpu = BPF_CORE_READ(p, wake_cpu); 229 | } else { 230 | struct task_struct__old *p_old = (void *)p; 231 | cpu = BPF_CORE_READ(p_old, cpu); 232 | } 233 | pid = BPF_CORE_READ(p, pid); 234 | BPF_CORE_READ_STR_INTO(&comm, p, comm); 235 | 236 | running = bpf_map_lookup_elem(&sched_switch, &pid); 237 | 238 | if (LINUX_KERNEL_VERSION < KERNEL_VERSION(6, 8, 0)) { 239 | struct sched_avg__pre68 *avg_old = (void *)&se->avg; 240 | util_est_enqueued = BPF_PROBE_READ(avg_old, util_est.enqueued); 241 | util_est_ewma = BPF_PROBE_READ(avg_old, util_est.ewma); 242 | } else { 243 | util_est_enqueued = BPF_CORE_READ(se, avg.util_est); 244 | util_est_ewma = 0; 245 | } 246 | 247 | e = bpf_ringbuf_reserve(&task_pelt_rb, sizeof(*e), 0); 248 | if (e) { 249 | e->ts = bpf_ktime_get_boot_ns(); 250 | e->cpu = cpu; 251 | e->pid = pid; 252 | BPF_CORE_READ_STR_INTO(&e->comm, p, comm); 253 | e->load_avg = -1; 254 | e->runnable_avg = -1; 255 | e->util_avg = -1; 256 | e->util_est_enqueued = util_est_enqueued & ~UTIL_AVG_UNCHANGED; 257 | e->util_est_ewma = util_est_ewma; 258 | e->uclamp_min = -1; 259 | e->uclamp_max = -1; 260 | if (running) 261 | e->running = 1; 262 | else 263 | e->running = 0; 264 | bpf_ringbuf_submit(e, 0); 265 | } 266 | } 267 | 268 | return 0; 269 | } 270 | 271 | SEC("raw_tp/pelt_cfs_tp") 272 | int BPF_PROG(handle_pelt_cfs, struct cfs_rq *cfs_rq) 273 | { 274 | if (cfs_rq_is_root(cfs_rq)) { 275 | struct rq *rq = rq_of(cfs_rq); 276 | int cpu = BPF_CORE_READ(rq, cpu); 277 | struct rq_pelt_event *e; 278 | 279 | unsigned long uclamp_min = -1; 280 | unsigned long uclamp_max = -1; 281 | 282 | if (bpf_core_field_exists(rq->uclamp[UCLAMP_MIN].value)) 283 | uclamp_min = BPF_CORE_READ(rq, uclamp[UCLAMP_MIN].value); 284 | if (bpf_core_field_exists(rq->uclamp[UCLAMP_MAX].value)) 285 | uclamp_max = BPF_CORE_READ(rq, uclamp[UCLAMP_MAX].value); 286 | 287 | bpf_printk("cfs: [CPU%d] uclamp_min = %lu uclamp_max = %lu", 288 | cpu, uclamp_min, uclamp_max); 289 | 290 | e = bpf_ringbuf_reserve(&rq_pelt_rb, sizeof(*e), 0); 291 | if (e) { 292 | e->ts = bpf_ktime_get_boot_ns(); 293 | e->cpu = cpu; 294 | e->type = PELT_TYPE_CFS; 295 | e->load_avg = BPF_CORE_READ(cfs_rq, avg.load_avg); 296 | e->runnable_avg = BPF_CORE_READ(cfs_rq, avg.runnable_avg); 297 | e->util_avg = BPF_CORE_READ(cfs_rq, avg.util_avg); 298 | e->util_est_enqueued = -1; 299 | e->util_est_ewma = -1; 300 | e->uclamp_min = uclamp_min; 301 | e->uclamp_max = uclamp_max; 302 | bpf_ringbuf_submit(e, 0); 303 | } 304 | } 305 | 306 | return 0; 307 | } 308 | 309 | SEC("raw_tp/sched_util_est_cfs_tp") 310 | int BPF_PROG(handle_util_est_cfs, struct cfs_rq *cfs_rq) 311 | { 312 | if (cfs_rq_is_root(cfs_rq)) { 313 | unsigned long util_est_enqueued, util_est_ewma; 314 | struct rq *rq = rq_of(cfs_rq); 315 | int cpu = BPF_CORE_READ(rq, cpu); 316 | struct rq_pelt_event *e; 317 | 318 | 319 | if (LINUX_KERNEL_VERSION < KERNEL_VERSION(6, 8, 0)) { 320 | struct sched_avg__pre68 *avg_old = (void *)&cfs_rq->avg; 321 | util_est_enqueued = BPF_PROBE_READ(avg_old, util_est.enqueued); 322 | util_est_ewma = BPF_PROBE_READ(avg_old, util_est.ewma); 323 | } else { 324 | util_est_enqueued = BPF_CORE_READ(cfs_rq, avg.util_est); 325 | util_est_ewma = 0; 326 | } 327 | 328 | bpf_printk("cfs: [CPU%d] util_est.enqueued = %lu util_est.ewma = %lu", 329 | cpu, util_est_enqueued, util_est_ewma); 330 | 331 | e = bpf_ringbuf_reserve(&rq_pelt_rb, sizeof(*e), 0); 332 | if (e) { 333 | e->ts = bpf_ktime_get_boot_ns(); 334 | e->cpu = cpu; 335 | e->load_avg = -1; 336 | e->runnable_avg = -1; 337 | e->util_avg = -1; 338 | e->util_est_enqueued = util_est_enqueued & ~UTIL_AVG_UNCHANGED; 339 | e->util_est_ewma = util_est_ewma; 340 | e->uclamp_min = -1; 341 | e->uclamp_max = -1; 342 | bpf_ringbuf_submit(e, 0); 343 | } 344 | } 345 | 346 | return 0; 347 | } 348 | 349 | SEC("raw_tp/pelt_rt_tp") 350 | int BPF_PROG(handle_pelt_rt, struct rq *rq) 351 | { 352 | int cpu = BPF_CORE_READ(rq, cpu); 353 | struct rq_pelt_event *e; 354 | 355 | if (!bpf_core_field_exists(rq->avg_rt)) 356 | return 0; 357 | 358 | unsigned long util_avg = BPF_CORE_READ(rq, avg_rt.util_avg); 359 | 360 | e = bpf_ringbuf_reserve(&rq_pelt_rb, sizeof(*e), 0); 361 | if (e) { 362 | e->ts = bpf_ktime_get_boot_ns(); 363 | e->cpu = cpu; 364 | e->type = PELT_TYPE_RT; 365 | e->load_avg = -1; 366 | e->runnable_avg = -1; 367 | e->util_avg = util_avg; 368 | e->util_est_enqueued = -1; 369 | e->util_est_ewma = -1; 370 | e->uclamp_min = -1; 371 | e->uclamp_max = -1; 372 | bpf_ringbuf_submit(e, 0); 373 | } 374 | 375 | return 0; 376 | } 377 | 378 | SEC("raw_tp/pelt_dl_tp") 379 | int BPF_PROG(handle_pelt_dl, struct rq *rq) 380 | { 381 | int cpu = BPF_CORE_READ(rq, cpu); 382 | struct rq_pelt_event *e; 383 | 384 | if (!bpf_core_field_exists(rq->avg_dl)) 385 | return 0; 386 | 387 | unsigned long util_avg = BPF_CORE_READ(rq, avg_dl.util_avg); 388 | 389 | e = bpf_ringbuf_reserve(&rq_pelt_rb, sizeof(*e), 0); 390 | if (e) { 391 | e->ts = bpf_ktime_get_boot_ns(); 392 | e->cpu = cpu; 393 | e->type = PELT_TYPE_DL; 394 | e->load_avg = -1; 395 | e->runnable_avg = -1; 396 | e->util_avg = util_avg; 397 | e->util_est_enqueued = -1; 398 | e->util_est_ewma = -1; 399 | e->uclamp_min = -1; 400 | e->uclamp_max = -1; 401 | bpf_ringbuf_submit(e, 0); 402 | } 403 | 404 | return 0; 405 | } 406 | 407 | SEC("raw_tp/pelt_irq_tp") 408 | int BPF_PROG(handle_pelt_irq, struct rq *rq) 409 | { 410 | int cpu = BPF_CORE_READ(rq, cpu); 411 | struct rq_pelt_event *e; 412 | 413 | if (!bpf_core_field_exists(rq->avg_irq)) 414 | return 0; 415 | 416 | unsigned long util_avg = BPF_CORE_READ(rq, avg_irq.util_avg); 417 | 418 | e = bpf_ringbuf_reserve(&rq_pelt_rb, sizeof(*e), 0); 419 | if (e) { 420 | e->ts = bpf_ktime_get_boot_ns(); 421 | e->cpu = cpu; 422 | e->type = PELT_TYPE_IRQ; 423 | e->load_avg = -1; 424 | e->runnable_avg = -1; 425 | e->util_avg = util_avg; 426 | e->util_est_enqueued = -1; 427 | e->util_est_ewma = -1; 428 | e->uclamp_min = -1; 429 | e->uclamp_max = -1; 430 | bpf_ringbuf_submit(e, 0); 431 | } 432 | 433 | return 0; 434 | } 435 | 436 | SEC("raw_tp/pelt_thermal_tp") 437 | int BPF_PROG(handle_pelt_thermal, struct rq *rq) 438 | { 439 | int cpu = BPF_CORE_READ(rq, cpu); 440 | struct rq_pelt_event *e; 441 | 442 | if (!bpf_core_field_exists(rq->avg_thermal)) 443 | return 0; 444 | 445 | unsigned long load_avg = BPF_CORE_READ(rq, avg_thermal.load_avg); 446 | 447 | e = bpf_ringbuf_reserve(&rq_pelt_rb, sizeof(*e), 0); 448 | if (e) { 449 | e->ts = bpf_ktime_get_boot_ns(); 450 | e->cpu = cpu; 451 | e->type = PELT_TYPE_THERMAL; 452 | e->load_avg = load_avg; 453 | e->runnable_avg = -1; 454 | e->util_avg = -1; 455 | e->util_est_enqueued = -1; 456 | e->util_est_ewma = -1; 457 | e->uclamp_min = -1; 458 | e->uclamp_max = -1; 459 | bpf_ringbuf_submit(e, 0); 460 | } 461 | 462 | return 0; 463 | } 464 | 465 | SEC("raw_tp/sched_update_nr_running_tp") 466 | int BPF_PROG(handle_sched_update_nr_running, struct rq *rq, int change) 467 | { 468 | int cpu = BPF_CORE_READ(rq, cpu); 469 | struct rq_nr_running_event *e; 470 | 471 | int nr_running = BPF_CORE_READ(rq, nr_running); 472 | 473 | bpf_printk("[CPU%d] nr_running = %d change = %d", 474 | cpu, nr_running, change); 475 | 476 | e = bpf_ringbuf_reserve(&rq_nr_running_rb, sizeof(*e), 0); 477 | if (e) { 478 | e->ts = bpf_ktime_get_boot_ns(); 479 | e->cpu = cpu; 480 | e->nr_running = nr_running; 481 | e->change = change; 482 | bpf_ringbuf_submit(e, 0); 483 | } 484 | 485 | return 0; 486 | } 487 | 488 | SEC("raw_tp/sched_switch") 489 | int BPF_PROG(handle_sched_switch, bool preempt, 490 | struct task_struct *prev, struct task_struct *next) 491 | { 492 | int cpu; 493 | if (bpf_core_field_exists(prev->wake_cpu)) { 494 | cpu = BPF_CORE_READ(prev, wake_cpu); 495 | } else { 496 | struct task_struct__old *prev_old = (void*)prev; 497 | cpu = BPF_CORE_READ(prev_old, cpu); 498 | } 499 | struct sched_switch_event *e; 500 | char comm[TASK_COMM_LEN]; 501 | int running = 1; 502 | pid_t pid; 503 | 504 | pid = BPF_CORE_READ(prev, pid); 505 | bpf_map_delete_elem(&sched_switch, &pid); 506 | 507 | BPF_CORE_READ_STR_INTO(&comm, prev, comm); 508 | bpf_printk("[CPU%d] comm = %s running = %d", 509 | cpu, comm, 0); 510 | 511 | pid = BPF_CORE_READ(next, pid); 512 | bpf_map_update_elem(&sched_switch, &pid, &running, BPF_ANY); 513 | 514 | BPF_CORE_READ_STR_INTO(&comm, next, comm); 515 | bpf_printk("[CPU%d] comm = %s running = %d", 516 | cpu, comm, 1); 517 | 518 | e = bpf_ringbuf_reserve(&sched_switch_rb, sizeof(*e), 0); 519 | if (e) { 520 | e->ts = bpf_ktime_get_boot_ns(); 521 | e->cpu = cpu; 522 | e->pid = BPF_CORE_READ(prev, pid); 523 | BPF_CORE_READ_STR_INTO(&e->comm, prev, comm); 524 | e->running = 0; 525 | bpf_ringbuf_submit(e, 0); 526 | } 527 | 528 | e = bpf_ringbuf_reserve(&sched_switch_rb, sizeof(*e), 0); 529 | if (e) { 530 | e->ts = bpf_ktime_get_boot_ns(); 531 | e->cpu = cpu; 532 | e->pid = BPF_CORE_READ(next, pid); 533 | BPF_CORE_READ_STR_INTO(&e->comm, next, comm); 534 | e->running = 1; 535 | bpf_ringbuf_submit(e, 0); 536 | } 537 | 538 | return 0; 539 | } 540 | 541 | SEC("raw_tp/sched_process_free") 542 | int BPF_PROG(handle_sched_process_free, struct task_struct *p) 543 | { 544 | struct task_pelt_event *e; 545 | char comm[TASK_COMM_LEN]; 546 | pid_t pid; 547 | int cpu; 548 | 549 | if (bpf_core_field_exists(p->wake_cpu)) { 550 | cpu = BPF_CORE_READ(p, wake_cpu); 551 | } else { 552 | struct task_struct__old *p_old = (void *)p; 553 | cpu = BPF_CORE_READ(p_old, cpu); 554 | } 555 | pid = BPF_CORE_READ(p, pid); 556 | BPF_CORE_READ_STR_INTO(&comm, p, comm); 557 | 558 | e = bpf_ringbuf_reserve(&task_pelt_rb, sizeof(*e), 0); 559 | if (e) { 560 | e->ts = bpf_ktime_get_boot_ns(); 561 | e->cpu = cpu; 562 | e->pid = pid; 563 | BPF_CORE_READ_STR_INTO(&e->comm, p, comm); 564 | e->load_avg = 0; 565 | e->runnable_avg = 0; 566 | e->util_avg = -0; 567 | e->util_est_enqueued = 0; 568 | e->util_est_ewma = 0; 569 | e->uclamp_min = 0; 570 | e->uclamp_max = 0; 571 | e->running = 0; 572 | bpf_ringbuf_submit(e, 0); 573 | } 574 | 575 | return 0; 576 | } 577 | 578 | SEC("raw_tp/cpu_frequency") 579 | int BPF_PROG(handle_cpu_frequency, unsigned int frequency, unsigned int cpu) 580 | { 581 | struct freq_idle_event *e; 582 | int idle_state = -1; 583 | 584 | bpf_printk("[CPU%d] freq = %u idle_state = %u", 585 | cpu, frequency, idle_state); 586 | 587 | e = bpf_ringbuf_reserve(&freq_idle_rb, sizeof(*e), 0); 588 | if (e) { 589 | e->ts = bpf_ktime_get_boot_ns(); 590 | e->cpu = cpu; 591 | e->frequency = frequency; 592 | e->idle_state = idle_state; 593 | e->idle_miss = 0; 594 | bpf_ringbuf_submit(e, 0); 595 | } 596 | 597 | return 0; 598 | } 599 | 600 | SEC("raw_tp/cpu_idle") 601 | int BPF_PROG(handle_cpu_idle, unsigned int state, unsigned int cpu) 602 | { 603 | int idle_state = (int)state; 604 | unsigned int frequency = 0; 605 | struct freq_idle_event *e; 606 | 607 | bpf_printk("[CPU%d] freq = %u idle_state = %u", 608 | cpu, frequency, idle_state); 609 | 610 | e = bpf_ringbuf_reserve(&freq_idle_rb, sizeof(*e), 0); 611 | if (e) { 612 | e->ts = bpf_ktime_get_boot_ns(); 613 | e->cpu = cpu; 614 | e->frequency = frequency; 615 | e->idle_state = idle_state; 616 | e->idle_miss = 0; 617 | bpf_ringbuf_submit(e, 0); 618 | } 619 | 620 | return 0; 621 | } 622 | 623 | SEC("raw_tp/cpu_idle_miss") 624 | int BPF_PROG(handle_cpu_idle_miss, unsigned int cpu, 625 | unsigned int state, bool below) 626 | { 627 | int idle_state = (int)state; 628 | unsigned int frequency = 0; 629 | struct freq_idle_event *e; 630 | 631 | bpf_printk("[CPU%d] freq = %u idle_state = %u", 632 | cpu, frequency, idle_state); 633 | 634 | e = bpf_ringbuf_reserve(&freq_idle_rb, sizeof(*e), 0); 635 | if (e) { 636 | e->ts = bpf_ktime_get_boot_ns(); 637 | e->cpu = cpu; 638 | e->frequency = frequency; 639 | e->idle_state = idle_state; 640 | e->idle_miss = below ? -1 : 1; 641 | bpf_ringbuf_submit(e, 0); 642 | } 643 | 644 | return 0; 645 | } 646 | 647 | SEC("raw_tp/softirq_entry") 648 | int BPF_PROG(handle_softirq_entry, unsigned int vec_nr) 649 | { 650 | int cpu = bpf_get_smp_processor_id(); 651 | u64 ts = bpf_ktime_get_boot_ns(); 652 | bpf_map_update_elem(&softirq_entry, &cpu, &ts, BPF_ANY); 653 | 654 | return 0; 655 | } 656 | 657 | SEC("raw_tp/softirq_exit") 658 | int BPF_PROG(handle_softirq_exit, unsigned int vec_nr) 659 | { 660 | int cpu = bpf_get_smp_processor_id(); 661 | u64 exit_ts = bpf_ktime_get_boot_ns(); 662 | struct softirq_event *e; 663 | u64 entry_ts, *ts; 664 | 665 | ts = bpf_map_lookup_elem(&softirq_entry, &cpu); 666 | if (!ts) 667 | return 0; 668 | bpf_map_delete_elem(&softirq_entry, &cpu); 669 | 670 | entry_ts = *ts; 671 | 672 | e = bpf_ringbuf_reserve(&softirq_rb, sizeof(*e), 0); 673 | if (e) { 674 | e->ts = entry_ts; 675 | e->cpu = cpu; 676 | copy_softirq(e->softirq, vec_nr); 677 | e->duration = exit_ts - entry_ts; 678 | bpf_ringbuf_submit(e, 0); 679 | } 680 | 681 | return 0; 682 | } 683 | 684 | SEC("kprobe/_nohz_idle_balance.isra.0") 685 | int BPF_PROG(handle_nohz_idle_balance_entry, struct rq *rq) 686 | { 687 | int this_cpu = bpf_get_smp_processor_id(); 688 | int lb_cpu = BPF_CORE_READ(rq, cpu); 689 | u64 ts = bpf_ktime_get_boot_ns(); 690 | struct lb_event *e; 691 | 692 | int key = LB_NOHZ_IDLE_BALANCE << 16 | this_cpu; 693 | bpf_map_update_elem(&lb_map, &key, &lb_cpu, BPF_ANY); 694 | 695 | e = bpf_ringbuf_reserve(&lb_rb, sizeof(*e), 0); 696 | if (e) { 697 | e->ts = ts; 698 | e->this_cpu = this_cpu; 699 | e->lb_cpu = lb_cpu; 700 | e->phase = LB_NOHZ_IDLE_BALANCE; 701 | e->entry = true; 702 | e->overloaded = BPF_CORE_READ(rq, rd, overload); 703 | e->overutilized = BPF_CORE_READ(rq, rd, overutilized); 704 | e->misfit_task_load = -1; 705 | bpf_ringbuf_submit(e, 0); 706 | } 707 | 708 | return 0; 709 | } 710 | 711 | SEC("kretprobe/_nohz_idle_balance.isra.0") 712 | int BPF_PROG(handle_nohz_idle_balance_exit) 713 | { 714 | int this_cpu = bpf_get_smp_processor_id(); 715 | u64 ts = bpf_ktime_get_boot_ns(); 716 | struct lb_event *e; 717 | 718 | int key = LB_NOHZ_IDLE_BALANCE << 16 | this_cpu; 719 | int *lb_cpu = bpf_map_lookup_elem(&lb_map, &key); 720 | if (!lb_cpu) 721 | return 0; 722 | bpf_map_delete_elem(&lb_map, &key); 723 | 724 | e = bpf_ringbuf_reserve(&lb_rb, sizeof(*e), 0); 725 | if (e) { 726 | e->ts = ts; 727 | e->this_cpu = this_cpu; 728 | e->lb_cpu = *lb_cpu; 729 | e->phase = LB_NOHZ_IDLE_BALANCE; 730 | e->entry = false; 731 | e->overloaded = -1; 732 | e->overutilized = -1; 733 | e->misfit_task_load = -1; 734 | bpf_ringbuf_submit(e, 0); 735 | } 736 | 737 | return 0; 738 | } 739 | 740 | SEC("kprobe/run_rebalance_domains") 741 | int BPF_PROG(handle_run_rebalance_domains_entry) 742 | { 743 | int this_cpu = bpf_get_smp_processor_id(); 744 | u64 ts = bpf_ktime_get_boot_ns(); 745 | struct lb_event *e; 746 | 747 | e = bpf_ringbuf_reserve(&lb_rb, sizeof(*e), 0); 748 | if (e) { 749 | e->ts = ts; 750 | e->this_cpu = this_cpu; 751 | e->lb_cpu = this_cpu; 752 | e->phase = LB_RUN_REBALANCE_DOMAINS; 753 | e->entry = true; 754 | e->overloaded = -1; 755 | e->overutilized = -1; 756 | e->misfit_task_load = -1; 757 | bpf_ringbuf_submit(e, 0); 758 | } 759 | 760 | return 0; 761 | } 762 | 763 | SEC("kretprobe/run_rebalance_domains") 764 | int BPF_PROG(handle_run_rebalance_domains_exit) 765 | { 766 | int this_cpu = bpf_get_smp_processor_id(); 767 | u64 ts = bpf_ktime_get_boot_ns(); 768 | struct lb_event *e; 769 | 770 | e = bpf_ringbuf_reserve(&lb_rb, sizeof(*e), 0); 771 | if (e) { 772 | e->ts = ts; 773 | e->this_cpu = this_cpu; 774 | e->lb_cpu = this_cpu; 775 | e->phase = LB_RUN_REBALANCE_DOMAINS; 776 | e->entry = false; 777 | e->overloaded = -1; 778 | e->overutilized = -1; 779 | e->misfit_task_load = -1; 780 | bpf_ringbuf_submit(e, 0); 781 | } 782 | 783 | return 0; 784 | } 785 | 786 | static void gen_sched_domain_stats(struct rq *rq, enum cpu_idle_type idle, 787 | struct lb_sd_stats *sd_stats) 788 | { 789 | struct sched_domain *sd = BPF_CORE_READ(rq, sd); 790 | int cpu = BPF_CORE_READ(rq, cpu); 791 | int i = 0; 792 | 793 | bool sched_idle = BPF_CORE_READ(rq, nr_running) == BPF_CORE_READ(rq, cfs.idle_h_nr_running); 794 | sched_idle = sched_idle && BPF_CORE_READ(rq, nr_running); 795 | bool busy = idle != CPU_IDLE && !sched_idle; 796 | 797 | sd_stats->cpu = cpu; 798 | 799 | while (sd && i < MAX_SD_LEVELS) { 800 | unsigned int interval = BPF_CORE_READ(sd, balance_interval); 801 | if (busy) 802 | interval *= BPF_CORE_READ(sd, busy_factor); 803 | 804 | sd_stats->level[i] = BPF_CORE_READ(sd, level); 805 | sd_stats->balance_interval[i] = interval; 806 | 807 | sd = BPF_CORE_READ(sd, parent); 808 | i++; 809 | } 810 | 811 | sd_stats->level[i] = -1; 812 | sd_stats->balance_interval[i] = 0; 813 | } 814 | 815 | SEC("kprobe/rebalance_domains") 816 | int BPF_PROG(handle_rebalance_domains_entry, struct rq *rq, enum cpu_idle_type idle) 817 | { 818 | int this_cpu = bpf_get_smp_processor_id(); 819 | int lb_cpu = BPF_CORE_READ(rq, cpu); 820 | u64 ts = bpf_ktime_get_boot_ns(); 821 | struct lb_event *e; 822 | 823 | int key = LB_REBALANCE_DOMAINS << 16 | this_cpu; 824 | bpf_map_update_elem(&lb_map, &key, &lb_cpu, BPF_ANY); 825 | 826 | e = bpf_ringbuf_reserve(&lb_rb, sizeof(*e), 0); 827 | if (e) { 828 | e->ts = ts; 829 | e->this_cpu = this_cpu; 830 | e->lb_cpu = lb_cpu; 831 | e->phase = LB_REBALANCE_DOMAINS; 832 | e->entry = true; 833 | e->overloaded = BPF_CORE_READ(rq, rd, overload); 834 | e->overutilized = BPF_CORE_READ(rq, rd, overutilized); 835 | e->misfit_task_load = BPF_CORE_READ(rq, misfit_task_load); 836 | gen_sched_domain_stats(rq, idle, &e->sd_stats); 837 | bpf_ringbuf_submit(e, 0); 838 | } 839 | 840 | return 0; 841 | } 842 | 843 | SEC("kretprobe/rebalance_domains") 844 | int BPF_PROG(handle_rebalance_domains_exit) 845 | { 846 | int this_cpu = bpf_get_smp_processor_id(); 847 | u64 ts = bpf_ktime_get_boot_ns(); 848 | struct lb_event *e; 849 | 850 | int key = LB_REBALANCE_DOMAINS << 16 | this_cpu; 851 | int *lb_cpu = bpf_map_lookup_elem(&lb_map, &key); 852 | if (!lb_cpu) 853 | return 0; 854 | bpf_map_delete_elem(&lb_map, &key); 855 | 856 | e = bpf_ringbuf_reserve(&lb_rb, sizeof(*e), 0); 857 | if (e) { 858 | e->ts = ts; 859 | e->this_cpu = this_cpu; 860 | e->lb_cpu = *lb_cpu; 861 | e->phase = LB_REBALANCE_DOMAINS; 862 | e->entry = false; 863 | e->overloaded = -1; 864 | e->overutilized = -1; 865 | e->misfit_task_load = -1; 866 | bpf_ringbuf_submit(e, 0); 867 | } 868 | 869 | return 0; 870 | } 871 | 872 | SEC("kprobe/balance_fair") 873 | int BPF_PROG(handle_balance_fair_entry, struct rq *rq) 874 | { 875 | int this_cpu = bpf_get_smp_processor_id(); 876 | int lb_cpu = BPF_CORE_READ(rq, cpu); 877 | u64 ts = bpf_ktime_get_boot_ns(); 878 | struct lb_event *e; 879 | 880 | int key = LB_BALANCE_FAIR << 16 | this_cpu; 881 | bpf_map_update_elem(&lb_map, &key, &lb_cpu, BPF_ANY); 882 | 883 | e = bpf_ringbuf_reserve(&lb_rb, sizeof(*e), 0); 884 | if (e) { 885 | e->ts = ts; 886 | e->this_cpu = this_cpu; 887 | e->lb_cpu = lb_cpu; 888 | e->phase = LB_BALANCE_FAIR; 889 | e->entry = true; 890 | e->overloaded = BPF_CORE_READ(rq, rd, overload); 891 | e->overutilized = BPF_CORE_READ(rq, rd, overutilized); 892 | e->misfit_task_load = BPF_CORE_READ(rq, misfit_task_load); 893 | bpf_ringbuf_submit(e, 0); 894 | } 895 | 896 | return 0; 897 | } 898 | 899 | SEC("kretprobe/balance_fair") 900 | int BPF_PROG(handle_balance_fair_exit) 901 | { 902 | int this_cpu = bpf_get_smp_processor_id(); 903 | u64 ts = bpf_ktime_get_boot_ns(); 904 | struct lb_event *e; 905 | 906 | int key = LB_BALANCE_FAIR << 16 | this_cpu; 907 | int *lb_cpu = bpf_map_lookup_elem(&lb_map, &key); 908 | if (!lb_cpu) 909 | return 0; 910 | bpf_map_delete_elem(&lb_map, &key); 911 | 912 | e = bpf_ringbuf_reserve(&lb_rb, sizeof(*e), 0); 913 | if (e) { 914 | e->ts = ts; 915 | e->this_cpu = this_cpu; 916 | e->lb_cpu = *lb_cpu; 917 | e->phase = LB_BALANCE_FAIR; 918 | e->entry = false; 919 | e->overloaded = -1; 920 | e->overutilized = -1; 921 | e->misfit_task_load = -1; 922 | bpf_ringbuf_submit(e, 0); 923 | } 924 | 925 | return 0; 926 | } 927 | 928 | SEC("kprobe/pick_next_task_fair") 929 | int BPF_PROG(handle_pick_next_task_fair_entry, struct rq *rq) 930 | { 931 | int this_cpu = bpf_get_smp_processor_id(); 932 | int lb_cpu = BPF_CORE_READ(rq, cpu); 933 | u64 ts = bpf_ktime_get_boot_ns(); 934 | struct lb_event *e; 935 | 936 | int key = LB_PICK_NEXT_TASK_FAIR << 16 | this_cpu; 937 | bpf_map_update_elem(&lb_map, &key, &lb_cpu, BPF_ANY); 938 | 939 | e = bpf_ringbuf_reserve(&lb_rb, sizeof(*e), 0); 940 | if (e) { 941 | e->ts = ts; 942 | e->this_cpu = this_cpu; 943 | e->lb_cpu = lb_cpu; 944 | e->phase = LB_PICK_NEXT_TASK_FAIR; 945 | e->entry = true; 946 | e->overloaded = BPF_CORE_READ(rq, rd, overload); 947 | e->overutilized = BPF_CORE_READ(rq, rd, overutilized); 948 | e->misfit_task_load = BPF_CORE_READ(rq, misfit_task_load); 949 | bpf_ringbuf_submit(e, 0); 950 | } 951 | 952 | return 0; 953 | } 954 | 955 | SEC("kretprobe/pick_next_task_fair") 956 | int BPF_PROG(handle_pick_next_task_fair_exit) 957 | { 958 | int this_cpu = bpf_get_smp_processor_id(); 959 | u64 ts = bpf_ktime_get_boot_ns(); 960 | struct lb_event *e; 961 | 962 | int key = LB_PICK_NEXT_TASK_FAIR << 16 | this_cpu; 963 | int *lb_cpu = bpf_map_lookup_elem(&lb_map, &key); 964 | if (!lb_cpu) 965 | return 0; 966 | bpf_map_delete_elem(&lb_map, &key); 967 | 968 | e = bpf_ringbuf_reserve(&lb_rb, sizeof(*e), 0); 969 | if (e) { 970 | e->ts = ts; 971 | e->this_cpu = this_cpu; 972 | e->lb_cpu = *lb_cpu; 973 | e->phase = LB_PICK_NEXT_TASK_FAIR; 974 | e->entry = false; 975 | e->overloaded = -1; 976 | e->overutilized = -1; 977 | e->misfit_task_load = -1; 978 | bpf_ringbuf_submit(e, 0); 979 | } 980 | 981 | return 0; 982 | } 983 | 984 | SEC("kprobe/newidle_balance") 985 | int BPF_PROG(handle_newidle_balance_entry, struct rq *rq) 986 | { 987 | int this_cpu = bpf_get_smp_processor_id(); 988 | int lb_cpu = BPF_CORE_READ(rq, cpu); 989 | u64 ts = bpf_ktime_get_boot_ns(); 990 | struct lb_event *e; 991 | 992 | int key = LB_NEWIDLE_BALANCE << 16 | this_cpu; 993 | bpf_map_update_elem(&lb_map, &key, &lb_cpu, BPF_ANY); 994 | 995 | e = bpf_ringbuf_reserve(&lb_rb, sizeof(*e), 0); 996 | if (e) { 997 | e->ts = ts; 998 | e->this_cpu = this_cpu; 999 | e->lb_cpu = lb_cpu; 1000 | e->phase = LB_NEWIDLE_BALANCE; 1001 | e->entry = true; 1002 | e->overloaded = BPF_CORE_READ(rq, rd, overload); 1003 | e->overutilized = BPF_CORE_READ(rq, rd, overutilized); 1004 | e->misfit_task_load = BPF_CORE_READ(rq, misfit_task_load); 1005 | bpf_ringbuf_submit(e, 0); 1006 | } 1007 | 1008 | return 0; 1009 | } 1010 | 1011 | SEC("kretprobe/newidle_balance") 1012 | int BPF_PROG(handle_newidle_balance_exit) 1013 | { 1014 | int this_cpu = bpf_get_smp_processor_id(); 1015 | u64 ts = bpf_ktime_get_boot_ns(); 1016 | struct lb_event *e; 1017 | 1018 | int key = LB_NEWIDLE_BALANCE << 16 | this_cpu; 1019 | int *lb_cpu = bpf_map_lookup_elem(&lb_map, &key); 1020 | if (!lb_cpu) 1021 | return 0; 1022 | bpf_map_delete_elem(&lb_map, &key); 1023 | 1024 | e = bpf_ringbuf_reserve(&lb_rb, sizeof(*e), 0); 1025 | if (e) { 1026 | e->ts = ts; 1027 | e->this_cpu = this_cpu; 1028 | e->lb_cpu = *lb_cpu; 1029 | e->phase = LB_NEWIDLE_BALANCE; 1030 | e->entry = false; 1031 | e->overloaded = -1; 1032 | e->overutilized = -1; 1033 | e->misfit_task_load = -1; 1034 | bpf_ringbuf_submit(e, 0); 1035 | } 1036 | 1037 | return 0; 1038 | } 1039 | 1040 | SEC("kprobe/load_balance") 1041 | int BPF_PROG(handle_load_balance_entry, int lb_cpu, struct rq *lb_rq, 1042 | struct sched_domain *sd, enum cpu_idle_type idle) 1043 | { 1044 | int this_cpu = bpf_get_smp_processor_id(); 1045 | u64 ts = bpf_ktime_get_boot_ns(); 1046 | struct lb_event *e; 1047 | 1048 | int key = LB_LOAD_BALANCE << 16 | this_cpu; 1049 | bpf_map_update_elem(&lb_map, &key, &lb_cpu, BPF_ANY); 1050 | 1051 | e = bpf_ringbuf_reserve(&lb_rb, sizeof(*e), 0); 1052 | if (e) { 1053 | e->ts = ts; 1054 | e->this_cpu = this_cpu; 1055 | e->lb_cpu = lb_cpu; 1056 | e->phase = LB_LOAD_BALANCE; 1057 | e->entry = true; 1058 | e->overloaded = BPF_CORE_READ(lb_rq, rd, overload); 1059 | e->overutilized = BPF_CORE_READ(lb_rq, rd, overutilized); 1060 | e->misfit_task_load = BPF_CORE_READ(lb_rq, misfit_task_load); 1061 | bpf_ringbuf_submit(e, 0); 1062 | } 1063 | 1064 | return 0; 1065 | } 1066 | 1067 | SEC("kretprobe/load_balance") 1068 | int BPF_PROG(handle_load_balance_exit) 1069 | { 1070 | int this_cpu = bpf_get_smp_processor_id(); 1071 | u64 ts = bpf_ktime_get_boot_ns(); 1072 | struct lb_event *e; 1073 | 1074 | int key = LB_LOAD_BALANCE << 16 | this_cpu; 1075 | int *lb_cpu = bpf_map_lookup_elem(&lb_map, &key); 1076 | if (!lb_cpu) 1077 | return 0; 1078 | bpf_map_delete_elem(&lb_map, &key); 1079 | 1080 | e = bpf_ringbuf_reserve(&lb_rb, sizeof(*e), 0); 1081 | if (e) { 1082 | e->ts = ts; 1083 | e->this_cpu = this_cpu; 1084 | e->lb_cpu = *lb_cpu; 1085 | e->phase = LB_LOAD_BALANCE; 1086 | e->entry = false; 1087 | e->overloaded = -1; 1088 | e->overutilized = -1; 1089 | e->misfit_task_load = -1; 1090 | bpf_ringbuf_submit(e, 0); 1091 | } 1092 | 1093 | return 0; 1094 | } 1095 | 1096 | SEC("raw_tp/ipi_send_cpu") 1097 | int BPF_PROG(handle_ipi_send_cpu, int cpu, void *callsite, void *callback) 1098 | { 1099 | u64 ts = bpf_ktime_get_boot_ns(); 1100 | struct ipi_event *e; 1101 | 1102 | e = bpf_ringbuf_reserve(&ipi_rb, sizeof(*e), 0); 1103 | if (e) { 1104 | e->ts = ts; 1105 | e->from_cpu = bpf_get_smp_processor_id(); 1106 | e->target_cpu = cpu; 1107 | e->callsite = callsite; 1108 | e->callback = callback; 1109 | bpf_ringbuf_submit(e, 0); 1110 | } 1111 | 1112 | return 0; 1113 | } 1114 | -------------------------------------------------------------------------------- /sched-analyzer.c: -------------------------------------------------------------------------------- 1 | /* SPDX-License-Identifier: GPL-2.0 */ 2 | /* Copyright (C) 2022 Qais Yousef */ 3 | #include 4 | #include 5 | #include 6 | #include 7 | #include 8 | #include 9 | 10 | #include "parse_argp.h" 11 | #include "parse_kallsyms.h" 12 | #include "perfetto_wrapper.h" 13 | 14 | #include "sched-analyzer-events.h" 15 | #include "sched-analyzer.skel.h" 16 | 17 | #ifdef DEBUG 18 | #define pr_debug(...) fprintf(__VA_ARGS__) 19 | #else 20 | #define pr_debug(...) 21 | #endif 22 | 23 | #define clamp(val, lo, hi) ((val) >= (hi) ? (hi) : ((val) <= (lo) ? (lo) : (val))) 24 | 25 | static volatile bool exiting = false; 26 | 27 | static void sig_handler(int sig) 28 | { 29 | exiting = true; 30 | } 31 | 32 | static bool ignore_pid_comm(pid_t pid, char *comm) 33 | { 34 | unsigned int i; 35 | 36 | if (!sa_opts.num_pids && !sa_opts.num_comms) 37 | return false; 38 | 39 | for (i = 0; i < sa_opts.num_pids; i++) 40 | if (sa_opts.pid[i] == pid) 41 | return false; 42 | 43 | for (i = 0; i < sa_opts.num_comms; i++) 44 | if (strstr(comm, sa_opts.comm[i])) 45 | return false; 46 | 47 | return true; 48 | } 49 | 50 | static int handle_rq_pelt_event(void *ctx, void *data, size_t data_sz) 51 | { 52 | struct rq_pelt_event *e = data; 53 | 54 | if (sa_opts.load_avg_cpu && e->load_avg != -1) 55 | trace_cpu_load_avg(e->ts, e->cpu, e->load_avg); 56 | 57 | if (sa_opts.runnable_avg_cpu && e->runnable_avg != -1) 58 | trace_cpu_runnable_avg(e->ts, e->cpu, e->runnable_avg); 59 | 60 | if (e->type == PELT_TYPE_THERMAL){ 61 | if (sa_opts.load_avg_thermal) 62 | trace_cpu_load_avg_thermal(e->ts, e->cpu, e->load_avg); 63 | } 64 | 65 | if (e->util_avg != -1) { 66 | switch (e->type) { 67 | case PELT_TYPE_CFS: 68 | if (sa_opts.util_avg_cpu) { 69 | trace_cpu_util_avg(e->ts, e->cpu, e->util_avg); 70 | if (e->uclamp_min != -1 && e->uclamp_max != -1) { 71 | unsigned long uclamped_avg = clamp(e->util_avg, 72 | e->uclamp_min, 73 | e->uclamp_max); 74 | trace_cpu_uclamped_avg(e->ts, e->cpu, uclamped_avg); 75 | } 76 | } 77 | break; 78 | case PELT_TYPE_RT: 79 | if (sa_opts.util_avg_rt) 80 | trace_cpu_util_avg_rt(e->ts, e->cpu, e->util_avg); 81 | break; 82 | case PELT_TYPE_DL: 83 | if (sa_opts.util_avg_dl) 84 | trace_cpu_util_avg_dl(e->ts, e->cpu, e->util_avg); 85 | break; 86 | case PELT_TYPE_IRQ: 87 | if (sa_opts.util_avg_irq) 88 | trace_cpu_util_avg_irq(e->ts, e->cpu, e->util_avg); 89 | break; 90 | default: 91 | fprintf(stderr, "Unexpected PELT type: %d\n", e->type); 92 | break; 93 | } 94 | } 95 | 96 | if (sa_opts.util_est_cpu && e->util_est_enqueued != -1) 97 | trace_cpu_util_est_enqueued(e->ts, e->cpu, e->util_est_enqueued); 98 | 99 | return 0; 100 | } 101 | 102 | static int handle_task_pelt_event(void *ctx, void *data, size_t data_sz) 103 | { 104 | struct task_pelt_event *e = data; 105 | 106 | if (ignore_pid_comm(e->pid, e->comm)) 107 | return 0; 108 | 109 | if (sa_opts.load_avg_task && e->load_avg != -1) 110 | trace_task_load_avg(e->ts, e->comm, e->pid, e->load_avg); 111 | 112 | if (sa_opts.runnable_avg_task && e->runnable_avg != -1) 113 | trace_task_runnable_avg(e->ts, e->comm, e->pid, e->runnable_avg); 114 | 115 | if (sa_opts.util_avg_task && e->util_avg != -1) { 116 | trace_task_util_avg(e->ts, e->comm, e->pid, e->util_avg); 117 | if (e->uclamp_min != -1 && e->uclamp_max != -1) { 118 | unsigned long uclamped_avg = clamp(e->util_avg, 119 | e->uclamp_min, 120 | e->uclamp_max); 121 | trace_task_uclamped_avg(e->ts, e->comm, e->pid, uclamped_avg); 122 | } 123 | } 124 | 125 | if (sa_opts.util_est_task && e->util_est_enqueued != -1) { 126 | trace_task_util_est_enqueued(e->ts, e->comm, e->pid, e->util_est_enqueued); 127 | trace_task_util_est_ewma(e->ts, e->comm, e->pid, e->util_est_ewma); 128 | } 129 | 130 | return 0; 131 | } 132 | 133 | static int handle_rq_nr_running_event(void *ctx, void *data, size_t data_sz) 134 | { 135 | struct rq_nr_running_event *e = data; 136 | 137 | if (sa_opts.cpu_nr_running) 138 | trace_cpu_nr_running(e->ts, e->cpu, e->nr_running); 139 | 140 | return 0; 141 | } 142 | 143 | static int handle_sched_switch_event(void *ctx, void *data, size_t data_sz) 144 | { 145 | struct sched_switch_event *e = data; 146 | 147 | if (ignore_pid_comm(e->pid, e->comm)) 148 | return 0; 149 | 150 | /* Reset load_avg to 0 for !running */ 151 | if (!e->running && sa_opts.util_avg_task) 152 | trace_task_load_avg(e->ts, e->comm, e->pid, 0); 153 | 154 | /* Reset util_avg to 0 for !running */ 155 | if (!e->running && sa_opts.util_avg_task) { 156 | trace_task_util_avg(e->ts, e->comm, e->pid, 0); 157 | trace_task_uclamped_avg(e->ts, e->comm, e->pid, 0); 158 | } 159 | 160 | /* Reset util_est to 0 for !running */ 161 | if (!e->running && sa_opts.util_est_task) { 162 | trace_task_util_est_enqueued(e->ts, e->comm, e->pid, 0); 163 | trace_task_util_est_ewma(e->ts, e->comm, e->pid, 0); 164 | } 165 | 166 | return 0; 167 | } 168 | 169 | static int handle_freq_idle_event(void *ctx, void *data, size_t data_sz) 170 | { 171 | struct freq_idle_event *e = data; 172 | 173 | if (sa_opts.cpu_idle) { 174 | trace_cpu_idle(e->ts, e->cpu, e->idle_state); 175 | if (e->idle_miss) 176 | trace_cpu_idle_miss(e->ts, e->cpu, e->idle_state, e->idle_miss); 177 | } 178 | 179 | return 0; 180 | } 181 | 182 | static int handle_softirq_event(void *ctx, void *data, size_t data_sz) 183 | { 184 | struct softirq_event *e = data; 185 | (void)e; 186 | return 0; 187 | } 188 | 189 | static int handle_lb_event(void *ctx, void *data, size_t data_sz) 190 | { 191 | struct lb_event *e = data; 192 | char *phase = "unknown"; 193 | 194 | switch (e->phase) { 195 | case LB_NOHZ_IDLE_BALANCE: 196 | phase = "_nohz_idle_balance()"; 197 | break; 198 | case LB_RUN_REBALANCE_DOMAINS: 199 | phase = "run_rebalance_domains()"; 200 | break; 201 | case LB_REBALANCE_DOMAINS: 202 | phase = "rebalance_domains()"; 203 | if (e->entry) 204 | trace_lb_sd_stats(e->ts, &e->sd_stats); 205 | break; 206 | case LB_BALANCE_FAIR: 207 | phase = "balance_fair()"; 208 | break; 209 | case LB_PICK_NEXT_TASK_FAIR: 210 | phase = "pick_next_task_fair()"; 211 | break; 212 | case LB_NEWIDLE_BALANCE: 213 | phase = "newidle_balance()"; 214 | break; 215 | case LB_LOAD_BALANCE: 216 | phase = "load_balance()"; 217 | break; 218 | } 219 | 220 | if (e->overloaded != -1) 221 | trace_lb_overloaded(e->ts, e->overloaded); 222 | 223 | if (e->overutilized != -1) 224 | trace_lb_overutilized(e->ts, e->overutilized); 225 | 226 | if (e->misfit_task_load != -1) 227 | trace_lb_misfit(e->ts, e->lb_cpu, e->misfit_task_load); 228 | 229 | if (e->entry) 230 | trace_lb_entry(e->ts, e->this_cpu, e->lb_cpu, phase); 231 | else 232 | trace_lb_exit(e->ts, e->this_cpu, e->lb_cpu); 233 | return 0; 234 | } 235 | 236 | static int handle_ipi_event(void *ctx, void *data, size_t data_sz) 237 | { 238 | struct ipi_event *e = data; 239 | 240 | trace_ipi_send_cpu(e->ts, e->from_cpu, e->target_cpu, 241 | find_kallsyms(e->callsite), e->callsite, 242 | find_kallsyms(e->callback), e->callback); 243 | 244 | return 0; 245 | } 246 | 247 | #define INIT_EVENT_RB(event) struct ring_buffer *event##_rb = NULL 248 | 249 | #define CREATE_EVENT_RB(event) do { \ 250 | event##_rb = ring_buffer__new(bpf_map__fd(skel->maps.event##_rb), \ 251 | handle_##event##_event, NULL, NULL); \ 252 | if (!event##_rb) { \ 253 | err = -1; \ 254 | fprintf(stderr, "Failed to create " #event " ringbuffer\n"); \ 255 | goto cleanup; \ 256 | } \ 257 | } while(0) 258 | 259 | #define DESTROY_EVENT_RB(event) do { \ 260 | ring_buffer__free(event##_rb); \ 261 | } while(0) 262 | 263 | #define POLL_EVENT_RB(event) do { \ 264 | err = ring_buffer__poll(event##_rb, 1000); \ 265 | if (err == -EINTR) { \ 266 | err = 0; \ 267 | break; \ 268 | } \ 269 | if (err < 0) { \ 270 | fprintf(stderr, "Error polling " #event " ring buffer: %d\n", err); \ 271 | break; \ 272 | } \ 273 | pr_debug(stdout, "[" #event "] consumed %d events\n", err); \ 274 | } while(0) 275 | 276 | #define INIT_EVENT_THREAD(event) pthread_t event##_tid; int event##_err = -1 277 | 278 | #define CREATE_EVENT_THREAD(event) do { \ 279 | event##_err = pthread_create(&event##_tid, NULL, event##_thread_fn, NULL); \ 280 | if (event##_err) { \ 281 | fprintf(stderr, "Failed to create " #event " thread: %d\n", event##_err); \ 282 | goto cleanup; \ 283 | } \ 284 | } while(0) 285 | 286 | #define DESTROY_EVENT_THREAD(event) do { \ 287 | err = event##_err ? 0 : pthread_join(event##_tid, NULL); \ 288 | if (err) \ 289 | fprintf(stderr, "Failed to destory " #event " thread: %d\n", err); \ 290 | } while(0) 291 | 292 | #define EVENT_THREAD_FN(event) \ 293 | void *event##_thread_fn(void *data) \ 294 | { \ 295 | int err; \ 296 | INIT_EVENT_RB(event); \ 297 | CREATE_EVENT_RB(event); \ 298 | while (!exiting) { \ 299 | POLL_EVENT_RB(event); \ 300 | usleep(10000); \ 301 | } \ 302 | cleanup: \ 303 | DESTROY_EVENT_RB(event); \ 304 | return NULL; \ 305 | } 306 | 307 | 308 | /* 309 | * All events require to access this variable to get access to the ringbuffer. 310 | * Make it available for all event##_thread_fn. 311 | */ 312 | struct sched_analyzer_bpf *skel; 313 | 314 | /* 315 | * Define a pthread function handler for each event 316 | */ 317 | EVENT_THREAD_FN(rq_pelt) 318 | EVENT_THREAD_FN(task_pelt) 319 | EVENT_THREAD_FN(rq_nr_running) 320 | EVENT_THREAD_FN(sched_switch) 321 | EVENT_THREAD_FN(freq_idle) 322 | EVENT_THREAD_FN(softirq) 323 | EVENT_THREAD_FN(lb) 324 | EVENT_THREAD_FN(ipi) 325 | 326 | int main(int argc, char **argv) 327 | { 328 | INIT_EVENT_THREAD(rq_pelt); 329 | INIT_EVENT_THREAD(task_pelt); 330 | INIT_EVENT_THREAD(rq_nr_running); 331 | INIT_EVENT_THREAD(sched_switch); 332 | INIT_EVENT_THREAD(freq_idle); 333 | INIT_EVENT_THREAD(softirq); 334 | INIT_EVENT_THREAD(lb); 335 | INIT_EVENT_THREAD(ipi); 336 | int err; 337 | 338 | err = argp_parse(&argp, argc, argv, 0, NULL, NULL); 339 | if (err) 340 | return err; 341 | 342 | if (sa_opts.ipi) 343 | parse_kallsyms(); 344 | 345 | init_perfetto(); 346 | 347 | signal(SIGINT, sig_handler); 348 | signal(SIGTERM, sig_handler); 349 | 350 | skel = sched_analyzer_bpf__open(); 351 | if (!skel) { 352 | fprintf(stderr, "Failed to open and load BPF skeleton\n"); 353 | return 1; 354 | } 355 | 356 | /* Initialize BPF global variables */ 357 | skel->bss->sa_opts = sa_opts; 358 | 359 | if (!sa_opts.load_avg_cpu && !sa_opts.runnable_avg_cpu && !sa_opts.util_avg_cpu) 360 | bpf_program__set_autoload(skel->progs.handle_pelt_cfs, false); 361 | if (!sa_opts.load_avg_task && !sa_opts.runnable_avg_task && !sa_opts.util_avg_task) 362 | bpf_program__set_autoload(skel->progs.handle_pelt_se, false); 363 | if (!sa_opts.util_avg_rt) 364 | bpf_program__set_autoload(skel->progs.handle_pelt_rt, false); 365 | if (!sa_opts.util_avg_dl) 366 | bpf_program__set_autoload(skel->progs.handle_pelt_dl, false); 367 | if (!sa_opts.util_avg_irq) 368 | bpf_program__set_autoload(skel->progs.handle_pelt_irq, false); 369 | if (!sa_opts.load_avg_thermal) 370 | bpf_program__set_autoload(skel->progs.handle_pelt_thermal, false); 371 | if (!sa_opts.util_est_cpu) 372 | bpf_program__set_autoload(skel->progs.handle_util_est_cfs, false); 373 | if (!sa_opts.util_est_task) 374 | bpf_program__set_autoload(skel->progs.handle_util_est_se, false); 375 | if (!sa_opts.cpu_nr_running) 376 | bpf_program__set_autoload(skel->progs.handle_sched_update_nr_running, false); 377 | if (!sa_opts.cpu_idle) { 378 | bpf_program__set_autoload(skel->progs.handle_cpu_idle, false); 379 | bpf_program__set_autoload(skel->progs.handle_cpu_idle_miss, false); 380 | } 381 | if (!sa_opts.load_balance) { 382 | bpf_program__set_autoload(skel->progs.handle_run_rebalance_domains_exit, false); 383 | bpf_program__set_autoload(skel->progs.handle_run_rebalance_domains_entry, false); 384 | bpf_program__set_autoload(skel->progs.handle_run_rebalance_domains_exit, false); 385 | bpf_program__set_autoload(skel->progs.handle_rebalance_domains_entry, false); 386 | bpf_program__set_autoload(skel->progs.handle_rebalance_domains_exit, false); 387 | bpf_program__set_autoload(skel->progs.handle_balance_fair_entry, false); 388 | bpf_program__set_autoload(skel->progs.handle_balance_fair_exit, false); 389 | bpf_program__set_autoload(skel->progs.handle_pick_next_task_fair_entry, false); 390 | bpf_program__set_autoload(skel->progs.handle_pick_next_task_fair_exit, false); 391 | bpf_program__set_autoload(skel->progs.handle_newidle_balance_entry, false); 392 | bpf_program__set_autoload(skel->progs.handle_newidle_balance_exit, false); 393 | bpf_program__set_autoload(skel->progs.handle_load_balance_entry, false); 394 | bpf_program__set_autoload(skel->progs.handle_load_balance_exit, false); 395 | } 396 | if (!sa_opts.ipi) 397 | bpf_program__set_autoload(skel->progs.handle_ipi_send_cpu, false); 398 | 399 | /* Make sure we zero out PELT signals for tasks when they exit */ 400 | if (!sa_opts.load_avg_task && !sa_opts.runnable_avg_task && !sa_opts.util_avg_task && !sa_opts.util_est_task) 401 | bpf_program__set_autoload(skel->progs.handle_sched_process_free, false); 402 | 403 | /* We can't reliably attach to those yet, so always disable them */ 404 | bpf_program__set_autoload(skel->progs.handle_nohz_idle_balance_entry, false); 405 | bpf_program__set_autoload(skel->progs.handle_nohz_idle_balance_exit, false); 406 | 407 | /* 408 | * Were used for old csv mode, no longer used but keep the traces lying 409 | * around but disabled for now. 410 | */ 411 | bpf_program__set_autoload(skel->progs.handle_cpu_frequency, false); 412 | bpf_program__set_autoload(skel->progs.handle_softirq_entry, false); 413 | bpf_program__set_autoload(skel->progs.handle_softirq_exit, false); 414 | 415 | /* 416 | * Was used to zero out pelt signals when task is not running. 417 | */ 418 | bpf_program__set_autoload(skel->progs.handle_sched_switch, false); 419 | 420 | 421 | err = sched_analyzer_bpf__load(skel); 422 | if (err) { 423 | fprintf(stderr, "Failed to load and verify BPF skeleton\n"); 424 | goto cleanup; 425 | } 426 | 427 | err = sched_analyzer_bpf__attach(skel); 428 | if (err) { 429 | fprintf(stderr, "Failed to attach BPF skeleton\n"); 430 | goto cleanup; 431 | } 432 | 433 | CREATE_EVENT_THREAD(rq_pelt); 434 | CREATE_EVENT_THREAD(task_pelt); 435 | CREATE_EVENT_THREAD(rq_nr_running); 436 | CREATE_EVENT_THREAD(sched_switch); 437 | CREATE_EVENT_THREAD(freq_idle); 438 | CREATE_EVENT_THREAD(softirq); 439 | CREATE_EVENT_THREAD(lb); 440 | CREATE_EVENT_THREAD(ipi); 441 | 442 | printf("Collecting data, CTRL+c to stop\n"); 443 | 444 | start_perfetto_trace(); 445 | 446 | while (!exiting) { 447 | sleep(1); 448 | } 449 | 450 | stop_perfetto_trace(); 451 | 452 | printf("\rCollected %s/%s\n", sa_opts.output_path, sa_opts.output); 453 | 454 | cleanup: 455 | DESTROY_EVENT_THREAD(rq_pelt); 456 | DESTROY_EVENT_THREAD(task_pelt); 457 | DESTROY_EVENT_THREAD(rq_nr_running); 458 | DESTROY_EVENT_THREAD(sched_switch); 459 | DESTROY_EVENT_THREAD(freq_idle); 460 | DESTROY_EVENT_THREAD(softirq); 461 | DESTROY_EVENT_THREAD(lb); 462 | DESTROY_EVENT_THREAD(ipi); 463 | sched_analyzer_bpf__destroy(skel); 464 | return err < 0 ? -err : 0; 465 | } 466 | -------------------------------------------------------------------------------- /screenshots/sched-analyzer-screenshot-ipi.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/qais-yousef/sched-analyzer/c7a47a2e55049b2d99129b2c73711e1292292d12/screenshots/sched-analyzer-screenshot-ipi.png -------------------------------------------------------------------------------- /screenshots/sched-analyzer-screenshot-pelt-filtered.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/qais-yousef/sched-analyzer/c7a47a2e55049b2d99129b2c73711e1292292d12/screenshots/sched-analyzer-screenshot-pelt-filtered.png -------------------------------------------------------------------------------- /screenshots/sched-analyzer-screenshot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/qais-yousef/sched-analyzer/c7a47a2e55049b2d99129b2c73711e1292292d12/screenshots/sched-analyzer-screenshot.png --------------------------------------------------------------------------------