├── .gitignore ├── Dockerfile ├── LICENSE ├── Makefile ├── README.md ├── bpf ├── __init__.py ├── bpf_monitor.c ├── tcp_monitor.c └── vfs_monitor.c ├── config.yaml ├── deep_mon.py ├── setup.py └── userspace ├── __init__.py ├── bpf_collector.py ├── container_info.py ├── curse.py ├── default_config.yaml ├── disk_collector.py ├── mem_collector.py ├── monitor_main.py ├── net_collector.py ├── proc_topology.py ├── process_info.py ├── process_table.py ├── rapl ├── __init__.py └── rapl.py └── sample_controller.py /.gitignore: -------------------------------------------------------------------------------- 1 | *.swp 2 | venv 3 | *.pyc 4 | ./idea 5 | .vscode/ -------------------------------------------------------------------------------- /Dockerfile: -------------------------------------------------------------------------------- 1 | FROM ubuntu:18.04 2 | LABEL maintainer="Rolando Brondolin" 3 | 4 | RUN apt-get clean && apt-get update \ 5 | && apt-get install -y python3 python3-pip locales locales-all libelf1 \ 6 | && rm -rf /var/lib/apt/lists/* 7 | 8 | RUN pip3 install --upgrade pip && pip3 install numpy pyyaml docker 9 | 10 | 11 | #Needed by Curse to print unicode characters to the terminal 12 | ENV LC_ALL en_US.UTF-8 13 | ENV LANG en_US.UTF-8 14 | ENV LANGUAGE en_US.UTF-8 15 | 16 | #install bcc and ddsketch 17 | 18 | RUN buildDeps='python python-pip wget curl apt-transport-https git bison build-essential cmake flex libedit-dev libllvm6.0 llvm-6.0-dev libclang-6.0-dev zlib1g-dev libelf-dev' \ 19 | && apt-get update && apt-get install -y $buildDeps \ 20 | && git clone https://github.com/iovisor/bcc.git \ 21 | && mkdir bcc/build \ 22 | && cd bcc/build \ 23 | && cmake .. \ 24 | && make \ 25 | && make install \ 26 | && cmake -DPYTHON_CMD=python3 .. \ 27 | && cd src/python/ \ 28 | && make \ 29 | && make install \ 30 | && cd / \ 31 | && rm -r bcc \ 32 | && git clone --branch v1.0 https://github.com/DataDog/sketches-py.git \ 33 | && cd sketches-py \ 34 | && python3 setup.py install \ 35 | && cd / \ 36 | && rm -r sketches-py \ 37 | && apt-get purge -y --auto-remove $buildDeps \ 38 | && rm -rf /var/lib/apt/lists/* 39 | 40 | WORKDIR /home 41 | 42 | RUN mkdir /home/deep_mon 43 | ADD bpf /home/deep_mon/bpf 44 | ADD userspace /home/deep_mon/userspace 45 | ADD deep_mon.py /home/deep_mon/ 46 | ADD setup.py /home 47 | 48 | #Install plugin dependencies 49 | RUN pip3 install . && rm -rf /home/deep_mon && rm setup.py 50 | 51 | # "-u" forces the binary I/O layers of stdout and stderr to be unbuffered and 52 | # is needed to avoid truncated output in Docker 53 | ENV PYTHONUNBUFFERED="on" 54 | CMD ["deep-mon"] 55 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | GNU GENERAL PUBLIC LICENSE 2 | Version 3, 29 June 2007 3 | 4 | Copyright (C) 2007 Free Software Foundation, Inc. 5 | Everyone is permitted to copy and distribute verbatim copies 6 | of this license document, but changing it is not allowed. 7 | 8 | Preamble 9 | 10 | The GNU General Public License is a free, copyleft license for 11 | software and other kinds of works. 12 | 13 | The licenses for most software and other practical works are designed 14 | to take away your freedom to share and change the works. By contrast, 15 | the GNU General Public License is intended to guarantee your freedom to 16 | share and change all versions of a program--to make sure it remains free 17 | software for all its users. We, the Free Software Foundation, use the 18 | GNU General Public License for most of our software; it applies also to 19 | any other work released this way by its authors. You can apply it to 20 | your programs, too. 21 | 22 | When we speak of free software, we are referring to freedom, not 23 | price. Our General Public Licenses are designed to make sure that you 24 | have the freedom to distribute copies of free software (and charge for 25 | them if you wish), that you receive source code or can get it if you 26 | want it, that you can change the software or use pieces of it in new 27 | free programs, and that you know you can do these things. 28 | 29 | To protect your rights, we need to prevent others from denying you 30 | these rights or asking you to surrender the rights. Therefore, you have 31 | certain responsibilities if you distribute copies of the software, or if 32 | you modify it: responsibilities to respect the freedom of others. 33 | 34 | For example, if you distribute copies of such a program, whether 35 | gratis or for a fee, you must pass on to the recipients the same 36 | freedoms that you received. You must make sure that they, too, receive 37 | or can get the source code. And you must show them these terms so they 38 | know their rights. 39 | 40 | Developers that use the GNU GPL protect your rights with two steps: 41 | (1) assert copyright on the software, and (2) offer you this License 42 | giving you legal permission to copy, distribute and/or modify it. 43 | 44 | For the developers' and authors' protection, the GPL clearly explains 45 | that there is no warranty for this free software. For both users' and 46 | authors' sake, the GPL requires that modified versions be marked as 47 | changed, so that their problems will not be attributed erroneously to 48 | authors of previous versions. 49 | 50 | Some devices are designed to deny users access to install or run 51 | modified versions of the software inside them, although the manufacturer 52 | can do so. This is fundamentally incompatible with the aim of 53 | protecting users' freedom to change the software. The systematic 54 | pattern of such abuse occurs in the area of products for individuals to 55 | use, which is precisely where it is most unacceptable. Therefore, we 56 | have designed this version of the GPL to prohibit the practice for those 57 | products. If such problems arise substantially in other domains, we 58 | stand ready to extend this provision to those domains in future versions 59 | of the GPL, as needed to protect the freedom of users. 60 | 61 | Finally, every program is threatened constantly by software patents. 62 | States should not allow patents to restrict development and use of 63 | software on general-purpose computers, but in those that do, we wish to 64 | avoid the special danger that patents applied to a free program could 65 | make it effectively proprietary. To prevent this, the GPL assures that 66 | patents cannot be used to render the program non-free. 67 | 68 | The precise terms and conditions for copying, distribution and 69 | modification follow. 70 | 71 | TERMS AND CONDITIONS 72 | 73 | 0. Definitions. 74 | 75 | "This License" refers to version 3 of the GNU General Public License. 76 | 77 | "Copyright" also means copyright-like laws that apply to other kinds of 78 | works, such as semiconductor masks. 79 | 80 | "The Program" refers to any copyrightable work licensed under this 81 | License. Each licensee is addressed as "you". "Licensees" and 82 | "recipients" may be individuals or organizations. 83 | 84 | To "modify" a work means to copy from or adapt all or part of the work 85 | in a fashion requiring copyright permission, other than the making of an 86 | exact copy. The resulting work is called a "modified version" of the 87 | earlier work or a work "based on" the earlier work. 88 | 89 | A "covered work" means either the unmodified Program or a work based 90 | on the Program. 91 | 92 | To "propagate" a work means to do anything with it that, without 93 | permission, would make you directly or secondarily liable for 94 | infringement under applicable copyright law, except executing it on a 95 | computer or modifying a private copy. Propagation includes copying, 96 | distribution (with or without modification), making available to the 97 | public, and in some countries other activities as well. 98 | 99 | To "convey" a work means any kind of propagation that enables other 100 | parties to make or receive copies. Mere interaction with a user through 101 | a computer network, with no transfer of a copy, is not conveying. 102 | 103 | An interactive user interface displays "Appropriate Legal Notices" 104 | to the extent that it includes a convenient and prominently visible 105 | feature that (1) displays an appropriate copyright notice, and (2) 106 | tells the user that there is no warranty for the work (except to the 107 | extent that warranties are provided), that licensees may convey the 108 | work under this License, and how to view a copy of this License. If 109 | the interface presents a list of user commands or options, such as a 110 | menu, a prominent item in the list meets this criterion. 111 | 112 | 1. Source Code. 113 | 114 | The "source code" for a work means the preferred form of the work 115 | for making modifications to it. "Object code" means any non-source 116 | form of a work. 117 | 118 | A "Standard Interface" means an interface that either is an official 119 | standard defined by a recognized standards body, or, in the case of 120 | interfaces specified for a particular programming language, one that 121 | is widely used among developers working in that language. 122 | 123 | The "System Libraries" of an executable work include anything, other 124 | than the work as a whole, that (a) is included in the normal form of 125 | packaging a Major Component, but which is not part of that Major 126 | Component, and (b) serves only to enable use of the work with that 127 | Major Component, or to implement a Standard Interface for which an 128 | implementation is available to the public in source code form. A 129 | "Major Component", in this context, means a major essential component 130 | (kernel, window system, and so on) of the specific operating system 131 | (if any) on which the executable work runs, or a compiler used to 132 | produce the work, or an object code interpreter used to run it. 133 | 134 | The "Corresponding Source" for a work in object code form means all 135 | the source code needed to generate, install, and (for an executable 136 | work) run the object code and to modify the work, including scripts to 137 | control those activities. However, it does not include the work's 138 | System Libraries, or general-purpose tools or generally available free 139 | programs which are used unmodified in performing those activities but 140 | which are not part of the work. For example, Corresponding Source 141 | includes interface definition files associated with source files for 142 | the work, and the source code for shared libraries and dynamically 143 | linked subprograms that the work is specifically designed to require, 144 | such as by intimate data communication or control flow between those 145 | subprograms and other parts of the work. 146 | 147 | The Corresponding Source need not include anything that users 148 | can regenerate automatically from other parts of the Corresponding 149 | Source. 150 | 151 | The Corresponding Source for a work in source code form is that 152 | same work. 153 | 154 | 2. Basic Permissions. 155 | 156 | All rights granted under this License are granted for the term of 157 | copyright on the Program, and are irrevocable provided the stated 158 | conditions are met. This License explicitly affirms your unlimited 159 | permission to run the unmodified Program. The output from running a 160 | covered work is covered by this License only if the output, given its 161 | content, constitutes a covered work. This License acknowledges your 162 | rights of fair use or other equivalent, as provided by copyright law. 163 | 164 | You may make, run and propagate covered works that you do not 165 | convey, without conditions so long as your license otherwise remains 166 | in force. You may convey covered works to others for the sole purpose 167 | of having them make modifications exclusively for you, or provide you 168 | with facilities for running those works, provided that you comply with 169 | the terms of this License in conveying all material for which you do 170 | not control copyright. Those thus making or running the covered works 171 | for you must do so exclusively on your behalf, under your direction 172 | and control, on terms that prohibit them from making any copies of 173 | your copyrighted material outside their relationship with you. 174 | 175 | Conveying under any other circumstances is permitted solely under 176 | the conditions stated below. Sublicensing is not allowed; section 10 177 | makes it unnecessary. 178 | 179 | 3. Protecting Users' Legal Rights From Anti-Circumvention Law. 180 | 181 | No covered work shall be deemed part of an effective technological 182 | measure under any applicable law fulfilling obligations under article 183 | 11 of the WIPO copyright treaty adopted on 20 December 1996, or 184 | similar laws prohibiting or restricting circumvention of such 185 | measures. 186 | 187 | When you convey a covered work, you waive any legal power to forbid 188 | circumvention of technological measures to the extent such circumvention 189 | is effected by exercising rights under this License with respect to 190 | the covered work, and you disclaim any intention to limit operation or 191 | modification of the work as a means of enforcing, against the work's 192 | users, your or third parties' legal rights to forbid circumvention of 193 | technological measures. 194 | 195 | 4. Conveying Verbatim Copies. 196 | 197 | You may convey verbatim copies of the Program's source code as you 198 | receive it, in any medium, provided that you conspicuously and 199 | appropriately publish on each copy an appropriate copyright notice; 200 | keep intact all notices stating that this License and any 201 | non-permissive terms added in accord with section 7 apply to the code; 202 | keep intact all notices of the absence of any warranty; and give all 203 | recipients a copy of this License along with the Program. 204 | 205 | You may charge any price or no price for each copy that you convey, 206 | and you may offer support or warranty protection for a fee. 207 | 208 | 5. Conveying Modified Source Versions. 209 | 210 | You may convey a work based on the Program, or the modifications to 211 | produce it from the Program, in the form of source code under the 212 | terms of section 4, provided that you also meet all of these conditions: 213 | 214 | a) The work must carry prominent notices stating that you modified 215 | it, and giving a relevant date. 216 | 217 | b) The work must carry prominent notices stating that it is 218 | released under this License and any conditions added under section 219 | 7. This requirement modifies the requirement in section 4 to 220 | "keep intact all notices". 221 | 222 | c) You must license the entire work, as a whole, under this 223 | License to anyone who comes into possession of a copy. This 224 | License will therefore apply, along with any applicable section 7 225 | additional terms, to the whole of the work, and all its parts, 226 | regardless of how they are packaged. This License gives no 227 | permission to license the work in any other way, but it does not 228 | invalidate such permission if you have separately received it. 229 | 230 | d) If the work has interactive user interfaces, each must display 231 | Appropriate Legal Notices; however, if the Program has interactive 232 | interfaces that do not display Appropriate Legal Notices, your 233 | work need not make them do so. 234 | 235 | A compilation of a covered work with other separate and independent 236 | works, which are not by their nature extensions of the covered work, 237 | and which are not combined with it such as to form a larger program, 238 | in or on a volume of a storage or distribution medium, is called an 239 | "aggregate" if the compilation and its resulting copyright are not 240 | used to limit the access or legal rights of the compilation's users 241 | beyond what the individual works permit. Inclusion of a covered work 242 | in an aggregate does not cause this License to apply to the other 243 | parts of the aggregate. 244 | 245 | 6. Conveying Non-Source Forms. 246 | 247 | You may convey a covered work in object code form under the terms 248 | of sections 4 and 5, provided that you also convey the 249 | machine-readable Corresponding Source under the terms of this License, 250 | in one of these ways: 251 | 252 | a) Convey the object code in, or embodied in, a physical product 253 | (including a physical distribution medium), accompanied by the 254 | Corresponding Source fixed on a durable physical medium 255 | customarily used for software interchange. 256 | 257 | b) Convey the object code in, or embodied in, a physical product 258 | (including a physical distribution medium), accompanied by a 259 | written offer, valid for at least three years and valid for as 260 | long as you offer spare parts or customer support for that product 261 | model, to give anyone who possesses the object code either (1) a 262 | copy of the Corresponding Source for all the software in the 263 | product that is covered by this License, on a durable physical 264 | medium customarily used for software interchange, for a price no 265 | more than your reasonable cost of physically performing this 266 | conveying of source, or (2) access to copy the 267 | Corresponding Source from a network server at no charge. 268 | 269 | c) Convey individual copies of the object code with a copy of the 270 | written offer to provide the Corresponding Source. This 271 | alternative is allowed only occasionally and noncommercially, and 272 | only if you received the object code with such an offer, in accord 273 | with subsection 6b. 274 | 275 | d) Convey the object code by offering access from a designated 276 | place (gratis or for a charge), and offer equivalent access to the 277 | Corresponding Source in the same way through the same place at no 278 | further charge. You need not require recipients to copy the 279 | Corresponding Source along with the object code. If the place to 280 | copy the object code is a network server, the Corresponding Source 281 | may be on a different server (operated by you or a third party) 282 | that supports equivalent copying facilities, provided you maintain 283 | clear directions next to the object code saying where to find the 284 | Corresponding Source. Regardless of what server hosts the 285 | Corresponding Source, you remain obligated to ensure that it is 286 | available for as long as needed to satisfy these requirements. 287 | 288 | e) Convey the object code using peer-to-peer transmission, provided 289 | you inform other peers where the object code and Corresponding 290 | Source of the work are being offered to the general public at no 291 | charge under subsection 6d. 292 | 293 | A separable portion of the object code, whose source code is excluded 294 | from the Corresponding Source as a System Library, need not be 295 | included in conveying the object code work. 296 | 297 | A "User Product" is either (1) a "consumer product", which means any 298 | tangible personal property which is normally used for personal, family, 299 | or household purposes, or (2) anything designed or sold for incorporation 300 | into a dwelling. In determining whether a product is a consumer product, 301 | doubtful cases shall be resolved in favor of coverage. For a particular 302 | product received by a particular user, "normally used" refers to a 303 | typical or common use of that class of product, regardless of the status 304 | of the particular user or of the way in which the particular user 305 | actually uses, or expects or is expected to use, the product. A product 306 | is a consumer product regardless of whether the product has substantial 307 | commercial, industrial or non-consumer uses, unless such uses represent 308 | the only significant mode of use of the product. 309 | 310 | "Installation Information" for a User Product means any methods, 311 | procedures, authorization keys, or other information required to install 312 | and execute modified versions of a covered work in that User Product from 313 | a modified version of its Corresponding Source. The information must 314 | suffice to ensure that the continued functioning of the modified object 315 | code is in no case prevented or interfered with solely because 316 | modification has been made. 317 | 318 | If you convey an object code work under this section in, or with, or 319 | specifically for use in, a User Product, and the conveying occurs as 320 | part of a transaction in which the right of possession and use of the 321 | User Product is transferred to the recipient in perpetuity or for a 322 | fixed term (regardless of how the transaction is characterized), the 323 | Corresponding Source conveyed under this section must be accompanied 324 | by the Installation Information. But this requirement does not apply 325 | if neither you nor any third party retains the ability to install 326 | modified object code on the User Product (for example, the work has 327 | been installed in ROM). 328 | 329 | The requirement to provide Installation Information does not include a 330 | requirement to continue to provide support service, warranty, or updates 331 | for a work that has been modified or installed by the recipient, or for 332 | the User Product in which it has been modified or installed. Access to a 333 | network may be denied when the modification itself materially and 334 | adversely affects the operation of the network or violates the rules and 335 | protocols for communication across the network. 336 | 337 | Corresponding Source conveyed, and Installation Information provided, 338 | in accord with this section must be in a format that is publicly 339 | documented (and with an implementation available to the public in 340 | source code form), and must require no special password or key for 341 | unpacking, reading or copying. 342 | 343 | 7. Additional Terms. 344 | 345 | "Additional permissions" are terms that supplement the terms of this 346 | License by making exceptions from one or more of its conditions. 347 | Additional permissions that are applicable to the entire Program shall 348 | be treated as though they were included in this License, to the extent 349 | that they are valid under applicable law. If additional permissions 350 | apply only to part of the Program, that part may be used separately 351 | under those permissions, but the entire Program remains governed by 352 | this License without regard to the additional permissions. 353 | 354 | When you convey a copy of a covered work, you may at your option 355 | remove any additional permissions from that copy, or from any part of 356 | it. (Additional permissions may be written to require their own 357 | removal in certain cases when you modify the work.) You may place 358 | additional permissions on material, added by you to a covered work, 359 | for which you have or can give appropriate copyright permission. 360 | 361 | Notwithstanding any other provision of this License, for material you 362 | add to a covered work, you may (if authorized by the copyright holders of 363 | that material) supplement the terms of this License with terms: 364 | 365 | a) Disclaiming warranty or limiting liability differently from the 366 | terms of sections 15 and 16 of this License; or 367 | 368 | b) Requiring preservation of specified reasonable legal notices or 369 | author attributions in that material or in the Appropriate Legal 370 | Notices displayed by works containing it; or 371 | 372 | c) Prohibiting misrepresentation of the origin of that material, or 373 | requiring that modified versions of such material be marked in 374 | reasonable ways as different from the original version; or 375 | 376 | d) Limiting the use for publicity purposes of names of licensors or 377 | authors of the material; or 378 | 379 | e) Declining to grant rights under trademark law for use of some 380 | trade names, trademarks, or service marks; or 381 | 382 | f) Requiring indemnification of licensors and authors of that 383 | material by anyone who conveys the material (or modified versions of 384 | it) with contractual assumptions of liability to the recipient, for 385 | any liability that these contractual assumptions directly impose on 386 | those licensors and authors. 387 | 388 | All other non-permissive additional terms are considered "further 389 | restrictions" within the meaning of section 10. If the Program as you 390 | received it, or any part of it, contains a notice stating that it is 391 | governed by this License along with a term that is a further 392 | restriction, you may remove that term. If a license document contains 393 | a further restriction but permits relicensing or conveying under this 394 | License, you may add to a covered work material governed by the terms 395 | of that license document, provided that the further restriction does 396 | not survive such relicensing or conveying. 397 | 398 | If you add terms to a covered work in accord with this section, you 399 | must place, in the relevant source files, a statement of the 400 | additional terms that apply to those files, or a notice indicating 401 | where to find the applicable terms. 402 | 403 | Additional terms, permissive or non-permissive, may be stated in the 404 | form of a separately written license, or stated as exceptions; 405 | the above requirements apply either way. 406 | 407 | 8. Termination. 408 | 409 | You may not propagate or modify a covered work except as expressly 410 | provided under this License. Any attempt otherwise to propagate or 411 | modify it is void, and will automatically terminate your rights under 412 | this License (including any patent licenses granted under the third 413 | paragraph of section 11). 414 | 415 | However, if you cease all violation of this License, then your 416 | license from a particular copyright holder is reinstated (a) 417 | provisionally, unless and until the copyright holder explicitly and 418 | finally terminates your license, and (b) permanently, if the copyright 419 | holder fails to notify you of the violation by some reasonable means 420 | prior to 60 days after the cessation. 421 | 422 | Moreover, your license from a particular copyright holder is 423 | reinstated permanently if the copyright holder notifies you of the 424 | violation by some reasonable means, this is the first time you have 425 | received notice of violation of this License (for any work) from that 426 | copyright holder, and you cure the violation prior to 30 days after 427 | your receipt of the notice. 428 | 429 | Termination of your rights under this section does not terminate the 430 | licenses of parties who have received copies or rights from you under 431 | this License. If your rights have been terminated and not permanently 432 | reinstated, you do not qualify to receive new licenses for the same 433 | material under section 10. 434 | 435 | 9. Acceptance Not Required for Having Copies. 436 | 437 | You are not required to accept this License in order to receive or 438 | run a copy of the Program. Ancillary propagation of a covered work 439 | occurring solely as a consequence of using peer-to-peer transmission 440 | to receive a copy likewise does not require acceptance. However, 441 | nothing other than this License grants you permission to propagate or 442 | modify any covered work. These actions infringe copyright if you do 443 | not accept this License. Therefore, by modifying or propagating a 444 | covered work, you indicate your acceptance of this License to do so. 445 | 446 | 10. Automatic Licensing of Downstream Recipients. 447 | 448 | Each time you convey a covered work, the recipient automatically 449 | receives a license from the original licensors, to run, modify and 450 | propagate that work, subject to this License. You are not responsible 451 | for enforcing compliance by third parties with this License. 452 | 453 | An "entity transaction" is a transaction transferring control of an 454 | organization, or substantially all assets of one, or subdividing an 455 | organization, or merging organizations. If propagation of a covered 456 | work results from an entity transaction, each party to that 457 | transaction who receives a copy of the work also receives whatever 458 | licenses to the work the party's predecessor in interest had or could 459 | give under the previous paragraph, plus a right to possession of the 460 | Corresponding Source of the work from the predecessor in interest, if 461 | the predecessor has it or can get it with reasonable efforts. 462 | 463 | You may not impose any further restrictions on the exercise of the 464 | rights granted or affirmed under this License. For example, you may 465 | not impose a license fee, royalty, or other charge for exercise of 466 | rights granted under this License, and you may not initiate litigation 467 | (including a cross-claim or counterclaim in a lawsuit) alleging that 468 | any patent claim is infringed by making, using, selling, offering for 469 | sale, or importing the Program or any portion of it. 470 | 471 | 11. Patents. 472 | 473 | A "contributor" is a copyright holder who authorizes use under this 474 | License of the Program or a work on which the Program is based. The 475 | work thus licensed is called the contributor's "contributor version". 476 | 477 | A contributor's "essential patent claims" are all patent claims 478 | owned or controlled by the contributor, whether already acquired or 479 | hereafter acquired, that would be infringed by some manner, permitted 480 | by this License, of making, using, or selling its contributor version, 481 | but do not include claims that would be infringed only as a 482 | consequence of further modification of the contributor version. For 483 | purposes of this definition, "control" includes the right to grant 484 | patent sublicenses in a manner consistent with the requirements of 485 | this License. 486 | 487 | Each contributor grants you a non-exclusive, worldwide, royalty-free 488 | patent license under the contributor's essential patent claims, to 489 | make, use, sell, offer for sale, import and otherwise run, modify and 490 | propagate the contents of its contributor version. 491 | 492 | In the following three paragraphs, a "patent license" is any express 493 | agreement or commitment, however denominated, not to enforce a patent 494 | (such as an express permission to practice a patent or covenant not to 495 | sue for patent infringement). To "grant" such a patent license to a 496 | party means to make such an agreement or commitment not to enforce a 497 | patent against the party. 498 | 499 | If you convey a covered work, knowingly relying on a patent license, 500 | and the Corresponding Source of the work is not available for anyone 501 | to copy, free of charge and under the terms of this License, through a 502 | publicly available network server or other readily accessible means, 503 | then you must either (1) cause the Corresponding Source to be so 504 | available, or (2) arrange to deprive yourself of the benefit of the 505 | patent license for this particular work, or (3) arrange, in a manner 506 | consistent with the requirements of this License, to extend the patent 507 | license to downstream recipients. "Knowingly relying" means you have 508 | actual knowledge that, but for the patent license, your conveying the 509 | covered work in a country, or your recipient's use of the covered work 510 | in a country, would infringe one or more identifiable patents in that 511 | country that you have reason to believe are valid. 512 | 513 | If, pursuant to or in connection with a single transaction or 514 | arrangement, you convey, or propagate by procuring conveyance of, a 515 | covered work, and grant a patent license to some of the parties 516 | receiving the covered work authorizing them to use, propagate, modify 517 | or convey a specific copy of the covered work, then the patent license 518 | you grant is automatically extended to all recipients of the covered 519 | work and works based on it. 520 | 521 | A patent license is "discriminatory" if it does not include within 522 | the scope of its coverage, prohibits the exercise of, or is 523 | conditioned on the non-exercise of one or more of the rights that are 524 | specifically granted under this License. You may not convey a covered 525 | work if you are a party to an arrangement with a third party that is 526 | in the business of distributing software, under which you make payment 527 | to the third party based on the extent of your activity of conveying 528 | the work, and under which the third party grants, to any of the 529 | parties who would receive the covered work from you, a discriminatory 530 | patent license (a) in connection with copies of the covered work 531 | conveyed by you (or copies made from those copies), or (b) primarily 532 | for and in connection with specific products or compilations that 533 | contain the covered work, unless you entered into that arrangement, 534 | or that patent license was granted, prior to 28 March 2007. 535 | 536 | Nothing in this License shall be construed as excluding or limiting 537 | any implied license or other defenses to infringement that may 538 | otherwise be available to you under applicable patent law. 539 | 540 | 12. No Surrender of Others' Freedom. 541 | 542 | If conditions are imposed on you (whether by court order, agreement or 543 | otherwise) that contradict the conditions of this License, they do not 544 | excuse you from the conditions of this License. If you cannot convey a 545 | covered work so as to satisfy simultaneously your obligations under this 546 | License and any other pertinent obligations, then as a consequence you may 547 | not convey it at all. For example, if you agree to terms that obligate you 548 | to collect a royalty for further conveying from those to whom you convey 549 | the Program, the only way you could satisfy both those terms and this 550 | License would be to refrain entirely from conveying the Program. 551 | 552 | 13. Use with the GNU Affero General Public License. 553 | 554 | Notwithstanding any other provision of this License, you have 555 | permission to link or combine any covered work with a work licensed 556 | under version 3 of the GNU Affero General Public License into a single 557 | combined work, and to convey the resulting work. The terms of this 558 | License will continue to apply to the part which is the covered work, 559 | but the special requirements of the GNU Affero General Public License, 560 | section 13, concerning interaction through a network will apply to the 561 | combination as such. 562 | 563 | 14. Revised Versions of this License. 564 | 565 | The Free Software Foundation may publish revised and/or new versions of 566 | the GNU General Public License from time to time. Such new versions will 567 | be similar in spirit to the present version, but may differ in detail to 568 | address new problems or concerns. 569 | 570 | Each version is given a distinguishing version number. If the 571 | Program specifies that a certain numbered version of the GNU General 572 | Public License "or any later version" applies to it, you have the 573 | option of following the terms and conditions either of that numbered 574 | version or of any later version published by the Free Software 575 | Foundation. If the Program does not specify a version number of the 576 | GNU General Public License, you may choose any version ever published 577 | by the Free Software Foundation. 578 | 579 | If the Program specifies that a proxy can decide which future 580 | versions of the GNU General Public License can be used, that proxy's 581 | public statement of acceptance of a version permanently authorizes you 582 | to choose that version for the Program. 583 | 584 | Later license versions may give you additional or different 585 | permissions. However, no additional obligations are imposed on any 586 | author or copyright holder as a result of your choosing to follow a 587 | later version. 588 | 589 | 15. Disclaimer of Warranty. 590 | 591 | THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY 592 | APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT 593 | HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY 594 | OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, 595 | THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR 596 | PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM 597 | IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF 598 | ALL NECESSARY SERVICING, REPAIR OR CORRECTION. 599 | 600 | 16. Limitation of Liability. 601 | 602 | IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING 603 | WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS 604 | THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY 605 | GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE 606 | USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF 607 | DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD 608 | PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS), 609 | EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF 610 | SUCH DAMAGES. 611 | 612 | 17. Interpretation of Sections 15 and 16. 613 | 614 | If the disclaimer of warranty and limitation of liability provided 615 | above cannot be given local legal effect according to their terms, 616 | reviewing courts shall apply local law that most closely approximates 617 | an absolute waiver of all civil liability in connection with the 618 | Program, unless a warranty or assumption of liability accompanies a 619 | copy of the Program in return for a fee. 620 | 621 | END OF TERMS AND CONDITIONS 622 | 623 | How to Apply These Terms to Your New Programs 624 | 625 | If you develop a new program, and you want it to be of the greatest 626 | possible use to the public, the best way to achieve this is to make it 627 | free software which everyone can redistribute and change under these terms. 628 | 629 | To do so, attach the following notices to the program. It is safest 630 | to attach them to the start of each source file to most effectively 631 | state the exclusion of warranty; and each file should have at least 632 | the "copyright" line and a pointer to where the full notice is found. 633 | 634 | DEEP-mon 635 | Copyright (C) 2020 Brondolin Rolando 636 | 637 | This file is part of DEEP-mon 638 | 639 | DEEP-mon is free software: you can redistribute it and/or modify 640 | it under the terms of the GNU General Public License as published by 641 | the Free Software Foundation, either version 3 of the License, or 642 | (at your option) any later version. 643 | 644 | DEEP-mon is distributed in the hope that it will be useful, 645 | but WITHOUT ANY WARRANTY; without even the implied warranty of 646 | MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 647 | GNU General Public License for more details. 648 | 649 | You should have received a copy of the GNU General Public License 650 | along with this program. If not, see . 651 | 652 | Also add information on how to contact you by electronic and paper mail. 653 | 654 | If the program does terminal interaction, make it output a short 655 | notice like this when it starts in an interactive mode: 656 | 657 | deep-mon Copyright (C) 2020 Project Hyppo 658 | This program comes with ABSOLUTELY NO WARRANTY; for details type `show w'. 659 | This is free software, and you are welcome to redistribute it 660 | under certain conditions; type `show c' for details. 661 | 662 | The hypothetical commands `show w' and `show c' should show the appropriate 663 | parts of the General Public License. Of course, your program's commands 664 | might be different; for a GUI interface, you would use an "about box". 665 | 666 | You should also get your employer (if you work as a programmer) or school, 667 | if any, to sign a "copyright disclaimer" for the program, if necessary. 668 | For more information on this, and how to apply and follow the GNU GPL, see 669 | . 670 | 671 | The GNU General Public License does not permit incorporating your program 672 | into proprietary programs. If your program is a subroutine library, you 673 | may consider it more useful to permit linking proprietary applications with 674 | the library. If this is what you want to do, use the GNU Lesser General 675 | Public License instead of this License. But first, please read 676 | . 677 | -------------------------------------------------------------------------------- /Makefile: -------------------------------------------------------------------------------- 1 | # DEEP-mon 2 | # Copyright (C) 2020 Brondolin Rolando 3 | # 4 | # This file is part of DEEP-mon 5 | # 6 | # DEEP-mon is free software: you can redistribute it and/or modify 7 | # it under the terms of the GNU General Public License as published by 8 | # the Free Software Foundation, either version 3 of the License, or 9 | # (at your option) any later version. 10 | # 11 | # DEEP-mon is distributed in the hope that it will be useful, 12 | # but WITHOUT ANY WARRANTY; without even the implied warranty of 13 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 14 | # GNU General Public License for more details. 15 | # 16 | # You should have received a copy of the GNU General Public License 17 | # along with this program. If not, see . 18 | 19 | 20 | # HELP 21 | # This will output the help for each task 22 | # thanks to https://marmelab.com/blog/2016/02/29/auto-documented-makefile.html 23 | .PHONY: help run stop build 24 | 25 | help: ## This help. 26 | @awk 'BEGIN {FS = ":.*?## "} /^[a-zA-Z_-]+:.*?## / {printf "\033[36m%-30s\033[0m %s\n", $$1, $$2}' $(MAKEFILE_LIST) 27 | 28 | .DEFAULT_GOAL := help 29 | 30 | # DOCKER TASKS 31 | run: ## Run a standalone image with text UI 32 | sudo docker run -it --rm --privileged --name deep-mon -v /lib/modules:/lib/modules:ro -v /usr/src:/usr/src:ro -v /etc/localtime:/etc/localtime:ro -v /sys/kernel/debug:/sys/kernel/debug:rw -v /proc:/host/proc:ro -v ${PWD}/config.yaml:/home/config.yaml -v /var/run/docker.sock:/var/run/docker.sock --net host deep-mon 33 | 34 | explore: ## Run a standalone image with bash to check stuff 35 | sudo docker run -it --rm --privileged --name deep-mon -v /lib/modules:/lib/modules:ro -v /usr/src:/usr/src:ro -v /etc/localtime:/etc/localtime:ro -v /sys/kernel/debug:/sys/kernel/debug:rw -v /proc:/host/proc:ro -v ${PWD}/config.yaml:/home/config.yaml -v /var/run/docker.sock:/var/run/docker.sock --net host deep-mon bash 36 | 37 | 38 | build: ## Build a standalone image 39 | sudo docker build . -t "deep-mon" 40 | 41 | build-no-cache: ## Build a standalone image without cache 42 | sudo docker build . -t "deep-mon" --no-cache 43 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # DEEP-mon 2 | 3 | Dynamic and Energy Efficient Power monitor (DEEP-mon) is an eBPF based monitoring tool to measure power consumption and performance of Docker containers. DEEP-mon started in 2017 as a research project of [NECSTLab](https://necst.it) at Politecnico di Milano with the goal of being able to measure the power consumption of each container running in a given host. The project then expanded and now DEEP-mon is able to measure power consumption, performance counters, CPU usage, Memory usage, Network I/O, and File I/O, all displayed in a curses UI terminal window. 4 | 5 | ## Getting started 6 | 7 | ### Requirements 8 | 9 | DEEP-mon currently runs on Docker and it is almost self contained. Requirements are: 10 | 11 | - Linux kernel >= 4.13 12 | - kernel headers 13 | - make 14 | - docker 15 | - intel_rapl module (for power consumption on Intel® processors) 16 | 17 | We tested DEEP-mon on Ubuntu 16.04 and 18.04. The power attribution feature of DEEP-mon targets the SMT capability of Intel® Sandy bridge and Ivy bridge server class processors. For other architectures a new attribution factor should be tested and used (more details [here](https://ieeexplore.ieee.org/abstract/document/8425477) and [here](https://www.usenix.org/conference/atc14/technical-sessions/presentation/zhai)). 18 | 19 | ### Build and Run 20 | 21 | To build the container, within the root folder of the project type: 22 | 23 | ```bash 24 | make build 25 | ``` 26 | 27 | To run the container, within the root folder of the project type: 28 | 29 | ```bash 30 | make run 31 | ``` 32 | 33 | ## Bug reports 34 | 35 | For bug reports or feature requests feel free to create an [issue](https://github.com/necst/DEEP-mon/issues). 36 | Please make sure that the same problem wasn't reported already. 37 | 38 | ## Contributing 39 | 40 | Contribution is welcome! 41 | 42 | * Create a pull request containing bug fixes or new features. 43 | * [Propose](https://github.com/necst/DEEP-mon/issues/new) new functions and improvements 44 | 45 | ## DEEP-mon roadmap: 46 | * fix performance issue with memory metrics 47 | * experimental measurement tool -> record stuff, single machine + distributed 48 | * frequency sampling 49 | * improve parameter injection 50 | * set attribution ratio in config 51 | * improve curse UI 52 | * documentation (code + md files) 53 | * tests 54 | * add k8s pod name, deployments, services (e.g. readable k8s data) 55 | * one UI per cluster (daemonset), server w/data, cli 56 | * web UI 57 | 58 | ## Research 59 | 60 | As we said at the beginning, this work comes form the research conducted at [NECSTLab](https://necst.it). If you use this tool for your research, please cite the following papers: 61 | 62 | * Brondolin, Rolando, Tommaso Sardelli, and Marco D. Santambrogio. "Deep-mon: Dynamic and energy efficient power monitoring for container-based infrastructures." 2018 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW). IEEE, 2018. (download [here](https://ieeexplore.ieee.org/abstract/document/8425477)) 63 | 64 | * Rolando Brondolin and Marco D Santambrogio. "A black-box monitoring approach to measure microservices runtime performance." ACM Transactions on Architecture and Code Optimization (TACO), 17(4):1-26, 2020. (download [here](https://doi.org/10.1145/3418899)) 65 | 66 | -------------------------------------------------------------------------------- /bpf/__init__.py: -------------------------------------------------------------------------------- 1 | """ 2 | DEEP-mon 3 | Copyright (C) 2020 Brondolin Rolando 4 | 5 | This file is part of DEEP-mon 6 | 7 | DEEP-mon is free software: you can redistribute it and/or modify 8 | it under the terms of the GNU General Public License as published by 9 | the Free Software Foundation, either version 3 of the License, or 10 | (at your option) any later version. 11 | 12 | DEEP-mon is distributed in the hope that it will be useful, 13 | but WITHOUT ANY WARRANTY; without even the implied warranty of 14 | MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 15 | GNU General Public License for more details. 16 | 17 | You should have received a copy of the GNU General Public License 18 | along with this program. If not, see . 19 | """ 20 | -------------------------------------------------------------------------------- /bpf/bpf_monitor.c: -------------------------------------------------------------------------------- 1 | /* 2 | DEEP-mon 3 | Copyright (C) 2020 Brondolin Rolando 4 | 5 | This file is part of DEEP-mon 6 | 7 | DEEP-mon is free software: you can redistribute it and/or modify 8 | it under the terms of the GNU General Public License as published by 9 | the Free Software Foundation, either version 3 of the License, or 10 | (at your option) any later version. 11 | 12 | DEEP-mon is distributed in the hope that it will be useful, 13 | but WITHOUT ANY WARRANTY; without even the implied warranty of 14 | MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 15 | GNU General Public License for more details. 16 | 17 | You should have received a copy of the GNU General Public License 18 | along with this program. If not, see . 19 | */ 20 | 21 | #include 22 | 23 | /** 24 | * In the rest of the code we are going to use a selector to read and write 25 | * events. The idea is that while userspace is reading events 26 | * for selector 0 we write events using selector 1 and vice versa. 27 | * This process in handled by the bpf_selector variable. 28 | */ 29 | #define SELECTOR_DIM 2 // Use a binary selector 30 | 31 | /** 32 | * Slots are array cells we use to store events following the selector idea. 33 | * Let's take as an example NUM_SOCKETS=2 (2 physical CPUs) and let's keep 34 | * SELECTOR_DIM=2 as defined above. In this case the array would have four slots 35 | * and will be partitioned in the following way: 36 | * [ CPU_0 SELECTOR_0 | CPU_0 SELECTOR_1 | CPU_1 SELECTOR_0 | CPU_1 SELECTOR_1 ] 37 | */ 38 | #define NUM_SLOTS NUM_SOCKETS * SELECTOR_DIM 39 | 40 | /** 41 | * pid_status is used to store information about pid X 42 | * bpf_selector is used for the slot selection described above and is set 43 | * in the BPF context. PCM arrays like cycles and IR are written by BPF but 44 | * read and initialized from user space. 45 | */ 46 | struct pid_status { 47 | int pid; /**< Process ID */ 48 | int tgid; 49 | char comm[TASK_COMM_LEN]; /**< Process name */ 50 | u64 weighted_cycles[NUM_SLOTS]; /**< Number of weighted cycles executed by the process */ 51 | u64 cycles[SELECTOR_DIM]; /**< Number of unhalted core cycles executed by the process */ 52 | u64 instruction_retired[SELECTOR_DIM]; /**< Number of instructions executed by the process */ 53 | u64 cache_misses[SELECTOR_DIM]; /**< Number of Last Level Cache misses executed by the process */ 54 | u64 cache_refs[SELECTOR_DIM]; /**< Number of Last Level Cache references executed by the process */ 55 | u64 time_ns[SELECTOR_DIM]; /**< Execution time of the process (in ns) */ 56 | unsigned int bpf_selector; /**< Slot selector */ 57 | u64 ts[SELECTOR_DIM]; /**< Timestamp of the latest update */ 58 | }; 59 | 60 | /** 61 | * proc_topology is used to store information about the underlying topology 62 | * of CPUS. 63 | * We distinguish processors, cores (physical core) and ht/thread (soft cores). 64 | * Siblings cores are two soft cores residing on the same physical core. 65 | */ 66 | struct proc_topology { 67 | u64 ht_id; 68 | u64 sibling_id; 69 | u64 core_id; 70 | u64 processor_id; 71 | u64 cycles_core; 72 | u64 cycles_core_delta_sibling; 73 | u64 cycles_thread; 74 | u64 instruction_thread; 75 | u64 cache_misses; 76 | u64 cache_refs; 77 | u64 ts; 78 | int running_pid; 79 | }; 80 | 81 | /** 82 | * sched_switch_args is the payload of a sched_switch tracepoint event 83 | * as defined in the Linux kernel. 84 | */ 85 | struct sched_switch_args { 86 | __u64 pad; // regs after 4.x? 87 | char prev_comm[16]; 88 | int prev_pid; 89 | int prev_prio; 90 | long long prev_state; 91 | char next_comm[16]; 92 | int next_pid; 93 | int next_prio; 94 | }; 95 | struct sched_process_exec_args { 96 | __u64 pad; // regs after 4.x? 97 | char filename[4]; 98 | int pid; 99 | int old_pid; 100 | }; 101 | struct sched_process_fork_args { 102 | __u64 pad; // regs after 4.x? 103 | char parent_comm[16]; 104 | int parent_pid; 105 | char child_comm[16]; 106 | int child_pid; 107 | }; 108 | struct sched_process_exit_args { 109 | __u64 pad; // regs after 4.x? 110 | char comm[16]; 111 | int pid; 112 | int prio; 113 | }; 114 | 115 | #ifdef DEBUG 116 | struct error_code { 117 | int err; 118 | }; 119 | 120 | BPF_PERF_OUTPUT(err); 121 | #endif 122 | 123 | #ifdef PERFORMANCE_COUNTERS 124 | BPF_PERF_ARRAY(cycles_core, NUM_CPUS); 125 | BPF_PERF_ARRAY(cycles_thread, NUM_CPUS); 126 | BPF_PERF_ARRAY(instr_thread, NUM_CPUS); 127 | BPF_PERF_ARRAY(cache_misses, NUM_CPUS); 128 | BPF_PERF_ARRAY(cache_refs, NUM_CPUS); 129 | #endif 130 | BPF_HASH(processors, u64, struct proc_topology); 131 | BPF_HASH(pids, int, struct pid_status); 132 | BPF_HASH(idles, u64, struct pid_status); 133 | 134 | /** 135 | * conf struct has 4 integer keys initialized in user space 136 | * 0: current bpf selector 137 | * 1: old bpf selector 138 | * 2: timeslice (dynamic window duration) 139 | * 3: switch count (global context switch counter). Used to compute the window size 140 | */ 141 | #define BPF_SELECTOR_INDEX 0 142 | #define BPF_SELECTOR_INDEX_OLD 1 143 | #define BPF_TIMESLICE 2 144 | #define BPF_SWITCH_COUNT 3 145 | BPF_ARRAY(conf, u32, 4); 146 | 147 | /* 148 | * timestamp array to store the last timestamp of a given time slot 149 | */ 150 | BPF_ARRAY(global_timestamps, u64, SELECTOR_DIM); 151 | 152 | 153 | /** 154 | * STEP_MIN and STEP_MAX are the lower and upper bound for the duration 155 | * of the dynamic window (interval between two reads from user space) 156 | * They are expressed in nanoseconds so their range is 1-4 second. 157 | * BEWARE: Changing the step in userspace means invalidate the last sample 158 | */ 159 | #define STEP_MIN 1000000000 160 | #define STEP_MAX 4000000000 161 | 162 | #define HAPPY_FACTOR 11/20 163 | #define STD_FACTOR 1 164 | 165 | /* 166 | * Errors code for userspace debug 167 | */ 168 | #define BPF_PROCEED_WITH_DEBUG_MODE -1 169 | #define BPF_SELECTOR_NOT_IN_PLACE -2 170 | #define OLD_BPF_SELECTOR_NOT_IN_PLACE -3 171 | #define TIMESTEP_NOT_IN_PLACE -4 172 | #define CORRUPTED_TOPOLOGY_MAP -5 173 | #define WRONG_SIBLING_TOPOLOGY_MAP -6 174 | #define THREAD_MIGRATED_UNEXPECTEDLY -7 175 | 176 | 177 | static void send_error(struct sched_switch_args *ctx, int err_code) { 178 | #ifdef DEBUG 179 | struct error_code error; 180 | error.err = err_code; 181 | err.perf_submit(ctx, &error, sizeof(error)); 182 | #endif 183 | } 184 | 185 | static void send_perf_error(struct bpf_perf_event_data *ctx, int err_code) { 186 | #ifdef DEBUG 187 | struct error_code error; 188 | error.err = err_code; 189 | err.perf_submit(ctx, &error, sizeof(error)); 190 | #endif 191 | } 192 | 193 | static inline int update_cycles_count(void *ctx, 194 | int old_pid, u32 bpf_selector, u32 step, u64 processor_id, 195 | #ifdef PERFORMANCE_COUNTERS 196 | u64 thread_cycles_sample, u64 core_cycles_sample, 197 | u64 instruction_retired_thread, u64 cache_misses_thread, 198 | u64 cache_refs_thread, 199 | #endif 200 | u64 ts) { 201 | 202 | int ret = 0; 203 | 204 | // Fetch more data about processor where the sched_switch happened 205 | struct proc_topology topology_info; 206 | ret = bpf_probe_read(&topology_info, sizeof(topology_info), processors.lookup(&processor_id)); 207 | if(ret!= 0 || topology_info.ht_id > NUM_CPUS) { 208 | send_error(ctx, CORRUPTED_TOPOLOGY_MAP); 209 | return 0; 210 | } 211 | 212 | // Create the struct to hold the pid status we are going to fetch 213 | struct pid_status status_old; 214 | status_old.pid = -1; 215 | 216 | /** 217 | * Fetch the status of the exiting pid. 218 | * If the pid is 0 then use the idles perf_hash 219 | */ 220 | if(old_pid == 0) { 221 | ret = bpf_probe_read(&status_old, sizeof(status_old), idles.lookup(&(processor_id))); 222 | } else { 223 | ret = bpf_probe_read(&status_old, sizeof(status_old), pids.lookup(&(old_pid))); 224 | } 225 | 226 | if(ret != 0) { 227 | // no data for this thread, for now do not account data 228 | return 0; 229 | } 230 | 231 | if (topology_info.running_pid != status_old.pid) { 232 | // we have some issues 233 | send_error(ctx, THREAD_MIGRATED_UNEXPECTEDLY); 234 | return 0; 235 | } 236 | 237 | #ifdef PERFORMANCE_COUNTERS 238 | // Retrieving information of the sibling processor 239 | u64 sibling_id = topology_info.sibling_id; 240 | struct proc_topology sibling_info; 241 | ret = bpf_probe_read(&sibling_info, sizeof(sibling_info), processors.lookup(&(sibling_id))); 242 | 243 | if(ret != 0) { 244 | // Wrong info on topology, do nothing 245 | send_error(ctx, WRONG_SIBLING_TOPOLOGY_MAP); 246 | return 0; 247 | } 248 | 249 | /** 250 | * Instead of adding stuff directly, given that we don't have 251 | * the measure of the sibling thread cycles, we are summing up 252 | * the information on our side to the core cycles of the sibling 253 | */ 254 | // Update sibling process info 255 | if(sibling_info.running_pid > 0 && old_pid > 0 && core_cycles_sample > sibling_info.cycles_core) { 256 | sibling_info.cycles_core_delta_sibling += core_cycles_sample - sibling_info.cycles_core; 257 | } 258 | sibling_info.cycles_core = core_cycles_sample; 259 | processors.update(&sibling_id, &sibling_info); 260 | #endif 261 | /** 262 | * Get back to our pid and our processor 263 | * Update the data for proc_topology and pid info 264 | * Take the ts marking the beginning of execution of the exiting pid 265 | */ 266 | u64 last_ts_pid_in = 0; 267 | //trick the compiler with loop unrolling 268 | #pragma clang loop unroll(full) 269 | for(int array_index = 0; array_index 0) { 315 | // old_time = topology_info.ts; 316 | // old_thread_cycles = topology_info.cycles_thread; 317 | // cycles_core_delta_sibling = topology_info.cycles_core_delta_sibling; 318 | // old_instruction_retired = topology_info.instruction_thread; 319 | // old_cache_misses = topology_info.cache_misses; 320 | // old_cache_refs = topology_info.cache_refs; 321 | // //update the last slice of concurrent execution inside two sibling hyperthreads 322 | // if (old_pid > 0 && sibling_info.running_pid > 0 && core_cycles_sample > topology_info.cycles_core) { 323 | // cycles_core_delta_sibling += core_cycles_sample - topology_info.cycles_core; 324 | // } 325 | // } 326 | 327 | if (topology_info.ts > 0) { 328 | // update per process measurements (aka IR, cache misses, cycles not weighted) 329 | #pragma clang loop unroll(full) 330 | for(int array_index = 0; array_index= topology_info.instruction_thread) { 334 | status_old.instruction_retired[array_index] += instruction_retired_thread - topology_info.instruction_thread; 335 | } else { 336 | send_error(ctx, old_pid); 337 | } 338 | if (cache_misses_thread >= topology_info.cache_misses) { 339 | status_old.cache_misses[array_index] += cache_misses_thread - topology_info.cache_misses; 340 | } else { 341 | send_error(ctx, old_pid); 342 | } 343 | if (cache_refs_thread >= topology_info.cache_refs) { 344 | status_old.cache_refs[array_index] += cache_refs_thread - topology_info.cache_refs; 345 | } else { 346 | send_error(ctx, old_pid); 347 | } 348 | if (thread_cycles_sample >= topology_info.cycles_thread){ 349 | status_old.cycles[array_index] += thread_cycles_sample - topology_info.cycles_thread; 350 | } else { 351 | send_error(ctx, old_pid); 352 | } 353 | #endif 354 | status_old.time_ns[array_index] += ts - topology_info.ts; 355 | status_old.ts[array_index] = ts; 356 | } 357 | } 358 | } 359 | 360 | #ifdef PERFORMANCE_COUNTERS 361 | // trick the compiler with loop unrolling 362 | // update weighted cycles for our pid 363 | if (topology_info.ts > 0) { 364 | #pragma clang loop unroll(full) 365 | for(int array_index = 0; array_index topology_info.cycles_thread){ 369 | u64 cycle1 = thread_cycles_sample - topology_info.cycles_thread; 370 | u64 cycle_overlap = topology_info.cycles_core_delta_sibling; 371 | u64 cycle_non_overlap = cycle1 > topology_info.cycles_core_delta_sibling ? cycle1 - topology_info.cycles_core_delta_sibling : 0; 372 | status_old.weighted_cycles[array_index] += cycle_non_overlap + cycle_overlap*HAPPY_FACTOR; 373 | } else { 374 | send_error(ctx, old_pid); 375 | } 376 | } 377 | } 378 | } 379 | #endif 380 | // update the pid status in our hashmap 381 | if(old_pid == 0) { 382 | status_old.tgid = bpf_get_current_pid_tgid() >> 32; 383 | idles.update(&processor_id, &status_old); 384 | } else { 385 | status_old.tgid = bpf_get_current_pid_tgid() >> 32; 386 | pids.update(&old_pid, &status_old); 387 | } 388 | 389 | return 0; 390 | } 391 | 392 | int trace_switch(struct sched_switch_args *ctx) { 393 | 394 | // Keys for the conf hash 395 | int selector_key = BPF_SELECTOR_INDEX; 396 | int old_selector_key = BPF_SELECTOR_INDEX_OLD; 397 | int step_key = BPF_TIMESLICE; 398 | int switch_count_key = BPF_SWITCH_COUNT; 399 | 400 | // Slot iterator for the selector 401 | int array_index = 0; 402 | 403 | // Binary selector to avoid event overwriting 404 | unsigned int bpf_selector = 0; 405 | int ret = 0; 406 | ret = bpf_probe_read(&bpf_selector, sizeof(bpf_selector), conf.lookup(&selector_key)); 407 | // If selector is not in place correctly, signal debug error and stop tracing routine 408 | if (ret!= 0 || bpf_selector > 1) { 409 | send_error(ctx, BPF_SELECTOR_NOT_IN_PLACE); 410 | return 0; 411 | } 412 | 413 | // Retrieve general switch count 414 | unsigned int switch_count = 0; 415 | ret = 0; 416 | ret = bpf_probe_read(&switch_count, sizeof(switch_count), conf.lookup(&switch_count_key)); 417 | 418 | /** 419 | * Retrieve old selector to update switch count correctly 420 | * If the current selector is still active increase the switch count 421 | * otherwise reset the count and update the current selector 422 | */ 423 | unsigned int old_bpf_selector = 0; 424 | ret = 0; 425 | ret = bpf_probe_read(&old_bpf_selector, sizeof(old_bpf_selector), conf.lookup(&old_selector_key)); 426 | if (ret!= 0 || old_bpf_selector > 1) { 427 | send_error(ctx, OLD_BPF_SELECTOR_NOT_IN_PLACE); 428 | return 0; 429 | } else if(old_bpf_selector != bpf_selector) { 430 | switch_count = 1; 431 | conf.update(&old_selector_key, &bpf_selector); 432 | } else { 433 | switch_count++; 434 | } 435 | conf.update(&switch_count_key, &switch_count); 436 | 437 | /** 438 | * Retrieve sampling step (dynamic window) 439 | * We need this because later on we check the timestamp of the 440 | * context switch, if it's higher than ts_begin_window + step 441 | * we need to account the current pcm in the next time window (so in 442 | * another bpf selector w.r.t. to the current one) 443 | * BEWARE: increasing the step in userspace means that the next 444 | * sample is invalid. Reducing the step in userspace is not 445 | * an issue, it discards data that has already been collected 446 | */ 447 | unsigned int step = 1000000000; 448 | ret = bpf_probe_read(&step, sizeof(step), conf.lookup(&step_key)); 449 | if (ret!= 0 || step < STEP_MIN || step > STEP_MAX) { 450 | send_error(ctx, TIMESTEP_NOT_IN_PLACE); 451 | return 0; 452 | } 453 | 454 | /** 455 | * Get the id of the processor where the sched_switch happened. 456 | * Collect cycles and IR samples from perf arrays. 457 | * Save the timestamp and store the exiting pid 458 | */ 459 | u64 processor_id = bpf_get_smp_processor_id(); 460 | #ifdef PERFORMANCE_COUNTERS 461 | u64 thread_cycles_sample = cycles_thread.perf_read(processor_id); 462 | u64 core_cycles_sample = cycles_core.perf_read(processor_id); 463 | u64 instruction_retired_thread = instr_thread.perf_read(processor_id); 464 | u64 cache_misses_thread = cache_misses.perf_read(processor_id); 465 | u64 cache_refs_thread = cache_refs.perf_read(processor_id); 466 | #endif 467 | u64 ts = bpf_ktime_get_ns(); 468 | int current_pid = ctx->prev_pid; 469 | 470 | if (ret == 0) { 471 | #ifdef PERFORMANCE_COUNTERS 472 | update_cycles_count(ctx, current_pid, bpf_selector, step, processor_id, thread_cycles_sample, core_cycles_sample, instruction_retired_thread, cache_misses_thread, cache_refs_thread, ts); 473 | #else 474 | update_cycles_count(ctx, current_pid, bpf_selector, step, processor_id, ts); 475 | #endif 476 | } 477 | 478 | // Fetch more data about processor where the sched_switch happened 479 | ret = 0; 480 | struct proc_topology topology_info; 481 | ret = bpf_probe_read(&topology_info, sizeof(topology_info), processors.lookup(&processor_id)); 482 | if(ret!= 0 || topology_info.ht_id > NUM_CPUS) { 483 | send_error(ctx, CORRUPTED_TOPOLOGY_MAP); 484 | return 0; 485 | } 486 | 487 | // 488 | // handle new scheduled process 489 | // 490 | int new_pid = ctx->next_pid; 491 | struct pid_status status_new; 492 | if(new_pid == 0) { 493 | ret = bpf_probe_read(&status_new, sizeof(status_new), idles.lookup(&(processor_id))); 494 | } else { 495 | ret = bpf_probe_read(&status_new, sizeof(status_new), pids.lookup(&(new_pid))); 496 | } 497 | //If no status for PID, then create one, otherwise update selector 498 | if(ret) { 499 | bpf_probe_read(&(status_new.comm), sizeof(status_new.comm), ctx->next_comm); 500 | 501 | #pragma clang loop unroll(full) 502 | for(array_index = 0; array_index 1) { 558 | // //send_error(ctx, BPF_SELECTOR_NOT_IN_PLACE); 559 | // return 0; 560 | // } 561 | // 562 | // unsigned int step = 1000000000; 563 | // ret = bpf_probe_read(&step, sizeof(step), conf.lookup(&step_key)); 564 | // if (ret!= 0 || step < STEP_MIN || step > STEP_MAX) { 565 | // //send_error(ctx, TIMESTEP_NOT_IN_PLACE); 566 | // return 0; 567 | // } 568 | 569 | char comm[16]; 570 | bpf_probe_read(&(comm), sizeof(comm), ctx->comm); 571 | int pid = ctx->pid; 572 | u64 ts = bpf_ktime_get_ns(); 573 | u64 processor_id = bpf_get_smp_processor_id(); 574 | 575 | // 576 | // // if (ret==0) { 577 | // // u64 thread_cycles_sample = cycles_thread.perf_read(processor_id); 578 | // // u64 core_cycles_sample = cycles_core.perf_read(processor_id); 579 | // // u64 instruction_retired_thread = instr_thread.perf_read(processor_id); 580 | // // update_cycles_count(ctx, pid, bpf_selector, step, processor_id, thread_cycles_sample, core_cycles_sample, instruction_retired_thread, ts); 581 | // // } 582 | // 583 | // //account data to the father 584 | // u64 tgid_pid = bpf_get_current_pid_tgid(); 585 | // int tgid = tgid_pid >> 32; 586 | // 587 | // struct pid_status tg_status; 588 | // if(tgid == 0) { 589 | // ret = bpf_probe_read(&tg_status, sizeof(tg_status), idles.lookup(&(processor_id))); 590 | // } else { 591 | // ret = bpf_probe_read(&tg_status, sizeof(tg_status), pids.lookup(&(tgid))); 592 | // } 593 | // 594 | // if(ret == 0) { 595 | // 596 | // //retrieve data about the exiting pid 597 | // struct pid_status status_old; 598 | // status_old.pid = -1; 599 | // 600 | // /** 601 | // * Fetch the status of the exiting pid. 602 | // * If the pid is 0 then use the idles perf_hash 603 | // */ 604 | // if(pid == 0) { 605 | // ret = bpf_probe_read(&status_old, sizeof(status_old), idles.lookup(&(processor_id))); 606 | // } else { 607 | // ret = bpf_probe_read(&status_old, sizeof(status_old), pids.lookup(&(pid))); 608 | // } 609 | // 610 | // if(ret == 0) { 611 | // 612 | // // do the summation for all the sockets when the data 613 | // // for a given socket are updated 614 | // 615 | // #pragma clang loop unroll(full) 616 | // for(int array_index = 0; array_index ts) { 622 | // if(last_ts_pid_in + step > ts){ 623 | // tg_status.ts[array_index] = ts; 624 | // tg_status.cycles[array_index] += status_old.cycles[array_index]; 625 | // tg_status.weighted_cycles[array_index] += status_old.weighted_cycles[array_index]; 626 | // tg_status.instruction_retired[array_index] += status_old.instruction_retired[array_index]; 627 | // tg_status.time_ns[array_index] += status_old.time_ns[array_index]; 628 | // tg_status.bpf_selector = bpf_selector; 629 | // } 630 | // } else { 631 | // if(last_ts_pid_in + step > ts){ 632 | // tg_status.ts[array_index] = ts; 633 | // tg_status.cycles[array_index] = status_old.cycles[array_index]; 634 | // tg_status.weighted_cycles[array_index] = status_old.weighted_cycles[array_index]; 635 | // tg_status.instruction_retired[array_index] = status_old.instruction_retired[array_index]; 636 | // tg_status.time_ns[array_index] = status_old.time_ns[array_index]; 637 | // tg_status.bpf_selector = bpf_selector; 638 | // } 639 | // } 640 | // } 641 | // } 642 | // 643 | // if(tgid == 0) { 644 | // idles.update(&processor_id, &tg_status); 645 | // } else { 646 | // pids.update(&tgid, &tg_status); 647 | // } 648 | // } 649 | // } 650 | 651 | //send_error(ctx, tgid); 652 | //send_error(ctx, pid); 653 | 654 | //remove the pid from the table if there 655 | pids.delete(&pid); 656 | 657 | struct proc_topology topology_info; 658 | bpf_probe_read(&topology_info, sizeof(topology_info), processors.lookup(&processor_id)); 659 | 660 | topology_info.running_pid = 0; 661 | topology_info.ts = ts; 662 | #ifdef PERFORMANCE_COUNTERS 663 | topology_info.cycles_thread = cycles_thread.perf_read(processor_id); 664 | topology_info.cycles_core = cycles_core.perf_read(processor_id); 665 | topology_info.instruction_thread = instr_thread.perf_read(processor_id); 666 | topology_info.cache_misses = cache_misses.perf_read(processor_id); 667 | topology_info.cache_refs = cache_refs.perf_read(processor_id); 668 | topology_info.cycles_core_delta_sibling = 0; 669 | #endif 670 | 671 | processors.update(&processor_id, &topology_info); 672 | 673 | return 0; 674 | } 675 | 676 | int timed_trace(struct bpf_perf_event_data *perf_ctx) { 677 | 678 | // Keys for the conf hash 679 | int selector_key = BPF_SELECTOR_INDEX; 680 | int old_selector_key = BPF_SELECTOR_INDEX_OLD; 681 | int step_key = BPF_TIMESLICE; 682 | int switch_count_key = BPF_SWITCH_COUNT; 683 | 684 | // Slot iterator for the selector 685 | int array_index = 0; 686 | 687 | // Binary selector to avoid event overwriting 688 | unsigned int bpf_selector = 0; 689 | int ret = 0; 690 | ret = bpf_probe_read(&bpf_selector, sizeof(bpf_selector), conf.lookup(&selector_key)); 691 | // If selector is not in place correctly, signal debug error and stop tracing routine 692 | if (ret!= 0 || bpf_selector > 1) { 693 | send_perf_error(perf_ctx, BPF_SELECTOR_NOT_IN_PLACE); 694 | return 0; 695 | } 696 | 697 | 698 | // Retrieve general switch count 699 | unsigned int switch_count = 0; 700 | ret = 0; 701 | ret = bpf_probe_read(&switch_count, sizeof(switch_count), conf.lookup(&switch_count_key)); 702 | 703 | /** 704 | * Retrieve old selector to update switch count correctly 705 | */ 706 | unsigned int old_bpf_selector = 0; 707 | ret = 0; 708 | ret = bpf_probe_read(&old_bpf_selector, sizeof(old_bpf_selector), conf.lookup(&old_selector_key)); 709 | if (ret!= 0 || old_bpf_selector > 1) { 710 | send_perf_error(perf_ctx, OLD_BPF_SELECTOR_NOT_IN_PLACE); 711 | return 0; 712 | } else if(old_bpf_selector != bpf_selector) { 713 | switch_count = 1; 714 | conf.update(&switch_count_key, &switch_count); 715 | conf.update(&old_selector_key, &bpf_selector); 716 | } 717 | 718 | /** 719 | * Retrieve sampling step (dynamic window) 720 | * We need this because later on we check the timestamp of the 721 | * context switch, if it's higher than ts_begin_window + step 722 | * we need to account the current pcm in the next time window (so in 723 | * another bpf selector w.r.t. to the current one) 724 | * BEWARE: increasing the step in userspace means that the next 725 | * sample is invalid. Reducing the step in userspace is not 726 | * an issue, it discards data that has already been collected 727 | */ 728 | unsigned int step = 1000000000; 729 | ret = bpf_probe_read(&step, sizeof(step), conf.lookup(&step_key)); 730 | if (ret!= 0 || step < STEP_MIN || step > STEP_MAX) { 731 | send_perf_error(perf_ctx, TIMESTEP_NOT_IN_PLACE); 732 | return 0; 733 | } 734 | 735 | int current_pid = bpf_get_current_pid_tgid(); 736 | 737 | /* Read the values of the performance counters to update the data 738 | * inside our hashmap 739 | */ 740 | u64 processor_id = bpf_get_smp_processor_id(); 741 | #ifdef PERFORMANCE_COUNTERS 742 | u64 thread_cycles_sample = cycles_thread.perf_read(processor_id); 743 | u64 core_cycles_sample = cycles_core.perf_read(processor_id); 744 | u64 instruction_retired_thread = instr_thread.perf_read(processor_id); 745 | u64 cache_misses_thread = cache_misses.perf_read(processor_id); 746 | u64 cache_refs_thread = cache_refs.perf_read(processor_id); 747 | #endif 748 | u64 ts = bpf_ktime_get_ns(); 749 | 750 | if (ret == 0) { 751 | #ifdef PERFORMANCE_COUNTERS 752 | update_cycles_count(perf_ctx, current_pid, bpf_selector, step, processor_id, thread_cycles_sample, core_cycles_sample, instruction_retired_thread, cache_misses_thread, cache_refs_thread, ts); 753 | #else 754 | update_cycles_count(perf_ctx, current_pid, bpf_selector, step, processor_id, ts); 755 | #endif 756 | } 757 | 758 | 759 | // Fetch more data about processor we are currently dealing with 760 | ret = 0; 761 | struct proc_topology topology_info; 762 | ret = bpf_probe_read(&topology_info, sizeof(topology_info), processors.lookup(&processor_id)); 763 | if(ret!= 0 || topology_info.ht_id > NUM_CPUS) { 764 | send_perf_error(perf_ctx, CORRUPTED_TOPOLOGY_MAP); 765 | return 0; 766 | } 767 | 768 | //update topology info since we are forcing the update with a timer 769 | topology_info.running_pid = current_pid; 770 | topology_info.ts = ts; 771 | #ifdef PERFORMANCE_COUNTERS 772 | topology_info.cycles_thread = thread_cycles_sample; 773 | topology_info.cycles_core_delta_sibling = 0; 774 | topology_info.cycles_core = core_cycles_sample; 775 | topology_info.instruction_thread = instruction_retired_thread; 776 | topology_info.cache_misses = cache_misses_thread; 777 | topology_info.cache_refs = cache_refs_thread; 778 | #endif 779 | processors.update(&processor_id, &topology_info); 780 | 781 | global_timestamps.update(&bpf_selector, &ts); 782 | 783 | send_perf_error(perf_ctx, BPF_PROCEED_WITH_DEBUG_MODE); 784 | 785 | return 0; 786 | } 787 | -------------------------------------------------------------------------------- /bpf/vfs_monitor.c: -------------------------------------------------------------------------------- 1 | /* 2 | DEEP-mon 3 | Copyright (C) 2020 Brondolin Rolando 4 | 5 | This file is part of DEEP-mon 6 | 7 | DEEP-mon is free software: you can redistribute it and/or modify 8 | it under the terms of the GNU General Public License as published by 9 | the Free Software Foundation, either version 3 of the License, or 10 | (at your option) any later version. 11 | 12 | DEEP-mon is distributed in the hope that it will be useful, 13 | but WITHOUT ANY WARRANTY; without even the implied warranty of 14 | MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 15 | GNU General Public License for more details. 16 | 17 | You should have received a copy of the GNU General Public License 18 | along with this program. If not, see . 19 | */ 20 | 21 | #include 22 | #include 23 | #include 24 | #include 25 | #include 26 | #include 27 | #include 28 | 29 | struct val_t { 30 | u32 sz; 31 | u64 ts; 32 | u32 name_len; 33 | char name[DNAME_INLINE_LEN]; 34 | char parent1[DNAME_INLINE_LEN]; 35 | char parent2[DNAME_INLINE_LEN]; 36 | }; 37 | 38 | struct val_pid_t { 39 | u32 pid; 40 | u64 num_r; 41 | u64 num_w; 42 | u64 bytes_r; 43 | u64 bytes_w; 44 | u64 sum_ts_deltas; 45 | }; 46 | 47 | struct val_file_t { 48 | u64 num_r; 49 | u64 num_w; 50 | u64 bytes_r; 51 | u64 bytes_w; 52 | }; 53 | 54 | struct key_file_t { 55 | char name[DNAME_INLINE_LEN]; 56 | char parent1[DNAME_INLINE_LEN]; 57 | char parent2[DNAME_INLINE_LEN]; 58 | }; 59 | 60 | 61 | BPF_HASH(counts_by_pid, pid_t, struct val_pid_t); 62 | BPF_HASH(counts_by_file, struct key_file_t, struct val_file_t); 63 | BPF_HASH(entryinfo, pid_t, struct val_t); 64 | 65 | int trace_rw_entry(struct pt_regs *ctx, struct file *file, char __user *buf, size_t count) { 66 | u32 tgid = bpf_get_current_pid_tgid() >> 32; 67 | u32 pid = bpf_get_current_pid_tgid(); 68 | int mode = file->f_inode->i_mode; 69 | struct dentry *de = file->f_path.dentry; 70 | if (de->d_name.len == 0 || !S_ISREG(mode)) 71 | return 0; 72 | // store size and timestamp by pid 73 | struct val_t val = {}; 74 | val.sz = count; 75 | val.ts = bpf_ktime_get_ns(); 76 | struct qstr d_name = de->d_name; 77 | val.name_len = d_name.len; 78 | 79 | bpf_probe_read(&val.name, sizeof(val.name), d_name.name); 80 | 81 | struct dentry *parent = de->d_parent; 82 | if (parent) { 83 | struct qstr parent_name = parent->d_name; 84 | bpf_probe_read(&val.parent1, sizeof(val.parent1), parent_name.name); 85 | 86 | struct dentry *second_parent = parent->d_parent; 87 | 88 | struct qstr second_parent_name = second_parent->d_name; 89 | bpf_probe_read(&val.parent2, sizeof(val.parent2), second_parent_name.name); 90 | } 91 | 92 | entryinfo.update(&pid, &val); 93 | return 0; 94 | } 95 | 96 | static int trace_rw_return(struct pt_regs *ctx, int type) { 97 | struct val_t *valp; 98 | u32 pid = bpf_get_current_pid_tgid(); 99 | 100 | //searches for key value and discards request if not found 101 | valp = entryinfo.lookup(&pid); 102 | if (valp == 0) { 103 | return 0; 104 | } 105 | 106 | //calculates delta and removes key 107 | u64 delta_us = (bpf_ktime_get_ns() - valp->ts) / 1000; 108 | entryinfo.delete(&pid); 109 | 110 | struct val_pid_t *val_pid, zero_pid = {}; 111 | val_pid = counts_by_pid.lookup_or_init(&pid, &zero_pid); 112 | if (val_pid) { 113 | if (type == 0) { 114 | val_pid->num_r++; 115 | val_pid->bytes_r += valp->sz; 116 | } else { 117 | val_pid->num_w++; 118 | val_pid->bytes_w += valp->sz; 119 | } 120 | val_pid->sum_ts_deltas += delta_us; 121 | val_pid->pid = pid; 122 | } 123 | 124 | struct key_file_t file_key = {}; 125 | bpf_probe_read(&file_key.name, sizeof(file_key.name), valp->name); 126 | bpf_probe_read(&file_key.parent1, sizeof(file_key.parent1), valp->parent1); 127 | bpf_probe_read(&file_key.parent2, sizeof(file_key.parent2), valp->parent2); 128 | 129 | struct val_file_t *val_file, zero_file = {}; 130 | val_file = counts_by_file.lookup_or_init(&file_key, &zero_file); 131 | 132 | if (val_file) { 133 | if (type == 0) { 134 | val_file->num_r++; 135 | val_file->bytes_r += valp->sz; 136 | } else { 137 | val_file->num_w++; 138 | val_file->bytes_w += valp->sz; 139 | } 140 | } 141 | return 0; 142 | } 143 | 144 | int trace_read_return(struct pt_regs *ctx) { 145 | return trace_rw_return(ctx, 0); 146 | } 147 | int trace_write_return(struct pt_regs *ctx) { 148 | return trace_rw_return(ctx, 1); 149 | } 150 | -------------------------------------------------------------------------------- /config.yaml: -------------------------------------------------------------------------------- 1 | window_mode: "fixed" 2 | output_format: "curses" 3 | debug_mode: False 4 | send_thread_data: False 5 | net_monitor: True 6 | nat_trace: False 7 | print_net_details: False 8 | dynamic_tcp_client_port_masking: True 9 | power_measure: True 10 | memory_measure: False 11 | disk_measure: True 12 | file_measure: True 13 | -------------------------------------------------------------------------------- /deep_mon.py: -------------------------------------------------------------------------------- 1 | """ 2 | DEEP-mon 3 | Copyright (C) 2020 Brondolin Rolando 4 | 5 | This file is part of DEEP-mon 6 | 7 | DEEP-mon is free software: you can redistribute it and/or modify 8 | it under the terms of the GNU General Public License as published by 9 | the Free Software Foundation, either version 3 of the License, or 10 | (at your option) any later version. 11 | 12 | DEEP-mon is distributed in the hope that it will be useful, 13 | but WITHOUT ANY WARRANTY; without even the implied warranty of 14 | MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 15 | GNU General Public License for more details. 16 | 17 | You should have received a copy of the GNU General Public License 18 | along with this program. If not, see . 19 | """ 20 | 21 | import click 22 | import yaml 23 | try: 24 | from yaml import CLoader as Loader 25 | except ImportError: 26 | from yaml import Loader 27 | 28 | if __name__ == '__main__': 29 | from userspace.monitor_main import MonitorMain 30 | from userspace.curse import Curse 31 | else: 32 | from .userspace.monitor_main import MonitorMain 33 | from .userspace.curse import Curse 34 | 35 | # Load config file with default values 36 | config = {} 37 | try: 38 | with open('/home/config.yaml', 'r') as config_file: 39 | config = yaml.load(config_file, Loader=yaml.FullLoader) 40 | except IOError: 41 | try: 42 | with open('userspace/default_config.yaml', 'r') as default_config_file: 43 | config = yaml.load(default_config_file, Loader=yaml.FullLoader) 44 | except IOError: 45 | print("Couldn't find a config file, check your path") 46 | config = {} 47 | 48 | CONTEXT_SETTINGS = dict( 49 | default_map=config 50 | ) 51 | 52 | @click.command(context_settings=CONTEXT_SETTINGS) 53 | @click.option('--window-mode', '-w') 54 | @click.option('--output-format', '-o') 55 | @click.option('--debug-mode', '-d') 56 | @click.option('--net_monitor', '-n') 57 | @click.option('--nat_trace') 58 | @click.option('--print_net_details') 59 | @click.option('--dynamic_tcp_client_port_masking') 60 | @click.option('--power_measure') 61 | @click.option('--memory_measure') 62 | @click.option('--disk_measure') 63 | @click.option('--file_measure') 64 | def main(window_mode, output_format, debug_mode, net_monitor, nat_trace, print_net_details, dynamic_tcp_client_port_masking, power_measure, memory_measure, disk_measure, file_measure): 65 | monitor = MonitorMain(output_format, window_mode, debug_mode, net_monitor, nat_trace, print_net_details, dynamic_tcp_client_port_masking, power_measure, memory_measure, disk_measure, file_measure) 66 | if output_format == 'curses': 67 | curse = Curse(monitor, power_measure, net_monitor, memory_measure, disk_measure, file_measure) 68 | curse.start() 69 | elif output_format == 'console': 70 | monitor.monitor_loop() 71 | 72 | if __name__ == '__main__': 73 | main() 74 | -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | """ 2 | DEEP-mon 3 | Copyright (C) 2020 Brondolin Rolando 4 | 5 | This file is part of DEEP-mon 6 | 7 | DEEP-mon is free software: you can redistribute it and/or modify 8 | it under the terms of the GNU General Public License as published by 9 | the Free Software Foundation, either version 3 of the License, or 10 | (at your option) any later version. 11 | 12 | DEEP-mon is distributed in the hope that it will be useful, 13 | but WITHOUT ANY WARRANTY; without even the implied warranty of 14 | MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 15 | GNU General Public License for more details. 16 | 17 | You should have received a copy of the GNU General Public License 18 | along with this program. If not, see . 19 | """ 20 | 21 | from setuptools import setup, find_packages 22 | 23 | setup( 24 | name='deep-mon', 25 | version='0.1dev', 26 | packages=['deep_mon', 'deep_mon.bpf','deep_mon.userspace','deep_mon.userspace.rapl'], 27 | package_data={'deep_mon.bpf': ['*.c'], 'deep_mon.userspace': ['*.yaml']}, 28 | include_package_data=True, 29 | install_requires=[ 30 | 'Click', 31 | ], 32 | entry_points=''' 33 | [console_scripts] 34 | deep-mon=deep_mon.deep_mon:main 35 | ''', 36 | ) 37 | -------------------------------------------------------------------------------- /userspace/__init__.py: -------------------------------------------------------------------------------- 1 | """ 2 | DEEP-mon 3 | Copyright (C) 2020 Brondolin Rolando 4 | 5 | This file is part of DEEP-mon 6 | 7 | DEEP-mon is free software: you can redistribute it and/or modify 8 | it under the terms of the GNU General Public License as published by 9 | the Free Software Foundation, either version 3 of the License, or 10 | (at your option) any later version. 11 | 12 | DEEP-mon is distributed in the hope that it will be useful, 13 | but WITHOUT ANY WARRANTY; without even the implied warranty of 14 | MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 15 | GNU General Public License for more details. 16 | 17 | You should have received a copy of the GNU General Public License 18 | along with this program. If not, see . 19 | """ 20 | -------------------------------------------------------------------------------- /userspace/bpf_collector.py: -------------------------------------------------------------------------------- 1 | """ 2 | DEEP-mon 3 | Copyright (C) 2020 Brondolin Rolando 4 | 5 | This file is part of DEEP-mon 6 | 7 | DEEP-mon is free software: you can redistribute it and/or modify 8 | it under the terms of the GNU General Public License as published by 9 | the Free Software Foundation, either version 3 of the License, or 10 | (at your option) any later version. 11 | 12 | DEEP-mon is distributed in the hope that it will be useful, 13 | but WITHOUT ANY WARRANTY; without even the implied warranty of 14 | MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 15 | GNU General Public License for more details. 16 | 17 | You should have received a copy of the GNU General Public License 18 | along with this program. If not, see . 19 | """ 20 | 21 | from bcc import BPF, PerfType, PerfHWConfig, PerfSWConfig 22 | from .proc_topology import BpfProcTopology 23 | from .proc_topology import ProcTopology 24 | from .process_info import BpfPidStatus 25 | from .process_info import SocketProcessItem 26 | from .process_info import ProcessInfo 27 | from .sample_controller import SampleController 28 | import ctypes as ct 29 | import json 30 | import multiprocessing 31 | import os 32 | import time 33 | 34 | 35 | class bcolors: 36 | RED = '\033[91m' 37 | GREEN = '\033[92m' 38 | YELLOW = '\033[93m' 39 | BLUE = '\033[94m' 40 | MAGENTA = '\033[95m' 41 | CYAN = '\033[96m' 42 | WHITE = '\033[97m' 43 | ENDC = '\033[0m' 44 | BOLD = '\033[1m' 45 | UNDERLINE = '\033[4m' 46 | 47 | 48 | class BpfSample: 49 | 50 | def __init__(self, max_ts, total_time, sched_switch_count, timeslice, 51 | total_active_power, pid_dict, cpu_cores): 52 | self.max_ts = max_ts 53 | self.total_execution_time = total_time 54 | self.sched_switch_count = sched_switch_count 55 | self.timeslice = timeslice 56 | self.total_active_power = total_active_power 57 | self.pid_dict = pid_dict 58 | self.cpu_cores = cpu_cores 59 | 60 | def get_max_ts(self): 61 | return self.max_ts 62 | 63 | def get_total_execution_time(self): 64 | return self.total_execution_time 65 | 66 | def get_sched_switch_count(self): 67 | return self.sched_switch_count 68 | 69 | def get_timeslice(self): 70 | return self.timeslice 71 | 72 | def get_total_active_power(self): 73 | return self.total_active_power 74 | 75 | def get_pid_dict(self): 76 | return self.pid_dict 77 | 78 | def get_cpu_cores(self): 79 | return self.cpu_cores 80 | 81 | def __str__(self): 82 | str_representation = "" 83 | 84 | for key, value in sorted(self.pid_dict.items()): 85 | str_representation = str_representation + str(value) + "\n" 86 | 87 | str_representation = str_representation + self.get_log_line() 88 | 89 | return str_representation 90 | 91 | def get_log_dict(self): 92 | d = {} 93 | d["PROC TIME"] = "{:.3f}".format(self.total_execution_time) 94 | d["SCHED SWITCH COUNT"] = str(self.sched_switch_count) 95 | d["TIMESLICE"] = str(self.timeslice / 1000000000) 96 | d["TOTAL PACKAGE ACTIVE POWER"] = "{:.3f}".format(self.total_active_power["package"]) 97 | d["TOTAL CORE ACTIVE POWER"] = "{:.3f}".format(self.total_active_power["core"]) 98 | d["TOTAL DRAM ACTIVE POWER"] = "{:.3f}".format(self.total_active_power["dram"]) 99 | return d 100 | 101 | def get_log_line(self): 102 | str_representation = ( 103 | bcolors.YELLOW + "PROC TIME: " + bcolors.ENDC 104 | + "{:.3f}".format(self.total_execution_time) 105 | + "\t" + bcolors.YELLOW + "SCHED SWITCH COUNT: " + bcolors.ENDC 106 | + str(self.sched_switch_count) 107 | + "\t" + bcolors.YELLOW + "TIMESLICE: " + bcolors.ENDC 108 | + str(self.timeslice / 1000000000) + "s" 109 | + "\n\t" + bcolors.GREEN + "TOTAL PACKAGE ACTIVE POWER:\t" + bcolors.ENDC 110 | + "{:.3f}".format(self.total_active_power["package"]) 111 | + "\n\t" + bcolors.GREEN + "TOTAL CORE ACTIVE POWER:\t" + bcolors.ENDC 112 | + "{:.3f}".format(self.total_active_power["core"]) 113 | + "\n\t" + bcolors.GREEN + "TOTAL DRAM ACTIVE POWER:\t" + bcolors.ENDC 114 | + "{:.3f}".format(self.total_active_power["dram"]) 115 | ) 116 | return str_representation 117 | 118 | def get_log_json(self): 119 | d = {"PROC TIME": str(self.total_execution_time), 120 | "SCHED SWITCH COUNT": str(self.sched_switch_count), 121 | "TIMESLICE": str(self.timeslice), 122 | "TOTAL PACKAGE ACTIVE POWER": str(self.total_active_power["package"]), 123 | "TOTAL CORE ACTIVE POWER": str(self.total_active_power["core"]), 124 | "TOTAL DRAM ACTIVE POWER": str(self.total_active_power["dram"]) 125 | } 126 | return json.dumps(d, indent=4) 127 | 128 | 129 | class ErrorCode(ct.Structure): 130 | _fields_ = [("err", ct.c_int)] 131 | 132 | class BPFErrors: 133 | error_dict = {-1: "BPF_PROCEED_WITH_DEBUG_MODE", \ 134 | -2: "BPF_SELECTOR_NOT_IN_PLACE", \ 135 | -3: "OLD_BPF_SELECTOR_NOT_IN_PLACE", \ 136 | -4: "TIMESTEP_NOT_IN_PLACE", \ 137 | -5: "CORRUPTED_TOPOLOGY_MAP", \ 138 | -6: "WRONG_SIBLING_TOPOLOGY_MAP", \ 139 | -7: "THREAD_MIGRATED_UNEXPECTEDLY"} 140 | 141 | 142 | class BpfCollector: 143 | 144 | def __init__(self, topology, debug, power_measure): 145 | self.topology = topology 146 | self.debug = debug 147 | self.power_measure = power_measure 148 | bpf_code_path = os.path.dirname(os.path.abspath(__file__)) \ 149 | + "/../bpf/bpf_monitor.c" 150 | if debug is False: 151 | if self.power_measure == True: 152 | self.bpf_program = BPF(src_file=bpf_code_path, \ 153 | cflags=["-DNUM_CPUS=%d" % multiprocessing.cpu_count(), \ 154 | "-DNUM_SOCKETS=%d" % len(self.topology.get_sockets()), \ 155 | "-DPERFORMANCE_COUNTERS"]) 156 | else: 157 | self.bpf_program = BPF(src_file=bpf_code_path, \ 158 | cflags=["-DNUM_CPUS=%d" % multiprocessing.cpu_count(), \ 159 | "-DNUM_SOCKETS=%d" % len(self.topology.get_sockets())]) 160 | else: 161 | self.bpf_program = BPF(src_file=bpf_code_path, \ 162 | cflags=["-DNUM_CPUS=%d" % multiprocessing.cpu_count(), \ 163 | "-DNUM_SOCKETS=%d" % len(self.topology.get_sockets()), \ 164 | "-DDEBUG"]) 165 | 166 | self.processors = self.bpf_program.get_table("processors") 167 | self.pids = self.bpf_program.get_table("pids") 168 | self.idles = self.bpf_program.get_table("idles") 169 | self.bpf_config = self.bpf_program.get_table("conf") 170 | self.bpf_global_timestamps = self.bpf_program.get_table("global_timestamps") 171 | self.selector = 0 172 | self.SELECTOR_DIM = 2 173 | self.timeslice = 1000000000 174 | self.timed_capture = False 175 | 176 | #self.bpf_program["cpu_cycles"].open_perf_event(PerfType.HARDWARE, \ 177 | # PerfHWConfig.CPU_CYCLES) 178 | # 4 means RAW_TYPE 179 | # int("73003c",16) is the hex for UNHALTED_CORE_CYCLES for any thread 180 | # int("53003c",16) is the hex for UNHALTED_CORE_CYCLES 181 | # int("5300c0",16) is the hex for INSTRUCTION_RETIRED 182 | if self.power_measure == True: 183 | self.bpf_program["cycles_core"].open_perf_event(4, int("73003c",16)) 184 | self.bpf_program["cycles_thread"].open_perf_event(4, int("53003c",16)) 185 | self.bpf_program["instr_thread"].open_perf_event(4, int("5300c0",16)) 186 | self.bpf_program["cache_misses"].open_perf_event(PerfType.HARDWARE, PerfHWConfig.CACHE_MISSES) 187 | self.bpf_program["cache_refs"].open_perf_event(PerfType.HARDWARE, PerfHWConfig.CACHE_REFERENCES) 188 | 189 | 190 | def print_event(self, cpu, data, size): 191 | event = ct.cast(data, ct.POINTER(ErrorCode)).contents 192 | if event.err >= 0: 193 | print("core: " + str(cpu) + " topology counters overflow or initialized with pid: " + str(event.err)) 194 | elif event.err < -1: 195 | # exclude the BPF_PROCEED_WITH_DEBUG_MODE event, since it is used 196 | # just to advance computation for the timed capture 197 | print("core: " + str(cpu) + " " + str(BPFErrors.error_dict[event.err])) 198 | 199 | 200 | def start_capture(self, timeslice): 201 | for key, value in self.topology.get_new_bpf_topology().items(): 202 | self.processors[ct.c_ulonglong(key)] = value 203 | 204 | self.timed_capture = False 205 | self.timeslice = timeslice 206 | self.bpf_config[ct.c_int(0)] = ct.c_uint(self.selector) # current selector 207 | self.bpf_config[ct.c_int(1)] = ct.c_uint(self.selector) # old selector 208 | self.bpf_config[ct.c_int(2)] = ct.c_uint(self.timeslice) # timeslice 209 | self.bpf_config[ct.c_int(3)] = ct.c_uint(0) # switch count 210 | 211 | if self.debug == True: 212 | self.bpf_program["err"].open_perf_buffer(self.print_event, page_cnt=256) 213 | 214 | self.bpf_program.attach_tracepoint(tp="sched:sched_switch", \ 215 | fn_name="trace_switch") 216 | self.bpf_program.attach_tracepoint(tp="sched:sched_process_exit", \ 217 | fn_name="trace_exit") 218 | 219 | def start_timed_capture(self, count=0, frequency=0): 220 | if frequency: 221 | sample_freq = frequency 222 | sample_period = 0 223 | self.timeslice = int((1 / float(frequency)) * 1000000000) 224 | elif count: 225 | sample_freq = 0 226 | sample_period = count 227 | self.timeslice = int(sample_period * 1000000000) 228 | else: 229 | # If user didn't specify anything, use default 49Hz sampling 230 | sample_freq = 49 231 | sample_period = 0 232 | self.timeslice = int((1 / float(frequency)) * 1000000000) 233 | 234 | self.timed_capture = True 235 | 236 | for key, value in self.topology.get_new_bpf_topology().items(): 237 | self.processors[ct.c_ulonglong(key)] = value 238 | 239 | self.bpf_config[ct.c_int(0)] = ct.c_uint(self.selector) # current selector 240 | self.bpf_config[ct.c_int(1)] = ct.c_uint(self.selector) # old selector 241 | self.bpf_config[ct.c_int(2)] = ct.c_uint(self.timeslice) # timeslice 242 | self.bpf_config[ct.c_int(3)] = ct.c_uint(0) # switch count 243 | 244 | 245 | if self.debug == True: 246 | self.bpf_program["err"].open_perf_buffer(self.print_event, page_cnt=256) 247 | 248 | self.bpf_program.attach_tracepoint(tp="sched:sched_switch", \ 249 | fn_name="trace_switch") 250 | self.bpf_program.attach_tracepoint(tp="sched:sched_process_exit", \ 251 | fn_name="trace_exit") 252 | self.bpf_program.attach_perf_event(ev_type=PerfType.SOFTWARE, 253 | ev_config=PerfSWConfig.CPU_CLOCK, fn_name="timed_trace", 254 | sample_period=sample_period, sample_freq=sample_freq) 255 | 256 | def stop_capture(self): 257 | self.bpf_program.detach_tracepoint(tp="sched:sched_switch") 258 | self.bpf_program.detach_tracepoint(tp="sched:sched_process_exit") 259 | 260 | def get_new_sample(self, sample_controller, rapl_monitor): 261 | sample = self._get_new_sample(rapl_monitor) 262 | if not self.timed_capture: 263 | sample_controller.compute_sleep_time(sample.get_sched_switch_count()) 264 | self.timeslice = sample_controller.get_timeslice() 265 | self.bpf_config[ct.c_int(2)] = ct.c_uint(self.timeslice) # timeslice 266 | 267 | if self.debug == True: 268 | self.bpf_program.kprobe_poll() 269 | 270 | return sample 271 | 272 | def _get_new_sample(self, rapl_monitor): 273 | 274 | total_execution_time = 0.0 275 | sched_switch_count = self.bpf_config[ct.c_int(3)].value 276 | tsmax = 0 277 | 278 | # Initialize the weighted cycles for each core to 0 279 | total_weighted_cycles = [] 280 | for socket in self.topology.get_sockets(): 281 | total_weighted_cycles.append(0) 282 | 283 | # We use a binary selector so that while userspace is reading events 284 | # using selector 0 we write events using selector 1 and vice versa. 285 | # Here we initialize it to 0 and set the number of slots used for 286 | # read/write equal to the number of sockets * the number of selectors 287 | read_selector = 0 288 | total_slots_length = len(self.topology.get_sockets())*self.SELECTOR_DIM 289 | 290 | # Every time we get a new sample we want to switch the selector we are using 291 | if self.selector == 0: 292 | self.selector = 1 293 | read_selector = 0 294 | else: 295 | self.selector = 0 296 | read_selector = 1 297 | 298 | rapl_measurement = [] 299 | package_diff = 0 300 | core_diff = 0 301 | dram_diff = 0 302 | if self.power_measure == True: 303 | # Get new sample from rapl right before changing selector in eBPF 304 | rapl_measurement = rapl_monitor.get_rapl_measure() 305 | 306 | package_diff = rapl_measurement["package"] 307 | core_diff = rapl_measurement["core"] 308 | dram_diff = rapl_measurement["dram"] 309 | 310 | # Propagate the update of the selector to the eBPF program 311 | self.bpf_config[ct.c_int(0)] = ct.c_uint(self.selector) 312 | 313 | pid_dict = {} 314 | 315 | tsmax = self.bpf_global_timestamps[ct.c_int(read_selector)].value 316 | 317 | 318 | # Add the count of clock cycles for each active process to the total 319 | # number of clock cycles of the socket 320 | for key, data in self.pids.items(): 321 | if data.ts[read_selector] + self.timeslice > tsmax: 322 | total_execution_time = total_execution_time + float(data.time_ns[read_selector])/1000000 323 | 324 | if self.power_measure == True: 325 | for multisocket_selector in range(read_selector, total_slots_length, self.SELECTOR_DIM): 326 | # Compute the number of total weighted cycles per socket 327 | cycles_index = int(multisocket_selector/self.SELECTOR_DIM) 328 | if data.ts[read_selector] + self.timeslice > tsmax: 329 | total_weighted_cycles[cycles_index] = total_weighted_cycles[cycles_index] + data.weighted_cycles[multisocket_selector] 330 | 331 | # Add the count of clock cycles for each idle process to the total 332 | # number of clock cycles of the socket 333 | for key, data in self.idles.items(): 334 | if data.ts[read_selector] + self.timeslice > tsmax: 335 | total_execution_time = total_execution_time + float(data.time_ns[read_selector])/1000000 336 | 337 | if self.power_measure == True: 338 | for multisocket_selector in range(read_selector, total_slots_length, self.SELECTOR_DIM): 339 | # Compute the number of total weighted cycles per socket 340 | cycles_index = int(multisocket_selector/self.SELECTOR_DIM) 341 | if data.ts[read_selector] + self.timeslice > tsmax: 342 | total_weighted_cycles[cycles_index] = total_weighted_cycles[cycles_index] + data.weighted_cycles[multisocket_selector] 343 | 344 | if self.power_measure == True: 345 | # Compute package/core/dram power in mW from RAPL samples 346 | package_power = [package_diff[skt].power_milliw() 347 | for skt in self.topology.get_sockets()] 348 | core_power = [core_diff[skt].power_milliw() 349 | for skt in self.topology.get_sockets()] 350 | dram_power = [dram_diff[skt].power_milliw() 351 | for skt in self.topology.get_sockets()] 352 | total_power = { 353 | "package": sum(package_power), 354 | "core": sum(core_power), 355 | "dram": sum(dram_power) 356 | } 357 | else: 358 | total_power = { 359 | "package": 0, 360 | "core": 0, 361 | "dram": 0 362 | } 363 | 364 | for key, data in self.pids.items(): 365 | 366 | proc_info = ProcessInfo(len(self.topology.get_sockets())) 367 | proc_info.set_pid(data.pid) 368 | proc_info.set_tgid(data.tgid) 369 | proc_info.set_comm(data.comm) 370 | proc_info.set_cycles(data.cycles[read_selector]) 371 | proc_info.set_instruction_retired(data.instruction_retired[read_selector]) 372 | proc_info.set_cache_misses(data.cache_misses[read_selector]) 373 | proc_info.set_cache_refs(data.cache_refs[read_selector]) 374 | proc_info.set_time_ns(data.time_ns[read_selector]) 375 | add_proc = False 376 | 377 | for multisocket_selector in range(read_selector, total_slots_length, self.SELECTOR_DIM): 378 | 379 | if data.ts[read_selector] + self.timeslice > tsmax: 380 | socket_info = SocketProcessItem() 381 | socket_info.set_weighted_cycles(data.weighted_cycles[multisocket_selector]) 382 | socket_info.set_ts(data.ts[read_selector]) 383 | proc_info.set_socket_data(int(multisocket_selector/self.SELECTOR_DIM), socket_info) 384 | add_proc = True 385 | 386 | if add_proc: 387 | pid_dict[data.pid] = proc_info 388 | 389 | if self.power_measure == True: 390 | proc_info.set_power(self._get_pid_power(proc_info, total_weighted_cycles, core_power)) 391 | else: 392 | proc_info.set_power(0) 393 | proc_info.compute_cpu_usage_millis(float(total_execution_time), multiprocessing.cpu_count()) 394 | 395 | for key, data in self.idles.items(): 396 | 397 | proc_info = ProcessInfo(len(self.topology.get_sockets())) 398 | proc_info.set_pid(data.pid) 399 | proc_info.set_tgid(-1 * (1 + int(key.value))) 400 | proc_info.set_comm(data.comm) 401 | proc_info.set_cycles(data.cycles[read_selector]) 402 | proc_info.set_instruction_retired(data.instruction_retired[read_selector]) 403 | proc_info.set_cache_misses(data.cache_misses[read_selector]) 404 | proc_info.set_cache_refs(data.cache_refs[read_selector]) 405 | proc_info.set_time_ns(data.time_ns[read_selector]) 406 | add_proc = False 407 | 408 | for multisocket_selector in range(read_selector, total_slots_length, self.SELECTOR_DIM): 409 | 410 | if data.ts[read_selector] + self.timeslice > tsmax: 411 | 412 | socket_info = SocketProcessItem() 413 | socket_info.set_weighted_cycles(data.weighted_cycles[multisocket_selector]) 414 | socket_info.set_ts(data.ts[read_selector]) 415 | proc_info.set_socket_data(int(multisocket_selector/self.SELECTOR_DIM), socket_info) 416 | add_proc = True 417 | 418 | if add_proc: 419 | pid_dict[-1 * (1 + int(key.value))] = proc_info 420 | if self.power_measure == True: 421 | proc_info.set_power(self._get_pid_power(proc_info, total_weighted_cycles, core_power)) 422 | else: 423 | proc_info.set_power(0) 424 | proc_info.compute_cpu_usage_millis(float(total_execution_time), multiprocessing.cpu_count()) 425 | 426 | return BpfSample(tsmax, total_execution_time, sched_switch_count, self.timeslice, total_power, pid_dict, self.topology.get_hyperthread_count()) 427 | 428 | def _get_pid_power(self, pid, total_cycles, core_power): 429 | 430 | pid_power = 0 431 | for socket in self.topology.get_sockets(): 432 | if float(total_cycles[socket]) > 0: 433 | pid_power = pid_power + (core_power[socket] * \ 434 | (float(pid.get_socket_data(socket).get_weighted_cycles()) \ 435 | / float(total_cycles[socket]))) 436 | return pid_power 437 | -------------------------------------------------------------------------------- /userspace/container_info.py: -------------------------------------------------------------------------------- 1 | """ 2 | DEEP-mon 3 | Copyright (C) 2020 Brondolin Rolando 4 | 5 | This file is part of DEEP-mon 6 | 7 | DEEP-mon is free software: you can redistribute it and/or modify 8 | it under the terms of the GNU General Public License as published by 9 | the Free Software Foundation, either version 3 of the License, or 10 | (at your option) any later version. 11 | 12 | DEEP-mon is distributed in the hope that it will be useful, 13 | but WITHOUT ANY WARRANTY; without even the implied warranty of 14 | MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 15 | GNU General Public License for more details. 16 | 17 | You should have received a copy of the GNU General Public License 18 | along with this program. If not, see . 19 | """ 20 | 21 | import json 22 | import time 23 | from .net_collector import TransactionData 24 | from .net_collector import TransactionType 25 | from .net_collector import TransactionRole 26 | import numpy as np 27 | from ddsketch.ddsketch import DDSketch 28 | 29 | class bcolors: 30 | RED = '\033[91m' 31 | GREEN = '\033[92m' 32 | YELLOW = '\033[93m' 33 | BLUE = '\033[94m' 34 | MAGENTA = '\033[95m' 35 | CYAN = '\033[96m' 36 | WHITE = '\033[97m' 37 | ENDC = '\033[0m' 38 | BOLD = '\033[1m' 39 | UNDERLINE = '\033[4m' 40 | 41 | class ContainerInfo: 42 | 43 | def __init__(self, container_id): 44 | self.container_id = container_id 45 | self.container_name = None 46 | self.container_image = None 47 | self.container_labels = None 48 | 49 | self.cycles = 0 50 | self.weighted_cycles = 0 51 | self.instruction_retired = 0 52 | self.cache_misses = 0 53 | self.cache_refs = 0 54 | self.time_ns = 0 55 | self.power = 0.0 56 | self.cpu_usage = 0.0 57 | self.pid_set = set() 58 | self.timestamp = 0 59 | self.network_transactions = [] 60 | self.nat_rules = [] 61 | 62 | #memory metrics 63 | self.mem_RSS = 0 64 | self.mem_PSS = 0 65 | self.mem_USS = 0 66 | #disk metrics 67 | self.kb_r = 0 68 | self.kb_w = 0 69 | self.num_r = 0 70 | self.num_w = 0 71 | self.disk_avg_lat = 0 72 | 73 | self.tcp_transaction_count = 0 74 | self.tcp_transaction_count_client = 0 75 | self.tcp_transaction_count_server = 0 76 | self.tcp_byte_tx = 0 77 | self.tcp_byte_rx = 0 78 | self.tcp_avg_latency = 0 79 | self.tcp_avg_latency_client = 0 80 | self.tcp_avg_latency_server = 0 81 | self.tcp_percentiles = [] 82 | self.tcp_percentiles_client = [] 83 | self.tcp_percentiles_server = [] 84 | 85 | self.http_transaction_count = 0 86 | self.http_transaction_count_client = 0 87 | self.http_transaction_count_server = 0 88 | self.http_byte_tx = 0 89 | self.http_byte_rx = 0 90 | self.http_avg_latency = 0 91 | self.http_avg_latency_client = 0 92 | self.http_avg_latency_server = 0 93 | self.http_percentiles = [] 94 | self.http_percentiles_client = [] 95 | self.http_percentiles_server = [] 96 | 97 | self.pct = [50,75,90,99,99.9,99.99,99.999] 98 | 99 | self.network_threads = 0 100 | self.weighted_threads = 0 101 | self.weighted_cpus = [] 102 | 103 | def add_weighted_cycles(self, new_cycles): 104 | self.weighted_cycles = self.weighted_cycles + new_cycles 105 | 106 | def add_cycles(self, new_cycles): 107 | self.cycles = self.cycles + new_cycles 108 | 109 | def add_time_ns(self, new_time_ns): 110 | self.time_ns = self.time_ns + new_time_ns 111 | 112 | def add_power(self, new_power): 113 | self.power = self.power + float(new_power) 114 | 115 | def add_instructions(self, new_instructions): 116 | self.instruction_retired = self.instruction_retired + new_instructions 117 | 118 | def add_cache_misses(self, new_cache_misses): 119 | self.cache_misses = self.cache_misses + new_cache_misses 120 | 121 | def add_cache_refs(self, new_cache_refs): 122 | self.cache_refs = self.cache_refs + new_cache_refs 123 | 124 | def add_cpu_usage(self, cpu_usage): 125 | self.cpu_usage = self.cpu_usage + float(cpu_usage) 126 | self.add_weighted_cpu_usage(cpu_usage) 127 | 128 | def add_pid(self, new_pid): 129 | self.pid_set.add(new_pid) 130 | 131 | def add_network_transactions(self, transaction_list): 132 | self.network_transactions.extend(transaction_list) 133 | self.network_threads = self.network_threads + 1 134 | 135 | def add_nat_rules(self, nat_list): 136 | self.nat_rules.extend(nat_list) 137 | 138 | def set_container_name(self, container_name): 139 | self.container_name = container_name 140 | 141 | def set_container_image(self, container_image): 142 | self.container_image = container_image 143 | 144 | def set_container_labels(self, container_labels): 145 | self.container_labels = container_labels 146 | 147 | def set_mem_RSS(self, rss): 148 | self.mem_RSS = rss 149 | 150 | def set_mem_PSS(self, pss): 151 | self.mem_PSS = pss 152 | 153 | def set_mem_USS(self, uss): 154 | self.mem_USS = uss 155 | 156 | def set_disk_kb_r(self, kb_r): 157 | self.kb_r = kb_r 158 | 159 | def set_disk_kb_w(self, kb_w): 160 | self.kb_w = kb_w 161 | 162 | def set_disk_num_r(self, num_r): 163 | self.num_r = num_r 164 | 165 | def set_disk_num_w(self, num_w): 166 | self.num_w = num_w 167 | 168 | def set_disk_avg_lat(self, avg_lat): 169 | self.disk_avg_lat = avg_lat 170 | 171 | def add_weighted_cpu_usage(self, cpu_usage): 172 | self.weighted_cpus.append(cpu_usage) 173 | max = 0 174 | #compute max 175 | for usage in self.weighted_cpus: 176 | if max < usage: 177 | max = usage 178 | 179 | # how many max do we have here? 180 | maxes = 0 181 | bin = 0 182 | for usage in self.weighted_cpus: 183 | bin = bin + usage 184 | if bin >= max: 185 | maxes = maxes + 1 186 | bin = bin - max 187 | 188 | self.weighted_threads = maxes 189 | 190 | def compute_aggregate_network_metrics(self): 191 | if self.network_transactions != []: 192 | http_transactions = DDSketch() 193 | http_transactions_client = DDSketch() 194 | http_transactions_server = DDSketch() 195 | tcp_transactions = DDSketch() 196 | tcp_transactions_client = DDSketch() 197 | tcp_transactions_server = DDSketch() 198 | 199 | for transaction in self.network_transactions: 200 | if transaction.type == TransactionType.ipv4_http or transaction.type == TransactionType.ipv6_http: 201 | self.http_transaction_count = self.http_transaction_count + transaction.get_transaction_count() 202 | self.http_byte_rx = self.http_byte_rx + transaction.get_byte_rx() 203 | self.http_byte_tx = self.http_byte_tx + transaction.get_byte_tx() 204 | self.http_avg_latency = self.http_avg_latency + transaction.get_avg_latency() * transaction.get_transaction_count() 205 | http_transactions.merge(transaction.get_samples()) 206 | 207 | if transaction.role == TransactionRole.client: 208 | self.http_transaction_count_client = self.http_transaction_count_client + transaction.get_transaction_count() 209 | self.http_avg_latency_client = self.http_avg_latency_client + transaction.get_avg_latency() * transaction.get_transaction_count() 210 | http_transactions_client.merge(transaction.get_samples()) 211 | else: 212 | self.http_transaction_count_server = self.http_transaction_count_server + transaction.get_transaction_count() 213 | self.http_avg_latency_server = self.http_avg_latency_server + transaction.get_avg_latency() * transaction.get_transaction_count() 214 | http_transactions_server.merge(transaction.get_samples()) 215 | 216 | else: 217 | self.tcp_transaction_count = self.tcp_transaction_count + transaction.get_transaction_count() 218 | self.tcp_byte_rx = self.tcp_byte_rx + transaction.get_byte_rx() 219 | self.tcp_byte_tx = self.tcp_byte_tx + transaction.get_byte_tx() 220 | self.tcp_avg_latency = self.tcp_avg_latency + transaction.get_avg_latency() * transaction.get_transaction_count() 221 | tcp_transactions.merge(transaction.get_samples()) 222 | 223 | if transaction.role == TransactionRole.client: 224 | self.tcp_transaction_count_client = self.tcp_transaction_count_client + transaction.get_transaction_count() 225 | self.tcp_avg_latency_client = self.tcp_avg_latency_client + transaction.get_avg_latency() * transaction.get_transaction_count() 226 | tcp_transactions_client.merge(transaction.get_samples()) 227 | else: 228 | self.tcp_transaction_count_server = self.tcp_transaction_count_server + transaction.get_transaction_count() 229 | self.tcp_avg_latency_server = self.tcp_avg_latency_server + transaction.get_avg_latency() * transaction.get_transaction_count() 230 | tcp_transactions_server.merge(transaction.get_samples()) 231 | 232 | if self.http_transaction_count > 0: 233 | self.http_avg_latency = self.http_avg_latency / float(self.http_transaction_count) 234 | self.http_percentiles = self.compute_container_percentiles(http_transactions) 235 | 236 | if self.http_transaction_count_client > 0: 237 | self.http_avg_latency_client = self.http_avg_latency_client / float(self.http_transaction_count_client) 238 | self.http_percentiles_client = self.compute_container_percentiles(http_transactions_client) 239 | if self.http_transaction_count_server > 0: 240 | self.http_avg_latency_server = self.http_avg_latency_server / float(self.http_transaction_count_server) 241 | self.http_percentiles_server = self.compute_container_percentiles(http_transactions_server) 242 | 243 | if self.tcp_transaction_count > 0: 244 | self.tcp_avg_latency = self.tcp_avg_latency / float(self.tcp_transaction_count) 245 | self.tcp_percentiles = self.compute_container_percentiles(tcp_transactions) 246 | 247 | if self.tcp_transaction_count_client > 0: 248 | self.tcp_avg_latency_client = self.tcp_avg_latency_client / float(self.tcp_transaction_count_client) 249 | self.tcp_percentiles_client = self.compute_container_percentiles(tcp_transactions_client) 250 | if self.tcp_transaction_count_server > 0: 251 | self.tcp_avg_latency_server = self.tcp_avg_latency_server / float(self.tcp_transaction_count_server) 252 | self.tcp_percentiles_server = self.compute_container_percentiles(tcp_transactions_server) 253 | 254 | def compute_container_percentiles(self, latency_sketch): 255 | out = [] 256 | for p in self.pct: 257 | out.append(latency_sketch.get_quantile_value(p/100)) 258 | return out 259 | 260 | def set_timestamp(self, ts): 261 | self.timestamp = ts 262 | 263 | def set_last_ts(self, ts): 264 | if(self.timestamp < ts): 265 | self.timestamp = ts 266 | 267 | def get_container_name(self): 268 | return self.container_name 269 | 270 | def get_container_image(self): 271 | return self.container_image 272 | 273 | def get_container_labels(self): 274 | return self.container_labels 275 | 276 | def get_cycles(self): 277 | return self.cycles 278 | 279 | def get_weighted_cycles(self): 280 | return self.weighted_cycles 281 | 282 | def get_instruction_retired(self): 283 | return self.instruction_retired 284 | 285 | def get_cache_misses(self): 286 | return self.cache_misses 287 | 288 | def get_cache_refs(self): 289 | return self.cache_refs 290 | 291 | def get_time_ns(self): 292 | return self.time_ns 293 | 294 | def get_power(self): 295 | return self.power 296 | 297 | def get_cpu_usage(self): 298 | return self.cpu_usage 299 | 300 | def get_pid_set(self): 301 | return self.pid_set 302 | 303 | def get_timestamp(self): 304 | return self.timestamp 305 | 306 | def get_network_transactions(self): 307 | return self.network_transactions 308 | 309 | def get_mem_RSS(self): 310 | return self.mem_RSS 311 | 312 | def get_mem_PSS(self): 313 | return self.mem_PSS 314 | 315 | def get_mem_USS(self): 316 | return self.mem_USS 317 | 318 | def get_kb_r(self): 319 | return self.kb_r 320 | 321 | def get_kb_w(self): 322 | return self.kb_w 323 | 324 | def get_num_r(self): 325 | return self.num_r 326 | 327 | def get_num_w(self): 328 | return self.num_w 329 | 330 | def get_disk_avg_lat(self): 331 | return self.disk_avg_lat 332 | 333 | def get_http_transaction_count(self): 334 | return self.http_transaction_count 335 | 336 | def get_http_byte_tx(self): 337 | return self.http_byte_tx 338 | 339 | def get_http_byte_rx(self): 340 | return self.http_byte_rx 341 | 342 | def get_http_avg_latency(self): 343 | return self.http_avg_latency 344 | 345 | def get_tcp_transaction_count(self): 346 | return self.tcp_transaction_count 347 | 348 | def get_tcp_byte_tx(self): 349 | return self.tcp_byte_tx 350 | 351 | def get_tcp_byte_rx(self): 352 | return self.tcp_byte_rx 353 | 354 | def get_tcp_avg_latency(self): 355 | return self.tcp_avg_latency 356 | 357 | def get_rewritten_network_transactions(self): 358 | 359 | for index in range(len(self.network_transactions)): 360 | transaction = self.network_transactions[index] 361 | 362 | # find if there are nat rules to be added or substituted 363 | src_modified = False 364 | dst_modified = False 365 | for nat_rule in self.nat_rules: 366 | # start with transaction src and look at both ends of nat rules 367 | if src_modified == False and nat_rule.get_saddr() == transaction.get_saddr() and nat_rule.get_lport() == transaction.get_lport(): 368 | # rewrite transaction source 369 | transaction.set_saddr(nat_rule.get_daddr()) 370 | transaction.set_lport(nat_rule.get_dport()) 371 | src_modified = True 372 | # print(nat_rule) 373 | # 374 | # if src_modified == False and nat_rule.get_daddr() == transaction.get_saddr() and nat_rule.get_dport() == transaction.get_lport(): 375 | # # rewrite transaction source 376 | # transaction.set_saddr(nat_rule.get_saddr()) 377 | # transaction.set_lport(nat_rule.get_lport()) 378 | # src_modified = True 379 | # # print(nat_rule) 380 | 381 | # if dst_modified == False and nat_rule.get_saddr() == transaction.get_daddr() and nat_rule.get_lport() == transaction.get_dport(): 382 | # # rewrite transaction source 383 | # transaction.set_daddr(nat_rule.get_daddr()) 384 | # transaction.set_dport(nat_rule.get_dport()) 385 | # dst_modified = True 386 | # # print(nat_rule) 387 | 388 | if dst_modified == False and nat_rule.get_daddr() == transaction.get_daddr() and nat_rule.get_dport() == transaction.get_dport(): 389 | # rewrite transaction source 390 | transaction.set_daddr(nat_rule.get_saddr()) 391 | transaction.set_dport(nat_rule.get_lport()) 392 | dst_modified = True 393 | # print(nat_rule) 394 | 395 | if src_modified and dst_modified: 396 | break 397 | self.network_transactions[index] = transaction 398 | 399 | return self.network_transactions 400 | 401 | 402 | def get_nat_rules(self): 403 | return self.nat_rules 404 | 405 | def get_http_percentiles(self): 406 | return [self.pct, self.http_percentiles] 407 | 408 | def get_tcp_percentiles(self): 409 | return [self.pct, self.tcp_percentiles] 410 | 411 | def to_dict(self): 412 | return {'container_id': self.container_id, 413 | 'cycles': self.cycles, 414 | 'weighted_cycles': self.weighted_cycles, 415 | 'instruction_retired': self.instruction_retired, 416 | 'cache_misses': self.cache_misses, 417 | 'cache_refs': self.cache_refs, 418 | 'cycles': self.cycles, 419 | 'time_ns': self.time_ns, 420 | 'power': self.power, 421 | 'cpu_usage': self.cpu_usage, 422 | 'pid_set': self.pid_set 423 | } 424 | 425 | def to_json(self): 426 | d = self.to_dict() 427 | d['pid_set'] = list(d['pid_set']) 428 | return json.dumps(d, indent=4) 429 | 430 | def __str__(self): 431 | fmt = '{:<28} {:<32} {:<34} {:<34} {:<34} {:<34} {:<38} {:<30} {:<30}' 432 | output_line = fmt.format ( 433 | bcolors.BLUE + "ID: " + bcolors.ENDC 434 | + self.container_id, 435 | bcolors.BLUE + "CYCLES: " + bcolors.ENDC 436 | + str(self.cycles), 437 | bcolors.BLUE + "W_CYCLES: " + bcolors.ENDC 438 | + str(self.weighted_cycles), 439 | bcolors.BLUE + "INSTR RET: " + bcolors.ENDC 440 | + str(self.instruction_retired), 441 | bcolors.BLUE + "CACHE MISS: " + bcolors.ENDC 442 | + str(self.cache_misses), 443 | bcolors.BLUE + "CACHE REFS: " + bcolors.ENDC 444 | +str(self.cache_refs), 445 | bcolors.BLUE + "EXEC TIME (s): " + bcolors.ENDC 446 | + '{:.5f}'.format(self.time_ns / 1000000000), 447 | bcolors.BLUE + "CPU USAGE: " + bcolors.ENDC 448 | + '{:.3f}'.format(self.cpu_usage), 449 | bcolors.GREEN + "TOTAL POWER (mW): " + bcolors.ENDC 450 | + '{:.3f}'.format(self.power) 451 | ) 452 | 453 | if self.mem_RSS > 0: 454 | fmt = '{:<20} {:<23} {:<23} {:<23}' 455 | output_line = output_line + "\n" + fmt.format( 456 | bcolors.GREEN + "\tMemory (kB): " + bcolors.ENDC, 457 | bcolors.BLUE + "RSS: " + bcolors.ENDC 458 | + str(self.mem_RSS), 459 | bcolors.BLUE + "PSS: " + bcolors.ENDC 460 | + str(self.mem_PSS), 461 | bcolors.BLUE + "USS: " + bcolors.ENDC 462 | + str(self.mem_USS) 463 | ) 464 | 465 | if (self.kb_r > 0 or self.kb_w > 0): 466 | fmt = '{:<20} {:<23} {:<23} {:<23} {:<23} {:23}' 467 | output_line = output_line + "\n" + fmt.format( 468 | bcolors.GREEN + "\tDisk Stats: " + bcolors.ENDC, 469 | bcolors.BLUE + "Kb R: " + bcolors.ENDC 470 | + str(self.kb_r), 471 | bcolors.BLUE + "Kb W: " + bcolors.ENDC 472 | + str(self.kb_w), 473 | bcolors.BLUE + "NUM R: " + bcolors.ENDC 474 | + str(self.num_r), 475 | bcolors.BLUE + "NUM W: " + bcolors.ENDC 476 | + str(self.num_w), 477 | bcolors.BLUE + "AVG LAT (ms): " + bcolors.ENDC 478 | + str(round(self.disk_avg_lat,3)) 479 | ) 480 | 481 | if self.http_transaction_count > 0: 482 | fmt = '{:<5} {:<32} {:<34} {:<34} {:<34}' 483 | output_line = output_line + "\n" + fmt.format( 484 | bcolors.BLUE + "--->" + bcolors.ENDC, 485 | bcolors.BLUE + "HTTP_T_COUNT: " + bcolors.ENDC 486 | + str(self.http_transaction_count), 487 | bcolors.BLUE + "HTTP_BYTE_SENT: " + bcolors.ENDC 488 | + str(self.http_byte_tx), 489 | bcolors.BLUE + "HTTP_BYTE_RECV: " + bcolors.ENDC 490 | + str(self.http_byte_rx), 491 | bcolors.BLUE + "HTTP_AVG_LATENCY (ms): " + bcolors.ENDC 492 | + '{:.3f}'.format(self.http_avg_latency) 493 | ) 494 | fmt = '{:<5} {:<30} {:<30} {:<30} {:<30} {:<30} {:<30} {:<30}' 495 | output_line = output_line + "\n" + fmt.format( 496 | bcolors.BLUE + "--->" + bcolors.ENDC, 497 | bcolors.BLUE + "50p: " + bcolors.ENDC + '{:.5f}'.format(self.http_percentiles[0]), 498 | bcolors.BLUE + "75p: " + bcolors.ENDC + '{:.5f}'.format(self.http_percentiles[1]), 499 | bcolors.BLUE + "90p: " + bcolors.ENDC + '{:.5f}'.format(self.http_percentiles[2]), 500 | bcolors.BLUE + "99p: " + bcolors.ENDC + '{:.5f}'.format(self.http_percentiles[3]), 501 | bcolors.BLUE + "99.9p: " + bcolors.ENDC + '{:.5f}'.format(self.http_percentiles[4]), 502 | bcolors.BLUE + "99.99p: " + bcolors.ENDC + '{:.5f}'.format(self.http_percentiles[5]), 503 | bcolors.BLUE + "99.999p: " + bcolors.ENDC + '{:.5f}'.format(self.http_percentiles[6]), 504 | ) 505 | 506 | if self.tcp_transaction_count > 0: 507 | fmt = '{:<5} {:<32} {:<34} {:<34} {:<34}' 508 | output_line = output_line + "\n" + fmt.format( 509 | bcolors.BLUE + "--->" + bcolors.ENDC, 510 | bcolors.BLUE + "TCP_T_COUNT: " + bcolors.ENDC 511 | + str(self.tcp_transaction_count), 512 | bcolors.BLUE + "TCP_BYTE_SENT: " + bcolors.ENDC 513 | + str(self.tcp_byte_tx), 514 | bcolors.BLUE + "TCP_BYTE_RECV: " + bcolors.ENDC 515 | + str(self.tcp_byte_rx), 516 | bcolors.BLUE + "TCP_AVG_LATENCY (ms): " + bcolors.ENDC 517 | + '{:.3f}'.format(self.tcp_avg_latency) 518 | ) 519 | fmt = '{:<5} {:<30} {:<30} {:<30} {:<30} {:<30} {:<30} {:<30}' 520 | output_line = output_line + "\n" + fmt.format( 521 | bcolors.BLUE + "--->" + bcolors.ENDC, 522 | bcolors.BLUE + "50p: " + bcolors.ENDC + '{:.5f}'.format(self.tcp_percentiles[0]), 523 | bcolors.BLUE + "75p: " + bcolors.ENDC + '{:.5f}'.format(self.tcp_percentiles[1]), 524 | bcolors.BLUE + "90p: " + bcolors.ENDC + '{:.5f}'.format(self.tcp_percentiles[2]), 525 | bcolors.BLUE + "99p: " + bcolors.ENDC + '{:.5f}'.format(self.tcp_percentiles[3]), 526 | bcolors.BLUE + "99.9p: " + bcolors.ENDC + '{:.5f}'.format(self.tcp_percentiles[4]), 527 | bcolors.BLUE + "99.99p: " + bcolors.ENDC + '{:.5f}'.format(self.tcp_percentiles[5]), 528 | bcolors.BLUE + "99.999p: " + bcolors.ENDC + '{:.5f}'.format(self.tcp_percentiles[6]), 529 | ) 530 | return output_line 531 | -------------------------------------------------------------------------------- /userspace/curse.py: -------------------------------------------------------------------------------- 1 | """ 2 | DEEP-mon 3 | Copyright (C) 2020 Brondolin Rolando 4 | 5 | This file is part of DEEP-mon 6 | 7 | DEEP-mon is free software: you can redistribute it and/or modify 8 | it under the terms of the GNU General Public License as published by 9 | the Free Software Foundation, either version 3 of the License, or 10 | (at your option) any later version. 11 | 12 | DEEP-mon is distributed in the hope that it will be useful, 13 | but WITHOUT ANY WARRANTY; without even the implied warranty of 14 | MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 15 | GNU General Public License for more details. 16 | 17 | You should have received a copy of the GNU General Public License 18 | along with this program. If not, see . 19 | """ 20 | 21 | import curses 22 | import time 23 | import locale 24 | 25 | class Curse: 26 | def __init__(self, monitor, power_measure, net_monitor, memory_measure, disk_measure, file_measure): 27 | locale.setlocale(locale.LC_ALL, '') 28 | self.monitor = monitor 29 | self.pages = [] 30 | self.displayed_metric = 'default' 31 | self.highlighted_line_index = 0 32 | self.start_display_index = 0 33 | self.end_display_index = 0 34 | self.initialize_metrics(power_measure, net_monitor, memory_measure, disk_measure, file_measure) 35 | 36 | def start(self): 37 | curses.wrapper(self.main) 38 | 39 | def initialize_metrics(self, power, net, memory, disk, files): 40 | self.pages.append('default') 41 | if power: 42 | self.pages.append("power") 43 | if memory: 44 | self.pages.append("memory") 45 | if disk: 46 | self.pages.append("disk") 47 | if files: 48 | self.pages.append("file") 49 | if net: 50 | self.pages.append("tcp") 51 | self.pages.append("tcp percentiles") 52 | self.pages.append("http") 53 | self.pages.append("http percentiles") 54 | 55 | def set_sample(self, sample): 56 | self.sample = sample 57 | 58 | def title_line(self, cx): 59 | title_win = curses.newwin(1,cx,0,0) 60 | 61 | title_str = "DEEP-mon" 62 | title_win.bkgd(" ", curses.color_pair(9)) 63 | title_win.addstr(0,int(cx/2-len(title_str)/2), title_str, curses.color_pair(9)) 64 | 65 | #title_win.addstr(0,0, str(self.start_display_index)+" "+str(self.end_display_index)+" "+str(self.highlighted_line_index), curses.color_pair(9)) 66 | title_win.noutrefresh() 67 | 68 | def last_line(self, cx,cy): 69 | locale.setlocale(locale.LC_ALL, '') 70 | 71 | last_line_win = curses.newwin(1,cx,cy-1,0) 72 | last_line_str = ("Press 'q' to exit, "+chr(8592)+" or "+chr(8594)+" to change metrics page, "+chr(8593)+" or "+chr(8595)+" to change line.").encode("UTF-8") 73 | last_line_win.bkgd(" ", curses.color_pair(4)) 74 | last_line_win.addstr(0,0, last_line_str, curses.color_pair(4)) 75 | last_line_win.addstr(0, cx-2, chr(9731).encode('UTF-8'), curses.color_pair(4)) 76 | last_line_win.noutrefresh() 77 | 78 | def persistent_info(self, cx, cy, log_dict): 79 | first_column_label_length = 21 80 | second_column_label_length = 29 81 | new_win = curses.newwin(3,cx,1,0) 82 | 83 | new_win.addstr(0,0, "SAMPLE TIMESLICE:", curses.color_pair(6)) 84 | new_win.addstr(1,0, "SCHED SWITCH COUNT:", curses.color_pair(6)) 85 | new_win.addstr(2,0, "PROC TIME:", curses.color_pair(6)) 86 | 87 | new_win.addstr(0, first_column_label_length, "%-9s" %(log_dict["TIMESLICE"]+"s")) 88 | new_win.addstr(1, first_column_label_length, "%-9s" %(log_dict["SCHED SWITCH COUNT"])) 89 | new_win.addstr(2, first_column_label_length, "%-9s" %(log_dict["PROC TIME"])) 90 | 91 | new_win.addstr(0,int(cx/2), "TOTAL PACKAGE ACTIVE POWER:", curses.color_pair(2)) 92 | new_win.addstr(1,int(cx/2), "TOTAL CORE ACTIVE POWER:", curses.color_pair(2)) 93 | new_win.addstr(2,int(cx/2), "TOTAL DRAM ACTIVE POWER:", curses.color_pair(2)) 94 | 95 | new_win.addstr(0, int(cx/2+second_column_label_length), "%-9s" %(log_dict["TOTAL PACKAGE ACTIVE POWER"])) 96 | new_win.addstr(1, int(cx/2+second_column_label_length), "%-9s" %(log_dict["TOTAL CORE ACTIVE POWER"])) 97 | new_win.addstr(2, int(cx/2+second_column_label_length), "%-9s" %(log_dict["TOTAL DRAM ACTIVE POWER"])) 98 | 99 | new_win.noutrefresh() 100 | 101 | def label_line(self, cx): 102 | label_win = curses.newwin(2,cx,4,0) 103 | label_win.bkgd(" ", curses.color_pair(7) | curses.A_REVERSE) 104 | 105 | label_win.addstr(0,0, "Displaying %s metrics" %(self.displayed_metric)) 106 | label_win.addstr(0,cx-10, "Page %d/%d" %(self.pages.index(self.displayed_metric)+1, len(self.pages))) 107 | 108 | if (self.displayed_metric == 'default'): 109 | label_win.addstr(1,0, "%12s %40s %12s %12s" % ( 110 | "CONTAINER_ID", "CONTAINER_NAME", "EXEC TIME(s)", "CPU USAGE" 111 | )) 112 | elif (self.displayed_metric == 'power'): 113 | label_win.addstr(1,0, "%12s %40s %11s %11s %10s %10s %9s %10s" % ( 114 | "CONTAINER_ID", "CONTAINER_NAME", "CYCLES", "W_CYCLES", "INSTR_RET", "CACHE_MISS", "CACHE_REF", "TOT_POWER" 115 | )) 116 | elif (self.displayed_metric == 'memory'): 117 | label_win.addstr(1,0, "%12s %40s %11s %11s %11s" % ( 118 | "CONTAINER_ID", "CONTAINER_NAME", "RSS (Kb)", "PSS (Kb)", "USS (Kb)" 119 | )) 120 | elif (self.displayed_metric == 'disk'): 121 | label_win.addstr(1,0, "%12s %40s %11s %11s %11s %11s %11s" % ( 122 | "CONTAINER_ID", "CONTAINER_NAME", "Kb_R", "Kb_W", "NUM_R", "NUM_W", "AVG_LAT(ms)" 123 | )) 124 | elif (self.displayed_metric == 'tcp'): 125 | label_win.addstr(1,0, "%12s %40s %13s %14s %14s %13s" % ( 126 | "CONTAINER_ID", "CONTAINER_NAME", "TCP_T_COUNT", "TCP_BYTE_SENT", "TCP_BYTE_RECV", "AVG_LAT(ms)" 127 | )) 128 | elif (self.displayed_metric == 'http'): 129 | label_win.addstr(1,0, "%12s %40s %13s %14s %14s %13s" % ( 130 | "CONTAINER_ID", "CONTAINER_NAME", "HTTP_T_COUNT", "HTTP_BYTE_SENT", "HTTP_BYTE_RECV", "AVG_LAT(ms)" 131 | )) 132 | elif (self.displayed_metric == 'tcp percentiles'): 133 | label_win.addstr(1,0, "%12s %40s %8s %8s %8s %8s %8s %8s %8s" % ( 134 | "CONTAINER_ID", "CONTAINER_NAME", "50p", "75p", "90p", "99p", "99.9p", "99.99p", "99.999p" 135 | )) 136 | elif (self.displayed_metric == 'http percentiles'): 137 | label_win.addstr(1,0, "%12s %40s %8s %8s %8s %8s %8s %8s %8s" % ( 138 | "CONTAINER_ID", "CONTAINER_NAME", "50p", "75p", "90p", "99p", "99.9p", "99.99p", "99.999p" 139 | )) 140 | elif (self.displayed_metric == 'file'): 141 | label_win.addstr(1,0, "%11s %11s %11s %11s %s" % ( 142 | "Kb_R", "Kb_W", "NUM_R", "NUM_W", "FILE NAME" 143 | )) 144 | 145 | 146 | label_win.noutrefresh() 147 | 148 | def metrics_window(self, cx, cy, container_list, file_dict): 149 | metrics_win = curses.newwin(cy-6,cx,6,0) 150 | 151 | counter = 0 152 | if self.displayed_metric != "file": 153 | for key, value in sorted(container_list.items()): 154 | if (counter == self.highlighted_line_index): 155 | color = curses.color_pair(4) 156 | else: 157 | color = curses.color_pair(8) 158 | if (self.start_display_index <= counter < self.end_display_index): 159 | metrics_win.addstr(counter-self.start_display_index, 0, "%12s " %key, color) 160 | 161 | if value.get_container_name() != None: 162 | metrics_win.addstr(counter-self.start_display_index, 13, "%41s" % value.get_container_name().ljust(40)[0:40], color) 163 | else: 164 | metrics_win.addstr(counter-self.start_display_index, 13, "%41s" %"", color) 165 | 166 | if self.displayed_metric == 'default': 167 | metrics_win.addstr(counter-self.start_display_index, 54, str.ljust("%12s %12s " % ( 168 | '{:.5f}'.format(value.get_time_ns() / 1000000000.0), 169 | '{:.2f}'.format(value.get_cpu_usage()) 170 | ),cx-54), color) 171 | 172 | elif self.displayed_metric == 'power': 173 | metrics_win.addstr(counter-self.start_display_index, 54, str.ljust("%11d %11d %10d %10s %9s %8smW" % ( 174 | value.get_cycles(), value.get_weighted_cycles(), 175 | value.get_instruction_retired(), 176 | value.get_cache_misses(), value.get_cache_refs(), 177 | '{:.2f}'.format(value.get_power()) 178 | ),cx-54), color) 179 | 180 | elif self.displayed_metric == 'memory': 181 | metrics_win.addstr(counter-self.start_display_index, 54, str.ljust("%11s %11s %11s" % ( 182 | str(value.get_mem_RSS()), str(value.get_mem_PSS()), str(value.get_mem_USS()) 183 | ),cx-54), color) 184 | 185 | elif self.displayed_metric == 'disk': 186 | metrics_win.addstr(counter-self.start_display_index, 54, str.ljust("%11s %11s %11s %11s %11s" % ( 187 | str(value.get_kb_r()), str(value.get_kb_w()), 188 | str(value.get_num_r()), str(value.get_num_w()), 189 | '{:.3f}'.format(value.get_disk_avg_lat()) 190 | ),cx-54), color) 191 | 192 | elif self.displayed_metric == 'http': 193 | metrics_win.addstr(counter-self.start_display_index, 54, str.ljust("%13s %14s %14s %13s" % ( 194 | str(value.get_http_transaction_count()), str(value.get_http_byte_tx()), 195 | str(value.get_http_byte_rx()), '{:.2f}'.format(value.get_http_avg_latency()), 196 | ),cx-54), color) 197 | 198 | elif self.displayed_metric == 'tcp': 199 | metrics_win.addstr(counter-self.start_display_index, 54, str.ljust("%13s %14s %14s %13s" % ( 200 | str(value.get_tcp_transaction_count()), str(value.get_tcp_byte_tx()), 201 | str(value.get_tcp_byte_rx()), '{:.2f}'.format(value.get_tcp_avg_latency()), 202 | ),cx-54), color) 203 | 204 | elif self.displayed_metric == 'http percentiles': 205 | pct_val = value.get_http_percentiles()[1] 206 | if len(pct_val) == 7: 207 | metrics_win.addstr(counter-self.start_display_index, 54, str.ljust("%8s %8s %8s %8s %8s %8s %8s" % ( 208 | '{:.1f}'.format(pct_val[0]), '{:.1f}'.format(pct_val[1]), 209 | '{:.1f}'.format(pct_val[2]), '{:.1f}'.format(pct_val[3]), 210 | '{:.1f}'.format(pct_val[4]), '{:.1f}'.format(pct_val[5]), 211 | '{:.1f}'.format(pct_val[6]) 212 | ),cx-54), color) 213 | else: 214 | metrics_win.addstr(counter-self.start_display_index, 54, str.ljust("%8s %8s %8s %8s %8s %8s %8s" % ( 215 | '0', '0', '0', '0', '0', '0', '0' 216 | ),cx-54), color) 217 | 218 | elif self.displayed_metric == 'tcp percentiles': 219 | pct_val = value.get_tcp_percentiles()[1] 220 | if len(pct_val) == 7: 221 | metrics_win.addstr(counter-self.start_display_index, 54, str.ljust("%8s %8s %8s %8s %8s %8s %8s" % ( 222 | '{:.1f}'.format(pct_val[0]), '{:.1f}'.format(pct_val[1]), 223 | '{:.1f}'.format(pct_val[2]), '{:.1f}'.format(pct_val[3]), 224 | '{:.1f}'.format(pct_val[4]), '{:.1f}'.format(pct_val[5]), 225 | '{:.1f}'.format(pct_val[6]) 226 | ),cx-54), color) 227 | else: 228 | metrics_win.addstr(counter-self.start_display_index, 54, str.ljust("%8s %8s %8s %8s %8s %8s %8s" % ( 229 | '0', '0', '0', '0', '0', '0', '0' 230 | ),cx-54), color) 231 | counter += 1 232 | else: 233 | for key, value in reversed(sorted(file_dict.items(), key=lambda counts: counts[1].get_kb_r()+counts[1].get_kb_w())): 234 | if (counter == self.highlighted_line_index): 235 | color = curses.color_pair(4) 236 | else: 237 | color = curses.color_pair(8) 238 | if (self.start_display_index <= counter < self.end_display_index): 239 | str_key = key 240 | if (len(key)>cx-50): 241 | str_key= ".."+key[-(cx-50):] 242 | metrics_win.addstr(counter-self.start_display_index, 0, str.ljust("%11s %11s %11s %11s %s" % ( 243 | str(file_dict[key].get_kb_r()), str(file_dict[key].get_kb_w()), 244 | str(file_dict[key].get_num_r()), str(file_dict[key].get_num_w()), 245 | str_key),cx), color) 246 | counter += 1 247 | 248 | metrics_win.noutrefresh() 249 | 250 | def _reset_window_indices(self, stdscr): 251 | yx = stdscr.getmaxyx() 252 | cy = yx[0] 253 | self.highlighted_line_index = 0 254 | self.start_display_index = 0 255 | self.end_display_index = cy-7 256 | 257 | def main(self, stdscr): 258 | if self.monitor.get_window_mode() == 'dynamic': 259 | time_to_sleep = self.monitor.get_sample_controller().get_sleep_time() 260 | else: 261 | time_to_sleep = 1 262 | 263 | stdscr.nodelay(True) 264 | stdscr.timeout(100) 265 | curses.curs_set(False) 266 | 267 | bg_color = curses.COLOR_BLACK 268 | if curses.has_colors(): 269 | curses.init_pair(1, curses.COLOR_RED, bg_color) 270 | curses.init_pair(2, curses.COLOR_GREEN, bg_color) 271 | curses.init_pair(3, curses.COLOR_BLUE, bg_color) 272 | curses.init_pair(4, bg_color, curses.COLOR_WHITE) 273 | curses.init_pair(5, curses.COLOR_MAGENTA, bg_color) 274 | curses.init_pair(6, curses.COLOR_YELLOW, bg_color) 275 | curses.init_pair(7, curses.COLOR_CYAN, bg_color) 276 | curses.init_pair(8, curses.COLOR_WHITE, bg_color) 277 | curses.init_pair(9, curses.COLOR_WHITE, curses.COLOR_RED) 278 | 279 | previous_time = time.time() 280 | sample_array = self.monitor.get_sample() 281 | sample = sample_array[0] 282 | container_list = sample_array[1] 283 | 284 | yx = stdscr.getmaxyx() 285 | cx = yx[1] 286 | cy = yx[0] 287 | 288 | self.start_display_index = 0 289 | self.end_display_index = cy-7 290 | 291 | while True: 292 | start_time = time.time() 293 | curses.napms(5) 294 | 295 | if (start_time - previous_time > time_to_sleep): 296 | sample_array = self.monitor.get_sample() 297 | sample = sample_array[0] 298 | container_list = sample_array[1] 299 | file_dict = sample_array[4] 300 | previous_time = time.time() 301 | if self.monitor.get_window_mode() == 'dynamic': 302 | time_to_sleep = self.monitor.get_sample_controller().get_sleep_time() \ 303 | - (time.time() - start_time) 304 | else: 305 | time_to_sleep = 1 - (time.time() - start_time) 306 | 307 | yx = stdscr.getmaxyx() 308 | cx = yx[1] 309 | cy = yx[0] 310 | 311 | if (cx >= 120 and cy >= 7): 312 | self.title_line(cx) 313 | self.persistent_info(cx,cy, sample.get_log_dict()) 314 | self.metrics_window(cx,cy, container_list, file_dict) 315 | self.label_line(cx) 316 | self.last_line(cx,cy) 317 | else: 318 | stdscr.clear() 319 | stdscr.addstr(5,1, "Window too small, try to resize :(") 320 | stdscr.refresh() 321 | 322 | ch = stdscr.getch() 323 | 324 | if ch == ord('q'): 325 | return 0 326 | elif ch == curses.KEY_LEFT: 327 | self.displayed_metric = self.pages[(self.pages.index(self.displayed_metric)-1) % len(self.pages)] 328 | self._reset_window_indices(stdscr) 329 | elif ch == curses.KEY_RIGHT: 330 | self.displayed_metric = self.pages[(self.pages.index(self.displayed_metric)+1) % len(self.pages)] 331 | self._reset_window_indices(stdscr) 332 | 333 | elif ch == curses.KEY_UP: 334 | if self.highlighted_line_index >= self.start_display_index and self.highlighted_line_index > 0: 335 | self.highlighted_line_index -= 1 336 | if self.highlighted_line_index == self.start_display_index-1 and self.start_display_index > 0: 337 | self.start_display_index -= 1 338 | self.end_display_index -= 1 339 | elif ch == curses.KEY_DOWN: 340 | if (self.displayed_metric != 'file' and self.highlighted_line_index < min(self.end_display_index, len(container_list)-1)): 341 | self.highlighted_line_index += 1 342 | elif (self.displayed_metric == 'file' and self.highlighted_line_index < min(self.end_display_index, len(file_dict)-1)): 343 | self.highlighted_line_index += 1 344 | if (self.displayed_metric != 'file' and self.highlighted_line_index >= (cy-7) and self.end_display_index < len(container_list)): 345 | self.start_display_index += 1 346 | self.end_display_index += 1 347 | elif (self.displayed_metric == 'file' and self.highlighted_line_index >= (cy-7) and self.end_display_index < len(file_dict)): 348 | self.start_display_index += 1 349 | self.end_display_index += 1 350 | 351 | elif ch == curses.KEY_RESIZE: 352 | self._reset_window_indices(stdscr) 353 | 354 | 355 | curses.doupdate() 356 | -------------------------------------------------------------------------------- /userspace/default_config.yaml: -------------------------------------------------------------------------------- 1 | window_mode: "fixed" 2 | output_format: "console" 3 | debug_mode: False 4 | send_thread_data: False 5 | net_monitor: True 6 | nat_trace: False 7 | print_net_details: False 8 | dynamic_tcp_client_port_masking: True 9 | power_measure: True 10 | memory_measure: True 11 | disk_measure: True 12 | file_measure: True 13 | -------------------------------------------------------------------------------- /userspace/disk_collector.py: -------------------------------------------------------------------------------- 1 | """ 2 | DEEP-mon 3 | Copyright (C) 2020 Brondolin Rolando 4 | 5 | This file is part of DEEP-mon 6 | 7 | DEEP-mon is free software: you can redistribute it and/or modify 8 | it under the terms of the GNU General Public License as published by 9 | the Free Software Foundation, either version 3 of the License, or 10 | (at your option) any later version. 11 | 12 | DEEP-mon is distributed in the hope that it will be useful, 13 | but WITHOUT ANY WARRANTY; without even the implied warranty of 14 | MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 15 | GNU General Public License for more details. 16 | 17 | You should have received a copy of the GNU General Public License 18 | along with this program. If not, see . 19 | """ 20 | 21 | from bcc import BPF 22 | import os 23 | import json 24 | 25 | class DiskCollector: 26 | def __init__(self, monitor_disk, monitor_file): 27 | self.monitor_file = monitor_file 28 | self.monitor_disk = monitor_disk 29 | self.disk_sample = None 30 | self.disk_monitor = None 31 | self.proc_path = "/host/proc" 32 | self.proc_files = [f for f in os.listdir(self.proc_path) if os.path.isfile(os.path.join(self.proc_path, f))] 33 | self.number_files_to_keep = 10 34 | 35 | def start_capture(self): 36 | bpf_code_path = os.path.dirname(os.path.abspath(__file__)) \ 37 | + "/../bpf/vfs_monitor.c" 38 | #DNAME_INLINE_LEN = 32 # linux/dcache.h 39 | self.disk_monitor = BPF(src_file=bpf_code_path, cflags=["-DNAME_INLINE_LEN=%d" % 32]) 40 | self.disk_monitor.attach_kprobe(event="vfs_read", fn_name="trace_rw_entry") 41 | self.disk_monitor.attach_kretprobe(event="vfs_read", fn_name="trace_read_return") 42 | 43 | self.disk_monitor.attach_kprobe(event="vfs_write", fn_name="trace_rw_entry") 44 | self.disk_monitor.attach_kretprobe(event="vfs_write", fn_name="trace_write_return") 45 | 46 | def _include_file_path(self, file_name, file_parent, file_parent2): 47 | file_name = file_name.decode("utf-8") 48 | file_parent = file_parent.decode("utf-8") 49 | file_parent2 = file_parent2.decode("utf-8") 50 | 51 | if (file_parent == "/"): 52 | if (file_name in self.proc_files or file_name.isdigit()): 53 | return False 54 | return "/"+file_name 55 | if (file_parent2 == "/"): 56 | if (file_parent in self.proc_files or file_parent.isdigit()): 57 | return False 58 | return "/"+file_parent+"/"+file_name 59 | if (file_parent2 in self.proc_files or file_parent2.isdigit()): 60 | return False 61 | return file_parent2+"/"+file_parent+"/"+file_name 62 | 63 | def get_sample(self): 64 | disk_dict = {} 65 | if (self.monitor_disk): 66 | disk_counts = self.disk_monitor["counts_by_pid"] 67 | for k,v in disk_counts.items(): 68 | key = int(v.pid) 69 | disk_dict[key] = {} 70 | disk_dict[key]["kb_r"] = int(v.bytes_r/1000) 71 | disk_dict[key]["kb_w"] = int(v.bytes_w/1000) 72 | disk_dict[key]["num_r"] = int(v.num_r) 73 | disk_dict[key]["num_w"] = int(v.num_w) 74 | disk_dict[key]["avg_lat"] = float(v.sum_ts_deltas) / 1000 / (v.num_r+v.num_w) 75 | disk_dict[key]["container_ID"] = "---others---" 76 | if (os.path.exists(os.path.join(self.proc_path,str(v.pid),"cgroup"))): 77 | try: 78 | with open(os.path.join(self.proc_path, str(v.pid), 'cgroup'), 'r') as f: 79 | for line in f: 80 | line_array = line.split("/") 81 | if len(line_array) > 1 and \ 82 | len(line_array[len(line_array) -1]) == 65: 83 | disk_dict[key]["container_ID"] = line_array[len(line_array) -1][:-1] 84 | break 85 | except IOError: 86 | continue 87 | # systemd Docker 88 | try: 89 | with open(os.path.join(self.proc_path, str(v.pid), 'cgroup'), 'r') as f: 90 | for line in f: 91 | line_array = line.split("/") 92 | if len(line_array) > 1 \ 93 | and "docker-" in line_array[len(line_array) -1] \ 94 | and ".scope" in line_array[len(line_array) -1]: 95 | 96 | new_id = line_array[len(line_array) -1].replace("docker-", "") 97 | new_id = new_id.replace(".scope", "") 98 | if len(new_id) == 65: 99 | disk_dict[key]["container_ID"] = new_id 100 | break 101 | except IOError: 102 | continue 103 | 104 | disk_dict = self._aggregate_metrics_by_container(disk_dict) 105 | disk_counts.clear() 106 | 107 | file_dict = {} 108 | if (self.monitor_file): 109 | counter = 0 110 | file_counts = self.disk_monitor.get_table("counts_by_file") 111 | for k, v in reversed(sorted(file_counts.items(), key=lambda counts_f: (counts_f[1].bytes_r+counts_f[1].bytes_w))): 112 | if (self._include_file_path(k.name, k.parent1, k.parent2) != False) and counter < self.number_files_to_keep: 113 | key = self._include_file_path(k.name, k.parent1, k.parent2) 114 | file_dict[key] = FileInfo() 115 | file_dict[key].set_file_path(key) 116 | file_dict[key].set_kb_r(int(v.bytes_r/1000)) 117 | file_dict[key].set_kb_w(int(v.bytes_w/1000)) 118 | file_dict[key].set_num_r(int(v.num_r)) 119 | file_dict[key].set_num_w(int(v.num_w)) 120 | file_dict[key].set_file_id(counter) 121 | counter+=1 122 | 123 | file_counts.clear() 124 | 125 | 126 | aggregate_dict = {} 127 | aggregate_dict['file_sample'] = file_dict 128 | aggregate_dict['disk_sample'] = disk_dict 129 | return aggregate_dict 130 | 131 | def _aggregate_metrics_by_container(self, disk_sample): 132 | container_dict = dict() 133 | for pid in disk_sample: 134 | shortened_ID = disk_sample[pid]["container_ID"][:12] 135 | if shortened_ID not in container_dict: 136 | container_dict[shortened_ID] = {} 137 | container_dict[shortened_ID]["full_ID"] = disk_sample[pid]["container_ID"] 138 | container_dict[shortened_ID]["kb_r"] = 0 139 | container_dict[shortened_ID]["kb_w"] = 0 140 | container_dict[shortened_ID]["num_r"] = 0 141 | container_dict[shortened_ID]["num_w"] = 0 142 | container_dict[shortened_ID]["avg_lat"] = 0 143 | container_dict[shortened_ID]["pids"] = [] 144 | container_dict[shortened_ID]["kb_r"] += disk_sample[pid]["kb_r"] 145 | container_dict[shortened_ID]["kb_w"] += disk_sample[pid]["kb_w"] 146 | container_dict[shortened_ID]["num_r"] += disk_sample[pid]["num_r"] 147 | container_dict[shortened_ID]["num_w"] += disk_sample[pid]["num_w"] 148 | container_dict[shortened_ID]["num_w"] += disk_sample[pid]["num_w"] 149 | container_dict[shortened_ID]["avg_lat"] += disk_sample[pid]["avg_lat"] 150 | container_dict[shortened_ID]["pids"].append(pid) 151 | for k,v in container_dict.items(): 152 | container_dict[k]["avg_lat"] = container_dict[k]["avg_lat"] / len(container_dict[k]["pids"]) 153 | 154 | return container_dict 155 | 156 | 157 | 158 | class FileInfo: 159 | def __init__(self): 160 | self.file_path = "" 161 | self.kb_r = 0 162 | self.kb_w = 0 163 | self.num_r = 0 164 | self.num_w = 0 165 | self.file_id = 0 166 | 167 | def get_file_path(self): 168 | return self.file_path 169 | 170 | def get_kb_r(self): 171 | return self.kb_r 172 | 173 | def get_kb_w(self): 174 | return self.kb_w 175 | 176 | def get_num_r(self): 177 | return self.num_r 178 | 179 | def get_num_w(self): 180 | return self.num_w 181 | 182 | def get_file_id(self): 183 | return self.file_id 184 | 185 | def set_file_id(self, file_id): 186 | self.file_id = file_id 187 | 188 | def set_file_path(self, file_path): 189 | self.file_path = file_path 190 | 191 | def set_kb_r(self, kbr): 192 | self.kb_r = kbr 193 | 194 | def set_kb_w(self, kbw): 195 | self.kb_w = kbw 196 | 197 | def set_num_r(self, numr): 198 | self.num_r = numr 199 | 200 | def set_num_w(self, numw): 201 | self.num_w = numw 202 | -------------------------------------------------------------------------------- /userspace/mem_collector.py: -------------------------------------------------------------------------------- 1 | """ 2 | DEEP-mon 3 | Copyright (C) 2020 Brondolin Rolando 4 | 5 | This file is part of DEEP-mon 6 | 7 | DEEP-mon is free software: you can redistribute it and/or modify 8 | it under the terms of the GNU General Public License as published by 9 | the Free Software Foundation, either version 3 of the License, or 10 | (at your option) any later version. 11 | 12 | DEEP-mon is distributed in the hope that it will be useful, 13 | but WITHOUT ANY WARRANTY; without even the implied warranty of 14 | MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 15 | GNU General Public License for more details. 16 | 17 | You should have received a copy of the GNU General Public License 18 | along with this program. If not, see . 19 | """ 20 | 21 | import os 22 | 23 | class MemCollector: 24 | def __init__ (self): 25 | self.mem_dictionary = dict() 26 | self.proc_path = "/host/proc" 27 | 28 | def get_mem_dictionary(self): 29 | self.mem_dictionary = self._aggregate_mem_metrics(self._get_sample()) 30 | return self.mem_dictionary 31 | 32 | def _get_pid_list(self): 33 | pid_list = os.listdir(self.proc_path) 34 | return [int(x) for x in pid_list if x.isdigit()] 35 | 36 | def _aggregate_mem_metrics(self, mem_sample): 37 | container_dict = dict() 38 | for pid in mem_sample: 39 | shortened_ID = mem_sample[pid]["container_ID"][:12] 40 | if shortened_ID not in container_dict: 41 | container_dict[shortened_ID] = {} 42 | container_dict[shortened_ID]["full_ID"] = mem_sample[pid]["container_ID"] 43 | container_dict[shortened_ID]["RSS"] = 0 44 | container_dict[shortened_ID]["PSS"] = 0 45 | container_dict[shortened_ID]["USS"] = 0 46 | container_dict[shortened_ID]["pids"] = [] 47 | container_dict[shortened_ID]["RSS"] += mem_sample[pid]["RSS"] 48 | container_dict[shortened_ID]["PSS"] += mem_sample[pid]["PSS"] 49 | container_dict[shortened_ID]["USS"] += mem_sample[pid]["USS"] 50 | container_dict[shortened_ID]["pids"].append(pid) 51 | return container_dict 52 | 53 | def _get_sample(self): 54 | pid_dict = dict() 55 | for pid in self._get_pid_list(): 56 | pid_dict[pid] = {} 57 | pid_dict[pid]["RSS"] = 0 58 | pid_dict[pid]["USS"] = 0 59 | pid_dict[pid]["PSS"] = 0 60 | pid_dict[pid]["container_ID"] = "---others---" 61 | #USS and PSS from smaps_rollup 62 | if (os.path.exists(os.path.join(self.proc_path,str(pid),"smaps_rollup"))): 63 | try: 64 | with open(os.path.join(self.proc_path,str(pid),"smaps_rollup"),"r") as f: 65 | for line in f: 66 | s = line.replace(" ","").replace("\n","").split(':') 67 | if (s[0] == "Rss"): 68 | pid_dict[pid]["RSS"] = int(s[1][:-2]) 69 | elif (s[0] == "Pss"): 70 | pid_dict[pid]["PSS"] = int(s[1][:-2]) 71 | elif (s[0] == "Private_Clean" or s[0] == "Private_Dirty" or s[0] == "Private_Hugetlb"): 72 | pid_dict[pid]["USS"] += int(s[1][:-2]) 73 | except IOError: 74 | continue 75 | #USS and PSS from smaps when smaps_rollup isn't in proc 76 | else: 77 | try: 78 | with open(os.path.join(self.proc_path,str(pid),"smaps"),"r") as f: 79 | for line in f: 80 | s = line.replace(" ","").replace("\n","").split(':') 81 | if (s[0] == "Pss"): 82 | pid_dict[pid]["RSS"] += int(s[1][:-2]) 83 | elif (s[0] == "Pss"): 84 | pid_dict[pid]["PSS"] += int(s[1][:-2]) 85 | elif (s[0] == "Private_Clean" or s[0] == "Private_Dirty" or s[0] == "Private_Hugetlb"): 86 | pid_dict[pid]["USS"] += int(s[1][:-2]) 87 | except IOError: 88 | continue 89 | #assign container ID from proc 90 | if (os.path.exists(os.path.join(self.proc_path,str(pid),"cgroup"))): 91 | try: 92 | with open(os.path.join(self.proc_path, str(pid), 'cgroup'), 'r') as f: 93 | for line in f: 94 | line_array = line.split("/") 95 | if len(line_array) > 1 and \ 96 | len(line_array[len(line_array) -1]) == 65: 97 | pid_dict[pid]["container_ID"] = line_array[len(line_array) -1][:-1] 98 | except IOError: 99 | continue 100 | # systemd Docker 101 | try: 102 | with open(os.path.join(self.proc_path, str(pid), 'cgroup'), 'r') as f: 103 | for line in f: 104 | line_array = line.split("/") 105 | if len(line_array) > 1 \ 106 | and "docker-" in line_array[len(line_array) -1] \ 107 | and ".scope" in line_array[len(line_array) -1]: 108 | 109 | new_id = line_array[len(line_array) -1].replace("docker-", "") 110 | new_id = new_id.replace(".scope", "") 111 | if len(new_id) == 65: 112 | pid_dict[pid]["container_ID"] = new_id 113 | except IOError: 114 | continue 115 | 116 | return pid_dict 117 | -------------------------------------------------------------------------------- /userspace/monitor_main.py: -------------------------------------------------------------------------------- 1 | """ 2 | DEEP-mon 3 | Copyright (C) 2020 Brondolin Rolando 4 | 5 | This file is part of DEEP-mon 6 | 7 | DEEP-mon is free software: you can redistribute it and/or modify 8 | it under the terms of the GNU General Public License as published by 9 | the Free Software Foundation, either version 3 of the License, or 10 | (at your option) any later version. 11 | 12 | DEEP-mon is distributed in the hope that it will be useful, 13 | but WITHOUT ANY WARRANTY; without even the implied warranty of 14 | MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 15 | GNU General Public License for more details. 16 | 17 | You should have received a copy of the GNU General Public License 18 | along with this program. If not, see . 19 | """ 20 | 21 | from .bpf_collector import BpfCollector 22 | from .proc_topology import ProcTopology 23 | from .sample_controller import SampleController 24 | from .process_table import ProcTable 25 | from .net_collector import NetCollector 26 | from .mem_collector import MemCollector 27 | from .disk_collector import DiskCollector 28 | from .rapl.rapl import RaplMonitor 29 | import os 30 | import socket 31 | import time 32 | import yaml 33 | try: 34 | from yaml import CLoader as Loader 35 | except ImportError: 36 | from yaml import Loader 37 | 38 | class MonitorMain(): 39 | 40 | def __init__(self, output_format, window_mode, debug_mode, net_monitor, nat_trace, print_net_details, dynamic_tcp_client_port_masking, power_measure, memory_measure, disk_measure, file_measure): 41 | self.output_format = output_format 42 | self.window_mode = window_mode 43 | # TODO: Don't hardcode the frequency 44 | self.frequency = 1 45 | 46 | self.topology = ProcTopology() 47 | self.collector = BpfCollector(self.topology, debug_mode, power_measure) 48 | self.sample_controller = SampleController(self.topology.get_hyperthread_count()) 49 | self.process_table = ProcTable() 50 | self.rapl_monitor = RaplMonitor(self.topology) 51 | self.started = False 52 | 53 | self.print_net_details = print_net_details 54 | self.net_monitor = net_monitor 55 | self.dynamic_tcp_client_port_masking = dynamic_tcp_client_port_masking 56 | self.net_collector = None 57 | 58 | self.mem_measure = memory_measure 59 | self.mem_collector = None 60 | 61 | self.disk_measure = disk_measure 62 | self.file_measure = file_measure 63 | self.disk_collector = None 64 | 65 | if self.net_monitor: 66 | self.net_collector = NetCollector(trace_nat = nat_trace, dynamic_tcp_client_port_masking=dynamic_tcp_client_port_masking) 67 | 68 | if self.mem_measure: 69 | self.mem_collector = MemCollector() 70 | 71 | if self.disk_measure or self.file_measure: 72 | self.disk_collector = DiskCollector(disk_measure, file_measure) 73 | 74 | def get_window_mode(self): 75 | return self.window_mode 76 | 77 | def get_sample_controller(self): 78 | return self.sample_controller 79 | 80 | def _start_bpf_program(self, window_mode): 81 | if window_mode == 'dynamic': 82 | self.collector.start_capture(self.sample_controller.get_timeslice()) 83 | if self.net_monitor: 84 | self.net_collector.start_capture() 85 | if (self.disk_measure or self.file_measure): 86 | self.disk_collector.start_capture() 87 | elif window_mode == 'fixed': 88 | self.collector.start_timed_capture(frequency=self.frequency) 89 | if self.net_monitor: 90 | self.net_collector.start_capture() 91 | if (self.disk_measure or self.file_measure): 92 | self.disk_collector.start_capture() 93 | else: 94 | print("Please provide a window mode") 95 | 96 | 97 | def get_sample(self): 98 | if not self.started: 99 | self._start_bpf_program(self.window_mode) 100 | self.started = True 101 | 102 | sample = self.collector.get_new_sample(self.sample_controller, self.rapl_monitor) 103 | # clear metrics for the new sample 104 | self.process_table.reset_metrics_and_evict_stale_processes(sample.get_max_ts()) 105 | # add stuff to cumulative process table 106 | 107 | mem_dict = None 108 | disk_dict = None 109 | file_dict = {} 110 | 111 | if self.mem_collector: 112 | mem_dict = self.mem_collector.get_mem_dictionary() 113 | if self.disk_measure or self.file_measure: 114 | aggregate_disk_sample = self.disk_collector.get_sample() 115 | if self.disk_collector: 116 | disk_dict = aggregate_disk_sample['disk_sample'] 117 | if self.file_measure: 118 | file_dict = aggregate_disk_sample['file_sample'] 119 | 120 | nat_data = [] 121 | if self.net_monitor: 122 | net_sample = self.net_collector.get_sample() 123 | self.process_table.add_process_from_sample(sample, \ 124 | net_dictionary=net_sample.get_pid_dictionary(), \ 125 | nat_dictionary=net_sample.get_nat_dictionary()) 126 | else: 127 | self.process_table.add_process_from_sample(sample) 128 | 129 | # Now, extract containers! 130 | container_list = self.process_table.get_container_dictionary(mem_dict, disk_dict) 131 | 132 | return [sample, container_list, self.process_table.get_proc_table(), nat_data, file_dict] 133 | 134 | 135 | def monitor_loop(self): 136 | if self.window_mode == 'dynamic': 137 | time_to_sleep = self.sample_controller.get_sleep_time() 138 | else: 139 | time_to_sleep = 1 / self.frequency 140 | 141 | while True: 142 | 143 | if time_to_sleep > 0: 144 | time.sleep(time_to_sleep) 145 | start_time = time.time() 146 | 147 | sample_array = self.get_sample() 148 | sample = sample_array[0] 149 | container_list = sample_array[1] 150 | 151 | if self.output_format == "json": 152 | for key, value in container_list.items(): 153 | print(value.to_json()) 154 | print(sample.get_log_json()) 155 | 156 | elif self.output_format == "console": 157 | if self.print_net_details: 158 | nat_data = sample_array[3] 159 | for nat_rule in nat_data: 160 | print(nat_rule) 161 | 162 | for key, value in sorted(container_list.items()): 163 | print(value) 164 | 165 | if self.print_net_details: 166 | for item in value.get_network_transactions(): 167 | print(item) 168 | for item in value.get_nat_rules(): 169 | print(item) 170 | 171 | print('│') 172 | print('└─╼', end='\t') 173 | print(sample.get_log_line()) 174 | print() 175 | print() 176 | 177 | if self.window_mode == 'dynamic': 178 | time_to_sleep = self.sample_controller.get_sleep_time() \ 179 | - (time.time() - start_time) 180 | else: 181 | time_to_sleep = 1 / self.frequency - (time.time() - start_time) 182 | -------------------------------------------------------------------------------- /userspace/net_collector.py: -------------------------------------------------------------------------------- 1 | """ 2 | DEEP-mon 3 | Copyright (C) 2020 Brondolin Rolando 4 | 5 | This file is part of DEEP-mon 6 | 7 | DEEP-mon is free software: you can redistribute it and/or modify 8 | it under the terms of the GNU General Public License as published by 9 | the Free Software Foundation, either version 3 of the License, or 10 | (at your option) any later version. 11 | 12 | DEEP-mon is distributed in the hope that it will be useful, 13 | but WITHOUT ANY WARRANTY; without even the implied warranty of 14 | MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 15 | GNU General Public License for more details. 16 | 17 | You should have received a copy of the GNU General Public License 18 | along with this program. If not, see . 19 | """ 20 | 21 | from bcc import BPF 22 | import ctypes as ct 23 | import numpy as np 24 | from socket import inet_ntop, AF_INET, AF_INET6 25 | from struct import pack 26 | from collections import namedtuple 27 | import os 28 | from ddsketch.ddsketch import DDSketch 29 | 30 | 31 | from enum import Enum 32 | 33 | HTTPSessionKey = namedtuple('HTTPSession', ['saddr', 'lport', 'daddr', 'dport', 'path']) 34 | TCPSessionKey = namedtuple('TCPSession', ['saddr', 'lport', 'daddr', 'dport']) 35 | TCPEndpointKey = namedtuple('TCPEndpoint', ['addr', 'port']) 36 | 37 | def get_ipv4_endpoint_key(k): 38 | return TCPEndpointKey(addr=inet_ntop(AF_INET, pack("I", k.addr)), 39 | port=k.port) 40 | 41 | def get_ipv6_endpoint_key(k): 42 | return TCPEndpointKey(addr=inet_ntop(AF_INET6, k.addr), 43 | port=k.port) 44 | 45 | def get_ipv4_session_key(k): 46 | return TCPSessionKey(saddr=inet_ntop(AF_INET, pack("I", k.saddr)), 47 | lport=k.lport, 48 | daddr=inet_ntop(AF_INET, pack("I", k.daddr)), 49 | dport=k.dport) 50 | 51 | def get_ipv6_session_key(k): 52 | return TCPSessionKey(saddr=inet_ntop(AF_INET6, k.saddr), 53 | lport=k.lport, 54 | daddr=inet_ntop(AF_INET6, k.daddr), 55 | dport=k.dport) 56 | 57 | def get_ipv4_http_session_key(k): 58 | return HTTPSessionKey(saddr=inet_ntop(AF_INET, pack("I", k.saddr)), 59 | lport=k.lport, 60 | daddr=inet_ntop(AF_INET, pack("I", k.daddr)), 61 | dport=k.dport, 62 | path=k.http_payload) 63 | 64 | def get_ipv6_http_session_key(k): 65 | return HTTPSessionKey(saddr=inet_ntop(AF_INET6, k.saddr), 66 | lport=k.lport, 67 | daddr=inet_ntop(AF_INET6, k.daddr), 68 | dport=k.dport, 69 | path=k.http_payload) 70 | 71 | def get_session_key_by_type(k, type): 72 | if type is TransactionType.ipv4_tcp: 73 | return get_ipv4_session_key(k) 74 | elif type is TransactionType.ipv4_http: 75 | return get_ipv4_http_session_key(k) 76 | elif type is TransactionType.ipv6_tcp: 77 | return get_ipv6_session_key(k) 78 | elif type is TransactionType.ipv6_http: 79 | return get_ipv6_http_session_key(k) 80 | return None 81 | 82 | class TransactionType(Enum): 83 | ipv4_tcp = 0 84 | ipv4_http = 1 85 | ipv6_tcp = 2 86 | ipv6_http = 3 87 | 88 | class TransactionRole(Enum): 89 | client = -1 90 | server = 1 91 | 92 | class TransactionData: 93 | 94 | def __init__(self, type, role, saddr, lport, daddr, dport, transaction_count, byte_rx, byte_tx): 95 | self.type = type 96 | self.role = role 97 | self.saddr = saddr 98 | self.lport = lport 99 | self.daddr = daddr 100 | self.dport = dport 101 | self.t_count = transaction_count 102 | self.byte_rx = byte_rx 103 | self.byte_tx = byte_tx 104 | self.avg = 0 105 | self.p50 = 0 106 | self.p75 = 0 107 | self.p90 = 0 108 | self.p99 = 0 109 | self.p99_9 = 0 110 | self.p99_99 = 0 111 | self.p99_999 = 0 112 | self.http_path = "" 113 | self.samples = [] 114 | 115 | def load_latencies(self, latency_sketch, total_time, transaction_count): 116 | self.samples = latency_sketch 117 | self.avg = float(total_time) / float(transaction_count * 1000000) 118 | 119 | self.p50 = latency_sketch.get_quantile_value(0.5) 120 | self.p75 = latency_sketch.get_quantile_value(0.75) 121 | self.p90 = latency_sketch.get_quantile_value(0.9) 122 | self.p99 = latency_sketch.get_quantile_value(0.99) 123 | self.p99_9 = latency_sketch.get_quantile_value(0.999) 124 | self.p99_99 = latency_sketch.get_quantile_value(0.9999) 125 | self.p99_999 = latency_sketch.get_quantile_value(0.99999) 126 | 127 | def load_http_path(self, path): 128 | self.http_path = path 129 | 130 | def get_type(self): 131 | return self.type 132 | 133 | def get_type_str_no_ip(self): 134 | if self.type is TransactionType.ipv6_tcp or self.type is TransactionType.ipv4_tcp: 135 | return "tcp" 136 | else: 137 | return "http" 138 | 139 | def get_role(self): 140 | return self.role 141 | 142 | def get_role_str(self): 143 | if self.role is TransactionRole.client: 144 | return "client" 145 | else: 146 | return "server" 147 | 148 | def get_saddr(self): 149 | return self.saddr 150 | 151 | def get_lport(self): 152 | return self.lport 153 | 154 | def get_daddr(self): 155 | return self.daddr 156 | 157 | def get_dport(self): 158 | return self.dport 159 | 160 | def get_transaction_count(self): 161 | return self.t_count 162 | 163 | def get_byte_rx(self): 164 | return self.byte_rx 165 | 166 | def get_byte_tx(self): 167 | return self.byte_tx 168 | 169 | def get_avg_latency(self): 170 | return self.avg 171 | 172 | def get_percentiles(self): 173 | return [self.p50, self.p75, self.p90, self.p99, self.p99_9, self.p99_99, self.p99_999] 174 | 175 | def get_http_path(self): 176 | return self.http_path 177 | 178 | def get_samples(self): 179 | return self.samples 180 | 181 | def set_saddr(self, saddr): 182 | self.saddr = saddr 183 | 184 | def set_lport(self, lport): 185 | self.lport = lport 186 | 187 | def set_daddr(self, daddr): 188 | self.daddr = daddr 189 | 190 | def set_dport(self, dport): 191 | self.dport = dport 192 | 193 | 194 | def __str__(self): 195 | role_str = "" 196 | if self.role is TransactionRole.server: 197 | role_str = "server" 198 | else: 199 | role_str = "client" 200 | 201 | output_str = "" 202 | if self.type == TransactionType.ipv4_http or self.type == TransactionType.ipv6_http: 203 | fmt = '{:<8} {:<40} {:<40} {:<20} {:<20} {:<20} {:<25} {:<68}' 204 | output_str = fmt.format( 205 | role_str, 206 | "SRC: " + str(self.saddr) + ":" + str(self.lport), 207 | "DST: " + str(self.daddr) + ":" + str(self.dport), 208 | "T_COUNT: " + str(self.t_count), 209 | "BYTE_TX: " + str(self.byte_tx), 210 | "BYTE_RX: " + str(self.byte_rx), 211 | "LAT_AVG (ms): " + '{:.5f}'.format(self.avg), 212 | str(self.http_path) 213 | ) 214 | 215 | else: 216 | fmt = '{:<8} {:<40} {:<40} {:<20} {:<20} {:<20} {:<25}' 217 | output_str = fmt.format( 218 | role_str, 219 | "SRC: " + str(self.saddr) + ":" + str(self.lport), 220 | "DST: " + str(self.daddr) + ":" + str(self.dport), 221 | "T_COUNT: " + str(self.t_count), 222 | "BYTE_TX: " + str(self.byte_tx), 223 | "BYTE_RX: " + str(self.byte_rx), 224 | "LAT_AVG (ms): " + '{:.5f}'.format(self.avg) 225 | ) 226 | 227 | fmt = '{:<5} {:<30} {:<30} {:<30} {:<30} {:<30} {:<30} {:<30}' 228 | output_str = output_str + "\n" + fmt.format( 229 | "--->", 230 | "50p: " + '{:.5f}'.format(self.p50), 231 | "75p: " + '{:.5f}'.format(self.p75), 232 | "90p: " + '{:.5f}'.format(self.p90), 233 | "99p: " + '{:.5f}'.format(self.p99), 234 | "99.9p: " + '{:.5f}'.format(self.p99_9), 235 | "99.99p: " + '{:.5f}'.format(self.p99_99), 236 | "99.999p: " + '{:.5f}'.format(self.p99_999), 237 | ) 238 | 239 | return output_str 240 | 241 | 242 | 243 | class NatData: 244 | def __init__(self, type, saddr, lport, daddr, dport): 245 | self.type = type 246 | self.saddr = saddr 247 | self.lport = lport 248 | self.daddr = daddr 249 | self.dport = dport 250 | 251 | def get_type(self): 252 | return self.type 253 | 254 | def get_saddr(self): 255 | return self.saddr 256 | 257 | def get_lport(self): 258 | return self.lport 259 | 260 | def get_daddr(self): 261 | return self.daddr 262 | 263 | def get_dport(self): 264 | return self.dport 265 | 266 | def __str__(self): 267 | 268 | fmt = '{:<10} {:<40} {:<40}' 269 | output_str = fmt.format( 270 | "NAT RULE", 271 | "SRC: " + str(self.saddr) + ":" + str(self.lport), 272 | "DST: " + str(self.daddr) + ":" + str(self.dport), 273 | ) 274 | 275 | return output_str 276 | 277 | 278 | 279 | class NetSample: 280 | 281 | def __init__(self, pid_dictionary, nat_dictionary, nat_list, host_transaction_count, host_byte_tx, host_byte_rx): 282 | self.pid_dictionary = pid_dictionary 283 | self.nat_dictionary = nat_dictionary 284 | self.host_transaction_count = host_transaction_count 285 | self.host_byte_tx = host_byte_tx 286 | self.host_byte_rx = host_byte_rx 287 | self.nat_list = nat_list 288 | 289 | def get_pid_dictionary(self): 290 | return self.pid_dictionary 291 | 292 | def get_nat_dictionary(self): 293 | return self.nat_dictionary 294 | 295 | def get_host_transaction_count(self): 296 | return self.host_transaction_count 297 | 298 | def get_host_byte_tx(self): 299 | return self.host_byte_tx 300 | 301 | def get_host_byte_rx(self): 302 | return self.host_byte_rx 303 | 304 | def get_nat_list(self): 305 | return self.nat_list 306 | 307 | 308 | 309 | class NetCollector: 310 | 311 | def __init__(self, trace_nat=False, dynamic_tcp_client_port_masking=False): 312 | self.ebpf_tcp_monitor = None 313 | self.nat = trace_nat 314 | self.dynamic_tcp_client_port_masking = dynamic_tcp_client_port_masking 315 | 316 | # define hash tables, skip endpoints and connections for now 317 | # as they self manage and self clean in eBPF code 318 | self.ipv4_summary = [None, None] 319 | self.ipv6_summary = [None, None] 320 | self.ipv4_http_summary = [None, None] 321 | self.ipv6_http_summary = [None, None] 322 | self.rewritten_rules = None 323 | self.rewritten_rules_6 = None 324 | 325 | self.ipv4_latency = [None, None] 326 | self.ipv6_latency = [None, None] 327 | self.ipv4_http_latency = [None, None] 328 | self.ipv6_http_latency = [None, None] 329 | 330 | self.latency_index_max = 240 331 | self.bucket_count = 15 332 | self.latency_bucket_size = 16 333 | 334 | self.tcp_dyn_masking_threshold = 10 335 | 336 | def start_capture(self): 337 | bpf_code_path = os.path.dirname(os.path.abspath(__file__)) \ 338 | + "/../bpf/tcp_monitor.c" 339 | 340 | cflags = ["-DLATENCY_SAMPLES=%d" % self.latency_index_max, \ 341 | "-DLATENCY_BUCKET_SIZE=%d" % self.latency_bucket_size, \ 342 | "-DBUCKET_COUNT=%d" % self.bucket_count] 343 | if self.nat: 344 | cflags.append("-DBYPASS") 345 | cflags.append("-DREVERSE_BYPASS") 346 | if BPF.tracepoint_exists("sock", "inet_sock_set_state"): 347 | cflags.append("-DSET_STATE_4_16") 348 | elif BPF.tracepoint_exists("tcp", "tcp_set_state"): 349 | cflags.append("-DSET_STATE_4_15") 350 | else: 351 | cflags.append("-DSET_STATE_KPROBE") 352 | if self.dynamic_tcp_client_port_masking: 353 | cflags.append("-DDYN_TCP_CLIENT_PORT_MASKING") 354 | cflags.append("-DDYN_TCP_CLIENT_PORT_MASKING_THRESHOLD=%d" % self.tcp_dyn_masking_threshold) 355 | 356 | # print(cflags) 357 | 358 | self.ebpf_tcp_monitor = BPF(src_file=bpf_code_path, cflags=cflags) 359 | 360 | self.ipv4_summary[0] = self.ebpf_tcp_monitor["ipv4_summary"] 361 | self.ipv6_summary[0] = self.ebpf_tcp_monitor["ipv6_summary"] 362 | self.ipv4_http_summary[0] = self.ebpf_tcp_monitor["ipv4_http_summary"] 363 | self.ipv6_http_summary[0] = self.ebpf_tcp_monitor["ipv6_http_summary"] 364 | self.ipv4_summary[1] = self.ebpf_tcp_monitor["ipv4_summary_1"] 365 | self.ipv6_summary[1] = self.ebpf_tcp_monitor["ipv6_summary_1"] 366 | self.ipv4_http_summary[1] = self.ebpf_tcp_monitor["ipv4_http_summary_1"] 367 | self.ipv6_http_summary[1] = self.ebpf_tcp_monitor["ipv6_http_summary_1"] 368 | self.rewritten_rules = self.ebpf_tcp_monitor["rewritten_rules"] 369 | self.rewritten_rules_6 = self.ebpf_tcp_monitor["rewritten_rules_6"] 370 | 371 | self.ipv4_latency[0] = self.ebpf_tcp_monitor["ipv4_latency"] 372 | self.ipv6_latency[0] = self.ebpf_tcp_monitor["ipv6_latency"] 373 | self.ipv4_http_latency[0] = self.ebpf_tcp_monitor["ipv4_http_latency"] 374 | self.ipv6_http_latency[0] = self.ebpf_tcp_monitor["ipv6_http_latency"] 375 | self.ipv4_latency[1] = self.ebpf_tcp_monitor["ipv4_latency_1"] 376 | self.ipv6_latency[1] = self.ebpf_tcp_monitor["ipv6_latency_1"] 377 | self.ipv4_http_latency[1] = self.ebpf_tcp_monitor["ipv4_http_latency_1"] 378 | self.ipv6_http_latency[1] = self.ebpf_tcp_monitor["ipv6_http_latency_1"] 379 | 380 | self.bpf_config = self.ebpf_tcp_monitor["conf"] 381 | self.selector = 0 382 | self.bpf_config[ct.c_int(0)] = ct.c_uint(self.selector) 383 | 384 | def get_sample(self): 385 | #iterate over summary tables 386 | pid_dict = {} 387 | nat_dict = {} 388 | nat_list = [] 389 | host_transaction_count = 0 390 | host_byte_tx = 0 391 | host_byte_rx = 0 392 | 393 | bucket_count = 0 394 | 395 | old_selector = self.selector 396 | 397 | if self.selector == 0: 398 | self.selector = 1 399 | else: 400 | self.selector = 0 401 | self.bpf_config[ct.c_int(0)] = ct.c_uint(self.selector) 402 | 403 | # set the types and tables to iterate on 404 | transaction_types = [TransactionType.ipv4_tcp, TransactionType.ipv6_tcp, TransactionType.ipv4_http, TransactionType.ipv6_http] 405 | transaction_tables = [self.ipv4_summary[old_selector], self.ipv6_summary[old_selector], self.ipv4_http_summary[old_selector], self.ipv6_http_summary[old_selector]] 406 | transaction_latencies = [self.ipv4_latency[old_selector], self.ipv6_latency[old_selector], self.ipv4_http_latency[old_selector], self.ipv6_http_latency[old_selector]] 407 | 408 | # transaction_types = [TransactionType.ipv4_http, TransactionType.ipv6_http] 409 | # transaction_tables = [self.ipv4_http_summary, self.ipv6_http_summary] 410 | for i in range(0,len(transaction_types)): 411 | transaction_type = transaction_types[i] 412 | transaction_table = transaction_tables[i] 413 | transaction_latency = transaction_latencies[i] 414 | 415 | latency_data = {} 416 | 417 | # retrieve latency reservoir data 418 | for key, value in transaction_latency.items(): 419 | formatted_key = get_session_key_by_type(key, transaction_type) 420 | sketch = latency_data[formatted_key] = latency_data.get(formatted_key, DDSketch()) 421 | 422 | for i in range(0, self.latency_bucket_size): 423 | if value.latency_vector[i] > 0: 424 | sketch.add(float(value.latency_vector[i]) / 1000000) 425 | bucket_count = bucket_count+1 426 | # if value.latency_vector[0] > 0: 427 | # sketch.add(float(value.latency_vector[0]) / 1000000) 428 | # bucket_count = bucket_count+1 429 | # print(latency_data) 430 | 431 | for key, value in transaction_table.items(): 432 | data_item = None 433 | formatted_key = get_session_key_by_type(key, transaction_type) 434 | if value.status == 0 and self.nat: 435 | # we found a nat rule, use the appropriate object 436 | data_item = NatData(transaction_type, formatted_key.saddr, formatted_key.lport, formatted_key.daddr, formatted_key.dport) 437 | nat_list.append(data_item) 438 | # add the nat rule to the pid 439 | if int(value.pid) in nat_dict: 440 | nat_dict[int(value.pid)].append(data_item) 441 | else: 442 | nat_dict[int(value.pid)] = [data_item] 443 | 444 | else: 445 | role = None 446 | if int(value.status) == -1: 447 | role = TransactionRole.client; 448 | elif int(value.status) == 1: 449 | role = TransactionRole.server; 450 | 451 | data_item = TransactionData(transaction_type, role, formatted_key.saddr, formatted_key.lport, formatted_key.daddr, formatted_key.dport, int(value.transaction_count), int(value.byte_rx), int(value.byte_tx)) 452 | try: 453 | data_item.load_latencies(latency_data[formatted_key], int(value.time), int(value.transaction_count)) 454 | except KeyError: 455 | # skip item if we lost it somehow 456 | continue 457 | 458 | if transaction_type == TransactionType.ipv4_http or transaction_type == TransactionType.ipv6_http: 459 | data_item.load_http_path(str(key.http_payload)) 460 | 461 | # sum up host metrics 462 | host_transaction_count = host_transaction_count + int(value.transaction_count) 463 | host_byte_tx = host_byte_tx + int(value.byte_tx) 464 | host_byte_rx = host_byte_rx + int(value.byte_rx) 465 | 466 | # add the data to the pid 467 | if int(value.pid) in pid_dict: 468 | pid_dict[int(value.pid)].append(data_item) 469 | else: 470 | pid_dict[int(value.pid)] = [data_item] 471 | 472 | #print(len(self.ipv4_summary[old_selector])) 473 | # print(len(self.ebpf_tcp_monitor["recv_cache"])) 474 | # print(len(self.ebpf_tcp_monitor["ipv4_endpoints"])) 475 | # print(len(self.ebpf_tcp_monitor["ipv4_connections"])) 476 | #print(len(self.ebpf_tcp_monitor["ipv6_connections"])) 477 | # print(len(self.ipv6_summary)) 478 | # print(len(self.ipv4_http_summary)) 479 | # print(len(self.ipv6_http_summary)) 480 | # print(len(self.ipv4_latency[old_selector])) 481 | # print(len(self.ipv4_http_summary[old_selector]) + len(self.ipv4_summary[old_selector]) + len(self.ipv6_http_summary[old_selector]) + len(self.ipv6_summary[old_selector])) 482 | # print(len(self.ipv4_http_latency[old_selector]) + len(self.ipv4_latency[old_selector]) + len(self.ipv6_http_latency[old_selector]) + len(self.ipv6_latency[old_selector])) 483 | # print(bucket_count) 484 | try: 485 | # clear tables for next sample 486 | self.ipv4_summary[old_selector].clear() 487 | self.ipv6_summary[old_selector].clear() 488 | self.ipv4_http_summary[old_selector].clear() 489 | self.ipv6_http_summary[old_selector].clear() 490 | except Exception as e: 491 | print(e) 492 | # try to clean rewritten rules as for each packet the useful nat rules 493 | # are rewritten inside the tables automatically 494 | try: 495 | self.rewritten_rules.clear() 496 | self.rewritten_rules_6.clear() 497 | except Exception as e: 498 | print(e) 499 | 500 | try: 501 | # clear also reservoir hashmaps 502 | self.ipv4_latency[old_selector].clear() 503 | self.ipv6_latency[old_selector].clear() 504 | self.ipv4_http_latency[old_selector].clear() 505 | self.ipv6_http_latency[old_selector].clear() 506 | except Exception as e: 507 | print(e) 508 | 509 | return NetSample(pid_dict, nat_dict, nat_list, host_transaction_count, host_byte_tx, host_byte_rx) 510 | -------------------------------------------------------------------------------- /userspace/proc_topology.py: -------------------------------------------------------------------------------- 1 | """ 2 | DEEP-mon 3 | Copyright (C) 2020 Brondolin Rolando 4 | 5 | This file is part of DEEP-mon 6 | 7 | DEEP-mon is free software: you can redistribute it and/or modify 8 | it under the terms of the GNU General Public License as published by 9 | the Free Software Foundation, either version 3 of the License, or 10 | (at your option) any later version. 11 | 12 | DEEP-mon is distributed in the hope that it will be useful, 13 | but WITHOUT ANY WARRANTY; without even the implied warranty of 14 | MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 15 | GNU General Public License for more details. 16 | 17 | You should have received a copy of the GNU General Public License 18 | along with this program. If not, see . 19 | """ 20 | 21 | import multiprocessing 22 | import os 23 | import ctypes as ct 24 | 25 | class BpfProcTopology(ct.Structure): 26 | _fields_ = [("ht_id", ct.c_ulonglong), 27 | ("sibling_id", ct.c_ulonglong), 28 | ("core_id", ct.c_ulonglong), 29 | ("processor_id", ct.c_ulonglong), 30 | ("cycles_core", ct.c_ulonglong), 31 | ("cycles_core_delta_sibling", ct.c_ulonglong), 32 | ("cycles_thread", ct.c_ulonglong), 33 | ("cache_misses", ct.c_ulonglong), 34 | ("cache_refs", ct.c_ulonglong), 35 | ("instr_thread", ct.c_ulonglong), 36 | ("ts", ct.c_ulonglong), 37 | ("running_pid", ct.c_int)] 38 | 39 | class ProcTopology: 40 | processors_path = '/proc/cpuinfo' 41 | 42 | def __init__(self): 43 | # parse /proc/cpuinfo to obtain processor topology 44 | ht_id = 0 45 | sibling_id = 0 46 | core_id = 0 47 | processor_id = 0 48 | 49 | #core elem is organized as ht_id, sibling_id, core_id, processor_id 50 | self.coresDict = {} 51 | self.socket_set = set() 52 | 53 | with open(ProcTopology.processors_path) as f: 54 | for line in f: 55 | sp = line.split(" ") 56 | if "processor\t" in sp[0]: 57 | ht_id = int(sp[1]) 58 | if "physical" in sp[0] and "id\t" in sp[1]: 59 | processor_id = int(sp[2]) 60 | self.socket_set.add(processor_id) 61 | if "core" in sp[0] and "id\t\t" in sp[1]: 62 | core_id = int(sp[2]) 63 | found = False 64 | for key, value in self.coresDict.items(): 65 | if value[2] == core_id and value[3] == processor_id: 66 | found = True 67 | value[1] = ht_id 68 | self.coresDict[ht_id] = [ht_id, value[0], core_id, \ 69 | processor_id] 70 | break 71 | if not found: 72 | self.coresDict[ht_id] = [ht_id, -1, core_id, processor_id] 73 | 74 | def print_topology(self): 75 | for key, value in self.coresDict.items(): 76 | print(value) 77 | 78 | def get_topology(self): 79 | return self.coresDict 80 | 81 | def get_sockets(self): 82 | return self.socket_set 83 | 84 | def get_hyperthread_count(self): 85 | return len(self.coresDict) 86 | 87 | def get_new_bpf_topology(self): 88 | bpf_dict = {} 89 | for key,value in self.coresDict.items(): 90 | core = BpfProcTopology(ct.c_ulonglong(value[0]), \ 91 | ct.c_ulonglong(value[1]), \ 92 | ct.c_ulonglong(value[2]), \ 93 | ct.c_ulonglong(value[3]), \ 94 | ct.c_ulonglong(0), \ 95 | ct.c_ulonglong(0), \ 96 | ct.c_ulonglong(0), \ 97 | ct.c_ulonglong(0), \ 98 | ct.c_ulonglong(0), \ 99 | ct.c_ulonglong(0), \ 100 | ct.c_ulonglong(0), \ 101 | ct.c_int(0)) 102 | bpf_dict[key] = core 103 | return bpf_dict 104 | -------------------------------------------------------------------------------- /userspace/process_info.py: -------------------------------------------------------------------------------- 1 | """ 2 | DEEP-mon 3 | Copyright (C) 2020 Brondolin Rolando 4 | 5 | This file is part of DEEP-mon 6 | 7 | DEEP-mon is free software: you can redistribute it and/or modify 8 | it under the terms of the GNU General Public License as published by 9 | the Free Software Foundation, either version 3 of the License, or 10 | (at your option) any later version. 11 | 12 | DEEP-mon is distributed in the hope that it will be useful, 13 | but WITHOUT ANY WARRANTY; without even the implied warranty of 14 | MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 15 | GNU General Public License for more details. 16 | 17 | You should have received a copy of the GNU General Public License 18 | along with this program. If not, see . 19 | """ 20 | 21 | import ctypes as ct 22 | from .net_collector import TransactionData 23 | import json 24 | 25 | 26 | class BpfPidStatus(ct.Structure): 27 | TASK_COMM_LEN = 16 28 | socket_size = 0 29 | _fields_ = [("pid", ct.c_int), 30 | ("tgid", ct.c_int), 31 | ("comm", ct.c_char * TASK_COMM_LEN), 32 | ("weighted_cycles", ct.c_ulonglong * 2 * socket_size), 33 | ("cycles", ct.c_ulonglong * 2), 34 | ("instruction_retired", ct.c_ulonglong * 2), 35 | ("cache_misses", ct.c_ulonglong * 2), 36 | ("cache_refs", ct.c_ulonglong * 2), 37 | ("time_ns", ct.c_ulonglong * 2), 38 | ("bpf_selector", ct.c_int), 39 | ("ts", ct.c_ulonglong * 2 * socket_size)] 40 | 41 | def __init__(self, socket_size): 42 | self.socket_size = socket_size 43 | 44 | class SocketProcessItem: 45 | 46 | def __init__(self, weighted_cycles = 0, ts = 0): 47 | self.weighted_cycles = weighted_cycles 48 | self.ts = ts 49 | 50 | def set_weighted_cycles(self, weighted_cycles): 51 | self.weighted_cycles = weighted_cycles 52 | 53 | def set_ts(self, ts): 54 | self.ts = ts 55 | 56 | def get_weighted_cycles(self): 57 | return self.weighted_cycles 58 | 59 | def get_ts(self): 60 | return self.ts 61 | 62 | def reset(self): 63 | self.weighted_cycles = 0 64 | self.ts = 0 65 | 66 | def __str__(self): 67 | return "ts: " + str(self.ts) \ 68 | + " w:" + str(self.weighted_cycles) 69 | 70 | class ProcessInfo: 71 | 72 | def __init__(self, num_sockets): 73 | self.pid = -1 74 | self.tgid = -1 75 | self.comm = "" 76 | self.power = 0.0 77 | self.cpu_usage = 0.0 78 | self.socket_data = [] 79 | self.cgroup_id = "" 80 | self.container_id = "" 81 | 82 | self.instruction_retired = 0 83 | self.cycles = 0 84 | self.cache_misses = 0 85 | self.cache_refs = 0 86 | self.time_ns = 0 87 | 88 | self.network_transactions = [] 89 | self.nat_rules = [] 90 | 91 | for i in range(0, num_sockets): 92 | self.socket_data.append(SocketProcessItem()) 93 | 94 | def set_pid(self, pid): 95 | self.pid = pid 96 | 97 | def set_tgid(self, tgid): 98 | self.tgid = tgid 99 | 100 | def set_comm(self, comm): 101 | self.comm = comm 102 | 103 | def set_power(self, power): 104 | self.power = float(power) 105 | 106 | def set_cpu_usage(self, cpu_usage): 107 | self.cpu_usage = float(cpu_usage) 108 | 109 | def set_instruction_retired(self, instruction_retired): 110 | self.instruction_retired = instruction_retired 111 | 112 | def set_cycles(self, cycles): 113 | self.cycles = cycles 114 | 115 | def set_cache_misses(self, cache_misses): 116 | self.cache_misses = cache_misses 117 | 118 | def set_cache_refs(self, cache_refs): 119 | self.cache_refs = cache_refs 120 | 121 | def set_time_ns(self, time_ns): 122 | self.time_ns = time_ns 123 | 124 | def compute_cpu_usage_millis(self, total_execution_time_millis, total_cores): 125 | self.cpu_usage = 0 126 | if total_execution_time_millis != 0: 127 | self.cpu_usage = float((self.time_ns/1000000) \ 128 | / total_execution_time_millis * total_cores * 100) # percentage moved to other percentage 129 | 130 | def set_socket_data_array(self, socket_data_array): 131 | self.socket_data = socket_data_array 132 | 133 | def set_socket_data(self, socket_index, socket_data): 134 | self.socket_data[socket_index] = socket_data 135 | 136 | def set_raw_socket_data(self, socket_index, weighted_cycles, ts): 137 | self.socket_data[socket_index] = \ 138 | SocketProcessItem(weighted_cycles, instruction_retired, time_ns, ts) 139 | 140 | def set_cgroup_id(self, cgroup_id): 141 | self.cgroup_id = cgroup_id 142 | 143 | def set_container_id(self, container_id): 144 | self.container_id = container_id 145 | 146 | def set_network_transactions(self, network_transactions): 147 | self.network_transactions = network_transactions 148 | 149 | def set_nat_rules(self, nat_rules): 150 | self.nat_rules = nat_rules 151 | 152 | 153 | def reset_data(self): 154 | self.instruction_retired = 0 155 | self.cycles = 0 156 | self.cache_misses = 0 157 | self.time_ns = 0 158 | self.network_transactions = [] 159 | self.nat_rules = [] 160 | for item in self.socket_data: 161 | item.reset() 162 | 163 | def get_pid(self): 164 | return self.pid 165 | 166 | def get_tgid(self): 167 | return self.tgid 168 | 169 | def get_comm(self): 170 | return self.comm 171 | 172 | def get_power(self): 173 | return self.power 174 | 175 | def get_cpu_usage(self): 176 | return self.cpu_usage 177 | 178 | def get_instruction_retired(self): 179 | return self.instruction_retired 180 | 181 | def get_cycles(self): 182 | return self.cycles 183 | 184 | def get_cache_misses(self): 185 | return self.cache_misses 186 | 187 | def get_cache_refs(self): 188 | return self.cache_refs 189 | 190 | def get_time_ns(self): 191 | return self.time_ns 192 | 193 | def get_socket_data(self, socket_index = -1): 194 | if socket_index < 0: 195 | return self.socket_data 196 | return self.socket_data[socket_index] 197 | 198 | def get_cgroup_id(self): 199 | return self.cgroup_id 200 | 201 | def get_container_id(self): 202 | return self.container_id 203 | 204 | def get_aggregated_weighted_cycles(self): 205 | aggregated = 0 206 | for item in self.socket_data: 207 | aggregated = aggregated + item.get_weighted_cycles() 208 | return aggregated 209 | 210 | def get_last_ts(self): 211 | max_ts = 0 212 | for item in self.socket_data: 213 | if max_ts < item.get_ts(): 214 | max_ts = item.get_ts() 215 | return max_ts 216 | 217 | def get_network_transactions(self): 218 | return self.network_transactions 219 | 220 | def get_nat_rules(self): 221 | return self.nat_rules 222 | 223 | 224 | def __str__(self): 225 | str_rep = str(self.pid) + " comm: " + str(self.comm) \ 226 | + " c_id: " + self.container_id + " p: " + str(self.power) \ 227 | + " u: " + str(self.cpu_usage) 228 | 229 | for socket_item in self.socket_data: 230 | str_rep = str_rep + " " + str(socket_item) 231 | 232 | return str_rep 233 | -------------------------------------------------------------------------------- /userspace/process_table.py: -------------------------------------------------------------------------------- 1 | """ 2 | DEEP-mon 3 | Copyright (C) 2020 Brondolin Rolando 4 | 5 | This file is part of DEEP-mon 6 | 7 | DEEP-mon is free software: you can redistribute it and/or modify 8 | it under the terms of the GNU General Public License as published by 9 | the Free Software Foundation, either version 3 of the License, or 10 | (at your option) any later version. 11 | 12 | DEEP-mon is distributed in the hope that it will be useful, 13 | but WITHOUT ANY WARRANTY; without even the implied warranty of 14 | MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 15 | GNU General Public License for more details. 16 | 17 | You should have received a copy of the GNU General Public License 18 | along with this program. If not, see . 19 | """ 20 | 21 | from .process_info import ProcessInfo 22 | from .bpf_collector import BpfSample 23 | from .container_info import ContainerInfo 24 | import os 25 | import docker 26 | 27 | class ProcTable: 28 | 29 | def __init__(self): 30 | self.proc_table = {} 31 | self.docker_client = docker.from_env() 32 | 33 | # remove processes that did not receive updates in the last 8 seconds 34 | def reset_metrics_and_evict_stale_processes(self, ts): 35 | evicted_keys = [] 36 | 37 | for proc_table_key, proc_table_value in self.proc_table.items(): 38 | if proc_table_value.get_last_ts() + 8000000000 < ts: 39 | evicted_keys.append(proc_table_key) 40 | else: 41 | proc_table_value.set_power(0) 42 | proc_table_value.set_cpu_usage(0) 43 | proc_table_value.reset_data() 44 | 45 | #remove evicted keys 46 | for k in evicted_keys: 47 | self.proc_table.pop(k, None) 48 | 49 | 50 | def add_process(self, proc_info): 51 | self.proc_table[proc_info.get_pid()] = proc_info 52 | 53 | 54 | def add_process_from_sample(self, sample, net_dictionary=None, nat_dictionary=None): 55 | # reset counters for each entries 56 | for key, value in sample.get_pid_dict().items(): 57 | if key in self.proc_table: 58 | # process already there, check if comm is the same 59 | if value.get_comm() == self.proc_table[key].get_comm(): 60 | # ok, update stuff 61 | self.proc_table[key].set_power(value.get_power()) 62 | self.proc_table[key].set_cpu_usage(value.get_cpu_usage()) 63 | self.proc_table[key].set_instruction_retired(value.get_instruction_retired()) 64 | self.proc_table[key].set_cycles(value.get_cycles()) 65 | self.proc_table[key].set_cache_misses(value.get_cache_misses()) 66 | self.proc_table[key].set_cache_refs(value.get_cache_refs()) 67 | self.proc_table[key].set_time_ns(value.get_time_ns()) 68 | self.proc_table[key].set_socket_data_array(value.get_socket_data()) 69 | 70 | else: 71 | # process is changed, replace entry and find cgroup_id 72 | value.set_cgroup_id(self.find_cgroup_id(key, value.tgid)) 73 | value.set_container_id(value.get_cgroup_id()[0:12]) 74 | self.proc_table[key] = value 75 | else: 76 | # new process, add it and find cgroup_id 77 | value.set_cgroup_id(self.find_cgroup_id(key, value.tgid)) 78 | value.set_container_id(value.get_cgroup_id()[0:12]) 79 | self.proc_table[key] = value 80 | if net_dictionary and key in net_dictionary: 81 | self.proc_table[key].set_network_transactions(net_dictionary[key]) 82 | if nat_dictionary and key in nat_dictionary: 83 | self.proc_table[key].set_nat_rules(nat_dictionary[key]) 84 | 85 | def find_cgroup_id(self, pid, tgid): 86 | # exclude idle 87 | if pid < 0: 88 | return "----idle----" 89 | 90 | for id in [pid, tgid]: 91 | 92 | #scan proc folder searching for the pid 93 | for path in ['/host/proc', '/proc']: 94 | try: 95 | # Non-systemd Docker 96 | with open(os.path.join(path, str(id), 'cgroup'), 'r') as f: 97 | for line in f: 98 | line_array = line.split("/") 99 | if len(line_array) > 1 and \ 100 | len(line_array[len(line_array) -1]) == 65: 101 | return line_array[len(line_array) -1] 102 | except IOError: 103 | continue 104 | 105 | for path in ['/host/proc', '/proc']: 106 | try: 107 | # systemd Docker 108 | with open(os.path.join(path, str(id), 'cgroup'), 'r') as f: 109 | for line in f: 110 | line_array = line.split("/") 111 | if len(line_array) > 1 \ 112 | and "docker-" in line_array[len(line_array) -1] \ 113 | and ".scope" in line_array[len(line_array) -1]: 114 | 115 | new_id = line_array[len(line_array) -1].replace("docker-", "") 116 | new_id = new_id.replace(".scope", "") 117 | if len(new_id) == 65: 118 | return new_id 119 | 120 | except IOError: # proc has already terminated 121 | continue 122 | return "---others---" 123 | 124 | def get_proc_table(self): 125 | return self.proc_table 126 | 127 | def get_container_dictionary(self, mem_dictionary = None, disk_dictionary = None): 128 | container_dict = {} 129 | # not_a_container = ContainerInfo("---others---") 130 | # idle = ContainerInfo("----idle----") 131 | # container_dict["---others---"] = not_a_container 132 | # container_dict["----idle----"] = idle 133 | 134 | for key, value in self.proc_table.items(): 135 | if value.container_id != "": 136 | if value.container_id not in container_dict: 137 | container_dict[value.container_id] = ContainerInfo(value.container_id) 138 | 139 | if value.container_id not in ["---others---", "----idle----"]: 140 | #retrieve info from docker 141 | container = self.docker_client.containers.get(value.container_id) 142 | container_dict[value.container_id].set_container_name(str(container.name)) 143 | container_dict[value.container_id].set_container_image(str(container.image)) 144 | container_dict[value.container_id].set_container_labels(container.labels) 145 | 146 | container_dict[value.container_id].add_cycles(value.get_cycles()) 147 | container_dict[value.container_id].add_weighted_cycles(value.get_aggregated_weighted_cycles()) 148 | container_dict[value.container_id].add_instructions(value.get_instruction_retired()) 149 | container_dict[value.container_id].add_cache_misses(value.get_cache_misses()) 150 | container_dict[value.container_id].add_cache_refs(value.get_cache_refs()) 151 | container_dict[value.container_id].add_time_ns(value.get_time_ns()) 152 | container_dict[value.container_id].add_power(value.get_power()) 153 | container_dict[value.container_id].add_cpu_usage(value.get_cpu_usage()) 154 | container_dict[value.container_id].add_pid(value.get_pid()) 155 | container_dict[value.container_id].set_last_ts(value.get_last_ts()) 156 | container_dict[value.container_id].add_network_transactions(value.get_network_transactions()) 157 | container_dict[value.container_id].add_nat_rules(value.get_nat_rules()) 158 | 159 | # aggregate stuff at the container level 160 | for key, value in container_dict.items(): 161 | value.compute_aggregate_network_metrics() 162 | 163 | if mem_dictionary: 164 | for key,value in container_dict.items(): 165 | if key in mem_dictionary: 166 | value.set_mem_RSS(mem_dictionary[key]["RSS"]) 167 | value.set_mem_PSS(mem_dictionary[key]["PSS"]) 168 | value.set_mem_USS(mem_dictionary[key]["USS"]) 169 | 170 | if disk_dictionary: 171 | for key,value in container_dict.items(): 172 | if key in disk_dictionary: 173 | value.set_disk_kb_r(disk_dictionary[key]["kb_r"]) 174 | value.set_disk_kb_w(disk_dictionary[key]["kb_w"]) 175 | value.set_disk_num_r(disk_dictionary[key]["num_r"]) 176 | value.set_disk_num_w(disk_dictionary[key]["num_w"]) 177 | value.set_disk_avg_lat(disk_dictionary[key]["avg_lat"]) 178 | 179 | 180 | return container_dict 181 | -------------------------------------------------------------------------------- /userspace/rapl/__init__.py: -------------------------------------------------------------------------------- 1 | """ 2 | DEEP-mon 3 | Copyright (C) 2020 Brondolin Rolando 4 | 5 | This file is part of DEEP-mon 6 | 7 | DEEP-mon is free software: you can redistribute it and/or modify 8 | it under the terms of the GNU General Public License as published by 9 | the Free Software Foundation, either version 3 of the License, or 10 | (at your option) any later version. 11 | 12 | DEEP-mon is distributed in the hope that it will be useful, 13 | but WITHOUT ANY WARRANTY; without even the implied warranty of 14 | MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 15 | GNU General Public License for more details. 16 | 17 | You should have received a copy of the GNU General Public License 18 | along with this program. If not, see . 19 | """ 20 | -------------------------------------------------------------------------------- /userspace/rapl/rapl.py: -------------------------------------------------------------------------------- 1 | """ 2 | DEEP-mon 3 | Copyright (C) 2020 Brondolin Rolando 4 | 5 | This file is part of DEEP-mon 6 | 7 | DEEP-mon is free software: you can redistribute it and/or modify 8 | it under the terms of the GNU General Public License as published by 9 | the Free Software Foundation, either version 3 of the License, or 10 | (at your option) any later version. 11 | 12 | DEEP-mon is distributed in the hope that it will be useful, 13 | but WITHOUT ANY WARRANTY; without even the implied warranty of 14 | MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 15 | GNU General Public License for more details. 16 | 17 | You should have received a copy of the GNU General Public License 18 | along with this program. If not, see . 19 | """ 20 | 21 | from collections import namedtuple 22 | from datetime import datetime 23 | 24 | 25 | class RaplReader(): 26 | 27 | def _read_sysfs_file(self, path): 28 | try: 29 | with open(path, "r") as f: 30 | contents = f.read().strip() 31 | return contents 32 | except EnvironmentError: 33 | return "0" 34 | 35 | def read_energy_core_sample(self, package=0): 36 | energy = int(self._read_sysfs_file("/sys/class/powercap/intel-rapl/" + 37 | "intel-rapl:{}/intel-rapl:{}:0/energy_uj".format(package, package))) 38 | return RaplSample(energy, datetime.now()) 39 | 40 | def read_energy_dram_sample(self, package=0): 41 | energy = int(self._read_sysfs_file("/sys/class/powercap/intel-rapl/" + 42 | "intel-rapl:{}/intel-rapl:{}:1/energy_uj".format(package, package))) 43 | return RaplSample(energy, datetime.now()) 44 | 45 | def read_energy_package_sample(self, package=0): 46 | energy = int(self._read_sysfs_file("/sys/class/powercap/intel-rapl/" + 47 | "intel-rapl:{}/energy_uj".format(package))) 48 | return RaplSample(energy, datetime.now()) 49 | 50 | 51 | class RaplSample(): 52 | def __init__(self, energy, timestamp): 53 | self.energy_uj = energy 54 | self.sample_time = timestamp 55 | 56 | @property 57 | def energy(self): 58 | return self.energy_uj 59 | 60 | @property 61 | def time(self): 62 | return self.sample_time 63 | 64 | def __sub__(self, other): 65 | energy_diff = self.energy_uj - other.energy_uj 66 | delta_time = (self.sample_time - other.sample_time).total_seconds() 67 | # this is overflow! 68 | if energy_diff < 0 and delta_time > 0: 69 | energy_diff = 2**32 + self.energy_uj - other.energy_uj 70 | return RaplDiff(energy_diff, delta_time) 71 | 72 | 73 | class RaplDiff(): 74 | def __init__(self, energy, time): 75 | self.energy_uj = energy 76 | self.duration = time 77 | 78 | @property 79 | def energy(self): 80 | return self.energy_uj 81 | 82 | def power_w(self): 83 | # Convert from microJ to J and return power consumption 84 | return (self.energy_uj / 1000000) / self.duration 85 | 86 | def power_milliw(self): 87 | # Convert from microJ to milliJ and return power consumption 88 | return (self.energy_uj / 1000) / self.duration 89 | 90 | def power_microw(self): 91 | return self.energy_uj / self.duration 92 | 93 | 94 | class RaplMonitor(): 95 | 96 | def __init__(self, topology): 97 | self.rapl_reader = RaplReader() 98 | self.topology = topology 99 | self.sample_core = [self.rapl_reader.read_energy_core_sample(skt) 100 | for skt in self.topology.get_sockets()] 101 | self.sample_pkg = [self.rapl_reader.read_energy_package_sample(skt) 102 | for skt in self.topology.get_sockets()] 103 | self.sample_dram = [self.rapl_reader.read_energy_dram_sample(skt) 104 | for skt in self.topology.get_sockets()] 105 | 106 | def take_sample_package(self): 107 | package_sample = [self.rapl_reader.read_energy_package_sample(skt) 108 | for skt in self.topology.get_sockets()] 109 | return package_sample 110 | 111 | def take_sample_core(self): 112 | core_sample = [self.rapl_reader.read_energy_core_sample(skt) 113 | for skt in self.topology.get_sockets()] 114 | return core_sample 115 | 116 | def take_sample_dram(self): 117 | dram_sample = [self.rapl_reader.read_energy_dram_sample(skt) 118 | for skt in self.topology.get_sockets()] 119 | return dram_sample 120 | 121 | def diff_samples(self, final_sample, initial_sample): 122 | rapl_diff = [final_sample[skt] - initial_sample[skt] 123 | for skt in self.topology.get_sockets()] 124 | return rapl_diff 125 | 126 | def get_rapl_measure(self): 127 | ret = {} 128 | package_sample = self.take_sample_package() 129 | core_sample = self.take_sample_core() 130 | dram_sample = self.take_sample_dram() 131 | 132 | ret["package"] = self.diff_samples(package_sample, self.sample_pkg) 133 | ret["core"] = self.diff_samples(core_sample, self.sample_core) 134 | ret["dram"] = self.diff_samples(dram_sample, self.sample_dram) 135 | 136 | self.sample_pkg = package_sample 137 | self.sample_core = core_sample 138 | self.sample_dram = dram_sample 139 | 140 | return ret 141 | -------------------------------------------------------------------------------- /userspace/sample_controller.py: -------------------------------------------------------------------------------- 1 | """ 2 | DEEP-mon 3 | Copyright (C) 2020 Brondolin Rolando 4 | 5 | This file is part of DEEP-mon 6 | 7 | DEEP-mon is free software: you can redistribute it and/or modify 8 | it under the terms of the GNU General Public License as published by 9 | the Free Software Foundation, either version 3 of the License, or 10 | (at your option) any later version. 11 | 12 | DEEP-mon is distributed in the hope that it will be useful, 13 | but WITHOUT ANY WARRANTY; without even the implied warranty of 14 | MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 15 | GNU General Public License for more details. 16 | 17 | You should have received a copy of the GNU General Public License 18 | along with this program. If not, see . 19 | """ 20 | 21 | class SampleController: 22 | 23 | def __init__(self, processors): 24 | self.timeslice = 1000000000 25 | self.sleep_time = 1 26 | self.processors = processors 27 | 28 | def compute_sleep_time(self, sched_switches): 29 | if sched_switches/(self.processors*self.sleep_time)< 100: 30 | self.sleep_time = 4 31 | self.timeslice = 4000000000 32 | elif sched_switches/(self.processors*self.sleep_time)< 200: 33 | self.sleep_time = 3 34 | self.timeslice = 3000000000 35 | elif sched_switches/(self.processors*self.sleep_time)< 300: 36 | self.sleep_time = 2 37 | self.timeslice = 2000000000 38 | else: 39 | self.sleep_time = 1 40 | self.timeslice = 1000000000 41 | 42 | def get_sleep_time(self): 43 | return self.sleep_time 44 | 45 | def get_timeslice(self): 46 | return self.timeslice 47 | --------------------------------------------------------------------------------