├── README.md ├── common.py ├── gdbpmp.py ├── gdbtypes.py └── tracer.py /README.md: -------------------------------------------------------------------------------- 1 | # gdbpmp - A GDB Based Wallclock Profiler 2 | 3 | gdbpmp is a wall clock time-based profiler based loosely on Mak 4 | Nazečić-Andrlon's gdbprof. 5 | 6 | ### Rationale 7 | 8 | gdbpmp expands on gdbprof in a number of ways: 9 | 10 | * gdbpmp can be run from the command-line or from inside GDB. 11 | * Sample data can be saved and viewed at a later time. 12 | * Samples are gathered independently for every thread. 13 | * Filters can be used to limit which threads samples are gathered from. 14 | * CTRL+C can be used to stop sample gathering along with a specified number of samples to collect before exiting. 15 | * Callgraph depth can be limited a percentage threshold. 16 | * Inverse callgraphs can be printed. 17 | 18 | ### Caveats 19 | 20 | Some features are still being implemented. KeyboardInterrupt handling still 21 | a bit wonky. 22 | 23 | ### Help 24 | ``` 25 | usage: gdbpmp.py [-h] (-i INPUT | -p PID) [-s SLEEP] [-n SAMPLES] [-m MATCH] 26 | [-x EXCLUDE] [-o OUTPUT] [-g GDB_PATH] [-t THRESHOLD] [-v] 27 | [-d] [-w MAX_WIDTH] [-r] 28 | 29 | optional arguments: 30 | -h, --help show this help message and exit 31 | -i INPUT, --input INPUT 32 | Read collected samples from this file. 33 | -p PID, --pid PID PID of the process to connect to. 34 | -s SLEEP, --sleep SLEEP 35 | Period of time to sleep between samples in seconds. 36 | -n SAMPLES, --samples SAMPLES 37 | The number of samples to collect. 38 | -m MATCH, --match MATCH 39 | A comma separated list of strings to match with 40 | threads 41 | -x EXCLUDE, --exclude EXCLUDE 42 | A comma separated list of strings to match when 43 | excluding threads. 44 | -o OUTPUT, --output OUTPUT 45 | Write collected samples to this file. 46 | -g GDB_PATH, --gdb_path GDB_PATH 47 | Path to the GDB executable. 48 | -t THRESHOLD, --threshold THRESHOLD 49 | Ignore results below the threshold when making the 50 | callgraph. 51 | -v, --invert Print inverted callgraph. 52 | -d, --detachattach Use the detach/attach trace mode. Slower, but more 53 | compatible. 54 | -w MAX_WIDTH, --max_width MAX_WIDTH 55 | Set the display width (default is terminal width) 56 | -r, --truncate Truncate lines to the terminal width 57 | ``` 58 | ### Example 59 | ``` 60 | $ ./gdbpmp.py -p 18965 -n 100 -m bstore_kv_sync,bstore_kv_final -o gdbpmp.data 61 | Attaching to process 18965...Done. 62 | Gathering Samples............................................................. 63 | ....................................... 64 | Profiling complete with 100 samples. 65 | ``` 66 | ``` 67 | $ ./gdbpmp.py -i gdbpmp.data 68 | 69 | 70 | Thread: 1 (ceph-osd) - 0 samples 71 | 72 | 73 | Thread: 2 (signal_handler) - 0 samples 74 | 75 | 76 | Thread: 3 (osd_srv_agent) - 0 samples 77 | 78 | 79 | Thread: 4 (safe_timer) - 0 samples 80 | 81 | 82 | Thread: 5 (safe_timer) - 0 samples 83 | 84 | 85 | Thread: 6 (safe_timer) - 0 samples 86 | 87 | 88 | Thread: 7 (safe_timer) - 0 samples 89 | 90 | 91 | Thread: 8 (finisher) - 0 samples 92 | 93 | 94 | Thread: 9 (fn_anonymous) - 0 samples 95 | 96 | 97 | Thread: 10 (osd_srv_heartbt) - 0 samples 98 | 99 | 100 | Thread: 11 (tp_osd_cmd) - 0 samples 101 | 102 | 103 | Thread: 12 (tp_osd_disk) - 0 samples 104 | 105 | 106 | Thread: 13 (tp_osd_tp) - 0 samples 107 | 108 | 109 | Thread: 14 (tp_osd_tp) - 0 samples 110 | 111 | 112 | Thread: 15 (tp_osd_tp) - 0 samples 113 | 114 | 115 | Thread: 16 (tp_osd_tp) - 0 samples 116 | 117 | 118 | Thread: 17 (tp_osd_tp) - 0 samples 119 | 120 | 121 | Thread: 18 (tp_osd_tp) - 0 samples 122 | 123 | 124 | Thread: 19 (tp_osd_tp) - 0 samples 125 | 126 | 127 | Thread: 20 (tp_osd_tp) - 0 samples 128 | 129 | 130 | Thread: 21 (tp_osd_tp) - 0 samples 131 | 132 | 133 | Thread: 22 (tp_osd_tp) - 0 samples 134 | 135 | 136 | Thread: 23 (tp_osd_tp) - 0 samples 137 | 138 | 139 | Thread: 24 (tp_osd_tp) - 0 samples 140 | 141 | 142 | Thread: 25 (tp_osd_tp) - 0 samples 143 | 144 | 145 | Thread: 26 (tp_osd_tp) - 0 samples 146 | 147 | 148 | Thread: 27 (tp_osd_tp) - 0 samples 149 | 150 | 151 | Thread: 28 (tp_osd_tp) - 0 samples 152 | 153 | 154 | Thread: 29 (tp_peering) - 0 samples 155 | 156 | 157 | Thread: 30 (tp_peering) - 0 samples 158 | 159 | 160 | Thread: 31 (safe_timer) - 0 samples 161 | 162 | 163 | Thread: 32 (fn_anonymous) - 0 samples 164 | 165 | 166 | Thread: 33 (safe_timer) - 0 samples 167 | 168 | 169 | Thread: 34 (ms_local) - 0 samples 170 | 171 | 172 | Thread: 35 (ms_dispatch) - 0 samples 173 | 174 | 175 | Thread: 36 (ms_local) - 0 samples 176 | 177 | 178 | Thread: 37 (ms_dispatch) - 0 samples 179 | 180 | 181 | Thread: 38 (ms_local) - 0 samples 182 | 183 | 184 | Thread: 39 (ms_dispatch) - 0 samples 185 | 186 | 187 | Thread: 40 (ms_local) - 0 samples 188 | 189 | 190 | Thread: 41 (ms_dispatch) - 0 samples 191 | 192 | 193 | Thread: 42 (ms_local) - 0 samples 194 | 195 | 196 | Thread: 43 (ms_dispatch) - 0 samples 197 | 198 | 199 | Thread: 44 (ms_local) - 0 samples 200 | 201 | 202 | Thread: 45 (ms_dispatch) - 0 samples 203 | 204 | 205 | Thread: 46 (ms_local) - 0 samples 206 | 207 | 208 | Thread: 47 (ms_dispatch) - 0 samples 209 | 210 | 211 | Thread: 48 (bstore_mempool) - 0 samples 212 | 213 | 214 | Thread: 49 (bstore_kv_final) - 100 samples 215 | 216 | + 100.00% clone 217 | + 100.00% start_thread 218 | + 100.00% BlueStore::KVFinalizeThread::entry 219 | + 100.00% BlueStore::_kv_finalize_thread 220 | + 65.00% std::condition_variable::wait(std::unique_lock&) 221 | | + 65.00% pthread_cond_wait@@GLIBC_2.3.2 222 | + 22.00% BlueStore::deferred_try_submit 223 | | + 20.00% BlueStore::_deferred_submit_unlock 224 | | | + 16.00% KernelDevice::aio_submit 225 | | | | + 16.00% aio_queue_t::submit_batch 226 | | | | + 16.00% io_submit 227 | | | + 3.00% KernelDevice::aio_write 228 | | | | + 1.00% should_gather 229 | | | | + 1.00% push_back 230 | | | | | + 1.00% _M_insert 231 | | | | | + 1.00% _M_create_node 232 | | | | | + 1.00% construct, aio_t> 233 | | | | | + 1.00% _List_node 234 | | | | | + 1.00% aio_t 235 | | | | | + 1.00% ceph::buffer::list::list(ceph::buffer::list&&) 236 | | | | | + 1.00% ceph::buffer::list::iterator_impl::advance 237 | | | | + 1.00% prepare_iov > 238 | | | + 1.00% PerfCounters::inc 239 | | + 1.00% std::vector, std::allocator > >::reserve 240 | | + 1.00% capacity 241 | + 11.00% BlueStore::_txc_state_proc 242 | | + 7.00% BlueStore::_txc_finish 243 | | | + 3.00% BlueStore::BufferSpace::finish_write 244 | | | | + 3.00% erase 245 | | | | + 3.00% std::_Rb_tree > >, std::_Select1st > > >, std::less, mempool::pool_allocator<(mempool::pool_index_t)4, std::pair > > > >::erase 246 | | | | + 3.00% erase 247 | | | | + 3.00% _M_erase_aux 248 | | | | + 3.00% clear 249 | | | | + 3.00% std::_Rb_tree > >, std::_Select1st > > >, std::less, mempool::pool_allocator<(mempool::pool_index_t)4, std::pair > > > >::_M_erase 250 | | | | + 3.00% _M_destroy_node 251 | | | | + 2.00% destroy 252 | | | | | + 2.00% ~_Rb_tree_node 253 | | | | | + 2.00% ~pair 254 | | | | | + 2.00% ~unique_ptr 255 | | | | | + 2.00% operator() 256 | | | | | + 1.00% ~Buffer 257 | | | | | | + 1.00% ~list 258 | | | | | | + 1.00% ~list 259 | | | | | | + 1.00% ~_List_base 260 | | | | | | + 1.00% std::_List_base >::_M_clear 261 | | | | | | + 1.00% destroy > 262 | | | | | | + 1.00% ~_List_node 263 | | | | | + 1.00% BlueStore::Buffer::operator delete 264 | | | | | + 1.00% deallocate 265 | | | | | + 1.00% operator-= 266 | | | | + 1.00% _M_put_node 267 | | | | + 1.00% deallocate 268 | | | | + 1.00% tc_deletearray 269 | | | + 2.00% BlueStore::TransContext::~TransContext 270 | | | | + 2.00% ~TransContext 271 | | | | + 1.00% ~bluestore_deferred_transaction_t 272 | | | | | + 1.00% ~list 273 | | | | | + 1.00% ~_List_base 274 | | | | | + 1.00% std::_List_base >::_M_clear 275 | | | | | + 1.00% destroy > 276 | | | | | + 1.00% ~_List_node 277 | | | | | + 1.00% ~bluestore_deferred_op_t 278 | | | | | + 1.00% ~list 279 | | | | | + 1.00% ~list 280 | | | | | + 1.00% ~_List_base 281 | | | | | + 1.00% std::_List_base >::_M_clear 282 | | | | | + 1.00% _M_put_node 283 | | | | | + 1.00% deallocate 284 | | | | | + 1.00% tc_delete 285 | | | | | + 1.00% tcmalloc::ThreadCache::ListTooLong(tcmalloc::ThreadCache::FreeList*, unsigned long) 286 | | | | | + 1.00% tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*, unsigned long, int) 287 | | | | + 1.00% ~IOContext 288 | | | | + 1.00% pthread_cond_destroy@@GLIBC_2.3.2 289 | | | + 1.00% tc_deletearray 290 | | | | + 1.00% tcmalloc::ThreadCache::ListTooLong(tcmalloc::ThreadCache::FreeList*, unsigned long) 291 | | | | + 1.00% tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*, unsigned long, int) 292 | | | + 1.00% end 293 | | | + 1.00% end 294 | | + 3.00% BlueStore::_txc_committed_kv 295 | | | + 2.00% queue 296 | | | | + 1.00% push_back 297 | | | | | + 1.00% std::vector >::_M_emplace_back_aux 298 | | | | + 1.00% Mutex::Lock 299 | | | | + 1.00% pthread_mutex_lock 300 | | | | + 1.00% _L_lock_791 301 | | | | + 1.00% __lll_lock_wait 302 | | | + 1.00% tc_delete 303 | | + 1.00% log_state_latency 304 | | + 1.00% ceph_clock_now 305 | + 2.00% BlueStore::DeferredBatch::~DeferredBatch 306 | + 2.00% ~DeferredBatch 307 | + 2.00% ~IOContext 308 | + 2.00% ~list 309 | + 2.00% ~_List_base 310 | + 2.00% std::_List_base >::_M_clear 311 | + 2.00% destroy > 312 | + 2.00% ~_List_node 313 | + 2.00% ~aio_t 314 | + 2.00% ~list 315 | + 2.00% ~list 316 | + 2.00% ~_List_base 317 | + 2.00% std::_List_base >::_M_clear 318 | + 1.00% destroy > 319 | | + 1.00% ~_List_node 320 | | + 1.00% ~ptr 321 | | + 1.00% ceph::buffer::ptr::release 322 | | + 1.00% ceph::buffer::raw_posix_aligned::~raw_posix_aligned 323 | | + 1.00% ~raw_posix_aligned 324 | | + 1.00% ~raw 325 | | + 1.00% ~map 326 | | + 1.00% ~_Rb_tree 327 | | + 1.00% std::_Rb_tree, std::pair const, std::pair >, std::_Select1st const, std::pair > >, std::less >, std::allocator const, std::pair > > >::_M_erase 328 | | + 1.00% _M_destroy_node 329 | | + 1.00% _M_put_node 330 | | + 1.00% deallocate 331 | | + 1.00% tc_delete 332 | | + 1.00% tcmalloc::ThreadCache::ListTooLong(tcmalloc::ThreadCache::FreeList*, unsigned long) 333 | | + 1.00% tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*, unsigned long, int) 334 | + 1.00% _M_put_node 335 | + 1.00% deallocate 336 | + 1.00% tc_delete 337 | + 1.00% tcmalloc::ThreadCache::ListTooLong(tcmalloc::ThreadCache::FreeList*, unsigned long) 338 | + 1.00% tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*, unsigned long, int) 339 | 340 | Thread: 50 (bstore_kv_sync) - 100 samples 341 | 342 | + 100.00% clone 343 | + 100.00% start_thread 344 | + 100.00% BlueStore::KVSyncThread::entry 345 | + 100.00% BlueStore::_kv_sync_thread 346 | + 69.00% RocksDBStore::submit_transaction 347 | | + 69.00% RocksDBStore::submit_common 348 | | + 69.00% rocksdb::DBImpl::Write 349 | | + 69.00% rocksdb::DBImpl::WriteImpl 350 | | + 36.00% rocksdb::DBImpl::WriteToWAL 351 | | | + 36.00% rocksdb::DBImpl::WriteToWAL 352 | | | + 35.00% rocksdb::log::Writer::AddRecord 353 | | | | + 35.00% rocksdb::log::Writer::EmitPhysicalRecord 354 | | | | + 26.00% rocksdb::crc32c::ExtendImpl 355 | | | | | + 26.00% Slow_CRC32 356 | | | | + 8.00% rocksdb::WritableFileWriter::Append 357 | | | | + 8.00% rocksdb::WritableFileWriter::WriteBuffered 358 | | | | + 8.00% BlueRocksWritableFile::Append 359 | | | | + 8.00% append 360 | | | | + 8.00% append 361 | | | | + 8.00% memcpy 362 | | | | + 8.00% __memcpy_ssse3_back 363 | | | + 1.00% back 364 | | | + 1.00% end 365 | | | + 1.00% _Deque_iterator 366 | | + 31.00% rocksdb::WriteBatchInternal::InsertInto 367 | | | + 31.00% rocksdb::WriteBatch::Iterate 368 | | | + 30.00% rocksdb::MemTableInserter::PutCF 369 | | | | + 29.00% rocksdb::MemTable::Add 370 | | | | | + 21.00% rocksdb::InlineSkipList::Insert 371 | | | | | | + 21.00% rocksdb::InlineSkipList::RecomputeSpliceLevels 372 | | | | | | + 21.00% rocksdb::InlineSkipList::FindSpliceForLevel 373 | | | | | | + 20.00% KeyIsAfterNode 374 | | | | | | + 20.00% rocksdb::MemTable::KeyComparator::operator() 375 | | | | | | + 11.00% rocksdb::InternalKeyComparator::Compare 376 | | | | | | | + 9.00% rocksdb::(anonymous namespace)::BytewiseComparatorImpl::Compare 377 | | | | | | | + 9.00% compare 378 | | | | | | | + 9.00% __memcmp_sse4_1 379 | | | | | | + 7.00% GetLengthPrefixedSlice 380 | | | | | + 7.00% __memcpy_ssse3_back 381 | | | | | + 1.00% rocksdb::EncodeVarint32 382 | | | | + 1.00% SeekToColumnFamily 383 | | | | + 1.00% rocksdb::ColumnFamilyMemTablesImpl::Seek 384 | | | | + 1.00% rocksdb::ColumnFamilySet::GetDefault 385 | | | + 1.00% rocksdb::MemTableInserter::DeleteCF 386 | | | + 1.00% DeleteImpl 387 | | | + 1.00% rocksdb::MemTable::Add 388 | | | + 1.00% rocksdb::(anonymous namespace)::SkipListRep::Allocate 389 | | | + 1.00% AllocateKey 390 | | | + 1.00% rocksdb::InlineSkipList::AllocateNode 391 | | | + 1.00% StashHeight 392 | | + 1.00% rocksdb::InstrumentedMutex::Lock 393 | | + 1.00% rocksdb::DBImpl::PreprocessWrite 394 | + 24.00% RocksDBStore::submit_transaction_sync 395 | | + 24.00% RocksDBStore::submit_common 396 | | + 24.00% rocksdb::DBImpl::Write 397 | | + 24.00% rocksdb::DBImpl::WriteImpl 398 | | + 21.00% rocksdb::DBImpl::WriteToWAL 399 | | | + 21.00% rocksdb::WritableFileWriter::Sync 400 | | | + 21.00% rocksdb::WritableFileWriter::SyncInternal 401 | | | + 21.00% BlueRocksWritableFile::Sync 402 | | | + 21.00% fsync 403 | | | + 21.00% BlueFS::_fsync 404 | | | + 19.00% BlueFS::_flush_bdev_safely 405 | | | | + 19.00% BlueFS::wait_for_aio 406 | | | | + 19.00% IOContext::aio_wait 407 | | | | + 19.00% std::condition_variable::wait(std::unique_lock&) 408 | | | | + 19.00% pthread_cond_wait@@GLIBC_2.3.2 409 | | | | + 1.00% __pthread_mutex_cond_lock 410 | | | | + 1.00% _L_cond_lock_792 411 | | | | + 1.00% __lll_lock_wait 412 | | | + 2.00% BlueFS::_flush 413 | | | + 2.00% BlueFS::_flush_range 414 | | | + 2.00% KernelDevice::aio_submit 415 | | | + 2.00% aio_queue_t::submit_batch 416 | | | + 2.00% io_submit 417 | | + 2.00% rocksdb::DBImpl::MarkLogsSynced 418 | | | + 1.00% SignalAll 419 | | | + 1.00% rocksdb::port::CondVar::SignalAll 420 | | + 1.00% rocksdb::WriteBatchInternal::InsertInto 421 | | + 1.00% rocksdb::WriteBatch::Iterate 422 | | + 1.00% rocksdb::MemTableInserter::SingleDeleteCF 423 | | + 1.00% DeleteImpl 424 | | + 1.00% rocksdb::MemTable::Add 425 | | + 1.00% rocksdb::InlineSkipList::Insert 426 | | + 1.00% rocksdb::InlineSkipList::RecomputeSpliceLevels 427 | | + 1.00% rocksdb::InlineSkipList::FindSpliceForLevel 428 | | + 1.00% KeyIsAfterNode 429 | | + 1.00% rocksdb::MemTable::KeyComparator::operator() 430 | | + 1.00% rocksdb::InternalKeyComparator::Compare 431 | | + 1.00% rocksdb::(anonymous namespace)::BytewiseComparatorImpl::Compare 432 | + 2.00% std::condition_variable::wait(std::unique_lock&) 433 | | + 2.00% pthread_cond_wait@@GLIBC_2.3.2 434 | + 2.00% BlueStore::_txc_applied_kv 435 | + 1.00% operator std::__atomic_base::__int_type 436 | + 1.00% load 437 | 438 | Thread: 51 (finisher) - 0 samples 439 | 440 | 441 | Thread: 52 (dfin) - 0 samples 442 | 443 | 444 | Thread: 53 (rocksdb:bg0) - 0 samples 445 | 446 | 447 | Thread: 54 (rocksdb:bg0) - 0 samples 448 | 449 | 450 | Thread: 55 (bstore_aio) - 0 samples 451 | 452 | 453 | Thread: 56 (bstore_aio) - 0 samples 454 | 455 | 456 | Thread: 57 (bstore_aio) - 0 samples 457 | 458 | 459 | Thread: 58 (bstore_aio) - 0 samples 460 | 461 | 462 | Thread: 59 (safe_timer) - 0 samples 463 | 464 | 465 | Thread: 60 (safe_timer) - 0 samples 466 | 467 | 468 | Thread: 61 (safe_timer) - 0 samples 469 | 470 | 471 | Thread: 62 (safe_timer) - 0 samples 472 | 473 | 474 | Thread: 63 (ceph-osd) - 0 samples 475 | 476 | 477 | Thread: 64 (admin_socket) - 0 samples 478 | 479 | 480 | Thread: 65 (service) - 0 samples 481 | 482 | 483 | Thread: 66 (msgr-worker-2) - 0 samples 484 | 485 | 486 | Thread: 67 (msgr-worker-1) - 0 samples 487 | 488 | 489 | Thread: 68 (msgr-worker-0) - 0 samples 490 | 491 | 492 | Thread: 69 (log) - 0 samples 493 | ``` 494 | -------------------------------------------------------------------------------- /common.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | # Copyright (c) 2017 Mark Nelson 3 | 4 | import argparse 5 | import pickle 6 | import shutil 7 | from gdbtypes import GDBFunction, GDBThread 8 | 9 | def parse_args(args=None): 10 | parser = argparse.ArgumentParser() 11 | group = parser.add_mutually_exclusive_group(required=True) 12 | group.add_argument('-i', '--input', required=False, type=str, help='Read collected samples from this file.') 13 | group.add_argument('-p', '--pid', required=False, type=int, help='PID of the process to connect to.') 14 | 15 | parser.add_argument('-s', '--sleep', required=False, type=float, default=0.01, help='Period of time to sleep between samples in seconds.') 16 | parser.add_argument('-n', '--samples', required=False, type=int, default=1000, help='The number of samples to collect.') 17 | parser.add_argument('-m', '--match', required=False, type=str, help='A comma separated list of strings to match with threads') 18 | parser.add_argument('-x', '--exclude', required=False, type=str, help='A comma separated list of strings to match when excluding threads.') 19 | parser.add_argument('-o', '--output', required=False, type=str, help='Write collected samples to this file.') 20 | parser.add_argument('-g', '--gdb_path', required=False, type=str, default='/usr/bin/gdb', help='Path to the GDB executable.') 21 | parser.add_argument('-t', '--threshold', required=False, type=float, default=0.1, help='Ignore results below the threshold when making the callgraph.') 22 | parser.add_argument('-v', '--invert', required=False, action='store_true', help='Print inverted callgraph.') 23 | parser.add_argument('-d', '--detachattach', required=False, action='store_true', help='Use the detach/attach trace mode. Slower, but more compatible.') 24 | parser.add_argument('-w', '--max_width', required=False, type=int, default=shutil.get_terminal_size().columns, help='Set the display width (default is terminal width)') 25 | parser.add_argument('-r', '--truncate', required=False, action='store_true', help="Truncate lines to the terminal width") 26 | 27 | if args: 28 | return parser.parse_args(args) 29 | else: 30 | return parser.parse_args() 31 | 32 | def print_callgraph(ctx, threads): 33 | print(""); 34 | for thn, gdbth in sorted(threads.items()): 35 | samples = gdbth.function.get_samples(True) 36 | print("") 37 | print(("Thread: %s (%s) - %s samples " % (gdbth.num, gdbth.name, samples))) 38 | print("") 39 | gdbth.function.print_percent(ctx, "", samples, True) 40 | 41 | def print_inverted_callgraph(ctx, threads): 42 | print("") 43 | for thn, gdbth in sorted(threads.items()): 44 | samples = gdbth.function.get_samples(True) 45 | igdbth = invert_thread(gdbth) 46 | print("") 47 | print(("Thread: %s (%s) - %s samples " % (igdbth.num, igdbth.name, samples))) 48 | print("") 49 | igdbth.function.print_percent(ctx, "", samples, False) 50 | 51 | def invert_thread(gdbth): 52 | inverted_func = GDBFunction(None, 2) 53 | invert_function(gdbth.function, inverted_func) 54 | return GDBThread(gdbth.name, gdbth.num, gdbth.ptid, inverted_func) 55 | 56 | def invert_function(input_func, base_func): 57 | ret = [] 58 | 59 | # If we spent time directly in this function, add it to the base_func 60 | if input_func.count > 0: 61 | func = base_func.get_or_add_func(input_func.name) 62 | func.count += input_func.count 63 | ret.append((func, input_func.count)) 64 | 65 | # iterate over the subfunctions 66 | for subfunction in input_func.subfunctions: 67 | tree_funcs = invert_function(subfunction, base_func) 68 | 69 | for (tree_func, count) in tree_funcs: 70 | func = tree_func.get_or_add_func(input_func.name) 71 | func.count += count 72 | ret.append((func, count)) 73 | 74 | return ret 75 | 76 | def dump_threads(threads, filename): 77 | with open(filename, 'wb') as handle: 78 | pickle.dump(threads, handle, protocol=pickle.HIGHEST_PROTOCOL) 79 | 80 | def load_threads(filename): 81 | with open(filename, 'rb') as handle: 82 | return pickle.load(handle) 83 | -------------------------------------------------------------------------------- /gdbpmp.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | # Copyright (c) 2017 Mark Nelson 3 | 4 | import sys 5 | import os 6 | 7 | try: 8 | import common 9 | except ImportError: 10 | sys.path.append(os.path.dirname(__file__)) 11 | import common 12 | 13 | import gdbtypes 14 | import subprocess 15 | import signal 16 | import time 17 | 18 | def main(): 19 | ctx = common.parse_args() 20 | if ctx.input: 21 | threads = common.load_threads(ctx.input) 22 | if ctx.invert: 23 | common.print_inverted_callgraph(ctx, threads) 24 | else: 25 | common.print_callgraph(ctx, threads) 26 | elif ctx.pid: 27 | this_path = os.path.realpath(__file__) 28 | pargs = ' '.join(sys.argv[1:]) 29 | 30 | args = [ctx.gdb_path, "-q", "--ex", "source %s" % this_path, "--ex", "gdbpmp %s" % pargs] 31 | proc = subprocess.Popen(args) 32 | 33 | try: 34 | while proc.poll() is None: 35 | time.sleep(0.1) 36 | 37 | except KeyboardInterrupt: 38 | proc.send_signal(signal.SIGINT) 39 | while proc.poll() is None: 40 | time.sleep(0.1) 41 | 42 | if __name__ == "__main__": 43 | try: 44 | import tracer 45 | except ImportError: 46 | main() 47 | sys.exit(0) 48 | sys.setrecursionlimit(5000) 49 | tracer.ProfileCommand() 50 | -------------------------------------------------------------------------------- /gdbtypes.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | # Copyright (c) 2017 Mark Nelson 3 | 4 | class GDBThread: 5 | def __init__(self, name, num, ptid, function): 6 | self.name = name 7 | self.num = num 8 | self.ptid = ptid 9 | self.function = function 10 | 11 | class GDBFunction: 12 | 13 | def __init__(self, name, indent): 14 | self.name = name 15 | self.indent = indent 16 | self.subfunctions = [] 17 | 18 | # count of times we terminated here 19 | self.count = 0 20 | 21 | def add_count(self): 22 | self.count += 1 23 | 24 | def get_samples(self, include_sub): 25 | _count = self.count 26 | if include_sub: 27 | for function in self.subfunctions: 28 | _count += function.get_samples(include_sub) 29 | return _count 30 | 31 | def get_percent(self, total, include_sub): 32 | return 100.0 * self.get_samples(include_sub) / total 33 | 34 | def get_name(self): 35 | return self.name; 36 | 37 | def get_func(self, name): 38 | for function in self.subfunctions: 39 | if function.get_name() == name: 40 | return function 41 | return None 42 | 43 | def get_or_add_func(self, name): 44 | function = self.get_func(name); 45 | if function is not None: 46 | return function; 47 | function = GDBFunction(name, self.indent) 48 | self.subfunctions.append(function) 49 | return function 50 | 51 | def fprint(self, ctx, line): 52 | print(line[0:ctx.max_width]) 53 | 54 | def print_samples(self, ctx, depth, include_sub): 55 | self.fprint(("%s%s - %s" % (' ' * (self.indent * depth), self.get_samples(include_sub), self.name))) 56 | for function in self.subfunctions: 57 | function.print_samples(ctx, depth+1) 58 | 59 | def print_percent(self, ctx, prefix, total, include_sub): 60 | subfunctions = {} 61 | for function in self.subfunctions: 62 | v = function.get_percent(total, include_sub) 63 | if function.name is None: 64 | function.name = "???" 65 | if v is None: 66 | v = "???" 67 | subfunctions[function.name] = v 68 | 69 | depth = 0 70 | for name, value in sorted(list(subfunctions.items()), key= lambda kv: (kv[1], kv[0]), reverse=True): 71 | line_prefix = "%s%s%0.2f%% " % (prefix, "+ ", value) 72 | nl_prefix = "%s%s" % (prefix," " * (len(line_prefix)-len(prefix))) 73 | line = name 74 | if (ctx.max_width > 0 and ctx.max_width < len(line_prefix) + len(line)): 75 | # output line will be longer than the max width 76 | line_max = ctx.max_width - len(line_prefix); 77 | if (ctx.truncate): 78 | # truncate it 79 | self.fprint(ctx, line_prefix + line[0:line_max-3] + "...") 80 | else: 81 | # or writap it 82 | self.fprint(ctx, line_prefix + line[0:line_max]) 83 | line = line[line_max:] 84 | 85 | while (line_max > 0 and len(line) > line_max): 86 | self.fprint(ctx, nl_prefix + line[0:line_max]) 87 | line = line[line_max:] 88 | if (len(line) > 0): 89 | self.fprint(ctx, nl_prefix + line) 90 | else: 91 | # output line is shorter than the max width, so print it 92 | self.fprint(ctx, "%s%s" % (line_prefix, line)) 93 | 94 | # Do not descend below the threshold 95 | if value < ctx.threshold: 96 | continue; 97 | 98 | new_prefix = '' 99 | if depth + 1 == len(self.subfunctions): 100 | new_prefix += ' ' 101 | else: 102 | new_prefix += '|' 103 | 104 | self.get_func(name).print_percent(ctx, prefix + new_prefix, total, include_sub) 105 | depth += 1 106 | 107 | def add_frame(self, frame): 108 | if frame is None: 109 | self.count += 1 110 | else: 111 | function = self.get_or_add_func(frame.name()) 112 | function.add_frame(frame.older()) 113 | 114 | def inverse_add_frame(self, frame): 115 | if frame is None: 116 | self.count += 1 117 | else: 118 | function = self.get_or_add_func(frame.name()) 119 | function.inverse_add_frame(frame.newer()) 120 | 121 | -------------------------------------------------------------------------------- /tracer.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | # -*- coding: utf-8 -*- 3 | # Copyright (c) 2012 Mak Nazečić-Andrlon 4 | # Copyright (c) 2017 Mark Nelson 5 | # 6 | # Permission is hereby granted, free of charge, to any person obtaining a copy 7 | # of this software and associated documentation files (the "Software"), to deal 8 | # in the Software without restriction, including without limitation the rights 9 | # to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 10 | # copies of the Software, and to permit persons to whom the Software is 11 | # furnished to do so, subject to the following conditions: 12 | # 13 | # The above copyright notice and this permission notice shall be included in all 14 | # copies or substantial portions of the Software. 15 | # 16 | # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 17 | # IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 18 | # FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 19 | # AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 20 | # LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 21 | # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 22 | # SOFTWARE. 23 | 24 | from time import sleep 25 | import gdb 26 | import os 27 | import signal 28 | import pickle 29 | import common 30 | from gdbtypes import GDBThread, GDBFunction 31 | 32 | class ProfileCommand(gdb.Command): 33 | def __init__(self): 34 | super(ProfileCommand, self).__init__("gdbpmp", 35 | gdb.COMMAND_RUNNING, gdb.COMPLETE_NONE, True) 36 | self.threads = {} 37 | 38 | def invoke(self, argument, from_tty): 39 | detatch_and_kill = 0 40 | self.dont_repeat() 41 | ctx = common.parse_args(gdb.string_to_argv(argument)) 42 | if ctx.input: 43 | threads = common.load_threads(ctx.input) 44 | common.print_callgraph(threads) 45 | return 46 | 47 | gdb.execute("set print thread-events off") 48 | gdb.execute("set print inferior-events off") 49 | gdb.execute("set pagination off") 50 | 51 | if ctx.pid and ctx.pid != gdb.selected_inferior().pid: 52 | detach_and_kill = 1 53 | gdb.write("Attaching to process %d..." % ctx.pid) 54 | gdb.flush(gdb.STDOUT) 55 | if gdb.selected_inferior().pid != 0: 56 | gdb.execute("detach", to_string=True) 57 | os.kill(ctx.pid, signal.SIGSTOP) 58 | gdb.execute("attach %d" % ctx.pid, to_string=True) 59 | os.kill(ctx.pid, signal.SIGCONT) 60 | gdb.execute("continue", to_string=True) 61 | gdb.write("Done.\n") 62 | gdb.flush(gdb.STDOUT) 63 | 64 | def detach_attach(): 65 | pid = gdb.selected_inferior().pid 66 | os.kill(pid, signal.SIGSTOP) # Make sure the process does nothing until 67 | # it's reattached. 68 | gdb.execute("detach", to_string=True) 69 | os.kill(pid, signal.SIGCONT) 70 | sleep(ctx.sleep) 71 | os.kill(pid, signal.SIGSTOP) 72 | gdb.execute("attach %d" % pid, to_string=True) 73 | os.kill(pid, signal.SIGCONT) 74 | gdb.execute("continue", to_string=True) 75 | 76 | def breaking_continue_handler(event): 77 | sleep(ctx.sleep) 78 | os.kill(gdb.selected_inferior().pid, signal.SIGINT) 79 | 80 | samples = 0 81 | gdb.write("Gathering Samples") 82 | gdb.flush(gdb.STDOUT) 83 | try: 84 | for i in range(0, ctx.samples): 85 | # Some processes handle signals in strange ways. For those 86 | # processes we may have to fully detach from the process, 87 | # sleep, and then reattach. This is quite slow. For other 88 | # processes we may be able to tell gdb to continue, sleep, 89 | # and then send SIGINT to let GDB stop the process. 90 | if ctx.detachattach: 91 | detach_attach() 92 | else: 93 | gdb.events.cont.connect(breaking_continue_handler) 94 | gdb.execute("continue", to_string=True) 95 | gdb.events.cont.disconnect(breaking_continue_handler) 96 | for inf in gdb.inferiors(): 97 | inum = inf.num 98 | for th in inf.threads(): 99 | thn = th.num 100 | if thn not in self.threads: 101 | f = GDBFunction(None, 2) 102 | self.threads[thn] = GDBThread(th.name, thn, th.ptid, f) 103 | if ctx.match and not any(m in th.name for m in ctx.match.split(',')): 104 | pass 105 | elif ctx.exclude and any(m in th.name for m in ctx.exclude.split(',')): 106 | pass 107 | else: 108 | th.switch() 109 | frame = gdb.newest_frame() 110 | while (frame.older() != None): 111 | frame = frame.older() 112 | self.threads[thn].function.inverse_add_frame(frame) 113 | 114 | samples += 1 115 | gdb.write(".") 116 | gdb.flush(gdb.STDOUT) 117 | except KeyboardInterrupt: 118 | pass 119 | finally: 120 | print(("\nProfiling complete with %d samples." % samples)) 121 | if ctx.output: 122 | common.dump_threads(self.threads, ctx.output) 123 | else: 124 | common.print_callgraph(self.threads) 125 | 126 | if detach_and_kill: 127 | gdb.execute("detach", to_string=True) 128 | gdb.execute("quit") 129 | else: 130 | pid = gdb.selected_inferior().pid 131 | os.kill(pid, signal.SIGSTOP) # Make sure the process does nothing until 132 | # it's reattached. 133 | gdb.execute("detach", to_string=True) 134 | gdb.execute("attach %d" % pid, to_string=True) 135 | os.kill(pid, signal.SIGCONT) 136 | gdb.execute("continue", to_string=True) 137 | 138 | --------------------------------------------------------------------------------