├── .gitignore ├── LICENSE ├── README.md ├── scripts ├── callgraph ├── callgraph.stp ├── common.py ├── memtrace ├── memtrace.stp ├── py-callgraph ├── py-callgraph.stp ├── sample └── sample.stp ├── static └── flamegraph_expanded.png ├── tapset ├── python2 │ └── util.stp └── python3 │ └── util.stp └── tests ├── bootstrap ├── example.py └── smoketest /.gitignore: -------------------------------------------------------------------------------- 1 | tapset/python2/py_library.stpm 2 | tapset/python3/py_library.stpm 3 | */*.pyc 4 | tests/testenv/ 5 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | The MIT License (MIT) 2 | 3 | Copyright (c) 2016 Eben Freeman 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | 2 | Do you need to profile a Python program that spends a lot of its time in C extension code? Debug a memory leak? Or understand a particular codepath in an unfamiliar codebase? 3 | 4 | [SystemTap](https://sourceware.org/systemtap/) is a powerful and flexible Linux tracing tool. It can be used to great effect to solve these problems and more. However, some leg work is required. Here are a few utilities to help trace Python programs. 5 | 6 | (I gave a talk about this at PyBay 2016, discussing some of the background, motivation, and implementation in depth. [Slides here](https://speakerdeck.com/emfree/python-tracing-superpowers-with-systems-tools)) 7 | 8 | # Getting started 9 | 10 | This will only work with CPython on Linux. Python 2 and 3 are both supported. 11 | 12 | 1. Install SystemTap. I recommend just building the latest version from source. 13 | ``` 14 | sudo apt-get install -y gcc g++ gettext libdw-dev linux-headers-`uname -r` 15 | git clone git://sourceware.org/git/systemtap.git 16 | cd systemtap 17 | ./configure && make && sudo make install 18 | ``` 19 | 20 | 2. You'll need to run a CPython binary that contains debugging symbols. Many distributions ship one; `apt-get install python-dbg` will give you a `python-dbg` binary on Debian or Ubuntu.[1] 21 | If your program relies on any C extension modules, you'll need to rebuild those against the new binary. If you're using `virtualenv`, this is straightforward: 22 | ``` 23 | virtualenv -p /usr/bin/python-dbg venv 24 | . venv/bin/activate 25 | # install your project's dependencies. 26 | ``` 27 | 28 | 3. For profiling support, clone [https://github.com/brendangregg/FlameGraph](https://github.com/brendangregg/FlameGraph) and add it to your `$PATH`. 29 | 30 | 31 | In general, SystemTap scripts need to be run as root. 32 | 33 | 34 | # Careful! 35 | Tracing overhead can vary dramatically! SystemTap bugs could crash your system! Test in a safe environment before you use any of this in production! 36 | 37 | 38 | # Examples 39 | 40 | 41 | ## CPU profiling 42 | 43 | ![A flamegraph showing user, interpreter and kernel stacks.](/static/flamegraph_expanded.png) 44 | -- An example flame graph combining Python, interpreter, and kernel call stacks. 45 | 46 | 47 | ### Basic Usage 48 | 49 | To profile a running process $PID for 60 seconds, run 50 | 51 | ``` 52 | scripts/sample -x $PID -t 60 | flamegraph.pl --colors=java > profile.svg 53 | ``` 54 | 55 | If you're using Python 3, pass `--py3`: 56 | 57 | ``` 58 | scripts/sample --py3 -x $PID -t 60 | flamegraph.pl --colors=java > profile.svg 59 | ``` 60 | 61 | ### Rationale 62 | 63 | There are a [number](https://github.com/joerick/pyinstrument) [of](https://github.com/bdarnell/plop) [sampling](https://github.com/vmprof/vmprof-python) [profiler](https://github.com/nylas/nylas-perftools) [implementations](https://github.com/what-studio/profiling) available for Python, and it's easy to roll your own if you don't like any of them. The most common strategy is to sample the Python call stack from within the interpreter. But this approach has two limitations: 64 | 65 | * Calls into C extension code are largely invisible 66 | * You'll need to integrate the profiler into your application code. 67 | 68 | In contrast, tools like Linux `perf` can profile unmodified native binaries. But using `perf` on a Python program will only give you C call stacks in the interpreter, and little insight into what your _Python_ code is doing. 69 | 70 | With SystemTap, we can get something of the best of both worlds. We don't need to change any application code, and the resultant profile transparently combines native and Python callstacks. 71 | 72 | 73 | ## Memory allocation tracing 74 | 75 | Run 76 | ``` 77 | scripts/memtrace -x $PID -t 60 78 | ``` 79 | to trace all Python object memory allocations for 60 seconds. At the end, for all surviving objects, the allocation timestamp and traceback will be printed. This can help track down memory leaks. 80 | 81 | 82 | 83 | ## C Function execution tracing 84 | 85 | Run 86 | ``` 87 | scripts/callgraph -x $PID -t $TRIGGER -n 20 88 | ``` 89 | to trace 20 executions of function $TRIGGER, and print a microsecond-timed callgraph. $TRIGGER should be a C function in the interpreter (so this is pretty low level). 90 | 91 | 92 | --- 93 | 94 | 95 | [1] The "debug" Python binary is built with `--with-pydebug`, which also builds the interpreter with `-O0`, and enables the debug memory allocator. Those changes can negatively affect application performance, when all we really need here is a binary with DWARF debugging symbols in it. If this is a factor, consider building your own Python binary instead. E.g. 96 | ``` 97 | export $VER=2.7.11 98 | wget https://www.python.org/ftp/python/$VER/Python-$VER.tar.xz 99 | tar -xvJf Python-$VER.tar.xz 100 | cd Python-$VER 101 | ./configure CFLAGS='-g -fno-omit-frame-pointer' --prefix=/opt/python-$VER-dbg 102 | make 103 | sudo make install 104 | ``` 105 | -------------------------------------------------------------------------------- /scripts/callgraph: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | from __future__ import print_function 3 | import argparse 4 | import subprocess 5 | import sys 6 | from common import abspath, build_stap_args, gen_tapset_macros 7 | 8 | 9 | def main(): 10 | argparser = argparse.ArgumentParser() 11 | argparser.add_argument('-x', '--pid', help='PID to profile', required=True) 12 | argparser.add_argument('-n', '--ncalls', 13 | help='Number of times to record function execution', 14 | default='20') 15 | argparser.add_argument('-t', '--trigger', help='Trigger function', 16 | required=True) 17 | argparser.add_argument('--py3', action='store_true', 18 | help='Pass when profiling Python 3 programs') 19 | argparser.add_argument('-v', '--verbose', action='store_true') 20 | args, extra_args = argparser.parse_known_args() 21 | main_pid = str(args.pid) 22 | 23 | if args.py3: 24 | tapset_dir = abspath('../tapset/python3') 25 | else: 26 | tapset_dir = abspath('../tapset/python2') 27 | 28 | gen_tapset_macros(main_pid, tapset_dir) 29 | 30 | stap_cmd = ['stap', abspath('callgraph.stp'), args.trigger, args.ncalls, 31 | '-I', tapset_dir] 32 | 33 | stap_cmd.extend(build_stap_args(main_pid)) 34 | 35 | limits=['-D', 'MAXSTRINGLEN=4096', '-D', 'MAXBACKTRACE=200', 36 | '-D', 'MAXMAPENTRIES=10240'] 37 | 38 | stap_cmd.extend(limits) 39 | stap_cmd.extend(extra_args) 40 | 41 | if args.verbose: 42 | print(" ".join(stap_cmd)) 43 | p = subprocess.Popen(stap_cmd) 44 | p.wait() 45 | if p.returncode != 0: 46 | print("Error running stap script (exit code {}). " 47 | "You may need to pass --py3.".format(p.returncode), file=sys.stderr) 48 | 49 | 50 | if __name__ == '__main__': 51 | main() 52 | -------------------------------------------------------------------------------- /scripts/callgraph.stp: -------------------------------------------------------------------------------- 1 | #! /usr/bin/env stap 2 | 3 | 4 | global ncalls = 0; 5 | global tracing = 0; 6 | global maxcalls = $2; 7 | 8 | function trace(entry_p) { 9 | if (tracing == 1) { 10 | printf("%s%s%s\n", 11 | thread_indent (entry_p), 12 | (entry_p>0?"->":"<-"), 13 | ppfunc ()) 14 | } 15 | } 16 | 17 | probe begin { println("tracing"); } 18 | 19 | probe process.function("*").call { 20 | trace(1) 21 | } 22 | 23 | probe process.function("*").return { 24 | trace(-1) 25 | } 26 | 27 | 28 | probe process.function(@1).call { 29 | ncalls++; 30 | tracing = 1; 31 | println("\n\n"); 32 | } 33 | 34 | probe process.function(@1).return { 35 | tracing = 0; 36 | if (ncalls == maxcalls) { 37 | exit(); 38 | } 39 | } 40 | -------------------------------------------------------------------------------- /scripts/common.py: -------------------------------------------------------------------------------- 1 | import os 2 | import re 3 | import subprocess 4 | 5 | shared_library_re = re.compile("\.so[0-9\.]*$") 6 | libpython_re = re.compile(r'\t.* => (.*libpython.*) \(0x') 7 | 8 | 9 | def abspath(f): 10 | return os.path.join(os.path.dirname(__file__), f) 11 | 12 | 13 | def gen_tapset_macros(pid, tapset_dir): 14 | binary_path = os.path.realpath('/proc/{}/exe'.format(pid)) 15 | python_lib_path = binary_path 16 | lines = subprocess.check_output(["ldd", binary_path]).splitlines() 17 | for line in lines: 18 | m = libpython_re.match(line) 19 | if not m: 20 | continue 21 | python_lib_path = m.group(1) 22 | break 23 | 24 | with open(os.path.join(tapset_dir, 'py_library.stpm'), 'w') as f: 25 | f.write('@define PYTHON_LIBRARY %( "{}" %)'.format(python_lib_path)) 26 | 27 | 28 | def child_pids(main_pid): 29 | children = subprocess.Popen(['pgrep', '--parent', main_pid], 30 | stdout=subprocess.PIPE).communicate()[0] 31 | return children.splitlines() 32 | 33 | 34 | def shared_libs(main_pid, child_pids): 35 | binary_path = os.path.realpath('/proc/{}/exe'.format(main_pid)) 36 | shared_libs = {binary_path, "kernel"} 37 | 38 | pids = [main_pid] + child_pids 39 | 40 | # Try to automatically load symbols for any shared libraries 41 | # the process and corresponding subprocesses (if any) are using. 42 | for pid in pids: 43 | with open("/proc/{}/maps".format(pid), "r") as fhandler: 44 | for line in fhandler: 45 | line = line.strip() 46 | cols = line.split(None, 5) 47 | if len(cols) != 6: 48 | continue 49 | lib_path = cols[5] 50 | if shared_library_re.findall(lib_path): 51 | shared_libs.add(lib_path) 52 | return shared_libs 53 | 54 | 55 | def build_stap_args(main_pid): 56 | args = ['-x', main_pid] 57 | 58 | children = child_pids(main_pid) 59 | for pid in children: 60 | args.extend(("-x", pid)) 61 | 62 | for lib in shared_libs(main_pid, children): 63 | if lib: 64 | args.extend(('-d', lib)) 65 | return args 66 | -------------------------------------------------------------------------------- /scripts/memtrace: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | from __future__ import print_function 3 | import argparse 4 | import subprocess 5 | import sys 6 | from common import abspath, build_stap_args, gen_tapset_macros 7 | 8 | 9 | def main(): 10 | argparser = argparse.ArgumentParser() 11 | argparser.add_argument('-x', '--pid', help='PID to profile', required=True) 12 | argparser.add_argument('-t', '--time', default='30', 13 | help='Time to run tracer, in seconds') 14 | argparser.add_argument('--py3', action='store_true', 15 | help='Pass when profiling Python 3 programs') 16 | argparser.add_argument('-v', '--verbose', action='store_true') 17 | args, extra_args = argparser.parse_known_args() 18 | main_pid = str(args.pid) 19 | 20 | stapscript = abspath('memtrace.stp') 21 | if args.py3: 22 | tapset_dir = abspath('../tapset/python3') 23 | else: 24 | tapset_dir = abspath('../tapset/python2') 25 | 26 | gen_tapset_macros(main_pid, tapset_dir) 27 | 28 | stap_cmd = ['stap', abspath('memtrace.stp'), args.time, '-I', tapset_dir] 29 | 30 | stap_cmd.extend(build_stap_args(main_pid)) 31 | 32 | limits=['-D', 'MAXSTRINGLEN=4096', '-D', 'MAXBACKTRACE=200', 33 | '-D', 'MAXMAPENTRIES=10240'] 34 | 35 | stap_cmd.extend(limits) 36 | stap_cmd.extend(extra_args) 37 | 38 | if args.verbose: 39 | print(" ".join(stap_cmd)) 40 | p = subprocess.Popen(stap_cmd) 41 | p.wait() 42 | if p.returncode != 0: 43 | print("Error running stap script (exit code {}). " 44 | "You may need to pass --py3.".format(p.returncode), file=sys.stderr) 45 | 46 | 47 | if __name__ == '__main__': 48 | main() 49 | -------------------------------------------------------------------------------- /scripts/memtrace.stp: -------------------------------------------------------------------------------- 1 | global alloc_bt 2 | global alloc_time 3 | 4 | probe process.function("_PyObject_Malloc").return { 5 | alloc_bt[$return] = sprint_merged_stack() 6 | alloc_time[$return] = gettimeofday_ms() 7 | } 8 | 9 | probe process.function("_PyObject_Free").return { 10 | delete alloc_bt[$p] 11 | delete alloc_time[$p] 12 | } 13 | 14 | probe timer.s($1) { 15 | foreach (ptr in alloc_bt) { 16 | printf("%s %d\n", alloc_bt[ptr], alloc_time[ptr]) 17 | } 18 | exit() 19 | } 20 | -------------------------------------------------------------------------------- /scripts/py-callgraph: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | from __future__ import print_function 3 | import argparse 4 | import subprocess 5 | import sys 6 | from common import abspath, build_stap_args, gen_tapset_macros 7 | 8 | 9 | def main(): 10 | argparser = argparse.ArgumentParser() 11 | argparser.add_argument('-x', '--pid', help='PID to profile', required=True) 12 | argparser.add_argument('-n', '--ncalls', 13 | help='Number of times to record function execution', 14 | default='20') 15 | argparser.add_argument('-t', '--trigger', help='Trigger function') 16 | argparser.add_argument('--py3', action='store_true', 17 | help='Pass when profiling Python 3 programs') 18 | argparser.add_argument('-v', '--verbose', action='store_true') 19 | args, extra_args = argparser.parse_known_args() 20 | main_pid = str(args.pid) 21 | 22 | if args.py3: 23 | tapset_dir = abspath('../tapset/python3') 24 | else: 25 | tapset_dir = abspath('../tapset/python2') 26 | 27 | gen_tapset_macros(main_pid, tapset_dir) 28 | 29 | trigger = args.trigger or '' 30 | 31 | stap_cmd = ['stap', abspath('py-callgraph.stp'), args.ncalls, trigger, 32 | '-I', tapset_dir] 33 | 34 | stap_cmd.extend(build_stap_args(main_pid)) 35 | 36 | limits=['-D', 'MAXSTRINGLEN=4096', '-D', 'MAXBACKTRACE=200', 37 | '-D', 'MAXMAPENTRIES=10240'] 38 | 39 | stap_cmd.extend(limits) 40 | stap_cmd.extend(extra_args) 41 | 42 | if args.verbose: 43 | print(" ".join(stap_cmd)) 44 | p = subprocess.Popen(stap_cmd) 45 | p.wait() 46 | if p.returncode != 0: 47 | print("Error running stap script (exit code {}). " 48 | "You may need to pass --py3.".format(p.returncode), file=sys.stderr) 49 | 50 | 51 | if __name__ == '__main__': 52 | main() 53 | -------------------------------------------------------------------------------- /scripts/py-callgraph.stp: -------------------------------------------------------------------------------- 1 | #! /usr/bin/env stap 2 | 3 | 4 | global ncalls = 0 5 | global tracing = 0 6 | global maxcalls = $1 7 | 8 | probe begin { 9 | if ($# < 2) { 10 | printf("nargs %d\n", $#) 11 | tracing = 1 12 | } 13 | } 14 | 15 | probe process.function("PyEval_EvalFrameEx").call { 16 | func = func_name($f) 17 | %( $# > 1 %? 18 | if (func == @2) { 19 | tracing = 1 20 | ncalls++ 21 | } 22 | %) 23 | if (tracing == 1) { 24 | printf("%s%s%s\n", thread_indent(1), "->", func) 25 | } 26 | } 27 | 28 | probe process.function("PyEval_EvalFrameEx").return { 29 | func = func_name($f) 30 | if (tracing == 1) { 31 | printf("%s%s%s\n", thread_indent(-1), "<-", func) 32 | } 33 | 34 | %( $# > 1 %? 35 | if (func == @2) { 36 | tracing = 0 37 | if (ncalls == maxcalls) { 38 | exit() 39 | } 40 | } 41 | %) 42 | } 43 | -------------------------------------------------------------------------------- /scripts/sample: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | from __future__ import print_function 3 | import argparse 4 | import subprocess 5 | import sys 6 | from common import abspath, build_stap_args, gen_tapset_macros 7 | 8 | 9 | def main(): 10 | argparser = argparse.ArgumentParser() 11 | argparser.add_argument('-x', '--pid', help='PID to profile', required=True) 12 | argparser.add_argument('-t', '--time', default='30', 13 | help='Time to run profiler, in seconds') 14 | argparser.add_argument('--py3', action='store_true', 15 | help='Pass when profiling Python 3 programs') 16 | argparser.add_argument('-v', '--verbose', action='store_true') 17 | args, extra_args = argparser.parse_known_args() 18 | main_pid = str(args.pid) 19 | 20 | if args.py3: 21 | tapset_dir = abspath('../tapset/python3') 22 | else: 23 | tapset_dir = abspath('../tapset/python2') 24 | 25 | gen_tapset_macros(main_pid, tapset_dir) 26 | 27 | stap_cmd = ['stap', abspath('sample.stp'), args.time, '-I', tapset_dir] 28 | 29 | stap_cmd.extend(build_stap_args(main_pid)) 30 | 31 | limits = ['-D', 'MAXSTRINGLEN=4096', '-D', 'MAXBACKTRACE=200', 32 | '-D', 'MAXMAPENTRIES=10240'] 33 | 34 | stap_cmd.extend(limits) 35 | stap_cmd.extend(extra_args) 36 | 37 | if args.verbose: 38 | print(" ".join(stap_cmd)) 39 | p = subprocess.Popen(stap_cmd) 40 | p.wait() 41 | if p.returncode != 0: 42 | print("Error running stap script (exit code {}). " 43 | "You may need to pass --py3." 44 | "".format(p.returncode), file=sys.stderr) 45 | 46 | 47 | if __name__ == '__main__': 48 | main() 49 | -------------------------------------------------------------------------------- /scripts/sample.stp: -------------------------------------------------------------------------------- 1 | global stack_agg 2 | 3 | probe perf.sw.task_clock.process { 4 | stack_agg[sprint_merged_stack()] <<< 1; 5 | } 6 | 7 | 8 | probe timer.s($1) { 9 | foreach (st in stack_agg) { 10 | printf("%s %d\n", st, @count(stack_agg[st])); 11 | } 12 | exit(); 13 | } 14 | -------------------------------------------------------------------------------- /static/flamegraph_expanded.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/emfree/systemtap-python-tools/badc02f886cd738b3afd1ba143e6b84e6408b280/static/flamegraph_expanded.png -------------------------------------------------------------------------------- /tapset/python2/util.stp: -------------------------------------------------------------------------------- 1 | global pystack; 2 | global ustack; 3 | 4 | function extract_string(py_string) { 5 | return user_string(@cast(py_string, "PyStringObject", @PYTHON_LIBRARY)->ob_sval); 6 | } 7 | 8 | 9 | function func_name (frame) { 10 | return extract_string(@cast(frame, "PyFrameObject", @PYTHON_LIBRARY)->f_code->co_name); 11 | } 12 | 13 | function filename (frame) { 14 | return extract_string(@cast(frame, "PyFrameObject", @PYTHON_LIBRARY)->f_code->co_filename); 15 | } 16 | 17 | 18 | function unpack_pystack () { 19 | delete pystack 20 | i = 0; 21 | frame = current_python_frame() 22 | while (frame != 0) { 23 | pystack[i] = sprintf("%s(%s)", func_name(frame), filename(frame)); 24 | frame = @cast(frame, "PyFrameObject", @PYTHON_LIBRARY)->f_back; 25 | i++; 26 | } 27 | } 28 | 29 | function unpack_ustack () { 30 | delete ustack; 31 | i = 0; 32 | 33 | // try throwing the kernel stack in there too 34 | if (!user_mode()) { 35 | kbt = backtrace(); 36 | addr = tokenize(kbt, " "); 37 | while (addr != "") { 38 | ustack[i] = symname(strtol(addr, 16)) . "_[k]"; 39 | i++; 40 | addr = tokenize("", " "); 41 | } 42 | } 43 | 44 | bt = ubacktrace(); 45 | addr = tokenize(bt, " "); 46 | while (addr != "") { 47 | ustack[i] = usymname(strtol(addr, 16)); 48 | i++; 49 | addr = tokenize("", " "); 50 | } 51 | } 52 | 53 | function current_python_frame() { 54 | pythreadstate = @var("_PyThreadState_Current@Python/pystate.c", @PYTHON_LIBRARY); 55 | if (pythreadstate == 0) { 56 | return 0 57 | } 58 | return @cast(pythreadstate, "PyThreadState", @PYTHON_LIBRARY)->frame; 59 | } 60 | 61 | 62 | function sprint_merged_stack () { 63 | unpack_pystack(); 64 | unpack_ustack(); 65 | merged_stack = ""; 66 | pystack_idx = 0; 67 | foreach(idx+ in ustack) { 68 | if (ustack[idx] == "PyEval_EvalFrameEx") { 69 | merged_stack = ";" . pystack[pystack_idx] . merged_stack; 70 | pystack_idx++; 71 | } else { 72 | merged_stack = ";" . ustack[idx] . merged_stack; 73 | } 74 | } 75 | return merged_stack; 76 | } 77 | 78 | function sprint_pystack () { 79 | unpack_pystack(); 80 | stack = ""; 81 | foreach (idx+ in pystack) { 82 | stack = ";" . pystack[idx] . stack; 83 | } 84 | return stack; 85 | } 86 | -------------------------------------------------------------------------------- /tapset/python3/util.stp: -------------------------------------------------------------------------------- 1 | global pystack; 2 | global ustack; 3 | 4 | 5 | function py3_extract_string(py_unicodeobject) { 6 | // The actual character buffer is stored right after the PyASCIIObject 7 | // struct. See Include/unicodeobject.h in the Python source for details. 8 | // TODO: assuming ASCII here 9 | data = &@cast(py_unicodeobject, "PyASCIIObject", @PYTHON_LIBRARY)[1]; 10 | return user_string(data); 11 | } 12 | 13 | 14 | function func_name (frame) { 15 | return py3_extract_string(@cast(frame, "PyFrameObject", @PYTHON_LIBRARY)->f_code->co_name); 16 | } 17 | 18 | function filename (frame) { 19 | return py3_extract_string(@cast(frame, "PyFrameObject", @PYTHON_LIBRARY)->f_code->co_filename); 20 | } 21 | 22 | 23 | function unpack_pystack () { 24 | delete pystack; 25 | i = 0; 26 | frame = current_python_frame(); 27 | while (frame != 0) { 28 | pystack[i] = sprintf("%s(%s)", func_name(frame), filename(frame)); 29 | frame = @cast(frame, "PyFrameObject", @PYTHON_LIBRARY)->f_back; 30 | i++; 31 | } 32 | } 33 | 34 | function unpack_ustack () { 35 | delete ustack; 36 | i = 0; 37 | 38 | // try throwing the kernel stack in there too 39 | if (!user_mode()) { 40 | kbt = backtrace(); 41 | addr = tokenize(kbt, " "); 42 | while (addr != "") { 43 | ustack[i] = symname(strtol(addr, 16)) . "_[k]"; 44 | i++; 45 | addr = tokenize("", " "); 46 | } 47 | } 48 | 49 | bt = ubacktrace(); 50 | addr = tokenize(bt, " "); 51 | while (addr != "") { 52 | ustack[i] = usymname(strtol(addr, 16)); 53 | i++; 54 | addr = tokenize("", " "); 55 | } 56 | 57 | } 58 | 59 | 60 | function current_python_frame() { 61 | addr = &@var("_PyThreadState_Current@Python/pystate.c", @PYTHON_LIBRARY); 62 | v = @cast(addr, "_Py_atomic_address", @PYTHON_LIBRARY)->_value; 63 | if (v == 0) { 64 | // This only works if the GIL has been initialized :( 65 | addr = &@var("gil_last_holder@Python/ceval.c", @PYTHON_LIBRARY); 66 | v = @cast(addr, "_Py_atomic_address", @PYTHON_LIBRARY)->_value; 67 | } 68 | frame = @cast(v, "PyThreadState", @PYTHON_LIBRARY)->frame; 69 | return frame; 70 | } 71 | 72 | 73 | function sprint_merged_stack () { 74 | unpack_ustack(); 75 | try { 76 | unpack_pystack(); 77 | have_pystack = 1; 78 | } catch { 79 | have_pystack = 0; 80 | } 81 | merged_stack = ""; 82 | pystack_idx = 0; 83 | foreach(idx+ in ustack) { 84 | if ((ustack[idx] == "PyEval_EvalFrameEx") && have_pystack) { 85 | merged_stack = ";" . pystack[pystack_idx] . merged_stack; 86 | pystack_idx++; 87 | } else { 88 | merged_stack = ";" . ustack[idx] . merged_stack; 89 | } 90 | } 91 | return merged_stack; 92 | } 93 | 94 | function sprint_pystack () { 95 | unpack_pystack(); 96 | stack = ""; 97 | foreach (idx+ in pystack) { 98 | stack = ";" . pystack[idx] . stack; 99 | } 100 | return stack; 101 | } 102 | -------------------------------------------------------------------------------- /tests/bootstrap: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | set -e 3 | 4 | apt-get install -y python-dbg python3-dbg virtualenv gcc g++ libdw-dev linux-headers-`uname -r` 5 | 6 | if ! which stap 7 | then 8 | git clone git://sourceware.org/git/systemtap.git /tmp/systemtap 9 | cd /tmp/systemtap 10 | ./configure && make && make install 11 | fi 12 | 13 | virtualenv -p python-dbg testenv 14 | -------------------------------------------------------------------------------- /tests/example.py: -------------------------------------------------------------------------------- 1 | def callee_a(): 2 | pass 3 | 4 | def callee_b(): 5 | callee_c() 6 | 7 | def callee_c(): 8 | pass 9 | 10 | def caller(): 11 | callee_a() 12 | callee_b() 13 | callee_a() 14 | 15 | while True: 16 | caller() 17 | -------------------------------------------------------------------------------- /tests/smoketest: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | set -e 3 | 4 | trap 'kill $(jobs -p)' EXIT 5 | 6 | testenv/bin/python example.py & 7 | PID=$! 8 | ../scripts/sample -t 1 -x $PID 9 | --------------------------------------------------------------------------------