├── debian ├── compat ├── source │ └── format ├── install ├── .gitignore ├── rules ├── control ├── changelog └── copyright ├── .clang-format ├── m4 ├── .gitignore ├── ax_append_flag.m4 └── ax_check_compile_flag.m4 ├── autogen.sh ├── Makefile.am ├── travis └── install-build-deps.sh ├── .gitignore ├── tests ├── sleep.sh ├── sleeper.py ├── exit_early.py ├── dijkstra.py └── test_end_to_end.py ├── src ├── frob2.cc ├── frob3.cc ├── frame.cc ├── aslr.h ├── thread.cc ├── Makefile.am ├── posix.h ├── version.h ├── exc.h ├── namespace.h ├── ptrace.h ├── pyfrob.h ├── aslr.cc ├── thread.h ├── frame.h ├── posix.cc ├── namespace.cc ├── ptrace.cc ├── symbol.h ├── pyfrob.cc ├── symbol.cc ├── frob.cc └── pyflame.cc ├── .travis.yml ├── runtests.sh ├── pyflame.man ├── configure.ac ├── utils └── flame-chart-json ├── LICENSE └── README.md /debian/compat: -------------------------------------------------------------------------------- 1 | 9 2 | -------------------------------------------------------------------------------- /.clang-format: -------------------------------------------------------------------------------- 1 | BasedOnStyle: Google 2 | -------------------------------------------------------------------------------- /debian/source/format: -------------------------------------------------------------------------------- 1 | 3.0 (native) 2 | -------------------------------------------------------------------------------- /m4/.gitignore: -------------------------------------------------------------------------------- 1 | libtool.m4 2 | lt*.m4 3 | -------------------------------------------------------------------------------- /autogen.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | autoreconf --install 3 | -------------------------------------------------------------------------------- /debian/install: -------------------------------------------------------------------------------- 1 | usr/bin 2 | utils/flame-chart-json usr/bin 3 | -------------------------------------------------------------------------------- /debian/.gitignore: -------------------------------------------------------------------------------- 1 | *.debhelper 2 | *.log 3 | *.substvars 4 | *-dbg/ 5 | autoreconf* 6 | files 7 | pyflame/ 8 | tmp/ 9 | 10 | -------------------------------------------------------------------------------- /Makefile.am: -------------------------------------------------------------------------------- 1 | SUBDIRS = src 2 | EXTRA_DIST = autogen.sh LICENSE README.md 3 | ACLOCAL_AMFLAGS = -I m4 4 | 5 | man1_MANS = pyflame.man 6 | 7 | clean-local: 8 | rm -f core.* pyflame 9 | 10 | test: 11 | bash runtests.sh 12 | -------------------------------------------------------------------------------- /travis/install-build-deps.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | sudo apt-get install autotools-dev g++ pkg-config 3 | 4 | if [ "$PYTHONVERSION" = "python3" ]; then 5 | sudo apt-get install python3-dev 6 | else 7 | sudo apt-get install python-dev 8 | fi 9 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | *.a 2 | *.la 3 | *.lo 4 | *.o 5 | *.svg 6 | .cache/ 7 | Makefile 8 | __pycache__/ 9 | libtool 10 | src/pyflame 11 | *.m4 12 | *.in 13 | autom4te.cache/ 14 | build-aux/ 15 | *.log 16 | config.status 17 | configure 18 | .deps/ 19 | *~ 20 | src/stamp-h1 21 | src/config.h -------------------------------------------------------------------------------- /debian/rules: -------------------------------------------------------------------------------- 1 | #!/usr/bin/make -f 2 | # -*- makefile -*- 3 | # Sample debian/rules that uses debhelper. 4 | # This file was originally written by Joey Hess and Craig Small. 5 | # As a special exception, when this file is copied by dh-make into a 6 | # dh-make output file, you may use that output file without restriction. 7 | # This special exception was added by Craig Small in version 0.37 of dh-make. 8 | 9 | # Uncomment this to turn on verbose mode. 10 | #export DH_VERBOSE=1 11 | 12 | %: 13 | dh $@ --with autoreconf 14 | 15 | # Need to make this work on Precise somehow... 16 | override_dh_auto_test: 17 | -------------------------------------------------------------------------------- /debian/control: -------------------------------------------------------------------------------- 1 | Source: pyflame 2 | Priority: extra 3 | Maintainer: Evan Klitzke 4 | Build-Depends: debhelper (>= 9.0.0), autotools-dev, dh-autoreconf, pkg-config, python-dev, g++ 5 | Standards-Version: 3.9.3 6 | Section: libs 7 | X-Python-Version: >= 2.6 8 | 9 | Package: pyflame 10 | Architecture: any 11 | Multi-Arch: same 12 | Depends: ${shlibs:Depends}, ${misc:Depends}, libstdc++6, python 13 | Description: Python flamegraph tool 14 | 15 | Package: pyflame-dbg 16 | Architecture: any 17 | Multi-Arch: same 18 | Depends: ${shlibs:Depends}, ${misc:Depends}, ${pyflame:Version} 19 | Recommends: python-dbg 20 | Description: Python flamegraph tool (debug package) 21 | -------------------------------------------------------------------------------- /debian/changelog: -------------------------------------------------------------------------------- 1 | pyflame (1.2.1) unstable; urgency=medium 2 | 3 | * New newest 1.2.1 tag 4 | * Various changes to the test suite 5 | 6 | -- Evan Klitzke Tue, 01 Nov 2016 22:14:32 -0700 7 | 8 | pyflame (1.2.0) unstable; urgency=medium 9 | 10 | * Get newest 1.2.0 tag 11 | 12 | -- Evan Klitzke Tue, 01 Nov 2016 14:33:36 -0700 13 | 14 | pyflame (1.1) unstable; urgency=medium 15 | 16 | * Lots of updates 17 | 18 | -- Evan Klitzke Fri, 07 Oct 2016 09:44:26 -0700 19 | 20 | pyflame (1.0) unstable; urgency=medium 21 | 22 | * Initial release. 23 | 24 | -- Evan Klitzke Tue, 09 Aug 2016 10:30:37 -0700 25 | -------------------------------------------------------------------------------- /tests/sleep.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | # 3 | # Copyright 2016 Uber Technologies, Inc. 4 | # 5 | # Licensed under the Apache License, Version 2.0 (the "License"); 6 | # you may not use this file except in compliance with the License. 7 | # You may obtain a copy of the License at 8 | # 9 | # http://www.apache.org/licenses/LICENSE-2.0 10 | # 11 | # Unless required by applicable law or agreed to in writing, software 12 | # distributed under the License is distributed on an "AS IS" BASIS, 13 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 14 | # See the License for the specific language governing permissions and 15 | # limitations under the License. 16 | 17 | sleep 1 18 | -------------------------------------------------------------------------------- /src/frob2.cc: -------------------------------------------------------------------------------- 1 | // Copyright 2016 Uber Technologies, Inc. 2 | // 3 | // Licensed under the Apache License, Version 2.0 (the "License"); 4 | // you may not use this file except in compliance with the License. 5 | // You may obtain a copy of the License at 6 | // 7 | // http://www.apache.org/licenses/LICENSE-2.0 8 | // 9 | // Unless required by applicable law or agreed to in writing, software 10 | // distributed under the License is distributed on an "AS IS" BASIS, 11 | // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | // See the License for the specific language governing permissions and 13 | // limitations under the License. 14 | 15 | #define PYFLAME_PY_VERSION 2 16 | 17 | #include "./frob.cc" 18 | -------------------------------------------------------------------------------- /src/frob3.cc: -------------------------------------------------------------------------------- 1 | // Copyright 2016 Uber Technologies, Inc. 2 | // 3 | // Licensed under the Apache License, Version 2.0 (the "License"); 4 | // you may not use this file except in compliance with the License. 5 | // You may obtain a copy of the License at 6 | // 7 | // http://www.apache.org/licenses/LICENSE-2.0 8 | // 9 | // Unless required by applicable law or agreed to in writing, software 10 | // distributed under the License is distributed on an "AS IS" BASIS, 11 | // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | // See the License for the specific language governing permissions and 13 | // limitations under the License. 14 | 15 | #define PYFLAME_PY_VERSION 3 16 | 17 | #include "./frob.cc" 18 | -------------------------------------------------------------------------------- /.travis.yml: -------------------------------------------------------------------------------- 1 | sudo: required 2 | 3 | language: cpp 4 | 5 | matrix: 6 | include: 7 | - compiler: gcc 8 | python: "2.7" 9 | dist: trusty 10 | addons: 11 | apt: 12 | packages: 13 | - python-dev 14 | env: PYTHONVERSION=python2 15 | - compiler: gcc 16 | python: "3.4" 17 | dist: trusty 18 | addons: 19 | apt: 20 | packages: 21 | - python3-dev 22 | env: PYTHONVERSION=python3 23 | - compiler: gcc 24 | python: "3.5" 25 | dist: trusty 26 | addons: 27 | apt: 28 | packages: 29 | - python3-dev 30 | env: PYTHONVERSION=python3 31 | 32 | script: 33 | - sudo sysctl kernel.yama.ptrace_scope=0 34 | - unset PYTHON_CFLAGS 35 | - ./autogen.sh 36 | - ./configure 37 | - make 38 | - make test 39 | -------------------------------------------------------------------------------- /src/frame.cc: -------------------------------------------------------------------------------- 1 | // Copyright 2016 Uber Technologies, Inc. 2 | // 3 | // Licensed under the Apache License, Version 2.0 (the "License"); 4 | // you may not use this file except in compliance with the License. 5 | // You may obtain a copy of the License at 6 | // 7 | // http://www.apache.org/licenses/LICENSE-2.0 8 | // 9 | // Unless required by applicable law or agreed to in writing, software 10 | // distributed under the License is distributed on an "AS IS" BASIS, 11 | // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | // See the License for the specific language governing permissions and 13 | // limitations under the License. 14 | 15 | #include "./frame.h" 16 | 17 | namespace pyflame { 18 | std::ostream &operator<<(std::ostream &os, const Frame &frame) { 19 | os << frame.file() << ':' << frame.name() << ':' << frame.line(); 20 | return os; 21 | } 22 | } // namespace pyflame 23 | -------------------------------------------------------------------------------- /src/aslr.h: -------------------------------------------------------------------------------- 1 | // Copyright 2016 Uber Technologies, Inc. 2 | // 3 | // Licensed under the Apache License, Version 2.0 (the "License"); 4 | // you may not use this file except in compliance with the License. 5 | // You may obtain a copy of the License at 6 | // 7 | // http://www.apache.org/licenses/LICENSE-2.0 8 | // 9 | // Unless required by applicable law or agreed to in writing, software 10 | // distributed under the License is distributed on an "AS IS" BASIS, 11 | // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | // See the License for the specific language governing permissions and 13 | // limitations under the License. 14 | 15 | #pragma once 16 | 17 | #include 18 | 19 | #include 20 | 21 | namespace pyflame { 22 | // Find libpython2.7.so and its offset for an ASLR process. 23 | size_t LocateLibPython(pid_t pid, const std::string &hint, std::string *path); 24 | } // namespace pyflame 25 | -------------------------------------------------------------------------------- /debian/copyright: -------------------------------------------------------------------------------- 1 | Format: http://dep.debian.net/deps/dep5 2 | Upstream-Name: pyflame 3 | Source: https://github.com/uber/pyflame 4 | 5 | Files: * 6 | Copyright: 2016 Uber Technologies, Inc. 7 | License: Apache-2.0 8 | 9 | License: Apache-2.0 10 | Licensed under the Apache License, Version 2.0 (the "License"); 11 | you may not use this file except in compliance with the License. 12 | You may obtain a copy of the License at 13 | . 14 | http://www.apache.org/licenses/LICENSE-2.0 15 | . 16 | Unless required by applicable law or agreed to in writing, software 17 | distributed under the License is distributed on an "AS IS" BASIS, 18 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 19 | See the License for the specific language governing permissions and 20 | limitations under the License. 21 | . 22 | On Debian systems, the complete text of the Apache License 2.0 can 23 | be found in "/usr/share/common-licenses/Apache-2.0" 24 | -------------------------------------------------------------------------------- /src/thread.cc: -------------------------------------------------------------------------------- 1 | // Copyright 2016 Uber Technologies, Inc. 2 | // 3 | // Licensed under the Apache License, Version 2.0 (the "License"); 4 | // you may not use this file except in compliance with the License. 5 | // You may obtain a copy of the License at 6 | // 7 | // http://www.apache.org/licenses/LICENSE-2.0 8 | // 9 | // Unless required by applicable law or agreed to in writing, software 10 | // distributed under the License is distributed on an "AS IS" BASIS, 11 | // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | // See the License for the specific language governing permissions and 13 | // limitations under the License. 14 | 15 | #include "./thread.h" 16 | 17 | namespace pyflame { 18 | std::ostream &operator<<(std::ostream &os, const Thread &thread) { 19 | os << thread.id() << ':' << std::endl; 20 | for (const auto &frame : thread.frames()) { 21 | os << frame << std::endl; 22 | } 23 | return os; 24 | } 25 | } // namespace pyflame 26 | -------------------------------------------------------------------------------- /tests/sleeper.py: -------------------------------------------------------------------------------- 1 | # Copyright 2016 Uber Technologies, Inc. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | 15 | import os 16 | import sys 17 | import time 18 | 19 | 20 | def main(): 21 | sys.stdout.write('%d\n' % (os.getpid(),)) 22 | sys.stdout.flush() 23 | while True: 24 | time.sleep(0.1) 25 | target = time.time() + 0.1 26 | while time.time() < target: 27 | pass 28 | 29 | 30 | if __name__ == '__main__': 31 | main() 32 | -------------------------------------------------------------------------------- /src/Makefile.am: -------------------------------------------------------------------------------- 1 | # Here be dragons. 2 | # 3 | # The way this code is structured is the code that know about Python interpreter 4 | # internals is in frob.cc. This file isn't compiled directly, instead there are 5 | # two files frob{2,3}.cc that define preprocessor macros to compile frob.cc for 6 | # python2 and python3, respectively. 7 | # 8 | # The libtool magic here makes it so that frob2.cc is compiled with python2 9 | # flags and frob3.cc is compiled with python3 flags. 10 | 11 | bin_PROGRAMS = pyflame 12 | pyflame_SOURCES = aslr.cc frame.cc thread.cc namespace.cc posix.cc ptrace.cc pyflame.cc pyfrob.cc symbol.cc 13 | pyflame_LDADD = 14 | 15 | noinst_LTLIBRARIES = 16 | 17 | if ENABLE_PY2 18 | libfrob2_la_SOURCES = frob2.cc 19 | libfrob2_la_CXXFLAGS = $(PY2_CFLAGS) 20 | noinst_LTLIBRARIES += libfrob2.la 21 | pyflame_LDADD += libfrob2.la 22 | endif 23 | 24 | if ENABLE_PY3 25 | libfrob3_la_SOURCES = frob3.cc 26 | libfrob3_la_CXXFLAGS = $(PY3_CFLAGS) 27 | noinst_LTLIBRARIES += libfrob3.la 28 | pyflame_LDADD += libfrob3.la 29 | endif 30 | -------------------------------------------------------------------------------- /src/posix.h: -------------------------------------------------------------------------------- 1 | // Copyright 2016 Uber Technologies, Inc. 2 | // 3 | // Licensed under the Apache License, Version 2.0 (the "License"); 4 | // you may not use this file except in compliance with the License. 5 | // You may obtain a copy of the License at 6 | // 7 | // http://www.apache.org/licenses/LICENSE-2.0 8 | // 9 | // Unless required by applicable law or agreed to in writing, software 10 | // distributed under the License is distributed on an "AS IS" BASIS, 11 | // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | // See the License for the specific language governing permissions and 13 | // limitations under the License. 14 | 15 | #pragma once 16 | 17 | #include 18 | #include 19 | 20 | #include 21 | 22 | namespace pyflame { 23 | int OpenRdonly(const char *path); 24 | void Close(int fd); 25 | 26 | void Fstat(int fd, struct stat *buf); 27 | void Lstat(const char *path, struct stat *buf); 28 | 29 | void SetNs(int fd); 30 | 31 | std::string ReadLink(const char *path); 32 | } // namespace pyflame 33 | -------------------------------------------------------------------------------- /src/version.h: -------------------------------------------------------------------------------- 1 | // Copyright 2016 Uber Technologies, Inc. 2 | // 3 | // Licensed under the Apache License, Version 2.0 (the "License"); 4 | // you may not use this file except in compliance with the License. 5 | // You may obtain a copy of the License at 6 | // 7 | // http://www.apache.org/licenses/LICENSE-2.0 8 | // 9 | // Unless required by applicable law or agreed to in writing, software 10 | // distributed under the License is distributed on an "AS IS" BASIS, 11 | // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | // See the License for the specific language governing permissions and 13 | // limitations under the License. 14 | 15 | #pragma once 16 | 17 | namespace { 18 | #ifdef __VERSION__ 19 | #ifdef __GNUC__ 20 | const char kBuildNote[] = 21 | "Compiled " __DATE__ ", " __TIME__ " by GCC " __VERSION__; 22 | #else 23 | const char kBuildNote[] = 24 | "Compiled " __DATE__ ", " __TIME__ " by C compiler " __VERSION__; 25 | #endif 26 | #else 27 | const char kBuildNote[] = "Compiled " __DATE__ ", " __TIME__; 28 | #endif 29 | } // namespace 30 | -------------------------------------------------------------------------------- /src/exc.h: -------------------------------------------------------------------------------- 1 | // Copyright 2016 Uber Technologies, Inc. 2 | // 3 | // Licensed under the Apache License, Version 2.0 (the "License"); 4 | // you may not use this file except in compliance with the License. 5 | // You may obtain a copy of the License at 6 | // 7 | // http://www.apache.org/licenses/LICENSE-2.0 8 | // 9 | // Unless required by applicable law or agreed to in writing, software 10 | // distributed under the License is distributed on an "AS IS" BASIS, 11 | // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | // See the License for the specific language governing permissions and 13 | // limitations under the License. 14 | 15 | #pragma once 16 | 17 | #include 18 | #include 19 | 20 | namespace pyflame { 21 | class FatalException : public std::runtime_error { 22 | public: 23 | explicit FatalException(const std::string &what_arg) 24 | : std::runtime_error(what_arg) {} 25 | }; 26 | 27 | class PtraceException : public std::runtime_error { 28 | public: 29 | explicit PtraceException(const std::string &what_arg) 30 | : std::runtime_error(what_arg) {} 31 | }; 32 | } // namespace pyflame 33 | -------------------------------------------------------------------------------- /src/namespace.h: -------------------------------------------------------------------------------- 1 | // Copyright 2016 Uber Technologies, Inc. 2 | // 3 | // Licensed under the Apache License, Version 2.0 (the "License"); 4 | // you may not use this file except in compliance with the License. 5 | // You may obtain a copy of the License at 6 | // 7 | // http://www.apache.org/licenses/LICENSE-2.0 8 | // 9 | // Unless required by applicable law or agreed to in writing, software 10 | // distributed under the License is distributed on an "AS IS" BASIS, 11 | // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | // See the License for the specific language governing permissions and 13 | // limitations under the License. 14 | 15 | #pragma once 16 | 17 | #include 18 | 19 | namespace pyflame { 20 | // Implementation of a Linux filesystem namespace 21 | class Namespace { 22 | public: 23 | Namespace() = delete; 24 | explicit Namespace(pid_t pid); 25 | ~Namespace(); 26 | 27 | // Get a file descriptor in the namespace 28 | int Open(const char *path); 29 | 30 | private: 31 | int out_; // file descriptor that lets us return to our original namespace 32 | int in_; // file descriptor that lets us enter the target namespace 33 | }; 34 | } // namespace pyflame 35 | -------------------------------------------------------------------------------- /runtests.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | set -ex 4 | 5 | ENVDIR="./test_env" 6 | 7 | # Run tests using pip; $1 = python version 8 | run_pip_tests() { 9 | virtualenv -p "$1" "${ENVDIR}" 10 | trap "rm -rf ${ENVDIR}" EXIT 11 | 12 | . "${ENVDIR}/bin/activate" 13 | pip install --upgrade pip 14 | pip install pytest 15 | py.test tests/ 16 | 17 | # clean up the trap 18 | rm -rf "${ENVDIR}" EXIT 19 | trap "" EXIT 20 | } 21 | 22 | # See if we can run the pip tests with this Python version 23 | try_pip_tests() { 24 | if which "$1" &>/dev/null; then 25 | run_pip_tests "$1" 26 | fi 27 | } 28 | 29 | # This runs the tests for building an RPM 30 | run_fedora_tests() { 31 | py.test-2 tests/ 32 | py.test-3 tests/ 33 | } 34 | 35 | if [ "$1" = "fedora" ]; then 36 | # If the first arg is fedora, don't use Pip 37 | run_fedora_tests 38 | elif [ $# -eq 1 ]; then 39 | # Run the tests for a particular version of python 40 | run_pip_tests "$1" 41 | elif [ -n "$PYTHONVERSION" ]; then 42 | # Run the tests for $PYTHONVERSION 43 | run_pip_tests "$PYTHONVERSION" 44 | else 45 | # Try various places where we might find Python 46 | for py in python{,2,3}; do 47 | try_pip_tests "$py" 48 | done 49 | fi 50 | -------------------------------------------------------------------------------- /src/ptrace.h: -------------------------------------------------------------------------------- 1 | // Copyright 2016 Uber Technologies, Inc. 2 | // 3 | // Licensed under the Apache License, Version 2.0 (the "License"); 4 | // you may not use this file except in compliance with the License. 5 | // You may obtain a copy of the License at 6 | // 7 | // http://www.apache.org/licenses/LICENSE-2.0 8 | // 9 | // Unless required by applicable law or agreed to in writing, software 10 | // distributed under the License is distributed on an "AS IS" BASIS, 11 | // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | // See the License for the specific language governing permissions and 13 | // limitations under the License. 14 | 15 | #pragma once 16 | 17 | #include 18 | #include 19 | 20 | #include 21 | #include 22 | 23 | namespace pyflame { 24 | // attach to a process 25 | void PtraceAttach(pid_t pid); 26 | 27 | // detach a process 28 | void PtraceDetach(pid_t pid); 29 | 30 | // read the long word at an address 31 | long PtracePeek(pid_t pid, unsigned long addr); 32 | 33 | // peek a null-terminated string 34 | std::string PtracePeekString(pid_t pid, unsigned long addr); 35 | 36 | // peek some number of bytes 37 | std::unique_ptr PtracePeekBytes(pid_t pid, unsigned long addr, 38 | size_t nbytes); 39 | } // namespace pyflame 40 | -------------------------------------------------------------------------------- /tests/exit_early.py: -------------------------------------------------------------------------------- 1 | # Copyright 2016 Uber Technologies, Inc. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | 15 | import argparse 16 | import os 17 | import sys 18 | import time 19 | 20 | 21 | def main(): 22 | parser = argparse.ArgumentParser() 23 | parser.add_argument('-s', '--silent', action='store_true') 24 | args = parser.parse_args() 25 | if not args.silent: 26 | sys.stdout.write('%d\n' % (os.getpid(),)) 27 | sys.stdout.flush() 28 | max_time = time.time() + 2 29 | while True: 30 | time.sleep(0.1) 31 | target = time.time() + 0.1 32 | while True: 33 | now = time.time() 34 | if now >= max_time: 35 | return 36 | if now >= target: 37 | break 38 | 39 | 40 | if __name__ == '__main__': 41 | main() 42 | -------------------------------------------------------------------------------- /src/pyfrob.h: -------------------------------------------------------------------------------- 1 | // Copyright 2016 Uber Technologies, Inc. 2 | // 3 | // Licensed under the Apache License, Version 2.0 (the "License"); 4 | // you may not use this file except in compliance with the License. 5 | // You may obtain a copy of the License at 6 | // 7 | // http://www.apache.org/licenses/LICENSE-2.0 8 | // 9 | // Unless required by applicable law or agreed to in writing, software 10 | // distributed under the License is distributed on an "AS IS" BASIS, 11 | // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | // See the License for the specific language governing permissions and 13 | // limitations under the License. 14 | 15 | #pragma once 16 | 17 | #include "./symbol.h" 18 | #include "./thread.h" 19 | 20 | // This abstracts the representation of py2/py3 21 | namespace pyflame { 22 | 23 | // Get the threads. Each thread stack will be in reverse order (most recent frame first). 24 | typedef std::vector (*get_threads_t)(pid_t, PyAddresses); 25 | 26 | // Frobber to get python stack stuff; this encapsulates all of the Python 27 | // interpreter logic. 28 | class PyFrob { 29 | public: 30 | PyFrob(pid_t pid) : pid_(pid), addrs_() {} 31 | 32 | // Must be called before GetThreads() to detect the Python version 33 | void DetectPython(); 34 | 35 | // Get the current frame list. 36 | std::vector GetThreads(); 37 | 38 | private: 39 | pid_t pid_; 40 | PyAddresses addrs_; 41 | get_threads_t get_threads_; 42 | }; 43 | 44 | } // namespace pyflame 45 | -------------------------------------------------------------------------------- /pyflame.man: -------------------------------------------------------------------------------- 1 | .\" Process this file with 2 | .\" groff -man -Tascii foo.1 3 | .\" 4 | .TH PYFLAME 1 "OCTOBER 2016" Linux "User Manuals" 5 | .SH NAME 6 | pyflame \- A Ptracing Profiler For Python 7 | .SH SYNOPSIS 8 | .B pyflame [options] 9 | .I PID 10 | 11 | .B pyflame [-t|--trace] [options] 12 | .I command arg1 arg2... 13 | .SH DESCRIPTION 14 | .B pyflame 15 | profiles a Python process using 16 | .BR ptrace (2) 17 | to extract the stack trace. There are two modes. In the default mode 18 | .B pyflame 19 | will attach to a running process to get profiling data. If, instead, the 20 | .B -t 21 | or 22 | .B --trace 23 | options are given, 24 | .B pyflame 25 | will instead run a command and trace it to completion. 26 | 27 | Since 28 | .B pyflame 29 | is implemented using 30 | .BR ptrace (2) 31 | it has the same permissions model as programs like 32 | .BR strace (1) 33 | or 34 | .BR gdb (1) 35 | . 36 | .SH OPTIONS 37 | .TP 38 | .BR \-h ", " \-\-help 39 | Display help. 40 | .TP 41 | .BR \-s ", " \-\-seconds 42 | Run the profiler for this many seconds. 43 | .TP 44 | .BR \-r ", " \-\-rate 45 | Sample the process at this frequency. This should be a fractional value in seconds, so 46 | .B -r 0.001 47 | would mean to sample the process 1000 times a second, i.e. once every millisecond. 48 | .TP 49 | .BR \-t ", " \-\-trace 50 | Run pyflame in trace mode. 51 | .TP 52 | .BR \-T ", " \-\-timestamp 53 | Print the timestamp for each stack. This is useful for generating "flame chart" profiles. 54 | .TP 55 | .BR \-v ", " \-\-version 56 | Print the version. 57 | .TP 58 | .BR \-x ", " \-\-exclude-idle 59 | Exclude "idle" time. 60 | .SH BUGS 61 | If you find them, please report them on GitHub! 62 | .SH AUTHOR 63 | Evan Klitzke 64 | -------------------------------------------------------------------------------- /src/aslr.cc: -------------------------------------------------------------------------------- 1 | // Copyright 2016 Uber Technologies, Inc. 2 | // 3 | // Licensed under the Apache License, Version 2.0 (the "License"); 4 | // you may not use this file except in compliance with the License. 5 | // You may obtain a copy of the License at 6 | // 7 | // http://www.apache.org/licenses/LICENSE-2.0 8 | // 9 | // Unless required by applicable law or agreed to in writing, software 10 | // distributed under the License is distributed on an "AS IS" BASIS, 11 | // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | // See the License for the specific language governing permissions and 13 | // limitations under the License. 14 | 15 | #include "./aslr.h" 16 | #include "./exc.h" 17 | 18 | #include 19 | #include 20 | #include 21 | 22 | namespace pyflame { 23 | // Find libpython2.7.so and its offset for an ASLR process 24 | size_t LocateLibPython(pid_t pid, const std::string &hint, std::string *path) { 25 | std::ostringstream ss; 26 | ss << "/proc/" << pid << "/maps"; 27 | std::ifstream fp(ss.str()); 28 | std::string line; 29 | std::string elf_path; 30 | while (std::getline(fp, line)) { 31 | if (line.find(hint) != std::string::npos && 32 | line.find(" r-xp ") != std::string::npos) { 33 | size_t pos = line.find('/'); 34 | if (pos == std::string::npos) { 35 | throw FatalException("Did not find libpython absolute path"); 36 | } 37 | *path = line.substr(pos); 38 | pos = line.find('-'); 39 | if (pos == std::string::npos) { 40 | throw FatalException("Did not find libpython virtual memory address"); 41 | } 42 | return std::strtoul(line.substr(0, pos).c_str(), nullptr, 16); 43 | } 44 | } 45 | return 0; 46 | } 47 | } // namespace pyflame 48 | -------------------------------------------------------------------------------- /src/thread.h: -------------------------------------------------------------------------------- 1 | // Copyright 2016 Uber Technologies, Inc. 2 | // 3 | // Licensed under the Apache License, Version 2.0 (the "License"); 4 | // you may not use this file except in compliance with the License. 5 | // You may obtain a copy of the License at 6 | // 7 | // http://www.apache.org/licenses/LICENSE-2.0 8 | // 9 | // Unless required by applicable law or agreed to in writing, software 10 | // distributed under the License is distributed on an "AS IS" BASIS, 11 | // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | // See the License for the specific language governing permissions and 13 | // limitations under the License. 14 | 15 | #pragma once 16 | 17 | #include 18 | 19 | #include 20 | #include 21 | #include 22 | #include 23 | #include 24 | 25 | #include "./frame.h" 26 | 27 | namespace pyflame { 28 | 29 | class Thread { 30 | public: 31 | Thread() = delete; 32 | Thread(const Thread &other) 33 | : id_(other.id_), is_current_(other.is_current_), frames_(other.frames_) {} 34 | Thread(const long id, const bool is_current, const std::vector frames) 35 | : id_(id), is_current_(is_current), frames_(frames) {} 36 | 37 | inline const long id() const { return id_; } 38 | inline const bool is_current() const { return is_current_; } 39 | inline const std::vector &frames() const { return frames_; } 40 | 41 | inline bool operator==(const Thread &other) const { 42 | return id_ == other.id_ && is_current_ == other.is_current_ && frames_ == other.frames_; 43 | } 44 | 45 | private: 46 | long id_; 47 | bool is_current_; 48 | std::vector frames_; 49 | }; 50 | 51 | std::ostream &operator<<(std::ostream &os, const Thread &thread); 52 | } // namespace pyflame 53 | -------------------------------------------------------------------------------- /src/frame.h: -------------------------------------------------------------------------------- 1 | // Copyright 2016 Uber Technologies, Inc. 2 | // 3 | // Licensed under the Apache License, Version 2.0 (the "License"); 4 | // you may not use this file except in compliance with the License. 5 | // You may obtain a copy of the License at 6 | // 7 | // http://www.apache.org/licenses/LICENSE-2.0 8 | // 9 | // Unless required by applicable law or agreed to in writing, software 10 | // distributed under the License is distributed on an "AS IS" BASIS, 11 | // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | // See the License for the specific language governing permissions and 13 | // limitations under the License. 14 | 15 | #pragma once 16 | 17 | #include 18 | 19 | #include 20 | #include 21 | #include 22 | #include 23 | #include 24 | 25 | #include "./namespace.h" 26 | 27 | namespace pyflame { 28 | 29 | class Frame { 30 | public: 31 | Frame() = delete; 32 | Frame(const Frame &other) 33 | : file_(other.file_), name_(other.name_), line_(other.line_) {} 34 | Frame(const std::string &file, const std::string &name, size_t line) 35 | : file_(file), name_(name), line_(line) {} 36 | 37 | inline const std::string &file() const { return file_; } 38 | inline const std::string &name() const { return name_; } 39 | inline size_t line() const { return line_; } 40 | 41 | inline bool operator==(const Frame &other) const { 42 | return file_ == other.file_ && line_ == other.line_; 43 | } 44 | 45 | private: 46 | std::string file_; 47 | std::string name_; 48 | size_t line_; 49 | }; 50 | 51 | std::ostream &operator<<(std::ostream &os, const Frame &frame); 52 | 53 | typedef std::vector frames_t; 54 | 55 | struct FrameHash { 56 | size_t operator()(const frames_t &frames) const { 57 | size_t hash = 0; 58 | for (size_t i = 0; i < frames.size(); i++) { 59 | hash ^= std::hash()(i); 60 | hash ^= std::hash()(frames[i].file()); 61 | } 62 | return hash; 63 | } 64 | }; 65 | 66 | struct FrameTS { 67 | std::chrono::system_clock::time_point ts; 68 | frames_t frames; 69 | }; 70 | } // namespace pyflame 71 | -------------------------------------------------------------------------------- /configure.ac: -------------------------------------------------------------------------------- 1 | AC_PREREQ([2.68]) 2 | AC_INIT([pyflame], [1.2.1], [evan@uber.com]) 3 | AC_CONFIG_AUX_DIR([build-aux]) 4 | AC_CONFIG_HEADERS([src/config.h]) 5 | AC_CONFIG_MACRO_DIR([m4]) 6 | AC_CONFIG_SRCDIR([src/pyflame.cc]) 7 | 8 | AM_INIT_AUTOMAKE([dist-bzip2 foreign subdir-objects -Wall -Werror]) 9 | 10 | # Checks for programs. 11 | AC_PROG_CXX 12 | AC_PROG_CC 13 | AC_PROG_INSTALL 14 | AM_PROG_AR 15 | 16 | LT_INIT 17 | 18 | # Fail early if the user tries to build on OS X 19 | AC_CHECK_HEADERS([linux/ptrace.h], [], [AC_MSG_ERROR([Pyflame only supports Linux hosts])]) 20 | 21 | AC_LANG([C++]) 22 | 23 | # This is commented out because it doesn't work right on Ubuntu 12.04 (Precise) 24 | # AX_CXX_COMPILE_STDCXX_11 25 | AX_CHECK_COMPILE_FLAG(["-std=c++11"], 26 | [AX_APPEND_FLAG(["-std=c++11"], [CXXFLAGS])], 27 | [AX_CHECK_COMPILE_FLAG(["-std=c++0x"], 28 | [AX_APPEND_FLAG(["-std=c++0x"], [CXXFLAGS])], 29 | [AC_MSG_ERROR([failed to detect C++11 support])])]) 30 | 31 | AX_CHECK_COMPILE_FLAG([-Wall], 32 | [AX_APPEND_FLAG([-Wall], [CXXFLAGS])]) 33 | 34 | # Checks for libraries. 35 | 36 | # Checks for header files. 37 | AC_CHECK_HEADERS([fcntl.h limits.h unistd.h]) 38 | 39 | # Checks for typedefs, structures, and compiler characteristics. 40 | # AC_CHECK_HEADER_STDBOOL 41 | 42 | # Checks for library functions. 43 | AC_FUNC_FORK 44 | AC_FUNC_LSTAT_FOLLOWS_SLASHED_SYMLINK 45 | AC_FUNC_MMAP 46 | AC_CHECK_FUNCS([memmove munmap strerror strtol strtoul]) 47 | 48 | PKG_CHECK_MODULES([PY2], [python], [enable_py2="yes"], [AC_MSG_WARN([Building without Python 2 support])]) 49 | AM_CONDITIONAL([ENABLE_PY2], [test x"$enable_py2" = xyes]) 50 | AM_COND_IF([ENABLE_PY2], [AC_DEFINE([ENABLE_PY2], [1], [Python2 is enabled])]) 51 | 52 | PKG_CHECK_MODULES([PY3], [python3], [enable_py3="yes"], [AC_MSG_WARN([Building without Python 3 support])]) 53 | AM_CONDITIONAL([ENABLE_PY3], [test x"$enable_py3" = xyes]) 54 | AM_COND_IF([ENABLE_PY3], [AC_DEFINE([ENABLE_PY3], [1], [Python3 is enabled])]) 55 | 56 | AC_CONFIG_FILES([Makefile 57 | src/Makefile]) 58 | AC_REVISION([m4_esyscmd_s([git describe --always])]) 59 | AC_OUTPUT 60 | -------------------------------------------------------------------------------- /src/posix.cc: -------------------------------------------------------------------------------- 1 | // Copyright 2016 Uber Technologies, Inc. 2 | // 3 | // Licensed under the Apache License, Version 2.0 (the "License"); 4 | // you may not use this file except in compliance with the License. 5 | // You may obtain a copy of the License at 6 | // 7 | // http://www.apache.org/licenses/LICENSE-2.0 8 | // 9 | // Unless required by applicable law or agreed to in writing, software 10 | // distributed under the License is distributed on an "AS IS" BASIS, 11 | // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | // See the License for the specific language governing permissions and 13 | // limitations under the License. 14 | 15 | #include "./posix.h" 16 | 17 | #include 18 | #include 19 | #include 20 | #include 21 | #include 22 | 23 | #include 24 | #include 25 | 26 | #include "./exc.h" 27 | 28 | namespace pyflame { 29 | int OpenRdonly(const char *path) { 30 | int fd = open(path, O_RDONLY); 31 | if (fd < 0) { 32 | std::ostringstream ss; 33 | ss << "Failed to open " << path << ": " << strerror(errno); 34 | throw FatalException(ss.str()); 35 | } 36 | return fd; 37 | } 38 | 39 | void Close(int fd) { 40 | if (fd < 0) { 41 | return; 42 | } 43 | while (close(fd) == -1) 44 | ; 45 | } 46 | 47 | void Fstat(int fd, struct stat *buf) { 48 | if (fstat(fd, buf) < 0) { 49 | std::ostringstream ss; 50 | ss << "Failed to fstat file descriptor " << fd << ": " << strerror(errno); 51 | throw FatalException(ss.str()); 52 | } 53 | } 54 | 55 | void Lstat(const char *path, struct stat *buf) { 56 | if (lstat(path, buf) < 0) { 57 | std::ostringstream ss; 58 | ss << "Failed to lstat path " << path << ": " << strerror(errno); 59 | throw FatalException(ss.str()); 60 | } 61 | } 62 | 63 | void SetNs(int fd) { 64 | if (setns(fd, 0)) { 65 | std::ostringstream ss; 66 | ss << "Failed to setns " << fd << ": " << strerror(errno); 67 | throw FatalException(ss.str()); 68 | } 69 | } 70 | 71 | std::string ReadLink(const char *path) { 72 | char buf[PATH_MAX]; 73 | ssize_t nbytes = readlink(path, buf, sizeof(buf)); 74 | if (nbytes < 0) { 75 | std::ostringstream ss; 76 | ss << "Failed to read symlink " << path << ": " << strerror(errno); 77 | throw FatalException(ss.str()); 78 | } 79 | buf[nbytes] = '\0'; 80 | return {buf, static_cast(nbytes)}; 81 | } 82 | } // namespace pyflame 83 | -------------------------------------------------------------------------------- /m4/ax_append_flag.m4: -------------------------------------------------------------------------------- 1 | # =========================================================================== 2 | # http://www.gnu.org/software/autoconf-archive/ax_append_flag.html 3 | # =========================================================================== 4 | # 5 | # SYNOPSIS 6 | # 7 | # AX_APPEND_FLAG(FLAG, [FLAGS-VARIABLE]) 8 | # 9 | # DESCRIPTION 10 | # 11 | # FLAG is appended to the FLAGS-VARIABLE shell variable, with a space 12 | # added in between. 13 | # 14 | # If FLAGS-VARIABLE is not specified, the current language's flags (e.g. 15 | # CFLAGS) is used. FLAGS-VARIABLE is not changed if it already contains 16 | # FLAG. If FLAGS-VARIABLE is unset in the shell, it is set to exactly 17 | # FLAG. 18 | # 19 | # NOTE: Implementation based on AX_CFLAGS_GCC_OPTION. 20 | # 21 | # LICENSE 22 | # 23 | # Copyright (c) 2008 Guido U. Draheim 24 | # Copyright (c) 2011 Maarten Bosmans 25 | # 26 | # This program is free software: you can redistribute it and/or modify it 27 | # under the terms of the GNU General Public License as published by the 28 | # Free Software Foundation, either version 3 of the License, or (at your 29 | # option) any later version. 30 | # 31 | # This program is distributed in the hope that it will be useful, but 32 | # WITHOUT ANY WARRANTY; without even the implied warranty of 33 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General 34 | # Public License for more details. 35 | # 36 | # You should have received a copy of the GNU General Public License along 37 | # with this program. If not, see . 38 | # 39 | # As a special exception, the respective Autoconf Macro's copyright owner 40 | # gives unlimited permission to copy, distribute and modify the configure 41 | # scripts that are the output of Autoconf when processing the Macro. You 42 | # need not follow the terms of the GNU General Public License when using 43 | # or distributing such scripts, even though portions of the text of the 44 | # Macro appear in them. The GNU General Public License (GPL) does govern 45 | # all other use of the material that constitutes the Autoconf Macro. 46 | # 47 | # This special exception to the GPL applies to versions of the Autoconf 48 | # Macro released by the Autoconf Archive. When you make and distribute a 49 | # modified version of the Autoconf Macro, you may extend this special 50 | # exception to the GPL to apply to your modified version as well. 51 | 52 | #serial 6 53 | 54 | AC_DEFUN([AX_APPEND_FLAG], 55 | [dnl 56 | AC_PREREQ(2.64)dnl for _AC_LANG_PREFIX and AS_VAR_SET_IF 57 | AS_VAR_PUSHDEF([FLAGS], [m4_default($2,_AC_LANG_PREFIX[FLAGS])]) 58 | AS_VAR_SET_IF(FLAGS,[ 59 | AS_CASE([" AS_VAR_GET(FLAGS) "], 60 | [*" $1 "*], [AC_RUN_LOG([: FLAGS already contains $1])], 61 | [ 62 | AS_VAR_APPEND(FLAGS,[" $1"]) 63 | AC_RUN_LOG([: FLAGS="$FLAGS"]) 64 | ]) 65 | ], 66 | [ 67 | AS_VAR_SET(FLAGS,[$1]) 68 | AC_RUN_LOG([: FLAGS="$FLAGS"]) 69 | ]) 70 | AS_VAR_POPDEF([FLAGS])dnl 71 | ])dnl AX_APPEND_FLAG 72 | -------------------------------------------------------------------------------- /src/namespace.cc: -------------------------------------------------------------------------------- 1 | // Copyright 2016 Uber Technologies, Inc. 2 | // 3 | // Licensed under the Apache License, Version 2.0 (the "License"); 4 | // you may not use this file except in compliance with the License. 5 | // You may obtain a copy of the License at 6 | // 7 | // http://www.apache.org/licenses/LICENSE-2.0 8 | // 9 | // Unless required by applicable law or agreed to in writing, software 10 | // distributed under the License is distributed on an "AS IS" BASIS, 11 | // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | // See the License for the specific language governing permissions and 13 | // limitations under the License. 14 | 15 | #include "./namespace.h" 16 | 17 | #include 18 | #include 19 | #include 20 | #include 21 | #include 22 | 23 | #include 24 | #include 25 | #include 26 | 27 | #include "./exc.h" 28 | #include "./posix.h" 29 | 30 | namespace { 31 | const char kOurMnt[] = "/proc/self/ns/mnt"; 32 | } 33 | 34 | namespace pyflame { 35 | Namespace::Namespace(pid_t pid) : out_(-1), in_(-1) { 36 | struct stat in_st; 37 | std::ostringstream os; 38 | os << "/proc/" << pid << "/ns/mnt"; 39 | const std::string their_mnt = os.str(); 40 | 41 | struct stat out_st; 42 | Lstat(kOurMnt, &out_st); 43 | // Since Linux 3.8 symbolic links are used. 44 | if (S_ISLNK(out_st.st_mode)) { 45 | char our_name[PATH_MAX]; 46 | ssize_t ourlen = readlink(kOurMnt, our_name, sizeof(our_name)); 47 | if (ourlen < 0) { 48 | std::ostringstream ss; 49 | ss << "Failed to readlink " << kOurMnt << ": " << strerror(errno); 50 | throw FatalException(ss.str()); 51 | } 52 | our_name[ourlen] = '\0'; 53 | 54 | char their_name[PATH_MAX]; 55 | ssize_t theirlen = readlink(their_mnt.c_str(), their_name, 56 | sizeof(their_name)); 57 | if (theirlen < 0) { 58 | std::ostringstream ss; 59 | ss << "Failed to readlink " << their_mnt.c_str() << ": " 60 | << strerror(errno); 61 | throw FatalException(ss.str()); 62 | } 63 | their_name[theirlen] = '\0'; 64 | 65 | if (strcmp(our_name, their_name) != 0) { 66 | out_ = OpenRdonly(kOurMnt); 67 | in_ = OpenRdonly(their_mnt.c_str()); 68 | } 69 | } else { 70 | // Before Linux 3.8 these are hard links. 71 | out_ = OpenRdonly(kOurMnt); 72 | Fstat(out_, &out_st); 73 | 74 | in_ = OpenRdonly(os.str().c_str()); 75 | Fstat(in_, &in_st); 76 | if (out_st.st_ino == in_st.st_ino) { 77 | Close(out_); 78 | Close(in_); 79 | out_ = in_ = -1; 80 | } 81 | } 82 | } 83 | 84 | int Namespace::Open(const char *path) { 85 | if (in_ != -1) { 86 | SetNs(in_); 87 | int fd = open(path, O_RDONLY); 88 | SetNs(out_); 89 | return fd; 90 | } else { 91 | return open(path, O_RDONLY); 92 | } 93 | } 94 | 95 | Namespace::~Namespace() { 96 | if (out_) { 97 | Close(out_); 98 | Close(in_); 99 | } 100 | } 101 | } // namespace pyflame 102 | -------------------------------------------------------------------------------- /src/ptrace.cc: -------------------------------------------------------------------------------- 1 | // Copyright 2016 Uber Technologies, Inc. 2 | // 3 | // Licensed under the Apache License, Version 2.0 (the "License"); 4 | // you may not use this file except in compliance with the License. 5 | // You may obtain a copy of the License at 6 | // 7 | // http://www.apache.org/licenses/LICENSE-2.0 8 | // 9 | // Unless required by applicable law or agreed to in writing, software 10 | // distributed under the License is distributed on an "AS IS" BASIS, 11 | // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | // See the License for the specific language governing permissions and 13 | // limitations under the License. 14 | 15 | #include "./ptrace.h" 16 | 17 | #include 18 | #include 19 | #include 20 | #include 21 | #include 22 | 23 | #include 24 | #include 25 | 26 | #include "./exc.h" 27 | 28 | namespace pyflame { 29 | void PtraceAttach(pid_t pid) { 30 | if (ptrace(PTRACE_ATTACH, pid, 0, 0)) { 31 | std::ostringstream ss; 32 | ss << "Failed to attach to PID " << pid << ": " << strerror(errno); 33 | throw PtraceException(ss.str()); 34 | } 35 | if (wait(nullptr) == -1) { 36 | std::ostringstream ss; 37 | ss << "Failed to wait on PID " << pid << ": " << strerror(errno); 38 | throw PtraceException(ss.str()); 39 | } 40 | } 41 | 42 | void PtraceDetach(pid_t pid) { 43 | if (ptrace(PTRACE_DETACH, pid, 0, 0)) { 44 | std::ostringstream ss; 45 | ss << "Failed to detach PID " << pid << ": " << strerror(errno); 46 | throw PtraceException(ss.str()); 47 | } 48 | } 49 | 50 | long PtracePeek(pid_t pid, unsigned long addr) { 51 | errno = 0; 52 | const long data = ptrace(PTRACE_PEEKDATA, pid, addr, 0); 53 | if (data == -1 && errno != 0) { 54 | std::ostringstream ss; 55 | ss << "Failed to PTRACE_PEEKDATA at " << reinterpret_cast(addr) 56 | << ": " << strerror(errno); 57 | throw PtraceException(ss.str()); 58 | } 59 | return data; 60 | } 61 | 62 | std::string PtracePeekString(pid_t pid, unsigned long addr) { 63 | std::ostringstream dump; 64 | unsigned long off = 0; 65 | while (true) { 66 | const long val = PtracePeek(pid, addr + off); 67 | 68 | // XXX: this can be micro-optimized, c.f. 69 | // https://graphics.stanford.edu/~seander/bithacks.html#ZeroInWord 70 | const std::string chunk(reinterpret_cast(&val), sizeof(val)); 71 | dump << chunk.c_str(); 72 | if (chunk.find_first_of('\0') != std::string::npos) { 73 | break; 74 | } 75 | off += sizeof(val); 76 | } 77 | return dump.str(); 78 | } 79 | 80 | std::unique_ptr PtracePeekBytes(pid_t pid, unsigned long addr, 81 | size_t nbytes) { 82 | // align the buffer to a word size 83 | if (nbytes % sizeof(long)) { 84 | nbytes = (nbytes / sizeof(long) + 1) * sizeof(long); 85 | } 86 | std::unique_ptr bytes(new uint8_t[nbytes]); 87 | 88 | size_t off = 0; 89 | while (off < nbytes) { 90 | const long val = PtracePeek(pid, addr + off); 91 | memmove(bytes.get() + off, &val, sizeof(val)); 92 | off += sizeof(val); 93 | } 94 | return bytes; 95 | } 96 | } // namespace pyflame 97 | -------------------------------------------------------------------------------- /m4/ax_check_compile_flag.m4: -------------------------------------------------------------------------------- 1 | # =========================================================================== 2 | # http://www.gnu.org/software/autoconf-archive/ax_check_compile_flag.html 3 | # =========================================================================== 4 | # 5 | # SYNOPSIS 6 | # 7 | # AX_CHECK_COMPILE_FLAG(FLAG, [ACTION-SUCCESS], [ACTION-FAILURE], [EXTRA-FLAGS], [INPUT]) 8 | # 9 | # DESCRIPTION 10 | # 11 | # Check whether the given FLAG works with the current language's compiler 12 | # or gives an error. (Warnings, however, are ignored) 13 | # 14 | # ACTION-SUCCESS/ACTION-FAILURE are shell commands to execute on 15 | # success/failure. 16 | # 17 | # If EXTRA-FLAGS is defined, it is added to the current language's default 18 | # flags (e.g. CFLAGS) when the check is done. The check is thus made with 19 | # the flags: "CFLAGS EXTRA-FLAGS FLAG". This can for example be used to 20 | # force the compiler to issue an error when a bad flag is given. 21 | # 22 | # INPUT gives an alternative input source to AC_COMPILE_IFELSE. 23 | # 24 | # NOTE: Implementation based on AX_CFLAGS_GCC_OPTION. Please keep this 25 | # macro in sync with AX_CHECK_{PREPROC,LINK}_FLAG. 26 | # 27 | # LICENSE 28 | # 29 | # Copyright (c) 2008 Guido U. Draheim 30 | # Copyright (c) 2011 Maarten Bosmans 31 | # 32 | # This program is free software: you can redistribute it and/or modify it 33 | # under the terms of the GNU General Public License as published by the 34 | # Free Software Foundation, either version 3 of the License, or (at your 35 | # option) any later version. 36 | # 37 | # This program is distributed in the hope that it will be useful, but 38 | # WITHOUT ANY WARRANTY; without even the implied warranty of 39 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General 40 | # Public License for more details. 41 | # 42 | # You should have received a copy of the GNU General Public License along 43 | # with this program. If not, see . 44 | # 45 | # As a special exception, the respective Autoconf Macro's copyright owner 46 | # gives unlimited permission to copy, distribute and modify the configure 47 | # scripts that are the output of Autoconf when processing the Macro. You 48 | # need not follow the terms of the GNU General Public License when using 49 | # or distributing such scripts, even though portions of the text of the 50 | # Macro appear in them. The GNU General Public License (GPL) does govern 51 | # all other use of the material that constitutes the Autoconf Macro. 52 | # 53 | # This special exception to the GPL applies to versions of the Autoconf 54 | # Macro released by the Autoconf Archive. When you make and distribute a 55 | # modified version of the Autoconf Macro, you may extend this special 56 | # exception to the GPL to apply to your modified version as well. 57 | 58 | #serial 4 59 | 60 | AC_DEFUN([AX_CHECK_COMPILE_FLAG], 61 | [AC_PREREQ(2.64)dnl for _AC_LANG_PREFIX and AS_VAR_IF 62 | AS_VAR_PUSHDEF([CACHEVAR],[ax_cv_check_[]_AC_LANG_ABBREV[]flags_$4_$1])dnl 63 | AC_CACHE_CHECK([whether _AC_LANG compiler accepts $1], CACHEVAR, [ 64 | ax_check_save_flags=$[]_AC_LANG_PREFIX[]FLAGS 65 | _AC_LANG_PREFIX[]FLAGS="$[]_AC_LANG_PREFIX[]FLAGS $4 $1" 66 | AC_COMPILE_IFELSE([m4_default([$5],[AC_LANG_PROGRAM()])], 67 | [AS_VAR_SET(CACHEVAR,[yes])], 68 | [AS_VAR_SET(CACHEVAR,[no])]) 69 | _AC_LANG_PREFIX[]FLAGS=$ax_check_save_flags]) 70 | AS_VAR_IF(CACHEVAR,yes, 71 | [m4_default([$2], :)], 72 | [m4_default([$3], :)]) 73 | AS_VAR_POPDEF([CACHEVAR])dnl 74 | ])dnl AX_CHECK_COMPILE_FLAGS 75 | -------------------------------------------------------------------------------- /src/symbol.h: -------------------------------------------------------------------------------- 1 | // Copyright 2016 Uber Technologies, Inc. 2 | // 3 | // Licensed under the Apache License, Version 2.0 (the "License"); 4 | // you may not use this file except in compliance with the License. 5 | // You may obtain a copy of the License at 6 | // 7 | // http://www.apache.org/licenses/LICENSE-2.0 8 | // 9 | // Unless required by applicable law or agreed to in writing, software 10 | // distributed under the License is distributed on an "AS IS" BASIS, 11 | // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | // See the License for the specific language governing permissions and 13 | // limitations under the License. 14 | 15 | #pragma once 16 | 17 | #include 18 | 19 | #include 20 | #include 21 | #include 22 | #include 23 | #include 24 | 25 | #include "./exc.h" 26 | #include "./namespace.h" 27 | 28 | #if (__WORDSIZE == 64) 29 | #define ehdr_t Elf64_Ehdr 30 | #define shdr_t Elf64_Shdr 31 | #define dyn_t Elf64_Dyn 32 | #define sym_t Elf64_Sym 33 | #define ARCH_ELFCLASS ELFCLASS64 34 | #elif (__WORDSIZE == 32) 35 | #define ehdr_t Elf32_Ehdr 36 | #define shdr_t Elf32_Shdr 37 | #define dyn_t Elf32_Dyn 38 | #define sym_t Elf32_Sym 39 | #define ARCH_ELFCLASS ELFCLASS32 40 | #else 41 | static_assert(false, "unknown build environment"); 42 | #endif 43 | 44 | namespace pyflame { 45 | 46 | // The Python interpreter version 47 | enum class PyVersion { Unknown = 0, Py2 = 2, Py3 = 3 }; 48 | 49 | // Symbols 50 | struct PyAddresses { 51 | unsigned long tstate_addr; 52 | unsigned long interp_head_addr; 53 | 54 | PyAddresses() : tstate_addr(0), interp_head_addr(0) {} 55 | 56 | PyAddresses operator+(const unsigned long base) const { 57 | PyAddresses res; 58 | res.tstate_addr = this->tstate_addr == 0 ? 0 : this->tstate_addr + base; 59 | res.interp_head_addr = this->interp_head_addr == 0 ? 0 : this->interp_head_addr + base; 60 | return res; 61 | } 62 | 63 | bool is_valid() const { 64 | return this->tstate_addr != 0; 65 | } 66 | }; 67 | 68 | // Representation of an ELF file. 69 | class ELF { 70 | public: 71 | ELF() : addr_(nullptr), length_(0), dynamic_(-1), dynstr_(-1), dynsym_(-1), strtab_(-1), symtab_(-1) {} 72 | ~ELF() { Close(); } 73 | 74 | // Open a file 75 | void Open(const std::string &target, Namespace *ns); 76 | 77 | // Close the file; normally the destructor will do this for you. 78 | void Close(); 79 | 80 | // Parse the ELF sections. 81 | void Parse(); 82 | 83 | // Find the DT_NEEDED fields. This is similar to the ldd(1) command. 84 | std::vector NeededLibs(); 85 | 86 | // Get the address of _PyThreadState_Current & interp_head, and the Python version 87 | PyAddresses GetAddresses(PyVersion *version); 88 | 89 | private: 90 | void *addr_; 91 | size_t length_; 92 | int dynamic_, dynstr_, dynsym_, strtab_, symtab_; 93 | 94 | inline const ehdr_t *hdr() const { 95 | return reinterpret_cast(addr_); 96 | } 97 | 98 | inline const shdr_t *shdr(int idx) const { 99 | if (idx < 0) { 100 | std::ostringstream ss; 101 | ss << "Illegal shdr index: " << idx; 102 | throw FatalException(ss.str()); 103 | } 104 | return reinterpret_cast(p() + hdr()->e_shoff + 105 | idx * hdr()->e_shentsize); 106 | } 107 | 108 | inline unsigned long p() const { 109 | return reinterpret_cast(addr_); 110 | } 111 | 112 | inline const char *strtab(int offset) const { 113 | const shdr_t *strings = shdr(hdr()->e_shstrndx); 114 | return reinterpret_cast(p() + strings->sh_offset + offset); 115 | } 116 | 117 | inline const char *dynstr(int offset) const { 118 | const shdr_t *strings = shdr(dynstr_); 119 | return reinterpret_cast(p() + strings->sh_offset + offset); 120 | } 121 | 122 | void WalkTable(int sym, int str, bool &have_version, PyVersion *version, PyAddresses &addrs); 123 | }; 124 | } // namespace pyflame 125 | -------------------------------------------------------------------------------- /utils/flame-chart-json: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | 3 | import sys 4 | import json 5 | 6 | """ Generate JSON for visualizing Flame Charts using Chrome CPU Profiler 7 | This takes input from Pyflame which is of the format 8 | timestamp_1 9 | callstack_1 10 | ... 11 | timestamp_n 12 | callstack_n 13 | 14 | With each callstack is of below format 15 | file_path:function_name:line_no;...file_path:function_name:line_no; 16 | 17 | Then converts them to JSON format which can be visualized as Flame Charts using 18 | Chrome CPU profiler 19 | 20 | USAGE: cat pyflame.out | flame-chart-json > flame_chart.cpuprofile 21 | You can also pipe pyflame output directly. 22 | Note: Chrome CPU profiler expects file with extension ".cpuprofile". This format 23 | is supported on Chrome Version 54.0.2840.71 24 | 25 | """ 26 | 27 | 28 | class FCJson(object): 29 | 30 | def __init__(self): 31 | self.node_id = 1 32 | self.prof = {} 33 | self.prev_ts = 0 34 | self.delta_ts = 2000 35 | self.IDLE_NODE_ID = 2 36 | 37 | # init root 38 | self.prof['nodes'] = [] 39 | node = {} 40 | self.fill_node(node, "(root)") 41 | node['children'] = [] 42 | self.prof['nodes'].append(node) 43 | 44 | # init idle node 45 | idle_node = {} 46 | self.fill_node(idle_node, "(idle)") 47 | idle_node['children'] = [] 48 | self.prof['nodes'].append(idle_node) 49 | self.prof['nodes'][0]['children'].append(self.node_id - 1) 50 | 51 | # init rest 52 | self.prof['startTime'] = 0 53 | self.prof['endTime'] = 5000 54 | self.prof['samples'] = [] 55 | self.prof['timeDeltas'] = [] 56 | 57 | def fill_node(self, node, func_name, fill_child_id=False): 58 | node['id'] = self.node_id 59 | node['callFrame'] = {} 60 | node['callFrame']['functionName'] = func_name 61 | node['callFrame']['scriptId'] = "0" 62 | node['callFrame']['url'] = "" 63 | node['hitCount'] = 1 64 | self.node_id += 1 65 | if fill_child_id is True: 66 | node['children'] = [] 67 | node['children'].append(self.node_id) 68 | 69 | def create_cs(self, cs): 70 | add_child = True 71 | cs_len = len(cs) 72 | # Do for each of the stack frames in the callstack 73 | for i, frame in enumerate(cs): 74 | # idle case can be optimized if required in future 75 | if frame == '': 76 | continue 77 | 78 | # Store the stack frame 79 | node = {} 80 | if (i >= cs_len - 2): 81 | add_child = False 82 | 83 | self.fill_node(node, frame, add_child) 84 | 85 | # Add the node to prof 86 | self.prof['nodes'].append(node) 87 | 88 | def create_node(self, ts, cs): 89 | prev_node_id = self.node_id 90 | 91 | self.create_cs(cs.strip().split(';')) 92 | 93 | self.prof['samples'].append(self.node_id - 1) 94 | if self.prev_ts is not 0: 95 | self.prof['timeDeltas'].append(ts-self.prev_ts-1000) 96 | else: 97 | self.prof['timeDeltas'].append(0) 98 | 99 | # Idle is made as the root of all the callstacks so that we display the 100 | # flame chart correctly. I did not find a better way to do this 101 | self.prof['samples'].append(self.IDLE_NODE_ID) 102 | self.prof['timeDeltas'].append(1000) 103 | 104 | self.prev_ts = ts 105 | # add the node_id of the first frame of cs to the children of root & 106 | # idle nodes. 107 | self.prof['nodes'][0]['children'].append(prev_node_id) 108 | self.prof['nodes'][1]['children'].append(prev_node_id) 109 | 110 | def start(self): 111 | first_ts = 0 112 | last_ts = 0 113 | is_first_ln = True 114 | for line in sys.stdin: 115 | # 1st line should be timestamp & 2nd line callstack 116 | if is_first_ln: 117 | ts = int(line) 118 | else: 119 | cs = line 120 | 121 | # Keep track of timestamps 122 | if first_ts == 0: 123 | first_ts = ts 124 | last_ts = ts 125 | 126 | if is_first_ln is False: 127 | if cs is None: 128 | break 129 | 130 | is_first_ln = True 131 | self.create_node(ts, cs) 132 | self.delta_ts += 1000 133 | else: 134 | is_first_ln = False 135 | 136 | # update the end time based on the timestamps, and reduce it by half 137 | # because Chrome is doubling the endTime while displaying the flamechart 138 | self.prof['endTime'] = (last_ts - first_ts)/2 + 1000 139 | 140 | print(json.dumps(self.prof)) 141 | 142 | if __name__ == "__main__": 143 | fc = FCJson() 144 | fc.start() 145 | -------------------------------------------------------------------------------- /src/pyfrob.cc: -------------------------------------------------------------------------------- 1 | // Copyright 2016 Uber Technologies, Inc. 2 | // 3 | // Licensed under the Apache License, Version 2.0 (the "License"); 4 | // you may not use this file except in compliance with the License. 5 | // You may obtain a copy of the License at 6 | // 7 | // http://www.apache.org/licenses/LICENSE-2.0 8 | // 9 | // Unless required by applicable law or agreed to in writing, software 10 | // distributed under the License is distributed on an "AS IS" BASIS, 11 | // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | // See the License for the specific language governing permissions and 13 | // limitations under the License. 14 | 15 | #include "./pyfrob.h" 16 | 17 | #include 18 | 19 | #include "./aslr.h" 20 | #include "./config.h" 21 | #include "./exc.h" 22 | #include "./namespace.h" 23 | #include "./posix.h" 24 | #include "./symbol.h" 25 | 26 | #define FROB_FUNCS \ 27 | std::vector GetThreads(pid_t pid, PyAddresses addr); 28 | 29 | namespace pyflame { 30 | namespace { 31 | // locate within libpython 32 | PyAddresses AddressesFromLibPython(pid_t pid, const std::string &libpython, 33 | Namespace *ns, PyVersion *version) { 34 | std::string elf_path; 35 | const size_t offset = LocateLibPython(pid, libpython, &elf_path); 36 | if (offset == 0) { 37 | std::ostringstream ss; 38 | ss << "Failed to locate libpython named " << libpython; 39 | throw FatalException(ss.str()); 40 | } 41 | 42 | ELF pyelf; 43 | pyelf.Open(elf_path, ns); 44 | pyelf.Parse(); 45 | const PyAddresses addrs = pyelf.GetAddresses(version); 46 | if (!addrs.is_valid()) { 47 | throw FatalException("Failed to locate addresses"); 48 | } 49 | return addrs + offset; 50 | } 51 | 52 | PyAddresses Addrs(pid_t pid, Namespace *ns, PyVersion *version) { 53 | std::ostringstream ss; 54 | ss << "/proc/" << pid << "/exe"; 55 | ELF target; 56 | target.Open(ReadLink(ss.str().c_str()), ns); 57 | target.Parse(); 58 | 59 | // There's two different cases here. The default way Python is compiled you 60 | // get a "static" build which means that you get a big several-megabytes 61 | // Python executable that has all of the symbols statically built in. For 62 | // instance, this is how Python is built on Debian and Ubuntu. This is the 63 | // easiest case to handle, since in this case there are no tricks, we just 64 | // need to find the symbol in the ELF file. 65 | // 66 | // There's also a configure option called --enable-shared where you get a 67 | // small several-kilobytes Python executable that links against a 68 | // several-megabytes libpython2.7.so. This is how Python is built on Fedora. 69 | // If that's the case we need to do some fiddly things to find the true symbol 70 | // location. 71 | // 72 | // The code here attempts to detect if the executable links against 73 | // libpython2.7.so, and if it does the libpython variable will be filled with 74 | // the full soname. That determines where we need to look to find our symbol 75 | // table. 76 | 77 | PyAddresses addrs = target.GetAddresses(version); 78 | if (addrs.is_valid()) { 79 | return addrs; 80 | } 81 | 82 | std::string libpython; 83 | for (const auto &lib : target.NeededLibs()) { 84 | if (lib.find("libpython") != std::string::npos) { 85 | libpython = lib; 86 | break; 87 | } 88 | } 89 | if (!libpython.empty()) { 90 | return AddressesFromLibPython(pid, libpython, ns, version); 91 | } 92 | // A process like uwsgi may use dlopen() to load libpython... let's just guess 93 | // that the DSO is called libpython2.7.so 94 | // 95 | // XXX: this won't work if the embedding language is Python 3 96 | return AddressesFromLibPython(pid, "libpython2.7.so", ns, version); 97 | } 98 | } // namespace 99 | 100 | #ifdef ENABLE_PY2 101 | namespace py2 { 102 | FROB_FUNCS 103 | } 104 | #endif 105 | 106 | #ifdef ENABLE_PY3 107 | namespace py3 { 108 | FROB_FUNCS 109 | } 110 | #endif 111 | 112 | void PyFrob::DetectPython() { 113 | Namespace ns(pid_); 114 | bool matched = false; 115 | PyVersion version = PyVersion::Unknown; 116 | addrs_ = Addrs(pid_, &ns, &version); 117 | 118 | switch (version) { 119 | case PyVersion::Unknown: 120 | break; // to appease -Wall 121 | case PyVersion::Py2: 122 | #ifdef ENABLE_PY2 123 | get_threads_ = py2::GetThreads; 124 | matched = true; 125 | #endif 126 | break; 127 | case PyVersion::Py3: 128 | #ifdef ENABLE_PY3 129 | get_threads_ = py3::GetThreads; 130 | matched = true; 131 | #endif 132 | break; 133 | } 134 | if (!matched) { 135 | std::ostringstream os; 136 | os << "Target is Python " << static_cast(version) 137 | << ", which is not supported by this pyflame build."; 138 | throw FatalException(os.str()); 139 | } 140 | } 141 | 142 | std::vector PyFrob::GetThreads() { 143 | return get_threads_(pid_, addrs_); 144 | } 145 | } // namespace pyflame 146 | -------------------------------------------------------------------------------- /src/symbol.cc: -------------------------------------------------------------------------------- 1 | // Copyright 2016 Uber Technologies, Inc. 2 | // 3 | // Licensed under the Apache License, Version 2.0 (the "License"); 4 | // you may not use this file except in compliance with the License. 5 | // You may obtain a copy of the License at 6 | // 7 | // http://www.apache.org/licenses/LICENSE-2.0 8 | // 9 | // Unless required by applicable law or agreed to in writing, software 10 | // distributed under the License is distributed on an "AS IS" BASIS, 11 | // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | // See the License for the specific language governing permissions and 13 | // limitations under the License. 14 | 15 | #include "./symbol.h" 16 | 17 | #include 18 | #include 19 | #include 20 | #include 21 | #include 22 | 23 | #include 24 | #include 25 | #include 26 | 27 | #include "./posix.h" 28 | 29 | namespace pyflame { 30 | void ELF::Close() { 31 | if (addr_ != nullptr) { 32 | munmap(addr_, length_); 33 | addr_ = nullptr; 34 | } 35 | } 36 | 37 | // mmap the file 38 | void ELF::Open(const std::string &target, Namespace *ns) { 39 | Close(); 40 | int fd; 41 | if (ns != nullptr) { 42 | fd = ns->Open(target.c_str()); 43 | } else { 44 | fd = open(target.c_str(), O_RDONLY); 45 | } 46 | if (fd == -1) { 47 | std::ostringstream ss; 48 | ss << "Failed to open ELF file " << target << ": " << strerror(errno); 49 | throw FatalException(ss.str()); 50 | } 51 | length_ = lseek(fd, 0, SEEK_END); 52 | addr_ = mmap(nullptr, length_, PROT_READ, MAP_SHARED, fd, 0); 53 | pyflame::Close(fd); 54 | if (addr_ == MAP_FAILED) { 55 | std::ostringstream ss; 56 | ss << "Failed to mmap " << target << ": " << strerror(errno); 57 | throw FatalException(ss.str()); 58 | } 59 | 60 | if (hdr()->e_ident[EI_MAG0] != ELFMAG0 || 61 | hdr()->e_ident[EI_MAG1] != ELFMAG1 || 62 | hdr()->e_ident[EI_MAG2] != ELFMAG2 || 63 | hdr()->e_ident[EI_MAG3] != ELFMAG3) { 64 | std::ostringstream ss; 65 | ss << "File " << target << " does not have correct ELF magic header"; 66 | throw FatalException(ss.str()); 67 | } 68 | if (hdr()->e_ident[EI_CLASS] != ARCH_ELFCLASS) { 69 | throw FatalException("ELF class does not match host architecture"); 70 | } 71 | } 72 | 73 | void ELF::Parse() { 74 | // skip the first section since it must be of type SHT_NULL 75 | for (uint16_t i = 1; i < hdr()->e_shnum; i++) { 76 | const shdr_t *s = shdr(i); 77 | switch (s->sh_type) { 78 | case SHT_STRTAB: 79 | if (strcmp(strtab(s->sh_name), ".dynstr") == 0) { 80 | dynstr_ = i; 81 | } else if (strcmp(strtab(s->sh_name), ".strtab") == 0) { 82 | strtab_ = i; 83 | } 84 | break; 85 | case SHT_DYNSYM: 86 | dynsym_ = i; 87 | break; 88 | case SHT_DYNAMIC: 89 | dynamic_ = i; 90 | break; 91 | case SHT_SYMTAB: 92 | symtab_ = i; 93 | break; 94 | } 95 | } 96 | if (dynamic_ == -1) { 97 | throw FatalException("Failed to find section .dynamic"); 98 | } else if (dynstr_ == -1) { 99 | throw FatalException("Failed to find section .dynstr"); 100 | } else if (dynsym_ == -1) { 101 | throw FatalException("Failed to find section .dynsym"); 102 | } 103 | } 104 | 105 | std::vector ELF::NeededLibs() { 106 | // Get all of the strings 107 | std::vector needed; 108 | const shdr_t *s = shdr(dynamic_); 109 | const shdr_t *d = shdr(dynstr_); 110 | for (uint16_t i = 0; i < s->sh_size / s->sh_entsize; i++) { 111 | const dyn_t *dyn = 112 | reinterpret_cast(p() + s->sh_offset + i * s->sh_entsize); 113 | if (dyn->d_tag == DT_NEEDED) { 114 | needed.push_back( 115 | reinterpret_cast(p() + d->sh_offset + dyn->d_un.d_val)); 116 | } 117 | } 118 | return needed; 119 | } 120 | 121 | void ELF::WalkTable(int sym, int str, bool &have_version, PyVersion *version, PyAddresses &addrs) { 122 | const shdr_t *s = shdr(sym); 123 | const shdr_t *d = shdr(str); 124 | for (uint16_t i = 0; i < s->sh_size / s->sh_entsize; i++) { 125 | if (have_version && addrs.tstate_addr && addrs.interp_head_addr) { 126 | break; 127 | } 128 | 129 | const sym_t *sym = 130 | reinterpret_cast(p() + s->sh_offset + i * s->sh_entsize); 131 | const char *name = 132 | reinterpret_cast(p() + d->sh_offset + sym->st_name); 133 | if (!addrs.tstate_addr && strcmp(name, "_PyThreadState_Current") == 0) { 134 | addrs.tstate_addr = static_cast(sym->st_value); 135 | } else if (!addrs.interp_head_addr && strcmp(name, "interp_head") == 0) { 136 | addrs.interp_head_addr = static_cast(sym->st_value); 137 | } else if (!have_version) { 138 | if (strcmp(name, "PyString_Type") == 0) { 139 | // if we find PyString_Type, it's python 2 140 | have_version = true; 141 | *version = PyVersion::Py2; 142 | } else if (strcmp(name, "PyBytes_Type") == 0) { 143 | // if we find PyBytes_Type, it's python 3 144 | have_version = true; 145 | *version = PyVersion::Py3; 146 | } 147 | } 148 | } 149 | } 150 | 151 | PyAddresses ELF::GetAddresses(PyVersion *version) { 152 | bool have_version = false; 153 | PyAddresses addrs; 154 | WalkTable(dynsym_, dynstr_, have_version, version, addrs); 155 | if (symtab_ >= 0 && strtab_ >= 0) { 156 | WalkTable(symtab_, strtab_, have_version, version, addrs); 157 | } 158 | return addrs; 159 | } 160 | } // namespace pyflame 161 | -------------------------------------------------------------------------------- /src/frob.cc: -------------------------------------------------------------------------------- 1 | // Copyright 2016 Uber Technologies, Inc. 2 | // 3 | // Licensed under the Apache License, Version 2.0 (the "License"); 4 | // you may not use this file except in compliance with the License. 5 | // You may obtain a copy of the License at 6 | // 7 | // http://www.apache.org/licenses/LICENSE-2.0 8 | // 9 | // Unless required by applicable law or agreed to in writing, software 10 | // distributed under the License is distributed on an "AS IS" BASIS, 11 | // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | // See the License for the specific language governing permissions and 13 | // limitations under the License. 14 | 15 | // XXX: This file isn't compiled directly. It's included by frob2.cc or 16 | // frob3.cc, which define PYFLAME_PY_VERSION. Since Makefile.am for more 17 | // information. 18 | 19 | #include 20 | #include 21 | #include 22 | #include 23 | #include 24 | #include 25 | #include 26 | #include 27 | 28 | #include 29 | #include 30 | 31 | #include "./symbol.h" 32 | #include "./config.h" 33 | #include "./exc.h" 34 | #include "./ptrace.h" 35 | #include "./pyfrob.h" 36 | 37 | // why would this not be true idk 38 | static_assert(sizeof(long) == sizeof(void *), "wat platform r u on"); 39 | 40 | namespace pyflame { 41 | 42 | #if PYFLAME_PY_VERSION == 2 43 | namespace py2 { 44 | unsigned long StringSize(unsigned long addr) { 45 | return addr + offsetof(PyStringObject, ob_size); 46 | } 47 | 48 | unsigned long StringData(unsigned long addr) { 49 | return addr + offsetof(PyStringObject, ob_sval); 50 | } 51 | #elif PYFLAME_PY_VERSION == 3 52 | namespace py3 { 53 | unsigned long StringSize(unsigned long addr) { 54 | return addr + offsetof(PyVarObject, ob_size); 55 | } 56 | 57 | unsigned long StringData(unsigned long addr) { 58 | // this works only if the filename is all ascii *fingers crossed* 59 | return addr + sizeof(PyASCIIObject); 60 | } 61 | #else 62 | static_assert(false, "uh oh, bad PYFLAME_PY_VERSION"); 63 | #endif 64 | 65 | // Extract the line number from the code object. Python uses a compressed table 66 | // data structure to store line numbers. See: 67 | // 68 | // https://svn.python.org/projects/python/trunk/Objects/lnotab_notes.txt 69 | // 70 | // This is essentially an implementation of PyFrame_GetLineNumber / 71 | // PyCode_Addr2Line. 72 | size_t GetLine(pid_t pid, unsigned long frame, unsigned long f_code) { 73 | const long f_trace = PtracePeek(pid, frame + offsetof(_frame, f_trace)); 74 | if (f_trace) { 75 | return static_cast( 76 | PtracePeek(pid, frame + offsetof(_frame, f_lineno)) & 77 | std::numeric_limits::max()); 78 | } 79 | 80 | const int f_lasti = PtracePeek(pid, frame + offsetof(_frame, f_lasti)) & 81 | std::numeric_limits::max(); 82 | const long co_lnotab = 83 | PtracePeek(pid, f_code + offsetof(PyCodeObject, co_lnotab)); 84 | 85 | int size = 86 | PtracePeek(pid, StringSize(co_lnotab)) & std::numeric_limits::max(); 87 | int line = PtracePeek(pid, f_code + offsetof(PyCodeObject, co_firstlineno)) & 88 | std::numeric_limits::max(); 89 | const std::unique_ptr tbl = 90 | PtracePeekBytes(pid, StringData(co_lnotab), size); 91 | size /= 2; // since we increment twice in each loop iteration 92 | const uint8_t *p = tbl.get(); 93 | int addr = 0; 94 | while (--size >= 0) { 95 | addr += *p++; 96 | if (addr > f_lasti) { 97 | break; 98 | } 99 | line += *p++; 100 | } 101 | return static_cast(line); 102 | } 103 | 104 | // This method will fill the stack trace. Normally in the C API there are some 105 | // methods that you can use to extract the filename and line number from a frame 106 | // object. We implement the same logic here just using PTRACE_PEEKDATA. In 107 | // principle we could also execute code in the context of the process, but this 108 | // approach is harder to mess up. 109 | void FollowFrame(pid_t pid, unsigned long frame, std::vector *stack) { 110 | const long f_code = PtracePeek(pid, frame + offsetof(_frame, f_code)); 111 | const long co_filename = 112 | PtracePeek(pid, f_code + offsetof(PyCodeObject, co_filename)); 113 | const std::string filename = PtracePeekString(pid, StringData(co_filename)); 114 | const long co_name = 115 | PtracePeek(pid, f_code + offsetof(PyCodeObject, co_name)); 116 | const std::string name = PtracePeekString(pid, StringData(co_name)); 117 | stack->push_back({filename, name, GetLine(pid, frame, f_code)}); 118 | 119 | const long f_back = PtracePeek(pid, frame + offsetof(_frame, f_back)); 120 | if (f_back != 0) { 121 | FollowFrame(pid, f_back, stack); 122 | } 123 | } 124 | 125 | std::vector GetThreads(pid_t pid, PyAddresses addrs) { 126 | unsigned long istate = 0; 127 | 128 | // First try to get interpreter state via dereferencing _PyThreadState_Current. 129 | // This won't work if the main thread doesn't hold the GIL (_Current will be null). 130 | const long tstate = PtracePeek(pid, addrs.tstate_addr); 131 | if (tstate != 0) { 132 | istate = static_cast( 133 | PtracePeek(pid, tstate + offsetof(PyThreadState, interp))); 134 | // Secondly try to get it via the static interp_head symbol, if we managed to find it: 135 | // - interp_head is not strictly speaking part of the public API so it might get removed! 136 | // - interp_head is not part of the dynamic symbol table, so e.g. strip will drop it 137 | } else if (addrs.interp_head_addr != 0) { 138 | istate = static_cast( 139 | PtracePeek(pid, addrs.interp_head_addr)); 140 | } 141 | 142 | if (istate == 0) { 143 | return {}; 144 | } 145 | 146 | std::vector threads; 147 | unsigned long chain_next_addr = istate + offsetof(PyInterpreterState, tstate_head); 148 | do { 149 | const unsigned long chain_tstate = static_cast(PtracePeek(pid, chain_next_addr)); 150 | if (chain_tstate == 0) break; 151 | 152 | const long id = PtracePeek(pid, chain_tstate + offsetof(PyThreadState, thread_id)); 153 | const bool is_current = chain_tstate == addrs.tstate_addr; 154 | 155 | // dereference the frame 156 | const unsigned long frame_addr = static_cast( 157 | PtracePeek(pid, chain_tstate + offsetof(PyThreadState, frame))); 158 | 159 | std::vector stack; 160 | if (frame_addr != 0) { 161 | FollowFrame(pid, frame_addr, &stack); 162 | } 163 | 164 | threads.push_back(Thread(id, is_current, stack)); 165 | 166 | chain_next_addr = chain_tstate + offsetof(PyThreadState, next); 167 | } while (1); 168 | 169 | return threads; 170 | } 171 | } // namespace py2/py3 172 | } // namespace pyflame 173 | -------------------------------------------------------------------------------- /tests/dijkstra.py: -------------------------------------------------------------------------------- 1 | # Copyright 2016 Uber Technologies, Inc. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | 15 | import argparse 16 | import logging 17 | import os 18 | import random 19 | import sys 20 | import threading 21 | 22 | 23 | log = logging.getLogger('dijkstra') 24 | 25 | 26 | try: 27 | range = xrange 28 | except NameError: 29 | pass 30 | 31 | 32 | class Graph(object): 33 | """Representation of a sparse graph.""" 34 | 35 | def __init__(self, width, height): 36 | self.width = width 37 | self.height = height 38 | self.initial = None 39 | self.goal = None 40 | self.filled = set() 41 | 42 | @classmethod 43 | def generate(cls, width, height, count): 44 | graph = cls(width, height) 45 | for _ in range(count): 46 | while True: 47 | x, y = graph.random_unfilled() 48 | if (x, y) not in graph: 49 | break 50 | graph.fill_node(x, y) 51 | possibilities = [] 52 | for xx in (-1, 0, 1): 53 | for yy in (-1, 0, 1): 54 | possibilities.append((xx, yy)) 55 | added = 0 56 | random.shuffle(possibilities) 57 | for px, py in possibilities: 58 | xx = x + px 59 | yy = y + py 60 | if not graph.valid(xx, yy): 61 | continue 62 | if (xx, yy) not in graph: 63 | graph.fill_node(xx, yy) 64 | added += 1 65 | if added == 3: 66 | break 67 | x = xx 68 | y = yy 69 | graph.initial = graph.random_unfilled() 70 | while True: 71 | goal = graph.random_unfilled() 72 | if goal != graph.initial: 73 | graph.goal = goal 74 | break 75 | return graph 76 | 77 | def random_unfilled(self): 78 | while True: 79 | x = random.randint(0, self.width - 1) 80 | y = random.randint(0, self.height - 1) 81 | if (x, y) not in self.filled: 82 | return (x, y) 83 | 84 | def fill_node(self, x, y): 85 | self.filled.add((x, y)) 86 | 87 | def valid(self, x, y): 88 | if x < 0 or y < 0: 89 | return False 90 | if x >= self.width or y >= self.height: 91 | return False 92 | return True 93 | 94 | def dist(self, x, y): 95 | gx, gy = self.goal 96 | dx = gx - x 97 | dy = gy - y 98 | return dx*dx + dy*dy 99 | 100 | def __str__(self): 101 | return '%s(%d, %d, %s) initial=%s goal=%s' % ( 102 | self.__class__.__name__, self.width, self.height, 103 | sorted(self.filled), self.initial, self.goal) 104 | 105 | def __contains__(self, elem): 106 | return elem in self.filled 107 | 108 | 109 | def dijkstra(graph): 110 | solution = None 111 | via = {graph.initial: None} 112 | candidates = [] 113 | x, y = graph.initial 114 | for xx in (-1, 0, 1): 115 | for yy in (-1, 0, 1): 116 | px = x + xx 117 | py = y + yy 118 | point = (px, py) 119 | if graph.valid(px, py) and point not in graph and point not in via: 120 | d = graph.dist(px, py) 121 | candidates.append((d, point)) 122 | via[point] = graph.initial 123 | while candidates: 124 | candidates.sort(reverse=True) 125 | d, point = candidates.pop() 126 | if d == 0: 127 | solution = [point] 128 | while True: 129 | next_point = via[point] 130 | solution.append(next_point) 131 | if next_point == graph.initial: 132 | break 133 | else: 134 | point = next_point 135 | solution.reverse() 136 | break 137 | else: 138 | x, y = point 139 | for xx in (-1, 0, 1): 140 | for yy in (-1, 0, 1): 141 | px = x + xx 142 | py = y + yy 143 | new_point = (px, py) 144 | if graph.valid(px, py)\ 145 | and new_point not in graph\ 146 | and new_point not in via: 147 | d = graph.dist(px, py) 148 | candidates.append((d, new_point)) 149 | via[new_point] = point 150 | return solution 151 | 152 | 153 | def run(): 154 | """Run Dijkstra's algorithm.""" 155 | graph = Graph.generate(100, 100, 80) 156 | log.info('initial = %s', graph.initial) 157 | log.info('goal = %s', graph.goal) 158 | solution = dijkstra(graph) 159 | solution_len = 0 if solution is None else len(solution) 160 | log.info('solution = %s, len = %d', solution, solution_len) 161 | 162 | 163 | def run_times(quiet, times): 164 | """Run Dijkstra's algorithm in a loop.""" 165 | if not quiet: 166 | sys.stdout.write('%d\n' % (os.getpid(),)) 167 | sys.stdout.flush() 168 | if times <= 0: 169 | while True: 170 | run() 171 | else: 172 | for _ in range(times): 173 | run() 174 | 175 | 176 | def main(): 177 | parser = argparse.ArgumentParser() 178 | parser.add_argument('-q', '--quiet', action='store_true', 179 | help='Be quiet') 180 | parser.add_argument('-v', '--verbose', action='store_true', 181 | help='Be verbose') 182 | parser.add_argument('-t', '--threads', type=int, default=1, 183 | help='Number of threads') 184 | parser.add_argument('-n', '--num', type=int, default=0, 185 | help='Number of iterations') 186 | args = parser.parse_args() 187 | 188 | logging.basicConfig() 189 | if args.verbose: 190 | log.setLevel(logging.DEBUG) 191 | 192 | if args.threads == 1: 193 | run_times(args.quiet, args.num) 194 | else: 195 | threads = [] 196 | for _ in range(args.threads): 197 | t = threading.Thread(target=run_times, args=(args.quiet, args.num)) 198 | t.start() 199 | threads.append(t) 200 | for i, t in enumerate(threads): 201 | log.info('joined thread %d', i) 202 | t.join() 203 | 204 | 205 | if __name__ == '__main__': 206 | main() 207 | -------------------------------------------------------------------------------- /tests/test_end_to_end.py: -------------------------------------------------------------------------------- 1 | # Copyright 2016 Uber Technologies, Inc. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | 15 | import contextlib 16 | import pytest 17 | import re 18 | import subprocess 19 | 20 | 21 | IDLE_RE = re.compile(r'^\(idle\) \d+$') 22 | FLAMEGRAPH_RE = re.compile(r'^.+ \d+$') 23 | TS_IDLE_RE = re.compile(r'\(idle\)') 24 | # Matches strings of the form 25 | # './tests/sleeper.py::31;./tests/sleeper.py:main:26;' 26 | TS_FLAMEGRAPH_RE = re.compile(r'[^[^\d]+\d+;]*') 27 | TS_RE = re.compile(r'\d+') 28 | 29 | 30 | @contextlib.contextmanager 31 | def proc(argv, wait_for_pid=True): 32 | # start the process and wait for it to print its pid... we explicitly do 33 | # this instead of using the pid attribute so we can ensure that the process 34 | # is initialized 35 | proc = subprocess.Popen(argv, stdout=subprocess.PIPE) 36 | if wait_for_pid: 37 | proc.stdout.readline() 38 | 39 | try: 40 | yield proc 41 | finally: 42 | proc.kill() 43 | 44 | 45 | def python_proc(test_file): 46 | return proc(['python', './tests/%s' % (test_file,)]) 47 | 48 | 49 | @pytest.yield_fixture 50 | def dijkstra(): 51 | with python_proc('dijkstra.py') as p: 52 | yield p 53 | 54 | 55 | @pytest.yield_fixture 56 | def sleeper(): 57 | with python_proc('sleeper.py') as p: 58 | yield p 59 | 60 | 61 | @pytest.yield_fixture 62 | def exit_early(): 63 | with python_proc('exit_early.py') as p: 64 | yield p 65 | 66 | 67 | @pytest.yield_fixture 68 | def not_python(): 69 | with proc(['./tests/sleep.sh'], wait_for_pid=False) as p: 70 | yield p 71 | 72 | 73 | def communicate(proc): 74 | out, err = proc.communicate() 75 | if isinstance(out, bytes): 76 | out = out.decode('utf-8') 77 | if isinstance(err, bytes): 78 | err = err.decode('utf-8') 79 | return out, err 80 | 81 | 82 | def test_monitor(dijkstra): 83 | """Basic test for the monitor mode.""" 84 | proc = subprocess.Popen(['./src/pyflame', str(dijkstra.pid)], 85 | stdout=subprocess.PIPE, 86 | stderr=subprocess.PIPE, 87 | universal_newlines=True) 88 | out, err = communicate(proc) 89 | assert not err 90 | assert proc.returncode == 0 91 | lines = out.split('\n') 92 | assert lines.pop(-1) == '' # output should end in a newline 93 | for line in lines: 94 | assert FLAMEGRAPH_RE.match(line) is not None 95 | 96 | 97 | def test_idle(sleeper): 98 | """Basic test for idle processes.""" 99 | proc = subprocess.Popen(['./src/pyflame', str(sleeper.pid)], 100 | stdout=subprocess.PIPE, 101 | stderr=subprocess.PIPE, 102 | universal_newlines=True) 103 | out, err = communicate(proc) 104 | assert not err 105 | assert proc.returncode == 0 106 | lines = out.split('\n') 107 | assert lines.pop(-1) == '' # output should end in a newline 108 | has_idle = False 109 | for line in lines: 110 | assert FLAMEGRAPH_RE.match(line) is not None 111 | if IDLE_RE.match(line): 112 | has_idle = True 113 | assert has_idle 114 | 115 | 116 | def test_exclude_idle(sleeper): 117 | """Basic test for idle processes.""" 118 | proc = subprocess.Popen(['./src/pyflame', '-x', str(sleeper.pid)], 119 | stdout=subprocess.PIPE, 120 | stderr=subprocess.PIPE, 121 | universal_newlines=True) 122 | out, err = communicate(proc) 123 | assert not err 124 | assert proc.returncode == 0 125 | lines = out.split('\n') 126 | assert lines.pop(-1) == '' # output should end in a newline 127 | for line in lines: 128 | assert FLAMEGRAPH_RE.match(line) is not None 129 | assert not IDLE_RE.match(line) 130 | 131 | 132 | def test_exit_early(exit_early): 133 | proc = subprocess.Popen(['./src/pyflame', '-s', '10', str(exit_early.pid)], 134 | stdout=subprocess.PIPE, 135 | stderr=subprocess.PIPE) 136 | out, err = communicate(proc) 137 | assert not err 138 | assert proc.returncode == 0 139 | lines = out.split('\n') 140 | assert lines.pop(-1) == '' # output should end in a newline 141 | for line in lines: 142 | assert FLAMEGRAPH_RE.match(line) or IDLE_RE.match(line) 143 | 144 | 145 | def test_sample_not_python(not_python): 146 | proc = subprocess.Popen(['./src/pyflame', str(not_python.pid)], 147 | stdout=subprocess.PIPE, 148 | stderr=subprocess.PIPE) 149 | out, err = communicate(proc) 150 | assert not out 151 | assert err.startswith('Failed to locate libpython') 152 | assert proc.returncode == 1 153 | 154 | 155 | def test_trace(): 156 | proc = subprocess.Popen(['./src/pyflame', '-t', 157 | 'python', 'tests/exit_early.py', '-s'], 158 | stdout=subprocess.PIPE, 159 | stderr=subprocess.PIPE) 160 | out, err = communicate(proc) 161 | assert not err 162 | assert proc.returncode == 0 163 | lines = out.split('\n') 164 | assert lines.pop(-1) == '' # output should end in a newline 165 | for line in lines: 166 | assert FLAMEGRAPH_RE.match(line) or IDLE_RE.match(line) 167 | 168 | 169 | def test_trace_not_python(): 170 | proc = subprocess.Popen(['./src/pyflame', '-t', './tests/sleep.sh'], 171 | stdout=subprocess.PIPE, 172 | stderr=subprocess.PIPE) 173 | out, err = communicate(proc) 174 | assert not out 175 | assert err.startswith('Failed to locate libpython') 176 | assert proc.returncode == 1 177 | 178 | 179 | def test_pyflame_a_pyflame(): 180 | proc = subprocess.Popen(['./src/pyflame', '-t', './src/pyflame'], 181 | stdout=subprocess.PIPE, 182 | stderr=subprocess.PIPE) 183 | out, err = communicate(proc) 184 | assert not out 185 | assert err.startswith('You tried to pyflame a pyflame') 186 | assert proc.returncode == 1 187 | 188 | 189 | def test_pyflame_nonexistent_file(): 190 | proc = subprocess.Popen(['./src/pyflame', '-t', '/no/such/file'], 191 | stdout=subprocess.PIPE, 192 | stderr=subprocess.PIPE) 193 | out, err = communicate(proc) 194 | assert not out 195 | assert 'Child process exited with status' in err 196 | assert proc.returncode == 1 197 | 198 | 199 | def test_trace_no_arg(): 200 | proc = subprocess.Popen(['./src/pyflame', '-t'], 201 | stdout=subprocess.PIPE, 202 | stderr=subprocess.PIPE) 203 | out, err = communicate(proc) 204 | assert not out 205 | assert err.startswith('Usage: ') 206 | assert proc.returncode == 1 207 | 208 | 209 | def test_sample_no_arg(): 210 | proc = subprocess.Popen(['./src/pyflame'], 211 | stdout=subprocess.PIPE, 212 | stderr=subprocess.PIPE) 213 | out, err = communicate(proc) 214 | assert not out 215 | assert err.startswith('Usage: ') 216 | assert proc.returncode == 1 217 | 218 | 219 | def test_sample_extra_args(): 220 | proc = subprocess.Popen(['./src/pyflame', 'foo', 'bar'], 221 | stdout=subprocess.PIPE, 222 | stderr=subprocess.PIPE) 223 | out, err = communicate(proc) 224 | assert not out 225 | assert err.startswith('Usage: ') 226 | assert proc.returncode == 1 227 | 228 | 229 | @pytest.mark.parametrize('pid', [(1,), (0,)]) 230 | def test_permission_error(pid): 231 | # pid 1 = EPERM 232 | # pid 0 = ESRCH 233 | proc = subprocess.Popen(['./src/pyflame', str(pid)], 234 | stdout=subprocess.PIPE, 235 | stderr=subprocess.PIPE) 236 | out, err = communicate(proc) 237 | assert not out 238 | assert err.startswith('Failed to attach to PID') 239 | assert proc.returncode == 1 240 | 241 | 242 | def test_include_ts(sleeper): 243 | """Basic test for timestamp processes.""" 244 | proc = subprocess.Popen(['./src/pyflame', '-T', str(sleeper.pid)], 245 | stdout=subprocess.PIPE, 246 | stderr=subprocess.PIPE, 247 | universal_newlines=True) 248 | out, err = proc.communicate() 249 | assert not err 250 | assert proc.returncode == 0 251 | lines = out.split('\n') 252 | assert lines.pop(-1) == '' # output should end in a newline 253 | for line in lines: 254 | assert (TS_FLAMEGRAPH_RE.match(line) or 255 | TS_RE.match(line) or TS_IDLE_RE.match(line)) 256 | 257 | 258 | def test_include_ts_exclude_idle(sleeper): 259 | """Basic test for timestamp processes.""" 260 | proc = subprocess.Popen(['./src/pyflame', '-T', '-x', str(sleeper.pid)], 261 | stdout=subprocess.PIPE, 262 | stderr=subprocess.PIPE, 263 | universal_newlines=True) 264 | out, err = proc.communicate() 265 | assert not err 266 | assert proc.returncode == 0 267 | lines = out.split('\n') 268 | assert lines.pop(-1) == '' # output should end in a newline 269 | for line in lines: 270 | assert not TS_IDLE_RE.match(line) 271 | assert (TS_FLAMEGRAPH_RE.match(line) or TS_RE.match(line)) 272 | -------------------------------------------------------------------------------- /src/pyflame.cc: -------------------------------------------------------------------------------- 1 | // Copyright 2016 Uber Technologies, Inc. 2 | // 3 | // Licensed under the Apache License, Version 2.0 (the "License"); 4 | // you may not use this file except in compliance with the License. 5 | // You may obtain a copy of the License at 6 | // 7 | // http://www.apache.org/licenses/LICENSE-2.0 8 | // 9 | // Unless required by applicable law or agreed to in writing, software 10 | // distributed under the License is distributed on an "AS IS" BASIS, 11 | // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | // See the License for the specific language governing permissions and 13 | // limitations under the License. 14 | 15 | #include 16 | #include 17 | #include 18 | 19 | #include 20 | #include 21 | #include 22 | #include 23 | #include 24 | #include 25 | #include 26 | #include 27 | #include 28 | 29 | #include "./config.h" 30 | #include "./exc.h" 31 | #include "./thread.h" 32 | #include "./ptrace.h" 33 | #include "./pyfrob.h" 34 | #include "./version.h" 35 | 36 | // FIXME: this logic should be moved to configure.ac 37 | #if !defined(ENABLE_PY2) && !defined(ENABLE_PY3) 38 | static_assert(false, "Need Python2 or Python3 support to build"); 39 | #endif 40 | 41 | using namespace pyflame; 42 | 43 | namespace { 44 | const char usage_str[] = 45 | ("Usage: pyflame [options] \n" 46 | " pyflame [-t|--trace] command arg1 arg2...\n" 47 | "\n" 48 | "General Options:\n" 49 | " -h, --help Show help\n" 50 | " -s, --seconds=SECS How many seconds to run for (default 1)\n" 51 | " -r, --rate=RATE Sample rate, as a fractional value of seconds " 52 | "(default 0.001)\n" 53 | " -t, --trace Trace a child process\n" 54 | " -T, --timestamp Include timestamps for each stacktrace\n" 55 | " -v, --version Show the version\n" 56 | " -x, --exclude-idle Exclude idle time from statistics\n"); 57 | 58 | typedef std::unordered_map buckets_t; 59 | 60 | // Prints all stack traces 61 | void PrintFrames(const std::vector &call_stacks, size_t idle) { 62 | if (idle) { 63 | std::cout << "(idle) " << idle << "\n"; 64 | } 65 | // Put the call stacks into buckets 66 | buckets_t buckets; 67 | for (const auto &call_stack : call_stacks) { 68 | auto bucket = buckets.find(call_stack.frames); 69 | if (bucket == buckets.end()) { 70 | buckets.insert(bucket, {call_stack.frames, 1}); 71 | } else { 72 | bucket->second++; 73 | } 74 | } 75 | // process the frames 76 | for (const auto &kv : buckets) { 77 | if (kv.first.empty()) { 78 | std::cerr << "fatal error\n"; 79 | return; 80 | } 81 | auto last = kv.first.rend(); 82 | last--; 83 | for (auto it = kv.first.rbegin(); it != last; ++it) { 84 | std::cout << *it << ";"; 85 | } 86 | std::cout << *last << " " << kv.second << "\n"; 87 | } 88 | } 89 | 90 | // Prints all stack traces with timestamps 91 | void PrintFramesTS(const std::vector &call_stacks) { 92 | for (const auto &call_stack : call_stacks) { 93 | std::cout << std::chrono::duration_cast( 94 | call_stack.ts.time_since_epoch()) 95 | .count() 96 | << "\n"; 97 | // Handle idle 98 | if (call_stack.frames.empty()) { 99 | std::cout << "(idle)\n"; 100 | continue; 101 | } 102 | // Print the call stack 103 | for (auto it = call_stack.frames.rbegin(); it != call_stack.frames.rend(); 104 | ++it) { 105 | std::cout << *it << ";"; 106 | } 107 | std::cout << "\n"; 108 | } 109 | } 110 | 111 | inline bool IsPyflame(const std::string &str) { 112 | return str.find("pyflame") != std::string::npos; 113 | } 114 | } // namespace 115 | 116 | int main(int argc, char **argv) { 117 | bool trace = false; 118 | bool include_idle = true; 119 | bool include_ts = false; 120 | double seconds = 1; 121 | double sample_rate = 0.001; 122 | for (;;) { 123 | static struct option long_options[] = { 124 | {"help", no_argument, 0, 'h'}, 125 | {"rate", required_argument, 0, 'r'}, 126 | {"seconds", required_argument, 0, 's'}, 127 | {"trace", no_argument, 0, 't'}, 128 | {"timestamp", no_argument, 0, 'T'}, 129 | {"version", no_argument, 0, 'v'}, 130 | {"exclude-idle", no_argument, 0, 'x'}, 131 | {0, 0, 0, 0}}; 132 | int option_index = 0; 133 | int c = getopt_long(argc, argv, "hr:s:tTvx", long_options, &option_index); 134 | if (c == -1) { 135 | break; 136 | } 137 | switch (c) { 138 | case 0: 139 | if (long_options[option_index].flag != 0) { 140 | // if the option set a flag, do nothing 141 | break; 142 | } 143 | break; 144 | case 'h': 145 | std::cout << usage_str; 146 | return 0; 147 | break; 148 | case 'r': 149 | sample_rate = std::stod(optarg); 150 | break; 151 | case 's': 152 | seconds = std::stod(optarg); 153 | break; 154 | case 't': 155 | trace = true; 156 | seconds = -1; 157 | goto finish_arg_parse; 158 | break; 159 | case 'T': 160 | include_ts = true; 161 | break; 162 | case 'v': 163 | std::cout << PACKAGE_STRING << "\n\n"; 164 | std::cout << kBuildNote << "\n"; 165 | return 0; 166 | break; 167 | case 'x': 168 | include_idle = false; 169 | break; 170 | case '?': 171 | // getopt_long should already have printed an error message 172 | break; 173 | default: 174 | abort(); 175 | } 176 | } 177 | finish_arg_parse: 178 | const std::chrono::microseconds interval{ 179 | static_cast(sample_rate * 1000000)}; 180 | pid_t pid; 181 | if (trace) { 182 | if (optind == argc) { 183 | std::cerr << usage_str; 184 | return 1; 185 | } 186 | if (IsPyflame(argv[optind])) { 187 | std::cerr << "You tried to pyflame a pyflame, naughty!\n"; 188 | return 1; 189 | } 190 | // In trace mode, all of the remaining arguments are a command to run. We 191 | // fork and have the child run the command; the parent traces. 192 | pid = fork(); 193 | if (pid == -1) { 194 | perror("fork()"); 195 | return 1; 196 | } else if (pid == 0) { 197 | // child; run the command 198 | if (execvp(argv[optind], argv + optind)) { 199 | perror("execlp()"); 200 | return 1; 201 | } 202 | } else { 203 | // parent; wait for the child to not be pyflame 204 | std::ostringstream os; 205 | os << "/proc/" << pid << "/comm"; 206 | for (;;) { 207 | int status; 208 | pid_t pid_status = waitpid(pid, &status, WNOHANG); 209 | if (pid_status == -1) { 210 | perror("waitpid()"); 211 | return 1; 212 | } else if (pid_status > 0) { 213 | std::cerr << "Child process exited with status " 214 | << WEXITSTATUS(pid_status) << "\n"; 215 | return 1; 216 | } 217 | 218 | std::ifstream ifs(os.str()); 219 | std::string line; 220 | std::getline(ifs, line); 221 | 222 | // If the child is not named pyflame we break, otherwise we sleep and 223 | // retry. Hopefully this is not an infinite loop, since we already 224 | // checked that the child should not have been pyflame. All bets are off 225 | // if the child tries to be extra pathological (e.g. immediately exec a 226 | // pyflame). 227 | if (!IsPyflame(line)) { 228 | break; 229 | } 230 | std::this_thread::sleep_for(interval); 231 | } 232 | } 233 | } else { 234 | // there should be one remaining argument: the pid to trace 235 | if (optind != argc - 1) { 236 | std::cerr << usage_str; 237 | return 1; 238 | } 239 | pid = static_cast(std::strtol(argv[argc - 1], nullptr, 10)); 240 | if (pid > std::numeric_limits::max() || 241 | pid < std::numeric_limits::min()) { 242 | std::cerr << "PID " << pid << " is out of valid PID range.\n"; 243 | return 1; 244 | } 245 | } 246 | 247 | std::vector call_stacks; 248 | size_t idle = 0; 249 | try { 250 | PtraceAttach(pid); 251 | PyFrob frobber(pid); 252 | frobber.DetectPython(); 253 | 254 | const std::chrono::microseconds interval{ 255 | static_cast(sample_rate * 1000000)}; 256 | bool check_end = seconds >= 0; 257 | auto end = std::chrono::system_clock::now() + 258 | std::chrono::microseconds(static_cast(seconds * 1000000)); 259 | for (;;) { 260 | auto now = std::chrono::system_clock::now(); 261 | std::vector threads = frobber.GetThreads(); 262 | /*for (const auto &thread : threads) { 263 | std::cout << thread << std::endl; 264 | }*/ 265 | auto current = std::find_if(threads.begin(), threads.end(), [](Thread& thread) -> bool { return thread.is_current(); }); 266 | if (current == threads.end()) { 267 | if (include_idle) { 268 | idle++; 269 | // Time stamp empty call stacks only if required. Since lots of time 270 | // the process will be idle, this is a good optimization to have 271 | if (include_ts) { 272 | call_stacks.push_back({now, {}}); 273 | } 274 | } 275 | } else { 276 | call_stacks.push_back({now, current->frames()}); 277 | } 278 | 279 | if ((check_end) && (now + interval >= end)) { 280 | break; 281 | } 282 | PtraceDetach(pid); 283 | std::this_thread::sleep_for(interval); 284 | PtraceAttach(pid); 285 | } 286 | if (!include_ts) { 287 | PrintFrames(call_stacks, idle); 288 | } else { 289 | PrintFramesTS(call_stacks); 290 | } 291 | } catch (const PtraceException &exc) { 292 | // If the process terminates early then we just print the stack traces up 293 | // until that point in time. 294 | if (!call_stacks.empty() || idle) { 295 | if (!include_ts) { 296 | PrintFrames(call_stacks, idle); 297 | } else { 298 | PrintFramesTS(call_stacks); 299 | } 300 | } else { 301 | std::cerr << exc.what() << std::endl; 302 | return 1; 303 | } 304 | } catch (const std::exception &exc) { 305 | std::cerr << exc.what() << std::endl; 306 | return 1; 307 | } 308 | return 0; 309 | } 310 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | 2 | Apache License 3 | Version 2.0, January 2004 4 | http://www.apache.org/licenses/ 5 | 6 | TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 7 | 8 | 1. Definitions. 9 | 10 | "License" shall mean the terms and conditions for use, reproduction, 11 | and distribution as defined by Sections 1 through 9 of this document. 12 | 13 | "Licensor" shall mean the copyright owner or entity authorized by 14 | the copyright owner that is granting the License. 15 | 16 | "Legal Entity" shall mean the union of the acting entity and all 17 | other entities that control, are controlled by, or are under common 18 | control with that entity. For the purposes of this definition, 19 | "control" means (i) the power, direct or indirect, to cause the 20 | direction or management of such entity, whether by contract or 21 | otherwise, or (ii) ownership of fifty percent (50%) or more of the 22 | outstanding shares, or (iii) beneficial ownership of such entity. 23 | 24 | "You" (or "Your") shall mean an individual or Legal Entity 25 | exercising permissions granted by this License. 26 | 27 | "Source" form shall mean the preferred form for making modifications, 28 | including but not limited to software source code, documentation 29 | source, and configuration files. 30 | 31 | "Object" form shall mean any form resulting from mechanical 32 | transformation or translation of a Source form, including but 33 | not limited to compiled object code, generated documentation, 34 | and conversions to other media types. 35 | 36 | "Work" shall mean the work of authorship, whether in Source or 37 | Object form, made available under the License, as indicated by a 38 | copyright notice that is included in or attached to the work 39 | (an example is provided in the Appendix below). 40 | 41 | "Derivative Works" shall mean any work, whether in Source or Object 42 | form, that is based on (or derived from) the Work and for which the 43 | editorial revisions, annotations, elaborations, or other modifications 44 | represent, as a whole, an original work of authorship. For the purposes 45 | of this License, Derivative Works shall not include works that remain 46 | separable from, or merely link (or bind by name) to the interfaces of, 47 | the Work and Derivative Works thereof. 48 | 49 | "Contribution" shall mean any work of authorship, including 50 | the original version of the Work and any modifications or additions 51 | to that Work or Derivative Works thereof, that is intentionally 52 | submitted to Licensor for inclusion in the Work by the copyright owner 53 | or by an individual or Legal Entity authorized to submit on behalf of 54 | the copyright owner. For the purposes of this definition, "submitted" 55 | means any form of electronic, verbal, or written communication sent 56 | to the Licensor or its representatives, including but not limited to 57 | communication on electronic mailing lists, source code control systems, 58 | and issue tracking systems that are managed by, or on behalf of, the 59 | Licensor for the purpose of discussing and improving the Work, but 60 | excluding communication that is conspicuously marked or otherwise 61 | designated in writing by the copyright owner as "Not a Contribution." 62 | 63 | "Contributor" shall mean Licensor and any individual or Legal Entity 64 | on behalf of whom a Contribution has been received by Licensor and 65 | subsequently incorporated within the Work. 66 | 67 | 2. Grant of Copyright License. Subject to the terms and conditions of 68 | this License, each Contributor hereby grants to You a perpetual, 69 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 70 | copyright license to reproduce, prepare Derivative Works of, 71 | publicly display, publicly perform, sublicense, and distribute the 72 | Work and such Derivative Works in Source or Object form. 73 | 74 | 3. Grant of Patent License. Subject to the terms and conditions of 75 | this License, each Contributor hereby grants to You a perpetual, 76 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 77 | (except as stated in this section) patent license to make, have made, 78 | use, offer to sell, sell, import, and otherwise transfer the Work, 79 | where such license applies only to those patent claims licensable 80 | by such Contributor that are necessarily infringed by their 81 | Contribution(s) alone or by combination of their Contribution(s) 82 | with the Work to which such Contribution(s) was submitted. If You 83 | institute patent litigation against any entity (including a 84 | cross-claim or counterclaim in a lawsuit) alleging that the Work 85 | or a Contribution incorporated within the Work constitutes direct 86 | or contributory patent infringement, then any patent licenses 87 | granted to You under this License for that Work shall terminate 88 | as of the date such litigation is filed. 89 | 90 | 4. Redistribution. You may reproduce and distribute copies of the 91 | Work or Derivative Works thereof in any medium, with or without 92 | modifications, and in Source or Object form, provided that You 93 | meet the following conditions: 94 | 95 | (a) You must give any other recipients of the Work or 96 | Derivative Works a copy of this License; and 97 | 98 | (b) You must cause any modified files to carry prominent notices 99 | stating that You changed the files; and 100 | 101 | (c) You must retain, in the Source form of any Derivative Works 102 | that You distribute, all copyright, patent, trademark, and 103 | attribution notices from the Source form of the Work, 104 | excluding those notices that do not pertain to any part of 105 | the Derivative Works; and 106 | 107 | (d) If the Work includes a "NOTICE" text file as part of its 108 | distribution, then any Derivative Works that You distribute must 109 | include a readable copy of the attribution notices contained 110 | within such NOTICE file, excluding those notices that do not 111 | pertain to any part of the Derivative Works, in at least one 112 | of the following places: within a NOTICE text file distributed 113 | as part of the Derivative Works; within the Source form or 114 | documentation, if provided along with the Derivative Works; or, 115 | within a display generated by the Derivative Works, if and 116 | wherever such third-party notices normally appear. The contents 117 | of the NOTICE file are for informational purposes only and 118 | do not modify the License. You may add Your own attribution 119 | notices within Derivative Works that You distribute, alongside 120 | or as an addendum to the NOTICE text from the Work, provided 121 | that such additional attribution notices cannot be construed 122 | as modifying the License. 123 | 124 | You may add Your own copyright statement to Your modifications and 125 | may provide additional or different license terms and conditions 126 | for use, reproduction, or distribution of Your modifications, or 127 | for any such Derivative Works as a whole, provided Your use, 128 | reproduction, and distribution of the Work otherwise complies with 129 | the conditions stated in this License. 130 | 131 | 5. Submission of Contributions. Unless You explicitly state otherwise, 132 | any Contribution intentionally submitted for inclusion in the Work 133 | by You to the Licensor shall be under the terms and conditions of 134 | this License, without any additional terms or conditions. 135 | Notwithstanding the above, nothing herein shall supersede or modify 136 | the terms of any separate license agreement you may have executed 137 | with Licensor regarding such Contributions. 138 | 139 | 6. Trademarks. This License does not grant permission to use the trade 140 | names, trademarks, service marks, or product names of the Licensor, 141 | except as required for reasonable and customary use in describing the 142 | origin of the Work and reproducing the content of the NOTICE file. 143 | 144 | 7. Disclaimer of Warranty. Unless required by applicable law or 145 | agreed to in writing, Licensor provides the Work (and each 146 | Contributor provides its Contributions) on an "AS IS" BASIS, 147 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 148 | implied, including, without limitation, any warranties or conditions 149 | of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A 150 | PARTICULAR PURPOSE. You are solely responsible for determining the 151 | appropriateness of using or redistributing the Work and assume any 152 | risks associated with Your exercise of permissions under this License. 153 | 154 | 8. Limitation of Liability. In no event and under no legal theory, 155 | whether in tort (including negligence), contract, or otherwise, 156 | unless required by applicable law (such as deliberate and grossly 157 | negligent acts) or agreed to in writing, shall any Contributor be 158 | liable to You for damages, including any direct, indirect, special, 159 | incidental, or consequential damages of any character arising as a 160 | result of this License or out of the use or inability to use the 161 | Work (including but not limited to damages for loss of goodwill, 162 | work stoppage, computer failure or malfunction, or any and all 163 | other commercial damages or losses), even if such Contributor 164 | has been advised of the possibility of such damages. 165 | 166 | 9. Accepting Warranty or Additional Liability. While redistributing 167 | the Work or Derivative Works thereof, You may choose to offer, 168 | and charge a fee for, acceptance of support, warranty, indemnity, 169 | or other liability obligations and/or rights consistent with this 170 | License. However, in accepting such obligations, You may act only 171 | on Your own behalf and on Your sole responsibility, not on behalf 172 | of any other Contributor, and only if You agree to indemnify, 173 | defend, and hold each Contributor harmless for any liability 174 | incurred by, or claims asserted against, such Contributor by reason 175 | of your accepting any such warranty or additional liability. 176 | 177 | END OF TERMS AND CONDITIONS 178 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | [![Build Status](https://api.travis-ci.org/uber/pyflame.svg?branch=master)](https://travis-ci.org/uber/pyflame) 2 | 3 | # Pyflame: A Ptracing Profiler For Python 4 | 5 | Pyflame is a tool for 6 | generating [flame graphs](https://github.com/brendangregg/FlameGraph) for Python 7 | processes. Pyflame is different from existing Python profilers because it 8 | doesn't require explicit instrumentation: it will work with any running Python 9 | process! Pyflame works by using 10 | the [ptrace(2)](http://man7.org/linux/man-pages/man2/ptrace.2.html) system call 11 | to analyze the currently-executing stack trace for a Python process. 12 | 13 | Learn more by reading 14 | [the Uber Engineering blog post about Pyflame](http://eng.uber.com/pyflame/). 15 | 16 | ![pyflame](https://cloud.githubusercontent.com/assets/2734/17949703/8ef7d08c-6a0b-11e6-8bbd-41f82086d862.png) 17 | 18 | ## Installing 19 | 20 | To build Pyflame you will need a C++ compiler with basic C++11 support. Pyflame 21 | is known to compile on versions of GCC as old as GCC 4.6. You'll also need GNU 22 | Autotools ([GNU Autoconf](https://www.gnu.org/software/autoconf/autoconf.html) 23 | and [GNU Automake](https://www.gnu.org/software/automake/automake.html)) if 24 | you're building from the Git repository. 25 | 26 | From Git you would compile like so: 27 | 28 | ```bash 29 | ./autogen.sh 30 | ./configure # Plus any options like --prefix. 31 | make 32 | make install 33 | ``` 34 | 35 | ### Fedora 36 | 37 | The following command should install the necessary packages to build on Fedora: 38 | 39 | ```bash 40 | # Install build dependencies on Fedora. 41 | sudo dnf install autoconf automake gcc-c++ python-devel libtool 42 | ``` 43 | 44 | ### Debian 45 | 46 | The following command should install the necessary packages to build on Debian 47 | (or Ubuntu): 48 | 49 | ```bash 50 | # Install build dependencies on Debian or Ubuntu. 51 | sudo apt-get install autoconf automake autotools-dev g++ pkg-config python-dev libtool 52 | ``` 53 | 54 | If you'd like to build a Debian package there's already a `debian/` directory at 55 | the root of this project. We'd like to remove this, as per the 56 | [upstream Debian packaging guidelines](https://wiki.debian.org/UpstreamGuide). 57 | If you can help get this project packaged in Debian please let us know. 58 | 59 | ### Arch Linux 60 | 61 | You can install pyflame from [AUR](https://aur.archlinux.org/packages/pyflame-git/). 62 | 63 | ## Usage 64 | 65 | After compiling Pyflame you'll get a small executable called `pyflame` (which 66 | will be in the `src/` directory if you haven't run `make install`). The most 67 | basic usage is: 68 | 69 | ```bash 70 | # Profile PID for 1s, sampling every 1ms. 71 | pyflame PID 72 | ``` 73 | 74 | The `pyflame` command will send data to stdout that is suitable for using with 75 | Brendan Gregg's `flamegraph.pl` tool (which you can 76 | get [here](https://github.com/brendangregg/FlameGraph)). Therefore a typical 77 | command pipeline might be like this: 78 | 79 | ```bash 80 | # Generate flame graph for pid 12345; assumes flamegraph.pl is in your $PATH. 81 | pyflame 12345 | flamegraph.pl > myprofile.svg 82 | ``` 83 | 84 | You can also change the sample time and sampling frequency: 85 | 86 | ```bash 87 | # Profile PID for 60 seconds, sampling every 10ms. 88 | pyflame -s 60 -r 0.10 PID 89 | ``` 90 | 91 | ### Trace Mode 92 | 93 | Sometimes you want to trace a process from start to finish. An example would be 94 | tracing the run of a test suite. Pyflame supports this use case. To use it, you 95 | invoke Pyflame like this: 96 | 97 | ```bash 98 | # Trace a given command until completion. 99 | pyflame [regular pyflame options] -t command arg1 arg2... 100 | ``` 101 | 102 | Frequently the value of `command` will actually be `python`, but it could be 103 | something else like `uwsgi` or `py.test`. For instance, here's how Pyflame can 104 | be used to trace its own test suite: 105 | 106 | ```bash 107 | # Trace the Pyflame test suite, a.k.a. pyflameception! 108 | pyflame -t py.test tests/ 109 | ``` 110 | 111 | Beware that when using the trace mode the stdout/stderr of the pyflame process 112 | and the traced process will be mixed. This means if the traced process sends 113 | data to stdout you may need to filter it somehow before sending the output to 114 | `flamegraph.pl`. 115 | 116 | ### Timestamp ("Flame Chart") Mode 117 | 118 | Pyflame can also generate data with timestamps which can be used to 119 | generate ["flame charts"](https://addyosmani.com/blog/devtools-flame-charts/) 120 | that can be viewed in Chrome. This is controlled with the `-T` option. 121 | 122 | Use `utils/flame-chart-json` to generate the JSON data required for viewing 123 | Flame Charts using the Chrome CPU profiler. 124 | 125 | ```bash 126 | Usage: cat | flame-chart-json > .cpuprofile 127 | (or) pyflame [regular pyflame options] | flame-chart-json > .cpuprofile 128 | ``` 129 | 130 | Then load the resulting .cpuprofile file from chrome CPU profiler to view Flame Chart. 131 | 132 | ## FAQ 133 | 134 | ### What Is "(idle)" Time? 135 | 136 | From time to time the Python interpreter will have nothing to do other than wait 137 | for I/O to complete. This will typically happen when the Python interpreter is 138 | waiting for network operations to finish. In this scenario Pyflame will report 139 | the time as "idle". 140 | 141 | If you don't want to include this time you can use the invocation `pyflame -x`. 142 | 143 | ### Are BSD / OS X / macOS Supported? 144 | 145 | No, these aren't supported. Someone who is proficient with low-level C 146 | programming can probably get BSD to work, as described in issue #3. It is 147 | probably much more difficult to adapt this code to work on OS X/macOS since the 148 | current code assumes that the host 149 | uses [ELF](https://en.wikipedia.org/wiki/Executable_and_Linkable_Format) files 150 | as the executable file format for the Python interpreter. 151 | 152 | ### What Are These Ptrace Permissions Errors? 153 | 154 | Because it's so powerful, the `ptrace(2)` system call is locked down by default 155 | in various situations by different Linux distributions. In order to use ptrace 156 | these conditions must be met: 157 | 158 | * You must have the 159 | [`SYS_PTRACE` capability](http://man7.org/linux/man-pages/man7/capabilities.7.html) (which 160 | is denied by default within Docker images). 161 | * The kernel must not have `kernel.yama.ptrace_scope` set to a value that is 162 | too restrictive. 163 | 164 | In both scenarios you'll also find that `strace` and `gdb` do not work as 165 | expected. 166 | 167 | #### Ptrace Errors Within Docker Containers 168 | 169 | By default Docker images do not have the `SYS_PTRACE` capability. When you 170 | invoke `docker run` try using the `--cap-add SYS_PTRACE` option: 171 | 172 | ```bash 173 | # Allows processes within the Docker container to use ptrace. 174 | docker run --cap-add SYS_PTRACE ... 175 | ``` 176 | 177 | You can also use [capsh(1)](http://man7.org/linux/man-pages/man1/capsh.1.html) 178 | to list your current capabilities: 179 | 180 | ```bash 181 | # You should see cap_sys_ptrace in the "Bounding set". 182 | capsh --print 183 | ``` 184 | 185 | Further note that by design you do not need to run Pyflame from within a Docker 186 | container. If you have sufficient permissions (i.e. you are root, or the same 187 | UID as the Docker process) Pyflame can be run from outside of the container and 188 | inspect a process inside the container. That said, Pyflame will certainly work 189 | within containers if that's how you want to use it. 190 | 191 | #### Ptrace Errors Outside Docker Containers Or When Not Using Docker 192 | 193 | If you're not in a Docker container, or you're not using Docker at all, ptrace 194 | permissions errors are likely related to you having too restrictive a value set 195 | for the `kernel.yama.ptrace_scope` sysfs knob. 196 | 197 | Debian Jessie ships with `ptrace_scope` set to 1 by default, which will prevent 198 | unprivileged users from attaching to already running processes. 199 | 200 | To see the current value of this setting: 201 | 202 | ```bash 203 | # Prints the current value for the ptrace_scope setting. 204 | sysctl kernel.yama.ptrace_scope 205 | ``` 206 | 207 | If you see a value other than 0 you may want to change it. Note that by doing 208 | this you'll affect the security of your system. Please read 209 | [the relevant kernel documentation](https://www.kernel.org/doc/Documentation/security/Yama.txt) 210 | for a comprehensive discussion of the possible settings and what you're 211 | changing. If you want to completely disable the ptrace settings and get 212 | "classic" permissions (i.e. root can ptrace anything, unprivileged users can 213 | ptrace processes with the same user id) then use: 214 | 215 | ```bash 216 | # Use this if you want "classic" ptrace permissions. 217 | sudo sysctl kernel.yama.ptrace_scope=0 218 | ``` 219 | 220 | #### Ptrace With SELinux 221 | 222 | If you're using SELinux, 223 | [you may have problems with ptrace](https://fedoraproject.org/wiki/Features/SELinuxDenyPtrace). 224 | To check if ptrace is disabled: 225 | 226 | ```bash 227 | # Check if SELinux is denying ptrace. 228 | getsebool deny_ptrace 229 | ``` 230 | 231 | If you'd like to enable it: 232 | 233 | ```bash 234 | # Enable ptrace under SELinux. 235 | setsebool -P deny_ptrace 0 236 | ``` 237 | 238 | ## Python 3 Support 239 | 240 | This mostly works: if you have the Python 3 headers installed on your system, 241 | the configure script should detect the presence of Python 3 and use it. Please 242 | report any bugs related to Python 3 detection if you find them (particularly if 243 | you have Python 3 headers installed, but the build system isn't finding them). 244 | 245 | There is one known 246 | bug: 247 | [Pyflame can only decode ASCII filenames in Python 3](https://github.com/uber/pyflame/issues/2). 248 | The issue has more details, if you want to help fix it. 249 | 250 | ## Hacking 251 | 252 | This section will explain the Pyflame code for people who are interested in 253 | contributing source code patches. 254 | 255 | The code style in Pyflame (mostly) conforms to 256 | the [Google C++ Style Guide](https://google.github.io/styleguide/cppguide.html). 257 | Additionally, all of the source code is formatted 258 | with [clang-format](http://clang.llvm.org/docs/ClangFormat.html). There's a 259 | `.clang-format` file checked into the root of this repository which will make 260 | `clang-format` do the right thing. 261 | 262 | The Linux-specific code is be mostly restricted to the files `src/aslr.*`, 263 | `src/namespace.*`, and `src/ptrace.*`. If you want to port Pyflame to another 264 | Unix you will probably only need to modify these files. 265 | 266 | You can run the test suite locally like this: 267 | 268 | ```bash 269 | # Run the Pyflame test suite. 270 | make test 271 | ``` 272 | 273 | ## Legal and Licensing 274 | 275 | Pyflame is [free software](https://www.gnu.org/philosophy/free-sw.en.html) 276 | licensed under the 277 | [Apache License, Version 2.0][]. 278 | 279 | [Apache License, Version 2.0]: LICENSE 280 | --------------------------------------------------------------------------------