├── requirements.txt ├── docs_src ├── _static │ └── .empty ├── _templates │ └── .empty ├── requirements.txt ├── assets │ ├── timeplot.png │ ├── flame_graph.png │ ├── split_time_plot.png │ ├── zoomed_flow_graph.png │ ├── distant_flow_graph.png │ └── flame_graph_sandwich.png ├── api │ ├── heapprof.rst │ ├── reader.rst │ ├── types.rst │ ├── flow_graph.rst │ ├── lowlevel.rst │ └── index.rst ├── Makefile ├── make.bat ├── license.md ├── releasing.md ├── quickstart.md ├── index.rst ├── contributing.md ├── conf.py ├── code_of_conduct.md ├── using_heapprof.md └── advanced_heapprof.md ├── heapprof ├── tests │ ├── __init__.py │ ├── end_to_end_test.py │ └── si_prefix_test.py ├── __main__.py ├── types.py ├── __init__.py └── _si_prefix.py ├── LICENSE.md ├── CODE_OF_CONDUCT.md ├── .gitattributes ├── MANIFEST.in ├── docs ├── objects.inv ├── _static │ ├── file.png │ ├── minus.png │ ├── plus.png │ ├── documentation_options.js │ └── pygments.css ├── _images │ ├── timeplot.png │ ├── flame_graph.png │ ├── split_time_plot.png │ ├── zoomed_flow_graph.png │ ├── distant_flow_graph.png │ └── flame_graph_sandwich.png ├── _sources │ ├── api │ │ ├── heapprof.rst.txt │ │ ├── reader.rst.txt │ │ ├── types.rst.txt │ │ ├── flow_graph.rst.txt │ │ ├── lowlevel.rst.txt │ │ └── index.rst.txt │ ├── license.md.txt │ ├── releasing.md.txt │ ├── quickstart.md.txt │ ├── index.rst.txt │ ├── contributing.md.txt │ ├── code_of_conduct.md.txt │ ├── using_heapprof.md.txt │ └── advanced_heapprof.md.txt ├── _config.yml ├── .buildinfo ├── search.html ├── py-modindex.html ├── api │ ├── index.html │ └── heapprof.html ├── license.html ├── releasing.html ├── contributing.html └── quickstart.html ├── Gemfile ├── pyproject.toml ├── tools ├── requirements.txt ├── _common.py ├── docs.py └── lint.py ├── _heapprof ├── abstract_profiler.cc ├── README.md ├── reentrant_scope.cc ├── stats_gatherer.h ├── malloc_patch.h ├── simple_hash.h ├── scoped_object.h ├── abstract_profiler.h ├── reentrant_scope.h ├── stats_gatherer.cc ├── sampler.cc ├── sampler.h ├── profiler.h ├── profiler.cc ├── file_format.h ├── util.cc ├── port.h ├── malloc_patch.cc └── util.h ├── mypy.ini ├── _config.yml ├── CPPLINT.cfg ├── .flake8 ├── RELEASE_NOTES.md ├── .pylintrc ├── .gitignore ├── README.md ├── .circleci └── config.yml ├── setup.py └── Gemfile.lock /requirements.txt: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /docs_src/_static/.empty: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /docs_src/_templates/.empty: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /heapprof/tests/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /LICENSE.md: -------------------------------------------------------------------------------- 1 | docs_src/license.md -------------------------------------------------------------------------------- /CODE_OF_CONDUCT.md: -------------------------------------------------------------------------------- 1 | docs_src/code_of_conduct.md -------------------------------------------------------------------------------- /.gitattributes: -------------------------------------------------------------------------------- 1 | docs/build linguist-generated=true 2 | -------------------------------------------------------------------------------- /MANIFEST.in: -------------------------------------------------------------------------------- 1 | include README.md 2 | recursive-include _heapprof *.h *.cc 3 | -------------------------------------------------------------------------------- /docs_src/requirements.txt: -------------------------------------------------------------------------------- 1 | recommonmark 2 | sphinx 3 | sphinx-nameko-theme 4 | -------------------------------------------------------------------------------- /docs/objects.inv: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/humu/heapprof/HEAD/docs/objects.inv -------------------------------------------------------------------------------- /Gemfile: -------------------------------------------------------------------------------- 1 | source 'https://rubygems.org' 2 | gem 'github-pages', group: :jekyll_plugins 3 | -------------------------------------------------------------------------------- /docs/_static/file.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/humu/heapprof/HEAD/docs/_static/file.png -------------------------------------------------------------------------------- /docs/_static/minus.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/humu/heapprof/HEAD/docs/_static/minus.png -------------------------------------------------------------------------------- /docs/_static/plus.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/humu/heapprof/HEAD/docs/_static/plus.png -------------------------------------------------------------------------------- /pyproject.toml: -------------------------------------------------------------------------------- 1 | [build-system] 2 | requires = ["setuptools", "wheel", "cmake", "skbuild"] 3 | -------------------------------------------------------------------------------- /docs/_images/timeplot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/humu/heapprof/HEAD/docs/_images/timeplot.png -------------------------------------------------------------------------------- /tools/requirements.txt: -------------------------------------------------------------------------------- 1 | black 2 | cpplint 3 | flake8 4 | isort 5 | mypy 6 | recommonmark 7 | sphinx 8 | -------------------------------------------------------------------------------- /docs/_images/flame_graph.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/humu/heapprof/HEAD/docs/_images/flame_graph.png -------------------------------------------------------------------------------- /docs_src/assets/timeplot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/humu/heapprof/HEAD/docs_src/assets/timeplot.png -------------------------------------------------------------------------------- /docs/_images/split_time_plot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/humu/heapprof/HEAD/docs/_images/split_time_plot.png -------------------------------------------------------------------------------- /docs_src/assets/flame_graph.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/humu/heapprof/HEAD/docs_src/assets/flame_graph.png -------------------------------------------------------------------------------- /docs/_images/zoomed_flow_graph.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/humu/heapprof/HEAD/docs/_images/zoomed_flow_graph.png -------------------------------------------------------------------------------- /tools/_common.py: -------------------------------------------------------------------------------- 1 | import os 2 | 3 | REPO_ROOT = os.path.abspath(os.path.join(os.path.dirname(__file__), '..')) 4 | -------------------------------------------------------------------------------- /docs/_images/distant_flow_graph.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/humu/heapprof/HEAD/docs/_images/distant_flow_graph.png -------------------------------------------------------------------------------- /docs/_images/flame_graph_sandwich.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/humu/heapprof/HEAD/docs/_images/flame_graph_sandwich.png -------------------------------------------------------------------------------- /docs_src/assets/split_time_plot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/humu/heapprof/HEAD/docs_src/assets/split_time_plot.png -------------------------------------------------------------------------------- /docs_src/assets/zoomed_flow_graph.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/humu/heapprof/HEAD/docs_src/assets/zoomed_flow_graph.png -------------------------------------------------------------------------------- /docs_src/assets/distant_flow_graph.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/humu/heapprof/HEAD/docs_src/assets/distant_flow_graph.png -------------------------------------------------------------------------------- /docs_src/api/heapprof.rst: -------------------------------------------------------------------------------- 1 | The top-level heapprof package 2 | ============================== 3 | 4 | .. automodule:: heapprof 5 | -------------------------------------------------------------------------------- /docs_src/assets/flame_graph_sandwich.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/humu/heapprof/HEAD/docs_src/assets/flame_graph_sandwich.png -------------------------------------------------------------------------------- /_heapprof/abstract_profiler.cc: -------------------------------------------------------------------------------- 1 | #include "_heapprof/abstract_profiler.h" 2 | 3 | // Empty file to make the Python setuputils happy. 4 | -------------------------------------------------------------------------------- /docs/_sources/api/heapprof.rst.txt: -------------------------------------------------------------------------------- 1 | The top-level heapprof package 2 | ============================== 3 | 4 | .. automodule:: heapprof 5 | -------------------------------------------------------------------------------- /docs_src/api/reader.rst: -------------------------------------------------------------------------------- 1 | heapprof.reader: The high-level API 2 | =================================== 3 | 4 | .. automodule:: heapprof.reader 5 | -------------------------------------------------------------------------------- /docs_src/api/types.rst: -------------------------------------------------------------------------------- 1 | heapprof.types: Common definitions 2 | ================================== 3 | 4 | .. automodule:: heapprof.types 5 | 6 | -------------------------------------------------------------------------------- /mypy.ini: -------------------------------------------------------------------------------- 1 | [mypy] 2 | ignore_missing_imports = True 3 | follow_imports = silent 4 | check_untyped_defs = True 5 | disallow_untyped_calls = True 6 | -------------------------------------------------------------------------------- /docs/_config.yml: -------------------------------------------------------------------------------- 1 | # Config file for Jekyll serving; generated by tools/docs.py 2 | baseurl: /heapprof 3 | include: 4 | - /_static 5 | - /_images 6 | -------------------------------------------------------------------------------- /docs/_sources/api/reader.rst.txt: -------------------------------------------------------------------------------- 1 | heapprof.reader: The high-level API 2 | =================================== 3 | 4 | .. automodule:: heapprof.reader 5 | -------------------------------------------------------------------------------- /docs/_sources/api/types.rst.txt: -------------------------------------------------------------------------------- 1 | heapprof.types: Common definitions 2 | ================================== 3 | 4 | .. automodule:: heapprof.types 5 | 6 | -------------------------------------------------------------------------------- /docs_src/api/flow_graph.rst: -------------------------------------------------------------------------------- 1 | heapprof.flow\_graph: The FlowGraph class 2 | ========================================= 3 | 4 | .. automodule:: heapprof.flow_graph 5 | -------------------------------------------------------------------------------- /docs/_sources/api/flow_graph.rst.txt: -------------------------------------------------------------------------------- 1 | heapprof.flow\_graph: The FlowGraph class 2 | ========================================= 3 | 4 | .. automodule:: heapprof.flow_graph 5 | -------------------------------------------------------------------------------- /docs_src/api/lowlevel.rst: -------------------------------------------------------------------------------- 1 | heapprof.lowlevel: The low-level API 2 | ==================================== 3 | 4 | .. automodule:: heapprof.lowlevel 5 | :members: 6 | :undoc-members: 7 | :show-inheritance: 8 | -------------------------------------------------------------------------------- /docs/_sources/api/lowlevel.rst.txt: -------------------------------------------------------------------------------- 1 | heapprof.lowlevel: The low-level API 2 | ==================================== 3 | 4 | .. automodule:: heapprof.lowlevel 5 | :members: 6 | :undoc-members: 7 | :show-inheritance: 8 | -------------------------------------------------------------------------------- /docs_src/api/index.rst: -------------------------------------------------------------------------------- 1 | The heapprof API 2 | ==================================== 3 | 4 | .. toctree:: 5 | :maxdepth: 2 6 | :caption: Contents: 7 | 8 | heapprof 9 | reader 10 | types 11 | flow_graph 12 | lowlevel 13 | -------------------------------------------------------------------------------- /docs/_sources/api/index.rst.txt: -------------------------------------------------------------------------------- 1 | The heapprof API 2 | ==================================== 3 | 4 | .. toctree:: 5 | :maxdepth: 2 6 | :caption: Contents: 7 | 8 | heapprof 9 | reader 10 | types 11 | flow_graph 12 | lowlevel 13 | -------------------------------------------------------------------------------- /docs/.buildinfo: -------------------------------------------------------------------------------- 1 | # Sphinx build info version 1 2 | # This file hashes the configuration used when building these files. When it is not found, a full rebuild will be done. 3 | config: 1dc0a1126b663d12160c08d7cd135214 4 | tags: 645f666f9bcd5a90fca523b33c5a78b7 5 | -------------------------------------------------------------------------------- /_config.yml: -------------------------------------------------------------------------------- 1 | # Config file for Jekyll serving. This just points it at the docs constructed by Sphinx, because 2 | # Sphinx can do fancy things like autogenerating docs from docstrings. 3 | baseurl: /heapprof 4 | source: docs 5 | include: 6 | - /_static 7 | - /_images 8 | -------------------------------------------------------------------------------- /CPPLINT.cfg: -------------------------------------------------------------------------------- 1 | linelength=100 2 | # build/include_subdir is excluded until we can figure out how to tell it about the standard Python 3 | # include dirs. 4 | filter=-legal/copyright,-build/include_subdir 5 | exclude_files=venv 6 | exclude_files=build 7 | exclude_files=dist 8 | exclude_files=.eggs 9 | -------------------------------------------------------------------------------- /.flake8: -------------------------------------------------------------------------------- 1 | [flake8] 2 | exclude = 3 | # Node and Python library code 4 | node_modules, 5 | venv, 6 | # Compilation artifacts 7 | .git, 8 | build, 9 | dist, 10 | .mypy_cache, 11 | .eggs, 12 | max-line-length = 100 13 | ignore = E111,E114,E252,E203,W503,E501 14 | select = C,E,F,W,B,B950 15 | -------------------------------------------------------------------------------- /RELEASE_NOTES.md: -------------------------------------------------------------------------------- 1 | # Release Notes 2 | 3 | ## 1.0 4 | 5 | * 1.0.0 (2019-08-12) Initial release. 6 | * 1.0.1 (2019-08-20) Add support for WIN64 (via binary wheels) and Linux (via source wheels, 7 | as the glibc requirements for ABSL don't allow use of manylinux) 8 | * 1.0.2 (2020-03-03) Security patches (update dependencies); typo fix. 9 | -------------------------------------------------------------------------------- /docs/_static/documentation_options.js: -------------------------------------------------------------------------------- 1 | var DOCUMENTATION_OPTIONS = { 2 | URL_ROOT: document.getElementById("documentation_options").getAttribute('data-url_root'), 3 | VERSION: '', 4 | LANGUAGE: 'None', 5 | COLLAPSE_INDEX: false, 6 | FILE_SUFFIX: '.html', 7 | HAS_SOURCE: true, 8 | SOURCELINK_SUFFIX: '.txt', 9 | NAVIGATION_WITH_KEYS: false 10 | }; -------------------------------------------------------------------------------- /_heapprof/README.md: -------------------------------------------------------------------------------- 1 | # The C++ Bit 2 | 3 | This directory contains the `_heapprof` package, the C++ part of heapprof. The top level of this 4 | module is heapprof.cc, which declares the API; profiler.* contains the "brains" of the heap 5 | profiler, while malloc_patch.* contains the logic required to hook the profiler onto the PEP445 6 | malloc hooks correctly. (A task, alas, more subtle than it at first seems!) 7 | -------------------------------------------------------------------------------- /_heapprof/reentrant_scope.cc: -------------------------------------------------------------------------------- 1 | #include "_heapprof/reentrant_scope.h" 2 | 3 | Py_tss_t ReentrantScope::in_malloc_ = Py_tss_NEEDS_INIT; 4 | 5 | ReentrantScope::ReentrantScope() { 6 | PyThread_tss_create(&in_malloc_); 7 | const void *ptr = PyThread_tss_get(&in_malloc_); 8 | if (ptr == nullptr) { 9 | is_top_level_ = true; 10 | // Py_True is totally arbitrary; we just need any non-nullptr PyObject. 11 | PyThread_tss_set(&in_malloc_, Py_True); 12 | } else { 13 | assert(ptr == Py_True); 14 | is_top_level_ = false; 15 | } 16 | } 17 | 18 | ReentrantScope::~ReentrantScope() { 19 | if (is_top_level_) { 20 | PyThread_tss_set(&in_malloc_, nullptr); 21 | } 22 | } 23 | -------------------------------------------------------------------------------- /docs_src/Makefile: -------------------------------------------------------------------------------- 1 | # Minimal makefile for Sphinx documentation 2 | # 3 | 4 | # You can set these variables from the command line, and also 5 | # from the environment for the first two. 6 | SPHINXOPTS ?= 7 | SPHINXBUILD ?= sphinx-build 8 | SOURCEDIR = . 9 | BUILDDIR = build 10 | 11 | # Put it first so that "make" without argument is like "make help". 12 | help: 13 | @$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) 14 | 15 | .PHONY: help Makefile 16 | 17 | # Catch-all target: route all unknown targets to Sphinx using the new 18 | # "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS). 19 | %: Makefile 20 | export PYTHONPATH=$(PYTHONPATH) 21 | @$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) 22 | -------------------------------------------------------------------------------- /_heapprof/stats_gatherer.h: -------------------------------------------------------------------------------- 1 | #ifndef _HEAPPROF_STATS_GATHERER_H__ 2 | #define _HEAPPROF_STATS_GATHERER_H__ 3 | 4 | #include 5 | 6 | #include "_heapprof/abstract_profiler.h" 7 | 8 | // StatsGatherer is an AbstractProfiler which collects distribution statistics 9 | // about the sizes of mallocs, and prints them out to stderr when the profiler 10 | // is stopped. This is useful for figuring out the best choice of sampling rate 11 | // parameters to pick for proper profiling. 12 | class StatsGatherer : public AbstractProfiler { 13 | public: 14 | StatsGatherer(); 15 | virtual ~StatsGatherer(); 16 | 17 | void HandleMalloc(void *ptr, size_t size); 18 | 19 | private: 20 | struct BinStats { 21 | int num_allocs = 0; 22 | size_t total_bytes = 0; 23 | }; 24 | 25 | // Keyed by ceil(log2(alloc size)) 26 | std::map bins_; 27 | }; 28 | 29 | #endif // _HEAPPROF_STATS_GATHERER_H__ 30 | -------------------------------------------------------------------------------- /.pylintrc: -------------------------------------------------------------------------------- 1 | [MESSAGES CONTROL] 2 | 3 | # Currently, we just use pylint for name checks, and flake8 for everything else: 4 | disable=all 5 | enable=invalid-name 6 | 7 | [BASIC] 8 | 9 | # Class names must be UpperCamelCase: 10 | class-rgx=_?_?[A-Z][a-zA-Z0-9]+$ 11 | 12 | # "Constants" are just top-level variables, so we allow both: 13 | const-rgx=_?_?([a-z][a-zA-Z0-9]*|[A-Z_][A-Z0-9_]*)$ 14 | 15 | attr-rgx=_?_?([a-z][a-zA-Z0-9]*|[A-Z_][A-Z0-9_]*)$ 16 | 17 | # Class attributes can be named like camelCase or UPPER_CASE: 18 | class-attribute-rgx=_?_?([a-z][a-zA-Z0-9]*|[A-Z_][A-Z0-9_]*)$ 19 | 20 | function-rgx=_?_?[a-z][a-zA-Z0-9]*$ 21 | method-rgx=(_?_?[a-z][a-zA-Z0-9]*|__.+__|test[A-Za-z]_[a-zA-Z0-9_]+)$ 22 | 23 | # Only one "_" for argument names: 24 | argument-rgx=_?[a-z][a-zA-Z0-9]*$ 25 | variable-rgx=_?[a-z][a-zA-Z0-9]*$ 26 | 27 | # "inlinevars" are like `x` in `for x in y:`: 28 | inlinevar-rgx=[a-z][a-zA-Z0-9]*$ 29 | -------------------------------------------------------------------------------- /docs_src/make.bat: -------------------------------------------------------------------------------- 1 | @ECHO OFF 2 | 3 | pushd %~dp0 4 | 5 | REM Command file for Sphinx documentation 6 | 7 | if "%SPHINXBUILD%" == "" ( 8 | set SPHINXBUILD=sphinx-build 9 | ) 10 | set SOURCEDIR=. 11 | set BUILDDIR=_build 12 | 13 | if "%1" == "" goto help 14 | 15 | %SPHINXBUILD% >NUL 2>NUL 16 | if errorlevel 9009 ( 17 | echo. 18 | echo.The 'sphinx-build' command was not found. Make sure you have Sphinx 19 | echo.installed, then set the SPHINXBUILD environment variable to point 20 | echo.to the full path of the 'sphinx-build' executable. Alternatively you 21 | echo.may add the Sphinx directory to PATH. 22 | echo. 23 | echo.If you don't have Sphinx installed, grab it from 24 | echo.http://sphinx-doc.org/ 25 | exit /b 1 26 | ) 27 | 28 | %SPHINXBUILD% -M %1 %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O% 29 | goto end 30 | 31 | :help 32 | %SPHINXBUILD% -M help %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O% 33 | 34 | :end 35 | popd 36 | -------------------------------------------------------------------------------- /docs_src/license.md: -------------------------------------------------------------------------------- 1 | # MIT License 2 | 3 | Copyright (c) 2019 Humu, Inc. 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /docs/_sources/license.md.txt: -------------------------------------------------------------------------------- 1 | # MIT License 2 | 3 | Copyright (c) 2019 Humu, Inc. 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /_heapprof/malloc_patch.h: -------------------------------------------------------------------------------- 1 | #ifndef _HEAPPROF_MALLOC_PATCH_H__ 2 | #define _HEAPPROF_MALLOC_PATCH_H__ 3 | 4 | #include "_heapprof/abstract_profiler.h" 5 | 6 | // These functions do the work of connecting and disconnecting a Profiler 7 | // from the Python malloc hooks. In a normal world -- the world envisioned by 8 | // PEP445 -- these functions would be too simple to require their own file, but 9 | // as so often happens, the best-laid schemes of mice and API designers gang aft 10 | // a-gley. 11 | 12 | // Attach a profiler to the malloc hooks and start profiling. This function 13 | // takes ownership of the profiler state; it will be deleted when it is 14 | // detached. 15 | void AttachProfiler(AbstractProfiler *profiler); 16 | 17 | // Detach the profiler from the malloc hooks and stop profiling. It is not an 18 | // error to call this if there is no active profiling. 19 | void DetachProfiler(); 20 | 21 | // Test if profiling is active. 22 | bool IsProfilerAttached(); 23 | 24 | // Called to initialize this subset of the module, after the module as a whole 25 | // is created. Returns true on success; false is fatal. 26 | bool MallocPatchInit(); 27 | 28 | #endif // _HEAPPROF_MALLOC_PATCH_H__ 29 | -------------------------------------------------------------------------------- /_heapprof/simple_hash.h: -------------------------------------------------------------------------------- 1 | #ifndef _HEAPPROF_SIMPLE_HASH_H__ 2 | #define _HEAPPROF_SIMPLE_HASH_H__ 3 | 4 | // This hash function is based on the hash used for tuples in 5 | // Object/tupleobject.c. 6 | class SimpleHash { 7 | public: 8 | SimpleHash() : acc_(kPrime5), count_(0) {} 9 | 10 | void add(uint32_t value) { 11 | acc_ += value * kPrime2; 12 | acc_ = ((acc_ << 13) | (acc_ >> 19)); 13 | acc_ *= kPrime1; 14 | count_ += 1; 15 | } 16 | 17 | void add(void *ptr) { 18 | const intptr_t word = reinterpret_cast(ptr); 19 | add(static_cast(word)); 20 | if (sizeof(intptr_t) == 8) { 21 | add(static_cast(word >> 32)); 22 | } 23 | } 24 | 25 | uint32_t get() const { 26 | const uint32_t value = acc_ + (count_ ^ (kPrime5 ^ 3527539UL)); 27 | if (value == 0xffffffff) { 28 | return 1546275796; 29 | } else { 30 | return value; 31 | } 32 | } 33 | 34 | private: 35 | uint32_t acc_; 36 | uint32_t count_; 37 | 38 | static const uint32_t kPrime1 = 2654435761UL; 39 | static const uint32_t kPrime2 = 2246822519UL; 40 | static const uint32_t kPrime5 = 374761393UL; 41 | }; 42 | 43 | #endif // _HEAPPROF_SIMPLE_HASH_H__ 44 | -------------------------------------------------------------------------------- /docs_src/releasing.md: -------------------------------------------------------------------------------- 1 | # Release instructions 2 | 3 | This document is for heapprof maintainers only. It assumes that the release is being cut from an OS 4 | X machine, since there's no good way to remotely build those wheels yet. To cut a new release: 5 | 6 | 1. Check in a changelist which increments the version number in setup.py and updates 7 | RELEASE_NOTES.md. The version number should follow the [semantic 8 | versioning](https://packaging.python.org/guides/distributing-packages-using-setuptools/#semantic-versioning-preferred) 9 | scheme. 10 | 1. Clean the local image by running `git clean -xfd` 11 | 1. Build the source wheel by running `python setup.py sdist` 12 | 1. Build the OS X wheel by running `python setup.py bdist_wheel` 13 | 1. Go to `https://circleci.com/gh/humu/heapprof/tree/master` and click on the link for the 14 | most recent `windows_64` job. For each of them, click on the "Artifacts" tab 15 | and download the corresponding wheel. Move it to your `dist` directory along with the wheels 16 | created in the previous step. 17 | 1. Run `twine upload dist/*` to push the new repository version. 18 | 1. Go to `https://pypi.org/project/heapprof/` and verify that the new version is live. 19 | -------------------------------------------------------------------------------- /docs/_sources/releasing.md.txt: -------------------------------------------------------------------------------- 1 | # Release instructions 2 | 3 | This document is for heapprof maintainers only. It assumes that the release is being cut from an OS 4 | X machine, since there's no good way to remotely build those wheels yet. To cut a new release: 5 | 6 | 1. Check in a changelist which increments the version number in setup.py and updates 7 | RELEASE_NOTES.md. The version number should follow the [semantic 8 | versioning](https://packaging.python.org/guides/distributing-packages-using-setuptools/#semantic-versioning-preferred) 9 | scheme. 10 | 1. Clean the local image by running `git clean -xfd` 11 | 1. Build the source wheel by running `python setup.py sdist` 12 | 1. Build the OS X wheel by running `python setup.py bdist_wheel` 13 | 1. Go to `https://circleci.com/gh/humu/heapprof/tree/master` and click on the link for the 14 | most recent `windows_64` job. For each of them, click on the "Artifacts" tab 15 | and download the corresponding wheel. Move it to your `dist` directory along with the wheels 16 | created in the previous step. 17 | 1. Run `twine upload dist/*` to push the new repository version. 18 | 1. Go to `https://pypi.org/project/heapprof/` and verify that the new version is live. 19 | -------------------------------------------------------------------------------- /_heapprof/scoped_object.h: -------------------------------------------------------------------------------- 1 | #ifndef _HEAPPROF_SCOPED_OBJECT_H__ 2 | #define _HEAPPROF_SCOPED_OBJECT_H__ 3 | 4 | #include 5 | #include "Python.h" 6 | 7 | // This is a simple wrapper around a PyObject which owns a reference. It's a 8 | // more C++ish (and less bug-prone) way to grab a scoped reference to an object. 9 | // Its syntax deliberately mimics that of std::unique_ptr. 10 | class ScopedObject { 11 | public: 12 | explicit ScopedObject(PyObject *o) : o_(o) {} 13 | ~ScopedObject() { Py_XDECREF(o_); } 14 | 15 | PyObject &operator*() { return *o_; } 16 | const PyObject &operator*() const { return *o_; } 17 | explicit operator bool() const noexcept { return o_ != nullptr; } 18 | PyObject *operator->() const noexcept { return o_; } 19 | 20 | PyObject *get() const noexcept { return o_; } 21 | PyObject *release() noexcept { 22 | PyObject *o = o_; 23 | o_ = nullptr; 24 | return o; 25 | } 26 | void reset(PyObject *o) noexcept { 27 | Py_XDECREF(o_); 28 | o_ = o; 29 | } 30 | void swap(ScopedObject &other) noexcept { 31 | PyObject *o = other.o_; 32 | other.o_ = o_; 33 | o_ = o; 34 | } 35 | 36 | private: 37 | PyObject *o_; 38 | }; 39 | 40 | #endif // _HEAPPROF_SCOPED_OBJECT_H__ 41 | -------------------------------------------------------------------------------- /heapprof/__main__.py: -------------------------------------------------------------------------------- 1 | import argparse 2 | import ast 3 | import os 4 | import sys 5 | from typing import Dict, Optional, cast 6 | 7 | import heapprof 8 | 9 | parser = argparse.ArgumentParser("heapprof") 10 | parser.add_argument( 11 | "-m", 12 | "--mode", 13 | choices=("profile", "stats"), 14 | help="Profiler mode selection", 15 | default="profile", 16 | ) 17 | parser.add_argument("-o", "--output", help="Output file base", default="hprof") 18 | parser.add_argument("--sample", help="Sampling rate dictionary") 19 | parser.add_argument("command", nargs="+") 20 | args = parser.parse_args() 21 | 22 | sampleRate: Optional[Dict[int, float]] = None 23 | if args.sample: 24 | parsed = ast.literal_eval(args.sample) 25 | assert isinstance(parsed, dict) 26 | sampleRate = cast(Dict[int, float], parsed) 27 | 28 | progname = args.command[0] 29 | sys.path.insert(0, os.path.dirname(progname)) 30 | sys.argv[:] = args.command 31 | print(f'Heap profiler: Running {" ".join(sys.argv)}') 32 | with open(progname, "rb") as fp: 33 | code = compile(fp.read(), progname, "exec") 34 | globs = { 35 | "__file__": progname, 36 | "__name__": "__main__", 37 | "__package__": None, 38 | "__cached__": None, 39 | } 40 | 41 | if args.mode == "profile": 42 | heapprof.start(args.output, sampleRate) 43 | elif args.mode == "stats": 44 | heapprof.gatherStats() 45 | else: 46 | raise ValueError("Unknown mode [{args.mode}]") 47 | 48 | try: 49 | exec(code, globs, None) 50 | finally: 51 | heapprof.stop() 52 | -------------------------------------------------------------------------------- /_heapprof/abstract_profiler.h: -------------------------------------------------------------------------------- 1 | #ifndef _HEAPPROF_ABSTRACT_PROFILER_H__ 2 | #define _HEAPPROF_ABSTRACT_PROFILER_H__ 3 | 4 | #include 5 | #include 6 | 7 | // An AbstractProfiler is something that does actual profiling. For now, we 8 | // don't have any fancy registration mechanism to refer to these by name; 9 | // instead, each AbstractProfiler is used by a different exposed Python method 10 | // defined in heapprof.cc. 11 | // 12 | // AbstractProfiler implementations should be thread-compatible, but need not be 13 | // thread-safe. 14 | class AbstractProfiler { 15 | public: 16 | // NB: The Python dynamic loader seems to get upset if any classes are purely 17 | // abstract; it wants _all_ the symbols to be loadable. ::shrug:: 18 | virtual ~AbstractProfiler() {} 19 | 20 | // These may each assume that ptr is not null. 21 | virtual void HandleMalloc(void *ptr, size_t size) {} 22 | virtual void HandleFree(void *ptr) {} 23 | 24 | // The default implementation is good for most cases. We could be more clever 25 | // here, treating oldptr == nullptr as a malloc, oldptr == newptr as a malloc 26 | // of the delta size, and oldptr != newptr as a free + malloc, but this would 27 | // run into trouble if oldptr weren't selected by sampling. This is almost as 28 | // good and way easier. This method may assume that newptr != nullptr, but 29 | // oldptr may be null. 30 | virtual void HandleRealloc(void *oldptr, void *newptr, size_t size) { 31 | assert(newptr != nullptr); 32 | if (oldptr != nullptr) { 33 | HandleFree(oldptr); 34 | } 35 | HandleMalloc(newptr, size); 36 | } 37 | }; 38 | 39 | #endif // _HEAPPROF_ABSTRACT_PROFILER_H__ 40 | -------------------------------------------------------------------------------- /_heapprof/reentrant_scope.h: -------------------------------------------------------------------------------- 1 | #ifndef _HEAPPROF_REENTRANT_SCOPE_H__ 2 | #define _HEAPPROF_REENTRANT_SCOPE_H__ 3 | 4 | #include "Python.h" 5 | #include "pythread.h" 6 | 7 | // Python separates its memory management into several "domains," each of which 8 | // has their own malloc, realloc, calloc, and free functions. In _malloc_patch, 9 | // we're going to instrument all of these domains with calls to the profiler -- 10 | // but there's one catch. Some of the handlers which we're about to instrument 11 | // call *each other*; e.g., the OBJECT domain's malloc sometimes falls back to 12 | // the MEMORY domain's malloc. When this happens, we want to make sure that 13 | // we're only profiling at the topmost scope, so that we don't double-count 14 | // memory. 15 | // 16 | // ReentrantScope is a simple solution to this problem; each malloc-type 17 | // function keeps one of these in scope, and its is_top_level() method reveals 18 | // whether we're at the top level of the malloc call (and so should profile) or 19 | // not. Under the hood, this is done with a simple thread-local variable. 20 | class ReentrantScope { 21 | public: 22 | ReentrantScope(); 23 | ~ReentrantScope(); 24 | 25 | bool is_top_level() const { return is_top_level_; } 26 | 27 | private: 28 | // This is a thread-local variable which indicates if we're already inside a 29 | // memory allocation call. It's set by an outermost call, and reset on its 30 | // exit, and we only do profiling at that outermost // level. As this is 31 | // PyObject-valued, we'll use for its two values IN_MALLOC and nullptr. 32 | static Py_tss_t in_malloc_; 33 | 34 | bool is_top_level_; 35 | }; 36 | 37 | #endif // _HEAPPROF_REENTRANT_SCOPE_H__ 38 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | # Byte-compiled / optimized / DLL files 2 | __pycache__/ 3 | *.py[cod] 4 | *$py.class 5 | 6 | # C extensions 7 | *.so 8 | 9 | # Distribution / packaging 10 | .Python 11 | build/ 12 | develop-eggs/ 13 | dist/ 14 | downloads/ 15 | eggs/ 16 | .eggs/ 17 | lib/ 18 | lib64/ 19 | parts/ 20 | sdist/ 21 | var/ 22 | wheels/ 23 | *.egg-info/ 24 | .installed.cfg 25 | *.egg 26 | MANIFEST 27 | 28 | # PyInstaller 29 | # Usually these files are written by a python script from a template 30 | # before PyInstaller builds the exe, so as to inject date/other infos into it. 31 | *.manifest 32 | *.spec 33 | 34 | # Installer logs 35 | pip-log.txt 36 | pip-delete-this-directory.txt 37 | 38 | # Unit test / coverage reports 39 | htmlcov/ 40 | .tox/ 41 | .coverage 42 | .coverage.* 43 | .cache 44 | nosetests.xml 45 | coverage.xml 46 | *.cover 47 | .hypothesis/ 48 | .pytest_cache/ 49 | 50 | # Translations 51 | *.mo 52 | *.pot 53 | 54 | # Django stuff: 55 | *.log 56 | local_settings.py 57 | db.sqlite3 58 | 59 | # Flask stuff: 60 | instance/ 61 | .webassets-cache 62 | 63 | # Scrapy stuff: 64 | .scrapy 65 | 66 | # Sphinx documentation 67 | docs/_build/ 68 | 69 | # PyBuilder 70 | target/ 71 | 72 | # Jupyter Notebook 73 | .ipynb_checkpoints 74 | 75 | # pyenv 76 | .python-version 77 | 78 | # celery beat schedule file 79 | celerybeat-schedule 80 | 81 | # SageMath parsed files 82 | *.sage.py 83 | 84 | # Environments 85 | .env 86 | .venv 87 | env/ 88 | venv/ 89 | ENV/ 90 | env.bak/ 91 | venv.bak/ 92 | 93 | # Spyder project settings 94 | .spyderproject 95 | .spyproject 96 | 97 | # Rope project settings 98 | .ropeproject 99 | 100 | # mkdocs documentation 101 | /site 102 | 103 | # mypy 104 | .mypy_cache/ 105 | 106 | # vi 107 | .*.swp 108 | 109 | # jekyll 110 | /_site/ 111 | .sass-cache/ 112 | -------------------------------------------------------------------------------- /_heapprof/stats_gatherer.cc: -------------------------------------------------------------------------------- 1 | #include "_heapprof/stats_gatherer.h" 2 | #include 3 | #include 4 | #include 5 | #include 6 | #include 7 | #include "_heapprof/util.h" 8 | 9 | StatsGatherer::StatsGatherer() {} 10 | 11 | StatsGatherer::~StatsGatherer() { 12 | struct StatsString { 13 | std::string size; 14 | std::string num_allocs; 15 | std::string total_bytes; 16 | }; 17 | 18 | // These initial values are the lengths of the titles for the columns. 19 | int size_len = 4; 20 | int allocs_len = 5; 21 | int total_len = 5; 22 | 23 | std::vector stats; 24 | int prev_size = 0; 25 | for (auto it = bins_.begin(); it != bins_.end(); ++it) { 26 | const int new_size = 1 << it->first; 27 | StatsString summary{ 28 | std::to_string(prev_size + 1) + " - " + std::to_string(new_size), 29 | std::to_string(it->second.num_allocs), 30 | std::to_string(it->second.total_bytes)}; 31 | size_len = std::max(size_len, summary.size.size()); 32 | allocs_len = std::max(allocs_len, summary.num_allocs.size()); 33 | total_len = std::max(total_len, summary.total_bytes.size()); 34 | stats.push_back(summary); 35 | 36 | prev_size = new_size; 37 | } 38 | 39 | fprintf(stderr, "-------------------------------------------\n"); 40 | fprintf(stderr, "HEAP USAGE SUMMARY\n"); 41 | fprintf(stderr, "%*s %*s %*s\n", size_len, "Size", allocs_len, "Count", 42 | total_len, "Bytes"); 43 | for (auto it = stats.begin(); it != stats.end(); ++it) { 44 | fprintf(stderr, "%*s %*s %*s\n", size_len, it->size.c_str(), allocs_len, 45 | it->num_allocs.c_str(), total_len, it->total_bytes.c_str()); 46 | } 47 | } 48 | 49 | void StatsGatherer::HandleMalloc(void* ptr, size_t size) { 50 | BinStats& stats = bins_[Log2RoundUp(size)]; 51 | stats.num_allocs++; 52 | stats.total_bytes += size; 53 | } 54 | -------------------------------------------------------------------------------- /heapprof/types.py: -------------------------------------------------------------------------------- 1 | from typing import Dict, List, NamedTuple, Union 2 | 3 | 4 | class RawTraceLine(NamedTuple): 5 | """A RawTraceLine represents a single line of code.""" 6 | 7 | filename: str 8 | lineno: int 9 | 10 | def __str__(self) -> str: 11 | if self.lineno: 12 | return f'{self.filename}:{self.lineno}' 13 | else: 14 | return self.filename 15 | 16 | @classmethod 17 | def parse(cls, value: Union['RawTraceLine', str]) -> 'RawTraceLine': 18 | if isinstance(value, RawTraceLine): 19 | return value 20 | filename, linestr = value.rsplit(':', 1) 21 | return cls(filename, int(linestr)) 22 | 23 | 24 | # A RawTrace is the simplest form of a raw stack trace. 25 | RawTrace = List[RawTraceLine] 26 | 27 | 28 | class TraceLine(NamedTuple): 29 | """A TraceLine is a RawTraceLine plus the actual line of code. These can be fetched from HPM 30 | files so long as the source code is also present; doing so is (for obvious reasons) more 31 | expensive than just working with RawTraces, but can make nicer stack traces. 32 | """ 33 | 34 | # The filename 35 | filename: str 36 | # The line number 37 | lineno: int 38 | # The actual line of code 39 | fileline: str 40 | 41 | 42 | HeapTrace = List[TraceLine] 43 | 44 | 45 | class Snapshot(NamedTuple): 46 | """A Snapshot represents the state of the heap at a single moment in time. These are the basic 47 | elements of .hpc files. 48 | """ 49 | 50 | # The timestamp of this snapshot, relative to program start, in seconds. 51 | relativeTime: float 52 | 53 | # Memory usage, as a map from the trace index at which memory was allocated (which can be 54 | # resolved into a proper raw or full stack trace using an HPM) to live bytes in memory at 55 | # this time. 56 | usage: Dict[int, int] 57 | 58 | def totalUsage(self) -> int: 59 | return sum(self.usage.values()) 60 | -------------------------------------------------------------------------------- /heapprof/tests/end_to_end_test.py: -------------------------------------------------------------------------------- 1 | import os 2 | import time 3 | import unittest 4 | from tempfile import TemporaryDirectory 5 | 6 | import heapprof 7 | 8 | 9 | class EndToEndTest(unittest.TestCase): 10 | def testGatherStats(self) -> None: 11 | # Basically a smoke test -- make sure it runs. 12 | heapprof.gatherStats() 13 | self.assertTrue(heapprof.isProfiling()) 14 | list(range(10000)) 15 | heapprof.stop() 16 | 17 | def testHeapProfiler(self) -> None: 18 | with TemporaryDirectory() as path: 19 | hpxFile = os.path.join(path, "hprof") 20 | 21 | # Do something to make a heap profile 22 | heapprof.start(hpxFile) 23 | list(range(100_000)) 24 | time.sleep(0.05) 25 | list(range(100_000)) 26 | heapprof.stop() 27 | 28 | # NB: Most uses don't really need to use a with statement here -- it cleans up on 29 | # __del__ -- , but on some platforms the TemporaryDirectory exit will fail if the 30 | # reader is still open. 31 | with heapprof.Reader(hpxFile) as reader: 32 | # No digest yet, so we can't check elapsed time. 33 | with self.assertRaises(AssertionError): 34 | reader.elapsedTime() 35 | 36 | # Make a digest with 10-millisecond intervals and no rounding. 37 | reader.makeDigest(timeInterval=0.01, precision=0, verbose=True) 38 | 39 | self.assertGreaterEqual(reader.elapsedTime(), 0.05) 40 | self.assertAlmostEqual(reader.snapshotInterval(), 0.01) 41 | for index, snapshot in enumerate(reader.snapshots()): 42 | self.assertAlmostEqual(index * reader.snapshotInterval(), snapshot.relativeTime) 43 | # It would be nice to do some more sophisticated testing here. 44 | self.assertGreater(len(snapshot.usage), 0) 45 | 46 | 47 | if __name__ == "__main__": 48 | unittest.main() 49 | -------------------------------------------------------------------------------- /docs_src/quickstart.md: -------------------------------------------------------------------------------- 1 | # heapprof quickstart 2 | 3 | You can install heapprof by running 4 | 5 | `pip install heapprof` 6 | 7 | The simplest way to get a heap profile for your program is to run 8 | 9 | `python -m heapprof -o -- mycommand.py args...` 10 | 11 | This will run your command and write the output to the files `.hpm` and `.hpd`. 12 | (Collectively, the "hpx files") Alternatively, you can control heapprof programmatically: 13 | 14 | ``` 15 | import heapprof 16 | 17 | heapprof.start('filename') 18 | ... do something ... 19 | heapprof.stop() 20 | ``` 21 | 22 | You can analyze hpx files with the analysis and visualization tools built in to heapprof, or use its 23 | APIs to dive deeper into your program's memory usage yourself. For example, you might do this with 24 | the built-in visualization tools: 25 | 26 | ``` 27 | import heapprof 28 | r = heapprof.read('filename') 29 | 30 | # Generate a plot of total memory usage over time. This command requires that you first pip install 31 | # matplotlib. 32 | r.timePlot().pyplot() 33 | 34 | # Looks like something interested happened 300 seconds in. You can look at the output as a Flame 35 | # graph and view it in a tool like speedscope.app. 36 | r.writeFlameGraph('flame-300.txt', 300) 37 | 38 | # Or you can look at the output as a Flow graph and view it using graphviz. (See graphviz.org) 39 | r.flowGraphAt(300).asDotFile('300.dot') 40 | 41 | # Maybe you'd like to compare three times and see how memory changed. This produces a multi-time 42 | # Flow graph. 43 | r.compare('compare.dot', r.flowGraphAt(240), r.flowGraphAt(300), r.flowGraphAt(500)) 44 | 45 | # Or maybe you found some lines of code where memory use seemed to shift interestingly. 46 | r.timePlot({ 47 | 'read_data': '/home/wombat/myproject/io.py:361', 48 | 'write_buffer': '/home/wombat/myproject/io.py:582', 49 | }) 50 | ``` 51 | 52 | One thing you'll notice in the above is that visualization tools require you to install extra 53 | packages -- that's a way to keep heapprof's dependencies down. 54 | 55 | To learn more, continue on to [Using heapprof](using_heapprof.md) 56 | -------------------------------------------------------------------------------- /docs/_sources/quickstart.md.txt: -------------------------------------------------------------------------------- 1 | # heapprof quickstart 2 | 3 | You can install heapprof by running 4 | 5 | `pip install heapprof` 6 | 7 | The simplest way to get a heap profile for your program is to run 8 | 9 | `python -m heapprof -o -- mycommand.py args...` 10 | 11 | This will run your command and write the output to the files `.hpm` and `.hpd`. 12 | (Collectively, the "hpx files") Alternatively, you can control heapprof programmatically: 13 | 14 | ``` 15 | import heapprof 16 | 17 | heapprof.start('filename') 18 | ... do something ... 19 | heapprof.stop() 20 | ``` 21 | 22 | You can analyze hpx files with the analysis and visualization tools built in to heapprof, or use its 23 | APIs to dive deeper into your program's memory usage yourself. For example, you might do this with 24 | the built-in visualization tools: 25 | 26 | ``` 27 | import heapprof 28 | r = heapprof.read('filename') 29 | 30 | # Generate a plot of total memory usage over time. This command requires that you first pip install 31 | # matplotlib. 32 | r.timePlot().pyplot() 33 | 34 | # Looks like something interested happened 300 seconds in. You can look at the output as a Flame 35 | # graph and view it in a tool like speedscope.app. 36 | r.writeFlameGraph('flame-300.txt', 300) 37 | 38 | # Or you can look at the output as a Flow graph and view it using graphviz. (See graphviz.org) 39 | r.flowGraphAt(300).asDotFile('300.dot') 40 | 41 | # Maybe you'd like to compare three times and see how memory changed. This produces a multi-time 42 | # Flow graph. 43 | r.compare('compare.dot', r.flowGraphAt(240), r.flowGraphAt(300), r.flowGraphAt(500)) 44 | 45 | # Or maybe you found some lines of code where memory use seemed to shift interestingly. 46 | r.timePlot({ 47 | 'read_data': '/home/wombat/myproject/io.py:361', 48 | 'write_buffer': '/home/wombat/myproject/io.py:582', 49 | }) 50 | ``` 51 | 52 | One thing you'll notice in the above is that visualization tools require you to install extra 53 | packages -- that's a way to keep heapprof's dependencies down. 54 | 55 | To learn more, continue on to [Using heapprof](using_heapprof.md) 56 | -------------------------------------------------------------------------------- /docs_src/index.rst: -------------------------------------------------------------------------------- 1 | heapprof: A Logging Heap Profiler 2 | ================================= 3 | 4 | .. image:: https://img.shields.io/badge/python-3.7-blue.svg 5 | :target: https://www.python.org/downloads/release/python-374/ 6 | .. image:: https://img.shields.io/badge/Contributor%20Covenant-v1.4%20adopted-ff69b4.svg 7 | :target: code_of_conduct.html 8 | .. image:: https://img.shields.io/badge/License-MIT-yellow.svg 9 | :target: license.html 10 | .. image:: https://circleci.com/gh/humu/heapprof/tree/master.svg?style=svg&circle-token=1557bfcabda0155d6505a45e3f00d4a71a005565 11 | :target: https://circleci.com/gh/humu/heapprof/tree/master 12 | 13 | heapprof is a logging, sampling heap profiler for Python 3.7+. 14 | 15 | * "Logging" means that as the program runs, it steadily generates a log of memory allocation and 16 | release events. This means that you can easily look at memory usage as a function of time. 17 | * "Sampling" means that it can record only a statistically random sample of memory events. This 18 | improves performance dramatically while writing logs, and (with the right parameters) sacrifices 19 | almost no accuracy. 20 | 21 | It comes with a suite of visualization and analysis tools (including time plots, flame graphs, and 22 | flow graphs), as well as an API for doing your own analyses of the results. 23 | 24 | heapprof is complementary to `tracemalloc `_, 25 | which is a snapshotting heap profiler. The difference is that tracemalloc keeps track of live memory 26 | internally, and only writes snapshots when its snapshot() function is called; this means it has 27 | slightly lower overhead, but you have to know the moments at which you'll want a snapshot before the 28 | program starts. This makes it particularly useful for finding leaks (from the snapshot at program 29 | exit), but not as good for understanding events like memory spikes. 30 | 31 | .. toctree:: 32 | :maxdepth: 2 33 | :caption: Contents: 34 | 35 | quickstart 36 | using_heapprof 37 | visualizing_results 38 | advanced_heapprof 39 | api/index 40 | contributing 41 | license 42 | 43 | .. toctree:: 44 | :hidden: 45 | 46 | code_of_conduct 47 | -------------------------------------------------------------------------------- /docs/_sources/index.rst.txt: -------------------------------------------------------------------------------- 1 | heapprof: A Logging Heap Profiler 2 | ================================= 3 | 4 | .. image:: https://img.shields.io/badge/python-3.7-blue.svg 5 | :target: https://www.python.org/downloads/release/python-374/ 6 | .. image:: https://img.shields.io/badge/Contributor%20Covenant-v1.4%20adopted-ff69b4.svg 7 | :target: code_of_conduct.html 8 | .. image:: https://img.shields.io/badge/License-MIT-yellow.svg 9 | :target: license.html 10 | .. image:: https://circleci.com/gh/humu/heapprof/tree/master.svg?style=svg&circle-token=1557bfcabda0155d6505a45e3f00d4a71a005565 11 | :target: https://circleci.com/gh/humu/heapprof/tree/master 12 | 13 | heapprof is a logging, sampling heap profiler for Python 3.7+. 14 | 15 | * "Logging" means that as the program runs, it steadily generates a log of memory allocation and 16 | release events. This means that you can easily look at memory usage as a function of time. 17 | * "Sampling" means that it can record only a statistically random sample of memory events. This 18 | improves performance dramatically while writing logs, and (with the right parameters) sacrifices 19 | almost no accuracy. 20 | 21 | It comes with a suite of visualization and analysis tools (including time plots, flame graphs, and 22 | flow graphs), as well as an API for doing your own analyses of the results. 23 | 24 | heapprof is complementary to `tracemalloc `_, 25 | which is a snapshotting heap profiler. The difference is that tracemalloc keeps track of live memory 26 | internally, and only writes snapshots when its snapshot() function is called; this means it has 27 | slightly lower overhead, but you have to know the moments at which you'll want a snapshot before the 28 | program starts. This makes it particularly useful for finding leaks (from the snapshot at program 29 | exit), but not as good for understanding events like memory spikes. 30 | 31 | .. toctree:: 32 | :maxdepth: 2 33 | :caption: Contents: 34 | 35 | quickstart 36 | using_heapprof 37 | visualizing_results 38 | advanced_heapprof 39 | api/index 40 | contributing 41 | license 42 | 43 | .. toctree:: 44 | :hidden: 45 | 46 | code_of_conduct 47 | -------------------------------------------------------------------------------- /_heapprof/sampler.cc: -------------------------------------------------------------------------------- 1 | #include "_heapprof/sampler.h" 2 | #include 3 | #include "_heapprof/scoped_object.h" 4 | #include "_heapprof/util.h" 5 | 6 | Sampler::Range::Range(Py_ssize_t m, double p) : max_bytes(m) { 7 | if (p == 0) { 8 | probability = 0; 9 | } else if (p == 1) { 10 | probability = UINT_FAST32_MAX; 11 | } else { 12 | probability = static_cast(p * UINT_FAST32_MAX); 13 | } 14 | } 15 | 16 | Sampler::Sampler(PyObject *sampling_rate) { 17 | if (!PyMapping_Check(sampling_rate)) { 18 | PyErr_SetString(PyExc_TypeError, "samplingRate is not a Dict[int, float]"); 19 | return; 20 | } 21 | 22 | { 23 | ScopedObject items(PyMapping_Items(sampling_rate)); 24 | 25 | for (int i = 0, len = PyList_GET_SIZE(items.get()); i < len; ++i) { 26 | PyObject *item = PyList_GET_ITEM(items.get(), i); 27 | Py_ssize_t max_size; 28 | double probability; 29 | if (!PyArg_ParseTuple(item, "nd", &max_size, &probability)) { 30 | // Exception already set. 31 | return; 32 | } 33 | if (max_size < 0) { 34 | PyErr_Format(PyExc_ValueError, 35 | "%zd is not a valid memory allocation size.", max_size); 36 | return; 37 | } 38 | if (probability < 0 || probability > 1) { 39 | PyErr_Format( 40 | PyExc_ValueError, 41 | "%f is not a valid probability; it must be in the range [0, 1].", 42 | probability); 43 | return; 44 | } 45 | ranges_.push_back(Range(max_size, probability)); 46 | } 47 | } 48 | 49 | sort(ranges_.begin(), ranges_.end()); 50 | 51 | // Safety check: Make sure there are no repeated entries. 52 | if (ranges_.size() > 1) { 53 | auto it = ranges_.begin(); 54 | Py_ssize_t last_size = it->max_bytes; 55 | for (++it; it != ranges_.end(); ++it) { 56 | if (it->max_bytes == last_size) { 57 | PyErr_SetString(PyExc_ValueError, 58 | "Repeated size entry in samplingRate"); 59 | return; 60 | } 61 | last_size = it->max_bytes; 62 | } 63 | } 64 | 65 | ok_ = true; 66 | } 67 | 68 | void Sampler::WriteStateToFile(int fd) const { 69 | WriteVarintToFile(fd, ranges_.size()); 70 | for (auto it = ranges_.begin(); it != ranges_.end(); ++it) { 71 | WriteFixed64ToFile(fd, it->max_bytes); 72 | WriteFixed32ToFile(fd, it->probabilityAsUint32()); 73 | } 74 | } 75 | -------------------------------------------------------------------------------- /heapprof/tests/si_prefix_test.py: -------------------------------------------------------------------------------- 1 | import unittest 2 | 3 | from heapprof._si_prefix import siPrefixString 4 | 5 | 6 | class SIPrefixTest(unittest.TestCase): 7 | def testDecimal(self): 8 | self.assertEqual('1050.00', siPrefixString(1050, threshold=1.1, precision=2)) 9 | self.assertEqual('1.05k', siPrefixString(1050, threshold=1.05, precision=2)) 10 | self.assertEqual('1.1', siPrefixString(1.1, threshold=1.1, precision=1)) 11 | self.assertEqual('2.0M', siPrefixString(2000000, threshold=1.1, precision=1)) 12 | self.assertEqual('500m', siPrefixString(0.5, threshold=1.1, precision=0)) 13 | self.assertEqual('1.1μ', siPrefixString(1.1e-6, threshold=1.1, precision=1)) 14 | 15 | def testBinary(self): 16 | self.assertEqual('1050.00', siPrefixString(1050, threshold=1.1, precision=2, binary=True)) 17 | self.assertEqual('1.1k', siPrefixString(1130, threshold=1.1, precision=1, binary=True)) 18 | self.assertEqual('1.1', siPrefixString(1.1, threshold=1.1, precision=1, binary=True)) 19 | self.assertEqual( 20 | '2.0M', siPrefixString(2 * 1024 * 1024, threshold=1.1, precision=1, binary=True) 21 | ) 22 | # NB that 0.5 = 512 * (1024)^-1. 23 | self.assertEqual('512m', siPrefixString(0.5, threshold=1.1, precision=0, binary=True)) 24 | self.assertEqual('1.2μ', siPrefixString(1.1e-6, threshold=1.1, precision=1, binary=True)) 25 | 26 | def testIEC(self): 27 | self.assertEqual( 28 | '1050.00', siPrefixString(1050, threshold=1.1, precision=2, binary=True, iecFormat=True) 29 | ) 30 | self.assertEqual( 31 | '1.1Ki', siPrefixString(1130, threshold=1.1, precision=1, binary=True, iecFormat=True) 32 | ) 33 | self.assertEqual( 34 | '1.1', siPrefixString(1.1, threshold=1.1, precision=1, binary=True, iecFormat=True) 35 | ) 36 | self.assertEqual( 37 | '2.0Mi', 38 | siPrefixString( 39 | 2 * 1024 * 1024, threshold=1.1, precision=1, binary=True, iecFormat=True 40 | ), 41 | ) 42 | self.assertEqual( 43 | '512mi', siPrefixString(0.5, threshold=1.1, precision=0, binary=True, iecFormat=True) 44 | ) 45 | self.assertEqual( 46 | '1.2μi', siPrefixString(1.1e-6, threshold=1.1, precision=1, binary=True, iecFormat=True) 47 | ) 48 | 49 | 50 | if __name__ == '__main__': 51 | unittest.main() 52 | -------------------------------------------------------------------------------- /_heapprof/sampler.h: -------------------------------------------------------------------------------- 1 | #ifndef _HEAPPROF_SAMPLER_H__ 2 | #define _HEAPPROF_SAMPLER_H__ 3 | 4 | #include 5 | #include 6 | #include 7 | #include "Python.h" 8 | 9 | // A Sampler maintains the map of sampling probabilities per allocation size. 10 | // This will probably require some performance tuning. 11 | class Sampler { 12 | public: 13 | // Construct a new Sampler. The argument must be a dict from int to double. 14 | explicit Sampler(PyObject* sampling_rate); 15 | ~Sampler() {} 16 | 17 | // Tests validity after construction. If this is false, the Python exception 18 | // has been set. 19 | bool ok() { return ok_; } 20 | 21 | // Decide if we should profile an allocation of the given size. 22 | bool Sample(int alloc_size); 23 | 24 | // Write the parameters of this sampler to a file. 25 | void WriteStateToFile(int fd) const; 26 | 27 | private: 28 | struct Range { 29 | Range(Py_ssize_t m, double p); 30 | Range& operator=(const Range& other) { 31 | max_bytes = other.max_bytes; 32 | probability = other.probability; 33 | return *this; 34 | } 35 | bool operator<(const Range& other) const { 36 | return max_bytes < other.max_bytes; 37 | } 38 | 39 | uint32_t probabilityAsUint32() const { 40 | if (probability == 0) { 41 | return 0; 42 | } else if (probability == UINT_FAST32_MAX) { 43 | return UINT32_MAX; 44 | } else { 45 | const double p = static_cast(probability) / UINT_FAST32_MAX; 46 | return static_cast(p * UINT32_MAX); 47 | } 48 | } 49 | 50 | Py_ssize_t max_bytes; 51 | uint_fast32_t probability; 52 | }; 53 | 54 | // Sorted by max_bytes. 55 | std::vector ranges_; 56 | 57 | // Our RNG. minstd_rand uses Lehmer's generator, which is very fast and more 58 | // than good enough for our purposes. 59 | std::minstd_rand rng_ = std::minstd_rand(); 60 | 61 | bool ok_ = false; 62 | }; 63 | 64 | // Inline because this is called for every malloc. 65 | inline bool Sampler::Sample(int alloc_size) { 66 | // ranges_ is small enough that a linear search is more efficient than a 67 | // binary one. 68 | for (auto it = ranges_.begin(); it != ranges_.end(); ++it) { 69 | if (it->max_bytes > alloc_size) { 70 | // Our sampling probability is it->probability 71 | if (it->probability == 0) { 72 | return false; 73 | } else if (it->probability == UINT_FAST32_MAX) { 74 | return true; 75 | } else { 76 | const bool result = rng_() < it->probability; 77 | return result; 78 | } 79 | } 80 | } 81 | 82 | // If it's bigger than all the ranges, we always interpret it as having 83 | // probability 1. 84 | return true; 85 | } 86 | 87 | #endif // _HEAPPROF_SAMPLER_H__ 88 | -------------------------------------------------------------------------------- /_heapprof/profiler.h: -------------------------------------------------------------------------------- 1 | #ifndef _HEAPPROF_PROFILER_H__ 2 | #define _HEAPPROF_PROFILER_H__ 3 | 4 | #include 5 | #include 6 | #include 7 | #include "Python.h" 8 | #include "_heapprof/abstract_profiler.h" 9 | #include "_heapprof/sampler.h" 10 | #include "_heapprof/util.h" 11 | 12 | // Profiler is the heart of heap profiling. When the profiler is on, a 13 | // singleton Profiler exists, and is patched into the malloc calls using 14 | // the standard hooks, by the logic in heapprof.cc; to end profiling, it is 15 | // simply unpatched and deleted. 16 | // 17 | // All the real work happens in this file, as the patched calls trigger 18 | // Profiler's Handle{Malloc, Realloc, Free} methods, which are the things 19 | // that generate actual log entries and so on. 20 | 21 | // This class is thread-compatible, but not thread-safe; callers are responsible 22 | // for ensuring its methods are not called in parallel. (See _malloc_patch.cc 23 | // for how this is done, and why we don't do the seemingly simple thing of 24 | // making this class thread-safe instead) 25 | class Profiler : public AbstractProfiler { 26 | public: 27 | // Takes ownership of the sampler. 28 | Profiler(const char *filebase, Sampler *sampler); 29 | virtual ~Profiler(); 30 | 31 | // These each require that ptr (newptr) not be nullptr. 32 | virtual void HandleMalloc(void *ptr, size_t size); 33 | virtual void HandleFree(void *ptr); 34 | 35 | // Verifies validity of the profiler after construction. If false, the 36 | // exception is already set. 37 | bool ok() const { return ok_; } 38 | 39 | private: 40 | // The information we store for a live pointer. 41 | struct LivePointer { 42 | // The trace at which it was allocated. 43 | uint32_t traceindex; 44 | // The size of the memory allocated. (NB: This is the size as returned by 45 | // malloc, which alas is *not* the size as actually pulled by malloc; but 46 | // there's no API-agnostic way to find out what the particular malloc 47 | // implementation on this machine actually did.) 48 | size_t size; 49 | }; 50 | 51 | std::unique_ptr sampler_; 52 | 53 | // File pointers to the two output files. 54 | ScopedFile metadata_file_; 55 | ScopedFile data_file_; 56 | 57 | // The time of the previous event. 58 | struct timespec last_clock_; 59 | 60 | // The next trace index we'll assign. Note that trace index 0 is defined to be 61 | // "the bogus trace index." 62 | uint32_t next_trace_index_ = 1; 63 | 64 | // A hash map from tracefp to trace index. 65 | std::unordered_map trace_index_; 66 | 67 | // Data about currently live pointers. 68 | std::unordered_map live_set_; 69 | 70 | bool ok_ = false; 71 | 72 | // Get the current trace index. 73 | int GetTraceIndex(); 74 | }; 75 | 76 | #endif // _HEAPPROF_PROFILER_H__ 77 | -------------------------------------------------------------------------------- /docs_src/contributing.md: -------------------------------------------------------------------------------- 1 | [![Contributor Covenant](https://img.shields.io/badge/Contributor%20Covenant-v1.4%20adopted-ff69b4.svg)](code_of_conduct.md) 2 | 3 | # Contributing to heapprof 4 | 5 | heapprof is an open source project distributed under the [MIT License](license.md). Discussions, 6 | questions, and feature requests should be done via the 7 | [GitHub issues page](https://github.com/humu-com/heapprof/issues). 8 | 9 | Pull requests for bugfixes and features are welcome! 10 | 11 | * Generally, you should discuss features or API changes on the tracking issue first, to make sure 12 | everyone is aligned on direction. 13 | * Lint and style: 14 | - Python code should follow PEP8+[Black](https://github.com/python/black) 15 | formatting, and should pass mypy with strict type checking. 16 | - C/C++ code should follow the 17 | [Google C++ Style Guide](https://google.github.io/styleguide/cppguide.html). 18 | - You can check a client against the same lint checks run by the continuous integration test by 19 | running `python tools/lint.py`; if you add `--fix`, it will try to fix any errors it can 20 | in-place. 21 | * Unittests are highly desired and should be invocable by `setup.py test`. 22 | * Documentation: Any changes should be reflected in the documentation! Remember: 23 | - All Python modules should have clear docstrings for all public methods and variables. Classes 24 | should be organized with the interface first, and implementation details later. 25 | - If you're adding any new top-level modules, add corresponding .rst files in `docs/api` and link 26 | to them from `docs/api/index.rst`. 27 | - Any other changes should be reflected in the documentation in `docs`. 28 | - After changing the documentation, you can rebuild the HTML by running `python tools/docs.py`. 29 | - If you want to see how the documentation looks before pushing, install 30 | [Jekyll](https://jekyllrb.com/docs/installation/) locally, then from the root directory of 31 | this repository, run `bundle exec jekyll serve`. This will serve the same images that will be 32 | served in production on `localhost:4000`. 33 | 34 | ## Code of Conduct 35 | 36 | Most importantly, heapprof is released with a [Contributor Code of Conduct](code_of_conduct.md). By 37 | participating in this product, you agree to abide by its terms. This code also governs behavior on 38 | the mailing list. We take this very seriously, and will enforce it gleefully. 39 | 40 | ## Desiderata 41 | 42 | Some known future features that we'll probably want: 43 | 44 | * Provide additional file formats of output to work with other kinds of visualization. 45 | * Measure and tune system performance. 46 | * Make the process of picking sampling rates less manual. 47 | * Add support for more platforms. (Win32? Android? iOS?) 48 | * Implement a CircleCI orb for Windows with Python + CMake to speed up builds. 49 | -------------------------------------------------------------------------------- /docs/_sources/contributing.md.txt: -------------------------------------------------------------------------------- 1 | [![Contributor Covenant](https://img.shields.io/badge/Contributor%20Covenant-v1.4%20adopted-ff69b4.svg)](code_of_conduct.md) 2 | 3 | # Contributing to heapprof 4 | 5 | heapprof is an open source project distributed under the [MIT License](license.md). Discussions, 6 | questions, and feature requests should be done via the 7 | [GitHub issues page](https://github.com/humu-com/heapprof/issues). 8 | 9 | Pull requests for bugfixes and features are welcome! 10 | 11 | * Generally, you should discuss features or API changes on the tracking issue first, to make sure 12 | everyone is aligned on direction. 13 | * Lint and style: 14 | - Python code should follow PEP8+[Black](https://github.com/python/black) 15 | formatting, and should pass mypy with strict type checking. 16 | - C/C++ code should follow the 17 | [Google C++ Style Guide](https://google.github.io/styleguide/cppguide.html). 18 | - You can check a client against the same lint checks run by the continuous integration test by 19 | running `python tools/lint.py`; if you add `--fix`, it will try to fix any errors it can 20 | in-place. 21 | * Unittests are highly desired and should be invocable by `setup.py test`. 22 | * Documentation: Any changes should be reflected in the documentation! Remember: 23 | - All Python modules should have clear docstrings for all public methods and variables. Classes 24 | should be organized with the interface first, and implementation details later. 25 | - If you're adding any new top-level modules, add corresponding .rst files in `docs/api` and link 26 | to them from `docs/api/index.rst`. 27 | - Any other changes should be reflected in the documentation in `docs`. 28 | - After changing the documentation, you can rebuild the HTML by running `python tools/docs.py`. 29 | - If you want to see how the documentation looks before pushing, install 30 | [Jekyll](https://jekyllrb.com/docs/installation/) locally, then from the root directory of 31 | this repository, run `bundle exec jekyll serve`. This will serve the same images that will be 32 | served in production on `localhost:4000`. 33 | 34 | ## Code of Conduct 35 | 36 | Most importantly, heapprof is released with a [Contributor Code of Conduct](code_of_conduct.md). By 37 | participating in this product, you agree to abide by its terms. This code also governs behavior on 38 | the mailing list. We take this very seriously, and will enforce it gleefully. 39 | 40 | ## Desiderata 41 | 42 | Some known future features that we'll probably want: 43 | 44 | * Provide additional file formats of output to work with other kinds of visualization. 45 | * Measure and tune system performance. 46 | * Make the process of picking sampling rates less manual. 47 | * Add support for more platforms. (Win32? Android? iOS?) 48 | * Implement a CircleCI orb for Windows with Python + CMake to speed up builds. 49 | -------------------------------------------------------------------------------- /docs_src/conf.py: -------------------------------------------------------------------------------- 1 | # Configuration file for the Sphinx documentation builder. 2 | # 3 | # This file only contains a selection of the most common options. For a full 4 | # list see the documentation: 5 | # http://www.sphinx-doc.org/en/master/config 6 | 7 | # -- Path setup -------------------------------------------------------------- 8 | 9 | # If extensions (or modules to document with autodoc) are in another directory, 10 | # add these directories to sys.path here. If the directory is relative to the 11 | # documentation root, use os.path.abspath to make it absolute, like shown here. 12 | # 13 | import os 14 | import sys 15 | 16 | import sphinx_nameko_theme 17 | from recommonmark.transform import AutoStructify 18 | 19 | sys.path.insert(0, os.path.abspath('../heapprof')) 20 | sys.path.insert(0, os.path.abspath('.')) 21 | 22 | 23 | # -- Project information ----------------------------------------------------- 24 | 25 | project = 'heapprof' 26 | copyright = '2019, Yonatan Zunger' 27 | author = 'Yonatan Zunger' 28 | 29 | 30 | # -- General configuration --------------------------------------------------- 31 | 32 | # Add any Sphinx extension module names here, as strings. They can be 33 | # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom 34 | # ones. 35 | extensions = [ 36 | 'sphinx.ext.autodoc', 37 | 'sphinx.ext.coverage', 38 | # This allows it to parse normal indentation in docstrings. 39 | 'sphinx.ext.napoleon', 40 | # This allows it to parse .md files as inputs. 41 | 'recommonmark', 42 | ] 43 | 44 | # Add any paths that contain templates here, relative to this directory. 45 | templates_path = ['_templates'] 46 | 47 | # List of patterns, relative to source directory, that match files and 48 | # directories to ignore when looking for source files. 49 | # This pattern also affects html_static_path and html_extra_path. 50 | exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store'] 51 | 52 | 53 | # -- Options for HTML output ------------------------------------------------- 54 | 55 | # The theme to use for HTML and HTML Help pages. See the documentation for 56 | # a list of builtin themes. 57 | html_theme_path = [sphinx_nameko_theme.get_html_theme_path()] 58 | html_theme = 'nameko' 59 | 60 | # Add any paths that contain custom static files (such as style sheets) here, 61 | # relative to this directory. They are copied after the builtin static files, 62 | # so a file named "default.css" will overwrite the builtin "default.css". 63 | html_static_path = ['_static'] 64 | 65 | autodoc_default_options = { 66 | 'members': True, 67 | 'member-order': 'bysource', 68 | 'undoc-members': True, 69 | 'show-inheritance': True, 70 | } 71 | 72 | 73 | def setup(app): 74 | app.add_config_value( 75 | 'recommonmark_config', 76 | {'enable_auto_toc_tree': True, 'auto_toc_tree_section': 'Contents'}, 77 | True, 78 | ) 79 | app.add_transform(AutoStructify) 80 | -------------------------------------------------------------------------------- /tools/docs.py: -------------------------------------------------------------------------------- 1 | import copy 2 | import logging 3 | import os 4 | import shutil 5 | import subprocess 6 | import sys 7 | from typing import List 8 | 9 | from _common import REPO_ROOT 10 | 11 | 12 | def findCommand(commandName: str) -> bool: 13 | return ( 14 | subprocess.run(f'which {commandName}', shell=True, stdout=subprocess.DEVNULL).returncode 15 | == 0 16 | ) 17 | 18 | 19 | def ensureRequirements() -> bool: 20 | # Ensure that everything required is installed. 21 | logging.info('Checking requirements') 22 | try: 23 | import sphinx # noqa 24 | import recommonmark # noqa 25 | except ImportError: 26 | reqsFile = os.path.abspath(os.path.join(REPO_ROOT, 'tools/requirements.txt')) 27 | relPath = os.path.relpath(os.getcwd(), reqsFile) 28 | logging.error(f'Required PIP packages are missing. Please run `pip install -r {relPath}') 29 | return False 30 | 31 | if not findCommand('make'): 32 | logging.error( 33 | 'make is not installed on your machine! Please ensure you have a working ' 34 | 'development environment with all the standard tools.' 35 | ) 36 | return False 37 | 38 | return True 39 | 40 | 41 | def clearOldSite() -> None: 42 | logging.info('Clearing old site images') 43 | shutil.rmtree(os.path.join(REPO_ROOT, '_build'), ignore_errors=True) 44 | shutil.rmtree(os.path.join(REPO_ROOT, 'docs'), ignore_errors=True) 45 | 46 | 47 | def rebuildSphinx() -> None: 48 | # Rebuilds docs/build from docs/. The master config for this is in docs/conf.py and 49 | # docs/Makefile. 50 | logging.info('Recompiling HTML from Sphinx') 51 | # Important trick: Make sure that even if the heapprof package is pip installed, we *don't* use 52 | # it: instead, 'import heapprof' should pull in REPO_ROOT/heapprof, so that you build from the 53 | # current client, not from the last time you ran 'python setup.py install' or the like. 54 | pythonPath: List[str] = [] 55 | for pathEntry in sys.path: 56 | if 'site-packages/heapprof' not in pathEntry: 57 | pythonPath.append(pathEntry) 58 | pythonPath.append(REPO_ROOT) 59 | 60 | env = copy.copy(os.environ) 61 | env['PYTHONPATH'] = ':'.join(pythonPath) 62 | 63 | subprocess.run( 64 | 'make html', cwd=os.path.join(REPO_ROOT, 'docs_src'), shell=True, check=True, env=env 65 | ) 66 | 67 | 68 | def configureSite() -> None: 69 | # While Jekyll would have no problem finding the docs in any directory, the Jekyll that runs as 70 | # part of GitHub Pages seems a bit more specific. 71 | shutil.move(os.path.join(REPO_ROOT, 'docs_src/build/html'), os.path.join(REPO_ROOT, 'docs')) 72 | shutil.rmtree(os.path.join(REPO_ROOT, 'docs_src/build')) 73 | 74 | with open(os.path.join(REPO_ROOT, 'docs/_config.yml'), 'w') as jekyllConfig: 75 | jekyllConfig.write( 76 | '# Config file for Jekyll serving; generated by tools/docs.py\n' 77 | 'baseurl: /heapprof\n' 78 | 'include:\n' 79 | ' - /_static\n' 80 | ' - /_images\n' 81 | ) 82 | 83 | 84 | def main(): 85 | logging.getLogger().addHandler(logging.StreamHandler()) 86 | logging.getLogger().setLevel(logging.INFO) 87 | 88 | if not ensureRequirements(): 89 | return 90 | 91 | clearOldSite() 92 | rebuildSphinx() 93 | configureSite() 94 | 95 | 96 | if __name__ == '__main__': 97 | main() 98 | -------------------------------------------------------------------------------- /docs/search.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | Search — heapprof documentation 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 43 | 44 |
45 |
46 |
47 |
48 | 49 |

Search

50 |
51 | 52 |

53 | Please activate JavaScript to enable the search 54 | functionality. 55 |

56 |
57 |

58 | From here you can search these documents. Enter your search 59 | words into the box below and click "search". Note that the search 60 | function will automatically search for all of the words. Pages 61 | containing fewer words won't appear in the result list. 62 |

63 |
64 | 65 | 66 | 67 |
68 | 69 |
70 | 71 |
72 | 73 |
74 |
75 |
76 | 84 |
85 |
86 | 87 | 88 | -------------------------------------------------------------------------------- /heapprof/__init__.py: -------------------------------------------------------------------------------- 1 | from typing import Dict, Optional 2 | 3 | import _heapprof 4 | 5 | # Types exposed as part of the API -- check out their .py files to learn more! 6 | from .flow_graph import FlowGraph # noqa 7 | from .reader import Reader 8 | from .types import (HeapTrace, RawTrace, RawTraceLine, Snapshot, # noqa 9 | TraceLine) 10 | 11 | # This default sampling rate was determined through some trial and error. However, it may or may not 12 | # be the right one for any particular case. 13 | DEFAULT_SAMPLING_RATE = {128: 1e-4, 8192: 0.1} 14 | 15 | 16 | def start(filebase: str, samplingRate: Optional[Dict[int, float]] = None) -> None: 17 | """Start heapprof in profiling (normal) mode. 18 | 19 | Args: 20 | filebase: The outputs will be written to filebase.{hpm, hpd}, a pair of local files which 21 | can later be read using the HeapProfile class. NB that these must be local files for 22 | performance reasons. 23 | samplingRate: A dict from byte size to sampling probability. Each byte size is interpreted 24 | as the upper bound of the range, and the sampling probability for byte sizes larger than 25 | the largest range given is always 1; thus the default value means to profile allocations 26 | of 1-127 bytes at 1 in 10,000, to profile allocations of 128-8,191 bytes at 1 in 10, and 27 | to profile all allocations of 8,192 bytes or more. 28 | 29 | Raises: 30 | TypeError: If samplingRate is not a mapping of the appropriate type. 31 | ValueError: If samplingRate contains repeated entries. 32 | RuntimeError: If the profiler is already running. 33 | """ 34 | _heapprof.startProfiler( 35 | filebase, samplingRate if samplingRate is not None else DEFAULT_SAMPLING_RATE 36 | ) 37 | 38 | 39 | def gatherStats() -> None: 40 | """Start heapprof in stats gathering mode. 41 | 42 | When the profiler is stopped, this will print out statistics on the size distribution of memory 43 | allocations. This can be useful for choosing sampling rates for profiling. 44 | """ 45 | _heapprof.startStats() 46 | 47 | 48 | def stop() -> None: 49 | """Stop the heap profiler. 50 | 51 | NB that if the program exits, this will be implicitly called. 52 | """ 53 | _heapprof.stop() 54 | 55 | 56 | def isProfiling() -> bool: 57 | """Test if the heap profiler is currently running.""" 58 | return _heapprof.isProfiling() 59 | 60 | 61 | def read(filebase: str, timeInterval: float = 60, precision: float = 0.01) -> Reader: 62 | """Open a reader, and create a digest for it if needed. 63 | 64 | Args: 65 | filebase: The name of the file to open; the same as the argument passed to start(). 66 | 67 | Args which apply only if you're creating the digest (i.e., opening it for the first time): 68 | timeInterval: The time interval between successive snapshots to store in the digest, 69 | in seconds. 70 | precision: At each snapshot, stack traces totalling up to this fraction of total 71 | memory used at that frame may be dropped into the "other stack trace" bucket. 72 | This can greatly shrink the size of the digest at no real cost in usefulness. 73 | Must be in [0, 1); a value of zero means nothing is dropped. 74 | """ 75 | r = Reader(filebase) 76 | if not r.hasDigest(): 77 | r.makeDigest(timeInterval=timeInterval, precision=precision, verbose=True) 78 | return r 79 | -------------------------------------------------------------------------------- /_heapprof/profiler.cc: -------------------------------------------------------------------------------- 1 | #include "_heapprof/profiler.h" 2 | #include 3 | #include 4 | #include 5 | #include "Python.h" 6 | #include "_heapprof/file_format.h" 7 | #include "_heapprof/port.h" 8 | #include "_heapprof/simple_hash.h" 9 | #include "_heapprof/util.h" 10 | #include "frameobject.h" 11 | 12 | ////////////////////////////////////////////////////////////////////////////////////////////////// 13 | // Profiler 14 | 15 | Profiler::Profiler(const char *filebase, Sampler *sampler) 16 | : sampler_(sampler), 17 | metadata_file_(filebase, ".hpm", true), 18 | data_file_(filebase, ".hpd", true) { 19 | if (!metadata_file_ || !data_file_) { 20 | return; 21 | } 22 | 23 | // Initialize our clock and write initial metadata. 24 | gettime(&last_clock_); 25 | WriteMetadata(metadata_file_, last_clock_, *sampler_); 26 | ok_ = true; 27 | } 28 | 29 | Profiler::~Profiler() {} 30 | 31 | void Profiler::HandleMalloc(void *ptr, size_t size) { 32 | assert(ptr != nullptr); 33 | if (!sampler_->Sample(size)) { 34 | return; 35 | } 36 | struct timespec timestamp; 37 | gettime(×tamp); 38 | const uint32_t traceindex = GetTraceIndex(); 39 | live_set_[ptr] = {traceindex, size}; 40 | WriteEvent(data_file_, &last_clock_, timestamp, traceindex, size, true); 41 | } 42 | 43 | void Profiler::HandleFree(void *ptr) { 44 | assert(ptr != nullptr); 45 | auto live_ptr = live_set_.find(ptr); 46 | // This means that this ptr wasn't sampled in HandleMalloc. 47 | if (live_ptr == live_set_.end()) { 48 | return; 49 | } 50 | struct timespec timestamp; 51 | gettime(×tamp); 52 | WriteEvent(data_file_, &last_clock_, timestamp, live_ptr->second.traceindex, 53 | live_ptr->second.size, false); 54 | live_set_.erase(live_ptr); 55 | } 56 | 57 | // Get a unique fingerprint of the current Python stack trace. NB that this 58 | // fingerprint will never be committed to disk, so it only needs to be unique 59 | // within the scope of this program execution; that means we can safely hash 60 | // just the code pointers. 61 | static uint32_t GetTraceFP() { 62 | const PyThreadState *tstate = PyGILState_GetThisThreadState(); 63 | if (tstate == nullptr) { 64 | // This is really weird and we should figure out what the appropriate error 65 | // handling is. 66 | return 0; 67 | } 68 | 69 | SimpleHash tracefp; 70 | for (PyFrameObject *pyframe = tstate->frame; pyframe != nullptr; 71 | pyframe = pyframe->f_back) { 72 | if (!SkipFrame(pyframe)) { 73 | tracefp.add(pyframe->f_code); 74 | tracefp.add(static_cast(PyFrame_GetLineNumber(pyframe))); 75 | } 76 | } 77 | return tracefp.get(); 78 | } 79 | 80 | int Profiler::GetTraceIndex() { 81 | const uint32_t tracefp = GetTraceFP(); 82 | if (PREDICT_FALSE(tracefp == 0)) { 83 | return 0; 84 | } 85 | 86 | const auto it = trace_index_.find(tracefp); 87 | if (PREDICT_TRUE(it != trace_index_.end())) { 88 | return it->second; 89 | } 90 | 91 | // First time we've seen this tracefp! Write it out to the metadata file, add 92 | // its new index, and return that. 93 | uint32_t new_index = next_trace_index_++; 94 | // If we can't write a stack trace, or if the trace index overflowed, give 95 | // this tracefp the "invalid index" value. 96 | if (PREDICT_FALSE(!WriteRawTrace(metadata_file_) || new_index & kHighBits)) { 97 | new_index = 0; 98 | } 99 | trace_index_[tracefp] = new_index; 100 | return new_index; 101 | } 102 | -------------------------------------------------------------------------------- /docs_src/code_of_conduct.md: -------------------------------------------------------------------------------- 1 | # Contributor Covenant Code of Conduct 2 | 3 | ### Our Pledge 4 | 5 | In the interest of fostering an open and welcoming environment, we as 6 | contributors and maintainers pledge to making participation in our project and 7 | our community a harassment-free experience for everyone, regardless of age, body 8 | size, disability, ethnicity, sex characteristics, gender identity and expression, 9 | level of experience, education, socio-economic status, nationality, personal 10 | appearance, race, religion, or sexual identity and orientation. 11 | 12 | ### Our Standards 13 | 14 | Examples of behavior that contributes to creating a positive environment 15 | include: 16 | 17 | * Using welcoming and inclusive language 18 | * Being respectful of differing viewpoints and experiences 19 | * Gracefully accepting constructive criticism 20 | * Focusing on what is best for the community 21 | * Showing empathy towards other community members 22 | 23 | Examples of unacceptable behavior by participants include: 24 | 25 | * The use of sexualized language or imagery and unwelcome sexual attention or 26 | advances 27 | * Trolling, insulting/derogatory comments, and personal or political attacks 28 | * Public or private harassment 29 | * Publishing others' private information, such as a physical or electronic 30 | address, without explicit permission 31 | * Other conduct which could reasonably be considered inappropriate in a 32 | professional setting 33 | 34 | ### Our Responsibilities 35 | 36 | Project maintainers are responsible for clarifying the standards of acceptable 37 | behavior and are expected to take appropriate and fair corrective action in 38 | response to any instances of unacceptable behavior. 39 | 40 | Project maintainers have the right and responsibility to remove, edit, or 41 | reject comments, commits, code, wiki edits, issues, and other contributions 42 | that are not aligned to this Code of Conduct, or to ban temporarily or 43 | permanently any contributor for other behaviors that they deem inappropriate, 44 | threatening, offensive, or harmful. 45 | 46 | ### Scope 47 | 48 | This Code of Conduct applies within all project spaces, and it also applies when 49 | an individual is representing the project or its community in public spaces. 50 | Examples of representing a project or community include using an official 51 | project e-mail address, posting via an official social media account, or acting 52 | as an appointed representative at an online or offline event. Representation of 53 | a project may be further defined and clarified by project maintainers. 54 | 55 | ### Enforcement 56 | 57 | Instances of abusive, harassing, or otherwise unacceptable behavior may be 58 | reported by contacting the project team at `opensource@humu.com`. All 59 | complaints will be reviewed and investigated and will result in a response that 60 | is deemed necessary and appropriate to the circumstances. The project team is 61 | obligated to maintain confidentiality with regard to the reporter of an incident. 62 | Further details of specific enforcement policies may be posted separately. 63 | 64 | Project maintainers who do not follow or enforce the Code of Conduct in good 65 | faith may face temporary or permanent repercussions as determined by other 66 | members of the project's leadership. 67 | 68 | ### Attribution 69 | 70 | This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4, 71 | available at https://www.contributor-covenant.org/version/1/4/code-of-conduct.html 72 | 73 | [homepage]: https://www.contributor-covenant.org 74 | 75 | For answers to common questions about this code of conduct, see 76 | https://www.contributor-covenant.org/faq 77 | -------------------------------------------------------------------------------- /docs/_sources/code_of_conduct.md.txt: -------------------------------------------------------------------------------- 1 | # Contributor Covenant Code of Conduct 2 | 3 | ### Our Pledge 4 | 5 | In the interest of fostering an open and welcoming environment, we as 6 | contributors and maintainers pledge to making participation in our project and 7 | our community a harassment-free experience for everyone, regardless of age, body 8 | size, disability, ethnicity, sex characteristics, gender identity and expression, 9 | level of experience, education, socio-economic status, nationality, personal 10 | appearance, race, religion, or sexual identity and orientation. 11 | 12 | ### Our Standards 13 | 14 | Examples of behavior that contributes to creating a positive environment 15 | include: 16 | 17 | * Using welcoming and inclusive language 18 | * Being respectful of differing viewpoints and experiences 19 | * Gracefully accepting constructive criticism 20 | * Focusing on what is best for the community 21 | * Showing empathy towards other community members 22 | 23 | Examples of unacceptable behavior by participants include: 24 | 25 | * The use of sexualized language or imagery and unwelcome sexual attention or 26 | advances 27 | * Trolling, insulting/derogatory comments, and personal or political attacks 28 | * Public or private harassment 29 | * Publishing others' private information, such as a physical or electronic 30 | address, without explicit permission 31 | * Other conduct which could reasonably be considered inappropriate in a 32 | professional setting 33 | 34 | ### Our Responsibilities 35 | 36 | Project maintainers are responsible for clarifying the standards of acceptable 37 | behavior and are expected to take appropriate and fair corrective action in 38 | response to any instances of unacceptable behavior. 39 | 40 | Project maintainers have the right and responsibility to remove, edit, or 41 | reject comments, commits, code, wiki edits, issues, and other contributions 42 | that are not aligned to this Code of Conduct, or to ban temporarily or 43 | permanently any contributor for other behaviors that they deem inappropriate, 44 | threatening, offensive, or harmful. 45 | 46 | ### Scope 47 | 48 | This Code of Conduct applies within all project spaces, and it also applies when 49 | an individual is representing the project or its community in public spaces. 50 | Examples of representing a project or community include using an official 51 | project e-mail address, posting via an official social media account, or acting 52 | as an appointed representative at an online or offline event. Representation of 53 | a project may be further defined and clarified by project maintainers. 54 | 55 | ### Enforcement 56 | 57 | Instances of abusive, harassing, or otherwise unacceptable behavior may be 58 | reported by contacting the project team at `opensource@humu.com`. All 59 | complaints will be reviewed and investigated and will result in a response that 60 | is deemed necessary and appropriate to the circumstances. The project team is 61 | obligated to maintain confidentiality with regard to the reporter of an incident. 62 | Further details of specific enforcement policies may be posted separately. 63 | 64 | Project maintainers who do not follow or enforce the Code of Conduct in good 65 | faith may face temporary or permanent repercussions as determined by other 66 | members of the project's leadership. 67 | 68 | ### Attribution 69 | 70 | This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4, 71 | available at https://www.contributor-covenant.org/version/1/4/code-of-conduct.html 72 | 73 | [homepage]: https://www.contributor-covenant.org 74 | 75 | For answers to common questions about this code of conduct, see 76 | https://www.contributor-covenant.org/faq 77 | -------------------------------------------------------------------------------- /_heapprof/file_format.h: -------------------------------------------------------------------------------- 1 | #ifndef _HEAPPROF_FILE_FORMAT_H__ 2 | #define _HEAPPROF_FILE_FORMAT_H__ 3 | 4 | #include 5 | #include "Python.h" 6 | #include "_heapprof/sampler.h" 7 | #include "frameobject.h" 8 | 9 | /////////////////////////////////////////////////////////////////////////////// 10 | // .hpm files: the metadata of a profile. 11 | // This consists of a metadata header, followed by a sequence of "raw traces." 12 | // Raw traces contain filenames and line numbers, and can be converted to nicer 13 | // structures by the wrapping Python code. 14 | 15 | // Write the metadata header to an .hpm file. 16 | void WriteMetadata(int fd, const struct timespec &start_clock, 17 | const Sampler &sampler); 18 | 19 | // Read the metadata header from an .hpm file. This will either return a tuple 20 | // (double initial_clock, Dict[int, double] sample_rate), or return nullptr and 21 | // set the exception. 22 | PyObject *ReadMetadata(int fd); 23 | 24 | // Write the current Python stack trace as a raw trace to the indicated file 25 | // descriptor. Returns false if there is no such trace! 26 | bool WriteRawTrace(int fd); 27 | 28 | // Read a single raw trace from the given file descriptor. Returns a 29 | // List[Tuple[str, int]] on success, or nullptr + raises an EOFError. 30 | PyObject *ReadRawTrace(int fd); 31 | 32 | // Check if a certain stack frame should be ignored when building and analyzing 33 | // stack traces. 34 | bool SkipFrame(PyFrameObject *pyframe); 35 | 36 | /////////////////////////////////////////////////////////////////////////////// 37 | // .hpd files: the raw log of events. 38 | // This consists of a sequence of event entries, each of which encodes a 39 | // timestamp, a trace index (which is a 1-based index into the list of traces in 40 | // the .hpm file), a number of bytes, and whether it's an alloc or free. 41 | 42 | // Bit constants for the index words in an event. 43 | const uint32_t kDeltaIsNegative = 0x80000000; 44 | const uint32_t kOperationIsFree = 0x40000000; 45 | const uint32_t kHighBits = (kDeltaIsNegative | kOperationIsFree); 46 | 47 | // Write a single heap event to the indicated file descriptor. last_clock is the 48 | // clock value of the previous event written; it will be updated by this method. 49 | // alloc is true for an allocation, or false for a free. 50 | void WriteEvent(int fd, struct timespec *last_clock, 51 | const struct timespec ×tamp, uint32_t traceindex, 52 | size_t size, bool alloc); 53 | 54 | // Read a single heap event from the given file descriptor, given a value for 55 | // the timestamp of the previous event. Returns either a tuple (double 56 | // delta-time, int traceindex, int size), or nullptr + raises an EOFError. 57 | PyObject *ReadEvent(int fd); 58 | 59 | /////////////////////////////////////////////////////////////////////////////// 60 | // .hpc files: a digested version of an .hpd file. 61 | // This file is created after profiling is over; it combines the events in an 62 | // .hpd file into a sequence of time snapshots, which can be random-accessed. 63 | 64 | // Read filebase.hpd and create filebase.hpc. 65 | bool MakeDigestFile(const char *filebase, int interval_msec, double precision, 66 | bool verbose); 67 | 68 | // Read the metadata and index from a .hpc file. Returns a 69 | // Tuple[float, float, List[int]], giving the initial time, the time delta 70 | // between frames, and a list of byte offsets for each frame in the file. 71 | PyObject *ReadDigestMetadata(int fd); 72 | 73 | // Read a single snapshot from a digest. The offset is one of the entries in the 74 | // list given in the metadata. The result is a dict from trace index to number 75 | // of live bytes at that instant. 76 | PyObject *ReadDigestEntry(int fd, Py_ssize_t offset); 77 | 78 | #endif // _HEAPPROF_FILE_FORMAT_H__ 79 | -------------------------------------------------------------------------------- /tools/lint.py: -------------------------------------------------------------------------------- 1 | import argparse 2 | import os 3 | import subprocess 4 | import sys 5 | from typing import List, Optional, Set 6 | 7 | from _common import REPO_ROOT 8 | 9 | EXCLUDE_NAMES = {'build', 'dist'} 10 | 11 | 12 | def addFileToList(filename: str, pyFiles: List[str], cppFiles: List[str]) -> None: 13 | parts = os.path.basename(filename).rsplit('.', 1) 14 | if len(parts) < 2: 15 | return 16 | 17 | suffix = parts[-1] 18 | if suffix == 'py': 19 | pyFiles.append(filename) 20 | elif suffix in ('c', 'cpp', 'cc', 'h', 'hpp'): 21 | cppFiles.append(filename) 22 | 23 | 24 | def findFiles( 25 | rootDir: str, pyFiles: List[str], cppFiles: List[str], seenDirs: Optional[Set[str]] = None 26 | ) -> None: 27 | """Find all Python and C++ files for us to lint.""" 28 | seenDirs = seenDirs or set() 29 | rootDir = os.path.abspath(rootDir) 30 | seenDirs.add(rootDir) 31 | 32 | for dirEntry in os.scandir(rootDir): 33 | if dirEntry.name in EXCLUDE_NAMES or dirEntry.name.startswith('.'): 34 | continue 35 | 36 | if dirEntry.is_dir() and dirEntry.path not in seenDirs: 37 | findFiles(dirEntry.path, pyFiles, cppFiles, seenDirs) 38 | elif dirEntry.is_file(): 39 | addFileToList(dirEntry.path, pyFiles, cppFiles) 40 | 41 | 42 | def runCommand(*command: str) -> bool: 43 | pythonPath = ':'.join(sys.path) 44 | try: 45 | subprocess.check_call([f'PYTHONPATH="{pythonPath}"', *command], cwd=REPO_ROOT, shell=True) 46 | return True 47 | except subprocess.CalledProcessError: 48 | return False 49 | 50 | 51 | def lintPyFiles(files: List[str]) -> bool: 52 | if not files: 53 | return True 54 | 55 | ok = True 56 | if not runCommand('python', '-m', 'flake8', *files): 57 | ok = False 58 | if not runCommand('python', '-m', 'isort', '--atomic', '--check-only', *files): 59 | ok = False 60 | if not runCommand('python', '-m', 'black', '--check', *files): 61 | ok = False 62 | if not runCommand('python', '-m', 'mypy', *files): 63 | ok = False 64 | 65 | print(f'Lint of {len(files)} Python files {"successful" if ok else "failed"}!') 66 | return ok 67 | 68 | 69 | def fixPyFiles(files: List[str]) -> None: 70 | if not files: 71 | return 72 | 73 | runCommand('python', '-m', 'isort', '--atomic', *files) 74 | runCommand('python', '-m', 'black', *files) 75 | 76 | 77 | def lintCppFiles(files: List[str]) -> bool: 78 | if not files: 79 | return True 80 | 81 | ok = runCommand('python', '-m', 'cpplint', '--quiet', *files) 82 | print(f'Lint of {len(files)} C/C++ files {"successful" if ok else "failed"}!') 83 | return ok 84 | 85 | 86 | def fixCppFiles(files: List[str]) -> None: 87 | if not files: 88 | return 89 | 90 | return runCommand('clang-format', '-i', '-style=Google', *files) 91 | 92 | 93 | if __name__ == '__main__': 94 | parser = argparse.ArgumentParser('linter') 95 | parser.add_argument('--fix', action='store_true', help='Fix files in-place') 96 | parser.add_argument('files', nargs='*', help='Files to lint; default all') 97 | args = parser.parse_args() 98 | 99 | pyFiles: List[str] = [] 100 | cppFiles: List[str] = [] 101 | if args.files: 102 | for file in args.files: 103 | addFileToList(file, pyFiles, cppFiles) 104 | else: 105 | findFiles(REPO_ROOT, pyFiles, cppFiles) 106 | 107 | # If there was a fix request, run that first. 108 | if args.fix: 109 | fixPyFiles(pyFiles) 110 | fixCppFiles(cppFiles) 111 | 112 | ok = lintPyFiles(pyFiles) 113 | if not lintCppFiles(cppFiles): 114 | ok = False 115 | 116 | if not ok: 117 | sys.exit(1) 118 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | [![Python 3.7](https://img.shields.io/badge/python-3.7-blue.svg)](https://www.python.org/downloads/release/python-374/) 2 | [![Contributor Covenant](https://img.shields.io/badge/Contributor%20Covenant-v1.4%20adopted-ff69b4.svg)](https://humu.github.io/heapprof/code_of_conduct.html) 3 | [![MIT License](https://img.shields.io/badge/License-MIT-yellow.svg)](https://humu.github.io/heapprof/license.html) 4 | [![CircleCI](https://circleci.com/gh/humu/heapprof/tree/master.svg?style=svg&circle-token=1557bfcabda0155d6505a45e3f00d4a71a005565)](https://circleci.com/gh/humu/heapprof/tree/master) 5 | 6 | # heapprof: A Logging Heap Profiler 7 | 8 | heapprof is a logging, sampling heap profiler for Python 3.7+. 9 | 10 | * "Logging" means that as the program runs, it steadily generates a log of memory allocation and 11 | release events. This means that you can easily look at memory usage as a function of time. 12 | * "Sampling" means that it can record only a statistically random sample of memory events. This 13 | improves performance dramatically while writing logs, and (with the right parameters) sacrifices 14 | almost no accuracy. 15 | 16 | It comes with a suite of visualization and analysis tools (including time plots, flame graphs, and 17 | flow graphs), as well as an API for doing your own analyses of the results. 18 | 19 | [![screenshot of split time plot](https://humu.github.io/heapprof/_images/split_time_plot.png)](https://humu.github.io/heapprof/visualizing_results.html) 20 | 21 | heapprof is complementary to [tracemalloc](https://docs.python.org/3/library/tracemalloc.html), 22 | which is a snapshotting heap profiler. The difference is that tracemalloc keeps track of live memory 23 | internally, and only writes snapshots when its snapshot() function is called; this means it has 24 | slightly lower overhead, but you have to know the moments at which you'll want a snapshot before the 25 | program starts. This makes it particularly useful for finding leaks (from the snapshot at program 26 | exit), but not as good for understanding events like memory spikes. 27 | 28 | You can install heapprof with `pip install heapprof`. heapprof is released under the 29 | [MIT License](https://humu.github.io/heapprof/license.html). 30 | 31 | You can read all the documentation at [humu.github.io/heapprof](https://humu.github.io/heapprof). 32 | 33 | ## Navigating the Repository 34 | 35 | If you're trying to find something in the GitHub repository, here's a brief directory (since, like 36 | most Python packages, this is a maze of twisty subdirectories, all different): 37 | 38 | * `heapprof` contains the Python package itself. (The API and visualization logic) 39 | * `_heapprof` contains the C/C++ package. (The core profiling logic) 40 | * `docs_src` contains the sources for the documentation, mostly as `.md` and `.rst` files. 41 | * `docs` contains the compiled HTML version of `docs_src`, created with `tools/docs.py` and checked 42 | in. 43 | * `tools` contains tools useful when modifying heapprof itself. 44 | * And then there are the configuration files for all the tools: 45 | * `setup.py` is the master build configuration for the PIP package. 46 | * `.flake8` and `.pylintrc` are the configuration for Python linting. 47 | * `CPPLINT.cfg` is the configuration for C/C++ linting. 48 | * `mypy.ini` is the configuration for Python type checking. 49 | * `Gemfile` is for setting up Jekyll for documentation hosting. 50 | * `_config.yml` is the configuration for Jekyll serving. 51 | * `docs/Makefile` and `docs/conf.py` are the configuration for building the HTML docs image via 52 | Sphinx. 53 | * `.circleci` is the configuration for continuous integration testing. 54 | * `pyproject.toml` and the root `requirements.txt` make `setuptools` happy. 55 | * Additional directories which are .gitignored but which show up during use: 56 | * `build` contains C/C++ dependencies and their compiled images; it's managed by `setup.py`. 57 | * `_site` contains the final Jekyll site which is served for documentation; it's created if you 58 | run `bundle exec jekyll serve` to run the docs web server locally. 59 | -------------------------------------------------------------------------------- /_heapprof/util.cc: -------------------------------------------------------------------------------- 1 | #include "_heapprof/util.h" 2 | #include 3 | 4 | #define VARINT_BUFFER_SIZE MAX_UNSIGNED_VARINT_SIZE(uint64_t) 5 | 6 | static uint8_t g_varint_buffer[VARINT_BUFFER_SIZE]; 7 | 8 | void WriteVarintToFile(int fd, uint64_t value) { 9 | const uint8_t *end = UnsafeAppendVarint(g_varint_buffer, value); 10 | write(fd, g_varint_buffer, end - g_varint_buffer); 11 | } 12 | 13 | void WriteFixed32ToFile(int fd, uint32_t value) { 14 | const uint32_t data = absl::ghtonl(value); 15 | write(fd, &data, sizeof(uint32_t)); 16 | } 17 | 18 | void WriteFixed64ToFile(int fd, uint64_t value) { 19 | const uint64_t data = absl::ghtonll(value); 20 | write(fd, &data, sizeof(uint64_t)); 21 | } 22 | 23 | bool ReadFixed32FromFile(int fd, uint32_t *value) { 24 | const int bytes_read = read(fd, value, sizeof(uint32_t)); 25 | if (bytes_read != sizeof(uint32_t)) { 26 | PyErr_SetString(PyExc_EOFError, ""); 27 | return false; 28 | } 29 | *value = absl::gntohl(*value); 30 | return true; 31 | } 32 | 33 | bool ReadFixed64FromFile(int fd, uint64_t *value) { 34 | const int bytes_read = read(fd, value, sizeof(uint64_t)); 35 | if (bytes_read != sizeof(uint64_t)) { 36 | PyErr_SetString(PyExc_EOFError, ""); 37 | return false; 38 | } 39 | *value = absl::gntohll(*value); 40 | return true; 41 | } 42 | 43 | bool ReadVarintFromFile(int fd, uint64_t *value) { 44 | // Read the maximum number of bytes which could potentially be valid right 45 | // now; we'll put back the unused bytes later. 46 | const int bytes_read = read(fd, g_varint_buffer, VARINT_BUFFER_SIZE); 47 | *value = 0; 48 | int pos = 0; 49 | while (pos < bytes_read) { 50 | *value |= (g_varint_buffer[pos] & 0x7f) << (7 * pos); 51 | if (g_varint_buffer[pos] & 0x80) { 52 | pos++; 53 | } else { 54 | // Done! Put the remaining bytes back on the file. 55 | lseek(fd, -(bytes_read - pos - 1), SEEK_CUR); 56 | return true; 57 | } 58 | } 59 | // If we get here, the read failed! 60 | if (pos == bytes_read) { 61 | PyErr_SetString(PyExc_ValueError, 62 | "Found a varint which could not decode into a uint64"); 63 | } else { 64 | PyErr_SetString(PyExc_EOFError, ""); 65 | } 66 | return false; 67 | } 68 | 69 | bool WriteStringToFile(int fd, PyObject *value) { 70 | Py_ssize_t len; 71 | const char *cstr = PyUnicode_AsUTF8AndSize(value, &len); 72 | if (cstr == nullptr) { 73 | // Exception already set. 74 | return false; 75 | } 76 | WriteVarintToFile(fd, len); 77 | write(fd, cstr, len); 78 | return true; 79 | } 80 | 81 | PyObject *ReadStringFromFile(int fd) { 82 | uint64_t len; 83 | if (!ReadVarintFromFile(fd, &len)) { 84 | // Exception already set. 85 | return nullptr; 86 | } 87 | const Py_ssize_t size = static_cast(len); 88 | 89 | char *buffer = reinterpret_cast(malloc(size)); 90 | if (buffer == nullptr) { 91 | PyErr_Format(PyExc_MemoryError, 92 | "Failed to allocate %zd bytes for string read", size); 93 | return nullptr; 94 | } 95 | const int bytes_read = read(fd, buffer, size); 96 | if (bytes_read < size) { 97 | free(buffer); 98 | PyErr_SetString(PyExc_EOFError, ""); 99 | return nullptr; 100 | } 101 | 102 | // Succeed or fail, we'll return the result of this. 103 | PyObject *result = PyUnicode_DecodeUTF8(buffer, size, "strict"); 104 | 105 | free(buffer); 106 | return result; 107 | } 108 | 109 | static inline int ScopedFileMode(bool write) { 110 | return write ? WRITE_MODE : READ_MODE; 111 | } 112 | 113 | ScopedFile::ScopedFile(const char *filebase, const char *suffix, bool write) 114 | : filename_(std::string(filebase) + suffix), 115 | fd_(open(filename_.c_str(), ScopedFileMode(write), 0600)), 116 | delete_(false) { 117 | if (fd_ == -1) { 118 | PyErr_SetFromErrnoWithFilenameObject( 119 | PyExc_OSError, 120 | PyUnicode_FromStringAndSize(filename_.c_str(), filename_.length())); 121 | } 122 | } 123 | 124 | ScopedFile::~ScopedFile() { 125 | if (fd_ != -1 && close(fd_) == -1) { 126 | PyErr_SetFromErrno(PyExc_OSError); 127 | return; 128 | } 129 | if (delete_ && unlink(filename_.c_str()) == -1) { 130 | PyErr_SetFromErrno(PyExc_OSError); 131 | return; 132 | } 133 | } 134 | -------------------------------------------------------------------------------- /_heapprof/port.h: -------------------------------------------------------------------------------- 1 | #ifndef _HEAPPROF_PORT_H__ 2 | #define _HEAPPROF_PORT_H__ 3 | 4 | // System portability library, with various things so this can compile on a range of machines. 5 | 6 | // All Windows platforms need this soonest. 7 | #ifdef _WIN64 8 | #include 9 | #elif _WIN32 10 | // TODO, maybe support this someday? It would require a lot more care with integer sizes in various 11 | // places, but 32-bit machines are a dying breed. 12 | #error heapprof does not currently support WIN32 builds. 13 | #endif 14 | 15 | //////////////////////////////////////////////////////////////////////////////////////////// 16 | // Time-related definitions 17 | 18 | // Include this on all machines, but append things to it in some cases. 19 | #include 20 | 21 | // Define clock_gettime etc for Windows boxen. 22 | #ifdef _WIN64 23 | 24 | inline void gettime(struct timespec *spec) { 25 | // This function returns the number of 100-nanosecond intervals (decashakes) since midnight 26 | // January 1st, 1601 UTC. This is a rather interesting combination of base and unit, being 27 | // roughly what you would need to describe nuclear chain reactions around the time of the 28 | // Protestant Reformation. 29 | uint64_t wintime; 30 | GetSystemTimeAsFileTime(reinterpret_cast(&wintime)); 31 | 32 | // Convert to decashakes since the UNIX Epoch. 33 | const int64_t since_epoch = wintime - 116444736000000000LL; 34 | 35 | spec->tv_sec = since_epoch / 10000000LL; 36 | spec->tv_nsec = (since_epoch % 10000000LL) * 100; 37 | } 38 | 39 | #else 40 | 41 | // POSIX systems just have a function for this. 42 | inline void gettime(struct timespec *spec) { 43 | clock_gettime(CLOCK_REALTIME, spec); 44 | } 45 | 46 | #endif 47 | 48 | //////////////////////////////////////////////////////////////////////////////////////////// 49 | // Bitwise operations, bytesex, and CPU branching. 50 | 51 | // A note here: We need to achieve some portable operations which aren't yet 52 | // available in the C++ standard, but the portable logic for them is 53 | // depressingly long. So we include ABSL, which has extremely nice 54 | // implementations of them. However, these are in the base/internal directory, 55 | // because the ABSL team hasn't decided to make them formally part of the spec 56 | // yet. At some point, these are definitely going to be moved out of internal, 57 | // just Not Quite Yet. (Signed, the original author of endian.h and quite a bit 58 | // of the other stuff in this directory; sigh. -- zunger@) 59 | #include "absl/base/internal/bits.h" 60 | // This file defines ghton[size] and gntoh[size]. 61 | #include "absl/base/internal/endian.h" 62 | 63 | // C++20 will have a standardized version of this. Until then, we use 64 | // compiler-specific directives, which are notably missing in MSVC. 65 | #if __clang__ || __GNUC__ 66 | #define PREDICT_FALSE(expr) __builtin_expect(static_cast(expr), 0) 67 | #define PREDICT_TRUE(expr) __builtin_expect(static_cast(expr), 1) 68 | #else 69 | #define PREDICT_FALSE(expr) (expr) 70 | #define PREDICT_TRUE(expr) (expr) 71 | #endif 72 | 73 | // Return ceil(log2(x)). 74 | inline int Log2RoundUp(uint64_t x) { 75 | return x ? 64 - absl::base_internal::CountLeadingZeros64(x - 1) : 0; 76 | } 77 | 78 | //////////////////////////////////////////////////////////////////////////////////////////// 79 | // File system access 80 | 81 | #ifdef _WIN64 82 | // Seriously, Microsoft? You don't have pwrite? Normal people implement write *on top of* pwrite. 83 | inline ssize_t pwrite(int fd, const void *buf, size_t nbytes, off_t offset) { 84 | const off_t pos = lseek(fd, 0, SEEK_CUR); 85 | lseek(fd, offset, SEEK_SET); 86 | const ssize_t written = write(fd, buf, nbytes); 87 | lseek(fd, pos, SEEK_SET); 88 | return written; 89 | } 90 | 91 | // _O_BINARY has no POSIX equivalent, but if you don't set it, it will default to a text mode that 92 | // will do "helpful" things like translate 0x0a to 0x0d0a when you write it. 93 | #define WRITE_MODE _O_WRONLY | _O_CREAT | _O_TRUNC | _O_BINARY 94 | #define READ_MODE _O_RDONLY | _O_BINARY 95 | 96 | #else // Non-Windows machines 97 | 98 | // Defines write() etc on Unices. 99 | #include 100 | 101 | #define WRITE_MODE O_WRONLY | O_CREAT | O_TRUNC 102 | #define READ_MODE O_RDONLY 103 | 104 | #endif // Switch over platforms 105 | 106 | #endif // _HEAPPROF_PORT_H__ 107 | -------------------------------------------------------------------------------- /.circleci/config.yml: -------------------------------------------------------------------------------- 1 | # CircleCI 2.0 configuration file 2 | # 3 | # Check https://circleci.com/docs/2.0/configuration-reference/ for more details. 4 | # 5 | version: 2.1 6 | 7 | executors: 8 | humu-ci-docker-image: 9 | docker: 10 | # specify the version you desire here 11 | - image: circleci/python:3.7.2 12 | 13 | # Specify service dependencies here if necessary 14 | # CircleCI maintains a library of pre-built images 15 | # documented at https://circleci.com/docs/2.0/circleci-images/ 16 | # - image: circleci/mongo:3.4.4 17 | 18 | orbs: 19 | win: circleci/windows@1.0.0 20 | 21 | jobs: 22 | test: 23 | executor: humu-ci-docker-image 24 | working_directory: ~/repo 25 | steps: 26 | - checkout 27 | 28 | # Install dependencies, with caching. 29 | - restore_cache: 30 | keys: 31 | - v1-dependencies-{{ checksum "requirements.txt" }} 32 | # fallback to using the latest cache if no exact match is found 33 | - v1-dependencies- 34 | - run: 35 | name: install_dependencies 36 | command: | 37 | python -m venv venv 38 | . venv/bin/activate 39 | pip install --upgrade pip setuptools cmake wheel 40 | - save_cache: 41 | paths: 42 | - ./venv 43 | key: v1-dependencies-{{ checksum "requirements.txt" }} 44 | 45 | - run: 46 | name: run_tests 47 | command: | 48 | . venv/bin/activate 49 | python setup.py test 50 | 51 | - store_artifacts: 52 | path: test-reports 53 | destination: test-reports 54 | 55 | 56 | lint: 57 | executor: humu-ci-docker-image 58 | working_directory: ~/repo 59 | steps: 60 | - checkout 61 | 62 | # Install dependencies, with caching. 63 | - restore_cache: 64 | keys: 65 | - v1-dependencies-{{ checksum "tools/requirements.txt" }} 66 | # fallback to using the latest cache if no exact match is found 67 | - v1-dependencies- 68 | - run: 69 | name: install_dependencies 70 | command: | 71 | python -m venv venv 72 | . venv/bin/activate 73 | pip install --upgrade pip 74 | pip install -r tools/requirements.txt 75 | - save_cache: 76 | paths: 77 | - ./venv 78 | key: v1-dependencies-{{ checksum "tools/requirements.txt" }} 79 | 80 | - run: 81 | name: run_lint 82 | command: | 83 | . venv/bin/activate 84 | python tools/lint.py 85 | 86 | - store_artifacts: 87 | path: test-reports 88 | destination: test-reports 89 | 90 | # Use this to cross-compile a Windows binary and make a wheel. The resulting artifacts from 91 | # release candidates go into PIP. 92 | windows_64: 93 | executor: 94 | name: win/vs2019 95 | steps: 96 | - checkout 97 | - restore_cache: 98 | keys: 99 | - win-python-3.7.4 100 | - run: 101 | name: install_python 102 | command: | 103 | choco upgrade python --version 3.7.4 -c C:\downloadcache -y 104 | choco upgrade cmake -c C:\downloadcache -y --installargs 'ADD_CMAKE_TO_PATH=System' 105 | - run: 106 | name: upgrade_python 107 | command: | 108 | refreshenv 109 | Python.exe -m pip install --upgrade pip setuptools wheel 110 | - save_cache: 111 | paths: 112 | - C:\downloadcache 113 | key: win-python-3.7.4 114 | - run: 115 | name: test_and_build 116 | command: | 117 | refreshenv 118 | Python.exe setup.py test 119 | Python.exe setup.py bdist_wheel 120 | - store_artifacts: 121 | path: dist 122 | destination: wheel 123 | - store_artifacts: 124 | path: test-reports 125 | destination: test-reports 126 | 127 | 128 | workflows: 129 | version: 2 130 | test: 131 | jobs: 132 | - lint 133 | - test 134 | - windows_64 135 | 136 | 137 | # Only notify slack about build failures/fixes on master! 138 | # https://discuss.circleci.com/t/only-notify-for-some-branches/10244 139 | experimental: 140 | notify: 141 | branches: 142 | only: 143 | - master 144 | -------------------------------------------------------------------------------- /docs/_static/pygments.css: -------------------------------------------------------------------------------- 1 | .highlight .hll { background-color: #ffffcc } 2 | .highlight { background: #f8f8f8; } 3 | .highlight .c { color: #888 } /* Comment */ 4 | .highlight .err { color: #a40000; border: 1px solid #ef2929 } /* Error */ 5 | .highlight .g { color: #000000 } /* Generic */ 6 | .highlight .k { color: #004461; font-weight: bold } /* Keyword */ 7 | .highlight .l { color: #000000 } /* Literal */ 8 | .highlight .n { color: #000000 } /* Name */ 9 | .highlight .o { color: #582800 } /* Operator */ 10 | .highlight .x { color: #000000 } /* Other */ 11 | .highlight .p { color: #000000; font-weight: bold } /* Punctuation */ 12 | .highlight .cm { color: #8f5902; font-style: italic } /* Comment.Multiline */ 13 | .highlight .cp { color: #8f5902 } /* Comment.Preproc */ 14 | .highlight .c1 { color: #8f5902; font-style: italic } /* Comment.Single */ 15 | .highlight .cs { color: #8f5902; font-style: italic } /* Comment.Special */ 16 | .highlight .gd { color: #a40000 } /* Generic.Deleted */ 17 | .highlight .ge { color: #000000; font-style: italic } /* Generic.Emph */ 18 | .highlight .gr { color: #ef2929 } /* Generic.Error */ 19 | .highlight .gh { color: #000080; font-weight: bold } /* Generic.Heading */ 20 | .highlight .gi { color: #00A000 } /* Generic.Inserted */ 21 | .highlight .go { color: #888888 } /* Generic.Output */ 22 | .highlight .gp { color: #745334 } /* Generic.Prompt */ 23 | .highlight .gs { color: #000000; font-weight: bold } /* Generic.Strong */ 24 | .highlight .gu { color: #800080; font-weight: bold } /* Generic.Subheading */ 25 | .highlight .gt { color: #a40000; font-weight: bold } /* Generic.Traceback */ 26 | .highlight .kc { color: #004461; font-weight: bold } /* Keyword.Constant */ 27 | .highlight .kd { color: #004461; font-weight: bold } /* Keyword.Declaration */ 28 | .highlight .kn { color: #004461; font-weight: bold } /* Keyword.Namespace */ 29 | .highlight .kp { color: #004461; font-weight: bold } /* Keyword.Pseudo */ 30 | .highlight .kr { color: #004461; font-weight: bold } /* Keyword.Reserved */ 31 | .highlight .kt { color: #004461; font-weight: bold } /* Keyword.Type */ 32 | .highlight .ld { color: #000000 } /* Literal.Date */ 33 | .highlight .m { color: #990000 } /* Literal.Number */ 34 | .highlight .s { color: #3b7c03 } /* Literal.String */ 35 | .highlight .na { color: #c4a000 } /* Name.Attribute */ 36 | .highlight .nb { color: #004461 } /* Name.Builtin */ 37 | .highlight .nc { color: #000000 } /* Name.Class */ 38 | .highlight .no { color: #000000 } /* Name.Constant */ 39 | .highlight .nd { color: #888888 } /* Name.Decorator */ 40 | .highlight .ni { color: #ce5c00 } /* Name.Entity */ 41 | .highlight .ne { color: #cc0000; font-weight: bold } /* Name.Exception */ 42 | .highlight .nf { color: #000000 } /* Name.Function */ 43 | .highlight .nl { color: #f57900 } /* Name.Label */ 44 | .highlight .nn { color: #000000 } /* Name.Namespace */ 45 | .highlight .nx { color: #000000 } /* Name.Other */ 46 | .highlight .py { color: #000000 } /* Name.Property */ 47 | .highlight .nt { color: #004461; font-weight: bold } /* Name.Tag */ 48 | .highlight .nv { color: #000000 } /* Name.Variable */ 49 | .highlight .ow { color: #004461; font-weight: bold } /* Operator.Word */ 50 | .highlight .w { color: #f8f8f8; text-decoration: underline } /* Text.Whitespace */ 51 | .highlight .mf { color: #990000 } /* Literal.Number.Float */ 52 | .highlight .mh { color: #990000 } /* Literal.Number.Hex */ 53 | .highlight .mi { color: #990000 } /* Literal.Number.Integer */ 54 | .highlight .mo { color: #990000 } /* Literal.Number.Oct */ 55 | .highlight .sb { color: #4e9a06 } /* Literal.String.Backtick */ 56 | .highlight .sc { color: #4e9a06 } /* Literal.String.Char */ 57 | .highlight .sd { color: #8f5902 } /* Literal.String.Doc */ 58 | .highlight .s2 { color: #4e9a06 } /* Literal.String.Double */ 59 | .highlight .se { color: #4e9a06 } /* Literal.String.Escape */ 60 | .highlight .sh { color: #4e9a06 } /* Literal.String.Heredoc */ 61 | .highlight .si { color: #4e9a06 } /* Literal.String.Interpol */ 62 | .highlight .sx { color: #4e9a06 } /* Literal.String.Other */ 63 | .highlight .sr { color: #4e9a06 } /* Literal.String.Regex */ 64 | .highlight .s1 { color: #4e9a06 } /* Literal.String.Single */ 65 | .highlight .ss { color: #4e9a06 } /* Literal.String.Symbol */ 66 | .highlight .bp { color: #3465a4 } /* Name.Builtin.Pseudo */ 67 | .highlight .vc { color: #000000 } /* Name.Variable.Class */ 68 | .highlight .vg { color: #000000 } /* Name.Variable.Global */ 69 | .highlight .vi { color: #000000 } /* Name.Variable.Instance */ 70 | .highlight .il { color: #990000 } /* Literal.Number.Integer.Long */ 71 | -------------------------------------------------------------------------------- /docs/py-modindex.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | Python Module Index — heapprof documentation 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 41 | 42 |
43 |
44 |
45 |
46 | 47 | 48 |

Python Module Index

49 | 50 |
51 | h 52 |
53 | 54 | 55 | 56 | 58 | 59 | 61 | 64 | 65 | 66 | 69 | 70 | 71 | 74 | 75 | 76 | 79 | 80 | 81 | 84 |
 
57 | h
62 | heapprof 63 |
    67 | heapprof.flow_graph 68 |
    72 | heapprof.lowlevel 73 |
    77 | heapprof.reader 78 |
    82 | heapprof.types 83 |
85 | 86 | 87 |
88 |
89 |
90 | 108 |
109 |
110 | 111 | 112 | -------------------------------------------------------------------------------- /docs/api/index.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | The heapprof API — heapprof documentation 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 46 | 47 |
48 |
49 |
50 |
51 | 52 | 65 | 66 | 67 |
68 |
69 |
70 | 97 |
98 |
99 | 100 | 101 | -------------------------------------------------------------------------------- /docs/license.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | MIT License — heapprof documentation 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 46 | 47 |
48 |
49 |
50 |
51 | 52 |
53 |

MIT License

54 |

Copyright (c) 2019 Humu, Inc.

55 |

Permission is hereby granted, free of charge, to any person obtaining a copy 56 | of this software and associated documentation files (the “Software”), to deal 57 | in the Software without restriction, including without limitation the rights 58 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 59 | copies of the Software, and to permit persons to whom the Software is 60 | furnished to do so, subject to the following conditions:

61 |

The above copyright notice and this permission notice shall be included in all 62 | copies or substantial portions of the Software.

63 |

THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 64 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 65 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 66 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 67 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 68 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 69 | SOFTWARE.

70 |
71 | 72 | 73 |
74 |
75 |
76 | 103 |
104 |
105 | 106 | 107 | -------------------------------------------------------------------------------- /docs_src/using_heapprof.md: -------------------------------------------------------------------------------- 1 | # Using heapprof 2 | 3 | The basic flow of using heapprof is: 4 | 5 | 1. [Optional] Adjust parameters like the [sampling rate](advanced_heapprof.md#controlling-sampling). 6 | 1. Run your program with heapprof turned on. This generates .hpm (metadata) and .hpd (data) files. 7 | 1. Open the output files with `heapprof.read()` and create a .hpc (digest) file. 8 | 1. Analyze the resulting data with visualization tools, or do your own poking with the API. 9 | 10 | The reason heapprof needs so many files is that it's doing everything it can to minimize overhead 11 | while your program is running. By writing two files (one with things like stack traces, the other 12 | with the individual events that happen) instead of one, it can minimize how much time is spent on 13 | disk I/O; by deferring the work of combining individual events into easily-understood histories, it 14 | both saves CPU and I/O during runtime, and gives the analysis tools more flexibility via access to 15 | the raw data. 16 | 17 | This means that at least the first time you open a set of .hpx files, you'll need to spend some time 18 | processing it; this can range from seconds to minutes, depending on just how much data you've 19 | accumulated. 20 | 21 | > **Tip:** Partially-written .hpd and .hpm files are valid; this means you can start running 22 | > analysis while your program is still going, and it shouldn't interfere with either your program or 23 | > with further data collection! Partially-written digest files are also valid; this means that you 24 | > can ctrl-C digest building partway through, and you'll get a perfectly valid digest up to the 25 | > timestamp where you stopped it. You can always replace the digest by calling 26 | > `heapprof.Reader.makeDigest()` again, with new options. 27 | 28 | ## Installation and system requirements 29 | 30 | To install heapprof, simply run `pip install heapprof`. Some of heapprof's data visualization 31 | features require additional packages as well: 32 | 33 | * For time plots, you will also need [matplotlib](https://matplotlib.org/). In most cases, you can 34 | install this by simply running `pip install matplotlib`. 35 | * For flow graphs, you will also need [graphviz](https://www.graphviz.org/). graphviz is not a 36 | Python package; although there are Python packages with similar names that wrap it, all of them 37 | require that you install the underlying tool. See the graphviz site for how best to get it on 38 | your platform. 39 | * For flame graphs, you can either use [speedscope.app](https://www.speedscope.app/) as a web 40 | application (and just send your flame data there), or you can install and run it locally with 41 | `npm install -g speedscope`. (See its [GitHub site](https://github.com/jlfwong/speedscope) for 42 | more installation details if you aren't familiar with running local JavaScript code) 43 | 44 | > #### System Requirements 45 | > heapprof is designed to work with Python 3.7 or greater, using the CPython runtime. If you are using 46 | > an older version of Python or a less-common runtime, this package won't work, and you'll get strange 47 | > errors if you try. 48 | > 49 | > The current heapprof release includes binaries for Linux x86_64, OS X, and 64-bit Windows. On 50 | > other platforms, you can try compiling from a source distribution; if you have to make portability 51 | > changes (which you almost certainly will), those would be good to commit back to the main branch. 52 | 53 | 54 | ## Running your code 55 | 56 | heapprof can be invoked either programmatically or from the command line. Programmatically, you call 57 | 58 | ``` 59 | import heapprof 60 | 61 | heapprof.start('/some/path/to/output') 62 | ... do something ... 63 | heapprof.stop() 64 | ``` 65 | 66 | You don't need to do fancy exception-guarding; if the program exits before heapprof.stop() is 67 | called, it will clean up automatically. The path you provide to `heapprof.start` is known as the 68 | "filebase;" output will be written to filebase.hpd, filebase.hpm, and so on. 69 | 70 | If you want to wrap an entire program execution in heapprof, it's even simpler: 71 | 72 | ``` 73 | python -m heapprof -o /output/file -- myprogram.py ... 74 | ``` 75 | 76 | You can pass arguments to your Python code just like you usually would. 77 | 78 | ## Making a digest 79 | 80 | The files output by heapprof directly contain a sequence of events: timestamps and heap locations at 81 | which memory was allocated or freed. Nearly all ways to analyze this rely on instead having a 82 | picture of how memory looked as a function of time. A digest file is simply a precomputed time 83 | history: when it's built, all the events are scanned, and at every given time interval, a `Snapshot` 84 | is written. Each Snapshot is simply a map from stack traces at which memory was allocated, to the 85 | number of live bytes in memory at that instant which were allocated from that location in code. 86 | 87 | When you open the output, if a digest doesn't already exist, one is created for you: 88 | 89 | ``` 90 | import heapprof 91 | 92 | r = heapprof.read('/output/file') 93 | ``` 94 | 95 | This function returns a [`heapprof.Reader`](api/reader), which is your main interface to all 96 | the data. 97 | 98 | Once you have a reader and a digest, you can start [visualizing and interpreting the 99 | results](visualizing_results.md). 100 | -------------------------------------------------------------------------------- /docs/_sources/using_heapprof.md.txt: -------------------------------------------------------------------------------- 1 | # Using heapprof 2 | 3 | The basic flow of using heapprof is: 4 | 5 | 1. [Optional] Adjust parameters like the [sampling rate](advanced_heapprof.md#controlling-sampling). 6 | 1. Run your program with heapprof turned on. This generates .hpm (metadata) and .hpd (data) files. 7 | 1. Open the output files with `heapprof.read()` and create a .hpc (digest) file. 8 | 1. Analyze the resulting data with visualization tools, or do your own poking with the API. 9 | 10 | The reason heapprof needs so many files is that it's doing everything it can to minimize overhead 11 | while your program is running. By writing two files (one with things like stack traces, the other 12 | with the individual events that happen) instead of one, it can minimize how much time is spent on 13 | disk I/O; by deferring the work of combining individual events into easily-understood histories, it 14 | both saves CPU and I/O during runtime, and gives the analysis tools more flexibility via access to 15 | the raw data. 16 | 17 | This means that at least the first time you open a set of .hpx files, you'll need to spend some time 18 | processing it; this can range from seconds to minutes, depending on just how much data you've 19 | accumulated. 20 | 21 | > **Tip:** Partially-written .hpd and .hpm files are valid; this means you can start running 22 | > analysis while your program is still going, and it shouldn't interfere with either your program or 23 | > with further data collection! Partially-written digest files are also valid; this means that you 24 | > can ctrl-C digest building partway through, and you'll get a perfectly valid digest up to the 25 | > timestamp where you stopped it. You can always replace the digest by calling 26 | > `heapprof.Reader.makeDigest()` again, with new options. 27 | 28 | ## Installation and system requirements 29 | 30 | To install heapprof, simply run `pip install heapprof`. Some of heapprof's data visualization 31 | features require additional packages as well: 32 | 33 | * For time plots, you will also need [matplotlib](https://matplotlib.org/). In most cases, you can 34 | install this by simply running `pip install matplotlib`. 35 | * For flow graphs, you will also need [graphviz](https://www.graphviz.org/). graphviz is not a 36 | Python package; although there are Python packages with similar names that wrap it, all of them 37 | require that you install the underlying tool. See the graphviz site for how best to get it on 38 | your platform. 39 | * For flame graphs, you can either use [speedscope.app](https://www.speedscope.app/) as a web 40 | application (and just send your flame data there), or you can install and run it locally with 41 | `npm install -g speedscope`. (See its [GitHub site](https://github.com/jlfwong/speedscope) for 42 | more installation details if you aren't familiar with running local JavaScript code) 43 | 44 | > #### System Requirements 45 | > heapprof is designed to work with Python 3.7 or greater, using the CPython runtime. If you are using 46 | > an older version of Python or a less-common runtime, this package won't work, and you'll get strange 47 | > errors if you try. 48 | > 49 | > The current heapprof release includes binaries for Linux x86_64, OS X, and 64-bit Windows. On 50 | > other platforms, you can try compiling from a source distribution; if you have to make portability 51 | > changes (which you almost certainly will), those would be good to commit back to the main branch. 52 | 53 | 54 | ## Running your code 55 | 56 | heapprof can be invoked either programmatically or from the command line. Programmatically, you call 57 | 58 | ``` 59 | import heapprof 60 | 61 | heapprof.start('/some/path/to/output') 62 | ... do something ... 63 | heapprof.stop() 64 | ``` 65 | 66 | You don't need to do fancy exception-guarding; if the program exits before heapprof.stop() is 67 | called, it will clean up automatically. The path you provide to `heapprof.start` is known as the 68 | "filebase;" output will be written to filebase.hpd, filebase.hpm, and so on. 69 | 70 | If you want to wrap an entire program execution in heapprof, it's even simpler: 71 | 72 | ``` 73 | python -m heapprof -o /output/file -- myprogram.py ... 74 | ``` 75 | 76 | You can pass arguments to your Python code just like you usually would. 77 | 78 | ## Making a digest 79 | 80 | The files output by heapprof directly contain a sequence of events: timestamps and heap locations at 81 | which memory was allocated or freed. Nearly all ways to analyze this rely on instead having a 82 | picture of how memory looked as a function of time. A digest file is simply a precomputed time 83 | history: when it's built, all the events are scanned, and at every given time interval, a `Snapshot` 84 | is written. Each Snapshot is simply a map from stack traces at which memory was allocated, to the 85 | number of live bytes in memory at that instant which were allocated from that location in code. 86 | 87 | When you open the output, if a digest doesn't already exist, one is created for you: 88 | 89 | ``` 90 | import heapprof 91 | 92 | r = heapprof.read('/output/file') 93 | ``` 94 | 95 | This function returns a [`heapprof.Reader`](api/reader), which is your main interface to all 96 | the data. 97 | 98 | Once you have a reader and a digest, you can start [visualizing and interpreting the 99 | results](visualizing_results.md). 100 | -------------------------------------------------------------------------------- /docs/releasing.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | Release instructions — heapprof documentation 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 38 | 39 |
40 |
41 |
42 |
43 | 44 |
45 |

Release instructions

46 |

This document is for heapprof maintainers only. It assumes that the release is being cut from an OS 47 | X machine, since there’s no good way to remotely build those wheels yet. To cut a new release:

48 |
    49 |

  1. Check in a changelist which increments the version number in setup.py and updates 50 | RELEASE_NOTES.md. The version number should follow the semantic 51 | versioning 52 | scheme.

    2. 53 |

  2. Clean the local image by running git clean -xfd

    3. 54 |

  3. Build the source wheel by running python setup.py sdist

    4. 55 |

  4. Build the OS X wheel by running python setup.py bdist_wheel

    5. 56 |

  5. Go to https://circleci.com/gh/humu/heapprof/tree/master and click on the link for the 57 | most recent windows_64 job. For each of them, click on the “Artifacts” tab 58 | and download the corresponding wheel. Move it to your dist directory along with the wheels 59 | created in the previous step.

    6. 60 |

  6. Run twine upload dist/* to push the new repository version.

    7. 61 |

  7. Go to https://pypi.org/project/heapprof/ and verify that the new version is live.

    8. 62 |
63 |
64 | 65 | 66 |
67 |
68 |
69 | 94 |
95 |
96 | 97 | 98 | -------------------------------------------------------------------------------- /_heapprof/malloc_patch.cc: -------------------------------------------------------------------------------- 1 | #include "_heapprof/malloc_patch.h" 2 | #include 3 | #include "Python.h" 4 | #include "_heapprof/reentrant_scope.h" 5 | #include "pythread.h" 6 | 7 | // The logic of this file is heavily inspired by that of _tracemalloc.c, whose 8 | // brave authors have fought their way through the morass of why the simple 9 | // approach given in PEP445's example 3 doesn't actually work. There are two 10 | // subtle bits: 11 | // 12 | // (1) The OBJ and MEM malloc Handlers are called with the GIL held, but the RAW 13 | // Handler is called without it held, so we need our own locking in just that 14 | // case. (See ProfilerLock, below) (2) Various allocators can and do call 15 | // *other* allocators; e.g., the OBJ allocator may call the MEM allocator 16 | // depending on how many bytes it's asked for. When this happens, we need to 17 | // make sure that we only profile the topmost call, to avoid double-counting. 18 | // This is Handled with ReentrantScope. 19 | 20 | // Our global profiler state. 21 | static std::unique_ptr g_profiler; 22 | 23 | // The underlying allocators that we're going to wrap. This gets filled in with 24 | // meaningful content during AttachProfiler. 25 | static struct { 26 | PyMemAllocatorEx raw; 27 | PyMemAllocatorEx mem; 28 | PyMemAllocatorEx obj; 29 | } g_base_allocators; 30 | 31 | // The MEM and OBJ domain allocators are always called with the GIL held, so we 32 | // have serialization guaranteed. But the RAW allocator is called with the GIL 33 | // in an unknown state, and it would be Very Bad to try to grab it ourselves. So 34 | // to ensure that we serialize all access to g_profiler, we use a separate 35 | // mutex in that case. ProfilerLock is a simple scoped lock. 36 | class ProfilerLock { 37 | public: 38 | explicit ProfilerLock(void *ctx) : is_raw_(ctx == &g_base_allocators.raw) { 39 | if (is_raw_) { 40 | PyThread_acquire_lock(lock_, 1); 41 | } 42 | } 43 | 44 | ~ProfilerLock() { 45 | if (is_raw_) { 46 | PyThread_release_lock(lock_); 47 | } 48 | } 49 | 50 | static void ModuleInit() { lock_ = PyThread_allocate_lock(); } 51 | 52 | private: 53 | static PyThread_type_lock lock_; 54 | const bool is_raw_; 55 | }; 56 | PyThread_type_lock ProfilerLock::lock_ = nullptr; 57 | 58 | // The wrapped methods with which we will replace the standard malloc, etc. In 59 | // each case, ctx will be a pointer to the appropriate base allocator. 60 | 61 | static void *WrappedMalloc(void *ctx, size_t size) { 62 | ReentrantScope scope; 63 | PyMemAllocatorEx *alloc = reinterpret_cast(ctx); 64 | void *ptr = alloc->malloc(alloc->ctx, size); 65 | if (ptr && scope.is_top_level()) { 66 | ProfilerLock l(ctx); 67 | g_profiler->HandleMalloc(ptr, size); 68 | } 69 | return ptr; 70 | } 71 | 72 | static void *WrappedCalloc(void *ctx, size_t nelem, size_t elsize) { 73 | ReentrantScope scope; 74 | PyMemAllocatorEx *alloc = reinterpret_cast(ctx); 75 | void *ptr = alloc->calloc(alloc->ctx, nelem, elsize); 76 | if (ptr && scope.is_top_level()) { 77 | ProfilerLock l(ctx); 78 | g_profiler->HandleMalloc(ptr, nelem * elsize); 79 | } 80 | return ptr; 81 | } 82 | 83 | static void *WrappedRealloc(void *ctx, void *ptr, size_t new_size) { 84 | ReentrantScope scope; 85 | PyMemAllocatorEx *alloc = reinterpret_cast(ctx); 86 | void *ptr2 = alloc->realloc(alloc->ctx, ptr, new_size); 87 | if (ptr2 && scope.is_top_level()) { 88 | ProfilerLock l(ctx); 89 | g_profiler->HandleRealloc(ptr, ptr2, new_size); 90 | } 91 | return ptr2; 92 | } 93 | 94 | static void WrappedFree(void *ctx, void *ptr) { 95 | ReentrantScope scope; 96 | PyMemAllocatorEx *alloc = reinterpret_cast(ctx); 97 | alloc->free(alloc->ctx, ptr); 98 | if (scope.is_top_level()) { 99 | ProfilerLock l(ctx); 100 | g_profiler->HandleFree(ptr); 101 | } 102 | } 103 | 104 | /* Our API */ 105 | 106 | void AttachProfiler(AbstractProfiler *profiler) { 107 | g_profiler.reset(profiler); 108 | 109 | PyMemAllocatorEx alloc; 110 | alloc.malloc = WrappedMalloc; 111 | alloc.calloc = WrappedCalloc; 112 | alloc.realloc = WrappedRealloc; 113 | alloc.free = WrappedFree; 114 | 115 | // Grab the base allocators 116 | PyMem_GetAllocator(PYMEM_DOMAIN_RAW, &g_base_allocators.raw); 117 | PyMem_GetAllocator(PYMEM_DOMAIN_MEM, &g_base_allocators.mem); 118 | PyMem_GetAllocator(PYMEM_DOMAIN_OBJ, &g_base_allocators.obj); 119 | 120 | // And repoint allocation at our wrapped methods! 121 | alloc.ctx = &g_base_allocators.raw; 122 | PyMem_SetAllocator(PYMEM_DOMAIN_RAW, &alloc); 123 | 124 | alloc.ctx = &g_base_allocators.mem; 125 | PyMem_SetAllocator(PYMEM_DOMAIN_MEM, &alloc); 126 | 127 | alloc.ctx = &g_base_allocators.obj; 128 | PyMem_SetAllocator(PYMEM_DOMAIN_OBJ, &alloc); 129 | } 130 | 131 | void DetachProfiler() { 132 | if (IsProfilerAttached()) { 133 | PyMem_SetAllocator(PYMEM_DOMAIN_RAW, &g_base_allocators.raw); 134 | PyMem_SetAllocator(PYMEM_DOMAIN_MEM, &g_base_allocators.mem); 135 | PyMem_SetAllocator(PYMEM_DOMAIN_OBJ, &g_base_allocators.obj); 136 | 137 | g_profiler.reset(nullptr); 138 | } 139 | } 140 | 141 | bool IsProfilerAttached() { return (g_profiler != nullptr); } 142 | 143 | bool MallocPatchInit() { 144 | ProfilerLock::ModuleInit(); 145 | return true; 146 | } 147 | -------------------------------------------------------------------------------- /_heapprof/util.h: -------------------------------------------------------------------------------- 1 | #ifndef _HEAPPROF_UTIL_H__ 2 | #define _HEAPPROF_UTIL_H__ 3 | 4 | #include 5 | #include 6 | #include 7 | #include 8 | #include "Python.h" 9 | #include "_heapprof/port.h" 10 | 11 | ////////////////////////////////////////////////////////////////////////////////////////////////// 12 | // File I/O helpers 13 | 14 | // A ScopedFile is a scoped file descriptor that closes (and optionally 15 | // deletes!) on exit. 16 | class ScopedFile { 17 | public: 18 | // Create a ScopedFile for either reading or writing, depending on the 19 | // parameter. After constructing it, if this is false, the file failed to open 20 | // and the exception has been set. 21 | ScopedFile(const char *filebase, const char *extension, bool write); 22 | ~ScopedFile(); 23 | 24 | operator int() const { return fd_; } 25 | operator bool() const { return fd_ != -1; } 26 | const std::string &filename() const { return filename_; } 27 | 28 | // If delete-on-exit is set, this file will not just be closed on exit, it 29 | // will be deleted. You can use this to manage files that should only be 30 | // preserved if writing them works. 31 | void set_delete_on_exit(bool v) { delete_ = v; } 32 | 33 | private: 34 | const std::string filename_; 35 | const int fd_; 36 | bool delete_; 37 | }; 38 | 39 | ////////////////////////////////////////////////////////////////////////////////////////////////// 40 | // Data writing helpers 41 | // Because heap profiling is pretty performance-critical, we pick an output 42 | // format that is as easy as possible to write. The helpers below help assemble 43 | // the buffers that we'll dump to disk. NB that the methods whose name begins 44 | // with "Unsafe" are exactly what they say: they DO NOT DO ANY BOUNDS CHECKING 45 | // and expect their callers to have worked that out for them. 46 | 47 | // A signed type has 8*sizeof(T) - 1 bits in its max value. This would encode to 48 | // ceil((8*sizeof(T) - 1) / 7) bytes in varint coding, or (8*sizeof(T)-1+6)/7 49 | // bytes. 50 | #define MAX_SIGNED_VARINT_SIZE(signed_type) ((8 * sizeof(signed_type) + 5) / 7) 51 | #define MAX_UNSIGNED_VARINT_SIZE(us_type) ((8 * sizeof(us_type) + 6) / 7) 52 | 53 | // Write 'value' as a varint to the end of buffer, without doing any bounds 54 | // checking. (It's assumed that the caller has already guaranteed this!) Returns 55 | // a pointer immediately beyond that which was written. This code is based on 56 | // the method used in protobuf. 57 | inline uint8_t *UnsafeAppendVarint(uint8_t *buffer, int value) { 58 | assert(value >= 0); 59 | // Common case: small value, one byte. 60 | if (value < 0x80) { 61 | buffer[0] = static_cast(value); 62 | return buffer + 1; 63 | } 64 | // Pretty common case: two byte value. 65 | buffer[0] = static_cast(value | 0x80); 66 | value >>= 7; 67 | if (value < 0x80) { 68 | buffer[1] = static_cast(value); 69 | return buffer + 2; 70 | } 71 | // Generic case: read lots of bytes. 72 | buffer++; 73 | do { 74 | *buffer = static_cast(value | 0x80); 75 | value >>= 7; 76 | ++buffer; 77 | } while (PREDICT_FALSE(value >= 0x80)); 78 | *buffer++ = static_cast(value); 79 | return buffer; 80 | } 81 | 82 | // True if x is a UINT32-aligned pointer. 83 | #define UINT32_ALIGNED(x) ((reinterpret_cast(x) & 0x3) == 0) 84 | 85 | // Append value, in network byte order, to the buffer, and return a pointer 86 | // beyond where the write happened. No bounds checking is performed. 87 | inline uint8_t *UnsafeAppendFixed32(uint8_t *buffer, uint32_t value) { 88 | if (PREDICT_TRUE(UINT32_ALIGNED(buffer))) { 89 | *reinterpret_cast(buffer) = absl::ghtonl(value); 90 | } else { 91 | const uint32_t norm = absl::ghtonl(value); 92 | memcpy(buffer, &norm, sizeof(uint32_t)); 93 | } 94 | return buffer + sizeof(uint32_t); 95 | } 96 | 97 | // The read/write functions below are thread-hostile; they must only be called 98 | // from a single thread at a time. 99 | // All of these functions set the Python exception whenever they fail. 100 | 101 | // Write single numbers to a file. 102 | void WriteVarintToFile(int fd, uint64_t value); 103 | void WriteFixed32ToFile(int fd, uint32_t value); 104 | void WriteFixed64ToFile(int fd, uint64_t value); 105 | 106 | // Read single numbers from a file. If they return false, they also set the 107 | // Python exception. 108 | bool ReadVarintFromFile(int fd, uint64_t *value); 109 | bool ReadFixed32FromFile(int fd, uint32_t *value); 110 | bool ReadFixed64FromFile(int fd, uint64_t *value); 111 | 112 | // Write a string to a file as varint length + bytes. Fails (and sets the 113 | // exception) only if the passed argument isn't a valid string. 114 | bool WriteStringToFile(int fd, PyObject *value); 115 | // Returns nullptr and sets the exception on failure; returns a reference owned 116 | // by the caller on success. 117 | PyObject *ReadStringFromFile(int fd); 118 | 119 | // Set result = stop - start. result->tv_nsec is guaranteed to be in [0, 1B) 120 | inline void DeltaTime(const struct timespec &start, const struct timespec &stop, 121 | struct timespec *result) { 122 | result->tv_sec = stop.tv_sec - start.tv_sec; 123 | result->tv_nsec = stop.tv_nsec - start.tv_nsec; 124 | if (result->tv_nsec < 0) { 125 | result->tv_nsec += 1000000000L; 126 | result->tv_sec -= 1; 127 | } 128 | } 129 | 130 | #endif // _HEAPPROF_UTIL_H__ 131 | -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | import distutils.spawn 2 | import importlib 3 | import os 4 | import sys 5 | from distutils.command.build_ext import build_ext as _build_ext # type: ignore 6 | from typing import Optional 7 | 8 | from setuptools import Extension, find_packages, setup 9 | 10 | with open( 11 | os.path.join(os.path.abspath(os.path.dirname(__file__)), "README.md"), encoding="utf-8" 12 | ) as f: 13 | long_description = f.read() 14 | 15 | WINDOWS = sys.platform in ('win32', 'cygwin') 16 | CMAKE = 'CMake' if WINDOWS else 'cmake' 17 | EGGS_DIR = './.eggs' 18 | 19 | 20 | def findEgg(prefix: str) -> str: 21 | """Find an egg whose name begins with the given string. Raises if this egg does not exist or is 22 | not unique. 23 | """ 24 | found: Optional[str] = None 25 | for dirEntry in os.scandir(EGGS_DIR): 26 | if dirEntry.is_dir() and dirEntry.name.startswith(prefix): 27 | if found is not None: 28 | raise ValueError( 29 | f'Found multiple eggs beginning with "{prefix}": {found} and ' 30 | f'{dirEntry.name}' 31 | ) 32 | found = dirEntry.name 33 | if found is None: 34 | raise ValueError(f'Found no egg beginning with "{prefix}"') 35 | return os.path.join(EGGS_DIR, found) 36 | 37 | 38 | def findCMake() -> str: 39 | # If cmake is already installed and available, we're good. 40 | if distutils.spawn.find_executable(CMAKE): 41 | return CMAKE 42 | 43 | # If cmake was only present because it's in setup_requires, that means that the egg is 44 | # downloaded, but the binary is hidden inside! Get its path. 45 | sys.path.append(findEgg('cmake-')) 46 | module = importlib.import_module('cmake') 47 | 48 | # The path of the cmake binary! 49 | cmake = os.path.join(module.CMAKE_BIN_DIR, CMAKE) 50 | # Ensure it's executable; this isn't always true in the wheel. 51 | os.chmod(cmake, 0o777) 52 | return cmake 53 | 54 | 55 | # Our C++ library depends on ABSL. This insane monkey-patch is the simplest way I can figure out to 56 | # actually make that dependency happen. 57 | class BuildExtWithABSL(_build_ext): 58 | def run(self) -> None: 59 | if not os.path.exists("build/absl"): 60 | self.mkpath("build/absl") 61 | self.spawn(["git", "clone", "https://github.com/abseil/abseil-cpp.git", "build/absl"]) 62 | 63 | cmake = findCMake() 64 | 65 | pwd = os.getcwd() 66 | os.chdir("build/absl") 67 | self.spawn([cmake, "."]) 68 | self.spawn([cmake, "--build", ".", "--target", "base"]) 69 | os.chdir(pwd) 70 | 71 | super().run() 72 | 73 | 74 | cppmodule = Extension( 75 | "_heapprof", 76 | sources=[ 77 | "_heapprof/abstract_profiler.cc", 78 | "_heapprof/file_format.cc", 79 | "_heapprof/heapprof.cc", 80 | "_heapprof/malloc_patch.cc", 81 | "_heapprof/profiler.cc", 82 | "_heapprof/reentrant_scope.cc", 83 | "_heapprof/sampler.cc", 84 | "_heapprof/stats_gatherer.cc", 85 | "_heapprof/util.cc", 86 | ], 87 | depends=[ 88 | "_heapprof/abstract_profiler.h", 89 | "_heapprof/file_format.h", 90 | "_heapprof/malloc_patch.h", 91 | "_heapprof/port.h", 92 | "_heapprof/profiler.h", 93 | "_heapprof/reentrant_scope.h", 94 | "_heapprof/sampler.h", 95 | "_heapprof/scoped_object.h", 96 | "_heapprof/simple_hash.h", 97 | "_heapprof/stats_gatherer.h", 98 | "_heapprof/util.h", 99 | ], 100 | include_dirs=[os.getcwd(), "build/absl"], 101 | library_dirs=["build\\absl\\absl\\base\\Debug" if WINDOWS else "build/absl/absl/base"], 102 | libraries=["absl_base"], 103 | define_macros=[("PY_SSIZE_T_CLEAN", None)], 104 | extra_compile_args=["" if WINDOWS else "-std=c++11"], 105 | ) 106 | 107 | setup( 108 | # About this project 109 | name="heapprof", 110 | version="1.0.2", 111 | description="Logging heap profiler", 112 | long_description=long_description, 113 | long_description_content_type="text/markdown", 114 | keywords="development profiling memory", 115 | author="Yonatan Zunger", 116 | author_email="zunger@humu.com", 117 | license="MIT", 118 | classifiers=[ 119 | "Development Status :: 5 - Production/Stable", 120 | "License :: OSI Approved :: MIT License", 121 | "Intended Audience :: Developers", 122 | "Programming Language :: Python :: 3.7", 123 | "Programming Language :: Python :: Implementation :: CPython", 124 | ], 125 | project_urls={ 126 | "Documentation": "https://humu.github.io/heapprof", 127 | "Source": "https://github.com/humu/heapprof", 128 | "Tracker": "https://github.com/humu/heapprof/issues", 129 | }, 130 | # I suppose I should be glad that distutils has a certain amount of monkey-patching ability 131 | # built-in? 132 | cmdclass={"build_ext": BuildExtWithABSL}, 133 | # The actual contents 134 | # NB: This has API requirements that only run on 3.7 and above. It's probably possible to make a 135 | # version of this run on 3.4 or above if anyone really wants to. 136 | python_requires=">=3.7", 137 | ext_modules=[cppmodule], 138 | packages=find_packages(exclude=["tests", "tools", "docs", "docs_src"]), 139 | setup_requires=["cmake"], 140 | # Testing 141 | test_suite="nose.collector", 142 | tests_require=["nose"], 143 | ) 144 | -------------------------------------------------------------------------------- /docs_src/advanced_heapprof.md: -------------------------------------------------------------------------------- 1 | # Advanced heapprof 2 | 3 | As you use heapprof more, you'll want to understand the corner cases of memory allocation better. 4 | Here are some general advanced tips worth knowing: 5 | 6 | * The performance overhead of heapprof hasn't yet been measured. From a rough eyeball estimate, it 7 | seems to be significant during the initial import of modules (because those generate so many 8 | distinct stack traces) but fairly low (similar to cProfile) during code execution. This will 9 | need to be measured and performance presumably tuned. 10 | * The .hpx file format is optimized around minimizing overhead at runtime. The idea is that the 11 | profiler continuously writes to the two open file descriptors, and relies on the kernel's 12 | buffering in the file system implementation to minimize that impact; to use that buffering most 13 | effectively, it's important to minimize the size of data written, so that flushes are rare. This 14 | is why the wire encoding (cf file_format.*) tends towards things like varints, which use a bit 15 | more CPU but reduce bytes on the wire. This also helps keep the sizes of the generated files 16 | under control. 17 | * The profiler very deliberately uses C++ native types, not Python data types, for its internal 18 | operations. This has two advantages: pure C++ types are faster and more compact, (because of 19 | the simpler memory management model), and they eliminate the risk of weird recursions if the 20 | heap profiler were to try to call any of the Python allocators. NB, however, that this means 21 | that the heap profiler does not include its own memory allocation in its output! 22 | * More generally, the heap profiler only profiles calls to the Python memory allocators; C/C++ 23 | modules which allocate memory separately from that are not counted. This can lead to 24 | discrepancies between the output of heapprof and total system usage. 25 | * Furthermore, all real malloc() implementations generally allocate more bytes than requested under 26 | the hood (e.g., to guarantee memory alignment of the result; see e.g. 27 | [this function](https://github.com/gperftools/gperftools/blob/master/src/common.cc#L77) in 28 | tcmalloc). Unfortunately, there is no implementation-independent way to find out how many bytes 29 | were actually allocated, either from the underlying C/C++ allocators or from the higher-level 30 | Python allocators. This means that the heap measured by heapprof will be the "logical" heap 31 | size, which is strictly less than the heap size requested by the process from the kernel. 32 | However, it is that latter size which is monitored by external systems such as the out-of-memory 33 | (OOM) process killers in sandbox environments. 34 | 35 | ## Controlling Sampling 36 | 37 | The sampling rate controls the probability with which heap events are written. Too high a sampling 38 | rate, and the overhead of writing the data will stop your app, or the amount of data written will 39 | overload your disk; too low a sampling rate, and you won't get a clear picture of events. 40 | 41 | heapprof defines sampling rates as a `Dict[int, float]`, which maps the upper range of byte sizes to 42 | sampling probabilities. For example, the default sampling rate is `{128: 1e-4, 8192: 0.1}`. This 43 | means that allocations from 1-127 bytes get sampled at 1 in 10,000; allocations from 128-8191 bytes 44 | get sampled at 1 in 10; and allocations of 8192 bytes or more are always written, without sampling. 45 | These values have proven useful for some programs, but they probably aren't right for everything. 46 | 47 | As heapprof is in its early days, its tools for picking sampling rates are somewhat manual. The best 48 | way to do this is to run heapprof in "stats gathering" mode: you can do this either with 49 | 50 | `python -m heapprof --mode stats -- mycommand.py args ...` 51 | 52 | or programmatically by calling `heapprof.gatherStats()` instead of `heapprof.start(filename)`. In 53 | this mode, rather than generating .hpx files, it will build up a distribution of allocation sizes, 54 | and print it to stderr at profiling stop. The result will look something like this: 55 | 56 | ``` 57 | ------------------------------------------- 58 | HEAP USAGE SUMMARY 59 | Size Count Bytes 60 | 1 - 1 138553 6646 61 | 2 - 2 30462 60924 62 | 3 - 4 3766 14076 63 | 5 - 8 794441 5293169 64 | 9 - 16 1553664 23614125 65 | 17 - 32 17465509 509454895 66 | 33 - 64 27282873 1445865086 67 | 65 - 128 9489792 801787796 68 | 129 - 256 3506871 567321439 69 | 257 - 512 436393 143560935 70 | 513 - 1024 347668 257207137 71 | 1025 - 2048 410159 466041685 72 | 2049 - 4096 135294 348213256 73 | 4097 - 8192 194711 1026937305 74 | 8193 - 16384 27027 278236057 75 | 16385 - 32768 8910 183592671 76 | 32769 - 65536 4409 200267665 77 | 65537 - 131072 2699 228614855 78 | 131073 - 262144 1478 277347497 79 | 262145 - 524288 1093 306727390 80 | 524289 - 1048576 104 75269351 81 | 1048577 - 2097152 58 83804159 82 | 2097153 - 4194304 37 106320012 83 | 4194305 - 8388608 8 44335352 84 | 8388609 - 16777216 6 69695438 85 | 16777217 - 33554432 3 55391152 86 | ``` 87 | 88 | This tells us that there was a huge number of allocations of 256 bytes or less, which means that we 89 | can use a small sampling rate and still get good data, perhaps 1e-5. There seems to be a spike in 90 | memory usage in the 4096-8192 byte range, and generally the 256-8192 byte range has a few hundred 91 | thousand allocations, so we could sample it at a rate of 0.1 or 0.01. Beyond that, the counts drop 92 | off radically, and so sampling would be a bad idea. This suggests a sampling rate of 93 | `{256: 1e-5, 8192: 0.1}` for this program. You can set this by running 94 | 95 | `python -m heapprof -o --sample '{256:1e-5, 8192:0.1} -- mycommand.py args...` 96 | 97 | or 98 | 99 | `heapprof.start('filename', {256: 1e-5, 8192: 0.1})` 100 | 101 | Some tips for choosing a good sampling rate: 102 | 103 | * The most expensive part of logging isn't writing the events, it's writing the stack traces. 104 | Generally, very small allocations happen at a huge variety of stack traces (nearly every time 105 | you instantiate a Python variable!), but larger ones are far less common. This means that it's 106 | usually very important to keep the sampling rate low for very small byte sizes -- say, no more 107 | than 1e-4 below 64 bytes, and preferably up to 128 bytes -- but much less important to keep it 108 | low for larger byte sizes. 109 | * The only reason you want to keep the sampling rate low is for performance; if at any point you can 110 | get away with a bigger sampling rate, err on that side. 111 | -------------------------------------------------------------------------------- /docs/_sources/advanced_heapprof.md.txt: -------------------------------------------------------------------------------- 1 | # Advanced heapprof 2 | 3 | As you use heapprof more, you'll want to understand the corner cases of memory allocation better. 4 | Here are some general advanced tips worth knowing: 5 | 6 | * The performance overhead of heapprof hasn't yet been measured. From a rough eyeball estimate, it 7 | seems to be significant during the initial import of modules (because those generate so many 8 | distinct stack traces) but fairly low (similar to cProfile) during code execution. This will 9 | need to be measured and performance presumably tuned. 10 | * The .hpx file format is optimized around minimizing overhead at runtime. The idea is that the 11 | profiler continuously writes to the two open file descriptors, and relies on the kernel's 12 | buffering in the file system implementation to minimize that impact; to use that buffering most 13 | effectively, it's important to minimize the size of data written, so that flushes are rare. This 14 | is why the wire encoding (cf file_format.*) tends towards things like varints, which use a bit 15 | more CPU but reduce bytes on the wire. This also helps keep the sizes of the generated files 16 | under control. 17 | * The profiler very deliberately uses C++ native types, not Python data types, for its internal 18 | operations. This has two advantages: pure C++ types are faster and more compact, (because of 19 | the simpler memory management model), and they eliminate the risk of weird recursions if the 20 | heap profiler were to try to call any of the Python allocators. NB, however, that this means 21 | that the heap profiler does not include its own memory allocation in its output! 22 | * More generally, the heap profiler only profiles calls to the Python memory allocators; C/C++ 23 | modules which allocate memory separately from that are not counted. This can lead to 24 | discrepancies between the output of heapprof and total system usage. 25 | * Furthermore, all real malloc() implementations generally allocate more bytes than requested under 26 | the hood (e.g., to guarantee memory alignment of the result; see e.g. 27 | [this function](https://github.com/gperftools/gperftools/blob/master/src/common.cc#L77) in 28 | tcmalloc). Unfortunately, there is no implementation-independent way to find out how many bytes 29 | were actually allocated, either from the underlying C/C++ allocators or from the higher-level 30 | Python allocators. This means that the heap measured by heapprof will be the "logical" heap 31 | size, which is strictly less than the heap size requested by the process from the kernel. 32 | However, it is that latter size which is monitored by external systems such as the out-of-memory 33 | (OOM) process killers in sandbox environments. 34 | 35 | ## Controlling Sampling 36 | 37 | The sampling rate controls the probability with which heap events are written. Too high a sampling 38 | rate, and the overhead of writing the data will stop your app, or the amount of data written will 39 | overload your disk; too low a sampling rate, and you won't get a clear picture of events. 40 | 41 | heapprof defines sampling rates as a `Dict[int, float]`, which maps the upper range of byte sizes to 42 | sampling probabilities. For example, the default sampling rate is `{128: 1e-4, 8192: 0.1}`. This 43 | means that allocations from 1-127 bytes get sampled at 1 in 10,000; allocations from 128-8191 bytes 44 | get sampled at 1 in 10; and allocations of 8192 bytes or more are always written, without sampling. 45 | These values have proven useful for some programs, but they probably aren't right for everything. 46 | 47 | As heapprof is in its early days, its tools for picking sampling rates are somewhat manual. The best 48 | way to do this is to run heapprof in "stats gathering" mode: you can do this either with 49 | 50 | `python -m heapprof --mode stats -- mycommand.py args ...` 51 | 52 | or programmatically by calling `heapprof.gatherStats()` instead of `heapprof.start(filename)`. In 53 | this mode, rather than generating .hpx files, it will build up a distribution of allocation sizes, 54 | and print it to stderr at profiling stop. The result will look something like this: 55 | 56 | ``` 57 | ------------------------------------------- 58 | HEAP USAGE SUMMARY 59 | Size Count Bytes 60 | 1 - 1 138553 6646 61 | 2 - 2 30462 60924 62 | 3 - 4 3766 14076 63 | 5 - 8 794441 5293169 64 | 9 - 16 1553664 23614125 65 | 17 - 32 17465509 509454895 66 | 33 - 64 27282873 1445865086 67 | 65 - 128 9489792 801787796 68 | 129 - 256 3506871 567321439 69 | 257 - 512 436393 143560935 70 | 513 - 1024 347668 257207137 71 | 1025 - 2048 410159 466041685 72 | 2049 - 4096 135294 348213256 73 | 4097 - 8192 194711 1026937305 74 | 8193 - 16384 27027 278236057 75 | 16385 - 32768 8910 183592671 76 | 32769 - 65536 4409 200267665 77 | 65537 - 131072 2699 228614855 78 | 131073 - 262144 1478 277347497 79 | 262145 - 524288 1093 306727390 80 | 524289 - 1048576 104 75269351 81 | 1048577 - 2097152 58 83804159 82 | 2097153 - 4194304 37 106320012 83 | 4194305 - 8388608 8 44335352 84 | 8388609 - 16777216 6 69695438 85 | 16777217 - 33554432 3 55391152 86 | ``` 87 | 88 | This tells us that there was a huge number of allocations of 256 bytes or less, which means that we 89 | can use a small sampling rate and still get good data, perhaps 1e-5. There seems to be a spike in 90 | memory usage in the 4096-8192 byte range, and generally the 256-8192 byte range has a few hundred 91 | thousand allocations, so we could sample it at a rate of 0.1 or 0.01. Beyond that, the counts drop 92 | off radically, and so sampling would be a bad idea. This suggests a sampling rate of 93 | `{256: 1e-5, 8192: 0.1}` for this program. You can set this by running 94 | 95 | `python -m heapprof -o --sample '{256:1e-5, 8192:0.1} -- mycommand.py args...` 96 | 97 | or 98 | 99 | `heapprof.start('filename', {256: 1e-5, 8192: 0.1})` 100 | 101 | Some tips for choosing a good sampling rate: 102 | 103 | * The most expensive part of logging isn't writing the events, it's writing the stack traces. 104 | Generally, very small allocations happen at a huge variety of stack traces (nearly every time 105 | you instantiate a Python variable!), but larger ones are far less common. This means that it's 106 | usually very important to keep the sampling rate low for very small byte sizes -- say, no more 107 | than 1e-4 below 64 bytes, and preferably up to 128 bytes -- but much less important to keep it 108 | low for larger byte sizes. 109 | * The only reason you want to keep the sampling rate low is for performance; if at any point you can 110 | get away with a bigger sampling rate, err on that side. 111 | -------------------------------------------------------------------------------- /heapprof/_si_prefix.py: -------------------------------------------------------------------------------- 1 | import math 2 | from typing import Tuple, Union 3 | 4 | 5 | """Tools for rendering numbers using SI prefixes, e.g. 1,023,404 as 1.02M. 6 | 7 | This should really be in its own library, but we'll include it as part of heapprof for now. 8 | 9 | This is subtle because there are a few different things, all referred to as "SI Prefixes," and it's 10 | really important to choose the right one. 11 | 12 | "Decimal" SI prefixes use 1k=1,000; "Binary" SI prefixes use 1k=1,024. 13 | 14 | - Any physical quantity: Always use decimal. 15 | - A number of bytes, in the context of networks (e.g. bps): Always use decimal 16 | - A number of bytes, in the context of a hard disk or SSD capacity: See below 17 | - A number of bytes, in any other context (e.g. storage): Always use binary 18 | 19 | Binary SI prefixes can be rendered in two ways: either identically to decimal SI prefixes, or in 20 | IEC60027 format, which looks like "Ki", "Mi", etc. The IEC format thus eliminates any ambiguity. 21 | 22 | Yes, this can be dangerous. This has led to major outages, especially when computing how much 23 | network bandwidth you need, where dividing a number of (binary, memory) MB by the time you need does 24 | *not* give you the number of Mbps you need. 25 | 26 | Despite that, IEC prefixes are not commonly used. This is largely because they look kind of silly, 27 | and their full names are even sillier: "kibibytes," "mebibytes," etc. Even though this standard has 28 | been around for nearly two decades, it's widely ignored, and people generally rely on context to 29 | tell them what's what. 30 | 31 | Hard disk and SSD capacities (the numbers you often see on the box) use *neither* SI nor IEC. They 32 | sometimes use a weird sort of hybrid; e.g., "1.44M" floppies actually hold (1440) * (1024) bytes. 33 | But most of the time, they use complete nonsense, with the numbers on the box being generated by 34 | marketing departments, not engineering departments. The technical term for these values is 35 | "barefaced lies." The only way to get the actual capacity of a disk is to format and mount it, and 36 | then check the number of bytes available, which should then be expressed using binary prefixes, 37 | like any other storage value. 38 | """ 39 | 40 | 41 | def siPrefixString( 42 | value: Union[int, float], 43 | threshold: float = 1.1, 44 | precision: int = 1, 45 | binary: bool = False, 46 | iecFormat: bool = False, 47 | failOnOverflow: bool = False, 48 | ) -> str: 49 | """Render a string with SI prefixes. 50 | 51 | Args: 52 | threshold: Switch to the "next prefix up" once you hit this many times that prefix. For 53 | example, if threshold=1.1 and binary=False, then 1,050 will be rendered as 1050, but 54 | 1,100 will be rendered as 1.1k. 55 | precision: The number of decimal places to preserve right of the point. 56 | binary: If true, treat these as binary (base=1024); if false, as decimal (base=1000). 57 | iecFormat: If binary is true, then generate IEC prefixes; otherwise, use SI. Ignored if 58 | binary is false. 59 | failOnOverflow: If true and the value exceeds the range of defined SI prefixes, raise an 60 | OverflowError; otherwise, return a non-prefix-based rendering like '1.1 << 120' (for 61 | binary) or '5.7e66' (for decimal). Which of these is appropriate really depends on how 62 | you're using the string. 63 | """ 64 | exponent, coefficient = _siExponent(value, 1024 if binary else 1000, threshold) 65 | try: 66 | return '%.*f' % (precision, coefficient) + _prefix(exponent, binary and iecFormat) 67 | except IndexError: 68 | # Most uses are unlikely to run out of SI prefixes anytime soon, but you never know. There 69 | # are official proposals about how SI is likely to be extended if and when this is required, 70 | # but they're overall pretty boring. I personally suggest we continue atto, zepto, yocto 71 | # with arpo, hico, and goucho, with positive prefixes arpa, hica, and 72 | # goucha. 73 | if failOnOverflow: 74 | raise OverflowError(f'The value {value} exceeds all defined SI prefixes') 75 | elif binary: 76 | return '%.*f << %d' % (precision, coefficient, 10 * exponent) 77 | else: 78 | return '%.*fe%d' % (precision, coefficient, exponent) 79 | 80 | 81 | def bytesString(value: Union[int, float], iecFormat: bool = False) -> str: 82 | """A helper for the common task of rendering 'value' as a (binary) prefix plus B for bytes. It 83 | may seem counterintuitive, given the comment above, that the default for iecFormat is false, 84 | but despite all the various arguments for its use, IEC prefices remain very rarely-used in 85 | practice, even by professionals and even in the fields where it matters. I suspect that the 86 | underlying reason for this is that the names ("kibi," "mebi," etc) sound just plain silly, 87 | causing nobody to want to use them, causing the places where they *are* used to seem overly 88 | stuffy, like ISO standards docs or people trying to correct others for ending sentences with 89 | prepositions. 90 | 91 | Doing that is completely fine, by the way. Like many "rules" of English grammar, that one was 92 | made up out of whole cloth in the early 20th century by people who didn't actually understand 93 | linguistics very well, but who really liked defining a "proper" English so that they could 94 | police class boundaries and look down on people who "spoke wrong." This is precisely the sort 95 | of prescriptivist nonsense up with which we should never put. 96 | """ 97 | return f'{siPrefixString(value, binary=True, iecFormat=iecFormat, failOnOverflow=True)}B' 98 | 99 | 100 | def _siExponent(value: Union[int, float], base: int, threshold: float) -> Tuple[int, float]: 101 | """Pull out the SI exponent of a number. This will return a pair of numbers (e, p) such that 102 | value = p * base^e, and p >= threshold. 103 | """ 104 | # TODO: This could be made more efficient. 105 | if value == 0: 106 | return (0, 0) 107 | 108 | comparison = value / threshold 109 | exponent = math.floor(math.log(comparison) / math.log(base)) 110 | coefficient = value * math.pow(base, -exponent) 111 | return (exponent, coefficient) 112 | 113 | 114 | NEGATIVE_PREFIXES = 'mμnpfazy' # milli, micro, nano, pico, femto, atto, zepto, yocto 115 | POSITIVE_SI_PREFIXES = 'kMGTPEZY' # kilo, mega, giga, tera, peta, eta, zetta, yotta 116 | POSITIVE_IEC_PREFIXES = 'KMGTPEZY' 117 | 118 | 119 | def _prefix(power: int, iec: bool) -> str: 120 | if power == 0: 121 | return '' 122 | elif power > 0: 123 | if iec: 124 | return POSITIVE_IEC_PREFIXES[power - 1] + 'i' 125 | else: 126 | return POSITIVE_SI_PREFIXES[power - 1] 127 | elif iec: 128 | return NEGATIVE_PREFIXES[-power - 1] + 'i' 129 | else: 130 | return NEGATIVE_PREFIXES[-power - 1] 131 | -------------------------------------------------------------------------------- /Gemfile.lock: -------------------------------------------------------------------------------- 1 | GEM 2 | remote: https://rubygems.org/ 3 | specs: 4 | activesupport (4.2.11.1) 5 | i18n (~> 0.7) 6 | minitest (~> 5.1) 7 | thread_safe (~> 0.3, >= 0.3.4) 8 | tzinfo (~> 1.1) 9 | addressable (2.6.0) 10 | public_suffix (>= 2.0.2, < 4.0) 11 | coffee-script (2.4.1) 12 | coffee-script-source 13 | execjs 14 | coffee-script-source (1.11.1) 15 | colorator (1.1.0) 16 | commonmarker (0.17.13) 17 | ruby-enum (~> 0.5) 18 | concurrent-ruby (1.1.5) 19 | dnsruby (1.61.2) 20 | addressable (~> 2.5) 21 | em-websocket (0.5.1) 22 | eventmachine (>= 0.12.9) 23 | http_parser.rb (~> 0.6.0) 24 | ethon (0.12.0) 25 | ffi (>= 1.3.0) 26 | eventmachine (1.2.7) 27 | execjs (2.7.0) 28 | faraday (0.15.4) 29 | multipart-post (>= 1.2, < 3) 30 | ffi (1.11.1) 31 | forwardable-extended (2.6.0) 32 | gemoji (3.0.1) 33 | github-pages (198) 34 | activesupport (= 4.2.11.1) 35 | github-pages-health-check (= 1.16.1) 36 | jekyll (= 3.8.5) 37 | jekyll-avatar (= 0.6.0) 38 | jekyll-coffeescript (= 1.1.1) 39 | jekyll-commonmark-ghpages (= 0.1.5) 40 | jekyll-default-layout (= 0.1.4) 41 | jekyll-feed (= 0.11.0) 42 | jekyll-gist (= 1.5.0) 43 | jekyll-github-metadata (= 2.12.1) 44 | jekyll-mentions (= 1.4.1) 45 | jekyll-optional-front-matter (= 0.3.0) 46 | jekyll-paginate (= 1.1.0) 47 | jekyll-readme-index (= 0.2.0) 48 | jekyll-redirect-from (= 0.14.0) 49 | jekyll-relative-links (= 0.6.0) 50 | jekyll-remote-theme (= 0.3.1) 51 | jekyll-sass-converter (= 1.5.2) 52 | jekyll-seo-tag (= 2.5.0) 53 | jekyll-sitemap (= 1.2.0) 54 | jekyll-swiss (= 0.4.0) 55 | jekyll-theme-architect (= 0.1.1) 56 | jekyll-theme-cayman (= 0.1.1) 57 | jekyll-theme-dinky (= 0.1.1) 58 | jekyll-theme-hacker (= 0.1.1) 59 | jekyll-theme-leap-day (= 0.1.1) 60 | jekyll-theme-merlot (= 0.1.1) 61 | jekyll-theme-midnight (= 0.1.1) 62 | jekyll-theme-minimal (= 0.1.1) 63 | jekyll-theme-modernist (= 0.1.1) 64 | jekyll-theme-primer (= 0.5.3) 65 | jekyll-theme-slate (= 0.1.1) 66 | jekyll-theme-tactile (= 0.1.1) 67 | jekyll-theme-time-machine (= 0.1.1) 68 | jekyll-titles-from-headings (= 0.5.1) 69 | jemoji (= 0.10.2) 70 | kramdown (= 1.17.0) 71 | liquid (= 4.0.0) 72 | listen (= 3.1.5) 73 | mercenary (~> 0.3) 74 | minima (= 2.5.0) 75 | nokogiri (>= 1.8.5, < 2.0) 76 | rouge (= 2.2.1) 77 | terminal-table (~> 1.4) 78 | github-pages-health-check (1.16.1) 79 | addressable (~> 2.3) 80 | dnsruby (~> 1.60) 81 | octokit (~> 4.0) 82 | public_suffix (~> 3.0) 83 | typhoeus (~> 1.3) 84 | html-pipeline (2.11.1) 85 | activesupport (>= 2) 86 | nokogiri (>= 1.4) 87 | http_parser.rb (0.6.0) 88 | i18n (0.9.5) 89 | concurrent-ruby (~> 1.0) 90 | jekyll (3.8.5) 91 | addressable (~> 2.4) 92 | colorator (~> 1.0) 93 | em-websocket (~> 0.5) 94 | i18n (~> 0.7) 95 | jekyll-sass-converter (~> 1.0) 96 | jekyll-watch (~> 2.0) 97 | kramdown (~> 1.14) 98 | liquid (~> 4.0) 99 | mercenary (~> 0.3.3) 100 | pathutil (~> 0.9) 101 | rouge (>= 1.7, < 4) 102 | safe_yaml (~> 1.0) 103 | jekyll-avatar (0.6.0) 104 | jekyll (~> 3.0) 105 | jekyll-coffeescript (1.1.1) 106 | coffee-script (~> 2.2) 107 | coffee-script-source (~> 1.11.1) 108 | jekyll-commonmark (1.3.1) 109 | commonmarker (~> 0.14) 110 | jekyll (>= 3.7, < 5.0) 111 | jekyll-commonmark-ghpages (0.1.5) 112 | commonmarker (~> 0.17.6) 113 | jekyll-commonmark (~> 1) 114 | rouge (~> 2) 115 | jekyll-default-layout (0.1.4) 116 | jekyll (~> 3.0) 117 | jekyll-feed (0.11.0) 118 | jekyll (~> 3.3) 119 | jekyll-gist (1.5.0) 120 | octokit (~> 4.2) 121 | jekyll-github-metadata (2.12.1) 122 | jekyll (~> 3.4) 123 | octokit (~> 4.0, != 4.4.0) 124 | jekyll-mentions (1.4.1) 125 | html-pipeline (~> 2.3) 126 | jekyll (~> 3.0) 127 | jekyll-optional-front-matter (0.3.0) 128 | jekyll (~> 3.0) 129 | jekyll-paginate (1.1.0) 130 | jekyll-readme-index (0.2.0) 131 | jekyll (~> 3.0) 132 | jekyll-redirect-from (0.14.0) 133 | jekyll (~> 3.3) 134 | jekyll-relative-links (0.6.0) 135 | jekyll (~> 3.3) 136 | jekyll-remote-theme (0.3.1) 137 | jekyll (~> 3.5) 138 | rubyzip (>= 1.2.1, < 3.0) 139 | jekyll-sass-converter (1.5.2) 140 | sass (~> 3.4) 141 | jekyll-seo-tag (2.5.0) 142 | jekyll (~> 3.3) 143 | jekyll-sitemap (1.2.0) 144 | jekyll (~> 3.3) 145 | jekyll-swiss (0.4.0) 146 | jekyll-theme-architect (0.1.1) 147 | jekyll (~> 3.5) 148 | jekyll-seo-tag (~> 2.0) 149 | jekyll-theme-cayman (0.1.1) 150 | jekyll (~> 3.5) 151 | jekyll-seo-tag (~> 2.0) 152 | jekyll-theme-dinky (0.1.1) 153 | jekyll (~> 3.5) 154 | jekyll-seo-tag (~> 2.0) 155 | jekyll-theme-hacker (0.1.1) 156 | jekyll (~> 3.5) 157 | jekyll-seo-tag (~> 2.0) 158 | jekyll-theme-leap-day (0.1.1) 159 | jekyll (~> 3.5) 160 | jekyll-seo-tag (~> 2.0) 161 | jekyll-theme-merlot (0.1.1) 162 | jekyll (~> 3.5) 163 | jekyll-seo-tag (~> 2.0) 164 | jekyll-theme-midnight (0.1.1) 165 | jekyll (~> 3.5) 166 | jekyll-seo-tag (~> 2.0) 167 | jekyll-theme-minimal (0.1.1) 168 | jekyll (~> 3.5) 169 | jekyll-seo-tag (~> 2.0) 170 | jekyll-theme-modernist (0.1.1) 171 | jekyll (~> 3.5) 172 | jekyll-seo-tag (~> 2.0) 173 | jekyll-theme-primer (0.5.3) 174 | jekyll (~> 3.5) 175 | jekyll-github-metadata (~> 2.9) 176 | jekyll-seo-tag (~> 2.0) 177 | jekyll-theme-slate (0.1.1) 178 | jekyll (~> 3.5) 179 | jekyll-seo-tag (~> 2.0) 180 | jekyll-theme-tactile (0.1.1) 181 | jekyll (~> 3.5) 182 | jekyll-seo-tag (~> 2.0) 183 | jekyll-theme-time-machine (0.1.1) 184 | jekyll (~> 3.5) 185 | jekyll-seo-tag (~> 2.0) 186 | jekyll-titles-from-headings (0.5.1) 187 | jekyll (~> 3.3) 188 | jekyll-watch (2.2.1) 189 | listen (~> 3.0) 190 | jemoji (0.10.2) 191 | gemoji (~> 3.0) 192 | html-pipeline (~> 2.2) 193 | jekyll (~> 3.0) 194 | kramdown (1.17.0) 195 | liquid (4.0.0) 196 | listen (3.1.5) 197 | rb-fsevent (~> 0.9, >= 0.9.4) 198 | rb-inotify (~> 0.9, >= 0.9.7) 199 | ruby_dep (~> 1.2) 200 | mercenary (0.3.6) 201 | mini_portile2 (2.4.0) 202 | minima (2.5.0) 203 | jekyll (~> 3.5) 204 | jekyll-feed (~> 0.9) 205 | jekyll-seo-tag (~> 2.1) 206 | minitest (5.11.3) 207 | multipart-post (2.1.1) 208 | nokogiri (1.10.8) 209 | mini_portile2 (~> 2.4.0) 210 | octokit (4.14.0) 211 | sawyer (~> 0.8.0, >= 0.5.3) 212 | pathutil (0.16.2) 213 | forwardable-extended (~> 2.6) 214 | public_suffix (3.1.1) 215 | rb-fsevent (0.10.3) 216 | rb-inotify (0.10.0) 217 | ffi (~> 1.0) 218 | rouge (2.2.1) 219 | ruby-enum (0.7.2) 220 | i18n 221 | ruby_dep (1.5.0) 222 | rubyzip (2.2.0) 223 | safe_yaml (1.0.5) 224 | sass (3.7.4) 225 | sass-listen (~> 4.0.0) 226 | sass-listen (4.0.0) 227 | rb-fsevent (~> 0.9, >= 0.9.4) 228 | rb-inotify (~> 0.9, >= 0.9.7) 229 | sawyer (0.8.2) 230 | addressable (>= 2.3.5) 231 | faraday (> 0.8, < 2.0) 232 | terminal-table (1.8.0) 233 | unicode-display_width (~> 1.1, >= 1.1.1) 234 | thread_safe (0.3.6) 235 | typhoeus (1.3.1) 236 | ethon (>= 0.9.0) 237 | tzinfo (1.2.5) 238 | thread_safe (~> 0.1) 239 | unicode-display_width (1.6.0) 240 | 241 | PLATFORMS 242 | ruby 243 | 244 | DEPENDENCIES 245 | github-pages 246 | 247 | BUNDLED WITH 248 | 1.17.2 249 | -------------------------------------------------------------------------------- /docs/contributing.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | Contributing to heapprof — heapprof documentation 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 46 | 47 |
48 |
49 |
50 |
51 | 52 |

53 |
54 |

Contributing to heapprof

55 |

heapprof is an open source project distributed under the MIT License. Discussions, 56 | questions, and feature requests should be done via the 57 | GitHub issues page.

58 |

Pull requests for bugfixes and features are welcome!

59 |
    60 |

  • Generally, you should discuss features or API changes on the tracking issue first, to make sure 61 | everyone is aligned on direction.

    • 62 |

  • Lint and style:

    63 |
      64 |

    • Python code should follow PEP8+Black 65 | formatting, and should pass mypy with strict type checking.

      • 66 |

    • C/C++ code should follow the 67 | Google C++ Style Guide.

      • 68 |

    • You can check a client against the same lint checks run by the continuous integration test by 69 | running python tools/lint.py; if you add --fix, it will try to fix any errors it can 70 | in-place.

      • 71 |
    72 |
    • 73 |

  • Unittests are highly desired and should be invocable by setup.py test.

    • 74 |

  • Documentation: Any changes should be reflected in the documentation! Remember:

    75 |
      76 |

    • All Python modules should have clear docstrings for all public methods and variables. Classes 77 | should be organized with the interface first, and implementation details later.

      • 78 |

    • If you’re adding any new top-level modules, add corresponding .rst files in docs/api and link 79 | to them from docs/api/index.rst.

      • 80 |

    • Any other changes should be reflected in the documentation in docs.

      • 81 |

    • After changing the documentation, you can rebuild the HTML by running python tools/docs.py.

      • 82 |

    • If you want to see how the documentation looks before pushing, install 83 | Jekyll locally, then from the root directory of 84 | this repository, run bundle exec jekyll serve. This will serve the same images that will be 85 | served in production on localhost:4000.

      • 86 |
    87 |
    • 88 |
89 |
90 |

Code of Conduct

91 |

Most importantly, heapprof is released with a Contributor Code of Conduct. By 92 | participating in this product, you agree to abide by its terms. This code also governs behavior on 93 | the mailing list. We take this very seriously, and will enforce it gleefully.

94 |
95 |
96 |

Desiderata

97 |

Some known future features that we’ll probably want:

98 |
    99 |

  • Provide additional file formats of output to work with other kinds of visualization.

    • 100 |

  • Measure and tune system performance.

    • 101 |

  • Make the process of picking sampling rates less manual.

    • 102 |

  • Add support for more platforms. (Win32? Android? iOS?)

    • 103 |

  • Implement a CircleCI orb for Windows with Python + CMake to speed up builds.

    • 104 |
105 |
106 |
107 | 108 | 109 |
110 |
111 |
112 | 148 |
149 |
150 | 151 | 152 | -------------------------------------------------------------------------------- /docs/api/heapprof.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | The top-level heapprof package — heapprof documentation 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 47 | 48 |
49 |
50 |
51 |
52 | 53 |
54 |

The top-level heapprof package

55 |
56 |
57 | heapprof.start(filebase: str, samplingRate: Optional[Dict[int, float]] = None) → None
58 |

Start heapprof in profiling (normal) mode.

59 |
60 |
Parameters
61 |
    62 |

  • filebase – The outputs will be written to filebase.{hpm, hpd}, a pair of local files which 63 | can later be read using the HeapProfile class. NB that these must be local files for 64 | performance reasons.

    • 65 |

  • samplingRate – A dict from byte size to sampling probability. Each byte size is interpreted 66 | as the upper bound of the range, and the sampling probability for byte sizes larger than 67 | the largest range given is always 1; thus the default value means to profile allocations 68 | of 1-127 bytes at 1 in 10,000, to profile allocations of 128-8,191 bytes at 1 in 10, and 69 | to profile all allocations of 8,192 bytes or more.

    • 70 |
71 |
72 |
Raises
73 |
    74 |

  • TypeError – If samplingRate is not a mapping of the appropriate type.

    • 75 |

  • ValueError – If samplingRate contains repeated entries.

    • 76 |

  • RuntimeError – If the profiler is already running.

    • 77 |
78 |
79 |
80 |
81 | 82 |
83 |
84 | heapprof.gatherStats() → None
85 |

Start heapprof in stats gathering mode.

86 |

When the profiler is stopped, this will print out statistics on the size distribution of memory 87 | allocations. This can be useful for choosing sampling rates for profiling.

88 |
89 | 90 |
91 |
92 | heapprof.stop() → None
93 |

Stop the heap profiler.

94 |

NB that if the program exits, this will be implicitly called.

95 |
96 | 97 |
98 |
99 | heapprof.isProfiling() → bool
100 |

Test if the heap profiler is currently running.

101 |
102 | 103 |
104 |
105 | heapprof.read(filebase: str, timeInterval: float = 60, precision: float = 0.01) → heapprof.reader.Reader
106 |

Open a reader, and create a digest for it if needed.

107 |
108 |
Parameters
109 |

filebase – The name of the file to open; the same as the argument passed to start().

110 |
111 |
112 |
113 |
Args which apply only if you’re creating the digest (i.e., opening it for the first time):
114 |
timeInterval: The time interval between successive snapshots to store in the digest,

in seconds.

115 |
116 |
precision: At each snapshot, stack traces totalling up to this fraction of total

memory used at that frame may be dropped into the “other stack trace” bucket. 117 | This can greatly shrink the size of the digest at no real cost in usefulness. 118 | Must be in [0, 1); a value of zero means nothing is dropped.

119 |
120 |
121 |
122 |
123 |
124 | 125 |
126 | 127 | 128 |
129 |
130 |
131 | 160 |
161 |
162 | 163 | 164 | -------------------------------------------------------------------------------- /docs/quickstart.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | heapprof quickstart — heapprof documentation 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 46 | 47 |
48 |
49 |
50 |
51 | 52 |
53 |

heapprof quickstart

54 |

You can install heapprof by running

55 |

pip install heapprof

56 |

The simplest way to get a heap profile for your program is to run

57 |

python -m heapprof -o <filename> -- mycommand.py args...

58 |

This will run your command and write the output to the files <filename>.hpm and <filename>.hpd. 59 | (Collectively, the “hpx files”) Alternatively, you can control heapprof programmatically:

60 | 
import heapprof
 61 | 
 62 | heapprof.start('filename')
 63 | ... do something ...
 64 | heapprof.stop()
 65 |
66 |
67 |

You can analyze hpx files with the analysis and visualization tools built in to heapprof, or use its 68 | APIs to dive deeper into your program’s memory usage yourself. For example, you might do this with 69 | the built-in visualization tools:

70 | 
import heapprof
 71 | r = heapprof.read('filename')
 72 | 
 73 | # Generate a plot of total memory usage over time. This command requires that you first pip install
 74 | # matplotlib.
 75 | r.timePlot().pyplot()
 76 | 
 77 | # Looks like something interested happened 300 seconds in. You can look at the output as a Flame
 78 | # graph and view it in a tool like speedscope.app.
 79 | r.writeFlameGraph('flame-300.txt', 300)
 80 | 
 81 | # Or you can look at the output as a Flow graph and view it using graphviz. (See graphviz.org)
 82 | r.flowGraphAt(300).asDotFile('300.dot')
 83 | 
 84 | # Maybe you'd like to compare three times and see how memory changed. This produces a multi-time
 85 | # Flow graph.
 86 | r.compare('compare.dot', r.flowGraphAt(240), r.flowGraphAt(300), r.flowGraphAt(500))
 87 | 
 88 | # Or maybe you found some lines of code where memory use seemed to shift interestingly.
 89 | r.timePlot({
 90 |   'read_data': '/home/wombat/myproject/io.py:361',
 91 |   'write_buffer': '/home/wombat/myproject/io.py:582',
 92 | })
 93 |
94 |
95 |

One thing you’ll notice in the above is that visualization tools require you to install extra 96 | packages – that’s a way to keep heapprof’s dependencies down.

97 |

To learn more, continue on to Using heapprof

98 |
99 | 100 | 101 |
102 |
103 |
104 | 131 |
132 |
133 | 134 | 135 | --------------------------------------------------------------------------------