Introduction

pymemtrace provides tools for tracking and understanding Python memory usage at different levels, at different granularities and with different runtime costs.

Full documentation: https://pymemtrace.readthedocs.io

pymemtrace Tools

The tools provided by pymemtrace are:

process_tree

A command line tool that shows the total memory usage of a process and its child processes at regular time intervals. It can log this data to a JSON file for later analysis. See some process_tree examples

process

A very lightweight way of logging the total memory usage of a single process at regular time intervals. It can plot memory over time with plotting programs such as gnuplot. See some process examples

cPyMemTrace

This module, written in C, provides real time logging of Python and C actions:

• pymemtrace.cPyMemTrace.Profile is a memory tracer written in C that can report total memory usage for every function call/return for both C and Python sections. This is more suitable for logging C code, for example Python’s C extensions.

• pymemtrace.cPyMemTrace.Trace is a memory tracer written in C that can report total memory usage for every function call/return/line for Python sections. This is more suitable for logging pure Python code.

• pymemtrace.cPyMemTrace.ReferenceTracing can report every object allocation and de-allocation with Reference Tracing. This is quite invasive but the API allows this to filter out most of the noise or target specific types of interest. (Python 3.13+ only).

See some cPyMemTrace examples and a technical note on cPyMemTrace.

DTrace

There are a number of D scripts that can trace the low level malloc() and free() system calls and report how much memory was allocated and by whom. See some DTrace examples and a technical note on DTrace.

trace_malloc

A convenience wrapper around the Python standard library tracemalloc module. This can report Python memory usage by module and line compensating for the cost of tracemalloc. This can take memory snapshots before and after code blocks and show the change on memory caused by that code. See some trace_malloc examples

debug_malloc_stats

Awrapper around the sys._debugmallocstats function that can take snapshots of memory before and after code execution and report the significant differences of the Python small object allocator. See some debug_malloc_stats examples

Tool Characteristics

Each tool can be characterised by:

• Memory Granularity: In how much detail is a memory change is observed. An example of coarse memory granularity is measuring the Resident Set Size (RSS) which is normally in chunks of 4096 bytes. An example of fine memory granularity is recording every malloc() and free().

• Execution Granularity: In how much code detail is the memory change observed. An example of coarse execution granularity is measuring the memory usage every second. An example of fine execution granularity is recording the memory usage every Python line.

• Memory Cost: How much extra memory the tool needs.

• Execution Cost: How much the execution time is increased.

Clearly there are trade-offs between these depending on the problem you are trying to solve.

Firstly granularity:

Tool Granularity

Tool	Memory Granularity	Execution Granularity
process_tree	RSS.	Regular time intervals.
process	RSS.	Regular time intervals.
cPyMemTrace.Profile	RSS.	Per Python line, Python function and return. C function call and return.
cPyMemTrace.Trace	RSS.	Per Python line, Python function and return. Python Opcode and exception.
cPyMemTrace.ReferenceTracing	RSS.	Every object allocation/de-allocation.
DTrace	Every malloc() and free().	Per function call and return.
trace_malloc	Every Python object.	Per Python line, per function call.
debug_malloc_stats	Python memory pool.	Snapshots the CPython memory pool either side of a block of code.

Secondly cost:

Tool Cost

Tool	Memory Cost	Execution Cost
process_tree	Near zero.	Near zero.
process	Near zero.	Near zero.
cPyMemTrace.Profile	Near zero.	10x to 40x.
cPyMemTrace.Trace	Near zero.	20x to 60x.
cPyMemTrace.ReferenceTracing	Near zero.	2x to 80x.
DTrace	Minimal.	90x to 100x.
trace_malloc	Significant but compensated.	900x for small objects, 6x for large objects.
debug_malloc_stats	Minimal.	+2000x for small objects, 12x for large objects.

Installation

To install pymemtrace, run this command in your terminal:

$ pip install pymemtrace

Licence

Python memory tracing.

• Free software: MIT license

• Documentation: https://pymemtrace.readthedocs.io.

• Project: https://github.com/paulross/pymemtrace.

Credits

Phil Smith (AHL) with whom a casual lunch time chat lead to the creation of an earlier, but quite different implementation, of cPyMemTrace in pure Python.

This package was created with Cookiecutter and the audreyr/cookiecutter-pypackage project template.