pymemtrace.debug_malloc_stats

This is a wrapper around sys._debugmallocstats which writes to C stderr. We capture this and can diff two calls to sys._debugmallocatats.

sys._debugmallocatats Implementation (Python 3.9)

sys__debugmallocstats_impl(PyObject *module)

Calls:

_PyObject_DebugMallocStats(stderr))

Then:

_PyObject_DebugTypeStats(stderr);

Memory Usage by the Python Object Allocator

_PyObject_DebugMallocStats is defined in Objects/obmalloc.c:

Which dumps out the arenas/pools/blocks.

Memory Usage by Type

_PyObject_DebugTypeStats is defined in Objects/object.c

This calls:

_PyDict_DebugMallocStats(out); // in Objects/dictobject.c, just calls _PyDebugAllocatorStats in Objects/obmalloc.c
_PyFloat_DebugMallocStats(out); // in Objects/dictobject.c, just calls _PyDebugAllocatorStats in Objects/obmalloc.c
_PyFrame_DebugMallocStats(out); // etc.
_PyList_DebugMallocStats(out);
_PyTuple_DebugMallocStats(out); // in Objects/tupleobject.c, calls _PyDebugAllocatorStats in Objects/obmalloc.c for (i = 1; i < PyTuple_MAXSAVESIZE; i++)

Note that only dict, float, frame, list, tuple are reported.

class pymemtrace.debug_malloc_stats.DebugMallocArenas(debug_malloc: bytes)

Decomposes this:

# arenas allocated total           =                2,033
# arenas reclaimed                 =                2,015
# arenas highwater mark            =                   18
# arenas allocated current         =                   18
18 arenas * 262144 bytes/arena     =            4,718,592

Into values for: ntimes_arena_allocated, narenas, narenas_highwater, ARENA_SIZE.

Infers: arenas_reclaimed, arenas_total.

From Object/obmalloc.c:

(void)printone(out, "# arenas allocated total", ntimes_arena_allocated);
(void)printone(out, "# arenas reclaimed", ntimes_arena_allocated - narenas);
(void)printone(out, "# arenas highwater mark", narenas_highwater);
(void)printone(out, "# arenas allocated current", narenas);

/* Total number of times malloc() called to allocate an arena. */
static size_t ntimes_arena_allocated = 0;
// b'# arenas allocated total           =                2,033'

/* High water mark (max value ever seen) for narenas_currently_allocated. */
static size_t narenas_highwater = 0;
// b'# arenas highwater mark            =                   18'

/* # of arenas actually allocated. */
size_t narenas = 0;
// b'# arenas allocated current         =                   18'

PyOS_snprintf(buf, sizeof(buf), "%" PY_FORMAT_SIZE_T "u arenas * %d bytes/arena", narenas, ARENA_SIZE);
(void)printone(out, buf, narenas * ARENA_SIZE);
// b'18 arenas * 262144 bytes/arena     =            4,718,592'
// Simple calculation: 18 * 262144 = 4718592

__init__(debug_malloc: bytes)

Constructor, decomposes this:

# arenas allocated total        -> self.ntimes_arena_allocated
# arenas reclaimed              -> self.arenas_reclaimed
# arenas highwater mark         -> self.narenas_highwater
# arenas allocated current      -> self.narenas
18 arenas * 262144 bytes/arena  -> self.narenas * self.arena_size  = self.arenas_total

__repr__()

Returns a string similar to sys._debugmallocstats.

__weakref__

list of weak references to the object (if defined)

class pymemtrace.debug_malloc_stats.DebugMallocPoolsBlocks(debug_malloc: bytes)

Decomposes this:

# bytes in allocated blocks        =            4,280,848
# bytes in available blocks        =               70,368
63 unused pools * 4096 bytes       =              258,048
# bytes lost to pool headers       =               52,272
# bytes lost to quantization       =               57,056
# bytes lost to arena alignment    =                    0
Total                              =            4,718,592

From Object/obmalloc.c:

total = printone(out, "# bytes in allocated blocks", allocated_bytes);
total += printone(out, "# bytes in available blocks", available_bytes);

PyOS_snprintf(buf, sizeof(buf), "%u unused pools * %d bytes", numfreepools, POOL_SIZE);
total += printone(out, buf, (size_t)numfreepools * POOL_SIZE);

total += printone(out, "# bytes lost to pool headers", pool_header_bytes);
total += printone(out, "# bytes lost to quantization", quantization);
total += printone(out, "# bytes lost to arena alignment", arena_alignment);
(void)printone(out, "Total", total);

Extracts: allocated_bytes, available_bytes, numfreepools, POOL_SIZE, pool_header_bytes, quantization, arena_alignment.

Infers: unused_pool_total, TOTAL.

__init__(debug_malloc: bytes)

__repr__()

Returns a string similar to sys._debugmallocstats.

__weakref__

list of weak references to the object (if defined)

pool_overhead(num_pools: int) → int

Returns the POOL_OVERHEAD as self.pool_header_bytes // the number of pools.

self.pool_header_bytes comes from:

# bytes lost to pool headers       =               51,264

Number of pools comes from the sum of num pools from DebugMallocStat.num_pools

class pymemtrace.debug_malloc_stats.DebugMallocStat(block_class: int, size: int, num_pools: int, blocks_in_use: int, avail_blocks: int)

Represents a single line in the malloc stats section. For example:

class   size   num pools   blocks in use  avail blocks
-----   ----   ---------   -------------  ------------
    0     16           2             297           209

Nomenclature is from _PyObject_DebugMallocStats(stderr)) in Objects/obmalloc.c. Typical implementation:

for (i = 0; i < numclasses; ++i) {
    size_t p = numpools[i];
    size_t b = numblocks[i];
    size_t f = numfreeblocks[i];
    uint size = INDEX2SIZE(i);
    if (p == 0) {
        assert(b == 0 && f == 0);
        continue;
    }
    fprintf(out, "%5u %6u "
                    "%11" PY_FORMAT_SIZE_T "u "
                    "%15" PY_FORMAT_SIZE_T "u "
                    "%13" PY_FORMAT_SIZE_T "u\n",
            i, size, p, b, f);
    allocated_bytes += b * size;
    available_bytes += f * size;
    pool_header_bytes += p * POOL_OVERHEAD;
    quantization += p * ((POOL_SIZE - POOL_OVERHEAD) % size);
}
fputc('\n', out);

__getnewargs__()

Return self as a plain tuple. Used by copy and pickle.

static __new__(_cls, block_class: int, size: int, num_pools: int, blocks_in_use: int, avail_blocks: int)

Create new instance of DebugMallocStat(block_class, size, num_pools, blocks_in_use, avail_blocks)

__repr__()

Representation of self of the form:

0     16           4             777           235

_asdict()

Return a new dict which maps field names to their values.

classmethod _make(iterable)

Make a new DebugMallocStat object from a sequence or iterable

_replace(**kwds)

Return a new DebugMallocStat object replacing specified fields with new values

avail_blocks: int

Alias for field number 4

block_class: int

Alias for field number 0

blocks_in_use: int

Alias for field number 3

num_pools: int

Alias for field number 2

size: int

Alias for field number 1

class pymemtrace.debug_malloc_stats.DebugTypeStat(free_count: int, object_type: str, bytes_each: int, bytes_total: int)

Represents a single line from sys._debugmallocstats.

Decomposed from a line such as:

4 free PyCFunctionObjects * 56 bytes each =                  224

See _PyObject_DebugTypeStats(stderr); in Objects/obmalloc.c

__getnewargs__()

Return self as a plain tuple. Used by copy and pickle.

static __new__(_cls, free_count: int, object_type: str, bytes_each: int, bytes_total: int)

Create new instance of DebugTypeStat(free_count, object_type, bytes_each, bytes_total)

__repr__()

Returns a string of the form of these lines:

      4 free PyCFunctionObjects * 56 bytes each =                  224
           9 free PyDictObjects * 48 bytes each =                  432
          5 free PyFloatObjects * 24 bytes each =                  120
         0 free PyFrameObjects * 368 bytes each =                    0
          80 free PyListObjects * 40 bytes each =                3,200
         8 free PyMethodObjects * 48 bytes each =                  384
  7 free 1-sized PyTupleObjects * 32 bytes each =                  224
 52 free 2-sized PyTupleObjects * 40 bytes each =                2,080
  1 free 3-sized PyTupleObjects * 48 bytes each =                   48
0 free 10-sized PyTupleObjects * 104 bytes each =                    0

_asdict()

Return a new dict which maps field names to their values.

classmethod _make(iterable)

Make a new DebugTypeStat object from a sequence or iterable

_replace(**kwds)

Return a new DebugTypeStat object replacing specified fields with new values

bytes_each: int

Alias for field number 2

bytes_total: int

Alias for field number 3

free_count: int

Alias for field number 0

object_type: str

Alias for field number 1

class pymemtrace.debug_malloc_stats.DiffSysDebugMallocStats

Context manager that compares two snapshots of sys._getdebugmallocstats() and can provide a diff between them.

__enter__()

Enters the context manager taking a snapshot of sys._getdebugmallocstats().

__exit__(exc_type, exc_val, exc_tb)

Exits the context manager taking a snapshot of sys._getdebugmallocstats().

__init__()

__weakref__

list of weak references to the object (if defined)

diff() → str

Returns the difference between two snapshots.

pymemtrace.debug_malloc_stats.POOL_OVERHEAD = 48

This value is initially approximate. In Object/obmalloc.c:

#define POOL_OVERHEAD   _Py_SIZE_ROUND_UP(sizeof(struct pool_header), ALIGNMENT)

We can calculate this from the sum of num_pools divided into '# bytes lost to pool headers'. This is done whenever a SysDebugMallocStats is created.

pymemtrace.debug_malloc_stats.POOL_SIZE = 4096

In Object/obmalloc.c:

#define POOL_SIZE               SYSTEM_PAGE_SIZE        /* must be 2^N */

pymemtrace.debug_malloc_stats.RE_DEBUG_MALLOC_ARENAS_SUMMARY_LINE = re.compile(b'^(\\d+) arenas \\* (\\d+) bytes/arena\\s+=\\s+(.+)$')

Matches:

b'18 arenas * 262144 bytes/arena     =            4,718,592'

Decomposed to extract the two integers, the total is computed.

pymemtrace.debug_malloc_stats.RE_DEBUG_MALLOC_HEADER_LINE = re.compile(b'^Small block threshold = (\\d+), in (\\d+) size classes\\.$')

Matches:

b'Small block threshold = 512, in 32 size classes.'

Decomposed to extract the two integers.

pymemtrace.debug_malloc_stats.RE_DEBUG_MALLOC_POOLS_SUMMARY_LINE = re.compile(b'^(\\d+) unused pools \\* (\\d+) bytes\\s+=\\s+(.+)$')

Matches:

b'63 unused pools * 4096 bytes       =              258,048'

Decomposed to extract the two integers, the total is computed.

pymemtrace.debug_malloc_stats.RE_DEBUG_MALLOC_STATS_LINE = re.compile(b'^\\s+(\\d+)\\s+(\\d+)\\s+(\\d+)\\s+(\\d+)\\s+(\\d+)$')

Matches:

b'    0     16           2             294           212'

Decomposed to extract the five integers.

pymemtrace.debug_malloc_stats.RE_DEBUG_MALLOC_TYPE_LINE = re.compile(b'^\\s*(\\d+) free (.+?) \\* (\\d+) bytes each =\\s+(.+)$')

Matches:

b'   34 free 2-sized PyTupleObjects * 40 bytes each =                1,360'

NOTE: commas in last value caused by printone() in Object/obmalloc.c printone is used in many places where there is message = value such as memory pool totals and type memory information.

class pymemtrace.debug_malloc_stats.SysDebugMallocStats(debug_malloc: bytes = b'')

This decomposes the output of sys._debugmallocstats into these areas:

• A list of malloc stats showing the pools and blocks.

• Descriptions of arenas.

• Descriptions of pools and blocks.

• A list of malloc usage by (some) types.

This class takes a snapshot of the debug malloc stats from sys._debugmallocstats. Importantly it can identify the difference between two snapshots.

__init__(debug_malloc: bytes = b'')

Constructor, this optionally takes a bytes object for testing. If nothing supplied this gets the bytes object from sys._debugmallocstats.

__repr__()

Representation of self similar to the output of sys._debugmallocstats

__weakref__

list of weak references to the object (if defined)

has_object_type(object_type: bytes)

Return True if the object type is present.

object_types() → KeysView[bytes]

Return all the known object types.

type_stat(object_type: bytes) → DebugTypeStat

Return the DebugTypeStat for the named object type. May raise an KeyError if the object_type doe not exist.

pymemtrace.debug_malloc_stats.diff_debug_malloc_stat(a: DebugMallocStat, b: DebugMallocStat) → str

Takes two DebugMallocStat objects and returns a string with the difference. The string is of similar format to the input from sys._debugmallocstats.

pymemtrace.debug_malloc_stats.diff_debug_type_stat(a: DebugTypeStat, b: DebugTypeStat) → str

Takes two DebugMallocStat objects and returns a string with the difference. The string is of similar format to the input from sys._debugmallocstats.

pymemtrace.debug_malloc_stats.diff_debugmallocstats(a_stats: SysDebugMallocStats, b_stats: SysDebugMallocStats)

This takes two SysDebugMallocStats objects and identifies what is different between them. The diff is a list of lines of identical form to sys._debugmallocstats() with ‘+’ or ‘-’ where appropriate. Lines that are the same are omitted.

pymemtrace.debug_malloc_stats.get_debugmallocstats() → bytes

Invokes sys._debugmallocstats and captures the output as bytes.

pymemtrace.debug_malloc_stats.last_value_as_int(line: bytes) → int

Returns that last value of the line such as:

b'# arenas allocated total           =                2,033'