trace_malloc Examples

trace_malloc contains some utility wrappers around the tracemalloc module. It can compensate for the memory used by tracemalloc module.

These Python examples are in pymemtrace.examples.ex_trace_malloc

Using trace_malloc Directly

Adding 1Mb Strings

Here is an example of adding 1Mb strings to a list under the watchful eye of trace_malloc.TraceMalloc:

from pymemtrace import trace_malloc

list_of_strings = []
print(f'example_trace_malloc_for_documentation()')
with trace_malloc.TraceMalloc('filename') as tm:
    for i in range(8):
        list_of_strings.append(' ' * 1024**2)
print(f' tm.memory_start={tm.memory_start}')
print(f'tm.memory_finish={tm.memory_finish}')
print(f'         tm.diff={tm.diff}')
for stat in tm.statistics:
    print(stat)

Typical output is:

example_trace_malloc_for_documentation()
 tm.memory_start=13072
tm.memory_finish=13800
         tm.diff=8388692
pymemtrace/examples/ex_trace_malloc.py:0: size=8194 KiB (+8193 KiB), count=16 (+10), average=512 KiB
/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/tracemalloc.py:0: size=6464 B (+504 B), count=39 (+10), average=166 B
Documents/workspace/pymemtrace/pymemtrace/trace_malloc.py:0: size=3076 B (-468 B), count=10 (-1), average=308 B
/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/logging/__init__.py:0: size=16.3 KiB (-128 B), count=49 (-2), average=340 B
/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/abc.py:0: size=3169 B (+0 B), count=30 (+0), average=106 B
/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/posixpath.py:0: size=480 B (+0 B), count=1 (+0), average=480 B
/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/threading.py:0: size=168 B (+0 B), count=2 (+0), average=84 B
/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/_weakrefset.py:0: size=72 B (+0 B), count=1 (+0), average=72 B

To eliminate the lines that is caused by tracemalloc itself change the last two lines to:

for stat in tm.net_statistics:
    print(stat)

Which removes the line:

/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/tracemalloc.py:0: size=6464 B (+504 B), count=39 (+10), average=166 B

Using trace_malloc as a Decorator

trace_malloc provides a function decorator that can log the tracemalloc memory usage caused by execution a function. For example:

from pymemtrace import trace_malloc

@trace_malloc.trace_malloc_log(logging.INFO)
def example_decorator_for_documentation(list_of_strings):
    for i in range(8):
        list_of_strings.append(create_string(1024**2))

list_of_strings = []
example_decorator_for_documentation(list_of_strings)

Would log something like the following:

2020-11-15 18:37:39,194 -   trace_malloc.py#87   - 10121 - (MainThread) - INFO     - TraceMalloc memory delta: 8,389,548 for "example_decorator_for_documentation()"

Cost of trace_malloc

In pymemtrace.examples.ex_trace_malloc are some timing examples of creating a list of strings of varying size with and without trace_malloc.TraceMalloc. Here are some typical results:

Using key_type ‘filename’

Times in µs tracing filename

Task

Without trace_malloc.TraceMalloc

With trace_malloc.TraceMalloc

Ratio

256 byte strings

5.7

4800

x840

1024 byte strings

18

5100

x290

1Mb strings

1100

6300

x5.8

Using key_type ‘lineno’

Times in µs tracing lineno

Task

Without trace_malloc.TraceMalloc

With trace_malloc.TraceMalloc

Ratio

256 byte strings

5.7

5000

x880

1024 byte strings

18

5200

x290

1Mb strings

1100

6400

x5.9

Using key_type ‘traceback’

Times in µs tracing traceback

Task

Without trace_malloc.TraceMalloc

With trace_malloc.TraceMalloc

Ratio

256 byte strings

5.7

5000

x880

1024 byte strings

18

5100

x290

1Mb strings

1100

6300

x5.9