trace_malloc
Examples¶
trace_malloc
contains some utility wrappers around the tracemalloc
module.
It can compensate for the memory used by tracemalloc
module.
These Python examples are in pymemtrace.examples.ex_trace_malloc
Using trace_malloc
Directly¶
Adding 1Mb Strings¶
Here is an example of adding 1Mb strings to a list under the watchful eye of trace_malloc.TraceMalloc
:
from pymemtrace import trace_malloc
list_of_strings = []
print(f'example_trace_malloc_for_documentation()')
with trace_malloc.TraceMalloc('filename') as tm:
for i in range(8):
list_of_strings.append(' ' * 1024**2)
print(f' tm.memory_start={tm.memory_start}')
print(f'tm.memory_finish={tm.memory_finish}')
print(f' tm.diff={tm.diff}')
for stat in tm.statistics:
print(stat)
Typical output is:
example_trace_malloc_for_documentation()
tm.memory_start=13072
tm.memory_finish=13800
tm.diff=8388692
pymemtrace/examples/ex_trace_malloc.py:0: size=8194 KiB (+8193 KiB), count=16 (+10), average=512 KiB
/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/tracemalloc.py:0: size=6464 B (+504 B), count=39 (+10), average=166 B
Documents/workspace/pymemtrace/pymemtrace/trace_malloc.py:0: size=3076 B (-468 B), count=10 (-1), average=308 B
/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/logging/__init__.py:0: size=16.3 KiB (-128 B), count=49 (-2), average=340 B
/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/abc.py:0: size=3169 B (+0 B), count=30 (+0), average=106 B
/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/posixpath.py:0: size=480 B (+0 B), count=1 (+0), average=480 B
/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/threading.py:0: size=168 B (+0 B), count=2 (+0), average=84 B
/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/_weakrefset.py:0: size=72 B (+0 B), count=1 (+0), average=72 B
To eliminate the lines that is caused by tracemalloc
itself change the last two lines to:
for stat in tm.net_statistics:
print(stat)
Which removes the line:
/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/tracemalloc.py:0: size=6464 B (+504 B), count=39 (+10), average=166 B
Using trace_malloc
as a Decorator¶
trace_malloc
provides a function decorator that can log the tracemalloc memory usage caused by execution a function.
For example:
from pymemtrace import trace_malloc
@trace_malloc.trace_malloc_log(logging.INFO)
def example_decorator_for_documentation(list_of_strings):
for i in range(8):
list_of_strings.append(create_string(1024**2))
list_of_strings = []
example_decorator_for_documentation(list_of_strings)
Would log something like the following:
2020-11-15 18:37:39,194 - trace_malloc.py#87 - 10121 - (MainThread) - INFO - TraceMalloc memory delta: 8,389,548 for "example_decorator_for_documentation()"
Cost of trace_malloc
¶
In pymemtrace.examples.ex_trace_malloc
are some timing examples of creating a list of strings of varying size
with and without trace_malloc.TraceMalloc
.
Here are some typical results:
Using key_type ‘filename’¶
Task |
Without |
With |
Ratio |
---|---|---|---|
256 byte strings |
5.7 |
4800 |
x840 |
1024 byte strings |
18 |
5100 |
x290 |
1Mb strings |
1100 |
6300 |
x5.8 |
Using key_type ‘lineno’¶
Task |
Without |
With |
Ratio |
---|---|---|---|
256 byte strings |
5.7 |
5000 |
x880 |
1024 byte strings |
18 |
5200 |
x290 |
1Mb strings |
1100 |
6400 |
x5.9 |
Using key_type ‘traceback’¶
Task |
Without |
With |
Ratio |
---|---|---|---|
256 byte strings |
5.7 |
5000 |
x880 |
1024 byte strings |
18 |
5100 |
x290 |
1Mb strings |
1100 |
6300 |
x5.9 |