Graham King

Solvitas perambulum

Finding memory leaks in Python with objgraph

software
Summary
To identify memory leaks in Python, use the `objgraph` module, which simplifies the process. First, install the module using `pip install objgraph`. Insert `import pdb; pdb.set_trace()` at the critical point in your code to enter the debugger. Once in the debugger, use `objgraph.show_growth(limit=10)` to baseline memory usage. Execute the function or code that causes the leak, and call `objgraph.show_growth(limit=10)` again to see the new or increased objects. To trace these objects' references, use `objgraph.find_backref_chain()` combined with `objgraph.show_chain()`, specifying a file name to visualize the ownership chain in a generated graph, facilitating easy identification of the memory leak source.

After a frustrating time trying to find a memory leak in my Python code with guppy / heappy hpy, I tried objgraph and, wow, it makes it so easy! Here’s what you do:

pip install objgraph

At the relevant point in your code, add an import pdb; pdb.set_trace() to drop into the debugger. Then just follow the docs on finding memory leaks with objgraph. In short you do:

>> import objgraph
>> objgraph.show_growth(limit=10)   # Start counting

Ignore that output. Call the function that leaks memory, iterate once through you loop, whatever you need to do to make your program consume more memory. Now call show_growth again:

>> my_leaky_func()
>> objgraph.show_growth(limit=10)   # Stop and show change

This time it shows the difference between now and the last time you called it. Those extra objects are the problem.

Finally you need to find where the reference to those leaky objects is being held:

>> import inspect, random
>> objgraph.show_chain(
...     objgraph.find_backref_chain(
...         random.choice(objgraph.by_type('MyBigFatObject')),
...         inspect.ismodule),
...     filename='chain.png')

That generates a lovely graph in your current directory, showing the ownership chain. Now wasn’t that easy? Thanks Marius Gedminas!