Skip to content

Profiling Python Code

View or edit on GitHub

This page is synchronized from doc/Profiling-Python-Code.md. Last modified on 2025-12-09 00:30 CET by Trase Admin. Please view or edit the original file there; changes should be reflected here after a midnight build (CET time), or manually triggering it with a GitHub action (link).

Profiling Python Code

Run the code in a profiler

The result of this is that a .pstats file will be written. There are a few options to do this:

  • Use PyCharm. There is a little icon up in the top right (right of 'Run' and 'Debug'). PyCharm will run your configuration in the profiler and write the file.

  • Execute from the command-line. If your code path is runnable from the command-line (for example a pytest or a Django management command), there is an easy way to attach the profiler:

python -m cProfile -o stats.pstats <usual command>

For example, python -m cProfile -o stats.pstats manage.py calculatevariantadvantage. One downside of doing this is that things like the pytest test setup will also be included in the .pstats file.

  • Wrap your code. Here we alter our code to be wrapped in a profiler. This is particularly useful when you want to test a code block in the web server: wrap that code, then fire off the request.
import cProfile

profile = cProfile.Profile()
profile.enable()
# ... do the code
profile.disable()
profile.dump_stats("stats.pstats")

Visualize

The .pstats file you generated contains a lot of information. We need to summarize this in a useful way to understand it. Again, there are a few options! Here are just some of then:

  • Use PyCharm. There is an inbuilt viewer!

  • Use Python's REPL tool. This is good for generating a quick table:

    ```bash $ python -m pstats stats.pstats Welcome to the profile statistics browser. stats.pstats% sort cumtime stats.pstats% stats 10 Tue May 28 15:23:46 2019 db.pstats

    9559767 function calls (9375585 primitive calls) in 13.545 seconds

Ordered by: cumulative time
List reduced from 16132 to 10 due to restriction <10>

ncalls  tottime  percall  cumtime  percall filename:lineno(function)
3119/1    0.022    0.000   13.602   13.602 {built-in method builtins.exec}
     1    0.000    0.000   13.602   13.602 manage.py:2(<module>)
...

    ```

  • Use gprof2dot. This command-line tool will generate a .dot of a call graph for you. There is a script in the codebase that additionally converts this file to a .png. It requires gprof2dot and graphviz, first install this 

    brew install gprof2dot graphviz  # e.g. on a mac...

    Then, you can generate a png file:

    trase/admin/scripts/pstats2png.sh stats.pstats stats.png --node-thres=10

    Any options supplied after the output image filename are passed through to gprof2dot. In the example above we are filtering out 10% of the smallest nodes.

  • Use SnakeViz. This Python package runs a little webserver that allows you to interactively browse a neat visualization.

    pip install snakeviz
snakeviz stats.pstat
    - Use QCacheGrind. This is a GUI program that allows you to browse the file interactively. First download and install QCacheGrind:

Then, convert the .pstat file to a format that QCacheGrind can understand:

pip install pyprof2calltree
pyprof2calltree -i stats.pstats -o cachegrind.output

Finally, open in QCacheGrind

qcachegrind cachegrind.output
# or
kcachegrind cachegrind.output