debugging - Get stacktrace from stuck python process that does not accept signals

Question

I have to run a legacy Zope2 website and have some grievance with it. The biggest issue is that, occasionally, it just locks up, running at 100% CPU load and not answering to requests anymore. While the problem isn't reproducible on a regular basis, one page containing 3 dynamic graphs triggers it sometimes, so I suspect some kind of race condition that leads to an endless loop or a stuck busywait.

The problem is, I have not yet found a way to debug this thing. There's nothing in the Zope logs and nothing in the system logs. I tried the suggestions from this question to get a stacktrace, but the only signal that has any effect is SIGKILL.

Is there another possibility to find out where exactly the process is when it gets stuck?

Answer 1

You can print out a nice stack trace using pyrasite.

First, you'll need to have gdb installed.

# Redhat, CentOS, etc
$ yum install gdb

# Ubuntu, Debian, etc
$ apt-get update && apt-get install gdb

Then, install pyrasite.

$ pip install pyrasite

Use ps or some other method to find the process ID for the stuck python process and run pyrasite-shell with it.

# Assuming process ID is 12345
$ pyrasite-shell 12345

You should now see a python REPL. Run the following in the REPL to see stack traces for all threads.

import sys, traceback
for thread_id, frame in sys._current_frames().items():
    print 'Stack for thread {}'.format(thread_id)
    traceback.print_stack(frame)
    print ''