python - Exception TypeError warning sometimes shown, sometimes not when using throw method of generator

Question

There is this code:

class MyException(Exception):
  pass

def gen():
  for i in range(3):
    try:
      yield i
    except MyException:
      print("MyException!")


a = gen()
next(a) 
a.throw(MyException)

Running this code:

$ python3.3 main.py
MyException!
$ python3.3 main.py
MyException!
Exception TypeError: TypeError('catching classes that do not inherit from BaseException is not allowed',) in <generator object gen at 0xb712efa4> ignored
$ python3.3 main.py
MyException!
$ python3.3 main.py
MyException!
$ python3.3 main.py
MyException!
Exception TypeError: TypeError('catching classes that do not inherit from BaseException is not allowed',) in <generator object gen at 0xb714afa4> ignored

The thing which I don't understand is why sometimes there is printed this Exception TypeError warning. Is there something wrong with custom exception?

Answer 1

You are seeing a __del__ hook misbehaving somewhere.

The TypeError is being thrown while shutting down, as the Python interpreter is exiting everything is deleted and any exceptions thrown in a __del__ deconstructor hook are being ignored (but are printed).

On exit, Python clears everything in the namespace by rebinding everything to None, but the order in which this happens is not set. The still running generator is closed (a.close() is called) when deleted, which triggers a GeneratorExit exception in the generator, which Python tests against your except MyException: line. If, however, MyException has already been cleared up and Python sees except None: the TypeError is thrown and you see that message printed.

You can trigger the error without exiting Python by adding:

MyException = None
del a

If you use list(a) and consume the rest of the generator, or explicitly close the generator with a.close() before Python exits and deletes MyException, the error message goes away.

Another work-around would be to handle GeneratorExit first:

def gen():
  for i in range(3):
    try:
      yield i
    except GeneratorExit:
      return
    except MyException:
      print("MyException!")

and Python will not evaluate the next except handler.

The error cannot be reproduced with Python 3.2 or earlier, so it looks like hash randomization (introduced in Python 3.3) randomizes the order objects are cleared; this certainly explains why you see the error only on some of your runs, but not on earlier Python runs where the hash order is fixed.

Note that the interaction of .__del__() hooks and other global objects in Python is documented with a big red warning in the .__del__() documentation:

Warning: Due to the precarious circumstances under which __del__() methods are invoked, exceptions that occur during their execution are ignored, and a warning is printed to sys.stderr instead. Also, when __del__() is invoked in response to a module being deleted (e.g., when execution of the program is done), other globals referenced by the __del__() method may already have been deleted or in the process of being torn down (e.g. the import machinery shutting down). For this reason, __del__() methods should do the absolute minimum needed to maintain external invariants. Starting with version 1.5, Python guarantees that globals whose name begins with a single underscore are deleted from their module before other globals are deleted; if no other references to such globals exist, this may help in assuring that imported modules are still available at the time when the __del__() method is called.