Strange class leakage

One of the ways in which code reloading can be implemented in Python, with respect to classes defined within managed scripts, is by updating the existing class in place.

The process resembles the following..

1. The code reloading framework is started up, loading a main copy of every script file registered with it.
2. The user modifies a script file.
3. The code reloading framework detects and reacts to the file modification.
4. The new version of the script file is executed into a dictionary.
5. For every class in the dictionary that already existed..
   1. The original version of the class is retrieved from the main copy.
   2. Each valid dictionary entry in the new class is moved into the original class, overwriting whatever it had. If the entry is a function, it is rebuilt so that its internal func_globals dictionary points to that of the main copy, rather than the new copy.
6. The new version of the script file is discarded and garbage collected, now that the main version has been updated to resemble it.

It is a lot less legwork than the alternate approach, which is wandering around the references in the garbage collector and replacing references to the original class.

Leakage

I use the code reloading framework in a MUD framework. One of the project's tasks that I am working on at the moment, is locating the commands which a logged in player can execute.

The original approach was to simply iterate over a given namespace and check if one of the contents was an instance of the Command class. However, as there are now multiple namespace locations where commands may be located, I decided to try an alternate approach. Instead I would find use the garbage collector to find all the subclasses of the Command class, and obtain the commands that way.

for class_ in pysupport.FindSubclasses(Command, inclusive=True):
    for verb in class_.__verbs__:
        if verb in self.verbs:
            logger.warning("Duplicate '%s': %s", verb, class_)
            continue

Each time I modified the rehash command, I started to see a ghost class appearing in the output of this logging action. So for the umpteenth time, I wrote some code to dig through the referrers in the garbage collector, in order to work out what was holding the references.

Tracking references

By passing the ghost classes to PrintReferrers, I see the following..

5 REFERRERS FOR <type 'classobj'> __builtin__.Rehash 0x20a84b0
  SKIPPED/is-local-frame 0x20ebec8 <frame object at 0x020EBEC8>
  SKIPPED/is-seen-list 0x20a9e90 <type 'list'>
  SKIPPED/is-local-frame 0x20f7ae0 <frame object at 0x020F7AE0>
  6 REFERRERS FOR <type 'list'> [ ... ] 0x20ae1e8
    5 REFERRERS FOR <type 'listiterator'> 0x1c33110
      SKIPPED/is-seen-list 0x20a9e90 <type 'list'>
      SKIPPED/is-local-frame 0x1b1fc88 <frame object at 0x01B1FC88>
      SKIPPED/is-referrer-list 0x20a9c10 <type 'list'>
      SKIPPED/is-local-frame 0x1b1fe30 <frame object at 0x01B1FE30>
      SKIPPED/is-local-frame 0x20f7ae0 <frame object at 0x020F7AE0>
    SKIPPED/is-local-frame 0x20ebec8 <frame object at 0x020EBEC8>
    SKIPPED/is-seen-list 0x20a9e90 <type 'list'>
    SKIPPED/is-referrer-list 0x20a96c0 <type 'list'>
    SKIPPED/is-local-frame 0x1b1fc88 <frame object at 0x01B1FC88>
    SKIPPED/is-local-frame 0x20f7ae0 <frame object at 0x020F7AE0>
  5 REFERRERS FOR <type 'dict'> { ... } 0x20aba50
    SKIPPED/is-local-frame 0x20ebec8 <frame object at 0x020EBEC8>
    SKIPPED/is-seen-list 0x20a9e90 <type 'list'>
    SKIPPED/is-referrer-list 0x20a96c0 <type 'list'>
    SKIPPED/is-local-frame 0x20cc188 <frame object at 0x020CC188>
    5 REFERRERS FOR <type 'function'> 0x20a5eb0
      SKIPPED/is-seen-list 0x20a9e90 <type 'list'>
      SKIPPED/is-local-frame 0x20cc188 <frame object at 0x020CC188>
      SKIPPED/is-referrer-list 0x20a9490 <type 'list'>
      SKIPPED/is-local-frame 0x20cc330 <frame object at 0x020CC330>
      5 REFERRERS FOR <type 'dict'> { ... } 0x20ab9c0
        SKIPPED/is-seen-list 0x20a9e90 <type 'list'>
        SKIPPED/is-local-frame 0x20cc330 <frame object at 0x020CC330>
        SKIPPED/is-referrer-list 0x20a9ad0 <type 'list'>
        SKIPPED/is-local-frame 0x20cf078 <frame object at 0x020CF078>
        SKIPPED/seen 0x20a84b0 __builtin__.Rehash

I've trimmed some redundant information out by hand. But what it shows, is that the class has a function that has a dictionary, that has a reference to the class. Seeing what the contents of the trimmed containers were, it indicates that the normal func_globals reference is creating a circular reference and keeping the class alive.

All classes leak?

The first thing I did was rule out that this was simply the way that Python worked.

I created a script containing..

class AClass:
    def AFunction(self):
        pass

And then repeatedly reloaded the class in the interpreter..

>>> d = {}
>>> execfile("test.py", d, d)
>>> d = {}
>>> execfile("test.py", d, d)
>>> d = {}
>>> execfile("test.py", d, d)
>>> d = {}
>>> execfile("test.py", d, d)
>>> import gc, types
>>> for v in gc.get_objects():
...     if type(v) is types.ClassType and v.__name__ == "AClass":
...             print v
...
__builtin__.AClass
__builtin__.AClass
__builtin__.AClass
__builtin__.AClass
>>> gc.collect()
12
>>> for v in gc.get_objects():
...     if type(v) is types.ClassType and v.__name__ == "AClass":
...             print v
...
__builtin__.AClass

So outside of my framework Python does the correct thing.

Conclusion

Some possible reasons for what is going on:

• My reference printing function is skipping something it shouldn't (not seeing it).
• How I am calling execfile is creating this circular reference in a way that is somehow different (not seeing it).

For now, I will enter a defect and have the new script file clear its dictionary when it is garbage collected, which fixes the problem.

Writing this reference tracking code for the umpteenth time makes me wonder if something similar, but both tested and proven, is out there.