Counting Pythons In My Sleep

18 Nov 2005

In my professional life, I work with COM (pity me). One of the things I really like about JavaScript and Java is the fact that they use garbage collection rather than reference counting.

I just discovered that Python also uses reference counting, but includes a garbage collector. From my primitive tests, the garbage collector doesn’t seem to be turned on by default.

class RefCountTester(object):
    def __del__(self):
        print "Goodbye, cruel world: %r" % self

>>> r1= RefCountTester()
>>> r2= RefCountTester()
>>> r1.r2= r2
>>> r2.r1= r1
>>> del r1
>>> del r2

The code above is what I used to test the basic reference counting. I deliberately set up a cycle in the hopes that the garbage collector would kick in, but no!

I checked to see whether the garbage collector is enabled, and Python reports that it is. However, I haven’t seen either of these objects get collected yet.

Will the garbage collector call the __del__ method?

For modules which may rely on garbage collection, is it likely that I’ll run into systems (Python 2.4 and above only) which might have garbage collection turned off?

8 Comments

  1. luca
    Posted 19 Nov 2005 at 1:04 am | Permalink

    I was researching exactly the same thing some time ago, and the garbarge collector works, but __del__ is not called for cycles. The problem is that __del__ could rely on the other object on the cycle, so there’s no safe way to call the __del__ method. Suppose you have:

    class RefCountTester(object):
    def __del__(self):
        print "Goodbye, cruel world: %r, %r" % (self, self.r)
r1= RefCountTester()
r2= RefCountTester()
r1.r = r2
r2.r = r1
del r1
del r2

    In this case, if you first destroy r1, you print “Goodby, cruel world: [r1], [r2]” whithout problem, and then you have to destroy r2, you print “Goodby, cruel world: [r2], ???”, r1 is gone, what do you print? Maybe you could call all the __del__ methods in the cycle before actually freeing the memory of those object, but I don’t know the side effects of that. Anyway, I’m allmost sure taht the cycles are really collected and that the __del__ methods are not called and that this is the reason.

  2. Jeff Watkins
    Posted 19 Nov 2005 at 7:52 am | Permalink

    Luca, I’d figured there had to be something like this. So basically, the objects were getting collected, but I couldn’t tell because their “destructors” weren’t getting called. I suppose that’s OK. I never put anything in __del__. I’d never even written a __del__ method until this test. So I don’t think I’m losing out.

    This does mean I can be a little more free in designing the Python ORM I’ve been working on lately. I deliberately made an effort to prevent cycles, when I guess I don’t really have to.

  3. Andrew Dalke
    Posted 19 Nov 2005 at 10:14 am | Permalink

    From the docs at http://www.python.org/doc/2.2.3/ext/refcounts.html

    “”"The cycle detector is able to detect garbage cycles and can reclaim them so long as there are no finalizers implemented in Python (__del__() methods). When there are such finalizers, the detector exposes the cycles through the gc module (specifically, the garbage variable in that module). The gc module also exposes a way to run the detector (the collect() function), as well as configuration interfaces and the ability to disable the detector at runtime.”"”

    >>> class RefCountTester(object):
...     def __del__(self):
...         print "Goodbye, cruel world: %r, %r" % (self, self.r)
... 
>>> r1= RefCountTester()
>>> r2= RefCountTester()
>>> r1.r = r2
>>> r2.r = r1
>>> del r1
>>> del r2
>>> import gc
>>> gc.garbage 
[,
 ]
>>> del gc.garbage[0].r
>>> del gc.garbage[:]
Goodbye, cruel world: ,

>>>
  4. Matt
    Posted 19 Nov 2005 at 10:21 am | Permalink

    Any Java programmer learning Python should check out PJE’s Python Is Not Java and Java is not Python, either posts.

  5. Jeff Watkins
    Posted 19 Nov 2005 at 11:41 am | Permalink

    Matt, I’ll definitely check out Python is Not Java. Thanks for the link.

    Andrew, that makes sense. Since I never really implement destructors, I don’t suppose I’ll have any problem. Thanks for the explanation.

  6. Bob Ippolito
    Posted 19 Nov 2005 at 1:12 pm | Permalink

    The GC is always on, but it doesn’t collect cyclic garbage for objects that implement del, your demonstration is the only scenario which doesn’t work.

    If you need to implement a destructor like that, do it via a weakref callback instead… which (assuming new enough version of Python) should be able to do it correctly for cyclic garbage. Generally though, just don’t write code that needs destructors because you never know when it’ll get called.

  7. Shalabh
    Posted 22 Nov 2005 at 1:46 am | Permalink

    The weakref module will also help you test if an object has been garbage collected. While you’re implementing an ORM, also look at weak key and weak value dictionaries (in weakref module). They might help you cache objects (if you want) so that you can return an existing instance, if a reference is held somewhere in user code, otherwise create a new instance and return it.

  8. Jeff Watkins
    Posted 22 Nov 2005 at 7:35 am | Permalink

    Shalabh, actually the cache that the Context maintains keeps strong references to objects. I need to be able to write changes to the database when the Context is committed. But I was thinking of keeping a weak reference up until you make your first change. Then, one the object is dirty, keep a strong reference to make certain I can update the database.