Friday, 29 April 2011

Python lazyimporting

At work, we have a modified version of this lazyimport module. Originally, I used it to work out which modules were not actually ever made use of, but the suggestion was made that it should be enabled by default. And it turns out that enabling it does result in worthwhile memory savings, something which is very useful when you have a budget, but it's not all plain sailing..

We've encountered problems along the way, one of which I will mention while it is fresh for anyone who might be using, or planning to use the same code.

static PyObject *
get_warnings_attr(const char *attr)
    static PyObject *warnings_str = NULL;
    PyObject *all_modules;
    PyObject *warnings_module;
    int result;

    if (warnings_str == NULL) {
        warnings_str = PyString_InternFromString("warnings");
        if (warnings_str == NULL)
            return NULL;

    all_modules = PyImport_GetModuleDict();
    result = PyDict_Contains(all_modules, warnings_str);
    if (result == -1 || result == 0)
        return NULL;

    warnings_module = PyDict_GetItem(all_modules, warnings_str);
    if (!PyObject_HasAttrString(warnings_module, attr))
            return NULL;
    return PyObject_GetAttrString(warnings_module, attr);

The problem is that PyDict_GetItem returns a borrowed reference, then the subsequent call to PyObject_HasAttrString causes the replacement of the lazy loading stub with the real imported module (and the garbage collection of the stub), and finally.. PyObject_GetAttrString chokes on its clobbered reference. Strangely, this only happens for certain people consistently in binaries they themselves built which are linked against the Python static library. The solution is that the warnings module needs to be excluded from lazy importing, or else this code will crash.

If there were a central repository for this module and a maintainer, I'd get our code released so that others could benefit from our changes (or we could get feedback on them).

Anyone else using this incredibly cool piece of code?