Merging Stackless Python 3.1

Releasing a new version of Stackless Python is never a straightforward experience. There are always a range of presumably avoidable problems that seem to pop up and slow the process down.

This time the problems encountered were:

• Pre-existing failures in the Python test suite.
• Changes in the standard Python pickling unit test approach.
• Dynamic behaviour in the unittest.py script involving itself in the Stackless function pickling code.
• TortoiseSVN giving a range of incomprehensible errors.

Pre-existing standard unit test failures

When there are failures in the Python unit tests in the Stackless build, a good step to tracking them down is to define STACKLESS_OFF and compile the Stackless source code into standard Python. If the tests fail there, then it might be a mismerge, or it might be that the tests fail in the standard Python branch that was merged from. After compiling and testing the standard Python branch and not seeing test failures there, that then points to a mismerge.

In this case, the same tests failed to pass in the standard Python branch. While these false negatives increase the time and effort required to do the merge, they are of course better than having mismerged. They can also be ignored, as the goal of Stackless is to be backwards compatible with standard Python...

The first Python unit test failure was in the distutils tests. It looks like the exported symbol prefix for module initialisation has changed from init to PyInit_, but a unit test had not been changed to generate a method with the new prefix.

test_distutils
xxmodule.c
   Creating library c:\users\richard\appdata\local\temp\tmpbc9aj_\Debug\users\richard\appdata\local\temp\tmpbc9aj_\xx_d.lib and object c:\users\richard\appdata\local\temp\tmpbc9aj_\Debug\users\richard\appdata\local\temp\tmpbc9aj_\xx_d.exp
foo.c
LINK : error LNK2001: unresolved external symbol PyInit_foo
c:\users\richard\appdata\local\temp\tmp6u2yky\tempt\users\richard\appdata\local\temp\tmpa9gpqp\foo_d.lib : fatal error LNK1120: 1 unresolved externals
[37422 refs]

D:\SVN\_python\python-branches\py3k-export\py3k>exit 1

D:\SVN\_python\python-branches\py3k-export\py3k>exit 0
test test_distutils failed -- Traceback (most recent call last):
  File "D:\SVN\_python\python-branches\py3k-export\py3k\lib\distutils\msvc9compiler.py", line 635, in link
    self.spawn([self.linker] + ld_args)
  File "D:\SVN\_python\python-branches\py3k-export\py3k\lib\distutils\ccompiler.py", line 981, in spawn
    spawn(cmd, dry_run=self.dry_run)
  File "D:\SVN\_python\python-branches\py3k-export\py3k\lib\distutils\spawn.py", line 36, in spawn
    _spawn_nt(cmd, search_path, dry_run=dry_run)
  File "D:\SVN\_python\python-branches\py3k-export\py3k\lib\distutils\spawn.py", line 77, in _spawn_nt
    "command '%s' failed with exit status %d" % (cmd[0], rc))
distutils.errors.DistutilsExecError: command '"c:\Program Files\Microsoft Visual Studio 9.0\VC\BIN\link.exe"' failed with exit status 1120

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "D:\SVN\_python\python-branches\py3k-export\py3k\lib\distutils\tests\test_build_ext.py", line 320, in test_get_outputs
    cmd.run()
  File "D:\SVN\_python\python-branches\py3k-export\py3k\lib\distutils\command\build_ext.py", line 347, in run
    self.build_extensions()
  File "D:\SVN\_python\python-branches\py3k-export\py3k\lib\distutils\command\build_ext.py", line 456, in build_extensions
    self.build_extension(ext)
  File "D:\SVN\_python\python-branches\py3k-export\py3k\lib\distutils\command\build_ext.py", line 543, in build_extension
    target_lang=language)
  File "D:\SVN\_python\python-branches\py3k-export\py3k\lib\distutils\ccompiler.py", line 791, in link_shared_object
    extra_preargs, extra_postargs, build_temp, target_lang)
  File "D:\SVN\_python\python-branches\py3k-export\py3k\lib\distutils\msvc9compiler.py", line 637, in link
    raise LinkError(msg)
distutils.errors.LinkError: command '"c:\Program Files\Microsoft Visual Studio 9.0\VC\BIN\link.exe"' failed with exit status 1120

The second unit test failure was within the socket unit tests, something about binding them.

test_socket
test test_socket failed -- Traceback (most recent call last):
  File "D:\SVN\_python\python-branches\py3k-export\py3k\lib\test\test_socket.py", line 498, in testSockName
    sock.bind(("0.0.0.0", port))
socket.error: [Errno 10048] Only one usage of each socket address (protocol/network address/port) is normally permitted

Legitimate standard unit test failures

Recompiling the merged source code with Stackless enabled showed an additional unit test failure in test_pickle.py. Stackless allows pickling of running code and in order to do this, it alters what gets pickled. In this case, the fix was simply pickling the problematic object, taking the result and and updating the unit test to include it. Of course, the case where Stackless is disabled still needs to be handled.

try:
    import stackless
    DATA4 = b'\x80\x02cstackless._wrap\nrange\nq\x00K\x00K\x05K\x01\x87q\x01Rq\x02)b.'
except:
    DATA4 = b'\x80\x02c__builtin__\nxrange\nq\x00K\x00K\x05K\x01\x87q\x01Rq\x02.'

Stackless pickling unit test failure

There was one failure in the Stackless unit tests, to do with pickling a recursive function. Editing the test and reconciling what comes out of pickletools.dis with the test failure stack trace made it clear what the problem was. It was dynamic behaviour in the unittest.py module, because there were self references in the recursive function being pickled, the test suite instance was being included with the pickled function. On unpickling, a __getattr__ hook in the _WritelnDecorator text UI test running support class was entering an infinite loop. The correct fix here was to remove the self reference to stop the test case instance being dragged into the pickle.

TortoiseSVN problems

It used to be that TortoiseSVN just worked. I'd go through the trauma of merging a Python branch and building an installer, and then I could rely on it to do its job. But these days, every time I do a merge, it seems that TortoiseSVN gets flakier and flakier.

When I went to commit the merged Stackless code into our py3k branch, TortoiseSVN complained about how a new file from the original Python branch was already present. I tried to do a "clean up" on the relevant directory, which did not work. I tried to exclude the relevant directory from the commit operation, which resulted in another unclear and apparently irrelevant error to do with recursive commits needing to be done a certain way. Eventually, what worked was reverting the directory and readding it.

With the merged changes committed, there was one more code change to do. The configure script is a unix script and is generated from another file, configure.in. It does not merge well, so the best approach is to not merge it and once all the other merges are checked in, fetch the branch on a Linux machine and rebuild configure there. Then I download the regenerated file to my local machine via the web, and commit it locally. Normally is as straightforward as it sounds and just works, but this time, TortoiseSVN complained about inconsistent line endings. The file has the subversion property of native line endings, and in the past it was just accepted, but not any more. What worked was loading the file up in a decent editor that allowed me to save it out in "unix" format.

Doing the release

I am not releasing Stackless Python 3.1 yet. In order to allow extensions to be compatible with Python builds, there is an official version of Visual Studio that needs to be used for both of these things. And I do not have access to an install of Visual Studio 2008 at this time. At some later stage, I might have the time and interest to download Visual C++ Express 2008, and build the relevant binaries. But I cannot guarantee that the express version of Visual Studio can build installers anyway, which might make it pointless.

So, at this time, Stackless Python 3.1 is not officially released.

IOCP-based sockets with ctypes in Python: 7

Previous post: IOCP-based sockets with ctypes in Python: 6

In the last attempt, I managed to get a minimal amount of standard socket code making use of the functionality Stackless Python provides, working within my script. But if I am to make a proper replacement socket module, I need to be able to use it in the same way stacklesssocket.py is used. To this end, the use case I want to support is Andrew Dalke's "more naturally usable" example script.

As can be seen in the script, the following code was the way the replacement socket module was put in place:

import stacklesssocket
sys.modules["socket"] = stacklesssocket

Back in Python 2.5 when it was being developed, the socket module was a lot simpler and the approach of making a complete replacement module was feasible. These days however, in later versions of Python 2 and also in Python 3, the socket module has become somewhat more complicated and the approach of leaving the standard version in place and substituting parts became practical. This lead to new code to substitute in that functionality:

import stacklesssocket
stacklesssocket.install()

The goal

Andrew's example script, the supporting of which is my goal, contains the following code:

import sys
import stacklesssocket
import stackless

stacklesssocket.install()

import urllib
import time

def download(uri):
     t1 = time.time()
     f = urllib.urlopen(uri)
     s = f.read()
     t2 = time.time()
     print "Downloaded", uri, "in", "%.1f" % (t2-t1), "seconds"
     return t2-t1


print " === Serial === "
t1 = time.time()
download("http://www.stackless.com/wiki/Tasklets")
download("http://www.stackless.com/wiki/Channels")
t2 = time.time()
print " --->", t2-t1

print " === Parallel === "
t1 = time.time()
stackless.tasklet(download)("http://www.stackless.com/wiki/Tasklets")
stackless.tasklet(download)("http://www.stackless.com/wiki/Channels")
stackless.run()
t2 = time.time()
print " --->", t2-t1

The most important aspect of this, is how it uses another higher level standard library module (urllib) that in turn uses the standard socket module. By using my replacement functionality in this way, I can verify how compatible and stable it might be.

The implementation

The original stacklesssocket.py module was based on asyncore, another framework that wraps select. Back when I wrote it, I was glad to have it there to use, but the fact is that adopting a framework brings in dependencies and abitrary abstraction. If I had the time and interest, I would rewrite stacklesssocket.py to use select directly.

At this time, I am developing for Stackless Python 2.6.2. Right off the bat, it was obvious that this new module was going to be somewhat simpler. There were several substitutions to the standard socket module for stacklesssocket.py, but in this case it looks like I just need to subsitute my own _realsocket object.

def install():
    stdsocket._realsocket = socket

The first obstacle was my lack of a connect method on my replacement socket object. Checking out the available functions on MSDN, ConnectEx was the one which suited my overlapped IO needs. However, I was unable to find it exported from any Winsock related DLL. It turns out, you need to call WSAIoctl in order to get a pointer to it. I had been avoiding calling WSAIoctl as I assumed it was only used to make faster function calls, but as it is the gateway to functionality I need that is no longer something I can do.

int WSAIoctl(
  __in   SOCKET s,
  __in   DWORD dwIoControlCode,
  __in   LPVOID lpvInBuffer,
  __in   DWORD cbInBuffer,
  __out  LPVOID lpvOutBuffer,
  __in   DWORD cbOutBuffer,
  __out  LPDWORD lpcbBytesReturned,
  __in   LPWSAOVERLAPPED lpOverlapped,
  __in   LPWSAOVERLAPPED_COMPLETION_ROUTINE lpCompletionRoutine
);

The corresponding ctypes definition is as follows:

WSAIoctl = windll.Ws2_32.WSAIoctl
WSAIoctl.argtypes = (SOCKET, DWORD, c_void_p, DWORD, c_void_p, DWORD, POINTER(DWORD), POINTER(OVERLAPPED), c_void_p)
WSAIoctl.restype = c_int

The appropriate dwIoControlCode value to get a function pointer is SIO_GET_EXTENSION_FUNCTION_POINTER. The value of lpvInBuffer to indicate that the pointer to ConnectEx is the one I want, is the following GUID.

{ 0x25a207b9,0xddf3,0x4660, { 0x8e,0xe9,0x76,0xe5,0x8c,0x74,0x06,0x3e }}

I'm not doing this as an overlapped operation, as I can't see how it could end up blocking in any meaningful way. This resulted in the following code in the replacement sockets __init__ method:

        self.wsFnConnectEx = ConnectExFunc()
        dwBytes = DWORD()
        ret = WSAIoctl(_socket, SIO_GET_EXTENSION_FUNCTION_POINTER, byref(WSAID_CONNECTEX), sizeof(WSAID_CONNECTEX), byref(self.wsFnConnectEx), sizeof(self.wsFnConnectEx), byref(dwBytes), cast(0, POINTER(OVERLAPPED)), 0)
        if ret == SOCKET_ERROR:
            err = WSAGetLastError()
            closesocket(ret)
            raise WinError(err)

With this out of the way, I can now move onto fleshing out the connect method.

    def connect(self, address):
        host, port = address

        self.bind(("0.0.0.0", 0))

        sa = sockaddr_in()
        sa.sin_family = AF_INET
        sa.sin_addr.s_addr = inet_addr(host)
        sa.sin_port = htons(port)

        bytesSent = DWORD(0)
        ovConnect = OVERLAPPED()
        c = ovConnect.channel = stackless.channel()

        ret = self.wsFnConnectEx(self._socket, sa, sizeof(sa), 0, 0, NULL, byref(ovConnect))
        if ret == FALSE:
            err = WSAGetLastError()
            # The operation was successful and is currently in progress.  Ignore this error...
            if err != ERROR_IO_PENDING:
                raise WinError()
            
        activeIO[self._socket] = c
        c.receive()

ConnectEx was not working on the first attempt, it turns out you need to bind a socket to a local address before you can connect using it. Oh well, time to retry Andrew's script and see how I go.

This time, the lack of a sendall method was preventing the script from running. sendall just continues calling send for the same set of data, until all of it has been sent.

    def sendall(self, msg, flags=None):
        bytesSent = self.send(msg)
        while bytesSent < len(msg):
            bytesSent += self.send(msg[bytesSent:])

At this point I decided to take note of the fact that it is possible for operations that are performed with an overlapped object, may actually complete right away. However, my mistake was in assuming that if this happened, a completion packet wouldn't arrive through GetQueuedCompletionStatus. This led to crashes where I examined the channel attribute on arrived packets, where the overlapped object had already been garbage collected due to the exit of the function that started the operation.

In order to clean things up, I decided to store a reference to the overlapped object ina global variable and in addition move the channel reference I hold also outside and alongside of that overlapped object. I could have stored the socket handle in the overlapped object, but since CreateIOCompletionPort allows me to associate a completion key with the socket, I store it there instead.

        # Bind the socket to the shared IO completion port.
        CreateIoCompletionPort(_socket, hIOCP, _socket, NULL)

The idea was that I could then ignore packets that arrived for earlier operations which had actually completed immediately. Of course, logic having gotten a result for an IO operation may try another which might this time block. This means that unwanted packets for the operations that had completed immediately would come in and assume they were for the followup blocking operation. Errors ensued. So each completed packet needs to be able to be matched up with the operation state stored for it.

This left connect in the following state:

    def connect(self, address):
        host, port = address

        self.bind(("0.0.0.0", 0))

        sa = sockaddr_in()
        sa.sin_family = AF_INET
        sa.sin_addr.s_addr = inet_addr(host)
        sa.sin_port = htons(port)

        bytesSent = DWORD(0)
        ovConnect = OVERLAPPED()
        opID = ovConnect.opID = self.__getnextopid()
        c = stackless.channel()
        c.preference = 0
        ovConnect.label = "connect"

        activeOps[(self._socket, opID)] = (c, ovConnect)

        ret = self.wsFnConnectEx(self._socket, sa, sizeof(sa), 0, 0, NULL, byref(ovConnect))
        if ret == FALSE:
            err = WSAGetLastError()
            if err != ERROR_IO_PENDING:
                raise WinError()
            
            c.receive()

The other overlapped IO operations also needed to be changed to match, along with _DispatchIOCP.

def _DispatchIOCP():
    numberOfBytes = DWORD()
    completionKey = c_ulong()
    ovCompletedPtr = POINTER(OVERLAPPED)()

    def _GetCompletionChannel(completionKey, overlappedPtr):
        _socket = completionKey.value
        opID = ovCompletedPtr.contents.opID
        k = _socket, opID
        c, ovRef = activeOps[k]
        del activeOps[k]

        return c

    while stackless.runcount > 2:
        while True:
            stackless.schedule()

            c = None
            ret = GetQueuedCompletionStatus(hIOCP, byref(numberOfBytes), byref(completionKey), byref(ovCompletedPtr), 50)
            if ret == FALSE:
                err = WSAGetLastError()
                if err == WAIT_TIMEOUT:
                    continue

                if not bool(ovCompletedPtr):
                    raise WinError(err)

                c = _GetCompletionChannel(completionKey, ovCompletedPtr)
                c.send_exception(WinError, err)
                continue

            break

        c = _GetCompletionChannel(completionKey, ovCompletedPtr)
        if c.balance == -1:
            c.send(numberOfBytes.value)

One note is that accessing ovCompletedPtr.contents will error if the value is NULL, the safe way with ctypes to ascertain whether the value is NULL is to check if the boolean value is False.

Running Andrew's script at this point results in it exiting cleanly, but mid-execution. The reason for this is the condition stackless.runcount > 2. The theory is that this should keep the loop running as long as there are at least two tasklets in the scheduler, this is just plain wrong and ill thought out. The fact is that at any given time, at least for Andrew's script, the runcount is only likely to be one. The reason for this is that all the other tasklets doing IO will be outside of the scheduler, blocked on a channel. So the lowest possible valid value for runcount that will require continued looping will be one.

Taking this into account, I added a weakref dictionary to track the socket objects which were live and added it as an additional reason to loop. This in no way addresses the above problem, but it does the trick. Andrew's script now runs, and gives the following output:

 === Serial ===
Downloaded http://www.stackless.com/wiki/Tasklets in 3.4 seconds
Downloaded http://www.stackless.com/wiki/Channels in 3.9 seconds
 ---> 7.26900005341
 === Parallel ===
Downloaded http://www.stackless.com/wiki/Channels in 3.3 seconds
Downloaded http://www.stackless.com/wiki/Tasklets in 3.5 seconds
 ---> 3.51300001144

Success (of a sort)!

Of course the module in no way provides the full functionality of the standard version yet. Trivial aspects no-one in their right mind should be using, like UDP and so forth, are completely unsupported.

Script source code: 06 - Polished Socket.py

Sims bakery: Lambs fry and bacon pie

Driving past Sims bakery this morning, I bought several interesting things. I always keep an eye out for interesting pies, and lambs fry and bacon sounded pretty tasty.



The inside is pretty disappointing and there is very little bacon or lambs fry. Most of the pie was a thick glutinous translucent "gravy" which was pretty tasteless.