IOCP based sockets with ctypes in Python: 1

As I was reminded of recently, select in the Python library only handles 512 sockets at a time. This has given me the interest to spend a little time writing socket IO completion port support using ctypes.

In order to write code that uses C-level DLL functionality with ctypes, it requires that all structures, constants and functions are redefined at the Python level. This can be a somewhat tedious process, slowing down the progress that an interactive prompt otherwise allows. Developing an extension module in C or C++ would probably be faster, but then tradeoffs would be made in terms of framework structure and other inherent costs.

Defining a function is straightforward, given the obstacle of defining required but currently undefined structures. For instance, this is the prototype for AcceptEx:

BOOL AcceptEx(
 __in   SOCKET sListenSocket,
 __in   SOCKET sAcceptSocket,
 __in   PVOID lpOutputBuffer,
 __in   DWORD dwReceiveDataLength,
 __in   DWORD dwLocalAddressLength,
 __in   DWORD dwRemoteAddressLength,
 __out  LPDWORD lpdwBytesReceived,
 __in   LPOVERLAPPED lpOverlapped
);

In order to specify this function definition with ctypes, the OVERLAPPED structure needs to be defined first in this way.

class _US(Structure):
    _fields_ = [
        ("Offset",          DWORD),
        ("OffsetHigh",      DWORD),
    ]

class _U(Union):
    _fields_ = [
        ("s",               _US),
        ("Pointer",         c_void_p),
    ]

    _anonymous_ = ("s",)

class OVERLAPPED(Structure):
    _fields_ = [
        ("Internal",        POINTER(ULONG)),
        ("InternalHigh",    POINTER(ULONG)),

        ("u",               _U),

        ("hEvent",          HANDLE),

        # Custom fields.
        ("channel",         py_object),
    ]

    _anonymous_ = ("u",)

I wrote that several years ago, and like most ctypes-based code, it took a reasonable amount of time. Interestingly, I've seen at least one other project has copied it verbatim (minus the Stackless 'channel' field) in order to avoid this effort. Given a definition for this structure, the function definition can also be written. The first step is to get a reference to the function from the DLL it is implemented within.

AcceptEx = windll.Mswsock.AcceptEx

Next the types of arguments it takes are specified.

AcceptEx.argtypes = (SOCKET, SOCKET, c_void_p, DWORD, DWORD, DWORD, POINTER(DWORD), POINTER(OVERLAPPED))

And finally, the type of the value it returns.

AcceptEx.restype = BOOL

At this point, AcceptEx is callable and can be used as needed. But to use it requires that the functions which create sockets also be defined, and whatever other functions you may need.

I often find myself writing all the definitions consecutively in a text editor. This is long and time consuming and you have no idea whether the code actually works. It is easy to make trivial mistakes like specifying the wrong DLL for the function to be obtained from. A much more rewarding approach in my experience is to write the code line by line in a interactive Python console, and once each aspect works, compose them into a script in a text editor. This script can be pasted into a new console when you resume the effort, next you find the time and interest.