IOCP based sockets with ctypes in Python: 4

Previous post: IOCP-based sockets with ctypes in Python: 3

The next thing I want to do with this, is to take what I have written so far and reshape it into a simple naive server. It will accept any connection made to it and then set aside that connection (and ignore it), ready to accept the next one.

I don't really need to care what IO completion packets get queued for me to receive on my IO completion port. But for the sake of extending my code base a little, I will set the completion key for each socket associated with the port anyway.

The original socket that listens for connections will get a unique value that has been set aside.

LISTEN_COMPLETION_KEY = 90L
CreateIoCompletionPort(listenSocket, hIOCP, LISTEN_COMPLETION_KEY, NULL)

And unique keys will be allocated on demand for the sockets of incoming connections which will get accepted.

currentCompletionKey = 100L

def CreateCompletionKey():
    global currentCompletionKey
    v = currentCompletionKey
    currentCompletionKey += 1L
    return v

Accepted connections that are set aside to be ignored are stored in a designated dictionary.

overlappedByKey = {}

Because I will need to accept an arbitrary number of incoming connections, rather than just the one the previous version of this script accepted, I've gathered the relevant parts of the script into a function with three additions.

The first addition is the allocation of a unique completion key for the new accept socket.

ovKey = CreateCompletionKey()

The second addition is setting this as the completion key for the socket when it is associated with the port.

CreateIoCompletionPort(_acceptSocket, hIOCP, ovKey, NULL)

The third addition is the returning of three values. The key and the socket are needed, for identifying what socket completion packets relate to and doing further operations on respectively. But as someone on Stack Overflow noted, I also need to hold onto a reference to the OVERLAPPED object used, otherwise Python may garbage collect it at some point, perhaps when the function exits.

return ovKey, _acceptSocket, _ovAccept

The completed function is as follows:

def CreateAcceptSocket():
    ret = WSASocket(AF_INET, SOCK_STREAM, IPPROTO_TCP, None, 0, WSA_FLAG_OVERLAPPED)
    if ret == INVALID_SOCKET:
        err = WSAGetLastError()
        closesocket(listenSocket)
        CloseHandle(hIOCP)
        WSACleanup()
        raise WinError(err)

    _acceptSocket = ret
    ovKey = CreateCompletionKey()

    dwBytesReceived = DWORD()
    _ovAccept = OVERLAPPED()

    ret = AcceptEx(listenSocket, _acceptSocket, outputBuffer, dwReceiveDataLength, dwLocalAddressLength, dwRemoteAddressLength, byref(dwBytesReceived), byref(_ovAccept))
    if ret == FALSE:
        err = WSAGetLastError()
        # The operation was successful and is currently in progress.  Ignore this error...
        if err != ERROR_IO_PENDING:
            closesocket(_acceptSocket)
            closesocket(listenSocket)
            CloseHandle(hIOCP)
            WSACleanup()
            raise WinError(err)

    # Bind the accept socket to the IO completion port.
    CreateIoCompletionPort(_acceptSocket, hIOCP, ovKey, NULL)
    return ovKey, _acceptSocket, _ovAccept

This function is incorporated into a another that waits for a connection to be assigned that new accept socket. Within it, an inner loop repeatedly polls for a completed packet rather than waiting indefinitely until one is received, for reasons that will be described later.

def Loop():
    global acceptSocket
    acceptKey, acceptSocket, ovAccept = CreateAcceptSocket()

    numberOfBytes = DWORD()
    completionKey = c_ulong()
    ovCompletedPtr = POINTER(OVERLAPPED)()

    while 1:
        ret = GetQueuedCompletionStatus(hIOCP, byref(numberOfBytes), byref(completionKey), byref(ovCompletedPtr), 500)
        if ret == FALSE:
            err = WSAGetLastError()
            if err == WAIT_TIMEOUT:
                continue
            Cleanup()
            raise WinError(err)
        break

    if completionKey.value != LISTEN_COMPLETION_KEY:
        Cleanup()
        raise Exception("Unexpected completion key", completionKey, "expected", LISTEN_COMPLETION_KEY)

    overlappedByKey[acceptKey] = acceptSocket

Now I can loop connection after connection as they come in. The only error that should propagate up to this logic is the keyboard interrupt exception that is generated when someone presses control-c, and it looks much better for it to be handled cleanly than displaying the exception.

try:
    while 1:
        Loop()
except KeyboardInterrupt:
    Cleanup()

Problems encountered

Python receives and handles signals, like the pressing of control-c, on its main thread. Because my script is not multithreaded it is running within that main thread and if external functions (like GetQueuedCompletionStatus) are called, the thread gets blocked until they return.

Initially I called GetQueuedCompletionStatus with an infinite timeout.

ret = GetQueuedCompletionStatus(hIOCP, byref(numberOfBytes), byref(completionKey), byref(ovCompletedPtr), INFINITE)

This meant that the Python main thread only had a chance to run when new connections were made, so pressing control-c in the console was not handled until that happened.

Because I want to be able to interrupt my script at any time, I changed the call to happen repeatedly with a short timeout, allowing signals were to be handled promptly.

ret = GetQueuedCompletionStatus(hIOCP, byref(numberOfBytes), byref(completionKey), byref(ovCompletedPtr), 500)

Another problem was my misinterpretation of how CreateIoCompletionPort is used. The completion key parameter is declared as a pointer.

HANDLE WINAPI CreateIoCompletionPort(
  __in      HANDLE FileHandle,
  __in_opt  HANDLE ExistingCompletionPort,
  __in      ULONG_PTR CompletionKey,
  __in      DWORD NumberOfConcurrentThreads
);

So I assumed that I needed to store the key I wanted to use and pass a pointer to it. The value passed is the completion key, not a pointer. I made this same mistake in the past when I wrote an overlapped file IO solution. In order to avoid confusing ctypes, I define it ignoring the pointer part.

CreateIoCompletionPort.argtypes = (HANDLE, HANDLE, c_ulong, DWORD)

Another misinterpretation I made was assuming that when I had an overlapped AcceptEx call in progress, the completion key I would receive when a connection was accepted would be the one for the relevant accept socket. It turns out that the key given for each accept packet completion is actually the one for the listen socket.

Next step

A good next step to achieve the goal of flesh out writing to sockets and handling remote disconnections is extending this into an echo server. So, that is what I will do. The function Loop really needs to be renamed to AcceptConnection along the way.

Next post: IOCP based sockets with ctypes in Python: 5
Script source code: 03 - Serving.py