Saturday 25 July 2009

IOCP based sockets with ctypes in Python: 4

Previous post: IOCP-based sockets with ctypes in Python: 3

The next thing I want to do with this, is to take what I have written so far and reshape it into a simple naive server. It will accept any connection made to it and then set aside that connection (and ignore it), ready to accept the next one.

I don't really need to care what IO completion packets get queued for me to receive on my IO completion port. But for the sake of extending my code base a little, I will set the completion key for each socket associated with the port anyway.

The original socket that listens for connections will get a unique value that has been set aside.

LISTEN_COMPLETION_KEY = 90L
CreateIoCompletionPort(listenSocket, hIOCP, LISTEN_COMPLETION_KEY, NULL)
And unique keys will be allocated on demand for the sockets of incoming connections which will get accepted.
currentCompletionKey = 100L

def CreateCompletionKey():
global currentCompletionKey
v = currentCompletionKey
currentCompletionKey += 1L
return v
Accepted connections that are set aside to be ignored are stored in a designated dictionary.
overlappedByKey = {}
Because I will need to accept an arbitrary number of incoming connections, rather than just the one the previous version of this script accepted, I've gathered the relevant parts of the script into a function with three additions.

The first addition is the allocation of a unique completion key for the new accept socket.
ovKey = CreateCompletionKey()
The second addition is setting this as the completion key for the socket when it is associated with the port.
CreateIoCompletionPort(_acceptSocket, hIOCP, ovKey, NULL)
The third addition is the returning of three values. The key and the socket are needed, for identifying what socket completion packets relate to and doing further operations on respectively. But as someone on Stack Overflow noted, I also need to hold onto a reference to the OVERLAPPED object used, otherwise Python may garbage collect it at some point, perhaps when the function exits.
return ovKey, _acceptSocket, _ovAccept
The completed function is as follows:
def CreateAcceptSocket():
ret = WSASocket(AF_INET, SOCK_STREAM, IPPROTO_TCP, None, 0, WSA_FLAG_OVERLAPPED)
if ret == INVALID_SOCKET:
err = WSAGetLastError()
closesocket(listenSocket)
CloseHandle(hIOCP)
WSACleanup()
raise WinError(err)

_acceptSocket = ret
ovKey = CreateCompletionKey()

dwBytesReceived = DWORD()
_ovAccept = OVERLAPPED()

ret = AcceptEx(listenSocket, _acceptSocket, outputBuffer, dwReceiveDataLength, dwLocalAddressLength, dwRemoteAddressLength, byref(dwBytesReceived), byref(_ovAccept))
if ret == FALSE:
err = WSAGetLastError()
# The operation was successful and is currently in progress. Ignore this error...
if err != ERROR_IO_PENDING:
closesocket(_acceptSocket)
closesocket(listenSocket)
CloseHandle(hIOCP)
WSACleanup()
raise WinError(err)

# Bind the accept socket to the IO completion port.
CreateIoCompletionPort(_acceptSocket, hIOCP, ovKey, NULL)
return ovKey, _acceptSocket, _ovAccept
This function is incorporated into a another that waits for a connection to be assigned that new accept socket. Within it, an inner loop repeatedly polls for a completed packet rather than waiting indefinitely until one is received, for reasons that will be described later.
def Loop():
global acceptSocket
acceptKey, acceptSocket, ovAccept = CreateAcceptSocket()

numberOfBytes = DWORD()
completionKey = c_ulong()
ovCompletedPtr = POINTER(OVERLAPPED)()

while 1:
ret = GetQueuedCompletionStatus(hIOCP, byref(numberOfBytes), byref(completionKey), byref(ovCompletedPtr), 500)
if ret == FALSE:
err = WSAGetLastError()
if err == WAIT_TIMEOUT:
continue
Cleanup()
raise WinError(err)
break

if completionKey.value != LISTEN_COMPLETION_KEY:
Cleanup()
raise Exception("Unexpected completion key", completionKey, "expected", LISTEN_COMPLETION_KEY)

overlappedByKey[acceptKey] = acceptSocket
Now I can loop connection after connection as they come in. The only error that should propagate up to this logic is the keyboard interrupt exception that is generated when someone presses control-c, and it looks much better for it to be handled cleanly than displaying the exception.
try:
while 1:
Loop()
except KeyboardInterrupt:
Cleanup()
Problems encountered

Python receives and handles signals, like the pressing of control-c, on its main thread. Because my script is not multithreaded it is running within that main thread and if external functions (like GetQueuedCompletionStatus) are called, the thread gets blocked until they return.

Initially I called GetQueuedCompletionStatus with an infinite timeout.
ret = GetQueuedCompletionStatus(hIOCP, byref(numberOfBytes), byref(completionKey), byref(ovCompletedPtr), INFINITE)
This meant that the Python main thread only had a chance to run when new connections were made, so pressing control-c in the console was not handled until that happened.

Because I want to be able to interrupt my script at any time, I changed the call to happen repeatedly with a short timeout, allowing signals were to be handled promptly.
ret = GetQueuedCompletionStatus(hIOCP, byref(numberOfBytes), byref(completionKey), byref(ovCompletedPtr), 500)
Another problem was my misinterpretation of how CreateIoCompletionPort is used. The completion key parameter is declared as a pointer.
HANDLE WINAPI CreateIoCompletionPort(
__in HANDLE FileHandle,
__in_opt HANDLE ExistingCompletionPort,
__in ULONG_PTR CompletionKey,
__in DWORD NumberOfConcurrentThreads
);
So I assumed that I needed to store the key I wanted to use and pass a pointer to it. The value passed is the completion key, not a pointer. I made this same mistake in the past when I wrote an overlapped file IO solution. In order to avoid confusing ctypes, I define it ignoring the pointer part.
CreateIoCompletionPort.argtypes = (HANDLE, HANDLE, c_ulong, DWORD)
Another misinterpretation I made was assuming that when I had an overlapped AcceptEx call in progress, the completion key I would receive when a connection was accepted would be the one for the relevant accept socket. It turns out that the key given for each accept packet completion is actually the one for the listen socket.

Next step

A good next step to achieve the goal of flesh out writing to sockets and handling remote disconnections is extending this into an echo server. So, that is what I will do. The function Loop really needs to be renamed to AcceptConnection along the way.

Next post: IOCP based sockets with ctypes in Python: 5
Script source code: 03 - Serving.py

Friday 24 July 2009

IOCP based sockets with ctypes in Python: 3

Previous post: IOCP-based sockets with ctypes in Python: 2

The listen socket needs to be associated with an IO completion port. First I need to create one using CreateIoCompletionPort.

HANDLE WINAPI CreateIoCompletionPort(
__in HANDLE FileHandle,
__in_opt HANDLE ExistingCompletionPort,
__in ULONG_PTR CompletionKey,
__in DWORD NumberOfConcurrentThreads
);
Defined using ctypes in this way.
CreateIoCompletionPort = windll.kernel32.CreateIoCompletionPort
CreateIoCompletionPort.argtypes = (HANDLE, HANDLE, c_ulong, DWORD)
CreateIoCompletionPort.restype = HANDLE
Now I can create the IO completion port.
NULL = c_ulong()
INVALID_HANDLE_VALUE = HANDLE(-1)
NULL_HANDLE = HANDLE(0)

hIOCP = CreateIoCompletionPort(INVALID_HANDLE_VALUE, NULL_HANDLE, NULL, NULL)
if hIOCP == 0:
err = WSAGetLastError()
closesocket(listenSocket)
WSACleanup()
raise WinError(err)
The listen socket I created needs to be associated with that IO completion port.
CreateIoCompletionPort(listenSocket, hIOCP, NULL, NULL)
The created IO completion port needs to be closed at some point (when all the sockets associated with it have been closed) with CloseHandle.
BOOL WINAPI CloseHandle(
__in HANDLE hObject
);
Defined with ctypes.
CloseHandle = windll.kernel32.CloseHandle
CloseHandle.argtypes = (HANDLE,)
CloseHandle.restype = BOOL
Now I can add the AcceptEx definitions I previously wrote.
class _US(Structure):
_fields_ = [
("Offset", DWORD),
("OffsetHigh", DWORD),
]

class _U(Union):
_fields_ = [
("s", _US),
("Pointer", c_void_p),
]
_anonymous_ = ("s",)

class OVERLAPPED(Structure):
_fields_ = [
("Internal", POINTER(ULONG)),
("InternalHigh", POINTER(ULONG)),
("u", _U),
("hEvent", HANDLE),
# Custom fields.
("channel", py_object),
]
_anonymous_ = ("u",)

AcceptEx = windll.Mswsock.AcceptEx
AcceptEx.argtypes = (SOCKET, SOCKET, c_void_p, DWORD, DWORD, DWORD, POINTER(DWORD), POINTER(OVERLAPPED))
AcceptEx.restype = BOOL
In order to call AcceptEx, a socket needs to be precreated for the incoming connection to be assigned to.
ret = WSASocket(AF_INET, SOCK_STREAM, IPPROTO_TCP, None, 0, WSA_FLAG_OVERLAPPED)
if ret == INVALID_SOCKET:
err = WSAGetLastError()
closesocket(listenSocket)
CloseHandle(hIOCP)
WSACleanup()
raise WinError(err)

acceptSocket = ret
And then the AcceptEx call can be made.
FALSE = 0
ERROR_IO_PENDING = 997

dwReceiveDataLength = 0
dwLocalAddressLength = sizeof(sockaddr_in)/2
dwRemoteAddressLength = sizeof(sockaddr_in)/2
outputBuffer = create_string_buffer(2 * sizeof(sockaddr_in))
dwBytesReceived = DWORD()
ovAccept = OVERLAPPED()

ret = AcceptEx(listenSocket, acceptSocket, outputBuffer, dwReceiveDataLength, dwLocalAddressLength, dwRemoteAddressLength, byref(dwBytesReceived), byref(ovAccept))
if ret == FALSE:
err = WSAGetLastError()
# The operation was successful and is currently in progress. Ignore this error...
if err != ERROR_IO_PENDING:
closesocket(acceptSocket)
closesocket(listenSocket)
CloseHandle(hIOCP)
WSACleanup()
raise WinError(err)
The accept socket also needs to be associated with the IO completion port.
CreateIoCompletionPort(acceptSocket, hIOCP, NULL, NULL)
Now incoming connections should be queued on the IO completion port for me to poll using GetQueuedCompletionStatus.
BOOL WINAPI GetQueuedCompletionStatus(
__in HANDLE CompletionPort,
__out LPDWORD lpNumberOfBytes,
__out PULONG_PTR lpCompletionKey,
__out LPOVERLAPPED *lpOverlapped,
__in DWORD dwMilliseconds
);
Which specified with ctypes is..
GetQueuedCompletionStatus = windll.kernel32.GetQueuedCompletionStatus
GetQueuedCompletionStatus.argtypes = (HANDLE, POINTER(DWORD), POINTER(c_ulong), POINTER(POINTER(OVERLAPPED)), DWORD)
GetQueuedCompletionStatus.restype = BOOL
For the purposes of writing a test script, I'm going to wait for 10 seconds for an incoming connection.
numberOfBytes = DWORD()
completionKey = c_ulong()
ovCompletedPtr = POINTER(OVERLAPPED)()

ret = GetQueuedCompletionStatus(hIOCP, byref(numberOfBytes), byref(completionKey), byref(ovCompletedPtr), 10000)
if ret == FALSE:
err = WSAGetLastError()
closesocket(acceptSocket)
closesocket(listenSocket)
CloseHandle(hIOCP)
WSACleanup()
raise WinError(err)
Now to run the script. I opened the listening socket on localhost port 10101, so telnet localhost 10101 will make a connection. My first attempt is going to let the GetQueuedCompletionStatus time out.
Traceback (most recent call last):
File "02 - Accepting.py", line 264, in
WSACleanup()
WindowsError: [Error 258] The wait operation timed out.
That is to be expected. Now to rerun the script, but do the telnet before the GetQueuedCompletionStatus call times out.
Traceback (most recent call last):
File "02 - Accepting.py", line 265, in
raise WinError(err)
WindowsError: [Error 122] The data area passed to a system call is too small.
Oops. Rereading the AcceptEx documentation, it suggests that the space needed for an address should be "at least 16 bytes more than the maximum address length for the transport protocol in use". Ah, my address length calculation was just wrong anyway, regardless of the lack of extra 16 bytes. The following lines are changed:
dwReceiveDataLength = 0
dwLocalAddressLength = sizeof(sockaddr_in) + 16
dwRemoteAddressLength = sizeof(sockaddr_in) + 16
outputBuffer = create_string_buffer(dwReceiveDataLength + dwLocalAddressLength + dwRemoteAddressLength)
I also need to be able to at least naively determine whether the overlapped object I get is one I created. The only one I created was ovAccept, so I misuse my custom channel attribute to store a string.
ovAccept = OVERLAPPED()
ovAccept.channel = "accept"
And add some extra output to the script on success.
print "SUCCESS"
print ovCompletedPtr.contents.channel
Now to paste the script into a python console and see what the output is.
>>> print "SUCCESS"
SUCCESS
>>> print ovCompletedPtr.contents.channel
accept
Now I'll send a message to the accepted connection before disconnecting them. WSASend looks like the right function to call.
int WSASend(
__in SOCKET s,
__in LPWSABUF lpBuffers,
__in DWORD dwBufferCount,
__out LPDWORD lpNumberOfBytesSent,
__in DWORD dwFlags,
__in LPWSAOVERLAPPED lpOverlapped,
__in LPWSAOVERLAPPED_COMPLETION_ROUTINE lpCompletionRoutine
);
And the new structure WSABUF.
struct WSABUF {
u_long len;
char FAR *buf;
}
Which gives the following ctypes definitions.
class WSABUF(Structure):
_fields_ = [
("len", c_ulong),
("buf", c_char_p),
]

WSASend = windll.Ws2_32.WSASend
WSASend.argtypes = (SOCKET, POINTER(WSABUF), DWORD, POINTER(DWORD), DWORD, POINTER(OVERLAPPED), c_void_p)
WSASend.restype = c_int
And now to send the message on the accepted connection.
msg = "Disconnecting you.."
sendBuffer = (WSABUF * 1)()
sendBuffer[0].buf = msg
sendBuffer[0].len = len(msg)
bytesSent = DWORD()
ovSend = OVERLAPPED()
ovSend.channel = "send"
ret = WSASend(acceptSocket, cast(sendBuffer, POINTER(WSABUF)), 1, byref(bytesSent), 0, byref(ovSend), 0)
if ret != 0:
err = WSAGetLastError()
# The operation was successful and is currently in progress. Ignore this error...
if err != ERROR_IO_PENDING:
closesocket(acceptSocket)
closesocket(listenSocket)
CloseHandle(hIOCP)
WSACleanup()
raise WinError(err)
I'll just repeat the earlier poll for the sake of simplicity.
ret = GetQueuedCompletionStatus(hIOCP, byref(numberOfBytes), byref(completionKey), byref(ovCompletedPtr), 10000)
if ret == FALSE:
err = WSAGetLastError()
closesocket(acceptSocket)
closesocket(listenSocket)
CloseHandle(hIOCP)
WSACleanup()
raise WinError(err)
The received overlapped object should be the one for the send. Print some output to confirm I got the right one.
if ovCompletedPtr.contents.channel == "send":
print "SUCCESS"
else:
print "FAILURE: Unrecognised overlapped object"
At this point the message should have been sent to the accepted connection so I'll now disconnect them and exit.
closesocket(acceptSocket)
closesocket(listenSocket)
CloseHandle(hIOCP)
WSACleanup()
Pasting the script into an open Python console gives the following output:
SUCCESS
And the console where I telneted sees the following output:
Disconnecting you..

Connection to host lost.
That's the initial goal achieved. The final goal is to have low-level IOCP networking code which can be wrapped using Stackless Python with a socket module interface so that it can be used in place of stacklesssocket.py.

Next post: IOCP-based sockets with ctypes in Python: 4
Script source code: 02 - Accepting.py

Edited at 6:25PM on 2009-07-25:

Was:
CreateIoCompletionPort = windll.kernel32.CreateIoCompletionPort
CreateIoCompletionPort.argtypes = (HANDLE, HANDLE, POINTER(c_ulong), DWORD)
CreateIoCompletionPort.restype = HANDLE
Changed to:
CreateIoCompletionPort = windll.kernel32.CreateIoCompletionPort
CreateIoCompletionPort.argtypes = (HANDLE, HANDLE, c_ulong, DWORD)
CreateIoCompletionPort.restype = HANDLE
The pointer while correct in matching the C++ definition had confused me and was leading me to pass in a pointer to a value rather than the value itself, in later code based on this.

Thursday 23 July 2009

Curing bacon: Day 1

My father worked as a butcher when he was younger, and over the years he kept preparing meat for himself. I remember going to my grandfather's and making mince meat by putting lumps of meat through a mincer. Making sausages in my grandfather's garage, stuffing sausage mix into casings and smoking them. Smoking salmon in a home-made smoker. All very useful things to be able to do for yourself.

So, since he has a leg of pork and I was helping make it into bacon, this time I decided to photograph the process so I can better remember how to do it.

The first step is to create a 55% ratio salt to sugar mix for rubbing into the leg of pork.

2009-07-23 - Curing bacon - 01 - Salt and sugar

Ensure the salt and sugar combination is thoroughly mixed.

2009-07-23 - Curing bacon - 02

Cut the leg of pork open. In this case, the pork is thin enough that the cure should penetrate it, so stuff doesn't need to be injected into it as it would otherwise.

2009-07-23 - Curing bacon - 03 - Leg of pork

Sprinkle some salt in the curing container.

2009-07-23 - Curing bacon - 04

Rub the mix into the first side of the pork. Ensure the mix is rubbed thoroughly into all the nooks and crannies.

2009-07-23 - Curing bacon - 06

Flip the pork over and rub the mix into the other side.

2009-07-23 - Curing bacon - 08 - Other side

Throw the pork into the curing container.

2009-07-23 - Curing bacon - 10

Sprinkle some more mix over the top.

2009-07-23 - Curing bacon - 11

Throw the curing container into the fridge.

2009-07-23 - Curing bacon - 14 - Sit overnight

Leave it until tomorrow when the next steps occur. Something about how saltpeter is now illegal and they have better things to use these days. Also it needs to be turned each day.

Next post: Curing bacon: Day 2

Wednesday 22 July 2009

IOCP based sockets with ctypes in Python: 2

Previous post: IOCP-based sockets with ctypes in Python: 1

My current goal is to get basic asynchronous socket IO working using Winsock via ctypes. So this post documents the steps I have made today, in defining Windows functions in an interactive Python console and using them as I go so that I have a working tested result.

I have a standard set of import statements I have used to get everything I need directly available from ctypes.

from ctypes import windll, pythonapi
from ctypes import c_bool, c_char, c_ubyte, c_int, c_uint, c_short, c_ushort, c_long, c_ulong, c_void_p, byref, c_char_p, Structure, Union, py_object, POINTER, pointer
from ctypes.wintypes import HANDLE, ULONG, DWORD, BOOL, LPCSTR, LPCWSTR, WinError, WORD
Before Winsock can be used, a call to WSAStartup needs to be made to initiate it for the current process.
int WSAStartup(
__in WORD wVersionRequested,
__out LPWSADATA lpWSAData
);
And before this function can be defined, the WSADATA structure it references needs to have already been defined. And the definition of WSADATA requires the values of the constants WSADESCRIPTION_LEN and WSASYS_STATUS_LEN which need to be searched for in Windows header files.
WSADESCRIPTION_LEN = 256
WSASYS_STATUS_LEN = 128

class WSADATA(Structure):
_fields_ = [
("wVersion", WORD),
("wHighVersion", WORD),
("szDescription", c_char * (WSADESCRIPTION_LEN+1)),
("szSystemStatus", c_char * (WSASYS_STATUS_LEN+1)),
("iMaxSockets", c_ushort),
("iMaxUdpDg", c_ushort),
("lpVendorInfo", c_char_p),
]
At this point, the prerequisites are present so that the WSAStartup function can also be defined.
WSAStartup = windll.Ws2_32.WSAStartup
WSAStartup.argtypes = (WORD, POINTER(WSADATA))
WSAStartup.restype = c_int
With WSAStartup defined, it can now be called.
def MAKEWORD(bLow, bHigh):
return (bHigh << 8) + bLow

wsaData = WSADATA()
LP_WSADATA = POINTER(WSADATA)
ret = WSAStartup(MAKEWORD(2, 2), LP_WSADATA(wsaData))
if ret != 0:
raise WinError(ret)
This works in the console and I can move onto creating sockets.

The first step is to define the WSASocket function.
SOCKET WSASocket(
__in int af,
__in int type,
__in int protocol,
__in LPWSAPROTOCOL_INFO lpProtocolInfo,
__in GROUP g,
__in DWORD dwFlags
);
This is as straightforward as it gets, once the datatype of GROUP is tracked down in the Windows header files. WSAPROTOCOL can be dismissed with a NULL value. Defining socket would be easier but it doesn't allow me to pass in a flag indicating the created socket should work in an overlapped manner.
GROUP = c_uint
SOCKET = c_uint

WSASocket = windll.Ws2_32.WSASocketA
WSASocket.argtypes = (c_int, c_int, c_int, c_void_p, GROUP, DWORD)
WSASocket.restype = SOCKET
And after locating the constants needed for the arguments from the Windows header files, I can now call the function.
AF_INET = 2
SOCK_STREAM = 1
IPPROTO_TCP = 6
WSA_FLAG_OVERLAPPED = 0x01
INVALID_SOCKET = ~0

ret = WSASocket(AF_INET, SOCK_STREAM, IPPROTO_TCP, None, 0, WSA_FLAG_OVERLAPPED)
if ret == INVALID_SOCKET:
raise WinError()
I'll be reusing 'ret' so I'll store it in another variable.
listenSocket = ret
The next step will be to listen on a socket and accept incoming connections on it, then to read data from other sockets whose connections are accepted. So now I need to call listen on the socket I have just created. So to define it..
int listen(
__in SOCKET s,
__in int backlog
);
listen = windll.Ws2_32.listen
listen.argtypes = (SOCKET, c_int)
listen.restype = BOOL
And to call it..
SOMAXCONN = 0x7fffffff

ret = listen(listenSocket, SOMAXCONN)
if ret != 0:
raise WinError()
Oops.
WindowsError: [Error 10022] An invalid argument was supplied.
listenSocket should be a valid socket, so back to the documentation.
A descriptor identifying a bound, unconnected socket.
Oh, that's right, I need to bind it first.
int bind(
__in SOCKET s,
__in const struct sockaddr *name,
__in int namelen
);
I don't have a definition for sockaddr, so that needs to be made first.
struct in_addr {
union {
struct {
u_char s_b1,s_b2,s_b3,s_b4;
} S_un_b;
struct {
u_short s_w1,s_w2;
} S_un_w;
u_long S_addr;
} S_un;
}

struct sockaddr_in {
short sin_family;
u_short sin_port;
struct in_addr sin_addr;
char sin_zero[8];
};
And the corresponding Python definition.
class _UN_b(Structure):
_fields_ = [
("s_b1", c_ubyte),
("s_b2", c_ubyte),
("s_b3", c_ubyte),
("s_b4", c_ubyte),
]

class _UN_w(Structure):
_fields_ = [
("s_w1", c_ushort),
("s_w2", c_ushort),
]

class _UN(Structure):
_fields_ = [
("S_un_b", _UN_b),
("S_un_w", _UN_w),
("S_addr", c_ulong),
]

class in_addr(Union):
_fields_ = [
("S_un", _UN),
]
_anonymous_ = ("S_un",)

class sockaddr_in(Structure):
_fields_ = [
("sin_family", c_short),
("sin_port", c_ushort),
("sin_addr", in_addr),
("szDescription", c_char * 8),
]

sockaddr_inp = POINTER(sockaddr_in)
The definition seems to work, now to define bind and try it out.
bind = windll.Ws2_32.bind
bind.argtypes = (SOCKET, POINTER(sockaddr_in), c_int)
bind.restype = c_int
However, the values for sin_addr and sin_port need to be calculated and adjusted to suit the relevant data structures in sockaddr_in.
u_short WSAAPI htons(
__in u_short hostshort
);

struct hostent* FAR gethostbyname(
__in const char *name
);

struct hostent {
char FAR * h_name;
char FAR FAR **h_aliases;
short h_addrtype;
short h_length;
char FAR FAR **h_addr_list;
}
Which require the following definitions.
class hostent(Structure):
_fields_ = [
("h_name", c_charp),
("h_aliases", POINTER(c_charp)),
("h_addrtype", c_short),
("h_length", c_short),
("h_addr_list", POINTER(c_charp)),
]

hostentp = POINTER(hostent)

gethostbyname = windll.Ws2_32.gethostbyname
gethostbyname.argtypes = (c_char_p,)
gethostbyname.restype = hostentp

inet_addr = windll.Ws2_32.inet_addr
inet_addr.argtypes = (c_char_p,)
inet_addr.restype = c_ulong

inet_ntoa = windll.Ws2_32.inet_ntoa
inet_ntoa.argtypes = (in_addr,)
inet_ntoa.restype = c_char_p

htons = windll.Ws2_32.htons
htons.argtypes = (c_ushort,)
htons.restype = c_ushort
This should cover it, and the calling of bind should now be possible. A sockaddr_in structure needs to be passed to it, so the data needed to populate that structure needs to be located first.
hostdata = gethostbyname("")
ip = inet_ntoa(cast(hostdata.contents.h_addr_list, POINTER(in_addr)).contents)
Something is going wrong here.
ValueError: Procedure probably called with too many arguments (8 bytes in excess)
Some testing..
ia = cast(hostdata.contents.h_addr_list, POINTER(in_addr)).contents
print sizeof(ia)
print sizeof(in_addr)
This gives the following output.
12
12
So, whatever is going wrong is not obvious. This is already taking long enough, so I will stick with localhost as the ip address for now.
port = 10101
ip = "127.0.0.1"

sa = sockaddr_in()
sa.sin_family = AF_INET
sa.sin_addr.S_addr = inet_addr(ip)
sa.sin_port = htons(port)

SOCKET_ERROR = -1

ret = bind(listenSocket, sockaddr_inp(sa), sizeof(sa))
if ret == SOCKET_ERROR:
raise WinError()
This works fine, now to try listening again.
ret = listen(listenSocket, SOMAXCONN)
if ret != 0:
raise WinError()
And this now works, Windows pops up the standard dialog window mentioning that Python has been blocked from accepting incoming connections and asks if I want to allow it. After allowing Python to accept incoming connections, port 10101 can now be telneted to, which demonstrates that the socket is now actually be listened to. The next step is to accept connections, which can wait until whenever I get the time to finish that code.

Next post: IOCP-based sockets with ctypes in Python: 3
Script source code: 01 - Listening.py

IOCP based sockets with ctypes in Python: 1

As I was reminded of recently, select in the Python library only handles 512 sockets at a time. This has given me the interest to spend a little time writing socket IO completion port support using ctypes.

In order to write code that uses C-level DLL functionality with ctypes, it requires that all structures, constants and functions are redefined at the Python level. This can be a somewhat tedious process, slowing down the progress that an interactive prompt otherwise allows. Developing an extension module in C or C++ would probably be faster, but then tradeoffs would be made in terms of framework structure and other inherent costs.

Defining a function is straightforward, given the obstacle of defining required but currently undefined structures. For instance, this is the prototype for AcceptEx:

BOOL AcceptEx(
__in SOCKET sListenSocket,
__in SOCKET sAcceptSocket,
__in PVOID lpOutputBuffer,
__in DWORD dwReceiveDataLength,
__in DWORD dwLocalAddressLength,
__in DWORD dwRemoteAddressLength,
__out LPDWORD lpdwBytesReceived,
__in LPOVERLAPPED lpOverlapped
);
In order to specify this function definition with ctypes, the OVERLAPPED structure needs to be defined first in this way.
class _US(Structure):
_fields_ = [
("Offset", DWORD),
("OffsetHigh", DWORD),
]

class _U(Union):
_fields_ = [
("s", _US),
("Pointer", c_void_p),
]

_anonymous_ = ("s",)

class OVERLAPPED(Structure):
_fields_ = [
("Internal", POINTER(ULONG)),
("InternalHigh", POINTER(ULONG)),

("u", _U),

("hEvent", HANDLE),

# Custom fields.
("channel", py_object),
]

_anonymous_ = ("u",)
I wrote that several years ago, and like most ctypes-based code, it took a reasonable amount of time. Interestingly, I've seen at least one other project has copied it verbatim (minus the Stackless 'channel' field) in order to avoid this effort. Given a definition for this structure, the function definition can also be written. The first step is to get a reference to the function from the DLL it is implemented within.
AcceptEx = windll.Mswsock.AcceptEx
Next the types of arguments it takes are specified.
AcceptEx.argtypes = (SOCKET, SOCKET, c_void_p, DWORD, DWORD, DWORD, POINTER(DWORD), POINTER(OVERLAPPED))
And finally, the type of the value it returns.
AcceptEx.restype = BOOL
At this point, AcceptEx is callable and can be used as needed. But to use it requires that the functions which create sockets also be defined, and whatever other functions you may need.

I often find myself writing all the definitions consecutively in a text editor. This is long and time consuming and you have no idea whether the code actually works. It is easy to make trivial mistakes like specifying the wrong DLL for the function to be obtained from. A much more rewarding approach in my experience is to write the code line by line in a interactive Python console, and once each aspect works, compose them into a script in a text editor. This script can be pasted into a new console when you resume the effort, next you find the time and interest.