IOCP based sockets with ctypes in Python: 2

Previous post: IOCP-based sockets with ctypes in Python: 1

My current goal is to get basic asynchronous socket IO working using Winsock via ctypes. So this post documents the steps I have made today, in defining Windows functions in an interactive Python console and using them as I go so that I have a working tested result.

I have a standard set of import statements I have used to get everything I need directly available from ctypes.

from ctypes import windll, pythonapi
from ctypes import c_bool, c_char, c_ubyte, c_int, c_uint, c_short, c_ushort, c_long, c_ulong, c_void_p, byref, c_char_p, Structure, Union, py_object, POINTER, pointer
from ctypes.wintypes import HANDLE, ULONG, DWORD, BOOL, LPCSTR, LPCWSTR, WinError, WORD

Before Winsock can be used, a call to WSAStartup needs to be made to initiate it for the current process.

int WSAStartup(
  __in   WORD wVersionRequested,
  __out  LPWSADATA lpWSAData
);

And before this function can be defined, the WSADATA structure it references needs to have already been defined. And the definition of WSADATA requires the values of the constants WSADESCRIPTION_LEN and WSASYS_STATUS_LEN which need to be searched for in Windows header files.

WSADESCRIPTION_LEN = 256
WSASYS_STATUS_LEN = 128

class WSADATA(Structure):
    _fields_ = [
        ("wVersion",        WORD),
        ("wHighVersion",    WORD),
        ("szDescription",   c_char * (WSADESCRIPTION_LEN+1)),
        ("szSystemStatus",  c_char * (WSASYS_STATUS_LEN+1)),
        ("iMaxSockets",     c_ushort),
        ("iMaxUdpDg",       c_ushort),
        ("lpVendorInfo",    c_char_p),
    ]

At this point, the prerequisites are present so that the WSAStartup function can also be defined.

WSAStartup = windll.Ws2_32.WSAStartup
WSAStartup.argtypes = (WORD, POINTER(WSADATA))
WSAStartup.restype = c_int

With WSAStartup defined, it can now be called.

def MAKEWORD(bLow, bHigh):
    return (bHigh << 8) + bLow

wsaData = WSADATA()
LP_WSADATA = POINTER(WSADATA)
ret = WSAStartup(MAKEWORD(2, 2), LP_WSADATA(wsaData))
if ret != 0:
    raise WinError(ret)

This works in the console and I can move onto creating sockets.

The first step is to define the WSASocket function.

SOCKET WSASocket(
  __in  int af,
  __in  int type,
  __in  int protocol,
  __in  LPWSAPROTOCOL_INFO lpProtocolInfo,
  __in  GROUP g,
  __in  DWORD dwFlags
);

This is as straightforward as it gets, once the datatype of GROUP is tracked down in the Windows header files. WSAPROTOCOL can be dismissed with a NULL value. Defining socket would be easier but it doesn't allow me to pass in a flag indicating the created socket should work in an overlapped manner.

GROUP = c_uint
SOCKET = c_uint

WSASocket = windll.Ws2_32.WSASocketA
WSASocket.argtypes = (c_int, c_int, c_int, c_void_p, GROUP, DWORD)
WSASocket.restype = SOCKET

And after locating the constants needed for the arguments from the Windows header files, I can now call the function.

AF_INET = 2
SOCK_STREAM = 1
IPPROTO_TCP = 6
WSA_FLAG_OVERLAPPED = 0x01
INVALID_SOCKET = ~0

ret = WSASocket(AF_INET, SOCK_STREAM, IPPROTO_TCP, None, 0, WSA_FLAG_OVERLAPPED)
if ret == INVALID_SOCKET:
    raise WinError()

I'll be reusing 'ret' so I'll store it in another variable.

listenSocket = ret

The next step will be to listen on a socket and accept incoming connections on it, then to read data from other sockets whose connections are accepted. So now I need to call listen on the socket I have just created. So to define it..

int listen(
  __in  SOCKET s,
  __in  int backlog
);

listen = windll.Ws2_32.listen
listen.argtypes = (SOCKET, c_int)
listen.restype = BOOL

And to call it..

SOMAXCONN = 0x7fffffff

ret = listen(listenSocket, SOMAXCONN)
if ret != 0:
    raise WinError()

Oops.

WindowsError: [Error 10022] An invalid argument was supplied.

listenSocket should be a valid socket, so back to the documentation.

A descriptor identifying a bound, unconnected socket.

Oh, that's right, I need to bind it first.

int bind(
  __in  SOCKET s,
  __in  const struct sockaddr *name,
  __in  int namelen
);

I don't have a definition for sockaddr, so that needs to be made first.

struct in_addr {
  union {
    struct {
      u_char s_b1,s_b2,s_b3,s_b4;
    } S_un_b;
    struct {
      u_short s_w1,s_w2;
    } S_un_w;
    u_long S_addr;
  } S_un;
}

struct sockaddr_in {
        short   sin_family;
        u_short sin_port;
        struct  in_addr sin_addr;
        char    sin_zero[8];
};

And the corresponding Python definition.

class _UN_b(Structure):
    _fields_ = [
        ("s_b1",            c_ubyte),
        ("s_b2",            c_ubyte),
        ("s_b3",            c_ubyte),
        ("s_b4",            c_ubyte),
    ]

class _UN_w(Structure):
    _fields_ = [
        ("s_w1",            c_ushort),
        ("s_w2",            c_ushort),
    ]

class _UN(Structure):
    _fields_ = [
        ("S_un_b",          _UN_b),
        ("S_un_w",          _UN_w),
        ("S_addr",          c_ulong),
    ]

class in_addr(Union):
    _fields_ = [
        ("S_un",            _UN),
    ]
    _anonymous_ = ("S_un",)

class sockaddr_in(Structure):
    _fields_ = [
        ("sin_family",      c_short),
        ("sin_port",        c_ushort),
        ("sin_addr",        in_addr),
        ("szDescription",   c_char * 8),
    ]

sockaddr_inp = POINTER(sockaddr_in)

The definition seems to work, now to define bind and try it out.

bind = windll.Ws2_32.bind
bind.argtypes = (SOCKET, POINTER(sockaddr_in), c_int)
bind.restype = c_int

However, the values for sin_addr and sin_port need to be calculated and adjusted to suit the relevant data structures in sockaddr_in.

u_short WSAAPI htons(
  __in  u_short hostshort
);

struct hostent* FAR gethostbyname(
  __in  const char *name
);

struct hostent {
  char FAR *    h_name;
  char FAR  FAR **h_aliases;
  short         h_addrtype;
  short         h_length;
  char FAR  FAR **h_addr_list;
}

Which require the following definitions.

class hostent(Structure):
    _fields_ = [
        ("h_name",          c_charp),
        ("h_aliases",       POINTER(c_charp)),
        ("h_addrtype",      c_short),
        ("h_length",        c_short),
        ("h_addr_list",     POINTER(c_charp)),
    ]

hostentp = POINTER(hostent)

gethostbyname = windll.Ws2_32.gethostbyname
gethostbyname.argtypes = (c_char_p,)
gethostbyname.restype = hostentp

inet_addr = windll.Ws2_32.inet_addr
inet_addr.argtypes = (c_char_p,)
inet_addr.restype = c_ulong

inet_ntoa = windll.Ws2_32.inet_ntoa
inet_ntoa.argtypes = (in_addr,)
inet_ntoa.restype = c_char_p

htons = windll.Ws2_32.htons
htons.argtypes = (c_ushort,)
htons.restype = c_ushort

This should cover it, and the calling of bind should now be possible. A sockaddr_in structure needs to be passed to it, so the data needed to populate that structure needs to be located first.

hostdata = gethostbyname("")
ip = inet_ntoa(cast(hostdata.contents.h_addr_list, POINTER(in_addr)).contents)

Something is going wrong here.

ValueError: Procedure probably called with too many arguments (8 bytes in excess)

Some testing..

ia = cast(hostdata.contents.h_addr_list, POINTER(in_addr)).contents
print sizeof(ia)
print sizeof(in_addr)

This gives the following output.

12
12

So, whatever is going wrong is not obvious. This is already taking long enough, so I will stick with localhost as the ip address for now.

port = 10101
ip = "127.0.0.1"

sa = sockaddr_in()
sa.sin_family = AF_INET
sa.sin_addr.S_addr = inet_addr(ip)
sa.sin_port = htons(port)

SOCKET_ERROR = -1

ret = bind(listenSocket, sockaddr_inp(sa), sizeof(sa))
if ret == SOCKET_ERROR:
    raise WinError()

This works fine, now to try listening again.

ret = listen(listenSocket, SOMAXCONN)
if ret != 0:
    raise WinError()

And this now works, Windows pops up the standard dialog window mentioning that Python has been blocked from accepting incoming connections and asks if I want to allow it. After allowing Python to accept incoming connections, port 10101 can now be telneted to, which demonstrates that the socket is now actually be listened to. The next step is to accept connections, which can wait until whenever I get the time to finish that code.

Next post: IOCP-based sockets with ctypes in Python: 3
Script source code: 01 - Listening.py