Reconsidering thread blocking WRT microthreads

I meant to spend a good part of the day doing the long overdue Stackless 2.6.6 merge, but having forgotten my SSH key, I found myself at work on a public holiday with time set aside for it.. but with an extra obstacle in the way. So I moved onto another project -- looking into the reports that there are problems with my stackless socket module, and the xmlrpc standard library. I was however unable to reproduce the problems, and spent the time cleaning up the mechanisms that allow me to run the standard library unit tests against my module.

The bulk of these mechanisms are monkey-patchings of thread blocking operations. When any of these calls (time.sleep, select.select, ..) are made, they would block the Stackless scheduler running on that thread. So the general brute force solution is to make any of these calls simply block the current microthread, while the actual call is made in another thread. This is rather naive, but with a bit of special casing for hiding harmless errors caused by race conditions, for the purposes of running suites of unit tests it works out fine.

However, the more I work on something like this, the more uneasy I feel with the long held ideal of having a suite of these monkey-patching mechanisms which users can just install to make any use of the standard or third party libraries just work. It's not that it wouldn't just work for the most part, look at stackless socket, it has been a cobbled together mess for many years and it was more than enough for almost anyone who used it. It's that it is actually important for people writing code to know where the points their microthreads block are. Whether these are safe voluntary yields to the scheduler, where state may change before the microthread gets scheduled again and resumes. Or whether it is unsafe blocking of the thread, which may also block the mechanism which triggers the condition that would reawaken it.

With this in mind, now I am thinking of an alternate solution. Rather than making a library that monkey-patches all thread blocking calls into tasklet blocking calls, I am now leaning towards monkey-patching them to raise an exception. They would all be guarded against being called at all. Any use of them would have to be explicitly allowed. So you would for instance allow stackless socket to call select.select via asyncore.poll, but when you subsequently invoked xmlrpc_server.serve_forever it would error on its call to select.select.

At this stage, no farming off work is performed at all, guards would simply be added to applicable functions. And references to those functions would be tracked down, in order to ensure that there are no corner cases where potentially thread blocking callables escape into the wild. What I mean by this is where for instance threading imports time and stores a local reference to sleep before the guards are put in place. This is a case I had to fix today to get the xmlrpc unit tests to run against stackless socket.

Once this is implemented and tested, next steps might be a generic mechanism to farm guarded calls out to worker threads, and an embedded web server that summarises what blocking calls are being made and additional information that helps a programmer understand what is really going on in their Stackless application.

Right, time to start coding..

Edit: First draft is checked in. It has a minimal blacklist, and can currently only handle simple functions. Before moving onto the next steps listed above, I should make it handle more than that.

Roguelike MUD progress #5

Previous post: Roguelike MUD progress #4

In order to add some life to my game, I chose a monster that I've always found intriguing, the gelatinous cube.

Image linked to from over on wizards.com

Now, the gelatinous cube is something from Dungeons and Dragons, and so it reminded me of the time that the people who made it released some open content. Cue confusion trying to understand their license. Still confused. Just as confused now as I was before. I am going to assume that it is workable.

Anyway, the content is provided in the form of RTF documents, so at first I thought I was going to have to rough it old school like they did in the '90s. However, other people provide versions in more approachable formats, like CHM and its inferior brother HTML. And better yet, various forms of SQL database. That was where it all started to go wrong, and this became a prolonged diversion from making real substantial progress on the project.

Rather than using piecemeal bits and pieces of the D&D content in the SRD, as it is available in database format, there should be no reason I can't procedurally generate game content from a database structured form of the SRD data. So, time to choose a database engine, and then to see what I can do with the data.

First up was SQLite. It is the obvious choice because it comes bundled with Python these days and therefore is not an extra dependency, given I am writing my game in.. Python. However, I am somewhat spoilt by the tools that come with SQL Server, so I looked around for some similar tools for SQLite. One promising tool just did not work. The next most promising tool was the Firefox add-on SQLite Manager.


It is a basic form of what it is intended to be, but unfortunately it is open source quality. Clunky. A garish gradient for background colours. As a piece of work, the programmer did well and probably put a lot of time into it, but the clunkiness gets in the way of usability. I found it hard to get a good view of what was actually in the columns. And found myself wanting to use SQL Server Management Studio.

So I took a look at the stripped down versions of SQL Server, and decided that SQL Server Express 2008 was the one for me. Where SQLite comes out of the box both Python and requires no installation, unfortunately the installation process of SQL Server takes ages and if you have a shoddy Acer laptop like myself, makes it overheat. Two days later it was installed and working, because the first time I installed it I chose the wrong option, and working out how to change it after the fact or why things didn't work.. was a nightmare.


That's much better. Now, it is time to look for a module that will allow me to use SQL Server from Python. pymssql looks like the best candidate for this.

The actual data in the SRD database is just pieces of information stored in text fields. Any use of it, will have to be interpreted at the Python level, or abstracted through database views.

Most of the work I have done on the game engine itself, since my last post, has been bug fixes.

• For some reason Putty stopped working as a game client. It would not switch to character mode and stayed in line mode when I connected to the game with it. This also meant that it didn't switch to the correct character set, because the server was reacting to incoming sets of characters, not incoming full lines. This was because I was not doing the negotiation right, sadly I needed to run Putty under the Visual Studio debugger to find out how. Unfortunately, dithering looks crap in Putty, as the colours are ambiguous.
  
  
  The yellow tile here is the "gelatinous cube". However, it still looks like a dithered yellow and white tile despite actually being straight yellow.
• Code reloading fixes. These are mostly make-do fixes, my referrer searching skills are deteriorating.
• Cleaned up key press handling, in preparation for the input queueing. Unfortunately, the input queueing requires game design related choices. Like how long an action takes, is dependent on what the action is and what it means in-game.
• The unicode character set display was broken, fixed it up and tried to get it to display various interesting unicode characters. A wider variety of tiles through use of unicode looks like more trouble than it's worth unfortunately. The number of unicode characters that have the correct representation when displayed in Putty are unfortunately limited. There are however Roguelikes that use unicode, but that's another project for another day.
  
  

In theory, I should be able to spend a few hours on this in the next three days. It is Mid-autumn Festival here, so I have three days off work.

Next post: Roguelike MUD progress #6 - Chinese character sets