stacklessemu

I use mobile broadband, and reducing bandwidth usage saves me a lot of money.  So when I wanted to play with pypes after hearing that Yahoo Pipes was being shut down, I somehow ended up taking Peter Szabo's greenstackless.py module, improving it a little, and putting it on a package up on PyPI.

It provides a good coverage of the Stackless API.  It's already been used as a backend for Peter Szabo's syncless project, and with the improvements I've made (adding the stackless.run() method, and fixing a greenlet/tasklet circular reference bug), it's even more capable.  But the devil is in the details.

It doesn't support the pre-emptive tasklet interruption that Stackless does, and can't as that is implemented via modifications to the internal Python VM source code.  Which is one of the few modifications Stackless makes in it's capacity as a fork of Python.  It is possibly possible to support this, as I believe Jeff Senn way back noted he had written a similar module with pre-emptive support based on the tracing hooks (with the downside that debuggers are unusable).

It doesn't support the threading model of Stackless Python.  This is derived by having a scheduler per thread, and most tasklets belonging to that scheduler at some level have slices of the stack of that thread and are therefore symbiotically attached to it.  It should be very possible to support this, with additional work, but that's something to set aside for when it's needed.

It doesn't support the module level properties, that Stackless implements.  And it can't unless it relies on hacks suggested in places like StackOverflow where classes are injected into 'sys.modules' to act as modules.

Future work I'm tempted to do is working out whether I can add the tealet project, as an alternative backend to the greenlet backend it currently has.  Greenlets are of course the stack slicing of Stackless extracted into an extension module.  tealet is the "next generation" version of the stack slicing intended for Stackless Python.  They're not using it already because it was such a wide ranging low level change to the workings of Stackless, that adopting it introduces potential instability which is not desirable at this time.

Kristjan Valur implemented a greenlet emulation module for tealet (in the same repo), so it may be that using it might be as simple as running stacklessemu on tealet.greenlets on tealet.  It's a pity it wasn't uploaded to pypi so I can try it out, but I don't think that's in Kristjan's wheelhouse.

Roguelike MUD progress #7 - Web-based client

Previous post: Roguelike MUD progress #6

I set a mini-goal for myself, to display a curses-based program on a web page.

The first step was to find a no frills, easy to adopt and extend Python websocket solution. One I've stumbled on  few times, is the simple websockets client/server gist. I used a slightly fixed version of it as a base. The next step was to find decent web-based terminal emulation. There are frameworks available which ditch the terminal window, but they come with larger problems.  The best candidate is term.js.

So taking the simple websockets simple_websocket_client.html and adding term.js, then modifying simple_websocket_server.py, it was easy to get a simple echo working with embedded ANSI control sequences.

First, I tried to proxy the input and output from the nano editor. Unfortunately, doing this with the subprocess module resulted in nano outputting a message about not supporting redirection.  There's probably a way to do this, but too much bother and it is not really my goal anyway.

Next, I went for something simpler.  I did a simple proxying of input and output to a telnet connection. Specifically my Roguelike MUD codebase.  There were some teething problems.  Text incoming from the browser-based terminal through the websocket, is UTF-8 encoded.  And similarly, outgoing text has to be as well.  But responses from the MUD contain both ANSI escape sequences and telnet negotiation sequences.  In order to work around this, I stripped the telnet negotiation out of outgoing text, and modified the relatively basic websocket code to also send binary frames.

A working connection:

In-game display:

As the picture above shows, the ANSI sequences are working perfectly.  But the character set does not seem to contain the line drawing characters.  My game interface already has debugging tools which display accessible character sets.

The main character sets:

As can be seen above, the only character set with even a hint of line drawing characters, is the one exposed through the ESC(0 ANSI sequence.  Most of these sequences are faked by term.js, where known character sets are mocked up from the underlying unicode-supporting font.

One solution is to do what I do with Putty, which is to identify the connected client, and if it's one which supports unicode, then switch unicode on using terminal control sequences.  Modifying term.js to respond to the ENQ character (0x05), and send it's terminal name (xterm) was easy enough.

This is what I should be seeing now:

This is what I am actually seeing:

The correct unicode characters are being sent, but not being displayed correctly by term.js.  I don't expect this will be that difficult to correct when I get some more time to work on this.

Stackless Python 2.7.6

I've finally completed the last part of the Stackless Python release process for our fork of Python 2.7.6.  We're a little behind on a few releases, but the source code for all of them has been  stable and working in the Mercurial repository in a timely fashion.  Unfortunately, due to the political problems that have arisen with our Stackless 2.8 work, we've taken the repository private until we've addressed them.

Thanks to Robert Babiak for the MacOS installer, and Anselm Kruis for the Windows installers, the documentation and setting up readthedocs accessible through the documentation link below.

Links:

• Downloads.
• Documentation.

Peasauce, interactive disassembler.. someday

From Resource to Peasauce

I have a list of hobby projects that I've always wanted to find the time and energy to work on, and one of these is an interactive disassembler.  Years ago when people used a type of computer called "the Amiga", it had an extremely usable interactive disassembler called Resource.  You would point it at a file and it would turn the file into assembly language code and data, after which you would interactively disassemble some more, and finally make it write out a file containing the assembly language source code.

Disassembled and commented code in Resource.

Recently I finally managed to start using some spare time to work on "Peasauce", and have made a pretty good start.  What it can currently do isn't really useful yet, given that editing a disassembled file or exporting it isn't implemented yet. But being able to select a file, see the disassembled results and then scroll around it qualifies as interactive disassembling in the loosest sense.  So it's a good start.

Disassembled code in Peasauce.

In order to help make the code more general, in addition to Amiga executable files, it also loads those from the Atari ST and X68000 platforms. I've never used either, but they're both based on the Motorola 680x0 CPU family so it's a minimal amount of extra work over and above parsing a few semi or undocumented file structures.

UI

I initially started usng the Tkinter, because it comes with the standard library.  Unfortunately, out of the box it just wasn't flexible enough to work in the way I wanted.  As an alternative, I first looked at PySide but wasn't able to locate documentation to make it easy to work out if it suited my needs.  The website looks like it is in a transitional phase and several links to useful sounding documentation were dead unfortunately.  This left wxPython, which I've used before.  As a fringe benefit of having a small play around with PySide, I first encountered crashes and then with Anselm Kruis' recent patch to Stackless Python was able verify this decade long ABI incompatibility was fixed and that the crashes went away.

With wxPython, I'm currently using the ListCtrl where it simply knows how many list items there are and requests list items as needed for display.  Strangely, as the number of list items increase it gets slower, but that's likely my fault for some reason. As a short term solution while there's minimal interactivity, it's good enough, but the time is almost right for something more dynamic.

The bisect module

I was iterating over ordered lists looking for the right entry, and this was profiling as the place most time was spent.  A quick browse of the Python documentation introduced me to a module I'd never heard of before, the bisect module.  By keeping two lists instead of one, bisect let me look up objects in the second list by passing it the first list.  That makes it sound rather complicated, a simplified piece of example code is a lot more helpful.

 lookup_index = bisect.bisect_left(lookup_keys, lookup_key)
 if lookup_keys[lookup_idx] != lookup_key:
  lookup_value = lookup_values[lookup_idx-1]
 else:
  lookup_value = lookup_values[lookup_idx]

While I haven't actually looked, writing this up has made me suspect that if I also bung judicial use of this in the list control virtual item fetching, it might even solve the slowness problem with larger files.  Actually going away and looking at the code confirms it.  I'll need another indexing list though, and to keep that synchronised with the other two.

Opensauce

My laptop is a little unreliable and addressing that isn't on the table, so a good way to ensure I don't lose the source code should anything happen, is to throw it up on github with everyone else's half completed projects where it can also sit mouldering and abandoned.  Or slowly be developed further, of course.

Link: Peasauce on github.

Stackless Python and Stack Overflow

I often search for programming related information, and gladly use the Stack Overflow based results.  More often than not, the answer I am looking for is already there.  This unfortunately means that people blindly post their questions there, without it occurring to them that there might be a better place where actual experts are present.  This better place is the Stackless Python mailing list.

For projects where the number of knowledgeable users present on Stack Overflow is high enough, it obviously serves as an invaluable resource.  For a lesser used niche project like Stackless Python, not so much.

Python garbage collection crash

I run the unit tests for the Windows debug build of Stackless Python 2.7.2, the tests pass, but on exiting the interpreter crashes.  The problem is that the garbage collector finds a 1 element list, where that element has either been already garbage collected or never initialised.  Logically, the following occurs.

How do you track down the cause of this crash?

Gyp and libuv

One open source project that I've heard a bit about recently is libuv.  A cross-platform asynchronous IO library, it is something that I can plug into Kristjan Valur's stacklesslib module.  And all the work writing the Python bindings has already been done by Saúl Ibarra Corretgé in his pyuv project.

Anyway, I spent some time today getting it compiling for Windows and thought I would post a note about gyp, so that I could remember to avoid the same problems in future.  Gyp is a pretty handy build system. Just define the gist of what needs to be compiled in a mildly arcane mark-up, and it will churn out Makefiles, Visual Studio projects and solutions and a range of other things.  However, as with most programming related things, it is the little details that trip you up and cause you to waste your time.

Missing include files

If you are compiling with Visual Studio 2008, and it includes inttypes.h or stdint.h, errors will occur as these files will not be present.

...
libuv\include\uv.h(54): fatal error C1083: Cannot open include file: 'stdint.h': No such file or directory
...

Apparently the solution to this, after a lot of googling on what constitutes a wi-fi connection over here in New Zealand,  is to download something like the custom msinttypes versions of these files.   And then bung them somewhere Visual Studio will find it.. wherever that is.

Overriding gyp settings

If you are compiling a static library with Visual Studio using gyp, it will start with a range of default settings.  You will not be able to override these settings no matter where you put your own custom versions in your gyp configuration file.  The specific problem I had was that Python extensions are compiled with a RuntimeLibrary setting of MultiThreadedDLL and libuv is compiled with a different setting of MultiThreaded.  These are of course incompatible, and a variety of compilation errors are caused by it.

...
LIBCMT.lib(crt0dat.obj) : error LNK2005: __amsg_exit already defined in MSVCRT.lib(MSVCR90.dll)
...
LINK : warning LNK4098: defaultlib 'MSVCRT' conflicts with use of other libs; use /NODEFAULTLIB:library
LINK : warning LNK4098: defaultlib 'LIBCMT' conflicts with use of other libs; use /NODEFAULTLIB:library
LIBCMT.lib(crt0.obj) : error LNK2001: unresolved external symbol _main
build\lib.win32-2.7\pyuv.pyd : fatal error LNK1120: 1 unresolved externals

In order to force this gyp setting to change from the default, it needs to be defined within a conditional expression.  The default setting is also a conditional expression, and .. well.. work it out.  Maybe it's in the documentation somewhere, but not in a clearly stated way that I could find.

Scheduler deadlock

One common approach to creating tasklets, is to wrap the callable that will run within the newly created tasklet in a function that captures exceptions.

Something like the following:

import traceback

def create_tasklet(f, *args, **kwargs):
    try:
        return f(*args, **kwargs)
    except BaseException, e:
        # Let TaskletExit, SystemExit and KeyboardInterrupt raise up.
        if isinstance(e, SystemExit, KeyboardInterrupt):
            raise e
        traceback.print_exc()

Another exception that is worth handling especially, is RuntimeError. In the case where the last tasklet tries to block, the scheduler will instead raise a RuntimeError on it in order to prevent deadlock. Catching this exception and showing what the main tasklet is doing, and why it is blocked, is helpful. That is assuming you are running the scheduler in the main tasklet.

Something like the following:

import traceback

def create_tasklet(f, *args, **kwargs):
    try:
        return f(*args, **kwargs)
    except BaseException, e:
        if isinstance(e, RuntimeError) and e[0].startswith("Deadlock:"):
            print "Deadlocked tasklet, the main tasklet is doing the following:"
            traceback.print_stack(stackless.main.frame)
        # Let TaskletExit, SystemExit and KeyboardInterrupt raise up.
        elif isinstance(e, SystemExit, KeyboardInterrupt):
            raise e
        traceback.print_exc()

For future reference, the situation where I needed this was where I was polling for completed asynchronous file IO events. In theory this was programmed to do so in a way where it would schedule any tasklets blocked waiting for their IO to complete, so they would run when the scheduler was pumped next. However, due to a race condition the main tasklet sent the results of the IO operation to the channel before the tasklet doing the operation itself blocked to wait for those results. Then when the other tasklet itself blocked, the deadlock situation occurred.

The licensing situation of Greenlet

In order to use Stackless Python and gain what it offers, a custom version of Python needs to be installed. You may know about Greenlet which offers part of what Stackless does, and which can be installed as an extension module for your normal Python installation.

The MIT licensing of Greenlet

When it was extracted from Stackless Python, it was checked into the Py or PyPy repository or something. Py and PyPy are licensed under the MIT license, but as an oversight the licensing of the Greenlet code was never clarified. From there, it was handed off and maintained by Bob Ippolito. And from there, it made its way to the bitbucket repo. Along the way it inherited the MIT license.

The actual licensing of Greenlet

Greenlet contains parts of Stackless Python, specifically the assembler code related to switching between coroutines. Stackless Python as a derivative of regular Python has always just gone with the license of Python itself, so these files are still licensed under the Python license. Ideally the easy solution would be to get Christian Tismer to relicense them under the MIT license, but its not that easy. Between the time Christian wrote the original set of these files, and the time when they were extracted for Greenlet, other people (including myself) contributed both additional files and patches. To really relicense the code, all of these people would need to give approval (as I understand it). Who are all these people? I don't know.

So despite the MIT license that comes with it, and the declaration that it is MIT licensed on the Pypi page and elsewhere, to my understanding the project is effectively licensed under two licenses. For the files which came from Stackless, the Python license applies. And for the files which came from Armin Rigo, the MIT license implicitly applies.

And?

There's a bug on it.

A double licensing mish-mash isn't desirable for the project, and the easiest step forward is relicensing to the Python license to work around that as broached in the bug. I was going to offer to do the work of tracking down all the contributors and sorting this out.. but due to circumstances beyond my control I no longer have the time.

Enthought's reStructuredText editor

With the Stackless Python 3.1.2 branch more or less working on both Ubuntu and Windows, I can concentrate on doing the few remaining tasks. One of these is to translate the Stackless documentation to speak the Python 3.x syntactical dialect. Because there's always some exercise in procrastination to choose, rather than get on with a task, I choose to look for aids in editing reStructuredText on Windows.

There are numerous excellent and free text editors on Windows, unfortunately none of the one non-niche ones support reStructuredText. Why is this? There are commercial text editors that support syntax highlighting for it, but all text editors are at some level an arbitrary set of features with an arbitrary look that work in an arbitrary way. The feature of restructured text highlighting isn't worth the price in effort and convenience of using another arbitrary text editor, let alone paying money to do so.

Searching for free custom targeted editors, led to three candidates:

• ReSTedit - Looks unmaintained, and doesn't look user friendly enough to warrant spending an unknown amount of time.
• rsted - Looked really good, but as Catherine Devlin describes, the process of getting it working is a long and torturous one.
• reSTinPeace - Lots of dependencies including PyQT, only source available.. sorry but I've wasted enough time on that rabbit hole with rsted.

rsted

In the screenshots on the blog post, this looks very well done albeit clunky in the way open source GUI toolkits are. While a standalone text editor for reStructuredText editing is non-ideal and frankly cumbersome, the functionality this promises seems more than work it.

Here's the glorious screenshot from the Enthought blog post:

However, let me save you some pain, do not waste time trying to compile and install this. Following the instructions given on Catherine's blog, and also by one of her posters I encountered lots of problems getting the dependencies installed. And in doing so, it turns out those instructions are now wrong. rsted does not come in the AppTools egg anymore. And should you fetch and run the rsted source after installing all the dependencies, you probably end up without a working preview pane.

My instance of rsted is still sitting there waiting for the heat death of the universe to occur. At this point, I am questioning whether I really need WYSIWYG editing of reStructuredText. The only thing that would convince me to try this editor again, is a Windows installer that just installed this tool ready for use. But what I would really like, is a plugin for one of the existing non-niche open source Windows text editors. Of course, neither of these things are likely to happen.. so time to get back to editing raw unhighlighted reStructuredText.

Preparing Stackless Python 2.7.1

I used to do the releases of Stackless Python on my laptop at home, but as it is an Acer, it is prone to instantly overheating and shutting down at the slightest sign of having to perform useful work. Play a movie? No worries. Synchronise an SVN repository? Time to shutdown! Given that and with everyone on holiday, today is a good day to go into work and do it on my work computer. There's also a decent internet connection here.

The process of building and releasing Python is rather easy these days. You can just execute the buildbot scripts and most of the work is done. After that, it's just a matter of dealing with the one or two parts of the process that seem to be.. strangely incompatible with the rest of the process.

Oftentimes I will have to get cabarc.exe sorted out. The MSI installer generation script won't seem to find it. In this case, it seems to have been replaced with another solution called makecab.exe in things like the Windows SDK. A quick google hit Stack Overflow which gave a roundabout way to obtain it. Namely obtaining some WindowsXP-KB838079-SupportTools-ENU.exe file, then extracting a CAB file from it. The given extract command does not exist, so I used 7zip to get cabarc.exe.



Installing the Windows SDK is rather a rough process. Wait, no latest .NET? Something won't be installed, and you will have to reinstall or something. Okay, I don't understand so exit to go and install the requested dependency.


I think they've given up on the notion anyone bothers to sit there and read the EULA.


In the past, I've gone through all kind of complicated shenanigans to get my data uploaded to the Stackless web site. I don't have any ready way of offering the files for download to be fetched from the Stackless site, so I upload it to another site and from there ftp it. I've at one time in the past tunneled FTP over an SSH connection, but that's still cumbersome. This time, I did a bit of googling and finding no clear explanation decided to download pscp.exe and see what it offered me.

pscp.exe python-2.7.1-stackless.msi richard@stackless.com:/home/richard/stackless/

That couldn't have been more straightforward and easy. It basically took advantage of Pageant running with my private key added, and transparently authenticated and copied the file into place. Robert's your father's brother.

unittest.py quirk

Now I am using Python 2.7, and whatever form of unittest module that comes with it. Maybe this is fixed over in the (still not relevant to me) 3.x branches. Anyway, oftentimes I will create some simple mocking in the setUp method of my test case, and then the test method will error in a way that is unrelated to my mocking. But error handling of tests happens before tearDown is called and it will collide with my mocking, so instead of finding out about what test failed I find out about this collision.

It goes something like this.

class MyTESTSSSS(unittest.TestCase):
    def setUp(self):
        self.oldOpen = open
        __builtin__.open = ReplacementOpenFunc

    def testSomething(self):
        1/0

Of course, that may not reproduce the problem, it's just for the purpose of illustration. Anyway, what's my point.. oh, in order to work around this, I now have to structure my tests in a way that enforces pre-error teardown.

class MyTESTSSSS(unittest.TestCase):
    def setUp(self):
        self.oldOpen = open
        __builtin__.open = ReplacementOpenFunc

    def preErrorTearDown(self):
        __builtin__.open = self.oldOpen

    def testSomething(self):
        try:
            self._testSomething()
        finally:
            self.preErrorTearDown()

    def _testSomething(self):
        1/0

Anyone else encounter this problem and take any different approaches to recovering from it?

If the answer involves a mocking framework, decorators, or context managers, then it is the wrong answer for me :-)

test_repr.py

Note to self. test_repr.py in the Python test suite does lots of long file name based tests. However, you don't need to have your source code under too many sub-directories on Windows before a test hits the path length limit and fails unwittingly. I think in my case it was c:\Users\MyName\Stackless\stackless-branches\release27-maint\release27-maint-export\release27-maint\...

Reading from non-blocking sockets

Until recently, I had never actually used my Stackless socket module for anything other than my own hobbyist projects. These are done out purely of interest, and the limited networking traffic they receive is solely me testing them. I had also never received any complaints about the throughput of the module. So it was a complete surprise to me when I learnt that I was doing non-blocking IO incorrectly with Python's socket module.

I had always assumed that when a call to select indicated there was data to be read, this meant that it was okay to do a recv call. And for my purposes, this appeared to work fine. But when you are polling select thirty times a second, and the packet sizes that are received are consistently around 16 KB, that limits the throughput to 280 KB/s. For instance, an application on an embedded device might be fetching 115 MB in updates. This would take 4 minutes.

It turns out that when select indicates there is data waiting to be read on a socket, you can call recv until EWOULDBLOCK is raised. Who knew? Probably someone who had read appropriate documentation, this document seems to be pretty pertinent.

With the above implemented, the current implementation reads the 115 MB in 13 seconds. And that is with it limiting the number of recv calls within each poll of asyncore, in order to yield in a timely fashion to the application that is pumping it.

Reconsidering thread blocking WRT microthreads

I meant to spend a good part of the day doing the long overdue Stackless 2.6.6 merge, but having forgotten my SSH key, I found myself at work on a public holiday with time set aside for it.. but with an extra obstacle in the way. So I moved onto another project -- looking into the reports that there are problems with my stackless socket module, and the xmlrpc standard library. I was however unable to reproduce the problems, and spent the time cleaning up the mechanisms that allow me to run the standard library unit tests against my module.

The bulk of these mechanisms are monkey-patchings of thread blocking operations. When any of these calls (time.sleep, select.select, ..) are made, they would block the Stackless scheduler running on that thread. So the general brute force solution is to make any of these calls simply block the current microthread, while the actual call is made in another thread. This is rather naive, but with a bit of special casing for hiding harmless errors caused by race conditions, for the purposes of running suites of unit tests it works out fine.

However, the more I work on something like this, the more uneasy I feel with the long held ideal of having a suite of these monkey-patching mechanisms which users can just install to make any use of the standard or third party libraries just work. It's not that it wouldn't just work for the most part, look at stackless socket, it has been a cobbled together mess for many years and it was more than enough for almost anyone who used it. It's that it is actually important for people writing code to know where the points their microthreads block are. Whether these are safe voluntary yields to the scheduler, where state may change before the microthread gets scheduled again and resumes. Or whether it is unsafe blocking of the thread, which may also block the mechanism which triggers the condition that would reawaken it.

With this in mind, now I am thinking of an alternate solution. Rather than making a library that monkey-patches all thread blocking calls into tasklet blocking calls, I am now leaning towards monkey-patching them to raise an exception. They would all be guarded against being called at all. Any use of them would have to be explicitly allowed. So you would for instance allow stackless socket to call select.select via asyncore.poll, but when you subsequently invoked xmlrpc_server.serve_forever it would error on its call to select.select.

At this stage, no farming off work is performed at all, guards would simply be added to applicable functions. And references to those functions would be tracked down, in order to ensure that there are no corner cases where potentially thread blocking callables escape into the wild. What I mean by this is where for instance threading imports time and stores a local reference to sleep before the guards are put in place. This is a case I had to fix today to get the xmlrpc unit tests to run against stackless socket.

Once this is implemented and tested, next steps might be a generic mechanism to farm guarded calls out to worker threads, and an embedded web server that summarises what blocking calls are being made and additional information that helps a programmer understand what is really going on in their Stackless application.

Right, time to start coding..

Edit: First draft is checked in. It has a minimal blacklist, and can currently only handle simple functions. Before moving onto the next steps listed above, I should make it handle more than that.

Roguelike MUD progress #4

Previous post: Roguelike MUD progress #3

Almost no progress on the game today. Instead, I was distracted by two different problems with my code reloading framework. Link to SVN revision.

1. Use of slots:

   class Tile(object):
       __slots__ = [ "character", "fgColour", "bgColour", "flags" ]

   This also seems to be an unsolved problem in the xreload module, which was written for normal Python namespaces.
   
   TypeError: descriptor 'flags' for 'Tile' objects doesn't apply to 'Tile' object
   
   I've not got any better ideas, and I am only using __slots__ for the sake of it. So the solution is to stop using them.


2. File-local instances of the classes it defines:

   START_TILE = Tile("x", isMarker=True)
   SPAWN_TILE = Tile("s", isMarker=True)
   WALL_TILE  = Tile("X", isPassable=False, isOpaque=True)
   WALL1_TILE = Tile("1", isPassable=False, isOpaque=True)
   WALL2_TILE = Tile("2", isPassable=False, isOpaque=True)
   DOOR_TILE  = Tile("D")
   FLOOR_TILE = Tile(" ")
   
   mapTilesByCharacter = {}
   def IndexTiles():
       for k, v in globals().iteritems():
           if k.endswith("_TILE"):
               mapTilesByCharacter[v.character] = v
   IndexTiles()
   
   CHAR_TILE_TEMPLATE = Tile("@", isPassable=False)
   DRAGON_TILE = Tile("&", isPassable=False)

   I've had to use a custom framework that has xreload-based code reloading, and encountered this same scenario there. In that case, it was because the class was registered with a external factory and subsequently used to churn out bad instances. These then hit the following error.
   
   unbound method BLAH must be called with BLAH instance as first argument (got BLAH instead)
   
   Solving that leaked class problem was simple, through locating the references to the throwaway new class, and substituting the old reference for them. Replacing instances though, is a little more complicated.
   
   This error has gone away in 3.x I am told, which is a pity. As it serves as a good indication that there instances or class references have leaked outside of the code reloading system.

Chances are that I am going to have to try and solve these cases for xreload, as well as in my custom framework.

Next post: Roguelike MUD progress #5

Roguelike MUD progress #3

Previous post: Roguelike MUD progress #2.

Field of view emphasis is now row based, rather than tile based. However, I've also added the ability for tiles to have attached foreground and background colours. Really, the rendering needs to break all this down and minimise the number of switches made using different escape codes. For now, use of colour is limited, so updating the display does not seem slow or flickery like it was in the past before I batched updates.

I've also added simple NPCs, in this case three small green dragons that wander around the dungeon. For now, that's all they do, but they're good at highlighting rendering bugs.



Current TODO list:

1. Add bursts of fire, using different tiles and colours to make use of a wide range of colours than the basic eight through dithering. It might be a good idea to write the tile/colour choosing logic in a way that can optionally use the xterm 256 colour mode, if the client can be detected as supporting it.
2. Add option for NPCs to choose to attack, rather than move, when they are ticked.
3. Add option for players to attack, rather than move.
4. At the moment, player keypresses are executed as they come in from the client. They should instead be queued and executed at a fixed rate.

Next post: Roguelike MUD progress #4

Roguelike MUD progress #2

Previous post: Roguelike MUD progress.

I wasn't feeling very motivated when the time I had set aside tonight to work on this came around, but once I got into it I made pretty good progress. The bugs I listed yesterday are now fixed, and additionally the field of view emphasis is working. It's annoying that the stupid mistakes are the ones that take the longest to track down. In this case, absently writing or instead of and.

if y in self.drawRangesNew:
    minX, maxX = self.drawRangesNew[y]
    if x >= minX or x <= maxX:
        return True

Maybe I should reconsider minX <= x <= maxX, although it never quite seems right.


The FoV changes are not as optimal as they could be. Every tile in the post-move FoV tile set is individually marked up with escape codes, I should instead simply mark each row of qualifying tiles in one go.

Current TODO list:

1. Clean up the FoV emphasis to be row-based, rather than tile-based.
2. Add some objects and entities to the world, that can be interacted with. Entities should move to add life to the world.

Next post: Roguelike MUD progress #3

Roguelike MUD progress

I finally found some more time to work on my roguelike MUD project. Tonight I managed to get proper multi-player support in, so that concurrently logged in players have their view of the game updated as other game objects change position. Up to this point the players mostly shared the world representation, and had separate state defining what was in it.


While the changes required are more or less fine, some didn't fit well into the existing game framework. I need to put some thought into cleaning those up.

Achievable next steps should hopefully be:

1. Cleaning up the bugs.
   - Fix the incorrect menu related message that appears in the top telnet window shown in the screenshot.
   - When a player quits the game, the display of other observing players does not update to reflect it.
   - When an object moves, observing players are defined as those who have visited the tile the object moved to and currently have it on their display. Instead only observing players who have the object in their field of vision should see the movement.
2. Polish the field of vision support.
   - The tiles that are in the field of vision should be distinct from those that are not (Smart Kobold uses this approach to good effect).
3. Widen the corridors so players can pass each other.
4. Add entities controlled by AI that move around of their own volition.

Really, in this day and age a game like this should be browser based (like 91). But getting side tracked into a non-game related endeavour to implement an equivalent canvas based display with a AJAX or COMET connection to the backend server, is something I have no time for just now.

Next post: Roguelike MUD progress #2

stacklesssocket.py, greenlet and blocking operations

It seems like many years ago now, that I wrote stacklesssocket as a hobby project for whatever reason I have since forgotten. By monkey patching a matching synchronous interface to asynchronous resources (i.e. asyncore) over the socket module, a lot of existing Python code that used the socket module would just run when used in an application that also used Stackless Python's tasklets. Andrew Dalke's example illustrates this.


I now have a number of hobby projects that only get changed when I have the time, energy and interest to work on them. And as such, this module would only get attention when those other projects used it and encountered a problem. This didn't stop other people from using the module. Whether they still use it I do not know, but the other day I saw that a business that at least used to base their service on it got bought by Google. It is also used internally at C.C.P. temporarily in a situation where their Stackless IO library (still pending open source-ification) is not supported.

Recently, I received an offer of sponsorship to do something I have wanted to do for a long time. I won't mention who is sponsoring me, without getting their permission first. Anyway, the central task within the sponsored work was to get the Python standard library unit tests running against it, and to get the tests (at least the non-UDP ones) passing. This is now done, and apart from two particular UDP unit tests, the module passes the tests for Python 2.6 and 2.7. The module is also compatible with Python 2.5, and won't work any worse with that version than it did before. There are at least some API differences (which exceptions are raised) that it does not specifically support when run against Python 2.5. It is interesting to note however, that given there are only two UDP test failures, it is a good sign that generally the new module changes are naturally correct.

The Python unit test suite uses a lot of thread blocking calls, which also block the Stackless scheduler that manages the tasklets running on the given thread. And often, those tasklets need to do something for the thread blocking call to get what it needs. So, this is not Stackless compatible behaviour. The most common of these calls are time.sleep, select.select and calling acquire on a thread lock. In order to get these working, I've implemented a threading-related module to also do monkey-patching to hand off the work into another thread. Something that also required more work that I would have liked, was working around the constraints that Stackless enforces when scheduling in another thread (there is a scheduler per-thread) and that thread is blocked while waiting for a channel operation. There aren't many users of Stackless' scheduler per thread support, and a few of the rare corner cases need some usage to clean up exactly how they should work.

While it has been an additional goal since stacklessocket was first created to eventually implement a matching suite of modules that can be used to turn any thread blocking operation to a tasklet one, unfortunately I've had to spend my time elsewhere. But we are now with the addition of the threading-related module part of the way there, even if only to a need-driven level so that the Python unit tests will run. I've been envious that gevent has a suite of modules like this, although I have no idea of the extent to which they work. My involvement with greenlet has been to write a module that provided a Stackless-like interface for it, when it was first released. Beyond that, my involvement with anything greenlet related has been to sending an email to a tengentially related mailing list, related to noting that the Python license more than likely applies to any source code that was extracted from Stackless Python (and therefore it can't be MIT licensed within greenlet), both demonstrating my lack of knowledge about how it is used and potentially annoying other people with license complications.

Anyway, if you use this module, I suggest giving the latest version a try. Let me know how it works for you.