Live coding / code reloading for Python

Several months ago there was a spate of interest in live coding with Python. Those who experimented with it reasonably took the approach of continually calling the reload method. However this only worked when code was implemented in a stateless way and how to code within this limitation was covered in detail in the blog posts about it.

I commented on one of the later posts (although my link placement was a little mixed-up) gathering together enough information needed which happened to be publically available and could be used to write the support needed for live coding without the mentioned limitation. To recap that information:

• Detection of changed files:
  - Cross-platform.
  - Windows.
• Updating class instances on reload.
  

But the general interest in this sort of thing died down and no-one stepped forward to do the work needed. I certainly was not going to do it, as while in theory it would work, it wasn't the approach I would take anyway. Tracking instances so that they can be individually updated seems like much more work than should be required.

A more usable solution would be to update the class itself which would transparently make the changes available to all instances. Though actually implementing this is not as straightforward as one might hope. For instance how functions from classes even unbound are indirectly linked to the class they came from is one of the little hidden surprises of how Python is implemented.

And this "more usable solution" is what I have released, combined with the code from the links above to handle the detection of file changes:

Live coding v 1.0

However there was a design decision made which need to be taken into account. Namely that this is only made possible by using a custom importing solution. All the code which is have automatic code reloading has to be under this solution. Why it was used I will cover after I show how to add a directory containing Python scripts to be made available

This will add a directory of code as importable modules:

import livecoding
cm = livecoding.CodeManager()
cm.AddDirectory("c:\app\server", "applib")

Let's say that 'c:\app' contains the following directories and files:

server\services\networking.py
server\objects\session.py

Where 'networking.py' specifies the class NetworkService and 'session.py' specifies the class Session. Under the algorithm used to map directory contents to an importable namespace, NetworkService would be placed as applib.services.NetworkService and Session would be placed as applib.objects.Session.

So these classes could be imported in the normal manner:

from applib.services import NetworkService
from applib.objects import Session

So why use the custom importing solution? Well, it makes updating loaded code with changes easier because as the library controls the importing it knows what to update and how. But why not look into getting a similar system working with the standard module system? Not interested, sorry. My fellow author and I have worked a lot with custom importing systems and found them to be much easier to work with than the standard module system. And over time we have come to prefer the system which this library implements.

I also cobbled together another library filechanges which comes as part of the download or can also be obtained from the SVN archive. This is just a collection of the file change detection code found in the links I originally gathered above. The livecoding library can be told to use this to provide built in file change detection.

import livecoding
cm = livecoding.CodeManager(detectChanges=True)
cm.AddDirectory("c:\app\server", "applib")

Which completes the collection of functionality which makes up the v1.0 release of this library. However there is still work remaining to do. There is the incomplete override feature, clean up needed regarding custom methods of handling the file update events and more methods of file change detection other than the only one which currently works (the cross-platform one).

Things of interest

This post to The Python Papers blog reminded me that I have been intending to release a library a friend and I have written which does most of the work to allow live coding in Python. Writing an article about it might be a good way to get the library out there, and hopefully get some discussion about its shortcomings. Reading a thread on the PyGame mailing list about allowing your game's users to use Python as a scripting language, and how there was no secure way to do this, it seemed like an interesting exercise to see how far the safe expression execution recipe could be taken to facilitate it. It would also be interesting to experiment with that and write an article about it.

Stackless Python

Stackless Python is in a much better position than it used to be for someone approaching it for the first time. Between Grant Olson's tutorial, the wiki pages like the ones for tasklets, the scheduler and the examples, there is something there for them to work from. But it is still a long way off from being approachable and I think this shows in the amount of use it sees. I recently realised just how hard it was for someone to get the benefits they should be getting out of Stackless.

Andrew Dalke was playing around my Stackless-compatible socket module (which can be found on the examples page in the wiki) and tested it by having two pieces of Python code executing in parallel doing socket operations on the same thread (you can see Andrew's code in this post). Andrew was quite impressed by this and justifiably so. But the fact that this is not possible with Stackless, out of the box, is indicative of how Stackless could be more approachable. The problem, and therefore reason why we need a replacement socket module, is that any blocking call (like normal socket calls) blocks the Python interpreter and therefore also blocks the Stackless scheduler.

The best benefit from using Stackless is gained by being able to write arbitrary code in a straightforward synchronous manner and being able to run it in parallel with other similarly written code. But when all the most common resources which you will want to use (most often file or socket IO) can't be used without blocking all your other running code, your code might look better but you're losing the benefits you should be getting naturally from running your code in parallel as tasklets.

Ok. So you now know to use the "Stackless-compatible" socket module. And most of your code which uses the socket module works in parallel (I'll let the 'most' sit for now). But what about file IO? What about subprocess calls? And any other blocking calls? It is possible to do as I did and use asynchronous IO (i.e. for sockets the asyncore module), wrap it with Stackless channels and provide a replacement module. But not all blocking calls have equivalent forms usable in an asynchronous manner.

This is a problem I have always been blind to with Stackless. On one hand I use Stackless Python every workday in my employment at CCP Games, and have done for the last five years. But we provide our own Stackless compatible socket and file IO and it has always just been there doing its thing. Then there is the personal programming projects I have used Stackless in, I eventually evolved some code which wrapped asyncore to do the socket IO asynchronously in the background and didn't think much about it. Eventually I cleaned up this code and released it as the Stackless-compatible socket module, but only to support my MUD example code.

That Stackless would be more approachable and naturally usable for having this module built into it, was not something that occurred to me. Andrew suggested a solution where a module would monkeypatch the Python runtime and replace all the blocking calls where it was possible to do so with Stackless compatible versions which worked asynchronously in the background. I've done a little work on a module like this and while it is a start, it still needs a lot more work.

One last thought. Where calls which block have no asynchronous alternative exposed to the interpreter which we can use to replace them, I lean towards feeling like this is something which should be built into Python. As compared to using ctypes to access asynchronous file IO, like my current monkeypatching module does for Windows and its IO completion ports. I am thinking this because of the new generator coroutines which Python has acquired. Isn't this IO problem something generators share? I have never had a situation where I have found a need to use them, so I don't know for sure.

Python for the Nintendo DS revisited

Around a year has passed since I ported Python to the Nintendo DS and occasionally I get an email asking about it. A few asked whether there was any support for the DS' hardware. Others asked about whether it could handle other things (which would also require hardware support). And at one point someone was planning to do the work themselves to add that hardware support.

I knew I would never have the time to work on the support needed for the port of Python to be of any real use. For me, the work involved in porting it and getting it working as much as it does was the interesting part, especially given that it was Stackless Python which was ported. And it looks like it is unlikely to ever see the extra work it needs to make it usable.

Baldur's Gate 2 / Brad Wardell interview

The interview with Brad Wardell on Gamasutra touched something interesting. Why were there never any more games like Baldur's Gate / Icewind Dale?

GS: ... I was talking with Simon [Carless] and we were talking about how you make games for a very specific market. Do you ever think you'd like to do something with more mass appeal?

BW: ... There's a market for that. We also want to make a role-playing game at some point. Not in an Elder Scrolls style, but more of a Baldur's Gate kind of thing.

GS: Yeah, that would be very popular. There's been a big movement away from that kind of thing.

BW: Didn't Baldur's Gate 2 sell like a Gajillion copies?

GS: It was huge. I don't know why they haven't made any more. They made Baldur's Gate 2 and Icewind Dale.

Earlier in the conversation, Brad refers to Master of Orion 3 and how the developers shouldn't have pursued their own vision, given what the previous versions were like and what players expected. If he applies this in the more general sense to actually making a game with the same appealing elements that Baldur's Gate 2 had, rather than just making something with the same look and feel (the look and feel difference between it and KOTOR for instance), Stardock might actually make a game I would want to play / buy for a change.

I have to take a look at the Infinity Engine modding scene and see where they are in terms of a editing toolset someday. This looks like a good site to start with.

Backing up a gmail account

There are a lot of interesting and useful services on the internet these days, like gmail and blogger. But when I take advantage of them, I cannot help but feel pained by the fact I am entering my content/information into a something controlled by someone else. There is a certain satisfaction to having a copy of your own information, knowing that if the service goes away, or changes it terms of service, you still have everything you created to do with as you wish.

To ensure that I have a copy of the information in my gmail account, I have been keeping an eye out for a way to back it up. It was easy to find references to downloading the emails via POP, but that seemed a second rate solution. How do I know it will back up the emails I sent? What about the labels I have added to threads? So, ideally, I wanted something that would download all of the original messages, the labels, the threads and the labels assigned to threads.

This last week, I found a little time to play around with libgmail. This is a Python library which seems to do the same thing that the gmail interface does, in order to acquire the same forms of information that the interface is provided with. It is a surprisingly small and simple library, but it was complete enough for me to use to write a Python script to do the depth of backing up I wanted. The script currently allows:

• Fetching the raw emails for each message in my account (whether ones I have sent or received).
• Noting which thread each messages belongs to.
• Noting which labels each thread has.
• Noting all the labels I have created.
• Using the previously backed up data to only download the changes, which is important because gmail locks down accounts in which it detects "unusual usage".

The backed up data is stored in the Python pickle file format. This is not a problem at all, because writing a Python script to do pretty much anything with the data, takes a minute at most. Including going over all the emails looking for specific things, extracting an email, etc.

One of the things I was interested to see, was that the size of all my emails came to about 80 megabytes, the same amount of space that gmail claimed my account was using. I had always doubted whether this was calculated using a fair method and assumed it was overestimated, or took into account how much space they used for all the indexing information for my account. I was also surprised to see how many emails I had, although alot came from mailing lists. 2018 threads in total, with around 11000 emails between them.

Backing up my account marks the emails in my inbox as having been read, which given all the features in gmail to make reading threaded conversations easy, makes it difficult to follow a thread. So it needs to be used after you have read all your emails.

Anyway, it is a load off my mind to have this backup tool written. I have also offered it as a demo script to the contact address for libgmail. Since I have blogged about it though, I have decided to add it to a google code hosted project. You can find it here:

http://code.google.com/p/gmail-backup/

Porting Python to the Nintendo DS

What was involved

Initially, when I decided to port Python to the Nintendo DS, I had no idea what was involved. It was a kind of spur of the moment thing. There is a development environment, devkitarm (part of devkitpro), which allows compilation of homebrew projects as 'roms' which can be loaded onto a Nintendo DS and executed. A part of this environment is support for standard output, and basic text output. Another homebrew coder (Headspin) had written a rom which allowed keyboard input through a picture of a keyboard on the bottom screen where you tap on keys with the stylus to write that text on the upper screen. So, I just compiled the Python source code against this basic setup.

And apart from some straightforward cross-platform compilation issues, which were easily addressable by an existing patch, it was that easy. Python compiled and ran on the Nintendo DS as is.

However, the Stackless Python features, involved a little more work. For Stackless to switch between tasklets by replacing sections of the stack, it needs to be able to do two things, at the assembly level.

1. Get the current address of the stack pointer.
2. Change the current address of the stack pointer.

The other supported platforms, except for x64 under Visual Studio, did this using inline assembly. And the reason that x64 used masm and a standalone assembly file, was because this combination of programming tools and platform do not support inline assembler. So, first, I tried the common easy approach, writing inline assembler.

static int slp_switch(void) {
    register int *stackref, stsizediff;
    __asm__ volatile ("" : : : REGS_TO_SAVE);
    __asm__ ("mov %0, sp" : "=g" (stackref) : );
    {
        SLP_SAVE_STATE(stackref, stsizediff);
        __asm__ volatile (
            "mov r0, %0\n"
            "add sp, sp, r0\n"
            : /* no outputs */
            : "g" (stsizediff)
            : "r0"
        );
        SLP_RESTORE_STATE();
        return 0;
    }
    __asm__ volatile ("" : : : REGS_TO_SAVE);
}

However, this did not seem to work. After littering the Stackless source code with printf statements and rerunning it over and over, adding more and more of them, I still had no idea why. So the next step was to try and run it in a debugger. Insight (a graphical frontend to gdb) comes with devkitarm. However, it does not emulate the DS platform, which means roms cannot be run within it. But it is handy, when it works, for looking at disassembly of functions within a compiled binary. But this was not an option, because the version of Insight that came with devkitarm crashed when a binary was loaded into it. The next step was to try the emulators, some of which sport debugging functionality. But this was also a lot more work than I expected.

• Only two of the debuggers, Dualis and Desmume, would even run my rom. Even then, Dualis would crash to desktop when my rom crashed inside it. Desmume was a little better, and would not crash, when my rom did inside it. This was a little better, because I could see the printf statements that had happened up to the point the rom crashed.
  
• There are no breakpoints. This meant that in order to get the emulator to pause at a specific point in the code, I had to add a loop which held it there for a prolonged period of time. Desmume had single stepping, and would step for a given number of instructions. But Dualis did not. What it did have, that Desmume did not, was syntax highlighting.

What the disassembly in the emulators, especially the more readable Dualis, showed me, was the reason my Stackless support did not work. The code generated by gcc was preserving the stack register (in register r7) which made the inline assembly ineffectual. And it made the Stackless tasklet switching unstable, the stack was being restored with the contents from the tasket being switched to, but the stack pointer itself could not be adjusted to match because gcc prevented this from happening. I could alter the inline assembly to set r7 directly, and this did work, but it was a hack and could not necessarily be relied on to work consistently. This left one option, write the whole switching function in assembler.

The support for masm/x64 was my base for this. I basically wrote equivalent code in ARM thumb assembly (I chose ARM thumb over normal ARM assembler because while it is slower, it generates smaller code and memory is at a premium on the DS). Once I had this written, it worked perfectly.

Lessons learned

• Porting a shell-based application to the Nintendo DS, given a method of input, a method of output and a display for the output to go to, is relatively straightforward. Especially to the degree which Headspin's keyboard code and libnds do.
• Adding support to Stackless Python for another platform is relatively straightforward. The hardest part of my experience with it, was caused because of the lack of debugging support for Nintendo DS homebrew roms.
• Just because Python is on another platform, doesn't mean it is much use. It just shows that for it to be much use, the DS hardware needs to be exposed in some way, so that games and applications can be driven by it.
  

Next steps

As the last of the lessons learned suggests, for Python on the DS to be any use, the DS hardware needs to be exposed. And the best way to do this, is probably to port PyGame. Why PyGame? Because SDL has been ported to the DS for the most part (sound and threading still need work, sound because it relies on the threading support) and PyGame from what I hear is a wrapper around SDL.