Wednesday, 11 November 2009

Comparing Go and Stackless Python

Google has just released a new programming language, called Go. Written by Russ Cox, amongst others, it wraps a custom programming language around low-level functionality very similar to that present in his libtask. With the ability to launch functions as microthreads, and the ability to switch between them using channels, they provide functionality similar to that of Stackless Python.

This post is intended to serve as a comparison of how microthreads and channels are used in two languages that feature them. It is not intended to advocate the choice of one over the other, nor is it guaranteed to be full and complete.

Starting a worker function as a microthread

The availability of lightweight threads that can be used without regard for the resource usage they might incur, means that among other things work can be farmed off to other microthreads while the current microthread does its own thing.

Stackless Python

channel = stackless.channel()

def wrapper(argument, channel):
result = longCalculation(argument)
channel.send(result)

stackless.tasklet(wrapper)(17, channel)
# Do other work in the current tasklet until the channel has a result.
result = channel.receive()
Go
c := make(chan int);

func wrapper(a int, c chan int) {
result := longCalculation(a);
c <- result;
}

go wrapper(17, c);
// Do other work in the current goroutine until the channel has a result.
x := <-c;
There are several things to note from this, including how the different languages handle microthread and channel creation, and the different syntax used respectively.

Creating a microthread

When a given function (in this case wrapper) is to be started as a microthread, the arguments to be passed into it (17 and the channel reference) need to be provided as well. These are set aside for use when the microthread is first scheduled, and the given function starts execution within it.

Stackless Python

A Stackless Python microthread is called a tasklet.
stackless.tasklet(wrapper)(17, channel)
Advantages:
  • Creation of microthreads happens in a function call, returning a reference to the created instance. The instance can be manipulated, allowing amongst other things explicit interruption and killing of the microthread.
    def engage_worker():
    c = stackless.channel()

    def worker():
    # Acquire some result..
    c.send(result)

    worker_tasklet = stackless.tasklet(worker)()
    # Do some work before requesting the result..
    if c.balance != 0:
    # Return the acquired result that is waiting.
    return c.receive()

    # The worker tasklet is still busy and we do not want
    # to wait for it, so abort it and return nothing.
    worker_tasklet.kill()
Go

A Go microthread is called a goroutine.
go wrapper(17, c)
Disadvantages:
  • There does not appear to be a way to store and operate on created microthreads. So the act of creating a microthread as a worker, but manually killing it before its work is complete, appears to be impossible.
One key difference between Go and Stackless Python, is how the tasklet is inserted into the scheduler. In Go, the go keyword explicitly indicates the microthread is being scheduled. While in Stackless Python, the passing of arguments to be used provides the tasklet with the last information it needs to run, and in doing so the tasklet is implicitly inserted into the scheduler.

Creating a channel

In both languages, it is possible to create channels to be used for communication between the microthreads.

Stackless Python
channel = stackless.channel()
Go
c := make(chan int);
Channel operations

Superficially at least, both kinds of channels are similar, allowing the sending and receiving of values through them in much the same way.

Stackless Python

Sending:
channel.send(value)
Receiving:
result = channel.receive()
Go

Sending:
c <- value;
Receiving:
result := <-c;
Microthread memory usage

One of the advantages of using these types of microthreads, is that they do not have the memory requirements that proper operating system threads do. Instead of having one or more megabytes set aside for possible use as a stack, they instead have at most several kilobytes set aside for them.

Stackless Python

Stackless tasklets use as their stack the actual stack of the operating system thread they were created in. This means that when a tasklet blocks and it is set aside to let others run, a chunk of memory is allocated from the heap, and the portion of the stack that has been used by it is copied into that chunk. Then the allocated chunk belonging to the next tasklet to be run is copied back onto the stack, and the chunk freed.

Advantages:
  • Blocked microthreads only use as much memory as they actually used.
  • C function calls can be intermixed with the Python function calls in the call stack of a blocked microthread. This could for instance involve a Python function invoking a C function, which then calls back into Python resulting in the blocking occuring before the stack is unwound.
Disadvantages:
  • Microthreads are linked to the thread they were created in and cannot continue running in any other thread.
It is possible to migrate microthreads from one operating system thread to another in Stackless Python, with the use of its ability to pickle blocked microthreads. However, every function that results in a system call that blocks the interpreter until it completes, would need to be monkey-patched to invoke the migration process. A better solution might be to monkey-patch the relevant functions to do the system calls asynchronously, for instance in the same way as the Stackless socket library does.

Go

Advantages:
  • Microthreads are not linked to the thread they were created in and if that thread is blocked for a system call on behalf of a given microthread, the other microthreads can be migrated to another thread and can continue running there.
Disadvantages:
  • It is not possible to call into C code and have it call back into Go code.
  • For C code to be usable with Go, it needs to be compiled with custom C compilers.
As noted in the Go source code, currently the language defaults to running in single-threaded mode due to multi-threaded operation being unstable. This means that the blocking of a thread for a system call on behalf of a given microthread would in fact block the execution of all the other microthreads until the call is completed.

Monday, 5 October 2009

Stackless Python merge backlog

Over the weekend, I merged and released 2.6.3 and 3.1.1. In addition, I built an installer for 3.1 and also released that. I'm not entirely happy to see that it looks like the 2.6.3 release was rushed and there is talk of releasing 2.6.4 to make up for it.

In any case, I also made a list of the outstanding merges and releases that need to be done.

  • 2.4.5
  • 2.4.6
  • 2.5.3
  • 2.5.4
Hopefully I will find the time to get these done sometime soon, but they'll take a back seat to releases of higher versions, including the fixer-upper release of 2.6.4 should it happen.

Sunday, 4 October 2009

Building a Windows Python installer

It's pretty easy to build an installer for Python these days (2.6 onwards). Yesterday, I built three, one for each version of Python I released for Stackless Python (2.6.3, 3.1 and 3.1.1).

After retrieving the source code for the version of Python you want to build an installer for, you also want to retrieve and compile the source code for all the dependencies (bzip2, sqlite, tcl..). This is almost completely automated for you by the scripts used for the Python buildbots, although you'll want to be using a Visual Studio command prompt so that the compiler is available to the script.

cd release26-maint
Tools\buildbot\external.bat
After executing this script, the dependency source code has been downloaded and built to some extent. You still need to read external.bat and reexecute the TCL and TK compilation commands with DEBUG=1 removed.

Given that you are of course using Windows, you need to do some manual compilation.
cd Tools\msi
nmake -f msisupport.mak
cd ..\..\PC
nmake -f icons.mak
And also build the documentation, to produce the extremely helpful CHM file. The first command retrieves the source code for the documentation generating dependencies, the second builds the CHM file. The documentation build scripts work with versions of Python earlier than 3.0, so you need to make sure that the scripts know where to find a suitable version.
cd Doc
set PYTHON=c:\python26\python.exe
make checkout
make htmlhelp
Next the release build needs to be done in Visual Studio. This actually needs to be done twice, as there are failures (most likely due to dependencies in the build process) in the first attempt.

And with everything now prepared, the installer can be built. As with the documentation, the version of Python used needs to be pre-3.0 and it needs to have the pywin32 extension installed.
cd Tools\msi
c:\python26\python.exe msi.py
If this errors complaining cabarc.exe cannot be found, then execute the SetEnv.cmd script provided by the Microsoft Platform SDK you have installed before rerunning msi.py.

And that's it. At this stage, you'll have a Python installer of your very own, perhaps even with a customised standard library. You can edit some variables at the top of msi.py if you want the built installer to be customised to some extent.

One thing to keep in mind is that this build process is the one used to create standard Python installers. Any installer built using this process for a given version of Python (2.6, 3.1, ...) will be considered identical to any other installer built for that version of Python. Have the official Python 2.6 installed? Then to install Stackless Python 2.6 or your own custom version of 2.6, you will need to uninstall the existing version of 2.6 before you an install a different one. Rather than just being a legacy of the installer build process, this actually serves a purpose. Python installs its DLL in Windows' System32 directory, for 2.6, it will be named python26.dll. It just isn't practical to change the DLL name for a custom Python installation, so the custom installers being identified as the same for Windows' purposes is actually beneficial.