Wednesday, 23 December 2009

DOM key event confusion

There are two standard sets of events that can be used to detect key presses, the first is onkeypress and the second is both onkeyup and onkeydown. Initially I went with onkeypress for my terminal, just to keep it simple.

When you are wanting to receive presses of what are called special keys, using onkeypress becomes a problem. In Firefox you can intercept the pressing of the backspace key, but in Internet Explorer this event never receives it and instead the browser goes back to the previously viewed page. Other keys that onkeypress does not handle in IE, include the cursor keys.

In order to work around this, I switched to onkeyup and onkeydown. Now all key presses, normal or special, work perfectly well in both browsers. However, there is another problem. All keys received are upper case characters, and various keys on the keyboard give symbols rather than their expected characters. The ',' key for instance gives the '¼' character.

Typing the following:

This is a test..,,,.M,,,,
Gives the following output:
THIS IS A TEST¾¾¼¼¼¾M¼¼¼¼
I made countless web searches trying to work out why this was. I even tried manually searching around Stack Overflow. I found web pages where someone was asking how to make the character received from onkeydown upper case, which implied that the problem was on my end. I was pretty much resigned to the existing behaviour, and was planning to use onkeypress to get the entered characters and onkeydown/onkeyup to get the special key presses. However it seemed like a hack, to work around the fact I didn't know what the problem was.

A last ditch question in ##javascript on the FreeNode IRC server yielded the following information:
[16:59] <deltab> chglpnts: keydown and keyup are lower-level, just identifying which *keys* have been pressed; keypress comes later, after the specific combination of keys has been converted into a character
[17:00] <chglpnts> deltab: Ah, I wondered if that was the case, but could find no references. How do I convert for instance 1/4 into , or is the standard way to use keydown/keyup for special keys but still use keypress for normal ones?
[17:00] <deltab> chglpnts: use both
I wonder where this this difference is clearly stated. Searching w3.org for onkeydown does not yield any relevant pages. It is stated in the Detecting keystrokes article on the quirksmode web site, although I was unable to distinguish the meaning until I was already aware of it.

In any case, I am now back on track..

CSS attribute matching and browsers

The goal is to have a blinking cursor. One approach I tried out, was through the use of attribute selectors, where making the cursor visible or not would be a matter of changing an attribute value.

Here's the HTML element:

<font id="cursor" class="cursor" shown="yes">
And the javascript to change the custom "shown" attribute:
if (cursorElement.getAttribute("shown") == "yes")
cursorElement.setAttribute("shown", "no")
else
cursorElement.setAttribute("shown", "yes")
With the following style sheet entries:
.cursor[shown="yes"] {
color: #7799CC;
background-color: rgb(241,128,22);
}

.cursor[shown="no"] {
color: white;
background-color: #7799CC;
}
This works perfectly in Firefox 3.5.6, but Internet Explorer 8.0 completely refuses to recognise the custom attribute. I'm not sure if there is a way in which this can be made to work in IE without defining custom DTD entries, but for now, I've gone back to an approach where I change the element class to make it blink.

Sunday, 20 December 2009

A step towards a Comet-based terminal

While I already have a rough and dirty web page that allows text to be entered, sent to the server and then echoed to anyone viewing the page, it isn't really inspiring enough to take any further. It's overly complicated and hacked together by an advanced manner of cut and pasting, from a selection of code found somewhere on the internet. In order to proceed with any project based on the functionality it is supposed to provide, I need a clean replacement for it.

Motivation

Code does not write itself. Although it was raining today, I needed to find a hook to get myself interested in this project. So, in order to motivate myself to implement a replacement from scratch, I took my inspiration from the AmigaDOS shell.


A thing of marvel, my goal was to get something that looked and appeared to work as close to it as possible.

Challenges

I had a little trouble getting the text to wrap right. Whenever a word which was longer than the width of the window was entered, the text entry continued off unseen outside the right of the window, resulting in the addition of a horizontal scrollbar. With a little CSS research, this was fixed by adding the following two lines:

  word-wrap:   break-word;
overflow-y: scroll;
There is, I have just noticed, one remaining bug. Because the console text is directly manipulated HTML, consecutive spaces are not displayed. I need to add the following:
  white-space:  pre-wrap
And finally, when you've entered enough lines to bring the vertical scrollbar into play, it doesn't automatically scroll. I still need to look into that.

Result

Eventually, I had to decide whether I was going to concentrate on functionality or the detail of its appearance, and functionality was the obvious winner. In its current state, the terminal supports the following:
  • Dragging the "window".
  • Text entry.
  • Movement of the cursor within the current line being entered.
  • The enter key starting a new line.
  • The backspace key removing the character before the cursor.
  • The del key removing the character under the cursor.
You can try it here.. although at this stage, it has only been tested in Firefox 3.5.5.

Saturday, 19 December 2009

WebSocket, js.io and replacing Comet

The prototype of my web-based game client uses Comet for the client-server communication. Recently, I have encountered more and more articles about WebSocket. As it is a cleaner and simpler way of doing what Comet does, I decided to look into the feasibility of adopting it now.

The options

Using WebSockets is not an option at this time. They are not supported to a level which makes them a viable solution, and only bleeding edge versions of the major browsers currently have the required support.

However, there is another possibility. That is, to use a solution which emulates the WebSocket interface as a layer over Comet. I've stumbled across two candidates in my browsing, the first being Kaazing Gateway, and the second js.io. I imagined that adopting one of these, or something similar, would enable me to write cleaner, simpler code that was future proof.

Possible solutions

I can write off Kaazing Gateway immediately. As a commercial product with a large download, the cost of adopting it is immediately higher than I am willing to pay. While it may suit the needs it was designed for, for my needs, there is no reason that the code required should not be lightweight and minimal.

js.io sounded much more promising. However, as a project, it is in a state where its usage is opaque and unclear to interested parties like myself. There is no documentation and the web site refers the user to the source code. It turns out that it both does and does not provides what I am looking for, but the latter aspect was also partly based on my lack of comprehension about the full range of functionality provided by WebSockets.

Researching js.io

Ideally, I wanted a client-side script that simply wrapped the Comet functionality, and allowed it to be used through a WebSocket interface. At least until using WebSocket itself became a viable option. Despite descriptions I had read of js.io painting it as something resembling this, I was unable to find anything within the existing browsable source code which looked suitable.

However, one of the many web searches I did found an older obsoleted script. It turns out that the earlier version of js.io provides exactly what I am looking for. Well, almost.

The earlier version of js.io layers on top of other solutions which provide a required TCPSocket object, specifically Lightstreamer and Sprocket. The default implementation of TCPSocket, which js.io falls back on looking for, is the one provided by Orbited. Unfortunately, these solutions are servers and frameworks, which are too heavyweight to be suitable for my needs.

What next?

Gutting Orbited is one option, but then I am effectively forking it, and the onus is on me to keep an eye on fixes and changes its maintainers make. Probably the best choice at this stage is to stick with Comet.

Monday, 14 December 2009

Documentation for Stackless Python

Stackless Python hasn't had proper documentation. Anyone wanting to use it has had to glean the information they needed from a range of sources, including:

  • The mailing list.
  • Example code that has been written for it, much by me, including my widely used stacklesssocket.py library.
  • The tutorial that was written for it by Grant Olson.
As I volunteer a lot of time and effort towards doing required tasks for Stackless, anything that no-one else has been willing to do, by default stands out as something I need to try and find time for myself.

Recently, I have been wanting to work on some personal projects and unfortunately, Stackless' need for documentation is something I am always aware of. In order to be able to better concentrate on my personal projects, I resolved to bite the bullet and write the documentation, to get it out of the way.

After a week and a bit of work, I have finally finished working on the initial release of documentation for Stackless. As it is late at night here, and I need to call it a night, I have not checked it in or uploaded it to stackless.com yet, but you can see the mirrored files on my web site.

If you find this of use, or for that matter, anything else I have worked on, like the stacklesssocket.py module. Then please consider donating something.

Reverse engineering Amiga software

When I was young I had to walk to school, and it was ten miles both ways, in bare feet, always raining, with snow coming at me sideways on gravel roads. Oh, and everything was a lot more interesting then, because nostalgia hadn't been invented yet.

Living in New Zealand, television shows and movies made their way over here quite a while after they had debuted wherever they originated from. And so did computer games of course. Unfortunately, the only way to get access to software in a manner that didn't require a lifestyle consisting of yacht ownership, smoking jacket wearing or coke sniffing through one hundred dollar bills, was buying pirated disks through the mail.

Reverse engineering

One of the most interesting parts of the pirated software, was the material added by the groups which had originally pirated it. Take for example this video taken from the first disk of Pools of Darkness.


The Amiga console supported an extended range of ANSI commands, some of which allowed moving the cursor an arbitrary number of pixels. By printing a character, then moving the cursor so the next printed character would overwrite some of the last, and repeating this, an image could be rendered in the console. The slow rendering of the image is also a nice effect.

There was one commercial disassembler available for the Amiga, named ReSource. Like IDA Pro, it is an interactive disassembler, an environment where code can be progressively reverse engineered over time.

Here's a screenshot of the SKID_ROW program disassembled in ReSource:

Interestingly, the slow rendering effect is the maximum speed of the console, as the screenshot shows that the complete text to be displayed is written to the console as one lump.

A wiki

In any case, there's a wealth of interesting material preserved from the early days of the Amiga when it was still a contender. If you share my interest in examining it, the most convenient way to do so is by running WinUAE, and perhaps running ReSource within that. But there is also domain knowledge in the tip and tricks to being able to access some of the data, and also what tools can be used in which ways.

I suggested that it would be nice to collect this knowledge on a forum and someone responded by suggesting I start a wiki on one of those sites that are around these days. On a whim, I decided to do so, but in order to make it both easier and more appealing for others to jump in and add their own content, I seeded it with all the information I could find or recall.

It is all very well to start a wiki and throw some information in it. But there must be some level of required information to be present, before others with an interest will consider it a viable place to participate. Or before the effort required on their part to add whatever they may have a whim to, is sufficiently reduced perhaps. I spent a day and a half filling this wiki, before announcing it. It will be interesting to see whether anyone jumps on board.

Link: Reverse engineering for the Commodore Amiga

On a related note

The effect of the text rendering shown above, reminds me of the demo DOS by Andromeda. Embedded Youtube video follows..


Let's face it, AmigaDOS in Kickstart v1.3 just looked plain cool with its blue background and white lines.

So I married an axe murderer

In university, one of the films my friends and I enjoyed the most, was So I Married an Axe Murderer. In it, Mike Myers performs poetry, with some kind of band thing going on.



Today, I came across a link to a segment on Conan O'Brien's show, where William Shatner reads from Sarah Palin's book in the same manner. Then Sarah Palin comes out and surprises him, reading from his autobiography. Both read with the musical accompaniment.



Brilliant.

Sunday, 13 December 2009

My housing situation

Illustrated using other people's pictures..

A mouse in the kitchen


Cockroaches in the kitchen


Burglars visiting the neighbours, but stealing computers.. not ham

Sunday, 22 November 2009

Sriracha chili sauce

One of the foodstuffs more readily available in the United States, was sriracha chili sauce. Pretty much any supermarket I went to had the standard rooster brand. To get it in New Zealand, I need to visit an asian supermarket.

The asian supermarket down the road and round the corner from my flat sells two brands, Mr Number One and Shark Brand.

Mr Number One

2009-11-22 - Sriracha - 01 - Squirt bottle


Interestingly, this has "Tương Ớt Sriracha" on the bottle. The wikipedia page I linked to above says that this is the rooster brand. Not sure why they would use a different label in New Zealand, as the rooster one has such a good reputation.

Shark Brand

2009-11-22 - Sriracha - 02 - Large bottle


20% sugar?! WTF? I might have to stop putting sriracha on my food. Mr Number One has 1g of sugar in a 5g serving, which is the same amount. Still, it could be worse - it could contain high fructose corn syrup.

I'd prefer to photograph the bottles standing up on the kitchen bench, but I'm afraid I'd catch a herd of cockroaches swarming across the wall in the background. If not them, then unsightly stains of unknown foods from the past.

Seasoned Anchovy with Sesame Seed

I caved and tried the Thai seasoned anchovy snacks today. It's strange that such a large amount of anchovy can be so cheap, given how expensive anchovies are in the tin.

2009-11-21 - Groceries - 03 - Seasoned Anchovy with Sesame Seed


At first they taste like harðfiskur, the traditional Icelandic dried fish. Except rather than a large hard lump which you have to tear mouthfuls from, each is a nice bite size. The next flavour that comes through is the sweet sesame coating. In fact, after the first mouthful that is the only flavour that comes through and tends to become overpowering. It makes me wish they sold an unseasoned variety.

2009-11-21 - Groceries - 05 - Seasoned Anchovy with Sesame Seed


I used to love potato chips, but these days they don't taste like food anymore. Instead they just feel like processed gunk in my mouth. Despite the cloying flavour, seasoned anchovies are a nice alternative. I'll definitely have to try the other flavour, which I think was chilli seasoning.

Using Steam in New Zealand

When I buy a game online, I expect to be able to download it as fast as my connection will allow. In New Zealand, living close to some sort of cabinet where proximity is supposed to mean faster connections, this means a standard download speed of ~500k/s.

Steam unfortunately has a standard download speed of ~50k/s.


However, it is a much cheaper way to buy games. I saved $23NZ buying it through Steam instead of through a local store.

Saturday, 21 November 2009

Trader Joe's cheddar cheeses

Besides junk food, the other thing I looked for at Trader Joe's was cheddar cheese. In most of the standard supermarkets, like Target or Walmart, decent cheddar cheese was either expensive or not sold (respectively). Target actually sold Irish products, whether butter or cheddar cheese, but it was as I said expensive.

I don't remember what any of these were like. I think one stood out as having a mature cheddar flavour, and the rest were lacklustre.

Black Diamond vintage reserve cheddar cheese

2009-05-14 - Cheese - Black Diamond


Seaside rugged mature English Cheddar

2009-05-17 - Cheese - Seaside rugged mature english cheddar


Carr Valley Wildflower Cheddar Cheese

2009-05-18 - Cheese - Carr Valley Wildflower Cheddar


Trader Joe's English Coastal Cheddar Cheese

2009-05-21 - Cheese - Trader Joe's English Coastal Cheddar

Asian supermarket groceries

One of the things about shopping in New Zealand, is that a lot of things seem to be cheaper at the asian supermarkets. The one I go to, is a nice walk down the road and round the corner.

A good thing about shopping at an asian supermarket is that they do not have the standard New Zealand junk food sitting about, that I can weaken enough to buy. However, they do have a bizarre range of asian junk food, whether from Thailand, Korea, Japan, China or wherever.

My personal rule is to avoid buying anything chinese because of all the news I have read about pollution and low manufacturing standards in China.

The lot

This only came to $14.20NZD ($10.27USD). Most of that price is the luxury items, which were the Pepero, pickled chilli and the anchovy snacks. I haven't been to the gym since I left the United States, so it is just as well those potatoes promise to give me big muscles.

2009-11-21 - Groceries - 01 - All


Pepero Almond and Chocolate

I was browsing the japanese aisle and as I like chocolate, I had to stop and consider the Pocky on the shelf. Fortunately, Pocky is too expensive for the likes of me, so I bought its cheaper cousin Pepero. Pepero was $1.68NZ, where Pocky was over twice the price.

2009-11-21 - Groceries - 02 - Pepero Almond and Chocolate


The sticks more resemble a small frozen chocolate ice cream, than a chocolate biscuit snack. They were okay, but nothing special.

2009-11-21 - Groceries - 04 - Pepero Almond and Chocolate Sticks


Seasoned Anchovy with Sesame Seed

I mainly picked this up because I recall someone on a cooking show I watched, whether Rick Stein or Anthony Bordain, remarked how some asian culture ate these as a snack. Most of the other snacks, and there are shelves and shelves of them at the asian supermarket, are too alien for me or from China. Because of the cooking show recommendation, I threw these in the basket.

2009-11-21 - Groceries - 03 - Seasoned Anchovy with Sesame Seed


My current plan is to corner a european flatmate and to convince them to act as my taster. When I have junk food open, I tend to eat the lot in sitting. If these are as strong and salty as I expect, they might be self-limiting in how quickly I consume them, which would be a good thing. But until I find reason to open them, I won't know.

Trader Joe's fancy junk food

Trader Joe's has a lot of interesting food, but not a lot that is particularly special. Whenever I had the chance to shop there, I tended to stick to the cheeses and the fancy junk food. This is a haul which I picked up in the company of Liza.

One of the things I miss about the United States is the variety of goods that come with being such a large country. As strange as it sounds I did most of my organic shopping at Walmart. New Zealand junk food has its bright spots, but for the most part it is a pretty lacklustre.

In any case, Trader Joe's was my go to place for interesting junk food that didn't contain crap like high fructose corn syrup.

Trader Joe's Almond Clusters

2009-03-23 - Trader Joe's Almond Clusters - 01 - The box


As a snack, these were okay but nothing special. Just a almondy tasting nut with a chocolatey coating. I can't say that either aspect stood out and I wouldn't recommend them unless you want a generic chocolate nut that doesn't taste cheap.

A closer look:

2009-03-23 - Trader Joe's Almond Clusters - 02 - Contents


Trader Joe's Dark Chocolate Covered Chile Spiced Dried Mango

2009-03-23 - Trader Joe's Chile Spiced Dried Mango - 01 - The box


The mango was more there for texture than flavour, the key thing that stood out was the tang of the chile and the bitterness of the dark chocolate. Definitely worth trying.

A closer look:

2009-03-23 - Trader Joe's Chile Spiced Dried Mango - 02 - Contents


Trader Joe's Thai Lime and Chili Cashews

2009-03-23 - Trader Joe's Thai Lime and Chile Cashews - 01 - The bag


The idea of these is something that could possibly be extremely nice. But in practice they are just okay. I'd recommend them over other generic flavoured nuts and would have picked them up again if I wanted something spicy and tangy.

A closer look:

2009-03-23 - Trader Joe's Thai Lime and Chile Cashews - 02 - Some

Kleinur, the Icelandic donut

One of the many foods I miss from Iceland, are kleinur. Kleinur have the taste and texture of stale donut. The only detriment they have, is that margarine is listed amongst the ingredients.

There's a list of ingredients I avoid, simply because they either taste inferior or the word on them is not good. Among these are high fructose corn syrup, monosodium glutamate and margarine. I used to go to Walmart grocery shopping and by not buying anything with high fructose corn syrup in, it left me with only dozens of items on the shelf that I could buy. Strangely, these tended to be the ones with short lists of commonsense ingredients, as compared to long lists that were chemical and additive related.

In any case, until I realised that kleinur were made from margarine, I used to consider them to be one of the better junk foods I have come across. These were smuggled into the United States, to Stone Mountain and me, from Iceland by a friend. This was done several times by different people, so I do not recall who exactly brought these to me.

The bag:

2009-02-19 - Kleinur - 01 - The bag


Some kleinur separated from the herd:

2009-02-19 - Kleinur - 02 - One


Innards:

2009-02-19 - Kleinur - 03 - Innards


As much as I like kleinur, the margarine is a killer. I should probably look into making my own donuts, let them get stale and see how they compare :-)

Why would anyone buy Krispy Kreme donuts?

Why would anyone buy Krispy Kreme donuts, especially when you have Dunkin Donuts?

We have Dunkin Donuts here in New Zealand, but it doesn't have the same variety of interesting flavours and the donuts are about four times the price. In Stone Mountain, fortunately or perhaps unfortunately, Dunkin Donuts were just down the road and around the corner from me. So, every other Sunday morning, I'd wander down get a cheap coffee and a box of donuts.

However, some day before I got into this habit, Baddi and I were out and about wasting a day and made the decision to pull into Krispy Kreme. He bought a box of glazed donuts and I bought a box of unglazed ones.

Now, an unglazed donut should still have the texture and flavour of a donut. But it turns out an unglazed Krispy Kreme donut has the texture of air and the flavour of nothing, a supreme accomplishment of the american industrial process. A real unglazed donut should taste something like the Icelandic kleinur.

Here's the box of leftovers:

2009-01-26 - Krispy Kreme - 01 - The box


And here's the sole surviving unglazed donut:

2009-01-26 - Krispy Kreme - 02 - One donut left


Where the other unglazed donuts went, I do not know. Neither of us wanted them, it would probably be more nutritious to eat plastic.

Saying goodbye to my fridge in Stone Mountain

I have been meaning to keep track of all the junk food that I have eaten, as part of a plan to keep an eye on my diet. While taking some pictures of New Zealand junk food, I found some old pictures of junk food I had at some stage in the United States.

In this case, it appears I took pictures of my fridge. Before blogging about them, I took a quick look to see what exactly was in them, and I am surprised to see that the least healthy thing present is the magical American milk which is on par with Twinkies for strangely long expiry dates.

Organic vegetables, organic yogurt, imported cheddar cheeses, organic eggs, organic wheatgerm, organic tomato sauce, pepsi max and more. The largesse that comes from having a well-paying job. What was I thinking there, coke zero tastes better than pepsi max..

The main bit:

2009-04-26 - Fridge - 01 - The main bit


And the door:

2009-04-26 - Fridge - 02 - The door


I bought several bottles of the sriracha sauce to take back to New Zealand, but unfortunately weight constraints meant that I could not take them with me and gave them away. Fortunately, the nearest asian supermarket here sells several different varieties which all seem to taste pretty good.

Sims bakery: Tea buns

For a long time, the best bakery in Ashburton was Sims bakery over the bridge in Tinwald. They had a wide variety of appealing food, including cheese steps (white thickly sliced loaves with a tasty melted cheese topping), cheese buns (serving-sized buns also coated with tasty melted cheese) and tea buns which I am showing here. Unfortunately, very few of their remaining products stand out as anything special anymore.

Like chelsea buns, tea buns are a glazed bun with icing on top. They don't particularly taste of anything, but as a diabetic after dinner treat with some extra fattening butter spread within them, they still do the trick.

The remains of a packet of tea buns:

2009-07-31 - Sims bakery - 05 - Tea buns

One of the buns isolated from the herd:

2009-07-31 - Sims bakery - 06 - Tea bun

I can't say I would go out of my way to get to Sims for my junk food anymore, although I would still head there for bakery-style loaves of bread.

Wednesday, 11 November 2009

Comparing Go and Stackless Python

Google has just released a new programming language, called Go. Written by Russ Cox, amongst others, it wraps a custom programming language around low-level functionality very similar to that present in his libtask. With the ability to launch functions as microthreads, and the ability to switch between them using channels, they provide functionality similar to that of Stackless Python.

This post is intended to serve as a comparison of how microthreads and channels are used in two languages that feature them. It is not intended to advocate the choice of one over the other, nor is it guaranteed to be full and complete.

Starting a worker function as a microthread

The availability of lightweight threads that can be used without regard for the resource usage they might incur, means that among other things work can be farmed off to other microthreads while the current microthread does its own thing.

Stackless Python

channel = stackless.channel()

def wrapper(argument, channel):
result = longCalculation(argument)
channel.send(result)

stackless.tasklet(wrapper)(17, channel)
# Do other work in the current tasklet until the channel has a result.
result = channel.receive()
Go
c := make(chan int);

func wrapper(a int, c chan int) {
result := longCalculation(a);
c <- result;
}

go wrapper(17, c);
// Do other work in the current goroutine until the channel has a result.
x := <-c;
There are several things to note from this, including how the different languages handle microthread and channel creation, and the different syntax used respectively.

Creating a microthread

When a given function (in this case wrapper) is to be started as a microthread, the arguments to be passed into it (17 and the channel reference) need to be provided as well. These are set aside for use when the microthread is first scheduled, and the given function starts execution within it.

Stackless Python

A Stackless Python microthread is called a tasklet.
stackless.tasklet(wrapper)(17, channel)
Advantages:
  • Creation of microthreads happens in a function call, returning a reference to the created instance. The instance can be manipulated, allowing amongst other things explicit interruption and killing of the microthread.
    def engage_worker():
    c = stackless.channel()

    def worker():
    # Acquire some result..
    c.send(result)

    worker_tasklet = stackless.tasklet(worker)()
    # Do some work before requesting the result..
    if c.balance != 0:
    # Return the acquired result that is waiting.
    return c.receive()

    # The worker tasklet is still busy and we do not want
    # to wait for it, so abort it and return nothing.
    worker_tasklet.kill()
Go

A Go microthread is called a goroutine.
go wrapper(17, c)
Disadvantages:
  • There does not appear to be a way to store and operate on created microthreads. So the act of creating a microthread as a worker, but manually killing it before its work is complete, appears to be impossible.
One key difference between Go and Stackless Python, is how the tasklet is inserted into the scheduler. In Go, the go keyword explicitly indicates the microthread is being scheduled. While in Stackless Python, the passing of arguments to be used provides the tasklet with the last information it needs to run, and in doing so the tasklet is implicitly inserted into the scheduler.

Creating a channel

In both languages, it is possible to create channels to be used for communication between the microthreads.

Stackless Python
channel = stackless.channel()
Go
c := make(chan int);
Channel operations

Superficially at least, both kinds of channels are similar, allowing the sending and receiving of values through them in much the same way.

Stackless Python

Sending:
channel.send(value)
Receiving:
result = channel.receive()
Go

Sending:
c <- value;
Receiving:
result := <-c;
Microthread memory usage

One of the advantages of using these types of microthreads, is that they do not have the memory requirements that proper operating system threads do. Instead of having one or more megabytes set aside for possible use as a stack, they instead have at most several kilobytes set aside for them.

Stackless Python

Stackless tasklets use as their stack the actual stack of the operating system thread they were created in. This means that when a tasklet blocks and it is set aside to let others run, a chunk of memory is allocated from the heap, and the portion of the stack that has been used by it is copied into that chunk. Then the allocated chunk belonging to the next tasklet to be run is copied back onto the stack, and the chunk freed.

Advantages:
  • Blocked microthreads only use as much memory as they actually used.
  • C function calls can be intermixed with the Python function calls in the call stack of a blocked microthread. This could for instance involve a Python function invoking a C function, which then calls back into Python resulting in the blocking occuring before the stack is unwound.
Disadvantages:
  • Microthreads are linked to the thread they were created in and cannot continue running in any other thread.
It is possible to migrate microthreads from one operating system thread to another in Stackless Python, with the use of its ability to pickle blocked microthreads. However, every function that results in a system call that blocks the interpreter until it completes, would need to be monkey-patched to invoke the migration process. A better solution might be to monkey-patch the relevant functions to do the system calls asynchronously, for instance in the same way as the Stackless socket library does.

Go

Advantages:
  • Microthreads are not linked to the thread they were created in and if that thread is blocked for a system call on behalf of a given microthread, the other microthreads can be migrated to another thread and can continue running there.
Disadvantages:
  • It is not possible to call into C code and have it call back into Go code.
  • For C code to be usable with Go, it needs to be compiled with custom C compilers.
As noted in the Go source code, currently the language defaults to running in single-threaded mode due to multi-threaded operation being unstable. This means that the blocking of a thread for a system call on behalf of a given microthread would in fact block the execution of all the other microthreads until the call is completed.

Monday, 5 October 2009

Stackless Python merge backlog

Over the weekend, I merged and released 2.6.3 and 3.1.1. In addition, I built an installer for 3.1 and also released that. I'm not entirely happy to see that it looks like the 2.6.3 release was rushed and there is talk of releasing 2.6.4 to make up for it.

In any case, I also made a list of the outstanding merges and releases that need to be done.

  • 2.4.5
  • 2.4.6
  • 2.5.3
  • 2.5.4
Hopefully I will find the time to get these done sometime soon, but they'll take a back seat to releases of higher versions, including the fixer-upper release of 2.6.4 should it happen.

Sunday, 4 October 2009

Building a Windows Python installer

It's pretty easy to build an installer for Python these days (2.6 onwards). Yesterday, I built three, one for each version of Python I released for Stackless Python (2.6.3, 3.1 and 3.1.1).

After retrieving the source code for the version of Python you want to build an installer for, you also want to retrieve and compile the source code for all the dependencies (bzip2, sqlite, tcl..). This is almost completely automated for you by the scripts used for the Python buildbots, although you'll want to be using a Visual Studio command prompt so that the compiler is available to the script.

cd release26-maint
Tools\buildbot\external.bat
After executing this script, the dependency source code has been downloaded and built to some extent. You still need to read external.bat and reexecute the TCL and TK compilation commands with DEBUG=1 removed.

Given that you are of course using Windows, you need to do some manual compilation.
cd Tools\msi
nmake -f msisupport.mak
cd ..\..\PC
nmake -f icons.mak
And also build the documentation, to produce the extremely helpful CHM file. The first command retrieves the source code for the documentation generating dependencies, the second builds the CHM file. The documentation build scripts work with versions of Python earlier than 3.0, so you need to make sure that the scripts know where to find a suitable version.
cd Doc
set PYTHON=c:\python26\python.exe
make checkout
make htmlhelp
Next the release build needs to be done in Visual Studio. This actually needs to be done twice, as there are failures (most likely due to dependencies in the build process) in the first attempt.

And with everything now prepared, the installer can be built. As with the documentation, the version of Python used needs to be pre-3.0 and it needs to have the pywin32 extension installed.
cd Tools\msi
c:\python26\python.exe msi.py
If this errors complaining cabarc.exe cannot be found, then execute the SetEnv.cmd script provided by the Microsoft Platform SDK you have installed before rerunning msi.py.

And that's it. At this stage, you'll have a Python installer of your very own, perhaps even with a customised standard library. You can edit some variables at the top of msi.py if you want the built installer to be customised to some extent.

One thing to keep in mind is that this build process is the one used to create standard Python installers. Any installer built using this process for a given version of Python (2.6, 3.1, ...) will be considered identical to any other installer built for that version of Python. Have the official Python 2.6 installed? Then to install Stackless Python 2.6 or your own custom version of 2.6, you will need to uninstall the existing version of 2.6 before you an install a different one. Rather than just being a legacy of the installer build process, this actually serves a purpose. Python installs its DLL in Windows' System32 directory, for 2.6, it will be named python26.dll. It just isn't practical to change the DLL name for a custom Python installation, so the custom installers being identified as the same for Windows' purposes is actually beneficial.

Friday, 2 October 2009

The looming spectre of data corruption?

A few years ago, I was watching the video of a presentation where one of the Subversion developers listed reasons that drove them to create it. One which stood out as being particularly interesting was file corruption in CVS. It went something along the lines of CVS having no verification that the data within it was correct, so it was possible for checked in files to become silently corrupt. I was aware that storage media is to some degree unreliable, but this made me more aware of one of the possible problems that might occur.

Given that corruption of files on storage media is a possibility, even if it might be a rare occurrence, is it worth doing something about? Is there a effective solution to detect that it has happened and to correct the corruption, that is practical to adopt? The case where storage media failure renders a file unreadable can be ignored, if it is considered worth insuring against, other solutions more suited to dealing with it can be used.

PAR2 data verification and repair tools, like QuickPar, might be one possible solution. A script or application that monitors storage media and generates recovery data for files, perhaps also even periodically verifying the files once in a while, actually wouldn't be that difficult to write.

QuickPar screenshot
Okay, so this isn't a problem that people generally worry about. After all, it is hard enough to find the time and energy to ensure that you are backing up all of your important data as it is. But it is something I can see being worth doing for piece of mind.

Monday, 7 September 2009

Thanks to the Python Software Foundation

In order to maintain Stackless Python, I need to have access to the Microsoft development tools in order to do things like build installers. The free versions available these days are great, but unfortunately, they don't provide the range of functionality I need. And as such, I have been unable to release installers for recent releases.


Fortunately, Steve Holden has been arranging for Microsoft to donate licenses to people who support use of Python on Windows, and I just received an email today letting me know that I have an MSDN subscription for the next year. You can see the two versions of Visual Studio required for Python in the screenshot above (Visual Studio 2003 .NET and 2008).

C.C.P. was unable to lend me some of their internal licenses, but they did offer to buy me copies of the required versions of Visual Studio. While this would be nicer for me personally, the MSDN subscription gives me access to other things like Windows 7 which is nicer for Stackless Python, assuming I have the time to make use of them.

Anyway, thanks again to Steve and the PSF!

Edit: 2009-09-08: Added picture of validated MSDN subscription contents and comments about VS version availability.

Friday, 21 August 2009

Buying a second-hand car: WoF

As I am looking at buying a second-hand car, I've been doing a bit of research about the options. One thing I didn't know, was what someone posted on Emigrate NZ:

No one is allowed to sell a car without a WOF which is less than one month old (exception being brand new cars that dont require wofs for a few years as they are brand new).

Usually that means the car you are buying has to have a new wof before you buy it.

It is possible to sell a car without a WOF, but only if it is sold "as is"
Doing a quick google for "second hand car wof site:govt.nz" gives me this link on the Land Transport NZ web site corroborating the claim.

Thursday, 6 August 2009

Sims bakery: Steak and kidney pie

Another interesting pie that I bought with the lambs fry and bacon one, was a steak and kidney pie. On the outside it looks like any other potentially satisfying pie.

2009-07-31 - Sims bakery - 03 - Steak and kidney pie

But the oozing innards unfortunately resemble its sibling. There was too little meat substance and too much gravy. Unlike the gravy in the lambs fry pie, the steak and kidney gravy was not floury and pasty, and had some flavour. But the texture the predominance of gravy brought to the pie just didn't make it a good one.

2009-07-31 - Sims bakery - 04 - Steak and kidney pie innards

Sims also do an interesting sounding Venison pie, but at a whopping price of $6.00NZ with what experience dictates will be a distinct lack of meatiness, I'm not springing for it.

Mollweide projections

Distracted!

With working code to generate mollweide projections of my planetary surface and display it in a pyglet window, I got a little distracted. I ditched the window with its default width and height and generated images of a width and height with correct aspect ratio. Given there are 360 degrees of longitude and 180 degrees of latitude, the width needed to be twice the height. With this working, I randomly generated seeds for the noise and images of the corresponding projection for each seed. I kind of wanted to find a nice looking surface with many separate islands, and not just one or two planet wrapping islands.

A simple loop generates the images. I don't really need to use the Wichmann Hill generator that comes with Python, but I've been using it get to get small extractable seeds so that I can reseed the generator and get reproducible sequences. Irrelevantly, I am not illustrating this in the following code snippet.

    rg = random.WichmannHill()
for i in range(2000):
v = int(rg.random() * sys.maxint)
terrain.set_noise_seed(v)
generate_mollweide_image("mollweide/water-%011d.jpg" % v, 1000, 500)
Keep in mind that the projection attaches to the sides and not the top and bottom. So an island that features on the edge of the left or right hand side will wrap to the other side.

75893359 (like the majority of other seeds) produces a surface with one overly large island wrapping the lower half of the planet.


1320502026 produces a nicer planet with one larger island, several of medium size and a scattering of smaller ones.


While 1898084921 produces one huge island, it is unique and interesting in having two huge inland bodies of water.


Lakes, mountains and a range of interesting islands of different sizes are present in 2029406243.


And there I am distracted into looking for interesting planetary surfaces again. At least it resulted in me optimising the code from taking 31 seconds to generate the height data of a given seed (640x480) to taking 1.8 seconds to generate the height data for a larger area (1000x500).
    def generate_mollweide_image(fileName, width, height):
global heights, height_min, height_max

print "generating heights"
t = time.time()
xscale = width
yscale = int(height * 1.3) # Cover more lines and leave no black ones.
heights = terrain.mollweide_projection(8, xscale, yscale, width, height)
height_min = min(heights, key=lambda x: sys.maxint if x is None else x)
height_max = max(heights, key=lambda x: -sys.maxint if x is None else x)
dt = time.time() - t
print "generated heights", dt, "seconds", (height_min, height_max)

print "generating colours"
t = time.time()
imageData = generate_image_data(width, height)
dt = time.time() - t
print "generated colours", dt, "seconds"

print "saving image"
mapImage = pyglet.image.create(width, height)
mapImage.set_data('RGB', width * 3, imageData.tostring())
mapImage.save(fileName)
All of the height generation now happens in a C extension (terrain.mollweide_projection), which contributed to half of that optimisation.

Here's the original Python height generation routine:
def generate_mollweide_heights_xy():
global heights, height_min, height_max

heights = [ None ] * (w.height * w.width)

xscale = w.width
yscale = int(w.height * 1.3) # Cover more lines and leave no black ones.

longitude0 = 0.0

for yi in xrange(yscale):
v = float(yi) / yscale

# Stay within the domain of -pi/2 to pi/2
latitude = (v - 0.5) * math.pi

dtheta = 0.149
theta = latitude
pi_sin_lat = math.pi * math.sin(latitude)
i = 0
while math.fabs(dtheta) > 5e-8:
if i == 30:
break
dtheta = -(theta + math.sin(theta) - pi_sin_lat) / (1.0 + math.cos(theta))
theta += dtheta
i += 1
theta = theta / 2.0

for xi in xrange(xscale):
u = float(xi) / xscale

xj = (u - 0.5) * math.cos(theta)
yj = math.sin(theta)

xc = int((xj + 0.5) * w.width)
yc = int(((yj + 1.0) / 2.0) * w.height)

longitude = u * math.pi * 2.0 - longitude0
x = math.cos(longitude) * math.cos(latitude)
y = math.sin(longitude) * math.cos(latitude)
z = math.sin(latitude)

heights[yc * w.width + xc] = terrain.fBm(8, x, y, z)

filtered_heights = [ x for x in heights if x is not None ]
height_min = min(filtered_heights)
height_max = max(filtered_heights)
It might be useful for someone who would also otherwise try and plug in the formulae present at Wikipedia or Mathworld.

I couldn't get the inverse transformation to produce anything that looked vaguely correct. And the forward transformation was much the same until I ditched a few elements and figured out the domain of the variables.

The corresponding C extension function:
static PyObject *terrain_mollweide(PyObject *self, PyObject *args) {
PyObject *list, *ob;
int octaveCount;
float u, v, du, dv;
float longitude, latitude;
float dtheta, theta, sin_lat, cos_lat, cos_theta;
float x, y, z;
float xf, yf, xiterations, yiterations;
int i, width, height, arr_size, arr_idx;
float *arr;

if (!PyArg_Parse(args, "(iffii)", &octaveCount, &xiterations, &yiterations, &width, &height))
return NULL;

arr_size = width * height;
arr = malloc(arr_size * sizeof(float));
if (arr == NULL) {
PyErr_SetString(PyExc_RuntimeError, "failed to allocate working memory");
return NULL;
}

for (i = 0; i < arr_size; i++)
arr[i] = -1.0f;

du = 1.0f / xiterations;
dv = 1.0f / yiterations;

v = 0.0f;
while (v < 1.0f) {
latitude = (v - 0.5f) * F_PI;

theta = latitude;
sin_lat = sinf(latitude);
cos_lat = cosf(latitude);

i = 0;
dtheta = 0.149f;
while (fabs(dtheta) > 4.9e-08f) {
if (i == 30)
break;
dtheta = -(theta + sinf(theta) - F_PI * sin_lat) / (1.0f + cosf(theta));
theta += dtheta;
i += 1;
}
theta = theta / 2.0f;
yf = ((sinf(theta) + 1.0f) / 2.0f);

cos_theta = cosf(theta);
u = 0.0f;
while (u < 1.0f) {
xf = (u - 0.5f) * cos_theta + 0.5f;

longitude = u * F_PI * 2.0f;
x = cosf(longitude) * cos_lat;
y = sinf(longitude) * cos_lat;
z = sin_lat;

arr_idx = width * (int)(yf * height) + (int)(xf * width);
if (arr_idx < 0 && arr_idx >= arr_size) {
free(arr);
PyErr_SetString(PyExc_RuntimeError, "internal out of range error");
return NULL;
}

arr[arr_idx] = fBm(octaveCount, x, y, z);

u += du;
} /* while(u < 1.0f) */

v += dv;
} /* while(v < 1.0f) */

list = PyList_New(width * height);
if (list == NULL) {
free(arr);
return NULL;
}

for (i = 0; i < arr_size; i++) {
if (arr[i] == -1.0f) {
Py_INCREF(Py_None);
ob = Py_None;
} else {
ob = PyFloat_FromDouble(arr[i]);
if (ob == NULL) {
Py_DECREF(list);
free(arr);
return NULL;
}
}
PyList_SET_ITEM(list, i, ob);
}

free(arr);
return list;
}
The image generation and colouring is now the slowest part of the process. Even worse is that the array and string shenanigans that pyglet insists on encounters MemoryError exceptions with larger sized projections. But really, neither of these things matter as the map projection is serving its purpose.

Wednesday, 5 August 2009

Map projections with Pyglet

In 2001, I wrote code to generate fractal-based landscapes and wanted to visualise what it produced using map projections. The projection I was most familiar with was the mercator projection, and while it showed that the generation was working, it wasn't ideal for visualisation due to the distortion present within it. So I played around with other projections and eventually settled on an orthographic projection. The end result was an animated gif of around a megabyte in size that showed a generated planet rotating. Unfortunately, at some point in time, the code was lost.

With a bit of time to spare to create some similar code, this time I decided on using the mollweide projection. Pygame has always been a little too complicated, sure there is documentation, but more often than not when I try and use a feature there will be an non-obvious little detail that prevents it from working correctly if at all. So this time, I decided to use Pyglet.

I've used Pyglet in the past, and had a simple script that used OpenGL to display a grid and move around it on the XZ plane. Unpromisingly, this script no longer worked and the grid wouldn't display. It turns out that Pyglet has changed the way their window events work. My resize event was being called:

@window.event
def on_resize(width, height):
# Override the default on_resize handler to create a 3D projection
glViewport(0, 0, width, height)
glMatrixMode(GL_PROJECTION)
glLoadIdentity()
gluPerspective(60.0, width / float(height), 0.1, 1000.0)
glMatrixMode(GL_MODELVIEW)
But it turns out that now if you do not return pyglet.event.EVENT_HANDLED from this event, Pyglet decides to assert some default behaviour. Some helpful bloke from the Pyglet mailing list pointed me in the right direction with this.

During the period where I was unaware that this non-backwards compatible change had occurred I tried two different examples from the Pyglet documentation. Both errored due to bugs in the current release of Pyglet (1.1.3). It seems that Pyglet isn't in a stable state at the moment, and given that 1.1.3 is a maintenance release or something, I am not sure it ever is. I tracked down bug reports for the problems and noted they were considered to be fixed in the trunk, both at different points in time months before 1.1.3 was released. It would be nice to be able to see a list of all the known bugs with the current release (especially since any I encounter already seem to be fixed) so that I as a user do not have to waste my time unexpectedly experiencing them, tracking down the problem and trying to report them. The best approach with regard to Pyglet when I experience a bug seems to be ignoring the bug reporting system and just looking to see if a fix has been made in the trunk and manually patching it into my own code.

I mentioned something above about PyGame. I'd also like to say that sure there is documentation for Pyglet, but more often than not when I try and use a feature there will be an non-obvious little detail that prevents it from working correctly if at all.

In any case, eventually I reached the point where I had written code to do a mollweide projection and had it displaying in the PyGame window, looking the following way.



I am also saving out the images I generate, in order to take not of different approaches and mistakes made during the process of development. By itself Pyglet can only save out PNG images, so this was what it saved out to a PNG file.



Now one picture is a little bright and the other is a little dark. The PNG images saved by Pyglet have darker colours. I initially tried to compensate by continually brightening the map colours. The brighter image, which is identical to what is displayed by Pyglet, was saved out as a JPG image after I had installed PIL. If PIL is present, than Pyglet can handle a wider range of image formats.

Saturday, 1 August 2009

Merging Stackless Python 3.1

Releasing a new version of Stackless Python is never a straightforward experience. There are always a range of presumably avoidable problems that seem to pop up and slow the process down.

This time the problems encountered were:

  • Pre-existing failures in the Python test suite.
  • Changes in the standard Python pickling unit test approach.
  • Dynamic behaviour in the unittest.py script involving itself in the Stackless function pickling code.
  • TortoiseSVN giving a range of incomprehensible errors.
Pre-existing standard unit test failures

When there are failures in the Python unit tests in the Stackless build, a good step to tracking them down is to define STACKLESS_OFF and compile the Stackless source code into standard Python. If the tests fail there, then it might be a mismerge, or it might be that the tests fail in the standard Python branch that was merged from. After compiling and testing the standard Python branch and not seeing test failures there, that then points to a mismerge.

In this case, the same tests failed to pass in the standard Python branch. While these false negatives increase the time and effort required to do the merge, they are of course better than having mismerged. They can also be ignored, as the goal of Stackless is to be backwards compatible with standard Python...

The first Python unit test failure was in the distutils tests. It looks like the exported symbol prefix for module initialisation has changed from init to PyInit_, but a unit test had not been changed to generate a method with the new prefix.
test_distutils
xxmodule.c
Creating library c:\users\richard\appdata\local\temp\tmpbc9aj_\Debug\users\richard\appdata\local\temp\tmpbc9aj_\xx_d.lib and object c:\users\richard\appdata\local\temp\tmpbc9aj_\Debug\users\richard\appdata\local\temp\tmpbc9aj_\xx_d.exp
foo.c
LINK : error LNK2001: unresolved external symbol PyInit_foo
c:\users\richard\appdata\local\temp\tmp6u2yky\tempt\users\richard\appdata\local\temp\tmpa9gpqp\foo_d.lib : fatal error LNK1120: 1 unresolved externals
[37422 refs]

D:\SVN\_python\python-branches\py3k-export\py3k>exit 1

D:\SVN\_python\python-branches\py3k-export\py3k>exit 0
test test_distutils failed -- Traceback (most recent call last):
File "D:\SVN\_python\python-branches\py3k-export\py3k\lib\distutils\msvc9compiler.py", line 635, in link
self.spawn([self.linker] + ld_args)
File "D:\SVN\_python\python-branches\py3k-export\py3k\lib\distutils\ccompiler.py", line 981, in spawn
spawn(cmd, dry_run=self.dry_run)
File "D:\SVN\_python\python-branches\py3k-export\py3k\lib\distutils\spawn.py", line 36, in spawn
_spawn_nt(cmd, search_path, dry_run=dry_run)
File "D:\SVN\_python\python-branches\py3k-export\py3k\lib\distutils\spawn.py", line 77, in _spawn_nt
"command '%s' failed with exit status %d" % (cmd[0], rc))
distutils.errors.DistutilsExecError: command '"c:\Program Files\Microsoft Visual Studio 9.0\VC\BIN\link.exe"' failed with exit status 1120

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "D:\SVN\_python\python-branches\py3k-export\py3k\lib\distutils\tests\test_build_ext.py", line 320, in test_get_outputs
cmd.run()
File "D:\SVN\_python\python-branches\py3k-export\py3k\lib\distutils\command\build_ext.py", line 347, in run
self.build_extensions()
File "D:\SVN\_python\python-branches\py3k-export\py3k\lib\distutils\command\build_ext.py", line 456, in build_extensions
self.build_extension(ext)
File "D:\SVN\_python\python-branches\py3k-export\py3k\lib\distutils\command\build_ext.py", line 543, in build_extension
target_lang=language)
File "D:\SVN\_python\python-branches\py3k-export\py3k\lib\distutils\ccompiler.py", line 791, in link_shared_object
extra_preargs, extra_postargs, build_temp, target_lang)
File "D:\SVN\_python\python-branches\py3k-export\py3k\lib\distutils\msvc9compiler.py", line 637, in link
raise LinkError(msg)
distutils.errors.LinkError: command '"c:\Program Files\Microsoft Visual Studio 9.0\VC\BIN\link.exe"' failed with exit status 1120
The second unit test failure was within the socket unit tests, something about binding them.
test_socket
test test_socket failed -- Traceback (most recent call last):
File "D:\SVN\_python\python-branches\py3k-export\py3k\lib\test\test_socket.py", line 498, in testSockName
sock.bind(("0.0.0.0", port))
socket.error: [Errno 10048] Only one usage of each socket address (protocol/network address/port) is normally permitted
Legitimate standard unit test failures

Recompiling the merged source code with Stackless enabled showed an additional unit test failure in test_pickle.py. Stackless allows pickling of running code and in order to do this, it alters what gets pickled. In this case, the fix was simply pickling the problematic object, taking the result and and updating the unit test to include it. Of course, the case where Stackless is disabled still needs to be handled.
try:
import stackless
DATA4 = b'\x80\x02cstackless._wrap\nrange\nq\x00K\x00K\x05K\x01\x87q\x01Rq\x02)b.'
except:
DATA4 = b'\x80\x02c__builtin__\nxrange\nq\x00K\x00K\x05K\x01\x87q\x01Rq\x02.'
Stackless pickling unit test failure

There was one failure in the Stackless unit tests, to do with pickling a recursive function. Editing the test and reconciling what comes out of pickletools.dis with the test failure stack trace made it clear what the problem was. It was dynamic behaviour in the unittest.py module, because there were self references in the recursive function being pickled, the test suite instance was being included with the pickled function. On unpickling, a __getattr__ hook in the _WritelnDecorator text UI test running support class was entering an infinite loop. The correct fix here was to remove the self reference to stop the test case instance being dragged into the pickle.

TortoiseSVN problems

It used to be that TortoiseSVN just worked. I'd go through the trauma of merging a Python branch and building an installer, and then I could rely on it to do its job. But these days, every time I do a merge, it seems that TortoiseSVN gets flakier and flakier.

When I went to commit the merged Stackless code into our py3k branch, TortoiseSVN complained about how a new file from the original Python branch was already present. I tried to do a "clean up" on the relevant directory, which did not work. I tried to exclude the relevant directory from the commit operation, which resulted in another unclear and apparently irrelevant error to do with recursive commits needing to be done a certain way. Eventually, what worked was reverting the directory and readding it.

With the merged changes committed, there was one more code change to do. The configure script is a unix script and is generated from another file, configure.in. It does not merge well, so the best approach is to not merge it and once all the other merges are checked in, fetch the branch on a Linux machine and rebuild configure there. Then I download the regenerated file to my local machine via the web, and commit it locally. Normally is as straightforward as it sounds and just works, but this time, TortoiseSVN complained about inconsistent line endings. The file has the subversion property of native line endings, and in the past it was just accepted, but not any more. What worked was loading the file up in a decent editor that allowed me to save it out in "unix" format.

Doing the release

I am not releasing Stackless Python 3.1 yet. In order to allow extensions to be compatible with Python builds, there is an official version of Visual Studio that needs to be used for both of these things. And I do not have access to an install of Visual Studio 2008 at this time. At some later stage, I might have the time and interest to download Visual C++ Express 2008, and build the relevant binaries. But I cannot guarantee that the express version of Visual Studio can build installers anyway, which might make it pointless.

So, at this time, Stackless Python 3.1 is not officially released.

Friday, 31 July 2009

IOCP-based sockets with ctypes in Python: 7

Previous post: IOCP-based sockets with ctypes in Python: 6

In the last attempt, I managed to get a minimal amount of standard socket code making use of the functionality Stackless Python provides, working within my script. But if I am to make a proper replacement socket module, I need to be able to use it in the same way stacklesssocket.py is used. To this end, the use case I want to support is Andrew Dalke's "more naturally usable" example script.

As can be seen in the script, the following code was the way the replacement socket module was put in place:

import stacklesssocket
sys.modules["socket"] = stacklesssocket
Back in Python 2.5 when it was being developed, the socket module was a lot simpler and the approach of making a complete replacement module was feasible. These days however, in later versions of Python 2 and also in Python 3, the socket module has become somewhat more complicated and the approach of leaving the standard version in place and substituting parts became practical. This lead to new code to substitute in that functionality:
import stacklesssocket
stacklesssocket.install()
The goal

Andrew's example script, the supporting of which is my goal, contains the following code:
import sys
import stacklesssocket
import stackless

stacklesssocket.install()

import urllib
import time

def download(uri):
t1 = time.time()
f = urllib.urlopen(uri)
s = f.read()
t2 = time.time()
print "Downloaded", uri, "in", "%.1f" % (t2-t1), "seconds"
return t2-t1


print " === Serial === "
t1 = time.time()
download("http://www.stackless.com/wiki/Tasklets")
download("http://www.stackless.com/wiki/Channels")
t2 = time.time()
print " --->", t2-t1

print " === Parallel === "
t1 = time.time()
stackless.tasklet(download)("http://www.stackless.com/wiki/Tasklets")
stackless.tasklet(download)("http://www.stackless.com/wiki/Channels")
stackless.run()
t2 = time.time()
print " --->", t2-t1
The most important aspect of this, is how it uses another higher level standard library module (urllib) that in turn uses the standard socket module. By using my replacement functionality in this way, I can verify how compatible and stable it might be.

The implementation

The original stacklesssocket.py module was based on asyncore, another framework that wraps select. Back when I wrote it, I was glad to have it there to use, but the fact is that adopting a framework brings in dependencies and abitrary abstraction. If I had the time and interest, I would rewrite stacklesssocket.py to use select directly.

At this time, I am developing for Stackless Python 2.6.2. Right off the bat, it was obvious that this new module was going to be somewhat simpler. There were several substitutions to the standard socket module for stacklesssocket.py, but in this case it looks like I just need to subsitute my own _realsocket object.
def install():
stdsocket._realsocket = socket
The first obstacle was my lack of a connect method on my replacement socket object. Checking out the available functions on MSDN, ConnectEx was the one which suited my overlapped IO needs. However, I was unable to find it exported from any Winsock related DLL. It turns out, you need to call WSAIoctl in order to get a pointer to it. I had been avoiding calling WSAIoctl as I assumed it was only used to make faster function calls, but as it is the gateway to functionality I need that is no longer something I can do.
int WSAIoctl(
__in SOCKET s,
__in DWORD dwIoControlCode,
__in LPVOID lpvInBuffer,
__in DWORD cbInBuffer,
__out LPVOID lpvOutBuffer,
__in DWORD cbOutBuffer,
__out LPDWORD lpcbBytesReturned,
__in LPWSAOVERLAPPED lpOverlapped,
__in LPWSAOVERLAPPED_COMPLETION_ROUTINE lpCompletionRoutine
);
The corresponding ctypes definition is as follows:
WSAIoctl = windll.Ws2_32.WSAIoctl
WSAIoctl.argtypes = (SOCKET, DWORD, c_void_p, DWORD, c_void_p, DWORD, POINTER(DWORD), POINTER(OVERLAPPED), c_void_p)
WSAIoctl.restype = c_int
The appropriate dwIoControlCode value to get a function pointer is SIO_GET_EXTENSION_FUNCTION_POINTER. The value of lpvInBuffer to indicate that the pointer to ConnectEx is the one I want, is the following GUID.
{ 0x25a207b9,0xddf3,0x4660, { 0x8e,0xe9,0x76,0xe5,0x8c,0x74,0x06,0x3e }}
I'm not doing this as an overlapped operation, as I can't see how it could end up blocking in any meaningful way. This resulted in the following code in the replacement sockets __init__ method:
        self.wsFnConnectEx = ConnectExFunc()
dwBytes = DWORD()
ret = WSAIoctl(_socket, SIO_GET_EXTENSION_FUNCTION_POINTER, byref(WSAID_CONNECTEX), sizeof(WSAID_CONNECTEX), byref(self.wsFnConnectEx), sizeof(self.wsFnConnectEx), byref(dwBytes), cast(0, POINTER(OVERLAPPED)), 0)
if ret == SOCKET_ERROR:
err = WSAGetLastError()
closesocket(ret)
raise WinError(err)
With this out of the way, I can now move onto fleshing out the connect method.
    def connect(self, address):
host, port = address

self.bind(("0.0.0.0", 0))

sa = sockaddr_in()
sa.sin_family = AF_INET
sa.sin_addr.s_addr = inet_addr(host)
sa.sin_port = htons(port)

bytesSent = DWORD(0)
ovConnect = OVERLAPPED()
c = ovConnect.channel = stackless.channel()

ret = self.wsFnConnectEx(self._socket, sa, sizeof(sa), 0, 0, NULL, byref(ovConnect))
if ret == FALSE:
err = WSAGetLastError()
# The operation was successful and is currently in progress. Ignore this error...
if err != ERROR_IO_PENDING:
raise WinError()

activeIO[self._socket] = c
c.receive()
ConnectEx was not working on the first attempt, it turns out you need to bind a socket to a local address before you can connect using it. Oh well, time to retry Andrew's script and see how I go.

This time, the lack of a sendall method was preventing the script from running. sendall just continues calling send for the same set of data, until all of it has been sent.
    def sendall(self, msg, flags=None):
bytesSent = self.send(msg)
while bytesSent < len(msg):
bytesSent += self.send(msg[bytesSent:])
At this point I decided to take note of the fact that it is possible for operations that are performed with an overlapped object, may actually complete right away. However, my mistake was in assuming that if this happened, a completion packet wouldn't arrive through GetQueuedCompletionStatus. This led to crashes where I examined the channel attribute on arrived packets, where the overlapped object had already been garbage collected due to the exit of the function that started the operation.

In order to clean things up, I decided to store a reference to the overlapped object ina global variable and in addition move the channel reference I hold also outside and alongside of that overlapped object. I could have stored the socket handle in the overlapped object, but since CreateIOCompletionPort allows me to associate a completion key with the socket, I store it there instead.
        # Bind the socket to the shared IO completion port.
CreateIoCompletionPort(_socket, hIOCP, _socket, NULL)
The idea was that I could then ignore packets that arrived for earlier operations which had actually completed immediately. Of course, logic having gotten a result for an IO operation may try another which might this time block. This means that unwanted packets for the operations that had completed immediately would come in and assume they were for the followup blocking operation. Errors ensued. So each completed packet needs to be able to be matched up with the operation state stored for it.

This left connect in the following state:
    def connect(self, address):
host, port = address

self.bind(("0.0.0.0", 0))

sa = sockaddr_in()
sa.sin_family = AF_INET
sa.sin_addr.s_addr = inet_addr(host)
sa.sin_port = htons(port)

bytesSent = DWORD(0)
ovConnect = OVERLAPPED()
opID = ovConnect.opID = self.__getnextopid()
c = stackless.channel()
c.preference = 0
ovConnect.label = "connect"

activeOps[(self._socket, opID)] = (c, ovConnect)

ret = self.wsFnConnectEx(self._socket, sa, sizeof(sa), 0, 0, NULL, byref(ovConnect))
if ret == FALSE:
err = WSAGetLastError()
if err != ERROR_IO_PENDING:
raise WinError()

c.receive()
The other overlapped IO operations also needed to be changed to match, along with _DispatchIOCP.
def _DispatchIOCP():
numberOfBytes = DWORD()
completionKey = c_ulong()
ovCompletedPtr = POINTER(OVERLAPPED)()

def _GetCompletionChannel(completionKey, overlappedPtr):
_socket = completionKey.value
opID = ovCompletedPtr.contents.opID
k = _socket, opID
c, ovRef = activeOps[k]
del activeOps[k]

return c

while stackless.runcount > 2:
while True:
stackless.schedule()

c = None
ret = GetQueuedCompletionStatus(hIOCP, byref(numberOfBytes), byref(completionKey), byref(ovCompletedPtr), 50)
if ret == FALSE:
err = WSAGetLastError()
if err == WAIT_TIMEOUT:
continue

if not bool(ovCompletedPtr):
raise WinError(err)

c = _GetCompletionChannel(completionKey, ovCompletedPtr)
c.send_exception(WinError, err)
continue

break

c = _GetCompletionChannel(completionKey, ovCompletedPtr)
if c.balance == -1:
c.send(numberOfBytes.value)
One note is that accessing ovCompletedPtr.contents will error if the value is NULL, the safe way with ctypes to ascertain whether the value is NULL is to check if the boolean value is False.

Running Andrew's script at this point results in it exiting cleanly, but mid-execution. The reason for this is the condition stackless.runcount > 2. The theory is that this should keep the loop running as long as there are at least two tasklets in the scheduler, this is just plain wrong and ill thought out. The fact is that at any given time, at least for Andrew's script, the runcount is only likely to be one. The reason for this is that all the other tasklets doing IO will be outside of the scheduler, blocked on a channel. So the lowest possible valid value for runcount that will require continued looping will be one.

Taking this into account, I added a weakref dictionary to track the socket objects which were live and added it as an additional reason to loop. This in no way addresses the above problem, but it does the trick. Andrew's script now runs, and gives the following output:
 === Serial ===
Downloaded http://www.stackless.com/wiki/Tasklets in 3.4 seconds
Downloaded http://www.stackless.com/wiki/Channels in 3.9 seconds
---> 7.26900005341
=== Parallel ===
Downloaded http://www.stackless.com/wiki/Channels in 3.3 seconds
Downloaded http://www.stackless.com/wiki/Tasklets in 3.5 seconds
---> 3.51300001144
Success (of a sort)!

Of course the module in no way provides the full functionality of the standard version yet. Trivial aspects no-one in their right mind should be using, like UDP and so forth, are completely unsupported.

Script source code: 06 - Polished Socket.py