Tuesday, 16 September 2008

PyPy: Using the Visual C++ Toolkit Compiler 2003

I've been meaning to experiment with PyPy for a while. There are a couple of projects I have in mind which might find some of the functionality it provides useful:

  • A text-based MUD programming environment using the sandboxing to allow programming and invocation of Python.
  • A reimplementation of the MudOS driver using translation to interpret LPC.
Before I start playing with it, I want to build the C translation of the PyPy Python interpreter with Stackless functionality. I'm going to note the steps I had to take, in order to get this to work in this post, for the purpose of reference.

The compilation PyPy does through its distutils module needs to be done with the same version of Visual Studio that the Python installation it is using was compiled with. Visual Studio .NET 2003 (7.1) is the proper Windows compilation environment for Python 2.5. I don't have VS .NET 2003 but I do have the free MS Toolkit Compiler set up for Python development. In order to get distutils to recognise it, I followed the instructions which Mike Fletcher provides.

I've already installed the pre-compiled Boehm garbage collection files, which the PyPy web site provides.

The command I need to execute in order to translate PyPy the way I want is: "c:\python25\python.exe --stackless translate.py --opt=3 targetpypystandalone.py".

It errors to a debugging prompt claiming it can't find 'windows.h'. This is my fault due to the way I have the MS Toolkit set up. In order to compile things with it I need to open the Microsoft Platform SDK for Windows Server 2003 R2 / Windows XP 32 bit / Retail build environment DOS window, then execute the BAT file which adds the environment variables for the Toolkit over and above that. I had just opened a normal DOS window and executed the BAT file before trying to compile.

It errors to a debugging prompt claiming it can't find 'zlib.h'. Apparently some option is set to enable zlib support in the process. I can't immediately see how to turn it off and I am not sure that I don't want zlib support. So I download zlib 1.2.3 and compile the source code, adding its path to the 'Include' environment variable and restart the compilation process

It errors to a debugging prompt claiming it can't find 'bzlib.h'. Same deal as with zlib. I already have the bz2 downloaded and compiled, as it is a dependency need for compilation of Python, which I need to do for Stackless. So I just add its path to the 'Include' environment variable and restart the compilation process.

It errors to a debugging prompt claimign it can't find 'bz2.lib'. This makes sense. However, one problem is that this is not what the lib is called in the output of the build process in my environment, it gets compiled as 'libbz2.lib'. I copy and rename it to 'bz2.lib' and add the path to the 'Lib' environment variable and then restart the compilation process.

Success. The compilation proceeds and during the process, a mandelbrot is drawn with characters in the DOS window as the PyPy web site describes. It is still compiling at this point and lots of generic looking cack is being printed to the window.

PyPy compilation output fractal
Updated 17th September 2008:

The compilation process errored. I am compiling against a Stackless Python installation. This includes Stackless include files which are in a subdirectory, and I didn't add that subdirectory to my 'Include' environment variable. So the compilation failed after all that time, because of this.

There were two warnings in the compilation at the point of failure, where unknown command line options were found: '-fprofile-generate' and '-frandom-seed=testing_1.c'. This seems to be harmless, so ignoring them.

Updated 18th September 2008:

Another error, where 'zlib.lib' could not be found. There's a pattern emerging here. Added the zlib directory to the 'Lib' environment variable.

Next 'python25.lib' could not be found. Added the Python installation libs directory to the 'Lib' environment variable.

End result is a compiled 'pypy-c.exe'. Ignoring my laziness in populating the 'Lib' and 'Include' environment variables with entries which I should have known were needed, the process was as painless as a MS Toolkit compilation can be expected to be.

The next step should be creating a very simple frontend and getting that to compile.