Releasing Stackless Python 2.6.1

I've spent the last seven and a half hours building and releasing Stackless Python 2.6.1. For the last release or two I have been able to get CCP to let me allocate working time to do this, since their framework embeds Stackless, but I haven't been paying attention to the python-dev mailing list and I've been ignoring the other mailing list where the release discussion takes place so I wasn't able to schedule the working time ahead of the fact. Here's a summary of the effort which is involved in this process.

The first step is to merge in all the changes up to the point where Python 2.6.1 was released, into the corresponding Stackless branch:

• Worked out that revision 67220 was the last one integrated from the Python release26-maint branch.
• Integrated up to revision 67220.
• Had to edit conflicts in two files. 'configure' which is a generated file, reverted the changes to that and will generate it on a non-Windows machine after I check in the changes. 'typeobject.c' where a new function is being used called 'call_attribute' which is replacing the use of a generic call API function, this touches the Stackless "soft switching" changes.
• Compiled the source code in Visual Studio 2008 Professional. CCP have allowed me to install it at home in order to build Stackless there. There was an error in 'typeobject.c' where TortoiseSVN shat on the file and left merge information in place.
• Ran the standard unit tests. There were three failures and some stacktraces which looked dodgy after tests which didn't fail. I rebuild the source code with Stackless disabled and saw most of these again when I ran the unit tests again. The only remaining problem is a stacktrace which happens between 'test_cpickle' and 'test_cprofile' complaining about a recursion depth runtime error. Crossing my fingers that the problem is not in the pickling support, as that pickling isn't the most fun part of the Python code to debug.
• Ran 'test_cpickle' directly and it definitely prints out the stacktrace. Ran the unit test again with the verbose flag and saw that it was happening in 'test_issue2702' test within the 'cPickleDeepRecursive' test case. Hoping that that it's some verification of proper behaviour which doesn't make sense in Stackless. Well, the test says "This should raise a RecursionLimit", given that, this is just Stackless specific noise.
• I'm happy with the Python unit tests, now I need to run the Stackless unit tests. They all pass fine.
• The merge looks like a success. If I really wanted to be pedantic, I'd look into that Pickling noise. But given I am hoping to release Stackless Python 3000 or whatever the newfangled Python that was released is called, it isn't worth doing.
• Time to check it in, so I can check it out remotely on a Linux box to fix the 'configure' problem.
• The changes to 'configure' were negligible. Checked it in and made a tag for the 2.6.1 state.

In order to build the Windows installer, I need to build all the dependencies like bzip2, OpenSSL and so forth:

• Exported the Stackless 'release26-maint' within a sub-directory of an arbitrarily named 'export-release26-maint' directory, so I can isolate the external dependency builds there. The build in the 'release26-maint' directory looks in its parent directory for those dependencies.
  
• Did a debug build within 'export-release26-maint/release26-maint' as I expected the buildbot external dependency building process to use Python.
• Entered the 'export-release26-maint/release26-maint' directory and executed 'Tools\buildbot\external.bat' file which takes care of the tedium of obtaining and building these dependencies. Python scripts weren't actually involved. Thanks to whomever came up with this timesaver.
• Rebuilt the debug build, with the dependencies theoretically in place, it should now do a full compilation.
• The '_ssl' and '_hashlib' projects failed due to their not being able to find some 'nasmw' command. This is addressed in the _ssl section of 'PCbuild/readme.txt'. It mentions the need to install nasm and points to the relevant web site. I've been here before and the 'nasmw.exe' executable is a myth, I've never been able to find it. I just download the binaries with the 'nasm.exe' executable and rename that one, placing it in my 'system32' directory. It seems to allow the compilation to proceed, so fingers crossed. Watching some Countryfile and drinking a delicious diet Pepsi Max while I am waiting. I've would never accuse the Python developers of writing 100% reliable or accurate build instructions, sometimes a lot of faffing around is involved wondering how they ever managed to make a release in the first place. I'd moan on about it a bit more, but let's be honest, I'm not going to bother submitting patches.
  
• '_ssl' compiled, but there's still one failure. Rebuilt the solution and the now compiled '_ssl' allowed '_hashlib' to now compile.
• This means I have successfully built a debug Stackless Python 2.6 and all its dependencies. I'm now going to build the release version and then run the unit tests for each. Maybe there will be some problems with the unit tests which involve the dependencies, or in the release build in ways which were not present in the debug build.
• The buildbot script for some reason does not do release builds of tcl and tk. I need to do both manually. In the case of tk, it wasn't enough to use the same command lines from the 'readme.txt'. I also needed to give a command line argument of 'TCLDIR=..\..\tcl-8.5.2.1'.
• Running the Python unit tests with the release build. I needed to take the tk and tcl dlls from my existing Python 2.6.1 install and put them in the 'PCbuild' directory to run the 'test_tcl' test suite. The test runs were consistent with previous results.
• Running the Stackless unit tests with the release build. The tests all passed successfully.
• Running the Python unit tests with the debug build. Acceptable.
• Running the Stackless unit tests with the debug build. The tests all passed successfully.

Now to actually build the installer:

• The first step is to run 'Tools\msi\msi.py'. This, as usual chokes, as it depends on the pywin32 extensions. The only way I know to get these in place are to install them into my existing Python 2.6.1 installation, and then to copy the 'site-packages' directories into the corresponding sub-directory in my build directory.
• 'msi.py' expects to find a directory matching 'tcl8*' above the 'release26-maint' directory. However, the directory is actually named 'tcl-8*', so.. I need to fix this inconsistency in 'msi.py'. I made the same fix for tk while I was there. Tix is also specified there, but I don't know where that comes from, so I've commented it out.
• 'msilib.py' errors in 'make_short' with an assertion. This always happens, and is meaningless, although I forget the reasons why. I comment this out. Why things like this aren't hit by the Python releasers, I do not know.
  
• 'msi.py' errors unable to find 'python26.chm'. I'm not even going down this rabbit hole again. Even when I can get the separate documentation build process to generate the chm prerequisites, the tool to generate the chm file chokes on the input. I just grab it from the Python 2.6.1 installation. There's no Stackless customisation to the documentation anyway. This file in the installation is coincidentally called 'python261.chm' in any case.
  
• This reminds me I need to customise 'msi.py' and have it do an 'official' build. This should get it looking for the '261' chm file I epxect. Now 'msi.py' is choking on there not being a suitable uuid entry for '2.6.1150'. This uuid was added days later to the 'uuid.py' module, so I hacked it in. I can understand why this happened, there was probably a code freeze at the time.
  
• Now it is asking me for 'python26.chm'. But the one which came with the install was 'python261.chm'. I want to conform to the existing install, but.. I'm getting weary so I will just rename the file to give it what it asks and move on.
• 'cabarc.exe' not found in registry (it never is). And it fails to find 'cabarc.exe' in the PATH environment variable. I executed the platform SDK 'SetEnv.cmd' script and it added the entries I needed.

Now I have a 'python-2.6.1.msi' installer, which is actually for Stackless. I'll rename it and put it up on the Stackless downloads web page.