Releasing Stackless Python 3.0.1

Continues on from Merging Stackless Python 3.0.1, part 2.

After the merging process is complete, the next step is to do the release. The release process follows these steps:

1. Compile the source code using the appropriate version of Visual Studio.
2. Generate an archive of files which can be extracted onto an existing Python installation.
3. Obtain and build all the dependencies for Python.
4. Build an installer.
5. Upload the archive and installer to the Stackless host.
6. Generate an archive of the Stackless source code on the Stackless host.
7. Add a news item to the Stackless web site.
8. Post an announcement to the Stackless mailing list.

Building the binaries archive

Christian Tismer wrote a script which automates this process.

1. Change the current directory to the PCbuild directory.
2. Execute publish_binaries.py. This will create a stackless-python-30.zip archive and a stackless-python-30.zip.md5.py MD5 checksum verifying script.

Building the installer

The Stackless installer needs to include all of the dependencies, the same extension modules and DLLs as the Python installer and the documentation.

1. Go to the Programs / Microsoft Platform SDK for Windows Server 2003 R2 / Open Build Environment Window / Windows XP 32-bit Build Environment / Set Windows XP 32-bit Build Environment (Retail) menu. This will open a build environment DOS prompt which will have the correct paths set up for commands like cabarc.
   
   
   • This may make compiler commands available, but they may not be for the correct version of Visual Studio. The Visual C++ environment variable script for the correct version of Visual Studio should be executed. For Python version 2.6 and above (so 3.0 as well), this is Visual Studio 2008.
     
     Execute C:\Program Files\Microsoft Visual Studio 9.0\VC\vcvarsall.bat.
   
   
2. Change the current directory to the PC directory.
3. Execute nmake -f icons.mak.
4. Change the current directory to the Tools\msi directory.
5. Execute nmake -f msisupport.mak.

At this point the dependencies and documentation need to be built before proceeding.

Building Python's dependencies

Nothing makes a prolonged and tedious process more fun than additional tedium, like manually downloading and working out how to compile the correct versions of other projects. Fortunately, to some degree that tedium is alleviated through the ability to use the buildbot scripts to automate it.

1. Download and install the Python installer for the matching version of Python.
2. Build the Python source code. Note the projects that failed to compile, lacking their dependencies.
3. Change the current directory to the top level of the source code.
4. Execute Tools\buildbot\external.bat. This is the script used for the buildbot, and it automates installing and compiling those dependencies.
5. Build the Python source code again. The projects previously lacking dependencies should now have them and will compile successfully.
   
   
   • The SSL project may fail to compile due the following error: The process cannot access the file because it is being used by another process.

     2> copy .\crypto\buildinf.h tmp32\buildinf.h
     2>        1 file(s) copied.
     2> copy .\crypto\opensslconf.h inc32\openssl\opensslconf.h
     2>        1 file(s) copied.

     Rebuild repeatedly until this race condition is avoided.
   
   
   • The dependencies for the tkinter project, tcl and tk, will not have been compiled in release mode. The PCbuild\README.txt file instructs the user to look at the Tools\buildbot\external.bat script, and execute the same commands but without the DEBUG=1 flag.
     
     There's a script build_tkinter.py in PCbuild which refers to the directories tcl8.5.2 and tk8.5.2. However, the buildbot script does not use these directory names. It uses tcl-8.5.2.1 and tk-8.5.2.0. Something to look at later on.


6. Run the Stackless unit tests and verify they pass.
7. Run the Python unit tests and verify they pass.
   
   
   • Tests which depend on the presence of DLLs in the PCbuild directory may have been skipped, like test_bz2 and test_tcl.Where these DLLs are supposed to come from, I do not know. They don't get compiled by the buildbot script, and there are no pointers on where they should be obtained from.
     
     For the lack of any better idea, these can be copied from the Python installation

At this point, both release and debug solutions should build completely.

Building the documentation

Before 3.0.1, I had yet to be able to build the Python documentation. The need to install a LaTeX solution, which I never seemed to be able to get to work. A new restructured text based system, which would error halfway through. And if it didn't, then the CHM compiler would crash. But like most other parts of the release process, this part is also slowly getting improved.

1. Change the current directory to the Doc directory.
2. Reading 'Doc\README.txt' suggests running make update then make htmlhelp.
   
   
   • make update just updates local copies of the documentation dependency source code. Reading the make.bat script shows that this just does an SVN update, and that you need to do a corresponding make checkout first.
   
   
   • The documentation building scripts are pre-Python 3.0, an existing older version of Python needs to be used. README.txt does not indicate this.


3. Execute set PYTHON=c:\python26\python.
4. Execute make.bat.
   
   
   • The build process may fail due to the following error: Unable to open build\htmlhelp\pydoc.hhp. This is because build\htmlhelp\python301.hhp is what is actually generated. make.bat should be edited to refer to the correct file.
   
   
5. Execute make.bat again, if the previous attempt failed.

There should now be a build\htmlhelp\python301.chm. If the documentation building process fails, then just take the standard approach and copy over the chm file from the normal Python installation done earlier.

Building the installer continued

At this point, the installer can now be built.

1. Change the current directory to the Tools\msi directory.
2. Execute msi.py after some minor changes to the msi related scripts.
   
   
   • For some reason, the first time I went through this process, I interpreted the msi README.txt file to meant that msi.py needs to be executed with the build of Python which had been compiled and was to be installed (i.e. ..\..\PCbuild\python.exe). However, given that the code in the msi directory has not been converted to work with Python 3.0, this implies I was wrong. Another hint is that the build process depends on the pywin32 extensions being installed. I used to copy these over to the site-packages directory of an existing install to the corresponding directory of the Stackless build. But really, the commonsense approach is to use the existing install directly given that there is no linkage between the script and what version of Python is used to execute it.
   
   
   • The generated installer needs to be customised to represent a release. This means editing msi.py and setting full_current_version to "3.0.1" and snapshot to 0.
   
   
   • Often releases of Python are made without adding the product code entries, or adding them correctly, to uuids.py. So if an error happens where the product code needed for the installer is not found, they can always be obtained from the latest code in the SVN repository.
   
   
3. Execute c:\python26\python.exe msi.py. This will generate python-3.0.1.msi.
4. Rename python-3.0.1.msi to python-3.0.1-stackless.msi
5. Uninstall the Python 3.0.1 installation and verify that the Stackless Python 3.0.1 installer installs correctly.

The installer is ready to release for people to use.

Future directions

This is the first time I have ever been able to build the Python documentation successfully. As long as this success is repeatable, it might be time to add Stackless Python related sections to the Python documentation within the Stackless branches and add generating custom documentation to the release process.

Also, in an ideal world, I would take the points above that present ideas for improvement which might be contributed back to the Python developers and well.. submit feedback or patches. Laziness however, dictates that it is less effort to suck it up and do these things every time, rather than give myself more obligations which require me to spend time on them and distract me from the ones I already have.

Merging Stackless Python 3.0.1, part 2

Continues on from merging Stackless Python 3.0.1.

I wasted a lot of time already knowing what the problem was, trying to work out why there was a failure in the Python distutils test suite. I had even made a note about how, as the problem wasn't happening when I compiled the original Python 3.0.1 source code, it was more than likely because of a mismerge. Then I proceeded to play with the code I had until I noticed code differences accidentally.

There are a certain set of actions I know to do, when merging Python changes into the Stackless branches. Yet, for some reason, I just pushed them out of my mind. It reminds me of a recent article about checklists, and the difference they've made to doctors and pilots. It's hardly brain surgery, but I should really make a checklist for effective merging and print it out to put somewhere handy.

1. Merge from the relevant Python branch, into the corresponding Stackless branch using Tortoise SVN.
2. Compile the resulting source code.
3. Run the Python unit tests.
4. If there are Python unit test failures:
   1. Recompile the merged source code with STACKLESS_OFF defined. This should compile a version of Python which is identical to source Python branch.
   2. If there are Python unit test failures:
      1. Get a copy of the original Python branch and compile it.
      2. Run the Python unit tests.
      3. If there are failures, then the problem is not with Stackless. Maybe Python isn't familar with the nuances of the compilation environment being used. Maybe the Python unit test is faulty. It doesn't matter, this problem can be ignored.
   3. Otherwise, check if there was a mismerge.
      1. Install WinMerge if it is not already installed.
      2. Feed it the merged code and the original Python branch
      3. Add line-based filters until all the source control tags are excluded, like $Revision and $Id.
      4. Verify that all the differences are valid Stackless changes.
      5. Hand copy over any files from the original branch which were not merged by Tortoise SVN. Correct any mismerged lines.
5. Run the Stackless unit tests and fix any problems.

Tortoise SVN used to be a lot more reliable. Unmerged files weren't the only problems I encountered during the merging process. There was one file which had been added to the original Python branch, which Tortoise SVN refused to add.

Merging Stackless Python 3.0.1

Part of the Stackless Python release backlog.

Preparation

1. Updating Tortoise SVN.
   • I would have skipped this step, but the web site claims that bugs have been fixed with Tortoise Merge.
   • Unfortunately, I should have still skipped this step, as what I forgot was that I use Perforce Merge which can be freely obtained and used for merging with Tortoise SVN.
   • A reboot was required to finish the installation.
2. Updating my local Stackless SVN repositories (release30-maint).
   • I might have last worked on it at work, or Kristjan Valur might have checked in fixes or merges in the meantime. Check-in emails are sent to the Stackless-checkins mailing list which I subscribe to, so I know no merging has taken place but that there have been fixes which I may not have. It is a quick process, so it is no big deal to do it just in case.
3. Updating my local Python SVN repository (release30-maint).
   • I am not 100% sure about the release process for Python 3.0.1 but I suspect that the development is done within the py3k branch, with selective merging made into release30-maint, where releases are then made from. All I really need to do is get the latest of the 3.0 release branch, then merge from that into the corresponding Stackless branch.
4. Ascertaining the release revision.
   • Looking at the checkin comment for the revision of the Python SVN r301 tag, it says Copied from python/branches/release30-maint, r69557.
   • This means I need to merge up to that revision in the Python release30-maint branch, into the Stackless release30-maint branch.
5. Ascertaining the merge revision range.
   • I need to know the highest revision of the Python release30-maint branch which has been merged into the Stackless release30-maint branch.
   • Looking at the history of the Stackless release30-maint branch, the last relevant comment by me is This brings release30-maint to the release tag level. Not very helpful. This means I need to look at the Python r30 tag, which has a comment of Copied from python/branches/release30-maint, r67505.
   • This gives a merge revision range of r67505 to r69557.

Merging

1. Right click on the Stackless release30-maint directory and select Merge from the Tortoise SVN context sub-menu.
2. Selected "Merge a range of revisions" and clicked the Next button.
3. Ensured that svn+ssh://pythondev@svn.python.org/python/branches/release30-maint was selected in the "URL to merge from" box.
4. Entered 67505-69557 in the "Revision range to merge" box and clicked the Next button.
5. Scratched my head at the "Merge Options" page and just clicked on the Test merge button.
6. There were 22 conflicts and various other things which were problematic, after the gigabyte of traffic it took. I decided to just do a merge, in order to not waste any more bandwidth.
7. All of the conflicts have been in files which Stackless does not modify. The documentation restructured text files for instance. Not sure what is going on there, maybe I determined the wrong merge range. There is certainly a lot of merging going on. Oh well, another gigabyte gone.
8. Skipped missing target: D:\SVN\_python\stackless-branches\release30-maint\Tools\msi\crtlicense.txt. This is not a very helpful message. If I had to guess, it would be another hint at this being a misconfigured merge. Manually copied the file over, as you do.
9. I am going to do a preliminary build with the unreliable Visual Studio 2008, in order to determine whether the merge resulted in anything usable. It should compile fine, but compilation correctness problems will show up when the unit tests are run.
10. Created a new non-version controlled release30-maint-export directory beside the version controlled SVN release30-maint one. Right clicked on release30-maint and selected "Export" from the Tortoise SVN context menu. Exported a copy of the versioned files from release30-maint into the new release30-maint-export directory.
11. Compiled release30-maint-export with Visual Studio 2008. I am not compiling the dependencies, but it will compile the basics of the interpreter and the DLLs, which I can use to run the unit tests.
12. The compilation was successful with only the dependencies erroring. Normally, I have to fight with Visual Studio, to get the correct version of it to open a solution file. Then things like the socket module fail to compile, indicating something is different with my environment than the one others use. This was the easiest compilation I have had in a long time. Fixes must have made it into this release, which addressed my past compilation problems.
13. Time to run the Stackless unit tests. I am running these first as they take no time at all compared to the Python unit tests.
14. Unfortunately the Stackless unit tests crashed. I am running the debug version of Python, so it might be an assertion. Will run the Python unit tests and see how they respond, before checking that out.
15. Python unit test failure 1:

    test test_descr failed -- Traceback (most recent call last):
      File "D:\SVN\_python\stackless-branches\release30-maint-export\release30-maint
      \lib\test\test_descr.py", line 3891, in test_method_wrapper
        self.assertEqual(l.__add__, l.__add__)
    AssertionError:  != 
    <method-wrapper '__add__' of list object at 0x05054D38>

    It is possible that this is caused by the incorrect compilation results of my Visual Studio 2008 installation.
16. Python unit test failure 2:

    ERROR: test_build_ext (distutils.tests.test_build_ext.BuildExtTestCase)
    ----------------------------------------------------------------------
    Traceback (most recent call last):
      File "D:\SVN\_python\stackless-branches\release30-maint-export\release30-maint
      \lib\distutils\tests\test_build_ext.py", line 22, in setUp
        os.mkdir(self.tmp_dir)
    WindowsError: [Error 183] Cannot create a file when that file already exists:
    'D:\\SVN\\_python\\stackless-branches\\release30-maint-export\\release30-maint
    \\lib\\distutils\\tests\\xx'

    This does not look like a compilation problem, more likely a problem with the unit test itself. Or something to do with my Windows Vista operating system installation.
17. Revisiting the Stackless unit test crash and attaching to the running Visual Studio instance I used to compile, the crash happens in the garbage collection when some object is being removed from a list in a dictionary which a frame being deallocated has the last reference to.

So, there are potentially two bugs to fix. I need to compile this source code at work tomorrow, in order to see what the correctly compiling Visual Studio compiler does. But for now, I can consider this merge a success, and can check in the result to the python.org SVN repository.

EDIT: Work compilation results.

I compiled the checked in source code at work, with the install of Visual Studio 2008 there. Strangely, it gives the same results as my home install. The same crash and unit test failures.

Stackless Python release backlog

Quite a lot of work is required to bring Stackless Python completely up to date. In the past, I followed the Python-dev mailing list and when upcoming releases were discussed, I found time to synchronise a matching Stackless Python release. But being disillusioned about the direction Python is headed combined with forgetting to check it out on a regular basis, I have unfortunately fallen behind with matching releases of older versions of Python.

This is the current list of outstanding release related tasks:

Python 3.0.1	Merge, bug fix, release	Visual Studio 2008
Python 3.0.0	Re-release	Visual Studio 2008
Python 2.6.1	Re-release	Visual Studio 2008
Python 2.5.4	Merge, release	Visual Studio 2003
Python 2.5.3	Merge, release	Visual Studio 2003
Python 2.4.5	Merge, release	Visual Studio 2003

Merging and Compilation

Merging is not a big deal these days. Most of the problems in the past were merging mistakes on my part and when the bugs were located, going back to the source code revealed what I had done wrong. The other big merging obstacle in the past was the merge from 2.3 to 2.4, this was significantly more work than any merging effort since, including what 3.0 involved.

Where the effort lies now is in the compilation. To some degree the process of compiling the Python dependencies, but mostly the unreliability of the Microsoft development tools.

Before Windows installers were provided for Stackless, the standard release process for Windows binaries was to compile the DLLs, archive them and upload the archive to the web site. It was a pretty straightforward process, Christian Tismer had even written a script to do it all. But when providing a Windows installer became part of the release process, this mean that all the dependencies of Python needed to be compiled so that they could be packaged by the installation process. This is still a lot of work, but it used to be more before the source code for the correct versions of the dependencies were checked into the Python SVN repository, and scripts were provided to automate part of the building process.

The most reliable Microsoft development tool I have installed, is the free VS.NET 2003 Toolkit which only contains a non-optimising compiler. Due to compatibility reasons, the DLLs for versions of Python up to and including 2.5 are required to be compiled with VS.NET 2003. My toolkit install will, following the steps documented on the linked wiki page, compile binaries which pass the Python and Stackless unit tests, and also build a working installer. The toolkit is a little cumbersome to install and configure with all its own dependencies, but after that process has been completed once, it doesn't need to be repeated.

Visual Studio 2008 is the compiler used for Python 2.6 and 3.0. This I have a full installation of both at home, and at work. The installation at home will not compile binaries that pass the Python unit tests reliably, but it will build a working installer. The installation at work will compile binaries that pass the Python unit tests reliably, but will not build a working installer. I do not own this however, and have a work license covering the home install, so it is a little cumbersome to reinstall it. And there's also the concern that if I reinstall it, things will be even more broken.

So for now, I will merge and build an installer for 2.6 and 3.0 at home. And build the binaries which get distributed in the installer at work. Not an ideal solution, but it is going to have to do.

Priorities

Python 3.0.1 is the highest priority. The older versions of Python before 2.6 are not commonly used, and the requests for the releases which do not have existing Stackless versions already, are very rare. Making a priority of the latest releases helps make it clear that Stackless is still maintained and has not stagnated. While Python 2.3 was the latest version, there was a long period when no-one had the time to keep Stackless up to date, and this made it look abandoned and was a source of frustration to existing users. The older releases are a bonus if I have the spare time to do them.

Edit: Linked to the 3.0.1 merge post.