Merging Stackless Python 3.0.1, part 2

Continues on from merging Stackless Python 3.0.1.

I wasted a lot of time already knowing what the problem was, trying to work out why there was a failure in the Python distutils test suite. I had even made a note about how, as the problem wasn't happening when I compiled the original Python 3.0.1 source code, it was more than likely because of a mismerge. Then I proceeded to play with the code I had until I noticed code differences accidentally.

There are a certain set of actions I know to do, when merging Python changes into the Stackless branches. Yet, for some reason, I just pushed them out of my mind. It reminds me of a recent article about checklists, and the difference they've made to doctors and pilots. It's hardly brain surgery, but I should really make a checklist for effective merging and print it out to put somewhere handy.

1. Merge from the relevant Python branch, into the corresponding Stackless branch using Tortoise SVN.
2. Compile the resulting source code.
3. Run the Python unit tests.
4. If there are Python unit test failures:
   1. Recompile the merged source code with STACKLESS_OFF defined. This should compile a version of Python which is identical to source Python branch.
   2. If there are Python unit test failures:
      1. Get a copy of the original Python branch and compile it.
      2. Run the Python unit tests.
      3. If there are failures, then the problem is not with Stackless. Maybe Python isn't familar with the nuances of the compilation environment being used. Maybe the Python unit test is faulty. It doesn't matter, this problem can be ignored.
   3. Otherwise, check if there was a mismerge.
      1. Install WinMerge if it is not already installed.
      2. Feed it the merged code and the original Python branch
      3. Add line-based filters until all the source control tags are excluded, like $Revision and $Id.
      4. Verify that all the differences are valid Stackless changes.
      5. Hand copy over any files from the original branch which were not merged by Tortoise SVN. Correct any mismerged lines.
5. Run the Stackless unit tests and fix any problems.

Tortoise SVN used to be a lot more reliable. Unmerged files weren't the only problems I encountered during the merging process. There was one file which had been added to the original Python branch, which Tortoise SVN refused to add.

Merging Stackless Python 3.0.1

Part of the Stackless Python release backlog.

Preparation

1. Updating Tortoise SVN.
   • I would have skipped this step, but the web site claims that bugs have been fixed with Tortoise Merge.
   • Unfortunately, I should have still skipped this step, as what I forgot was that I use Perforce Merge which can be freely obtained and used for merging with Tortoise SVN.
   • A reboot was required to finish the installation.
2. Updating my local Stackless SVN repositories (release30-maint).
   • I might have last worked on it at work, or Kristjan Valur might have checked in fixes or merges in the meantime. Check-in emails are sent to the Stackless-checkins mailing list which I subscribe to, so I know no merging has taken place but that there have been fixes which I may not have. It is a quick process, so it is no big deal to do it just in case.
3. Updating my local Python SVN repository (release30-maint).
   • I am not 100% sure about the release process for Python 3.0.1 but I suspect that the development is done within the py3k branch, with selective merging made into release30-maint, where releases are then made from. All I really need to do is get the latest of the 3.0 release branch, then merge from that into the corresponding Stackless branch.
4. Ascertaining the release revision.
   • Looking at the checkin comment for the revision of the Python SVN r301 tag, it says Copied from python/branches/release30-maint, r69557.
   • This means I need to merge up to that revision in the Python release30-maint branch, into the Stackless release30-maint branch.
5. Ascertaining the merge revision range.
   • I need to know the highest revision of the Python release30-maint branch which has been merged into the Stackless release30-maint branch.
   • Looking at the history of the Stackless release30-maint branch, the last relevant comment by me is This brings release30-maint to the release tag level. Not very helpful. This means I need to look at the Python r30 tag, which has a comment of Copied from python/branches/release30-maint, r67505.
   • This gives a merge revision range of r67505 to r69557.

Merging

1. Right click on the Stackless release30-maint directory and select Merge from the Tortoise SVN context sub-menu.
2. Selected "Merge a range of revisions" and clicked the Next button.
3. Ensured that svn+ssh://pythondev@svn.python.org/python/branches/release30-maint was selected in the "URL to merge from" box.
4. Entered 67505-69557 in the "Revision range to merge" box and clicked the Next button.
5. Scratched my head at the "Merge Options" page and just clicked on the Test merge button.
6. There were 22 conflicts and various other things which were problematic, after the gigabyte of traffic it took. I decided to just do a merge, in order to not waste any more bandwidth.
7. All of the conflicts have been in files which Stackless does not modify. The documentation restructured text files for instance. Not sure what is going on there, maybe I determined the wrong merge range. There is certainly a lot of merging going on. Oh well, another gigabyte gone.
8. Skipped missing target: D:\SVN\_python\stackless-branches\release30-maint\Tools\msi\crtlicense.txt. This is not a very helpful message. If I had to guess, it would be another hint at this being a misconfigured merge. Manually copied the file over, as you do.
9. I am going to do a preliminary build with the unreliable Visual Studio 2008, in order to determine whether the merge resulted in anything usable. It should compile fine, but compilation correctness problems will show up when the unit tests are run.
10. Created a new non-version controlled release30-maint-export directory beside the version controlled SVN release30-maint one. Right clicked on release30-maint and selected "Export" from the Tortoise SVN context menu. Exported a copy of the versioned files from release30-maint into the new release30-maint-export directory.
11. Compiled release30-maint-export with Visual Studio 2008. I am not compiling the dependencies, but it will compile the basics of the interpreter and the DLLs, which I can use to run the unit tests.
12. The compilation was successful with only the dependencies erroring. Normally, I have to fight with Visual Studio, to get the correct version of it to open a solution file. Then things like the socket module fail to compile, indicating something is different with my environment than the one others use. This was the easiest compilation I have had in a long time. Fixes must have made it into this release, which addressed my past compilation problems.
13. Time to run the Stackless unit tests. I am running these first as they take no time at all compared to the Python unit tests.
14. Unfortunately the Stackless unit tests crashed. I am running the debug version of Python, so it might be an assertion. Will run the Python unit tests and see how they respond, before checking that out.
15. Python unit test failure 1:

    test test_descr failed -- Traceback (most recent call last):
      File "D:\SVN\_python\stackless-branches\release30-maint-export\release30-maint
      \lib\test\test_descr.py", line 3891, in test_method_wrapper
        self.assertEqual(l.__add__, l.__add__)
    AssertionError:  != 
    <method-wrapper '__add__' of list object at 0x05054D38>

    It is possible that this is caused by the incorrect compilation results of my Visual Studio 2008 installation.
16. Python unit test failure 2:

    ERROR: test_build_ext (distutils.tests.test_build_ext.BuildExtTestCase)
    ----------------------------------------------------------------------
    Traceback (most recent call last):
      File "D:\SVN\_python\stackless-branches\release30-maint-export\release30-maint
      \lib\distutils\tests\test_build_ext.py", line 22, in setUp
        os.mkdir(self.tmp_dir)
    WindowsError: [Error 183] Cannot create a file when that file already exists:
    'D:\\SVN\\_python\\stackless-branches\\release30-maint-export\\release30-maint
    \\lib\\distutils\\tests\\xx'

    This does not look like a compilation problem, more likely a problem with the unit test itself. Or something to do with my Windows Vista operating system installation.
17. Revisiting the Stackless unit test crash and attaching to the running Visual Studio instance I used to compile, the crash happens in the garbage collection when some object is being removed from a list in a dictionary which a frame being deallocated has the last reference to.

So, there are potentially two bugs to fix. I need to compile this source code at work tomorrow, in order to see what the correctly compiling Visual Studio compiler does. But for now, I can consider this merge a success, and can check in the result to the python.org SVN repository.

EDIT: Work compilation results.

I compiled the checked in source code at work, with the install of Visual Studio 2008 there. Strangely, it gives the same results as my home install. The same crash and unit test failures.

Stackless Python release backlog

Quite a lot of work is required to bring Stackless Python completely up to date. In the past, I followed the Python-dev mailing list and when upcoming releases were discussed, I found time to synchronise a matching Stackless Python release. But being disillusioned about the direction Python is headed combined with forgetting to check it out on a regular basis, I have unfortunately fallen behind with matching releases of older versions of Python.

This is the current list of outstanding release related tasks:

Python 3.0.1	Merge, bug fix, release	Visual Studio 2008
Python 3.0.0	Re-release	Visual Studio 2008
Python 2.6.1	Re-release	Visual Studio 2008
Python 2.5.4	Merge, release	Visual Studio 2003
Python 2.5.3	Merge, release	Visual Studio 2003
Python 2.4.5	Merge, release	Visual Studio 2003

Merging and Compilation

Merging is not a big deal these days. Most of the problems in the past were merging mistakes on my part and when the bugs were located, going back to the source code revealed what I had done wrong. The other big merging obstacle in the past was the merge from 2.3 to 2.4, this was significantly more work than any merging effort since, including what 3.0 involved.

Where the effort lies now is in the compilation. To some degree the process of compiling the Python dependencies, but mostly the unreliability of the Microsoft development tools.

Before Windows installers were provided for Stackless, the standard release process for Windows binaries was to compile the DLLs, archive them and upload the archive to the web site. It was a pretty straightforward process, Christian Tismer had even written a script to do it all. But when providing a Windows installer became part of the release process, this mean that all the dependencies of Python needed to be compiled so that they could be packaged by the installation process. This is still a lot of work, but it used to be more before the source code for the correct versions of the dependencies were checked into the Python SVN repository, and scripts were provided to automate part of the building process.

The most reliable Microsoft development tool I have installed, is the free VS.NET 2003 Toolkit which only contains a non-optimising compiler. Due to compatibility reasons, the DLLs for versions of Python up to and including 2.5 are required to be compiled with VS.NET 2003. My toolkit install will, following the steps documented on the linked wiki page, compile binaries which pass the Python and Stackless unit tests, and also build a working installer. The toolkit is a little cumbersome to install and configure with all its own dependencies, but after that process has been completed once, it doesn't need to be repeated.

Visual Studio 2008 is the compiler used for Python 2.6 and 3.0. This I have a full installation of both at home, and at work. The installation at home will not compile binaries that pass the Python unit tests reliably, but it will build a working installer. The installation at work will compile binaries that pass the Python unit tests reliably, but will not build a working installer. I do not own this however, and have a work license covering the home install, so it is a little cumbersome to reinstall it. And there's also the concern that if I reinstall it, things will be even more broken.

So for now, I will merge and build an installer for 2.6 and 3.0 at home. And build the binaries which get distributed in the installer at work. Not an ideal solution, but it is going to have to do.

Priorities

Python 3.0.1 is the highest priority. The older versions of Python before 2.6 are not commonly used, and the requests for the releases which do not have existing Stackless versions already, are very rare. Making a priority of the latest releases helps make it clear that Stackless is still maintained and has not stagnated. While Python 2.3 was the latest version, there was a long period when no-one had the time to keep Stackless up to date, and this made it look abandoned and was a source of frustration to existing users. The older releases are a bonus if I have the spare time to do them.

Edit: Linked to the 3.0.1 merge post.