Releasing Stackless Python 3.0.1

Continues on from Merging Stackless Python 3.0.1, part 2.

After the merging process is complete, the next step is to do the release. The release process follows these steps:

1. Compile the source code using the appropriate version of Visual Studio.
2. Generate an archive of files which can be extracted onto an existing Python installation.
3. Obtain and build all the dependencies for Python.
4. Build an installer.
5. Upload the archive and installer to the Stackless host.
6. Generate an archive of the Stackless source code on the Stackless host.
7. Add a news item to the Stackless web site.
8. Post an announcement to the Stackless mailing list.

Building the binaries archive

Christian Tismer wrote a script which automates this process.

1. Change the current directory to the PCbuild directory.
2. Execute publish_binaries.py. This will create a stackless-python-30.zip archive and a stackless-python-30.zip.md5.py MD5 checksum verifying script.

Building the installer

The Stackless installer needs to include all of the dependencies, the same extension modules and DLLs as the Python installer and the documentation.

1. Go to the Programs / Microsoft Platform SDK for Windows Server 2003 R2 / Open Build Environment Window / Windows XP 32-bit Build Environment / Set Windows XP 32-bit Build Environment (Retail) menu. This will open a build environment DOS prompt which will have the correct paths set up for commands like cabarc.
   
   
   • This may make compiler commands available, but they may not be for the correct version of Visual Studio. The Visual C++ environment variable script for the correct version of Visual Studio should be executed. For Python version 2.6 and above (so 3.0 as well), this is Visual Studio 2008.
     
     Execute C:\Program Files\Microsoft Visual Studio 9.0\VC\vcvarsall.bat.
   
   
2. Change the current directory to the PC directory.
3. Execute nmake -f icons.mak.
4. Change the current directory to the Tools\msi directory.
5. Execute nmake -f msisupport.mak.

At this point the dependencies and documentation need to be built before proceeding.

Building Python's dependencies

Nothing makes a prolonged and tedious process more fun than additional tedium, like manually downloading and working out how to compile the correct versions of other projects. Fortunately, to some degree that tedium is alleviated through the ability to use the buildbot scripts to automate it.

1. Download and install the Python installer for the matching version of Python.
2. Build the Python source code. Note the projects that failed to compile, lacking their dependencies.
3. Change the current directory to the top level of the source code.
4. Execute Tools\buildbot\external.bat. This is the script used for the buildbot, and it automates installing and compiling those dependencies.
5. Build the Python source code again. The projects previously lacking dependencies should now have them and will compile successfully.
   
   
   • The SSL project may fail to compile due the following error: The process cannot access the file because it is being used by another process.

     2> copy .\crypto\buildinf.h tmp32\buildinf.h
     2>        1 file(s) copied.
     2> copy .\crypto\opensslconf.h inc32\openssl\opensslconf.h
     2>        1 file(s) copied.

     Rebuild repeatedly until this race condition is avoided.
   
   
   • The dependencies for the tkinter project, tcl and tk, will not have been compiled in release mode. The PCbuild\README.txt file instructs the user to look at the Tools\buildbot\external.bat script, and execute the same commands but without the DEBUG=1 flag.
     
     There's a script build_tkinter.py in PCbuild which refers to the directories tcl8.5.2 and tk8.5.2. However, the buildbot script does not use these directory names. It uses tcl-8.5.2.1 and tk-8.5.2.0. Something to look at later on.


6. Run the Stackless unit tests and verify they pass.
7. Run the Python unit tests and verify they pass.
   
   
   • Tests which depend on the presence of DLLs in the PCbuild directory may have been skipped, like test_bz2 and test_tcl.Where these DLLs are supposed to come from, I do not know. They don't get compiled by the buildbot script, and there are no pointers on where they should be obtained from.
     
     For the lack of any better idea, these can be copied from the Python installation

At this point, both release and debug solutions should build completely.

Building the documentation

Before 3.0.1, I had yet to be able to build the Python documentation. The need to install a LaTeX solution, which I never seemed to be able to get to work. A new restructured text based system, which would error halfway through. And if it didn't, then the CHM compiler would crash. But like most other parts of the release process, this part is also slowly getting improved.

1. Change the current directory to the Doc directory.
2. Reading 'Doc\README.txt' suggests running make update then make htmlhelp.
   
   
   • make update just updates local copies of the documentation dependency source code. Reading the make.bat script shows that this just does an SVN update, and that you need to do a corresponding make checkout first.
   
   
   • The documentation building scripts are pre-Python 3.0, an existing older version of Python needs to be used. README.txt does not indicate this.


3. Execute set PYTHON=c:\python26\python.
4. Execute make.bat.
   
   
   • The build process may fail due to the following error: Unable to open build\htmlhelp\pydoc.hhp. This is because build\htmlhelp\python301.hhp is what is actually generated. make.bat should be edited to refer to the correct file.
   
   
5. Execute make.bat again, if the previous attempt failed.

There should now be a build\htmlhelp\python301.chm. If the documentation building process fails, then just take the standard approach and copy over the chm file from the normal Python installation done earlier.

Building the installer continued

At this point, the installer can now be built.

1. Change the current directory to the Tools\msi directory.
2. Execute msi.py after some minor changes to the msi related scripts.
   
   
   • For some reason, the first time I went through this process, I interpreted the msi README.txt file to meant that msi.py needs to be executed with the build of Python which had been compiled and was to be installed (i.e. ..\..\PCbuild\python.exe). However, given that the code in the msi directory has not been converted to work with Python 3.0, this implies I was wrong. Another hint is that the build process depends on the pywin32 extensions being installed. I used to copy these over to the site-packages directory of an existing install to the corresponding directory of the Stackless build. But really, the commonsense approach is to use the existing install directly given that there is no linkage between the script and what version of Python is used to execute it.
   
   
   • The generated installer needs to be customised to represent a release. This means editing msi.py and setting full_current_version to "3.0.1" and snapshot to 0.
   
   
   • Often releases of Python are made without adding the product code entries, or adding them correctly, to uuids.py. So if an error happens where the product code needed for the installer is not found, they can always be obtained from the latest code in the SVN repository.
   
   
3. Execute c:\python26\python.exe msi.py. This will generate python-3.0.1.msi.
4. Rename python-3.0.1.msi to python-3.0.1-stackless.msi
5. Uninstall the Python 3.0.1 installation and verify that the Stackless Python 3.0.1 installer installs correctly.

The installer is ready to release for people to use.

Future directions

This is the first time I have ever been able to build the Python documentation successfully. As long as this success is repeatable, it might be time to add Stackless Python related sections to the Python documentation within the Stackless branches and add generating custom documentation to the release process.

Also, in an ideal world, I would take the points above that present ideas for improvement which might be contributed back to the Python developers and well.. submit feedback or patches. Laziness however, dictates that it is less effort to suck it up and do these things every time, rather than give myself more obligations which require me to spend time on them and distract me from the ones I already have.