Python packaging pain
Being in China with a wireless internet connection that drops in and out, I want to be able to just download and install software with the minimum amount of complication and fuss, if I need to. Tonight I wanted to download sphinx so that I could generate some documentation. First step was googling "sphinx" and seeing what the website had to say.
I've used easy_install in the past. I don't remember much about it, but I remember it both errored and installed things in ways I didn't quite understand. After a bit of searching, I concluded that it didn't seem to come with Python 2.7, so I decided to go with the second option and downloaded sphinx from the Python Package Index.
More egg things and source code. I don't understand those egg things and using them without being prepared to spend several hours working around non-working packaging systems is just not worth it. At least, that's what my past experience with them has led me to believe. So I went with the source code, in the hope I could just extract it and use it. After downloading the file, I consulted the README file:
Use ``setup.py``::So the only option is to engage the "magic" packaging systems, whether I want to or not.
python setup.py build
sudo python setup.py install
D:\Sphinx-1.0.7>c:\Python27\python.exe setup.py buildThe "magic" is in progress.
---------------------------------------------------------------------------
This script requires setuptools version 0.6c9 to run (even to display
help). I will attempt to download it for you (from
http://pypi.python.org/packages/2.7/s/setuptools/), but
you may need to enable firewall access for this script first.
I will start the download in 15 seconds.
(Note: if this machine does not have network access, please obtain the file
http://pypi.python.org/packages/2.7/s/setuptools/setuptools-0.6c9-py2.7.egg
and place it in this directory before rerunning this script.)
---------------------------------------------------------------------------
Downloading http://pypi.python.org/packages/2.7/s/setuptools/setuptools-0.6c9-py2.7.egg
Traceback (most recent call last):Well, that's my "once every couple of years" encounter with Python packaging over and done with. I think I'll go back to easier and more straightforward solutions that involve just using source code directly. Sure, I could look into this and try and sort it out for myself, or google to see if someone else has encountered it. But I've been down this rabbit hole before, with almost exactly the same problem with the packaging system several years ago. Fortunately, I can just abandon this hoop jumping and copy this source code from the SVN exports that the Python documentation building process does (since I build it for Stackless Python), and adapt the script that comes with it.
File "setup.py", line 6, in
ez_setup.use_setuptools()
File "D:\Sphinx-1.0.7\ez_setup.py", line 95, in use_
setuptools
return do_download()
File "D:\Sphinx-1.0.7\ez_setup.py", line 89, in do_d
ownload
egg = download_setuptools(version, download_base, to_dir, download_delay)
File "D:\Sphinx-1.0.7\ez_setup.py", line 150, in dow
nload_setuptools
src = urllib2.urlopen(url)
File "c:\Python27\lib\urllib2.py", line 126, in urlopen
return _opener.open(url, data, timeout)
File "c:\Python27\lib\urllib2.py", line 398, in open
response = meth(req, response)
File "c:\Python27\lib\urllib2.py", line 511, in http_response
'http', request, response, code, msg, hdrs)
File "c:\Python27\lib\urllib2.py", line 436, in error
return self._call_chain(*args)
File "c:\Python27\lib\urllib2.py", line 370, in _call_chain
result = func(*args)
File "c:\Python27\lib\urllib2.py", line 519, in http_error_default
raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
urllib2.HTTPError: HTTP Error 404: Not Found
Yes, Python packaging sucks. I wasted way too much time struggling with it, both as a user and as a maintainer.
ReplyDeletePractical solution: Every time you install Python on a machine, install `distribute` and `pip` right away. Then `pip install sphinx` will work without hassle.
The hard way:
ReplyDelete==========
1. Download and install setuptools-0.6c11.win32-py2.7.exe from http://pypi.python.org/pypi/setuptools
2. Go to C:\Python27\Scripts
3. Execute:
easy_install docutils
easy_install jinja2
easy_install sphinx
easy_install pygments (if you like to have source code highlighting support)
The easy way:
==========
easy_install pip
pip install sphinx
easy_install usually works, especially for big projects like this. It's very easy to get working on Linux, somewhat simple on Mac, and hard on Windows.
ReplyDeleteIf you want to have package management on Windows, use ActivePython. It comes with a good package manager that is adapted to the needs of Windows users.
I agree Python's packaging is not the best in the world, but you're making it too difficult for yourself.
ReplyDeleteInstall setuptools (http://pypi.python.org/pypi/setuptools/0.6c11) which is basically the standard add-on for Python folks. It gives you eggs and easy_install and whatnot. Don't worry about it, just install it on all your Python development machines.
Then "easy_install sphinx"
If you're fine with an old version, you can
ReplyDeletesudo apt-get install python-sphinx.
It seems to me that if you're not going to spend the time to learn about the ecosystem you have no right to complain. I'm a relatively new Pythonista and I'll admit to having some rather large issues with the whole system you've described. Spending a few hours, reading the guide to Python packaging, and most-especially using pip though have made everything completely flawless. I just issued pip install sphinx from my Windows system, and 2 minutes later it has installed both sphinx, and the pygments package it depends upon. In short read the docs, just as you had to to when you were a newbie like me, and you'll be good.
ReplyDeleteeasy_install it isn't installed with Python. In Windows, you have to install Setuptools. Look for it in pypi, there is an .exe file.
ReplyDeleteIn other OS, it is already packaged (in debian/ubuntu is python-setuptools, in arch is setuptools)
So let me get this straight, people use this packaging system and refer to its commands directly and offer their software through it almost entirely, and they don't seem to realise that this is arcane and obscure.
ReplyDeleteQ is right, this is an ecosystem. It isn't a natural part of Python, it is a layer transparent to its normal users but obtuse to anyone else. For a packaging system to be adopted as easy, it needs to be a part of the Python distribution. For people to see these 'egg' things and read about how they should just use 'easy_install' is ridiculous. Take a look at that sphinx screenshot, 'easy_install -U ...' is only worth something if you already understand where it comes from and how to use it already. That's not easy, it's confusing.
But this is besides the point. I have jumped through these hoops and installed 'easy_install' in the past. It does not work. Take a look at that exception I pasted. That is the same kind of nonsense I had to suffer through with 'easy_install'. I do not try and use these things very often, but every time I do, I get these same errors!
I don't have time to waste on these second party systems that do not work. When Python comes with a built in packaging system, as it should, which has been tested and actually works without requiring the user to debug the packaging system itself. Then I'll give it another chance.
Thanks for the suggestion of ActiveState distribution Rafe, if I did not use the Stackless build of Python, I would be right there.
I'm curious about the failure mode here. I have seen errors in setuptools.py, but I've never seen a 404 from PyPI like that. It looks like your network is broken.
ReplyDeleteYou *can* go download every dependency individually and then 'setup.py install' them, but that's equally error-prone and much more tedious, so it makes sense that it would be worthwhile for projects to try to be some automation. (Also, easy_install is pre-installed on Linux and MacOS, so usually one doesn't even need to do the extra step of installing setuptools.) Based on what little I can see here, the GFW is eating your packets and faking error response codes for valid URLs, so the experience would have been largely the same without the packaging magic; the fact that the dependency was a packaging dependency was almost incidental. Sphinx has several other dependencies which could have failed in the same way. Heck, sphinx itself could have failed like that.
I'm not a big fan of setuptools, and especially not of things that require you to 'sudo easy_install' anything. But, usually this kind of thing works, especially on Windows, where permissions generally aren't an issue.
Glyph: if I try and access http://pypi.python.org/packages/2.7/s/setuptools/setuptools-0.6c9-py2.7.egg with Chrome via VPN bypassing the GFW, I get the same failure to load it as if I were behind the GFW. I can download from PyPI just fine behind the GFW otherwise.
ReplyDeletetelnet pypi.python.org 80 and doing a GET /... gives.
... 404 Not Found ... Not Found ... The requested URL /packages/2.7/s/setuptools/setuptools-0.6c9-py2.7.egg was not found on this server.
Yes, installing stuff really is painful. It would be best if you didn't have to install anything and have these applications fully "portable."
ReplyDeleteSee my blog for a similar discussion:
http://blog.ccpgames.com/kristjan/?p=42
I am very unsure of what exactly are you smoking. Python packaging is not perfect (I also think that easy_install or pip should be standard in Python), but for the usecase that you present here it is better than sliced bread.
ReplyDeleteeasy_install sphinx
just works for 99% of the people. Actually you are the first person that I have ever heard of it not working. This in not arcane or obscure, it is very progressive and efficient - you get automatic download, version checking, dependency resolution and installation.
You get the error because you are trying to download an old version. Just go to http://pypi.python.org/pypi/setuptools and get the newest .exe file, install that and you will have working easy_install. You should not specify versions, the easy_install will look up the latest version on its own. The newer version of the same principle is http://pypi.python.org/pypi/pip
Python here follows the example set by Debian Linux which is now also being copied by Apple, Google and Microsoft - offer all software from a central repository so that it is easily installable by name and so that it can be centrally managed (removed, upgraded, ...) from a unified interface.
Aigars, I do not smoke and never have. Read through what you wrote, this is not straightforward. What an obstacle for users to have to go through.
ReplyDeleteAnd let me repeat for the third time - I have installed easy_install in the past and used it to install packages and it did not work. It did not work. It failed to find a dependency.
It may all just work for you, and if it does I am glad. But it costs you nothing to go around claiming it is simple and should all just work, when it is not a reality that is correct in my experience.
I thank you for the information you gave, which is useful for someone who is willing to jump through hoops. However, yet again it has proven to be unworth my time and with no reward for me.
"So the only option is to engage the "magic" packaging systems, whether I want to or not."
ReplyDeleteNot much magic about it. It basically just compiles pyc files and copies them and the source to where it should be.
"The "magic" is in progress."
Actually not. That's not a part of the packaging system. Setuptools is an extension to the packaging system, and Sphinx uses it. So does many other packages. The automatic install of it is kinda magic, but *not* part of the packaging system.
Setuptools does include some values of magic though, like automatic download and install of dependencies.
"I think I'll go back to easier and more straightforward solutions that involve just using source code directly."
Well, is that easier? Then you need to download all the packages that Sphinx depends on, and copy it's source codes to your site-package manually. Is that really easier? In some cases you might need to compile extensions. Manually....
One thing I do agree with you though: source .egg files. Completely pointless.
I don't think this has been noticed
ReplyDeletehowever there is no setuptools/0.6c9
for python 2.7
http://pypi.python.org/pypi/setuptools/0.6c9#downloads
which is why the error is occuring
downgrade to python 2.6