29.6 Popular Third-Party Software

In this section, we list some of the most popular third-party add-ons to Python. Some are small yet deeply useful modules, others are full fledged applications with massive internal complexity. Each is what we consider a good tool.

URLs change, so don't be dissapointed if the URLs we mention are no longer valid by the time you type them in. Instead, go to Google and type python name of the package—you're more than likely to find it.

29.6.1 Interfaces to Windows and the MacOS

While each operating system provides a wide variety of interfaces, Unix and related operating systems like Linux tend to provide that interface through command-line tools and special-purpose files, both of which tend to vary too much across versions to allow for useful programmatic interfaces. Windows and Macintosh use a more API-oriented approach, and as a result make the operating system more naturally accessible from a programming language like Python. There are Python interfaces to pretty much every corner of Windows and the Macintosh APIs.

29.6.1.1 Windows

Core Python comes with some interfaces to basic Windows interfaces like the os module for basic operating system functions and the _regedit low-level API to the Windows Registry. Serious Windows programming, however, requires access to many more Windows libraries. Most of these are exposed by the win32all package by Mark Hammond. win32all is available either as an add-on to the Python distribution from http://www.python.org, or bundled as part of ActivePython from ActiveState (http://www.ActiveState.com/Python). win32all also includes a Window-only IDE for Python, Pythonwin.

Not all Windows APIs are exposed by win32all. Should you wish to use one of these, you can use Thomas Heller's ctypes module, which provides a foreign function interface from Python to dynamically loaded shared libraries. ctypes is described below.

29.6.1.2 Macintosh

The Macintosh port of Python, maintained by Jack Jansen, and available at http://www.cwi.nl/~jack/macpython.html, comes in two kinds as of this writing. There is a new version that runs from the Mac OS X command line, as well as a version which runs on Mac OS 9 or OS X, although there are plans to merge the two. The documentation for the Macintosh library, the Macintosh Library Modules, is part of the standard library reference. Recent versions of Mac OS X come with Python preinstalled.

29.6.2 Special-Purpose Libraries

There are some domains of computing that lend themselves to well-defined libraries. These typically involve specialized data types (e.g., arrays of numbers, dates) or special-purpose algorithms (e.g., three-dimensional graphics engines). In this section we survey some of the most popular such libraries.

29.6.2.1 Scientific computing libraries

Python has a significant following in the scientific and engineering fields. Its fairly mathematically reasonable syntax and ease of extensibility make it a good match for scientists who need to combine easy to use syntax with the ability to attach high-power computational engines.

The granddaddy of scientific libraries in Python is called Numeric Python, NumPy, or Numeric for short. Numeric consists of a set of Python and C extension modules that provide very powerful and high performance array operations. With Numeric, one can do "en masse" operations on very large multidimensional arrays of numbers much faster than one can do in pure Python. Numeric is the current standard solution for doing such operations in Python, although many are looking to its proposed replacement, numarray.

Here's an example of typical NumPy code, numpytest.py, and one representation of the data in generates:

from Numeric import *              
coords = arange(-6, 6, .02)               # Create a range of coordinates.
xs = sin(coords)                          # Take the sine of all of the x's.
ys = cos(coords)*exp(-coords*coords/18.0) # Take a complex function of the y's.
zx = xs * ys[:,NewAxis]                   # Multiply the x row with the y column.

If you remember your math, you might figure out that xs is an array of the sines of the numbers between -6 and 6, and ys is an array of the cosines of those same numbers scaled by an exponential function centered at 0. zs is simply the outer product of those two arrays of numbers. If you're curious as to what that might look like, you could convert the array zs into an image and obtain the image shown in Figure 29-1.

Figure 29-1. Graphical representation of the array zs in numpytest.py

NumPy lets you manipulate very large arrays of numbers very efficiently. The preceding code runs orders of magnitude faster than comparable code using large lists of numbers and uses a fraction of the memory. Many Python users never have to deal with these kinds of issues, but many scientists and engineers require such capabilities daily.

Numarray (http://stsdas.stsci.edu/numarray/) is a reimplementation of the basic functionality of Numeric, while aiming to fix some of the problems with the original having to do with type conversions, extensibility, memory efficiency and other factors. It's too early to tell, but our sources indicate that numarray is well positioned to take over Numeric's market share in a nice, progressive, and orchestrated transition.

For those interested in doing more with numbers, the SciPy project is worth looking at. SciPy supplements the core array-crunching modules with a variety of high-level scientific and engineering modules, such as solvers, optimizers, statistical routines, and a very interesting inlining tool that compiles expressions using C++ in a semi-transparent way, called weave. SciPy also includes the most Pythonic scientific graphing software, called Chaco. Information about SciPy can be found at its web site, http://www.scipy.org.

Finally, there are other scientific libraries for Python not listed above—some for dealing with Fortran libraries, some for doing specialized processing in the fields of chemistry, genetics, etc. Use the resources mentioned to look for software in your particular field.

29.6.2.2 Relational database interfaces

Relational databases are a very common mode of storing and manipulating data, although as we've seen in this book, there are other, simpler ways of storing at least small amounts of data (such as using pickle, or the related marshal and shelve modules—see their documentation for details). All popular scripting languages have interfaces to common database engines, and Python is no exception. There is even a standard interface that allows Python programmers to talk to a variety of databases in a database-independent way, called the DB-API. Details on the DB-API, including references to current modules, are available at http://www.python.org/topics/database/. You will find that there are interfaces to Oracle, Postgres, MySQL, SAP DB, Informix, and many others.

There are other, less standard databases that are worth knowing about in case you don't have a particular reason to use a SQL server. The two most commonly used ones are Gadfly and MetaKit. Gadfly (http://gadfly.sourceforge.net/gadfly.html), is a SQL server written entirely in Python (with some C extensions), which provides a very simple SQL solution for people who have relatively small databases (Gadfly stores its data in memory), and don't require multiuser capabilities, but still want transactional support. MetaKit (http://www.equi4.com/metakit/python.html) is a C++ database engine with very good Python bindings. MetaKit has a more record-oriented view of the world, and is remarkably fast at a remarkable set of operations. A third and more recent entry half-way between Gadfly and full-fledged SQL databases is PySQLite, an interface to the popular embedded SQL engine SQLite.

A final player worth mentioning here is the set of tools from eGenix.com. These include mxODBC (an interface to the ODBC database layer), mxDateTime (a high-power date/time library), and other add-ons. These are all available at http://www.egenix.com).

29.6.2.3 Graphics libraries

There are several types of packages that can help produce graphical output of various kinds, from image processing tools to game frameworks to visualization toolkits.

29.6.2.3.1 Python Imaging Library (PIL)

The Python Imaging Library is an extensive framework written by Fredrik Lundh for creating, manipulating, converting, and saving bitmapped images in a variety of formats such as GIF, JPEG, and PNG. It has interfaces to Tk and Pythonwin, so that one can use either Tk widgets or Pythonwin code to display PIL-generated images. Alternatively, the images can be saved to disk in a variety of formats. The home for PIL is at http://www.pythonware.com.

29.6.2.3.2 PyGame and PyOpenGL

Two of the most popular selections for graphics work are PyGame for two-dimensional and gaming software, and PyOpenGL for 3-D graphics, especially in the area of scientific visualization. Pygame (http://pygame.org) is a set of Python modules written on top of the free, cross-platform SDL library. It has been used for a variety of game and nongame software, and provides excellent interfaces to the screen, the mouse, the keyboard and sound output. People have written some remarkable games with PyGame, as well as used it for nongaming applications that have heavy emphasis on game-like animations, controls etc. A simple example of PyGame programming is the following snippet, which plays the first five seconds of the first track of the first CD player it finds, and then ejects it.

import pygame
pygame.cdrom.init(  )
cd_object = pygame.cdrom.CD(pygame.cdrom.get_count(  )-1)
cd_object.init(  )
if cd_object.get_track_audio(i):
    audio_track_found = 1
    cd_object.play(i)
    pygame.time.delay(5000)
cd_object.eject(  )
cd_object.quit(  )

PyOpenGL is a fairly different tool—it is a wrapper around OpenGL, the standard for cross-platform high-performance 2-D and 3-D graphics. PyOpenGL, available from http://pyopengl.sourceforge.net, provides Python-level interfaces to a very large number of OpenGL APIs as well as related libraries such as GLU, GLUT, and more. PyOpenGL, especially when combined with Numeric Python arrays and used with a graphics card that provides hardware acceleration of OpenGL, can produce stunningly fast graphics. Anyone looking to do scientific or engineering visualization should explore this option.

29.6.3 Interfaces to GUI Toolkits

Working with a GUI toolkit is a requirement for many people who write end-user- software. The decision of which GUI toolkit to use is typically quite a complex one, involving such factors as portability, performance, look and feel, licensing and cost, threading models, etc. Suffice it to say that Python has interfaces to all of the popular GUI toolkits on the major platforms.

There are several popular cross-platform toolkits with Python interfaces:

• Tk has an interface that ships with standard Python called Tkinter. While Tk's widget set isn't as rich as others, there are ways to extend it using megawidget tools such as Pmw (http://pmw.sf.net).

• wxPython, the interface to the wxWindows GUI toolkit, has become very popular of late (see http://www.wxPython.org for details).

• PyQt on top of Qt (see http://www.riverbankcomputing.co.uk/pyqt/ for details; note the somewhat restrictive licensing terms and purchasing options).

Another cross-platform solution is:

• FXPy (http://fxpy.sourceforge.net/), the bindings to the FOX GUI library

For the most native Windows look and feel, you can use either low-level Win32 calls through the win32ui and win32gui modules, or the higher-level Microsoft Foundation Classes (MFC). Both interfaces are available with win32all, either standalone or bundled with ActivePython.

Jython users can use the native Java toolkits (AWT and Swing).

Unix/Linux users can also use:

• Python-GTK (http://www.daa.com.au/~james/pygtk/), bindings to GTK + and GNOME.

• PyKDE, bindings for the KDE desktop environment (http://www.riverbankcomputing.co.uk/pykde/index.php).

Several other toolkit interfaces are available, but have been omitted here because they are not as popular or battle-tested.

29.6.4 Interfaces to C/C++/Fortran

Many of the libraries mentioned are wrappers around third-party libraries written in a compiled language like C, C++ or Fortran. There are several options available to those wishing to expose such libraries to Python.

29.6.4.1 SWIG and f2py

The oldest, but still very useful package is SWIG (Simple Wrapper Interface Generator), available from http://www.swig.org. SWIG, unlike all of the other packages mentioned in this section, works with other languages in addition to Python. SWIG takes (a sometimes tweaked version of ) C or C++ header files and automatically produces the interfacing code that exposes the C/C++ APIs to Python. It is especially well-suited to large code bases with clean APIs, such as OpenGL, the Windows API, and other well-defined codebases. SWIG makes writing such interfaces quite simple even for nonexperts, and takes care of many of the tedious details such as reference counting, error checking and the like.

People interested in wrapping Fortran 77/90/95 should investigate f2py, available from http://cens.ioc.ee/projects/f2py2e/.

29.6.4.2 CXX and Boost::Python

Sometimes one wants to write extensions in C or C++. C extension writers tend to use the API that comes with Python (documented in Extending and Embedding the Python Interpreter, which is part of the standard documentation set, available at http://www.python.org/doc/current/ext/ext.html). C++ extension writers tend to use one of two popular tools, CXX and Boost::Python. CXX was designed to write extensions in C++ instead of C. Standard CXX code looks like a hybrid of C++ and Python—one manipulates Python-style data structures using C++ syntax. CXX takes care of reference counting and of mapping C++ exceptions to Python and vice-versa. Boost::Python is a more recent library, part of the well-respected Boost C++ library, and probably more actively maintained than CXX as of this writing. Boost::Python is a framework for interfacing Python and C++, designed to let you to quickly and seamlessly expose C++ classes functions and objects to Python, and vice-versa, using no special tools. Because it aims to wrap C++ interfaces unintrusively, you should not have to change the C++ code at all in order to wrap it. To compare the two, CXX is probably the right choice when trying to write an extension from scratch whose only purpose is to be a Python extension, while Boost::Python is more appropriate for wrapping C++ code and making a Python interface to that interface. Your mileage may vary, and you are encouraged to study both before embarking on a significant project.

29.6.4.3 Pyrex, Weave, and Psyco

SWIG and Boost::Python are designed mostly to wrap existing codes. CXX, while it can be used to interface to C++ libraries, is more of a straight port of the C extension mechanism to C++. Given that many C extensions are written strictly to optimize away some bottlenecks, an alternative approach is to use C or C++ to write special-purpose code strictly for performance reasons. Three other systems target this case: Pyrex, Weave and Psyco.

Pyrex is a language specially designed to write extensions for Python. Pyrex can be summarized as Python with C data types. It's best illustrated with a simple example:

def primes(int kmax):
    cdef int n, k, i
    cdef int p[1000]
    result = [  ]
    if kmax > 1000:
        kmax = 1000
    k = 0
    n = 2
    while k < kmax:
        i = 0
        while i < k and n % p[i] <> 0:
            i = i + 1
        if i == k:
            p[k] = n
            k = k + 1
            result.append(n)
        n = n + 1
    return result

When Pyrex runs this code, it notices the cdef statements that tell it that n, k and i are C integers, and that p is an array of 1,000 C integers. The rest of the function is straight Python code, but thanks to those declarations, Pyrex can compile the code to C, making the key loop run much faster than if this same program had run in Python.

Weave takes a different approach to solve the same fundamental problem—how to insert C/C++ code inside of otherwise perfectly fine, logical Python code. Unlike Pyrex, Weave uses no new syntax, but instead requires you to insert C or C++ code in a Python string, and to call out to a specialized module to do compilation and loading behind the scenes. So the straightforward but slow binary search routine written in Python as:

def binary_search(seq, t):
        min = 0; max = len(seq) - 1
        while 1:
            if max < min:
                return -1
            m = (min  + max)  / 2
            if seq[m] < t: 
                min = m  + 1 
            elif seq[m] > t: 
                max = m  - 1 
            else:
                return m

can be written in Weave as:

def c_int_binary_search(seq,t):
    # Do a little type checking in Python.
    assert(type(t) == type(1))
    assert(type(seq) == type([  ]))
    
    # Now the C code
    code = """
           # Line 29 "binary_search.py"
           int val, m, min = 0;  
           int max = seq.length(  ) - 1;
           PyObject *return_val; 
           for(;;) {
               if (max < min) { 
                   return_val =  Py::new_reference_to(Py::Int(-1)); 
                   break;
               } 
               m =  (min + max) /2;
               val = py_to_int(PyList_GetItem(seq.ptr(  ),m),"val"); 
               if (val  < t) 
                   min = m  + 1;
               else if (val >  t)
                   max = m - 1;
               else {
                    return_val = Py::new_reference_to(Py::Int(m));
                    break;
                }
           }
           """
    return inline(code,['seq','t'])

Weave is part of SciPy, available from http://www.scipy.org/site_content/weave.

Psyco is the most ambitious of any of the projects mentioned here, and, as a result the least practically useful at this stage. Psyco is a specializing compiler for Python. The details are much beyond the level of this book, and Psyco isn't of much use today, but it offers great promise for the future, including the possibility of making Python code run faster than C code, which is considered the benchmark for interpreted languages. The curious can read up on the current Psyco project at http://psyco.sourceforge.net/, as well as a related project, pypy, which aims to write a new implementation of Python in Python, and to build Psyco-style technology to compile it (see http://codespeak.net/pypy/).

29.6.4.4 ctypes

The latest entry in this set of tools is ctypes. ctypes uses a completely different approach to interfacing—unlike SWIG, Boost::Python or the standard C extension module system, ctypes is designed to call directly from Python into shared libraries. Using platform-specific tricks (ctypes currently runs only on Windows, Linux and OS/X, although more platforms may be added), ctypes lets you load arbitrary shared libraries and call arbitrary functions, as long as you know the signature of these functions:

>>> from ctypes import cdll, c_double
>>> printf = cdll.msvcrt.printf
>>> printf("An int %d, a double %f\n", 1234, c_double(3.14))
An int 1234, a double 3.140000

There is a lot more to ctypes that makes this approach realistic for a wide variety of applications. See http://starship.python.net/crew/theller/ctypes.html for details.

29.6.5 Little Gems

In this section, we'll list some modules that, while not necessarily deeply complex or major efforts, provide nice solutions to common problems.

The platform module (part of the standard library in Python 2.3 but available for earlier python version at http://www.egenix.com/files/python/platform.py) is a module that lets you find out many more subtle variations of what your current platform, compared to sys.platform. Instead of just saying win32, it lets you know that you're running on e.g. "Windows-2000-5.0.2195-SP3". Sometimes this sort of information is crucial, and it's hard to identify without code like that in platform.py.

turtle.py, which is part of the standard distribution but unknown to many, provides a simple LOGO-like system for interactive drawing and beginning programmers. Built on Tkinter, it provides a nice first step for teaching programming.

Trent Mick's go (at http://starship.python.net/crew/tmick/) is a simple command line tool that makes it trivial to build bookmarks of often-used directories. Instead of having to remember long path names, you can teach go which directories you use often and then use two-word commands to go there (go home, go python23, etc.).

29.6.6 Packaging Tools

Python developers who wish to share their Python code with other Python developers should learn how to use the distutils package, which lets people build distributions of their programs that are easy for other developers to install. distutils is documented as part of the standard Python library (http://www.python.org/doc/current/dist/dist.html).

Python developers who are building standalone applications for use by non-Python developers have a variety of approaches available to them. To make it easy for end users, it is generally a good idea to ship a single file that installs everything the users need—this may include all or a part of the Python distribution. While you can do it the hard way, by shipping the hundreds of files that make up this distribution, it is much easier to use a tool that packages the program and the part of the standard distribution that the program needs.

The Python distribution includes such a program, called freeze, which, while it should work, has far fewer features than the alternatives.

29.6.6.1 Installer

The most popular packaging tool is probably Gordon McMillan's Installer tool. Installer inspects your source code to find out all of the modules that it imports, as well as the modules that they import and so on, until it has built a list of all of the files it thinks your program needs. Those files, including the shared libraries that contain the Python interpreter and extension modules you use, are then packaged in an executable, which you can ship to your users. When the executable is run, it unpacks its contents in a temporary directory and runs the code from there. Installer works on Windows and Linux.

29.6.6.2 py2exe

Windows users have an alternative to Installer that has a more standard approach—it leverages the existing distutils mechanisms for defining the contents of a package, extends it with a few special purpose options, but generally performs the same task as Installer. py2exe has specialized support for packaging Windows NT Services.