26.5 Core Language Summary

Congratulations! This concludes your look at the core Python programming language. If you've gotten this far, you may consider yourself an Official Python Programmer (and should feel free to add Python to your resume the next time you dig it out). You've already seen just about everything there is to see in the language itself—all in much more depth than many practicing Python programmers. In Part II through Part VII of the book, you studied built-in types, statements, and exceptions, as well as tools used to build-up larger program units—functions, modules, and classes and explored design issues, OOP, program architecture, and more.

26.5.1 The Python Toolset

From this point forward, your future Python career will largely consist of becoming proficient with the toolset available for application-level Python programmming. You'll find this to be an ongoing task. The standard library, for example, contains some 200 modules and the public domain offers more tools still. Because new tools appear constantly, it's possible to spend a decade or more becoming proficient in all these tools. We speak from personal experience here.

In general, Python provides a hierarchy of tool sets:

Built-ins

Built-in types like strings, lists, and dictionaries make it easy to write simple programs fast.

Python extensions

For more demanding tasks, you can extend Python, by writing your own functions, modules, and classes.

C extensions

Although not covered in this book, Python can also be extended with modules written in C or C++.

Because Python layers its tool sets, you can decide how deeply your programs need to delve into this hierarchy for any given task—use built-ins for simple scripts, add Python-coded extensions for larger systems, and code C extensions for advanced work. You've covered the first two of these categories above in this book already, and that's plenty to do substantial programming in Python.

The next part of this book takes you on a tour of standard modules and common tasks in Python. Table 26-1 summarizes some of the sources of built-in or existing functionality available to Python programmers, and topics you'll explore in the remainder of this book. Up until now, most of our examples have been very small and self-contained. We wrote them that way on purpose to help you master the basics. But now that you know all about the core language, it's time to start learning how to use Python's built-in interfaces to do real work. You'll find that with a simple language like Python, common tasks are often much easier than you might expect.

Table 26-1. Python's toolbox categories

Category	Examples
Object types	Lists, dictionaries, files, strings
Functions	len, range, apply, open
Exceptions	IndexError, KeyError
Modules	string, os, Tkinter, pickle, re
Attributes	__dict__, - name- , __class__
Peripheral tools	NumPy, SWIG, Jython, PythonWin, etc.

26.5.2 Development Tools for Larger Projects

Finally, once you've mastered the basics, you'll find that your Python programs become substantially larger than the examples you've experimented with so far. For developing larger systems, a set of development tools is available in both Python and the public domain. You've seen some of these in action, and we've talked about others. To help you on your way, here is a summary of some of the most commonly used tools in this domain, many of which you've already seen:

PyDoc and docstrings

We introduced the PyDoc help function and HTML interfaces in Chapter 11. PyDoc provides a documentation system for your modules and objects, and integrates with the docstings feature of Python. PyDoc is a standard part of the Python system. See the library manual for more details. Be sure to also refer back to the documentation source hints listed in Chapter 11, for Python information resources in general.

PyChecker

Because Python is such a dynamic language, some programming errors are not reported until your program runs (e.g., syntax errors are caught when a file is run or imported). This isn't a big downside—like most languages, it just means that you have to test your Python code before shipping it. Furthermore, Python's dynamic nature, automatic error messages, and exception model, makes it easier and quicker to find and fix errors than in some languages (unlike C, Python does not crash on errors).

However, the PyChecker system provides support for catching a large set of common errors ahead of time, before your script runs. It serves similar roles to the "lint" program in C development. Some Python groups run their code through PyChecker prior to testing or delivery, to catch any lurking potential problems. In fact, the Python standard library is regularly run through PyChecker before release. PyChecker is a third party package; find it at either http://www.python.org, or the Vaults of Parnassus web site.

PyUnit (a.k.a. unittest)

In Part V, we demonstrated how to add self-test code to Python files, by using the __name__ =='__main__' trick at the bottom of the file. For more advanced testing purposes, Python comes with two testing support tools. The first, PyUnit (called unittest in the library manual), provides an object-oriented class framework, for specifying and customizing test cases and expected results. It mimics the JUnit framework for Java. This is a large class-based system; see the Python library manual for details.

Doctest

The doctest standard library module provides a second and simpler approach to regression testing. It is based upon the docstrings feature of Python. Roughly, under doctest, you cut and paste a log of an interactive testing session into the docstrings of your source files. Doctest then extracts your docstrings, parses out test cases and results, and reruns the tests to verify the expected results. Doctest's operation can be tailored in a variety of ways; see the library manual for more on its operation.

IDEs

We discussed IDEs for Python in Chapter 3. IDEs, such as IDLE, provide a graphical environment for editing, running, debugging, and browsing your Python programs. Some advance IDEs such as Komodo support additional development tasks, such as source control integration, interactive GUI builders, project files, and more. See Chapter 3, the text editors page at http://www.python.org, and the Vaults of Parnassus web site for more on available IDEs and GUI builders for Python.

Profilers

Because Python is so high-level and dynamic, intuitions about performance gleaned from experience with other languages is usually wrong. To truly isolate performance bottlenecks in your code, you need to either add timing logic with clock tools in the time module, or run your code under the profile module.

profile is a standard library module that implements a source code profiler for Python; it runs a string of code you provide (e.g., a script file import, or a call to a function), and then, by default, prints a report to the standard output stream that gives performance statistics—number of calls to each function, time spent in each function, and more. The profile module can be customized in various ways; for example, it can save run statistics to a file, to be later analyzed with the pstats module.

Debuggers

The Python standard library also includes a command line source code debugger module, called pdb. This module works much like a command line C language debugger (e.g., dbx, gdb). You import the module, start running code by calling a pdb function (e.g., pdb.run("main( )")), and then type debugging commands from an interactive prompt. Because IDEs such as IDLE include point-and-click debugging interfaces, pdb seems to be used infrequently today; see Chapter 3 for tips on using IDLE's debugging GUI interfaces.^[3]

^[3] To be honest, IDLE's debugger is not used very often either. Most practicing Python programmers end up debugging their code by inserting strategic print statements, and running again. Because turnaround from change to execution is so quick in Python, adding prints is usually faster than either typing pdb debugger commands, or starting a GUI debugging session. Another valid debugging technique is to do nothing at all—because Python prints useful error messages instead of crashing on program errors, you usually get enough information to analyze and repair errors.

Shipping options

In Chapter 2, we introduced common tools for packaging Python programs. Py2Exe, Installer, and freeze, can package byte-code and the Python Virtual Machine into "frozen binary" stand alone executables, which don't require that Python be installed in the target machine, and fully hide your system's code. In addition, you learned in Chapter 2 and Part V that Python programs may be shipped in their source (.py) or byte-code (.pyc) forms, and import hooks support special packaging techniques such as zip files and byte-code encryption. A system known as distutils also provides packaging options for Python modules and packages, and C-coded extensions; see the Python manuals for more details.

Optimization options

For optimizing your programs, the Psyco system described in Chapter 2 (and still experimental), provides a just-in-time compiler for Python byte-code to binary machine code. You may also occasionally see .pyo optimized byte-code files, generated and run with the -O Python command line flag discussed in Chapter 15; because this provides a very modest performance boost, it is not commonly used. As a last resort, you can also move parts of your program to a compiled language such as C to boost performance; see the book Programming Python and the Python standard manuals for more on C extensions. In general, Python's speed also improves over time, so be sure to upgrade to the most recent release when possible (Version 2.3 has been clocked at 15-20% faster them 2.2).

Other hints for larger projects

Finally, we've met a variety of language features that tend to become more useful once you start coding larger projects. Among these are: module packages (Chapter 17); class-based exceptions (Chapter 25); class pseudo-private attributes (Chapter 23); documentation strings (Chapter 11); module path configuration files (Chapter 15); hiding names from from* with __all__ lists and _X style names (Chapter 18); adding self-test code with the - name- =='__main__' trick (Chapter 17); using common design rules for functions and modules (Chapter 5 and Chapter 6); and so on.

For additional large-scale Python development tools available in the public domain, be sure to also browse the pages at the Vaults of Parnassus web site.