17.4 A Tale of Three Systems

The only time package imports are actually required, though, is in order to resolve ambiguities that may arise when multiple programs are installed on a single machine. This is something of an install issue, but can also become a concern in general practice. Let's turn to a hypothetical scenario to illustrate.

Suppose that a programmer develops a Python program that contains a file called utilities.py for common utility code, and a top-level file named main.py that users launch to start the program. All over this program, its files say import utilities to load and use the common code. When this program is shipped, it arrives as a single tar or zip file containing all the program's files; when it is installed, it unpacks all its files into a single directory named system1 on the target machine:

system1\
    utilities.py        # Common utility functions, classes
    main.py             # Launch this to start the program.
    other.py            # Import utilities to load my tools

Now, suppose that a second programmer does the same thing: he or she develops a different program with files utilities.py and main.py, and uses import utilities to load the common code file again. When this second system is fetched and installed, its files unpack into a new directory called system2 somewhere on the receiving machine, such that its files do not overwrite same-named files from the first system. Eventually, both systems become so popular that they wind up commonly installed in the same computer:

system2\
    utilities.py        # Common utilities
    main.py             # Launch this to run.
    other.py            # Imports utilities

So far, there's no problem: both systems can coexist or run on the same machine. In fact, we don't even need to configure the module search path to use these programs—because Python always searches the home directory first (that is, the directory containing the top-level file), imports in either system's files will automatically see all the files in that system's directory. For instance, if you click on system1\main.py, all imports will search system1 first. Similarly, if you launch system2\main.py, then system2 is searched first instead. Remember, module search path settings are only needed to import across directory boundaries.

But now, suppose that after you've installed these two programs on your machine, you decide that you'd like to use code in the utilities.py files of either of the two in a system of your own. It's common utility code, after all, and Python code by nature wants to be reused. You want to be able to say the following from code that you're writing in a third directory:

import utilities
utilities.func('spam')

to load one of the two files. And now the problem starts to materialize. To make this work at all, you'll have to set the module search path to include the directories containing the utilities.py files. But which directory do you put first in the path—system1 or system2?

The problem is the linear nature of the search path; it is always scanned left to right. No matter how long you may ponder this dilemma, you will always get utilities.py from the directory listed first (leftmost) on the search path. As is, you'll never be able to import it from the other directory at all. You could try changing sys.path within your script before each import operation, but that's both extra work, and highly error-prone. By default, you're stuck.

And this is the issue that packages actually fix. Rather than installing programs as a flat list of files in standalone directories, package and install them as subdirectories, under a common root. For instance, you might organize all the code in this example as an install hierarchy that looks like this:

root\
    system1\
        __init__.py
        utilities.py
        main.py
        other.py
    system2\
        __init__.py
        utilities.py
        main.py
        other.py
    system3\                 # Here or elsewhere
        __init__.py           # Your new code here
        myfile.py

Why You Will Care: Module Packages Now that packages are a standard part of Python, it's common to see larger third-party extensions shipped as a set of package directories, rather than a flat list of modules. The win32all Windows extensions package for Python, for instance, was one of the first to jump on the package bandwagon. Many of its utility modules reside in packages imported with paths; for instance, to load client-side COM tools: from win32com.client import constants, Dispatch this line fetches names from the client module of the win32com package (an install subdirectory). Package imports are also pervasive in code run under the Jython Java-based implementation of Python, because Java libraries are organized into a hierarchy as well. We'll see more of COM and Jython later in Part VIII. In recent Python releases, the email and XML tools are also organized into packaged subdirectories in the standard library.

Now, add just the common root directory to your search path. If your code's imports are relative to this common root, you can import either system's utility file with package imports—the enclosing directory name makes the path (and hence the module reference) unique. In fact, you can import both utility files in the same module, as long as you use the import statement and repeat the full path each time you reference the utility modules:

import system1.utilities
import system2.utilities
system1.utilities.function('spam')
system2.utilities.function('eggs')

Notice that __init__.py files were added to the system1 and system2 directories to make this work, but not to the root: only directories listed within import statements require these files.

Technically, your system3 directory doesn't have to be under root—just the packages of code from which you will import. However, because you never know when your own modules might be useful in other programs, you might as well place them under the common root to avoid similar name-collision problems in the future.

Also, notice that both of the two original systems' imports will keep working as is and unchanged: because their home directory is searched first, the addition of the common root on the search path is irrelevent to code in system1 and system2. They can keep saying just import utilities and expect to find their own file. Moreover, if you're careful to unpack all your Python systems under the common root like this, path configuration becomes simple: you'll only need to add the common root, once.