21.5 Namespaces: The Whole Story

Now that we've seen class and instance objects, the Python namespace story is complete; for reference, let's quickly summarize all the rules used to resolve names. The first things you need to remember are that qualified and unqualified names are treated differently, and that some scopes serve to initialize object namespaces:

• Unqualified names (e.g., X) deal with scopes.

• Qualified attribute names (e.g., object.X) use object namespaces.

• Some scopes initialize object namespaces (modules and classes).

21.5.1 Unqualified Names: Global Unless Assigned

Unqualified names follow the LEGB lexical scoping rules outlined for functions in Chapter 13:

Assignment: X = value

Makes names local: creates or changes name X in the current local scope, unless declared global

Reference: X

Looks for name X in the current local scope, then any and all enclosing functions, then the current global scope, then the built-in scope

21.5.2 Qualified Names: Object Namespaces

Q ualified names refer to attributes of specific objects and obey the rules for modules and classes. For class and class instance objects, the reference rules are augmented to include the inheritance search procedure:

Assignment: object.X = value

Creates or alters the attribute name X in the namespace of the object being qualified, and no other. Inheritance tree climbing only happens on attribute reference, not on attribute assignment.

Reference: object.X

For class-based objects, searches for the attribute name X in the object, then in all accessible classes above it, using the inheritance search procedure. For non-class objects such as modules, fetches X from object directly.

21.5.3 Assignments Classify Names

With distinct search procedures for qualified and unqualified names, and multiple lookup layers for both, it can sometimes be confusing to know where a name will wind up going. In Python, the place where you assign a name is crucial—it fully determines which scope or which object a name will reside in. File manynames.py illustrates and summarizes how this translates to code:

X = 11                  # Module (global) name/attribute

class c:
    X = 22              # Class attribute
    def m(self):
        X = 33          # Local variable in method
        self.X = 44     # Instance attribute

def f(  ):
    X = 55              # Local variable in function

def g(  ):
    print X             # Access module X (11)

Because this file assigns the same name, X, in five different locations, there are actually five completely different Xs in this program. From top to bottom, the assignments to X names generate a module attribute, a class attribute, a local variable in a method, an instance attribute, and a local in a function.

You should take enough time to study this example carefully, because it collects ideas we've been exploring throughout the last few parts of this book. When it makes sense to you, you will have achieved Python namespace nirvana. Of course, an alternative route to nirvana is to simply run this and see what happens. Here's the remainder of this file, which makes an instance, and prints all the Xs that it can fetch:

obj = c(  )
obj.m(  )

print obj.X             # 44: instance
print c.X               # 22: class     (a.k.a. obj.X if no X in instance)
print X                 # 11: module    (a.k.a. manynames.X outside file)

#print c.m.X            # FAILS: only visible in method
#print f.X              # FAILS: only visible in function

Notice that we can go through the class to fetch its attribute (c.X), but can never fetch local variables in functions or methods from outside their def statements. Locals are only visible to other code within the def, and in fact only live in memory while a call to the function or method is executing.

21.5.4 Namespace Dictionaries

In Chapter 16, we learned that module namespaces are actually implemented as dictionaries and exposed with the built-in __dict__ attribute. The same holds for class and instance objects: attribute qualification is really a dictionary indexing operation internally, and attribute inheritance is just a matter of searching linked dictionaries. In fact, instance and class objects are mostly just dictionaries with links inside Python. Python exposes these dictionaries, as well as the links between them, for use in advanced roles (e.g., for coding tools).

To help you understand how attributes work internally, let's work through an interactive session that traces the way namespace dictionaries grow when classes are involved. First, let's define a superclass and a subclass with methods that will store data in their instances:

>>> class super:
...     def hello(self):
...         self.data1 = 'spam'
...
>>> class sub(super):
...     def hola(self):
...         self.data2 = 'eggs'

When we make an instance of the subclass, the instance starts out with an empty namespace dictionary, but has links back to the class for the inheritance search to follow. In fact, the inheritance tree is explicitly available in special attributes, which you can inspect: instances have a __class__ attribute that links to their class, and classes have a __bases__ attribute that is a tuple containing links to higher superclasses:

>>> X = sub(  )
>>> X.__dict__
{  }

>>> X.__class__
<class __main__.sub at 0x00A48448>

>>> sub.__bases__
(<class __main__.super at 0x00A3E1C8>,)

>>> super.__bases__
(  )

As classes assign to self attributes, they populate the instance object—that is, attributes wind up in the instance's attribute namespace dictionary, not in the classes. Instance object namespaces record data that can vary from instance to instance, and self is a hook into that namespace:

>>> Y = sub(  )

>>> X.hello(  )
>>> X.__dict__
{'data1': 'spam'}

>>> X.hola(  )
>>> X.__dict__
{'data1': 'spam', 'data2': 'eggs'}
 
>>> sub.__dict__
{'__module__': '__main__', '__doc__': None, 'hola': <function hola at
 0x00A47048>}

>>> super.__dict__
{'__module__': '__main__', 'hello': <function hello at 0x00A3C5A8>,
 '__doc__': None}

>>> sub.__dict__.keys(  ), super.__dict__.keys(  )
(['__module__', '__doc__', 'hola'], ['__module__', 'hello', '__doc__'])

>>> Y.__dict__
{  }

Notice the extra underscore names in the class dictionaries; these are set by Python automatically. Most are not used in typical programs, but some are utilized by tools (e.g., __doc__ holds the docstrings discussed in Chapter 11).

Also observe that Y, a second instance made at the start of this series, still has an empty namespace dictionary at the end, even though X's has been populated by assignments in methods. Each instance is an independent namespace dictionary, which starts out empty, and can record completely different attributes than other instances of the same class.

Now, because attributes are actually dictionary keys inside Python, there are really two ways to fetch and assign their values—by qualification, or key indexing:

>>> X.data1, X.__dict__['data1']
('spam', 'spam')

>>> X.data3 = 'toast'
>>> X.__dict__
{'data1': 'spam', 'data3': 'toast', 'data2': 'eggs'}

>>> X.__dict__['data3'] = 'ham'
>>> X.data3
'ham'

This equivalence only applies to attributes attached to the instance, though. Because attribute qualification also performs inheritance, it can access attributes that namespace dictionary indexing cannot. The inherited attribute X.hello, for instance, cannot be had by X.__dict__['hello'].

And finally, here is the built-in dir function we met in Chapter 3 and Chapter 11 at work on class and instance objects. This function works on anything with attributes: dir(object) is similar to an object.__dict__.keys( ) call. Notice though, that dir sorts its list, and includes some system attributes; as of Python 2.2, dir also collects inherited attributes automatically.^[6]

^[6] The content of attribute dictionaries and dir call results is prone to change over time. For example, because Python now allows built-in types to be subclassed like classes, the contents of dir results for built-in types expanded to include operator overloading methods. In general, attribute names with leading and trailing double underscores are interpreter-specific. More on type subclasses in Chapter 23.

>>> X.__dict__
{'data1': 'spam', 'data3': 'ham', 'data2': 'eggs'}
>>> X.__dict__.keys(  )
['data1', 'data3', 'data2']

>>>> dir(X)
['__doc__', '__module__', 'data1', 'data2', 'data3', 'hello', 'hola']
>>> dir(sub)
['__doc__', '__module__', 'hello', 'hola']
>>> dir(super)
['__doc__', '__module__', 'hello']

Experiment with these special attributes on your own to get a better feel for how namespaces actually do their attribute business. Even if you will never use these in the kinds of programs you write, it helps demystify the notion of namespaces in general when you see that they are just normal dictionaries.

21.5.5 Namespace Links

The prior section introduced the special __class__ and __bases__ instance and class attributes without really telling why you might care about them. In short, they allow you to inspect inheritance hierarchies within your own code. For example, they can be used to display a class tree, as in the following example, file classtree.py:

def classtree(cls, indent):
    print '.'*indent, cls.__name__        # Print class name here.
    for supercls in cls.__bases__:        # Recur to all superclasses
        classtree(supercls, indent+3)         # May visit super > once

def instancetree(inst):
    print 'Tree of', inst                     # Show instance.
    classtree(inst.__class__, 3)          # Climb to its class.

def selftest(  ):
    class A: pass
    class B(A): pass
    class C(A): pass
    class D(B,C): pass
    class E: pass
    class F(D,E): pass
    instancetree(B(  ))
    instancetree(F(  ))
    
if __name__ == '__main__': selftest(  )

The classtree function in this script is recursive—it prints a class's name using __name__, and then climbs up to superclasses by calling itself. This allows the function to traverse arbitrarily shaped class trees; the recursion climbs to the top, and stops at root superclasses that have empty __bases__. Most of this file is self-test code; when run standalone, it builds an empty class tree, makes two instance from it, and prints their class tree structures:

% python classtree.py
Tree of <__main__.B instance at 0x00ACB438>
... B
...... A
Tree of <__main__.F instance at 0x00AC4DA8>
... F
...... D
......... B
............ A
......... C
............ A
...... E

Here, indentation marked by periods is used to denote class tree height. We can import these functions anywhere we want a quick class tree display:

>>> class Emp: pass
>>> class Person(Emp): pass
>>> bob = Person(  )
>>> import classtree
>>> classtree.instancetree(bob)
Tree of <__main__.Person instance at 0x00AD34E8>
... Person
...... Emp

Of course, we could improve on this output format, and perhaps even sketch it in a GUI display. Whether or not you will ever code or use such tools, this example demonstrates one of the many ways that we can make use of special attributes that expose interpreter internals. We'll meet another when we code a general purpose attribute listing class, in Seciton 22.6 of Chapter 22 .