5.1 More Than One Copy Is "a Good Thing"

We begin by exploring directory replication. This feature hasn't been standardized yet; our example focuses on the OpenLDAP project. The concepts and principles that I will present are applicable to all LDAP directories, but the examples themselves are specific to OpenLDAP.

Because LDAP replication is vendor-specific at the moment, it is not possible to replicate data from one vendor's server to another vendor's server. It is possible to achieve cross-vendor replication by using tricks such as parsing a change log, but these tricks are themselves vendor-dependent.

The LDAP Duplication/Replication/Update Protocols (LDUP) Working Group of the IETF attempted to define a standardized replication protocol that would allow for interoperability between all LDAPv3-compliant servers. However, there appears to be more demand for an LDAP client update protocol (LCUP) that would allow clients to synchronize a local cache of a directory as well as be notified of updates. Details of the group's progress can be found at http://www.ietf.org/html.charters/ldup-charter.html.

A frequently asked question is: "When should I install a replica for all or part of my directory?" The answer depends heavily on your particular environment. Here are a few symptoms that indicate the need for directory replicas:

• If one application makes heavy use of the directory and slows down the server's response to other client applications, you may want to consider installing a replica and dedicating the second server solely to the application that is causing the congestion.

• If the directory server does not have enough CPU power to handle the number of requests it is receiving, installing a replica can improve response time. You may also wish to install several read-only replicas and use some means of load balancing, such as round-robin DNS or a virtual server software package. Before taking this route, make sure that the limiting factor is CPU and not other finite resources such as memory or disk I/O.

• If a group of clients is located on the other side of a slow network link, installing a local replica (i.e., a replica that is close to the clients making the request) will decrease traffic over the link and improve response time for the remote clients.

• If the directory server cannot be taken offline for backups, consider implementing a read-only replica to provide service while the master server is taken down for backups or normal maintenance.

• If your directory is a critical part of the services provided by your network, using replicas can help provide failover and redundancy.

Once the questions of "When?" and "Why?" have been answered, the next question is "How?" The OpenLDAP project uses the original design for replication that was implemented by the University of Michigan's LDAP server. This design uses a secondary daemon (slurpd) to process a change log written by the standalone LDAP server (slapd). slurpd then forwards the changes to the replica's local slapd instance using normal LDAP modify commands. Figure 5-1 displays the relationship between slapd and slurpd on the master directory server and the replica.

Figure 5-1. Dependencies between slapd, the change log, and the slurpd replication helper

5.1.1 Building slurpd

When you built the OpenLDAP package in Chapter 3, you didn't build the slurpd binary. To build slurpd, you must pass the command-line option —enable-slurpd to the configure script. Note that thread support is required for slurpd; there is no nonthreaded version.

After running ./configure with the —enable-slurpd option, executing make creates the server in the servers/slurpd/ subdirectory of the OpenLDAP source code tree. This binary can then be copied to the same location as slapd. You can copy slurpd to the appropriate location by hand or by running make install. All of the examples in this chapter assume that slurpd has been installed in the default location of /usr/local/libexec/.

5.1.2 Replication in a Nutshell

Before implementing replication in a directory, you must have a working LDAP master server. The directory server built in previous chapters will be used as a starting point. Once the master slapd server has been created, implementing a replica server can be accomplished by following these steps:

This quick list glosses over a few details, such as how to configure a server to send updates and how to configure a server to accept updates from a master server. Let's start with step 2 (if step 1 is not obvious, refer to Chapter 4 for a refresher on starting and stopping slapd).

5.1.3 Configuring the Master Server

To configure your master server to log changes that can be processed by the slurpd helper daemon, you need to add two directives to the database section of slapd.conf. It is possible to give slurpd its own configuration file using the -f command-line option, but because of the producer/consumer relationship between slapd and slurpd, the most common setup is to use a single configuration file for both daemons. Here's the database configuration developed in the previous chapter:

## -- master slapd --
#######################################################
## Define the beginning of example database.
database        bdb
      
## Define the root suffix you serve.
suffix          "dc=plainjoe,dc=org"
      
## Define a root DN for superuser privileges.
rootdn          "cn=Manager,dc=plainjoe,dc=org"
      
## Define the password used with rootdn. This is the Base64-encoded MD5 hash of
## "secret."
rootpw          {SSHA}2aksIaicAvwc+DhCrXUFlhgWsbBJPLxy
      
## Directory containing the database files
directory       /var/ldap/plainjoe.org
      
## Files should be created rw for the owner **only**.
mode            0600
      
## Indexes to maintain
index           objectClass     eq
index           cn              pres,eq
      
## db tuning parameters; cache 2,000 entries in memory
cachesize       2000
      
# Simple ACL granting read access to the world
access to * 
     by * read

First, you need to add the name of the log file in which slapd will record all LDAP modifications. This is specified using the replogfile parameter.

## -- master slapd --
# Specify the location of the file to append changes to.
replogfile     /var/ldap/slapd.replog

The slurpd daemon will use only the first instance of the replogfile parameter in a configuration file. On a server that is configured to hold multiple databases, all of these partitions must use a single replogfile value. The best way to do this is to define the replogfile parameter in the configuration file's global section. In this way, the value will be used for all databases defined later in slapd.conf. The alternative is to start a separate slurpd instance, each with its own configuration file, for each database section.

The second parameter you need to add informs slurpd where to send the changes. You add this parameter, replica, just below the replogfile directive.

## -- master slapd --
# Set the hostname and bind credentials used to propagate the changes in the
# replogfile.
replica      host=replica1.plainjoe.org:389
             suffix="dc=plainjoe,dc=org"
             binddn="cn=replica,dc=plainjoe,dc=org"
             credentials=MyPass
             bindmethod=simple
             tls=yes

replica specifies the host and port to which the data should be sent, the portion of the partition to be replicated, the DN to use when binding to the replicated server, any credentials that are acquired, and information about the binding method and protocols. Note that the binddn used in the replica directive must possess write access to the slave server. The most common binddn to use is the the rootdn specified in the replica's slapd.conf. However, any DN that has the appropriate level of access (possibly granted by ACLs on the directory) will work. The credentials parameter specifies the password used for simple binds and SASL binds to the slave server. The bindmethod option for the replica directive accepts one of the following two values: simple or sasl. Examples using SASL accounts with OpenLDAP will be presented later in this chapter.

In our example, the slave server replica1.plainjoe.org must be listening on the default LDAP port of 389. TLS is enabled (tls=yes) to protect the privacy of information as it is replicated. Because the LDAP connection will be made on port 389, it is essential that the slave server support the StartTLS extended command to ensure secure replication. Regardless of how careful you are about securing the data in your directory, if you are replicating the directory to a slave server over an insecure link and do not use some type of transport layer security such as TLS or IPSec, all of your efforts will be in vain.

5.1.4 Configuring the Replica Server

The first step in creating a replica is to initialize its database with a current copy of the directory from the master server. There are two ways to accomplish this:

• Copy the master's database files to the replica.

• Export the master's database to an LDIF file and reimport the entries into the replica.

There are a few restrictions to keep in mind when copying the actual database files from the master to the slave server:

• Both hosts must have the same (or compatible) versions of the DBM libraries.

• In most cases, both hosts must use the same byte ordering (little-endian versus big-endian).

• Some methods of copying DBM sparse files, such as using cp, will fill in the holes, resulting in much larger files on the replica host.

For these reasons, a more general way to transfer the master's database is to export the database to an LDIF file using slapcat. The file can then be imported into the replica's directory using the slapadd command. Using the slapd.conf file from your directory, the following commands initialize the replica's directory:

root@master# slapcat -b "dc=plainjoe,dc=org" -l contents.ldif
      
...copy contents.ldif to the slave server...
      
root@replica1# slapadd -l contents.ldif

In this example, specifying the base suffix with the -b option is not necessary because your server contains only one partition. Had there been more than one database section in slapd.conf, the base suffix would have specified which partition to dump. Specifying a base suffix isn't necessary when you import the file into the replica because slapadd parses the configuration file and places the data into the first database to match the base suffix of the LDIF entries.

Once the data has been copied to the slave server, it is time to update the replica's slapd.conf to accept updates from the master server. The global section of the replica's configuration file will be identical to the master server's. However, certain pieces of information, such as the server's public certificate, should be unique to the slave. The database section of the slave's slapd.conf will also be identical, minus the replication parameters and with an appropriate local rootdn and rootpw. For the purposes of this chapter, the slave's database section contains the following rootdn and rootpw:

## -- slave slapd --
## replica's administrative DN
rootdn     "cn=replica,dc=plainjoe,dc=org"
      
## Salted Secure Hash version of MyPass
rootpw     {SSHA}SMKnFPO435G+QstIzNGb4RGjT0KLz2TV

To make the server act as a slave, you must add two parameters to the configuration. Just as the master must know where to send updates, the slave server must know who is authorized to make these changes. This is done by defining an updatedn:

## -- slave slapd --
## Define the DN that will be used by the master slurpd to replicate data. Normally,
## this is the rootdn of the slave server or, at the minimum, a DN that is allowed
## write access to all entries via an ACL.
updatedn     "cn=replica,dc=plainjoe,dc=org"

Because a slave server contains only replicated data and OpenLDAP currently supports only a single master replication system (i.e., updates must be made on the master directory server), the slave server requires an LDAP URL (updateref) that points clients to the master directory server. The slave refers clients to the master when clients send modification requests. Here's the appropriate addition to the database section of the replica's slapd.conf:

## -- slave slapd --
## Specify the LDAP URL of the master server, which can accept update requests.
updateref     ldap://pogo.plainjoe.org

Development versions of OpenLDAP support an experimental implementation of a multimaster replication protocol. Multimaster replication means that changes to the directory can be accepted at any replica: the replica propagates the changes to all servers containing copies of the partition. Multimaster replication is not covered in this book.

This completes step 4 of the configuration process. Steps 5 and 6 are to launch the master and slave's slapd processes using the procedure described in earlier chapters.

5.1.5 slurpd's replogfile

As Figure 5-1 illustrates, the slurpd daemon processes the change log written by slapd. The replication log uses a format similar to the LDIF examples used throughout this book. After reading the replogfile, slurpd copies the entry to its own replay log. The location of the slurpd.replog file can be controlled using the -t command-line option when starting slurpd. In a default installation, slurpd.replog will be stored in /usr/local/var/openldap-slurp/replica/.

The following log entry in the replogfile was generated when the email address for cn=Jerry Carter,ou=people,dc=plainjoe,dc=org was changed to jcarter@nowhere.com:

replica: pogo.plainjoe.org
time: 975434259
dn: cn=jerry carter,ou=People,dc=plainjoe,dc=org
changetype: modify
replace: mail
mail: jcarter@nowhere.com
-
replace: entryCSN
entryCSN: 2002110403:55:49Z#0x0001#0#0000
-
replace: modifiersName
modifiersName: cn=Manager,dc=plainjoe,dc=org
-
replace: modifyTimestamp
modifyTimestamp: 20001128175739Z
-

While only one attribute value was changed, the replication log entry updates four attributes: mail (as expected), plus modifiersName, modifyTimestamp, and entryCSN. These last three attributes, two of which are described in RFC 2251, are maintained by the LDAP server and cannot be modified by clients.

The log entry also specifies two additional values not used in normal LDIF entries. The replica directive defines the host to which the change should be propagated. There can be multiple replica lines if a partition will be synchronized to several directory slaves. The time parameter defines the timestamp of the entry in seconds since 1970 GMT. slurpd maintains the timestamp of the most recently read change, which prevents it from reparsing entries it has already processed. This state information is stored in a status file named slurpd.status in the same directory as the slurpd.replog file.

slurpd reads entries in the replication log file one at a time and propagates the changes using basic LDAP commands (e.g., add, modify, delete, modrdn, etc.). If a change cannot be made, slurpd writes the entry and reason for the failure to a reject log named <hostname:port>.rej in the same directory as the slurpd.replog file. It's the administrator's responsibility to read this log and figure out how to handle changes that were not made. There are many possible reasons for failure—for example, a conflict between the schemas on the master and slave servers, access control entries that aren't set up appropriately, etc.

Certain errors, such as a network problem in the connection from the server to the slave, cause slurpd to requeue a modification. However, any entry written to the reject log cannot be replicated in its current state. These entries will require manual intervention on a case-by-case basis. If the update resulted in an object schema violation, perhaps one of the schema files was left out of the configuration file. If the replica arrived in a state inconsistent with the directory master, attempting to add an entry that already existed would also result in an error. These types of circumstances can be very data-specific, and the examples given here represent only a few of the possible causes. In any case, the appropriate response to an error isn't to try to update the slave by hand; rather, you should figure out why the update can't take place automatically, and fix the appropriate configuration file.

Our setup will utilize slurpd as a daemon that monitors the replication log file and propagates changes periodically. However, it can also operate as a command-line tool to process a single change log and exit. This is referred to as "one shot" mode, and it's invaluable for dealing with rejected entries. A complete list of command-line options used when starting slurpd is given in Table 5-1.

Table 5-1. Command-line options for slurpd

Option	Description
-d integer	Specifies the debugging information to be written to the log. See the discussion of the loglevel slapd.conf parameter for more information on the debugging flags.
-f "config filename"	Specifies the location and filename of the slapd.conf file to be used.
-r replogfile	Specifies the location and filename of the replication log to be processed. The -r switch is often used in conjunction with the slurpd's one-shot mode to process a particular file.
-o	Executes slurpd in one-shot mode to process a single replication log and exit.
-t directory	Specifies the directory to be used for temporary files such as lock and status files. The default is to store the files in /usr/local/var/openldap-slurp/replica/.
-k "srvtab file"	Specifies the location of the srvtab file when using LDAPv2 Kerberos 4 binds to a slave slapd server.

To complete the configuration of your directory replica, you must start the slurpd daemon on the master server. To do so, execute slurpd as root from a shell prompt:

root@master# /usr/local/libexec/slurpd

From this point on, any changes made to the master directory will be replicated to the slave server. In current 2.x versions, the slurpd thread responsible for monitoring the replication log checks every three seconds for updates and propagates these updates as necessary. This interval is not configurable from the command line nor from slapd.conf. However, if less frequent updates are required, slurpd could be run manually using the -o argument to process the replication log when desired.

Using a Replica in a Backup Plan Backing up a directory that must be available 24 hours a day can require special arrangements. By using replication, a slave server can act as a backup server. Even if the backup plan relies on offline storage, a directory replica can be very helpful. If you stop the slurpd daemon on the master server, the replica server contains a read-only copy of the directory at that point in time. Of course, the master slapd continues to write changes to the replogfile. You can now use slapcat on the slave to dump the database files to LDIF while slapd is running because data is guaranteed not to change in mid-backup. Once the backup is complete, restarting slurpd ensures that all changes made to the master while the replica was being backed up will propagated, bringing the slave in sync with the current state of the directory.