Before you start to configure a cluster, it is worth explaining how to view the finished product. For this purpose use the crm_mon utility that will display the current state of an active cluster. It can show the cluster status by node or by resource and can be used in either single-shot or dynamically-updating mode. Using this tool, you can examine the state of the cluster for irregularities and see how it responds when you cause or simulate failures.

Details on all the available options can be obtained using the crm_mon --help command.

There is a basic warning for updating the cluster configuration:

	Rules For Updating the Configuration
Never edit the cib.xml file manually, otherwise the cluster will refuse to use the configuration. Instead, always use the cibadmin tool to change your configuration.

To modify your cluster configuration, use the cibadmin command which talks to a running cluster. With cibadmin, you can query, add, remove, update or replace any part of the configuration. All changes take effect immediately and there is no need to perform a reload-like operation.

The simplest way of using cibadmin is a three-step procedure:

If you only want to modify the resources section, do the following to avoid modifying any other part of the configuration:

cibadmin --cib_query --obj_type resources > /tmp/tmp.xml
vi /tmp/tmp.xml
cibadmin --cib_replace --obj_type resources --xml-file /tmp/tmp.xml

Sometimes it is necessary to delete an object quickly. This can be done in three easy steps:

Some common tasks can also be performed with one of the higher level tools that avoid the need to read or edit XML. Run the following command to enable STONITH, for example:

crm_attribute --attr-name stonith-enabled --attr-value true

Or to see if somenode is allowed to run resources, there is:

crm_standby --get-value --node-uname somenode

Or to find the current location of my-test-rsc one can use:

crm_resource --locate --resource my-test-rsc

It is not necessary to modify a real cluster in order to test the effect of the configuration changes. Do the following to test your modifications:

The tool uses the same library as the live cluster to show the impact it would have done. Its output, in addition to a significant amount of logging, is stored in two files, tmp.graph and tmp.dot. Both files are representations of the same thing—the cluster’s response to your changes. In the graph file the complete transition is stored, containing a list of all actions, their parameters and their prerequisites. The transition graph is not very easy to read. Therefore, the tool also generates a Graphviz dot-file representing the same information.

All LSB scripts are commonly found in the directory /etc/init.d. They must have several actions implemented, which are at least start, stop, restart, reload, force-reload, and status as explained in http://www.linux-foundation.org/spec/refspecs/LSB_1.3.0/gLSB/gLSB/iniscrptact.html.

The configuration of those services is not standardized. If you intend to use an LSB script with Heartbeat, make sure that you understand how the respective script is configured. Often you can find some documentation to this in the documentation of the respective package in /usr/share/doc/packages/<package_name>.

When used by Heartbeat, the service should not be touched by other means. This means that it should not be started or stopped on boot, reboot, or manually. However, if you want to check if the service is configured properly, start it manually, but make sure that it is stopped again before Heartbeat takes over.

Before using an LSB resource, make sure that the configuration of this resource is present and identical on all cluster nodes. The configuration is not managed by Heartbeat. You must take care of that yourself.

All OCF agents are located in /usr/lib/ocf/resource.d/heartbeat/. These are small programs that have a functionality similar to that of LSB scripts. However, the configuration is always done with environment variables. All OCF Resource Agents are required to have at least the actions start, stop, status, monitor, and meta-data. The meta-data action retrieves information about how to configure the agent. For example, if you want to know more about the IPaddr agent, use the command:

/usr/lib/ocf/resource.d/heartbeat/IPaddr meta-data

The output is lengthy information in a simple XML format. You can validate the output with the ra-api-1.dtd DTD. Basically this XML format has three sections—first several common descriptions, second all the available parameters, and last the available actions for this agent.

A typical parameter of a OCF RA as shown with the meta-data command looks like this:

<!DOCTYPE resource-agent SYSTEM "ra-api-1.dtd">
<resource-agent name="apache"> 
  <!-- Some elements omitted -->
  <parameter name="ip" unique="1" required="1">
    <longdesc lang="en">
The IPv4 address to be configured in dotted quad notation, for example
"192.168.1.1".
    </longdesc>
    <shortdesc lang="en">IPv4 address</shortdesc>
    <content type="string" default="" />
  </parameter>
</resource-agent>

This is part of the IPaddr RA. The information about how to configure the parameter of this RA can be read as follows:

Find a configuration example for this RA at Chapter 3, Setting Up a Simple Resource.

To set up the NFS server, three resources are needed: a file system resource, a drbd resource, and a group of an NFS server and an IP address. You can write each of the resource configurations to a separate file then load them to the cluster with cibadmin -C -o resources -x resource_configuration_file.

<primitive id="filesystem_resource" class="ocf" provider="heartbeat" type="Filesystem">
  <instance_attributes id="ia-filesystem_1">
    <attributes>
      <nvpair id="filesystem-nv-1" name="device" value="/dev/drbd0"/>
      <nvpair id="filesystem-nv-2" name="directory" value="/srv/failover"/>
      <nvpair id="filesystem-nv-3" name="fstype" value="reiserfs"/>
    </attributes>
  </instance_attributes>
</primitive>
    

<master_slave id="drbd_resource" ordered="false">
  <meta_attributes>
    <attributes>
      <nvpair id="drbd-nv-1" name="clone_max" value="2"/> 
      <nvpair id="drbd-nv-2" name="clone_node_max" value="1"/>
      <nvpair id="drbd-nv-3" name="master_max" value="1"/>
      <nvpair id="drbd-nv-4" name="master_node_max" value="1"/>
      <nvpair id="drbd-nv-5" name="notify" value="yes"/>
    </attributes>
  </meta_attributes>
  <primitive id="drbd_r0" class="ocf" provider="heartbeat" type="drbd">
    <instance_attributes id="ia-drbd_1">
      <attributes>
        <nvpair id="drbd-nv-5" name="drbd_resource" value="r0"/>
      </attributes>
    </instance_attributes>
  </primitive>
</master_slave>

<group id="nfs_group">
  <primitive id="nfs_resource" class="lsb" type="nfsserver"/>
  <primitive id="ip_resource" class="ocf" provider="heartbeat" 
    type="IPaddr">
    <instance_attributes id="ia-ipaddr_1">
      <attributes>
        <nvpair id="ipaddr-nv-1" name="ip" value="10.10.0.1"/>
      </attributes>
    </instance_attributes>
  </primitive>
</group>

This type of constraint may be added multiple times for each resource. All rsc_location constraints are evaluated for a given resource. A simple example that increases the probability to run a resource with the ID filesystem_1 on the node with the name earth to 100 would be the following:

<rsc_location id="filesystem_1_location" rsc="filesystem_1">
   <rule id="pref_filesystem_1" score="100">
      <expression attribute="#uname" operation="eq" value="earth"/>
   </rule>
</rsc_location>

The rsc_colocation constraint is used to define what resources should run on the same or on different hosts. It is not possible to give a score other than INFINITY or -INFINITY, defining resources to run together always or never to run together. For example, to run the two resources with the IDs filesystem_resource and nfs_group always on the same host, use the following constraint:

<rsc_colocation 
  id="nfs_on_filesystem"
  to="filesystem_resource"
  from="nfs_group" 
  score="INFINITY"/>

For a master slave configuration, it is necessary to know if the current node is a master in addition to running the resource locally. This can be checked with an additional to_role or from_role attribute.

Sometimes it is necessary to provide an order in which services must start. For example, you cannot mount a file system before the device is available to a system. Ordering constraints can be used to start or stop a service right before or after a different resource meets a special condition, such as being started, stopped, or promoted to master. An ordering constraint looks like the following:

<rsc_order id="nfs_after_filesystem" from="group_nfs" action="start"
      to="filesystem_resource" to_action="start" type="after"/>

With type="after", the action of the from resource is done after the action of the to resource.

The example used for this chapter is quite useless without additional constraints. It is essential that all resources run on the same machine as the master of the drbd resource. Another thing that is critical is that the drbd resource must be master before any other resource starts. Trying to mount the drbd device when drbd is not master simply fails. The constraints that must be fulfilled look like the following: