The packages needed for configuring and managing a cluster are included in the High Availability installation pattern, available with the High Availability Extension.

Check if High Availability Extension is installed as an add-on to SUSE Linux Enterprise Server 11 SP2 on each of the cluster nodes and if the High Availability pattern is installed on each of the machines as described in Section 3.3, “Installation as Add-on”.

To communicate with each other, all nodes belonging to the same cluster need to use the same bindnetaddr, mcastaddr and mcastport as described in Section 3.5, “Manual Cluster Setup (YaST)”.

Check if the communication channels and options configured in /etc/corosync/corosync.conf are the same for all cluster nodes.

In case you use encrypted communication, check if the /etc/corosync/authkey file is available on all cluster nodes.

All corosync.conf settings with the exception of nodeid must be the same; authkey files on all nodes must be identical.

If the mcastport used for communication between the cluster nodes is blocked by the firewall, the nodes cannot see each other. When configuring the initial setup with YaST as described in , the firewall settings are usually automatically adjusted.

To make sure the mcastport is not blocked by the firewall, check the settings in /etc/sysconfig/SuSEfirewall2 on each node. Alternatively, start the YaST firewall module on each cluster node. After clicking Allowed Service+Advanced, add the mcastport to the list of allowed UDP Ports and confirm your changes.

Check the OpenAIS status on each cluster node with /etc/init.d/openais status. In case OpenAIS is not running, start it by executing /etc/init.d/openais start.

The lrmd daemon does not log recurring monitor operations unless an error occurred. Logging all recurring operatings would produce too much noise. Therefore recurring monitor operations are logged only once an hour.

Add the --verbose parameter to your commands. If you do that multiple times, the debug output becomes quite verbose. See /var/log/messages for useful hints.

Use the crm_mon command. The following displays the resource operation history (option -o) and inactive resources (-r):

crm_mon -o -r

The display is refreshed when the status changes (to cancel this press Ctrl+C.) An example may look like:

Refresh in 10s...

============
Last updated: Mon Jan 19 08:56:14 2009
Current DC: d42 (d42)
3 Nodes configured.
3 Resources configured.
============

Online: [ d230 d42 ]
OFFLINE: [ clusternode-1 ]

Full list of resources:

Clone Set: o2cb-clone
         Stopped: [  o2cb:0 o2cb:1o2cb:2 ]
Clone Set: dlm-clone
         Stopped [ dlm:0 dlm:1 dlm:2 ]
mySecondIP      (ocf::heartbeat:IPaddr):        Stopped

Operations:
* Node d230:
   aa: migration-threshold=1000000
    + (5) probe: rc=0 (ok)
    + (37) stop: rc=0 (ok)
    + (38) start: rc=0 (ok)
    + (39) monitor: interval=15000ms rc=0 (ok)
* Node d42:
   aa: migration-threshold=1000000
    + (3) probe: rc=0 (ok)
    + (12) stop: rc=0 (ok)

First get your node online, then check your resources and operations.

The Configuration Explained PDF covers three different recovery types in the How Does the Cluster Interpret the OCF Return Codes? section. It is available at http://www.clusterlabs.org/doc/ .

Use the following commands :

crm resource list
crm resource cleanup rscid [node]

If you leave out the node, the resource is cleaned on all nodes. More information can be found in Section 7.4.2, “Cleaning Up Resources”.

Use the command crm resource list to display your current resources.

To check an OCF script use ocf-tester, for instance:

ocf-tester -n ip1 -o ip=YOUR_IP_ADDRESS \
  /usr/lib/ocf/resource.d/heartbeat/IPaddr

Use -o multiple times for more parameters. The list of required and optional parameters can be obtained by running crm ra info AGENT, for example:

crm ra info ocf:heartbeat:IPaddr

Before running ocf-tester, make sure the resource is not managed by the cluster.

If your cluster is a two node cluster, killing one node will leave the remaining node without quorum. Unless you set the no-quorum-policy property to ignore, nothing happens. For two-node clusters you need:

property no-quorum-policy="ignore"

Another possibility is that the killed node is considered unclean. Then it is necessary to fence it. If the stonith resource is not operational or does not exist, the remaining node will waiting for the fencing to happen. The fencing timeouts are typically high, so it may take quite a while to see any obvious sign of problems (if ever).

Yet another possible explanation is that a resource is simply not allowed to run on this node. That may be due to a failure which happened in the past and which was not “cleaned”. Or it may be due to an earlier administrative action, i.e. a location constraint with a negative score. Such a location constraint is for instance inserted by the crm resource migrate command.

If there are no location constraints for a resource, its placement is subject to an (almost) random node choice. You are well advised to always express a preferred node for resources. That does not mean that you need to specify location preferences for all resources. One preference suffices for a set of related (colocated) resources. A node preference looks like this:

location rsc-prefers-node1 rsc 100: node1

Start (or enable) operation includes checking the status of the device. If the device is not ready, the STONITH resource will fail to start.

At the same time the STONITH plugin will be asked to produce a host list. If this list is empty, there is no point in running a STONITH resource which cannot shoot anything. The name of the host on which STONITH is running is filtered from the list, since the node cannot shoot itself.

If you want to use single-host management devices such as lights-out devices, make sure that the stonith resource is not allowed to run on the node which it is supposed to to fence. Use an infinitely negative location node preference (constraint). The cluster will move the stonith resource to another place where it can start, but not before informing you.

Each STONITH resource must provide a host list. This list may be inserted by hand in the STONITH resource configuration or retrieved from the device itself, for instance from outlet names. That depends on the nature of the STONITH plugin. stonithd uses the list to find out which STONITH resource can fence the target node. Only if the node appears in the list can the STONITH resource shoot (fence) the node.

If stonithd does not find the node in any of the host lists provided by running STONITH resources, it will ask stonithd instances on other nodes. If the target node does not show up in the host lists of other stonithd instances, the fencing request ends in a timeout at the originating node.

Power management devices may give up if there is too much broadcast traffic. Space out the monitor operations. Given that fencing is necessary only once in a while (and hopefully never), checking the device status once a few hours is more than enough.

Also, some of these devices may refuse to talk to more than one party at the same time. This may be a problem if you keep a terminal or browser session open while the cluster tries to test the status.

Use the command pssh for this task. If necessary, install pssh. Create a file (for example hosts.txt) where you collect all your IP addresses or hostnames you want to visit. Make sure you can log in with ssh to each host listed in your hosts.txt file. If everything is correctly prepared, execute pssh and use the hosts.txt file (option -h) and the interactive mode (option -i) as shown in this example:

pssh -i -h hosts.txt "ls -l /corosync/*.conf"
[1] 08:28:32 [SUCCESS] root@venus.example.com
-rw-r--r-- 1 root root 1480 Nov 14 13:37 /etc/corosync/corosync.conf
[2] 08:28:32 [SUCCESS] root@192.168.2.102
-rw-r--r-- 1 root root 1480 Nov 14 13:37 /etc/corosync/corosync.conf

To check the current state of your cluster, use one of the programs crm_mon or crm status. This displays the current DC as well as all the nodes and resources known by the current node.

There could be several reasons:

Check /var/log/messages for the following line:

Jan 12 09:58:55 clusternode2 lrmd: [3487]: info: RA output: (o2cb:1:start:stderr) 2009/01/12_09:58:55 
  ERROR: Could not load ocfs2_stackglue
Jan 12 16:04:22 clusternode2 modprobe: FATAL: Module ocfs2_stackglue not found.

In this case the kernel module ocfs2_stackglue.ko is missing. Install the package ocfs2-kmp-default, ocfs2-kmp-pae or ocfs2-kmp-xen, depending on the installed kernel.

Use the tool hb_report to create a report. The tool is used to compile:

Usually run hb_report with the following command:

hb_report -f 0:00 -n earth -n venus

The command extracts all information since 0am on the hosts earth and venus and creates a tar.bz2 archive named hb_report-DATE.tar.bz2 in the current directory, for example, hb_report-Wed-03-Mar-2012. If you are only interested in a specific time frame, add the end time with the -t option.

	Remove Sensitive Information
The hb_report tool tries to remove any sensitive information from the CIB and the peinput files, however, it can not do everything. If you have more sensitive information, supply additional patterns. The logs and the crm_mon, ccm_tool, and crm_verify output are not sanitized. Before sharing your data in any way, check the archive and remove all information you do not want to expose.

Customize the command execution with further options. For example, if you have an OpenAIS cluster, you certainly want to add the option -A. In case you have another user who has permissions to the cluster, use the -u option and specify this user (in addition to root and hacluster). Further options can be found in the manpage of hb_report.

After hb_report analyzed all the relevant log files and created the directory (or archive), check the logs for an uppercase ERROR string. The most important files in the top level directory of the report are:

Node-specific files are stored in a subdirectory named by the node's name.