Contents
Abstract
This chapter describes how to install and set up SUSE® Linux Enterprise High Availability Extension 11 SP2 from scratch. Choose between an automatic setup which allows you to have a cluster up and running within a few minutes (with the choice to adjust any options later on) or decide for a manual setup, allowing you to set your individual options right at the beginning.
Refer to chapter Appendix D, Upgrading Your Cluster to the Latest Product Version if you want to migrate an existing cluster that runs an older version of SUSE Linux Enterprise High Availability Extension or if you want to update any software packages on nodes that are part of a running cluster.
The term “existing cluster” is used to refer to any cluster which consists of at least one node. Existing clusters have a basic Corosync configuration that defines the communication channels, but they do not necessarily have resource configuration yet.
A technology used for a one-to-many communication within a network that can be used for cluster communication. If multicast does not comply with your corporate IT policy, use unicast instead.
![]() | Switches and Multicast |
|---|---|
If you want to use multicast for cluster communication, make sure your switches support multicast. | |
mcastaddr)
IP address to use for multicasting (either IPv4 or IPv6). You can use any multicast address in your private network.
mcastport)
The port to use for cluster communication.
A technology for sending messages to a single network destination. In Corosync, unicast is implemented as UDP-unicast (UDPU).
bindnetaddr)
The network address to bind to. To ease sharing configuration files across the cluster, OpenAIS uses network interface netmask to mask only the address bits that are used for routing the network.
![]() | Network Address for All Nodes |
|---|---|
As the same Corosync configuration will be used on all nodes, make
sure to use a network address as
| |
Corosync supports the Totem Redundant Ring Protocol. It allows the use of multiple redundant local-area networks for resilience against partial or total network faults. This way, cluster communication can still be kept up as long as a single network is operational. For more information, refer to http://www.rcsc.de/pdf/icdcs02.pdf.
When having defined redundant communication channels in Corosync,
use RRP to tell the cluster how to use these interfaces. RRP can have
three modes (rrp_mode): if set to
active, Corosync uses both interfaces actively.
If set to passive, Corosync uses the second
interface only if the first ring fails. If rrp_mode is set to
none, RRP is disabled.
A synchronization tool that can be used to replicate configuration
files across all nodes in the cluster. Csync2 can handle any number
of hosts, sorted into synchronization groups. Each synchronization
group has its own list of member hosts and its include/exclude
patterns that define which files should be synchronized in the
synchronization group. The groups, the hostnames belonging to each
group, and the include/exclude rules for each group are specified in
the Csync2 configuration file,
/etc/csync2/csync2.cfg.
For authentication, Csync2 uses the IP addresses and pre-shared keys within a synchronization group. You need to generate one key file for each synchronization group and copy it to all group members.
For more information about Csync2, refer to http://oss.linbit.com/csync2/paper.pdf
To synchronize the connection status between cluster nodes, the High Availability Extension
uses the
conntrack-tools.
They allow to interact with the in-kernel Connection Tracking System
for enabling stateful packet inspection for
iptables. For detailed information, refer to
http://conntrack-tools.netfilter.org/.
AutoYaST is a system for installing one or more SUSE Linux Enterprise systems automatically and without user intervention. On SUSE Linux Enterprise you can create an AutoYaST profile that contains installation and configuration data. The profile tells AutoYaST what to install and how to configure the installed system to get a ready-to-use system in the end. This profile can then be used for mass deployment in different ways (for example, to clone existing cluster nodes).
For detailed instructions on how to use AutoYaST in various scenarios, see the SUSE Linux Enterprise 11 SP2 Deployment Guide, available at http://www.suse.com/documentation. Refer to chapter Automated Installation.
The following basic steps are needed for installation and initial cluster setup.
Install the software packages with YaST. Alternatively, you can install them from the command line with zypper:
zypper in -t pattern sles_ha
Initial Cluster Setup:
After installing the software on all nodes that will be part of your cluster, the following steps are needed to initially configure the cluster.
Optional: Defining Authentication Settings
Transferring the Configuration to All Nodes.
Whereas the configuration of Csync2 is done on one node only, the
services Csync2 and
xinetd need to be
started on all nodes.
Optional: Synchronizing Connection Status Between Cluster Nodes
Bringing the Cluster Online. The OpenAIS/Corosync service needs to be started on all nodes.
The cluster setup steps can either be executed automatically (with bootstrap scripts) or manually (with the YaST cluster module or from command line).
If you decide for an automatic cluster setup, refer to Section 3.4, “Automatic Cluster Setup (sleha-bootstrap)”.
For a manual setup (or for adjusting any options after the automatic setup), refer to Section 3.5, “Manual Cluster Setup (YaST)”.
You can also use a combination of both setup methods, for example: set up one node with YaST cluster and then use sleha-join to integrate more nodes.
Existing nodes can also be cloned for mass deployment with AutoYaST. The cloned nodes will have the same packages installed and the same system configuration. For details, refer to Section 3.6, “Mass Deployment with AutoYaST”.
The packages needed for configuring and managing a cluster with the
High Availability Extension are included in the High Availability installation pattern.
This pattern is only available after SUSE Linux Enterprise High Availability Extension has been installed as
add-on to SUSE Linux Enterprise Server. For information on how to install add-on products, see
the SUSE Linux Enterprise 11 SP2 Deployment Guide, available at
http://www.suse.com/documentation/sles11. Refer to
chapter Installing Add-On Products.
Procedure 3.1. Installing the High Availability Pattern¶
Start YaST as root user and select +.
Alternatively, start the YaST package manager as root on a
command line with yast2 sw_single.
From the list, select and activate the pattern in the pattern list.
Click to start installing the packages.
![]() | |
The software packages needed for High Availability clusters are not automatically copied to the cluster nodes. | |
Install the High Availability pattern on all machines that will be part of your cluster.
If you do not want to install SUSE® Linux Enterprise Server 11 SP2 and SUSE Linux Enterprise High Availability Extension 11 SP2 manually on all nodes that will be part of your cluster, use AutoYaST to clone existing nodes. For more information, refer to Section 3.6, “Mass Deployment with AutoYaST”.
The sleha-bootstrap package
provides everything you need to get a one-node cluster up and running and
to make other nodes join with the following basic steps:
With sleha-init, define the basic parameters needed for cluster communication and (optionally) set up a STONITH mechanism to protect your shared storage. This leaves you with a running one-node cluster.
With sleha-join, add more nodes to your cluster.
Both commands execute bootstrap scripts that require only a minimum of time and manual intervention. Any options set during the bootstrap process can be modified later with the YaST cluster module.
Before starting the automatic setup, make sure that the following prerequisites are fulfilled on all nodes that will participate in the cluster:
Prerequisites
The requirements listed in Section 2.2, “Software Requirements” and Section 2.4, “Other Requirements” are fulfilled.
The sleha-bootstrap package
is installed.
The network is configured according to your needs. For example, a private network is available for cluster communication and network device bonding is configured. For information on bonding, refer to Chapter 11, Network Device Bonding.
If you want to use SBD for your shared storage, you need one shared block device for SBD. The block device need not be formatted. For more information, refer to Chapter 17, Storage Protection.
All nodes must be able to see the shared storage via the same paths
(/dev/disk/by-path/... or
/dev/disk/by-id/...).
Procedure 3.2. Automatically Setting Up the First Node¶
The sleha-init command checks for configuration of NTP and guides you through configuration of the cluster communication layer (Corosync), and (optionally) through the configuration of SBD to protect your shared storage:
Log in as root to the physical or virtual machine you want to use
as cluster node.
Start the bootstrap script by executing
sleha-init
If NTP has not been configured to start at boot time, a message appears.
If you decide to continue anyway, the script will automatically generate keys for SSH access and for the Csync2 synchronization tool and start the services needed for both.
To configure the cluster communication layer (Corosync):
Enter a network address to bind to. By default, the script will
propose the network address of eth0.
Alternatively, enter a different network address, for example the
address of bond0.
Enter a multicast address. The script proposes a random address that you can use as default.
Enter a multicast port. The script proposes 5405
as default.
To configure SBD (optional), enter a persistent path to the partition of your block device that you want to use for SBD. The path must be consistent across all nodes in the cluster.
Finally, the script will start the OpenAIS service to bring the one-node cluster online and enable the Web management interface Hawk. The URL to use for Hawk is displayed on the screen.
For any details of the setup process, check
/var/log/sleha-bootstrap.log.
You now have a running one-node cluster. If you want to check the cluster status or start managing resources, proceed by logging in to one of the user interfaces, Hawk or the Pacemaker GUI. For more information, refer to Chapter 6, Configuring and Managing Cluster Resources (Web Interface) and Chapter 5, Configuring and Managing Cluster Resources (GUI).
![]() | Secure Password |
|---|---|
The bootstrap procedure creates a linux user named
passwd hacluster | |
Procedure 3.3. Adding Nodes to an Existing Cluster¶
If you have a cluster up and running (with one or more nodes), add more cluster nodes with the sleha-join bootstrap script. The script only needs access to an existing cluster node and will complete the basic setup on the current machine automatically.
If you have configured the existing cluster nodes with the YaST cluster module, make sure the following prerequisites are fulfilled before you run sleha-join:
The root user on the existing nodes has SSH keys in place for
passwordless login.
Csync2 is configured on the existing nodes. For details, refer to Procedure 3.8, “Configuring Csync2 with YaST”.
If you are logged in to the first node via Hawk, you can follow the changes in cluster status and view the resources being activated in the Web interface.
Log in as root to the physical or virtual machine supposed to
join the cluster.
Start the bootstrap script by executing:
sleha-join
If NTP has not been configured to start at boot time, a message appears.
If you decide to continue anyway, you will be prompted for the IP address of an existing node. Enter the IP address.
If you have not already configured a passwordless SSH access between
both machines, you will also be prompted for the root password of
the existing node.
After logging in to the specified node, the script will copy the Corosync configuration, configure SSH and Csync2, and will bring the current machine online as new cluster node. Apart from that, it will start the service needed for Hawk. If you have configured shared storage with OCFS2, it will also automatically create the mountpoint directory for the OCFS2 file system.
Repeat the steps above for all machines you want to add to the cluster.
For details of the process, check
/var/log/sleha-bootstrap.log.
![]() | Check no-quorum-policy |
|---|---|
After adding all nodes, check if you need to adjust the
| |
See Section 3.2, “Overview” for an overview of all steps for initial setup.
The following sections guide you through each of the setup steps, using
the YaST cluster module. To access it, start YaST as root and
select +. Alternatively, start the module from command line with
yast2 cluster.
If you start the cluster module for the first time, it appears as wizard, guiding you through all the steps necessary for basic setup. Otherwise, click the categories on the left panel to access the configuration options for each step.
The YaST cluster module automatically opens the ports in the firewall
that are needed for cluster communication on the current machine. The
configuration is written to
/etc/sysconfig/SuSEfirewall2.d/services/cluster.
Note that some options in the YaST cluster module apply only to the current node, whereas others may automatically be transferred to all nodes. Find detailed information about this in the following sections.
For successful communication between the cluster nodes, define at least one communication channel.
![]() | Redundant Communication Paths |
|---|---|
However, it is highly recommended to set up cluster communication via two or more redundant paths. This can be done via:
If possible, choose network device bonding. | |
Procedure 3.4. Defining the First Communication Channel¶
For communication between the cluster nodes, use either multicast (UDP) or unicast (UDPU).
In the YaST cluster module, switch to the category.
To use multicast:
Set the protocol to
UDP.
Define the . Set the value to the subnet you will use for cluster multicast.
Define the .
Define the .
Wit the values entered above, you have now defined
one communication channel for the cluster. In
multicast mode, the same bindnetaddr,
mcastaddr, and
mcastport will be used for all cluster
nodes. All nodes in the cluster will know each other by using the
same multicast address. For different clusters, use different
multicast addresses.
To use unicast:
Set the protocol to
UDPU.
Define the . Set the value to the subnet you will use for cluster unicast.
Define the .
For unicast communication, Corosync needs to know the IP addresses of all nodes in the cluster. For each node that will be part of the cluster, click and enter its IP address. To modify or remove any addresses of cluster members, use the or buttons.
Activate to automatically generate a unique ID for every cluster node.
If you modified any options for an existing cluster, confirm your
changes and close the cluster module. YaST writes the configuration
to /etc/corosync/corosync.conf.
If needed, define a second communication channel as described below. Or click and proceed with Procedure 3.6, “Enabling Secure Authentication”.
Procedure 3.5. Defining a Redundant Communication Channel¶
If network device bonding cannot be used for any reason, the second best choice is to define a redundant communication channel (a second ring) in Corosync. That way, two physically separate networks can be used for communication. In case one network fails, the cluster nodes can still communicate via the other network.
![]() | Redundant Rings and /etc/hosts |
|---|---|
If multiple rings are configured, each node can have multiple IP
addresses. This needs to be reflected in the
| |
In the YaST cluster module, switch to the category.
Activate . The redundant channel must use the same protocol as the first communication channel you defined.
If you use multicast, define the , the and the for the redundant channel.
If you use unicast, define the , the and enter the IP addresses of all nodes that will be part of the cluster.
Now you have defined an additional communication channel in Corosync
that will form a second token-passing ring. In
/etc/corosync/corosync.conf, the primary ring
(the first channel you have configured) gets the ringnumber
0, the second ring (redundant channel) the
ringnumber 1.
To tell Corosync how and when to use the different channels, select
the you want to use
(active or passive). For more
information about the modes, refer to
Redundant Ring Protocol (RRP) or click .
As soon as RRP is used, the Stream Control Transmission Protocol
(SCTP) is used for communication between the nodes (instead of TCP).
The High Availability Extension monitors the status of the current rings and automatically
re-enables redundant rings after faults. Alternatively, you can also
check the ring status manually with
corosync-cfgtool. View the available options with
-h.
If only one communication channel is defined,
is automatically disabled (value
none).
If you modified any options for an existing cluster, confirm your
changes and close the cluster module. YaST writes the configuration
to /etc/corosync/corosync.conf.
For further cluster configuration, click and proceed with Section 3.5.3, “Defining Authentication Settings”.
Find an example file for a UDP setup in
/etc/corosync/corosync.conf.example. An example for
UDPU setup is available in
/etc/corosync/corosync.conf.example.udpu.
The next step is to define the authentication settings for the cluster. You can use HMAC/SHA1 authentication that requires a shared secret used to protect and authenticate messages. The authentication key (password) you specify will be used on all nodes in the cluster.
Procedure 3.6. Enabling Secure Authentication¶
In the YaST cluster module, switch to the category.
Activate .
For a newly created cluster, click . An authentication key is created and written to
/etc/corosync/authkey.
If you want the current machine to join an existing cluster, do not
generate a new key file. Instead, copy the
/etc/corosync/authkey from one of the nodes to
the current machine (either manually or with Csync2).
If you modified any options for an existing cluster, confirm your
changes and close the cluster module. YaST writes the configuration
to /etc/corosync/corosync.conf.
For further cluster configuration, proceed with Section 3.5.4, “Configuring Services”.
In the YaST cluster module define whether to start certain services on a node at boot time. You can also use the module to start and stop the services manually. To bring the cluster nodes online and start the cluster resource manager, OpenAIS must be running as a service.
Procedure 3.7. Enabling OpenAIS and mgmtd¶
In the YaST cluster module, switch to the category.
To start OpenAIS each time this cluster node is booted, select the respective option in the group. If you select in the group, you must start OpenAIS manually each time this node is booted. To start OpenAIS manually, use the rcopenais start command.
If you want to use the Pacemaker GUI for configuring, managing and monitoring cluster resources, activate . This daemon is needed for the GUI.
To start or stop OpenAIS immediately, click the respective button.
If you modified any options for an existing cluster node, confirm your changes and close the cluster module. Note that the configuration only applies to the current machine, not to all cluster nodes.
For further cluster configuration, click and proceed with Section 3.5.5, “Transferring the Configuration to All Nodes”.
Instead of copying the resulting configuration files to all nodes manually, use the csync2 tool for replication across all nodes in the cluster.
This requires the following basic steps:
Procedure 3.8. Configuring Csync2 with YaST¶
In the YaST cluster module, switch to the category.
To specify the synchronization group, click in the group and enter the local hostnames of all nodes in your cluster. For each node, you must use exactly the strings that are returned by the hostname command.
Click to create a key file
for the synchronization group. The key file is written to
/etc/csync2/key_hagroup. After it has been
created, it must be copied manually to all members of the cluster.
To populate the list with the files that usually need to be synchronized among all nodes, click .
If you want to , or files from the list of files to be synchronized use the respective buttons. You must enter the absolute pathname for each file.
Activate Csync2 by clicking . This will execute chkconfig csync2 to start Csync2 automatically at boot time.
If you modified any options for an existing cluster, confirm your
changes and close the cluster module. YaST then writes the Csync2
configuration to /etc/csync2/csync2.cfg. To start
the synchronization process now, proceed with
Procedure 3.9, “Synchronizing the Configuration Files with Csync2”.
For further cluster configuration, click and proceed with Section 3.5.6, “Synchronizing Connection Status Between Cluster Nodes”.
Procedure 3.9. Synchronizing the Configuration Files with Csync2¶
To successfully synchronize the files with Csync2, make sure that the following prerequisites are met:
The same Csync2 configuration is available on all nodes. Copy the
file /etc/csync2/csync2.cfg manually to all
nodes after you have configured it as described in
Procedure 3.8, “Configuring Csync2 with YaST”. It is
recommended to include this file in the list of files to be
synchronized with Csync2.
Copy the /etc/csync2/key_hagroup file you have
generated on one node in Step 3
to all nodes in the cluster as it is needed for
authentication by Csync2. However, do not regenerate the file on the
other nodes as it needs to be the same file on all nodes.
Both Csync2 and xinetd must
be running on all nodes.
![]() | Starting Services at Boot Time |
|---|---|
Execute the following commands on all nodes to make both services
start automatically at boot time and to start
chkconfig csync2 on chkconfig xinetd on rcxinetd start | |
On the node that you want to copy the configuration from, execute the following command:
csync2 -xvThis will synchronize all the files once by pushing them to the other nodes. If all files are synchronized successfully, Csync2 will finish with no errors.
If one or several files that are to be synchronized have been modified on other nodes (not only on the current one), Csync2 will report a conflict. You will get an output similar to the one below:
While syncing file /etc/corosync/corosync.conf: ERROR from peer hex-14: File is also marked dirty here! Finished with 1 errors.
If you are sure that the file version on the current node is the “best” one, you can resolve the conflict by forcing this file and resynchronizing:
csync2 -f /etc/corosync/corosync.conf csync2 -x
For more information on the Csync2 options, run
csync2 -help.
![]() | Pushing Synchronization After Any Changes |
|---|---|
Csync2 only pushes changes. It does not continuously synchronize files between the nodes.
Each time you update files that need to be synchronized, you have to
push the changes to the other nodes: Run
csync2 | |
To enable stateful packet inspection for iptables, configure and use the conntrack tools with the following basic steps:
Configuring a resource for
conntrackd (class:
ocf, provider: heartbeat). If
you use Hawk to add the resource, use the default values proposed by
Hawk.
After configuring the conntrack tools, you can use them for load balancing with Linux Virtual Server.
Procedure 3.10. Configuring the conntrackd with YaST¶
Use the YaST cluster module to configure the user-space
conntrackd. It needs a
dedicated network interface that is not used for other communication
channels. The daemon can be started via a resource agent afterward.
In the YaST cluster module, switch to the category.
Select a for synchronizing the connection status. The IPv4 address of the selected interface is automatically detected and shown in YaST. It must already be configured and it must support multicast.
Define the to be used for synchronizing the connection status.
In , define a numeric ID for the group to synchronize the connection status to.
Click to
create the configuration file for
conntrackd.
Confirm your changes and close the cluster module. If you have done the initial cluster setup exclusively with the YaST cluster module, you have now completed the basic configuration steps. Proceed with Section 3.5.7, “Bringing the Cluster Online”.
After the initial cluster configuration is done, start the OpenAIS/Corosync service on each cluster node to bring the stack online:
Procedure 3.11. Starting OpenAIS/Corosync and Checking the Status
Log in to an existing node.
Check if the service is already running:
rcopenais status
If not, start OpenAIS/Corosync now:
rcopenais start
Repeat the steps above for each of the cluster nodes.
On one of the nodes, check the cluster status with the following command:
crm_mon
If all nodes are online, the output should be similar to the following:
============ Last updated: Thu Nov 10 13:37:48 2011 Last change: Wed Nov 9 17:40:06 2011 by hacluster via cibadmin on auckland Stack: openais Current DC: auckland - partition with quorum Version: 1.1.6-2d8fad5daab7e592069215f144cc765def450309 2 Nodes configured, 2 expected votes 8 Resources configured. ============ Online: [ e231 e229 ]
This output indicates that the cluster resource manager is started and is ready to manage resources.
After the basic configuration is done and the nodes are online, you can start to configure cluster resources, using one of the cluster management tools like the crm shell, the Pacemaker GUI, or the HA Web Konsole. For more information, refer to the following chapters.
The following procedure is suitable for deploying cluster nodes which are clones of an already existing node. The cloned nodes will have the same packages installed and the same system configuration.
Procedure 3.12. Cloning a Cluster Node with AutoYaST¶
![]() | Identical Hardware |
|---|---|
This scenario assumes you are rolling out SUSE Linux Enterprise High Availability Extension 11 SP2 to a set of machines with exactly the same hardware configuration. | |
If you need to deploy cluster nodes on non-identical hardware, refer to the Rule-Based Autoinstallation section in the SUSE Linux Enterprise 11 SP2 Deployment Guide, available at http://www.suse.com/documentation.
Make sure the node you want to clone is correctly installed and configured. For details, refer to Section 3.3, “Installation as Add-on”, and Section 3.4, “Automatic Cluster Setup (sleha-bootstrap)” or Section 3.5, “Manual Cluster Setup (YaST)”, respectively.
Follow the description outlined in the SUSE Linux Enterprise 11 SP2 Deployment Guide for simple mass installation. This includes the following basic steps:
Creating an AutoYaST profile. Use the AutoYaST GUI to create and modify a profile based on the existing system configuration. In AutoYaST, choose the module and click the button. If needed, adjust the configuration in the other modules and save the resulting control file as XML.
Determining the source of the AutoYaST profile and the parameter to pass to the installation routines for the other nodes.
Determining the source of the SUSE Linux Enterprise Server and SUSE Linux Enterprise High Availability Extension installation data.
Determining and setting up the boot scenario for autoinstallation.
Passing the command line to the installation routines, either by
adding the parameters manually or by creating an
info file.
Starting and monitoring the autoinstallation process.
After the clone has been successfully installed, execute the following steps to make the cloned node join the cluster:
Procedure 3.13. Bringing the Cloned Node Online¶
Transfer the key configuration files from the already configured nodes to the cloned node with Csync2 as described in Section 3.5.5, “Transferring the Configuration to All Nodes”.
To bring the node online, start the OpenAIS service on the cloned node as described in Section 3.5.7, “Bringing the Cluster Online”.
The cloned node will now join the cluster because the
/etc/corosync/corosync.conf file has been applied to
the cloned node via Csync2. The CIB is automatically synchronized among
the cluster nodes.