Chapter 6. Configuring and Managing Cluster Resources (Web Interface)

Chapter 6. Configuring and Managing Cluster Resources (Web Interface)¶

Contents

6.1. Hawk—Overview
6.2. Configuring Global Cluster Options
6.3. Configuring Cluster Resources
6.4. Managing Cluster Resources
6.5. Troubleshooting

Abstract

In addition to the crm command line tool and the Pacemaker GUI, the High Availability Extension also comes with the HA Web Konsole (Hawk), a Web-based user interface for management tasks. It allows you to monitor and administer your Linux cluster from non-Linux machines as well. Furthermore, it is the ideal solution in case your system only provides a minimal graphical user interface.

This chapter introduces Hawk and covers basic tasks for configuring and managing cluster resources: modifying global cluster options, creating basic and advanced types of resources (groups and clones), configuring constraints, specifying failover nodes and failback nodes, configuring resource monitoring, starting, cleaning up or removing resources, and migrating resources manually. For detailed analysis of the cluster status, Hawk generates a cluster report (hb_report). You can view the cluster history or explore potential failure scenarios with the simulator.

The Web interface is included in the hawk package. It must be installed on all cluster nodes you want to connect to with Hawk. On the machine from which you want to access a cluster node using Hawk, you only need a (graphical) Web browser with JavaScript and cookies enabled to establish the connection.

User Authentication

	User Authentication
To log in to the cluster from Hawk, the respective user must be a member of the `haclient` group. The installation creates a Linux user named `hacluster` and adds the user to the `haclient` group. Before using Hawk, either set a password for the `hacluster` user or create a new user which is a member of the `haclient` group. Do this on every node you will connect to with Hawk.

To log in to the cluster from Hawk, the respective user must be a member of the haclient group. The installation creates a Linux user named hacluster and adds the user to the haclient group.

Before using Hawk, either set a password for the hacluster user or create a new user which is a member of the haclient group.

Do this on every node you will connect to with Hawk.

6.1. Hawk—Overview¶

Learn how to start Hawk, how to log in to the cluster and get to know the Hawk basics.

6.1.1. Starting Hawk and Logging in¶

Procedure 6.1. Starting Hawk¶

To use Hawk, the respective Web service must be started on the node that you want to connect to via the Web interface. For communication, the standard HTTPS protocol and port 7630 is used.

On the node you want to connect to, open a shell and log in as root.
Check the status of the service by entering
```
rchawk status
```
If the service is not running, start it with
```
rchawk start
```
If you want Hawk to start automatically at boot time, execute the following command:
```
chkconfig hawk on
```

On any machine, start a Web browser and make sure that JavaScript and cookies are enabled.

Point it at the IP address or hostname of any cluster node running the Hawk Web service, or the address of any IPaddr(2) resource that you may have configured:

https://IPaddress:7630/

	Certificate Warning
Depending on your browser and browser options, you may get a certificate warning when trying to access the URL for the first time. This is because Hawk uses a self-signed certificate that is not considered trustworthy per default. To proceed anyway, you can add an exception in the browser to bypass the warning. To avoid the warning in the first place, the self-signed certificate can also be replaced with a certificate signed by an official Certificate Authority. For information on how to do so, refer to Replacing the Self-Signed Certificate.

Certificate Warning

Depending on your browser and browser options, you may get a certificate warning when trying to access the URL for the first time. This is because Hawk uses a self-signed certificate that is not considered trustworthy per default.

To proceed anyway, you can add an exception in the browser to bypass the warning. To avoid the warning in the first place, the self-signed certificate can also be replaced with a certificate signed by an official Certificate Authority. For information on how to do so, refer to Replacing the Self-Signed Certificate.

On the Hawk login screen, enter the Username and Password of the hacluster user (or of any other user that is a member of the haclient group) and click Log In.
The Cluster Status screen appears, displaying the status of your cluster nodes and resources similar to the output of the crm_mon.

6.1.2. Main Screen: Cluster Status¶

After logging in, Hawk displays the Cluster Status screen. It shows a summary with the most important global cluster parameters and the status of your cluster nodes and resources. The following color code is used for status display of nodes and resources:

Hawk Color Code¶

Green: OK. For example, the resource is running or the node is online.
Red: Bad, unclean. For example, the resource has failed or the node was not shut down cleanly.
Yellow: In transition. For example, the node is currently being shut down.
Gray: Not running, but the cluster expects it to be running. For example, nodes that the administrator has stopped or put into standby mode. Also nodes that are offline are displayed in gray (if they have been shut down cleanly).

If a resource has failed, a failure message with the details is shown in red at the top of the screen.

Figure 6.1. Hawk—Cluster Status (Summary View)

The Cluster Status screen refreshes itself in near real-time. Choose between the following views:

Summary View: Shows the most important global cluster parameters and the status of your cluster nodes and resources at the same time. To see more details for a certain node or resource, click the respective entries that are marked as links.
Tree View: Presents an expandable view of the most important global cluster parameters and the status of your cluster nodes and resources. Click the arrows to expand or collapse the elements belonging to the respective category. In contrast to the Summary View this view not only shows the IDs and status of resources but also the type (for example, primitive, clone, or group).
Table View: This view is especially useful for larger clusters, because it shows in a concise way which resources are currently running on which node. Inactive nodes or resources are also displayed.

To perform basic operator tasks on nodes and resources (like starting or stopping resources, bringing nodes online, or viewing details), click the wrench icon next to the respective node or resource to access a context menu.

For more complex tasks like configuring resources, constraints, or global cluster options, use the navigation bar at the left hand side. From there, you can also generate a cluster report (hb_report), view the cluster history, or explore potential failure scenarios with the simulator.

Available Functions in Hawk

	Available Functions in Hawk
By default, users logged in as `root` or `hacluster` have full read-write access to all cluster configuration tasks. However, Access Control Lists can be used to define fine-grained access permissions. If ACLs are enabled in the CRM, the available functions in Hawk depend on the user role and access permissions assigned to you. In addition, the following functions in Hawk can only be executed by the user `hacluster`: Generating a `hb_report`. Using the history explorer. Viewing recent events for nodes or resources.

By default, users logged in as root or hacluster have full read-write access to all cluster configuration tasks. However, Access Control Lists can be used to define fine-grained access permissions.

If ACLs are enabled in the CRM, the available functions in Hawk depend on the user role and access permissions assigned to you. In addition, the following functions in Hawk can only be executed by the user hacluster:

Generating a hb_report.
Using the history explorer.
Viewing recent events for nodes or resources.

6.2. Configuring Global Cluster Options¶

Global cluster options control how the cluster behaves when confronted with certain situations. They are grouped into sets and can be viewed and modified with cluster management tools like Pacemaker GUI, Hawk, and crm shell. The predefined values can be kept in most cases. However, to make key functions of your cluster work correctly, you need to adjust the following parameters after basic cluster setup:

Procedure 6.2. Modifying Global Cluster Options¶

Start a Web browser and log in to the cluster as described in Section 6.1.1, “Starting Hawk and Logging in”.
In the left navigation bar, select Cluster Properties to view the global cluster options and their current values.
Depending on your cluster requirements, set no-quorum-policy to the appropriate value.
If you need to disable fencing for any reasons, deselect stonith-enabled.
No Support Without STONITH
A cluster without STONITH enabled is not supported.
To remove a cluster property, click the minus icon next to the property.
To add a new property, choose one from the drop-down list and click the plus icon.
Confirm your changes.

Figure 6.2. Hawk—Global Cluster Properties

6.3. Configuring Cluster Resources¶

As a cluster administrator, you need to create cluster resources for every resource or application you run on servers in your cluster. Cluster resources can include Web sites, mail servers, databases, file systems, virtual machines, and any other server-based applications or services you want to make available to users at all times.

For an overview of resource types you can create, refer to Section 4.2.3, “Types of Resources”. Apart from the basic specification of a resource (ID, class, provider, and type), you can add or modify the following parameters during or after creation of a resource:

Instance attributes (parameters) determine which instance of a service the resource controls. For more information, refer to Section 4.2.7, “Instance Attributes (Parameters)”.
When creating a resource, Hawk automatically shows any required parameters. Edit them to get a valid resource configuration.
Meta attributes tell the CRM how to treat a specific resource. For more information, refer to Section 4.2.6, “Resource Options (Meta Attributes)”.
When creating a resource, Hawk automatically lists the important meta attributes for that resource (for example, the target-role attribute that defines the initial state of a resource. By default, it is set to Stopped, so the resource will not start immediately).
Operations are needed for resource monitoring. For more information, refer to Section 4.2.8, “Resource Operations”.
When creating a resource, Hawk displays the most important resource operations (monitor, start, and stop).

6.3.1. Configuring Resources with the Setup Wizard¶

The High Availability Extension comes with a predefined set of resources for some frequently used cluster scenarios, for example, setting up a highly available Web server. Find the predefined sets in the hawk-templates package. You can also define your own wizard templates. For detailed information, refer to http://hg.clusterlabs.org/pacemaker/hawk/file/tip/doc/wizard.txt. Hawk provides a wizard that guides you through all configuration steps.

Procedure 6.3. Using the Setup Wizard¶

Start a Web browser and log in to the cluster as described in Section 6.1.1, “Starting Hawk and Logging in”.
In the left navigation bar, select Setup Wizard. The Cluster Setup Wizard shows a list of available resource sets. If you click an entry, Hawk displays a short help text about the resource set.
Select the resource set you want to configure and click Next.
Follow the instructions on the screen. If you need information about an option, click it to display a short help text in Hawk.

Figure 6.3. Hawk—Setup Wizard

6.3.2. Creating Simple Cluster Resources¶

To create the most basic type of a resource, proceed as follows:

Procedure 6.4. Adding Primitive Resources¶

Start a Web browser and log in to the cluster as described in Section 6.1.1, “Starting Hawk and Logging in”.
In the left navigation bar, select Resources. The Resources screen shows categories for all types of resources. It lists any resources that are already defined.
Select the Primitive category and click the plus icon.
Specify the resource:
1. Enter a unique Resource ID.
2. From the Class list, select the resource agent class you want to use for the resource: heartbeat, lsb, ocf or stonith. For more information, see Section 4.2.2, “Supported Resource Agent Classes”.
3. If you selected ocf as class, specify the Provider of your OCF resource agent. The OCF specification allows multiple vendors to supply the same resource agent.
4. From the Type list, select the resource agent you want to use (for example, IPaddr or Filesystem). A short description for this resource agent is displayed.
  The selection you get in the Type list depends on the Class (and for OCF resources also on the Provider) you have chosen.
Hawk automatically shows any required parameters for the resource plus an empty drop-down list that you can use to specify an additional parameter.
To define Parameters (instance attributes) for the resource:
1. Enter values for each required parameter. A short help text is displayed as soon as you click the input field next to a parameter.
2. To completely remove a parameter, click the minus icon next to the parameter.
3. To add another parameter, click the blank drop-down list, select a parameter and enter a value for it.
Hawk automatically shows the most important resource Operations and proposes default values. If you do not modify any settings here, Hawk will add the proposed operations and their default values as soon as you confirm your changes.
For details on how to modify, add or remove operations, refer to Procedure 6.13, “Adding or Modifying Monitor Operations”.
Hawk automatically lists the most important meta attributes for the resource, for example target-role.
To modify or add Meta Attributes:
1. To set a (different) value for an attribute, select one from the drop-down list next to the attribute.
2. To completely remove a meta attribute, click the minus icon next to it.
3. To add another meta attribute, click the empty drop-down list and select an attribute. The default value for the attribute is displayed. If needed, change it as described above.
Click Create Resource to finish the configuration. A message at the top of the screen shows if the resource was successfully created or not.

Figure 6.4. Hawk—Primitive Resource

6.3.3. Creating STONITH Resources¶

	No Support Without STONITH
A cluster without STONITH is not supported.

By default, the global cluster option stonith-enabled is set to true: If no STONITH resources have been defined, the cluster will refuse to start any resources. Configure one or more STONITH resources to complete the STONITH setup. While they are configured similar to other resources, the behavior of STONITH resources is different in some respects. For details refer to Section 9.3, “STONITH Configuration”.

Procedure 6.5. Adding a STONITH Resource¶

Start a Web browser and log in to the cluster as described in Section 6.1.1, “Starting Hawk and Logging in”.
In the left navigation bar, select Resources. The Resources screen shows categories for all types of resources and lists all defined resources.
Select the Primitive category and click the plus icon.
Specify the resource:
1. Enter a unique Resource ID.
2. From the Class list, select the resource agent class stonith.
3. From the Type list, select the STONITH plug-in for controlling your STONITH device. A short description for this plug-in is displayed.
Hawk automatically shows the required Parameters for the resource. Enter values for each parameter.
Hawk displays the most important resource Operations and proposes default values. If you do not modify any settings here, Hawk will add the proposed operations and their default values as soon as you confirm.
Adopt the default Meta Attributes settings if there is no reason to change them.
Confirm your changes to create the STONITH resource.

To complete your fencing configuration, add constraints, use clones or both. For more details, refer to Chapter 9, Fencing and STONITH.

6.3.4. Using Resource Templates¶

If you want to create lots of resources with similar configurations, defining a resource template is the easiest way. Once defined, it can be referenced in primitives or in certain types of constraints. For detailed information about function and use of resource templates, refer to Section 4.4.3, “Resource Templates and Constraints”.

Start a Web browser and log in to the cluster as described in Section 6.1.1, “Starting Hawk and Logging in”.
In the left navigation bar, select Resources. The Resources screen shows categories for all types of resources plus a Template category.
Select the Template category and click the plus icon.
Enter a Template ID.
Specify the resource template as you would specify a primitive. Follow , starting with Step 4.b.
Click Create Resource to finish the configuration. A message at the top of the screen shows if the resource template was successfully created.

Figure 6.5. Hawk—Resource Template

Procedure 6.6. Referencing Resource Templates¶

Start a Web browser and log in to the cluster as described in Section 6.1.1, “Starting Hawk and Logging in”.
To reference the newly created resource template in a primitive, follow these steps:
1. In the left navigation bar, select Resources. The Resources screen shows categories for all types of resources. It lists all defined resources.
2. Select the Primitive category and click the plus icon.
3. Enter a unique Resource ID.
4. Activate Use Template and, from the drop-down list, select the template to reference.
5. If needed, specify further Parameters, Operations, or Meta Attributes as described in Procedure 6.4, “Adding Primitive Resources”.
To reference the newly created resource template in colocational or order constraints, proceed as described in Procedure 6.8, “Adding or Modifying Colocational or Order Constraints”.

6.3.5. Configuring Resource Constraints¶

After you have configured all resources, specify how the cluster should handle them correctly. Resource constraints let you specify on which cluster nodes resources can run, in which order resources will be loaded, and what other resources a specific resource depends on.

For an overview of available types of constraints, refer to Section 4.4.1, “Types of Constraints”. When defining constraints, you also need to specify scores. For more information on scores and their implications in the cluster, see Section 4.4.2, “Scores and Infinity”.

Learn how to create the different types of constraints in the following procedures.

Procedure 6.7. Adding or Modifying Location Constraints¶

For location constraints, specify a constraint ID, resource, score and node:

Start a Web browser and log in to the cluster as described in Section 6.1.1, “Starting Hawk and Logging in”.
In the left navigation bar, select Constraints. The Constraints screen shows categories for all types of constraints. It lists all defined constraints.
To add a new Location constraint, click the plus icon in the respective category.
To modify an existing constraint, click the wrench icon next to the constraint and select Edit Constraint.
Enter a unique Constraint ID. When modifying existing constraints, the ID is already defined.
Select the Resource for which to define the constraint. The list shows the IDs of all resources that have been configured for the cluster.
Set the Score for the constraint. Positive values indicate the resource can run on the Node you specify in the next step. Negative values mean it should not run on that node. Setting the score to INFINITY forces the resource to run on the node. Setting it to -INFINITY means the resources must not run on the node.
Select the Node for the constraint.
Click Create Constraint to finish the configuration. A message at the top of the screen shows if the constraint was successfully created.

Figure 6.6. Hawk—Location Constraint

Procedure 6.8. Adding or Modifying Colocational or Order Constraints¶

For both types of constraints specify a constraint ID and a score, then add resources to a dependency chain:

Start a Web browser and log in to the cluster as described in Section 6.1.1, “Starting Hawk and Logging in”.
In the left navigation bar, select Constraints. The Constraints screen shows categories for all types of constraints and lists all defined constraints.
To add a new Colocation or Order constraint, click the plus icon in the respective category.
To modify an existing constraint, click the wrench icon next to the constraint and select Edit Constraint.
Enter a unique Constraint ID. When modifying existing constraints, the ID is already defined.
Define a Score.
For colocation constraints, the score determines the location relationship between the resources. Positive values indicate the resources should run on the same node. Negative values indicate the resources should not run on the same node. Setting the score to INFINITY forces the resources to run on the same node. Setting it to -INFINITY means the resources must not run on the same node. The score will be combined with other factors to decide where to put the resource.
For order constraints, the constraint is mandatory if the score is greater than zero, otherwise it is only a suggestion. The default value is INFINITY.
For order constraints, you can usually keep the option Symmetrical enabled. This specifies that resources are stopped in reverse order.
To define the resources for the constraint, follow these steps:
1. Select a resource from the list Add resource to constraint. The list shows the IDs of all resources and all resource templates configured for the cluster.
2. To add the selected resource, click the plus icon next to the list. A new list appears beneath. Select the next resource from the list. As both colocation and order constraints define a dependency between resources, you need at least two resources.
3. Select one of the remaining resources from the list Add resource to constraint. Click the plus icon to add the resource.
  Now you have two resources in a dependency chain.
  If you have defined an order constraint, the topmost resource will start first, then the second etc. Usually the resources will be stopped in reverse order.
  However, if you have defined a colocation constraint, the arrow icons between the resources reflect their dependency, but not their start order. As the topmost resource depends on the next resource and so on, the cluster will first decide where to put the last resource, then place the depending ones based on that decision. If the constraint cannot be satisﬁed, the cluster may decide not to allow the resource to run at all.
4. Add as many resources as needed for your colocation or order constraint.
5. If you want to swap the order of two resources, click the double arrow at the right hand side of the resources to swap the resources in the dependency chain.
If needed, specify further parameters for each resource, like the role (Master, Slave, Started, or Stopped).
Click Create Constraint to finish the configuration. A message at the top of the screen shows if the constraint was successfully created.

Figure 6.7. Hawk—Colocation Constraint

As an alternative format for defining colocation or ordering constraints, you can use resource sets. They have the same ordering semantics as groups.

Procedure 6.9. Using Resource Sets for Colocation or Order Constraints¶

Start a Web browser and log in to the cluster as described in Section 6.1.1, “Starting Hawk and Logging in”.
Define colocation or order constraints as described in Procedure 6.8, “Adding or Modifying Colocational or Order Constraints”.
When you have added the resources to the dependency chain, you can put them into a resource set by clicking the chain icon at the right hand side. A resource set is visualized by a frame around the resources belonging to a set.
You can also add multiple resources to a resource set or create multiple resource sets.
To extract a resource from a resource set, click the scissors icon above the respective resource. The resource will be removed from the set and put back into the dependency chain at its original place.
Confirm your changes to finish the constraint configuration.

For more information on configuring constraints and detailed background information about the basic concepts of ordering and colocation, refer to the documentation available at http://www.clusterlabs.org/doc/ and http://www.clusterlabs.org/wiki/Documentation , respectively:

Pacemaker Explained , chapter Resource Constraints
Collocation Explained
Ordering Explained

Procedure 6.10. Removing Constraints¶

Start a Web browser and log in to the cluster as described in Section 6.1.1, “Starting Hawk and Logging in”.
In the left navigation bar, select Constraints. The Constraints screen shows categories for all types of constraints and lists all defined constraints.
Click the wrench icon next to a constraint and select Remove Constraint.

6.3.6. Specifying Resource Failover Nodes¶

A resource will be automatically restarted if it fails. If that cannot be achieved on the current node, or it fails N times on the current node, it will try to fail over to another node. You can define a number of failures for resources (a migration-threshold), after which they will migrate to a new node. If you have more than two nodes in your cluster, the node to which a particular resource fails over is chosen by the High Availability software.

You can specify a specific node to which a resource will fail over by proceeding as follows:

Start a Web browser and log in to the cluster as described in Section 6.1.1, “Starting Hawk and Logging in”.
Configure a location constraint for the resource as described in Procedure 6.7, “Adding or Modifying Location Constraints”.
Add the migration-threshold meta attribute to the resource as described in , Step 7 and enter a Value for the migration-threshold. The value should be positive and less than INFINITY.
If you want to automatically expire the failcount for a resource, add the failure-timeout meta attribute to the resource as described in , Step 7 and enter a Value for the failure-timeout.
If you want to specify additional failover nodes with preferences for a resource, create additional location constraints.

The process flow regarding migration thresholds and failcounts is demonstrated in Example 4.2, “Migration Threshold—Process Flow”.

Instead of letting the failcount for a resource expire automatically, you can also clean up failcounts for a resource manually at any time. Refer to Section 6.4.2, “Cleaning Up Resources” for details.

6.3.7. Specifying Resource Failback Nodes (Resource Stickiness)¶

A resource may fail back to its original node when that node is back online and in the cluster. To prevent this or to specify a different node for the resource to fail back to, change the stickiness value of the resource. You can either specify the resource stickiness when creating it or afterwards.

For the implications of different resource stickiness values, refer to Section 4.4.5, “Failback Nodes”.

Procedure 6.11. Specifying Resource Stickiness¶

Start a Web browser and log in to the cluster as described in Section 6.1.1, “Starting Hawk and Logging in”.
Add the resource-stickiness meta attribute to the resource as described in , Step 7.
Specify a value between -INFINITY and INFINITY for the resource-stickiness.

6.3.8. Configuring Placement of Resources Based on Load Impact¶

Not all resources are equal. Some, such as Xen guests, require that the node hosting them meets their capacity requirements. If resources are placed so that their combined needs exceed the provided capacity, the performance of the resources diminishes or they fail.

To take this into account, the High Availability Extension allows you to specify the following parameters:

The capacity a certain node provides.
The capacity a certain resource requires.
An overall strategy for placement of resources.

Utilization attributes are used to configure both the resource's requirements and the capacity a node provides. The High Availability Extension now also provides means to detect and configure both node capacity and resource requirements automatically. For more details and a configuration example, refer to Section 4.4.6, “Placing Resources Based on Their Load Impact”.

To display a node's capacity values (defined via utilization attributes) as well as the capacity currently consumed by resources running on the node, switch to the Cluster Status screen in Hawk and select the node you are interested in. Click the wrench icon next to the node and select Show Details.

Figure 6.8. Hawk—Viewing a Node's Capacity Values

After you have configured the capacities your nodes provide and the capacities your resources require, you need to set the placement strategy in the global cluster options, otherwise the capacity configurations have no effect. Several strategies are available to schedule the load: for example, you can concentrate it on as few nodes as possible, or balance it evenly over all available nodes. For more information, refer to Section 4.4.6, “Placing Resources Based on Their Load Impact”.

Procedure 6.12. Setting the Placement Strategy¶

Start a Web browser and log in to the cluster as described in Section 6.1.1, “Starting Hawk and Logging in”.
In the left navigation bar, select Cluster Properties to view the global cluster options and their current values.
From the Add new property drop-down list, choose placement-strategy.
Depending on your requirements, set Placement Strategy to the appropriate value.
Click the plus icon to add the new cluster property including its value.
Confirm your changes.

6.3.9. Configuring Resource Monitoring¶

The High Availability Extension can not only detect a node failure, but also when an individual resource on a node has failed. If you want to ensure that a resource is running, configure resource monitoring for it. For resource monitoring, specify a timeout and/or start delay value, and an interval. The interval tells the CRM how often it should check the resource status. You can also set particular parameters, such as Timeout for start or stop operations.

Procedure 6.13. Adding or Modifying Monitor Operations¶

Start a Web browser and log in to the cluster as described in Section 6.1.1, “Starting Hawk and Logging in”.
In the left navigation bar, select Resources. The Resources screen shows categories for all types of resources and lists all defined resources.
Select the resource to modify, click the wrench icon next to it and select Edit Resource. The resource definition is displayed. Hawk automatically shows the most important resource operations (monitor, start, stop) and proposes default values.
To change the values for an operation:
1. Click the pen icon next to the operation.
2. In the dialog that opens, specify the following values:
  - Enter a timeout value in seconds. After the specified timeout period, the operation will be treated as failed. The PE will decide what to do or execute what you specified in the On Fail field of the monitor operation.
  - For monitoring operations, define the monitoring interval in seconds.
  If needed, use the empty drop-down list at the bottom of the monitor dialog to add more parameters, like On Fail (what to do if this action fails?) or Requires (what conditions need to be fulfilled before this action occurs?).
3. Confirm your changes to close the dialog and to return to the Edit Resource screen.
To completely remove an operation, click the minus icon next to it.
To add another operation, click the empty drop-down list and select an operation. A default value for the operation is displayed. If needed, change it as described above.
Click Apply Changes to finish the configuration. A message at the top of the screen shows if the resource was successfully updated or not.

For the processes which take place if the resource monitor detects a failure, refer to Section 4.3, “Resource Monitoring”.

6.3.10. Configuring a Cluster Resource Group¶

Some cluster resources depend on other components or resources and require that each component or resource starts in a specific order and runs on the same server. To simplify this configuration we support the concept of groups.

For an example of a resource group and more information about groups and their properties, refer to Section 4.2.5.1, “Groups”.

	Empty Groups
Groups must contain at least one resource, otherwise the configuration is not valid. In Hawk, primitives cannot be created or modified while creating a group. Before adding a group, create primitives and configure them as desired. For details, refer to Procedure 6.4, “Adding Primitive Resources”.

Procedure 6.14. Adding a Resource Group¶

Start a Web browser and log in to the cluster as described in Section 6.1.1, “Starting Hawk and Logging in”.
In the left navigation bar, select Resources. The Resources screen shows categories for all types of resources and lists all defined resources.
Select the Group category and click the plus icon.
Enter a unique Group ID.
To define the group members, select one or multiple entries in the list of Available Primitives and click the < icon to add them to the Group Children list. Any new group members are added to the bottom of the list. To define the order of the group members, you currently need to add and remove them in the order you desire.
If needed, modify or add Meta Attributes as described in , Step 7.
Click Create Group to finish the configuration. A message at the top of the screen shows if the group was successfully created.

Figure 6.9. Hawk—Resource Group

6.3.11. Configuring a Clone Resource¶

If you want certain resources to run simultaneously on multiple nodes in your cluster, configure these resources as a clones. For example, cloning makes sense for resources like STONITH and cluster file systems like OCFS2. You can clone any resource provided. Cloning is supported by the resource’s Resource Agent. Clone resources may be configured differently depending on which nodes they are running on.

For an overview of the available types of resource clones, refer to Section 4.2.5.2, “Clones”.

	Sub-resources for Clones
Clones can either contain a primitive or a group as sub-resources. In Hawk, sub-resources cannot be created or modified while creating a clone. Before adding a clone, create sub-resources and configure them as desired. For details, refer to Procedure 6.4, “Adding Primitive Resources” or Procedure 6.14, “Adding a Resource Group”.

Procedure 6.15. Adding or Modifying Clones¶

Start a Web browser and log in to the cluster as described in Section 6.1.1, “Starting Hawk and Logging in”.
In the left navigation bar, select Resources. The Resources screen shows categories for all types of resources and lists all defined resources.
Select the Clone category and click the plus icon.
Enter a unique Clone ID.
From the Child Resource list, select the primitive or group to use as a sub-resource for the clone.
If needed, modify or add Meta Attributes as described in , Step 7.
Click Create Clone to finish the configuration. A message at the top of the screen shows if the clone was successfully created.

Figure 6.10. Hawk—Clone Resource

6.4. Managing Cluster Resources¶

In addition to configuring your cluster resources, Hawk allows you to manage existing resources from the Cluster Status screen. For a general overview of the screen, its different views and the color code used for status information, refer to Section 6.1.2, “Main Screen: Cluster Status”.

Basic resource operations can be executed from any cluster status view. Both Tree View and Table View let you access the individual resources directly. However, in the Summary View you need to click the links in the resources category first to display the resource details.

6.4.1. Starting Resources¶

Before you start a cluster resource, make sure it is set up correctly. For example, if you want to use an Apache server as a cluster resource, set up the Apache server first and complete the Apache configuration before starting the respective resource in your cluster.

Do Not Touch Services Managed by the Cluster

	Do Not Touch Services Managed by the Cluster
When managing a resource via the High Availability Extension, the same resource must not be started or stopped otherwise (outside of the cluster, for example manually or on boot or reboot). The High Availability Extension software is responsible for all service start or stop actions. However, if you want to check if the service is configured properly, start it manually, but make sure that it is stopped again before High Availability takes over. For interventions in resources that are currently managed by the cluster, set the resource to `unmanaged mode` first as described in Procedure 6.21, “Changing Management Mode of Resources”.

When managing a resource via the High Availability Extension, the same resource must not be started or stopped otherwise (outside of the cluster, for example manually or on boot or reboot). The High Availability Extension software is responsible for all service start or stop actions.

However, if you want to check if the service is configured properly, start it manually, but make sure that it is stopped again before High Availability takes over.

For interventions in resources that are currently managed by the cluster, set the resource to unmanaged mode first as described in Procedure 6.21, “Changing Management Mode of Resources”.

When creating a resource with Hawk, you can set its initial state with the target-role meta attribute. If you set its value to stopped, the resource does not start automatically after being created.

Procedure 6.16. Starting A New Resource¶

Start a Web browser and log in to the cluster as described in Section 6.1.1, “Starting Hawk and Logging in”.
In the left navigation bar, select Cluster Status.
In one of the individual resource views, click the wrench icon next to the resource and select Start. To continue, confirm the message that appears. As soon as the resource has started, Hawk changes the resource's color to green and shows on which node it is running.

6.4.2. Cleaning Up Resources¶

A resource will be automatically restarted if it fails, but each failure increases the resource's failcount.

If a migration-threshold has been set for the resource, the node will no longer run the resource when the number of failures reaches the migration threshold.

A resource's failcount can either be reset automatically (by setting a failure-timeout option for the resource) or you can reset it manually as described below.

Procedure 6.17. Cleaning Up A Resource¶

Start a Web browser and log in to the cluster as described in Section 6.1.1, “Starting Hawk and Logging in”.
In the left navigation bar, select Cluster Status.
In one of the individual resource views, click the wrench icon next to the failed resource and select Clean Up. To continue, confirm the message that appears.
This executes the commands crm_resource -C and crm_failcount -D for the specified resource on the specified node.

For more information, see the man pages of crm_resource and crm_failcount.

6.4.3. Removing Cluster Resources¶

If you need to remove a resource from the cluster, follow the procedure below to avoid configuration errors:

	Removing Referenced Resources
A cluster resource cannot be removed if its ID is referenced by any constraint. If you cannot delete a resource, check where the resource ID is referenced and remove the resource from the constraint first.

Procedure 6.18. Removing a Cluster Resource¶

Start a Web browser and log in to the cluster as described in Section 6.1.1, “Starting Hawk and Logging in”.
In the left navigation bar, select Cluster Status.
Clean up the resource on all nodes as described in Procedure 6.17, “Cleaning Up A Resource”.
In one of the individual resource views, click the wrench icon next to the resource and select Stop. To continue, confirm the message that appears.
Remove all constraints that relate to the resource, otherwise removing the resource will not be possible. For details, refer to Procedure 6.10, “Removing Constraints”.
If the resource is stopped, click the wrench icon next to it and select Delete Resource.

6.4.4. Migrating Cluster Resources¶

As mentioned in Section 6.3.6, “Specifying Resource Failover Nodes”, the cluster will fail over (migrate) resources automatically in case of software or hardware failures—according to certain parameters you can define (for example, migration threshold or resource stickiness). Apart from that, you can also manually migrate a resource to another node in the cluster (or decide to just move it away from the current node and leave the decision where to put it to the cluster).

Procedure 6.19. Manually Migrating a Resource¶

Start a Web browser and log in to the cluster as described in Section 6.1.1, “Starting Hawk and Logging in”.
In the left navigation bar, select Cluster Status.
In one of the individual resource views, click the wrench icon next to the resource and select Move.
In the new window, select the node to which to move the resource.
This creates a location constraint with an INFINITY score for the destination node.
Alternatively, select to move the resource Away from current node. This creates a location constraint with a -INFINITY score for the current node.
Click OK to confirm the migration.

To allow a resource to move back again, proceed as follows:

Procedure 6.20. Clearing a Migration Constraint¶

Start a Web browser and log in to the cluster as described in Section 6.1.1, “Starting Hawk and Logging in”.
In the left navigation bar, select Cluster Status.
In one of the individual resource views, click the wrench icon next to the resource and select Drop Relocation Rule. To continue, confirm the message that appears.
This uses the crm_resource -U command. The resource can move back to its original location or it may stay where it is (depending on resource stickiness).

For more information, see the crm_resource man page or Pacemaker Explained, available from http://www.clusterlabs.org/doc/ . Refer to section Resource Migration.

6.4.5. Changing Management Mode of Resources¶

When a resource is being managed by the cluster, it must not be touched otherwise (outside of the cluster). For maintenance of individual resources, you can set the respective resources to an unmanaged mode that allows you to modify the resource outside of the cluster.

Procedure 6.21. Changing Management Mode of Resources¶

Start a Web browser and log in to the cluster as described in Section 6.1.1, “Starting Hawk and Logging in”.
In the left navigation bar, select Resources. Select the resource you want to put in unmanaged mode, click the wrench icon next to the resource and select Edit Resource.
Open the Meta Attributes category.
From the empty drop-down list, select the is-managed attribute and click the plus icon to add it.
Deactivate the checkbox next to is-managed to put the resource in unmanaged mode and confirm your changes.
After you have finished the maintenance task for that resource, reactivate the checkbox next to the is-managed attribute for that resource.
From this point on, the resource will be managed by the High Availability Extension software again.

6.4.6. Viewing the Cluster History¶

Hawk provides the following possibilities to view past events on the cluster (on different levels and in varying detail).

Procedure 6.22. Viewing Recent Events of Nodes or Resources¶

Start a Web browser and log in to the cluster as described in Section 6.1.1, “Starting Hawk and Logging in”.
In the left navigation bar, select Cluster Status.
In the Tree View or Table View, click the wrench icon next to the resource or node you are interested in and select View Recent Events.
The dialog that opens shows the events of the last hour.

Procedure 6.23. Viewing Transitions with the History Explorer¶

The History Explorer provides transition information for a time frame that you can define:

Start a Web browser and log in to the cluster as described in Section 6.1.1, “Starting Hawk and Logging in”.
In the left navigation bar, select History Explorer.
By default, the period to explore is set to the last 24 hours. To modify this, set another Start Time and End Time.
Click Display to start collecting transition data.

Figure 6.11. Hawk—History Report

The following information is displayed:

The time line of all past transitions in the cluster.
The pe-input files for each transition. For each transition, the cluster saves a copy of the state which is provided to the policy engine as input. The path to this archive is logged. The files are only generated on the Designated Coordinator (DC), but as the DC can change, there may be pe-input files from several nodes listed in the History Explorer. The files show what the Policy Engine (PE) planned to do.
Graph, XML representation and details of each transition.
If you choose to show the Graph, the PE is reinvoked (using the pe-input files), and generates a graphical visualization of the transition. Alternatively, you can view the XML representation of the graph. Clicking Details shows snippets of var/log/messages that belong to that particular transition.

Figure 6.12. Hawk History Report—Transition Graph

6.4.7. Exploring Potential Failure Scenarios¶

Hawk provides a Simulator that allows you to explore failure scenarios before they happen. After switching to the simulator mode, change the status of nodes or execute multiple resource operations to see how the cluster would behave should these events occur.

Procedure 6.24. Switching to Simulator Mode¶

Start a Web browser and log in to the cluster as described in Section 6.1.1, “Starting Hawk and Logging in”.
In the left navigation bar, select Simulator.
Hawk's background changes color to indicate the simulator is active. A simulator dialog opens in the bottom right hand corner of the screen. Its title Simulator (initial state) indicates that Cluster Status screen still reflects the current state of the cluster.
To simulate status change of a node:
1. Click +Node in the simulator control dialog.
2. Select the Node you want to manipulate and select its target State.
3. Confirm your changes to add them to the queue of events listed in the controller dialog below Injected State.
To simulate a resource operation:
1. Click +Op in the simulator control dialog.
2. Select the Resource to manipulate and the Operation to simulate. If necessary, define an Interval. Select the Node on which to run the operation and the targeted Result.
3. Confirm your changes to add them to the queue of events listed in the controller dialog below Injected State.
Repeat the previous steps for any other events you wish to simulate.
To remove any of the events listed in Injected State, select the entry and click the minus icon beneath the list.
To start the simulation, click Run in the simulator control dialog. The Cluster Status screen displays the simulated events. For example, if you marked a node as unclean, it will now be shown offline, and all its resources will be stopped. The simulator control dialog changes to Simulator (final state).
To view more detailed information:
1. To see log snippets of what occurred, click the Details link in the simulator dialog.
2. To see the transition graph, click the Graph link.
3. To display the initial CIB state, click CIB (in).
4. To see what the CIB would look like after the transition, click CIB (out).
To start from scratch with a new simulation, use the Reset button.
To exit the simulation mode, close the simulator control dialog. The Cluster Status screen switches back to its normal color and displays the current cluster state.

Figure 6.13. Hawk—Simulator

6.4.8. Generating a Cluster Report¶

For analysis and diagnosis of problems occurring on the cluster, Hawk can generate a cluster report that collects information from all nodes in the cluster.

Procedure 6.25. Generating a hb_report¶

Start a Web browser and log in to the cluster as described in Section 6.1.1, “Starting Hawk and Logging in”.
In the left navigation bar, select Generate hb_report.
By default, the period to examine is the last hour. To modify this, set another Start Time and End Time.
Click Generate.
After the report has been created, download the *.tar.bz2 file by clicking the respective link.

6.5. Troubleshooting¶

Hawk Log Files

Find the Hawk log files in /srv/www/hawk/log. Check these files in case you cannot access Hawk.

If you have trouble starting or stopping a resource with Hawk, check the Pacemaker log messages. By default, Pacemaker logs to /var/log/messages.

Authentication Fails

If you cannot log in to Hawk with a new user that is a member of the haclient group (or if you experience delays until Hawk accepts logins from this user), stop the nscd daemon with rcnscd stop and try again.

Replacing the Self-Signed Certificate

To avoid the warning about the self-signed certificate on first Hawk startup, replace the automatically created certificate with your own certificate or a certificate that was signed by an official Certificate Authority (CA).

The certificate is stored in /etc/lighttpd/certs/hawk-combined.pem and contains both key and certificate. After you have created or received your new key and certificate, combine them by executing the following command:

cat keyfile certificatefile > /etc/lighttpd/certs/hawk-combined.pem

Change the permissions to make the file only accessible by root:

chown root.root /etc/lighttpd/certs/hawk-combined.pem
chmod 600 /etc/lighttpd/certs/hawk-combined.pem

Login to Hawk Fails After Using History Explorer/hb_report

Depending on the period of time you defined in the History Explorer or hb_report and the events that took place in the cluster during this time, Hawk might collect an extensive amount of information stored in log files in the /tmp directory. This might consume the remaining free disk space on your node. In case Hawk should not respond after using the History Explorer or hb_report, check the hard disk of your cluster node and remove the respective log files.

	No Support Without STONITH
A cluster without STONITH enabled is not supported.