CRM CLI (command line interface) tool
======================================
Dejan_Muhamedagic,_Yan_Gao dejan@suse.de,ygao@novell.com
v0.9

The CRM (a.k.a Pacemaker) is a Cluster Resource Manager which
implements the cluster configuration provided by the user in CIB
(Cluster Information Base). The CIB is a set of instructions
coded in XML. Editing the CIB is a challenge, not only due to its
complexity and a wide variety of options, but also because XML is
not exactly user friendly.

.Note
**************************
I do understand that there are people capable of
dealing with XML without an intermediary.
**************************

There are currently three options to manage the CIB, listed here
in a decreasing order of user-friendliness:

- the GUI (`hb_gui`)
- a set of command line tools
- `cibadmin(8)`

The GUI is very popular and it has recently seen a lot of good
development. For some it is going to be (or remain) the first
choice in cluster management.

The command line tools, lead by `crm_resource(8)`, are capable of
performing almost any kind of CIB transformation. The usage is,
however, plagued by the notorious weakness common to all UNIX
tools: a multitude of options, necessary for operation and yet
very hard to remember. Usage is also inconsistent at times.

The `cibadmin` is the ultimate CIB management tool: it applies
chunks of XML written by the user or generated by another tool to
the CIB. Very difficult to use without extensive training. Or
should I say drill. May be unnerving as well, in particular due
to sometimes cryptic error messages.

== Design goals

The CLI provides a consistent and unified interface to
CIB/cluster management. It uses the command line tools where
possible and may resort to XML and `cibadmin` when there is no
other option. That is the easiest way to ensure compatibility
between different management tools.

It may be used either as an interactive shell or for single
commands directly on the shell's command line. It is also
possible to feed it a set of commands from standard input, thus
turning it into a scripting tool. Templates with ready made
configurations may help people to learn about the cluster
configuration or facilitate testing procedures.

The CLI may also be used for the CIB description and generation.
A file containing a set of CLI instructions may be applied to the
CLI tool to generate a complete CIB.

The new shadow CIB feature may also be put to use. The user may
work on one of the shadow CIBs and then apply (or commit) it in a
single step to the cluster.
 
It should also allow deployment of raw XML which may come either
from files or network.

Several modes of operation are available to restrict the set of
features depending on the user's proficiency.

The CLI is line oriented: every command must start and finish
on the same line. It is possible to use a continuation character
to write one command in two or more lines.

The CLI must run on one of the cluster nodes.

.Note
**************************
Even though all sensible configurations (and most of those that
are not) are going to be supported by the CLI, I suspect that it
may still happen that certain XML constructs may confuse the
tool. When that happens, please file a bug report.

The CLI will not try to update the objects it does not
understand. Of course, it is always possible to edit such objects
in the XML format.
**************************

== Introduction to the user interface

Arguably the most important aspect of such a program is the user
interface. We begin with an informal introduction so that the
reader may get acquainted with it and get a general feeling of
the tool. It is probably best just to give some examples:

1. Command line (one-shot) use:

        # crm resource stop www_app

2. Interactive use:

        # crm
        crm(live)# resource
        crm(live) resource# unmanage tetris_1
        crm(live) resource# up
        crm(live)# node standby node4

3. Cluster configuration:

    # crm<<EOF
    configure
      erase
      primitive disk0 iscsi \
        params portal=192.168.2.108:3260 target=iqn.2008-07.com.suse:disk0
      primitive fs0 Filesystem \
        params device=/dev/disk/by-label/disk0 directory=/disk0 fstype=ext3
      primitive internal_ip IPaddr params ip=192.168.1.101
      primitive apache apache \
        params configfile=/disk0/etc/apache2/site0.conf \
        op monitor interval=60s timeout=30s
      primitive apcfence stonith:apcsmart \
        params ttydev=/dev/ttyS0 hostlist="node1 node2" \
        op start timeout=60s \
        op monitor interval=30m timeout=60s
      primitive pingd pingd \
        params name=pingd dampen=5s multiplier=100 host_list="r1 r2"
      # cluster layout
      group internal_www \
        disk0 fs0 internal_ip apache
      clone fence apcfence \
        meta globally-unique=false clone-max=2 clone-node-max=1
      clone conn pingd \
        meta globally-unique=false clone-max=2 clone-node-max=1
      location node_pref internal_www \
        rule 50: #uname eq node1 \
        rule pingd: defined pingd
      property stonith-enabled=true
      property stonith-action=shutdown
    end
    EOF

If you've ever done a CRM style configuration, you should be able
to understand the above examples without much difficulties.  The
CLI should provide a means to manage the cluster efficiently or
put together a configuration in a concise manner.

The `(live)` string in the prompt signifies that the current CIB
in use is the cluster live configuration. It is possible to also
use the so-called shadow CIBs, i.e. configurations which are
stored in files and aren't active, but may be applied at any time
to the cluster.

Since the CIB is hierarchical such is the interface too. There
are several levels and entering each of them enables the user to
use a certain set of commands.

== Reference

We define a small and simple language. Most commands consist of
just a list of simple tokens. The only complex constructs are
found at the `configure` level.

The syntax is described in a somewhat informal manner: `<>`
denotes a string, `[]` means that the construct is optional, the
ellipsis (`...`) signifies that the previous construct may be
repeated, `|` means pick one of many, and the rest are literals
(strings, `:`, `=`).

=== `cib`

This level is for management of shadow CIBs.

All the commands are implemented using `cib_shadow(8)` and the
`CIB_shadow` environment variable. The user prompt always
includes the name of the currently active shadow or the live CIB.

==== `list`

List existing shadow CIBs.

Usage:
...............
        list
...............

==== `new/delete`

Create a new shadow CIB or delete an existing one. On `new`, 
the live cluster configuration is copied.

Usage:
...............
        new <cib>
        delete <cib>
...............

==== `reset`

Copy the current cluster configuration into the shadow CIB.

Usage:
...............
        reset <cib>
...............

==== `use`

Choose a CIB. Leave out the CIB name to switch to the running
CIB.

Usage:
...............
        use [<cib>]
...............

==== `diff`

Print differences between the current cluster configuration and
the active shadow CIB.

Usage:
...............
        diff
...............

==== `commit`

Apply a shadow CIB to the cluster.

Usage:
...............
        commit <cib>
...............

=== `ra`

This level contains commands which show various information about
the installed resource agents. It is available both at the top
level and at the `configure` level.

==== `classes`

Print all resource agents' classes and, where appropriate, a list
of available providers.

Usage:
...............
        classes
...............

==== `list`

List available resource agents for the given class. If the class
is `ocf`, supply a provider to get agents which are available
only from that provider.

Usage:
...............
        list <class> [<provider>]
...............
Example:
...............
        list ocf pacemaker
...............

==== `meta`

Show the meta-data of a resource agent type. This is where users
can find information on how to use a resource agent.

Usage:
...............
        meta <type> <class> [<provider>]
...............
Example:
...............
        meta apache ocf
        meta ipmilan stonith
...............

==== `providers`

List providers for a resource agent type.

Usage:
...............
        providers <type>
...............
Example:
...............
        providers apache
...............

=== `resource`

At this level resources may be managed.

All (or almost all) commands are implemented with the CRM tools
such as `crm_resource(8)`.

==== `status`

Print resource status. If the resource parameter is left out
status of all resources is printed.

Usage:
...............
        status [<rsc>]
...............

==== `start/stop`

Start/stop a resource using the `target-role` attribute.

Usage:
...............
        start <rsc>
        stop <rsc>
...............

==== `manage/unmanage`

Manage/unmanage a resource using the `is-managed` attribute.

Usage:
...............
        manage <rsc>
        unmanage <rsc>
...............

==== `migrate/unmigrate`

Migrate a resource to a different node or remove the constraint
generated by the previous migrate command. If node is left out, the
resource is migrated by creating a constraint which prevents it from
running on the current node.

Usage:
...............
        migrate <rsc> [<node>]
        unmigrate <rsc>
...............

==== `param`

Show/edit/delete a parameter of a resource.

Usage:
...............
        param <rsc> set <param> <value>
        param <rsc> delete <param>
        param <rsc> show <param>
...............
Example:
...............
        param ip_0 show ip
...............

==== `meta`

Show/edit/delete a meta attribute of a resource. Currently, all
meta attributes of a resource may be managed with other commands
such as `resource stop`.

Usage:
...............
        meta <rsc> set <attr> <value>
        meta <rsc> delete <attr>
        meta <rsc> show <attr>
...............
Example:
...............
        meta ip_0 set target-role stopped
...............

==== `failcount`

Show/edit/delete the failcount of a resource.

Usage:
...............
        failcount <rsc> set <node> <value>
        failcount <rsc> delete <node>
        failcount <rsc> show <node>
...............
Example:
...............
        failcount fs_0 delete node2
...............

==== `cleanup`

Cleanup resource status. Typically done after the resource has
temporarily failed. If a node is omitted, cleanup on all nodes.
If there are many nodes, the command may take a while.

Usage:
...............
        cleanup <rsc> [<node>]
...............

==== `refresh`

Refresh CIB from the LRM status.

Usage:
...............
        refresh [<node>]
...............

==== `reprobe`

Probe for resources not started by the CRM.

Usage:
...............
        reprobe [<node>]
...............

=== `node`

Node management and status commands.

==== `show`

Show a node definition. If the node parameter is omitted then all
nodes are shown.

Usage:
...............
        show [<node>]
...............

==== `standby/online`

Set a node to standby or online status. The node parameter
defaults to the node where the command is run.

Usage:
...............
        standby [<node>]
        online [<node>]
...............

==== `delete`

Delete a node. This command will remove the node from the CIB
and, in case the heartbeat stack is running, run hb_delnode too.

Usage:
...............
        delete <node>
...............

==== `attribute`

Edit node attributes. This kind of attribute should refer to
relatively static properties, such as memory size.

Usage:
...............
        attribute <node> set <attr> <value>
        attribute <node> delete <attr>
        attribute <node> show <attr>
...............
Example:
...............
        attribute node_1 set memory_size 4096
...............

==== `status-attr`

Edit node attributes which are in the CIB status section, i.e.
attributes which hold properties of a more volatile nature. One
typical example is attribute generated by the `pingd` utility.

Usage:
...............
        status-attr <node> set <attr> <value>
        status-attr <node> delete <attr>
        status-attr <node> show <attr>
...............
Example:
...............
        status-attr node_1 show pingd
...............

=== `options`

The user may set various options for the CLI program itself.

==== `skill-level`

Based on the skill-level setting, the user is allowed to use only
a subset of commands. There are three levels: operator,
administrator, and expert. The operator level allows only
commands at the `resource` and `node` levels, but not editing
or deleting resources. The administrator may do that and may also
configure the cluster at the `configure` level and manage the
shadow CIBs. The expert may do all.

Usage:
...............
        skill-level level
        
        level :: operator | administrator | expert
...............

==== `user`

Sufficient privileges are necessary in order to manage a
cluster: programs such as `crm_verify` or `crm_resource` and,
ultimately, `cibadmin` have to be run either as `root` or as the
CRM owner user (typically `hacluster`). You don't have to worry
about that if you run `crm` as `root`. A more secure way is to
run the program with your usual privileges, set this option to
the appropriate user (such as `hacluster`), and setup the
`sudoers` file.

Usage:
...............
        user system-user
...............
Example:
...............
        user hacluster
...............

==== `editor`

The `edit` command invokes an editor. Use this to specify your
prefered editor program. If not set, it will default to either
the value of the `EDITOR` environment variable or to one of the
standard UNIX editors (`vi`,`emacs`,`nano`).

Usage:
...............
        editor program
...............
Example:
...............
        editor vim
...............

==== `pager`

The `view` command displays text through a pager. Use this to
specify your prefered pager program. If not set, it will default
to either the value of the `PAGER` environment variable or to one
of the standard UNIX system pagers (`less`,`more`,`pg`).

==== `show`

Display all current settings.

==== `save`

Save current settings to the rc file (`$HOME/.crm.rc`). On
further `crm` runs, the rc file is automatically read and parsed.

=== `configure`

This level enables all CIB object definition commands.

The configuration may be logically divided into four parts:
nodes, resources, constraints, and (cluster) properties and
attributes.  Each of these commands support one or more basic CIB
objects.

Nodes and attributes describing nodes are managed using the
`node` command.

Commands for resources are:

- `primitive`
- `group`
- `clone`
- `ms` (master-slave)

There are three types of constraints:

- `location`
- `colocation`
- `order`

Finally, there are the cluster properties, resource meta
attributes defaults, and operations defaults. All are just a set
of attributes. These attributes are managed by the following
commands:

- `property`
- `rsc_defaults`
- `op_defaults`

The changes applied to the current CIB only on ending the
configuration session or using the `commit` command.

==== `node`

The node command describes a cluster node. Nodes in the CIB are
commonly created automatically by the CRM. Hence, you should not
need to do this yourself unless you also want to define node
attributes. Note that it is also possible to manage node
attributes at the `node` level.

Usage:
...............
        node <uname>[:<type>]
          [attributes <param>=<value> [<param>=<value>...]]

        type :: normal | member | ping
...............
Example:
...............
        node node1
        node big_node attributes memory=64
...............

==== `primitive`

The primitive command describes a resource. It may be referenced
only once in group, clone, or master-slave objects. If it's not
referenced, then it is placed as a single resource in the CIB.

Operations may be specified in three ways. "Anonymous" as a
simple list of "op" specifications. Use that if you don't want to
reference the set of operations elsewhere. That's by far the most
common way to define operations. If reusing operation sets is
desired, use the "operations" keyword along with the id to give
the operations set a name and the id-ref to reference another set
of operations.

Usage:
...............
        primitive <rsc> [<class>:[<provider>:]]<type>
          [params <param>=<value> [<param>=<value>...]]
          [meta <attribute>=<value> [<attribute>=<value>...]]
          [operations id_spec
            [op op_type [<attribute>=<value>...] ...]]

        id_spec :: $id=<id> | $id-ref=<id>
        op_type :: start | stop | monitor
...............
Example:
...............
        primitive apcfence stonith:apcsmart \
          params ttydev=/dev/ttyS0 hostlist="node1 node2" \
          op start timeout=60s \
          op monitor interval=30m timeout=60s

        primitive www8 apache \
          params configfile=/etc/apache/www8.conf \
          operations $id-ref=apache_ops
...............

==== `group`

The `group` command creates a group of resources.

Usage:
...............
        group <name> <rsc> [<rsc>...]
          [params <param>=<value> [<param>=<value>...]]
          [meta <attribute>=<value> [<attribute>=<value>...]]
...............
Example:
...............
        group internal_www disk0 fs0 internal_ip apache \
          meta target_role=stopped
...............

==== `clone`

The `clone` command creates a resource clone. It may contain a
single primitive resource or one group of resources.

Usage:
...............
        clone <name> <rsc>
          [params <param>=<value> [<param>=<value>...]]
          [meta <attribute>=<value> [<attribute>=<value>...]]
...............
Example:
...............
        clone cl_fence apc_1 \
          meta clone-node-max=1 globally-unique=false
...............

==== `ms` (master/slave)

The `ms` command creates a master/slave resource type. It may contain a
single primitive resource or one group of resources.

Usage:
...............
        ms <name> <rsc>
          [params <param>=<value> [<param>=<value>...]]
          [meta <attribute>=<value> [<attribute>=<value>...]]
...............
Example:
...............
        ms disk1 drbd1 \
          meta notify=true globally-unique=false
...............

==== `location`

`location` defines the preference of nodes for the given
resource. The location constraints consist of one or more rules
which specify a score to be awarded if the rule matches.

Usage:
...............
        location <id> <rsc>
          rule [id_spec] [$role=<role>] <score>: <expression>
          [rule [id_spec] [$role=<role>] <score>: <expression> ...]

        id_spec :: $id=<id> | $id-ref=<id>
        score :: <number> | <attribute> | [-]inf
        expression :: <single_exp> [bool_op <simple_exp> ...]
                      | <date_expr>
        bool_op :: or | and
        single_exp :: <attribute> [type:]<binary_op> <value>
                      | <unary_op> <attribute>
        type :: string | version | number
        binary_op :: lt | gt | lte | gte | eq | ne
        unary_op :: defined | not_defined

        date_expr :: date_op <start> [<end>]  (TBD)
...............
Examples:
...............
        location conn_1 internal_www \
          rule 50: #uname eq node1 \
          rule pingd: defined pingd

        location conn_2 dummy_float \
          rule -inf: not_defined pingd or pingd lte 0
...............

==== `colocation`

This constraint expresses the placement relation between two
resources.

Usage:
...............
        colocation <id> <score>: <rsc>[:<role>] <rsc>[:<role>]
...............
Example:
...............
        colocation dummy_and_apache -inf: apache dummy
...............

==== `order`

This constraint expresses the order of actions on two resources.

Usage:
...............
        order <id> score-type: <first-rsc>[:<action>] <then-rsc>[:<action>]
          [symmetrical=<bool>]

        score-type :: advisory | mandatory | <score>
...............
Example:
...............
        order c_apache_1 mandatory: apache:start ip_1
...............

==== `property`

Set the cluster (`crm_config`) options.

Usage:
...............
        property [$id=<set_id>] <option>=<value> [<option>=<value> ...]
...............
Example:
...............
        property stonith-enabled=true
...............

==== `rsc_defaults`

Set defaults for the resource meta attributes.

Usage:
...............
        rsc_defaults [$id=<set_id>] <option>=<value> [<option>=<value> ...]
...............
Example:
...............
        rsc_defaults failure-timeout=3m
...............

==== `op_defaults`

Set defaults for the operations meta attributes.

Usage:
...............
        op_defaults [$id=<set_id>] <option>=<value> [<option>=<value> ...]
...............
Example:
...............
        op_defaults record-pending=true
...............

=== `show`

The `show` command displays objects. It may display all objects
or a set of objects. The user may also choose to see only objects
which were changed.
Optionally, the XML code may be displayed instead of the CLI
representation.

Usage:
...............
        show [xml] [<id> ...]
        show [xml] changed
...............

=== `edit`

This command invokes the editor with the object description. As
with the `show` command, the user may choose to edit all objects
or a set of objects.
If the user insists, he or she may edit the XML edition of the
object.

Usage:
...............
        edit [xml] [<id> ...]
        edit [xml] changed
...............

=== `delete`

Delete one or more objects. If an object to be deleted belongs to
a container object, such as a group, and it is the only resource
in that container, then the container is deleted as well.

Usage:
...............
        delete <id> [<id>...]
...............

=== `rename`

Rename an object. It is recommended to use this command to rename
a resource, because it will take care of updating all related
constraints. Changing ids with the edit command won't have the
same effect.

If you want to rename a resource, it must be stopped.

Usage:
...............
        rename <old_id> <new_id>
...............

=== `refresh`

Refresh the internal structures from the CIB. All changes made
during this session are gone.

Usage:
...............
        refresh
...............

=== `erase`

The `erase` clears all configuration at the current level.

Usage:
...............
        erase
...............

.Warning
**************************
If you haven't already noticed, this command is very destructive.
**************************

=== `ptest`

Show PE (policy engine) motions using ptest.

A CIB is constructed using the current user edited version and
the status from the current CIB. This CIB is run through ptest
to show changes. If you have graphviz installed and X11 session,
dotty is run to display the changes graphically.

Usage:
...............
        ptest
...............

=== `commit`

Commit the current configuration to the CIB in use. As noted
elsewhere, commands in a configure session don't have immediate
effect on the CIB. All changes are applied at one point in time,
either using `commit` or when the user leaves the configure
level. In case the CIB in use changed in the meantime, presumably
by somebody else, the CLI will refuse to apply the changes. If
you know that it's fine to still apply them add `force`.

Usage:
...............
        commit [force]
...............

=== `upgrade`

If you get the `CIB not supported` error, which typically means
that the current CIB version is coming from the older release,
you may try to upgrade it to the latest revision. The command
to perform the upgrade is:

    cibadmin --upgrade --force

If we don't recognize the current CIB as the old one, but you're
sure that it is, you may force the command.

Usage:
...............
        upgrade [force]
...............

=== `verify`

Verify the contents of the CIB which would be committed.

Usage:
...............
        verify
...............

=== `save`

Save the configuration of the current level to a file.
Optionally, as XML.

Usage:
...............
        save [xml] <file>
...............
Example:
...............
        save myfirstcib.txt
...............

=== `load`

Load a part of configuration (or all of it) from a local file or
a network URL. The `replace` method replaces the current
configuration with the one from the source. The `update` tries to
import the contents into the current configuration.
The file may be a CLI file or an XML file.

Usage:
...............
        load [xml] method URL

        method :: replace | update
...............
Example:
...............
        load xml update myfirstcib.xml
        load xml replace http://storage.big.com/cibs/bigcib.xml
...............

=== `template`

User may be assisted in the cluster configuration by a template
prepared in advance. Templates consist of a typical ready
configuration which may be edited to suit particular user needs.

This command opens a template and loads it into the editor (it is
similar to a `load` followed by `edit`). The user may then fill
out the missing parts or replace the existing options and use it
as the cluster configuration.

The current configuration is replaced by the template.

Usage:
...............
        template [xml] <url>
...............
Example:
...............
        template two-apaches.txt
...............

.Note
****************************
A template may contain a complete configuration, or, say, a set
of resources, or a single object definition. It is up to the user
to pick the right one. However, even if a template from a
different level is loaded it may be edited to fit the current
object level.
****************************

=== `end`

The `end` command ends the current level and the user moves to
the parent level. If invoked at the `enter` level, the user
signals that they finished with the object description. This
command is available everywhere.

Usage:
...............
        end
...............

=== `help`

The `help` command prints help for the current level or for the
specified topic (command). This command is available everywhere.

Usage:
...............
        help [<topic>]
...............

=== `quit`

Leave the program.

//////////////////////
 vim:ts=4:sw=4:expandtab:
//////////////////////
