Contents
Abstract
DRBD allows you to create a mirror of two block devices that are located at two different sites across an IP network. When used with OpenAIS, DRBD supports distributed high-availability Linux clusters. This chapter shows you how to install and set up DRBD.
DRBD replicates data on the primary device to the secondary device in a way that ensures that both copies of the data remain identical. Think of it as a networked RAID 1. It mirrors data in real-time, so its replication occurs continuously. Applications do not need to know that in fact their data is stored on different disks.
![]() | Unencrypted Data |
|---|---|
The data traffic between mirrors is not encrypted. For secure data exchange, you should deploy a Virtual Private Network (VPN) solution for the connection. | |
DRBD is a Linux kernel module and sits between the I/O scheduler at the lower end and the file system at the upper end, see Figure 13.1, “Position of DRBD within Linux”. To communicate with DRBD, users use the high-level command drbdadm. For maximum flexibility DRBD comes with the low-level tool drbdsetup.
DRBD allows you to use any block device supported by Linux, usually:
partition or complete hard disk
software RAID
Logical Volume Manager (LVM)
Enterprise Volume Management System (EVMS)
By default, DRBD uses the TCP port 7780 and higher for communication between DRBD nodes. Make sure that your firewall does not prevent communication on this port.
You must set up the DRBD devices before creating file systems on them.
Everything pertaining to user data should be done solely via the
/dev/drbd_ device and
not on the raw device, as DRBD uses the last 128 MB of the raw device for
metadata. Make sure to create file systems only on the
R/dev/drbd<n> device and not on the raw device.
For example, if the raw device is 1024 MB in size, the DRBD device has only 896 MB available for data, with 128 MB hidden and reserved for the metadata. Any attempt to access the space between 896 MB and 1024 MB fails because it is not available for user data.
To install the needed packages for DRBD, install the High Availability Extension Add-On product on both SUSE Linux Enterprise Server machines in your networked cluster as described in Part I, “Installation and Setup”. Installing High Availability Extension also installs the DRBD program files.
If you do not need the complete cluster stack but just want to use DRBD,
Table 13.1, “DRBD RPM Packages” contains a list of all RPM
packages for DRBD. During the last version the
drbd package was split into seperate packages.
Table 13.1. DRBD RPM Packages
|
Filename |
Explanation |
|---|---|
|
|
Convenience package, split into other |
|
|
Programmable bash completion support for drbdadm |
|
|
Heartbeat resource agent for DRBD (only needed for Heartbeat) |
|
|
Kernel module for DRBD (needed) |
|
|
Xen kernel module for DRBD |
|
|
udev integration scripts for DRBD, managing symlinks to DRBD devices
in |
|
|
Management utilities for DRBD (needed) |
|
|
Pacemaker resource agent for DRBD |
|
|
Xen block device management script for DRBD |
|
|
YaST DRBD Configuration (recommended) |
To simplify the work with drbdadm, use the Bash
complection support in the RPM package
drbd-bash-completion. If you want to enable it
in your current shell session, insert the following command:
source /etc/bash_completion.d/drbdadm.sh
To use it permanently for root, create a file
/root/.bashrc and insert the previous line.
![]() | |
The following procedure uses the server names jupiter and venus, and the cluster resource name r0. It sets up jupiter as the primary node. Make sure to modify the instructions to use your own nodes and filenames. | |
Before you start configuring DRBD, make sure your block devices in your Linux nodes are ready and paritioned (if needed). The following procedure assumes you have two nodes, jupiter and venus, and they use the TCP port 7780. Make sure this port is open in your firewall.
To set up DRBD manually, proceed as follows:
Procedure 13.1. Manually Configure DRBD
Log in as user root.
Change DRBD's configuration files:
Open the file /etc/drbd.conf and insert the
following lines, if not available:
include "drbd.d/global_common.conf"; include "drbd.d/*.res";
Beginning with DRBD 8.3 the configuration file is split into separate
files, located under the directory /etc/drbd.d.
Open the file /etc/drbd.d/global_common.conf. It
contains already some pre-definied values. Go to the
startup section and insert these lines:
startup {
# wfc-timeout degr-wfc-timeout outdated-wfc-timeout
# wait-after-sb;
wfc-timeout 1;
degr-wfc-timeout 1;
}These options are used to reduce the timeouts when booting, see http://www.drbd.org/users-guide-emb/re-drbdconf.html for more details.
Create the file /etc/drbd.d/r0.res, change the
lines according to your situation, and save it:
resource r0 {
device /dev/drbd_r0 minor 0;
disk /dev/sda1;
meta-disk internal;
on jupiter {
address 192.168.1.10:7780;
}
on venus {
address 192.168.1.11:7780;
}
syncer {
rate 7M;
}
}
Name of the resource. It is recommended to use resource names like
| |
The device name for DRBD and its minor number. It is recommended to
start with | |
The device that is replicated between nodes. Note, in this example
the devices are the same on both nodes. If you need different
devices, move the | |
The meta-disk parameter usually contains the value
| |
The | |
The IP address and port number of the respective node. Each resource needs an individual port, usually starting with 7780. | |
The syncronization rate. Set it to one third of your bandwith. It only limits the resyncronization, not the mirroring. |
Check the syntax of your configuration file(s). If the following command returns an error, verify your files:
drbdadm dump all
Copy the DRBD configuration files to the other node:
scp /etc/drbd.conf venus:/etc/ scp /etc/drbd.d/* venus:/etc/drbd.d/
Initialize the meta data on both systems by entering the following on each node.
drbdadm -- --ignore-sanity-checks create-md r0 rcdrbd start
If your disk contains already a filesystem that you do not need anymore, destroy the filesystem structure with the following command and repeat this step:
dd if=/dev/zero of=/dev/sdb1 count=10000
Watch the DRBD status by entering the following on each node:
rcdrbd status
You should get something like this:
drbd driver loaded OK; device status: version: 8.3.7 (api:88/proto:86-91) GIT-hash: ea9e28dbff98e331a62bcbcc63a6135808fe2917 build by phil@fat-tyre, 2010-01-13 17:17:27 m:res cs ro ds p mounted fstype 0:r0 Connected Secondary/Secondary Inconsistent/Inconsistent C
Start the resync process on your intented primary node (jupiter in this case):
drbdadm -- --overwrite-data-of-peer primary r0
Check the status again with rcdrbd status and you get:
... m:res cs ro ds p mounted fstype 0:r0 Connected Primary/Secondary UpToDate/UpToDate C
The status in the ds row (disk status) must be
UpToDate on both nodes.
Set jupiter as primary node:
drbdadm primary r0
Create your filesystem on top of your DRBD device, for example:
mkfs.ext3 /dev/drbd_r0
Mount the file system and use it:
mount /dev/drbd_r0 /mnt/
If the install and configuration procedures worked as expected, you are ready to run a basic test of the DRBD functionality. This test also helps with understanding how the software works.
Test the DRBD service on jupiter.
Open a terminal console, then log in as
root.
Create a mount point on jupiter, such as
/srv/r0mount:
mkdir -p /srv/r0mount
Mount the drbd device:
mount -o rw /dev/drbd0 /srv/r0mount
Create a file from the primary node:
touch /srv/r0mount/from_node1
Test the DRBD service on venus.
Open a terminal console, then log in as
root.
Unmount the disk on jupiter:
umount /srv/r0mount
Downgrade the DRBD service on jupiter by typing the following command on jupiter:
drbdadm secondary r0
On venus, promote the DRBD service to primary:
drbdadm primary r0
On venus, check to see if venus is primary:
rcdrbd status
On venus, create a mount point such as
/srv/r0mount:
mkdir /srv/r0mount
On venus, mount the DRBD device:
mount -o rw /dev/drbd0 /srv/r0mount
Verify that the file you created on jupiter is viewable.
ls /srv/r0mount
The /srv/r0mount/from_node1 file should be
listed.
If the service is working on both nodes, the DRBD setup is complete.
Set up jupiter as the primary again.
Dismount the disk on venus by typing the following command on venus:
umount /srv/r0mount
Downgrade the DRBD service on venus by typing the following command on venus:
drbdadm secondary r0
On jupiter, promote the DRBD service to primary:
drbdadm primary r0
On jupiter, check to see if jupiter is primary:
rcdrbd status
To get the service to automatically start and fail over if the server has a problem, you can set up DRBD as a high availability service with OpenAIS. For information about installing and configuring OpenAIS for SUSE Linux Enterprise 11 see Part II, “Configuration and Administration”.
There are several ways to tune DRBD:
Use an external disk for your metadata. This speeds up your connection.
Create a udev rule to change the read ahead of the DRBD device. Save
the following line in the file
/etc/udev/rules.d/82-dm-ra.rules and change the
read_ahead_kb value to your workload:
ACTION=="add", KERNEL=="dm-*", ATTR{bdi/read_ahead_kb}="4100"This line works only, when you use LVM.
Activate bmbv on Linux software RAID systems. Use the following line in
the common disk section of your DRBD configuration, usually in
/etc/drbd.d/global_common.conf:
disk {
use-bmbv;
}The drbd setup involves many different components and problems may arise from different sources. The following sections cover several common scenarios and recommends various solutions.
If the initial drbd setup does not work as
expected, there is probably something wrong with your configuration.
To get information about the configuration:
Open a terminal console, then log in as root.
Test the configuration file by running drbdadm with the -d option. Enter the following command:
drbdadm -d adjust r0
In a dry run of the adjust option, drbdadm compares the actual configuration of the DRBD resource with your DRBD configuration file, but it does not execute the calls. Review the output to make sure you know the source and cause of any errors.
If there are errors in the /etc/drbd.d/* and
drbd.conf files, correct them before continuing.
If the partitions and settings are correct, run drbdadm again without the -d option.
drbdadm adjust r0
This applies the configuration file to the DRBD resource.
For DRBD, hostnames are case sensitive (Node0
would be a different host than node0).
If you have several network devices and want to use a dedicated network
device, the hostname will likely not resolve to the used IP address. In
this case, use the parameter disable-ip-verification.
If your system is unable to connect to the peer, this might be a problem with your local firewall. By default, DRBD uses the TCP port 7788 to access the other node. Make sure that this port is accessible on both nodes.
In cases when DRBD does not know which of the real devices holds the
latest data, it changes to a split brain condition. In this case, the
respective DRBD subsystems come up as secondary and do not connect to
each other. In this case, the following message is written to
/var/log/messages:
Split-Brain detected, dropping connection!
To resolve this situation, enter the following on the node which has data to be discarded:
drbdadm secondary r0 drbdadm -- --discard-my-data connect r0
On the node which has the latest data enter the following:
drbdadm connect r0
The following open source resources are available for DRBD:
The project home page http://www.drbd.org.
http://clusterlabs.org/wiki/DRBD_HowTo_1.0 by the Linux Pacemaker Cluster Stack Project.
The following man pages for DRBD are available in the distribution: drbd(8), drbddisk(8), drbdsetup(8), drbdsetup(8), drbdadm(8), drbd.conf(5).
Find a commented example configuration for DRBD at
/usr/share/doc/packages/drbd/drbd.conf