Pacemaker PCS Cookbook for Managing Linux Clusters
Srini Rao
Pacemaker and pcs on NFS servers managing cluster resources
This cookbook mostly use various commands to show how to manage pacemaker cluster resources.
Primarly five types of resources configured in our pacemaker configurations:
vip
Virtual ip resource managed by pacemaker
luksOpenLvm
Luks LVM open resource for lvm that has setup with Luks
Filesystem
xfs file system resource that need to be monitored
nfs-server
nfs server resource
LVM
Logical volume manager resource
Before get started, here is the cluster current status.
[A3][root@nfs3 ~]# pcs status
Cluster name: nfs
Last updated: Fri Sep 16 12:44:41 2016
Last change: Tue Aug 16 17:59:49
2016 by root via crmd on nfs4
Stack: cman
Current DC: nfs3 (version 1.1.14-8.el6-70404b0) - partition with quorum
2 nodes and 43 resources configured
Online: [ nfs3 nfs4 ]
Full list of resources:
Resource Group: ha-nfsserver
vip
(ocf::heartbeat:IPaddr2):
Started nfs3
luksOpen-3600a098038303753723f494743616663p1_lvm (ocf::heartbeat:luksOpenLvm):
Started nfs3
luksOpen-3600a098038303753723f494743616669p1_lvm (ocf::heartbeat:luksOpenLvm):
Started nfs3
lvm_iscsi1
(ocf::heartbeat:LVM):
Started nfs3
fs-vg_iscsi1-lv_iscsi1
(ocf::heartbeat:Filesystem): Started nfs3
pseudo-vg_iscsi1-lv_iscsi1
(ocf::heartbeat:Filesystem): Started nfs3
luksOpen-3600a098038303753723f49474361666cp1_lvm (ocf::heartbeat:luksOpenLvm):
Started nfs3
luksOpen-3600a098038303753723f49474361662fp1_lvm (ocf::heartbeat:luksOpenLvm):
Started nfs3
lvm_iscsi2
(ocf::heartbeat:LVM):
Started nfs3
fs-vg_iscsi2-lv_iscsi2
(ocf::heartbeat:Filesystem): Started nfs3
pseudo-vg_iscsi2-lv_iscsi2
(ocf::heartbeat:Filesystem): Started nfs3
luksOpen-3600a098038303753723f494743616661p1_lvm (ocf::heartbeat:luksOpenLvm):
Started nfs3
luksOpen-3600a098038303753723f494743616662p1_lvm (ocf::heartbeat:luksOpenLvm):
Started nfs3
lvm_iscsi3
(ocf::heartbeat:LVM):
Started nfs3
fs-vg_iscsi3-lv_iscsi3
(ocf::heartbeat:Filesystem): Started nfs3
pseudo-vg_iscsi3-lv_iscsi3
(ocf::heartbeat:Filesystem): Started nfs3
luksOpen-3600a098038303753723f494743616664p1_lvm (ocf::heartbeat:luksOpenLvm):
Started nfs3
luksOpen-3600a098038303753723f494743616665p1_lvm (ocf::heartbeat:luksOpenLvm):
Pacemaker PCS Cookbook for Managing Linux Clusters
Srini Rao
Started nfs3
lvm_iscsi4
(ocf::heartbeat:LVM):
Started nfs3
fs-vg_iscsi4-lv_iscsi4
(ocf::heartbeat:Filesystem): Started nfs3
pseudo-vg_iscsi4-lv_iscsi4
(ocf::heartbeat:Filesystem): Started nfs3
Create a resource
To create a new resource
#pcs resource create fs-vg_iscsi7-lv_iscsi7 ocf:heartbeat:Filesystem
device=/dev/mapper/vg_iscsi7-lv_iscsi7 directory=/iscsi7 fstype=xfs
fast_stop="no" force_unmount="safe" op stop on-fail=stop timeout=200 op monitor onfail=stop timeout=200 OCF_CHECK_LEVEL=10
Delete a resource
To delete a resource
[A3][root@nfs3 ~]# pcs delete resource fs-vg_iscsi7-lv_iscsi7
Show the resource
[A3][root@nfs3 ~]# pcs resource show fs-vg_iscsi7-lv_iscsi7
Resource: fs-vg_iscsi7-lv_iscsi7 (class=ocf provider=heartbeat type=Filesystem)
Attributes: device=/dev/mapper/vg_iscsi7-lv_iscsi7 directory=/iscsi7 fstype=xfs
Operations: monitor interval=30s (fs-vg_iscsi7-lv_iscsi7-monitor-interval-30s)
start interval=0s timeout=20s (fs-vg_iscsi7-lv_iscsi7-start-interval-0s)
stop interval=0s timeout=20s (fs-vg_iscsi7-lv_iscsi7-stop-interval-0s)
Manually Moving Resources Around the Cluster
To mover a resource to any other nodes
[A3][root@nfs3 ~]# pcs resource move fs-vg_iscsi7-lv_iscsi7
To move a resource to a nfs4 (other node)
[A3][root@nfs3 ~]#
pcs resource move fs-vg_iscsi7-lv_iscsi7 nfs4
To mover a resource to its origional node nfs3
[A3][root@nfs3 ~]#
pcs resource clear fs-vg_iscsi7-lv_iscsi7
Note: You can also use the move command to move resource back to its origional node, however, it
won't clear the constraint that move command generated. Thus, it's better to use 'resource clear' to move
back to its normal status.
Pacemaker PCS Cookbook for Managing Linux Clusters
Srini Rao
Note2: When moving a resource, any other resources that has constraint to the resource to be moved
will get moved too.
Moving Resources Due to Failure
By default, there is no threshold defined, so pacemaker will move resource to other nodes whenever it
fails. To define a threadhold to 3,run
# pcs resource meta fs-vg_iscsi7-lv_iscsi7 migration-threshold=3
The command above defines the resource fs-vg_iscsi7-lv_iscsi7 to move after 3 failures.
Note: after a resource move due to failure, it will not run on the origional node until the failcount is
reset, or failure timeout reached.
To set all resource threshold to 3, so all resources in the cluster will move after 3 times fails
# pcs resource defaults migration-threshold=10
To show current failcount
[A3][root@nfs3 ~]# pcs resource failcount show luksOpen3600a098038303753723f49474361665ap1_lvm
No failcounts for luksOpen-3600a098038303753723f49474361665ap1_lvm
To clear the failcount, run
[A3][root@nfs3 ~]#resource failcount reset luksOpen3600a098038303753723f49474361665ap1_lvm
Note: the threshold only works when in normal mode, not for start and stop operation.
Start failures cause the failcount to be set to INFINITY and thus always cause the resource to move
immediately.
Stop failures are slightly different and crucial. If a resource fails to stop and STONITH is enabled, then
the cluster will fence the node in order to be able to start the resource elsewhere. If STONITH is not
enabled, then the cluster has no way to continue and will not try to start the resource elsewhere, but will
try to stop it again after the failure timeout.
Disabling, and Banning Cluster Resources
To stop a nfs-server resource on a node and don't want it get started on other nodes.
#pcs resource disabled nfs-server
To start a nfs-server resource on a node and back the resource to normal state
#pcs resource enabled nfs-server
To ban a resource on a node
#pcs resource ban nfs-server nfs4
If no node specified, it's banned on current node
Pacemaker PCS Cookbook for Managing Linux Clusters
Srini Rao
To remove the ban constraint, run
#pcs resource clear nfs-server
To debug a resrouce start
#pcs resource debug-start nfs-server
Disabling a Monitor Operations
To disable monitor operation for a resource
#pcs resource update pseudo-vg_iscsi7-lv_iscsi7 op monitor enabled="false"
To enable monitor operation for a resource
#pcs resource update pseudo-vg_iscsi7-lv_iscsi7 op monitor enabled="true"
To permenant stop a resource monitoring , just delete the monitoring
Managed Resources
To set a resource to unmanaged state, compare to the resource deletion, unmanaged resource is still in
the cluster configuration, but pacemaker doesn't manage it.
#pcs resource unmanage nfs-server
To set a resource to managed state
#pcs resource manage nfs-server
