video on demandArticle Listing 1 oct2006.tar
Sun Cluster 3.x Quorum Issue
Peter van der Weerd
Clustering software usually consists of a collection of scripts and binaries that unconfigure an interface, bring down an application, unmount some file systems, give away a group of disks, and reverse this procedure on some other machine. This goes for all Unix cluster solutions. Of course, there are some differences on different levels. Different vendors use different storage products and software to manage devices, have their own ideas about establishing and maintaining membership between the clustered machines, and so on.
So, if Unix clustering is so straightforward and common practice, why am I writing an article on it? Good question. In this article, I will not list all pros and cons of all different cluster products. I will describe one con of one Unix cluster product, elaborate on it a bit, and come up with a script that could help. Specifically, I will cover the quorum issue in Sun Microsystems' Sun Cluster 3.x product. Even though Sun, to my mind, has one of the most advanced cluster solutions in the field, there is a drawback. This drawback is the quorum device issue, or to be a little more exact, the ignored issue of losing the disk that is your quorum device.
Quorum Device
You may already know about the quorum device issue, but just to make sure, here's a short recap. Unix clusters all have some sort of heartbeat protocol that uses either a dedicated network or all available networks to communicate and establish "membership". Membership means that both, or in the case of more than two nodes, all nodes have to be aware of each other at all times. You cannot have one node that is unable to communicate with its cluster buddies and decides to run applications on its own.
A key concept in Unix clusters is high availability, but you do not want your applications to be doubly available. Access and changes to your data should at all times be controlled by one source, or at least monitored by one source to prevent two individual instances from changing your data without knowing about each other. Data integrity is considered more important than availability.
If, at any time, membership can no longer be guaranteed, an important decision should be made -- which part of "broken" cluster should continue, and which part should instantly die to avoid messed up data?
This is where the quorum device comes in. To avoid a so-called partitioned cluster, both take a run for the quorum device, reserve it, and start counting the number of votes they have. For example, in a two-node cluster, the number of votes present when membership is ok, is three. One vote for each node, and one vote for the quorum device. This makes a total of three. When membership is gone, one node will succeed in reserving the quorum device, the other node will not succeed. This will leave the slower node with only one vote. One vote is less than half of all possible votes. The faster node has two votes, which is more than half of the votes. It's easy. If one node has more than half of the votes, it will run all applications; if a node has only half or even less than half of the possible votes, it will panic.
This reservation process differs according to product. Since this article deals with the Sun Microsystems solution, I will concentrate on their method. Sun Cluster quorum disk reservation is a two-step process that begins with SCSI-reservation. But this SCSI-reservation is only valid and ruling for as long as the node is up and running. If the "surviving" node were to be reset, the SCSI-reservation would be gone for a short while. At this point, the other node, being panicked and all, would be sitting there, waiting for the opportunity to be able to reserve the quorum disk. This is not what Sun Cluster wants. Sun Cluster says "the last node to leave the cluster should be the first one to join", meaning that Sun Cluster wants to prevent a node that doesn't have the most recent configuration from starting a cluster on its own. So, SCSI-reservation is not enough.
Reservation Key
Each node joining the cluster puts its reservation key on the quorum disk. Putting reservation keys on disks is supported in SCSI-III and is called "Persistent Group Reservation". Once membership is established, all nodes will have their keys on the quorum disk. In the case of heartbeat/membership loss, the keys of the nodes that panic are removed from the quorum disk. If, at any time, a node that panicked out of the cluster tries to reserve the quorum disk, it will see that its key is not on the disk and politely inform you that the disk is owned by someone else. It will wait until the other nodes want to establish membership and do nothing. This is how Sun Cluster prevents older configurations from starting a cluster.
More than Half of the Votes
What if, in a three-node cluster, two nodes die? Would you want the surviving node to panic? Probably not. But in the scenario of needing more than half of the possible votes, the node would definitely die. In a three-node cluster, there are four possible votes: three nodes and a quorum disk. When two of the three nodes decide to leave, that one remaining node is left with only two out of four votes, which is not more than half. Solution: the quorum disk vote is the number of nodes minus one. In a three-node cluster, the total number of votes would then be five. Now, if two nodes fail, the surviving node would have its own vote plus the two votes of that disk, which amounts to three, which is more than half. Additionally, it removes the other nodes' reservation keys from the disk to make sure that it is boss at all times.
What If the Quorum Disk Dies?
Now, here we have an issue. Some call this a single point of failure, but it is not. If the quorum disk fails, nothing happens. If you set up your storage correctly, your data is mirrored, so losing a single disk will not affect your data. Where vote counts are concerned, you have no problem either: losing the quorum disk will not leave your nodes with half or less than half of the votes. A single point of failure it is definitely not. So what is the problem? The problem is that there is no daemon that monitors your quorum disk and creates a new one when the old one dies. If your quorum disk is dead and you reboot a node, the other node will not have enough votes left and will panic. This is not what you want. You want a new quorum device the minute the original one dies, because the quorum functionality is not something that can be mirrored.
The Solution
Make sure that you do not need a quorum disk. Sun Cluster has recently decided to let you create heartbeat networks the way you want to. So you can use every interface on your cluster nodes to transport heartbeat. It is not very likely that you will lose all network interfaces at the same time. And if you do lose all network connection, what's the availability then? No network means no clients. Unfortunately, Sun Cluster will not let you build a two-node cluster without a quorum disk at this time.
A Cluster
That was the theory bit. Now, let's have a look at a real example. Assume we have a two-node cluster with one quorum disk. But, which disk is the quorum disk? The average customer does not set up a Sun Cluster; Sun sets it up for them. Imagine then that you are the average customer and want to know which disk is your quorum disk. Do the following:
node1#/usr/cluster/bin/scstat -q
-- Quorum Summary --
Quorum votes possible: 3
Quorum votes needed: 2
Quorum votes present: 3
-- Quorum Votes by Node --
Node Name Present Possible Status
--------- ------- -------- ------
Node votes: node1 1 1 Online
Node votes: node2 1 1 Online
-- Quorum Votes by Device --
Device Name Present Possible Status
----------- ------- -------- ------
Device votes: /dev/did/rdsk/d4s2 1 1 Online
Don't let the odd name "d4s2" confuse you. This is not a Solaris Volume Manager device name. It is a so-called "Did" name. In a Sun Cluster environment, all devices have a unique name. So, device d4 is the same device on node1 as it is on node2. It may well be that the original device file name on node1 is c1t3d0 where it is c2t3d0 on node2. This way the controller information on both nodes has become irrelevant.
It is advisable to have data on your quorum disk. Make sure your quorum disk is part of a mirror that is used by a clustered application. Sun Cluster will only detect that a disk is broken when it cannot be accessed anymore. SCSI errors will not occur if you do not access the disk. A quorum device that is part of an active mirror in a clustered environment is a good quorum device. You will see when it is broken because the scstat command will tell you that it is offline.
A quorum device that is offline is a failure, and the vote count is prone to disaster. One less vote (a node that dies) and your applications will not be available anymore because the other node will panic and not be able to achieve membership for as long as the dead node remains dead. You will have to create a new quorum disk before that happens. To do so, you must first choose a new device then remove the old one. Obviously, you cannot remove the only quorum device you have. You might compromise quorum, even though the device is broken. So, just go with the flow and create a new quorum disk once you see that the present quorum disk is offline.
Broken Quorum Disk
We break the disk...it is broken:
node1#/usr/cluster/bin/scstat -q
(output skipped)
-- Quorum Votes by Device --
Device Name Present Possible Status
----------- ------- -------- ------
Device votes: /dev/did/rdsk/d4s2 0 1 Offline
As you can see, the disk is offline and obviously not available to function as a tie breaker in the case of heartbeat loss. We double-check by running the Sun Cluster diskpath monitor "scdpm" and letting it collect all failed devices:
scdpm -p all|grep Fail
node1:/dev/did/rdsk/d4 Fail
node2:/dev/did/rdsk/d4 Fail
Select New Quorum Disk
Before you can select a new quorum disk, you must determine which disks are available as quorum disks. The new quorum disk must be a disk that is accessible by both cluster nodes. It should be a shared disk:
scdidadm -L
1 node2:/dev/rdsk/c0t2d0 /dev/did/rdsk/d1
2 node2:/dev/rdsk/c0t0d0 /dev/did/rdsk/d2
3 node2:/dev/rdsk/c1t9d0 /dev/did/rdsk/d3
3 node1:/dev/rdsk/c1t9d0 /dev/did/rdsk/d3
4 node2:/dev/rdsk/c1t10d0 /dev/did/rdsk/d4
4 node1:/dev/rdsk/c1t10d0 /dev/did/rdsk/d4
5 node2:/dev/rdsk/c1t11d0 /dev/did/rdsk/d5
5 node1:/dev/rdsk/c1t11d0 /dev/did/rdsk/d5
6 node2:/dev/rdsk/c1t12d0 /dev/did/rdsk/d6
6 node1:/dev/rdsk/c1t12d0 /dev/did/rdsk/d6
7 node2:/dev/rdsk/c1t13d0 /dev/did/rdsk/d7
7 node1:/dev/rdsk/c1t13d0 /dev/did/rdsk/d7
8 node2:/dev/rdsk/c1t14d0 /dev/did/rdsk/d8
8 node1:/dev/rdsk/c1t14d0 /dev/did/rdsk/d8
9 node1:/dev/rdsk/c0t0d0 /dev/did/rdsk/d9
This list shows all Device IDs as well as the official device file name of all disks on both nodes. All drives that appear twice in the list are shared devices. This means you can pick any of these disks as long as the diskpath monitor thinks they are ok:
Drive d4 is broken, so we pick the next in the list if it is still ok:
scdpm -l all:all
(output skipped)
node1:/dev/did/rdsk/d4 Fail
node1:/dev/did/rdsk/d5 Ok
node1:/dev/did/rdsk/d6 Ok
(output skipped)
Obviously, drive d5 is ok. So, we select that one:
scconf -a -q globaldev=d5
We check whether d5 is now a valid quorum disk:
scstat -q
(output skipped)
-- Quorum Votes by Device --
Device Name Present Possible Status
----------- ------- -------- ------
Device votes: /dev/did/rdsk/d4s2 0 1 Offline
Device votes: /dev/did/rdsk/d5s2 1 1 Online
(output skipped)
Now, we can remove the old quorum device:
scconf -r -q globaldev=d4
At this point, we are back to where we were before the disk broke. Until this new quorum disk breaks, we can rest assured that in the case of heartbeat loss, one node will survive by grabbing the quorum disk and establishing maximum vote. Unfortunately, we cannot go around checking the health of the quorum disk all the time. So, it might be an idea to write a script to do this for us (see Listing 1).
This script will check whether your quorum disk is still okay, every 5 minutes. If the quorum disk is not okay, it will take the first shared disk from a list, until all disks are gone. And, if all disks are gone, you will very likely have other things to worry about than your quorum disk.
Conclusion
Since virtually all Unix cluster solutions use a voting mechanism in combination with quorum disks, there is probably something about quorum disks that makes them either popular or inevitable. To avoid disappointments when such a disk fails, it is good to take precautions. I hope this article will help a bit.
Peter van der Weerd works as a freelance Solaris, HP-UX, and Linux trainer in Europe.