I've spent the past 6 hours trying to configure a cluster of Linux machines. Before I talk about the current sad state of affairs, let me give you a brief history of Linux clustering.
In the beginning, there was Heartbeat. Released in 1998, it was a very simple clustering system. It only supported two nodes, and it managed a simple set of "resources" that could be running on either machine. It wasn't very powerful, but it also just worked.
Meanwhile Sistina (later RedHat) was creating cman, their own clustering system. cman was so bad that it was completely replaced with cman 2, which was a somewhat modular system where cman was a thin layer of configuration utilities that controlled the OpenAIS cluster library.
Heartbeat meanwhile had developed into Heartbeat 2, with support for more than 2 nodes and much more complete support for resource management. But the big feature it doesn't have is distributed locking support, which is a system for allowing synchronization between all the nodes in the cluster. Thus, I started looking at cman.
Around this time, Heartbeat development started to stagnate. So, some of the developers split heartbeat into a new tool called "pacemaker" that just manages the resources in the cluster and a new heartbeat that manages cluster membership. Then, there was an unholy matrimony of pacemaker with openais, where it gained support for using either heartbeat or openais as its cluster membership library.
Then things got even more interesting when openais decided to split into two modules, a new cluster membership library called "corosync" and a cluster utility library called (the new) "openais". Pacemaker doesn't require any of the utilities from the new openais, but it still calls corosync "openais" when it is using corosync.
(While doing research for this blog post, I also discovered that Heartbeat development has picked up again and they're about to do a release of Heartbeat 3.0.)
So, it is in this confusing and incestuous world of clustering software that I have tried to set up a clvm cluster. clvm is a piece of software that allows you to share a single physical disk (in our case, an iSCSI-connected RAID) between multiple computers that are simultaneously accessing it. clvm delightfully supports four different clustering systems: cman, openais, corosync, and gulm. We had been running clvm with cman, but it is extremely prone to deadlocks, which is particularly bad because cman's distributed locking is implemented in the Linux kernel, and there is no way to reset the state of the locks without completely rebooting your machine.
evanbro and I have been working on getting clvm, openais, corosync, and pacemaker running on Ubuntu Hardy for Invirt. After backporting the current versions of all of the software, we discovered that none of the documentation out there matched the current state of the software (it almost all assumes that "openais" is a single, unsplit program), and worse, most of the software isn't documented at all.The corosync documentation says that you can write a simple configuration file and start "corosync" on all the nodes, and they will all find each other and form a cluster. Hah. While the log messages that get to syslog certainly indicate that the nodes find each other, all the client utilities (such as corosync-cfgtool, or more importantly, clvmd) fail with error code 6 ("SA_AIS_ERR_TRY_AGAIN"). There is, of course, no documentation anywhere that tells you what you should do when you encounter this error. After pounding on it for a while, including both enabling and disabling the openais plugins in corosync, we were able to get clvmd to start and configure itself (we did this by disabling openais but enabling the openais_lck plugin for distributed locking), but performing any operation simultaneously in two places immediately and repeatedly caused a deadlock.
Did I mention that pacemaker, despite running on not its original clustering software, just worked the minute we turned it on on openais?
Ugh. As keithw would say, "I bemoan the state of Computer Science".
