Linux cluster monitoring

Introduction

Linux cluster is a group of Linux computers or nodes, storage devices that work together and are managed as a single system. A traditional clustering configuration has two nodes that are connected to shared storage (typically a SAN). With Linux clustering, an application is run on one node, and clustering software is used to monitor its operation.

A Linux cluster provides faster processing speed, larger storage capacity, better data integrity, greater reliability and wider availability of resources.

Failover

Failover is a process. Whenever a primary system, network or a database fails or is abnormally terminated, then a Failover acts as a standby which helps resume these operations.

Failover Cluster

Failover cluster is a set of servers that work together to provide High Availability (HA) or Continuous availability (CA). As mentioned earlier, if one of the servers goes down another node in the cluster can take over its workload with minimal or no downtime. Some failover clusters use physical servers whereas others involve virtual machines (VMs).

CA clusters allow users to access and work on the services and applications without any incidence of timeouts (100% availability), in case of a server failure. HA clusters, on the other hand, may cause a short hiatus in the service, but system recovers automatically with minimum downtime and no data loss.

A cluster is a set of two or more nodes (servers) that transmit data for processing through cables or a dedicated secure network. Even load balancing, storage or concurrent/parallel processing is possible through other clustering technologies.

If you look at the above image, Node 1 and Node 2 have common shared storage. Whenever one node goes down, the other one will pick up from there. These two nodes have one virtual IP that all other clients connect to.

Let us take a look at the two failover clusters, namely High Availability Failover Clusters and Continuous Availability Failover Clusters.

High Availability Failover Clusters

In case of High AVailability Failover Clusters, a set of servers share data and resources in the system. All the nodes have access to the shared storage.

High availability clusters also include a monitoring connection that servers use to check the “heartbeat” or health of the other servers. At any time, at least one of the nodes in a cluster is active, while at least one is passive.

Continuous Availability Failover Clusters

This system consists of multiple systems that share a single copy of a computer’s operating system. Software commands issued by one system are also executed on the other systems. In case of a failover, the user can check critical data in a transaction.

There are a few Failover Cluster types like Linux Server Failover Cluster (WSFC), VMware Failover Clusters, SQL Server Failover Clusters, and Red Hat Linux Failover Clusters.

Pre-Requisites

1. OpsRamp Classic Gateway 11.0 and above (or) OpsRamp Cluster gateway
2. Ensure that “adapter integrations'' addon is enabled in client configuration. Once enabled you can see Linux Fail-over Cluster integration under Setup -> Integrations -> Adapter section.
3. Credentials : root / non-root privileges with a member of “haclient” group.
4. Cluster management: Pacemaker.
5. Accessibility: All nodes within a cluster should be accessible by a single credential set.
6. For non-root users: Update “~/.bashrc” file with “pcs” command path across all cluster nodes.
   Ex: export PATH=$PATH:/usr/sbin -> as a new line in ~/.bashrc file.

RGManager(Non-Pacemaker)

Pre-Requisites

1. Credentials : should provide access to both root and non-root users.
2. Cluster management: RGManager
3. Accessibility: All the nodes within a cluster should be accessible by a single credential set.
4. For non-root users: Update the following commands in “etc/sudoers” file to provide access for non-root users to execute these commands.

“/usr/sbin/cman_tool nodes,/usr/sbin/cman_tool status,/usr/sbin/clustat -l,/sbin/service cman status,/sbin/service rgmanager status,/sbin/service corosync status,/usr/sbin/dmidecode -s system-uuid,/bin/cat /sys/class/dmi/id/product_serial”

Note: Usually a linux cluster will be configured with a virtual-ip normally called as cluster-virtual-ip.We use this Ip for adding configurations during the installation of integration.

If the cluster-virtual-ip is not configured give the ip address of the reachable node associated with the cluster

Risks, Limitations & Assumptions

• Use pacemaker software to build local 3-node Linux failover cluster.
• Use cluster specific bash commands to fetch cluster related details. (ex: pcs status command)
• Each node’s /etc/hosts file should contain DNS details of all other nodes in a cluster.
• The credential should have read-only permission to fetch data out of pcs commands/pacemaker specific commands.
• All the nodes of a cluster should have the same credential-set.
• App can handle below 2 different types of alerts
  • UnknownHost Exception
  • JSch Exception
• Using metric as error type. In this case 2 different metrics mentioned in above point.
• App will send a critical alert for the very first poll when it encounters the above issues and the subsequent alert will be a recovery alert whenever the app doesn’t come across above issues in subsequent polls.
• App can’t control monitoring pause/resume actions based on above alerts.
• Platform has support to enable/disable configuration. So when a particular notification is generated then the customer can take action using UI/Commands.

Install the integration

1. From All Clients, select a client
2. Go to Setup > Integrations > Integrations.
3. From Available Integrations, select Adapter > Linux Fail-over Cluster. Then Install Linux Fail-over Cluster Integration pop-up appears.
   Note: Ensure that Adapter addon is enabled at client and partner levels.

   

4. Enter the following information:

Configure the integration

1. In CONFIGURATION section, click +Add.
2. On Create Adapter Configuration, enter
   • Name: Configuration name.
   • IP Address / Host Name: IP address or host name of the target.
   • Notification Alerts: Select TRUE or FALSE.
     Notes:
     • By default False is selected.
     • If you select TRUE, App will handle Critical/Recovery failure alert notifications for Connectivity, Authentication Exceptions.
3. From the Credentials section, select Custom, enter Username and Password.
   Note: These credentials are required to communicate with the target (cluster).
4. From the Resource Types & Metrics section, select the metrics and configure for availability and alert conditions, for Cluster & Server.
5. In the Discovery Schedule section, configure how frequently the discovery action should trigger. Select Recurrence Pattern to add one of the following patterns:
   
   • Minutes
   • Hourly
   • Daily
   • Weekly
   • Monthly
6. In the Monitoring Schedule section, configure how frequently the monitoring action should trigger.
7. Click Save.
   

   

   

   

   
   
   After saving the configuration, the resources are discovered and monitoring is done as specified in the configuration profile.
   The configuration is saved and displayed on the page.

   

   
   You can perform the actions manually, like Discovery, Monitoring or even Disable the configuration.
   The discovered resource(s) are displayed in the Infrastructure page under “Cluster”, with Native Resource Type as Linux Cluster
   

   

   
   Note: The cluster nodes are displayed under Components.

Note: You can select Pacemaker or RGManager from the Cluster Type drop-down.

Supported Metrics

Resource Type: Cluster

Pacemaker

RG Manager (Non-Pacemaker)

Risks, Limitations & Assumptions

• Component level threshold configuration is not possible.
• Resource level metric threshold customization and frequency setting are not possible.
• Usability issues in app configuration page while adding/editing.
• Optional configuration parameters cannot be defined.
• App upgrade is a manual process without version change.