Veritas cluster monitoring

Introduction

Veritas Cluster Server is a high-availability cluster software for unix, linux and Microsoft windows computer systems. It is an availability clustering solution that manages the availability of application services, such as Db2 database, by enabling application failover. The state of each individual cluster node and its associated software services are regularly monitored.

Failover

Failover is a process. Whenever a primary system, network or a database fails or is abnormally terminated, then a Failover acts as a standby which helps resume these operations.

Failover Cluster

Failover cluster is a set of servers that work together to provide High Availability (HA) or Continuous availability (CA). As mentioned earlier, if one of the servers goes down another node in the cluster can take over its workload with minimal or no downtime. Some failover clusters use physical servers whereas others involve virtual machines (VMs).

CA clusters allow users to access and work on the services and applications without any incidence of timeouts (100% availability), in case of a server failure. HA clusters, on the other hand, may cause a short hiatus in the service, but system recovers automatically with minimum downtime and no data loss.

A cluster is a set of two or more nodes (servers) that transmit data for processing through cables or a dedicated secure network. Even load balancing, storage or concurrent/parallel processing is possible through other clustering technologies.

The above image shows an application that runs on a primary or master server. A dedicated redundant server is present to take over on any failure. The redundant server is not configured to perform any other functions. The redundant server is on stand-by with full performance capability.

Veritas Cluster Server (VCS)

Veritas Cluster Server is a high-availability cluster software for Unix, Linux and Microsoft Windows computer systems.

Veritas Cluster Server connects multiple, independent systems into a management framework for increased availability. Each system or node runs its own operating system and cooperates at the software level to form a cluster. VCS links commodity hardware with intelligent software to provide application failover and control. So when a node or a monitored application fails, other nodes take over and bring up services elsewhere in the cluster.

How VCS detects failure

VCS detects failure by issuing specific commands or scripts to monitor the overall health of an application. VCS also determines the health of underlying resources supporting the application, such as network interfaces or file systems.

VCS uses a redundant network heartbeat to differentiate between the loss of a system and the loss of communication between systems.

How VCS ensures application availability

When VCS detects an node or application failure, VCS brings application services up on a different node in a cluster. VCS virtualizes IP addresses and system names, so client systems continue to access the application without any interruption.

Integration is done with Veritas Cluster using SSH and shell script.

Pre-requisites

• OpsRamp Classic Gateway 10.0 and above.

• Ensure that “adapter integrations” add-on is enabled in client configuration. Once enabled you can see Veritas Cluster integration under Setup -> Integrations -> Adapter section.

• It is mandatory to provide administrator (or) operator level VCS credentials in input configuration. VCS credentials are required to fetch veritas_cluster_group_State, veritas_cluster_group_Status, veritas_cluster_node_State, veritas_cluster_resource_State, veritas_cluster_resource_Status, veritas_cluster_group_failover_Status metric details.

• In order to get Additional metrics ( veritas_cluster_lltLinks_State, veritas_cluster_lltInterface_Status), you need to provide permissions for the non-root SSH user. So, you need to make an entry as below in “/etc/sudoers”.

      cat /etc/sudoers
  
      ## Allow root to run any commands anywhere 
      root    ALL=(ALL)     ALL
      content    ALL=NOPASSWD:/usr/sbin/lltstat -n,/usr/sbin/lltstat -nvv configured
  
      

• In case root SSH Credentials are provided, there is no need to configure the second prerequisite.

• All nodes IP Address should be publicly available or it should be configured in /etc/hosts of all the existing nodes in the Veritas Cluster.

      cat /etc/hosts
      [root@centos-node1 ~]# cat /etc/hosts
      127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
      ::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
      172.26.1.25 centos-node1
      172.26.1.26 centos-node2
  
      

• non-root user running VCS commands must have a home directory on the system on which the VCS commands will be run.

      pwd
      [content@centos-node1 ~]$ pwd
      /home/content
  
      

Resource Hierarchy

    Veritas Cluster
                → Server

Default monitoring configurations

Veritas Cluster application has default Global Device Management Policies, Global Templates, Global Monitors and Global metrics in OpsRamp. Users can customize these default monitoring configurations as per their business use cases by cloning respective global templates and global Device Management Policies. OpsRamp recommends doing this activity before installing the application to avoid noise alerts and data.

1. Default Global Device Management Policies

   OpsRamp has a Global Device Management Policy for each Native Type of VERITAS-CLUSTER. We can find those Device Management Policies at Setup -> Resources -> Device Management Policies, search with suggested names in global scope. Each Device Management Policy follows below naming convention:

   {appName nativeType - version}

   Ex: veritas-cluster Veritas Cluster - 1 (i.e, appName = veritas-cluster, nativeType = Veritas Cluster, version = 1)

2. Default Global Templates

   OpsRamp has a Global template for each Native Type of VERITAS-CLUSTER. We can find those templates at Setup -> Monitoring -> Templates, search with suggested names in global scope. Each template follows below naming convention:

{ appName nativeType 'Template' - version}

Ex: veritas-cluster Veritas Cluster Template - 1 (i.e, appName = veritas-cluster, nativeType = Veritas Cluster, version = 1)

3. Default Global Monitors

   OpsRamp has a Global Monitors for each Native Type which has monitoring support. You can find those monitors at Setup -> Monitoring -> Monitors, search with suggested names in global scope. Each Monitors follows below naming convention:

   {monitorKey appName nativeType - version}

   Ex: Veritas Cluster Monitor veritas-cluster Veritas Cluster 1 (i.e, monitorKey = Veritas Cluster Monitor, appName = veritas-cluster, nativeType = Veritas Cluster, version = 1)

Configure and install the integration

1. From All Clients, select a client.
2. Go to Setup > Integrations and Apps.
3. Click Manage Apps.
   Notes:
   • If there are already installed applications, it will redirect to the INSTALLED APPS page where all the installed applications are displayed.
   • If there are no installed applications, it will navigate to the ADD APP page.

4. Click + ADD on the INSTALLED INTEGRATIONS page. The AVAILABLE INTEGRATIONS AND APPS page displays all the available applications along with the newly created application with the version.
   Note: You can even search for the application using the search option available. Also you can use the All Categories option to search.

5. Click ADD in the Veritas Cluster application.
6. In the Configuration page, click + ADD. The Add Configuration page appears.
7. Enter the below mentioned BASIC INFORMATION:

Note: Select App Failure Notifications: if turned on, you will be notified in case of an application failure that is, Connectivity Exception, Authentication Exception.

8. Select the below mentioned Custom Attribute:

Note: The custom attribute that you add here will be assigned to all the resources that are created by the integration. You can add a maximum of five custom attributes (key and value pair).

9. In the RESOURCE TYPE section, select:
   • ALL: All the existing and future resources will be discovered.
   • SELECT: You can select one or multiple resources to be discovered.
10. In the DISCOVERY SCHEDULE section, select Recurrence Pattern to add one of the following patterns:
    • Minutes
    • Hourly
    • Daily
    • Weekly
    • Monthly
11. Click ADD.

12. Now the configuration is saved and displayed on the configurations page after you save it. From the same page, you may Edit and Remove the created configuration..
13. Click Next.
14. In the Installation page, select an existing registered profile, and click FINISH.

The application is now installed and displayed on the INSTALLED INTEGRATION page. Use the search field to find the installed application.

Modify the Configuration

• See Modify an Installed Integration or Application article.

  Note: Select the Veritas Cluster application.

View the Veritas Cluster details

The discovered resource(s) are displayed in Infrastructure > Resources > Cluster, with Native Resource Type as Veritas Cluster. You can navigate to the Attributes tab to view the discovery details, and the Metrics tab to view the metric details for Veritas Cluster.

View Resource Metrics

To confirm Veritas Cluster monitoring, review the following:

• Metric graphs: A graph is plotted for each metric that is enabled in the configuration.
• Alerts: Alerts are generated for metrics that are configured as defined for integration.

Supported Metrics

Resource Type: Cluster

Resource Type: Server

Risks, Limitations & Assumptions

• As of now supporting only Linux based Veritas failover cluster.
• When we add two configurations with same end device details (like IP, credentials), we might observe gaps in the graphs due to internal VCS login and logouts parallely on the same device.
• Component level threshold configuration is not possible.
• Resource level metric threshold customization and frequency setting are not possible.
• Usability issues in app configuration page while adding/editing.
• Optional configuration parameters cannot be defined.
• Application upgrade is a manual process without version change.