HDFS DataNode

A DataNode stores data in HadoopFileSystem. A functional filesystem has more than one DataNode, with data replicated across them.

Prerequisite

For Virtual Machines, install the Linux Agent.

Configuring the credentials

Configure the credentials in the directory /opt/opsramp/agent/conf/app.d/creds.yaml

hdfs-datanode:
- name: hdfs-datanode
  user: <username>
  pwd: <Password>
  encoding-type: plain
  labels:
    key1: val1
    key2: val2

Configuring the application

Virtual machine

Configure the application in the directory /opt/opsramp/agent/conf/app/discovery/auto-detection.yaml

- name: hdfs-datanode
  instance-checks:
    process-check:
      - datanode
    port-check:
      - 50075
      - 9864
    uri-check:
      - "jmx?qry=Hadoop:service=DataNode,name=FSDatasetState*"
      - "jmx?qry=java.lang:type=OperatingSystem"

Docker environment

Configure the application in the directory /opt/opsramp/agent/conf/app/discovery/auto-container-detection.yaml

- name: hdfs-datanode
  container-checks:
    image-check:
      - datanode
    port-check:
      - 50075
      - 9864

Kubernetes environment

Configure the application in config.yaml

- name: hdfs-datanode
  container-checks:
    image-check:
      - datanode
    port-check:
      - 50075
      - 9864

Validate

Go to Resources under the Infrastructure tab to check if your resources are onboarded and the metrics are collected.

Supported metrics

OpsRamp Metric	Metric Display Name	Unit
hdfs_datanode_dfs_remaining The remaining disk space left	Dfs Remaining	bytes
hdfs_datanode_dfs_remaining_percent The remaining disk space left	Dfs Remaining Percent	percent
hdfs_datanode_dfs_capacity Capacity of the disk	Dfs Capacity	bytes
hdfs_datanode_dfs_used Disk usage	Dfs Used	bytes
hdfs_datanode_dfs_used_percent Disk usage	Dfs Used Percent	percent
hdfs_datanode_cache_capacity Capacity of the cache	Cache Capacity	bytes
hdfs_datanode_cache_used Cache used	Cache Used	bytes
hdfs_datanode_num_failed_volumes Number of failed volumes	Num Failed Volumes
hdfs_datanode_last_volume_failure_date Date/time of the last volume failure since epoch	Last Volume Failure Date	milliseconds
hdfs_datanode_estimated_capacity_lost_total Estimated capacity lost	Estimated Capacity Lost Total	bytes
hdfs_datanode_num_blocks_cached Number of blocks cached	Num Blocks Cached	Blocks
hdfs_datanode_num_blocks_failed_to_cache Number of blocks that failed to cache	Num Blocks Failed To Cache	Blocks
hdfs_datanode_num_blocks_failed_to_uncache Number of failed blocks to remove from cache	Num Blocks Failed To Uncache	Blocks
hdfs_datanode_process_cpu_load CPU Load of the process	Process Cpu Load	Load