Apache Kafka is an open-source stream-processing software platform developed by the Apache Software Foundation, written in Scala and Java. The project aims to provide a unified, high-throughput, low-latency platform for handling real-time data feeds.
Prerequisite
- Install Kafka and configure the JMX Port using the environment variable.
export JMX_PORT=5199 && ./bin/kafka-server-start.sh -daemon config/server.properties
- For Virtual Machines, install the Linux Agent.
Configuring the credentials
Configure the credentials in the directory /opt/opsramp/agent/conf/app.d/creds.yaml
kafka:
- name: kafka
user: <username>
pwd: <Password>
encoding-type: plain
labels:
key1: val1
key2: val2
Configuring the application
Virtual machine
Configure the application in the directory /opt/opsramp/agent/conf/app/discovery/auto-detection.yaml
- name: kafka
instance-checks:
service-check:
- kafka
process-check:
- kafka
port-check:
- 9092
mon-type: "jmx"
misc:
jmx-port: "5199"
Docker environment
Configure the application in the directory /opt/opsramp/agent/conf/app/discovery/auto-container-detection.yaml
- name: kafka
container-checks:
image-check:
- kafka
port-check:
- 9092
mon-type: "jmx"
misc:
jmx-port: "5199"
Kubernetes environment
Configure the application in config.yaml
- name: kafka
container-checks:
image-check:
- kafka
port-check:
- 9092
mon-type: "jmx"
misc:
jmx-port: "5199"
Validate
Go to Resources under the Infrastructure tab to check if your resources are onboarded and the metrics are collected.
Supported metrics
| OpsRamp Metric | Metric Display Name | Unit |
|---|---|---|
| kafka_channel_request_queue_size Number of queued requests | Request Queue Size | Requests |
| kafka_channel_response_queue_size Number of queued responses | Response Queue Size | Responses |
| kafka_controller_active_controller_count Number of active controller on broker | Active Controller Count | Controllers |
| kafka_fetch_requests_delayed Requests delayed in the fetch purgatory | Fetch Delayed Requests | Requests |
| kafka_fetch_requests_purgatory_delayed Requests delayed in the fetch purgatory | Fetch Delayed Requests | Requests |
| kafka_fetch_requests_purgatory_size Requests waiting in the fetch purgatory | Fetch Purgatory Size | Requests |
| kafka_fetch_requests_size Requests waiting in the fetch purgatory | Fetch Purgatory Size | Requests |
| kafka_jvm_gc_collection_count Number of garbage objects collected | Kafka JVM GC collection_count | Objects collected |
| kafka_jvm_gc_collection_time Time taken for collection of the garbage objects | Kafka JVM GC collection_time | Seconds |
| kafka_jvm_mem_heap_committed Heap memory committed for the server | Kafka JVM Mem heap_committed | megabytes |
| kafka_jvm_mem_heap_used Heap memory usage of the server | Kafka JVM Mem heap_used | megabytes |
| kafka_jvm_mem_non_heap_committed Non-heap memory committed for the server | Kafka JVM Mem non_heap_committed | megabytes |
| kafka_jvm_mem_non_heap_used Non-heap memory usage of the server | Kafka JVM Mem non_heap_used | megabytes |
| kafka_jvm_open_fds Number of Open file descriptors of the server | Kafka JVM OpenFDs | Open FDs |
| kafka_jvm_threads Number of threads | Kafka JVM Threads | Threads |
| kafka_jvm_uptime Uptime of the server | Uptime | Minutes |
| kafka_log_flush_rate Log flush rate and time | LogFlush Rate And Time | Log flushes |
| kafka_metrics_controlled_shutdown_req_queue_time Refers to the time for which the request is waiting in queue | Controlled Shutdown Request Queue Time | Request time,ms |
| kafka_metrics_controlled_shutdown_requests Request rate | Controlled Shutdown Requests | Requests / sec |
| kafka_metrics_controlled_shutdown_resp_send_time Time to send the response | Controlled Shutdown Response Send Time | Request time,ms |
| kafka_metrics_controlled_shutdown_total_time Request total time | Controlled Shutdown Total Time | Total time,ms |
| kafka_metrics_fetch_consumer_req_queue_time Refers to the time for which the request is waiting in the request queue | Fetch Consumer Request Queue Time | milliseconds |
| kafka_metrics_fetch_consumer_resp_send_time Time to send the response | Fetch Consumer Response Send Time | milliseconds |
| kafka_metrics_fetch_consumer_total_time Request total time | Fetch Consumer Total Time | milliseconds |
| kafka_metrics_fetch_follower_local_time Time taken to process the request at the leader | Fetch Follower Local Time | milliseconds |
| kafka_metrics_fetch_follower_resp_queue_time Time taken by the request to wait in the response queue | Fetch Follower Response Queue Time | milliseconds |
| kafka_metrics_fetch_requests Request rate | Fetch Requests | Requests / second |
| kafka_metrics_leader_isr_local_time Time taken to process the request at the leader | Leader And Isr Local Time | milliseconds |
| kafka_metrics_leader_isr_remote_time Time taken by the request to wait for the follower | Leader And Isr Remote Time | milliseconds |
| kafka_metrics_metadata_req_queue_time Time taken by the request to wait in the request queue | Metadata Request Queue Time | milliseconds |
| kafka_metrics_offset_commit_remote_time Time taken by the request to wait for the follower | Offset Commit Remote Time | milliseconds |
| kafka_metrics_offset_commit_resp_queue_time Time taken by the request to wait in the response queue | Offset Commit Response Queue Time | milliseconds |
| kafka_metrics_offsets_req_queue_time Time taken by the request to wait in the request queue | Offsets Request Queue Time | milliseconds |
| kafka_metrics_offsets_resp_queue_time Time taken by the request to wait in the response queue | Offsets Response Queue Time | milliseconds |
| kafka_metrics_produce_remote_time Time taken by the request to wait for the follower | Producer Remote Time | milliseconds |
| kafka_metrics_stop_replica_total_time Request total time | Stop Replica Total Time | milliseconds |
| kafka_metrics_update_metadata_remote_time Time taken by the request to wait for the follower | Update Metadata Remote Time | milliseconds |
| kafka_metrics_update_metadata_requests Request rate | Update Metadata Requests | Requests / second |
| kafka_net_bytes_in Bytes in rate | Bytes In | bytes / second |
| kafka_net_bytes_out Bytes out rate | Bytes Out | bytes / second |
| kafka_net_bytes_rejected Bytes Rejected | Bytes Rejected | bytes / second |
| kafka_net_messages_in Messages in rate | Messages In | Messages / second |
| kafka_producer_requests_delayed Requests delayed in the producer purgatory | Producer Delayed Requests | Requests |
| kafka_producer_requests_purgatory_delayed Requests delayed in the producer purgatory | Producer Delayed Requests | Requests |
| kafka_producer_requests_purgatory_size Requests waiting in the producer purgatory | Producer Purgatory Size | Requests |
| kafka_producer_requests_size Requests waiting in the producer purgatory | Producer Purgatory Size | Requests |
| kafka_replication_isr_expands ISR expansion rate | ISR Expands | Exapands / second |
| kafka_replication_isr_shrinks ISR shrink rate | ISR Shrinks | Shrinks / second |
| kafka_replication_leader_count Leader replica counts | Leader Count | Total Leaders |
| kafka_replication_leader_elections Leader election rate | Leader Election Rate And Time | Leader elections |
| kafka_replication_max_lag Maximum lag in messages between follower and leader replicas | Replication Max Lag | Messages |
| kafka_replication_partitions Partition counts | Partition Count | Total Partitions |
| kafka_replication_unclean_leader_elections Unclean leader election rate | Unclean Leader Elections | Elections / second |
| kafka_replication_under_replicated_partitions Number of under replicated partitions(|ISR| < |all replicas|) | Under Replicated Partitions | Under Replicated Partitions |
| kafka_request_fetch_failed Failed fetch requests rate | Failed Fetch Requests | Requests / second |
| kafka_request_fetch_time_99percentile Time for produce requests for 99th percentile | Fetch Total Time 99percentile | Requests / second |
| kafka_request_fetch_time_avg Request total time | Fetch Total Time | milliseconds |
| kafka_request_handler_avg_idle_pct Average fraction of time when the request handler threads are idle | Request Handler Threads Idle Time | Fraction |
| kafka_request_metadata_time_99percentile Time for metadata requests for 99th percentile | Metadata 99percentile Time | milliseconds |
| kafka_request_metadata_time_avg Request total time | Metadata Total Time | milliseconds |
| kafka_request_offsets_time_99percentile Time for offset requests for 99th percentile | Offsets Request Time 99percentile | milliseconds |
| kafka_request_offsets_time_avg Average time for an offset request | Offsets Request Time | milliseconds |
| kafka_request_produce_failed Failed producer requests rate | Failed Produce Requests | Requests / second |
| kafka_request_produce_time_99percentile Time for produce requests for 99th percentile | Produce Request Time 99percentile | Request / second |
| kafka_request_produce_time_avg Average time for a produce request | Produce Request Time | Request / second |
| kafka_request_update_metadata_time_99percentile Time for update metadata requests for 99th percentile | Update Metadata 99percentile Time | milliseconds |
| kafka_request_update_metadata_time_avg Request total time | Update Metadata Total Time | milliseconds |