Prometheus K8s SD Receiver

Introduction

The Prometheus Kubernetes Service Discovery (K8s SD) receiver allows you to collect metrics from any software or workload that exposes Prometheus-compatible metric endpoints. Kubernetes service discovery configurations enable the receiver to dynamically retrieve and update scrape targets and their associated labels using the Kubernetes REST API, ensuring continuous synchronization with the current cluster state. It also supports modifying target and metric labels through the relabel_configs property, enabling flexible filtering, enrichment, and normalization of collected metrics.

One of the following role types can be configured to discover targets:

Pod role – available meta labels

Meta Label	Description
__meta_kubernetes_namespace	The namespace of the pod object.
__meta_kubernetes_pod_name	The name of the pod object.
__meta_kubernetes_pod_ip	The pod IP of the pod object.
__meta_kubernetes_pod_label_<labelname>	Each label from the pod object, with any unsupported characters converted to an underscore.
__meta_kubernetes_pod_labelpresent_<labelname>	Set to true for each label from the pod object, with any unsupported characters converted to an underscore.
__meta_kubernetes_pod_annotation_<annotationname>	Each annotation from the pod object.
__meta_kubernetes_pod_annotationpresent_<annotationname>	Set to true for each annotation from the pod object.
__meta_kubernetes_pod_container_init	Set to true if the container is an InitContainer.
__meta_kubernetes_pod_container_name	The name of the container the target address points to.
__meta_kubernetes_pod_container_id	The ID of the container the target address points to. Format: <type>://<container_id>.
__meta_kubernetes_pod_container_image	The image used by the container.
__meta_kubernetes_pod_container_port_name	The name of the container port.
__meta_kubernetes_pod_container_port_number	The port number of the container.
__meta_kubernetes_pod_container_port_protocol	The protocol of the container port.
__meta_kubernetes_pod_ready	Indicates whether the pod is in a ready state (true or false).
__meta_kubernetes_pod_phase	The current pod phase: Pending, Running, Succeeded, Failed, or Unknown.
__meta_kubernetes_pod_node_name	The name of the node where the pod is scheduled.
__meta_kubernetes_pod_host_ip	The host IP address of the pod.
__meta_kubernetes_pod_uid	The unique UID of the pod object.
__meta_kubernetes_pod_controller_kind	The kind of controller managing the pod.
__meta_kubernetes_pod_controller_name	The name of the pod controller.

Service role – available meta labels

Meta Label	Description
__meta_kubernetes_namespace	The namespace of the service object.
__meta_kubernetes_service_annotation_<annotationname>	Each annotation from the service object.
__meta_kubernetes_service_annotationpresent_<annotationname>	Set to true for each annotation of the service object.
__meta_kubernetes_service_cluster_ip	The cluster IP address of the service (not applicable to services of type ExternalName).
__meta_kubernetes_service_loadbalancer_ip	The IP address of the load balancer (applies to services of type LoadBalancer).
__meta_kubernetes_service_external_name	The DNS name of the service (applies to services of type ExternalName).
__meta_kubernetes_service_label_<labelname>	Each label from the service object, with any unsupported characters converted to an underscore.
__meta_kubernetes_service_labelpresent_<labelname>	Set to true for each label of the service object, with any unsupported characters converted to an underscore.
__meta_kubernetes_service_name	The name of the service object.
__meta_kubernetes_service_port_name	The name of the service port for the target.
__meta_kubernetes_service_port_number	The port number of the service port for the target.
__meta_kubernetes_service_port_protocol	The protocol of the service port for the target.
__meta_kubernetes_service_type	The type of the service.

Steps to Configure

Step 1: Check for Existing ConfigMap

Run the following command to see if the opsramp-workload-metric-user-config ConfigMap already exists:

kubectl get configmap opsramp-workload-metric-user-config -n <agent-installed-namespace>

Step 2: Create a New ConfigMap (if not available)

If this is your first setup, create a workload-master ConfigMap. Use the following template and update it as per your workload requirements:

Example 1 - role: service

apiVersion: v1
kind: ConfigMap
metadata:
  name: opsramp-workload-metric-user-config
  namespace: opsramp-agent
data:
  workloads: |
    my-app-svc/prometheus-k8s-sd:
    - job_name: my-app-svc-1
      scrape_interval: 90s
      metrics_path: "/metrics"
      scheme: "http"
      kubernetes_sd_configs:
        - role: service
          namespaces:
            names:
              - opsramp-agent
      relabel_configs:
        - action: replace
          source_labels: ["__meta_kubernetes_service_name"]
          target_label: "opsramp_service_name"

Example 2 - role: pod

apiVersion: v1
kind: ConfigMap
metadata:
  name: opsramp-workload-metric-user-config
  namespace: opsramp-agent
data:
  workloads: |
    my-app-pod/prometheus-k8s-sd:
    - job_name: my-app-pod-1
      scrape_interval: 90s
      metrics_path: "/metrics"
      scheme: "http"
      kubernetes_sd_configs:
        - role: pod
          namespaces:
            names:
              - opsramp-agent
          selectors:
            - role: pod
              label: "app=opsramp-agent-master"
      relabel_configs:
        - action: replace
          source_labels: [__meta_kubernetes_pod_name]
          target_label: opsramp_pod_name
        - action: replace
          source_labels: [__meta_kubernetes_namespace]
          target_label: opsramp_namespace

Note

Ensure there are no extra spaces at the end of any line in the manifest. Extra spaces can cause rendering issues.

Step 3: Apply the ConfigMap

kubectl apply -f <path-to-your-manifest-file> -n <agent-installed-namespace>

Step 4: Verify in OpsRamp Portal

After a few minutes, check metrics in the OpsRamp Portal > Dashboards > Dashboard.

Step 5: Set Up Alerts and Availability Rules

Once metrics are visible, you can configure alerts and availability rules in the OpsRamp portal. Edit the relevant ConfigMaps to adjust workload or application-specific monitoring.

Important Note:

This receiver gives user flexibility to modify / delete / add the metric labels for corresponding workloads.
The targetPodSelector field should NOT be used with this receiver.
Under relabel_configs section, the following action are supported:
- [ "replace", "keep", "drop", "lowercase", "uppercase", "keepequal", "dropequal", "hashmod", "labelmap", "labeldrop", "labelkeep" ]
Mandatory fields in user workload configuration:
- [ "job_name", "kubernetes_sd_configs" ]