When the Prometheus server is installed inside a Kubernetes cluster, the pods that we would like to include in the monitoring will have to have their Prometheus metrics endpoint exposed. A kubernetes service should be created for each pod sharing the exporter’s port along with “prometheus.io/scrape” = “true” annotation which will tell the Prometheus server that this service is actually pointed to an exporter. The prometheus server then automatically scrapes the target service and eventually the pod’s running exporter.

In this example the pod’s app label is prometheus-exporter which was added to selector -> app section of the yaml code below and the rest of the naming has been changed accordingly.

apiVersion: v1
kind: Service
metadata:
  name: prometheus-exporter-service
  annotations:
    prometheus.io/scrape: "true"
  labels:
    app: prometheus-exporter
    instance: prometheus-exporter
spec:
  type: ClusterIP
  selector:
    app: prometheus-exporter
  sessionAffinity: ClientIP
  ports:
    - protocol: TCP
      port: 9696
      targetPort: 9696

There are 2 notable settings here which is crucial for the successful scraping.

1. Define prometheus.io/scrape: “true” in annotations. This will tell the Prometheus server to scrape this service.
2. Define instance: prometheus-exporter in labels. This is important because by default the Prometheus server will pick up the internal ip and the service port for the instance name and it will display like that on Grafana which is not really a user friendly way of displaying data. So instead of 10.0.0.180:9696 the instance name defined will show up on Grafana.

If you use terraform to create your kubernetes services just like I would do, use the code below to do so.

resource "kubernetes_service" "prometheus_exporter_service" {
  metadata {
    name      = "prometheus-exporter-service"
    namespace = "prometheus-exporter"
    annotations = {
      "prometheus.io/scrape" = "true"
    }
    labels = {
      app      = "prometheus-exporter"
      instance = "prometheus-exporter"
    }
  }
  spec {
    type = "ClusterIP"
    selector = {
      app      = "prometheus-exporter"
    }
    session_affinity = "ClientIP"
    port {
      port        = 9696
      target_port = 9696
      protocol    = "TCP"
    }
  }
}