Setting up Data source

---

This guide provides sample scrape job to be used for Prometheus deployment’s scrape-configmap and service monitors to be used with Prometheus operator for the different architectural topologies for integrating Prometheus (connecting a data source link) with Chaos center.

Before you begin

To setup a data source for a chaos center project, you must know about open observability and data source considerations in Litmus 2.0

Topologies

Listed below are three among many topologies in which a data source can be setup for collecting agent cluster's metrics along with chaos metrics for chaos center.

For Control plane agent / Self agent with Prometheus - scraping chaos-exporter metrics and events along with other metrics

Data flow and topological diagram

Scrape jobs:

• Chaos exporter

- job_name: 'chaos-exporter'
  static_configs:
    - targets: ['chaos-exporter.litmus.svc.cluster.local:8080']
  relabel_configs:
    - target_label: instance
      replacement: 'chaos-exporter-service'

• Kube state metrics exporter

- job_name: 'kube-state-metrics'
  static_configs:
    - targets: ['kube-state-metrics.monitoring.svc.cluster.local:8080']

• Node exporter
• Black box exporter
• cAdvisor

Detailed setup guide for Prometheus deployment with scrape configuration

Service monitors:

• Chaos exporter

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: chaos-exporter
  labels:
    k8s-app: chaos-exporter
  namespace: litmus
spec:
  jobLabel: app
  selector:
    matchLabels:
      app: chaos-exporter
  namespaceSelector:
    matchNames:
      - litmus
  endpoints:
    - port: tcp
      interval: 1s
      metricRelabelings:
        - targetLabel: instance
          replacement: 'chaos-exporter-service'

• Kube state metrics exporter

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    app.kubernetes.io/name: kube-state-metrics
    app.kubernetes.io/version: latest
    k8s-app: kube-state-metrics
  name: kube-state-metrics
  namespace: monitoring
spec:
  endpoints:
    - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
      honorLabels: true
      interval: 30s
      port: http-metrics
      relabelings:
        - action: labeldrop
          regex: (pod|service|endpoint|namespace)
      scheme: http
      scrapeTimeout: 30s
      tlsConfig:
        insecureSkipVerify: true
    - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
      interval: 30s
      port: telemetry
      scheme: http
      tlsConfig:
        insecureSkipVerify: true
  jobLabel: app.kubernetes.io/name
  selector:
    matchLabels:
      app.kubernetes.io/name: kube-state-metrics

• Node exporter
• Black box exporter

Detailed setup guide for Prometheus operator with service monitors

Note:

• Scrape jobs as per (above links) can also be added under this additionalScrapeConfigs for the Prometheus community’s Kube prometheus stack

• Pod labels to be used with Prometheus operator installed via helm as part of the Prometheus community’s Kube prometheus stack configured for podMonitors or with Prometheus’s inbuilt Kubernetes service discovery

Chaos exporter

labels

labels:
  app: chaos-exporter
  release: prometheus-stack

pod monitor

apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: chaos-exporter-monitor
  namespace: monitoring
  labels:
    release: prometheus-stack
spec:
  selector:
    matchLabels:
      app: chaos-exporter
  namespaceSelector:
    matchNames:
      - litmus
  podMetricsEndpoints:
    - port: tcp
    - interval: 1s
      metricRelabelings:
        - targetLabel: instance
          replacement: 'chaos-exporter-service'

Black box exporter

labels

labels:
  app: prometheus-blackbox-exporter
  release: prometheus-stack

pod monitor

apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: black-box-exporter-monitor
  namespace: monitoring
  labels:
    release: prometheus-stack
spec:
  selector:
    matchLabels:
      app: prometheus-blackbox-exporter
  namespaceSelector:
    matchNames:
      - monitoring
  podMetricsEndpoints:
    - port: http
    - interval: 1s

Detailed setup guide for Prometheus operator with pod monitors

For Multiple agents with multiple prometheus instances - scraping chaos-exporter metrics and events along with other metrics

Data flow and topological diagram

• Scrape job and service monitor remain same as in case of Control plane agent / self agent, the individual Prometheus instances can be connected as separate data sources to the Chaos center. Separate dashboards can be created by selecting specific agents and their corresponding data source which is essentially a Prometheus time series database, collecting metrics from the agent cluster for application / infra metrics, chaos events and chaos verdicts.

For Multiple agents with single prometheus - scraping chaos-exporter metrics and events along with other metrics

• Separate dashboards can be created by selecting specific agents and the data source which is essentially a Prometheus time series database, collecting metrics from the agent cluster for application / infra metrics, chaos events and chaos verdicts.

Data flow and topological diagram

Scrape jobs:

• Chaos-exporter - agent-1

- job_name: 'chaos-exporter-agent-1'
  static_configs:
    - targets: ['<AGENT_1_CHAOS_EXPORTER_SERVICE_PUBLIC_HOSTNAME/IP:8080>']
  relabel_configs:
    - target_label: instance
      replacement: 'chaos-exporter-service'

• Chaos-exporter - agent-2

- job_name: 'chaos-exporter-agent-2'
  static_configs:
    - targets: ['<AGENT_2_CHAOS_EXPORTER_SERVICE_PUBLIC_HOSTNAME/IP:8080>']
  relabel_configs:
    - target_label: instance
      replacement: 'chaos-exporter-service'

• Kube state metrics exporter - agent-1

- job_name: 'kube-state-metrics-agent-1'
  static_configs:
    - targets: ['<AGENT_1_KUBE_STATE_METRICS_EXPORTER_SERVICE_PUBLIC_HOSTNAME/IP:8080>']

• Kube state metrics exporter - agent-2

- job_name: 'kube-state-metrics-agent-2'
  static_configs:
    - targets: ['<AGENT_2_KUBE_STATE_METRICS_EXPORTER_SERVICE_PUBLIC_HOSTNAME/IP:8080>']

• Node exporter
• Black box exporter
• cAdvisor

Detailed setup guide for Prometheus deployment with scrape configuration

Service endpoint, spec and monitors:

• Chaos-exporter - agent-1

kind: Service
apiVersion: v1
metadata:
  name: chaos-exporter-agent-1
  namespace: monitoring
spec:
  type: ClusterIP
  ports:
    - name: tcp
      port: 8080
      targetPort: 8080
---
kind: Endpoints
apiVersion: v1
metadata:
  name: chaos-exporter-agent-1
  namespace: monitoring
subsets:
  - addresses:
      - ip: <AGENT_1_CHAOS_EXPORTER_SERVICE_PUBLIC_HOSTNAME/IP>
    ports:
      - name: tcp
        port: 8080
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    k8s-app: chaos-exporter-agent-1
  name: chaos-exporter-agent-1
  namespace: monitoring
spec:
  jobLabel: app
  selector:
    matchLabels:
      app: chaos-exporter-agent-1
  namespaceSelector:
    matchNames:
      - monitoring
  endpoints:
    - interval: 1s
      port: tcp
      metricRelabelings:
        - targetLabel: instance
          replacement: 'chaos-exporter-service'

• Chaos-exporter - agent-2

kind: Service
apiVersion: v1
metadata:
  name: chaos-exporter-agent-2
  namespace: monitoring
spec:
  type: ClusterIP
  ports:
    - name: tcp
      port: 8080
      targetPort: 8080
---
kind: Endpoints
apiVersion: v1
metadata:
  name: chaos-exporter-agent-2
  namespace: monitoring
subsets:
  - addresses:
      - ip: <AGENT_2_CHAOS_EXPORTER_SERVICE_PUBLIC_HOSTNAME/IP>
    ports:
      - name: tcp
        port: 8080
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    k8s-app: chaos-exporter-agent-2
  name: chaos-exporter-agent-2
  namespace: monitoring
spec:
  jobLabel: app
  selector:
    matchLabels:
      app: chaos-exporter-agent-2
  namespaceSelector:
    matchNames:
      - monitoring
  endpoints:
    - interval: 1s
      port: tcp
      metricRelabelings:
        - targetLabel: instance
          replacement: 'chaos-exporter-service'

• Kube state metrics exporter - agent-1

kind: Service
apiVersion: v1
metadata:
  name: kube-state-metrics-agent-1
  namespace: monitoring
spec:
  type: ClusterIP
  ports:
    - name: tcp
      port: 8080
      targetPort: 8080
---
kind: Endpoints
apiVersion: v1
metadata:
  name: kube-state-metrics-agent-1
  namespace: monitoring
subsets:
  - addresses:
      - ip: <AGENT_1_KUBE_STATE_METRICS_EXPORTER_SERVICE_PUBLIC_HOSTNAME/IP>
    ports:
      - name: tcp
        port: 8080
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    k8s-app: kube-state-metrics-agent-1
  name: kube-state-metrics-agent-1
  namespace: monitoring
spec:
  jobLabel: app
  selector:
    matchLabels:
      app: kube-state-metrics-agent-1
  namespaceSelector:
    matchNames:
      - monitoring
  endpoints:
    - interval: 30s
      port: tcp

• Kube state metrics exporter - agent-2

kind: Service
apiVersion: v1
metadata:
  name: kube-state-metrics-agent-2
  namespace: monitoring
spec:
  type: ClusterIP
  ports:
    - name: tcp
      port: 8080
      targetPort: 8080
---
kind: Endpoints
apiVersion: v1
metadata:
  name: kube-state-metrics-agent-2
  namespace: monitoring
subsets:
  - addresses:
      - ip: <AGENT_2_KUBE_STATE_METRICS_EXPORTER_SERVICE_PUBLIC_HOSTNAME/IP>
    ports:
      - name: tcp
        port: 8080
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    k8s-app: kube-state-metrics-agent-2
  name: kube-state-metrics-agent-2
  namespace: monitoring
spec:
  jobLabel: app
  selector:
    matchLabels:
      app: kube-state-metrics-agent-2
  namespaceSelector:
    matchNames:
      - monitoring
  endpoints:
    - interval: 30s
      port: tcp

• Node exporter
• Black box exporter

Detailed setup guide for Prometheus operator with service monitors

Resources

Learn more

• Observability Setup
• Configure Data Source
• Manage Application Dashboards