Monitor Jenkins with G.A.P on K8s cluster

The goal of this page is to show you how to monitor Jenkins which is running on a K8s cluster.

G.A.P stands for Grafana, AlertManager, Prometheus

Pre-requisite

• Before everything, you have to have the Jenkins running on a K8s cluster. Please see the step by step guide to deploy and configure Jenkins on K8s.

• Install the "Prometheus" plugin on Jenkins. The Plugin will expose the Prometheus endpoint

Install Prometheus

• Create a prometheus configuration YAML file which is basically telling prometheus what jobs to run and how to run

prometheus-config.yaml

global:
  scrape_interval: 15s # By default, scrape targets every 15 seconds.

  # Attach these labels to any time series or alerts when communicating with
  # external systems (federation, remote storage, Alertmanager).
  external_labels:
    monitor: 'pks-monitor'

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'
    # Override the global default and scrape targets from this job every 5 seconds.
    scrape_interval: 5s
    static_configs:
      - targets: ['localhost:9090']
  - job_name: 'jenkins-lb'
    scrape_interval: 5s
    metrics_path: '/prometheus'
    basic_auth:
      username: '<jenkins_username>'
      password: '<jenkins_password>'
    static_configs:
      - targets: ['jenkins-lb:80']
  - job_name: 'node-exporter'
    scrape_interval: 5s
    static_configs:
      - targets: ['node-exporter-service:9100']
  - job_name: 'kube-state-metrics'
    scrape_interval: 5s
    static_configs:
      - targets: ['kube-state-metrics.kube-system.svc.cluster.local:8080']
  - job_name: 'scheduler-service'
    scrape_timeout: 15s
    static_configs:
      - targets: ['scheduler-service.kube-system.svc.cluster.local:10251']

• job_name: 'prometheus': Tells prometheus to scrape itself

• job_name: 'jenkins-lb': Configures prometheus to scrape Jenkins metrics

• job_name: 'node-exporter': Configures prometheus to scrape the node-exporter-service. The service is deployed separately

• job_name: 'kube-state-metrics': Configures prometheus to scrape data from kube-state-metrics

• job_name: 'scheduler-service': Configure prometheus to monitor the kube-scheduler. Some K8s service provider(PKS) might hide the kube-scheduler, so you won't see it from kubectl.

Reference: https://sysdig.com/blog/kubernetes-monitoring-prometheus/