Configure Jenkins to run on K8S

Jenkins An open source automation server which helps people to build, deploy and automate their tasks. I am using it for the following purposes:

• Treat as the cron job which runs some tasks periodically

• Treat as a Pipeline which runs several long time integration test

Jenkins is based on a master-slave structure, master runs as the task scheduler, and slaves run as the task agent. If you setup Jenkins on a single node cluster, master and salves are on the same node. In my case, I deploy Jenkins on a K8s cluster. By doing so, you could expect the master runs as a pod on a node which is scheduled by K8s scheduler, and Jenkins jobs would be running as pods as well.

Pre-requisition

• Setup A running k8s cluster

• Deploy Jenkins with the following YAMLs

00-ns.yaml

apiVersion: v1
kind: Namespace
metadata:
  name: pks-testbed

01-service-account.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  name: jenkins-service-account
  namespace: pks-testbed
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "true"
  creationTimestamp: null
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
  name: jenkins
rules:
- apiGroups: ['']
  resources: ['events',
              'namespaces',
              'nodes',
              'nodes/stats',
              'pods',
              'services']
  verbs: ['get', 'list', 'watch', 'create', 'delete']
- apiGroups: ['extensions']
  resources: ['deployments']
  verbs: ['get', 'list', 'watch']
- nonResourceURLs: ["/metrics"]
  verbs:
  - get
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: jenkins
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: jenkins
subjects:
- kind: ServiceAccount
  name: jenkins-service-account
  namespace: pks-testbed

02-deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: jenkins-deployment
  namespace: pks-testbed
spec:
  selector:
    matchLabels:
      app: jenkins
  replicas: 1
  template:
    metadata:
      labels:
        app: jenkins
    spec:
      serviceAccountName: jenkins-service-account
      containers:
      - name: jenkins
        image: jenkins/jenkins:lts
        imagePullPolicy: Always
        ports:
        - name: http-port
          containerPort: 8080
        - name: jnlp-port
          containerPort: 50000

03-service.yaml

apiVersion: v1
kind: Service
metadata:
  name: jenkins-np
  namespace: pks-testbed
spec:
  type: NodePort
  selector:
    app: jenkins
  ports:
  - name: http
    port: 80
    targetPort: 8080
    nodePort: 30036
    protocol: TCP
---
apiVersion: v1
kind: Service
metadata:
  name: jenkins-lb
  namespace: pks-testbed
spec:
  type: LoadBalancer
  selector:
    app: jenkins
  ports:
  - name: http
    port: 80
    targetPort: 8080
    protocol: TCPs

After deploy the Jenkins, you should be able to access it from your node IP or external IP from your load balancer.

Configurations

• Install the kubernetes plugin

• Go to "Manage Jenkins" -> "Configure System" -> "Add new cloud", choose "Kubernetes"

• Run "kubectl cluster-info" to get the "Kubernetes URL"

• Get the IP of the pod on which Jenkins master is running on:

kubo@jumper:~/jenkins$ kubectl get pods -n pks-testbed
NAME                                 READY   STATUS    RESTARTS   AGE
jenkins-deployment-cb7bfb86c-mvkl6   1/1     Running   0          45h

kubo@jumper:~/jenkins$ kubectl describe pod jenkins-deployment-cb7bfb86c-mvkl6 -n pks-testbed
Name:               jenkins-deployment-cb7bfb86c-mvkl6
Namespace:          pks-testbed
Priority:           0
PriorityClassName:  <none>
Node:               ec78dee2-3652-4c45-8621-1877e66856bc/30.0.3.4
Start Time:         Fri, 29 Mar 2019 01:22:55 -0700
Labels:             app=jenkins
                    pod-template-hash=cb7bfb86c
Annotations:        <none>
Status:             Running
IP:                 40.0.0.2
Controlled By:      ReplicaSet/jenkins-deployment-cb7bfb86c
......

Use freestyle project

• Adding pod template here is required for Jenkins freestyle projects to be run on a pod.

  • Under the "Image" section, Click on "Add Pod Template"

  • Add "Containers" is a little bit tricky. By default Jenkins will use a container named "jnlp" to run your job even you don't add any "Containers" in configurations. If you add another container whose name is different from "jnlp", Jenkins will bring up a pod with two containers(jnlp + the_other_one), and the script you put in your "Execute Shell" section will only be executed in "jnlp". If you want to run your shell script in your own container, you have to build you own slave image, and add the container as "jnlp". By doing this, the default jnlp image from Jenkins will be replaced.

Use pipeline

The more flexible solution to use different docker images for different job is to use Jenkins Pipeline.

• Install "Pipeline" plugin, then you will be able to add a new item as pipeline. (Official Doc)
• In your pipeline script, you could run any shell script in a specific container

def label = "mypod-${UUID.randomUUID().toString()}"
podTemplate(label: label) {
  node(label) {
    stage('Run shell') {
      container('mycontainer') {
        sh 'echo hello world'
      }
    }
  }
}

Advanced Example

def pod_name = "${JOB_NAME}-${UUID.randomUUID().toString()}"
def pod_label = "${JOB_NAME}"
def docker_image = params.docker_image

podTemplate(
        name: pod_name,
        label: pod_label,
        containers: [
                containerTemplate(name: 'hello-jenkins',
                        image: docker_image,
                        alwaysPullImage: true,
                        ttyEnabled: true,
                        command: 'cat')
        ]
)
{
    node(pod_label) {
        stage('Provision') {
            container('hello-jenkins') {
              sh ```
              echo hello-world
              ```  
            }
        }
    }
}

If use pipeline, you could configure the pot template, container template in pipeline script instead of doing it from "Manage Jenkins" UI portal.