Custom Resources

What is CRs (Custom resources)

According to the official documentation, we have the definition of custom resources as follow:

"A resource is an endpoint in the Kubernetes API that stores a collection of API objects of a certain kind. For example, the built-in pods resource contains a collection of Pod objects.

A custom resource is an extension of the Kubernetes API that is not necessarily available in a default Kubernetes installation. It represents a customization of a particular Kubernetes installation. However, many core Kubernetes functions are now built using custom resources, making Kubernetes more modular.

Custom resources can appear and disappear in a running cluster through dynamic registration, and cluster admins can update custom resources independently of the cluster itself. Once a custom resource is installed, users can create and access its objects using kubectl, just as they do for built-in resources like Pods."

Declare and Create CRD

CRD is more like the class in object oriental programming, and CR is more like the instance of the class. Below is an example from official K8s documentation:

crontab-crd.yaml

apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  # name must match the spec fields below, and be in the form: <plural>.<group>
  name: crontabs.stable.example.com
spec:
  # group name to use for REST API: /apis/<group>/<version>
  group: stable.example.com
  # list of versions supported by this CustomResourceDefinition
  versions:
    - name: v1
      # Each version can be enabled/disabled by Served flag.
      served: true
      # One and only one version must be marked as the storage version.
      storage: true
  # either Namespaced or Cluster
  scope: Namespaced
  names:
    # plural name to be used in the URL: /apis/<group>/<version>/<plural>
    plural: crontabs
    # singular name to be used as an alias on the CLI and for display
    singular: crontab
    # kind is normally the CamelCased singular type. Your resource manifests use this.
    kind: CronTab
    # shortNames allow shorter string to match your resource on the CLI
    shortNames:
    - ct

• metadata.name must be match the spec fields with <plural>.<group>

• spec :

  • group : Naming a group is the first thing you want to do when you are to implement CRD, because you do not want to overlap with existing core group. In your own API group, you could have as many resoureces as you want, and they may have the same name as may exist in other API group

  • versions : You could support multiple versions of custom resources you have developed. But only one must be marked as the storage version. See K8s official documentation for more details.
  • versions:
      - name: v1beta1
        # Each version can be enabled/disabled by Served flag.
        served: true
        # One and only one version must be marked as the storage version.
        storage: true
      - name: v1
        served: true
        storage: false

  • scope : The CRD can be either namespaced or cluster-scoped. <TBA>

  • names : This section is very straight forward, just need to follow the naming convention

Create the CRD

• kubectl apply -f crontab-crd.yaml

customresourcedefinition.apiextensions.k8s.io/crontabs.stable.example.com created

• kubectl get crd crontabs.stable.example.com -o yaml

apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"apiextensions.k8s.io/v1beta1","kind":"CustomResourceDefinition","metadata":{"annotations":{},"name":"crontabs.stable.example.com"},"spec":{"group":"stable.example.com","names":{"kind":"CronTab","plural":"crontabs","shortNames":["ct"],"singular":"crontab"},"scope":"Namespaced","versions":[{"name":"v1","served":true,"storage":true}]}}
  creationTimestamp: "2019-05-10T05:49:35Z"
  generation: 1
  name: crontabs.stable.example.com
  resourceVersion: "3291467"
  selfLink: /apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions/crontabs.stable.example.com
  uid: 64399ad4-72e7-11e9-ab47-0050569e670d
spec:
  conversion:
    strategy: None
  group: stable.example.com
  names:
    kind: CronTab
    listKind: CronTabList
    plural: crontabs
    shortNames:
    - ct
    singular: crontab
  scope: Namespaced
  version: v1
  versions:
  - name: v1
    served: true
    storage: true
status:
  acceptedNames:
    kind: CronTab
    listKind: CronTabList
    plural: crontabs
    shortNames:
    - ct
    singular: crontab
  conditions:
  - lastTransitionTime: "2019-05-10T05:49:35Z"
    message: no conflicts found
    reason: NoConflicts
    status: "True"
    type: NamesAccepted
  - lastTransitionTime: null
    message: the initial names have been accepted
    reason: InitialNamesAccepted
    status: "True"
    type: Established
  storedVersions:
  - v1

kubectl caches discovery content by default for 10 minutes, using the ~/.kube/cache/discovery directory. Also, it can take up to 10 minutes for kubectl to see a new resource, such as the CRD we defined above. However, on a cache miss—that is, when kubectl can not identify the resource name in a command—it will immediately re-discover it.

Create custom objects

my-crontab.yaml

apiVersion: "stable.example.com/v1"
kind: CronTab
metadata:
  name: my-new-cron-object
spec:
  cronSpec: "* * * * */5"
  image: my-awesome-cron-image

• kubectl apply -f my-crontab.yaml

crontab.stable.example.com/my-new-cron-object created

View custom objects

• curl http://127.0.0.1:8001/apis/stable.example.com/v1/namespaces/default/crontabs (You need to run kubectl proxy priro to this command)

{"apiVersion":"stable.example.com/v1","items":[{"apiVersion":"stable.example.com/v1","kind":"CronTab","metadata":{"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"stable.example.com/v1\",\"kind\":\"CronTab\",\"metadata\":{\"annotations\":{},\"name\":\"my-new-cron-object\",\"namespace\":\"default\"},\"spec\":{\"cronSpec\":\"* * * * */5\",\"image\":\"my-awesome-cron-image\"}}\n"},"creationTimestamp":"2019-05-10T06:18:07Z","generation":1,"name":"my-new-cron-object","namespace":"default","resourceVersion":"3293543","selfLink":"/apis/stable.example.com/v1/namespaces/default/crontabs/my-new-cron-object","uid":"6071d640-72eb-11e9-ab47-0050569e670d"},"spec":{"cronSpec":"* * * * */5","image":"my-awesome-cron-image"}}],"kind":"CronTabList","metadata":{"continue":"","resourceVersion":"3293799","selfLink":"/apis/stable.example.com/v1/namespaces/default/crontabs"}}

• kubectl get crontab

NAME                 AGE
my-new-cron-object   6m

• kubectl get ct -o yaml

apiVersion: v1
items:
- apiVersion: stable.example.com/v1
  kind: CronTab
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"stable.example.com/v1","kind":"CronTab","metadata":{"annotations":{},"name":"my-new-cron-object","namespace":"default"},"spec":{"cronSpec":"* * * * */5","image":"my-awesome-cron-image"}}
    creationTimestamp: "2019-05-10T06:18:07Z"
    generation: 1
    name: my-new-cron-object
    namespace: default
    resourceVersion: "3293543"
    selfLink: /apis/stable.example.com/v1/namespaces/default/crontabs/my-new-cron-object
    uid: 6071d640-72eb-11e9-ab47-0050569e670d
  spec:
    cronSpec: '* * * * */5'
    image: my-awesome-cron-image
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

• We could also monitor the resources creation and updates via the API endpoint.

curl -f http://127.0.0.1:8001/apis/stable.example.com/v1/namespaces/default/crontabs?watch=true&resourceVersion=3296084

If you open another terminal and run kubectl delete ct my-new-cron-object to delete the CRD object you just created, you will see the live console output.

{"type":"ADDED","object":{"apiVersion":"stable.example.com/v1","kind":"CronTab","metadata":{"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"stable.example.com/v1\",\"kind\":\"CronTab\",\"metadata\":{\"annotations\":{},\"name\":\"my-new-cron-object\",\"namespace\":\"default\"},\"spec\":{\"cronSpec\":\"* * * * */5\",\"image\":\"my-awesome-cron-image\"}}\n"},"creationTimestamp":"2019-05-10T06:53:01Z","generation":1,"name":"my-new-cron-object","namespace":"default","resourceVersion":"3296084","selfLink":"/apis/stable.example.com/v1/namespaces/default/crontabs/my-new-cron-object","uid":"40a1c61e-72f0-11e9-ab47-0050569e670d"},"spec":{"cronSpec":"* * * * */5","image":"my-awesome-cron-image"}}}
{"type":"DELETED","object":{"apiVersion":"stable.example.com/v1","kind":"CronTab","metadata":{"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"stable.example.com/v1\",\"kind\":\"CronTab\",\"metadata\":{\"annotations\":{},\"name\":\"my-new-cron-object\",\"namespace\":\"default\"},\"spec\":{\"cronSpec\":\"* * * * */5\",\"image\":\"my-awesome-cron-image\"}}\n"},"creationTimestamp":"2019-05-10T06:53:01Z","generation":1,"name":"my-new-cron-object","namespace":"default","resourceVersion":"3296173","selfLink":"/apis/stable.example.com/v1/namespaces/default/crontabs/my-new-cron-object","uid":"40a1c61e-72f0-11e9-ab47-0050569e670d"},"spec":{"cronSpec":"* * * * */5","image":"my-awesome-cron-image"}}}

View CRD objects in etcd

The example below is for PKS environment

• SSH onto Master node

• master/47dab855-6720-4549-a7b6-e9d1319c1ca6:~# sudo -i
  master/47dab855-6720-4549-a7b6-e9d1319c1ca6:~# alias etcdctlv3='ETCDCTL_API=3 /var/vcap/packages/etcd/bin/etcdctl --cert=/var/vcap/jobs/etcd/config/etcdctl.crt --key=/var/vcap/jobs/etcd/config/etcdctl.key --cacert=/var/vcap/jobs/etcd/config/etcdctl-ca.crt --endpoints=https://master-0.etcd.cfcr.internal:2379';

• master/47dab855-6720-4549-a7b6-e9d1319c1ca6:~# etcdctlv3 get '' --keys-only --prefix

/registry/apiextensions.k8s.io/customresourcedefinitions/clustermetricsinks.apps.pivotal.io

/registry/apiextensions.k8s.io/customresourcedefinitions/clustersinks.apps.pivotal.io

/registry/apiextensions.k8s.io/customresourcedefinitions/crontabs.stable.example.com

/registry/apiextensions.k8s.io/customresourcedefinitions/metricsinks.apps.pivotal.io

/registry/apiextensions.k8s.io/customresourcedefinitions/sinks.apps.pivotal.io

• master/47dab855-6720-4549-a7b6-e9d1319c1ca6:~# etcdctlv3 get '/registry/stable.example.com/crontabs/default/my-new-cron-object' 
  /registry/stable.example.com/crontabs/default/my-new-cron-object 
  {"apiVersion":"stable.example.com/v1","kind":"CronTab","metadata":{"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"stable.example.com/v1\",\"kind\":\"CronTab\",\"metadata\":{\"annotations\":{},\"name\":\"my-new-cron-object\",\"namespace\":\"default\"},\"spec\":{\"cronSpec\":\"    /5\",\"image\":\"my-awesome-cron-image\"}}\n"},"creationTimestamp":"2019-05-10T06:58:14Z","generation":1,"name":"my-new-cron-object","namespace":"default","uid":"fb06b742-72f0-11e9-ab47-0050569e670d"},"spec":{"cronSpec":"    /5","image":"my-awesome-cron-image"}}