# Custom Resources

## What is CRs (Custom resources)

According to the official documentation, we have the definition of custom resources as follow:

"_A resource is an endpoint in the_ [_Kubernetes API_](https://kubernetes.io/docs/reference/using-api/api-overview/) _that stores a collection of_ [_API objects_](https://kubernetes.io/docs/concepts/overview/working-with-objects/kubernetes-objects/) _of a certain kind. For example, the built-in pods resource contains a collection of Pod objects._

_A custom resource is an extension of the Kubernetes API that is not necessarily available in a default Kubernetes installation. It represents a customization of a particular Kubernetes installation. However, many core Kubernetes functions are now built using custom resources, making Kubernetes more modular._

_Custom resources can appear and disappear in a running cluster through dynamic registration, and cluster admins can update custom resources independently of the cluster itself. Once a custom resource is installed, users can create and access its objects using_ [_kubectl_](https://kubernetes.io/docs/user-guide/kubectl-overview/)_, just as they do for built-in resources like Pods._"

## Declare and Create CRD

CRD is more like the class in object oriental programming, and CR is more like the instance of the class. Below is an example from official K8s [documentation](https://kubernetes.io/docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definitions/):

{% code title="crontab-crd.yaml" %}
```yaml
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  # name must match the spec fields below, and be in the form: <plural>.<group>
  name: crontabs.stable.example.com
spec:
  # group name to use for REST API: /apis/<group>/<version>
  group: stable.example.com
  # list of versions supported by this CustomResourceDefinition
  versions:
    - name: v1
      # Each version can be enabled/disabled by Served flag.
      served: true
      # One and only one version must be marked as the storage version.
      storage: true
  # either Namespaced or Cluster
  scope: Namespaced
  names:
    # plural name to be used in the URL: /apis/<group>/<version>/<plural>
    plural: crontabs
    # singular name to be used as an alias on the CLI and for display
    singular: crontab
    # kind is normally the CamelCased singular type. Your resource manifests use this.
    kind: CronTab
    # shortNames allow shorter string to match your resource on the CLI
    shortNames:
    - ct
```
{% endcode %}

* `metadata.name` must be match the spec fields with `<plural>.<group>`
* `spec` :
  * `group` : Naming a group is the first thing you want to do when you are to implement CRD, because you do not want to overlap with existing core group. In your own API group, you could have as many resoureces as you want, and they may have the same name as may exist in other API group
  * `versions` : You could support multiple versions of custom resources you have developed. But **only one must be marked as the storage version.** See K8s official [documentation](https://kubernetes.io/docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definition-versioning/) for more details.
  *

      ```yaml
      versions:
        - name: v1beta1
          # Each version can be enabled/disabled by Served flag.
          served: true
          # One and only one version must be marked as the storage version.
          storage: true
        - name: v1
          served: true
          storage: false
      ```
  * `scope` : The CRD can be either namespaced or cluster-scoped. \<TBA>
  * `names` : This section is very straight forward, just need to follow the naming convention

### Create the CRD

* `kubectl apply -f crontab-crd.yaml`

`customresourcedefinition.apiextensions.k8s.io/crontabs.stable.example.com created`&#x20;

* `kubectl get crd crontabs.stable.example.com -o yaml`&#x20;

```yaml
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"apiextensions.k8s.io/v1beta1","kind":"CustomResourceDefinition","metadata":{"annotations":{},"name":"crontabs.stable.example.com"},"spec":{"group":"stable.example.com","names":{"kind":"CronTab","plural":"crontabs","shortNames":["ct"],"singular":"crontab"},"scope":"Namespaced","versions":[{"name":"v1","served":true,"storage":true}]}}
  creationTimestamp: "2019-05-10T05:49:35Z"
  generation: 1
  name: crontabs.stable.example.com
  resourceVersion: "3291467"
  selfLink: /apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions/crontabs.stable.example.com
  uid: 64399ad4-72e7-11e9-ab47-0050569e670d
spec:
  conversion:
    strategy: None
  group: stable.example.com
  names:
    kind: CronTab
    listKind: CronTabList
    plural: crontabs
    shortNames:
    - ct
    singular: crontab
  scope: Namespaced
  version: v1
  versions:
  - name: v1
    served: true
    storage: true
status:
  acceptedNames:
    kind: CronTab
    listKind: CronTabList
    plural: crontabs
    shortNames:
    - ct
    singular: crontab
  conditions:
  - lastTransitionTime: "2019-05-10T05:49:35Z"
    message: no conflicts found
    reason: NoConflicts
    status: "True"
    type: NamesAccepted
  - lastTransitionTime: null
    message: the initial names have been accepted
    reason: InitialNamesAccepted
    status: "True"
    type: Established
  storedVersions:
  - v1
```

{% hint style="info" %}
`kubectl` caches discovery content by default for 10 minutes, using the `~/.kube/cache/discovery` directory. Also, it can take up to 10 minutes for `kubectl` to see a new resource, such as the CRD we defined above. However, on a cache miss—that is, when `kubectl` can not identify the resource name in a command—it will immediately re-discover it.
{% endhint %}

### Create custom objects

{% code title="my-crontab.yaml" %}
```yaml
apiVersion: "stable.example.com/v1"
kind: CronTab
metadata:
  name: my-new-cron-object
spec:
  cronSpec: "* * * * */5"
  image: my-awesome-cron-image
```
{% endcode %}

* `kubectl apply -f my-crontab.yaml`&#x20;

`crontab.stable.example.com/my-new-cron-object created`&#x20;

### View custom objects

* `curl http://127.0.0.1:8001/apis/stable.example.com/v1/namespaces/default/crontabs` (You need to run `kubectl proxy` priro to this command)

```
{"apiVersion":"stable.example.com/v1","items":[{"apiVersion":"stable.example.com/v1","kind":"CronTab","metadata":{"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"stable.example.com/v1\",\"kind\":\"CronTab\",\"metadata\":{\"annotations\":{},\"name\":\"my-new-cron-object\",\"namespace\":\"default\"},\"spec\":{\"cronSpec\":\"* * * * */5\",\"image\":\"my-awesome-cron-image\"}}\n"},"creationTimestamp":"2019-05-10T06:18:07Z","generation":1,"name":"my-new-cron-object","namespace":"default","resourceVersion":"3293543","selfLink":"/apis/stable.example.com/v1/namespaces/default/crontabs/my-new-cron-object","uid":"6071d640-72eb-11e9-ab47-0050569e670d"},"spec":{"cronSpec":"* * * * */5","image":"my-awesome-cron-image"}}],"kind":"CronTabList","metadata":{"continue":"","resourceVersion":"3293799","selfLink":"/apis/stable.example.com/v1/namespaces/default/crontabs"}}
```

* `kubectl get crontab`&#x20;

```
NAME                 AGE
my-new-cron-object   6m
```

* `kubectl get ct -o yaml`&#x20;

```yaml
apiVersion: v1
items:
- apiVersion: stable.example.com/v1
  kind: CronTab
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"stable.example.com/v1","kind":"CronTab","metadata":{"annotations":{},"name":"my-new-cron-object","namespace":"default"},"spec":{"cronSpec":"* * * * */5","image":"my-awesome-cron-image"}}
    creationTimestamp: "2019-05-10T06:18:07Z"
    generation: 1
    name: my-new-cron-object
    namespace: default
    resourceVersion: "3293543"
    selfLink: /apis/stable.example.com/v1/namespaces/default/crontabs/my-new-cron-object
    uid: 6071d640-72eb-11e9-ab47-0050569e670d
  spec:
    cronSpec: '* * * * */5'
    image: my-awesome-cron-image
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""
```

* We could also monitor the resources creation and updates via the API endpoint.

`curl -f http://127.0.0.1:8001/apis/stable.example.com/v1/namespaces/default/crontabs?watch=true&resourceVersion=3296084`

If you open another terminal and run `kubectl delete ct my-new-cron-object` to delete the CRD object you just created, you will see the live console output.

```
{"type":"ADDED","object":{"apiVersion":"stable.example.com/v1","kind":"CronTab","metadata":{"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"stable.example.com/v1\",\"kind\":\"CronTab\",\"metadata\":{\"annotations\":{},\"name\":\"my-new-cron-object\",\"namespace\":\"default\"},\"spec\":{\"cronSpec\":\"* * * * */5\",\"image\":\"my-awesome-cron-image\"}}\n"},"creationTimestamp":"2019-05-10T06:53:01Z","generation":1,"name":"my-new-cron-object","namespace":"default","resourceVersion":"3296084","selfLink":"/apis/stable.example.com/v1/namespaces/default/crontabs/my-new-cron-object","uid":"40a1c61e-72f0-11e9-ab47-0050569e670d"},"spec":{"cronSpec":"* * * * */5","image":"my-awesome-cron-image"}}}
{"type":"DELETED","object":{"apiVersion":"stable.example.com/v1","kind":"CronTab","metadata":{"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"stable.example.com/v1\",\"kind\":\"CronTab\",\"metadata\":{\"annotations\":{},\"name\":\"my-new-cron-object\",\"namespace\":\"default\"},\"spec\":{\"cronSpec\":\"* * * * */5\",\"image\":\"my-awesome-cron-image\"}}\n"},"creationTimestamp":"2019-05-10T06:53:01Z","generation":1,"name":"my-new-cron-object","namespace":"default","resourceVersion":"3296173","selfLink":"/apis/stable.example.com/v1/namespaces/default/crontabs/my-new-cron-object","uid":"40a1c61e-72f0-11e9-ab47-0050569e670d"},"spec":{"cronSpec":"* * * * */5","image":"my-awesome-cron-image"}}}
```

### View CRD objects in etcd

{% hint style="info" %}
The example below is for PKS environment&#x20;
{% endhint %}

* SSH onto Master node
* ```
  master/47dab855-6720-4549-a7b6-e9d1319c1ca6:~# sudo -i
  master/47dab855-6720-4549-a7b6-e9d1319c1ca6:~# alias etcdctlv3='ETCDCTL_API=3 /var/vcap/packages/etcd/bin/etcdctl --cert=/var/vcap/jobs/etcd/config/etcdctl.crt --key=/var/vcap/jobs/etcd/config/etcdctl.key --cacert=/var/vcap/jobs/etcd/config/etcdctl-ca.crt --endpoints=https://master-0.etcd.cfcr.internal:2379';
  ```
* ```
  master/47dab855-6720-4549-a7b6-e9d1319c1ca6:~# etcdctlv3 get '' --keys-only --prefix
  ```

```
/registry/apiextensions.k8s.io/customresourcedefinitions/clustermetricsinks.apps.pivotal.io

/registry/apiextensions.k8s.io/customresourcedefinitions/clustersinks.apps.pivotal.io

/registry/apiextensions.k8s.io/customresourcedefinitions/crontabs.stable.example.com

/registry/apiextensions.k8s.io/customresourcedefinitions/metricsinks.apps.pivotal.io

/registry/apiextensions.k8s.io/customresourcedefinitions/sinks.apps.pivotal.io
```

* ```
  master/47dab855-6720-4549-a7b6-e9d1319c1ca6:~# etcdctlv3 get '/registry/stable.example.com/crontabs/default/my-new-cron-object' 
  /registry/stable.example.com/crontabs/default/my-new-cron-object 
  {"apiVersion":"stable.example.com/v1","kind":"CronTab","metadata":{"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"stable.example.com/v1\",\"kind\":\"CronTab\",\"metadata\":{\"annotations\":{},\"name\":\"my-new-cron-object\",\"namespace\":\"default\"},\"spec\":{\"cronSpec\":\"    /5\",\"image\":\"my-awesome-cron-image\"}}\n"},"creationTimestamp":"2019-05-10T06:58:14Z","generation":1,"name":"my-new-cron-object","namespace":"default","uid":"fb06b742-72f0-11e9-ab47-0050569e670d"},"spec":{"cronSpec":"    /5","image":"my-awesome-cron-image"}}
  ```
