Kubernetes Example: leverage CRD to provision Google Cloud SQL DBs on demand

The aim is to implement a controller to automatically provision a SQL database managed by Google Cloud to deployment when a user adds a specific annotation to a Deployment and delete it when it is no longer needed.

Moreover, the controller should automatically inject into the Pod belonging to the deployment the information needed to connect to the database, such as the IP of the instance, user and passwords.

The controller will take care of handling correctly passwords storing them in a Secret whose life cycle will be bond to the DB itself.

The solution is scalable and extendable, for example, at some point we could add a different cloud provider simply by adding a new cloud API layer. (Take into consideration to improve the controller adding support for a new annotation `dbbroker-cloud-provider: AWS` to implement hybrid cloud. Create a PR and let’s discuss it).

Difficulty: Medium — Time required: 30 minutes — Author: Paolo Gallina

Ingredients:

• A running Kubernetes cluster, if you do not have one available, minikube will work just fine.
• A Google Cloud Project, if you don’t have one spare or you do not want to mess up your production, you can always make use of the initial free credit.
• A valid Google Cloud service account capable to manage your SQL instances.
• Kubebuilder installed if you want to try from scratch or my code downloaded in your Go folder to directly test the solution.
• Basic understanding of GoLang.

Moreover, if you do not have any knowledge of controllers and you have never used Kubebuilder, I would recommend you to first check out my previous article that aimed to introduce CRDs, controllers and how to leverage Kubebuilder to get started with them.

Directions:

The task might seem difficult and complex, however, keeping it as simple as possible, we will have to code merely the following three elements:

1. A custom resource to represent the DB instance into the cluster linking their lifecycles, I decided to call it dbbroker.
2. A controller to manage the reconcile process of the dbbroker object, to create the Db and to inject into the Pod the information needed to connect.
3. A controller to span a dbbroker object if a user adds the required annotation and to delete it if removed.

Reconciliation pattern

All controllers follow the reconciliation pattern: basically, each time the object is modified the controller check if the status is the expected one given the specifications, if not, it tries to converge the status to the desirable one.

Reconciler implements interfaces that attempt to reconcile the actual state of the world with the desired one by triggering actions.

Code the dbbroker CRD and implement the needed logic

The idea is to link the lifecycle of the DB to a CRD called dbbroker in order to hide the complexity of both credential management and of DB creation. To do so we will listen for every change on the dbbroker instances:

• upon creation of the dbbroker, the DB instance will be created as well.
• once the DB object is created we will have to wait till the actual Db instance is up and running to populate the info inside the Pods and set the dbbroker as initialized.
• on deletion, we will garbage collect the resource in the cloud.

Define the new resource as follows:

To connect to GoogleCloud leverage the official GO client, that, although it is currently a beta, it is really simple and it is going to be the official way to interact with GCP.

The authentication with the Google Cloud Platform will be managed by the client out of the box, the only action required from our side is to export an environment variable pointing to a GCP service account token having the cloud SQL admin permissions:

export GOOGLE_APPLICATION_CREDENTIALS="/path/to/MyToken.json

Each time a dbbroker instance gets modified this flow is called and in case the status is different from initialized we will call an idempotent function to trigger the creation of the DB.

Idempotence is the property of certain operations in computer science whereby they can be applied multiple times without changing the result beyond the initial application.

Note that for sake of simplicity the specifications are hardcoded. (Why don’t you try to improve the solution making all the fields configurable by annotations? I will be very happy to review any PR going in this direction)

The Root password is generated and injected directly into the instance making use of a secret stored in the same namespace, thus it never leaves the cluster and the namespace. For each dbbroker there will be one DB and one secret, the information stored will be exposed as environment variables in the target Pod.

The secret is owned by the dbbroker resource, in this way once deleted it will be cleaned automatically and we can forget about it, Kubenetes will manage its lifecycle for us thanks to the OwnerReference.

Add also a no-Root user, that should be used during normal operation by the service running into the pod, both username and password should be randomly generated:

Each time a new DB is created we will have to wait till the instance is up and running and the IP of the endpoint published by Google Cloud.

The function will try recursively to fetch the IP each time waiting more time. After 10 retries the reconciliation will fail and the process will start from scratch, otherwise, there would be the possibility to incur into a deadlock.

Populate Information into the deployment
Once the Db is up and running all the data needed in order to connect will be known by the controller, however, it still has to pass it to the Pod. So far the passwords are stored into the secret owned by the dbbroker instance and the endpoint and the username are saved into the fields of the status.

Expose the information into the container as environment variables. To do so, first of all, fetch the deployment by the name and the namespace, information saved into the dbbroker specs.

The core of the logic of the last snippet is the function checkIfInfoMissingAndPopulate. It takes as argument the pointer to the specs of the first container of the deployment and if the information is not injected, it populates them and returns true to trigger the update of the deployment.

An example of how to inject an environment variable taking the values from a secret and from the status of the dbbroker instance follows:

(Consider to improve the solution giving the possibility to the user to choose in which container to inject credentials: 0,1,.., all of them)

DeleteOldDb

Keep in mind that the dbbroker CR is totally managed, it should be created and deleted automatically: the user should never act directly on the DB or on the resource owned by it. Therefore the deletion of the resource will be triggered in a normal scenario merely by the controller once the annotation `dbbroker-db-required:true` is removed or the deployment deleted.

In order to avoid a proliferation of old unused DBs (that will affect our GCP bill at the end of the month) we should make sure to get rid of the DBs once the dbbroker resource is deleted:

As we stated already the Secret will be garbaged collected by Kubernetes and we can forget about it.

Creation of dbbroker instance

If you arrived till this point, then from the beginning you have for sure one question in your mind: “who is creating the dbbroker instance in the first place?”

— A new controller, of course!

This time the controller is not listening on changes on the dbbroker but on all the deployments of our cluster, if needed, we can restrict the behavior for the sake of performances or due to security concerns.

We have again our reconciliation flow:

• when the annotation is spotted we trigger the creation of the DB if not already present. The DB will be called deploymentName-randomstring since in the Google Cloud Platform, unluckily we cannot recreate a DB having the same name without waiting some time
• if the DB is already present we check if any info is missing, if so, we fetch them from the dbbroker instance and the secret.
• If the annotation is removed we delete the dbbroker instance and clean the Pod environment variables from the Specs.

Once again the dbbroker will be owned by the deployment thanks to the OwnerReferences, if someone deletes the deployment, then the dbbroker is deleted and the secret as well on cascade delete.

The creation of the dbbroker can be implemented as follows:

Notice that we added a label to the dbbroker instance in order to fetch it through a getList without knowing its actual name since it has been partially randomly generated.

DEMO

Remember to export all the needed environment variables before running the controller and to add in the config/config.yaml the id of your google cloud project:

➜ export GOOGLE_APPLICATION_CREDENTIALS = ”/users/paologallina/Downloads/yourToken.json”
➜ GOPATH = $HOME/go
➜ make; make install; make run

Let’s create a simple deployment with the annotation, it will trigger the creation of all the resources:

 apiVersion: extensions/v1beta1
 kind: Deployment
 metadata:
 annotations:
 dbbroker: managed
 dbbroker-db-required: ‘true’
...

The controller is triggered and the dbbroker created:

The dbbroker has been created, check form the status if it has been initialized already and notice its owner from the specs:

The secret as well has been created, therefore we can check the deployment looking for the injected information:

The deployment Specs:

If we exec into the pod we will notice all the environment variables already available to any process running:

The last step is to check out the Google Console and, YES, here it is the DB!

Let’s try to clean out the DB, removing the annotation or simply by getting rid of the deployment itself.

➜ dbbroker git:(master) ✗ kubectl delete deployment nginx
deployment.extensions “nginx” deleted
➜ dbbroker git:(master) ✗ kubuectl get secrets
zsh: command not found: kubuectl
➜ dbbroker git:(master) ✗ kubectl get secrets
NAME TYPE DATA AGE
default-token-nkljh kubernetes.io/service-account-token 3 38d

Everything got deleted, as expected, in our GCP project as well!

Disclaimer

For the sake of readability, the code shown is not always the one actually used inside the controller, some error checks are taken away and some parts collapsed, please check the full implementation here.

The solution has been developed not to be deployed in a production environment, but merely to show some of the most common patterns.

In the future, a video Demo will be added to the article.

Additional tool used

• Viper to deal with configuration
• Goland as IDE
• My previous article