Recently, I configured Kubernetes secrets for a Rails app to be stored in git for deploying the app using the GitOps approach for one of our clients. This blog post is about the approach I followed to store the Kubernetes secrets for GitOps.
Storing Kubernetes manifest on git
We store the deployment, configmap, ingress & service
YAML files in Gitlab.
Hence some part of GitOps is already done. For the
had to first export secrets from the Kubernetes cluster to yaml since it's not
stored anywhere else using:
kubectl get secrets my-app-secret -o yaml > secrets.yaml
This is how secrets.yaml content looks like:
apiVersion: v1 data: user: cGFzcw== kind: Secret metadata: name: my-app-secret type: Opaque
In Kubernetes, secrets are store encoded in base64 format, which can be decoded easily. When you follow GitOps and secrets are to be stored on git, they can't be stored in base64 format, they need to be encrypted & stored on git. I came across kubeseal to solve this.
What is Kubeseal?
Its a tool used for encrypting Kubernetes secrets. Kubeseal will be used to make SealedSecrets as templates for secrets. A SealedSecret will be a CRD(Custom Resource Definition) that can be decrypted only by the kubeseal controller running on your Kubernetes cluster. The controller then creates a Kubernetes secret on the cluster.
You can install kubseal by following the instructions from its README.md.
The kubeseal consits of two parts.
Client - Installed locally.
Controller - Installed on the remote cluster.
For the controller to decrypt the
SealedSecret it needs a certificate. While
creating a SealedSecret locally we will use the certificate to encrypt it
locally. Following is the command to fetch the certificate.
kubeseal --fetch-cert > staging.pem
Now with this certificate we can create a
echo -n foo | kubeseal --raw --from-file secrets.yml --cert staging.pem -o yaml --name mysecret
but we need a
SealedSecret resource as yaml instead of just a secret. We need
to pass a Kubernetes secret.yaml file for kubeseal to create a
SealedSecret for us.
kubeseal --cert staging.pem --format=yaml < secrets.yml
apiVersion: bitnami.com/v1alpha1 kind: SealedSecret metadata: creationTimestamp: null name: my-app-secret namespace: default spec: encryptedData: my-db-pass: AgChw/NSDxfhun7kWZZnXHR6zj... template: metadata: creationTimestamp: null name: my-app-secret namespace: default type: Opaque
Now I can output it to a yaml file as:
kubeseal --cert staging.pem --format=yaml < secrets.yml > sealedsecret.yml
The above steps are helpful when we don't have full access to the Kubernetes cluster for using the kubeseal controller directly.
Other ways to create
There is another way to generate a sealedsecret with kubeseal command, we need to pass secret.yaml to kubeseal command as input. Note that we are not passing the certificate manually this time, the controller will fetch it automatically.
kubectl create secret generic app-secret --dry-run --from-literal=foo=bar -o yaml | \ kubeseal \ --controller-name=kubeseal-sealed-secrets \ --controller-namespace=kubeseal \ --format yaml > sealedsecret.yaml
this will create a sealedsecret.yaml. The sealedsecret.yaml can be committed to git along with other kubernetes manifest files for GitOps. Using these manifest files we can deploy our app using any CD tool. In our case, we deployed using Argo CD, an opensource GitOps continuous delivery tool for Kubernetes.
When the sealedsecrets.yaml is deployed to the Kubernetes cluster the controller will unseal the SealedSecret automatically & create a secret. It will also watch for any changes to the SealedSecret & will update the corresponding secret.