Introduction
When working with Azure Kubernetes Service (AKS), you are going to at some point need to think about how to handle secrets, for example, you may need to set environment variables on one of your containers which allow connection to a database, you do not want to hard code these values for two main reasons:
1. Storing credentials in Git is a security risk.
2. Hardcoding values restricts your image from being flexible, what I mean by this is your image may have the ability to connect to multiple databases, if you hard code the values, you would require an image for every database/environment, where-as if you set the image to read from environment variables, these can be set at container runtime, thus only one image is required for all environments.
That is where Azure Key Vault comes in… we can store secrets (credentials) and then use these secrets to set environment variables at container runtime in AKS.
In this article we are going to go through the steps required to integrate Azure Key Vault with AKS so we can set environment variables using Azure Key Vault Provider for Secrets Store CSI Driver.
Requirements
- Contributor Role on a Azure Subscription
- Azure Cloud Shell
- Azure CLI
- Kubectl
Create Azure Kubernetes Service
First things first, lets create an Azure Kubernetes Service!
az group create --name jf-aks-rg --location uksouth
az aks create -g jf-aks-rg --name jf-aks --node-count 1 --enable-addons azure-keyvault-secrets-provider --enable-managed-identity --generate-ssh-keys
az aks get-credentials -g jf-aks-rg --name jf-aks
The first command above creates a resource group called jf-aks-rg. The second command creates an AKS cluster named jf-aks with the azure-keyvault-secrets-provider addon enabled. The final command gets the AKS credentials so we can manage the cluster through kubectl.
During the creation of the AKS cluster, a managed identity is automatically created named azurekeyvaultsecretsprovider-jf-aks and assigned to the Kubelet.
After the AKS creation is complete, if we run the following command:
kubectl get pods -n kube-system
We should see 4 pods have been created that make up the azure-keyvault-secrets-provider addon:
Create Azure Key Vault
Now lets create a Key Vault to store our secrets!
az keyvault create --name jf-aks-kv -g jf-aks-rg --location uksouth
az keyvault secret set --vault-name "jf-aks-kv"--name "demosecret" --value "ThisIsMyDemoSecret!"
The two commands above create a Key Vault named jf-aks-kv with a secret inside named demosecret with a value of ThisIsMyDemoSecret!
Grant AKS Access to Key Vault
Now we have both the AKS cluster and Key Vault created, we need to allow the AKS cluster to access the Key Vault to get secrets, to do so, we need to amend the access policy of the Key Vault.
az aks show -g jf-aks-rg -n jf-aks --query "addonProfiles.azureKeyvaultSecretsProvider.identity.objectId"
az keyvault set-policy --name jf-aks-kv --object-id <OUTPUT OF ABOVE COMMAND> --secret-permissions "get"
The first command above fetches the ObjectId of the managed identity assigned to the Key Vault addon. The second command then sets the access policy to allow this managed identity to GET secrets from the Key Vault.
Create Secret Provider Class
Now we have granted AKS access to our Key Vault, we need to create a SecretProviderClass in AKS, this is where we define all of the secrets we want to fetch from the Key Vault.
Firstly, we need to get the ClientId of the Key Vault addon managed identity, note this is different to the ObjectId we fetched earlier:
az aks show -g jf-aks-rg -n jf-aks --query "addonProfiles.azureKeyvaultSecretsProvider.identity.clientId"
Please create a file named secrets.yml with the contents below:
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
name: "kv-secret-provider" # name given to secret provider class
spec:
provider: azure
secretObjects:
- secretName: aks-secret # name given to our kubernetes secret
type: Opaque
data:
- objectName: demosecret # must match objectName below
key: demosecret # this can be called what you want, this is to reference this object.
parameters:
usePodIdentity: "false"
useVMManagedIdentity: "true" # set to true
userAssignedIdentityID: "<CLIENT ID OF KEY VAULT ADDON MANAGED IDENTITY" # set to the client id of the key vault addon managed identity fetched above
keyvaultName: "<KEY VAULT NAME>" # name of your key vault
objects: |
array:
- |
objectName: demosecret # name of secret in key vault
objectType: secret # object types: secret, key, or cert
objectVersion: "" # defaults to latest if not specified
tenantId: "<AZURE TENANT ID>" # the tenant id of your key vault
What we are doing above is fetching the secret from our Key Vault and storing it into a Kubernetes Secret. Please refer to the comments in the above YAML file for more detail on each selector.
Now apply this configuration using the following command:
kubectl apply -f secrets.yml
Inject Secrets as Environment Variables
So now we have created our SecretProviderClass, next is to create a Pod and inject our environment variables into the container using our secret. For this demo I am just using a public image from Docker.
Create a pod.yml file with the contents below:
kind: Pod
metadata:
name: docker-getting-started
spec:
containers:
- name: docker-getting-started
image: docker/getting-started
env:
- name: MY_KV_SECRET # environment variable to set inside container
valueFrom:
secretKeyRef:
name: aks-secret # name of our kubernetes secret
key: demosecret # key specified in kubernetes secret
volumeMounts:
- name: secret-volume # same name as volume below
mountPath: "/mnt/secrets"
readOnly: true
volumes:
- name: secret-volume # given name to volume
csi:
driver: secrets-store.csi.k8s.io
readOnly: true
volumeAttributes:
secretProviderClass: kv-secret-provider # name of secret provider class created in previous step
Now apply this configuration using the following command:
kubectl apply -f pod.yml
If we run the following command we should see 1 pod running:
kubectl get pods
Now to see if our secret has successfully been injected as an environment variable, run the following command to execute printenv inside the container we just created:
kubectl exec -it docker-getting-started -- printenv
As you can see below, our environment variable MY_KV_SECRET has been successfully set with the value from our Key Vault secret!
Jim Ferrari 