Secret handling has always been a pain point for me over the past few years. Most of the time, I ended up using services with features I didn’t need, and even worse, wasting money on them. That’s why I decided to spend some time doing proper research to find the right tool. Nowadays, FinOps has become a huge topic, and I’m starting to understand why. It seems like cloud providers design their pricing to keep you from understanding what’s really going on. But that’s off-topic (allow me a little rant :)).

The features I wanted in a secret management service were:

• A central place to store secrets
• Ability to audit secret usage
• A straightforward process

Just the basic features — nothing complex. First of all, I did some quick research on the services provided by the major cloud providers, especially from a cost perspective. My requirements were to store a maximum of 50 secrets, nothing fancy in terms of encryption, easy integration with Kubernetes, and fewer than 10K API calls monthly:

There was the option to store Vault Open Source on the Kubernetes cluster. However, you’d have to think about how to handle the sealed/unsealed process, think about encryption for data at rest, and so on. Honestly, it felt like overkill for my current needs, and it went against my principle of keeping things simple.

I consistently avoided other smaller cloud services I didn’t have experience with. Now, I guess you’re wondering what I chose, right? Jokes aside, the choice was obvious in terms of cost. Plus, all the services listed had more than enough basic features, so I decided to stick with Azure Key Vault.

Azure Key Vault

#

I don t want to dive too deep into the setup of Azure Key Vault. The official doc here: https://learn.microsoft.com/en-us/azure/key-vault/general/quick-create-portal definitely explains the steps better than I can. Be sure to set up a proper user with the least privileged policies.

There are different approaches you can follow to connect to the service from outside, but I decided to use a Service Principal with read permissions to the Azure Key Vault service.

By CLI, you can create both the Service Principal and the policy:

az ad sp create-for-rbac --name <service-principal-name> --skip-assignment

This will output:

{
  "appId": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
  "displayName": "<service-principal-name>",
  "name": "http://<service-principal-name>",
  "password": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
  "tenant": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
}

Then, grab the appId and attach the reading permission to the Service Principal:

az keyvault set-policy --name <your-key-vault-name> --spn <your-service-principal-appId> --secret-permissions get list

After that, I created all the secrets via UI. You can also create them via CLI or set up IaC (Infrastructure as Code) for that. Maybe I’ll dig into this in the next few days.

External Secret Operator

#

Finally, here’s the juicy part — setting up the integration with the Kubernetes cluster. I heard good things about the External Secret Operator, both for its simplicity and its integration capabilities. So, I decided to give it a try and integrate Azure Key Vault with it.

For the installation part, there is a handy helm chart you can use. I integrated it into my FluxCD GitOps setup; you can check out the configuration in my homelab repo here.

After installation, you need to handle the configuration. The External Secret Operator offers two CRs (Custom Resources) for connecting to an external secret service:

• ClusterSecretStore
• SecretStore

The first allows you to use this connection across all namespaces, while the second is tied to a specific namespace. I opted for the cluster-scoped one because I didn’t need a namespaced connection:

apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: secret-store
spec:
  provider:
    # provider type: azure keyvault
    azurekv:
      # azure tenant ID, see: https://docs.microsoft.com/en-us/azure/active-directory/fundamentals/active-directory-how-to-find-tenant
      tenantId: "1ed4ffe0-36d4-476f-bc06-741936223e6a"
      # URL of your vault instance, see: https://docs.microsoft.com/en-us/azure/key-vault/general/about-keys-secrets-certificates
      vaultUrl: "https://homelab-dev-alfonso.vault.azure.net"
      authSecretRef:
        # points to the secret that contains
        # the azure service principal credentials
        clientId:
          name: azure-secret-sp
          key: ClientID
          namespace: external-secret-operator
        clientSecret:
          name: azure-secret-sp
          key: ClientSecret
          namespace: external-secret-operator

With this connection, you can create secrets using the ExternalSecret CR. For example, here’s a basic-auth secret:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: wallabag-db-creds-external
spec:
  refreshInterval: 1h
  secretStoreRef:
    kind: ClusterSecretStore
    name: secret-store-dev
  target:
    name: wallabag-db-creds
    creationPolicy: Owner
    template:
      type: kubernetes.io/basic-auth
  data:
    # name of the SECRET in the Azure KV (no prefix is by default a SECRET)
    - secretKey: username
      remoteRef:
        key: wallabag-db-username
    - secretKey: password
      remoteRef:
        key: wallabag-symphony-password

And a general secret:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: wallabag-storage-creds
spec:
  refreshInterval: 1h
  secretStoreRef:
    kind: ClusterSecretStore
    name: secret-store-dev
  target:
    name: azure-creds
    creationPolicy: Owner
  data:
    - secretKey: wallabag-connection-string
      remoteRef:
        key: wallabag-connection-string

The reconciliation is pretty fast. One thing that could be improved is the logging. Sometimes, the reconciliation didn’t happen because of a configuration mistake on my end, and the logs were a bit misleading. That said, it works flawlessly, and the process is really streamlined.

I highly suggest giving this operator a try.