Skip to content

Secrets Management With External Secrets Operator and 1Password (part 2)

In this second part for my Secrets Management with External Secrets Operator (ESO) and 1Password, I will be detailing how I configured my ESO deployment through GitOps using Flux, Kustomization, and Secrets resources. You can read the first part here: Secrets Management With External Secrets Operator and 1Password (part 1).

A recap on why ESO: the goal of the ESO operator is to synchronize secrets from these external sources into Kubernetes secrets, so they can be more easily accessed and used throughout the cluster.

All of these configuration files can be found in my homelab GitHub repository located here: https://github.com/cyberwatchdoug/homelab/tree/main

First let's recap and take another look at the structure of the GitOps repository. I will be focusing on describing the remaining files in this post, as well as how they work together with those previously detailed in part 1 found here which together deploy a declarative configuration for secrets management in my homelab using GitOps.

File and Folder Structure

infrastructure
├───base
│   └───external-secrets
│       │   deployment-crds.yaml
│       │   deployment-crs.yaml
│       │   kustomization.yaml
│       │   namespace.yaml
│       │   release.yaml
│       │   repositories.yaml
│       │
│       └───crs
│               cluster-secret-store.yaml
│               kustomization.yaml
│
└───staging
    └───external-secrets
            kustomization.yaml

In this part 2, we are concerned with the files in the infrastructure/base/external-secrets/crs/ folder as well as the deployment-crds.yaml and deployment-crs.yaml files.

Secrets Management for ESO

In part 1, I did not make mention of credentials that would be needed in order to use ESO to access the external provider. As best practice, we do not want this token hard coded into our infrastructure manifests. The simplest is storing the credentials needed inside a Kubernetes Secret store, which can be manually created, or fully automated as part of any infrastructure or configuration automations when setting up your K8s nodes. I won't cover this in detail here but will likely have a future post with more information.

I'll point out where the Secret store is referenced in the file below.

Deploying Custom Resource Definitions (CRDs)

As pointed out in part 1, there is a race condition we are trying to have control over when Custom Resources (CRs) are created compared to the Custom Resource Definitions (CRDs). The problem is that we have two controllers working to deploy ESO. One is the HelmController, and the other is the KustomizeController. As both controllers manage resources independently, there is no control over making one wait for the other. What is needed is a process to control the CRDs being deployed before the CRs.

The first step in controlling this process was shown in part 1, where the deployment.yaml file instructed the HelmController not to install CRDs. So the next step is naturally to create the Kustomization file to deploy the CRDs, which is the deployment-crds.yaml file listed below.

The deployment-crds.yaml file defines a Flux Kustomization resource in the flux-system namespace. It instructs Flux to apply manifests found in the /deploy/crds directory of the referenced GitRepository we created in the repositories.yaml file. In this case, it points to the GitHub repo for ESO itself. There is a lone file in that directory called bundle.yaml. This file, at well over 27,000 lines, deploys all the CRDs required for the CRs.

deployment-crds.yaml

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: external-secrets-crds
  namespace: flux-system
spec:
  interval: 10m
  path: ./deploy/crds
  prune: true
  sourceRef:
    kind: GitRepository
    name: external-secrets

Deploying Custom Resources (CRs)

For the deployment-crs.yaml we define another Flux Kustomization resource in the flux-system namespace. This file instructs Flux to apply resources found in the ./infrastructure/base/external/crs/ directory. But the magic is that it will only do this after ensuring the external-secrets-crs dependency is present. The source referenced is the flux-system GitRepository, which is the same repo that FluxCD watches to synchronize the cluster. So a reference to itself!

So the last step, detailed in the next section, is to place all the desired CRs to be deployed, for example the cluster-secret-store.yaml for reaching the 1Password SDK.

deployment-crs.yaml

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: external-secrets-crs
  namespace: flux-system
spec:
  dependsOn:
    - name: external-secrets-crds
  interval: 10m
  path: ./infrastructure/base/external/crs
  prune: true
  sourceRef:
    kind: GitRepository
    name: flux-system

CRs to Deploy

As pointed out, within the ./infrastructure/base/external/crs directory are all the CRs to be deployed in our cluster. For my cluster, this is my lone manifest with details on connecting with the 1Password SDK to my vault.

This file does a few things. First it's of kind ClusterSecretStore meaning it is available cluster-wide. To restrict it you can also use SecretStore to bind it to a single namespace. Next it declares the provider, onepasswordSDK as well as the vault name to be accessed. And finally it declares the secret reference to be used for authentication to the 1Password SDK.

cluster-secret-store.yaml

apiVersion: external-secrets.io/v1
kind: ClusterSecretStore
metadata:
  name: onep-store
spec:
  provider:
    onepasswordSDK:
      vault: Homelab
      auth:
        serviceAccountSecretRef:
          name: onep-creds
          key: OnePToken
          namespace: external-secrets

Finally, a Kustomization file in the crs directory ensures FluxCD deploys all manifests within the folder:

kustomization.yaml (./infrastructure/base/external-secrets/crs)

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - cluster-secret-store.yaml

Wrap-Up

To wrap things up: using FluxCD to orchestrate the deployment of ESO and its associated resources ensures a reliable and repeatable process for secrets management in Kubernetes. By separating the deployment of CRDs and CRs and leveraging GitOps principles, you gain fine-grained control over resource application order and configuration drift. This approach not only enhances security by keeping sensitive credentials out of your manifests, but also streamlines operations and makes your homelab infrastructure more robust and maintainable. Stay tuned for future posts where I’ll dive deeper into automating secret creation and integrating additional providers!