Skip to content

Secrets Management With External Secrets Operator and 1Password (part 1)

In this first part for my Secrets Management with External Secrets Operator (ESO) and 1Password series, I'm going to detail how to get ESO deployed through GitOps using Flux, Kustomization resources, and Helm resources. All of these configuration files can be found in my homelab GitHub repository located here: https://github.com/cyberwatchdoug/homelab/tree/main

What exactly is External Secrets Operator, and why should we use it? Great question. ESO is a Kubernetes operator that solves the dilemma of secrets management in Kubernetes from external sources. The list of providers is lengthy, but it includes important players like AWS, Google, Azure, HashiCorp, CyberArk, and 1Password. The goal of this operator is to synchronize secrets from these external sources into Kubernetes secrets, so they can be more easily accessed and used throughout the cluster.

First, let's take a look at the structure for this GitOps repository. In this post I'll detail most of these, with some detailed in my part 2 found here.

File and Folder Structure

infrastructure
├───base
│   └───external-secrets
│       │   deployment-crds.yaml
│       │   deployment-crs.yaml
│       │   kustomization.yaml
│       │   namespace.yaml
│       │   release.yaml
│       │   repositories.yaml
│       │
│       └───crs
│               cluster-secret-store.yaml
│               kustomization.yaml
│
└───staging
    └───external-secrets
            kustomization.yaml

Deploying External Secrets Operator

Kustomization Files Everywhere

Like most things in my homelab, it all starts with a Kustomization YAML file. In this case, I have a file infrastructure.yaml located in the clusters/staging/ directory, which is the same place where the apps.yaml Kustomization file lives. The sole purpose for this file is a Flux Kustomization resource which tells Flux to continuously apply and manage all Kubernetes manifests found in the path: field. In my case, it points to infrastructure/staging/.

infrastructure.yaml

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: infrastructure
  namespace: flux-system
spec:
  interval: 1h
  sourceRef:
    kind: GitRepository
    name: flux-system
  path: "./infrastructure/staging"
  prune: true
  timeout: 1m
  retryInterval: 1m

Up next is, you guessed it, another Kustomization YAML file! There is currently only one operator folder in the infrastructure directory: external-secrets. More to come later, but since this post is specific to ESO, it's all we need to discuss.

There is also only one file in the infrastructure/staging/external-secrets/ folder: the Kustomization YAML file. But this one is different from the one above. Can you spot the main difference? In the apiVersion, the one above is based on the Kustomize spec from FluxCD, while this one here is based on the Kustomize spec from Kubernetes. This can be confusing at first, because both FluxCD and Kubernetes have a Kustomization kind, even though they are technically completely different objects.

The important thing to know here is that the apiVersion provided will decide which Kubernetes controller gets the hand off.

The one detailed below (the K8s Kustomization) is for the native Kustomize API group and is used for templating and to organize our YAML manifests with patches (future topic). The only declaration inside is the resources section, which defines where the template needs to look for additional Kustomization files. In this case, it directs to the ../../base/external-secrets/ folder, which is two levels up (the ../../ bit).

kustomization.yaml infrastructure/staging/external-secrets

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ../../base/external-secrets/

And the last Kustomization YAML file for this section is another K8s Kustomize object reference. Like the previous one, it declares resources the template needs to go find and apply to the cluster.

kustomization.yaml infrastructure/base/external-secrets/

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - namespace.yaml
  - repositories.yaml
  - deployment-crds.yaml
  - release.yaml
  - deployment-crs.yaml

Namespace Setup

First in the list is our namespace.yaml which does just that: creates the namespace for ESO named external-secrets. In Kubernetes, a namespace is the mechanism for isolating groups of resources. In our case, we are isolating all these external-secrets resources together.

namespace.yaml

apiVersion: v1
kind: Namespace
metadata:
  name: external-secrets

Helm Setup

To get started using Helm, you'll first need to make sure your Kubernetes cluster has it installed. In my case, k3s comes with it included. You can check if your Kubernetes cluster has Helm by running the command helm version.

For my homelab, I'm configuring the Helm Repository using a FluxCD object HelmRepository that provides the url specific for ESO Helm charts. Because it's a FluxCD object, the HelmRepository source will be managed by Flux.

To list all Helm repos managed by Flux, use this command: flux get sources helm --all-namespace.

repositories.yaml

apiVersion: source.toolkit.fluxcd.io/v1
kind: HelmRepository
metadata:
  name: external-secrets
  namespace: flux-system
spec:
  interval: 10m
  url: https://charts.external-secrets.io
---
apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  name: external-secrets
  namespace: flux-system
spec:
  interval: 10m
  ref:
    tag: v0.20.0
  url: https://github.com/external-secrets/external-secrets

Now that we have declared the Helm repo that we want Flux to manage, we need another file: release.yaml. This defines a Flux HelmRelease resource object for deploying ESO into the Kubernetes cluster (using Helm for deployment in the end).

You'll see that we reference the HelmRepository created in the previous YAML file. We are also setting installCRDs: false to prevent a special race condition where the Custom Resources (CRs), namely SecretStore, and ClusterSecretStore, could be deployed before the Custom Resource Definitions (CRDs) which are needed to recognize them (the CRs). This will be explained in more detail in part 2.

release.yaml

apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: external-secrets
  namespace: flux-system
spec:
  # Override release name to avoid pattern Namespace-Release
  releaseName: external-secrets
  targetNamespace: external-secrets
  interval: 10m
  chart:
    spec:
      chart: external-secrets
      version: 0.20.1
      sourceRef:
        kind: HelmRepository
        name: external-secrets
        namespace: flux-system
  values:
    installCRDs: false
  install:
    createNamespace: true

Wrap-Up

With the foundational setup for External Secrets Operator in place, my homelab now has a GitOps-driven approach to deploying and managing ESO in my Kubernetes cluster using Flux and Helm. In part 2, I'll dive deeper into configuring ESO to actually sync secrets from 1Password, including the setup of ClusterSecretStores and the necessary Custom Resources. I'll also explain in more detail the race condition I alluded to. All of this to ensure you can securely manage secrets in Kubernetes clusters end-to-end.