HashiCorp Vault + External Secrets Operator on Kubernetes

A complete production setup guide for HashiCorp Vault on Kubernetes with External Secrets Operator (ESO). This covers HA Vault deployment, Kubernetes auth, Redis secret management, TLS configuration, Vault Agent with GitLab CI, and user/policy management.

---

Architecture Overview

Vault (HA) → External Secrets Operator → Kubernetes Secrets → Application Pods

---

Step 1: Deploy HashiCorp Vault

1.1 Create Namespace and Install Vault

# Create vault namespace
kubectl create namespace vault

# Add Helm repository
helm repo add hashicorp https://helm.releases.hashicorp.com
helm repo update

Create the values file:

cat > vault-values.yaml << EOF
global:
  enabled: true
  tlsDisable: true

server:
  image:
    repository: hashicorp/vault
    tag: 1.15.0

  ha:
    enabled: true
    replicas: 1
    raft:
      enabled: true
      setNodeId: true

  service:
    enabled: true
    type: ClusterIP
    port: 8200

ui:
  enabled: true
  serviceType: ClusterIP

resources:
  requests:
    memory: "256Mi"
    cpu: "250m"
  limits:
    memory: "512Mi"
    cpu: "500m"
EOF

# Install Vault
helm install vault hashicorp/vault -n vault -f vault-values.yaml

1.2 Initialize Vault

kubectl exec -n vault vault-0 -- vault operator init \
  -key-shares=1 \
  -key-threshold=1 \
  -format=json > vault-init.json

# Extract keys and token
ROOT_TOKEN=$(jq -r '.root_token' vault-init.json)
UNSEAL_KEY=$(jq -r '.unseal_keys_b64[0]' vault-init.json)

echo "Root Token: $ROOT_TOKEN"
echo "Unseal Key: $UNSEAL_KEY"

1.3 Unseal Vault

for pod in vault-0; do
  kubectl exec -n vault $pod -- vault operator unseal $UNSEAL_KEY
done

Expected output:

Key                     Value
Seal Type               shamir
Initialized             true
Sealed                  false
Storage Type            raft
HA Enabled              true
HA Mode                 active

# Verify status
kubectl exec -n vault vault-0 -- vault status

1.4 Configure Vault

# Login
kubectl exec -n vault vault-0 -- vault login $ROOT_TOKEN

# Enable Kubernetes auth
kubectl exec -n vault vault-0 -- vault auth enable kubernetes

# Configure Kubernetes auth
kubectl exec -n vault vault-0 -- vault write auth/kubernetes/config \
  kubernetes_host="https://192.168.17.128:6443" \
  kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt \
  token_reviewer_jwt=@/var/run/secrets/kubernetes.io/serviceaccount/token

# Enable KV secrets engine
kubectl exec -n vault vault-0 -- vault secrets enable -path=secret kv-v2

---

Step 2: Deploy External Secrets Operator

2.1 Install ESO

kubectl create namespace external-secrets

helm repo add external-secrets https://charts.external-secrets.io
helm repo update

helm install external-secrets \
  external-secrets/external-secrets \
  -n external-secrets \
  --set installCRDs=true

2.2 Create Service Account and RBAC

cat > eso-rbac.yaml << EOF
apiVersion: v1
kind: ServiceAccount
metadata:
  name: external-secrets
  namespace: external-secrets
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: external-secrets
rules:
- apiGroups: [""]
  resources: ["secrets"]
  verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
- apiGroups: [""]
  resources: ["services"]
  verbs: ["get"]
- apiGroups: ["external-secrets.io"]
  resources: ["externalsecrets"]
  verbs: ["get", "list", "watch"]
- apiGroups: ["external-secrets.io"]
  resources: ["externalsecrets/status"]
  verbs: ["get", "update", "patch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: external-secrets
subjects:
- kind: ServiceAccount
  name: external-secrets
  namespace: external-secrets
roleRef:
  kind: ClusterRole
  name: external-secrets
  apiGroup: rbac.authorization.k8s.io
EOF

kubectl apply -f eso-rbac.yaml

2.3 Create ESO Token Secret

cat > eso-token-secret.yaml << EOF
apiVersion: v1
kind: Secret
metadata:
  name: external-secrets-token
  namespace: external-secrets
  annotations:
    kubernetes.io/service-account.name: external-secrets
type: kubernetes.io/service-account-token
EOF

kubectl apply -f eso-token-secret.yaml

---

Step 3: Configure Vault for ESO Access

3.1 Create Redis Secrets in Vault

# Create redis credentials
kubectl exec -n vault vault-0 -- vault kv put secret/redis/main \
  password="redis-super-secret-password-123" \
  host="redis.redis.svc.cluster.local" \
  port="6379" \
  username="redis-user"

# Create redis config
kubectl exec -n vault vault-0 -- vault kv put secret/redis/config \
  maxmemory="1gb" \
  timeout="300" \
  requirepass="true"

# Verify
kubectl exec -n vault vault-0 -- vault kv list secret/
kubectl exec -n vault vault-0 -- vault kv get secret/redis/main

3.2 Create Vault Policy and Role

# Create global policy for ESO
kubectl exec -n vault vault-0 -- sh -c '
vault policy write eso-global-policy - <<EOF
path "secret/data/*" {
  capabilities = ["read"]
}
path "secret/metadata/*" {
  capabilities = ["list"]
}
EOF
'

# Create Kubernetes role for ESO
kubectl exec -n vault vault-0 -- \
  vault write auth/kubernetes/role/eso-global-role \
  bound_service_account_names=external-secrets \
  bound_service_account_namespaces=external-secrets \
  policies=eso-global-policy \
  ttl=1h

3.3 Create ClusterSecretStore

cat > vault-global.yaml << EOF
apiVersion: external-secrets.io/v1
kind: ClusterSecretStore
metadata:
  name: vault-global
spec:
  provider:
    vault:
      server: "http://vault.vault.svc.cluster.local:8200"
      path: "secret"
      version: "v2"
      auth:
        kubernetes:
          mountPath: "kubernetes"
          role: "eso-global-role"
EOF

kubectl apply -f vault-global.yaml

---

Step 4: Create Redis ExternalSecrets

kubectl create namespace redis

ExternalSecret — Redis Credentials

cat > redis-external-secret.yaml << EOF
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: redis-credentials
  namespace: redis
spec:
  refreshInterval: "1h"
  secretStoreRef:
    name: vault-global
    kind: ClusterSecretStore
  target:
    name: redis-credentials
    creationPolicy: Owner
  data:
  - secretKey: REDIS_PASSWORD
    remoteRef:
      key: redis/main
      property: password
  - secretKey: REDIS_HOST
    remoteRef:
      key: redis/main
      property: host
  - secretKey: REDIS_PORT
    remoteRef:
      key: redis/main
      property: port
  - secretKey: REDIS_USERNAME
    remoteRef:
      key: redis/main
      property: username
EOF

kubectl apply -f redis-external-secret.yaml

ExternalSecret — Redis Config

cat > redis-config-external-secret.yaml << EOF
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: redis-config
  namespace: redis
spec:
  refreshInterval: "1h"
  secretStoreRef:
    name: vault-global
    kind: ClusterSecretStore
  target:
    name: redis-config
    creationPolicy: Owner
  data:
  - secretKey: MAX_MEMORY
    remoteRef:
      key: redis/config
      property: maxmemory
  - secretKey: TIMEOUT
    remoteRef:
      key: redis/config
      property: timeout
  - secretKey: REQUIRE_PASS
    remoteRef:
      key: redis/config
      property: requirepass
EOF

kubectl apply -f redis-config-external-secret.yaml

Tip: To force an ExternalSecret to sync immediately:

kubectl annotate es <es_name> -n redis force-sync="$(date +%s)" --overwrite

---

Step 5: Deploy Redis Using Vault Secrets

cat > redis-deployment.yaml << EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: redis
  namespace: redis
  labels:
    app: redis
spec:
  replicas: 1
  selector:
    matchLabels:
      app: redis
  template:
    metadata:
      labels:
        app: redis
    spec:
      containers:
      - name: redis
        image: redis:7-alpine
        command: ["redis-server"]
        args:
        - "--requirepass"
        - "\$(REDIS_PASSWORD)"
        - "--maxmemory"
        - "\$(MAX_MEMORY)"
        - "--timeout"
        - "\$(TIMEOUT)"
        ports:
        - containerPort: 6379
        env:
        - name: REDIS_PASSWORD
          valueFrom:
            secretKeyRef:
              name: redis-credentials
              key: REDIS_PASSWORD
        - name: MAX_MEMORY
          valueFrom:
            secretKeyRef:
              name: redis-config
              key: MAX_MEMORY
        - name: TIMEOUT
          valueFrom:
            secretKeyRef:
              name: redis-config
              key: TIMEOUT
        volumeMounts:
        - name: redis-secrets
          mountPath: "/secrets"
          readOnly: true
        resources:
          requests:
            memory: "128Mi"
            cpu: "100m"
          limits:
            memory: "256Mi"
            cpu: "200m"
      volumes:
      - name: redis-secrets
        secret:
          secretName: redis-credentials
---
apiVersion: v1
kind: Service
metadata:
  name: redis
  namespace: redis
spec:
  selector:
    app: redis
  ports:
  - port: 6379
    targetPort: 6379
EOF

kubectl apply -f redis-deployment.yaml

---

Step 6: Verification and Testing

# Check ExternalSecrets status
kubectl get externalsecrets -n redis

# Check Kubernetes secrets
kubectl get secrets -n redis

# Check ESO logs
kubectl logs -n external-secrets deployment/external-secrets -f

# Check Redis deployment
kubectl get pods -n redis

# Verify secret content
kubectl get secret redis-credentials -n redis -o jsonpath='{.data.REDIS_PASSWORD}' | base64 -d
echo ""
kubectl get secret redis-credentials -n redis -o jsonpath='{.data.REDIS_HOST}' | base64 -d
echo ""

# Verify secrets used by Redis
kubectl exec -n redis deployment/redis -- env | grep REDIS

Test Secret Update Flow

# Update secret in Vault
kubectl exec -n vault vault-0 -- vault kv put secret/redis/main \
  password="new-updated-password-456" \
  host="redis.redis.svc.cluster.local" \
  port="6379" \
  username="redis-user"

# Watch for Kubernetes secret update (up to 1 minute)
kubectl get secret redis-credentials -n redis -w -o yaml

---

Enabling HTTPS / TLS

Generate Certificates

mkdir -p vault-certs && cd vault-certs

# CA key and cert
openssl genrsa -out ca.key 2048
openssl req -new -x509 -days 365 -key ca.key -out ca.crt \
  -subj "/C=US/ST=State/L=City/O=Organization/CN=Vault CA"

# Vault server key and CSR
openssl genrsa -out vault.key 2048
openssl req -new -key vault.key -out vault.csr \
  -subj "/C=US/ST=State/L=City/O=Organization/CN=vault.vault.svc.cluster.local" \
  -addext "subjectAltName = DNS:vault.vault.svc.cluster.local,DNS:vault,DNS:localhost,IP:127.0.0.1"

# Sign with CA
openssl x509 -req -days 365 -in vault.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out vault.crt \
  -extfile <(printf "subjectAltName=DNS:vault.vault.svc.cluster.local,DNS:vault,DNS:localhost,IP:127.0.0.1")

# Create Kubernetes TLS secret
kubectl create secret generic vault-tls -n vault \
  --from-file=vault.crt=vault.crt \
  --from-file=vault.key=vault.key \
  --from-file=ca.crt=ca.crt

Upgrade Vault with TLS Values

cat > vault-values-https.yaml << EOF
global:
  enabled: true
  tlsDisable: false

injector:
  enabled: true

server:
  image:
    repository: hashicorp/vault
    tag: 1.15.0
  extraEnvironmentVars:
    VAULT_CACERT: /vault/userconfig/vault-tls/ca.crt
    VAULT_TLSCERT: /vault/userconfig/vault-tls/vault.crt
    VAULT_TLSKEY: /vault/userconfig/vault-tls/vault.key
  volumes:
    - name: userconfig-vault-tls
      secret:
        defaultMode: 420
        secretName: vault-tls
  volumeMounts:
    - mountPath: /vault/userconfig/vault-tls
      name: userconfig-vault-tls
      readOnly: true
  ha:
    enabled: true
    replicas: 1
    raft:
      enabled: true
      setNodeId: true
      config: |
        cluster_name = "vault-integrated-storage"
        ui = true
        listener "tcp" {
          tls_disable = 0
          address = "[::]:8200"
          cluster_address = "[::]:8201"
          tls_cert_file = "/vault/userconfig/vault-tls/vault.crt"
          tls_key_file  = "/vault/userconfig/vault-tls/vault.key"
          tls_client_ca_file = "/vault/userconfig/vault-tls/ca.crt"
        }
        storage "raft" {
          path = "/vault/data"
        }
        disable_mlock = true
        service_registration "kubernetes" {}

ui:
  enabled: true
  serviceType: ClusterIP
EOF

# Scale down injector if running
kubectl scale deployment vault-agent-injector -n vault --replicas=0

helm upgrade vault hashicorp/vault -n vault -f vault-values-https.yaml

# Restart vault pods
for pod in vault-0; do
  kubectl delete pod -n vault $pod
done

Update ClusterSecretStore for HTTPS

apiVersion: external-secrets.io/v1
kind: ClusterSecretStore
metadata:
  name: vault-global
spec:
  provider:
    vault:
      server: "https://vault.vault.svc.cluster.local:8200"
      path: "secret"
      version: "v2"
      caProvider:
        type: "Secret"
        name: "vault-tls"
        key: "ca.crt"
        namespace: "vault"
      auth:
        kubernetes:
          mountPath: "kubernetes"
          role: "eso-global-role"

---

Vault with PostgreSQL Backend

vault-values.yaml (Postgres storage)

global:
  tlsDisable: true

server:
  ha:
    enabled: true
    replicas: 3
    config: |-
      ui = true

      listener "tcp" {
        tls_disable = 1
        address = "[::]:8200"
        cluster_address = "[::]:8201"
      }

      storage "postgresql" {
        connection_url = "postgres://vault:123456@192.168.17.128:5432/vault_db?sslmode=disable"
        table = "vault_kv_store"
        ha_enabled = true
        ha_table = "vault_ha_locks"
      }

  service:
    enabled: true
    type: ClusterIP

ui:
  enabled: true
  serviceType: NodePort

resources:
  requests:
    memory: "256Mi"
    cpu: "250m"
  limits:
    memory: "512Mi"
    cpu: "500m"

Create Required PostgreSQL Tables

CREATE TABLE vault_kv_store (
  parent_path TEXT COLLATE "C" NOT NULL,
  path        TEXT COLLATE "C",
  key         TEXT COLLATE "C",
  value       BYTEA,
  CONSTRAINT pkey PRIMARY KEY (path, key)
);

CREATE INDEX parent_path_idx ON vault_kv_store (parent_path);

CREATE TABLE vault_ha_locks (
  ha_key       TEXT COLLATE "C" NOT NULL,
  ha_identity  TEXT COLLATE "C" NOT NULL,
  ha_value     TEXT COLLATE "C",
  valid_until  TIMESTAMP WITH TIME ZONE NOT NULL,
  CONSTRAINT ha_key PRIMARY KEY (ha_key)
);

GRANT ALL PRIVILEGES ON DATABASE vault_db TO vault;
GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA public TO vault;

---

Vault Agent with GitLab CI

Install Vault Binary

curl -fsSL https://releases.hashicorp.com/vault/1.20.3/vault_1.20.3_linux_amd64.zip -o vault.zip
unzip vault.zip
mv vault /usr/bin

Configure vault-agent-config.hcl

exit_after_auth = false
pid_file = "/tmp/vault-agent.pid"

auto_auth {
  method "approle" {
    config = {
      role_id_file_path   = "/etc/vault/role_id"
      secret_id_file_path = "/etc/vault/secret_id"
    }
  }

  sink "file" {
    config = {
      path = "/etc/vault/agent-token"
    }
  }
}

vault {
  address = "https://192.168.17.128:31624"
  tls_skip_verify = true
}

Create AppRole and Policy

# Enable AppRole auth
kubectl exec -n vault vault-0 -- vault auth enable approle

# Create policy
cat > gitlab-policy.hcl <<EOF
path "docker-secret/data/credentials" {
  capabilities = ["read"]
}
EOF

kubectl cp gitlab-policy.hcl vault/vault-0:/tmp/gitlab-policy.hcl
kubectl exec -n vault vault-0 -- vault policy write gitlab-policy /tmp/gitlab-policy.hcl

# Create role
vault write auth/approle/role/gitlab-role \
  token_policies="gitlab-policy" \
  token_ttl=1h \
  token_max_ttl=4h

# Get RoleID and SecretID
vault read auth/approle/role/gitlab-role/role-id
vault write -f auth/approle/role/gitlab-role/secret-id

Set Up Agent Files

echo "<your-role-id>"   > /etc/vault/role_id
echo "<your-secret-id>" > /etc/vault/secret_id

sudo chown -R gitlab-runner:gitlab-runner /workdir/vault-agent
sudo chmod -R 755 /workdir/vault-agent
chmod 644 /etc/vault/agent-token

Systemd Service for Vault Agent

[Unit]
Description=HashiCorp Vault Agent
Requires=network-online.target
After=network-online.target

[Service]
User=root
Group=root
ExecStart=/usr/bin/vault agent -config=/workdir/vault-agent/vault-agent-config.hcl
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure
RestartSec=5s
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

Create Test Secret in Vault

vault secrets enable -path=docker-secret kv-v2

vault kv put docker-secret/credentials \
  username="myuser" \
  password="dckr_pat_xxxx"

GitLab CI Pipeline

stages:
  - push

variables:
  VAULT_ADDR: "https://192.168.17.128:31624"
  VAULT_TOKEN_FILE: "/etc/vault/agent-token"
  VAULT_SKIP_VERIFY: "true"
  TAG_NAME: ""

before_script:
  - echo "Setting up environment for Vault integration..."
  - export VAULT_TOKEN=$(cat ${VAULT_TOKEN_FILE})
  - export TAG_NAME="v$CI_PIPELINE_ID"

push_image:
  stage: push
  script:
    - |
      export DOCKER_USERNAME=$(vault kv get -field=username docker-secret/credentials) && \
      export DOCKER_PASSWORD=$(vault kv get -field=password docker-secret/credentials) && \
      echo "Logging in to Docker..."
    - echo $DOCKER_PASSWORD | docker login -u $DOCKER_USERNAME --password-stdin
  tags:
    - build

---

User and Policy Management

Create ACL Policies

In the Vault UI go to Policies → ACL Policies → Create ACL Policy:

# devopsfullpolicy
path "secret/metadata/*" {
  capabilities = ["list"]
}
path "secret/data/redis" {
  capabilities = ["create", "read", "update", "list"]
}

# dbreadpolicy
path "secret/metadata/*" {
  capabilities = ["list"]
}
path "secret/data/db_secret" {
  capabilities = ["read", "list"]
}

Create Groups, Entities, and Users

1. Groups: Go to Access → Groups → Create Group. Create devopsfull with devopsfullpolicy, and dbreadonly with dbreadpolicy.
2. Entities: Go to Access → Entities → Create Entity. Create entities for each user (e.g., mubin, shams).
3. Aliases: Add aliases to each entity with the auth method set to userpass.

# Create userpass users
kubectl exec -n vault vault-0 -- vault write auth/userpass/users/mubin password="test123"
kubectl exec -n vault vault-0 -- vault write auth/userpass/users/shams password="test123"

4. Attach entities to groups via Access → Groups → Edit Group.

Generate Token for a User

kubectl exec -n vault vault-0 -- vault login -method=userpass \
  username=shams \
  password="test123" \
  -format=json > shams-token.json

# Use the client_token field from the output
cat shams-token.json | jq -r '.auth.client_token'

---

Step 7: Cleanup

kubectl delete namespace redis
kubectl delete namespace external-secrets
kubectl delete namespace vault
helm delete vault -n vault
helm delete external-secrets -n external-secrets
rm -f vault-init.json vault-values.yaml eso-rbac.yaml vault-global.yaml redis-*.yaml

---

Summary

This setup provides a complete production-grade secret management pipeline:

| Component | Role | |-----------|------| | HashiCorp Vault | Central secret store with HA Raft or PostgreSQL backend | | External Secrets Operator | Auto-syncs secrets from Vault to Kubernetes | | ClusterSecretStore | Global Vault connection config for all namespaces | | ExternalSecret | Namespace-scoped secret mapping | | Vault Agent | Token provider for external tools like GitLab CI |

Secret flow: Vault → ESO → Kubernetes Secrets → Application Pods