Secrets Management

Why Secrets Management Matters

HashiCorp Vault

Vault is an identity-based secrets and encryption management system. It provides a unified interface to any secret while providing tight access control and recording a detailed audit log.

Vault Architecture

• Server: Stateless process that handles API requests, authentication, and policy enforcement
• Storage Backend: Durable storage for encrypted data — Consul (HA), Integrated Raft (recommended), DynamoDB, GCS
• Auth Methods: How clients authenticate — Kubernetes, AWS IAM, GCP IAM, AppRole, OIDC, LDAP, GitHub
• Secret Engines: Plugins that generate, store, or transform secrets — KV, PKI, Database, AWS, GCP, SSH, Transit
• Policies: HCL or JSON rules defining what a token can access
• Audit Devices: Log all requests and responses — file, syslog, socket

Vault KV Secrets Engine v2

# Enable KV v2
vault secrets enable -path=secret kv-v2

# Write a secret
vault kv put secret/myapp/database \
  username="app_user" \
  password="s3cr3t-p@ssw0rd" \
  host="postgres.internal:5432"

# Read a secret
vault kv get secret/myapp/database

# Read specific version
vault kv get -version=3 secret/myapp/database

# Read as JSON (for scripting)
vault kv get -format=json secret/myapp/database | jq '.data.data'

# List secrets
vault kv list secret/myapp/

# Delete (soft delete - retains metadata and versions)
vault kv delete secret/myapp/database

# Permanently destroy a version
vault kv destroy -versions=1,2 secret/myapp/database

# Enable versioning with custom settings
vault kv metadata put \
  -max-versions=10 \
  -delete-version-after=720h \
  secret/myapp/database

PKI Secrets Engine (Internal CA)

# Enable and configure PKI engine as Root CA
vault secrets enable pki
vault secrets tune -max-lease-ttl=87600h pki

# Generate root certificate
vault write -field=certificate pki/root/generate/internal \
  common_name="Example Internal CA" \
  ttl=87600h \
  key_type=ec \
  key_bits=384 > ca.crt

# Configure CRL and OCSP URLs
vault write pki/config/urls \
  issuing_certificates="https://vault.internal:8200/v1/pki/ca" \
  crl_distribution_points="https://vault.internal:8200/v1/pki/crl"

# Create a role for issuing certs
vault write pki/roles/internal-services \
  allowed_domains="internal,svc.cluster.local" \
  allow_subdomains=true \
  max_ttl=8760h \
  key_type=ec \
  key_bits=256

# Issue a certificate
vault write pki/issue/internal-services \
  common_name="myapp.internal" \
  alt_names="myapp.svc.cluster.local" \
  ttl=2160h

Dynamic Secrets — Database

# Enable the database secrets engine
vault secrets enable database

# Configure PostgreSQL connection
vault write database/config/myapp-postgres \
  plugin_name=postgresql-database-plugin \
  allowed_roles="app-role,readonly-role" \
  connection_url="postgresql://{{username}}:{{password}}@postgres.internal:5432/myapp" \
  username="vault_admin" \
  password="vault-admin-password"

# Create a role that generates dynamic credentials
vault write database/roles/app-role \
  db_name=myapp-postgres \
  creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' \
    VALID UNTIL '{{expiration}}'; GRANT SELECT, INSERT, UPDATE, DELETE ON ALL TABLES \
    IN SCHEMA public TO \"{{name}}\";" \
  revocation_statements="DROP ROLE IF EXISTS \"{{name}}\";" \
  default_ttl=1h \
  max_ttl=24h

# Application requests dynamic credentials
vault read database/creds/app-role
# Returns: username=v-app-role-xyz123, password=A1B2C3..., lease_duration=1h

Kubernetes Auth Method

# Enable Kubernetes auth
vault auth enable kubernetes

# Configure with cluster info
vault write auth/kubernetes/config \
  kubernetes_host="https://kubernetes.default.svc:443" \
  kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt \
  token_reviewer_jwt=@/var/run/secrets/kubernetes.io/serviceaccount/token \
  issuer="https://kubernetes.default.svc.cluster.local"

# Create a policy
vault policy write myapp-policy - <<EOF
path "secret/data/myapp/*" {
  capabilities = ["read", "list"]
}
path "database/creds/app-role" {
  capabilities = ["read"]
}
EOF

# Create auth role binding KSA to policy
vault write auth/kubernetes/role/myapp \
  bound_service_account_names=myapp-sa \
  bound_service_account_namespaces=production \
  policies=myapp-policy \
  ttl=1h

Vault Agent Sidecar Injection

# Pod annotation for Vault Agent sidecar injection
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
  namespace: production
spec:
  template:
    metadata:
      annotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/role: "myapp"
        vault.hashicorp.com/agent-inject-secret-config: "secret/data/myapp/config"
        vault.hashicorp.com/agent-inject-template-config: |
          {{- with secret "secret/data/myapp/config" -}}
          export DB_HOST="{{ .Data.data.db_host }}"
          export DB_PASSWORD="{{ .Data.data.db_password }}"
          export API_KEY="{{ .Data.data.api_key }}"
          {{- end }}
        vault.hashicorp.com/agent-pre-populate-only: "false"
        vault.hashicorp.com/agent-inject-status: "update"
    spec:
      serviceAccountName: myapp-sa
      containers:
        - name: myapp
          image: myapp:latest
          command: ["/bin/sh", "-c"]
          args: ["source /vault/secrets/config && ./myapp"]

SOPS — Secrets OPerationS

SOPS is a tool for encrypting files. It supports AWS KMS, GCP KMS, Azure Key Vault, age, and PGP. Unlike Vault, SOPS is file-based — ideal for storing encrypted secrets in Git.

SOPS Configuration (.sops.yaml)

# .sops.yaml — place in repo root
creation_rules:
  # Production secrets: require AWS KMS (region-specific)
  - path_regex: .*/production/.*\.(yaml|env)$
    kms: arn:aws:kms:ap-southeast-1:123456789012:key/mrk-abc123def456
    gcp_kms: projects/my-project/locations/asia-southeast1/keyRings/sops-keyring/cryptoKeys/sops-key

  # Staging: use GCP KMS
  - path_regex: .*/staging/.*\.yaml$
    gcp_kms: projects/my-staging-project/locations/global/keyRings/sops/cryptoKeys/sops-key

  # Developer local: use age
  - path_regex: .*/dev/.*\.yaml$
    age: age1ql3z7hjy54pw3hyww5ayyfg7zqgvc7w3j2elw8zmrj2kg5sfn9aqmcac8p

  # All other files: require multiple keys (m-of-n)
  - path_regex: .*\.enc\.yaml$
    kms:
      - arn: arn:aws:kms:ap-southeast-1:123456789012:key/key1
      - arn: arn:aws:kms:ap-southeast-1:123456789012:key/key2
    shamir_threshold: 2

SOPS Encrypt/Decrypt Commands

# Encrypt a YAML file in-place
sops --encrypt --in-place secrets/production/database.yaml

# Encrypt only specific keys (leave others in plaintext)
sops --encrypt \
  --encrypted-regex "^(password|secret|token|key)$" \
  --in-place secrets/production/config.yaml

# Decrypt to stdout (for piping)
sops --decrypt secrets/production/database.yaml

# Decrypt to a file
sops --decrypt --output /tmp/decrypted.yaml secrets/production/database.yaml

# Edit an encrypted file (opens in $EDITOR, re-encrypts on save)
sops secrets/production/database.yaml

# Rotate encryption keys (re-encrypt with new keys)
sops rotate --kms arn:aws:kms:ap-southeast-1:123456789012:key/new-key-id \
  --in-place secrets/production/database.yaml

# Encrypt a .env file
sops --encrypt .env.production > .env.production.enc

SOPS with Helm

# Install helm-secrets plugin
helm plugin install https://github.com/jkroepke/helm-secrets

# Directory structure
helmfile.yaml
secrets/
  production/
    values.enc.yaml   # SOPS-encrypted

# Deploy with decrypted values
helm secrets upgrade myapp ./charts/myapp \
  --namespace production \
  -f values.yaml \
  -f secrets/production/values.enc.yaml

# helmfile.yaml
releases:
  - name: myapp
    chart: ./charts/myapp
    values:
      - values.yaml
      - secrets://secrets/production/values.enc.yaml

AWS Secrets Manager

Create and Access Secrets

# Create a secret
aws secretsmanager create-secret \
  --name "production/myapp/database" \
  --description "Database credentials for myapp production" \
  --secret-string '{"username":"app_user","password":"s3cr3t-p@ssw0rd","host":"prod-db.cluster.ap-southeast-1.rds.amazonaws.com"}' \
  --kms-key-id arn:aws:kms:ap-southeast-1:123456789012:key/mrk-abc123 \
  --tags Key=Environment,Value=production Key=Application,Value=myapp

# Get secret value
aws secretsmanager get-secret-value \
  --secret-id "production/myapp/database" \
  --query SecretString \
  --output text | jq .

# Update a secret
aws secretsmanager put-secret-value \
  --secret-id "production/myapp/database" \
  --secret-string '{"username":"app_user","password":"new-s3cr3t","host":"prod-db.rds.amazonaws.com"}'

# List versions
aws secretsmanager list-secret-version-ids \
  --secret-id "production/myapp/database"

Python SDK Access

import boto3
import json
from botocore.exceptions import ClientError

def get_secret(secret_name: str, region: str = "ap-southeast-1") -> dict:
    """Retrieve and parse a JSON secret from AWS Secrets Manager."""
    client = boto3.client("secretsmanager", region_name=region)

    try:
        response = client.get_secret_value(SecretId=secret_name)
    except ClientError as e:
        error_code = e.response["Error"]["Code"]
        if error_code == "ResourceNotFoundException":
            raise ValueError(f"Secret {secret_name} not found") from e
        elif error_code == "AccessDeniedException":
            raise PermissionError(f"No access to secret {secret_name}") from e
        raise

    secret_string = response.get("SecretString")
    if secret_string:
        return json.loads(secret_string)
    raise ValueError("Secret is binary, not string")

# Usage
db_creds = get_secret("production/myapp/database")
db_url = f"postgresql://{db_creds['username']}:{db_creds['password']}@{db_creds['host']}/myapp"

Automatic Rotation with Lambda

# Enable rotation with a managed Lambda rotation function
aws secretsmanager rotate-secret \
  --secret-id "production/myapp/database" \
  --rotation-lambda-arn arn:aws:lambda:ap-southeast-1:123456789012:function:SecretsManagerRDSPostgreSQLRotationSingleUser \
  --rotation-rules AutomaticallyAfterDays=30

# IAM policy for application to read secrets
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowSecretsAccess",
      "Effect": "Allow",
      "Action": [
        "secretsmanager:GetSecretValue",
        "secretsmanager:DescribeSecret"
      ],
      "Resource": "arn:aws:secretsmanager:ap-southeast-1:123456789012:secret:production/myapp/*",
      "Condition": {
        "StringEquals": {
          "aws:RequestedRegion": "ap-southeast-1"
        }
      }
    },
    {
      "Sid": "AllowKMSDecrypt",
      "Effect": "Allow",
      "Action": ["kms:Decrypt", "kms:GenerateDataKey"],
      "Resource": "arn:aws:kms:ap-southeast-1:123456789012:key/mrk-abc123"
    }
  ]
}

GCP Secret Manager

# Create a secret
echo -n "s3cr3t-value" | gcloud secrets create my-api-key \
  --data-file=- \
  --replication-policy=user-managed \
  --locations=asia-southeast1 \
  --labels=env=production,app=myapp

# Add a new version (rotation)
echo -n "new-s3cr3t-value" | gcloud secrets versions add my-api-key \
  --data-file=-

# Access the latest version
gcloud secrets versions access latest --secret=my-api-key

# Access a specific version
gcloud secrets versions access 3 --secret=my-api-key

# Grant a service account access to a secret
gcloud secrets add-iam-policy-binding my-api-key \
  --member="serviceAccount:[email protected]" \
  --role="roles/secretmanager.secretAccessor"

# Disable an old version after rotation
gcloud secrets versions disable 1 --secret=my-api-key

GCP Secret Manager in Python

from google.cloud import secretmanager

def access_secret(project_id: str, secret_id: str, version: str = "latest") -> str:
    """Access a secret version from GCP Secret Manager."""
    client = secretmanager.SecretManagerServiceClient()
    name = f"projects/{project_id}/secrets/{secret_id}/versions/{version}"
    response = client.access_secret_version(request={"name": name})
    return response.payload.data.decode("UTF-8")

# Usage in application startup
import os
PROJECT_ID = os.environ.get("GCP_PROJECT_ID", "my-project")
DB_PASSWORD = access_secret(PROJECT_ID, "myapp-db-password")
API_KEY = access_secret(PROJECT_ID, "myapp-third-party-api-key")

Kubernetes Secrets

Enable Envelope Encryption for etcd

# EncryptionConfiguration for EKS / self-managed clusters
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
      - configmaps
    providers:
      - kms:
          name: aws-encryption-provider
          endpoint: unix:///var/run/kmsplugin/socket.sock
          cachesize: 1000
          timeout: 3s
      - identity: {}  # Fallback for decryption of older data

External Secrets Operator (ESO)

ESO is a Kubernetes operator that syncs secrets from external secret management systems (AWS SM, GCP SM, Vault, Azure Key Vault) into Kubernetes Secrets. Applications read native K8s Secrets — no SDK changes needed.

# SecretStore: Connection to AWS Secrets Manager
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: aws-secrets-manager
  namespace: production
spec:
  provider:
    aws:
      service: SecretsManager
      region: ap-southeast-1
      auth:
        jwt:
          serviceAccountRef:
            name: external-secrets-sa  # Uses IRSA / Workload Identity
---
# ExternalSecret: Sync a specific secret
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: myapp-database-credentials
  namespace: production
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets-manager
    kind: SecretStore
  target:
    name: myapp-db-secret  # K8s Secret name to create
    creationPolicy: Owner
    template:
      type: Opaque
      data:
        DATABASE_URL: "postgresql://{{ .username }}:{{ .password }}@{{ .host }}/myapp"
        DB_PASSWORD: "{{ .password }}"
  data:
    - secretKey: username
      remoteRef:
        key: production/myapp/database
        property: username
    - secretKey: password
      remoteRef:
        key: production/myapp/database
        property: password
    - secretKey: host
      remoteRef:
        key: production/myapp/database
        property: host

# ClusterSecretStore: Vault backend (cluster-wide)
apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: vault-backend
spec:
  provider:
    vault:
      server: "https://vault.internal:8200"
      path: "secret"
      version: "v2"
      auth:
        kubernetes:
          mountPath: "kubernetes"
          role: "external-secrets"
          serviceAccountRef:
            name: "external-secrets-sa"
            namespace: "external-secrets"
---
# ExternalSecret from Vault
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: myapp-vault-secrets
  namespace: production
spec:
  refreshInterval: 15m
  secretStoreRef:
    name: vault-backend
    kind: ClusterSecretStore
  target:
    name: myapp-secrets
    creationPolicy: Owner
  dataFrom:
    - extract:
        key: myapp/config  # Extracts all key-value pairs from this Vault path

Secret Rotation Patterns

Zero-Downtime Rotation

# AWS: Automated rotation with version staging
# AWS SM uses AWSPENDING, AWSCURRENT, AWSPREVIOUS staging labels

# Step 1: Create new version (AWSPENDING)
aws secretsmanager put-secret-value \
  --secret-id production/myapp/api-key \
  --secret-string "new-api-key-value" \
  --version-stages AWSPENDING

# Step 2: Test the new secret works
# (Lambda rotation function does this)

# Step 3: Move AWSCURRENT to AWSPREVIOUS, AWSPENDING to AWSCURRENT
aws secretsmanager update-secret-version-stage \
  --secret-id production/myapp/api-key \
  --version-stage AWSCURRENT \
  --move-to-version-id <new-version-id> \
  --remove-from-version-id <current-version-id>

# Applications that cache secrets will fall back to AWSPREVIOUS if needed
# Old secret remains accessible as AWSPREVIOUS for a grace period