Injecting Secrets into Infrastructure: Vault, KMS, and SSM Patterns

Infrastructure needs secrets. Databases need passwords, applications need API keys, services need certificates. The question is how the secrets get from your secret store to the infrastructure that needs them — without leaking the secrets along the way and without checking them into Git.

This post is the patterns for injecting secrets into infrastructure: what works, what looks secure but is not, and the architectural choices that determine your blast radius if something goes wrong.

The Two Distribution Models

Two patterns dominate:

Push. The deployment system writes secrets into the workload's environment at deploy time. Terraform reads a secret from Vault and sets it as an EC2 user data variable, for example.

Pull. The workload fetches its own secrets at runtime. A pod uses a sidecar that talks to Vault and writes secrets to a file.

Push is simpler. Pull is more secure (secrets do not pass through the deployment system).

For modern setups, pull is usually preferred. The deployment system never sees the secret values — it only configures the workload to fetch them.

Vault as the Source of Truth

HashiCorp Vault dominates the secret-store space. Other options (AWS Secrets Manager, GCP Secret Manager, Azure Key Vault, Conjur) exist; Vault is the most flexible and most popular for multi-cloud or hybrid setups.

Vault provides:

• Centralized secret storage
• Authentication backends (Kubernetes, AWS IAM, OIDC, etc.)
• Dynamic secrets (Vault generates a fresh database credential per request)
• Lease and revocation (secrets have TTLs)
• Audit logging (every secret access is logged)
• Encryption as a service (encrypt without storing the key)

For cloud-native deployments: the cloud-provider secret manager is simpler, but Vault is more featureful.

Vault Agent for Kubernetes

The canonical pattern for getting secrets into Kubernetes pods uses Vault Agent as a sidecar.

apiVersion: v1
kind: Pod
metadata:
  annotations:
    vault.hashicorp.com/agent-inject: "true"
    vault.hashicorp.com/role: "my-app"
    vault.hashicorp.com/agent-inject-secret-db: "database/creds/my-app"
    vault.hashicorp.com/agent-inject-template-db: |
      {{ with secret "database/creds/my-app" -}}
      DB_USERNAME="{{ .Data.username }}"
      DB_PASSWORD="{{ .Data.password }}"
      {{- end }}
spec:
  containers:
    - name: my-app
      image: my-app:1.0
      command: ["sh", "-c", "source /vault/secrets/db && /app/start.sh"]

The Vault Agent (injected as an init container and sidecar) authenticates to Vault, fetches secrets, and writes them to /vault/secrets/db for the application to read. The application never has Vault credentials directly.

For long-running applications, the Agent refreshes secrets periodically. Database credentials with short TTLs rotate automatically.

External Secrets Operator

For Kubernetes-native secret management, External Secrets Operator (ESO) bridges Kubernetes Secrets to external stores.

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: my-app-secrets
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets-manager
    kind: SecretStore
  target:
    name: my-app-secrets
  data:
    - secretKey: db_password
      remoteRef:
        key: prod/my-app/database
        property: password

ESO syncs the external secret to a Kubernetes Secret. Pods consume the Kubernetes Secret normally. The advantage: applications do not need vault-specific clients.

ESO supports many backends: AWS Secrets Manager, GCP Secret Manager, Azure Key Vault, HashiCorp Vault, Akeyless, and others. It is the right pattern for "we want Kubernetes Secrets but we do not want the source of truth to be in etcd."

Cloud-Native Patterns

For workloads on a single cloud, the cloud's secret manager + native integration is the simplest.

AWS:

• Secrets Manager stores secrets
• IAM roles authorize access
• ECS task definitions inject secrets directly: secrets: [{name: "DB_PASSWORD", valueFrom: "arn:aws:secretsmanager:..."}]
• Lambda has similar integration

GCP:

• Secret Manager stores secrets
• Workload Identity authorizes pod access
• Direct mounting via the Secret Manager CSI driver

Azure:

• Key Vault stores secrets
• Managed Identities for authorization
• Direct integration with App Services

For pure single-cloud deployments, these are simpler than running Vault. For multi-cloud or hybrid, Vault or ESO is the right call.

What Never to Do

A few patterns that need to die:

Secrets in environment variables in deployment configs. A Kubernetes Deployment YAML with value: "supersecret" is a secret in your Git repository. Every developer who has read access to the repo has read access to the secret. No.

Secrets in CI environment variables that are echoed in logs. A common pattern: echo "Deploying with TOKEN=$TOKEN" to confirm the token is set. The token is now in logs forever. Mask or do not log.

Secrets in container images. RUN echo "API_KEY=xyz" > /etc/api.conf. The secret is in every layer of every image forever. No.

Secrets in code repositories. Even encrypted. Encryption keys leak; encrypted secrets become plaintext eventually. Just do not commit them.

Long-lived API keys for service authentication. Use short-lived tokens (OIDC, IAM roles, service account tokens) instead.

Rotation

Secrets should rotate. Long-lived secrets that never rotate are credentials waiting to leak.

Patterns for rotation:

Dynamic secrets. Vault's killer feature. The application requests a database credential; Vault creates a fresh user, returns the password, and revokes after a TTL. No long-lived credentials to leak.

Scheduled rotation. A weekly job rotates the secret in the store; deployments pick it up on the next deploy or restart.

On-demand rotation. A button (or runbook) rotates a specific secret immediately. Used after a suspected leak or personnel change.

For most secrets: scheduled rotation is enough. For high-stakes ones (root database passwords, signing keys): dynamic if possible, otherwise frequent rotation.

Audit and Observability

Every secret access should be logged.

• Vault audit logs record every access with who, when, what.
• AWS Secrets Manager records access in CloudTrail.
• GCP Secret Manager logs to Cloud Logging.

These logs are essential for incident response. When a secret leaks, you want to know who and what accessed it before the breach.

Alerts on unusual access patterns: a spike in secret reads, access from unfamiliar identities, access at unusual times. These often indicate compromise.

Network Boundaries

The secrets store should be reachable only by the things that need it. Common patterns:

• Vault accessible only inside the VPC, not from the public internet.
• Cloud secret managers accessible only via VPC endpoints (no public traffic).
• Service-to-Vault authentication via short-lived tokens, not long-lived credentials.

A secret store that is publicly accessible is a target. Even with strong authentication, network-level isolation is defense in depth.

Bootstrapping

The chicken-and-egg problem: how does the workload authenticate to the secret store?

The pattern: use the platform's identity primitive.

• Kubernetes: the pod's service account becomes its identity. Vault's Kubernetes auth method verifies the service account token.
• AWS: the instance's IAM role authenticates to AWS Secrets Manager.
• GCP: the workload's identity (Workload Identity) authenticates to Secret Manager.

These platforms provide identity that is hard to forge. The bootstrap problem is solved by trusting the platform's identity assertion.

What State Looks Like in Terraform

Terraform with secrets:

# Read a secret from Vault at apply time
data "vault_generic_secret" "db_password" {
  path = "secret/prod/database"
}

resource "aws_db_instance" "main" {
  username = "app_user"
  password = data.vault_generic_secret.db_password.data["password"]
}

The secret enters Terraform state. The state file now contains the secret in plaintext.

Mitigations:

• Encrypt state at rest (S3 SSE).
• Restrict who can read state.
• Avoid passing secrets through Terraform when possible — let the workload fetch them at runtime.

The cleanest pattern: Terraform creates the secret in the secret store; workloads fetch the secret directly from the store. Terraform writes; the workload reads. Terraform state has limited exposure.

When Things Go Wrong

Despite the best practices, secrets do leak. When they do:

1. Rotate the secret immediately. The leaked credential is no longer valid.
2. Audit access logs. Did anyone use the leaked credential before rotation?
3. Rotate adjacent secrets. If one secret leaked, others in the same context might have too.
4. Find the leak source. A bad commit, a logging mistake, a misconfigured endpoint.

The faster you can rotate secrets, the less damage a leak does. Practice rotation; do not let the first rotation happen under incident pressure.

A Practical Setup

For a typical Kubernetes deployment on AWS:

1. AWS Secrets Manager stores secrets.
2. External Secrets Operator syncs to Kubernetes Secrets.
3. Kubernetes Secrets mount into pods as env vars or files.
4. IRSA authorizes ESO to read from Secrets Manager.
5. Scheduled rotation for database credentials via Secrets Manager's built-in rotation.
6. CloudTrail logs all access.
7. State encrypted in S3 with versioning.

This setup is achievable in a week. The patterns scale from a single service to a large fleet.

For multi-cloud or with Vault: similar pattern with Vault in place of Secrets Manager. The principles are the same.

Designing secrets management for a new platform or auditing an existing setup that has accumulated risks? We help teams pick patterns that match their threat model and operational maturity. scopeforged.com