Secrets Management in Production

Secrets;API keys, database passwords, encryption keys;are the crown jewels of your application. Hardcoding them in source code or configuration files is a security disaster waiting to happen. This guide covers modern secrets management practices.

The Problem with Secrets

What Goes Wrong

Many teams start by putting secrets in configuration files or environment variables committed to version control. This approach seems convenient but creates serious security risks.

# Don't do this
DATABASE_PASSWORD=supersecret123  # In .env committed to git
API_KEY=sk_live_abc123           # In source code

Consequences:

• Secrets in git history forever
• Leaked in logs, error messages
• Shared across environments
• No audit trail
• Difficult rotation

Secrets Management Principles

1. Never Commit Secrets

Your .gitignore should exclude all files that might contain secrets. The .env.example file provides a template without actual values.

# .gitignore
.env
.env.*
!.env.example
*.pem
*.key
credentials.json

2. Encrypt at Rest

Secrets should be encrypted in storage, not just protected by access control.

3. Audit Access

Every secret access should be logged: who, when, what.

4. Rotate Regularly

Secrets should be rotatable without application changes.

5. Principle of Least Privilege

Applications should only access secrets they need.

HashiCorp Vault

Architecture

HashiCorp Vault is the industry standard for secrets management. It provides a central place to store, access, and distribute secrets with fine-grained access control.

Application → Vault Agent → Vault Server → Storage Backend
                                        ↓
                                   Secrets Engine
                                   (KV, Database, AWS, etc.)

Basic Setup

Getting started with Vault involves enabling a secrets engine and storing your first secret. The following commands demonstrate the key-value secrets engine.

# Start Vault server
vault server -dev

# Enable KV secrets engine
vault secrets enable -path=secret kv-v2

# Store a secret
vault kv put secret/myapp/database \
    username=dbuser \
    password=supersecret

# Retrieve a secret
vault kv get secret/myapp/database

Dynamic Database Credentials

One of Vault's most powerful features is generating short-lived database credentials on demand. Instead of a single long-lived password, each application instance gets unique credentials that automatically expire.

# Configure database secrets engine
vault secrets enable database

vault write database/config/postgresql \
    plugin_name=postgresql-database-plugin \
    connection_url="postgresql://{{username}}:{{password}}@db:5432/myapp" \
    allowed_roles="myapp-role" \
    username="vault" \
    password="vaultpass"

# Create role with TTL
vault write database/roles/myapp-role \
    db_name=postgresql \
    creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; GRANT SELECT ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
    default_ttl="1h" \
    max_ttl="24h"

With this configuration, each credential lease lasts only one hour. If credentials are compromised, the exposure window is limited.

Laravel Integration

Integrating Vault with Laravel requires a service that fetches secrets and configures your application at boot time. This approach keeps secrets out of your configuration files entirely.

// config/services.php
return [
    'vault' => [
        'addr' => env('VAULT_ADDR', 'http://127.0.0.1:8200'),
        'token' => env('VAULT_TOKEN'),
    ],
];

// App\Services\VaultService
class VaultService
{
    private Client $client;

    public function __construct()
    {
        $this->client = new Client([
            'base_uri' => config('services.vault.addr'),
            'headers' => [
                'X-Vault-Token' => config('services.vault.token'),
            ],
        ]);
    }

    public function getSecret(string $path): array
    {
        $response = $this->client->get("/v1/secret/data/{$path}");
        $data = json_decode($response->getBody(), true);
        return $data['data']['data'];
    }

    public function getDatabaseCredentials(): array
    {
        $response = $this->client->get('/v1/database/creds/myapp-role');
        $data = json_decode($response->getBody(), true);

        return [
            'username' => $data['data']['username'],
            'password' => $data['data']['password'],
            'lease_duration' => $data['lease_duration'],
        ];
    }
}

// Service Provider
class VaultServiceProvider extends ServiceProvider
{
    public function boot(VaultService $vault)
    {
        if (app()->environment('production')) {
            $dbCreds = $vault->getDatabaseCredentials();

            config([
                'database.connections.mysql.username' => $dbCreds['username'],
                'database.connections.mysql.password' => $dbCreds['password'],
            ]);
        }
    }
}

The service provider only fetches secrets in production, allowing local development to use standard .env files.

AWS Secrets Manager

Storing Secrets

AWS Secrets Manager provides a managed solution that integrates well with other AWS services. Secrets are stored as JSON, making it easy to group related values.

# Create secret
aws secretsmanager create-secret \
    --name myapp/production/database \
    --secret-string '{"username":"dbuser","password":"supersecret"}'

# Update secret
aws secretsmanager put-secret-value \
    --secret-id myapp/production/database \
    --secret-string '{"username":"dbuser","password":"newsecret"}'

Laravel Integration

The AWS SDK makes it straightforward to fetch secrets at application boot time. You can fetch multiple secrets in parallel to minimize startup latency.

// App\Services\AwsSecretsService
use Aws\SecretsManager\SecretsManagerClient;

class AwsSecretsService
{
    private SecretsManagerClient $client;

    public function __construct()
    {
        $this->client = new SecretsManagerClient([
            'region' => config('services.aws.region'),
            'version' => 'latest',
        ]);
    }

    public function getSecret(string $secretId): array
    {
        $result = $this->client->getSecretValue([
            'SecretId' => $secretId,
        ]);

        return json_decode($result['SecretString'], true);
    }
}

// Usage in config
class AppServiceProvider extends ServiceProvider
{
    public function boot()
    {
        if (app()->environment('production')) {
            $secrets = app(AwsSecretsService::class);

            $dbSecrets = $secrets->getSecret('myapp/production/database');
            config([
                'database.connections.mysql.username' => $dbSecrets['username'],
                'database.connections.mysql.password' => $dbSecrets['password'],
            ]);

            $stripeSecrets = $secrets->getSecret('myapp/production/stripe');
            config([
                'services.stripe.secret' => $stripeSecrets['secret_key'],
            ]);
        }
    }
}

Note how secrets are organized by environment (production) and service (database, stripe). This naming convention makes it clear which secrets belong where.

Automatic Rotation

AWS Secrets Manager can automatically rotate secrets using Lambda functions. This example shows a rotation function for database credentials.

# Lambda rotation function
import boto3
import json
import string
import secrets

def lambda_handler(event, context):
    secret_id = event['SecretId']
    step = event['Step']

    client = boto3.client('secretsmanager')

    if step == 'createSecret':
        # Generate new password
        new_password = ''.join(
            secrets.choice(string.ascii_letters + string.digits)
            for _ in range(32)
        )

        current = client.get_secret_value(SecretId=secret_id)
        secret_data = json.loads(current['SecretString'])
        secret_data['password'] = new_password

        client.put_secret_value(
            SecretId=secret_id,
            SecretString=json.dumps(secret_data),
            VersionStage='AWSPENDING'
        )

    elif step == 'setSecret':
        # Update the database password
        pending = client.get_secret_value(
            SecretId=secret_id,
            VersionStage='AWSPENDING'
        )
        secret_data = json.loads(pending['SecretString'])
        update_database_password(secret_data)

    elif step == 'finishSecret':
        # Mark new version as current
        client.update_secret_version_stage(
            SecretId=secret_id,
            VersionStage='AWSCURRENT',
            MoveToVersionId=event['ClientRequestToken']
        )

The rotation happens in stages: first the new secret is created and marked as pending, then it is applied to the database, and finally it becomes the current version. This staged approach ensures no downtime during rotation.

Kubernetes Secrets

Basic Secrets

Kubernetes provides a native secrets mechanism. While not as feature-rich as Vault, it integrates seamlessly with Kubernetes workloads.

apiVersion: v1
kind: Secret
metadata:
  name: db-credentials
type: Opaque
stringData:
  username: dbuser
  password: supersecret

Using Secrets in Pods

You can inject secrets as environment variables or mount them as files. Environment variables are simpler but less secure since they may appear in logs or process listings.

apiVersion: v1
kind: Pod
metadata:
  name: myapp
spec:
  containers:
    - name: app
      image: myapp:latest
      env:
        - name: DB_USERNAME
          valueFrom:
            secretKeyRef:
              name: db-credentials
              key: username
        - name: DB_PASSWORD
          valueFrom:
            secretKeyRef:
              name: db-credentials
              key: password

External Secrets Operator

For production workloads, the External Secrets Operator bridges Kubernetes with external secret managers like AWS Secrets Manager or Vault. This gives you the best of both worlds: centralized secret management with native Kubernetes integration.

# Pull from AWS Secrets Manager
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: db-credentials
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets-manager
    kind: SecretStore
  target:
    name: db-credentials
  data:
    - secretKey: username
      remoteRef:
        key: myapp/production/database
        property: username
    - secretKey: password
      remoteRef:
        key: myapp/production/database
        property: password

The refreshInterval ensures secrets are automatically updated when they change in the external store.

Encryption Keys

Key Management

For sensitive data encryption, use envelope encryption. A data encryption key (DEK) encrypts the actual data, and a key encryption key (KEK) from your KMS encrypts the DEK.

// Envelope encryption pattern
class EncryptionService
{
    public function encrypt(string $plaintext): array
    {
        // Generate data encryption key (DEK)
        $dek = random_bytes(32);

        // Encrypt data with DEK
        $nonce = random_bytes(SODIUM_CRYPTO_SECRETBOX_NONCEBYTES);
        $ciphertext = sodium_crypto_secretbox($plaintext, $nonce, $dek);

        // Encrypt DEK with master key (KEK) from KMS
        $encryptedDek = $this->kms->encrypt($dek);

        return [
            'ciphertext' => base64_encode($ciphertext),
            'nonce' => base64_encode($nonce),
            'encrypted_dek' => base64_encode($encryptedDek),
        ];
    }

    public function decrypt(array $envelope): string
    {
        // Decrypt DEK with KMS
        $dek = $this->kms->decrypt(base64_decode($envelope['encrypted_dek']));

        // Decrypt data with DEK
        return sodium_crypto_secretbox_open(
            base64_decode($envelope['ciphertext']),
            base64_decode($envelope['nonce']),
            $dek
        );
    }
}

Envelope encryption means you can rotate the KEK without re-encrypting all data. Simply re-encrypt the DEKs with the new KEK.

Secret Rotation

Zero-Downtime Rotation

Rotating secrets without downtime requires a grace period where both old and new secrets are valid. This service implements that pattern for API keys.

class SecretRotationService
{
    public function rotateApiKey(): void
    {
        // 1. Generate new key
        $newKey = Str::random(64);

        // 2. Store both keys (grace period)
        $this->vault->put('api/keys', [
            'current' => $newKey,
            'previous' => $this->vault->get('api/keys')['current'],
        ]);

        // 3. Update external services
        $this->updateExternalServices($newKey);

        // 4. After grace period, remove old key
        dispatch(new RemoveOldApiKey())->delay(now()->addHours(24));
    }
}

// Validation accepts both during rotation
class ApiKeyValidator
{
    public function validate(string $key): bool
    {
        $keys = $this->vault->get('api/keys');
        return $key === $keys['current'] || $key === $keys['previous'];
    }
}

The 24-hour grace period gives external systems time to update to the new key. During this window, both keys are accepted.

Auditing

Access Logging

Every access to secrets should be logged for security auditing and compliance. Wrap your secrets service with an auditing decorator.

class AuditedVaultService
{
    public function getSecret(string $path): array
    {
        $secret = $this->vault->getSecret($path);

        Log::channel('audit')->info('Secret accessed', [
            'path' => $path,
            'user' => auth()->id(),
            'ip' => request()->ip(),
            'timestamp' => now()->toIso8601String(),
        ]);

        return $secret;
    }
}

Store audit logs separately from application logs and retain them according to your compliance requirements.

Best Practices Checklist

• Never commit secrets to version control
• Use a secrets manager (Vault, AWS Secrets Manager)
• Encrypt secrets at rest and in transit
• Implement secret rotation
• Audit all secret access
• Use short-lived credentials when possible
• Separate secrets by environment
• Use service accounts, not personal credentials
• Implement emergency rotation procedures
• Regular security audits

Conclusion

Secrets management is foundational to application security. Use dedicated secrets managers like HashiCorp Vault or AWS Secrets Manager instead of environment files. Implement rotation, audit access, and follow the principle of least privilege. The investment in proper secrets management pays off in reduced security risk and easier compliance.