Secrets Rotation: Automating Credential Lifecycle Management

In most organizations, secrets — database passwords, API keys, TLS certificates, signing keys — are rotated in one of two scenarios: after a suspected compromise, or when someone on the security team finally gets around to auditing them. Neither is acceptable.

A credential that was set once and never changed is a credential that, when eventually leaked, gives attackers an indefinitely valid key. Automating rotation eliminates the maintenance burden and makes frequent rotation practical.

Why Rotation Matters

Secrets can be leaked through multiple channels without anyone noticing:

• Committed to source control (even briefly, before reverting)
• Logged in plain text by a debugging statement
• Included in error messages sent to a monitoring service
• Exposed by a misconfigured secrets manager
• Exfiltrated by a supply chain compromise
• Visible in a screenshot shared on Slack

If you rotate secrets regularly, a leaked secret from 90 days ago is already expired. The window of exposure is limited to the rotation interval.

AWS Secrets Manager: Automated Rotation

AWS Secrets Manager supports automated rotation via Lambda functions. When you configure rotation, Secrets Manager calls your Lambda on the rotation schedule, passing four distinct steps:

# Lambda function for rotating a database password
import boto3
import json
import string
import secrets
import pymysql

def lambda_handler(event, context):
    arn = event['SecretId']
    token = event['ClientRequestToken']
    step = event['Step']

    client = boto3.client('secretsmanager')

    if step == 'createSecret':
        create_secret(client, arn, token)
    elif step == 'setSecret':
        set_secret(client, arn, token)
    elif step == 'testSecret':
        test_secret(client, arn, token)
    elif step == 'finishSecret':
        finish_secret(client, arn, token)

def create_secret(client, arn, token):
    """Generate a new password and store as AWSPENDING."""
    try:
        # Check if AWSPENDING already exists (rotation may have been interrupted)
        client.get_secret_value(SecretId=arn, VersionStage='AWSPENDING')
        return
    except client.exceptions.ResourceNotFoundException:
        pass

    # Get current secret to preserve non-password fields
    current = json.loads(client.get_secret_value(SecretId=arn)['SecretString'])

    # Generate new password
    alphabet = string.ascii_letters + string.digits + '!@#$%^&*'
    new_password = ''.join(secrets.choice(alphabet) for _ in range(32))

    current['password'] = new_password

    client.put_secret_value(
        SecretId=arn,
        ClientRequestToken=token,
        SecretString=json.dumps(current),
        VersionStages=['AWSPENDING'],
    )

def set_secret(client, arn, token):
    """Apply the new password to the database."""
    current = json.loads(client.get_secret_value(SecretId=arn)['SecretString'])
    pending = json.loads(
        client.get_secret_value(SecretId=arn, VersionStage='AWSPENDING')['SecretString']
    )

    connection = pymysql.connect(
        host=current['host'],
        user=current['username'],
        password=current['password'],
        database=current['dbname'],
    )

    with connection.cursor() as cursor:
        cursor.execute(
            "ALTER USER %s@'%%' IDENTIFIED BY %s",
            (pending['username'], pending['password'])
        )
    connection.commit()

def test_secret(client, arn, token):
    """Verify the new credentials actually work."""
    pending = json.loads(
        client.get_secret_value(SecretId=arn, VersionStage='AWSPENDING')['SecretString']
    )

    # This will raise an exception if the credentials don't work
    connection = pymysql.connect(
        host=pending['host'],
        user=pending['username'],
        password=pending['password'],
        database=pending['dbname'],
    )
    connection.close()

def finish_secret(client, arn, token):
    """Promote AWSPENDING to AWSCURRENT."""
    metadata = client.describe_secret(SecretId=arn)
    current_version = next(
        version for version, stages in metadata['VersionIdsToStages'].items()
        if 'AWSCURRENT' in stages
    )

    client.update_secret_version_stage(
        SecretId=arn,
        VersionStage='AWSCURRENT',
        MoveToVersionId=token,
        RemoveFromVersionId=current_version,
    )

Configure this rotation in Terraform:

resource "aws_secretsmanager_secret_rotation" "db_password" {
  secret_id           = aws_secretsmanager_secret.db_password.id
  rotation_lambda_arn = aws_lambda_function.rotate_secret.arn

  rotation_rules {
    automatically_after_days = 30
  }
}

Application-Side: Reading Rotating Secrets

Rotation only works if your application reads the current secret value dynamically rather than caching it at startup. A database password cached on boot is the old password 24 hours after rotation.

// Bad: secret cached at application boot
class DatabaseConfig
{
    private static string $password;

    public static function getPassword(): string
    {
        if (!isset(self::$password)) {
            self::$password = env('DB_PASSWORD'); // Cached forever
        }
        return self::$password;
    }
}

// Better: fetch from secrets manager with cache and automatic refresh
class SecretsManager
{
    private array $cache = [];
    private array $cacheExpiry = [];
    private int $cacheTtl = 300; // 5 minutes

    public function get(string $secretName): array
    {
        if ($this->isCached($secretName)) {
            return $this->cache[$secretName];
        }

        $value = $this->fetchFromAws($secretName);
        $this->cache[$secretName] = $value;
        $this->cacheExpiry[$secretName] = time() + $this->cacheTtl;

        return $value;
    }

    private function isCached(string $secretName): bool
    {
        return isset($this->cache[$secretName])
            && $this->cacheExpiry[$secretName] > time();
    }

    private function fetchFromAws(string $secretName): array
    {
        $client = new SecretsManagerClient(['region' => config('aws.region')]);

        $result = $client->getSecretValue(['SecretId' => $secretName]);

        return json_decode($result['SecretString'], true);
    }
}

For database connections, handle credential rotation gracefully by catching authentication failures and retrying with a fresh credential:

class RotatingDatabaseManager
{
    public function getConnection(): PDO
    {
        try {
            return $this->createConnection($this->secrets->get('prod/db/primary'));
        } catch (PDOException $e) {
            // Authentication failure — secrets may have just rotated
            if (str_contains($e->getMessage(), 'Access denied')) {
                $this->secrets->invalidateCache('prod/db/primary');
                return $this->createConnection($this->secrets->get('prod/db/primary'));
            }
            throw $e;
        }
    }
}

HashiCorp Vault: Dynamic Secrets

Vault takes a more powerful approach: rather than rotating stored passwords, it generates on-demand credentials that expire automatically.

Configure Vault's database secrets engine:

# Enable the database secrets engine
vault secrets enable database

# Configure a MySQL connection
vault write database/config/my-mysql \
    plugin_name=mysql-database-plugin \
    connection_url="{{username}}:{{password}}@tcp(db.internal:3306)/" \
    allowed_roles="app-role" \
    username="vault-admin" \
    password="vault-admin-password"

# Create a role that generates credentials with specific grants
vault write database/roles/app-role \
    db_name=my-mysql \
    creation_statements="CREATE USER '{{name}}'@'%' IDENTIFIED BY '{{password}}'; \
                         GRANT SELECT, INSERT, UPDATE ON mydb.* TO '{{name}}'@'%';" \
    revocation_statements="DROP USER IF EXISTS '{{name}}'@'%';" \
    default_ttl="1h" \
    max_ttl="24h"

Your application requests credentials when it needs a database connection:

class VaultDatabaseProvider
{
    private ?array $credentials = null;
    private ?int $expiresAt = null;

    public function getCredentials(): array
    {
        if ($this->credentials && $this->expiresAt > time() + 300) {
            return $this->credentials;
        }

        $response = Http::withToken($this->vaultToken)
            ->get("https://vault.internal/v1/database/creds/app-role");

        $data = $response->json();
        $this->credentials = $data['data'];
        $this->expiresAt = time() + $data['lease_duration'];

        return $this->credentials;
    }
}

Vault creates a real database user with a short TTL. When the TTL expires, Vault revokes it automatically. Your application never holds long-lived credentials.

Rotating API Keys

For API keys issued to external partners or services, build rotation into the key lifecycle:

class ApiKeyRotationService
{
    public function rotate(ApiKey $currentKey): ApiKey
    {
        DB::transaction(function () use ($currentKey, &$newKey) {
            // Create the new key
            $newKey = ApiKey::create([
                'user_id' => $currentKey->user_id,
                'scopes' => $currentKey->scopes,
                'name' => $currentKey->name . ' (rotated)',
                'expires_at' => now()->addDays(90),
                'previous_key_id' => $currentKey->id,
            ]);

            // Grace period: old key still works for 7 days
            $currentKey->update([
                'expires_at' => now()->addDays(7),
                'rotation_notified_at' => now(),
            ]);

            // Notify the key owner
            $currentKey->user->notify(new ApiKeyRotatedNotification($currentKey, $newKey));
        });

        return $newKey;
    }

    public function revokeExpiredKeys(): int
    {
        return ApiKey::where('expires_at', '<', now())
            ->whereNull('revoked_at')
            ->update(['revoked_at' => now()]);
    }
}

TLS Certificate Rotation

Expired TLS certificates cause outages. Automate renewal with Let's Encrypt and certbot:

# Install certbot and auto-renew
certbot certonly --nginx -d example.com -d www.example.com

# certbot installs a systemd timer or cron job for renewal
# Verify it exists:
systemctl list-timers | grep certbot

# Test renewal without actually renewing
certbot renew --dry-run

For wildcard certificates and internal CA, use cert-manager in Kubernetes:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: api-tls
  namespace: production
spec:
  secretName: api-tls-secret
  duration: 2160h  # 90 days
  renewBefore: 720h # Renew 30 days before expiry
  dnsNames:
    - api.example.com
  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer

cert-manager handles renewal automatically and updates the Kubernetes secret. Your deployments pick up the new certificate without manual intervention.

Secrets Rotation Checklist

For each class of secret in your system, document:

[ ] What is this secret? (database password, API key, signing key, TLS cert)
[ ] Where is it stored? (Secrets Manager, Vault, .env file, Kubernetes secret)
[ ] What rotation interval is appropriate? (daily, 30 days, 90 days, on demand)
[ ] Is rotation automated? If not, who rotates it and how?
[ ] How does the application get notified of rotation? (cache invalidation, restart)
[ ] What happens if rotation fails? (fallback, alerting, rollback)
[ ] Is there a grace period where both old and new credentials work?
[ ] Where is rotation logged? (audit trail for compliance)

Manual rotation is not a process — it is a hope. Automate your credential lifecycle and make rotation a non-event.

Building secure, reliable systems? We help teams deliver software they can trust. scopeforged.com