Rotation is easy to describe and hard to ship without downtime. The mechanics — click "new key", paste it somewhere — take thirty seconds. What breaks production is everything around them: the cron job nobody remembered, the old laptop still holding a copy, the partner webhook still signed with the previous secret, the container image with the value baked in at build time.

This is an order-of-operations guide. The single rule that prevents most rotation incidents: the new credential must be live everywhere before the old one dies. Everything below is a consequence of that rule.

1. The golden sequence

  1. Inventory every place the secret lives.
  2. Issue the new credential with narrow scope.
  3. Deploy new values everywhere, old one still valid.
  4. Verify real traffic succeeds on the new credential.
  5. Revoke the old credential.
  6. Document what you did, when, and who owns it.

Steps 3 and 5 are separated by an overlap window — a period where both credentials work. That window is the entire safety mechanism. Skipping it converts a routine maintenance task into an outage.

Revoke-first is correct in exactly one case: the key is being actively abused right now and the cost of continued abuse exceeds the cost of downtime. A leaked Stripe secret key draining a balance, an AWS key spawning mining instances. Even then, open a vendor support channel before you revoke, so you have a path back if you break something you didn't expect.

Why inventory is the hard step

Everyone underestimates this. A single API key in a modest production system commonly lives in:

  • The CI provider's secret store (GitHub Actions, GitLab CI, CircleCI)
  • The runtime environment (ECS task definition, Kubernetes Secret, Heroku config, Vercel/Netlify env vars)
  • A staging environment with a separate copy that drifted months ago
  • Two or three developer laptops, in .env files and shell profiles
  • A local secret manager or password manager entry
  • Terraform or Pulumi state, if it was ever passed as a variable
  • A container image, if someone used ARG/ENV at build time
  • Serverless function config, per region
  • A monitoring or alerting integration nobody owns
  • Someone's Slack DM from onboarding

Grep is your friend, and it should run against more than your main repo:

# Search working tree for the key's distinctive prefix (not the full secret)
rg -n --hidden --glob '!.git' 'sk_live_|AKIA|ghp_|glpat-'

# Search all branches and history, not just HEAD
git log --all -p -S 'AKIA' --oneline

# Check whether the value is baked into a built image
docker history --no-trunc myimage:latest | rg -i 'ARG|ENV'

Write the inventory down. You will need it again in six months, and the list is the deliverable that makes the next rotation cheap.

2. Scope the new credential smaller than the old one

Rotation is the natural moment to reduce blast radius, because you are already touching every consumer. If the old key was a root-ish token used by four services, issue four scoped keys instead of one replacement.

Concretely:

  • Stripe: restricted keys with per-resource read/write permissions instead of a full secret key.
  • AWS: an IAM role assumed via OIDC instead of a user with static keys. If you must keep static keys, attach a policy scoped to the specific resources, not *.
  • GitHub: a fine-grained personal access token scoped to specific repositories, or better, a GitHub App installation token.
  • Databases: a per-service user with only the tables and verbs it needs, not the owner account.

One key per consumer costs slightly more bookkeeping and buys you the ability to rotate one service without touching the others. It also makes the next leak far cheaper to contain, because revoking one key doesn't take down everything.

3. Stripe: keys and webhook signing secrets

These are two different rotations with different mechanics, and conflating them is a common mistake.

API keys. Stripe supports rolling a secret key with a grace period rather than an instant cut: you generate the replacement and choose how long the old key stays valid. That grace window is your overlap — use it instead of trying to coordinate a simultaneous deploy. Deploy the new key, watch for errors, then let the old one expire (or expire it early once you're confident).

Webhook signing secrets (whsec_…) are per-endpoint and verify inbound requests, so the overlap has to live in your verifier:

// Accept either secret during the migration window.
import Stripe from 'stripe'

const SECRETS = [
  process.env.STRIPE_WEBHOOK_SECRET,
  process.env.STRIPE_WEBHOOK_SECRET_PREVIOUS,
].filter(Boolean)

export function verify(rawBody, signatureHeader) {
  for (const secret of SECRETS) {
    try {
      return Stripe.webhooks.constructEvent(rawBody, signatureHeader, secret)
    } catch {
      // Try the next secret; only fail if none validate.
    }
  }
  throw new Error('No configured webhook secret validated this signature')
}

Deploy the dual-accept verifier first. Then roll the secret in the Stripe dashboard. Then remove STRIPE_WEBHOOK_SECRET_PREVIOUS in a follow-up deploy. Three deploys, zero dropped events.

Note the ordering trap: if you roll the secret before shipping the dual-accept code, every webhook between the roll and your deploy fails signature verification. Stripe will retry, so you may not lose data permanently, but you will page someone.

Current mechanics: Stripe API keys · Webhook signatures.

4. AWS access keys

The best rotation is the one you never have to do. Long-lived AKIA… keys are worth eliminating outright:

  • GitHub Actions → AWS: use OIDC to assume a role. No stored credential at all. (Configuring OIDC in AWS)
  • EC2 / ECS / Lambda: instance profiles and task roles, delivered through the metadata service.
  • Local development: IAM Identity Center (aws sso login) issues short-lived credentials.

When you genuinely need static keys, IAM allows two access keys per user, and that limit exists precisely to enable overlap:

# 1. Create the second key (returns AccessKeyId + SecretAccessKey once — capture it now)
aws iam create-access-key --user-name ci-deployer

# 2. Roll it out to every consumer, then verify it actually works
AWS_ACCESS_KEY_ID=AKIANEW... AWS_SECRET_ACCESS_KEY=... \
  aws sts get-caller-identity

# 3. Confirm the OLD key has stopped being used before you touch it
aws iam get-access-key-last-used --access-key-id AKIAOLD...

# 4. Disable — reversible, unlike deletion
aws iam update-access-key --user-name ci-deployer \
  --access-key-id AKIAOLD... --status Inactive

# 5. After a quiet period, delete
aws iam delete-access-key --user-name ci-deployer --access-key-id AKIAOLD...

Step 3 is the one people skip. get-access-key-last-used tells you the date, region, and service of the last call made with that key. If it shows activity from five minutes ago, something is still using it and you have not finished step 2.

The Inactive state between disable and delete is a deliberate checkpoint: if something breaks, flipping back to Active is one command. Deletion is not reversible.

Watch CloudTrail during the window for AccessDenied events referencing the old key, and set a calendar reminder for the deletion — an indefinitely-Inactive key is clutter that hides the next real problem.

AWS guidance: Rotating access keys.

5. GitHub tokens

# Verify what a token can actually do before you rely on it.
# The x-oauth-scopes response header lists granted scopes for classic PATs.
curl -sS -I -H "Authorization: Bearer $NEW_TOKEN" https://api.github.com/user \
  | rg -i 'x-oauth-scopes|x-ratelimit-limit'

# Update an Actions secret without opening a browser
gh secret set NPM_DEPLOY_TOKEN --body "$NEW_TOKEN" --repo myorg/myrepo

# Trigger a workflow that exercises the token, and watch it
gh workflow run deploy.yml --repo myorg/myrepo
gh run watch --repo myorg/myrepo

Order matters here too: set the secret, run a workflow that actually uses it, confirm green, then revoke the old token in GitHub settings. Revoking first means your next deploy is the thing that discovers the problem.

Prefer fine-grained PATs scoped to specific repositories over classic PATs with broad scopes. For anything long-lived and org-wide, a GitHub App with an installation token is better still: tokens are short-lived by construction, so rotation becomes automatic rather than a task on someone's list.

Docs: Managing personal access tokens.

6. Rolling the value through CI and orchestrators

Updating the secret store is not the same as updating the running process. Almost every runtime reads environment variables once at startup.

# Kubernetes: update the Secret, then force pods to pick it up.
kubectl create secret generic app-secrets \
  --from-literal=STRIPE_SECRET_KEY="$NEW_KEY" \
  --dry-run=client -o yaml | kubectl apply -f -

# Env-var-mounted Secrets do NOT hot-reload. Restart deliberately:
kubectl rollout restart deployment/api
kubectl rollout status deployment/api --timeout=120s

A Secret mounted as a volume is eventually updated in place by the kubelet, but the application still has to notice and re-read the file. Mounted as environment variables, it never updates without a restart. Know which one you're using before you declare the rotation complete.

The same applies elsewhere: ECS needs a new task definition revision and a service update; Lambda needs a configuration update; systemd services need systemctl restart after the EnvironmentFile changes; a long-lived cron job may hold a stale value until its next invocation.

7. Verify with traffic, not with optimism

"The deploy went green" is not verification. A key used only by a nightly job will look fine for twenty-three hours.

Before revoking, confirm:

  • Positive signal: a real request succeeded using the new credential. Vendor dashboards usually show last-used-at per key; that timestamp moving is the proof you want.
  • Negative signal: the old credential's last-used timestamp has stopped advancing.
  • Low-frequency consumers have run at least once. Trigger nightly jobs manually rather than waiting.
  • Error rates are flat — no new 401/403 in application logs.
# Cheap synthetic check you can run in a loop during the window
while true; do
  curl -sS -o /dev/null -w '%{http_code} ' \
    -H "Authorization: Bearer $NEW_KEY" https://api.vendor.com/v1/ping
  sleep 30
done

8. Purge the old value from developer machines

Rotation that leaves the dead secret scattered across laptops has only half worked. The old value is no longer dangerous once revoked, but the habit of stale local copies is what makes the next leak likely.

After revoking:

  • Delete the value from local .env files — the most commonly forgotten location.
  • Update your vault entry rather than adding a second one. If you keep history, label it explicitly as retired with a date, so nobody pastes it into a terminal at 2am.
  • Clear stale copies from password managers, notes apps, and shell history:
# Check whether a secret ever landed in shell history
rg -n 'sk_live_|AKIA' ~/.zsh_history ~/.bash_history 2>/dev/null

This is the part a local vault makes tractable. With PassStore, the canonical value lives in one encrypted entry per project; rotating means editing that entry, not hunting for every file that happens to contain the string. The vault health audit surfaces entries you haven't touched in a long time, which is a decent proxy for "overdue for rotation."

9. When rotation is incident response

If you are rotating because a key leaked, the sequence inverts — containment outranks uptime. Start here instead:

The short version: revoke and reissue before you clean Git history. Rewriting history is housekeeping. It does nothing about the copy an attacker already has, and time spent on git filter-repo while a live key sits in someone else's clone is time spent on the wrong problem.

How often should keys be rotated?

There is no universal interval, and calendar-driven rotation is weaker than most policies imply. Rotating a key every 90 days does nothing about a compromise on day 4.

Better triggers:

  • Someone with access leaves the team or project — rotate immediately.
  • A key appears anywhere it shouldn't — a log, a screenshot, a ticket, a Slack channel, a Git commit.
  • The key can't be scoped down and grants broad access — rotate more often precisely because the blast radius is large.
  • A vendor reports suspicious activity.
  • Your provider supports short-lived credentials — then adopt those and stop rotating manually.

Fixed intervals are still worth keeping as a backstop for credentials you cannot make short-lived, because they force you to prove the rotation path still works. A rotation procedure nobody has exercised in a year is not a procedure; it's a document.

Rotation readiness, per credential

QuestionIf the answer is "no"
Can two of these credentials be valid at once?Plan a maintenance window; you cannot do zero-downtime overlap
Does the vendor expose last-used-at?Add your own logging before rotating, or you're rotating blind
Is every consumer inventoried?Do not start; finish the inventory first
Can you disable before deleting?Treat revocation as irreversible and verify harder
Could this be a short-lived credential instead?Fix that instead of building a rotation habit