Key Rotation Procedures

Organization: ZKProva System: ZKP-Powered Portable Credit Union Identity SOC 2 Criteria: CC6.6 (Management of Credentials) Document Version: 1.0 Effective Date: 2026-02-28 Classification: Confidential Review Cadence: Quarterly (next review: 2026-05-31)

---

Table of Contents

1. Key Inventory
2. Rotation Procedures
3. Application Cache Caveat
4. Audit Trail
5. Emergency Rotation
6. Document Control

---

Key Inventory

Key / Secret	Algorithm	Storage Location	Rotation Schedule	Last Rotated	Next Rotation
JWT Signing Key	HS256	AWS Secrets Manager (zkprova-secrets)	90 days	Initial deployment	Q2 2026
AES-256-GCM Encryption Key	AES-256-GCM	AWS Secrets Manager (zkprova-secrets)	180 days	Initial deployment	Q2 2026
Ed25519 Issuer Signing Key	Ed25519 (DID:key)	AWS Secrets Manager (zkprova-secrets)	Annual	Initial deployment	Q1 2027
TLS Certificates	RSA 2048 / ECDHE 256	AWS Certificate Manager (ACM)	Automatic (ACM managed)	Auto-renewed	Auto-renewed
GHCR PAT (CI/CD)	N/A	GitHub Actions Secrets	90 days	Per CI setup	Q2 2026
Redis Auth Token	N/A	AWS Secrets Manager / Terraform variable	180 days	Initial deployment	Q2 2026
Database Password	N/A	AWS Secrets Manager / Terraform variable (TF_VAR_db_password)	180 days	Initial deployment	Q2 2026
Webhook HMAC Secrets	HMAC-SHA256	PostgreSQL (per-lender)	Per lender request	N/A	On request
SES SMTP Credentials	N/A	IAM (zkprova-ses-smtp user)	180 days	Initial deployment	Q2 2026

Key Validation at Startup

The application enforces key strength at startup via Pydantic validators in backend/app/config.py:

• jwt_secret_key: Minimum 32 characters. Known development values rejected when debug=False.
• encryption_key: Exactly 64 hex characters (32 bytes). Known development values rejected when debug=False.
• issuer_private_key_hex: Exactly 64 hex characters (32 bytes Ed25519 seed).

---

Rotation Procedures

JWT Signing Key (90-day rotation)

Risk: A compromised JWT key allows an attacker to forge authentication tokens for any user role (member, lender, admin).

Procedure — Zero-Downtime Dual-Accept Window:

1. Generate new key:

   python3 -c "import secrets; print(secrets.token_urlsafe(48))"
   

2. Update Secrets Manager with both keys:

   # Store new key as primary
   aws secretsmanager update-secret \
     --secret-id zkprova-secrets \
     --secret-string '{"jwt_secret_key": "<new-key>", "jwt_secret_key_previous": "<old-key>", ...}'
   

3. Deploy with dual-accept logic:

4. The application first validates incoming JWTs against the primary key
5. If validation fails, it retries against jwt_secret_key_previous

6. New tokens are always signed with the primary key

7. Rolling restart to pick up new secrets:

   kubectl rollout restart deployment/zkprova-backend -n default
   kubectl rollout status deployment/zkprova-backend -n default
   

8. Verify:

9. Existing sessions continue to work (validated against previous key)
10. New logins receive tokens signed with the new key

11. Run smoke tests: ./scripts/smoke-test.sh

12. Remove old key after the dual-accept window expires (access token lifetime = 30 minutes, refresh token = 7 days; wait at least 7 days):

    aws secretsmanager update-secret \
      --secret-id zkprova-secrets \
      --secret-string '{"jwt_secret_key": "<new-key>", ...}'
    

13. Final restart:

    kubectl rollout restart deployment/zkprova-backend -n default
    

14. Log rotation event in audit service and update this document's "Last Rotated" column.

Rollback: If issues arise, swap the primary and previous keys in Secrets Manager and restart.

AES-256-GCM Encryption Key (180-day rotation)

Risk: A compromised encryption key allows decryption of all credential data stored in PostgreSQL.

Procedure — Re-Encryption Migration:

1. Generate new key:

   python3 -c "import secrets; print(secrets.token_hex(32))"
   

2. Update Secrets Manager with both keys:

   aws secretsmanager update-secret \
     --secret-id zkprova-secrets \
     --secret-string '{"encryption_key": "<new-key-hex>", "encryption_key_previous": "<old-key-hex>", ...}'
   

3. Deploy with dual-key decryption:

4. Decrypt attempts use the primary key first
5. If decryption fails (wrong key), retry with encryption_key_previous

6. All new encryptions use the primary key

7. Rolling restart:

   kubectl rollout restart deployment/zkprova-backend -n default
   

8. Run re-encryption migration:

   # One-time script to re-encrypt all credentials with the new key
   # This reads each credential, decrypts with old key, re-encrypts with new key
   # Run during low-traffic window
   kubectl exec -it deployment/zkprova-backend -- python -m scripts.reencrypt_credentials
   

9. Verify: Spot-check credential decryption with new key only. Confirm no credentials remain encrypted with the old key.

10. Remove old key from Secrets Manager after re-encryption is verified complete.

11. Final restart and audit log entry.

Rollback: Keep the old key in Secrets Manager until re-encryption is verified. Both keys remain valid during the migration window.

Ed25519 Issuer Signing Key (Annual rotation)

Risk: A compromised Ed25519 key allows forging W3C Verifiable Credentials and DID:key identifiers. This is the most sensitive key in the system.

Procedure — Coordinated DID:key Update:

1. Generate new Ed25519 keypair:

   python3 -c "
   from cryptography.hazmat.primitives.asymmetric.ed25519 import Ed25519PrivateKey
   key = Ed25519PrivateKey.generate()
   print('Private (hex):', key.private_bytes_raw().hex())
   print('Public (hex):', key.public_bytes_raw().hex())
   "
   

2. Derive new DID:key identifier from the new public key (per W3C DID:key specification).

3. Notify all partner credit unions and lenders:

4. Provide the new DID:key identifier
5. Announce the transition timeline (minimum 30-day notice)

6. Both old and new DIDs will be accepted during transition

7. Update Secrets Manager:

   aws secretsmanager update-secret \
     --secret-id zkprova-secrets \
     --secret-string '{"issuer_private_key_hex": "<new-key-hex>", "issuer_private_key_hex_previous": "<old-key-hex>", ...}'
   

8. Deploy with dual-key verification:

9. New credentials signed with the new key and new DID:key
10. Verification accepts proofs from credentials signed by either key

11. Old credentials remain valid until their natural expiry

12. Rolling restart:

    kubectl rollout restart deployment/zkprova-backend -n default
    

13. Transition period (90 days):

14. Monitor verification success rates for old vs. new credentials

15. Encourage members to request re-issuance of credentials with the new key

16. Decommission old key after transition period. Remove from Secrets Manager. Update DID document to revoke old DID:key.

17. Audit log entry and document update.

Rollback: During transition period, revert to old key as primary if issues with new key are discovered.

Future improvement: HSM integration for Ed25519 keys (Gap #6, target Q3 2026) will provide hardware-backed key protection and simplified rotation.

TLS Certificates (Automatic via ACM)

AWS Certificate Manager handles certificate renewal automatically for *.zkprova.com. DNS validation via Route53 ensures seamless renewal.

Verification: Periodically check certificate expiry:

aws acm describe-certificate --certificate-arn <arn> \
  --query 'Certificate.NotAfter'

No manual rotation required.

GHCR PAT (90-day rotation)

1. Generate new Personal Access Token in GitHub Settings with read:packages and write:packages scopes
2. Update GitHub Actions repository secret GHCR_PAT
3. Verify CI pipeline can push/pull images: trigger a test workflow run
4. Revoke old token in GitHub Settings

Redis Auth Token (180-day rotation)

1. Generate new token:

   python3 -c "import secrets; print(secrets.token_urlsafe(32))"
   

2. Update ElastiCache auth token:

   aws elasticache modify-replication-group \
     --replication-group-id zkprova-redis \
     --auth-token <new-token> \
     --auth-token-update-strategy ROTATE
   

   This enables a dual-accept window where both old and new tokens are accepted.

3. Update application secret:

   aws secretsmanager update-secret \
     --secret-id zkprova-secrets \
     --secret-string '{"redis_url": "rediss://:<new-token>@<endpoint>:6379/0", ...}'
   

4. Rolling restart:

   kubectl rollout restart deployment/zkprova-backend -n default
   

5. Finalize rotation:

   aws elasticache modify-replication-group \
     --replication-group-id zkprova-redis \
     --auth-token <new-token> \
     --auth-token-update-strategy SET
   

Database Password (180-day rotation)

1. Generate new password:

   python3 -c "import secrets; print(secrets.token_urlsafe(32))"
   

2. Update RDS master password:

   aws rds modify-db-instance \
     --db-instance-identifier zkprova-db \
     --master-user-password <new-password> \
     --apply-immediately
   

3. Update application secret with new DATABASE_URL in Secrets Manager.

4. Rolling restart:

   kubectl rollout restart deployment/zkprova-backend -n default
   

5. Verify: /health endpoint confirms DB connectivity.

---

Application Cache Caveat

The application uses @lru_cache decorators in two critical locations:

1. backend/app/services/secrets.py — get_secrets_backend() caches the secrets backend singleton
2. backend/app/config.py — get_settings() caches the Pydantic Settings object (which reads secrets at initialization)

Implication: After rotating any secret in AWS Secrets Manager or environment variables, running pods will continue using the cached (old) values. A kubectl rollout restart is required to force pods to re-read secrets.

Procedure for every rotation:

# After updating the secret in Secrets Manager:
kubectl rollout restart deployment/zkprova-backend -n default
kubectl rollout status deployment/zkprova-backend -n default

# PDB (minAvailable=2) ensures at least 2 pods remain serving during restart
# Rolling restart replaces pods one at a time — no downtime

Monitoring: After restart, verify: - All pods reach Running state: kubectl get pods - Health check passes: curl https://api.zkprova.com/health - Smoke tests pass: ./scripts/smoke-test.sh - No error spike in Sentry

---

Audit Trail

Every key rotation event is recorded through two channels:

1. Application Audit Log (PostgreSQL)

The audit service (backend/app/services/audit.py) records rotation events with:

Field	Value
event_type	key.rotated
actor_id	Engineer performing the rotation
actor_type	admin
resource_type	secret
resource_id	Key identifier (e.g., jwt_secret_key, encryption_key)
details	{"reason": "<scheduled/emergency>", "rotation_date": "<ISO 8601>"}
ip_address	Source IP of the engineer

Audit logs are append-only with 7-year retention.

2. Structured Logs (structlog)

The audit service dual-writes to structured JSON logs regardless of DB write success. These logs are forwarded to CloudWatch Logs for centralized analysis.

3. AWS CloudTrail

All AWS Secrets Manager API calls (UpdateSecret, GetSecretValue) are automatically logged in CloudTrail with IAM principal, timestamp, and source IP.

---

Emergency Rotation

In the event of a confirmed or suspected key compromise (P0 incident):

1. Do not wait for the standard rotation window. Initiate immediate rotation.
2. Follow the standard procedure for the compromised key type, but skip the dual-accept window transition period.
3. For JWT key compromise: Immediate rotation invalidates all active sessions. All users must re-authenticate. Accept the service disruption.
4. For AES key compromise: Immediate rotation + emergency re-encryption. If the attacker has exfiltrated encrypted data, the old key must be considered compromised — re-encrypted data with the new key is safe.
5. For Ed25519 key compromise: Immediately revoke the old DID:key. Issue new credentials to all members. Notify all lenders to reject proofs from the compromised DID. This is the highest-severity key rotation scenario.
6. Document the emergency rotation as part of the incident post-mortem (see Incident Response Plan).

---

Document Control

Version	Date	Author	Description
1.0	2026-02-28	ZKProva Engineering	Initial key rotation procedures

---

This document satisfies SOC 2 Trust Service Criteria CC6.6 (Management of Credentials). It is reviewed quarterly and updated after each key rotation event.