Red team exercise exposed weak SSH key rotation — lessons learned
Just wrapped a pentest on our production infrastructure and found something concerning: SSH keys weren't being rotated regularly enough. Our red teamers compromised a junior dev's key from 18 months ago that was still active across 12 servers.
We've now implemented:
- Mandatory key rotation every 90 days
- Automated key revocation on employee offboarding
- Central key management with HashiCorp Vault
- Audit logging of all key operations
Has anyone else dealt with this at scale? Using Vault in a multi-region setup (AWS + Hetzner) and wondering if there are gotchas I should know about before full rollout. Also curious about key derivation strategies — ECDSA or ED25519 preferred in your environments?
Edited at 26 Mar 2026, 08:55
Good catch. One thing we learned the hard way: Vault's great but don't sleep on the SSH CA model instead. Way simpler to scale than managing individual keys—issue short-lived certs on-demand, no rotation headaches. We cut our key management overhead in half. For offboarding especially, you just let certs expire naturally instead of manually revoking everywhere.
Ah, good point about the SSH CA model—we actually looked at that but our devs were pretty attached to key-based auth already. That said, we're definitely considering cert-based for new services going forward. Did you find it easier to get team buy-in once you switched?
18 months is brutal. One thing we didn't see mentioned—did you audit where those old keys were being used? We found keys duplicated in deploy scripts, CI/CD configs, and personal laptops long after they should've been purged. The rotation policy is solid, but without inventory enforcement you're playing whack-a-mole. Consider adding a pre-rotation scan to catch these orphaned copies before they become a liability again.
The offboarding piece is huge—we missed that too. Pro tip: don't just revoke keys on exit day. Grep your entire codebase (including git history!) for hardcoded keys immediately after. We found 6 prod keys buried in old deployment scripts that nobody remembered existed. Also consider agent forwarding restrictions if you haven't already—limits blast radius if a bastion does get compromised.