Secrets rotation on EKS without triggering rolling restarts?

We're running about 15 microservices on a managed K8s cluster (EKS) and I'm trying to figure out the best way to handle secrets rotation without causing pod restarts across the board.

Currently we're using Kubernetes secrets synced from AWS Secrets Manager via External Secrets Operator. The problem is every time a secret rotates, ESO updates the K8s secret, which triggers a rolling restart if you have reloader watching it. For most services that's fine, but we have a few stateful workloads where a restart during peak hours causes issues.

Has anyone set up a more granular approach? I've been looking at HashiCorp Vault with the agent injector so the app can pick up new secrets without a restart, but the operational overhead of running Vault in production gives me pause.

Open to suggestions. We're on EKS 1.29, Flux for GitOps, and everything is Helm-based.

Have you considered using a sidecar pattern with a secrets polling mechanism instead of relying on reloader? We do this with a custom Go binary that checks AWS Secrets Manager every 30s and hot-reloads creds into memory without bouncing the pod.

Alternatively, if you're using External Secrets, you could disable reloader on those specific deployments and instead implement graceful secret refresh logic in your app code itself. Most frameworks support reloading from env vars or config files without a restart. For stateful workloads like databases, you might also want to look at Vault's dynamic secrets—they handle rotation at the connection level without your app even knowing about it.

Fwiw, the rolling restart approach doesn't scale well past a few services. What language/framework are your stateful workloads using? That might influence the best solution here.

That's a good point, I hadn't considered the sidecar approach for hot-reloading. Do you handle the 30s polling overhead well on your stateful services, or does it depend on the workload type? Might be worth testing that out before ripping out reloader entirely.

Instead of polling, have you looked at using AWS Secrets Manager native event notifications? You can trigger Lambda → SNS → your app webhook when a secret rotates, then your app reloads creds without pod restart. Beats polling overhead and gives you sub-second rotation. Docs are sparse but works well for DB creds especially.