GitLab CI/CD suddenly timing out on Docker builds—anyone else seeing this?

Hey folks, started noticing our CI/CD pipeline is timing out on Docker image builds as of this morning. Same builds that were finishing in ~3 mins are now hitting 15+ mins and getting killed.

We're on GitLab 16.8, using a shared runner pool on a Hetzner dedicated server. Nothing changed on our end—no new dependencies, no bigger images. Just suddenly slow.

Checked disk space (40% free), CPU looks normal, Docker daemon is responsive. Ran the builds manually on the runner and they're fast. Only slow through GitLab.

Anyone run into this recently? Thinking it might be a GitLab caching issue or maybe runner config got borked somehow?

Check if Docker's build cache got corrupted. Try docker builder prune -a on the runner and clear GitLab's cache with CI_DEBUG_TRACE=true on next run. Also worth checking if Hetzner did maintenance or if your runner's disk I/O degraded—run iostat -x 1 5 while a build's running to see if it's bottlenecked there. That 3→15min jump screams I/O or cache issue to me.

Good call on the builder prune—just ran that and cleared about 8GB of dangling images. Kicked off a fresh build and it's already back to normal speed. Cheers! Definitely didn't think to check the build cache since the daemon seemed fine.

Glad you found it! Btw, worth setting up a scheduled job to prune that cache weekly—Docker's layer caching can be a silent killer on shared runners. You could add something like docker builder prune -a -f --keep-state to a cron job, or let BuildKit handle it with --build-context flags. Since you're on Hetzner, you've got the disk space, just need the discipline. https://docs.docker.com/build/buildkit/

Ah nice catch! One thing that bit us similarly—if you're using BuildKit (which is the default in newer Docker versions), check if DOCKER_BUILDKIT=1 is set inconsistently across your runner environment. We had one runner with it enabled and others without, causing wildly different cache behavior. Worth checking your runner config to make sure the BuildKit setting is consistent. Also might want to add that prune to a cron job so it doesn't sneak up again.

Update: set up the weekly prune job like suggested and builds are consistently under 3mins now. The 8GB cache bloat was killing performance. Thanks for the quick diagnosis!