GitLab CI hanging on Docker image pulls, killing our deploy times

hey everyone, we've been running into this issue for the past week where our CI/CD pipelines are just... hanging. specifically when pulling the base Docker image at the start of each job.

we're on GitLab CI with our own runners (docker executor) hosted on a Hetzner dedicated server. image pulls that used to take 30 seconds are now timing out after 5 minutes. nothing changed on our end that i can see.

running docker pull manually from the runner machine works fine, completes in under a minute. but inside the CI job context it's incredibly slow. we tried:

• bumping the timeout from 10m to 30m (band-aid fix, not ideal)
• pulling from a different registry (same issue)
• docker system prune to clear cache
• checking network connectivity (all good)

has anyone seen this before? is this a known GitLab CI issue or am I missing something obvious?

Check if Docker's image pull is actually hitting a registry rate limit or getting stuck on DNS resolution. Run docker pull --verbose on the runner and pipe it to a log—you'll see exactly where it's hanging.

Also, Hetzner's network can sometimes have weird MTU issues. Try ping -M do -s 1472 registry.hub.docker.com from your runner to see if there's packet fragmentation happening. If that's the culprit, lowering Docker's MTU in /etc/docker/daemon.json can fix it fast.

One more thing: check your GitLab Runner's concurrent job limit—if you're pulling the same image across multiple jobs simultaneously, the daemon might be throttling. What does docker info show for pool size?

good point, I'll run that verbose pull and check the logs. never thought about DNS being the culprit—we did have some network tweaks done on the Hetzner side last week that I didn't connect to this. let me dig into that first.

Since manual pulls work fine, check if it's a GitLab Runner caching or credential issue. Try gitlab-runner verify on that machine and look for any auth errors. Also worth checking if the runner's Docker daemon has --max-concurrent-downloads set too low (defaults to 3). You can bump it in /etc/docker/daemon.json and restart the daemon. That network change at Hetzner might've also affected MTU or DNS—try docker pull --platform linux/amd64 to rule out manifest resolution issues.