GitLab CI suddenly slow after March updates, anyone else seeing this?

so we upgraded our gitlab runners to the latest stable last week and now builds that normally take 2-3 mins are hitting 8-10 mins consistently. docker layer caching seems fine, no resource contention on the runner nodes themselves.

rolled back one runner to the previous version and it's back to normal speeds. pretty sure it's something in the 24.3 release causing this.

anyone else on the latest gitlab experiencing slowdowns? wondering if it's worth filing an issue or if there's a known workaround. we're running about 150 builds/day so this is actually costing us real time.

Check your runner's log level and see if there's extra debug logging enabled after the upgrade—I've seen that tank performance hard. Also worth checking the CI_DEBUG_TRACE var and any new prometheus metrics collection they might've added. If it's reproducible on rollback, def file the issue with GitLab, include the timeline and version numbers. They're usually pretty responsive to perf regressions.

good catch! yeah I checked the log levels and they looked normal, but I'll dig into the prometheus stuff—hadn't thought about that. Already filed an issue on gitlab, but gonna try disabling some of the new telemetry features on that one rolled-back runner to see if that's the culprit.

Did you check if the runner is using a different executor after the upgrade? I had something similar where updating bumped us from docker to docker+machine and the overhead killed performance. Also worth checking your config.toml for any new defaults around pull_policy or image pulling—24.3 might've changed those. Definitely worth reporting if it's reproducible across multiple runners.

Did you compare the actual runner binary versions? I had this happen where a patch upgrade silently swapped the base image and docker wasn't pre-cached anymore. Run gitlab-runner --version on both versions and check the release notes for any build environment changes. Also worth checking if BuildKit got enabled by default—that can add overhead initially if your build cache is cold.

Have you checked if the runner's docker daemon itself got updated as a side effect? I had this exact issue—gitlab-runner upgrade pulled in a newer docker version that changed how it handles layer caching. Try docker info on both runners and compare the storage driver and version. Also worth checking if the new runner is using buildkit by default now (DOCKER_BUILDKIT env var). Could easily add 3-5x overhead if that flipped without you knowing.