GitLab pipelines experiencing longer than expected execution timesOperational

Components

Background Processing, GitLab Duo

Locations

Google Compute Engine



September 4, 2024 05:57 UTC
[Resolved] Pipeline functionality has been restored and is working as expected based on our monitoring. We are marking this incident as resolved. For more updates, please follow: gitlab.com/gitlab-com/gl-infra/production/-/issues/18489 Additionally, we have reset the compute monthly minutes due to this incident. For details on this action, see: gitlab.com/gitlab-com/gl-infra/production/-/issues/18497

September 4, 2024 00:13 UTC
[Monitoring] Pipeline functionality has been restored. We have opted to reset the compute monthly minutes due to this incident. That work has started and you can view it here: gitlab.com/gitlab-com/gl-infra/production/-/issues/18497. See incident issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/18489

September 3, 2024 23:12 UTC
[Monitoring] Pipeline functionality has been restored. We have opted to reset the compute monthly min due to this incident. Please view the change request here. gitlab.com/gitlab-com/gl-infra/production/-/issues/18497. See incident issue : gitlab.com/gitlab-com/gl-infra/production/-/issues/18489

September 3, 2024 22:02 UTC
[Monitoring] Pipeline functionality has been restored. We have opted to reset the compute monthly min due to this incident. Please view the change request here. gitlab.com/gitlab-com/gl-infra/production/-/issues/18497.

September 3, 2024 19:46 UTC
[Monitoring] We have mitigation in place and pipelines are resolving normally. Please See the issue for more information: gitlab.com/gitlab-com/gl-infra/production/-/issues/18489

September 3, 2024 18:20 UTC
[Monitoring] We have mitigation in place and pipelines are resolving normally. Please See the issue for more information: gitlab.com/gitlab-com/gl-infra/production/-/issues/18489

September 3, 2024 18:04 UTC
[Monitoring] We are continuing to monitor and we recommend cancelling jobs that may have become stuck during the incident and we should see them process. See the issue for more information: gitlab.com/gitlab-com/gl-infra/production/-/issues/18489

September 3, 2024 16:57 UTC
[Monitoring] We are continuing to monitor and we recommend cancelling jobs that may have become stuck during the incident and we should see them process. See the issue for more information: gitlab.com/gitlab-com/gl-infra/production/-/issues/18489

September 3, 2024 15:41 UTC
[Monitoring] No material updates at this time. We are continuing to monitor and we recommend cancelling jobs that may have become stuck during the incident. See the issue for more information: gitlab.com/gitlab-com/gl-infra/production/-/issues/18489

September 3, 2024 13:59 UTC
[Monitoring] We are continuing to monitor. We recommend cancelling jobs that may have become stuck during the incident. See the issue for more information: gitlab.com/gitlab-com/gl-infra/production/-/issues/18489

September 3, 2024 12:26 UTC
[Monitoring] Background job processing has been restored. We are continuing to monitor while we investigate further into the root cause. See the issue for more information: gitlab.com/gitlab-com/gl-infra/production/-/issues/18489

September 3, 2024 11:40 UTC
[Investigating] No material updates at this time. We are continuing to investigate and monitor background job processing. See the issue for more information: gitlab.com/gitlab-com/gl-infra/production/-/issues/18489

September 3, 2024 10:59 UTC
[Investigating] We are seeing improvements in background job processing and operations are recovering back to normal. Delays in email processing are still possible, and we are continuing with our investigation. See the issue for more information: gitlab.com/gitlab-com/gl-infra/production/-/issues/18489

September 3, 2024 10:04 UTC
[Investigating] No current updates at this time. We are still working to isolate the problem but took measures to add relief. We will provide an update once the root cause has been identified. See the issue for more information: gitlab.com/gitlab-com/gl-infra/production/-/issues/18489.

September 3, 2024 09:38 UTC
[Investigating] We are still working to isolate the problem to a specific background job. See the issue for more information: gitlab.com/gitlab-com/gl-infra/production/-/issues/18489.

September 3, 2024 09:15 UTC
[Investigating] We are trying to isolate the problem to a number of background jobs contributing to this problem and we are taking them offline for some time. AI feature such as Duo Chat may also be impacted by this. See the issue for more information: gitlab.com/gitlab-com/gl-infra/production/-/issues/18489.

September 3, 2024 08:52 UTC
[Investigating] No material updates at this time - our investigation into Sidekiq issues is ongoing. See the issue for more information: gitlab.com/gitlab-com/gl-infra/production/-/issues/18489.

September 3, 2024 08:37 UTC
[Investigating] We are investigating issues with Sidekiq. See the issue for more information: gitlab.com/gitlab-com/gl-infra/production/-/issues/18489.

September 3, 2024 08:13 UTC
[Identified] We hare still investigating the issue. The background processes may take longer to be picked up and executed than usual. This can result in emails not being delivered and commits not showing up in merge request activity. CI/CD jobs may be stuck in pending state. Issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/18489

September 3, 2024 07:51 UTC
[Investigating] We are still investigating the cause of extended GitLab pipeline execution times. Our team continues to work on resolving this issue. We appreciate your patience and will update when we have more information. Issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/18489

September 3, 2024 07:18 UTC
[Investigating] We are currently investigating reports of GitLab pipelines taking longer than usual to complete. Our team is aware of the issue and actively working to identify the cause of these delays. Issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/18489

Back to current status