Pipelines not completingOperational

Components

Background Processing

Locations

Google Compute Engine



October 9, 2024 17:33 UTC
[Resolved] The MR reverting the offending commit that caused this issue has been deployed to production. This incident is now resolved. See gitlab.com/gitlab-com/gl-infra/production/-/issues/18676 for all the details.

October 9, 2024 10:16 UTC
[Monitoring] We have enabled a feature flag that will prevent new pipelines and Merge Requests from being impacted. Pipelines and MR that were previously stuck will need to be recreated. See gitlab.com/gitlab-com/gl-infra/production/-/issues/18676

October 9, 2024 09:28 UTC
[Identified] We have identified a potential commit that could have caused this incident. We are working on reverting it. See gitlab.com/gitlab-com/gl-infra/production/-/issues/18676 for details.

October 9, 2024 09:23 UTC
[Investigating] Some Merge requests are still impacted by this incident. Merge requests are stuck with the message "Your merge request is almost ready!". The workaround is to recreate the impacted Merge Request. See gitlab.com/gitlab-com/gl-infra/production/-/issues/18676 for details.

October 9, 2024 08:26 UTC
[Investigating] We are moving this incident back to Investigating, as after resolving the Redis saturation, we are still seeing Pipelines not progressing between stages. Merge Requests are functioning normally now. See gitlab.com/gitlab-com/gl-infra/production/-/issues/18676 for details.

October 9, 2024 04:25 UTC
[Monitoring] We have resolved the Redis saturation issue and see that all pipelines are functioning normally now. We will continue to monitor while continuing investigation into other aspects. See gitlab.com/gitlab-com/gl-infra/production/-/issues/18676 for details.

October 9, 2024 04:18 UTC
[Resolved] We have resolved the Redis saturation issue and see that all pipelines are functioning normally now. We will continue to monitor while continuing investigation into other aspects. See gitlab.com/gitlab-com/gl-infra/production/-/issues/18676 for details.

October 9, 2024 03:33 UTC
[Monitoring] We have resolved the Redis saturation issue and see that all pipelines are functioning normally now. We will continue to monitor while continuing investigation into other aspects. See gitlab.com/gitlab-com/gl-infra/production/-/issues/18676 for details.

October 9, 2024 02:33 UTC
[Investigating] No material updates to report. We continue investigations and will provide further updates in 1 hour. Please follow gitlab.com/gitlab-com/gl-infra/production/-/issues/18676 for the latest.

October 9, 2024 01:26 UTC
[Investigating] No material updates to report. We continue to investigate potential causes of this issue. Please follow gitlab.com/gitlab-com/gl-infra/production/-/issues/18676 for the latest updates.

October 9, 2024 01:02 UTC
[Investigating] Investigation continues. Remember, restarting the pipeline on individual jobs may serve as a workaround. See gitlab.com/gitlab-com/gl-infra/production/-/issues/18676.

October 9, 2024 00:02 UTC
[Investigating] Reminder: If you have urgent pipelines affected by the ongoing issue, you can cancel and retry them. A retry may make the pipeline run properly. We're still working on a permanent fix. See gitlab.com/gitlab-com/gl-infra/production/-/issues/18676.

October 8, 2024 23:57 UTC
[Investigating] Despite resolving the Redis saturation issue, we're still receiving reports of hanging pipelines on GitLab.com. We understand the frustration this causes and are urgently investigating. Updates at gitlab.com/gitlab-com/gl-infra/production/-/issues/18676.

October 8, 2024 20:24 UTC
[Monitoring] After alleviating the saturation reported in our Redis infrastructure we are seeing readings go back to healthy levels. We will monitor now for new occurrences and reports to confirm this is resolved. Please review gitlab.com/gitlab-com/gl-infra/production/-/issues/18676 for details.

October 8, 2024 19:23 UTC
[Investigating] We're taking corrective actions to alleviate some unexpected pressure in one of our Redis shards, which may alleviate the problem. See gitlab.com/gitlab-com/gl-infra/production/-/issues/18676 for more details.

October 8, 2024 18:29 UTC
[Investigating] We are currently investigating errors on a specific Redis shard that may be contributing to the problem. For the latest updates and detailed information, please check gitlab.com/gitlab-com/gl-infra/production/-/issues/18676.

October 8, 2024 17:55 UTC
[Investigating] Investigation continues. Remember restarting the pipeline or individual jobs may serve as a workaround. See gitlab.com/gitlab-com/gl-infra/production/-/issues/18676.

October 8, 2024 17:25 UTC
[Investigating] Work is ongoing to identify the root of this issue, including an analysis of our Redis infrastructure. Please continue to follow gitlab.com/gitlab-com/gl-infra/production/-/issues/18676.

October 8, 2024 16:48 UTC
[Investigating] Work is still in progress to identify the cause of service degradation to pipelines. Update frequency will be increased to every 30 minutes or as soon as we have material updates to share. Please follow gitlab.com/gitlab-com/gl-infra/production/-/issues/18676 for more information.

October 8, 2024 16:23 UTC
[Investigating] We continue to investigate potential causes of this issue. Please follow gitlab.com/gitlab-com/gl-infra/production/-/issues/18676 for the latest updates.

October 8, 2024 15:39 UTC
[Investigating] We are currently investigating an issue where pipelines are not progressing between stages. This causes them to remain in a running state indefinitely, and merge requests are not being marked as ready. gitlab.com/gitlab-com/gl-infra/production/-/issues/18676

Back to current status