Sidekiq degredationOperational

Components

Website, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions, Background Processing

Locations

Google Compute Engine



January 30, 2024 23:02 UTC
[Resolved] The feature flag responsible for the issue has been disabled by default and operations are back to normal. This incident is considered resolved. Further details can be found in the incident issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/17504

January 30, 2024 22:02 UTC
[Identified] We've identified the issue and have narrowed the problem to a recent feature flag that created slow queries caused by specific import workers. These slow queries lead to database saturation. To mitigate the issue, the team has disabled the feature flag. We are now re-processing jobs from these workers. This issue is considered mitigated. Further updates will be minimal until there is a permanent code fix in place. More details in gitlab.com/gitlab-com/gl-infra/production/-/issues/17504

January 30, 2024 20:22 UTC
[Investigating] No material update at this time. Our team is still investigating the root cause.

January 30, 2024 19:54 UTC
[Investigating] We've temporarily disabled some features of the GitHub import functionality. Users might experience issues with GitHub imports at this time while our team continues to investigate.

January 30, 2024 19:28 UTC
[Investigating] Sidekiq has largely recovered and jobs are processing. Our team is still investigating the root cause.

January 30, 2024 19:08 UTC
[Investigating] Our team is still investigating the root cause of this issue. We'll continue to provide updates as more information is available.

January 30, 2024 18:51 UTC
[Investigating] We're currently seeing some sidekiq degradation that appears to be impacting some pipelines and merge requests. Our team is investigating. More information in gitlab.com/gitlab-com/gl-infra/production/-/issues/17504

Back to current status