Sidekiq job delays resulting in updates to MR approval policies being ineffectiveOperational

Components

Website, API

Locations

Google Compute Engine



October 4, 2024 06:59 UTC
[Resolved] Our monitoring shows no further issues after the fix was applied. We are resolving this issue and any further updates will be in: gitlab.com/gitlab-com/gl-infra/production/-/issues/18660

October 3, 2024 22:23 UTC
[Monitoring] The fix has been successfully merged, but we are still waiting for the changes to reach production. We will continue to monitor Sidekiq’s performance, and the next update will be provided once the incident has been mitigated. For more details, visit gitlab.com/gitlab-com/gl-infra/production/-/issues/18660

October 3, 2024 21:04 UTC
[Monitoring] The fix has been successfully merged. We will continue to monitor Sidekiq’s performance to ensure everything is functioning as expected. For details, visit gitlab.com/gitlab-com/gl-infra/production/-/issues/18660

October 3, 2024 20:00 UTC
[Monitoring] The fix is expected to be merged soon. In the meantime, we’ll continue monitoring Sidekiq’s performance. Updates to MR approval policies will remain delayed until the issue is fully resolved. For more details, visit gitlab.com/gitlab-com/gl-infra/production/-/issues/18660

October 3, 2024 19:02 UTC
[Monitoring] We are still awaiting the deployment of the fix and will continue to monitor Sidekiq’s performance in the meantime. Updates to MR approval policies are delayed until the issue is resolved. For more details, visit gitlab.com/gitlab-com/gl-infra/production/-/issues/18660

October 3, 2024 18:02 UTC
[Monitoring] We continue to closely monitor Sidekiq’s performance while awaiting the deployment of the fix. In the meantime, updates to MR approval policies remain delayed. More details about this incident can be found at gitlab.com/gitlab-com/gl-infra/production/-/issues/18660

October 3, 2024 17:06 UTC
[Monitoring] We are actively monitoring the performance of Sidekiq until the fix is deployed. Updates to merge request approval policies are still experiencing delays. Details about this incident can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/18660

October 3, 2024 15:55 UTC
[Monitoring] We're continuing to monitor the performance of Sidekiq until the fix is deployed. Updates to MR approval policies are still processed with delays. The MR to fix the root cause is currently waiting to be merged. More details about this incident can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/18660

October 3, 2024 14:46 UTC
[Monitoring] We're continuing to see improvements and are monitoring until a fix is deployed. Updates to MR approval policies are still processed with delays. More details about this incident can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/18660

October 3, 2024 14:28 UTC
[Monitoring] We've identified the root cause and are recovering from a high volume of request traffic and are seeing improvements. Updates to MR approval policies are currently processing but with delays. More details about this incident can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/18660

October 3, 2024 14:12 UTC
[Investigating] We are currently investigating issues with updating MR Approval policies as a result of Sidekiq performance degradation. More details about this incident can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/18660.

Back to current status