Sidekiq processing is delayedOperational

Components

Background Processing

Locations

Google Compute Engine



November 23, 2023 19:38 UTC
[Resolved] The backlog of jobs has been completed. Security Scan Result Policies are working as expected. This incident is now considered resolved.

November 23, 2023 15:52 UTC
[Monitoring] Our team has merged a code change for the issue into production and we've re-enabled the affected worker. We are working through a backlog of jobs and will provide an update once it is completed.

November 22, 2023 23:25 UTC
[Monitoring] No material updates at this time. We'll provide additional updates once more information is available.

November 22, 2023 22:17 UTC
[Monitoring] While the issue has been mitigated, our team has disabled a worker that impacts the availability of the Security Scan Result Policies. This can impact the ability to enable/disable and update policies. Our team is working on a code change to fix the root issue.

November 22, 2023 21:18 UTC
[Resolved] The mitigation has been successful and the issue is considered resolved. Our team will continue to work on additional action items. More details in gitlab.com/gitlab-com/gl-infra/production/-/issues/17168

November 22, 2023 21:05 UTC
[Monitoring] Our team has disabled the offending worker and performance is improving. We are continuing to monitor the state of the background processing.

November 22, 2023 20:49 UTC
[Identified] Our team has identified the root cause as related to a specific worker. We are continuing to work on mitigation efforts.

November 22, 2023 20:19 UTC
[Investigating] Our team is still continuing the investigation. Additional details can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/17168

November 22, 2023 19:45 UTC
[Investigating] Our team is continuing to investigate the root cause. We believe to have identified a query that is driving high CPU usage and saturating sidekiq.

November 22, 2023 19:20 UTC
[Investigating] Our team is still investigating the delays with sidekiq processing. This may affect jobs getting picked up, or the ability to create or update merge requests. More details in gitlab.com/gitlab-com/gl-infra/production/-/issues/17168

November 22, 2023 19:06 UTC
[Investigating] Our team is currently investigating some sidekiq performance issues which is causing some delays in jobs processing.

Back to current status