November 23, 2023 19:38 UTC
[Resolved] The backlog of jobs has been completed. Security Scan Result Policies are working as expected. This incident is now considered resolved.
November 23, 2023 15:52 UTC
[Monitoring] Our team has merged a code change for the issue into production and we've re-enabled the affected worker. We are working through a backlog of jobs and will provide an update once it is completed.
November 22, 2023 23:25 UTC
[Monitoring] No material updates at this time. We'll provide additional updates once more information is available.
November 22, 2023 22:17 UTC
[Monitoring] While the issue has been mitigated, our team has disabled a worker that impacts the availability of the Security Scan Result Policies. This can impact the ability to enable/disable and update policies. Our team is working on a code change to fix the root issue.
November 22, 2023 21:18 UTC
November 22, 2023 21:05 UTC
[Monitoring] Our team has disabled the offending worker and performance is improving. We are continuing to monitor the state of the background processing.
November 22, 2023 20:49 UTC
[Identified] Our team has identified the root cause as related to a specific worker. We are continuing to work on mitigation efforts.
November 22, 2023 20:19 UTC
November 22, 2023 19:45 UTC
[Investigating] Our team is continuing to investigate the root cause. We believe to have identified a query that is driving high CPU usage and saturating sidekiq.
November 22, 2023 19:20 UTC
November 22, 2023 19:06 UTC
[Investigating] Our team is currently investigating some sidekiq performance issues which is causing some delays in jobs processing.