All Systems Operational

Updated a few seconds ago

Back to current status

Status History

Filter: CI/CD - Hosted runners on macOS (Clear)



April 2024

Runners are not picking up jobs causing 500 errors

April 11, 2024 14:12 UTC

Incident Status

Partial Service Disruption


Components

CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions


Locations

Google Compute Engine




April 11, 2024 14:12 UTC
[Resolved] This incident has been resolved and jobs are being picked up without any errors. More information can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/17819.

April 11, 2024 13:31 UTC
[Monitoring] The rollback has completed and we are seeing the jobs being picked up as normal. We're continuing to monitor to ensure the problem does not recur. Details in gitlab.com/gitlab-com/gl-infra/production/-/issues/17819

April 11, 2024 13:21 UTC
[Identified] We have started to roll back the relevant changes and anticipate that this should complete within 45 minutes. Details in - gitlab.com/gitlab-com/gl-infra/production/-/issues/17819

April 11, 2024 13:00 UTC
[Identified] We have started to roll back the relevant changes. Details in - gitlab.com/gitlab-com/gl-infra/production/-/issues/17819

April 11, 2024 12:30 UTC
[Identified] We have identified the cause of this problem and are starting to roll back the relevant changes. Details in - gitlab.com/gitlab-com/gl-infra/production/-/issues/17819

April 11, 2024 12:18 UTC
[Investigating] No material updates to report. We're continuing to investigate the issue. Tracking in gitlab.com/gitlab-com/gl-infra/production/-/issues/17819

April 11, 2024 12:03 UTC
[Investigating] We're currently investigating issues with jobs not being picked up by runners. More details in - gitlab.com/gitlab-com/gl-infra/production/-/issues/17819

March 2024

Elevated queue size on Shared Runners

March 15, 2024 18:40 UTC

Incident Status

Degraded Performance


Components

CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS


Locations

Google Compute Engine, AWS




March 15, 2024 18:40 UTC
[Resolved] Job pickup delays and slowness on running jobs have not been observed for some time, and the team does not anticipate the problem returning. Additional information will be shared in gitlab.com/gitlab-com/gl-infra/production/-/issues/17724

March 15, 2024 14:35 UTC
[Monitoring] As things remain stable, we will continue to monitor the situation and will provide further updates as necessary. Full details are available in gitlab.com/gitlab-com/gl-infra/production/-/issues/17724

March 15, 2024 13:34 UTC
[Monitoring] There are no material updates to report. We will continue to monitor and post updates hourly. Full details are available in gitlab.com/gitlab-com/gl-infra/production/-/issues/17724

March 15, 2024 12:33 UTC
[Monitoring] Things remain stable, with no material updates at the time. We are continuing to monitor the situation, and will provide hourly updates. Full details are available in gitlab.com/gitlab-com/gl-infra/production/-/issues/17724

March 15, 2024 11:30 UTC
[Monitoring] No material updates at the time. We are continuing to monitor the situation, and will provide hourly updates. Full details are available in gitlab.com/gitlab-com/gl-infra/production/-/issues/17724

March 15, 2024 10:29 UTC
[Monitoring] The impact from this incident appears to now be mitigated, however we will continue to monitor the situation and will provide further updates as necessary. Full details are available in gitlab.com/gitlab-com/gl-infra/production/-/issues/17724

March 15, 2024 09:26 UTC
[Monitoring] The impact from this incident appears low, however we are continuing to monitor the situation, and will post another update in one hour. Full details are available in gitlab.com/gitlab-com/gl-infra/production/-/issues/17724

March 15, 2024 08:15 UTC
[Monitoring] We are continuing to monitor this situation, with no additional details to add at this time. Full details are available in gitlab.com/gitlab-com/gl-infra/production/-/issues/17724

March 15, 2024 07:11 UTC
[Identified] No material updates to report. We will continue to monitor to the situation for now. Details in gitlab.com/gitlab-com/gl-infra/production/-/issues/17724

March 15, 2024 05:18 UTC
[Identified] We have narrowed down the potential root cause but we will continue the investigation. No further impact on CI/CD pipeline execution. Next update will be provided in 1 hour. Details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/17724

March 15, 2024 03:35 UTC
[Investigating] We are continuing to monitor the extended runner queue size. Impact on CI/CD pipeline execution does not appear to be significant at this time. A further update will be provided in 1 hour. Details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/17724

March 15, 2024 02:31 UTC
[Investigating] Analysis continues. A further update will be provided in 1 hour, or sooner should there be a significant update to report. Details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/17724

March 15, 2024 01:57 UTC
[Investigating] The investigation is ongoing, we continue to assess the root cause of the extended queue size with the help of additional resources. A further update will be provided in 30 minutes. Details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/17724

March 15, 2024 01:34 UTC
[Investigating] No material updates at this time. Investigation into the increasing runner queue size is ongoing. Details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/17724

March 15, 2024 01:13 UTC
[Investigating] We are continuing to investigate the root cause of the elevated queue size. For more information: gitlab.com/gitlab-com/gl-infra/production/-/issues/17724

March 15, 2024 00:55 UTC
[Investigating] No material update at this time. Our team is still investigating the root cause.

March 15, 2024 00:42 UTC
[Investigating] We are investigating an issue with elevated queue size on our GitLab.com runners. Users may experience slowness with runners picking up jobs, or running jobs. For more information: gitlab.com/gitlab-com/gl-infra/production/-/issues/17724

January 2024

Sidekiq degredation

January 30, 2024 23:02 UTC

Sidekiq degredationDegraded Performance

Incident Status

Degraded Performance


Components

Website, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions, Background Processing


Locations

Google Compute Engine




January 30, 2024 23:02 UTC
[Resolved] The feature flag responsible for the issue has been disabled by default and operations are back to normal. This incident is considered resolved. Further details can be found in the incident issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/17504

January 30, 2024 22:02 UTC
[Identified] We've identified the issue and have narrowed the problem to a recent feature flag that created slow queries caused by specific import workers. These slow queries lead to database saturation. To mitigate the issue, the team has disabled the feature flag. We are now re-processing jobs from these workers. This issue is considered mitigated. Further updates will be minimal until there is a permanent code fix in place. More details in gitlab.com/gitlab-com/gl-infra/production/-/issues/17504

January 30, 2024 20:22 UTC
[Investigating] No material update at this time. Our team is still investigating the root cause.

January 30, 2024 19:54 UTC
[Investigating] We've temporarily disabled some features of the GitHub import functionality. Users might experience issues with GitHub imports at this time while our team continues to investigate.

January 30, 2024 19:28 UTC
[Investigating] Sidekiq has largely recovered and jobs are processing. Our team is still investigating the root cause.

January 30, 2024 19:08 UTC
[Investigating] Our team is still investigating the root cause of this issue. We'll continue to provide updates as more information is available.

January 30, 2024 18:51 UTC
[Investigating] We're currently seeing some sidekiq degradation that appears to be impacting some pipelines and merge requests. Our team is investigating. More information in gitlab.com/gitlab-com/gl-infra/production/-/issues/17504

November 2023

Performance issues affecting processing of MRs

November 20, 2023 18:49 UTC

Incident Status

Degraded Performance


Components

CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions, Background Processing


Locations

Google Compute Engine, AWS




November 20, 2023 18:49 UTC
[Resolved] Performance of CI pipelines and merge requests has fully returned to normal and shows no further signs of degradation. The status page is being marked resolved, and future updates can be found in the production issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/17158

November 20, 2023 18:05 UTC
[Monitoring] Performance of CI job pickup and merge requests has returned to a normal state. The incident response team will continue monitoring and reviewing the rolled back changes. Next update in an hour, unless there is anything to report prior.

November 20, 2023 17:21 UTC
[Investigating] We have started to notice some signs of recovery after the rollback. Investigation into root causes is still ongoing to identify the source of the CI and merge request performance problems.

November 20, 2023 16:50 UTC
[Investigating] Rollback to an earlier deployment has completed. No material updates in the CI job pickup slowness has been observed yet. Next update will be in 30 minutes unless there is anything to report sooner.

November 20, 2023 16:32 UTC
[Investigating] The rollback to an earlier deployment is in progress and a review of all changes is in progress to identify the root cause of slow CI job pickup. Additional support from other teams is being brought in to assist.

November 20, 2023 16:12 UTC
[Investigating] The incident response team is continuing to investigate the slow CI job pickup. A rollback to a previous deployment is in progress as a potential mitigating solution.

November 20, 2023 15:53 UTC
[Investigating] We are actively investigating an issue where CI jobs are being picked up slowly affecting performance of MR's. More details here: gitlab.com/gitlab-com/gl-infra/production/-/issues/17158





Back to current status