June 2024

2024-06-11: Intermittent failures in starting up pipeline

June 11, 2024 09:32 UTC

2024-06-11: Intermittent failures in starting up pipelineDegraded Performance

Incident Status

Degraded Performance

Components

Background Processing

Locations

Google Compute Engine

---

June 11, 2024 09:32 UTC

[Resolved] This incident has been resolved and pipelines are starting up without errors now. More details are available here: gitlab.com/gitlab-com/gl-infra/production/-/issues/18136

June 11, 2024 08:26 UTC

[Monitoring] We are seeing normal functionality again after taking a mitigating action, we will continue monitoring the situation. If you are still seeing these issues, please comment in the production issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/18136

June 11, 2024 08:02 UTC

[Identified] We are still investigating the issue with pipelines not automatically triggering. This issue only affects groups with IP restrictions enabled. For updates, please refer to issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/18136

June 11, 2024 07:46 UTC

[Identified] We are still investigating the issue with pipelines not automatically triggering. This issue only affects groups with IP restrictions enabled. For updates, please refer to issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/18136

June 11, 2024 06:48 UTC

[Identified] We have identified the cause of the pipeline triggering issue and are actively working on a fix. For ongoing updates, please refer to the tracking issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/18136

June 11, 2024 06:18 UTC

[Investigating] We are currently experiencing an issue where pipelines are not automatically triggering as expected. However, they can still be triggered manually. For ongoing updates please refer to the tracking issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/18136

May 2024

Code Suggestions failing with 403 errors

May 24, 2024 12:34 UTC

Code Suggestions failing with 403 errorsPartial Service Disruption

Incident Status

Partial Service Disruption

Components

API

Locations

Google Compute Engine

---

May 24, 2024 12:34 UTC

[Resolved] Monitoring has not surfaced any further errors, so the incident is now considered resolved. See gitlab.com/gitlab-com/gl-infra/production/-/issues/18062 for more details

May 24, 2024 11:15 UTC

[Monitoring] We've identified the cause and applied a fix, and we are now no longer seeing failing requests. We will continue to monitor. See gitlab.com/gitlab-com/gl-infra/production/-/issues/18062 for more details

May 24, 2024 10:55 UTC

[Investigating] No material updates at this time; our investigation is ongoing. Details will be posted in: gitlab.com/gitlab-com/gl-infra/production/-/issues/18062

May 24, 2024 10:38 UTC

[Investigating] We are aware of a large percentage of GitLab Duo Code Suggestions requests failing with error 403. All IDE plugins are impacted. Details will be provided in: gitlab.com/gitlab-com/gl-infra/production/-/issues/18062

Degraded Performance Causing Intermittent 503 Errors

May 22, 2024 05:20 UTC

Degraded Performance Causing Intermittent 503 ErrorsDegraded Performance

Incident Status

Degraded Performance

Components

Website

Locations

Google Compute Engine

---

May 22, 2024 05:20 UTC

[Resolved] The issue has been resolved, and the website is now performing normally without any issues. The full timeline of this incident is available here: gitlab.com/gitlab-com/gl-infra/production/-/issues/18046

May 22, 2024 04:47 UTC

[Monitoring] The services are being recovered, and we will continue to monitor the situation before marking the affected components as operational. For more details see gitlab.com/gitlab-com/gl-infra/production/-/issues/18046

May 22, 2024 04:13 UTC

[Investigating] We are investigating an increase in error rates while using GitLab.com. For more details see gitlab.com/gitlab-com/gl-infra/production/-/issues/18046

Pipeline jobs not being picked up and 500 errors

May 17, 2024 02:14 UTC

Pipeline jobs not being picked up and 500 errorsPartial Service Disruption

Incident Status

Partial Service Disruption

Components

Website, API, CI/CD - Hosted runners on Linux, Background Processing

Locations

Google Compute Engine

---

May 17, 2024 02:14 UTC

[Resolved] Ongoing monitoring is showing that all systems are working without issue. Google has also resolved their incident. We will mark this incident as resolved. See following issue for more information: gitlab.com/gitlab-com/gl-infra/production/-/issues/18023

May 17, 2024 00:48 UTC

[Monitoring] We are seeing normal functionality again in our logging but we will continue monitoring. If you are still seeing these issues, please comment in the production issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/18023

May 17, 2024 00:24 UTC

[Identified] We have identified that these outages are impacted by the Google Cloud Service disruptions. For more information, see gitlab.com/gitlab-com/gl-infra/production/-/issues/18023. To follow Google Cloud status information, see status.cloud.google.com/incidents/xVSEV3kVaJBmS7SZbnre#RP1d9aZLNFZEJmTBk8e1

May 17, 2024 00:01 UTC

[Investigating] GitLab is investigating reports of pipelines not being picked up and 500 errors. We have two incidents open and we are working to understand the cause of these errors and whether they are related. We are investigating if this is related to this Google issue: status.cloud.google.com/incidents/xVSEV3kVaJBmS7SZbnre#RP1d9aZLNFZEJmTBk8e1 Issue links: gitlab.com/gitlab-com/gl-infra/production/-/issues/18024, gitlab.com/gitlab-com/gl-infra/production/-/issues/18023

April 2024

Stuck merge requests

April 29, 2024 15:06 UTC

Stuck merge requestsPartial Service Disruption

Incident Status

Partial Service Disruption

Components

Website

Locations

Google Compute Engine

---

April 29, 2024 15:06 UTC

[Resolved] We are closing this incident because the system is operational. There are still 1489 merge requests in an incorrect state. We are working on correcting them and discussing options in the issue. More details are available in gitlab.com/gitlab-com/gl-infra/production/-/issues/17921.

April 29, 2024 11:29 UTC

[Identified] We are working on unlocking the affected merge requests. A workaround for the "Merge request is not mergeable" error is to close and reopen the merge request.

April 29, 2024 11:04 UTC

[Investigating] We had an increased rate in web service errors. The errors cleared, but some merge requests remained in a locked state. We are working on unlocking these merge requests, more details are available in gitlab.com/gitlab-com/gl-infra/production/-/issues/17921.

SaaS Runners unable to pull from Microsoft Container Registry

April 24, 2024 10:37 UTC

SaaS Runners unable to pull from Microsoft Container RegistryPartial Service Disruption

Incident Status

Partial Service Disruption

Components

CI/CD - Hosted runners on Linux

Locations

Google Compute Engine

---

April 24, 2024 10:37 UTC

[Resolved] This incident has been resolved after switching the runners to another zone. More details are available in gitlab.com/gitlab-com/gl-infra/production/-/issues/17896.

April 24, 2024 09:13 UTC

[Identified] We have confirmed that there is a network problem when accessing the container registry, please continue to follow gitlab.com/gitlab-com/gl-infra/production/-/issues/17896 for further details.

April 24, 2024 03:56 UTC

[Investigating] No material updates at this time. Our investigation is ongoing. We will provide an update once we've received advice from our cloud providers. Please see incident issue for more details: gitlab.com/gitlab-com/gl-infra/production/-/issues/17896

April 24, 2024 03:28 UTC

[Investigating] We have requested assistance from our cloud hosting providers to investigate the connectivity issues. Investigation is still ongoing. Please see incident issue for more details: gitlab.com/gitlab-com/gl-infra/production/-/issues/17896

April 24, 2024 03:05 UTC

[Investigating] We are investigating a potential connectivity issue between Microsoft Container Registry and our SaaS GitLab Runners. Please see incident issue for more details: gitlab.com/gitlab-com/gl-infra/production/-/issues/17896

April 24, 2024 02:41 UTC

[Investigating] We're currently investigating reports of GitLab SaaS Runners failing to pull images from Microsoft Container Registry (mcr.microsoft.com). Please see incident issue for more details: gitlab.com/gitlab-com/gl-infra/production/-/issues/17896

Downstream Pipelines Returning Errors

April 23, 2024 15:19 UTC

Downstream Pipelines Returning ErrorsDegraded Performance

Incident Status

Degraded Performance

Components

CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions

Locations

Google Compute Engine

---

April 23, 2024 15:19 UTC

[Resolved] This incident has been resolved and pipelines are running without errors now. More details are available in gitlab.com/gitlab-com/gl-infra/production/-/issues/17885.

April 22, 2024 21:56 UTC

[Identified] A fix for the issue has been merged and will be available in the next deployment tomorrow morning. Full details here gitlab.com/gitlab-com/gl-infra/production/-/issues/17885.

April 22, 2024 20:43 UTC

[Identified] We've identified the cause of the issue and preparations for a fix are ongoing. Details in gitlab.com/gitlab-com/gl-infra/production/-/issues/17885.

April 22, 2024 18:51 UTC

[Identified] We've identified the cause of the issue and are preparing options to deploy a fix. Details in gitlab.com/gitlab-com/gl-infra/production/-/issues/17885.

April 22, 2024 18:24 UTC

[Identified] We confirmed that the issue is isolated to a small number of namespaces and are continuing to investigate. Details in gitlab.com/gitlab-com/gl-infra/production/-/issues/17885.

April 22, 2024 17:56 UTC

[Investigating] We're investigating reports of parent-child pipeline setups returning an error and failing to create a downstream pipeline. Details in gitlab.com/gitlab-com/gl-infra/production/-/issues/17885.

Errors managing Kubernetes agents

April 23, 2024 15:13 UTC

Errors managing Kubernetes agentsPartial Service Disruption

Incident Status

Partial Service Disruption

Components

Website, API

Locations

Google Compute Engine

---

April 23, 2024 15:13 UTC

[Resolved] The fix for this issue was deployed and this incident has been resolved. More information is available in gitlab.com/gitlab-com/gl-infra/production/-/issues/17887.

April 23, 2024 03:36 UTC

[Identified] A fix has been merged and will be available in the next deployment tomorrow morning. Please follow gitlab.com/gitlab-com/gl-infra/production/-/issues/17887 for more details

April 23, 2024 02:52 UTC

[Identified] We are in the process of merging code changes to fix the issue. Next status update will be once the Merge Request is merged. Please follow gitlab.com/gitlab-com/gl-infra/production/-/issues/17887 for more details

April 23, 2024 02:23 UTC

[Identified] We have identified the potential cause of the issue and we are working on mitigating the issue. See gitlab.com/gitlab-com/gl-infra/production/-/issues/17887 for more details

April 23, 2024 02:08 UTC

[Investigating] No material updates to report. We're continuing to investigate the issue. Tracking in gitlab.com/gitlab-com/gl-infra/production/-/issues/17887

April 23, 2024 01:50 UTC

[Investigating] Some users are experiencing errors managing Kubernetes agents, or listing existing agents. We are actively investigating the issue. Details can be found in: gitlab.com/gitlab-com/gl-infra/production/-/issues/17887

Dependency Proxy 500 errors

April 18, 2024 15:32 UTC

Dependency Proxy 500 errorsDegraded Performance

Incident Status

Degraded Performance

Components

Container Registry

Locations

Google Compute Engine

Postmortem

https://gitlab.com/gitlab-com/gl-infra/production/-/issues/17851

---

April 18, 2024 15:32 UTC

[Resolved] This incident is resolved now. See gitlab.com/gitlab-com/gl-infra/production/-/issues/17851 for more details

April 18, 2024 05:00 UTC

[Monitoring] There's no material to update at the moment. We'll continue to monitor the situation. See gitlab.com/gitlab-com/gl-infra/production/-/issues/17851 for more details

April 18, 2024 02:51 UTC

[Monitoring] The rollback has completed successfully. We have confirmed that we no longer see reports of this issue but we will continue monitoring. See gitlab.com/gitlab-com/gl-infra/production/-/issues/17851 for more details

April 17, 2024 20:29 UTC

[Identified] Work is underway to revert the problematic changes. We do not expect this change to land in production for 8-12 hours. Once we have confirmed the fix has been deployed, we will post an update here. We sincerely apologize for the delay. Details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/17851

April 17, 2024 18:59 UTC

[Identified] We believe to have identified the issue. We are working on a revert to address the problem. Details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/17851

April 17, 2024 18:18 UTC

[Investigating] No material updates at this time. We continue to investigate the issue. Details will be provided in: gitlab.com/gitlab-com/gl-infra/production/-/issues/17851

April 17, 2024 17:55 UTC

[Investigating] No material updates at this time. Our investigation is ongoing. Details will be posted in: gitlab.com/gitlab-com/gl-infra/production/-/issues/17851

April 17, 2024 17:37 UTC

[Investigating] Our investigation is ongoing. This appears to be impacting only a small portion of users. Details will be posted in: gitlab.com/gitlab-com/gl-infra/production/-/issues/17851

April 17, 2024 17:13 UTC

[Investigating] No material updates at this time. Our investigation is ongoing. Details will be posted in: gitlab.com/gitlab-com/gl-infra/production/-/issues/17851

April 17, 2024 16:56 UTC

[Investigating] We are aware of some jobs encountering 500 errors when attempting to pull images via the Dependency Proxy. We are investigating. Details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/17851

500 errors during SAML sign in

April 18, 2024 04:06 UTC

500 errors during SAML sign inPartial Service Disruption

Incident Status

Partial Service Disruption

Components

SAML SSO - GitLab SaaS

Locations

Google Compute Engine

---

April 18, 2024 04:06 UTC

[Resolved] This incident is now resolved. There have been no further reports of SAML SSO on GitLab.com throwing 500 errors. For more details, please see gitlab.com/gitlab-com/gl-infra/production/-/issues/17856

April 18, 2024 01:42 UTC

[Monitoring] Rollback of the change that caused the incident has completed. We expect the issue to be resolved but will continue monitoring. For more details, see gitlab.com/gitlab-com/gl-infra/production/-/issues/17856 .

April 18, 2024 01:21 UTC

[Identified] We have started a rollback and expect it to complete around 02:15AM UTC April 18 (about 1 hour). For more details, see gitlab.com/gitlab-com/gl-infra/production/-/issues/17856.

April 18, 2024 01:07 UTC

[Identified] We have identified the potential cause of the issue and we are working on mitigating the issue. See gitlab.com/gitlab-com/gl-infra/production/-/issues/17856 for more details

April 18, 2024 00:43 UTC

[Investigating] Investigations are ongoing. Please see gitlab.com/gitlab-com/gl-infra/production/-/issues/17856 for more details.

April 18, 2024 00:13 UTC

[Investigating] We are of some users receiving a 500 error when signing in via SAML. We are actively investigating the issue. Details can be found in: gitlab.com/gitlab-com/gl-infra/production/-/issues/17856

Runners are not picking up jobs causing 500 errors

April 11, 2024 14:12 UTC

Runners are not picking up jobs causing 500 errorsPartial Service Disruption

Incident Status

Partial Service Disruption

Components

CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions

Locations

Google Compute Engine

---

April 11, 2024 14:12 UTC

[Resolved] This incident has been resolved and jobs are being picked up without any errors. More information can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/17819.

April 11, 2024 13:31 UTC

[Monitoring] The rollback has completed and we are seeing the jobs being picked up as normal. We're continuing to monitor to ensure the problem does not recur. Details in gitlab.com/gitlab-com/gl-infra/production/-/issues/17819

April 11, 2024 13:21 UTC

[Identified] We have started to roll back the relevant changes and anticipate that this should complete within 45 minutes. Details in - gitlab.com/gitlab-com/gl-infra/production/-/issues/17819

April 11, 2024 13:00 UTC

[Identified] We have started to roll back the relevant changes. Details in - gitlab.com/gitlab-com/gl-infra/production/-/issues/17819

April 11, 2024 12:30 UTC

[Identified] We have identified the cause of this problem and are starting to roll back the relevant changes. Details in - gitlab.com/gitlab-com/gl-infra/production/-/issues/17819

April 11, 2024 12:18 UTC

[Investigating] No material updates to report. We're continuing to investigate the issue. Tracking in gitlab.com/gitlab-com/gl-infra/production/-/issues/17819

April 11, 2024 12:03 UTC

[Investigating] We're currently investigating issues with jobs not being picked up by runners. More details in - gitlab.com/gitlab-com/gl-infra/production/-/issues/17819

Degraded performance causing intermittent 500 errors

April 8, 2024 15:12 UTC

Degraded performance causing intermittent 500 errorsService Disruption

Incident Status

Service Disruption

Components

Website, Git Operations

Locations

Google Compute Engine

---

April 8, 2024 15:12 UTC

[Resolved] This incident has been resolved. More information can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/17790

April 8, 2024 14:37 UTC

[Identified] We have identified the cause for the increase in error rates and have taken steps to mitigate the source of the errors. For more details see gitlab.com/gitlab-com/gl-infra/production/-/issues/17790

April 8, 2024 14:21 UTC

[Investigating] We are still investigating an increase in error rates on some projects when viewing MRs and project details. For more details see gitlab.com/gitlab-com/gl-infra/production/-/issues/17790

April 8, 2024 14:02 UTC

[Investigating] We are still investigating the increase in error rates on some projects when viewing MRs and project details. For more details see gitlab.com/gitlab-com/gl-infra/production/-/issues/17790

April 8, 2024 13:42 UTC

[Investigating] We are investigating an increase in error rates on some projects when viewing MRs and project details. For more details see gitlab.com/gitlab-com/gl-infra/production/-/issues/17790

500 errors upon groups creation via API

April 4, 2024 13:17 UTC

500 errors upon groups creation via APIPartial Service Disruption

Incident Status

Partial Service Disruption

Components

API

Locations

Google Compute Engine

---

April 4, 2024 13:17 UTC

[Resolved] This incident has been resolved and the API endpoint is now operational. More information can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/17775.

April 3, 2024 23:08 UTC

[Identified] The fix has been merged and t is currently in the middle of our auto-deploy process. It shall land into production in the next deploy cycle. Next update once fix is deployed to production. See gitlab.com/gitlab-com/gl-infra/production/-/issues/17775 for more details.

April 3, 2024 19:28 UTC

[Identified] We are working on a code Merge Request to fix this issue. Next status update will be once the Merge Request is merged. Please follow gitlab.com/gitlab-com/gl-infra/production/-/issues/17775 for details.

April 3, 2024 19:05 UTC

[Identified] We originally reported project creation was also affected by this incident, however, we would like to clarify that we've only found evidence of group creation being affected. See gitlab.com/gitlab-com/gl-infra/production/-/issues/17775 for details.

April 3, 2024 18:49 UTC

[Identified] We have identified the recent deprecation of the "emails_disabled" attribute in favor of "emails_enabled" attribute as the cause for this issue and we are working on a solution. See gitlab.com/gitlab-com/gl-infra/production/-/issues/17775#note_1844428763 for more details.

April 3, 2024 18:16 UTC

[Investigating] API calls to create groups/projects on GitLab.com are receiving a 500 response if the "emails_disabled" attribute is specified. Please follow gitlab.com/gitlab-com/gl-infra/production/-/issues/17775 for more details

Customers Portal maintenance mode

April 1, 2024 10:49 UTC

Customers Portal maintenance modePartial Service Disruption

Incident Status

Partial Service Disruption

Components

GitLab Customers Portal

Locations

Google Compute Engine

---

April 1, 2024 10:49 UTC

[Resolved] This incident has been resolved and the customer portal is now operational. More information can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/17767.

April 1, 2024 10:39 UTC

[Identified] The Customers Portal (customers.gitlab.com) is currently in maintenance mode due to a 3rd party API having issues at the moment. For more information see gitlab.com/gitlab-com/gl-infra/production/-/issues/17767

March 2024

Vulnerability Report Background Processing Degradation

March 26, 2024 02:28 UTC

Vulnerability Report Background Processing DegradationDegraded Performance

Incident Status

Degraded Performance

Components

Background Processing

Locations

Google Compute Engine

Postmortem

https://gitlab.com/gitlab-com/gl-infra/production/-/issues/17754

---

March 26, 2024 02:28 UTC

[Resolved] A fix has been deployed, and background processing of vulnerability reports is now operating normally. This incident has been resolved.

March 25, 2024 20:42 UTC

[Identified] We are currently in the process of deploying a solution aimed at resolving this incident. Our next update will be posted in 6 hours. More details in gitlab.com/gitlab-com/gl-infra/production/-/issues/17754

March 25, 2024 14:44 UTC

[Investigating] We're currently investigating issues with the background processing which updates the vulnerability report. The report is not being updated from new pipeline runs as of 13:28 UTC on 2024-03-25. We are working to restore this background processing. Existing report data continues to function as normal but may be outdated. Next update will be posted in 6 hours. More details in - gitlab.com/gitlab-com/gl-infra/production/-/issues/17754

Package registry upload issues

March 23, 2024 00:03 UTC

Package registry upload issuesDegraded Performance

Incident Status

Degraded Performance

Components

API

Locations

Google Compute Engine

---

March 23, 2024 00:03 UTC

[Resolved] GitLab is no longer observing errors with artifact and package repository uploads. We are resolving this incident at this time. More details in gitlab.com/gitlab-com/gl-infra/production/-/issues/17749

March 22, 2024 23:47 UTC

[Monitoring] GitLab believes that issues with uploads to the package registry and artifact uploads have been identified and resolved. We are monitoring for the next fifteen minutes to ensure that no further issues are observed before we resolve this incident. More details in gitlab.com/gitlab-com/gl-infra/production/-/issues/17749

March 22, 2024 23:04 UTC

[Investigating] GitLab believes that package registry and artifact issues is related to inbound HTTP2 connections, which was exposed by the CloudFlare setting "HTTP/2 to origin". We have disabled this feature and are monitoring to verify uploads are working. We will update in 30 minutes. More details in gitlab.com/gitlab-com/gl-infra/production/-/issues/17749

March 22, 2024 21:16 UTC

[Investigating] GitLab is continuing to investigate artifact and package registry upload issues. We are continuing work at understanding the root cause of these upload errors. We will post our next update in one hour. More details in gitlab.com/gitlab-com/gl-infra/production/-/issues/17749

March 22, 2024 20:44 UTC

[Investigating] GitLab is continuing to investigate artifact and package registry upload issues. GitLab is monitoring the issue and working CloudFlare to understand these connection issues. We will post our next update in 30 minutes. More details in gitlab.com/gitlab-com/gl-infra/production/-/issues/17749

March 22, 2024 20:05 UTC

[Investigating] GitLab is continuing to investigate artifact and package registry upload issues. GitLab is monitoring the issue and working CloudFlare to understand these connection issues. We will post our next update in 30 minutes. More details in gitlab.com/gitlab-com/gl-infra/production/-/issues/17749

March 22, 2024 19:27 UTC

[Investigating] GitLab is continuing to investigate artifact and package registry upload issues. GitLab is monitoring the issue and working CloudFlare to understand these connection issues. We will post our next update in 30 minutes. More details in gitlab.com/gitlab-com/gl-infra/production/-/issues/17749

March 22, 2024 19:09 UTC

[Investigating] GitLab is continuing to investigate artifact and package registry upload issues. GitLab is monitoring the issue and working CloudFlare to understand these connection issues. More details in gitlab.com/gitlab-com/gl-infra/production/-/issues/17749

March 22, 2024 18:51 UTC

[Investigating] GitLab is continuing to investigate artifact and package registry upload issues. We have reached out to CloudFlare so we can understand the connection issues we are observing. More details in gitlab.com/gitlab-com/gl-infra/production/-/issues/17749

March 22, 2024 18:35 UTC

[Investigating] GitLab is continuing to investigate artifact and package registry upload issues. We have reached out to CloudFlare so we can understand the connection issues we are observing. More details in gitlab.com/gitlab-com/gl-infra/production/-/issues/17749

March 22, 2024 18:15 UTC

[Investigating] GitLab is investigating issues with package and artifact uploads. We have observed elevated error rates with uploads and we are working to understand the root cause. More details in gitlab.com/gitlab-com/gl-infra/production/-/issues/17749

Elevated queue size on Shared Runners

March 15, 2024 18:40 UTC

Elevated queue size on Shared RunnersDegraded Performance

Incident Status

Degraded Performance

Components

CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS

Locations

Google Compute Engine, AWS

---

March 15, 2024 18:40 UTC

[Resolved] Job pickup delays and slowness on running jobs have not been observed for some time, and the team does not anticipate the problem returning. Additional information will be shared in gitlab.com/gitlab-com/gl-infra/production/-/issues/17724

March 15, 2024 14:35 UTC

[Monitoring] As things remain stable, we will continue to monitor the situation and will provide further updates as necessary. Full details are available in gitlab.com/gitlab-com/gl-infra/production/-/issues/17724

March 15, 2024 13:34 UTC

[Monitoring] There are no material updates to report. We will continue to monitor and post updates hourly. Full details are available in gitlab.com/gitlab-com/gl-infra/production/-/issues/17724

March 15, 2024 12:33 UTC

[Monitoring] Things remain stable, with no material updates at the time. We are continuing to monitor the situation, and will provide hourly updates. Full details are available in gitlab.com/gitlab-com/gl-infra/production/-/issues/17724

March 15, 2024 11:30 UTC

[Monitoring] No material updates at the time. We are continuing to monitor the situation, and will provide hourly updates. Full details are available in gitlab.com/gitlab-com/gl-infra/production/-/issues/17724

March 15, 2024 10:29 UTC

[Monitoring] The impact from this incident appears to now be mitigated, however we will continue to monitor the situation and will provide further updates as necessary. Full details are available in gitlab.com/gitlab-com/gl-infra/production/-/issues/17724

March 15, 2024 09:26 UTC

[Monitoring] The impact from this incident appears low, however we are continuing to monitor the situation, and will post another update in one hour. Full details are available in gitlab.com/gitlab-com/gl-infra/production/-/issues/17724

March 15, 2024 08:15 UTC

[Monitoring] We are continuing to monitor this situation, with no additional details to add at this time. Full details are available in gitlab.com/gitlab-com/gl-infra/production/-/issues/17724

March 15, 2024 07:11 UTC

[Identified] No material updates to report. We will continue to monitor to the situation for now. Details in gitlab.com/gitlab-com/gl-infra/production/-/issues/17724

March 15, 2024 05:18 UTC

[Identified] We have narrowed down the potential root cause but we will continue the investigation. No further impact on CI/CD pipeline execution. Next update will be provided in 1 hour. Details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/17724

March 15, 2024 03:35 UTC

[Investigating] We are continuing to monitor the extended runner queue size. Impact on CI/CD pipeline execution does not appear to be significant at this time. A further update will be provided in 1 hour. Details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/17724

March 15, 2024 02:31 UTC

[Investigating] Analysis continues. A further update will be provided in 1 hour, or sooner should there be a significant update to report. Details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/17724

March 15, 2024 01:57 UTC

[Investigating] The investigation is ongoing, we continue to assess the root cause of the extended queue size with the help of additional resources. A further update will be provided in 30 minutes. Details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/17724

March 15, 2024 01:34 UTC

[Investigating] No material updates at this time. Investigation into the increasing runner queue size is ongoing. Details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/17724

March 15, 2024 01:13 UTC

[Investigating] We are continuing to investigate the root cause of the elevated queue size. For more information: gitlab.com/gitlab-com/gl-infra/production/-/issues/17724

March 15, 2024 00:55 UTC

[Investigating] No material update at this time. Our team is still investigating the root cause.

March 15, 2024 00:42 UTC

[Investigating] We are investigating an issue with elevated queue size on our GitLab.com runners. Users may experience slowness with runners picking up jobs, or running jobs. For more information: gitlab.com/gitlab-com/gl-infra/production/-/issues/17724

Security policies giving 500 error Security Scans may fail

March 2, 2024 00:22 UTC

Security policies giving 500 error Security Scans may failDegraded Performance

Incident Status

Degraded Performance

Components

Website

Locations

Google Compute Engine

---

March 2, 2024 00:22 UTC

[Resolved] The revert has been completed and our logs show no indication of the the Policies page 500 error. This incident has been resolved. More details can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/17690

March 1, 2024 22:24 UTC

[Identified] Currently, Security Policies are showing a 500 error and may not be accessible. Security Scans may fail at this time as well. Revert is almost complete. It has passed staging and is on its way to production. A workaround is available on the issue. gitlab.com/gitlab-com/gl-infra/production/-/issues/17690

March 1, 2024 19:58 UTC

[Identified] Currently, Security Policies are showing a 500 error and may not be accessible. Security Scans may fail at this time as well. Revert has started waiting for it to complete. A workaround is available on the issue. gitlab.com/gitlab-com/gl-infra/production/-/issues/17690

March 1, 2024 18:41 UTC

[Identified] Currently, Security Policies are showing a 500 error and may not be accessible. Security Scans may fail at this time as well. Revert has started waiting for it to complete. A workaround is available on the issue. gitlab.com/gitlab-com/gl-infra/production/-/issues/17690

March 1, 2024 18:33 UTC

[Investigating] Currently, Security Policies are throwing a 500 error and may not be accessible. We are currently investigating this issue. Please see gitlab.com/gitlab-com/gl-infra/production/-/issues/17690

500 error when signing into GitLab.com with SAML

March 2, 2024 00:21 UTC

500 error when signing into GitLab.com with SAMLPartial Service Disruption

Incident Status

Partial Service Disruption

Components

SAML SSO - GitLab SaaS

Locations

Google Compute Engine

---

March 2, 2024 00:21 UTC

[Resolved] The revert has been completed and our logs show no indication of the Policies page 500 error. This incident has been resolved. More details can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/17690

February 28, 2024 14:56 UTC

[Resolved] This incident has been resolved and SAML login is now operational. More information can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/17672

February 28, 2024 14:29 UTC

[Monitoring] We have applied a fix and are now monitoring the state for 1 hour before marking as resolved. More details can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/17672.

February 28, 2024 11:39 UTC

[Identified] We are on track in terms of resolution. The fix should be present on GitLab.com within the next 3h. While this affects a subset of SAML providers (including Akamai MFA, Cisco Duo, Keycloak, NetIQ, PingOne MFA) one workaround is to temporary disable SAML authentication. More details can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/17672.

February 28, 2024 10:06 UTC

[Identified] We've identified the underlying issue and are working on resolving it. The expected time to resolution is about 4h from now. More details can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/17672.

February 28, 2024 09:13 UTC

[Identified] We've identified the cause of the issue and are working on resolving it. While this affects a subset of SAML providers - the current workaround would be to temporary disable SAML until the issue is resolved. More details in gitlab.com/gitlab-com/gl-infra/production/-/issues/17672

February 28, 2024 06:51 UTC

[Monitoring] We are seeing report of users getting 500 errors when signing into their GitLab.com group with SAML SSO. We have identified the problem and reverted the change that introduce the error. We'll continue to monitor the problem. See more information in this issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/17672

February 2024

Bad gateway errors

February 27, 2024 13:10 UTC

Bad gateway errorsDegraded Performance

Incident Status

Degraded Performance

Components

Website

Locations

Google Compute Engine

---

February 27, 2024 13:10 UTC

[Resolved] This incident has been resolved. More information can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/17667

February 27, 2024 12:44 UTC

[Monitoring] The service status has been restored, however we are still looking into the root cause of the issue. More details can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/17667

February 27, 2024 12:18 UTC

[Investigating] We are currently investigating 502 errors on GitLab.com. More details about this incident can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/17667