All Systems Operational

Updated a few seconds ago

Back to current status

Status History

Filter: CI/CD - Hosted runners on Linux (Clear)



August 2024

CI/CD token allowlist selection failing

August 27, 2024 14:52 UTC

CI/CD token allowlist selection failingPartial Service Disruption

Incident Status

Partial Service Disruption


Components

Website, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions


Locations

Google Compute Engine, AWS




August 27, 2024 14:52 UTC
[Resolved] The revert MR has been deployed to production, and we see that CI/CD token allowlist project selection renders properly now. Details will be provided in the incident issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/18446

August 27, 2024 03:55 UTC
[Identified] We have identified an issue that is causing CI/CD token allowlist project selection to fail. We are in the process of reverting the change that caused this. For more details see gitlab.com/gitlab-com/gl-infra/production/-/issues/18446.

Some requests timing out

August 23, 2024 03:59 UTC

Incident Status

Operational


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions, SAML SSO - GitLab SaaS, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, docs.gitlab.com, Canary, Product Analytics - Configurator (Beta), Product Analytics - Collector (Beta), Product Analytics - Background processing (Beta), Product Analytics - Queries (Beta), GitLab Duo


Locations

Google Compute Engine, Digital Ocean, Zendesk, AWS




August 23, 2024 03:59 UTC
[Resolved] The issue has been resolved and all functionality on GitLab.com is nominal. Please see gitlab.com/gitlab-com/gl-infra/production/-/issues/18435 for more details.

August 23, 2024 01:53 UTC
[Monitoring] The issue has been identified and the incident resolved. We will continue monitoring for a while.

August 23, 2024 01:27 UTC
[Identified] We have identified an underlying cause and are working to mitigate it. See gitlab.com/gitlab-com/gl-infra/production/-/issues/18435 for more details.

August 23, 2024 00:34 UTC
[Investigating] We have noticed that there are delays in web UI updates for example, MRs not showing up after submission, We are continuing to investigate - please see gitlab.com/gitlab-com/gl-infra/production/-/issues/18435 for more details.

August 23, 2024 00:27 UTC
[Investigating] Some requests are slow or timing out due to DB saturation. We are investigating. Please see [this issue](gitlab.com/gitlab-com/gl-infra/production/-/issues/18435) for further details.

July 2024

GitLab.com is unavailable

July 11, 2024 18:28 UTC

GitLab.com is unavailableService Disruption

Incident Status

Service Disruption


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions, SAML SSO - GitLab SaaS, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, docs.gitlab.com, Canary, Product Analytics - Configurator (Beta), Product Analytics - Collector (Beta), Product Analytics - Background processing (Beta), Product Analytics - Queries (Beta), GitLab Duo


Locations

Google Compute Engine, Digital Ocean, Zendesk, AWS




July 11, 2024 18:28 UTC
[Resolved] This incident is now resolved and no further errors are being seen on GitLab.com. Our investigation points to a disk performance issue. You can see further details and corrective actions in gitlab.com/gitlab-com/gl-infra/production/-/issues/18269.

July 11, 2024 15:51 UTC
[Investigating] Our investigations are ongoing. We will post our next update when we have more information. Please follow the incident issue for updates here: gitlab.com/gitlab-com/gl-infra/production/-/issues/18269

July 11, 2024 15:28 UTC
[Investigating] No material updates to report. We're continuing to investigate the issue. Please follow the incident issue for updates here: gitlab.com/gitlab-com/gl-infra/production/-/issues/18269

July 11, 2024 15:00 UTC
[Investigating] We are seeing improved service availability and investigations continue for this problem. Please follow the incident issue for updates here: gitlab.com/gitlab-com/gl-infra/production/-/issues/18269

July 11, 2024 14:43 UTC
[Investigating] We are seeing improved service availability and investigations continue for this problem. We are planning to make the incident issue publicly available as soon as possible.

July 11, 2024 14:29 UTC
[Investigating] We are investigating a problem with gitlab.com being unavailable for some users. Our investigations continue and we will post further status updates shortly.

July 11, 2024 14:18 UTC
[Investigating] We are investigating a problem with gitlab.com being unavailable for some users. We will post further status updates shortly.

May 2024

Pipeline jobs not being picked up and 500 errors

May 17, 2024 02:14 UTC

Incident Status

Partial Service Disruption


Components

Website, API, CI/CD - Hosted runners on Linux, Background Processing


Locations

Google Compute Engine




May 17, 2024 02:14 UTC
[Resolved] Ongoing monitoring is showing that all systems are working without issue. Google has also resolved their incident. We will mark this incident as resolved. See following issue for more information: gitlab.com/gitlab-com/gl-infra/production/-/issues/18023

May 17, 2024 00:48 UTC
[Monitoring] We are seeing normal functionality again in our logging but we will continue monitoring. If you are still seeing these issues, please comment in the production issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/18023

May 17, 2024 00:24 UTC
[Identified] We have identified that these outages are impacted by the Google Cloud Service disruptions. For more information, see gitlab.com/gitlab-com/gl-infra/production/-/issues/18023. To follow Google Cloud status information, see status.cloud.google.com/incidents/xVSEV3kVaJBmS7SZbnre#RP1d9aZLNFZEJmTBk8e1

May 17, 2024 00:01 UTC
[Investigating] GitLab is investigating reports of pipelines not being picked up and 500 errors. We have two incidents open and we are working to understand the cause of these errors and whether they are related. We are investigating if this is related to this Google issue: status.cloud.google.com/incidents/xVSEV3kVaJBmS7SZbnre#RP1d9aZLNFZEJmTBk8e1 Issue links: gitlab.com/gitlab-com/gl-infra/production/-/issues/18024, gitlab.com/gitlab-com/gl-infra/production/-/issues/18023

April 2024

SaaS Runners unable to pull from Microsoft Container Registry

April 24, 2024 10:37 UTC

Incident Status

Partial Service Disruption


Components

CI/CD - Hosted runners on Linux


Locations

Google Compute Engine




April 24, 2024 10:37 UTC
[Resolved] This incident has been resolved after switching the runners to another zone. More details are available in gitlab.com/gitlab-com/gl-infra/production/-/issues/17896.

April 24, 2024 09:13 UTC
[Identified] We have confirmed that there is a network problem when accessing the container registry, please continue to follow gitlab.com/gitlab-com/gl-infra/production/-/issues/17896 for further details.

April 24, 2024 03:56 UTC
[Investigating] No material updates at this time. Our investigation is ongoing. We will provide an update once we've received advice from our cloud providers. Please see incident issue for more details: gitlab.com/gitlab-com/gl-infra/production/-/issues/17896

April 24, 2024 03:28 UTC
[Investigating] We have requested assistance from our cloud hosting providers to investigate the connectivity issues. Investigation is still ongoing. Please see incident issue for more details: gitlab.com/gitlab-com/gl-infra/production/-/issues/17896

April 24, 2024 03:05 UTC
[Investigating] We are investigating a potential connectivity issue between Microsoft Container Registry and our SaaS GitLab Runners. Please see incident issue for more details: gitlab.com/gitlab-com/gl-infra/production/-/issues/17896

April 24, 2024 02:41 UTC
[Investigating] We're currently investigating reports of GitLab SaaS Runners failing to pull images from Microsoft Container Registry (mcr.microsoft.com). Please see incident issue for more details: gitlab.com/gitlab-com/gl-infra/production/-/issues/17896

Downstream Pipelines Returning Errors

April 23, 2024 15:19 UTC

Incident Status

Degraded Performance


Components

CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions


Locations

Google Compute Engine




April 23, 2024 15:19 UTC
[Resolved] This incident has been resolved and pipelines are running without errors now. More details are available in gitlab.com/gitlab-com/gl-infra/production/-/issues/17885.

April 22, 2024 21:56 UTC
[Identified] A fix for the issue has been merged and will be available in the next deployment tomorrow morning. Full details here gitlab.com/gitlab-com/gl-infra/production/-/issues/17885.

April 22, 2024 20:43 UTC
[Identified] We've identified the cause of the issue and preparations for a fix are ongoing. Details in gitlab.com/gitlab-com/gl-infra/production/-/issues/17885.

April 22, 2024 18:51 UTC
[Identified] We've identified the cause of the issue and are preparing options to deploy a fix. Details in gitlab.com/gitlab-com/gl-infra/production/-/issues/17885.

April 22, 2024 18:24 UTC
[Identified] We confirmed that the issue is isolated to a small number of namespaces and are continuing to investigate. Details in gitlab.com/gitlab-com/gl-infra/production/-/issues/17885.

April 22, 2024 17:56 UTC
[Investigating] We're investigating reports of parent-child pipeline setups returning an error and failing to create a downstream pipeline. Details in gitlab.com/gitlab-com/gl-infra/production/-/issues/17885.

Runners are not picking up jobs causing 500 errors

April 11, 2024 14:12 UTC

Incident Status

Partial Service Disruption


Components

CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions


Locations

Google Compute Engine




April 11, 2024 14:12 UTC
[Resolved] This incident has been resolved and jobs are being picked up without any errors. More information can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/17819.

April 11, 2024 13:31 UTC
[Monitoring] The rollback has completed and we are seeing the jobs being picked up as normal. We're continuing to monitor to ensure the problem does not recur. Details in gitlab.com/gitlab-com/gl-infra/production/-/issues/17819

April 11, 2024 13:21 UTC
[Identified] We have started to roll back the relevant changes and anticipate that this should complete within 45 minutes. Details in - gitlab.com/gitlab-com/gl-infra/production/-/issues/17819

April 11, 2024 13:00 UTC
[Identified] We have started to roll back the relevant changes. Details in - gitlab.com/gitlab-com/gl-infra/production/-/issues/17819

April 11, 2024 12:30 UTC
[Identified] We have identified the cause of this problem and are starting to roll back the relevant changes. Details in - gitlab.com/gitlab-com/gl-infra/production/-/issues/17819

April 11, 2024 12:18 UTC
[Investigating] No material updates to report. We're continuing to investigate the issue. Tracking in gitlab.com/gitlab-com/gl-infra/production/-/issues/17819

April 11, 2024 12:03 UTC
[Investigating] We're currently investigating issues with jobs not being picked up by runners. More details in - gitlab.com/gitlab-com/gl-infra/production/-/issues/17819

March 2024

Elevated queue size on Shared Runners

March 15, 2024 18:40 UTC

Incident Status

Degraded Performance


Components

CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS


Locations

Google Compute Engine, AWS




March 15, 2024 18:40 UTC
[Resolved] Job pickup delays and slowness on running jobs have not been observed for some time, and the team does not anticipate the problem returning. Additional information will be shared in gitlab.com/gitlab-com/gl-infra/production/-/issues/17724

March 15, 2024 14:35 UTC
[Monitoring] As things remain stable, we will continue to monitor the situation and will provide further updates as necessary. Full details are available in gitlab.com/gitlab-com/gl-infra/production/-/issues/17724

March 15, 2024 13:34 UTC
[Monitoring] There are no material updates to report. We will continue to monitor and post updates hourly. Full details are available in gitlab.com/gitlab-com/gl-infra/production/-/issues/17724

March 15, 2024 12:33 UTC
[Monitoring] Things remain stable, with no material updates at the time. We are continuing to monitor the situation, and will provide hourly updates. Full details are available in gitlab.com/gitlab-com/gl-infra/production/-/issues/17724

March 15, 2024 11:30 UTC
[Monitoring] No material updates at the time. We are continuing to monitor the situation, and will provide hourly updates. Full details are available in gitlab.com/gitlab-com/gl-infra/production/-/issues/17724

March 15, 2024 10:29 UTC
[Monitoring] The impact from this incident appears to now be mitigated, however we will continue to monitor the situation and will provide further updates as necessary. Full details are available in gitlab.com/gitlab-com/gl-infra/production/-/issues/17724

March 15, 2024 09:26 UTC
[Monitoring] The impact from this incident appears low, however we are continuing to monitor the situation, and will post another update in one hour. Full details are available in gitlab.com/gitlab-com/gl-infra/production/-/issues/17724

March 15, 2024 08:15 UTC
[Monitoring] We are continuing to monitor this situation, with no additional details to add at this time. Full details are available in gitlab.com/gitlab-com/gl-infra/production/-/issues/17724

March 15, 2024 07:11 UTC
[Identified] No material updates to report. We will continue to monitor to the situation for now. Details in gitlab.com/gitlab-com/gl-infra/production/-/issues/17724

March 15, 2024 05:18 UTC
[Identified] We have narrowed down the potential root cause but we will continue the investigation. No further impact on CI/CD pipeline execution. Next update will be provided in 1 hour. Details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/17724

March 15, 2024 03:35 UTC
[Investigating] We are continuing to monitor the extended runner queue size. Impact on CI/CD pipeline execution does not appear to be significant at this time. A further update will be provided in 1 hour. Details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/17724

March 15, 2024 02:31 UTC
[Investigating] Analysis continues. A further update will be provided in 1 hour, or sooner should there be a significant update to report. Details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/17724

March 15, 2024 01:57 UTC
[Investigating] The investigation is ongoing, we continue to assess the root cause of the extended queue size with the help of additional resources. A further update will be provided in 30 minutes. Details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/17724

March 15, 2024 01:34 UTC
[Investigating] No material updates at this time. Investigation into the increasing runner queue size is ongoing. Details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/17724

March 15, 2024 01:13 UTC
[Investigating] We are continuing to investigate the root cause of the elevated queue size. For more information: gitlab.com/gitlab-com/gl-infra/production/-/issues/17724

March 15, 2024 00:55 UTC
[Investigating] No material update at this time. Our team is still investigating the root cause.

March 15, 2024 00:42 UTC
[Investigating] We are investigating an issue with elevated queue size on our GitLab.com runners. Users may experience slowness with runners picking up jobs, or running jobs. For more information: gitlab.com/gitlab-com/gl-infra/production/-/issues/17724

February 2024

Intermittent job failures with GitLab.com SaaS runners

February 21, 2024 17:54 UTC

Incident Status

Partial Service Disruption


Components

CI/CD - Hosted runners on Linux


Locations

Google Compute Engine




February 21, 2024 17:54 UTC
[Resolved] The issue has been resolved and the affected services are now operating normally. Further details are available in - gitlab.com/gitlab-com/gl-infra/production/-/issues/17636

February 21, 2024 17:03 UTC
[Monitoring] We have taken steps to mitigate the problem and are now monitoring. The next update will be in 60 minutes unless there is anything to report sooner. Tracking in - gitlab.com/gitlab-com/gl-infra/production/-/issues/17636

February 21, 2024 16:32 UTC
[Investigating] We're currently investigating a known problem with SaaS runners which may cause CI jobs to fail intermittently. Next update will be in 30 minutes unless there is anything to report sooner. Tracking in - gitlab.com/gitlab-com/gl-infra/production/-/issues/17636

Intermittent job failures with GitLab.com SaaS runners

February 16, 2024 03:09 UTC

Incident Status

Partial Service Disruption


Components

CI/CD - Hosted runners on Linux


Locations

Google Compute Engine




February 16, 2024 03:09 UTC
[Resolved] The issue has been resolved and affected services are now operating normally. Further details are available in - gitlab.com/gitlab-com/gl-infra/production/-/issues/17603

February 16, 2024 02:47 UTC
[Monitoring] The cause of the problem has been identified and subsequent action has been taken to resolve it. We're seeing a return to normal operation and are continuing to monitor before declaring the incident as resolved. Tracking in - gitlab.com/gitlab-com/gl-infra/production/-/issues/17603

February 16, 2024 02:37 UTC
[Investigating] We're currently investigating a known SSL certificate problem with SaaS runners which may cause CI jobs to fail intermittently. Tracking in - gitlab.com/gitlab-com/gl-infra/production/-/issues/17603

January 2024

Sidekiq degredation

January 30, 2024 23:02 UTC

Sidekiq degredationDegraded Performance

Incident Status

Degraded Performance


Components

Website, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions, Background Processing


Locations

Google Compute Engine




January 30, 2024 23:02 UTC
[Resolved] The feature flag responsible for the issue has been disabled by default and operations are back to normal. This incident is considered resolved. Further details can be found in the incident issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/17504

January 30, 2024 22:02 UTC
[Identified] We've identified the issue and have narrowed the problem to a recent feature flag that created slow queries caused by specific import workers. These slow queries lead to database saturation. To mitigate the issue, the team has disabled the feature flag. We are now re-processing jobs from these workers. This issue is considered mitigated. Further updates will be minimal until there is a permanent code fix in place. More details in gitlab.com/gitlab-com/gl-infra/production/-/issues/17504

January 30, 2024 20:22 UTC
[Investigating] No material update at this time. Our team is still investigating the root cause.

January 30, 2024 19:54 UTC
[Investigating] We've temporarily disabled some features of the GitHub import functionality. Users might experience issues with GitHub imports at this time while our team continues to investigate.

January 30, 2024 19:28 UTC
[Investigating] Sidekiq has largely recovered and jobs are processing. Our team is still investigating the root cause.

January 30, 2024 19:08 UTC
[Investigating] Our team is still investigating the root cause of this issue. We'll continue to provide updates as more information is available.

January 30, 2024 18:51 UTC
[Investigating] We're currently seeing some sidekiq degradation that appears to be impacting some pipelines and merge requests. Our team is investigating. More information in gitlab.com/gitlab-com/gl-infra/production/-/issues/17504

DIND not working in Shared Runners after update

January 18, 2024 15:03 UTC

Incident Status

Partial Service Disruption


Components

CI/CD - Hosted runners on Linux


Locations

Google Compute Engine




January 18, 2024 15:03 UTC
[Resolved] The rollback was successful in restoring service. Runner service has been restored. More information: gitlab.com/gitlab-com/gl-infra/production/-/issues/17422

January 18, 2024 14:53 UTC
[Monitoring] The shared runner update has been successfully rolled back. Positive reports have been received from some users. We are still monitoring to confirm if errors have subsided. More information: gitlab.com/gitlab-com/gl-infra/production/-/issues/17422

January 18, 2024 14:39 UTC
[Identified] Shared runner rollback is still in progress. For More information: gitlab.com/gitlab-com/gl-infra/production/-/issues/17422

January 18, 2024 14:15 UTC
[Identified] We’re aware of an issue with Shared Runners and a rollback is in progress. For more information: gitlab.com/gitlab-com/gl-infra/production/-/issues/17422

December 2023

SaaS runners erroring out when queuing jobs

December 15, 2023 21:00 UTC

Incident Status

Degraded Performance


Components

CI/CD - Hosted runners on Linux


Locations

Google Compute Engine




December 15, 2023 21:00 UTC
[Resolved] Docker has been updated and functionality has been restored. Customers are now able to use lastest and 24.0.7 tags for Docker. Marking the issue as resolved for more information see issue gitlab.com/gitlab-com/gl-infra/production/-/issues/17283

December 15, 2023 08:45 UTC
[Identified] We have identified the cause of the issue introduced with the latest Docker image (24.0.7 tag) and are looking into ways to address it. We will post the next update when we have identified a solution to this change gitlab.com/gitlab-com/gl-infra/production/-/issues/17283

December 15, 2023 08:07 UTC
[Identified] We are working on compatibility issues with the latest Docker image (24.0.7 tag) for the shared runner component. Affected users, kindly use the workaround mentioned in the previous update. For more info: gitlab.com/gitlab-com/gl-infra/production/-/issues/17283.

December 15, 2023 07:26 UTC
[Identified] We've identified the cause of the issue being the latest/24.0.7 tag of Docker image used. The workaround is to pin a tag that is 24.0.6 and before. For more info: gitlab.com/gitlab-com/gl-infra/production/-/issues/17283.

December 15, 2023 07:03 UTC
[Investigating] We are primarily seeing SaaS Linux small runners affected. Our investigation is ongoing. More information on gitlab.com/gitlab-com/gl-infra/production/-/issues/17283.

December 15, 2023 06:43 UTC
[Investigating] We are investigating an issue with elevated errors on our SaaS runners being able to queue jobs. For more information: gitlab.com/gitlab-com/gl-infra/production/-/issues/17283

November 2023

Performance issues affecting processing of MRs

November 20, 2023 18:49 UTC

Incident Status

Degraded Performance


Components

CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions, Background Processing


Locations

Google Compute Engine, AWS




November 20, 2023 18:49 UTC
[Resolved] Performance of CI pipelines and merge requests has fully returned to normal and shows no further signs of degradation. The status page is being marked resolved, and future updates can be found in the production issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/17158

November 20, 2023 18:05 UTC
[Monitoring] Performance of CI job pickup and merge requests has returned to a normal state. The incident response team will continue monitoring and reviewing the rolled back changes. Next update in an hour, unless there is anything to report prior.

November 20, 2023 17:21 UTC
[Investigating] We have started to notice some signs of recovery after the rollback. Investigation into root causes is still ongoing to identify the source of the CI and merge request performance problems.

November 20, 2023 16:50 UTC
[Investigating] Rollback to an earlier deployment has completed. No material updates in the CI job pickup slowness has been observed yet. Next update will be in 30 minutes unless there is anything to report sooner.

November 20, 2023 16:32 UTC
[Investigating] The rollback to an earlier deployment is in progress and a review of all changes is in progress to identify the root cause of slow CI job pickup. Additional support from other teams is being brought in to assist.

November 20, 2023 16:12 UTC
[Investigating] The incident response team is continuing to investigate the slow CI job pickup. A rollback to a previous deployment is in progress as a potential mitigating solution.

November 20, 2023 15:53 UTC
[Investigating] We are actively investigating an issue where CI jobs are being picked up slowly affecting performance of MR's. More details here: gitlab.com/gitlab-com/gl-infra/production/-/issues/17158

Configured Services in GitLab CI Not Receiving Variables from .gitlab-ci.yml

November 16, 2023 07:49 UTC

Incident Status

Partial Service Disruption


Components

CI/CD - Hosted runners on Linux, CI/CD - Hosted runners for GitLab community contributions


Locations

Google Compute Engine




November 16, 2023 07:49 UTC
[Resolved] The rollback has been successfully completed, and variables are now being correctly passed to the services specified in the CI as expected. This incident has been resolved. For more details, please visit: gitlab.com/gitlab-com/gl-infra/production/-/issues/17143

November 16, 2023 07:03 UTC
[Identified] Our team is currently working on a rollback of the recent changes. For more details, please visit: gitlab.com/gitlab-com/gl-infra/production/-/issues/17143

November 16, 2023 06:39 UTC
[Identified] The revert merge request (gitlab.com/gitlab-org/gitlab/-/merge_requests/137084) has been successfully merged. Our team is now working on deploying these changes to production. For more details, please see: gitlab.com/gitlab-com/gl-infra/production/-/issues/17143

November 16, 2023 06:15 UTC
[Identified] Our team has identified the probable root cause of the issue and is currently working on a revert merge request. In the meantime, a workaround is to specify the variables in the global section. More details here: gitlab.com/gitlab-com/gl-infra/production/-/issues/17143

November 16, 2023 05:57 UTC
[Investigating] We are actively investigating an issue where services specified in .gitlab-ci.yml are not picking up environment variables as expected. More details here: gitlab.com/gitlab-com/gl-infra/production/-/issues/17143

September 2023

Main DB Cluster PostgreSQL 14 Upgrade

September 9, 2023 17:25 UTC

Description

We will be undergoing scheduled maintenance to our database layer. The maintenance will take up to 6 hours, from 14:00 UTC to 20:00 UTC. GitLab.com will continue to be available during the maintenance window. We apologize in advance for any inconvenience this may cause. See gitlab.com/gitlab-com/gl-infra/production/-/issues/16266 for details.


Components

Website, API, Git Operations, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners for GitLab community contributions


Locations

Google Compute Engine


Schedule

September 9, 2023 14:00 - September 9, 2023 20:00 UTC



September 9, 2023 17:25 UTC
[Update] GitLab.com planned maintenance for the database layer is complete. We'll be monitoring the platform to ensure all systems are functioning correctly. Thank you for your patience.

September 9, 2023 16:12 UTC
[Update] GitLab.com planned maintenance for the database layer is almost complete. We’re continuing to verify that all systems are functioning correctly. Thank you for your patience. See <gitlab.com/gitlab-com/gl-infra/production/-/issues/16266>

September 9, 2023 13:01 UTC
[Update] We will be undergoing scheduled maintenance to our main database layer in 1 hour. The maintenance will take up to 5 hours starting from 14:00 UTC to 19:00 UTC. GitLab.com will be available but users may experience degraded performance during the maintenance window. We apologize in advance for any inconvenience this may cause. See gitlab.com/gitlab-com/gl-infra/production/-/issues/16266

September 9, 2023 10:08 UTC
[Update] We will be undergoing scheduled maintenance to our main database layer in 3 hours. The maintenance will take up to 5 hours starting from 14:00 UTC to 19:00 UTC. GitLab.com will be available but users may experience degraded performance during the maintenance window. We apologize in advance for any inconvenience this may cause. See gitlab.com/gitlab-com/gl-infra/production/-/issues/16266

September 8, 2023 13:59 UTC
[Update] Reminder: Tomorrow, we will be undergoing scheduled maintenance to our main database layer. The maintenance will take up to 5 hours starting from 14:00 UTC to 19:00 UTC. GitLab.com will be available but users may experience degraded performance during the maintenance window. We apologize in advance for any inconvenience this may cause. See gitlab.com/gitlab-com/gl-infra/production/-/issues/16266

September 6, 2023 12:13 UTC
[Update] In 3 days, we will be undergoing scheduled maintenance to our main database layer. The maintenance will take up to 5 hours starting from 14:00 UTC to 19:00 UTC. GitLab.com will be available but users may experience degraded performance during the maintenance window. We apologize in advance for any inconvenience this may cause. See #16266

August 31, 2023 00:01 UTC
[Update] Next week, we will be undergoing scheduled maintenance to our main database layer. The maintenance will take up to 5 hours starting from 14:00 UTC to 19:00 UTC. GitLab.com will be available but users may experience degraded performance during the maintenance window. We apologize in advance for any inconvenience this may cause. See <gitlab.com/gitlab-com/gl-infra/production/-/issues/14403>

August 2023

Main DB Cluster PostgreSQL 14 Upgrade

August 26, 2023 19:51 UTC

Description

We will be undergoing scheduled maintenance to our database layer. The maintenance will take up to 5 hours, from 14:00 UTC to 19:00 UTC. GitLab.com will continue to be available during the maintenance window. We apologize in advance for any inconvenience this may cause. See gitlab.com/gitlab-com/gl-infra/production/-/issues/14403 for details.


Components

Website, API, Git Operations, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners for GitLab community contributions


Locations

Google Compute Engine


Schedule

August 26, 2023 14:00 - August 26, 2023 19:00 UTC



August 26, 2023 19:51 UTC
[Update] GitLab.com rollback for the database layer is complete, and we're back up and running. We'll be monitoring the platform to ensure all systems are functioning correctly. Thank you for your patience.

August 26, 2023 19:21 UTC
[Update] GitLab.com rollback for the database layer is almost complete. The site is back up and running although we're continuing to verify that all systems are functioning correctly. Thank you for your patience.

August 26, 2023 18:29 UTC
[Update] Due to an issue during the planned maintenance for the database layer, we have initiated a rollback of the changes. We will send another update within the next 30 minutes.

August 26, 2023 14:11 UTC
[Update] We will be starting scheduled maintenance to our main database layer now. The maintenance will take up to 5 hours. GitLab.com will be available but users may experience degraded performance during the maintenance window. We apologize in advance for any inconvenience this may cause. See gitlab.com/gitlab-com/gl-infra/production/-/issues/14403

August 26, 2023 13:01 UTC
[Update] We will be undergoing scheduled maintenance to our main database layer in 1 hour. The maintenance will take up to 5 hours starting from 14:00 UTC to 19:00 UTC. GitLab.com will be available but users may experience degraded performance during the maintenance window. We apologize in advance for any inconvenience this may cause. See gitlab.com/gitlab-com/gl-infra/production/-/issues/14403

August 26, 2023 11:06 UTC
[Update] We will be undergoing scheduled maintenance to our main database layer in 3 hours. The maintenance will take up to 5 hours starting from 14:00 UTC to 19:00 UTC. GitLab.com will be available but users may experience degraded performance during the maintenance window. We apologize in advance for any inconvenience this may cause. See gitlab.com/gitlab-com/gl-infra/production/-/issues/14403

August 25, 2023 15:41 UTC
[Update] Reminder: Tomorrow, we will be undergoing scheduled maintenance to our main database layer. The maintenance will take up to 5 hours starting from 14:00 UTC to 19:00 UTC. GitLab.com will be available but users may experience degraded performance during the maintenance window. We apologize in advance for any inconvenience this may cause. See gitlab.com/gitlab-com/gl-infra/production/-/issues/14403

August 23, 2023 12:09 UTC
[Update] In 3 days, we will be undergoing scheduled maintenance to our main database layer. The maintenance will take up to 5 hours starting from 14:00 UTC to 19:00 UTC. GitLab.com will be available but users may experience degraded performance during the maintenance window. We apologize in advance for any inconvenience this may cause. See gitlab.com/gitlab-com/gl-infra/production/-/issues/14403

August 20, 2023 21:13 UTC
[Update] Next week, we will be undergoing scheduled maintenance to our main database layer. The maintenance will take up to 5 hours starting from 14:00 UTC to 19:00 UTC. GitLab.com will be available but users may experience degraded performance during the maintenance window. We apologize in advance for any inconvenience this may cause. See <gitlab.com/gitlab-com/gl-infra/production/-/issues/14403>

Delays in Background Processes

August 23, 2023 18:52 UTC

Delays in Background ProcessesDegraded Performance

Incident Status

Degraded Performance


Components

CI/CD - Hosted runners on Linux


Locations

Google Compute Engine




August 23, 2023 18:52 UTC
[Resolved] Job processing has returned to normal and this incident has now been resolved. We'll continue to work on corrective actions to prevent future occurrences. Details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/16226

August 23, 2023 18:27 UTC
[Identified] The job queue continues to recover. We are monitoring the queue and will provide updates until the issue has been fully mitigated. Details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/16226

August 23, 2023 18:05 UTC
[Identified] The backlog of pending jobs continues to improve. Details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/16226

August 23, 2023 17:43 UTC
[Identified] We continue to see improvements in job processing and are monitoring the jobs queue. Details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/16226

August 23, 2023 17:21 UTC
[Identified] Thank you again for your patience as we continue our mitigation efforts. We are beginning to see some improvements to job processing but will continue to monitor job queues. Details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/16226

August 23, 2023 17:01 UTC
[Identified] No further updates at this time. We are continuing our mitigation efforts and will provide additional updates when available. Details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/16226

August 23, 2023 16:42 UTC
[Identified] We continue to experience delays in job processing. Thank you for your patience as we work on mitigating this issue. Details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/16226

August 23, 2023 16:26 UTC
[Identified] We have identified the problem and continue to work towards mitigating the issue. More details in gitlab.com/gitlab-com/gl-infra/production/-/issues/16226

August 23, 2023 16:08 UTC
[Investigating] No material updates to report. We are still working on mitigating the issue. More details can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/16226

August 23, 2023 15:50 UTC
[Investigating] We are still working on mitigating the issue. Users might also experience errors in the UI. More details can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/16226

August 23, 2023 15:27 UTC
[Investigating] We are currently working on mitigating the issue. More details can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/16226

August 23, 2023 15:13 UTC
[Investigating] We continue seeing delays in jobs getting picked by the runners, and are investigating options to perform a rollback of recent changes. More details in gitlab.com/gitlab-com/gl-infra/production/-/issues/16226

August 23, 2023 14:58 UTC
[Investigating] Our team is still actively investigating the problem. More details in gitlab.com/gitlab-com/gl-infra/production/-/issues/16226

August 23, 2023 14:34 UTC
[Investigating] We're observing an increase in the pending jobs queue for the SaaS runners `small` shard, users might experience a delay in jobs getting picked by the runners. More details in gitlab.com/gitlab-com/gl-infra/production/-/issues/16226

CI Cluster PostgreSQL 14 Upgrade

August 12, 2023 16:51 UTC

Description

We will be undergoing scheduled maintenance to our CI database layer. The maintenance will take up to 5 hours, from 14:00 UTC to 19:00 UTC. GitLab.com will continue to be available during the maintenance window. We apologize in advance for any inconvenience this may cause. See gitlab.com/gitlab-com/gl-infra/production/-/issues/16093 for details.


Components

API, Git Operations, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners for GitLab community contributions


Locations

Google Compute Engine


Schedule

August 12, 2023 14:00 - August 12, 2023 19:00 UTC



August 12, 2023 16:51 UTC
[Update] GitLab.com planned maintenance for the database layer is complete. We'll be monitoring the platform to ensure all systems are functioning correctly. Thank you for your patience.

August 12, 2023 16:32 UTC
[Update] GitLab.com planned maintenance for the database layer is complete. We'll be monitoring the platform to ensure all systems are functioning correctly. Thank you for your patience.

August 12, 2023 16:28 UTC
[Update] The scheduled maintenance to our ci database layer is ongoing. The maintenance will take up to 5 hours starting from 14:00 UTC to 19:00 UTC. GitLab.com will be available but users may experience degraded performance during the maintenance window. We apologize in advance for any inconvenience this may cause. See <gitlab.com/gitlab-com/gl-infra/production/-/issues/15945>

August 12, 2023 15:48 UTC
[Update] GitLab.com planned maintenance for the database layer is almost complete. We're continuing to verify that all systems are functioning correctly. Thank you for your patience.

August 12, 2023 13:00 UTC
[Update] We will be undergoing scheduled maintenance to our ci database layer in 1 hour. The maintenance will take up to 5 hours starting from 14:00 UTC to 19:00 UTC. GitLab.com will be available but users may experience degraded performance during the maintenance window. We apologize in advance for any inconvenience this may cause. See <gitlab.com/gitlab-com/gl-infra/production/-/issues/15945>

August 12, 2023 11:00 UTC
[Update] We will be undergoing scheduled maintenance to our ci database layer in 3 hours. The maintenance will take up to 5 hours starting from 14:00 UTC to 19:00 UTC. GitLab.com will be available but users may experience degraded performance during the maintenance window. We apologize in advance for any inconvenience this may cause. See <gitlab.com/gitlab-com/gl-infra/production/-/issues/15945>

August 12, 2023 04:29 UTC
[Update] Reminder: Today, we will be undergoing scheduled maintenance to our CI database layer. The maintenance will take up to 5 hours starting from 14:00 UTC to 19:00 UTC. GitLab.com will be available but users may experience degraded performance during the maintenance window. We apologize in advance for any inconvenience this may cause. See gitlab.com/gitlab-com/gl-infra/production/-/issues/16093

August 9, 2023 15:42 UTC
[Update] In 3 days, we will be undergoing scheduled maintenance to our ci database layer. The maintenance will take up to 5 hours starting from 14:00 UTC to 19:00 UTC. GitLab.com will be available but users may experience degraded performance during the maintenance window. We apologize in advance for any inconvenience this may cause. See <gitlab.com/gitlab-com/gl-infra/production/-/issues/15945>

July 2023

Performance issues on GitLab.com

July 15, 2023 04:51 UTC

Incident Status

Degraded Performance


Components

API, Git Operations, Container Registry, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners for GitLab community contributions, Background Processing, Canary


Locations

Google Compute Engine




July 15, 2023 04:51 UTC
[Resolved] This incident has been marked as resolved, as we don't see any outstanding issues with GitLab.com. A full timeline of this incident can be found in our Incident Report available at gitlab.com/gitlab-com/gl-infra/production/-/issues/16042

July 15, 2023 03:29 UTC
[Monitoring] We will update the status of this incident to `Operational` for now..

July 15, 2023 03:26 UTC
[Monitoring] The rollback has been completed. We are observing that services have recovered. We will continue to monitor for any issues. More details in gitlab.com/gitlab-com/gl-infra/production/-/issues/16042

July 15, 2023 03:02 UTC
[Identified] We are working on performing a rollback on GitLab.com. Next expected update will be in 30 minutes. More details in gitlab.com/gitlab-com/gl-infra/production/-/issues/16042

July 15, 2023 02:52 UTC
[Identified] We are continuing to observe performance issues on GitLab.com. We are investigating options to perform a rollback. More details in gitlab.com/gitlab-com/gl-infra/production/-/issues/16042

July 15, 2023 02:18 UTC
[Identified] No material updates to report at this time. Infrastructure teams are continuing investigations. Details in gitlab.com/gitlab-com/gl-infra/production/-/issues/16042

July 15, 2023 01:45 UTC
[Identified] We have identified the problem and mitigated the worst of it. Our infrastructure team is continuing investigations. Details in gitlab.com/gitlab-com/gl-infra/production/-/issues/16042.

July 15, 2023 01:13 UTC
[Investigating] We have detected a problem related to GitLab.com services and are actively investigating. More details to follow.





Back to current status