All Systems Operational

Updated a few seconds ago

Back to current status

Status History

Filter: CI/CD - Hosted runners for GitLab community contributions (Clear)



July 2024

GitLab.com is unavailable

July 11, 2024 18:28 UTC

GitLab.com is unavailableService Disruption

Incident Status

Service Disruption


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions, SAML SSO - GitLab SaaS, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, docs.gitlab.com, Canary, Product Analytics - Configurator (Beta), Product Analytics - Collector (Beta), Product Analytics - Background processing (Beta), Product Analytics - Queries (Beta), GitLab Duo


Locations

Google Compute Engine, Digital Ocean, Zendesk, AWS




July 11, 2024 18:28 UTC
[Resolved] This incident is now resolved and no further errors are being seen on GitLab.com. Our investigation points to a disk performance issue. You can see further details and corrective actions in gitlab.com/gitlab-com/gl-infra/production/-/issues/18269.

July 11, 2024 15:51 UTC
[Investigating] Our investigations are ongoing. We will post our next update when we have more information. Please follow the incident issue for updates here: gitlab.com/gitlab-com/gl-infra/production/-/issues/18269

July 11, 2024 15:28 UTC
[Investigating] No material updates to report. We're continuing to investigate the issue. Please follow the incident issue for updates here: gitlab.com/gitlab-com/gl-infra/production/-/issues/18269

July 11, 2024 15:00 UTC
[Investigating] We are seeing improved service availability and investigations continue for this problem. Please follow the incident issue for updates here: gitlab.com/gitlab-com/gl-infra/production/-/issues/18269

July 11, 2024 14:43 UTC
[Investigating] We are seeing improved service availability and investigations continue for this problem. We are planning to make the incident issue publicly available as soon as possible.

July 11, 2024 14:29 UTC
[Investigating] We are investigating a problem with gitlab.com being unavailable for some users. Our investigations continue and we will post further status updates shortly.

July 11, 2024 14:18 UTC
[Investigating] We are investigating a problem with gitlab.com being unavailable for some users. We will post further status updates shortly.

April 2024

Downstream Pipelines Returning Errors

April 23, 2024 15:19 UTC

Incident Status

Degraded Performance


Components

CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions


Locations

Google Compute Engine




April 23, 2024 15:19 UTC
[Resolved] This incident has been resolved and pipelines are running without errors now. More details are available in gitlab.com/gitlab-com/gl-infra/production/-/issues/17885.

April 22, 2024 21:56 UTC
[Identified] A fix for the issue has been merged and will be available in the next deployment tomorrow morning. Full details here gitlab.com/gitlab-com/gl-infra/production/-/issues/17885.

April 22, 2024 20:43 UTC
[Identified] We've identified the cause of the issue and preparations for a fix are ongoing. Details in gitlab.com/gitlab-com/gl-infra/production/-/issues/17885.

April 22, 2024 18:51 UTC
[Identified] We've identified the cause of the issue and are preparing options to deploy a fix. Details in gitlab.com/gitlab-com/gl-infra/production/-/issues/17885.

April 22, 2024 18:24 UTC
[Identified] We confirmed that the issue is isolated to a small number of namespaces and are continuing to investigate. Details in gitlab.com/gitlab-com/gl-infra/production/-/issues/17885.

April 22, 2024 17:56 UTC
[Investigating] We're investigating reports of parent-child pipeline setups returning an error and failing to create a downstream pipeline. Details in gitlab.com/gitlab-com/gl-infra/production/-/issues/17885.

Runners are not picking up jobs causing 500 errors

April 11, 2024 14:12 UTC

Incident Status

Partial Service Disruption


Components

CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions


Locations

Google Compute Engine




April 11, 2024 14:12 UTC
[Resolved] This incident has been resolved and jobs are being picked up without any errors. More information can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/17819.

April 11, 2024 13:31 UTC
[Monitoring] The rollback has completed and we are seeing the jobs being picked up as normal. We're continuing to monitor to ensure the problem does not recur. Details in gitlab.com/gitlab-com/gl-infra/production/-/issues/17819

April 11, 2024 13:21 UTC
[Identified] We have started to roll back the relevant changes and anticipate that this should complete within 45 minutes. Details in - gitlab.com/gitlab-com/gl-infra/production/-/issues/17819

April 11, 2024 13:00 UTC
[Identified] We have started to roll back the relevant changes. Details in - gitlab.com/gitlab-com/gl-infra/production/-/issues/17819

April 11, 2024 12:30 UTC
[Identified] We have identified the cause of this problem and are starting to roll back the relevant changes. Details in - gitlab.com/gitlab-com/gl-infra/production/-/issues/17819

April 11, 2024 12:18 UTC
[Investigating] No material updates to report. We're continuing to investigate the issue. Tracking in gitlab.com/gitlab-com/gl-infra/production/-/issues/17819

April 11, 2024 12:03 UTC
[Investigating] We're currently investigating issues with jobs not being picked up by runners. More details in - gitlab.com/gitlab-com/gl-infra/production/-/issues/17819

January 2024

Sidekiq degredation

January 30, 2024 23:02 UTC

Sidekiq degredationDegraded Performance

Incident Status

Degraded Performance


Components

Website, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions, Background Processing


Locations

Google Compute Engine




January 30, 2024 23:02 UTC
[Resolved] The feature flag responsible for the issue has been disabled by default and operations are back to normal. This incident is considered resolved. Further details can be found in the incident issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/17504

January 30, 2024 22:02 UTC
[Identified] We've identified the issue and have narrowed the problem to a recent feature flag that created slow queries caused by specific import workers. These slow queries lead to database saturation. To mitigate the issue, the team has disabled the feature flag. We are now re-processing jobs from these workers. This issue is considered mitigated. Further updates will be minimal until there is a permanent code fix in place. More details in gitlab.com/gitlab-com/gl-infra/production/-/issues/17504

January 30, 2024 20:22 UTC
[Investigating] No material update at this time. Our team is still investigating the root cause.

January 30, 2024 19:54 UTC
[Investigating] We've temporarily disabled some features of the GitHub import functionality. Users might experience issues with GitHub imports at this time while our team continues to investigate.

January 30, 2024 19:28 UTC
[Investigating] Sidekiq has largely recovered and jobs are processing. Our team is still investigating the root cause.

January 30, 2024 19:08 UTC
[Investigating] Our team is still investigating the root cause of this issue. We'll continue to provide updates as more information is available.

January 30, 2024 18:51 UTC
[Investigating] We're currently seeing some sidekiq degradation that appears to be impacting some pipelines and merge requests. Our team is investigating. More information in gitlab.com/gitlab-com/gl-infra/production/-/issues/17504

December 2023

Self hosted Pipelines Failing with Cert Errors

December 14, 2023 10:22 UTC

Incident Status

Operational


Components

CI/CD - Hosted runners for GitLab community contributions


Locations

Google Compute Engine




December 14, 2023 10:22 UTC
[Resolved] The issue is resolved. For self-hosted runners, the previous workaround may need to be re-applied as we have updated GitLab.com certificate. Please see possible workarounds in gitlab.com/gitlab-com/gl-infra/production/-/issues/17265

December 14, 2023 09:28 UTC
[Monitoring] We are updating the certificate on GitLab.com. If you are experiencing self-hosted runner connection errors, please see possible workarounds in gitlab.com/gitlab-com/gl-infra/production/-/issues/17265

December 14, 2023 08:24 UTC
[Monitoring] We are investigating an issue with connection issues for self-hosted runners and users using the AWS OIDC provider due to an SSL certificate change. If you are experiencing self-hosted runner connection errors, please see possible workarounds in gitlab.com/gitlab-com/gl-infra/production/-/issues/17265

December 14, 2023 05:47 UTC
[Monitoring] We continue to monitor the situation, and are looking into a solution to reduce the need for either of the two workarounds mentioned in our previous updates. For more information: gitlab.com/gitlab-com/gl-infra/production/-/issues/17265.

December 14, 2023 00:52 UTC
[Monitoring] GitLab.com shared runners may be affected by the switch to LetsEncrypt certs if they use container images that do not recognise LetsEncrypt's CA certificates. If you are facing this issue, please update the container image. More details: gitlab.com/gitlab-com/gl-infra/production/-/issues/17265

December 13, 2023 22:43 UTC
[Monitoring] We have a workaround in place and are monitoring the incident. The current workaround is to reboot the runners to restore functionality. We will keep this alert in monitoring for now. This is affecting Saas self-hosted runners. Shared runners are not affected.

December 13, 2023 22:03 UTC
[Monitoring] We are currently monitoring the situation. Self-hosted runners on Saas are experiencing failed pipelines due to certificate errors. Saas runners are not affected. The current workaround is to reboot the runner to refresh SSL and restore service. gitlab.com/gitlab-com/gl-infra/production/-/issues/17265

December 13, 2023 21:44 UTC
[Identified] Further investigation reveals this issue to be affecting specifically self-hosted runners on Saas. Saas runners are not affected. Current workaround is to reboot the runner to refresh the SSL.

December 13, 2023 21:26 UTC
[Investigating] We are still investigating an issue with SSL certificates causing pipelines to fail on self-manged. As a workaround, we can try rebooting the runners to update the SSL on cache. We are still investigating the issue gitlab.com/gitlab-com/gl-infra/production/-/issues/17265

December 13, 2023 20:52 UTC
[Investigating] We have made a change to our Cloudflare certificates and self-hosted runners may have trouble running pipelines on self-managed. Please stand by while we are investigating this issue.

November 2023

Performance issues affecting processing of MRs

November 20, 2023 18:49 UTC

Incident Status

Degraded Performance


Components

CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions, Background Processing


Locations

Google Compute Engine, AWS




November 20, 2023 18:49 UTC
[Resolved] Performance of CI pipelines and merge requests has fully returned to normal and shows no further signs of degradation. The status page is being marked resolved, and future updates can be found in the production issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/17158

November 20, 2023 18:05 UTC
[Monitoring] Performance of CI job pickup and merge requests has returned to a normal state. The incident response team will continue monitoring and reviewing the rolled back changes. Next update in an hour, unless there is anything to report prior.

November 20, 2023 17:21 UTC
[Investigating] We have started to notice some signs of recovery after the rollback. Investigation into root causes is still ongoing to identify the source of the CI and merge request performance problems.

November 20, 2023 16:50 UTC
[Investigating] Rollback to an earlier deployment has completed. No material updates in the CI job pickup slowness has been observed yet. Next update will be in 30 minutes unless there is anything to report sooner.

November 20, 2023 16:32 UTC
[Investigating] The rollback to an earlier deployment is in progress and a review of all changes is in progress to identify the root cause of slow CI job pickup. Additional support from other teams is being brought in to assist.

November 20, 2023 16:12 UTC
[Investigating] The incident response team is continuing to investigate the slow CI job pickup. A rollback to a previous deployment is in progress as a potential mitigating solution.

November 20, 2023 15:53 UTC
[Investigating] We are actively investigating an issue where CI jobs are being picked up slowly affecting performance of MR's. More details here: gitlab.com/gitlab-com/gl-infra/production/-/issues/17158

Configured Services in GitLab CI Not Receiving Variables from .gitlab-ci.yml

November 16, 2023 07:49 UTC

Incident Status

Partial Service Disruption


Components

CI/CD - Hosted runners on Linux, CI/CD - Hosted runners for GitLab community contributions


Locations

Google Compute Engine




November 16, 2023 07:49 UTC
[Resolved] The rollback has been successfully completed, and variables are now being correctly passed to the services specified in the CI as expected. This incident has been resolved. For more details, please visit: gitlab.com/gitlab-com/gl-infra/production/-/issues/17143

November 16, 2023 07:03 UTC
[Identified] Our team is currently working on a rollback of the recent changes. For more details, please visit: gitlab.com/gitlab-com/gl-infra/production/-/issues/17143

November 16, 2023 06:39 UTC
[Identified] The revert merge request (gitlab.com/gitlab-org/gitlab/-/merge_requests/137084) has been successfully merged. Our team is now working on deploying these changes to production. For more details, please see: gitlab.com/gitlab-com/gl-infra/production/-/issues/17143

November 16, 2023 06:15 UTC
[Identified] Our team has identified the probable root cause of the issue and is currently working on a revert merge request. In the meantime, a workaround is to specify the variables in the global section. More details here: gitlab.com/gitlab-com/gl-infra/production/-/issues/17143

November 16, 2023 05:57 UTC
[Investigating] We are actively investigating an issue where services specified in .gitlab-ci.yml are not picking up environment variables as expected. More details here: gitlab.com/gitlab-com/gl-infra/production/-/issues/17143

September 2023

Main DB Cluster PostgreSQL 14 Upgrade

September 9, 2023 17:25 UTC

Description

We will be undergoing scheduled maintenance to our database layer. The maintenance will take up to 6 hours, from 14:00 UTC to 20:00 UTC. GitLab.com will continue to be available during the maintenance window. We apologize in advance for any inconvenience this may cause. See gitlab.com/gitlab-com/gl-infra/production/-/issues/16266 for details.


Components

Website, API, Git Operations, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners for GitLab community contributions


Locations

Google Compute Engine


Schedule

September 9, 2023 14:00 - September 9, 2023 20:00 UTC



September 9, 2023 17:25 UTC
[Update] GitLab.com planned maintenance for the database layer is complete. We'll be monitoring the platform to ensure all systems are functioning correctly. Thank you for your patience.

September 9, 2023 16:12 UTC
[Update] GitLab.com planned maintenance for the database layer is almost complete. We’re continuing to verify that all systems are functioning correctly. Thank you for your patience. See <gitlab.com/gitlab-com/gl-infra/production/-/issues/16266>

September 9, 2023 13:01 UTC
[Update] We will be undergoing scheduled maintenance to our main database layer in 1 hour. The maintenance will take up to 5 hours starting from 14:00 UTC to 19:00 UTC. GitLab.com will be available but users may experience degraded performance during the maintenance window. We apologize in advance for any inconvenience this may cause. See gitlab.com/gitlab-com/gl-infra/production/-/issues/16266

September 9, 2023 10:08 UTC
[Update] We will be undergoing scheduled maintenance to our main database layer in 3 hours. The maintenance will take up to 5 hours starting from 14:00 UTC to 19:00 UTC. GitLab.com will be available but users may experience degraded performance during the maintenance window. We apologize in advance for any inconvenience this may cause. See gitlab.com/gitlab-com/gl-infra/production/-/issues/16266

September 8, 2023 13:59 UTC
[Update] Reminder: Tomorrow, we will be undergoing scheduled maintenance to our main database layer. The maintenance will take up to 5 hours starting from 14:00 UTC to 19:00 UTC. GitLab.com will be available but users may experience degraded performance during the maintenance window. We apologize in advance for any inconvenience this may cause. See gitlab.com/gitlab-com/gl-infra/production/-/issues/16266

September 6, 2023 12:13 UTC
[Update] In 3 days, we will be undergoing scheduled maintenance to our main database layer. The maintenance will take up to 5 hours starting from 14:00 UTC to 19:00 UTC. GitLab.com will be available but users may experience degraded performance during the maintenance window. We apologize in advance for any inconvenience this may cause. See #16266

August 31, 2023 00:01 UTC
[Update] Next week, we will be undergoing scheduled maintenance to our main database layer. The maintenance will take up to 5 hours starting from 14:00 UTC to 19:00 UTC. GitLab.com will be available but users may experience degraded performance during the maintenance window. We apologize in advance for any inconvenience this may cause. See <gitlab.com/gitlab-com/gl-infra/production/-/issues/14403>

August 2023

Main DB Cluster PostgreSQL 14 Upgrade

August 26, 2023 19:51 UTC

Description

We will be undergoing scheduled maintenance to our database layer. The maintenance will take up to 5 hours, from 14:00 UTC to 19:00 UTC. GitLab.com will continue to be available during the maintenance window. We apologize in advance for any inconvenience this may cause. See gitlab.com/gitlab-com/gl-infra/production/-/issues/14403 for details.


Components

Website, API, Git Operations, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners for GitLab community contributions


Locations

Google Compute Engine


Schedule

August 26, 2023 14:00 - August 26, 2023 19:00 UTC



August 26, 2023 19:51 UTC
[Update] GitLab.com rollback for the database layer is complete, and we're back up and running. We'll be monitoring the platform to ensure all systems are functioning correctly. Thank you for your patience.

August 26, 2023 19:21 UTC
[Update] GitLab.com rollback for the database layer is almost complete. The site is back up and running although we're continuing to verify that all systems are functioning correctly. Thank you for your patience.

August 26, 2023 18:29 UTC
[Update] Due to an issue during the planned maintenance for the database layer, we have initiated a rollback of the changes. We will send another update within the next 30 minutes.

August 26, 2023 14:11 UTC
[Update] We will be starting scheduled maintenance to our main database layer now. The maintenance will take up to 5 hours. GitLab.com will be available but users may experience degraded performance during the maintenance window. We apologize in advance for any inconvenience this may cause. See gitlab.com/gitlab-com/gl-infra/production/-/issues/14403

August 26, 2023 13:01 UTC
[Update] We will be undergoing scheduled maintenance to our main database layer in 1 hour. The maintenance will take up to 5 hours starting from 14:00 UTC to 19:00 UTC. GitLab.com will be available but users may experience degraded performance during the maintenance window. We apologize in advance for any inconvenience this may cause. See gitlab.com/gitlab-com/gl-infra/production/-/issues/14403

August 26, 2023 11:06 UTC
[Update] We will be undergoing scheduled maintenance to our main database layer in 3 hours. The maintenance will take up to 5 hours starting from 14:00 UTC to 19:00 UTC. GitLab.com will be available but users may experience degraded performance during the maintenance window. We apologize in advance for any inconvenience this may cause. See gitlab.com/gitlab-com/gl-infra/production/-/issues/14403

August 25, 2023 15:41 UTC
[Update] Reminder: Tomorrow, we will be undergoing scheduled maintenance to our main database layer. The maintenance will take up to 5 hours starting from 14:00 UTC to 19:00 UTC. GitLab.com will be available but users may experience degraded performance during the maintenance window. We apologize in advance for any inconvenience this may cause. See gitlab.com/gitlab-com/gl-infra/production/-/issues/14403

August 23, 2023 12:09 UTC
[Update] In 3 days, we will be undergoing scheduled maintenance to our main database layer. The maintenance will take up to 5 hours starting from 14:00 UTC to 19:00 UTC. GitLab.com will be available but users may experience degraded performance during the maintenance window. We apologize in advance for any inconvenience this may cause. See gitlab.com/gitlab-com/gl-infra/production/-/issues/14403

August 20, 2023 21:13 UTC
[Update] Next week, we will be undergoing scheduled maintenance to our main database layer. The maintenance will take up to 5 hours starting from 14:00 UTC to 19:00 UTC. GitLab.com will be available but users may experience degraded performance during the maintenance window. We apologize in advance for any inconvenience this may cause. See <gitlab.com/gitlab-com/gl-infra/production/-/issues/14403>

CI Cluster PostgreSQL 14 Upgrade

August 12, 2023 16:51 UTC

Description

We will be undergoing scheduled maintenance to our CI database layer. The maintenance will take up to 5 hours, from 14:00 UTC to 19:00 UTC. GitLab.com will continue to be available during the maintenance window. We apologize in advance for any inconvenience this may cause. See gitlab.com/gitlab-com/gl-infra/production/-/issues/16093 for details.


Components

API, Git Operations, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners for GitLab community contributions


Locations

Google Compute Engine


Schedule

August 12, 2023 14:00 - August 12, 2023 19:00 UTC



August 12, 2023 16:51 UTC
[Update] GitLab.com planned maintenance for the database layer is complete. We'll be monitoring the platform to ensure all systems are functioning correctly. Thank you for your patience.

August 12, 2023 16:32 UTC
[Update] GitLab.com planned maintenance for the database layer is complete. We'll be monitoring the platform to ensure all systems are functioning correctly. Thank you for your patience.

August 12, 2023 16:28 UTC
[Update] The scheduled maintenance to our ci database layer is ongoing. The maintenance will take up to 5 hours starting from 14:00 UTC to 19:00 UTC. GitLab.com will be available but users may experience degraded performance during the maintenance window. We apologize in advance for any inconvenience this may cause. See <gitlab.com/gitlab-com/gl-infra/production/-/issues/15945>

August 12, 2023 15:48 UTC
[Update] GitLab.com planned maintenance for the database layer is almost complete. We're continuing to verify that all systems are functioning correctly. Thank you for your patience.

August 12, 2023 13:00 UTC
[Update] We will be undergoing scheduled maintenance to our ci database layer in 1 hour. The maintenance will take up to 5 hours starting from 14:00 UTC to 19:00 UTC. GitLab.com will be available but users may experience degraded performance during the maintenance window. We apologize in advance for any inconvenience this may cause. See <gitlab.com/gitlab-com/gl-infra/production/-/issues/15945>

August 12, 2023 11:00 UTC
[Update] We will be undergoing scheduled maintenance to our ci database layer in 3 hours. The maintenance will take up to 5 hours starting from 14:00 UTC to 19:00 UTC. GitLab.com will be available but users may experience degraded performance during the maintenance window. We apologize in advance for any inconvenience this may cause. See <gitlab.com/gitlab-com/gl-infra/production/-/issues/15945>

August 12, 2023 04:29 UTC
[Update] Reminder: Today, we will be undergoing scheduled maintenance to our CI database layer. The maintenance will take up to 5 hours starting from 14:00 UTC to 19:00 UTC. GitLab.com will be available but users may experience degraded performance during the maintenance window. We apologize in advance for any inconvenience this may cause. See gitlab.com/gitlab-com/gl-infra/production/-/issues/16093

August 9, 2023 15:42 UTC
[Update] In 3 days, we will be undergoing scheduled maintenance to our ci database layer. The maintenance will take up to 5 hours starting from 14:00 UTC to 19:00 UTC. GitLab.com will be available but users may experience degraded performance during the maintenance window. We apologize in advance for any inconvenience this may cause. See <gitlab.com/gitlab-com/gl-infra/production/-/issues/15945>

July 2023

Performance issues on GitLab.com

July 15, 2023 04:51 UTC

Incident Status

Degraded Performance


Components

API, Git Operations, Container Registry, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners for GitLab community contributions, Background Processing, Canary


Locations

Google Compute Engine




July 15, 2023 04:51 UTC
[Resolved] This incident has been marked as resolved, as we don't see any outstanding issues with GitLab.com. A full timeline of this incident can be found in our Incident Report available at gitlab.com/gitlab-com/gl-infra/production/-/issues/16042

July 15, 2023 03:29 UTC
[Monitoring] We will update the status of this incident to `Operational` for now..

July 15, 2023 03:26 UTC
[Monitoring] The rollback has been completed. We are observing that services have recovered. We will continue to monitor for any issues. More details in gitlab.com/gitlab-com/gl-infra/production/-/issues/16042

July 15, 2023 03:02 UTC
[Identified] We are working on performing a rollback on GitLab.com. Next expected update will be in 30 minutes. More details in gitlab.com/gitlab-com/gl-infra/production/-/issues/16042

July 15, 2023 02:52 UTC
[Identified] We are continuing to observe performance issues on GitLab.com. We are investigating options to perform a rollback. More details in gitlab.com/gitlab-com/gl-infra/production/-/issues/16042

July 15, 2023 02:18 UTC
[Identified] No material updates to report at this time. Infrastructure teams are continuing investigations. Details in gitlab.com/gitlab-com/gl-infra/production/-/issues/16042

July 15, 2023 01:45 UTC
[Identified] We have identified the problem and mitigated the worst of it. Our infrastructure team is continuing investigations. Details in gitlab.com/gitlab-com/gl-infra/production/-/issues/16042.

July 15, 2023 01:13 UTC
[Investigating] We have detected a problem related to GitLab.com services and are actively investigating. More details to follow.

CI Cluster PostgreSQL 14 Upgrade

July 8, 2023 04:34 UTC

Description

We will be undergoing scheduled maintenance to our CI database layer. GitLab.com will continue to be available during the maintenance window. We apologize in advance for any inconvenience this may cause. More details: gitlab.com/gitlab-com/gl-infra/production/-/issues/15945


Components

Website, API, Git Operations, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners for GitLab community contributions


Locations

Google Compute Engine


Schedule

July 8, 2023 14:00 - July 8, 2023 19:00 UTC



July 8, 2023 04:34 UTC
[Update] Database maintenance on our CI cluster, originally scheduled for tomorrow, has been postponed. Once a new date has been determined, we will post a new maintenance entry. Details can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/15945.

July 8, 2023 04:33 UTC
[Update] Database maintenance on our CI cluster, originally scheduled for tomorrow, has been postponed. Once a new date has been determined, we will post a new maintenance entry. Details can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/15945.

July 7, 2023 14:54 UTC
[Update] Tomorrow, we will perform a scheduled maintenance to our CI database layer starting from 14:00 UTC to 19:00 UTC, during which users may experience degraded performance. See gitlab.com/gitlab-com/gl-infra/production/-/issues/15945

July 5, 2023 15:31 UTC
[Update] In 3 days we will perform a scheduled maintenance to our CI database layer starting from 14:00 UTC to 19:00 UTC, during which users may experience degraded performance. See gitlab.com/gitlab-com/gl-infra/production/-/issues/15945

June 2023

Main DB Cluster PostgreSQL 14 Upgrade

June 29, 2023 14:38 UTC

Description

We will be undergoing scheduled maintenance to our database layer. GitLab.com will continue to be available during the maintenance window. We apologize in advance for any inconvenience this may cause. More details: gitlab.com/gitlab-com/gl-infra/production/-/issues/14403


Components

Website, API, Git Operations, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners for GitLab community contributions


Locations

Google Compute Engine


Schedule

July 8, 2023 14:00 - July 8, 2023 19:00 UTC



June 29, 2023 14:38 UTC
[Update] We will be rescheduling the Main DB Cluster upgrade, and a new date will be communicated soon. More details: gitlab.com/gitlab-com/gl-infra/production/-/issues/14403

CI Cluster PostgreSQL 14 Upgrade

June 24, 2023 18:39 UTC

Description

We will be undergoing scheduled maintenance to our CI database layer. GitLab.com will continue to be available during the maintenance window. We apologize in advance for any inconvenience this may cause. More details: gitlab.com/gitlab-com/gl-infra/production/-/issues/14402


Components

Website, API, Git Operations, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners for GitLab community contributions


Locations

Google Compute Engine


Schedule

June 24, 2023 14:00 - June 24, 2023 19:00 UTC



June 24, 2023 18:39 UTC
[Update] GitLab.com rollback for the database layer is complete, and we're back up and running. We'll be monitoring the platform to ensure all systems are functioning correctly. Thank you for your patience.

June 24, 2023 17:54 UTC
[Update] Due to an issue during the planned maintenance for the database layer, we have initiated a rollback of the changes. We will send another update within the next 30 minutes.

June 24, 2023 14:10 UTC
[Update] We now are starting the maintenance to our ci database layer. The maintenance will take up to 5 hours starting from 14:00 UTC to 19:00 UTC. GitLab.com will be available but users may experience degraded performance during the maintenance window. We apologize in advance for any inconvenience this may cause. See <gitlab.com/gitlab-com/gl-infra/production/-/issues/14402>

June 24, 2023 13:02 UTC
[Update] We will be undergoing scheduled maintenance to our ci database layer in 1 hour. The maintenance will take up to 5 hours starting from 14:00 UTC to 19:00 UTC. GitLab.com will be available but users may experience degraded performance during the maintenance window. We apologize in advance for any inconvenience this may cause. See <gitlab.com/gitlab-com/gl-infra/production/-/issues/14402>

June 24, 2023 11:12 UTC
[Update] We will perform a scheduled maintenance to our CI database layer in 3 hours, starting from 14:00 UTC to 19:00 UTC, during which users may experience degraded performance. See gitlab.com/gitlab-com/gl-infra/production/-/issues/14402

June 23, 2023 08:12 UTC
[Update] Tomorrow we will perform a scheduled maintenance to our CI database layer starting from 14:00 UTC to 19:00 UTC, during which users may experience degraded performance. See gitlab.com/gitlab-com/gl-infra/production/-/issues/14402

June 21, 2023 14:56 UTC
[Update] In 3 days, we will be undergoing scheduled maintenance to our ci database layer. The maintenance will take up to 5 hours starting from 14:00 UTC to 19:00 UTC. GitLab.com will be available but users may experience degraded performance during the maintenance window. We apologize in advance for any inconvenience this may cause. See gitlab.com/gitlab-com/gl-infra/production/-/issues/14402

June 17, 2023 14:00 UTC
[Update] Next week, we will be undergoing scheduled maintenance to our ci database layer. The maintenance will take up to 5 hours starting from 14:00 UTC to 19:00 UTC. GitLab.com will be available but users may experience degraded performance during the maintenance window. We apologize in advance for any inconvenience this may cause. See gitlab.com/gitlab-com/gl-infra/production/-/issues/14402

June 10, 2023 11:01 UTC
[Update] In 2 weeks, we will be undergoing scheduled maintenance to our CI database layer. GitLab.com will continue to be available during the maintenance window. We apologize in advance for any inconvenience this may cause. More details: gitlab.com/gitlab-com/gl-infra/production/-/issues/14402

Slow requests on GitLab.com

June 7, 2023 13:32 UTC

Slow requests on GitLab.comPartial Service Disruption

Incident Status

Partial Service Disruption


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners for GitLab community contributions, SAML SSO - GitLab SaaS, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, docs.gitlab.com, Canary


Locations

Google Compute Engine, Digital Ocean, Zendesk, AWS




June 7, 2023 13:32 UTC
[Resolved] We have identified the root cause, have seen recovery and are therefore resolving this incident. More information in: gitlab.com/gitlab-com/gl-infra/production/-/issues/14839

June 7, 2023 13:21 UTC
[Investigating] Customers are experiencing slow requests on GitLab.com. More information in: gitlab.com/gitlab-com/gl-infra/production/-/issues/14839

May 2023

Hardware upgrade for the GitLab.com datastore

May 27, 2023 04:06 UTC

Description

In 2 weeks, as part of a planned maintenance window, we will perform a hardware upgrade for the GitLab.com datastore. Users may experience temporary 50X errors for a brief period during this window. See gitlab.com/gitlab-com/gl-infra/production/-/issues/10694


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners for GitLab community contributions, SAML SSO - GitLab SaaS, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, docs.gitlab.com, Canary


Locations

Google Compute Engine, Digital Ocean, Zendesk, AWS


Schedule

May 27, 2023 03:00 - May 27, 2023 04:00 UTC



May 27, 2023 04:06 UTC
[Update] GitLab.com's database layer maintenance is complete now, and we're fully back up and running. We'll be monitoring the platform to ensure all systems are functioning correctly. Thank you for your patience.

May 27, 2023 03:41 UTC
[Update] The database switchover is now complete, we expect all SQL statements to be routed to the new nodes. The site is back up and we're continuing to verify that all systems are functioning correctly. Thank you for your patience.

May 27, 2023 03:00 UTC
[Update] The database maintenance is starting now. The database Primary node switchover should happen at any moment in the next hour and during the switchover users might see 50X errors for a very brief span of time. For details see gitlab.com/gitlab-com/gl-infra/production/-/issues/10694

May 27, 2023 01:51 UTC
[Update] We will be conducting a maintenance database activity in an hour, from 03:00 to 04:00 UTC. Users might see 50X errors for a very brief span of time during this window. For details see gitlab.com/gitlab-com/gl-infra/production/-/issues/10694

May 26, 2023 02:27 UTC
[Update] We will be conducting a maintenance database activity Tomorrow, 2023-05-27, from 03:00 to 04:00 UTC. Users might see 50X errors for a very brief span of time during this window. For details see gitlab.com/gitlab-com/gl-infra/production/-/issues/10694

May 24, 2023 02:14 UTC
[Update] Correction: We will be conducting a maintenance database activity this Saturday, 2023-05-27. from 03:00 to 04:00 UTC. Users might see 50X errors for a very brief span of time during this window. For details see gitlab.com/gitlab-com/gl-infra/production/-/issues/10694

May 24, 2023 02:05 UTC
[Update] We will be conducting a maintenance database activity this Saturday, 2023-05-20, from 03:00 to 04:00 UTC. Users might see 50X errors for a very brief span of time during this window. For details see gitlab.com/gitlab-com/gl-infra/production/-/issues/10694

May 20, 2023 03:58 UTC
[Update] The maintenance is rescheduled to next Saturday, 2023-05-27, from 03:00 to 04:00 UTC. There is currently a long database migration running to fix an incident (gitlab.com/gitlab-com/gl-infra/production/-/issues/14468) that we can't risk to interrupt. We apologise for any inconvenience.

May 20, 2023 03:07 UTC
[Update] The scheduled maintenance will be not be starting at 03:00 UTC, and has been delayed. More details soon to follow.

May 20, 2023 02:00 UTC
[Update] We will be conducting a maintenance database activity in an hour, from 03:00 to 04:00 UTC. Users might see 50X errors for a very brief span of time during this window. For details see gitlab.com/gitlab-com/gl-infra/production/-/issues/10694

May 20, 2023 01:00 UTC
[Update] We will be conducting a maintenance database activity in 2 hours, from 03:00 to 04:00 UTC. Users might see 50X errors for a very brief span of time during this window. For details see gitlab.com/gitlab-com/gl-infra/production/-/issues/10694

May 19, 2023 02:00 UTC
[Update] We will be conducting a maintenance database activity tomorrow, 2023-05-20, from 03:00 to 04:00 UTC. Users might see 50X errors for a very brief span of time during this window. For details see gitlab.com/gitlab-com/gl-infra/production/-/issues/10694

May 17, 2023 02:00 UTC
[Update] We will be conducting a maintenance database activity this Saturday, 2023-05-20, from 03:00 to 04:00 UTC. Users might see 50X errors for a very brief span of time during this window. For details see gitlab.com/gitlab-com/gl-infra/production/-/issues/10694

May 12, 2023 02:00 UTC
[Update] Next week, as part of a planned maintenance window on 2023-05-20 from 03:00 to 04:00 UTC, We will perform a hardware upgrade for the GitLab.com datastore. Users may experience temporary 50X errors for a brief period during this window. See gitlab.com/gitlab-com/gl-infra/production/-/issues/10694

April 2023

Some users seeing 403 errors on pipelines and jobs

April 3, 2023 02:18 UTC

Incident Status

Operational


Components

CI/CD - Hosted runners on Linux, CI/CD - Hosted runners for GitLab community contributions


Locations

Google Compute Engine




April 3, 2023 02:18 UTC
[Resolved] There have been no further issues during the monitoring phase. More details about this incident can be found here: gitlab.com/gitlab-com/gl-infra/production/-/issues/8647

April 3, 2023 01:12 UTC
[Monitoring] Some users were affected by an incident affecting GitLab pipelines and runners. The incident was resolved by 2023-04-03 01:00 UTC. We will continue monitoring. Further details available here: gitlab.com/gitlab-com/gl-infra/production/-/issues/8647

March 2023

License Scanning temporarily disabled

March 31, 2023 21:23 UTC

License Scanning temporarily disabledPartial Service Disruption

Incident Status

Partial Service Disruption


Components

CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners for GitLab community contributions


Locations

Google Compute Engine




March 31, 2023 21:23 UTC
[Resolved] After monitoring, GitLab has seen no further issues with license scanning and we are resolving the incident. Further details can be found here: gitlab.com/gitlab-org/gitlab/-/issues/398679

March 31, 2023 01:48 UTC
[Monitoring] License scanning has now been re-enabled. The optimizations in place have proven to reduce significantly the performance issues on the database. We will continue to monitor. Further details can be found here: gitlab.com/gitlab-org/gitlab/-/issues/398679

March 30, 2023 07:20 UTC
[Identified] We have deployed a merge request to address the issue. We are currently reviewing this and will continue to monitor the situation. You can refer to the incident issue gitlab.com/gitlab-com/gl-infra/production/-/issues/8593 for more details.

March 28, 2023 01:09 UTC
[Identified] Our engineering team has a merge request to fix this issue; it is being reviewed and can be followed at gitlab.com/gitlab-org/gitlab/-/merge_requests/115773. At this time we anticipate the fix should be merged within one to two days. Explicitly including the license scanning template in your CI configuration will allow scans to run correctly. The cu

March 24, 2023 19:41 UTC
[Identified] Our engineering team continues to work to restore full license scanning operation. Explicitly including the license scanning template in your CI configuration will allow scans to run correctly. Next update will be Monday, March 27th unless there is news to share sooner.

March 23, 2023 19:57 UTC
[Identified] A resolution approach has been decided upon, but it will take time to implement. You can include the license scanning template to ensure proper detection while our engineering team continues working. Expect the next update in 24 hours unless there's new information before.

March 23, 2023 18:35 UTC
[Identified] We are evaluating several potential resolutions. Next update in one hour. To ensure your license scan runs correctly, explicitly include the template per the documentation here: docs.gitlab.com/ee/user/compliance/license_compliance/#include-the-license-scanning-template

March 23, 2023 18:01 UTC
[Identified] No licenses will be detected unless the license scanning template is explicitly included in your .gitlab-ci.yml file. Our engineering team is working to restore full functionality. Details in this issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/8593

Performance problems with CI

March 14, 2023 20:03 UTC

Performance problems with CIDegraded Performance

Incident Status

Degraded Performance


Components

Website, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners for GitLab community contributions


Locations

Google Compute Engine




March 14, 2023 20:03 UTC
[Resolved] Our team confirmed that load averages are back to normal and the affected CI components are back to operational status, for more information: gitlab.com/gitlab-com/gl-infra/production/-/issues/8541

March 14, 2023 19:07 UTC
[Monitoring] Our team is still monitoring the load to make sure that the loads get back to normal values. For more information: gitlab.com/gitlab-com/gl-infra/production/-/issues/8541

March 14, 2023 18:29 UTC
[Monitoring] Our team is monitoring the load to make sure that the loads get back to normal values. For more information: gitlab.com/gitlab-com/gl-infra/production/-/issues/8541

March 14, 2023 18:02 UTC
[Monitoring] Disabling the feature flag resulted in a decrease in load in Patroni, our team is monitoring the load to make sure that the loads get back to normal values. For more information: gitlab.com/gitlab-com/gl-infra/production/-/issues/8541

March 14, 2023 17:46 UTC
[Identified] Our team has identified a recently enabled feature that is possibly causing the load issue with Patroni. We have disabled the feature and are currently working on evaluating the performance changes. gitlab.com/gitlab-com/gl-infra/production/-/issues/8541

March 14, 2023 17:25 UTC
[Investigating] Our teams are assessing recent partition tables to the database to determine if a recently implemented change on the database is creating the loads on Patroni, for more information please see gitlab.com/gitlab-com/gl-infra/production/-/issues/8541

March 14, 2023 17:10 UTC
[Investigating] Our teams are still investigating the root cause of the issue, currently, we are analyzing loads in our databases to pinpoint the saturation on Patroni. For more information please see gitlab.com/gitlab-com/gl-infra/production/-/issues/8541

March 14, 2023 16:50 UTC
[Investigating] No material updates at the moment. Our teams are still investigating the root cause of the problem. For more information please see gitlab.com/gitlab-com/gl-infra/production/-/issues/8541

March 14, 2023 16:31 UTC
[Investigating] There is an ongoing incident with our CI Postgres cluster. Some CI jobs are getting delayed. For more information please see the incident (gitlab.com/gitlab-com/gl-infra/production/-/issues/8541)

February 2023

Degraded performance on Gitlab.com

February 27, 2023 21:50 UTC

Incident Status

Operational


Components

Website, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners for GitLab community contributions


Locations

Google Compute Engine




February 27, 2023 21:50 UTC
[Resolved] We have identified the root cause, and are preparing preventative actions. Our services have recovered and we now consider this incident to be resolved. Details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/8471

February 27, 2023 21:36 UTC
[Monitoring] At this time there are no further material updates to provide. We are continuing to monitor the situation. Details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/8471

February 27, 2023 19:59 UTC
[Monitoring] No material updates to provide. We are continuing to monitor. Details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/8471

February 27, 2023 18:52 UTC
[Monitoring] Systems have returned to normal and we are continuing to monitor. We plan to share an update in 1 hour. Details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/8471

February 27, 2023 17:41 UTC
[Monitoring] Our services have recovered. We are continuing to monitor and investigate the root cause. Details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/8471

February 27, 2023 17:18 UTC
[Investigating] Users may encounter degraded performance or intermittent errors on GitLab.com. We are currently investigating. Details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/8471

February 27, 2023 17:16 UTC
[Investigating] Users may encounter degraded performance or intermittent errors on GitLab.com. We are currently investigating. Details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/8471





Back to current status