Active Issue

Updated a few seconds ago

Back to current status

Status History

Filter: Azure (Clear)



February 2022

Customers.gitlab.com maintenance

February 5, 2022 16:10 UTC

Description

The Customers Portal will be in maintenance mode on Saturday February 5th for 2 hours. We will be performing a server migration


Components

GitLab Customers Portal


Locations

Azure


Schedule

February 5, 2022 14:00 - February 5, 2022 16:00 UTC



February 5, 2022 16:10 UTC
[Update] customers.gitlab.com maintenance finished successfully. Thank you for your patience!

February 5, 2022 15:35 UTC
[Update] customers.gitlab.com maintenance is complete now. Site is back up, and we're continuing to verify that all systems are functioning correctly.

February 5, 2022 14:00 UTC
[Update] customers.gitlab.com planned maintenance is starting now. See you on the other side!

January 10, 2022 11:34 UTC
[Update] The maintenance taking place on the 5th of February from 14:00 UTC to 16:00 UTC will only affect the Customers Portal; customers.gitlab.com. All other GitLab services are expected to be functioning as usual.

January 10, 2022 09:07 UTC
[Update] We have decided to reschedule the server migration for February the 5th. The maintenance will start at 14:00 UTC and last for 2 hrs.

November 2021

Scheduled Maintenance on customers.gitlab.com

November 14, 2021 06:21 UTC

Description

customers.gitlab.com will be unavailable during the maintenance window due to a 3rd-party API set to maintenance mode. Customers won't be able to manage their subscriptions or purchase new ones.


Components

GitLab Customers Portal


Locations

Azure


Schedule

November 13, 2021 02:00 - November 13, 2021 05:00 UTC



November 14, 2021 06:21 UTC
[Update] All testing has been completed and customers.gitlab.com is now fully functional. Thank you for your patience.

November 14, 2021 05:22 UTC
[Update] We are still testing to ensure full functionality of the customers.gitlab.com site. Next update when testing is completed.

November 14, 2021 04:45 UTC
[Update] Maintenance is completed on 3rd party API, we are completing tests. Expect full functionality of customers.gitlab.com in about 30mins.

November 14, 2021 03:02 UTC
[Update] The customers.gitlab.com site is unavailable due to a 3rd-party API set to maintenance mode. You will not be able to manage subscriptions or purchase new one during this time.

September 2021

Increased error rates across all GitLab services

September 12, 2021 21:48 UTC

Incident Status

Service Disruption


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners for GitLab community contributions, SAML SSO - GitLab SaaS, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, Canary


Locations

Google Compute Engine, Azure, Digital Ocean, Zendesk, AWS




September 12, 2021 21:48 UTC
[Resolved] After investigation we determined that the elevated error rates were caused by database connection saturation; further details will be in this issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/5514

September 12, 2021 21:23 UTC
[Monitoring] GitLab performance and error rates have returned to normal; we are investigating the root cause to understand the likelyhood it will reoccur.

September 12, 2021 21:04 UTC
[Investigating] We are investigating slow response times and increased error rates across all GitLab services. Incident: gitlab.com/gitlab-com/gl-infra/production/-/issues/5514

August 2021

Elevated error rates on GitLab.com

August 11, 2021 17:51 UTC

Elevated error rates on GitLab.comPartial Service Disruption

Incident Status

Partial Service Disruption


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners for GitLab community contributions, SAML SSO - GitLab SaaS, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, Canary


Locations

Google Compute Engine, Azure, Digital Ocean, Zendesk, AWS




August 11, 2021 17:51 UTC
[Resolved] As GitLab.com is operating normally, we'll now be marking this incident as resolved. Further investigation efforts and details surrounding the issue can be found in: gitlab.com/gitlab-com/gl-infra/production/-/issues/5333

August 11, 2021 17:44 UTC
[Investigating] No material updates to report - our investigation is still ongoing. Details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/5333

August 11, 2021 17:27 UTC
[Investigating] All GitLab.com services are operating normally, but our investigation into the cause of the errors continues. Details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/5333

August 11, 2021 17:18 UTC
[Investigating] At 17:04 UTC, there was a 5 minute elevated error ratio event for all GitLab.com services. It has recovered, but we are investigating. Details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/5333

Increased error rates across all services

August 10, 2021 10:39 UTC

Incident Status

Partial Service Disruption


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners for GitLab community contributions, SAML SSO - GitLab SaaS, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, Canary


Locations

Google Compute Engine, Azure, Digital Ocean, Zendesk, AWS




August 10, 2021 10:39 UTC
[Resolved] The issue has been resolved, and we can confirm that all services on GitLab.com are fully operational. A full timeline of this incident and more details are available in gitlab.com/gitlab-com/gl-infra/production/-/issues/5324

August 10, 2021 10:05 UTC
[Monitoring] We are back to normal error rates across all services. We keep monitoring the situation. Refer to gitlab.com/gitlab-com/gl-infra/production/-/issues/5324 for more updates.

August 10, 2021 09:52 UTC
[Investigating] We are investigating increased error rates across all services, see gitlab.com/gitlab-com/gl-infra/production/-/issues/5324 for more details

GitLab performance degraded

August 6, 2021 12:48 UTC

Incident Status

Operational


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners for GitLab community contributions, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, Canary


Locations

Google Compute Engine, Azure, Digital Ocean, Zendesk, AWS




August 6, 2021 12:48 UTC
[Resolved] From 12:05 - 12:15 UTC, we saw a system degradation across all of our components, which seems to have recovered at the moment. Our engineers are investigating. Further details are available in: gitlab.com/gitlab-com/gl-infra/production/-/issues/5306

July 2021

Issue Uploading Artifacts from CI

July 21, 2021 01:39 UTC

Incident Status

Service Disruption


Components

Website, API, Git Operations, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners for GitLab community contributions, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, Canary


Locations

Google Compute Engine, Azure, Digital Ocean, Zendesk, AWS




July 21, 2021 01:39 UTC
[Resolved] This incident is now resolved and we are no longer seeing any errors. We will continue to investigate the cause internally. Full timeline of this incident and more details available in gitlab.com/gitlab-com/gl-infra/production/-/issues/5194

July 21, 2021 01:12 UTC
[Monitoring] The rollout has completed and we are no longer seeing errors with the upload. This has been applied to all our servers. We are continuously monitoring this further for any issues.

July 21, 2021 01:04 UTC
[Identified] The rollout has completed on two of our clusters. This will continue on our last few clusters and we will post further updates once completed. More details at gitlab.com/gitlab-com/gl-infra/production/-/issues/5194

July 21, 2021 00:44 UTC
[Identified] We are currently working on a recent configuration rollback that affected uploading CI artifacts. More details at gitlab.com/gitlab-com/gl-infra/production/-/issues/5194

July 21, 2021 00:28 UTC
[Investigating] We have identified a failure in uploading artifacts from CI which is currently showing a 400 error. We are investigating the issue at gitlab.com/gitlab-com/gl-infra/production/-/issues/5194 and will post more updates.

June 2021

Short service disruption on GitLab.com

June 21, 2021 21:34 UTC

Incident Status

Degraded Performance


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners for GitLab community contributions, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, Canary


Locations

Google Compute Engine, Azure, Digital Ocean, Zendesk, AWS




June 21, 2021 21:34 UTC
[Resolved] Our team has confirmed that the disruption is now gone, and that services are back to normal, Investigation will continue on gitlab.com/gitlab-com/gl-infra/production/-/issues/4945 for additional details and root cause analysis.

June 21, 2021 21:14 UTC
[Investigating] GitLab.com is up and operational, but from 20:53 to 20:58, we saw an increase in errors on GitLab.com web, API, and Git services that were a short disruption, now gone. Investigation is being done on gitlab.com/gitlab-com/gl-infra/production/-/issues/4945

May 2021

Unscheduled maintenance - Switchover of our primary database

May 8, 2021 14:59 UTC

Description

We will be conducting a switchover of our primary database at 14:45 UTC to mitigate a potential instability. Users will be unable to access GitLab.com for up to 2 minutes during this switchover. Please check gitlab.com/gitlab-com/gl-infra/production/-/issues/4528


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, Canary


Locations

Google Compute Engine, Azure, Digital Ocean, Zendesk, AWS


Schedule

May 7, 2021 14:45 - May 7, 2021 14:50 UTC



May 8, 2021 14:59 UTC
[Update] The primary database switchover is done and only affected Gitlab.com for 4 seconds. All systems are up and operational. For more details, see gitlab.com/gitlab-com/gl-infra/production/-/issues/4528

May 8, 2021 14:53 UTC
[Update] The primary database switchover is complete. For more details, please see gitlab.com/gitlab-com/gl-infra/production/-/issues/4528

March 2021

Possible Database & Redis degradation

March 18, 2021 15:06 UTC

Incident Status

Degraded Performance


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, Canary


Locations

Google Compute Engine, Azure, Digital Ocean, Zendesk, AWS




March 18, 2021 15:06 UTC
[Resolved] Our services have fully recovered and the incident is now mitigated. For the full root cause analysis follow our investigation at gitlab.com/gitlab-com/gl-infra/production/-/issues/4011

March 18, 2021 14:57 UTC
[Investigating] No material updates; we continue to see recovery across our services. Our engineers are still investigating the root cause. More details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/4011

March 18, 2021 14:40 UTC
[Investigating] We're seeing a recovery across our different services and continuing to investigate to confirm the root cause of the degradation. Details in issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/4011

March 18, 2021 14:23 UTC
[Investigating] We're investigating a possible service degradation across our Redis and DB components. Users might be experiencing delays in CI/CD pipelines. Details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/4011

Database latency resulting in 500 errors

March 15, 2021 14:19 UTC

Incident Status

Degraded Performance


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, Canary


Locations

Google Compute Engine, Azure, Digital Ocean, Zendesk, AWS




March 15, 2021 14:19 UTC
[Resolved] While the root cause is still being investigated, we have seen no further alerts and GitLab.com is fully operational. We are therefore marking this incident as resolved. For further details please check: gitlab.com/gitlab-com/gl-infra/production/-/issues/3962

March 15, 2021 12:51 UTC
[Monitoring] Gitlab.com appears to be stable, with no additional updates to share. Our engineers are still monitoring our application and investigating the root cause of this problem. Details in issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/3962

March 15, 2021 12:35 UTC
[Investigating] No material updates, our engineers are investigating the root cause. We're currently waiting for more database replicas to come online to help with the load. Details in issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/3962

March 15, 2021 12:15 UTC
[Investigating] We're still experiencing 500 errors on GitLab.com, investigation is underway. Details in issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/3962

March 15, 2021 11:51 UTC
[Investigating] We're still investigating the overall cause of the issue and are currently checking if this could be related to database re-indexing of a particular table. Details in issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/3962

March 15, 2021 11:29 UTC
[Investigating] We are experiencing database latency resulting in customers seeing 500 errors when trying to reach GitLab.com. We are currently investigating. Details in issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/3962

Intermittent issue on GitLab.com

March 1, 2021 05:19 UTC

Incident Status

Degraded Performance


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, Canary


Locations

Google Compute Engine, Azure, Digital Ocean, Zendesk, AWS




March 1, 2021 05:19 UTC
[Resolved] It seems the error rate has not gone back up as the monthly CI minutes reset job has completed. All the affected components are back to operating normally. We will post further updates and fix in the incident issue gitlab.com/gitlab-com/gl-infra/production/-/issues/3823

March 1, 2021 03:13 UTC
[Monitoring] It seems the CI minutes resetting jobs have finished. We're continuing to monitor the performance.

March 1, 2021 02:34 UTC
[Identified] We're expecting for the error to gradually recover on its own over the next 30 minutes. We're exploring a permanence solution so that it doesn't come up during the next month CI minute reset jobs.

March 1, 2021 02:01 UTC
[Identified] We notice there's degraded performance on GitLab.com. It seems to relate to our monthly CI minutes reset jobs. While the performance is recovering, we're exploring possible solution. You can read more about our investigation on gitlab.com/gitlab-com/gl-infra/production/-/issues/3823

January 2021

customers.gitlab.com is slow

January 26, 2021 22:47 UTC

customers.gitlab.com is slowDegraded Performance

Incident Status

Degraded Performance


Components

GitLab Customers Portal


Locations

Azure




January 26, 2021 22:47 UTC
[Resolved] The incident is fully resolved. We're no longer experiencing issues and are operating normally.

January 26, 2021 21:46 UTC
[Monitoring] We've identified the root cause and have mitigated the performance issues at this time. We are continuing to monitor service availability and performance of the customers portal.

January 26, 2021 21:28 UTC
[Investigating] Our customers.gitlab.com portal currently is currently having performance issues for some users/locations. We are investigating now. Details at: gitlab.com/gitlab-com/gl-infra/production/-/issues/3425

License and customers application is down

January 21, 2021 22:57 UTC

Incident Status

Partial Service Disruption


Components

GitLab Customers Portal


Locations

Azure




January 21, 2021 22:57 UTC
[Resolved] We've confirmed that both the customers.gitlab.com and license.gitlab.com applications will remain healthy. We're still working behind the scenes to tidy up our configs, but the incident has been fully resolved.

January 21, 2021 22:11 UTC
[Monitoring] All previously affected services have been restored and are online. We're currently monitoring their performance to ensure no issues recur. Details at gitlab.com/gitlab-com/gl-infra/production/-/issues/3398.

January 21, 2021 21:55 UTC
[Monitoring] The service has been restored back and our team is currently monitoring the performance of the affected applications. customers.gitlab.com and license.gitlab.com are back online additional information will be given through our incident review gitlab.com/gitlab-com/gl-infra/production/-/issues/3398

January 21, 2021 21:45 UTC
[Investigating] No material updates at the moment, our engineering team is currently researching at our database for the related causes of the problem. It has been identified that customers.gitlab.com and license.gitlab.com are affected by this problem.

January 21, 2021 21:31 UTC
[Investigating] Our engineers have detected that our internal application to generate licenses is experiencing issues that are currently under investigation, This does not impact currently issued licenses however it will prevent new ones to be created.

October 2020

SSO warnings

October 28, 2020 01:45 UTC

SSO warningsOperational

Incident Status

Operational


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, Canary


Locations

Google Compute Engine, Azure, Digital Ocean, Zendesk, AWS




October 28, 2020 01:45 UTC
[Resolved] Closing incident per updates on gitlab.com/gitlab-com/gl-infra/production/-/issues/2916

October 28, 2020 00:35 UTC
[Investigating] We’ve received reports that on SSH pushes users were getting an unintended message. This has been remediated.

Degraded performance related to GCS issues

October 24, 2020 21:56 UTC

Incident Status

Degraded Performance


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, Canary


Locations

Google Compute Engine, Azure, Digital Ocean, Zendesk, AWS




October 24, 2020 21:56 UTC
[Resolved] GitLab.com Registry is back to normal.

October 24, 2020 18:19 UTC
[Monitoring] GitLab.com Registry appears to be recovering. We are continuing to monitor its status along with GCS.

October 24, 2020 17:15 UTC
[Identified] We are continuing to monitor issues with the Registry on GitLab.com and looking for updates from our provider.

October 24, 2020 16:28 UTC
[Identified] We are continuing to monitor the issues with Registry on GitLab.com and are working with our provider on those issues.

October 24, 2020 16:04 UTC
[Identified] We are continuing to investigate issues with GitLab Pages and Registry on GitLab.com which appear to be related to underlying problems with GCS.

October 24, 2020 15:43 UTC
[Investigating] It appears GitLab pages may be affected. Other GitLab.com services appear to operating correctly.

October 24, 2020 15:40 UTC
[Investigating] We are experiencing degraded performance probably related to GCS issues. We are investigating in gitlab.com/gitlab-com/gl-infra/production/-/issues/2887

GitLab.com is experiencing degraded availability because of high database load.

October 24, 2020 11:32 UTC

Incident Status

Service Disruption


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, Canary


Locations

Google Compute Engine, Azure, Digital Ocean, Zendesk, AWS




October 24, 2020 11:32 UTC
[Resolved] GitLab.com is back to operating normally.

October 24, 2020 11:19 UTC
[Monitoring] GitLab.com is back to operating normally. We are continuing to monitor on the incident issue.

October 24, 2020 10:57 UTC
[Identified] We are continuing to investigate the origin of some slow database queries. We are tracking on gitlab.com/gitlab-com/gl-infra/production/-/issues/2885.

October 24, 2020 10:27 UTC
[Identified] We have identified a particular endpoint that was related to the issues and have disabled it temporarily on GitLab.com. The application appears to be recovering, but we are continuing to investigate.

October 24, 2020 10:02 UTC
[Investigating] We are continuing to investigate issues related to high load on the DB cluster for GitLab.com.

October 24, 2020 09:36 UTC
[Investigating] GitLab.com is experiencing degraded availability because of high database load. We are investigating.

Potential DDoS on Gitlab.com

October 23, 2020 14:53 UTC

Incident Status

Operational


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, Canary


Locations

Google Compute Engine, Azure, Digital Ocean, Zendesk, AWS




October 23, 2020 14:53 UTC
[Resolved] Thank you all of your patience, we can confirm that the DDoS incident is now fully mitigated, Gitlab.com is fully operational and secure.

October 23, 2020 13:58 UTC
[Monitoring] GitLab.com is continuing to operate normally. We've identified the source of an apparent attack and have mitigated its impact. We will continue investigating this incident and how we can further lessen the impact of attacks like these.

October 23, 2020 13:23 UTC
[Investigating] GitLab.com experienced a brief disruption due to a potential DDoS attempt. GitLab.com is now operating as normal and we are continuing to look into the source of the disruption.

August 2020

Routing failures from some locations impacting GitLab.com

August 30, 2020 16:59 UTC

Incident Status

Partial Service Disruption


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, Canary


Locations

Google Compute Engine, Azure, Digital Ocean, Zendesk, AWS




August 30, 2020 16:59 UTC
[Resolved] This incident is now resolved.

August 30, 2020 14:51 UTC
[Monitoring] Cloudflare is reporting this issue to be mitigated. We will monitor the situation.

August 30, 2020 13:26 UTC
[Identified] There is an ongoing issue with the CenturyLink ISP that is impacting some traffic to GitLab.com. We are tracking this incident here: cloudflarestatus.com/incidents/hptvkprkvp23.

customers.gitlab.com outage due to dependency outage

August 24, 2020 16:36 UTC

Incident Status

Service Disruption


Components

GitLab Customers Portal


Locations

Azure




August 24, 2020 16:36 UTC
[Resolved] customers.gitlab.com is fully operational. Please see gitlab.com/gitlab-com/gl-infra/production/-/issues/2559 for additional details.

August 24, 2020 16:34 UTC
[Monitoring] We're still monitoring Customers Portal (customers.gitlab.com) but the issue is mitigated and the portal operational. We expect closure surrounding this soon and will continue to update.

August 24, 2020 15:22 UTC
[Monitoring] Customers Portal (customers.gitlab.com) is now available. We've been actively monitoring for failures and will continue to monitor. Production issue for details: gitlab.com/gitlab-com/gl-infra/production/-/issues/2559

August 24, 2020 14:25 UTC
[Identified] Customers Portal (customers.gitlab.com) is still unavailable due to an outage in one of its dependencies. The good news is that the dependency service is rolling out mitigation measures, so hopefully we will be back on track as well soon. Stay tuned.

August 24, 2020 13:44 UTC
[Identified] Customers Portal (customers.gitlab.com) is still unavailable due to an outage in one of its dependencies. Issue for tracking: gitlab.com/gitlab-com/gl-infra/production/-/issues/2559

August 24, 2020 12:53 UTC
[Identified] customers.gitlab.com is experiencing an outage due to an outage in one of its dependencies. We're continuing to monitor the situation and will keep you updated.





Back to current status