GitLab.com disruption due to database issues related to failover Operational

Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com

Locations

Google Compute Engine, Azure, Digital Ocean, Zendesk, AWS



March 30, 2020 07:14 UTC
[Resolved] GitLab.com is operating normally and we have not seen a recurrence of the issue during our latest monitoring period. Our apologies for any inconvenience. We will be doing a root cause analysis (RCA) here: gitlab.com/gitlab-com/gl-infra/production/-/issues/1865

March 30, 2020 06:37 UTC
[Monitoring] The affected database replicas are now back to normal operation. We're monitoring the issue now and investigate to ensure this doesn't recur. Details can be found at gitlab.com/gitlab-com/gl-infra/production/-/issues/1865

March 30, 2020 06:29 UTC
[Identified] Our db replicas are continuing to recover and performance is now improving on GitLab.com.

March 30, 2020 06:13 UTC
[Identified] GitLab.com is operational, but our engineers are still working on bringing all DB replicas fully online.

March 30, 2020 06:02 UTC
[Identified] The recovery is still in process while we work to bring our other database replicas online.

March 30, 2020 05:53 UTC
[Identified] A few of other database replicas have recovered and we are continuing to bring other replicas back online.

March 30, 2020 05:43 UTC
[Identified] We are now getting replicas back online and GitLab.com should be starting recover.

March 30, 2020 05:36 UTC
[Identified] We are still working to bring our db replicas online so you may still experience slowdowns in your experience on GitLab.com at this point.

March 30, 2020 05:24 UTC
[Identified] We are continuing to work on bringing our db replicas back online.

March 30, 2020 05:08 UTC
[Identified] We are still facing slowdowns in some requests as we work on resolving the issue with our databases.

March 30, 2020 04:53 UTC
[Identified] We are tracking the incident on gitlab.com/gitlab-com/gl-infra/production/-/issues/1865.

March 30, 2020 04:43 UTC
[Identified] GitLab.com is experiencing a service disruption related to an automatic DB failover which did not fully succeed. We are currently working on bringing replicas back online.

Back to current status