GitLab System Status

August 2018

Maintenance for dry run of GCP Migration August 6, 2018 23:18 UTC: Maintenance for dry run of GCP MigrationPlanned Maintenance
Description
We will perform another short maintenance window similar to July 28 to validate updates to our procedure.

Components
Website, API, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, Background Processing

Locations
Google Compute Engine, Azure, Digital Ocean

Schedule
August 4, 2018 13:00 - August 4, 2018 14:00 UTC

August 6, 2018 23:18 UTC
[Update] Cleaning up maintenance from Aug 4.

August 6, 2018 23:17 UTC
[Update] Cleaning up maintenance from Aug 4.

Elevated Number of 500 Errors August 2, 2018 19:27 UTC: Elevated Number of 500 ErrorsPartial Service Disruption
Incident Status
Partial Service Disruption

Components
Website, API

Locations
Azure

Postmortem
https://gitlab.com/gitlab-com/infrastructure/issues/4646

August 2, 2018 19:27 UTC
[Resolved] We have completed the deploy and patched the issue with deploy tokens. Please check out gitlab.com/gitlab-com/infrastructure/issues/4646 for information going forward.

August 2, 2018 17:59 UTC
[Identified] The elevated errors were related to a deploy that was in flight. We are watching our systems as that deploy finishes.

August 2, 2018 17:27 UTC
[Investigating] We are seeing an elevated number of 500 errors when attempting to clone a repository via deploy tokens and are currently investigating.

Maintenance for GCP Migration July 21 July 21, 2018 14:46 UTC: Maintenance for GCP Migration July 21Planned Maintenance
Description
On July 21, 2018, we will be performing maintenance related to our GCP migration: about.gitlab.com/2018/07/19/gcp-move-update The window will start at 13:00 UTC and last for 1 hour.

Components
Website, API, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, Background Processing

Locations
Google Compute Engine, Azure, Digital Ocean

Schedule
July 21, 2018 13:00 - July 21, 2018 14:00 UTC

July 21, 2018 14:46 UTC
[Update] Our maintenance is complete. All GitLab.com systems are running normally.

July 21, 2018 14:18 UTC
[Update] GitLab.com should now be available - we will continue to monitor the system now that maintenance is complete to make sure all systems are healthy.

July 21, 2018 14:11 UTC
[Update] We are wrapping up the maintenance window for our practice run o of the GCP switchover.

July 21, 2018 13:13 UTC
[Update] Maintenance has started for our dry run of GitLab.com GCP switchover.

Site Slowdown July 11, 2018 16:51 UTC: Site SlowdownPartial Service Disruption
Incident Status
Partial Service Disruption

Components
Website, API, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, Background Processing

Locations
Google Compute Engine, Azure, Digital Ocean

July 11, 2018 16:51 UTC
[Resolved] The rollback is complete and systems are back to normal operations.

July 11, 2018 16:05 UTC
[Monitoring] We are continuing to monitor our environment with the rollback.

July 11, 2018 15:39 UTC
[Monitoring] The rollback is proceeding and we are monitoring the recovery of our systems.

July 11, 2018 15:11 UTC
[Identified] A change introduced in the 11.1.0 RC7 deployment lead to high sequential scans in the database, affecting load. We are rolling back to 11.1.0 RC4.

July 11, 2018 15:06 UTC
[Investigating] While deploying 11.1.0 RC7 to we experienced higher than normal database load which is causing site degradation - we are investigating to pinpoint the cause.

NFS Backend Failures July 2, 2018 04:36 UTC: NFS Backend FailuresPartial Service Disruption
Incident Status
Partial Service Disruption

Components
Website, API, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, Background Processing

Locations
Google Compute Engine, Azure, Digital Ocean

July 2, 2018 04:36 UTC
[Resolved] All hosts are back online and we have seen no more errors presented in the backend storage servers.

July 2, 2018 04:19 UTC
[Monitoring] We have recovered all of our backend storage servers and are not monitoring their status.

July 2, 2018 03:56 UTC
[Identified] Our provider is having issues with some of the nodes which has introduced instability into our backend storage nodes. This affects all of GitLab.