December 2018

We are facing high error rates at Gitlab.com

December 10, 2018 10:42 UTC

We are facing high error rates at Gitlab.comOperational

Incident Status

Operational

Components

Website, API, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, Background Processing

Locations

Google Compute Engine, Digital Ocean

---

December 10, 2018 10:42 UTC

[Resolved] Gitlab.com is back to normal.

December 10, 2018 10:33 UTC

[Identified] We find out the issue, and we are executing a rollback.

December 10, 2018 10:18 UTC

[Investigating] We find out the issue, and we are executing a rollback.

Patroni migration - Database High Availability

December 8, 2018 16:39 UTC

Patroni migration - Database High AvailabilityPlanned Maintenance

Description

GitLab.com planned maintenance for migration to Patroni is starting. See you on the other side!

Components

Website, API, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, Background Processing

Locations

Google Compute Engine, Digital Ocean

Schedule

December 8, 2018 13:00 - December 8, 2018 13:30 UTC

---

December 8, 2018 16:39 UTC

[Update] Maintenance was completed at 14:30 UTC

December 8, 2018 16:38 UTC

[Update] Maintenance was started at 1300 UTC

November 2018

API disruption

November 29, 2018 21:20 UTC

API disruptionService Disruption

Incident Status

Service Disruption

Components

Website, API, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, Background Processing

Locations

Google Compute Engine, Digital Ocean

---

November 29, 2018 21:20 UTC

[Resolved] All GitLab.com api backends have remained healthy. We will link an RCA and actions to gitlab.com/gitlab-com/gl-infra/production/issues/592. Apologies for the inconvenience.

November 29, 2018 20:56 UTC

[Investigating] Starting at 20:20 UTC ending at 20:38 UTC, we observed all of the backends for our API nodes unavailable. We are tracking in gitlab.com/gitlab-com/gl-infra/production/issues/592. We're continuing to investigate.

November 29, 2018 20:37 UTC

[Investigating] We are currently investigating an issue with our API.

CI Runners not respecting tags

November 12, 2018 23:48 UTC

CI Runners not respecting tagsPartial Service Disruption

Incident Status

Partial Service Disruption

Components

CI/CD - Hosted runners on Linux

Locations

Google Compute Engine, Digital Ocean

---

November 12, 2018 23:48 UTC

[Resolved] The patch has been applied and CI runners are now properly respecting job tags.

November 12, 2018 21:15 UTC

[Identified] With the deploy of GitLab 11.5.0 RC4 to GitLab.com, the CI runners stopped respecting job tags. We have prepared a patch and are deploying it to GitLab.com. For more information, please see gitlab.com/gitlab-com/gl-infra/production/issues/564

October 2018

Git HTTP Authentication Problems

October 26, 2018 17:14 UTC

Git HTTP Authentication ProblemsPartial Service Disruption

Incident Status

Partial Service Disruption

Components

Website, API, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, Background Processing

Locations

Google Compute Engine, Digital Ocean

---

October 26, 2018 17:14 UTC

[Resolved] Resolving incident after monitoring.

October 26, 2018 15:56 UTC

[Monitoring] The fix for the https authentication issues has been deployed and we are monitoring. Git over https should again be working.

October 26, 2018 15:48 UTC

[Identified] We are starting to roll out the patch for the git https issues to our fleet.

October 26, 2018 15:23 UTC

[Identified] We have applied a patch to our canary for the git over https issues and are verifying. The rest of the production fleet soon.

October 26, 2018 15:01 UTC

[Identified] We are testing a fix which should resolve the https authentication issues with git operations.

October 26, 2018 12:32 UTC

[Identified] We have identified an issue which is causing all Git HTTPS authentication for users using private access tokens - including all 2FA users - to fail. Apologies for any inconvenience. We are urgently working on a fix. Please switch to Git SSH or password authentication while we work on a resolution. Further details available at gitlab.com/gitlab-com/gl-infra/production/issues/536

CI Queue performance problems

October 26, 2018 14:10 UTC

CI Queue performance problemsPartial Service Disruption

Incident Status

Partial Service Disruption

Components

CI/CD - Hosted runners on Linux

Locations

Google Compute Engine, Digital Ocean

Postmortem

https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/5375

---

October 26, 2018 14:10 UTC

[Resolved] The root cause of CI Queue performance problems was resolved and the system is back now to normal capacity. We will continue to monitor it, but CI jobs should be processed without additional slowness. For more information see gitlab.com/gitlab-com/gl-infra/infrastructure/issues/5375.

October 24, 2018 19:33 UTC

[Monitoring] The load on CI Queues went down and the system is operable again since last 2 hours. We've also prepared and are validating the fix for the issue. We're still monitoring the system until the issue will be finally resolved.

October 24, 2018 12:59 UTC

[Identified] We've identified the root cause of CI queues performance problem. We're now working on a fix.

October 23, 2018 20:35 UTC

[Monitoring] At this moment the performance problems of CI Queues settled down. We'll continue to monitoring it as well as investigating to find out the root cause of the issue. We're sorry for any inconveniences.

October 23, 2018 13:13 UTC

[Investigating] CI queuing times are much bigger than usual. It looks like there is a big slowdown of auto-scaled machines creation events. We're investigating the problem.

September 2018

Redis Maintenance for Memory Management

September 29, 2018 08:30 UTC

Redis Maintenance for Memory ManagementPlanned Maintenance

Description

We will be executing maintenance of the GitLab.com redis cluster. While we don't expect any interruption of service there may be a brief time of increased errors. Please see gitlab.com/gitlab-com/gl-infra/production/issues/487 for more information.

Components

Website, API, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, Background Processing

Locations

Google Compute Engine, Digital Ocean

Schedule

September 29, 2018 08:00 - September 29, 2018 08:30 UTC

---

September 29, 2018 08:30 UTC

[Update] We have finished our Redis maintenance window and all services are operating as normal.

September 29, 2018 08:00 UTC

[Update] We are starting maintenance of our redis fleet which may result in a brief interruption of service on gitlab.com. We expect minimal service interruption during the maintenance.

August 2018

GitLab.com unavailable

August 17, 2018 19:21 UTC

GitLab.com unavailableService Disruption

Incident Status

Service Disruption

Components

Website, API, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, Background Processing

Locations

Google Compute Engine, Digital Ocean

---

August 17, 2018 19:21 UTC

[Resolved] Closing the incident. Systems have been normal since 17:08 UTC.

August 17, 2018 17:25 UTC

[Monitoring] GitLab issue gitlab.com/gitlab-com/production/issues/425 and working doc: docs.google.com/document/d/1p0pLaASameWw_l_DTeWhzkUVXWE8N8MUtJK4fmkCxHg/edit#

August 17, 2018 17:22 UTC

[Monitoring] We experienced an issue with pgbouncer and various application servers and api nodes became unhealthy. The immediate issue should be resolved. We are monitoring and investigating the cause of the event.

August 17, 2018 17:06 UTC

[Investigating] We are investigating issues with the site.

Maintenance for GCP Migration August 11

August 11, 2018 13:01 UTC

Maintenance for GCP Migration August 11Planned Maintenance

Description

On Saturday August 11, 2018, we will be performing maintenance to do our GCP migration. The window will start at 10:00 UTC and last for 2 hours.

Components

Website, API, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, Background Processing

Locations

Google Compute Engine, Azure, Digital Ocean

Schedule

August 11, 2018 10:00 - August 11, 2018 12:00 UTC

---

August 11, 2018 13:01 UTC

[Update] Our maintenance window is complete. GitLab.com is running from GCP! We will continue to monitor systems over the weekend.

August 11, 2018 12:29 UTC

[Update] GitLab.com planned maintenance for migration to @GCPcloud is almost complete. GitLab.com is available although we're continuing to verify that all systems are functioning correctly.

August 11, 2018 12:02 UTC

[Update] Maintenance update 6- Internal QA phase 1 passed. We are doing final configuration changes before making GitLab.com available. We will continue to verify that all systems are functioning correctly.

August 11, 2018 11:43 UTC

[Update] Maintenance update 5- Internal QA is in progress. We will do a final confidence check soon. Given the current timings, we may go for another 30 minutes to 12:30 UTC for this maintenance window.

August 11, 2018 11:23 UTC

[Update] Maintenance update 4- updating GitLab config in GCP is nearly done. Our internal QA which will start soon.

August 11, 2018 11:02 UTC

[Update] Maintenance update 3- we are promoting the GCP database and updating GitLab config in GCP to prep for our internal QA which should start soon.

August 11, 2018 10:41 UTC

[Update] Maintenance update 2- GCP side is up to date from Azure. Starting reconfig to point to GCP prod soon. GitLab.com will remain down while we verify our new production setup.

August 11, 2018 10:20 UTC

[Update] Maintenance update - we have stopped access to GItLab.com and are making sure all items and data are synchronized and good in GCP.

August 11, 2018 10:01 UTC

[Update] GCP Switchover maintenance is starting. Working doc: docs.google.com/document/d/1CzkieGnqJStAh3pMwgg-v62-HTpBdFDoP9smGaFE1Ko

Maintenance for dry run of GCP Migration

August 6, 2018 23:18 UTC

Maintenance for dry run of GCP MigrationPlanned Maintenance

Description

We will perform another short maintenance window similar to July 28 to validate updates to our procedure.

Components

Website, API, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, Background Processing

Locations

Google Compute Engine, Azure, Digital Ocean

Schedule

August 4, 2018 13:00 - August 4, 2018 14:00 UTC

---

August 6, 2018 23:18 UTC

[Update] Cleaning up maintenance from Aug 4.

August 6, 2018 23:17 UTC

[Update] Cleaning up maintenance from Aug 4.

July 2018

Maintenance for GCP Migration July 21

July 21, 2018 14:46 UTC

Maintenance for GCP Migration July 21Planned Maintenance

Description

On July 21, 2018, we will be performing maintenance related to our GCP migration: about.gitlab.com/2018/07/19/gcp-move-update The window will start at 13:00 UTC and last for 1 hour.

Components

Website, API, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, Background Processing

Locations

Google Compute Engine, Azure, Digital Ocean

Schedule

July 21, 2018 13:00 - July 21, 2018 14:00 UTC

---

July 21, 2018 14:46 UTC

[Update] Our maintenance is complete. All GitLab.com systems are running normally.

July 21, 2018 14:18 UTC

[Update] GitLab.com should now be available - we will continue to monitor the system now that maintenance is complete to make sure all systems are healthy.

July 21, 2018 14:11 UTC

[Update] We are wrapping up the maintenance window for our practice run o of the GCP switchover.

July 21, 2018 13:13 UTC

[Update] Maintenance has started for our dry run of GitLab.com GCP switchover.

Site Slowdown

July 11, 2018 16:51 UTC

Site SlowdownPartial Service Disruption

Incident Status

Partial Service Disruption

Components

Website, API, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, Background Processing

Locations

Google Compute Engine, Azure, Digital Ocean

---

July 11, 2018 16:51 UTC

[Resolved] The rollback is complete and systems are back to normal operations.

July 11, 2018 16:05 UTC

[Monitoring] We are continuing to monitor our environment with the rollback.

July 11, 2018 15:39 UTC

[Monitoring] The rollback is proceeding and we are monitoring the recovery of our systems.

July 11, 2018 15:11 UTC

[Identified] A change introduced in the 11.1.0 RC7 deployment lead to high sequential scans in the database, affecting load. We are rolling back to 11.1.0 RC4.

July 11, 2018 15:06 UTC

[Investigating] While deploying 11.1.0 RC7 to we experienced higher than normal database load which is causing site degradation - we are investigating to pinpoint the cause.

NFS Backend Failures

July 2, 2018 04:36 UTC

NFS Backend FailuresPartial Service Disruption

Incident Status

Partial Service Disruption

Components

Website, API, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, Background Processing

Locations

Google Compute Engine, Azure, Digital Ocean

---

July 2, 2018 04:36 UTC

[Resolved] All hosts are back online and we have seen no more errors presented in the backend storage servers.

July 2, 2018 04:19 UTC

[Monitoring] We have recovered all of our backend storage servers and are not monitoring their status.

July 2, 2018 03:56 UTC

[Identified] Our provider is having issues with some of the nodes which has introduced instability into our backend storage nodes. This affects all of GitLab.