March 2019

gitlab-com/gl-infra/production#733

March 18, 2019 18:44 UTC

gitlab-com/gl-infra/production#733Degraded Performance

Incident Status

Degraded Performance

Components

Website, API, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, Background Processing, Support Services, packages.gitlab.com

Locations

Google Compute Engine, Digital Ocean, Zendesk, AWS

---

March 18, 2019 18:44 UTC

[Resolved] We've stabilized and we're resolving the issue. All operations are normal.

March 18, 2019 18:24 UTC

[Monitoring] We've relieved the pressure on our build infrastructure and systems are operating normally. We will continue to monitor for stability before issuing issuing an all clear.

March 18, 2019 18:05 UTC

[Identified] We've narrowed the issue to our build infrastructure. 500 errors are still occurring, but in low numbers. Users might continue to experience issues or delays with their build pipelines.

March 18, 2019 17:34 UTC

[Investigating] We're still seeing a slightly elevated error count, but the error rate is declining. API users are those most likely to be affected by this incident. We apologize for any degraded performance and will update you again soon.

March 18, 2019 17:13 UTC

[Investigating] We are currently seeing a high rate of errors on GitLab.com. Our infrastructure team is investigating.

Upgrade of packages.gitlab.com

March 5, 2019 17:25 UTC

Upgrade of packages.gitlab.comPlanned Maintenance

Description

We will be upgrading packages.gitlab.com to the most recent version of PackageCloud. During this time, downloads may not be available. See gitlab.com/gitlab-com/gl-infra/production/issues/705 for more info.

Components

packages.gitlab.com

Locations

AWS

Schedule

March 4, 2019 21:00 - March 4, 2019 22:00 UTC

---

March 5, 2019 17:25 UTC

[Update] The migrations have completed and the maintenance is complete.

March 5, 2019 16:54 UTC

[Update] The database migration is currently updating the downloads table. Downloading packages will not be available until this is complete. Based on the timing of the other migrations, this should take around 10 minutes.

March 5, 2019 13:47 UTC

[Update] The migration is still ongoing due to long running database migrations. Packages are still available to download. You can follow the progress on gitlab.com/gitlab-com/gl-infra/production/issues/705

March 5, 2019 01:02 UTC

[Update] The maintenance is ongoing due to a long running migration. Packages are still available to download.

March 4, 2019 22:00 UTC

[Update] The maintenance is ongoing and taking longer than anticipated.

March 4, 2019 21:00 UTC

[Update] We are now beginning our upgrade of packages.gitlab.com. During this time download of packages may be unavailable.