August 2019

We are investigating higher latencies and durations for job processing.

August 8, 2019 15:45 UTC

We are investigating higher latencies and durations for job processing.Degraded Performance

Incident Status

Degraded Performance

Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com

Locations

Google Compute Engine, Azure, Zendesk, AWS

---

August 8, 2019 15:45 UTC

[Resolved] The issue is resolved.

August 8, 2019 15:16 UTC

[Monitoring] Update on issue gitlab.com/gitlab-com/gl-infra/production/issues/1039. The problem is solved. We are monitoring the queues.

August 8, 2019 15:00 UTC

[Monitoring] Update on issue gitlab.com/gitlab-com/gl-infra/production/issues/1039. The problem is solved. We are monitoring the queues.

August 8, 2019 14:54 UTC

[Identified] Update on issue gitlab.com/gitlab-com/gl-infra/production/issues/1039. The queues are clearing. We are processing the queues.

August 8, 2019 14:37 UTC

[Identified] Update on issue gitlab.com/gitlab-com/gl-infra/production/issues/1039. The queues are clearing. We are processing the queues.

August 8, 2019 14:21 UTC

[Identified] Update on issue gitlab.com/gitlab-com/gl-infra/production/issues/1039. The queues are clearing. We are processing the queues.

August 8, 2019 14:06 UTC

[Identified] Update on issue gitlab.com/gitlab-com/gl-infra/production/issues/1039. We have found delays again, and the queues are growing.

August 8, 2019 13:48 UTC

[Identified] Update on issue gitlab.com/gitlab-com/gl-infra/production/issues/1039. We have found delays again, and the queues are growing.

August 8, 2019 13:28 UTC

[Identified] Update on issue gitlab.com/gitlab-com/gl-infra/production/issues/1039. We have found delays again, and the queues are growing.

August 8, 2019 13:20 UTC

[Identified] Update on issue gitlab.com/gitlab-com/gl-infra/production/issues/1039. We found the problem, and we are working mitigating, and the queues are clearing.

August 8, 2019 13:04 UTC

[Identified] Update on issue gitlab.com/gitlab-com/gl-infra/production/issues/1039. We found the problem, and we are working mitigating, and the queues are clearing.

August 8, 2019 12:46 UTC

[Identified] Update on issue gitlab.com/gitlab-com/gl-infra/production/issues/1039. We found the problem, and we are working mitigating, and the queues are clearing.

August 8, 2019 12:31 UTC

[Identified] Update on issue gitlab.com/gitlab-com/gl-infra/production/issues/1039. We found the problem, and we are working mitigating, and the queues are clearing.

August 8, 2019 12:16 UTC

[Identified] Update on issue gitlab.com/gitlab-com/gl-infra/production/issues/1039. We found the problem, and we are working mitigating, and the queues are clearing.

August 8, 2019 12:00 UTC

[Identified] Update on issue gitlab.com/gitlab-com/gl-infra/production/issues/1039. We found the problem, and we are working mitigating, and the queues are clearing.

August 8, 2019 11:52 UTC

[Identified] Update on issue #1039. We found the problem, and we are working mitigating and the queues are going down.

August 8, 2019 11:29 UTC

[Identified] Update on issue #1039. We applied the hot patch, and we keep with the investigation.

August 8, 2019 11:11 UTC

[Identified] Update on issue #1039. We applied the hot patch and we keep with the investigation.

August 8, 2019 10:51 UTC

[Identified] Update on issue #1039. We are issuing a hot patch.

August 8, 2019 10:25 UTC

[Identified] Update on issue #1039. We are issuing a hot patch.

August 8, 2019 10:07 UTC

[Identified] Update on issue #1039. We are issuing a hot patch.

August 8, 2019 10:02 UTC

[Identified] We are issuing a hot patch.

August 8, 2019 09:45 UTC

[Identified] We are issuing a hot patch.

August 8, 2019 09:28 UTC

[Identified] The situation is stabilizing, and we are issuing a hot patch.

August 8, 2019 09:10 UTC

[Investigating] The situation is stabilizing, we are investigating higher latencies and durations for job processing.

August 8, 2019 08:54 UTC

[Investigating] The situation is stabilizing, we still under investigation of the root cause of the issue.

August 8, 2019 08:39 UTC

[Investigating] The situation is stabilizing, we still under investigation of the root cause of the issue.

August 8, 2019 08:15 UTC

[Investigating] We are investigating higher latencies and durations for job processing.

August 8, 2019 07:56 UTC

[Investigating] We are investigating higher latencies and durations for job processing.

August 8, 2019 07:21 UTC

[Investigating] We are investigating higher latencies and durations for job processing.

We are executing an update on the backend of the platform

August 7, 2019 21:50 UTC

We are executing an update on the backend of the platformOperational

Incident Status

Operational

Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com

Locations

Google Compute Engine, Azure, Zendesk, AWS

Postmortem

https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/7468

---

August 7, 2019 21:50 UTC

[Resolved] IncidentReview now available at: gitlab.com/gitlab-com/gl-infra/infrastructure/issues/7468

August 7, 2019 19:47 UTC

[Identified] We are preparing the final procedure to execute in the production environment.

August 7, 2019 19:06 UTC

[Identified] We are executing the final procedure in the staging environment.

August 7, 2019 17:50 UTC

[Identified] We are executing tests in the staging environment.

August 7, 2019 17:30 UTC

[Identified] We are executing a review of the upgrade that we will perform on production.

August 7, 2019 17:13 UTC

[Identified] We are executing a review of the upgrade that we will perform on production.

August 7, 2019 16:21 UTC

[Investigating] We are under investigation of the best approach to renew the certificates.

August 7, 2019 16:02 UTC

[Investigating] Status on the incident #1037, we are updating certificates on backend and expect no impact in production

behaviour change in new docker version

August 6, 2019 12:11 UTC

behaviour change in new docker version Operational

Incident Status

Operational

Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com

Locations

Google Compute Engine, Azure, Zendesk, AWS

---

August 6, 2019 12:11 UTC

[Resolved] We are closing this incident which was kept open for informational purposes.

August 2, 2019 10:15 UTC

[Identified] A small number of users are reporting runner issues who are using docker:dind. If you are seeing errors with jobs that use the docker:dind image please see gitlab.com/gitlab-com/gl-infra/production/issues/1023 for how to workaround the issue in your CI configuration.

Database Maintenance

August 5, 2019 23:23 UTC

Database MaintenancePlanned Maintenance

Description

We are executing maintenance in the database disks, no services will be impacted.

Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com

Locations

Google Compute Engine, Azure, Zendesk, AWS

Schedule

August 5, 2019 22:00 - August 5, 2019 23:03 UTC

---

August 5, 2019 23:23 UTC

[Update] The maintenance has been completed successfully.

August 5, 2019 23:04 UTC

[Update] Maintenance on the databases disks. No systems will be impacted.

We are experiencing degraded performance

August 5, 2019 15:03 UTC

We are experiencing degraded performanceOperational

Incident Status

Operational

Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com

Locations

Google Compute Engine, Azure, Zendesk, AWS

---

August 5, 2019 15:03 UTC

[Resolved] All waiting notifications have been send out and queues are back to zero.

August 5, 2019 14:34 UTC

[Monitoring] The issue is resolved. We are monitoring.

August 5, 2019 14:19 UTC

[Investigating] We are under investigation of the issue, that might generate delay to receive notifications for issues and merge requests.

August 5, 2019 13:51 UTC

[Investigating] We are under investigation of the issue, that might generate delay to receive notifications for issues and merge requests.

August 5, 2019 13:17 UTC

[Investigating] We are experiencing degraded performance, that might generate delay to receive notifications and execute merge requests.

July 2019

job queue takes longer to process jobs

July 31, 2019 16:39 UTC

job queue takes longer to process jobsDegraded Performance

Incident Status

Degraded Performance

Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com

Locations

Google Compute Engine, Azure, Zendesk, AWS

---

July 31, 2019 16:39 UTC

[Resolved] Pipeline queues are back to normal. We apologize for any delays you may have experienced with your CI jobs.

July 31, 2019 16:13 UTC

[Monitoring] The pipeline_processing queue is back to normal levels and we will continue to monitor levels.

July 31, 2019 15:47 UTC

[Identified] We've identified the slowdown and the queue is falling. We are tuning settings to relieve back pressure. Additional updates at gitlab.com/gitlab-com/gl-infra/production/issues/1014

July 31, 2019 14:45 UTC

[Investigating] We are investigating an issue with a job queue taking longer to process jobs.

We are experiencing degraded performance

July 31, 2019 11:56 UTC

We are experiencing degraded performanceDegraded Performance

Incident Status

Degraded Performance

Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com

Locations

Google Compute Engine, Azure, Zendesk, AWS

---

July 31, 2019 11:56 UTC

[Resolved] The degraded performance issue is resolved.

July 31, 2019 11:19 UTC

[Investigating] We are still under investigation.

July 31, 2019 10:48 UTC

[Investigating] We are experiencing load problems on one of our Git file server backends. This may lead to latency issues for some users. We are investigating….

Delay in reply-to-email processing on GitLab.com 2019-07-24

July 25, 2019 00:17 UTC

Delay in reply-to-email processing on GitLab.com 2019-07-24Operational

Incident Status

Operational

Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com

Locations

Google Compute Engine, Azure, Zendesk, AWS

---

July 25, 2019 00:17 UTC

[Resolved] Queues for email processing have cleared and are now normal.

July 25, 2019 00:11 UTC

[Monitoring] We found a delay in Service Desk and reply-to-email processing on GitLab.com today on July 24 that is now resolved. Tracking is on issue: gitlab.com/gitlab-com/gl-infra/production/issues/992.

Database Failover

July 12, 2019 00:06 UTC

Database FailoverPlanned Maintenance

Description

This evening we'll be failing over our primary database. Any long running queries exceeding are unlikely to finish and this could result in API errors or partial loading of web pages. We're taking every precaution to ensure minimal disruption and we will notify you here once the maintenance is complete.

Components

Website, API, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com

Locations

Google Compute Engine, Azure, Zendesk, AWS

Schedule

July 11, 2019 23:00 - July 11, 2019 23:15 UTC

---

July 12, 2019 00:06 UTC

[Update] The primary is stable and the maintenance is complete.

July 11, 2019 23:12 UTC

[Update] The database gracefully failed over. We're currently monitoring the health of the new primary.

Database Replica Replacement

July 11, 2019 12:07 UTC

Database Replica ReplacementOperational

Incident Status

Operational

Components

Website, API, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com

Locations

Google Compute Engine, Azure, Digital Ocean, Zendesk, AWS

---

July 11, 2019 12:07 UTC

[Resolved] Replica created.

July 11, 2019 11:08 UTC

[Identified] We are under maintenance for a secondary database, due replication issue. No users impacted.

We are currently investigating delayed execution of mirror jobs.

July 2, 2019 17:06 UTC

We are currently investigating delayed execution of mirror jobs.Degraded Performance

Incident Status

Degraded Performance

Components

Website, API, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com

Locations

Google Compute Engine, Azure, Digital Ocean, Zendesk, AWS

---

July 2, 2019 17:06 UTC

[Resolved] We're all caught up with the delayed execution of pull mirror jobs. dashboards.gitlab.com/d/_MKRXrSmk/pull-mirrors?orgId=1&refresh=30s. We apologize for the inconvenience. The original issue has been moved into our production tracker. Ongoing conversation will take place there - gitlab.com/gitlab-com/gl-infra/production/issues/934.

July 2, 2019 17:06 UTC

[Resolved] All pull mirrors have been processed and the queue is back to normal operations since 4pm UTC.

July 2, 2019 15:05 UTC

[Investigating] We are currently investigating delayed execution of repository mirror jobs. See gitlab.com/gitlab-com/gl-infra/infrastructure/issues/7155 for details.

June 2019

increased delays for CI jobs

June 27, 2019 10:12 UTC

increased delays for CI jobsDegraded Performance

Incident Status

Degraded Performance

Components

Website, API, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com

Locations

Google Compute Engine, Azure, Digital Ocean, Zendesk, AWS

---

June 27, 2019 10:12 UTC

[Resolved] CI jobs are not delayed anymore since 6/26 14:00 UTC.

June 26, 2019 14:43 UTC

[Investigating] CI pipeline delays are improving now. We are still investigating the root cause. See gitlab.com/gitlab-com/gl-infra/production/issues/922 for details.

June 26, 2019 14:10 UTC

[Investigating] We are investigating delays for jobs in CI pipelines.

git operations over https are slow

June 20, 2019 19:12 UTC

git operations over https are slowDegraded Performance

Incident Status

Degraded Performance

Components

Website, API, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com

Locations

Google Compute Engine, Azure, Digital Ocean, Zendesk, AWS

---

June 20, 2019 19:12 UTC

[Resolved] We're operating normally again. We are continuing the search for a root cause analysis and will update in gitlab.com/gitlab-com/gl-infra/production/issues/912.

June 20, 2019 17:52 UTC

[Monitoring] We were unable to determine the root cause of the problem, but we've seen the latencies return to normal levels. We will continue monitoring for spikes and we'll be carefully listening for user reports.

June 20, 2019 17:04 UTC

[Investigating] We are still investigating the git access slowdowns for a limited number of repositories.

June 20, 2019 16:31 UTC

[Investigating] We've narrowed the impact to specific projects, including gitlab-ce. The majority of users are not impacted by this issue.

June 20, 2019 16:14 UTC

[Investigating] We've confirmed the latency issues, but we're still investigating. Thanks for your patience.

June 20, 2019 15:49 UTC

[Investigating] We're investigating reports of slow git operations over https connections.

May 2019

Observing git-over-ssh errors

May 23, 2019 19:22 UTC

Observing git-over-ssh errorsOperational

Incident Status

Operational

Components

Website, API, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, Background Processing, Support Services, packages.gitlab.com

Locations

Google Compute Engine, Digital Ocean, Zendesk, AWS

---

May 23, 2019 19:22 UTC

[Resolved] We've determined the issue is no longer impacting users and are marking this issue resolved.

May 23, 2019 17:05 UTC

[Monitoring] We are continuing to monitor git over ssh performance. Response latency and error ratios are back to normal, and we've not observed any other metrics that indicate any other operations on GitLab.com were impacted.

May 23, 2019 16:29 UTC

[Investigating] We're observing increased errors on GitLab.com where git connections via ssh are unexpectedly failing. The issue is being tracked in gitlab.com/gitlab-com/gl-infra/production/issues/844.

Issues with credentials for customers previously logged in via SSO

May 22, 2019 16:05 UTC

Issues with credentials for customers previously logged in via SSO Operational

Incident Status

Operational

Components

Website, API, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, Background Processing, Support Services, packages.gitlab.com

Locations

Google Compute Engine, Digital Ocean, Zendesk, AWS

---

May 22, 2019 16:05 UTC

[Resolved] We haven't received any additional reports of issues with SSO authentication. Both login and runner credentials are operating normally, again. Our apologies to anyone who was impacted–and our thanks to the few who brought this issue to our attention!

May 22, 2019 15:15 UTC

[Monitoring] We've received customer confirmation that the configuration change we reverted has resolved their issues. We'll continue monitoring with our support team.

May 22, 2019 14:58 UTC

[Investigating] We've reverted a setting that was forcing new sessions for SSO authentication. But, we're still investigating, as we've yet to find definitive metrics indicating the issue as resolved.

May 22, 2019 14:39 UTC

[Investigating] We're currently investigating customer reports of issues with CI runners and user logins. If you're using SSO you may be impacted.

We are investigating performance issues on our site.

May 17, 2019 15:26 UTC

We are investigating performance issues on our site.Degraded Performance

Incident Status

Degraded Performance

Components

Website, API, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, Background Processing, Support Services, packages.gitlab.com

Locations

Google Compute Engine, Digital Ocean, Zendesk, AWS

---

May 17, 2019 15:26 UTC

[Resolved] The performance issues are fully resolved. And the site is fully operational.

May 17, 2019 13:10 UTC

[Monitoring] The rollback is still ongoing and the website performance is not fully recovered yet.

May 17, 2019 11:42 UTC

[Monitoring] We are seeing performance of GitLab.com going back to normal levels while the rollback is going on. See gitlab.com/gitlab-com/gl-infra/production/issues/832 for more details.

May 17, 2019 10:37 UTC

[Investigating] We are rolling back the changes that lead to the performance degradation. See gitlab.com/gitlab-com/gl-infra/production/issues/832 for more details.

May 17, 2019 09:58 UTC

[Investigating] We are continuing to investigate the performance issues. See gitlab.com/gitlab-com/gl-infra/production/issues/832 for more details.

May 17, 2019 08:20 UTC

[Investigating] We are investigating performance issues. See gitlab.com/gitlab-com/gl-infra/production/issues/832 for more details.

April 2019

Scheduled jobs not triggering

April 26, 2019 23:44 UTC

Scheduled jobs not triggeringDegraded Performance

Incident Status

Degraded Performance

Components

Website, API, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, Background Processing, Support Services, packages.gitlab.com

Locations

Google Compute Engine, Digital Ocean, Zendesk, AWS

---

April 26, 2019 23:44 UTC

[Resolved] Scheduled jobs are now getting triggered as expected. Please check: gitlab.com/gitlab-com/gl-infra/production/issues/805 for further details.

April 26, 2019 07:48 UTC

[Investigating] We are investigating an issue with scheduled jobs not getting triggered. Please follow: gitlab.com/gitlab-com/gl-infra/production/issues/805 for further details and investigation.

Degraded service availability

April 24, 2019 16:21 UTC

Degraded service availability Partial Service Disruption

Incident Status

Partial Service Disruption

Components

Website, API, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, Background Processing, Support Services, packages.gitlab.com

Locations

Google Compute Engine, Digital Ocean, Zendesk, AWS

---

April 24, 2019 16:21 UTC

[Resolved] Our cloud provider resolved the underlying inconsistency within their infrastructure 3h ago and we started our remaining job processor as of 30min ago. We are not seeing any further issue. Details: gitlab.com/gitlab-com/gl-infra/production/issues/802

April 24, 2019 11:32 UTC

[Monitoring] The jobs that have been stuck are all caught up and processed now. We are monitoring the issue on our end while we wait to get further update from our cloud provider. For details: gitlab.com/gitlab-com/gl-infra/production/issues/802

April 24, 2019 08:36 UTC

[Identified] We believe we have a good lead on what might be happening and waiting to hear back from our provider for an update. The error rates have dropped down drastically and users should be seeing improvements. Details: gitlab.com/gitlab-com/gl-infra/production/issues/802

April 24, 2019 06:40 UTC

[Investigating] We are investigating an issue within our infrastructure that is causing a degraded service availability. Current known symptoms users might see are intermittent error 500s when trying to do certain operations that involve DB writes. Follow: gitlab.com/gitlab-com/gl-infra/production/issues/802 for details.

March 2019

ssh connections - IP blocked

March 20, 2019 15:51 UTC

ssh connections - IP blockedOperational

Incident Status

Operational

Components

Website, API, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, Background Processing, Support Services, packages.gitlab.com

Locations

Google Compute Engine, Digital Ocean, Zendesk, AWS

---

March 20, 2019 15:51 UTC

[Resolved] We just blocked an IP that was hogging ssh connections. We apologize to anyone who may have had recent trouble executing git commands over ssh!

GitLab.com not responding

March 18, 2019 22:58 UTC

GitLab.com not respondingService Disruption

Incident Status

Service Disruption

Components

Website, API, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, Background Processing, Support Services, packages.gitlab.com

Locations

Google Compute Engine, Digital Ocean, Zendesk, AWS

---

March 18, 2019 22:58 UTC

[Resolved] GitLab.com has been operating normally with the temporary remediations in place. RCA will be on gitlab.com/gitlab-com/gl-infra/infrastructure/issues/6407.

March 18, 2019 20:53 UTC

[Monitoring] GitLab.com is again operating normally and we will continue to monitor while we patch the affected area. Thanks for bearing with us!

March 18, 2019 20:35 UTC

[Identified] We have identified the source of the issues with the requests and we are tracking on gitlab.com/gitlab-com/gl-infra/production/issues/735

March 18, 2019 20:23 UTC

[Investigating] We are investigating slow queries on our database which appear to be related to the higher error rates and slow requests.

March 18, 2019 20:05 UTC

[Investigating] We are currently investigating elevated error rates on GitLab.com web and api requests.