July 2019

Delay in reply-to-email processing on GitLab.com 2019-07-24

July 25, 2019 00:17 UTC

Delay in reply-to-email processing on GitLab.com 2019-07-24Operational

Incident Status

Operational

Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com

Locations

Google Compute Engine, Azure, Zendesk, AWS

---

July 25, 2019 00:17 UTC

[Resolved] Queues for email processing have cleared and are now normal.

July 25, 2019 00:11 UTC

[Monitoring] We found a delay in Service Desk and reply-to-email processing on GitLab.com today on July 24 that is now resolved. Tracking is on issue: gitlab.com/gitlab-com/gl-infra/production/issues/992.

Database Failover

July 12, 2019 00:06 UTC

Database FailoverPlanned Maintenance

Description

This evening we'll be failing over our primary database. Any long running queries exceeding are unlikely to finish and this could result in API errors or partial loading of web pages. We're taking every precaution to ensure minimal disruption and we will notify you here once the maintenance is complete.

Components

Website, API, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com

Locations

Google Compute Engine, Azure, Zendesk, AWS

Schedule

July 11, 2019 23:00 - July 11, 2019 23:15 UTC

---

July 12, 2019 00:06 UTC

[Update] The primary is stable and the maintenance is complete.

July 11, 2019 23:12 UTC

[Update] The database gracefully failed over. We're currently monitoring the health of the new primary.

Database Replica Replacement

July 11, 2019 12:07 UTC

Database Replica ReplacementOperational

Incident Status

Operational

Components

Website, API, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com

Locations

Google Compute Engine, Azure, Digital Ocean, Zendesk, AWS

---

July 11, 2019 12:07 UTC

[Resolved] Replica created.

July 11, 2019 11:08 UTC

[Identified] We are under maintenance for a secondary database, due replication issue. No users impacted.

We are currently investigating delayed execution of mirror jobs.

July 2, 2019 17:06 UTC

We are currently investigating delayed execution of mirror jobs.Degraded Performance

Incident Status

Degraded Performance

Components

Website, API, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com

Locations

Google Compute Engine, Azure, Digital Ocean, Zendesk, AWS

---

July 2, 2019 17:06 UTC

[Resolved] We're all caught up with the delayed execution of pull mirror jobs. dashboards.gitlab.com/d/_MKRXrSmk/pull-mirrors?orgId=1&refresh=30s. We apologize for the inconvenience. The original issue has been moved into our production tracker. Ongoing conversation will take place there - gitlab.com/gitlab-com/gl-infra/production/issues/934.

July 2, 2019 17:06 UTC

[Resolved] All pull mirrors have been processed and the queue is back to normal operations since 4pm UTC.

July 2, 2019 15:05 UTC

[Investigating] We are currently investigating delayed execution of repository mirror jobs. See gitlab.com/gitlab-com/gl-infra/infrastructure/issues/7155 for details.

June 2019

increased delays for CI jobs

June 27, 2019 10:12 UTC

increased delays for CI jobsDegraded Performance

Incident Status

Degraded Performance

Components

Website, API, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com

Locations

Google Compute Engine, Azure, Digital Ocean, Zendesk, AWS

---

June 27, 2019 10:12 UTC

[Resolved] CI jobs are not delayed anymore since 6/26 14:00 UTC.

June 26, 2019 14:43 UTC

[Investigating] CI pipeline delays are improving now. We are still investigating the root cause. See gitlab.com/gitlab-com/gl-infra/production/issues/922 for details.

June 26, 2019 14:10 UTC

[Investigating] We are investigating delays for jobs in CI pipelines.

git operations over https are slow

June 20, 2019 19:12 UTC

git operations over https are slowDegraded Performance

Incident Status

Degraded Performance

Components

Website, API, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com

Locations

Google Compute Engine, Azure, Digital Ocean, Zendesk, AWS

---

June 20, 2019 19:12 UTC

[Resolved] We're operating normally again. We are continuing the search for a root cause analysis and will update in gitlab.com/gitlab-com/gl-infra/production/issues/912.

June 20, 2019 17:52 UTC

[Monitoring] We were unable to determine the root cause of the problem, but we've seen the latencies return to normal levels. We will continue monitoring for spikes and we'll be carefully listening for user reports.

June 20, 2019 17:04 UTC

[Investigating] We are still investigating the git access slowdowns for a limited number of repositories.

June 20, 2019 16:31 UTC

[Investigating] We've narrowed the impact to specific projects, including gitlab-ce. The majority of users are not impacted by this issue.

June 20, 2019 16:14 UTC

[Investigating] We've confirmed the latency issues, but we're still investigating. Thanks for your patience.

June 20, 2019 15:49 UTC

[Investigating] We're investigating reports of slow git operations over https connections.

May 2019

Observing git-over-ssh errors

May 23, 2019 19:22 UTC

Observing git-over-ssh errorsOperational

Incident Status

Operational

Components

Website, API, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, Background Processing, Support Services, packages.gitlab.com

Locations

Google Compute Engine, Digital Ocean, Zendesk, AWS

---

May 23, 2019 19:22 UTC

[Resolved] We've determined the issue is no longer impacting users and are marking this issue resolved.

May 23, 2019 17:05 UTC

[Monitoring] We are continuing to monitor git over ssh performance. Response latency and error ratios are back to normal, and we've not observed any other metrics that indicate any other operations on GitLab.com were impacted.

May 23, 2019 16:29 UTC

[Investigating] We're observing increased errors on GitLab.com where git connections via ssh are unexpectedly failing. The issue is being tracked in gitlab.com/gitlab-com/gl-infra/production/issues/844.

Issues with credentials for customers previously logged in via SSO

May 22, 2019 16:05 UTC

Issues with credentials for customers previously logged in via SSO Operational

Incident Status

Operational

Components

Website, API, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, Background Processing, Support Services, packages.gitlab.com

Locations

Google Compute Engine, Digital Ocean, Zendesk, AWS

---

May 22, 2019 16:05 UTC

[Resolved] We haven't received any additional reports of issues with SSO authentication. Both login and runner credentials are operating normally, again. Our apologies to anyone who was impacted–and our thanks to the few who brought this issue to our attention!

May 22, 2019 15:15 UTC

[Monitoring] We've received customer confirmation that the configuration change we reverted has resolved their issues. We'll continue monitoring with our support team.

May 22, 2019 14:58 UTC

[Investigating] We've reverted a setting that was forcing new sessions for SSO authentication. But, we're still investigating, as we've yet to find definitive metrics indicating the issue as resolved.

May 22, 2019 14:39 UTC

[Investigating] We're currently investigating customer reports of issues with CI runners and user logins. If you're using SSO you may be impacted.

We are investigating performance issues on our site.

May 17, 2019 15:26 UTC

We are investigating performance issues on our site.Degraded Performance

Incident Status

Degraded Performance

Components

Website, API, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, Background Processing, Support Services, packages.gitlab.com

Locations

Google Compute Engine, Digital Ocean, Zendesk, AWS

---

May 17, 2019 15:26 UTC

[Resolved] The performance issues are fully resolved. And the site is fully operational.

May 17, 2019 13:10 UTC

[Monitoring] The rollback is still ongoing and the website performance is not fully recovered yet.

May 17, 2019 11:42 UTC

[Monitoring] We are seeing performance of GitLab.com going back to normal levels while the rollback is going on. See gitlab.com/gitlab-com/gl-infra/production/issues/832 for more details.

May 17, 2019 10:37 UTC

[Investigating] We are rolling back the changes that lead to the performance degradation. See gitlab.com/gitlab-com/gl-infra/production/issues/832 for more details.

May 17, 2019 09:58 UTC

[Investigating] We are continuing to investigate the performance issues. See gitlab.com/gitlab-com/gl-infra/production/issues/832 for more details.

May 17, 2019 08:20 UTC

[Investigating] We are investigating performance issues. See gitlab.com/gitlab-com/gl-infra/production/issues/832 for more details.

April 2019

Scheduled jobs not triggering

April 26, 2019 23:44 UTC

Scheduled jobs not triggeringDegraded Performance

Incident Status

Degraded Performance

Components

Website, API, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, Background Processing, Support Services, packages.gitlab.com

Locations

Google Compute Engine, Digital Ocean, Zendesk, AWS

---

April 26, 2019 23:44 UTC

[Resolved] Scheduled jobs are now getting triggered as expected. Please check: gitlab.com/gitlab-com/gl-infra/production/issues/805 for further details.

April 26, 2019 07:48 UTC

[Investigating] We are investigating an issue with scheduled jobs not getting triggered. Please follow: gitlab.com/gitlab-com/gl-infra/production/issues/805 for further details and investigation.

Degraded service availability

April 24, 2019 16:21 UTC

Degraded service availability Partial Service Disruption

Incident Status

Partial Service Disruption

Components

Website, API, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, Background Processing, Support Services, packages.gitlab.com

Locations

Google Compute Engine, Digital Ocean, Zendesk, AWS

---

April 24, 2019 16:21 UTC

[Resolved] Our cloud provider resolved the underlying inconsistency within their infrastructure 3h ago and we started our remaining job processor as of 30min ago. We are not seeing any further issue. Details: gitlab.com/gitlab-com/gl-infra/production/issues/802

April 24, 2019 11:32 UTC

[Monitoring] The jobs that have been stuck are all caught up and processed now. We are monitoring the issue on our end while we wait to get further update from our cloud provider. For details: gitlab.com/gitlab-com/gl-infra/production/issues/802

April 24, 2019 08:36 UTC

[Identified] We believe we have a good lead on what might be happening and waiting to hear back from our provider for an update. The error rates have dropped down drastically and users should be seeing improvements. Details: gitlab.com/gitlab-com/gl-infra/production/issues/802

April 24, 2019 06:40 UTC

[Investigating] We are investigating an issue within our infrastructure that is causing a degraded service availability. Current known symptoms users might see are intermittent error 500s when trying to do certain operations that involve DB writes. Follow: gitlab.com/gitlab-com/gl-infra/production/issues/802 for details.

March 2019

ssh connections - IP blocked

March 20, 2019 15:51 UTC

ssh connections - IP blockedOperational

Incident Status

Operational

Components

Website, API, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, Background Processing, Support Services, packages.gitlab.com

Locations

Google Compute Engine, Digital Ocean, Zendesk, AWS

---

March 20, 2019 15:51 UTC

[Resolved] We just blocked an IP that was hogging ssh connections. We apologize to anyone who may have had recent trouble executing git commands over ssh!

GitLab.com not responding

March 18, 2019 22:58 UTC

GitLab.com not respondingService Disruption

Incident Status

Service Disruption

Components

Website, API, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, Background Processing, Support Services, packages.gitlab.com

Locations

Google Compute Engine, Digital Ocean, Zendesk, AWS

---

March 18, 2019 22:58 UTC

[Resolved] GitLab.com has been operating normally with the temporary remediations in place. RCA will be on gitlab.com/gitlab-com/gl-infra/infrastructure/issues/6407.

March 18, 2019 20:53 UTC

[Monitoring] GitLab.com is again operating normally and we will continue to monitor while we patch the affected area. Thanks for bearing with us!

March 18, 2019 20:35 UTC

[Identified] We have identified the source of the issues with the requests and we are tracking on gitlab.com/gitlab-com/gl-infra/production/issues/735

March 18, 2019 20:23 UTC

[Investigating] We are investigating slow queries on our database which appear to be related to the higher error rates and slow requests.

March 18, 2019 20:05 UTC

[Investigating] We are currently investigating elevated error rates on GitLab.com web and api requests.

gitlab-com/gl-infra/production#733

March 18, 2019 18:44 UTC

gitlab-com/gl-infra/production#733Degraded Performance

Incident Status

Degraded Performance

Components

Website, API, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, Background Processing, Support Services, packages.gitlab.com

Locations

Google Compute Engine, Digital Ocean, Zendesk, AWS

---

March 18, 2019 18:44 UTC

[Resolved] We've stabilized and we're resolving the issue. All operations are normal.

March 18, 2019 18:24 UTC

[Monitoring] We've relieved the pressure on our build infrastructure and systems are operating normally. We will continue to monitor for stability before issuing issuing an all clear.

March 18, 2019 18:05 UTC

[Identified] We've narrowed the issue to our build infrastructure. 500 errors are still occurring, but in low numbers. Users might continue to experience issues or delays with their build pipelines.

March 18, 2019 17:34 UTC

[Investigating] We're still seeing a slightly elevated error count, but the error rate is declining. API users are those most likely to be affected by this incident. We apologize for any degraded performance and will update you again soon.

March 18, 2019 17:13 UTC

[Investigating] We are currently seeing a high rate of errors on GitLab.com. Our infrastructure team is investigating.

January 2019

Investigating slow Git interactions on GitLab.com

January 17, 2019 17:43 UTC

Investigating slow Git interactions on GitLab.comDegraded Performance

Incident Status

Degraded Performance

Components

Website, API, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, Background Processing, Support Services

Locations

Google Compute Engine, Digital Ocean, Zendesk

---

January 17, 2019 17:43 UTC

[Resolved] All GitLab.com services are operating normally again and more notes have been added to gitlab.com/gitlab-com/gl-infra/production/issues/657 related to our investigation.

January 17, 2019 17:04 UTC

[Monitoring] Error rates on GitLab.com have gone back to normal levels and all services are operating normally. We will continue to investigate and update on gitlab.com/gitlab-com/gl-infra/production/issues/657.

January 17, 2019 16:44 UTC

[Investigating] We are currently investigating slow interactions with git on GitLab.com.