All Systems Operational

Updated a few seconds ago

Back to current status

Status History

Filter: Azure (Clear)



August 2019

Service degradation and high latency after database failover

August 14, 2019 18:05 UTC

Incident Status

Operational


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com


Locations

Google Compute Engine, Azure, Zendesk, AWS




August 14, 2019 18:05 UTC
[Resolved] Issue closed.

August 14, 2019 15:43 UTC
[Monitoring] We are still monitoring GitLab.com. API and web requests latencies are still good and CI pending queues continue to look better.

August 14, 2019 15:15 UTC
[Monitoring] We are continuing to monitor the recovery of GitLab.com. CI pending jobs queues are continuing to recover. Web and API request latencies are back at normal levels.

August 14, 2019 14:52 UTC
[Monitoring] The failed DB node has been re-added and we are monitoring recovery. We are fixing an issue with our monitoring now before we call an all clear.

August 14, 2019 14:02 UTC
[Identified] We've repaired the failed database node and added it back into the load balancer rotation. We've also created an additional read-replica, and once it's ready we'll add it too. Latency continues to be an issue.

August 14, 2019 12:47 UTC
[Identified] We are facing degraded performance on GitLab.com. We are working to restore the failed database node. We are tracking the issue in gitlab.com/gitlab-com/gl-infra/production/issues/1054.

August 14, 2019 12:05 UTC
[Identified] We are facing errors and latency in the API requests. We are investigating our database read-replica load balancing mechanism as it is not behaving as expected to distribute the load when a node leaves the rotation. We are tracking the issue in gitlab.com/gitlab-com/gl-infra/production/issues/1054.

August 14, 2019 11:30 UTC
[Identified] We are investigating our database read-replica load balancing mechanism as it is not behaving as expected to distribute the load when a node leaves the rotation. We are tracking the issue in gitlab.com/gitlab-com/gl-infra/production/issues/1054.

August 14, 2019 10:49 UTC
[Identified] At the moment we found performance problems with another database read-only node, we remove it from the cluster. We are tracking the issue in gitlab.com/gitlab-com/gl-infra/production/issues/1054.

August 14, 2019 10:28 UTC
[Identified] We are facing errors and latency in the API requests. At the moment we are restoring the failed database node. We are tracking the issue in gitlab.com/gitlab-com/gl-infra/production/issues/1054.

August 14, 2019 09:54 UTC
[Identified] We are facing a slowdown in some requests to the database. At the moment we are restoring the failed database node. Tracking the issue in gitlab.com/gitlab-com/gl-infra/production/issues/1054

August 14, 2019 09:21 UTC
[Identified] We identified a database failover. At the moment we are mitigating the side effects. Tracking the issue in gitlab.com/gitlab-com/gl-infra/production/issues/1054

August 14, 2019 08:53 UTC
[Investigating] We had a brief downtime to most of our services, and we are currently investigating.

Degraded Performance of Web requests on GitLab.com

August 14, 2019 16:44 UTC

Incident Status

Degraded Performance


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com


Locations

Google Compute Engine, Azure, Zendesk, AWS




August 14, 2019 16:44 UTC
[Resolved] All clear. We're confident we have identified the underlying cause and will perform a thorough incident review in issue gitlab.com/gitlab-com/gl-infra/infrastructure/issues/7543. Our apologies to all impacted users. We're dedicated to making your experience on GitLab.com better!

August 14, 2019 16:29 UTC
[Monitoring] We're nearly back to normal. Job queues are at their expected depths, but we've not quite reached optimal processing speeds. We'll continue to monitor and expect to resolve the incident shortly.

August 14, 2019 16:05 UTC
[Monitoring] CI/CD job queues continue to fall. We're keeping a watchful eye on the database and jobs as levels trend back toward normal. At the current rate, we'll be operating at normal levels in ~15 minutes.

August 13, 2019 15:29 UTC
[Resolved] All systems are operating normally again. We identified a runaway CI pipeline that generated contention for writes on commonly accessed database table. The jobs in the pipeline have completed and we're investigating ways to safeguard similar actions from having this type impact on the platform. We apologies for any inconvenience you may have incurred!

August 13, 2019 15:01 UTC
[Monitoring] Web request latency on GitLab.com is back at operation levels. We'll continue to monitor. Thanks for your patience!

August 13, 2019 14:48 UTC
[Identified] We are continuing to investigate degraded performance on GitLab.com web requests. We have identified a pattern of requests that are related. We continue to track on gitlab.com/gitlab-com/gl-infra/production/issues/1052.

August 13, 2019 14:22 UTC
[Investigating] We are continuing to investigate degraded performance on web requests to GitLab.com.

August 13, 2019 13:58 UTC
[Investigating] GitLab.com saw a small dip degrading performance on web requests in the last 15 minutes. Latencies have recovered, but we are investigating. We will track on gitlab.com/gitlab-com/gl-infra/production/issues/1052.

We are experiencing degraded performance

August 9, 2019 12:59 UTC

Incident Status

Operational


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com


Locations

Google Compute Engine, Azure, Zendesk, AWS




August 9, 2019 12:59 UTC
[Resolved] Update on issue gitlab.com/gitlab-com/gl-infra/production/issues/1046. The incident is resolved.

August 9, 2019 11:56 UTC
[Identified] Update on issue gitlab.com/gitlab-com/gl-infra/production/issues/1046. We do not have any impact on users, and we are looking for improvements that will be listed on the incident report before closing.

August 9, 2019 11:31 UTC
[Investigating] Update on issue gitlab.com/gitlab-com/gl-infra/production/issues/1046. We are experiencing a slowdown in requests to the database, and we are investigating.

August 9, 2019 11:14 UTC
[Investigating] Update on issue gitlab.com/gitlab-com/gl-infra/production/issues/1046. We are experiencing a slowdown in requests to the database, and we are investigating.

August 9, 2019 10:49 UTC
[Investigating] Update on issue gitlab.com/gitlab-com/gl-infra/production/issues/1046. We are experiencing a slowdown in requests to the database, and we are investigating.

August 9, 2019 10:24 UTC
[Investigating] Update on issue gitlab.com/gitlab-com/gl-infra/production/issues/1046. We are experiencing a slowdown, and we are investigating.

August 9, 2019 10:06 UTC
[Investigating] Update on issue gitlab.com/gitlab-com/gl-infra/production/issues/1046. We are experiencing a slowdown, and we are investigating.

August 9, 2019 09:45 UTC
[Investigating] We are experiencing a slowdown and we are investigating.

Potential Service Disruptions

August 9, 2019 03:23 UTC

Description

Early morning, Friday 09 August at 02:00 UTC we will be performing maintenance to apply configuration changes on Consul agents. Hosts responsible for maintaining database connections rely on consul advertising the primary database In the event the clients lose connectivity to the Consul servers, users may see errors as a result of the lost connectivity. We expect the maintenance will be brief and that all systems will resume normal operations after only a few minutes. Please see gitlab.com/gitlab-com/gl-infra/production/issues/1042 for more details for more information.


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com


Locations

Google Compute Engine, Azure, Zendesk, AWS


Schedule

August 9, 2019 02:00 - August 9, 2019 02:05 UTC



August 9, 2019 03:23 UTC
[Update] We've completed the maintenance without incurring any disruption of services. Thanks for following along! gitlab.com/gitlab-com/gl-infra/production/issues/1042 is now closed.

August 9, 2019 02:25 UTC
[Update] Maintenance is complete. We are monitoring operations, but so far have not observed an increased rate in errors as a result of the change. It appears there was no impact to users.

August 9, 2019 02:11 UTC
[Update] We've pushed the change back 10 minutes. Any disruptions will be observed at 02:20 UTC.

August 9, 2019 02:03 UTC
[Update] We're beginning the maintenance. If service is disrupted, errors will begin to occur at 02:10 UTC.

We are investigating that some merge requests are getting closed inadvertently

August 8, 2019 18:55 UTC

Incident Status

Operational


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com


Locations

Google Compute Engine, Azure, Zendesk, AWS




August 8, 2019 18:55 UTC
[Resolved] gitlab.com/gitlab-com/gl-infra/production/issues/1042 is now closed. Please follow the linked Incident Report for additional details.

August 8, 2019 18:55 UTC
[Resolved] gitlab.com/gitlab-com/gl-infra/production/issues/1042 is now closed. Please follow the linked Incident Report for additional details.

August 8, 2019 15:25 UTC
[Investigating] Update on issue gitlab.com/gitlab-com/gl-infra/production/issues/1040. We have no more reports of errors, and we are gathering data of affected users before closing the incident report.

August 8, 2019 15:11 UTC
[Investigating] Update on issue gitlab.com/gitlab-com/gl-infra/production/issues/1040. We are investigating that some merge requests are getting closed inadvertently.

August 8, 2019 14:54 UTC
[Investigating] Update on issue gitlab.com/gitlab-com/gl-infra/production/issues/1040. We are investigating that some merge requests are getting closed inadvertently.

August 8, 2019 14:38 UTC
[Investigating] Update on issue gitlab.com/gitlab-com/gl-infra/production/issues/1040. We are investigating that some merge requests are getting closed inadvertently.

August 8, 2019 14:23 UTC
[Investigating] Update on issue gitlab.com/gitlab-com/gl-infra/production/issues/1040. We are investigating that some merge requests are getting closed inadvertently.

August 8, 2019 14:07 UTC
[Investigating] Update on issue gitlab.com/gitlab-com/gl-infra/production/issues/1040. We are investigating that some merge requests are getting closed inadvertently.

August 8, 2019 13:49 UTC
[Investigating] Update on issue gitlab.com/gitlab-com/gl-infra/production/issues/1040. We are investigating that some merge requests are getting closed inadvertently.

August 8, 2019 13:35 UTC
[Investigating] Update on issue gitlab.com/gitlab-com/gl-infra/production/issues/1040. We are investigating that some merge requests are getting closed inadvertently.

August 8, 2019 13:19 UTC
[Investigating] We are investigating that some merge requests are getting closed inadvertently

We are investigating higher latencies and durations for job processing.

August 8, 2019 15:45 UTC

Incident Status

Degraded Performance


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com


Locations

Google Compute Engine, Azure, Zendesk, AWS




August 8, 2019 15:45 UTC
[Resolved] The issue is resolved.

August 8, 2019 15:16 UTC
[Monitoring] Update on issue gitlab.com/gitlab-com/gl-infra/production/issues/1039. The problem is solved. We are monitoring the queues.

August 8, 2019 15:00 UTC
[Monitoring] Update on issue gitlab.com/gitlab-com/gl-infra/production/issues/1039. The problem is solved. We are monitoring the queues.

August 8, 2019 14:54 UTC
[Identified] Update on issue gitlab.com/gitlab-com/gl-infra/production/issues/1039. The queues are clearing. We are processing the queues.

August 8, 2019 14:37 UTC
[Identified] Update on issue gitlab.com/gitlab-com/gl-infra/production/issues/1039. The queues are clearing. We are processing the queues.

August 8, 2019 14:21 UTC
[Identified] Update on issue gitlab.com/gitlab-com/gl-infra/production/issues/1039. The queues are clearing. We are processing the queues.

August 8, 2019 14:06 UTC
[Identified] Update on issue gitlab.com/gitlab-com/gl-infra/production/issues/1039. We have found delays again, and the queues are growing.

August 8, 2019 13:48 UTC
[Identified] Update on issue gitlab.com/gitlab-com/gl-infra/production/issues/1039. We have found delays again, and the queues are growing.

August 8, 2019 13:28 UTC
[Identified] Update on issue gitlab.com/gitlab-com/gl-infra/production/issues/1039. We have found delays again, and the queues are growing.

August 8, 2019 13:20 UTC
[Identified] Update on issue gitlab.com/gitlab-com/gl-infra/production/issues/1039. We found the problem, and we are working mitigating, and the queues are clearing.

August 8, 2019 13:04 UTC
[Identified] Update on issue gitlab.com/gitlab-com/gl-infra/production/issues/1039. We found the problem, and we are working mitigating, and the queues are clearing.

August 8, 2019 12:46 UTC
[Identified] Update on issue gitlab.com/gitlab-com/gl-infra/production/issues/1039. We found the problem, and we are working mitigating, and the queues are clearing.

August 8, 2019 12:31 UTC
[Identified] Update on issue gitlab.com/gitlab-com/gl-infra/production/issues/1039. We found the problem, and we are working mitigating, and the queues are clearing.

August 8, 2019 12:16 UTC
[Identified] Update on issue gitlab.com/gitlab-com/gl-infra/production/issues/1039. We found the problem, and we are working mitigating, and the queues are clearing.

August 8, 2019 12:00 UTC
[Identified] Update on issue gitlab.com/gitlab-com/gl-infra/production/issues/1039. We found the problem, and we are working mitigating, and the queues are clearing.

August 8, 2019 11:52 UTC
[Identified] Update on issue #1039. We found the problem, and we are working mitigating and the queues are going down.

August 8, 2019 11:29 UTC
[Identified] Update on issue #1039. We applied the hot patch, and we keep with the investigation.

August 8, 2019 11:11 UTC
[Identified] Update on issue #1039. We applied the hot patch and we keep with the investigation.

August 8, 2019 10:51 UTC
[Identified] Update on issue #1039. We are issuing a hot patch.

August 8, 2019 10:25 UTC
[Identified] Update on issue #1039. We are issuing a hot patch.

August 8, 2019 10:07 UTC
[Identified] Update on issue #1039. We are issuing a hot patch.

August 8, 2019 10:02 UTC
[Identified] We are issuing a hot patch.

August 8, 2019 09:45 UTC
[Identified] We are issuing a hot patch.

August 8, 2019 09:28 UTC
[Identified] The situation is stabilizing, and we are issuing a hot patch.

August 8, 2019 09:10 UTC
[Investigating] The situation is stabilizing, we are investigating higher latencies and durations for job processing.

August 8, 2019 08:54 UTC
[Investigating] The situation is stabilizing, we still under investigation of the root cause of the issue.

August 8, 2019 08:39 UTC
[Investigating] The situation is stabilizing, we still under investigation of the root cause of the issue.

August 8, 2019 08:15 UTC
[Investigating] We are investigating higher latencies and durations for job processing.

August 8, 2019 07:56 UTC
[Investigating] We are investigating higher latencies and durations for job processing.

August 8, 2019 07:21 UTC
[Investigating] We are investigating higher latencies and durations for job processing.

We are executing an update on the backend of the platform

August 7, 2019 21:50 UTC

Incident Status

Operational


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com


Locations

Google Compute Engine, Azure, Zendesk, AWS




August 7, 2019 21:50 UTC
[Resolved] IncidentReview now available at: gitlab.com/gitlab-com/gl-infra/infrastructure/issues/7468

August 7, 2019 19:47 UTC
[Identified] We are preparing the final procedure to execute in the production environment.

August 7, 2019 19:06 UTC
[Identified] We are executing the final procedure in the staging environment.

August 7, 2019 17:50 UTC
[Identified] We are executing tests in the staging environment.

August 7, 2019 17:30 UTC
[Identified] We are executing a review of the upgrade that we will perform on production.

August 7, 2019 17:13 UTC
[Identified] We are executing a review of the upgrade that we will perform on production.

August 7, 2019 16:21 UTC
[Investigating] We are under investigation of the best approach to renew the certificates.

August 7, 2019 16:02 UTC
[Investigating] Status on the incident #1037, we are updating certificates on backend and expect no impact in production

behaviour change in new docker version

August 6, 2019 12:11 UTC

Incident Status

Operational


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com


Locations

Google Compute Engine, Azure, Zendesk, AWS




August 6, 2019 12:11 UTC
[Resolved] We are closing this incident which was kept open for informational purposes.

August 2, 2019 10:15 UTC
[Identified] A small number of users are reporting runner issues who are using docker:dind. If you are seeing errors with jobs that use the docker:dind image please see gitlab.com/gitlab-com/gl-infra/production/issues/1023 for how to workaround the issue in your CI configuration.

Database Maintenance

August 5, 2019 23:23 UTC

Database MaintenancePlanned Maintenance

Description

We are executing maintenance in the database disks, no services will be impacted.


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com


Locations

Google Compute Engine, Azure, Zendesk, AWS


Schedule

August 5, 2019 22:00 - August 5, 2019 23:03 UTC



August 5, 2019 23:23 UTC
[Update] The maintenance has been completed successfully.

August 5, 2019 23:04 UTC
[Update] Maintenance on the databases disks. No systems will be impacted.

We are experiencing degraded performance

August 5, 2019 15:03 UTC

Incident Status

Operational


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com


Locations

Google Compute Engine, Azure, Zendesk, AWS




August 5, 2019 15:03 UTC
[Resolved] All waiting notifications have been send out and queues are back to zero.

August 5, 2019 14:34 UTC
[Monitoring] The issue is resolved. We are monitoring.

August 5, 2019 14:19 UTC
[Investigating] We are under investigation of the issue, that might generate delay to receive notifications for issues and merge requests.

August 5, 2019 13:51 UTC
[Investigating] We are under investigation of the issue, that might generate delay to receive notifications for issues and merge requests.

August 5, 2019 13:17 UTC
[Investigating] We are experiencing degraded performance, that might generate delay to receive notifications and execute merge requests.

July 2019

job queue takes longer to process jobs

July 31, 2019 16:39 UTC

Incident Status

Degraded Performance


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com


Locations

Google Compute Engine, Azure, Zendesk, AWS




July 31, 2019 16:39 UTC
[Resolved] Pipeline queues are back to normal. We apologize for any delays you may have experienced with your CI jobs.

July 31, 2019 16:13 UTC
[Monitoring] The pipeline_processing queue is back to normal levels and we will continue to monitor levels.

July 31, 2019 15:47 UTC
[Identified] We've identified the slowdown and the queue is falling. We are tuning settings to relieve back pressure. Additional updates at gitlab.com/gitlab-com/gl-infra/production/issues/1014

July 31, 2019 14:45 UTC
[Investigating] We are investigating an issue with a job queue taking longer to process jobs.

We are experiencing degraded performance

July 31, 2019 11:56 UTC

Incident Status

Degraded Performance


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com


Locations

Google Compute Engine, Azure, Zendesk, AWS




July 31, 2019 11:56 UTC
[Resolved] The degraded performance issue is resolved.

July 31, 2019 11:19 UTC
[Investigating] We are still under investigation.

July 31, 2019 10:48 UTC
[Investigating] We are experiencing load problems on one of our Git file server backends. This may lead to latency issues for some users. We are investigating….

Delay in reply-to-email processing on GitLab.com 2019-07-24

July 25, 2019 00:17 UTC

Incident Status

Operational


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com


Locations

Google Compute Engine, Azure, Zendesk, AWS




July 25, 2019 00:17 UTC
[Resolved] Queues for email processing have cleared and are now normal.

July 25, 2019 00:11 UTC
[Monitoring] We found a delay in Service Desk and reply-to-email processing on GitLab.com today on July 24 that is now resolved. Tracking is on issue: gitlab.com/gitlab-com/gl-infra/production/issues/992.

Database Failover

July 12, 2019 00:06 UTC

Database FailoverPlanned Maintenance

Description

This evening we'll be failing over our primary database. Any long running queries exceeding are unlikely to finish and this could result in API errors or partial loading of web pages. We're taking every precaution to ensure minimal disruption and we will notify you here once the maintenance is complete.


Components

Website, API, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com


Locations

Google Compute Engine, Azure, Zendesk, AWS


Schedule

July 11, 2019 23:00 - July 11, 2019 23:15 UTC



July 12, 2019 00:06 UTC
[Update] The primary is stable and the maintenance is complete.

July 11, 2019 23:12 UTC
[Update] The database gracefully failed over. We're currently monitoring the health of the new primary.

Database Replica Replacement

July 11, 2019 12:07 UTC

Incident Status

Operational


Components

Website, API, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com


Locations

Google Compute Engine, Azure, Digital Ocean, Zendesk, AWS




July 11, 2019 12:07 UTC
[Resolved] Replica created.

July 11, 2019 11:08 UTC
[Identified] We are under maintenance for a secondary database, due replication issue. No users impacted.

We are currently investigating delayed execution of mirror jobs.

July 2, 2019 17:06 UTC

Incident Status

Degraded Performance


Components

Website, API, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com


Locations

Google Compute Engine, Azure, Digital Ocean, Zendesk, AWS




July 2, 2019 17:06 UTC
[Resolved] We're all caught up with the delayed execution of pull mirror jobs. dashboards.gitlab.com/d/_MKRXrSmk/pull-mirrors?orgId=1&refresh=30s. We apologize for the inconvenience. The original issue has been moved into our production tracker. Ongoing conversation will take place there - gitlab.com/gitlab-com/gl-infra/production/issues/934.

July 2, 2019 17:06 UTC
[Resolved] All pull mirrors have been processed and the queue is back to normal operations since 4pm UTC.

July 2, 2019 15:05 UTC
[Investigating] We are currently investigating delayed execution of repository mirror jobs. See gitlab.com/gitlab-com/gl-infra/infrastructure/issues/7155 for details.

June 2019

increased delays for CI jobs

June 27, 2019 10:12 UTC

increased delays for CI jobsDegraded Performance

Incident Status

Degraded Performance


Components

Website, API, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com


Locations

Google Compute Engine, Azure, Digital Ocean, Zendesk, AWS




June 27, 2019 10:12 UTC
[Resolved] CI jobs are not delayed anymore since 6/26 14:00 UTC.

June 26, 2019 14:43 UTC
[Investigating] CI pipeline delays are improving now. We are still investigating the root cause. See gitlab.com/gitlab-com/gl-infra/production/issues/922 for details.

June 26, 2019 14:10 UTC
[Investigating] We are investigating delays for jobs in CI pipelines.

git operations over https are slow

June 20, 2019 19:12 UTC

Incident Status

Degraded Performance


Components

Website, API, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com


Locations

Google Compute Engine, Azure, Digital Ocean, Zendesk, AWS




June 20, 2019 19:12 UTC
[Resolved] We're operating normally again. We are continuing the search for a root cause analysis and will update in gitlab.com/gitlab-com/gl-infra/production/issues/912.

June 20, 2019 17:52 UTC
[Monitoring] We were unable to determine the root cause of the problem, but we've seen the latencies return to normal levels. We will continue monitoring for spikes and we'll be carefully listening for user reports.

June 20, 2019 17:04 UTC
[Investigating] We are still investigating the git access slowdowns for a limited number of repositories.

June 20, 2019 16:31 UTC
[Investigating] We've narrowed the impact to specific projects, including gitlab-ce. The majority of users are not impacted by this issue.

June 20, 2019 16:14 UTC
[Investigating] We've confirmed the latency issues, but we're still investigating. Thanks for your patience.

June 20, 2019 15:49 UTC
[Investigating] We're investigating reports of slow git operations over https connections.

Issue with trials on customers.gitlab.com

June 14, 2019 05:38 UTC

Incident Status

Partial Service Disruption


Components

GitLab Customers Portal


Locations

Azure




June 14, 2019 05:38 UTC
[Resolved] Thanks for your patience. We've resolved the issue with users being unable to initiate Gold trials on GitLab.com. Details can be found in gitlab.com/gitlab-com/gl-infra/production/issues/897.

June 13, 2019 22:22 UTC
[Identified] We are aware of the issues creating Gold trials on GitLab.com. Please contact support at support.gitlab.com in the interim to initiate a trial.

August 2018

Maintenance for GCP Migration August 11

August 11, 2018 13:01 UTC

Description

On Saturday August 11, 2018, we will be performing maintenance to do our GCP migration. The window will start at 10:00 UTC and last for 2 hours.


Components

Website, API, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, Background Processing


Locations

Google Compute Engine, Azure, Digital Ocean


Schedule

August 11, 2018 10:00 - August 11, 2018 12:00 UTC



August 11, 2018 13:01 UTC
[Update] Our maintenance window is complete. GitLab.com is running from GCP! We will continue to monitor systems over the weekend.

August 11, 2018 12:29 UTC
[Update] GitLab.com planned maintenance for migration to @GCPcloud is almost complete. GitLab.com is available although we're continuing to verify that all systems are functioning correctly.

August 11, 2018 12:02 UTC
[Update] Maintenance update 6- Internal QA phase 1 passed. We are doing final configuration changes before making GitLab.com available. We will continue to verify that all systems are functioning correctly.

August 11, 2018 11:43 UTC
[Update] Maintenance update 5- Internal QA is in progress. We will do a final confidence check soon. Given the current timings, we may go for another 30 minutes to 12:30 UTC for this maintenance window.

August 11, 2018 11:23 UTC
[Update] Maintenance update 4- updating GitLab config in GCP is nearly done. Our internal QA which will start soon.

August 11, 2018 11:02 UTC
[Update] Maintenance update 3- we are promoting the GCP database and updating GitLab config in GCP to prep for our internal QA which should start soon.

August 11, 2018 10:41 UTC
[Update] Maintenance update 2- GCP side is up to date from Azure. Starting reconfig to point to GCP prod soon. GitLab.com will remain down while we verify our new production setup.

August 11, 2018 10:20 UTC
[Update] Maintenance update - we have stopped access to GItLab.com and are making sure all items and data are synchronized and good in GCP.

August 11, 2018 10:01 UTC
[Update] GCP Switchover maintenance is starting. Working doc: docs.google.com/document/d/1CzkieGnqJStAh3pMwgg-v62-HTpBdFDoP9smGaFE1Ko





Back to current status