All Systems Operational

Updated a few seconds ago

Back to current status

Status History

Filter: Zendesk (Clear)



November 2019

Increased latencies on gitlab.com

November 18, 2019 11:51 UTC

Incident Status

Degraded Performance


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com


Locations

Google Compute Engine, Azure, Digital Ocean, Zendesk, AWS




November 18, 2019 11:51 UTC
[Resolved] We've alleviated the congestion around web database connections. See gitlab.com/gitlab-com/gl-infra/production/issues/1373 for more details.

November 18, 2019 10:59 UTC
[Monitoring] We've rebalanced a database configuration to favor web and API connections, which are the most latency sensitive to our users. We've seen an immediate improvement and we're monitoring closely.

November 18, 2019 09:53 UTC
[Investigating] We are seeing increased latencies on GitLab.com. Investigation of the issues is taking place in gitlab.com/gitlab-com/gl-infra/production/issues/1373.

Higher response times and error rates

November 6, 2019 16:04 UTC

Incident Status

Degraded Performance


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com


Locations

Google Compute Engine, Azure, Digital Ocean, Zendesk, AWS




November 6, 2019 16:04 UTC
[Resolved] We've resolved the issue and will conduct a full review in :gitlab.com/gitlab-com/gl-infra/infrastructure/issues/8368.

November 6, 2019 15:20 UTC
[Monitoring] GitLab.com is currently operational, but we're monitoring the environment closely. A project import with a number of LFS objects in their repo was creating high latencies in the import queue after it created slow database queries. We hope to resolve the issue shortly.

November 6, 2019 13:51 UTC
[Identified] We are continuing to investigate disruptions on GitLab.com. We are tracking problems on gitlab.com/gitlab-com/gl-infra/production/issues/1327. GitLab.com is currently up, but we are continuing to monitor its health.

November 6, 2019 13:33 UTC
[Investigating] We are intermittently unavailable and we are investigating the cause. We are tracking on gitlab.com/gitlab-com/gl-infra/production/issues/1327

November 6, 2019 13:29 UTC
[Investigating] We are intermittently unavailable and we are investigating the cause. We are tracking on gitlab.com/gitlab-com/gl-infra/production/issues/1327

November 6, 2019 11:37 UTC
[Monitoring] We have identified and disabled a feature flag that was possibly related to the slower requests. We are tracking on gitlab.com/gitlab-com/gl-infra/production/issues/1327.

November 6, 2019 11:03 UTC
[Investigating] We are experiencing higher response times and error rates at the moment and are investigating the root cause in gitlab.com/gitlab-com/gl-infra/production/issues/1327.

October 2019

Database Failover

October 29, 2019 11:05 UTC

Incident Status

Operational


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com


Locations

Google Compute Engine, Azure, Digital Ocean, Zendesk, AWS




October 29, 2019 11:05 UTC
[Resolved] We confirmed all services operating normally.

October 29, 2019 09:54 UTC
[Monitoring] We experienced a database failover leading to a short spike of errors on GitLab.com. The situation is back to normal and we are further investigating in gitlab.com/gitlab-com/gl-infra/production/issues/1285.

Delays in job processing

October 25, 2019 14:04 UTC

Delays in job processingDegraded Performance

Incident Status

Degraded Performance


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com


Locations

Google Compute Engine, Azure, Digital Ocean, Zendesk, AWS




October 25, 2019 14:04 UTC
[Resolved] A patch was pushed yesterday evening to fix the root cause of the issue. See gitlab.com/gitlab-org/gitlab/commit/b4037524908171800e92d72a4f12eca5ce5e7972. CI shared runners are operational.

October 24, 2019 23:16 UTC
[Monitoring] We've cleared out another problematic build that caused a resurgence in the issue and are applying a patch to fix the underlying problem. Details in: gitlab.com/gitlab-org/gitlab/issues/34860 and gitlab.com/gitlab-org/gitlab/merge_requests/19124

October 24, 2019 20:41 UTC
[Monitoring] We're seeing vast improvements in job queue times for Shared Runners on GitLab.com. Service levels are nearing normal operation and we're now monitoring to ensure the issue does not recur.

October 24, 2019 19:18 UTC
[Identified] We are still seeing issues with job queue processing and are continuing to work towards getting the matter fully resolved. Tracking in gitlab.com/gitlab-com/gl-infra/production/issues/1275.

October 24, 2019 17:52 UTC
[Resolved] CI jobs On shared runners are fully operational again. We apologize for any delays you may have experienced.

October 24, 2019 15:34 UTC
[Monitoring] Shared runner CI jobs are starting and our queues are slowly coming down. We expect to achieve normal levels within 90 minutes. We'll continue to monitor and will update once we're fully operational again.

October 24, 2019 14:46 UTC
[Identified] We've identified an issue where malformed data from a project import began throwing errors and is preventing some CI pipelines from starting. We've canceled the pipelines in question and are monitoring metrics.

October 24, 2019 12:59 UTC
[Investigating] The job durations are still higher than usual. We are continuing to investigate the situation.

October 24, 2019 12:39 UTC
[Monitoring] Jobs duration times are looking good again. We are still monitoring and investigating the root cause of the durations in gitlab.com/gitlab-com/gl-infra/production/issues/1275.

October 24, 2019 11:31 UTC
[Investigating] We are currently seeing delays in CI job processing and are investigating.

gitlab.com outage

October 23, 2019 17:05 UTC

gitlab.com outageService Disruption

Incident Status

Service Disruption


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com


Locations

Google Compute Engine, Azure, Digital Ocean, Zendesk, AWS




October 23, 2019 17:05 UTC
[Resolved] The incident is resolved. We are conducting our review in gitlab.com/gitlab-com/gl-infra/infrastructure/issues/8247.

October 23, 2019 16:02 UTC
[Monitoring] We've alleviated the memory pressure on our Redis cluster and we'll be monitoring for the next hour before sounding the all clear. All systems are operating normally.

October 23, 2019 13:30 UTC
[Identified] We confirmed the issues were caused by failures with our Redis cluster. We observed unusual activity that contributed to OOM errors on Redis. We'll be continuing to report our findings in an incident review issue: gitlab.com/gitlab-com/gl-infra/infrastructure/issues/8247.

October 23, 2019 12:17 UTC
[Investigating] While the site is up again, we are investigating problems with our redis cluster as the root cause.

October 23, 2019 11:56 UTC
[Resolved] The site is flapping again. We are investigating the root cause in gitlab.com/gitlab-com/gl-infra/production/issues/1272.

October 23, 2019 11:39 UTC
[Investigating] The site is up again. We are still checking for the root cause of the short outage.

October 23, 2019 11:35 UTC
[Investigating] We are experiencing an outage of gitlab.com and are investigating the root cause.

September 2019

Elevated Error Rates on GitLab.com

September 20, 2019 11:23 UTC

Incident Status

Operational


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com


Locations

Google Compute Engine, Azure, Zendesk, AWS




September 20, 2019 11:23 UTC
[Resolved] A fix to gitaly was made in gitlab.com/gitlab-org/gitaly/merge_requests/1492

September 18, 2019 09:07 UTC
[Monitoring] A single Gitaly file-server on GitLab.com went down briefly, leading to a momentary spike in errors. Service has been restored, but we are investigating the cause. gitlab.com/gitlab-com/gl-infra/production/issues/1165

August 2019

Partial degradation on the performance of git repositories

August 29, 2019 13:58 UTC

Incident Status

Degraded Performance


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com


Locations

Google Compute Engine, Azure, Zendesk, AWS




August 29, 2019 13:58 UTC
[Resolved] The partial degradation performance has been resolved. Thank you for your patience!

August 29, 2019 13:45 UTC
[Identified] We have identified the abuse pattern, and we are executing the corrective actions. We are tracking on gitlab.com/gitlab-com/gl-infra/production/issues/1099

August 29, 2019 13:00 UTC
[Investigating] We are observing some performance degradation on one storage node, due to possible abuse activity.

Degraded Performance of Web requests on GitLab.com

August 20, 2019 16:24 UTC

Incident Status

Degraded Performance


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com


Locations

Google Compute Engine, Azure, Zendesk, AWS




August 20, 2019 16:24 UTC
[Resolved] GitLab.com web and API request latencies are back at normal levels.

August 20, 2019 15:16 UTC
[Monitoring] GitLab.com web and API request latencies are back at normal levels. We'll continue to monitor the health of the requests as the day continues.

August 20, 2019 14:58 UTC
[Identified] GitLab.com requests are still slightly degraded. We are continuing to validate ideas on the one affected read-only database and it's pgbouncer cpu usage.

August 20, 2019 14:39 UTC
[Identified] We are continuing to test ideas to improve performance the 1 readonly DB node that is slowing web and API requests on GitLab.com.

August 20, 2019 14:19 UTC
[Identified] GitLab.com is still slightly degraded and we continue to investigate with notes on gitlab.com/gitlab-com/gl-infra/production/issues/1073

August 20, 2019 13:59 UTC
[Identified] We are investigating increased queued connections on one of our readonly databases and are continuing investigation.

August 20, 2019 13:40 UTC
[Investigating] We are continuing to investigate degraded web performance on GitLab.com. Tracking on gitlab.com/gitlab-com/gl-infra/production/issues/1073

August 20, 2019 13:25 UTC
[Investigating] We are investigating degraded performance on web requests to GitLab.com.

Service degradation and high latency after database failover

August 14, 2019 18:05 UTC

Incident Status

Operational


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com


Locations

Google Compute Engine, Azure, Zendesk, AWS




August 14, 2019 18:05 UTC
[Resolved] Issue closed.

August 14, 2019 15:43 UTC
[Monitoring] We are still monitoring GitLab.com. API and web requests latencies are still good and CI pending queues continue to look better.

August 14, 2019 15:15 UTC
[Monitoring] We are continuing to monitor the recovery of GitLab.com. CI pending jobs queues are continuing to recover. Web and API request latencies are back at normal levels.

August 14, 2019 14:52 UTC
[Monitoring] The failed DB node has been re-added and we are monitoring recovery. We are fixing an issue with our monitoring now before we call an all clear.

August 14, 2019 14:02 UTC
[Identified] We've repaired the failed database node and added it back into the load balancer rotation. We've also created an additional read-replica, and once it's ready we'll add it too. Latency continues to be an issue.

August 14, 2019 12:47 UTC
[Identified] We are facing degraded performance on GitLab.com. We are working to restore the failed database node. We are tracking the issue in gitlab.com/gitlab-com/gl-infra/production/issues/1054.

August 14, 2019 12:05 UTC
[Identified] We are facing errors and latency in the API requests. We are investigating our database read-replica load balancing mechanism as it is not behaving as expected to distribute the load when a node leaves the rotation. We are tracking the issue in gitlab.com/gitlab-com/gl-infra/production/issues/1054.

August 14, 2019 11:30 UTC
[Identified] We are investigating our database read-replica load balancing mechanism as it is not behaving as expected to distribute the load when a node leaves the rotation. We are tracking the issue in gitlab.com/gitlab-com/gl-infra/production/issues/1054.

August 14, 2019 10:49 UTC
[Identified] At the moment we found performance problems with another database read-only node, we remove it from the cluster. We are tracking the issue in gitlab.com/gitlab-com/gl-infra/production/issues/1054.

August 14, 2019 10:28 UTC
[Identified] We are facing errors and latency in the API requests. At the moment we are restoring the failed database node. We are tracking the issue in gitlab.com/gitlab-com/gl-infra/production/issues/1054.

August 14, 2019 09:54 UTC
[Identified] We are facing a slowdown in some requests to the database. At the moment we are restoring the failed database node. Tracking the issue in gitlab.com/gitlab-com/gl-infra/production/issues/1054

August 14, 2019 09:21 UTC
[Identified] We identified a database failover. At the moment we are mitigating the side effects. Tracking the issue in gitlab.com/gitlab-com/gl-infra/production/issues/1054

August 14, 2019 08:53 UTC
[Investigating] We had a brief downtime to most of our services, and we are currently investigating.

Degraded Performance of Web requests on GitLab.com

August 14, 2019 16:44 UTC

Incident Status

Degraded Performance


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com


Locations

Google Compute Engine, Azure, Zendesk, AWS




August 14, 2019 16:44 UTC
[Resolved] All clear. We're confident we have identified the underlying cause and will perform a thorough incident review in issue gitlab.com/gitlab-com/gl-infra/infrastructure/issues/7543. Our apologies to all impacted users. We're dedicated to making your experience on GitLab.com better!

August 14, 2019 16:29 UTC
[Monitoring] We're nearly back to normal. Job queues are at their expected depths, but we've not quite reached optimal processing speeds. We'll continue to monitor and expect to resolve the incident shortly.

August 14, 2019 16:05 UTC
[Monitoring] CI/CD job queues continue to fall. We're keeping a watchful eye on the database and jobs as levels trend back toward normal. At the current rate, we'll be operating at normal levels in ~15 minutes.

August 13, 2019 15:29 UTC
[Resolved] All systems are operating normally again. We identified a runaway CI pipeline that generated contention for writes on commonly accessed database table. The jobs in the pipeline have completed and we're investigating ways to safeguard similar actions from having this type impact on the platform. We apologies for any inconvenience you may have incurred!

August 13, 2019 15:01 UTC
[Monitoring] Web request latency on GitLab.com is back at operation levels. We'll continue to monitor. Thanks for your patience!

August 13, 2019 14:48 UTC
[Identified] We are continuing to investigate degraded performance on GitLab.com web requests. We have identified a pattern of requests that are related. We continue to track on gitlab.com/gitlab-com/gl-infra/production/issues/1052.

August 13, 2019 14:22 UTC
[Investigating] We are continuing to investigate degraded performance on web requests to GitLab.com.

August 13, 2019 13:58 UTC
[Investigating] GitLab.com saw a small dip degrading performance on web requests in the last 15 minutes. Latencies have recovered, but we are investigating. We will track on gitlab.com/gitlab-com/gl-infra/production/issues/1052.

We are experiencing degraded performance

August 9, 2019 12:59 UTC

Incident Status

Operational


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com


Locations

Google Compute Engine, Azure, Zendesk, AWS




August 9, 2019 12:59 UTC
[Resolved] Update on issue gitlab.com/gitlab-com/gl-infra/production/issues/1046. The incident is resolved.

August 9, 2019 11:56 UTC
[Identified] Update on issue gitlab.com/gitlab-com/gl-infra/production/issues/1046. We do not have any impact on users, and we are looking for improvements that will be listed on the incident report before closing.

August 9, 2019 11:31 UTC
[Investigating] Update on issue gitlab.com/gitlab-com/gl-infra/production/issues/1046. We are experiencing a slowdown in requests to the database, and we are investigating.

August 9, 2019 11:14 UTC
[Investigating] Update on issue gitlab.com/gitlab-com/gl-infra/production/issues/1046. We are experiencing a slowdown in requests to the database, and we are investigating.

August 9, 2019 10:49 UTC
[Investigating] Update on issue gitlab.com/gitlab-com/gl-infra/production/issues/1046. We are experiencing a slowdown in requests to the database, and we are investigating.

August 9, 2019 10:24 UTC
[Investigating] Update on issue gitlab.com/gitlab-com/gl-infra/production/issues/1046. We are experiencing a slowdown, and we are investigating.

August 9, 2019 10:06 UTC
[Investigating] Update on issue gitlab.com/gitlab-com/gl-infra/production/issues/1046. We are experiencing a slowdown, and we are investigating.

August 9, 2019 09:45 UTC
[Investigating] We are experiencing a slowdown and we are investigating.

Potential Service Disruptions

August 9, 2019 03:23 UTC

Description

Early morning, Friday 09 August at 02:00 UTC we will be performing maintenance to apply configuration changes on Consul agents. Hosts responsible for maintaining database connections rely on consul advertising the primary database In the event the clients lose connectivity to the Consul servers, users may see errors as a result of the lost connectivity. We expect the maintenance will be brief and that all systems will resume normal operations after only a few minutes. Please see gitlab.com/gitlab-com/gl-infra/production/issues/1042 for more details for more information.


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com


Locations

Google Compute Engine, Azure, Zendesk, AWS


Schedule

August 9, 2019 02:00 - August 9, 2019 02:05 UTC



August 9, 2019 03:23 UTC
[Update] We've completed the maintenance without incurring any disruption of services. Thanks for following along! gitlab.com/gitlab-com/gl-infra/production/issues/1042 is now closed.

August 9, 2019 02:25 UTC
[Update] Maintenance is complete. We are monitoring operations, but so far have not observed an increased rate in errors as a result of the change. It appears there was no impact to users.

August 9, 2019 02:11 UTC
[Update] We've pushed the change back 10 minutes. Any disruptions will be observed at 02:20 UTC.

August 9, 2019 02:03 UTC
[Update] We're beginning the maintenance. If service is disrupted, errors will begin to occur at 02:10 UTC.

We are investigating that some merge requests are getting closed inadvertently

August 8, 2019 18:55 UTC

Incident Status

Operational


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com


Locations

Google Compute Engine, Azure, Zendesk, AWS




August 8, 2019 18:55 UTC
[Resolved] gitlab.com/gitlab-com/gl-infra/production/issues/1042 is now closed. Please follow the linked Incident Report for additional details.

August 8, 2019 18:55 UTC
[Resolved] gitlab.com/gitlab-com/gl-infra/production/issues/1042 is now closed. Please follow the linked Incident Report for additional details.

August 8, 2019 15:25 UTC
[Investigating] Update on issue gitlab.com/gitlab-com/gl-infra/production/issues/1040. We have no more reports of errors, and we are gathering data of affected users before closing the incident report.

August 8, 2019 15:11 UTC
[Investigating] Update on issue gitlab.com/gitlab-com/gl-infra/production/issues/1040. We are investigating that some merge requests are getting closed inadvertently.

August 8, 2019 14:54 UTC
[Investigating] Update on issue gitlab.com/gitlab-com/gl-infra/production/issues/1040. We are investigating that some merge requests are getting closed inadvertently.

August 8, 2019 14:38 UTC
[Investigating] Update on issue gitlab.com/gitlab-com/gl-infra/production/issues/1040. We are investigating that some merge requests are getting closed inadvertently.

August 8, 2019 14:23 UTC
[Investigating] Update on issue gitlab.com/gitlab-com/gl-infra/production/issues/1040. We are investigating that some merge requests are getting closed inadvertently.

August 8, 2019 14:07 UTC
[Investigating] Update on issue gitlab.com/gitlab-com/gl-infra/production/issues/1040. We are investigating that some merge requests are getting closed inadvertently.

August 8, 2019 13:49 UTC
[Investigating] Update on issue gitlab.com/gitlab-com/gl-infra/production/issues/1040. We are investigating that some merge requests are getting closed inadvertently.

August 8, 2019 13:35 UTC
[Investigating] Update on issue gitlab.com/gitlab-com/gl-infra/production/issues/1040. We are investigating that some merge requests are getting closed inadvertently.

August 8, 2019 13:19 UTC
[Investigating] We are investigating that some merge requests are getting closed inadvertently

We are investigating higher latencies and durations for job processing.

August 8, 2019 15:45 UTC

Incident Status

Degraded Performance


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com


Locations

Google Compute Engine, Azure, Zendesk, AWS




August 8, 2019 15:45 UTC
[Resolved] The issue is resolved.

August 8, 2019 15:16 UTC
[Monitoring] Update on issue gitlab.com/gitlab-com/gl-infra/production/issues/1039. The problem is solved. We are monitoring the queues.

August 8, 2019 15:00 UTC
[Monitoring] Update on issue gitlab.com/gitlab-com/gl-infra/production/issues/1039. The problem is solved. We are monitoring the queues.

August 8, 2019 14:54 UTC
[Identified] Update on issue gitlab.com/gitlab-com/gl-infra/production/issues/1039. The queues are clearing. We are processing the queues.

August 8, 2019 14:37 UTC
[Identified] Update on issue gitlab.com/gitlab-com/gl-infra/production/issues/1039. The queues are clearing. We are processing the queues.

August 8, 2019 14:21 UTC
[Identified] Update on issue gitlab.com/gitlab-com/gl-infra/production/issues/1039. The queues are clearing. We are processing the queues.

August 8, 2019 14:06 UTC
[Identified] Update on issue gitlab.com/gitlab-com/gl-infra/production/issues/1039. We have found delays again, and the queues are growing.

August 8, 2019 13:48 UTC
[Identified] Update on issue gitlab.com/gitlab-com/gl-infra/production/issues/1039. We have found delays again, and the queues are growing.

August 8, 2019 13:28 UTC
[Identified] Update on issue gitlab.com/gitlab-com/gl-infra/production/issues/1039. We have found delays again, and the queues are growing.

August 8, 2019 13:20 UTC
[Identified] Update on issue gitlab.com/gitlab-com/gl-infra/production/issues/1039. We found the problem, and we are working mitigating, and the queues are clearing.

August 8, 2019 13:04 UTC
[Identified] Update on issue gitlab.com/gitlab-com/gl-infra/production/issues/1039. We found the problem, and we are working mitigating, and the queues are clearing.

August 8, 2019 12:46 UTC
[Identified] Update on issue gitlab.com/gitlab-com/gl-infra/production/issues/1039. We found the problem, and we are working mitigating, and the queues are clearing.

August 8, 2019 12:31 UTC
[Identified] Update on issue gitlab.com/gitlab-com/gl-infra/production/issues/1039. We found the problem, and we are working mitigating, and the queues are clearing.

August 8, 2019 12:16 UTC
[Identified] Update on issue gitlab.com/gitlab-com/gl-infra/production/issues/1039. We found the problem, and we are working mitigating, and the queues are clearing.

August 8, 2019 12:00 UTC
[Identified] Update on issue gitlab.com/gitlab-com/gl-infra/production/issues/1039. We found the problem, and we are working mitigating, and the queues are clearing.

August 8, 2019 11:52 UTC
[Identified] Update on issue #1039. We found the problem, and we are working mitigating and the queues are going down.

August 8, 2019 11:29 UTC
[Identified] Update on issue #1039. We applied the hot patch, and we keep with the investigation.

August 8, 2019 11:11 UTC
[Identified] Update on issue #1039. We applied the hot patch and we keep with the investigation.

August 8, 2019 10:51 UTC
[Identified] Update on issue #1039. We are issuing a hot patch.

August 8, 2019 10:25 UTC
[Identified] Update on issue #1039. We are issuing a hot patch.

August 8, 2019 10:07 UTC
[Identified] Update on issue #1039. We are issuing a hot patch.

August 8, 2019 10:02 UTC
[Identified] We are issuing a hot patch.

August 8, 2019 09:45 UTC
[Identified] We are issuing a hot patch.

August 8, 2019 09:28 UTC
[Identified] The situation is stabilizing, and we are issuing a hot patch.

August 8, 2019 09:10 UTC
[Investigating] The situation is stabilizing, we are investigating higher latencies and durations for job processing.

August 8, 2019 08:54 UTC
[Investigating] The situation is stabilizing, we still under investigation of the root cause of the issue.

August 8, 2019 08:39 UTC
[Investigating] The situation is stabilizing, we still under investigation of the root cause of the issue.

August 8, 2019 08:15 UTC
[Investigating] We are investigating higher latencies and durations for job processing.

August 8, 2019 07:56 UTC
[Investigating] We are investigating higher latencies and durations for job processing.

August 8, 2019 07:21 UTC
[Investigating] We are investigating higher latencies and durations for job processing.

We are executing an update on the backend of the platform

August 7, 2019 21:50 UTC

Incident Status

Operational


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com


Locations

Google Compute Engine, Azure, Zendesk, AWS




August 7, 2019 21:50 UTC
[Resolved] IncidentReview now available at: gitlab.com/gitlab-com/gl-infra/infrastructure/issues/7468

August 7, 2019 19:47 UTC
[Identified] We are preparing the final procedure to execute in the production environment.

August 7, 2019 19:06 UTC
[Identified] We are executing the final procedure in the staging environment.

August 7, 2019 17:50 UTC
[Identified] We are executing tests in the staging environment.

August 7, 2019 17:30 UTC
[Identified] We are executing a review of the upgrade that we will perform on production.

August 7, 2019 17:13 UTC
[Identified] We are executing a review of the upgrade that we will perform on production.

August 7, 2019 16:21 UTC
[Investigating] We are under investigation of the best approach to renew the certificates.

August 7, 2019 16:02 UTC
[Investigating] Status on the incident #1037, we are updating certificates on backend and expect no impact in production

behaviour change in new docker version

August 6, 2019 12:11 UTC

Incident Status

Operational


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com


Locations

Google Compute Engine, Azure, Zendesk, AWS




August 6, 2019 12:11 UTC
[Resolved] We are closing this incident which was kept open for informational purposes.

August 2, 2019 10:15 UTC
[Identified] A small number of users are reporting runner issues who are using docker:dind. If you are seeing errors with jobs that use the docker:dind image please see gitlab.com/gitlab-com/gl-infra/production/issues/1023 for how to workaround the issue in your CI configuration.

Database Maintenance

August 5, 2019 23:23 UTC

Database MaintenancePlanned Maintenance

Description

We are executing maintenance in the database disks, no services will be impacted.


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com


Locations

Google Compute Engine, Azure, Zendesk, AWS


Schedule

August 5, 2019 22:00 - August 5, 2019 23:03 UTC



August 5, 2019 23:23 UTC
[Update] The maintenance has been completed successfully.

August 5, 2019 23:04 UTC
[Update] Maintenance on the databases disks. No systems will be impacted.

We are experiencing degraded performance

August 5, 2019 15:03 UTC

Incident Status

Operational


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com


Locations

Google Compute Engine, Azure, Zendesk, AWS




August 5, 2019 15:03 UTC
[Resolved] All waiting notifications have been send out and queues are back to zero.

August 5, 2019 14:34 UTC
[Monitoring] The issue is resolved. We are monitoring.

August 5, 2019 14:19 UTC
[Investigating] We are under investigation of the issue, that might generate delay to receive notifications for issues and merge requests.

August 5, 2019 13:51 UTC
[Investigating] We are under investigation of the issue, that might generate delay to receive notifications for issues and merge requests.

August 5, 2019 13:17 UTC
[Investigating] We are experiencing degraded performance, that might generate delay to receive notifications and execute merge requests.

July 2019

job queue takes longer to process jobs

July 31, 2019 16:39 UTC

Incident Status

Degraded Performance


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com


Locations

Google Compute Engine, Azure, Zendesk, AWS




July 31, 2019 16:39 UTC
[Resolved] Pipeline queues are back to normal. We apologize for any delays you may have experienced with your CI jobs.

July 31, 2019 16:13 UTC
[Monitoring] The pipeline_processing queue is back to normal levels and we will continue to monitor levels.

July 31, 2019 15:47 UTC
[Identified] We've identified the slowdown and the queue is falling. We are tuning settings to relieve back pressure. Additional updates at gitlab.com/gitlab-com/gl-infra/production/issues/1014

July 31, 2019 14:45 UTC
[Investigating] We are investigating an issue with a job queue taking longer to process jobs.

We are experiencing degraded performance

July 31, 2019 11:56 UTC

Incident Status

Degraded Performance


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com


Locations

Google Compute Engine, Azure, Zendesk, AWS




July 31, 2019 11:56 UTC
[Resolved] The degraded performance issue is resolved.

July 31, 2019 11:19 UTC
[Investigating] We are still under investigation.

July 31, 2019 10:48 UTC
[Investigating] We are experiencing load problems on one of our Git file server backends. This may lead to latency issues for some users. We are investigating….





Back to current status