All Systems Operational

Updated a few seconds ago

Back to current status

Status History

Filter: Background Processing (Clear)



November 2023

Performance issues affecting processing of MRs

November 20, 2023 18:49 UTC

Incident Status

Degraded Performance


Components

CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions, Background Processing


Locations

Google Compute Engine, AWS




November 20, 2023 18:49 UTC
[Resolved] Performance of CI pipelines and merge requests has fully returned to normal and shows no further signs of degradation. The status page is being marked resolved, and future updates can be found in the production issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/17158

November 20, 2023 18:05 UTC
[Monitoring] Performance of CI job pickup and merge requests has returned to a normal state. The incident response team will continue monitoring and reviewing the rolled back changes. Next update in an hour, unless there is anything to report prior.

November 20, 2023 17:21 UTC
[Investigating] We have started to notice some signs of recovery after the rollback. Investigation into root causes is still ongoing to identify the source of the CI and merge request performance problems.

November 20, 2023 16:50 UTC
[Investigating] Rollback to an earlier deployment has completed. No material updates in the CI job pickup slowness has been observed yet. Next update will be in 30 minutes unless there is anything to report sooner.

November 20, 2023 16:32 UTC
[Investigating] The rollback to an earlier deployment is in progress and a review of all changes is in progress to identify the root cause of slow CI job pickup. Additional support from other teams is being brought in to assist.

November 20, 2023 16:12 UTC
[Investigating] The incident response team is continuing to investigate the slow CI job pickup. A rollback to a previous deployment is in progress as a potential mitigating solution.

November 20, 2023 15:53 UTC
[Investigating] We are actively investigating an issue where CI jobs are being picked up slowly affecting performance of MR's. More details here: gitlab.com/gitlab-com/gl-infra/production/-/issues/17158

October 2023

Delays in background and CI processing

October 25, 2023 18:08 UTC

Incident Status

Degraded Performance


Components

Website, Background Processing


Locations

Google Compute Engine




October 25, 2023 18:08 UTC
[Resolved] GitLab identified problematic usage patterns that lead to DB saturation. We took action to mitigate the problems caused by these usage patterns and will be investigating how to mitigate the impact of similar usage patterns in the future. We will post further updates to the incident issue. Incident issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/17030

October 25, 2023 17:30 UTC
[Monitoring] The rollback is done and we are monitoring. Background processing should be back to normal levels. We see that CI jobs are being picked up and workers are working through the CI queues; users may see delays in jobs being picked up but they are being processed. Incident issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/17030

October 25, 2023 16:25 UTC
[Investigating] We are continuing to investigate the root cause for CI and Merge Request performance issues; the rollback has completed and we have identified a possible proximate cause for DB saturation. We have made changes to mitigate DB saturation and are monitoring to understand if our changes have been effective. Incident issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/17030

Decreased bulk importer functionality

October 6, 2023 00:23 UTC

Decreased bulk importer functionalityPartial Service Disruption

Incident Status

Partial Service Disruption


Components

Background Processing


Locations

Google Compute Engine




October 6, 2023 00:23 UTC
[Resolved] This incident has been resolved. The bulk import functionality was enabled and is now operational.

October 5, 2023 23:14 UTC
[Identified] A fix of the bulk import functionality has been implemented and we will re-enable the feature shortly.

September 22, 2023 13:59 UTC
[Identified] The bulk import functionality is still disabled as it requires further testing. The next update will be made once testing and deployment of the fix has been successfully made.

September 21, 2023 06:12 UTC
[Identified] The bulk import functionality has been disabled pending the deployment of a fix to the identified problem. The next update will made in 24 hours once deployment and testing of the fix has been successfully made.

September 21, 2023 05:31 UTC
[Investigating] We are continuing to investigate the causes of the reduced bulk import functionality, however there is no further information to share at present. A further update will be provided in 60 minutes.

September 21, 2023 05:10 UTC
[Investigating] We are seeing some issues with the bulk import functionality and we are currently investigating the cause. An update will be provided in 15 minutes.

July 2023

Performance issues on GitLab.com

July 15, 2023 04:51 UTC

Incident Status

Degraded Performance


Components

API, Git Operations, Container Registry, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners for GitLab community contributions, Background Processing, Canary


Locations

Google Compute Engine




July 15, 2023 04:51 UTC
[Resolved] This incident has been marked as resolved, as we don't see any outstanding issues with GitLab.com. A full timeline of this incident can be found in our Incident Report available at gitlab.com/gitlab-com/gl-infra/production/-/issues/16042

July 15, 2023 03:29 UTC
[Monitoring] We will update the status of this incident to `Operational` for now..

July 15, 2023 03:26 UTC
[Monitoring] The rollback has been completed. We are observing that services have recovered. We will continue to monitor for any issues. More details in gitlab.com/gitlab-com/gl-infra/production/-/issues/16042

July 15, 2023 03:02 UTC
[Identified] We are working on performing a rollback on GitLab.com. Next expected update will be in 30 minutes. More details in gitlab.com/gitlab-com/gl-infra/production/-/issues/16042

July 15, 2023 02:52 UTC
[Identified] We are continuing to observe performance issues on GitLab.com. We are investigating options to perform a rollback. More details in gitlab.com/gitlab-com/gl-infra/production/-/issues/16042

July 15, 2023 02:18 UTC
[Identified] No material updates to report at this time. Infrastructure teams are continuing investigations. Details in gitlab.com/gitlab-com/gl-infra/production/-/issues/16042

July 15, 2023 01:45 UTC
[Identified] We have identified the problem and mitigated the worst of it. Our infrastructure team is continuing investigations. Details in gitlab.com/gitlab-com/gl-infra/production/-/issues/16042.

July 15, 2023 01:13 UTC
[Investigating] We have detected a problem related to GitLab.com services and are actively investigating. More details to follow.

June 2023

Delays in Background Processes

June 8, 2023 12:30 UTC

Delays in Background ProcessesPartial Service Disruption

Incident Status

Partial Service Disruption


Components

Website, API, Background Processing


Locations

Google Compute Engine




June 8, 2023 12:30 UTC
[Resolved] Our Engineers applied the fix and redundant pipeline cancellation is now enabled again. We will mark this incident as resolved.

June 6, 2023 18:59 UTC
[Monitoring] The issue is mitigated on Gitlab.com. A fix is being deployed to Gitlab.com. We do not anticipate further updates until the fix is deployed. For further details, check gitlab.com/gitlab-com/gl-infra/production/-/issues/14758

June 6, 2023 14:13 UTC
[Monitoring] Our engineers have identified a potential fix, which is currently being tested. In the meantime, we keep monitoring the situation on the GitLab.com.

June 6, 2023 11:04 UTC
[Monitoring] No material updates to report. Our Engineers keep monitoring the systems, and working on a code fix to enable redundant pipeline cancellation again. For details, see gitlab.com/gitlab-com/gl-infra/production/-/issues/14758

June 6, 2023 09:49 UTC
[Monitoring] The background processes backlog is clear and all features are back to normal but redundant pipeline cancellation is still disabled. We will keep monitoring the systems. For details see gitlab.com/gitlab-com/gl-infra/production/-/issues/14758

June 6, 2023 09:23 UTC
[Identified] Our Engineers have identified the problem cause and implemented a fix. But we expect multiple pipelines were triggered without automatically canceling the old ones. For details, check: gitlab.com/gitlab-com/gl-infra/production/-/issues/14758

June 6, 2023 09:02 UTC
[Investigating] We are currently seeing delays in background processes, which is affecting Merge Requests and other features. For more details, check: gitlab.com/gitlab-com/gl-infra/production/-/issues/14758

Slow requests on GitLab.com

June 7, 2023 13:32 UTC

Slow requests on GitLab.comPartial Service Disruption

Incident Status

Partial Service Disruption


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners for GitLab community contributions, SAML SSO - GitLab SaaS, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, docs.gitlab.com, Canary


Locations

Google Compute Engine, Digital Ocean, Zendesk, AWS




June 7, 2023 13:32 UTC
[Resolved] We have identified the root cause, have seen recovery and are therefore resolving this incident. More information in: gitlab.com/gitlab-com/gl-infra/production/-/issues/14839

June 7, 2023 13:21 UTC
[Investigating] Customers are experiencing slow requests on GitLab.com. More information in: gitlab.com/gitlab-com/gl-infra/production/-/issues/14839

May 2023

Hardware upgrade for the GitLab.com datastore

May 27, 2023 04:06 UTC

Description

In 2 weeks, as part of a planned maintenance window, we will perform a hardware upgrade for the GitLab.com datastore. Users may experience temporary 50X errors for a brief period during this window. See gitlab.com/gitlab-com/gl-infra/production/-/issues/10694


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners for GitLab community contributions, SAML SSO - GitLab SaaS, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, docs.gitlab.com, Canary


Locations

Google Compute Engine, Digital Ocean, Zendesk, AWS


Schedule

May 27, 2023 03:00 - May 27, 2023 04:00 UTC



May 27, 2023 04:06 UTC
[Update] GitLab.com's database layer maintenance is complete now, and we're fully back up and running. We'll be monitoring the platform to ensure all systems are functioning correctly. Thank you for your patience.

May 27, 2023 03:41 UTC
[Update] The database switchover is now complete, we expect all SQL statements to be routed to the new nodes. The site is back up and we're continuing to verify that all systems are functioning correctly. Thank you for your patience.

May 27, 2023 03:00 UTC
[Update] The database maintenance is starting now. The database Primary node switchover should happen at any moment in the next hour and during the switchover users might see 50X errors for a very brief span of time. For details see gitlab.com/gitlab-com/gl-infra/production/-/issues/10694

May 27, 2023 01:51 UTC
[Update] We will be conducting a maintenance database activity in an hour, from 03:00 to 04:00 UTC. Users might see 50X errors for a very brief span of time during this window. For details see gitlab.com/gitlab-com/gl-infra/production/-/issues/10694

May 26, 2023 02:27 UTC
[Update] We will be conducting a maintenance database activity Tomorrow, 2023-05-27, from 03:00 to 04:00 UTC. Users might see 50X errors for a very brief span of time during this window. For details see gitlab.com/gitlab-com/gl-infra/production/-/issues/10694

May 24, 2023 02:14 UTC
[Update] Correction: We will be conducting a maintenance database activity this Saturday, 2023-05-27. from 03:00 to 04:00 UTC. Users might see 50X errors for a very brief span of time during this window. For details see gitlab.com/gitlab-com/gl-infra/production/-/issues/10694

May 24, 2023 02:05 UTC
[Update] We will be conducting a maintenance database activity this Saturday, 2023-05-20, from 03:00 to 04:00 UTC. Users might see 50X errors for a very brief span of time during this window. For details see gitlab.com/gitlab-com/gl-infra/production/-/issues/10694

May 20, 2023 03:58 UTC
[Update] The maintenance is rescheduled to next Saturday, 2023-05-27, from 03:00 to 04:00 UTC. There is currently a long database migration running to fix an incident (gitlab.com/gitlab-com/gl-infra/production/-/issues/14468) that we can't risk to interrupt. We apologise for any inconvenience.

May 20, 2023 03:07 UTC
[Update] The scheduled maintenance will be not be starting at 03:00 UTC, and has been delayed. More details soon to follow.

May 20, 2023 02:00 UTC
[Update] We will be conducting a maintenance database activity in an hour, from 03:00 to 04:00 UTC. Users might see 50X errors for a very brief span of time during this window. For details see gitlab.com/gitlab-com/gl-infra/production/-/issues/10694

May 20, 2023 01:00 UTC
[Update] We will be conducting a maintenance database activity in 2 hours, from 03:00 to 04:00 UTC. Users might see 50X errors for a very brief span of time during this window. For details see gitlab.com/gitlab-com/gl-infra/production/-/issues/10694

May 19, 2023 02:00 UTC
[Update] We will be conducting a maintenance database activity tomorrow, 2023-05-20, from 03:00 to 04:00 UTC. Users might see 50X errors for a very brief span of time during this window. For details see gitlab.com/gitlab-com/gl-infra/production/-/issues/10694

May 17, 2023 02:00 UTC
[Update] We will be conducting a maintenance database activity this Saturday, 2023-05-20, from 03:00 to 04:00 UTC. Users might see 50X errors for a very brief span of time during this window. For details see gitlab.com/gitlab-com/gl-infra/production/-/issues/10694

May 12, 2023 02:00 UTC
[Update] Next week, as part of a planned maintenance window on 2023-05-20 from 03:00 to 04:00 UTC, We will perform a hardware upgrade for the GitLab.com datastore. Users may experience temporary 50X errors for a brief period during this window. See gitlab.com/gitlab-com/gl-infra/production/-/issues/10694

April 2023

Increased number of queued background jobs

April 19, 2023 16:44 UTC

Incident Status

Degraded Performance


Components

Website, Background Processing


Locations

Google Compute Engine




April 19, 2023 16:44 UTC
[Resolved] Our team has increased Sidekiq capacity and background jobs are processing normally. This issue is now resolved. More details in gitlab.com/gitlab-com/gl-infra/production/-/issues/8751

April 19, 2023 16:21 UTC
[Monitoring] We're still monitoring this incident closely as queue-lengths are returning to normal.

April 19, 2023 15:36 UTC
[Monitoring] The root cause was identified, and a mitigation rolled out. The queue-lengths are starting to normalize. We'll continue to monitor the situation closely.

April 19, 2023 15:03 UTC
[Identified] We are seeing an increase of queued background jobs and have started working on a mitigation. More details can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/8751

February 2023

Delays in background processing

February 22, 2023 19:39 UTC

Incident Status

Degraded Performance


Components

Background Processing


Locations

Google Compute Engine




February 22, 2023 19:39 UTC
[Resolved] Job processing is now at normal levels and this incident has now been resolved. We'll continue to work on corrective actions to prevent future occurrences.

February 22, 2023 18:14 UTC
[Resolved] Job processing is continuing to return to normal. We are planning corrective actions to prevent this particular problem in the future. Additional information in gitlab.com/gitlab-com/gl-infra/production/-/issues/8450.

February 22, 2023 17:57 UTC
[Monitoring] We have identified an increase in jobs as a potential root cause. The recovery is continuing and we are monitoring.

February 22, 2023 17:39 UTC
[Investigating] We are continuing to investigate. While we have not yet identified a root cause, we are observing a slow recovery that may improve the behavior for some users. We will continue to update gitlab.com/gitlab-com/gl-infra/production/-/issues/8450.

February 22, 2023 17:25 UTC
[Investigating] We have detected a delay in background job processing and are actively investigating. MRs and pipeline creation may be impacted. The production issue has more information: gitlab.com/gitlab-com/gl-infra/production/-/issues/8450

January 2023

High Error Rate on Select GitLab.com

January 5, 2023 11:24 UTC

Incident Status

Service Disruption


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners for GitLab community contributions, SAML SSO - GitLab SaaS, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, Canary


Locations

Google Compute Engine, Digital Ocean, Zendesk, AWS




January 5, 2023 11:24 UTC
[Resolved] Our Engineers identified the root cause and addressed it with a fix. The incident is now mitigated and users are expected to work with MRs without encountering any errors. For details, check gitlab.com/gitlab-com/gl-infra/production/-/issues/8201

January 5, 2023 09:18 UTC
[Monitoring] Services continue to recover and customers reported successful logins though SAML. However, some users are still facing errors with MRs. Please check the issue for the workaround: gitlab.com/gitlab-com/gl-infra/production/-/issues/8201#note_1228554821

January 5, 2023 08:45 UTC
[Monitoring] The error rates are decreasing, but users might still experience errors when authenticating through SAML and working with MRs. Investigation is still ongoing. For details, check gitlab.com/gitlab-com/gl-infra/production/-/issues/8201

January 5, 2023 08:24 UTC
[Monitoring] We are seeing the error rate decreasing and the services recovering. Our Engineers will keep monitoring the systems. For more details, please check gitlab.com/gitlab-com/gl-infra/production/-/issues/8201

January 5, 2023 08:07 UTC
[Investigating] We are currently seeing a high rate of errors on GitLab.com. Our infrastructure team is currently investigating. More information can be found here: gitlab.com/gitlab-com/gl-infra/production/-/issues/8201

November 2022

GitLab.com is unavailable

November 30, 2022 02:04 UTC

GitLab.com is unavailableDegraded Performance

Incident Status

Degraded Performance


Components

API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners for GitLab community contributions, SAML SSO - GitLab SaaS, Background Processing, Canary


Locations

Google Compute Engine




November 30, 2022 02:04 UTC
[Resolved] GitLab.com fully operational. Root cause: An unexpected deployment behaviour affected the Consul cluster that several of our services depend on (eg. Patroni)

November 30, 2022 01:27 UTC
[Monitoring] Monitoring situation, there was a deployment error for GitLab.com: gitlab.com/gitlab-com/gl-infra/production/-/issues/8097

November 30, 2022 01:10 UTC
[Monitoring] GitLab.com is back up, all systems operational.

November 30, 2022 00:54 UTC
[Investigating] We are currently investigating a GitLab.com service disruption

November 30, 2022 00:48 UTC
[Investigating] We have detected a problem related to GitLab.com services and are actively investigating. More details to follow.

September 2022

GitLab.com maintenance 2022-09-03 11:00 to 14:00 UTC

September 3, 2022 14:16 UTC

Description

We will be performing maintenance on GitLab.com's production database which is anticipated to include a service downtime of up to 180 minutes between 11:00am to 2:00pm UTC on Saturday 2022-09-03. During this time, users will experience complete service disruption. Improving the performance and reliability of GitLab.com has always been a top priority for GitLab. While we continuously make iterative improvements to GitLab and our production architecture, we anticipate making a larger change to improve the scalability and reliability of GitLab.com: performing necessary Operating System maintenance to our database clusters. You can read more about this change in our blog: about.gitlab.com/blog/2022/08/12/upgrading-database-os


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners for GitLab community contributions, SAML SSO - GitLab SaaS, Background Processing, GitLab Customers Portal, Canary


Locations

Google Compute Engine


Schedule

September 3, 2022 11:00 - September 3, 2022 14:00 UTC



September 3, 2022 14:16 UTC
[Update] GitLab.com's database layer maintenance upgrade is complete now, and we're back up and running. We'll be monitoring the platform to ensure all systems are functioning correctly. Thank you for your patience.

September 3, 2022 14:13 UTC
[Update] GitLab.com planned maintenance for the database layer is almost complete. The site is back up and running although we're continuing to verify that all systems are functioning correctly. Thank you for your patience.

September 3, 2022 13:55 UTC
[Update] We are still performing tests to ensure full functionality of the gitlab.com site before bringing it up. Expect full functionality of gitlab.com in about 30mins

September 3, 2022 10:59 UTC
[Update] GitLab.com is now shutting down for planned maintenance to our database layer for the next 3 hours. See you on the other side! More details in about.gitlab.com/blog/2022/08/12/upgrading-database-os

August 31, 2022 16:00 UTC
[Update] Reminder: We will be performing maintenance on GitLab.com's production database which is anticipated to include a complete service downtime of up to 180 minutes between 11:00am to 2:00pm UTC on Saturday 2022-09-03. You can read more in about.gitlab.com/blog/2022/08/12/upgrading-database-os

August 2022

GitLab.com slow to respond

August 26, 2022 06:21 UTC

GitLab.com slow to respondDegraded Performance

Incident Status

Degraded Performance


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners for GitLab community contributions, SAML SSO - GitLab SaaS, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, Canary


Locations

Google Compute Engine, Digital Ocean, Zendesk, AWS




August 26, 2022 06:21 UTC
[Resolved] We are no longer seeing connectivity issues for GitLab.com after a period of monitoring. We therefore believe that this incident has been resolved. Details in : gitlab.com/gitlab-com/gl-infra/production/-/issues/7656

August 26, 2022 05:08 UTC
[Monitoring] All the services seems to have recovered. We will continue to monitor the performance and provide the updates here. More information on the root cause can be found here: gitlab.com/gitlab-com/gl-infra/production/-/issues/7656

August 26, 2022 04:57 UTC
[Investigating] We are still investigating the root cause of this issue. More details here: gitlab.com/gitlab-com/gl-infra/production/-/issues/7656

August 26, 2022 04:55 UTC
[Investigating] We are noticing some slow response issues with GitLab.com.

July 2022

Scheduled GitLab.com production database maintenance

July 2, 2022 08:51 UTC

Description

We will be performing maintenance on GitLab.com's production database which is anticipated to include a service downtime of up to 120 minutes between Saturday, 2022-07-02, 06:00am to 08:00am UTC. During this time users will experience complete service disruption. Improving the performance and reliability of GitLab.com has always been a top priority for GitLab. While we continuously make iterative improvements to GitLab and our production architecture, we anticipate making a larger change to improve the scalability and reliability of GitLab.com: We are splitting our single PostgreSQL database into a main and a ci database. You can read more about this change in our blog: about.gitlab.com/blog/2022/06/02/splitting-database-into-main-and-ci


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners for GitLab community contributions, SAML SSO - GitLab SaaS, Background Processing, GitLab Customers Portal


Locations

Google Compute Engine


Schedule

July 2, 2022 05:00 - July 2, 2022 09:00 UTC



July 2, 2022 08:51 UTC
[Update] GitLab.com's database layer maintenance upgrade is complete now, and we're back up and running. We'll be monitoring the platform to ensure all systems are functioning correctly. Thank you for your patience.

July 2, 2022 07:43 UTC
[Update] GitLab.com planned maintenance for the database layer is almost complete. The site is back up and running although we're continuing to verify that all systems are functioning correctly. Thank you for your patience.

July 2, 2022 06:00 UTC
[Update] GitLab.com is now shutting down for planned maintenance for database layer improvements for the next 2 hours. See you on the other side!

July 2, 2022 05:00 UTC
[Update] GitLab.com will undergo maintenance in 1 hour. Please note that any CI jobs that start before the maintenance window but complete during the window period (up to 2 hours) will fail and may need to be started again.

July 1, 2022 05:45 UTC
[Update] Reminder: Tomorrow we will be undergoing scheduled maintenance to our database layer. We expect the GitLab.com site to be unavailable for up to 2 hours. Starting time: 2022-07-02 06:00 UTC.

June 29, 2022 10:01 UTC
[Update] We wanted to clarify that this scheduled maintenance will also impact Pages hosted with GitLab Pages on GitLab.com. Pages will be unavailable for up to 2 hours starting from 2022-07-02 06:00 UTC. We apologize in advance for any inconvenience this may cause.

June 29, 2022 05:50 UTC
[Update] In 3 days time, we will be undergoing some scheduled maintenance to our database layer so we expect the GitLab.com site to be unavailable for up to 2 hours starting from 2022-07-02 06:00 UTC. We apologize in advance for any inconvenience this may cause.

June 2022

Spike in error rate on gitlab.com

June 23, 2022 07:19 UTC

Incident Status

Operational


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners for GitLab community contributions, SAML SSO - GitLab SaaS, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, Canary


Locations

Google Compute Engine, Digital Ocean, Zendesk, AWS




June 23, 2022 07:19 UTC
[Resolved] Error rates have returned to normal levels. Follow-up discussion and actions are being discussed here: gitlab.com/gitlab-com/gl-infra/production/-/issues/7310

June 23, 2022 06:46 UTC
[Monitoring] We are investigating a brief spike in errors that lasted about 20 minutes, we are monitoring the recovery and will put more detail in gitlab.com/gitlab-com/gl-infra/production/-/issues/7310

Connectivity issues for GitLab.com

June 21, 2022 08:43 UTC

Incident Status

Service Disruption


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners for GitLab community contributions, SAML SSO - GitLab SaaS, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, Canary


Locations

Google Compute Engine, Digital Ocean, Zendesk, AWS




June 21, 2022 08:43 UTC
[Resolved] We are no longer seeing connectivity issues for GitLab.com after a period of monitoring. We therefore believe that this incident has been resolved. Details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/7288

June 21, 2022 08:00 UTC
[Monitoring] Services seem to be back to normal, and we continue monitoring. Details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/7288

June 21, 2022 07:45 UTC
[Identified] We are seeing services slowly recovering and continue monitoring issues with our CDN provider.

June 21, 2022 07:22 UTC
[Identified] No material updates at this point. We are continuing to monitor issues with our CDN provider

June 21, 2022 07:02 UTC
[Identified] We are continuing to monitor connectivity issues from some regions due to connectivity problems from our CDN provider.

June 21, 2022 06:52 UTC
[Investigating] We are investigating connectivity issues for GitLab.com, this is affecting all services.

Gitlab.com is having slowness issues

June 15, 2022 23:28 UTC

Incident Status

Degraded Performance


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners for GitLab community contributions, SAML SSO - GitLab SaaS, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, Canary


Locations

Google Compute Engine, Digital Ocean, Zendesk, AWS




June 15, 2022 23:28 UTC
[Resolved] We are no longer seeing performance issues on GitLab.com after a period of monitoring. We believe the incident has now been resolved. More details can be found here: gitlab.com/gitlab-com/gl-infra/production/-/issues/7258

June 15, 2022 23:12 UTC
[Identified] Our teams had identified the root cause of the slowness and are discussing preventive actions for it. Performance is back to normal.

June 15, 2022 22:53 UTC
[Investigating] Our teams detected a performance degradation for all components for GitLab.com, it appears this degradation is now stabilizing back to normal performance. We are still investigating.

June 15, 2022 22:36 UTC
[Investigating] Requests to GitLab.com are seeing elevated error rates and slowness, we are investigating.

Repository mirroring delays

June 9, 2022 14:31 UTC

Repository mirroring delaysDegraded Performance

Incident Status

Degraded Performance


Components

Background Processing


Locations

Google Compute Engine




June 9, 2022 14:31 UTC
[Resolved] After a period of monitoring, we are no longer seeing errors with the Pull Mirror feature. This incident has been marked as resolved. More details here: gitlab.com/gitlab-com/gl-infra/production/-/issues/7223

June 9, 2022 12:34 UTC
[Monitoring] After observing the Pull Mirror processing and verifying that the fix worked, we marked the incident as mitigated. We'll continue to monitor the Pull Mirror processing.

June 9, 2022 10:24 UTC
[Monitoring] Deployment of the fix to production is complete. This will be monitored for some time before considering that the issue is solved. Details in gitlab.com/gitlab-com/gl-infra/production/-/issues/7223

June 9, 2022 09:26 UTC
[Identified] The potential fix has been promoted and is currently being deployed to production. After that, we will be actively monitoring to confirm expected results. For more details, please refer to gitlab.com/gitlab-com/gl-infra/production/-/issues/7223

June 9, 2022 07:04 UTC
[Identified] No material updates at this time - we're still waiting for the deployment to complete. Following update will be in the next hour or if we have more update. For the details, please refer to gitlab.com/gitlab-com/gl-infra/production/-/issues/7223

June 9, 2022 04:57 UTC
[Identified] We're waiting for the deployment to complete. Following update will be in the next two hours or if we have more update. For the details, please refer to gitlab.com/gitlab-com/gl-infra/production/-/issues/7223

June 9, 2022 02:39 UTC
[Identified] We've found minor issues with the deployment and resolved them. Currently we are waiting for the deploy to complete. Following update will be in the next two hours. For the details, please refer to gitlab.com/gitlab-com/gl-infra/production/-/issues/7223

June 9, 2022 00:57 UTC
[Identified] We've identified the cause of the issue and are working on resolving it. We are still waiting for the deployment to finish. For further details, please refer to gitlab.com/gitlab-com/gl-infra/production/-/issues/7223.

June 8, 2022 23:06 UTC
[Investigating] No material updates at this time - our investigation and monitoring is ongoing. We are waiting for the fix to be deployed. More information on gitlab.com/gitlab-com/gl-infra/production/-/issues/7223

June 8, 2022 21:15 UTC
[Investigating] We’ve been investigating and believe we have identified a root cause of the delayed repository mirroring. We are working to confirm our understanding and resolve the issue. For additional information see gitlab.com/gitlab-com/gl-infra/production/-/issues/7223

June 8, 2022 17:25 UTC
[Investigating] We are still investigating the cause of the delays in repository mirroring. The issue seems to be only impacting pull mirrors. For additional information see gitlab.com/gitlab-com/gl-infra/production/-/issues/7223

June 8, 2022 15:07 UTC
[Investigating] No material updates to report. Investigation is still underway. More details in gitlab.com/gitlab-com/gl-infra/production/-/issues/7223

June 8, 2022 14:51 UTC
[Investigating] Investigation of the delays in repository mirroring is still ongoing. More details: gitlab.com/gitlab-com/gl-infra/production/-/issues/7223

June 8, 2022 14:32 UTC
[Investigating] We are still investigating the cause of the delays in repository mirroring and no material update at this time, for additional information see gitlab.com/gitlab-com/gl-infra/production/-/issues/7223

June 8, 2022 14:01 UTC
[Investigating] We're investigating unusually high delays in our repository mirroring feature. Details in gitlab.com/gitlab-com/gl-infra/production/-/issues/7223

Intermittent failures in outbound emails

June 9, 2022 14:22 UTC

Incident Status

Partial Service Disruption


Components

Background Processing


Locations

Google Compute Engine




June 9, 2022 14:22 UTC
[Resolved] We are no longer seeing any errors in our email sending infrastructure after a period of monitoring. We believe the incident has now been resolved. More details can be found here: gitlab.com/gitlab-com/gl-infra/production/-/issues/7231

June 9, 2022 13:39 UTC
[Monitoring] Our email delivery service provider, Mailgun, is currently having an incident but a fix is being implemented on their end: status.mailgun.com/incidents/gcm3zw1lvfl4. We are currently seeing a drop in the error rates and will continue to monitor.

June 9, 2022 13:12 UTC
[Investigating] We are observing intermittent failures in our email sending infrastructure and are currently investigating. For more details, see: gitlab.com/gitlab-com/gl-infra/production/-/issues/7231

April 2022

Elevated Error Rate on GitLab.com

April 28, 2022 07:37 UTC

Incident Status

Service Disruption


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners for GitLab community contributions, SAML SSO - GitLab SaaS, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, Canary


Locations

Google Compute Engine, Digital Ocean, Zendesk, AWS




April 28, 2022 07:37 UTC
[Resolved] We transitioned this incident to resolved. Check the issue for more details: gitlab.com/gitlab-com/gl-infra/production/-/issues/6933

April 28, 2022 07:23 UTC
[Monitoring] We have identified the cause to this incident and do not anticipate any additional impacts at this time. We are continuing to monitor services as well as complete additional investigation. More details here: gitlab.com/gitlab-com/gl-infra/production/-/issues/6933

April 28, 2022 06:34 UTC
[Investigating] We are still investigating the root cause. The error rates on affected systems recovering, and we are monitoring to ensure that the issue does not recur.

April 28, 2022 06:12 UTC
[Investigating] We're investigating increased error rates across GitLab-com services, some users might be experiencing 500 errors intermittently, more info can be found in here: gitlab.com/gitlab-com/gl-infra/production/-/issues/6933





Back to current status