Active Issue

Updated a few seconds ago

Back to current status

Status History

Filter: Background Processing (Clear)



February 2025

PG minor upgrade 16.6 in Production

February 15, 2025 07:49 UTC

Description

Scheduled maintenance to patch our DB layer to PostgreSQL 16.6. 2025-02-15 02:00-08:00 UTC. GitLab.com will be available throughout this time, but users may experience errors on requests. We apologize in advance for any inconvenience caused. See <gitlab.com/gitlab-com/gl-infra/production/-/issues/19120>


Components

Website, API, Container Registry, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions, Background Processing


Locations

Google Compute Engine, AWS


Schedule

February 15, 2025 03:00 - February 15, 2025 09:00 UTC



February 15, 2025 07:49 UTC
[Update] Gitlab.com database maintenance was performed, the platform is considered stable. We'll continue to monitor for any performance issues. Thank you for your patience. See <gitlab.com/gitlab-com/gl-infra/production/-/issues/19120>

February 15, 2025 04:07 UTC
[Update] Gitlab.com scheduled maintenance of our database layer has started. GitLab.com should be available during the whole period but we expect availability and performance impact until 2025-02-15 09:00 UTC. See gitlab.com/gitlab-com/gl-infra/production/-/issues/19120

February 15, 2025 03:01 UTC
[Update] Gitlab.com scheduled maintenance of our database layer has started. GitLab.com should be available during the whole period but we expect availability and performance impact until 2025-02-15 09:00 UTC. See gitlab.com/gitlab-com/gl-infra/production/-/issues/19120

February 15, 2025 02:48 UTC
[Update] Scheduled database maintenance will now start at 2025-02-15 03:00 UTC and expected to run to 2025-02-15 09:00 UTC. See gitlab.com/gitlab-com/gl-infra/production/-/issues/19120

February 15, 2025 01:54 UTC
[Update] Due to an ongoing incident we are delaying the maintenance window by 2 hours to investigate the impact. We currently plan to start the maintenance window at 04:00 UTC.

February 14, 2025 01:45 UTC
[Update] We will be undergoing scheduled maintenance of our databases tomorrow. There may be some availability and performance impact between 2025-02-15 02:00 UTC and 2025-02-15 08:00 UTC. See gitlab.com/gitlab-com/gl-infra/production/-/issues/19120

February 12, 2025 01:13 UTC
[Update] We will be undergoing scheduled maintenance of our databases in 3 days. There may be some availability and performance impact between 2025-02-15 02:00 UTC and 2025-02-15 08:00 UTC. See gitlab.com/gitlab-com/gl-infra/production/-/issues/19120

February 8, 2025 08:37 UTC
[Update] Scheduled maintenance in 1 week: Upgrading DB layer to PostgreSQL 16.6 on Feb 15, 02:00-08:00 UTC. GitLab.com will stay up but users may experience errors. See gitlab.com/gitlab-com/gl-infra/production/-/issues/19120

February 1, 2025 02:42 UTC
[Update] Scheduled maintenance in 2 weeks: Upgrading DB layer to PostgreSQL 16.6 on Feb 15, 02:00-08:00 UTC. GitLab.com will stay up but may experience errors. Sorry for any inconvenience. See gitlab.com/gitlab-com/gl-infra/production/-/issues/19120

January 2025

Accessing gitlab.com results in 500 errors

January 6, 2025 16:42 UTC

Incident Status

Partial Service Disruption


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions, SAML SSO - GitLab SaaS, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, docs.gitlab.com, Canary, Product Analytics - Configurator (Beta), Product Analytics - Collector (Beta), Product Analytics - Background processing (Beta), Product Analytics - Queries (Beta), GitLab Duo


Locations

Google Compute Engine, Digital Ocean, Zendesk, AWS




January 6, 2025 16:42 UTC
[Resolved] We've identified the cause of the errors and the errors have returned to normal. The incident is now resolved. More information on the incident can be found in this issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/19079

January 6, 2025 15:57 UTC
[Monitoring] We've identified the cause of the errors, and the errors have returned to normal. The incident is now mitigated, but we're continuing to monitor for any further impact. More information on the incident can be found in this issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/19079

January 6, 2025 15:27 UTC
[Monitoring] We're continuing to monitor for further errors related to the incident. We see further decreases in the errors.

January 6, 2025 15:17 UTC
[Monitoring] We saw elevated 500 errors on gitlab.com which have then decreased. We're continuing to monitor for further errors

December 2024

Intermittent errors across GitLab.com

December 20, 2024 19:47 UTC

Intermittent errors across GitLab.comPartial Service Disruption

Incident Status

Partial Service Disruption


Components

Website, GitLab Pages, Background Processing


Locations

Google Compute Engine




December 20, 2024 19:47 UTC
[Resolved] GitLab.com has remained stable throughout our monitoring period and we are now marking this incident as resolved. Please see the following issue for further information related to this incident: gitlab.com/gitlab-com/gl-infra/production/-/issues/19033

December 20, 2024 00:08 UTC
[Monitoring] We are continuing to monitor the issue at this point, with further recovery being observed. Follow gitlab.com/gitlab-com/gl-infra/production/-/issues/19033 for more details.

December 19, 2024 23:32 UTC
[Monitoring] Mitigation steps have been completed. You should see delayed tasks start to recover now. Follow gitlab.com/gitlab-com/gl-infra/production/-/issues/19033 for more details.

December 19, 2024 23:09 UTC
[Identified] Mitigation steps are progressing and GitLab.com responsiveness is looking good. Follow gitlab.com/gitlab-com/gl-infra/production/-/issues/19033 for more details.

December 19, 2024 21:58 UTC
[Identified] Users may still see delays when interacting with GitLab.com. We are still actively working to mitigate the issue. Details will be posted in: gitlab.com/gitlab-com/gl-infra/production/-/issues/19033

December 19, 2024 21:11 UTC
[Identified] Our work to fully mitigate the underlying issue is still ongoing. Details will be posted in: gitlab.com/gitlab-com/gl-infra/production/-/issues/19033

December 19, 2024 20:31 UTC
[Identified] No material updates at this time. We have seen in overall improvement but are still are still working to fully mitigate the issue. Details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/19033

December 19, 2024 19:51 UTC
[Identified] Our initial changes have helped alleviate the issue. Our efforts are still ongoing as this is not fully mitigated. Details will be posted in: gitlab.com/gitlab-com/gl-infra/production/-/issues/19033

December 19, 2024 19:14 UTC
[Identified] We have implemented mitigating changes. Services should begin to recover. We are continuing to adjust and monitor. Details will be posted in: gitlab.com/gitlab-com/gl-infra/production/-/issues/19033

December 19, 2024 18:37 UTC
[Identified] No material updates at this time. We are still working to mitigate the issue. Details will be posted in: gitlab.com/gitlab-com/gl-infra/production/-/issues/19033

December 19, 2024 18:09 UTC
[Identified] We are still working to mitigate the issue. Users may still encounter errors or delays on GitLab.com. Details will be posted in: gitlab.com/gitlab-com/gl-infra/production/-/issues/19033

December 19, 2024 17:46 UTC
[Identified] We believe to have identified the cause of this incident. We are working on a mitigation strategy. Users may see errors or delays when interacting with GitLab.com. Details will be posted in: gitlab.com/gitlab-com/gl-infra/production/-/issues/19033

December 19, 2024 17:31 UTC
[Investigating] We are investigating reports of intermittent errors across GitLab.com and GitLab Pages. Details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/19033

Project mirror disabled due to excessive notifications

December 3, 2024 20:26 UTC

Incident Status

Degraded Performance


Components

Website, Background Processing


Locations

Google Compute Engine




December 3, 2024 20:26 UTC
[Resolved] After seeing no further issues arise during our monitoring period, we are considering this incident resolved. Please review gitlab.com/gitlab-com/gl-infra/production/-/issues/18929 for more details.

December 3, 2024 18:49 UTC
[Monitoring] Project mirroring has been re-enabled on GitLab.com and we are monitoring to make sure no further issues arise. See gitlab.com/gitlab-com/gl-infra/production/-/issues/18929 for further details.

December 3, 2024 15:51 UTC
[Identified] We've turned off the functionality relating to schedules for project mirroring. Project mirroring will be reenabled once we resolve this issue. Mirrored projects will not be updated during this time.

December 3, 2024 14:47 UTC
[Identified] We've turned off the functionality that sends out email updates temporarily for project mirrors. We are continuing to investigate this incident.

December 3, 2024 14:04 UTC
[Investigating] We are currently investigating emails being sent for older project mirrors and imports. More details about this incident can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/18929

October 2024

Sidekiq queue lengths increasing causing delays in MR and pipeline processing

October 16, 2024 15:02 UTC

Incident Status

Partial Service Disruption


Components

Website, Background Processing


Locations

Google Compute Engine




October 16, 2024 15:02 UTC
[Resolved] Our monitoring shows no further issues in Sidekiq performance after the mitigation steps were taken. We are resolving this issue and any further updates will be in: gitlab.com/gitlab-com/gl-infra/production/-/issues/18720

October 16, 2024 13:47 UTC
[Monitoring] We are continuing to see the queue recover, and are monitoring Sidekiq performance while recovery is complete. More details about this incident can be found in: gitlab.com/gitlab-com/gl-infra/production/-/issues/18720

October 16, 2024 12:49 UTC
[Monitoring] We are continuing to see the queue recover, and we are monitoring performance while recovery is complete. More details about this incident can be found in: gitlab.com/gitlab-com/gl-infra/production/-/issues/18720

October 16, 2024 12:14 UTC
[Identified] We've identified issues with delays in MR and pipeline processing as a result of Sidekiq queue lengths increasing. We're starting to see the queue recover. More details about this incident can be found in: gitlab.com/gitlab-com/gl-infra/production/-/issues/18720

Pipelines not completing

October 9, 2024 17:33 UTC

Pipelines not completingPartial Service Disruption

Incident Status

Partial Service Disruption


Components

Background Processing


Locations

Google Compute Engine




October 9, 2024 17:33 UTC
[Resolved] The MR reverting the offending commit that caused this issue has been deployed to production. This incident is now resolved. See gitlab.com/gitlab-com/gl-infra/production/-/issues/18676 for all the details.

October 9, 2024 10:16 UTC
[Monitoring] We have enabled a feature flag that will prevent new pipelines and Merge Requests from being impacted. Pipelines and MR that were previously stuck will need to be recreated. See gitlab.com/gitlab-com/gl-infra/production/-/issues/18676

October 9, 2024 09:28 UTC
[Identified] We have identified a potential commit that could have caused this incident. We are working on reverting it. See gitlab.com/gitlab-com/gl-infra/production/-/issues/18676 for details.

October 9, 2024 09:23 UTC
[Investigating] Some Merge requests are still impacted by this incident. Merge requests are stuck with the message "Your merge request is almost ready!". The workaround is to recreate the impacted Merge Request. See gitlab.com/gitlab-com/gl-infra/production/-/issues/18676 for details.

October 9, 2024 08:26 UTC
[Investigating] We are moving this incident back to Investigating, as after resolving the Redis saturation, we are still seeing Pipelines not progressing between stages. Merge Requests are functioning normally now. See gitlab.com/gitlab-com/gl-infra/production/-/issues/18676 for details.

October 9, 2024 04:25 UTC
[Monitoring] We have resolved the Redis saturation issue and see that all pipelines are functioning normally now. We will continue to monitor while continuing investigation into other aspects. See gitlab.com/gitlab-com/gl-infra/production/-/issues/18676 for details.

October 9, 2024 04:18 UTC
[Resolved] We have resolved the Redis saturation issue and see that all pipelines are functioning normally now. We will continue to monitor while continuing investigation into other aspects. See gitlab.com/gitlab-com/gl-infra/production/-/issues/18676 for details.

October 9, 2024 03:33 UTC
[Monitoring] We have resolved the Redis saturation issue and see that all pipelines are functioning normally now. We will continue to monitor while continuing investigation into other aspects. See gitlab.com/gitlab-com/gl-infra/production/-/issues/18676 for details.

October 9, 2024 02:33 UTC
[Investigating] No material updates to report. We continue investigations and will provide further updates in 1 hour. Please follow gitlab.com/gitlab-com/gl-infra/production/-/issues/18676 for the latest.

October 9, 2024 01:26 UTC
[Investigating] No material updates to report. We continue to investigate potential causes of this issue. Please follow gitlab.com/gitlab-com/gl-infra/production/-/issues/18676 for the latest updates.

October 9, 2024 01:02 UTC
[Investigating] Investigation continues. Remember, restarting the pipeline on individual jobs may serve as a workaround. See gitlab.com/gitlab-com/gl-infra/production/-/issues/18676.

October 9, 2024 00:02 UTC
[Investigating] Reminder: If you have urgent pipelines affected by the ongoing issue, you can cancel and retry them. A retry may make the pipeline run properly. We're still working on a permanent fix. See gitlab.com/gitlab-com/gl-infra/production/-/issues/18676.

October 8, 2024 23:57 UTC
[Investigating] Despite resolving the Redis saturation issue, we're still receiving reports of hanging pipelines on GitLab.com. We understand the frustration this causes and are urgently investigating. Updates at gitlab.com/gitlab-com/gl-infra/production/-/issues/18676.

October 8, 2024 20:24 UTC
[Monitoring] After alleviating the saturation reported in our Redis infrastructure we are seeing readings go back to healthy levels. We will monitor now for new occurrences and reports to confirm this is resolved. Please review gitlab.com/gitlab-com/gl-infra/production/-/issues/18676 for details.

October 8, 2024 19:23 UTC
[Investigating] We're taking corrective actions to alleviate some unexpected pressure in one of our Redis shards, which may alleviate the problem. See gitlab.com/gitlab-com/gl-infra/production/-/issues/18676 for more details.

October 8, 2024 18:29 UTC
[Investigating] We are currently investigating errors on a specific Redis shard that may be contributing to the problem. For the latest updates and detailed information, please check gitlab.com/gitlab-com/gl-infra/production/-/issues/18676.

October 8, 2024 17:55 UTC
[Investigating] Investigation continues. Remember restarting the pipeline or individual jobs may serve as a workaround. See gitlab.com/gitlab-com/gl-infra/production/-/issues/18676.

October 8, 2024 17:25 UTC
[Investigating] Work is ongoing to identify the root of this issue, including an analysis of our Redis infrastructure. Please continue to follow gitlab.com/gitlab-com/gl-infra/production/-/issues/18676.

October 8, 2024 16:48 UTC
[Investigating] Work is still in progress to identify the cause of service degradation to pipelines. Update frequency will be increased to every 30 minutes or as soon as we have material updates to share. Please follow gitlab.com/gitlab-com/gl-infra/production/-/issues/18676 for more information.

October 8, 2024 16:23 UTC
[Investigating] We continue to investigate potential causes of this issue. Please follow gitlab.com/gitlab-com/gl-infra/production/-/issues/18676 for the latest updates.

October 8, 2024 15:39 UTC
[Investigating] We are currently investigating an issue where pipelines are not progressing between stages. This causes them to remain in a running state indefinitely, and merge requests are not being marked as ready. gitlab.com/gitlab-com/gl-infra/production/-/issues/18676

GitLab slow background processing

October 8, 2024 11:42 UTC

Incident Status

Degraded Performance


Components

Background Processing


Locations

Google Compute Engine




October 8, 2024 11:42 UTC
[Resolved] We are resolving the slowness with background jobs issue gitlab.com/gitlab-com/gl-infra/production/-/issues/18677 and we will follow up with stuck MRs and pipelines in gitlab.com/gitlab-com/gl-infra/production/-/issues/18676

October 8, 2024 10:42 UTC
[Identified] We have identified the cause of the background job saturation and the services are recovering. More information in: gitlab.com/gitlab-com/gl-infra/production/-/issues/18676

October 8, 2024 10:27 UTC
[Investigating] We are currently investigating slowness with background jobs which might impact Merge Requests and Pipelines. More information in: gitlab.com/gitlab-com/gl-infra/production/-/issues/18676

September 2024

GitLab slow background processing

September 10, 2024 17:22 UTC

Incident Status

Degraded Performance


Components

Background Processing, GitLab Duo


Locations

Google Compute Engine




September 10, 2024 17:22 UTC
[Resolved] This incident is now resolved. The background jobs queue is now empty, and all stuck jobs should be being processed. If you still see any stuck jobs, restarting the job should work to unblock it. More details about this incident can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/18538.

September 10, 2024 17:02 UTC
[Monitoring] The background jobs queue is now empty. All stuck jobs should be being processed. If you still see any stuck jobs, restarting the job should work to unblock it. We are monitoring the current status of the application. More details about this incident can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/18538.

September 10, 2024 15:47 UTC
[Identified] We are working on resolving the background processing issues. GitLab Duo responses might be affected due to slow background processing. More details about this incident can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/18538.

September 10, 2024 15:14 UTC
[Investigating] No material updates at this time. More details can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/18538.

September 10, 2024 14:57 UTC
[Investigating] GitLab CI functionality is partially unavailable right now due to issues with background processing. We are working on a mitigation. More details can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/18538.

September 10, 2024 14:31 UTC
[Investigating] We are currently investigating issues with background processing and working on a temporary mitigation strategy. More details can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/18538.

September 10, 2024 14:08 UTC
[Investigating] We are currently investigating issues with background processing. More details about this incident can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/18538.

GitLab pipelines experiencing longer than expected execution times

September 4, 2024 05:57 UTC

Incident Status

Operational


Components

Background Processing, GitLab Duo


Locations

Google Compute Engine




September 4, 2024 05:57 UTC
[Resolved] Pipeline functionality has been restored and is working as expected based on our monitoring. We are marking this incident as resolved. For more updates, please follow: gitlab.com/gitlab-com/gl-infra/production/-/issues/18489 Additionally, we have reset the compute monthly minutes due to this incident. For details on this action, see: gitlab.com/gitlab-com/gl-infra/production/-/issues/18497

September 4, 2024 00:13 UTC
[Monitoring] Pipeline functionality has been restored. We have opted to reset the compute monthly minutes due to this incident. That work has started and you can view it here: gitlab.com/gitlab-com/gl-infra/production/-/issues/18497. See incident issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/18489

September 3, 2024 23:12 UTC
[Monitoring] Pipeline functionality has been restored. We have opted to reset the compute monthly min due to this incident. Please view the change request here. gitlab.com/gitlab-com/gl-infra/production/-/issues/18497. See incident issue : gitlab.com/gitlab-com/gl-infra/production/-/issues/18489

September 3, 2024 22:02 UTC
[Monitoring] Pipeline functionality has been restored. We have opted to reset the compute monthly min due to this incident. Please view the change request here. gitlab.com/gitlab-com/gl-infra/production/-/issues/18497.

September 3, 2024 19:46 UTC
[Monitoring] We have mitigation in place and pipelines are resolving normally. Please See the issue for more information: gitlab.com/gitlab-com/gl-infra/production/-/issues/18489

September 3, 2024 18:20 UTC
[Monitoring] We have mitigation in place and pipelines are resolving normally. Please See the issue for more information: gitlab.com/gitlab-com/gl-infra/production/-/issues/18489

September 3, 2024 18:04 UTC
[Monitoring] We are continuing to monitor and we recommend cancelling jobs that may have become stuck during the incident and we should see them process. See the issue for more information: gitlab.com/gitlab-com/gl-infra/production/-/issues/18489

September 3, 2024 16:57 UTC
[Monitoring] We are continuing to monitor and we recommend cancelling jobs that may have become stuck during the incident and we should see them process. See the issue for more information: gitlab.com/gitlab-com/gl-infra/production/-/issues/18489

September 3, 2024 15:41 UTC
[Monitoring] No material updates at this time. We are continuing to monitor and we recommend cancelling jobs that may have become stuck during the incident. See the issue for more information: gitlab.com/gitlab-com/gl-infra/production/-/issues/18489

September 3, 2024 13:59 UTC
[Monitoring] We are continuing to monitor. We recommend cancelling jobs that may have become stuck during the incident. See the issue for more information: gitlab.com/gitlab-com/gl-infra/production/-/issues/18489

September 3, 2024 12:26 UTC
[Monitoring] Background job processing has been restored. We are continuing to monitor while we investigate further into the root cause. See the issue for more information: gitlab.com/gitlab-com/gl-infra/production/-/issues/18489

September 3, 2024 11:40 UTC
[Investigating] No material updates at this time. We are continuing to investigate and monitor background job processing. See the issue for more information: gitlab.com/gitlab-com/gl-infra/production/-/issues/18489

September 3, 2024 10:59 UTC
[Investigating] We are seeing improvements in background job processing and operations are recovering back to normal. Delays in email processing are still possible, and we are continuing with our investigation. See the issue for more information: gitlab.com/gitlab-com/gl-infra/production/-/issues/18489

September 3, 2024 10:04 UTC
[Investigating] No current updates at this time. We are still working to isolate the problem but took measures to add relief. We will provide an update once the root cause has been identified. See the issue for more information: gitlab.com/gitlab-com/gl-infra/production/-/issues/18489.

September 3, 2024 09:38 UTC
[Investigating] We are still working to isolate the problem to a specific background job. See the issue for more information: gitlab.com/gitlab-com/gl-infra/production/-/issues/18489.

September 3, 2024 09:15 UTC
[Investigating] We are trying to isolate the problem to a number of background jobs contributing to this problem and we are taking them offline for some time. AI feature such as Duo Chat may also be impacted by this. See the issue for more information: gitlab.com/gitlab-com/gl-infra/production/-/issues/18489.

September 3, 2024 08:52 UTC
[Investigating] No material updates at this time - our investigation into Sidekiq issues is ongoing. See the issue for more information: gitlab.com/gitlab-com/gl-infra/production/-/issues/18489.

September 3, 2024 08:37 UTC
[Investigating] We are investigating issues with Sidekiq. See the issue for more information: gitlab.com/gitlab-com/gl-infra/production/-/issues/18489.

September 3, 2024 08:13 UTC
[Identified] We hare still investigating the issue. The background processes may take longer to be picked up and executed than usual. This can result in emails not being delivered and commits not showing up in merge request activity. CI/CD jobs may be stuck in pending state. Issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/18489

September 3, 2024 07:51 UTC
[Investigating] We are still investigating the cause of extended GitLab pipeline execution times. Our team continues to work on resolving this issue. We appreciate your patience and will update when we have more information. Issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/18489

September 3, 2024 07:18 UTC
[Investigating] We are currently investigating reports of GitLab pipelines taking longer than usual to complete. Our team is aware of the issue and actively working to identify the cause of these delays. Issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/18489

GitLab.com Connectivity Issues

September 3, 2024 05:34 UTC

Incident Status

Service Disruption


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions, SAML SSO - GitLab SaaS, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, docs.gitlab.com, Canary, Product Analytics - Configurator (Beta), Product Analytics - Collector (Beta), Product Analytics - Background processing (Beta), Product Analytics - Queries (Beta), GitLab Duo


Locations

Google Compute Engine, Digital Ocean, Zendesk, AWS




September 3, 2024 05:34 UTC
[Resolved] All GitLab services have been restored. We are actively monitoring the system, and everything appears normal at this time. For more details, please follow the issue at: gitlab.com/gitlab-com/gl-infra/production/-/issues/18490

September 3, 2024 05:19 UTC
[Monitoring] We've identified and addressed the root cause of the issue. All GitLab services have been restored, and we are actively monitoring the system. Follow the issue for more details : gitlab.com/gitlab-com/gl-infra/production/-/issues/18490

September 3, 2024 05:09 UTC
[Investigating] GitLab.com is currently experiencing connectivity issues due to a high error rate. Our infrastructure team is investigating. Issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/18490

August 2024

Some requests timing out

August 23, 2024 03:59 UTC

Incident Status

Operational


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions, SAML SSO - GitLab SaaS, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, docs.gitlab.com, Canary, Product Analytics - Configurator (Beta), Product Analytics - Collector (Beta), Product Analytics - Background processing (Beta), Product Analytics - Queries (Beta), GitLab Duo


Locations

Google Compute Engine, Digital Ocean, Zendesk, AWS




August 23, 2024 03:59 UTC
[Resolved] The issue has been resolved and all functionality on GitLab.com is nominal. Please see gitlab.com/gitlab-com/gl-infra/production/-/issues/18435 for more details.

August 23, 2024 01:53 UTC
[Monitoring] The issue has been identified and the incident resolved. We will continue monitoring for a while.

August 23, 2024 01:27 UTC
[Identified] We have identified an underlying cause and are working to mitigate it. See gitlab.com/gitlab-com/gl-infra/production/-/issues/18435 for more details.

August 23, 2024 00:34 UTC
[Investigating] We have noticed that there are delays in web UI updates for example, MRs not showing up after submission, We are continuing to investigate - please see gitlab.com/gitlab-com/gl-infra/production/-/issues/18435 for more details.

August 23, 2024 00:27 UTC
[Investigating] Some requests are slow or timing out due to DB saturation. We are investigating. Please see [this issue](gitlab.com/gitlab-com/gl-infra/production/-/issues/18435) for further details.

Sidekiq processing delays

August 16, 2024 20:29 UTC

Sidekiq processing delaysDegraded Performance

Incident Status

Degraded Performance


Components

Website, Background Processing


Locations

Google Compute Engine




August 16, 2024 20:29 UTC
[Resolved] This incident is considered resolved. Thanks for your patience! Details will be provided in the incident issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/18407

August 16, 2024 20:05 UTC
[Identified] While background processing has stabilized, our team is still investigating the root cause. Updates will be provided when more information is available.

August 16, 2024 19:40 UTC
[Identified] Our team has identified and disabled a worker that was impacting performance. Sidekiq processing is improving and we are continuing to monitor the state of the background processing.

August 16, 2024 19:27 UTC
[Investigating] Our team is still investigating the processing delays. Some users might experience some errors with creating merge requests or other delays with processing background jobs.

August 16, 2024 19:07 UTC
[Investigating] Our team is currently investigating some sidekiq performance issues which is causing some delays in jobs processing. More information will be in the incident issue gitlab.com/gitlab-com/gl-infra/production/-/issues/18407

July 2024

GitLab.com is unavailable

July 11, 2024 18:28 UTC

GitLab.com is unavailableService Disruption

Incident Status

Service Disruption


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions, SAML SSO - GitLab SaaS, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, docs.gitlab.com, Canary, Product Analytics - Configurator (Beta), Product Analytics - Collector (Beta), Product Analytics - Background processing (Beta), Product Analytics - Queries (Beta), GitLab Duo


Locations

Google Compute Engine, Digital Ocean, Zendesk, AWS




July 11, 2024 18:28 UTC
[Resolved] This incident is now resolved and no further errors are being seen on GitLab.com. Our investigation points to a disk performance issue. You can see further details and corrective actions in gitlab.com/gitlab-com/gl-infra/production/-/issues/18269.

July 11, 2024 15:51 UTC
[Investigating] Our investigations are ongoing. We will post our next update when we have more information. Please follow the incident issue for updates here: gitlab.com/gitlab-com/gl-infra/production/-/issues/18269

July 11, 2024 15:28 UTC
[Investigating] No material updates to report. We're continuing to investigate the issue. Please follow the incident issue for updates here: gitlab.com/gitlab-com/gl-infra/production/-/issues/18269

July 11, 2024 15:00 UTC
[Investigating] We are seeing improved service availability and investigations continue for this problem. Please follow the incident issue for updates here: gitlab.com/gitlab-com/gl-infra/production/-/issues/18269

July 11, 2024 14:43 UTC
[Investigating] We are seeing improved service availability and investigations continue for this problem. We are planning to make the incident issue publicly available as soon as possible.

July 11, 2024 14:29 UTC
[Investigating] We are investigating a problem with gitlab.com being unavailable for some users. Our investigations continue and we will post further status updates shortly.

July 11, 2024 14:18 UTC
[Investigating] We are investigating a problem with gitlab.com being unavailable for some users. We will post further status updates shortly.

Increase in GitLab.com's Ci jobs processing time

July 2, 2024 16:01 UTC

Incident Status

Operational


Components

Website, API, Background Processing


Locations

Google Compute Engine




July 2, 2024 16:01 UTC
[Resolved] This incident is now mitigated. More details here: gitlab.com/gitlab-com/gl-infra/production/-/issues/18229

July 2, 2024 15:43 UTC
[Monitoring] We rolled back a recent change to mitigate the issue. Performance is now back to normal. More details here: gitlab.com/gitlab-com/gl-infra/production/-/issues/18229

July 2, 2024 15:28 UTC
[Investigating] We're noticing an increase in GitLab.com's Ci jobs processing time. We're now investigating this issue. More details here: gitlab.com/gitlab-com/gl-infra/production/-/issues/18229

July 2, 2024 15:26 UTC
[Investigating] We're noticing an increase in GitLab.com's Ci jobs processing time. We're now investigating this issue. More details here: gitlab.com/gitlab-com/gl-infra/production/-/issues/18229

June 2024

2024-06-11: Intermittent failures in starting up pipeline

June 11, 2024 09:32 UTC

Incident Status

Degraded Performance


Components

Background Processing


Locations

Google Compute Engine




June 11, 2024 09:32 UTC
[Resolved] This incident has been resolved and pipelines are starting up without errors now. More details are available here: gitlab.com/gitlab-com/gl-infra/production/-/issues/18136

June 11, 2024 08:26 UTC
[Monitoring] We are seeing normal functionality again after taking a mitigating action, we will continue monitoring the situation. If you are still seeing these issues, please comment in the production issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/18136

June 11, 2024 08:02 UTC
[Identified] We are still investigating the issue with pipelines not automatically triggering. This issue only affects groups with IP restrictions enabled. For updates, please refer to issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/18136

June 11, 2024 07:46 UTC
[Identified] We are still investigating the issue with pipelines not automatically triggering. This issue only affects groups with IP restrictions enabled. For updates, please refer to issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/18136

June 11, 2024 06:48 UTC
[Identified] We have identified the cause of the pipeline triggering issue and are actively working on a fix. For ongoing updates, please refer to the tracking issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/18136

June 11, 2024 06:18 UTC
[Investigating] We are currently experiencing an issue where pipelines are not automatically triggering as expected. However, they can still be triggered manually. For ongoing updates please refer to the tracking issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/18136

May 2024

Pipeline jobs not being picked up and 500 errors

May 17, 2024 02:14 UTC

Incident Status

Partial Service Disruption


Components

Website, API, CI/CD - Hosted runners on Linux, Background Processing


Locations

Google Compute Engine




May 17, 2024 02:14 UTC
[Resolved] Ongoing monitoring is showing that all systems are working without issue. Google has also resolved their incident. We will mark this incident as resolved. See following issue for more information: gitlab.com/gitlab-com/gl-infra/production/-/issues/18023

May 17, 2024 00:48 UTC
[Monitoring] We are seeing normal functionality again in our logging but we will continue monitoring. If you are still seeing these issues, please comment in the production issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/18023

May 17, 2024 00:24 UTC
[Identified] We have identified that these outages are impacted by the Google Cloud Service disruptions. For more information, see gitlab.com/gitlab-com/gl-infra/production/-/issues/18023. To follow Google Cloud status information, see status.cloud.google.com/incidents/xVSEV3kVaJBmS7SZbnre#RP1d9aZLNFZEJmTBk8e1

May 17, 2024 00:01 UTC
[Investigating] GitLab is investigating reports of pipelines not being picked up and 500 errors. We have two incidents open and we are working to understand the cause of these errors and whether they are related. We are investigating if this is related to this Google issue: status.cloud.google.com/incidents/xVSEV3kVaJBmS7SZbnre#RP1d9aZLNFZEJmTBk8e1 Issue links: gitlab.com/gitlab-com/gl-infra/production/-/issues/18024, gitlab.com/gitlab-com/gl-infra/production/-/issues/18023

March 2024

Vulnerability Report Background Processing Degradation

March 26, 2024 02:28 UTC

Incident Status

Degraded Performance


Components

Background Processing


Locations

Google Compute Engine




March 26, 2024 02:28 UTC
[Resolved] A fix has been deployed, and background processing of vulnerability reports is now operating normally. This incident has been resolved.

March 25, 2024 20:42 UTC
[Identified] We are currently in the process of deploying a solution aimed at resolving this incident. Our next update will be posted in 6 hours. More details in gitlab.com/gitlab-com/gl-infra/production/-/issues/17754

March 25, 2024 14:44 UTC
[Investigating] We're currently investigating issues with the background processing which updates the vulnerability report. The report is not being updated from new pipeline runs as of 13:28 UTC on 2024-03-25. We are working to restore this background processing. Existing report data continues to function as normal but may be outdated. Next update will be posted in 6 hours. More details in - gitlab.com/gitlab-com/gl-infra/production/-/issues/17754

January 2024

Sidekiq degredation

January 30, 2024 23:02 UTC

Sidekiq degredationDegraded Performance

Incident Status

Degraded Performance


Components

Website, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions, Background Processing


Locations

Google Compute Engine




January 30, 2024 23:02 UTC
[Resolved] The feature flag responsible for the issue has been disabled by default and operations are back to normal. This incident is considered resolved. Further details can be found in the incident issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/17504

January 30, 2024 22:02 UTC
[Identified] We've identified the issue and have narrowed the problem to a recent feature flag that created slow queries caused by specific import workers. These slow queries lead to database saturation. To mitigate the issue, the team has disabled the feature flag. We are now re-processing jobs from these workers. This issue is considered mitigated. Further updates will be minimal until there is a permanent code fix in place. More details in gitlab.com/gitlab-com/gl-infra/production/-/issues/17504

January 30, 2024 20:22 UTC
[Investigating] No material update at this time. Our team is still investigating the root cause.

January 30, 2024 19:54 UTC
[Investigating] We've temporarily disabled some features of the GitHub import functionality. Users might experience issues with GitHub imports at this time while our team continues to investigate.

January 30, 2024 19:28 UTC
[Investigating] Sidekiq has largely recovered and jobs are processing. Our team is still investigating the root cause.

January 30, 2024 19:08 UTC
[Investigating] Our team is still investigating the root cause of this issue. We'll continue to provide updates as more information is available.

January 30, 2024 18:51 UTC
[Investigating] We're currently seeing some sidekiq degradation that appears to be impacting some pipelines and merge requests. Our team is investigating. More information in gitlab.com/gitlab-com/gl-infra/production/-/issues/17504

Email delivery delays

January 25, 2024 17:45 UTC

Email delivery delaysDegraded Performance

Incident Status

Degraded Performance


Components

Background Processing


Locations

Google Compute Engine




January 25, 2024 17:45 UTC
[Resolved] No further email delays have been reported by our users while our new IP's continue to warm up for the next couple of days. We consider this incident resolved until new reports are received. See gitlab.com/gitlab-com/gl-infra/production/-/issues/17404 for details.

January 19, 2024 23:34 UTC
[Monitoring] Earlier this week, we added additional IPs to our mail server. As these IPs go through the warm-up phase, we anticipate ongoing improvements. Expect additional updates on Monday, unless notable changes necessitate earlier communication. Details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/17404

January 19, 2024 04:28 UTC
[Monitoring] We continue to monitor email delivery for improvements after mitigation steps have been implemented. Details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/17404

January 19, 2024 04:24 UTC
[Monitoring] We continue to monitor email delivery for improvements after mitigation steps have been implemented. Details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/17404

January 18, 2024 18:15 UTC
[Monitoring] Although there are signs of improvement in email delivery delays, reports of delayed emails from certain users persist. We aim to identify additional options for further mitigation and continue to monitor the situation closely. Details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/17404

January 18, 2024 00:30 UTC
[Monitoring] Indications are that the email delivery delay has improved, however we are still receiving reports of late arriving email by some users. We are continuing our analysis to determine options to further mitigate the delayed delivery.

January 17, 2024 00:31 UTC
[Monitoring] Further monitoring indicates an improvement since the changes were implemented. We will continue to monitor and provide an update once we have concluded a measurable success. Details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/17404

January 16, 2024 23:31 UTC
[Identified] We have initiated a change aimed at resolving the email delays. The full implementation of this change may take 24-48 hours to show results. We will continue to monitor the situation and provide the next update once we confirm the effectiveness of the changes. Details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/17404

January 16, 2024 22:02 UTC
[Identified] We're working on implementing mitigation measures to address the email delays. Our team is monitoring the situation, and we will provide an update as soon as additional information becomes available. Details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/17404

January 16, 2024 19:55 UTC
[Identified] Our team continues to work on resolving this issue as swiftly as possible. We will post the next update when we have significant developments to share. Details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/17404

January 16, 2024 18:44 UTC
[Identified] We are actively working to address the email delivery delays. We appreciate your patience as we continue to work towards a solution. Next update will be in an hour. Details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/17404

January 16, 2024 17:27 UTC
[Identified] We continue to address email delivery delays. While there is no new information to share at this time, we are actively working on a resolution. Another update will be provided in one hour. Details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/17404

January 16, 2024 16:19 UTC
[Identified] We have identified an issue causing delays in email delivery and are actively working to resolve the issue. We anticipate providing an update within the next hour. Details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/17404

January 16, 2024 15:31 UTC
[Investigating] No material updates to report. We are still working with the vendor to identify the cause. Next update will be posted when we know more. For more information: gitlab.com/gitlab-com/gl-infra/production/-/issues/17404

January 16, 2024 15:05 UTC
[Investigating] We are working with the email vendor to identify the cause. For more information: gitlab.com/gitlab-com/gl-infra/production/-/issues/17404

January 16, 2024 14:40 UTC
[Investigating] We are still actively investing reports of delayed emails. More information: gitlab.com/gitlab-com/gl-infra/production/-/issues/17404

January 16, 2024 14:10 UTC
[Investigating] We have identified a problem where emails are being delayed intermittently. More information: gitlab.com/gitlab-com/gl-infra/production/-/issues/17404.

November 2023

Sidekiq processing is delayed

November 23, 2023 19:38 UTC

Sidekiq processing is delayedDegraded Performance

Incident Status

Degraded Performance


Components

Background Processing


Locations

Google Compute Engine




November 23, 2023 19:38 UTC
[Resolved] The backlog of jobs has been completed. Security Scan Result Policies are working as expected. This incident is now considered resolved.

November 23, 2023 15:52 UTC
[Monitoring] Our team has merged a code change for the issue into production and we've re-enabled the affected worker. We are working through a backlog of jobs and will provide an update once it is completed.

November 22, 2023 23:25 UTC
[Monitoring] No material updates at this time. We'll provide additional updates once more information is available.

November 22, 2023 22:17 UTC
[Monitoring] While the issue has been mitigated, our team has disabled a worker that impacts the availability of the Security Scan Result Policies. This can impact the ability to enable/disable and update policies. Our team is working on a code change to fix the root issue.

November 22, 2023 21:18 UTC
[Resolved] The mitigation has been successful and the issue is considered resolved. Our team will continue to work on additional action items. More details in gitlab.com/gitlab-com/gl-infra/production/-/issues/17168

November 22, 2023 21:05 UTC
[Monitoring] Our team has disabled the offending worker and performance is improving. We are continuing to monitor the state of the background processing.

November 22, 2023 20:49 UTC
[Identified] Our team has identified the root cause as related to a specific worker. We are continuing to work on mitigation efforts.

November 22, 2023 20:19 UTC
[Investigating] Our team is still continuing the investigation. Additional details can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/17168

November 22, 2023 19:45 UTC
[Investigating] Our team is continuing to investigate the root cause. We believe to have identified a query that is driving high CPU usage and saturating sidekiq.

November 22, 2023 19:20 UTC
[Investigating] Our team is still investigating the delays with sidekiq processing. This may affect jobs getting picked up, or the ability to create or update merge requests. More details in gitlab.com/gitlab-com/gl-infra/production/-/issues/17168

November 22, 2023 19:06 UTC
[Investigating] Our team is currently investigating some sidekiq performance issues which is causing some delays in jobs processing.





Back to current status