All Systems Operational

Updated a few seconds ago

Back to current status

Status History



October 2024

Sidekiq job delays resulting in updates to MR approval policies being ineffective

October 4, 2024 06:59 UTC

Incident Status

Partial Service Disruption


Components

Website, API


Locations

Google Compute Engine




October 4, 2024 06:59 UTC
[Resolved] Our monitoring shows no further issues after the fix was applied. We are resolving this issue and any further updates will be in: gitlab.com/gitlab-com/gl-infra/production/-/issues/18660

October 3, 2024 22:23 UTC
[Monitoring] The fix has been successfully merged, but we are still waiting for the changes to reach production. We will continue to monitor Sidekiq’s performance, and the next update will be provided once the incident has been mitigated. For more details, visit gitlab.com/gitlab-com/gl-infra/production/-/issues/18660

October 3, 2024 21:04 UTC
[Monitoring] The fix has been successfully merged. We will continue to monitor Sidekiq’s performance to ensure everything is functioning as expected. For details, visit gitlab.com/gitlab-com/gl-infra/production/-/issues/18660

October 3, 2024 20:00 UTC
[Monitoring] The fix is expected to be merged soon. In the meantime, we’ll continue monitoring Sidekiq’s performance. Updates to MR approval policies will remain delayed until the issue is fully resolved. For more details, visit gitlab.com/gitlab-com/gl-infra/production/-/issues/18660

October 3, 2024 19:02 UTC
[Monitoring] We are still awaiting the deployment of the fix and will continue to monitor Sidekiq’s performance in the meantime. Updates to MR approval policies are delayed until the issue is resolved. For more details, visit gitlab.com/gitlab-com/gl-infra/production/-/issues/18660

October 3, 2024 18:02 UTC
[Monitoring] We continue to closely monitor Sidekiq’s performance while awaiting the deployment of the fix. In the meantime, updates to MR approval policies remain delayed. More details about this incident can be found at gitlab.com/gitlab-com/gl-infra/production/-/issues/18660

October 3, 2024 17:06 UTC
[Monitoring] We are actively monitoring the performance of Sidekiq until the fix is deployed. Updates to merge request approval policies are still experiencing delays. Details about this incident can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/18660

October 3, 2024 15:55 UTC
[Monitoring] We're continuing to monitor the performance of Sidekiq until the fix is deployed. Updates to MR approval policies are still processed with delays. The MR to fix the root cause is currently waiting to be merged. More details about this incident can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/18660

October 3, 2024 14:46 UTC
[Monitoring] We're continuing to see improvements and are monitoring until a fix is deployed. Updates to MR approval policies are still processed with delays. More details about this incident can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/18660

October 3, 2024 14:28 UTC
[Monitoring] We've identified the root cause and are recovering from a high volume of request traffic and are seeing improvements. Updates to MR approval policies are currently processing but with delays. More details about this incident can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/18660

October 3, 2024 14:12 UTC
[Investigating] We are currently investigating issues with updating MR Approval policies as a result of Sidekiq performance degradation. More details about this incident can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/18660.

September 2024

Main DB Upgrade

September 26, 2024 03:54 UTC

Main DB UpgradePlanned Maintenance

Description

We will be undergoing scheduled maintenance to our database layer. GitLab.com will continue to be available during the maintenance window. For more details, see gitlab.com/gitlab-com/gl-infra/dbre/-/issues/227


Components

Website, API, Git Operations, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions


Locations

Google Compute Engine, AWS


Schedule

October 25, 2024 00:00 - October 30, 2024 01:00 UTC



September 26, 2024 03:54 UTC
[Update] Scheduled maintenance on October 25 to our database layer has now been cancelled. For more details, see gitlab.com/gitlab-com/gl-infra/dbre/-/issues/227

CI DB upgrade

September 26, 2024 03:50 UTC

CI DB upgradePlanned Maintenance

Description

Next week, we will be undergoing scheduled maintenance to our CI database layer. The maintenance will start at 2024-09-29 00:00 UTC and should finish at 2024-10-02 02:00 UTC (including performance regression observability period). GitLab.com will be available during the whole period as the maintenance should be seamless and transparent for the application. We apologize in advance for any inconvenience this may cause. See <gitlab.com/gitlab-com/gl-infra/production/-/issues/18593>


Components

Website, API, Git Operations, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions


Locations

Google Compute Engine, AWS


Schedule

September 27, 2024 00:00 - October 2, 2024 01:00 UTC



September 26, 2024 03:50 UTC
[Update] Maintenance scheduled for 2024-09-29 on our CI database layer, has now been cancelled. For more information: gitlab.com/gitlab-com/gl-infra/production/-/issues/18593

Duo Availability Degraded

September 25, 2024 20:03 UTC

Duo Availability DegradedDegraded Performance

Incident Status

Degraded Performance


Components

GitLab Duo


Locations

Google Compute Engine




September 25, 2024 20:03 UTC
[Resolved] Our team has not seen any further issues. This incident is now resolved. More details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/18620

September 25, 2024 18:31 UTC
[Monitoring] GitLab has seen error rates from GitLab DUO going down after an upstream incident. We are continuing to monitor until we are confident the issues are resolved. More details here: gitlab.com/gitlab-com/gl-infra/production/-/issues/18620

September 25, 2024 15:36 UTC
[Identified] We are experiencing errors on GitLab DUO features related to the ongoing Anthropic incident. Find more details here: gitlab.com/gitlab-com/gl-infra/production/-/issues/18620

Slowness on GitLab.com

September 21, 2024 17:55 UTC

Slowness on GitLab.comPartial Service Disruption

Incident Status

Partial Service Disruption


Components

Website, API, Git Operations


Locations

Google Compute Engine




September 21, 2024 17:55 UTC
[Resolved] Our monitoring has show no further issues after our change. We are resolving this issue and any further updates will be in: gitlab.com/gitlab-com/gl-infra/production/-/issues/18596

September 21, 2024 14:44 UTC
[Monitoring] We have identified the issue, and have made a change. We continue to see improvements. We will now continue to monitor, and if nothing changes, the next update can be expected in 3 hours. More details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/18596

September 21, 2024 14:09 UTC
[Identified] We are recovering from a high volume of request traffic and are seeing improvements. More information in: gitlab.com/gitlab-com/gl-infra/production/-/issues/18596

September 21, 2024 13:45 UTC
[Investigating] We are seeing slow responses from our API, and we are still investigating. More information in: gitlab.com/gitlab-com/gl-infra/production/-/issues/18596

September 21, 2024 13:27 UTC
[Investigating] No material updates at this point. Our Engineers are still investigating. More information in: gitlab.com/gitlab-com/gl-infra/production/-/issues/18596

September 21, 2024 13:10 UTC
[Investigating] We are currently investigating problems with slowness on GitLab.com. More information in: gitlab.com/gitlab-com/gl-infra/production/-/issues/18596

Intermittent 503 errors

September 20, 2024 21:40 UTC

Incident Status

Operational


Components

Website, API, Canary


Locations

Google Compute Engine




September 20, 2024 21:40 UTC
[Resolved] GitLab has not seen any further errors related to this incident in our monitoring and we are resolving the incident. Any further updates are in the incident issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/18591.

September 20, 2024 20:53 UTC
[Monitoring] Our rollout has completed and we believe that the issue is mitigated. We are moving the incident to Monitoring to validate the fix. gitlab.com/gitlab-com/gl-infra/production/-/issues/18591

September 20, 2024 15:08 UTC
[Identified] No material updates to report. The roll out of the fix is still ongoing. More details are available in gitlab.com/gitlab-com/gl-infra/production/-/issues/18591.

September 20, 2024 13:54 UTC
[Identified] We are seeing an uptick in errors with git operations. The roll out of the fix is still ongoing. More details are available in gitlab.com/gitlab-com/gl-infra/production/-/issues/18591.

September 20, 2024 13:12 UTC
[Identified] We are seeing an increased number of 503 errors in the UI and API. We have identified the issue and are rolling out a fix. More details are available in gitlab.com/gitlab-com/gl-infra/production/-/issues/18591.

"View job currently using resource" button returns 500 error

September 20, 2024 19:10 UTC

Incident Status

Degraded Performance


Components

CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions


Locations

Google Compute Engine, AWS




September 20, 2024 19:10 UTC
[Resolved] GitLab has verified we are no longer seeing 500 errors from the View Job button. We are resolving the incident. Any further updates will be posted in the incident issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/18530

September 17, 2024 02:35 UTC
[Monitoring] A fix has been deployed and is being verified, further updates will be provided upon completion. Details available in: gitlab.com/gitlab-com/gl-infra/production/-/issues/18530

September 12, 2024 20:47 UTC
[Identified] The fix is known and will be deployed soon. There are some existing pipelines that are still affected, and our team is working on resolving them. Affected users can view the open issue here: gitlab.com/gitlab-com/gl-infra/production/-/issues/18530

September 11, 2024 20:26 UTC
[Identified] No updates at this time. There are some existing pipelines that are still affected, and our team is working on resolving them. Affected users can view the open issue here: gitlab.com/gitlab-com/gl-infra/production/-/issues/18530

September 11, 2024 17:40 UTC
[Identified] A fix has been deployed to production. New pipelines are unaffected. However, there are some existing pipelines that are still affected, and our team is working on resolving them. Affected users can view the open issue here: gitlab.com/gitlab-com/gl-infra/production/-/issues/18530

September 11, 2024 09:32 UTC
[Resolved] This incident has been resolved. Affected users can view the open issue here: gitlab.com/gitlab-com/gl-infra/production/-/issues/18530 If you have CI pipelines that are stuck as a result of this issue, please cancel them so they don't continue to use up your CI minutes quota. We’re actively looking into remediating CI minutes usage for the impacted jobs. If you believe you are impacted by this incident, please open a support ticket here: support.gitlab.com

September 10, 2024 19:33 UTC
[Identified] A fix has been deployed to production. New pipelines are unaffected. However, there are some existing pipelines that are still affected, and our team is working on resolving them. Affected users can view the open issue here: gitlab.com/gitlab-org/gitlab/-/issues/483290 If you continue to see the issue, please contact support here: support.gitlab.com

September 9, 2024 22:25 UTC
[Identified] At this time there are no further updates. Affected users can view the open issue here: gitlab.com/gitlab-org/gitlab/-/issues/483290 If you deleted a pipeline that had a resource group and are now seeing stuck jobs, please contact support.

September 9, 2024 20:56 UTC
[Identified] Some users are seeing 500 errors when clicking the "View job currently using resource" button. Users may also see jobs stuck in "canceling" state after deleting a pipeline that was deploying to a resource group. The fix is already in the works. Affected users can view the open issue here: gitlab.com/gitlab-org/gitlab/-/issues/483290 If you deleted a pipeline that had a resource group and are now seeing stuck jobs, please contact support.

Slowness on GitLab.com

September 17, 2024 02:19 UTC

Incident Status

Operational


Components

Website


Locations

Google Compute Engine




September 17, 2024 02:19 UTC
[Resolved] This issue is now resolved. Details can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/18562.

September 13, 2024 21:21 UTC
[Monitoring] No further updates at this time. We're monitoring the situation. Please see the issue here: gitlab.com/gitlab-com/gl-infra/production/-/issues/18562

September 13, 2024 19:46 UTC
[Monitoring] The cause of the slowness has been identified and corrected. We're monitoring the situation. Please see the issue here: gitlab.com/gitlab-com/gl-infra/production/-/issues/18562

September 13, 2024 19:03 UTC
[Investigating] We've received reports of slowness on GitLab.com. We're currently investigating. Please see the issue here: gitlab.com/gitlab-com/gl-infra/production/-/issues/18562

September 13, 2024 19:03 UTC
[Investigating] We've received reports of slowness on GitLab.com. We're currently investigating. Please see the issue here: gitlab.com/gitlab-com/gl-infra/production/-/issues/18562

PDF rendering issues on GitLab.com

September 16, 2024 17:56 UTC

Incident Status

Degraded Performance


Components

Website


Locations

Google Compute Engine




September 16, 2024 17:56 UTC
[Resolved] GitLab has resolved PDF rendering issues. We are following up to prevent this kind of issue in the future; futher details in gitlab.com/gitlab-com/gl-infra/production/-/issues/18574

September 16, 2024 17:50 UTC
[Investigating] GitLab is investigating PDF rendering issues on GitLab.com. gitlab.com/gitlab-com/gl-infra/production/-/issues/18574

Missing Access Tokens

September 12, 2024 06:53 UTC

Missing Access TokensPartial Service Disruption

Incident Status

Partial Service Disruption


Components

API


Locations

Google Compute Engine




September 12, 2024 06:53 UTC
[Resolved] We have reached out to all affected customers on remedying the problem. We apologize for any inconvenience this may cause. This incident is now marked as Resolved. More information can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/18548.

September 11, 2024 20:16 UTC
[Monitoring] No updates to report at this time. We are currently reaching out to customers impacted as to remedy the problem. More details can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/18548.

September 11, 2024 12:11 UTC
[Monitoring] We've resolved the issue with the cron job responsible for revoking tokens. We are proactively reaching out to customers impacted as to remedy the problem. More details can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/18548.

September 11, 2024 11:53 UTC
[Identified] We've identified cause of the issue and have stopped the cron job responsible. To resolve the issue please recreate the token. More details can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/18548.

September 11, 2024 11:29 UTC
[Investigating] We are currently investigating issues with missing access tokens. More details about this incident can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/18548

Registry DB upgrade

September 11, 2024 22:51 UTC

Registry DB upgradePlanned Maintenance

Description

We will be undergoing scheduled maintenance to our database layer. GitLab.com will continue to be available during the maintenance window. For more details, see gitlab.com/gitlab-com/gl-infra/dbre/-/issues/225


Components

Website, API, Git Operations, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions


Locations

Google Compute Engine, AWS


Schedule

September 6, 2024 00:00 - September 11, 2024 01:00 UTC



September 11, 2024 22:51 UTC
[Update] Planned maintenance for the registry database layer on GitLab.com is complete. Registry database is now running Postgres v16. The rollback window for the Registry database was closed on approximately 2024-09-12 1:00 UTC.

September 8, 2024 01:51 UTC
[Update] Planned maintenance for the registry database layer on GitLab.com is complete. Registry database is now running Postgres v16. Rollback window for this planned maintenance ends 2024-09-11 00:00 UTC. We'll be monitoring the platform during this time to ensure all systems are functioning correctly.

September 8, 2024 00:11 UTC
[Update] Scheduled maintenance to our registry database layer has started. The maintenance should finish at 2024-09-11 02:00 UTC UTC (including performance regression observability period). GitLab.com will be available during the whole period as the maintenance should be seamless and transparent for the application. We apologize in advance for any inconvenience this may cause. See issue for more information: gitlab.com/gitlab-com/gl-infra/production/-/issues/18447

September 5, 2024 05:53 UTC
[Update] In 3 days, we will be undergoing scheduled maintenance to our registry database layer. The maintenance will start at 2024-09-08 00:00 UTC UTC and should finish at 2024-09-11 02:00 UTC UTC (including performance regression observability period). GitLab.com will be available during the whole period as the maintenance should be seamless and transparent for the application. We apologize in advance for any inconvenience this may cause. See <gitlab.com/gitlab-com/gl-infra/production/-/issues/18447>

Some jobs to running longer than the defined timeout

September 11, 2024 09:37 UTC

Incident Status

Operational


Components

CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions


Locations

Google Compute Engine, AWS




September 11, 2024 09:37 UTC
[Resolved] This incident has been resolved. Affected users can view the open issue here: gitlab.com/gitlab-com/gl-infra/production/-/issues/18533 If you have CI pipelines that are stuck as a result of this issue, please cancel them so they don't continue to use up your CI minutes quota. We’re actively looking into remediating CI minutes usage for the impacted jobs. If you believe you are impacted by this incident, please open a support ticket here: support.gitlab.com

September 10, 2024 22:28 UTC
[Monitoring] An incident earlier today caused some jobs to run longer than the defined timeout. This lead to excessive usage of CI minutes. Affected users can view the open issue here: gitlab.com/gitlab-com/gl-infra/production/-/issues/18533 If you have CI pipelines that are stuck as a result of this issue, please cancel them so they don't continue to use up your CI minutes quota. We’re actively looking into remediating CI minutes usage for the impacted jobs. If you believe you are impacted by this incident, please open a support ticket here: support.gitlab.com

September 10, 2024 21:48 UTC
[Monitoring] An incident earlier today caused some jobs to run longer than the defined timeout. This lead to excessive usage of CI minutes. Affected users can view the open issue here: gitlab.com/gitlab-com/gl-infra/production/-/issues/18533 We’re actively looking into remediating CI minutes usage for the impacted jobs. If you believe you are impacted by this incident, please open a support ticket. support.gitlab.com

GitLab slow background processing

September 10, 2024 17:22 UTC

Incident Status

Degraded Performance


Components

Background Processing, GitLab Duo


Locations

Google Compute Engine




September 10, 2024 17:22 UTC
[Resolved] This incident is now resolved. The background jobs queue is now empty, and all stuck jobs should be being processed. If you still see any stuck jobs, restarting the job should work to unblock it. More details about this incident can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/18538.

September 10, 2024 17:02 UTC
[Monitoring] The background jobs queue is now empty. All stuck jobs should be being processed. If you still see any stuck jobs, restarting the job should work to unblock it. We are monitoring the current status of the application. More details about this incident can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/18538.

September 10, 2024 15:47 UTC
[Identified] We are working on resolving the background processing issues. GitLab Duo responses might be affected due to slow background processing. More details about this incident can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/18538.

September 10, 2024 15:14 UTC
[Investigating] No material updates at this time. More details can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/18538.

September 10, 2024 14:57 UTC
[Investigating] GitLab CI functionality is partially unavailable right now due to issues with background processing. We are working on a mitigation. More details can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/18538.

September 10, 2024 14:31 UTC
[Investigating] We are currently investigating issues with background processing and working on a temporary mitigation strategy. More details can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/18538.

September 10, 2024 14:08 UTC
[Investigating] We are currently investigating issues with background processing. More details about this incident can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/18538.

GitLab connectivity issues

September 10, 2024 13:40 UTC

Incident Status

Operational


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions


Locations

Google Compute Engine




September 10, 2024 13:40 UTC
[Resolved] This incident has been resolved. More information can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/18535.

September 10, 2024 13:33 UTC
[Monitoring] The website has recovered are and we are now monitoring before marking as resolved. More details can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/18535.

September 10, 2024 13:24 UTC
[Investigating] No material updates at this time - our investigation is ongoing. More details can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/18535

September 10, 2024 13:10 UTC
[Investigating]

September 10, 2024 13:08 UTC
[Investigating] We are currently investigating connectivity issues to GitLab.com. For more information see gitlab.com/gitlab-com/gl-infra/production/-/issues/18535

Some projects and repositories are not accessible

September 6, 2024 11:19 UTC

Incident Status

Partial Service Disruption


Components

Website


Locations

Google Compute Engine




September 6, 2024 11:19 UTC
[Resolved] This incident has been resolved and projects are accessible. More details can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/18514.

September 6, 2024 10:46 UTC
[Monitoring] We have taken action to mitigate the issue and are now monitoring before marking as resolved. More details about this incident can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/18514.

September 6, 2024 10:35 UTC
[Identified] We've identified the cause of the issue and are working on resolving it. More details about this incident can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/18514.

September 6, 2024 10:15 UTC
[Investigating] We are currently investigating issues where some projects and repositories are not accessible. More details about this incident can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/18514.

Customers Portal is down (customers.gitlab.com)

September 4, 2024 20:07 UTC

Incident Status

Service Disruption


Components

GitLab Customers Portal


Locations

Google Compute Engine




September 4, 2024 20:07 UTC
[Resolved] Functionality has been restored to the customer portal (customers.gitlab.com).

September 4, 2024 19:46 UTC
[Monitoring] A fix has been implemented and we are monitoring the results.

September 4, 2024 19:04 UTC
[Investigating] The Customers Portal (customers.gitlab.com) is in maintenance mode due to a 3rd-party API being down. The billing pages in GitLab.com may be affected.

GitLab pipelines experiencing longer than expected execution times

September 4, 2024 05:57 UTC

Incident Status

Operational


Components

Background Processing, GitLab Duo


Locations

Google Compute Engine




September 4, 2024 05:57 UTC
[Resolved] Pipeline functionality has been restored and is working as expected based on our monitoring. We are marking this incident as resolved. For more updates, please follow: gitlab.com/gitlab-com/gl-infra/production/-/issues/18489 Additionally, we have reset the compute monthly minutes due to this incident. For details on this action, see: gitlab.com/gitlab-com/gl-infra/production/-/issues/18497

September 4, 2024 00:13 UTC
[Monitoring] Pipeline functionality has been restored. We have opted to reset the compute monthly minutes due to this incident. That work has started and you can view it here: gitlab.com/gitlab-com/gl-infra/production/-/issues/18497. See incident issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/18489

September 3, 2024 23:12 UTC
[Monitoring] Pipeline functionality has been restored. We have opted to reset the compute monthly min due to this incident. Please view the change request here. gitlab.com/gitlab-com/gl-infra/production/-/issues/18497. See incident issue : gitlab.com/gitlab-com/gl-infra/production/-/issues/18489

September 3, 2024 22:02 UTC
[Monitoring] Pipeline functionality has been restored. We have opted to reset the compute monthly min due to this incident. Please view the change request here. gitlab.com/gitlab-com/gl-infra/production/-/issues/18497.

September 3, 2024 19:46 UTC
[Monitoring] We have mitigation in place and pipelines are resolving normally. Please See the issue for more information: gitlab.com/gitlab-com/gl-infra/production/-/issues/18489

September 3, 2024 18:20 UTC
[Monitoring] We have mitigation in place and pipelines are resolving normally. Please See the issue for more information: gitlab.com/gitlab-com/gl-infra/production/-/issues/18489

September 3, 2024 18:04 UTC
[Monitoring] We are continuing to monitor and we recommend cancelling jobs that may have become stuck during the incident and we should see them process. See the issue for more information: gitlab.com/gitlab-com/gl-infra/production/-/issues/18489

September 3, 2024 16:57 UTC
[Monitoring] We are continuing to monitor and we recommend cancelling jobs that may have become stuck during the incident and we should see them process. See the issue for more information: gitlab.com/gitlab-com/gl-infra/production/-/issues/18489

September 3, 2024 15:41 UTC
[Monitoring] No material updates at this time. We are continuing to monitor and we recommend cancelling jobs that may have become stuck during the incident. See the issue for more information: gitlab.com/gitlab-com/gl-infra/production/-/issues/18489

September 3, 2024 13:59 UTC
[Monitoring] We are continuing to monitor. We recommend cancelling jobs that may have become stuck during the incident. See the issue for more information: gitlab.com/gitlab-com/gl-infra/production/-/issues/18489

September 3, 2024 12:26 UTC
[Monitoring] Background job processing has been restored. We are continuing to monitor while we investigate further into the root cause. See the issue for more information: gitlab.com/gitlab-com/gl-infra/production/-/issues/18489

September 3, 2024 11:40 UTC
[Investigating] No material updates at this time. We are continuing to investigate and monitor background job processing. See the issue for more information: gitlab.com/gitlab-com/gl-infra/production/-/issues/18489

September 3, 2024 10:59 UTC
[Investigating] We are seeing improvements in background job processing and operations are recovering back to normal. Delays in email processing are still possible, and we are continuing with our investigation. See the issue for more information: gitlab.com/gitlab-com/gl-infra/production/-/issues/18489

September 3, 2024 10:04 UTC
[Investigating] No current updates at this time. We are still working to isolate the problem but took measures to add relief. We will provide an update once the root cause has been identified. See the issue for more information: gitlab.com/gitlab-com/gl-infra/production/-/issues/18489.

September 3, 2024 09:38 UTC
[Investigating] We are still working to isolate the problem to a specific background job. See the issue for more information: gitlab.com/gitlab-com/gl-infra/production/-/issues/18489.

September 3, 2024 09:15 UTC
[Investigating] We are trying to isolate the problem to a number of background jobs contributing to this problem and we are taking them offline for some time. AI feature such as Duo Chat may also be impacted by this. See the issue for more information: gitlab.com/gitlab-com/gl-infra/production/-/issues/18489.

September 3, 2024 08:52 UTC
[Investigating] No material updates at this time - our investigation into Sidekiq issues is ongoing. See the issue for more information: gitlab.com/gitlab-com/gl-infra/production/-/issues/18489.

September 3, 2024 08:37 UTC
[Investigating] We are investigating issues with Sidekiq. See the issue for more information: gitlab.com/gitlab-com/gl-infra/production/-/issues/18489.

September 3, 2024 08:13 UTC
[Identified] We hare still investigating the issue. The background processes may take longer to be picked up and executed than usual. This can result in emails not being delivered and commits not showing up in merge request activity. CI/CD jobs may be stuck in pending state. Issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/18489

September 3, 2024 07:51 UTC
[Investigating] We are still investigating the cause of extended GitLab pipeline execution times. Our team continues to work on resolving this issue. We appreciate your patience and will update when we have more information. Issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/18489

September 3, 2024 07:18 UTC
[Investigating] We are currently investigating reports of GitLab pipelines taking longer than usual to complete. Our team is aware of the issue and actively working to identify the cause of these delays. Issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/18489

GitLab.com Connectivity Issues

September 3, 2024 05:34 UTC

Incident Status

Service Disruption


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions, SAML SSO - GitLab SaaS, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, docs.gitlab.com, Canary, Product Analytics - Configurator (Beta), Product Analytics - Collector (Beta), Product Analytics - Background processing (Beta), Product Analytics - Queries (Beta), GitLab Duo


Locations

Google Compute Engine, Digital Ocean, Zendesk, AWS




September 3, 2024 05:34 UTC
[Resolved] All GitLab services have been restored. We are actively monitoring the system, and everything appears normal at this time. For more details, please follow the issue at: gitlab.com/gitlab-com/gl-infra/production/-/issues/18490

September 3, 2024 05:19 UTC
[Monitoring] We've identified and addressed the root cause of the issue. All GitLab services have been restored, and we are actively monitoring the system. Follow the issue for more details : gitlab.com/gitlab-com/gl-infra/production/-/issues/18490

September 3, 2024 05:09 UTC
[Investigating] GitLab.com is currently experiencing connectivity issues due to a high error rate. Our infrastructure team is investigating. Issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/18490

August 2024

Embedding DB Upgrade

August 31, 2024 04:58 UTC

Embedding DB UpgradePlanned Maintenance

Description

We will be undergoing scheduled maintenance to our database layer. GitLab.com will continue to be available during the maintenance window. For more details, see gitlab.com/gitlab-com/gl-infra/dbre/-/issues/228


Components

Website, API, Git Operations, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions


Locations

Google Compute Engine, AWS


Schedule

October 25, 2024 00:00 - October 30, 2024 01:00 UTC



August 31, 2024 04:58 UTC
[Update] We are cancelling the Embedding database upgrade. See gitlab.com/gitlab-com/gl-infra/dbre/-/issues/228 for details. All other DB upgrade schedules will go ahead as planned.

CI/CD token allowlist selection failing

August 27, 2024 14:52 UTC

CI/CD token allowlist selection failingPartial Service Disruption

Incident Status

Partial Service Disruption


Components

Website, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions


Locations

Google Compute Engine, AWS




August 27, 2024 14:52 UTC
[Resolved] The revert MR has been deployed to production, and we see that CI/CD token allowlist project selection renders properly now. Details will be provided in the incident issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/18446

August 27, 2024 03:55 UTC
[Identified] We have identified an issue that is causing CI/CD token allowlist project selection to fail. We are in the process of reverting the change that caused this. For more details see gitlab.com/gitlab-com/gl-infra/production/-/issues/18446.





Back to current status