Active Issue

Updated a few seconds ago

Back to current status

Status History

Filter: GitLab Duo (Clear)



January 2025

Accessing gitlab.com results in 500 errors

January 6, 2025 16:42 UTC

Incident Status

Partial Service Disruption


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions, SAML SSO - GitLab SaaS, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, docs.gitlab.com, Canary, Product Analytics - Configurator (Beta), Product Analytics - Collector (Beta), Product Analytics - Background processing (Beta), Product Analytics - Queries (Beta), GitLab Duo


Locations

Google Compute Engine, Digital Ocean, Zendesk, AWS




January 6, 2025 16:42 UTC
[Resolved] We've identified the cause of the errors and the errors have returned to normal. The incident is now resolved. More information on the incident can be found in this issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/19079

January 6, 2025 15:57 UTC
[Monitoring] We've identified the cause of the errors, and the errors have returned to normal. The incident is now mitigated, but we're continuing to monitor for any further impact. More information on the incident can be found in this issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/19079

January 6, 2025 15:27 UTC
[Monitoring] We're continuing to monitor for further errors related to the incident. We see further decreases in the errors.

January 6, 2025 15:17 UTC
[Monitoring] We saw elevated 500 errors on gitlab.com which have then decreased. We're continuing to monitor for further errors

December 2024

Duo Chat is not working for most cases in VSCode and JetBrains

December 11, 2024 13:37 UTC

Incident Status

Partial Service Disruption


Components

GitLab Duo


Locations

Google Compute Engine




December 11, 2024 13:37 UTC
[Resolved] This incident has been resolved and Duo Chat in IDEs is now fully operational. More information can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/18980

December 11, 2024 11:51 UTC
[Monitoring] The fix has been deployed to production and we are no longer seeing these errors. We will continue monitoring. More details gitlab.com/gitlab-com/gl-infra/production/-/issues/18980

December 11, 2024 11:05 UTC
[Identified] The deployment to production has started and we are expecting it to complete in approximately 1 hour. See gitlab.com/gitlab-com/gl-infra/production/-/issues/18980

December 11, 2024 02:43 UTC
[Identified] No material update to report. Working toward getting identified fix applied. ETA 6-8 hours. See gitlab.com/gitlab-com/gl-infra/production/-/issues/18980

December 10, 2024 22:26 UTC
[Identified] This update is to clarify the scope of affected users. The incident only affects workflows configured using OAuth authentication. Users who authenticate using an access token remain unaffected.

December 10, 2024 21:17 UTC
[Identified] Our team has identified the issue and is working on a fix. Duo Chat may be temporarily unavailable in editor extensions until a fix is merged. We will provide an update once the fix has been applied.

December 10, 2024 19:31 UTC
[Investigating] Our team is still investigating the issue. We'll provide additional updates as more information becomes available.

December 10, 2024 16:52 UTC
[Investigating] Current impact is identifying that Duo Chat is not working for most cases in VSCode and JetBrains. We're still investigating and will provide updates as more information becomes available.

December 10, 2024 16:47 UTC
[Investigating] No material updates at this time. Our team is continuing to investigate the issue.

December 10, 2024 16:17 UTC
[Investigating] No material updates at this time. Our team is still investigating the issue.

December 10, 2024 15:55 UTC
[Investigating] Users are experiencing issues accessing Duo Chat functionality within VSCode. Our engineering team is actively investigating the root cause in gitlab.com/gitlab-com/gl-infra/production/-/issues/18980

November 2024

Elevated errors in Duo Chat

November 15, 2024 09:20 UTC

Elevated errors in Duo ChatDegraded Performance

Incident Status

Degraded Performance


Components

GitLab Duo


Locations

Google Compute Engine




November 15, 2024 09:20 UTC
[Resolved] The revert MR has now been deployed to production and Duo Chat is working as expected Details to be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/18858

November 15, 2024 01:05 UTC
[Identified] The revert MR has been merged and is awaiting deployment to production. A further update will be provided once deployment has completed. Details to be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/18858

November 15, 2024 00:39 UTC
[Identified] The root cause has been identified and a reversion MR is being prepared. Further details can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/18858

November 15, 2024 00:26 UTC
[Investigating] We are currently investigating errors from Duo Chat (A9999). Further details can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/18858

September 2024

Duo Availability Degraded

September 25, 2024 20:03 UTC

Duo Availability DegradedDegraded Performance

Incident Status

Degraded Performance


Components

GitLab Duo


Locations

Google Compute Engine




September 25, 2024 20:03 UTC
[Resolved] Our team has not seen any further issues. This incident is now resolved. More details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/18620

September 25, 2024 18:31 UTC
[Monitoring] GitLab has seen error rates from GitLab DUO going down after an upstream incident. We are continuing to monitor until we are confident the issues are resolved. More details here: gitlab.com/gitlab-com/gl-infra/production/-/issues/18620

September 25, 2024 15:36 UTC
[Identified] We are experiencing errors on GitLab DUO features related to the ongoing Anthropic incident. Find more details here: gitlab.com/gitlab-com/gl-infra/production/-/issues/18620

GitLab slow background processing

September 10, 2024 17:22 UTC

Incident Status

Degraded Performance


Components

Background Processing, GitLab Duo


Locations

Google Compute Engine




September 10, 2024 17:22 UTC
[Resolved] This incident is now resolved. The background jobs queue is now empty, and all stuck jobs should be being processed. If you still see any stuck jobs, restarting the job should work to unblock it. More details about this incident can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/18538.

September 10, 2024 17:02 UTC
[Monitoring] The background jobs queue is now empty. All stuck jobs should be being processed. If you still see any stuck jobs, restarting the job should work to unblock it. We are monitoring the current status of the application. More details about this incident can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/18538.

September 10, 2024 15:47 UTC
[Identified] We are working on resolving the background processing issues. GitLab Duo responses might be affected due to slow background processing. More details about this incident can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/18538.

September 10, 2024 15:14 UTC
[Investigating] No material updates at this time. More details can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/18538.

September 10, 2024 14:57 UTC
[Investigating] GitLab CI functionality is partially unavailable right now due to issues with background processing. We are working on a mitigation. More details can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/18538.

September 10, 2024 14:31 UTC
[Investigating] We are currently investigating issues with background processing and working on a temporary mitigation strategy. More details can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/18538.

September 10, 2024 14:08 UTC
[Investigating] We are currently investigating issues with background processing. More details about this incident can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/18538.

GitLab pipelines experiencing longer than expected execution times

September 4, 2024 05:57 UTC

Incident Status

Operational


Components

Background Processing, GitLab Duo


Locations

Google Compute Engine




September 4, 2024 05:57 UTC
[Resolved] Pipeline functionality has been restored and is working as expected based on our monitoring. We are marking this incident as resolved. For more updates, please follow: gitlab.com/gitlab-com/gl-infra/production/-/issues/18489 Additionally, we have reset the compute monthly minutes due to this incident. For details on this action, see: gitlab.com/gitlab-com/gl-infra/production/-/issues/18497

September 4, 2024 00:13 UTC
[Monitoring] Pipeline functionality has been restored. We have opted to reset the compute monthly minutes due to this incident. That work has started and you can view it here: gitlab.com/gitlab-com/gl-infra/production/-/issues/18497. See incident issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/18489

September 3, 2024 23:12 UTC
[Monitoring] Pipeline functionality has been restored. We have opted to reset the compute monthly min due to this incident. Please view the change request here. gitlab.com/gitlab-com/gl-infra/production/-/issues/18497. See incident issue : gitlab.com/gitlab-com/gl-infra/production/-/issues/18489

September 3, 2024 22:02 UTC
[Monitoring] Pipeline functionality has been restored. We have opted to reset the compute monthly min due to this incident. Please view the change request here. gitlab.com/gitlab-com/gl-infra/production/-/issues/18497.

September 3, 2024 19:46 UTC
[Monitoring] We have mitigation in place and pipelines are resolving normally. Please See the issue for more information: gitlab.com/gitlab-com/gl-infra/production/-/issues/18489

September 3, 2024 18:20 UTC
[Monitoring] We have mitigation in place and pipelines are resolving normally. Please See the issue for more information: gitlab.com/gitlab-com/gl-infra/production/-/issues/18489

September 3, 2024 18:04 UTC
[Monitoring] We are continuing to monitor and we recommend cancelling jobs that may have become stuck during the incident and we should see them process. See the issue for more information: gitlab.com/gitlab-com/gl-infra/production/-/issues/18489

September 3, 2024 16:57 UTC
[Monitoring] We are continuing to monitor and we recommend cancelling jobs that may have become stuck during the incident and we should see them process. See the issue for more information: gitlab.com/gitlab-com/gl-infra/production/-/issues/18489

September 3, 2024 15:41 UTC
[Monitoring] No material updates at this time. We are continuing to monitor and we recommend cancelling jobs that may have become stuck during the incident. See the issue for more information: gitlab.com/gitlab-com/gl-infra/production/-/issues/18489

September 3, 2024 13:59 UTC
[Monitoring] We are continuing to monitor. We recommend cancelling jobs that may have become stuck during the incident. See the issue for more information: gitlab.com/gitlab-com/gl-infra/production/-/issues/18489

September 3, 2024 12:26 UTC
[Monitoring] Background job processing has been restored. We are continuing to monitor while we investigate further into the root cause. See the issue for more information: gitlab.com/gitlab-com/gl-infra/production/-/issues/18489

September 3, 2024 11:40 UTC
[Investigating] No material updates at this time. We are continuing to investigate and monitor background job processing. See the issue for more information: gitlab.com/gitlab-com/gl-infra/production/-/issues/18489

September 3, 2024 10:59 UTC
[Investigating] We are seeing improvements in background job processing and operations are recovering back to normal. Delays in email processing are still possible, and we are continuing with our investigation. See the issue for more information: gitlab.com/gitlab-com/gl-infra/production/-/issues/18489

September 3, 2024 10:04 UTC
[Investigating] No current updates at this time. We are still working to isolate the problem but took measures to add relief. We will provide an update once the root cause has been identified. See the issue for more information: gitlab.com/gitlab-com/gl-infra/production/-/issues/18489.

September 3, 2024 09:38 UTC
[Investigating] We are still working to isolate the problem to a specific background job. See the issue for more information: gitlab.com/gitlab-com/gl-infra/production/-/issues/18489.

September 3, 2024 09:15 UTC
[Investigating] We are trying to isolate the problem to a number of background jobs contributing to this problem and we are taking them offline for some time. AI feature such as Duo Chat may also be impacted by this. See the issue for more information: gitlab.com/gitlab-com/gl-infra/production/-/issues/18489.

September 3, 2024 08:52 UTC
[Investigating] No material updates at this time - our investigation into Sidekiq issues is ongoing. See the issue for more information: gitlab.com/gitlab-com/gl-infra/production/-/issues/18489.

September 3, 2024 08:37 UTC
[Investigating] We are investigating issues with Sidekiq. See the issue for more information: gitlab.com/gitlab-com/gl-infra/production/-/issues/18489.

September 3, 2024 08:13 UTC
[Identified] We hare still investigating the issue. The background processes may take longer to be picked up and executed than usual. This can result in emails not being delivered and commits not showing up in merge request activity. CI/CD jobs may be stuck in pending state. Issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/18489

September 3, 2024 07:51 UTC
[Investigating] We are still investigating the cause of extended GitLab pipeline execution times. Our team continues to work on resolving this issue. We appreciate your patience and will update when we have more information. Issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/18489

September 3, 2024 07:18 UTC
[Investigating] We are currently investigating reports of GitLab pipelines taking longer than usual to complete. Our team is aware of the issue and actively working to identify the cause of these delays. Issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/18489

GitLab.com Connectivity Issues

September 3, 2024 05:34 UTC

Incident Status

Service Disruption


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions, SAML SSO - GitLab SaaS, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, docs.gitlab.com, Canary, Product Analytics - Configurator (Beta), Product Analytics - Collector (Beta), Product Analytics - Background processing (Beta), Product Analytics - Queries (Beta), GitLab Duo


Locations

Google Compute Engine, Digital Ocean, Zendesk, AWS




September 3, 2024 05:34 UTC
[Resolved] All GitLab services have been restored. We are actively monitoring the system, and everything appears normal at this time. For more details, please follow the issue at: gitlab.com/gitlab-com/gl-infra/production/-/issues/18490

September 3, 2024 05:19 UTC
[Monitoring] We've identified and addressed the root cause of the issue. All GitLab services have been restored, and we are actively monitoring the system. Follow the issue for more details : gitlab.com/gitlab-com/gl-infra/production/-/issues/18490

September 3, 2024 05:09 UTC
[Investigating] GitLab.com is currently experiencing connectivity issues due to a high error rate. Our infrastructure team is investigating. Issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/18490

August 2024

Some requests timing out

August 23, 2024 03:59 UTC

Incident Status

Operational


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions, SAML SSO - GitLab SaaS, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, docs.gitlab.com, Canary, Product Analytics - Configurator (Beta), Product Analytics - Collector (Beta), Product Analytics - Background processing (Beta), Product Analytics - Queries (Beta), GitLab Duo


Locations

Google Compute Engine, Digital Ocean, Zendesk, AWS




August 23, 2024 03:59 UTC
[Resolved] The issue has been resolved and all functionality on GitLab.com is nominal. Please see gitlab.com/gitlab-com/gl-infra/production/-/issues/18435 for more details.

August 23, 2024 01:53 UTC
[Monitoring] The issue has been identified and the incident resolved. We will continue monitoring for a while.

August 23, 2024 01:27 UTC
[Identified] We have identified an underlying cause and are working to mitigate it. See gitlab.com/gitlab-com/gl-infra/production/-/issues/18435 for more details.

August 23, 2024 00:34 UTC
[Investigating] We have noticed that there are delays in web UI updates for example, MRs not showing up after submission, We are continuing to investigate - please see gitlab.com/gitlab-com/gl-infra/production/-/issues/18435 for more details.

August 23, 2024 00:27 UTC
[Investigating] Some requests are slow or timing out due to DB saturation. We are investigating. Please see [this issue](gitlab.com/gitlab-com/gl-infra/production/-/issues/18435) for further details.

Some Duo AI features not working correctly

August 9, 2024 00:56 UTC

Incident Status

Partial Service Disruption


Components

GitLab Duo


Locations

Google Compute Engine




August 9, 2024 00:56 UTC
[Resolved] GitLab is no longer seeing errors from Duo Chat which is inline with our upstream Anthropic provider status. For further details, see the incident page at gitlab.com/gitlab-com/gl-infra/production/-/issues/18377 and the upstream status page at status.anthropic.com

August 8, 2024 17:40 UTC
[Monitoring] GitLab has been continuing to monitor Duo Chat error rates, which have been decreasing over time. We will continue to monitor until the upstream Anthropic outage is resolved; we will only update here if this status changes. For more details, see the incident page at gitlab.com/gitlab-com/gl-infra/production/-/issues/18377 and the status page of the upstream provider at status.anthropic.com

August 8, 2024 16:19 UTC
[Monitoring] We are continuing to monitor reports of Duo Chat features returning errors due to an upstream outage. For more details, see the incident page at gitlab.com/gitlab-com/gl-infra/production/-/issues/18377 and the upstream status page at status.anthropic.com

August 8, 2024 16:01 UTC
[Investigating] We are seeing reports of Duo features returning errors due to an upstream outage. For more details, see the incident page at gitlab.com/gitlab-com/gl-infra/production/-/issues/18377 and the status page of the upstream provider at status.anthropic.com

July 2024

Some Duo AI features not working correctly

July 31, 2024 14:31 UTC

Incident Status

Partial Service Disruption


Components

GitLab Duo


Locations

Google Compute Engine




July 31, 2024 14:31 UTC
[Resolved] The issue has been resolved. Details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/18348

July 31, 2024 14:13 UTC
[Investigating] Some Duo AI features may not be working correctly. Engineers are investigating. Details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/18348

GitLab.com is unavailable

July 11, 2024 18:28 UTC

GitLab.com is unavailableService Disruption

Incident Status

Service Disruption


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions, SAML SSO - GitLab SaaS, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, docs.gitlab.com, Canary, Product Analytics - Configurator (Beta), Product Analytics - Collector (Beta), Product Analytics - Background processing (Beta), Product Analytics - Queries (Beta), GitLab Duo


Locations

Google Compute Engine, Digital Ocean, Zendesk, AWS




July 11, 2024 18:28 UTC
[Resolved] This incident is now resolved and no further errors are being seen on GitLab.com. Our investigation points to a disk performance issue. You can see further details and corrective actions in gitlab.com/gitlab-com/gl-infra/production/-/issues/18269.

July 11, 2024 15:51 UTC
[Investigating] Our investigations are ongoing. We will post our next update when we have more information. Please follow the incident issue for updates here: gitlab.com/gitlab-com/gl-infra/production/-/issues/18269

July 11, 2024 15:28 UTC
[Investigating] No material updates to report. We're continuing to investigate the issue. Please follow the incident issue for updates here: gitlab.com/gitlab-com/gl-infra/production/-/issues/18269

July 11, 2024 15:00 UTC
[Investigating] We are seeing improved service availability and investigations continue for this problem. Please follow the incident issue for updates here: gitlab.com/gitlab-com/gl-infra/production/-/issues/18269

July 11, 2024 14:43 UTC
[Investigating] We are seeing improved service availability and investigations continue for this problem. We are planning to make the incident issue publicly available as soon as possible.

July 11, 2024 14:29 UTC
[Investigating] We are investigating a problem with gitlab.com being unavailable for some users. Our investigations continue and we will post further status updates shortly.

July 11, 2024 14:18 UTC
[Investigating] We are investigating a problem with gitlab.com being unavailable for some users. We will post further status updates shortly.

June 2024

Duo Chat Returning Error Code A1001

June 27, 2024 20:39 UTC

Duo Chat Returning Error Code A1001Partial Service Disruption

Incident Status

Partial Service Disruption


Components

GitLab Duo


Locations

Google Compute Engine




June 27, 2024 20:39 UTC
[Resolved] We have confirmed that the fix has been deployed, so users should no longer experience errors with Duo Chat.

June 27, 2024 16:36 UTC
[Identified] Monitoring of the deployment of the fix continues. In the meantime, issuing a /clear to Duo Chat can be used to temporarily clear the error.

June 27, 2024 00:48 UTC
[Identified] It appears that the deployment of the fix has not completed as was previously understood. Users may still be experiencing the original issue. We're currently monitoring the deployment and a status update will be provided upon it's completion.

June 26, 2024 23:19 UTC
[Resolved] A fix for the issue has been deployed and users should no longer see error code A1001 when interacting with Duo Chat.

June 25, 2024 23:21 UTC
[Identified] Work is continuing on the development of a fix for the issue. Further updates will be provided on the status page once we have a timeline for the deployment of the fix.

June 25, 2024 17:44 UTC
[Identified] We have identified an issue with the /summarize feature of Duo Chat returning an error when used on Epics and Issues. A fix is currently being developed.





Back to current status