All Systems Operational

Updated a few seconds ago

Back to current status

Status History



April 2025

gitlab-runner package repository GPG public key expired

April 29, 2025 05:47 UTC

Incident Status

Partial Service Disruption


Components

CI/CD - Self-managed runners


Locations

Self-Managed Runner Connectivity




April 29, 2025 05:47 UTC
[Resolved] We have updated our public GPG key so any signature validations should now work correctly. More details are available here: gitlab.com/gitlab-com/gl-infra/production/-/issues/19725

April 28, 2025 23:25 UTC
[Identified] There are no further material updates to report at this time. We are analyzing the impact and expect to have further updates in the next several hours. More details are available here: gitlab.com/gitlab-com/gl-infra/production/-/issues/19725

April 27, 2025 04:32 UTC
[Identified] There are no material updates to report at this time. We are analyzing the impact and expect to have further updates in the next several hours. More details are available here: gitlab.com/gitlab-com/gl-infra/production/-/issues/19725

April 27, 2025 03:24 UTC
[Identified] We are aware that the current GPG public key for gitlab-runner packages has expired, we are working to refresh the key. Further updates will be provided accordingly.

April 27, 2025 02:59 UTC
[Investigating] We are aware that the current GPG public key for gitlab-runner packages has expired, we are working to refresh the key.

GitLab Duo Chat responding with error message

April 28, 2025 18:05 UTC

Incident Status

Operational


Components

GitLab Duo


Locations

Google Compute Engine




April 28, 2025 18:05 UTC
[Resolved] The cause of the Duo Chat error messages was discovered, and the fix has been deployed to production. Testing shows this is now resolved. If you still see errors with Duo Chat, please contact Support. See more: gitlab.com/gitlab-com/gl-infra/production/-/issues/19730

April 28, 2025 17:47 UTC
[Monitoring] The cause of the GitLab Duo Chat error messages has been discovered, and the fix has been deployed to production. Deployment should take ~20 minutes. See more: gitlab.com/gitlab-com/gl-infra/production/-/issues/19730

April 28, 2025 17:32 UTC
[Identified] The cause of the GitLab Duo Chat error messages has been discovered and we are working on a fix. See more: gitlab.com/gitlab-com/gl-infra/production/-/issues/19730

April 28, 2025 17:11 UTC
[Investigating] We're aware of an issue with GitLab Duo Chat responding to all queries with an A9999 error message. We're currently investigating. Please see gitlab.com/gitlab-com/gl-infra/production/-/issues/19730

April 28, 2025 17:08 UTC
[Investigating] We're aware of an issue with GitLab Duo Chat responding to all queries with an A9999 error message. We're currently investigating. Please see gitlab.com/gitlab-com/gl-infra/production/-/issues/19730

Intermittent issues interacting with gitlab-org/gitlab and any of its forks on GitLab.com

April 28, 2025 00:39 UTC

Incident Status

Operational


Components

Website, Git Operations


Locations

Google Compute Engine




April 28, 2025 00:39 UTC
[Resolved] Changes were implemented and after monitoring this incident is now resolved. For further details see gitlab.com/gitlab-com/gl-infra/production/-/issues/19691

April 22, 2025 03:21 UTC
[Investigating] Users may experience intermittent issues interacting with gitlab-org/gitlab and any of its forks on GitLab.com. 500 errors may occur and we are investigating

Decomposition of security database in GitLab.com

April 19, 2025 14:10 UTC

Description

This is a scheduled maintenance event to conduct the decomposition of tables relating to Gitlab's Security Risk Management features to a new DB cluster in GitLab.com's infrastructure to improve scalability and reliability. We expect very little disruption during this process. There may be increased latency and/or requests errors for a small duration, but we do not expect any downtime during transition.


Components

Website, API, Git Operations, Container Registry, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions, CI/CD - Self-managed runners, Background Processing


Locations

Google Compute Engine, AWS, Self-Managed Runner Connectivity


Schedule

April 19, 2025 06:00 - April 19, 2025 12:00 UTC



April 19, 2025 14:10 UTC
[Update] The scheduled maintenance is complete. For further details refer to - gitlab.com/gitlab-com/gl-infra/data-access/dbo/dbo-issue-tracker/-/issues/336

April 19, 2025 09:27 UTC
[Update] Gitlab.com SEC database decomposition was performed. We'll continue to monitor for any performance issues until the end of the maintenance window. Thank you for your patience. For further details refer to - gitlab.com/gitlab-com/gl-infra/data-access/dbo/dbo-issue-tracker/-/issues/336

April 19, 2025 06:00 UTC
[Update] Maintenance scheduled for the decomposition of the security database is starting. Further details can be found on this issue: gitlab.com/gitlab-com/gl-infra/data-access/dbo/dbo-issue-tracker/-/issues/336

Elevated Error Rate on GitLab.com

April 15, 2025 23:07 UTC

Incident Status

Degraded Performance


Components

API, CI/CD


Locations

Google Compute Engine




April 15, 2025 23:07 UTC
[Resolved] After a period of monitoring where a stable error rate was observed, we believe the issue is now resolved.

April 15, 2025 22:52 UTC
[Investigating] Error rates have remained stable, but we're continuing to monitor due to the previous recurrence of the issue. Details in gitlab.com/gitlab-com/gl-infra/production/-/issues/19675.

April 15, 2025 20:18 UTC
[Investigating] We have seen a recurrence of the issue and are working towards a resolution for users seeing 500 errors through the API and CI jobs. Details in gitlab.com/gitlab-com/gl-infra/production/-/issues/19675.

April 15, 2025 17:33 UTC
[Resolved] After a period of monitoring we have observed that the error rate has remained at expected levels. This issue is now resolved.

April 15, 2025 16:57 UTC
[Investigating] We have seen significant improvement in the error rate, it's now back to normal levels. We are continuing to monitor.

April 15, 2025 16:43 UTC
[Investigating] We are investigating an issue causing users to receive 500 or 502 responses from API calls and CI jobs. Details in gitlab.com/gitlab-com/gl-infra/production/-/issues/19675.

Errors when purchasing add-ons in the customer portal

April 11, 2025 12:16 UTC

Incident Status

Degraded Performance


Components

GitLab Customers Portal


Locations

Google Compute Engine




April 11, 2025 12:16 UTC
[Resolved] The incident has been resolved, and customers will be able to purchase add-ons in the customer portal again. Details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/19650

April 11, 2025 10:30 UTC
[Identified] No material updates to report. A fix is on the way, and we will post again once the MR to revert the issue has been deployed. Details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/19650

April 11, 2025 09:44 UTC
[Identified] Customers might experience errors when purchasing add-ons in the customer portal. We have identified the root cause, and are waiting for an MR to revert the issue. Details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/19650

Change in OAuth grant support

April 11, 2025 03:59 UTC

Description

GitLab.com is improving the security of OAuth Resource Owner Password Credentials grant type and will require client authentication for all ROPC requests, effective April 8, 2025. More information about this change can be read here: about.gitlab.com/blog/2025/04/01/improving-oauth-ropc-security-on-gitlab-com


Components

API


Locations

Google Compute Engine


Schedule

April 1, 2025 13:30 - April 11, 2025 13:30 UTC



April 11, 2025 03:59 UTC
[Update] Scheduled maintenance is complete. GitLab.com has improved the security of OAuth Resource Owner Password Credentials grant type to authentication for all ROPC requests. More information can be found here: about.gitlab.com/blog/2025/04/01/improving-oauth-ropc-security-on-gitlab-com

April 1, 2025 13:37 UTC
[Update] GitLab.com is improving the security of OAuth Resource Owner Password Credentials grant type and will require client authentication for all ROPC requests, effective April 8, 2025. More information about this change can be read here: about.gitlab.com/blog/2025/04/01/improving-oauth-ropc-security-on-gitlab-com

Delay in background job processing

April 10, 2025 09:44 UTC

Incident Status

Operational


Components

Background Processing


Locations

Google Compute Engine




April 10, 2025 09:44 UTC
[Resolved] The incident has been mitigated and we see no further delays with Audit Event Streaming. Details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/19637

April 9, 2025 17:50 UTC
[Monitoring] Job processing is continuing without delay. However, as a result of our mitigation efforts users may see delays in Audit Event Streaming.

April 9, 2025 16:19 UTC
[Monitoring] We encountered a brief delay in job processing that we believe to have mitigated. We will continue to monitor the situation. Details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/19637

Runner Performance Issues

April 8, 2025 21:03 UTC

Runner Performance IssuesDegraded Performance

Incident Status

Degraded Performance


Components

CI/CD


Locations

Google Compute Engine




April 8, 2025 21:03 UTC
[Resolved] After a period of monitoring we can confirm that the issue has been resolved.

April 8, 2025 19:31 UTC
[Monitoring] We're seeing significant improvement in the job queue backlog and believe we have mitigated the issue. We'll be monitoring for a time to ensure the issue doesn't recur.

April 8, 2025 19:12 UTC
[Identified] We've confirmed that a symptom of this issue is pre-defined CI variables being unavailable on the Pipeline Creation screen in the UI.

April 8, 2025 18:38 UTC
[Identified] We've made adjustments and are noticing the backlog of CI jobs is decreasing, but we're continuing to monitor. Details in gitlab.com/gitlab-com/gl-infra/production/-/issues/19629.

April 8, 2025 17:42 UTC
[Investigating] We are investigating reports of pipeline slowness and CI job completion issues. Details in gitlab.com/gitlab-com/gl-infra/production/-/issues/19629.

Elevated workload on Sidekiq due to large exports

April 1, 2025 10:07 UTC

Incident Status

Partial Service Disruption


Components

Git Operations, Background Processing


Locations

Google Compute Engine




April 1, 2025 10:07 UTC
[Resolved] We've monitored the queue, and it continued to perform at 100%. We're moving this incident to resolved.

April 1, 2025 09:16 UTC
[Monitoring] We've identified the root cause of the elevated workload on Sidekiq caused by large exports. We've implemented a fix and are monitoring the queue which is now running at 100%.

April 1, 2025 07:32 UTC
[Investigating] No substantial updates at this time. Our team is still investigating the issue.

April 1, 2025 06:48 UTC
[Investigating] We are seeing elevated workload on Sidekiq due to large exports. There might be increased wait on running repository exports. We are currently investigating the issue.

March 2025

Viewing jobs artifacts returns 404

March 29, 2025 00:22 UTC

Viewing jobs artifacts returns 404Partial Service Disruption

Incident Status

Partial Service Disruption


Components

Website


Locations

Google Compute Engine




March 29, 2025 00:22 UTC
[Resolved] Some users may have continued to see intermittent 404 when accessing job artifacts or fetching packages from the GitLab registry up until around 20:20 UTC when the fix was fully deployed to all our environments. See gitlab.com/gitlab-com/gl-infra/production/-/issues/19565.

March 28, 2025 11:07 UTC
[Resolved] After a monitoring period, we confirmed the 404 errors when accessing artifacts are not seen anymore, this incident is resolved. More details at gitlab.com/gitlab-com/gl-infra/production/-/issues/19565

March 28, 2025 10:11 UTC
[Monitoring] The 404 error when accessing artifacts is now fixed following the roll back of the latest deployment. We will monitor the behaviour in the next hour. More details at gitlab.com/gitlab-com/gl-infra/production/-/issues/19565

March 28, 2025 09:06 UTC
[Investigating] No material updates to report. We are still investigating the root cause. Please follow gitlab.com/gitlab-com/gl-infra/production/-/issues/19565 for more details

March 28, 2025 08:30 UTC
[Investigating] No material updates to report. We are still investigating the root cause. Please follow gitlab.com/gitlab-com/gl-infra/production/-/issues/19565 for more details

March 28, 2025 08:01 UTC
[Investigating] There is no materials to update. We are still investigating the root cause. Please follow gitlab.com/gitlab-com/gl-infra/production/-/issues/19565 for more details

March 28, 2025 07:43 UTC
[Investigating] Some users report downloading artifacts as a workaround isn't working. We're still investigate the root cause. Please follow gitlab.com/gitlab-com/gl-infra/production/-/issues/19565 for more details

March 28, 2025 07:24 UTC
[Investigating] You can download artifacts instead of viewing the in the WebUI as a workaround while we investigate this further. Please follow gitlab.com/gitlab-com/gl-infra/production/-/issues/19565 for more details

March 28, 2025 07:09 UTC
[Investigating] We are still investigating and narrowing down the possible cause. Please follow gitlab.com/gitlab-com/gl-infra/production/-/issues/19565 for more details

March 28, 2025 06:50 UTC
[Investigating] There's no materials for update as we are still investigating the root cause of the problem. Please follow gitlab.com/gitlab-com/gl-infra/production/-/issues/19565 for more details

March 28, 2025 06:34 UTC
[Investigating] There's no materials for update as we are still investigating the root cause of the problem. Please follow gitlab.com/gitlab-com/gl-infra/production/-/issues/19565 for more details

March 28, 2025 05:52 UTC
[Investigating] Users report getting 404 error when viewing jobs artifacts. We are investigating the problem. Please follow gitlab.com/gitlab-com/gl-infra/production/-/issues/19565 for more details

Increased errors for SaaS Windows Runners

March 27, 2025 06:47 UTC

Incident Status

Partial Service Disruption


Components

CI/CD - Hosted runners on Windows


Locations

Google Compute Engine




March 27, 2025 06:47 UTC
[Resolved] We do not see anymore failure during the past hour. We are marking this incident as resolved. Please see gitlab.com/gitlab-com/gl-infra/production/-/issues/19553 for more details

March 27, 2025 05:26 UTC
[Monitoring] We have rolled back the change and see that Windows Runners have successfully started. We'll continue to monitor the Windows Runner. Please follow gitlab.com/gitlab-com/gl-infra/production/-/issues/19553 for more details.

March 27, 2025 04:22 UTC
[Identified] We have identified the root cause and we are reverting the change that caused the incident. Please follow gitlab.com/gitlab-com/gl-infra/production/-/issues/19553 for more detail

March 27, 2025 03:43 UTC
[Investigating] We are seeing an increased rate of errors for Windows SaaS Runners. This is causing pipeline jobs to fail. We are investigating possible causes and mitigation actions. More details available in the incident issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/19553

Duo settings page is Blank for GitLab.com users

March 27, 2025 04:07 UTC

Incident Status

Operational


Components

Website, GitLab Duo


Locations

Google Compute Engine




March 27, 2025 04:07 UTC
[Resolved] A fix is deployed to production. The GitLab Duo seats assignment page should be available now. Please see gitlab.com/gitlab-com/gl-infra/production/-/issues/19550 for more details.

March 26, 2025 20:04 UTC
[Identified] We continue to monitor the revert MR's deploying status. Next update on this incident will be once the revert makes it into production. Continue following gitlab.com/gitlab-com/gl-infra/production/-/issues/19550 for more details for more details.

March 26, 2025 18:25 UTC
[Identified] As a workaround, users can access the settings page by typing in the URL directly, for example: gitlab.com/groups<namespace>/-/settings/gitlab_duo/seat_utilization Please follow gitlab.com/gitlab-com/gl-infra/production/-/issues/19550 for more details.

March 26, 2025 18:21 UTC
[Identified] Navigating to the "Settings -> GitLab Duo" page on a top-level group on GitLab.com renders a blank page. We have identified the cause and are in the process of deploying a revert for it.

Issue and Merge Request Descriptions not updating

March 25, 2025 17:16 UTC

Incident Status

Operational


Components

Website, Canary


Locations

Google Compute Engine




March 25, 2025 17:16 UTC
[Resolved] Our Merge Request with the fix has now been deployed to production and we have confirmed the issue is no longer seen in GitLab.com. This incident is now resolved.

March 25, 2025 07:16 UTC
[Identified] The fix has been merged and we're waiting for deployment to production. More details can be found in the incident issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/19535

March 25, 2025 04:35 UTC
[Identified] We have a fix in place. It has been approved and waiting to be merged. More details can be found in the incident issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/19535

March 25, 2025 03:31 UTC
[Identified] We have identified the root cause. Rolling back a recent change has mitigated the impact. We're currently investigating options to resolve. More details can be found in the incident issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/19535

March 25, 2025 02:58 UTC
[Identified] Some users may have experienced being unable to modify Merge Request and Issue descriptions. We have rolled back a recent change and are currently investigating further. More details can be found in the incident issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/19535

UI delayed status updates for completed pipelines

March 24, 2025 18:46 UTC

Incident Status

Partial Service Disruption


Components

Website, API


Locations

Google Compute Engine




March 24, 2025 18:46 UTC
[Resolved] Job processing is back to operational levels and we have not seen the issue resurface. We are considering this incident resolved, but investigation on the root cause will continue on the incident issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/19531.

March 24, 2025 17:50 UTC
[Monitoring] After gradually re-enabling the suspected worker to a 100% capacity, the issue has not re-appeared. We will monitor the health of our background processing to confirm its good status.

March 24, 2025 17:17 UTC
[Identified] We have stopped a worker contributing to the database saturation issue. Impact should no longer be perceived. We are working on cautiously re-enabling the worker to prevent the issue from re-appearing.

March 24, 2025 16:51 UTC
[Investigating] We continue investigating the database saturation issue in our infrastructure. GitLab.com users may notice a delay between pipeline completion and UI updates.

March 24, 2025 16:36 UTC
[Investigating] We are seeing signs of database saturation and believe this to be the cause of the issue.

March 24, 2025 16:15 UTC
[Investigating] Pipeline completion status may display with a delay in the user interface. Real-time pipeline execution continues normally, but status updates in the UI might lag behind actual completion events.

Issues pulling the latest gemnasium image

March 18, 2025 15:55 UTC

Incident Status

Degraded Performance


Components

Container Registry


Locations

Google Compute Engine




March 18, 2025 15:55 UTC
[Resolved] We've pushed the Gemnasium 5.8.8 image, everyone can now pull this. Everything is back to normal. For more information: gitlab.com/gitlab-com/gl-infra/production/-/issues/19498

March 18, 2025 14:23 UTC
[Identified] We've implemented a fix, and we're performing final verifications before releasing it. For more information: gitlab.com/gitlab-com/gl-infra/production/-/issues/19498

March 18, 2025 13:36 UTC
[Identified] Users retrying the jobs should succeed now as long as they specify any of these versions 5, 5.8, or latest, rather than 5.8.8. This is a quick temporary fix. We're currently working on a more permanent fix. For more information: gitlab.com/gitlab-com/gl-infra/production/-/issues/19498

March 18, 2025 12:58 UTC
[Identified] We are receiving some reports about users being unable to download the latest gemnasium images. We've identified the issue, and we're working on a fix. For more information: gitlab.com/gitlab-com/gl-infra/production/-/issues/19498

GitLab Duo outage

March 18, 2025 09:43 UTC

GitLab Duo outageService Disruption

Incident Status

Service Disruption


Components

GitLab Duo


Locations

Google Compute Engine




March 18, 2025 09:43 UTC
[Resolved] Service has been fully restored For more information - gitlab.com/gitlab-com/gl-infra/production/-/issues/19496

March 18, 2025 09:08 UTC
[Monitoring] An upstream fix has been implemented by Anthropic and service has been restored. We will continue to monitor the situation For more information - gitlab.com/gitlab-com/gl-infra/production/-/issues/19496

March 18, 2025 08:49 UTC
[Identified] GitLab Duo is not working on GitLab.com, Self-Managed, and Dedicated instances due to Anthropic being down. Per their Status Page, the issue has been identified and a fix is being implemented - status.anthropic.com For more information - gitlab.com/gitlab-com/gl-infra/production/-/issues/19496

Jobs queued for a long time

March 13, 2025 19:26 UTC

Jobs queued for a long timeDegraded Performance

Incident Status

Degraded Performance


Components

CI/CD - Hosted runners on Linux


Locations

Google Compute Engine




March 13, 2025 19:26 UTC
[Resolved] Operations have returned to normal and jobs are getting picked up as expected. This incident is now resolved.

March 13, 2025 19:14 UTC
[Identified] Our team has identified the issue and jobs are processing normally. Our team is continuing to monitor.

March 13, 2025 18:40 UTC
[Investigating] No substantial updates at this time. Our team is continuing to investigate.

March 13, 2025 18:08 UTC
[Investigating] We've identified that the build queue size has grown, impacting some jobs from getting picked up. Our team is continuing to investigate the issue to identify the root cause.

March 13, 2025 17:37 UTC
[Investigating] We're receiving reports of jobs being queued for an excessive amount of time. Our team is currently investigating.

Pipelines are delayed in reporting completion

March 10, 2025 18:30 UTC

Incident Status

Degraded Performance


Components

Background Processing


Locations

Google Compute Engine




March 10, 2025 18:30 UTC
[Resolved] Our team has been monitoring and all operations have returned to normal. This incident is now resolved.

March 10, 2025 17:59 UTC
[Identified] We are no longer seeing delays on new jobs. For existing jobs in a stuck state, cancelling and retrying the job should resolve the problem. If you can’t cancel and restart a pipeline, cancelling an early job in the pipeline and restarting it may also recover the pipeline.

March 10, 2025 17:44 UTC
[Investigating] Our team has identified some issues that contributed to PG Bouncer saturation and increased sidekiq load. While this has decreased significantly, we're continuing to investigate the root cause.

March 10, 2025 17:21 UTC
[Investigating] No substantial updates at this time. Our team is still investigating the issue.

March 10, 2025 16:52 UTC
[Investigating] We are receiving reports that pipelines are taking a long time to complete. Our team is investigating.

Posts are being marked as edited by "Ghost User"

March 10, 2025 05:10 UTC

Incident Status

Degraded Performance


Components

Website


Locations

Google Compute Engine




March 10, 2025 05:10 UTC
[Resolved] The fix has now been deployed to production. Anyone still seeing 'Ghost User' edits on issues, please refresh browser cache. If the problem continues for you please reach out to support. for more information: gitlab.com/gitlab-com/gl-infra/production/-/issues/19440

March 7, 2025 21:21 UTC
[Identified] Users may see "Ghost User" edits on issues if the new issue look is enabled, as well as on epics and tasks. This is purely a display issue. The fix is in the works and will be fully implemented by 2025-03-10. We will update the status at that time. Please see here for more information: gitlab.com/gitlab-com/gl-infra/production/-/issues/19440





Back to current status