Active Issue

Updated a few seconds ago

Back to current status

Status History

Filter: Website (Clear)



February 2025

PG minor upgrade 16.6 in Production

February 15, 2025 07:49 UTC

Description

Scheduled maintenance to patch our DB layer to PostgreSQL 16.6. 2025-02-15 02:00-08:00 UTC. GitLab.com will be available throughout this time, but users may experience errors on requests. We apologize in advance for any inconvenience caused. See <gitlab.com/gitlab-com/gl-infra/production/-/issues/19120>


Components

Website, API, Container Registry, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions, Background Processing


Locations

Google Compute Engine, AWS


Schedule

February 15, 2025 03:00 - February 15, 2025 09:00 UTC



February 15, 2025 07:49 UTC
[Update] Gitlab.com database maintenance was performed, the platform is considered stable. We'll continue to monitor for any performance issues. Thank you for your patience. See <gitlab.com/gitlab-com/gl-infra/production/-/issues/19120>

February 15, 2025 04:07 UTC
[Update] Gitlab.com scheduled maintenance of our database layer has started. GitLab.com should be available during the whole period but we expect availability and performance impact until 2025-02-15 09:00 UTC. See gitlab.com/gitlab-com/gl-infra/production/-/issues/19120

February 15, 2025 03:01 UTC
[Update] Gitlab.com scheduled maintenance of our database layer has started. GitLab.com should be available during the whole period but we expect availability and performance impact until 2025-02-15 09:00 UTC. See gitlab.com/gitlab-com/gl-infra/production/-/issues/19120

February 15, 2025 02:48 UTC
[Update] Scheduled database maintenance will now start at 2025-02-15 03:00 UTC and expected to run to 2025-02-15 09:00 UTC. See gitlab.com/gitlab-com/gl-infra/production/-/issues/19120

February 15, 2025 01:54 UTC
[Update] Due to an ongoing incident we are delaying the maintenance window by 2 hours to investigate the impact. We currently plan to start the maintenance window at 04:00 UTC.

February 14, 2025 01:45 UTC
[Update] We will be undergoing scheduled maintenance of our databases tomorrow. There may be some availability and performance impact between 2025-02-15 02:00 UTC and 2025-02-15 08:00 UTC. See gitlab.com/gitlab-com/gl-infra/production/-/issues/19120

February 12, 2025 01:13 UTC
[Update] We will be undergoing scheduled maintenance of our databases in 3 days. There may be some availability and performance impact between 2025-02-15 02:00 UTC and 2025-02-15 08:00 UTC. See gitlab.com/gitlab-com/gl-infra/production/-/issues/19120

February 8, 2025 08:37 UTC
[Update] Scheduled maintenance in 1 week: Upgrading DB layer to PostgreSQL 16.6 on Feb 15, 02:00-08:00 UTC. GitLab.com will stay up but users may experience errors. See gitlab.com/gitlab-com/gl-infra/production/-/issues/19120

February 1, 2025 02:42 UTC
[Update] Scheduled maintenance in 2 weeks: Upgrading DB layer to PostgreSQL 16.6 on Feb 15, 02:00-08:00 UTC. GitLab.com will stay up but may experience errors. Sorry for any inconvenience. See gitlab.com/gitlab-com/gl-infra/production/-/issues/19120

Errors managing Kubernetes agents

February 11, 2025 15:23 UTC

Errors managing Kubernetes agentsPartial Service Disruption

Incident Status

Partial Service Disruption


Components

Website, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions, CI/CD - Self-managed runners


Locations

Google Compute Engine, AWS, Self-Managed Runner Connectivity




February 11, 2025 15:23 UTC
[Resolved] The fix has been successfully deployed. Our monitoring confirms that the service has been restored to normal operation with no further errors observed. Thank you for your patience during this disruption. If you experience any further issues, please contact our Support team.

February 11, 2025 14:39 UTC
[Monitoring] We are currently rolling out a fix to production. Our team is actively monitoring the deployment and its effectiveness. Please continue to follow our status page for the latest updates.

February 11, 2025 10:24 UTC
[Identified] The ongoing incident investigation has been marked as confidential. While the original incident report will no longer be publicly visible, we continue to actively work on the resolution. Service impact remains unchanged. Further updates will be available on the status page - status.gitlab.com

February 11, 2025 09:35 UTC
[Identified] We identified the root cause of the incident, and we are reverting the relevant MR. The estimated time for the fix: around 5 hours from now. More details: gitlab.com/gitlab-com/gl-infra/production/-/issues/19278

February 11, 2025 09:00 UTC
[Investigating] Some users are experiencing errors managing Kubernetes agents. Details can be found in: gitlab.com/gitlab-com/gl-infra/production/-/issues/19278

Degraded performance of the WebIDE

February 6, 2025 10:34 UTC

Incident Status

Degraded Performance


Components

Website


Locations

Google Compute Engine




February 6, 2025 10:34 UTC
[Resolved] We have fully recovered so we are marking this as resolved now. For details see gitlab.com/gitlab-com/gl-infra/production/-/issues/19244

February 6, 2025 09:29 UTC
[Monitoring] We are seeing recovery, and are going to continue monitoring. For details see gitlab.com/gitlab-com/gl-infra/production/-/issues/19244

February 6, 2025 09:14 UTC
[Investigating] We have noticed degraded performance of the WebIDE and are investigating. See gitlab.com/gitlab-com/gl-infra/production/-/issues/19244

January 2025

Merge request diffs for comments unavailable for parts of December / January

January 20, 2025 12:09 UTC

Incident Status

Partial Service Disruption


Components

Website


Locations

Google Compute Engine




January 20, 2025 12:09 UTC
[Resolved] After a monitoring period, we confirmed that the issue is now resolved and the diffs are showing again as expected. Please reach out to our Support team if you encounter issues.

January 20, 2025 03:02 UTC
[Monitoring] No material updates to report. Our team continue to monitor the fix. Another update will be provided in 24 hours.

January 19, 2025 01:09 UTC
[Monitoring] No material updates to report. We're continuing to assess the scope of impact and monitoring the fix. Another update will be provided in 24 hours.

January 18, 2025 02:34 UTC
[Monitoring] No material updates to report. We are still monitoring the fix and we will provide an update in 24 hours.

January 17, 2025 09:26 UTC
[Monitoring] The deploy to production has finished and image diff comments are rendered correctly again. Our engineers are monitoring the situation. The next update can be expected in 12 hours.

January 16, 2025 17:15 UTC
[Identified] The fix for Merge Request diff comments has been deployed to production. We are still working on a fix for image diff comments. MR: gitlab.com/gitlab-org/gitlab/-/merge_requests/178143. We will update once this has been deployed to production.

January 16, 2025 08:29 UTC
[Identified] A fix has been merged and is getting deployed to production. We will update the status page again, once the fix has been deployed successfully.

January 16, 2025 00:08 UTC
[Identified] We have been analyzing impacted merge request diff comments; we have noticed that some diff notes do not have a Reply form. We are working on a fix in this Merge Request: gitlab.com/gitlab-org/gitlab/-/merge_requests/178057

January 15, 2025 16:54 UTC
[Identified] We have verified the fix in production and it has been rolled out to all GitLab.com projects. The majority of comment display issues have been fixed and we are working on a followup to resolve any remaining issues.

January 15, 2025 08:07 UTC
[Identified] A fix for impacted diff comments has been merged, and is getting deployed to production. In our next status update, we will notify when the fix is in production.

January 15, 2025 00:27 UTC
[Identified] The issue where diff comments are sometimes not showing in Merge Requests appears to be a display issue. A initial fix was deployed on Jan 8 to prevent further occurrences. A fix for impacted diff comments is in progress.

January 14, 2025 20:29 UTC
[Identified] We are aware of an issue where diffs are not showing in some cases on merge requests created between December 9 and Jan 8. We are continuing to research and will update when we have further updates.

January 14, 2025 16:24 UTC
[Identified] We're aware of an issue where merge request comments created between Dec 9th and Jan 8th may contain comments that are no longer associated with a diff.

Runner 17.8.0 incorrectly tagged

January 16, 2025 20:46 UTC

Runner 17.8.0 incorrectly taggedPartial Service Disruption

Incident Status

Partial Service Disruption


Components

Website, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions, CI/CD - Self-managed runners


Locations

Google Compute Engine, Self-Managed Runner Connectivity




January 16, 2025 20:46 UTC
[Resolved] GitLab has published a corrected image for GitLab Runner 17.8; the image can be used normally. Further investigations will be documented in gitlab.com/gitlab-com/gl-infra/production/-/issues/19129.

January 16, 2025 20:04 UTC
[Investigating] GitLab is working on uploading a corrected version of the 17.8.0 Runner image. We will provide an update once we have confirmed the update has been deployed.

January 16, 2025 19:38 UTC
[Investigating] GitLab is investigating an issue where the image for Runner 17.8.0 was not tagged correctly when uploaded to the Container Registry. We recommend continuing to use the Runner 17.7 image until the Runner 17.8 image can be tagged correctly.

Subscription plan disruptions on GitLab.com

January 15, 2025 22:51 UTC

Incident Status

Partial Service Disruption


Components

Website


Locations

Google Compute Engine




January 15, 2025 22:51 UTC
[Resolved] Our monitoring is now complete and we haven't seen additional reports of subscription plans on GitLab.com being impacted. We'll now be resolving this incident. Please reach out to our Support team if you encounter issues.

January 15, 2025 08:18 UTC
[Monitoring] No material updates to report. We are still monitoring the fix and we will provide an update in 12 hours.

January 14, 2025 21:56 UTC
[Monitoring] A fix has been implemented and impacted subscription plans should now be restored. Our team will be monitoring this fix over the next few hours.

January 14, 2025 20:20 UTC
[Identified] At the moment, there are no material updates at this time. A fix has been deployed and investigations are ongoing. We will update when we have a substantive update.

January 14, 2025 14:38 UTC
[Identified] No material updates at this point. A fix has been deployed, but investigations are still ongoing. The next update can be expected in 2 hours.

January 14, 2025 12:31 UTC
[Identified] We have deployed a fix, but investigations are still ongoing. The next update can be expected in 2 hours.

January 14, 2025 10:24 UTC
[Identified] No updates at this time. The next status update can be expected in 2 hours.

January 14, 2025 08:25 UTC
[Identified] No material updates at this point. The next update can be expected in 2 hours.

January 14, 2025 06:11 UTC
[Identified] No material updates at this time. We are working to reinstate the subscriptions as quickly as possible.

January 14, 2025 04:57 UTC
[Identified] A small number of GitLab.com customers are experiencing subscription plan disruptions as a result of a routine audit of our subscription records. Our team is currently working on reinstating these.

Accessing gitlab.com results in 500 errors

January 6, 2025 16:42 UTC

Incident Status

Partial Service Disruption


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions, SAML SSO - GitLab SaaS, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, docs.gitlab.com, Canary, Product Analytics - Configurator (Beta), Product Analytics - Collector (Beta), Product Analytics - Background processing (Beta), Product Analytics - Queries (Beta), GitLab Duo


Locations

Google Compute Engine, Digital Ocean, Zendesk, AWS




January 6, 2025 16:42 UTC
[Resolved] We've identified the cause of the errors and the errors have returned to normal. The incident is now resolved. More information on the incident can be found in this issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/19079

January 6, 2025 15:57 UTC
[Monitoring] We've identified the cause of the errors, and the errors have returned to normal. The incident is now mitigated, but we're continuing to monitor for any further impact. More information on the incident can be found in this issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/19079

January 6, 2025 15:27 UTC
[Monitoring] We're continuing to monitor for further errors related to the incident. We see further decreases in the errors.

January 6, 2025 15:17 UTC
[Monitoring] We saw elevated 500 errors on gitlab.com which have then decreased. We're continuing to monitor for further errors

Experiencing some slowness accessing Gitlab.com

January 3, 2025 22:55 UTC

Incident Status

Operational


Components

Website


Locations

Google Compute Engine




January 3, 2025 22:55 UTC
[Resolved] Mitigation has been implemented and GitLab.com has remained stable throughout our monitoring period and we are now marking this incident as resolved.

January 3, 2025 22:09 UTC
[Monitoring] We experienced some slowness around 9:38PM UTC time at around 10 min in duration. That has since been resolved, but we are still investigating the root cause.

January 3, 2025 21:50 UTC
[Investigating] We experienced some slowness around 9:38PM UTC time around 10 min in duration. That has since been resolved but we are still investigating the root cause.

December 2024

Intermittent errors across GitLab.com

December 20, 2024 19:47 UTC

Intermittent errors across GitLab.comPartial Service Disruption

Incident Status

Partial Service Disruption


Components

Website, GitLab Pages, Background Processing


Locations

Google Compute Engine




December 20, 2024 19:47 UTC
[Resolved] GitLab.com has remained stable throughout our monitoring period and we are now marking this incident as resolved. Please see the following issue for further information related to this incident: gitlab.com/gitlab-com/gl-infra/production/-/issues/19033

December 20, 2024 00:08 UTC
[Monitoring] We are continuing to monitor the issue at this point, with further recovery being observed. Follow gitlab.com/gitlab-com/gl-infra/production/-/issues/19033 for more details.

December 19, 2024 23:32 UTC
[Monitoring] Mitigation steps have been completed. You should see delayed tasks start to recover now. Follow gitlab.com/gitlab-com/gl-infra/production/-/issues/19033 for more details.

December 19, 2024 23:09 UTC
[Identified] Mitigation steps are progressing and GitLab.com responsiveness is looking good. Follow gitlab.com/gitlab-com/gl-infra/production/-/issues/19033 for more details.

December 19, 2024 21:58 UTC
[Identified] Users may still see delays when interacting with GitLab.com. We are still actively working to mitigate the issue. Details will be posted in: gitlab.com/gitlab-com/gl-infra/production/-/issues/19033

December 19, 2024 21:11 UTC
[Identified] Our work to fully mitigate the underlying issue is still ongoing. Details will be posted in: gitlab.com/gitlab-com/gl-infra/production/-/issues/19033

December 19, 2024 20:31 UTC
[Identified] No material updates at this time. We have seen in overall improvement but are still are still working to fully mitigate the issue. Details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/19033

December 19, 2024 19:51 UTC
[Identified] Our initial changes have helped alleviate the issue. Our efforts are still ongoing as this is not fully mitigated. Details will be posted in: gitlab.com/gitlab-com/gl-infra/production/-/issues/19033

December 19, 2024 19:14 UTC
[Identified] We have implemented mitigating changes. Services should begin to recover. We are continuing to adjust and monitor. Details will be posted in: gitlab.com/gitlab-com/gl-infra/production/-/issues/19033

December 19, 2024 18:37 UTC
[Identified] No material updates at this time. We are still working to mitigate the issue. Details will be posted in: gitlab.com/gitlab-com/gl-infra/production/-/issues/19033

December 19, 2024 18:09 UTC
[Identified] We are still working to mitigate the issue. Users may still encounter errors or delays on GitLab.com. Details will be posted in: gitlab.com/gitlab-com/gl-infra/production/-/issues/19033

December 19, 2024 17:46 UTC
[Identified] We believe to have identified the cause of this incident. We are working on a mitigation strategy. Users may see errors or delays when interacting with GitLab.com. Details will be posted in: gitlab.com/gitlab-com/gl-infra/production/-/issues/19033

December 19, 2024 17:31 UTC
[Investigating] We are investigating reports of intermittent errors across GitLab.com and GitLab Pages. Details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/19033

Redis Cluster Saturation

December 18, 2024 16:30 UTC

Redis Cluster SaturationPartial Service Disruption

Incident Status

Partial Service Disruption


Components

Website


Locations

Google Compute Engine




December 18, 2024 16:30 UTC
[Resolved] We have confirmed our mitigation efforts were successful. We are now marking this incident as resolved. Details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/19025

December 18, 2024 15:48 UTC
[Monitoring] We have identified the cause of the issue and have taken the necessary measures to mitigate it. We are now monitoring Redis before marking the incident as resolved. More details here: gitlab.com/gitlab-com/gl-infra/production/-/issues/19025

December 18, 2024 15:05 UTC
[Investigating] We're experiencing saturation in our Redis cluster. The investigation is still ongoing. Users might get 500 errors intermittently. More details here: gitlab.com/gitlab-com/gl-infra/production/-/issues/19025

Importing with GitHub, Bitbucket Server, and Gitea importer offline

December 18, 2024 14:27 UTC

Incident Status

Security Issue


Components

Website, API


Locations

Google Compute Engine




December 18, 2024 14:27 UTC
[Resolved] Import with GitHub, Bitbucket Server, and Gitea has now been re-enabled and importers are back online.

November 22, 2024 11:59 UTC
[Investigating] Import with GitHub, Bitbucket Server, and Gitea remains offline and we continue efforts to restore functionality. Migration by Direct Transfer is now re-enabled with improved functionality on GitLab.com. See details at docs.gitlab.com/ee/user/project/import/index.html#user-contribution-and-membership-mapping

October 9, 2024 08:22 UTC
[Investigating] Migration by direct transfer and importing with GitHub, Bitbucket Server, and Gitea importer is currently offline. We are working to restore the import functionality. We do not have an estimated resolution date at this time.

August 13, 2024 08:11 UTC
[Investigating] Migration by direct transfer and importing with GitHub, Bitbucket Server, and Gitea importer is currently offline and we are investigating the cause. We will update the status page when more information is available.

July 11, 2024 06:29 UTC
[Investigating] Migration by direct transfer and importing with GitHub, Bitbucket Server, and Gitea importer is currently offline and we are investigating the cause. We will update the status page when more information becomes available.

July 11, 2024 06:22 UTC
[Investigating] Migration by direct transfer and importing with GitHub, Bitbucket Server, and Gitea importer is currently offline and we are currently investigating the cause. We will update the status page when more information becomes available.

Fargate runner error - file name too long

December 4, 2024 16:50 UTC

Incident Status

Partial Service Disruption


Components

Website, CI/CD - Hosted runners for GitLab community contributions


Locations

Google Compute Engine




December 4, 2024 16:50 UTC
[Resolved] As no new user reports have been received during our monitoring period we consider this incident resolved. See gitlab.com/gitlab-com/gl-infra/production/-/issues/18939 for the full incident history.

December 4, 2024 15:46 UTC
[Monitoring] We have disabled the feature flag and are now monitoring the issue for 1 hour before marking as resolved. More details can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/18939.

December 4, 2024 15:42 UTC
[Identified] We've identified cause of the issue and are working on resolving it. More details can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/18939.

December 4, 2024 15:16 UTC
[Investigating] We are currently investigating issues with GitLab runners with Fargate driver returning "file name too long" errors. More details about this incident can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/18939

Project mirror disabled due to excessive notifications

December 3, 2024 20:26 UTC

Incident Status

Degraded Performance


Components

Website, Background Processing


Locations

Google Compute Engine




December 3, 2024 20:26 UTC
[Resolved] After seeing no further issues arise during our monitoring period, we are considering this incident resolved. Please review gitlab.com/gitlab-com/gl-infra/production/-/issues/18929 for more details.

December 3, 2024 18:49 UTC
[Monitoring] Project mirroring has been re-enabled on GitLab.com and we are monitoring to make sure no further issues arise. See gitlab.com/gitlab-com/gl-infra/production/-/issues/18929 for further details.

December 3, 2024 15:51 UTC
[Identified] We've turned off the functionality relating to schedules for project mirroring. Project mirroring will be reenabled once we resolve this issue. Mirrored projects will not be updated during this time.

December 3, 2024 14:47 UTC
[Identified] We've turned off the functionality that sends out email updates temporarily for project mirrors. We are continuing to investigate this incident.

December 3, 2024 14:04 UTC
[Investigating] We are currently investigating emails being sent for older project mirrors and imports. More details about this incident can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/18929

November 2024

Duplicate Merge Request Events

November 26, 2024 19:52 UTC

Duplicate Merge Request EventsDegraded Performance

Incident Status

Degraded Performance


Components

Website


Locations

Google Compute Engine




November 26, 2024 19:52 UTC
[Resolved] We have confirmed that duplicate merge request events have stopped being created. Please see the production issue for details: gitlab.com/gitlab-com/gl-infra/production/-/issues/18904

November 26, 2024 19:04 UTC
[Monitoring] A fix has been rolled out and merge request events should no longer be duplicated. See the production issue for more details: gitlab.com/gitlab-com/gl-infra/production/-/issues/18904

November 26, 2024 18:43 UTC
[Identified] Users may see duplicated events on merge requests, or failed merge attempts and merge request updates. Please see gitlab.com/gitlab-com/gl-infra/production/-/issues/18904 for more information.

Error when installing GitLab for Jira Cloud app

November 19, 2024 22:18 UTC

Incident Status

Partial Service Disruption


Components

Website


Locations

Google Compute Engine




November 19, 2024 22:18 UTC
[Resolved] We have verified that GitLab for Jira Cloud application issues have been fixed. We are resolving this incident; any further updates will be posted to gitlab.com/gitlab-com/gl-infra/production/-/issues/18872

November 19, 2024 22:02 UTC
[Monitoring] The fix has been deployed to production and has been tested; we believe that issues with the GitLab for Jira Cloud application are resolved; we are monitoring to confirm. See more details in gitlab.com/gitlab-com/gl-infra/production/-/issues/18872

November 19, 2024 14:19 UTC
[Identified] The fix has been merged and is currently being deployed to production. We will confirm when deployment has completed or provide another update sooner if relevant. See more details in gitlab.com/gitlab-com/gl-infra/production/-/issues/18872

November 19, 2024 12:51 UTC
[Identified] The merge request to fix the issue is still in progress, but expected to complete shortly. We will confirm when it has been merged or provide another update sooner if relevant. See more details in gitlab.com/gitlab-com/gl-infra/production/-/issues/18872

November 19, 2024 12:08 UTC
[Identified] We've identified the root cause and are currently working on a merge request to fix the issue. See more details in gitlab.com/gitlab-com/gl-infra/production/-/issues/18872

November 19, 2024 11:27 UTC
[Investigating] We're aware of an issue with installing the GitLab for Jira Cloud app from Atlassian Marketplace and are investigating. See more details in gitlab.com/gitlab-com/gl-infra/production/-/issues/18872

500 errors on user preferences pages

November 18, 2024 15:45 UTC

Incident Status

Degraded Performance


Components

Website


Locations

Google Compute Engine




November 18, 2024 15:45 UTC
[Resolved] This incident has been resolved as we are no longer seeing any issues on the User Preferences page after a period of monitoring. For more information please see: gitlab.com/gitlab-com/gl-infra/production/-/issues/18867

November 18, 2024 15:18 UTC
[Monitoring] The fix has been deployed to production and the User Preferences page should be accessible again. We're now monitoring to ensure the issue has been resolved. For more information please see: gitlab.com/gitlab-com/gl-infra/production/-/issues/18867

November 18, 2024 14:07 UTC
[Identified] Deployment to production is currently ongoing, but is expected to complete shortly. We will provide another update once deployment is completed. For more information please see: gitlab.com/gitlab-com/gl-infra/production/-/issues/18867

November 18, 2024 13:16 UTC
[Identified] We have deployed on canary and the revert fixed the issue with accessing the User Preferences page. We will confirm once deployment to production is complete within the next ~40 minutes. For more information please see: gitlab.com/gitlab-com/gl-infra/production/-/issues/18867

November 18, 2024 11:58 UTC
[Identified] The deployment process to resolve the issue is still ongoing. We will confirm once the deployment is complete or provide another progress update within ~1 hour. For more information please see: gitlab.com/gitlab-com/gl-infra/production/-/issues/18867

November 18, 2024 09:24 UTC
[Identified] A fix is currently being deployed and is expected to complete in the next ~3 hours. Another update will be provided at 12:00 UTC or once the deployment has completed. For more information please see: gitlab.com/gitlab-com/gl-infra/production/-/issues/18867

November 18, 2024 07:57 UTC
[Identified] The fix is awaiting deployment, which is expected to occur before 12:00 UTC. Once the fix has deployed the User Preferences page will be monitored to ensure the issue has been resolved. For more information please see: gitlab.com/gitlab-com/gl-infra/production/-/issues/18867

November 18, 2024 07:34 UTC
[Identified] A fix has been merged and it will be deployed shortly - once it has been deployed we will monitor the number of 500s occurring on the settings page to ensure the issue is resolved. For more information please see: gitlab.com/gitlab-com/gl-infra/production/-/issues/18867

November 18, 2024 07:18 UTC
[Identified] A fix has been implemented and is being tested before being merged. For more information please see: gitlab.com/gitlab-com/gl-infra/production/-/issues/18867

November 18, 2024 07:02 UTC
[Identified] We are seeing a number of 500 errors occurring when users attempt to navigate to the user profile settings in the UI - We have identified the issue and are working on a fix. The issue is being tracked in the incident issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/18867

MAIN DB cluster upgrade

November 5, 2024 11:02 UTC

MAIN DB cluster upgrade Planned Maintenance

Description

Next week, we will be undergoing scheduled maintenance to our MAIN database layer. The maintenance will start at 2024-11-03 06:00 UTC and should finish at 2024-11-05 11:00 UTC (including performance regression observability period). GitLab.com will be available during the whole period as the maintenance should be seamless and transparent for the application. We apologize in advance for any inconvenience this may cause. See <gitlab.com/gitlab-com/gl-infra/production/-/issues/18747>


Components

Website, API, Git Operations, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions


Locations

Google Compute Engine, AWS


Schedule

November 3, 2024 06:00 - November 5, 2024 11:00 UTC



November 5, 2024 11:02 UTC
[Update] Gitlab.com MAIN database upgrade is now complete with all systems are functioning correctly. Thank you for your patience. See gitlab.com/gitlab-com/gl-infra/production/-/issues/18747

November 3, 2024 09:01 UTC
[Update] Gitlab.com MAIN database upgrade was performed. We'll continue to monitor for any performance issues until the end of the maintenance window. Thank you for your patience. See <gitlab.com/gitlab-com/gl-infra/production/-/issues/18747>

November 3, 2024 06:05 UTC
[Update] Gitlab.com scheduled maintenance of our MAIN database layer have started. GitLab.com should be available during the whole period. See <gitlab.com/gitlab-com/gl-infra/production/-/issues/18747>

November 3, 2024 05:00 UTC
[Update] MAIN DB layer maintenance will start in 1h from now at 2024-11-03 06:00 UTC. GitLab.com will remains available. Details: gitlab.com/gitlab-com/gl-infra/production/-/issues/18747

November 2, 2024 05:00 UTC
[Update] Scheduled maintenance on MAIN DB layer: Start: 2024-11-03 06:00 UTC End: 2024-11-05 11:00 UTC (incl. monitoring) GitLab.com remains available. Maintenance should be seamless. We apologize for any inconvenience. Details: gitlab.com/gitlab-com/gl-infra/production/-/issues/18747

October 30, 2024 06:00 UTC
[Update] In three days, we will be undergoing scheduled maintenance to our MAIN database layer. The maintenance will start at 2024-11-03 06:00 UTC and should finish at 2024-11-05 11:00 UTC (including performance and regression observability period). GitLab.com will be available during the whole period as the maintenance should be seamless and transparent for the application. We apologize in advance for any inconvenience this may cause. See: gitlab.com/gitlab-com/gl-infra/production/-/issues/18747

October 2024

CI DB cluster upgrade

October 30, 2024 01:37 UTC

CI DB cluster upgradePlanned Maintenance

Description

Next week, we will be undergoing scheduled maintenance to our CI database layer. The maintenance will start at 2024-10-26 06:00 UTC and should finish at 2024-10-29 11:00 UTC (including performance regression observability period). GitLab.com will be available during the whole period as the maintenance should be seamless and transparent for the application. We apologize in advance for any inconvenience this may cause. See <gitlab.com/gitlab-com/gl-infra/production/-/issues/18639>


Components

Website, API, Git Operations, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions


Locations

Google Compute Engine, AWS


Schedule

October 26, 2024 06:00 - October 29, 2024 11:00 UTC



October 30, 2024 01:37 UTC
[Update] GitLab.com upgrade of the CI database layer is now complete with all systems are functioning correctly. Thank you for your patience. See gitlab.com/gitlab-com/gl-infra/production/-/issues/18639

October 27, 2024 08:47 UTC
[Update] GitLab.com upgrade of the CI database layer is complete. We'll be monitoring the platform to ensure all systems are functioning correctly. Thank you for your patience.

October 26, 2024 06:01 UTC
[Update] We are starting maintenance to the CI database layer, and should finish at 2024-10-29 11:00 UTC (including performance regression and observability period). GitLab.com will be available during the whole period as the maintenance should be seamless. See gitlab.com/gitlab-com/gl-infra/production/-/issues/18639

October 25, 2024 19:03 UTC
[Update] Reminder: Tomorrow, we will be undergoing scheduled maintenance to our CI database layer. The maintenance will start at 2024-10-26 06:00 UTC and should finish at 2024-10-29 11:00 UTC (including performance regression observability period). GitLab.com will be available during the whole period as the maintenance should be seamless and transparent for the application. We apologize in advance for any inconvenience this may cause. See <gitlab.com/gitlab-com/gl-infra/production/-/issues/18639>

October 23, 2024 17:25 UTC
[Update] In 3 days, we will be undergoing scheduled maintenance to our CI database layer. The maintenance will start at 2024-10-26 06:00 UTC and should finish at 2024-10-29 11:00 UTC (including performance regression observability period). GitLab.com will be available during the whole period as the maintenance should be seamless and transparent for the application. We apologize in advance for any inconvenience this may cause. See <gitlab.com/gitlab-com/gl-infra/production/-/issues/18639>

Self-managed runners are failing after runners are upgraded to 17.5

October 18, 2024 21:18 UTC

Incident Status

Partial Service Disruption


Components

Website, CI/CD - Hosted runners for GitLab community contributions


Locations

Google Compute Engine




October 18, 2024 21:18 UTC
[Resolved] Runner version 17.5.1 has been released and includes a fix for the "failed to get user home dir: $HOME is not defined" error. No action besides upgrading Runner is required. For more details see: gitlab.com/gitlab-com/gl-infra/production/-/issues/18732

October 18, 2024 10:25 UTC
[Identified] We have identified the problem and are working on releasing a patch for version 17.5. We will send a new update once the patch is available. For more information please see gitlab.com/gitlab-com/gl-infra/production/-/issues/18732.

October 18, 2024 09:49 UTC
[Investigating] Self-managed runners are failing with the error "failed to get user home dir: $HOME is not defined" after upgrading the GitLab runner to version 17.5. The current workaround is to downgrade the affected runners back to version 17.4. More details are available in gitlab.com/gitlab-org/gitlab-runner/-/issues/38252.

Sidekiq queue lengths increasing causing delays in MR and pipeline processing

October 16, 2024 15:02 UTC

Incident Status

Partial Service Disruption


Components

Website, Background Processing


Locations

Google Compute Engine




October 16, 2024 15:02 UTC
[Resolved] Our monitoring shows no further issues in Sidekiq performance after the mitigation steps were taken. We are resolving this issue and any further updates will be in: gitlab.com/gitlab-com/gl-infra/production/-/issues/18720

October 16, 2024 13:47 UTC
[Monitoring] We are continuing to see the queue recover, and are monitoring Sidekiq performance while recovery is complete. More details about this incident can be found in: gitlab.com/gitlab-com/gl-infra/production/-/issues/18720

October 16, 2024 12:49 UTC
[Monitoring] We are continuing to see the queue recover, and we are monitoring performance while recovery is complete. More details about this incident can be found in: gitlab.com/gitlab-com/gl-infra/production/-/issues/18720

October 16, 2024 12:14 UTC
[Identified] We've identified issues with delays in MR and pipeline processing as a result of Sidekiq queue lengths increasing. We're starting to see the queue recover. More details about this incident can be found in: gitlab.com/gitlab-com/gl-infra/production/-/issues/18720





Back to current status