Active Incident

Updated a few seconds ago

Back to current status

Status History

Filter: API (Clear)



February 2025

PG minor upgrade 16.6 in Production

February 15, 2025 07:49 UTC

Description

Scheduled maintenance to patch our DB layer to PostgreSQL 16.6. 2025-02-15 02:00-08:00 UTC. GitLab.com will be available throughout this time, but users may experience errors on requests. We apologize in advance for any inconvenience caused. See <gitlab.com/gitlab-com/gl-infra/production/-/issues/19120>


Components

Website, API, Container Registry, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions, Background Processing


Locations

Google Compute Engine, AWS


Schedule

February 15, 2025 03:00 - February 15, 2025 09:00 UTC



February 15, 2025 07:49 UTC
[Update] Gitlab.com database maintenance was performed, the platform is considered stable. We'll continue to monitor for any performance issues. Thank you for your patience. See <gitlab.com/gitlab-com/gl-infra/production/-/issues/19120>

February 15, 2025 04:07 UTC
[Update] Gitlab.com scheduled maintenance of our database layer has started. GitLab.com should be available during the whole period but we expect availability and performance impact until 2025-02-15 09:00 UTC. See gitlab.com/gitlab-com/gl-infra/production/-/issues/19120

February 15, 2025 03:01 UTC
[Update] Gitlab.com scheduled maintenance of our database layer has started. GitLab.com should be available during the whole period but we expect availability and performance impact until 2025-02-15 09:00 UTC. See gitlab.com/gitlab-com/gl-infra/production/-/issues/19120

February 15, 2025 02:48 UTC
[Update] Scheduled database maintenance will now start at 2025-02-15 03:00 UTC and expected to run to 2025-02-15 09:00 UTC. See gitlab.com/gitlab-com/gl-infra/production/-/issues/19120

February 15, 2025 01:54 UTC
[Update] Due to an ongoing incident we are delaying the maintenance window by 2 hours to investigate the impact. We currently plan to start the maintenance window at 04:00 UTC.

February 14, 2025 01:45 UTC
[Update] We will be undergoing scheduled maintenance of our databases tomorrow. There may be some availability and performance impact between 2025-02-15 02:00 UTC and 2025-02-15 08:00 UTC. See gitlab.com/gitlab-com/gl-infra/production/-/issues/19120

February 12, 2025 01:13 UTC
[Update] We will be undergoing scheduled maintenance of our databases in 3 days. There may be some availability and performance impact between 2025-02-15 02:00 UTC and 2025-02-15 08:00 UTC. See gitlab.com/gitlab-com/gl-infra/production/-/issues/19120

February 8, 2025 08:37 UTC
[Update] Scheduled maintenance in 1 week: Upgrading DB layer to PostgreSQL 16.6 on Feb 15, 02:00-08:00 UTC. GitLab.com will stay up but users may experience errors. See gitlab.com/gitlab-com/gl-infra/production/-/issues/19120

February 1, 2025 02:42 UTC
[Update] Scheduled maintenance in 2 weeks: Upgrading DB layer to PostgreSQL 16.6 on Feb 15, 02:00-08:00 UTC. GitLab.com will stay up but may experience errors. Sorry for any inconvenience. See gitlab.com/gitlab-com/gl-infra/production/-/issues/19120

January 2025

Accessing gitlab.com results in 500 errors

January 6, 2025 16:42 UTC

Incident Status

Partial Service Disruption


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions, SAML SSO - GitLab SaaS, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, docs.gitlab.com, Canary, Product Analytics - Configurator (Beta), Product Analytics - Collector (Beta), Product Analytics - Background processing (Beta), Product Analytics - Queries (Beta), GitLab Duo


Locations

Google Compute Engine, Digital Ocean, Zendesk, AWS




January 6, 2025 16:42 UTC
[Resolved] We've identified the cause of the errors and the errors have returned to normal. The incident is now resolved. More information on the incident can be found in this issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/19079

January 6, 2025 15:57 UTC
[Monitoring] We've identified the cause of the errors, and the errors have returned to normal. The incident is now mitigated, but we're continuing to monitor for any further impact. More information on the incident can be found in this issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/19079

January 6, 2025 15:27 UTC
[Monitoring] We're continuing to monitor for further errors related to the incident. We see further decreases in the errors.

January 6, 2025 15:17 UTC
[Monitoring] We saw elevated 500 errors on gitlab.com which have then decreased. We're continuing to monitor for further errors

December 2024

Importing with GitHub, Bitbucket Server, and Gitea importer offline

December 18, 2024 14:27 UTC

Incident Status

Security Issue


Components

Website, API


Locations

Google Compute Engine




December 18, 2024 14:27 UTC
[Resolved] Import with GitHub, Bitbucket Server, and Gitea has now been re-enabled and importers are back online.

November 22, 2024 11:59 UTC
[Investigating] Import with GitHub, Bitbucket Server, and Gitea remains offline and we continue efforts to restore functionality. Migration by Direct Transfer is now re-enabled with improved functionality on GitLab.com. See details at docs.gitlab.com/ee/user/project/import/index.html#user-contribution-and-membership-mapping

October 9, 2024 08:22 UTC
[Investigating] Migration by direct transfer and importing with GitHub, Bitbucket Server, and Gitea importer is currently offline. We are working to restore the import functionality. We do not have an estimated resolution date at this time.

August 13, 2024 08:11 UTC
[Investigating] Migration by direct transfer and importing with GitHub, Bitbucket Server, and Gitea importer is currently offline and we are investigating the cause. We will update the status page when more information is available.

July 11, 2024 06:29 UTC
[Investigating] Migration by direct transfer and importing with GitHub, Bitbucket Server, and Gitea importer is currently offline and we are investigating the cause. We will update the status page when more information becomes available.

July 11, 2024 06:22 UTC
[Investigating] Migration by direct transfer and importing with GitHub, Bitbucket Server, and Gitea importer is currently offline and we are currently investigating the cause. We will update the status page when more information becomes available.

November 2024

Database query timeout on groups API

November 6, 2024 12:59 UTC

Database query timeout on groups APIPartial Service Disruption

Incident Status

Partial Service Disruption


Components

API


Locations

Google Compute Engine




November 6, 2024 12:59 UTC
[Resolved] After a period of monitoring, we have confirmed that the issue has been resolvedSee the GitLab incident issue for more details: gitlab.com/gitlab-com/gl-infra/production/-/issues/18820

November 6, 2024 12:50 UTC
[Monitoring] The fix has now successfully been deployed to Production and we will continue to monitor this situation. See the GitLab incident issue for more details: gitlab.com/gitlab-com/gl-infra/production/-/issues/18820

November 6, 2024 11:15 UTC
[Identified] Deployment of this fix is continuing, and more updates will follow shortly. Please see gitlab.com/gitlab-com/gl-infra/production/-/issues/18820 for more information.

November 6, 2024 09:28 UTC
[Identified] A fix has been identified, and will be deployed to production shortly. More updates will follow, but please see gitlab.com/gitlab-com/gl-infra/production/-/issues/18820 for more information.

November 6, 2024 08:36 UTC
[Investigating] Investigation of this issue is continuing, with no material updates to report. Please follow gitlab.com/gitlab-com/gl-infra/production/-/issues/18820 for more information.

November 6, 2024 08:12 UTC
[Investigating] No material updates to report. We are continuing to investigate potential causes of this issue. Please follow gitlab.com/gitlab-com/gl-infra/production/-/issues/18820 for more information.

November 6, 2024 07:44 UTC
[Investigating] No material updates to report. We are continuing to investigate potential causes of this issue. Please follow gitlab.com/gitlab-com/gl-infra/production/-/issues/18820 for more information.

November 6, 2024 07:20 UTC
[Investigating] We previously resolved this issue, but with the recent increase in traffic, we're noticing a spike in error rates for the Groups API. We're re-opening this incident for further investigation. Details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/18820

November 6, 2024 03:20 UTC
[Resolved] After a period of monitoring, we have confirmed that the issue has been resolved after the rollout of the feature flag. See the GitLab incident issue for more details: gitlab.com/gitlab-com/gl-infra/production/-/issues/18811

November 6, 2024 01:45 UTC
[Monitoring] A fix has successfully been deployed to Production. The corresponding Feature Flag is being rolled-out gradually. We will continue to monitor. See the GitLab incident issue for more details: gitlab.com/gitlab-com/gl-infra/production/-/issues/18811

November 5, 2024 23:47 UTC
[Identified] A fix is currently being rolled out to production. We are still waiting for the pipeline to complete. Once completed, we will continue to monitor.

November 5, 2024 22:46 UTC
[Identified] A fix is currently being rolled out to production. Once complete, the issue should be mitigated, and we will continue to monitor.

November 5, 2024 21:37 UTC
[Identified] A fix is currently being rolled out to production. Once complete, the issue should be mitigated, and we will continue to monitor.

November 5, 2024 19:56 UTC
[Identified] We have identified the issue and are in the process of Merging a fix! We will update you once that fix has merged.

November 5, 2024 18:39 UTC
[Identified] We have identified the issue and are in the process of Merging a fix! Please stand by for updates.

November 5, 2024 17:35 UTC
[Identified] We have identified the issue and are still working with our team to resolve it. The next update is in an hour. Please standby.

November 5, 2024 16:54 UTC
[Identified] We have identified the issue and are still working with our team to resolve it. Please stand by for updates.

November 5, 2024 16:01 UTC
[Identified] The root cause has been identified, and our engineers are working on a fix. Please stand by for further updates.

November 5, 2024 15:50 UTC
[Investigating] Investigation is continuing, we will provide another update in 15 minutes.

November 5, 2024 15:35 UTC
[Investigating] Our engineers continue to investigate the root cause, please stand by for further updates.

November 5, 2024 15:20 UTC
[Investigating] Our engineers are continuing their investigation. Please stand by for further updates.

November 5, 2024 15:04 UTC
[Investigating] Our engineers continue to investigate the root cause, please stand by for further updates.

November 5, 2024 14:49 UTC
[Investigating] Investigation is continuing, we will provide another update in 15 minutes.

November 5, 2024 14:33 UTC
[Investigating] We have identified an issue with the Groups API, when the min_access_level is included in the API Call. Investigation is proceeding, and we will post an update in 15 minutes.

MAIN DB cluster upgrade

November 5, 2024 11:02 UTC

MAIN DB cluster upgrade Planned Maintenance

Description

Next week, we will be undergoing scheduled maintenance to our MAIN database layer. The maintenance will start at 2024-11-03 06:00 UTC and should finish at 2024-11-05 11:00 UTC (including performance regression observability period). GitLab.com will be available during the whole period as the maintenance should be seamless and transparent for the application. We apologize in advance for any inconvenience this may cause. See <gitlab.com/gitlab-com/gl-infra/production/-/issues/18747>


Components

Website, API, Git Operations, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions


Locations

Google Compute Engine, AWS


Schedule

November 3, 2024 06:00 - November 5, 2024 11:00 UTC



November 5, 2024 11:02 UTC
[Update] Gitlab.com MAIN database upgrade is now complete with all systems are functioning correctly. Thank you for your patience. See gitlab.com/gitlab-com/gl-infra/production/-/issues/18747

November 3, 2024 09:01 UTC
[Update] Gitlab.com MAIN database upgrade was performed. We'll continue to monitor for any performance issues until the end of the maintenance window. Thank you for your patience. See <gitlab.com/gitlab-com/gl-infra/production/-/issues/18747>

November 3, 2024 06:05 UTC
[Update] Gitlab.com scheduled maintenance of our MAIN database layer have started. GitLab.com should be available during the whole period. See <gitlab.com/gitlab-com/gl-infra/production/-/issues/18747>

November 3, 2024 05:00 UTC
[Update] MAIN DB layer maintenance will start in 1h from now at 2024-11-03 06:00 UTC. GitLab.com will remains available. Details: gitlab.com/gitlab-com/gl-infra/production/-/issues/18747

November 2, 2024 05:00 UTC
[Update] Scheduled maintenance on MAIN DB layer: Start: 2024-11-03 06:00 UTC End: 2024-11-05 11:00 UTC (incl. monitoring) GitLab.com remains available. Maintenance should be seamless. We apologize for any inconvenience. Details: gitlab.com/gitlab-com/gl-infra/production/-/issues/18747

October 30, 2024 06:00 UTC
[Update] In three days, we will be undergoing scheduled maintenance to our MAIN database layer. The maintenance will start at 2024-11-03 06:00 UTC and should finish at 2024-11-05 11:00 UTC (including performance and regression observability period). GitLab.com will be available during the whole period as the maintenance should be seamless and transparent for the application. We apologize in advance for any inconvenience this may cause. See: gitlab.com/gitlab-com/gl-infra/production/-/issues/18747

October 2024

Runner authentication verification API errors

October 31, 2024 22:50 UTC

Incident Status

Partial Service Disruption


Components

API


Locations

Google Compute Engine




October 31, 2024 22:50 UTC
[Resolved] After a period of monitoring we confirmed that the issue has been resolved.

October 31, 2024 22:39 UTC
[Monitoring] We've manually applied a fix that has stopped the error rate and mitigated the incident. We'll be monitoring for a time to ensure the issue doesn't recur. Details: gitlab.com/gitlab-com/gl-infra/production/-/issues/18792.

October 31, 2024 18:51 UTC
[Identified] We're finalizing the next method we'll be using to mitigate the incident and will be attempting another fix shortly. Details in gitlab.com/gitlab-com/gl-infra/production/-/issues/18792.

October 31, 2024 17:42 UTC
[Identified] Deployment of the fix has completed, but it is not having the impact we hoped for. We're currently weighing options for a second fix. Details in gitlab.com/gitlab-com/gl-infra/production/-/issues/18792.

October 31, 2024 14:50 UTC
[Identified] We have identified and pushed the fix, and it is currently in the process of being deployed. For more details, see the production issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/18792

October 31, 2024 09:19 UTC
[Identified] We have identified the cause of the errors, and are currently working on the fix. For more details, see the incident issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/18792

October 31, 2024 08:42 UTC
[Investigating] We are seeing elevated error rates on the POST /runners/verify endpoint. We are currently investigating. See the infrastructure issue for details: gitlab.com/gitlab-com/gl-infra/production/-/issues/18792

CI DB cluster upgrade

October 30, 2024 01:37 UTC

CI DB cluster upgradePlanned Maintenance

Description

Next week, we will be undergoing scheduled maintenance to our CI database layer. The maintenance will start at 2024-10-26 06:00 UTC and should finish at 2024-10-29 11:00 UTC (including performance regression observability period). GitLab.com will be available during the whole period as the maintenance should be seamless and transparent for the application. We apologize in advance for any inconvenience this may cause. See <gitlab.com/gitlab-com/gl-infra/production/-/issues/18639>


Components

Website, API, Git Operations, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions


Locations

Google Compute Engine, AWS


Schedule

October 26, 2024 06:00 - October 29, 2024 11:00 UTC



October 30, 2024 01:37 UTC
[Update] GitLab.com upgrade of the CI database layer is now complete with all systems are functioning correctly. Thank you for your patience. See gitlab.com/gitlab-com/gl-infra/production/-/issues/18639

October 27, 2024 08:47 UTC
[Update] GitLab.com upgrade of the CI database layer is complete. We'll be monitoring the platform to ensure all systems are functioning correctly. Thank you for your patience.

October 26, 2024 06:01 UTC
[Update] We are starting maintenance to the CI database layer, and should finish at 2024-10-29 11:00 UTC (including performance regression and observability period). GitLab.com will be available during the whole period as the maintenance should be seamless. See gitlab.com/gitlab-com/gl-infra/production/-/issues/18639

October 25, 2024 19:03 UTC
[Update] Reminder: Tomorrow, we will be undergoing scheduled maintenance to our CI database layer. The maintenance will start at 2024-10-26 06:00 UTC and should finish at 2024-10-29 11:00 UTC (including performance regression observability period). GitLab.com will be available during the whole period as the maintenance should be seamless and transparent for the application. We apologize in advance for any inconvenience this may cause. See <gitlab.com/gitlab-com/gl-infra/production/-/issues/18639>

October 23, 2024 17:25 UTC
[Update] In 3 days, we will be undergoing scheduled maintenance to our CI database layer. The maintenance will start at 2024-10-26 06:00 UTC and should finish at 2024-10-29 11:00 UTC (including performance regression observability period). GitLab.com will be available during the whole period as the maintenance should be seamless and transparent for the application. We apologize in advance for any inconvenience this may cause. See <gitlab.com/gitlab-com/gl-infra/production/-/issues/18639>

Errors on work item updates

October 10, 2024 17:44 UTC

Errors on work item updatesPartial Service Disruption

Incident Status

Partial Service Disruption


Components

Website, API


Locations

Google Compute Engine




October 10, 2024 17:44 UTC
[Resolved] The MR to remove a database foreign key has been deployed to the production. This incident is now resolved. For more details, visit gitlab.com/gitlab-com/gl-infra/production/-/issues/18692.

October 10, 2024 06:21 UTC
[Identified] We are preparing a MR to remove a database foreign key that is causing this issue. The next update will be provided once the incident has been mitigated. For more details, visit gitlab.com/gitlab-com/gl-infra/production/-/issues/18692

October 10, 2024 05:01 UTC
[Investigating] We are currently investigating an error with Work Item updates. More details about this incident can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/18692

Sidekiq job delays resulting in updates to MR approval policies being ineffective

October 4, 2024 06:59 UTC

Incident Status

Partial Service Disruption


Components

Website, API


Locations

Google Compute Engine




October 4, 2024 06:59 UTC
[Resolved] Our monitoring shows no further issues after the fix was applied. We are resolving this issue and any further updates will be in: gitlab.com/gitlab-com/gl-infra/production/-/issues/18660

October 3, 2024 22:23 UTC
[Monitoring] The fix has been successfully merged, but we are still waiting for the changes to reach production. We will continue to monitor Sidekiq’s performance, and the next update will be provided once the incident has been mitigated. For more details, visit gitlab.com/gitlab-com/gl-infra/production/-/issues/18660

October 3, 2024 21:04 UTC
[Monitoring] The fix has been successfully merged. We will continue to monitor Sidekiq’s performance to ensure everything is functioning as expected. For details, visit gitlab.com/gitlab-com/gl-infra/production/-/issues/18660

October 3, 2024 20:00 UTC
[Monitoring] The fix is expected to be merged soon. In the meantime, we’ll continue monitoring Sidekiq’s performance. Updates to MR approval policies will remain delayed until the issue is fully resolved. For more details, visit gitlab.com/gitlab-com/gl-infra/production/-/issues/18660

October 3, 2024 19:02 UTC
[Monitoring] We are still awaiting the deployment of the fix and will continue to monitor Sidekiq’s performance in the meantime. Updates to MR approval policies are delayed until the issue is resolved. For more details, visit gitlab.com/gitlab-com/gl-infra/production/-/issues/18660

October 3, 2024 18:02 UTC
[Monitoring] We continue to closely monitor Sidekiq’s performance while awaiting the deployment of the fix. In the meantime, updates to MR approval policies remain delayed. More details about this incident can be found at gitlab.com/gitlab-com/gl-infra/production/-/issues/18660

October 3, 2024 17:06 UTC
[Monitoring] We are actively monitoring the performance of Sidekiq until the fix is deployed. Updates to merge request approval policies are still experiencing delays. Details about this incident can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/18660

October 3, 2024 15:55 UTC
[Monitoring] We're continuing to monitor the performance of Sidekiq until the fix is deployed. Updates to MR approval policies are still processed with delays. The MR to fix the root cause is currently waiting to be merged. More details about this incident can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/18660

October 3, 2024 14:46 UTC
[Monitoring] We're continuing to see improvements and are monitoring until a fix is deployed. Updates to MR approval policies are still processed with delays. More details about this incident can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/18660

October 3, 2024 14:28 UTC
[Monitoring] We've identified the root cause and are recovering from a high volume of request traffic and are seeing improvements. Updates to MR approval policies are currently processing but with delays. More details about this incident can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/18660

October 3, 2024 14:12 UTC
[Investigating] We are currently investigating issues with updating MR Approval policies as a result of Sidekiq performance degradation. More details about this incident can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/18660.

September 2024

Main DB Upgrade

September 26, 2024 03:54 UTC

Main DB UpgradePlanned Maintenance

Description

We will be undergoing scheduled maintenance to our database layer. GitLab.com will continue to be available during the maintenance window. For more details, see gitlab.com/gitlab-com/gl-infra/dbre/-/issues/227


Components

Website, API, Git Operations, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions


Locations

Google Compute Engine, AWS


Schedule

October 25, 2024 00:00 - October 30, 2024 01:00 UTC



September 26, 2024 03:54 UTC
[Update] Scheduled maintenance on October 25 to our database layer has now been cancelled. For more details, see gitlab.com/gitlab-com/gl-infra/dbre/-/issues/227

CI DB upgrade

September 26, 2024 03:50 UTC

CI DB upgradePlanned Maintenance

Description

Next week, we will be undergoing scheduled maintenance to our CI database layer. The maintenance will start at 2024-09-29 00:00 UTC and should finish at 2024-10-02 02:00 UTC (including performance regression observability period). GitLab.com will be available during the whole period as the maintenance should be seamless and transparent for the application. We apologize in advance for any inconvenience this may cause. See <gitlab.com/gitlab-com/gl-infra/production/-/issues/18593>


Components

Website, API, Git Operations, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions


Locations

Google Compute Engine, AWS


Schedule

September 27, 2024 00:00 - October 2, 2024 01:00 UTC



September 26, 2024 03:50 UTC
[Update] Maintenance scheduled for 2024-09-29 on our CI database layer, has now been cancelled. For more information: gitlab.com/gitlab-com/gl-infra/production/-/issues/18593

Slowness on GitLab.com

September 21, 2024 17:55 UTC

Slowness on GitLab.comPartial Service Disruption

Incident Status

Partial Service Disruption


Components

Website, API, Git Operations


Locations

Google Compute Engine




September 21, 2024 17:55 UTC
[Resolved] Our monitoring has show no further issues after our change. We are resolving this issue and any further updates will be in: gitlab.com/gitlab-com/gl-infra/production/-/issues/18596

September 21, 2024 14:44 UTC
[Monitoring] We have identified the issue, and have made a change. We continue to see improvements. We will now continue to monitor, and if nothing changes, the next update can be expected in 3 hours. More details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/18596

September 21, 2024 14:09 UTC
[Identified] We are recovering from a high volume of request traffic and are seeing improvements. More information in: gitlab.com/gitlab-com/gl-infra/production/-/issues/18596

September 21, 2024 13:45 UTC
[Investigating] We are seeing slow responses from our API, and we are still investigating. More information in: gitlab.com/gitlab-com/gl-infra/production/-/issues/18596

September 21, 2024 13:27 UTC
[Investigating] No material updates at this point. Our Engineers are still investigating. More information in: gitlab.com/gitlab-com/gl-infra/production/-/issues/18596

September 21, 2024 13:10 UTC
[Investigating] We are currently investigating problems with slowness on GitLab.com. More information in: gitlab.com/gitlab-com/gl-infra/production/-/issues/18596

Intermittent 503 errors

September 20, 2024 21:40 UTC

Incident Status

Operational


Components

Website, API, Canary


Locations

Google Compute Engine




September 20, 2024 21:40 UTC
[Resolved] GitLab has not seen any further errors related to this incident in our monitoring and we are resolving the incident. Any further updates are in the incident issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/18591.

September 20, 2024 20:53 UTC
[Monitoring] Our rollout has completed and we believe that the issue is mitigated. We are moving the incident to Monitoring to validate the fix. gitlab.com/gitlab-com/gl-infra/production/-/issues/18591

September 20, 2024 15:08 UTC
[Identified] No material updates to report. The roll out of the fix is still ongoing. More details are available in gitlab.com/gitlab-com/gl-infra/production/-/issues/18591.

September 20, 2024 13:54 UTC
[Identified] We are seeing an uptick in errors with git operations. The roll out of the fix is still ongoing. More details are available in gitlab.com/gitlab-com/gl-infra/production/-/issues/18591.

September 20, 2024 13:12 UTC
[Identified] We are seeing an increased number of 503 errors in the UI and API. We have identified the issue and are rolling out a fix. More details are available in gitlab.com/gitlab-com/gl-infra/production/-/issues/18591.

Missing Access Tokens

September 12, 2024 06:53 UTC

Missing Access TokensPartial Service Disruption

Incident Status

Partial Service Disruption


Components

API


Locations

Google Compute Engine




September 12, 2024 06:53 UTC
[Resolved] We have reached out to all affected customers on remedying the problem. We apologize for any inconvenience this may cause. This incident is now marked as Resolved. More information can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/18548.

September 11, 2024 20:16 UTC
[Monitoring] No updates to report at this time. We are currently reaching out to customers impacted as to remedy the problem. More details can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/18548.

September 11, 2024 12:11 UTC
[Monitoring] We've resolved the issue with the cron job responsible for revoking tokens. We are proactively reaching out to customers impacted as to remedy the problem. More details can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/18548.

September 11, 2024 11:53 UTC
[Identified] We've identified cause of the issue and have stopped the cron job responsible. To resolve the issue please recreate the token. More details can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/18548.

September 11, 2024 11:29 UTC
[Investigating] We are currently investigating issues with missing access tokens. More details about this incident can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/18548

Registry DB upgrade

September 11, 2024 22:51 UTC

Registry DB upgradePlanned Maintenance

Description

We will be undergoing scheduled maintenance to our database layer. GitLab.com will continue to be available during the maintenance window. For more details, see gitlab.com/gitlab-com/gl-infra/dbre/-/issues/225


Components

Website, API, Git Operations, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions


Locations

Google Compute Engine, AWS


Schedule

September 6, 2024 00:00 - September 11, 2024 01:00 UTC



September 11, 2024 22:51 UTC
[Update] Planned maintenance for the registry database layer on GitLab.com is complete. Registry database is now running Postgres v16. The rollback window for the Registry database was closed on approximately 2024-09-12 1:00 UTC.

September 8, 2024 01:51 UTC
[Update] Planned maintenance for the registry database layer on GitLab.com is complete. Registry database is now running Postgres v16. Rollback window for this planned maintenance ends 2024-09-11 00:00 UTC. We'll be monitoring the platform during this time to ensure all systems are functioning correctly.

September 8, 2024 00:11 UTC
[Update] Scheduled maintenance to our registry database layer has started. The maintenance should finish at 2024-09-11 02:00 UTC UTC (including performance regression observability period). GitLab.com will be available during the whole period as the maintenance should be seamless and transparent for the application. We apologize in advance for any inconvenience this may cause. See issue for more information: gitlab.com/gitlab-com/gl-infra/production/-/issues/18447

September 5, 2024 05:53 UTC
[Update] In 3 days, we will be undergoing scheduled maintenance to our registry database layer. The maintenance will start at 2024-09-08 00:00 UTC UTC and should finish at 2024-09-11 02:00 UTC UTC (including performance regression observability period). GitLab.com will be available during the whole period as the maintenance should be seamless and transparent for the application. We apologize in advance for any inconvenience this may cause. See <gitlab.com/gitlab-com/gl-infra/production/-/issues/18447>

GitLab connectivity issues

September 10, 2024 13:40 UTC

Incident Status

Operational


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions


Locations

Google Compute Engine




September 10, 2024 13:40 UTC
[Resolved] This incident has been resolved. More information can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/18535.

September 10, 2024 13:33 UTC
[Monitoring] The website has recovered are and we are now monitoring before marking as resolved. More details can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/18535.

September 10, 2024 13:24 UTC
[Investigating] No material updates at this time - our investigation is ongoing. More details can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/18535

September 10, 2024 13:10 UTC
[Investigating]

September 10, 2024 13:08 UTC
[Investigating] We are currently investigating connectivity issues to GitLab.com. For more information see gitlab.com/gitlab-com/gl-infra/production/-/issues/18535

GitLab.com Connectivity Issues

September 3, 2024 05:34 UTC

Incident Status

Service Disruption


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions, SAML SSO - GitLab SaaS, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, docs.gitlab.com, Canary, Product Analytics - Configurator (Beta), Product Analytics - Collector (Beta), Product Analytics - Background processing (Beta), Product Analytics - Queries (Beta), GitLab Duo


Locations

Google Compute Engine, Digital Ocean, Zendesk, AWS




September 3, 2024 05:34 UTC
[Resolved] All GitLab services have been restored. We are actively monitoring the system, and everything appears normal at this time. For more details, please follow the issue at: gitlab.com/gitlab-com/gl-infra/production/-/issues/18490

September 3, 2024 05:19 UTC
[Monitoring] We've identified and addressed the root cause of the issue. All GitLab services have been restored, and we are actively monitoring the system. Follow the issue for more details : gitlab.com/gitlab-com/gl-infra/production/-/issues/18490

September 3, 2024 05:09 UTC
[Investigating] GitLab.com is currently experiencing connectivity issues due to a high error rate. Our infrastructure team is investigating. Issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/18490

August 2024

Embedding DB Upgrade

August 31, 2024 04:58 UTC

Embedding DB UpgradePlanned Maintenance

Description

We will be undergoing scheduled maintenance to our database layer. GitLab.com will continue to be available during the maintenance window. For more details, see gitlab.com/gitlab-com/gl-infra/dbre/-/issues/228


Components

Website, API, Git Operations, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions


Locations

Google Compute Engine, AWS


Schedule

October 25, 2024 00:00 - October 30, 2024 01:00 UTC



August 31, 2024 04:58 UTC
[Update] We are cancelling the Embedding database upgrade. See gitlab.com/gitlab-com/gl-infra/dbre/-/issues/228 for details. All other DB upgrade schedules will go ahead as planned.

Some requests timing out

August 23, 2024 03:59 UTC

Incident Status

Operational


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions, SAML SSO - GitLab SaaS, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, docs.gitlab.com, Canary, Product Analytics - Configurator (Beta), Product Analytics - Collector (Beta), Product Analytics - Background processing (Beta), Product Analytics - Queries (Beta), GitLab Duo


Locations

Google Compute Engine, Digital Ocean, Zendesk, AWS




August 23, 2024 03:59 UTC
[Resolved] The issue has been resolved and all functionality on GitLab.com is nominal. Please see gitlab.com/gitlab-com/gl-infra/production/-/issues/18435 for more details.

August 23, 2024 01:53 UTC
[Monitoring] The issue has been identified and the incident resolved. We will continue monitoring for a while.

August 23, 2024 01:27 UTC
[Identified] We have identified an underlying cause and are working to mitigate it. See gitlab.com/gitlab-com/gl-infra/production/-/issues/18435 for more details.

August 23, 2024 00:34 UTC
[Investigating] We have noticed that there are delays in web UI updates for example, MRs not showing up after submission, We are continuing to investigate - please see gitlab.com/gitlab-com/gl-infra/production/-/issues/18435 for more details.

August 23, 2024 00:27 UTC
[Investigating] Some requests are slow or timing out due to DB saturation. We are investigating. Please see [this issue](gitlab.com/gitlab-com/gl-infra/production/-/issues/18435) for further details.

July 2024

GitLab.com is unavailable

July 11, 2024 18:28 UTC

GitLab.com is unavailableService Disruption

Incident Status

Service Disruption


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions, SAML SSO - GitLab SaaS, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, docs.gitlab.com, Canary, Product Analytics - Configurator (Beta), Product Analytics - Collector (Beta), Product Analytics - Background processing (Beta), Product Analytics - Queries (Beta), GitLab Duo


Locations

Google Compute Engine, Digital Ocean, Zendesk, AWS




July 11, 2024 18:28 UTC
[Resolved] This incident is now resolved and no further errors are being seen on GitLab.com. Our investigation points to a disk performance issue. You can see further details and corrective actions in gitlab.com/gitlab-com/gl-infra/production/-/issues/18269.

July 11, 2024 15:51 UTC
[Investigating] Our investigations are ongoing. We will post our next update when we have more information. Please follow the incident issue for updates here: gitlab.com/gitlab-com/gl-infra/production/-/issues/18269

July 11, 2024 15:28 UTC
[Investigating] No material updates to report. We're continuing to investigate the issue. Please follow the incident issue for updates here: gitlab.com/gitlab-com/gl-infra/production/-/issues/18269

July 11, 2024 15:00 UTC
[Investigating] We are seeing improved service availability and investigations continue for this problem. Please follow the incident issue for updates here: gitlab.com/gitlab-com/gl-infra/production/-/issues/18269

July 11, 2024 14:43 UTC
[Investigating] We are seeing improved service availability and investigations continue for this problem. We are planning to make the incident issue publicly available as soon as possible.

July 11, 2024 14:29 UTC
[Investigating] We are investigating a problem with gitlab.com being unavailable for some users. Our investigations continue and we will post further status updates shortly.

July 11, 2024 14:18 UTC
[Investigating] We are investigating a problem with gitlab.com being unavailable for some users. We will post further status updates shortly.





Back to current status