Active Issue

Updated a few seconds ago

Back to current status

Status History

Filter: Git Operations (Clear)



January 2025

Accessing gitlab.com results in 500 errors

January 6, 2025 16:42 UTC

Incident Status

Partial Service Disruption


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions, SAML SSO - GitLab SaaS, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, docs.gitlab.com, Canary, Product Analytics - Configurator (Beta), Product Analytics - Collector (Beta), Product Analytics - Background processing (Beta), Product Analytics - Queries (Beta), GitLab Duo


Locations

Google Compute Engine, Digital Ocean, Zendesk, AWS




January 6, 2025 16:42 UTC
[Resolved] We've identified the cause of the errors and the errors have returned to normal. The incident is now resolved. More information on the incident can be found in this issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/19079

January 6, 2025 15:57 UTC
[Monitoring] We've identified the cause of the errors, and the errors have returned to normal. The incident is now mitigated, but we're continuing to monitor for any further impact. More information on the incident can be found in this issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/19079

January 6, 2025 15:27 UTC
[Monitoring] We're continuing to monitor for further errors related to the incident. We see further decreases in the errors.

January 6, 2025 15:17 UTC
[Monitoring] We saw elevated 500 errors on gitlab.com which have then decreased. We're continuing to monitor for further errors

December 2024

Intermittent timeouts for requests from some Utah IP addresses

December 20, 2024 02:51 UTC

Incident Status

Partial Service Disruption


Components

Git Operations, Container Registry


Locations

Google Compute Engine




December 20, 2024 02:51 UTC
[Resolved] We are now marking this incident as resolved. Details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/19017

December 19, 2024 01:48 UTC
[Monitoring] Google's network engineers have identified the issue affecting the "us-west3" region and deployed a fix. We will continue monitoring. For questions and more details, please follow gitlab.com/gitlab-com/gl-infra/production/-/issues/19017.

December 18, 2024 22:15 UTC
[Investigating] No material updates at this time. Cloudlfare and Google are still investigating these routing issues. We will provide additional information as it becomes available. Details will be posted to: gitlab.com/gitlab-com/gl-infra/production/-/issues/19017

December 18, 2024 03:57 UTC
[Investigating] Our Infrastructure Engineers have been working with CloudFlare and Google and have identified a routing issue between GCP "us-west3" and Cloudflare. Investigation is continuing. Please follow gitlab.com/gitlab-com/gl-infra/production/-/issues/19017 for more details.

December 17, 2024 22:37 UTC
[Investigating] No material updates at this time. Requests originating from the Salt Lake City, Utah region may still see intermittent timeouts. We are working with our third party vendor to investigate further. We will provide updates as info becomes available. Details: gitlab.com/gitlab-com/gl-infra/production/-/issues/19017

December 17, 2024 13:45 UTC
[Investigating] No material updates at the moment. We continue to work with a third party vendor to get more updates and waiting for feedback from affected customers. Details to follow in: gitlab.com/gitlab-com/gl-infra/production/-/issues/19017

December 16, 2024 23:49 UTC
[Investigating] We have received some reports of requests timing out for IP addresses originating in Salt Lake City, Utah. We are working with a third party vendor to identify the issue. Details to follow in: gitlab.com/gitlab-com/gl-infra/production/-/issues/19017

November 2024

MAIN DB cluster upgrade

November 5, 2024 11:02 UTC

MAIN DB cluster upgrade Planned Maintenance

Description

Next week, we will be undergoing scheduled maintenance to our MAIN database layer. The maintenance will start at 2024-11-03 06:00 UTC and should finish at 2024-11-05 11:00 UTC (including performance regression observability period). GitLab.com will be available during the whole period as the maintenance should be seamless and transparent for the application. We apologize in advance for any inconvenience this may cause. See <gitlab.com/gitlab-com/gl-infra/production/-/issues/18747>


Components

Website, API, Git Operations, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions


Locations

Google Compute Engine, AWS


Schedule

November 3, 2024 06:00 - November 5, 2024 11:00 UTC



November 5, 2024 11:02 UTC
[Update] Gitlab.com MAIN database upgrade is now complete with all systems are functioning correctly. Thank you for your patience. See gitlab.com/gitlab-com/gl-infra/production/-/issues/18747

November 3, 2024 09:01 UTC
[Update] Gitlab.com MAIN database upgrade was performed. We'll continue to monitor for any performance issues until the end of the maintenance window. Thank you for your patience. See <gitlab.com/gitlab-com/gl-infra/production/-/issues/18747>

November 3, 2024 06:05 UTC
[Update] Gitlab.com scheduled maintenance of our MAIN database layer have started. GitLab.com should be available during the whole period. See <gitlab.com/gitlab-com/gl-infra/production/-/issues/18747>

November 3, 2024 05:00 UTC
[Update] MAIN DB layer maintenance will start in 1h from now at 2024-11-03 06:00 UTC. GitLab.com will remains available. Details: gitlab.com/gitlab-com/gl-infra/production/-/issues/18747

November 2, 2024 05:00 UTC
[Update] Scheduled maintenance on MAIN DB layer: Start: 2024-11-03 06:00 UTC End: 2024-11-05 11:00 UTC (incl. monitoring) GitLab.com remains available. Maintenance should be seamless. We apologize for any inconvenience. Details: gitlab.com/gitlab-com/gl-infra/production/-/issues/18747

October 30, 2024 06:00 UTC
[Update] In three days, we will be undergoing scheduled maintenance to our MAIN database layer. The maintenance will start at 2024-11-03 06:00 UTC and should finish at 2024-11-05 11:00 UTC (including performance and regression observability period). GitLab.com will be available during the whole period as the maintenance should be seamless and transparent for the application. We apologize in advance for any inconvenience this may cause. See: gitlab.com/gitlab-com/gl-infra/production/-/issues/18747

October 2024

CI DB cluster upgrade

October 30, 2024 01:37 UTC

CI DB cluster upgradePlanned Maintenance

Description

Next week, we will be undergoing scheduled maintenance to our CI database layer. The maintenance will start at 2024-10-26 06:00 UTC and should finish at 2024-10-29 11:00 UTC (including performance regression observability period). GitLab.com will be available during the whole period as the maintenance should be seamless and transparent for the application. We apologize in advance for any inconvenience this may cause. See <gitlab.com/gitlab-com/gl-infra/production/-/issues/18639>


Components

Website, API, Git Operations, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions


Locations

Google Compute Engine, AWS


Schedule

October 26, 2024 06:00 - October 29, 2024 11:00 UTC



October 30, 2024 01:37 UTC
[Update] GitLab.com upgrade of the CI database layer is now complete with all systems are functioning correctly. Thank you for your patience. See gitlab.com/gitlab-com/gl-infra/production/-/issues/18639

October 27, 2024 08:47 UTC
[Update] GitLab.com upgrade of the CI database layer is complete. We'll be monitoring the platform to ensure all systems are functioning correctly. Thank you for your patience.

October 26, 2024 06:01 UTC
[Update] We are starting maintenance to the CI database layer, and should finish at 2024-10-29 11:00 UTC (including performance regression and observability period). GitLab.com will be available during the whole period as the maintenance should be seamless. See gitlab.com/gitlab-com/gl-infra/production/-/issues/18639

October 25, 2024 19:03 UTC
[Update] Reminder: Tomorrow, we will be undergoing scheduled maintenance to our CI database layer. The maintenance will start at 2024-10-26 06:00 UTC and should finish at 2024-10-29 11:00 UTC (including performance regression observability period). GitLab.com will be available during the whole period as the maintenance should be seamless and transparent for the application. We apologize in advance for any inconvenience this may cause. See <gitlab.com/gitlab-com/gl-infra/production/-/issues/18639>

October 23, 2024 17:25 UTC
[Update] In 3 days, we will be undergoing scheduled maintenance to our CI database layer. The maintenance will start at 2024-10-26 06:00 UTC and should finish at 2024-10-29 11:00 UTC (including performance regression observability period). GitLab.com will be available during the whole period as the maintenance should be seamless and transparent for the application. We apologize in advance for any inconvenience this may cause. See <gitlab.com/gitlab-com/gl-infra/production/-/issues/18639>

September 2024

Main DB Upgrade

September 26, 2024 03:54 UTC

Main DB UpgradePlanned Maintenance

Description

We will be undergoing scheduled maintenance to our database layer. GitLab.com will continue to be available during the maintenance window. For more details, see gitlab.com/gitlab-com/gl-infra/dbre/-/issues/227


Components

Website, API, Git Operations, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions


Locations

Google Compute Engine, AWS


Schedule

October 25, 2024 00:00 - October 30, 2024 01:00 UTC



September 26, 2024 03:54 UTC
[Update] Scheduled maintenance on October 25 to our database layer has now been cancelled. For more details, see gitlab.com/gitlab-com/gl-infra/dbre/-/issues/227

CI DB upgrade

September 26, 2024 03:50 UTC

CI DB upgradePlanned Maintenance

Description

Next week, we will be undergoing scheduled maintenance to our CI database layer. The maintenance will start at 2024-09-29 00:00 UTC and should finish at 2024-10-02 02:00 UTC (including performance regression observability period). GitLab.com will be available during the whole period as the maintenance should be seamless and transparent for the application. We apologize in advance for any inconvenience this may cause. See <gitlab.com/gitlab-com/gl-infra/production/-/issues/18593>


Components

Website, API, Git Operations, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions


Locations

Google Compute Engine, AWS


Schedule

September 27, 2024 00:00 - October 2, 2024 01:00 UTC



September 26, 2024 03:50 UTC
[Update] Maintenance scheduled for 2024-09-29 on our CI database layer, has now been cancelled. For more information: gitlab.com/gitlab-com/gl-infra/production/-/issues/18593

Slowness on GitLab.com

September 21, 2024 17:55 UTC

Slowness on GitLab.comPartial Service Disruption

Incident Status

Partial Service Disruption


Components

Website, API, Git Operations


Locations

Google Compute Engine




September 21, 2024 17:55 UTC
[Resolved] Our monitoring has show no further issues after our change. We are resolving this issue and any further updates will be in: gitlab.com/gitlab-com/gl-infra/production/-/issues/18596

September 21, 2024 14:44 UTC
[Monitoring] We have identified the issue, and have made a change. We continue to see improvements. We will now continue to monitor, and if nothing changes, the next update can be expected in 3 hours. More details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/18596

September 21, 2024 14:09 UTC
[Identified] We are recovering from a high volume of request traffic and are seeing improvements. More information in: gitlab.com/gitlab-com/gl-infra/production/-/issues/18596

September 21, 2024 13:45 UTC
[Investigating] We are seeing slow responses from our API, and we are still investigating. More information in: gitlab.com/gitlab-com/gl-infra/production/-/issues/18596

September 21, 2024 13:27 UTC
[Investigating] No material updates at this point. Our Engineers are still investigating. More information in: gitlab.com/gitlab-com/gl-infra/production/-/issues/18596

September 21, 2024 13:10 UTC
[Investigating] We are currently investigating problems with slowness on GitLab.com. More information in: gitlab.com/gitlab-com/gl-infra/production/-/issues/18596

Registry DB upgrade

September 11, 2024 22:51 UTC

Registry DB upgradePlanned Maintenance

Description

We will be undergoing scheduled maintenance to our database layer. GitLab.com will continue to be available during the maintenance window. For more details, see gitlab.com/gitlab-com/gl-infra/dbre/-/issues/225


Components

Website, API, Git Operations, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions


Locations

Google Compute Engine, AWS


Schedule

September 6, 2024 00:00 - September 11, 2024 01:00 UTC



September 11, 2024 22:51 UTC
[Update] Planned maintenance for the registry database layer on GitLab.com is complete. Registry database is now running Postgres v16. The rollback window for the Registry database was closed on approximately 2024-09-12 1:00 UTC.

September 8, 2024 01:51 UTC
[Update] Planned maintenance for the registry database layer on GitLab.com is complete. Registry database is now running Postgres v16. Rollback window for this planned maintenance ends 2024-09-11 00:00 UTC. We'll be monitoring the platform during this time to ensure all systems are functioning correctly.

September 8, 2024 00:11 UTC
[Update] Scheduled maintenance to our registry database layer has started. The maintenance should finish at 2024-09-11 02:00 UTC UTC (including performance regression observability period). GitLab.com will be available during the whole period as the maintenance should be seamless and transparent for the application. We apologize in advance for any inconvenience this may cause. See issue for more information: gitlab.com/gitlab-com/gl-infra/production/-/issues/18447

September 5, 2024 05:53 UTC
[Update] In 3 days, we will be undergoing scheduled maintenance to our registry database layer. The maintenance will start at 2024-09-08 00:00 UTC UTC and should finish at 2024-09-11 02:00 UTC UTC (including performance regression observability period). GitLab.com will be available during the whole period as the maintenance should be seamless and transparent for the application. We apologize in advance for any inconvenience this may cause. See <gitlab.com/gitlab-com/gl-infra/production/-/issues/18447>

GitLab connectivity issues

September 10, 2024 13:40 UTC

Incident Status

Operational


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions


Locations

Google Compute Engine




September 10, 2024 13:40 UTC
[Resolved] This incident has been resolved. More information can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/18535.

September 10, 2024 13:33 UTC
[Monitoring] The website has recovered are and we are now monitoring before marking as resolved. More details can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/18535.

September 10, 2024 13:24 UTC
[Investigating] No material updates at this time - our investigation is ongoing. More details can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/18535

September 10, 2024 13:10 UTC
[Investigating]

September 10, 2024 13:08 UTC
[Investigating] We are currently investigating connectivity issues to GitLab.com. For more information see gitlab.com/gitlab-com/gl-infra/production/-/issues/18535

GitLab.com Connectivity Issues

September 3, 2024 05:34 UTC

Incident Status

Service Disruption


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions, SAML SSO - GitLab SaaS, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, docs.gitlab.com, Canary, Product Analytics - Configurator (Beta), Product Analytics - Collector (Beta), Product Analytics - Background processing (Beta), Product Analytics - Queries (Beta), GitLab Duo


Locations

Google Compute Engine, Digital Ocean, Zendesk, AWS




September 3, 2024 05:34 UTC
[Resolved] All GitLab services have been restored. We are actively monitoring the system, and everything appears normal at this time. For more details, please follow the issue at: gitlab.com/gitlab-com/gl-infra/production/-/issues/18490

September 3, 2024 05:19 UTC
[Monitoring] We've identified and addressed the root cause of the issue. All GitLab services have been restored, and we are actively monitoring the system. Follow the issue for more details : gitlab.com/gitlab-com/gl-infra/production/-/issues/18490

September 3, 2024 05:09 UTC
[Investigating] GitLab.com is currently experiencing connectivity issues due to a high error rate. Our infrastructure team is investigating. Issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/18490

August 2024

Embedding DB Upgrade

August 31, 2024 04:58 UTC

Embedding DB UpgradePlanned Maintenance

Description

We will be undergoing scheduled maintenance to our database layer. GitLab.com will continue to be available during the maintenance window. For more details, see gitlab.com/gitlab-com/gl-infra/dbre/-/issues/228


Components

Website, API, Git Operations, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions


Locations

Google Compute Engine, AWS


Schedule

October 25, 2024 00:00 - October 30, 2024 01:00 UTC



August 31, 2024 04:58 UTC
[Update] We are cancelling the Embedding database upgrade. See gitlab.com/gitlab-com/gl-infra/dbre/-/issues/228 for details. All other DB upgrade schedules will go ahead as planned.

Some requests timing out

August 23, 2024 03:59 UTC

Incident Status

Operational


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions, SAML SSO - GitLab SaaS, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, docs.gitlab.com, Canary, Product Analytics - Configurator (Beta), Product Analytics - Collector (Beta), Product Analytics - Background processing (Beta), Product Analytics - Queries (Beta), GitLab Duo


Locations

Google Compute Engine, Digital Ocean, Zendesk, AWS




August 23, 2024 03:59 UTC
[Resolved] The issue has been resolved and all functionality on GitLab.com is nominal. Please see gitlab.com/gitlab-com/gl-infra/production/-/issues/18435 for more details.

August 23, 2024 01:53 UTC
[Monitoring] The issue has been identified and the incident resolved. We will continue monitoring for a while.

August 23, 2024 01:27 UTC
[Identified] We have identified an underlying cause and are working to mitigate it. See gitlab.com/gitlab-com/gl-infra/production/-/issues/18435 for more details.

August 23, 2024 00:34 UTC
[Investigating] We have noticed that there are delays in web UI updates for example, MRs not showing up after submission, We are continuing to investigate - please see gitlab.com/gitlab-com/gl-infra/production/-/issues/18435 for more details.

August 23, 2024 00:27 UTC
[Investigating] Some requests are slow or timing out due to DB saturation. We are investigating. Please see [this issue](gitlab.com/gitlab-com/gl-infra/production/-/issues/18435) for further details.

July 2024

GitLab.com is unavailable

July 11, 2024 18:28 UTC

GitLab.com is unavailableService Disruption

Incident Status

Service Disruption


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions, SAML SSO - GitLab SaaS, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, docs.gitlab.com, Canary, Product Analytics - Configurator (Beta), Product Analytics - Collector (Beta), Product Analytics - Background processing (Beta), Product Analytics - Queries (Beta), GitLab Duo


Locations

Google Compute Engine, Digital Ocean, Zendesk, AWS




July 11, 2024 18:28 UTC
[Resolved] This incident is now resolved and no further errors are being seen on GitLab.com. Our investigation points to a disk performance issue. You can see further details and corrective actions in gitlab.com/gitlab-com/gl-infra/production/-/issues/18269.

July 11, 2024 15:51 UTC
[Investigating] Our investigations are ongoing. We will post our next update when we have more information. Please follow the incident issue for updates here: gitlab.com/gitlab-com/gl-infra/production/-/issues/18269

July 11, 2024 15:28 UTC
[Investigating] No material updates to report. We're continuing to investigate the issue. Please follow the incident issue for updates here: gitlab.com/gitlab-com/gl-infra/production/-/issues/18269

July 11, 2024 15:00 UTC
[Investigating] We are seeing improved service availability and investigations continue for this problem. Please follow the incident issue for updates here: gitlab.com/gitlab-com/gl-infra/production/-/issues/18269

July 11, 2024 14:43 UTC
[Investigating] We are seeing improved service availability and investigations continue for this problem. We are planning to make the incident issue publicly available as soon as possible.

July 11, 2024 14:29 UTC
[Investigating] We are investigating a problem with gitlab.com being unavailable for some users. Our investigations continue and we will post further status updates shortly.

July 11, 2024 14:18 UTC
[Investigating] We are investigating a problem with gitlab.com being unavailable for some users. We will post further status updates shortly.

June 2024

429 rate limiting errors

June 20, 2024 10:30 UTC

429 rate limiting errorsDegraded Performance

Incident Status

Degraded Performance


Components

Website, Git Operations


Locations

Google Compute Engine




June 20, 2024 10:30 UTC
[Resolved] This incident has been resolved. More information can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/18173.

June 20, 2024 10:21 UTC
[Monitoring] We identified the root cause and have applied a fix, we are monitoring the recovery. More details can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/18173.

June 20, 2024 09:59 UTC
[Investigating] We are investigating an increase in requests being rate limited resulting in 429 errors. For more information see gitlab.com/gitlab-com/gl-infra/production/-/issues/18173

Multiple reports of authentication failures accessing GitLab.com

June 18, 2024 09:03 UTC

Incident Status

Operational


Components

Website, API, Git Operations, Container Registry


Locations

Google Compute Engine




June 18, 2024 09:03 UTC
[Resolved] This incident has been resolved. More information can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/18003.

May 30, 2024 13:02 UTC
[Monitoring] We have temporarily increased the rate limits for unauthenticated requests, to alleviate errors resulting from the token lifetime limits implementation. We are continuing to monitor and adjust these rate limits as appropriate.

May 14, 2024 16:12 UTC
[Monitoring] We have temporarily increased the rate limits for unauthenticated requests, to alleviate errors resulting from the token lifetime limits implementation. Please see the issue for more details: gitlab.com/gitlab-com/gl-infra/production/-/issues/18003

May 14, 2024 03:27 UTC
[Identified] Token lifetime limits have been implemented on GitLab.com. If you are experiencing increased authentication errors accessing GitLab.com, please see our blog post: about.gitlab.com/blog/2023/10/25/access-token-lifetime-limits for details on how to mitigate problems with authentication.

April 2024

Degraded performance causing intermittent 500 errors

April 8, 2024 15:12 UTC

Incident Status

Service Disruption


Components

Website, Git Operations


Locations

Google Compute Engine




April 8, 2024 15:12 UTC
[Resolved] This incident has been resolved. More information can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/17790

April 8, 2024 14:37 UTC
[Identified] We have identified the cause for the increase in error rates and have taken steps to mitigate the source of the errors. For more details see gitlab.com/gitlab-com/gl-infra/production/-/issues/17790

April 8, 2024 14:21 UTC
[Investigating] We are still investigating an increase in error rates on some projects when viewing MRs and project details. For more details see gitlab.com/gitlab-com/gl-infra/production/-/issues/17790

April 8, 2024 14:02 UTC
[Investigating] We are still investigating the increase in error rates on some projects when viewing MRs and project details. For more details see gitlab.com/gitlab-com/gl-infra/production/-/issues/17790

April 8, 2024 13:42 UTC
[Investigating] We are investigating an increase in error rates on some projects when viewing MRs and project details. For more details see gitlab.com/gitlab-com/gl-infra/production/-/issues/17790

October 2023

GitLab.com availability issues

October 30, 2023 18:33 UTC

Incident Status

Service Disruption


Components

Website, API, Git Operations


Locations

Google Compute Engine




October 30, 2023 18:33 UTC
[Resolved] Import functionality has been restored and operation of GitLab.com is fully returned to normal. We're marking the incident resolved, please see the issue for further details: gitlab.com/gitlab-com/gl-infra/production/-/issues/17054

October 30, 2023 17:58 UTC
[Monitoring] The incident response team has mitigated the root cause of the performance impacts and is now working to restore full import functionality. Monitoring is ongoing to ensure there are no further impacts. Issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/17054

October 30, 2023 17:28 UTC
[Identified] Performance continues to improve and the team is continuing to work on mitigations to prevent future impact.

October 30, 2023 17:09 UTC
[Identified] We continue to work towards full mitigation. Performance of GitLab.com is returning, though project import remains partially disabled while the team investigates.

October 30, 2023 16:54 UTC
[Identified] A likely cause of the database load has been identified. Our incident response team has temporarily disabled part of the project import feature while we continue to investigate.

October 30, 2023 16:32 UTC
[Investigating] We are seeing signs of improvement, and the team has narrowed the potential causes. Mitigation and additional investigation are in progress.

October 30, 2023 16:23 UTC
[Investigating] We've confirmed potential impacts to Git operations and API as well. Investigation still ongoing.

October 30, 2023 16:10 UTC
[Investigating] The incident response team is continuing to investigate causes of the increased database usage, additional teams are being brought in to assist.

October 30, 2023 15:55 UTC
[Investigating] We are investigating increased database usage as a primary cause. We will update as more information becomes available.

October 30, 2023 15:39 UTC
[Investigating] We are currently investigating availability issues with on GitLab.com. We will update as more information becomes available.

September 2023

Elevated error reates and slow response times

September 13, 2023 09:29 UTC

Incident Status

Degraded Performance


Components

Website, API, Git Operations, Container Registry


Locations

Google Compute Engine




September 13, 2023 09:29 UTC
[Resolved] The issue has been resolved and services are operating normally. The investigation summary will be posted in gitlab.com/gitlab-com/gl-infra/production/-/issues/16359

September 13, 2023 08:23 UTC
[Monitoring] We're now seeing error rates and performance returning back to normal operation. We're continuing to monitor to ensure the problem does not recur. Details in gitlab.com/gitlab-com/gl-infra/production/-/issues/16359

September 13, 2023 08:09 UTC
[Investigating] We are investigating an increase in errors and slower response times on GitLab.com. For details see gitlab.com/gitlab-com/gl-infra/production/-/issues/16359

Main DB Cluster PostgreSQL 14 Upgrade

September 9, 2023 17:25 UTC

Description

We will be undergoing scheduled maintenance to our database layer. The maintenance will take up to 6 hours, from 14:00 UTC to 20:00 UTC. GitLab.com will continue to be available during the maintenance window. We apologize in advance for any inconvenience this may cause. See gitlab.com/gitlab-com/gl-infra/production/-/issues/16266 for details.


Components

Website, API, Git Operations, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners for GitLab community contributions


Locations

Google Compute Engine


Schedule

September 9, 2023 14:00 - September 9, 2023 20:00 UTC



September 9, 2023 17:25 UTC
[Update] GitLab.com planned maintenance for the database layer is complete. We'll be monitoring the platform to ensure all systems are functioning correctly. Thank you for your patience.

September 9, 2023 16:12 UTC
[Update] GitLab.com planned maintenance for the database layer is almost complete. We’re continuing to verify that all systems are functioning correctly. Thank you for your patience. See <gitlab.com/gitlab-com/gl-infra/production/-/issues/16266>

September 9, 2023 13:01 UTC
[Update] We will be undergoing scheduled maintenance to our main database layer in 1 hour. The maintenance will take up to 5 hours starting from 14:00 UTC to 19:00 UTC. GitLab.com will be available but users may experience degraded performance during the maintenance window. We apologize in advance for any inconvenience this may cause. See gitlab.com/gitlab-com/gl-infra/production/-/issues/16266

September 9, 2023 10:08 UTC
[Update] We will be undergoing scheduled maintenance to our main database layer in 3 hours. The maintenance will take up to 5 hours starting from 14:00 UTC to 19:00 UTC. GitLab.com will be available but users may experience degraded performance during the maintenance window. We apologize in advance for any inconvenience this may cause. See gitlab.com/gitlab-com/gl-infra/production/-/issues/16266

September 8, 2023 13:59 UTC
[Update] Reminder: Tomorrow, we will be undergoing scheduled maintenance to our main database layer. The maintenance will take up to 5 hours starting from 14:00 UTC to 19:00 UTC. GitLab.com will be available but users may experience degraded performance during the maintenance window. We apologize in advance for any inconvenience this may cause. See gitlab.com/gitlab-com/gl-infra/production/-/issues/16266

September 6, 2023 12:13 UTC
[Update] In 3 days, we will be undergoing scheduled maintenance to our main database layer. The maintenance will take up to 5 hours starting from 14:00 UTC to 19:00 UTC. GitLab.com will be available but users may experience degraded performance during the maintenance window. We apologize in advance for any inconvenience this may cause. See #16266

August 31, 2023 00:01 UTC
[Update] Next week, we will be undergoing scheduled maintenance to our main database layer. The maintenance will take up to 5 hours starting from 14:00 UTC to 19:00 UTC. GitLab.com will be available but users may experience degraded performance during the maintenance window. We apologize in advance for any inconvenience this may cause. See <gitlab.com/gitlab-com/gl-infra/production/-/issues/14403>

August 2023

Main DB Cluster PostgreSQL 14 Upgrade

August 26, 2023 19:51 UTC

Description

We will be undergoing scheduled maintenance to our database layer. The maintenance will take up to 5 hours, from 14:00 UTC to 19:00 UTC. GitLab.com will continue to be available during the maintenance window. We apologize in advance for any inconvenience this may cause. See gitlab.com/gitlab-com/gl-infra/production/-/issues/14403 for details.


Components

Website, API, Git Operations, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners for GitLab community contributions


Locations

Google Compute Engine


Schedule

August 26, 2023 14:00 - August 26, 2023 19:00 UTC



August 26, 2023 19:51 UTC
[Update] GitLab.com rollback for the database layer is complete, and we're back up and running. We'll be monitoring the platform to ensure all systems are functioning correctly. Thank you for your patience.

August 26, 2023 19:21 UTC
[Update] GitLab.com rollback for the database layer is almost complete. The site is back up and running although we're continuing to verify that all systems are functioning correctly. Thank you for your patience.

August 26, 2023 18:29 UTC
[Update] Due to an issue during the planned maintenance for the database layer, we have initiated a rollback of the changes. We will send another update within the next 30 minutes.

August 26, 2023 14:11 UTC
[Update] We will be starting scheduled maintenance to our main database layer now. The maintenance will take up to 5 hours. GitLab.com will be available but users may experience degraded performance during the maintenance window. We apologize in advance for any inconvenience this may cause. See gitlab.com/gitlab-com/gl-infra/production/-/issues/14403

August 26, 2023 13:01 UTC
[Update] We will be undergoing scheduled maintenance to our main database layer in 1 hour. The maintenance will take up to 5 hours starting from 14:00 UTC to 19:00 UTC. GitLab.com will be available but users may experience degraded performance during the maintenance window. We apologize in advance for any inconvenience this may cause. See gitlab.com/gitlab-com/gl-infra/production/-/issues/14403

August 26, 2023 11:06 UTC
[Update] We will be undergoing scheduled maintenance to our main database layer in 3 hours. The maintenance will take up to 5 hours starting from 14:00 UTC to 19:00 UTC. GitLab.com will be available but users may experience degraded performance during the maintenance window. We apologize in advance for any inconvenience this may cause. See gitlab.com/gitlab-com/gl-infra/production/-/issues/14403

August 25, 2023 15:41 UTC
[Update] Reminder: Tomorrow, we will be undergoing scheduled maintenance to our main database layer. The maintenance will take up to 5 hours starting from 14:00 UTC to 19:00 UTC. GitLab.com will be available but users may experience degraded performance during the maintenance window. We apologize in advance for any inconvenience this may cause. See gitlab.com/gitlab-com/gl-infra/production/-/issues/14403

August 23, 2023 12:09 UTC
[Update] In 3 days, we will be undergoing scheduled maintenance to our main database layer. The maintenance will take up to 5 hours starting from 14:00 UTC to 19:00 UTC. GitLab.com will be available but users may experience degraded performance during the maintenance window. We apologize in advance for any inconvenience this may cause. See gitlab.com/gitlab-com/gl-infra/production/-/issues/14403

August 20, 2023 21:13 UTC
[Update] Next week, we will be undergoing scheduled maintenance to our main database layer. The maintenance will take up to 5 hours starting from 14:00 UTC to 19:00 UTC. GitLab.com will be available but users may experience degraded performance during the maintenance window. We apologize in advance for any inconvenience this may cause. See <gitlab.com/gitlab-com/gl-infra/production/-/issues/14403>





Back to current status