Active Issue

Updated a few seconds ago

Back to current status

Status History

Filter: GitLab Pages (Clear)



January 2025

Accessing gitlab.com results in 500 errors

January 6, 2025 16:42 UTC

Incident Status

Partial Service Disruption


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions, SAML SSO - GitLab SaaS, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, docs.gitlab.com, Canary, Product Analytics - Configurator (Beta), Product Analytics - Collector (Beta), Product Analytics - Background processing (Beta), Product Analytics - Queries (Beta), GitLab Duo


Locations

Google Compute Engine, Digital Ocean, Zendesk, AWS




January 6, 2025 16:42 UTC
[Resolved] We've identified the cause of the errors and the errors have returned to normal. The incident is now resolved. More information on the incident can be found in this issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/19079

January 6, 2025 15:57 UTC
[Monitoring] We've identified the cause of the errors, and the errors have returned to normal. The incident is now mitigated, but we're continuing to monitor for any further impact. More information on the incident can be found in this issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/19079

January 6, 2025 15:27 UTC
[Monitoring] We're continuing to monitor for further errors related to the incident. We see further decreases in the errors.

January 6, 2025 15:17 UTC
[Monitoring] We saw elevated 500 errors on gitlab.com which have then decreased. We're continuing to monitor for further errors

December 2024

Intermittent errors across GitLab.com

December 20, 2024 19:47 UTC

Intermittent errors across GitLab.comPartial Service Disruption

Incident Status

Partial Service Disruption


Components

Website, GitLab Pages, Background Processing


Locations

Google Compute Engine




December 20, 2024 19:47 UTC
[Resolved] GitLab.com has remained stable throughout our monitoring period and we are now marking this incident as resolved. Please see the following issue for further information related to this incident: gitlab.com/gitlab-com/gl-infra/production/-/issues/19033

December 20, 2024 00:08 UTC
[Monitoring] We are continuing to monitor the issue at this point, with further recovery being observed. Follow gitlab.com/gitlab-com/gl-infra/production/-/issues/19033 for more details.

December 19, 2024 23:32 UTC
[Monitoring] Mitigation steps have been completed. You should see delayed tasks start to recover now. Follow gitlab.com/gitlab-com/gl-infra/production/-/issues/19033 for more details.

December 19, 2024 23:09 UTC
[Identified] Mitigation steps are progressing and GitLab.com responsiveness is looking good. Follow gitlab.com/gitlab-com/gl-infra/production/-/issues/19033 for more details.

December 19, 2024 21:58 UTC
[Identified] Users may still see delays when interacting with GitLab.com. We are still actively working to mitigate the issue. Details will be posted in: gitlab.com/gitlab-com/gl-infra/production/-/issues/19033

December 19, 2024 21:11 UTC
[Identified] Our work to fully mitigate the underlying issue is still ongoing. Details will be posted in: gitlab.com/gitlab-com/gl-infra/production/-/issues/19033

December 19, 2024 20:31 UTC
[Identified] No material updates at this time. We have seen in overall improvement but are still are still working to fully mitigate the issue. Details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/19033

December 19, 2024 19:51 UTC
[Identified] Our initial changes have helped alleviate the issue. Our efforts are still ongoing as this is not fully mitigated. Details will be posted in: gitlab.com/gitlab-com/gl-infra/production/-/issues/19033

December 19, 2024 19:14 UTC
[Identified] We have implemented mitigating changes. Services should begin to recover. We are continuing to adjust and monitor. Details will be posted in: gitlab.com/gitlab-com/gl-infra/production/-/issues/19033

December 19, 2024 18:37 UTC
[Identified] No material updates at this time. We are still working to mitigate the issue. Details will be posted in: gitlab.com/gitlab-com/gl-infra/production/-/issues/19033

December 19, 2024 18:09 UTC
[Identified] We are still working to mitigate the issue. Users may still encounter errors or delays on GitLab.com. Details will be posted in: gitlab.com/gitlab-com/gl-infra/production/-/issues/19033

December 19, 2024 17:46 UTC
[Identified] We believe to have identified the cause of this incident. We are working on a mitigation strategy. Users may see errors or delays when interacting with GitLab.com. Details will be posted in: gitlab.com/gitlab-com/gl-infra/production/-/issues/19033

December 19, 2024 17:31 UTC
[Investigating] We are investigating reports of intermittent errors across GitLab.com and GitLab Pages. Details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/19033

November 2024

MAIN DB cluster upgrade

November 5, 2024 11:02 UTC

MAIN DB cluster upgrade Planned Maintenance

Description

Next week, we will be undergoing scheduled maintenance to our MAIN database layer. The maintenance will start at 2024-11-03 06:00 UTC and should finish at 2024-11-05 11:00 UTC (including performance regression observability period). GitLab.com will be available during the whole period as the maintenance should be seamless and transparent for the application. We apologize in advance for any inconvenience this may cause. See <gitlab.com/gitlab-com/gl-infra/production/-/issues/18747>


Components

Website, API, Git Operations, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions


Locations

Google Compute Engine, AWS


Schedule

November 3, 2024 06:00 - November 5, 2024 11:00 UTC



November 5, 2024 11:02 UTC
[Update] Gitlab.com MAIN database upgrade is now complete with all systems are functioning correctly. Thank you for your patience. See gitlab.com/gitlab-com/gl-infra/production/-/issues/18747

November 3, 2024 09:01 UTC
[Update] Gitlab.com MAIN database upgrade was performed. We'll continue to monitor for any performance issues until the end of the maintenance window. Thank you for your patience. See <gitlab.com/gitlab-com/gl-infra/production/-/issues/18747>

November 3, 2024 06:05 UTC
[Update] Gitlab.com scheduled maintenance of our MAIN database layer have started. GitLab.com should be available during the whole period. See <gitlab.com/gitlab-com/gl-infra/production/-/issues/18747>

November 3, 2024 05:00 UTC
[Update] MAIN DB layer maintenance will start in 1h from now at 2024-11-03 06:00 UTC. GitLab.com will remains available. Details: gitlab.com/gitlab-com/gl-infra/production/-/issues/18747

November 2, 2024 05:00 UTC
[Update] Scheduled maintenance on MAIN DB layer: Start: 2024-11-03 06:00 UTC End: 2024-11-05 11:00 UTC (incl. monitoring) GitLab.com remains available. Maintenance should be seamless. We apologize for any inconvenience. Details: gitlab.com/gitlab-com/gl-infra/production/-/issues/18747

October 30, 2024 06:00 UTC
[Update] In three days, we will be undergoing scheduled maintenance to our MAIN database layer. The maintenance will start at 2024-11-03 06:00 UTC and should finish at 2024-11-05 11:00 UTC (including performance and regression observability period). GitLab.com will be available during the whole period as the maintenance should be seamless and transparent for the application. We apologize in advance for any inconvenience this may cause. See: gitlab.com/gitlab-com/gl-infra/production/-/issues/18747

September 2024

GitLab connectivity issues

September 10, 2024 13:40 UTC

Incident Status

Operational


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions


Locations

Google Compute Engine




September 10, 2024 13:40 UTC
[Resolved] This incident has been resolved. More information can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/18535.

September 10, 2024 13:33 UTC
[Monitoring] The website has recovered are and we are now monitoring before marking as resolved. More details can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/18535.

September 10, 2024 13:24 UTC
[Investigating] No material updates at this time - our investigation is ongoing. More details can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/18535

September 10, 2024 13:10 UTC
[Investigating]

September 10, 2024 13:08 UTC
[Investigating] We are currently investigating connectivity issues to GitLab.com. For more information see gitlab.com/gitlab-com/gl-infra/production/-/issues/18535

GitLab.com Connectivity Issues

September 3, 2024 05:34 UTC

Incident Status

Service Disruption


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions, SAML SSO - GitLab SaaS, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, docs.gitlab.com, Canary, Product Analytics - Configurator (Beta), Product Analytics - Collector (Beta), Product Analytics - Background processing (Beta), Product Analytics - Queries (Beta), GitLab Duo


Locations

Google Compute Engine, Digital Ocean, Zendesk, AWS




September 3, 2024 05:34 UTC
[Resolved] All GitLab services have been restored. We are actively monitoring the system, and everything appears normal at this time. For more details, please follow the issue at: gitlab.com/gitlab-com/gl-infra/production/-/issues/18490

September 3, 2024 05:19 UTC
[Monitoring] We've identified and addressed the root cause of the issue. All GitLab services have been restored, and we are actively monitoring the system. Follow the issue for more details : gitlab.com/gitlab-com/gl-infra/production/-/issues/18490

September 3, 2024 05:09 UTC
[Investigating] GitLab.com is currently experiencing connectivity issues due to a high error rate. Our infrastructure team is investigating. Issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/18490

August 2024

Some requests timing out

August 23, 2024 03:59 UTC

Incident Status

Operational


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions, SAML SSO - GitLab SaaS, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, docs.gitlab.com, Canary, Product Analytics - Configurator (Beta), Product Analytics - Collector (Beta), Product Analytics - Background processing (Beta), Product Analytics - Queries (Beta), GitLab Duo


Locations

Google Compute Engine, Digital Ocean, Zendesk, AWS




August 23, 2024 03:59 UTC
[Resolved] The issue has been resolved and all functionality on GitLab.com is nominal. Please see gitlab.com/gitlab-com/gl-infra/production/-/issues/18435 for more details.

August 23, 2024 01:53 UTC
[Monitoring] The issue has been identified and the incident resolved. We will continue monitoring for a while.

August 23, 2024 01:27 UTC
[Identified] We have identified an underlying cause and are working to mitigate it. See gitlab.com/gitlab-com/gl-infra/production/-/issues/18435 for more details.

August 23, 2024 00:34 UTC
[Investigating] We have noticed that there are delays in web UI updates for example, MRs not showing up after submission, We are continuing to investigate - please see gitlab.com/gitlab-com/gl-infra/production/-/issues/18435 for more details.

August 23, 2024 00:27 UTC
[Investigating] Some requests are slow or timing out due to DB saturation. We are investigating. Please see [this issue](gitlab.com/gitlab-com/gl-infra/production/-/issues/18435) for further details.

July 2024

GitLab.com is unavailable

July 11, 2024 18:28 UTC

GitLab.com is unavailableService Disruption

Incident Status

Service Disruption


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions, SAML SSO - GitLab SaaS, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, docs.gitlab.com, Canary, Product Analytics - Configurator (Beta), Product Analytics - Collector (Beta), Product Analytics - Background processing (Beta), Product Analytics - Queries (Beta), GitLab Duo


Locations

Google Compute Engine, Digital Ocean, Zendesk, AWS




July 11, 2024 18:28 UTC
[Resolved] This incident is now resolved and no further errors are being seen on GitLab.com. Our investigation points to a disk performance issue. You can see further details and corrective actions in gitlab.com/gitlab-com/gl-infra/production/-/issues/18269.

July 11, 2024 15:51 UTC
[Investigating] Our investigations are ongoing. We will post our next update when we have more information. Please follow the incident issue for updates here: gitlab.com/gitlab-com/gl-infra/production/-/issues/18269

July 11, 2024 15:28 UTC
[Investigating] No material updates to report. We're continuing to investigate the issue. Please follow the incident issue for updates here: gitlab.com/gitlab-com/gl-infra/production/-/issues/18269

July 11, 2024 15:00 UTC
[Investigating] We are seeing improved service availability and investigations continue for this problem. Please follow the incident issue for updates here: gitlab.com/gitlab-com/gl-infra/production/-/issues/18269

July 11, 2024 14:43 UTC
[Investigating] We are seeing improved service availability and investigations continue for this problem. We are planning to make the incident issue publicly available as soon as possible.

July 11, 2024 14:29 UTC
[Investigating] We are investigating a problem with gitlab.com being unavailable for some users. Our investigations continue and we will post further status updates shortly.

July 11, 2024 14:18 UTC
[Investigating] We are investigating a problem with gitlab.com being unavailable for some users. We will post further status updates shortly.

June 2023

Slow requests on GitLab.com

June 7, 2023 13:32 UTC

Slow requests on GitLab.comPartial Service Disruption

Incident Status

Partial Service Disruption


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners for GitLab community contributions, SAML SSO - GitLab SaaS, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, docs.gitlab.com, Canary


Locations

Google Compute Engine, Digital Ocean, Zendesk, AWS




June 7, 2023 13:32 UTC
[Resolved] We have identified the root cause, have seen recovery and are therefore resolving this incident. More information in: gitlab.com/gitlab-com/gl-infra/production/-/issues/14839

June 7, 2023 13:21 UTC
[Investigating] Customers are experiencing slow requests on GitLab.com. More information in: gitlab.com/gitlab-com/gl-infra/production/-/issues/14839

May 2023

Hardware upgrade for the GitLab.com datastore

May 27, 2023 04:06 UTC

Description

In 2 weeks, as part of a planned maintenance window, we will perform a hardware upgrade for the GitLab.com datastore. Users may experience temporary 50X errors for a brief period during this window. See gitlab.com/gitlab-com/gl-infra/production/-/issues/10694


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners for GitLab community contributions, SAML SSO - GitLab SaaS, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, docs.gitlab.com, Canary


Locations

Google Compute Engine, Digital Ocean, Zendesk, AWS


Schedule

May 27, 2023 03:00 - May 27, 2023 04:00 UTC



May 27, 2023 04:06 UTC
[Update] GitLab.com's database layer maintenance is complete now, and we're fully back up and running. We'll be monitoring the platform to ensure all systems are functioning correctly. Thank you for your patience.

May 27, 2023 03:41 UTC
[Update] The database switchover is now complete, we expect all SQL statements to be routed to the new nodes. The site is back up and we're continuing to verify that all systems are functioning correctly. Thank you for your patience.

May 27, 2023 03:00 UTC
[Update] The database maintenance is starting now. The database Primary node switchover should happen at any moment in the next hour and during the switchover users might see 50X errors for a very brief span of time. For details see gitlab.com/gitlab-com/gl-infra/production/-/issues/10694

May 27, 2023 01:51 UTC
[Update] We will be conducting a maintenance database activity in an hour, from 03:00 to 04:00 UTC. Users might see 50X errors for a very brief span of time during this window. For details see gitlab.com/gitlab-com/gl-infra/production/-/issues/10694

May 26, 2023 02:27 UTC
[Update] We will be conducting a maintenance database activity Tomorrow, 2023-05-27, from 03:00 to 04:00 UTC. Users might see 50X errors for a very brief span of time during this window. For details see gitlab.com/gitlab-com/gl-infra/production/-/issues/10694

May 24, 2023 02:14 UTC
[Update] Correction: We will be conducting a maintenance database activity this Saturday, 2023-05-27. from 03:00 to 04:00 UTC. Users might see 50X errors for a very brief span of time during this window. For details see gitlab.com/gitlab-com/gl-infra/production/-/issues/10694

May 24, 2023 02:05 UTC
[Update] We will be conducting a maintenance database activity this Saturday, 2023-05-20, from 03:00 to 04:00 UTC. Users might see 50X errors for a very brief span of time during this window. For details see gitlab.com/gitlab-com/gl-infra/production/-/issues/10694

May 20, 2023 03:58 UTC
[Update] The maintenance is rescheduled to next Saturday, 2023-05-27, from 03:00 to 04:00 UTC. There is currently a long database migration running to fix an incident (gitlab.com/gitlab-com/gl-infra/production/-/issues/14468) that we can't risk to interrupt. We apologise for any inconvenience.

May 20, 2023 03:07 UTC
[Update] The scheduled maintenance will be not be starting at 03:00 UTC, and has been delayed. More details soon to follow.

May 20, 2023 02:00 UTC
[Update] We will be conducting a maintenance database activity in an hour, from 03:00 to 04:00 UTC. Users might see 50X errors for a very brief span of time during this window. For details see gitlab.com/gitlab-com/gl-infra/production/-/issues/10694

May 20, 2023 01:00 UTC
[Update] We will be conducting a maintenance database activity in 2 hours, from 03:00 to 04:00 UTC. Users might see 50X errors for a very brief span of time during this window. For details see gitlab.com/gitlab-com/gl-infra/production/-/issues/10694

May 19, 2023 02:00 UTC
[Update] We will be conducting a maintenance database activity tomorrow, 2023-05-20, from 03:00 to 04:00 UTC. Users might see 50X errors for a very brief span of time during this window. For details see gitlab.com/gitlab-com/gl-infra/production/-/issues/10694

May 17, 2023 02:00 UTC
[Update] We will be conducting a maintenance database activity this Saturday, 2023-05-20, from 03:00 to 04:00 UTC. Users might see 50X errors for a very brief span of time during this window. For details see gitlab.com/gitlab-com/gl-infra/production/-/issues/10694

May 12, 2023 02:00 UTC
[Update] Next week, as part of a planned maintenance window on 2023-05-20 from 03:00 to 04:00 UTC, We will perform a hardware upgrade for the GitLab.com datastore. Users may experience temporary 50X errors for a brief period during this window. See gitlab.com/gitlab-com/gl-infra/production/-/issues/10694

January 2023

404 error when accessing docs.gitlab.com

January 31, 2023 11:24 UTC

Incident Status

Partial Service Disruption


Components

GitLab Pages


Locations

Google Compute Engine




January 31, 2023 11:24 UTC
[Resolved] The website docs.gitlab.com has been monitored for the last 30 minutes and have returned to operating normally. More details are available in gitlab.com/gitlab-com/gl-infra/production/-/issues/8329

January 31, 2023 10:46 UTC
[Monitoring] The website docs.gitlab.com is accessible again. More details about this incident can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/8329

January 31, 2023 10:34 UTC
[Investigating] We are still investigating 404 error message on docs.gitlab.com More details about this incident can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/8329

January 31, 2023 10:13 UTC
[Investigating] We are currently investigating 404 error messages from docs.gitlab.com. More details about this incident can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/8329

High Error Rate on Select GitLab.com

January 5, 2023 11:24 UTC

Incident Status

Service Disruption


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners for GitLab community contributions, SAML SSO - GitLab SaaS, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, Canary


Locations

Google Compute Engine, Digital Ocean, Zendesk, AWS




January 5, 2023 11:24 UTC
[Resolved] Our Engineers identified the root cause and addressed it with a fix. The incident is now mitigated and users are expected to work with MRs without encountering any errors. For details, check gitlab.com/gitlab-com/gl-infra/production/-/issues/8201

January 5, 2023 09:18 UTC
[Monitoring] Services continue to recover and customers reported successful logins though SAML. However, some users are still facing errors with MRs. Please check the issue for the workaround: gitlab.com/gitlab-com/gl-infra/production/-/issues/8201#note_1228554821

January 5, 2023 08:45 UTC
[Monitoring] The error rates are decreasing, but users might still experience errors when authenticating through SAML and working with MRs. Investigation is still ongoing. For details, check gitlab.com/gitlab-com/gl-infra/production/-/issues/8201

January 5, 2023 08:24 UTC
[Monitoring] We are seeing the error rate decreasing and the services recovering. Our Engineers will keep monitoring the systems. For more details, please check gitlab.com/gitlab-com/gl-infra/production/-/issues/8201

January 5, 2023 08:07 UTC
[Investigating] We are currently seeing a high rate of errors on GitLab.com. Our infrastructure team is currently investigating. More information can be found here: gitlab.com/gitlab-com/gl-infra/production/-/issues/8201

November 2022

GitLab.com is unavailable

November 30, 2022 02:04 UTC

GitLab.com is unavailableDegraded Performance

Incident Status

Degraded Performance


Components

API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners for GitLab community contributions, SAML SSO - GitLab SaaS, Background Processing, Canary


Locations

Google Compute Engine




November 30, 2022 02:04 UTC
[Resolved] GitLab.com fully operational. Root cause: An unexpected deployment behaviour affected the Consul cluster that several of our services depend on (eg. Patroni)

November 30, 2022 01:27 UTC
[Monitoring] Monitoring situation, there was a deployment error for GitLab.com: gitlab.com/gitlab-com/gl-infra/production/-/issues/8097

November 30, 2022 01:10 UTC
[Monitoring] GitLab.com is back up, all systems operational.

November 30, 2022 00:54 UTC
[Investigating] We are currently investigating a GitLab.com service disruption

November 30, 2022 00:48 UTC
[Investigating] We have detected a problem related to GitLab.com services and are actively investigating. More details to follow.

September 2022

GitLab.com maintenance 2022-09-03 11:00 to 14:00 UTC

September 3, 2022 14:16 UTC

Description

We will be performing maintenance on GitLab.com's production database which is anticipated to include a service downtime of up to 180 minutes between 11:00am to 2:00pm UTC on Saturday 2022-09-03. During this time, users will experience complete service disruption. Improving the performance and reliability of GitLab.com has always been a top priority for GitLab. While we continuously make iterative improvements to GitLab and our production architecture, we anticipate making a larger change to improve the scalability and reliability of GitLab.com: performing necessary Operating System maintenance to our database clusters. You can read more about this change in our blog: about.gitlab.com/blog/2022/08/12/upgrading-database-os


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners for GitLab community contributions, SAML SSO - GitLab SaaS, Background Processing, GitLab Customers Portal, Canary


Locations

Google Compute Engine


Schedule

September 3, 2022 11:00 - September 3, 2022 14:00 UTC



September 3, 2022 14:16 UTC
[Update] GitLab.com's database layer maintenance upgrade is complete now, and we're back up and running. We'll be monitoring the platform to ensure all systems are functioning correctly. Thank you for your patience.

September 3, 2022 14:13 UTC
[Update] GitLab.com planned maintenance for the database layer is almost complete. The site is back up and running although we're continuing to verify that all systems are functioning correctly. Thank you for your patience.

September 3, 2022 13:55 UTC
[Update] We are still performing tests to ensure full functionality of the gitlab.com site before bringing it up. Expect full functionality of gitlab.com in about 30mins

September 3, 2022 10:59 UTC
[Update] GitLab.com is now shutting down for planned maintenance to our database layer for the next 3 hours. See you on the other side! More details in about.gitlab.com/blog/2022/08/12/upgrading-database-os

August 31, 2022 16:00 UTC
[Update] Reminder: We will be performing maintenance on GitLab.com's production database which is anticipated to include a complete service downtime of up to 180 minutes between 11:00am to 2:00pm UTC on Saturday 2022-09-03. You can read more in about.gitlab.com/blog/2022/08/12/upgrading-database-os

August 2022

GitLab.com slow to respond

August 26, 2022 06:21 UTC

GitLab.com slow to respondDegraded Performance

Incident Status

Degraded Performance


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners for GitLab community contributions, SAML SSO - GitLab SaaS, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, Canary


Locations

Google Compute Engine, Digital Ocean, Zendesk, AWS




August 26, 2022 06:21 UTC
[Resolved] We are no longer seeing connectivity issues for GitLab.com after a period of monitoring. We therefore believe that this incident has been resolved. Details in : gitlab.com/gitlab-com/gl-infra/production/-/issues/7656

August 26, 2022 05:08 UTC
[Monitoring] All the services seems to have recovered. We will continue to monitor the performance and provide the updates here. More information on the root cause can be found here: gitlab.com/gitlab-com/gl-infra/production/-/issues/7656

August 26, 2022 04:57 UTC
[Investigating] We are still investigating the root cause of this issue. More details here: gitlab.com/gitlab-com/gl-infra/production/-/issues/7656

August 26, 2022 04:55 UTC
[Investigating] We are noticing some slow response issues with GitLab.com.

Unexpected locked projects warning

August 23, 2022 21:39 UTC

Incident Status

Service Disruption


Components

Website, API, Git Operations, Container Registry, GitLab Pages


Locations

Google Compute Engine




August 23, 2022 21:39 UTC
[Resolved] This incident is now considered fully resolved and will be closed. Thanks for your patience! Please see the production issue for full details: gitlab.com/gitlab-com/gl-infra/production/-/issues/7638

August 23, 2022 21:06 UTC
[Monitoring] This incident has been resolved. We'll be monitoring for 30 minutes prior to fully closing the incident. See the production issue for details: gitlab.com/gitlab-com/gl-infra/production/-/issues/7638

August 23, 2022 19:05 UTC
[Monitoring] GitLab.com has been operational since we've mitigated the issue. Users should not be impacted at this time. We're currently working towards a full resolution. Details in gitlab.com/gitlab-com/gl-infra/production/-/issues/7638

August 23, 2022 18:39 UTC
[Identified] We'll be providing less updates until there is a material update to report on. Our team is still working towards a resolution. More details in gitlab.com/gitlab-com/gl-infra/production/-/issues/7638

August 23, 2022 17:53 UTC
[Identified] No material updates. Our team is still working on a full resolution to this issue. Details in gitlab.com/gitlab-com/gl-infra/production/-/issues/7638

August 23, 2022 17:20 UTC
[Identified] We're still working towards a full resolution on this issue. Users should not see any impact at this time. gitlab.com/gitlab-com/gl-infra/production/-/issues/7638

August 23, 2022 16:43 UTC
[Identified] We've identified the root cause and we're working towards a full resolution for this issue. Details in the production issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/7638

August 23, 2022 16:15 UTC
[Investigating] While projects should be unblocked, our team is still investigating the root cause. We’ll be moving to a longer cadence for updates during our investigation. More details in gitlab.com/gitlab-com/gl-infra/production/-/issues/7638

August 23, 2022 15:49 UTC
[Investigating] Our team has implemented a temporary fix to this issue. Projects that were previously locked should now be unlocked. Our team is continuing to investigate the root cause. More details are available in our production issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/7638

August 23, 2022 15:28 UTC
[Investigating] Our investigation is still ongoing. More details are available in our production issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/7638

August 23, 2022 15:12 UTC
[Investigating] Our team is continuing to investigate this issue. More details are available in our production issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/7638

August 23, 2022 15:00 UTC
[Investigating] We are still investigating the root cause of this issue. More details here: gitlab.com/gitlab-com/gl-infra/production/-/issues/7638

August 23, 2022 14:35 UTC
[Investigating] We are currently investigating an issue with projects displaying a "locked project" warning. More details here: gitlab.com/gitlab-com/gl-infra/production/-/issues/7638

July 2022

A subset GitLab.com Pages sites not loading properly

July 15, 2022 00:13 UTC

Incident Status

Partial Service Disruption


Components

GitLab Pages


Locations

Google Compute Engine




July 15, 2022 00:13 UTC
[Resolved] Fix has been applied to Production and the issue has been resolved. A full timeline of this incident and more details available at gitlab.com/gitlab-com/gl-infra/production/-/issues/7438.

July 14, 2022 23:46 UTC
[Monitoring] Fix has been deployed to Production. We continue to monitor impact on GitLab Pages. See gitlab.com/gitlab-com/gl-infra/production/-/issues/7438 for more details.

July 14, 2022 22:45 UTC
[Monitoring] Our mitigation measures are back in place, GitLab Pages sites should be loading correctly once again. We continue to monitor until the fix to the underlying issue is deployed into production. See gitlab.com/gitlab-com/gl-infra/production/-/issues/7438 for more details.

July 14, 2022 22:26 UTC
[Identified] As part of testing the fix to this issue, the previous mitigation measure was removed and the problem is present again. We work on mitigating it back ASAP. ETA for definitive fix in production is ~1 hour. See gitlab.com/gitlab-com/gl-infra/production/-/issues/7438.

July 14, 2022 18:38 UTC
[Monitoring] This incident is currently mitigated. The underlying problem will be fixed by an MR pending to be deployed into production within the next hours. Next update in 5 hours or sooner. See gitlab.com/gitlab-com/gl-infra/production/-/issues for further details.

July 14, 2022 17:19 UTC
[Monitoring] GitLab Pages version has been rolled back on GitLab.com and we are noticing improvements on the served sites, we continue to monitor to ensure the issue is mitigated for all use-cases. See gitlab.com/gitlab-com/gl-infra/production/-/issues/7438 for more details.

July 14, 2022 17:01 UTC
[Investigating] The investigation is still ongoing. We have an MR in flight to roll GitLab Pages back to a version we believe may mitigate the issue. Please stay tuned on gitlab.com/gitlab-com/gl-infra/production/-/issues/7438.

July 14, 2022 16:41 UTC
[Investigating] Investigation is still ongoing. We have identified a potential cause and are currently considering reverting the suspect changes. See more details on gitlab.com/gitlab-com/gl-infra/production/-/issues/7438.

July 14, 2022 16:17 UTC
[Investigating] We continue to investigate the cause of the Gitlab Pages sites errors analyzing recent merge requests and also investigating with GCP. See gitlab.com/gitlab-com/gl-infra/production/-/issues/7438 for further details.

July 14, 2022 15:58 UTC
[Investigating] Initial status update emphasized docs.gitlab.com as the affected site, this update is to clarify that there are more Pages-hosted sites affected by this, continue to track the incident at gitlab.com/gitlab-com/gl-infra/production/-/issues/7438

July 14, 2022 15:52 UTC
[Investigating] We continue to investigate the rendering errors affecting sites hosted on GitLab Pages. For more information see gitlab.com/gitlab-com/gl-infra/production/-/issues/7438 for incident tracking.

July 14, 2022 15:25 UTC
[Investigating] docs.gitlab.com (and possibly other pages sites) are showing partial or non-renderable pages, see gitlab.com/gitlab-com/gl-infra/production/-/issues/7438 for further details.

Scheduled GitLab.com production database maintenance

July 2, 2022 08:51 UTC

Description

We will be performing maintenance on GitLab.com's production database which is anticipated to include a service downtime of up to 120 minutes between Saturday, 2022-07-02, 06:00am to 08:00am UTC. During this time users will experience complete service disruption. Improving the performance and reliability of GitLab.com has always been a top priority for GitLab. While we continuously make iterative improvements to GitLab and our production architecture, we anticipate making a larger change to improve the scalability and reliability of GitLab.com: We are splitting our single PostgreSQL database into a main and a ci database. You can read more about this change in our blog: about.gitlab.com/blog/2022/06/02/splitting-database-into-main-and-ci


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners for GitLab community contributions, SAML SSO - GitLab SaaS, Background Processing, GitLab Customers Portal


Locations

Google Compute Engine


Schedule

July 2, 2022 05:00 - July 2, 2022 09:00 UTC



July 2, 2022 08:51 UTC
[Update] GitLab.com's database layer maintenance upgrade is complete now, and we're back up and running. We'll be monitoring the platform to ensure all systems are functioning correctly. Thank you for your patience.

July 2, 2022 07:43 UTC
[Update] GitLab.com planned maintenance for the database layer is almost complete. The site is back up and running although we're continuing to verify that all systems are functioning correctly. Thank you for your patience.

July 2, 2022 06:00 UTC
[Update] GitLab.com is now shutting down for planned maintenance for database layer improvements for the next 2 hours. See you on the other side!

July 2, 2022 05:00 UTC
[Update] GitLab.com will undergo maintenance in 1 hour. Please note that any CI jobs that start before the maintenance window but complete during the window period (up to 2 hours) will fail and may need to be started again.

July 1, 2022 05:45 UTC
[Update] Reminder: Tomorrow we will be undergoing scheduled maintenance to our database layer. We expect the GitLab.com site to be unavailable for up to 2 hours. Starting time: 2022-07-02 06:00 UTC.

June 29, 2022 10:01 UTC
[Update] We wanted to clarify that this scheduled maintenance will also impact Pages hosted with GitLab Pages on GitLab.com. Pages will be unavailable for up to 2 hours starting from 2022-07-02 06:00 UTC. We apologize in advance for any inconvenience this may cause.

June 29, 2022 05:50 UTC
[Update] In 3 days time, we will be undergoing some scheduled maintenance to our database layer so we expect the GitLab.com site to be unavailable for up to 2 hours starting from 2022-07-02 06:00 UTC. We apologize in advance for any inconvenience this may cause.

June 2022

Spike in error rate on gitlab.com

June 23, 2022 07:19 UTC

Incident Status

Operational


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners for GitLab community contributions, SAML SSO - GitLab SaaS, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, Canary


Locations

Google Compute Engine, Digital Ocean, Zendesk, AWS




June 23, 2022 07:19 UTC
[Resolved] Error rates have returned to normal levels. Follow-up discussion and actions are being discussed here: gitlab.com/gitlab-com/gl-infra/production/-/issues/7310

June 23, 2022 06:46 UTC
[Monitoring] We are investigating a brief spike in errors that lasted about 20 minutes, we are monitoring the recovery and will put more detail in gitlab.com/gitlab-com/gl-infra/production/-/issues/7310

Connectivity issues for GitLab.com

June 21, 2022 08:43 UTC

Incident Status

Service Disruption


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners for GitLab community contributions, SAML SSO - GitLab SaaS, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, Canary


Locations

Google Compute Engine, Digital Ocean, Zendesk, AWS




June 21, 2022 08:43 UTC
[Resolved] We are no longer seeing connectivity issues for GitLab.com after a period of monitoring. We therefore believe that this incident has been resolved. Details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/7288

June 21, 2022 08:00 UTC
[Monitoring] Services seem to be back to normal, and we continue monitoring. Details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/7288

June 21, 2022 07:45 UTC
[Identified] We are seeing services slowly recovering and continue monitoring issues with our CDN provider.

June 21, 2022 07:22 UTC
[Identified] No material updates at this point. We are continuing to monitor issues with our CDN provider

June 21, 2022 07:02 UTC
[Identified] We are continuing to monitor connectivity issues from some regions due to connectivity problems from our CDN provider.

June 21, 2022 06:52 UTC
[Investigating] We are investigating connectivity issues for GitLab.com, this is affecting all services.

Gitlab.com is having slowness issues

June 15, 2022 23:28 UTC

Incident Status

Degraded Performance


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners for GitLab community contributions, SAML SSO - GitLab SaaS, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, Canary


Locations

Google Compute Engine, Digital Ocean, Zendesk, AWS




June 15, 2022 23:28 UTC
[Resolved] We are no longer seeing performance issues on GitLab.com after a period of monitoring. We believe the incident has now been resolved. More details can be found here: gitlab.com/gitlab-com/gl-infra/production/-/issues/7258

June 15, 2022 23:12 UTC
[Identified] Our teams had identified the root cause of the slowness and are discussing preventive actions for it. Performance is back to normal.

June 15, 2022 22:53 UTC
[Investigating] Our teams detected a performance degradation for all components for GitLab.com, it appears this degradation is now stabilizing back to normal performance. We are still investigating.

June 15, 2022 22:36 UTC
[Investigating] Requests to GitLab.com are seeing elevated error rates and slowness, we are investigating.





Back to current status