Active Issue

Updated a few seconds ago

Back to current status

Status History

Filter: Digital Ocean (Clear)



January 2025

Accessing gitlab.com results in 500 errors

January 6, 2025 16:42 UTC

Incident Status

Partial Service Disruption


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions, SAML SSO - GitLab SaaS, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, docs.gitlab.com, Canary, Product Analytics - Configurator (Beta), Product Analytics - Collector (Beta), Product Analytics - Background processing (Beta), Product Analytics - Queries (Beta), GitLab Duo


Locations

Google Compute Engine, Digital Ocean, Zendesk, AWS




January 6, 2025 16:42 UTC
[Resolved] We've identified the cause of the errors and the errors have returned to normal. The incident is now resolved. More information on the incident can be found in this issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/19079

January 6, 2025 15:57 UTC
[Monitoring] We've identified the cause of the errors, and the errors have returned to normal. The incident is now mitigated, but we're continuing to monitor for any further impact. More information on the incident can be found in this issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/19079

January 6, 2025 15:27 UTC
[Monitoring] We're continuing to monitor for further errors related to the incident. We see further decreases in the errors.

January 6, 2025 15:17 UTC
[Monitoring] We saw elevated 500 errors on gitlab.com which have then decreased. We're continuing to monitor for further errors

September 2024

GitLab.com Connectivity Issues

September 3, 2024 05:34 UTC

Incident Status

Service Disruption


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions, SAML SSO - GitLab SaaS, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, docs.gitlab.com, Canary, Product Analytics - Configurator (Beta), Product Analytics - Collector (Beta), Product Analytics - Background processing (Beta), Product Analytics - Queries (Beta), GitLab Duo


Locations

Google Compute Engine, Digital Ocean, Zendesk, AWS




September 3, 2024 05:34 UTC
[Resolved] All GitLab services have been restored. We are actively monitoring the system, and everything appears normal at this time. For more details, please follow the issue at: gitlab.com/gitlab-com/gl-infra/production/-/issues/18490

September 3, 2024 05:19 UTC
[Monitoring] We've identified and addressed the root cause of the issue. All GitLab services have been restored, and we are actively monitoring the system. Follow the issue for more details : gitlab.com/gitlab-com/gl-infra/production/-/issues/18490

September 3, 2024 05:09 UTC
[Investigating] GitLab.com is currently experiencing connectivity issues due to a high error rate. Our infrastructure team is investigating. Issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/18490

August 2024

Some requests timing out

August 23, 2024 03:59 UTC

Incident Status

Operational


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions, SAML SSO - GitLab SaaS, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, docs.gitlab.com, Canary, Product Analytics - Configurator (Beta), Product Analytics - Collector (Beta), Product Analytics - Background processing (Beta), Product Analytics - Queries (Beta), GitLab Duo


Locations

Google Compute Engine, Digital Ocean, Zendesk, AWS




August 23, 2024 03:59 UTC
[Resolved] The issue has been resolved and all functionality on GitLab.com is nominal. Please see gitlab.com/gitlab-com/gl-infra/production/-/issues/18435 for more details.

August 23, 2024 01:53 UTC
[Monitoring] The issue has been identified and the incident resolved. We will continue monitoring for a while.

August 23, 2024 01:27 UTC
[Identified] We have identified an underlying cause and are working to mitigate it. See gitlab.com/gitlab-com/gl-infra/production/-/issues/18435 for more details.

August 23, 2024 00:34 UTC
[Investigating] We have noticed that there are delays in web UI updates for example, MRs not showing up after submission, We are continuing to investigate - please see gitlab.com/gitlab-com/gl-infra/production/-/issues/18435 for more details.

August 23, 2024 00:27 UTC
[Investigating] Some requests are slow or timing out due to DB saturation. We are investigating. Please see [this issue](gitlab.com/gitlab-com/gl-infra/production/-/issues/18435) for further details.

July 2024

GitLab.com is unavailable

July 11, 2024 18:28 UTC

GitLab.com is unavailableService Disruption

Incident Status

Service Disruption


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions, SAML SSO - GitLab SaaS, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, docs.gitlab.com, Canary, Product Analytics - Configurator (Beta), Product Analytics - Collector (Beta), Product Analytics - Background processing (Beta), Product Analytics - Queries (Beta), GitLab Duo


Locations

Google Compute Engine, Digital Ocean, Zendesk, AWS




July 11, 2024 18:28 UTC
[Resolved] This incident is now resolved and no further errors are being seen on GitLab.com. Our investigation points to a disk performance issue. You can see further details and corrective actions in gitlab.com/gitlab-com/gl-infra/production/-/issues/18269.

July 11, 2024 15:51 UTC
[Investigating] Our investigations are ongoing. We will post our next update when we have more information. Please follow the incident issue for updates here: gitlab.com/gitlab-com/gl-infra/production/-/issues/18269

July 11, 2024 15:28 UTC
[Investigating] No material updates to report. We're continuing to investigate the issue. Please follow the incident issue for updates here: gitlab.com/gitlab-com/gl-infra/production/-/issues/18269

July 11, 2024 15:00 UTC
[Investigating] We are seeing improved service availability and investigations continue for this problem. Please follow the incident issue for updates here: gitlab.com/gitlab-com/gl-infra/production/-/issues/18269

July 11, 2024 14:43 UTC
[Investigating] We are seeing improved service availability and investigations continue for this problem. We are planning to make the incident issue publicly available as soon as possible.

July 11, 2024 14:29 UTC
[Investigating] We are investigating a problem with gitlab.com being unavailable for some users. Our investigations continue and we will post further status updates shortly.

July 11, 2024 14:18 UTC
[Investigating] We are investigating a problem with gitlab.com being unavailable for some users. We will post further status updates shortly.

June 2023

Slow requests on GitLab.com

June 7, 2023 13:32 UTC

Slow requests on GitLab.comPartial Service Disruption

Incident Status

Partial Service Disruption


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners for GitLab community contributions, SAML SSO - GitLab SaaS, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, docs.gitlab.com, Canary


Locations

Google Compute Engine, Digital Ocean, Zendesk, AWS




June 7, 2023 13:32 UTC
[Resolved] We have identified the root cause, have seen recovery and are therefore resolving this incident. More information in: gitlab.com/gitlab-com/gl-infra/production/-/issues/14839

June 7, 2023 13:21 UTC
[Investigating] Customers are experiencing slow requests on GitLab.com. More information in: gitlab.com/gitlab-com/gl-infra/production/-/issues/14839

May 2023

Hardware upgrade for the GitLab.com datastore

May 27, 2023 04:06 UTC

Description

In 2 weeks, as part of a planned maintenance window, we will perform a hardware upgrade for the GitLab.com datastore. Users may experience temporary 50X errors for a brief period during this window. See gitlab.com/gitlab-com/gl-infra/production/-/issues/10694


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners for GitLab community contributions, SAML SSO - GitLab SaaS, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, docs.gitlab.com, Canary


Locations

Google Compute Engine, Digital Ocean, Zendesk, AWS


Schedule

May 27, 2023 03:00 - May 27, 2023 04:00 UTC



May 27, 2023 04:06 UTC
[Update] GitLab.com's database layer maintenance is complete now, and we're fully back up and running. We'll be monitoring the platform to ensure all systems are functioning correctly. Thank you for your patience.

May 27, 2023 03:41 UTC
[Update] The database switchover is now complete, we expect all SQL statements to be routed to the new nodes. The site is back up and we're continuing to verify that all systems are functioning correctly. Thank you for your patience.

May 27, 2023 03:00 UTC
[Update] The database maintenance is starting now. The database Primary node switchover should happen at any moment in the next hour and during the switchover users might see 50X errors for a very brief span of time. For details see gitlab.com/gitlab-com/gl-infra/production/-/issues/10694

May 27, 2023 01:51 UTC
[Update] We will be conducting a maintenance database activity in an hour, from 03:00 to 04:00 UTC. Users might see 50X errors for a very brief span of time during this window. For details see gitlab.com/gitlab-com/gl-infra/production/-/issues/10694

May 26, 2023 02:27 UTC
[Update] We will be conducting a maintenance database activity Tomorrow, 2023-05-27, from 03:00 to 04:00 UTC. Users might see 50X errors for a very brief span of time during this window. For details see gitlab.com/gitlab-com/gl-infra/production/-/issues/10694

May 24, 2023 02:14 UTC
[Update] Correction: We will be conducting a maintenance database activity this Saturday, 2023-05-27. from 03:00 to 04:00 UTC. Users might see 50X errors for a very brief span of time during this window. For details see gitlab.com/gitlab-com/gl-infra/production/-/issues/10694

May 24, 2023 02:05 UTC
[Update] We will be conducting a maintenance database activity this Saturday, 2023-05-20, from 03:00 to 04:00 UTC. Users might see 50X errors for a very brief span of time during this window. For details see gitlab.com/gitlab-com/gl-infra/production/-/issues/10694

May 20, 2023 03:58 UTC
[Update] The maintenance is rescheduled to next Saturday, 2023-05-27, from 03:00 to 04:00 UTC. There is currently a long database migration running to fix an incident (gitlab.com/gitlab-com/gl-infra/production/-/issues/14468) that we can't risk to interrupt. We apologise for any inconvenience.

May 20, 2023 03:07 UTC
[Update] The scheduled maintenance will be not be starting at 03:00 UTC, and has been delayed. More details soon to follow.

May 20, 2023 02:00 UTC
[Update] We will be conducting a maintenance database activity in an hour, from 03:00 to 04:00 UTC. Users might see 50X errors for a very brief span of time during this window. For details see gitlab.com/gitlab-com/gl-infra/production/-/issues/10694

May 20, 2023 01:00 UTC
[Update] We will be conducting a maintenance database activity in 2 hours, from 03:00 to 04:00 UTC. Users might see 50X errors for a very brief span of time during this window. For details see gitlab.com/gitlab-com/gl-infra/production/-/issues/10694

May 19, 2023 02:00 UTC
[Update] We will be conducting a maintenance database activity tomorrow, 2023-05-20, from 03:00 to 04:00 UTC. Users might see 50X errors for a very brief span of time during this window. For details see gitlab.com/gitlab-com/gl-infra/production/-/issues/10694

May 17, 2023 02:00 UTC
[Update] We will be conducting a maintenance database activity this Saturday, 2023-05-20, from 03:00 to 04:00 UTC. Users might see 50X errors for a very brief span of time during this window. For details see gitlab.com/gitlab-com/gl-infra/production/-/issues/10694

May 12, 2023 02:00 UTC
[Update] Next week, as part of a planned maintenance window on 2023-05-20 from 03:00 to 04:00 UTC, We will perform a hardware upgrade for the GitLab.com datastore. Users may experience temporary 50X errors for a brief period during this window. See gitlab.com/gitlab-com/gl-infra/production/-/issues/10694

January 2023

High Error Rate on Select GitLab.com

January 5, 2023 11:24 UTC

Incident Status

Service Disruption


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners for GitLab community contributions, SAML SSO - GitLab SaaS, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, Canary


Locations

Google Compute Engine, Digital Ocean, Zendesk, AWS




January 5, 2023 11:24 UTC
[Resolved] Our Engineers identified the root cause and addressed it with a fix. The incident is now mitigated and users are expected to work with MRs without encountering any errors. For details, check gitlab.com/gitlab-com/gl-infra/production/-/issues/8201

January 5, 2023 09:18 UTC
[Monitoring] Services continue to recover and customers reported successful logins though SAML. However, some users are still facing errors with MRs. Please check the issue for the workaround: gitlab.com/gitlab-com/gl-infra/production/-/issues/8201#note_1228554821

January 5, 2023 08:45 UTC
[Monitoring] The error rates are decreasing, but users might still experience errors when authenticating through SAML and working with MRs. Investigation is still ongoing. For details, check gitlab.com/gitlab-com/gl-infra/production/-/issues/8201

January 5, 2023 08:24 UTC
[Monitoring] We are seeing the error rate decreasing and the services recovering. Our Engineers will keep monitoring the systems. For more details, please check gitlab.com/gitlab-com/gl-infra/production/-/issues/8201

January 5, 2023 08:07 UTC
[Investigating] We are currently seeing a high rate of errors on GitLab.com. Our infrastructure team is currently investigating. More information can be found here: gitlab.com/gitlab-com/gl-infra/production/-/issues/8201

August 2022

GitLab.com slow to respond

August 26, 2022 06:21 UTC

GitLab.com slow to respondDegraded Performance

Incident Status

Degraded Performance


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners for GitLab community contributions, SAML SSO - GitLab SaaS, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, Canary


Locations

Google Compute Engine, Digital Ocean, Zendesk, AWS




August 26, 2022 06:21 UTC
[Resolved] We are no longer seeing connectivity issues for GitLab.com after a period of monitoring. We therefore believe that this incident has been resolved. Details in : gitlab.com/gitlab-com/gl-infra/production/-/issues/7656

August 26, 2022 05:08 UTC
[Monitoring] All the services seems to have recovered. We will continue to monitor the performance and provide the updates here. More information on the root cause can be found here: gitlab.com/gitlab-com/gl-infra/production/-/issues/7656

August 26, 2022 04:57 UTC
[Investigating] We are still investigating the root cause of this issue. More details here: gitlab.com/gitlab-com/gl-infra/production/-/issues/7656

August 26, 2022 04:55 UTC
[Investigating] We are noticing some slow response issues with GitLab.com.

June 2022

Spike in error rate on gitlab.com

June 23, 2022 07:19 UTC

Incident Status

Operational


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners for GitLab community contributions, SAML SSO - GitLab SaaS, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, Canary


Locations

Google Compute Engine, Digital Ocean, Zendesk, AWS




June 23, 2022 07:19 UTC
[Resolved] Error rates have returned to normal levels. Follow-up discussion and actions are being discussed here: gitlab.com/gitlab-com/gl-infra/production/-/issues/7310

June 23, 2022 06:46 UTC
[Monitoring] We are investigating a brief spike in errors that lasted about 20 minutes, we are monitoring the recovery and will put more detail in gitlab.com/gitlab-com/gl-infra/production/-/issues/7310

Connectivity issues for GitLab.com

June 21, 2022 08:43 UTC

Incident Status

Service Disruption


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners for GitLab community contributions, SAML SSO - GitLab SaaS, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, Canary


Locations

Google Compute Engine, Digital Ocean, Zendesk, AWS




June 21, 2022 08:43 UTC
[Resolved] We are no longer seeing connectivity issues for GitLab.com after a period of monitoring. We therefore believe that this incident has been resolved. Details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/7288

June 21, 2022 08:00 UTC
[Monitoring] Services seem to be back to normal, and we continue monitoring. Details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/7288

June 21, 2022 07:45 UTC
[Identified] We are seeing services slowly recovering and continue monitoring issues with our CDN provider.

June 21, 2022 07:22 UTC
[Identified] No material updates at this point. We are continuing to monitor issues with our CDN provider

June 21, 2022 07:02 UTC
[Identified] We are continuing to monitor connectivity issues from some regions due to connectivity problems from our CDN provider.

June 21, 2022 06:52 UTC
[Investigating] We are investigating connectivity issues for GitLab.com, this is affecting all services.

Gitlab.com is having slowness issues

June 15, 2022 23:28 UTC

Incident Status

Degraded Performance


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners for GitLab community contributions, SAML SSO - GitLab SaaS, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, Canary


Locations

Google Compute Engine, Digital Ocean, Zendesk, AWS




June 15, 2022 23:28 UTC
[Resolved] We are no longer seeing performance issues on GitLab.com after a period of monitoring. We believe the incident has now been resolved. More details can be found here: gitlab.com/gitlab-com/gl-infra/production/-/issues/7258

June 15, 2022 23:12 UTC
[Identified] Our teams had identified the root cause of the slowness and are discussing preventive actions for it. Performance is back to normal.

June 15, 2022 22:53 UTC
[Investigating] Our teams detected a performance degradation for all components for GitLab.com, it appears this degradation is now stabilizing back to normal performance. We are still investigating.

June 15, 2022 22:36 UTC
[Investigating] Requests to GitLab.com are seeing elevated error rates and slowness, we are investigating.

April 2022

Elevated Error Rate on GitLab.com

April 28, 2022 07:37 UTC

Incident Status

Service Disruption


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners for GitLab community contributions, SAML SSO - GitLab SaaS, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, Canary


Locations

Google Compute Engine, Digital Ocean, Zendesk, AWS




April 28, 2022 07:37 UTC
[Resolved] We transitioned this incident to resolved. Check the issue for more details: gitlab.com/gitlab-com/gl-infra/production/-/issues/6933

April 28, 2022 07:23 UTC
[Monitoring] We have identified the cause to this incident and do not anticipate any additional impacts at this time. We are continuing to monitor services as well as complete additional investigation. More details here: gitlab.com/gitlab-com/gl-infra/production/-/issues/6933

April 28, 2022 06:34 UTC
[Investigating] We are still investigating the root cause. The error rates on affected systems recovering, and we are monitoring to ensure that the issue does not recur.

April 28, 2022 06:12 UTC
[Investigating] We're investigating increased error rates across GitLab-com services, some users might be experiencing 500 errors intermittently, more info can be found in here: gitlab.com/gitlab-com/gl-infra/production/-/issues/6933

September 2021

Increased error rates across all GitLab services

September 12, 2021 21:48 UTC

Incident Status

Service Disruption


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners for GitLab community contributions, SAML SSO - GitLab SaaS, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, Canary


Locations

Google Compute Engine, Azure, Digital Ocean, Zendesk, AWS




September 12, 2021 21:48 UTC
[Resolved] After investigation we determined that the elevated error rates were caused by database connection saturation; further details will be in this issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/5514

September 12, 2021 21:23 UTC
[Monitoring] GitLab performance and error rates have returned to normal; we are investigating the root cause to understand the likelyhood it will reoccur.

September 12, 2021 21:04 UTC
[Investigating] We are investigating slow response times and increased error rates across all GitLab services. Incident: gitlab.com/gitlab-com/gl-infra/production/-/issues/5514

August 2021

Elevated error rates on GitLab.com

August 11, 2021 17:51 UTC

Elevated error rates on GitLab.comPartial Service Disruption

Incident Status

Partial Service Disruption


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners for GitLab community contributions, SAML SSO - GitLab SaaS, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, Canary


Locations

Google Compute Engine, Azure, Digital Ocean, Zendesk, AWS




August 11, 2021 17:51 UTC
[Resolved] As GitLab.com is operating normally, we'll now be marking this incident as resolved. Further investigation efforts and details surrounding the issue can be found in: gitlab.com/gitlab-com/gl-infra/production/-/issues/5333

August 11, 2021 17:44 UTC
[Investigating] No material updates to report - our investigation is still ongoing. Details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/5333

August 11, 2021 17:27 UTC
[Investigating] All GitLab.com services are operating normally, but our investigation into the cause of the errors continues. Details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/5333

August 11, 2021 17:18 UTC
[Investigating] At 17:04 UTC, there was a 5 minute elevated error ratio event for all GitLab.com services. It has recovered, but we are investigating. Details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/5333

Increased error rates across all services

August 10, 2021 10:39 UTC

Incident Status

Partial Service Disruption


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners for GitLab community contributions, SAML SSO - GitLab SaaS, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, Canary


Locations

Google Compute Engine, Azure, Digital Ocean, Zendesk, AWS




August 10, 2021 10:39 UTC
[Resolved] The issue has been resolved, and we can confirm that all services on GitLab.com are fully operational. A full timeline of this incident and more details are available in gitlab.com/gitlab-com/gl-infra/production/-/issues/5324

August 10, 2021 10:05 UTC
[Monitoring] We are back to normal error rates across all services. We keep monitoring the situation. Refer to gitlab.com/gitlab-com/gl-infra/production/-/issues/5324 for more updates.

August 10, 2021 09:52 UTC
[Investigating] We are investigating increased error rates across all services, see gitlab.com/gitlab-com/gl-infra/production/-/issues/5324 for more details

GitLab performance degraded

August 6, 2021 12:48 UTC

Incident Status

Operational


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners for GitLab community contributions, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, Canary


Locations

Google Compute Engine, Azure, Digital Ocean, Zendesk, AWS




August 6, 2021 12:48 UTC
[Resolved] From 12:05 - 12:15 UTC, we saw a system degradation across all of our components, which seems to have recovered at the moment. Our engineers are investigating. Further details are available in: gitlab.com/gitlab-com/gl-infra/production/-/issues/5306

July 2021

Issue Uploading Artifacts from CI

July 21, 2021 01:39 UTC

Incident Status

Service Disruption


Components

Website, API, Git Operations, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners for GitLab community contributions, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, Canary


Locations

Google Compute Engine, Azure, Digital Ocean, Zendesk, AWS




July 21, 2021 01:39 UTC
[Resolved] This incident is now resolved and we are no longer seeing any errors. We will continue to investigate the cause internally. Full timeline of this incident and more details available in gitlab.com/gitlab-com/gl-infra/production/-/issues/5194

July 21, 2021 01:12 UTC
[Monitoring] The rollout has completed and we are no longer seeing errors with the upload. This has been applied to all our servers. We are continuously monitoring this further for any issues.

July 21, 2021 01:04 UTC
[Identified] The rollout has completed on two of our clusters. This will continue on our last few clusters and we will post further updates once completed. More details at gitlab.com/gitlab-com/gl-infra/production/-/issues/5194

July 21, 2021 00:44 UTC
[Identified] We are currently working on a recent configuration rollback that affected uploading CI artifacts. More details at gitlab.com/gitlab-com/gl-infra/production/-/issues/5194

July 21, 2021 00:28 UTC
[Investigating] We have identified a failure in uploading artifacts from CI which is currently showing a 400 error. We are investigating the issue at gitlab.com/gitlab-com/gl-infra/production/-/issues/5194 and will post more updates.

June 2021

Short service disruption on GitLab.com

June 21, 2021 21:34 UTC

Incident Status

Degraded Performance


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners for GitLab community contributions, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, Canary


Locations

Google Compute Engine, Azure, Digital Ocean, Zendesk, AWS




June 21, 2021 21:34 UTC
[Resolved] Our team has confirmed that the disruption is now gone, and that services are back to normal, Investigation will continue on gitlab.com/gitlab-com/gl-infra/production/-/issues/4945 for additional details and root cause analysis.

June 21, 2021 21:14 UTC
[Investigating] GitLab.com is up and operational, but from 20:53 to 20:58, we saw an increase in errors on GitLab.com web, API, and Git services that were a short disruption, now gone. Investigation is being done on gitlab.com/gitlab-com/gl-infra/production/-/issues/4945

May 2021

Unscheduled maintenance - Switchover of our primary database

May 8, 2021 14:59 UTC

Description

We will be conducting a switchover of our primary database at 14:45 UTC to mitigate a potential instability. Users will be unable to access GitLab.com for up to 2 minutes during this switchover. Please check gitlab.com/gitlab-com/gl-infra/production/-/issues/4528


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, Canary


Locations

Google Compute Engine, Azure, Digital Ocean, Zendesk, AWS


Schedule

May 7, 2021 14:45 - May 7, 2021 14:50 UTC



May 8, 2021 14:59 UTC
[Update] The primary database switchover is done and only affected Gitlab.com for 4 seconds. All systems are up and operational. For more details, see gitlab.com/gitlab-com/gl-infra/production/-/issues/4528

May 8, 2021 14:53 UTC
[Update] The primary database switchover is complete. For more details, please see gitlab.com/gitlab-com/gl-infra/production/-/issues/4528

March 2021

Possible Database & Redis degradation

March 18, 2021 15:06 UTC

Incident Status

Degraded Performance


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, Canary


Locations

Google Compute Engine, Azure, Digital Ocean, Zendesk, AWS




March 18, 2021 15:06 UTC
[Resolved] Our services have fully recovered and the incident is now mitigated. For the full root cause analysis follow our investigation at gitlab.com/gitlab-com/gl-infra/production/-/issues/4011

March 18, 2021 14:57 UTC
[Investigating] No material updates; we continue to see recovery across our services. Our engineers are still investigating the root cause. More details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/4011

March 18, 2021 14:40 UTC
[Investigating] We're seeing a recovery across our different services and continuing to investigate to confirm the root cause of the degradation. Details in issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/4011

March 18, 2021 14:23 UTC
[Investigating] We're investigating a possible service degradation across our Redis and DB components. Users might be experiencing delays in CI/CD pipelines. Details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/4011





Back to current status