All Systems Operational

Updated a few seconds ago

Back to current status

Status History

Filter: AWS (Clear)



July 2024

Users unable to download NuGet packages

July 2, 2024 15:11 UTC

Incident Status

Degraded Performance


Components

packages.gitlab.com


Locations

AWS




July 2, 2024 15:11 UTC
[Resolved] This incident was resolved by rolling back a feature flag. We've got confirmation from users that packages are being downloaded succesfully again. More details here: gitlab.com/gitlab-com/gl-infra/production/-/issues/18228

July 2, 2024 14:45 UTC
[Investigating] We have received reports from users unable to download NuGet packages on GitLab.com. We're investigating the issue at the moment. More details here: gitlab.com/gitlab-com/gl-infra/production/-/issues/18228

March 2024

Elevated queue size on Shared Runners

March 15, 2024 18:40 UTC

Incident Status

Degraded Performance


Components

CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS


Locations

Google Compute Engine, AWS




March 15, 2024 18:40 UTC
[Resolved] Job pickup delays and slowness on running jobs have not been observed for some time, and the team does not anticipate the problem returning. Additional information will be shared in gitlab.com/gitlab-com/gl-infra/production/-/issues/17724

March 15, 2024 14:35 UTC
[Monitoring] As things remain stable, we will continue to monitor the situation and will provide further updates as necessary. Full details are available in gitlab.com/gitlab-com/gl-infra/production/-/issues/17724

March 15, 2024 13:34 UTC
[Monitoring] There are no material updates to report. We will continue to monitor and post updates hourly. Full details are available in gitlab.com/gitlab-com/gl-infra/production/-/issues/17724

March 15, 2024 12:33 UTC
[Monitoring] Things remain stable, with no material updates at the time. We are continuing to monitor the situation, and will provide hourly updates. Full details are available in gitlab.com/gitlab-com/gl-infra/production/-/issues/17724

March 15, 2024 11:30 UTC
[Monitoring] No material updates at the time. We are continuing to monitor the situation, and will provide hourly updates. Full details are available in gitlab.com/gitlab-com/gl-infra/production/-/issues/17724

March 15, 2024 10:29 UTC
[Monitoring] The impact from this incident appears to now be mitigated, however we will continue to monitor the situation and will provide further updates as necessary. Full details are available in gitlab.com/gitlab-com/gl-infra/production/-/issues/17724

March 15, 2024 09:26 UTC
[Monitoring] The impact from this incident appears low, however we are continuing to monitor the situation, and will post another update in one hour. Full details are available in gitlab.com/gitlab-com/gl-infra/production/-/issues/17724

March 15, 2024 08:15 UTC
[Monitoring] We are continuing to monitor this situation, with no additional details to add at this time. Full details are available in gitlab.com/gitlab-com/gl-infra/production/-/issues/17724

March 15, 2024 07:11 UTC
[Identified] No material updates to report. We will continue to monitor to the situation for now. Details in gitlab.com/gitlab-com/gl-infra/production/-/issues/17724

March 15, 2024 05:18 UTC
[Identified] We have narrowed down the potential root cause but we will continue the investigation. No further impact on CI/CD pipeline execution. Next update will be provided in 1 hour. Details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/17724

March 15, 2024 03:35 UTC
[Investigating] We are continuing to monitor the extended runner queue size. Impact on CI/CD pipeline execution does not appear to be significant at this time. A further update will be provided in 1 hour. Details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/17724

March 15, 2024 02:31 UTC
[Investigating] Analysis continues. A further update will be provided in 1 hour, or sooner should there be a significant update to report. Details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/17724

March 15, 2024 01:57 UTC
[Investigating] The investigation is ongoing, we continue to assess the root cause of the extended queue size with the help of additional resources. A further update will be provided in 30 minutes. Details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/17724

March 15, 2024 01:34 UTC
[Investigating] No material updates at this time. Investigation into the increasing runner queue size is ongoing. Details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/17724

March 15, 2024 01:13 UTC
[Investigating] We are continuing to investigate the root cause of the elevated queue size. For more information: gitlab.com/gitlab-com/gl-infra/production/-/issues/17724

March 15, 2024 00:55 UTC
[Investigating] No material update at this time. Our team is still investigating the root cause.

March 15, 2024 00:42 UTC
[Investigating] We are investigating an issue with elevated queue size on our GitLab.com runners. Users may experience slowness with runners picking up jobs, or running jobs. For more information: gitlab.com/gitlab-com/gl-infra/production/-/issues/17724

November 2023

Performance issues affecting processing of MRs

November 20, 2023 18:49 UTC

Incident Status

Degraded Performance


Components

CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions, Background Processing


Locations

Google Compute Engine, AWS




November 20, 2023 18:49 UTC
[Resolved] Performance of CI pipelines and merge requests has fully returned to normal and shows no further signs of degradation. The status page is being marked resolved, and future updates can be found in the production issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/17158

November 20, 2023 18:05 UTC
[Monitoring] Performance of CI job pickup and merge requests has returned to a normal state. The incident response team will continue monitoring and reviewing the rolled back changes. Next update in an hour, unless there is anything to report prior.

November 20, 2023 17:21 UTC
[Investigating] We have started to notice some signs of recovery after the rollback. Investigation into root causes is still ongoing to identify the source of the CI and merge request performance problems.

November 20, 2023 16:50 UTC
[Investigating] Rollback to an earlier deployment has completed. No material updates in the CI job pickup slowness has been observed yet. Next update will be in 30 minutes unless there is anything to report sooner.

November 20, 2023 16:32 UTC
[Investigating] The rollback to an earlier deployment is in progress and a review of all changes is in progress to identify the root cause of slow CI job pickup. Additional support from other teams is being brought in to assist.

November 20, 2023 16:12 UTC
[Investigating] The incident response team is continuing to investigate the slow CI job pickup. A rollback to a previous deployment is in progress as a potential mitigating solution.

November 20, 2023 15:53 UTC
[Investigating] We are actively investigating an issue where CI jobs are being picked up slowly affecting performance of MR's. More details here: gitlab.com/gitlab-com/gl-infra/production/-/issues/17158

August 2023

Degraded performance of packages.gitlab.com

August 31, 2023 19:11 UTC

Incident Status

Degraded Performance


Components

packages.gitlab.com


Locations

AWS




August 31, 2023 19:11 UTC
[Resolved] There has been no further sign of performance degradation on packages.gitlab.com since traffic returned to normal. Thanks for your patience! See gitlab.com/gitlab-com/gl-infra/production/-/issues/16281 for details.

August 31, 2023 18:34 UTC
[Monitoring] Performance on packages.gitlab.com is returning to normal. Our engineers have identified the cause and are monitoring. See gitlab.com/gitlab-com/gl-infra/production/-/issues/16281

August 31, 2023 18:22 UTC
[Identified] We're seeing degraded performance on packages.gitlab.com due to increased traffic. Some customers may be experiencing intermittent 5xx responses or timeouts. See gitlab.com/gitlab-com/gl-infra/production/-/issues/16281 for more details.

June 2023

Slow requests on GitLab.com

June 7, 2023 13:32 UTC

Slow requests on GitLab.comPartial Service Disruption

Incident Status

Partial Service Disruption


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners for GitLab community contributions, SAML SSO - GitLab SaaS, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, docs.gitlab.com, Canary


Locations

Google Compute Engine, Digital Ocean, Zendesk, AWS




June 7, 2023 13:32 UTC
[Resolved] We have identified the root cause, have seen recovery and are therefore resolving this incident. More information in: gitlab.com/gitlab-com/gl-infra/production/-/issues/14839

June 7, 2023 13:21 UTC
[Investigating] Customers are experiencing slow requests on GitLab.com. More information in: gitlab.com/gitlab-com/gl-infra/production/-/issues/14839

May 2023

Hardware upgrade for the GitLab.com datastore

May 27, 2023 04:06 UTC

Description

In 2 weeks, as part of a planned maintenance window, we will perform a hardware upgrade for the GitLab.com datastore. Users may experience temporary 50X errors for a brief period during this window. See gitlab.com/gitlab-com/gl-infra/production/-/issues/10694


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners for GitLab community contributions, SAML SSO - GitLab SaaS, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, docs.gitlab.com, Canary


Locations

Google Compute Engine, Digital Ocean, Zendesk, AWS


Schedule

May 27, 2023 03:00 - May 27, 2023 04:00 UTC



May 27, 2023 04:06 UTC
[Update] GitLab.com's database layer maintenance is complete now, and we're fully back up and running. We'll be monitoring the platform to ensure all systems are functioning correctly. Thank you for your patience.

May 27, 2023 03:41 UTC
[Update] The database switchover is now complete, we expect all SQL statements to be routed to the new nodes. The site is back up and we're continuing to verify that all systems are functioning correctly. Thank you for your patience.

May 27, 2023 03:00 UTC
[Update] The database maintenance is starting now. The database Primary node switchover should happen at any moment in the next hour and during the switchover users might see 50X errors for a very brief span of time. For details see gitlab.com/gitlab-com/gl-infra/production/-/issues/10694

May 27, 2023 01:51 UTC
[Update] We will be conducting a maintenance database activity in an hour, from 03:00 to 04:00 UTC. Users might see 50X errors for a very brief span of time during this window. For details see gitlab.com/gitlab-com/gl-infra/production/-/issues/10694

May 26, 2023 02:27 UTC
[Update] We will be conducting a maintenance database activity Tomorrow, 2023-05-27, from 03:00 to 04:00 UTC. Users might see 50X errors for a very brief span of time during this window. For details see gitlab.com/gitlab-com/gl-infra/production/-/issues/10694

May 24, 2023 02:14 UTC
[Update] Correction: We will be conducting a maintenance database activity this Saturday, 2023-05-27. from 03:00 to 04:00 UTC. Users might see 50X errors for a very brief span of time during this window. For details see gitlab.com/gitlab-com/gl-infra/production/-/issues/10694

May 24, 2023 02:05 UTC
[Update] We will be conducting a maintenance database activity this Saturday, 2023-05-20, from 03:00 to 04:00 UTC. Users might see 50X errors for a very brief span of time during this window. For details see gitlab.com/gitlab-com/gl-infra/production/-/issues/10694

May 20, 2023 03:58 UTC
[Update] The maintenance is rescheduled to next Saturday, 2023-05-27, from 03:00 to 04:00 UTC. There is currently a long database migration running to fix an incident (gitlab.com/gitlab-com/gl-infra/production/-/issues/14468) that we can't risk to interrupt. We apologise for any inconvenience.

May 20, 2023 03:07 UTC
[Update] The scheduled maintenance will be not be starting at 03:00 UTC, and has been delayed. More details soon to follow.

May 20, 2023 02:00 UTC
[Update] We will be conducting a maintenance database activity in an hour, from 03:00 to 04:00 UTC. Users might see 50X errors for a very brief span of time during this window. For details see gitlab.com/gitlab-com/gl-infra/production/-/issues/10694

May 20, 2023 01:00 UTC
[Update] We will be conducting a maintenance database activity in 2 hours, from 03:00 to 04:00 UTC. Users might see 50X errors for a very brief span of time during this window. For details see gitlab.com/gitlab-com/gl-infra/production/-/issues/10694

May 19, 2023 02:00 UTC
[Update] We will be conducting a maintenance database activity tomorrow, 2023-05-20, from 03:00 to 04:00 UTC. Users might see 50X errors for a very brief span of time during this window. For details see gitlab.com/gitlab-com/gl-infra/production/-/issues/10694

May 17, 2023 02:00 UTC
[Update] We will be conducting a maintenance database activity this Saturday, 2023-05-20, from 03:00 to 04:00 UTC. Users might see 50X errors for a very brief span of time during this window. For details see gitlab.com/gitlab-com/gl-infra/production/-/issues/10694

May 12, 2023 02:00 UTC
[Update] Next week, as part of a planned maintenance window on 2023-05-20 from 03:00 to 04:00 UTC, We will perform a hardware upgrade for the GitLab.com datastore. Users may experience temporary 50X errors for a brief period during this window. See gitlab.com/gitlab-com/gl-infra/production/-/issues/10694

March 2023

packages.gitlab.com is unavailable

March 31, 2023 08:56 UTC

Incident Status

Service Disruption


Components

packages.gitlab.com


Locations

AWS




March 31, 2023 08:56 UTC
[Resolved] We are no longer seeing errors from packages.gitlab.com. We will set this as resolved now. We are continuing investigations for the root cause and will post further updates in gitlab.com/gitlab-com/gl-infra/production/-/issues/8635

March 31, 2023 07:51 UTC
[Monitoring] We are no longer seeing errors from packages.gitlab.com. We are still investigating the cause of the issue. We will post further updates as more information is available in the next couple of hours. Details: gitlab.com/gitlab-com/gl-infra/production/-/issues/8635

March 31, 2023 06:16 UTC
[Monitoring] Access to packages.gitlab.com is restored. While the service has been restored, our team is continuing to investigate the root cause of the issue. We will move this to monitoring state for now. Details: gitlab.com/gitlab-com/gl-infra/production/-/issues/8635

March 31, 2023 05:48 UTC
[Investigating] We do not have any material updates to report. However, we want to assure you that our team is continuing to investigate the issue. We will send another update in the next hour. Details: gitlab.com/gitlab-com/gl-infra/production/-/issues/8635

March 31, 2023 05:27 UTC
[Investigating] We are continuing to investigate the cause of the problem on packages.gitlab.com and will post further updates as more information is available. Details: gitlab.com/gitlab-com/gl-infra/production/-/issues/8635

March 31, 2023 05:03 UTC
[Investigating] We are currently experiencing an issue on packages.gitlab.com. This service is currently not available, and our team is currently looking into this. Kindly refer to this link for more details: gitlab.com/gitlab-com/gl-infra/production/-/issues/8635

Reliability improvements on packages.gitlab.com

March 18, 2023 13:00 UTC

Description

We are making a series of improvements to the packages.gitlab.com infrastructure starting with the database requiring service downtime.


Components

packages.gitlab.com


Locations

AWS


Schedule

March 18, 2023 11:30 - March 18, 2023 13:00 UTC



March 18, 2023 13:00 UTC
[Update] Scheduled maintenance is complete.

March 18, 2023 11:30 UTC
[Update] Scheduled maintenance is starting.

January 2023

High Error Rate on Select GitLab.com

January 5, 2023 11:24 UTC

Incident Status

Service Disruption


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners for GitLab community contributions, SAML SSO - GitLab SaaS, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, Canary


Locations

Google Compute Engine, Digital Ocean, Zendesk, AWS




January 5, 2023 11:24 UTC
[Resolved] Our Engineers identified the root cause and addressed it with a fix. The incident is now mitigated and users are expected to work with MRs without encountering any errors. For details, check gitlab.com/gitlab-com/gl-infra/production/-/issues/8201

January 5, 2023 09:18 UTC
[Monitoring] Services continue to recover and customers reported successful logins though SAML. However, some users are still facing errors with MRs. Please check the issue for the workaround: gitlab.com/gitlab-com/gl-infra/production/-/issues/8201#note_1228554821

January 5, 2023 08:45 UTC
[Monitoring] The error rates are decreasing, but users might still experience errors when authenticating through SAML and working with MRs. Investigation is still ongoing. For details, check gitlab.com/gitlab-com/gl-infra/production/-/issues/8201

January 5, 2023 08:24 UTC
[Monitoring] We are seeing the error rate decreasing and the services recovering. Our Engineers will keep monitoring the systems. For more details, please check gitlab.com/gitlab-com/gl-infra/production/-/issues/8201

January 5, 2023 08:07 UTC
[Investigating] We are currently seeing a high rate of errors on GitLab.com. Our infrastructure team is currently investigating. More information can be found here: gitlab.com/gitlab-com/gl-infra/production/-/issues/8201

August 2022

GitLab.com slow to respond

August 26, 2022 06:21 UTC

GitLab.com slow to respondDegraded Performance

Incident Status

Degraded Performance


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners for GitLab community contributions, SAML SSO - GitLab SaaS, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, Canary


Locations

Google Compute Engine, Digital Ocean, Zendesk, AWS




August 26, 2022 06:21 UTC
[Resolved] We are no longer seeing connectivity issues for GitLab.com after a period of monitoring. We therefore believe that this incident has been resolved. Details in : gitlab.com/gitlab-com/gl-infra/production/-/issues/7656

August 26, 2022 05:08 UTC
[Monitoring] All the services seems to have recovered. We will continue to monitor the performance and provide the updates here. More information on the root cause can be found here: gitlab.com/gitlab-com/gl-infra/production/-/issues/7656

August 26, 2022 04:57 UTC
[Investigating] We are still investigating the root cause of this issue. More details here: gitlab.com/gitlab-com/gl-infra/production/-/issues/7656

August 26, 2022 04:55 UTC
[Investigating] We are noticing some slow response issues with GitLab.com.

June 2022

Spike in error rate on gitlab.com

June 23, 2022 07:19 UTC

Incident Status

Operational


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners for GitLab community contributions, SAML SSO - GitLab SaaS, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, Canary


Locations

Google Compute Engine, Digital Ocean, Zendesk, AWS




June 23, 2022 07:19 UTC
[Resolved] Error rates have returned to normal levels. Follow-up discussion and actions are being discussed here: gitlab.com/gitlab-com/gl-infra/production/-/issues/7310

June 23, 2022 06:46 UTC
[Monitoring] We are investigating a brief spike in errors that lasted about 20 minutes, we are monitoring the recovery and will put more detail in gitlab.com/gitlab-com/gl-infra/production/-/issues/7310

Connectivity issues for GitLab.com

June 21, 2022 08:43 UTC

Incident Status

Service Disruption


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners for GitLab community contributions, SAML SSO - GitLab SaaS, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, Canary


Locations

Google Compute Engine, Digital Ocean, Zendesk, AWS




June 21, 2022 08:43 UTC
[Resolved] We are no longer seeing connectivity issues for GitLab.com after a period of monitoring. We therefore believe that this incident has been resolved. Details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/7288

June 21, 2022 08:00 UTC
[Monitoring] Services seem to be back to normal, and we continue monitoring. Details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/7288

June 21, 2022 07:45 UTC
[Identified] We are seeing services slowly recovering and continue monitoring issues with our CDN provider.

June 21, 2022 07:22 UTC
[Identified] No material updates at this point. We are continuing to monitor issues with our CDN provider

June 21, 2022 07:02 UTC
[Identified] We are continuing to monitor connectivity issues from some regions due to connectivity problems from our CDN provider.

June 21, 2022 06:52 UTC
[Investigating] We are investigating connectivity issues for GitLab.com, this is affecting all services.

Gitlab.com is having slowness issues

June 15, 2022 23:28 UTC

Incident Status

Degraded Performance


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners for GitLab community contributions, SAML SSO - GitLab SaaS, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, Canary


Locations

Google Compute Engine, Digital Ocean, Zendesk, AWS




June 15, 2022 23:28 UTC
[Resolved] We are no longer seeing performance issues on GitLab.com after a period of monitoring. We believe the incident has now been resolved. More details can be found here: gitlab.com/gitlab-com/gl-infra/production/-/issues/7258

June 15, 2022 23:12 UTC
[Identified] Our teams had identified the root cause of the slowness and are discussing preventive actions for it. Performance is back to normal.

June 15, 2022 22:53 UTC
[Investigating] Our teams detected a performance degradation for all components for GitLab.com, it appears this degradation is now stabilizing back to normal performance. We are still investigating.

June 15, 2022 22:36 UTC
[Investigating] Requests to GitLab.com are seeing elevated error rates and slowness, we are investigating.

April 2022

Elevated Error Rate on GitLab.com

April 28, 2022 07:37 UTC

Incident Status

Service Disruption


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners for GitLab community contributions, SAML SSO - GitLab SaaS, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, Canary


Locations

Google Compute Engine, Digital Ocean, Zendesk, AWS




April 28, 2022 07:37 UTC
[Resolved] We transitioned this incident to resolved. Check the issue for more details: gitlab.com/gitlab-com/gl-infra/production/-/issues/6933

April 28, 2022 07:23 UTC
[Monitoring] We have identified the cause to this incident and do not anticipate any additional impacts at this time. We are continuing to monitor services as well as complete additional investigation. More details here: gitlab.com/gitlab-com/gl-infra/production/-/issues/6933

April 28, 2022 06:34 UTC
[Investigating] We are still investigating the root cause. The error rates on affected systems recovering, and we are monitoring to ensure that the issue does not recur.

April 28, 2022 06:12 UTC
[Investigating] We're investigating increased error rates across GitLab-com services, some users might be experiencing 500 errors intermittently, more info can be found in here: gitlab.com/gitlab-com/gl-infra/production/-/issues/6933

September 2021

Increased error rates across all GitLab services

September 12, 2021 21:48 UTC

Incident Status

Service Disruption


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners for GitLab community contributions, SAML SSO - GitLab SaaS, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, Canary


Locations

Google Compute Engine, Azure, Digital Ocean, Zendesk, AWS




September 12, 2021 21:48 UTC
[Resolved] After investigation we determined that the elevated error rates were caused by database connection saturation; further details will be in this issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/5514

September 12, 2021 21:23 UTC
[Monitoring] GitLab performance and error rates have returned to normal; we are investigating the root cause to understand the likelyhood it will reoccur.

September 12, 2021 21:04 UTC
[Investigating] We are investigating slow response times and increased error rates across all GitLab services. Incident: gitlab.com/gitlab-com/gl-infra/production/-/issues/5514

August 2021

Elevated error rates on GitLab.com

August 11, 2021 17:51 UTC

Elevated error rates on GitLab.comPartial Service Disruption

Incident Status

Partial Service Disruption


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners for GitLab community contributions, SAML SSO - GitLab SaaS, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, Canary


Locations

Google Compute Engine, Azure, Digital Ocean, Zendesk, AWS




August 11, 2021 17:51 UTC
[Resolved] As GitLab.com is operating normally, we'll now be marking this incident as resolved. Further investigation efforts and details surrounding the issue can be found in: gitlab.com/gitlab-com/gl-infra/production/-/issues/5333

August 11, 2021 17:44 UTC
[Investigating] No material updates to report - our investigation is still ongoing. Details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/5333

August 11, 2021 17:27 UTC
[Investigating] All GitLab.com services are operating normally, but our investigation into the cause of the errors continues. Details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/5333

August 11, 2021 17:18 UTC
[Investigating] At 17:04 UTC, there was a 5 minute elevated error ratio event for all GitLab.com services. It has recovered, but we are investigating. Details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/5333

Increased error rates across all services

August 10, 2021 10:39 UTC

Incident Status

Partial Service Disruption


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners for GitLab community contributions, SAML SSO - GitLab SaaS, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, Canary


Locations

Google Compute Engine, Azure, Digital Ocean, Zendesk, AWS




August 10, 2021 10:39 UTC
[Resolved] The issue has been resolved, and we can confirm that all services on GitLab.com are fully operational. A full timeline of this incident and more details are available in gitlab.com/gitlab-com/gl-infra/production/-/issues/5324

August 10, 2021 10:05 UTC
[Monitoring] We are back to normal error rates across all services. We keep monitoring the situation. Refer to gitlab.com/gitlab-com/gl-infra/production/-/issues/5324 for more updates.

August 10, 2021 09:52 UTC
[Investigating] We are investigating increased error rates across all services, see gitlab.com/gitlab-com/gl-infra/production/-/issues/5324 for more details

GitLab performance degraded

August 6, 2021 12:48 UTC

Incident Status

Operational


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners for GitLab community contributions, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, Canary


Locations

Google Compute Engine, Azure, Digital Ocean, Zendesk, AWS




August 6, 2021 12:48 UTC
[Resolved] From 12:05 - 12:15 UTC, we saw a system degradation across all of our components, which seems to have recovered at the moment. Our engineers are investigating. Further details are available in: gitlab.com/gitlab-com/gl-infra/production/-/issues/5306

July 2021

Issue Uploading Artifacts from CI

July 21, 2021 01:39 UTC

Incident Status

Service Disruption


Components

Website, API, Git Operations, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners for GitLab community contributions, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, Canary


Locations

Google Compute Engine, Azure, Digital Ocean, Zendesk, AWS




July 21, 2021 01:39 UTC
[Resolved] This incident is now resolved and we are no longer seeing any errors. We will continue to investigate the cause internally. Full timeline of this incident and more details available in gitlab.com/gitlab-com/gl-infra/production/-/issues/5194

July 21, 2021 01:12 UTC
[Monitoring] The rollout has completed and we are no longer seeing errors with the upload. This has been applied to all our servers. We are continuously monitoring this further for any issues.

July 21, 2021 01:04 UTC
[Identified] The rollout has completed on two of our clusters. This will continue on our last few clusters and we will post further updates once completed. More details at gitlab.com/gitlab-com/gl-infra/production/-/issues/5194

July 21, 2021 00:44 UTC
[Identified] We are currently working on a recent configuration rollback that affected uploading CI artifacts. More details at gitlab.com/gitlab-com/gl-infra/production/-/issues/5194

July 21, 2021 00:28 UTC
[Investigating] We have identified a failure in uploading artifacts from CI which is currently showing a 400 error. We are investigating the issue at gitlab.com/gitlab-com/gl-infra/production/-/issues/5194 and will post more updates.

June 2021

Short service disruption on GitLab.com

June 21, 2021 21:34 UTC

Incident Status

Degraded Performance


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners for GitLab community contributions, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, Canary


Locations

Google Compute Engine, Azure, Digital Ocean, Zendesk, AWS




June 21, 2021 21:34 UTC
[Resolved] Our team has confirmed that the disruption is now gone, and that services are back to normal, Investigation will continue on gitlab.com/gitlab-com/gl-infra/production/-/issues/4945 for additional details and root cause analysis.

June 21, 2021 21:14 UTC
[Investigating] GitLab.com is up and operational, but from 20:53 to 20:58, we saw an increase in errors on GitLab.com web, API, and Git services that were a short disruption, now gone. Investigation is being done on gitlab.com/gitlab-com/gl-infra/production/-/issues/4945





Back to current status