All Systems Operational

Updated a few seconds ago

Back to current status

Status History

Filter: CI/CD - Hosted runners on Windows (Clear)



April 2024

Runners are not picking up jobs causing 500 errors

April 11, 2024 14:12 UTC

Incident Status

Partial Service Disruption


Components

CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions


Locations

Google Compute Engine




April 11, 2024 14:12 UTC
[Resolved] This incident has been resolved and jobs are being picked up without any errors. More information can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/17819.

April 11, 2024 13:31 UTC
[Monitoring] The rollback has completed and we are seeing the jobs being picked up as normal. We're continuing to monitor to ensure the problem does not recur. Details in gitlab.com/gitlab-com/gl-infra/production/-/issues/17819

April 11, 2024 13:21 UTC
[Identified] We have started to roll back the relevant changes and anticipate that this should complete within 45 minutes. Details in - gitlab.com/gitlab-com/gl-infra/production/-/issues/17819

April 11, 2024 13:00 UTC
[Identified] We have started to roll back the relevant changes. Details in - gitlab.com/gitlab-com/gl-infra/production/-/issues/17819

April 11, 2024 12:30 UTC
[Identified] We have identified the cause of this problem and are starting to roll back the relevant changes. Details in - gitlab.com/gitlab-com/gl-infra/production/-/issues/17819

April 11, 2024 12:18 UTC
[Investigating] No material updates to report. We're continuing to investigate the issue. Tracking in gitlab.com/gitlab-com/gl-infra/production/-/issues/17819

April 11, 2024 12:03 UTC
[Investigating] We're currently investigating issues with jobs not being picked up by runners. More details in - gitlab.com/gitlab-com/gl-infra/production/-/issues/17819

March 2024

Elevated queue size on Shared Runners

March 15, 2024 18:40 UTC

Incident Status

Degraded Performance


Components

CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS


Locations

Google Compute Engine, AWS




March 15, 2024 18:40 UTC
[Resolved] Job pickup delays and slowness on running jobs have not been observed for some time, and the team does not anticipate the problem returning. Additional information will be shared in gitlab.com/gitlab-com/gl-infra/production/-/issues/17724

March 15, 2024 14:35 UTC
[Monitoring] As things remain stable, we will continue to monitor the situation and will provide further updates as necessary. Full details are available in gitlab.com/gitlab-com/gl-infra/production/-/issues/17724

March 15, 2024 13:34 UTC
[Monitoring] There are no material updates to report. We will continue to monitor and post updates hourly. Full details are available in gitlab.com/gitlab-com/gl-infra/production/-/issues/17724

March 15, 2024 12:33 UTC
[Monitoring] Things remain stable, with no material updates at the time. We are continuing to monitor the situation, and will provide hourly updates. Full details are available in gitlab.com/gitlab-com/gl-infra/production/-/issues/17724

March 15, 2024 11:30 UTC
[Monitoring] No material updates at the time. We are continuing to monitor the situation, and will provide hourly updates. Full details are available in gitlab.com/gitlab-com/gl-infra/production/-/issues/17724

March 15, 2024 10:29 UTC
[Monitoring] The impact from this incident appears to now be mitigated, however we will continue to monitor the situation and will provide further updates as necessary. Full details are available in gitlab.com/gitlab-com/gl-infra/production/-/issues/17724

March 15, 2024 09:26 UTC
[Monitoring] The impact from this incident appears low, however we are continuing to monitor the situation, and will post another update in one hour. Full details are available in gitlab.com/gitlab-com/gl-infra/production/-/issues/17724

March 15, 2024 08:15 UTC
[Monitoring] We are continuing to monitor this situation, with no additional details to add at this time. Full details are available in gitlab.com/gitlab-com/gl-infra/production/-/issues/17724

March 15, 2024 07:11 UTC
[Identified] No material updates to report. We will continue to monitor to the situation for now. Details in gitlab.com/gitlab-com/gl-infra/production/-/issues/17724

March 15, 2024 05:18 UTC
[Identified] We have narrowed down the potential root cause but we will continue the investigation. No further impact on CI/CD pipeline execution. Next update will be provided in 1 hour. Details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/17724

March 15, 2024 03:35 UTC
[Investigating] We are continuing to monitor the extended runner queue size. Impact on CI/CD pipeline execution does not appear to be significant at this time. A further update will be provided in 1 hour. Details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/17724

March 15, 2024 02:31 UTC
[Investigating] Analysis continues. A further update will be provided in 1 hour, or sooner should there be a significant update to report. Details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/17724

March 15, 2024 01:57 UTC
[Investigating] The investigation is ongoing, we continue to assess the root cause of the extended queue size with the help of additional resources. A further update will be provided in 30 minutes. Details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/17724

March 15, 2024 01:34 UTC
[Investigating] No material updates at this time. Investigation into the increasing runner queue size is ongoing. Details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/17724

March 15, 2024 01:13 UTC
[Investigating] We are continuing to investigate the root cause of the elevated queue size. For more information: gitlab.com/gitlab-com/gl-infra/production/-/issues/17724

March 15, 2024 00:55 UTC
[Investigating] No material update at this time. Our team is still investigating the root cause.

March 15, 2024 00:42 UTC
[Investigating] We are investigating an issue with elevated queue size on our GitLab.com runners. Users may experience slowness with runners picking up jobs, or running jobs. For more information: gitlab.com/gitlab-com/gl-infra/production/-/issues/17724

January 2024

Sidekiq degredation

January 30, 2024 23:02 UTC

Sidekiq degredationDegraded Performance

Incident Status

Degraded Performance


Components

Website, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions, Background Processing


Locations

Google Compute Engine




January 30, 2024 23:02 UTC
[Resolved] The feature flag responsible for the issue has been disabled by default and operations are back to normal. This incident is considered resolved. Further details can be found in the incident issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/17504

January 30, 2024 22:02 UTC
[Identified] We've identified the issue and have narrowed the problem to a recent feature flag that created slow queries caused by specific import workers. These slow queries lead to database saturation. To mitigate the issue, the team has disabled the feature flag. We are now re-processing jobs from these workers. This issue is considered mitigated. Further updates will be minimal until there is a permanent code fix in place. More details in gitlab.com/gitlab-com/gl-infra/production/-/issues/17504

January 30, 2024 20:22 UTC
[Investigating] No material update at this time. Our team is still investigating the root cause.

January 30, 2024 19:54 UTC
[Investigating] We've temporarily disabled some features of the GitHub import functionality. Users might experience issues with GitHub imports at this time while our team continues to investigate.

January 30, 2024 19:28 UTC
[Investigating] Sidekiq has largely recovered and jobs are processing. Our team is still investigating the root cause.

January 30, 2024 19:08 UTC
[Investigating] Our team is still investigating the root cause of this issue. We'll continue to provide updates as more information is available.

January 30, 2024 18:51 UTC
[Investigating] We're currently seeing some sidekiq degradation that appears to be impacting some pipelines and merge requests. Our team is investigating. More information in gitlab.com/gitlab-com/gl-infra/production/-/issues/17504

November 2023

Performance issues affecting processing of MRs

November 20, 2023 18:49 UTC

Incident Status

Degraded Performance


Components

CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions, Background Processing


Locations

Google Compute Engine, AWS




November 20, 2023 18:49 UTC
[Resolved] Performance of CI pipelines and merge requests has fully returned to normal and shows no further signs of degradation. The status page is being marked resolved, and future updates can be found in the production issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/17158

November 20, 2023 18:05 UTC
[Monitoring] Performance of CI job pickup and merge requests has returned to a normal state. The incident response team will continue monitoring and reviewing the rolled back changes. Next update in an hour, unless there is anything to report prior.

November 20, 2023 17:21 UTC
[Investigating] We have started to notice some signs of recovery after the rollback. Investigation into root causes is still ongoing to identify the source of the CI and merge request performance problems.

November 20, 2023 16:50 UTC
[Investigating] Rollback to an earlier deployment has completed. No material updates in the CI job pickup slowness has been observed yet. Next update will be in 30 minutes unless there is anything to report sooner.

November 20, 2023 16:32 UTC
[Investigating] The rollback to an earlier deployment is in progress and a review of all changes is in progress to identify the root cause of slow CI job pickup. Additional support from other teams is being brought in to assist.

November 20, 2023 16:12 UTC
[Investigating] The incident response team is continuing to investigate the slow CI job pickup. A rollback to a previous deployment is in progress as a potential mitigating solution.

November 20, 2023 15:53 UTC
[Investigating] We are actively investigating an issue where CI jobs are being picked up slowly affecting performance of MR's. More details here: gitlab.com/gitlab-com/gl-infra/production/-/issues/17158

September 2023

Main DB Cluster PostgreSQL 14 Upgrade

September 9, 2023 17:25 UTC

Description

We will be undergoing scheduled maintenance to our database layer. The maintenance will take up to 6 hours, from 14:00 UTC to 20:00 UTC. GitLab.com will continue to be available during the maintenance window. We apologize in advance for any inconvenience this may cause. See gitlab.com/gitlab-com/gl-infra/production/-/issues/16266 for details.


Components

Website, API, Git Operations, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners for GitLab community contributions


Locations

Google Compute Engine


Schedule

September 9, 2023 14:00 - September 9, 2023 20:00 UTC



September 9, 2023 17:25 UTC
[Update] GitLab.com planned maintenance for the database layer is complete. We'll be monitoring the platform to ensure all systems are functioning correctly. Thank you for your patience.

September 9, 2023 16:12 UTC
[Update] GitLab.com planned maintenance for the database layer is almost complete. We’re continuing to verify that all systems are functioning correctly. Thank you for your patience. See <gitlab.com/gitlab-com/gl-infra/production/-/issues/16266>

September 9, 2023 13:01 UTC
[Update] We will be undergoing scheduled maintenance to our main database layer in 1 hour. The maintenance will take up to 5 hours starting from 14:00 UTC to 19:00 UTC. GitLab.com will be available but users may experience degraded performance during the maintenance window. We apologize in advance for any inconvenience this may cause. See gitlab.com/gitlab-com/gl-infra/production/-/issues/16266

September 9, 2023 10:08 UTC
[Update] We will be undergoing scheduled maintenance to our main database layer in 3 hours. The maintenance will take up to 5 hours starting from 14:00 UTC to 19:00 UTC. GitLab.com will be available but users may experience degraded performance during the maintenance window. We apologize in advance for any inconvenience this may cause. See gitlab.com/gitlab-com/gl-infra/production/-/issues/16266

September 8, 2023 13:59 UTC
[Update] Reminder: Tomorrow, we will be undergoing scheduled maintenance to our main database layer. The maintenance will take up to 5 hours starting from 14:00 UTC to 19:00 UTC. GitLab.com will be available but users may experience degraded performance during the maintenance window. We apologize in advance for any inconvenience this may cause. See gitlab.com/gitlab-com/gl-infra/production/-/issues/16266

September 6, 2023 12:13 UTC
[Update] In 3 days, we will be undergoing scheduled maintenance to our main database layer. The maintenance will take up to 5 hours starting from 14:00 UTC to 19:00 UTC. GitLab.com will be available but users may experience degraded performance during the maintenance window. We apologize in advance for any inconvenience this may cause. See #16266

August 31, 2023 00:01 UTC
[Update] Next week, we will be undergoing scheduled maintenance to our main database layer. The maintenance will take up to 5 hours starting from 14:00 UTC to 19:00 UTC. GitLab.com will be available but users may experience degraded performance during the maintenance window. We apologize in advance for any inconvenience this may cause. See <gitlab.com/gitlab-com/gl-infra/production/-/issues/14403>

August 2023

Main DB Cluster PostgreSQL 14 Upgrade

August 26, 2023 19:51 UTC

Description

We will be undergoing scheduled maintenance to our database layer. The maintenance will take up to 5 hours, from 14:00 UTC to 19:00 UTC. GitLab.com will continue to be available during the maintenance window. We apologize in advance for any inconvenience this may cause. See gitlab.com/gitlab-com/gl-infra/production/-/issues/14403 for details.


Components

Website, API, Git Operations, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners for GitLab community contributions


Locations

Google Compute Engine


Schedule

August 26, 2023 14:00 - August 26, 2023 19:00 UTC



August 26, 2023 19:51 UTC
[Update] GitLab.com rollback for the database layer is complete, and we're back up and running. We'll be monitoring the platform to ensure all systems are functioning correctly. Thank you for your patience.

August 26, 2023 19:21 UTC
[Update] GitLab.com rollback for the database layer is almost complete. The site is back up and running although we're continuing to verify that all systems are functioning correctly. Thank you for your patience.

August 26, 2023 18:29 UTC
[Update] Due to an issue during the planned maintenance for the database layer, we have initiated a rollback of the changes. We will send another update within the next 30 minutes.

August 26, 2023 14:11 UTC
[Update] We will be starting scheduled maintenance to our main database layer now. The maintenance will take up to 5 hours. GitLab.com will be available but users may experience degraded performance during the maintenance window. We apologize in advance for any inconvenience this may cause. See gitlab.com/gitlab-com/gl-infra/production/-/issues/14403

August 26, 2023 13:01 UTC
[Update] We will be undergoing scheduled maintenance to our main database layer in 1 hour. The maintenance will take up to 5 hours starting from 14:00 UTC to 19:00 UTC. GitLab.com will be available but users may experience degraded performance during the maintenance window. We apologize in advance for any inconvenience this may cause. See gitlab.com/gitlab-com/gl-infra/production/-/issues/14403

August 26, 2023 11:06 UTC
[Update] We will be undergoing scheduled maintenance to our main database layer in 3 hours. The maintenance will take up to 5 hours starting from 14:00 UTC to 19:00 UTC. GitLab.com will be available but users may experience degraded performance during the maintenance window. We apologize in advance for any inconvenience this may cause. See gitlab.com/gitlab-com/gl-infra/production/-/issues/14403

August 25, 2023 15:41 UTC
[Update] Reminder: Tomorrow, we will be undergoing scheduled maintenance to our main database layer. The maintenance will take up to 5 hours starting from 14:00 UTC to 19:00 UTC. GitLab.com will be available but users may experience degraded performance during the maintenance window. We apologize in advance for any inconvenience this may cause. See gitlab.com/gitlab-com/gl-infra/production/-/issues/14403

August 23, 2023 12:09 UTC
[Update] In 3 days, we will be undergoing scheduled maintenance to our main database layer. The maintenance will take up to 5 hours starting from 14:00 UTC to 19:00 UTC. GitLab.com will be available but users may experience degraded performance during the maintenance window. We apologize in advance for any inconvenience this may cause. See gitlab.com/gitlab-com/gl-infra/production/-/issues/14403

August 20, 2023 21:13 UTC
[Update] Next week, we will be undergoing scheduled maintenance to our main database layer. The maintenance will take up to 5 hours starting from 14:00 UTC to 19:00 UTC. GitLab.com will be available but users may experience degraded performance during the maintenance window. We apologize in advance for any inconvenience this may cause. See <gitlab.com/gitlab-com/gl-infra/production/-/issues/14403>

CI Cluster PostgreSQL 14 Upgrade

August 12, 2023 16:51 UTC

Description

We will be undergoing scheduled maintenance to our CI database layer. The maintenance will take up to 5 hours, from 14:00 UTC to 19:00 UTC. GitLab.com will continue to be available during the maintenance window. We apologize in advance for any inconvenience this may cause. See gitlab.com/gitlab-com/gl-infra/production/-/issues/16093 for details.


Components

API, Git Operations, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners for GitLab community contributions


Locations

Google Compute Engine


Schedule

August 12, 2023 14:00 - August 12, 2023 19:00 UTC



August 12, 2023 16:51 UTC
[Update] GitLab.com planned maintenance for the database layer is complete. We'll be monitoring the platform to ensure all systems are functioning correctly. Thank you for your patience.

August 12, 2023 16:32 UTC
[Update] GitLab.com planned maintenance for the database layer is complete. We'll be monitoring the platform to ensure all systems are functioning correctly. Thank you for your patience.

August 12, 2023 16:28 UTC
[Update] The scheduled maintenance to our ci database layer is ongoing. The maintenance will take up to 5 hours starting from 14:00 UTC to 19:00 UTC. GitLab.com will be available but users may experience degraded performance during the maintenance window. We apologize in advance for any inconvenience this may cause. See <gitlab.com/gitlab-com/gl-infra/production/-/issues/15945>

August 12, 2023 15:48 UTC
[Update] GitLab.com planned maintenance for the database layer is almost complete. We're continuing to verify that all systems are functioning correctly. Thank you for your patience.

August 12, 2023 13:00 UTC
[Update] We will be undergoing scheduled maintenance to our ci database layer in 1 hour. The maintenance will take up to 5 hours starting from 14:00 UTC to 19:00 UTC. GitLab.com will be available but users may experience degraded performance during the maintenance window. We apologize in advance for any inconvenience this may cause. See <gitlab.com/gitlab-com/gl-infra/production/-/issues/15945>

August 12, 2023 11:00 UTC
[Update] We will be undergoing scheduled maintenance to our ci database layer in 3 hours. The maintenance will take up to 5 hours starting from 14:00 UTC to 19:00 UTC. GitLab.com will be available but users may experience degraded performance during the maintenance window. We apologize in advance for any inconvenience this may cause. See <gitlab.com/gitlab-com/gl-infra/production/-/issues/15945>

August 12, 2023 04:29 UTC
[Update] Reminder: Today, we will be undergoing scheduled maintenance to our CI database layer. The maintenance will take up to 5 hours starting from 14:00 UTC to 19:00 UTC. GitLab.com will be available but users may experience degraded performance during the maintenance window. We apologize in advance for any inconvenience this may cause. See gitlab.com/gitlab-com/gl-infra/production/-/issues/16093

August 9, 2023 15:42 UTC
[Update] In 3 days, we will be undergoing scheduled maintenance to our ci database layer. The maintenance will take up to 5 hours starting from 14:00 UTC to 19:00 UTC. GitLab.com will be available but users may experience degraded performance during the maintenance window. We apologize in advance for any inconvenience this may cause. See <gitlab.com/gitlab-com/gl-infra/production/-/issues/15945>

July 2023

Performance issues on GitLab.com

July 15, 2023 04:51 UTC

Incident Status

Degraded Performance


Components

API, Git Operations, Container Registry, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners for GitLab community contributions, Background Processing, Canary


Locations

Google Compute Engine




July 15, 2023 04:51 UTC
[Resolved] This incident has been marked as resolved, as we don't see any outstanding issues with GitLab.com. A full timeline of this incident can be found in our Incident Report available at gitlab.com/gitlab-com/gl-infra/production/-/issues/16042

July 15, 2023 03:29 UTC
[Monitoring] We will update the status of this incident to `Operational` for now..

July 15, 2023 03:26 UTC
[Monitoring] The rollback has been completed. We are observing that services have recovered. We will continue to monitor for any issues. More details in gitlab.com/gitlab-com/gl-infra/production/-/issues/16042

July 15, 2023 03:02 UTC
[Identified] We are working on performing a rollback on GitLab.com. Next expected update will be in 30 minutes. More details in gitlab.com/gitlab-com/gl-infra/production/-/issues/16042

July 15, 2023 02:52 UTC
[Identified] We are continuing to observe performance issues on GitLab.com. We are investigating options to perform a rollback. More details in gitlab.com/gitlab-com/gl-infra/production/-/issues/16042

July 15, 2023 02:18 UTC
[Identified] No material updates to report at this time. Infrastructure teams are continuing investigations. Details in gitlab.com/gitlab-com/gl-infra/production/-/issues/16042

July 15, 2023 01:45 UTC
[Identified] We have identified the problem and mitigated the worst of it. Our infrastructure team is continuing investigations. Details in gitlab.com/gitlab-com/gl-infra/production/-/issues/16042.

July 15, 2023 01:13 UTC
[Investigating] We have detected a problem related to GitLab.com services and are actively investigating. More details to follow.

June 2023

Slow requests on GitLab.com

June 7, 2023 13:32 UTC

Slow requests on GitLab.comPartial Service Disruption

Incident Status

Partial Service Disruption


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners for GitLab community contributions, SAML SSO - GitLab SaaS, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, docs.gitlab.com, Canary


Locations

Google Compute Engine, Digital Ocean, Zendesk, AWS




June 7, 2023 13:32 UTC
[Resolved] We have identified the root cause, have seen recovery and are therefore resolving this incident. More information in: gitlab.com/gitlab-com/gl-infra/production/-/issues/14839

June 7, 2023 13:21 UTC
[Investigating] Customers are experiencing slow requests on GitLab.com. More information in: gitlab.com/gitlab-com/gl-infra/production/-/issues/14839

May 2023

Hardware upgrade for the GitLab.com datastore

May 27, 2023 04:06 UTC

Description

In 2 weeks, as part of a planned maintenance window, we will perform a hardware upgrade for the GitLab.com datastore. Users may experience temporary 50X errors for a brief period during this window. See gitlab.com/gitlab-com/gl-infra/production/-/issues/10694


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners for GitLab community contributions, SAML SSO - GitLab SaaS, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, docs.gitlab.com, Canary


Locations

Google Compute Engine, Digital Ocean, Zendesk, AWS


Schedule

May 27, 2023 03:00 - May 27, 2023 04:00 UTC



May 27, 2023 04:06 UTC
[Update] GitLab.com's database layer maintenance is complete now, and we're fully back up and running. We'll be monitoring the platform to ensure all systems are functioning correctly. Thank you for your patience.

May 27, 2023 03:41 UTC
[Update] The database switchover is now complete, we expect all SQL statements to be routed to the new nodes. The site is back up and we're continuing to verify that all systems are functioning correctly. Thank you for your patience.

May 27, 2023 03:00 UTC
[Update] The database maintenance is starting now. The database Primary node switchover should happen at any moment in the next hour and during the switchover users might see 50X errors for a very brief span of time. For details see gitlab.com/gitlab-com/gl-infra/production/-/issues/10694

May 27, 2023 01:51 UTC
[Update] We will be conducting a maintenance database activity in an hour, from 03:00 to 04:00 UTC. Users might see 50X errors for a very brief span of time during this window. For details see gitlab.com/gitlab-com/gl-infra/production/-/issues/10694

May 26, 2023 02:27 UTC
[Update] We will be conducting a maintenance database activity Tomorrow, 2023-05-27, from 03:00 to 04:00 UTC. Users might see 50X errors for a very brief span of time during this window. For details see gitlab.com/gitlab-com/gl-infra/production/-/issues/10694

May 24, 2023 02:14 UTC
[Update] Correction: We will be conducting a maintenance database activity this Saturday, 2023-05-27. from 03:00 to 04:00 UTC. Users might see 50X errors for a very brief span of time during this window. For details see gitlab.com/gitlab-com/gl-infra/production/-/issues/10694

May 24, 2023 02:05 UTC
[Update] We will be conducting a maintenance database activity this Saturday, 2023-05-20, from 03:00 to 04:00 UTC. Users might see 50X errors for a very brief span of time during this window. For details see gitlab.com/gitlab-com/gl-infra/production/-/issues/10694

May 20, 2023 03:58 UTC
[Update] The maintenance is rescheduled to next Saturday, 2023-05-27, from 03:00 to 04:00 UTC. There is currently a long database migration running to fix an incident (gitlab.com/gitlab-com/gl-infra/production/-/issues/14468) that we can't risk to interrupt. We apologise for any inconvenience.

May 20, 2023 03:07 UTC
[Update] The scheduled maintenance will be not be starting at 03:00 UTC, and has been delayed. More details soon to follow.

May 20, 2023 02:00 UTC
[Update] We will be conducting a maintenance database activity in an hour, from 03:00 to 04:00 UTC. Users might see 50X errors for a very brief span of time during this window. For details see gitlab.com/gitlab-com/gl-infra/production/-/issues/10694

May 20, 2023 01:00 UTC
[Update] We will be conducting a maintenance database activity in 2 hours, from 03:00 to 04:00 UTC. Users might see 50X errors for a very brief span of time during this window. For details see gitlab.com/gitlab-com/gl-infra/production/-/issues/10694

May 19, 2023 02:00 UTC
[Update] We will be conducting a maintenance database activity tomorrow, 2023-05-20, from 03:00 to 04:00 UTC. Users might see 50X errors for a very brief span of time during this window. For details see gitlab.com/gitlab-com/gl-infra/production/-/issues/10694

May 17, 2023 02:00 UTC
[Update] We will be conducting a maintenance database activity this Saturday, 2023-05-20, from 03:00 to 04:00 UTC. Users might see 50X errors for a very brief span of time during this window. For details see gitlab.com/gitlab-com/gl-infra/production/-/issues/10694

May 12, 2023 02:00 UTC
[Update] Next week, as part of a planned maintenance window on 2023-05-20 from 03:00 to 04:00 UTC, We will perform a hardware upgrade for the GitLab.com datastore. Users may experience temporary 50X errors for a brief period during this window. See gitlab.com/gitlab-com/gl-infra/production/-/issues/10694

March 2023

License Scanning temporarily disabled

March 31, 2023 21:23 UTC

License Scanning temporarily disabledPartial Service Disruption

Incident Status

Partial Service Disruption


Components

CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners for GitLab community contributions


Locations

Google Compute Engine




March 31, 2023 21:23 UTC
[Resolved] After monitoring, GitLab has seen no further issues with license scanning and we are resolving the incident. Further details can be found here: gitlab.com/gitlab-org/gitlab/-/issues/398679

March 31, 2023 01:48 UTC
[Monitoring] License scanning has now been re-enabled. The optimizations in place have proven to reduce significantly the performance issues on the database. We will continue to monitor. Further details can be found here: gitlab.com/gitlab-org/gitlab/-/issues/398679

March 30, 2023 07:20 UTC
[Identified] We have deployed a merge request to address the issue. We are currently reviewing this and will continue to monitor the situation. You can refer to the incident issue gitlab.com/gitlab-com/gl-infra/production/-/issues/8593 for more details.

March 28, 2023 01:09 UTC
[Identified] Our engineering team has a merge request to fix this issue; it is being reviewed and can be followed at gitlab.com/gitlab-org/gitlab/-/merge_requests/115773. At this time we anticipate the fix should be merged within one to two days. Explicitly including the license scanning template in your CI configuration will allow scans to run correctly. The cu

March 24, 2023 19:41 UTC
[Identified] Our engineering team continues to work to restore full license scanning operation. Explicitly including the license scanning template in your CI configuration will allow scans to run correctly. Next update will be Monday, March 27th unless there is news to share sooner.

March 23, 2023 19:57 UTC
[Identified] A resolution approach has been decided upon, but it will take time to implement. You can include the license scanning template to ensure proper detection while our engineering team continues working. Expect the next update in 24 hours unless there's new information before.

March 23, 2023 18:35 UTC
[Identified] We are evaluating several potential resolutions. Next update in one hour. To ensure your license scan runs correctly, explicitly include the template per the documentation here: docs.gitlab.com/ee/user/compliance/license_compliance/#include-the-license-scanning-template

March 23, 2023 18:01 UTC
[Identified] No licenses will be detected unless the license scanning template is explicitly included in your .gitlab-ci.yml file. Our engineering team is working to restore full functionality. Details in this issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/8593

Performance problems with CI

March 14, 2023 20:03 UTC

Performance problems with CIDegraded Performance

Incident Status

Degraded Performance


Components

Website, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners for GitLab community contributions


Locations

Google Compute Engine




March 14, 2023 20:03 UTC
[Resolved] Our team confirmed that load averages are back to normal and the affected CI components are back to operational status, for more information: gitlab.com/gitlab-com/gl-infra/production/-/issues/8541

March 14, 2023 19:07 UTC
[Monitoring] Our team is still monitoring the load to make sure that the loads get back to normal values. For more information: gitlab.com/gitlab-com/gl-infra/production/-/issues/8541

March 14, 2023 18:29 UTC
[Monitoring] Our team is monitoring the load to make sure that the loads get back to normal values. For more information: gitlab.com/gitlab-com/gl-infra/production/-/issues/8541

March 14, 2023 18:02 UTC
[Monitoring] Disabling the feature flag resulted in a decrease in load in Patroni, our team is monitoring the load to make sure that the loads get back to normal values. For more information: gitlab.com/gitlab-com/gl-infra/production/-/issues/8541

March 14, 2023 17:46 UTC
[Identified] Our team has identified a recently enabled feature that is possibly causing the load issue with Patroni. We have disabled the feature and are currently working on evaluating the performance changes. gitlab.com/gitlab-com/gl-infra/production/-/issues/8541

March 14, 2023 17:25 UTC
[Investigating] Our teams are assessing recent partition tables to the database to determine if a recently implemented change on the database is creating the loads on Patroni, for more information please see gitlab.com/gitlab-com/gl-infra/production/-/issues/8541

March 14, 2023 17:10 UTC
[Investigating] Our teams are still investigating the root cause of the issue, currently, we are analyzing loads in our databases to pinpoint the saturation on Patroni. For more information please see gitlab.com/gitlab-com/gl-infra/production/-/issues/8541

March 14, 2023 16:50 UTC
[Investigating] No material updates at the moment. Our teams are still investigating the root cause of the problem. For more information please see gitlab.com/gitlab-com/gl-infra/production/-/issues/8541

March 14, 2023 16:31 UTC
[Investigating] There is an ongoing incident with our CI Postgres cluster. Some CI jobs are getting delayed. For more information please see the incident (gitlab.com/gitlab-com/gl-infra/production/-/issues/8541)

February 2023

Degraded performance on Gitlab.com

February 27, 2023 21:50 UTC

Incident Status

Operational


Components

Website, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners for GitLab community contributions


Locations

Google Compute Engine




February 27, 2023 21:50 UTC
[Resolved] We have identified the root cause, and are preparing preventative actions. Our services have recovered and we now consider this incident to be resolved. Details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/8471

February 27, 2023 21:36 UTC
[Monitoring] At this time there are no further material updates to provide. We are continuing to monitor the situation. Details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/8471

February 27, 2023 19:59 UTC
[Monitoring] No material updates to provide. We are continuing to monitor. Details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/8471

February 27, 2023 18:52 UTC
[Monitoring] Systems have returned to normal and we are continuing to monitor. We plan to share an update in 1 hour. Details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/8471

February 27, 2023 17:41 UTC
[Monitoring] Our services have recovered. We are continuing to monitor and investigate the root cause. Details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/8471

February 27, 2023 17:18 UTC
[Investigating] Users may encounter degraded performance or intermittent errors on GitLab.com. We are currently investigating. Details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/8471

February 27, 2023 17:16 UTC
[Investigating] Users may encounter degraded performance or intermittent errors on GitLab.com. We are currently investigating. Details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/8471

January 2023

Elevated error rates on CI Runner

January 5, 2023 17:11 UTC

Incident Status

Degraded Performance


Components

CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners for GitLab community contributions


Locations

Google Compute Engine




January 5, 2023 17:11 UTC
[Resolved] Our Engineers have identified the root cause and implemented a fix. This incident has been resolved. More information can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/8204

January 5, 2023 16:59 UTC
[Monitoring] Jobs started after 15:20 UTC until 16:07 UTC may error in reporting their status or job trace.  Retrying the jobs will succeed. Our engineers will continue to monitor. For more details, please check gitlab.com/gitlab-com/gl-infra/production/-/issues/8204

January 5, 2023 16:41 UTC
[Investigating] Starting at 15:20 UTC, we started having elevated error rates for runners trying to write to job logs or status. The error rates have since stabilized and our engineers continue to investigate the root cause. For more details, please check gitlab.com/gitlab-com/gl-infra/production/-/issues/8204

High Error Rate on Select GitLab.com

January 5, 2023 11:24 UTC

Incident Status

Service Disruption


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners for GitLab community contributions, SAML SSO - GitLab SaaS, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, Canary


Locations

Google Compute Engine, Digital Ocean, Zendesk, AWS




January 5, 2023 11:24 UTC
[Resolved] Our Engineers identified the root cause and addressed it with a fix. The incident is now mitigated and users are expected to work with MRs without encountering any errors. For details, check gitlab.com/gitlab-com/gl-infra/production/-/issues/8201

January 5, 2023 09:18 UTC
[Monitoring] Services continue to recover and customers reported successful logins though SAML. However, some users are still facing errors with MRs. Please check the issue for the workaround: gitlab.com/gitlab-com/gl-infra/production/-/issues/8201#note_1228554821

January 5, 2023 08:45 UTC
[Monitoring] The error rates are decreasing, but users might still experience errors when authenticating through SAML and working with MRs. Investigation is still ongoing. For details, check gitlab.com/gitlab-com/gl-infra/production/-/issues/8201

January 5, 2023 08:24 UTC
[Monitoring] We are seeing the error rate decreasing and the services recovering. Our Engineers will keep monitoring the systems. For more details, please check gitlab.com/gitlab-com/gl-infra/production/-/issues/8201

January 5, 2023 08:07 UTC
[Investigating] We are currently seeing a high rate of errors on GitLab.com. Our infrastructure team is currently investigating. More information can be found here: gitlab.com/gitlab-com/gl-infra/production/-/issues/8201

November 2022

GitLab.com is unavailable

November 30, 2022 02:04 UTC

GitLab.com is unavailableDegraded Performance

Incident Status

Degraded Performance


Components

API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners for GitLab community contributions, SAML SSO - GitLab SaaS, Background Processing, Canary


Locations

Google Compute Engine




November 30, 2022 02:04 UTC
[Resolved] GitLab.com fully operational. Root cause: An unexpected deployment behaviour affected the Consul cluster that several of our services depend on (eg. Patroni)

November 30, 2022 01:27 UTC
[Monitoring] Monitoring situation, there was a deployment error for GitLab.com: gitlab.com/gitlab-com/gl-infra/production/-/issues/8097

November 30, 2022 01:10 UTC
[Monitoring] GitLab.com is back up, all systems operational.

November 30, 2022 00:54 UTC
[Investigating] We are currently investigating a GitLab.com service disruption

November 30, 2022 00:48 UTC
[Investigating] We have detected a problem related to GitLab.com services and are actively investigating. More details to follow.

September 2022

GitLab.com maintenance 2022-09-03 11:00 to 14:00 UTC

September 3, 2022 14:16 UTC

Description

We will be performing maintenance on GitLab.com's production database which is anticipated to include a service downtime of up to 180 minutes between 11:00am to 2:00pm UTC on Saturday 2022-09-03. During this time, users will experience complete service disruption. Improving the performance and reliability of GitLab.com has always been a top priority for GitLab. While we continuously make iterative improvements to GitLab and our production architecture, we anticipate making a larger change to improve the scalability and reliability of GitLab.com: performing necessary Operating System maintenance to our database clusters. You can read more about this change in our blog: about.gitlab.com/blog/2022/08/12/upgrading-database-os


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners for GitLab community contributions, SAML SSO - GitLab SaaS, Background Processing, GitLab Customers Portal, Canary


Locations

Google Compute Engine


Schedule

September 3, 2022 11:00 - September 3, 2022 14:00 UTC



September 3, 2022 14:16 UTC
[Update] GitLab.com's database layer maintenance upgrade is complete now, and we're back up and running. We'll be monitoring the platform to ensure all systems are functioning correctly. Thank you for your patience.

September 3, 2022 14:13 UTC
[Update] GitLab.com planned maintenance for the database layer is almost complete. The site is back up and running although we're continuing to verify that all systems are functioning correctly. Thank you for your patience.

September 3, 2022 13:55 UTC
[Update] We are still performing tests to ensure full functionality of the gitlab.com site before bringing it up. Expect full functionality of gitlab.com in about 30mins

September 3, 2022 10:59 UTC
[Update] GitLab.com is now shutting down for planned maintenance to our database layer for the next 3 hours. See you on the other side! More details in about.gitlab.com/blog/2022/08/12/upgrading-database-os

August 31, 2022 16:00 UTC
[Update] Reminder: We will be performing maintenance on GitLab.com's production database which is anticipated to include a complete service downtime of up to 180 minutes between 11:00am to 2:00pm UTC on Saturday 2022-09-03. You can read more in about.gitlab.com/blog/2022/08/12/upgrading-database-os

August 2022

GitLab.com slow to respond

August 26, 2022 06:21 UTC

GitLab.com slow to respondDegraded Performance

Incident Status

Degraded Performance


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners for GitLab community contributions, SAML SSO - GitLab SaaS, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, Canary


Locations

Google Compute Engine, Digital Ocean, Zendesk, AWS




August 26, 2022 06:21 UTC
[Resolved] We are no longer seeing connectivity issues for GitLab.com after a period of monitoring. We therefore believe that this incident has been resolved. Details in : gitlab.com/gitlab-com/gl-infra/production/-/issues/7656

August 26, 2022 05:08 UTC
[Monitoring] All the services seems to have recovered. We will continue to monitor the performance and provide the updates here. More information on the root cause can be found here: gitlab.com/gitlab-com/gl-infra/production/-/issues/7656

August 26, 2022 04:57 UTC
[Investigating] We are still investigating the root cause of this issue. More details here: gitlab.com/gitlab-com/gl-infra/production/-/issues/7656

August 26, 2022 04:55 UTC
[Investigating] We are noticing some slow response issues with GitLab.com.

Some manual pipelines leading to errors

August 18, 2022 19:45 UTC

Incident Status

Degraded Performance


Components

CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners for GitLab community contributions


Locations

Google Compute Engine




August 18, 2022 19:45 UTC
[Resolved] We've successfully deployed the patch, and this incident is now resolved. More information on the root cause can be found here: gitlab.com/gitlab-com/gl-infra/production/-/issues/7618 We will follow up on this incident with a post-mortem review in the future.

August 18, 2022 10:47 UTC
[Identified] Customers using manually triggered pipelines may encounter errors. A patch is being deployed and more details are here: gitlab.com/gitlab-com/gl-infra/production/-/issues/7618

July 2022

Scheduled GitLab.com production database maintenance

July 2, 2022 08:51 UTC

Description

We will be performing maintenance on GitLab.com's production database which is anticipated to include a service downtime of up to 120 minutes between Saturday, 2022-07-02, 06:00am to 08:00am UTC. During this time users will experience complete service disruption. Improving the performance and reliability of GitLab.com has always been a top priority for GitLab. While we continuously make iterative improvements to GitLab and our production architecture, we anticipate making a larger change to improve the scalability and reliability of GitLab.com: We are splitting our single PostgreSQL database into a main and a ci database. You can read more about this change in our blog: about.gitlab.com/blog/2022/06/02/splitting-database-into-main-and-ci


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners for GitLab community contributions, SAML SSO - GitLab SaaS, Background Processing, GitLab Customers Portal


Locations

Google Compute Engine


Schedule

July 2, 2022 05:00 - July 2, 2022 09:00 UTC



July 2, 2022 08:51 UTC
[Update] GitLab.com's database layer maintenance upgrade is complete now, and we're back up and running. We'll be monitoring the platform to ensure all systems are functioning correctly. Thank you for your patience.

July 2, 2022 07:43 UTC
[Update] GitLab.com planned maintenance for the database layer is almost complete. The site is back up and running although we're continuing to verify that all systems are functioning correctly. Thank you for your patience.

July 2, 2022 06:00 UTC
[Update] GitLab.com is now shutting down for planned maintenance for database layer improvements for the next 2 hours. See you on the other side!

July 2, 2022 05:00 UTC
[Update] GitLab.com will undergo maintenance in 1 hour. Please note that any CI jobs that start before the maintenance window but complete during the window period (up to 2 hours) will fail and may need to be started again.

July 1, 2022 05:45 UTC
[Update] Reminder: Tomorrow we will be undergoing scheduled maintenance to our database layer. We expect the GitLab.com site to be unavailable for up to 2 hours. Starting time: 2022-07-02 06:00 UTC.

June 29, 2022 10:01 UTC
[Update] We wanted to clarify that this scheduled maintenance will also impact Pages hosted with GitLab Pages on GitLab.com. Pages will be unavailable for up to 2 hours starting from 2022-07-02 06:00 UTC. We apologize in advance for any inconvenience this may cause.

June 29, 2022 05:50 UTC
[Update] In 3 days time, we will be undergoing some scheduled maintenance to our database layer so we expect the GitLab.com site to be unavailable for up to 2 hours starting from 2022-07-02 06:00 UTC. We apologize in advance for any inconvenience this may cause.





Back to current status