Active Issue

Updated a few seconds ago

Back to current status

Status History

Filter: CI/CD - Hosted runners on Linux (Clear)



February 2025

PG minor upgrade 16.6 in Production

February 15, 2025 07:49 UTC

Description

Scheduled maintenance to patch our DB layer to PostgreSQL 16.6. 2025-02-15 02:00-08:00 UTC. GitLab.com will be available throughout this time, but users may experience errors on requests. We apologize in advance for any inconvenience caused. See <gitlab.com/gitlab-com/gl-infra/production/-/issues/19120>


Components

Website, API, Container Registry, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions, Background Processing


Locations

Google Compute Engine, AWS


Schedule

February 15, 2025 03:00 - February 15, 2025 09:00 UTC



February 15, 2025 07:49 UTC
[Update] Gitlab.com database maintenance was performed, the platform is considered stable. We'll continue to monitor for any performance issues. Thank you for your patience. See <gitlab.com/gitlab-com/gl-infra/production/-/issues/19120>

February 15, 2025 04:07 UTC
[Update] Gitlab.com scheduled maintenance of our database layer has started. GitLab.com should be available during the whole period but we expect availability and performance impact until 2025-02-15 09:00 UTC. See gitlab.com/gitlab-com/gl-infra/production/-/issues/19120

February 15, 2025 03:01 UTC
[Update] Gitlab.com scheduled maintenance of our database layer has started. GitLab.com should be available during the whole period but we expect availability and performance impact until 2025-02-15 09:00 UTC. See gitlab.com/gitlab-com/gl-infra/production/-/issues/19120

February 15, 2025 02:48 UTC
[Update] Scheduled database maintenance will now start at 2025-02-15 03:00 UTC and expected to run to 2025-02-15 09:00 UTC. See gitlab.com/gitlab-com/gl-infra/production/-/issues/19120

February 15, 2025 01:54 UTC
[Update] Due to an ongoing incident we are delaying the maintenance window by 2 hours to investigate the impact. We currently plan to start the maintenance window at 04:00 UTC.

February 14, 2025 01:45 UTC
[Update] We will be undergoing scheduled maintenance of our databases tomorrow. There may be some availability and performance impact between 2025-02-15 02:00 UTC and 2025-02-15 08:00 UTC. See gitlab.com/gitlab-com/gl-infra/production/-/issues/19120

February 12, 2025 01:13 UTC
[Update] We will be undergoing scheduled maintenance of our databases in 3 days. There may be some availability and performance impact between 2025-02-15 02:00 UTC and 2025-02-15 08:00 UTC. See gitlab.com/gitlab-com/gl-infra/production/-/issues/19120

February 8, 2025 08:37 UTC
[Update] Scheduled maintenance in 1 week: Upgrading DB layer to PostgreSQL 16.6 on Feb 15, 02:00-08:00 UTC. GitLab.com will stay up but users may experience errors. See gitlab.com/gitlab-com/gl-infra/production/-/issues/19120

February 1, 2025 02:42 UTC
[Update] Scheduled maintenance in 2 weeks: Upgrading DB layer to PostgreSQL 16.6 on Feb 15, 02:00-08:00 UTC. GitLab.com will stay up but may experience errors. Sorry for any inconvenience. See gitlab.com/gitlab-com/gl-infra/production/-/issues/19120

Pipelines fail with "Unable to create pipeline" error

February 14, 2025 22:13 UTC

Incident Status

Partial Service Disruption


Components

CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions, CI/CD - Self-managed runners


Locations

Google Compute Engine, AWS, Self-Managed Runner Connectivity




February 14, 2025 22:13 UTC
[Resolved] This fix has been successfully deployed and we have confirmed the issue is no longer occurring. We appreciate your patience while we worked to address this. We will mark this incident as resolved. Details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/19310

February 14, 2025 19:43 UTC
[Identified] We expect the revert MR to land in production within the next 3 hours. Impacted users can find a workaround within the incident issue. We will continue to provide updates here as needed. Details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/19310

February 14, 2025 17:06 UTC
[Identified] We expect the revert MR to land in production within the next 6 hours. Additional updates will be posted here as needed. If you are one the subset of users impacted by this change, please see this issue for a workaround and other additional details: gitlab.com/gitlab-com/gl-infra/production/-/issues/19310

February 14, 2025 16:11 UTC
[Identified] The revert MR to address the issue has been merged. We are still waiting for this to land in production. These changes will be re-introduced to the Dependency-Scanning.latest.gitlab-ci.yml template, where breaking changes can be expected. More details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/19310

February 14, 2025 15:25 UTC
[Identified] We have created a Merge Request to revert recent changes to the Dependency-Scanning.gitlab-ci.yml template. Continue following this incident issue for further updates: gitlab.com/gitlab-com/gl-infra/production/-/issues/19310

February 14, 2025 14:52 UTC
[Identified] We are currently experiencing issues with failing GitLab.com pipelines that include the Dependency-Scanning.gitlab-ci.yml template. We have identified the root cause and are preparing to revert the MR. For a workaround and more details, please see: gitlab.com/gitlab-com/gl-infra/production/-/issues/19310

Errors managing Kubernetes agents

February 11, 2025 15:23 UTC

Errors managing Kubernetes agentsPartial Service Disruption

Incident Status

Partial Service Disruption


Components

Website, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions, CI/CD - Self-managed runners


Locations

Google Compute Engine, AWS, Self-Managed Runner Connectivity




February 11, 2025 15:23 UTC
[Resolved] The fix has been successfully deployed. Our monitoring confirms that the service has been restored to normal operation with no further errors observed. Thank you for your patience during this disruption. If you experience any further issues, please contact our Support team.

February 11, 2025 14:39 UTC
[Monitoring] We are currently rolling out a fix to production. Our team is actively monitoring the deployment and its effectiveness. Please continue to follow our status page for the latest updates.

February 11, 2025 10:24 UTC
[Identified] The ongoing incident investigation has been marked as confidential. While the original incident report will no longer be publicly visible, we continue to actively work on the resolution. Service impact remains unchanged. Further updates will be available on the status page - status.gitlab.com

February 11, 2025 09:35 UTC
[Identified] We identified the root cause of the incident, and we are reverting the relevant MR. The estimated time for the fix: around 5 hours from now. More details: gitlab.com/gitlab-com/gl-infra/production/-/issues/19278

February 11, 2025 09:00 UTC
[Investigating] Some users are experiencing errors managing Kubernetes agents. Details can be found in: gitlab.com/gitlab-com/gl-infra/production/-/issues/19278

January 2025

Runner 17.8.0 incorrectly tagged

January 16, 2025 20:46 UTC

Runner 17.8.0 incorrectly taggedPartial Service Disruption

Incident Status

Partial Service Disruption


Components

Website, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions, CI/CD - Self-managed runners


Locations

Google Compute Engine, Self-Managed Runner Connectivity




January 16, 2025 20:46 UTC
[Resolved] GitLab has published a corrected image for GitLab Runner 17.8; the image can be used normally. Further investigations will be documented in gitlab.com/gitlab-com/gl-infra/production/-/issues/19129.

January 16, 2025 20:04 UTC
[Investigating] GitLab is working on uploading a corrected version of the 17.8.0 Runner image. We will provide an update once we have confirmed the update has been deployed.

January 16, 2025 19:38 UTC
[Investigating] GitLab is investigating an issue where the image for Runner 17.8.0 was not tagged correctly when uploaded to the Container Registry. We recommend continuing to use the Runner 17.7 image until the Runner 17.8 image can be tagged correctly.

Accessing gitlab.com results in 500 errors

January 6, 2025 16:42 UTC

Incident Status

Partial Service Disruption


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions, SAML SSO - GitLab SaaS, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, docs.gitlab.com, Canary, Product Analytics - Configurator (Beta), Product Analytics - Collector (Beta), Product Analytics - Background processing (Beta), Product Analytics - Queries (Beta), GitLab Duo


Locations

Google Compute Engine, Digital Ocean, Zendesk, AWS




January 6, 2025 16:42 UTC
[Resolved] We've identified the cause of the errors and the errors have returned to normal. The incident is now resolved. More information on the incident can be found in this issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/19079

January 6, 2025 15:57 UTC
[Monitoring] We've identified the cause of the errors, and the errors have returned to normal. The incident is now mitigated, but we're continuing to monitor for any further impact. More information on the incident can be found in this issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/19079

January 6, 2025 15:27 UTC
[Monitoring] We're continuing to monitor for further errors related to the incident. We see further decreases in the errors.

January 6, 2025 15:17 UTC
[Monitoring] We saw elevated 500 errors on gitlab.com which have then decreased. We're continuing to monitor for further errors

December 2024

Runner registration returning 500 errors

December 11, 2024 13:34 UTC

Incident Status

Degraded Performance


Components

CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions


Locations

Google Compute Engine




December 11, 2024 13:34 UTC
[Resolved] This incident has been resolved and runner registration is now fully operational. More information can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/18984

December 11, 2024 11:47 UTC
[Monitoring] The fix has been deployed to production and we are no longer seeing these errors. We will continue monitoring. More details gitlab.com/gitlab-com/gl-infra/production/-/issues/18984

December 11, 2024 11:06 UTC
[Identified] The deployment to production has started and we are expecting it to complete in approximately 1 hour. More details gitlab.com/gitlab-com/gl-infra/production/-/issues/18984

December 11, 2024 04:56 UTC
[Identified] A fix for this problem has been merged into the codebase - it should become available on GitLab.com in the next 6-7 hours. For more information please see gitlab.com/gitlab-com/gl-infra/production/-/issues/18984

December 11, 2024 04:36 UTC
[Identified] As a workaround is available (using an authentication token to register a new runner) the status of this incident has been downgraded to 'Partial Service Disruption'

December 11, 2024 04:10 UTC
[Identified] Work to fix the deprecated runner registration method continues. Workaround: register runner with an authentication token. More details gitlab.com/gitlab-com/gl-infra/production/-/issues/18984

December 11, 2024 03:39 UTC
[Identified] Work continues to fix the deprecated runner registration method. More details gitlab.com/gitlab-com/gl-infra/production/-/issues/18984 Workaround: register runner with an authentication token.

December 11, 2024 02:59 UTC
[Identified] A workaround has been identified to register a runner using an authentication token: docs.gitlab.com/runner/register/#register-with-a-runner-authentication-token Incident only impacting the deprecated runner registration method.

December 11, 2024 02:50 UTC
[Identified] Work continues for a fix to the identified cause. More details gitlab.com/gitlab-com/gl-infra/production/-/issues/18984

December 11, 2024 02:22 UTC
[Identified] We are seeing 500 errors with runner registrations. We have identified the cause, and are working on a fix. More details gitlab.com/gitlab-com/gl-infra/production/-/issues/18984

Some CI jobs are failing to run due to insufficient permissions

December 5, 2024 22:50 UTC

Incident Status

Partial Service Disruption


Components

CI/CD - Hosted runners on Linux


Locations

Google Compute Engine




December 5, 2024 22:50 UTC
[Resolved] No more reports of this error have been received. We conclude the monitoring period and consider this incident resolved. See gitlab.com/gitlab-com/gl-infra/production/-/issues/18952 for details.

December 5, 2024 22:32 UTC
[Monitoring] We would like to clarify that this problem affected some jobs on GitLab.com groups that used the "Restrict group access by IP address" feature. See gitlab.com/gitlab-com/gl-infra/production/-/issues/18952.

December 5, 2024 21:52 UTC
[Monitoring] Configuration changes have been put in place in our infrastructure to potentially mitigate this issue. We are currently monitoring our logs and user reports for confirmation. See gitlab.com/gitlab-com/gl-infra/production/-/issues/18952

December 5, 2024 21:40 UTC
[Investigating] We believe the cause for CI clone failure is related to Group-level IP restrictions. We're reviewing internal logs to confirm. See gitlab.com/gitlab-com/gl-infra/production/-/issues/18952 for more.

December 5, 2024 21:18 UTC
[Investigating] We see reports of some CI Jobs failing to clone repositories with an "insufficient permissions" error. See gitlab.com/gitlab-com/gl-infra/production/-/issues/18952 for details on the investigation.

GitLab-hosted runners with the gitlab-org-docker tag are offline

December 4, 2024 22:20 UTC

Incident Status

Partial Service Disruption


Components

CI/CD - Hosted runners on Linux


Locations

Google Compute Engine




December 4, 2024 22:20 UTC
[Resolved] This incident is now resolved. Please make sure not to use the "gitlab-org-docker" tag for your workloads as they are intended for gitlab-org projects only. See gitlab.com/gitlab-com/gl-infra/production/-/issues/18945.

December 4, 2024 22:19 UTC
[Monitoring] Runner performance metrics are back to normal levels and jobs are being properly picked up. See gitlab.com/gitlab-com/gl-infra/production/-/issues/18945 for the full details.

December 4, 2024 21:29 UTC
[Monitoring] We have pushed a potential fix and see signs of recovery from the affected Runners. We will continue to monitor this to ensure jobs are properly picked up. See gitlab.com/gitlab-com/gl-infra/production/-/issues/18945.

December 4, 2024 21:10 UTC
[Investigating] We have potentially identified the commit that caused this disruption in the Runner network. Investigation continues. See gitlab.com/gitlab-com/gl-infra/production/-/issues/18945 for details.

December 4, 2024 20:46 UTC
[Investigating] We have found traces of connectivity issues in our Runner network infrastructure. We continue our investigation. See gitlab.com/gitlab-com/gl-infra/production/-/issues/18945 for details.

December 4, 2024 20:10 UTC
[Investigating] We continue to investigate our Runner infrastructure to determine the cause of the issue. Please review gitlab.com/gitlab-com/gl-infra/production/-/issues/18945 for full details.

December 4, 2024 19:53 UTC
[Investigating] The "gitlab-org-docker" tag is meant for gitlab-org projects only and not for customer workloads. As a preliminary potential fix, please remove the tag from your affected jobs and retry them. See: gitlab.com/gitlab-com/gl-infra/production/-/issues/18945

December 4, 2024 19:38 UTC
[Investigating] Jobs tagged with the "gitlab-org-docker" tag are stuck in a "Pending" status as the runners are currently offline. Please see gitlab.com/gitlab-com/gl-infra/production/-/issues/18945 for further details.

November 2024

MAIN DB cluster upgrade

November 5, 2024 11:02 UTC

MAIN DB cluster upgrade Planned Maintenance

Description

Next week, we will be undergoing scheduled maintenance to our MAIN database layer. The maintenance will start at 2024-11-03 06:00 UTC and should finish at 2024-11-05 11:00 UTC (including performance regression observability period). GitLab.com will be available during the whole period as the maintenance should be seamless and transparent for the application. We apologize in advance for any inconvenience this may cause. See <gitlab.com/gitlab-com/gl-infra/production/-/issues/18747>


Components

Website, API, Git Operations, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions


Locations

Google Compute Engine, AWS


Schedule

November 3, 2024 06:00 - November 5, 2024 11:00 UTC



November 5, 2024 11:02 UTC
[Update] Gitlab.com MAIN database upgrade is now complete with all systems are functioning correctly. Thank you for your patience. See gitlab.com/gitlab-com/gl-infra/production/-/issues/18747

November 3, 2024 09:01 UTC
[Update] Gitlab.com MAIN database upgrade was performed. We'll continue to monitor for any performance issues until the end of the maintenance window. Thank you for your patience. See <gitlab.com/gitlab-com/gl-infra/production/-/issues/18747>

November 3, 2024 06:05 UTC
[Update] Gitlab.com scheduled maintenance of our MAIN database layer have started. GitLab.com should be available during the whole period. See <gitlab.com/gitlab-com/gl-infra/production/-/issues/18747>

November 3, 2024 05:00 UTC
[Update] MAIN DB layer maintenance will start in 1h from now at 2024-11-03 06:00 UTC. GitLab.com will remains available. Details: gitlab.com/gitlab-com/gl-infra/production/-/issues/18747

November 2, 2024 05:00 UTC
[Update] Scheduled maintenance on MAIN DB layer: Start: 2024-11-03 06:00 UTC End: 2024-11-05 11:00 UTC (incl. monitoring) GitLab.com remains available. Maintenance should be seamless. We apologize for any inconvenience. Details: gitlab.com/gitlab-com/gl-infra/production/-/issues/18747

October 30, 2024 06:00 UTC
[Update] In three days, we will be undergoing scheduled maintenance to our MAIN database layer. The maintenance will start at 2024-11-03 06:00 UTC and should finish at 2024-11-05 11:00 UTC (including performance and regression observability period). GitLab.com will be available during the whole period as the maintenance should be seamless and transparent for the application. We apologize in advance for any inconvenience this may cause. See: gitlab.com/gitlab-com/gl-infra/production/-/issues/18747

October 2024

CI DB cluster upgrade

October 30, 2024 01:37 UTC

CI DB cluster upgradePlanned Maintenance

Description

Next week, we will be undergoing scheduled maintenance to our CI database layer. The maintenance will start at 2024-10-26 06:00 UTC and should finish at 2024-10-29 11:00 UTC (including performance regression observability period). GitLab.com will be available during the whole period as the maintenance should be seamless and transparent for the application. We apologize in advance for any inconvenience this may cause. See <gitlab.com/gitlab-com/gl-infra/production/-/issues/18639>


Components

Website, API, Git Operations, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions


Locations

Google Compute Engine, AWS


Schedule

October 26, 2024 06:00 - October 29, 2024 11:00 UTC



October 30, 2024 01:37 UTC
[Update] GitLab.com upgrade of the CI database layer is now complete with all systems are functioning correctly. Thank you for your patience. See gitlab.com/gitlab-com/gl-infra/production/-/issues/18639

October 27, 2024 08:47 UTC
[Update] GitLab.com upgrade of the CI database layer is complete. We'll be monitoring the platform to ensure all systems are functioning correctly. Thank you for your patience.

October 26, 2024 06:01 UTC
[Update] We are starting maintenance to the CI database layer, and should finish at 2024-10-29 11:00 UTC (including performance regression and observability period). GitLab.com will be available during the whole period as the maintenance should be seamless. See gitlab.com/gitlab-com/gl-infra/production/-/issues/18639

October 25, 2024 19:03 UTC
[Update] Reminder: Tomorrow, we will be undergoing scheduled maintenance to our CI database layer. The maintenance will start at 2024-10-26 06:00 UTC and should finish at 2024-10-29 11:00 UTC (including performance regression observability period). GitLab.com will be available during the whole period as the maintenance should be seamless and transparent for the application. We apologize in advance for any inconvenience this may cause. See <gitlab.com/gitlab-com/gl-infra/production/-/issues/18639>

October 23, 2024 17:25 UTC
[Update] In 3 days, we will be undergoing scheduled maintenance to our CI database layer. The maintenance will start at 2024-10-26 06:00 UTC and should finish at 2024-10-29 11:00 UTC (including performance regression observability period). GitLab.com will be available during the whole period as the maintenance should be seamless and transparent for the application. We apologize in advance for any inconvenience this may cause. See <gitlab.com/gitlab-com/gl-infra/production/-/issues/18639>

Pipelines using SaaS Runners stuck on Pending status

October 9, 2024 08:38 UTC

Incident Status

Service Disruption


Components

CI/CD - Hosted runners on Linux


Locations

Google Compute Engine




October 9, 2024 08:38 UTC
[Resolved] Pipeline jobs are now being picked up by Runners based on our monitoring. We are marking this incident as resolved. See gitlab.com/gitlab-com/gl-infra/production/-/issues/18679

October 8, 2024 23:31 UTC
[Monitoring] We've discovered this incident also impacted self-managed Runners when attempting to verify their credentials on GitLab.com. 404 errors were generated, preventing job execution. See gitlab.com/gitlab-com/gl-infra/production/-/issues/18679.

October 8, 2024 22:39 UTC
[Monitoring] Next update for this incident will be once the revert MR is deployed to production. Please follow gitlab.com/gitlab-com/gl-infra/production/-/issues/18679 for more details.

October 8, 2024 22:38 UTC
[Monitoring] Pipelines processed by GitLab-hosted runners are back to normal operation after the rollback. A revert MR is in process to be deployed to production which will address the root cause of the issue. See gitlab.com/gitlab-com/gl-infra/production/-/issues/18679.

October 8, 2024 22:02 UTC
[Monitoring] We have performed a deployment rollback and previously stuck pipelines are being picked up. We will start a monitoring window for 30 minutes to confirm incident resolution. See gitlab.com/gitlab-com/gl-infra/production/-/issues/18679.

October 8, 2024 21:44 UTC
[Identified] We are receiving reports of CI Pipelines failing to start and stuck on Pending status. This is only affecting pipelines using GitLab-hosted runners. A fix has already been identified and should be deployed to production soon. Please see gitlab.com/gitlab-com/gl-infra/production/-/issues/18679.

September 2024

Main DB Upgrade

September 26, 2024 03:54 UTC

Main DB UpgradePlanned Maintenance

Description

We will be undergoing scheduled maintenance to our database layer. GitLab.com will continue to be available during the maintenance window. For more details, see gitlab.com/gitlab-com/gl-infra/dbre/-/issues/227


Components

Website, API, Git Operations, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions


Locations

Google Compute Engine, AWS


Schedule

October 25, 2024 00:00 - October 30, 2024 01:00 UTC



September 26, 2024 03:54 UTC
[Update] Scheduled maintenance on October 25 to our database layer has now been cancelled. For more details, see gitlab.com/gitlab-com/gl-infra/dbre/-/issues/227

CI DB upgrade

September 26, 2024 03:50 UTC

CI DB upgradePlanned Maintenance

Description

Next week, we will be undergoing scheduled maintenance to our CI database layer. The maintenance will start at 2024-09-29 00:00 UTC and should finish at 2024-10-02 02:00 UTC (including performance regression observability period). GitLab.com will be available during the whole period as the maintenance should be seamless and transparent for the application. We apologize in advance for any inconvenience this may cause. See <gitlab.com/gitlab-com/gl-infra/production/-/issues/18593>


Components

Website, API, Git Operations, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions


Locations

Google Compute Engine, AWS


Schedule

September 27, 2024 00:00 - October 2, 2024 01:00 UTC



September 26, 2024 03:50 UTC
[Update] Maintenance scheduled for 2024-09-29 on our CI database layer, has now been cancelled. For more information: gitlab.com/gitlab-com/gl-infra/production/-/issues/18593

"View job currently using resource" button returns 500 error

September 20, 2024 19:10 UTC

Incident Status

Degraded Performance


Components

CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions


Locations

Google Compute Engine, AWS




September 20, 2024 19:10 UTC
[Resolved] GitLab has verified we are no longer seeing 500 errors from the View Job button. We are resolving the incident. Any further updates will be posted in the incident issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/18530

September 17, 2024 02:35 UTC
[Monitoring] A fix has been deployed and is being verified, further updates will be provided upon completion. Details available in: gitlab.com/gitlab-com/gl-infra/production/-/issues/18530

September 12, 2024 20:47 UTC
[Identified] The fix is known and will be deployed soon. There are some existing pipelines that are still affected, and our team is working on resolving them. Affected users can view the open issue here: gitlab.com/gitlab-com/gl-infra/production/-/issues/18530

September 11, 2024 20:26 UTC
[Identified] No updates at this time. There are some existing pipelines that are still affected, and our team is working on resolving them. Affected users can view the open issue here: gitlab.com/gitlab-com/gl-infra/production/-/issues/18530

September 11, 2024 17:40 UTC
[Identified] A fix has been deployed to production. New pipelines are unaffected. However, there are some existing pipelines that are still affected, and our team is working on resolving them. Affected users can view the open issue here: gitlab.com/gitlab-com/gl-infra/production/-/issues/18530

September 11, 2024 09:32 UTC
[Resolved] This incident has been resolved. Affected users can view the open issue here: gitlab.com/gitlab-com/gl-infra/production/-/issues/18530 If you have CI pipelines that are stuck as a result of this issue, please cancel them so they don't continue to use up your CI minutes quota. We’re actively looking into remediating CI minutes usage for the impacted jobs. If you believe you are impacted by this incident, please open a support ticket here: support.gitlab.com

September 10, 2024 19:33 UTC
[Identified] A fix has been deployed to production. New pipelines are unaffected. However, there are some existing pipelines that are still affected, and our team is working on resolving them. Affected users can view the open issue here: gitlab.com/gitlab-org/gitlab/-/issues/483290 If you continue to see the issue, please contact support here: support.gitlab.com

September 9, 2024 22:25 UTC
[Identified] At this time there are no further updates. Affected users can view the open issue here: gitlab.com/gitlab-org/gitlab/-/issues/483290 If you deleted a pipeline that had a resource group and are now seeing stuck jobs, please contact support.

September 9, 2024 20:56 UTC
[Identified] Some users are seeing 500 errors when clicking the "View job currently using resource" button. Users may also see jobs stuck in "canceling" state after deleting a pipeline that was deploying to a resource group. The fix is already in the works. Affected users can view the open issue here: gitlab.com/gitlab-org/gitlab/-/issues/483290 If you deleted a pipeline that had a resource group and are now seeing stuck jobs, please contact support.

Registry DB upgrade

September 11, 2024 22:51 UTC

Registry DB upgradePlanned Maintenance

Description

We will be undergoing scheduled maintenance to our database layer. GitLab.com will continue to be available during the maintenance window. For more details, see gitlab.com/gitlab-com/gl-infra/dbre/-/issues/225


Components

Website, API, Git Operations, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions


Locations

Google Compute Engine, AWS


Schedule

September 6, 2024 00:00 - September 11, 2024 01:00 UTC



September 11, 2024 22:51 UTC
[Update] Planned maintenance for the registry database layer on GitLab.com is complete. Registry database is now running Postgres v16. The rollback window for the Registry database was closed on approximately 2024-09-12 1:00 UTC.

September 8, 2024 01:51 UTC
[Update] Planned maintenance for the registry database layer on GitLab.com is complete. Registry database is now running Postgres v16. Rollback window for this planned maintenance ends 2024-09-11 00:00 UTC. We'll be monitoring the platform during this time to ensure all systems are functioning correctly.

September 8, 2024 00:11 UTC
[Update] Scheduled maintenance to our registry database layer has started. The maintenance should finish at 2024-09-11 02:00 UTC UTC (including performance regression observability period). GitLab.com will be available during the whole period as the maintenance should be seamless and transparent for the application. We apologize in advance for any inconvenience this may cause. See issue for more information: gitlab.com/gitlab-com/gl-infra/production/-/issues/18447

September 5, 2024 05:53 UTC
[Update] In 3 days, we will be undergoing scheduled maintenance to our registry database layer. The maintenance will start at 2024-09-08 00:00 UTC UTC and should finish at 2024-09-11 02:00 UTC UTC (including performance regression observability period). GitLab.com will be available during the whole period as the maintenance should be seamless and transparent for the application. We apologize in advance for any inconvenience this may cause. See <gitlab.com/gitlab-com/gl-infra/production/-/issues/18447>

Some jobs to running longer than the defined timeout

September 11, 2024 09:37 UTC

Incident Status

Operational


Components

CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions


Locations

Google Compute Engine, AWS




September 11, 2024 09:37 UTC
[Resolved] This incident has been resolved. Affected users can view the open issue here: gitlab.com/gitlab-com/gl-infra/production/-/issues/18533 If you have CI pipelines that are stuck as a result of this issue, please cancel them so they don't continue to use up your CI minutes quota. We’re actively looking into remediating CI minutes usage for the impacted jobs. If you believe you are impacted by this incident, please open a support ticket here: support.gitlab.com

September 10, 2024 22:28 UTC
[Monitoring] An incident earlier today caused some jobs to run longer than the defined timeout. This lead to excessive usage of CI minutes. Affected users can view the open issue here: gitlab.com/gitlab-com/gl-infra/production/-/issues/18533 If you have CI pipelines that are stuck as a result of this issue, please cancel them so they don't continue to use up your CI minutes quota. We’re actively looking into remediating CI minutes usage for the impacted jobs. If you believe you are impacted by this incident, please open a support ticket here: support.gitlab.com

September 10, 2024 21:48 UTC
[Monitoring] An incident earlier today caused some jobs to run longer than the defined timeout. This lead to excessive usage of CI minutes. Affected users can view the open issue here: gitlab.com/gitlab-com/gl-infra/production/-/issues/18533 We’re actively looking into remediating CI minutes usage for the impacted jobs. If you believe you are impacted by this incident, please open a support ticket. support.gitlab.com

GitLab connectivity issues

September 10, 2024 13:40 UTC

Incident Status

Operational


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions


Locations

Google Compute Engine




September 10, 2024 13:40 UTC
[Resolved] This incident has been resolved. More information can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/18535.

September 10, 2024 13:33 UTC
[Monitoring] The website has recovered are and we are now monitoring before marking as resolved. More details can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/18535.

September 10, 2024 13:24 UTC
[Investigating] No material updates at this time - our investigation is ongoing. More details can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/18535

September 10, 2024 13:10 UTC
[Investigating]

September 10, 2024 13:08 UTC
[Investigating] We are currently investigating connectivity issues to GitLab.com. For more information see gitlab.com/gitlab-com/gl-infra/production/-/issues/18535

GitLab.com Connectivity Issues

September 3, 2024 05:34 UTC

Incident Status

Service Disruption


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions, SAML SSO - GitLab SaaS, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, docs.gitlab.com, Canary, Product Analytics - Configurator (Beta), Product Analytics - Collector (Beta), Product Analytics - Background processing (Beta), Product Analytics - Queries (Beta), GitLab Duo


Locations

Google Compute Engine, Digital Ocean, Zendesk, AWS




September 3, 2024 05:34 UTC
[Resolved] All GitLab services have been restored. We are actively monitoring the system, and everything appears normal at this time. For more details, please follow the issue at: gitlab.com/gitlab-com/gl-infra/production/-/issues/18490

September 3, 2024 05:19 UTC
[Monitoring] We've identified and addressed the root cause of the issue. All GitLab services have been restored, and we are actively monitoring the system. Follow the issue for more details : gitlab.com/gitlab-com/gl-infra/production/-/issues/18490

September 3, 2024 05:09 UTC
[Investigating] GitLab.com is currently experiencing connectivity issues due to a high error rate. Our infrastructure team is investigating. Issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/18490

August 2024

Embedding DB Upgrade

August 31, 2024 04:58 UTC

Embedding DB UpgradePlanned Maintenance

Description

We will be undergoing scheduled maintenance to our database layer. GitLab.com will continue to be available during the maintenance window. For more details, see gitlab.com/gitlab-com/gl-infra/dbre/-/issues/228


Components

Website, API, Git Operations, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions


Locations

Google Compute Engine, AWS


Schedule

October 25, 2024 00:00 - October 30, 2024 01:00 UTC



August 31, 2024 04:58 UTC
[Update] We are cancelling the Embedding database upgrade. See gitlab.com/gitlab-com/gl-infra/dbre/-/issues/228 for details. All other DB upgrade schedules will go ahead as planned.

CI/CD token allowlist selection failing

August 27, 2024 14:52 UTC

CI/CD token allowlist selection failingPartial Service Disruption

Incident Status

Partial Service Disruption


Components

Website, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions


Locations

Google Compute Engine, AWS




August 27, 2024 14:52 UTC
[Resolved] The revert MR has been deployed to production, and we see that CI/CD token allowlist project selection renders properly now. Details will be provided in the incident issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/18446

August 27, 2024 03:55 UTC
[Identified] We have identified an issue that is causing CI/CD token allowlist project selection to fail. We are in the process of reverting the change that caused this. For more details see gitlab.com/gitlab-com/gl-infra/production/-/issues/18446.





Back to current status