Active Issue

Updated a few seconds ago

Back to current status

Status History

Filter: CI/CD - Hosted runners on macOS (Clear)



February 2025

PG minor upgrade 16.6 in Production

February 15, 2025 07:49 UTC

Description

Scheduled maintenance to patch our DB layer to PostgreSQL 16.6. 2025-02-15 02:00-08:00 UTC. GitLab.com will be available throughout this time, but users may experience errors on requests. We apologize in advance for any inconvenience caused. See <gitlab.com/gitlab-com/gl-infra/production/-/issues/19120>


Components

Website, API, Container Registry, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions, Background Processing


Locations

Google Compute Engine, AWS


Schedule

February 15, 2025 03:00 - February 15, 2025 09:00 UTC



February 15, 2025 07:49 UTC
[Update] Gitlab.com database maintenance was performed, the platform is considered stable. We'll continue to monitor for any performance issues. Thank you for your patience. See <gitlab.com/gitlab-com/gl-infra/production/-/issues/19120>

February 15, 2025 04:07 UTC
[Update] Gitlab.com scheduled maintenance of our database layer has started. GitLab.com should be available during the whole period but we expect availability and performance impact until 2025-02-15 09:00 UTC. See gitlab.com/gitlab-com/gl-infra/production/-/issues/19120

February 15, 2025 03:01 UTC
[Update] Gitlab.com scheduled maintenance of our database layer has started. GitLab.com should be available during the whole period but we expect availability and performance impact until 2025-02-15 09:00 UTC. See gitlab.com/gitlab-com/gl-infra/production/-/issues/19120

February 15, 2025 02:48 UTC
[Update] Scheduled database maintenance will now start at 2025-02-15 03:00 UTC and expected to run to 2025-02-15 09:00 UTC. See gitlab.com/gitlab-com/gl-infra/production/-/issues/19120

February 15, 2025 01:54 UTC
[Update] Due to an ongoing incident we are delaying the maintenance window by 2 hours to investigate the impact. We currently plan to start the maintenance window at 04:00 UTC.

February 14, 2025 01:45 UTC
[Update] We will be undergoing scheduled maintenance of our databases tomorrow. There may be some availability and performance impact between 2025-02-15 02:00 UTC and 2025-02-15 08:00 UTC. See gitlab.com/gitlab-com/gl-infra/production/-/issues/19120

February 12, 2025 01:13 UTC
[Update] We will be undergoing scheduled maintenance of our databases in 3 days. There may be some availability and performance impact between 2025-02-15 02:00 UTC and 2025-02-15 08:00 UTC. See gitlab.com/gitlab-com/gl-infra/production/-/issues/19120

February 8, 2025 08:37 UTC
[Update] Scheduled maintenance in 1 week: Upgrading DB layer to PostgreSQL 16.6 on Feb 15, 02:00-08:00 UTC. GitLab.com will stay up but users may experience errors. See gitlab.com/gitlab-com/gl-infra/production/-/issues/19120

February 1, 2025 02:42 UTC
[Update] Scheduled maintenance in 2 weeks: Upgrading DB layer to PostgreSQL 16.6 on Feb 15, 02:00-08:00 UTC. GitLab.com will stay up but may experience errors. Sorry for any inconvenience. See gitlab.com/gitlab-com/gl-infra/production/-/issues/19120

Pipelines fail with "Unable to create pipeline" error

February 14, 2025 22:13 UTC

Incident Status

Partial Service Disruption


Components

CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions, CI/CD - Self-managed runners


Locations

Google Compute Engine, AWS, Self-Managed Runner Connectivity




February 14, 2025 22:13 UTC
[Resolved] This fix has been successfully deployed and we have confirmed the issue is no longer occurring. We appreciate your patience while we worked to address this. We will mark this incident as resolved. Details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/19310

February 14, 2025 19:43 UTC
[Identified] We expect the revert MR to land in production within the next 3 hours. Impacted users can find a workaround within the incident issue. We will continue to provide updates here as needed. Details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/19310

February 14, 2025 17:06 UTC
[Identified] We expect the revert MR to land in production within the next 6 hours. Additional updates will be posted here as needed. If you are one the subset of users impacted by this change, please see this issue for a workaround and other additional details: gitlab.com/gitlab-com/gl-infra/production/-/issues/19310

February 14, 2025 16:11 UTC
[Identified] The revert MR to address the issue has been merged. We are still waiting for this to land in production. These changes will be re-introduced to the Dependency-Scanning.latest.gitlab-ci.yml template, where breaking changes can be expected. More details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/19310

February 14, 2025 15:25 UTC
[Identified] We have created a Merge Request to revert recent changes to the Dependency-Scanning.gitlab-ci.yml template. Continue following this incident issue for further updates: gitlab.com/gitlab-com/gl-infra/production/-/issues/19310

February 14, 2025 14:52 UTC
[Identified] We are currently experiencing issues with failing GitLab.com pipelines that include the Dependency-Scanning.gitlab-ci.yml template. We have identified the root cause and are preparing to revert the MR. For a workaround and more details, please see: gitlab.com/gitlab-com/gl-infra/production/-/issues/19310

Errors managing Kubernetes agents

February 11, 2025 15:23 UTC

Errors managing Kubernetes agentsPartial Service Disruption

Incident Status

Partial Service Disruption


Components

Website, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions, CI/CD - Self-managed runners


Locations

Google Compute Engine, AWS, Self-Managed Runner Connectivity




February 11, 2025 15:23 UTC
[Resolved] The fix has been successfully deployed. Our monitoring confirms that the service has been restored to normal operation with no further errors observed. Thank you for your patience during this disruption. If you experience any further issues, please contact our Support team.

February 11, 2025 14:39 UTC
[Monitoring] We are currently rolling out a fix to production. Our team is actively monitoring the deployment and its effectiveness. Please continue to follow our status page for the latest updates.

February 11, 2025 10:24 UTC
[Identified] The ongoing incident investigation has been marked as confidential. While the original incident report will no longer be publicly visible, we continue to actively work on the resolution. Service impact remains unchanged. Further updates will be available on the status page - status.gitlab.com

February 11, 2025 09:35 UTC
[Identified] We identified the root cause of the incident, and we are reverting the relevant MR. The estimated time for the fix: around 5 hours from now. More details: gitlab.com/gitlab-com/gl-infra/production/-/issues/19278

February 11, 2025 09:00 UTC
[Investigating] Some users are experiencing errors managing Kubernetes agents. Details can be found in: gitlab.com/gitlab-com/gl-infra/production/-/issues/19278

January 2025

Runner 17.8.0 incorrectly tagged

January 16, 2025 20:46 UTC

Runner 17.8.0 incorrectly taggedPartial Service Disruption

Incident Status

Partial Service Disruption


Components

Website, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions, CI/CD - Self-managed runners


Locations

Google Compute Engine, Self-Managed Runner Connectivity




January 16, 2025 20:46 UTC
[Resolved] GitLab has published a corrected image for GitLab Runner 17.8; the image can be used normally. Further investigations will be documented in gitlab.com/gitlab-com/gl-infra/production/-/issues/19129.

January 16, 2025 20:04 UTC
[Investigating] GitLab is working on uploading a corrected version of the 17.8.0 Runner image. We will provide an update once we have confirmed the update has been deployed.

January 16, 2025 19:38 UTC
[Investigating] GitLab is investigating an issue where the image for Runner 17.8.0 was not tagged correctly when uploaded to the Container Registry. We recommend continuing to use the Runner 17.7 image until the Runner 17.8 image can be tagged correctly.

Accessing gitlab.com results in 500 errors

January 6, 2025 16:42 UTC

Incident Status

Partial Service Disruption


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions, SAML SSO - GitLab SaaS, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, docs.gitlab.com, Canary, Product Analytics - Configurator (Beta), Product Analytics - Collector (Beta), Product Analytics - Background processing (Beta), Product Analytics - Queries (Beta), GitLab Duo


Locations

Google Compute Engine, Digital Ocean, Zendesk, AWS




January 6, 2025 16:42 UTC
[Resolved] We've identified the cause of the errors and the errors have returned to normal. The incident is now resolved. More information on the incident can be found in this issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/19079

January 6, 2025 15:57 UTC
[Monitoring] We've identified the cause of the errors, and the errors have returned to normal. The incident is now mitigated, but we're continuing to monitor for any further impact. More information on the incident can be found in this issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/19079

January 6, 2025 15:27 UTC
[Monitoring] We're continuing to monitor for further errors related to the incident. We see further decreases in the errors.

January 6, 2025 15:17 UTC
[Monitoring] We saw elevated 500 errors on gitlab.com which have then decreased. We're continuing to monitor for further errors

December 2024

Runner registration returning 500 errors

December 11, 2024 13:34 UTC

Incident Status

Degraded Performance


Components

CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions


Locations

Google Compute Engine




December 11, 2024 13:34 UTC
[Resolved] This incident has been resolved and runner registration is now fully operational. More information can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/18984

December 11, 2024 11:47 UTC
[Monitoring] The fix has been deployed to production and we are no longer seeing these errors. We will continue monitoring. More details gitlab.com/gitlab-com/gl-infra/production/-/issues/18984

December 11, 2024 11:06 UTC
[Identified] The deployment to production has started and we are expecting it to complete in approximately 1 hour. More details gitlab.com/gitlab-com/gl-infra/production/-/issues/18984

December 11, 2024 04:56 UTC
[Identified] A fix for this problem has been merged into the codebase - it should become available on GitLab.com in the next 6-7 hours. For more information please see gitlab.com/gitlab-com/gl-infra/production/-/issues/18984

December 11, 2024 04:36 UTC
[Identified] As a workaround is available (using an authentication token to register a new runner) the status of this incident has been downgraded to 'Partial Service Disruption'

December 11, 2024 04:10 UTC
[Identified] Work to fix the deprecated runner registration method continues. Workaround: register runner with an authentication token. More details gitlab.com/gitlab-com/gl-infra/production/-/issues/18984

December 11, 2024 03:39 UTC
[Identified] Work continues to fix the deprecated runner registration method. More details gitlab.com/gitlab-com/gl-infra/production/-/issues/18984 Workaround: register runner with an authentication token.

December 11, 2024 02:59 UTC
[Identified] A workaround has been identified to register a runner using an authentication token: docs.gitlab.com/runner/register/#register-with-a-runner-authentication-token Incident only impacting the deprecated runner registration method.

December 11, 2024 02:50 UTC
[Identified] Work continues for a fix to the identified cause. More details gitlab.com/gitlab-com/gl-infra/production/-/issues/18984

December 11, 2024 02:22 UTC
[Identified] We are seeing 500 errors with runner registrations. We have identified the cause, and are working on a fix. More details gitlab.com/gitlab-com/gl-infra/production/-/issues/18984

November 2024

MAIN DB cluster upgrade

November 5, 2024 11:02 UTC

MAIN DB cluster upgrade Planned Maintenance

Description

Next week, we will be undergoing scheduled maintenance to our MAIN database layer. The maintenance will start at 2024-11-03 06:00 UTC and should finish at 2024-11-05 11:00 UTC (including performance regression observability period). GitLab.com will be available during the whole period as the maintenance should be seamless and transparent for the application. We apologize in advance for any inconvenience this may cause. See <gitlab.com/gitlab-com/gl-infra/production/-/issues/18747>


Components

Website, API, Git Operations, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions


Locations

Google Compute Engine, AWS


Schedule

November 3, 2024 06:00 - November 5, 2024 11:00 UTC



November 5, 2024 11:02 UTC
[Update] Gitlab.com MAIN database upgrade is now complete with all systems are functioning correctly. Thank you for your patience. See gitlab.com/gitlab-com/gl-infra/production/-/issues/18747

November 3, 2024 09:01 UTC
[Update] Gitlab.com MAIN database upgrade was performed. We'll continue to monitor for any performance issues until the end of the maintenance window. Thank you for your patience. See <gitlab.com/gitlab-com/gl-infra/production/-/issues/18747>

November 3, 2024 06:05 UTC
[Update] Gitlab.com scheduled maintenance of our MAIN database layer have started. GitLab.com should be available during the whole period. See <gitlab.com/gitlab-com/gl-infra/production/-/issues/18747>

November 3, 2024 05:00 UTC
[Update] MAIN DB layer maintenance will start in 1h from now at 2024-11-03 06:00 UTC. GitLab.com will remains available. Details: gitlab.com/gitlab-com/gl-infra/production/-/issues/18747

November 2, 2024 05:00 UTC
[Update] Scheduled maintenance on MAIN DB layer: Start: 2024-11-03 06:00 UTC End: 2024-11-05 11:00 UTC (incl. monitoring) GitLab.com remains available. Maintenance should be seamless. We apologize for any inconvenience. Details: gitlab.com/gitlab-com/gl-infra/production/-/issues/18747

October 30, 2024 06:00 UTC
[Update] In three days, we will be undergoing scheduled maintenance to our MAIN database layer. The maintenance will start at 2024-11-03 06:00 UTC and should finish at 2024-11-05 11:00 UTC (including performance and regression observability period). GitLab.com will be available during the whole period as the maintenance should be seamless and transparent for the application. We apologize in advance for any inconvenience this may cause. See: gitlab.com/gitlab-com/gl-infra/production/-/issues/18747

October 2024

CI DB cluster upgrade

October 30, 2024 01:37 UTC

CI DB cluster upgradePlanned Maintenance

Description

Next week, we will be undergoing scheduled maintenance to our CI database layer. The maintenance will start at 2024-10-26 06:00 UTC and should finish at 2024-10-29 11:00 UTC (including performance regression observability period). GitLab.com will be available during the whole period as the maintenance should be seamless and transparent for the application. We apologize in advance for any inconvenience this may cause. See <gitlab.com/gitlab-com/gl-infra/production/-/issues/18639>


Components

Website, API, Git Operations, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions


Locations

Google Compute Engine, AWS


Schedule

October 26, 2024 06:00 - October 29, 2024 11:00 UTC



October 30, 2024 01:37 UTC
[Update] GitLab.com upgrade of the CI database layer is now complete with all systems are functioning correctly. Thank you for your patience. See gitlab.com/gitlab-com/gl-infra/production/-/issues/18639

October 27, 2024 08:47 UTC
[Update] GitLab.com upgrade of the CI database layer is complete. We'll be monitoring the platform to ensure all systems are functioning correctly. Thank you for your patience.

October 26, 2024 06:01 UTC
[Update] We are starting maintenance to the CI database layer, and should finish at 2024-10-29 11:00 UTC (including performance regression and observability period). GitLab.com will be available during the whole period as the maintenance should be seamless. See gitlab.com/gitlab-com/gl-infra/production/-/issues/18639

October 25, 2024 19:03 UTC
[Update] Reminder: Tomorrow, we will be undergoing scheduled maintenance to our CI database layer. The maintenance will start at 2024-10-26 06:00 UTC and should finish at 2024-10-29 11:00 UTC (including performance regression observability period). GitLab.com will be available during the whole period as the maintenance should be seamless and transparent for the application. We apologize in advance for any inconvenience this may cause. See <gitlab.com/gitlab-com/gl-infra/production/-/issues/18639>

October 23, 2024 17:25 UTC
[Update] In 3 days, we will be undergoing scheduled maintenance to our CI database layer. The maintenance will start at 2024-10-26 06:00 UTC and should finish at 2024-10-29 11:00 UTC (including performance regression observability period). GitLab.com will be available during the whole period as the maintenance should be seamless and transparent for the application. We apologize in advance for any inconvenience this may cause. See <gitlab.com/gitlab-com/gl-infra/production/-/issues/18639>

September 2024

Main DB Upgrade

September 26, 2024 03:54 UTC

Main DB UpgradePlanned Maintenance

Description

We will be undergoing scheduled maintenance to our database layer. GitLab.com will continue to be available during the maintenance window. For more details, see gitlab.com/gitlab-com/gl-infra/dbre/-/issues/227


Components

Website, API, Git Operations, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions


Locations

Google Compute Engine, AWS


Schedule

October 25, 2024 00:00 - October 30, 2024 01:00 UTC



September 26, 2024 03:54 UTC
[Update] Scheduled maintenance on October 25 to our database layer has now been cancelled. For more details, see gitlab.com/gitlab-com/gl-infra/dbre/-/issues/227

CI DB upgrade

September 26, 2024 03:50 UTC

CI DB upgradePlanned Maintenance

Description

Next week, we will be undergoing scheduled maintenance to our CI database layer. The maintenance will start at 2024-09-29 00:00 UTC and should finish at 2024-10-02 02:00 UTC (including performance regression observability period). GitLab.com will be available during the whole period as the maintenance should be seamless and transparent for the application. We apologize in advance for any inconvenience this may cause. See <gitlab.com/gitlab-com/gl-infra/production/-/issues/18593>


Components

Website, API, Git Operations, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions


Locations

Google Compute Engine, AWS


Schedule

September 27, 2024 00:00 - October 2, 2024 01:00 UTC



September 26, 2024 03:50 UTC
[Update] Maintenance scheduled for 2024-09-29 on our CI database layer, has now been cancelled. For more information: gitlab.com/gitlab-com/gl-infra/production/-/issues/18593

"View job currently using resource" button returns 500 error

September 20, 2024 19:10 UTC

Incident Status

Degraded Performance


Components

CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions


Locations

Google Compute Engine, AWS




September 20, 2024 19:10 UTC
[Resolved] GitLab has verified we are no longer seeing 500 errors from the View Job button. We are resolving the incident. Any further updates will be posted in the incident issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/18530

September 17, 2024 02:35 UTC
[Monitoring] A fix has been deployed and is being verified, further updates will be provided upon completion. Details available in: gitlab.com/gitlab-com/gl-infra/production/-/issues/18530

September 12, 2024 20:47 UTC
[Identified] The fix is known and will be deployed soon. There are some existing pipelines that are still affected, and our team is working on resolving them. Affected users can view the open issue here: gitlab.com/gitlab-com/gl-infra/production/-/issues/18530

September 11, 2024 20:26 UTC
[Identified] No updates at this time. There are some existing pipelines that are still affected, and our team is working on resolving them. Affected users can view the open issue here: gitlab.com/gitlab-com/gl-infra/production/-/issues/18530

September 11, 2024 17:40 UTC
[Identified] A fix has been deployed to production. New pipelines are unaffected. However, there are some existing pipelines that are still affected, and our team is working on resolving them. Affected users can view the open issue here: gitlab.com/gitlab-com/gl-infra/production/-/issues/18530

September 11, 2024 09:32 UTC
[Resolved] This incident has been resolved. Affected users can view the open issue here: gitlab.com/gitlab-com/gl-infra/production/-/issues/18530 If you have CI pipelines that are stuck as a result of this issue, please cancel them so they don't continue to use up your CI minutes quota. We’re actively looking into remediating CI minutes usage for the impacted jobs. If you believe you are impacted by this incident, please open a support ticket here: support.gitlab.com

September 10, 2024 19:33 UTC
[Identified] A fix has been deployed to production. New pipelines are unaffected. However, there are some existing pipelines that are still affected, and our team is working on resolving them. Affected users can view the open issue here: gitlab.com/gitlab-org/gitlab/-/issues/483290 If you continue to see the issue, please contact support here: support.gitlab.com

September 9, 2024 22:25 UTC
[Identified] At this time there are no further updates. Affected users can view the open issue here: gitlab.com/gitlab-org/gitlab/-/issues/483290 If you deleted a pipeline that had a resource group and are now seeing stuck jobs, please contact support.

September 9, 2024 20:56 UTC
[Identified] Some users are seeing 500 errors when clicking the "View job currently using resource" button. Users may also see jobs stuck in "canceling" state after deleting a pipeline that was deploying to a resource group. The fix is already in the works. Affected users can view the open issue here: gitlab.com/gitlab-org/gitlab/-/issues/483290 If you deleted a pipeline that had a resource group and are now seeing stuck jobs, please contact support.

Registry DB upgrade

September 11, 2024 22:51 UTC

Registry DB upgradePlanned Maintenance

Description

We will be undergoing scheduled maintenance to our database layer. GitLab.com will continue to be available during the maintenance window. For more details, see gitlab.com/gitlab-com/gl-infra/dbre/-/issues/225


Components

Website, API, Git Operations, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions


Locations

Google Compute Engine, AWS


Schedule

September 6, 2024 00:00 - September 11, 2024 01:00 UTC



September 11, 2024 22:51 UTC
[Update] Planned maintenance for the registry database layer on GitLab.com is complete. Registry database is now running Postgres v16. The rollback window for the Registry database was closed on approximately 2024-09-12 1:00 UTC.

September 8, 2024 01:51 UTC
[Update] Planned maintenance for the registry database layer on GitLab.com is complete. Registry database is now running Postgres v16. Rollback window for this planned maintenance ends 2024-09-11 00:00 UTC. We'll be monitoring the platform during this time to ensure all systems are functioning correctly.

September 8, 2024 00:11 UTC
[Update] Scheduled maintenance to our registry database layer has started. The maintenance should finish at 2024-09-11 02:00 UTC UTC (including performance regression observability period). GitLab.com will be available during the whole period as the maintenance should be seamless and transparent for the application. We apologize in advance for any inconvenience this may cause. See issue for more information: gitlab.com/gitlab-com/gl-infra/production/-/issues/18447

September 5, 2024 05:53 UTC
[Update] In 3 days, we will be undergoing scheduled maintenance to our registry database layer. The maintenance will start at 2024-09-08 00:00 UTC UTC and should finish at 2024-09-11 02:00 UTC UTC (including performance regression observability period). GitLab.com will be available during the whole period as the maintenance should be seamless and transparent for the application. We apologize in advance for any inconvenience this may cause. See <gitlab.com/gitlab-com/gl-infra/production/-/issues/18447>

Some jobs to running longer than the defined timeout

September 11, 2024 09:37 UTC

Incident Status

Operational


Components

CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions


Locations

Google Compute Engine, AWS




September 11, 2024 09:37 UTC
[Resolved] This incident has been resolved. Affected users can view the open issue here: gitlab.com/gitlab-com/gl-infra/production/-/issues/18533 If you have CI pipelines that are stuck as a result of this issue, please cancel them so they don't continue to use up your CI minutes quota. We’re actively looking into remediating CI minutes usage for the impacted jobs. If you believe you are impacted by this incident, please open a support ticket here: support.gitlab.com

September 10, 2024 22:28 UTC
[Monitoring] An incident earlier today caused some jobs to run longer than the defined timeout. This lead to excessive usage of CI minutes. Affected users can view the open issue here: gitlab.com/gitlab-com/gl-infra/production/-/issues/18533 If you have CI pipelines that are stuck as a result of this issue, please cancel them so they don't continue to use up your CI minutes quota. We’re actively looking into remediating CI minutes usage for the impacted jobs. If you believe you are impacted by this incident, please open a support ticket here: support.gitlab.com

September 10, 2024 21:48 UTC
[Monitoring] An incident earlier today caused some jobs to run longer than the defined timeout. This lead to excessive usage of CI minutes. Affected users can view the open issue here: gitlab.com/gitlab-com/gl-infra/production/-/issues/18533 We’re actively looking into remediating CI minutes usage for the impacted jobs. If you believe you are impacted by this incident, please open a support ticket. support.gitlab.com

GitLab connectivity issues

September 10, 2024 13:40 UTC

Incident Status

Operational


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions


Locations

Google Compute Engine




September 10, 2024 13:40 UTC
[Resolved] This incident has been resolved. More information can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/18535.

September 10, 2024 13:33 UTC
[Monitoring] The website has recovered are and we are now monitoring before marking as resolved. More details can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/18535.

September 10, 2024 13:24 UTC
[Investigating] No material updates at this time - our investigation is ongoing. More details can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/18535

September 10, 2024 13:10 UTC
[Investigating]

September 10, 2024 13:08 UTC
[Investigating] We are currently investigating connectivity issues to GitLab.com. For more information see gitlab.com/gitlab-com/gl-infra/production/-/issues/18535

GitLab.com Connectivity Issues

September 3, 2024 05:34 UTC

Incident Status

Service Disruption


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions, SAML SSO - GitLab SaaS, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, docs.gitlab.com, Canary, Product Analytics - Configurator (Beta), Product Analytics - Collector (Beta), Product Analytics - Background processing (Beta), Product Analytics - Queries (Beta), GitLab Duo


Locations

Google Compute Engine, Digital Ocean, Zendesk, AWS




September 3, 2024 05:34 UTC
[Resolved] All GitLab services have been restored. We are actively monitoring the system, and everything appears normal at this time. For more details, please follow the issue at: gitlab.com/gitlab-com/gl-infra/production/-/issues/18490

September 3, 2024 05:19 UTC
[Monitoring] We've identified and addressed the root cause of the issue. All GitLab services have been restored, and we are actively monitoring the system. Follow the issue for more details : gitlab.com/gitlab-com/gl-infra/production/-/issues/18490

September 3, 2024 05:09 UTC
[Investigating] GitLab.com is currently experiencing connectivity issues due to a high error rate. Our infrastructure team is investigating. Issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/18490

August 2024

Embedding DB Upgrade

August 31, 2024 04:58 UTC

Embedding DB UpgradePlanned Maintenance

Description

We will be undergoing scheduled maintenance to our database layer. GitLab.com will continue to be available during the maintenance window. For more details, see gitlab.com/gitlab-com/gl-infra/dbre/-/issues/228


Components

Website, API, Git Operations, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions


Locations

Google Compute Engine, AWS


Schedule

October 25, 2024 00:00 - October 30, 2024 01:00 UTC



August 31, 2024 04:58 UTC
[Update] We are cancelling the Embedding database upgrade. See gitlab.com/gitlab-com/gl-infra/dbre/-/issues/228 for details. All other DB upgrade schedules will go ahead as planned.

CI/CD token allowlist selection failing

August 27, 2024 14:52 UTC

CI/CD token allowlist selection failingPartial Service Disruption

Incident Status

Partial Service Disruption


Components

Website, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions


Locations

Google Compute Engine, AWS




August 27, 2024 14:52 UTC
[Resolved] The revert MR has been deployed to production, and we see that CI/CD token allowlist project selection renders properly now. Details will be provided in the incident issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/18446

August 27, 2024 03:55 UTC
[Identified] We have identified an issue that is causing CI/CD token allowlist project selection to fail. We are in the process of reverting the change that caused this. For more details see gitlab.com/gitlab-com/gl-infra/production/-/issues/18446.

Some requests timing out

August 23, 2024 03:59 UTC

Incident Status

Operational


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions, SAML SSO - GitLab SaaS, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, docs.gitlab.com, Canary, Product Analytics - Configurator (Beta), Product Analytics - Collector (Beta), Product Analytics - Background processing (Beta), Product Analytics - Queries (Beta), GitLab Duo


Locations

Google Compute Engine, Digital Ocean, Zendesk, AWS




August 23, 2024 03:59 UTC
[Resolved] The issue has been resolved and all functionality on GitLab.com is nominal. Please see gitlab.com/gitlab-com/gl-infra/production/-/issues/18435 for more details.

August 23, 2024 01:53 UTC
[Monitoring] The issue has been identified and the incident resolved. We will continue monitoring for a while.

August 23, 2024 01:27 UTC
[Identified] We have identified an underlying cause and are working to mitigate it. See gitlab.com/gitlab-com/gl-infra/production/-/issues/18435 for more details.

August 23, 2024 00:34 UTC
[Investigating] We have noticed that there are delays in web UI updates for example, MRs not showing up after submission, We are continuing to investigate - please see gitlab.com/gitlab-com/gl-infra/production/-/issues/18435 for more details.

August 23, 2024 00:27 UTC
[Investigating] Some requests are slow or timing out due to DB saturation. We are investigating. Please see [this issue](gitlab.com/gitlab-com/gl-infra/production/-/issues/18435) for further details.

July 2024

GitLab.com is unavailable

July 11, 2024 18:28 UTC

GitLab.com is unavailableService Disruption

Incident Status

Service Disruption


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions, SAML SSO - GitLab SaaS, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, docs.gitlab.com, Canary, Product Analytics - Configurator (Beta), Product Analytics - Collector (Beta), Product Analytics - Background processing (Beta), Product Analytics - Queries (Beta), GitLab Duo


Locations

Google Compute Engine, Digital Ocean, Zendesk, AWS




July 11, 2024 18:28 UTC
[Resolved] This incident is now resolved and no further errors are being seen on GitLab.com. Our investigation points to a disk performance issue. You can see further details and corrective actions in gitlab.com/gitlab-com/gl-infra/production/-/issues/18269.

July 11, 2024 15:51 UTC
[Investigating] Our investigations are ongoing. We will post our next update when we have more information. Please follow the incident issue for updates here: gitlab.com/gitlab-com/gl-infra/production/-/issues/18269

July 11, 2024 15:28 UTC
[Investigating] No material updates to report. We're continuing to investigate the issue. Please follow the incident issue for updates here: gitlab.com/gitlab-com/gl-infra/production/-/issues/18269

July 11, 2024 15:00 UTC
[Investigating] We are seeing improved service availability and investigations continue for this problem. Please follow the incident issue for updates here: gitlab.com/gitlab-com/gl-infra/production/-/issues/18269

July 11, 2024 14:43 UTC
[Investigating] We are seeing improved service availability and investigations continue for this problem. We are planning to make the incident issue publicly available as soon as possible.

July 11, 2024 14:29 UTC
[Investigating] We are investigating a problem with gitlab.com being unavailable for some users. Our investigations continue and we will post further status updates shortly.

July 11, 2024 14:18 UTC
[Investigating] We are investigating a problem with gitlab.com being unavailable for some users. We will post further status updates shortly.

April 2024

Downstream Pipelines Returning Errors

April 23, 2024 15:19 UTC

Incident Status

Degraded Performance


Components

CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions


Locations

Google Compute Engine




April 23, 2024 15:19 UTC
[Resolved] This incident has been resolved and pipelines are running without errors now. More details are available in gitlab.com/gitlab-com/gl-infra/production/-/issues/17885.

April 22, 2024 21:56 UTC
[Identified] A fix for the issue has been merged and will be available in the next deployment tomorrow morning. Full details here gitlab.com/gitlab-com/gl-infra/production/-/issues/17885.

April 22, 2024 20:43 UTC
[Identified] We've identified the cause of the issue and preparations for a fix are ongoing. Details in gitlab.com/gitlab-com/gl-infra/production/-/issues/17885.

April 22, 2024 18:51 UTC
[Identified] We've identified the cause of the issue and are preparing options to deploy a fix. Details in gitlab.com/gitlab-com/gl-infra/production/-/issues/17885.

April 22, 2024 18:24 UTC
[Identified] We confirmed that the issue is isolated to a small number of namespaces and are continuing to investigate. Details in gitlab.com/gitlab-com/gl-infra/production/-/issues/17885.

April 22, 2024 17:56 UTC
[Investigating] We're investigating reports of parent-child pipeline setups returning an error and failing to create a downstream pipeline. Details in gitlab.com/gitlab-com/gl-infra/production/-/issues/17885.





Back to current status