Active Issue

Updated a few seconds ago

Back to current status

Status History

Filter: AWS (Clear)



February 2025

PG minor upgrade 16.6 in Production

February 15, 2025 07:49 UTC

Description

Scheduled maintenance to patch our DB layer to PostgreSQL 16.6. 2025-02-15 02:00-08:00 UTC. GitLab.com will be available throughout this time, but users may experience errors on requests. We apologize in advance for any inconvenience caused. See <gitlab.com/gitlab-com/gl-infra/production/-/issues/19120>


Components

Website, API, Container Registry, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions, Background Processing


Locations

Google Compute Engine, AWS


Schedule

February 15, 2025 03:00 - February 15, 2025 09:00 UTC



February 15, 2025 07:49 UTC
[Update] Gitlab.com database maintenance was performed, the platform is considered stable. We'll continue to monitor for any performance issues. Thank you for your patience. See <gitlab.com/gitlab-com/gl-infra/production/-/issues/19120>

February 15, 2025 04:07 UTC
[Update] Gitlab.com scheduled maintenance of our database layer has started. GitLab.com should be available during the whole period but we expect availability and performance impact until 2025-02-15 09:00 UTC. See gitlab.com/gitlab-com/gl-infra/production/-/issues/19120

February 15, 2025 03:01 UTC
[Update] Gitlab.com scheduled maintenance of our database layer has started. GitLab.com should be available during the whole period but we expect availability and performance impact until 2025-02-15 09:00 UTC. See gitlab.com/gitlab-com/gl-infra/production/-/issues/19120

February 15, 2025 02:48 UTC
[Update] Scheduled database maintenance will now start at 2025-02-15 03:00 UTC and expected to run to 2025-02-15 09:00 UTC. See gitlab.com/gitlab-com/gl-infra/production/-/issues/19120

February 15, 2025 01:54 UTC
[Update] Due to an ongoing incident we are delaying the maintenance window by 2 hours to investigate the impact. We currently plan to start the maintenance window at 04:00 UTC.

February 14, 2025 01:45 UTC
[Update] We will be undergoing scheduled maintenance of our databases tomorrow. There may be some availability and performance impact between 2025-02-15 02:00 UTC and 2025-02-15 08:00 UTC. See gitlab.com/gitlab-com/gl-infra/production/-/issues/19120

February 12, 2025 01:13 UTC
[Update] We will be undergoing scheduled maintenance of our databases in 3 days. There may be some availability and performance impact between 2025-02-15 02:00 UTC and 2025-02-15 08:00 UTC. See gitlab.com/gitlab-com/gl-infra/production/-/issues/19120

February 8, 2025 08:37 UTC
[Update] Scheduled maintenance in 1 week: Upgrading DB layer to PostgreSQL 16.6 on Feb 15, 02:00-08:00 UTC. GitLab.com will stay up but users may experience errors. See gitlab.com/gitlab-com/gl-infra/production/-/issues/19120

February 1, 2025 02:42 UTC
[Update] Scheduled maintenance in 2 weeks: Upgrading DB layer to PostgreSQL 16.6 on Feb 15, 02:00-08:00 UTC. GitLab.com will stay up but may experience errors. Sorry for any inconvenience. See gitlab.com/gitlab-com/gl-infra/production/-/issues/19120

Pipelines fail with "Unable to create pipeline" error

February 14, 2025 22:13 UTC

Incident Status

Partial Service Disruption


Components

CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions, CI/CD - Self-managed runners


Locations

Google Compute Engine, AWS, Self-Managed Runner Connectivity




February 14, 2025 22:13 UTC
[Resolved] This fix has been successfully deployed and we have confirmed the issue is no longer occurring. We appreciate your patience while we worked to address this. We will mark this incident as resolved. Details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/19310

February 14, 2025 19:43 UTC
[Identified] We expect the revert MR to land in production within the next 3 hours. Impacted users can find a workaround within the incident issue. We will continue to provide updates here as needed. Details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/19310

February 14, 2025 17:06 UTC
[Identified] We expect the revert MR to land in production within the next 6 hours. Additional updates will be posted here as needed. If you are one the subset of users impacted by this change, please see this issue for a workaround and other additional details: gitlab.com/gitlab-com/gl-infra/production/-/issues/19310

February 14, 2025 16:11 UTC
[Identified] The revert MR to address the issue has been merged. We are still waiting for this to land in production. These changes will be re-introduced to the Dependency-Scanning.latest.gitlab-ci.yml template, where breaking changes can be expected. More details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/19310

February 14, 2025 15:25 UTC
[Identified] We have created a Merge Request to revert recent changes to the Dependency-Scanning.gitlab-ci.yml template. Continue following this incident issue for further updates: gitlab.com/gitlab-com/gl-infra/production/-/issues/19310

February 14, 2025 14:52 UTC
[Identified] We are currently experiencing issues with failing GitLab.com pipelines that include the Dependency-Scanning.gitlab-ci.yml template. We have identified the root cause and are preparing to revert the MR. For a workaround and more details, please see: gitlab.com/gitlab-com/gl-infra/production/-/issues/19310

Errors managing Kubernetes agents

February 11, 2025 15:23 UTC

Errors managing Kubernetes agentsPartial Service Disruption

Incident Status

Partial Service Disruption


Components

Website, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions, CI/CD - Self-managed runners


Locations

Google Compute Engine, AWS, Self-Managed Runner Connectivity




February 11, 2025 15:23 UTC
[Resolved] The fix has been successfully deployed. Our monitoring confirms that the service has been restored to normal operation with no further errors observed. Thank you for your patience during this disruption. If you experience any further issues, please contact our Support team.

February 11, 2025 14:39 UTC
[Monitoring] We are currently rolling out a fix to production. Our team is actively monitoring the deployment and its effectiveness. Please continue to follow our status page for the latest updates.

February 11, 2025 10:24 UTC
[Identified] The ongoing incident investigation has been marked as confidential. While the original incident report will no longer be publicly visible, we continue to actively work on the resolution. Service impact remains unchanged. Further updates will be available on the status page - status.gitlab.com

February 11, 2025 09:35 UTC
[Identified] We identified the root cause of the incident, and we are reverting the relevant MR. The estimated time for the fix: around 5 hours from now. More details: gitlab.com/gitlab-com/gl-infra/production/-/issues/19278

February 11, 2025 09:00 UTC
[Investigating] Some users are experiencing errors managing Kubernetes agents. Details can be found in: gitlab.com/gitlab-com/gl-infra/production/-/issues/19278

January 2025

Packages on GitLab.com (packages.gitlab.com) become unavailable sporadically with 502 errors.

January 10, 2025 03:54 UTC

Incident Status

Degraded Performance


Components

packages.gitlab.com


Locations

AWS




January 10, 2025 03:54 UTC
[Resolved] The fix for the sporadic unavailability of packages on GitLab.com (packages.gitlab.com) has completed successfully. We are now marking this incident as resolved. More about the incident and its progress can be read here: gitlab.com/gitlab-com/gl-infra/production/-/issues/19091

January 10, 2025 02:32 UTC
[Monitoring] The fix for the sporadic unavailability of packages on GitLab.com (packages.gitlab.com) is ongoing. It appears to be progressing as expected. More about the incident and its progress can be read here: gitlab.com/gitlab-com/gl-infra/production/-/issues/19091

January 9, 2025 21:55 UTC
[Monitoring] The fix for the sporadic unavailability of packages on GitLab.com (packages.gitlab.com) is ongoing. We expect it to take about 20 hours, and will post updates every 4 hours. More about the incident and its progress can be read here: gitlab.com/gitlab-com/gl-infra/production/-/issues/19091

January 9, 2025 19:53 UTC
[Monitoring] No updates at this time. The fix for the sporadic unavailability of packages on GitLab.com (packages.gitlab.com) is still processing. More about the incident and its progress can be read here: gitlab.com/gitlab-com/gl-infra/production/-/issues/19091

January 9, 2025 18:13 UTC
[Monitoring] The fix for the sporadic unavailability of packages on GitLab.com (packages.gitlab.com) has been implemented, and is processing now. More about the incident and its progress can be read here: gitlab.com/gitlab-com/gl-infra/production/-/issues/19091

January 9, 2025 16:30 UTC
[Identified] We're still working on implementing the fix for the sporadic unavailability of packages on GitLab.com (packages.gitlab.com). More about the incident and its progress can be read here: gitlab.com/gitlab-com/gl-infra/production/-/issues/19091

January 9, 2025 15:54 UTC
[Identified] We're still working on implementing the fix for the sporadic unavailability of packages on GitLab.com (packages.gitlab.com). More about the incident and its progress can be read here: gitlab.com/gitlab-com/gl-infra/production/-/issues/19091

January 9, 2025 15:22 UTC
[Identified] We're still working on implementing the fix for the sporadic unavailability of packages on GitLab.com (packages.gitlab.com). More about the incident and its updates can be read here: gitlab.com/gitlab-com/gl-infra/production/-/issues/19091

January 9, 2025 14:49 UTC
[Identified] We're still working on implementing the fix for the sporadic unavailability of packages on GitLab.com (packages.gitlab.com). More about the incident and its updates can be read here: gitlab.com/gitlab-com/gl-infra/production/-/issues/19091

January 9, 2025 14:17 UTC
[Identified] We're still working on implementing the fix for the sporadic unavailability of packages on GitLab.com (packages.gitlab.com). More about the incident and its updates can be read here: gitlab.com/gitlab-com/gl-infra/production/-/issues/19091

January 9, 2025 13:43 UTC
[Identified] We're working on implementing the fix for the sporadic unavailability of packages on GitLab.com (packages.gitlab.com). More about the incident and its updates can be read here: gitlab.com/gitlab-com/gl-infra/production/-/issues/19091

January 9, 2025 13:09 UTC
[Identified] We've identified the root cause for the sporadic unavailability of packages on GitLab.com (packages.gitlab.com) and working on a fix. More about the incident and its updates can be read here: gitlab.com/gitlab-com/gl-infra/production/-/issues/19091

January 9, 2025 12:48 UTC
[Investigating] We're still investigating the root cause for the sporadic unavailability of packages on GitLab.com (packages.gitlab.com). More about the incident and its updates can be read here: gitlab.com/gitlab-com/gl-infra/production/-/issues/19091

January 9, 2025 12:33 UTC
[Investigating] We're still investigating the root cause for the sporadic unavailability of packages on GitLab.com (packages.gitlab.com). More about the incident and its updates can be read here: gitlab.com/gitlab-com/gl-infra/production/-/issues/19091

January 9, 2025 12:17 UTC
[Investigating] We're still investigating the root cause for the sporadic unavailability of packages on GitLab.com (packages.gitlab.com). More about the incident and its updates can be read here: gitlab.com/gitlab-com/gl-infra/production/-/issues/19091

January 9, 2025 12:02 UTC
[Investigating] We're still investigating the root cause for the sporadic unavailability of packages on GitLab.com (packages.gitlab.com). More about the incident and its updates can be read here: gitlab.com/gitlab-com/gl-infra/production/-/issues/19091

January 9, 2025 11:45 UTC
[Investigating] We are currently investigating an incident that is causing packages on GitLab.com (packages.gitlab.com) to become unavailable sporadically with 502 errors. More about the incident can be read here: gitlab.com/gitlab-com/gl-infra/production/-/issues/19091

Accessing gitlab.com results in 500 errors

January 6, 2025 16:42 UTC

Incident Status

Partial Service Disruption


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions, SAML SSO - GitLab SaaS, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, docs.gitlab.com, Canary, Product Analytics - Configurator (Beta), Product Analytics - Collector (Beta), Product Analytics - Background processing (Beta), Product Analytics - Queries (Beta), GitLab Duo


Locations

Google Compute Engine, Digital Ocean, Zendesk, AWS




January 6, 2025 16:42 UTC
[Resolved] We've identified the cause of the errors and the errors have returned to normal. The incident is now resolved. More information on the incident can be found in this issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/19079

January 6, 2025 15:57 UTC
[Monitoring] We've identified the cause of the errors, and the errors have returned to normal. The incident is now mitigated, but we're continuing to monitor for any further impact. More information on the incident can be found in this issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/19079

January 6, 2025 15:27 UTC
[Monitoring] We're continuing to monitor for further errors related to the incident. We see further decreases in the errors.

January 6, 2025 15:17 UTC
[Monitoring] We saw elevated 500 errors on gitlab.com which have then decreased. We're continuing to monitor for further errors

November 2024

Update the CA of certificates for number of hostnames

November 7, 2024 01:55 UTC

Description

To maximize device compatibility going forward, GitLab will update the CA of certificates used by a number of hostnames to Google Trust Services. For full details please refer to the CR issue : gitlab.com/gitlab-com/gl-infra/production/-/issues/18625


Components

packages.gitlab.com


Locations

AWS


Schedule

November 6, 2024 00:00 - November 7, 2024 00:00 UTC



November 7, 2024 01:55 UTC
[Update] GitLab.com planned maintenance for updating certificate CAs has completed. There were no reported problems during the monitoring period in the maintenance window. Thank you for your patience.

November 6, 2024 23:44 UTC
[Update] Maintenance started as of November 6, 2024 00:00 and is expected to end November 7, 2024 00:00 UTC

November 6, 2024 02:48 UTC
[Update] Maintenance is underway to update the CA of certificates used by a number of hostnames to Google Trust Services. No customer impact is expected from this change. We'll continue to monitor until the end of the maintenance window. For full details please refer to the CR issue : gitlab.com/gitlab-com/gl-infra/production/-/issues/18625

November 4, 2024 00:39 UTC
[Update] Due to the ongoing production change lock this maintenance has been rescheduled, to begin 2024-11-06 00:00 to 2024-11-07 00:00 UTC We apologize for the inconvenience; this update has been reflected on the infrastructure issue as well: gitlab.com/gitlab-com/gl-infra/production/-/issues/18655

October 30, 2024 14:01 UTC
[Update] Due to circumstances beyond our control, the maintenance has been rescheduled from the original timeframe of 2024-10-31 - 2024-11-01 to 2024-11-04 - 2024-11-05. We apologize for the inconvenience; this update has been reflected on the infrastructure issue as well: gitlab.com/gitlab-com/gl-infra/production/-/issues/18655

MAIN DB cluster upgrade

November 5, 2024 11:02 UTC

MAIN DB cluster upgrade Planned Maintenance

Description

Next week, we will be undergoing scheduled maintenance to our MAIN database layer. The maintenance will start at 2024-11-03 06:00 UTC and should finish at 2024-11-05 11:00 UTC (including performance regression observability period). GitLab.com will be available during the whole period as the maintenance should be seamless and transparent for the application. We apologize in advance for any inconvenience this may cause. See <gitlab.com/gitlab-com/gl-infra/production/-/issues/18747>


Components

Website, API, Git Operations, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions


Locations

Google Compute Engine, AWS


Schedule

November 3, 2024 06:00 - November 5, 2024 11:00 UTC



November 5, 2024 11:02 UTC
[Update] Gitlab.com MAIN database upgrade is now complete with all systems are functioning correctly. Thank you for your patience. See gitlab.com/gitlab-com/gl-infra/production/-/issues/18747

November 3, 2024 09:01 UTC
[Update] Gitlab.com MAIN database upgrade was performed. We'll continue to monitor for any performance issues until the end of the maintenance window. Thank you for your patience. See <gitlab.com/gitlab-com/gl-infra/production/-/issues/18747>

November 3, 2024 06:05 UTC
[Update] Gitlab.com scheduled maintenance of our MAIN database layer have started. GitLab.com should be available during the whole period. See <gitlab.com/gitlab-com/gl-infra/production/-/issues/18747>

November 3, 2024 05:00 UTC
[Update] MAIN DB layer maintenance will start in 1h from now at 2024-11-03 06:00 UTC. GitLab.com will remains available. Details: gitlab.com/gitlab-com/gl-infra/production/-/issues/18747

November 2, 2024 05:00 UTC
[Update] Scheduled maintenance on MAIN DB layer: Start: 2024-11-03 06:00 UTC End: 2024-11-05 11:00 UTC (incl. monitoring) GitLab.com remains available. Maintenance should be seamless. We apologize for any inconvenience. Details: gitlab.com/gitlab-com/gl-infra/production/-/issues/18747

October 30, 2024 06:00 UTC
[Update] In three days, we will be undergoing scheduled maintenance to our MAIN database layer. The maintenance will start at 2024-11-03 06:00 UTC and should finish at 2024-11-05 11:00 UTC (including performance and regression observability period). GitLab.com will be available during the whole period as the maintenance should be seamless and transparent for the application. We apologize in advance for any inconvenience this may cause. See: gitlab.com/gitlab-com/gl-infra/production/-/issues/18747

October 2024

CI DB cluster upgrade

October 30, 2024 01:37 UTC

CI DB cluster upgradePlanned Maintenance

Description

Next week, we will be undergoing scheduled maintenance to our CI database layer. The maintenance will start at 2024-10-26 06:00 UTC and should finish at 2024-10-29 11:00 UTC (including performance regression observability period). GitLab.com will be available during the whole period as the maintenance should be seamless and transparent for the application. We apologize in advance for any inconvenience this may cause. See <gitlab.com/gitlab-com/gl-infra/production/-/issues/18639>


Components

Website, API, Git Operations, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions


Locations

Google Compute Engine, AWS


Schedule

October 26, 2024 06:00 - October 29, 2024 11:00 UTC



October 30, 2024 01:37 UTC
[Update] GitLab.com upgrade of the CI database layer is now complete with all systems are functioning correctly. Thank you for your patience. See gitlab.com/gitlab-com/gl-infra/production/-/issues/18639

October 27, 2024 08:47 UTC
[Update] GitLab.com upgrade of the CI database layer is complete. We'll be monitoring the platform to ensure all systems are functioning correctly. Thank you for your patience.

October 26, 2024 06:01 UTC
[Update] We are starting maintenance to the CI database layer, and should finish at 2024-10-29 11:00 UTC (including performance regression and observability period). GitLab.com will be available during the whole period as the maintenance should be seamless. See gitlab.com/gitlab-com/gl-infra/production/-/issues/18639

October 25, 2024 19:03 UTC
[Update] Reminder: Tomorrow, we will be undergoing scheduled maintenance to our CI database layer. The maintenance will start at 2024-10-26 06:00 UTC and should finish at 2024-10-29 11:00 UTC (including performance regression observability period). GitLab.com will be available during the whole period as the maintenance should be seamless and transparent for the application. We apologize in advance for any inconvenience this may cause. See <gitlab.com/gitlab-com/gl-infra/production/-/issues/18639>

October 23, 2024 17:25 UTC
[Update] In 3 days, we will be undergoing scheduled maintenance to our CI database layer. The maintenance will start at 2024-10-26 06:00 UTC and should finish at 2024-10-29 11:00 UTC (including performance regression observability period). GitLab.com will be available during the whole period as the maintenance should be seamless and transparent for the application. We apologize in advance for any inconvenience this may cause. See <gitlab.com/gitlab-com/gl-infra/production/-/issues/18639>

September 2024

Main DB Upgrade

September 26, 2024 03:54 UTC

Main DB UpgradePlanned Maintenance

Description

We will be undergoing scheduled maintenance to our database layer. GitLab.com will continue to be available during the maintenance window. For more details, see gitlab.com/gitlab-com/gl-infra/dbre/-/issues/227


Components

Website, API, Git Operations, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions


Locations

Google Compute Engine, AWS


Schedule

October 25, 2024 00:00 - October 30, 2024 01:00 UTC



September 26, 2024 03:54 UTC
[Update] Scheduled maintenance on October 25 to our database layer has now been cancelled. For more details, see gitlab.com/gitlab-com/gl-infra/dbre/-/issues/227

CI DB upgrade

September 26, 2024 03:50 UTC

CI DB upgradePlanned Maintenance

Description

Next week, we will be undergoing scheduled maintenance to our CI database layer. The maintenance will start at 2024-09-29 00:00 UTC and should finish at 2024-10-02 02:00 UTC (including performance regression observability period). GitLab.com will be available during the whole period as the maintenance should be seamless and transparent for the application. We apologize in advance for any inconvenience this may cause. See <gitlab.com/gitlab-com/gl-infra/production/-/issues/18593>


Components

Website, API, Git Operations, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions


Locations

Google Compute Engine, AWS


Schedule

September 27, 2024 00:00 - October 2, 2024 01:00 UTC



September 26, 2024 03:50 UTC
[Update] Maintenance scheduled for 2024-09-29 on our CI database layer, has now been cancelled. For more information: gitlab.com/gitlab-com/gl-infra/production/-/issues/18593

"View job currently using resource" button returns 500 error

September 20, 2024 19:10 UTC

Incident Status

Degraded Performance


Components

CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions


Locations

Google Compute Engine, AWS




September 20, 2024 19:10 UTC
[Resolved] GitLab has verified we are no longer seeing 500 errors from the View Job button. We are resolving the incident. Any further updates will be posted in the incident issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/18530

September 17, 2024 02:35 UTC
[Monitoring] A fix has been deployed and is being verified, further updates will be provided upon completion. Details available in: gitlab.com/gitlab-com/gl-infra/production/-/issues/18530

September 12, 2024 20:47 UTC
[Identified] The fix is known and will be deployed soon. There are some existing pipelines that are still affected, and our team is working on resolving them. Affected users can view the open issue here: gitlab.com/gitlab-com/gl-infra/production/-/issues/18530

September 11, 2024 20:26 UTC
[Identified] No updates at this time. There are some existing pipelines that are still affected, and our team is working on resolving them. Affected users can view the open issue here: gitlab.com/gitlab-com/gl-infra/production/-/issues/18530

September 11, 2024 17:40 UTC
[Identified] A fix has been deployed to production. New pipelines are unaffected. However, there are some existing pipelines that are still affected, and our team is working on resolving them. Affected users can view the open issue here: gitlab.com/gitlab-com/gl-infra/production/-/issues/18530

September 11, 2024 09:32 UTC
[Resolved] This incident has been resolved. Affected users can view the open issue here: gitlab.com/gitlab-com/gl-infra/production/-/issues/18530 If you have CI pipelines that are stuck as a result of this issue, please cancel them so they don't continue to use up your CI minutes quota. We’re actively looking into remediating CI minutes usage for the impacted jobs. If you believe you are impacted by this incident, please open a support ticket here: support.gitlab.com

September 10, 2024 19:33 UTC
[Identified] A fix has been deployed to production. New pipelines are unaffected. However, there are some existing pipelines that are still affected, and our team is working on resolving them. Affected users can view the open issue here: gitlab.com/gitlab-org/gitlab/-/issues/483290 If you continue to see the issue, please contact support here: support.gitlab.com

September 9, 2024 22:25 UTC
[Identified] At this time there are no further updates. Affected users can view the open issue here: gitlab.com/gitlab-org/gitlab/-/issues/483290 If you deleted a pipeline that had a resource group and are now seeing stuck jobs, please contact support.

September 9, 2024 20:56 UTC
[Identified] Some users are seeing 500 errors when clicking the "View job currently using resource" button. Users may also see jobs stuck in "canceling" state after deleting a pipeline that was deploying to a resource group. The fix is already in the works. Affected users can view the open issue here: gitlab.com/gitlab-org/gitlab/-/issues/483290 If you deleted a pipeline that had a resource group and are now seeing stuck jobs, please contact support.

Registry DB upgrade

September 11, 2024 22:51 UTC

Registry DB upgradePlanned Maintenance

Description

We will be undergoing scheduled maintenance to our database layer. GitLab.com will continue to be available during the maintenance window. For more details, see gitlab.com/gitlab-com/gl-infra/dbre/-/issues/225


Components

Website, API, Git Operations, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions


Locations

Google Compute Engine, AWS


Schedule

September 6, 2024 00:00 - September 11, 2024 01:00 UTC



September 11, 2024 22:51 UTC
[Update] Planned maintenance for the registry database layer on GitLab.com is complete. Registry database is now running Postgres v16. The rollback window for the Registry database was closed on approximately 2024-09-12 1:00 UTC.

September 8, 2024 01:51 UTC
[Update] Planned maintenance for the registry database layer on GitLab.com is complete. Registry database is now running Postgres v16. Rollback window for this planned maintenance ends 2024-09-11 00:00 UTC. We'll be monitoring the platform during this time to ensure all systems are functioning correctly.

September 8, 2024 00:11 UTC
[Update] Scheduled maintenance to our registry database layer has started. The maintenance should finish at 2024-09-11 02:00 UTC UTC (including performance regression observability period). GitLab.com will be available during the whole period as the maintenance should be seamless and transparent for the application. We apologize in advance for any inconvenience this may cause. See issue for more information: gitlab.com/gitlab-com/gl-infra/production/-/issues/18447

September 5, 2024 05:53 UTC
[Update] In 3 days, we will be undergoing scheduled maintenance to our registry database layer. The maintenance will start at 2024-09-08 00:00 UTC UTC and should finish at 2024-09-11 02:00 UTC UTC (including performance regression observability period). GitLab.com will be available during the whole period as the maintenance should be seamless and transparent for the application. We apologize in advance for any inconvenience this may cause. See <gitlab.com/gitlab-com/gl-infra/production/-/issues/18447>

Some jobs to running longer than the defined timeout

September 11, 2024 09:37 UTC

Incident Status

Operational


Components

CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions


Locations

Google Compute Engine, AWS




September 11, 2024 09:37 UTC
[Resolved] This incident has been resolved. Affected users can view the open issue here: gitlab.com/gitlab-com/gl-infra/production/-/issues/18533 If you have CI pipelines that are stuck as a result of this issue, please cancel them so they don't continue to use up your CI minutes quota. We’re actively looking into remediating CI minutes usage for the impacted jobs. If you believe you are impacted by this incident, please open a support ticket here: support.gitlab.com

September 10, 2024 22:28 UTC
[Monitoring] An incident earlier today caused some jobs to run longer than the defined timeout. This lead to excessive usage of CI minutes. Affected users can view the open issue here: gitlab.com/gitlab-com/gl-infra/production/-/issues/18533 If you have CI pipelines that are stuck as a result of this issue, please cancel them so they don't continue to use up your CI minutes quota. We’re actively looking into remediating CI minutes usage for the impacted jobs. If you believe you are impacted by this incident, please open a support ticket here: support.gitlab.com

September 10, 2024 21:48 UTC
[Monitoring] An incident earlier today caused some jobs to run longer than the defined timeout. This lead to excessive usage of CI minutes. Affected users can view the open issue here: gitlab.com/gitlab-com/gl-infra/production/-/issues/18533 We’re actively looking into remediating CI minutes usage for the impacted jobs. If you believe you are impacted by this incident, please open a support ticket. support.gitlab.com

GitLab.com Connectivity Issues

September 3, 2024 05:34 UTC

Incident Status

Service Disruption


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions, SAML SSO - GitLab SaaS, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, docs.gitlab.com, Canary, Product Analytics - Configurator (Beta), Product Analytics - Collector (Beta), Product Analytics - Background processing (Beta), Product Analytics - Queries (Beta), GitLab Duo


Locations

Google Compute Engine, Digital Ocean, Zendesk, AWS




September 3, 2024 05:34 UTC
[Resolved] All GitLab services have been restored. We are actively monitoring the system, and everything appears normal at this time. For more details, please follow the issue at: gitlab.com/gitlab-com/gl-infra/production/-/issues/18490

September 3, 2024 05:19 UTC
[Monitoring] We've identified and addressed the root cause of the issue. All GitLab services have been restored, and we are actively monitoring the system. Follow the issue for more details : gitlab.com/gitlab-com/gl-infra/production/-/issues/18490

September 3, 2024 05:09 UTC
[Investigating] GitLab.com is currently experiencing connectivity issues due to a high error rate. Our infrastructure team is investigating. Issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/18490

August 2024

Embedding DB Upgrade

August 31, 2024 04:58 UTC

Embedding DB UpgradePlanned Maintenance

Description

We will be undergoing scheduled maintenance to our database layer. GitLab.com will continue to be available during the maintenance window. For more details, see gitlab.com/gitlab-com/gl-infra/dbre/-/issues/228


Components

Website, API, Git Operations, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions


Locations

Google Compute Engine, AWS


Schedule

October 25, 2024 00:00 - October 30, 2024 01:00 UTC



August 31, 2024 04:58 UTC
[Update] We are cancelling the Embedding database upgrade. See gitlab.com/gitlab-com/gl-infra/dbre/-/issues/228 for details. All other DB upgrade schedules will go ahead as planned.

CI/CD token allowlist selection failing

August 27, 2024 14:52 UTC

CI/CD token allowlist selection failingPartial Service Disruption

Incident Status

Partial Service Disruption


Components

Website, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions


Locations

Google Compute Engine, AWS




August 27, 2024 14:52 UTC
[Resolved] The revert MR has been deployed to production, and we see that CI/CD token allowlist project selection renders properly now. Details will be provided in the incident issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/18446

August 27, 2024 03:55 UTC
[Identified] We have identified an issue that is causing CI/CD token allowlist project selection to fail. We are in the process of reverting the change that caused this. For more details see gitlab.com/gitlab-com/gl-infra/production/-/issues/18446.

Some requests timing out

August 23, 2024 03:59 UTC

Incident Status

Operational


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions, SAML SSO - GitLab SaaS, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, docs.gitlab.com, Canary, Product Analytics - Configurator (Beta), Product Analytics - Collector (Beta), Product Analytics - Background processing (Beta), Product Analytics - Queries (Beta), GitLab Duo


Locations

Google Compute Engine, Digital Ocean, Zendesk, AWS




August 23, 2024 03:59 UTC
[Resolved] The issue has been resolved and all functionality on GitLab.com is nominal. Please see gitlab.com/gitlab-com/gl-infra/production/-/issues/18435 for more details.

August 23, 2024 01:53 UTC
[Monitoring] The issue has been identified and the incident resolved. We will continue monitoring for a while.

August 23, 2024 01:27 UTC
[Identified] We have identified an underlying cause and are working to mitigate it. See gitlab.com/gitlab-com/gl-infra/production/-/issues/18435 for more details.

August 23, 2024 00:34 UTC
[Investigating] We have noticed that there are delays in web UI updates for example, MRs not showing up after submission, We are continuing to investigate - please see gitlab.com/gitlab-com/gl-infra/production/-/issues/18435 for more details.

August 23, 2024 00:27 UTC
[Investigating] Some requests are slow or timing out due to DB saturation. We are investigating. Please see [this issue](gitlab.com/gitlab-com/gl-infra/production/-/issues/18435) for further details.

July 2024

"Package Manifest is not coherent" errors

July 11, 2024 20:26 UTC

Incident Status

Partial Service Disruption


Components

packages.gitlab.com


Locations

AWS




July 11, 2024 20:26 UTC
[Resolved] The two MR's that were opened to address this incident are deployed and live on GitLab.com. This incident is now resolved. Some users may still see the error due to npm auto-corrections and we are adding documentation on how to handle it. See gitlab.com/gitlab-com/gl-infra/production/-/issues/18253 for more details.

July 10, 2024 19:44 UTC
[Identified] Some users are receiving the error message "Package Manifest is not coherent" when attempting to publish NPM packages. A second MR has been opened to address another underlying bug that was discovered while addressing the root cause. Once this MR is merged and deployed, the incident will be resolved. Affected users can subscribe to the issue for updates here: gitlab.com/gitlab-com/gl-infra/production/-/issues/18253

July 10, 2024 14:41 UTC
[Identified] Some users are receiving the error message "Package Manifest is not coherent" when attempting to publish NPM packages. We have identified the cause, and are working on a fix. Possible workarounds have been added to the incident - gitlab.com/gitlab-com/gl-infra/production/-/issues/18253#note_1992957468

July 9, 2024 23:23 UTC
[Identified] Some users are receiving the error message "Package Manifest is not coherent" when attempting to publish NPM packages. The cause is known, the fix has been merged and will be deployed to production in the next deployment. Affected users can subscribe to the issue for updates here: gitlab.com/gitlab-com/gl-infra/production/-/issues/18253

GitLab.com is unavailable

July 11, 2024 18:28 UTC

GitLab.com is unavailableService Disruption

Incident Status

Service Disruption


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions, SAML SSO - GitLab SaaS, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, docs.gitlab.com, Canary, Product Analytics - Configurator (Beta), Product Analytics - Collector (Beta), Product Analytics - Background processing (Beta), Product Analytics - Queries (Beta), GitLab Duo


Locations

Google Compute Engine, Digital Ocean, Zendesk, AWS




July 11, 2024 18:28 UTC
[Resolved] This incident is now resolved and no further errors are being seen on GitLab.com. Our investigation points to a disk performance issue. You can see further details and corrective actions in gitlab.com/gitlab-com/gl-infra/production/-/issues/18269.

July 11, 2024 15:51 UTC
[Investigating] Our investigations are ongoing. We will post our next update when we have more information. Please follow the incident issue for updates here: gitlab.com/gitlab-com/gl-infra/production/-/issues/18269

July 11, 2024 15:28 UTC
[Investigating] No material updates to report. We're continuing to investigate the issue. Please follow the incident issue for updates here: gitlab.com/gitlab-com/gl-infra/production/-/issues/18269

July 11, 2024 15:00 UTC
[Investigating] We are seeing improved service availability and investigations continue for this problem. Please follow the incident issue for updates here: gitlab.com/gitlab-com/gl-infra/production/-/issues/18269

July 11, 2024 14:43 UTC
[Investigating] We are seeing improved service availability and investigations continue for this problem. We are planning to make the incident issue publicly available as soon as possible.

July 11, 2024 14:29 UTC
[Investigating] We are investigating a problem with gitlab.com being unavailable for some users. Our investigations continue and we will post further status updates shortly.

July 11, 2024 14:18 UTC
[Investigating] We are investigating a problem with gitlab.com being unavailable for some users. We will post further status updates shortly.

Intermit 500 errors on packages.gitlab.com

July 5, 2024 22:09 UTC

Incident Status

Operational


Components

packages.gitlab.com


Locations

AWS




July 5, 2024 22:09 UTC
[Resolved] Service has stabilized after the implemented fix. Access should be fully restored to packages.gitlab.com site. Marking as resolved.

July 5, 2024 21:24 UTC
[Monitoring] We have implemented a possible fix. Service has been restored to packages.gitlab.com . Users should no longer see a 500 error at login. We will continue to monitor packages.gitlab.com to make sure it's stable before we mark it as resolved.

July 5, 2024 20:37 UTC
[Investigating] packages.gitlab.com is experiencing intermit 500 errors. We are currently in the process of investigating the cause at this time. This will not allow new downloads of our GitLab product. We will update once we have more information or at the hour.

July 5, 2024 19:51 UTC
[Investigating] packages.gitlab.com is experiencing intermit 500 errors. We are currently in the process of investigating the cause at this time. This will not allow new downloads of our GitLab product. We will update once we have more information or at the hour.

July 5, 2024 18:43 UTC
[Investigating] packages.gitlab.com is experiencing intermit 500 errors. We are currently in the process of investigating the cause at this time. This will not allow new downloads of our GitLab product.

July 5, 2024 18:39 UTC
[Investigating] packages.gitlab.com is experiencing intermit 500 errors. We are currently in the process of investigating the cause at this time. This will not allow new downloads of our GitLab product.





Back to current status