All Systems Operational

Updated a few seconds ago

Back to current status

Status History

Filter: API (Clear)



July 2024

GitLab.com is unavailable

July 11, 2024 18:28 UTC

GitLab.com is unavailableService Disruption

Incident Status

Service Disruption


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions, SAML SSO - GitLab SaaS, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, docs.gitlab.com, Canary, Product Analytics - Configurator (Beta), Product Analytics - Collector (Beta), Product Analytics - Background processing (Beta), Product Analytics - Queries (Beta), GitLab Duo


Locations

Google Compute Engine, Digital Ocean, Zendesk, AWS




July 11, 2024 18:28 UTC
[Resolved] This incident is now resolved and no further errors are being seen on GitLab.com. Our investigation points to a disk performance issue. You can see further details and corrective actions in gitlab.com/gitlab-com/gl-infra/production/-/issues/18269.

July 11, 2024 15:51 UTC
[Investigating] Our investigations are ongoing. We will post our next update when we have more information. Please follow the incident issue for updates here: gitlab.com/gitlab-com/gl-infra/production/-/issues/18269

July 11, 2024 15:28 UTC
[Investigating] No material updates to report. We're continuing to investigate the issue. Please follow the incident issue for updates here: gitlab.com/gitlab-com/gl-infra/production/-/issues/18269

July 11, 2024 15:00 UTC
[Investigating] We are seeing improved service availability and investigations continue for this problem. Please follow the incident issue for updates here: gitlab.com/gitlab-com/gl-infra/production/-/issues/18269

July 11, 2024 14:43 UTC
[Investigating] We are seeing improved service availability and investigations continue for this problem. We are planning to make the incident issue publicly available as soon as possible.

July 11, 2024 14:29 UTC
[Investigating] We are investigating a problem with gitlab.com being unavailable for some users. Our investigations continue and we will post further status updates shortly.

July 11, 2024 14:18 UTC
[Investigating] We are investigating a problem with gitlab.com being unavailable for some users. We will post further status updates shortly.

Migration by direct transfer and importing with GitHub, Bitbucket Server, and Gitea importer offline

July 8, 2024 11:27 UTC

Incident Status

Service Disruption


Components

Website, API


Locations

Google Compute Engine




July 8, 2024 11:27 UTC
[Resolved] A fix has been implemented and the service is now stable. The Bulk import direct transfer functionality is available again.

July 2, 2024 16:05 UTC
[Investigating] Work is continuing on the development of a fix for the issue. Further updates will be provided on the status page once we have a timeline for the deployment of the fix.

June 28, 2024 04:55 UTC
[Investigating] Work is continuing on the development of a fix for the issue. Further updates will be provided on the status page once we have a timeline for the deployment of the fix.

June 7, 2024 14:34 UTC
[Investigating] We are working to re-enable these features and further updates will be available on the status page - status.gitlab.com

June 6, 2024 09:13 UTC
[Investigating] Migration by direct transfer is currently offline and we are currently investigating the cause. We will update the status page when more information becomes available.

June 6, 2024 04:23 UTC
[Investigating] Bulk import direct transfer functionality is currently offline and we are currently investigating the cause. We will update the status page when more information becomes available.

June 6, 2024 00:28 UTC
[Investigating] Bulk import functionality is currently offline and we are currently investigating the cause. We will update the status page when more information becomes available.

Increase in GitLab.com's Ci jobs processing time

July 2, 2024 16:01 UTC

Incident Status

Operational


Components

Website, API, Background Processing


Locations

Google Compute Engine




July 2, 2024 16:01 UTC
[Resolved] This incident is now mitigated. More details here: gitlab.com/gitlab-com/gl-infra/production/-/issues/18229

July 2, 2024 15:43 UTC
[Monitoring] We rolled back a recent change to mitigate the issue. Performance is now back to normal. More details here: gitlab.com/gitlab-com/gl-infra/production/-/issues/18229

July 2, 2024 15:28 UTC
[Investigating] We're noticing an increase in GitLab.com's Ci jobs processing time. We're now investigating this issue. More details here: gitlab.com/gitlab-com/gl-infra/production/-/issues/18229

July 2, 2024 15:26 UTC
[Investigating] We're noticing an increase in GitLab.com's Ci jobs processing time. We're now investigating this issue. More details here: gitlab.com/gitlab-com/gl-infra/production/-/issues/18229

June 2024

Multiple reports of authentication failures accessing GitLab.com

June 18, 2024 09:03 UTC

Incident Status

Operational


Components

Website, API, Git Operations, Container Registry


Locations

Google Compute Engine




June 18, 2024 09:03 UTC
[Resolved] This incident has been resolved. More information can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/18003.

May 30, 2024 13:02 UTC
[Monitoring] We have temporarily increased the rate limits for unauthenticated requests, to alleviate errors resulting from the token lifetime limits implementation. We are continuing to monitor and adjust these rate limits as appropriate.

May 14, 2024 16:12 UTC
[Monitoring] We have temporarily increased the rate limits for unauthenticated requests, to alleviate errors resulting from the token lifetime limits implementation. Please see the issue for more details: gitlab.com/gitlab-com/gl-infra/production/-/issues/18003

May 14, 2024 03:27 UTC
[Identified] Token lifetime limits have been implemented on GitLab.com. If you are experiencing increased authentication errors accessing GitLab.com, please see our blog post: about.gitlab.com/blog/2023/10/25/access-token-lifetime-limits for details on how to mitigate problems with authentication.

May 2024

Code Suggestions failing with 403 errors

May 24, 2024 12:34 UTC

Incident Status

Partial Service Disruption


Components

API


Locations

Google Compute Engine




May 24, 2024 12:34 UTC
[Resolved] Monitoring has not surfaced any further errors, so the incident is now considered resolved. See gitlab.com/gitlab-com/gl-infra/production/-/issues/18062 for more details

May 24, 2024 11:15 UTC
[Monitoring] We've identified the cause and applied a fix, and we are now no longer seeing failing requests. We will continue to monitor. See gitlab.com/gitlab-com/gl-infra/production/-/issues/18062 for more details

May 24, 2024 10:55 UTC
[Investigating] No material updates at this time; our investigation is ongoing. Details will be posted in: gitlab.com/gitlab-com/gl-infra/production/-/issues/18062

May 24, 2024 10:38 UTC
[Investigating] We are aware of a large percentage of GitLab Duo Code Suggestions requests failing with error 403. All IDE plugins are impacted. Details will be provided in: gitlab.com/gitlab-com/gl-infra/production/-/issues/18062

Pipeline jobs not being picked up and 500 errors

May 17, 2024 02:14 UTC

Incident Status

Partial Service Disruption


Components

Website, API, CI/CD - Hosted runners on Linux, Background Processing


Locations

Google Compute Engine




May 17, 2024 02:14 UTC
[Resolved] Ongoing monitoring is showing that all systems are working without issue. Google has also resolved their incident. We will mark this incident as resolved. See following issue for more information: gitlab.com/gitlab-com/gl-infra/production/-/issues/18023

May 17, 2024 00:48 UTC
[Monitoring] We are seeing normal functionality again in our logging but we will continue monitoring. If you are still seeing these issues, please comment in the production issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/18023

May 17, 2024 00:24 UTC
[Identified] We have identified that these outages are impacted by the Google Cloud Service disruptions. For more information, see gitlab.com/gitlab-com/gl-infra/production/-/issues/18023. To follow Google Cloud status information, see status.cloud.google.com/incidents/xVSEV3kVaJBmS7SZbnre#RP1d9aZLNFZEJmTBk8e1

May 17, 2024 00:01 UTC
[Investigating] GitLab is investigating reports of pipelines not being picked up and 500 errors. We have two incidents open and we are working to understand the cause of these errors and whether they are related. We are investigating if this is related to this Google issue: status.cloud.google.com/incidents/xVSEV3kVaJBmS7SZbnre#RP1d9aZLNFZEJmTBk8e1 Issue links: gitlab.com/gitlab-com/gl-infra/production/-/issues/18024, gitlab.com/gitlab-com/gl-infra/production/-/issues/18023

April 2024

Errors managing Kubernetes agents

April 23, 2024 15:13 UTC

Errors managing Kubernetes agentsPartial Service Disruption

Incident Status

Partial Service Disruption


Components

Website, API


Locations

Google Compute Engine




April 23, 2024 15:13 UTC
[Resolved] The fix for this issue was deployed and this incident has been resolved. More information is available in gitlab.com/gitlab-com/gl-infra/production/-/issues/17887.

April 23, 2024 03:36 UTC
[Identified] A fix has been merged and will be available in the next deployment tomorrow morning. Please follow gitlab.com/gitlab-com/gl-infra/production/-/issues/17887 for more details

April 23, 2024 02:52 UTC
[Identified] We are in the process of merging code changes to fix the issue. Next status update will be once the Merge Request is merged. Please follow gitlab.com/gitlab-com/gl-infra/production/-/issues/17887 for more details

April 23, 2024 02:23 UTC
[Identified] We have identified the potential cause of the issue and we are working on mitigating the issue. See gitlab.com/gitlab-com/gl-infra/production/-/issues/17887 for more details

April 23, 2024 02:08 UTC
[Investigating] No material updates to report. We're continuing to investigate the issue. Tracking in gitlab.com/gitlab-com/gl-infra/production/-/issues/17887

April 23, 2024 01:50 UTC
[Investigating] Some users are experiencing errors managing Kubernetes agents, or listing existing agents. We are actively investigating the issue. Details can be found in: gitlab.com/gitlab-com/gl-infra/production/-/issues/17887

500 errors upon groups creation via API

April 4, 2024 13:17 UTC

500 errors upon groups creation via APIPartial Service Disruption

Incident Status

Partial Service Disruption


Components

API


Locations

Google Compute Engine




April 4, 2024 13:17 UTC
[Resolved] This incident has been resolved and the API endpoint is now operational. More information can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/17775.

April 3, 2024 23:08 UTC
[Identified] The fix has been merged and t is currently in the middle of our auto-deploy process. It shall land into production in the next deploy cycle. Next update once fix is deployed to production. See gitlab.com/gitlab-com/gl-infra/production/-/issues/17775 for more details.

April 3, 2024 19:28 UTC
[Identified] We are working on a code Merge Request to fix this issue. Next status update will be once the Merge Request is merged. Please follow gitlab.com/gitlab-com/gl-infra/production/-/issues/17775 for details.

April 3, 2024 19:05 UTC
[Identified] We originally reported project creation was also affected by this incident, however, we would like to clarify that we've only found evidence of group creation being affected. See gitlab.com/gitlab-com/gl-infra/production/-/issues/17775 for details.

April 3, 2024 18:49 UTC
[Identified] We have identified the recent deprecation of the "emails_disabled" attribute in favor of "emails_enabled" attribute as the cause for this issue and we are working on a solution. See gitlab.com/gitlab-com/gl-infra/production/-/issues/17775#note_1844428763 for more details.

April 3, 2024 18:16 UTC
[Investigating] API calls to create groups/projects on GitLab.com are receiving a 500 response if the "emails_disabled" attribute is specified. Please follow gitlab.com/gitlab-com/gl-infra/production/-/issues/17775 for more details

March 2024

Package registry upload issues

March 23, 2024 00:03 UTC

Package registry upload issuesDegraded Performance

Incident Status

Degraded Performance


Components

API


Locations

Google Compute Engine




March 23, 2024 00:03 UTC
[Resolved] GitLab is no longer observing errors with artifact and package repository uploads. We are resolving this incident at this time. More details in gitlab.com/gitlab-com/gl-infra/production/-/issues/17749

March 22, 2024 23:47 UTC
[Monitoring] GitLab believes that issues with uploads to the package registry and artifact uploads have been identified and resolved. We are monitoring for the next fifteen minutes to ensure that no further issues are observed before we resolve this incident. More details in gitlab.com/gitlab-com/gl-infra/production/-/issues/17749

March 22, 2024 23:04 UTC
[Investigating] GitLab believes that package registry and artifact issues is related to inbound HTTP2 connections, which was exposed by the CloudFlare setting "HTTP/2 to origin". We have disabled this feature and are monitoring to verify uploads are working. We will update in 30 minutes. More details in gitlab.com/gitlab-com/gl-infra/production/-/issues/17749

March 22, 2024 21:16 UTC
[Investigating] GitLab is continuing to investigate artifact and package registry upload issues. We are continuing work at understanding the root cause of these upload errors. We will post our next update in one hour. More details in gitlab.com/gitlab-com/gl-infra/production/-/issues/17749

March 22, 2024 20:44 UTC
[Investigating] GitLab is continuing to investigate artifact and package registry upload issues. GitLab is monitoring the issue and working CloudFlare to understand these connection issues. We will post our next update in 30 minutes. More details in gitlab.com/gitlab-com/gl-infra/production/-/issues/17749

March 22, 2024 20:05 UTC
[Investigating] GitLab is continuing to investigate artifact and package registry upload issues. GitLab is monitoring the issue and working CloudFlare to understand these connection issues. We will post our next update in 30 minutes. More details in gitlab.com/gitlab-com/gl-infra/production/-/issues/17749

March 22, 2024 19:27 UTC
[Investigating] GitLab is continuing to investigate artifact and package registry upload issues. GitLab is monitoring the issue and working CloudFlare to understand these connection issues. We will post our next update in 30 minutes. More details in gitlab.com/gitlab-com/gl-infra/production/-/issues/17749

March 22, 2024 19:09 UTC
[Investigating] GitLab is continuing to investigate artifact and package registry upload issues. GitLab is monitoring the issue and working CloudFlare to understand these connection issues. More details in gitlab.com/gitlab-com/gl-infra/production/-/issues/17749

March 22, 2024 18:51 UTC
[Investigating] GitLab is continuing to investigate artifact and package registry upload issues. We have reached out to CloudFlare so we can understand the connection issues we are observing. More details in gitlab.com/gitlab-com/gl-infra/production/-/issues/17749

March 22, 2024 18:35 UTC
[Investigating] GitLab is continuing to investigate artifact and package registry upload issues. We have reached out to CloudFlare so we can understand the connection issues we are observing. More details in gitlab.com/gitlab-com/gl-infra/production/-/issues/17749

March 22, 2024 18:15 UTC
[Investigating] GitLab is investigating issues with package and artifact uploads. We have observed elevated error rates with uploads and we are working to understand the root cause. More details in gitlab.com/gitlab-com/gl-infra/production/-/issues/17749

February 2024

GitLab Agent timeout issues

February 27, 2024 04:01 UTC

GitLab Agent timeout issuesDegraded Performance

Incident Status

Degraded Performance


Components

API


Locations

Google Compute Engine




February 27, 2024 04:01 UTC
[Resolved] The issue has been resolved and the affected services are now operating normally. Further details are available in gitlab.com/gitlab-com/gl-infra/production/-/issues/17655

February 26, 2024 23:46 UTC
[Monitoring] The resolution has been merged. Waiting for builds to complete. We will continue to monitor. Connectivity does appear to be resolved at this time.

February 26, 2024 21:02 UTC
[Monitoring] We have a permanent fix ready to deploy. We are working through other issues preventing us from deploying this fix. For now, We will continue to monitor. Most connectivity does appear to be resolved at this time.

February 26, 2024 18:15 UTC
[Identified] We are in the process of resolving the issue. Work on this issue continues and may take around 2h-3h until it is fully deployed on GitLab.com. More details can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/17655

February 26, 2024 15:08 UTC
[Identified] We are in the process of resolving the issue. This process will take around 2h-4h until it is fully deployed on GitLab.com. More details can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/17655

February 26, 2024 12:22 UTC
[Identified] We are in the process of resolving the issue. More details can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/17655

February 26, 2024 11:55 UTC
[Identified] We are working on resolving GitLab Agent timeout issues. Users might experience 500 errors in CI jobs when using it. More details can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/17655

January 2024

System notes authored by a deleted user/bot prevent any Work Item discussions from being loaded

January 22, 2024 19:05 UTC

Incident Status

Partial Service Disruption


Components

Website, API


Locations

Google Compute Engine




January 22, 2024 19:05 UTC
[Resolved] We have now confirmed that the issue has been resolved by the implemented fix. Please review gitlab.com/gitlab-com/gl-infra/production/-/issues/17436 for details.

January 22, 2024 17:49 UTC
[Monitoring] The fix has now been confirmed to be in production. We will monitor the status of this incident to confirm it is solved. For more details continue to follow gitlab.com/gitlab-com/gl-infra/production/-/issues/17436.

January 22, 2024 17:29 UTC
[Identified] Making a slight correction to our previous update, the fix has been deployed to our pre-production environment and is pending to land on production. Follow gitlab.com/gitlab-com/gl-infra/production/-/issues/17436 for more details.

January 22, 2024 16:58 UTC
[Monitoring] The deployment of the fix to production is complete. We will continue to monitor the situation. For details, please see: gitlab.com/gitlab-com/gl-infra/production/-/issues/17436

January 22, 2024 09:52 UTC
[Identified] We are expecting the deployment of the fix to be in production in approx. 4.5 h. We will post our next update when the deployment is complete. For details, please see: gitlab.com/gitlab-com/gl-infra/production/-/issues/17436

January 22, 2024 09:15 UTC
[Identified] The MR has been merged, and we're waiting for the fix to be deployed. For details, please see: gitlab.com/gitlab-com/gl-infra/production/-/issues/17436

January 22, 2024 08:35 UTC
[Identified] We're waiting for the fix to be deployed. For details, please see: gitlab.com/gitlab-com/gl-infra/production/-/issues/17436

January 22, 2024 07:59 UTC
[Identified] We continue working on resolving the problem. A tentative fix is in progress. For details, please see: gitlab.com/gitlab-com/gl-infra/production/-/issues/17436

January 22, 2024 07:12 UTC
[Identified] We have identified an issue preventing discussion loading and are actively working to resolve it. We will provide an update within the next hour. Details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/17436

January 22, 2024 06:41 UTC
[Investigating] We became aware of an issue that the system notes by deleted users or bots prevent users from loading the discussions on the page. We are now investigating the root cause and will provide updates along the way. See gitlab.com/gitlab-com/gl-infra/production/-/issues/17436 for details.

November 2023

Merge requests incorrectly blocked by denied license

November 6, 2023 16:12 UTC

Incident Status

Partial Service Disruption


Components

Website, API


Locations

Google Compute Engine




November 6, 2023 16:12 UTC
[Resolved] The issue has been successfully resolved. As a result, the blockage of merge requests due to license denials is no longer expected to occur. For a comprehensive timeline of the incident, please refer to: gitlab.com/gitlab-com/gl-infra/production/-/issues/17082

November 6, 2023 16:10 UTC
[Resolved] The MR gitlab.com/gitlab-org/gitlab/-/merge_requests/135986 has been deployed to production, resolving this incident.

November 3, 2023 21:53 UTC
[Identified] The planned mitigation has been verified and committed and will be deployed on Monday. In the meantime a workaround is to create a scan result policy approval to unblock your merge request. gitlab.com/gitlab-com/gl-infra/production/-/issues/17082#note_1634269721

November 3, 2023 21:16 UTC
[Identified] The incident response team has a potential mitigation path planned, and is working to deploy a fix as soon as it's verified. Details may be found in this issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/17082

November 3, 2023 20:43 UTC
[Identified] We are investigating reports of merge requests incorrectly being blocked due to denied licenses

October 2023

GitLab.com availability issues

October 30, 2023 18:33 UTC

Incident Status

Service Disruption


Components

Website, API, Git Operations


Locations

Google Compute Engine




October 30, 2023 18:33 UTC
[Resolved] Import functionality has been restored and operation of GitLab.com is fully returned to normal. We're marking the incident resolved, please see the issue for further details: gitlab.com/gitlab-com/gl-infra/production/-/issues/17054

October 30, 2023 17:58 UTC
[Monitoring] The incident response team has mitigated the root cause of the performance impacts and is now working to restore full import functionality. Monitoring is ongoing to ensure there are no further impacts. Issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/17054

October 30, 2023 17:28 UTC
[Identified] Performance continues to improve and the team is continuing to work on mitigations to prevent future impact.

October 30, 2023 17:09 UTC
[Identified] We continue to work towards full mitigation. Performance of GitLab.com is returning, though project import remains partially disabled while the team investigates.

October 30, 2023 16:54 UTC
[Identified] A likely cause of the database load has been identified. Our incident response team has temporarily disabled part of the project import feature while we continue to investigate.

October 30, 2023 16:32 UTC
[Investigating] We are seeing signs of improvement, and the team has narrowed the potential causes. Mitigation and additional investigation are in progress.

October 30, 2023 16:23 UTC
[Investigating] We've confirmed potential impacts to Git operations and API as well. Investigation still ongoing.

October 30, 2023 16:10 UTC
[Investigating] The incident response team is continuing to investigate causes of the increased database usage, additional teams are being brought in to assist.

October 30, 2023 15:55 UTC
[Investigating] We are investigating increased database usage as a primary cause. We will update as more information becomes available.

October 30, 2023 15:39 UTC
[Investigating] We are currently investigating availability issues with on GitLab.com. We will update as more information becomes available.

Connectivity problems when interacting with the GitLab KAS API

October 12, 2023 04:45 UTC

Incident Status

Operational


Components

API


Locations

Google Compute Engine




October 12, 2023 04:45 UTC
[Resolved] There has been no further sign of increase in errors in GitLab KAS API since we took steps to mitigate the problem. We will continue to investigate the root cause and all future updates will be found on the issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/16970

October 12, 2023 03:49 UTC
[Monitoring] No further impact has been seen since mitigation steps were taken for this. We will continue to investigate the root cause and all future updates will be found on the issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/16970

October 12, 2023 02:41 UTC
[Monitoring] No further impact has been seen since mitigation steps were taken. We will continue to investigate the root cause. For all updates refer to: gitlab.com/gitlab-com/gl-infra/production/-/issues/16970

October 12, 2023 00:48 UTC
[Investigating] The steps taken to mitigate the problem remain successful. We are continuing to investigate the root cause. The next update will be within the next 90 minutes. Additional information is available in the gitlab.com/gitlab-com/gl-infra/production/-/issues/16970 issue.

October 11, 2023 23:28 UTC
[Investigating] The steps taken to mitigate the problem remain successful. We are continuing to investigate the root cause. The next update will be within the next 90 minutes. Additional information is available in the gitlab.com/gitlab-com/gl-infra/production/-/issues/16970 issue.

October 11, 2023 22:00 UTC
[Investigating] We are continuing to investigate and have taken steps to mitigate the impact of this problem. The next update will be within the next 90 minutes. Additional information is available in the gitlab.com/gitlab-com/gl-infra/production/-/issues/16970 issue.

October 11, 2023 21:00 UTC
[Investigating] We are continuing to investigate the cause of the 429 errors caused by this problem. The next update will be within the next 60 minutes. Additional information is available in the gitlab.com/gitlab-com/gl-infra/production/-/issues/16970 issue.

October 11, 2023 20:01 UTC
[Investigating] We are continuing to investigate the cause of this problem. The next update will be within the next 60 minutes. Additional information is available in the gitlab.com/gitlab-com/gl-infra/production/-/issues/16970 issue.

October 11, 2023 19:27 UTC
[Investigating] We are continuing to investigate the cause of this problem. The next update will be within the next 30 minutes. Additional information is available in the gitlab.com/gitlab-com/gl-infra/production/-/issues/16970 issue.

October 11, 2023 19:08 UTC
[Investigating] We are continuing to investigate the cause of this problem.

October 11, 2023 18:58 UTC
[Investigating] We are investigating elevated error rates when interacting with the GitLab KAS API. More information is in the production issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/16970.

September 2023

Elevated error reates and slow response times

September 13, 2023 09:29 UTC

Incident Status

Degraded Performance


Components

Website, API, Git Operations, Container Registry


Locations

Google Compute Engine




September 13, 2023 09:29 UTC
[Resolved] The issue has been resolved and services are operating normally. The investigation summary will be posted in gitlab.com/gitlab-com/gl-infra/production/-/issues/16359

September 13, 2023 08:23 UTC
[Monitoring] We're now seeing error rates and performance returning back to normal operation. We're continuing to monitor to ensure the problem does not recur. Details in gitlab.com/gitlab-com/gl-infra/production/-/issues/16359

September 13, 2023 08:09 UTC
[Investigating] We are investigating an increase in errors and slower response times on GitLab.com. For details see gitlab.com/gitlab-com/gl-infra/production/-/issues/16359

Elevated error rates impacting API calls

September 12, 2023 16:38 UTC

Incident Status

Operational


Components

API


Locations

Google Compute Engine




September 12, 2023 16:38 UTC
[Resolved] Reverting the change had an immediate effect and error rates have returned to normal. This incident has been mitigated. Further details in gitlab.com/gitlab-com/gl-infra/production/-/issues/16355

September 12, 2023 16:32 UTC
[Identified] Our team has identified the likely root cause and the change has been reverted. We’re currently monitoring the situation but error rates are decreasing. gitlab.com/gitlab-com/gl-infra/production/-/issues/16355

September 12, 2023 16:20 UTC
[Investigating] Our team is currently investigating elevated errors rates that are impacting API calls. More information in gitlab.com/gitlab-com/gl-infra/production/-/issues/16355

Main DB Cluster PostgreSQL 14 Upgrade

September 9, 2023 17:25 UTC

Description

We will be undergoing scheduled maintenance to our database layer. The maintenance will take up to 6 hours, from 14:00 UTC to 20:00 UTC. GitLab.com will continue to be available during the maintenance window. We apologize in advance for any inconvenience this may cause. See gitlab.com/gitlab-com/gl-infra/production/-/issues/16266 for details.


Components

Website, API, Git Operations, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners for GitLab community contributions


Locations

Google Compute Engine


Schedule

September 9, 2023 14:00 - September 9, 2023 20:00 UTC



September 9, 2023 17:25 UTC
[Update] GitLab.com planned maintenance for the database layer is complete. We'll be monitoring the platform to ensure all systems are functioning correctly. Thank you for your patience.

September 9, 2023 16:12 UTC
[Update] GitLab.com planned maintenance for the database layer is almost complete. We’re continuing to verify that all systems are functioning correctly. Thank you for your patience. See <gitlab.com/gitlab-com/gl-infra/production/-/issues/16266>

September 9, 2023 13:01 UTC
[Update] We will be undergoing scheduled maintenance to our main database layer in 1 hour. The maintenance will take up to 5 hours starting from 14:00 UTC to 19:00 UTC. GitLab.com will be available but users may experience degraded performance during the maintenance window. We apologize in advance for any inconvenience this may cause. See gitlab.com/gitlab-com/gl-infra/production/-/issues/16266

September 9, 2023 10:08 UTC
[Update] We will be undergoing scheduled maintenance to our main database layer in 3 hours. The maintenance will take up to 5 hours starting from 14:00 UTC to 19:00 UTC. GitLab.com will be available but users may experience degraded performance during the maintenance window. We apologize in advance for any inconvenience this may cause. See gitlab.com/gitlab-com/gl-infra/production/-/issues/16266

September 8, 2023 13:59 UTC
[Update] Reminder: Tomorrow, we will be undergoing scheduled maintenance to our main database layer. The maintenance will take up to 5 hours starting from 14:00 UTC to 19:00 UTC. GitLab.com will be available but users may experience degraded performance during the maintenance window. We apologize in advance for any inconvenience this may cause. See gitlab.com/gitlab-com/gl-infra/production/-/issues/16266

September 6, 2023 12:13 UTC
[Update] In 3 days, we will be undergoing scheduled maintenance to our main database layer. The maintenance will take up to 5 hours starting from 14:00 UTC to 19:00 UTC. GitLab.com will be available but users may experience degraded performance during the maintenance window. We apologize in advance for any inconvenience this may cause. See #16266

August 31, 2023 00:01 UTC
[Update] Next week, we will be undergoing scheduled maintenance to our main database layer. The maintenance will take up to 5 hours starting from 14:00 UTC to 19:00 UTC. GitLab.com will be available but users may experience degraded performance during the maintenance window. We apologize in advance for any inconvenience this may cause. See <gitlab.com/gitlab-com/gl-infra/production/-/issues/14403>

August 2023

Main DB Cluster PostgreSQL 14 Upgrade

August 26, 2023 19:51 UTC

Description

We will be undergoing scheduled maintenance to our database layer. The maintenance will take up to 5 hours, from 14:00 UTC to 19:00 UTC. GitLab.com will continue to be available during the maintenance window. We apologize in advance for any inconvenience this may cause. See gitlab.com/gitlab-com/gl-infra/production/-/issues/14403 for details.


Components

Website, API, Git Operations, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners for GitLab community contributions


Locations

Google Compute Engine


Schedule

August 26, 2023 14:00 - August 26, 2023 19:00 UTC



August 26, 2023 19:51 UTC
[Update] GitLab.com rollback for the database layer is complete, and we're back up and running. We'll be monitoring the platform to ensure all systems are functioning correctly. Thank you for your patience.

August 26, 2023 19:21 UTC
[Update] GitLab.com rollback for the database layer is almost complete. The site is back up and running although we're continuing to verify that all systems are functioning correctly. Thank you for your patience.

August 26, 2023 18:29 UTC
[Update] Due to an issue during the planned maintenance for the database layer, we have initiated a rollback of the changes. We will send another update within the next 30 minutes.

August 26, 2023 14:11 UTC
[Update] We will be starting scheduled maintenance to our main database layer now. The maintenance will take up to 5 hours. GitLab.com will be available but users may experience degraded performance during the maintenance window. We apologize in advance for any inconvenience this may cause. See gitlab.com/gitlab-com/gl-infra/production/-/issues/14403

August 26, 2023 13:01 UTC
[Update] We will be undergoing scheduled maintenance to our main database layer in 1 hour. The maintenance will take up to 5 hours starting from 14:00 UTC to 19:00 UTC. GitLab.com will be available but users may experience degraded performance during the maintenance window. We apologize in advance for any inconvenience this may cause. See gitlab.com/gitlab-com/gl-infra/production/-/issues/14403

August 26, 2023 11:06 UTC
[Update] We will be undergoing scheduled maintenance to our main database layer in 3 hours. The maintenance will take up to 5 hours starting from 14:00 UTC to 19:00 UTC. GitLab.com will be available but users may experience degraded performance during the maintenance window. We apologize in advance for any inconvenience this may cause. See gitlab.com/gitlab-com/gl-infra/production/-/issues/14403

August 25, 2023 15:41 UTC
[Update] Reminder: Tomorrow, we will be undergoing scheduled maintenance to our main database layer. The maintenance will take up to 5 hours starting from 14:00 UTC to 19:00 UTC. GitLab.com will be available but users may experience degraded performance during the maintenance window. We apologize in advance for any inconvenience this may cause. See gitlab.com/gitlab-com/gl-infra/production/-/issues/14403

August 23, 2023 12:09 UTC
[Update] In 3 days, we will be undergoing scheduled maintenance to our main database layer. The maintenance will take up to 5 hours starting from 14:00 UTC to 19:00 UTC. GitLab.com will be available but users may experience degraded performance during the maintenance window. We apologize in advance for any inconvenience this may cause. See gitlab.com/gitlab-com/gl-infra/production/-/issues/14403

August 20, 2023 21:13 UTC
[Update] Next week, we will be undergoing scheduled maintenance to our main database layer. The maintenance will take up to 5 hours starting from 14:00 UTC to 19:00 UTC. GitLab.com will be available but users may experience degraded performance during the maintenance window. We apologize in advance for any inconvenience this may cause. See <gitlab.com/gitlab-com/gl-infra/production/-/issues/14403>

SidekiqQueueTooLarge - ElasticSearch - gitlab.com returning 500

August 26, 2023 03:38 UTC

Incident Status

Partial Service Disruption


Components

Website, API


Locations

Google Compute Engine




August 26, 2023 03:38 UTC
[Resolved] This incident has been resolved . Global search service is operating normally. A full timeline of this incident can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/16237

August 26, 2023 03:06 UTC
[Monitoring] Global search service is operating normally, and index is now up to date. More details can be found gitlab.com/gitlab-com/gl-infra/production/-/issues/16237

August 26, 2023 02:48 UTC
[Identified] Global search is restored. We’re now working to update the index. Until then, results will not show documents from the last 3 hours. More details can be found gitlab.com/gitlab-com/gl-infra/production/-/issues/16237

August 26, 2023 02:38 UTC
[Identified] We’re still working to restore global search. More details in gitlab.com/gitlab-com/gl-infra/production/-/issues/16237

August 26, 2023 02:13 UTC
[Identified] No further updates at this time. We continue to work towards mitigating the issue. Search continue to show some instability. Further details in gitlab.com/gitlab-com/gl-infra/production/-/issues/16237

August 26, 2023 01:35 UTC
[Identified] Search continue to show instability. Some requests are still failing. Our team is still actively working on this. More details in gitlab.com/gitlab-com/gl-infra/production/-/issues/16237

August 26, 2023 01:35 UTC
[Investigating] Search continue to show instability. Some requests are still failing. Our team is still actively working on this. More details in gitlab.com/gitlab-com/gl-infra/production/-/issues/16237

August 26, 2023 01:04 UTC
[Investigating] No material updates at this stage. Our team is still actively investigating the problem. More details in gitlab.com/gitlab-com/gl-infra/production/-/issues/16237

August 26, 2023 00:47 UTC
[Investigating] Our team is still actively investigating the problem. More details in gitlab.com/gitlab-com/gl-infra/production/-/issues/16237

August 26, 2023 00:25 UTC
[Investigating] We are aware of searches on gitlab.com returning 500. We are beginning investigation in to this now. For further information and status please see gitlab.com/gitlab-com/gl-infra/production/-/issues/16237

Slightly elevated error rates on Gitaly nodes

August 17, 2023 15:52 UTC

Incident Status

Degraded Performance


Components

Website, API, Git Operations


Locations

Google Compute Engine




August 17, 2023 15:52 UTC
[Resolved] This incident has been resolved. More information can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/16187.

August 17, 2023 14:57 UTC
[Monitoring] We have validated the release in our canary infrastructure. We are in the process of deploying the release production wide. More details can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/16187.

August 17, 2023 13:19 UTC
[Monitoring] We've deployed the fix to our canary infrastructure and validating if the issue is resolved. More details can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/16187.

August 17, 2023 11:14 UTC
[Monitoring] The mitigations steps are complete. We believe we also found the root cause of the issues. More details can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/16187.

August 17, 2023 09:41 UTC
[Identified] We are working on mitigating the issue. More details can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/16187.

August 17, 2023 08:23 UTC
[Identified] No current updates at this time, we are still rolling out the fix to mitigate the impact. More details can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/16187.

August 17, 2023 07:51 UTC
[Identified] We have identified a change that caused the incident and we're in the process of rolling out a fix to mitigate the impact. Meanwhile, we'll continue to manually monitor Gitaly nodes. More information on gitlab.com/gitlab-com/gl-infra/production/-/issues/16187

August 17, 2023 07:23 UTC
[Investigating] No materials for update. Our team is still actively investigate the problem. More information on gitlab.com/gitlab-com/gl-infra/production/-/issues/16187

August 17, 2023 06:08 UTC
[Investigating] We continue to see issue on Gitaly nodes. We're deploying another revert to a known working version. More information on gitlab.com/gitlab-com/gl-infra/production/-/issues/16187

August 17, 2023 04:54 UTC
[Investigating] While we continue to look for a root cause, we are reverting Gitaly to the last known good deployment. More information on gitlab.com/gitlab-com/gl-infra/production/-/issues/16187

August 17, 2023 03:27 UTC
[Investigating] No material updates at this time - our investigation is ongoing. More information on gitlab.com/gitlab-com/gl-infra/production/-/issues/16187.

August 17, 2023 02:13 UTC
[Investigating] All Gitaly nodes are operational. We are investigating and will update in the following hour. More details about this incident can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/16187.

August 17, 2023 01:27 UTC
[Investigating] We have noticed degraded performance with the Gitaly service and applied the fix. We are currently investigating for the root cause. More details about this incident can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/16187.





Back to current status