August 2024

Embedding DB Upgrade

August 31, 2024 04:58 UTC

Embedding DB UpgradePlanned Maintenance

Description

We will be undergoing scheduled maintenance to our database layer. GitLab.com will continue to be available during the maintenance window. For more details, see gitlab.com/gitlab-com/gl-infra/dbre/-/issues/228

Components

Website, API, Git Operations, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions

Locations

Google Compute Engine, AWS

Schedule

October 25, 2024 00:00 - October 30, 2024 01:00 UTC

---

August 31, 2024 04:58 UTC

[Update] We are cancelling the Embedding database upgrade. See gitlab.com/gitlab-com/gl-infra/dbre/-/issues/228 for details. All other DB upgrade schedules will go ahead as planned.

CI/CD token allowlist selection failing

August 27, 2024 14:52 UTC

CI/CD token allowlist selection failingPartial Service Disruption

Incident Status

Partial Service Disruption

Components

Website, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions

Locations

Google Compute Engine, AWS

Postmortem

https://gitlab.com/gitlab-com/gl-infra/production/-/issues/18446

---

August 27, 2024 14:52 UTC

[Resolved] The revert MR has been deployed to production, and we see that CI/CD token allowlist project selection renders properly now. Details will be provided in the incident issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/18446

August 27, 2024 03:55 UTC

[Identified] We have identified an issue that is causing CI/CD token allowlist project selection to fail. We are in the process of reverting the change that caused this. For more details see gitlab.com/gitlab-com/gl-infra/production/-/issues/18446.

Some requests timing out

August 23, 2024 03:59 UTC

Some requests timing outOperational

Incident Status

Operational

Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions, SAML SSO - GitLab SaaS, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, docs.gitlab.com, Canary, Product Analytics - Configurator (Beta), Product Analytics - Collector (Beta), Product Analytics - Background processing (Beta), Product Analytics - Queries (Beta), GitLab Duo

Locations

Google Compute Engine, Digital Ocean, Zendesk, AWS

---

August 23, 2024 03:59 UTC

[Resolved] The issue has been resolved and all functionality on GitLab.com is nominal. Please see gitlab.com/gitlab-com/gl-infra/production/-/issues/18435 for more details.

August 23, 2024 01:53 UTC

[Monitoring] The issue has been identified and the incident resolved. We will continue monitoring for a while.

August 23, 2024 01:27 UTC

[Identified] We have identified an underlying cause and are working to mitigate it. See gitlab.com/gitlab-com/gl-infra/production/-/issues/18435 for more details.

August 23, 2024 00:34 UTC

[Investigating] We have noticed that there are delays in web UI updates for example, MRs not showing up after submission, We are continuing to investigate - please see gitlab.com/gitlab-com/gl-infra/production/-/issues/18435 for more details.

August 23, 2024 00:27 UTC

[Investigating] Some requests are slow or timing out due to DB saturation. We are investigating. Please see [this issue](gitlab.com/gitlab-com/gl-infra/production/-/issues/18435) for further details.

Sidekiq processing delays

August 16, 2024 20:29 UTC

Sidekiq processing delaysDegraded Performance

Incident Status

Degraded Performance

Components

Website, Background Processing

Locations

Google Compute Engine

Postmortem

https://gitlab.com/gitlab-com/gl-infra/production/-/issues/18407

---

August 16, 2024 20:29 UTC

[Resolved] This incident is considered resolved. Thanks for your patience! Details will be provided in the incident issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/18407

August 16, 2024 20:05 UTC

[Identified] While background processing has stabilized, our team is still investigating the root cause. Updates will be provided when more information is available.

August 16, 2024 19:40 UTC

[Identified] Our team has identified and disabled a worker that was impacting performance. Sidekiq processing is improving and we are continuing to monitor the state of the background processing.

August 16, 2024 19:27 UTC

[Investigating] Our team is still investigating the processing delays. Some users might experience some errors with creating merge requests or other delays with processing background jobs.

August 16, 2024 19:07 UTC

[Investigating] Our team is currently investigating some sidekiq performance issues which is causing some delays in jobs processing. More information will be in the incident issue gitlab.com/gitlab-com/gl-infra/production/-/issues/18407

Documentation search is unavailable to some users

August 11, 2024 06:12 UTC

Documentation search is unavailable to some usersPartial Service Disruption

Incident Status

Partial Service Disruption

Components

docs.gitlab.com

Locations

Google Compute Engine

---

August 11, 2024 06:12 UTC

[Resolved] A temporary resolution has been deployed and testing indicates that search functionality has been restored.

August 10, 2024 06:15 UTC

[Identified] We’re continuing to investigate the issue with docs.gitlab.com Search functionality. We will provide an update once we have further information to share.

August 10, 2024 05:12 UTC

[Identified] We've determined that the issue is not limited to one geographic area - instead there is a third-party library request that is breaking the search functionality on docs.gitlab.com. There are several workarounds available while we evaluate potential solutions: - Searching the docs using a third party search engine such as Google - Navigating the docs directory within the repository ( gitlab.com/gitlab-org/gitlab/-/tree/master/doc ) - Using an adblocker extension such as uBlock Origin

August 10, 2024 03:44 UTC

[Investigating] Investigation into the unavailable search functionality continues. It appears this may be limited to users in certain geographical locations. Further updates will be provided when more information is known. Details to be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/18382.

August 10, 2024 03:11 UTC

[Investigating] We are aware that some users may not be able to perform searches on docs.gitlab.com. We are currently investigating the root cause - details to be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/18382.

Some Duo AI features not working correctly

August 9, 2024 00:56 UTC

Some Duo AI features not working correctlyPartial Service Disruption

Incident Status

Partial Service Disruption

Components

GitLab Duo

Locations

Google Compute Engine

---

August 9, 2024 00:56 UTC

[Resolved] GitLab is no longer seeing errors from Duo Chat which is inline with our upstream Anthropic provider status. For further details, see the incident page at gitlab.com/gitlab-com/gl-infra/production/-/issues/18377 and the upstream status page at status.anthropic.com

August 8, 2024 17:40 UTC

[Monitoring] GitLab has been continuing to monitor Duo Chat error rates, which have been decreasing over time. We will continue to monitor until the upstream Anthropic outage is resolved; we will only update here if this status changes. For more details, see the incident page at gitlab.com/gitlab-com/gl-infra/production/-/issues/18377 and the status page of the upstream provider at status.anthropic.com

August 8, 2024 16:19 UTC

[Monitoring] We are continuing to monitor reports of Duo Chat features returning errors due to an upstream outage. For more details, see the incident page at gitlab.com/gitlab-com/gl-infra/production/-/issues/18377 and the upstream status page at status.anthropic.com

August 8, 2024 16:01 UTC

[Investigating] We are seeing reports of Duo features returning errors due to an upstream outage. For more details, see the incident page at gitlab.com/gitlab-com/gl-infra/production/-/issues/18377 and the status page of the upstream provider at status.anthropic.com

All CVEs on the GitLab Advisory Database are returning 404s

August 8, 2024 04:18 UTC

All CVEs on the GitLab Advisory Database are returning 404sPartial Service Disruption

Incident Status

Partial Service Disruption

Components

Website

Locations

Google Compute Engine

---

August 8, 2024 04:18 UTC

[Resolved] We have determined that while CVEs on advisories.gitlab.com are returning 404s, this has no impact on the operations of gitlab.com. For more details, see gitlab.com/gitlab-com/gl-infra/production/-/issues/18374.

August 8, 2024 03:48 UTC

[Investigating] We are currently seeing that every CVE on advisories.gitlab.com is returning a 404. Investigation continues. For more details, see gitlab.com/gitlab-com/gl-infra/production/-/issues/18374

August 8, 2024 00:44 UTC

[Investigating] We are currently seeing that every CVE on advisories.gitlab.com is returning a 404. We are investigating the cause. For more details, see gitlab.com/gitlab-com/gl-infra/production/-/issues/18374

July 2024

Customers Portal Maintenance Mode

July 31, 2024 20:47 UTC

Customers Portal Maintenance ModePartial Service Disruption

Incident Status

Partial Service Disruption

Components

GitLab Customers Portal

Locations

Google Compute Engine

Postmortem

https://gitlab.com/gitlab-com/gl-infra/production/-/issues/18351

---

July 31, 2024 20:47 UTC

[Resolved] This incident has been resolved and the customer portal is now operational. More information can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/18351

July 31, 2024 20:28 UTC

[Monitoring] The Customers Portal is now accessible and operating normally. We are monitoring for the next fifteen minutes to ensure that no further issues are observed before we resolve this incident. Details in gitlab.com/gitlab-com/gl-infra/production/-/issues/18351

July 31, 2024 20:11 UTC

[Identified] The Customers Portal (customers.gitlab.com) is currently in maintenance mode due to a 3rd party API experiencing issues at the moment. For more information see gitlab.com/gitlab-com/gl-infra/production/-/issues/18351

Some Duo AI features not working correctly

July 31, 2024 14:31 UTC

Some Duo AI features not working correctlyPartial Service Disruption

Incident Status

Partial Service Disruption

Components

GitLab Duo

Locations

Google Compute Engine

Postmortem

https://gitlab.com/gitlab-com/gl-infra/production/-/issues/18348

---

July 31, 2024 14:31 UTC

[Resolved] The issue has been resolved. Details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/18348

July 31, 2024 14:13 UTC

[Investigating] Some Duo AI features may not be working correctly. Engineers are investigating. Details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/18348

Signup with OAuth cannot be completed

July 30, 2024 13:19 UTC

Signup with OAuth cannot be completedPartial Service Disruption

Incident Status

Partial Service Disruption

Components

Website

Locations

Google Compute Engine

Postmortem

https://gitlab.com/gitlab-com/gl-infra/production/-/issues/18341

---

July 30, 2024 13:19 UTC

[Resolved] The rollback has now completed. Users signing up with OAuth are now able to complete the verification process.

July 30, 2024 12:47 UTC

[Identified] There are no material updates at this time. The rollback is in progress, and we will post again, once complete.

July 30, 2024 12:07 UTC

[Identified] There are no material updates to report. Our engineers are still working on rolling back the change that introduced the issue.

July 30, 2024 11:25 UTC

[Identified] No material updates to report. The rollback of the change that introduced the issue is still ongoing.

July 30, 2024 10:40 UTC

[Identified] There are no material updates to report. We are still in the process of rolling back the change that introduced the issue.

July 30, 2024 10:01 UTC

[Identified] No material updates at this point. We have identified the MR that introduced the issue, and are still rolling back. More details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/18341

July 30, 2024 09:25 UTC

[Identified] We are currently seeing issues for customers using Signup with OAuth SSO. We are in the process of reverting the change that introduced this issue. More details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/18341

Merge Request incorrectly rejected by expression in merge commit message

July 23, 2024 11:04 UTC

Merge Request incorrectly rejected by expression in merge commit message Partial Service Disruption

Incident Status

Partial Service Disruption

Components

Website

Locations

Google Compute Engine

---

July 23, 2024 11:04 UTC

[Resolved] We have confirmed that the fix has resolved the incident, and it's reflected in the number of errors we monitored. This incident is now resolved. More information in this issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/18313

July 23, 2024 10:09 UTC

[Monitoring] We've now merged the fix to the production environment and are currently monitoring for errors. The progress could be followed in this issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/18313

July 23, 2024 06:52 UTC

[Identified] The rollback has been merged and deployed to our staging environment. We are currently awaiting the next release to production, which should resolve the issue. For more details and workarounds, please refer to this issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/18313

July 22, 2024 21:33 UTC

[Identified] The rollback is merged and in our staging environment, and the next release that will allow it to go to production is tomorrow, 2024-07-23. For details: gitlab.com/gitlab-com/gl-infra/production/-/issues/18313

July 22, 2024 20:18 UTC

[Identified] The rollback of the change causing merge requests to be blocked by commit message rules incorrectly is currently in staging and awaiting release to production. gitlab.com/gitlab-com/gl-infra/production/-/issues/18313

July 22, 2024 10:01 UTC

[Identified] Some users are seeing MRs being incorrectly rejected by commit message rules. We have identified the cause and are in the process of rolling back the relevant MR. For further details see gitlab.com/gitlab-com/gl-infra/production/-/issues/18313

Repository tree not loading all contents

July 17, 2024 14:00 UTC

Repository tree not loading all contentsDegraded Performance

Incident Status

Degraded Performance

Components

Website

Locations

Google Compute Engine

---

July 17, 2024 14:00 UTC

[Resolved] We have confirmed that the fix allows the repository content to fully load. The incident is resolved. For more details, see gitlab.com/gitlab-com/gl-infra/production/-/issues/18286

July 17, 2024 03:17 UTC

[Identified] A fix has been pushed and is currently in the process of being deployed. More details: gitlab.com/gitlab-com/gl-infra/production/-/issues/18286

July 17, 2024 01:30 UTC

[Identified] We have identified an MR that we believe introduced the issue. We are performing tests locally to confirm. More details: gitlab.com/gitlab-com/gl-infra/production/-/issues/18286

July 16, 2024 21:18 UTC

[Investigating] Investigation is still ongoing to fix this issue. We will post the next update once we have identified the cause. Please follow gitlab.com/gitlab-com/gl-infra/production/-/issues/18286 for more details.

July 16, 2024 20:03 UTC

[Investigating] We're investigating the possibility of a recent change to gitaly as related to this issue. Please continue to follow gitlab.com/gitlab-com/gl-infra/production/-/issues/18286 for more details.

July 16, 2024 19:04 UTC

[Investigating] We continue to investigate the cause of this repository content visibility issue. Make sure to enable notifications at gitlab.com/gitlab-com/gl-infra/production/-/issues/18286 to track progress.

July 16, 2024 17:55 UTC

[Investigating] We are working across different teams to narrow down the cause of this issue. We will be updating this incident every hour or as soon as we have material updates to share. See gitlab.com/gitlab-com/gl-infra/production/-/issues/18286.

July 16, 2024 17:11 UTC

[Investigating] No material updates to provide. We continue to investigate this issue at gitlab.com/gitlab-com/gl-infra/production/-/issues/18286.

July 16, 2024 16:51 UTC

[Investigating] We continue to investigate the cause for this issue. Please follow gitlab.com/gitlab-com/gl-infra/production/-/issues/18286 for more details.

July 16, 2024 15:33 UTC

[Investigating] A issue has been identified where listings of repository files are truncated after a hundred entries. We are currently investigating the cause. For more details, see gitlab.com/gitlab-com/gl-infra/production/-/issues/18286

Projects mistakenly labeled as scheduled for deletion

July 16, 2024 16:39 UTC

Projects mistakenly labeled as scheduled for deletionOperational

Incident Status

Operational

Components

Website

Locations

Google Compute Engine

Postmortem

https://gitlab.com/gitlab-com/gl-infra/production/-/issues/18282

---

July 16, 2024 16:39 UTC

[Resolved] We have confirmed that projects are no longer showing the misleading deletion message. This incident is now resolved. Please see gitlab.com/gitlab-com/gl-infra/production/-/issues/18282 for more details.

July 16, 2024 15:36 UTC

[Monitoring] The fix has been deployed, and we are currently monitoring the situation. For more details, see gitlab.com/gitlab-com/gl-infra/production/-/issues/18282

July 16, 2024 10:00 UTC

[Identified] A fix has been pushed and is currently being deployed. For more details, see gitlab.com/gitlab-com/gl-infra/production/-/issues/18282

July 16, 2024 02:24 UTC

[Identified] We have identified the MR that introduced the bug. We are going through the process to revert the MR. For more details, see gitlab.com/gitlab-com/gl-infra/production/-/issues/18282

July 16, 2024 02:06 UTC

[Identified] We are continuing to investigate solutions for this bug. For more details, see gitlab.com/gitlab-com/gl-infra/production/-/issues/18282

July 16, 2024 01:28 UTC

[Identified] We are continuing to investigate solutions for this bug. For more details, see gitlab.com/gitlab-com/gl-infra/production/-/issues/18282

July 16, 2024 00:36 UTC

[Identified] A UI bug is showing some free tier projects being scheduled for deletion. Projects are not actually scheduled for deletion unless a user marked them for deletion. For more details, see gitlab.com/gitlab-com/gl-infra/production/-/issues/18282

GitLab.com is unavailable

July 11, 2024 18:28 UTC

GitLab.com is unavailableService Disruption

Incident Status

Service Disruption

Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions, SAML SSO - GitLab SaaS, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, docs.gitlab.com, Canary, Product Analytics - Configurator (Beta), Product Analytics - Collector (Beta), Product Analytics - Background processing (Beta), Product Analytics - Queries (Beta), GitLab Duo

Locations

Google Compute Engine, Digital Ocean, Zendesk, AWS

Postmortem

https://gitlab.com/gitlab-com/gl-infra/production/-/issues/18269

---

July 11, 2024 18:28 UTC

[Resolved] This incident is now resolved and no further errors are being seen on GitLab.com. Our investigation points to a disk performance issue. You can see further details and corrective actions in gitlab.com/gitlab-com/gl-infra/production/-/issues/18269.

July 11, 2024 15:51 UTC

[Investigating] Our investigations are ongoing. We will post our next update when we have more information. Please follow the incident issue for updates here: gitlab.com/gitlab-com/gl-infra/production/-/issues/18269

July 11, 2024 15:28 UTC

[Investigating] No material updates to report. We're continuing to investigate the issue. Please follow the incident issue for updates here: gitlab.com/gitlab-com/gl-infra/production/-/issues/18269

July 11, 2024 15:00 UTC

[Investigating] We are seeing improved service availability and investigations continue for this problem. Please follow the incident issue for updates here: gitlab.com/gitlab-com/gl-infra/production/-/issues/18269

July 11, 2024 14:43 UTC

[Investigating] We are seeing improved service availability and investigations continue for this problem. We are planning to make the incident issue publicly available as soon as possible.

July 11, 2024 14:29 UTC

[Investigating] We are investigating a problem with gitlab.com being unavailable for some users. Our investigations continue and we will post further status updates shortly.

July 11, 2024 14:18 UTC

[Investigating] We are investigating a problem with gitlab.com being unavailable for some users. We will post further status updates shortly.

Migration by direct transfer and importing with GitHub, Bitbucket Server, and Gitea importer offline

July 8, 2024 11:27 UTC

Migration by direct transfer and importing with GitHub, Bitbucket Server, and Gitea importer offlineService Disruption

Incident Status

Service Disruption

Components

Website, API

Locations

Google Compute Engine

---

July 8, 2024 11:27 UTC

[Resolved] A fix has been implemented and the service is now stable. The Bulk import direct transfer functionality is available again.

July 2, 2024 16:05 UTC

[Investigating] Work is continuing on the development of a fix for the issue. Further updates will be provided on the status page once we have a timeline for the deployment of the fix.

June 28, 2024 04:55 UTC

[Investigating] Work is continuing on the development of a fix for the issue. Further updates will be provided on the status page once we have a timeline for the deployment of the fix.

June 7, 2024 14:34 UTC

[Investigating] We are working to re-enable these features and further updates will be available on the status page - status.gitlab.com

June 6, 2024 09:13 UTC

[Investigating] Migration by direct transfer is currently offline and we are currently investigating the cause. We will update the status page when more information becomes available.

June 6, 2024 04:23 UTC

[Investigating] Bulk import direct transfer functionality is currently offline and we are currently investigating the cause. We will update the status page when more information becomes available.

June 6, 2024 00:28 UTC

[Investigating] Bulk import functionality is currently offline and we are currently investigating the cause. We will update the status page when more information becomes available.

Increase in GitLab.com's Ci jobs processing time

July 2, 2024 16:01 UTC

Increase in GitLab.com's Ci jobs processing timeOperational

Incident Status

Operational

Components

Website, API, Background Processing

Locations

Google Compute Engine

Postmortem

https://gitlab.com/gitlab-com/gl-infra/production/-/issues/18229

---

July 2, 2024 16:01 UTC

[Resolved] This incident is now mitigated. More details here: gitlab.com/gitlab-com/gl-infra/production/-/issues/18229

July 2, 2024 15:43 UTC

[Monitoring] We rolled back a recent change to mitigate the issue. Performance is now back to normal. More details here: gitlab.com/gitlab-com/gl-infra/production/-/issues/18229

July 2, 2024 15:28 UTC

[Investigating] We're noticing an increase in GitLab.com's Ci jobs processing time. We're now investigating this issue. More details here: gitlab.com/gitlab-com/gl-infra/production/-/issues/18229

July 2, 2024 15:26 UTC

[Investigating] We're noticing an increase in GitLab.com's Ci jobs processing time. We're now investigating this issue. More details here: gitlab.com/gitlab-com/gl-infra/production/-/issues/18229

June 2024

Duo Chat Returning Error Code A1001

June 27, 2024 20:39 UTC

Duo Chat Returning Error Code A1001Partial Service Disruption

Incident Status

Partial Service Disruption

Components

GitLab Duo

Locations

Google Compute Engine

---

June 27, 2024 20:39 UTC

[Resolved] We have confirmed that the fix has been deployed, so users should no longer experience errors with Duo Chat.

June 27, 2024 16:36 UTC

[Identified] Monitoring of the deployment of the fix continues. In the meantime, issuing a /clear to Duo Chat can be used to temporarily clear the error.

June 27, 2024 00:48 UTC

[Identified] It appears that the deployment of the fix has not completed as was previously understood. Users may still be experiencing the original issue. We're currently monitoring the deployment and a status update will be provided upon it's completion.

June 26, 2024 23:19 UTC

[Resolved] A fix for the issue has been deployed and users should no longer see error code A1001 when interacting with Duo Chat.

June 25, 2024 23:21 UTC

[Identified] Work is continuing on the development of a fix for the issue. Further updates will be provided on the status page once we have a timeline for the deployment of the fix.

June 25, 2024 17:44 UTC

[Identified] We have identified an issue with the /summarize feature of Duo Chat returning an error when used on Epics and Issues. A fix is currently being developed.

502 Bad Gateway

June 20, 2024 20:23 UTC

502 Bad GatewayPartial Service Disruption

Incident Status

Partial Service Disruption

Components

Website

Locations

Google Compute Engine

---

June 20, 2024 20:23 UTC

[Resolved] This incident has been resolved. Full details can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/18175

June 20, 2024 20:08 UTC

[Monitoring] Our provider has implemented a fix and we are currently monitoring. We've seen minimal 502 responses in the last 30 minutes.

June 20, 2024 19:30 UTC

[Identified] One of our providers has identified connectivity issues that might be impacting users that are accessing GitLab.com. This is resulting in 502 Bad Gateway for a small amount of users. For more details please see gitlab.com/gitlab-com/gl-infra/production/-/issues/18175

June 20, 2024 19:12 UTC

[Investigating] We're investigating some 502 Bad Gateway (Cloudflare) errors some users are experiencing. More details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/18175

429 rate limiting errors

June 20, 2024 10:30 UTC

429 rate limiting errorsDegraded Performance

Incident Status

Degraded Performance

Components

Website, Git Operations

Locations

Google Compute Engine

---

June 20, 2024 10:30 UTC

[Resolved] This incident has been resolved. More information can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/18173.

June 20, 2024 10:21 UTC

[Monitoring] We identified the root cause and have applied a fix, we are monitoring the recovery. More details can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/18173.

June 20, 2024 09:59 UTC

[Investigating] We are investigating an increase in requests being rate limited resulting in 429 errors. For more information see gitlab.com/gitlab-com/gl-infra/production/-/issues/18173

Multiple reports of authentication failures accessing GitLab.com

June 18, 2024 09:03 UTC

Multiple reports of authentication failures accessing GitLab.comOperational

Incident Status

Operational

Components

Website, API, Git Operations, Container Registry

Locations

Google Compute Engine

---

June 18, 2024 09:03 UTC

[Resolved] This incident has been resolved. More information can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/18003.

May 30, 2024 13:02 UTC

[Monitoring] We have temporarily increased the rate limits for unauthenticated requests, to alleviate errors resulting from the token lifetime limits implementation. We are continuing to monitor and adjust these rate limits as appropriate.

May 14, 2024 16:12 UTC

[Monitoring] We have temporarily increased the rate limits for unauthenticated requests, to alleviate errors resulting from the token lifetime limits implementation. Please see the issue for more details: gitlab.com/gitlab-com/gl-infra/production/-/issues/18003

May 14, 2024 03:27 UTC

[Identified] Token lifetime limits have been implemented on GitLab.com. If you are experiencing increased authentication errors accessing GitLab.com, please see our blog post: about.gitlab.com/blog/2023/10/25/access-token-lifetime-limits for details on how to mitigate problems with authentication.