All Systems Operational

Updated a few seconds ago

Back to current status

Status History

Filter: SAML SSO - GitLab SaaS (Clear)



December 2021

GitLab.com production database reliability update

December 18, 2021 16:46 UTC

Description

During this maintenance we will be upgrading our GitLab.com production database to a minor version of PostgreSQL 12.9 to improve overall reliability. This maintenance will not require downtime and is not expected to otherwise impact performance or availability of GitLab.com functionality. We are providing notice of this work due to the critical nature of the database to our operations, even though we anticipate no impact.


Components

Website, API, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners for GitLab community contributions, SAML SSO - GitLab SaaS, Background Processing, Canary


Locations

Google Compute Engine


Schedule

December 18, 2021 15:00 - December 18, 2021 17:00 UTC



December 18, 2021 16:46 UTC
[Update] Maintenance work is now completed. The full anticipated work was not successfully completed, but no further changes will be executed today.

December 18, 2021 16:02 UTC
[Update] Maint Started 15:00 UTC

November 2021

SAML SSO returning too many redirects

November 10, 2021 18:54 UTC

SAML SSO returning too many redirectsPartial Service Disruption

Incident Status

Partial Service Disruption


Components

SAML SSO - GitLab SaaS


Locations

Google Compute Engine




November 10, 2021 18:54 UTC
[Resolved] We have confirmed the SSO redirect issue is no longer occurring and will now consider this resolved. Follow this issue for further updates: gitlab.com/gitlab-com/gl-infra/production/-/issues/5904

November 10, 2021 18:41 UTC
[Monitoring] We've identified the feature flag responsible for the problem and have reverted it. Users should no longer be encountering this redirect loop. Details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/5904

November 10, 2021 18:20 UTC
[Investigating] No material updates at this time - our investigation is ongoing. Users can still login via their Identity Provider directly. Details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/5904

November 10, 2021 18:07 UTC
[Investigating] Our investigation is still ongoing. SAML SSO users can still login via their Identity Provider directly. Details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/5904

November 10, 2021 17:56 UTC
[Investigating] We are currently investigating an issue wherein SAML SSO login attempts will be stuck in a redirect loop. Details and steps to workaround the issue can be found here: gitlab.com/gitlab-com/gl-infra/production/-/issues/5904

September 2021

Increased error rates across all GitLab services

September 12, 2021 21:48 UTC

Incident Status

Service Disruption


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners for GitLab community contributions, SAML SSO - GitLab SaaS, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, Canary


Locations

Google Compute Engine, Azure, Digital Ocean, Zendesk, AWS




September 12, 2021 21:48 UTC
[Resolved] After investigation we determined that the elevated error rates were caused by database connection saturation; further details will be in this issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/5514

September 12, 2021 21:23 UTC
[Monitoring] GitLab performance and error rates have returned to normal; we are investigating the root cause to understand the likelyhood it will reoccur.

September 12, 2021 21:04 UTC
[Investigating] We are investigating slow response times and increased error rates across all GitLab services. Incident: gitlab.com/gitlab-com/gl-infra/production/-/issues/5514

August 2021

Elevated error rates on GitLab.com

August 11, 2021 17:51 UTC

Elevated error rates on GitLab.comPartial Service Disruption

Incident Status

Partial Service Disruption


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners for GitLab community contributions, SAML SSO - GitLab SaaS, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, Canary


Locations

Google Compute Engine, Azure, Digital Ocean, Zendesk, AWS




August 11, 2021 17:51 UTC
[Resolved] As GitLab.com is operating normally, we'll now be marking this incident as resolved. Further investigation efforts and details surrounding the issue can be found in: gitlab.com/gitlab-com/gl-infra/production/-/issues/5333

August 11, 2021 17:44 UTC
[Investigating] No material updates to report - our investigation is still ongoing. Details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/5333

August 11, 2021 17:27 UTC
[Investigating] All GitLab.com services are operating normally, but our investigation into the cause of the errors continues. Details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/5333

August 11, 2021 17:18 UTC
[Investigating] At 17:04 UTC, there was a 5 minute elevated error ratio event for all GitLab.com services. It has recovered, but we are investigating. Details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/5333

SAML Group Links cause 404 on sign-in

August 10, 2021 22:47 UTC

Incident Status

Degraded Performance


Components

SAML SSO - GitLab SaaS


Locations

Google Compute Engine




August 10, 2021 22:47 UTC
[Resolved] This awareness incident will now be marked as resolved. For further details, see: gitlab.com/gitlab-com/gl-infra/production/-/issues/5320

August 9, 2021 21:37 UTC
[Monitoring] Moving to operational - we'll be leaving this incident open for the next 24 hours in order to ensure users are aware of the change. Further details gitlab.com/gitlab-com/gl-infra/production/-/issues/5320

August 9, 2021 20:47 UTC
[Monitoring] Due to a recent change in SAML behavior, users will receive a 404 and be removed from their group if SAML Group Sync is not properly configured. This is intended behavior - further details (including a workaround) can be found here: gitlab.com/gitlab-com/gl-infra/production/-/issues/5320

Increased error rates across all services

August 10, 2021 10:39 UTC

Incident Status

Partial Service Disruption


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners for GitLab community contributions, SAML SSO - GitLab SaaS, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, Canary


Locations

Google Compute Engine, Azure, Digital Ocean, Zendesk, AWS




August 10, 2021 10:39 UTC
[Resolved] The issue has been resolved, and we can confirm that all services on GitLab.com are fully operational. A full timeline of this incident and more details are available in gitlab.com/gitlab-com/gl-infra/production/-/issues/5324

August 10, 2021 10:05 UTC
[Monitoring] We are back to normal error rates across all services. We keep monitoring the situation. Refer to gitlab.com/gitlab-com/gl-infra/production/-/issues/5324 for more updates.

August 10, 2021 09:52 UTC
[Investigating] We are investigating increased error rates across all services, see gitlab.com/gitlab-com/gl-infra/production/-/issues/5324 for more details





Back to current status