Degraded performance for container registry on GitLab.comOperational

Components

Container Registry

Locations

Google Compute Engine


July 8, 2020 22:14 UTC
[Resolved] We've now completed permanently putting in place the temporary replacement load balancer servers. Container Registry service remains stable and this issue is now resolved but requires more incident review. We will have an incident review provided in this issue here: gitlab.com/gitlab-com/gl-infra/production/-/issues/2381

July 8, 2020 21:13 UTC
[Monitoring] Permanently putting in place the temporary replacement load balancers servers is still in progress. Container Registry service remains stable, we will send any further updates in this issue here: gitlab.com/gitlab-com/gl-infra/production/-/issues/2381

July 8, 2020 20:37 UTC
[Monitoring] We've made progress but are still working to permanently put in place the temporary replacement load balancers servers. Container Registry service remains stable, we will send any further updates in this issue here: gitlab.com/gitlab-com/gl-infra/production/-/issues/2381

July 8, 2020 20:06 UTC
[Monitoring] We're currently working to permanently put in place the temporary replacement load balancers servers. Container Registry service remains stable, we will send any further updates in this issue here: gitlab.com/gitlab-com/gl-infra/production/-/issues/2381

July 8, 2020 19:19 UTC
[Monitoring] We’ve found that once removing specific load balancer servers from the Container Registry service that recovery has started and requests are successful, this is a temporary solution but we’re monitoring stability. Post Mortem and permanent fixes will be summarized here: gitlab.com/gitlab-com/gl-infra/production/-/issues/2381

July 8, 2020 18:58 UTC
[Investigating] We're working to remove believed problem servers from the Container Registry service to continue troubleshooting these connectivity failures. If you have any additional feedback, please do share it with us here: gitlab.com/gitlab-com/gl-infra/production/-/issues/2381

July 8, 2020 18:38 UTC
[Investigating] A rollback of a change that correlates with the timing of this issue is being made as a potential fix. We're still investigating and working on other potential fixes. Our team appreciates your feedback, please do submit your experience here if you are affect by this issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/2381

July 8, 2020 18:16 UTC
[Investigating] Our team has just finished redeploying the Container Registry service, we are still investigating other possible fixes. Please do submit your experience here if you are affect by this issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/2381

July 8, 2020 17:54 UTC
[Investigating] We’re still diligently investigating these Container Registry connectivity issues between the service and its load balancers. More information will be added here: gitlab.com/gitlab-com/gl-infra/production/-/issues/2381

July 8, 2020 17:36 UTC
[Investigating] Error rates have continued to increase after initial mitigation. We’re working on further investigation into connectivity issues between the service and its load balancers. More information will be added here: gitlab.com/gitlab-com/gl-infra/production/-/issues/2381

July 8, 2020 17:11 UTC
[Monitoring] We are observing error rates increasing after initial mitigation and are continuing to work on those issues. More information will be added here: gitlab.com/gitlab-com/gl-infra/production/-/issues/2381

July 8, 2020 16:57 UTC
[Monitoring] Container Registry service on GitLab.com is starting to show recovery. We’re monitoring this recovery for stability. More information will be added here: gitlab.com/gitlab-com/gl-infra/production/-/issues/2381

July 8, 2020 16:42 UTC
[Investigating] Container Registry service on GitLab.com is still down. We’re working on standing up load balancers to attempt recovering the Container Registry service. More information will be added here: gitlab.com/gitlab-com/gl-infra/production/-/issues/2381

July 8, 2020 16:24 UTC
[Investigating] Container Registry service on GitLab.com is still down. Our team is working diligently to identify more information. Updates will be added here: gitlab.com/gitlab-com/gl-infra/production/-/issues/2381

July 8, 2020 16:09 UTC
[Investigating] No Material Update: We're still investigating this performance failure for Container Registry service on GitLab.com. More information will be added here: gitlab.com/gitlab-com/gl-infra/production/-/issues/2381

July 8, 2020 15:58 UTC
[Investigating] We're still investigating this performance failure for Container Registry service on GitLab.com. More information will be added here: gitlab.com/gitlab-com/gl-infra/production/-/issues/2381

July 8, 2020 15:46 UTC
[Investigating] The registry on GitLab.com is down.

July 8, 2020 15:40 UTC
[Investigating] The Registry service on GitLab.com is down. We are working on this issue, more information will be added here: gitlab.com/gitlab-com/gl-infra/production/-/issues/2381

July 8, 2020 15:33 UTC
[Investigating] The Registry service on GitLab.com is down. We are working on this issue, more information will be added here: gitlab.com/gitlab-com/gl-infra/production/-/issues/2381

July 8, 2020 15:21 UTC
[Investigating] We are investigating degraded performance with the container registry on GitLab, more information will be added here: gitlab.com/gitlab-com/gl-infra/production/-/issues/2381

Back to current status