Active Issue

Updated a few seconds ago

Back to current status

Status History

Filter: Container Registry (Clear)



February 2025

PG minor upgrade 16.6 in Production

February 15, 2025 07:49 UTC

Description

Scheduled maintenance to patch our DB layer to PostgreSQL 16.6. 2025-02-15 02:00-08:00 UTC. GitLab.com will be available throughout this time, but users may experience errors on requests. We apologize in advance for any inconvenience caused. See <gitlab.com/gitlab-com/gl-infra/production/-/issues/19120>


Components

Website, API, Container Registry, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions, Background Processing


Locations

Google Compute Engine, AWS


Schedule

February 15, 2025 03:00 - February 15, 2025 09:00 UTC



February 15, 2025 07:49 UTC
[Update] Gitlab.com database maintenance was performed, the platform is considered stable. We'll continue to monitor for any performance issues. Thank you for your patience. See <gitlab.com/gitlab-com/gl-infra/production/-/issues/19120>

February 15, 2025 04:07 UTC
[Update] Gitlab.com scheduled maintenance of our database layer has started. GitLab.com should be available during the whole period but we expect availability and performance impact until 2025-02-15 09:00 UTC. See gitlab.com/gitlab-com/gl-infra/production/-/issues/19120

February 15, 2025 03:01 UTC
[Update] Gitlab.com scheduled maintenance of our database layer has started. GitLab.com should be available during the whole period but we expect availability and performance impact until 2025-02-15 09:00 UTC. See gitlab.com/gitlab-com/gl-infra/production/-/issues/19120

February 15, 2025 02:48 UTC
[Update] Scheduled database maintenance will now start at 2025-02-15 03:00 UTC and expected to run to 2025-02-15 09:00 UTC. See gitlab.com/gitlab-com/gl-infra/production/-/issues/19120

February 15, 2025 01:54 UTC
[Update] Due to an ongoing incident we are delaying the maintenance window by 2 hours to investigate the impact. We currently plan to start the maintenance window at 04:00 UTC.

February 14, 2025 01:45 UTC
[Update] We will be undergoing scheduled maintenance of our databases tomorrow. There may be some availability and performance impact between 2025-02-15 02:00 UTC and 2025-02-15 08:00 UTC. See gitlab.com/gitlab-com/gl-infra/production/-/issues/19120

February 12, 2025 01:13 UTC
[Update] We will be undergoing scheduled maintenance of our databases in 3 days. There may be some availability and performance impact between 2025-02-15 02:00 UTC and 2025-02-15 08:00 UTC. See gitlab.com/gitlab-com/gl-infra/production/-/issues/19120

February 8, 2025 08:37 UTC
[Update] Scheduled maintenance in 1 week: Upgrading DB layer to PostgreSQL 16.6 on Feb 15, 02:00-08:00 UTC. GitLab.com will stay up but users may experience errors. See gitlab.com/gitlab-com/gl-infra/production/-/issues/19120

February 1, 2025 02:42 UTC
[Update] Scheduled maintenance in 2 weeks: Upgrading DB layer to PostgreSQL 16.6 on Feb 15, 02:00-08:00 UTC. GitLab.com will stay up but may experience errors. Sorry for any inconvenience. See gitlab.com/gitlab-com/gl-infra/production/-/issues/19120

January 2025

Accessing gitlab.com results in 500 errors

January 6, 2025 16:42 UTC

Incident Status

Partial Service Disruption


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions, SAML SSO - GitLab SaaS, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, docs.gitlab.com, Canary, Product Analytics - Configurator (Beta), Product Analytics - Collector (Beta), Product Analytics - Background processing (Beta), Product Analytics - Queries (Beta), GitLab Duo


Locations

Google Compute Engine, Digital Ocean, Zendesk, AWS




January 6, 2025 16:42 UTC
[Resolved] We've identified the cause of the errors and the errors have returned to normal. The incident is now resolved. More information on the incident can be found in this issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/19079

January 6, 2025 15:57 UTC
[Monitoring] We've identified the cause of the errors, and the errors have returned to normal. The incident is now mitigated, but we're continuing to monitor for any further impact. More information on the incident can be found in this issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/19079

January 6, 2025 15:27 UTC
[Monitoring] We're continuing to monitor for further errors related to the incident. We see further decreases in the errors.

January 6, 2025 15:17 UTC
[Monitoring] We saw elevated 500 errors on gitlab.com which have then decreased. We're continuing to monitor for further errors

December 2024

Intermittent timeouts for requests from some Utah IP addresses

December 20, 2024 02:51 UTC

Incident Status

Partial Service Disruption


Components

Git Operations, Container Registry


Locations

Google Compute Engine




December 20, 2024 02:51 UTC
[Resolved] We are now marking this incident as resolved. Details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/19017

December 19, 2024 01:48 UTC
[Monitoring] Google's network engineers have identified the issue affecting the "us-west3" region and deployed a fix. We will continue monitoring. For questions and more details, please follow gitlab.com/gitlab-com/gl-infra/production/-/issues/19017.

December 18, 2024 22:15 UTC
[Investigating] No material updates at this time. Cloudlfare and Google are still investigating these routing issues. We will provide additional information as it becomes available. Details will be posted to: gitlab.com/gitlab-com/gl-infra/production/-/issues/19017

December 18, 2024 03:57 UTC
[Investigating] Our Infrastructure Engineers have been working with CloudFlare and Google and have identified a routing issue between GCP "us-west3" and Cloudflare. Investigation is continuing. Please follow gitlab.com/gitlab-com/gl-infra/production/-/issues/19017 for more details.

December 17, 2024 22:37 UTC
[Investigating] No material updates at this time. Requests originating from the Salt Lake City, Utah region may still see intermittent timeouts. We are working with our third party vendor to investigate further. We will provide updates as info becomes available. Details: gitlab.com/gitlab-com/gl-infra/production/-/issues/19017

December 17, 2024 13:45 UTC
[Investigating] No material updates at the moment. We continue to work with a third party vendor to get more updates and waiting for feedback from affected customers. Details to follow in: gitlab.com/gitlab-com/gl-infra/production/-/issues/19017

December 16, 2024 23:49 UTC
[Investigating] We have received some reports of requests timing out for IP addresses originating in Salt Lake City, Utah. We are working with a third party vendor to identify the issue. Details to follow in: gitlab.com/gitlab-com/gl-infra/production/-/issues/19017

September 2024

GitLab connectivity issues

September 10, 2024 13:40 UTC

Incident Status

Operational


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions


Locations

Google Compute Engine




September 10, 2024 13:40 UTC
[Resolved] This incident has been resolved. More information can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/18535.

September 10, 2024 13:33 UTC
[Monitoring] The website has recovered are and we are now monitoring before marking as resolved. More details can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/18535.

September 10, 2024 13:24 UTC
[Investigating] No material updates at this time - our investigation is ongoing. More details can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/18535

September 10, 2024 13:10 UTC
[Investigating]

September 10, 2024 13:08 UTC
[Investigating] We are currently investigating connectivity issues to GitLab.com. For more information see gitlab.com/gitlab-com/gl-infra/production/-/issues/18535

GitLab.com Connectivity Issues

September 3, 2024 05:34 UTC

Incident Status

Service Disruption


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions, SAML SSO - GitLab SaaS, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, docs.gitlab.com, Canary, Product Analytics - Configurator (Beta), Product Analytics - Collector (Beta), Product Analytics - Background processing (Beta), Product Analytics - Queries (Beta), GitLab Duo


Locations

Google Compute Engine, Digital Ocean, Zendesk, AWS




September 3, 2024 05:34 UTC
[Resolved] All GitLab services have been restored. We are actively monitoring the system, and everything appears normal at this time. For more details, please follow the issue at: gitlab.com/gitlab-com/gl-infra/production/-/issues/18490

September 3, 2024 05:19 UTC
[Monitoring] We've identified and addressed the root cause of the issue. All GitLab services have been restored, and we are actively monitoring the system. Follow the issue for more details : gitlab.com/gitlab-com/gl-infra/production/-/issues/18490

September 3, 2024 05:09 UTC
[Investigating] GitLab.com is currently experiencing connectivity issues due to a high error rate. Our infrastructure team is investigating. Issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/18490

August 2024

Some requests timing out

August 23, 2024 03:59 UTC

Incident Status

Operational


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions, SAML SSO - GitLab SaaS, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, docs.gitlab.com, Canary, Product Analytics - Configurator (Beta), Product Analytics - Collector (Beta), Product Analytics - Background processing (Beta), Product Analytics - Queries (Beta), GitLab Duo


Locations

Google Compute Engine, Digital Ocean, Zendesk, AWS




August 23, 2024 03:59 UTC
[Resolved] The issue has been resolved and all functionality on GitLab.com is nominal. Please see gitlab.com/gitlab-com/gl-infra/production/-/issues/18435 for more details.

August 23, 2024 01:53 UTC
[Monitoring] The issue has been identified and the incident resolved. We will continue monitoring for a while.

August 23, 2024 01:27 UTC
[Identified] We have identified an underlying cause and are working to mitigate it. See gitlab.com/gitlab-com/gl-infra/production/-/issues/18435 for more details.

August 23, 2024 00:34 UTC
[Investigating] We have noticed that there are delays in web UI updates for example, MRs not showing up after submission, We are continuing to investigate - please see gitlab.com/gitlab-com/gl-infra/production/-/issues/18435 for more details.

August 23, 2024 00:27 UTC
[Investigating] Some requests are slow or timing out due to DB saturation. We are investigating. Please see [this issue](gitlab.com/gitlab-com/gl-infra/production/-/issues/18435) for further details.

July 2024

GitLab.com is unavailable

July 11, 2024 18:28 UTC

GitLab.com is unavailableService Disruption

Incident Status

Service Disruption


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners for GitLab community contributions, SAML SSO - GitLab SaaS, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, docs.gitlab.com, Canary, Product Analytics - Configurator (Beta), Product Analytics - Collector (Beta), Product Analytics - Background processing (Beta), Product Analytics - Queries (Beta), GitLab Duo


Locations

Google Compute Engine, Digital Ocean, Zendesk, AWS




July 11, 2024 18:28 UTC
[Resolved] This incident is now resolved and no further errors are being seen on GitLab.com. Our investigation points to a disk performance issue. You can see further details and corrective actions in gitlab.com/gitlab-com/gl-infra/production/-/issues/18269.

July 11, 2024 15:51 UTC
[Investigating] Our investigations are ongoing. We will post our next update when we have more information. Please follow the incident issue for updates here: gitlab.com/gitlab-com/gl-infra/production/-/issues/18269

July 11, 2024 15:28 UTC
[Investigating] No material updates to report. We're continuing to investigate the issue. Please follow the incident issue for updates here: gitlab.com/gitlab-com/gl-infra/production/-/issues/18269

July 11, 2024 15:00 UTC
[Investigating] We are seeing improved service availability and investigations continue for this problem. Please follow the incident issue for updates here: gitlab.com/gitlab-com/gl-infra/production/-/issues/18269

July 11, 2024 14:43 UTC
[Investigating] We are seeing improved service availability and investigations continue for this problem. We are planning to make the incident issue publicly available as soon as possible.

July 11, 2024 14:29 UTC
[Investigating] We are investigating a problem with gitlab.com being unavailable for some users. Our investigations continue and we will post further status updates shortly.

July 11, 2024 14:18 UTC
[Investigating] We are investigating a problem with gitlab.com being unavailable for some users. We will post further status updates shortly.

June 2024

Multiple reports of authentication failures accessing GitLab.com

June 18, 2024 09:03 UTC

Incident Status

Operational


Components

Website, API, Git Operations, Container Registry


Locations

Google Compute Engine




June 18, 2024 09:03 UTC
[Resolved] This incident has been resolved. More information can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/18003.

May 30, 2024 13:02 UTC
[Monitoring] We have temporarily increased the rate limits for unauthenticated requests, to alleviate errors resulting from the token lifetime limits implementation. We are continuing to monitor and adjust these rate limits as appropriate.

May 14, 2024 16:12 UTC
[Monitoring] We have temporarily increased the rate limits for unauthenticated requests, to alleviate errors resulting from the token lifetime limits implementation. Please see the issue for more details: gitlab.com/gitlab-com/gl-infra/production/-/issues/18003

May 14, 2024 03:27 UTC
[Identified] Token lifetime limits have been implemented on GitLab.com. If you are experiencing increased authentication errors accessing GitLab.com, please see our blog post: about.gitlab.com/blog/2023/10/25/access-token-lifetime-limits for details on how to mitigate problems with authentication.

April 2024

Dependency Proxy 500 errors

April 18, 2024 15:32 UTC

Dependency Proxy 500 errorsDegraded Performance

Incident Status

Degraded Performance


Components

Container Registry


Locations

Google Compute Engine




April 18, 2024 15:32 UTC
[Resolved] This incident is resolved now. See gitlab.com/gitlab-com/gl-infra/production/-/issues/17851 for more details

April 18, 2024 05:00 UTC
[Monitoring] There's no material to update at the moment. We'll continue to monitor the situation. See gitlab.com/gitlab-com/gl-infra/production/-/issues/17851 for more details

April 18, 2024 02:51 UTC
[Monitoring] The rollback has completed successfully. We have confirmed that we no longer see reports of this issue but we will continue monitoring. See gitlab.com/gitlab-com/gl-infra/production/-/issues/17851 for more details

April 17, 2024 20:29 UTC
[Identified] Work is underway to revert the problematic changes. We do not expect this change to land in production for 8-12 hours. Once we have confirmed the fix has been deployed, we will post an update here. We sincerely apologize for the delay. Details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/17851

April 17, 2024 18:59 UTC
[Identified] We believe to have identified the issue. We are working on a revert to address the problem. Details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/17851

April 17, 2024 18:18 UTC
[Investigating] No material updates at this time. We continue to investigate the issue. Details will be provided in: gitlab.com/gitlab-com/gl-infra/production/-/issues/17851

April 17, 2024 17:55 UTC
[Investigating] No material updates at this time. Our investigation is ongoing. Details will be posted in: gitlab.com/gitlab-com/gl-infra/production/-/issues/17851

April 17, 2024 17:37 UTC
[Investigating] Our investigation is ongoing. This appears to be impacting only a small portion of users. Details will be posted in: gitlab.com/gitlab-com/gl-infra/production/-/issues/17851

April 17, 2024 17:13 UTC
[Investigating] No material updates at this time. Our investigation is ongoing. Details will be posted in: gitlab.com/gitlab-com/gl-infra/production/-/issues/17851

April 17, 2024 16:56 UTC
[Investigating] We are aware of some jobs encountering 500 errors when attempting to pull images via the Dependency Proxy. We are investigating. Details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/17851

September 2023

Elevated error reates and slow response times

September 13, 2023 09:29 UTC

Incident Status

Degraded Performance


Components

Website, API, Git Operations, Container Registry


Locations

Google Compute Engine




September 13, 2023 09:29 UTC
[Resolved] The issue has been resolved and services are operating normally. The investigation summary will be posted in gitlab.com/gitlab-com/gl-infra/production/-/issues/16359

September 13, 2023 08:23 UTC
[Monitoring] We're now seeing error rates and performance returning back to normal operation. We're continuing to monitor to ensure the problem does not recur. Details in gitlab.com/gitlab-com/gl-infra/production/-/issues/16359

September 13, 2023 08:09 UTC
[Investigating] We are investigating an increase in errors and slower response times on GitLab.com. For details see gitlab.com/gitlab-com/gl-infra/production/-/issues/16359

July 2023

Performance issues on GitLab.com

July 15, 2023 04:51 UTC

Incident Status

Degraded Performance


Components

API, Git Operations, Container Registry, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners for GitLab community contributions, Background Processing, Canary


Locations

Google Compute Engine




July 15, 2023 04:51 UTC
[Resolved] This incident has been marked as resolved, as we don't see any outstanding issues with GitLab.com. A full timeline of this incident can be found in our Incident Report available at gitlab.com/gitlab-com/gl-infra/production/-/issues/16042

July 15, 2023 03:29 UTC
[Monitoring] We will update the status of this incident to `Operational` for now..

July 15, 2023 03:26 UTC
[Monitoring] The rollback has been completed. We are observing that services have recovered. We will continue to monitor for any issues. More details in gitlab.com/gitlab-com/gl-infra/production/-/issues/16042

July 15, 2023 03:02 UTC
[Identified] We are working on performing a rollback on GitLab.com. Next expected update will be in 30 minutes. More details in gitlab.com/gitlab-com/gl-infra/production/-/issues/16042

July 15, 2023 02:52 UTC
[Identified] We are continuing to observe performance issues on GitLab.com. We are investigating options to perform a rollback. More details in gitlab.com/gitlab-com/gl-infra/production/-/issues/16042

July 15, 2023 02:18 UTC
[Identified] No material updates to report at this time. Infrastructure teams are continuing investigations. Details in gitlab.com/gitlab-com/gl-infra/production/-/issues/16042

July 15, 2023 01:45 UTC
[Identified] We have identified the problem and mitigated the worst of it. Our infrastructure team is continuing investigations. Details in gitlab.com/gitlab-com/gl-infra/production/-/issues/16042.

July 15, 2023 01:13 UTC
[Investigating] We have detected a problem related to GitLab.com services and are actively investigating. More details to follow.

June 2023

Docker pull failures from registry.gitlab.com

June 29, 2023 03:39 UTC

Incident Status

Partial Service Disruption


Components

Container Registry


Locations

Google Compute Engine




June 29, 2023 03:39 UTC
[Resolved] This incident has been resolved and Container Registry is now fully operational. More information can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/15955.

June 29, 2023 01:57 UTC
[Monitoring] We have applied a fix and are now monitoring Container Registry for the next hour before marking the incident as resolved. More details can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/15955.

June 28, 2023 18:48 UTC
[Identified] Next status update will be once we confirm the fix has been deployed, or earlier if any material updates are in order. Incident issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/15955

June 28, 2023 18:44 UTC
[Identified] We have identified a workaround for two different scenarios that may help affected users go back to normal operation while the fix is deployed. Please see gitlab.com/gitlab-com/gl-infra/production/-/issues/15955#note_1449997741

June 28, 2023 18:10 UTC
[Identified] We continue to wait for the fix to be deployed into production. We are currently looking into a possible workaround to provide our users in the meantime. Continue following this incident at gitlab.com/gitlab-com/gl-infra/production/-/issues/15955

June 28, 2023 17:38 UTC
[Identified] Users started seeing failures when pulling images from registry.gitlab.com. The affecting MR has been identified and is in the process of being rolled back. More details in gitlab.com/gitlab-com/gl-infra/production/-/issues/15955

Registry Cluster PostgreSQL 14 Upgrade

June 10, 2023 16:59 UTC

Description

We will be undergoing scheduled maintenance to our Registry database layer. GitLab.com will continue to be available during the maintenance window. We apologize in advance for any inconvenience this may cause. More details: gitlab.com/gitlab-com/gl-infra/production/-/issues/11375


Components

Container Registry


Locations

Google Compute Engine


Schedule

June 10, 2023 14:00 - June 10, 2023 19:00 UTC



June 10, 2023 16:59 UTC
[Update] GitLab.com planned maintenance for the database layer is complete. We'll be monitoring the platform to ensure all systems are functioning correctly. Thank you for your patience.

June 10, 2023 16:36 UTC
[Update] GitLab.com planned maintenance for the database layer is almost complete. We're continuing to verify that all systems are functioning correctly. Thank you for your patience.

June 10, 2023 14:19 UTC
[Update] GitLab.com planned maintenance for the database layer is starting.

June 10, 2023 13:04 UTC
[Update] We will be undergoing scheduled maintenance to our registry database layer in 1 hour. The maintenance will take up to 5 hours starting from 14:00 UTC to 19:00 UTC. GitLab.com will be available but users may experience degraded performance during the maintenance window. We apologize in advance for any inconvenience this may cause. See <gitlab.com/gitlab-com/gl-infra/production/-/issues/11375>

June 10, 2023 13:03 UTC
[Update] We will be undergoing scheduled maintenance to our registry database layer in 3 hours. The maintenance will take up to 5 hours starting from 14:00 UTC to 19:00 UTC. GitLab.com will be available but users may experience degraded performance during the maintenance window. We apologize in advance for any inconvenience this may cause. See <gitlab.com/gitlab-com/gl-infra/production/-/issues/11375>

June 10, 2023 11:03 UTC
[Update] In 3 hours we will perform the scheduled maintenance to our registry database. It will take up to 5 hours, during which users may experience degraded performance. See gitlab.com/gitlab-com/gl-infra/production/-/issues/11375

June 9, 2023 14:06 UTC
[Update] Reminder: Tomorrow, we will be undergoing scheduled maintenance to our registry database layer. The maintenance will take up to 5 hours starting from 14:00 UTC to 19:00 UTC. GitLab.com will be available but users may experience degraded performance during the maintenance window. We apologize in advance for any inconvenience this may cause. See <gitlab.com/gitlab-com/gl-infra/production/-/issues/11375>

June 9, 2023 08:05 UTC
[Update] Tomorrow we will perform a scheduled maintenance to our registry database starting from 14:00 UTC to 19:00 UTC, during which users may experience degraded performance. See gitlab.com/gitlab-com/gl-infra/production/-/issues/11375

June 7, 2023 10:10 UTC
[Update] In 3 days, we will perform a scheduled maintenance to our Registry database layer. The maintenance will take up to 5 hours starting from 14:00 UTC to 19:00 UTC, during which GitLab.com will be available but users may experience degraded performance. We apologize in advance for any inconvenience this may cause. See gitlab.com/gitlab-com/gl-infra/production/-/issues/11375

Slow requests on GitLab.com

June 7, 2023 13:32 UTC

Slow requests on GitLab.comPartial Service Disruption

Incident Status

Partial Service Disruption


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners for GitLab community contributions, SAML SSO - GitLab SaaS, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, docs.gitlab.com, Canary


Locations

Google Compute Engine, Digital Ocean, Zendesk, AWS




June 7, 2023 13:32 UTC
[Resolved] We have identified the root cause, have seen recovery and are therefore resolving this incident. More information in: gitlab.com/gitlab-com/gl-infra/production/-/issues/14839

June 7, 2023 13:21 UTC
[Investigating] Customers are experiencing slow requests on GitLab.com. More information in: gitlab.com/gitlab-com/gl-infra/production/-/issues/14839

May 2023

Hardware upgrade for the GitLab.com datastore

May 27, 2023 04:06 UTC

Description

In 2 weeks, as part of a planned maintenance window, we will perform a hardware upgrade for the GitLab.com datastore. Users may experience temporary 50X errors for a brief period during this window. See gitlab.com/gitlab-com/gl-infra/production/-/issues/10694


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners for GitLab community contributions, SAML SSO - GitLab SaaS, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, docs.gitlab.com, Canary


Locations

Google Compute Engine, Digital Ocean, Zendesk, AWS


Schedule

May 27, 2023 03:00 - May 27, 2023 04:00 UTC



May 27, 2023 04:06 UTC
[Update] GitLab.com's database layer maintenance is complete now, and we're fully back up and running. We'll be monitoring the platform to ensure all systems are functioning correctly. Thank you for your patience.

May 27, 2023 03:41 UTC
[Update] The database switchover is now complete, we expect all SQL statements to be routed to the new nodes. The site is back up and we're continuing to verify that all systems are functioning correctly. Thank you for your patience.

May 27, 2023 03:00 UTC
[Update] The database maintenance is starting now. The database Primary node switchover should happen at any moment in the next hour and during the switchover users might see 50X errors for a very brief span of time. For details see gitlab.com/gitlab-com/gl-infra/production/-/issues/10694

May 27, 2023 01:51 UTC
[Update] We will be conducting a maintenance database activity in an hour, from 03:00 to 04:00 UTC. Users might see 50X errors for a very brief span of time during this window. For details see gitlab.com/gitlab-com/gl-infra/production/-/issues/10694

May 26, 2023 02:27 UTC
[Update] We will be conducting a maintenance database activity Tomorrow, 2023-05-27, from 03:00 to 04:00 UTC. Users might see 50X errors for a very brief span of time during this window. For details see gitlab.com/gitlab-com/gl-infra/production/-/issues/10694

May 24, 2023 02:14 UTC
[Update] Correction: We will be conducting a maintenance database activity this Saturday, 2023-05-27. from 03:00 to 04:00 UTC. Users might see 50X errors for a very brief span of time during this window. For details see gitlab.com/gitlab-com/gl-infra/production/-/issues/10694

May 24, 2023 02:05 UTC
[Update] We will be conducting a maintenance database activity this Saturday, 2023-05-20, from 03:00 to 04:00 UTC. Users might see 50X errors for a very brief span of time during this window. For details see gitlab.com/gitlab-com/gl-infra/production/-/issues/10694

May 20, 2023 03:58 UTC
[Update] The maintenance is rescheduled to next Saturday, 2023-05-27, from 03:00 to 04:00 UTC. There is currently a long database migration running to fix an incident (gitlab.com/gitlab-com/gl-infra/production/-/issues/14468) that we can't risk to interrupt. We apologise for any inconvenience.

May 20, 2023 03:07 UTC
[Update] The scheduled maintenance will be not be starting at 03:00 UTC, and has been delayed. More details soon to follow.

May 20, 2023 02:00 UTC
[Update] We will be conducting a maintenance database activity in an hour, from 03:00 to 04:00 UTC. Users might see 50X errors for a very brief span of time during this window. For details see gitlab.com/gitlab-com/gl-infra/production/-/issues/10694

May 20, 2023 01:00 UTC
[Update] We will be conducting a maintenance database activity in 2 hours, from 03:00 to 04:00 UTC. Users might see 50X errors for a very brief span of time during this window. For details see gitlab.com/gitlab-com/gl-infra/production/-/issues/10694

May 19, 2023 02:00 UTC
[Update] We will be conducting a maintenance database activity tomorrow, 2023-05-20, from 03:00 to 04:00 UTC. Users might see 50X errors for a very brief span of time during this window. For details see gitlab.com/gitlab-com/gl-infra/production/-/issues/10694

May 17, 2023 02:00 UTC
[Update] We will be conducting a maintenance database activity this Saturday, 2023-05-20, from 03:00 to 04:00 UTC. Users might see 50X errors for a very brief span of time during this window. For details see gitlab.com/gitlab-com/gl-infra/production/-/issues/10694

May 12, 2023 02:00 UTC
[Update] Next week, as part of a planned maintenance window on 2023-05-20 from 03:00 to 04:00 UTC, We will perform a hardware upgrade for the GitLab.com datastore. Users may experience temporary 50X errors for a brief period during this window. See gitlab.com/gitlab-com/gl-infra/production/-/issues/10694

GitLab.com - Container registry pulls failing with unknown blob

May 12, 2023 16:02 UTC

Incident Status

Operational


Components

Container Registry


Locations

Google Compute Engine




May 12, 2023 16:02 UTC
[Resolved] We've deployed a permanent fix for the container registry pulls failing. We are no longer seeing errors related to the incident. Details in gitlab.com/gitlab-com/gl-infra/production/-/issues/14260

May 12, 2023 14:58 UTC
[Monitoring] We're deploying a permanent fix for the container registry pulls failing. Details in gitlab.com/gitlab-com/gl-infra/production/-/issues/14260

May 12, 2023 14:27 UTC
[Identified] We're seeing a decrease in errors, but not a complete resolution is found yet. Details in gitlab.com/gitlab-com/gl-infra/production/-/issues/14260

May 12, 2023 14:04 UTC
[Identified] We've identified the cause of the problem on container registry pulls failing. We're working on deploying a workaround. Details in gitlab.com/gitlab-com/gl-infra/production/-/issues/14260

May 12, 2023 13:50 UTC
[Investigating] We're continuing to investigate the cause of the problem on container registry pulls failing. Details in gitlab.com/gitlab-com/gl-infra/production/-/issues/14260

May 12, 2023 13:39 UTC
[Investigating] We do not have any material updates to report on container registry pulls failing. However, we want to assure you that our team is continuing to investigate the issue. Details in gitlab.com/gitlab-com/gl-infra/production/-/issues/14260

May 12, 2023 13:26 UTC
[Investigating] We are continuing to investigate the cause of the problem on container registry pulls failing. Details in gitlab.com/gitlab-com/gl-infra/production/-/issues/14260

May 12, 2023 13:16 UTC
[Investigating] We're investigating reports of container registry pulls failing with unknown blob. Details in gitlab.com/gitlab-com/gl-infra/production/-/issues/14260#note_1388488922

May 12, 2023 13:14 UTC
[Investigating] We're investigating reports of container registry pulls failing with unknown blob. Details in gitlab.com/gitlab-com/gl-infra/production/-/issues/14260#note_1388488922

January 2023

High Error Rate on Select GitLab.com

January 5, 2023 11:24 UTC

Incident Status

Service Disruption


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners for GitLab community contributions, SAML SSO - GitLab SaaS, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, Canary


Locations

Google Compute Engine, Digital Ocean, Zendesk, AWS




January 5, 2023 11:24 UTC
[Resolved] Our Engineers identified the root cause and addressed it with a fix. The incident is now mitigated and users are expected to work with MRs without encountering any errors. For details, check gitlab.com/gitlab-com/gl-infra/production/-/issues/8201

January 5, 2023 09:18 UTC
[Monitoring] Services continue to recover and customers reported successful logins though SAML. However, some users are still facing errors with MRs. Please check the issue for the workaround: gitlab.com/gitlab-com/gl-infra/production/-/issues/8201#note_1228554821

January 5, 2023 08:45 UTC
[Monitoring] The error rates are decreasing, but users might still experience errors when authenticating through SAML and working with MRs. Investigation is still ongoing. For details, check gitlab.com/gitlab-com/gl-infra/production/-/issues/8201

January 5, 2023 08:24 UTC
[Monitoring] We are seeing the error rate decreasing and the services recovering. Our Engineers will keep monitoring the systems. For more details, please check gitlab.com/gitlab-com/gl-infra/production/-/issues/8201

January 5, 2023 08:07 UTC
[Investigating] We are currently seeing a high rate of errors on GitLab.com. Our infrastructure team is currently investigating. More information can be found here: gitlab.com/gitlab-com/gl-infra/production/-/issues/8201

November 2022

GitLab.com is unavailable

November 30, 2022 02:04 UTC

GitLab.com is unavailableDegraded Performance

Incident Status

Degraded Performance


Components

API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners for GitLab community contributions, SAML SSO - GitLab SaaS, Background Processing, Canary


Locations

Google Compute Engine




November 30, 2022 02:04 UTC
[Resolved] GitLab.com fully operational. Root cause: An unexpected deployment behaviour affected the Consul cluster that several of our services depend on (eg. Patroni)

November 30, 2022 01:27 UTC
[Monitoring] Monitoring situation, there was a deployment error for GitLab.com: gitlab.com/gitlab-com/gl-infra/production/-/issues/8097

November 30, 2022 01:10 UTC
[Monitoring] GitLab.com is back up, all systems operational.

November 30, 2022 00:54 UTC
[Investigating] We are currently investigating a GitLab.com service disruption

November 30, 2022 00:48 UTC
[Investigating] We have detected a problem related to GitLab.com services and are actively investigating. More details to follow.

October 2022

Intermittent 520 errors while cloning or browsing GitLab.com

October 13, 2022 19:32 UTC

Incident Status

Degraded Performance


Components

Website, Git Operations, Container Registry


Locations

Google Compute Engine




October 13, 2022 19:32 UTC
[Resolved] No further occurrences of this error have been reported during our monitoring period. We consider this incident resolved and will continue to work with our provider for further analysis. Please follow gitlab.com/gitlab-com/gl-infra/production/-/issues/7876 for more information.

October 13, 2022 18:28 UTC
[Monitoring] After initial investigation we believe the root cause is related to a provider incident that has since then been resolved. This also seems to have affected only a small subset of users. Please follow gitlab.com/gitlab-com/gl-infra/production/-/issues/7876 for more information.

October 13, 2022 17:48 UTC
[Investigating] Intermittent 520 errors are seen when attempting to clone from GitLab.com and browsing the Web portal. Both operations seem to work fine after retrying. Please follow gitlab.com/gitlab-com/gl-infra/production/-/issues/7876 for more information.

September 2022

GitLab.com maintenance 2022-09-03 11:00 to 14:00 UTC

September 3, 2022 14:16 UTC

Description

We will be performing maintenance on GitLab.com's production database which is anticipated to include a service downtime of up to 180 minutes between 11:00am to 2:00pm UTC on Saturday 2022-09-03. During this time, users will experience complete service disruption. Improving the performance and reliability of GitLab.com has always been a top priority for GitLab. While we continuously make iterative improvements to GitLab and our production architecture, we anticipate making a larger change to improve the scalability and reliability of GitLab.com: performing necessary Operating System maintenance to our database clusters. You can read more about this change in our blog: about.gitlab.com/blog/2022/08/12/upgrading-database-os


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners for GitLab community contributions, SAML SSO - GitLab SaaS, Background Processing, GitLab Customers Portal, Canary


Locations

Google Compute Engine


Schedule

September 3, 2022 11:00 - September 3, 2022 14:00 UTC



September 3, 2022 14:16 UTC
[Update] GitLab.com's database layer maintenance upgrade is complete now, and we're back up and running. We'll be monitoring the platform to ensure all systems are functioning correctly. Thank you for your patience.

September 3, 2022 14:13 UTC
[Update] GitLab.com planned maintenance for the database layer is almost complete. The site is back up and running although we're continuing to verify that all systems are functioning correctly. Thank you for your patience.

September 3, 2022 13:55 UTC
[Update] We are still performing tests to ensure full functionality of the gitlab.com site before bringing it up. Expect full functionality of gitlab.com in about 30mins

September 3, 2022 10:59 UTC
[Update] GitLab.com is now shutting down for planned maintenance to our database layer for the next 3 hours. See you on the other side! More details in about.gitlab.com/blog/2022/08/12/upgrading-database-os

August 31, 2022 16:00 UTC
[Update] Reminder: We will be performing maintenance on GitLab.com's production database which is anticipated to include a complete service downtime of up to 180 minutes between 11:00am to 2:00pm UTC on Saturday 2022-09-03. You can read more in about.gitlab.com/blog/2022/08/12/upgrading-database-os





Back to current status