Active Incident

Updated a few seconds ago

We continuously monitor gitlab.com and all its services. If there are any performance or service interruptions, an update will be posted here.
Delayed Job Processing on Shared RunnersOperational

Incident Status

Operational

Components

Website, CI/CD

Locations

Google Compute Engine


April 20, 2021 22:09 CEST
April 20, 2021 20:09 UTC
[Monitoring] Job processing on shared runners has recovered and we are continuing to monitor the situation. https://gitlab.com/gitlab-com/gl-infra/production/-/issues/4292

April 20, 2021 21:42 CEST
April 20, 2021 19:42 UTC
[Identified] Correction: The previous message meant to say "Please note that private runners are not affected." Shared runners are affected by this issue.

April 20, 2021 21:33 CEST
April 20, 2021 19:33 UTC
[Identified] We are experiencing an issue with delayed job processing on shared runners. Please note that shared runners are not affected. We have mitigated the issue and are waiting for the backlog to process. https://gitlab.com/gitlab-com/gl-infra/production/-/issues/4292
Issue with Auto DevOpsDegraded Performance

Incident Status

Degraded Performance

Components

CI/CD

Locations

Google Compute Engine


April 20, 2021 22:19 CEST
April 20, 2021 20:19 UTC
[Identified] No material updates to report, the rollback process is still underway. https://gitlab.com/gitlab-com/gl-infra/production/-/issues/4288

April 20, 2021 20:17 CEST
April 20, 2021 18:17 UTC
[Identified] No material updates to report, the rollback process is still underway. https://gitlab.com/gitlab-com/gl-infra/production/-/issues/4288

April 20, 2021 19:18 CEST
April 20, 2021 17:18 UTC
[Identified] The rollback of the impacting change is still underway. A workaround has been identified and may be implemented following the instructions detailed in the issue description here: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/4288

April 20, 2021 17:45 CEST
April 20, 2021 15:45 UTC
[Investigating] We are aware of an issue with Auto DevOps on GitLab.com and have identified the cause. A rollback of the change is being conducted and we will update you once the rollback is complete. https://gitlab.com/gitlab-com/gl-infra/production/-/issues/4288

Scheduled Maintenance

GitLab.com PostgreSQL Database UpgradePlanned Maintenance

Schedule

May 8, 2021 11:00 - 12:00 CEST
May 8, 2021 09:00 - 10:00 UTC

Components

Website, API, Git (ssh and https), Pages, Registry, CI/CD, Background Processing, Windows Runners (beta), Canary

Locations

Google Compute Engine

Description

We'll be upgrading GitLab.com PostgreSQL database to version 12. This maintenance will include primary transition of the cluster within a 1 hour window with downtime during part or all of the window. We intend to minimize the length of downtime, but you should plan for all services will be unavailable during the hour maintenance. Why are we doing this upgrade? 1. Keeping up with new Gitlab.com release support 2. General improvements available in newer versions of PostgreSQL 3. Specific improvements in the PostgreSQL query planner which have recently led to performance impacts and service disruptions for GitLab.com Why does it require downtime? The upgrade approach requires a short downtime while we transition all services from the existing PostgreSQL cluster to this new version 12 PostgreSQL cluster and ensure that all runtime and asynchronous functions are operating as expected. Additional information is available in this GitLab Issue: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/4037

April 16, 2021 19:47 CEST
April 16, 2021 17:47 UTC
[Update] In a final review of our plans for the PostgreSQL upgrade we have decided to postpone this maintenance until May 8th. This date is tentative and will be confirmed prior to April 23rd.

April 14, 2021 20:42 CEST
April 14, 2021 18:42 UTC
[Update] The GitLab.com PostgreSQL Database Upgrade maintenance will take place on Saturday April 17, 2021 09:00 - 10:00 UTC as previously announced.

Website




Operational

API




Operational

Git (ssh and https)




Operational

Pages




Operational

Registry




Operational

CI/CD




Degraded Performance

Background Processing




Operational

Support Services




Operational

packages.gitlab.com




Operational

customers.gitlab.com




Operational

version.gitlab.com




Operational

forum.gitlab.com




Operational

Windows Runners (beta)




Operational

Canary




Operational

dashboards.gitlab.com




Operational

External Services

AWS Route 53

AWS S3

Cloudflare

Docker

Dyn

Fastly

Google Cloud

Mailgun

Zendesk