All Systems Operational

Updated a few seconds ago

Back to current status

Status History

Filter: Git Operations (Clear)



August 2023

Slightly elevated error rates on Gitaly nodes

August 17, 2023 15:52 UTC

Incident Status

Degraded Performance


Components

Website, API, Git Operations


Locations

Google Compute Engine




August 17, 2023 15:52 UTC
[Resolved] This incident has been resolved. More information can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/16187.

August 17, 2023 14:57 UTC
[Monitoring] We have validated the release in our canary infrastructure. We are in the process of deploying the release production wide. More details can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/16187.

August 17, 2023 13:19 UTC
[Monitoring] We've deployed the fix to our canary infrastructure and validating if the issue is resolved. More details can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/16187.

August 17, 2023 11:14 UTC
[Monitoring] The mitigations steps are complete. We believe we also found the root cause of the issues. More details can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/16187.

August 17, 2023 09:41 UTC
[Identified] We are working on mitigating the issue. More details can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/16187.

August 17, 2023 08:23 UTC
[Identified] No current updates at this time, we are still rolling out the fix to mitigate the impact. More details can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/16187.

August 17, 2023 07:51 UTC
[Identified] We have identified a change that caused the incident and we're in the process of rolling out a fix to mitigate the impact. Meanwhile, we'll continue to manually monitor Gitaly nodes. More information on gitlab.com/gitlab-com/gl-infra/production/-/issues/16187

August 17, 2023 07:23 UTC
[Investigating] No materials for update. Our team is still actively investigate the problem. More information on gitlab.com/gitlab-com/gl-infra/production/-/issues/16187

August 17, 2023 06:08 UTC
[Investigating] We continue to see issue on Gitaly nodes. We're deploying another revert to a known working version. More information on gitlab.com/gitlab-com/gl-infra/production/-/issues/16187

August 17, 2023 04:54 UTC
[Investigating] While we continue to look for a root cause, we are reverting Gitaly to the last known good deployment. More information on gitlab.com/gitlab-com/gl-infra/production/-/issues/16187

August 17, 2023 03:27 UTC
[Investigating] No material updates at this time - our investigation is ongoing. More information on gitlab.com/gitlab-com/gl-infra/production/-/issues/16187.

August 17, 2023 02:13 UTC
[Investigating] All Gitaly nodes are operational. We are investigating and will update in the following hour. More details about this incident can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/16187.

August 17, 2023 01:27 UTC
[Investigating] We have noticed degraded performance with the Gitaly service and applied the fix. We are currently investigating for the root cause. More details about this incident can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/16187.

Unresponsive node causing slightly elevated error rates

August 17, 2023 00:10 UTC

Incident Status

Degraded Performance


Components

Website, API, Git Operations


Locations

Google Compute Engine




August 17, 2023 00:10 UTC
[Resolved] This incident has been resolved and the Gitaly node is now operational. More information can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/16187

August 16, 2023 23:57 UTC
[Investigating] The Gitaly service on one node has become unresponsive. We are troubleshooting the issue and gathering data to understand the root cause.

Slightly elevated error rates between 19:44 - 19:52 UTC

August 17, 2023 00:09 UTC

Incident Status

Operational


Components

Git Operations


Locations

Google Compute Engine




August 17, 2023 00:09 UTC
[Resolved] This incident has been resolved and the Gitaly node is now operational. More information can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/16187

August 16, 2023 20:22 UTC
[Resolved] A Gitaly host experienced a service restart and has been resolved. We will investigate the root cause for this issue, but at this time the host is responding normally and no further issues are anticipated.

CI Cluster PostgreSQL 14 Upgrade

August 12, 2023 16:51 UTC

Description

We will be undergoing scheduled maintenance to our CI database layer. The maintenance will take up to 5 hours, from 14:00 UTC to 19:00 UTC. GitLab.com will continue to be available during the maintenance window. We apologize in advance for any inconvenience this may cause. See gitlab.com/gitlab-com/gl-infra/production/-/issues/16093 for details.


Components

API, Git Operations, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners for GitLab community contributions


Locations

Google Compute Engine


Schedule

August 12, 2023 14:00 - August 12, 2023 19:00 UTC



August 12, 2023 16:51 UTC
[Update] GitLab.com planned maintenance for the database layer is complete. We'll be monitoring the platform to ensure all systems are functioning correctly. Thank you for your patience.

August 12, 2023 16:32 UTC
[Update] GitLab.com planned maintenance for the database layer is complete. We'll be monitoring the platform to ensure all systems are functioning correctly. Thank you for your patience.

August 12, 2023 16:28 UTC
[Update] The scheduled maintenance to our ci database layer is ongoing. The maintenance will take up to 5 hours starting from 14:00 UTC to 19:00 UTC. GitLab.com will be available but users may experience degraded performance during the maintenance window. We apologize in advance for any inconvenience this may cause. See <gitlab.com/gitlab-com/gl-infra/production/-/issues/15945>

August 12, 2023 15:48 UTC
[Update] GitLab.com planned maintenance for the database layer is almost complete. We're continuing to verify that all systems are functioning correctly. Thank you for your patience.

August 12, 2023 13:00 UTC
[Update] We will be undergoing scheduled maintenance to our ci database layer in 1 hour. The maintenance will take up to 5 hours starting from 14:00 UTC to 19:00 UTC. GitLab.com will be available but users may experience degraded performance during the maintenance window. We apologize in advance for any inconvenience this may cause. See <gitlab.com/gitlab-com/gl-infra/production/-/issues/15945>

August 12, 2023 11:00 UTC
[Update] We will be undergoing scheduled maintenance to our ci database layer in 3 hours. The maintenance will take up to 5 hours starting from 14:00 UTC to 19:00 UTC. GitLab.com will be available but users may experience degraded performance during the maintenance window. We apologize in advance for any inconvenience this may cause. See <gitlab.com/gitlab-com/gl-infra/production/-/issues/15945>

August 12, 2023 04:29 UTC
[Update] Reminder: Today, we will be undergoing scheduled maintenance to our CI database layer. The maintenance will take up to 5 hours starting from 14:00 UTC to 19:00 UTC. GitLab.com will be available but users may experience degraded performance during the maintenance window. We apologize in advance for any inconvenience this may cause. See gitlab.com/gitlab-com/gl-infra/production/-/issues/16093

August 9, 2023 15:42 UTC
[Update] In 3 days, we will be undergoing scheduled maintenance to our ci database layer. The maintenance will take up to 5 hours starting from 14:00 UTC to 19:00 UTC. GitLab.com will be available but users may experience degraded performance during the maintenance window. We apologize in advance for any inconvenience this may cause. See <gitlab.com/gitlab-com/gl-infra/production/-/issues/15945>

July 2023

Performance issues on GitLab.com

July 15, 2023 04:51 UTC

Incident Status

Degraded Performance


Components

API, Git Operations, Container Registry, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners for GitLab community contributions, Background Processing, Canary


Locations

Google Compute Engine




July 15, 2023 04:51 UTC
[Resolved] This incident has been marked as resolved, as we don't see any outstanding issues with GitLab.com. A full timeline of this incident can be found in our Incident Report available at gitlab.com/gitlab-com/gl-infra/production/-/issues/16042

July 15, 2023 03:29 UTC
[Monitoring] We will update the status of this incident to `Operational` for now..

July 15, 2023 03:26 UTC
[Monitoring] The rollback has been completed. We are observing that services have recovered. We will continue to monitor for any issues. More details in gitlab.com/gitlab-com/gl-infra/production/-/issues/16042

July 15, 2023 03:02 UTC
[Identified] We are working on performing a rollback on GitLab.com. Next expected update will be in 30 minutes. More details in gitlab.com/gitlab-com/gl-infra/production/-/issues/16042

July 15, 2023 02:52 UTC
[Identified] We are continuing to observe performance issues on GitLab.com. We are investigating options to perform a rollback. More details in gitlab.com/gitlab-com/gl-infra/production/-/issues/16042

July 15, 2023 02:18 UTC
[Identified] No material updates to report at this time. Infrastructure teams are continuing investigations. Details in gitlab.com/gitlab-com/gl-infra/production/-/issues/16042

July 15, 2023 01:45 UTC
[Identified] We have identified the problem and mitigated the worst of it. Our infrastructure team is continuing investigations. Details in gitlab.com/gitlab-com/gl-infra/production/-/issues/16042.

July 15, 2023 01:13 UTC
[Investigating] We have detected a problem related to GitLab.com services and are actively investigating. More details to follow.

503 Errors on GitLab.com

July 10, 2023 11:16 UTC

503 Errors on GitLab.comService Disruption

Incident Status

Service Disruption


Components

Git Operations


Locations

Google Compute Engine




July 10, 2023 11:16 UTC
[Resolved] This incident has been resolved and Git Operations for affected users are operational. A full timeline of this incident can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/15999

July 8, 2023 04:17 UTC
[Monitoring] A limited number of customers were impacted by a git configuration change earlier today. Affected customers have been notified. A review will be posted following our investigation here: gitlab.com/gitlab-com/gl-infra/production/-/issues/15999

July 8, 2023 03:12 UTC
[Identified] No material updates at this time. We are still actively working on restoring Git Operations to full functionality. Additional information will be provided as it becomes available within the following incident issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/15997

July 8, 2023 02:11 UTC
[Identified] No material updates at this time. We are still actively working on restoring Git Operations to full functionality. Additional information will be provided as it becomes available within the following incident issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/15997

July 8, 2023 01:06 UTC
[Identified] No material updates at this time. We are still actively working on restoring Git Operations to full functionality. Additional information will be provided as it becomes available within the following incident issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/15997

July 8, 2023 00:16 UTC
[Identified] No material updates at this time. We are still actively working on restoring Git Operations to full functionality. Additional information will be provided as it becomes available within the following incident issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/15997

July 7, 2023 23:08 UTC
[Identified] No material updates at this time. We are still actively working on restoring Git Operations to full functionality. Additional information will be provided as it becomes available within the following incident issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/15997

July 7, 2023 22:26 UTC
[Identified] We are actively working on restoring Git Operations to full functionality. We will provide additional information as it becomes available within the following incident issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/15997

July 7, 2023 21:26 UTC
[Identified] We're continuing our work to restore Git Operations fully. As additional details become available we will provide them within the incident issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/15997

July 7, 2023 20:51 UTC
[Identified] We're continuing our work to restore Git Operations. We will provide details as they become available in the incident issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/15997

July 7, 2023 20:27 UTC
[Identified] We've confirmed all services as restored except for Git Operations. Git Operations are partially disrupted and we're continuing to restore this service while validating data integrity. Details will be posted in: gitlab.com/gitlab-com/gl-infra/production/-/issues/15997

July 7, 2023 20:01 UTC
[Identified] We are now seeing a restoration of our container registry service. We have identified the root cause of the incident and are working to fully restore the remaining impacted services. Details will be available in: gitlab.com/gitlab-com/gl-infra/production/-/issues/15997

July 7, 2023 19:31 UTC
[Investigating] We have confirmed the status of additional services and updated the status page accordingly. We are still working to fully address the issue - details can be found here: gitlab.com/gitlab-com/gl-infra/production/-/issues/15997

July 7, 2023 19:14 UTC
[Investigating] We have implemented a fix bringing partial service restoration. Our status page has been updated to reflect the known current status of services. We are still investigating other services. Details will be posted in: gitlab.com/gitlab-com/gl-infra/production/-/issues/15997

July 7, 2023 18:46 UTC
[Investigating] We have implemented a fix to mitigate Web/API services. Investigation is ongoing for other services. Details will be provided in this issue after recovery: gitlab.com/gitlab-com/gl-infra/production/-/issues/15997

July 7, 2023 18:18 UTC
[Investigating] We continue to implement our fix to provide partial service restoration while additional investigation continues. We will provide more information here as it becomes available. Full details will be available here once availability is fully restored: gitlab.com/gitlab-com/gl-infra/production/-/issues/15997

July 7, 2023 18:04 UTC
[Investigating] Work continues to implement a fix to provide partial service restoration, in addition to our wider investigation. Once the service is fully restored, we will share detailed information regarding the incident here: gitlab.com/gitlab-com/gl-infra/production/-/issues/15997

July 7, 2023 17:51 UTC
[Investigating] We have identified a path forward for partial service restoration and are working to implement it now. Additional investigation is still ongoing. Once the service is fully restored, we will share detailed information regarding the incident here: gitlab.com/gitlab-com/gl-infra/production/-/issues/15997

July 7, 2023 17:31 UTC
[Investigating] Our investigation to determine the underlying cause of this incident is still in progress. When availability has been restored, we will provide more comprehensive information in the incident issue here: gitlab.com/gitlab-com/gl-infra/production/-/issues/15997

July 7, 2023 17:18 UTC
[Investigating] Our investigation into the root cause of the issue is still underway. Once availability has been restored, full details regarding the incident will be posted in the incident issue here: gitlab.com/gitlab-com/gl-infra/production/-/issues/15997

July 7, 2023 17:02 UTC
[Investigating] Our investigation is still underway. We sincerely apologize for any issues this has caused. We are aware the incident issue is not currently available. Details will be posted here once available: gitlab.com/gitlab-com/gl-infra/production/-/issues/15997

July 7, 2023 16:48 UTC
[Investigating] We are aware of multiple services impacted by this incident. Our investigation is still underway - we will provide updates as they become available. Details will be posted in: gitlab.com/gitlab-com/gl-infra/production/-/issues/15997

July 7, 2023 16:34 UTC
[Investigating] GitLab.com is encountering 503 errors when interacting with the site. Our investigation is underway. Details in: gitlab.com/gitlab-com/gl-infra/production/-/issues/15997

CI Cluster PostgreSQL 14 Upgrade

July 8, 2023 04:34 UTC

Description

We will be undergoing scheduled maintenance to our CI database layer. GitLab.com will continue to be available during the maintenance window. We apologize in advance for any inconvenience this may cause. More details: gitlab.com/gitlab-com/gl-infra/production/-/issues/15945


Components

Website, API, Git Operations, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners for GitLab community contributions


Locations

Google Compute Engine


Schedule

July 8, 2023 14:00 - July 8, 2023 19:00 UTC



July 8, 2023 04:34 UTC
[Update] Database maintenance on our CI cluster, originally scheduled for tomorrow, has been postponed. Once a new date has been determined, we will post a new maintenance entry. Details can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/15945.

July 8, 2023 04:33 UTC
[Update] Database maintenance on our CI cluster, originally scheduled for tomorrow, has been postponed. Once a new date has been determined, we will post a new maintenance entry. Details can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/15945.

July 7, 2023 14:54 UTC
[Update] Tomorrow, we will perform a scheduled maintenance to our CI database layer starting from 14:00 UTC to 19:00 UTC, during which users may experience degraded performance. See gitlab.com/gitlab-com/gl-infra/production/-/issues/15945

July 5, 2023 15:31 UTC
[Update] In 3 days we will perform a scheduled maintenance to our CI database layer starting from 14:00 UTC to 19:00 UTC, during which users may experience degraded performance. See gitlab.com/gitlab-com/gl-infra/production/-/issues/15945

June 2023

Main DB Cluster PostgreSQL 14 Upgrade

June 29, 2023 14:38 UTC

Description

We will be undergoing scheduled maintenance to our database layer. GitLab.com will continue to be available during the maintenance window. We apologize in advance for any inconvenience this may cause. More details: gitlab.com/gitlab-com/gl-infra/production/-/issues/14403


Components

Website, API, Git Operations, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners for GitLab community contributions


Locations

Google Compute Engine


Schedule

July 8, 2023 14:00 - July 8, 2023 19:00 UTC



June 29, 2023 14:38 UTC
[Update] We will be rescheduling the Main DB Cluster upgrade, and a new date will be communicated soon. More details: gitlab.com/gitlab-com/gl-infra/production/-/issues/14403

CI Cluster PostgreSQL 14 Upgrade

June 24, 2023 18:39 UTC

Description

We will be undergoing scheduled maintenance to our CI database layer. GitLab.com will continue to be available during the maintenance window. We apologize in advance for any inconvenience this may cause. More details: gitlab.com/gitlab-com/gl-infra/production/-/issues/14402


Components

Website, API, Git Operations, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners for GitLab community contributions


Locations

Google Compute Engine


Schedule

June 24, 2023 14:00 - June 24, 2023 19:00 UTC



June 24, 2023 18:39 UTC
[Update] GitLab.com rollback for the database layer is complete, and we're back up and running. We'll be monitoring the platform to ensure all systems are functioning correctly. Thank you for your patience.

June 24, 2023 17:54 UTC
[Update] Due to an issue during the planned maintenance for the database layer, we have initiated a rollback of the changes. We will send another update within the next 30 minutes.

June 24, 2023 14:10 UTC
[Update] We now are starting the maintenance to our ci database layer. The maintenance will take up to 5 hours starting from 14:00 UTC to 19:00 UTC. GitLab.com will be available but users may experience degraded performance during the maintenance window. We apologize in advance for any inconvenience this may cause. See <gitlab.com/gitlab-com/gl-infra/production/-/issues/14402>

June 24, 2023 13:02 UTC
[Update] We will be undergoing scheduled maintenance to our ci database layer in 1 hour. The maintenance will take up to 5 hours starting from 14:00 UTC to 19:00 UTC. GitLab.com will be available but users may experience degraded performance during the maintenance window. We apologize in advance for any inconvenience this may cause. See <gitlab.com/gitlab-com/gl-infra/production/-/issues/14402>

June 24, 2023 11:12 UTC
[Update] We will perform a scheduled maintenance to our CI database layer in 3 hours, starting from 14:00 UTC to 19:00 UTC, during which users may experience degraded performance. See gitlab.com/gitlab-com/gl-infra/production/-/issues/14402

June 23, 2023 08:12 UTC
[Update] Tomorrow we will perform a scheduled maintenance to our CI database layer starting from 14:00 UTC to 19:00 UTC, during which users may experience degraded performance. See gitlab.com/gitlab-com/gl-infra/production/-/issues/14402

June 21, 2023 14:56 UTC
[Update] In 3 days, we will be undergoing scheduled maintenance to our ci database layer. The maintenance will take up to 5 hours starting from 14:00 UTC to 19:00 UTC. GitLab.com will be available but users may experience degraded performance during the maintenance window. We apologize in advance for any inconvenience this may cause. See gitlab.com/gitlab-com/gl-infra/production/-/issues/14402

June 17, 2023 14:00 UTC
[Update] Next week, we will be undergoing scheduled maintenance to our ci database layer. The maintenance will take up to 5 hours starting from 14:00 UTC to 19:00 UTC. GitLab.com will be available but users may experience degraded performance during the maintenance window. We apologize in advance for any inconvenience this may cause. See gitlab.com/gitlab-com/gl-infra/production/-/issues/14402

June 10, 2023 11:01 UTC
[Update] In 2 weeks, we will be undergoing scheduled maintenance to our CI database layer. GitLab.com will continue to be available during the maintenance window. We apologize in advance for any inconvenience this may cause. More details: gitlab.com/gitlab-com/gl-infra/production/-/issues/14402

High Error Rate on Select GitLab.com Repositories

June 22, 2023 12:27 UTC

Incident Status

Operational


Components

Git Operations


Locations

Google Compute Engine




June 22, 2023 12:27 UTC
[Resolved] This incident has been marked as resolved. Details and timeline available in gitlab.com/gitlab-com/gl-infra/production/-/issues/15911

June 22, 2023 11:15 UTC
[Monitoring] Error rates have stabilized. We continue to monitor the situation closely. Read more on the investigation over at gitlab.com/gitlab-com/gl-infra/production/-/issues/15911

June 22, 2023 11:06 UTC
[Monitoring] While our SREs are investigating possible causes, error rates are already recovering. We continue to monitor the situation closely. For more details, check the following GitLab issue: gitlab.com/gitlab-com/gl-infra/production/-/issues/15911

June 22, 2023 10:59 UTC
[Investigating] We are currently seeing a high rate of errors for some repositories on GitLab.com. Our infrastructure team is currently investigating.

Slow requests on GitLab.com

June 7, 2023 13:32 UTC

Slow requests on GitLab.comPartial Service Disruption

Incident Status

Partial Service Disruption


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners for GitLab community contributions, SAML SSO - GitLab SaaS, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, docs.gitlab.com, Canary


Locations

Google Compute Engine, Digital Ocean, Zendesk, AWS




June 7, 2023 13:32 UTC
[Resolved] We have identified the root cause, have seen recovery and are therefore resolving this incident. More information in: gitlab.com/gitlab-com/gl-infra/production/-/issues/14839

June 7, 2023 13:21 UTC
[Investigating] Customers are experiencing slow requests on GitLab.com. More information in: gitlab.com/gitlab-com/gl-infra/production/-/issues/14839

May 2023

Hardware upgrade for the GitLab.com datastore

May 27, 2023 04:06 UTC

Description

In 2 weeks, as part of a planned maintenance window, we will perform a hardware upgrade for the GitLab.com datastore. Users may experience temporary 50X errors for a brief period during this window. See gitlab.com/gitlab-com/gl-infra/production/-/issues/10694


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners for GitLab community contributions, SAML SSO - GitLab SaaS, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, docs.gitlab.com, Canary


Locations

Google Compute Engine, Digital Ocean, Zendesk, AWS


Schedule

May 27, 2023 03:00 - May 27, 2023 04:00 UTC



May 27, 2023 04:06 UTC
[Update] GitLab.com's database layer maintenance is complete now, and we're fully back up and running. We'll be monitoring the platform to ensure all systems are functioning correctly. Thank you for your patience.

May 27, 2023 03:41 UTC
[Update] The database switchover is now complete, we expect all SQL statements to be routed to the new nodes. The site is back up and we're continuing to verify that all systems are functioning correctly. Thank you for your patience.

May 27, 2023 03:00 UTC
[Update] The database maintenance is starting now. The database Primary node switchover should happen at any moment in the next hour and during the switchover users might see 50X errors for a very brief span of time. For details see gitlab.com/gitlab-com/gl-infra/production/-/issues/10694

May 27, 2023 01:51 UTC
[Update] We will be conducting a maintenance database activity in an hour, from 03:00 to 04:00 UTC. Users might see 50X errors for a very brief span of time during this window. For details see gitlab.com/gitlab-com/gl-infra/production/-/issues/10694

May 26, 2023 02:27 UTC
[Update] We will be conducting a maintenance database activity Tomorrow, 2023-05-27, from 03:00 to 04:00 UTC. Users might see 50X errors for a very brief span of time during this window. For details see gitlab.com/gitlab-com/gl-infra/production/-/issues/10694

May 24, 2023 02:14 UTC
[Update] Correction: We will be conducting a maintenance database activity this Saturday, 2023-05-27. from 03:00 to 04:00 UTC. Users might see 50X errors for a very brief span of time during this window. For details see gitlab.com/gitlab-com/gl-infra/production/-/issues/10694

May 24, 2023 02:05 UTC
[Update] We will be conducting a maintenance database activity this Saturday, 2023-05-20, from 03:00 to 04:00 UTC. Users might see 50X errors for a very brief span of time during this window. For details see gitlab.com/gitlab-com/gl-infra/production/-/issues/10694

May 20, 2023 03:58 UTC
[Update] The maintenance is rescheduled to next Saturday, 2023-05-27, from 03:00 to 04:00 UTC. There is currently a long database migration running to fix an incident (gitlab.com/gitlab-com/gl-infra/production/-/issues/14468) that we can't risk to interrupt. We apologise for any inconvenience.

May 20, 2023 03:07 UTC
[Update] The scheduled maintenance will be not be starting at 03:00 UTC, and has been delayed. More details soon to follow.

May 20, 2023 02:00 UTC
[Update] We will be conducting a maintenance database activity in an hour, from 03:00 to 04:00 UTC. Users might see 50X errors for a very brief span of time during this window. For details see gitlab.com/gitlab-com/gl-infra/production/-/issues/10694

May 20, 2023 01:00 UTC
[Update] We will be conducting a maintenance database activity in 2 hours, from 03:00 to 04:00 UTC. Users might see 50X errors for a very brief span of time during this window. For details see gitlab.com/gitlab-com/gl-infra/production/-/issues/10694

May 19, 2023 02:00 UTC
[Update] We will be conducting a maintenance database activity tomorrow, 2023-05-20, from 03:00 to 04:00 UTC. Users might see 50X errors for a very brief span of time during this window. For details see gitlab.com/gitlab-com/gl-infra/production/-/issues/10694

May 17, 2023 02:00 UTC
[Update] We will be conducting a maintenance database activity this Saturday, 2023-05-20, from 03:00 to 04:00 UTC. Users might see 50X errors for a very brief span of time during this window. For details see gitlab.com/gitlab-com/gl-infra/production/-/issues/10694

May 12, 2023 02:00 UTC
[Update] Next week, as part of a planned maintenance window on 2023-05-20 from 03:00 to 04:00 UTC, We will perform a hardware upgrade for the GitLab.com datastore. Users may experience temporary 50X errors for a brief period during this window. See gitlab.com/gitlab-com/gl-infra/production/-/issues/10694

GitLab.com partial disruption

May 17, 2023 20:59 UTC

GitLab.com partial disruptionPartial Service Disruption

Incident Status

Partial Service Disruption


Components

Website, API, Git Operations


Locations

Google Compute Engine




May 17, 2023 20:59 UTC
[Resolved] Thanks for your patience. Our monitoring shows that everything is operating normally. You can review the incident in gitlab.com/gitlab-com/gl-infra/production/-/issues/14422

May 17, 2023 20:42 UTC
[Monitoring] We have addressed the issue, and have seen evidence that everything is operational again. We will monitor for 15 minutes before resolving. We will post a further update at that time.

May 17, 2023 20:38 UTC
[Identified] We have identified the certificate issue tied to the internal API. We continue to work towards resolving it. gitlab.com/gitlab-com/gl-infra/production/-/issues/14422

May 17, 2023 20:28 UTC
[Identified] We believe that we have identified the issue to be certificate related and are working on mitigating. For more, please see gitlab.com/gitlab-com/gl-infra/production/-/issues/14422

May 17, 2023 20:17 UTC
[Investigating] We're seeing errors on GitLab.com in relation to a number of features including WebIDE and scripts in runner jobs. We are investigating. Details in gitlab.com/gitlab-com/gl-infra/production/-/issues/14422

January 2023

High Error Rate on Select GitLab.com

January 5, 2023 11:24 UTC

Incident Status

Service Disruption


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners for GitLab community contributions, SAML SSO - GitLab SaaS, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, Canary


Locations

Google Compute Engine, Digital Ocean, Zendesk, AWS




January 5, 2023 11:24 UTC
[Resolved] Our Engineers identified the root cause and addressed it with a fix. The incident is now mitigated and users are expected to work with MRs without encountering any errors. For details, check gitlab.com/gitlab-com/gl-infra/production/-/issues/8201

January 5, 2023 09:18 UTC
[Monitoring] Services continue to recover and customers reported successful logins though SAML. However, some users are still facing errors with MRs. Please check the issue for the workaround: gitlab.com/gitlab-com/gl-infra/production/-/issues/8201#note_1228554821

January 5, 2023 08:45 UTC
[Monitoring] The error rates are decreasing, but users might still experience errors when authenticating through SAML and working with MRs. Investigation is still ongoing. For details, check gitlab.com/gitlab-com/gl-infra/production/-/issues/8201

January 5, 2023 08:24 UTC
[Monitoring] We are seeing the error rate decreasing and the services recovering. Our Engineers will keep monitoring the systems. For more details, please check gitlab.com/gitlab-com/gl-infra/production/-/issues/8201

January 5, 2023 08:07 UTC
[Investigating] We are currently seeing a high rate of errors on GitLab.com. Our infrastructure team is currently investigating. More information can be found here: gitlab.com/gitlab-com/gl-infra/production/-/issues/8201

November 2022

GitLab.com is unavailable

November 30, 2022 02:04 UTC

GitLab.com is unavailableDegraded Performance

Incident Status

Degraded Performance


Components

API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners for GitLab community contributions, SAML SSO - GitLab SaaS, Background Processing, Canary


Locations

Google Compute Engine




November 30, 2022 02:04 UTC
[Resolved] GitLab.com fully operational. Root cause: An unexpected deployment behaviour affected the Consul cluster that several of our services depend on (eg. Patroni)

November 30, 2022 01:27 UTC
[Monitoring] Monitoring situation, there was a deployment error for GitLab.com: gitlab.com/gitlab-com/gl-infra/production/-/issues/8097

November 30, 2022 01:10 UTC
[Monitoring] GitLab.com is back up, all systems operational.

November 30, 2022 00:54 UTC
[Investigating] We are currently investigating a GitLab.com service disruption

November 30, 2022 00:48 UTC
[Investigating] We have detected a problem related to GitLab.com services and are actively investigating. More details to follow.

Elevated error rate on GitLab.com

November 9, 2022 11:37 UTC

Incident Status

Service Disruption


Components

Website, Git Operations


Locations

Google Compute Engine




November 9, 2022 11:37 UTC
[Resolved] The issue has been mitigated. All services are operational.

November 9, 2022 10:54 UTC
[Monitoring] A fix was applied and the incident is now mitigated. Our Engineers will continue monitoring the systems. For more details please see: gitlab.com/gitlab-com/gl-infra/production/-/issues/8025

November 9, 2022 10:24 UTC
[Investigating] Our Engineers are still working on finding the root cause and implementing a fix. For more details please see gitlab.com/gitlab-com/gl-infra/production/-/issues/8025

November 9, 2022 09:53 UTC
[Investigating] We are seeing an increase in error rate on GitLab.com. Our Engineers are working on identifying the root cause and implementing a fix. For details, please see: gitlab.com/gitlab-com/gl-infra/production/-/issues/8025

October 2022

Intermittent 520 errors while cloning or browsing GitLab.com

October 13, 2022 19:32 UTC

Incident Status

Degraded Performance


Components

Website, Git Operations, Container Registry


Locations

Google Compute Engine




October 13, 2022 19:32 UTC
[Resolved] No further occurrences of this error have been reported during our monitoring period. We consider this incident resolved and will continue to work with our provider for further analysis. Please follow gitlab.com/gitlab-com/gl-infra/production/-/issues/7876 for more information.

October 13, 2022 18:28 UTC
[Monitoring] After initial investigation we believe the root cause is related to a provider incident that has since then been resolved. This also seems to have affected only a small subset of users. Please follow gitlab.com/gitlab-com/gl-infra/production/-/issues/7876 for more information.

October 13, 2022 17:48 UTC
[Investigating] Intermittent 520 errors are seen when attempting to clone from GitLab.com and browsing the Web portal. Both operations seem to work fine after retrying. Please follow gitlab.com/gitlab-com/gl-infra/production/-/issues/7876 for more information.

Elevated error rate on GitLab.com

October 12, 2022 09:38 UTC

Incident Status

Degraded Performance


Components

Website, API, Git Operations


Locations

Google Compute Engine




October 12, 2022 09:38 UTC
[Resolved] All the services seems to have recovered, and we no longer see any 5XX errors. Our Engineers are still investigating the root cause. More information can be found here: gitlab.com/gitlab-com/gl-infra/production/-/issues/7864

October 12, 2022 08:58 UTC
[Monitoring] We are seeing the error rate decreasing and the services recovering. Our Engineers will keep monitoring the systems. For more details, check gitlab.com/gitlab-com/gl-infra/production/-/issues/7864

October 12, 2022 08:38 UTC
[Investigating] Gitlab.com have experienced an increase in 5XX errors. Our Engineers have made some changes to eliminate the errors, and they are still investigating the root cause. For details, check gitlab.com/gitlab-com/gl-infra/production/-/issues/7864

September 2022

GitLab.com maintenance 2022-09-03 11:00 to 14:00 UTC

September 3, 2022 14:16 UTC

Description

We will be performing maintenance on GitLab.com's production database which is anticipated to include a service downtime of up to 180 minutes between 11:00am to 2:00pm UTC on Saturday 2022-09-03. During this time, users will experience complete service disruption. Improving the performance and reliability of GitLab.com has always been a top priority for GitLab. While we continuously make iterative improvements to GitLab and our production architecture, we anticipate making a larger change to improve the scalability and reliability of GitLab.com: performing necessary Operating System maintenance to our database clusters. You can read more about this change in our blog: about.gitlab.com/blog/2022/08/12/upgrading-database-os


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners for GitLab community contributions, SAML SSO - GitLab SaaS, Background Processing, GitLab Customers Portal, Canary


Locations

Google Compute Engine


Schedule

September 3, 2022 11:00 - September 3, 2022 14:00 UTC



September 3, 2022 14:16 UTC
[Update] GitLab.com's database layer maintenance upgrade is complete now, and we're back up and running. We'll be monitoring the platform to ensure all systems are functioning correctly. Thank you for your patience.

September 3, 2022 14:13 UTC
[Update] GitLab.com planned maintenance for the database layer is almost complete. The site is back up and running although we're continuing to verify that all systems are functioning correctly. Thank you for your patience.

September 3, 2022 13:55 UTC
[Update] We are still performing tests to ensure full functionality of the gitlab.com site before bringing it up. Expect full functionality of gitlab.com in about 30mins

September 3, 2022 10:59 UTC
[Update] GitLab.com is now shutting down for planned maintenance to our database layer for the next 3 hours. See you on the other side! More details in about.gitlab.com/blog/2022/08/12/upgrading-database-os

August 31, 2022 16:00 UTC
[Update] Reminder: We will be performing maintenance on GitLab.com's production database which is anticipated to include a complete service downtime of up to 180 minutes between 11:00am to 2:00pm UTC on Saturday 2022-09-03. You can read more in about.gitlab.com/blog/2022/08/12/upgrading-database-os

August 2022

GitLab.com slow to respond

August 26, 2022 06:21 UTC

GitLab.com slow to respondDegraded Performance

Incident Status

Degraded Performance


Components

Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on Windows, CI/CD - Hosted runners for GitLab community contributions, SAML SSO - GitLab SaaS, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, Canary


Locations

Google Compute Engine, Digital Ocean, Zendesk, AWS




August 26, 2022 06:21 UTC
[Resolved] We are no longer seeing connectivity issues for GitLab.com after a period of monitoring. We therefore believe that this incident has been resolved. Details in : gitlab.com/gitlab-com/gl-infra/production/-/issues/7656

August 26, 2022 05:08 UTC
[Monitoring] All the services seems to have recovered. We will continue to monitor the performance and provide the updates here. More information on the root cause can be found here: gitlab.com/gitlab-com/gl-infra/production/-/issues/7656

August 26, 2022 04:57 UTC
[Investigating] We are still investigating the root cause of this issue. More details here: gitlab.com/gitlab-com/gl-infra/production/-/issues/7656

August 26, 2022 04:55 UTC
[Investigating] We are noticing some slow response issues with GitLab.com.





Back to current status