rssLink RSS for all categories
 
icon_red
icon_green
icon_green
icon_orange
icon_red
icon_green
icon_green
icon_red
icon_red
icon_red
icon_red
icon_green
icon_green
icon_red
icon_green
icon_blue
icon_green
icon_green
icon_red
icon_orange
icon_green
icon_red
icon_green
icon_red
icon_orange
icon_green
icon_green
icon_green
icon_green
icon_green
icon_blue
icon_red
icon_green
 

FS#44655 — LBaaS - GRA

Attached to Project— Cloud
Incident
cloud
CLOSED
100%
• Incident summary / Résumé de l'incident : incident
• Start time / Heure de début : Estimated : 2020-05-20 12:20:00 UTC
• Impact / Périmètre affecté : Load Balancers
• Impact type / Type d'impact : delay delivery
• Estimated time to recovery / Temps de résolution estimé : 1 hour
• Actions undertaken / Actions entreprises : Ports creation process is stuck. We're unlocking the creation process.


Each Load balancer is assigned a port from our Cloud scheduler. This process is currently in a locked state so that few Load balancers delivered since yesteray 8PM are seeing delivery delays issues.
The team is currently working on it.
Date:  Friday, 22 May 2020, 10:51AM
Reason for closing:  Done
Comment by OVH - Wednesday, 20 May 2020, 15:19PM

Lock status has been resolved. We're recovering stucked load balancers


Comment by OVH - Wednesday, 20 May 2020, 15:47PM

We experience anormal long duration calls.


Comment by OVH - Wednesday, 20 May 2020, 20:37PM

root cause has been identified, we are currently rolling the deployment of the fix


Comment by OVH - Wednesday, 20 May 2020, 23:39PM

We've identified the root cause which is linked to the libVirt component. We are not able to quick fix this root cause and we will plan a maintenance later to address it. In the mean time, we've asked our internal customer (K8S) to rollback its Load Balancing strategy to avoid any impact for new services.

We're still catching up the situation for Customers that have spawned a LBaaS, and we are going to update this task until we have remaining impacted customers.


Comment by OVH - Thursday, 21 May 2020, 00:46AM

We're processing functional tests over all deployments


Comment by OVH - Thursday, 21 May 2020, 02:14AM

Tests are OK, we start the reschedule of a first batch of deployments.


Comment by OVH - Thursday, 21 May 2020, 05:59AM

Scheduled load balancers are doing fine. We continue the deployment.
We expect the situation to get back to normal in about 30minutes from now.


Comment by OVH - Thursday, 21 May 2020, 06:28AM

All load balancers are now fully functionnal.
We will publish the post mortem as soon as we can.