rssLink RSS for all categories
 
icon_blue
icon_green
icon_green
icon_orange
icon_red
icon_green
icon_green
icon_orange
icon_red
icon_red
icon_red
icon_green
icon_green
icon_red
icon_green
icon_red
icon_green
icon_green
icon_red
icon_red
icon_green
icon_red
icon_green
icon_green
icon_orange
icon_blue
icon_orange
icon_blue
icon_green
icon_blue
icon_red
icon_green
 

FS#37140 — Worker nodes creation

Attached to Project— Kubernetes
Incident
Backend / Core
CLOSED
100%
- Incident summary/Récumé de l'incident: Some nodes are unschedulable due to planned update on Kubernetes 11.5.
- Start time/Heure de début: 02-26-2019 6PM UTC
- Impact / Périmètre affecté: Nodes on Kubernetes cluster version 1.11.5
- Impact type / Type d'impact : Nodes management
- Estimated date to recovery / Date de résolution estimé : 02-27-2019 1AM UTC
- Actions undertaken / Actions entreprises : All of our nodes will be update to 11.7 to reduce the load on OpenStack API.
- Affected hosts / Hôtes affectés: Maximum 1 node per cluster.

Details :
New Kubelet version overload OpenStack API.
Few old nodes are unschedulable. New node deployment are blocked until resolution.
Date:  Tuesday, 16 July 2019, 16:45PM
Reason for closing:  Done
Comment by OVH - Tuesday, 26 February 2019, 00:56AM

Rootcause is identified.
We have started to roll a full update process to 11.7.
Our main goal is the reduction of load on OpenStack API.


Comment by OVH - Tuesday, 26 February 2019, 01:34AM

Successfully fixed 3 nodes.
Attempting fix on a 35 nodes.
Around 100 impacted nodes are still pending.


Comment by OVH - Tuesday, 26 February 2019, 01:54AM

Fix deployed on all impacted nodes.
Starting Kubelet gracefully.


Comment by OVH - Tuesday, 26 February 2019, 02:26AM

140 nodes updated.

Still the same load on Open Stack API.

Investigating.


Comment by OVH - Tuesday, 26 February 2019, 02:55AM

Waiting for OpenStack API to stabilize, requests count decreased a lot.