OVHcloud Network Status
Current status
Legend
- Operational
- Degraded performance
- Partial Outage
- Major Outage
- Under maintenance
37.187.28.248/249
Incident
Report for Network & Infrastructure
Resolved
Les switch 37.187.28.248 et 37.187.28.249 ont rebooté une après les autres.
Reason: Reset triggered due to HA policy of Reset
Service: eth_port_sec hap reset
Ce bug est connu par Cisco, nous allons effectuer la mise-à-jours de ce switch cette semaine.
Update(s):
Date: 2016-05-14 08:17:43 UTC
L'impact constaté pendant la mise à jour est de 9h25 à 10h00 GMT+2.
Tout semble rentré dans l'ordre dans nos outils de monitoring
Date: 2016-05-14 08:02:29 UTC
La config a été repush, les interfaces up sont désormais identiques sur les 2 switchs, la redondance est de nouveau assurée.
Nous contrôlons nos outils de monitoring pour valider que tout est rentré dans l'ordre.
Date: 2016-05-14 07:52:26 UTC
Certaines interfaces du FEX 101 posent problème sur le b (provisionning missing).
Nous réappliquons la configuration.
Date: 2016-05-14 07:48:31 UTC
Tous les ports sont up sur le 249. Les ports finisent de remonter sur le 248.
Date: 2016-05-14 07:45:51 UTC
Tous les FEX sont online sur les 2 Nexus.
Les ports remontent
Date: 2016-05-14 07:38:31 UTC
5 FEX sur 11 sont online sur le 249, les serveurs reviennent.
Date: 2016-05-14 07:35:23 UTC
la VPC est up, les FEX remontent petit à petit.
ETA : 10min
Date: 2016-05-14 07:34:30 UTC
Le 248 vient de rebooter, il est en train d'appliquer sa config.
Date: 2016-05-14 07:33:36 UTC
Le preload des FEX ne semblent pas s'être passé correctement. Les FEX rebootent après l'image download.
Date: 2016-05-14 07:30:59 UTC
Le switch 248 est en train de rebooter, les FEX raccrochent sur le 249 petit à petit.
Date: 2016-05-14 07:22:59 UTC
C'est parti pour le 248
sw.37.187.28.248# install all kickstart n5000-uk9-kickstart.7.1.3.N1.2a.bin system n5000-uk9.7.1.3.N1.2a.bin
Verifying image bootflash:/n5000-uk9-kickstart.7.1.3.N1.2a.bin for boot variable \"kickstart\".
[####################] 100% -- SUCCESS
Verifying image bootflash:/n5000-uk9.7.1.3.N1.2a.bin for boot variable \"system\".
[####################] 100% -- SUCCESS
Verifying image type.
[####################] 100% -- SUCCESS
Extracting \"system\" version from image bootflash:/n5000-uk9.7.1.3.N1.2a.bin.
[####################] 100% -- SUCCESS
Extracting \"kickstart\" version from image bootflash:/n5000-uk9-kickstart.7.1.3.N1.2a.bin.
[####################] 100% -- SUCCESS
Extracting \"bios\" version from image bootflash:/n5000-uk9.7.1.3.N1.2a.bin.
[####################] 100% -- SUCCESS
Extracting \"fexth\" version from image bootflash:/n5000-uk9.7.1.3.N1.2a.bin.
[####################] 100% -- SUCCESS
Performing module support checks.
[####################] 100% -- SUCCESS
Notifying services about system upgrade.
[####################] 100% -- SUCCESS
Compatibility check is done:
Module bootable Impact Install-type Reason
------ -------- -------------- ------------ ------
1 yes disruptive reset Incompatible image
3 yes disruptive reset Incompatible image
100 yes disruptive reset Incompatible image
101 yes disruptive reset Incompatible image
102 yes disruptive reset Incompatible image
103 yes disruptive reset Incompatible image
104 yes disruptive reset Incompatible image
105 yes disruptive reset Incompatible image
106 yes disruptive reset Incompatible image
107 yes disruptive reset Incompatible image
108 yes disruptive reset Incompatible image
109 yes disruptive reset Incompatible image
111 yes disruptive reset Incompatible image
Images will be upgraded according to following table:
Module Image Running-Version New-Version Upg-Required
------ ---------------- ---------------------- ---------------------- ------------
1 system 6.0(2)N2(2) 7.1(3)N1(2a) yes
1 kickstart 6.0(2)N2(2) 7.1(3)N1(2a) yes
1 bios v3.6.0(05/09/2012) v3.6.0(05/09/2012) no
1 power-seq v2.0 v3.0 yes
1 SFP-uC v1.1.0.0 v1.0.0.0 no
3 power-seq v2.0 v2.0 no
100 fexth 6.0(2)N2(2) 7.1(3)N1(2a) yes
101 fexth 6.0(2)N2(2) 7.1(3)N1(2a) yes
102 fexth 6.0(2)N2(2) 7.1(3)N1(2a) yes
103 fexth 6.0(2)N2(2) 7.1(3)N1(2a) yes
104 fexth 6.0(2)N2(2) 7.1(3)N1(2a) yes
105 fexth 6.0(2)N2(2) 7.1(3)N1(2a) yes
106 fexth 6.0(2)N2(2) 7.1(3)N1(2a) yes
107 fexth 6.0(2)N2(2) 7.1(3)N1(2a) yes
108 fexth 6.0(2)N2(2) 7.1(3)N1(2a) yes
109 fexth 6.0(2)N2(2) 7.1(3)N1(2a) yes
111 fexth 6.0(2)N2(2) 7.1(3)N1(2a) yes
1 microcontroller v1.2.0.1 v1.2.0.1 no
Switch will be reloaded for disruptive upgrade.
Do you want to continue with the installation (y/n)? [n] y
Date: 2016-05-14 07:20:11 UTC
le switch est à jour, les FEX sont cependant en version mismatch.
Nous lancons la mise sur le primaire afin de terminer la mise à jour de ce couple. Une coupure est possible lorsque les FEX réappliquerons leur MaJ et configuration des ports.
Date: 2016-05-14 07:11:49 UTC
sw.37.187.28.249# System is still initializing
Configuration mode is blocked until system is ready
Date: 2016-05-14 07:04:18 UTC
Pre-loading modules.
[This step might take upto 20 minutes to complete - please wait.]
[*Warning -- Please do not abort installation/reload or powercycle fexes*]
[####################] 100% -- SUCCESS
Finishing the upgrade, switch will reboot in 10 seconds.
Le sw 249 reboote
Date: 2016-05-14 06:56:58 UTC
Le preload des images des FEX sont en cours. Une fois le preload done, le switch rebootera dans la dernière version et nous upgraderons le switch primaire.
Date: 2016-05-14 06:53:28 UTC
La mise à jour du secondaire est en cours en disruptive:
sw.37.187.28.249# install all kickstart n5000-uk9-kickstart.7.1.3.N1.2a.bin system n5000-uk9.7.1.3.N1.2a.bin ?
ssi Boot-variable name
sw.37.187.28.249# install all kickstart n5000-uk9-kickstart.7.1.3.N1.2a.bin system n5000-uk9.7.1.3.N1.2a.bin
Verifying image bootflash:/n5000-uk9-kickstart.7.1.3.N1.2a.bin for boot variable \"kickstart\".
[####################] 100% -- SUCCESS
Verifying image bootflash:/n5000-uk9.7.1.3.N1.2a.bin for boot variable \"system\".
[####################] 100% -- SUCCESS
Verifying image type.
[####################] 100% -- SUCCESS
Extracting \"system\" version from image bootflash:/n5000-uk9.7.1.3.N1.2a.bin.
[####################] 100% -- SUCCESS
Extracting \"kickstart\" version from image bootflash:/n5000-uk9-kickstart.7.1.3.N1.2a.bin.
[####################] 100% -- SUCCESS
Extracting \"bios\" version from image bootflash:/n5000-uk9.7.1.3.N1.2a.bin.
[####################] 100% -- SUCCESS
Extracting \"fexth\" version from image bootflash:/n5000-uk9.7.1.3.N1.2a.bin.
[####################] 100% -- SUCCESS
Performing module support checks.
[####################] 100% -- SUCCESS
Notifying services about system upgrade.
[####################] 100% -- SUCCESS
Compatibility check is done:
Module bootable Impact Install-type Reason
------ -------- -------------- ------------ ------
1 yes disruptive reset Incompatible image
3 yes disruptive reset Incompatible image
100 yes disruptive reset Incompatible image
101 yes disruptive reset Incompatible image
102 yes disruptive reset Incompatible image
103 yes disruptive reset Incompatible image
104 yes disruptive reset Incompatible image
105 yes disruptive reset Incompatible image
106 yes disruptive reset Incompatible image
107 yes disruptive reset Incompatible image
108 yes disruptive reset Incompatible image
109 yes disruptive reset Incompatible image
111 yes disruptive reset Incompatible image
Images will be upgraded according to following table:
Module Image Running-Version New-Version Upg-Required
------ ---------------- ---------------------- ---------------------- ------------
1 system 6.0(2)N2(2) 7.1(3)N1(2a) yes
1 kickstart 6.0(2)N2(2) 7.1(3)N1(2a) yes
1 bios v3.6.0(05/09/2012) v3.6.0(05/09/2012) no
1 power-seq v2.0 v3.0 yes
1 SFP-uC v1.1.0.0 v1.0.0.0 no
3 power-seq v2.0 v2.0 no
100 fexth 6.0(2)N2(2) 7.1(3)N1(2a) yes
101 fexth 6.0(2)N2(2) 7.1(3)N1(2a) yes
102 fexth 6.0(2)N2(2) 7.1(3)N1(2a) yes
103 fexth 6.0(2)N2(2) 7.1(3)N1(2a) yes
104 fexth 6.0(2)N2(2) 7.1(3)N1(2a) yes
105 fexth 6.0(2)N2(2) 7.1(3)N1(2a) yes
106 fexth 6.0(2)N2(2) 7.1(3)N1(2a) yes
107 fexth 6.0(2)N2(2) 7.1(3)N1(2a) yes
108 fexth 6.0(2)N2(2) 7.1(3)N1(2a) yes
109 fexth 6.0(2)N2(2) 7.1(3)N1(2a) yes
111 fexth 6.0(2)N2(2) 7.1(3)N1(2a) yes
1 microcontroller v1.2.0.1 v1.2.0.1 no
Switch will be reloaded for disruptive upgrade.
Do you want to continue with the installation (y/n)? [n] y
Install is in progress, please wait.
Date: 2016-05-14 06:06:07 UTC
%SYSMGR-2-SERVICE_CRASHED: Service \"eth_port_sec\" (PID 4853) hasn't caught signal 6 (core will be saved).
%SYSMGR-2-HAP_FAILURE_SUP_RESET: System reset due to service \"eth_port_sec\" in vdc 1 has had a hap failure
Date: 2016-05-14 06:05:15 UTC
Nous continuons d'avoir des problèmes sur ces switchs. Nous upgradons en dernière version.
Date: 2016-05-14 03:09:09 UTC
Il s'agit du même problème hap_reset.
Reason: Reset triggered due to HA policy of Reset
Service: eth_port_sec hap reset
Si le problème se reproduit de nouveau dans les heures qui suivent, nous ferons la mise à jour à chaud.
Date: 2016-05-14 03:00:14 UTC
Nous avons rencontré de nouveau le problème sur le couple. Nous investiguons pour juger de l'urgence de l'upgrade ou non.
Reason: Reset triggered due to HA policy of Reset
Service: eth_port_sec hap reset
Ce bug est connu par Cisco, nous allons effectuer la mise-à-jours de ce switch cette semaine.
Update(s):
Date: 2016-05-14 08:17:43 UTC
L'impact constaté pendant la mise à jour est de 9h25 à 10h00 GMT+2.
Tout semble rentré dans l'ordre dans nos outils de monitoring
Date: 2016-05-14 08:02:29 UTC
La config a été repush, les interfaces up sont désormais identiques sur les 2 switchs, la redondance est de nouveau assurée.
Nous contrôlons nos outils de monitoring pour valider que tout est rentré dans l'ordre.
Date: 2016-05-14 07:52:26 UTC
Certaines interfaces du FEX 101 posent problème sur le b (provisionning missing).
Nous réappliquons la configuration.
Date: 2016-05-14 07:48:31 UTC
Tous les ports sont up sur le 249. Les ports finisent de remonter sur le 248.
Date: 2016-05-14 07:45:51 UTC
Tous les FEX sont online sur les 2 Nexus.
Les ports remontent
Date: 2016-05-14 07:38:31 UTC
5 FEX sur 11 sont online sur le 249, les serveurs reviennent.
Date: 2016-05-14 07:35:23 UTC
la VPC est up, les FEX remontent petit à petit.
ETA : 10min
Date: 2016-05-14 07:34:30 UTC
Le 248 vient de rebooter, il est en train d'appliquer sa config.
Date: 2016-05-14 07:33:36 UTC
Le preload des FEX ne semblent pas s'être passé correctement. Les FEX rebootent après l'image download.
Date: 2016-05-14 07:30:59 UTC
Le switch 248 est en train de rebooter, les FEX raccrochent sur le 249 petit à petit.
Date: 2016-05-14 07:22:59 UTC
C'est parti pour le 248
sw.37.187.28.248# install all kickstart n5000-uk9-kickstart.7.1.3.N1.2a.bin system n5000-uk9.7.1.3.N1.2a.bin
Verifying image bootflash:/n5000-uk9-kickstart.7.1.3.N1.2a.bin for boot variable \"kickstart\".
[####################] 100% -- SUCCESS
Verifying image bootflash:/n5000-uk9.7.1.3.N1.2a.bin for boot variable \"system\".
[####################] 100% -- SUCCESS
Verifying image type.
[####################] 100% -- SUCCESS
Extracting \"system\" version from image bootflash:/n5000-uk9.7.1.3.N1.2a.bin.
[####################] 100% -- SUCCESS
Extracting \"kickstart\" version from image bootflash:/n5000-uk9-kickstart.7.1.3.N1.2a.bin.
[####################] 100% -- SUCCESS
Extracting \"bios\" version from image bootflash:/n5000-uk9.7.1.3.N1.2a.bin.
[####################] 100% -- SUCCESS
Extracting \"fexth\" version from image bootflash:/n5000-uk9.7.1.3.N1.2a.bin.
[####################] 100% -- SUCCESS
Performing module support checks.
[####################] 100% -- SUCCESS
Notifying services about system upgrade.
[####################] 100% -- SUCCESS
Compatibility check is done:
Module bootable Impact Install-type Reason
------ -------- -------------- ------------ ------
1 yes disruptive reset Incompatible image
3 yes disruptive reset Incompatible image
100 yes disruptive reset Incompatible image
101 yes disruptive reset Incompatible image
102 yes disruptive reset Incompatible image
103 yes disruptive reset Incompatible image
104 yes disruptive reset Incompatible image
105 yes disruptive reset Incompatible image
106 yes disruptive reset Incompatible image
107 yes disruptive reset Incompatible image
108 yes disruptive reset Incompatible image
109 yes disruptive reset Incompatible image
111 yes disruptive reset Incompatible image
Images will be upgraded according to following table:
Module Image Running-Version New-Version Upg-Required
------ ---------------- ---------------------- ---------------------- ------------
1 system 6.0(2)N2(2) 7.1(3)N1(2a) yes
1 kickstart 6.0(2)N2(2) 7.1(3)N1(2a) yes
1 bios v3.6.0(05/09/2012) v3.6.0(05/09/2012) no
1 power-seq v2.0 v3.0 yes
1 SFP-uC v1.1.0.0 v1.0.0.0 no
3 power-seq v2.0 v2.0 no
100 fexth 6.0(2)N2(2) 7.1(3)N1(2a) yes
101 fexth 6.0(2)N2(2) 7.1(3)N1(2a) yes
102 fexth 6.0(2)N2(2) 7.1(3)N1(2a) yes
103 fexth 6.0(2)N2(2) 7.1(3)N1(2a) yes
104 fexth 6.0(2)N2(2) 7.1(3)N1(2a) yes
105 fexth 6.0(2)N2(2) 7.1(3)N1(2a) yes
106 fexth 6.0(2)N2(2) 7.1(3)N1(2a) yes
107 fexth 6.0(2)N2(2) 7.1(3)N1(2a) yes
108 fexth 6.0(2)N2(2) 7.1(3)N1(2a) yes
109 fexth 6.0(2)N2(2) 7.1(3)N1(2a) yes
111 fexth 6.0(2)N2(2) 7.1(3)N1(2a) yes
1 microcontroller v1.2.0.1 v1.2.0.1 no
Switch will be reloaded for disruptive upgrade.
Do you want to continue with the installation (y/n)? [n] y
Date: 2016-05-14 07:20:11 UTC
le switch est à jour, les FEX sont cependant en version mismatch.
Nous lancons la mise sur le primaire afin de terminer la mise à jour de ce couple. Une coupure est possible lorsque les FEX réappliquerons leur MaJ et configuration des ports.
Date: 2016-05-14 07:11:49 UTC
sw.37.187.28.249# System is still initializing
Configuration mode is blocked until system is ready
Date: 2016-05-14 07:04:18 UTC
Pre-loading modules.
[This step might take upto 20 minutes to complete - please wait.]
[*Warning -- Please do not abort installation/reload or powercycle fexes*]
[####################] 100% -- SUCCESS
Finishing the upgrade, switch will reboot in 10 seconds.
Le sw 249 reboote
Date: 2016-05-14 06:56:58 UTC
Le preload des images des FEX sont en cours. Une fois le preload done, le switch rebootera dans la dernière version et nous upgraderons le switch primaire.
Date: 2016-05-14 06:53:28 UTC
La mise à jour du secondaire est en cours en disruptive:
sw.37.187.28.249# install all kickstart n5000-uk9-kickstart.7.1.3.N1.2a.bin system n5000-uk9.7.1.3.N1.2a.bin ?
ssi Boot-variable name
sw.37.187.28.249# install all kickstart n5000-uk9-kickstart.7.1.3.N1.2a.bin system n5000-uk9.7.1.3.N1.2a.bin
Verifying image bootflash:/n5000-uk9-kickstart.7.1.3.N1.2a.bin for boot variable \"kickstart\".
[####################] 100% -- SUCCESS
Verifying image bootflash:/n5000-uk9.7.1.3.N1.2a.bin for boot variable \"system\".
[####################] 100% -- SUCCESS
Verifying image type.
[####################] 100% -- SUCCESS
Extracting \"system\" version from image bootflash:/n5000-uk9.7.1.3.N1.2a.bin.
[####################] 100% -- SUCCESS
Extracting \"kickstart\" version from image bootflash:/n5000-uk9-kickstart.7.1.3.N1.2a.bin.
[####################] 100% -- SUCCESS
Extracting \"bios\" version from image bootflash:/n5000-uk9.7.1.3.N1.2a.bin.
[####################] 100% -- SUCCESS
Extracting \"fexth\" version from image bootflash:/n5000-uk9.7.1.3.N1.2a.bin.
[####################] 100% -- SUCCESS
Performing module support checks.
[####################] 100% -- SUCCESS
Notifying services about system upgrade.
[####################] 100% -- SUCCESS
Compatibility check is done:
Module bootable Impact Install-type Reason
------ -------- -------------- ------------ ------
1 yes disruptive reset Incompatible image
3 yes disruptive reset Incompatible image
100 yes disruptive reset Incompatible image
101 yes disruptive reset Incompatible image
102 yes disruptive reset Incompatible image
103 yes disruptive reset Incompatible image
104 yes disruptive reset Incompatible image
105 yes disruptive reset Incompatible image
106 yes disruptive reset Incompatible image
107 yes disruptive reset Incompatible image
108 yes disruptive reset Incompatible image
109 yes disruptive reset Incompatible image
111 yes disruptive reset Incompatible image
Images will be upgraded according to following table:
Module Image Running-Version New-Version Upg-Required
------ ---------------- ---------------------- ---------------------- ------------
1 system 6.0(2)N2(2) 7.1(3)N1(2a) yes
1 kickstart 6.0(2)N2(2) 7.1(3)N1(2a) yes
1 bios v3.6.0(05/09/2012) v3.6.0(05/09/2012) no
1 power-seq v2.0 v3.0 yes
1 SFP-uC v1.1.0.0 v1.0.0.0 no
3 power-seq v2.0 v2.0 no
100 fexth 6.0(2)N2(2) 7.1(3)N1(2a) yes
101 fexth 6.0(2)N2(2) 7.1(3)N1(2a) yes
102 fexth 6.0(2)N2(2) 7.1(3)N1(2a) yes
103 fexth 6.0(2)N2(2) 7.1(3)N1(2a) yes
104 fexth 6.0(2)N2(2) 7.1(3)N1(2a) yes
105 fexth 6.0(2)N2(2) 7.1(3)N1(2a) yes
106 fexth 6.0(2)N2(2) 7.1(3)N1(2a) yes
107 fexth 6.0(2)N2(2) 7.1(3)N1(2a) yes
108 fexth 6.0(2)N2(2) 7.1(3)N1(2a) yes
109 fexth 6.0(2)N2(2) 7.1(3)N1(2a) yes
111 fexth 6.0(2)N2(2) 7.1(3)N1(2a) yes
1 microcontroller v1.2.0.1 v1.2.0.1 no
Switch will be reloaded for disruptive upgrade.
Do you want to continue with the installation (y/n)? [n] y
Install is in progress, please wait.
Date: 2016-05-14 06:06:07 UTC
%SYSMGR-2-SERVICE_CRASHED: Service \"eth_port_sec\" (PID 4853) hasn't caught signal 6 (core will be saved).
%SYSMGR-2-HAP_FAILURE_SUP_RESET: System reset due to service \"eth_port_sec\" in vdc 1 has had a hap failure
Date: 2016-05-14 06:05:15 UTC
Nous continuons d'avoir des problèmes sur ces switchs. Nous upgradons en dernière version.
Date: 2016-05-14 03:09:09 UTC
Il s'agit du même problème hap_reset.
Reason: Reset triggered due to HA policy of Reset
Service: eth_port_sec hap reset
Si le problème se reproduit de nouveau dans les heures qui suivent, nous ferons la mise à jour à chaud.
Date: 2016-05-14 03:00:14 UTC
Nous avons rencontré de nouveau le problème sur le couple. Nous investiguons pour juger de l'urgence de l'upgrade ou non.
This incident affected: Infrastructure || GRA (GRA1, GRA2, GRA3).