OVHcloud Network Status

Current status
Legend
  • Operational
  • Degraded performance
  • Partial Outage
  • Major Outage
  • Under maintenance
bhs4-17a/b-n56
Scheduled Maintenance Report for Network & Infrastructure
Completed
Nous allons mettre a jour ce couple de nexus en release 7.1.3.N1.2
L'intervention est planifiée le 4 février 2016 a partir de 8h00 am CET ( 2h00 am EST )

Le but est principalement du bug fix:
- Améliore l'interop entre le fex2348 et les chips réseau intel 10gBaseT x552/x557
- Fix des compteur d'interface sur le 2348
- Fix divers

La mise a jour se passera en 2 phases:
- La mise a jour du NXOS sur les nexus et les fex
- La mise a jour du PHY broadcom dans les fex2348 (Firmware bas niveau qui gère les eth)

Un sh install all impact montre que se sera non-disruptive en ISSU, la première phase sera donc hitless

bhs4-17a-n56# sh install all impact kickstart n6000-uk9-kickstart.7.1.3.N1.2.bin system n6000-uk9.7.1.3.N1.2.bin

Verifying image bootflash:/n6000-uk9-kickstart.7.1.3.N1.2.bin for boot variable \"kickstart\".
[####################] 100% -- SUCCESS

Verifying image bootflash:/n6000-uk9.7.1.3.N1.2.bin for boot variable \"system\".
[####################] 100% -- SUCCESS

Verifying image type.
[####################] 100% -- SUCCESS

Extracting \"system\" version from image bootflash:/n6000-uk9.7.1.3.N1.2.bin.
[####################] 100% -- SUCCESS

Extracting \"kickstart\" version from image bootflash:/n6000-uk9-kickstart.7.1.3.N1.2.bin.
[####################] 100% -- SUCCESS

Extracting \"bios\" version from image bootflash:/n6000-uk9.7.1.3.N1.2.bin.
[####################] 100% -- SUCCESS

Extracting \"fex4\" version from image bootflash:/n6000-uk9.7.1.3.N1.2.bin.
[####################] 100% -- SUCCESS

Performing module support checks.
[####################] 100% -- SUCCESS

Notifying services about system upgrade.
[####################] 100% -- SUCCESS



Compatibility check is done:
Module bootable Impact Install-type Reason
------ -------- -------------- ------------ ------
0 yes non-disruptive reset
2 yes non-disruptive rolling
100 yes non-disruptive rolling
101 yes non-disruptive rolling
102 yes non-disruptive rolling
103 yes non-disruptive rolling
104 yes non-disruptive rolling
105 yes non-disruptive rolling
106 yes non-disruptive rolling
107 yes non-disruptive rolling
108 yes non-disruptive rolling
109 yes non-disruptive rolling
110 yes non-disruptive rolling
112 yes non-disruptive rolling
113 yes non-disruptive rolling
114 yes non-disruptive rolling
115 yes non-disruptive rolling
116 yes non-disruptive rolling
117 yes non-disruptive rolling
118 yes non-disruptive rolling
119 yes non-disruptive rolling



Images will be upgraded according to following table:
Module Image Running-Version New-Version Upg-Required
------ ---------------- ---------------------- ---------------------- ------------
0 system 7.1(2)N1(1) 7.1(3)N1(2) yes
0 kickstart 7.1(2)N1(1) 7.1(3)N1(2) yes
0 bios v1.0.8(10/29/2014) v1.0.8(10/29/2014) no
0 power-seq SF-uC:39, SF-FPGA:6 SF-uC:37, SF-FPGA:5 no
0 iofpga v0.0.0.34 v0.0.0.34 no
2 power-seq SF-uC:23, SF-FPGA:7 SF-uC:21, SF-FPGA:6 no
2 iofpga v0.0.0.14 v0.0.0.14 no
100 fex4 7.1(2)N1(1) 7.1(3)N1(2) yes
101 fex4 7.1(2)N1(1) 7.1(3)N1(2) yes
102 fex4 7.1(2)N1(1) 7.1(3)N1(2) yes
103 fex4 7.1(2)N1(1) 7.1(3)N1(2) yes
104 fex4 7.1(2)N1(1) 7.1(3)N1(2) yes
105 fex4 7.1(2)N1(1) 7.1(3)N1(2) yes
106 fex4 7.1(2)N1(1) 7.1(3)N1(2) yes
107 fex4 7.1(2)N1(1) 7.1(3)N1(2) yes
108 fex4 7.1(2)N1(1) 7.1(3)N1(2) yes
109 fex4 7.1(2)N1(1) 7.1(3)N1(2) yes
110 fex4 7.1(2)N1(1) 7.1(3)N1(2) yes
112 fex4 7.1(2)N1(1) 7.1(3)N1(2) yes
113 fex4 7.1(2)N1(1) 7.1(3)N1(2) yes
114 fex4 7.1(2)N1(1) 7.1(3)N1(2) yes
115 fex4 7.1(2)N1(1) 7.1(3)N1(2) yes
116 fex4 7.1(2)N1(1) 7.1(3)N1(2) yes
117 fex4 7.1(2)N1(1) 7.1(3)N1(2) yes
118 fex4 7.1(2)N1(1) 7.1(3)N1(2) yes
119 fex4 7.1(2)N1(1) 7.1(3)N1(2) yes



Cependant, afin que le fex puisse mettre a jour le PHY, il faut obligatoirement un reload du fex (c'est un composant bas niveau qui ne peut pas être maj a chaud).
Nous ferons le reload fex par fex après la maj du NXOS, cela implique un downtime de 2-3min par fex.





Update(s):

Date: 2016-02-04 09:26:42 UTC
Le monitoring ne remonte plus de serveur non-joignable.

Nous sommes en webex avec Cisco pour analyser le crash.

Date: 2016-02-04 09:18:10 UTC
Aussi:
les fexs, en perdant les parents, ont reboote, on est maintenant dans la bonne Version du PHY sur les fex2348



Date: 2016-02-04 09:16:24 UTC
Les fex sont tous UP, les serveurs reviennent, nous faisons le tour du monitoring sur les machines restantes

bhs4-17b-n56# sh fex
FEX FEX FEX FEX Fex
Number Description State Model Serial
------------------------------------------------------------------------
100 fex100 Online N2K-C2348TQ-10GE FOC1930R12Z
101 fex101 Online N2K-C2348TQ-10GE FOC1930R1K8
102 fex102 Online N2K-C2348TQ-10GE FOC1930R1KG
103 fex103 Online N2K-C2348TQ-10GE FOC1930R1KT
104 fex104 Online N2K-C2348TQ-10GE FOC1930R42V
105 fex105 Online N2K-C2348TQ-10GE FOC1930R1BW
106 fex106 Online N2K-C2348TQ-10GE FOC1930R40U
107 fex107 Online N2K-C2348TQ-10GE FOC1930R40R
108 fex108 Online N2K-C2348TQ-10GE FOC1930R163
109 fex109 Online N2K-C2348TQ-10GE FOC1930R3ZQ
110 fex110 Online N2K-C2348TQ-10GE FOC1930R41W
112 fex112 Online N2K-C2348TQ-10GE FOC1930R10Y
113 fex113 Online N2K-C2348TQ-10GE FOC1930R42U
114 fex114 Online N2K-C2348TQ-10GE FOC1930R432
115 fex115 Online N2K-C2348TQ-10GE FOC1930R41X
116 fex116 Online N2K-C2348TQ-10GE FOC1930R1B2
117 fex117 Online N2K-C2348TQ-10GE FOC1944R0CU
118 fex118 Online N2K-C2348TQ-10GE FOC1944R06D
119 fex119 Online N2K-C2348TQ-10GE FOC1944R06R


Date: 2016-02-04 09:06:17 UTC
la vpc est up, les fex reviennent

bhs4-17a-n56# sh fex
FEX FEX FEX FEX Fex
Number Description State Model Serial
------------------------------------------------------------------------
100 fex100 Connected N2K-C2348TQ-10GE FOC1930R12Z
101 fex101 Connected N2K-C2348TQ-10GE FOC1930R1K8
102 fex102 Online N2K-C2348TQ-10GE FOC1930R1KG
103 fex103 Connected N2K-C2348TQ-10GE FOC1930R1KT
104 fex104 Connected N2K-C2348TQ-10GE FOC1930R42V
105 fex105 Connected N2K-C2348TQ-10GE FOC1930R1BW
106 fex106 Connected N2K-C2348TQ-10GE FOC1930R40U
107 fex107 Online N2K-C2348TQ-10GE FOC1930R40R
108 fex108 Connected N2K-C2348TQ-10GE FOC1930R163
109 fex109 Connected N2K-C2348TQ-10GE FOC1930R3ZQ
110 fex110 Connected N2K-C2348TQ-10GE FOC1930R41W
112 fex112 Connected N2K-C2348TQ-10GE FOC1930R10Y
114 fex114 Connected N2K-C2348TQ-10GE FOC1930R432
116 fex116 Online Sequence N2K-C2348TQ-10GE FOC1930R1B2
117 fex117 Connected N2K-C2348TQ-10GE FOC1944R0CU
118 fex118 Connected N2K-C2348TQ-10GE FOC1944R06D
119 fex119 Online N2K-C2348TQ-10GE FOC1944R06R
--- -------- Connected N2K-C2348TQ-10GE FOC1930R42U
--- -------- Connected N2K-C2348TQ-10GE FOC1930R41X


Date: 2016-02-04 08:55:11 UTC
le A vient de crasher !



bhs4-17a-n56# sh fex

Broadcast message from root (console) (Thu Feb 4 09:54:50 2016):

The system is going down for reboot NOW!



Date: 2016-02-04 08:54:26 UTC
l'install sur le B a crashe

on a perdu tt les fex

bhs4-17a-n56# sh fex
FEX FEX FEX FEX Fex
Number Description State Model Serial
------------------------------------------------------------------------
100 fex100 Offline Sequence N2K-C2348TQ-10GE FOC1930R12Z
101 fex101 Offline Sequence N2K-C2348TQ-10GE FOC1930R1K8
102 fex102 Offline Sequence N2K-C2348TQ-10GE FOC1930R1KG
103 fex103 Offline Sequence N2K-C2348TQ-10GE FOC1930R1KT
104 fex104 Offline Sequence N2K-C2348TQ-10GE FOC1930R42V
105 fex105 Offline Sequence N2K-C2348TQ-10GE FOC1930R1BW
106 fex106 Offline Sequence N2K-C2348TQ-10GE FOC1930R40U
107 fex107 Offline Sequence N2K-C2348TQ-10GE FOC1930R40R
108 fex108 Offline Sequence N2K-C2348TQ-10GE FOC1930R163
109 fex109 Offline Sequence N2K-C2348TQ-10GE FOC1930R3ZQ
110 fex110 Offline Sequence N2K-C2348TQ-10GE FOC1930R41W
112 fex112 Offline Sequence N2K-C2348TQ-10GE FOC1930R10Y
113 fex113 Offline Sequence N2K-C2348TQ-10GE FOC1930R42U
114 fex114 Offline Sequence N2K-C2348TQ-10GE FOC1930R432
115 fex115 Offline Sequence N2K-C2348TQ-10GE FOC1930R41X
116 fex116 Offline Sequence N2K-C2348TQ-10GE FOC1930R1B2
117 fex117 Offline Sequence N2K-C2348TQ-10GE FOC1944R0CU
118 fex118 Offline Sequence N2K-C2348TQ-10GE FOC1944R06D
119 fex119 Offline Sequence N2K-C2348TQ-10GE FOC1944R06R
bhs4-17a-n56#
bhs4-17a-n56#


Date: 2016-02-04 08:50:06 UTC
Do you want to continue with the installation (y/n)? [n] y

Install is in progress, please wait.


Date: 2016-02-04 08:45:52 UTC
le 17a est a jour

016 Feb 4 09:40:42.022 bhs4-17b-n56 %NOHMS-2-NOHMS_ENV_FEX_ONLINE: FEX-119 On-line
2016 Feb 4 09:40:43.738 bhs4-17b-n56 %PFMA-2-FEX_STATUS: Fex 119 is online
2016 Feb 4 09:41:29.415 bhs4-17b-n56 %VPC-2-VPC_ISSU_END: Peer vPC switch ISSU end, unlocking configuration


Nous débutons l'install sur le 17b


bhs4-17b-n56# install all system n6000-uk9.7.1.3.N1.2.bin kickstart n6000-uk9-kickstart.7.1.3.N1.2.bin

Verifying image bootflash:/n6000-uk9-kickstart.7.1.3.N1.2.bin for boot variable \"kickstart\".
[####################] 100% -- SUCCESS

Verifying image bootflash:/n6000-uk9.7.1.3.N1.2.bin for boot variable \"system\".
[####################] 100% -- SUCCESS

Verifying image type.
[########### ] 50%


Date: 2016-02-04 08:30:43 UTC
Module 108: Non-disruptive upgrading.
[####################] 100% -- SUCCESS

Module 109: Non-disruptive upgrading.
[####################] 100% -- SUCCESS

Module 110: Non-disruptive upgrading.
[# ] 0%


Date: 2016-02-04 08:16:31 UTC
Supervisor non-disruptive upgrade successful.

Pre-loading modules.
SUCCESS

Module 100: Non-disruptive upgrading.



Date: 2016-02-04 08:10:43 UTC
Continuing with installation process, please wait.
The login will be disabled until the installation is completed.

Performing supervisor state verification.
[####################] 100% -- SUCCESS

Supervisor non-disruptive upgrade successful.

Pre-loading modules.
[This step might take upto 20 minutes to complete - please wait.]
[*Warning -- Please do not abort installation/reload or powercycle fexes*]
[### ] 10%


Date: 2016-02-04 08:03:21 UTC
Do you want to continue with the installation (y/n)? [n] y

Install is in progress, please wait.




Date: 2016-02-04 07:58:58 UTC
here we go

bhs4-17a-n56# install all kickstart n6000-uk9-kickstart.7.1.3.N1.2.bin system n6000-uk9.7.1.3.N1.2.bin

Verifying image bootflash:/n6000-uk9-kickstart.7.1.3.N1.2.bin for boot variable \"kickstart\".
[####################] 100% -- SUCCESS

Verifying image bootflash:/n6000-uk9.7.1.3.N1.2.bin for boot variable \"system\".
[####################] 100% -- SUCCESS

Verifying image type.
[########### ] 50%


Date: 2016-02-04 07:20:14 UTC
Nous allons débuter l'intervention d'ici 15min~
Posted Feb 03, 2016 - 11:30 UTC
This scheduled maintenance affected: Infrastructure || BHS (BHS4).