OVHcloud Network Status

Current status
Legend
  • Operational
  • Degraded performance
  • Partial Outage
  • Major Outage
  • Under maintenance
gra1-102-n56-vrack
Scheduled Maintenance Report for Network & Infrastructure
Completed
Ce switch connait actuellement une forte croissance et, avec la version actuellement en place, nous ne pouvons voir les informations sur le trafic qui passe sur les liens.
Nous allons mettre a jour a chaud cet équipement afin de résoudre ce dysfonctionnement.

Périmètre de maintenance : Vrack de GRA
Date d'intervention : 08/02 a partir de 23h
Impact theorique : Null


Update(s):

Date: 2016-02-09 00:55:00 UTC
Tout est up !

On va envoyer les show tech / traces a cisco pour analyse

Date: 2016-02-09 00:40:58 UTC
shut/no shut sur les ports physique a fixe ( mais pas le shut/no shut du port-channel)
reload en cours du fex161

le reste des fexs est UP avec la bonne version du pHY

Date: 2016-02-09 00:34:25 UTC
nous continuons a remonter les fexs, tout se passe bien, sauf le pour le fex161
gra1-102-n56-vrack# sh int status | i fex161
Eth1/27 fex161 Dot1qMisC 1 full 10G 10Gbase-SR
Eth1/28 fex161 Dot1qMisC 1 full 10G 10Gbase-SR
Po161 fex161 noOperMem 1 auto auto --

nous avançons sur les fexs suivant et reviendrons sur lui a la fin

Date: 2016-02-09 00:33:35 UTC
gra1-102-n56-vrack# sh fex
FEX FEX FEX FEX Fex
Number Description State Model Serial
------------------------------------------------------------------------
150 fex150 Online N2K-C2348TQ-10GE FOC1930R15E
151 fex151 Online N2K-C2232TM-E-10GE SSI190300DF
152 fex152 Online N2K-C2348TQ-10GE FOC1930R516
153 fex153 Online N2K-C2348TQ-10GE FOC1930R0BU
154 fex154 Online N2K-C2348TQ-10GE FOC1930R1C4
155 fex155 Online N2K-C2232TM-E-10GE SSI190300KA


On passe au 5 suivants



Date: 2016-02-09 00:15:39 UTC
Les fex2348 n'ont pas reloades, on se retrouve avec la vieille version du firmware..

ici le fex150 qui a deja reload:
gra1-102-n56-vrack# attach fex 150
Attaching to FEX 150 ...
To exit type 'exit', to abort type '$.'
fex-150# dbgexec tib show port hi1
[snip]
ushort ucode_ver: 108h
[snip]

Les autres sont en 107h

Nous rebootons les fex en questions

Date: 2016-02-09 00:13:20 UTC
On up les fexs tranquillement, si on up tout les fex d'un coup, les buffers de comm interne entre le nexus et les fex explosent, les commandes collent et eth port-manager n'arrive pas a pusher la conf sur tt les interfaces

gra1-102-n56-vrack# sh fex
FEX FEX FEX FEX Fex
Number Description State Model Serial
------------------------------------------------------------------------
150 fex150 Connected N2K-C2348TQ-10GE FOC1930R15E
151 fex151 Connected N2K-C2232TM-E-10GE SSI190300DF
152 fex152 Online N2K-C2348TQ-10GE FOC1930R516
153 fex153 Online N2K-C2348TQ-10GE FOC1930R0BU
154 fex154 Connected N2K-C2348TQ-10GE FOC1930R1C4
155 fex155 Connected N2K-C2232TM-E-10GE SSI190300KA


Date: 2016-02-08 23:57:30 UTC
Traces prisent, nous avons couper les port-channels vers les fex, maintenant nous reloadons


Date: 2016-02-08 23:30:52 UTC
ça ne fix pas

nous prenons des traces avant de reloader.

Date: 2016-02-08 23:20:55 UTC
Le fex150 n'est pas revenu et a embarqué avec lui 4 autre fexs
gra1-102-n56-vrack# sh fex
FEX FEX FEX FEX Fex
Number Description State Model Serial
------------------------------------------------------------------------
151 fex151 Online N2K-C2232TM-E-10GE SSI190300DF
152 fex152 Offline N2K-C2348TQ-10GE FOC1930R516
153 fex153 Online N2K-C2348TQ-10GE FOC1930R0BU
154 fex154 Online N2K-C2348TQ-10GE FOC1930R1C4
155 fex155 Online N2K-C2232TM-E-10GE SSI190300KA
156 fex156 Online N2K-C2348TQ-10GE FOC1930R1J5
157 fex157 Online N2K-C2348TQ-10GE FOC1930R11Q
158 fex158 Online N2K-C2348TQ-10GE FOC1930R41J
159 fex159 Online N2K-C2348TQ-10GE FOC1930R40A
160 fex160 Online N2K-C2348TQ-10GE FOC1919R0Y0
161 fex161 Online N2K-C2348TQ-10GE FOC1930R12K
162 fex162 Offline N2K-C2348TQ-10GE FOC1930R41H
163 fex163 Offline N2K-C2348TQ-10GE FOC1930R1KN
164 fex164 Offline N2K-C2248TP-E-1GE FOX1930GE8W
165 fex165 Online N2K-C2348TQ-10GE FOC1930R11A
166 fex166 Online N2K-C2348TQ-10GE FOC1930R10K
167 fex167 Online N2K-C2232TM-E-10GE SSI1918021E
168 fex168 Online N2K-C2248TP-E-1GE FOX1913G0JD
169 fex169 Online N2K-C2348TQ-10GE FOC1939R0ZV


On test un reload hard sur le 150

Il n'y a de l'impact que sur le fex 150,152 et 162 a 164

Si ca ne fixe pas, nous devons reloader le nexus en entier.

Date: 2016-02-08 23:04:37 UTC
Le reload se fait fex par fex.

gra1-102-n56-vrack# sh fex | i 2348
150 fex150 Online N2K-C2348TQ-10GE FOC1930R15E
152 fex152 Online N2K-C2348TQ-10GE FOC1930R516
153 fex153 Online N2K-C2348TQ-10GE FOC1930R0BU
154 fex154 Online N2K-C2348TQ-10GE FOC1930R1C4
156 fex156 Online N2K-C2348TQ-10GE FOC1930R1J5
157 fex157 Online N2K-C2348TQ-10GE FOC1930R11Q
158 fex158 Online N2K-C2348TQ-10GE FOC1930R41J
159 fex159 Online N2K-C2348TQ-10GE FOC1930R40A
160 fex160 Online N2K-C2348TQ-10GE FOC1919R0Y0
161 fex161 Online N2K-C2348TQ-10GE FOC1930R12K
162 fex162 Online N2K-C2348TQ-10GE FOC1930R41H
163 fex163 Online N2K-C2348TQ-10GE FOC1930R1KN
165 fex165 Online N2K-C2348TQ-10GE FOC1930R11A
166 fex166 Online N2K-C2348TQ-10GE FOC1930R10K
169 fex169 Online N2K-C2348TQ-10GE FOC1939R0ZV


nous commençons avec le fex150

Date: 2016-02-08 23:03:42 UTC
Nous allons passer a l'autre partie de la maintenance, le reload des fex 2348 pour mettre a jour le PHY ( la partie bas niveau en charge des interfaces ).
La mise a jour du PHY est forcement disruptive et ne peut s'updater qu'avec un reload

Date: 2016-02-08 23:01:48 UTC
l'install est okay en non-disruptif
Module 168: Non-disruptive upgrading.
SUCCESS

Module 169: Non-disruptive upgrading.
SUCCESS

Install has been successful.

Date: 2016-02-08 22:50:44 UTC
Module 160: Non-disruptive upgrading.
[####################] 100% -- SUCCESS

Module 161: Non-disruptive upgrading.
[####################] 100% -- SUCCESS

Module 162: Non-disruptive upgrading.
[####################] 100% -- SUCCESS

Module 163: Non-disruptive upgrading.
[####################] 100% -- SUCCESS

Module 164: Non-disruptive upgrading.
[# ] 0%


Date: 2016-02-08 22:32:01 UTC
l'upgrade des fexs commence

Pre-loading modules.
[This step might take upto 20 minutes to complete - please wait.]
[*Warning -- Please do not abort installation/reload or powercycle fexes*]
[####################] 100% -- SUCCESS

Module 150: Non-disruptive upgrading.
[# ] 0%


Date: 2016-02-08 22:25:19 UTC
Le supervisor a reload ( hitless ISSU, pas de panic :) )

il est maintenant en 7.1.3.n1.2

ra1-102-n56-vrack# sh version
Cisco Nexus Operating System (NX-OS) Software
TAC support: http://www.cisco.com/tac
Documents: http://www.cisco.com/en/US/products/ps9372/tsd_products_support_series_home.html
Copyright (c) 2002-2016, Cisco Systems, Inc. All rights reserved.
The copyrights to certain works contained herein are owned by
other third parties and are used and distributed under license.
Some parts of this software are covered under the GNU Public
License. A copy of the license is available at
http://www.gnu.org/licenses/gpl.html.

Software
BIOS: version 2.1.2
Power Sequencer Firmware:
Module 1: v4.0
Module 1: v4.0
Fabric Power Sequencer Firmware: Module 1: version v4.0
Microcontroller Firmware: version v0.0.0.15
QSFP Microcontroller Firmware:
Module 1: v2.0.0.0
CXP Microcontroller Firmware:
Module not detected
kickstart: version 7.1(3)N1(2) <<<<<<<<<<<<<<<<<<<<<
system: version 7.1(3)N1(2) <<<<<<<<<<<<<<<<<<<<<
BIOS compile time: 07/16/2014
kickstart image file is: bootflash:///n6000-uk9-kickstart.7.1.3.N1.2.bin
kickstart compile time: 1/21/2016 23:00:00 [01/22/2016 10:48:28]
system image file is: bootflash:///n6000-uk9.7.1.3.N1.2.bin
system compile time: 1/21/2016 23:00:00 [01/22/2016 10:49:13]


Il pre-load les images sur les fexs:

Supervisor non-disruptive upgrade successful.

Pre-loading modules.
[This step might take upto 20 minutes to complete - please wait.]
[*Warning -- Please do not abort installation/reload or powercycle fexes*]
[### ] 10%



Date: 2016-02-08 22:15:22 UTC
Compatibility check is done:
Module bootable Impact Install-type Reason
------ -------- -------------- ------------ ------
1 yes non-disruptive reset
2 yes non-disruptive rolling
150 yes non-disruptive rolling
151 yes non-disruptive rolling
152 yes non-disruptive rolling
153 yes non-disruptive rolling
154 yes non-disruptive rolling
155 yes non-disruptive rolling
156 yes non-disruptive rolling
157 yes non-disruptive rolling
158 yes non-disruptive rolling
159 yes non-disruptive rolling
160 yes non-disruptive rolling
161 yes non-disruptive rolling
162 yes non-disruptive rolling
163 yes non-disruptive rolling
164 yes non-disruptive rolling
165 yes non-disruptive rolling
166 yes non-disruptive rolling
167 yes non-disruptive rolling
168 yes non-disruptive rolling
169 yes non-disruptive rolling



Images will be upgraded according to following table:
Module Image Running-Version New-Version Upg-Required
------ ---------------- ---------------------- ---------------------- ------------
1 system 7.1(2)N1(1) 7.1(3)N1(2) yes
1 kickstart 7.1(2)N1(1) 7.1(3)N1(2) yes
1 bios v2.1.0(02/24/2014) v2.1.2(07/16/2014) yes
1 power-seq v4.0 v4.0 no
1 fabric-power-seq v3.0 v4.0 yes
2 power-seq v4.0 v4.0 no
150 fex4 7.1(2)N1(1) 7.1(3)N1(2) yes
151 fexth 7.1(2)N1(1) 7.1(3)N1(2) yes
152 fex4 7.1(2)N1(1) 7.1(3)N1(2) yes
153 fex4 7.1(2)N1(1) 7.1(3)N1(2) yes
154 fex4 7.1(2)N1(1) 7.1(3)N1(2) yes
155 fexth 7.1(2)N1(1) 7.1(3)N1(2) yes
156 fex4 7.1(2)N1(1) 7.1(3)N1(2) yes
157 fex4 7.1(2)N1(1) 7.1(3)N1(2) yes
158 fex4 7.1(2)N1(1) 7.1(3)N1(2) yes
159 fex4 7.1(2)N1(1) 7.1(3)N1(2) yes
160 fex4 7.1(2)N1(1) 7.1(3)N1(2) yes
161 fex4 7.1(2)N1(1) 7.1(3)N1(2) yes
162 fex4 7.1(2)N1(1) 7.1(3)N1(2) yes
163 fex4 7.1(2)N1(1) 7.1(3)N1(2) yes
164 fexth 7.1(2)N1(1) 7.1(3)N1(2) yes
165 fex4 7.1(2)N1(1) 7.1(3)N1(2) yes
166 fex4 7.1(2)N1(1) 7.1(3)N1(2) yes
167 fexth 7.1(2)N1(1) 7.1(3)N1(2) yes
168 fexth 7.1(2)N1(1) 7.1(3)N1(2) yes
169 fex4 7.1(2)N1(1) 7.1(3)N1(2) yes
1 microcontroller v0.0.0.15 v0.0.0.15 no


Do you want to continue with the installation (y/n)? [n] y

Install is in progress, please wait.

Performing runtime checks.
[####################] 100% -- SUCCESS

Notifying services about the upgrade.
[####################] 100% -- SUCCESS


Date: 2016-02-08 22:05:48 UTC
nous débutons l'upgrade

gra1-102-n56-vrack# install all kickstart n6000-uk9-kickstart.7.1.3.N1.2.bin system n6000-uk9.7.1.3.N1.2.bin

Verifying image bootflash:/n6000-uk9-kickstart.7.1.3.N1.2.bin for boot variable \"kickstart\".
[####################] 100% -- SUCCESS

Verifying image bootflash:/n6000-uk9.7.1.3.N1.2.bin for boot variable \"system\".
[####################] 100% -- SUCCESS

Verifying image type.
[########### ] 50%
Posted Feb 08, 2016 - 15:06 UTC
This scheduled maintenance affected: Infrastructure || GRA (GRA1).