rssLink RSS for all categories
 
icon_red
icon_green
icon_green
icon_red
icon_red
icon_green
icon_green
icon_red
icon_red
icon_blue
icon_green
icon_green
icon_green
icon_orange
icon_green
icon_red
icon_green
icon_blue
icon_red
icon_red
icon_green
icon_orange
icon_green
icon_red
icon_orange
icon_green
icon_green
icon_green
icon_green
icon_green
icon_blue
icon_green
icon_red
 

FS#11635 — pcc-22-n5

Attached to Project— Dedicated Cloud
Incident
Rbx2a
CLOSED
100%
Ce n5 ne répond plus, nous investiguons.
le service est maintenu par le pcc-23-pcc.
Date:  Saturday, 20 September 2014, 15:21PM
Reason for closing:  Done
Comment by OVH - Saturday, 20 September 2014, 03:37AM

le N5 est UP, nous ferons le resync de la conf au matin.


Comment by OVH - Saturday, 20 September 2014, 07:27AM

le N5 est retombé, la gestion des fan est defailante, les 2 bloc fans sont neuf et pourtant sont tombé au même moment (d'apres les log)
on ne prend pas de risque, on remplace le N5 par un nouveau

[local7.crit] === : 2014 Sep 20 06:51:32 CEST: %NOHMS-2-NOHMS_ENV_ERR_FAN_SPEED: System minor alarm in fan tray 2: fan speed is out of range on fan 2. 9450 to 12600 rpm expected. 8794 rpm read
[local7.crit] === : 2014 Sep 20 06:51:32 CEST: %NOHMS-2-NOHMS_ENV_ERR_FANS_DOWN: System major alarm on fans: Multiple fans missing or failed
[local7.emerg] === : 2014 Sep 20 06:51:32 CEST: %PFMA-0-SYS_SHUTDOWN_FAN_REMOVAL: System shutdown in 60 seconds due to fan removal
[local7.crit] === : 2014 Sep 20 06:51:34 CEST: %NOHMS-2-NOHMS_ENV_ERR_FAN_RECOVERED: Recovered: System minor alarm in fan tray 1
[local7.crit] === : 2014 Sep 20 06:51:34 CEST: %NOHMS-2-NOHMS_ENV_ERR_FANS_UP: Recovered: System major alarm on fans: Multiple fans missing or failed
[local7.crit] === : 2014 Sep 20 06:51:35 CEST: %NOHMS-2-NOHMS_ENV_ERR_FAN_SPEED: System minor alarm in fan tray 1: fan speed is out of range on fan 3. 12158 to 16211 rpm expected. 6087 rpm read
[local7.crit] === : 2014 Sep 20 06:51:35 CEST: %NOHMS-2-NOHMS_ENV_ERR_FAN_SPEED: System minor alarm in fan tray 1: fan speed is out of range on fan 4. 9450 to 12600 rpm expected. 6601 rpm read
[local7.crit] === : 2014 Sep 20 06:51:35 CEST: %NOHMS-2-NOHMS_ENV_ERR_FANS_DOWN: System major alarm on fans: Multiple fans missing or failed
[local7.emerg] === : 2014 Sep 20 06:51:35 CEST: %PFMA-0-SYS_SHUTDOWN_FAN_REMOVAL: System shutdown in 60 seconds due to fan removal
[local7.emerg] === : 2014 Sep 20 06:51:40 CEST: %PFMA-0-SYS_SHUTDOWN_FAN_REMOVAL: System shutdown in 55 seconds due to fan removal
[local7.emerg] === : 2014 Sep 20 06:51:45 CEST: %PFMA-0-SYS_SHUTDOWN_FAN_REMOVAL: System shutdown in 50 seconds due to fan removal
[local7.emerg] === : 2014 Sep 20 06:51:50 CEST: %PFMA-0-SYS_SHUTDOWN_FAN_REMOVAL: System shutdown in 45 seconds due to fan removal
[local7.emerg] === : 2014 Sep 20 06:51:55 CEST: %PFMA-0-SYS_SHUTDOWN_FAN_REMOVAL: System shutdown in 40 seconds due to fan removal
[local7.emerg] === : 2014 Sep 20 06:52:00 CEST: %PFMA-0-SYS_SHUTDOWN_FAN_REMOVAL: System shutdown in 35 seconds due to fan removal
[local7.emerg] === : 2014 Sep 20 06:52:05 CEST: %PFMA-0-SYS_SHUTDOWN_FAN_REMOVAL: System shutdown in 30 seconds due to fan removal
[local7.emerg] === : 2014 Sep 20 06:52:10 CEST: %PFMA-0-SYS_SHUTDOWN_FAN_REMOVAL: System shutdown in 25 seconds due to fan removal
[local7.emerg] === : 2014 Sep 20 06:52:15 CEST: %PFMA-0-SYS_SHUTDOWN_FAN_REMOVAL: System shutdown in 20 seconds due to fan removal
[local7.emerg] === : 2014 Sep 20 06:52:20 CEST: %PFMA-0-SYS_SHUTDOWN_FAN_REMOVAL: System shutdown in 15 seconds due to fan removal
[local7.emerg] === : 2014 Sep 20 06:52:25 CEST: %PFMA-0-SYS_SHUTDOWN_FAN_REMOVAL: System shutdown in 10 seconds due to fan removal
[local7.emerg] === : 2014 Sep 20 06:52:30 CEST: %PFMA-0-SYS_SHUTDOWN_FAN_REMOVAL: System shutdown in 5 seconds due to fan removal
[local7.emerg] === : 2014 Sep 20 06:52:35 CEST: %PFMA-0-SYS_SHUTDOWN_FAN_REMOVAL: System shutdown in 0 seconds due to fan removal
[local7.emerg] === : 2014 Sep 20 06:52:35 CEST: Sep 20 06:52:35 %KERN-0-SYSTEM_MSG: Shutdown Ports.. - kernel


Comment by OVH - Saturday, 20 September 2014, 09:08AM

le N5 est remplacé, nous le mettons a jour avec la dernière version du NXOS validé pour PCC.


Comment by OVH - Saturday, 20 September 2014, 11:12AM

N5 a jour,
PCC resync la conf.


Comment by OVH - Saturday, 20 September 2014, 14:40PM

conf resync, j'allume les FEX un par un


Comment by OVH - Saturday, 20 September 2014, 14:47PM

le fex se met à jour
pcc-22-n5# sh fex
FEX FEX FEX FEX Fex
Number Description State Model Serial
------------------------------------------------------------------------
100 fex100|27C02 Image Download N2K-C2232PP-10GE SSI14210EWH


Comment by OVH - Saturday, 20 September 2014, 15:05PM

cc-22-n5(config-if-range)# sh fex
FEX FEX FEX FEX Fex
Number Description State Model Serial
------------------------------------------------------------------------
100 fex100|27C02 Online N2K-C2232PP-10GE SSI14210EWH
101 fex101|27C02 Image Download N2K-C2232PP-10GE SSI14210EPS
102 fex102|27C04 Online Sequence N2K-C2232PP-10GE SSI142203E8
103 fex103|27C05 Offline N2K-C2232PP-10GE SSI14270R32
104 tests |28D17 Image Download N2K-C2248TP-1GE SSI1423007F
105 fex105|28B26 Image Download N2K-C2248TP-1GE SSI14270E7E
106 fex106|28B25 Image Download N2K-C2248TP-1GE SSI14270R4Y
107 fex107|28B27 Image Download N2K-C2248TP-1GE SSI14270CJS
108 fex108|28B24 Image Download N2K-C2248TP-1GE SSI14270R7X
109 fex109|28B23 Offline N2K-C2248TP-1GE SSI14150XK0
110 fex110|28B22 Image Download N2K-C2248TP-1GE SSI142800E3


Comment by OVH - Saturday, 20 September 2014, 15:11PM

pcc-22-n5# sh fex
FEX FEX FEX FEX Fex
Number Description State Model Serial
------------------------------------------------------------------------
100 fex100|27C02 Online N2K-C2232PP-10GE SSI14210EWH
101 fex101|27C02 Online N2K-C2232PP-10GE SSI14210EPS
102 fex102|27C04 Online N2K-C2232PP-10GE SSI142203E8
103 fex103|27C05 Online N2K-C2232PP-10GE SSI14270R32
104 tests |28D17 Online N2K-C2248TP-1GE SSI1423007F
105 fex105|28B26 Online N2K-C2248TP-1GE SSI14270E7E
106 fex106|28B25 Online N2K-C2248TP-1GE SSI14270R4Y
107 fex107|28B27 Online N2K-C2248TP-1GE SSI14270CJS
108 fex108|28B24 Online N2K-C2248TP-1GE SSI14270R7X
109 fex109|28B23 Online N2K-C2248TP-1GE SSI14150XK0
110 fex110|28B22 Online N2K-C2248TP-1GE SSI142800E3


Comment by OVH - Saturday, 20 September 2014, 15:15PM

tout est okay, la redondance est de nouveau en place.