OVHcloud Network Status

Current status
Legend
  • Operational
  • Degraded performance
  • Partial Outage
  • Major Outage
  • Under maintenance
upgrade: vss-2-6k
Incident Report for Network & Infrastructure
Resolved
Nous allons mettre à jour le routeur vss-2-6k. Après minuit et avant 7h du matin.
Soit cette nuit (de lundi à mardi) soit demain nuit (ça va dependre si on est prêt).

Update(s):

Date: 2009-12-10 04:55:13 UTC

fini ...

heuressement que les routeurs n'ont pas d'oreille ... et ne comprennent pas
les emotions ... yaa ! saloperie !



Date: 2009-12-10 04:49:26 UTC
up. en SXI3.

comment generer 1 heure de panne pour essayer d'eviter (826-164) secondes de coupure net et previsible ...

64 bytes from 94.23.60.10: icmp_seq=161 ttl=60 time=4.12 ms
64 bytes from 94.23.60.10: icmp_seq=162 ttl=60 time=4.08 ms
64 bytes from 94.23.60.10: icmp_seq=164 ttl=60 time=4.04 ms

64 bytes from 94.23.60.10: icmp_seq=826 ttl=58 time=4.09 ms
64 bytes from 94.23.60.10: icmp_seq=827 ttl=58 time=4.06 ms
64 bytes from 94.23.60.10: icmp_seq=828 ttl=58 time=4.06 ms


Date: 2009-12-10 04:36:29 UTC
go !

Date: 2009-12-10 04:30:40 UTC
tout est syncro. c'est reparti pour un tour ...

Date: 2009-12-10 04:16:26 UTC
le 1er chassis est à nouveau up. on attend le 2ème.

en suite on va lui coller le SXI3 en dur puis reboot hard.

Date: 2009-12-10 03:41:39 UTC
*Dec 10 03:35:03.499: %ISSU_PROCESS-SW1_SP-3-IMAGE: Active is loading the wrong image [ disk0:s72033-advipservicesk9-mz.122-33.SXI3.bin ], expected image [ disk0:s72033-advipservicesk9-mz.122-33.SXI1.bin ]
*Dec 10 03:35:04.255: %RF-SW1_SP-5-RF_RELOAD: Shelf reload. Reason: Active is loading the wrong image
*Dec 10 03:35:04.255: %OIR-SW1_SP-6-CONSOLE: Changing console ownership to switch processor

saloperie !

Date: 2009-12-10 03:33:39 UTC
bon ça ne se passe pas comme prevu. on va devoir faire une mise à jour
avec un reboot hard et donc une coupure de ... 5-15 minutes, le temps
que tout revienne en place et le routeur reapprenne toutes les MAC.

c'est beau la technologie ...

allez ... prêt pour le reveil de 3000 personnes ?



Date: 2009-12-10 03:29:27 UTC
$ echo \"10000*6/60/60\" | bc -l
16.66666666666666666666

16 heures d'initiation ...

à 4h45 on fait un reboot hard ... tant pis pour la perfection
qui visiblement n'est pas de ce monde ... grr !

Date: 2009-12-10 03:25:57 UTC
Dec 10 04:24:48 GMT: %IDBINDEX_SYNC-4-RESERVE: Failed to lookup existing ifindex for an interface on the Standby, allocating a new ifindex from the Active (ifindex=1791, idbtype=SWIDB)
Dec 10 04:24:51 GMT: %IDBINDEX_SYNC-4-RESERVE: Failed to lookup existing ifindex for an interface on the Standby, allocating a new ifindex from the Active (ifindex=1791, idbtype=HWIDB)
Dec 10 04:24:54 GMT: %IDBINDEX_SYNC-4-RESERVE: Failed to lookup existing ifindex for an interface on the Standby, allocating a new ifindex from the Active (ifindex=1792, idbtype=SWIDB)
Dec 10 04:24:57 GMT: %IDBINDEX_SYNC-4-RESERVE: Failed to lookup existing ifindex for an interface on the Standby, allocating a new ifindex from the Active (ifindex=1792, idbtype=HWIDB)
Dec 10 04:25:00 GMT: %IDBINDEX_SYNC-4-RESERVE: Failed to lookup existing ifindex for an interface on the Standby, allocating a new ifindex from the Active (ifindex=1793, idbtype=SWIDB)
Dec 10 04:25:03 GMT: %IDBINDEX_SYNC-4-RESERVE: Failed to lookup existing ifindex for an interface on the Standby, allocating a new ifindex from the Active (ifindex=1793, idbtype=HWIDB)
Dec 10 04:25:06 GMT: %IDBINDEX_SYNC-4-RESERVE: Failed to lookup existing ifindex for an interface on the Standby, allocating a new ifindex from the Active (ifindex=1794, idbtype=SWIDB)
Dec 10 04:25:09 GMT: %IDBINDEX_SYNC-4-RESERVE: Failed to lookup existing ifindex for an interface on the Standby, allocating a new ifindex from the Active (ifindex=1794, idbtype=HWIDB)
Dec 10 04:25:12 GMT: %IDBINDEX_SYNC-4-RESERVE: Failed to lookup existing ifindex for an interface on the Standby, allocating a new ifindex from the Active (ifindex=1795, idbtype=SWIDB)
Dec 10 04:25:16 GMT: %IDBINDEX_SYNC-4-RESERVE: Failed to lookup existing ifindex for an interface on the Standby, allocating a new ifindex from the Active (ifindex=1795, idbtype=HWIDB)

il est bien décidé de tout syncroniser. même ce qui n'existe pas.

le rollback est configuré à 45 minutes ...

Date: 2009-12-10 03:22:03 UTC
j'espere qu'il ne va pas aller jusqu'à 12000 ...

vss-2-6k#sh idb all | i 1601
H 1019 1601 D,A,R . Port-channelXXX
S 1016 1601 U . Port-channelXXX
vss-2-6k#sh idb all | i 1602
H 1020 1602 D,A,R . Port-channelXXX
S 1017 1602 U . Port-channelXXX
vss-2-6k#sh idb all | i 1603
S 1018 1603 U . Port-channelXXX
vss-2-6k#sh idb all | i 1604
vss-2-6k#sh idb all | i 1605
vss-2-6k#sh idb all | i 1606
Maximum number of Software IDBs 12000. In use 1606.
Total IDBs 1606 1606
vss-2-6k#sh idb all | i 1607
vss-2-6k#

Date: 2009-12-10 03:07:05 UTC
Dec 10 04:05:51 GMT: %IDBINDEX_SYNC-4-RESERVE: Failed to lookup existing ifindex for an interface on the Standby, allocating a new ifindex from the Active (ifindex=1603, idbtype=HWIDB)
Dec 10 04:05:54 GMT: %IDBINDEX_SYNC-4-RESERVE: Failed to lookup existing ifindex for an interface on the Standby, allocating a new ifindex from the Active (ifindex=1604, idbtype=SWIDB)
Dec 10 04:05:57 GMT: %IDBINDEX_SYNC-4-RESERVE: Failed to lookup existing ifindex for an interface on the Standby, allocating a new ifindex from the Active (ifindex=1604, idbtype=HWIDB)
Dec 10 04:06:00 GMT: %IDBINDEX_SYNC-4-RESERVE: Failed to lookup existing ifindex for an interface on the Standby, allocating a new ifindex from the Active (ifindex=1605, idbtype=SWIDB)
Dec 10 04:06:03 GMT: %IDBINDEX_SYNC-4-RESERVE: Failed to lookup existing ifindex for an interface on the Standby, allocating a new ifindex from the Active (ifindex=1605, idbtype=HWIDB)
Dec 10 04:06:06 GMT: %IDBINDEX_SYNC-4-RESERVE: Failed to lookup existing ifindex for an interface on the Standby, allocating a new ifindex from the Active (ifindex=1606, idbtype=SWIDB)
Dec 10 04:06:09 GMT: %IDBINDEX_SYNC-4-RESERVE: Failed to lookup existing ifindex for an interface on the Standby, allocating a new ifindex from the Active (ifindex=1606, idbtype=HWIDB)
Dec 10 04:06:12 GMT: %IDBINDEX_SYNC-4-RESERVE: Failed to lookup existing ifindex for an interface on the Standby, allocating a new ifindex from the Active (ifindex=1607, idbtype=SWIDB)
Dec 10 04:06:16 GMT: %IDBINDEX_SYNC-4-RESERVE: Failed to lookup existing ifindex for an interface on the Standby, allocating a new ifindex from the Active (ifindex=1607, idbtype=HWIDB)
Dec 10 04:06:19 GMT: %IDBINDEX_SYNC-4-RESERVE: Failed to lookup existing ifindex for an interface on the Standby, allocating a new ifindex from the Active (ifindex=1608, idbtype=SWIDB)
Dec 10 04:06:22 GMT: %IDBINDEX_SYNC-4-RESERVE: Failed to lookup existing ifindex for an interface on the Standby, allocating a new ifindex from the Active (ifindex=1608, idbtype=HWIDB)
Dec 10 04:06:25 GMT: %IDBINDEX_SYNC-4-RESERVE: Failed to lookup existing ifindex for an interface on the Standby, allocating a new ifindex from the Active (ifindex=1609, idbtype=SWIDB)
Dec 10 04:06:28 GMT: %IDBINDEX_SYNC-4-RESERVE: Failed to lookup existing ifindex for an interface on the Standby, allocating a new ifindex from the Active (ifindex=1609, idbtype=HWIDB)
Dec 10 04:06:31 GMT: %IDBINDEX_SYNC-4-RESERVE: Failed to lookup existing ifindex for an interface on the Standby, allocating a new ifindex from the Active (ifindex=1610, idbtype=SWIDB)
Dec 10 04:06:34 GMT: %IDBINDEX_SYNC-4-RESERVE: Failed to lookup existing ifindex for an interface on the Standby, allocating a new ifindex from the Active (ifindex=1610, idbtype=HWIDB)
Dec 10 04:06:37 GMT: %IDBINDEX_SYNC-4-RESERVE: Failed to lookup existing ifindex for an interface on the Standby, allocating a new ifindex from the Active (ifindex=1611, idbtype=SWIDB)



Date: 2009-12-10 03:06:43 UTC
Dec 10 04:02:35 GMT: %CHKPT-SW2_SP-4-INVALID: Invalid checkpoint client ID (231).
Dec 10 04:02:35 GMT: %CHKPT-SW2_SP-4-INVALID: Invalid checkpoint client ID (232).
Dec 10 04:02:35 GMT: %CHKPT-SW2_SP-4-INVALID: Invalid checkpoint client ID (233).
Dec 10 04:02:35 GMT: %CHKPT-SW2_SP-4-INVALID: Invalid checkpoint client ID (234).



Date: 2009-12-10 02:38:00 UTC
On commence les opérations dans 10 minutes.
Posted Dec 07, 2009 - 21:50 UTC