rssLink RSS for all categories
 
icon_blue
icon_green
icon_green
icon_orange
icon_red
icon_green
icon_green
icon_orange
icon_red
icon_orange
icon_red
icon_green
icon_green
icon_blue
icon_red
icon_green
icon_green
icon_blue
icon_red
icon_red
icon_green
icon_red
icon_orange
icon_red
icon_orange
icon_green
icon_green
icon_red
icon_green
icon_blue
icon_green
icon_orange
 

FS#6241 — Upgrades ASR 9K Roubaix

Attached to Project— Reseau Internet et Baies
Maintenance
Tout le réseau
CLOSED
100%
Nous allons upgrader les 2 routeurs "core" ici à Roubaix en tête du réseau afin de pouvoir ajouter de la capacité. Ces chassis sont maintenant full en terme de linecard 8x10G. Nous allons passer aux nouvelles RSP 440 qui vont nous permettre de remplacer certaines de ces linecards par les nouvelles 24x10G. La procédure est la suivante:
- upgrade software
- remplacement des RSPs (arrêt/redémarrage complet du chassis nécessaire)
- remplacement de 2 des linecards 8G par des 24g

Ces interventions auront bien entendu lieu de nuit, on démarre cette nuit (du 10 au 11) avec rbx-g1-a9 à 00:00 CET. Pas d'impact sur le trafic à anticiper car nous allons forcer le trafic par rbx-g2 pendant la durée des travaux.
Date:  Tuesday, 31 January 2012, 14:29PM
Reason for closing:  Done
Comment by OVH - Tuesday, 10 January 2012, 23:54PM

Nous démarrons les travaux. Nous allons rerouter une partie du trafic des transits à Londres et Amsterdam vers Paris afin de délester les liens de rbx-g1 vers ldn-1/5 et ams-1/5.


Comment by OVH - Wednesday, 11 January 2012, 00:30AM

nous allons preparer le routeur à la mise en place
de la nouvelle version de IOS XR


Comment by OVH - Wednesday, 11 January 2012, 00:55AM

RP/0/RSP1/CPU0:rbx-g1-a9(admin)#show install inactive
Tue Jan 10 23:55:33.960 UTC
Secure Domain Router: Owner

There are no inactive packages on the SDR.

RP/0/RSP1/CPU0:rbx-g1-a9#sh install summary
Tue Jan 10 23:56:25.147 UTC
Active Packages:
disk0:asr9k-mini-p-4.0.1
disk0:asr9k-doc-p-4.0.1
disk0:asr9k-k9sec-p-4.0.1
disk0:asr9k-mpls-p-4.0.1
disk0:asr9k-mgbl-p-4.0.1
disk0:asr9k-mcast-p-4.0.1


Comment by OVH - Wednesday, 11 January 2012, 01:15AM

(admin) install add source disk0:// asr9k-doc-p.pie-4.2.0 asr9k-k9sec-p.pie-4.2.0 asr9k-mcast-p.pie-4.2.0 asr9k-mgbl-p.pie-4.2.0 asr9k-mini-p.pie-4.2.0 asr9k-mpls-p.pie-4.2.0 asr9k-upgrade-p.pie-4.2.0 synchronous


Comment by OVH - Wednesday, 11 January 2012, 01:20AM

pendant ce temps là sur le g2:

RP/0/RSP0/CPU0:rbx-g2-a9#delete disk0:ASR9K-iosxr-k9-4.1.1.tar

RP/0/RSP0/CPU0:rbx-g2-a9#sh install inactive
Wed Jan 11 00:19:21.016 UTC
Node 0/RSP0/CPU0 [RP] [SDR: Owner]
Boot Device: disk0:
Inactive Packages:
disk0:asr9k-upgrade-p-4.1.0
disk0:asr9k-mini-p-4.1.0
disk0:asr9k-doc-p-4.1.0
disk0:asr9k-k9sec-p-4.1.0
disk0:asr9k-mpls-p-4.1.0
disk0:asr9k-mgbl-p-4.1.0
disk0:asr9k-mcast-p-4.1.0
[...]
RP/0/RSP0/CPU0:rbx-g2-a9#admin
Wed Jan 11 00:19:38.546 UTC
RP/0/RSP0/CPU0:rbx-g2-a9(admin)#install remove inactive
Wed Jan 11 00:19:44.906 UTC
Install operation 26 '(admin) install remove inactive' started by user 'octave' via CLI at 00:19:45 UTC Wed Jan 11 2012.
Info: This operation will remove the following packages:
Info: disk0:asr9k-upgrade-p-4.1.0
Info: disk0:asr9k-doc-4.1.0
Info: disk0:asr9k-adv-video-4.1.0
Info: disk0:asr9k-sbc-4.1.0
Info: disk0:asr9k-k9sec-4.1.0
[...]
Proceed with removing these packages? [confirm]
The install operation will continue asynchronously.


Comment by OVH - Wednesday, 11 January 2012, 01:36AM

Nous avons changé le cost OSPF du g1 pour le sortir
du routage.


Comment by OVH - Wednesday, 11 January 2012, 01:37AM

c'est parti:

RP/0/RSP1/CPU0:rbx-g1-a9(admin)#install activate ?
disk0:asr9k-doc-p-4.2.0 Package to activate
disk0:asr9k-k9sec-p-4.2.0 Package to activate
disk0:asr9k-mcast-p-4.2.0 Package to activate
disk0:asr9k-mgbl-p-4.2.0 Package to activate
disk0:asr9k-mini-p-4.2.0 Package to activate
disk0:asr9k-mpls-p-4.2.0 Package to activate
disk0:asr9k-upgrade-p-4.2.0 Package to activate
id The id of the install add operation containing packages to activate

RP/0/RSP1/CPU0:rbx-g1-a9(admin)#install activate disk0:*4.2.0* synchronous
[...]
Proceed with this install operation (y/n)? [y]


Comment by OVH - Wednesday, 11 January 2012, 01:42AM

Proceed with this install operation (y/n)? [y]
Info: Install Method: Parallel Reload
Info: The changes made to software configurations will not be persistent across system reloads. Use the command '(admin) install commit' to make changes persistent.
Info: Please verify that the system is consistent following the software change using the following commands:
Info: show system verify
Info: install verify packages
Install operation 132 completed successfully at 00:41:37 UTC Wed Jan 11 2012.
RP/0/RSP1/CPU0:rbx-g1-a9(admin)#
RP/0/RSP1/CPU0:rbx-g1-a9(admin)#
RP/0/RSP1/CPU0:rbx-g1-a9(admin)#
RP/0/RSP1/CPU0:rbx-g1-a9(admin)#
RP/0/RSP1/CPU0:rbx-g1-a9(admin)#

ça y est ... il s'est rebooté


Comment by OVH - Wednesday, 11 January 2012, 01:47AM

This (D)RP Node is not ready or active for login /configuration

This (D)RP Node is not ready or active for login /configuration

This (D)RP Node is not ready or active for login /configuration

*** insthelper node 0/RSP0/CPU0: rebooting in 35 seconds due to reload upgrade 132 ...
[0x79701f673bab5] Record Reboot History, reboot cause = 0x4000014, descr = Cause: Install operation. Shutdown standby node gracefully. Process: redcon
Traceback: 4c223c30 4rebooting

Selecting ROMMON Image... B
DDR in Interleaved mode
POST 1 : PASSED : code 0 : DDR2 Memory Quick Test

CPU Reset Reason = 0x000d
POST 2 : PASSED : code 0 : FPGA Flash Image CRC Checks

Loading Field Programmable Devices:
FPGA 0-B PROGRAMMED : image: 0xff500028 - 0xff576cca, et: 117ms
FPGA 1-B PROGRAMMED : image: 0xff400028 - 0xff4d1034, et: 206ms
FPGA 2-B PROGRAMMED : image: 0xff100028 - 0xff276358, et: 369ms
FPGA 3-B PROGRAMMED : image: 0xff000028 - 0xff0454a8, et: 69ms

System Bootstrap, Version 1.05(20101118:025914) [ASR9K ROMMON],
Copyright (c) 1994-2010 by Cisco Systems, Inc.
Compiled Wed 17-Nov-10 18:59 by saurabja
[...]


Comment by OVH - Wednesday, 11 January 2012, 01:55AM

le routeur est up.

RP/0/RSP0/CPU0:rbx-g1-a9(admin)#sh install summary
Wed Jan 11 00:53:30.008 UTC
Default Profile:
SDRs:
Owner
Active Packages:
disk0:asr9k-upgrade-p-4.2.0
disk0:asr9k-mini-p-4.2.0
disk0:asr9k-doc-p-4.2.0
disk0:asr9k-k9sec-p-4.2.0
disk0:asr9k-mpls-p-4.2.0
disk0:asr9k-mgbl-p-4.2.0
disk0:asr9k-mcast-p-4.2.0

RP/0/RSP0/CPU0:rbx-g1-a9(admin)#install deactivate disk0:asr9k-upgrade-p-4.2.0 sync

RP/0/RSP0/CPU0:rbx-g1-a9(admin)#install remove disk0:asr9k-upgrade-p-4.2.0 sync
Wed Jan 11 00:56:34.030 UTC
Install operation 134 '(admin) install remove disk0:asr9k-upgrade-p-4.2.0 synchronous' started by user 'octave' via CLI at 00:56:34 UTC Wed Jan 11 2012.
Info: This operation will remove the following package:
Info: disk0:asr9k-upgrade-p-4.2.0
Info: After this install remove the following install rollback point will no longer be reachable, as the required packages will not be present:
Info: 132
Proceed with removing these packages? [confirm]
Install operation 134 completed successfully at 00:56:52 UTC Wed Jan 11 2012.


Comment by OVH - Wednesday, 11 January 2012, 01:59AM

RP/0/RSP0/CPU0:rbx-g1-a9(admin)#sh install summary
Wed Jan 11 00:58:56.412 UTC
Default Profile:
SDRs:
Owner
Active Packages:
disk0:asr9k-mini-p-4.2.0
disk0:asr9k-doc-p-4.2.0
disk0:asr9k-k9sec-p-4.2.0
disk0:asr9k-mpls-p-4.2.0
disk0:asr9k-mgbl-p-4.2.0
disk0:asr9k-mcast-p-4.2.0


Comment by OVH - Wednesday, 11 January 2012, 02:02AM

RP/0/RSP0/CPU0:rbx-g1-a9(admin)#sh install inactive
Wed Jan 11 00:59:08.043 UTC
Secure Domain Router: Owner

Node 0/RSP0/CPU0 [RP] [SDR: Owner]
Boot Device: disk0:
Inactive Packages:
disk0:asr9k-mini-p-4.0.1
disk0:asr9k-doc-p-4.0.1
disk0:asr9k-k9sec-p-4.0.1
disk0:asr9k-mpls-p-4.0.1
disk0:asr9k-mgbl-p-4.0.1
disk0:asr9k-mcast-p-4.0.1
[...]

RP/0/RSP0/CPU0:rbx-g1-a9(admin)#install remove ?
WORD <device><directory> (may include '*' or '?')
disk0:asr9K-doc-supp-4.0.1 Package to remove
disk0:asr9k-base-4.0.1 Package to remove
disk0:asr9k-cpp-4.0.1 Package to remove
[...]

RP/0/RSP0/CPU0:rbx-g1-a9(admin)#install remove inactive synchronous
Wed Jan 11 01:00:11.094 UTC
Install operation 136 '(admin) install remove inactive synchronous' started by user 'octave' via CLI at 01:00:11 UTC Wed Jan 11 2012.
Info: This operation will remove the following packages:
Info: disk0:asr9k-mpls-p-4.0.1
Info: disk0:asr9k-mini-p-4.0.1
Info: disk0:asr9k-mgbl-p-4.0.1
Info: disk0:asr9k-mcast-p-4.0.1
Info: disk0:asr9k-k9sec-p-4.0.1
Info: disk0:asr9k-doc-p-4.0.1
Info: After this install remove the following install rollback points will no longer be reachable, as the required packages will not be present:
Info: 114, 116
Proceed with removing these packages? [confirm]


Comment by OVH - Wednesday, 11 January 2012, 02:06AM

RP/0/RSP0/CPU0:rbx-g1-a9(admin)#upgrade hw-module fpd all location all
Wed Jan 11 01:04:46.813 UTC

***** UPGRADE WARNING MESSAGE: *****
* This upgrade operation has a maximum timout of 90 minutes. *
* If you are executing the cmd for one specific location and *
* card in that location reloads or goes down for some reason *
* you can press CTRL-C to get back the RP's prompt. *
* If you are executing the cmd for _all_ locations and a node *
* reloads or is down please allow other nodes to finish the *
* upgrade process before pressing CTRL-C. *

% RELOAD REMINDER:
- The upgrade operation of the target module will not interrupt its normal
operation. However, for the changes to take effect, the target module
will need to be manually reloaded after the upgrade operation. This can
be accomplished with the use of "hw-module <target> reload" command.
- If automatic reload operation is desired after the upgrade, please use
the "reload" option at the end of the upgrade command.
- The output of "show hw-module fpd location" command will not display
correct version information after the upgrade if the target module is
not reloaded.
NOTE: Chassis CLI will not be accessible while upgrade is in progress.
Continue? [confirm]
This can take some time for a full chassis.
Ensure that system is not power cycled during the upgrades.
Please consult the documentation for more information.
Continue ? [no]: yes


FPD upgrade in progress on some hardware, reload/configuration change
on those is not recommended as it might cause HW programming failure
and result in RMA of the hardware.


Comment by OVH - Wednesday, 11 January 2012, 02:15AM

Successfully upgraded fpga3 for A9K-RSP-4G on location 0/RSP0/CPU0 from 1.18 to 1.23
Successfully upgraded fpga3 for A9K-RSP-4G on location 0/RSP1/CPU0 from 1.18 to 1.23
FPD upgrade in progress. Max timeout remaining 89 min.
Successfully upgraded cpld1 for A9K-8T-L on location 0/4/CPU0 from 1.00 to 1.02
Successfully upgraded cpld4 for A9K-8T-L on location 0/4/CPU0 from 1.00 to 1.03
FPD upgrade in progress. Max timeout remaining 88 min.
FPD upgrade in progress. Max timeout remaining 87 min.
FPD upgrade in progress. Max timeout remaining 86 min.
FPD upgrade in progress. Max timeout remaining 85 min.
FPD upgrade in progress. Max timeout remaining 84 min.
Successfully upgraded fpga for A9K-8T-L on location 0/6/CPU0 from 1.00 to 1.02
Successfully upgraded cpld1 for A9K-8T-L on location 0/6/CPU0 from 1.00 to 1.02
Successfully upgraded cpld4 for A9K-8T-L on location 0/6/CPU0 from 1.00 to 1.03
Successfully upgraded fpga for A9K-8T-L on location 0/6/CPU0 from 1.00 to 1.02
FPD upgrade in progress. Max timeout remaining 83 min.
FPD upgrade in progress. Max timeout remaining 82 min.
Successfully upgraded fpga for A9K-8T-L on location 0/2/CPU0 from 1.00 to 1.02
Successfully upgraded cpld1 for A9K-8T-L on location 0/2/CPU0 from 1.00 to 1.02
Successfully upgraded cpld4 for A9K-8T-L on location 0/2/CPU0 from 1.00 to 1.03
Successfully upgraded fpga for A9K-8T-L on location 0/2/CPU0 from 1.00 to 1.02
Successfully upgraded fpga for A9K-8T-L on location 0/1/CPU0 from 1.00 to 1.02
Successfully upgraded cpld1 for A9K-8T-L on location 0/1/CPU0 from 1.00 to 1.02
Successfully upgraded cpld4 for A9K-8T-L on location 0/1/CPU0 from 1.00 to 1.03
Successfully upgraded fpga for A9K-8T-L on location 0/1/CPU0 from 1.00 to 1.02
FPD upgrade in progress. Max timeout remaining 81 min.
Successfully upgraded fpga for A9K-8T-L on location 0/5/CPU0 from 1.00 to 1.02
Successfully upgraded cpld1 for A9K-8T-L on location 0/5/CPU0 from 1.00 to 1.02
Successfully upgraded cpld4 for A9K-8T-L on location 0/5/CPU0 from 1.00 to 1.03
Successfully upgraded fpga for A9K-8T-L on location 0/5/CPU0 from 1.00 to 1.02
Successfully upgraded fpga for A9K-8T-L on location 0/0/CPU0 from 1.00 to 1.02
Successfully upgraded cpld1 for A9K-8T-L on location 0/0/CPU0 from 1.00 to 1.02
Successfully upgraded cpld4 for A9K-8T-L on location 0/0/CPU0 from 1.00 to 1.03
Successfully upgraded fpga for A9K-8T-L on location 0/0/CPU0 from 1.00 to 1.02
Successfully upgraded fpga for A9K-8T-L on location 0/7/CPU0 from 1.00 to 1.02
Successfully upgraded cpld1 for A9K-8T-L on location 0/7/CPU0 from 1.00 to 1.02
Successfully upgraded cpld4 for A9K-8T-L on location 0/7/CPU0 from 1.00 to 1.03
Successfully upgraded fpga for A9K-8T-L on location 0/7/CPU0 from 1.00 to 1.02
Successfully upgraded fpga for A9K-8T-L on location 0/3/CPU0 from 1.00 to 1.02
Successfully upgraded cpld1 for A9K-8T-L on location 0/3/CPU0 from 1.00 to 1.02
Successfully upgraded cpld4 for A9K-8T-L on location 0/3/CPU0 from 1.00 to 1.03
Successfully upgraded fpga for A9K-8T-L on location 0/3/CPU0 from 1.00 to 1.02


FPD upgrade has ended.


Comment by OVH - Wednesday, 11 January 2012, 02:18AM

on va maintenant sortir les RP2 du chassis g1 et
mettre en place les nouvelles RP440.


Comment by OVH - Wednesday, 11 January 2012, 02:21AM

c'est fait. le routeur est à nouveau down. on insert les nouvelles
cartes qu'on va reconfigurer from scratch puisqu'elles sont vierges


Comment by OVH - Wednesday, 11 January 2012, 02:34AM

suprise: il n'y a pas d'image du tout sur les RP440.
elles sont vierges de chez vierge

##########################################################
System Bootstrap, Version 0.46 [ASR9K x86 ROMMON],
Copyright (c) 1994-2011 by Cisco Systems, Inc.
Compiled on Thu 09/22/2011 11:18:01.84 by myekkar

Rommon : 0.46
Ibex Peak : 6
Jasper Forest: 1.0
Zen-JF : 0.7.92
CBC : Part 1=16.114, Part 2=16.114, Act Part=2
Laguna : 0.9.0
Dao : 0.8.0
UTI : 4.9
Timex : 0.1.1
Board : 4
==========================================================
Sending Boot Image validation request.
HIT CTRL-C to abort
-----------
Interface link did not come up. Timed out.
Boot Image validation: Link failure
Sending Boot Image validation request.
HIT CTRL-C to abort
-----------
Interface link did not come up. Timed out.
Boot Image validation: Link failure
Sending Boot Image validation request.
HIT CTRL-C to abort
-----------
Interface link did not come up. Timed out.
Boot Image validation: Link failure
Sending Boot Image validation request.
HIT CTRL-C to abort
-----------
Interface link did not come up. Timed out.
Boot Image validation: Link failure
Sending Boot Image validation request.
HIT CTRL-C to abort


Comment by OVH - Wednesday, 11 January 2012, 02:35AM

on remet les anciennes RP2 et on redemarre le g1.
on va lui remettre le trafic comme avant mais en 4.2
puis mettre à jour le g2.

puis on va regarder avec cisco comment on demarre
ces cartes sans image ..


Comment by OVH - Wednesday, 11 January 2012, 02:39AM

RP2 sont en boot.

Selecting ROMMON Image... B
DDR in Interleaved mode
POST 1 : PASSED : code 0 : DDR2 Memory Quick Test

CPU Reset Reason = 0x0002
POST 2 : PASSED : code 0 : FPGA Flash Image CRC Checks

Loading Field Programmable Devices:
FPGA 0-B PROGRAMMED : image: 0xff500028 - 0xff576cca, et: 117ms
FPGA 1-B PROGRAMMED : image: 0xff400028 - 0xff4d1034, et: 206ms
FPGA 2-B PROGRAMMED : image: 0xff100028 - 0xff276358, et: 369ms
FPGA 3-B PROGRAMMED : image: 0xff000028 - 0xff0454a8, et: 69ms

System Bootstrap, Version 1.05(20101118:025914) [ASR9K ROMMON],
Copyright (c) 1994-2010 by Cisco Systems, Inc.
Compiled Wed 17-Nov-10 18:59 by saurabja

CPUCtrl: 1.17 [00000001/00000011]
ClkCtrl: 1.23 [00000001/00000017]
IntCtrl: 1.15 [00000001/0000000f]
Punt: 1.5 [00000001/00000005]
CBC: 1.2
BID: 0x0006


PPC 8641D (partnum 0x8004), Revision 03.00, (Core Version 02.02)
M8641 CLKIN: 66 Mhz
Core Clock: 1333 Mhz
MPX Clock: 533 Mhz
LBC Clock: 33 Mhz

POST 3 : PASSED : code 0 : Slot ID/Board Type Validity
PCI-E1: Ready as Root Complex
PCI-E2: Ready as Root Complex


set_chassis_type: chassis_type=0xef02fe found=TRUE
ASR9K (8641D PPC) platform with 4096 Mb of main memory

program load complete, entry point: 0x100000, size: 0x2ac18
program load complete, entry point: 0x100000, size: 0x2ac18
MBI Candidate = disk0:asr9k-os-mbi-4.2.0/0x100000/mbiasr9k-rp.vm

CARD_SLOT_NUMBER: 4
CPU_INSTANCE: 1
MBI Validation starts ...
Missing or illegal ip address for variable IP_ADDRESS

Mgt LAN 0 interface is selected
tsec_init_hw: configuring FE (port 2) for: Auto Speed, Auto Duplex

tsec_init_interface: hardware initialization completed
Interface link changed state to UP.
Interface link state up.

MBI validation sending request.
HIT CTRL-C to abort
.........
No MBI confirmation received from dSC

AUTOBOOT: Boot string = disk0:asr9k-os-mbi-4.2.0/0x100000/mbiasr9k-rp.vm,1;
AUTOBOOT: autobootstate=0, autobootcount=0, cmd=boot disk0:asr9k-os-mbi-4.2.0/0x100000/mbiasr9k-rp.vm
program load complete, entry point: 0x100000, size: 0x2ac18

MBI size from header = 21578140,Bootflash resident MBI filesize = 21578140
..............................


Comment by OVH - Wednesday, 11 January 2012, 02:55AM

g1 est up. on lui change les cost.

puis on attaque le g2.

dans la journée on va voir avec cisco pourquoi la RP440
ne boote pas. d'apres les gens de cisco que nous avons
pu avoir à cette heure tardive, c'est pas normal.


Comment by OVH - Wednesday, 11 January 2012, 03:29AM

le process ospv3 a un pb:

Jan 11 03:28:25 rbx-g2-a9.fr.eu 88274: RP/0/RSP0/CPU0:Jan 11 02:28:25 UTC: ospf[1009]: %ROUTING-OSPF-3-INTERNALERR : Internal error, path id out of range
Jan 11 03:28:25 rbx-g2-a9.fr.eu 88275: RP/0/RSP0/CPU0:Jan 11 02:28:25 UTC: ospf[1009]: %ROUTING-OSPF-3-INTERNALERR : Internal error, path id out of range
Jan 11 03:28:25 rbx-g2-a9.fr.eu 88276: RP/0/RSP0/CPU0:Jan 11 02:28:25 UTC: ospf[1009]: %ROUTING-OSPF-3-INTERNALERR : Internal error, path id out of range
Jan 11 03:28:25 rbx-g2-a9.fr.eu 88277: RP/0/RSP0/CPU0:Jan 11 02:28:25 UTC: ospf[1009]: %ROUTING-OSPF-3-INTERNALERR : Internal error, path id out of range
Jan 11 03:28:25 rbx-g2-a9.fr.eu 88278: RP/0/RSP0/CPU0:Jan 11 02:28:25 UTC: ospf[1009]: %ROUTING-OSPF-3-INTERNALERR : Internal error, path id out of range
Jan 11 03:28:25 rbx-g2-a9.fr.eu 88279: RP/0/RSP0/CPU0:Jan 11 02:28:25 UTC: ospf[1009]: %ROUTING-OSPF-3-INTERNALERR : Internal error, path id out of range
Jan 11 03:28:25 rbx-g2-a9.fr.eu 88280: RP/0/RSP0/CPU0:Jan 11 02:28:25 UTC: ospf[1009]: %ROUTING-OSPF-3-INTERNALERR : Internal error, path id out of range
Jan 11 03:28:25 rbx-g2-a9.fr.eu 88281: RP/0/RSP0/CPU0:Jan 11 02:28:25 UTC: ospf[1009]: %ROUTING-OSPF-3-INTERNALERR : Internal error, path id out of range
Jan 11 03:28:25 rbx-g2-a9.fr.eu 88282: RP/0/RSP0/CPU0:Jan 11 02:28:25 UTC: ospf[1009]: %ROUTING-OSPF-3-INTERNALERR : Internal error, path id out of range
Jan 11 03:28:25 rbx-g2-a9.fr.eu 88283: RP/0/RSP0/CPU0:Jan 11 02:28:25 UTC: ospf[1009]: %ROUTING-OSPF-3-INTERNALERR : Internal error, path id out of range
Jan 11 03:28:25 rbx-g2-a9.fr.eu 88284: RP/0/RSP0/CPU0:Jan 11 02:28:25 UTC: ospf[1009]: %ROUTING-OSPF-3-INTERNALERR : Internal error, path id out of range
Jan 11 03:28:25 rbx-g2-a9.fr.eu 88285: RP/0/RSP0/CPU0:Jan 11 02:28:25 UTC: ospf[1009]: %ROUTING-OSPF-3-INTERNALERR : Internal error, path id out of range


Comment by OVH - Wednesday, 11 January 2012, 03:35AM

Nous avons du forcer un basculement sur la RSP1 et faire un reset du process opsfv3. La RSP0 est en cours de redémarrage.


Comment by OVH - Wednesday, 11 January 2012, 03:48AM

Activation des nouveaux paquets en cours sur G2:

Warning: - the set of packages being activated.
Info: This operation will reload the following nodes in parallel:
Info: 0/RSP0/CPU0 (RP) (SDR: Owner)
Info: 0/RSP1/CPU0 (RP) (SDR: Owner)
Info: 0/0/CPU0 (LC) (SDR: Owner)
Info: 0/1/CPU0 (LC) (SDR: Owner)
Info: 0/2/CPU0 (LC) (SDR: Owner)
Info: 0/3/CPU0 (LC) (SDR: Owner)
Info: 0/4/CPU0 (LC) (SDR: Owner)
Info: 0/5/CPU0 (LC) (SDR: Owner)
Info: 0/6/CPU0 (LC) (SDR: Owner)
Info: 0/7/CPU0 (LC) (SDR: Owner)
Proceed with this install operation (y/n)? [y]
Info: Install Method: Parallel Reload


Comment by OVH - Wednesday, 11 January 2012, 03:53AM

Le routeur est en cours de redémarrage


Comment by OVH - Wednesday, 11 January 2012, 04:16AM

Le routeur a correctement redémarré sur les nouvelles images. On met à jour les fpd:
FPD upgrade in progress. Max timeout remaining 89 min.


Comment by OVH - Wednesday, 11 January 2012, 04:22AM

L'upgrade des FPDs est terminé. On reboote une dernière fois tous les nodes.


Comment by OVH - Wednesday, 11 January 2012, 04:34AM

Le routeur est à jour et dans un état stable. On réactive le trafic dessus.


Comment by OVH - Wednesday, 11 January 2012, 04:44AM

Le trafic est réactivé. Les 2 routeurs rbx-g1 et g2 sont en fonctionnement normal.


Comment by OVH - Monday, 16 January 2012, 19:46PM

Nous reprenons les travaux ce soir à partir de 00:00 CET. Les 2 routeurs sont maintenant à jour au niveau soft et nous avons installé l'IOS XR sur les nouvelles RSPS 440. Nous allons les insérer cette nuit dans les chassis rbx-g1 et g2. Si tout se déroule correctement, nous pourrons insérer les nouvelles 24x10G.


Comment by OVH - Tuesday, 17 January 2012, 00:11AM

Nous commencons les travaux. On passe le trafic vers rbx-g2.


Comment by OVH - Tuesday, 17 January 2012, 00:27AM

Nous remplacons les cartes RSP.


Comment by OVH - Tuesday, 17 January 2012, 00:47AM

le routeur a booté. on est en train de lui repusher la configuration


Comment by OVH - Tuesday, 17 January 2012, 00:58AM

Le routeur a démarré correctement sur les nouvelles RSPs. Nous redescendons la config.


Comment by OVH - Tuesday, 17 January 2012, 01:05AM

g1 a été configuré. on va lui repasser le trafic


Comment by OVH - Tuesday, 17 January 2012, 01:20AM

Le trafic s'écoule normalement sur g1. On attaque g2.


Comment by OVH - Tuesday, 17 January 2012, 01:20AM

Nous remplacons les RSP sur rbx-g2


Comment by OVH - Tuesday, 17 January 2012, 01:34AM

Sur ces RSPS, une nouvelle config port console est nécessaire:
line console
exec-timeout 0 0
parity none
stopbits 1
length 60
access-class ingress 10
access-class egress 10


Comment by OVH - Tuesday, 17 January 2012, 01:39AM

On redescend la config sur g2.


Comment by OVH - Tuesday, 17 January 2012, 02:02AM

On réactive le trafic sur g2.


Comment by OVH - Tuesday, 17 January 2012, 02:15AM

Les 2 routeurs sont stables. On remplace la carte 0 de rbx-g2 par l'une des nouvelles carte 24x10G.


Comment by OVH - Tuesday, 17 January 2012, 02:22AM

RP/0/RSP0/CPU0:rbx-g2-a9#RP/0/RSP1/CPU0:Jan 17 01:21:39 UTC: dao_tmp[55]: Card Inserted 0/0/CPU0
RP/0/RSP0/CPU0:Jan 17 01:21:39 UTC: dao_tmp[55]: Card Inserted 0/0/CPU0
RP/0/RSP0/CPU0:Jan 17 01:21:50 UTC: shelfmgr[385]: %PLATFORM-SHELFMGR-3-MIXED_MEMORY_POWERED : Node 0/0/CPU0 is powered, system running mixed li
ode
RP/0/RSP0/CPU0:Jan 17 01:21:50 UTC: shelfmgr[385]: %PLATFORM-SHELFMGR-3-MIXED_MEMORY_ALARM : System enters mixed line card memory mode
RP/0/RSP0/CPU0:Jan 17 01:21:50 UTC: eem_ed_oir[190]: Messge received content : Event 0 NodeId: 0x801


Comment by OVH - Tuesday, 17 January 2012, 03:04AM

Nos optiques sont bien reconnues par les cartes mais n'émettent pas de lumière. Nous suspectons une incompatibilité.


Comment by OVH - Tuesday, 17 January 2012, 03:18AM

On fallback sur la 8x10G. Nous allons travailler avec cisco sur le problème de compatibilité des optiques et vérifier les implications exacts du message "MIXED_MEMORY_ALARM" remonté par la carte à l'insertion.


Comment by OVH - Tuesday, 17 January 2012, 04:00AM

Nous avons finalement pu faire fonctinner un autre type d'optique. On laisse tourner cette 24x10G afin de vérifier son fonctionnement. S'il n'y a pas d'autre soucis, nous insérerons les suivantes.


Comment by OVH - Thursday, 26 January 2012, 19:50PM

Nous allons upgrader la capa entre rbx-g1/g2-a9 et les rbx-g1/g2-n7. L'upgrade sur l'ensemble g2 est en cours. Sur g1, nous devons remplacer certaines cartes 8T par des 24x10G. Nous allons réaliser cette opération cette nuit à partir de 00:00 CET.


Comment by OVH - Friday, 27 January 2012, 00:02AM

Nous commencons l'intervention de remplacement des cartes 8x10G par les 24x10G.
Nous allons intervenir sur rbx-g2-a9 puis rbx-g1-a9.


Comment by OVH - Friday, 27 January 2012, 00:20AM

Les nouvelles cartes sont en prod sur rbx-g2. Même chose maintenant pour rbx-g1.


Comment by OVH - Friday, 27 January 2012, 01:08AM

Les nouvelles cartes sont en prod sur rbx-g1-a9.