OVHcloud Network Status
Current status
Legend
- Operational
- Degraded performance
- Partial Outage
- Major Outage
- Under maintenance
vac1
Incident
Report for Network & Infrastructure
Resolved
Le VAC1 ne fonctionne pas correctement. Arbor du VAC1
semble d'avoir un probleme. On vient de le couper.
L'anti-ddos est effectué sur le VAC2 et VAC3.
Update(s):
Date: 2015-09-21 11:46:42 UTC
Tout est rentré dans l'ordre. Nous remettons vac1 en marche.
Date: 2015-09-20 18:44:16 UTC
ça sent pas bon du tout. on est parti pour gerer
une panne hardware avec arbor.
On a fait differentes manipulations, on a retiré les
cartes une par une et ça ne marche toujours pas. A
chaque fois on a fait le reload du chassis. On s'est
appeçu que durant le reload les ports 10G ne passent
pas DOWN, on pense donc que le reload software ne
fait pas le vrai reload du chassis. On a donc arraché
très violament les cables d'alimentation de chassis
(3 cables !!) et on les a remis avec de l'amour dans
le geste. C'est mieux. Le chassis a reinitié toutes
les cartes. On surveille.
Date: 2015-09-20 18:42:00 UTC
Sep 20 18:38:31 vac1-10-tms blinky[30476]: [W] 'get_pktengine_config_version' to 'apm-1-0:8' failed on [Errno 111] Connection refused host:apm-1-0
Sep 20 18:38:31 vac1-10-tms blinky[30476]: [W] 'get_pktengine_config_version' to 'apm-1-0:19' failed on [Errno 111] Connection refused host:apm-1-0
Sep 20 18:38:31 vac1-10-tms blinky[30476]: [W] 'get_pktengine_config_version' to 'apm-0-0:4' failed on [Errno 111] Connection refused host:apm-0-0
Sep 20 18:38:31 vac1-10-tms blinky[30476]: [W] 'get_pktengine_config_version' to 'apm-1-0:15' failed on [Errno 111] Connection refused host:apm-1-0
Sep 20 18:38:31 vac1-10-tms blinky[30476]: [W] 'get_pktengine_config_version' to 'apm-1-0:17' failed on [Errno 111] Connection refused host:apm-1-0
Sep 20 18:38:31 vac1-10-tms blinky[30476]: [W] 'get_pktengine_config_version' to 'apm-0-1:19' failed on [Errno 111] Connection refused host:apm-0-1
Sep 20 18:38:31 vac1-10-tms blinky[30476]: [W] 'get_pktengine_config_version' to 'apm-0-1:20' failed on [Errno 111] Connection refused host:apm-0-1
Sep 20 18:38:31 vac1-10-tms blinky[30476]: [W] 'get_pktengine_config_version' to 'apm-1-0:6' failed on [Errno 111] Connection refused host:apm-1-0
Sep 20 18:38:31 vac1-10-tms blinky[30476]: [W] 'get_pktengine_config_version' to 'apm-0-0:20' failed on [Errno 111] Connection refused host:apm-0-0
Sep 20 18:38:31 vac1-10-tms blinky[30476]: [W] 'get_pktengine_config_version' to 'apm-1-1:0' failed on [Errno 111] Connection refused host:apm-1-1
Sep 20 18:38:31 vac1-10-tms blinky[30476]: [W] 'get_pktengine_config_version' to 'apm-1-1:3' failed on [Errno 111] Connection refused host:apm-1-1
Sep 20 18:38:31 vac1-10-tms blinky[30476]: [W] 'get_pktengine_config_version' to 'apm-0-1:18' failed on [Errno 111] Connection refused host:apm-0-1
Date: 2015-09-20 18:40:21 UTC
admin@vac1-10-tms:/# services tms show
Peakflow TMS state: stopped
admin@vac1-10-tms:/# services tms start
Starting Peakflow TMS services..done.
admin@vac1-10-tms:/#
Date: 2015-09-20 18:38:33 UTC
Sep 20 18:25:55 (none) python[6468]: [S] #SUBHOSTS-REBOOT found 3 apms
Sep 20 18:26:51 (none) python[6468]: [S] #SUBHOSTS-REBOOT verified apm-0-ipmc
Sep 20 18:26:51 (none) python[6468]: [S] #SUBHOSTS-REBOOT verified apm-1-ipmc
Sep 20 18:26:51 (none) python[6468]: [S] #SUBHOSTS-REBOOT verified apm-2-ipmc
Date: 2015-09-20 18:18:40 UTC
on va rebooter le bouzin.
admin@vac1-10-tms:/# services tms stop
Stopping Peakflow TMS services....................................................done.
admin@vac1-10-tms:/# re
now Reload without confirmation
Reload with confirmation
admin@vac1-10-tms:/# re now
094: Rebooting the system..
Broadcast message from root (pts/8) (Sun Sep 20 18:17:35 2015):
The system is going down for reboot NOW!
Connection to vac1-10-tms closed by remote host.
Connection to vac1-10-tms closed.
Date: 2015-09-20 18:14:51 UTC
Sep 20 09:40:01 apm-2-1 apm-1 pktengine[23602]: [W] #SEND-REPLY-FAILED -1 Resource temporarily unavailable
Sep 20 09:40:01 apm-1-0 apm-0 pktengine[3009]: [W] #SEND-REPLY-FAILED -1 Resource temporarily unavailable
Sep 20 09:40:01 apm-2-0 apm-0 pktengine[23610]: [W] #SEND-REPLY-FAILED -1 Resource temporarily unavailable
Sep 20 09:40:01 apm-2-1 apm-1 pktengine[23610]: [W] #SEND-REPLY-FAILED -1 Resource temporarily unavailable
Sep 20 09:40:01 apm-0-1 apm-1 pktengine[23583]: [W] #SEND-REPLY-FAILED -1 Resource temporarily unavailable
Sep 20 09:40:01 apm-0-0 apm-0 pktengine[23608]: [W] #SEND-REPLY-FAILED -1 Resource temporarily unavailable
Sep 20 09:40:01 apm-0-1 apm-1 pktengine[23618]: [W] #SEND-REPLY-FAILED -1 Resource temporarily unavailable
Date: 2015-09-20 18:08:24 UTC
Le routeur Cisco ne voit pas ces DOWN. Ca doit
être un probleme interne à Arbor.
Date: 2015-09-20 18:06:20 UTC
TMS Fault
Appliance: vac1-10-tms
Interface Link 'logical1' is 'Degraded' (Logical port DEGRADED, members: tmsx1.1=ACTIVE tmsx1.3=INACTIVE tmsx1.5=INACTIVE tmsx1.7=ACTIVE) Sep 20 19:58 - 20:00
(0:02) None Show all annotations
2161654 Alert 2161654 Medium TMS Fault
Appliance: vac1-10-tms
Interface Link 'logical1' is 'Degraded' (Logical port DEGRADED, members: tmsx1.1=ACTIVE tmsx1.3=INACTIVE tmsx1.5=INACTIVE tmsx1.7=INACTIVE) Sep 20 19:58 - 20:00
(0:02) None Show all annotations
2161653 Alert 2161653 Medium TMS Fault
Appliance: vac1-10-tms
Interface Link 'logical0' is 'Degraded' (Logical port DEGRADED, members: tmsx1.0=ACTIVE tmsx1.2=INACTIVE tmsx1.4=ACTIVE tmsx1.6=ACTIVE) Sep 20 19:58 - 20:00
(0:02) None Show all annotations
2161652 Alert 2161652 Medium TMS Fault
Appliance: vac1-10-tms
Interface Link 'logical0' is 'Degraded' (Logical port DEGRADED, members: tmsx1.0=INACTIVE tmsx1.2=INACTIVE tmsx1.4=INACTIVE tmsx1.6=ACTIVE) Sep 20 19:58 - 20:00
(0:02) None Show all annotations
2161616 Alert 2161616 Medium TMS Fault
Appliance: vac1-10-tms
Interface Link 'logical1' is 'Degraded' (Logical port DEGRADED, members: tmsx1.1=INACTIVE tmsx1.3=ACTIVE tmsx1.5=ACTIVE tmsx1.7=INACTIVE) Sep 20 19:49 - 19:51
(0:02) None Show all annotations
2161615 Alert 2161615 Medium TMS Fault
Appliance: vac1-10-tms
Interface Link 'logical1' is 'Degraded' (Logical port DEGRADED, members: tmsx1.1=INACTIVE tmsx1.3=ACTIVE tmsx1.5=ACTIVE tmsx1.7=ACTIVE) Sep 20 19:49 - 19:51
(0:02) None Show all annotations
2161614 Alert 2161614 Medium TMS Fault
Appliance: vac1-10-tms
Interface Link 'logical0' is 'Degraded' (Logical port DEGRADED, members: tmsx1.0=INACTIVE tmsx1.2=ACTIVE tmsx1.4=INACTIVE tmsx1.6=INACTIVE) Sep 20 19:49 - 19:51
(0:02) None Show all annotations
2161613 Alert 2161613 Medium TMS Fault
Appliance: vac1-10-tms
Interface Link 'logical0' is 'Degraded' (Logical port DEGRADED, members: tmsx1.0=ACTIVE tmsx1.2=ACTIVE tmsx1.4=ACTIVE tmsx1.6=INACTIVE) Sep 20 19:49 - 19:51
(0:02) None Show all annotations
2161612 Alert 2161612 Medium TMS Fault
Appliance: vac1-10-tms
Interface Link 'logical1' is 'Down' (Logical port INACTIVE, members: tmsx1.1=INACTIVE tmsx1.3=INACTIVE tmsx1.5=INACTIVE tmsx1.7=INACTIVE) Sep 20 19:49 - 19:51
(0:02) None Show all annotations
2161611 Alert 2161611 Medium TMS Fault
Appliance: vac1-10-tms
Interface Link 'logical0' is 'Down' (Logical port INACTIVE, members: tmsx1.0=INACTIVE tmsx1.2=INACTIVE tmsx1.4=INACTIVE tmsx1.6=INACTIVE) Sep 20 19:49 - 19:51
(0:02) None Show all annotations
semble d'avoir un probleme. On vient de le couper.
L'anti-ddos est effectué sur le VAC2 et VAC3.
Update(s):
Date: 2015-09-21 11:46:42 UTC
Tout est rentré dans l'ordre. Nous remettons vac1 en marche.
Date: 2015-09-20 18:44:16 UTC
ça sent pas bon du tout. on est parti pour gerer
une panne hardware avec arbor.
On a fait differentes manipulations, on a retiré les
cartes une par une et ça ne marche toujours pas. A
chaque fois on a fait le reload du chassis. On s'est
appeçu que durant le reload les ports 10G ne passent
pas DOWN, on pense donc que le reload software ne
fait pas le vrai reload du chassis. On a donc arraché
très violament les cables d'alimentation de chassis
(3 cables !!) et on les a remis avec de l'amour dans
le geste. C'est mieux. Le chassis a reinitié toutes
les cartes. On surveille.
Date: 2015-09-20 18:42:00 UTC
Sep 20 18:38:31 vac1-10-tms blinky[30476]: [W] 'get_pktengine_config_version' to 'apm-1-0:8' failed on [Errno 111] Connection refused host:apm-1-0
Sep 20 18:38:31 vac1-10-tms blinky[30476]: [W] 'get_pktengine_config_version' to 'apm-1-0:19' failed on [Errno 111] Connection refused host:apm-1-0
Sep 20 18:38:31 vac1-10-tms blinky[30476]: [W] 'get_pktengine_config_version' to 'apm-0-0:4' failed on [Errno 111] Connection refused host:apm-0-0
Sep 20 18:38:31 vac1-10-tms blinky[30476]: [W] 'get_pktengine_config_version' to 'apm-1-0:15' failed on [Errno 111] Connection refused host:apm-1-0
Sep 20 18:38:31 vac1-10-tms blinky[30476]: [W] 'get_pktengine_config_version' to 'apm-1-0:17' failed on [Errno 111] Connection refused host:apm-1-0
Sep 20 18:38:31 vac1-10-tms blinky[30476]: [W] 'get_pktengine_config_version' to 'apm-0-1:19' failed on [Errno 111] Connection refused host:apm-0-1
Sep 20 18:38:31 vac1-10-tms blinky[30476]: [W] 'get_pktengine_config_version' to 'apm-0-1:20' failed on [Errno 111] Connection refused host:apm-0-1
Sep 20 18:38:31 vac1-10-tms blinky[30476]: [W] 'get_pktengine_config_version' to 'apm-1-0:6' failed on [Errno 111] Connection refused host:apm-1-0
Sep 20 18:38:31 vac1-10-tms blinky[30476]: [W] 'get_pktengine_config_version' to 'apm-0-0:20' failed on [Errno 111] Connection refused host:apm-0-0
Sep 20 18:38:31 vac1-10-tms blinky[30476]: [W] 'get_pktengine_config_version' to 'apm-1-1:0' failed on [Errno 111] Connection refused host:apm-1-1
Sep 20 18:38:31 vac1-10-tms blinky[30476]: [W] 'get_pktengine_config_version' to 'apm-1-1:3' failed on [Errno 111] Connection refused host:apm-1-1
Sep 20 18:38:31 vac1-10-tms blinky[30476]: [W] 'get_pktengine_config_version' to 'apm-0-1:18' failed on [Errno 111] Connection refused host:apm-0-1
Date: 2015-09-20 18:40:21 UTC
admin@vac1-10-tms:/# services tms show
Peakflow TMS state: stopped
admin@vac1-10-tms:/# services tms start
Starting Peakflow TMS services..done.
admin@vac1-10-tms:/#
Date: 2015-09-20 18:38:33 UTC
Sep 20 18:25:55 (none) python[6468]: [S] #SUBHOSTS-REBOOT found 3 apms
Sep 20 18:26:51 (none) python[6468]: [S] #SUBHOSTS-REBOOT verified apm-0-ipmc
Sep 20 18:26:51 (none) python[6468]: [S] #SUBHOSTS-REBOOT verified apm-1-ipmc
Sep 20 18:26:51 (none) python[6468]: [S] #SUBHOSTS-REBOOT verified apm-2-ipmc
Date: 2015-09-20 18:18:40 UTC
on va rebooter le bouzin.
admin@vac1-10-tms:/# services tms stop
Stopping Peakflow TMS services....................................................done.
admin@vac1-10-tms:/# re
now Reload without confirmation
Reload with confirmation
admin@vac1-10-tms:/# re now
094: Rebooting the system..
Broadcast message from root (pts/8) (Sun Sep 20 18:17:35 2015):
The system is going down for reboot NOW!
Connection to vac1-10-tms closed by remote host.
Connection to vac1-10-tms closed.
Date: 2015-09-20 18:14:51 UTC
Sep 20 09:40:01 apm-2-1 apm-1 pktengine[23602]: [W] #SEND-REPLY-FAILED -1 Resource temporarily unavailable
Sep 20 09:40:01 apm-1-0 apm-0 pktengine[3009]: [W] #SEND-REPLY-FAILED -1 Resource temporarily unavailable
Sep 20 09:40:01 apm-2-0 apm-0 pktengine[23610]: [W] #SEND-REPLY-FAILED -1 Resource temporarily unavailable
Sep 20 09:40:01 apm-2-1 apm-1 pktengine[23610]: [W] #SEND-REPLY-FAILED -1 Resource temporarily unavailable
Sep 20 09:40:01 apm-0-1 apm-1 pktengine[23583]: [W] #SEND-REPLY-FAILED -1 Resource temporarily unavailable
Sep 20 09:40:01 apm-0-0 apm-0 pktengine[23608]: [W] #SEND-REPLY-FAILED -1 Resource temporarily unavailable
Sep 20 09:40:01 apm-0-1 apm-1 pktengine[23618]: [W] #SEND-REPLY-FAILED -1 Resource temporarily unavailable
Date: 2015-09-20 18:08:24 UTC
Le routeur Cisco ne voit pas ces DOWN. Ca doit
être un probleme interne à Arbor.
Date: 2015-09-20 18:06:20 UTC
TMS Fault
Appliance: vac1-10-tms
Interface Link 'logical1' is 'Degraded' (Logical port DEGRADED, members: tmsx1.1=ACTIVE tmsx1.3=INACTIVE tmsx1.5=INACTIVE tmsx1.7=ACTIVE) Sep 20 19:58 - 20:00
(0:02) None Show all annotations
2161654 Alert 2161654 Medium TMS Fault
Appliance: vac1-10-tms
Interface Link 'logical1' is 'Degraded' (Logical port DEGRADED, members: tmsx1.1=ACTIVE tmsx1.3=INACTIVE tmsx1.5=INACTIVE tmsx1.7=INACTIVE) Sep 20 19:58 - 20:00
(0:02) None Show all annotations
2161653 Alert 2161653 Medium TMS Fault
Appliance: vac1-10-tms
Interface Link 'logical0' is 'Degraded' (Logical port DEGRADED, members: tmsx1.0=ACTIVE tmsx1.2=INACTIVE tmsx1.4=ACTIVE tmsx1.6=ACTIVE) Sep 20 19:58 - 20:00
(0:02) None Show all annotations
2161652 Alert 2161652 Medium TMS Fault
Appliance: vac1-10-tms
Interface Link 'logical0' is 'Degraded' (Logical port DEGRADED, members: tmsx1.0=INACTIVE tmsx1.2=INACTIVE tmsx1.4=INACTIVE tmsx1.6=ACTIVE) Sep 20 19:58 - 20:00
(0:02) None Show all annotations
2161616 Alert 2161616 Medium TMS Fault
Appliance: vac1-10-tms
Interface Link 'logical1' is 'Degraded' (Logical port DEGRADED, members: tmsx1.1=INACTIVE tmsx1.3=ACTIVE tmsx1.5=ACTIVE tmsx1.7=INACTIVE) Sep 20 19:49 - 19:51
(0:02) None Show all annotations
2161615 Alert 2161615 Medium TMS Fault
Appliance: vac1-10-tms
Interface Link 'logical1' is 'Degraded' (Logical port DEGRADED, members: tmsx1.1=INACTIVE tmsx1.3=ACTIVE tmsx1.5=ACTIVE tmsx1.7=ACTIVE) Sep 20 19:49 - 19:51
(0:02) None Show all annotations
2161614 Alert 2161614 Medium TMS Fault
Appliance: vac1-10-tms
Interface Link 'logical0' is 'Degraded' (Logical port DEGRADED, members: tmsx1.0=INACTIVE tmsx1.2=ACTIVE tmsx1.4=INACTIVE tmsx1.6=INACTIVE) Sep 20 19:49 - 19:51
(0:02) None Show all annotations
2161613 Alert 2161613 Medium TMS Fault
Appliance: vac1-10-tms
Interface Link 'logical0' is 'Degraded' (Logical port DEGRADED, members: tmsx1.0=ACTIVE tmsx1.2=ACTIVE tmsx1.4=ACTIVE tmsx1.6=INACTIVE) Sep 20 19:49 - 19:51
(0:02) None Show all annotations
2161612 Alert 2161612 Medium TMS Fault
Appliance: vac1-10-tms
Interface Link 'logical1' is 'Down' (Logical port INACTIVE, members: tmsx1.1=INACTIVE tmsx1.3=INACTIVE tmsx1.5=INACTIVE tmsx1.7=INACTIVE) Sep 20 19:49 - 19:51
(0:02) None Show all annotations
2161611 Alert 2161611 Medium TMS Fault
Appliance: vac1-10-tms
Interface Link 'logical0' is 'Down' (Logical port INACTIVE, members: tmsx1.0=INACTIVE tmsx1.2=INACTIVE tmsx1.4=INACTIVE tmsx1.6=INACTIVE) Sep 20 19:49 - 19:51
(0:02) None Show all annotations