EU1.AMS core router farm outage at our IP transit provider (Serveruis). Onsite staff working on this issue.
** UPDATE ** Connectivity restored, no BGP flapping or packet loss, all metrics back to normal.
** UPDATE FROM DC NOC ** Service interruption: Some client in SDC2, Meppel had an interruption with a network connection.
Reason: Defective hardware of one of our customers was connected to Serverius network.
When the problem started our monitoring systems didn’t show us any abnormal information like volume or anomaly traffic. Even the Syslog system didn’t give us any clear information about root reason of the problem. To fix the problem we tried to restart Cisco router but after the restart, the same problems came back. Therefore all customer’s connections were disabled on the affected routers and we enabled them one by one until we found the uplinks which caused all problems. After a while, it was found and disconnected.
We are investigating which kind of abnormal traffic was generated, why the CPU on both Cisco routers was affected and how we can avoid this kind of problem in the future.
We apologize for the inconvenience.