Comments on: Internet Outage today

By: ravishan

ravishan — Wed, 23 Apr 2008 21:19:45 +0000

Steve, First off, I agree with you that we could have done a much better job communicating about the outage to the community. However, the two people who would normally direct the activities including the communication during such a crisis were both out sick at the time the incident occurred. Due to the nature of the problem, both of these staff members had to be involved and were directing the activities from home while the network staff were working to resolve the issue.

Despite the fact that we plan and try to communicate better during crises like this, sometimes it is very hard to do. For example, in this case, once we found out that there was a problem, the directive to the two network administrators was to try to resolve this ASAP because classes are going on and some of them may rely on internet connectivity. These administrators are the source of information about the problem, yet they are involved in calling the vendors (in this case Cisco) and internet provider (CEN) and trying to look at multiple devices, all at once.

It is impossible to get their attention to gather reliable information that we can pass on to the end users. The worst thing we can do is to communicate the wrong information to the end users, so we tend to be very conservative and wait till we have reliable information to let the users know.

It is also usually the case that the ITS staff list first gets some information about the problem before the university wide communication goes out. It was just the combination of events yesterday that made communication to go out late and we will take this as an opportunity to discuss how to better communicate in the event of emergencies such as these in the future.

By: Sarah

Sarah — Wed, 23 Apr 2008 02:57:51 +0000

Out of curiosity, why would anyone attack our network?

By: bro dude

bro dude — Wed, 23 Apr 2008 02:49:03 +0000

whoa. intense.

By: Stephen

Stephen — Wed, 23 Apr 2008 02:11:58 +0000

Thanks for the report, transparency can only help to restore confidence in ITS and the Wesleyan network connections.
However, the report is incomplete and far too late to provide much comfort. The internet difficulties were already apparent by 11:30 in the morning, when downloading a 2MB file took 5 minutes to complete. At 4PM the internet access was not an “outage”, it was a severe slow down, and it only affected access to the outside world (not internal Wesleyan connections).
Finally, and most distressingly, users were left for several hours wondering if the problem was unique to them, wondering if ITS was working to fix the problem,in fact wondering if ITS even had a clue that there was a problem. When we phoned our dedicated ITS support staff, it turns out THEY HAD BEEN TOLD NOTHING, MORE THAN TWO HOURS (!!!!!!!) AFTER THE PHONE CALL TO CEN. This lack of transparency (even within ITS!) shakes our faith in the network and in ITS to the core, because it conveys the impression that ITS does not know about problems that us simpleton users are experiencing. Prompt notification of network problems will give us confidence that our network is not a cobbled together, fragile mess. Please believe that we would all have far more confidence in ITS when we hear about problems than when we don’t hear about problems.