Internet Outage today
Apr. 22, 2008 by ravishan
At approximately 1:40 PM today, we experienced a network outage. As is customary, our network staff began diagnosing the problem and made an initial call to our internet connection provider, the Connecticut Education Network (CEN). The CEN Helpdesk informed us that they were not aware of any network issues.
Diagnosing the problem was extremely difficult because our network devices where so overwhelmed with incoming network traffic that our network staff were unable to get access to them. They restarted the devices, only to see them immediately become overwhelmed and unresponsive again. This pointed to a Denial of Service (DOS) attack on our network.
At approximately 2:30 PM, we logged another problem call with CEN, who acknowledged that there may be a network issue specific to our network. Soon thereafter, their engineers determined that one of our servers was being targeted for Denial of Service attack by external hosts. We immediately disconnected that host from the network, though this really does not help stop this type of attack. The attackers were still sending the network traffic targeted at this host, and our internet connection continued to be saturated with network packets that died ONLY after they came into our network.
The prescription for deflecting the attack calls for certain steps (which, for security reasons, we do not wish to elaborate on here) that were followed collaboratively by our staff and CEN staff.
Due to the complex nature of internet traffic routing, it took until 4:45 PM or so to completely deflect the attack.
During this time, several internal services were available and certain others were interrupted a few times. Internet connectivity both from and to Wesleyan was practically nonexistent during this period. Woodframe houses, whose connections to the internet go through Comcast, were not affected aside from being unable to access Wesleyan services during the outage.
We apologize for the inconvenience caused by this outage. We plan to investigate this incident further, work to understand the cause of the problem, and correct it to prevent future attacks like this.
Thanks for the report, transparency can only help to restore confidence in ITS and the Wesleyan network connections.
However, the report is incomplete and far too late to provide much comfort. The internet difficulties were already apparent by 11:30 in the morning, when downloading a 2MB file took 5 minutes to complete. At 4PM the internet access was not an “outage”, it was a severe slow down, and it only affected access to the outside world (not internal Wesleyan connections).
Finally, and most distressingly, users were left for several hours wondering if the problem was unique to them, wondering if ITS was working to fix the problem,in fact wondering if ITS even had a clue that there was a problem. When we phoned our dedicated ITS support staff, it turns out THEY HAD BEEN TOLD NOTHING, MORE THAN TWO HOURS (!!!!!!!) AFTER THE PHONE CALL TO CEN. This lack of transparency (even within ITS!) shakes our faith in the network and in ITS to the core, because it conveys the impression that ITS does not know about problems that us simpleton users are experiencing. Prompt notification of network problems will give us confidence that our network is not a cobbled together, fragile mess. Please believe that we would all have far more confidence in ITS when we hear about problems than when we don’t hear about problems.
whoa. intense.
Out of curiosity, why would anyone attack our network?
Steve, First off, I agree with you that we could have done a much better job communicating about the outage to the community. However, the two people who would normally direct the activities including the communication during such a crisis were both out sick at the time the incident occurred. Due to the nature of the problem, both of these staff members had to be involved and were directing the activities from home while the network staff were working to resolve the issue.
Despite the fact that we plan and try to communicate better during crises like this, sometimes it is very hard to do. For example, in this case, once we found out that there was a problem, the directive to the two network administrators was to try to resolve this ASAP because classes are going on and some of them may rely on internet connectivity. These administrators are the source of information about the problem, yet they are involved in calling the vendors (in this case Cisco) and internet provider (CEN) and trying to look at multiple devices, all at once.
It is impossible to get their attention to gather reliable information that we can pass on to the end users. The worst thing we can do is to communicate the wrong information to the end users, so we tend to be very conservative and wait till we have reliable information to let the users know.
It is also usually the case that the ITS staff list first gets some information about the problem before the university wide communication goes out. It was just the combination of events yesterday that made communication to go out late and we will take this as an opportunity to discuss how to better communicate in the event of emergencies such as these in the future.
Home based business and internet promoted by Jeff Paul marketing are running the grass-root level micro economics. Some times I feel it’s a blessing.
I would like to further stretch that internet marketing can create brand, loyal customers and socialize with customers
The topic discussed here is further discussed on jeffpaulportal.com