« An Internet Exchange in Utah | Main | High-performance network, low-performance hosts »
January 27, 2003
Network Blow-out
By now, most everyone has experienced or heard about the Microsoft SQL Server worm (Slammer, Saphire, etc) that affected computers and especially networks world-wide this past weekend.
UEN and our stakeholders responded pretty well. I was on-line when the attack happened, and Troy and I quickly identified what was happening and put in blocks to limit it's affects on our network, at least from the outside. It looks like the worm didn't have a huge impact on major backbones, but had some devestating effects on local and regional networks. The traffic from compromised machines within networks overwhelmed switches, routers and circuits and made the networks closest to those machines very unstable.
We had a preliminary post-mortem today. I found some of the initial observations quite interesting, and not what people might have thought previously:
- some of the compromised machines had been considered meticulously administrated by very competent sysadmins - the patch that closed this vulnerability seemed to be easy to overlook
- the Microsoft SQL Desktop Edition, included with potentially hundreds of applications including Visio, MSFT Office, and even some games, required a patch specific to the bundled application. Most of these vulnerabilities have no patch available, and the users are unaware they are running SQL Server.
- the attack was so fast (30 minutes or less), it overwhelmed many of the reporting systems that would have notified support staff. Connectivity between monitoring systems and notification systems, which uses the network, was affected by the worm.
- the attack was immediately obvious to people who were on-line at the time, but took automated systems 10-15 minutes to identify. By that time, most vulnerable machines where already compromised and having serious impact on the network
- in-band network management was mostly shut down by this attack. Even out-of-band management was seriously impacted, usually because the network devices were so overwhelmed by the worm traffic.
Some of my own observations about this attack:
- we can't rely on patching as an effective solution to this problem. Vendors need to make patching easier, and users need to be more effective at making sure patches are applied. But we need to protect the network as well, to limit the impact a worm like this can have.
- we need a disaster recovery plan to address this type of event. Though the response was pretty good, it could have been a lot more effective if it had been coordinated and planned in advance.
- the impact of this type of attack needs to be considered in plans for network convergence. I'm particularly concerned about how the coordinated response would have been handled had people been using a VoIP system, which would have been shut down by this worm. Contacting administrators of compromised machines who are on VoIP systems is a particular concern.
- what would have happened if this had occurred at 10am on Monday morning instead of 10pm on Friday night? Most networks were down for at least 4 hours, would that have been less in the day-time? How would a 2-4-hour complete network shut-down have impacted UEN stakeholders?
- this could easily have been used as a diversionary attack. Many network security mechanisms were completely overwhelmed or overcome, opening up the opportunity for other targetted security breaches.
- how can the network be protected from this kind of attack, and how can we identify and respond to it more quickly?
Unfortunately, these solutions and answers to these questions have to be developed quickly. This worm demonstrated an exciting opportunity for hackers, and already the "underground" is developing hybrid worms that will be even more damaging than this one was. There will be a SQLSlammer II in the coming months, how will we be better prepared for it?
Posted by pete at January 27, 2003 10:20 PM
Trackback Pings
TrackBack URL for this entry: