« A very long day, all around | Main | Cisco sets a record »

July 18, 2003

It just doesn't stop

After 48 hours with the Cisco problem, I spent a welcome morning away from it, at the new Weber State University campus in Layton. I met with Ted McGrath and Bill Clark and some other members of the WSU I.T. staff to plan out network connectivity to the new campus. It's a pretty exciting project for WSU and for UEN.

I planned to stop by the office briefly, and then try to catch up on my golf game, since I've only been on the links once this season. I realized quickly that wasn't going to happen.

  • The exploit for the Cisco virus was confirmed to be in the wild, and was rumored to be incorporated into Windows-based worms.
  • Progress on upgrading district offices was going pretty slowly, primarily due to conflicts with firewalls and other minor but frustrating issues.
  • In 60 hours, only about 40 routers out of some 400 we are responsible for had been upgraded, and that was with all of the UEN staff dedicated to the effort.

    I stayed around for the afternoon and worked on upgrading routers, putting a mostly-effective block in place, and cleaning up issues from previous work-around implementations.

    Frustration is mounting and patience is thinning. The network has been in various stages of unstability for 60 hours now. One of our ISPs had an outage for 3 hours due to the Cisco vulnerability. A lot of districts and schools have had multiple outages several times a day for the last two days. Important projects are being delayed, and the solution to this situation just seems to drag out longer and longer. The process of doing IOS upgrades is one of the most tedious and boring ones, and the thought of heading into another week for 8+ hours per day of IOS upgrades ... not exciting.

    But at least we can enjoy the weekend, knowing the most critical parts of the infrastructure have been taken care of.

    Posted by pete at July 18, 2003 8:43 PM