« The suspense is killing me | Main | It just doesn't stop »

July 17, 2003

A very long day, all around

A lot of Cisco customers have had a very tedious last 24 hours, dealing with one of the most serious security threats in a very long time. I am very tired.

So many things could have been handled so much better.

Cisco pre-released details to "select" customers last Saturday, almost a week ago. With some nasty NDA. Presumably part of the reason for the hush-hush was the seriousness (and probably trivialness) of the vulnerability. As far as I've been able to find out, these customers were not given any details about the nature of the vulnerability, they were only told that it was very serious and given a list of fixed releases.

Questions:

  • how much was Cisco compelled to act the way they did, and how much did they chose to do on their own? What information did they have discretion in releasing, and what information were they directed to (not) release?
  • why were critical details, such as the protocol types, released late the first day (about 15 hours after the first announcement), when had they been released earlier, most customers could have used them to put adequate interim protection in place, instead of having to race to get upgrades done
  • why didn't Cisco circulate the "recommended upgrades" list more widely? This could have been provided to many more customers in advance of the public announcement, without leaking details of the exploit.
  • why now? This was a bug going back more than 10 years. Other than Cisco discovering it internally, are there other reasons for making the time-table so short, and informing so few people before making it public?
  • most importantly, why was/is UEN not considered "critical infrastructure", who makes that determination, and how can we change it?

    UEN provides services to every public safety, public service and state government agency in the state, and roughly 25% of the state's population depends on our network every day. While we may not be "critical" on a national basis, we are undisputably in the top 2 or 3 most important networks in Utah.

    Because UEN was not selected for pre-release information, we were only able to collect rumors and fragments of information until Cisco made the annoucement public. Once the information was made available publically, we were racing as fast to protect our network as the miscreants were to exploit the vulnerability.

    In the first 24 hours, we were only able to upgrade the 15 most critical routers in the state, our PoP sites. That required at least 50 man-hours to accomplish, and had to be done during the day, causing disruption to a lot of distance-ed classes and at times taking whole groups of schools off-line for several hours. Each router upgrade takes about 3 man-hours between the coordination, the actual upgrade, and the clean-up of failed upgrades.

    We also had to spend a significant amount of time simultaneously educating our stakeholders, helping them find appropriate ugprades, supporting them with upgrades, and helping them communicate the right information to the users they support.

    Tomorrow, we start on the next group of routers at district offices, about another 40 or so routers. That will once again disrupt classes and other critical network services (including voice-over-IP inside districts), and will probably result in at least a few districts being off-line for a few hours when upgrades don't install correctly.

    By tomorrow, there will probably be an exploit available, so we may be simultaneously battling routers that are taken down by miscreants or script kiddies.

    And we still have maybe another 200 routers to upgrade, so we will continue into next week.

    Having our entire staff (and a lot of staff at our stakeholders) dedicated to just this effort is really setting back a lot of projects, at a time when we are busier than ever (summer is the busiest season for education technical people).

    Had we gotten pre-release information, we (like other major carriers) could have scheduled a lot of the maintenance during non-peak hours, and could have focused almost exclusively on taking care of our own devices. Then today, we could have spent most of our time supporting and educating our customers at a time when they really needed our help.

    We are fortunate that we were able to move faster than the miscreants have, and that the miscreants haven't exploited this faster than we could do upgrades. But I don't expect that to be a luxury we will have much longer.

    The next major vulnerability like this could easily turn out much worse, with the exploit being actively used against us before we can protect ourselves--unless we can get advanced notice early enough to protect the network before the miscreants get the same information.

    Posted by pete at July 17, 2003 11:08 PM

    Trackback Pings

    TrackBack URL for this entry:

    Comments

    Post a comment




    Remember Me?