« January 2003 | Main | March 2003 »

February 27, 2003

I used to be smart

I started my career just over 10 years ago, as a programmer at the University of Utah Mailing Bureau. Actually, I started there sticking labels to envelopes, and found my way (back) into programming. I'd done a lot of programming as a teenager (even made money off of it), but this was my first job as a programmer.

Like many technical people (and as I worked with technical people at a number of different IT jobs), I soon developed what can best be called a condescending attitude towards anyone who didn't share my interests and speak the same lingo, especially "management." I mean, how could anyone not be totally excited about their new computer and the opportunity to spend a few days or a week trying out all the new software, optimizing and customizing it? The only reason I could think that someone wouldn't know all of the intricacies of something like their email software was that they must be ... stupid. Which of course, fit the techie's perception of "management."

In the same way that teenagers become the adults they always criticized their parents for being, I realized recently that I have become just like the "stupid" people I used to criticize. As much as I'd like to spend the day fiddling with my desktop configuration or integrating some new tool into my email system, I realized a few years ago (as I think many technical people eventually do) that there just isn't enough time to do everything I want to.

I realize that when I talk to other technical people now, and they ask about my email configuration or my desktop set up that I don't have time to fiddle with or optimize, they see me the same way I used to see "management." Somehow, though I still have the same interests (and I'd like to believe the same skill) in what I used to tinker with, not having the time to do it makes me look ... stupid.

Posted by pete at 11:59 PM

My Boring Life ... Is Interesting

At our Brown Bag lunch today (I had late breakfast, so it was just a Brown Bag discussion for me) we had a stream-of-conscience discussion, whatever topic was interesting to the group. We got on the topic of blogs, which is becoming a regular topic of conversation at UEN. We were speculating on what some of the non-bloggers would write about in their blogs. One person mentioned that his life is so boring, he can't imagine reading his own blogs about his life, let along someone else reading them.

When I started my blog over Christmas vacation last year, I think I felt the same. I did have a few things to write about, but I wasn't sure people would be very interested in those, much less postings I would come up with once I drained the list of interesting ones.

I knew so little about blogging then. Or maybe I'm just more delusional now. Though I haven't been religious about posting every day like Jim, I've been surprised at how many people read my blog and more importantly, how many opportunities I have to talk to people about what I've written and what they've been blogging about. I've found that some of the topics I've blogged about that I thought were mundane, but were the only thing I could think of, ended up generating a lot of interest and discussion.

I found that my life wasn't nearly as boring as I thought, and that some of the things I'm interested in that I thought were uninteresting to most people ... well, they are still probably uninteresting to most people, but I found some people who were interested in some of the same things I am.

Jim reminds me that "if you want to meet interesting people, do interesting things and they will want to meet you." I think I am beginning to experience that, maybe just to a small degree.

Posted by pete at 11:42 AM

February 25, 2003

P2P collaboration

Was previously: blah, when I was testing Groove/Userland integration.

Peer-to-peer has been a controversial term almost since it was introduced to the world by Napster. As tens of millions of users learned they could freely share music, movies and software, those industries fought harder and harder against them in what appears to be a losing battle, with the businesses and organizations who own the networks and computers caught in the middle. Education and research organizations have been in the most uncomfortable position, trying to find a balance between enabling a promising new technology and discouraging improper and illegal activities.

There have been a few applications that have begun to demonstrate the power of peer-to-peer communications. Like Napster, these applications mirror the way people deal with each other as peers, or would like to, and make those interactions possible or more efficient. Groove Networks, the latest invention of Lotus Notes creator Ray Ozzie, enables individual users to collaborate the way Notes could only do if an organization supported its use. Groove integrates with other peer-to-peer collaboration tools such as IM, and uses a true peer-to-peer architecture instead of client-server like Notes. I don't know if Groove is going to be a huge success, but I think it's a good demonstration of the useful power of peer-to-peer technology.

Microsoft is poised to launch the next wave of interest in peer-to-peer. The Windows Peer-to-Peer Update and SDK enables developers to create and extend applications through peer-to-peer connectivity. Microsoft's first application built on this technology is Three Degrees, a group-oriented add-on for Messenger that enables shared music-listening (not music sharing, just listening), picture-viewing and a few other things that are more oriented to consumers than businesses. Microsoft is also studying how peer-to-peer applications perform on today's networks. But those functions always find applications for business users (not a year ago, IM was laughed at in the business environment, where today whole companies regularly use it to complement phones and email). It's not a stretch to imagine how Three Degrees fits into a business environment, and the synergy with Groove is too obvious to ignore.

It's exciting to finally have some tangible examples of how peer-to-peer is a viable and useful technology. With products like Groove and companies like Microsoft advocating and enabling peer-to-peer networking, it will be an exciting technology to watch develop.

Posted by pete at 10:27 PM

February 24, 2003

Next steps for Utah Community Internet Exchange

Since launching the Regional Exchange Peering Forum earlier this month, I've had a number of opportunities to talk to regional exchange operators in other parts of the country. I talked with Chris Caputo, the third member of the Seattle IX (SIX) about his experience creating a now quite successful exchange. I have talked with Jere Retzner, who is involved with the NWAX in Portland. I have read and participated in a number of other discussions about regional peering.

I think there are some critical next steps CommIX must take to continue growing:

There's a lot of work to do, but seeing what other regional exchanges are doing and have accomplished gives me hope and a lot of ideas. SIX and NWAX were not much different than CommIX, but have approached things differently than we have and as a result have achieved much more success. I think if we try some different things with CommIX and focus our efforts more efficiently, we will achieve much greater success.

Posted by pete at 10:40 PM

February 23, 2003

Commodity Networks

Jim and I have been talking about the commoditizing of networks for a while now. Last week, we had a conversation about how technologies in general can be commoditized, and what UEN's role might be in commoditizing technologies.

Many years ago, I first read Discipline of Market Leaders. The authors show how successful companies have had to focus their efforts on one of three disciplines: Operational Excellence, Product Leadership, or Customer Intimacy. I found the principles of this book useful in starting two companies. McDonald's is probably the best example of Operational Excellence, where your local mom'n'pop burger stand (assuming they have good food) may be focused more on customer intimacy.

When I started at UEN two years ago, I looked forward to fully exploring the discipline of customer intimacy. UEN has a pretty fixed customer base, so it seemed like the perfect opportunity to get to better understand our customers and how UEN could provide value-added services to them.

Last week, we talked about how UEN might be more valuable commoditizing the technologies than trying to find the ones we can most add value to. This would be an operationally excellent focus, instead of the customer intimacy that I had assumed best fit UEN.

After thinking about it for a while, I think that commoditizing technology fits UEN's mission much better than adding value does. Commoditizing implies wide availability, enabling as many people as possible to use a technology. It also implies reducing costs, both for purchase and use, so they can afford and justify using. These factors are both important to education.

We began last year to commoditize the network with the wide-spread deployment of inexpensive high-speed networks. Other technologies that might be more valuable as a commodity include network management, H.323 video conferencing, network security and voice-over-IP.

Posted by pete at 10:51 PM

February 22, 2003

Too many projects

Mike writes about Development Thrashing. I have been struggling with a similar problem, Project Thrashing, for at least the last year. There are so many things going on, and so few people to work with them, that I started to feel like I spent most of my time just switching between projects, and had little or no time left to actually work on projects.

For at least the last eight years, I have worked this way. It worked fine when I worked 80-90-hour weeks as a business owner and when I was at a start-up. I could do all of the meetings during "normal" business hours, then I did my project work during the "other" hours. This got pretty old for myself and my family. At UEN, I have tried to achieve better balance, and see if it's even possible to fit all of my work into the regular work-week (we are encouraged to work 40-45 hours per week, and managers are supposed to help manage work-loads to that level--though I know a lot of people work a lot more than 45 hours).

Things improved six months ago when Allen and Mike Downie joined the Network Engineering group. Finally I had some other help with the projects on my plate. But there are still a lot of projects that require my involvement, many of them to a very deep level. There are so many projects, and they are all so important and urgent. I find that I average 20-30 hours per week in meetings, planning, coordinating and discussing projects. The remaining hours are so spread out and so sporadic, they are almost useless to achieve any level of focus or momentum (and I spend most of them dealing with other non-project issues). The projects that I need to work on often move forward pretty slowly. Many people at UEN are frustrated about the same thing.

We are experimenting with a different approach, called a "Hot Team". Our first Hot project is a DNS redesign and clean-up, which will begin officially the week of March 3. During 3 weeks, over a 6-week period, a team of 4-8 people will focus half-days and full-days a week at a time on this single project, to create the focus and momentum and excitement to get through one of our oldest languishing projects. We will take breaks after each week, so we can address other individual responsibilities, but during those weeks, all other projects, meetings and responsibilities (within reason) will be set aside to focus on the DNS project.

I am looking forward to this experiment, I hope it will be a better way to get things done as a team. I will blog about it as it happens.

Posted by pete at 12:53 PM

February 19, 2003

Do-it Yourself Networking

I attended a municipal network planning meeting today. On the way back, I was thinking about how as networks become commoditized, it will be easier for organizations who typically don't own or run the network, to do that. This is happening in homes and neighborhoods, with wireless and phone- and power-line networking products. It is also happening in businesses, communities, education and government organizations.

Before an organization can explore the benefits of owning or running its own network, it has to overcome one major hurdle: realizing that networks are commodity, and that it doesn't take a rocket scientist or a goldmine to run your own network. Building your own fiber, or leasing or buying fiber from a carrier is cheaper than ever. If fiber's not right, buy/lease a light wave (lambda). Wireless and airless fiber are inexpensive, fast and reliable. Many organizations will find that using disruptive technologies such as GigE and lambdas with simple network designs enables them to operate (and control) a network very expensively.

David Isenberg points out that networks are driven by the human need to connect and interact with other humans. As I work with various groups to develop networks throughout the state and country, I am constantly amazed at how often collaboration happens. The Internet, as bitterly commercial as it may be, still relies on inter-network cooperation to ensure that anyone anywhere on any network can reach anyone else anywhere on any other network. This motivates or forces cooperation that is ultimately in the interests of consumers and humanity, as much as it might frustrate the profit-making abilities of network providers.

Network cooperation seems to be contagious. Cities and municipalities are cooperating to build big homogenous interconnected broadband networks that cover whole counties and states. Highly-competitive regional ISPs interconnect at regional exchanges to lower costs and improve network performance for their customers. Government organizations who may share little in common are investigating ways to collaborate to reduce network purchase and operational costs. Neighbors are using wireless and fiber to share the costs of bringing broadband connectivity into their homes. All of this because networks are becoming a commodity.

Posted by pete at 9:27 PM

February 17, 2003

Why local and regional peering is important

I have been actively involved with regional and local peering in Utah since the mid-90's. I often am asked (or ask myself, or am told), why is regional peering important? Many people who run large networks argue that regional peering is an interesting hobby for "insignificant" networks to engage in, but all real networks know that only a few peering locations are justifiable.

I beg to differ.

The Internet as we know it today was built around some fundamental assumptions or conditions:

Often people confuse what is with should be. I am often told that the network is the way it is because it should be that way, that it's the best way for it to be. I believe that the network is the way it is because that was the easiest, cheapest and fastest way to provide services that people would buy at the time. That was the first generation, going from 0 to 10mph. Now people have gotten a taste, and they want to see what 30mph might be like.

What have we learned in the last decade, in the first generation of the pervasive network?

When "the other foot falls" and users can get high-speed networks to their homes that enables many high-quality services to be used simultaneously, the network will look very different. Today's applications will use a small fraction of the overall network traffic. The majority of bandwidth will happen between users and other users and service providers in their communities. National-level traffic will be significant, but relatively small compared to local traffic. At the center of it all will be a local/regional traffic exchange, that will interconnect high-speed user networks with each other and with service providers.

A regional traffic exchange is critical to these new kinds of applications:

Did I say that we still don't know what happens when users get cheap, pervasive high-speed bandwidth? It can only make regional peering a more critical part of the network infrastructure.

Is regional peering only important in the future? Though my own experience with high-speed networks shows it'll become much more important as first-mile connectivity improves, regional peering fills some important roles now.

Did I say that I beg to differ about the importance of regional and local exchanges? I've been wrong before, but I'd have to say I've had lots of time to see what's happening and think about it, and I feel more right than ever. More people are starting to say the same things, too. I don't know that regional exchanges will replace national exchanges, but I think they will eventually fill a role just as noticable and critical.

Posted by pete at 6:00 PM

February 16, 2003

What is the Best Network?

David Isenberg is one of my favorite thinkers on the telecommunications industry. I have been reading and thinking about his essays and newsletters for a little over a year now.

I have been thinking about his essay "The Paradox of the Best Network" recently. This is a derivative essay which extends the thinking Isenberg wrote about in "The Rise of the Stupid Network", which caused him to leave AT&T.

The paradox is that everything that intuitively makes the network better, actually makes it worse. The best network, the one that will most successfully meet the users' needs, is the one that is least complicated, with the least number of features and lowest cost, and the greatest flexibility, that simply moves bits.

Communications networks have a more important job than generating return on investment — their value comes from their connectivity and from the services they enable. Therefore, the best network delivers bits in the largest volumes at the fastest speeds. In addition, the best network is the most open to new communications services; it closes off the fewest futures and elicits the most innovation.

The struggle any organization faces, whether it is a carrier or a service provider or enterprise, is that the Best Network interently is a commodity, and "implementing the new commodity network threatens the very basis of their business." This extends beyond the network itself to any network-based technology that becomes commoditized.

Technology support organizations must undergo constant, often painful transitions to let go of services and products that become commodized and focus on those that they can add value to. "To survive, the incumbents must become different businesses. But there’s no guarantee that they'll be the best companies to run the best network." Unfortunately, this is very often too much for an organization to do, and the transition is more brutal (failed company, replaced/outsourced IT staff, etc).

Perhaps the most difficult part of the transition is "loss of control." I believe this is classic "big part of a small pie vs. small part of a big pie", and it's just too tempting to keep the pie you already have and know and like so well. "The best technological network is also the most open political network. The best network is not only simple, low-cost, robust and innovation-friendly, it is also best at promoting a free, democratic, pluralistic, participatory society; a society in which people with new business ideas are free to fail and free to succeed in the marketplace." This is compounded by all-to-common attitudes towards customers/users inherent in most complex industries: we know what's good for them, and heaven forbid they learned enough to be dangerous. It's attitudes like this that open doors for competitors who are relatively more open to the disruptive needs of newly-empowered users.

Isenberg has a list of some of his favorite quotes. Some seem appropriate for this discussion:

"Long range planning does not deal with future decisions, but with the future of present decisions." Peter F. Drucker.

"You have to be willing to learn all over again, to reinvent yourself. You have to be stupid." Christos Cotsakos, CEO, E*Trade, in Inc, December 1997, p. 68.

"The light bulb was not invented by the candle industry looking to improve output." Joab Jackson in Washington Technology.

I paraphrase another, from Jack Welch, former CEO of General Electric: Control your destiny, or someone else will. He doesn't say "or your competitor will", "or your boss will"; he says "or someone else will." Anyone could change your destiny, and in fact there are probably many more people and organizations than you could imagine who actively, either directly or indirectly, are changing your destiny, and the destiny of your organization.

The ultimate paradox of the best network may be that those who most believe they are the best to create the best network, and have the greatest power and control to create the best network, are most unaware of how little control and experience they have to create and operate the best network. There are too many other forces, many from the users themselves, motivating the Rise of the Stupid Network. Only those network operators who embrace the change and capture the momentum to create a network that is "truly open to any and all services that want to use it, no matter how innovative or how unexpected" have a chance at the Best Network.

Posted by pete at 3:18 PM

February 14, 2003

VoIP and ENUM

Several sites are carrying the story that the US Commerce Department will support the ENUM standard, and is advocating that the US government support this standard to integrate the traditional phone network and voice-over-IP networks.ENUM is a technology that uses the Internet Domain Name System to associate phone numbers with VoIP phones, so (with other supporting services and technologies) users of either a traditional phone or a VoIP phone can dial to or from any other phone, whether it is on the phone network or the Internet.

At Next Generation Networks last fall, ENUM support was widely discussed as one of the critical elements to support wide-scale VoIP deployment and aggressive innovation in voice and voice integration applications. Several predictions were made that once ENUM was adopted, VoIP deployment and innovation would rapidly accelerate and VoIP would quickly become the predominant voice technology.

Without ENUM, VoIP systems are islands connected by the traditional phone network. Where VoIP systems do interconnect, the system administrators are required to develop numbering plans and directories to allow users on the individual systems to reach each other. Needless to say, these and other complications have kept VoIP systems at the edge of the phone network, primarily within the private networks maintained by individual organizations.

So, now that ENUM is gaining support, it'll be interesting to watch where VoIP goes, especially outside enterprise and institutional networks where VoIP is mostly deployed now.

Posted by pete at 11:43 AM | Comments (1)

February 13, 2003

QoS: FUD or fundamentally necessary?

Allen writes about IP QoS and FUD (Fear, Uncertainty and Doubt: a sales technique many geeks suspect is used to motivate purchases of less-necessary products/features).

The UEN Network Engineering team has been researching and designing QoS architectures for quite some time now. At least my own experience has led to an unexpected observation: I have yet to find anyone who is successfully using IP QoS in a wide-area network environment to solve actual problems. I've found that the people who are most adamant for QoS are ones who either get paid to sell it or have no actual network operations experience with it (just books, seminars and maybe labs). The people who actually have run QoS in their wide-area networks generally have not been successful and no longer use QoS. I've found many people who run applications that supposedly require QoS to operate successfully, who have deployed no QoS in their networks.

This recent thread on the NANOG mailing list shows the stark contrast between the few people who are operating networks that should benefit from QoS, and those who think in theory networks benefit from QoS. A quote from Bill Woodcock, who has deployed a very wide-scale VoIP network covering most major networks around the world:

QoS is completely unnecessary for VoIP. Doesn't appear to make a bit of
difference. Any relationship between the two is just FUD from people
who've never used VoIP.

We start to lab our QoS design next week, and plan several live-network pilot deployments shortly after that.

One of the things I am most concerned about deploying and operating a QoS-enabled networks is our (and the industry's) lack of visibility into application performance. Because QoS manipulates all applications, whether directly or indirectly, to ensure performance characteristics of specific applications, not being able to see what's happening with applications is a big problem. By deploying QoS, I may get some reassurances that "critical" applications such as voice and video are protected and guaranteed to perform to some level. But at the same time, I may break other "critical" applications, such as intranet/extranet applications that are just as important. If I can't track the performance of all applications (or at least critical ones), I have no way to tell if there even is a problem that QoS can solve it, or whether QoS makes the situation better or worse.

Some of the applications QoS was designed to help apparently have gotten pretty good at helping themselves. Since most of them use UDP, which has no built-in congestion management, these applications tend to stomp on existing network applications, most of which are TCP-based. Another quote from Bill Woodcock:

We've got plenty of the INOC-DBA phones on the ends of congested satellite
tail-circuits, and don't really have significant trouble. As has been
pointed out, the VoIP traffic may be stomping all over TCP traffic on the
same links, but it sounds good.

Looks like we have a lot to learn in the lab and in our network, about the benefits (if any) of QoS and the best ways to operate a QoS-enabled network.

Posted by pete at 7:39 PM

New UEN Blog

Another UEN blog: Rich Finlinson works in the public relations group of UEN, and I've looked forward to his starting a weblog. Hopefully this is the start of a trend that will tip the blog balance away from the geek side of UEN.

Posted by pete at 6:50 PM

NANOG 27 Follow-up

[I'm a few days behind due to my painting experience, travelling and then getting sick. I haven't quite got the balance down to maintain my blog while travelling. Someday.]

I attended NANOG 27 in Phoenix earlier this week. There was a lot of pertinent information and discussion. Understandably, in the wake of the Sapphire worm, there were several presentations and many discussions about network security.

The ISP Security BOF (unfortunately, no presentations are on-line and the session wasn't video-taped) was representative of the mood of this NANOG. Several networks presented the data they captured from the Sapphire worm, and the group discussed the implications of this kind of "flash" attack for network operators. The general concensus seemed to be:

I wrote a report on this NANOG, primarily intended for The Quilt Peering Project, and mostly focused on peering issues of interest to UEN and The Quilt.

I have posted pictures from NANOG and Phoenix. These were primarily intended for people back at UEN and home, respectively, so I hope you enjoy them, too.

Posted by pete at 5:53 PM

February 8, 2003

Painting, Patience and Perfection

I spent the day today painting doors. And enjoyed it.

Probably the most frustrating thing for me is home improvement. I spent most of my childhood working on home improvement projects. When I was 7 or 8, my family lived on a 7-acre farm outside Fairmont, North Dakota, which is a suburb of Wahpeton, which is a suburb of Fargo. I use the word suburb loosely. The farm was kind of a hobby for my dad, who taught (and still teaches) at the community college in Wahpeton. But seven acres provides a lot of opportunities for a 7-year-old to build stuff.

A few years later, we moved into one of the oldest houses in Wahpeton. It was cool to go to the town museum and see our house (at least part of it) on pictures over 100 years old. The house haunted us as our home-improvement project for the next nine years, before we gave up and moved to Fargo. But not before I learned about all there was to know about 2-phase and 3-phase electrical wiring, plumbing, carpentry, roofing, concrete work, framing, drywall and various other skills.

Needless to say, when I left home I looked forward to a life without home improvements. That lasted a year, until we bought our first house near Liberty Park, and my life has been a string of frustrating home-improvement projects since then. With an enjoyable break for the last couple of years. With our current home seven years old, it starts again.

This time, I decided I was going to enjoy it, even though something would happen that would frustrate me. What I hate most about home improvement is that I don't have the practice to do projects the way I'd like them to turn out, and by the time I'm done, I'm pretty far from what it should have looked like, and incredibly frustrated (about the only time I feel a strong urge to let out a string of profanities). But this time was different.

The paint sprayer took two weeks to fix. And then it kept getting clogged, because the last person hadn't cleaned it out. The doors wouldn't stand up very well. The paint wasn't thin enough and kept shooting goobers onto the doors. But I was going to enjoy it, even the mistakes.

Maybe painting is better for my personality. It's much more forgiving to amateurs. Sometimes. Two months ago we painted our master bathroom with some special-technique paint. That took four weeks and four coats to figure out how to do it. But still ended up being enjoyable.

Maybe I'll just stick to painting for a while.

Posted by pete at 10:31 PM

February 7, 2003

Sharing the Burden of Network Security

Bruce Schneier wrote today in the New York Times about the difficulties of relying strictly on patching to secure the network. I mentioned this issue in my follow-up to the Sapphire worm incident a few weeks ago.

I think it would be educational for network administrators to walk in the shoes of system administrators once in a while, to gain an appreciation of what they are asking of sysadmins. One system administrator I talked to explained what was required to close the vulnerability used by Sapphire:

Apparently the Service Pack 3 for MS SQL, which apparently makes this process much easier and more likely to succeed, was released just days after the Sapphire worm hit, and there is some speculation that the timing isn't coincidental.

So when network operators loudly proclaim that the problem would be fixed if system administrators would "just" patch their machines, they probably don't appreciate what they are asking for. Bruce appropriately suggests some of the responsibility lies with the network administrator:

Security professionals are quick to blame system administrators who don't install every patch. This is beginning to feel like blaming the victim. Perhaps such precautions should have been taken, but the real blame lies elsewhere.

Good security is resilient. It is resilient to user errors. It is resilient to network changes. And it is resilient to administrators who are not installing every patch. I have championed monitoring as a way to provide this resilient security. If you are monitoring your network carefully enough, you will catch a hacker regardless of what vulnerability he exploited to gain access. This needs to be done by security experts around the clock; attacks are often too subtle for computers to figure out and can come at any time.

and the software vendors:

In a perfect world, systems would rarely need security patches. The few patches they did need would automatically download, be easy to install, and always work. But network administrators are busy people, and networks are constantly changing. Vigilant monitoring does not "solve'' computer security, but it is a much more realistic way of providing resilient security.

There's a lot of work do be done by network and software vendors, network and system administrators, and policy-makers. Unfortunately, I don't think the necessary changes can be developed and implemented fast enough to avoid another Sapphire incident. There will be more reminders of how vulnerable our technology infrastructure is before we can implement any significant levels of protection.

Posted by pete at 11:13 AM

February 5, 2003

Mitnick revisited

Kevin Mitnick, probably the most notorious (maybe more by media reputation than actual acts) computer hacker, recently completed his probation and is well underway to becoming one of the fore-most (at least well-recognized) authorities on security. He recently agreed to a Slashdot interview, and posted his answers today.

This is a frustrating observation:

Breaking into systems and networks is much easier today than it was a decade ago. ... In today's world, anyone with an Internet connection can obtain "security assessment" tools and/or published proof-of-concept exploit code. This information can be used by an attacker to compromise his or her targets without even knowing how the tool works or the bug is exploited.

Almost a decade after my arrest, computer systems and networks are still being successfully attacked on a daily basis. The saying, "The more things change, the more things stay the same" comes to mind.

And:

As you know, security is not a product that can be purchased off the shelf, but consists of policies, people, processes, and technology. ... Unfortunately, too many organizations are lulled into a false sense of security when they acquire and implement typical security technologies, such as firewalls and antivirus software.

This is a good reminder: "There is no 'patch' for stupidity."

Posted by pete at 10:03 AM

February 3, 2003

Sapphire worm post-mortem

CAIDA has released a preliminary report on the Sapphire worm event of Jan 24. There will be plenty more discussion about this, and I'm sure the cracker community will be quick to remind us how easy it is to cause these problems.

Some very concerning observations about Sapphire:

The ominous conclusion sets out our tasks as IT professionals:

Formerly, small populations (<20,000 machines or less on the Internet) were not viewed as particularly vulnerable to worms, as the probability of finding a susceptible machine in any given scan is quite low. However, a worm which can infect a population of 75,000 hosts in 10 minutes can similarly infect a population of 20,000 hosts in under an hour. Thus, exploits for less popular software present a viable breeding ground for new worms.

Since high-speed worms are no longer simply a theoretical threat, worm defenses need to be automatic; there is no conceivable way for system administrators to respond to threats of this speed. Human-mediated filtering provides no benefit for actually limiting the number of infected machines. While the filtering may mitigate the overhead of the worm's continuing scan traffic, a more sophisticated worm might have stopped scanning once the entire susceptible population was infected, leaving itself dormant on over 75,000 machines to do harm at some future point. Had the worm's propagation lasted only 10 minutes, it would likely take hours or days of effort simply to identify the attack, and many compromised machines could never be identified.

The work to go from Code Red to Sapphire was significant. The effort required to modify Sapphire to be more damaging to the network and to vulnerable machines is incremental and in most cases trivial.

Network and IT infrastructure is so vulnerable right now, I think it unlikely that any significant changes to protect it can happen fast enough. Many of the changes require rethinking and redesigning products, software, and social systems. It is just too appealing and too trivial to not take advantage of the situation. I think we will get to know flash worms really well in the coming months.

Posted by pete at 5:02 PM

Last-mile vs First-mile

"Last-mile" and "first-mile", referencing the frustratingly constrained and obsolete piece of wire between a customer and the phone network, are often used interchangeably.

Came across this quote from Wired today, which gave me a good reason to start using "first-mile" instead of the apparently derogatory "last-mile".

Only a telephone company could have come up with a phrase like "the last mile." It requires the kind of self-importance fostered by years without competition to assume you are at the center of the universe - that the customer represents the last mile, not the first.

This is from the August 1998 issue, where (separately) one of my favorite pundits is quoted on the future of networks. Amazing how many things he hit dead on. If you haven't already read the piece that got him fired from AT&T and follow-ups to it, I'd highly recommend they're worth your time.

Posted by pete at 12:47 PM

golf tips