Pete Kruckenberg's Perfect World

« December 2002 | Main | February 2003 »

January 31, 2003

Dufus

I got the UEN Dufus Award for this week. As we like to quote Jim so often, we all reserve the right to make mistakes, but nobody said we can't make a big deal about it when it happens.

The Dufus Award is reserved for people who show exemplary behavior of how UEN shouldn't be. I got this award for getting things done, but not as a team.

It's a semi-humorous and mostly light-hearted way to point out mistakes that will happen on our way to becoming the organization we want to be.

Update: Jim wrote about the Dufus Award in his blog.

Updated update: I brought Krispy Kreme donuts on Monday as penance. All is forgiven.

Posted by pete at 11:35 AM

January 30, 2003

Community Internet Exchange

I am preparing my presention for NANOG next month. I have written a little about CommIX before. I will be part of a panel discussing recent developments at exchange points around the world. I have presented on CommIX at previous NANOG conferences, as well as at various conferences and meetings in Utah.

One of the points I hope to articulate at NANOG is that there is growing need for exchange points that focus on community needs. I realized this shortly after I started at UEN just over two years ago, when the Utah REP was re-branded the Community Internet Exchange. I found in discussions with government, education and business leaders that a local exchange point could meet the needs of more than just local Internet providers.

This is an important difference between CommIX and existing national exchange points. It partially answers the question I'm always asked at NANOG and similar events: why should carriers care about an "insignificant" regional exchange in Utah? The pat and overly simplified answer is: because their customers and regulators in that region care about it.

Posted by pete at 10:21 PM

January 29, 2003

The case for disbelief

Jim wrote Sunday about The Case for Belief. Though belief has many applications and implications, most often religious, inherently we are believing creatures who live and work based on our beliefs, secular and religious.

I'd like to make the case for disbelief (or unbelief), and the importance of disbelief before belief.

As a believer, I have found that often the most difficult barrier is getting others to believe in what I believe. This would be simple if they didn't believe already. The toughest person to convince and engage is one who believes differently than I do, or worse, that what I believe is wrong.

Worse yet is changing my own beliefs and disbeliefs. Many of these are so innate, they influence me subconciously in ways I am not even aware of.

Belief, whether it be religious or secular, in a vision or a person or a concept, is achieved through repeated recognitions of conflicting beliefs and disbeliefs, and replacing those with stronger beliefs. Though there are times when an irrefutable belief is formed quickly, my experience has been that belief is formed and strengthened as an iterative process of self-discovery and increased awareness.

Posted by pete at 10:29 PM

January 28, 2003

High-performance network, low-performance hosts

In the race to put in bigger, faster networks, application performance is usually overlooked or worse, assumed to scale with the network.

The two predominant IP protocols on the Internet are TCP and UDP. TCP is a well-behaved protocol that includes mechanisms ("slow-start" and "back-off") to enable lots of hosts to use the network without stomping on each other. TCP is a reliable transport protocol that ensures applications that everything transmitted will arrive intact at the other end. TCP accounts for 90+% of traffic in most networks. UDP has none of these features, and is used (most successfully) by applications that use a relatively small amount of bandwidth (typically < 10% overall ) and can handle unreliable connectivity.

The algorithms that make TCP behave well were designed for much slower and less-reliable networks than we have today. Similarly, Ethernet was designed for 10Mbps networks. Today's gigabit-per-second networks present some interesting challenges to high-performance applications.

The combination of TCP and Ethernet means that a typical host, in best-case-scenario, can achieve no more than 6.5Mb/s in a single transfer session. That isn't even possible in most cases due to poorly-written applications, operating systems, network drivers and CPU-intensive protocols.

Projects such as Web100, the Internet2 End-to-End Performance Initiative, TCP Tuning and Ethernet Jumbo Frames are focusing on this problem from various directions. Unfortunately, the changes they require take a long time to be implemented in today's networks. Guy Almes addressed this at Westnet, talking about some of the social issues around this issue.

This issue will become of increasing importance to UEN and our stakeholders this year. We are installing GigE and 100Mb circuits throughout our backbone and to many schools. It will be unfortunate if the applications are incapable of utilizing this bandwidth due to TCP, Ethernet and other limitations.

Posted by pete at 10:48 PM

January 27, 2003

Network Blow-out

By now, most everyone has experienced or heard about the Microsoft SQL Server worm (Slammer, Saphire, etc) that affected computers and especially networks world-wide this past weekend.

UEN and our stakeholders responded pretty well. I was on-line when the attack happened, and Troy and I quickly identified what was happening and put in blocks to limit it's affects on our network, at least from the outside. It looks like the worm didn't have a huge impact on major backbones, but had some devestating effects on local and regional networks. The traffic from compromised machines within networks overwhelmed switches, routers and circuits and made the networks closest to those machines very unstable.

We had a preliminary post-mortem today. I found some of the initial observations quite interesting, and not what people might have thought previously:

• some of the compromised machines had been considered meticulously administrated by very competent sysadmins - the patch that closed this vulnerability seemed to be easy to overlook

• the Microsoft SQL Desktop Edition, included with potentially hundreds of applications including Visio, MSFT Office, and even some games, required a patch specific to the bundled application. Most of these vulnerabilities have no patch available, and the users are unaware they are running SQL Server.

• the attack was so fast (30 minutes or less), it overwhelmed many of the reporting systems that would have notified support staff. Connectivity between monitoring systems and notification systems, which uses the network, was affected by the worm.

• the attack was immediately obvious to people who were on-line at the time, but took automated systems 10-15 minutes to identify. By that time, most vulnerable machines where already compromised and having serious impact on the network

• in-band network management was mostly shut down by this attack. Even out-of-band management was seriously impacted, usually because the network devices were so overwhelmed by the worm traffic.

Some of my own observations about this attack:

• we can't rely on patching as an effective solution to this problem. Vendors need to make patching easier, and users need to be more effective at making sure patches are applied. But we need to protect the network as well, to limit the impact a worm like this can have.

• we need a disaster recovery plan to address this type of event. Though the response was pretty good, it could have been a lot more effective if it had been coordinated and planned in advance.

• the impact of this type of attack needs to be considered in plans for network convergence. I'm particularly concerned about how the coordinated response would have been handled had people been using a VoIP system, which would have been shut down by this worm. Contacting administrators of compromised machines who are on VoIP systems is a particular concern.

• what would have happened if this had occurred at 10am on Monday morning instead of 10pm on Friday night? Most networks were down for at least 4 hours, would that have been less in the day-time? How would a 2-4-hour complete network shut-down have impacted UEN stakeholders?

• this could easily have been used as a diversionary attack. Many network security mechanisms were completely overwhelmed or overcome, opening up the opportunity for other targetted security breaches.

• how can the network be protected from this kind of attack, and how can we identify and respond to it more quickly?

Unfortunately, these solutions and answers to these questions have to be developed quickly. This worm demonstrated an exciting opportunity for hackers, and already the "underground" is developing hybrid worms that will be even more damaging than this one was. There will be a SQLSlammer II in the coming months, how will we be better prepared for it?

Posted by pete at 10:20 PM

January 24, 2003

An Internet Exchange in Utah

One of the (perhaps long-term) projects I am working on is creating a national Internet exchange point in Utah. This will be as significant to Utah as being the meet-point for four interstate highways, or the meet-point for the railroads in 1869. Utah's role in the history of national infrastructure has striking similarities to efforts underway at UEN, UofU, Intermountain GigaPOP and other places.

In 1995, a friend and I worked at Internet providers across the hall from each other. Because we bought service from different Internet backbones, the path between our workstations usually went through Virginia or California, because that's where the Internet traffic exchanges were located where Internet backbones interconnected. Even more frustrating, any congestion or outage anywhere along that 5,000-mile-plus path or at the exchanges would affect our traffic, between workstations less than fifty feet apart. Out of sheer frustration, we started the Utah Regional Exchange Point to interconnect our networks locally. Utah REP, now called the Utah Community Internet Exchange (CommIX), interconnects most of the major ISPs in Utah and the surrounding region.

Fast-forward to 2001. I met with Phil Windley, then CIO at the State of Utah, to discuss Utah REP amongst other things. Phil took the idea to Governor Leavitt, who is now probably the only governor who knows what Internet peering is. Phil came back to me with a much bigger idea, taking the same concept to a national level. Phil had found in his economic-development visits around the country that good Internet connectivity, and preferrably a national Internet exchange, are fundamental to network-based economic development.

Utah is in a good position (well, better than some other states) to make this happen. Most of the fiber routes across the Western U.S. meet in Utah. We are also central to the Western U.S., so a lot of the surrounding states are served by telecommunications services that pass through or originate in Utah. This is also (that I'm aware of) the only state-backed project of its kind, which provides a high level of financial security, neutrality and long-term focus.

UEN is actively pursuing the establishment of a national exchange in Utah. Our sponsorship of and participation with CommIX has united local and regional networks in a common interest for better connectivity. The CommIX model we are developing with Utah municipal networks furthers that local momentum. The national peering we began late last year expands our influence to a national level and begins directing national attention towards Utah. Baby steps.

Creating a national Internet exchange in Utah may take decades. I may be long gone (from UEN) by the time it happens. Our active involvement in national and regional networks such as WestNet, National Light Rail, TeraGrid, and Internet2, and Homeland Security projects could significantly accelerate that time-line. It can't get worse for trying, and the efforts I am involved in now meet immediate needs for UEN and also support this long-term vision.

Posted by pete at 10:13 PM

January 23, 2003

Blog Ideas

Many of the people I associate with are starting their own blogs. We are starting to look for other places to use blogs, and find different ways to make them effective. I've talked about other blogs that have been started recently at UEN.

Phil Windley suggested that we create a blog aggregator, a "Best of UEN Blogs" blog that would highlight the blogging activity going on at UEN. This would be a great addition to the blogging effort, and a motivation to get more people to get involved in "quality" blogging. I'm researching RSS aggregation software that would do this.

Other blogging ideas that have been suggested:

• Create a T-forum Weblog to share information between UEN stakeholders -- an extension of the regional, super-regional and state-wide T-forums that are held monthly, quarterly and annually

• Create a Tech Summit Weblog for information sharing regarding the up-coming UEN Tech Summit in May

• Install a Weblog server for use by all UEN employees (blog.uen.org), to make it easy and inexpensive for anyone to start and maintain a Weblog -- MovableType would work great for this

• Use RSS channels and news aggregation to allow stakeholders to easily sort through immense amounts of information (network management, etc) to get the relevant information they need from UEN -- an extension of the NOC Blog and other future UEN blogs

• Conduct building-wide Weblogging training to enable and encourage UEN employees to start their own Weblogs and contribute to the collaborative Best of UEN Blog

• Include Weblog training at the May Tech-Summit to show UEN stakeholders how to use Weblogs for personal publishing as well as collaborative group information-sharing

More ideas will be generated as we experiment with blogging in new situations and find where it is beneficial.

Posted by pete at 10:41 PM

January 22, 2003

Risks and Leverage

I was recently asked a thought-provoking question: "What would you attempt to do if you knew you could not fail?"

On one hand, this can be an inspiring question. I've found many problem-solving and strategic-thinking methodologies use a similar tool to help people and groups get past a perceived obstacle (for example, Who Moved My Cheese suggests that when you are afraid of the future, you think about what you would do if you weren't afraid, and then do it). What would I attempt to do if I knew I could not fail?

There's another side to this question. There's something about the fact that I could fail that makes it worth attempting. If I knew that I couldn't fail, it actually makes the attempt less interesting.

Though it's an interesting and maybe useful mental exercise to imagine away the obstacles that hold me back, the real world invariably seems to package the most interesting things with higher risk. Maybe it's a personality thing, but I like a healthy mix of higher-risk, higher-stakes projects along with the normal mix of things I do. The risks, as well as the results of these projects create their own energy and excitement that make them rewarding to be involved with.

I've also found that many of these projects provide huge benefits for the effort they require. Many do require a lot of effort, but the leverage effect amplifies the benefits from those efforts, when the project succeeds.

Part of what motivates me is my ability to create, find and participate in these higher-risk, high-leverage, high-return projects. This is one of the things I enjoy about UEN. The combination of a large education/research network and an affiliation with University of Utah and other universities in the state, amongst other things, makes for plenty of opportunities to turn a (relatively) small amount of effort and resources into an amazing result that benefits UEN stakeholders and usually many more people in the surrounding communities.

Posted by pete at 10:50 PM

January 21, 2003

Working together

We are starting a project this week to better understand and implement our UEN Shared Vision. In the manner of the master of self-improvement, we are focusing on one point each week. This week, we focus on "We work together to get things done."

My first question is: Why? Is this to make ourselves feel good, to make sure that everyone feels like they are included, that nobody is left out? I think there are places where that motivation is appropriate, but I don't think the work-place is the right place. We have jobs because we must get stuff done.

• Teams get things done better and faster than individuals could
• Teams accomplish things an individual couldn't

If a team isn't providing these benefits to a project and its members, a team approach may be worse than working individually.

• Teams create focus, excitement, energy and motivation

There is a big difference between the natural levels of communication and interaction and contributions that happen in any work environment (a passive team, just a bunch of people who get together), and the momentum and intensity associated with a Team (an active team, that creates and feeds on its own energy). Though naturally many people prefer the control of working individually, everyone prefers to be part of a team that is doing amazing things.

A "Hot Team" as described in Art of Innovation makes a huge difference in how employees perceive an organization, their level of excitement, committment and passion, and the productivity and success of the employees and organization. A Hot Team creates its own energy, excitement, momentum, motivation and success. Everyone wants to be part of a Hot Team. This has a direct positive result on the natural levels of interaction between employees and with customers, and how the organization is perceived by employees and customers.

I wonder if we would be more effective using a team approach on our project lists. The momentum, excitement and energy of small focused teams quickly completing major projects one at a time might be much more productive than easing forward a bit at a time on all projects simultaneously.

Posted by pete at 9:42 PM

Spam Conference

I've contemplated the problem of SPAM email before. (So far, 590 spam messages this month). Last week the first ever Spam Conference was held at MIT. Little surprise, this is a problem that a lot of people are frustrated with, so attendance and coverage was greater than expected.

Paul Graham gave an excellent presentation on the latest algorithms he has developed. A presentation from Matt Sergeant at MessageLabs also is good; they offer anti-virus, anti-porn and anti-spam filtering as a service. Slashdot posted a good summary of the conference.

I still have found my best results from SpamAssassin, DCC and Vipul's Razor. There is a lot of research happening in this area, and it's really making a difference. Of the 590 spam messages I've received through today, less than 20 weren't filtered. And I could get even better performance with some newer SPAM filtering technology. Maybe technology will make Lawrence Lessig's bold bet unnecessary.

Unfortunately, as Paul Graham points out, the spammers will probably just get smarter and bolder to find new ways to get to us.

Posted by pete at 8:47 PM

January 18, 2003

Constant Noise

I had a couple experiences this week where I found out that something that was supposed to have been done (by a vendor) hadn't gotten done. This isn't surprising, it happens all the time with people and vendors, but it is frustrating and embarrassing every time it happens.

It's especially frustrating when I am the one who is responsible internally, but I have to rely on someone external to resolve the problem. When my needs are competing against other customers', it usually means that every couple of months, things boil to an loud, embarrassing moment for us and the vendor, after which we get some results for a while, until things start building to the next boiling point.

This process seems so non-productive. Rather than encourage the vendor and improve our relationship, it seems to make things worse on both sides. Granted, there are always times when relationships need to be "reset", but if the more complicated issues can't get resolved in a positive way, you're both soon going to be looking for easier people to get along with.

Fortunately, most vendors seem to be able to deal with the more common issues that come up in a relationship. It's the oddities, the things that only come up once in a while, but usually are far more significant than the day-to-day problems. How to make sure these issues get attention, get moved along, get the priority you need, and ultimately get resolved to your satisfaction?

I think that a more effective method might be that of " constant noise ". Rather than being loud every few months, would there be a way that I could maintain a subtle, persistent noise level that would keep these issues in the forefront? I hate to nag, is there a more positive way to create this constant noise? And can I keep the noise from blending into the background or being selectively filtered?

One method I am going to try is the running issues list and status meeting. With some vendors this has to be more frequent than others, and I hate to add any more meetings than necessary. But keeping the focus on the issues on a regular basis, and exchanging status information about them in person or electronically, might be an effective way to create constant noise and avoid another quarterly lecture.

Posted by pete at 8:08 PM

What is Blogging?

George Siemens has posted two excellent pieces on The Art of Blogging, including a lot of the history and philosophy of blogging, as well as tips for the beginning blogger. I think these are good enough to make the Left Margin of my blog.

I have waxed poetic about blogs at least once in the recent past. I probably will again. I have been surprised to see how well blogging works, and who participates. Blogs are now driving face-to-face communications, as well as electronic asychronous ones. Several times this week I ran into people I don't talk to often and our discussion centered around something in my blog they had read. I travelled a few days this week, and used the UEN NOC blog to catch up on network events, which I then followed up on using Instant Messenging. Jim has told me about several great experiences related to people getting to know him through his blog.

I have gotten to know so many people through their blogs, better than I could have given the few opportunities we have to spend time together. Even people I spend a lot of time with, I've gotten to know better through their blogs. I try now to get everyone I know (well, most of them) to start and maintain a blog, so I can get to know them better.

One aspect of blogging I have really enjoyed is that it provides a great method for getting things out of my head and onto (virtual) paper on a regular basis. I suspect my blog is far more useful to myself as a thinking tool than it is to those (few) who read it.

Posted by pete at 7:25 PM

Culture of Innovation

Jeremy Zawodny writes about how to destroy a culture of innovation. There's a lot of discussion today about how companies are not innovative, maybe more than there was last decade about how companies were innovative. I suspect in both cases there was/is a lot more talking about theory than practice. But it goes to a question I've been thinking about and reading about recently: how do organizations create a culture of innovation?

We included innovation in our recently-developed UEN shared vision, but how does that go from wishful thinking to reality?

I think most people think they want to work in an innovative environment. Who wants to work for a stagnant lumbering company? Encouraging and embracing innovation requires a carefully-maintained balance of good management, motivated employees, teamwork, collaboration, risk-taking, "thinking outside the box" and other cliches, and a list of other subtle characteristics. And that only prepares an environment where innovation can happen; there is a whole other set of methods and processes and perspectives to enable effective innovation. Creating a culture of innovation is a lot of hard work; that may be the reason that most organizations struggle with consistent, pervasive innovation.

People generally are innovative. People want to improve things around them. Often innovation is thought of as only the big bold new ideas that win fame (and maybe fortune) for the organization and people who worked on them. Those certainly are part of innovation, but the thousands of smaller innovations that come from engaging everyone in innovation are much more important to the success of organization and individuals. Unfortunately, these are the innovations that often are not embraced because they aren't big, bold and exciting.

A culture of pervasive innovation starts with a pervasive attitude of constant improvement. People may be happy, but nobody is satisfied with how things are. Nothing is ever truly finished--only in stages, because in the process of building and using what we create, we are already seeing ways to make it better. The culture, from top down, has to support and encourage and embrace constant questioning, exploration and experimentation.

Posted by pete at 11:41 AM

January 16, 2003

Collaborative Security

I have been thinking about collaborative security the last week or so. This started with a posting on Dave Farber's Interesting People mailing list, about the National Cybersecurity Strategy encouraging the creation of an Cyberspace NOC by Internet operators:

ISPs, hardware and software vendors, IT security-related companies,
computer emergency response teams, and the ISACs,
together, should consider establishing a Cyberspace Network
Operations Center (Cyberspace NOC), physical or virtual, to share
information and ensure coordination to support the health and
reliability of Internet operations in the United States. Although it
would not be a government entity and would be managed by
the private sector, the Federal government should explore ways
in which it could cooperate with the Cyberspace NOC. (p. 13)

I participated in discussion on the NANOG mailing list about this topic, trying to find out if there was anything going on in this area. There are a number of failed efforts, and a few on-going basic efforts, but there seem to be some pretty substantial hurdles preventing any substantive collaboration between network operators.

While I've been at the Westnet conference, I've had a chance to discuss this topic with a number of people here. Today I was invited to give a spontaneous presentation and group discussion on the topic. These conversations have confirmed my initial findings, and also the interest in doing something about the problem.

I think there is an opportunity for education/research networks to explore this issue and demonstrate how collaborative security works. I will be putting together a white paper on this topic to circulate to the Westnet group, proposing that we undertake that effort. This could be particularly interesting because all of the Westnet institutions already share a network, and potentially will share a high-performance network soon.

Posted by pete at 10:47 AM

Abilene / Internet2 Update

I'm at the WestNet conference again today. Guy Almes is presenting a status report on Abilene plans for late 2002 and 2003.

The Abilene backbone is being upgraded with 10Gb/s lambdas, with about 40% of the backbone upgraded in 2002 and the rest this year. These are being connected to the new Juniper T640 routers. Abilene is very pleased with the performance of the new backbone and routers, and with the integration to the 1st generation (OC-48/Cisco) network. Wide-area 10Gb/s is becoming more pervasive, and a common method for building high-performance networks.

Abilene is running IPv6 natively now, with both Cisco and Juniper equipment. Internet2 would like to accelerate native connectivity to GigaPOPs and campuses. They are also peering with all other IPv6 and multicast networks, including commercial networks, to encourage more adoption.

Abilene iniatives for 2003 include several advanced network services: high-performance multicast; native, high-performance IPv6; resiliency, and security. There are also several iniatives to improve measurement of the backbone, one using iperf measurements from each Abilene node.

Guy makes the point that the IGMP snooping requirements placed on Ethernet switch vendors places unreasonable expectations on those products (cheap but sophisticated). IGMP v3 and IPv6 will exacerbate this situation. He suggests a more prudent approach might be to have the routers tell the switches about the multicast information they need. Cisco used this approach (CGMP) in the past when Ethernet switch CPUs couldn't support IGMP snooping, but has since moved more to IGMP snooping.

There's an interesting phenomenon happening in high-speed network development. Users perceive that networks are congested, and planners perceive that the network is underutilized and high-performance networks aren't needed. The source of the problem in both cases is probably under-performing network protocols and applications, packet loss and high latency. The Web100 project is working on improvements to protocols and applications to support high-performance networks. There are a lot of other institutions focusing on this problem, making improvements to IP and TCP to improve their performance.

Posted by pete at 10:01 AM

January 15, 2003

Westnet presentation

I presented at the WestNet conference in Phoenix today, with Jim. Our presentation was titled "Unnatural Contortions of Carriers", and we talked about our experiences and methods over the last two years to get Gigabit Ethernet services from telcos in Utah. Peter O'Neil presented later in the day about fiber-based network efforts happening in other parts of the country, and how other networks are having similar experiences. There are a lot of exciting network projects being proposed and developed by education/research consortiums, which I hope will impact Utah in a big and very good way this year.

I will post our presentation tomorrow when I have a > 31.2kbps link.

Update Our presentation has been posted to the Westnet web site. Powerpoint HTML Notes

Posted by pete at 11:09 PM

January 14, 2003

XML-based Network Management

Information management is probably the most important part of running a network. It's may also be the most complex aspect of running a network. Not only is there a lot of information to collect, correlate, graph, analyze, trend and store, but there are dozens of tools or more for each of those tasks. There are also configurations for each device to be designed, tested, implemented, verified, standardized and archived. There are few similarities between configurations from different vendors, not to mention for devices from the same vendor. Each software revision for the same device can have slightly different configuration syntax.

Getting all of this to work together is a complicated integration problem, without many good solutions. Though many of the communications between network devices and the network management system use standards-based protocols such as SNMP, the rest of the integration problem is left to a hodge-podge of screen-scraping scripts, Perl code, and custom API programming. Like most IT integration projects, few network management implementations successfully realize more than basic functionality because of these issues.

XML could be a powerful solution to many of these problems. Phil Shafer from Juniper Networks circulated a white paper on XML-based Network Management (XNM) in July, 2001. In October 2001, Rob Enns from Juniper presented the topic at NANOG. Juniper has since added XML to their router software.

XML-capable network devices enable intelligent configuration development and implementation, from any XML-enabled configuration manager. Rather than the screen-scraping (unreliable and difficult to maintain) used in almost all configuration managers today, XML creates standard mechanisms for multi-vendor as well as vendor-specific (using XSLT) configurations. XML could also integrate with other label-based configuration systems such as route filter systems, security systems, network element directories and configuration standards templates to assemble and update device-specific configurations. This would be a much more powerful successor to the much-hyped Directory-Enabled Network.

Juniper also proposes the use of XML instead of SNMP. This is also a powerful idea, but SNMP, though more complex, is already a pretty good solution to the data-collection problem. XML would be beneficial for more data-rich data exchange with network devices, but would most likely complement or supplement SNMP rather than replacing it.

XML would also be a great solution to the network-management-integration problem. Especially if those systems could share a single XML database, rather than each system maintaining its own independent database. This would allow for better n-tier NMS architectures, with best-of-breed components to handle each responsibility and reduced load on the network and network operators. This would facilitate development of more XML NMS components, instead of the monolithic NMS applications used today. There doesn't seem to be as much focus in this area, but it would probably be a natural result once routers and switches start speaking XML.

Juniper and other vendors are working to create an XML IETF working group, as part of the IETF Ops Area.

Posted by pete at 10:43 PM

January 13, 2003

Another Weblog (Boston, this time)

My brother started his weblog yesterday. He is a programmer and sysadmin for Tufts University in Boston.

Posted by pete at 11:07 PM

January 12, 2003

Strategic Planning

I was asked his week how I develop strategic plans. I've been involved in a lot of strategic planning with lots of organizations and people, so the methods I use have been developed organically through those experiences. I follow a method, but haven't ever documented it.

Each strategic plan may require modifications to this method, but generally I have found the following to be invaluable to creating a strategic plan:

• Imagine you're already there: this is the most critical step, because it generates much of the information for the plan. If what you are planning had already happened, how would things be different, how would people and processes change, how would it be used, etc. This is an exercise that requires imagination and focus. It's well worth doing, and thinking about, a lot.
• How you got there: once you image you have done what you plan to do, think backwards to today and figure out how you got there. Again, requires imagination and focus and a lot of thinking, but it will provide much of the information you need.
• Leverage points: find the things that allow you to create the greatest amount of momentum towards your destination with the least amount of effort. Use existing resources and processes wherever possible. Find and exploit synergies with other strategic plans.
• Plan for multiple possibilities: envision and plan for a worst case, best case and most likely outcome. This will help you prepare for unexpected circumstances.
• Milestones and turning points: identify the major milestones and decision points to make sure that there are checkpoints to ensure the plan is still on target, and that the desired outcome is still desired. These are also good times to update the plan, as your goal becomes more visible.
• Use straw-man iterative process: I typically use a method of developing a broad initial draft with a few people, and presenting that to a slightly larger group for feedback based on a gap analysis. Committees are too unwieldly to start a plan from scratch, but they are invaluable in the gap analysis and ultimate support of the plan. As the plan gets refined, it is presented to ever-larger groups.

These are a few of the major events in the development of a strategic plan. I spend a lot of my time planning, and refining my methods.

Posted by pete at 12:28 AM

January 11, 2003

Fantasy Blog

Phil Windley discovered one of the Weblogs I mentioned but didn't link to (I thought it was internal only, but Troy had a link to it). I didn't realize a NOC blog was one of Phil's fantasies, it just seemed like a good idea at the time Troy and I decided to set it up. It has proven very useful--and well-used--in the few days it's been running, and I hope to see it developed further. Phil has some good ideas to make it more useful.

Posted by pete at 11:22 PM

Why I Work at UEN

As I've thought about "What Should I Do With My Life?", I have thought a lot about my current job, and why I work at UEN.

I've been associated with UEN since the early '90's, when I bought Internet service from WestNet while at DSW. In 1996, UEN joined the Utah REP, a regional exchange point I co-founded with another Utah ISP. The next year, UEN became the sponsor for the REP, and I continued to work with UEN as part of that project even after I left the ISP community. Through knowing UEN, and particularly knowing that they have the closest thing to a big, complex network in Utah, I jumped at the chance to take a 30% paycut to work for UEN in 2001.

I'm close to my two-year anniversary. In spite of the market, which makes it easy to stay at a good job, my thoughts of leaving for a new job are at an all-time low. In spite of a challenging political and usually beaurocratic environment, I've found plenty to like about what I do:

• The Kids: never have I worked at a job that had such a good reason for existing. I've worked for plenty of companies that were self-deluded into thinking that somehow they made a difference and would change the world. We do change the world of tens of thousands of people in the most meaningful way, by providing educational opportunities to people who otherwise wouldn't get them.
• The Customers: for many companies, customer interaction is strictly a tool to close sales. Post-sales customer interaction is a cost of doing business, to be kept as low as possible. UEN is one of only a few places where I've gotten to explore and develop strong relationships with customers, and found how important and beneficial those relationships are to a successful organization.
• The Technology: in spite of tight dollars, the things we get to do with technology, and how we use techology, are incredibly interesting and challenging.
• The Environment: having worked only in the private sector, I'm still adjusting to the public sector. There are advantages and disadvantages to either, and I imagine it's equally frustrating to change from one to the other. The academic/research environment at the UofU and other customers supports an attitude of exploration and experimentation.
• The Opportunities: I've found that my entrepreneurial urges are just as satisfied in the public sector as they were in the private sector. The leverage of Education, the State of Utah, and other partners we work with is much more significant than any company I've worked with. I've been able to participate in or start initiatives such as Utah National Internet Exchange, Intermountain GigaPOP, Utah Network Research Center, Community Internet Exchange, and Utah Valley Community Network.
• The Education: I have learned a lot in the two years I've been at UEN. Inevitably, one has to learn patience and politics to get things done. I've learned a lot more about how people work together and what motivates people to change. And I've definitely learned a lot about what it takes organizationally and technically to engineer and operate a growing, complex network.

I look forward to the next year at UEN. We have accomplished a lot of things, and the next year will be even more interesting.

Posted by pete at 4:37 PM

Pervasive weblogs

I've been surprised to find out how many situations are a perfect use for a weblog, and how many people are interested in weblogging themselves.

My boss (Jim Stewart) had been talking for a long time about putting together a Director's Page. After my experience with my own weblog, I convinced him to start his own, which I think will eventually turn into the Director's Page. He's posting pretty much every day.

The security guy at UEN has always had a need for publishing lots of information, about his interests in security and neighborhood networking. Another perfect solution for weblogging. He posts regularly, usually daily.

We had an internal need at UEN to communicate network status information amongst a diverse group of people, and found that a weblog was the perfect solution. Now about a dozen people are posting regular daily and hourly updates on network issues.

My brother and I manage our family website, and we've always struggled with keeping the content interesting and current. A weblog is probably right for this, too.

Talking to my brother, he has been looking for something like this for a while. So I set up MovableType, which will accomodate multiple users managing multiple logs in different roles. This will support each person having multiple weblogs, as well as a single weblog for the family website.

This has been an interesting 2 weeks, since I started in earnest keeping a weblog, and promoting it to other people. I've found it interesting how much people are interested in what I write, as well as writing their own material.

Posted by pete at 12:08 AM

January 10, 2003

Value of Observance

I've started reading Art of Innovation. This book is written by the brother of the founder of Ideo, the leading design firm in America and one of the best in the world.

The first part of the book talks about how important it is to observe people actually using the product, as part of the design, instead of assuming or calculating how they will in theory use it.

I experienced this at a previous job. We had a team of some of the best systems management tools people design the best systems management systems they could think of, pretty much without restriction on cost or time. They took nine months and many million dollars and consulted with some of the best and most experienced people in the industry.

When they deployed the system to monitor the our data center, I spent a few days a week in the NOC, watching how people used the system. What I found was that almost every feature that was supposed to make this system revolutionary and worth tens of thousands per month to a customer, was either useless or worse, got in the way of knowing what was happening with the systems. Notifications went through so much processing, they typically took several minutes to show up. (The NOC eventually implemented a simple $199 off-the-shelf tool to deal with the delays, because it could identify and report an even in a matter of seconds.)

As I watched the way the NOC used the tool, I saw that what they found most valueable was the event console, the basic component of the system management software, which was the free give-away part of the system. Everything else was so complicated to use, it just got in the way.

I have tried to remember this lesson as I design systems. The ultimate measure of success is that the user can solve a problem or create a benefit with what I build, and that they find the product easy and convenient to use.

Posted by pete at 12:03 AM

January 8, 2003

Another Utah Weblog

Troy Jessup is the security guru at UEN and a good friend. Besides security, he has a personal interest in neighborhood networking and has a pretty successful one running in his neighborhood using fiber and wireless.

Posted by pete at 8:03 AM | Comments (1)

January 7, 2003

The next major disruptions in network technology

I spent part of the afternoon with a Fellow from the Monterey Naval Postgraduate Institute, talking about some of the projects he works on for secure, reliable networks. He has built networks for military applications for many years.

I'm not sure how much I can talk about from the discussion, but as we talked I realized that there will eventually be two changes that will fundamentally change how the Internet (and other networks, too) is structured. These are probably a few of many disruptive network requirements, and may not be the most significant distruptions (though I think they will rank high).

• The need for something other than best-effort. All Internet applications are designed for best-effort networks. IP has rudementary Quality of Service capabilities when compared with protocols such as ATM. IP may be able to guarantee that data will get transmitted, but it cannot guarantee when or how long it will take to get there. The Internet has been created around a best-effort paradigm, so when that isn't good enough for a substantial number of people and applications, many fundamental (though perhaps subtle) changes will have to occur in how Internet networks are designed and operated, and in the technology used in Internet networks.

• Broadband to the home and business. The Internet backbone is designed to meet the requirements of tens of millions of dial-up users (and a few broadband) accessing services which are mostly housed in a (relatively) few centralized data centers. The majority of Internet capacity is contained along a few fiber routes used for ISP backbones. As broadband is extended to the home, with millions of users connected at 100-1000x the bandwidth, and businesses with connections that are capable of hosting their intra- and extra-net services, the paradigm will shift significantly. Though the national backbone will still exist and be fundamental to the network, local and regional backbones and traffic exchanges will become just as important to accomodate localized services.

Several other ideas have come out of recent discussions like this, and I will be sharing those over the next few days.

Posted by pete at 11:59 PM

Individuals and Organizations

I have participated the last several months with the management team and the staff at UEN in what I think is a fairly uncommon process in most organizations. We have started practicing some of the principles of The Fifth Discipline, including dialoguing, developing shared vision, and recognizing and changing mental models. So far, some 1500 (wo)man-hours have been invested in the process, 1000 of those by the management team (it was 94 hours, but lots of people in the meetings), to develop a Shared Vision, Mission Statement, and Department Roles and Responsibilities.

Today we had an all-staff meeting where the results of the first round were presented and discussed. Each manager was asked to share his thoughts on the process, so I decided to share those in my post today (I spoke for 10 minutes, so these are just some of the highlights).

• Born of frustration: Many of the best companies achieved success because things got so bad (usually financially) that people were willing to take huge risks to try something else to see if it would work. Many people who have worked at UEN, myself included, have been so frustrated they have left, and those who stayed just endured. We chose to try something different. I hope that it will be the risk that makes UEN an amazing place to work.
• I came to UEN to get stuff done. One of my shortcomings is that I have a hard time being patient, and I want to get stuff done. I found out during this process that most UEN employees feel the same way. Apparently, we are all trying to get stuff done, and we all get frustrated that it can't get done faster. I have also started to better understand the value and importance of patience, and how necessary it is to allow people to adjust to the significant changes required to make UEN what it needs to be.
• I want to work for an organization that does amazing things. Part of how I perceive myself is how people think of the organization I work for. I don't want to work for an organization that is considered a dufus. I found out that everyone else feels the same way, and in fact many people came to UEN and have stayed at UEN specifically because of what we do. We want to create an organization that is recognized for doing amazing things.
• An organization does not exist independently of the people who work for it. It is a collective of individuals. What the organization is is what the individuals are who make it up. The ideas, values, opinions, attitudes and contributions of many individuals create an organization that exactly represents the sum of those individual attributes. We are the organization.
• Effective communication and building of ideas and vision are the basis to a group of individuals creating an effective organization. And getting stuff done, together.
• Real changes in people and organizations happen slowly and subtly over (long periods of) time. It's often difficult to measure them or perceive that they are happening. One way is to look back, and ask yourself if what you are doing today would have been possible for you as a person or organization to have done a year ago. I can see tremendous change in UEN looking at what we are able to do today (especially *today*) as an organization, that would have been impossible a year ago.
• Since it's hard to see change happening, it's difficult to believe it can happen. Belief has significant influence on your attitude towards change. If you are to contribute to and engage in the change, you have to believe that change is possible and that it is happening and that your participation will make it happen.
• We are the ones that make change happen. You are the one that makes change happen. It's not just the managers or the director that will change UEN into what we want it to be, it is each of us individually, who commit to the shared vision and mission. It is by our actions, by what we choose to do or not to, and how we act, that UEN will become what we want. There is nobody else who can do it.
• What you do, what I do, what we do, is what makes UEN what it is.

I am excited about the future we will create for ourselves and the organizations served by UEN. It has been a rewarding, if sometimes arduous, process, but looks to be well worth the effort.

Posted by pete at 11:41 PM

January 6, 2003

IPv6

There is quite a bit of IPv6 research happening on campus, and I recently got involved with using IPv6 at home. It reminds me a lot of the early days of Linux, when few people had heard about it, and even fewer had tried it.

Before getting IPv6-enabled, I (like most networkers in the US) thought IPv6 was a wait-and-see technology, that might be implemented in the next 20-30 years. I participated in an IPv6 training session put on by Internet2 in July, 2002, and had my first glimpse of how much is already happening with IPv6. Shortly afterwards I connected my home network to the IPv6 network. Now when I go to http://www.kame.net/ the turtle dances.

Since the US (and Europe) has most of the IPv4 address space, we will probably be the last ones to move to IPv6. Maybe it'll be like the metric system and we will run IPv4 for decades after everyone else is on IPv6. The rest of the world can't get IPv4 address space, so they have been very aggressive about implementing IPv6, especially in Asia.

Though it's probably not much more difficult to understand than IPv4 (most IPv4 concepts are re-used, usually better, in IPv6), I think the subtle differences will make the transition to IPv6 very complicated. Part of the difficulty will be that a lot of people who run and use networks know what to do with IPv4, but they don't necessarily know how or why it works. There will also be a lot of changes required to go from decimal addresses that you can remember and recite, to addresses that are hexadecimal and much longer.

I am working on plans to deploy IPv6 within the UEN backbone, hopefully this summer. My goal is that by the end of 2004 we would have removed all non-IP protocols from the UEN backbone (yes, we do still have some in aggregation routers) and hopefully from most of the networks connected to the backbone. As we approach 2006, my goal is to have IPv6 deployed pervasively through the network, and in active use by users, sysadmins, and network adminstrators. I will share the specifics of the plan here as I develop them over the next several weeks.

Posted by pete at 10:07 AM

January 5, 2003

What Should I Do With My Life?

New Years is traditionally the time that we think about where we are and where we'd like to go next year. Po Bronson has a timely essay in Fast Company, entitled "What Should I Do With My Life?", based on his book by the same name.

Too often I think we get caught up in playing The Game, and forget what it is we really want out of life. Those of us who worked (or started) in IT through the 90's experienced the frantic pace that culminated in the dot.com bubble. Many of us are still trying to figure out how the "real" world works. This essay gives some wise advise on the questions we should be asking ourselves, and where we should (and shouldn't) be looking for answers.

I enjoyed reading Jeremy Zawodny's personal response to this article.

The questions Mr. Bronson poses are not answered quickly or easily. I have often thought about them, and more often in the last few years. I started my professional career in the early 90's, just when IT became the "hot" industry. Many of my own perspectives and values about work were formed by the dot.com environment. As much as people talked about changing the world, the underlying motivation for pretty much everyone was money. Companies who had it had employees who would work to burn-out to get it, and companies who ran out of it died a quick death as employees jumped to the next funded startup opportunity (somehow a "pre-IPO" job offer just isn't as appealing as it was two years ago).

Those were the days. Even more, UEN has been a dramatic change from previous jobs, and has provoked a lot of thoughts about what I want to become and what I want to contribute to the community and industry. A lot of the artificial motivations that are prevalent in the corporate environment aren't at UEN, though there are plenty of frustrating attributes of a government job.

Since I started at UEN, I have accomplished some things I have wanted to for a long time. I leave the office most days by 5:30, and I spend a lot more time with my family. Strangely, I found that working a more rigorous schedule actually makes me more productive. I have found time to develop hobbies, and found that a balance of non-work things really helps me be more effective at work, and enjoy it more. Weird.

This year I expect to find new, deeper answers to the question, What Should I Do With My Life?. I will share what I find.

Posted by pete at 7:56 PM

January 4, 2003

Another Utah Weblog

Jim Stewart, UEN Technical Services Director and a good friend, started his weblog this week. He has some interesting perspectives on technology and life.

Posted by pete at 10:26 PM

January 2, 2003

Power of many

The human mind has a very difficult time comprehending big numbers. It's easy to visualize 100 or 1,000 or 10,000, but around 100,000 or 1,000,000 we have a hard time comprehending, and between 10,000,000 and 1,000,000,000 most of us start thinking in infinite terms (using comparisons like "you'd have to spend $115,000 per day every day of the year to spend $1 billion in a year"). Some people (like astronomers) might be better at comprehending big numbers.

One of the greatest powers, that is frequently underestimated, is the power of a lot of people doing the same thing or feeling the same way about something. I'm always amazed at how products that sell for under $5 generate billions in annual revenue for the companies that make them.

Somewhere around 650 million users are on the Internet now. That's a lot of people. The Internet has enabled those people to get together in ways that simply were not possible just 10 years ago. The combination of communications, applications and information enables people to find and participate in interest groups larger and more powerful than ever before.

Consider that a relatively small handful, around 1.5 million active Napster users, created the biggest threat to the music industry in its history. Napster did very little (if any) advertising, no telemarketing, no door-to-door sales. Ultimately what Napster users started will fundamentally change the entertainment industries, probably for the benefit of consumers. There are lots of other examples, especially with peer-to-peer technologies.

Spam (unsolicited junk email) has become quite a nuisance over the last few years, and looks to get even worse. In December, I received over 900 spam messages, an average of 30 per day. Not even accounting for the time it takes to delete those messages, they are a big security risk and some are quite embarrassing.

A lot of effort has gone into stopping spam, through technology, the legal system and many other methods. Most of these have focused on trying to classify a message based on the words it uses. Generally, I have found them to be difficult to maintain and mostly ineffective, and they will often miss spam messages and classify legitimate messages as spam.

I have been experimenting recently with some very powerful collaborative spam tools. These tools use the power of a few hundred thousand people (their mailboxes to be more specific) to quickly catalog new spam messages. As each message is reported (usually in real-time), a server tallies how many times it has been reported by other users and analyzes the message to assign it a spam score. This collaborative effort results in a spam filter that is 99% accurate. Of the 900 spam messages I received last month, only a few weren't properly identified as spam, and no legitimate messages were identified as spam. There is also an Outlook plug-in that works in the same way.

Some of the most interesting Internet applications are those (such as Weblogs) that leverage the power of the masses of Internet users. Spam blocking might not be the most interesting one, but it is one example of how simple a solution can be when it distributes the workload. This is one area of technology (and social) development I think will be exciting to watch in the future.

Posted by pete at 12:16 AM

January 1, 2003

Happy New Year

Happy New Year to you.

I have been able to spend a lot of time with family this holiday. One addition to the family get-together this year was including my brother's family from Boston via video conference. They were able to spend time with us Christmas Eve, open presents with us Christmas morning, and join in various family get-togethers. And it didn't feel very geeky, once everyone got used to it.

This was done with $85 worth of new technology (not counting the laptop, wireless network and broadband Internet link already in place), a laptop webcam from Logitech. I found that Windows/MSN Messenger has the best mix of usability, video and audio quality (the echo cancellation works very well), but ICUII has much better (and larger) video quality. The best results seemed to be from using Messenger for audio and ICUII for video. I'd also recommend this headset for personal video calls.

I have also experimented with Netmeeting and H.323 video, though the video quality and features of Netmeeting leave a lot to be desired (it hasn't been updated since Windows 98). In spite if that, I have been able to have some decent H.323 calls with Netmeeting to Polycom video stations. I will be trying out a higher-quality camera and H.323 software from Polycom in a few weeks; hopefully that will be a good H.323 solution.

I've taken some time over the holidays thinking and planning for 2003, and remembering the good things about 2002. I hope you have taken time to do the same.

Posted by pete at 5:46 PM