« TiVo 3x storage, in an hour | Main | To better customer service (Part 1) »
January 03, 2004
Prevent repeat tech mistakes, with tech
I love it when something technical actually works, especially when it's to keep me from looking stupid (again).
Several months ago, I was informed that I was supposed to be submitting quarterly reports on our Internet2 usage. Since a year ago. The report itself is simple (percentage of traffic used by each type of customer), but collecting the data is pretty rough. Since there's a lot of data to analyze, I set up a script to collect the data every day. Then, I decided to save myself some work by setting up a script that runs once a quarter and emails me the data for the report. Then I wouldn't have to remember to do the report.
January 1 when I checked my email, there was the data. Monday morning I'll put the report together, email it out, and maybe get kudos for having remembered to do it.
I've had a few other similar experiences recently. A few weeks ago our login authentication system was acting erradically. Just figuring out what was wrong was tough, since the problem only showed up sporadically, one out of every 4-20 login attempts. While diagnosing, I wrote a script that would repeatedly make login attempts, and from that we were able to find and resolve the problem (kind of--after lots of packet captures, we rebooted the authentication server and everything worked).
This was the second time in a few months that this problem had happened. So I took that script and set it up to make 100 login attempts three times daily and send out a notification if any failed. I don't think we'll be surprised by this issue again.
There are lots of other examples, of how programming a simple (well, usually simple) script has been a critical part of diagnosing a problem or proactively preventing a problem from happening again. Too bad this isn't a book I'm writing, so I could go on.
I've told many people that network engineers are only half as useful as they could be if they aren't programmers. It's issues like this that remind me how valuable it is to know Perl or another programming language well enough to resolve problems on the fly, and make the network more reliable even when it's a rare problem I'm resolving.
Posted by pete at January 3, 2004 08:59 AM
Trackback Pings
TrackBack URL for this entry:
Comments
I agree with the programming ability a needed asset. It can cause weird situations though. I needed to submit a daily virus report to various people. Since the mail handlers were Unix-based with a command-line virus scanner, it was trivial to write a Perl script to generate daily and monthly reports.
The problem is that my previous employer still thinks I have access to the network (the script signs my name at the bottom of the e-mail).
Posted by: joat at January 4, 2004 06:27 PM