Draconis Software Blog

In IT, many devices have an administrator password that, when forgotten, require some hoop-jumping to reset. Holding down the power button while booting the router/switch/etc, using a secret “back-door” password, or something similar is usually the norm. Not so with a Sony tape library, as this post at Everything Sysadmin shows. Talking to Sony tech support meant that, in order to reset the password on the tape library, it would cost $899 (escalating to Sony customer support dropped the price to $699):

The manual says that one can’t reset the password and I should call my dealer. I figured they just didn’t want the information spread around, so I contacted Sony and they gave me the right people to speak with. A very helpful person names Lucia informed me that I could send in the device and for only $899 they would reset the password. That seemed unreasonable, but escalating it brought me no joy. John Marshall, Customer Service/Support Manager at Sony … was very polite and friendly, but was not able to tell me the secrets to doing the process myself. I even offered to sign a non-disclosure if the process was secret. No luck. He offered to reduce the rate to $699 but that was unstatisfactory.

That’s an expensive password reset. Do you have a crazy password reset story? Let’s hear it!

Fetchmail is one of the core Linux applications that I use consistently — pretty much every email I get is passed through it before getting filtered, forwarded or backed up. It fills an important place in the email chain between the Mail Transfer Agent (or MTA) such as sendmail or qmail, and the Mail Delivery Agent (MDA) such as procmail (there’s also MUA and MSA — the architecture of email seems to demand MxA style acronyms). Fetchmail’s place in this chain is to retrieve emails from a mail server using protocols like POP3 and IMAP, and then deliver them to local or remote users with SMTP.

HowtoForge has a nice tutorial on getting setup with fetchmail. Fetchmail’s robust configuration system allows for a variety of setups. This allows for a number of uses:

1. Consolidating Accounts - It’s common nowadays for a person to have many email addresses, such as work, ISP, school, as well as plenty of free accounts. With fetchmail you can check all of these accounts and forward them to a single address.
2. Usage of text-based email clients - Since fetchmail can also deliver to a local account, it allows you to store all your email on a Linux machine and read it using a client like mutt, pine, elm and others. This means you can check your email from any computer just by SSHing to your machine.
3. Integration with mail filters - It’s easy to have messages be filtered before local delivery or remote forwarding. The Linux filter I use is procmail although there are many others. For example, you may want to filter out emails from a mailing list into a separate boxes. It also allows you to benefit from a spam filter that integrates with the mail delivery agent.

My personal email setup is rather complicated but it fits my needs very well. First emails are retrieved from my various accounts using fetchmail, and delivered to my local account with over Qmail. Procmail then filters out any spam, forwards the remaining emails to a separate email account, and then does further filtering before delivering to local mailbox files. The reason for doing this is it allows my email to be stored and backed up on my Linux machine, and if from any computer to SSH in and read email with mutt. But when I’m on my own computer, I can check the secondary account with Thunderbird, which gives me the benefits of a graphical mail client. It’s these kinds of personalized setups that fetchmail (along with the rest of the MxA family) allows.

I recently saw an article about the DomainKeys Identified Mail (DKIM) draft being accepted by the IETF as an official proposed standard (even though it happened back in February). I really hope the acceptance of this takes off, though the article seemed to show many large companies (who could probably benefit from it) non-committal.

DKIM is a simple means for verifying the origination of an email in an attempt to better track (and fight) spam and phishing messages. The method is simple: the sender encrypts the message body using it’s private key and stores this in the message header (non DKIM receivers, then, can safely ignore it and still deliver the message). A DKIM-enabled receiver looks up the originating domain’s record and extracts the public key. From Wikipedia: “The receiver can then decrypt the hash value in the header field and at the same time recalculate the hash value for the mail body that was received, from the point immediately following the “DomainKey-Signature:” header. If the two values match, this cryptographically proves that the mail did in fact originate at the purported domain, and has not been tampered with in transit.”

(Read the article)

Normally if I’m going to write a script to automate a task via cron, I’ll use Perl (or lately, Ruby). Recently though I needed to use PHP in order to take advantage of common code from a web application. I was expecting this to be an unpleasant experience, using PHP outside of web programming, its raison d’être. However I ended up being pleasantly surprised with the results.

There are a few cases where it may make sense to write scripts in PHP:

1. Code reuse. This was my reason for going with PHP. If a script is related to a web application written in PHP, it’s good practice to avoid reinventing as much functionality as possible. This also makes it easier to write tests directly in PHP.
2. Availability. PHP tends to be widely installed on most machines, especially those with a web server. If you’re not sure what type of system the script may be running on, PHP might be safest bet.
3. Widely known. There are a lot of languages out there nowadays, but PHP tends to a common denominator among developers due to its age and ease of learning. This is useful if the code is being written in a team that doesn’t have the same knowledge base. Even if someone doesn’t know PHP, it’s usually pretty easy to pick up given it’s similarity to languages like C, Java and Perl.

Luckily, PHP makes it very easy to move out of the web area into scripting. Recent versions of PHP include a Command Line Interface (CLI) which allows code to be run outside of a web server. The CLI also provides for parsing of command-line arguments and easy access to stdin/stdout/stderr. I had never used the PHP CLI before but was able to pick it up very quickly.

My foray into PHP scripting was also made a lot easier though various helper functions. PHP includes quite a few functions that can come in handy when writing a script or doing administration. For file I/O, the file_get_contents()/file_put_contents() pair of functions can be used to read or write the contents of a file in a single line of code, and I found them to be particularly useful. PHP’s functions for dealing with the filesystem, image parsing, string manipulation, and database processing are also helpful and may come in handy for the system administrator.

So the next time you need to write a script, consider using PHP, especially if it relates to a web application that’s written in the language. And if the decision to use PHP is forced upon you, don’t despair, as it might be a lot easier than you think.

Additional Reading:
Using PHP from the command line
SysAdmin to SysAdmin: Scripting admin tasks with PHP
System Administration with PHP

Comments (2)

I was working on a project recently that had a number of minor changes between revisions (a client of ours had made minor changes to a project we produced, keeping it under their own version control system), and I needed to sync these changes with a local copy. Since all of these changes were minor, and there were pieces in both versions I wanted to preserve, I would need to do diffs/merges by hand across each of the files (there were about two dozen). Well, diff/edit/close/repeat doesn’t sound like much fun, so I thought a more streamlined approach was necessary.

My favorite command-line editor is vim - no, I don’t mean to start a war here - and I knew I’d be using it quite a bit in merging these changes. In coming up with a streamlined approach, I found a vim plugin that does recursive diffs, splitting the screen up to show differences between each of the files, and allowing for streamlined sync’s in the process. The plugin is called DirDiff, and is written by William Lee, and it proved very useful.

To install, just download the plugin, and place it in your ~/.vim/plugin directory (create it, if you don’t already have one). Once installed, open vim and type :DirDiff [directory1] [directory2]. It found and diff’d each of the files in the directories correctly. It also has a nice feature of sync’ing files, one from the other, based on what you choose. I would have liked more advanced merge capabilities, as I couldn’t take chunks from one file and have them merge into the other, but in most cases, the plugin proved very useful.

Recently we had to deal with some really big log files. One of our clients had an Apache access log several gigs large, and needed to logically separate entries for a new analytics package. The easiest way, of course, is to write a simple Perl (or any other language, such as Ruby) script, iterate through the file line-by-line, and parse each line. The problem here was speed and system resources: opening the file took several minutes on a very fast, SCSI RAID box, and used a huge amount of RAM. We realized there had to be a better way.

(Read the article)

Recently, I needed to upgrade the Subversion client I had installed on my Mac (my personal development machine) to a more recent version. The issue that came up was an incompatibility between my IDE’s Subversion plugin and the command-line version I had previously installed.

When checking out files within the IDE (or even doing updates, for that matter), the sandbox would have its version flags switched, causing my command-line client to no longer work. It got frustrating, as different parts of the sandbox were done in different editors (a Java IDE for one project, a text editor for another, etc), and keeping my sandbox up-to-date was proving tedious.

So, the solution was to update the Subversion client on the command line. Doing so was very easy, though it required building from source (I haven’t been able to find a recent binary installer for the Mac).

(Read the article)

Comments (2)

I’ve generally been a fan of certification programs for systems administrators as a means for providing at least a basic idea of the competency of a potential hire. After reading this article at Linux.com, I’m not particularly surprised to see the number of certifications expected to increase (especially for GNU/Linux programs), though I have to wonder about it: the best sysadmins I’ve known didn’t have a single certification and weren’t particularly interested in getting one.

I see the whole certification process as having two main flaws (as least on the part of companies certifying their own products): (1) little pressure on the part of the certifying company to make the tests difficult or otherwise accurately prove a taker’s skills, and (2) lots of pressure on the part of the company to test the applicant’s knowledge of vendor-specific aspects. It seems to me that it’s in the interest of the certifying company to have lots of certified engineers out there who know the ins-and-outs of that company’s products and little about any competing products.

So, as someone who needs to hire a competent sysadmin, how does this help me? A potential sysadmin who’s certified as an MCSE or RHCE or whatever shows they can take a test and know the basics of that particular company’s product, but what about the millions of other things that sysadmin would be responsible for? Really, it seems these certifications are good for large companies interested in a sysadmin to manage many exact-same boxes and little else. For the majority of employers, especially startups and growing companies, I’d think someone who is more well-rounded in things they’d need to manage is much more useful (for instance, can a sysadmin fill in for a network admin should the need arise? do they understand infrastructure needs to make recommendations for expansion? etc). The best skill a sysadmin can have is the ability to learn as they go and adapt to changes. Having a certification in a particular OS doesn’t particularly help if its a heterogeneous network.

I think certification is a useful process but I’d like to see programs that are more comprehensive, easier to afford, and focus on general skills regardless of environment. What do you think? Have you gone through a certification process and, if so, how has it helped you? In the past, I’ve thought about getting certified myself, but never went through with it primarily for these reasons (as well as the cost).

The O’Reilly Sysadmin Blog has a nice piece on things sysadmins forget to do. Most of them relate to common tasks that fall by the wayside. I think programmers and sysadmins alike have a tendency to prefer interesting, more intellectually challenging activities over things like managing users or root access. The good thing about this is that these kinds of menial tasks can often be automated by programs like cfengine or puppet.

Even for those tasks that cannot be fully automated, such as keeping documentation, there are plenty of applications out there to assist in the process. For example, we’ve found that keeping documentation, notes and fixes in a wiki is a great way to keep information organized.

Personally, one of the tasks I often forget is to keep packages managed and up-to-date. Luckily, this is a lot easier than it used to be with programs like apt-get and yum.

Finally, I think we can all appreciate the reminder to always keep a friendly attitude: “If you want support from management, consider remembering that the user you offend today could wind up on the board of directors. Regardless of that possibility, system administrators should always remember that their clients are internal and if you want to keep your job, be good to your clients.”

Yesterday, we updated our workgroup server to the latest version of Ubuntu - Feisty Fawn - and so far we’ve been happy with it. We’re a little late in updating, though that was primarily us being careful on a production machine. The main reason for updating, actually, was due to an issue with PHP.

We’ve been doing some work in PHP lately that makes use of the GD library to do some interesting graphics manipulations. One of the components of the project made use of a transparency color for an image resource, and it seemed like there may have been a bug in the imagefill function: filling with a color of -1 caused the php processor to segfault. What was odd, was a different version (on my Mac), wasn’t crashing at all. This prompted the update to the latest Ubuntu. The latest version, 5.2.1, solved the problem and we were good to go.

Doing an Update (from Edgy to Feisty):
A distribution upgrade with Ubuntu is cake: simply edit your /etc/apt/sources.list file and replace “edgy” with “feisty”. Next, update the sources list (apt-get update will do it), and finally do a dist-upgrade (apt-get dist-upgrade). Check out this article for more details, as well as how to do a dist-upgrade using the GUI.