The following is what I came up with. It’s a ruby script that uses the delightful Hpricot gem to parse the xml. It takes one argument, the subversion username that you wish to retrieve the logs for. I hope that it’s useful for someone else! You can curl it from http://pastie.textmate.org/197763.txt if that makes it easier, too.

[ruby]
#!/usr/bin/ruby
require ‘rubygems’
require ‘hpricot’

username = ARGV[0]
if username.nil? || username == “”
puts “Please specify the username to cull log entries for.”
exit
end

puts “Requesting SVN log, this may take a bit.”

doc = IO.popen(“svn log –xml”) do |f|
Hpricot.XML(f)
end

entries = doc.search(“logentry”).find_all do |entry|
(entry/”author”).innerHTML == username
end

entries.each do |entry|
revision = entry.attributes["revision"]
author = (entry/”author”).innerHTML
date = (entry/”date”).innerHTML
msg = (entry/”msg”).innerHTML

puts “r#{revision} – #{author}”
puts “#{date}”
puts “#{msg}”
puts “-”*80
end
[/ruby]

/Developer/Tools/SetFile -a V /Volumes/Untitled

Replace the “Untitled” with whatever the volume name is listed as in /Volumes. Note also, that you will need to have Apple’s Developer Tools installed in order for this to work. It’s free, but a fairly large download. Hope you find this useful!

In addition, there’s a new wireless stack, a new Firewire stack (very cool), and a slew of new drivers and other changes. Check out details about this release here, then download the kernel. Enjoy!

BugZilla 3.0 stable was recently released

Second, grab a copy of the new BugZilla. As I’m installing on a Ubuntu box, I used the tarball, but there are different versions available for other operating systems (including, apparently, an unsupported automated installer, which I haven’t tried). Once you’ve downloaded it, move it to your web directory, unpack it, and name it something like bugzilla-3.0.

Third, copy the important stuff out of the old BugZilla (from the Upgrading to New Releases documentation):

bash$ cd bugzilla-3.0
bash$ cp -p ../bugzilla/localconfig* .
bash$ cp -pr ../bugzilla/data .

Note the “bugzilla” directory is your old BugZilla installation, while the bugzilla-3.0 is the new installation you just downloaded.

Fourth, run the “checksetup.pl” script in the bugzilla-3.0 directory. This script will go through your environment and alert you to any missing Perl modules, and update your database schema to the new version for BugZilla 3.0. I had to install several new Perl modules in order for the new version to work, which can be done using the command “perl -MCPAN -e ‘install [some::module]‘”, or you can just download the packages individually from CPAN and build/install them by hand.

One additional note about the checksetup.pl script: the new version of BugZilla is using a new data encoding for the database, which requires updating any existing data currently in your database to the new version (UTF-8, I believe). To do this, I needed to run the command “./contrib/recode.pl –guess”, which figured out my existing encoding and updated it to the new UTF-8 format. Check out the options for recode.pl (and consider trying the –dry-run flag first).

After running recode.pl, I needed to rerun my checksetup.pl to finish updating the database to the new version. Once done, my BugZilla installation was nearly complete. When going to the new BugZilla installation via my web browser, I noticed no images were appearing. It turned out to be a permissions problem. Make sure the owner/group and access rights are set properly for your BugZilla installation based on how you’re running Apache.

And that’s it! All my existing bug data appeared in the new version of BugZilla and I was ready to go.

In a world where spam is bound to hold such an important position,
methods of preventing it should also be given an increasing importance.
Some of the easiest and most widely used prevention methods are host
control solutions, Antispam applications and user education.

I’m particularly excited about Domain Keys Identified Mail (DKIM), which I wrote about recently. Such a method would be very useful in tracking down spammers and controlling the flood of junk they put out, though, as this article points out, it’s not without it’s limitations. “The main disadvantage of DKIM is that email messages can be significantly modified in certain situations (e.g. when being forwarded by list severs), causing the signature to be invalidated and the message to be rejected. A solution to this issue would be combining DomainKeys with SPF, because SPF is immune to modifications of the email data.”

Check the article out – it’s a good overview of what’s currently going on, and perhaps may inspire you to try a different approach to your current antispam efforts.

The manual says that one can’t reset the password and I should call my dealer. I figured they just didn’t want the information spread around, so I contacted Sony and they gave me the right people to speak with. A very helpful person names Lucia informed me that I could send in the device and for only $899 they would reset the password. That seemed unreasonable, but escalating it brought me no joy. John Marshall, Customer Service/Support Manager at Sony … was very polite and friendly, but was not able to tell me the secrets to doing the process myself. I even offered to sign a non-disclosure if the process was secret. No luck. He offered to reduce the rate to $699 but that was unstatisfactory.

That’s an expensive password reset. Do you have a crazy password reset story? Let’s hear it!

HowtoForge has a nice tutorial on getting setup with fetchmail. Fetchmail’s robust configuration system allows for a variety of setups. This allows for a number of uses:

1. Consolidating Accounts – It’s common nowadays for a person to have many email addresses, such as work, ISP, school, as well as plenty of free accounts. With fetchmail you can check all of these accounts and forward them to a single address.
2. Usage of text-based email clients – Since fetchmail can also deliver to a local account, it allows you to store all your email on a Linux machine and read it using a client like mutt, pine, elm and others. This means you can check your email from any computer just by SSHing to your machine.
3. Integration with mail filters – It’s easy to have messages be filtered before local delivery or remote forwarding. The Linux filter I use is procmail although there are many others. For example, you may want to filter out emails from a mailing list into a separate boxes. It also allows you to benefit from a spam filter that integrates with the mail delivery agent.

My personal email setup is rather complicated but it fits my needs very well. First emails are retrieved from my various accounts using fetchmail, and delivered to my local account with over Qmail. Procmail then filters out any spam, forwards the remaining emails to a separate email account, and then does further filtering before delivering to local mailbox files. The reason for doing this is it allows my email to be stored and backed up on my Linux machine, and if from any computer to SSH in and read email with mutt. But when I’m on my own computer, I can check the secondary account with Thunderbird, which gives me the benefits of a graphical mail client. It’s these kinds of personalized setups that fetchmail (along with the rest of the MxA family) allows.

DKIM is a simple means for verifying the origination of an email in an attempt to better track (and fight) spam and phishing messages. The method is simple: the sender encrypts the message body using it’s private key and stores this in the message header (non DKIM receivers, then, can safely ignore it and still deliver the message). A DKIM-enabled receiver looks up the originating domain’s record and extracts the public key. From Wikipedia: “The receiver can then decrypt the hash value in the header field and at the same time recalculate the hash value for the mail body that was received, from the point immediately following the “DomainKey-Signature:” header. If the two values match, this cryptographically proves that the mail did in fact originate at the purported domain, and has not been tampered with in transit.”

Using such a technique makes it nearly impossible to create forged emails like the kind hitting our inboxes purporting to be from PayPal, eBay, and others. From CNET: “In the long run, DomainKeys is more promising than existing antispam and antiphishing technologies, which rely on techniques like assembling a “blacklist” of known fraudsters or detecting such messages by trying to identify common characteristics.”

Microsoft has also put together a method for guaranteeing identity of email messages, called Sender ID, though it’s acceptance has been very slow (“because Microsoft initially did not agree to license patents in ways that are compatible with GNU General Public License”).

I think the DKIM technique towards spam/phishing message blocking is one of the most interesting and promising methods to come around. I’ve used Bayesian filters and blacklists, and spammers always find a way around it. But if we can get mail servers to all use this technique, I could easily see the eradication of nearly all phishing messages, and much-reduced spam messages (since it would be possible to track the sending habits of domains better).

What’s your take? In order for DKIM to be useful, both the sender and receiver must implement the DKIM method, which may mean upgrading your mail server software. One of the biggest downsides to this, unfortunately, is the extra overhead involved in email: each message requires cryptographic signing and a domain lookup, which can be quite taxing for already overworked email servers. Will you be using DKIM?

Additional Reading
Wikipedia: DomainKeys
CNET News: Promising antispam technique gets nod
DomainKeys Identified Mail Homepage
DKIM Signatures IETF Standards Draft

My favorite command-line editor is vim – no, I don’t mean to start a war here – and I knew I’d be using it quite a bit in merging these changes. In coming up with a streamlined approach, I found a vim plugin that does recursive diffs, splitting the screen up to show differences between each of the files, and allowing for streamlined sync’s in the process. The plugin is called DirDiff, and is written by William Lee, and it proved very useful.

To install, just download the plugin, and place it in your ~/.vim/plugin directory (create it, if you don’t already have one). Once installed, open vim and type irDiff [directory1] [directory2]. It found and diff’d each of the files in the directories correctly. It also has a nice feature of sync’ing files, one from the other, based on what you choose. I would have liked more advanced merge capabilities, as I couldn’t take chunks from one file and have them merge into the other, but in most cases, the plugin proved very useful.

Probably the first, and most helpful, thing to do in a situation like this is to split the log file down into smaller chunks. It reduces the time it takes to open the file, reduces the memory used when opening it, and just makes it easier all around. Using the split -l command allowed us to pull out a set number of lines (say 100k), keeping the line endings intact (rather than splitting on bytes, for instance). The split command is extremely fast, which is great for files of this size.

The next step was to write a Perl script to do our parsing. Pretty simple (here’s the template):
open(FILE, "file-aa");
foreach my $line (<file>) {
# parse using split/regex/etc
}
close(FILE);

Finally, to output the line again, we used the “echo” shell command, called from within the script. Though a little extra overhead is incurred from having to execute a shell command, it meant we didn’t have to deal with managing file handles, etc. Plus, there didn’t seem to be a lot of extra overhead involved, and the real bottleneck is with the disk anyway, not the CPU.

One useful trick was to buffer lines rather than writing every line immediately. This allowed us to read in, say, a thousand or so lines at a time, then append all of them to the appropriate new file(s) as a batch, making the write larger, but all at once, which our disks could handle much faster.

These tricks allowed us to split this huge log access log in under a day or so, when other methods (especially before using split) we projected to take closer to a week. With the splitting and buffering we managed to get some significant speed increases.

Have you ever dealt with huge log files before? What tricks have you come up with? Certainly the best is to avoid them altogether using appropriate rolling and other tricks.