The O’Reilly Sysadmin Blog has a nice piece on things sysadmins forget to do. Most of them relate to common tasks that fall by the wayside. I think programmers and sysadmins alike have a tendency to prefer interesting, more intellectually challenging activities over things like managing users or root access. The good thing about this is that these kinds of menial tasks can often be automated by programs like cfengine or puppet.

Even for those tasks that cannot be fully automated, such as keeping documentation, there are plenty of applications out there to assist in the process. For example, we’ve found that keeping documentation, notes and fixes in a wiki is a great way to keep information organized.

Personally, one of the tasks I often forget is to keep packages managed and up-to-date. Luckily, this is a lot easier than it used to be with programs like apt-get and yum.

Finally, I think we can all appreciate the reminder to always keep a friendly attitude: “If you want support from management, consider remembering that the user you offend today could wind up on the board of directors. Regardless of that possibility, system administrators should always remember that their clients are internal and if you want to keep your job, be good to your clients.”

We maintain a lot of projects in version control, both internal stuff (like our monitoring/management app RSP, and our website), as well as client projects (some of whom have direct access). Originally, we used CVS to handle versioning, but over time, migrated to Subversion, a great replacement. Since moving to Subversion, however, we’ve had to rethink how backups are done, as one of the key differences with Subversion is how it stores information on disk. Instead of storing flat files with versioning information inside each file, it instead uses either the Berkeley database or a custom file system (called FSFS, for a filesystem within the host’s filesystem). It gives the application a lot more flexibility (directories are properly versioned as well as files), but can make backups a bit tricky. Let’s take a look at how to backup a Subversion repository (it’s not too difficult).

The trick here is dealing with a live repository: since users could be making changes to this repository at any given time, doing a recursive filesystem copy isn’t ideal. If a user were making a change while this copy was taking place, the backup could be missing data, or could simply be corrupt, which defeats the purpose of making the backup in the first place! A better way is to either freeze the Subversion server temporarily, until the backup completes, or to do a hot-copy.

[Read more...]

Ruby gets a lot of (well deserved) press because of Ruby on Rails, but recently I’ve found it to also be an excellent choice for scripting tasks, jobs that I otherwise would have used Perl for.

The first benefit of using Ruby is that its built-in code blocks allow for simpler, shorter code. A Ruby block is basically an anonymous function that can be passed to another function as a parameter. For example, here’s how you could take an array of numbers and square each one:

[ruby]

nums = [1, 2, 3, 4, 5]
nums.collect { |x| x*x } # produces [1, 4, 9, 16, 25]
[/ruby]

Blocks are used for all sorts of purposes, like looping, iterating, sorting and mapping. Once you get used to them, they become quite intuitive and save a lot of coding time.

In particular I’ve found code blocks to very useful with database programming. Here’s an example that uses the mysql Ruby plugin to do a query:

[ruby]
require ‘mysql’

db = Mysql.new(“localhost”, “root”, “password”)
db.select_db(“a_database”)

result = db.query “select id,name from users”
result.each do |row|
puts “ID: #{id}, Name: #{name}”
end
[/ruby]

Note that “do/end” is a multi-line alternative to brackets.

In Ruby, everything is an object, even primitive types like numbers and strings. The language makes it quite easy to create your own classes as well, and because of this I find myself creating classes more often than I normally would while writing scripts. Currently in Perl, OO programming is still rather obtuse, so it’s often simpler just to use scalars, arrays and hashes in various combinations (for example, “a hash where the keys are names, and the values are arrays where the first element is a number”, etc). The problem with this is it can create brittle code that can make bugs common.

Ruby classes can be written inline with procedural statements, and are usually very short:

[ruby]
# A simple class to represent a person
class Person
attr_accessor :first_name, :last_name, :age

def initialize(first_name, last_name, age)
@first_name = first_name
@last_name = last_name
@age = age
end
end

joe = new Person(“Joe”, “Schmo”, 25)
[/ruby]

The “attr_accessor” line is a useful declaration that creates a private member variable along with public reader and writer methods.

Two common tasks of scripts are working with operating system calls, and string parsing/manipulation. In these aspects, Ruby works just like Perl. You can use hash marks (the ` character) to run a command return its output. For working with strings, Ruby regular expressions work just like Perl. Here’s an example that uses both:

[ruby]
# Look for a resolve error
resolve = `resolveip #{address}`
puts “Host not found” if resolve =~ /host not found/
[/ruby]

All of this being said, there are still many places where I feel that Perl is a more appropriate choice for sysadmins. While Ruby has a growing set of plugins, nothing can beat the breadth of Perl modules available through CPAN. For a very specific task (say, interfacing with the /proc filesystem) Perl is often a better choice simply because of the amount of publicly available code out there.

The next time you have to do some administrative scripting, consider giving Ruby a try. Even if you decide to stick with Perl (or whatever your language of choice), it’s a great way to try out Ruby if you’re not familiar with it.

Even a relatively inactive server can produce quite a large amount of logs, and a high use machine serving multiple functions will probably have so much log activity that it becomes impossible to discover anomalies, resource problems, or potential attacks. Luckily, this copious amount of data can be made manageable without too much trouble.

Consolidate logs across the network

If you have more than one machine to keep track of (and what sysadmin doesn’t) the log problem only multiplies. One thing that can help in this situation is remote logging. For example, syslog makes it easy to send logs over the network to a single source. In large networks this may be a dedicated logging machine. In any case, this at least somewhat reduces the complexity of the situation, although does not help with the sheer amount of information produced.

I’ll usually leave a window open with a connection to the central logging machine with a running output of logs (this can be done with `tail -f’). This helps me catch some issues as they occur, but it’s easy to miss entries if I’m not at the computer or not paying attention to the running log.

[Read more...]

There’s a nice article on Lifehacker about backing up your Mac using several free tools. As I’ve said time and again, backups are key! The article gives a good introduction to the various tools available for Mac backups.

As pointed out in the article, the next iteration of the MacOS will see a new feature, Time Machine, which will provide built-in revisioning services to documents in the MacOS (very cool), as well as the ability to backup to a different hard drive (including a remote-mounted disk). I’m looking forward to this (and the Spaces feature) especially, but until then, another tool needs to fill the need of backing up your Mac.

[Read more...]

I do a lot of data backups on CDs and DVDs, and I recently realized that I had no organized way of figuring out which files were located on which discs. Rather than looking for a piece of management software, I decided it would be easier to just write an app myself in Ruby on Rails.

Nowadays, agile development methodologies like such as Ruby on Rails allow you to development applications so quickly, that it’s often easier to write code yourself, even if a similar tool already exists.

Imagine if I had tried to use an existing tool to manage my backups, rather than do it myself. I would needed to take the time to:

• Research available tools and determine which one fits my needs
• Setup and install the program
• Customize it for my needs and environment
• Learn to use it

During the time it takes to complete these tasks, it’s just as easy (and frankly more fun) to code an app myself, which will be customized to my exact needs from the beginning, and requires no learning time.

As agile development becomes more and more used, I think this “do-it-yourself” practice will also become more common. Certainly for system administrators, who already spend time writing customized scripts and GUI apps. This is the topic of a recent IBM developerWorks article, Develop Web applications for local use, which encourages web application development in place of GUI or command-line applications.

An added benefit of this practice is that the developer might discover that the application fills a need felt by others, and can turn their local webapp into an actual website or product. That’s how RSP started. It started development as a tool to use internally, before we discovered that it could have a use to the outside world.

Backups, as always, are an important part of any sysadmin’s job. If you’ve been following this blog for some time, we’ve taken a look at doing automated backups on Linux, Solaris, Windows, and the MacOS, but each of these assumed a sysadmin would be setting up the backups (or at least someone with knowledge of cron, rsync, etc). As many first-time Linux users are getting started using Ubuntu, this article gives a step-by-step to users who may not be ready to setup an rsync system for themselves.

SBackup is a simple backup solution intended for desktop use. It can backup any subset of files and directories. Exclusions can be defined by regular expressions. A maximum individual file size limit can be defined. Backups may be saved to any local and remote directories that are supported by gnome-vfs. There is a Gnome GUI interface for configuration and restore.

If you’re looking for an easy way to backup your Ubuntu desktop machine, sbackup may be the way to go.