For a time, Twitter planned to allow firehose access through a service called Gnip, but in October stated that they would instead work on providing access themselves.  Since then, details have been sparse on the timeline or methods for which access will be given.

Last week however, some more information has been quietly released by means of an FAQ on the Twitter API website.  Here’s the question and answer in full (emphasis theirs):

When will the firehose be ready?

By late January, early February 2009. For at least Q1 2009, the “firehose” (the near-realtime stream of all public status updates on Twitter) will only be available to a small group of trusted partners. The firehose is a stream HTTP solution; a client connects to it and the stream begins, ceasing only when the client disconnects. Once we’re confident in the stability of the service, we’ll add partners on a case-by-case basis. We may allow a wider selection of clients to consume subsets of the public stream (that is, updates from a collection of user IDs or matching specific search terms). We do not intend to allow anonymous, unregulated public access to this stream for any number of legal, financial, and technical reasons.

There’s a few pieces of new information here.  First, that some kind of beta group will be given firehose access within a few weeks, using HTTP streams.  This sounds similar to the solution provided by Gnip to their users.

Perhaps more important though is the news that a full public stream of the type previously provided will not be returning.  While providing subsets of the public stream could be useful for things like groups, without the full firehose it’s very difficult (if not impossible) to provide a feature like real-time tracking, which has been eagerly awaited.

The three pieces of rationale for not making the firehose public (“legal, financial, and technical”) each bring up additional questions in turn.  Legally, is there a difference between providing public tweets in a full stream and providing tweets publicly by user (which requires knowing the username ahead of time)?  Do the financial motivations refer to saving money on servers and bandwidth, or by making money in providing access for a fee?  And technically, are the existing solutions (such as HTTP streams or XMPP) insufficient for the task, and if so, how?

First, I thought I’d outline the reasons for an alternative to Gmail.  Let me preface this by making it clear that I love Gmail – it’s a great service, but one that has some inherently odd choices that make it less than ideal for an email-heavy user.  Some of Gmail’s most irksome quirks:

• No such thing as folders.  Instead, everything is tagged, which doesn’t lend itself well to being used inside my mail client
• The “All Mail” special tag appearing in my mail client via IMAP, which appears as though I receive two of every email
• Removes duplicates of messages, which means if I send a message to myself, it will never appear in my inbox (instead, it’s always in my Sent Messages tag)
• Not completely reliable.

In terms of reliability, it’s very important to me that the service is always available, and always fast.  I want to receive messages moments after they’re sent – not hours later.  When checking for messages from my other accounts (via POP), Google uses some heuristics to determine how often it should check for new messages.  Unfortunately, this often means it won’t check for an hour or so, which can be very annoying.

So, I’ve been on the lookout for a Gmail replacement for some time, and thought I had found it in Apple’s revamped MobileMe service.  When it was launched, it appeared that Apple was doing email right, but as I’ve learned from using the service first-hand, I’ve merely traded improvements in some areas, for problems in others.  For instance:

• It can’t check foreign email accounts.  This wasn’t a complete deal-breaker, as I have some clever auto-forward email recipes setup which largely solves this problem.
• There’s no way to send mail as a non-MobileMe email account.  You can create multiple “personalities” that give you different @me.com addresses, but there’s no way to send as a non @me.com address.  This is a huge problem, and means I can’t stop using Gmail until this is added to MobileMe.
• It drives me crazy that the web interface often refreshes the entire page.  If I leave it open in a Safari tab for a little while, then return, it seems to refresh.  That’s annoying.
• Another annoyance factor is that whenever I return to the email interface (or whenever the page refreshes), all of my folders are expanded.  I have a LOT of email, and I keep them fairly well organized into many different folders.  I’d like the system to remember what folders I had kept closed.

I should mention that I am a big fan of the push email functionality I get with MobileMe (and is a major reason why I’m still using the service).  Also, the email web interface is clean and intuitive (and matches the Mail desktop app, which scores major points with me).  Compared to the other major email providers (Gmail, and Yahoo Mail, for instance), the MobileMe email interface wins hands-down.

Apple’s MobileMe is a fantastic service, but I just can’t switch yet.  So many different great features are all aligning to make for a killer email product, but the annoyances I’ve outlined are show-stoppers for me (and why I can’t dump Gmail just yet).  I’m more than happy to pay the MobileMe subscription service if Apple could just get these last few details smoothed out.

What’s your take?  Have you considered switching from Gmail to MobileMe?

1. Rather than restate all the steps I took, I’ll link to this post that I found useful. In particular, configuring Gmail’s Drafts, Junk, Trash and Sent folders to match those in Mail is nice.
2. The way Gmail implements IMAP wasn’t very intuitive to me and took some getting used to. For example, deleting a message from a mailbox doesn’t really delete it, it just removes that label (or at least that’s how it’s supposed to work; more on that later). I found this page helpful in getting used to how IMAP functions match Gmail actions.
3. I know this has probably been discussed ad nauseum, but I wish Gmail would loose the “labels” concept and move to “folders”. While I’ve eventually gotten used to it within Gmail, the problem is only made worse with IMAP. For example, if a message has two labels, you’ll see it in two folders, which means it’ll show as two unread messages.
4. I figured as long as I was setting all this up, I’d use Gmail to backup all my old messages, as described by Ryan. This meant copying over my old mbox files, importing them to Mail, and moving them into the appropriate folders in Gmail. It’s definitely a slow process, but you can view the progress from the Activity window (under Window -> Activity). There were a few times when the moving failed: I believe it was due to cases where messages had very large bodies. Although it failed partway through, I was able to start over without having any duplicate messages, so it looks like Gmail is smart enough not to add an exact duplicate.
5. To help speed the process a bit, I’d recommend turning off caching while doing any importing or moving of messages. This is done under “Keep messages for offline viewing” in the Advanced tab of the Gmail Account preferences. Once everything is set how you like it, you can turn cacheing back on. For a large number of messages the caching process can also take a while.
6. It took me a while to find a setup for my outgoing messages that works for me. In the past, I’ve had my email client set to automatically BCC myself on any emails I send. This way my messages get categorized properly, and my emails will be in the same mailbox as the rest of a thread. With Gmail, my sent messages were getting put into the Sent Mail folder in Gmail, which means I never got a chance to filter them into any labels. BCCing myself didn’t help, since Gmail already had a copy of the message in Sent Items. What I’ve done now is to remove Gmail’s Sent folder as Mail’s Sent mailbox, which means the only copy that Gmail receives is the one that gets BCCed, which means I can have it filtered however I like. Update: Turns out I was wrong on this.  Any email sent through Gmail’s SMTP will be added to “Sent Mail”, so BCCing won’t help.
7. An unfortunate annoyance with Gmail IMAP is that unless you have a message filtered to automatically archive (in Gmail context this means moving it to “All Mail”), it’ll show up twice in Mail: once in the Inbox, and once in “All Mail”. So far I haven’t found a way to avoid this.
8. The filters in Gmail are rather basic and limited, especially compared to Mail. For example, I don’t think there’s any way to match messages whose subjects start with a string, instead of just containing the string.
9. Rather than set up a complex set of filters for all the email I get, I got an idea from Ryan to only create filters for less important messages, like automated log messages. Anything important will to go my inbox, and then I can choose how to label it or to delete it. This also has the benefit that on my iPhone, I can just check my Inbox and have important new emails in one place.
10. Although Google Help says that deleting a message will just remove its label, it won’t necessarily work this way in Mail. Although deleting a message will remove the label (or if it’s in the Inbox, archive it), moving it to the Trash will actually put it in the Trash, which puts it in line for deletion. I think you could prevent this by not having Mail’s Trash mailbox be Gmail’s Trash, but the whole “deleting a message to remove the label” feature isn’t something I really need to use.
11. I love the new Todos in Mail, but getting them working in Gmail also required some changes. From what I can tell, you can have Mail store the Todos locally or on Gmail (although there doesn’t seem to be a simple Preferences change for this). If it’s on the server, each Todo is stored as an email, and if you want to specify a Calendar for an item, it means defining a new set of Calendars in iCal. If you store the items on the server, each corresponding message shows up in a nice human readable format, but unfortunately on the iPhone all you see is a Mime attachment. Hopefully in the future Apple will provide a simple way to access Todos on the iPhone.
12. Reading back on these notes, I see how complicated Gmail IMAP can be! I don’t think it’s quite ready for average users yet. That said, it’s nice to know that 1) I now have all my email accounts coming to one place, 2) my old emails are backed up online, and 3) I can access all my email from my computer, my phone, or a browser.

Here’s a quick overview of how to get all of your old emails into Gmail as painlessly as possible (and one way that preserves dates!).

Loading Emails
I spent some time looking into the different options I had for loading my email messages into Gmail. I have about 10 thousand or so that I wanted to keep stored (there’s a lot more than that sitting around in various mbox files, but these were the important ones). These 10 thousand or so messages comprised about one year of using the Apple Mail application on my Mac. So, poking around, I found a Python script by Mark Lyon called GML (for GMail Loader), which was designed to iterate through each message in an mbox (or other format) file and forward each message to GMail.

I managed to get the script working (one tip: be sure to export your messages by using the Save Archive option, which will create appropriately formatted mbox files for the script to read) and was able to import about the first thousand or so messages I had. Unfortunately, I ran into a couple of problems. First, I could only forward so many messages an hour due to a restriction set by my ISP’s SMTP server. Second, and most problematic, was that the GML script would actually forward each message, one-by-one, as if I were clicking the Forward button in my email client. This meant that, although the message still had it’s original date associated with it, the message would show up as having been originally sent on the date that I forwarded the message, not when it was actually received. This was a problem, and there had to be another way.

One idea I had was to setup my own pop server, where I would place all of my archived messages, and setup this server as an account in Gmail. Then, Gmail would connect to my mail server and load all of my old messages. I never got a chance to try it, but my guess is that it would work. Instead, another idea came up (courtesy of Costa): use IMAP.

Not long ago, Google announced the Gmail service would now support IMAP, allowing mail clients to connect to Gmail and use it as a kind of clearinghouse for all things email (IMAP stores everything on the server, allowing any number of clients to connect and always appear in sync with one another). If I added Gmail as an IMAP account to my Apple Mail program, could I then simply drag messages stored in the various folders into Gmail? The answer, luckily, was yes!

It was simple: I merely added Gmail as an IMAP account to my Apple Mail client, then just dragged folder-by-folder all the messages important to me onto my Gmail account. It took a while to transfer all the messages (be sure to open the Activity window in Mail to see your progress), but once it was done, all my old messages were safely stored in Gmail and had their original dates! Perfect!

Using IMAP
So now I’ve got my desktop Mail client using Gmail as my sole email account, along with my iPhone. Whenever I read a message on my desktop, it’s marked as read on my phone. And whenever I send an email from my phone, it appears in my Sent folder on my desktop. Everything stays in sync, and I’m very pleased with how it turned out.

One note: be sure to check out this page for details on fine-tuning your Gmail IMAP setup with Apple Mail and the iPhone. There’s a couple of neat little tricks to get your setup streamlined.

sudo su
[Enter your password]
bash
export ARCHFLAGS="-arch i386"
gem update

Note that you’ll want to change the i386 to the actual architecture you have (my MacBook, for instance, is an Intel processor, while you would want to use “ppc” for non-Intel Macs).

Once I had updated my gems, things started working fine again. For instance, mongrel_cluster was having issues configuring a new project, but after updating the gem using this method, it seemed to be working fine. Hope you find this useful!

Of course, don’t get the impression that Java is dead!  It’s anything but: kiosks and embedded devices run it, along with consumer-facing websites and plenty of enterprise software.  Many other technologies have been eating away at it over the years, and will continue to do so, unfortunately, for many of the reasons in the article.

Like Alex, I think the biggest issue Java faced (and got wrong) was with the web.  Before there was web 2.0 (and web 1.0 was still being explored), Java applets were the way to integrate interactivity inside web pages.  But applets were horribly slow, limited in their abilities, and complicated.  I think if Sun were able to do it all over again, they shouldn’t have focused on placing Java applets inside web pages, but focused more closely on making Java the core of web development.  Like Alex states, imagine how great it would be if we could manipulate the DOM using Java, say if it were an integrated part of the browser.

But, alas, that didn’t happen.  There’s still time to reinvent how Java and the web co-exist, but for the time being, it’s being edged out in favor of new technologies and languages.  So here’s to you, Java!

And I think Joel is completely right. Gmail, for one, has been stagnant for the last three years or so, and haven’t been preparing for the future. Check out this blog article from Lifehacker about a comparison between Gmail and Yahoo Mail. Their conclusion? Yahoo Mail has spent the last two years innovating and adding all sorts of new features, while Gmail has very little improvements (except, perhaps, incrementally increased storage levels).

As Joel has pointed out, the future of web applications is going to involve a standardized interface, providing UI-level features, interoperability, and a common platform for development. And there’s evidence of this happening. Several frameworks have come out recently that are generally very good: my favorites are YUI and prototype/scriptaculous. Very helpful, though the YUI is, I think, especially useful, as it provides a lot of very clean, useful interface-based functionality perfect for developing rich applications. But, it’s far from lightweight. And Joel’s point is: that’s okay.

I think we’re nearing a critical mass when more online software development will use an existing framework like YUI, so the common code between these sites will more likely be in users’ caches, therefore speeding up load times and reducing bandwidth costs. And, on top of that, there is likely to be further innovation to speed this up: building some amount of the framework code into browsers, say, or different, longer-lasting caching mechanisms strictly for Javascript framework files, or a whole number of other options.

Whatever happens, I think the biggest winners over the next several years will be with the developers who spend their time making the best applications, and let others (like browser developers) worry about keeping it fast. That’s not to say, “go out and make super-slow sites”, but rather to realize you shouldn’t spend all your time developing faster applications while neglecting features and other forward-looking progress.

If you’re not already aware, if you bought your iPhone within 10 days of an announced price break, you’re entitled to receive the difference from Apple (provided you claim this within another 14 days of the announcement).

Should Apple reduce its price on any shipped product within 10 calendar days of shipment, you may contact Apple Sales Support at 1-800-676-2775 to request a refund or credit of the difference between the price you were charged and the current selling price. To receive the refund or credit you must contact Apple within 14 business days of shipment.

Sadly, we at Draconis bought out iPhones 16 days before the announcement: 2 days later and we would have qualified for that rebate. But I’m not bitter over it: I love my iPhone, was willing to part with the full price without expecting any kind of rebate, and anyway, these things are out of our control. Anyone else in the same boat as us?

The blog is to keep much of the invotrak-related content in one place (without cluttering up this blog). It’ll have invotrak-related news (new features, notices, etc), as well as original content we think is of interest to users. If you’re interested in freelancing/entrepreneurship/etc, you may find the invotrak blog interesting.