Socks and Sandals

I was just reading about In/Out, 37signals’ internal Twitter clone over on their blog and it struck me how they sometimes mutilate context in their writings. First of all, DHH knows the Twitter guys well. Second of all, they didn’t even mention that it was inspired by Twitter and even went so far as to refute this in the comments. I guess a place that’s know for its “creativity” has to keep up the appearances of being innovative…

UPDATE: Apparently, as the comments state, I am wrong about this. My bad. Nevermind this post.

Today a real validation for the Hadoop effort came in the form of Yahoo! announcing that their production search is now running on Hadoop. This should go a long way to allaying others’ concerns about Hadoop’s speed, stability or scalability. Cheers, Yahoo!: You’ve done some excellent work!

This is getting really old, really fast. Now the Charlotte hotel I am staying in also proxies port 25 and fails to proxy SMTP AUTH. WTF? And they are using Symantec Mail Security, a product from the division where I used to work! Sheesh… Anyway, its really Mailstreet’s fault for not listening on 587 in the first place.

For those of you who hit up the T-Mobile WiFi access points at your local Starbucks at various points, I’ve found two things to watch out for:

• T-Mobile proxies port 25 (SMTP) traffic. This wouldn’t be horrendous if they didn’t fail to also proxy the SMTP AUTH commands. Anyone shipping their outgoing mail over port 25 and needing authentication is hosed. If you can, submit your mail over the alternate submission port (587) if your ESP supports it.
• The layout of the stores was not well regarded when Starbucks Corporate decided to install those new sandwich microwaves. As such, any place within about a 60 degree fan-out in front of the cooker will lose WiFi due to microwave interference while it is running. This is quite frequently between the hours of 7 and 11 AM (and even more extended on weekends) so plan to either sit outside or somewhere way off to the side of the microwave unit.

As well, the service is pretty expensive (on the order of a landline broadband connection) and not very fast. But, it does in a pinch. Tim Bray says Boingo is much faster (and cheaper) but I’ve yet to find a place that had this service around me.

When I started using Amazon’s EC2 service I realized pretty quickly that the traditional load-balancing solution of putting a big, honkin’ F5 BIG-IP in front of the servers wasn’t going to work out. Amazon doesn’t currently rent F5’s ;)

So, I went looking around for a DNS-based load-balancing solution that would be flexible enough to deal with the dynamic environment that EC2 provides. However, I pretty quickly found that the existing dynamic DNS APIs of most of the providers were not up to the task of programmatically updating a DNS record the way I needed. Specifically, I wanted to be able to register and deregister an EC2 instance with a round robin A record automatically upon instance startup and shutdown.

In the end, I was only able to find one dynamic DNS provider whose API was up to this task: Nettica.

Once I found them, it was pretty easy to wrap their SOAP-based API into a binary to drive this type of dynamic management of my DNS records. Apache Axis took care of turning the WSDL into Java (the only library that could do so across three programming languages, by the way) and the code for driving that wrapper was pretty simple. The end result of that effort is now open sourced for all to use.

The way I use the new Nettica client is to have the init script automatically register an instance on startup with the Nettica service. However, I remove the instance from Nettica a bit before shutting down the instance itself in order to deal with DNS caches that sustain the now-removed IP for longer than the specified TTL value. You might be able to automate this by throwing a sleep LARGENUMBER into the shutdown portion of the init script, but I haven’t tried this yet.

In any case, I hope people find this client helpful when using EC2. I look forward to hearing about experiences with it and improving it for others. I’m planning for a release pretty soon to add support for RPM packaging, as most of the AMIs used today appear to be RedHat RPM based. Watch this space for updates.

I’ve just released by code to update EC2 instances on start up and shut down with the Nettica DNS service. You can find it here:

http://code.google.com/p/netticadns/

With this code, you can set up your EC2 instances to automatically register/deregister themselves with the Nettica DNS service on start up and shut down. There’s no RPM packaging at this point, but there is a dpkg available for it.

I’d appreciate any feedback or patches. Have fun with it :)

I just gave a presentation about the Erlang programming language to the Philly Linux User Group on Wednesday night. The slides are available in the first link. It was an introductory talk focused towards a programming-oriented group. Let me know what you think :)

I’ve been asked to speak at Philly ETech this year, and guess what? I agreed!

I’m on the Web 2.0 track on March 29th at 3:30 PM. I’ll be talking about Comet, which is a technique for building super-low latency Web applications that allow for real-time collaboration between users.

The schedule is up and so is my abstract along with what some might call a bio. And, if anyone wants to go but can’t get a registration, I have one for both days so email me up and we can deal.

Wow, Sun is really hitting on all cylinders lately. Project Blackbox is an especially neat idea of putting a datacenter-sized array of boxes inside a portable box (in relative terms). I remember that Cringley was going off about Google building things like this for themself a while ago, but I never heard anything more about it. Now, anyone can grab one ;-) Man, they look good, too.

I recently gave a talk to the Philly Linux User Group regarding the internal workings of Google’s infrastructure. The slides are available here. The presentation was created with S5 so all that’s required to view them is a relatively recent browser.

The talk was pretty well received. I was somewhat surprised at the interest in the material. I have never worked for Google, so I got all my information from their own papers and various other Internet sources. A considerable amount of knowledge is available just at their Google Labs paper index, although they seem to have pared it down in size recently.