Socks and Sandals

I’m speaking about Hadoop today at 4PM EDT at the Philadelphia Emerging Technologies for the Enterprise conference. Its located at Drexel University. Come on down if you can make it; the speaker list is great and I’m sure it will be a great time for all!

I went up to Manhattan last night to attend the monthly NYC.rb Hackfest and it was pretty awesome! Lots of great people and conversation was had. I highly recommend checking it out if you get a chance to head up there.

...has arrived in the form of a blog post by one Lawrence Kesteloot. In this post, he attempts to show that languages are distinguished as either “production” or “toy” languages (troll terms if I’ve ever heard them) and then proceeds to slam SICP and Lisp and generally anything he doesn’t understand.

My biggest problem with this article is not the fact that someone with a masters degree in computer science doesn’t know what a fixed point is, but rather the fact that it nearly proves the Blub paradox to be true all by itself. Clearly, this author has become “institutionalized” in the Java mindset and construes brevity of expression with inscrutability. He maligns powerful languages features and elegance in favor of “understandability” by the average programmer and then implicitly marks himself as such by misconstruing boilerplate code that an IDE generates for him with readability. Ever read the code generated by Axis? No… why would you? That’s the least readable Java can be, in my estimation, but the point is that I don’t have to read it. Lisp excels at that kind of code generation task. The author seems to believe that thinking about the code you’re working with is a bad thing and that if it takes any effort at all to understand it must be bad. This thing we’re in is an art, people, not a science.

The speculation about Yahoo!’s rewrite of Viaweb also does not help his case. Yahoo! trucks in average programmers galore. Surely, there are many good and great ones there, too, just not in as great a number. As such, its not unbelievable that they could not find Lisp programmers: you typically have to be looking for something before you find it. Also, why would a Lisp guy want to work at Yahoo!? The coolest project they have going right now is written in Java. The kind of guy who hacks Lisp is unlikely to be pining for a job at Microsoft or IBM or even Yahoo! these days, nes pas? I do know some excellent “production” people (around Philly, even) that hack on Lisp and Scheme in production, but they appear to be the exception rather than the rule. Finally, I’m assuming that the author has never heard the fact that lots of the big financials choose a very high-level functional language when they need to make sure their systems are right. I bet this is partially so they can attract the very best people to those positions, as well.

I suppose the author might have also taken Amazon’s choice to dump Lisp as a sign that Lisp is not production. (assuming he was aware that it was, in fact, originally built in Lisp and C) I, instead, would take that as an indication that Amazon outgrew the ability to employ only the best people they could find. Business growth has a way of doing that to you. The distribution of talent and passion appear to be two somewhat uncorrelated power law curves and finding people at the top of both curves simultaneously is a task that will make anyone’s hair fall out and liver get harder. Plus, there are many business realities that make Blub languages more attractive for any number of reasons: talent pool, employee turnover, ramp-up time mitigation, cultural integration, etc. I write this article after having just dumped Ruby on Rails in favor of Java at my startup for some of these very reasons.

To this author I would only say that a language does not make something “production”. People do. Throw Alan Cox , Ingo Molnar and Guy Steele in a room and I bet they could write a better mousetrap in Brainf*ck if they were excited enough about it. Its never about the language. Its about the people. Full stop.

Last night I gave a presentation on Scala to the Philly Lambda group. The slides are available here. It was a pretty good time and about 15 people or so showed up for the talk. The space was graciously provided in the really cool-looking Cira Centre down at 30th St. by Aaron Feng of Algorithmics. Thanks to Aaron and all who attended; I had a great time presenting!

Someone had said the above statement to me today and it stuck in my mind for some reason. I was thinking about it tonight and something occurred to me.

I’ve heard a bunch of variations on this statement before today as I’m sure we all have. Often, when I hear this statement its coming from people who would like to be able (or allowed) to use Ruby in their work projects. These places are invariably ones in which Java is the primary language and any others are either frowned upon or outright barred. The management at these places, either technical or not, typically characterize this decision as smart given the large market for engineers that know Java, the availability of quality libraries for many purposes and the inherent generality of the language. All of these are certainly true and are pretty good business reasons to use Java.

These arguments ignore the ease of use of Ruby and the speed of development one can achieve with it. They are also a form of premature optimization, as well, although in this case of business interests, not source code. Most of these shops are doing 3-tier Web applications and, as such, they are missing out if they skip over Rails because it doesn’t run on Java. (it does, but whatever)

However, it was the statement that really hit. People saying that are indicating that different languages have different strengths. Even though Java now has regular expression capabilities, you’d still want to use Perl or Ruby for a big text processing job given how many built-in facilities those languages have for that task. You’d want to use Erlang for very-long-lived server applications over Java because it was tailor-made for that purpose and Java was not. All of these points lend themselves towards a heterogeneous language environment, using a language for what its good for and not for what its not.

I wonder, though: when people tell me this, they are usually rationalizing the use of Ruby with this statement. Not “rationalizing” in that they shouldn’t be using Ruby, but rather they are seeking to get leverage for Ruby so that it might one day be in the position that Java is now. I can certainly sympathize with this; Ruby is a sweet language. Alas, you can’t get there from here today, though. The main Ruby interpreter has major problems with memory and stability and its successors are still in their nascent stages. The runtime situation is just not that good so you can’t drop Java for everything yet if you need speed/low-memory in places.

How many of the people who have said this, though, would then reverse their position and advocate for Ruby being the only language for “maintainability” or “ease of introduction to the junior guys” reasons once a strong Ruby VM was available? And how many people said the same thing in reference to Java when C and C++ were king? It seems to me that these language trends are cyclical. The new hotness challenges the old-and-busted and eventually wins, only to become the new old-and-busted.

The irony here is that the statement is true in an absolute sense. One should use languages for the things they are good for and find different ones for things they are not. To not do so is to arbitrarily shorten not only your toolset but your very range of thought. (big up, Sapir-Whorf) Attempting to use one language for all your programming needs leads to ridiculous situations like the Kingdom of Nouns phenomenon. The other strangeness with this statement is that the people who say it generally never mention the really out-there languages like Lisp, Ocaml, Prolog or Smalltalk that are orders of magnitude better at certain things than more mainstream languages. Personally, I just hope people remember the irony when 10 years have passed and all new development at JPMorgan is in Ruby running on a Gemstone derived VM.

Apparently, its not just software engineers that can’t estimate how long a project will take . I thought maybe we were alone in that, given all of the high-tower thinkers spouting off about how other engineering disciplines always succeed in their endeavors and software rarely does. Looks like maybe no one’s got it all figured out just yet, eh?

This article was especially pertinent to me now since I am responsible for coming up with these kinds of schedules at work these days. I’m not too far along in my EBS implementation but I’m hoping that can offset some of the pain of project estimation.

I just got back from RubyConf 2007 tonight and it was awesome! The conference was a well-oiled machine with superb content and attendees. My only beef, and this is small, is that the hotel Wifi was a bit weak, but that’s to be expected.

Confreaks video-taped the whole conference, including RejectConf and they said the video will be available for streaming and download sometime next week. Their videos are awesome so you should definitely check them out.

The level of conversation in the hallways and taverns was just unreal. There were so many smart people there working on really great stuff that I didn’t know who to talk to next!

Evan’s keynote was probably the best talk of the conference for me, but I’m a bit biased towards Rubinius, given my previous issues with the MRI as it stands today . Francis Hwang’s talk was the most intriguing and made me think the most. Ryan Davis’s talk was packed and really entertaining. I missed Laurent’s talk but it was similarly packed and people said it was hyper-cool. Eric’s talk was really awesome, too, as he was giving us all his tips about how to stay productive: coming from a guy with 40+ open source projects under his belt, that’s definitely good info.

Some highlights for me, in no particular order:

• Got to meet Tim Bray and he’s just as sharp as I though, if not sharper
• Met some of the Twitter guys and they are really, really cool, down-to-earth dudes
• I finally got to meet Matz in person and I used the opportunity to interrogate him on the state of Ruby’s GC
• Ryan Davis and Francis Hwang gave me some really good tips on testing and I am very grateful for that
• Had a couple really good conversations with Evan Phoenix ; man
  • He’s one smart-ass cookie and a great showman, too
• Hung out with a bunch of really smart and interesting Rubyists, including Obie Fernandez and Steven Bristol
• Talked to Ezra a bunch about Merb and Engine Yard and got to meet Yehuda Katz
• Played in a video-taped Werewolf game with some of the best Rubyists in the world (who shall remain nameless at this time because the game became… uh, lets say “controversial”)

There wasn’t a lot of Rails-related stuff there (which was good) and the people who were there were really into Ruby, not just Rails. DHH didn’t even show up so the noise level was low.

I’m definitely going to be going to RubyConf 2008. Thanks to Rich, David and Chad for an awesome conference!

Joel Spolsky has an intriguing new article on one of the new features in FogBugz 6.0 called Evidence Based Scheduling.

I’ve actually had some experience with parts of this technique before, as I used it on a project while I was at Symantec. In my estimation, the project went much better because of this technique, partly because of the much better granularity afforded by this technique and the constant feedback given to myself and my superiors.

I also felt much better about my progress, both because I was completing tasks much faster (since they were smaller) and I had a much better handle on the scope of each task. It also did force me to think about the implementation issues ahead of time, something that Joel mentions and I think a lot of us gloss over and hand-wave away when dreaming up schedules and project plans.

Credit goes to Ross Fubini for being the first at Symantec to implement this style of project management.

One thing we didn’t do at Symantec was the the statistical dispersion of potential completion dates. Using a Monte Carlo simulation would indeed give you a much better feel for progress and allows for more realistic estimates of project health. We use statistical analysis at work quite frequently but it had never occurred to me to apply it to the software development process. Hindsight’s always 20-20, eh? Kudos to Joel for having the insight to apply this technique to software engineering.

As I am now responsible for the delivery of commercial software, I look for things of this nature to aid in the day-to-day engineering of projects. Based partly on my experiences and partly on the elegance of the idea, I do believe that EBS is very valuable and those responsible for shipping software should give it some serious consideration. You don’t even need FogBugz 6.0 to do it: an Excel spreadsheet or wiki software can be used to track the tasks and associated times and you can use GNU R to do the Monte Carlo simulations.

My friend Kyle recently turned me on to Project Euler. The site is a forum and list of problems and is a great way to get you into some math and numerics hacking examples. The problems are supposedly all solvable by a reasonably-powered computer in about a minute but there’s no proviso for you being able to code up the solution to them in that short a time :) You don’t get to see the forum for a particular problem until you’ve submitted the right answer to that problem, though, so it encourages you to work out at least a brute force solution before seeing what others did.

I’m pretty surprised at how many people chose “Assembler” as their preferred language. Me, I’m not that hardcore: I’m posting my problem solutions in Ruby as I finish each problem.

It would seem that Sun’s new chip, dubbed the Rock, will support hardware extentions for software transactional memories.

I tend to think of this as not such great news.

Now, I’m all for helping programmers in their work (being one myself) and making it easier to work with the parallel systems coming down the pike. However, I don’t see hardware STM support as a big step forward.

I do realize that people don’t change overnight and that all of that legacy code can’t be rewritten in a flash to take advantage of the newer, better ways always being introduced. I also realize that Sun is making the right business move in allowing their customers to “port” their code to these more parallel machines with the minimum of effort. That’s all well and good and I applaud them being the first to market on this, both from a business and a technical perspective. I’m sure it wasn’t an easy task, what they did. But making a big deal of this advance belies the ultimate truths involved here.

Its no secret that I’m a fan of message-passing concurrency in general and Erlang in specific. I believe that this is the way to go for the industry going forward. You are free to disagree, but I feel that MPC is a safer and cleaner way to go forward for the future of the industry. It removes a lot of the issues that are still problems with STM, just hidden so that only the VM and compiler guys have to worry about them. (at least initially) Not that it doesn’t introduce some of its own, but I find these to be preferable to the alternatives.

However, in the end, using STM doesn’t force programmers to write concurrent code. They are not thinking concurrently, only thinking about what to mark as atomic. This is still sequential programming. People using SQL have been thinking this way for years and I’d hardly call it concurrency-oriented. Raise your hand if you’ve worked on a project where someone kills database performance with some poorly-written transactions. All hands up?

We’re just going to have to go through this again later when the STM abstraction starts falling apart. Optimistic concurrency (the underlying premise of STMs) is only as good as the care taken to prevent lots of and/or many long transactions. Without this care, performance will suffer both in speed and concurrency. It is, after all, optimistic. Optimism just doesn’t seem to scale.

You can argue that the best programmers will do this beautifully, and I agree. But how many of us get to work with the best programmers? How many more of us have to deal with a bunch of code written by people who couldn’t wait to jump in their cars at five o’clock? To me, message-passing concurrency solves this problem in a more direct (if more initially brutal) method: it just plain forces the programmer to think concurrently. That’s the real key. Don’t mask it, make them face it.

STMs are essentially a stop-gap measure to stem the pain coming from the rise of multicore. But, like any drug, the effects will wear off and that pain will be felt. By using STMs, you’re borrowing from the future. Personally, I would think it better to just rip off the band-aid now and start learning and using the message-passing concurrency mindset today.