Using a Real Logger With Netty Will Cause Boss Thread to Explode

I came across a really interesting bug that was plaguing me for about a day and a half and I finally tracked it down today, with JIT assistance from the superhuman @brianm.

I have a fairly simple app that uses Netty for its underlying event handling. Just a Web service, no great shakes. But, there was this weird bug that I found during load testing whereby it would just stop all network activity after a few minutes for no discernable reason. No more connections could be established, existing connections did not finish, etc. At first, I thought it was maybe a bug in Tsung, my load testing tool, but that wasn’t the case.

I upped the logging and found this strange error that was happening right around the time of the network shutdown:

I googled around for this error and found only a few references to it, mostly having to do with a partially-uploaded JAR. My app is packaged in a fat JAR, so I found this unlikely, but I reuploaded it and verified the upload with the MD5 hash and still the error occurred and the network shut down.

I was pretty much out of other options at this point, since there were no other anomalies and YourKit was unable to find any deadlocks or such. I took a stab and sent the gist above to @brianm to see if he had seen it before. He immediately noted that the call to was bad news. He then suggested I try the “bazooka-squirrel approach”: if you can’t hit the squirrel with your .22 because the tree’s a mess, take out the whole tree with a bazooka ;-) In this case, the “tree” was the logging attempt in the backtrace.

However, I wasn’t using java.util.logging in the app. I was using logback. I checked out the source code of NioServerSocketPipelineSink and found that it was Netty using java.util.logging. Oh, really? At this point, java.util.logging was not configured. I configured it with a simple ConsoleLogger and all of a sudden *BLAM*… problem solved.

Netty is using java.util.logging which, if left unconfigured, was causing it to reach out into the fat JAR for the logging configuration which wasn’t there. When it did so, it caused an InternalError due to some kind of JVM security issue which blew up the server socket. Netty catches that and shuts down the boss thread. *sigh*

tl;dr You must configure java.util.logging to go somewhere or if there’s an exception in your boss thread it will decapitate Netty rather effectively.

MapReduce Patterns, Algorithms, and Use Cases

Great list of map/reduce patterns that everyone can use.

Tackling the Folklore Surrounding High Performance Computing

Great talk by Martin Thompson. Well worth the hour to watch.

If You’re Using Node.js, You’re Doing Life Wrong

This morning, on a conference mailing list, I made some disparaging remarks about Node.js (the title of this post, in fact). A couple people asked me why I felt that way. Rather than respond individually, I’ll just list my reasons here:


V8 is not server-class

At my current place, we have a Web crawler where some portions are leveraging Chrome/V8. Take a guess which component absolutely dominates the bug count and issue list? Not to mention the fact that its balls-ass slower than some straight un-optimized Scala. We’re looking to get rid of it completely ASAFP.

On a larger note, using JavaScript on the server-side seems kind of ridiculous. If Linden Labs came out with a server-side framework in LindenScript, would you use that? How about if Apple came out with a framework based on AppleScript?

Callback spaghetti is about the last pattern with which you’d ever want to write anything

I’ve written servers in just about every kind of server framework pattern there is. Node’s is the absolute worst. It provides the least amount of aid and comfort to the programmer and its nigh-on-impossible to follow the code 6 months later. The idea that the broken-ass concurrency model forced on Javascript by old browser implementations is actually a *good thing* gives me a fucking aneurism.

Non-blocking != fast != scalable

This is probably the most annoying point for me. First of all, scalability has very little to do with raw speed. Just because you’re fast does not mean you’re scalable. You know what’s fast? MySQL. You know what’s not scalable by itself? MySQL. The hype around Node.js on this issue makes me want to punch faces. Furthermore, Node.js isn’t even that fast. You can do much better with Scala and its a much nicer language, to boot. Oh, and nevermind about those extra CPUs you bought: you won’t be needing those. Events and non-blocking are instantaneous, so you can do everything on a single core, right?


The Ruby and Python communities are just now, many years after the hype has faded, learning that stuff like dependency injection and proper modularization are actually good things that help you maintain code over time. JavaScript has very little support for any of those nice things: it doesn’t even have namespaces, for chrissakes. Why would I want to repeat these same mistakes over and over again in a new language? Knowing a language is not the same as being able to maintain services written in that language over the long haul.


It seems to me that people who are really crazy about Node.js are people who only knew JavaScript to begin with and to whom none of the above would ever even occur. Perhaps, this is a case of Worse Is Better and I’ll eventually have to eat my words on this one. But the kind of misunderstanding going on in this video clip seems to pervade the Node.js hype and gives me rageface.

HeapAudit – JVM Memory Profiler for the Real World

HeapAudit is not a monitoring tool, but rather an engineering tool that collects actionable data – information sufficient for directly making code change improvements. It is created for the real world, applicable to live running production servers.

HeapAudit is a foursquare open source project designed for understanding JVM heap allocations. It is implemented as a Java agent built on top of ASM.

Looks really awesome. I’m probably going to try this out tomorrow on some $ORK systems.

GoLang Tutorials

Table of Contents

Section 1: Audience, Programming, Go Philosophies, Setup Audience
Checking we have it setup right Section 2 A step by step approach to Hello World in Go
Typical early errors while programming Go
Compiling and linking with Makefile Section 3 Memory, variables in memory, and pointers
Control structures - Go if else statement
Control structures - Go for loop, break, continue, range
Control structures - Go switch case statement
Control structures - Go defer statement
Multiple return values from Go functions Section 4 - Objected Oriented Programming in Go? Or lack thereof Structs in Go - instead of classes in Object Oriented Programming
Anonymous fields in structs - like object composition
Methods on structs
Inheritance and subclassing in Go - or its near likeness
Multiple inheritance in Go
Interfaces in Go
Interfaces in Go - Part 2: Aiding adaptable, evolutionary design NEW
Polymorphism in Go
Summary of object oriented programming goodness in Go, without the pain Section 5 - Goroutines, channels: parallel programming and communications Goroutines
Channels in Go
Channels in Go - range and select Section 6 - Web Programming Web Programming with Go - first web Hello world
Go Templates
Go Templates - Part 2
Go Templates - Part 3 - Template Sets
Using an external api in a Go web program - urlshortener from Google APIs
OAuth2: 3-legged authorization in a Go web application Section 7 - Google AppEngine for Go Google AppEngine for Go - Installation and Setup
Analysing a Google AppEnginge for Go program
Using an external api in a Go AppEngine program - urlshortener from Google APIs Miscellaneous Go packages and goinstall - creating and using your own packages in Go
gotest - unit testing and benchmarking Go programs
Formatting Go code with gofmt

Superman’s a Baby, but He’s Still Superman

Which means that the correct lesson the boy’s parents could have taught him was what it is  the boy does to make Superman think he can manipulate him, or even what it is about Superman that makes him act that way; but the one they went with, the one that will make him neurotic for the rest of his life, is that he’s a winner.

Automatically Populating .ssh/known_hosts

At that point I got busy with other things (most notably final preparations for the FreeBSD 9.0-RELEASE announcement) but on Sunday evening I sat down and wrote a much-needed shell script:

# ssh-knownhost hostname [fingerprint ...]
The ssh-knownhost script uses ssh-keyscan to download all the host keys for the specified hostname; uses ssh-keygen to compute their fingerprints; compares them to the list of fingerprints provided on the command-line; and adds any new host keys to ~/.ssh/known_hosts. Short, simple, and effective.

Could be really handy.

Why Warren Buffett Disdains the Private Equity Crowd

Calling this “Orwellian”: Buffett wrote that “private equity” is a “name that turns facts upside-down: A purchase of a business by these firms almost invariably results in dramatic reductions in the equity portion of the acquiree’s capital structure compared to that previously existing.


There are different approaches for getting to the user’s credentials, I will present the easiest here and I will concentrate on iOS:

  1. We register a custom NSURLProtocol for ‘keylogger://’ URLs. It is a dummy implementation which just makes sure that those URLs aren’t processed further by the framework.

  2. In the webView:didFinishLoad: method, inject some JavaScript into the loaded page. The JavaScript will attach a listener to every input element on the page and that listener will call a ‘keylogger://’ URL crafted by us which contains the character the user entered.

  3. In the shouldStartLoadWithRequest: method, we capture all of the ‘keylogger://’ requests and log the characters. Then we stop loading, because those URLs are just used to communicate between JS and Objective-C.