HAProxy to smooth things out
For a long time I've run a rails app behind nginx with 3 mongrels in a round robin configuration. It has worked for me pretty well. Recently I've added a few longish running reports and doubled my user count, and I've seen some sluggishness on my app. The issue is not that the 3 mongrels can't handle the load, the issue is that sometimes the requests get stacked up behind the long running process when there may be a couple of free mongrels just sitting there.
This is where HAProxy comes into the picture. HAProxy is an open source tcp/http load balancer. The magic that it gives us is the ability to only allow one connection at a time to our mongrels. So, if we have one long running process going, that mongrel will be skipped and the other 2 will be used until the first frees up. I've just started using this, but I think that it is making a difference already. You can watch a nice screencast from Mark Imbriaco of 37 signals on this approach.
My setup:
nginx ----> HAProxy ----> mongrels
Twitter