Benchmarking Ebb, Thin, Mongrel and FastCGI for Ruby on Rails hosting

Since day one of our Ruby on Rails hosting packages, we provide our customers with a Lighttpd webserver, which starts two Ruby processes. The lighttpd handles all static requests, and all other requests are handed over to the Ruby processes using the FastCGI protocol. Two years ago, we were convinced that this was the best and most stable way to offer shared hosting for Rails projects.

The last few weeks, there was a lot of buzz in the Rails-community about new ways to serve Rails projects. We first saw the improved Mongrel, later came Thin and recently Ebb, which claimed to be 10% faster than Thin and 30% faster than Mongrel.

There are three kinds of lies: lies, damned lies, and statistics. We wondered how much faster those new methods are, and how they compare to the classic Mongrel setup, or our Lighttpd/FastCGI combination.

In all testcases, we ran one front-end webserver and two back-end Rails processes.

Static content

We are convinced: static content should be handled by webservers that are optimized to serve static content. This has been one of the major advantages of our current setup: the lighttpd server will serve all static content itself, without passing it to the Rails backends. Quite a few Rails projects cache output to .html files, so these are blazing fast and will even work when the Rails environment is down or restarting. Let’s look a graph that shows the performance of Lighttpd, Ebb, Thin and nginx when used as a static file server. The tests were done using the Apache “ab” tool, against the homepage of a Mephisto blog, which gets cached to a single static .html file.

We see a clear pattern: If you have static content, serve it through a webserver, and not through your Ruby environment! The graph shows static content being served at +10000 requests per second serving 50 concurrent visitors! (There is also a lesson for application builders in this: Think about “caching” your output to plain .html pages!)

Mephisto search results

To test the “real” Rails performance of Ebb, Thin and Mongrel and to compare them to our setup, we tested the search results page of a Mephisto blog. These result-pages can’t be cached, and they require computing and rendering in Mephisto. This way, we move the performance testing away from the connection handling and serving of content towards the Rails/Ruby environment.

The results came a bit as a suprise! It seems that Mongrel is a lot faster than Ebb and Thin! All those new webservers claim they are a lot faster than Mongrel, while in fact, they are slower. We have a theory about the causes of this “anormality” and will give our opinion later on.

The second Thing we noticed was that our two-year-old setup of a Lighttpd 1.4 and Ruby fastcgi processes, is still amongst the fastest. In fact, we don’t see why we would switch to another config based on these results.

Radiant CMS results

Because of the odd result that Mongrel was, in fact, among the fastest environments in the previous test, we wanted to test “fast” Rails pages. The search result in Mephisto is slow (as it has to do a full text search over all the blog articles and does that in a non-optimized way). The Radiant CMS is very lean and fast, but doesn’t use HTML caching for its pages. An excellent candidate for our benchmarks.

The lighttpd + fastcgi setup blows the competition away! It’s faster at every level of concurrency. Another noticable point is the fact that Mongrel is amongst the slowest of the pack and Thin and Ebb are (as predicted by their authors) significantly faster than Mongrel.

Why is Mongrel sometimes amongst the fastest and sometimes amongst the slowest? We Think it’s because Mongrel is quite fast at processing Ruby code (as it is written in Ruby itself), but is quite slow at handling new connections, at processing the HTTP headers and signaling. The new and improved Ruby servers such as Thin and Ebb, are very fast and efficient when handling new requests and their connections, but are probably a bit slower in handling the actual Ruby code.

For pages that spend quite some time doing “Ruby” work, such as the Mephisto blog search results, this means Mongrel is faster, for small and speedy requests such as the Radiant-CMS homepage, Ebb and Thin are faster as the processing time is less important, and the time needed to handle the connections can make the difference.

Conclusions

Our main conclusion is that the choices we made two years ago about the default setup for our Rails customers, is still an excellent choice. It’s easy for the customer to setup or tune (the config files are very small), it’s very fast at serving static content, it’s very fast for non-intensive pages, and it’s doing well serving intensive pages.

For customers that have a (virtual) dedicated environment, there are two conclusions. The first is that you do want to handle all static requests using either lighttpd or nginx; you don’t want Mongrel, Ebb or Thin to serve static content. The second is that you need to test your application with a few different configurations and decide the best one for your needs.

And a short note: we performed these benchmarks to the best of our knowledge, but they still are simple benchmarks.