RidingTheClutch.com

I recently discovered Sinatra and and found that after a few tests with Apache Bench that it was pretty fast. It felt almost as fast as a Rack application. But how fast was it, really?

Whenever a new framework comes out people like to display a bunch of benchmarks showing how fast it can display “hello, world.” But I hadn’t seen many that compared several frameworks objectively. That’s where this post comes in. I wanted to compare the most popular frameworks and as a byproduct I also ended up testing the most popular templating languages. I ran everything against Ruby 1.9.1 since I wanted speed and so far it looks to be around 2x faster than 1.8.7.

So I ended up testing Rails vs. Rack vs. Sinatra, each of them using ERB, Builder, HAML and plain HTML templates. (I know that Camping has been around almost as long as Rails, and is pretty popular, but it’s not compatible with Ruby 1.9.1 as of yet.) Each framework would be served by Thin – generally considered the fastest way to serve Ruby apps right now. In addition I tested Rails served through Passenger and Apache. For comparison I wanted to test these frameworks against Apache and Nginx serving static HTML.

You can download the code for my tests at Github which includes the server setup scripts and web server configs I talk about below.

First I needed some hardware. I set up a medium-sized high cpu instance on Amazon’s EC2. I found a barebones Ubuntu 8.10 install (ami-0372946a). I installed several aptitude packages that allowed Ruby and Passenger to be built, along with the required gems to run the different frameworks. For the framework code I kept it simple: each created a variable to hold the time and then output that variable (along with some text) in each of the different frameworks. For example, the Rack/ERB template:

["\nrequire 'rubygems'\nrequire 'rack'\nrequire 'erb'\n \nclass Go\n  def call(env)\n    now = Time.now\n    [ 200, \n      {'Content-Type' => 'text/html'}, \n      ERB.new(\"<h1>Rack - ERB</h1><p>The time is:<%= now %>\").result(binding) ]\n  end\nend\n \nRack::Handler::Thin.run Go.new, :Port => 3000\n"]

For the actual tests I used Apache Bench. I know there’s a lot of controversy out there about how well AB actually works, whether it can create as many concurrent requests as you think you’re getting, etc. but for this test I thought it was adequate. I only went up to a maximum of 50 concurrent requests which I’ve read is a good maximum. I ran three different groups:

• 1000 requests, consecutive (one request goes out and returns before the next is sent)
• 1000 requests, 10 concurrent
• 10000 requests, 50 concurrent

For part 1 of this performance test I only hit a single instance of each framework, and tested from the server that was running the code. This removes any latency of the internet and will be an absolute pie-in-the-sky best case scenario. You will never see numbers this high in real life, sorry. :) One ssh session ran the server and the other ran Apache Bench. These were the only things running on the box at the time, besides standard kernel stuff.

I ran each test once to get the framework loaded into memory, then repeated it 10 times and recorded the average. I shut down that server and then started the next. I tried to keep everything as controlled as a could.

Here are the results (or the Google Doc ). Each column is the number of requests/per second. Each is an average of 10 runs using the Apache Bench settings in the column header (-n 1000 -c 10 means 1000 total requests, 10 concurrent). Numbers at the bottom of the column, below the double-line border, are the average of the numbers above. How to interpret these numbers

So, with one instance of a framework running, Rack is definitely the fastest. Rack serving plain HTML. As for templating languages, ERB and Builder are in a dead heat, but HAML isn’t that far behind. If you just need raw static serving speed, stick with nginx.

This is only part 1 of my Ultimate Ruby Performance Test. In part 2 I’ll put several instances of a framework behind Apache and Nginx and we should really start to see some benefit when doing the concurrent tests. Stay tuned!