Catalyst vs Rails vs Django Cook off

Today I began working on a new project and decided to benchmark Catalyst and Rails for fun. See how my new favorable framework does against Rails. I was a bit shocked at the results though. I guess this is worth mentioning in hope Catalyst can improve in it's Accessor Generation code. So here are the results:

Benchmark System
Celeron 1.8Ghz
1 Gig of Ram
FreeBSD-6

Interpreters:
Ruby - 1.8.5
Perl - 5.8.8

Frameworks:
Catalyst - 5.7003
Rails - 1.1.6

Run as:
Lighttpd: 1.4.13
FCGI: 3 max proc

Benchmarked as:
ab -n 1000 -c 100 http://siteurl.com/

h3. Some background

I specifically turned off sessions and did not use ActiveRecord/DBIC to keep it as fair as possible between the two frameworks. Both frameworks were run under Lighttpd and FCGI. I tried to keep this as apples to apples as possible.

So lets take a look at the results!

Rails:


Server Software:        lighttpd/1.4.13                                    
Server Hostname:        wansanity.com
Server Port:            9090

Document Path:          /main/index
Document Length:        2142 bytes

Concurrency Level:      100
Time taken for tests:   18.261 seconds
Complete requests:      1000
Failed requests:        0
Broken pipe errors:     0
Total transferred:      2296892 bytes
HTML transferred:       2143288 bytes
Requests per second:    54.76 [#/sec] (mean)
Time per request:       1826.10 [ms] (mean)
Time per request:       18.26 [ms] (mean, across all concurrent requests)
Transfer rate:          125.78 [Kbytes/sec] received

Connnection Times (ms)
            min  mean[+/-sd] median   max
Connect:       74   885 1742.9    138 11785
Processing:   172   661 1216.8    173  8195
Waiting:       84   661 1216.8    173  8194
Total:        172  1547 2123.8    330 11893

Percentage of the requests served within a certain time (ms)
50%    330
66%   1354
75%   2786
80%   3106
90%   4297
95%   6279
98%   8216
99%   9285
100%  11893 (last request)

Thats 54 connections / sec which is great. I have seen it peak at 70 connections/sec which is just awesome!

Catalyst:

	
Server Software:        lighttpd/1.4.13                                    
Server Hostname:        wansanity.com
Server Port:            80

Document Path:          /
Document Length:        2232 bytes

Concurrency Level:      100
Time taken for tests:   43.503 seconds
Complete requests:      1000
Failed requests:        0
Broken pipe errors:     0
Total transferred:      2401300 bytes
HTML transferred:       2238490 bytes
Requests per second:    22.99 [#/sec] (mean)
Time per request:       4350.30 [ms] (mean)
Time per request:       43.50 [ms] (mean, across all concurrent requests)
Transfer rate:          55.20 [Kbytes/sec] received

Connnection Times (ms)
              min  mean[+/-sd] median   max
Connect:       75   322  808.5     93  6028
Processing:   269  3804  851.8   3928  6754
Waiting:      192  3804  851.7   3928  6754
Total:        269  4126 1178.5   4186 10293

Percentage of the requests served within a certain time (ms)
  50%   4186
  66%   4384
  75%   4404
  80%   4424
  90%   5025
  95%   6422
  98%   7194
  99%   7709
 100%  10293 (last request)

22 connections / sec not exactly what I expected from a framework built on top of the fast Perl Interpreter.

Being a bit disappointed with the results, I investigated further.

So here are the perl dprof results.

	
%Time ExclSec CumulS #Calls sec/call Csec/c  Name
 0.00   0.605  4.128   1512   0.0004 0.0027  NEXT::AUTOLOAD
 0.00   0.373  0.373  25794   0.0000 0.0000  Class::Accessor::Fast::__ANON__
 0.00   0.235  0.235   1177   0.0002 0.0002  NEXT::ELSEWHERE::ancestors
 0.00   0.211  0.225      1   0.2107 0.2253  YAML::Type::code::BEGIN
 0.00   0.184  5.182     86   0.0021 0.0603  Catalyst::Engine::HTTP::_handler
 0.00   0.177  0.205   2583   0.0001 0.0001  File::Spec::Unix::canonpath
 0.00   0.164  0.309   1942   0.0001 0.0002  File::Spec::Unix::catdir
 0.00   0.156  2.408   3201   0.0000 0.0008  Catalyst::Action::__ANON__
 0.00   0.134  0.739     73   0.0018 0.0101  base::import
 0.00   0.129  0.136   5904   0.0000 0.0000  Class::Data::Inheritable::__ANON__
 0.00   0.109  0.814      7   0.0155 0.1163  main::BEGIN
 0.00   0.108  0.108   1323   0.0001 0.0001  HTTP::Headers::_header
 0.00   0.101  0.116     10   0.0101 0.0116  Template::Parser::BEGIN
 0.00   0.101  0.334     11   0.0092 0.0304  Catalyst::Engine::BEGIN
 0.00   0.101  0.295   1264   0.0001 0.0002  Path::Class::Dir::stringify

It seems like the main bottleneck in Catalyst 5.7003 is Next. Jrockway was kind enough to post some new code into Catalyst's trunk for me to try; a new replacement for Next - C3.

Here are the results with the C3 Plugin from Trunk

	
%Time ExclSec CumulS #Calls sec/call Csec/c  Name
 0.00   0.211  0.233      1   0.2106 0.2330  YAML::Type::code::BEGIN
 0.00   0.135  0.135   8035   0.0000 0.0000  Class::Accessor::Fast::__ANON__
 0.00   0.126  0.721     73   0.0017 0.0099  base::import
 0.00   0.109  0.116     10   0.0109 0.0116  Template::Parser::BEGIN
 0.00   0.108  0.805      7   0.0155 0.1150  main::BEGIN
 0.00   0.093  0.106      7   0.0133 0.0152  Catalyst::Engine::HTTP::Restarter:
                                             :Watcher::BEGIN
 0.00   0.090  0.105   1023   0.0001 0.0001  File::Spec::Unix::canonpath
 0.00   0.085  0.326     11   0.0077 0.0296  Catalyst::Engine::BEGIN
 0.00   0.081  0.905    196   0.0004 0.0046  Catalyst::execute
 0.00   0.069  0.120      8   0.0087 0.0150  Catalyst::Plugin::Server::XMLRPC::
                                             Request::BEGIN
 0.00   0.064  1.639    444   0.0001 0.0037  next::method
 0.00   0.061  0.313     32   0.0019 0.0098  Catalyst::BEGIN
 0.00   0.054  0.216      7   0.0077 0.0309  Template::Config::load
 0.00   0.054  0.189      4   0.0135 0.0473  HTTP::Body::OctetStream::BEGIN
 0.00   0.054  0.388      4   0.0135 0.0970  Gambit::BEGIN

So there you have it, the results with the C3 Plugin. It only made a slight difference by pushing the Catalyst benchmark score to 25 connections / sec.

I hope this benchmark can get some changes put into place for Catalyst's next release.

Conclusion

It seems like Rails is roughly 62% faster than Catalyst at this time. Keep in mind this benchmark does not take into account the ORM performance. This benchmark tests how quick the frameworks themselves dispatch methods and render views.

Also take into consideration when choosing a framework you need to look at the problem at hand. Catalyst can feed off Perl's vast CPAN resource library. Catalyst has features that Rails does not have. Catalyst's DBIC ORM supports multi-column primary keys and can do relationship mapping just by reading the schema! You don't even have to bother writing any has_many belongs_to definitions!

I am going to have to take a look into Django see how well it fairs in this benchmark. Perhaps an update on this?

Update Django Results


Server Software:        lighttpd/1.4.13                                    
Server Hostname:        fab40
Server Port:            9090

Document Path:          /
Document Length:        2235 bytes

Concurrency Level:      100
Time taken for tests:   13.643 seconds
Complete requests:      1000
Failed requests:        0
Broken pipe errors:     0
Total transferred:      2409769 bytes
HTML transferred:       2253459 bytes
Requests per second:    73.30 [#/sec] (mean)
Time per request:       1364.30 [ms] (mean)
Time per request:       13.64 [ms] (mean, across all concurrent requests)
Transfer rate:          176.63 [Kbytes/sec] received

Connnection Times (ms)
              min  mean[+/-sd] median   max
Connect:       76   483 1068.1    101  8666
Processing:   190   744  726.3    571  6088
Waiting:       93   744  726.4    572  6088
Total:        190  1227 1414.2    692  9606

Percentage of the requests served within a certain time (ms)
  50%    692
  66%    972
  75%   1209
  80%   1445
  90%   3282
  95%   4020
  98%   6414
  99%   8113
 100%   9606 (last request)

72 connections / sec! Amazing and the winner!

And anyone that disagrees with this can go ahead and look at the code for all three projects.

I have the least experience with django for your information

mst Please don't kill me'

Many thanks go out to jrockway to helping me point out the root cause of the bottleneck in Catalyst.

vic

Posted

testing

JayK

This post has caused a flurry of discussion in
several circles.

I myself have looked at the test, and performed
several simlar tests with both Catalyst and Rails
based loosely on the code rovided alongside
this entry.

That said, I strongly recommend that this test
not be used to decide between frameworks.
Ignoring the various non-performance related
reasons for choosing a framework, there are
issues with the test provided here. These are
my comments from another discussion, posted here
for completeness.

1) Because of the simplicity of the example apps,
and the lack of real-world configuration,
this can not be considered to be a real world
application test - at best it's a dispatcher test.

2) As a dispatcher test it fails for the
following reasons: a) all systems tested use the
templating system of the framework in question
which is significant enough overhead to completely
mask any dispatcher performance indicators.
b) There are only two actions present - which
is significantly less than any real project
would have.

3) Dispatchers are responsible for sorting out
where to send the request for processing. It is
a balancing act and the results reflect a trade-
off between speed at different levels of usage /
numbers of actions / complexity of uri matching.
So different design decisions will yield
different results. To really bench the
dispatcher you'd need to account for this by
providing tests at different depths of actions,
quantity of actions in the system, types of
actions (static vs. regex) etc.

4) You really need to request multiple URIs to
work the dispatcher out. Optimizations that
deal with a single uri being requested many
times could drastically skew results on any of
the platforms.

5) The configs were not 'clean' - among other
things, templates and other plugins are
loaded. The way that at least Catalyst works,
plugins can and do literally plug into the
dispatch cycle - whether they are used or not -
which means anything that is loaded can (and
often does) affect speed of the uri to action
dispatch, even if it's not 'in use'.

In short - While certainly opinion provoking..
we can't really derive any solid information
from the tests that were performed as they were
performed.

That said - I think it is thought provoking and
it encourages further investigation... Because I
too am surprised at the scale of discrepency
given what I know about perl and ruby... and
Cat and Rails.

I just hope that no-one takes this as a
definitive indication of performance of any of
the frameworks, and that the next round of
benchmarks that will no doubt be posted will
test the systems more completely.

Perrin Harkins

Please try sorting the dprof results by real time (the -r flag) instead of just CPU time. Otherwise, I/O wait is not taken into account and you may be chasing the wrong problems.

snake

Very cool! I've been looking at django, catalyst and rails. Not much benchmarks out there. Just flamewars and zealots. You did it! The first one came out with some interesting tests. Given that benchmarks among framworks are quite hard to come to conclusion, but very interesting results nonetheless.

victor

Well, do take this into consideration -- catalyst is the most flexible out of all the three frameworks and It has the most features thanks to CPAN.

Micha

What about Grails? :o)

Name
Email
Website
Body
Verification

Lets Get Dugg!