Hello World - Web Application Benchmarks
By: Joshua Chamas
Choosing the right way to build your web site has never been
more complex than it is today. It was easy at first, a bunch
of html, and a few cgis, and you would have yourself some of the
best stuff online.
We still have the static html and perl CGIs. But how do you
want your CGI ? You used to run your CGI as a separate
executable every web request, but now embedding the executable
in the server is all the rage. You can take your sluggish
perl CGI, and accelerate it enormously with
ActiveState's PerlEx
or
Doug MacEahern's mod_perl.
This is just the beginning. CGI is only one way to deploy
a web application. Now we can have the best of both worlds
and mix the html and code easily in embeded environments
like
Apache::ASP,
ASP PerlScript,
HTML::Embed,
and
Mason
to name a few.
You have already chosen to use perl, and for good reason,
namely rapid development time, often twice that of other
languages. But there are still some important questions that remain.
On what operating system, web server, and web application
environment will you chose to develop your site?
Back to the Benchmark
It is tough to answer these questions objectively, so a few of us,
on the ModPerl mailing list, thought it would be good to do some
benchmarks comparing the speed various web application
environments, web servers, operating systems, and, just
to be fair, different languages too!
The methodology used to compare these various web environments
is to create a simple Hello World program, created in
such a way as to be a fair comparion between raw code,
and embeded environments. The output of the script should
be roughly:
Content-Type: text/html
Hello World
The results are ranked by most web requests processed per second
per CPU clock cycles, controlling for some benchmarks using
500 MHz machines, and some only 233 MHz.
The benchmark results presented below were tested from over a
network with web clients making requests in HTTP 1.0, so that
the server was fully utilized serving the Hello World benchmark.
Much of the original data
for this article was not presented, as those benchmarks where the
server was tested with the client on the same machine do not
represent the maximum capacity of the web server being tested.
Results
What the benchmarks show well is the relative startup
speeds of each web application environment on various
operating systems and web servers. The speed of this Hello World
program would then be the upper limit on the speed of any web application
you would deploy under a similar environment.
What it does not
show is how this will translate to performance for Real World
applications, involving large code bases and potentially complex
system interactions like database driven html.
Of all the web
enviroments tested, PHP results are likely the least indicative of
real world performance for the bench, as it does not cache compiled scripts,
as other environments do. Its performance is reported to degrade more
than other environments as the complexity of the application increases,
increasing the runtime parsing that PHP must do.
You should be able to conclude then from these results which
operating system and web server is best for your web
application of choice, but not necessarily which web application
is the fastest for your web site needs. This latter analysis
begs futher testing of more robust benchmark.
The Best Benchmarks
This table shows the fastest benchmarks submitted for each
web application environment.
Note that Linux tends to
be faster than Solaris in general, and that perl cgis run
under modperl/registry tend to be as fast as embperl scripts,
but this does not show up here, as the fastest perl cgi
benchmark was done on Solaris.
Linux vs. Solaris
It may not be easy to compare how operating systems stack up
against each other, but one way is to look at how web application
environments perform across various operating systems.
You can definately see from this graph that Linux is faster than Solaris.
Web Application Rankings on Linux
These results show a nice cross section of various web application
environments on the Linux platforms.
Web Application Rankings on Solaris Intel
These results show how the perl web application environments compare
when controlling for the operating system. It gives a more honest
comparison, than the best benchmarks list.
Web Application Rankings on WinNT
| Code Type | Hits/sec/MHz | Hits/s | CPU | OS | Web Server | Application |
| ASP VBScript | | 540 | PII-266 x 2 | WinNT 4.0 SP4 | iis 4.0 | asp/vbscript |
| ASP VBScript | | 300 | PII-300 | WinNT 4.0 SP4 | iis 4.0 | asp/vbscript |
| ASP PerlScript | | 26 | PII-300 | WinNT 4.0 SP4 | iis 4.0 | asp/perlscript 515 |
Web Application Rankings All
Kudos
Many thanks to all those who submitted results and gave feedback
on the benchmark. Special thanks go out to:
Chip Turner
Jeffrey Baker
John Smirl
Kimbro Staken
Rasmus Lerdorf
and to Matt Sergeant
who master minded the Hello World benchmark in
the first place.
References
For the orginal data that this article is based on,
and some that didn't cleanly fit here, go to
http://www.chamas.com/hello_world.html.
It also has all the source code used for the benchmarks on the
various systems.