Vanilla on HHVM a wicked experiment....

x00

HHVM stands for HipHop Virtual Machine, for those who are not aware. It is what facebook uses to run its infrastructure.

Around 2008 facebook needed to optimize their architecture.

They needed something that would be more suitable for scale and performance than the Zend engine that PHP usually runs on.

Their first incarnation was a php to binary compiler called HPHPc which was open sourced in 2010, it was powerful but had limitations in terms of running existing web architecture, that wasn't specifically designed for it.

HHVM is an new execution engine for PHP which runs as virtual machine that JIT (just in time) compiles php. It is exclusively for 64 bit architecture.

However is HHVM is not just an execution engine, but is a web server in one. So you don't need apache or nginx necessarily, however you can also proxy with nginx or apache.

Given that I have been hearing great things about it recently and it is nearing total support for PHP 5.4 I wanted to try it out. I was curious as to how it would perform against some typical setups of apache and nginx in a load test.

This is not totally scientific, but wanted to idealize the test as far as possible. So I installed a fresh copy of Linux Mint 15. There is a binary packages for hhvm for several distros, however I found run smoothly most up to date master was, so compiling form source was the way to go. I will cover setup later.

But without further ado the the test and results:

Perquisites

• Processor: 4x Intel(R) Core(TM)2 Quad CPU Q8300 @ 2.50GHz
• Memory: 3275MB (1641MB used)
• Operating System: Linux Mint 15 Olivia (clean install)
• Vanilla install based on my hhvm mods (for all tests)

Test Subjects

• Apache/2.2.22 + PHP 5.4.9-4 Apache handler
• Nginx/1.2.6 + PHP 5.4.9-4 FPM FastCGI
• HipHop VM v2.4.0-dev (rel)

No real additional optimizations, only thing it to increase connections/files on nginx and fcgi proxy to make it comparable.

Test 1

Simulate browsing load using JMeter with a basic HTTP Get request to index.php?p=/discussions, 500 threads with a ramp up period of 2 seconds.

Results

Apache

Nginx

HHVM

Test 2

Simulate a discussion post load using JMeter with a basic HTTP POST request to index.php?p=/post/discussion/ then handle subsequent GET redirect to post, 300 threads with a ramp up period of 2 seconds.

Post params

• Name: Test Post
• Body: Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.
• CategoryID: 1

Predefined Cookies

• Vanilla
• Vanilla-Vv

Results

Apache

Nginx

HHVM

Conclusions

I must say I was pretty impressed HHVM is clearly very fast 3-4 times faster and the throughput 2.5 as much. On post+redirect it was astounding x3-5 throughput . But really it makes sense the code is optimized, so everything including the PDO/Databases connection is going to be very fast. the other are just servers they may be individualyl fast, but can't speed up the execution.

I love it when posts don't feel laboured, and all those posts 300 where posted an redirected in a flash.

None of these tests have any other features such APC or Memcached, which is for another day. Also fronting HHVP with other servers.

Nginx was not as good as apache, in the post operation which is disappointing, but bear in mind there is lot of configuration you can do regarding php-fpm and nginx. Also nginx has sensible default settlings and this is not really a 'sensible' test but a more extreme test.

I think most importantly point is HHVM it is pretty resilient. I test with with thousands thread an it held up ok, with stable output.

Todd

Awesome stuff. We just had @kasper build a vagrant machine to do some hhvm testing with in house. I think he has it up on github. We are very interested in hhvm and will be making sure our code runs well on it. Our initial tests didn't offer as much of a speedup as we'd hope so we have some investigation to do still. Please post back any more discoveries you have @x00.

PS: Take a screenshot with a scrollbar in it is a cruel trick

Kasper

Nice one @x00! Like Todd mentioned, we ran an Nginx + HHVM FastCGI combo (now part of Vagrant LNPP: https://github.com/kasperisager/vagrant-lnpp) and got a ~75% increase in throughput compared to Nginx + PHP-FPM. This was all done inside a 1024mb virtual machine running Ubuntu Precise (amd64) so it definitely wasn't bad – it just wasn't a mind-bogglingly fast as we had hoped.

x00

Big Caveat: HHVM is not yet 100% transferable, with infrastructure Zend and still has bugs and nuances. The is totality experimental not for production.

Like I said you will need a 64 bit OS that is supported. Folks I offer no support installing, as this is an advanced topic. If you are not familiar with how to compile software, and setting up web servers this is not for you.

I followed this roughly

https://github.com/facebook/hhvm/wiki/Building-and-installing-HHVM-on-Ubuntu-13.04

Don't use the prebuilt packages as they are older.

You can grab a copy of my tweeks, first cd to your web folder

git clone https://github.com/x00/Garden vanillatweaks
cd vanillatweaks
git chechout hhvm

You will need a basic configuration the server e.g. hhvm.hdf

Server {
  Port = 80
  SourceRoot = /var/www/
}

Eval {
  Jit = true
  EnableObjDestructCall = true
}
Log {
  Level = Error
  UseLogFile = true
  File = /var/log/hhvm/error.log
  Access {
    * {
      File = /var/log/hhvm/access.log
      Format = %h %l %u %t \"%r\" %>s %b
    }
  }
}

VirtualHost {
  * {
    Pattern = .*
    RewriteRules {
      dirindex {
        pattern = ^/(.*)/?$
        to = $1/index.php
        qsa = true
    }
  }
}

StaticFile {
  FilesMatch {
    * {
      pattern = .*\.(dll|exe)
      headers {
        * = Content-Disposition: attachment
      }
    }
  }
  Extensions {
    css = text/css
    gif = image/gif
    html = text/html
    jpe = image/jpeg
    jpeg = image/jpeg
    jpg = image/jpeg
    png = image/png
    tif = image/tiff
    tiff = image/tiff
    txt = text/plain
  }
}

stop other servers on that port and start the server

sudo ~/dev/hhvm/hphp/hhvm/hhvm --mode server --user web --config /etc/hhvm.hdf

you can also run it as daemon.

x00

One of the main changes is I used a later version of Smarty

http://www.smarty.net/files/Smarty-2.6.28.tar.gz

and extract it to library/vendors/Smarty-2.6.28

this basically deals with depreciated e modifier in preg_match and uses a proper callback.

here is a summery of the other mods.

--- a/applications/dashboard/models/class.usermodel.php
+++ b/applications/dashboard/models/class.usermodel.php
@@ -726,7 +726,15 @@ class UserModel extends Gdn_Model {
       $Join = GetValue('Join', $Options, array('Name', 'Email', 'Photo'));
       $UserPhotoDefaultUrl = function_exists('UserPhotoDefaultUrl');

-      foreach ($Data as &$Row) {
+         // workaround for https://github.com/facebook/hhvm/issues/1482
+         if(is_object($Data))
+               $RowResults = $Data->Result();
+      while(list($Index, $Current) = each($Data)){
+               if(is_array($Current)){
+                       $Row &= $RowResults[$Index];
+               }else{
+                       $Row = $Current;
+               } 
          foreach ($Prefixes as $Px) {
             $ID = GetValue($Px.'UserID', $Row);
             if (is_numeric($ID)) {
diff --git a/applications/vanilla/models/class.categorymodel.php b/applications/vanilla/models/class.categorymodel.php
index b9df951..6c44cb0 100755
--- a/applications/vanilla/models/class.categorymodel.php
+++ b/applications/vanilla/models/class.categorymodel.php
@@ -165,7 +165,16 @@ class CategoryModel extends Gdn_Model {
                }

       $Keys = array_reverse(array_keys($Data));
-      foreach ($Keys as $Key) {
+
+      //workaround for https://github.com/facebook/hhvm/issues/1482
+         if(is_object($Data))
+               $RowResults = $Data->Result();
+      while(list($Index, $Current) = each($Data)){
+               if(is_array($Current)){
+                       $Row &= $RowResults[$Index];
+               }else{
+                       $Row = $Current;
+               } 
          $Cat = $Data[$Key];
          $ParentID = $Cat['ParentCategoryID'];

@@ -301,7 +310,15 @@ class CategoryModel extends Gdn_Model {
     */
    public static function JoinCategories(&$Data, $Column = 'CategoryID', $Options = array()) {
       $Join = GetValue('Join', $Options, array('Name' => 'Category', 'PermissionCategoryID', 'UrlCode' => 'CategoryUrlCode'));
-      foreach ($Data as &$Row) {
+         // workaround for https://github.com/facebook/hhvm/issues/1482
+         if(is_object($Data))
+               $RowResults = $Data->Result();
+      while(list($Index, $Current) = each($Data)){
+               if(is_array($Current)){
+                       $Row &= $RowResults[$Index];
+               }else{
+                       $Row = $Current;
+               } 
          $ID = GetValue($Column, $Row);
          $Category = self::Categories($ID);
          foreach ($Join as $N => $V) {
@@ -1568,4 +1585,4 @@ class CategoryModel extends Gdn_Model {
       return Url($Result, $WithDomain);
    }

-}
\ No newline at end of file
+}
diff --git a/bootstrap.php b/bootstrap.php
index 02ac146..99b4638 100755
--- a/bootstrap.php
+++ b/bootstrap.php
@@ -140,7 +140,7 @@ Gdn::FactoryInstall(Gdn::AliasRouter, 'Gdn_Router');
 Gdn::FactoryInstall(Gdn::AliasDispatcher, 'Gdn_Dispatcher');

 // Smarty Templating Engine
-Gdn::FactoryInstall('Smarty', 'Smarty', PATH_LIBRARY.'/vendors/Smarty-2.6.25/libs/Smarty.class.php');
+Gdn::FactoryInstall('Smarty', 'Smarty', PATH_LIBRARY.'/vendors/Smarty-2.6.28/libs/Smarty.class.php');
 Gdn::FactoryInstall('ViewHandler.tpl', 'Gdn_Smarty');

+++ b/conf/bootstrap.after.php
@@ -0,0 +1,21 @@
+<?php if (!defined('APPLICATION')) exit();
+class Gdn_HHVPD{
+
+       function __construct(){
+               // class destructors are not automatically called in HHVM, 
+               // so will force in shutdown to ensure Config value are saved
+               // also ensures install persisits. 
+               register_shutdown_function(array($this, 'Cleanup'));
+       }
+
+       function Cleanup(){
+               Gdn::Config()->Shutdown();
+       }
+
+}
+
+$HHVPDestructor = new Gdn_HHVPD;
+
+
+
+

The issues addressed in these changed:

this bug that I found

https://github.com/facebook/hhvm/issues/1482

(It would be better if this is addressed by them, given how wide the widespread Traversable this use is in the fraework.)

And also the fact that class destructors are not called automatically in HHVM unless the the instance is explicitly nullified/unset. So I created a shutdown function to ensure Config is Shutdown so that it is saved/persists.

x00

@Kasper it would be interesting to compare compatibility mods / workarounds to get it running on hhvm.

Kasper

@x00: The only change I made to Vanilla was similar to what you did to work around https://github.com/facebook/hhvm/issues/1482. The shutdown function seems like something I'd want to implement as well.

Shadowdare

A lot of people have been switching over to Nginx with PHP-FPM and seeing news on HHVM is interesting. Would there be better performance overall for PHP programs with HHVM's own web server or HHVM-FastCGI with Nginx? For static files, I believe that HHVM-FastCGI with Nginx works more efficiently.

x00

regarding Nginx+HHVP vs Nginx+Php-fpm, according to the official blog in operations with a small computation fpm might be slightly faster but as it ramps up hhvp overtakes. So they do realize the have room for improvement there.

what was more the overriding impression is that HHVM was designed for higher loads in general. from my inti impress it seem to be stable so long as the code itself parses.

I think the main pint is facebook is faster, because their architecture is optimized for hhvp.

The documentation is somewhat lacking however.

Shadowdare

I have a test server which has Nginx 1.4.4, PHP 5.5.7, Percona Server 5.6, and WordPress 3.8 installed. PHP-FPM has been configured with Zend Optimizer+ enabled and XCache installed and configured properly for W3 Total Cache (W3TC) object cache configuration support.

After reading this discussion earlier today, I set up HHVM 2.3.2 on my test server and tried both HHVM by itself with a WordPress configuration guide on the HHVM blog and HHVM-FastCGI with Nginx. I couldn't try out the HHVM-Nightly build from January 6, 2014 because it returns HipHop Fatal error: Unexpected object type stdClass with the date_format function.

I agree that the documentation is somewhat lacking because the rewrite rule for WordPress shown in the guide on the blog didn't work well with some of the permalink settings. After some tweaking of the rewrite rule, I got the posts and pages to resolve correctly and it seemed to load a lot quicker than the initial setup.

Due to the somewhat lacking documentation, I couldn't figure out what the syntax to ignore the /wp-admin/ directory is, so the admin dashboard would not load up properly unless a specific page is specified in the URL like /wp-admin/index.php.

Since Nginx has been shown to handle static resources such as static HTML pages, images, and other files very well and I already had a working WordPress permalink rewrite rule, I am now testing out HHVM-FastCGI with Nginx.

Of course, this is with W3TC disabled. If you use W3TC with either disk or APC cache, the HipHop Warning: Parameter 1 to W3_Plugin_TotalCache::ob_callback() expected to be a reference, value given warning will show up in the log. I presume that the APC functions exist in HHVM, but they do nothing. So, this brings up the question of: is PHP-FPM with optimization and caching methods like Zend Optimizer+ and XCache (used by the PHP program) faster than just HHVM with its SQLite cache and not being able to use most of those caching methods?

I don't have time right now to do actual benchmarks, but so far everything loads fast now according to the average request time based on the auditor results in my web browser. Also, I agree that HHVM seems to be able to handle high load well, but since it's still in buggy development, many important functions aren't supported yet.

For example, if you use SMTP, you will get a f_stream_socket_enable_crypto is not going to be supported error, so you would have to set up the built-in mailer on your server. If you use a Linux distro with exim4, you can follow this tutorial: https://wiki.debian.org/GmailAndExim4#Using_Exim4_to_Send_Messages_through_Gmail

I look forward to what future updates will bring.

x00

this seems a apt discussion

https://github.com/facebook/hhvm/issues/1066

makes me optimistic things are going to improve.