HackerOne users: Testing against this community violates our program's Terms of Service and will result in your bounty being denied.
Vanilla Performance Discussion
I've just done some tests, comparing Vanilla vs. MyBB vs. FluxBB and I found out that although I love the concept and the community features of Vanilla, it's 10x slower compared to FluxBB when under heavy load. Is there anything I can do to make it faster? I'm using APC what is used by FluxBB hence why it can be so fast. Isn't there a mod or patch to make Vanilla use APC?
I've written my detailed results here:
http://www.lowendtalk.com/discussion/5323/vanilla-vs.-fluxbb-vs.-mybb-siege-test
0
Comments
APC you just turn on in php, you can use memecache. I recommend clearing activity periodically (this is something fixed in 2.1). Apache is slow, Vanilla performs better with nginx.
Performance is a complicated subject so I'm not going to get into huge detail, but when you are doing benchmarking all you are doing is accessing how it performs (using that request method) on that setup. You could have performance acceleration software that would actually slow down your site if you don't have enough memory or capacity.
grep is your friend.
I am on nginx / php-fpm / APC. Normally the website isn't slow, it's just slow compared to other forums in siege-test.
Vanilla response -c100: 1.94 secs
Vanilla response -c200: 2.03 secs
MyBB response -c100: 0.25 secs
MyBB response -c200: 0.73 secs
FluxBB response -c100: 0.04 secs
FluxBB response -c200: 0.08 secs
Are you running this benchmark from a clean install of each forum? I've just finished working on my box and the plain Vanilla 2.0 -c100 test is giving me 80.7 transactions per second on a Linode 512, which is three times faster than your results.
In your test on similar hardware Flux delivers 193 trans/sec which is superb, but it's not radically (10x) better than Vanilla. Considering the functionality of Vanilla and Garden it's very fast indeed, provided you've got a properly baked server stack.
Activity tend to slow thing down as it grows, prune careful but periodically with a cron job, especially with 2.0
You really don't need old activity that is transient. Just some messages.
grep is your friend.
Just for the sake of evaluating the benchmark posted in this thread I used Vanilla straight out of the box. This is without even minifying the CSS/JS, let alone disabling Activity or other software optimisations.
A "Prune Activity" Plugin would be great. With some timeframe options ("Keep last 365 days", "Keep last 6 months").
What page of each site were you hitting, and was the same amount of data (or any?) loaded into each forum before these benchmarks were done?
My analogy is a post saying "Guys, I just bought Car A and Car B. When I floored them both, Car A got to the end of the road faster." My first questions would be: Did you put either car past first gear? Was there turning involved? Can you tell us about this road? Can you offer us any context or insight about your results beyond "this thing went faster when I typed in this command"?
I'm obviously keenly interested in Vanilla's performance (and frankly very prepared to accept it isn't nearly the fastest kid in town at stock config - that hasn't been our foremost concern since we provide customized hosting for it which speeds it up considerably) but I don't see useful levels of info in what's been presented so far.
Unfortunately, activities can't be disabled in Vanilla. I wish they could be.
Im moving a couple of my big forums to vanilla. Love the architecture, but it is indeed slow compared to other forums
Im going to bite the bullet and move to vanilla anyway.
I hope version 3.x is all about performance improvements.
I just imported one of my big forums into vanilla and apache benchmarked it against the original old phpbb 2.0 version of the forum. Used 100 concurrent clients for the test.
phpbb 2.0: 303 requests per second
vanilla: 22 requests per second
I have tuned the hell out of my MySql database running on an SSD drive for this move, and I still get these low numbers. This is a pretty hefty dedicated server. Quad core xeon processors, lots of ram, etc.
Vanillas queries look super heavy and beat the hell out of the MySql engine.
I may have to custom hack to get the performance I want.
I hope the vanilla dev team really take this seriously. It just doesnt scale well right now.
Thats a big deal breaker for a lot of sites.
@jackjitsu What version did you test?
I recommend Varnish.
Version 2.0.18.8
Varnish will not help. Im already running on a highly tuned NGINX install with fast_cgi caching.
When I enable nginx caching on my wordpress sites I get 25,000 requests per second.
When I try this with Vanilla Forums it fails because the home page pushes out a Cookie and a no-cache header, both of which prevents caching by caching engines like nginx and varnish.
I did some more load testing with slow queries enabled in MYSQL to catch what goes slow under load. To my surprise, no query exceeded 3 seconds.
I was running HTOP and watching. Looks like Vanilla Forums is really heavy on the CPU's, not so much the DB which was my original assumption.
I NEED MORE POWER! LOL.
yes vanilla's queries aren’t that bad.
Never go on what a query looks like, optimised queries can look pretty heavy, and the optimisation is not just in the query itself.
grep is your friend.
If you would like to know what code is causing the performance issues, you could try using XDebug profiler and then a tool like KCacheGrind to inspect it, but beware that it may take a significant amount of time to both to collect and inspect the data.
My shop | About Me
@jackjitsu You will find 2.1 makes the necessary adjustments to run Varnish and probably fixes many of the queries you're concerned about. Varnish has been a tremendous asset to us.
We've done a lot of scaling optimizations which took load off of io, but has raised the load on cpu. This is due to having more calculation in the app rather than the database.
We will be improving the cpu performance in coming versions. We do have some inefficiencies due to some options in the code that don't get much use and we will be removing support for some of these edge cases. I'll be talking about this after 2.1 gets its final release.
One of our cpu bottlenecks though is rendering and that has to do with our theming flexibility. This is just going to be a fact of a Vanilla forum and I think it's a fair trade-off. We have a lot more power than phpbb or mybb and we're designed to work on more modern hardware.
We build our software for scale. This means that our software won't perform as well on lesser hardware or on small communities. However, when the community is large we have the ability to parallelize a lot of the code and as a result we can run massive communities.
I havent dug into the code yet to review the architecture yet. I guessed the rendering/templating was where most of the time is spent.
Everything is a trade off, and im willing to trade some CPU for all the awesome hooks and plugin architecture Vanilla provides. I installed a bunch of plugins, then deleted the plugin directory and the forum still ran. PhpBB would explode instantly.
A loosely coupled plugin architecture is an awesome thing, and worth some trade offs, but 10X CPU requirement is a bit much.
Hopefully this will get better with time.
Looked into Varnish.... now I see the trick used to make it work.
Varnish is being configured to IGNORE no-cache headers for at least 60 seconds.
Basically micro-caching. Which you can do in nginx as well.
sigh In any case, I will have to use nginx or varnish as a no-cache header ignoring proxy. I dont like this additional complexity
# Adjusting Varnish's caching - hold on to all cachable objects fo 24 hours.
# Objects declared explicibly as uncacheable are held for 60 seconds, which
# helps in the event of a sudden ridiculous rush of traffic.
if (beresp.ttl < 24h) {
if (beresp.http.Cache-Control ~ "(private|no-cache|no-store)") {
set beresp.ttl = 60s;
}
else {
set beresp.ttl = 24h;
}
}
OK.... so I did the equivalent of Varnish but with NGINX. No need for more software ;-)
Implemented a microcache using NGINX (Varnish replacement), proxying to the real NGINX instance on the back end port 8080. Used a tiny 2 second cache.
Went from 22 requests per second to 20,000.
Feeling better about this now.
NGINX users really should not use Varnish. Using NGINX<--->NGINX natively is a lot easier/faster.