(WIP) notes on scaling Vanilla on AWS EC2

jamesinc

I'm documenting something I've been doing recently, which is moving my forums from a VPS to AWS EC2 t2.micro instances that scale with demand using an Auto-Scaling Group (ASG). The database is hosted on RDS.

My current configuration uses S3FS to mount S3 buckets holding the contents of uploads/ and conf/ into the forum's filesystem.

Currently they sit behind an ELB, however I think I'll move to an ALB and map the uploads directory directly to the relevant S3 bucket

The server itself is rolled using Ubuntu 16.04, S3FS paths and forum codebase fetched out of my Git repo, and then snapshotted into an AMI. This is picked up in a Launch Configuration that is tied to the ASG.

The goal has been to do this without doing any AWS-specific code changes to the forum software, and so far it's been successful.

Will update again when I have the final system configuration sorted out.

jamesinc

Update: I misspoke in the original post; mapping uploads/ to an S3 bucket at path resolution time won't be done with an ALB but rather using CloudFront. My bad, stream of consciousness strikes again.

jamesinc

Update 2: have decided against using an S3 origin in CloudFront as it creates a pathing issue when the same bucket is being accessed by Vanilla on its EC2 instances.

Instead, I've simply created a behaviour for the uploads directory that is more permissive in terms of what it caches.

x00

Do you really want the massive expense of AWS EC2, which is expensive even to test simple things? Investigated this before didn't have very good success with this at all, a big money pit.

I think for anything other then custom design software or nodes that can operate independently, avoid.

There are simpler more responsive options like jelastic (up and down scaling vertical and horizontal) with a better interface. Yes it doe have the feel of a VPS and is PaaS than IaaS, but PaaS is more suitable for this and also they have expanded the software they support over the years. Unless you are doing somethign very custom, then it is a good option for a standard forum. EC2 makes things much more complicated.

Also you can much better manage your costs. Whereas calculating your cost in AWS is a headache.

jamesinc

Yes, no doubt there are simpler solutions, however I am an AWS DevOps engineer by day, so it is very easy to work with for me.

Not enough traffic data yet to see what cost is like, but my modelling showed it being similar price to my VPS setup.

I'm not necessarily recommending AWS as a solution, I'm just using this space to jot down how I am solving various challenges.

x00

OK makes sense. Yep this is advanced stuff for sure.

x00

I'm interested how this turns out becuase I had a lot of issues with this.

Linc

If you have specific notes on issues you run into with the core software supporting this, please file a detailed GitHub issue. I suspect you'll be fine, but this isn't an (exact) scenario we've run ourselves.

jamesinc

One issue I've encountered so far is that if I terminate SSL at the Cloudfront layer, it's not picked up in PHP. This leads to the forum writing links starting with http:// instead of https://. The root cause here is that Cloudfront reports SSL state using a custom header, CLOUDFRONT_FORWARDED_PROTO.

To get around this, I've added a new scheme check to library/core/class.request.php (from line 485)

        // Amazon CloudFront terminated SSL
        if (isset($_SERVER['CLOUDFRONT_FORWARDED_PROTO']) && strtolower($_SERVER['CLOUDFRONT_FORWARDED_PROTO']) == 'https') {
            $scheme = 'https';
        }

Disclaimer to others: find a better way to do this. Maybe rewrite headers in your .htaccess file?

Edit: I decided to submit a pull request for this.

Linc

We've seen a similar pattern using CloudFlare.

I don't think that code snipped is an appropriate way to address this generally. We've done an addon for CloudFlare support using a whitelisted IP range that adds a header but doesn't address the https issue. This use case is a bit more challenging.

jamesinc

@Linc can you elaborate on why this isn't an appropriate solution? Just because it's a vendor-specific header?

An alternative would be, in .htaccess:

SetEnvIf cloudfront-forwarded-proto https HTTPS=on

I thought adding it to class.request.php would be more appropriate, as that's where other forwarding headers are being tested.

Linc

Yes, it's primarily because we don't want vendor-specific kludges in core. I'd at least wanna see a hook put in place where an addon could handle this for AWS sites.

I can't think of a way to exploit switching the scheme by setting that header but putting conditions like that in core is a good way to start making things murky.

x00

this should be done at the server rules level IMO, not really a frameworks job.

Linc

I've asked our ops team for more information about this, since we must do something similar with our loadbalancers.

The correct way to do this is for the intercepting servers to set the X-Forwarded-Proto header, which should in turn set a HTTP_X_FORWARDED_PROTO server variable to https in your server config, which is what we support. I don't know why Cloudfront has chosen to set a proprietary header instead, but it's incorrect.

jamesinc

@Linc said:
I don't know why Cloudfront has chosen to set a proprietary header instead, but it's incorrect.

I think I understand why. A common configuration on Amazon is Cloudfront -> ELB -> EC2. The ELBs can also terminate HTTPS and already use the X_Forwarded_Proto header, so presumably it's just to make it unambiguous.