Day 3: VirtualHost Setup

As we near the proverbial hump-day of our 5 Day plan, a quick glance at where we’ve been thus far:

Because today is a pretty short day (well, it’s a full day, but my time to work on this was limited) I think all we’re gonna have time to tackle is setting up the VirtualHost. This definitely puts a little pressure on for next week (again, a few days off coming up), but we’ll do what we can do!

Off we go:

Setup the VirtualHost

As we stated in the goals, we want to run SSL-only, so I wanted to create a template we could use with the web_app provider we get courtesy of the apache2 cookbook. So, we’ll need a rewrite for anything non-SSL to SSL, and then I’m going to base the rest of the config on the .htaccess file provided by the HTML5 Boilerplate project. So far, I’ve only removed a little bit, so there’s a good change we’ll have to revisit this file later. Here’s our apache-ssl-only-boilerplate.erb:

Day 1: Starting the Server Build (Chef, Vagrant & EC2)

So, if you’re just tuning in, we’re at the start of building a large EC2 instance to run WordPress. You can read the details here. Today we’re going to tackle creating the actual server. To do this, we’re going to use Chef. I wrote a few intro to Chef pieces a few years ago, although they’re getting pretty dated at this point. Luckily, there’s plenty of resources online now for learning (like, this one from Opscode). I also hope that this exercise because a real-world exercise in using Chef for a pretty run-of-the-mill setup. From this point forward, I’ll be writing assumeing a base-level understanding of Chef, my apologies!

The Pre-reqs

First things first, we need some preliminary setup for the tools we’ll need to make this thing work.

Chef Organization

Because we’ve been using more Chef lately at Penn (and best practices have been appearing) I’m starting a new Chef organization for this project. Also, it will keep documenting everything for blogging purposes a lot easier. We keep a template organization git repo internally here that is based on the Opscode base repository with a few adjustments for working with team members here, and a Cheffile for Librarian that has a bunch of our internal cookbooks defined already. If you wanted to follow along at home, the Opscode base should work just fine. I created a new organization on Hosted Chef, configure knife appropriately, and ran a quick test:

% knife client list

Which connects successfully and just shows that my only configured client is the validator, which is correct.

Cloud in Five: WordPress on AWS

I’ve been meaning to get back to blogging a bit, so we’re going to try this format out. More-or-less the goal is to build and test something in the cloud in a week or so (let’s say 5 business days, because I have a few personal days coming up). Can I guarantee it will be a week every time? Nope! I can’t even say that it will be a week this time. But it’s an exercise in technology and writing that I think is worth trying out. So, on to our first challenge:

Build an SSD-backed WordPress on AWS Setup

For a while I’ve been curious about the performance you could squeeze from one of the EC2 SSD backed instances running WordPress. But why, you ask? There’s already companies that specialize in WordPress hosting. True, and we use one of the biggest of them for one of our web properties. We also run a number of sites on a clustered setup that we run internally. Initially, I had thought we would probably do one of those for a much larger project we have in the works, but I’ve got some reservations / lingering doubts:

Control: At the end of the day, this may just be an exercise in me wanting to control our own infrastructure again. Being unable to access all the things I need to effectively debug an issue can be frustrating. Having to convince someone that the problem you’re starting a ticket about does actually exist is frustrating. Feeling like your ticket is being ignored for large stretches of time is frustrating. I know, you get what you pay for, and there are trade-offs to not having to worry about servers in the middle of the night.

SSL: I want to use SSL exclusively. I know this isn’t the norm yet, but I think that if you have to go back and forth on a site, in this day and age, you should run completely secured. Even if you don’t have people authenticating, there still may be reasons why you want to do this. To run this way, I’d like to see CDN assets distributed this way, as well as enable things like Google’s SPDY protocol. This is where you start to lose some bonus features of the hosting providers, as they don’t all support SSL CDN, or SPDY. Additionally, just doing in some places starts the end-around of all the caching systems people tend to put in place for WordPress. And if SSL doesn’t…

Users: It seems to me that the most common approach to optimize WordPress to scale, quite naturally, confronts the most common use case: anonymous readers. Whether it’s in WordPress itself through the use of the popular caching plugins, or through a layer of nginx or varnish caches, one of the first configuration pieces you tend to see is “skip me if you’re logged in”. But what if you want to encourage people to create accounts and engage? What if those are actually the most important users you have? I think we need to provide them a experience that’s on par, if not better, then, say the Googlebot (although I do realize that we can’t ignore the Googlebot, it requires speed these days as well!)

Simplicity: Although I’m part of a large organization, our team is actually rather small, and what I’ll be proposing should, with the aid of configuration management, be a really simple setup compared to the distributed complexity (and latency!) you tend to end up with when you’re sharing space in, say, a VMWare environment with a variety of other users. I’m also hopeful that AWS reduces complexity by turning parts I would normally have to manage on a device (like say, load balancers) into a service I can configure via API. At the very least, it’s going to let me spin and and destroy computers loaded up with SSDs without any major time or monetary investment!

So, now that we’re through the doubts and goals, here’s my proposal:

Let’s build an optimized WordPress setup based around EC2’s hi1.4xlarge (SSD-backed) instance type.

Is it cheap? Nope. This isn’t a bargain basement setup, I want to take one of the best machines they offer and see what it can do. Here’s the specs (from Amazon’s list):

High I/O Quadruple Extra Large Instance

  • 60.5 GiB of memory
  • 35 EC2 Compute Units (16 virtual cores*)
  • 2 SSD-based volumes each with 1024 GB of instance storage
  • 64-bit platform
  • I/O Performance: Very High (10 Gigabit Ethernet)
  • Storage I/O Performance: Very High*
  • EBS-Optimized Available: No**
  • API name: hi1.4xlarge

Sounds powerful. So, what are we enabling? I’m hopeful that we’re going back to the good old days of one server handling all my stuff! DB Server on the same box as the assets as the webserver. Old fashioned, I know. I’ll be jettisoning what I could envision as a terribly complex setup of load balancers, web servers, NAS, db cluster, and memcached servers. So, we’ve got a terrabyte drive to hand to the database, and another to hand to the webserver. Of course, we’ll need redundancy, and that also needs to be part of this project, but the theory is that when everything is going great this box should be able to stand up to everything we could potentially throw at it (note: we’ve got years of our traffic data, so we won’t be guessing).

Approach

Now that we’ve outlined the goals, we need an approach. Here are the steps I’m envisioning (not necessarily in order), and a few bullets about each:

  • Using Chef, configure the server.
    • Share as much as possible about the setup
    • Have a Vagrant setup to use for testing the recipes
    • Install/configure MySQL to use one of the SSDs to its data store
    • Install/configure Apache & PHPFPM to serve from the other SSD
    • We’re going to run with the Ubuntu instances, because that’s where I’m most comfortable.
  • Install/Configure WordPress
  • Build it into the AWS architecture.
    • Use an Elastic Load Balancer
    • Setup CloudFront CDN
  • Add redundancy / failover / backup
    • Leverage EBS/S3 for persistent storage?
    • Build a backup (and restore) process
    • Setup a hot standby (probably not SSD backed)
  • Benchmark / Tune
    • WordPress Caching
    • PHP Tuning
    • MySQL Tuning
    • Build a Jmeter test suite

So, start your timers, we’re off! By tomorrow I hope to have at least tackled the complete server setup.

Average Apache Process Memory Usage

Exactly as the title suggests! A little shell snippet:

ps aux | grep apache2 | grep -v pts | awk '{ tot += $6; procs += 1; print $2,$6,$11 } END { print "TOTAL: ",tot,"/",procs,"=",tot/procs }'

You may need to swap out “apache2” for “httpd” depending on your server setup. Why did I need this? At the moment I’m monitoring the average memory use, and trying to determine how it changes as I adjust the max-requests that a prefork process can serve before it’s recycled. Once I have that number, I’ll adjust the maximum amount of processes accordingly to prevent swapping.

Chef: Rounding the Bend

Since you’ve already ready my intro to Chef, and well as my article on getting started (right?) we’re going to do away with the long recap and instead give you a “Preiviously on 24” style list:

  • You know what Chef is
  • You’ve installed and configured Chef
  • You’ve created a node
  • You’ve created a role
  • You’ve downloaded and used some existing cookbooks, changing default settings as necessary
  • Jack Bauer has muttered something about not having enough time

You can actually accomplish a fair amount just using the methods above, but eventually you’re going to need some functionality that goes beyond what the one-size-fits-all cookbooks can provide. A common reason that I’ve encountered for this is some recipes put in place a structure where you can easily extend them with your own templates, so we’ll look at that next.

Extending Recipes with Site Cookbooks

If you’ve been following along so far, you’ve got an empty site-cookbooks folder in the root of your chef repository (if you don’t, go ahead and create one). How does this work? Basically, you create a structure in there to mimic that of the cookbooks folder, and when knife uploads the cookbook, it uses the files it finds in site-cookbooks instead of those in cookbooks. To clarify: you only need to create the folders and files that you plan on overriding, or files that don’t exist in the cookbook. So, why don’t you just edit the cookbook itself? Actually, if it’s your own cookbook, that’s what you should do, but I’m talking about the case where you’re extending one you’ve found online. Granted, you could just as easily update it in the cookbooks folder, using git to manage your changes, but I personally thing it’s cleaner to use the site cookbooks method. This way, you can keep track of what you’ve written and/or changed. Additionally, if the author of the original enhances their cookbook while you’re off conquering the world with your new Chef setup, I think it’s easier to just replace the original in cookbooks, because you can use version control to see what’s changed and preserve all your own customizations.

So, on to the example. At the moment we have some basic functionality happening in our base_server.rb role file, but now we want to lock the machine down with an iptables firewall. Luckily, there’s a cookbook for that called, appopriately, iptables, so let’s vendor that cookbook with knife:

knife cookbook site vendor git -d

If you glance through the source of the cookbook, you’ll see that it’s creating a /etc/iptables.d directory, in which it will be placing rules, these rules are created by template files with a ‘definition’ call. Finally, the machine is locked down to only accept connections defined in those rule files. Two things worth noting here: First, this is our first look at a definition in Chef, so that warrants an explanation. To quote the chef wiki: “Definitions allow you to create new Resources by stringing together existing resources.” There’s some good examples on there as well, but we’re going to procede with ours. Here’s the relevant source from the definitions/iptables_rule.rb file:

define :iptables_rule, :enable => true, :source => nil, :variables => {} do
  template_source = params[:source] ? params[:source] : "#{params[:name]}.erb"
  
  template "/etc/iptables.d/#{params[:name]}" do
    source template_source
    mode 0644
    variables params[:variables]
    backup false
    notifies :run, resources(:execute => "rebuild-iptables")
    if params[:enable]
      action :create
    else
      action :delete
    end
  end
end

In short, it’s creating a new file in the iptables.d folder using a source based either on the name of the definition we’re creating or one we pass as a parameter, and enabling it. Why is this handy? Because instead of putting all that code in our recipe, we get a nice reusable snippet. Here’s how this is used in the default.rb recipe:

iptables_rule "all_established"
iptables_rule "all_icmp"

Sure beats writing all that code up there over and over again. Also, in Ruby fashion, it’s really easy to read. (I read: “Create an iptables rule for all established connections and for ping”). Just to fill in the last piece of the puzzle, here’s the template all_icmp.erb (which, if you’re following, is called by the iptables_rule definition):

# ICMP 
-A FWR -p icmp -j ACCEPT

Now, this is all well and good, except that most people are going to need rules for more then just established connections and ping. That brings us to the second thing worth noting about this cookbook: we need more templates! Enter: site cookbooks. For our example below, let’s extend this cookbook to include rule for a web server. To begin, create the appropriate structure (run from the site-cookbooks folder):

mkdir -p iptables/templates/default/
mkdir -p iptables/attributes
mkdir -p iptables/recipes

Let’s start with some simple templates for iptables including rules for http and https traffic from anywhere, as well as one for ssh (we do want to be able to administrate the box remotely, right?). Here’s the templates/default/all_http.erb file:

# HTTP
-A FWR -p tcp --dport 80 -j ACCEPT

Next, the template/default/all_https.erb file:

# HTTPS
-A FWR -p tcp --dport 443 -j ACCEPT

And finally, template/default/all_ssh.erb:

# SSH
-A FWR -p tcp --dport ssh -j ACCEPT

I suppose you could combine those into one template, but at this level I prefer to keep things as granular as possibly, so we can mix and match down the road. Now, let’s get tricky and try to apply one of the other things we know about Chef: the templates can be dynamic. So, let’s throw in some rules for locked down versions of those same services. Here’s templates/default/network_http.erb:

# HTTP Locked Down
<% @node&#91;:iptables&#93;&#91;:ssh&#93;&#91;:addresses&#93;.each do |address| %>
-A FWR -p tcp -s <%= address %> --dport 80 -j ACCEPT
<% end %>

And to match, templates/default/network_https.erb:

# HTTPS Locked Down
<% @node&#91;:iptables&#93;&#91;:ssh&#93;&#91;:addresses&#93;.each do |address| %>
-A FWR -p tcp -s <%= address %> --dport 443 -j ACCEPT
<% end %>

Finally, templates/default/network_ssh.erb:

# SSH Locked Down
<% @node&#91;:iptables&#93;&#91;:ssh&#93;&#91;:addresses&#93;.each do |address| %>
-A FWR -p tcp -s <%= address %> --dport 22 -j ACCEPT
<% end %>

Now we have a nice base to work with, some iptables templates we can mix and match as necessary. Of note: we’re introduced a node variable to the network rules, so we have to remember to cover that in our recipe. Also worth noting: most programmers are going to start to see repetition above, and may feel tempted to create an all_tcp rule with an additional “port” variable. Don’t let me stop you, it might make sense. Two reasons why I didn’t: 1) Down the road there could be more complicated services I’m defining in templates that could have multiple iptables rules, and I would prefer to have them in one template so that.. 2) each service remains a granular object, easy to read when defined in the recipe. I’m willing to sacrifice a little bit of repetition if it makes my recipes easier to read and administrate. Again, personal choice, and you’re a rugged individualist, so do your own thing if it makes you happy!

Moving on, I think recipes don’t have to be as granular (because we did so in the templates), so let’s create a recipe using these templates for “iptables::web” encompassing http and https, as well as a decision on which rules to use based on a node variable. Here’s our recipes/web.rb:

# Have we decided to lock down the node?
if node[:iptables][:web][:addresses].empty?
  # Use the all_ rules
  iptables_rule "all_http"
  iptables_rule "all_https"
  # Disable the network rules
  iptables_rule "network_http", :enable => false
  iptables_rule "network_https", :enable => false
else
  # Use the network rule
  iptables_rule "network_http"
  iptables_rule "network_https"
  # Disable the all traffic rules
  iptables_rule "all_http", :enable => false
  iptables_rule "all_https", :enable => false
end

Note: if we don’t do that enable => false bit, the file will remain on the server even if we remove the line later. Strange, I know. Moving on, one for ssh, “iptables::ssh” (recipes/ssh.rb):

# Have we decided to lock down the node?
if node[:iptables][:ssh][:addresses].empty?
  # Use the all_ssh rule
  iptables_rule "all_ssh"
  # Disable the network ssh rule
  iptables_rule "network_ssh", :enable => false
else
  # Use the network rule
  iptables_rule "network_ssh"
  # Disable the all traffic rule
  iptables_rule "all_ssh", :enable => false
end

Pretty simple right? If we’ve defined addresses on the node, use the lock down rules, otherwise, open the port up to the world. Finally, because we’ve introduced new node attributes, we need to create two attribute files to correspond to our new recipes. First, attributes/web.rb:

# Web Traffic Allowed Networks (IP/NETMASK)
default[:iptables][:web][:addresses] = Array.new

And one for SSH as well (attributes/ssh.rb):

# SSH Allowed Networks (IP/NETMASK)
default[:iptables][:ssh][:addresses] = Array.new

Awesome. Now the big finish: let’s add the ssh recipe to our base server, then create a new role for a web server that applies our “base server” configuration and then locks down the machine. In the real world, this would be part of a role that also configures your web servers. Coincidentally, the apache2 cookbook uses a very similar mechanism so if you’re anxious you can move ahead using the versatile “web_app” definition in that cookbook. Now our roles/base_server.rb looks like this (locking down ssh to an arbitrary subnet):

name "base_server"
description "Common Server Base Configuration"
run_list(
  "recipe[fail2ban]",
  "recipe[git]",
  "recipe[vim]",
  "recipe[ntp]",
  "recipe[iptables]",
  "recipe[iptables::ssh]"
)
default_attributes(
  "ntp" => {
    "servers" => ["timeserver1.upenn.edu", "timeserver2.upenn.edu", "timeserver3.upenn.edu"]
  },
  "resolver" => {
    "nameservers" => ["128.91.87.123", "128.91.91.87", "128.91.2.13"],
    "search" => "wharton.upenn.edu"
  },
  "postfix" => {
    "relayhost" => "SOME.RELAY.SERVER"
  },
  "iptables" => {
    "ssh" => { "addresses" => ["128.91.0.0/255.255.0.0", "130.91.0.0/255.255.0.0"] }
  }
)

Great, now lets create a more task specific role for a web server, building off of that base role. Because I want to prove it works, why don’t you go and download the cookbook for apache2. I’ll wait. Ok, let’s include it to do a base install so you can check easily. Here’s the new roles/web_server.rb role file:

name "web_server"
description "Generic Web Server"
run_list(
  "role[base_server]",
  "recipe[apache2]",
  "recipe[apache2::mod_ssl]",
  "recipe[iptables::web]"
)

There you have it. Note the “role[base_server]” line in run_list, that includes all the good stuff we have in our base server role (obvious, right?) Upload the cookbooks, update the roles, and try assigning the new web server role to a test node and you’re rolling!

Housekeeping

To complete this example, I need to include a few more things I did. This falls into a grey area for me because I actually think this belongs in the original recipe, which is something you’re likely to encounter as well as you extend cookbooks: When to override and when to patch? My approach is going to be fix in site cookbooks, and then submit a patch back to the author, if it gets included, great, update the cookbook and remove it from the site cookbook. Because I haven’t gotten this patch back to Opscode yet, and I want you to be able to use my examples if you’d like, here was the last piece of the iptables functionality I had to change.

The problem: iptables weren’t persisting through reboot. A pretty big issue! My solution was to add a script to the ifup.d folder that would call the rebuild-iptables script (created by the iptables rule) when network interfaces come up. To do this required a short template (templates/default/iptables.erb):

#!/bin/sh
/usr/sbin/rebuild-iptables

And a recipe to place it (recipes/on_boot):

if platform?("debian", "ubuntu")
  # Add a script to restore the rules on boot
  template "/etc/network/if-up.d/iptables" do
    source "iptables.erb"
    owner "root"
    group "root"
    mode 0755
  end
end

Now, include the “iptables::on_boot” recipe in your base_server.rb role and you’re good to go! In case you’re curious, I haven’t submitted the patch yet because I haven’t had time to test it on anything but the ubuntu boxes I’m running (thus the “if platform?” conditional), and I’d prefer to submit a patch that also works in the RHEL space as well. Opsode, don’t let this stop you from taking this and running!

Seems to me we’ve gone quite long again, so we’ll wrap this one up. Still to come: the wild wild west, creating your own cookbook in it’s entirety from scratch! Until then, enjoy extending recipes. At this point if you’ve been following along you’re already wielding a pretty powerful configuration tool!