Amtrak, B-Movies, Web Development, and other nonsense

Tag: AWS

Apache and HTTP/2 on AWS

I’ve spent the last few months building up a generic highly-available EC2 stack for our various on-premises applications. We’re using Apache as the webserver on EC2 since that’s what we’re familiar with. An interesting wrinkle is that, out-of-the-box, cURL and Safari didn’t work with WordPress running on this stack. cURL would return this cryptic error message:

curl: (92) HTTP/2 stream 1 was not closed cleanly: PROTOCOL_ERROR (err 1)

Safari refused to render the page at all, with this equally (un)helpful message:

"Safari can't open the page. The error is "The operation couldn't be completed. Protocol error" (NSPOSIXErrorDomain:100)"

Okay, that’s not great. Chrome and Firefox were fine; what’s going on?


Let’s take a step back and discuss the architecture. We have a CloudFront distribution forwarding HTTP and HTTPS requests to an Application Load Balancer in a public subnet. It supports both HTTP/1.1 and HTTP/2. It’s forwarding all traffic over port 443 to the EC2 instances in a private subnet. The EC2 instances are responsible for the TLS termination. Apache on the EC2 instances supports HTTP/1.1 and HTTP/2.


I started with the cURL problem, which led to a number of blind alleys before someone (I forget where) suggesting using the --http1.1 flag which confirmed that the problem was an HTTP/2 issue. That was helpful to a point, but with separate pieces of infrastructure in the mix–CloudFront, ALB, and EC2–I wasn’t sure where the underlying problem lay.

I backed out, and started researching the Safari failure mode, and ran across a good discussion on Serverfault which explained the messages from Safari and cURL. The underlying issue is that Apache is sending an h2c header over a connection that is already an HTTP/2 connection. This is invalid under RFC 7540; the degree to which clients respect that varies widely.  Given that in our configuration everything with the client is over TLS there’s no reason to send that header in the first place.


All signs pointed to Apache. We ship a slightly modified version of the default Apache configuration from the Amazon Linux 2 AMI. The HTTP/2 module is enabled by default, with the following configuration:

<IfModule mod_http2.c>
    Protocols h2 h2c http/1.1

There it is: h2, h2c, and http/1.1. Per the Apache documentation if you’re offering HTTP/2 and TLS your Protocols line should omit h2c. I made that change, and also explicitly unset the upgrade header:

<IfModule mod_http2.c>
    Protocols h2 http/1.1
    Header unset Upgrade

With that change and an Apache restart all is well. The lesson here is that the default Apache configuration is generally sane for most use cases, but you probably need to go through it and ensure that it’s reasonable for your use case. We didn’t have much experience with HTTP/2 at Lafayette prior to his project, else we’d have caught this sooner.

Creating an AWS Backup Plan using CDK

I’ve been doing a lot with the Amazon Web Services Cloud Development Kit (CDK) over the last few months. CDK lets you define all your infrastructure in code, such as Typescript, which is then compiled into a CloudFormation template. It’s pretty cool, but it’s also a new production changing rapidly and the documentation hasn’t always kept up. Below are my notes for setting up a backup strategy for an Elastic File System (EFS) using Amazon Backup.

High-level overview

With Amazon Backup, you will define a Backup Plan with one or more rules targeting various resources, and then store those backups in a Backup Vault. This means you’re going to need the following:

  1. A Backup Vault
  2. A Backup Plan
  3. A Backup Selection
  4. A resource to be backed up. For this example, an EFS instance.

You’re also going to need an Identity and Access Management (IAM) role with sufficient permissions to provision all the backup infrastructure.

IAM permissions

This was the trickiest aspect of the implementation. CDK build failures do not generally indicate which permission failed; sometimes they don’t even indicate which resource was in play. I encountered this error repeatedly trying to create a Backup Vault:

 2/16 | 3:03:21 PM | CREATE_FAILED | AWS::Backup::BackupVault | &amp;amp;lt;VaultName&amp;amp;gt; Insufficient privileges to perform this action. (Service: AWSBackup; Status Code: 403; Error Code: AccessDeniedException; Request ID: &amp;amp;lt;id&amp;amp;gt;)

Not much to go on, and especially puzzling given that I’d given my IAM user the backup:CreateBackupVault permission. Amazon has a page describing the permissions involved to manage AWS Backup programmatically, and there are actually seven different permissions that you need. Confusingly, several of the ARN examples given on that page are incorrect. To create the Backup Vault, the Backup Plan, the Backup Selection, and delegate the backup role correctly, I eventually crafted this policy snippet:

            "Sid": "BackupPolicy",
            "Effect": "Allow",
            "Action": [
            "Resource": [
            "Sid": "BackupStoragePolicy",
            "Effect": "Allow",
            "Action": [
            "Resource": "*"

Something to note here is that you can’t qualify everything by ARN, not easily anyway. The generated ARNs for backup plans appear to be completely random.

CDK code

This example presupposes that you have an EFS filesystem defined elsewhere, named efsUploads. The following code creates a Backup Vault, a Backup Plan, and targets that EFS filesystem by ARN. It also creates an IAM service role for running backups. The backup rules create daily backups in warm storage, retained for 35 days, and monthly backups in cold storage, retained for a year.

    const efsVault = new backup.CfnBackupVault(this, opts.siteName + 'BackupVault', {
      backupVaultName: opts.siteName,
    const efsBackupRole = new iam.Role(this, opts.siteName + 'EFSBackupRole', {
      assumedBy: new iam.ServicePrincipal(''),
      managedPolicies: [
    const efsBackup = new backup.CfnBackupPlan(this, opts.siteName + 'EFSBackupPlan', {
      backupPlan: {
        backupPlanName: opts.siteName + 'EFSBackupPlan',
        backupPlanRule: [
            ruleName: opts.siteName + 'DailyWarmBackup',
            lifecycle: {
              deleteAfterDays: 35
            targetBackupVault: efsVault.attrBackupVaultName,
            scheduleExpression: 'cron(0 8 * * ? *)',
            ruleName: opts.siteName + 'MonthlyColdBackup',
            lifecycle: {
              deleteAfterDays: 365,
              moveToColdStorageAfterDays: 30
            targetBackupVault: efsVault.attrBackupVaultName,
            scheduleExpression: 'cron(0 8 1 * ? *)',
    const efsBackupPlanSelection = new backup.CfnBackupSelection(this, opts.siteName + 'EFSBackupPlanSelection', {
      backupPlanId: efsBackup.attrBackupPlanId,
      backupSelection: {
        iamRoleArn: efsBackupRole.roleArn,
        selectionName: opts.siteName + 'EFS',
        resources: [
          'arn:aws:elasticfilesystem:' + this.region + ':' + this.account + ':file-system/' + efsUploads.ref,

There’s no way (that I know of) to get the ARN of the EFS instance programmatically, so you have to construct it. The AWS Backup library in CDK is still in developer preview, and all the constructs are still pretty low-level.