Apache and HTTP/2 on AWS

I’ve spent the last few months building up a generic highly-available EC2 stack for our various on-premises applications. We’re using Apache as the webserver on EC2 since that’s what we’re familiar with. An interesting wrinkle is that, out-of-the-box, cURL and Safari didn’t work with WordPress running on this stack. cURL would return this cryptic error message:

curl: (92) HTTP/2 stream 1 was not closed cleanly: PROTOCOL_ERROR (err 1)

Safari refused to render the page at all, with this equally (un)helpful message:

"Safari can't open the page. The error is "The operation couldn't be completed. Protocol error" (NSPOSIXErrorDomain:100)"

Okay, that’s not great. Chrome and Firefox were fine; what’s going on?

Architecture

Let’s take a step back and discuss the architecture. We have a CloudFront distribution forwarding HTTP and HTTPS requests to an Application Load Balancer in a public subnet. It supports both HTTP/1.1 and HTTP/2. It’s forwarding all traffic over port 443 to the EC2 instances in a private subnet. The EC2 instances are responsible for the TLS termination. Apache on the EC2 instances supports HTTP/1.1 and HTTP/2.

Troubleshooting

I started with the cURL problem, which led to a number of blind alleys before someone (I forget where) suggesting using the --http1.1 flag which confirmed that the problem was an HTTP/2 issue. That was helpful to a point, but with separate pieces of infrastructure in the mix–CloudFront, ALB, and EC2–I wasn’t sure where the underlying problem lay.

I backed out, and started researching the Safari failure mode, and ran across a good discussion on Serverfault which explained the messages from Safari and cURL. The underlying issue is that Apache is sending an h2c header over a connection that is already an HTTP/2 connection. This is invalid under RFC 7540; the degree to which clients respect that varies widely.  Given that in our configuration everything with the client is over TLS there’s no reason to send that header in the first place.

Resolution

All signs pointed to Apache. We ship a slightly modified version of the default Apache configuration from the Amazon Linux 2 AMI. The HTTP/2 module is enabled by default, with the following configuration:

<IfModule mod_http2.c>
    Protocols h2 h2c http/1.1
</IfModule>

There it is: h2, h2c, and http/1.1. Per the Apache documentation if you’re offering HTTP/2 and TLS your Protocols line should omit h2c. I made that change, and also explicitly unset the upgrade header:

<IfModule mod_http2.c>
    Protocols h2 http/1.1
    Header unset Upgrade
</IfModule>

With that change and an Apache restart all is well. The lesson here is that the default Apache configuration is generally sane for most use cases, but you probably need to go through it and ensure that it’s reasonable for your use case. We didn’t have much experience with HTTP/2 at Lafayette prior to his project, else we’d have caught this sooner.