WordPress and partial content

Eighteen months ago we had an anomalous problem where video playback didn’t work on some, but not all, of our WordPress multisites. Videos wouldn’t play, or would play but wouldn’t seek. The problem was confined to local uploads embedded in a page. Videos from YouTube played fine; if you viewed the video directly playback worked as expected.

The problem turned out to be long-standing issue with how ms-files.php served up files from pre-WordPress 3.5 multisites. Solutions had floated around for years. Our problem was describing the problem with enough specificity to actually find the right solution.

Past is prologue

Serving streaming video requires that the video, server, and web browser all play nice together. Years before we’d run into a problem with videos not “seeking” (letting you jump around) because older versions of Apache didn’t support partial content. That was solved by adding the Accept-Ranges header to our default Apache configuration:

Header set Accept-Ranges bytes

After WordPress allowed native audio and video formats in WordPress 3.6, we went on a long quest to identify the “best” set of codecs for people on our campus to use. We eventually landed on the right cross-browser solution, MP4 (H.264, AAC).

We also had problems with video playback controls not working because Apache didn’t support SVG out of the box. That was another quick fix in the default Apache configuration:

AddType image/svg+xml svg svgz
AddEncoding gzip svgz

Maybe it’s Chrome! Or Apache!

This is all to say that we knew video was hard, but that we thought we had the problem solved. Hence our puzzlement when it manifested in Fall 2015, but only on certain installations. Other installations on the same server worked fine. We went back over all our earlier fixes and verified that yes, they were in place. The videos were encoded correctly.

We lacked adequate nouns to describe our problem. Video playback failing, even with qualifiers about MP4 videos or Chrome, will give you the kitchen sink. We chased the idea that it was a Chrome bug for a while, leading us to Chrome won’t play certain mp3 files and endless canceled http requests when playing an mp4 in a video tag.

The one slender reed we had was a strange response code from the server:

$ curl -H Range:bytes=16- -I http://path/to/site/files/2015/12/My-First-Project.webm
HTTP/1.1 200 OK
Date: Fri, 11 Dec 2015 00:57:44 GMT
Server: Apache/2.2.15 (Red Hat)
X-Powered-By: PHP/5.3.3
Content-Length: 703864
Last-Modified: Wed, 09 Dec 2015 15:47:56 GMT
ETag: "4c74914a4fec74d7af8a5bf3846a96d6" 
Expires: Sun, 10 Feb 2019 10:44:24 GMT
Vary: Accept-Encoding
Accept-Ranges: bytes
X-UA-Compatible: IE=Edge
Connection: close
Content-Type: video/webm

We used cURL to simulate a streaming request by requesting a byte range. A 200 status code with Accept-Ranges: bytes set is nonsensical—it ought to be HTTP/1.1 206, for partial content. We knew what was wrong, but not why. For a while we thought it was an Apache bug, to the point that we were on the verge of inspecting the source used to compile Apache on our production box. Cooler heads prevailed after we proved that the correct header was returned when you bypassed WordPress and hit the file directly.

It was WordPress all along

Finally, I was able to recreate the problem in a fresh WordPress install which started with a pre-WordPress 3.5 multisite. Prior to 3.5 uploads were stored in wp-content/blogs.dir. For security reasons, these files were not served directly but via wp-includes/ms-files.php, courtesy of a rewrite rule in .htaccess. The ms-files.php handler does many bad things, but chief among them is setting an arbitrary cache expiration of a little over three years (100000000 seconds), and reading the file into memory and serving it with PHP’s internal readfile() function. This why we were getting the 200 status code.

One way to solve this would be to deprecate ms-files.php, and Mika Epstein wrote a good guide on how to do that. On large sites this could be pretty disruptive, and we had four multisite installations with 2000+ sites affected. That’s a big hammer when a small number of uploads are actually broken.

Fortunately, there’s a simpler solution. Now that we knew our problem was best described as “ms-files.php screws up partial content” we found lots of resources. The Apache module mod_xsendfile can be used to send proper headers when serving files via PHP. You need to install that, then add some configuration to any virtual host which needs the capability:

        XSendFile on
        XSendFilePath /path/to/your/blogs.dir

Finally, you need to add a line to your wp-config.php so that WordPress takes advantage of this capability:

define('WPMU_SENDFILE', true);

We didn’t find it necessary to modify our .htaccess file and we haven’t noticed any performance issues.

Recap

We battled this issue for months, off and on, hampered by our inability to articulate exactly what was wrong and misled by past experience with similar yet different issues. Long term, it serves as a reminder of why we encourage our faculty, staff, and students to hosts videos on external platforms.

Featured image by Paulae (Own work) [CC BY 3.0], via Wikimedia Commons