The HTTP protocol is quite simple. But many of us under-use it, programmatically speaking. There are many very simple performance mechanisms that are often forgotten. Many developers go for disabling HTTP caching completely, as they often don’t understand how to use it, and because it can cause weird bugs when used incorrectly.

But so much things are cacheable: pages, images, CSS, JavaScript, even many REST web services! Yes, even in this social web era where content changes faster than you can write, there’s still plenty of slow changing information, such as home pages, or lists of countries, regions and cities.

Efficiently using caching translates into:

  • Better response and loading time
  • Decreased load on the server
  • Better user experience

This article aims to present a simple explanation of the HTTP protocol and proper use of HTTP caching.

The HTTP Protocol basics

The HTTP protocol is a communication scheme between two or three actors: the server, the browser, and the often forgotten proxy.

First, to see the implicated headers, let’s have a look to the typical request …

GET / HTTP/1.1
Host: www.tommylacroix.com
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
If-Modified-Since: Thu, 17 Jul 2008 16:11:24 GMT
If-None-Match: "a9432-fc1b28423-cb122da"
User-Agent: Mozilla/5.0 (...) Gecko/2008052906 Firefox/3.0

… and response …

HTTP/1.1 200 OK
Date: Thu, 17 Jul 2008 16:11:24 GMT
Expires: Wed, 11 Jan 1984 05:00:00 GMT
Etag: "a9432-fc1b28423-cb122da"
Last-Modified: Thu, 17 Jul 2008 16:11:24 GMT
Cache-Control: private, must-revalidate, max-age=0
Vary: User-Agent
Content-Type: text/html; charset=UTF-8

<html>...</html>

Basic Caching

The response headers returned by the server give the browser and the proxy information to make caching decisions.

First, the Cache-Control header tells if the content is cacheable or not. If they’re not sent, it’s cacheable by default. Consider the following headers:

Cache-Control: public, max-age=86400

It basically says that the page is cacheable in a public scope, and that the content shouldn’t be kept in the cache without revalidation for more than 1 day (86400 seconds). The scope can be:

  • public: Cacheable by browsers and proxies, even if authenticated
  • Private: Cacheable by browsers only (and proxies, but only for requests from the same clients)
  • no-cache: Cacheable, but revalidation is required every time.
  • no-store: Not cacheable at all.

In addition, there are a few other keywords that you can add to your Cache-Control header. Keep in mind that you must separate multiple keywords by commas.

  • must-revalidate: Some proxies can be configured to ignore the Expires and maxage. This keyword forces them to always act like the resource was expired.
  • proxy-revalidate: Same as must-revalidate, but only for proxies.
  • s-maxage: Same as maxage, but only for proxies.

Finally, for HTTP/1.0 compatibility, you should send a Pragma header when using the Cache-Control no-cache directive.

Pragma: no-cache

Browsers and proxies cache resources based on their URL. You can take advantage of this in many ways at the design stage of your web site or web application. For example, you should put non private REST web service parameters (such as language, or country) in the URL:

http://www.somesite.com/webservice/regionslist/country/usa/language/en

Browsers and proxies also store three important information about the page: the Expires, Last-Modified and Etag headers.

  • The Expires header tells the browser and proxy, along with the max-age component of the Cache-Control header, until when this version of the content should be valid.
  • The Last-Modified header tells the browser and proxy the date and time of the last modification to this page. It isn’t always provided.
  • The Etag header, which stands for Entity Tag, gives the browser and proxy a unique identifier that describes the content it returned. If a page’s content changes, its Etag changes as well.

When you requested this page, your browser (and the proxy in the middle if there was one) did check if the page is cacheable, and since it was, stored a copy of the page associated with the URL, and the three metrics above. For subsequent requests, the server might use this cached copy.

Might? Yes, because the cached page won’t be good until the end of times. At some point, the browser (or the proxy) will check with the server if the page it has in cache is still valid. This is called revalidation, and the Expires HTTP response header along with the max-age component of the Cache-Control header control it.

When the browser or proxy revalidates a page, it sends information about its version: the Etag, and the Last-Modified the server sent when he cached the page. These are sent as If-Modified-Since, and If-None-Match, respectively:

GET /some-cachable-page HTTP/1.1
Host: www.tommylacroix.com
If-Modified-Since: Thu, 17 Jul 2008 16:11:24 GMT
If-None-Match: "a9432-fc1b28423-cb122da"
User-Agent: Mozilla/5.0 (...) Gecko/2008052906 Firefox/3.0

The server then compares this information with the Etag and the Last-Modified of the up-to-date page. If the browser’s cached copy appears to be valid, the server replies with the Expires and Etag (if available), headers, and no content:

HTTP/1.1 304 Not modified
Date: Thu, 17 Jul 2008 16:11:24 GMT
Expires: Wed, 11 Jan 2009 05:00:00 GMT
Etag: "a9432-fc1b28423-cb122da"

If the browser’s cached copy appears to be out-dated, the server replies with the whole page, as usual.

Specific Cases

Now, here comes the tricky cases. For the sake of clarity, I’ll use the following plot for the scenarios below:

« Bob, Alice and Gregg work in the same office. Their office is equiped with a caching web proxy. Bob and Gregg share the same computer with the same user (ok, not credible, so lets say it’s a under-financed non-profit organization), and Alice has her own (she’s the boss).  »

Scenario 1: Secure sessions

« Bob goes on his SuperSocial profile page, at http://www.supersocial.com/profile/. His browser and the office proxy will check if the page is cacheable, and it is. They both store a copy of the page associated with the URL.

But what happens when Gregg or Alice log in his/her SuperSocial’s profile page right after? The browser will give him/her Bob’s page! »

The Cache-Control, Expires, and the Etag headers. Setting the Expires header in the past and the max-age to zero will cause the browser to revalidate the content each time. If the cached content is valid, only a header with no content will be sent. This is slightly slower than full blown caching, but no as much as no caching at all.

We could also set the Cache-Control scope to private, as each copy of the page will obviously only be valid one user.

We should also use Etag, as the content returned for the same URL isn’t the same for Bob and for Gregg, so the Etag will change, and the browser will reload the page for Gregg.

Scenario 2: Content optimization

« The three unproductive workers browse a news site that has a wicked design. So wicked that the guys behind it had to make browser specific optimizations. Therefore, when you’re with Internet Explorer, the page is IE optimized, when you’re with Safari, some in-line CSS styles are different, and when you’re with Firefox, you get the regular page because this beauty is standard-compliant. Standard-what? Lets not digress…

When Bob and Gregg browse the site with IE from the same computer, it’s fine. But when Alice accesses it a short while after with Safari, she gets an ugly page as it’s IE optimized. The reason is, the server sent the IE version to Bob, and the proxy cached it. When Alice requests it, the proxy sends the IE page, thinking that it’s all fine. »

The Vary header. When the content of a page changes based on miscellaneous headers contained in the request, the server must tell the browser and the proxy using the Vary header. The Vary header basically says: If you cache this content, beware that if this header changes, the content might change as well, so use the URL with these fields to make sure you serve a valid version later.

Here’s an example. The request…

GET / HTTP/1.1
Host: www.tommylacroix.com
User-Agent: Mozilla/5.0 (...) Gecko/2008052906 Firefox/3.0

… and response …

HTTP/1.1 200 OK
Date: Thu, 17 Jul 2008 16:11:24 GMT
Expires: Wed, 11 Jan 2009 05:00:00 GMT
Last-Modified: Thu, 17 Jul 2008 16:11:24 GMT
Cache-Control: public, max-age=86400
Vary: User-Agent
Content-Type: text/html; charset=UTF-8

Code Snippet

Here’s a code snippet I wrote quite some time ago. It’s not pretty, but it works so I think it’s a good basic practical example.

/**
* setCacheHTTPHeaders
*
* @author	Tommy Lacroix
* @param	string	$privacy		Scope: public, private or no-cache
* @param	int	$lastModified		Unix timestamp of last page modification (optional)
* @param	int	$maxage			Maximum caching time before revalidation (optional)
* @param	string	$etag			Entity tag, page-content specific (optional)
* @return	bool				TRUE if content need to be sent, FALSE if no content need to be sent
*/
function setCachePolicy($privacy = 'public', $lastModified = false, $maxage = false, $etag = false) {
	// Sanitize privacy
	switch ($privacy) {
		case 'privacy':
		case 'public':
		case 'no-cache':
			break;
		default:
			$privacy = "public";
			break;
	}

	// Calculate expiry and max-age
	if (is_string($maxage)) { // Expiry is a string, interpret
		unset($m);
		if (preg_match('/^([0-9]+)([smhdwy])$/', $maxage, $m)) {
			$maxage = $m[1];
			switch ($m[2]) {
				case 's':	break;
				case 'm':	$maxage *= 60; 		break;
				case 'h':	$maxage *= 3600; 	break;
				case 'd':	$maxage *= 86400; 	break;
				case 'w':	$maxage *= 604800; 	break;
				case 'y':	$maxage *= 31536000; 	break;
			}
		}
	}
	if ($privacy != 'no-cache') {
		header('Expires: '.gmdate("r", time()+$maxage));
	} else {
		header('Expires: '.gmdate("r", time()-31536000));
		$maxage = 0;
	}

	// Send ETag headers
	if ($etag !== false) {
		header('ETag: "'.$etag.'"');
	}

	// Determine wheter we need to send content or not
	$outputContent = true;

	// Check ETag
	if ((isset($_SERVER['HTTP_IF_NONE_MATCH'])) && ($etag !== false)) {
		if ($_SERVER['HTTP_IF_NONE_MATCH'] == '"'.$etag.'"') {
			header($_SERVER['SERVER_PROTOCOL'].' 304 Not Modified');
			$outputContent = false;
		}
	}

	if ((isset($_SERVER['HTTP_IF_MODIFIED_SINCE'])) && ($lastModified !== false)) {
		$ifModifiedSince = strtotime($_SERVER['HTTP_IF_MODIFIED_SINCE']);
		if ($ifModifiedSince >= $lastModified) {
			header($_SERVER['SERVER_PROTOCOL'].' 304 Not Modified');
			$outputContent = false;
		}
	}

	// Remove content if output has been disabled
	if (!$outputContent) {
		// Return false: You don't need to send content
		return false;
	} else {
		// Send other headers
		header('Cache-Control: '.$privacy.', must-revalidate, post-check=0, pre-check=0, max-age='.$maxage);
		if ($privacy == 'no-cache') header('Pragma: no-cache');
		if ($lastModified !== false) header('Last-Modified: '.gmdate('r',$lastModified));

		// Return true: you need to send content
		return true;
	}
}

Conclusion

The caching HTTP headers are simple to implement, and provide a huge performance bonus. If done properly, this convert into a better user experience, as there’s less waiting, and more browsing.

Shall you like me to add other scenarios, feel free to drop me a line and I’ll see what I can do.

Further Reading


If you like this article, leaving a comment, tweeting ofr liking it is always appreciated.

category PHP, Usability Thursday 17 July 2008 Comment (0)

Leave a Reply