Tommy Lacroix

HTTP caching basics

tlacroix — Thu, 17 Jul 2008 18:56:56 +0000

The HTTP protocol is quite simple. But many of us under-use it, programmatically speaking. There are many very simple performance mechanisms that are often forgotten. Many developers go for disabling HTTP caching completely, as they often don’t understand how to use it, and because it can cause weird bugs when used incorrectly.

But so much things are cacheable: pages, images, CSS, JavaScript, even many REST web services! Yes, even in this social web era where content changes faster than you can write, there’s still plenty of slow changing information, such as home pages, or lists of countries, regions and cities.

Efficiently using caching translates into:

Better response and loading time
Decreased load on the server
Better user experience

This article aims to present a simple explanation of the HTTP protocol and proper use of HTTP caching.

The HTTP Protocol basics

The HTTP protocol is a communication scheme between two or three actors: the server, the browser, and the often forgotten proxy.

First, to see the implicated headers, let’s have a look to the typical request …

GET / HTTP/1.1
Host: www.tommylacroix.com
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
If-Modified-Since: Thu, 17 Jul 2008 16:11:24 GMT
If-None-Match: "a9432-fc1b28423-cb122da"
User-Agent: Mozilla/5.0 (...) Gecko/2008052906 Firefox/3.0

… and response …

HTTP/1.1 200 OK
Date: Thu, 17 Jul 2008 16:11:24 GMT
Expires: Wed, 11 Jan 1984 05:00:00 GMT
Etag: "a9432-fc1b28423-cb122da"
Last-Modified: Thu, 17 Jul 2008 16:11:24 GMT
Cache-Control: private, must-revalidate, max-age=0
Vary: User-Agent
Content-Type: text/html; charset=UTF-8

...

Basic Caching

The response headers returned by the server give the browser and the proxy information to make caching decisions.

First, the Cache-Control header tells if the content is cacheable or not. If they’re not sent, it’s cacheable by default. Consider the following headers:

Cache-Control: public, max-age=86400

It basically says that the page is cacheable in a public scope, and that the content shouldn’t be kept in the cache without revalidation for more than 1 day (86400 seconds). The scope can be:

public: Cacheable by browsers and proxies, even if authenticated
Private: Cacheable by browsers only (and proxies, but only for requests from the same clients)
no-cache: Cacheable, but revalidation is required every time.
no-store: Not cacheable at all.

In addition, there are a few other keywords that you can add to your Cache-Control header. Keep in mind that you must separate multiple keywords by commas.

must-revalidate: Some proxies can be configured to ignore the Expires and maxage. This keyword forces them to always act like the resource was expired.
proxy-revalidate: Same as must-revalidate, but only for proxies.
s-maxage: Same as maxage, but only for proxies.

Finally, for HTTP/1.0 compatibility, you should send a Pragma header when using the Cache-Control no-cache directive.

Pragma: no-cache

Browsers and proxies cache resources based on their URL. You can take advantage of this in many ways at the design stage of your web site or web application. For example, you should put non private REST web service parameters (such as language, or country) in the URL:

http://www.somesite.com/webservice/regionslist/country/usa/language/en

Browsers and proxies also store three important information about the page: the Expires, Last-Modified and Etag headers.

The Expires header tells the browser and proxy, along with the max-age component of the Cache-Control header, until when this version of the content should be valid.
The Last-Modified header tells the browser and proxy the date and time of the last modification to this page. It isn’t always provided.
The Etag header, which stands for Entity Tag, gives the browser and proxy a unique identifier that describes the content it returned. If a page’s content changes, its Etag changes as well.

When you requested this page, your browser (and the proxy in the middle if there was one) did check if the page is cacheable, and since it was, stored a copy of the page associated with the URL, and the three metrics above. For subsequent requests, the server might use this cached copy.

Might? Yes, because the cached page won’t be good until the end of times. At some point, the browser (or the proxy) will check with the server if the page it has in cache is still valid. This is called revalidation, and the Expires HTTP response header along with the max-age component of the Cache-Control header control it.

When the browser or proxy revalidates a page, it sends information about its version: the Etag, and the Last-Modified the server sent when he cached the page. These are sent as If-Modified-Since, and If-None-Match, respectively:

GET /some-cachable-page HTTP/1.1
Host: www.tommylacroix.com
If-Modified-Since: Thu, 17 Jul 2008 16:11:24 GMT
If-None-Match: "a9432-fc1b28423-cb122da"
User-Agent: Mozilla/5.0 (...) Gecko/2008052906 Firefox/3.0

The server then compares this information with the Etag and the Last-Modified of the up-to-date page. If the browser’s cached copy appears to be valid, the server replies with the Expires and Etag (if available), headers, and no content:

HTTP/1.1 304 Not modified
Date: Thu, 17 Jul 2008 16:11:24 GMT
Expires: Wed, 11 Jan 2009 05:00:00 GMT
Etag: "a9432-fc1b28423-cb122da"

If the browser’s cached copy appears to be out-dated, the server replies with the whole page, as usual.

Specific Cases

Now, here comes the tricky cases. For the sake of clarity, I’ll use the following plot for the scenarios below:

« Bob, Alice and Gregg work in the same office. Their office is equiped with a caching web proxy. Bob and Gregg share the same computer with the same user (ok, not credible, so lets say it’s a under-financed non-profit organization), and Alice has her own (she’s the boss). »

Scenario 1: Secure sessions

« Bob goes on his SuperSocial profile page, at http://www.supersocial.com/profile/. His browser and the office proxy will check if the page is cacheable, and it is. They both store a copy of the page associated with the URL.

But what happens when Gregg or Alice log in his/her SuperSocial’s profile page right after? The browser will give him/her Bob’s page! »

The Cache-Control, Expires, and the Etag headers. Setting the Expires header in the past and the max-age to zero will cause the browser to revalidate the content each time. If the cached content is valid, only a header with no content will be sent. This is slightly slower than full blown caching, but no as much as no caching at all.

We could also set the Cache-Control scope to private, as each copy of the page will obviously only be valid one user.

We should also use Etag, as the content returned for the same URL isn’t the same for Bob and for Gregg, so the Etag will change, and the browser will reload the page for Gregg.

Scenario 2: Content optimization

« The three unproductive workers browse a news site that has a wicked design. So wicked that the guys behind it had to make browser specific optimizations. Therefore, when you’re with Internet Explorer, the page is IE optimized, when you’re with Safari, some in-line CSS styles are different, and when you’re with Firefox, you get the regular page because this beauty is standard-compliant. Standard-what? Lets not digress…

When Bob and Gregg browse the site with IE from the same computer, it’s fine. But when Alice accesses it a short while after with Safari, she gets an ugly page as it’s IE optimized. The reason is, the server sent the IE version to Bob, and the proxy cached it. When Alice requests it, the proxy sends the IE page, thinking that it’s all fine. »

The Vary header. When the content of a page changes based on miscellaneous headers contained in the request, the server must tell the browser and the proxy using the Vary header. The Vary header basically says: If you cache this content, beware that if this header changes, the content might change as well, so use the URL with these fields to make sure you serve a valid version later.

Here’s an example. The request…

GET / HTTP/1.1
Host: www.tommylacroix.com
User-Agent: Mozilla/5.0 (...) Gecko/2008052906 Firefox/3.0

… and response …

HTTP/1.1 200 OK
Date: Thu, 17 Jul 2008 16:11:24 GMT
Expires: Wed, 11 Jan 2009 05:00:00 GMT
Last-Modified: Thu, 17 Jul 2008 16:11:24 GMT
Cache-Control: public, max-age=86400
Vary: User-Agent
Content-Type: text/html; charset=UTF-8

Code Snippet

Here’s a code snippet I wrote quite some time ago. It’s not pretty, but it works so I think it’s a good basic practical example.

/**
* setCacheHTTPHeaders
*
* @author	Tommy Lacroix
* @param	string	$privacy		Scope: public, private or no-cache
* @param	int	$lastModified		Unix timestamp of last page modification (optional)
* @param	int	$maxage			Maximum caching time before revalidation (optional)
* @param	string	$etag			Entity tag, page-content specific (optional)
* @return	bool				TRUE if content need to be sent, FALSE if no content need to be sent
*/
function setCachePolicy($privacy = 'public', $lastModified = false, $maxage = false, $etag = false) {
	// Sanitize privacy
	switch ($privacy) {
		case 'privacy':
		case 'public':
		case 'no-cache':
			break;
		default:
			$privacy = "public";
			break;
	}

	// Calculate expiry and max-age
	if (is_string($maxage)) { // Expiry is a string, interpret
		unset($m);
		if (preg_match('/^([0-9]+)([smhdwy])$/', $maxage, $m)) {
			$maxage = $m[1];
			switch ($m[2]) {
				case 's':	break;
				case 'm':	$maxage *= 60; 		break;
				case 'h':	$maxage *= 3600; 	break;
				case 'd':	$maxage *= 86400; 	break;
				case 'w':	$maxage *= 604800; 	break;
				case 'y':	$maxage *= 31536000; 	break;
			}
		}
	}
	if ($privacy != 'no-cache') {
		header('Expires: '.gmdate("r", time()+$maxage));
	} else {
		header('Expires: '.gmdate("r", time()-31536000));
		$maxage = 0;
	}

	// Send ETag headers
	if ($etag !== false) {
		header('ETag: "'.$etag.'"');
	}

	// Determine wheter we need to send content or not
	$outputContent = true;

	// Check ETag
	if ((isset($_SERVER['HTTP_IF_NONE_MATCH'])) && ($etag !== false)) {
		if ($_SERVER['HTTP_IF_NONE_MATCH'] == '"'.$etag.'"') {
			header($_SERVER['SERVER_PROTOCOL'].' 304 Not Modified');
			$outputContent = false;
		}
	}

	if ((isset($_SERVER['HTTP_IF_MODIFIED_SINCE'])) && ($lastModified !== false)) {
		$ifModifiedSince = strtotime($_SERVER['HTTP_IF_MODIFIED_SINCE']);
		if ($ifModifiedSince >= $lastModified) {
			header($_SERVER['SERVER_PROTOCOL'].' 304 Not Modified');
			$outputContent = false;
		}
	}

	// Remove content if output has been disabled
	if (!$outputContent) {
		// Return false: You don't need to send content
		return false;
	} else {
		// Send other headers
		header('Cache-Control: '.$privacy.', must-revalidate, post-check=0, pre-check=0, max-age='.$maxage);
		if ($privacy == 'no-cache') header('Pragma: no-cache');
		if ($lastModified !== false) header('Last-Modified: '.gmdate('r',$lastModified));

		// Return true: you need to send content
		return true;
	}
}

Conclusion

The caching HTTP headers are simple to implement, and provide a huge performance bonus. If done properly, this convert into a better user experience, as there’s less waiting, and more browsing.

Shall you like me to add other scenarios, feel free to drop me a line and I’ll see what I can do.

Super-charged Amazon SQS Sample in PHP

tlacroix — Fri, 11 Jul 2008 20:00:58 +0000

If you played with the pretty cool Amazon Web Services, you probably started off fetching a sample off the Developer’s Connection pages. At least, I did.

One cool sample for Simple Queue Service is the one by Justin@AWS. But (you saw it coming, I suppose), being anti-PEAR (for many reason, such as dependencies, weight, version conflicts, non-catchiness of the name, fructose intolerance, etc.), I just couldn’t leave the sample like that. Yes, this is me again reinventing the wheel.

The original library depended on two PEAR components:

Crypt_HMAC, for authentication
HTTP_Request, for HTTP calls

The Crypt_HMAC replacement

I replaced the Crypt_HMAC dependency by three potential routines: hash PECL, mhash, and Lance’s function (which you will never hear in a Numb3r’s episode, as it has nothing to do with a mathematician, but rather with a smart dude who posted a fully PHP HMAC code snippet in the PHP comments of the mhash() function).

The code goes as follow:

/**
 * HMAC function using hash PECL (http://ca.php.net/manual/en/book.hash.php)
 *
 * @author Tommy Lacroix
 * @param string $stringToSign
 * @return string	base64 encoded hmac
 * @internal
 */
private function hmac_hash($stringToSign) {
	return base64_encode(pack('H*',hash_hmac('sha1', $stringToSign, $this->secretKey)));
}	

/**
 * HMAC function using mhash (http://ca.php.net/manual/en/book.hash.php)
 *
 * @author Tommy Lacroix
 * @param string $stringToSign
 * @return string	base64 encoded hmac
 * @internal
 */
private function hmac_mhash($stringToSign) {
	return base64_encode(mhash(MHASH_SHA1, $stringToSign, $this->secretKey));
}

/**
 * HMAC function using Lance's function (http://ca.php.net/manual/en/function.mhash.php, see comments)
 *
 * @author Tommy Lacroix
 * @param string $stringToSign
 * @return string	base64 encoded hmac
 * @internal
 */
private function hmac_lance($stringToSign) {
    $b = 64;
	if (strlen($this->secretKey) > $b) {
		$key = pack("H*",sha1($this->secretKey));
	} else {
		$key = $this->secretKey;
	}
	$key  = str_pad($key, $b, chr(0x00));
	$ipad = str_pad('', $b, chr(0x36));
	$opad = str_pad('', $b, chr(0x5c));
	$k_ipad = $key ^ $ipad ;
	$k_opad = $key ^ $opad;

	return base64_encode(pack('H*', sha1($k_opad  . pack("H*",sha1($k_ipad . $stringToSign)))));
}

HTTP_Request replacement

I replaced the HTTP_Request dependency by two potential routines: cURL and URL wrappers.

The code goes as follow:

/**
 * Fetch with curl
 *
 * @author Tommy Lacroix
 * @param string 	$url
 * @param string 	$qs
 * @param bool		$post
 * @return array(output,httpCode)
 * @internal
 */
private function fetch_curl($url, $qs, $post) {
	$curl = curl_init();
	curl_setopt($curl, CURLOPT_RETURNTRANSFER, TRUE);
	curl_setopt($curl, CURLOPT_SSL_VERIFYPEER, FALSE);
	curl_setopt($curl, CURLOPT_HEADER, false);
	curl_setopt($curl, CURLOPT_POST, $post);
	curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);
	curl_setopt($curl, CURLOPT_FOLLOWLOCATION, false);
	curl_setopt($curl, CURLOPT_URL, $url . (!$post ? '?'.$qs : ''));
	if ($post) {
		curl_setopt($curl, CURLOPT_POSTFIELDS, $qs);
	}

	// Execute
	$output = array();
	$output[0] = curl_exec($curl);
	$output[1] = curl_getinfo($curl,CURLINFO_HTTP_CODE);

	// Close handle
	curl_close($curl);

	return $output;
}

/**
 * Fetch with urlwrappers
 *
 * @author Tommy Lacroix
 * @param string 	$url
 * @param string 	$qs
 * @param bool		$post
 * @return array(output,httpCode)
 * @internal
 */
private function fetch_urlwrappers($url, $qs, $post) {
	$output = array();

	if ($post) {
		$opts = array(
		  'http'=>array(
		    'method'=>"POST",
		    'header'=>"Content-type: application/x-www-form-urlencoded\r\n" .
		              "Content-length: " . strlen($qs),
		    'content'=>$qs
		  )
		);
		$context = stream_context_create($opts);
	} else {
		$opts = array(
		  'http'=>array(
		    'method'=>"GET"
		  )
		);
		$context = stream_context_create($opts);
		$url .= '?'.$qs;
	}

	$output[0] = '';
	$f = @fopen($url,'r',null,$context);
	if (!$f) {
		$output[0] = '';
		$output[1] = 404;
		return $output;
	}
	while (!feof($f)) {
		$output[0] .= fread($f,1024);
	}
	$meta_data = stream_get_meta_data($f);
	if (preg_match('/^HTTP\/1\.[01] ([0-9]{3})/',$meta_data['wrapper_data'][0],$m)) {
		$output[1] = $m[1];
	} else {
		$output[1] = false;
	}
	fclose($f);

	$output[2] = 'wrappers';

	return $output;
}

Detection and conclusion

Finally, the constructor includes a small detection mechanism that selects an HMAC and a fetching function:

/**
 * Constructor
 *
 * @author Justin@AWS
 * @author Tommy Lacroix
 * @param string $accessKey
 * @param string $secretKey
 * @param string $endpoint	http://queue.amazonaws.com
 * @param string $queueName	optional
 * @return SQSClient
 */
public function SQSClient($accessKey, $secretKey, $endpoint = 'http://queue.amazonaws.com', $queueName = '')
{
	.
	.
	.

	// Select fetch function
	// Call it with $this->{$this->fetchFunction}(...)
	if (function_exists('curl_init')) $this->fetchFunction = 'fetch_curl';
		else $this->fetchFunction = 'fetch_urlwrappers';

	// Select hash function
	// Call it with $this->{$this->hmacFunction}(...)
	if (function_exists('hash_hmac')) {
		$this->hmacFunction = 'hmac_hash';
	} else if (function_exists('mhash')) {
		$this->hmacFunction = 'hmac_mhash';
	} else if (class_exists('Crypt_HMAC')) {
		$this->hmacFunction = 'hmac_pear';
	} else {
		$this->hmacFunction = 'hmac_lance';
	}

	.
	.
	.
}

The complete modified sqsClient package can be downloaded below. The original Justin@AWS package is available here.

sqsClientPHP-2.1.zip (11k)

PHP based FLV meta and cue points reader/writer

tlacroix — Fri, 04 Jul 2008 21:30:26 +0000

Update (Aug 15, 2008): The AMF0Parser library has been updated to support the Date type in the metas, added among others by the “inlet media FLVTool2″ tool.

If you’ve done Flash banners or micro-sites that embed video with cue points for synchronization before, you know that it’s a pain in the arse. There doesn’t seem to be any tool around to modify the damn points once the file is encoded, so you’ve got to re-encode the file over and over to have your things synced correctly.

If you’ve tried to reverse engineer the FLV format before, you know that it’s a pain in the arse as well. The AMF0 format is anything but intuitive, and the documentation has been lacking for a long long time — although there’s apparently an SDK now. Luckily, there was OSFlash, and SabreAMF and AMFPHP that could be used as a documentation source.

If you recognize yourself, this might be your lucky day. After many hours of trial and error, I’ve finally been able to reverse engineer it, and to build a library that extract essential FLV information, reads the Meta, and allows you to write them back — including the cue points.

First, you’ll need to FLVInfo2 and the AMF0Parser libraries, below. The FLVInfo2 library has three main functions:

getInfo, which returns information gotten from analyzing the FLV file and it’s meta. Frame rate, bit rate, audio/video codec, width, height, etc.

(
    [signature] => 1
    [hasVideo] => 1
    [hasAudio] => 1
    [minimalFlashVersion] => 8
    [video] => stdClass Object
        (
            [codec] => 4
            [width] => 320
            [height] => 240
            [keyframeRatio] => 0.0180537208278
            [keyframeEvery] => 55.3902439024
            [fps] => 15
            [bitrate] => 448
            [codecStr] => On2 VP6
        )

    [audio] => stdClass Object
        (
            [codec] => 2
            [frequency] => 22
            [depth] => 16
            [channels] => 2
            [bitrate] => 48
            [codecStr] => MP3
        )
)

getMeta, which returns the meta data, and the cue points

Array
(
    [metas] => Array
        (
            [0] => Array
                (
                    [0] => onMetaData
                    [1] => Array
                        (
                            [duration] => 151.46
                            [width] => 320
                            [height] => 240
                            [videodatarate] => 500
                            [canSeekToEnd] => 1
                            [videocodecid] => 4
                            [audiodatarate] => 48
                            [audiocodecid] => 2
                            [framerate] => 15
                            [creationdate] => Tue Jun 17 13:06:15 2008
                            [Encoded_By] => orangetango Video Encoder
                            [Encoded_With] => orangetango Video Encoder
                            [metadatacreator] => orangetango FLV meta data writer
                        )
                )
        )
    [cuepoints] => Array
        (
            [0] => Array
                (
                    [0] => onCuePoint
                    [1] => Array
                        (
                            [name] => name1
                            [time] => 4.41
                            [type] => event
                        )
                )
            [1] => Array
                (
                    [0] => onCuePoint
                    [1] => Array
                        (
                            [name] => name1_end
                            [time] => 6.7
                            [type] => event
                        )
                )
        )
)

rewriteMeta, which inputs an FLV file, meta data and cue points, and outputs a new FLV file.

As a practical example, you find the rewriteTagsForFile function in the example.php file, that rewrites the meta data and gets the cue points from a FLVCoreCuePoints XML file.

flvinfo2.php (175k)

AMF0Parser.php (96k, updated Aug 15, 2008)

example.php (12k)

Flash Video logo found on Wikipedia.

CodeIgniter 1.6.x OpenID module

tlacroix — Wed, 25 Jun 2008 12:42:22 +0000

I got no IPod. I got no IPhone. But I’ve got an OpenID. I’m like a low class geek. But whatever. I’ve read a bit lately about OpenID and wanted to give it a try on one of my personal projects (CodeIgniter based, you bet).

I’ve stumbled upon Rémi Prévost — who’s blog a follow more or less consistently due to the lack of time — CIOpenID module. It’s basically a CI embed of Janrain’s PHP_OpenID.

His code is great and works well. But one thing I don’t like about it is that it requires a different bootstrap file (a modified version of index.php) and a somewhat hacked version of your typical .htaccess file. The reason is, CodeIgniter annihilates the $_GET variable during initialization, because GET queries aren’t secure (ok, this is overly simplified, but you get the idea).

Being who I am, and somewhat liking to reinvent the wheel during my free time, I used a different approach. Rather than using a different bootstrap file, I rather chose to use the pre_system hook.

The installation process is quite simple:

Put my ciopenid.php library in your /system/application/libraries directory.
Create /system/application/libraries/openid, and unzip PHP_OpenID 2.x.x’s Auth directory in it.
Open your /system/application/config/config.php file, and make sure that hooks are enabled:
```
$config['enable_hooks'] = TRUE;
```

Open your /system/application/config/hooks.php file, and add the following lines:

$hook['pre_system'][] = array(
   'class'    => 'ciopenid',
   'function' => 'pre_system_hook',
   'filename' => 'ciopenid.php',
   'filepath' => 'libraries',
   'params'   => null
   );

To make an authentication query to an OpenID URL in $openid:

$this->load->library('ciopenid');
try {
   $redirect_url = $this->ciopenid->authenticate($openid, site_url($this->uri->uri_string()));
   header("Location: ".$redirect_url);
} catch (Exception $e) {
   die('Error: '.$e->getMessage());
}

And to get the OpenID query response:

$this->load->library('ciopenid');
if (isset($this->ciopenid->response)) {
   switch ($this->ciopenid->response->status) {
      case Auth_OpenID_SUCCESS:
         $openid = $this->ciopenid->response->endpoint->claimed_id;
         print 'Success: '.$openid;
         break;
      case Auth_OpenID_CANCEL:
         print 'User cancelled';
         break;
      default:
         print 'Internal error';
         break;
   }
}

That’s it! Ooops, here’s the file…

ciopenid.zip (3k)

Enjoy!

Usability review of the week (April 4th, 08)

tlacroix — Sat, 05 Apr 2008 01:53:41 +0000

Getting back to good habits, here’s another issue of the usability review of the week. I should actually be called usability review of the day, as all the good stuff I came across was published on April 3rd.

Big impact, small changes on Amazon (04/03/2008)
Effective text only emails (04/03/2008)
A Refreshing Take on Usability (04/03/2008)

Shall you stumble on great usability pages, feel free to leave me a comment and I’ll give it a look.

Big impact, small changes on Amazon (April 3rd, 2008)

You probably didn’t notice unless you’re a savvy Amazon shopper (whether you shop for books or for usability practices) , but Amazon recently made subtle but great changes to the design. They improved readability of key information elements, and redesigned the up-selling space. Check the link below for a great before/after comparison (that I didn’t feel comfortable to rip even if I was going to credit the author).

Via: Future Now’s blog

Effective text only emails (April 3rd, 2008)

Text emails don’t have to be bad and boring. Okay, they’ve got to be boring. But still, when HTML emails aren’t an option, or even just to provide a text alternative for all these new non-HTML cell phone email readers out there, there are ways to make text emails look better:

Put links on new lines
Uses lists with asterisks as bullets
Try to keep it short
If you can’t keep it short, divide in sections
Call to action

Via: Shape Shed

A Refreshing Take on Usability (April 3rd, 2008)

This is more a funny heads up than a news, but it shows how well one can use available information to improve user experience.

Via: B#

Yet another Facebook conspiracy theory

tlacroix — Sun, 30 Mar 2008 15:56:13 +0000

Long time no write, I have been really busy in the last two weeks. Big news coming professionally. I’ll keep you posted on this.

Schedule aside, there’s an idea that has been on my mind for quite some time. Actually, since I saw the What happens in the Facebook stays in the Facebook Flash presentation, and after my advertising teacher told us something like : “Advertising on the web is great, because you can track the viewers behavior, including pre and post viewing”.

AdServer conspiracy

Well, Facebook knows a lot about you, and in advertising, the more you know about your viewer, the more relevant the ads you present him can be. Now, Facebook has an internal advertising platform, Facebook Social Ads, based on this. Nothing external, such as AdSense, yet.

But lets imagine that Facebook launches an external AdServer and builds a big network of small advertisers, like Google AdSense, or a small network of large advertisers.

First, as Facebook knows a lot about you, it simply has to set a cookie in your browser when you go on your profile, and reuse the cookie when displaying a banner on newyorktimes.com. The AdServer would then know that I’m a 27 years old male in a relationship with a university degree.

The force

This could lead to very interesting possibilities:

Segmented Ad Performance Calculation : Does you ad under-performs except with single gay females with a college degree? You can’t know now, but a Facebook AdServer could tell you.
Segmented Ad Presentation: Only present the ad to your target segments. So you could have a Monster ad presented only to unemployed viewers and viewers who didn’t update their work profile for the past 2 years.
Segmented Ad Selection : Present an ad more specific to the viewer’s segment. Creating various versions of the same banner is cheap, so if you can present a version more specific to a segment. So you could have an Obama ad targeted for:
- Afro-Americans
- Hispanic-Americans
- The bunch

The ethics

Sure, this raises many ethical questions, as it could be misused by the wrong people (you know, the bad guys in 24 and in Michael Moore’s films).

I remember (or I think I do remember — please let me know if I’m misquoting) that, while I was working at ZeroKnowledge Systems way back, Adam Shostack’s moto was: “Someday, soon enough, all databases will be interconnected”.

That’s why ZeroKnowledge developed their Freedom platform, allowing you to browse anonymously. But nobody cared at the time: querying a 500MB database took a few seconds on cluster, so no one could realistically store so much information in a usable way. That’s why ZK’s great idea and platform never really worked, business-wise. Too much, too soon.

But in the 2008 world were Google and Yahoo crawl — and even cache — a huge portion of the web, where 25% of adults in Quebec have a Facebook profile, and where interconnecting platforms is easier than ever with all the APIs around, this could be up and running tomorrow morning. Too little, too late?

Big brother is watching us.

Saturday morning: Bunker game

tlacroix — Sat, 15 Mar 2008 14:10:25 +0000

In this saturday morning, I’m pasting my copy, sipping my coffee and sniffing around other peoples blog looking for gems.

Well, no gems found this morning, only a little single-handed pleasure Flash game called Bunker. Free advice: don’t buy the shitty 400$ shot gun, it really sucks.

Usability review of the week (March 14th, 08)

tlacroix — Fri, 14 Mar 2008 19:06:06 +0000

Another crazy week, and not so much good usability articles appeared on my radar. Yet, there’s a few, and here they are.

Is Customer Experience Recession-Proof? (03/14/2008)
eCommerce Usability Review: Advanced Search Pages (03/13/2008)
11 Ways to Fill Your Shopper’s Cart (03/12/2008)
Google to start to implement site performance metrics in SEM Quality Score (03/07/2008)

Is Customer Experience Recession-Proof?

Apparently, when consumer spending drops, the fight over the remaining dollars intensifies. A new report from Forrester goes in that direction. According to the summary, “more than 80% of respondents said that improving the usability, usefulness and enjoyability of the online experience is more important this year.” Respondents forecast increased spending in Web Analytics (68%), Usability (53%) and Behavioral Research (51%).

Via: Research Central @ CIO Insight

eCommerce Usability Review: Advanced Search Pages

Yesterday, Varien published an advanced search page attributes review, that is somewhat a sequel to usability review on search page attributes. Same concept as before, the folks take a couple of very good, good and bad pages to display what advanced search engine pages out there allow the user to do. I’m particularilly a fan of the “refine search” concept,
as it is in harmony with the progressive disclosure principle.

Via: Varien’s eCommerce Cache Blog

11 Ways to Fill Your Shopper’s Cart

I liked this one as it helped complete my eCommerce site functionality checklist. Most of Stoney deGeyter’s tips are obvious if you already designed a few eCommerce sites (search engine, product comparison guides, product reviews, etc.), but the list is brief and well assembled, which makes it a good reference tool to save you some thinking time.

Via: Search Engine Guide

Google to start to implement site performance metrics in SEM Quality Score

Earlier this week, Google released a post indicating that they will start to consider the landing page load time in the calculation of the Quality Score. The guys at Traffick see it as the beginning of a trend, and I tend to agree after reading Yahoo’s usability patent filling last week.

Via: Traffick

JavaScript database with CRUD interface: TaffyDB

tlacroix — Wed, 12 Mar 2008 14:07:20 +0000

As the post title says, TaffyDB is a JavaScript database with a CRUD — Create, Read, Update, Delete — interface. Object oriented, under 10K and compatible with most AJAX frameworks around, TaffyDB is a cool tool to bring your AJAX applications to the next level.

The object usage is quite intuitive… see for yourself:

products.find({price:{lessthan:10},
              type:{not:"Book"}});

products.update({status:"NA"},
                {manufacturer:"XZYDesign"});

products.orderBy(
   ["type",{"price":"asce"},{"quantity":"asce"}]);

I’m definitely going to find a use for this in a project soon. Meanwhile, I’m still messing with it to seize its full power. Enjoy…!

Usability review of the week (March 7th, 08)

tlacroix — Fri, 07 Mar 2008 13:06:19 +0000

Here’s what I stumbled upon this week, on the web, about usability. The reviews are quite brief as this has been a crazy week for me.

Jakob Nielsen’s reports usability ROI decline(03/04/2008)
Yahoo Automates Usability Consulting (03/03/2008)
Study: Introductory Paragraphs and Tabs Don’t Aid Reading Comprehension Online (05/03/2008)
Measuring satisfaction: Beyond the usability questionnaire (03/03/2008)

Jakob Nielsen’s reports usability ROI decline

Jakob Nielsen reports that average the usability ROI of web sites declined from 135% (2002) to 83% (2008). Nielson gives a few explanations to the decline. Two of them are :

We have now harvested most of the low-hanging fruit from the truly horrible websites that dominated the lost decade of Web usability (approximately 1993–2003). In those early years, Web design was abominable — think splash screens, search that didn’t find anything, bloated graphics everywhere. The only good thing about these early designs was that they were so bad that it was easy for usability people to be heroes: even the smallest study would inevitably reveal several immense opportunities for improvement.

Via: Little Sprint Design

Yahoo Automates Usability Consulting

A recent Yahoo patent application lists usability factors a search engine could use to determine how usable a web page is. Why would Yahoo be interested in knowing? Quoting the authors of the document :

It can be important to make web pages easy and pleasing to use, which can be particularly important for web pages it is desired to monetize.

This may include, for example, advertisement-containing web pages (of a so-called “web portal,” for example), for which an advertiser pays money when a user views the web page and activates a link of the advertisement.

If such web pages are not easy and pleasing to use, the money-making potential of those web pages can be jeopardized. One conventional indication of whether a web page is easy and pleasing to use is called “clutter.”

I personally crave for a usability test engine to replace my old boring evaluation checklist. Next step would be a Google Analytics module that would analyze the browsing patterns.

Via: SEO by the SEA

Study: Introductory Paragraphs and Tabs Don’t Aid Reading Comprehension Online

A recent study by the University of Washington analyzes the impact of introductory paragraphs (presence of the paragraph, and presence of links in the paragraph), and tabbed navigation.

Cutting to the chase, users apparently like tabs, but introductory text might have no effect on comprehension. Anchored text is helps increasing comprehension — especially on how the concepts relate to each other — but decreases user satisfaction, especially when there’s no tabbed navigation present. I strongly encourage you to spend 5 minutes to read Faithful Web summary.
Via: Faithful Web

Measuring satisfaction: Beyond the usability questionnaire

David Travis from UserFocus describes the criticism-reluctancy problem in rational usability testing, and provides a method to overcome this problem.

This is very similar to a few focus group interview techniques used in marketing research, but applied to usability testing.

Via: UserFocus