I used queues many times in the past. First in first out (FIFO) queues for for mass email programs, last in first out (LIFO) queues for MicroBlogBuzz, and prioritized queues for a video compression platform, among other things.

After a lot of trial and error, storage engine changes, code optimization, benchmarking and swearing, I came up with this idea of cascaded queueing. In a nutshell, cascaded queueing is putting two queues back to back: a master queue that will act as a warehouse, and one or many slave queues that will act as distributors or brokers.

Available Queueing Methods

The queue messages can be stored in different ways, and each have their pros and cons. The goal of this post not being to describe queueing, but rather cascaded queueing, here’s a quick analysis of the most common ones.

• File system: Quick, persistent and flexible, but it’s locally hosted, and it requires a file system that supports file locking.
• Database (ie. MySQL): Persistent, networked and possibily distributed, but slow under high loads due to locking.
• Memory (ie. Memcached): Super fast for simple uses, can be distributed, but non persistent, and it’s simple nature renders prioritized queueing a bit complex.

There also are full blown queueing solutions providing an API to enqueue and dequeue messages:

• Message Queueing Platforms (ie. Apache ActiveMQ): Full blown, but too heavy and/or complex for most simple uses.
• External Message Queueing (ie. Amazon Simple Queue System aka SQS): Robust, simple and distributed, but on a pay per message and pay for bandwidth model, and there’s a REST payload for each request.

The main idea behind cascaded queueing is to attenuate the cons of one queueing storage (poor performance or increased payload) by using two queueing methods simultaneously. With cascaded queueing, you would choose a master queue method for robustness, and the slave queue methods for performance.

Theory Of Operation

As with single queue queueing, all incoming messages get queued in the master queue.

However, the slave queue(s) communicate with the master queue, in order to make sure they keep enough messages locally to provide the processing nodes with them in a timely fashion. So they lock messages in the master queue, and host them locally. There can be a great performance gain here if the master queue is more efficient to locks and send messages in batch.

The processing nodes then lock and request messages to the slave queue. If the locks are only valid for a given amount of time, the slave queue must ensure that the processing node’s lock on the slave queue message must not exceed the slave queue lock on the master queue message. The slave queue may extend the lock on the master queue message if possible, or simply not return that message.

Finally, upon deletion of a message from the slave queue by the processing nodes, the deletion is cascaded to the master queue so the message is definitely removed.

A slave queue data loss will not cause any message to be lost, as they still exist in the master queue. The slave queue can therefore be shut down and rebuilt at anytime, which makes the use of Memcached as the slave queue backend storage possible.

Test Driving It

I first used this technique with MySQL as storage engine for the master queue, and Memcached as storage for the slave queue.

The big advantage of MySQL is that getting 200 messages isn’t a lot slower than getting 1: the locking takes pretty much the same time. However, the average processing time of each of my messages was quite high, and so was the standard deviation: processing a message took 15 seconds on average, but could take anywhere between 1 second and 60 seconds. Thus, processing nodes locking 200 messages at once often resulted in inefficient message distribution among them.

On the other hand, even if I find that Memcached is extremely reliable, it’s non persistent by definition. If you loose one of four nodes, you lose the quarter of your messages. And it’s even worst if your memory gets full: the oldest messages are deleted. But it’s extremely fast, as there’s no disk IO.

Cascading MySQL to Memcached improved the queue performance a lot by removing almost all MySQL inter-locking dead times . Messages were be transferred in batches of 200 to the slave queue every time the queue was at 100 messages or lower, and they were be dequeued one by one by the processing nodes.

As I only had one processing environment, I hosted my slave queue and my master queue at the same place. But one could imagine a slave queue per processing environment as well.

Final note

Obviously, requeueing everything a second time, extending locks and cascading delete requests means extra payload and processing time. Moreover, if not implemented properly, you run the risk of seeing your slave queue repeatedly go empty, having your processing nodes waiting for it to be refilled.

Therefore, cascaded queueing is not the answer to all queueing problem, but rather a way to work around some problems encountered with some queueing methods.

The MP4Info class is a simple extensible PHP class reading the MP4 container’s frames (called boxes) to get various information, namely the video duration, the video/audio codecs, the width and the height, as well as the embedded XMP meta data.

The F4V format is better than the FLV format in many aspects:

• Better compression ratio
• Better quality at similar bit rates
• Open source compressors available, through ffmpeg and libh264 (Flash 8 On2 VP6 codec is proprietary)
• Decoding H.264 is a lot less CPU intensive on the host computer than decoding On2 VP6

It’s quite easy to use: simply load the JavaScript file at the end of your HTML file, and have it point on your JPEG image to an default X/Y position.

<script language="javascript" type="text/javascript" src="watermarker.js"></script>
<script language="javascript" type="text/javascript">
setWatermark("images/mywatermark.jpg",-4,-5);
</script>

And voila, you can toggle your JPEG overlay, move it around, change it’s alpha transparency and reset it to its default position.

A small demo is available here.

watermarker.js (7.8k)

For those not familiar with micro-blogging, it’s pretty much like blogging, but smaller — hence the micro prefix. Micro-blogging posts are very short, 140-160 characters that is. The most popular platform is Twitter, but new platforms are appearing, such as Pownce, Jaiku and Identica. Also, this year’s TechCrunch50 winner is the project oriented micro-blogging platform Yammer (which I didn’t get to try yet).

Back to MicroBlogBuzz, I started this little project part time while testing various APIs, and when stumbled on TwitterBuzz, which presents the most popular links on Twitter. However, TwitterBuzz only present the domain name, which isn’t really meaningful since most micro-bloggers use TinyURL or similar services to shorten their URL. TinyURL was presented as the Top Twitter link. And I thought it was kind of stupid. So I decided to do the same thing, but to follow the HTTP redirections to get to the final URL.

I got surprised by the quantity of data that I would collect. Around 1200 links every 15 minutes, and over 330,000 links and 400,000 comments in five days. I quickly ran into problems, as I built my database on InnoDB and foreign keys to keep things clean, and with this amount of data and the small server it runs on, well it can’t be clean and fast at the same time.

So I switched to MyISAM. But still, it wasn’t enough. And I added caching and smart HTTP headers. But still, it wasn’t enough. And I added preprocessing. And it was ok. At least for now, with my small 500 visitors per day.

Feel free to send me your comments and suggestions, by email, or even better, on Twitter

There are two solutions to this problem, but unfortunately, the inefficient one is often used by programmers, due to the lack of time, or to inexperience.

Imagine, for the sake of clarity, a simple dataset that goes as follow:

Array
(
    [0] => Array
        (
            [name] => Node 0
            [parent] => null
        )

    [1] => Array
        (
            [name] => Node 1
            [parent] => 4
        )

    [2] => Array
        (
            [name] => Node 2
            [parent] => 8
        )
    ...

The common approach is to use a recursive function that finds all the root nodes, then all their sub nodes, then all the sub nodes of their sub nodes, and so on.

function mapTree($dataset, $parent=null) {
	$tree = array();
	foreach ($dataset as $id=>$node) {
		if ($node['parent'] !== $parent) continue;
		$node['children'] = mapTree($dataset, $id);
		$tree[$id] = $node;
	}

	return $tree;
}

The problem with this method is that, in terms of calculation, the problem complexity is exponential: you need to search all the dataset for children nodes for each node. Thus, building a tree of 1,000 nodes is 100 times more complicated than building a tree of 100 nodes, because you will have to search 10 times more nodes, 10 times more often.

A more efficient way to build the tree is to use dereferencing. This allows to map the whole tree in a single pass, without recursion. The problem complexity then becomes linear rather than exponential, and 10 times more nodes equals to 10 times more time (although the time required by PHP to find a specific index of an array isn’t taken into account here).

function mapTree($dataset) {
	$tree = array();
	foreach ($dataset as $id=>&$node) {
		if ($node['parent'] === null) { // root node
			$tree[$id] = &$node;
		} else { // sub node
			if (!isset($dataset[$node['parent']]['children'])) $dataset[$node['parent']]['childs'] = array();
			$dataset[$node['parent']]['children'][$id] = &$node;
		}
	}

	return $tree;
}

The full PHP code of examples can be found below:

tree.zip (1.2k)

Tree image from Wikipedia.org

But so much things are cacheable: pages, images, CSS, JavaScript, even many REST web services! Yes, even in this social web era where content changes faster than you can write, there’s still plenty of slow changing information, such as home pages, or lists of countries, regions and cities.

Efficiently using caching translates into:

• Better response and loading time
• Decreased load on the server
• Better user experience

This article aims to present a simple explanation of the HTTP protocol and proper use of HTTP caching.

The HTTP Protocol basics

The HTTP protocol is a communication scheme between two or three actors: the server, the browser, and the often forgotten proxy.

First, to see the implicated headers, let’s have a look to the typical request …

GET / HTTP/1.1
Host: www.tommylacroix.com
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
If-Modified-Since: Thu, 17 Jul 2008 16:11:24 GMT
If-None-Match: "a9432-fc1b28423-cb122da"
User-Agent: Mozilla/5.0 (...) Gecko/2008052906 Firefox/3.0

… and response …

HTTP/1.1 200 OK
Date: Thu, 17 Jul 2008 16:11:24 GMT
Expires: Wed, 11 Jan 1984 05:00:00 GMT
Etag: "a9432-fc1b28423-cb122da"
Last-Modified: Thu, 17 Jul 2008 16:11:24 GMT
Cache-Control: private, must-revalidate, max-age=0
Vary: User-Agent
Content-Type: text/html; charset=UTF-8

<html>...</html>

Basic Caching

The response headers returned by the server give the browser and the proxy information to make caching decisions.

First, the Cache-Control header tells if the content is cacheable or not. If they’re not sent, it’s cacheable by default. Consider the following headers:

Cache-Control: public, max-age=86400

It basically says that the page is cacheable in a public scope, and that the content shouldn’t be kept in the cache without revalidation for more than 1 day (86400 seconds). The scope can be:

• public: Cacheable by browsers and proxies, even if authenticated
• Private: Cacheable by browsers only (and proxies, but only for requests from the same clients)
• no-cache: Cacheable, but revalidation is required every time.
• no-store: Not cacheable at all.

In addition, there are a few other keywords that you can add to your Cache-Control header. Keep in mind that you must separate multiple keywords by commas.

• must-revalidate: Some proxies can be configured to ignore the Expires and maxage. This keyword forces them to always act like the resource was expired.
• proxy-revalidate: Same as must-revalidate, but only for proxies.
• s-maxage: Same as maxage, but only for proxies.

Finally, for HTTP/1.0 compatibility, you should send a Pragma header when using the Cache-Control no-cache directive.

Pragma: no-cache

Browsers and proxies cache resources based on their URL. You can take advantage of this in many ways at the design stage of your web site or web application. For example, you should put non private REST web service parameters (such as language, or country) in the URL:

http://www.somesite.com/webservice/regionslist/country/usa/language/en

Browsers and proxies also store three important information about the page: the Expires, Last-Modified and Etag headers.

• The Expires header tells the browser and proxy, along with the max-age component of the Cache-Control header, until when this version of the content should be valid.
• The Last-Modified header tells the browser and proxy the date and time of the last modification to this page. It isn’t always provided.
• The Etag header, which stands for Entity Tag, gives the browser and proxy a unique identifier that describes the content it returned. If a page’s content changes, its Etag changes as well.

When you requested this page, your browser (and the proxy in the middle if there was one) did check if the page is cacheable, and since it was, stored a copy of the page associated with the URL, and the three metrics above. For subsequent requests, the server might use this cached copy.

Might? Yes, because the cached page won’t be good until the end of times. At some point, the browser (or the proxy) will check with the server if the page it has in cache is still valid. This is called revalidation, and the Expires HTTP response header along with the max-age component of the Cache-Control header control it.

When the browser or proxy revalidates a page, it sends information about its version: the Etag, and the Last-Modified the server sent when he cached the page. These are sent as If-Modified-Since, and If-None-Match, respectively:

GET /some-cachable-page HTTP/1.1
Host: www.tommylacroix.com
If-Modified-Since: Thu, 17 Jul 2008 16:11:24 GMT
If-None-Match: "a9432-fc1b28423-cb122da"
User-Agent: Mozilla/5.0 (...) Gecko/2008052906 Firefox/3.0

The server then compares this information with the Etag and the Last-Modified of the up-to-date page. If the browser’s cached copy appears to be valid, the server replies with the Expires and Etag (if available), headers, and no content:

HTTP/1.1 304 Not modified
Date: Thu, 17 Jul 2008 16:11:24 GMT
Expires: Wed, 11 Jan 2009 05:00:00 GMT
Etag: "a9432-fc1b28423-cb122da"

If the browser’s cached copy appears to be out-dated, the server replies with the whole page, as usual.

Specific Cases

Now, here comes the tricky cases. For the sake of clarity, I’ll use the following plot for the scenarios below:

« Bob, Alice and Gregg work in the same office. Their office is equiped with a caching web proxy. Bob and Gregg share the same computer with the same user (ok, not credible, so lets say it’s a under-financed non-profit organization), and Alice has her own (she’s the boss).  »

Scenario 1: Secure sessions

« Bob goes on his SuperSocial profile page, at http://www.supersocial.com/profile/. His browser and the office proxy will check if the page is cacheable, and it is. They both store a copy of the page associated with the URL.

But what happens when Gregg or Alice log in his/her SuperSocial’s profile page right after? The browser will give him/her Bob’s page! »

The Cache-Control, Expires, and the Etag headers. Setting the Expires header in the past and the max-age to zero will cause the browser to revalidate the content each time. If the cached content is valid, only a header with no content will be sent. This is slightly slower than full blown caching, but no as much as no caching at all.

We could also set the Cache-Control scope to private, as each copy of the page will obviously only be valid one user.

We should also use Etag, as the content returned for the same URL isn’t the same for Bob and for Gregg, so the Etag will change, and the browser will reload the page for Gregg.

Scenario 2: Content optimization

« The three unproductive workers browse a news site that has a wicked design. So wicked that the guys behind it had to make browser specific optimizations. Therefore, when you’re with Internet Explorer, the page is IE optimized, when you’re with Safari, some in-line CSS styles are different, and when you’re with Firefox, you get the regular page because this beauty is standard-compliant. Standard-what? Lets not digress…

When Bob and Gregg browse the site with IE from the same computer, it’s fine. But when Alice accesses it a short while after with Safari, she gets an ugly page as it’s IE optimized. The reason is, the server sent the IE version to Bob, and the proxy cached it. When Alice requests it, the proxy sends the IE page, thinking that it’s all fine. »

The Vary header. When the content of a page changes based on miscellaneous headers contained in the request, the server must tell the browser and the proxy using the Vary header. The Vary header basically says: If you cache this content, beware that if this header changes, the content might change as well, so use the URL with these fields to make sure you serve a valid version later.

Here’s an example. The request…

GET / HTTP/1.1
Host: www.tommylacroix.com
User-Agent: Mozilla/5.0 (...) Gecko/2008052906 Firefox/3.0

… and response …

HTTP/1.1 200 OK
Date: Thu, 17 Jul 2008 16:11:24 GMT
Expires: Wed, 11 Jan 2009 05:00:00 GMT
Last-Modified: Thu, 17 Jul 2008 16:11:24 GMT
Cache-Control: public, max-age=86400
Vary: User-Agent
Content-Type: text/html; charset=UTF-8

Code Snippet

Here’s a code snippet I wrote quite some time ago. It’s not pretty, but it works so I think it’s a good basic practical example.

/**
* setCacheHTTPHeaders
*
* @author	Tommy Lacroix
* @param	string	$privacy		Scope: public, private or no-cache
* @param	int	$lastModified		Unix timestamp of last page modification (optional)
* @param	int	$maxage			Maximum caching time before revalidation (optional)
* @param	string	$etag			Entity tag, page-content specific (optional)
* @return	bool				TRUE if content need to be sent, FALSE if no content need to be sent
*/
function setCachePolicy($privacy = 'public', $lastModified = false, $maxage = false, $etag = false) {
	// Sanitize privacy
	switch ($privacy) {
		case 'privacy':
		case 'public':
		case 'no-cache':
			break;
		default:
			$privacy = "public";
			break;
	}

	// Calculate expiry and max-age
	if (is_string($maxage)) { // Expiry is a string, interpret
		unset($m);
		if (preg_match('/^([0-9]+)([smhdwy])$/', $maxage, $m)) {
			$maxage = $m[1];
			switch ($m[2]) {
				case 's':	break;
				case 'm':	$maxage *= 60; 		break;
				case 'h':	$maxage *= 3600; 	break;
				case 'd':	$maxage *= 86400; 	break;
				case 'w':	$maxage *= 604800; 	break;
				case 'y':	$maxage *= 31536000; 	break;
			}
		}
	}
	if ($privacy != 'no-cache') {
		header('Expires: '.gmdate("r", time()+$maxage));
	} else {
		header('Expires: '.gmdate("r", time()-31536000));
		$maxage = 0;
	}

	// Send ETag headers
	if ($etag !== false) {
		header('ETag: "'.$etag.'"');
	}

	// Determine wheter we need to send content or not
	$outputContent = true;

	// Check ETag
	if ((isset($_SERVER['HTTP_IF_NONE_MATCH'])) && ($etag !== false)) {
		if ($_SERVER['HTTP_IF_NONE_MATCH'] == '"'.$etag.'"') {
			header($_SERVER['SERVER_PROTOCOL'].' 304 Not Modified');
			$outputContent = false;
		}
	}

	if ((isset($_SERVER['HTTP_IF_MODIFIED_SINCE'])) && ($lastModified !== false)) {
		$ifModifiedSince = strtotime($_SERVER['HTTP_IF_MODIFIED_SINCE']);
		if ($ifModifiedSince >= $lastModified) {
			header($_SERVER['SERVER_PROTOCOL'].' 304 Not Modified');
			$outputContent = false;
		}
	}

	// Remove content if output has been disabled
	if (!$outputContent) {
		// Return false: You don't need to send content
		return false;
	} else {
		// Send other headers
		header('Cache-Control: '.$privacy.', must-revalidate, post-check=0, pre-check=0, max-age='.$maxage);
		if ($privacy == 'no-cache') header('Pragma: no-cache');
		if ($lastModified !== false) header('Last-Modified: '.gmdate('r',$lastModified));

		// Return true: you need to send content
		return true;
	}
}

Conclusion

The caching HTTP headers are simple to implement, and provide a huge performance bonus. If done properly, this convert into a better user experience, as there’s less waiting, and more browsing.

Shall you like me to add other scenarios, feel free to drop me a line and I’ll see what I can do.

Further Reading

• Caching in HTTP, at W3
• HTTP Caching in Mozilla, at Mozilla

One cool sample for Simple Queue Service is the one by Justin@AWS. But (you saw it coming, I suppose), being anti-PEAR (for many reason, such as dependencies, weight, version conflicts, non-catchiness of the name, fructose intolerance, etc.), I just couldn’t leave the sample like that. Yes, this is me again reinventing the wheel.

The original library depended on two PEAR components:

• Crypt_HMAC, for authentication
• HTTP_Request, for HTTP calls

The Crypt_HMAC replacement

I replaced the Crypt_HMAC dependency by three potential routines: hash PECL, mhash, and Lance’s function (which you will never hear in a Numb3r’s episode, as it has nothing to do with a mathematician, but rather with a smart dude who posted a fully PHP HMAC code snippet in the PHP comments of the mhash() function).

The code goes as follow:

/**
 * HMAC function using hash PECL (http://ca.php.net/manual/en/book.hash.php)
 *
 * @author Tommy Lacroix
 * @param string $stringToSign
 * @return string	base64 encoded hmac
 * @internal
 */
private function hmac_hash($stringToSign) {
	return base64_encode(pack('H*',hash_hmac('sha1', $stringToSign, $this->secretKey)));
}	

/**
 * HMAC function using mhash (http://ca.php.net/manual/en/book.hash.php)
 *
 * @author Tommy Lacroix
 * @param string $stringToSign
 * @return string	base64 encoded hmac
 * @internal
 */
private function hmac_mhash($stringToSign) {
	return base64_encode(mhash(MHASH_SHA1, $stringToSign, $this->secretKey));
}

/**
 * HMAC function using Lance's function (http://ca.php.net/manual/en/function.mhash.php, see comments)
 *
 * @author Tommy Lacroix
 * @param string $stringToSign
 * @return string	base64 encoded hmac
 * @internal
 */
private function hmac_lance($stringToSign) {
    $b = 64;
	if (strlen($this->secretKey) > $b) {
		$key = pack("H*",sha1($this->secretKey));
	} else {
		$key = $this->secretKey;
	}
	$key  = str_pad($key, $b, chr(0x00));
	$ipad = str_pad('', $b, chr(0x36));
	$opad = str_pad('', $b, chr(0x5c));
	$k_ipad = $key ^ $ipad ;
	$k_opad = $key ^ $opad;

	return base64_encode(pack('H*', sha1($k_opad  . pack("H*",sha1($k_ipad . $stringToSign)))));
}

HTTP_Request replacement

I replaced the HTTP_Request dependency by two potential routines: cURL and URL wrappers.

The code goes as follow:

/**
 * Fetch with curl
 *
 * @author Tommy Lacroix
 * @param string 	$url
 * @param string 	$qs
 * @param bool		$post
 * @return array(output,httpCode)
 * @internal
 */
private function fetch_curl($url, $qs, $post) {
	$curl = curl_init();
	curl_setopt($curl, CURLOPT_RETURNTRANSFER, TRUE);
	curl_setopt($curl, CURLOPT_SSL_VERIFYPEER, FALSE);
	curl_setopt($curl, CURLOPT_HEADER, false);
	curl_setopt($curl, CURLOPT_POST, $post);
	curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);
	curl_setopt($curl, CURLOPT_FOLLOWLOCATION, false);
	curl_setopt($curl, CURLOPT_URL, $url . (!$post ? '?'.$qs : ''));
	if ($post) {
		curl_setopt($curl, CURLOPT_POSTFIELDS, $qs);
	}

	// Execute
	$output = array();
	$output[0] = curl_exec($curl);
	$output[1] = curl_getinfo($curl,CURLINFO_HTTP_CODE);

	// Close handle
	curl_close($curl);

	return $output;
}

/**
 * Fetch with urlwrappers
 *
 * @author Tommy Lacroix
 * @param string 	$url
 * @param string 	$qs
 * @param bool		$post
 * @return array(output,httpCode)
 * @internal
 */
private function fetch_urlwrappers($url, $qs, $post) {
	$output = array();

	if ($post) {
		$opts = array(
		  'http'=>array(
		    'method'=>"POST",
		    'header'=>"Content-type: application/x-www-form-urlencoded\r\n" .
		              "Content-length: " . strlen($qs),
		    'content'=>$qs
		  )
		);
		$context = stream_context_create($opts);
	} else {
		$opts = array(
		  'http'=>array(
		    'method'=>"GET"
		  )
		);
		$context = stream_context_create($opts);
		$url .= '?'.$qs;
	}

	$output[0] = '';
	$f = @fopen($url,'r',null,$context);
	if (!$f) {
		$output[0] = '';
		$output[1] = 404;
		return $output;
	}
	while (!feof($f)) {
		$output[0] .= fread($f,1024);
	}
	$meta_data = stream_get_meta_data($f);
	if (preg_match('/^HTTP\/1\.[01] ([0-9]{3})/',$meta_data['wrapper_data'][0],$m)) {
		$output[1] = $m[1];
	} else {
		$output[1] = false;
	}
	fclose($f);

	$output[2] = 'wrappers';

	return $output;
}

Detection and conclusion

Finally, the constructor includes a small detection mechanism that selects an HMAC and a fetching function:

/**
 * Constructor
 *
 * @author Justin@AWS
 * @author Tommy Lacroix
 * @param string $accessKey
 * @param string $secretKey
 * @param string $endpoint	http://queue.amazonaws.com
 * @param string $queueName	optional
 * @return SQSClient
 */
public function SQSClient($accessKey, $secretKey, $endpoint = 'http://queue.amazonaws.com', $queueName = '')
{
	.
	.
	.

	// Select fetch function
	// Call it with $this->{$this->fetchFunction}(...)
	if (function_exists('curl_init')) $this->fetchFunction = 'fetch_curl';
		else $this->fetchFunction = 'fetch_urlwrappers';

	// Select hash function
	// Call it with $this->{$this->hmacFunction}(...)
	if (function_exists('hash_hmac')) {
		$this->hmacFunction = 'hmac_hash';
	} else if (function_exists('mhash')) {
		$this->hmacFunction = 'hmac_mhash';
	} else if (class_exists('Crypt_HMAC')) {
		$this->hmacFunction = 'hmac_pear';
	} else {
		$this->hmacFunction = 'hmac_lance';
	}

	.
	.
	.
}

The complete modified sqsClient package can be downloaded below. The original Justin@AWS package is available here.

sqsClientPHP-2.1.zip (11k)

Update (Aug 15, 2008): The AMF0Parser library has been updated to support the Date type in the metas, added among others by the “inlet media FLVTool2″ tool.

Update (Jun 11, 2009): The project has been moved to Google Code. MP4Info, an F4V file (or more genericaly MP4 file) class is also available: check out blog post and project at Google Code.

If you’ve done Flash banners or micro-sites that embed video with cue points for synchronization before, you know that it’s a pain in the arse. There doesn’t seem to be any tool around to modify the damn points once the file is encoded, so you’ve got to re-encode the file over and over to have your things synced correctly.

If you’ve tried to reverse engineer the FLV format before, you know that it’s a pain in the arse as well. The AMF0 format is anything but intuitive, and the documentation has been lacking for a long long time — although there’s apparently an SDK now. Luckily, there was OSFlash, and SabreAMF and AMFPHP that could be used as a documentation source.

If you recognize yourself, this might be your lucky day. After many hours of trial and error, I’ve finally been able to reverse engineer it, and to build a library that extract essential FLV information, reads the Meta, and allows you to write them back — including the cue points.

First, you’ll need to FLVInfo2 and the AMF0Parser libraries, below. The FLVInfo2 library has three main functions:

• getInfo, which returns information gotten from analyzing the FLV file and it’s meta. Frame rate, bit rate, audio/video codec, width, height, etc.

  (
      [signature] => 1
      [hasVideo] => 1
      [hasAudio] => 1
      [minimalFlashVersion] => 8
      [video] => stdClass Object
          (
              [codec] => 4
              [width] => 320
              [height] => 240
              [keyframeRatio] => 0.0180537208278
              [keyframeEvery] => 55.3902439024
              [fps] => 15
              [bitrate] => 448
              [codecStr] => On2 VP6
          )
  
      [audio] => stdClass Object
          (
              [codec] => 2
              [frequency] => 22
              [depth] => 16
              [channels] => 2
              [bitrate] => 48
              [codecStr] => MP3
          )
  )

• getMeta, which returns the meta data, and the cue points

  Array
  (
      [metas] => Array
          (
              [0] => Array
                  (
                      [0] => onMetaData
                      [1] => Array
                          (
                              [duration] => 151.46
                              [width] => 320
                              [height] => 240
                              [videodatarate] => 500
                              [canSeekToEnd] => 1
                              [videocodecid] => 4
                              [audiodatarate] => 48
                              [audiocodecid] => 2
                              [framerate] => 15
                              [creationdate] => Tue Jun 17 13:06:15 2008
                              [Encoded_By] => orangetango Video Encoder
                              [Encoded_With] => orangetango Video Encoder
                              [metadatacreator] => orangetango FLV meta data writer
                          )
                  )
          )
      [cuepoints] => Array
          (
              [0] => Array
                  (
                      [0] => onCuePoint
                      [1] => Array
                          (
                              [name] => name1
                              [time] => 4.41
                              [type] => event
                          )
                  )
              [1] => Array
                  (
                      [0] => onCuePoint
                      [1] => Array
                          (
                              [name] => name1_end
                              [time] => 6.7
                              [type] => event
                          )
                  )
          )
  )

• rewriteMeta, which inputs an FLV file, meta data and cue points, and outputs a new FLV file.

As a practical example, you find the rewriteTagsForFile function in the example.php file, that rewrites the meta data and gets the cue points from a FLVCoreCuePoints XML file.

flvinfo2.php (20k , updated Dec 20, 2008)

AMF0Parser.php (16k, updated Dec 20, 2008)

example.php (12k)

Flash Video logo found on Wikipedia.

I’ve stumbled upon Rémi Prévost — who’s blog a follow more or less consistently due to the lack of time — CIOpenID module. It’s basically a CI embed of Janrain’s PHP_OpenID.

His code is great and works well. But one thing I don’t like about it is that it requires a different bootstrap file (a modified version of index.php) and a somewhat hacked version of your typical .htaccess file. The reason is, CodeIgniter annihilates the $_GET variable during initialization, because GET queries aren’t secure (ok, this is overly simplified, but you get the idea).

Being who I am, and somewhat liking to reinvent the wheel during my free time, I used a different approach. Rather than using a different bootstrap file, I rather chose to use the pre_system hook.

The installation process is quite simple:

1. Put my ciopenid.php library in your /system/application/libraries directory.
2. Create /system/application/libraries/openid, and unzip PHP_OpenID 2.x.x’s Auth directory in it.
3. Open your /system/application/config/config.php file, and make sure that hooks are enabled:

   $config['enable_hooks'] = TRUE;

4. Open your /system/application/config/hooks.php file, and add the following lines:

   $hook['pre_system'][] = array(
      'class'    => 'ciopenid',
      'function' => 'pre_system_hook',
      'filename' => 'ciopenid.php',
      'filepath' => 'libraries',
      'params'   => null
      );

To make an authentication query to an OpenID URL in $openid:

$this->load->library('ciopenid');
try {
   $redirect_url = $this->ciopenid->authenticate($openid, site_url($this->uri->uri_string()));
   header("Location: ".$redirect_url);
} catch (Exception $e) {
   die('Error: '.$e->getMessage());
}

And to get the OpenID query response:

$this->load->library('ciopenid');
if (isset($this->ciopenid->response)) {
   switch ($this->ciopenid->response->status) {
      case Auth_OpenID_SUCCESS:
         $openid = $this->ciopenid->response->endpoint->claimed_id;
         print 'Success: '.$openid;
         break;
      case Auth_OpenID_CANCEL:
         print 'User cancelled';
         break;
      default:
         print 'Internal error';
         break;
   }
}

That’s it! Ooops, here’s the file…

ciopenid.zip (3k)

Enjoy!

• Big impact, small changes on Amazon (04/03/2008)
• Effective text only emails (04/03/2008)
• A Refreshing Take on Usability (04/03/2008)

Shall you stumble on great usability pages, feel free to leave me a comment and I’ll give it a look.

Big impact, small changes on Amazon (April 3rd, 2008)

You probably didn’t notice unless you’re a savvy Amazon shopper (whether you shop for books or for usability practices) , but Amazon recently made subtle but great changes to the design. They improved readability of key information elements, and redesigned the up-selling space. Check the link below for a great before/after comparison (that I didn’t feel comfortable to rip even if I was going to credit the author).

Via: Future Now’s blog

Effective text only emails (April 3rd, 2008)

Text emails don’t have to be bad and boring. Okay, they’ve got to be boring. But still, when HTML emails aren’t an option, or even just to provide a text alternative for all these new non-HTML cell phone email readers out there, there are ways to make text emails look better:

• Put links on new lines
• Uses lists with asterisks as bullets
• Try to keep it short
• If you can’t keep it short, divide in sections
• Call to action

Via: Shape Shed

A Refreshing Take on Usability (April 3rd, 2008)

This is more a funny heads up than a news, but it shows how well one can use available information to improve user experience.

Via: B#