This Clicktale review is completely independent. We’ve used it on several projects where we wanted to better understand user interactions. In doing this we had a chance to look at both the way it collects data and how it reports this information. We’ve also had the opportunity to compare the results against other other software tracking tools, as well as other approaches to the same task such as physical eye tracking. We started using Clicktale based on many of the positive reviews we’d read, but on further investigation many of these were paid reviews (via affiliate commission).
To summarise our experience, we were quite disappointed with the results. The Clicktale reports seem to illustrate certain behaviours and user problems, but after some investigation we realised these weren’t problems.
Users were in fact behaving very differently to what this tool was telling us.
We wasted a lot of time investigating issues that we couldn’t reproduce, and worrying about defects which weren’t there.
Our problem was that we realised that some of the Clicktale data was wrong but the trouble was we didn’t know which bit.
Clicktale seems good for creating impressive looking but superficial “feel good” reports (look - users are using our web site!). The visitor recordings and heatmaps make for great eye candy to flash in front of a bewildered client or manager. Clicktale really blow away their competition both in terms of generating pretty results, and also explaining this well on their website. The site also explains how cheaper it is to use Clicktale vs Eye Tracking and actual user testing, and also state that the results will be close enough to be the same.
Where things fell apart for us was when we tried to use Clicktale to dig deeper. We wanted to analyse how people went through a series of forms, what they were interested in, and gain better insight into where people were getting confused or dropping off.
Clicktales main premise is that it will record what someone has done on your site and then show you a video style playback of what occurred. You can watch what they do and understand how real users behave. This information can also be summarised in a heatmap, which shows where users have clicked and pointed.
The problem is that there are a few technical issues with accurately recording users behaviour, which Clicktale hasn’t really solved.
Here’s a simple example, using the Twitter join form as an example. The first screencast is what a user actually sees:
This is (in my humble opinion) a pretty nice join form. It gives lots of clear feedback and helpful suggestions. The form uses Ajax to check if that username is available or in the right format, plus comes up with some helpful suggestions. In this example user tries a few and then chooses one of the suggested names.
This is how the same user behaviour on this form would look like via Clicktale. Note the absence of any kind of prompts or help text, and the pauses while the user reads the validation message (which isn’t visible).
If you were viewing this playback to get a sense of how the form was being used, you might conclude that the user is having real trouble with filling it in. Not only that, but none of the prompts are appearing and the form looks totally broken. Have we tested in all browsers? Is this a problem that occurs under load? What is this user doing differently to all our test results?
The reason for this discrepancy is that Clicktale isn’t actually taking a screencast of what the user is doing. Although you could be forgiven for thinking this - it is how it is explained quite clearly on their site.
See absolutely everything visitors do on your webpage. Watch recordings of your visitors full browsing sessions to discover exactly how they use your site. Its as if you’re looking over their shoulder!
Using Clicktale is not like looking over their shoulder. Clicktale records mouse position every second or so and keyboard strokes. Then, in a separate process, a Clicktale bot visits your site and takes a screenshot of that page. The actual playback is an animation of this screenshot and an image of a mouse cursor moving over the top of it. Similarly the heatmap uses this screenshot.
There are a few problems with this approach, but the primary one is that the screen that you’re seeing in the playback can be quite different to the one that your user just saw. For a site like this one (simple blog style site with static pages that are identical for every user) this approach works. If the site you’re testing has any features that change depending on what a user has done, it won’t.
A few of the ways it can be different:
None of these are particularly obscure or unusual elements for a web site to have- in fact I’d be surprised if most sites other than a basic static brochureware site wouldn’t use at least a few of these.
If it’s still unclear as to why this would be a problem, here is what I’d see if I used Clicktale to analyse how people are using the Bookdepository checkout process. As you can see the two screens are quite different. The Clicktale playback will still merrily animate a mouse cursor moving around, but as you can see the image would be completely wrong and misleading. Likewise for the various heatmaps and reports.
What a real user sees:
and what you’ll see in the Clicktale report:
Which is puzzling, because on the Clicktale site there are a lot of testimonials gushing about this amazing insight into behaviour which improves shopping and conversions. I’m not really sure how many of these sites don’t use some sort of cookie or session information to facilitate a checkout process.
OK, maybe Clicktale can provide insight in other ways like using the heatmaps to see what people are interested in. Again, we’ve been down the rabbit hole on this one, trying to solve problems that it turns out never actually existed.
Heatmaps run into the same problem as the playback issue above - the screenshot in the background might not be what the user was looking at. You might notice a lot of heatmap activity around the top of your page. That might be people clicking around the header, or if you have a dropdown menu they might be clicking on the menu items. Or they might have been clicking on a message area or banner ad that isn’t in the screenshot.
Now, I’ve always had a soft spot for a nice heatmap, but Clicktale ones have another problem - they’re not heatmaps of what people are looking at, they track where their mouses are pointing.
The Clicktale blog assures us that this isn’t anything to worry about, since there is a very strong correlation between where a user looks and where they point their mouse (84-88). My bullshit detectors went off when I read that. Google pointed me to a few other posts that pointed out that the 88 is based on a misreading of the research, and the figure is more like 30. I’d be surprised if it was even as high as 30, and if you believe the 88% please contact me as I have a very interesting investment scheme in Nigeria that will make you very rich. I promise.
Here’s a simple test: as you read this line of text, where is your mouse right now? Is it hovering over each word? I’m guessing not.
Say we take the 30% number as a broad average, what these heatmaps are actually telling us is where people are not looking, most of the time. Try explaining that to your client or manager.
I’ve raised this discrepancy with Clicktale between the report they cite and what they write, asking for more clarification about exactly where the information comes from. The response was the data was from Google
Ironically an instance where a user would look at where their mouse is would be as they are filling in a field or clicking a button. Typically things youd do when filling out a form (see issue above)
Other issues we hit:
At the end of the day, Clicktale wasn’t really a great fit. Were trying a more pragmatic approach - solve real problems rather than using a tool to find problems. Some examples:
We’ve had several discussions with the guys at Clicktale, they’ve provided some more information, and we’ve done some analysis of their recent updates. And they’d really like us to change this post.
We didn’t discover anything that changed our opinion of this software. In fact some of their new innovations are pretty scary. For example there is a cookie sniffer. But to make this work you need to turn of SSL. Which is a Very Bad Idea. The whole point of encrypting cookies and sessions is to stop other people intercepting that. It is almost that Clicktale comes from an alternate reality where the normal rules of web security and development practices don’t apply.
And to be fair it seems this criticism can be applied to many of the “alternatives to ClickTale” - we know a bit because they keep on contacting us to tell us how awesomely different they are to ClickTale.
Here’s the core problem:
Here’s what you can do:
Clicktale review which looks at how the technology works and some real world issues that you might face when implementing Clicktale
31 Jan 2012