Friday, March 23, 2007

A case about reporting vs. analysis

When designing a web site, we need to understand the demographics, psychographics and other characteristics of our audience. Elements to consider includes such technical considerations as the bandwidth, browser, and computer being used.

In this example, I will use Omniture SiteCatalyst but the same type of analysis could be done with any solution. For the sake of this demonstration, let's assume it's a B2B site, so visitors comes from very specific partner companies.

I was faced with a very good case where analysis, instead of reporting, made all the difference. Take a simple statement: 15% of the visitors to the web site use Windows 2000/IE 5. We know this could lead to slightly different design considerations.

If we are simply reporting, we're stuck with 15%... is it worth designing for this platform? How can we tell?

Since we suspect Windows 2000/IE 5 use might be declining, let's look at the usage trend for our visitors. We already have a clearer indication, going from nearly 16% down to 12% in 5 months. Plus the fact Omniture can tell us the Internet average is about 6%, indicating our site receives a larger than average share of Windows 2000 visitors.

Could there be just a handful of visitors (companies) using Windows 2000 and influencing this reporting? Now we need to create a Custom Datawarehouse Report using Omniture's Advanced Analysis. We can ask for the number of visits by OS, grouped by Company, with a breakdown by month. Once we get the CSV file loaded in Excel, a simple Pivot Table can give us a very good look at what is going on.

Now we can spot that only a couple of companies, some of them representing a fair share of the traffic, have much higher usage of Windows 2000 (42% in the above example). This makes sense because it appear one of those companies is a banking environment, and we know those corporate environment traditionally takes longer to adopt more recent technologies. Furthermore, we see there's also a clear negative trend, from 122 visits in October to 54 in February (while XP is generally growing).

Now that we have done a more thorough analysis, we're in a much better position to decide if we need to invest in designing for this platform. From a simple report stating "15%" with very limited context, we were able to identify more detailed behavior and address the impact on a limited subset of our visitors.

In the spirit of the 10/90 rule coined by Avinash Kaushik, getting the number was easy. The analysis is where the value is: how is the site structured? who's your audience? which other reports and metrics will help you get the right decision? Too often, people are complaining about the tool: it's not working, it's not configured correctly, it doesn't give me my numbers... I guess that's where experience and an analytical mindset can make all the difference.

(Ultimately, the recommendation was to avoid integrating costly exceptions in the design and the development process.)