Is it ethical to keep online case studies of clients that are not using your solution anymore? Or to present case studies in a way that could mislead your potential clients?
What I found out
I'm getting closer to the next version of my WASP Firefox extension, and to ease the testing process, I figured I could build a quality assurance test where I would visit client sites referred by each vendor. A list of high-profile clients shown with emphasis on the home page and case studies is a proven way to increase confidence and credibility. But here's what I found out...Google Analytics case studies
I was in for a surprise! Take the case of Google, which highlights a number of case studies:- BuildDirect Google Analytics
- Discount Tire: Google Analytics
- RE/MAX: Google Analytics
- Roche: Google Analytics
- Vueling: Google Analytics
- CKE Restaurants, for the Spicy Paris campaign: Google Analytics
- Financial Times: WebTrends live, Google Analytics is used for campaign landing pages
- Agency.com: WebSideStory HBX v1.4... no traces of Google Analytics
- Career Builder: Omniture SiteCatalyst, even CBCampus.com, the site referred in the case study, uses Omniture and as no traces of Google.
- Deckers: none identified (might be log based).
I'm sure Google Analytics is not the only solution provider in this situation. In fact, I've even found a number of vendors that are not using their own solution on their corporate web site...
What about others?
I took a look at a couple of the Omniture Customer Showcase and a random sample of 10 of those sites resulted in 100% Omniture usage (along with some double-tagging with Google Analytics). Same thing for VisualSciences WebSideStory Customers... I can stop there.My two-cents
- Can you put a date on the case studies? Being in the web analytics field, you know that what makes a good story is the context. Without any time reference, we loose a valuable contextual element that can easily lead to assumptions and misjudgment.
- Take their claims with a grain of salt (and pepper). Everybody is a "market leader", and an "industry first" at something. Cut the crap and do your homework: does the solution fit your needs, not those of others!
- As they say at Microsoft: "Eat your own dog found": if a vendor is not even using its own tool (and especially if using Google Analytics!), how can you believe their solution is anything but useless? (This is especially true of smaller players, but even some of the major vendors are using both their own and Google Analytics!).
WASP can help you detect which web analytics solution is being used by top sites and competitors. And it can help you insure the quality of your own site tagging.
Named one of the most influential industry contributors by the Digital Analytics Association. With over twenty years’ experience empowering organizations to analyze and optimize their online channels, Stéphane has cemented his position as a leading voice for online analytics and optimization.
4 comments:
Hey Stephane - this does sound a bit like Google bashing to me. Is there a reason for this? As you might expect I would like to comment :)
[Quick disclaimer for others reading: As you know I have seen both sides of the fence when it comes to working with Google - first from the outside running SEO/Adwords campaigns for clients, and then from the inside running the web analytics team for Google in Europe. Sure I am a Googlephile - I like their approach and vision, but that does not mean I am not critical of the company.]
Coming back to your post. Firstly, if a client in the past was happy enough with a product to publicly put their name to the vendor's marketing material, then that is still valid even if they later change to a different vendor for what ever reason. Yes, if its 10 years our of date it will be stale, but that hardly applies here.
Secondly, I would argue that case studies for free tools actually have a higher value than those for paid tools. As you know, there are very significant financial incentives for clients to agree to be a case study for paid vendors, whilst there are none for free tools e.g. Google, Yahoo Analytics etc.
Thirdly, as I understand it you only looked for tracking codes on home pages of the case study references. For large multi-national enterprises, it is quite common for each country/brand to operate independently of the rest of the company. For example, TUI in Germany may operate a different tracking tool than TUI in the UK. In my experience, few companies have a global tracking tool - IBM (Coremetrics), Sony BMG (GA), P&G (GA), EMI Music (GA) are a few. So whilst you might not see the tracking code on the corporate home page, that does not mean someone in the business is not getting real benefit from the tool they have control over for their part of the business.
I agree of course, client portfolios and case studies should also be taken with a grain of salt.
My 2 cents worth.
Brian
Thanks for your comment Brian. If you look at the post date, that was published a long time ago (August 2007), when GA didn't have the strengths it has today. And sadly, looking back at the post, I agree it sounded like Google bashing...
I really appreciate your comments, and I would like to add to them:
1 & 3) if a client has agreed to put their name upfront, that's great and valuable. At the time, I was surprised to see so few of the mentioned companies really had GA deployed. Using it for a campaign or a small site was not what those case studies seemed to imply. That being said, I was also shocked to find out how many other vendors were not even using their own tools on their sites!
Note that shortly after this post, Google asked for "success stories" they later used as replacements/additions to the case studies that were there at the time.
2) I disagree with your second POV. Google is not giving away GA without any financial incentive... That could be an interesting debate. Especially now that we see AdWords stats being merged into GA.
And finally, the most important point when I titled was "be wary of case studies", be it GA or any other vendors. Case studies are marketing material, not independent reviews. Companies should still do their homework and see if GA (or any other vendor) fits their needs.
Regards,
Stephane
Ah - now I see the date. Apologies for being so trigger happy that day. In fact not only GA, but the marketing teams at Google have come a long way in that time period...!
Just to clarify, there are NO financial incentives for working with Google as a Success Story/Case Study, other than having an experienced person (i.e. a member of my Team at the time), work with you to understand your website performance.
That said, better understanding usually equals an improved ROI, so I guess you could say that indirectly there is a financial incentive for the client. However, that is very different to the point I was making wrt paid tools.
Lets agree to disagree on that one :)
No problem Brian, in fact, I appreciate this kind of comments because it allow us to continue the conversation and clarify our thoughts.
When you said "financial incentives" I didn't view it from the client perspective, but from the vendor POV. I think for any vendor (paid or free), delivering case studies showcasing high-value, high-visibility clients necessarily takes its roots from a financial incentive.
From the client POV, I would find it ethically inappropriate for a vendor to offer a financial incentive to get showcased. On this point, I totally agree with you.
Stéphane
Post a Comment