Tuesday, 3 February, 2009

Postulates of Web Analytics

2/03/2009  Stéphane Hamel  6 comments

I must admit I stole the idea from Prof. Raymond Hames of the Department of Anthropology & Geography at University of Nebraska-Lincoln: "Goals and Characteristics of Science". I think approaching web analytics with the same scientific rigueur makes a lot of sense!

"The aim of science is to seek the simplest explanation of complex facts. ...Seek simplicity and distrust it."
--A. N. Whitehead, The Concept of Nature
"... when one's thoughts about the way things are supposed to be conflict with what seems to be the actual truth of the matter, we reexamine those notions which led us to erroneous predictions ..."
--Niles Eldridge, Time Frames

The 15 axioms of web analytics

  1. Web analytics is logical (utilizing the appropriate form of logic), reasonable, and rational.
  2. Web analytics makes well-defined claims based upon the best available evidence.
  3. Hypothesis must be falsifable.
    (Falsifiability (or refutability or testability) is the logical possibility that an assertion can be shown false by an observation or an experiment.)
  4. Web analytics requires that claims be examined by qualified peers.
    (Or challenged by the HiPPO - "Highest Paid Person Opinion"!)
  5. Web analytics views unexplained gaps in theories or evidence with suspicion.
  6. Exercise caution both in performing experiments, and in examining and evaluating evidence.
  7. Experiments should be repeatable under similar circumstances.
    (Apply multivariate testing methodology whenever possible.)
  8. Web analytics requires efforts at objectivity, both in control of variables and of biases.
  9. Web analytics does not accept coincidence or unlinked or unproven correlations as proofs.
  10. Web analytics does not accept undocumented anecdotal evidence as good proof by itself.
  11. Web analytics demands extraordinarily good evidence for extraordinary unconventional claims.
  12. Web analytics favors parsimony: that the simplest adequate explanation is preferred.
    (Favor continuous improvement/evolution rather than absolute certainty or truth)
  13. Web analytics demands the honest use of the scientific method and truthful reports.
  14. Web analytics demands every effort be made to control or assess all (appropriate) variables.
  15. Web analytics needs the uninhibited exchange of ideas and greatest possible discourse of the material and knowledge.
When I read this quote I thought of web analytics, multivariate testing and behavioral targeting (or even just cookies!) and how obscure and incomprehensible those things are to most people:
"Although science does not ascribe an absolutism to its laws, it does not follow that one theory is as good as another.... To the general public, uneducated in even elementary science, and used to many technical miracles, the scientific-sounding jargon and forceful arguments of cranks are often convincing."
-- G. D. Goodman



What do you think? Do you agree with those axioms? Should they be formulated differently or are there any others?

Posted in: ,,,

6 comments:

I think the axioms are well worth aiming for, but web analytics as it stands today is lacking rigorous scientific theoretical foundations. Before we can begin calling web analytics a science, it has to have proper theoretical foundation. I don't think we have that yet. Elements of the work do rely on real science, however, there are still more elements which are not. Brandt Dainow touched this topic when he criticized (rightly so, in my opinion) standardization work done by the WAA.

@ Anonymous: In that case, I don't think I want WA to be a science. The center of our work IMHO should be the needs of our customers. If one definition of simple metrics like page views or visits suits a customer A and a customer B has completely different needs, he should use metrics that do him the most good - and not follow a definiton he does not need or want.

The science in WA is in methodology, is in conclusions from measured data and is in conclusions from the known bias in this data. Science is not dogmatic defininitons and standards.

Nicolas: very good point! I was even wondering if the initial (anonymous) comment didn't come from M.Dainow himself! :)

I'm currently working on my MBA thesis, which will be a literature review of web analytics. Honestly, so far there has been lots of books on the topic, but few (if any?) were written with the academic rigor of a scientific approach. It seems most of those books were written by consultants for "visibility/credibility" purposes. What do you think?

Hm, I'm not sure, what a scientific approach on WA really _is_.

Maybe

1) A categorization of methods, their development over time and their distribution over countries/cultures/clusters.

2) A list of methods and an analysis of their success and failure to reach several goals within set restrictions.

3) An inquiry of broader (marketing) theories for specific technology-dependent parts as particular character of WA. [there is a heavy axiom in this]

4) A reversal of 2) - Define goals and restrictions and develop methods to measure the goals or overcome the restrictions.

5) Describe where online marketing, marketing controlling, general controlling, PR etc. belong to the field of WA and where they definitely do not belong to WA. (which would soon create schools fighting each other - hence create a solid basis for a scientific career :-) )

Nicolas: I think all of those points would be valid paths to explore! There are lots of opportunities for research! I've been thinking if doing a Doctorate in the field would be worth it, and the deeper I look, the more obvious it becomes... but that would be a huge endeavor!

@Stephane: I can assure the anonymous comment is not from Mr. Dainow.
I agree with your observations about the web analytics literature; so far I have not come across books matching the academic rigor of scientific approach.

@Nicolas & Stephane: The point I wanted to make with my original comment is that web analytics should not be called science until it meets the criteria. The discussion of usefulness of web analytics is a different matter. As a practitioner, I can attest it can be useful in many respects. Even many of the books written on the subject have been most useful :-).

Post a Comment