Tuesday, February 3, 2009

Postulates of Web Analytics

I must admit I stole the idea from Prof. Raymond Hames of the Department of Anthropology & Geography at University of Nebraska-Lincoln: "Goals and Characteristics of Science". I think approaching web analytics with the same scientific rigueur makes a lot of sense!
"The aim of science is to seek the simplest explanation of complex facts. ...Seek simplicity and distrust it."
--A. N. Whitehead, The Concept of Nature
"... when one's thoughts about the way things are supposed to be conflict with what seems to be the actual truth of the matter, we reexamine those notions which led us to erroneous predictions ..."
--Niles Eldridge, Time Frames

The 15 axioms of web analytics

  1. Web analytics is logical (utilizing the appropriate form of logic), reasonable, and rational.
  2. Web analytics makes well-defined claims based upon the best available evidence.
  3. Hypothesis must be falsifable.
    (Falsifiability (or refutability or testability) is the logical possibility that an assertion can be shown false by an observation or an experiment.)
  4. Web analytics requires that claims be examined by qualified peers.
    (Or challenged by the HiPPO - "Highest Paid Person Opinion"!)
  5. Web analytics views unexplained gaps in theories or evidence with suspicion.
  6. Exercise caution both in performing experiments, and in examining and evaluating evidence.
  7. Experiments should be repeatable under similar circumstances.
    (Apply multivariate testing methodology whenever possible.)
  8. Web analytics requires efforts at objectivity, both in control of variables and of biases.
  9. Web analytics does not accept coincidence or unlinked or unproven correlations as proofs.
  10. Web analytics does not accept undocumented anecdotal evidence as good proof by itself.
  11. Web analytics demands extraordinarily good evidence for extraordinary unconventional claims.
  12. Web analytics favors parsimony: that the simplest adequate explanation is preferred.
    (Favor continuous improvement/evolution rather than absolute certainty or truth)
  13. Web analytics demands the honest use of the scientific method and truthful reports.
  14. Web analytics demands every effort be made to control or assess all (appropriate) variables.
  15. Web analytics needs the uninhibited exchange of ideas and greatest possible discourse of the material and knowledge.
When I read this quote I thought of web analytics, multivariate testing and behavioral targeting (or even just cookies!) and how obscure and incomprehensible those things are to most people:
"Although science does not ascribe an absolutism to its laws, it does not follow that one theory is as good as another.... To the general public, uneducated in even elementary science, and used to many technical miracles, the scientific-sounding jargon and forceful arguments of cranks are often convincing."
-- G. D. Goodman

What do you think? Do you agree with those axioms? Should they be formulated differently or are there any others?