In the last few years we’ve all heard amazing stories of how “big data” can make uncanny predictions about users (e.g., Target knowing when someone is pregnant based on their purchases). However, there are just as many stories of when analytics gets it wrong (e.g., Google Flu Trends overestimating flu cases in 2013). It shouldn’t be at all surprising when predictions based strictly on click analysis (sometimes with demographic data thrown in) gets it wrong because analytics tells us only the WHAT, not the WHY. In the case of Google Flu Trends, the massive media coverage influenced people to search on the keywords that previously indicated one was coming down with the flu but now simply meant people were curious about it. Our logs were missing the WHY behind the keywords.
At Google, we conduct an annual Experience Sampling Methodology (ESM) study with a large sample of our users recording their needs, context, and experience to capture the WHY we cannot see in our query stream. Over a five day period, participants tell us about their experiences throughout their day and submit photos to explain things words alone cannot capture. Doing this over a several month period every year allows us to monitor changes in user’s subjective and objective behavior for a clearer picture than analytics alone.