The Subtle Sources of Sampling Bias Hiding in Your Data

Read the full story in MIT Sloan Management Review.

When a group of Boston College studentsi started an analytics project using data about UFO sightings, they thought they might learn more about visits from little green creatures — or at least something about how weather or movie releases influence human perceptions of those extraterrestrials, similar to The Economist’s findings about sightings being linked to times when people are “nursing their fourth beer” leading to a “close encounter of the slurred kind.”

Instead, the students learned about sampling bias.

UFO sighting reports have increased exponentially since the National UFO Reporting Center first started tracking them in 1974. But this might not mean that we are in fact getting more visitors from outer space.

When the reporting center first began, communicating a sighting required making a telephone call to file a report. When the internet became publicly available, people could report a sighting using an online form. Unsurprisingly, increases in the number of sightings correspond with public adoption of the internet. When reporting UFOs became easier and cheaper, more data about sightings became available. Increased access to data fundamentally changes the sample.

Decreased costs of data collection are great; we have much more data than ever before. But managers must be careful to understand how the data was generated to be able to make good decisions.

Unfortunately, in business, the sources of bias can be far subtler than in this UFO example. What should managers consider as they work to gain insight from increasingly available data?

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s