Author: Scottie Todd

When performing data analysis, it can be easy to slide into a few traps and end up making mistakes. Diligence is essential, and it’s wise to keep an eye out for the following 7 potential mistakes you can make. These include:

• Sampling bias
• Cherry-picking
• Disclosing metrics
• Overfitting
• Focusing only on the numbers
• Solution bias
• Communicating poorly

 Let’s take a look at why each one can be problematic and how you might be able to avoid these issues.

The Why

Sampling bias occurs when a non-representative sample is used. For example, a political campaign might sample 1,300 voters only to find out that one political party’s members are dramatically overrepresented in the pool. Sampling bias should be avoided because it can weigh the analysis too far in one particular direction.

Cherry-picking happens when data is stacked to support a particular hypothesis. It’s one of the more intentional problems that appear on this list because there’s always a temptation to give the analysis a nudge in the “right” direction. Not only is cherry-picking unethical, but it may have more serious consequences in fields like public policy, engineering, and health.

Disclosing metrics is a problem because a metric becomes useless once subjects know its value. This ends up creating problems like the habit in the education field of teaching to what’s on standardized tests. A similar problem occurred in the early days of internet search when websites started flooding their content with keywords to game the way pages were ranked.

Overfitting tends to happen during the analysis process. Someone might have a model, for example, and the curve produced by the model seems to be predictive. Unfortunately, the curve is only a curve because the data fits the model. The failure of the model may only become apparent, however, when the model is compared to future observations that aren’t so well-fitted.

Focusing only on the numbers is worrisome because it can have adverse real-world consequences. For example, existing social biases can be fed into models. A company handling lending might produce a model that induces geographic bias by using data derived from biased sources. The numbers may look clean and neat, but the underlying biases can be socially and economically turbulent.

Solution bias can be thought of as the gentler cousin of cherry-picking. With solution bias, a solution might be so cool, interesting or elegant that it’s hard not to fall in love with. Unfortunately, the solution might be wrong, and appropriate levels of scientific and mathematical rigor might not be applied because refuting the solution would just seem disheartening.

Communicating poorly is more problematic than you might expect. Producing analysis is one thing, but conveying findings in an accessible manner to people who didn’t participate in the project is critical. Data scientists need to be comfortable with producing elegant and engaging dashboards, charts and other work products to ensure their findings are well-communicated.

How to Avoid These Problems

Process and diligence are your primary weapons in combating mistakes in data analysis. First, you must have a process in place that emphasizes the importance of getting things right. When you’re creating a data science experiment, there need to be checks in place that will force you to stop and consider things like:

• Where is the data coming from?
• Are there known biases in the data?
• Can you screen the data for problems?
• Who is checking everybody’s work?
• When will results be re-analyzed to verify integrity?
• Are there ethical, social, economic or moral implications that need to be examined more closely before starting?

Diligence is also essential. You should be looking at concerns about whether:

• You have a large and representative enough sample to work with
• There are more rigorous ways to conduct the analysis
• How you’ll make sure analysts are following properly outlined procedures

Tackling a data science project requires sufficient and ample planning. You also have to consider ways to refine your work and to keep improving your processes over time. It takes commitment, but a group with the right culture can do a better job of steering clear of avoidable mistakes.

Back to blog homepage

Data literacy is quickly becoming one of the most crucial skills in a world that’s increasingly dominated by stats, business processes, machine learning, computing, and AI. Unfortunately, a massive skills gap has developed, producing adverse effects within every company that uses data in its operations. 

Issues of data literacy have emerged all the way up to the C suites of multinational corporations, with one study showing that just 24% of business decision-makers have the necessary skills. Worse, there appears to be zero generational benefits for folks who grew up in the data age, with digital natives posting a 22% rate of acceptable data literacy. The U.S. lags Europe, and worldwide the issue is problematic in even the best of environments. 

Only 17% of businesses report openly encouraging data literacy training, even though employees have almost uniformly expressed a desire for it. Likewise, only 36% of the same businesses report providing any incentives, such as higher salaries, for employees to upgrade their skills.

Defining Data Literacy

Among the trickier issues is defining what data literacy means. It’s important to distinguish data literacy from digital literacy. Many people handle basic digital skills well, but they commonly lack grounding in reading and comprehending data. There is also a high level of dependency on machines to get the answers right without really knowing what the implications of such attitudes are.

Data literacy in business predominantly consists of three main areas. These are:

• Data science
• Programming
• Analytics

Each of these fields is underpinned by the discipline of statistics. That means people reading data, for example, need to be able to understand and apply concepts like:

• Sample size
• Regression
• Correlation
• Bias

They also need to be familiar with issues such as data preparation, analysis, archiving and presentation. Even professionals who are highly skilled at preparing and analyzing data, for example, may lack skills on the visualization side of the equation. It’s important to understand how particular presentations of data may aid or hurt comprehension, especially when the audience is being introduced to the information for the first time. As nit-picky as it sounds, the precise choice of one style of chart versus another imposes biases and creates possible comprehension issues.

What’s the Cost?

Unsurprisingly, all this comes at a cost. An estimate from the Data Literacy Project indicates that for a company valued at over $10 billion, the cost of the data literacy gap may cause economic damage to the order of 5% of the firm’s market cap. That represents more than $500 million worth of value. Despite these severe numbers, only 8% of businesses reported making significant changes to how they approach their data issues.

In addition to the immediate costs, there are also opportunity costs which are much harder to estimate. The absence of data-driven decision-making at many companies means those organizations are falling behind more progressive competitors in fostering data cultures. It’s not uncommon for businesses to lack data management titles such as a Chief Data Officer, this signifies the absence of governance overseeing many organizations’ data strategies. This also highlights organizational issues that aren’t preparing employees to be assuming those data executive roles 10 and 20 years from now.

How to Close the Gap

There are two areas where change can immediately occur. These are:

• Training
• Providing incentives

Solving these two issues is well within the means of modern companies. As previously stated, the majority of employees at companies are eager to join a data literacy effort through training. Likewise, providing incentives is not only logical in terms of encouraging training, but it becomes a retention issue. After all, you don’t want to spend money training people who end up at other organizations solely based on better pay.

At a larger scale, the adoption of a data-centric culture is critical. Data can’t just be a work product that’s presented to the higher-ups. From the C suites on down, every company needs to understand how to read data, interpret it and talk about its implications. With a focused effort, your organization can become one that starts taking advantage of new opportunities.

Back to blog homepage