Hearing that your organization has dark data can make you think of your data as quite ominous and menacing. Saying that data is dark, however, is closer in meaning to what people are talking about when they say a room is dark. What they mean is there’s the potential for someone to switch the light on and make what was unseen visible.
What is Dark Data?
Every operation in the world produces data, and most of those entities record at least some of it regardless of whether they make further use of it. For example, many businesses collect information about sales, inventories, losses, and profits just to satisfy the basic reporting requirements for taxes and how their companies are set up. You might also have a complete customer service department that’s producing data all the time through daily chats, emails, and many other forms of communication. Even maintaining a social media presence means creating data.
Such data is considered dark if it isn’t put to other uses. Shining a light on dark data can allow a company to:
- Conduct analysis
- Creating sellable data products
- Learn about relationships
- Supply insights to decision-makers
By definition, dark data is an unutilized resource. Owning dark data is like keeping things in storage that never or rarely get used. In other words, if you tolerate the existence of dark data within your organization, you’re at risk of leaving money on the table. In fact, you may be taking a loss on dark data because you’re storing it without first turning to monetization.
How to Bring Dark Data into the Light
The first order of business is figuring out exactly what your organization has in the way of data sources. Some things will be fairly obvious, such as turning up sales data from a POS and inventory numbers from an ICS. Other data sources may be trickier to find, but they can be discovered by:
- Surveying your team members to learn what data different departments collect
- Conducting audits of computing systems to identify databases, log files, and spreadsheets
- Scanning through social media feeds, including direct messages from customers
- Collecting corporate data, such as financial statements and email correspondence
- Studying call records
It’s also wise to think about places where you could be collecting more data. For example, a customer service system that isn’t sending out surveys is letting a perfectly good opportunity go to waste.
What to Do Now That You See the Data
The second order of business is figuring out how to draw more insights from your data. Companies accomplish this by:
- Creating data lakes and providing access to them
- Auditing databases for potentially useful information
- Implementing or expanding data science projects
- Developing data-centric corporate cultural practices
- Adding resources to do machine learning and stats work
- Hiring new professionals who can dig into the data
Much of this hinges on moving forward with a data-centric culture. Even if you already feel that you have one, there’s a lot to be said for looking at who your team members are and how you can use dark data with experienced data users at your disposal.
The third order of business is establishing goals for your projects. If you run a company that has potential legal risk exposure due to compliance problems involving laws like HIPAA and the GDPR, for example, you might analyze the very way your organization stores information. A company that collects huge amounts of anonymous data from millions of users might figure out how to package that data into sellable information products, such as reports or datasets. You may even cut costs by determining what data is useless, potentially removing terabytes of information from storage.
Conclusion
Modern organizations collect so much data that it’s hard for them to clearly imagine what they have at their disposal. It’s important to take an extensive look at all the ways dark data may be residing in your systems. By being a bit more aggressive and imaginative, you can find ways to improve processes, cut costs, and even drive profits.
