Data can end up in disparate spots for a variety of reasons. Deliberate actions can be taken in the interest of not leaving all your eggs in one basket. Some organizations end up in a sort of data drift, rolling out servers and databases for different projects until each bit of data is its own island in a massive archipelago.
Regardless of how things got this way at your operation, there are a number of dangers and challenges to this sort of setup. Let’s take a look at why disparate data can end up being a business killer.
Multiple Points of Failure
At first blush, this can seem like a pro. The reality, however, is that cloud computing and cluster servers have made it possible to keep your data in a single pool while not leaving it subject to multiple points of failure.
Leaving your data in disparate servers poses a number of problems. First, there’s a risk that the failure of any one system might wipe information out for good. Second, it can be difficult to collect data from all of the available sources unless you have them accurately mapped out. Finally, you may end up with idle resources operating and wasting energy long after they’ve outlived their utility.
It’s best to get everything onto a single system. If you want some degree of failure tolerance beyond using clouds or clusters, you can set up a separate archive to store data at specific stages of projects. Once your systems are brought up to speed, you’ll also begin to see significant cost savings as old or excess servers go offline.
Inconsistency
With data spread out across multiple systems, there’s a real risk that things won’t be properly synchronized. At best this ends up being inefficient. At worst it may lead to errors getting into your finished work products. For example, an older dataset from the wrong server might end up used by your analytics packages. Without the right checks in place, the data could be analyzed and out into reports, producing flawed business intelligence and decision-making.
Likewise, disparate data can lead to inconsistency in situations where multiple teams are working. One group may have its own datasets that don’t line up with what another team is using. By centralizing your efforts, you can guarantee that all teams will be working with the same data.
Bear in mind that inconsistency can get very far out of hand. If you need to audit data for legal purposes, for example, you may find data that has been retained too long, poorly anonymized or misused. With everything centralized, you’ll have a better chance of catching such problems before they create trouble.
Security Risks
More systems means more targets. That opens you up to more potential spots where hackers might get their hands on sensitive data. Similarly, you’re stuck with the challenge of patching multiple servers when exploits are found. In the worst scenario, you may not even notice a breach because you’re trying to juggle too many balls at the same time. Simply put, it’s a lot of work just to end up doing things the wrong way.
Turf Wars and Company Culture
When different departments in control of different data silos, it’s likely that different groups will start to see the data within their control as privileged. It’s rare that such an attitude is beneficial in a company that’s trying to develop a data-centric culture. Although you’ll want access to be limited to appropriate parties, there’s a big difference between doing that in a structured and well-administrated manner versus having it as the de facto reality of a fractured infrastructure.
Depending on how culturally far apart the departments in a company are, these clashes in culture can create major friction. One department may have an entirely different set of security tools. This can make it difficult to get threat monitor onto a single, network-wide system that protects everyone.
Conflicts between interfaces can also make it difficult for folks to share. By building a single data pool, you can ensure greater interoperability between departments.
Conclusion
Consolidating your data systems allows you to create a tighter and more efficient operation. Security can be improved rapidly, and monitoring of a single collection of data will allow you to devote more resources to the task. A unified data pool can also foster the right culture in a company. It takes an investment of time and effort to get the disparate data systems under control, but the payoff is worth it.
Some members of r/datasets on Reddit have released a dataset of all comments on the site dating back to 2005. The datasets are categorized by year and are available to download for free by anyone and it could be a fun project to analyze the data and see what could be discovered about reddit commenters.
