Category: Big Data

With available data in the modern world emerging on a second-by-second basis, one of the most prominent areas of interest for organizations is the creation of continuous intelligence. A report from Gartner indicates that by 2022 more than half of all major business systems will employ continuous intelligence in their operations.

While this sounds like an exciting opportunity for any data-centric enterprise, you might wonder, though, what the pros and cons of utilizing continuous intelligence may be. Let’s break our analysis down along those lines to examine how a business might employ this emerging technology.

What Are the Pros of Using Continuous Intelligence?

By far the biggest pro of collecting and developing business intelligence continuously is greater opportunity. A company will have the opportunity to:

• Constantly monitor developments
• Stay ahead of less agile competitors
• Look at issues with the most contemporaneous data possible
• Produce analytics and work products on an on-demand basis

Consider how a bank might approach the use of continuous intelligence to handle fraud detection. As information about transactions flows in, the bank’s analysis operations would continuously stream data to machine learning algorithms. More importantly, as bad actors try to sneak around existing filters by adapting their tactics, the fraud detection system will note the tactical changes and adjust accordingly. Rather than waiting for customers to complain or relying on filters that can only be adapted sporadically, the bank will have a constantly evolving response to constantly evolving threats.

Enhanced predictability is also a major pro for continuous intelligence. A seaborne oil transportation business, for example, might use continuous monitoring of local news and social media feeds to detect upticks in pirate activity before tankers move into more dangerous regions. Decision-makers could be delivered a list of options that might include things like routing changes in response to threats, flying in additional security personnel, or notifying police and military forces in the area.

Optimization is a big opportunity for continuous intelligence. Global supply chains have come under increasing strains in recent years. Continuous intelligence systems can scan data regarding demand, supply, geopolitical activity, macroeconomic trends, and news reports to distill what the state of the market is. Rather than waiting for shelves to be empty or warehouses to be overflowing, the company could adapt in real-time to ensure goods are sufficiently meeting needs.

What Are the Cons of Using Continuous Intelligence?

The volume of data to be analyzed poses several potential hazards to organizations that depend on continuous intelligence. In particular, there’s a danger that voluminous data will create conditions where noise starts to look like a pattern. Quality control is a significant job when using continuous intelligence as a decision-making tool, and it’s critical to employ data scientists who understand the importance of always questioning the inputs and outputs.

Common human errors in handling analytics can creep in as well. With a continuous intelligence system constantly supplying insights, it’s easy to fall into the trap of treating the machine like it’s an oracle.

Additionally, dirty data is also a huge problem. If a system is feasting on continuous inputs, it can be hard to perform the necessary amount of quality control. Consider what happens if a data vendor changes the formatting of their XML documents. If the changes are implemented without notice, there’s a real risk that the continuous intelligence system will keep humming along while generating ill-informed insights.

How to Put Continuous Intelligence to Work

The adoption of continuous intelligence tools is a fact of the business world. If you want to dive into using it, it’s worth noting that you should already have a mature data operation in place. A firm that isn’t already generating numerous business intelligence insights should ease into continuous models due to the many real-time complexities involved.

Big Data capacity has to be built out in a way that exceeds what many operations currently use to produce BI. A company has to create an operating environment where bandwidth, connection speeds, and hardware are sufficient to sustain continuous data collection. For an enterprise that already has a robust Big Data department, though, continuous intelligence represents a chance to take their game to the next level.

Back to blog homepage

Having the skills needed to perform data science work is immensely beneficial in a wide range of industries and job functions. But at some point, it is also advantageous to develop a thought process that allows you to tackle problems like a data scientist. Here are 7 steps you can take to start thinking like one.

1. Understand How the Project Lifecycle Works

Every project needs to be guided through a lifecycle that goes from preparation to building and then on to finishing it. Preparation means setting goals, exploring the available data, and assessing how you’ll do the job. Building requires planning, analyzing problems, optimizing your approach, and then building viable code. Finally, finishing requires you to perform revisions, deliver the project, and wrap up loose ends. The lifecycle installs rails around the project to ensure it doesn’t suffer from mission creep.

2. Know How Time Factors into Cost-Benefit Analysis

Scraping the web for all the data you need may prove to be time-consuming, especially if the data needs to be aggressively cleaned up. On the other hand, purchasing data from a vendor can be expensive in terms of capital. There’s rarely a perfect balance between time and money so try to be receptive to which is more important on a particular project.

3. Know Why You’ve Chosen a Specific Programming Language

All programming languages have their unique strengths and weaknesses. For example, MATLAB is a very powerful language, but it often comes with licensing issues. Java handles work with a high level of precision, but it can be cumbersome. R is an excellent choice for people who need core math functions, but it can be limiting when it comes to more advanced functionality. It is essential to think about how your choice of a programming language will influence the outcome of your project.

4. Learn How to Think Outside of Your Segment of Data Science

 It’s easy to get caught in the trap of thinking certain processes are somehow more academically valid than ones aimed at the consumer market or vice versa. While something like A/B testing can feel very simple and grounded in the consumer sector, it may have applications to projects that are seemingly more technically advanced. Be open-minded in digesting information from sectors that are different from your own.

5. Appreciate Why Convincing Others is Important

Another common trap in data science is to just stay in your lane. Being a zealous advocate for your projects can make a difference in terms of getting approval and resources for them.

Develop relationships that encourage the two-way transmission of ideas and arguments. If you’re in a leadership position at a company, foster conversations with individuals who are closer to where the data gets fed into the meat grinder of analysis. Likewise, those down the ladder should be confident in presenting their ideas to people further up the chain. A good project deserves a representative who’ll advocate for it.

6. Demand Clean Data at Every Stage of a Project

Especially when there’s pressure to deliver work products, cleaning up data can sometimes feel like a secondary concern. Oftentimes, data scientists get their inputs and outputs cleaned up to a condition of “good enough” to avoid additional mundane cleaning tasks.

Data sets rarely just go away when a job is done, and that’s simply good practice for the sake of retention, auditing, and reuse. But, that also means someone else may get stuck swimming through a data swamp when they were expecting a data lake. Leave every bit of data you encounter looking cleaner than you found it.

7. Know When to Apply Critical Thinking

Data science should never be a machine that continually goes through the motions and automatically spits out results. A slew of problems can emerge when a project is too results-oriented without an eye toward critical thinking. You should always be thinking about issues like:

• Overfitting
• Correlation vs. causation
• Bayesian inference
• Getting fooled by noise
• Independent replication of results

Welcome criticism and be prepared to ask others to show how they’ve applied critical thinking to their efforts. Doing so could very well save a project from a massive misstep.

Back to blog homepage

Going to work on a big data project can leave you wondering whether your organization is handling the job as effectively as possible. It’s wise to learn from some of the most common mistakes people make on these projects. Let’s look at 5 critical big data project mistakes and how you can avoid them.

Not Knowing How to Match Tools to Tasks

It’s tempting to want to deploy the most powerful resources available. This, however, can be problematic for a host of reasons. The potential mismatch between your team members’ skills and the tools you’re asking them to use is the most critical. For example, you don’t want to have your top business analyst struggling to figure out how to modify Python code.

The goal should always be to simplify projects by providing tools that match their skills well. If a learning curve is required, you’d much prefer to have non-technical analysts trying to figure out how to use a simpler tool. For example, if the only programming language choices are between Python and R, there’s no question you want the less technically inclined folks working with R.

Failing to Emphasize Data Quality

Nothing can wreck a big data project as quickly as poor quality. The worst of possible scenarios is that low-quality and poorly structured data is fed into the system at the collection phase, ends up being used to produce analysis, and makes its way into insights and visualizations. 

There’s no such thing as being too thorough in filtering quality issues at every stage. You’ll need to keep an eye out for problems like:

• Misaligned columns and rows in sources
• Characters that were either scrubbed or altered during processing
• Out-of-date data that needs to be fetched again
• Poorly sourced data from unreliable vendors
• Data used outside of acceptable licensing terms

Data Collection without Real Analysis

It’s easy to assemble a collection of data without really putting it to work. A company can accumulate a fair amount of useful data without doing analysis, after all. For example, there is usually some value in collecting customer service data even if you never run a serious analysis on it.

 If you don’t emphasize doing analysis, delivering sights and driving decision-making, though, you’re failing to capitalize on every available ounce of value from your data. You should be looking for:

• Patterns within the data
• Ways to benefit the end customer
• Insights to provide to decision-makers
• Suggestions that can be passed along

Most companies have logs of the activities of all of the users who visit their websites. Generally, these are only utilized to deal with security and performance problems after the fact. You can, however, use weblogs to identify UX failures, SEO problems, and response rates for email and social media marketing efforts.

Not Understanding How or Why to Use Metrics

The analysis necessarily noteworthy if it’s not tied to a set of meaningful and valuable metrics. In fact, you may need to run an analysis on the data you have available just to establish what your KPIs are. Fortunately, some tools can provide confidence intervals regarding which relationships in datasets are most likely to be relevant.

For example, a company may be looking at the daily unique users for a mobile app. Unfortunately, that company might end up missing unprincipled or inaccurate activity that causes inflation in those figures. It’s important in such a situation to look at metrics that draw straight lines to meaningful performance. Even if the numbers are legit, having a bunch of unprofitable users burning through your bandwidth is not contributing to the bottom line.

Underutilizing Automation

One of the best ways to recoup some of your team’s valuable time is to automate as much of the process as possible. While the machines will always require human supervision, you don’t want to see professionals spending large amounts of time handling mundane tasks like fetching and formatting data. Fortunately, machine learning tools can be quickly trained to handle jobs like formatting collected data. If at all possible, find a way to automate the time and attention intensive phases of projects.

Back to blog homepage