Get Started
Watch Demo
Get Started
Watch Demo

Category: Data Analytics

Categories
Big Data Data Analytics

The Advantages of Automation for Wastewater Treatment

Automation is everywhere these days, helping us to make better use of labor, time and other resources, plus leading to the development of cleaner, future-ready industries. It’s not a surprise to see automation edging into wastewater treatment: this is one of the most critical industrial-level activities on planet earth today where public health is concerned. Here are some of the ways technology is making this process more efficient and cost-effective.

Lower Energy Costs

Not surprisingly, energy use is the single biggest expense for wastewater treatment plants. Automating infrastructure provides 1 way to reduce energy expenditures associated with a number of critical water treatment processes. One example is the blowers located in holding basins, which keep the water aerated. Some estimates say blowers account for up to 60% of a treatment plant’s total energy consumption.

Automation can improve cost-effectiveness in this area through data collection. Instead associated with operating the blowers constantly, at the fixed speed, plants can use information about effluent levels in holding basins to apply air and remove solids only when it’s necessary to do so. This reduces energy costs, helps maintain a steady flow and reduces wear and tear on equipment.

Constant Access to Data and Ongoing Sampling

In wastewater treatment and many…

Read More on Dataflow

Categories
Big Data Data Analytics

How Artificial Intelligence Will Disrupt the Financial Sector

Artificial intelligence thrives with data. The more data you have, the better your algorithms will be. However , just having a lot of data is not sufficient anymore. You also need high-quality data, or in the words of Peter Norvig, you need better data:

“We don’t possess better algorithms, we just have a lot more data. More information beats clever algorithm, but better data beats more information. ” – Peter Norvig – Director of Research, Google

Nowadays, most organisations collect vast troves of data, but especially the financial sector is well-suited for also collecting high-quality data. Simply because of regulations and because a lot of data in the financial sector is structured data. There is also an abundance of data within public markets or even other external sources that can become linked for additional insights. As it seems, banks and insurance companies can benefit a lot from AI, if implemented correctly, of course.

Financial Institutions Have to Innovate

Besides, more and more consumers require financial institutions to innovate. They have become fatigued with overbearing fees to their manage capital and provide products such as credit. The below graph by State of AI clearly shows the difference in costs between traditionally managed wealth and automated management of wealth. As a…

Read More on Dataflow

Categories
Big Data Business Intelligence Data Analytics Data Monetization

7 Ways To Grow Your Business with Data Monetization

It’s estimated that by the year 2020 revenues around the world for big data and business analytics are going to exceed $203 billion. With all this earning potential, it makes sense to want to get your business “in on it.”

One of the best ways to do this is with data monetization. After all, data is the new currency.

In the past, businesses in the information technology sector have always been deriving value from data. However, the ability to effectively use and monetize data is now impacting virtually all types of business.

This means that driving value from data is something you can implement in your own business strategy. What many people may not realize is that this process can be extremely challenging.

As a result, you need to learn some helpful tools and actionable steps you can take to monetize data for your business.

If you are interested in learning more, then keep reading.

1. Decision Architecture

When thinking about analytics, the majority of organizations want to know how their business is performing, along with what information is needed to answer various performance questions. While this can help to inform and to describe what is taking place in the organization, it doesn’t enable any type of action.

Instead, the goal needs to be to capture the decision architecture of specific business problems. Once this is done, you can build analytics capabilities to create a diagnosis that enables decisions and actions. Leaders need to focus on making decisions that are based on data, rather than just answering questions about what already happened.

2. Stop Revenue Leaks

Busy healthcare providers, clinics, and hospitals can easily lose track of the services being rendered. Every procedure has an assigned code and description. Each of these often includes errors.

By using analytics, the organizations can identify patterns associated with procedures and codes, flagging patient invoices for possible errors or even missing charges. Intelligent data use can also help the organizations improve the ROI of their collections process.

3. Data Aggregation

The method that is at the very bottom of the pyramid, but that represents the biggest opportunity to earn, is data aggregation.

This means taking data from various sources, including your business, and merging it together to create a larger, integrated picture. While the data sources on their own may be interesting, when they are combined, they become valuable.

An example of this would be your credit report. The information credit bureaus aggregate, such as the credit cards you have, if you have a mortgage, and if you pay your bills on time, can be sold for a profit.

By aggregating this information into a single report, the information can be sold to interested parties. While there isn’t a lot of money in this, it’s still money.

4. Infer Customer Satisfaction

Many organizations use social media and survey sentiment to understand the levels of customer satisfaction. By combining data from several sources, airlines can now infer how satisfied a customer is based on factors, like where they are sitting.

This process requires information to be aggregated from several sources. However, in the airline example, you can use the information to determine if a customer is going to fly with you again, and if not, offer a free upgrade or other incentives.

5. Embrace a New Revenue Model

Today, data is actively changing relationships companies have with customers. Manufacturers of tangible goods are now supplementing the products they sell with flexible software options and services to offer customers new choices and new revenue streams.

Additionally, these companies are providing much higher levels of personalization. Across several industries, new economic models are starting to be explored – like replacing an auto fleet with self-driving cars.

In this example, rather than selling data, people are going to pay you to solve a problem or to provide answers. This is a unique revenue model.

The value lies in the fact that you have married your data to the mission of a business and solving a problem that businesses have. This is what is going to generate revenue.

6. Detect Piracy and Fraud

Most online retailers sell products on several different websites. Supplemental sales channels typically include eBay, Amazon.com and other online marketplaces maintained by larger retailers, like Best Buy and Walmart.

Selling through these channels is extremely data-intensive, since the customer types, products, and pricing can vary greatly across the channels. In some case, the price discrepancies are so large that they signal possible piracy or fraud.

If you sell across dozens of e-commerce websites, then consider building databases of your own products and your unique pricing. You can then compare this to existing expected pricing data, allowing you to detect stolen goods or suppliers who are mispricing their goods.

With this information, it’s possible to go to the marketplace and make a report stating that they believe someone is selling stolen items.

How Can You Use Data Monetization Methods for Your Business?

Data monetization is an ever-evolving concept that offers opportunities to earn profits by providing information to others. Your business can take advantage of this by utilizing the tips and information here.

The fact is, there are already countless businesses, in all industries, that are currently using data monetization. Now is the time to begin doing so, too, as it offers huge revenue stream potential.

If you are convinced that data monetization is something you want to use for your company, then contact us. We can provide you with help and information about how this process works.

Categories
Big Data Data Analytics

How Social Media Data Can Boost Your Sales

Social media data is one of the richest sources of information available to modern marketers, influencers, website operators and data scientists. One of the challenges, though, is finding the right way for your operation to harness that power. Let’s take a look at how social media data can boost your sales.

The Raw Data in Social Media

There are plenty of ways to deploy data analysis tools to both mine data and derive insights from it. These include looking at data points like:

  • Shares and likes
  • Mentions
  • Hashtags
  • Click-thrus to URLs
  • Addition and loss of followers
  • Demographic groups
  • Influencer networks

It’s important to not obsess about the vanity metrics, though. All the followers in the world don’t mean much if they’re not translating into sales. For example, tracking codes need to be embedded with URLs to verify that social media followers are moving into the marketing funnel. By using embedded referral codes specifically designed for your social media presence, you can keep tabs on whether followers are converting.

Finding useful sources of data is also important. There are plenty of free options, such as pulling marketing data from:

  • Facebook Insights
  • Google Analytics
  • Twitter Analytics
  • LinkedIn Analytics

Some social media companies, such as Instagram, also offer paid access to their data. In many cases, however, it’s possible to pull data using other solutions, such as web scrapers.

If your setup is properly configured, you should be able to track engagement as it moves through your marketing funnel. For example, your Twitter-specific referral code will show up in both Twitter Analytics and Google Analytics, making it easier to tie user behavior to particular campaigns.

Developing Insights from Social Media Data

The best pool of information means nothing if you can’t use data analysis tools to derive insights from it. Foremost, you need to know what goals your business is shooting for. You can make a checklist that covers things like:

  • Acquiring new customers
  • Developing a more widely recognized brand
  • Making decisions based on social media data
  • Responding better to customer concerns
  • Fostering a superior customer experience

Let’s say your business wants to focus on social media as a way to quickly identify customer complaints. One great thing about social media is that folks quitting your brand might not call your customer support hotline to express their discontent, but you can bet they’ll complain to their friends online about your company’s products and services.

One way companies take advantage of this is sentiment analysis. This is a data-driven decision-making tool that focuses on gathering data regarding positive, negative and neutral statements that people make about companies online. By regularly scanning social media, these firms are able to “read the room” at a global scale. Instead of letting customer anger fester out of sight, sentiment analysis allows companies to get out in front of problems.

There is also plenty of information hiding in the networks that folks form on social media. Marketing data can be developed by creating network maps of their social associations. For example, a retailer that wants to build an influencer campaign on Instagram wants to know which users are going to spread ideas the fastest. They can then supply those Instagram influencers with:

  • Early access to product details
  • Marketing and brand materials
  • Product demos and samples
  • Immediate access to top-tier customer and technical support
  • Opportunities to meet with key players
  • Invitations to company-sponsored events

Driving Business Decisions

Using marketing data should not be seen as a one-way street. There’s a lot that can be learned by monitoring the social media sphere. Trend analysis, for example, can allow companies to get ahead of what people are excited about. A clothing company might focus on analyzing trends coming into each of the fashion seasons, allowing them to handle ordering issues like:

  • Choosing quantities
  • Conveying customer demands to overseas buyers
  • Establishing transport times to put products in stores in time for trends to peak
  • Re-ordering items that are expected to sell out

It’s important to develop a data-driven culture at a company in order to make the most of social media data. Stakeholders and decision-makers shouldn’t be stuck wondering what the social media budget is actually doing. By deploying dashboards, data scientists at companies can provide real-time, engaging insights to those parties. In no time at all, folks who once questioned data and social media expenditures will be checking the dashboards on their cellphones to see how campaigns are unfolding.

Building this sort of data-centric business culture requires an investment. Infrastructure has to be put in place to ensure data scientists on your team have the servers they need to pull data, clean it up, analyze it and generate insights. Done the right way, though, building out this sort of infrastructure can help you get a better grasp on how customers interact with your brands, products and services.

Read More Here

Categories
Big Data Data Analytics Data Quality

What You Need to Know About Monetizing Healthcare Data

Healthcare services providers generate huge amounts of data in the course of any given year. Many organizations, though, see this work as a source of financial losses. In a more modern view of the situation, however, all this healthcare data maintenance can be seen as a potential way to decrease losses and to create profit centers. Let’s explore some of the ways data monetization can benefit a business in the healthcare industry.

Ethical and Legal Concerns with Data Monetization

HIPAA is, rightly, the dominant issue when dealing with the legality of any monetization effort, but not as much as one might think. Bear in mind that anonymization, when performed competently, does cover the confidentiality issues related to HIPAA.

The more concerning problem is on the ethical side of the equation. In particular, efforts to anonymize data need to focus on ensuring identifying factors, such as addresses, Social Security numbers and even uniquely assigned identifiers aren’t traceable to any one patient. This can be surprisingly challenging, as evidenced by work from MIT researchers that found anonymized datasets could be mapped to individuals based on location data and networks.

When setting up data sets, you definitely want to discuss these worries in detail with the parties handling them. Other stakeholders, including doctors, patients and your organization’s lawyers should be included in the process.

One solution worth considering is asking patients to opt in to information sharing. This requires creating a framework that guarantees the confidentiality of the data, and there also needs to be legal language that explains patients’ rights in detail. Such documents should always include an opt-in process that requires a patient to clearly indicate their interest and to provide their consent. This is absolutely essential if you’re going to be monetizing data by selling to third parties.

Reducing Losses

Much of the low-hanging fruit in the industry comes from situations where data analysis can provide insights regarding losses. In the simplest form, this comes from streamlining processes, such as:

  • Scheduling appointments between doctors and patients more efficiently
  • Avoiding duplication of medical efforts
  • Preventing potential slip-ups
  • Maintaining contact with patients about screenings and check-ups

There’s also another level at which healthcare data can be utilized to spur the well being of patients. For example, insurance carriers and hospitals mining patient data have discovered trends among their customers where preventive measures can be taken. By providing those customers with medical devices, preventative screenings and other forms of care, they’ve reduced costs by avoiding more expensive, radical and reactionary solutions.

Healthcare data can also be utilized to establish the efficacy of medical options. Rather than relying on old habits and tried-and-true solutions, professionals can utilize large-scale to develop insights about which drugs produce the best outcomes for the dollar. Data is also employed in researching:

  • Genomic and epigenetic concerns
  • Pharmacology studies
  • New drug discoveries and treatments

Developing Healthcare Data Profit Centers

While HIPAA rules limit the amount of specific data that can be sold in relation to patient care, anonymized data is still a strong potential profit center. Researchers, insurance companies, government agencies and marketers are all looking for information that can fuel their own data analysis. This sort of data can benefit groups that are trying to develop:

  • Economic models
  • Government policies
  • Metastudies
  • Information regarding rare disease
  • Trend analysis

Packaging data for third parties carries with it several concerns that need to be taken seriously. Foremost, it’s important that all patient data be scrubbed of any identifying features. Secondly, large banks of data become targets for hackers, and it’s important to secure all your systems. Thirdly, aggregation of anonymous data will likely demand some investment in bringing in qualified data scientists, establishing in-house standards and building out computing resources.

There is also the cultural component that comes with all efforts to become more data-centric. Stakeholders need to be brought in on monetization efforts, and it’s critical to confirm they are on board with the technical, cultural, legal and ethical requirements of the process. While you don’t want to clear out folks who have honest objections, there usually are situations where stakeholders have to be bought out of contracts or given severance packages. Your goal during a monetization push should be to develop a larger organizational commitment to doing it well.

A commitment to data and monetization takes time. Resources and talent have to be accumulated, and data often has to be prepped for outside consumption. This means taking into account data consumers’ concerns about data lineage, unique identifiers and other information that allows them to do their job well. Being able to present both internal stakeholders and third parties with finished products can make offerings significantly more appealing.

Plenty of thought goes into monetizing data from a healthcare organization. In time, though, a portion of your business that seems like it costs you money can end up curtailing losses and generating new sources of revenue.

Read more similar content here

Categories
Big Data Data Analytics Data Quality

What is Data Lineage & Why is it Important?

In the world of data analytics in 2019, keeping tabs on where bits of information came from, how they were processed and where they ended up at is more important than ever. This concept is boiled down to two words: data lineage. Just as a dog breeder would want to the lineage of a pooch they’re paying for, folks in the business intelligence sector want to know the lineage of the data that shows up in a final work product. Let’s look at the what, the why and the how of this process.

What is Data Lineage?

The simplest form of lineage for data is indexing items with unique keys that follow them everywhere. From the moment a piece of data is entered into a system, it should be tagged with a unique identifier that will follow it through every process it’s subjected to. This will ensure that all data points can be tracked across departments, systems and even data centers.

The concept can be extended significantly. Meta-data about entries can include information regarding:

  • Original publication dates
  • Names of authors
  • Copyright attributions
  • The date of the original entry
  • Any subsequently dates when it was accessed or modified
  • Parties that accessed or modified the data
  • Analytics methods that were used to process the data

In other words, the lineage functions as a pedigree that allows anyone looking at it to evaluate where it came from and how it got where it is today.

Why Does This Matter?

Within the context of business intelligence, there will always be questions about the inputs that went into a final product. Individual data points can be reviewed to discover problems with processes or to show how transformations occurred. This allows folks to:

  • Perform quality control on both the data and analytics techniques
  • Explain how particular insights were arrived at
  • Consider alternative approaches
  • Refine techniques
  • Mine older sources of data using new technologies

When someone wants to pull a specific anecdote from the data, the lineage allows them to get very granular, too. In the NBA of 2019, for example, shot location data is used to study players, set defenses and even choose when and where to shoot. If a coach wants to cite an example, they can look through the lineage for a shot in order to find film to pull up.

The same logic applies in many business use cases. An insurance company may be trying to find ways to deal with specific kinds of claims. No amount of data in the world is going to have the narrative power of a particular anecdote. In presenting insights, data scientists can enhance their presentations by honing in on a handful of data points that really highlight the ideas they’re trying to convey. This might include:

  • Providing quotes from adjuster’s reports
  • Comparing specifics of an incident to more generalized insights
  • Showing how the numbers align
  • Talking about what still needs to be studied

Data governance is also becoming a bigger deal with each passing year. Questions about privacy and anonymization can be answered based on the lineage of a company’s data. Knowing what the entire life cycle of a piece of information is ultimately enhances trust both within an organization and with the larger public.

Cost savings may be discovered along the way, too. Verification can be sped up by having a good lineage already available. Errors like duplication are more likely to be discovered and to be found sooner, ultimately improving both the quality and speed of a process. If a data set is outdated, it will be more evident based on its lineage.

The How

Talking about data lineage in the abstract is one thing. Implementing sensible and practical policies is another.

Just as data analytics demands a number of particular cultural changes within an organization, caring about lineage takes that one step further. It entails being able to:

  • Document where all the company’s data came from
  • Account for who has used it and how
  • Explain why certain use cases were explored
  • Vouch for the governance of the data with a high level of confidence

At a technical level, databases have to be configured to make tracking lineage possible. Data architecture takes on new meaning under these circumstances, and systems have to be designed from the start with lineage in mind. This can often be a major undertaking when confronting banks of older data. If it’s implemented in the acquisition and use of new data, though, it can save a ton of headaches.

Conclusion

Tracking the lineage of a company’s data allows it to handle a wide array of tasks more professionally and competently. This is especially the case when pulling data from outsides sources, particularly when paying for third-party data. Not only is caring about lineage the right thing to do, but it also has a strong business case to back it up.

READ MORE HERE

Categories
Big Data Data Analytics

Your Company Needs to Be Data Driven, Here’s Why

Data driven decision making is an increasingly important part of the modern business landscape. Amazingly enough, 58% of business leaders who responded to one survey said that the majority of their decisions are still based on gut feelings or experience rather than data. While the human element will never be eliminated from the process of making decisions, there’s a strong argument for an organization focusing on developing a data driven attitude.

Data Driven Decision Making Frequently Fixes Biases

For most industries, making money is a question of discovering what hasn’t yet been exploited by other companies. Spotting and exploiting these sorts of inefficiencies allows firms to gain first-mover advantages.

The folks who run call centers at Xerox Services turned to big data to reassess how they pick job candidates for interviews. The initial proposed solution based on the data left some managers downright shocked. In some cases, the system was actually sending in people with no relevant prior experience. It also singled out individuals who were on four or more social networks to not be sent in. As the program moved forward, though, attrition rates for new hires dropped 20%.

How did this happen? Data driven decision making often moves companies past human biases. Human hiring managers frequently look for signals that feel relevant but aren’t. The machines cut out all the noise of human interaction, focusing on results rather than imputing biases.

The Data Analytics Arms Race

In some industries, building out data analytics capabilities is well on its way to being an arms race between companies. The NBA has been revolutionized by analytics, with the league utilizing technologies derived from missile-tracking systems to keep tabs on every footstep and dribble made in each game. A league that was once dominated by the slam dunk rapidly switched to 3-pt shooting, and the Golden State Warriors are widely considered the first champion built on hard data. Other teams have since been racing to catch up.

On Wall Street, companies that use programmatic trading and algorithms are considered dinosaurs stuck in the 1980s. Private equity has long since moved beyond learning from the past and is now focused on predictive data analytics. One high-frequency trading firm posted a profit in 1237 out of 1238 trading days. It’s easy to see why “data scientist” is the hottest job trend in finance.

Data Driven Marketing

Some sectors have found the concurrent rise of social media and big data to be the confluence of events they required to get out in front of the competition. For large corporations, this has allowed them to target niches that were often unreachable. If you’ve walked through the grocery store and read the packages, there’s a good chance you’ve seen data driven marketing in action. Brands like Betty Crocker and General Mills now frequently emphasize niche selling points such as “non-GMO” and “gluten-free.” These selling propositions were designed by sifting through social media data to find what concerns drove consumer decisions. The brands then adjusted their marketing to have appeal to both the general public and niche markets, allowing them to maximize their exposure without making massive investments in advertising. Instead, they changed a few things on their packages.

Cutting Costs

The difference between a profitable year and a bad one often boils down to nothing more than costs. Nearly 50% of Fortune 1000 firms say they’ve started data driven initiatives to cut expenses and seen a return on the investment.

In the fashion world, using big data to track trends has become a key part of the purchasing process. No one wants to be sitting on inventory because they made the wrong buy or bought at the wrong moment. Timing this out can be challenging, too, as most retailers depend on global supply chains to bring purchased inventory from overseas to target markets. By monitoring social media trends, for example, a fashion retailer can send real-time data to a buyer in Bangladesh informing them of what styles are trending and how strongly. That can be distilled to data that enables a buyer on the other side of the planet to determine everything from purchasing volume to shipping method.

Becoming a Data Driven Operation

It’s not enough to want your company become a data driven organization. You need to lay out a plan that gets you there. This includes:

  • Fostering a culture that values data
  • Putting standards in place
  • Hiring professionals with big data skills
  • Educating stakeholders about the advantage of driving decisions with data
  • Building out the necessary infrastructure, particularly computer servers
  • Adjusting hiring practices to incorporate big data skills
  • Opening up the discussion to all parties from top to bottom

The move to a data-centric worldview also means getting tough about things. Companies often end up using severance packages to ship out folks who refuse to get on board with the changes. This requires a hard look at why certain people are employed and whether they can adjust to the new reality.

Ultimately, a data driven approach is about competitiveness. Other companies are already doing it and succeeding. The sooner your operation becomes one that values data, the sooner it can attract the right candidates for jobs and become more competitive.

Categories
Business Intelligence Data Analytics

What is the Difference Between Business Intelligence, Data Warehousing, and Data Analytics?

When listening to discussions of many of the core concepts of the big data world, it often can feel like being caught in a hurricane of technobabble and buzzwords. Three of the most relevant concepts to understand, though, are data warehousing, data analysis, and business intelligence (BI).

Individually, each of these concepts engenders one-third of an overall process. When that process comes together, a company can more efficiently collect data, analyze it and turn it into actionable information for decision-makers at all levels of an operation.

The What

Data warehousing is the most straightforward of the three concepts to understand. As the term suggests, it’s the process of taking collected data in a company and storing it in places where it can be kept secure and accessible. This means having access to either on-site database servers or off-site cloud storage platforms.

Data analysis is the process of scanning through the available data an organization has in order to produce insights. Many people misuse this concept interchangeably with BI. The distinction is that data analysis tools help professionals handle the tasks of:

  • Acquiring data from sources
  • Prepping data for analysis
  • Confirming data integrity
  • Identifying statistically grounded methods for gaining insights
  • Using computing resources to rapidly cull massive amounts of data
  • Iterating through permutations of statistical models to generate insights
  • Verifying that any generated insights are statistically valid

Business intelligence is about taking the raw insights gained using those data analysis tools and turning them into actionable information. BI platforms are designed to provide visualizations and data to stakeholders. For example, a U.S. retailer might offer its buyers in China real-time data streams of insights derived from scanning millions of influencers’ feeds on Twitter, Instagram, Facebook and other social media platforms. This allows the buyers to look at the insights and quickly make decisions about what’s likely to sell well in the upcoming fashion season.

The How

All of this work calls for the support of folks who have experience in working with computing resources at large scales. There’s a lot more going on here than simply putting entries into a spreadsheet. The industry employs plenty of data scientists, computer programmers and IT professionals. Likewise, individuals with business backgrounds in consulting are often in high demand.

From end to end, a company has to build its training and hiring practices around fostering a culture that values big data and insights. Building such a culture often presents its own set of challenges, as many people prefer to make choices based on tastes, gut reactions and “eye tests.”

If you want an insight into how this process unfolds, look no further than the world of professional baseball. Few sports are now as driven by analytics as baseball. Starting at the turn of the century, small clubs that were strapped for cash began hunting for market inefficiencies. Two decades later, everyone in the business is using data analytics tools to make decisions. In 2019, the Houston Astros announced they were cutting their scouting department significantly while adding more people in analytics.

The Why

One of the classic examples of how statistically driven insights can defy expectations is the so-called Monty Hall problem. The original version of the show “Let’s Make a Deal” featured a game where a contestant had to choose one of three doors to win a prize like a new car. Behind one door was something no one wanted, such as a goat. Another door hid the car, and a third one hid a lesser prize.

After the contestant picked a door, the host would reveal what was behind one of the other doors. For the sake of dramatic tension, the host never showed the goat or the car in the first reveal. The host then would ask, “Do you want to change your pick?”

According to volumes of computer simulations and PhD-level stats papers, the answer should always be “yes.” By switching, the contestant improves their chance of winning from 1/3 to 2/3.

If that feels wrong to you, don’t feel bad. The answer is not intuitive. Most people assume the contestant has somewhere between a 1/3 and 1/2 chance when switching. Thousands of respected mathematicians even tried to refute the solution.

Lots of business decisions are basically the Monty Hall problem scaled into the thousands, millions or even billions. There are plenty of doors to pick from, and the goats far outnumber the cars. Also, you’re competing against numerous other contestants simultaneously.

Unless you need to pay a dowry, you probably don’t want that many goats. How do you improve your chances of finding the winning prize? You embrace the value of data warehousing, data analysis and business intelligence.

Categories
Data Analytics

Retail Analytics: Boost Your Business in One Day

Few industries are as well-positioned as retail to use data-driven systems to improve their bottom lines. Retail analysis is flush with sources of data, including information that can be derived from sales, inventory, traffic and marketing. Turning all this information into something useful, however, requires an understanding of where retail data systems fit into the bigger picture. Let’s take a look at the trend and how data analytics software may be used to boost your business.

How is it Affecting the Retail Industry?

As of 2019, omnichannel marketing and sales have become key features of how many retailers and customers interact. Even the simplest forms of this approach have changed what items are put into inventory, which customers are met with what appeals, how prices are chosen and how stores themselves are designed.

For example, let’s look at loss prevention systems that are used by many brick-and-mortar retailers. Using retail analysis methods, we can quickly spot which departments suffer the greatest losses. Items that are commonly stolen can be moved to spots where sales associates can see them. Closed areas of stores that allow for bad behavior can be opened up to observation. Patterns that might not be obvious to the average person can be discovered by comparing data across multiple stores.

Why Should Retailers Invest in Data Analysis?

Supply chains are being tightened up like never before. In the world of clothing sales, for example, you want to keep inventory purchases as close to trend spotting as possible. Retail data systems can dig deep into information gleaned from social media to empower buyers on the other side of the planet to make decisions about items to put in stores and on websites. The timing of trend data pulled from customer analysis will increase the chances that a trend will arrive in stores right before it’s ready to take off with the general public.

Personalization also offers many opportunities. Insights can be derived from mobile apps, website purchases, in-store sensors and post-of-sale units. Marketing appeals can then be tailored to the specific tastes and desires of the customer, such as offering coupon codes via text when the mobile app notices they’re within a certain driving range of a physical store.

All of this is data intensive. Customer analytics calls for a backend of systems that can store data securely and make it readily available to decision-makers in a timely manner.

Analyzing Customer Behavior

Good data science people approach customer analysis with a highly experimental attitude. Let’s say you want to determine the optimal layout for your store’s website. A/B testing methods can be utilized to discover how to maximize ROI. You simply serve multiple version of the website, and then you can use data analytics software to compare which versions kept folks on the site longest, drove sales and encouraged return engagement. Customer analytics can even be utilized to establish whether some customers should be pursued more aggressively with offers, sales and other incentives.

Using Predictive Analytics

Figuring out where to put money before the next sales season hits will be one of the biggest goals of many retail analysis efforts in 2019. In-store Wi-Fi offered for free can include opt-ins that allow data gathering and mining to be performed. These can then be used to determine which customers should be encouraged with loyalty programs, points offers and more. Metadata can even be employed to establish what the relationships are among different customers, allowing you to see how friends circles and families influence members.

Ultimately, you want to get to the point that predictive systems provide prescriptions. In addition to getting ahead of trends, decisions can be made about how many items to put on shelves, what times of day customer support is most needed and where to place salespeople in stores.

Processes will be increasingly tailored around the customer experience and ROI. Assortment analytics can be used to make recommendations regarding products that are frequently purchased together. This can be used, for example, to issue coupons at checkout that will encourage customers to come back soon. Similarly, website and app versions of stores can point customers toward product recommendations they’ll actually want.

Deriving these sorts of insights is not a light undertaking. Data needs to be accumulated in sufficient quantities to ensure that predictions actually track closely with results. A data-driven attitude has to be fostered throughout a business, and an eye always has to be kept on quality control. In time, though, a company can form a robust base to work from and to deliver value to both customers and internal stakeholders.

Categories
Data Analytics Data Quality Data Science Careers

I’m Outta Here: The Top Frustrations of a BI Engineer

The statements below first appeared in the r/BusinessIntelligence subreddit.

I have been working as a BI Developer/Consultant for the past 5 years after graduating from University. Many people are thinking about a career in this field. I thought I would offer my perspective of the problems I have faced and what led to my decision to move away from BI. Would love to hear any opinions/advice from others.

The first point I want to raise is that things have changed A LOT in BI/Data jobs over the past 5 years and not for the better. The job does not carry the same level of respect or ‘perceived’ value in an organization. Before you all murder me, let me explain. Data has more value than ever, I agree. However, the people who extract, clean, combine and deliver this data have much lower value. I am not sure why this has developed.


Advantages of BI/Data Careers

Job title of BI sounds fancy to most people. Salary ramp-up to mid level ($80k) on par or better than other IT/Business fields. (BI does cap out much earlier than other fields).

Easy to get into a low workload job as a Excel/PowerBI/Tableau data cruncher with a mid-level salary. Progress after that is very hard unless you make shifts to other areas.

Disadvantages of BI/Data Careers

Work that nobody wants to do gets dumped into the BI department. Its role is less well defined and it’s easy to sneak the mistakes of others into “the data department.” There’s no systematic way of managing the quality of what arrives in. Once we’ve taken custody of it and a few days have passed, it’s our problem. As if somehow 7,000 emails got turned into NULL in the 2 days since you sent me your file.

I once worked with a client that ran a yearly survey to gather data. They produced a report of top 100 companies and industry trends. Nobody in the client’s company wanted to sift through over 10,000 survey responses. Nobody wanted to clean data, extract insights from survey responses. So they just sent it.

This entire workload fell to us. the external consulting company, even with our $150-per-hour bill rate. It took us weeks of work and the company paid out quite a bit. Of course, remember I did not see $150-per-hour for this work, I just received my salary, which was in the $60k range. So who benefited and who overpaid?

Another example, this time from a large enterprise. Daily data loads extract data from [HR, finance, payroll, etc.] systems. New employees are sometimes set up with different/wrong values in different systems. This causes major issues in reporting/BI tools. Senior Management was quick to blame BI. They didn’t consider the inefficient processes, or mistakes at the operational level that led to this. The HR/Finance analysts don’t care about these issues. It got so bad, eventually setting up new employees in the HR system fell to BI analysts. They main reason was that they cared the most about the data.

The end users look at the data once a month if at all. The weekly emailed static reports often go unread. Instead the end users revert back to the prior solution where data is sourced by BI analysts manually. Guess what the reason was? End users find it boring to have to use cubes to browse data or PowerBI/Tableau to manipulate data. They prefer to file a request with the BI team and let them do that work, or have analysts send them a weekly email. Or simply sit in a meeting where someone else tells them what’s going on.

Salary cap to what BI developers can earn. I find that as a BI developer, my salary peaks at around 80% of what other types of developers earn at upper levels. Market rate for me is 90-100K (USD) in house and 100-120K (USD) consulting.

This is made worse by the number of senior SQL server/DBA/BI consultants (+20 experience) in the market. You don’t need more than 3/4 years experience with a BI toolset to get the job done properly. Yet I have been on many projects where clients have asked for someone with 12+ years experience. They’re later surprised when they learn someone with 4 years experience did the projects.

Job tied to a tool/industry. I was never sure why this matters so much. The ability to learn a new tool to get the job done is under-appreciated. I have worked in finance/retail/media and government BI. But I have been told I am not skilled enough to work in x industry or with y tool that varies slightly. Add to this jobs where I see people with masters or PhD level education doing BI Analyst work. People are on-average under-utilized, in my opinion.

BI testing. The most boring, manual, but most necessary part of any BI project.

Testing SQL business logic is painful because of the lack of automated testing solutions used across companies .

Testing with popular tools (PowerBI, Tableau) is nearly always manual . (Good luck testing complex finance dashboards with complex DAX business logic.)

Source system testing is non-existent. (What happens if you change the time zone in a source finance application. Does all the data for the user we extract change at a DB level as well?)

ETL testing (good luck testing 100+ SSIS packages).

Data Warehouse testing: all too often, complex business logic is piled on top of existing logic due to source system upgrades. cube/dashboard testing. No automated solutions exist. Mainly manual.

It’s rare to find business users who will agree to do testing properly. I have seen business users resign from jobs rather than sit and test large amounts of data manually.

While a career in BI is still very attractive to knowledge workers, I wanted to share the pitfalls. I hope my experience helps others. The space still has some maturing to do. If you get with the right organization, it can still be a great career. If they let you use the right data analysis tools, it can still be a win. The key is being able to quickly understand the environment and make quick decisions.

As an employee, you should be watchful for this, but you do have some choices . As a consultant – as I was/am – you’ll often get dragged into some of the worst environments to help fix things.

Expect that.

One can easily find themselves stuck cleaning data in Google Sheets for most of each day. It’s important to recognize the signs and signals of a good BI vs. a bad BI environment. My advice: look for places where business users are actively involved in BI projects. Companies that invest in their data, and in advanced AI tools. Places where they actually care about the outcome and respect the work you do. Because it’s important. You’re important.

Good luck out there.

The statements above first appeared in the r/BusinessIntelligence subreddit.

 

About Us
Free Tools
Contact Us

PO Box 90762
Lakeland, FL 33804-0762
813-499-9814

Watch Demo

© 2023 Inzata. All Rights Reserved.

Polk County Schools Case Study in Data Analytics

We’ll send it to your inbox immediately!

Polk County Case Study for Data Analytics Inzata Platform in School Districts

Get Your Guide

We’ll send it to your inbox immediately!

Guide to Cleaning Data with Excel & Google Sheets Book Cover by Inzata COO Christopher Rafter