Enterprise Data’s Dirty Little Secret

Tim Neill ON Jul 29, 2015

Business rarely execute on the entire data cycle in their attempts to get value from that data and make data-driven decisions. Our data model has four fundamental steps:

  1. Data Collection
  2. Data Storage
  3. Data Preparation & Optimization
  4. Reporting & Visualization

Failing to properly execute on any of those steps nearly always results in failed or invaluable analysis.

Let’s take the example of an F&B customer. They got reports from Micros POS systems that provided skeletal information about the consumption patterns of their customers on property. They also got Guest Metric analytics data that further broke down purchasing behavior and connected it with other customer data points.

They could not, however, provide meaningful insight into the purchase patterns or real value of customers and could not extract it to the level where they learned from the data – learned how to be smarter buyers or learned how to teach their staff what to sell.

The Problem

  • Corporations are collecting more data than ever before. They still do not know how to use it to improve how they operate as a business.
  • Corporations have invested in data stores and data reporting & visualization platforms
    • Data stores = Hadoop, Data Warehouses (EDWs?), Data Lakes, Databases of all stripes and colors, POS reports, Analytics
    • Reporting & Visualization Platforms = Tableau, MicroSTrategy, Cognos, Qlik, Adobe, etc.
  • But they do not derive answers or the real value from either
  • There is a gap. If you think about data coming out of data collection systems (like POS or ecommerce) and sitting on corporate IT assets and the reporting and visualization tools sitting in the hands of the business. There is a huge missing step.
  • IT wants to control access to data
  • Many times the business ‘owner’ cannot get to clean, joined, prepared data.
  • Almost all data being collected is unstructured or semi structured and has not been normalize – the result is that it is not ready for consumption.
    • E.g. financial data represented as a percentage and another is a real number or customer data that is tagged as first name, last name and in another source Last Name, First Name. All of it needs to be cleaned and normalized.
  • If you put crap data into most reporting and visualization platforms, you get crap out the other end.


  1. Corporations are collecting more data than ever before but doing a marginal job putting it to work for their business
  2. Breaking down organizational silos (aka making data accessible) and cleaning and preparing data is the first step
  3. Intelligently designed and orchestrated reporting/visualization provides access to critical business data empowering better decision making

Becoming “Data Intelligent”

  • Understand the business questions
  • Map your data ecosystem – understand what data you have and where it lives
  • Prepare your data for reporting
  • Connect data sources to the right questions and the right tools
  • Accelerate your ETL (Extract, Transform, Load) processes
  • Move discreet data/data sets to the cloud for reporting and visualization