The Most Important Retail Data Considerations for CPG IT Departments

During my 30-plus-year career in the consumer packaged goods (CPG) and retail industries, I have seen many different retail data collection methods that worked adequately for CPG companies. However, for those that failed, there was one major, common mistake and several “medium” mistakes. All were fixable and preventable, but many companies wanted to take shortcuts.

The major common mistake: IT departments allowed retail point-of-sale (POS) and inventory data to be analyzed and used by sales, marketing, supply chain, and finance teams that was not cleansed nor harmonized. This has been a crucial mistake for both B2B and B2C users.

Cleansing data prior to generating retail data reporting and analytics, especially for strategic, enterprise decision-making and collaboration between CPGs and retailers, is a critical step that should NEVER be skipped. Retail data coming from a retailer’s POS or technology system is seldom cleansed prior to being sent to or collected by CPG manufacturers. This “dirty” data, which even comes from large retailers, may be suitable for basic retail and CPG data analytics, but it should never be used for critical decision-making that can drive sales, revenue, and profitability.  

I have seen large CPG companies use uncleansed supply and demand data for their mission-critical systems and applications, and then, after-the-fact, realize that this less-than-stellar-quality data is practically useless. Having the ability to cleanse important data is vital for everyday decision-making, and I dare say, critical for reliable data inputs for Artificial Intelligence (AI) tools. Imagine your AI tool “learning” from lower-quality data—some organizations may never recover from that mistake.

WHAT RETAIL DATA IS MOST IMPORTANT TO A CPG?

At a minimum, ideal retail data sets collected should include an SKU or UPC number, selling price, sales units, on-hand inventory, and on-order inventory. The best-of-the-best companies also have at their immediate availability SKU and UPC numbers, online sales activity, item description, item retail price, item wholesale price, and location identifier. These more progressive companies will also capture key trade promotion-related data.

Of course, when it comes to data frequency, using daily POS and inventory data is tremendously better than using weekly data. Only a few still use monthly data, and even fewer use hourly data, but hourly data is closest to real-time data when available.

CLEANSING AND HARMONIZING DATA SHOULD BE A BEST PRACTICE

Retail Velocity has previously written about the importance of CPGs using harmonized retail data; however, using cleansed AND harmonized daily data at an item and store level will ensure greater success in today’s omnichannel retail world. Why? Because using cleansed and harmonized daily retail data is the only proven method for creating valid comparisons for products, retailers, and regions.

"Successful data harmonization is key to your future decision-making and creating unified, reliable data that can easily be shared and fully leveraged."

Basing insights on low-quality data creates serious risks for CPGs and retailers. When data is uncleansed and unharmonious it can lead to the greatest risk which is extremely poor business decision-making that can be costly—costly to a brand’s sales, bottom line, brand reputation, and retailer relationships.

Basic data harmonization involves manually matching “like-for-like.”  Expert-level retail data harmonization requires sophisticated mapping and the comprehensive alignment of fiscal calendars (manufacturer and retailers). Over the last 30 years, at Retail Velocity, we have seen data companies that have been around five, 10, and even 15 years that have not mastered the nuances of data collection and harmonization. Successful data harmonization is key to your future decision-making and creating unified, reliable data that can easily be shared and fully leveraged.  

Data latency is terrible for modern decision-making. When your retail data is not harmonized (i.e., it is erroneous or incomplete), you must relook at your data sources individually. These time-consuming tasks are counterproductive and delay successful decision-making and execution across your enterprise. Since data is dynamic, these delays likely cause latency and the need to reload previous data or ingest new data, thereby requiring IT departments to rerun the data cleansing and harmonization process.

DATA AUTOMATION: THE DIFFERENCE-MAKER

Automating the retail data collection and harmonization processes—for U.S. and international markets and for both physical stores and e-commerce—for IT personnel is the main difference between the novice and experienced solution providers. Centralized data models allow for efficient mapping and are proven to be more successful with Machine Learning (ML) methods, especially for automated data mapping. Retail Velocity’s retail data platform, VELOCITY^®, uses data models synchronously for ML and AI tools.

Many CPG companies with ERP systems focus on Master Data Management; however, data harmonization using VELOCITY takes this process one step further. VELOCITY cleanses and removes inaccuracies while significantly reducing inconsistencies from our client’s retail data sources. VELOCITY, properly configured by our engineers, creates harmony between these various data sources, thereby, showing the complete data picture as One Version of the Truth and allowing for comprehensive data to be immediately ready for data analysts and data scientists. This speed-to-insights approach has been Retail Velocity’s calling card for nearly 30 years.

BUY OR BUILD? AN EASY DECISION WHEN CONSIDERING LOWER COSTS AND SUPERIOR RESULTS

Various CPG companies have attempted to build in-house data collection and harmonization solutions or capabilities, or they have used novice-level competitors’ solutions with limited success. As an industry-leader with global enterprise accounts, our expertise and focus are on collecting, cleaning, and harmonizing data. Eliminating delays that cost money and reduce speed-to-market, as well as having expert-level engineers and consultants for your engagements, are logical reasons to choose VELOCITY (which has proven quantifiable results) during “build vs. buy” decisions.

Retail Velocity engineers are externally recognized experts in developing robust retail data adaptors, for both retailers and distributors. With over 625 available retail data adaptors that handle a variety of data formats, there are very little data collection, extraction, and integration challenges that Retail Velocity has not seen and successfully conquered. Many of our data adaptors are fully automated, requiring minimal manual intervention after initial setup.  

"Cleansing and harmonizing large volumes of disparate data requires expertise cultivated over years of real-life experience."

VELOCITY retail data adaptors are also equipped to manage data from a multitude of data sources, including flat files, EDI, Excel, portals and APIs. What sets Retail Velocity apart is not the file formats our VELOCITY solution can handle but the frequency of retail data it can handle, the granularity of the data, the automation of collecting/cleansing/harmonizing data, the ability to handle U.S. and international data, and the availability of yesterday’s data by 10:00 a.m. the next morning. Accomplishing all these areas simultaneously requires deep industry experience and expertise.  

Creating data adaptors for flat files and EDI data feeds is a novice-level capability that can easily be outsourced throughout the world. Data portals require medium levels of experience, yet many companies have failed or not even tried due to non-automated processes requiring significant manual intervention, high time-to-market costs, or lack of expertise. These failed attempts are exacerbated by an ever-changing retailer data landscape.

Cleansing and harmonizing large volumes of disparate data requires expertise cultivated over years of real-life experience, and building advanced portals and functionally efficient APIs is not for the fainthearted nor inexperienced IT personnel. Very few in-house IT departments can successfully build and automate portals and adaptors for weekly data, let alone daily data. Building these automated portals for daily data management and usage is only done by a select few.

CONCLUSION

As previously mentioned, retail data consumption, cleansing, and harmonization is tough, and when you want to also integrate vast amounts of enterprise data it intensifies the expertise needed—expertise that reduces numerous risks and costs. And when you consider the growing robustness of vendor portals and the amount of data being made available every day, the need for that expertise grows significantly.  

For the most impactful decision-making, the best consumer goods manufacturers need daily POS and inventory data—at the SKU level for every store location—for all SKUs. This is where syndicated data providers fail, especially for collecting and providing data (which is at best 40%-60% accurate) for all a manufacturer’s SKUs. Other solution providers fail on multiple levels including daily store-level data. In-house organizational staff also fail on multiple levels, including automated portals.

The costs of trying to clean and harmonize data in-house for a CPG will be high, even if using lower-cost offshore resources; therefore, time-to-market from proven, off-the-shelf configurable solutions have the highest ROI, while solutions utilizing proven industry experts have even higher ROI.

To learn more about how Retail Velocity can help CPG IT departments efficiently, effectively, and automatically collect, cleanse, and harmonize ALL their retail data to drive actionable insights and strategic decision-making, contact us today.