Retailer Data: Plenty Available, but What Do You Actually Need?

Since 1994, Retail Velocity has been helping consumer goods manufacturers ingest, cleanse, and harmonize data from hundreds of retailers. In the last five years, we have seen an explosion of available retail data, and thankfully, our data experts understand the major difference between the “explosion of data” and having the “best data to drive high ROI.” Unfortunately, many have the “Just grab all the available data and send it to us.” mentality, believing more is better. However, our decades of industry experience and expertise tells us this is an incorrect belief. The only data that should be ingested (and cleansed and harmonized) is the data needed to resolve issues and optimize decision-making. Read further for deeper examples and insights.

Retail Velocity’s modern data platform and techniques are unparalleled in the industry as they relate to ingesting, cleansing, and harmonizing large volumes of point of sale (POS) and inventory data in an “automated factory” model that also provides direct integration into data cubes, data warehouses, data lakes, and artificial intelligence (AI) tools. For those who may not know, Retail Velocity has been directly involved with AI tools for nearly 10 years, and our VELOCITY^® engine has had AI integration areas built in since 2014.

RETAIL DATA: WHAT’S THE BEST AVAILABLE?

Retail data is available in many forms. Basic data is typically available in flat files and EDI formats. This type of data is usually inventory and POS data which can be extremely limiting when trying to generate deeper, more valuable insights. Therefore, anyone wanting advanced insights should be ingesting data at a vendor portal or API level. We view the data in the basic-to-advanced complexity levels, as well as robustness of data to ingest, as follows:

5.) Flat Files or comma-delimited (CSV)
4.) EDI
3.) Microsoft Excel
2.) Portal
1.) API

Regardless of complexity level, all data collected should be cleansed and harmonized for proper and maximum usage, especially for reporting, data analytics, and actionable insights. Also, the cleansing and harmonizing process is infinitely more important if the retailer data is going to be fed into ERP, trade promotion, demand forecasting, or AI tools.

EVOLVING to VENDOR PORTALS AND API—THE ROAD TO SUCCESS

As of spring 2024, the vast majority of the top 60 U.S.-headquartered retailers offer portal- or API-level daily data at the SKU level per store. Walmart Luminate, Amazon, and Target fall into this category. Walmart Luminate is only for the U.S. market as of now, so many CPG manufacturers require both Luminate and Walmart Retail Link data to fully view their inventory and POS data activity.

"Expect retailers over the next five years to continue evolving from EDI to portals or from EDI to APIs, including charging manufacturers for this data."

VELOCITY^® is Retail Velocity’s main data ingestion engine and we offer 625+ retail data adaptors, of which at least 60 are portals, 10 are APIs, 35 are flat files, and 500+ are EDI. These numbers include U.S. and international markets, including physical stores and e-commerce. VELOCITY is also tuned to ingest syndicated data, census data, ERP data, trade promotion data, demand forecasting data, and weather data.  

Currently, we are witnessing the largest U.S. retailers evolving from EDI and AS2 to advanced portals or APIs. Non-coincidentally, CPG manufacturers that want daily POS and inventory data for more precise decision-making are getting the data from retailer portals or APIs. An important caveat and Retail Velocity opinion: Expect retailers over the next five years to continue evolving from EDI to portals or from EDI to APIs, including charging manufacturers for this data. The good news is that some organizations are finding out that this portal/API data can help them transform data into a profit center.

THE NEED FOR DATA AUTOMATION

There is a plethora of ways to receive flat files or EDI data, including lower-cost offshore resources or in-house IT staff, and very few of these have the proper expertise to build automated ingestion of portal or API data. The most difficult part is not ingesting the data; the most difficult part is automatically ingesting the data, automatically cleaning the data, automatically harmonizing the data, and automatically exporting the data into a data warehouse, data lake, or data cube.

An important item to note— your ingested B2B data should be in secured, U.S.-based data centers, data warehouses, or data lakes. This has proven to be the best method for data protection, including the legal systems. For this reason, all of Retail Velocity’s client data is in U.S.-based data centers.

Although flat files and EDI-formatted data may be easiest to receive, ingestion of this data is less challenging than its cleansing and harmonizing processes. Excel, portal, and API data have many nuances for ingestion only, and the cleansing process is critical prior to data harmonization. Once cleansed and harmonized, the data is properly ready for use.

A word of caution, if you want to receive advanced promotional data or advanced demand forecasting data, the ingesting/cleansing/harmonizing process should be performed by data professionals with the proper expertise. Fortunately, Retail Velocity has this nuanced expertise, plus the expertise to make this comprehensive data available in your preferred system(s) or applications.

We highly recommend, whenever possible, using portal or API data, which contains fields and measures not available in other data sets. This data will be much more meaningful for most CPG manufacturers.  You may contact us for information and expertise related to available portals, especially for North American needs. As previously mentioned, Walmart Luminate Charter, Amazon, Target Kiteworks, and Kroger, plus approximately 60 other U.S.-based CPG manufacturers have portals or APIs. Retail Velocity has built Amazon APIs for a total of 20 foreign countries.

The complexity associated with successfully building portals or APIs is relative to a few factors including 1.) which fields you need, 2.) do any of the fields need algorithmic “help,” 3.) does the portal require multifactor authentication (MFA), and 4.) will the retailer allow full automation eliminating human intervention. Note, this stated complexity is BEFORE the necessary cleansing and harmonizing begins.

One warning—due to the nature of retailer portals, using data virtualization models may not properly work. ETL models including reposting capabilities will prove more efficient.

PROVEN HIGH ROI BUSINESS CASES

The benefits of having robust data fine-tuned to maximize your business’ ROI are plentiful across the entire enterprise. A few proven business cases with high ROI involvement:

• Revenue Growth Management areas including Promotion Optimization, New Product Introduction, Promotion Analysis, and Pricing Analytics
• Supply Chain including Phantom Inventory, Out-of-Stock (OOS) Reduction, On-Shelf Availability (OSA), Demand Forecasting, Distribution Analysis, and Inventory Optimization
• Retail Execution including Field Service Management and Optimization, Store Merchandiser Execution, and Broker Management
• Forecasting Excellence related to Demand, Trade Promotion, Store-Level Analysis, and since 100% of SKUs would be available—better Sales Baselines
• Margin Measurement including Markdowns, Returns, and Transfers

One large recommendation: Receive 100% of your own SKUs whenever possible and combine them with syndicated data category information for extremely strong store and brand analysis capabilities.  

IT’S TIME TO FOCUS

Retail Velocity and our data engineers, some of whom are experts in “all things data,” continuously help our clients through complex situations in a consultative manner. These engineers are more than technologists or technicians. They can advise on improved processes, better uses for the available data, and better data analytics models. This advisory method enables your data scientists and business analysts to develop highly reliable insights.

A large and highly regarded consulting firm uses a 10/20/70 model which nicely coincides with Retail Velocity’s consultative approach. Paraphrased, the 10/20/70 model is 10% effort in building the model; 20% effort involving high-quality data, technology implementation, and innovations; and 70% effort focused on business transformation via insights. Retail Velocity has always separated data, analytics, and insights with our focus on data and analytics. This allows the business experts, with their deep knowledge about their organization’s tactics and strategy, to not spend any time or effort on collecting data nor providing the needed algorithms for any analytics. Our standard approach we have employed for over 30 years allows for our clients to 100% focus on developing insights and improving business processes.

Retail Velocity has one important “Call to Action” for you: Are you able to focus all your resources on insights and business improvements? If not, our VELOCITY^® solutions will enable this capability. Get in touch with us today.