Migrating Analytics from SAS to Hadoop
Retail organization needed to migrate a system that generates a propensity score that indicates how likely are customers to buy protection agreements on the home appliances that they purchased from SAS to Hadoop in order to cut costs and be able to scale to the growing size of the datasets.
01 Problem
Propensity score is used by call centers and technicians who are in customers' homes for repair or installation to get an idea about pitching protection agreements. SAS was used to calculate this score using customer data, orders, agreements, last 50 years of data. High SAS license costs on DB2 database. SAS was slow because based on real-time transactional data; SAS provided information after the fact. SAS programs going over terabytes of data and taking too long to resolve mathematical model.
02 Solutions
Convert business logic that was written in SAS into Pig. Source data related to customers, agreements and orders and transform it to fit analytics needs by going through 50 years of history to give score. Take the mathematical model written for SAS by data scientists and re-write in Pig. Pig handles all ETL logic in concise steps using MapReduce. Sourced all data, run Pig jobs, send scores to target interaction team. Use Mahout to identify. Distribute processing on 500 node Hadoop cluster so that the entire analysis happens in a few minutes. Targeted analysis occurs over night so ready for technicians the following day. SAS goes customer by customer. Hadoop goes by entire data set. This causes a change of mindset of how the data can be used.
03 Results
Ability to provide technicians with propensity score information while they are in customers’ homes. Change in mindset of how the data can be used compelled business to identify high cost SAS programs and convert to Pig. Once the data is in Hadoop, it can be reused for other analysis.
04 Tools Used
Hadoop
MapReduce
Pig
Mahout