Business Value in an Ocean of Data: Data Mining from a User Perspective

Bulcsú Fajszi – László Cser – Tamás Fehér

BUSINESS VALUE

in an OCEAN of DATA

Data Mining from a User Perspective

Authors of the book:

© Szabolcs Biró, András Bujna, László Cser,

Bulcsú Fajszi, Tamás Fehér, Miklós Hosszu,

István Járási, Ágnes Masanja, András Máté,

Krisztina Móra, Márton Szél, Márton Zimmer

Many of the authors who contributed to this book are experts of the Hiflylabs team.

Hiflylabs - We create business value in a world full of data.

www.hiflylabs.com

Editor: Prof. dr. László Cser

Language editor: Simon Milton

Cartoons: © András Bujna

Book cover:© Balázs Csöllei, 2013

© Alinea Press, 2013

ISBN 978-615-5303-05-0

All rights reserved.

Table of Contents

– Oh, good that you’re here Pam. You probably remember

on which page is Landing in fog.

Foreword

1.   Data Mining in General

1.1.   Why Data Mining?

1.2.   Technology for Uncovering Hidden Information

1.3.   Pitfalls and Difficulties

1.4.   Data Mining and Business

1.5.   Data Mining and Statistics

1.6.   Data Mining and Conventional Data Analytics

1.7.   Data Mining and Manual Analysis

1.8.   Business Value in an Ocean of Data

1.9.   The Structure of This Book

1.10.  The Authors of This Book

2.   Business Intelligence and Data Warehouses

2.1.   Business Intelligence (BI)

2.2.   Data Warehouses. From Source Systems to Data Mining

– Data Warehouses and Data Marts

2.3.   Metadata

2.4.   Exploitation

2.5.   OLAP

2.6.   Other Exploitation Technologies

3.   Lessons from Data Warehousing Projects

3.1.   Establishing a Corporate Data Warehouse

3.2.   The Conditions Necessary for a Successful Implementation

4.   The Methodology of Data Mining Projects

4.1.   Data Mining Projects

4.2.   Business-Oriented View

4.3.   Methodologies

4.4.   Success Factors for Data Mining

5.   A Typical Data Mining Problem: Predictive Modelling

5.1.   Predicting the Future

5.2.   The Predictive Modelling Process

5.3.   The Purpose of Predictive Modelling

5.4.   Scorecard

5.5.   Making Predictions Based on the Past

5.6.   Setting up an Analytical Environment, Collecting Data and Definitions

5.7.   Examining and Understanding the Data

5.8.   Data Modification, Preparing Data for Modelling, Modelling

5.9.   Building and Applying the Model

5.10  Applying the Final Model

6.   Segmentation

6.1.   The Purpose of Segmentation

6.2.   Customer Segmentation

6.3.   Elements of Customer Segmentation

6.4.   Preparation of Data

6.5.   Segmentation Base Table

6.6.   Segmentation and Profiling

6.7.   Segment Creation Methods

6.8.   Operating the Segmentation

7.   Behaviour Prediction (Early Warning)

7.1.   Another Task for Predictive Data Mining

7.2.   Exploring Behaviour

7.3.   Time Series: The Foundation of Behaviour Modelling

7.4.   Behaviour Modelling: The Practice

8.   Campaign Optimization

8.1.   Why Should Marketing Campaigns Be Optimized?

8.2.   The Business Goal of Campaigns

8.3.   The Main Question for Campaign Optimization

8.4.   Planning Campaigns

8.5.   Campaign Interactions

8.6.   The Utility Score

8.7.   Excessive Communication and Exclusion

8.8.   Constraints

8.9.   Allocation of Offers

8.10.  The Optimal Allocation

8.11.  Summary

9.   The Implications of Data Quality

9.1.   The Business Implications of Data Quality

9.2.   Data Quality Issues

9.3.   Improving Data Quality

9.4.   Methodologies and Algorithms for Data Cleaning

9.5.   Back-Testing

9.6.   An Example of Cleaning of Master Data: A Register of Prisoners of World War II

10.  Networks for Business Purposes

10.1.  Network Perspective

10.2.  About Networks

10.3.  Business Use of Networks

10.4.  Defining Networks

10.5.  Preparing Network Data

10.6.  Visualisation

10.7.  Analysis

10.8.  Network Simulation

10.9.  Storing Network Data

11.  Outlook

11.1.  Industrial and Commercial Applications

11.2.  Agricultural Applications

11.3.  Service and Trade

11.4.  Other Areas of Interest

Notes

Bibliography

Index

Introductions

Foreword

"Data mining has become a standard tool helping

both the business and scientific worlds.

It undergoes a continuous rapid development,

however, while overviewing it we emphasize our

own experience."

The Internet has completely changed most business functions like sales, distribution, customer service, marketing and even production. This change has resulted in two major consequences. Firstly, the barriers to enter new markets have become very low, and getting financed by Capital Ventures or Angel Investors is relatively easy. Secondly, competition takes place on a global level with hundreds or thousands of competitors.

The question is — How can you survive in this environment? How can you differentiate yourself?

I firmly believe the only way to gain a long-term sustainable competitive advantage is by having business intelligence. Other competitive factors are easy to copy or are not sustainable. So the real question faced by decision makers is — How can the company gain and take advantage of business intelligence?

This cannot be answered in just one sentence. That is why I suggest you read through this book. The authors have created an excellent body of work as they have managed to translate very math-intensive science into an easy-to-understand solution-oriented book. Several real-life examples will make it clear how you can improve your company’s profitability, identify hidden patterns or attain precise predictions of customer behaviour. By the end of the book you will be able to identify those key areas of your operations where you can apply data mining. Do not forget! It is not a method, it is a way of thinking! The data in your hands can be transformed into valuable information and your decisions will rely on analytics your competitors do not posses.

Do not hesitate! Start reading and earning more money!

Roland Szűcs

Former CFO of T-Systems Hungary

Data mining has, for a long time, been an emerging field of computer and information science for extracting relevant information from huge masses of data. In various application areas of science and technology, the amount of data has recently been exploding. Simultaneously, understanding and analyzing these ever growing data sets, the so called Big Data, has become more and more difficult or even impossible using traditional database management and analysis tools. In industrial applications, for instance, novel machine learning based information retrieval methods are needed in extracting important information from databases and measurements.

The textbook at hand, Business Value in an Ocean of Data (Data Mining from a User Perspective), approaches this important topic from user perspective. Success in complex data mining and analysis projects requires multidisciplinary expertise which usually is obtained through team work combining, e.g., data mining methodology, application knowledge, and data understanding in novel and innovative ways. In the development of data mining methods, the user should always be in the centre.

I know personally Professor László Cser from a long-term research cooperation in an international industrial data analysis project. Professor Cser is internationally highly acknowledged as a scientist and researcher. His multidisciplinary expertise in the field of this textbook is also widely appreciated in the scientific community. His experience and knowledge of practical problems in industrial environment are clearly visible in the book. Human thinking is often visually-oriented and, thus, visualisation plays an important role in data analysis and understanding. This is nicely emphasized in the various applications and case studies discussed in the text.

I am especially happy to see that the important aspect of user perspective in data mining has properly taken into account in the text. I assume that the textbook at hand is a valuable contribution to the field of data mining and analysis. It gives valuable examples to researchers and scientists as well as to developers of practical engineering applications of data mining.

Prof. Olli Simula

Dean of the Faculty of Information and Natural Sciences

Aalto University, Helsinki (Finland)

Recommendations

Modern business organizations face the challenge of a real data boom today. They can measure their environment and internal activities with unprecedented precision and frequency. Data miners can find real business value in this exponentially growing hill of data. This is a great book about how to find the value creating opportunities and how to utilize them. Reading it is a must for managers who understand that data became an extremely important source of competitive advantage in today's business.

György Bőgel

CEU Business School (Hungary)

This book provides an excellent overview over business intelligence applications. The authors explain both advanced algorithms and technical concepts at the core of BI vividly and comprehensibly, even for non-expert readers. Real-life examples illustrate the pitfalls of BI and data mining applications and offer advice on how to avoid them. Professionals working on BI-related problems as well as students in management and information systems will find this book to be a most valuable and entertaining source of information.

Prof. Franz Lehner

University of Passau (Germany)

As a practicing top manager, I regularly face business challenges in which the analysis of huge amount of data is required. It was important for me to recognize that in our business life, data mining and data warehouse are becoming extremely important to answer questions, make decisions and solve problems. Being responsible for the operative management of a multinational telecommunications company, I had the privilege to meet the authors of this book and to elaborate and build up a robust data warehouse and a management information system that proved to be very useful for the management team. This project was a great, real-life experience with the authors who demonstrated their unparalleled knowledge and competence in designing, building and operating such systems. This book is based on this knowledge and competence of the authors, and rightfully belongs in the library of every manager to be used every day!

Csaba Bőthe

CEO, IT Services Hungary (Hungary)

Data mining is a very helpful digital technology to discover the golden nuggets of information in a large amount of data. This book highlights the potential business value in data mining, and from the perspective of users, the book mainly elaborates data mining technology in the business field in a systematic way, which is exceedingly valuable to both individuals and corporations. Besides, the authors of this book are all experts in this field and have done voluminous research in the field concerned. In conclusion, this book is definitely worth reading.

Dr. Zhen Zhao

Jiao Tong University of Shanghai (China)

Prof. Laszlo Cser has worldwide reputation in data mining, among other fields. He recognized the importance of finding the right data in exponentially increasing galaxy of data years ago and the way how to convert it into information. Prof. Cser brilliantly combines the theory with practical application. I recommend this book not only for educators and researchers, but managers and specialists in every branch of economy.

István Lepsényi

CEO, Knorr-Bremse Hungary (Hungary)

The experienced authors provide a comprehensive overview able to serve as a guide to the perplexed in a crowded world of information.

The useful, logical and professional content is presented in a friendly style, illustrating through practical examples the importance and advantages of data mining.

I strongly recommend this book to engineers, applied scientists, researchers and businessmen battling with data mining proliferation, while searching for clear interpretations and ways for practical applications.

Prof. em. Zvi Katz

University of Johannesburg (South Africa)

1. Data Mining in General

"In the era of information abundance the question these

days really is how to access the golden nuggets of

information hidden in large amount of data."

– Tell me, Little Red Riding Hood: which wolf has swallowed your grandma?

1.1. Why Data Mining?

When, more than twenty years ago, data mining surfaced as a new trend in computer science, both the academic and business world kept a wary eye on it. It had been a common tendency for computer science to sell the same old thing in a different guise and at a higher price. The term ‘data mining’ came into use in reference to gold-diggers who sifted through massive amounts of worthless sand, only to discover tiny nuggets of valuable gold (see figure 1.1). It was only later that references to the classic work of miners entered the picture; to complicate things even more, the expression ‘drilling down’ was also used in ERP systems.

The analogy of data mining with gold-digging exists even today. The academic and business world is drowning in data but lacks information and is starving for real knowledge.

Figure 1.1. Is this like data mining? (A South African representation of gold panning)

Modern day database management systems are able to cope with gathering and storing data and forecasts indicate that they are likely to be able to keep up with the increasing demands that are projected. The importance of data preservation is not really in question; it is obvious even to laymen that an ever-growing sea of data carries with it a tremendous amount of useful information. The real question is how the treasure of knowledge within this sea (or rather ocean) can be accessed.

1.2. Technology for Uncovering Hidden Information

One possible definition of data mining is now presented: Data mining is the extraction of implicit and hidden information and rules and patterns from databases (with a practical use).

Statistical knowledge is the building block for data mining algorithms. In case of data mining, however, many practical aspects predominate the classic constraints that statistics is subject to.

Data mining has rapidly developed as an autonomous scientific field over the last years of the past millennium. This trend is explained by the following facts:

 The amount of data available for analysis is growing exponentially. Over the years, many databases have reached a size and complexity that conventional analytical tools are unable to cope with; the ever-increasing need for state-of-the-art data mining techniques is undeniable.

 High-performance hardware which has been developed recently now allows elaborate theories to be put into in practice. Fast processing units and efficient management of databases shorten the processing time of the complex algorithms utilized in data mining.

 In the meantime, novel data mining techniques and processes which are designed to uncover the treasured knowledge have also been developed. Further research and development calls for even faster equipment that will allow us to tackle problems with an even higher level of complexity (even though the level already reached makes the utility of data mining high).

The aforementioned factors account for the widespread use of data mining.

1.3. Pitfalls and Difficulties

Back in the last century, the authors of this book participated in a contest which was designed to find the best team of analysts and the best mining tools. One member of the organizing team proposed that the attribute names should be masked in order to make the challenge more intriguing, and thus enable the discovery of the most talented analysts. And why not? To the layman’s eye this would not contradict the use of the three main elements that comprise data mining — hardware, software and algorithms. But let us stop here for a moment! Is there really nothing else needed for successful data mining?

1.3.1. Popular Beliefs About Data Mining

In 1997, Information Week published an article about the exaggerated expectations related to data mining and identified the following popular beliefs:

 Data mining is able to procure results that are so staggeringly useful that they can immediately change the course of business.

 Data mining can substitute for having detailed knowledge about a given field of business.

 Data mining can automatically find the patterns that we seek without the need to define the exact questions in advance.

 Data mining utilizes unusual quantitative analytical techniques which have nothing in common with conventional modelling methods.

 Data mining is so extremely complex that only scientists are able to do it.

 Analyses can only be performed on large amounts of data; all available data sources should be involved.

 Data mining will disappear, giving way to the return of conventional business approaches to problem solving.

Just before proving these assumptions wrong, we will now explain a few of the typical pitfalls with analyses.

1.3.2. Analytical Booby Traps

Ideal Types

Let us assume that our goal is to identify a target group for an investment fund campaign for a large bank. Results show that the most responsive clients are those which already have deposited large sums of money. In this scenario, by contacting only the depositors of the very same bank, the campaign has a success rate of over 20 percent. This is certainly good news for the product manager of the investment fund. And how about the bank? Is it beneficial when, supported by the results of data mining, one product devours another’s clientele, thus moving money from deposits to investment funds? When presented to senior management, the issue was of course addressed by adapting the model to identify clients who would bring additional funds in to the bank. This group is called the ’ideal type’. In this case, a success rate of only 10 percent was possible — but resulted in additional millions for the bank when compared to the previous solution.

Be aware that, in a typical data mining project, almost half of the time used is spent on selecting the proper ideal type.

Siren Voices of the Mining Tool

When an agreement about the ideal type is finally reached, there are still other sorts of traps to avoid. Once, the current authors modelled mortgage applications in order to select targets for future campaigns. Naturally, the focus of the analysis was on client behaviour a few months prior to the application. Several indicators were created (see Chapter 7 for details) and entered into the data mining tool. The resulting model performed surprisingly well; with a hit rate of 40 percent it was almost too good to be true. The most important variable was found to be the difference between the amount of bank account charges (for online transfers and cash withdrawals) one and two months prior to the application for the loan.

Just before starting a mortgage campaign which targeted these clients with high numbers of bank transfers, we probed the procedure using mystery shopping — which finally revealed the secret.

When a potential applicant enters a branch of the bank, the requirements of the application are explained first and then basic information about the customer is gathered. If the mortgage is likely to be granted, the client is encouraged to make an advance payment on the house, or initiate refurbishment of the dwelling (as the funding is made available once invoices are verified).

In other words, the high number of transfers identified was not the cause but the effect of applying for a mortgage, even though in a time series the reverse is indicated. Causal relations must be revealed through human input. Data mining and machine learning cannot substitute for this input.

Non-usable Rules

In a credit scoring analysis what we look for is those typical features of defaulting clients that help us score potential clients at the moment they apply for credit based on the true risk they represent. At the primary stage of modelling in the example described above, no features were sorted; all the variables were put into the mining tool. So what was the main distinguishing feature between defaulting clients and the control group? The customer identifier. After double-checking the data, we verified that there was no error; the association rang true (for further details see Chapter 5).

Hopefully, no one in their right mind would create a credit scoring model based on the customer identifier assigned by the scoring system. The bank in question did not!

Overfitting

All the pattern recognition procedures are performed on available historical data. There is a slight chance that a particular pattern observed in the past will never occur again. This phenomenon is called ‘overfitting’. In Chapter 5 a detailed example of image processing is discussed concerning an attempt that was made to recognize camouflaged armoured combat vehicles. Curiously enough, results were exceptionally good, but it was later found that overfitting had caused the success.

After facing all these issues it is only natural to insist that the variables employed in data mining are used in a comprehensible way.

1.4. Data Mining and Business

In the following we make an attempt to identify the key factors which should be considered when undertaking data mining. Successfully dealing with them can result in clear values for business while avoiding the aforementioned pitfalls.

Business Understanding

Having a specific business goal is a necessity for starting any kind of project. Oftentimes, competition in business compels companies to make use of the new opportunities offered by data mining. Ideas about the need to innovate or simple curiosity also contribute to the kicking off of some projects. In some cases no compelling goal is present, only the suspicion that data mining might turn a profit for the company over time.

Achieving the business goal which has been identified is only possible if suitable data are available. In today’s typical business environment, data is superabundant but its quality is often questionable. Data quality affects the duration of data mining projects directly, as when excessive cleaning of data is needed this prolongs the data preparation phase. One approach to this could be the discarding of data of problematic quality, but such an approach may also degrade the final outcome.

The best-case scenario is when a properly maintained data warehouse is available for use where well-structured, quality controlled data resides. Having the support of aptly informed, motivated senior management is also essential when commencing an analysis. When members of the organization (especially management) do not feel convinced about the potential benefits of the project they will get, it is most unwise to proceed; eventually it will fail due to resistance within the company.

Expertise

Three main factors influence the success of data mining projects:

 proficiency and practice in data mining;

 business know how; and

 data insight.

The fields above are listed in order of the strength of their relationship with the specific problem. In most cases only those employees who work with data on a daily basis have data-specific knowledge (e.g. know details about the location of the information, data transformations, and code definitions). The much-needed business know-how is at the disposal of the analysts and experts in the given field. Oftentimes there is need to involve external consultants as well, especially for data mining expertise. Establishing a proficient and effective team of data scientists within a company is rare and does not usually pay off.

1.4.1. Tools

Currently, a number of software vendors are offering their data mining programmes with the promise that they will facilitate more efficient work. Do note however, that successful data mining involves much more than purchasing and introducing new software. Valuable results are only guaranteed with a deep knowledge of the necessary fields mentioned above. A piece of software may produce useful single click solutions, but the interpretation of these results is always up to analysts. Their task is to ask the proper questions, provide the proper data and select a suitable method for analysis. Also among their duties is the interpretation of results, decision-making, and identifying the direction for future inquiries. It is the analyst who makes the final distinction between valuable results, irrelevant information and errors, based on the results.

1.4.2. Business Value

The true value that data mining offers comes in the form of business decisions and actions. A data mining project’s exact worth is always measured by the utilization of its results.

Experience valued from best practices from previous projects has resulted in the induction of the rule of thirds; results of an analysis are considered good when two thirds of them prove the hypotheses that the experts of the given field had, while one third comprise new forms of knowledge. A much higher rate of novelty should raise suspicion and in most cases implies some sort of a mistake; experts are unlikely to have poor levels of knowledge of their respective fields. On the other hand, the revealing of only well-known associations brings no additional value.

Do note that, in some cases, looks can be deceiving. Occasionally, even valuable findings seem evident at first. Data mining can add value by empirically confirming such evidence, thus drawing a distinction between proven findings and the unverifiable rest.

In the end, business value turns into profit, covering the expenses of the data mining projects conducted. Obviously, financial evaluation of such costs and benefits can only be performed for a specific business situation. Nonetheless, in the following we attempt to cover the most common types of costs and benefits which influence the rate of return of data mining projects (see figure 1.2).

Data mining increases the efficiency of business processes. Some activities can be automated, thus increasing output at the same capacity (e.g. a greater number of credit applications may be evaluated or more analytical procedures can be undertaken without employing additional manpower). Practice shows that members of management, driven by the logic of bounded rationality, do not spend less time on preparing decisions. Thus a quicker analysis allows more information to be gathered and more informed decisions to be made. Processing speed is an important factor in the face of fierce competition.

Figure 1.2. Costs and benefits of a data mining project

Data mining brings additional business opportunities. Restructuring of customer behaviour patterns allows for customized solutions to be created in industries where mass services used to be common (e.g. banking, telecommunications, public and the retail sector). Finding the balance between the now-cherished desire for customization and the cutting of costs is the key to success as, according to the rule of economies of scale, a standardized service is substantially cheaper to provide.

Customer segmentation based on market analysis is a practical tool that helps increase purchasing affinity by targeting each client with the products most suitable for them. Apart from being cost-effective, targeted marketing means that fewer irrelevant approaches are made to customers.

Data mining may offer immaterial and not-so-tangible benefits as well — such as increases in brand identity and awareness, more satisfied customers and an improved level of employee satisfaction and be a cornerstone for further development. Engaging in data mining for the sole purpose of achieving these benefits does not usually pay off, but the importance of these benefits should not be neglected.

In practice, it is often not possible to estimate an accurate total return for a data mining project in advance for the following reasons:

 no sophisticated models are available for properly modelling the revenue-raising impacts of the expected results;

 forecasting the expected results of the project is close to impossible, especially as the goal is to reveal hidden associations.

On the other hand, aggregating the costs which emerge during a project is always easier and can potentially raise objections on the part of corporate management, even if total costs are dwarfed by projected profits. Mutual trust is more easily established if a low-cost small-scale pilot project is conducted first, focusing on a single business issue. Based on the experiences of a preliminary project, a more exact evaluation of the potential benefits of a large-scale data mining project can be derived.

1.4.2. Defining the Business Objective

The initial step in any analysis is to identify the main features of the business sector and survey market position, products and clientele, etc. It is also important to become familiar with the organizations and employees involved and set priorities and determine the focus of the project. Typically, many potential objectives arise — more than can be possibly heeded. Until a mutual agreement about objectives is reached within the organization, it is unwise to proceed to the next step. Having set objectives, preliminary time and cost calculations can be done and their approval by senior management marks the effective start of the project.

When we sit down with potential clients for the first time, we ask them what kind of knowledge they seek and what would help them to cut