What it means
Reporting tells you what happened, while data mining looks for structure that lets you predict what will happen or explain why it did. A sales report shows last month's churn rate, whereas a mining exercise identifies the combination of behaviours that precedes a cancellation.
The commercial value lies in acting on that pattern before the event. The common tasks are classification, meaning sorting records into groups such as likely to buy or not, regression, meaning predicting a number such as expected spend, clustering, meaning finding natural groupings without being told what to look for, and association, meaning finding things that occur together.
Most business uses fall into the first two, because they attach directly to a decision someone actually has to make. In practice the work is mostly preparation.
Pulling data from several systems, cleaning it, dealing with missing values and building sensible variables usually takes far longer than fitting the model, and it determines whether the result is useful. Teams that skip this and go straight to algorithms almost always produce impressive-looking models that fail in production.
The classic business measure is lift, meaning how much better the model is than choosing at random. If the top 10% of a customer list ranked by the model contains four times the response rate of the list as a whole, the model has a lift of four in that decile, and the campaign can be targeted at a fraction of the cost.
The risks are real. Mining a large dataset hard enough will always throw up relationships that are coincidental, which is why models must be tested on data they have never seen, and using personal data for purposes the customer never agreed to is both a legal and a reputational problem.
The activity has an unflattering cousin called data dredging, which means searching until something looks significant.
In practice
Real-world examples.
Example
A subscription streaming service builds a churn model from viewing frequency, device mix and support contacts. Customers scored in the highest risk band are sent a personalised recommendation email rather than a discount, and monthly churn in that band falls from 6.5% to 5.1%. The retained revenue is worth many times the cost of the modelling work.
Example
A commercial bank mines transaction data for invoice fraud and finds that payments to newly added suppliers in the last three days of a month are far more likely to be fraudulent. The rule is added to the payment approval workflow as a manual check. It flags about 40 payments a month, of which two or three turn out to be genuine attempts at fraud.
Example
A supermarket chain clusters loyalty card data and finds a segment that buys baby products but no fresh food. Rather than assuming these are new parents, the analysts establish that they are shoppers using the store for top-up trips and doing their main shop elsewhere. The insight redirects budget from baby vouchers to a fresh produce offer.
Formula
Calculation
Model lift = response rate of the targeted group / response rate of the whole population
A retailer has 500,000 customers on its mailing list and a baseline response rate of 2% to a catalogue mailing that costs $1.20 a pack to produce and post. Each response generates $45 of gross profit.
Mailing 50,000 customers chosen at random:
Responses = 50,000 x 2% = 1,000
Gross profit = 1,000 x $45 = $45,000
Mailing cost = 50,000 x $1.20 = $60,000
Net result = $45,000 - $60,000 = -$15,000
A mining model ranks the list, and the top decile responds at 8%, giving a lift of 8% / 2% = 4.
Responses = 50,000 x 8% = 4,000
Gross profit = 4,000 x $45 = $180,000
Mailing cost = 50,000 x $1.20 = $60,000
Net result = $180,000 - $60,000 = $120,000
The same budget turns a $15,000 loss into a $120,000 profit, purely by changing who receives the catalogue.Case study
Seen in the real world.
Brightgale Insurance is an illustrative, invented motor insurer used to show data mining applied to a real commercial question. It had 200,000 policies coming up for renewal each year at an average premium of $600, and a renewal rate of 68%, so 136,000 customers renewed.
The marketing team wanted to offer every renewing customer a 10% discount to lift retention. At $60 a policy that would have cost 136,000 x $60 = $8,160,000 a year, which the finance director refused to sign off without evidence.
A model built on claims history, tenure, price sensitivity and quote behaviour showed that roughly two thirds of renewing customers would have stayed at full price, leaving about 45,000 policyholders genuinely at risk of leaving. Targeting the discount at that group cost 45,000 x $60 = $2,700,000 and lifted the renewal rate to 74%, worth 12,000 extra policies at $600, or $7,200,000 of retained premium. The illustrative point is not the model's cleverness but the money: the same objective was met for $5,460,000 less, because the analysis identified who actually needed persuading.
Watch out
Common mistakes.
- Confusing data mining with reporting, when dashboards describe the past and mining builds models that rank or predict.
- Judging a model on how well it fits the data it was built on rather than on unseen test data, which hides overfitting.
- Assuming a discovered correlation is a cause, then changing a process on that basis without running any test.
Questions
People also ask.
How much data do you need?
Less than people assume for a simple classification task, since a few thousand well-labelled examples with a decent number of positive cases is often enough to beat guesswork.
Is data mining the same as machine learning?
They overlap heavily. Data mining is the broader business activity of finding useful patterns, while machine learning is the set of techniques most often used to do it.
What stops a project failing?
A named decision the model will feed and a person who will act on it, because models built without a decision attached almost never get used.
From the founder's library
Take it further with the book.
Build your financial confidence beyond this definition. Shihan's full-length guide, Accounting Fundamentals, takes the same plain-English approach and turns it into a complete, practical playbook for non-finance managers, business owners and students - with chapter-end quiz answers and presentation slides included.
25% off with code MMHQ25, applied at checkout. Priced in USD - checkout may show the equivalent in your local currency.
View the book and save 25%