Project 01 / Finance & Banking / Rule evaluation

Card Fraud and Disputes Performance Review

The true cost of getting detection wrong in both directions. 2.4 million card transactions, eight detection rules, twelve months.

Total cost of fraud

GBP 909k

12 months, all components

Reported to the board

GBP 135k

15% of the real cost

Net fraud loss

7.0 bps

vs ~6.9 bps published UK benchmark

Genuine transactions declined

99,807

32 genuine stopped per fraud caught

Recommended saving

GBP 118k

within a 20% fraud-loss tolerance

Five cost components. Only the first is reported today.

The finding in one paragraph

The fraud team reports gross fraud losses, which fell again this year. Those losses are GBP 135k. The full economic cost of the detection engine is GBP 909k, because the engine also declines 99,807 genuine transactions a year and pushes 222,754 alerts into a review queue. The single largest line is not fraud at all - it is the cost of reviewing alerts that are, in more than 99 cases out of 100, genuine customers trying to spend their own money.

Leave-one-out net contribution per rule.

The trade-off a fraud director actually has to sign.

The conclusion survives every version of the contested assumption.

Alert volume against precision. Bubble = unique catches.

Precision with 95% Wilson intervals. R04 fired 133 times: its interval runs from 2.1% to 9.5%, which is why it gets an interval and not a point estimate.

rule_id	rule_name	default_action	alert_volume	true_positives	precision_pct	precision_ci_low_pct	precision_ci_high_pct	recall_pct	false_positives_per_catch	unique_fraud_catches
R05	HIGH_RISK_MCC_VALUE	review	124702	257.0	0.206	0.182	0.233	5.33	484.2	29.0
R06	CNP_NEW_MERCHANT	review	117346	716.0	0.61	0.567	0.656	14.84	162.9	373.0
R03	GEO_IMPOSSIBLE_3H_CP	block	50658	796.0	1.571	1.467	1.683	16.5	62.6	485.0
R08	NOCTURNAL_BURST	review	44764	2045.0	4.568	4.379	4.766	42.38	20.9	783.0
R02	VELOCITY_5_IN_60MIN	block	26805	1190.0	4.439	4.199	4.693	24.66	21.5	173.0
R01	HIGH_VALUE_ECOM	block	11669	77.0	0.66	0.528	0.824	1.6	150.5	0.0
R07	ROUND_AMOUNT_CROSS_BORDER	block	483	0.0	0.0	0.0	0.789	0.0		0.0
R04	DORMANT_REACTIVATION	review	133	6.0	4.511	2.084	9.493	0.12	21.2	1.0

The metric that decides retirement is not precision

It is unique catches - the fraud a rule finds that no other rule found. R01 HIGH_VALUE_ECOM has respectable-looking precision and 77 catches a year, and its unique catch count is zero. Every single fraud it stops is also caught by at least one other rule, so switching it off loses no detection at all and removes 11,669 alerts.

R07 ROUND_AMOUNT_CROSS_BORDER produced 483 alerts and no confirmed fraud at all in twelve months. The honest way to report that is not 'precision 0%' but 'the 95% upper bound on its precision is 0.79%'.

Why the causal design mattered more than the arithmetic

The obvious comparison - declined customers against never-declined customers - says a false decline increases spend by 536%. It does not. Customers get declined because they are active, and comparing them against inactive customers measures activity, not the decline.

The design that works compares customers declined this month against customers who will be declined three months from now, matched on segment and CRM spend band, over identical calendar weeks. That returns -8.1% (95% CI -13.4% to -2.7%).

A placebo test on an event 90 days earlier returns +11.3% rather than zero, so some upward bias remains. The reported answer is therefore a range, -8.1% to -19.4%, and the cost model uses the conservative end. The true injected value was -9.75%, inside the range.

False decline rate by segment and channel.

segment	channel	genuine_txns	genuine_declined	false_decline_rate_pct	genuine_value_declined	fraud_caught	fp_per_catch
Basic	card_present_chip	47606	2056	4.32	274499.0	31	66.3
Basic	contactless	38144	1494	3.92	48472.0	97	15.4
Basic	ecommerce	53566	1037	1.94	432208.0	110	9.4
Basic	manual_entry	3958	145	3.66	14049.0	19	7.6
Mass	card_present_chip	308737	14420	4.67	2550423.0	140	103.0
Mass	contactless	224325	9257	4.13	334706.0	277	33.4
Mass	ecommerce	333879	8399	2.52	4534833.0	862	9.7
Mass	manual_entry	25081	1058	4.22	125637.0	77	13.7
Mass Affluent	card_present_chip	275076	13785	5.01	2917761.0	120	114.9
Mass Affluent	contactless	179327	8110	4.52	323729.0	190	42.7
Mass Affluent	ecommerce	284990	10074	3.53	6267300.0	410	24.6
Mass Affluent	manual_entry	21331	1022	4.79	159914.0	47	21.7
Premier	card_present_chip	180179	10116	5.61	2471558.0	73	138.6
Premier	contactless	104642	5196	4.97	222966.0	111	46.8
Premier	ecommerce	176830	8151	4.61	5341493.0	279	29.2
Premier	manual_entry	13372	730	5.46	128185.0	22	33.2
Student	card_present_chip	45140	1963	4.35	277955.0	35	56.1
Student	contactless	36882	1427	3.87	43453.0	63	22.7
Student	ecommerce	51195	914	1.79	369929.0	104	8.8
Student	manual_entry	3875	164	4.23	17565.0	14	11.7

Disputes split by class. First-party misuse is a third of case volume and is not a detection failure - it needs a different intervention entirely.

dispute_class	cases	avg_investigation_hours	total_investigation_hours	avg_handling_cost	total_operational_cost	total_write_off	total_cost
First-party misuse	651	1.26	819.0	49.74	40195.0	37292.0	77487.0
Third-party fraud	1520	0.67	1015.0	30.87	65163.0	93126.0	158289.0

Annual saving

GBP 118k

net of the fraud given up

Review alerts removed

26,114

per year

Investigator hours freed

1,959

1.2 FTE

False declines removed

12,964

genuine customers

Extra fraud loss accepted

GBP 32.2k

the risk this change carries

Every recommendation carries the risk it creates. A tuning paper without a risk column does not get signed off.

rule_id	rule_name	current_action	alerts_per_year	precision_pct	unique_catches	net_contribution_gbp	recommendation	rationale	risk_of_this_change
R05	HIGH_RISK_MCC_VALUE	review	124702	0.206	29	-198897	Retire	Costs GBP 198,897/yr more than it saves. 29 unique catches out of 124,702 alerts.	Loses GBP 9,169/yr of fraud prevention. Residual risk sits with counterfeit coverage.
R06	CNP_NEW_MERCHANT	review	117346	0.61	373	-102878	Retire	Costs GBP 102,878/yr more than it saves. 373 unique catches out of 117,346 alerts.	Loses GBP 81,471/yr of fraud prevention. Residual risk sits with card_not_present coverage.
R08	NOCTURNAL_BURST	review	44764	4.568	783	-41474	Retire	Costs GBP 41,474/yr more than it saves. 783 unique catches out of 44,764 alerts.	Loses GBP 49,442/yr of fraud prevention. Residual risk sits with lost_stolen coverage.
R01	HIGH_VALUE_ECOM	block	11669	0.66	0	-27977	Retire	Costs GBP 27,977/yr more than it saves. 0 unique catches out of 11,669 alerts.	Loses GBP 22,344/yr of fraud prevention. Residual risk sits with card_not_present coverage.
R07	ROUND_AMOUNT_CROSS_BORDER	block	483	0.0	0	-166	Keep under review	Net contribution is close to zero; the decision is inside the uncertainty of the cost assumptions.	Re-test after the next quarter of data.
R04	DORMANT_REACTIVATION	review	133	4.511	1	-113	Keep under review	Net contribution is close to zero; the decision is inside the uncertainty of the cost assumptions.	Re-test after the next quarter of data.
R02	VELOCITY_5_IN_60MIN	block	26805	4.439	173	22603	Keep unchanged	Net contribution GBP 22,603/yr.	None from this change.
R03	GEO_IMPOSSIBLE_3H_CP	block	50658	1.571	485	55237	Keep unchanged	Net contribution GBP 55,237/yr.	None from this change.

What this analysis cannot tell you

The rules were reconstructed from published fraud typologies, not read out of a production engine. A real engine has scores and sub-conditions this does not model.
Above-the-line testing only. Lowering a threshold requires re-scanning transactions that never alerted, which is done properly in Project 2.
The behavioural effect is measured over 60 days. Anything that shows up after that - attrition, reduced product take-up - is not captured and would make the false-decline cost larger, not smaller.
The analyst review outcome is modelled at a 78% catch rate and a 6% false-decline rate. Real dispositions would replace it.