Fraud Rules and ML Models: Design, Tune and Govern

Most fraud detection programmes run a mix of hand-written rules and machine learning models, and most of their problems come from how the two are designed, tuned and governed together rather than from the choice of algorithm. Rules are fast to deploy and easy to explain but decay as fraudsters adapt; models catch subtler patterns but depend on labels that arrive weeks late and on data that drifts. This guide covers when to use rules versus models, feature design, label delay, class imbalance, threshold tuning, champion and challenger testing, graph detection, drift monitoring and model risk governance under SR 11-7 and similar regimes. It is for fraud analysts, data scientists and model risk teams in banks, payments firms and digital businesses. For how to measure the outcome, see the Fraud Metrics and KPIs Guide; this page is about the detection logic itself.

Key takeaways

Rules and models do different jobs. Use rules for policy, hard limits and fast response to a new attack; use models for ranking risk across many weak signals.
Labels arrive late and incomplete. Chargebacks, scam claims and confirmed fraud can take weeks or months, so recent performance is always overstated unless you account for label maturity.
Accuracy is the wrong metric. With fraud rates well under 1%, a model that flags nothing looks almost perfectly accurate; tune on precision, recall and cost at a chosen threshold.
Every change is an experiment. Shadow mode and champion and challenger testing prevent a "better" model from quietly blocking good customers.
Fraud models are models. SR 11-7 defines a model broadly enough to cover fraud scoring, so inventory, validation, effective challenge and ongoing monitoring apply, including to vendor scores.

Rules versus models: choosing the right tool

A rules engine evaluates explicit conditions ("decline if the card was added less than 10 minutes ago and the shipping country differs from the issuing country"). A machine learning model learns a scoring function from labelled history and outputs a probability or score that a policy then acts on. Mature programmes run both in one decision flow: hard policy rules first, then a model score, then rules that turn the score into an action (approve, step up, review, decline).

Factor	Rules	Machine learning models
Time to deploy against a new attack	Hours	Days to weeks (needs labelled examples and validation)
Explainability	Direct: the condition is the reason	Needs reason codes or feature attribution
Handling many weak signals	Poor: rule sets grow and overlap	Strong: combines hundreds of features
Decay	Fast once fraudsters learn the threshold	Slower, but drifts with behaviour and products
Data needed	Domain knowledge and recent cases	Large labelled history with consistent definitions
Typical use	Regulatory and policy limits, sanctions, velocity caps, emergency blocks	Transaction, login, onboarding and payee risk scoring
Main failure mode	Rule sprawl and overlapping declines nobody owns	Silent performance loss and label leakage

Small organisations: start with a well-owned rule set plus a vendor score, and invest in labelling discipline before building in-house models.
Large organisations: build in-house models for core flows, keep rules for policy and rapid response, and retire rules the model already covers.
Rule hygiene: give every rule an owner, a purpose, a creation date, an expected hit rate and a review date. Measure each rule's marginal value: the fraud it catches that nothing else would.

Features and data

Model quality is mostly feature quality. The strongest fraud features describe change and context rather than static attributes.

Feature family	Examples	Watch out for
Velocity	Transactions, failed logins or new payees per account, card, device or IP in 1 hour, 24 hours, 7 days	Window aggregates must be computed exactly the same way in training and real time
Behavioural deviation	Amount versus the customer's 90-day median, new merchant category, unusual time of day	Thin history for new customers
Device and session	Device age on account, emulator or remote access indicators, typing cadence	Privacy notices and consent for device signals
Identity and account	Account age, recent password, email or phone changes, recent recovery events	Account changes are strong signals for takeover; feed them in near real time
Counterparty	Payee age, payee reported by other customers, merchant chargeback rate	Consortium data sharing terms
Network and graph	Shared devices, addresses or payment instruments across accounts	Compute cost and explainability

Leakage: never train on fields populated after the decision, such as a review outcome, a reversal flag or a later account closure. Leakage produces excellent offline results and poor live ones.
Training and serving parity: the most common production bug is a feature calculated differently in the batch training pipeline and the real-time scoring path. Log the exact feature values used at decision time and train on those.
Identity signals: recent changes to credentials, MFA factors and recovery details are among the best takeover features. See the Account Takeover Prevention Guide and the Transaction Monitoring and Behavioural Biometrics Guide.

Labels, label delay and class imbalance

A fraud label is the confirmed outcome of a past decision, and it is the weakest part of most fraud data.

Label delay: card disputes, scam reimbursement claims and first-party fraud surface weeks or months after the transaction. Choose a label maturity window (for example 90 days) and evaluate only on transactions older than it; report recent periods as provisional.
Selection bias: declined transactions never get a fraud outcome, so the model learns only from what was approved. Keep a small random holdout that bypasses some declines under strict loss limits, or use reviewed cases, so the model sees outcomes the current system blocks.
Label definitions: separate third-party fraud, first-party fraud, scams with customer authorisation and friendly fraud. Mixing them teaches a model contradictory patterns. See the First-Party Fraud Guide.
Imbalance: Google's machine learning course uses card fraud as its example, noting fraudulent purchases might be under 0.1% of examples. It describes downsampling the majority class and upweighting it by the same factor (for example downsample legitimate transactions by 25 and weight each by 25) so the model sees enough fraud while keeping a true sense of base rates.

Scoring, thresholds and tuning

A model produces a score; the business decides what to do at each score. That threshold is where most value is won or lost.

Right metrics: precision is the share of flagged cases that are fraud, recall the share of fraud caught, false positive rate the share of genuine activity wrongly flagged. Google's course shows that on a 1% positive dataset a model that always predicts "not fraud" scores 99% accuracy while catching nothing.
Cost-based thresholds: set thresholds by expected loss, not by a round score. Compare the value of fraud prevented against the cost of declined good customers, review effort and step-up friction at each candidate threshold.
Tiered actions: use several bands (approve, step-up authentication, manual review, decline) so the most uncertain cases get friction rather than a decline.
Capacity-aware review: set the review band to the number of cases the team can work within the decision deadline; an unworked queue is a silent approval.
Reason codes: return the top contributing features with every score so analysts can work cases quickly, customers can be given a meaningful explanation, and validators can spot a model leaning on a proxy it should not use. Keep the reason code mapping under change control alongside the model.
Fallback policy: decide in advance what happens when the model times out or a feature source fails: a conservative rule set, a default step-up, or approval under a lower limit. Test the fallback as often as the model.
False positives: track customer complaints, abandonment after step-up and repeat declines per customer. False positives are a fraud cost even if they never appear in loss figures.

Champion and challenger testing

Changing a rule or model changes who gets declined. Treat every change as a controlled experiment.

Offline back-test: score a mature, out-of-time sample that was not used in training. SR 11-7 describes back-testing as comparing outputs with actual outcomes on data not used in development.
Shadow mode: run the challenger alongside the champion on live traffic, logging scores without acting on them, for at least one full label maturity cycle where possible.
Controlled split: route a small random share of traffic (for example 5-10%) to the challenger's decisions, with loss limits and an automatic switch back.
Compare on mature labels: evaluate fraud caught, false positives, review volume and customer impact once labels have matured, not on week-one numbers.
Promote or roll back: promote through change control with sign-off; keep the previous champion deployable so rollback is a configuration change, not a rebuild.

Graph detection

Organised fraud reuses resources: the same device, phone number, address, payee account or payment card appears across many accounts. Graph methods model accounts and these shared attributes as connected nodes.

Link analysis: derive features such as "number of accounts sharing this device in 30 days" or "distance to a confirmed fraud account" and feed them to the main model.
Community detection: find clusters of accounts that behave as a ring, such as mule networks receiving and forwarding funds, and send whole clusters to investigation.
Cautions: shared attributes have innocent explanations (households, shared office networks, carrier-grade NAT), so use graph signals to prioritise investigation rather than to auto-decline. See the Fraud Data Sharing and Consortium Intelligence Guide for cross-institution links.

Drift and monitoring

NIST's AI Risk Management Framework notes that AI systems may need more frequent maintenance because of data, model or concept drift, and calls for monitoring the functionality and behaviour of deployed systems (MEASURE 2.4) and post-deployment monitoring plans including override, incident response and change management (MANAGE 4.1).

What to monitor	Signal	Typical response
Input drift	Population stability index or distribution shift on key features	Investigate source system changes before retraining
Score drift	Share of traffic in each score band changes week on week	Check thresholds still meet review capacity
Concept drift	Precision or recall falls on matured labels	Retrain or redevelop; add rules for the new pattern meanwhile
Rule decay	Rule hit rate or precision falls, or marginal value near zero	Retune or retire
Data pipeline health	Missing or default feature values, latency	Fail safe to a defined fallback policy
Override rate	Analysts reversing model decisions	Review for model weakness or training gaps

Metrics that show the detection stack is healthy: matured-label recall at the operating threshold, false positive rate or good-customer decline rate, review queue hit rate (share of reviewed cases confirmed as fraud), rule marginal value per rule, and time from new attack pattern to deployed control.

Governance: SR 11-7 and model risk management

The Federal Reserve's SR 11-7 (April 2011, issued with the OCC) defines a model as a quantitative method, system or approach that turns input data into quantitative estimates, including approaches that use expert judgement as long as the output is quantitative. A fraud score meets that definition; many organisations also treat complex rule sets as models or model-like tools.

Inventory: SR 11-7 expects an inventory of models in use, under development and recently retired. Include vendor scores and rule sets that drive automated declines.
Validation: the three core elements are conceptual soundness, ongoing monitoring (including benchmarking) and outcomes analysis (including back-testing). Validation applies equally to vendor models.
Effective challenge: critical review by objective, informed parties with the influence to force change. For fraud, that means validators independent of the team rewarded for loss reduction.
AML systems: the 2021 interagency statement (SR 21-8, Federal Reserve, FDIC and OCC) says model risk management principles may be appropriate for BSA/AML systems but do not require any specific framework.
UK banks: the PRA's SS1/23 model risk management principles (effective 17 May 2024) apply to UK banks and building societies with internal model approval, cover models used to inform business decisions regardless of technology, and include managing AI and machine learning risks.
EU AI Act: Annex III point 5(b) lists creditworthiness and credit scoring of natural persons as high risk "with the exception of AI systems used for the purpose of detecting financial fraud". Fraud detection is therefore carved out of that high-risk category, though data protection law still applies to automated decisions.

Activity	Reference
Model inventory and tiering	SR 11-7; PRA SS1/23 Principle 1 (model identification and model risk classification)
Development standards and documentation	SR 11-7; PRA SS1/23 Principle 3
Independent validation	SR 11-7 core validation elements; PRA SS1/23 Principle 4
Production monitoring and drift	NIST AI RMF MEASURE 2.4, MANAGE 4.1
Override, fallback and change control	NIST AI RMF MANAGE 4.1; PRA SS1/23 Principle 5 (model risk mitigants)

Common failure patterns

Grading on immature labels: a new model looks superb in its first month because its fraud has not been reported yet.
Rule sprawl: hundreds of overlapping rules with no owners, several declining the same good customers, none ever retired.
Training on yesterday's decisions: the model learns the old rules' blind spots because declined traffic never gets labels.
Feature mismatch in production: a velocity window computed differently online than in training quietly degrades scores.
Vendor score treated as a black box: no validation, no monitoring and no understanding of what changed when the vendor retrained.
Thresholds set once: review capacity, fraud mix and products change while the cut-off stays where it was set at launch.

Practitioner checklist

Map the full decision flow: policy rules, model scores and the rules that convert scores to actions.
Give every rule an owner, purpose, expected hit rate and review date; retire rules with no marginal value.
Define fraud label types and a label maturity window; evaluate only on matured data.
Log decision-time feature values and train on them to guarantee training and serving parity.
Keep a controlled holdout so the model can learn from traffic the current system declines.
Set thresholds by expected cost and review capacity, with tiered actions rather than a single cut-off.
Run every rule or model change through back-test, shadow mode and a controlled champion and challenger split.
Monitor input, score and concept drift, override rates and pipeline health with defined responses.
Put fraud models, vendor scores and decisioning rule sets in the model inventory with independent validation.

Fraud Rules and Machine Learning Models Guide: Design, Tune and Govern Detection