Differentially Private Fraud Detector Benchmarking

Benchmarks fraud detection models across institutions using subsample-and-aggregate methods or synthetic transaction graphs to preserve customer privacy with formal differential privacy guarantees.

The Problem

“Differentially Private Fraud Detector Benchmarking Across Financial Institutions”

Organizations face these key challenges:

1

Raw transaction graphs contain highly sensitive customer and relationship data

2

Simple anonymization is vulnerable to graph re-identification and linkage attacks

3

Institutions cannot easily compare fraud detectors on common realistic benchmarks

4

Privacy-preserving evaluation often degrades model utility if not carefully tuned

Impact When Solved

Enables cross-institution fraud model benchmarking without raw customer graph sharingProvides formal differential privacy accounting for benchmark outputsReduces compliance and legal barriers to collaborative model evaluationImproves confidence in fraud detector generalization across institutions

The Shift

Before AI~85% Manual

Human Does

  • •Negotiate data-sharing terms and approve limited benchmarking arrangements
  • •Prepare anonymized transaction datasets and manually remove sensitive fields
  • •Coordinate benchmark runs across institutions and collect results
  • •Review legal, compliance, and privacy risks before any data release

Automation

  • •Run fraud detection models on local or shared benchmark datasets
  • •Produce basic performance summaries from submitted benchmark results
  • •Flag obvious data quality issues in benchmark inputs
With AI~75% Automated

Human Does

  • •Set benchmarking scope, privacy budget limits, and participation rules
  • •Approve benchmark releases and review privacy-utility tradeoffs
  • •Handle exceptions involving unusual graph patterns, compliance concerns, or disputed results

AI Handles

  • •Execute standardized local benchmark jobs without exporting raw transaction graphs
  • •Generate differentially private aggregate scorecards or synthetic benchmark graphs
  • •Track privacy budget consumption and monitor benchmark quality across rounds
  • •Compare fraud detector performance across institutions and surface benchmark insights

Operating Intelligence

How it works

Humans set constraints. AI generates options.

Humans choose what moves forward.

Selections improve future generation quality.

Confidence88%
ArchetypeGenerate & Evaluate
Shape6-step branching
Human gates2
Autonomy
50%AI controls 3 of 6 steps

Who is in control at each step

Each column marks the operating owner for that step. AI-led actions sit above the divider, human decisions and feedback loops sit below it.

Loop shapebranching

Step 1

Define Constraints

Step 2

Generate

Step 3

Evaluate

Step 4

Select & Refine

Step 5

Deliver

Step 6

Feedback

AI lead

Autonomous execution

2AI
3AI
5AI
gate
gate

Human lead

Approval, override, feedback

1Human
4Human
6↺ Loop
AI-led step
Human-controlled step
Feedback loop
TL;DR

Humans define the constraints. AI generates and evaluates options. Humans select what ships. Outcomes train the next generation cycle.

The Loop

6 steps

1 operating angles mapped

Operational Depth

Technologies

Technologies commonly used in Differentially Private Fraud Detector Benchmarking implementations:

DP synthetic graph dataOther
4 mentions
Fraud detector benchmark experimentsOther
4 mentions
Subsample-and-aggregateOther
4 mentions
Differential privacy guaranteeOther
3 mentions

Key Players

Companies actively working on Differentially Private Fraud Detector Benchmarking solutions:

Privacy-preserving ML evaluation frameworksSecure internal-only fraud benchmarking

Real-World Use Cases

Differentially private fraud-detector benchmarking via subsample-and-aggregate or synthetic graphs

The idea is to add privacy protection when testing fraud models on secret transaction graphs, either by splitting data into pieces and combining noisy results or by creating privacy-safe fake graphs.

Privacy-preserving evaluation of graph ML under differential privacyimmature for deployment; current methods fail to preserve enough utility under dp.
10.0

Free access to this report