Fireside Chat: FinCrime Agent Benchmark and Evaluating AI for AML
What does it take for an AI agent to adjudicate AML alerts reliably enough to hold up with an examiner? In this session, Peter Piatetsky of Castellum.AI is joined by Matt Coates of Stripe and Tyler Nielsen of Rain to discuss the OCC’s new supervisory framework, how they test and govern agents in production and what FCA Bench revealed about general purpose LLMs.
Key takeaways
The OCC framework ties control intensity to consequences: The November 4, 2026 guidance names alert disposition as a use case and leaves it to examiners and institutions to define what human involvement looks like.
Agent accuracy and consistency are the two lenses that matter: An agent should reach the correct screening decision under your policies and return the same output whether it sees an alert 10 times or a million times.
Judge the agent and the human separately: Stripe evaluates agent outputs and investigator work against separate criteria, which puts accountability on the agent and has helped bring handle times down.
Point in time validation won’t be enough: Prompt changes and vendor updates can cause drift, so governance needs ongoing monitoring and consistency thresholds.
A purpose-built harness closes the accuracy gap: In FCA Bench, the best standalone model got 71.1% of cases fully correct, while the same models inside Castellum.AI’s harness reached up to 99.6% accuracy.
Pre-deployment testing starts with your human baseline: Match current precision, recall and handle times in shadow testing, then decide whether each mismatch is a blocker, a quick fix or an accepted risk.
Meet the Speakers
Tyler leads sanctions compliance strategy at Rain, a stablecoin infrastructure company. Before Rain, Tyler founded SET Global, a consultancy advising banks and fintechs on fincrime compliance. He previously held multiple roles in the US government around sanctions design, implementation and impact.
Peter Piatetsky
Co-Founder & CEO
Castellum.AI
Matthew Coates
Financial Crimes Supportability and Oversight Lead
Stripe
Tyler Nielsen
Head of OFAC and Sanctions Compliance
Rain
Peter leads strategy, growth and product at Castellum.AI, working closely with clients to implement risk-aligned solutions. Prior to co-founding Castellum.AI, Peter served at the US Treasury Department.
Matt is the Financial Crimes Supportability and Oversight Lead at Stripe, a global payments and financial infrastructure platform. Prior to Stripe, he built AML programs from the ground up at Etsy and Intuit across the US, EU and Brazil.
Featured Resource
FinCrime Agent Benchmark
The first benchmark to evaluate how AI models apply an institution’s AML/KYC policies, identify risk and adjudicate alerts. What’s inside the benchmark:
Model leaderboard: How 9 leading LLMs perform across 500 sanctions, PEPs and adverse media cases.
Standalone vs. harnessed results: How each model performs on its own and through Castellum.AI’s agentic harness.
Screening effectiveness: How many expected hits each model finds before adjudication begins.
Decision consistency: Whether a model reaches the same decision when it runs the same case 3 times.