Skip to content

The new BARC study is here

Download now

Near Real-Time Fraud Scoring with Exasol, Kafka, and Debezium

Sanjay Kumar
· · 12 mins read

Effective fraud detection depends on one thing above all else: fresh context. When a suspicious transaction happens, the signals that matter most; what the account did in the last five minutes, whether this merchant has ever appeared before, whether the channel just changed are only meaningful if they’re current. Stale data doesn’t just slow down detection; it changes what you can detect at all.

Yet most fraud detection systems still behave like reporting tools; they collect data overnight, process it by morning, and surface results by the next business day. By then, the window to act has long since closed.

That gap is what this project addresses. We built and open-sourced a near real-time fraud analytics pipeline using Exasol as the analytical core one that takes banking transaction events as they happen, moves them through a streaming infrastructure, and delivers fraud scores without the sprawling architecture that usually comes with real-time ambitions.

Explore the project: exasol-labs/industry-solutions: A collection of industry specific customer solutions and deployment patterns built on top of Exasol

In this post, we’ll cover:

  • why “batch first” fraud pipelines keep burning teams in practice,
  • how this pipeline is wired end to end, and
  • which pieces you can lift directly into your own environment.

Pipeline at a glance:

PostgreSQL (OLTP) → Debezium CDC → Kafka + Avro + Schema Registry → Exasol (RAW/CLEANSED/ANALYTICS) → Python UDF for fraud scoring

Why batch fraud pipelines fail in practice

Batch pipelines were never designed for fraud; they were designed for reports.

That’s not just a slogan. In most banks, the data stack started life as a reporting platform for finance and risk: nightly loads, a warehouse, a BI layer. When fraud teams needed data, building on that existing platform was the sensible decision. Ten years later, the sensible decision is showing its limits.

Three predictable problems show up again and again:

1. Fraud moves faster than the pipeline

Some patterns unfold in minutes:

  • bursts of small transactions just under common thresholds,
  • account takeovers followed by a quick drain,
  • coordinated hops through mule accounts.

If your main fraud view refreshes overnight, you’re always looking at what happened yesterday, not what’s happening right now. That’s fine for loss reporting; it’s not fine for stopping active fraud.

2. Investigators are working with stale context

Ask a fraud analyst what they want to see first and you’ll hear some version of:

“Show me everything that happened around this account or card in the last few minutes.”

Batch stacks struggle here. Analysts end up:

  • pulling yesterday’s warehouse view,
  • opening a separate system for authorizations,
  • checking a case management tool for prior alerts,

and then rebuilding the timeline by hand. By the time the picture is clear, the window to block or reverse a transaction may have closed.

3. Complexity quietly accumulates

Most batch fraud stacks grow by addition:

  • a new export for a model,
  • a hand‑rolled feature pipeline,
  • a separate environment for “near real‑time” scoring,
  • a couple of glue scripts to keep them all in sync.

Each change makes sense locally. Taken together, you end up with a system that has many moving parts, is fragile to schema changes, and takes serious engineering effort just to keep running. This reference pipeline is meant as the opposite of that: closer to real time, but small enough that a single team can understand and operate it.

The Full Architecture: What Was Built and Why

The pipeline connects four stages, each with a clear job to do. Before diving into each layer, here’s the high‑level flow:

  1. PostgreSQL – banking system of record
    This is the OLTP database for accounts, cards, customers, and transactions. Every card payment, transfer, and alert is written here first. Each of those writes is the starting point for the pipeline.
  2. Debezium – Change Data Capture without polling
    Debezium taps into PostgreSQL’s write‑ahead log and turns each committed change into a structured event. There’s no polling, no batch window. If a transaction is written, it’s captured and pushed into the stream almost immediately.
  3. Kafka + Avro + Schema Registry – governed event transport
    Kafka acts as the central event transport, a durable, ordered log of everything that has happened. It decouples the banking database from the analytical systems downstream, so each side can evolve independently. Every event also follows a defined schema, enforced at publish time. This prevents the all-too-common situation where one team changes a data field and quietly breaks every consumer downstream.
  4. Exasol – landing, cleansing, features, and scoring in one place
    Exasol is where the heavy lifting happens and this is where the architecture diverges from the typical approach. In a conventional setup, you would need separate systems to land the stream, clean the data, compute fraud features, and serve the model. Here, all of that happens inside Exasol. Incoming events land in a staging area, get merged into a clean historical record, flow into fraud features, and are scored by a model, all without leaving the platform.

The end‑to‑end pipeline looks like this:

PostgreSQL (banking database)
→ Debezium CDC (continuous event stream)
→ Kafka + Schema Registry (event transport + governed contracts)
→ Exasol (landing → cleansing → features → fraud scoring)

Three principles shaped these choices:

  • Speed over staleness. Events should go from write to score in minutes, not hours.
  • Consistency across teams. Schemas are enforced so one team can’t quietly break another.
  • Fewer moving parts. Every extra system is another failure mode; we consolidate wherever we can without sacrificing correctness.

For a deeper architectural walk‑through, see the project docs in the repo:
industry-solutions/realtime-banking-fraud-detection/docs/architecture-technical.md at main · exasol-labs/industry-solutions

The Journey of a Transaction

The easiest way to make this concrete is to follow a single transaction through the pipeline.

  1. A customer makes a payment
    The transaction is written to the PostgreSQL banking database along with the usual context: account, card, customer, merchant, channel, and so on.
  2. Debezium spots the change
    The write hits the database log. Debezium picks it up almost immediately and emits a structured event to Kafka; no cron job, no intermediate staging file, no polling lag.
  3. Kafka carries the event with guardrails
    The event travels through Kafka in its governed format, validated against a defined schema before any downstream system can consume it. This is a small but important step. In fraud detection, where many teams may be touching the same data, schema enforcement is the guardrail that prevents silent breakages from propagating through the pipeline. A producer-side change that would break a consumer gets caught at registration time, not three days later when someone notices scores have stopped updating.
  4. Exasol ingests the event from Kafka
    Exasol’s native Kafka connector reads the event and drops it into a staging table. From there, a merge process:
    • keeps only the latest version of each record,
    • applies updates and deletions correctly,
    • and writes the result into a stable analytical layer.

      That analytical layer is what all downstream logic trusts.
  5. Fraud features are computed in ExasolFrom there, fraud-relevant signals are computed directly inside Exasol. Not generic velocity counts, but the specific questions a fraud investigator would actually ask when a transaction looks wrong:
    • Was OTP bypassed on a foreign merchant POS transaction? A contactless or fallback swipe at an overseas terminal without step-up authentication is a pattern worth flagging independently of the amount.
    • Is this the first time this card has used contactless, Pay Later, or tokenised payment? Fraud frameworks used in Apple Pay and similar tokenised rails treat first-use of a new payment modality as an elevated-risk event, the cardholder may not have initiated it.
    • Did the device token change between the last transaction and this one? Token substitution, common in account takeover scenarios, is invisible to amount-based rules but shows up clearly in token lineage.
    • Is the merchant category code consistent with this card’s history? A card with three years of grocery and transit spend suddenly appearing at a foreign electronics retailer is a behavioural anomaly that velocity alone won’t catch.
    • Was the authentication method downgraded? A transaction that could have required biometric or 3DS confirmation but settled on signature or PIN-bypass at the terminal is worth scoring differently, the downgrade itself is a signal.

These questions are answered as SQL logic running against the data that’s already there:

SELECT txn_id, amount, ANALYTICS.FRAUD_SCORE_UDF(amount, txn_count_1h, avg_amount_30d, ...) AS fraud_probability FROM ANALYTICS.FRAUD_FEATURES WHERE score_status = 'PENDING';

Finally, a trained fraud model, stored inside Exasol alongside the data, is invoked to produce a score. The result is written back to the analytics layer, where it’s immediately available to investigators and downstream systems. From write to score, the pipeline can complete in minutes. Crucially, if the model file is ever missing, scoring fails visibly, there’s no silent fallback, no degraded output that goes unnoticed. That’s an intentional design choice: in fraud detection, a quietly broken scoring layer is more dangerous than an obviously broken one.

Running It Yourself

The entire streaming environment comes up with a couple of commands.

  1. Clone the repository and configure Exasol
  2. Start the stack
    From the project directory:
docker compose up -d --build bash deploy.sh

That spins up the banking database (pre-seeded with sample data), the event streaming infrastructure, schema governance, and a demo dashboard, all locally. Exasol runs as a separate Docker container, the one component outside the compose file. Start it independently, point the configuration to its host and port, and the provided SQL scripts handle the rest: data layers, connectors, and scoring model are all set up automatically.

If you’d rather see it before you run it, check out the recorded walkthrough that shows the full sequence from transaction insert to fraud score. It’s a useful reference for understanding what each step looks like in practice before you set up your own environment.

Once everything is running, a good first tour looks like this:

  • Insert a test transaction through the dashboard.
  • Watch the event appear in the Kafka stream monitor.
  • Trigger the Exasol import and feature refresh.
  • Run the scoring step.
  • Query the top‑scored transactions in Exasol.

The suspicious one surfaces at the top. More usefully, you can trace the same transaction record across every stage, from the banking database, through staging, into the cleansed layer, and finally as a scored fraud feature. That end-to-end lineage visibility is something teams often lack in their production systems, and seeing it in a working demo tends to prompt the right conversations.

From Demo to Production: What to Strengthen

This is a reference pipeline, designed to be understood, run, and adapted, not deployed as-is. For teams taking it further, four areas stand out.

Pipeline health monitoring. Basic visibility is included, but production needs active alerting. If events stop flowing or the pipeline falls behind, risk teams should know immediately, not after scores have quietly degraded.

Resilience to bad data. Real banking data is messy. A production pipeline needs an explicit strategy for late, missing, or malformed records rather than letting them fail silently.

Scheduled feature refresh with auditability. Features are refreshed on demand in the demo. In production, especially in a regulated environment, you’d schedule those runs and log when they happened, from which data offset, and who triggered them.

Model lifecycle management. The pipeline already tags each score with a model version. The natural next step is deploying new models without downtime and running two versions in parallel before fully committing, the foundation of a trustworthy scoring layer.

Why Exasol for This Pipeline

The decision to use Exasol as the analytical platform rather than assembling a collection of specialist tools is the architectural choice that most defines this pipeline, so it’s worth explaining directly. And it connects back to the original problem: the systems that fail at real-time fraud detection don’t usually fail because of a single bad component. They fail because of the accumulated cost of too many components.

A common alternative would involve a dedicated stream processing engine for feature computation, a feature store for serving those features to a model, a model serving layer for inference, and a warehouse for historical analysis with integration jobs holding them all together. Each of those components is a reasonable choice on its own. Together, they create a system with many boundaries to maintain, many places where data can silently drift out of sync, and significant operational overhead for a team that ultimately just wants to detect fraud faster.

This pipeline consolidates those responsibilities inside Exasol. Events land in Exasol, features are computed in Exasol, the model runs in Exasol, and historical analysis lives in Exasol. There is no data export step before scoring, no separate feature computation service, no latency accumulating at system boundaries.

For a banking fraud context specifically, Exasol’s columnar engine handles the join-heavy workload transaction enriched with account history, card behaviour, customer profile, and prior alerts efficiently and at analytical speed. The Python UDF capability means the model lives where the data is. And the reduction in moving parts means a small team can own and operate the pipeline without deep expertise spread across five different tools.

The result is an architecture that’s faster to reason about, easier to debug, and more straightforward to hand off; which, in fraud detection, translates directly into faster response times and fewer blind spots.

What Would You Add First?

The fraud signals in this reference pipeline are a deliberate starting point, not a ceiling. They’re common, well-understood indicators, enough to make the model useful and the architecture story clear. The more interesting question, for anyone adapting this to a real environment, is where to go next.

The fraud signals here are a starting point, not a ceiling. The easy wins are already in the data: transaction velocity over tighter windows, first-time merchants, channel shifts, cross-border flags. Beyond that, device and network reputation add meaningful lift; as do graph-based signals for spotting mule networks but those need additional data sources.

The repo is open, the extension points are documented, and the stack is meant to be modified.

Clone it, run it, bend it to your reality, and if you build something interesting on top, tell us what you added first.

Happy Exasoling!

Sanjay Kumar
Sanjay Kumar

Sanjay Kumar is a database and analytics professional at Exasol with a passion for query optimization, data engineering, and AI. He writes about practical solutions, performance tuning, semi-structured data, and the evolving role of AI in modern data platforms.