Reconciling 1,000,000 M-Pesa & Bank Transactions in Seconds with Rust & Apache DataFusion

In enterprise accounting and fintech operations, financial data reconciliation is frequently plagued by manual Excel VLOOKUP formulas, fragile CSV exports, and changing bank statement layouts. When transaction volumes scale past 100,000 records daily across M-Pesa Paybills, Card Gateways, and Bank Statements, traditional relational databases and manual spreadsheets collapse under I/O bottlenecks and memory constraints.

At Resarv, we built FinOps R² — a high-performance horizontal data resolution engine capable of matching over 1,000,000 multi-source transaction records per second using Rust and Apache DataFusion.

---

⚡ 1. Why Rust & Apache DataFusion?

Traditional reconciliation tools built on Python/Pandas or JVM-based frameworks suffer from heavy garbage collection pauses and high memory consumption when joining multi-gigabyte datasets.

// Core DataFusion In-Memory SQL Context Setup
use datafusion::prelude::*;
use std::sync::Arc;

pub async fn execute_reconciliation_pipeline(
    mpesa_df: DataFrame,
    bank_df: DataFrame,
) -> Result<DataFrame, DataFusionError> {
    let ctx = SessionContext::new();

    ctx.register_dataframe("mpesa_records", mpesa_df)?;
    ctx.register_dataframe("bank_records", bank_df)?;

    // Fast 1:1 & M:N Multi-Source Transaction Join
    let recon_sql = "
        SELECT 
            m.receipt_no AS mpesa_receipt,
            b.transaction_id AS bank_ref,
            m.amount,
            (m.amount - b.amount) AS variance
        FROM mpesa_records m
        INNER JOIN bank_records b
        ON m.receipt_no = b.mpesa_reference_code
        WHERE ABS(m.amount - b.amount) < 0.01
    ";

    ctx.sql(recon_sql).await
}

Key Architectural Advantages:

1. Zero-Cost Abstractions: Rust’s memory ownership model eliminates runtime Garbage Collection (GC) pauses during massive array-buffer operations.
2. Apache Arrow Memory Format: Apache DataFusion operates directly on Arrow columnar memory buffers, enabling SIMD vectorization and zero-copy memory transfers between ingestion stages and query execution.
3. Multi-Source Match Strategies: Supports 1:1 exact matching, Transitive matching across intermediate accounts, and M:N batch settlement grouping.

---

📄 2. Handling PDF Statement Parsing & LLM Schema Drift Protection

Bank statements are frequently delivered as non-standard PDF tables or CSV exports where column headers shift without notice (e.g. TransID changing to Receipt_No or Ref_Code).

┌───────────────────────────────────────┬────────────────────────────────────────────────────────┐
│ Ingestion Challenge                   │ FinOps R² Resolution Pattern                           │
├───────────────────────────────────────┼────────────────────────────────────────────────────────┤
│ Shifted CSV Column Headers            │ Dynamic LLM Schema Drift Protection mapping aliases.   │
├───────────────────────────────────────┼────────────────────────────────────────────────────────┤
│ Unstructured PDF Statement Tables     │ Rust PDF stream extraction into Arrow Column Buffers.  │
├───────────────────────────────────────┼────────────────────────────────────────────────────────┤
│ Out-of-Order Payment Callbacks        │ Double-Entry Audit Ledger & Maker/Checker verification.│
└───────────────────────────────────────┴────────────────────────────────────────────────────────┘

Dynamic Schema Normalization:

Before execution, incoming statement streams are processed by an LLM-assisted Schema Alignment layer that maps arbitrary header titles to standardized schema fields (transaction_id, amount, timestamp, party_b). This prevents formula crashes or missed matches when financial institutions alter export formats.

---

🔒 3. Audit Compliance: Double-Entry Immutable Ledgers

Speed means nothing without mathematical verification. FinOps R² enforces a strict Double-Entry Ledger Architecture where every matched or unmatched transaction produces an immutable audit record containing:

• Source Lineage Hash: Cryptographic hash of the raw ingested file row.
• Variance Calculations: Automatic detection of fee deductions or partial settlements.
• Maker/Checker Workspaces: Enforced approval workflows for manual variance resolution before posting to ERP ledgers.

---

📈 Summary

By combining Rust’s memory safety with Apache DataFusion’s columnar query engine, FinOps R² empowers finance teams to resolve millions of M-Pesa and bank transactions in seconds with zero manual spreadsheet entry.