Revenue Reconciliation & Settlement

Revenue reconciliation and settlement is the accounting spine that turns the priced fare decisions leaving the Fare Rule Validation & Calculation Engines — plus the acquirer, processor, and vendor settlement files that arrive hours or days later — into an auditable revenue ledger that ties out to the cent. Every tap the engine priced is a promise of money; settlement is where that promise is confirmed against what the bank actually moved. When the two disagree and nobody catches it, the failure is expensive: revenue leaks silently, a year-end audit fails, an inter-agency payout gets disputed, and the numbers reported to the National Transit Database no longer reflect reality.

Ownership of that reconciliation is layered. Revenue assurance analysts answer to internal and external auditors for the integrity of the ledger; transit operations owns the daily variance and the awkward morning conversation when yesterday’s cash does not match yesterday’s taps; and the Python developers who build the reconciliation code own the invariants — exact arithmetic, idempotent re-runs, and an immutable audit trail — that let the other two sign their names to a number. This page maps the domain, the settlement topology, the core matching pattern, and the compliance and operational edges that separate a spreadsheet that mostly balances from a reconciliation system an auditor will accept.

Overview

Settlement is a convergence problem. One stream of expected amounts flows in from the calculation engines — one priced decision per tap, already deterministic and replayable. A second stream of settled amounts flows in from the outside world: acquirer settlement files for EMV contactless taps, processor reports for account-based tickets, and closed-loop vendor extracts for stored-value media. Reconciliation is the daily act of matching those two streams, explaining every difference, netting the obligations between operators, and writing the result to a ledger that can never be quietly rewritten.

The load-bearing property of the whole pipeline is that the ledger is the single source of truth for money and it is append-only. Nothing downstream — not a netting run, not an NTD extract — ever mutates a prior entry; corrections are new compensating entries that reference what they correct. That is what makes the difference between a reconciliation you can defend in an audit and a running total you can only hope is right.

Domain taxonomy

Settlement has its own vocabulary, and the same word carries different weight for an analyst, an acquirer, and a clearinghouse. Three amounts anchor everything. The expected amount is what the calculation engine priced for a tap — the agency’s claim. The settled amount is what the acquirer or vendor actually reports moving for that transaction. The cleared amount is what remains owed between operators after netting mutual obligations. A healthy day has expected equal to settled equal to cleared for the vast majority of lines; reconciliation exists entirely to find and explain the lines where they diverge.

Every reconciled line falls into exactly one variance class. The classification is the product of a day’s work, and it drives who does what next.

Underneath the amounts sit the entities the reconciliation code reasons over. An obligation is a directed debt from one operator to another for a shared journey; netting collapses the mutual obligations in a clearing batch into a single directed transfer per operator pair. The atomic unit written to the ledger is the ledger entry: an immutable, hash-chained record binding one reconciliation decision to its inputs.

The direction of dependency matters as much as it does upstream in the engine. An ExpectedFare is derived and immutable, a SettledLine is externally given and immutable, and a ReconciliationLine binds a specific expected claim to a specific settled row and the tolerance and rule version used to match them, so the decision can be replayed months later during a dispute. The zone and product vocabulary the reconciler groups by is defined once upstream — the reconciliation core only reads fare products, it never invents them.

System architecture

The reconciliation system is not one service but a chain of narrow stages, each replayable from its input, mirroring the discipline that governs the calculation engines. Two clocks drive it: a heavy daily batch that runs after the settlement cutoff, and a lighter intraday flow that ingests files and stages expected fares as they land so the morning batch is mostly I/O-free.

The daily batch is the authoritative pass. After the settlement-day cutoff, it pulls the frozen set of expected fares for the settlement date, loads every settlement file that has arrived for that date, matches them line by line, classifies each result, runs inter-agency netting over the matched obligations, and appends the outcome to the ledger. The intraday flow does the un-glamorous preparation: it validates and decrypts incoming vendor files, deduplicates settlement rows, and buffers late arrivals so the batch sees a clean, deduplicated input set rather than a pile of overlapping extracts.

Each stage preserves the input that produced it. A ReconciliationLine carries a snapshot of both amounts and the matching parameters; a ClearingResult carries the obligation set it collapsed. That property is what lets an analyst re-derive any downstream figure from first principles instead of trusting a total whose provenance has been lost. The matching stage keys expected against settled on a composite of settlement batch, operator, and fare product — the same grain the acquirer files are cut on — so a mismatch is localized to a small bucket rather than smeared across the whole day.

The split between an intraday flow and a daily batch is deliberate, and it mirrors the validation-versus-calculation separation upstream. Ingestion is stateful and messy: it deals with decryption, deduplication, schema drift in vendor extracts, and files that dribble in over a settlement window. Matching is a pure function that should never have to care about any of that. By pushing all the I/O and cleanup into the intraday stage, the nightly batch becomes a deterministic, side-effect-light computation over frozen inputs — which is exactly the shape that makes it replayable and testable. The expected fares arrive continuously from the calculation engines and are staged as they land, so by cutoff the batch is dominated by matching arithmetic rather than data movement.

One consumer sits directly on the reconciliation output and one sits a stage later. Daily variance reporting reads the classified lines the instant the batch completes, surfacing the short-settled and unmatched buckets to analysts before the business day starts. Inter-agency clearing reads the matched obligations, nets them, and writes its own ledger entries. Both consumers treat the reconciliation result as read-only; neither reaches back to mutate a classification, because a variance, once written, is evidence.

Core implementation pattern

The heart of the section is the ReconciliationEngine: it indexes expected fares by a composite settlement key, walks the settled lines from an acquirer file, matches each against the expected claim, computes the variance with Decimal, and classifies the line. It is deliberately pure — it takes two input sets and returns a list of decisions plus the unmatched remainder — so it can run identically in the nightly batch and in a replay. Money is Decimal throughout, tolerance is an explicit minor-unit band rather than a float epsilon, failures raise a typed exception rather than returning a sentinel amount, and every decision is logged with structured context.

from __future__ import annotations

import logging
from collections import defaultdict
from dataclasses import dataclass, field
from datetime import date
from decimal import Decimal, InvalidOperation
from enum import Enum
from typing import Iterable

logger = logging.getLogger("transit.reconciliation")

ZERO = Decimal("0.00")


class VarianceClass(str, Enum):
    MATCHED = "MATCHED"
    SHORT_SETTLED = "SHORT_SETTLED"
    OVER_SETTLED = "OVER_SETTLED"
    UNMATCHED_EXPECTED = "UNMATCHED_EXPECTED"
    UNMATCHED_SETTLED = "UNMATCHED_SETTLED"


class ReconciliationError(Exception):
    """Raised when a line cannot be reconciled deterministically."""


@dataclass(frozen=True)
class SettlementKey:
    """The grain acquirer files are cut on; expected and settled share it."""
    settlement_batch: str
    operator_id: str
    fare_product: str


@dataclass(frozen=True)
class ExpectedFare:
    key: SettlementKey
    transaction_id: str
    expected_amount: Decimal      # minor-unit precise, never float
    settlement_date: date


@dataclass(frozen=True)
class SettledLine:
    key: SettlementKey
    transaction_id: str
    settled_amount: Decimal
    file_id: str


@dataclass(frozen=True)
class ReconciliationLine:
    key: SettlementKey
    transaction_id: str
    expected_amount: Decimal
    settled_amount: Decimal
    variance: Decimal             # settled - expected, signed
    variance_class: VarianceClass


class ReconciliationEngine:
    """Matches expected fares to settled lines and classifies each variance."""

    def __init__(self, tolerance: Decimal = ZERO) -> None:
        # Tolerance is an explicit minor-unit band (e.g. Decimal("0.00")
        # for exact match, Decimal("0.01") to absorb one-cent rounding).
        if tolerance < ZERO:
            raise ValueError("tolerance must be non-negative")
        self.tolerance = tolerance

    def _index_expected(
        self, expected: Iterable[ExpectedFare]
    ) -> dict[tuple[SettlementKey, str], ExpectedFare]:
        index: dict[tuple[SettlementKey, str], ExpectedFare] = {}
        for row in expected:
            idx = (row.key, row.transaction_id)
            if idx in index:
                # Two expected rows for one transaction is a pricing bug,
                # not a reconciliation input — fail loudly rather than pick one.
                raise ReconciliationError(
                    f"duplicate expected fare for {row.transaction_id}"
                )
            index[idx] = row
        return index

    def _classify(self, variance: Decimal) -> VarianceClass:
        if abs(variance) <= self.tolerance:
            return VarianceClass.MATCHED
        return (
            VarianceClass.OVER_SETTLED if variance > ZERO
            else VarianceClass.SHORT_SETTLED
        )

    def reconcile(
        self,
        expected: Iterable[ExpectedFare],
        settled: Iterable[SettledLine],
    ) -> tuple[list[ReconciliationLine], list[ExpectedFare]]:
        """Return classified lines plus expected fares that never settled.

        Idempotent: the same inputs always yield the same output, so a
        re-run of a settlement batch overwrites rather than double-counts.
        """
        index = self._index_expected(expected)
        matched_keys: set[tuple[SettlementKey, str]] = set()
        lines: list[ReconciliationLine] = []

        for line in settled:
            idx = (line.key, line.transaction_id)
            exp = index.get(idx)
            try:
                if exp is None:
                    # Settlement row with no priced decision behind it.
                    variance = line.settled_amount
                    vclass = VarianceClass.UNMATCHED_SETTLED
                    expected_amount = ZERO
                else:
                    matched_keys.add(idx)
                    expected_amount = exp.expected_amount
                    variance = line.settled_amount - expected_amount
                    vclass = self._classify(variance)
            except (InvalidOperation, TypeError) as exc:
                logger.error(
                    "reconcile.line.failed",
                    extra={"transaction_id": line.transaction_id, "error": str(exc)},
                )
                raise ReconciliationError(str(exc)) from exc

            lines.append(
                ReconciliationLine(
                    key=line.key,
                    transaction_id=line.transaction_id,
                    expected_amount=expected_amount,
                    settled_amount=line.settled_amount,
                    variance=variance,
                    variance_class=vclass,
                )
            )

        # Expected fares with no matching settlement row: revenue we booked
        # but the bank has not (yet) confirmed.
        unmatched_expected = [
            exp for idx, exp in index.items() if idx not in matched_keys
        ]
        logger.info(
            "reconcile.batch.done",
            extra={
                "matched": len(matched_keys),
                "settled_lines": len(lines),
                "unmatched_expected": len(unmatched_expected),
            },
        )
        return lines, unmatched_expected

Because the inputs are frozen and the matching is keyed rather than positional, the engine is order-independent and idempotent: feeding the same acquirer file twice produces the same classification set, which is precisely what makes a batch re-run safe. The UNMATCHED_EXPECTED remainder is not an error — it is the day’s open receivable, the fares booked whose settlement has not landed, and it becomes the aging queue the Daily Variance Reconciliation job carries forward.

Once lines are classified and matched, the signed variances aggregate into per-operator obligations. For a settlement period, the net obligation from operator $i$ to operator $j$ is the difference of the gross flows each direction:

$$N_{ij} = \sum_{k} o^{(k)}_{ij} - \sum_{k} o^{(k)}_{ji}$$

Only the sign and magnitude of $N_{ij}$ survives netting: one directed transfer per operator pair, instead of thousands of individual obligations moving cash back and forth. That collapse is the job of the Inter-Agency Settlement Clearing stage.

Security & compliance boundaries

A reconciliation system holds the system of record for money and ingests files that, for card-based fares, sit inside cardholder-data scope. Its security posture is a first-class requirement across four overlapping regimes.

PCI-DSS scope for card settlement data. Acquirer settlement files reference EMV contactless transactions, and any file that carries a primary account number — even truncated — drags the ingestion host into PCI-DSS scope. The reconciliation core is kept out of that scope by matching on tokenized transaction identifiers, never on PAN; the tokenization and PAN-handling perimeter is defined in AFC System Security Boundaries, and the reconciler consumes only the token. Files that arrive with clear PAN are quarantined and truncated at the boundary before a single line reaches the matcher.

Encryption at rest. Settlement files and the ledger are both encrypted with AES-256 at rest, with keys held in a dedicated KMS or HSM and rotated on a fixed schedule following NIST SP 800-57 key-management guidance. Vendor files are decrypted only in memory on the ingestion worker; the plaintext extract is never written to disk unencrypted, and the encrypted original is retained as the settlement-day evidence.

Immutable audit trail and segregation of duties. Every reconciliation and clearing decision writes an append-only, hash-chained ledger entry; corrections are compensating entries, never edits. The person who runs a reconciliation batch cannot also approve a manual adjustment to its output — segregation of duties is enforced in code, so a single compromised or mistaken operator cannot both create and bless a variance write-off. The hash-chaining scheme that makes tampering detectable is the subject of Ledger & Audit Trail Design.

NTD reporting obligations. Agencies receiving federal funds report fare revenue to the National Transit Database on a fixed schedule, and those figures are audited against the ledger. Because the ledger is the sole upstream source for the report, the NTD extract is a deterministic query over immutable entries rather than a hand-assembled spreadsheet — reproducible on demand and defensible line by line. That automation is covered in Compliance Reporting Automation.

Operational resilience

Reconciliation runs against the messiest inputs in the whole fare stack — files that arrive late, partially, twice, or not at all — so resilience is designed in, not bolted on.

Idempotent batch re-runs. A settlement batch is identified by its settlement date and file set, and re-running it is a supported, routine operation: an analyst who receives a corrected acquirer file re-runs the day and the keyed matcher overwrites the prior result deterministically rather than double-counting it. Because each ledger write references the batch identity and the input hashes, a re-run supersedes the earlier entries with a visible, chained correction instead of silently mutating them.

Late and partial vendor files. Settlement files do not all arrive on time, and some arrive in fragments. The pipeline treats a settlement date as open until every expected file has landed: unmatched-expected lines age in a receivable queue, and when a late file arrives, only its keys are re-reconciled rather than the whole day. A partial file is matched for the lines it contains and its absent lines stay in the aging queue, so a half-delivered extract never looks like a wave of leakage.

Replayable reconciliation. The entire pass is a pure function of its inputs, and every input is retained encrypted, so any settlement day can be replayed from source months later — during an audit, a dispute, or a bug post-mortem — and reproduce the original ledger entries byte-for-byte. Replay uses the same code path as the live batch, which keeps the two from drifting: there is no separate “audit mode” that could disagree with production.

Edge cases & reconciliation pitfalls

Most reconciliation incidents trace to a small, recurring set of edges. Each has a deterministic remedy, and each is the kind of thing that nets out in aggregate and hides on a dashboard until an auditor finds it.

Duplicate settlement rows. Acquirers occasionally emit the same transaction twice across two files, or a re-sent file overlaps a prior one. Deduplicate on the settlement transaction identifier at ingestion, before matching; a duplicate that survives to the matcher becomes a spurious OVER_SETTLED variance and, if written, overstates revenue.

Timezone and cutoff boundaries. The settlement day is defined by the acquirer’s cutoff, not local midnight, and a tap near that boundary can be priced on one calendar date but settled on the next. Reconciliation keys on the settlement date the acquirer assigns, not the tap date, so a late-evening tap does not appear as unmatched-expected on day one and unmatched-settled on day two. Every comparison runs in a single declared timezone.

Refunds and adjustments. A refund is a negative settled line that may arrive days after the original fare, and a fare adjustment can change the expected amount after the fact. Both are modeled as new lines that reference the original transaction, never as edits to it, so the ledger shows the full history — original charge, later refund — rather than a silently reduced figure.

Rounding across proration. When a shared-corridor fare is split across operators, each operator’s share is rounded to the cent, and the residual must be assigned deterministically — largest-remainder to the highest-weight operator — so the shares sum back exactly to the gross. A proration that drops a cent per multi-operator journey becomes a real, compounding settlement discrepancy across millions of trips; the netting stage carries the residual explicitly rather than letting it evaporate.

Chargebacks arriving late. An EMV contactless chargeback can land days or weeks after the fare settled and the ledger closed for that period. It is never a retroactive edit to a closed day; it is a compensating entry in the current period that references the original transaction, keeping each period’s ledger immutable while still reflecting the reversal. The end-to-end handling of these disputes lives in Chargeback Dispute Automation.

Tolerance that hides leakage. A matching tolerance wide enough to absorb one-cent proration rounding is prudent; a tolerance wide enough to absorb an interchange fee is a way to launder revenue leakage into MATCHED. The tolerance band is a policy value, versioned and auditable, never a convenience constant tuned until the exceptions queue looks empty. A short-settlement that is genuinely a fee should be classified, explained, and posted as a fee — not quietly matched away — because the day the fee schedule changes, an over-wide tolerance stops flagging the difference.

Foreign and interoperable media. A rider from a neighboring system taps on media the local reconciler has never priced, producing an UNMATCHED_SETTLED line with no expected fare behind it. These are not errors; they are inbound interoperability revenue that resolves through the clearinghouse rather than the local expected-fare set. Routing them to the netting stage instead of the exceptions queue keeps analysts from chasing settlements that were never supposed to match locally.

Explore this section

• Daily Variance Reconciliation — the daily batch that matches expected fares against settlement files, classifies every variance, and ages the unmatched receivables that carry forward.
• Inter-Agency Settlement Clearing — collapsing mutual operator obligations into one netted transfer per pair, with deterministic residual assignment across prorated shares.
• Ledger & Audit Trail Design — the append-only, hash-chained ledger model that makes every reconciliation decision immutable and forensically replayable.
• Compliance Reporting Automation — generating NTD and audit reports as deterministic queries over the ledger rather than hand-built spreadsheets.

Related

• Fare Rule Validation & Calculation Engines — the upstream source of the priced, deterministic fare decisions that become this section’s expected amounts.
• Fraud Detection & Revenue Protection — where unmatched and over-settled variances feed anomaly detection, and chargebacks originate.
• AFC System Security Boundaries — the tokenization and PAN-handling perimeter that keeps the reconciliation core out of PCI-DSS scope.

Part of the transit-fare.org fare automation reference, alongside the sibling sections Core Architecture & Fare Taxonomy, Fare Data Ingestion & GTFS-RT Sync, Fare Rule Validation & Calculation Engines, and Fraud Detection & Revenue Protection.