Inter-Agency Settlement Clearing

Inter-agency settlement clearing is the sub-problem inside Revenue Reconciliation & Settlement that answers a deceptively simple question: when several operators share riders, corridors, and free transfers, who owes whom, and how little cash has to actually move to settle it? A regional network where a rider taps a city bus, a commuter rail line, and a private shuttle on one journey generates fractional obligations in every direction at once. This page builds the engine that turns those millions of fractional claims into a small, provable set of settlement transfers routed through a clearinghouse.

The obligations themselves come from two upstream sources that this component treats as inputs, not concerns of its own. Trip-level revenue splits arrive from fare rules — a cross-operator free transfer, for instance, originates as a payout event flagged by Transfer Window Logic, where the origin operator collected the base fare and a second operator carried the rider for free. Corridor-level splits arrive from proration. The clearing engine’s job is narrower and financial: aggregate every bilateral claim into a matrix, reduce that matrix to net positions, and emit settlement instructions that conserve every cent.

Architecture: From Operator Ledgers to Net Payouts

Each operator posts what it believes it is owed by, and owes to, every counterparty over a settlement period. Those directed claims form an obligation matrix — a square grid where cell $(i, j)$ is the gross amount operator $i$ owes operator $j$. Netting collapses the matrix into one signed net position per operator, and the clearinghouse converts positions into the fewest cash movements that make everyone whole.

The financial invariant the whole component defends is conservation. If the network collected a fare, the sum of every operator’s net position must be exactly zero — every debit is someone else’s credit. A settlement run that violates that by even a cent has either invented money or destroyed it, and the failure is almost always a rounding artifact from proration or a float that should have been a Decimal. The engine below is built so that invariant is asserted, not assumed.

Prerequisites & Environment

This engine targets Python 3.11+ and depends only on the standard library for the settlement math — decimal, dataclasses, collections, and datetime. Money never touches float; every amount is a Decimal of integer minor units (cents) so that summing thousands of claims is exact. Optional production dependencies are a database driver (psycopg[binary]>=3.1) for reading posted obligations out of the ledger and pandas>=2.1 only for the analyst-facing settlement report, never for the arithmetic.

Window thresholds and transfer eligibility are resolved before a claim ever reaches this engine. A cross-operator transfer, for example, is already priced and flagged by the time it posts here; the clearing engine consumes the resulting obligation, it does not re-derive the fare rule.

Core Implementation

The engine accumulates directed obligations into a matrix keyed on (debtor, creditor), then reduces the matrix to one signed net position per operator. It refuses to produce a settlement unless the positions sum to zero. Turning those net positions into the minimal set of cash transfers is a distinct algorithm handled in its own guide (linked below); this class stops at the net-position ledger and its conservation proof.

from __future__ import annotations

import logging
from collections import defaultdict
from dataclasses import dataclass, field
from datetime import datetime
from decimal import Decimal, InvalidOperation
from typing import Iterable, Mapping

logger = logging.getLogger("transit.settlement.clearing")

ZERO = Decimal("0")


class SettlementError(Exception):
    """Base exception for settlement-clearing failures."""


class UnknownCounterpartyError(SettlementError):
    """A claim references an operator absent from the participant registry."""


class ConservationError(SettlementError):
    """Net positions failed to sum to zero — money was invented or lost."""


@dataclass(frozen=True)
class Obligation:
    """A single directed claim: debtor owes creditor amount_cents for a period."""

    debtor: str
    creditor: str
    amount_cents: Decimal          # positive minor units
    period_start: datetime
    period_end: datetime
    source_ref: str                # trip_id, corridor_id, or dispute ticket
    kind: str = "trip"             # 'trip' | 'corridor' | 'transfer' | 'adjustment'


@dataclass
class NetPosition:
    operator: str
    gross_owed_to_others: Decimal  # sum of this operator's debits
    gross_owed_by_others: Decimal  # sum of this operator's credits
    net_cents: Decimal = field(init=False)  # credit positive, debit negative

    def __post_init__(self) -> None:
        self.net_cents = self.gross_owed_by_others - self.gross_owed_to_others


class ClearingEngine:
    """Aggregates directed obligations into a matrix and nets them to
    per-operator positions, defending a zero-sum conservation invariant."""

    def __init__(self, participants: Iterable[str]) -> None:
        self.participants: set[str] = set(participants)
        if len(self.participants) < 2:
            raise SettlementError("A settlement needs at least two participants.")
        # matrix[(debtor, creditor)] -> accumulated Decimal cents
        self._matrix: dict[tuple[str, str], Decimal] = defaultdict(lambda: ZERO)
        self._claim_count = 0

    def post(self, ob: Obligation) -> None:
        """Accumulate one obligation into the matrix, validating strictly."""
        for role, op in (("debtor", ob.debtor), ("creditor", ob.creditor)):
            if op not in self.participants:
                raise UnknownCounterpartyError(
                    f"{role} {op!r} (ref {ob.source_ref}) is not a registered participant."
                )
        if ob.debtor == ob.creditor:
            logger.debug("Skipping self-obligation for %s (ref %s)", ob.debtor, ob.source_ref)
            return
        if not isinstance(ob.amount_cents, Decimal):
            raise SettlementError(f"amount_cents must be Decimal, got {type(ob.amount_cents).__name__}.")
        try:
            amount = ob.amount_cents.quantize(Decimal("1"))
        except InvalidOperation as exc:
            raise SettlementError(f"Non-integer cents in ref {ob.source_ref}: {ob.amount_cents}") from exc
        if amount < ZERO:
            # A negative claim is a reversal; flip its direction rather than
            # carrying a signed amount through the matrix.
            self._matrix[(ob.creditor, ob.debtor)] += -amount
        else:
            self._matrix[(ob.debtor, ob.creditor)] += amount
        self._claim_count += 1

    def post_all(self, obligations: Iterable[Obligation]) -> None:
        for ob in obligations:
            self.post(ob)

    def gross_matrix(self) -> Mapping[tuple[str, str], Decimal]:
        """Return the accumulated bilateral gross obligations (read-only view)."""
        return dict(self._matrix)

    def net_positions(self) -> dict[str, NetPosition]:
        """Reduce the matrix to one signed net position per operator and
        assert the zero-sum conservation invariant before returning."""
        owed_to: dict[str, Decimal] = defaultdict(lambda: ZERO)   # this op is debtor
        owed_by: dict[str, Decimal] = defaultdict(lambda: ZERO)   # this op is creditor
        for (debtor, creditor), amount in self._matrix.items():
            owed_to[debtor] += amount
            owed_by[creditor] += amount

        positions = {
            op: NetPosition(
                operator=op,
                gross_owed_to_others=owed_to[op],
                gross_owed_by_others=owed_by[op],
            )
            for op in self.participants
        }

        total = sum((p.net_cents for p in positions.values()), ZERO)
        if total != ZERO:
            raise ConservationError(
                f"Net positions sum to {total} cents, not zero, across "
                f"{self._claim_count} claims. Refusing to settle."
            )
        logger.info(
            "Netted %d claims across %d operators; conservation holds.",
            self._claim_count, len(self.participants),
        )
        return positions

Three design choices carry the correctness guarantees. Reversals are folded by direction rather than sign, so the matrix only ever holds non-negative gross amounts and a late credit note nets cleanly against the original charge. Every net position is derived from the same matrix in a single pass, which is what makes the conservation assertion meaningful — if debits and credits were tallied from separate inputs, a zero sum would prove nothing. And the engine refuses to hand back positions that fail conservation, converting a silent settlement discrepancy into a loud, traceable exception carrying the claim count.

Schema Validation & Edge Cases

Clean upstream events do not guarantee a clean settlement. The failures that reach a clearinghouse are almost all about money identity — a cent that appears twice, a cent that belongs to no one, or a counterparty that cannot be paid.

• Rounding across proration. A corridor fare split three ways rarely divides into whole cents. If each leg rounds independently, the reconstructed obligations sum to a cent above or below the collected fare, and that stray cent lands in the net positions as a conservation break. The fix is upstream — prorate with the largest-remainder method so the parts provably re-sum to the total before any claim is posted — but the engine’s conservation assertion is the backstop that catches a proration bug that slipped through.
• Disputed legs. A contested claim must not silently drop out of settlement, because removing one side of a bilateral pair breaks conservation. Model a dispute as an adjustment obligation that reverses the contested amount into a holdback account represented as a synthetic participant, so the books still balance while the dispute is adjudicated out of band, then post the resolution as a fresh obligation in the next period.
• Missing counterparties. A claim naming an operator absent from the participant registry is an existential error, not a rounding one. The engine raises UnknownCounterpartyError rather than accruing money toward an account that cannot receive a payout; a typo in an operator code must halt the run, not settle into a void.
• Idempotent posting. Settlement runs get retried. Because obligations accumulate, replaying the same claim double-counts it. Deduplicate on source_ref at the boundary — carry a processed-ref set, or rely on the ledger’s own append-only key — so a redelivered batch is a no-op rather than a double charge.
• Period boundaries. Obligations are scoped to a half-open UTC window [start, end). A claim whose timestamp sits exactly on a boundary must belong to exactly one run; overlapping inclusive windows are the classic cause of a leg settled twice across consecutive days.
• Zero-activity operators. Every registered participant appears in the output with a zero net position even if it posted no claims. Omitting idle operators makes a downstream reconciliation report ambiguous about whether an operator was quiet or simply dropped.

Integration Pattern

The clearing engine is the join point between daily reconciliation and the permanent record, and it is deliberately stateless about how obligations were derived. Its inputs are posted claims; its outputs are net positions and, one step further, settlement instructions.

On the way in, the obligations it nets are the same figures that Daily Variance Reconciliation has already checked against tap ledgers and vendor settlement files. Netting a claim that failed variance reconciliation would settle a number nobody trusts, so the ordering is strict: reconcile the day, then clear the period. A variance that resolves to a genuine correction re-enters the engine as an adjustment obligation in the following run rather than mutating a claim already settled.

On the way out, both the gross matrix and the netted positions are written through Ledger & Audit Trail Design as an append-only, hash-chained record. That is what makes a settlement forensically defensible months later: an operator disputing a payout can be walked from its net position back through the matrix cell to the individual source_ref — the specific corridor split or the specific cross-operator transfer flagged by Transfer Window Logic — that produced it. The engine never overwrites a prior run; a correction is a new, signed entry.

Performance & Scale Considerations

A metro network is a handful to a few dozen operators, so the matrix itself is tiny — at most a few hundred cells. The scale problem is not the netting; it is the volume of obligations flowing into it.

• Aggregate before you net. Millions of trip-level claims collapse into an $N \times N$ matrix whose size depends only on operator count, not claim count. Accumulate into the matrix in a single streaming pass over the day’s obligations rather than materializing them all in memory; the matrix is the bounded state, the claim stream is not.
• Stream the source. Read posted obligations from the ledger with a server-side cursor and feed them through post_all, so peak memory is the matrix plus the idempotency ref set, not the full claim table. A day of a large network is tens of millions of rows and does not belong in a DataFrame for this step.
• Partition by period, not by operator. Netting is inherently all-to-all across operators, so it cannot be sharded by counterparty without a reduce step. Parallelize across independent settlement periods instead — each day nets on its own worker — and only serialize the final cross-period adjustments.
• Keep the matrix exact. Decimal arithmetic is slower than float, but at a few hundred cells the cost is irrelevant and the exactness is non-negotiable. Do not be tempted to net in float and quantize at the end; the whole point is a provable zero sum.
• Bound the idempotency set. Scope the processed-source_ref set to the settlement window plus a redelivery horizon; a claim can only be redelivered inside the broker’s or batch job’s retry window, so the set need not grow without limit.

Operational Checklist

1. Register participants first. Confirm every operator code in the incoming claims exists in the clearinghouse participant registry before the run starts; an unknown counterparty must halt, not accrue.
2. Gate on variance reconciliation. Only net obligations for a period whose daily variance has already closed clean, so settlement never runs on numbers still under review.
3. Assert conservation on every run. Treat a non-zero net-position sum as a hard failure that blocks payout, and alert with the claim count and the offending period.
4. Deduplicate on source_ref. Verify the idempotency guard drops redelivered batches so a retried settlement run does not double-count.
5. Model disputes as adjustments. Route contested legs into a holdback participant rather than deleting them, keeping the books balanced while adjudication proceeds.
6. Write both matrix and positions to the audit ledger. Persist the gross matrix alongside the netted positions so any payout is traceable back to its originating legs.
7. Reconcile net positions to the clearinghouse cash movements. After the bank confirms transfers, match settled amounts back to the emitted positions and flag any operator whose received amount differs from its net.
8. Retain period provenance. Store the settlement window bounds with the run so a claim can be proven to belong to exactly one period during a dispute.

Related deep-dive guides

• Prorating Shared-Corridor Revenue Across Operators — splitting a collected corridor fare across operators by agreed weights with the largest-remainder method, so the shares that become obligations sum exactly to the fare.
• Building a Clearinghouse Netting Batch in Python — reducing the bilateral gross matrix to the minimal set of net settlement transfers while proving total in equals total out.

Related

• Daily Variance Reconciliation — the upstream check that clears each day’s figures before they are netted for settlement.
• Ledger & Audit Trail Design — the append-only, hash-chained record where the matrix and net positions are persisted.
• Transfer Window Logic — where a cross-operator free transfer originates as the payout obligation this engine nets.
• Threshold Tuning Frameworks — where transfer windows and grace periods that shape cross-operator obligations are versioned.
• Compliance Reporting Automation — the reporting layer that consumes settled positions for regulatory filings.

Part of Revenue Reconciliation & Settlement.