Compliance Reporting Automation

Regulatory revenue reporting is where an agency’s audited fare data leaves the building, and this page covers how to generate those filings deterministically from the ledger inside the Revenue Reconciliation & Settlement pipeline. In the United States the reference case is the National Transit Database (NTD): the annual and monthly revenue and ridership submissions that fund formula grants and that the FTA can audit line by line. A report that a human assembles by hand from spreadsheets is unreproducible and unfalsifiable; a report generated from the ledger by code that asserts its own totals tie back to the source is both. This page treats compliance reporting as a pure aggregation over an immutable ledger, not a quarterly fire drill.

The scope here is the aggregation-and-reconciliation layer: reading audited ledger entries, bucketing them by reporting period and fare category, and proving the report total equals the ledger total for that window before anything is filed. The end-to-end build of one such report — an NTD-style directly-generated fare revenue figure — is worked in the guide on generating NTD revenue reports from fare ledgers, which this component wraps in scheduling and validation.

The regulatory shape matters because it constrains the aggregation. NTD distinguishes directly-generated fare revenue — money the agency collected from riders at its own faregates and fareboxes — from purchased-transportation and other funding sources, and it wants that revenue broken out by mode and reconciled against reported ridership. An agency that reports a fare revenue figure it cannot trace back to individual audited transactions is one FTA data-quality review away from a finding, and formula-grant apportionment keys off exactly these numbers. So the automation’s job is not to produce a plausible number; it is to produce the number the ledger already contains, in the shape the form demands, with an assertion that the two are identical.

Architecture: From Ledger to Reconciled Report

A report is not produced by querying rows and summing them once. Entries are selected by period cutoff, aggregated by category, rendered into the regulatory shape, and then reconciled: the sum of the report’s line items must equal the independently computed ledger total for the same window, or the report is rejected rather than filed. The diagram below shows that path and the reconciliation gate that guards it.

The reconciliation invariant is a single equality the generator asserts before emitting. For a reporting window $P$, with line items $\ell$ over fare categories $C$ and the ledger control sum $L_P$:

$$\sum_{c \in C} \ell_c = L_P = \sum_{\substack{e \in \text{ledger} \\ e.occurred \in P}} e.amount$$

Both sides are computed in Decimal from the same selected entries by two independent code paths, so an equal result is strong evidence the aggregation dropped, duplicated, or misclassified nothing. When the two disagree, the difference is the audit lead — never a number to reconcile away by adjusting the report.

Prerequisites & Environment

This component targets Python 3.11+ and, like the ledger it reads, keeps its core on the standard library — decimal, datetime, and dataclasses — so a filing can be regenerated years later without a dependency archaeology project. It reads from the append-only store built in Ledger & Audit Trail Design and treats those entries as immutable inputs.

Because the report keys off occurred_utc, a correction booked after a period closes still lands in the period it economically belongs to. That restatement behavior is the hard part of the domain, and it is handled explicitly below rather than papered over.

One category deserves special attention before any code runs: what counts as revenue at all. A transfer credit that zeroes out a second-leg fare is not negative revenue to be netted against the mode that carried the rider for free — it is an allocation question the inter-agency layer resolves before the entry ever reaches this report. Refunds, on the other hand, genuinely reduce reported fare revenue and must net against the period the original fare occurred in. The category map that this component versions is therefore not cosmetic labeling; it encodes the policy decisions about which ledger kinds roll up to reported fare revenue, which are informational, and which belong to a different line of the submission entirely. Getting that map wrong does not fail the tie-out — both sides move together — which is precisely why the map is versioned, reviewed, and tested against known-good historical filings rather than trusted.

Core Implementation

The generator below selects entries by a half-open period, aggregates them by fare category into report lines with Decimal, computes an independent ledger control total over the same selection, and refuses to emit a report whose lines do not sum to that control total.

from __future__ import annotations

import logging
from dataclasses import dataclass
from datetime import datetime, timezone
from decimal import Decimal
from typing import Iterable, Mapping

logger = logging.getLogger("transit.compliance_report")


class ReconciliationMismatch(Exception):
    """Raised when report line items do not tie back to the ledger total."""


@dataclass(frozen=True)
class ReportableEntry:
    entry_key: str
    category: str          # e.g. 'adult_full', 'senior', 'transfer_credit'
    amount_minor: Decimal  # signed minor units
    occurred_utc: datetime


@dataclass(frozen=True)
class ReportLine:
    category: str
    amount_minor: Decimal
    entry_count: int


@dataclass(frozen=True)
class ComplianceReport:
    period_start: datetime
    period_end: datetime
    lines: tuple[ReportLine, ...]
    total_minor: Decimal
    ledger_control_minor: Decimal


class ComplianceReportGenerator:
    """Deterministic period aggregation that reconciles back to the ledger."""

    def __init__(self, report_timezone: timezone = timezone.utc) -> None:
        self._tz = report_timezone

    def _in_period(self, ts: datetime, start: datetime, end: datetime) -> bool:
        # Half-open [start, end): the entry at exactly `end` belongs to next period.
        ts_local = ts.astimezone(self._tz)
        return start <= ts_local < end

    def generate(
        self,
        entries: Iterable[ReportableEntry],
        *,
        period_start: datetime,
        period_end: datetime,
    ) -> ComplianceReport:
        if period_start.tzinfo is None or period_end.tzinfo is None:
            raise ValueError("period bounds must be timezone-aware")
        if period_end <= period_start:
            raise ValueError("period_end must be after period_start")

        buckets: dict[str, Decimal] = {}
        counts: dict[str, int] = {}
        control = Decimal("0")

        for entry in entries:
            if not isinstance(entry.amount_minor, Decimal):
                raise TypeError(f"amount for {entry.entry_key} is not Decimal")
            if not self._in_period(entry.occurred_utc, period_start, period_end):
                continue
            buckets[entry.category] = buckets.get(entry.category, Decimal("0")) + entry.amount_minor
            counts[entry.category] = counts.get(entry.category, 0) + 1
            control += entry.amount_minor  # independent running control total

        lines = tuple(
            ReportLine(category=cat, amount_minor=buckets[cat], entry_count=counts[cat])
            for cat in sorted(buckets)
        )
        report_total = sum((ln.amount_minor for ln in lines), Decimal("0"))

        if report_total != control:
            # Two independent sums of the same selection disagree: never file this.
            raise ReconciliationMismatch(
                f"report total {report_total} != ledger control {control} "
                f"for [{period_start.isoformat()}, {period_end.isoformat()})"
            )

        logger.info("report [%s,%s) total=%s lines=%d",
                    period_start.date(), period_end.date(), report_total, len(lines))
        return ComplianceReport(
            period_start=period_start,
            period_end=period_end,
            lines=lines,
            total_minor=report_total,
            ledger_control_minor=control,
        )

    def reconcile_against_ledger(
        self, report: ComplianceReport, ledger_period_totals: Mapping[str, Decimal],
    ) -> None:
        """Cross-check each report line against a categorized ledger total
        computed by the ledger module itself, not by this generator."""
        for line in report.lines:
            expected = ledger_period_totals.get(line.category, Decimal("0"))
            if line.amount_minor != expected:
                raise ReconciliationMismatch(
                    f"category {line.category}: report {line.amount_minor} "
                    f"!= ledger {expected}")

The design point is the two-path reconciliation. The line items are built by bucketing; the control total is accumulated independently in the same loop; and reconcile_against_ledger offers a third cross-check against category totals the ledger module computes on its own. A report only files when all three agree, so a classification bug surfaces as a raised ReconciliationMismatch, not a quietly wrong grant application.

Note what the generator deliberately does not do. It does not adjust, round, or reconcile-away a discrepancy; it raises. A compliance pipeline that silently corrects its own output to make the totals foot is worse than useless, because it converts a detectable data-quality problem into an undetectable one. The half-open period test lives in one place (_in_period) so the boundary convention cannot drift between the two sums, and the timezone is injected at construction so the same generator can produce reports for agencies on different reporting calendars without a fork. Everything about the class is arranged so that the only way to change the reported number is to change the ledger — which is append-only and hash-chained — or to change the period bounds, which are explicit arguments recorded on the emitted report.

Schema Validation & Edge Cases

Compliance reporting inherits a clean, verified ledger, but the reporting layer has its own hazards, and every one of them is a way to file a number that does not match reality.

• Period boundaries. The window is half-open [start, end) in a fixed reporting timezone. A fare tapped at exactly midnight on the boundary belongs to the next period, not both and not neither. Inclusive-both-ends is the classic double-count; inclusive-neither drops a day a year.
• Restatements. A prior-period correction is booked as a compensating entry with an occurred_utc inside the closed period. Regenerating that period’s report picks the correction up automatically because it keys on occurred_utc; the current period is unaffected. The filing process must then submit a restatement, not silently overwrite the original, so the audit trail of what was filed when is preserved.
• Late adjustments. A settlement received after a period closes but economically belonging to it is handled the same way: book it against the period it occurred in, regenerate, and restate. Never fold a late January adjustment into March just because that is when it arrived.
• Category mapping drift. The map from ledger kind/meta to NTD report lines is versioned. If the mapping changes mid-year, historical reports must regenerate against the mapping in force when the revenue occurred, exactly as tariff snapshots are resolved at the tap’s own timestamp in the AFC System Security Boundaries reconciliation engine.
• Verify before aggregate. Run the ledger’s chain verification before generating any report. Aggregating a tampered ledger produces a self-consistent but false filing; the whole value of the pipeline is that the source is provably intact.
• Revenue-to-ridership consistency. NTD wants fare revenue and unlinked passenger trips reported together, and a reviewer will notice if average fare per trip lurches between periods. The aggregation should surface implied average fare per mode as a derived check so an analyst catches a category-mapping regression — revenue landing on the wrong mode line — before the submission goes out, even when the grand total still ties.
• Zero-revenue categories. A category that had activity last period and none this period must still be representable, because a silently absent line reads as missing data rather than genuine zero. Emit the line with a Decimal("0") amount when the reporting form expects it, rather than dropping the key.

Integration Pattern

This component sits at the reporting end of the settlement pipeline and depends on the two components immediately upstream of it.

It reads directly from Ledger & Audit Trail Design: the report is an aggregation over that ledger’s entries, and the reconciliation gate asserts its totals equal the ledger’s own period balances. Keeping the report a pure function of the ledger is what makes it reproducible — regenerating a two-year-old NTD submission means replaying the same entries through the same code and getting the same number.

It also shares its ground truth with Daily Variance Reconciliation: variance reconciliation matches settlement files against the ledger daily, so by the time a period closes, the ledger the compliance report reads has already been reconciled against bank and processor totals day by day. The compliance layer therefore inherits a ledger whose external accuracy is continuously checked, and its own job narrows to deterministic aggregation and regulatory formatting rather than re-litigating whether the underlying revenue is right.

This separation of concerns is what makes the compliance report cheap to trust. If the daily reconciliation has already proven that each day’s ledger balance equals what the bank and card processor actually settled, then a period report that ties back to the ledger inherits that external validation transitively — the report equals the ledger, the ledger equals the money that moved. The compliance component never talks to a bank or a processor directly; it talks only to the ledger, and it depends on the daily layer having done the external matching. When the two are wired this way, an audit question about a reported figure decomposes cleanly: is the aggregation right (this component’s tie-out proves it) and is the ledger right (the daily reconciliation and the hash chain prove it). Neither question requires reconstructing anything from spreadsheets under audit pressure.

Performance & Scale Considerations

At an agency’s scale — tens of millions of fare entries per reporting year across modes and categories — report generation is an aggregation problem bounded by how the entries are read.

• Stream the selection. Aggregate in a single pass over a cursor filtered by occurred_utc, accumulating Decimal buckets, rather than loading a period into a DataFrame. Peak memory is the number of distinct categories, not the number of entries.
• Push the cutoff into storage. Filter the reporting window at the query layer with an index on occurred_utc, so the generator only ever sees candidate rows. A full-table scan per report does not survive contact with a multi-year ledger.
• Precompute closed-period control totals. Once a period closes and its report is filed, cache the signed control total alongside the ledger snapshot. A restatement then reconciles against a stored anchor instead of re-summing the entire history.
• Parallelize by mode, reconcile at the end. Categories and modes aggregate independently, so shard the pass across workers by mode and combine line items at the end — but always run the final tie-out on the combined total, never per shard, or a cross-shard misclassification hides.

Operational Checklist

1. Verify the chain first. Run ledger verification before every report generation and refuse to aggregate a ledger that does not validate.
2. Fix the period contract. Pin the reporting timezone and the half-open [start, end) convention in one place, and cover the midnight boundary with a test.
3. Key reports on occurred_utc. Assign every entry to the period it economically belongs to, independent of when it was booked.
4. Gate on tie-out. Never emit a report whose line items do not sum to the independent ledger control total; treat a mismatch as an audit lead, not a rounding nuisance.
5. Restate, never overwrite. Regenerate closed periods when corrections land, and file a restatement that preserves what was originally submitted and when.
6. Version the category map. Resolve historical reports against the fare-category mapping in force at the revenue’s own timestamp.
7. Sign and archive every filing. Store each emitted report with its control total and the ledger snapshot it was built from, so an FTA audit can be answered by regeneration, not reconstruction.
8. Keep money in Decimal. Assert Decimal on ingestion into the generator and forbid float in the reporting module.

Related deep-dive guides

• Generating NTD Revenue Reports from Fare Ledgers — building a National Transit Database directly-generated fare revenue report and tying it back to the ledger, including a late-adjustment restatement.

Related

• Ledger & Audit Trail Design — the audited, hash-chained ledger this component aggregates.
• Daily Variance Reconciliation — the daily check that keeps the ledger externally accurate before a period closes.
• Inter-Agency Settlement Clearing — cross-operator payouts that appear as categorized entries in the reported totals.
• AFC System Security Boundaries — the timestamp-snapshot pattern reused for versioned category resolution.

Part of Revenue Reconciliation & Settlement.