Fare Evasion Analytics

Fare evasion analytics is the batch, after-the-fact side of Fraud Detection & Revenue Protection: instead of scoring one tap in flight, it reads a whole period of gate-entry counts and validated paid taps and asks how much revenue quietly walked through the faregates. The output is not an alarm but a number — an estimated evasion rate per station, route, and time band, translated into a Decimal revenue-leakage figure with honest confidence bounds — that revenue analysts use to size recovery, justify gate hardening, and set enforcement rosters. This page builds that estimator as a reproducible pipeline rather than a spreadsheet, because a leakage figure that cannot be replayed cannot survive a budget review.

The scope here is deliberately aggregate. Real-time behavioural signals — a card tapping twice in impossible succession, a velocity spike — belong to Tap Pattern Anomaly Detection, which catches individual bad actors. Evasion analytics instead measures the population-level gap between how many people entered and how many paid, without ever identifying who evaded. The two are complementary: anomaly detection finds the fraud you can act on; evasion analytics measures the fraud you cannot yet see, and puts a defensible dollar value on closing it.

Architecture: From Counts to Sized Leakage

An evasion estimate is a subtraction with error bars around it. Two independent count streams — physical gate-entry counts from beam or optical sensors, and validated paid taps from the AFC ledger — are reconciled over the same station, direction, and time window. Their difference, once corrected for sensor bias and legitimate non-tap entries, is the estimated unpaid-entry count; multiplied by an average fare it becomes leakage; sliced by dimension it becomes an operational map of where the money is going.

The pipeline below shows how the two raw count streams converge into a corrected gap, a priced leakage figure, and its segmentation:

The central quantity is an evasion rate. If a station records $E$ physical entries over a period and $P$ validated paid taps, and a fraction $\alpha$ of entries are legitimately non-tapping (staff, wheelchair-gate escorts, courtesy passages) while the sensor systematically counts a fraction $\beta$ too few, the estimated evasion rate is

$$\hat{r} = 1 - \frac{P}{\left(\dfrac{E}{1-\beta}\right)\left(1-\alpha\right)}$$

Everything downstream — the leakage dollars, the segmentation, the confidence interval — is a consequence of that ratio and the uncertainty in the two adjustment factors it depends on.

The reason the method holds up is that the two count streams are collected by entirely independent systems. Gate-entry counts come from the physical sensor on the faregate — a beam, an optical counter, or the gate’s own cycle telemetry — and know nothing about fares. Paid taps come from the AFC ledger and know nothing about who physically passed. Because neither instrument can see the other, a systematic gap between them is hard to explain by anything except unpaid passage, once the two known biases are removed. That independence is also the method’s limit: it can measure the size of the gap but never name the rider in it, which is precisely the boundary that keeps evasion analytics an accounting exercise rather than a surveillance one.

Prerequisites & Environment

This component runs as an offline batch on Python 3.11+, using decimal for every money figure, statistics from the standard library for the interval math, and pandas>=2.1 only for the join-and-group-by stage over daily count tables. It reads two curated inputs — a gate-count table and the paid-tap ledger — that are already deduplicated and timezone-normalized upstream; the estimator does no ingestion of its own.

Because the estimate is only as trustworthy as $\alpha$ and $\beta$, both are treated as ranges, not point values. They are versioned alongside the gate hardware inventory so a sensor firmware change that shifts $\beta$ invalidates the affected periods rather than silently biasing a quarter of reporting. The calibration itself comes from periodic ground truth: a supervised manual count at a sample of gates over a known window gives an observed entry total to compare against the sensor’s, which fixes $\beta$, while a short observational study of who passes without tapping fixes $\alpha$ for that station type. These calibrations decay — a gate mechanism wears, a station’s rider mix shifts — so they carry an expiry, and an estimate built on a stale calibration is flagged rather than trusted.

Core Implementation

The estimator takes the reconciled counts for one station-period, applies the sensor and non-tap corrections, and returns an immutable estimate carrying both the point leakage and a low/high band derived from the plausible range of the correction factors. Every monetary value is Decimal; the rate math is kept in Decimal too so the point estimate and its bounds are reproducible to the cent.

from __future__ import annotations

import logging
from dataclasses import dataclass
from decimal import Decimal, ROUND_HALF_UP, InvalidOperation
from typing import Iterable

logger = logging.getLogger("transit.evasion_analytics")

CENTS = Decimal("1")  # leakage is reported in whole minor units


class EvasionEstimationError(Exception):
    """Base exception for evasion-rate estimation failures."""


class CountConsistencyError(EvasionEstimationError):
    """Raised when the input counts cannot yield a meaningful estimate."""


@dataclass(frozen=True)
class StationPeriod:
    """Reconciled counts for one station, direction and time window."""
    station_id: str
    route_id: str
    window_start_utc: str
    window_end_utc: str
    gate_entries: int          # raw physical entries from the sensor
    paid_taps: int             # settled paid taps from the AFC ledger
    avg_fare_cents: Decimal    # segment average fare, minor units
    # Calibration ranges: (low, nominal, high) for each correction factor.
    non_tap_allowance: tuple[Decimal, Decimal, Decimal]   # alpha
    sensor_reliability: tuple[Decimal, Decimal, Decimal]  # 1 - beta


@dataclass(frozen=True)
class EvasionEstimate:
    station_id: str
    route_id: str
    window_start_utc: str
    evasion_rate: Decimal          # nominal, 0..1
    unpaid_entries: int            # nominal corrected count
    leakage_cents: Decimal         # nominal leakage, minor units
    leakage_low_cents: Decimal     # conservative bound
    leakage_high_cents: Decimal    # aggressive bound
    flagged: bool                  # estimate outside a trustworthy range


class EvasionRateEstimator:
    """Batch estimator of fare-evasion rate and revenue leakage per segment."""

    def __init__(
        self,
        min_entries_for_estimate: int = 200,
        max_plausible_rate: Decimal = Decimal("0.60"),
    ) -> None:
        # Below this entry count a single-period ratio is statistical noise.
        self.min_entries = min_entries_for_estimate
        # An estimate above this is almost always a data fault, not real evasion.
        self.max_rate = max_plausible_rate

    def _corrected_entries(self, entries: int, reliability: Decimal, alpha: Decimal) -> Decimal:
        """Adjust raw sensor entries up for undercount, down for legitimate non-tap."""
        if reliability <= 0:
            raise CountConsistencyError("Sensor reliability must be positive.")
        # Scale up for the fraction the sensor missed, then keep only fare-liable entries.
        true_entries = Decimal(entries) / reliability
        fare_liable = true_entries * (Decimal("1") - alpha)
        return fare_liable

    def _rate(self, paid: int, fare_liable: Decimal) -> Decimal:
        if fare_liable <= 0:
            raise CountConsistencyError("Fare-liable entries resolved to zero or less.")
        rate = Decimal("1") - (Decimal(paid) / fare_liable)
        # Clamp: negative rates mean the sensor undercounted vs taps, not negative evasion.
        return max(Decimal("0"), rate)

    def _leakage(self, fare_liable: Decimal, paid: int, avg_fare: Decimal) -> Decimal:
        unpaid = fare_liable - Decimal(paid)
        if unpaid <= 0:
            return Decimal("0")
        return (unpaid * avg_fare).quantize(CENTS, rounding=ROUND_HALF_UP)

    def estimate(self, sp: StationPeriod) -> EvasionEstimate:
        try:
            if sp.gate_entries < 0 or sp.paid_taps < 0:
                raise CountConsistencyError("Counts cannot be negative.")
            if sp.avg_fare_cents <= 0:
                raise CountConsistencyError("Average fare must be positive.")

            alpha_lo, alpha_nom, alpha_hi = sp.non_tap_allowance
            rel_lo, rel_nom, rel_hi = sp.sensor_reliability

            # Nominal path.
            liable_nom = self._corrected_entries(sp.gate_entries, rel_nom, alpha_nom)
            rate_nom = self._rate(sp.paid_taps, liable_nom)
            leak_nom = self._leakage(liable_nom, sp.paid_taps, sp.avg_fare_cents)

            # Conservative bound: most legit non-tap, best sensor -> fewest unpaid entries.
            liable_low = self._corrected_entries(sp.gate_entries, rel_hi, alpha_hi)
            leak_low = self._leakage(liable_low, sp.paid_taps, sp.avg_fare_cents)

            # Aggressive bound: least legit non-tap, worst sensor -> most unpaid entries.
            liable_high = self._corrected_entries(sp.gate_entries, rel_lo, alpha_lo)
            leak_high = self._leakage(liable_high, sp.paid_taps, sp.avg_fare_cents)

            flagged = (
                sp.gate_entries < self.min_entries
                or rate_nom > self.max_rate
            )
            if flagged:
                logger.warning(
                    "Flagged estimate for %s window %s: rate=%.3f entries=%d",
                    sp.station_id, sp.window_start_utc, rate_nom, sp.gate_entries,
                )

            return EvasionEstimate(
                station_id=sp.station_id,
                route_id=sp.route_id,
                window_start_utc=sp.window_start_utc,
                evasion_rate=rate_nom.quantize(Decimal("0.0001")),
                unpaid_entries=int((liable_nom - Decimal(sp.paid_taps)).to_integral_value(ROUND_HALF_UP)),
                leakage_cents=leak_nom,
                leakage_low_cents=leak_low,
                leakage_high_cents=leak_high,
                flagged=flagged,
            )
        except (InvalidOperation, CountConsistencyError) as exc:
            logger.error("Estimate failed for %s: %s", sp.station_id, exc)
            raise

    def aggregate_leakage(self, estimates: Iterable[EvasionEstimate]) -> Decimal:
        """Sum nominal leakage across segments in Decimal, ignoring flagged rows."""
        total = Decimal("0")
        for est in estimates:
            if est.flagged:
                continue
            total += est.leakage_cents
        return total

Three decisions keep the figure defensible. The point estimate and both bounds run through the same correction algebra with only the factor values swapped, so the interval genuinely reflects calibration uncertainty rather than an arbitrary percentage padding. Negative rates are clamped to zero rather than reported, because a station where paid taps exceed corrected entries is a sensor or clock fault, not negative evasion. And flagged estimates are excluded from aggregate_leakage rather than dropped from the record, so an analyst can see the suspicious segment while it is kept out of the headline total.

The choice to model each correction factor as a (low, nominal, high) triple rather than a single number is what makes the output honest. A leakage figure quoted as a single dollar amount invites a false-precision argument in a budget meeting — someone will challenge the exact number and, when it moves under a plausible re-calibration, dismiss the whole estimate. Quoting $2,500 (range $1,900–$3,400) frames the conversation correctly: the uncertainty is stated up front, the decision is about whether even the conservative bound justifies action, and no single point can be picked apart. The conservative bound assumes the most favorable reading — the best sensor performance and the highest legitimate non-tap allowance — so a leakage figure that clears it is one an operator can defend on the record. Analysts who present only the nominal figure routinely lose the argument they should have won on the lower bound.

Schema Validation & Edge Cases

Clean counts are a precondition, but the ways two independently-collected count streams disagree are exactly where a naive subtraction invents or hides leakage.

• Sensor undercount and overcount. Beam sensors miss tailgaters who cross on one gate cycle; optical counters add phantom entries for luggage carts and prams. The sensor_reliability range models both directions — a reliability above 1.0 is legitimate for a sensor that overcounts — and the estimate is only as narrow as that range is calibrated.
• Legitimate non-tap groups. Staff passages, accessibility-gate escorts, police and emergency access, and free-travel scheme holders who use a swing gate all enter without a fare-liable tap. Set non_tap_allowance per station type: an interchange with a wide accessible gate has a materially higher $\alpha$ than a gated suburban platform, and applying a network average over-attributes their passages to evasion.
• Double counting. Paid taps must come from the settled ledger, never raw device logs, because a rider who taps, gets a read error, and taps again produces two device events but one settled fare. Counting raw taps deflates the apparent gap and understates leakage.
• Ungated and open-boarding stations. A station with no faregates has no entry count to difference against; it must be excluded from the gate-gap method entirely and estimated by a survey-based approach, not silently returned as zero evasion.
• Window alignment and timezone. The count window and the tap window must share one UTC boundary. At a rush-hour station a 15-minute skew moves thousands of entries across the boundary and manufactures a gap; align both to the same window_start_utc and reject any pair whose windows differ.
• Idempotency. Re-running a period must yield an identical estimate. Key each EvasionEstimate on (station_id, window_start_utc, sensor_calibration_version) so a recalibration produces a new, comparable row rather than overwriting history.

Integration Pattern

Evasion analytics sits between the operational fraud signal and the financial ledger, and it hands off in both directions. Its inputs — the validated paid taps — are the same settled events that Revenue Reconciliation & Settlement produces, so the estimator should read the reconciled ledger after daily variance has cleared, not the provisional feed. Running it against unreconciled taps means every settlement correction re-opens the evasion number, and the two reports never agree.

Downstream, a sustained high evasion rate at a specific station and time is a targeting signal, not just a KPI. Feed the segmented output to the enforcement roster and to Tap Pattern Anomaly Detection: the aggregate gap tells you where to look, and the real-time anomaly scorer tells you which media to act on once inspectors are deployed there. The segmentation itself is where the operational value concentrates — a network-wide evasion rate is a headline, but a rate broken down by station, entrance, and time band is a deployment plan. Two stations with identical daily leakage can demand opposite responses: one bleeding steadily across every hour points at a broken gate or a chronic tailgating geometry that hardware fixes, while one spiking only in the late-evening window points at staffing gaps that a roster change addresses more cheaply than a capital project. The leakage figure also flows into capital planning — a station whose aggressive-bound leakage exceeds the amortized cost of gate replacement is a hardware business case, and the confidence band is what keeps that case honest when finance pushes back on the point estimate. The single-station deep dive on estimating fare-evasion rates from gate-tap gaps is where the correction and interval math is worked end to end for one platform.

Performance & Scale Considerations

The estimator is cheap per segment and dominated entirely by the join that produces its inputs; a metro network is on the order of hundreds of stations times twenty-odd active hours times a handful of directions, which is tens of thousands of segment-rows per day, not millions.

• Group once, estimate many. Do the station-direction-hour group-by on the reconciled counts in a single vectorized pass, then iterate the resulting rows through estimate. The per-row Decimal math is exact but slow; there is no benefit to pushing it into the DataFrame, and doing so risks a silent float cast.
• Bound the join, not the estimate. Memory pressure lives in aligning the gate-count and tap tables. Stream both sorted by (station_id, window_start_utc) and merge; never cross-join a month of counts against a month of taps.
• Backfill by calibration version. When $\alpha$ or $\beta$ is recalibrated, only the affected station-periods need recomputation. Store the calibration version on each estimate so a backfill is a targeted rerun, not a full-history rebuild.
• Parallelize by station. Segments are independent, so shard the estimate stage by station_id across workers for a large backfill. The aggregation is a simple Decimal sum and can be reduced afterwards.
• Keep the raw counts. Store the pre-correction gate_entries and paid_taps alongside every estimate so a later calibration change can be re-derived without re-reading the source systems.

Operational Checklist

1. Read reconciled taps only. Confirm the estimator’s tap input is the settled ledger after daily variance has cleared, never the provisional feed.
2. Version the calibration. Pin non_tap_allowance and sensor_reliability per station type and stamp the version on every estimate; block any run against an uncalibrated sensor model.
3. Ground-truth the sensors. Schedule periodic manual counts at a sample of gates and feed the result back into the reliability range; alert when observed drift exceeds the modelled band.
4. Publish the interval, not just the point. Every leakage figure that leaves the team carries its low and high bound; a bare point estimate is not allowed in a financial report.
5. Quarantine flagged segments. Route flagged rows to a review queue and keep them out of the headline total until a human confirms the cause.
6. Exclude ungated stations explicitly. Maintain the list of open-boarding and ungated stops the gate-gap method cannot cover, and report them under a separate survey-based method.
7. Reconcile to settlement monthly. Confirm the sum of segment leakage moves consistently with the reconciled revenue trend; a divergence usually means a tap-source or window-alignment regression.
8. Retain raw counts. Store pre-correction counts with each estimate so recalibration is a re-derivation, not a re-collection.

Related deep-dive guides

• Estimating Fare Evasion Rates from Gate Tap Gaps — the full single-station method: correcting the gap, pricing the leakage in Decimal, and attaching a confidence interval.

Related

• Tap Pattern Anomaly Detection — the real-time counterpart that identifies which media to act on once analytics shows where to look.
• Revenue Reconciliation & Settlement — produces the settled tap ledger the estimator differences against.
• Chargeback & Dispute Automation — the other revenue-protection workflow, handling contactless disputes after settlement.
• Threshold Tuning Frameworks — where the flag thresholds and calibration ranges are versioned against historical logs.

Part of Fraud Detection & Revenue Protection.