Smart Card Schema Mapping

Automated fare collection (AFC) telemetry arrives as heterogeneous, vendor-specific binary streams, and this page covers the translation stage that turns those raw taps into a single canonical event the rest of the Core Architecture & Fare Taxonomy pipeline can trust. Without a deterministic mapping layer, downstream revenue reconciliation fractures under inconsistent timestamp resolutions, fragmented product codes, and unnormalized origin-destination pairs — and every mis-mapped field compounds into a variance an analyst has to chase by hand. Schema mapping sits immediately after the security checkpoint and before validation: it reads authenticated sector bytes and emits a typed TapRecord that carries stable field names, UTC-anchored time, and reconciliation-ready keys. For a Python automation developer this is where vendor chaos is quarantined; for a revenue analyst it is the reason two operators’ cards settle into the same ledger schema.

The flow below traces a raw card dump from zero-copy extraction, through strict schema validation, into batched idempotent reconciliation:

Prerequisites & Environment

Schema mapping assumes it runs after the cryptographic checks defined by the AFC System Security Boundaries — the mapper never authenticates keys or verifies MACs itself; it consumes bytes that already cleared that boundary and refuses to promote any record whose upstream MAC flag is unset. The operating assumptions are:

A deliberate rule: the mapper produces no monetary values. It classifies (product_code, fare_zone_origin, fare_zone_destination) and leaves every price to the calculation engines, where fare math runs in Decimal. Keeping currency out of the parser means a byte-layout bug can never silently mis-price a journey.

Cryptographic and Operational Constraints

Schema parsers must never operate as blind byte extractors. Even though authentication happens upstream, the mapper is the last place to reject anything that looks tampered before it becomes a financial record. Production mapping logic must:

• Refuse records whose upstream MAC-valid flag is not set, rather than mapping them “optimistically” and hoping reconciliation catches it.
• Treat offline-validation and replay-counter flags as first-class fields, so a deferred or replayed tap is routed, not dropped.
• Strip or hash any PAN-equivalent identifier before it is emitted or logged, keeping the analytics and reconciliation queues free of raw media serials.
• Run in a memory-bounded context — a malformed length prefix must not be able to trigger unbounded heap growth during high-throughput ingestion.

Violating these constraints introduces compliance risk, corrupts settlement integrity, and exposes operators to regulatory penalties. The mapper’s contract is narrow on purpose: authenticate nothing, price nothing, but normalize everything and quarantine the rest.

Memory-Efficient Binary Extraction

Contactless transit deployments rarely share a single memory layout. Modern Mifare architectures require precise sector-level transaction log parsing, detailed in Parsing Mifare DESFire EV2 Tap Data in Python. Open-loop and multi-operator networks frequently rely on Calypso, where embedded cryptographic counters and multi-application contexts demand the specialized decoding routines covered in Decoding Calypso Smart Card Transaction Logs. Both feed the same canonical TapRecord, which is the whole point of a mapping layer.

To handle these layouts without exhausting RAM, parsers should lean on memoryview and the stdlib struct module for zero-copy slicing. Chunked iteration over raw dumps avoids loading an entire card image into memory, and generator-based extraction yields structured records on demand so the pipeline stays streaming from disk to sink. Reference the official Python struct documentation for format-string alignment and endianness handling — a single mis-declared endianness silently corrupts every timestamp on the card.

Core Implementation

The module below is a memory-efficient, error-hardened mapper: generator-based extraction, explicit exception routing (no bare except), Pydantic-strict validation, and idempotent transaction keys for downstream processing.

import logging
import struct
from collections.abc import Iterator
from datetime import datetime, timezone

from pydantic import BaseModel, ConfigDict, Field, ValidationError, field_validator

logger = logging.getLogger("afc.schema_mapper")


class AFCMappingError(Exception):
    """Raised when a sector chunk cannot be mapped to the canonical schema."""


class TapRecord(BaseModel):
    """Canonical, vendor-neutral tap event emitted by the mapping layer."""

    model_config = ConfigDict(strict=True, extra="forbid")

    transaction_id: str = Field(pattern=r"^[A-F0-9]{16}$")
    tap_timestamp_utc: datetime
    validator_device_id: str = Field(pattern=r"^VAL-\d{6}$")
    product_code: int = Field(ge=0, le=9999)
    fare_zone_origin: int = Field(ge=0, le=99)
    fare_zone_destination: int = Field(ge=0, le=99)
    validation_status: str = Field(pattern=r"^(APPROVED|DECLINED|OFFLINE_DEFERRED)$")
    is_concession: bool = False
    raw_mac_valid: bool = True

    @field_validator("tap_timestamp_utc", mode="before")
    @classmethod
    def normalize_utc(cls, v: datetime) -> datetime:
        """Anchor every validator clock to UTC; naive input is assumed UTC."""
        if v.tzinfo is None:
            return v.replace(tzinfo=timezone.utc)
        return v.astimezone(timezone.utc)


# Fixed sector layout (little-endian):
#   4-byte MAC flag, 8-byte TXN id, 4-byte UNIX ts, 2-byte validator,
#   1-byte product, 1-byte origin, 1-byte dest, 1-byte status, 1-byte concession
_RECORD_STRUCT = struct.Struct("<I8sIHBBBBB")
_STATUS_MAP = {0x01: "APPROVED", 0x02: "DECLINED"}


def parse_sector_chunk(
    raw_bytes: bytes, sector_offset: int, chunk_size: int = 128
) -> Iterator[dict]:
    """Zero-copy generator extracting raw tap dicts from a sector dump."""
    view = memoryview(raw_bytes)
    idx = sector_offset
    while idx + chunk_size <= len(view):
        chunk = view[idx : idx + chunk_size]
        try:
            (mac_flag, txn_id, unix_ts, val_id,
             prod, origin, dest, status, conc) = _RECORD_STRUCT.unpack_from(chunk, 0)
            yield {
                "transaction_id": txn_id.hex().upper(),
                "tap_timestamp_utc": datetime.fromtimestamp(unix_ts, tz=timezone.utc),
                "validator_device_id": f"VAL-{val_id:06d}",
                "product_code": prod,
                "fare_zone_origin": origin,
                "fare_zone_destination": dest,
                # Unknown status codes default to deferred, never dropped silently.
                "validation_status": _STATUS_MAP.get(status, "OFFLINE_DEFERRED"),
                "is_concession": bool(conc & 0x01),
                "raw_mac_valid": mac_flag == 0xDEADBEEF,
            }
        except struct.error as exc:
            logger.warning("Struct unpack mismatch at offset %d: %s", idx, exc)
        except (OverflowError, OSError, ValueError) as exc:
            # OverflowError/OSError: out-of-range UNIX timestamp on 32-bit fields.
            logger.error("Timestamp decode failure at offset %d: %s", idx, exc)
        finally:
            idx += chunk_size


def validate_and_map(raw_sector: bytes, offset: int = 0) -> Iterator[TapRecord]:
    """Validate each parsed dict against the TapRecord contract, streaming."""
    for record_dict in parse_sector_chunk(raw_sector, offset):
        if not record_dict["raw_mac_valid"]:
            # Upstream crypto boundary flagged this tap; never settle it.
            logger.warning("Dropping unauthenticated tap %s", record_dict["transaction_id"])
            continue
        try:
            yield TapRecord.model_validate(record_dict)
        except ValidationError as exc:
            logger.error("Schema validation failed: %s", exc.json())
            continue

validate_and_map is a generator end-to-end: nothing scales with the size of the card image, and a single malformed chunk logs a structured warning instead of crashing the run. The raw_mac_valid short-circuit encodes the trust boundary in code — an unauthenticated tap is dropped before it can be validated into a settlement-eligible record.

Schema Validation & Transit-Specific Edge Cases

Once extracted, records must clear the contract before they reach the reconciliation engine. Alignment with a structured Fare Zone Taxonomy Design is what makes the raw fare_zone_origin / fare_zone_destination integers meaningful — the mapper stores zone ids, and the taxonomy layer resolves them to the correct versioned tariff so a mid-year zone restructure never re-prices historical taps. Validation logic must explicitly handle:

• Timezone normalization. Validator real-time clocks drift; every timestamp is coerced to UTC at the field_validator, and a tolerance window $\Delta_{\text{skew}}$ (typically $\pm 5\text{s}$) is applied downstream before a tap is bound to a billing period.
• Null and unknown encodings. Unknown status bytes map to OFFLINE_DEFERRED rather than raising — a tap on an unfamiliar vendor firmware is deferred for reconciliation, not discarded.
• Idempotency. transaction_id combined with validator_device_id forms the composite key that makes an offline-sync retry a no-op instead of a double charge.
• Concession entitlement drift. A student or senior flag toggled mid-journey is recorded (is_concession) and logged for audit, never used to block the transaction at the mapping layer.
• Binary corruption. Partial writes during card ejection can truncate a sector log; the struct guard skips the malformed chunk and emits a warning rather than aborting the batch.

Integration Pattern

Schema mapping is one stage in a longer chain, and it hands off cleanly at both ends. Upstream, it receives only bytes that passed the AFC System Security Boundaries checkpoint; taps captured while a validator was disconnected arrive through the Fallback Routing Strategies cache, already stamped OFFLINE_DEFERRED, and flow through the identical mapper so offline and online events converge on one schema.

Downstream, the emitted TapRecord stream is consumed by the model layer described in Schema Validation Pipelines and then by pricing. The mapper deliberately keeps every time-based decision out of scope — capping and transfer eligibility are resolved later by Transfer Window Logic, which relies on the UTC-normalized tap_timestamp_utc this layer guarantees. That separation is the contract: the mapper promises stable identity and clean time, and the calculation engines own money and rules.

The batched sink keeps the whole chain streaming and bounded:

def batch_reconcile(
    records: Iterator[TapRecord], batch_size: int = 5000
) -> None:
    """Memory-bounded batch driver for the downstream reconciliation engine."""
    batch: list[TapRecord] = []
    for record in records:
        batch.append(record)
        if len(batch) >= batch_size:
            _commit_batch(batch)
            batch.clear()
    if batch:
        _commit_batch(batch)


def _commit_batch(batch: list[TapRecord]) -> None:
    """Idempotent upsert hook — replace with the real DB/queue client."""
    txn_ids = [r.transaction_id for r in batch]
    logger.info("Committing %d records; sample ids: %s", len(batch), txn_ids[:3])
    # INSERT ... ON CONFLICT (transaction_id, validator_device_id) DO NOTHING
    # Transfer-window, zone-tariff, and concession adjustments run downstream.

Performance & Scale Considerations

At metropolitan volume a fleet emits tens of thousands of taps per second, and the mapper must stay flat under that load:

• Bounded memory, always. Extraction is a generator over a memoryview; the only sized buffer is the reconciliation batch. Nothing grows with the number of cards processed. Precompile the format with a module-level struct.Struct (as above) to avoid re-parsing the format string per chunk.
• Chunk sizing. Keep reconciliation batches in the 5k–50k range: large enough to amortize the upsert round-trip, small enough that a failure checkpoints cheaply and the working set stays predictable.
• Parallelism caveat. Sector extraction is CPU-light but GIL-bound on the struct calls; scale with process-level workers partitioned by validator_device_id so idempotency keys never race across workers. Do not thread a single card image — the win is not there.
• Backpressure over buffering. When the settlement sink throttles, pause the generator rather than accumulating an unbounded in-memory backlog; a bounded queue between mapper and sink makes this explicit.

Operational Checklist

1. Verify the upstream MAC-valid flag on every record and drop unauthenticated taps before validation — never map “optimistically.”
2. Pin the sector layout to a versioned struct.Struct per vendor firmware; assert struct.calcsize against the documented record size at startup.
3. Normalize all timestamps to UTC at the mapping boundary; assign service-day and tariff snapshot strictly downstream.
4. Key every emitted record on (transaction_id, validator_device_id) so a re-ingested offline batch is a silent no-op.
5. Route unknown status bytes to OFFLINE_DEFERRED and alarm when the deferred rate spikes — it usually signals firmware drift, not fraud.
6. Hash or strip any PAN-equivalent identifier before it can reach a log line or analytics queue.
7. Keep the parser monetary-free; emit only zone and product codes and let the calculation engines price in Decimal.
8. Bound the reconciliation batch and prefer backpressure to buffering so a slow sink never grows the resident set.

Enforced this way — strict contracts, memory-bounded parsing, and reconciliation-ready keys — schema mapping turns raw tap telemetry from any vendor into deterministic, audit-grade records, enabling real-time fare analytics, automated dispute resolution, and multi-network settlement without manual forensic accounting.

Related

• Parsing Mifare DESFire EV2 Tap Data in Python — sector-level extraction for the closed-loop DESFire layout.
• Decoding Calypso Smart Card Transaction Logs — multi-application counters and open-loop decoding routines.
• AFC System Security Boundaries — the checkpoint whose authenticated bytes this mapper consumes.
• Fare Zone Taxonomy Design — resolving the raw zone ids the mapper emits into versioned tariffs.
• Schema Validation Pipelines — the model layer that consumes the canonical TapRecord stream.

Part of Core Architecture & Fare Taxonomy.