Fee Schedule — Data Standardization & Normalization Spec

Why this matters

The scoring batch put a number on extraction quality (avg correctness 95.2, completeness 99.5) — but it also exposed where the data model itself lets anomalies through. The single biggest root cause is that fee_amount is a free-text string. Across 23,375 fees we see the same economic value written a dozen ways, and the sign of a rebate depends on parentheses that aren't always applied.

Real value diversity we measured in one column (fee_amount)

Same story in the tier columns: fee_monthly_volume holds 0-24,999 contracts, 0 to 199,999, 0.025% to 0.035% ADV, 0 - 1,500,000 Users — four different metrics and units in one text field. And fee_participant has a catch-all "Other" (4,520 rows) sitting next to real roles.

1 · Canonical fee record (normalized)

Keep the current row identity, but split every semantically-overloaded string into typed columns with controlled vocabularies. Free text is preserved as *_raw for provenance, never used for computation.

2 · The amount grammar (parse once, at write)

A deterministic parser normalizes fee_amount_raw → (amount_value, amount_type, amount_unit, amount_basis, direction). Rules, in order:

1. Sign: parentheses or fee_charge_type/category = rebate/credit ⇒ direction=rebate, amount_value < 0. This makes the sign explicit instead of relying on punctuation.
2. Strip & canonicalize: remove $, spaces, thousands commas; normalize .00040 → 0.0004.
3. Unit/basis from trailing prose: per share→per_share, per executed equivalent share→per_share + basis=executed_equivalent_share, /month→per_month, % of Dollar Value→percent + basis=dollar_value, ADV→basis=adv.
4. Ranges (0.025% to 0.035%) ⇒ amount_type=tiered; expand into rows or store bounds.
5. Non-numeric (varies, No Charge): No Charge/Free/$0 ⇒ amount_value=0, amount_type=fixed. varies ⇒ amount_type=varies and a required formula/note or it fails validation.

Worked examples — raw → normalized

// rebate, buried sign
raw: "($0.25)"
→ amount_value: -0.25
  direction: "rebate"
  amount_type: "fixed"
  amount_unit: "per_contract"

// percentage + basis
raw: "0.15% of Dollar Value"
→ amount_value: 0.15
  amount_type: "percentage"
  amount_unit: "percent"
  amount_basis: "dollar_value"

// micro per-share
raw: "$0.000035 per executed
      equivalent share"
→ amount_value: 0.000035
  amount_unit: "per_share"
  amount_basis:
    "executed_equivalent_share"

// un-actionable → must justify
raw: "varies"
→ amount_type: "varies"
  amount_value: null
  formula: REQUIRED
  // else validation fails

// tier text → bounds
raw vol: "0-24,999 contracts"
→ tier_metric: "contracts"
  tier_lower: 0
  tier_upper: 24999

// recurring
raw: "$0.15/month"
→ amount_value: 0.15
  amount_unit: "per_month"

3 · Controlled vocabularies

Every categorical column resolves to a fixed enum. Unknown values are not silently stored — they route to a review queue. This is what removes the "Other" (4,520) and N/A bleed.

4 · Validation & anomaly-prevention rules

These run as CHECK constraints + a pre-delivery validator. Each maps to a real anomaly the scoring batch found.

5 · Why this makes it queryable & actionable

Once amounts are typed and signed, the questions Jane Street and the team actually ask become one-line SQL instead of string-parsing:

-- effective maker rebate per venue (rebates are already negative)
SELECT exchange, AVG(amount_value) FROM fee
WHERE fee_action='add' AND direction='rebate' AND amount_unit='per_contract'
GROUP BY exchange;

-- every fee that changed vs the prior delivered version
SELECT f.* FROM fee f JOIN fee p USING (atomic_key)
WHERE f.version_id=$new AND p.version_id=$prev AND f.amount_value <> p.amount_value;

-- delivery gate: block anything low-confidence or unvalidated
SELECT version_id FROM fee
GROUP BY version_id
HAVING MIN(correctness) < 60 OR bool_or(source_chunk_index IS NULL);

None of these are possible today because WHERE amount < 0 can't be trusted and "varies" breaks aggregation.

6 · Confidence-gated delivery

The two stored scores become an operational gate, not just a report:

• Green — correctness ≥ 90 & completeness ≥ 95 & 0 validation failures ⇒ auto-deliver.
• Amber — 60–90, or isolated V1/V5 flags ⇒ deliver with a flagged-rows manifest for the analyst.
• Red — any fabricated row (V2), completeness < 80 (V7), or correctness < 60 ⇒ hold, route to review before it reaches Jane Street.

From this batch: 76/80 versions are Green, ~4 Amber (GEMX, LTSE, NYSE National, EDGX equities), 0 Red once the empty-fetch was re-run.

7 · Rollout

1. Add typed columns alongside existing strings (non-breaking); backfill via the amount parser over all delivered versions.
2. Shadow-validate — run V1–V8 in report-only mode for one delivery cycle; tune plausibility bands + participant lookup.
3. Enforce — flip CHECK constraints on; wire the confidence gate into the SFTP delivery step.
4. Re-score — fee-schedule-scoring-sftp is idempotent, so re-running after normalization refreshes scores cleanly with no double-writes.