Turning an opinion into a rating: methodology, evidence, and a public scored record

RuleScore is a deterministic, public opinion about contract language. To call it a rating, the grades have to predict something real: disputes, resolution delays, rule changes, refunds. We are building that evidence three ways, and we are being explicit about the limits of each.

First, a retrospective calibration: score each coded dispute case on the rules text as it was listed, blind to the outcome where possible, and read dispute/delay rates by grade band. Second — and this is the part that disciplines the claim — a CONTROL sample of non-disputed high-volume markets, the denominator without which a 'disputes-only' table proves nothing. We will publish no 'N times more likely' figure until that control set exists. Third, a forward log: live grades are hash-stamped with timestamps every scan, so in a few months there is a clean, pre-registered out-of-sample test that needs no historical reconstruction. That log is already accruing.

The honest constraint is that a clean historical backtest needs point-in-time rules text the firm largely does not have, because the point-in-time recorder was scoped down for data-rights reasons. We say so rather than imply a backtest we can't run. The calibration reads out with exact N, dates, method, and caveats once the control set is coded — published only if the relationship holds; if it does not, that is a model-improvement signal. The discipline is the credibility: a rating that can't be checked is an opinion with confidence.

Meanwhile the parts that are already real do the work: the methodology is fully public and reproducible, the dispute evidence is sourced and dated, the forward grade log runs daily, and the firm's own forecasts are registered and Brier-scored in the open — the first resolving at the June 17 FOMC. The rating is being earned in public, on the record, not asserted.