Announced Breaks Separate Conformal Reliability from Frequency Calibration

Announced Breaks Separate Conformal Reliability from Frequency Calibration

Karl Li, Columbia University – Accepted at NeurIPS 2026

---

Introduction

In sequential prediction settings—such as electricity markets, monetary policy forecasts, or any time‑series where a model may experience abrupt regime changes—maintaining valid predictive intervals is challenging. Traditional conformal inference guarantees per‑time coverage (reliability) but does not directly address frequency calibration, the long‑run proportion of errors matching a target miscoverage level. Karl Li’s paper proposes a novel reset rule triggered by an announced boundary (a pre‑declared point before a break occurs). This rule simultaneously targets two audits:

1. Conformal reliability – the expected deviation of the interval’s coverage from the nominal level at a fixed miscoverage (e.g., 0.1).
2. Frequency calibration – whether the empirical error rate over a closed window aligns with the target.

The reset mechanism leverages the post‑boundary empirical quantile, achieving minimax optimal per‑time rates for independent scores and offering provable frequency guarantees under bounded scores.

---

How the Reset Rule Works

1. Announced Boundary – Before the first potentially affected outcome, a boundary is announced. This signals that a break may occur, allowing the algorithm to prepare.
2. Per‑Time Audit – At each time step, the method computes a conformal set using the current calibration data. The reset replaces the calibration set with the empirical quantile of scores observed after the boundary.
3. Frequency Audit – Over a sliding window that starts at the boundary, the algorithm tracks the proportion of violations. The reset rule ensures that the deviation of this proportion from the target shrinks at a rate of \(O(1/n)\) when scores are bounded.
4. Recursion for Bounded Scores – When scores have a known bound, the reset can be applied recursively at each new announced break, preserving the frequency gap of order \(1/\text{count}\) while keeping the per‑time gap minimal.

The theoretical analysis shows that the post‑boundary empirical quantile is minimax for independent scores, meaning no other rule can uniformly achieve a lower worst‑case per‑time gap.

---

Empirical Highlights

• Synthetic Breaks: Across 50 forecasts after a large synthetic break, the reset rule’s per‑time gap was consistently lower than five competing adaptive conformal methods (including the authors’ own baselines). Every paired confidence band was below zero, indicating tighter coverage.
• Electricity Market Reform: On 68 unseen scores from a real‑world market reform, the reset’s confidence band was below zero against two methods, intersected zero against two (including adaptive conformal inference), and exactly matched zero against one method.
• Monetary Policy Decisions: Over the first 34 days after a policy shift, the reset’s pointwise band was above zero against all five benchmarks on 10 days, demonstrating superior frequency calibration during volatile periods.
• Multiple Announced Boundaries: Among nine announced boundaries, the reset rule cleared the smallest coverage difference detectable by its window, while all other methods exceeded the nominal miscoverage at the 50th forecast.

These results illustrate that the reset rule not only tightens intervals (better reliability) but also aligns long‑run error frequencies with the target, even under abrupt distributional changes.

---

Practical Implications

1. Operational Forecasting: Organizations that can pre‑announce potential regime shifts—e.g., power grid operators before scheduled maintenance—can adopt the reset rule to maintain trustworthy prediction intervals without sacrificing responsiveness.
2. Regulatory Compliance: In domains where calibrated risk estimates are mandated (financial stress testing, climate risk), the dual audit provides a clear, provable guarantee that both instantaneous and aggregate error rates meet standards.
3. Implementation Simplicity: The method requires only the empirical quantile of post‑boundary scores, avoiding complex weighting schemes or heavy hyper‑parameter tuning typical of adaptive conformal methods.
4. Extensibility: While the paper focuses on independent scores, the underlying principles could be extended to weakly dependent settings by adjusting the quantile estimation to account for autocorrelation.

---

Conclusion

Karl Li’s work bridges a gap between two central goals of sequential prediction—maintaining reliable per‑time coverage and achieving accurate frequency calibration—through a simple yet theoretically optimal reset rule triggered by announced boundaries. Empirical evaluations on synthetic breaks and real‑world datasets (electricity markets, monetary policy) demonstrate consistent advantages over existing adaptive conformal approaches. For practitioners facing abrupt regime changes, the reset rule offers a practical, provably optimal tool to keep predictive intervals both tight and well‑calibrated.