metrics_probabilistic#

Metrics for forecasts that predict probability distributions instead of single values.

Unlike deterministic forecasts that predict one value (e.g., “load will be 100 MW”), probabilistic forecasts predict a range of possible outcomes with their likelihoods (e.g., “80% chance load will be between 90-110 MW”). These metrics evaluate both how accurate these probability estimates are and how well-calibrated they are.

Key concepts:

  • Calibration: Do 90% prediction intervals actually contain the true value 90% of the time?

  • Sharpness: How narrow are the prediction intervals (more precise is better)?

  • Proper scoring: Metrics that reward honest probability estimates over gaming the system.

Functions#

crps(y_true, y_pred, quantiles[, sample_weights])

Calculate the Continuous Ranked Probability Score (CRPS) for probabilistic forecasts.

mean_absolute_calibration_error(y_true, ...)

Calculate the Mean Absolute Calibration Error (MACE) for probabilistic forecasts.

mean_pinball_loss(y_true, y_pred, quantiles)

Calculate the Mean Pinball Loss for quantile forecasts.

observed_probability(y_true, y_pred)

Calculate the observed probability (empirical quantile) of predicted values.

rcrps(y_true, y_pred, quantiles[, ...])

Calculate the relative Continuous Ranked Probability Score (rCRPS).