observed_probability#

openstef_beam.metrics.observed_probability(y_true: ndarray[tuple[Any, ...], dtype[floating]], y_pred: ndarray[tuple[Any, ...], dtype[floating]]) → float[source]#

Calculate the observed probability (empirical quantile) of predicted values.

This function determines what quantile the predicted values correspond to based on the observed outcomes. For well-calibrated forecasts, a prediction at the p-th quantile should have approximately p fraction of observations below it.

Parameters:

  • y_true (ndarray[tuple[Any, ...], dtype[floating]]) – Observed values with shape (num_samples,).

  • y_pred (ndarray[tuple[Any, ...], dtype[floating]]) – Predicted values with shape (num_samples,). These are typically predictions from a specific quantile level.

Returns:

The empirical quantile level as a float in [0, 1]. This represents the fraction of observations that fall below the predicted values.

Return type:

float

Example

Check calibration of median forecasts:

>>> import numpy as np
>>> y_true = np.array([95, 105, 100, 110, 90])
>>> y_pred = np.array([100, 100, 100, 100, 100])  # Median predictions
>>> obs_prob = observed_probability(y_true, y_pred)
>>> round(obs_prob, 1)  # Should be close to 0.5 for well-calibrated median
0.4

Note

This metric is fundamental for evaluating forecast calibration. Systematic deviations from expected quantile levels indicate overconfident or underconfident uncertainty estimates.

Parameters:

  • y_true (ndarray[tuple[Any, ...], dtype[floating]])

  • y_pred (ndarray[tuple[Any, ...], dtype[floating]])

Return type:

float