MeanAbsoluteCalibrationErrorProvider

class openstef_beam.evaluation.metric_providers.MeanAbsoluteCalibrationErrorProvider(**data: Any) → None

Bases: MetricProvider

Provides quantile calibration metrics.

Computes the observed probability for each quantile, which should match the quantile level for well-calibrated forecasts. The metric quantifies this by computing the mean absolute error between observed probabilities and predicted quantiles across all samples.

Parameters:

data (Any)

property metric_names: frozenset[str]

Declared metric names that this provider produces.

Override in subclasses to enable eager metric-name validation (e.g. in the hyperparameter tuner).

compute_probabilistic(y_true: ndarray[tuple[Any, ...], dtype[floating]], y_pred: ndarray[tuple[Any, ...], dtype[floating]], quantiles: ndarray[tuple[Any, ...], dtype[floating]]) → dict[Quantile | Literal['global'], dict[str, Annotated[float, BeforeValidator(func=_convert_none_to_nan, json_schema_input_type=PydanticUndefined)]]]

Compute mean absolute calibration error for probabilistic forecasts.

Parameters:

• y_true (ndarray[tuple[Any, ...], dtype[floating]]) – True values, 1D array of shape (num_samples,).

• y_pred (ndarray[tuple[Any, ...], dtype[floating]]) – Predicted values, 2D array of shape (num_samples, num_quantiles).

• quantiles (ndarray[tuple[Any, ...], dtype[floating]]) – Quantiles used for prediction, 1D array of shape (num_quantiles,).

• y_true

• y_pred

• quantiles

Returns:

QuantileMetricsDict containing global mean absolute calibration error metric.

Return type:

dict[Union[Quantile, Literal['global']], dict[str, float]]

model_config: ClassVar[ConfigDict] = {'arbitrary_types_allowed': False, 'extra': 'ignore', 'protected_namespaces': ()}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].