ObservedProbabilityProvider

class openstef_beam.evaluation.metric_providers.ObservedProbabilityProvider(**data: Any) → None

Bases: MetricProvider

Provides observed probability metrics.

Measures how often the actual value falls below the predicted value, which should match the quantile level for well-calibrated forecasts.

This metric is only useful as a global metric and not windowed.

Parameters:

data (Any)

property metric_names: frozenset[str]

Declared metric names that this provider produces.

Override in subclasses to enable eager metric-name validation (e.g. in the hyperparameter tuner).

compute_deterministic(y_true: ndarray[tuple[Any, ...], dtype[floating]], y_pred: ndarray[tuple[Any, ...], dtype[floating]], quantile: float) → dict[str, Annotated[float, BeforeValidator(func=_convert_none_to_nan, json_schema_input_type=PydanticUndefined)]]

Compute metrics for a single quantile prediction.

Must be implemented by subclasses that provide deterministic metrics (per quantile).

Parameters:

• y_true (ndarray[tuple[Any, ...], dtype[floating]]) – True values, 1D array of shape (num_samples,).

• y_pred (ndarray[tuple[Any, ...], dtype[floating]]) – Predicted values, 1D array of shape (num_samples,).

• quantile (float) – Quantile value for the prediction.

• y_true

• y_pred

• quantile

Return type:

dict[str, float]

model_config: ClassVar[ConfigDict] = {'arbitrary_types_allowed': False, 'extra': 'ignore', 'protected_namespaces': ()}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].