benchmark_pipeline#

Benchmark pipeline for systematic forecasting model evaluation.

Provides a standardized framework for running forecasting benchmarks across multiple targets, models, and evaluation metrics. Coordinates backtesting, evaluation, and analysis phases while managing parallel execution and result storage.

The benchmark pipeline follows a consistent workflow:

  1. Target acquisition from configurable providers

  2. Model training and backtesting for each target

  3. Evaluation against ground truth with configurable metrics

  4. Analysis and visualization of results

  5. Storage of results for comparison and reporting

Key components:

  • BenchmarkPipeline: Main orchestrator for benchmark execution

  • ForecasterFactory: Factory pattern for creating target-specific models

  • BenchmarkStorage: Pluggable storage backends for results

  • BenchmarkCallback: Extensible event handling for monitoring and customization

Functions#

read_evaluation_reports(targets, storage, ...)

Load evaluation reports for multiple targets from storage.

Classes#

BenchmarkContext(**data)

Context information passed to forecaster factories during benchmark execution.

BenchmarkPipeline(backtest_config, ...[, ...])

Orchestrates forecasting model benchmarks across multiple targets.