TensorRTProvider

class openstef_foundation_models.inference.TensorRTProvider(**data: Any) → None

Bases: BaseConfig

The TensorRT execution provider (NVIDIA, ahead-of-time engine build).

FP16 with a persistent engine cache is the recommended production path on NVIDIA hardware: the first run pays the engine-build cost, subsequent runs load the cached engine.

Parameters:

data (Any)

kind: Literal['tensorrt']

Discriminator tag for execution-provider type.

device_id: int

CUDA device index to run on.

fp16: bool

Enable FP16 precision for faster inference.

engine_cache_dir: Path | None

Directory to persist built TensorRT engines. When set, engine caching is enabled.

to_ort() → OrtProvider

Compile to an ONNX Runtime provider tuple.

Return type:

tuple[str, dict[str, object]]

Returns:

The TensorrtExecutionProvider with precision and engine-cache options.

model_config: ClassVar[ConfigDict] = {'arbitrary_types_allowed': False, 'extra': 'ignore', 'protected_namespaces': ()}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].