TensorRTProvider#
- class openstef_foundation_models.inference.providers.TensorRTProvider(*, kind: Literal['tensorrt'] = 'tensorrt', device_id: int = 0, fp16: bool = True, engine_cache_dir: Path | None = None)[source]
Bases:
BaseConfigThe TensorRT execution provider (NVIDIA, ahead-of-time engine build).
FP16 with a persistent engine cache is the recommended production path on NVIDIA hardware: the first run pays the engine-build cost, subsequent runs load the cached engine.
- kind: Literal['tensorrt']
- device_id: int
- fp16: bool
- to_ort() OrtProvider[source]
Compile to an ONNX Runtime provider tuple.
- model_config: ClassVar[ConfigDict] = {'arbitrary_types_allowed': False, 'extra': 'ignore', 'protected_namespaces': ()}
Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].