Overview

Model serving overview

Kubeflow supports two model serving systems that allow multi-framework model serving: KFServing and Seldon Core. Alternatively, you can use a standalone model serving system. This page gives an overview of the options, so that you can choose the framework that best supports your model serving requirements.

Multi-framework serving with KFServing or Seldon Core

KFServing and Seldon Core are both open source systems that allow multi-framework model serving. The following table compares KFServing and Seldon Core. A check mark () indicates that the system (KFServing or Seldon Core) supports the feature specified in that row.

Feature Sub-feature KFServing Seldon Core
Framework TensorFlow sample docs
XGBoost sample docs
scikit-learn sample docs
NVIDIA TensorRT Inference Server sample docs
ONNX sample docs
PyTorch sample
Graph Transformers sample docs
Combiners Roadmap sample
Routers including MAB Roadmap docs
Analytics Explanations sample docs
Scaling Knative sample
GPU AutoScaling sample
HPA docs
Custom Container sample docs
Language Wrappers Python, Java, R
Multi-Container docs
Rollout Canary sample docs
Shadow
Istio

Notes:

  • KFServing and Seldon Core share some technical features, including explainability (using Seldon Alibi Explain) and payload logging, as well as other areas.
  • A commercial product, Seldon Deploy, supports both KFServing and Seldon in production.
  • KFServing is part of the Kubeflow project ecosystem. Seldon Core is an external project supported within Kubeflow.

Further information:

TensorFlow Serving

For TensorFlow models you can use TensorFlow Serving for real-time prediction. However, if you plan to use multiple frameworks, you should consider KFServing or Seldon Core as described above.

NVIDIA TensorRT Inference Server

NVIDIA TensorRT Inference Server is a REST and GRPC service for deep-learning inferencing of TensorRT, TensorFlow and Caffe2 models. The server is optimized to deploy machine learning algorithms on both GPUs and CPUs at scale.

You can use NVIDIA TensorRT Inference Server as a standalone system, but you should consider KFServing as described above. KFServing includes support for NVIDIA TensorRT Inference Server.