Introduction

What is Kubeflow Pipelines?

Kubeflow Pipelines (KFP) is a platform for building and deploying portable and scalable machine learning (ML) workflows by using Docker containers.

KFP is available as a core component of Kubeflow or as a standalone installation. To quickly get started with a KFP deployment and usage example, see the Quickstart guide.

Objectives

The primary objectives of Kubeflow Pipelines are to enable the following:

  • End-to-end orchestration of ML workflows
  • Pipeline composability through reusable components and pipelines
  • Easy management, tracking, and visualization of pipeline definitions, pipeline runs, experiments, and ML artifacts
  • Efficient use of compute resources by eliminating redundant executions through caching
  • Cross-platform pipeline portability through a platform-neutral IR YAML pipeline definition

What is a pipeline?

A pipeline is the definition of a workflow with one or more steps called tasks. A task is defined by a single container execution and includes output parameters. Each task in a pipeline might also include input parameters. By specifying the output of one task as the input of another task, a pipeline author can form a computed acyclic graph (DAG) of tasks.

Pipelines are written in Python for an easy authoring experience, compiled to YAML for portability, and executed on Kubernetes for scalability.

What does using KFP look like?

At a high level, using KFP consists of the following steps:

  1. Author a pipeline with one or more components using the Python KFP SDK’s domain-specific language (DSL). You can author your own components or use prebuilt components provided by other authors.
  2. Compile the pipeline into a static representation (YAML) by using the KFP SDK’s DSL compiler.
  3. Submit the pipeline to run on the KFP backend. The KFP backend orchestrates the Kubernetes Pod creation and data passes, which are required to execute your workflow.
  4. View your runs, experiments, and ML artifacts on the KFP Dashboard.

Next steps

Feedback

Was this page helpful?