Run a Pipeline

Overview

Kubeflow Pipelines (KFP) provides several ways to trigger a pipeline run:

1. KFP Dashboard
2. KFP SDK Client
3. KFP CLI

Run Pipeline - KFP Dashboard

The first and easiest way to run a pipeline is by submitting it via the KFP dashboard.

To submit a pipeline for an immediate run:

1. Compile a pipeline to IR YAML.

2. From the “Pipelines” tab in the dashboard, select + Upload pipeline:

   

3. Upload the pipeline .yaml, .zip or .tar.gz file, populate the upload form, then click Create.

   

4. From the “Runs” tab, select + Create run:

   

5. Select the pipeline you want to run, populate the run form, then click Start:

   

Run Pipeline - KFP SDK Client

You may also programmatically submit pipeline runs from the KFP SDK client. The client supports two ways of submitting runs: from IR YAML or from a Python pipeline function.

For either approach, start by instantiating a kfp.Client():

import kfp

# TIP: you may need to authenticate with the KFP instance
kfp_client = kfp.Client()

To submit IR YAML for execution use the .create_run_from_pipeline_package() method:

#from kfp import compiler, dsl
#
#@dsl.component
#def add(a: float, b: float) -> float:
#   return a + b
#
#@dsl.pipeline(name="Add two Numbers")
#def add_pipeline(a: float, b: float):
#   add_task = add(a=a, b=b)
#
#compiler.Compiler().compile(
#    add_pipeline, 
#    package_path="./add-pipeline.yaml"
#)

kfp_client.create_run_from_pipeline_package(
    "./add-pipeline.yaml", 
    arguments={
        "a": 1,
        "b": 2,
    }
)

To submit a python pipeline function for execution use the .create_run_from_pipeline_func() convenience method, which wraps compilation and run submission into one method:

#from kfp import dsl
#
#@dsl.component
#def add(a: float, b: float) -> float:
#    return a + b
#
#@dsl.pipeline(name="Add two Numbers")
#def add_pipeline(a: float, b: float):
#    add_task = add(a=a, b=b)

kfp_client.create_run_from_pipeline_func(
    add_pipeline,
    arguments={
        "a": 1,
        "b": 2,
    }
)

Run Pipeline - KFP CLI

The kfp run create command allows you to submit a pipeline from the command line.

Here is the output of kfp run create --help:

kfp run create [OPTIONS] [ARGS]...

For example, the following command submits the ./path/to/pipeline.yaml IR YAML to the KFP backend:

kfp run create \
  --experiment-name "my-experiment" \
  --package-file "./path/to/pipeline.yaml"

For more information about the kfp CLI, please see:

• Use the CLI
• CLI Reference