Compile a Pipeline

Overview

To submit a pipeline for execution, you must compile it to YAML with the KFP SDK compiler.

In the following example, the compiler creates a file called pipeline.yaml, which contains a hermetic representation of your pipeline. The output is called an Intermediate Representation (IR) YAML, which is a serialized PipelineSpec protocol buffer message.

from kfp import compiler, dsl

@dsl.component
def comp(message: str) -> str:
    print(message)
    return message

@dsl.pipeline
def my_pipeline(message: str) -> str:
    """My ML pipeline."""
    return comp(message=message).output

compiler.Compiler().compile(my_pipeline, package_path='pipeline.yaml')

Because components are actually pipelines, you may also compile them to IR YAML:

@dsl.component
def comp(message: str) -> str:
    print(message)
    return message

compiler.Compiler().compile(comp, package_path='component.yaml')

You can view an example of IR YAML on GitHub. The contents of the file are not intended to be human-readable, however the comments at the top of the file provide a summary of the pipeline:

# PIPELINE DEFINITION
# Name: my-pipeline
# Description: My ML pipeline.
# Inputs:
#    message: str
# Outputs:
#    Output: str
...

Type checking

By default, the DSL compiler statically type checks your pipeline to ensure type consistency between components that pass data between one another. Static type checking helps identify component I/O inconsistencies without having to run the pipeline, shortening development iterations.

Specifically, the type checker checks for type equality between the type of data a component input expects and the type of the data provided. See Data Types for more information about KFP data types.

For example, for parameters, a list input may only be passed to parameters with a typing.List annotation. Similarly, a float may only be passed to parameters with a float annotation.

Input data types and annotations must also match for artifacts, with one exception: the Artifact type is compatible with all other artifact types. In this sense, the Artifact type is both the default artifact type and an artifact “any” type.

As described in the following section, you can disable type checking.

Compiler arguments

The Compiler.compile method accepts the following arguments:

Compiling for Kubernetes Native API Mode

When using Kubeflow Pipelines deployed in Kubernetes native API mode, you can compile pipelines directly to Kubernetes manifests using the KFP SDK. This mode stores pipeline definitions as Kubernetes Custom Resources and provides better integration with Kubernetes native tooling.

KubernetesManifestOptions Parameters

When using kubernetes_manifest_format=True, you can configure the generated Kubernetes resources using KubernetesManifestOptions:

Using KFP SDK with Kubernetes Native API Mode

The KFP SDK can compile pipelines directly to Kubernetes manifests for native deployment:

import kfp
from kfp import dsl
from kfp.compiler.compiler_utils import KubernetesManifestOptions

@dsl.component
def hello_world(name: str = "World") -> str:    
    print(f"Hello, {name}!")
    return f"Hello, {name}!"

@dsl.pipeline(
    name="hello-world-pipeline",
    description="A simple hello world pipeline"
)
def hello_world_pipeline(name: str = "Kubernetes Native") -> str:
    hello_task = hello_world(name=name)
    return hello_task.output

# Compile directly to Kubernetes manifests
kfp.compiler.Compiler().compile(
    pipeline_func=hello_world_pipeline,
    package_path="hello-world-pipeline-k8s.yaml",
    kubernetes_manifest_format=True,
    kubernetes_manifest_options=KubernetesManifestOptions(
        pipeline_name="hello-world-pipeline",
        pipeline_display_name="Hello World Pipeline",
        pipeline_version_name="hello-world-pipeline-v1",
        pipeline_version_display_name="Hello World Pipeline v1",
        namespace="kubeflow-user-example-com",
        include_pipeline_manifest=True
    )
)

Output Format

The compiled output will contain native Kubernetes manifests that can be directly applied to your cluster using kubectl apply -f hello-world-pipeline-k8s.yaml.

Generated Resources:

• Pipeline: Kubernetes Custom Resource (CRD) with API version pipelines.kubeflow.org/v2beta1
• PipelineVersion: Kubernetes Custom Resource (CRD) containing the actual pipeline specification

Example Output Structure:

apiVersion: pipelines.kubeflow.org/v2beta1
kind: Pipeline
metadata:
  name: hello-world-pipeline
  namespace: kubeflow-user-example-com
spec:
  displayName: Hello World Pipeline
---
apiVersion: pipelines.kubeflow.org/v2beta1
kind: PipelineVersion
metadata:
  name: hello-world-pipeline-v1
  namespace: kubeflow-user-example-com
spec:
  displayName: Hello World Pipeline v1
  pipelineName: hello-world-pipeline
  pipelineSpec:
    # Pipeline specification details
  platformSpec:
    # Platform specification details

Important Notes

• Namespace: Ensure the namespace in KubernetesManifestOptions matches where Kubeflow Pipelines is deployed
• API Version: The generated manifests use pipelines.kubeflow.org/v2beta1 API version
• Kubernetes Native Mode: This feature requires Kubeflow Pipelines to be deployed in Kubernetes Native API mode (version 2.14.0+)