Compile a pipeline

You can compile your pipeline or component to intermediate representation (IR) YAML. The IR YAML definition preserves a static representation of the pipeline or component. You can submit the YAML definition to the KFP backend for execution, or deserialize it using the KFP SDK for integration into another pipeline. (View an example on GitHub).

Note: Pipelines as well as components are authored in Python. A pipeline is a template representing a multistep workflow, whereas a component is a template representing a single step workflow.

Compile a pipeline

You can compile a pipeline or a template into IR YAML using the Compiler.compile method. To do this, follow these steps:

1. Define a simple pipeline:

   from kfp import compiler
   from kfp import dsl
   
   @dsl.component
   def addition_component(num1: int, num2: int) -> int:
     return num1 + num2
   
   @dsl.pipeline(name='addition-pipeline')
   def my_pipeline(a: int, b: int, c: int = 10):
     add_task_1 = addition_component(num1=a, num2=b)
     add_task_2 = addition_component(num1=add_task_1.output, num2=c)
   

2. Compile the pipeline to the file my_pipeline.yaml:

   cmplr = compiler.Compiler()
   cmplr.compile(my_pipeline, package_path='my_pipeline.yaml')
   

3. Compile the component addition_component to the file addition_component.yaml:

   cmplr.compile(addition_component, package_path='addition_component.yaml')
   

The Compiler.compile method accepts the following parameters:

IR YAML

The IR YAML is an intermediate representation of a compiled pipeline or component. It is an instance of the PipelineSpec protocol buffer message type, which is a platform-agnostic pipeline representation protocol. It is considered an intermediate representation because the KFP backend compiles PipelineSpec to Argo Workflow YAML as the final pipeline definition for execution.

Unlike the v1 component YAML, the IR YAML is not intended to be written directly. To learn how to author pipelines and components in KFP v2 similar to authoring component YAML in KFP v1, see Author a Pipeline: Custom Container Components.

The compiled IR YAML file contains the following sections: