Special Case: Importer Components

Unlike the other three authoring approaches, an importer component is not a general authoring style but a pre-baked component for a specific use case: loading a machine learning artifact from from a URI into the current pipeline and, as a result, into ML Metadata. This section assumes basic familiarity with KFP artifacts.

As described in Pipeline Basics, inputs to a task are typically outputs of an upstream task. When this is the case, artifacts are easily accessed on the upstream task using my_task.outputs['<output-key>']. The artifact is also registered in ML Metadata when it is created by the upstream task.

If you wish to use an existing artifact that was not generated by a task in the current pipeline, you can use a dsl.importer component to load the artifact from its URI.

You do not need to write an importer component; it can be imported from the dsl module and used directly:

from kfp import dsl

@dsl.pipeline
def my_pipeline():
    task = get_date_string()
    importer_task = dsl.importer(
        artifact_uri='gs://ml-pipeline-playground/shakespeare1.txt',
        artifact_class=dsl.Dataset,
        reimport=True,
        metadata={'date': task.output})
    other_component(dataset=importer_task.output)

In addition to an artifact_uri argument, you must provide an artifact_class argument to specify the type of the artifact.

The importer component permits setting artifact metadata via the metadata argument. Metadata can be constructed with outputs from upstream tasks, as is done for the 'date' value in the example pipeline.

You may also specify a boolean reimport argument. If reimport is False, KFP will check to see if the artifact has already been imported to ML Metadata and, if so, use it. This is useful for avoiding duplicative artifact entries in ML Metadata when multiple pipeline runs import the same artifact. If reimport is True, KFP will reimport the artifact as a new artifact in ML Metadata regardless of whether it was previously imported.