Best Practices for Designing Components

Designing and writing components for Kubeflow Pipelines

This page describes some recommended practices for designing components. For an application of these best practices, see the component development guide. If you’re new to pipelines, see the conceptual guides to pipelines and components.

General component design rules

  • Design your components with composability in mind. Think about upstream and downstream components. What formats to consume as inputs from the upstream components. What formats to use for output data so that downstream components can consume it.
  • Component code must use local files for input/output data (unless impossible
    • for example, Cloud ML Engine and BigQuery require Cloud Storage staging paths).
  • Components must be pure - they must not use any outside data except data that comes through inputs (unless impossible). Everything should either be inside the container or come from inputs. Network access is strongly discouraged unless that’s the explicit purpose of a component (for example, upload/download).

Writing component code

Writing tests

  • Follow the general rules section so that writing the tests is easier.
  • Use the unit testing libraries that are standard for the language you’re using.
  • Try to design the component code so that it can be tested using unit tests. Do not use network unless necessary

  • Prepare small input data files so that the component code can be tested in isolation. For example, for an ML prediction component prepare a small model and evaluation dataset.

  • Use testing best practices.

    • Test the expected behavior of the code. Don’t just verify that “nothing has changed”:

      • For training you can look at loss at final iteration.
      • For prediction you can look at the result metrics.
      • For data augmenting you can check for some desired post-invariants.
  • If the component cannot be tested locally or in isolation, then create a small proof-of-concept pipeline that tests the component. You can use conditionals to verify the output values of a particular task and only enable the “success” task if the results are expected.

Writing a Dockerfile

  • Follow the Docker best practices

  • Structure the Dockerfile so that the required packages are installed first and the main component scripts/binaries are added last. Ideally, split the Dockerfile in two parts (base image and component code) so that the main component image build is fast and more reliable (not require network access).

Writing a component specification YAML file

For the complete definition of a Kubeflow Pipelines component, see the component specification. When creating your component.yaml file, you can look at the definitions for some existing components.

  • Use the {inputValue: Input name} command-line placeholder for small values that should be directly inserted into the command-line.
  • Use the {outputPath: Output name} command-line placeholder for output file locations.
  • Specify the full command line in ‘command:’ instead of just arguments to the entry point.