Pipelines Standalone Deployment in GCP

Instructions to deploy Kubeflow Pipelines standalone to a cluster

As an alternative to deploying Kubeflow as a whole with many components including pipelines, you also have a choice to deploy only Kubeflow Pipelines. Follow the instructions below to deploy Kubeflow Pipelines standalone using the supplied kustomize manifests.

Knowledge about Kubernetes, kubectl and kustomize will help you understand this document better and be able to customize your deployment based on your needs.

Common prerequisites

These are common one time setups you need for all the instructions below:

Download kubectl CLI tool

Follow the kubectl installation guide.

You need kubectl version 1.14 or later, for native support of kustomize.

Configure kubectl to talk to your cluster

See the Google Kubernetes Engine (GKE) guide to configuring cluster access for kubectl.

Deploying Kubeflow Pipelines standalone to an existing cluster

  1. Deploy the latest version of Kubeflow Pipelines:

    export PIPELINE_VERSION=0.1.35
    kubectl apply -k github.com/kubeflow/pipelines//manifests/kustomize/env/dev?ref=$PIPELINE_VERSION
    
  2. Get the URL for the Kubeflow Pipelines UI :

    kubectl describe configmap inverse-proxy-config -n kubeflow | grep googleusercontent.com
    

Deploying Kubeflow Pipelines standalone from scratch

  1. Prepare a Kubernetes cluster:

    See the GKE guide to creating a cluster for Google Cloud Platform (GCP).

    Recommend using the following gcloud command to create a cluster that can run all pipeline samples:

    # The following parameters can be customized based on your needs.
    
    CLUSTER_NAME="kubeflow-pipelines-standalone"
    ZONE="us-central1-a"
    MACHINE_TYPE="n1-standard-2" # A machine with 2 CPUs and 7.50GB memory
    SCOPES="storage-rw,cloud-platform" # These scopes are needed for running some pipeline samples
    
    gcloud container clusters create $CLUSTER_NAME \
        --zone $ZONE \
        --machine-type $MACHINE_TYPE \
        --scopes $SCOPES \
        --num-nodes 2 \
        --max-nodes 5 \
        --min-nodes 2 \
        --enable-autoscaling
    

    Reference:

  2. Configure kubectl to talk to your newly created cluster. Refer to Configuring cluster access for kubectl.

  3. Deploy the latest version of Kubeflow Pipelines standalone to your cluster:

    export PIPELINE_VERSION=0.1.35
    kubectl apply -k github.com/kubeflow/pipelines//manifests/kustomize/env/dev?ref=$PIPELINE_VERSION
    

    Kubeflow Pipelines applications take a while (~3 minutes) to start.

  4. Get public URL of Pipelines UI and use it to access Kubeflow Pipelines:

    kubectl describe configmap inverse-proxy-config -n kubeflow | grep googleusercontent.com
    

Upgrade

  1. Configure kubectl to talk to your cluster. Refer to Configuring cluster access for kubectl.
  2. Upgrade to a version of Kubeflow Pipelines standalone you choose:

    export PIPELINE_VERSION=<version-you-want-to-upgrade-to>
    kubectl apply -k github.com/kubeflow/pipelines//manifests/kustomize/env/dev?ref=$PIPELINE_VERSION
    

    Check Kubeflow Pipelines github repo for available releases.

Customization

Customization can be done through kustomize overlays.

Note - The instruction below assume you installed kubectl v1.14.0 or later, which has native support of kustomize. To get latest kubectl, visit here

For the following instructions, first clone Kubeflow Pipelines repo, and use it as working directory.

Deploy on GCP with CloudSQL and Google Cloud Storage

This is recommended for production environments. See here for more details.

Change deployment namespace

To deploy Kubeflow Pipelines standalone in namespace FOO:

  1. Edit dev/kustomization.yaml or gcp/kustomization.yaml namespace section to FOO.
  2. Then run

    kubectl apply -k manifests/kustomize/env/dev
    # Or the following if using GCP Cloud SQL + Google Cloud Storage
    # kubectl apply -k manifests/kustomize/env/gcp
    

Disable the public endpoint

By default, the deployment installs an inverting proxy agent that exposes a public URL. If you want to skip installing it,

  1. Comment out the proxy component in the kustomization.yaml.
  2. Then run:

    kubectl apply -k manifests/kustomize/env/dev
    # Or the following if using GCP Cloud SQL + Google Cloud Storage
    # kubectl apply -k manifests/kustomize/env/gcp
    

The UI is still accessible by port-forwarding:

kubectl port-forward -n kubeflow svc/ml-pipeline-ui 8080:80

and open http://localhost:8080/.

Uninstall

You can uninstall Kubeflow Pipelines by:

  1. Configure kubectl to talk to your cluster. Refer to Configuring cluster access for kubectl.

  2. Uninstall Pipelines:

    export PIPELINE_VERSION=0.1.35
    kubectl delete -k github.com/kubeflow/pipelines//manifests/kustomize/env/dev?ref=$PIPELINE_VERSION
    

    Or if you deployed through kustomize:

    kubectl delete -k manifests/kustomize/env/dev
    # Or the following if using GCP Cloud SQL + Google Cloud Storage
    # kubectl delete -k manifests/kustomize/env/gcp
    

Best practices maintaining custom manifests

Maintain a repo for your manifests

Save the following to a source controlled repo.

File kustomization.yaml.

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
# Edit the following to change the deployment to your custom namespace.
namespace: kubeflow
# You can add other customizations here using kustomize.
# Edit ref in the following link to deploy a different version of Kubeflow Pipelines.
bases:
- github.com/kubeflow/pipelines//manifests/kustomize/env/dev?ref=0.1.35

How to deploy, upgrade and uninstall using the repo

Deploy: kubectl apply -k $YOUR_REPO

Upgrade:

  1. (Recommended) Back up your data storages for KFP.
  2. Edit ref=0.1.35 to a version you want to upgrade to.

    Check [Kubeflow Pipelines github repo](https://github.com/kubeflow/pipelines/releases) for available releases.
    
  3. Deploy: kubectl apply -k $YOUR_REPO.

Uninstall: kubectl delete -k $YOUR_REPO.

Further reading

Troubleshooting

Permission error installing Kubeflow Pipelines standalone to a cluster

Run:

kubectl create clusterrolebinding your-binding --clusterrole=cluster-admin --user=[your-user-name]

Pipeline samples that require “user-gcp-sa” secret

If sample pipeline requires a “user-gcp-sa” secret, you could create one by:

  1. First download the GCE VM service account token Document:

    gcloud iam service-accounts keys create application_default_credentials.json \
      --iam-account [SA-NAME]@[PROJECT-ID].iam.gserviceaccount.com
    
  2. Run:

    kubectl create secret -n [your-namespace] generic user-gcp-sa \
      --from-file=user-gcp-sa.json=application_default_credentials.json