Upgrading a Kubeflow Deployment

Upgrading your deployment to a later version of Kubeflow

Upgrading from Kubeflow v0.7.0 or later

Upgrading your Kubeflow deployment is supported if your deployment is v0.7.0 or later.

Prerequisites

  • Download the latest kfctl binary from the Kubeflow releases page.

  • Ensure that your Kubeflow namespace is annotated with the control-plane: kubeflow label. You can verify this by doing:

    kubectl get namespace $NAMESPACE -o yaml
    

    You should see something like:

    metadata:
      labels:
        control-plane: kubeflow
    

    If the label is not present, you can patch the namespace by doing

    kubectl label namespace $NAMESPACE control-plane=kubeflow
    
  • You must have a local Kubeflow application directory matching your current deployment. We’ll call this ${KF_DIR}.

    • This should be present if you deployed Kubeflow using the kfctl tool.
    • If you deployed Kubeflow using some other method (e.g. the Deployment UI), you will need to first create a local deployment directory:

      1. Follow the instructions for environment preparations.
        • Make sure that the ${PROJECT}, ${ZONE}, and ${KF_NAME} variables match exactly with your deployment.
        • For ${CONFIG_URI}, use the kfctl_gcp_iap file if your deployment uses Cloud IAP for authentication. Use the kfctl_gcp_basic_auth file if your deployment uses username and password.
      2. Create your local deployment files:

        mkdir -p ${KF_DIR}
        cd ${KF_DIR}
        kfctl build -V -f ${CONFIG_URI}
        

Upgrade instructions

Prepare your upgrade specification:

  1. Create an upgrade specification in the parent directory of your ${KF_DIR}. For example, to upgrade a v0.7.0 deployment to v0.7.1, use this specification. Your directory structure should look like:

    ${PARENT_DIR}
    |----${KF_DIR}
    |----kfctl_upgrade_spec.yaml
    
  2. Modify the contents of the upgrade specification for your deployment:

    apiVersion: kfupgrade.apps.kubeflow.org/v1alpha1
    kind: KfUpgrade
    metadata:
      name: kf-upgrade-v0.7.1
    spec:
      currentKfDef:
        # Replace with the name of your Kubeflow app
        name: kubeflow-app
        version: v0.7.0
      newKfDef:
        # Replace with the name of your kubeflow app
        name: kubeflow-app
        version: v0.7.1
      # Replace this with the path to the KfDef that you are upgrading to
      baseConfigPath: https://raw.githubusercontent.com/kubeflow/manifests/v0.7-branch/kfdef/kfctl_gcp_iap.0.7.1.yaml
    

If you don’t need to change any kustomize parameter values in your configuration, run the apply command now to upgrade your deployment:

kfctl apply -f ${UPGRADE_SPEC} -V

Alternatively you can follow these steps to change the configuration before applying the upgrade:

  1. Run a build command:

    kfctl build -f ${UPGRADE_SPEC} -V
    

    The above command creates a new Kubeflow application in the same directory, with a name in the form of a 7-character long hash value. The directory structure should look like:

    ${PARENT_DIR}
    |----${KF_DIR}
    |----${UPGRADE_DIR}
    |----kfctl_upgrade_spec.yaml
    

    You can examine and change the kustomize parameter values in ${UPGRADE_DIR}.

  2. Run the apply command to upgrade the deployment:

    kfctl apply -f ${UPGRADE_SPEC} -V
    

Upgrades from earlier versions of Kubeflow

For earlier versions, Kubeflow makes no promises of backwards compatibility or upgradeability. Nonetheless, here are some instructions for updating your deployment:

  1. Check your Kubeflow configuration directory (${KF_DIR}) into source control as a backup.

  2. Delete your existing Kubeflow cluster:

    kfctl delete -V 
    
  3. Download the kfctl v0.7.0 release from the Kubeflow releases page.

  4. Unpack the tar ball:

    tar -xvf kfctl_<release tag>_<platform>.tar.gz
    
  5. Update your kustomize manifests:

    export CONFIG_FILE=<the path to your Kubeflow config file>
    kfctl build -V -f ${CONFIG_FILE}
    

    The ${CONFIG_FILE} environment variable must contain the path to the Kubeflow configuration file in your ${KF_DIR} directory. For example, ${KF_DIR}/kfctl_k8s_istio.0.7.0.yaml or ${KF_DIR}/kfctl_existing_arrikto.yaml

  6. Re-apply any customizations that you need.

  7. Update the deployment:

     kfctl apply -V -f ${CONFIG_FILE}
    

Upgrading Kubeflow Pipelines

See the Kubeflow Pipelines upgrade guide.