Integration with Volcano for Batch Scheduling

Integration with Volcano for Batch Scheduling

Volcano is a batch system built on Kubernetes. It provides a suite of mechanisms currently missing from Kubernetes that are commonly required by many classes of batch & elastic workloads. With the integration with Volcano, Spark application pods can be scheduled for better scheduling efficiency.

Volcano components

Before using Kubernetes Operator for Apache Spark, with Volcano enabled, user need to ensure Volcano has been successfully installed in the same environment, please refer Quick Start Guide for Volcano installation.

Install Kubernetes Operator for Apache Spark with Volcano enabled

Within the help of Helm chart, Kubernetes Operator for Apache Spark with Volcano can be easily installed with the command below:

helm repo add spark-operator

helm install my-release spark-operator/spark-operator \
    --namespace spark-operator \
    --set webhook.enable=true \
    --set batchScheduler.enable=true

Run Spark Application with Volcano scheduler

Now, we can run an updated version of spark application (with batchScheduler configured), for instance:

kind: SparkApplication
  name: spark-pi
  namespace: default
  type: Scala
  mode: cluster
  image: spark:3.5.1
  imagePullPolicy: Always
  mainClass: org.apache.spark.examples.SparkPi
  mainApplicationFile: local:///opt/spark/examples/jars/spark-examples_2.12-v3.5.1.jar
  sparkVersion: 3.5.1
  batchScheduler: volcano # Note: the batch scheduler name must be specified with `volcano`
    type: Never
    - name: test-volume
        path: /tmp
        type: Directory
    cores: 1
    coreLimit: 1200m
    memory: 512m
      version: 3.5.1
    serviceAccount: spark
      - name: test-volume
        mountPath: /tmp
    cores: 1
    instances: 1
    memory: 512m
      version: 3.5.1
      - name: test-volume
        mountPath: "/tmp"

When running, the Pods Events can be used to verify that whether the pods have been scheduled via Volcano.

Type    Reason     Age   From                          Message
----    ------     ----  ----                          -------
Normal  Scheduled  23s   volcano                       Successfully assigned default/spark-pi-driver to integration-worker2

Technological detail

If SparkApplication is configured to run with Volcano, there are some details underground that make the two systems integrated:

  1. Kubernetes Operator for Apache Spark’s webhook will patch pods’ schedulerName according to the batchScheduler in SparkApplication Spec.
  2. Before submitting spark application, Kubernetes Operator for Apache Spark will create a Volcano native resource PodGrouphere for the whole application. and as a brief introduction, most of the Volcano’s advanced scheduling features, such as pod delay creation, resource fairness and gang scheduling are all depend on this resource. Also, a new pod annotation named will be added.
  3. Volcano scheduler will take over all of the pods that both have schedulerName and annotation correctly configured for scheduling.

Kubernetes Operator for Apache Spark enables end user to have fine-grained controlled on batch scheduling via attribute BatchSchedulerOptions. BatchSchedulerOptions is a string dictionary that different batch scheduler can utilize it to expose different attributes. For now, volcano support these attributes below:

queueUsed to specify which volcano queue will this spark application belongs tobatchSchedulerOptions:
    queue: “queue1”
priorityClassNameUsed to specify which priorityClass this spark application will usebatchSchedulerOptions:
    priorityClassName: “pri1”


Was this page helpful?