Banzai Cloud Logo Close
Home Benefits Blog Company Contact
Sign in
Author Laszlo Puskas

Apache Spark CI/CD workflow howto

TRY PIPELINE FOR FREE

Apache Spark on Kubernetes series:
Introduction to Spark on Kubernetes
Scaling Spark made simple on Kubernetes
The anatomy of Spark applications on Kubernetes
Monitoring Apache Spark with Prometheus
Apache Spark CI/CD workflow howto
Spark History Server on Kubernetes
Spark scheduling on Kubernetes demystified

Apache Zeppelin on Kubernetes series:
Running Zeppelin Spark notebooks on Kubernetes
Running Zeppelin Spark notebooks on Kubernetes - deep dive
CI/CD flow for Zeppelin notebooks

Apache Kafka on Kubernetes series:
Kafka on Kubernetes - using etcd

There is a newer version of Pipeline, 0.3.0 available. Accordingly, the value of this blogpost may have depreciated, so we highly recommend that you check Pipeline’s documentation for the latest howto.

Now that the first version of the Pipeline application has arrived, this blog post will quickly and concisely present how to a perform a basic CI/CD workflow setup using our platform. We have also prepared a few projects in our GitHub repo that can be used as starting points (see the links at the end of this post).

These example projects make use of resources (Spark clusters on k8s on AWS) provisioned on-demand during their CI/CD workflow. However, some workflow steps vary considerably (build details, run details).

This example will focus on Spark, but you should note that Pipeline is a generic microservice platform that’s not exclusively tied to use with big data workloads - is able to run any containerized, distributed workflow. Our next example in this series will involve databases, as we are moving towards JEE support, a key component of which is a persistent datastore.

To hook a Spark project into the CI/CD workflow of Banzai Cloud Pipeline, watch the video we’ve provided or follow the instructions below.

Getting Started

It’s easy to make the assumption that the source of a given Spark application is stored in GitHub and its user has an AWS account, since example CI flows run on EC2 instances provisioned on demand. Nevertheless, Pipeline is not tied to a particular cloud provider, but supports a variety of cloud and managed K8S providers (see Pipeline’s documentation).

Prerequisites

  1. Account on GitHub
  2. Repository on GitHub for the Spark application source code
  3. AWS account
  4. AWS EC2 key pair

Register OAuth on GitHub

Register an OAuth application on GitHub for the Pipeline CI/CD workflow.

For now, fill in the Authorization callback URL with a dummy value. This field will be updated once the Control Plane is up and running, using an IP address or DNS name.

Take note of the Client ID and Client Secret, as these are required to launch the Pipeline Control Plane.

Launch Pipeline Control Plane

The easiest way to run a Pipeline Control Plane is to use a Cloudformation template.

  • Fill in the following fields on the form:

    • Stack name
    • specify a name for the Control Plane deployment

    • Control Plane Instance Config
      • InstanceName - name of the EC2 instance that will host the Control Plane
      • ImageId - select the image id from the README
      • KeyName - specify your AWS EC2 key pair

    • Pipeline Credentials
      • Pipeline API Password - specify the password for accessing the Pipeline REST API exposed by the Pipeline PaaS. Please note your user name and password, as those will be required when setting secrets for GitHub repositories in the CI/CD workflow.

    • Banzai-Ci Credentials
      • Orgs - comma-separated list of Github organizations whose members to grant access to use Banzai Cloud Pipeline’s CI/CD workflow
      • GithubClient - GitHub OAuth Client Id
      • GithubSecret - Github OAuth Client Secret

    • Grafana Dashboard
      • Grafana Dashboard Password - specify a password for accessing the Grafana dashboard - defaults are application specific

    • Prometheus Dashboard
      • Prometheus Password - specify a password for accessing Prometheus, which collects cluster metrics

    • Advanced Pipeline Options
      • PipelineImageTag - specify 0.3.0 to use the current stable Pipeline release.

    • Slack Credentials

      • this section is optional. Complete this section to receive cluster-related alerts via Slack push notifications.
    • Alert SMTP Credentials

      • this section is optional. Fill it in to receive cluster-related alerts via email.
  • Finish the wizard to create a Control Plane instance.

  • Please note the PublicIP of the generated Stack. We refer to this as the PublicIP of the Control Plane.

  • Go back to your previously created GitHub OAuth application and modify it. Set the Authorization callback URL field to http://{control_plane_public_ip}/authorize

Define a .pipeline.yml pipeline workflow configuration for your Spark application

The steps of the workflow executed by the CI/CD flow are described in the .pipeline.yml file, which must be placed in the root directory of the Spark application’s source code. The file has to be pushed to the GitHub repo along with the source files of the application.

Here’s an example Spark application, spark-pi-example, which can be used to try out the CI/CD pipeline.

Note: To accomplish this, fork that repository into your own!

In order to set up your own spark application for the workflow, start by customizing the .pipeline.yml configuration file in spark-pi-example.

The following sections need to be modified:

  • the command for building your application
  remote_build:
    ...
    original_commands:
      - mvn clean package -s settings.xml
  • the Main class of your application
  run:
    ...
    spark_class: banzaicloud.SparkPi
  • the name of your application
  run:
    ...
    spark_app_name: sparkpi
  • the application artifact

This is the relative path to your Spark application’s jar. The jar is generated by this build command.

  run:
    ...
    spark_app_source: target/spark-pi-1.0-SNAPSHOT.jar
  • the application arguments
  run:
    ...
    spark_app_args: 1000

Grant access to desired GitHub organizations

Navigate to http://{control_plane_public_ip} in your web browser and grant access for the organizations that contain the GitHub repositories that you want to hook into the CI/CD workflow. Then click authorize access.

It may take some time for all of Pipeline’s services to fully initialize, and so the page may not load at first. If this happens, please wait a little and retry.

Hook repositories to CI/CD flow

Navigate to http://{control_plane_public_ip}, bringing you to the CI/CD user interface. Select Repositories from the top left menu. This lists all the repositories that the Pipeline has access to. Select the repositories you want to be hooked into the CI/CD flow.

CI/CD secrets

For the hooked repositories set the following secrets:

  • plugin_endpoint - specify http://{control_plane_public_ip}/pipeline/api/v1

  • plugin_username - specify the same user name you used for Pipeline Credentials

  • plugin_password - specify the same password you used for Pipeline Credentials

Submit your changes

Modify the source code of your Spark application. Commit changes and push them to the repository on GitHub. Pipeline is notified about commits through GitHub webhooks, and will trigger the flow described in the watched repositories’ .pipeline.yml file.

Monitor running workflows

Any running CI/CD jobs can be monitored and managed at http://{control_plane_public_ip}/account/repos

In order to check the logs of the different steps in the CI/CD workflow, click on the desired commit message in the UI.


That’s it! Once configured, the Spark application will be built, deployed and executed for every commit pushed to the project’s repository. You can check on the workflow’s progress by clicking on the small orange dot beside the commit in the GitHub UI.

Here are some of our git repos that contain example projects with pipeline workflow configurations:

If you’re interested in our technology and open source projects, follow us on GitHub, LinkedIn or Twitter:

Star

TRY PIPELINE FOR FREE

Comments

comments powered by Disqus