Banzai Cloud Logo Close
Home Benefits Blog Company Contact
Sign in
Author Sandor Magyari

The Horizontal Pod Autoscaler operator reloaded

TRY PIPELINE FOR FREE

One of the main advantages of the Pipeline platform is that it allows users to use their infrastructure cost effectively; Telescopes helps with cluster and machine instance recommendations, Hollowtrees enables SLA-aware cost reduction using spot instances, and autoscalers allow for multi-dimensional autoscaling based on custom metrics.

This post will highlight some new features of the Banzai Cloud Horizontal Pod Autoscaler Kubernetes Operator and the advanced automation supported by Pipeline - a new, forward-thinking way to operate Kubernetes clusters and autoscale deployments.

In case you missed our blog series about the different types of autoscaling features supported and automated by the Pipeline platform, please follow these earlier posts:

tl;dr:

  • An introduction to redesigned HPA annotations in the Horizontal Pod Autoscaler Kubernetes Operator
  • The Deletion of HPA resources in the event that annotations are completely removed
  • The HPA & Metrics Server recently released for Amazon EKS available in Pipeline
  • An introducion to the new Pipeline HPA API

HPA Operator

Redesigned HPA annotations

We redesigned the annotation naming scheme in order to provide us greater flexibility and to comply with Kubernetes annotation naming standards. Let’s see an example of the new annotation system in action:

hpa.autoscaling.banzaicloud.io/minReplicas:               "1",
hpa.autoscaling.banzaicloud.io/maxReplicas:               "3",
cpu.hpa.autoscaling.banzaicloud.io/targetAverageUtilization: "70"
memory.hpa.autoscaling.banzaicloud.io/targetAverageValue: "1000Mi"
pod.hpa.autoscaling.banzaicloud.io/custom.metric.name: "1000Mi"

As you can see, there’s a separate subdomain for cpu, memory and custom pod metrics. Now you can specify percentage or absolute quantity values for both cpu and memory metrics.

  • cpu.hpa.autoscaling.banzaicloud.io/targetAverageUtilization: "{targetAverageUtilizationPercentage}" - adds a Resource type metric for CPU with targetAverageUtilization set as specified, where targetAverageUtilizationPercentage should be an int value between [1-100]

  • cpu.hpa.autoscaling.banzaicloud.io/targetAverageValue: "{targetAverageValue}" - adds a Resource type metric for CPU with targetAverageValue set as specified, where targetAverageValue is a Quantity.

  • memory.hpa.autoscaling.banzaicloud.io/targetAverageUtilization: "{targetAverageUtilizationPercentage}" - adds a Resource type metric for memory with targetAverageUtilization set as specified, where targetAverageUtilizationPercentage should be an int value between [1-100].

  • memory.hpa.autoscaling.banzaicloud.io/targetAverageValue: "{targetAverageValue}" - adds a Resource type metric for memory with targetAverageValue set as specified, where targetAverageValue is a Quantity.

  • autoscale.pod/custom_metric_name: "{targetAverageValue}" - adds a Pods type metric with targetAverageValue set as specified, where targetAverageValue is a Quantity.

HPA & Metrics Servers available on Amazon EKS

Amazon just announced that they’ve enabled aggregations on the API Server, thus the HPA & Metrics Server is now available on EKS. Unsurprisingly, we’ve been keen to add this to Pipeline as well. As usual, the whole process is automated and our end users don’t need to do anything, since we install the Metrics Server automatically.

This is not a straight forward process, so here are a couple of notes for those of you who are going to attempt this manually, instead of using Pipeline:

  • be sure to use the latest (0.3.0) version of gcr.io/google_containers/metrics-server-amd64
  • don’t forget to configure security settings in order to allow the cluster control plane to communicate with pods running extension API servers on port 443, and, vice-versa, to allow pods running extension API servers on port 443 to receive communication from the cluster control plane as well. Checkout our latest Cloud Formation template for reference.

Pipeline HPA API

If you’d like to autoscale your Kubernetes deployments, by default, you’ll need to add the correct annotations manually. This is consistent with Pipeline as it existed a few months ago, and the first version of the Horizontal Pod Autoscaler Kubernetes Operator relied on these manual inputs as well. The process has been simplified in the current Pipeline user interface, as it generates these annotations, but the cluster operator still has to copy generated annotations into the right locations in the values.yaml files.

This process is better than the default HPA (especially now that we’re generating proper YAML snippets), but based on feedback from our customers, we’ve realized that there’s still room for improvement. So, we decided to create a separate page to manage HPAs for all Pipeline deployments, and, in order to automate that process, we designed a new Horizontal Pod Autoscaler API, which actually creates, updates and manages annotations on all Kubernetes Deployments and StafulSets.

Let’s examine a sample of how we might manage HPA for a deployment:

  • query resources contained in a released Helm chart, for example a K8s deployments:
GET {{url}}/api/v1/orgs/{{orgId}}/clusters/{{cluster_id}}/deployments/quaffing-dachshund/resources?resourceTypes=deployment
  • query HPAs bound to this deployment specifying deployment name in scaleTarget:
GET {{url}}/api/v1/orgs/{{orgId}}/clusters/{{cluster_id}}/hpa?scaleTarget=morbid-catfish-zeppelin
  • create and update HPAs
POST {{url}}/api/v1/orgs/{{orgId}}/clusters/{{cluster_id}}/hpa

{
	"scaleTarget": "morbid-catfish-zeppelin",
	"minReplicas": 1,
	"maxReplicas": 10,
	"cpu": {
        "targetAverageValueType": "percentage",
        "targetAverageValue": "60"
	},
	"memory": {
        "targetAverageValueType": "quantity",
        "targetAverageValue": "1100Mi"
	},
  "customMetrics": {
    "customMetricName": {
      "type": "pod",
      "targetAverageValue": "700m"
    },
    "customMetricName2": {
      "type": "pod",
      "targetAverageValue": "700m"
    }
  }
}
  • delete HPA
DELETE {{url}}/api/v1/orgs/{{orgId}}/clusters/{{cluster_id}}/hpa?scaleTarget=morbid-catfish-zeppelin

Our main goal with the Horizontal Pod Autoscaler Kubernetes Operator is to bypass the complexities of HPA. Pipeline platform users should only have to think about the best metrics for autoscaling based on their deployments, and Pipeline will automate the rest.

Happy Autoscaling!

If you’d like to learn more about Banzai Cloud and the Pipeline platform, check out our other posts on this blog, or the core building blocks of Pipeline, Hollowtrees, Telescopes and Bank-Vaults on Github, or follow us on LinkedIn or Twitter.

Star

TRY PIPELINE FOR FREE

Comments

comments powered by Disqus