Placeholder image

Lajos Papp

Fri, Apr 20, 2018


Control your AWS spendings with ChatOps

While we are building our open source, cloud agnostic Heroku / Cloud Foundry like Paas, Pipeline built on top of Kubernetes we launch lots of clusters on different cloud providers. Most of these clusters are launched on spot or preemptible instances and managed by Hollowtrees, however there are many smaller development clusters, control planes, instances and proof of concepts we regularly do and they are marginally related or launched with Pipeline. These are obliviously having an associated cost as well and we wanted to keep tight control on that as well.

Hollowtrees is managing clusters started by Pipeline and using some sophisticated spot cluster recommendations, however the billing alert and Slack bot can be used independently from Pipeline as a generic solution to control spending on AWS

tl;dr1:

We have automated the creation of an AWS billing alarm, you can get the stack from our GitHub: banzaicloud/aws-billing-alarm

tl;dr2:

We have open sourced a Slack chatbot to help you tightly control your AWS spendings. Get the Cloudformation stack template and the open source Golang lambda function from our GitHub: banzaicloud/lambda-slack-bot.

Read on for further details.

Controlling costs on AWS - available options

Controlling costs on AWS (or a cloud provider) takes some effort, but it definitely pays out. Usually people regularly perform these two steps manually:

  • check the actual month total cost (estimate)
  • check if there are any unused resources, which are not used anymore

Performing them manually is tiresome, so lets automate both:

  • create an AWS billing alarm which emails you if your estimated charges exceeds a limit
  • create a chatbot which helps to discover resources that are eligible for billing

AWS Billing Alarm

Creating an AWS billing alarm is pretty straightforward, and you should really use it. You can avoid surprise bills at the end of the month.

If you want to create a Billing alarm, we provide a cloudformation template on GitHub: banzaicloud/aws-billing-alarm

ChatOps

Now lets turn to the ChatOps part. For most of the AWS customers the charges are produced by EC2 instances. Have you ever shouted around the office like:

  • Has anybody left a running instance in us-west region?
  • Who started that m4.xlarge instance 2 weeks ago in eu-central-1?

One issue is, that on AWS console you always have to select a single specific region to work with resources. So if you want to check multiple regions, you have to repeat the process for each region.

SlackBot

Slack seems to be the one of the most used team communication platform, and it offers various extensibility. One of them is Slash Commands, which means once you have installed/configured the App (that is the slack term for extensions) you can just simple trigger your app by a slack message starting with “/”. Like:

/instances

The deployment process will be:

  • deploy a CloudformationStack (api gateway + lambda function + ec2 readOnly policy)
  • create a slack app
  • add a slash command pointing to the apiGateway endpoint

Deploy the stack from cli - quick way

If you have the aws-cli installed it is a single command:

export AWS_STACK_NAME=slackBot
aws cloudformation create-stack \
  --template-url https://s3-eu-west-1.amazonaws.com/lp-lambda-go/aws-slack-bot.json \
  --capabilities CAPABILITY_IAM \
  --stack-name $AWS_STACK_NAME

If you want to hack on the template you can find it on GitHub: banzaicloud/lambda-slack-bot

Deploy the stack from console - it takes longer than installing awscli

If you don’t have aws-cli installed, or you like to click and type, you can deploy from the console:

  • Start the Create Stack on AWS console: Launch Stack
  • click Next Select Template in the Select Template step.
  • click Next on the Specify Details step.
  • click Next on the Options step.
  • On the Review step, scroll down to Capabilities and check both I acknowledge ... checkbox
  • Create ChangeSet on the Transforms section
  • Once the transformation is done, scroll down and click on Execute

Verifying the deployment

Once the slackBot stack is deployed, watch the Output for the restAPI url.

export AWS_STACK_NAME=slackBot
aws cloudformation describe-stacks \
  --stack-name $AWS_STACK_NAME \
  --query 'Stacks[0].Outputs[0].OutputValue' \
  --out text

You can test the API endpoint via curl. The response to a GET will be the version

$ curl https://a1b2c3d4e5.execute-api.eu-west-1.amazonaws.com/Prod
version: "0.0.7"

Create a Slack App

  • Go to SlackApi / YourApps
  • Click on the green Create New App
  • Fill out the form:
    • App Name : AwsInstancesBot
    • Development Slack Workspace: Choose the workspace (community) where you want to use the bot from.
    • click Create App

Add a Slash Command

  • Choose: Slash Commands in the left menu
  • Click on Create New Command Button:
  • Fill the form:
    • Command: /instances
    • Request URL: https://a1b2c3d4e5.execute-api.eu-west-1.amazonaws.com/Prod
    • Short Description: Lists running ec2 instances
    • Usage Hint : /instances ascii
    • click Save

Activate the Bot on the Workspace

Although the Create a Slack App step seemingly attached the bot to the workspace, there is an extra step needed: - Settings / Basic Information - click on the Install your app to your workspace dropdown section - push the green Install App button

Now you are ready to use the new slash command. Try to write to the General channel: /instances ascii. You will see the ascii table of running instances

Configuring queried regions

The lambda function stores in an environment variable the coma separated list of regions for the EC2 instances you’d like to be listed. By default it contains all regions except ap-northeast-2. The reason is that it takes about 4 seconds if you include it, and without it takes about 1 sec.

Roadmap

Some possible direction this project can go:

  • Include a spent column, using the AWS price api, and spot price history to estimate the money burned by an instance
  • tag-bot: All instances should have an Owner represented by an EC2 tag. Instances without an owner could be regularly cleaned. A bot could be used to tag ownerless instances. Billing tools can use this tag, for cost monitoring/management.
  • listing/cleaning VPCs: VPCs are one of those resources which have endless dependencies. The console helps you if you want to delete with dependencies, but on the cli, or in the api, you have to discover, and pre-clean all dependencies.

tl;dr

Some additional useful resources used during development.

Resource Groups

Although you could use Resource Groups from the console to discover resources across regions, but it seems like there are two different types of resource groups:

  • classic groups
  • resource groups - saved groups (new)

The classic one seems to only exist on the console, and can do multi-region queries. While the new one, is available from the api/cli like: aws resource-groups list-groups. But that one which is programmable/scriptable seems to work only on regional level.

So you are left with repeating steps for each region you are interested, which points to automation.

TabWriter

Creating pretty ascii tables normally involves some sort of: fmt.Printf("%20s %20d", strVal, number). But even for that you have to calculate the column width. TabWriter comes to the rescue! You just wrap the output with TabWriter and all you have to do is print extra tab characters \t between fields, and the rest is taken care of. Again a nice example of hidden gems in the standard go library:

If you are interested in our technology and open source projects, follow us on GitHub, LinkedIn or Twitter:

Star



Comments

comments powered by Disqus