Banzai Cloud Logo Close
Home Benefits Blog Company Contact
Sign in

Cloud agnostic bucket management

TRY PIPELINE FOR FREE

Banzai Cloud’s Pipeline provides a platform which allows enterprises to develop, deploy and scale container-based applications. It leverages best-of-breed cloud components, such as Kubernetes, to create a highly productive, yet flexible environment for developers and operations teams alike. Strong security measures—multiple authentication backends, fine-grained authorization, dynamic secret management, automated secure communications between components using TLS, vulnerability scans, static code analysis, etc.—are a tier zero feature of the Pipeline platform, which we strive to automate and enable for all enterprises.

There are a lot of different storage solutions out there, but Object storage has emerged as the de facto leader in today’s fast-moving world of technology; it’s cost effective at scale, ensures data is available across multiple regions, and scales to petabytes in a highly available and highly durable manner. Today’s modern applications that are deployed to K8s produce and consume large datasets, which are stored in the object storages offered by cloud providers.

Pipeline simplifies the flow and management of object stores for those applications deployed through it. It provides a cloud agnostic unified API to create, list, delete, and check object storage on all major cloud providers, like Amazon, Google, Azure, Oracle, and Alibaba. Moreover, this API uses Bank-Vaults to store/retrieve secrets necessary for object storage-related operations.

There may be prerequisites to creating object stores on cloud providers that must be fulfilled before object stores can be created. Pipeline automates the steps required to create object storage on any cloud provider. Users need only specify the necessary input parameters and they can let Pipeline to take care of everything else.

As mentioned above, when Pipeline executes an object storage operation it talks to Bank-Vaults to retrieve credentials needed to engage the cloud provider’s API.

Object storage management flow in Pipeline looks like this:

  1. A user obtains a bearer token that is required by Pipeline’s REST API. This token can be obtained via login, first at http://{pipeline-url}/auth/github/login, then http://{pipeline-url}/api/v1/token by using a web browser.

  2. A user registers their cloud provider credentials in Pipeline, which are stored as secrets securely in Bank Vaults. Note, these credentials must have object storage creation privileges on the cloud provider.

  3. A user invokes the creation, deletion, checking, or listing of object stores in the API on Pipeline.

Also note, the create and delete API calls run asynchronously, as these might to be lengthy operations.

When listing object stores, a field called managed indicates whether a store is being managed via Pipeline or not. The delete operation is permitted only on those stores where "managed": true.

Cloud agnostic bucket management

Amazon S3 buckets

To create an S3 bucket

curl -X POST \
  'http://{pipeline-url}/api/v1/orgs/{orgId}/buckets' \
  -H 'Content-Type: application/json' \
  -H "Authorization: Bearer $TOKEN" \
  -d '{
	"secretName": "my-amazon-secret",
	"name": "my-s3-bucket1113",
	"properties": {
		"amazon": {
			"location": "eu-west-1"
		}
	}
}'

The above API call creates an S3 bucket on Amazon with the name my-s3-bucket1113 in eu-west-1, using those Amazon credentials referenced by the Vault secret identified as my-amazon-secret.

Verify if the Amazon S3 bucket was created

curl -X HEAD -I \
    'http://{pipeline-url}/api/v1/orgs/{orgId}/buckets/my-s3-bucket1113?cloudType=amazon&location=eu-west-1' \
    -H "Authorization: Bearer $TOKEN" \
    -H 'secretName: my-amazon-secret'

 HTTP/1.1 200 OK

Amazon console

Amazon S3 bucket

To list Amazon S3 buckets

curl -X GET \
    'http://{pipeline-url}/api/v1/orgs/1/buckets?cloudType=amazon&location=eu-west-1' \
    -H "Authorization: Bearer $TOKEN" \
    -H 'secretName: my-amazon-secret'

[
...
  {
    "name": "my-s3-bucket1113",
    "managed": true,
    "location": "eu-west-1"
  },
...
]

The above API call lists all Amazon S3 buckets that the credentials referenced by my-amazon-secret have listing access to. In addition to the name of the S3 bucket, it also returns the region the S3 bucket was originally created in.

To delete the Amazon S3 bucket

curl -X DELETE \
    'http://{pipeline-url}/api/v1/orgs/{orgId}/buckets/my-s3-bucket1113?cloudType=amazon&location=eu-west-1' \
    -H "Authorization: Bearer $TOKEN" \
    -H 'secretName: my-amazon-secret'

Google Cloud Storage buckets

To create a bucket in Google Cloud Storage

curl -X POST \
  'http://{pipeline-url}/api/v1/orgs/{orgId}/buckets' \
  -H 'Content-Type: application/json' \
  -H "Authorization: Bearer $TOKEN" \
  -d '{
	"secretName": "my-google-secret",
	"name": "my-gs-bucket1113",
	"properties": {
		"google": {
			"location": "eu"
		}
	}
}'

This creates a multi-regional bucket with the name my-gs-bucket1113 in eu location using the Google Cloud credentials referenced by my-google-secret. To create a regional bucket, pass a region name (e.g. europe-west1) as location.

Verify if the bucket was created

curl -X HEAD -I \
    'http://{pipeline-url}/api/v1/orgs/{orgId}/buckets/my-gs-bucket1113?cloudType=google' \
    -H "Authorization: Bearer $TOKEN" \
    -H 'secretName: my-google-secret'

 HTTP/1.1 200 OK

Google console

Google Storage bucket

To list Google Storage buckets

curl -X GET \
    'http://{pipeline-url}/api/v1/orgs/1/buckets?cloudType=google' \
    -H "Authorization: Bearer $TOKEN" \
    -H 'secretName: my-google-secret'

[
...
  {
    "name": "my-gs-bucket1113",
    "managed": true
  },
...
]

The above API call lists all the buckets on Google Cloud Storage that the credential referenced by my-google-secret has listing access to.

To delete the Google bucket

curl -X DELETE \
    'http://{pipeline-url}/api/v1/orgs/{orgId}/buckets/my-gs-bucket1113?cloudType=google' \
    -H "Authorization: Bearer $TOKEN" \
    -H 'secretName: my-google-secret'

Azure Blob Storage

To create an Azure Blob Storage

To create Azure Blob Storage a storage account and a resource group are required:

curl -X POST \
  'http://{pipeline-url}/api/v1/orgs/{orgId}/buckets' \
  -H 'Content-Type: application/json' \
  -H "Authorization: Bearer $TOKEN" \
  -d '{
	"secretName": "my-azure-secret",
	"name": "my-azure-blobstore1113",
	"properties": {
		"azure": {
			"location": "westeurope",
			"storageAccount": "mysa1113",
			"resourceGroup": "myrg1113"
		}
	}
}'

Pipeline verifies that the specified my-sa1113 storage account and my-rg1113 resource group exist, and creates them if they don’t. Only afterward does it create a my-azure-blobstore1113 Azure Blob Storage. Also, it retrieves the access key of the storage account and stores it as a Vault secret so that this accesss key is only available later, when other operations require the account access key.

Create a bucket on Azure

Verify if the bucket was created

curl -X HEAD -I \
    'http://{pipeline-url}/api/v1/orgs/{orgId}/buckets/my-azure-blobstore1113?cloudType=azure&resourceGroup=myrg1113&storageAccount=mysa1113' \
    -H "Authorization: Bearer $TOKEN" \
    -H 'secretName: my-azure-secret'

 HTTP/1.1 200 OK

Azure portal

Azure Blob Storage

To list blob storages on Azure

url -X GET \
    'http://{pipeline-url}/api/v1/orgs/1/buckets?cloudType=azure' \
    -H "Authorization: Bearer $TOKEN" \
    -H 'secretName: my-azure-secret'

[
  ...
  {
    "name": "my-azure-blobstore1113",
    "managed": true,
    "aks": {
      "resourceGroup": "myrg1113",
      "storageAccount": "mysa1113"
    }
  },
  ...
]

The above API call lists all blob stores on Azure that the credential referenced by my-azure-secret has listing access to. In addition to the name of a blob store, it also lists the resource group and storage account it belongs to.

To delete blob storage

In order to delete a blob store, we need the access key of the storage account that holds it. Pipeline then uses this access key, which was stored in Vault at the creation of the blob store.

curl -X DELETE \
    'http://{pipeline-url}/api/v1/orgs/1/buckets/my-azure-blobstore1113?cloudType=azure&resourceGroup=myrg1113&storageAccount=mysa1113' \
    -H "Authorization: Bearer $TOKEN" \
    -H 'secretName: my-azure-secret'

Delete bucket on Azure

Oracle Cloud Storage

To create a bucket in Oracle Cloud Object storage

curl -X POST \
  'http://{pipeline-url}/api/v1/orgs/{orgId}/buckets' \
  -H 'Content-Type: application/json' \
  -H "Authorization: Bearer $TOKEN" \
  -d '{
	"secretName": "my-oracle-secret",
	"name": "my-oracle-bucket1113",
	"properties": {
		"oracle": {
			"location": "eu-frankfurt-1"
		}
	}
}'

The API call above creates a bucket in Oracle Cloud Storage with name my-oracle-bucket1113 in eu-frankfurt-1, using the Oracle cloud credential referenced by the Vault secret identified by my-oracle-secret.

Verify that the bucket was created

curl -X HEAD -I \
    'http://{pipeline-url}/api/v1/orgs/{orgId}/buckets/my-oracle-bucket1113?cloudType=oracle&location=eu-frankfurt-1' \
    -H "Authorization: Bearer $TOKEN" \
    -H 'secretName: my-oracle-secret'

HTTP/1.1 200 OK

Oracle Cloud Console

Oracle Cloud Storage bucket

To list buckets on Oracle Cloud

curl -X GET \
  'http://{pipeline-url}/api/v1/orgs/1/buckets?cloudType=oracle' \
  -H "Authorization: Bearer $TOKEN" \
  -H "secretName: my-oracle-secret"

  [
  ...
  {
    "name": "my-oracle-bucket1113",
    "managed": true,
    "location": "eu-frankfurt-1"
  },
  ...
]

The above API call lists all buckets on Oracle Cloud that the credential referenced by my-oracle-secret has listing access to.

To delete the bucket

curl -X HEAD -I \
    'http://{pipeline-url}/api/v1/orgs/{orgId}/buckets/my-oracle-bucket1113?cloudType=oracle&location=eu-frankfurt-1' \
    -H "Authorization: Bearer $TOKEN" \
    -H 'secretName: my-oracle-secret'

Alibaba Object Storage

To create a bucket in Alibaba Cloud Object storage

curl -X POST \
  'http://{pipeline-url}/api/v1/orgs/{orgId}/buckets' \
  -H 'Content-Type: application/json' \
  -H "Authorization: Bearer $TOKEN" \
  -d '{
	"secretName": "my-alibaba-secret",
	"name": "my-ali-bucket1113",
	"properties": {
		"alibaba": {
			"location": "oss-eu-central-1"
		}
	}
}'

The API call above creates a bucket in Alibaba Storage with name my-ali-bucket1113 in oss-eu-central-1 using the Alibaba cloud credential referenced by the Vault secret identified by my-alibaba-secret.

Verify if the bucket was created

curl -X HEAD -I \
    'http://{pipeline-url}/api/v1/orgs/{orgId}/buckets/my-ali-bucket1113?cloudType=alibaba&location=oss-eu-central-1' \
    -H "Authorization: Bearer $TOKEN" \
    -H 'secretName: my-alibaba-secret'

HTTP/1.1 200 OK

Alibaba Cloud Console

Alibaba Cloud Storage bucket

To list buckets on Alibaba Cloud

curl -X GET \
    'http://{pipeline-url}/api/v1/orgs/1/buckets?cloudType=alibaba&location=oss-eu-central-1' \
    -H "Authorization: Bearer $TOKEN" \
    -H 'secretName: my-alibaba-secret'

[
  ...
  {
    "name": "my-ali-bucket1113",
    "managed": true
  },
  ...
]

The above API call lists all buckets on Alibaba Cloud that the credential referenced by my-alibaba-secret has listing access to.

To delete the bucket

curl -X DELETE \
    'http://{pipeline-url}/api/v1/orgs/{orgId}/buckets/my-ali-bucket1113?cloudType=alibaba&location=oss-eu-central-1' \
    -H "Authorization: Bearer $TOKEN" \
    -H 'secretName: my-alibaba-secret'

The Pipeline object storage management API definition

The OpenAPI 3.0.0 definition for the Pipeline’s object storage management API can be found here.

If you are interested in our technology and open source projects, follow us on GitHub, LinkedIn or Twitter:

Star

TRY PIPELINE FOR FREE

Comments

comments powered by Disqus