Banzai Cloud Logo Close
Home Products Benefits Blog Company Contact
Author Zsolt Varga

Pod level external HTTP(S) proxy configuration with Istio

Companies frequently use proxies to act as a link between an internal network and the Internet. This is often frustrating for employees, even non-IT ones, when they can’t access a specific site from the company network. For engineers it’s even more obnoxious, since they have to configure all kinds of compute infrastructure to connect to external networks via these proxies.

It’s debatable if this is the best way to harden corporate network security, but it’s still the most widely spread method to restrict outgoing traffic. The intentions behind introducing a proxy are usually good, as the main reason is almost always to prevent data breaches by adding an additional layer of security. Proxies come with other useful features, however. They can audit network traffic, filter undesired content or even act as a cache.

This post will talk about corporate proxies from an infrastructure engineer’s standpoint, when they have to deal with the configuration in an environment including Kubernetes and Istio.

tl;dr: 🔗︎

When utilizing dynamic services that run on Kubernetes and Istio, a proxy requires an increased level of configuration automation for all the components running behind it. We’re going to give a brief overview of how Backyards helps with the pod-level configuration of these settings in the Kubernetes cluster.

Backyards is the Banzai Cloud Istio distribution that just works. Backyards operationalizes the service mesh to bring deep observability, convenient management, and policy-based security to modern container-based applications.

Istio with proxy

Proxying on Kubernetes 🔗︎

The proxy configuration for client applications usually uses environment variables, such as HTTP_PROXY, HTTPS_PROXY and NO_PROXY.

Let’s assume we have the following proxy addresses:

  • http://proxy.example.com:3128/ for HTTP proxy
  • http://proxy.example.com:3129/ for HTTPS proxy

If that were the case, the following environment variables would need to be set in the YAML spec of a pod running on the Kubernetes cluster:

containers:
- env:
  - name: "HTTP_PROXY"
    value: "http://proxy.example.com:3128"
  - name: "HTTPS_PROXY"
    value: "http://proxy.example.com:3129"
  - name: "NO_PROXY"
    value: "svc,local"

Manually inserting those variables into each and every pod would be far too tedious and error-prone a process, and would only work in a very small cluster anyway.

Admission webhooks to the rescue 🔗︎

The better option to inject these environment variables to a pod’s containers is to use an admission webhook. Since Backyards already has a webhook to inject Istio sidecar proxies into pods, we can leverage and extend the existing webhook functionality to support proxy environment variable injection as well.

Backyards uses the open source Banzai Cloud Istio operator under the hood, which provides support for exactly the kind of feature we were just talking about; we can specify the settings of the external proxy in the Istio custom resource and Backyards will take care of the rest.

The corresponding Istio custom resource configuration is as follows:

spec:
  httpProxyEnvs:
    httpProxy: "http://proxy.example.com:3128"
    httpsProxy: "http://proxy.example.com:3129"
    noProxy: "svc,local"

Showtime 🔗︎

Setup 🔗︎

  1. Create a Kubernetes cluster.

    You can create a Kubernetes cluster with the Banzai Cloud Pipeline platform on-prem, or over any one of five different cloud providers.

  2. Point KUBECONFIG to your cluster.

  3. Install Istio with Backyards:

    curl https://getbackyards.sh | sh && backyards install -a
    

    Note: Backyards is Banzai Cloud’s Istio service mesh. You can test and evaluate it in non-production environments. Contact us if you’re interested in using Backyards in production.

    This step usually takes a few minutes, and installs Istio and Backyards on your cluster.

Deploy an HTTPS proxy 🔗︎

Let’s deploy a simple proxy inside the cluster to showcase external proxy usage for this example. Also, we’re going to use the pod IP address to make the scenario more realistic. This example uses Tinyproxy, but any proxy that supports HTTP CONNECT will do.

  1. Create an external namespace

Create a new namespace without the Istio sidecar injection enabled to simulate the proxy being outside of the cluster.

$ kubectl create namespace external
namespace/external created
  1. Deploy Tinyproxy
$ kubectl apply -f - <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: tinyproxy
  namespace: external
spec:
  replicas: 1
  selector:
    matchLabels:
      app: tinyproxy
  template:
    metadata:
      labels:
        app: tinyproxy
    spec:
      containers:
      - image: vimagick/tinyproxy
        imagePullPolicy: IfNotPresent
        name: tinyproxy
EOF
  1. Obtain the IP address of the proxy pod

Store the proxy pod IP in the PROXY_IP environment variable for later use.

$ export PROXY_IP=$(kubectl get pod -n external -l app=tinyproxy -o jsonpath='{@.items..podIP}')

Configure the proxy settings 🔗︎

  1. Set the proxy settings in the Istio custom resource

The default Tinyproxy container will listen on port 8888, so that’s the port that we’ll use.

$ kubectl -n istio-system patch istio mesh --type=merge --patch "
spec:
  httpProxyEnvs:
    httpProxy: "http://$PROXY_IP:8888"
    httpsProxy: "http://$PROXY_IP:8888"
    noProxy: "svc,local"

Deploy a simple pod to check outgoing connections 🔗︎

  1. Label the default namespace for automatic sidecar injection
$ backyards sp ai on default
INFO[0002] auto sidecar injection successfully set to namespace default
  1. Use the following snippet to create a test pod.
$ kubectl create -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
  name: curl-test
spec:
  containers:
  - name: curl
    image: curlimages/curl:7.70.0
    command: [ "/bin/sh", "-c", "--" ]
    args: [ "while true; do sleep 3000; done;" ]
EOF
  1. Check the environment variables of the created pod to see if the automatic injection worked.
$ kubectl -n default get pods curl-test -o yaml |grep "env:" -A 12|head -13
    env:
    - name: http_proxy
      value: http://10.20.1.127:8888
    - name: HTTP_PROXY
      value: http://10.20.1.127:8888
    - name: https_proxy
      value: http://10.20.1.127:8888
    - name: HTTPS_PROXY
      value: http://10.20.1.127:8888
    - name: no_proxy
      value: svc,local
    - name: NO_PROXY
      value: svc,local

The HTTP(S) proxy environment variables were automatically injected to the pod, and the variables are set in both uppercase and lowercase.

  1. Send a request from the curl-test pod in the default namespace to an external service.
$ kubectl exec -it curl-test -c curl -- sh -c "curl http://httpbin.org/headers"
{
  "headers": {
    "Accept": "*/*",
    "Host": "httpbin.org",
    "User-Agent": "curl/7.70.0-DEV",
    "X-Amzn-Trace-Id": "Root=1-5ef23da7-4c7406e8ef392f32b61d7ddb"
  }
}
  1. Check the access logs of the proxy for your request
$ kubectl -n external logs -l app=tinyproxy |grep CONNECT
CONNECT   Jun 23 17:36:27 [10]: Established connection to host "httpbin.org" using file descriptor 10.
CONNECT   Jun 23 17:36:38 [6]: Connect (file descriptor 9): ip-10-20-1-173.eu-west-1.compute.internal [10.20.1.173]
CONNECT   Jun 23 17:36:38 [6]: Request (file descriptor 9): GET http://httpbin.org/headers HTTP/1.1
CONNECT   Jun 23 17:36:39 [6]: Established connection to host "httpbin.org" using file descriptor 10.

Istio with proxy

The future: transparent external proxy with Envoy 🔗︎

An even better solution for the use case we’ve discussed would be to make it so the injected Envoy sidecar transparently supported routing through an external proxy. That way it would be totally unnecessary to configure via environment variables or provide support in the client application.

There is ongoing work to support such a feature in Envoy, and hopefully it will land in the near future. The CONNECT support is already merged, and there is an open issue about CONNECT support for HTTP/1.1 upstreams as well.

Once these features are merged and released upstream, Backyards will switch to its Envoy-based method of configuring external proxies.

‘Til then, happy proxying.

Never miss a post again!
Schedule a Backyards demo

If you are interested in our technology and open source projects, follow us on GitHub, LinkedIn, or Twitter, or get in touch on Slack: