• Zero Trust in AWS
• Zero Trust with Cloudflare
• Implementation
  • Cloudflare Side
    • A Note on manual setup
    • Tunnel
    • Access Application
  • AWS Side
    • Parameter Store
    • ECR
      • Cloudflared
      • Flask
    • ECS Cluster & VPC
    • ECS Service
      • Container Definitions
      • ECS Task
• See it in Action
• Conclusions

For the past few years, I’ve been gradually implementing and extending a Flask web app that I use to automate the management of some of my side projects (i.e., CloudSecList, CloudSecBooks).

This app started as a simple project with a few endpoints deployed locally via docker-compose, to a not-so-simple-anymore application that automates multiple parts of my side projects. I have now reached a state where I need more than a local deployment, as I might need to perform some quick operations when I don’t have my laptop with me. So I started looking at options to deploy it safely on AWS without exposing it directly to the public Internet, as it is an admin-like interface.

In this blog, I’ll explain how to use Cloudflare Tunnel to securely access a Flask web app running in a private subnet in ECS on Fargate, without exposing the app to the public Internet. No VPNs are required either.

Zero Trust in AWS

So, the main goal is to have a solution which allows access to a web app (Flask in this case, but it could be another framework or even language, as it doesn’t affect the final solution) running in ECS on Fargate, without either having it exposed to the public Internet or behind a VPN.

If we were on GCP, this could’ve easily been accomplished by Identity-Aware Proxy (IAP), a native service that implements a zero-trust model to control access to applications running on Google Cloud. But since the rest of my infrastructure is on AWS, I want to avoid going multi-cloud (😅), so IAP is out of the equation.

In AWS, this is more complex, as native services (like Cognito) are not so streamlined and well-integrated.

That’s where Cloudflare comes to the rescue. In Remotely Access your Kubernetes Lab with Cloudflare Tunnel, I previously described how to use Cloudflare Tunnel to connect a Kubernetes cluster to the Cloudflare network. Similarly, here we can use Cloudflare Tunnel again to provide Zero Trust access to our Flask app.

At a high level, it will look like the following:

Before jumping into the implementation details, let’s discuss what happens in the “Cloudflare” box of the diagram above.

Zero Trust with Cloudflare

We will be using two components of the Zero Trust offering of Cloudflare:

1. Tunnel: to enable private network connectivity.
2. Applications (Access): to enforce access control.

Cloudflare Tunnel provides a way to connect resources to Cloudflare without a publicly routable IP address. With Tunnel, traffic is not sent to an external IP. Instead, you deploy a lightweight daemon in your infrastructure (cloudflared) which creates an outbound-only connection to Cloudflare’s edge.

Cloudflare Applications, instead, can secure self-hosted and SaaS applications with Zero Trust rules which enforce authentication and access controls (among other security checks).

Implementation

The final solution I ended up with is shown in the diagram below:

For the remainder of this blog post, I will explain each component’s purpose and how it can be automatically deployed.

Cloudflare Side

Let’s start from the Cloudflare side of the picture. As mentioned above, we have two components to set up: an Application and a Tunnel.

But first, let me digress for a moment on Cloudflare’s own documentation and the manual setup they recommend.

A Note on manual setup

When I initially started looking at this problem, I, of course, started from Cloudflare’s documentation website.

Although their documentation is excellent for a manual deployment [1, 2, 3] (by following the doc and clicking/pasting around, you can indeed get a Tunnel up and running in ~10 minutes), it lacks a bit if you want to make the process repeatable and automated without human intervention (with Terraform for example).

In short, the manual steps they recommend are the following:

• From your instance/server, authenticate cloudflared against your Cloudflare account and zone:
  • cloudflared tunnel login will open a browser window and prompt you to log in to your Cloudflare account. After logging in to your account, select your hostname (example.com).
  • Once authenticated, cloudflared will generate an account certificate (cert.pem).
  • Here is how you can do it if you are running cloudflared as a Docker container (not easily referenced from their docs):

docker run --rm -it                                            \
      -v /tmp/cloudflared_config/:/home/nonroot/.cloudflared/  \
      cloudflare/cloudflared:2023.3.1                          \
      tunnel login https://flask.example.com
        

• Create a Tunnel and give it a name:
  • cloudflared tunnel create <NAME> will create a (for now, inactive) Tunnel with the name specified.
  • cloudflared will also generate a Tunnel credentials file, which stores the tunnel’s credentials in JSON format.
  • Here is how you can do it if you are running cloudflared as a Docker container:

docker run --rm -it                                            \
      -v /tmp/cloudflared_config/:/home/nonroot/.cloudflared/  \
      cloudflare/cloudflared:2023.3.1                          \
      tunnel create flask
        

• Start routing traffic:
  • cloudflared tunnel route dns <NAME> <hostname> assigns a CNAME record that points traffic to the Tunnel’s subdomain.
  • Here is how you can do it if you are running cloudflared as a Docker container:

docker run --rm -it                                            \
      -v /tmp/cloudflared_config/:/home/nonroot/.cloudflared/  \
      cloudflare/cloudflared:2023.3.1                          \
      tunnel route dns flask flask.example.com
        

• Run the Tunnel:
  • cloudflared tunnel run <NAME> starts proxying incoming traffic from the Tunnel to the services running locally on your origin.
  • Here is how you can do it if you are running cloudflared as a Docker container:

docker run --rm -it                                            \
      -v /tmp/cloudflared_config/:/home/nonroot/.cloudflared/  \
      cloudflare/cloudflared:2023.3.1                          \
      tunnel run flask
        

While I was starting to automate these steps, I found relevant Terraform resources to replace all steps but one. I couldn’t find a way to remove the manual browser-based login step until I realised that the PEM file is unnecessary if you create the Tunnel via Terraform (as I was planning)!

In fact, although not directly mentioned in the Cloudflare docs (actually, it is only implicitly inferred but not openly stated [1, 2, 3]), I discovered via a GitHub issue that it is indeed possible to run the Tunnel by providing only the JSON credentials file. That was the last hurdle gladly removed, also because it is not currently possible to rotate the PEM account certificate (the issue has been closed without resolution).

Done with the digression, from here onwards, we will focus exclusively on an automated deployment via Terraform.

Tunnel

The first component we are going to deploy is the Cloudflare Tunnel:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
#
# Tunnel
#
resource "cloudflare_tunnel" "flask" {
  account_id = var.cloudflare_account_id

  name   = var.tunnel_name
  secret = random_password.flask_token.result
}

# Random password to use as the tunnel secret
resource "random_password" "flask_token" {
  length  = 32
  special = false
}

#
# Tunnel Config
#
resource "cloudflare_tunnel_config" "flask" {
  account_id = var.cloudflare_account_id

  tunnel_id = cloudflare_tunnel.flask.id

  config {
    warp_routing {
      enabled = false
    }
    ingress_rule {
      service = var.tunnel_origin
    }
  }

}

#
# DNS routing for the tunnel:
#   flask.example.com -> <tunnel-UUID>.cfargotunnel.com
#
resource "cloudflare_record" "flask_tunnel" {
  zone_id = var.cloudflare_zone_id

  name  = var.tunnel_dnsname
  value = cloudflare_tunnel.flask.cname

  type    = "CNAME"
  proxied = "true"
  ttl     = 1
}

• Lines 4-9: define the actual Tunnel, specifying its name (var.tunnel_name, e.g. flask) and a random secret that will be used as the Tunnel’s password.
• Lines 20-34: provide the configuration for the Tunnel. The cloudflare_tunnel_config Terraform registry page contains a description of all the available options.
• Lines 40-49: provide routing for the Tunnel, which enables connectivity by pointing a public-facing hostname of our choosing (e.g., flask.example.com) to the private Tunnel endpoint (in the format of <tunnel-UUID>.cfargotunnel.com). This block has the same effect as manually running the tunnel route dns flask flask.example.com command we saw above.

Access Application

With the Tunnel defined (but not yet active, as you can see from the INACTIVE label in the screenshot above), we need to define an Application which will coordinate authentication and authorization to the Tunnel and, ultimately, to the Flask application.

1
2
3
4
5
6
7
8
9
10
11
resource "cloudflare_access_application" "flask" {
  account_id = var.cloudflare_account_id

  name   = var.cloudflare_app_name
  domain = var.tunnel_hostname

  type                 = "self_hosted"
  session_duration     = "24h"
  app_launcher_visible = true
  logo_url             = var.cloudflare_app_logo
}

• Line 4: a friendly name of the Access Application.
• Line 5: as domain, we specify the same public-facing hostname of our choosing (e.g., flask.example.com), which will be the entry point for the Tunnel.

Next, we can define an access policy for the Application. To keep things simple, I’m using email-based authentication (rather than integrating a full blown Identity Provider), and lines 9-11 below specify which email addresses are allowed to access the Application:

1
2
3
4
5
6
7
8
9
10
11
12
13
resource "cloudflare_access_policy" "flask" {
  account_id     = var.cloudflare_account_id
  application_id = cloudflare_access_application.flask.id

  name       = "Email Filter for ${var.cloudflare_app_name}"
  precedence = "1"
  decision   = "allow"

  include {
    email = var.cloudflare_app_allowed_emails
  }

}

And that’s it from the Cloudflare side. So now we can focus on the AWS resources.

AWS Side

On the AWS side, instead, we have a few more moving parts, which we will analyse next.

Parameter Store

First, we will need a place to store the Tunnel credentials, the authentication-related JSON data generated when you first create a Tunnel (whether with cloudflared tunnel create <NAME> or the cloudflare_tunnel Terraform resource).

Since AWS Secrets Manager charges per secret per month, I opted for AWS Systems Manager Parameter Store instead (which is free):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
resource "aws_ssm_parameter" "tunnel_credentials" {
  name        = var.parameter_credentials_name
  description = "${var.tunnel_name} Cloudflared Tunnel: credentials JSON"
  type        = "SecureString"

  value = <<EOF
{
  "AccountTag": "${var.cloudflare_account_id}",
  "TunnelSecret": "${cloudflare_tunnel.flask.secret}",
  "TunnelID": "${cloudflare_tunnel.flask.id}"
}
EOF

}

• Line 2: sets the parameter’s name (e.g., FLASK_CREDENTIALS).
• Lines 6-12: programmatically construct the JSON data, which contains the Cloudflare account ID (AccountTag), as well as the Tunnel’s token (TunnelSecret) and ID (TunnelID).

ECR

Next, we can prepare two ECR repositories to host the related container images, one for the Flask server itself and one for cloudflared, which will act as a sidecar (more on this later). The following snippet is self-explanatory:

resource "aws_ecr_repository" "cloudflared" {
  name                 = var.ecr_cloudflared_name
  image_tag_mutability = "MUTABLE"
}

resource "aws_ecr_repository" "flask" {
  name                 = var.ecr_flask_name
  image_tag_mutability = "MUTABLE"
}

Let’s look at how to build the images to store in ECR.

Cloudflared

So far, I’ve been mentioning this magic cloudflared a lot. But what actually is it?

cloudflared is the command-line client for Cloudflare Tunnel, a tunnelling daemon that proxies traffic from the Cloudflare network to your origin without requiring to allow direct traffic to the origin itself.

Let’s create a Dockerfile for an image we will push to the ECR repository defined above:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
# ==============================================================================
# Download cloudflared
# ==============================================================================
FROM debian:stretch-slim as build

# Install dependencies
RUN apt-get update                       && \
    apt-get install -y curl

# Download cloudflared
RUN curl -L https://github.com/cloudflare/cloudflared/releases/latest/download/cloudflared-linux-amd64 -o cloudflared  && \
    chmod +x cloudflared                && \
    mv cloudflared /usr/local/bin/

# ==============================================================================
# Run from distroless
# ==============================================================================
FROM alpine:3.12

# Get dependencies
COPY --from=build /usr/local/bin/cloudflared /usr/local/bin/cloudflared

# Copy script
WORKDIR /etc/cloudflared
COPY ./entrypoint.sh /etc/cloudflared/entrypoint.sh
RUN chmod +x /etc/cloudflared/entrypoint.sh

# Run script
ENTRYPOINT ["/etc/cloudflared/entrypoint.sh"]

• Lines 4-13: in the first stage of this multi-stage build, we install curl to download the latest cloudflared release from GitHub.
• Lines 18-29: in the final stage of the multi-stage build, we copy cloudflared from the previous step and copy and set the entrypoint below.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
#! /bin/sh

set -ueo pipefail

# ==============================================================================
# CONFIG
# ==============================================================================
#
# STATIC
#
PATH_CLOUDFLARED="/etc/cloudflared"
PATH_CREDENTIALS="${PATH_CLOUDFLARED}/credentials.json"
PATH_CONFIG="${PATH_CLOUDFLARED}/config.yml"

#
# ENV VARS
#
VAR_ORIGIN_URL=${ORIGIN_URL}
VAR_TUNNEL_UUID=${TUNNEL_UUID}

#
# SECRETS
#
VAR_TUNNEL_CREDENTIALS=${TUNNEL_CREDENTIALS}


# ==============================================================================
# PREPARE CLOUDFLARED CONFIG
# ==============================================================================
#
# Create folder
#
mkdir -p ${PATH_CLOUDFLARED}

#
# Fetch secrets
#
echo "[*] Fetching Cloudflared Tunnel: credentials JSON..."
echo "$VAR_TUNNEL_CREDENTIALS" > $PATH_CREDENTIALS

#
# Create config file
#
echo -e "tunnel: ${VAR_TUNNEL_UUID}
credentials-file: ${PATH_CREDENTIALS}
url: ${VAR_ORIGIN_URL}
no-autoupdate: true" > $PATH_CONFIG


# ==============================================================================
# RUN TUNNEL
# ==============================================================================

# Run tunnel
echo "[*] Starting tunnel..."
cloudflared tunnel --config ${PATH_CONFIG} run ${VAR_TUNNEL_UUID}

• Lines 11-24: fetching the environment variables.
• Line 39: fetching the secrets from SSM Parameter Store.
• Lines 44-47: creating the Tunnel config file, which specifies the Tunnel ID, its credentials file (fetched from SSM), and the URL of the origin.
• Lines 56: finally starting the Tunnel.

Flask

For this PoC, we are going to deploy a super-streamlined Flask app with just one endpoint to prove the concept:

from flask import Flask
from flask_restful import Resource, Api

app = Flask(__name__)
api = Api(app)

class WebApp(Resource):
    def get(self):
        return {'Hello': 'Flask deployed in ECS'}

api.add_resource(WebApp, '/hello')

if __name__ == '__main__':
    app.run(debug=True, host='0.0.0.0')

ECS Cluster & VPC

With the ancillary services ready, it’s time to prepare the AWS-related networking the two container images will need.

This mainly translates in:

• A dedicated ECS cluster running on Fargate.

• A dedicated VPC (10.0.0.0/16) containing:
  • A primary (10.0.1.0/24) and a secondary (10.0.2.0/24) subnet.
  • An Internet Gateway.
  • Gateway VPC Endpoints for DynamoDB and S3.
  • Interface VPC Endpoints, required to enable private connectivity with the other AWS services.
  • Relevant Security Groups.
  • A Route Table which defines the relevant routes, as shown below.

All this heavy lifting is done by a Terraform module I open-sourced as well. Those interested can check it out in the related GitHub repo I linked at the top of this article:

module "flask_cluster" {
  source = "../modules/aws-ecs-cluster"

  cluster_name = var.ecs_cluster_name
}

ECS Service

The core of this post is the ECS Service running Flask and its sidecar (cloudflared):

Container Definitions

First, we need to define the container definitions that will compose the Task:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
module "container_cloudflared" {
  source = "../modules/aws-ecs-container"

  container_name  = var.container_cloudflared_name
  container_image = "${aws_ecr_repository.cloudflared.repository_url}:latest"

  essential        = true
  container_cpu    = 256 # 0.25 vCPU
  container_memory = 512 # 512 MB

  secrets = [
    {
      name      = "TUNNEL_CREDENTIALS"
      valueFrom = aws_ssm_parameter.tunnel_credentials.arn
    },
  ]

  environment = [
    {
      name  = "ORIGIN_URL"
      value = var.tunnel_origin
    },
    {
      name  = "TUNNEL_UUID"
      value = cloudflare_tunnel.flask.id
    },
  ]

  container_depends_on = [
    {
      containerName = var.container_flask_name
      condition     = "START"
    }
  ]
}

module "container_flask" {
  source = "../modules/aws-ecs-container"

  container_name  = var.container_flask_name
  container_image = "${aws_ecr_repository.flask.repository_url}:latest"

  essential = true
  port_mappings = [
    {
      containerPort = 5000
      hostPort      = 5000
      protocol      = "tcp"
    }
  ]

}

• Line 4: references the container image stored in the cloudflared ECR repository described above.
• Lines 11-16: instruct the image to pull the Tunnel credentials from the TUNNEL_CREDENTIALS SSM Parameter.
• Lines 18-27: define other environment variables (like the origin URL and Tunnel UUID) needed by the cloudflared entrypoint.
• Lines 37-52: a simple definition for the Flask container that exposes port 5000 locally.

ECS Task

Finally, we can define the task definition for the web app we want to run:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
module "task" {
  source = "../modules/aws-ecs-task"

  # Main settings
  name_prefix      = var.service_name
  platform_version = "LATEST"

  # CloudWatch
  log_retention_in_days = 1

  # Networking
  cluster_id = module.flask_cluster.cluster_id
  vpc_id     = module.flask_cluster.vpc_id
  vpc_sg_id  = module.flask_cluster.sg_id
  private_subnet_ids = [
    module.flask_cluster.subnet_id,
    module.flask_cluster.secondary_subnet_id
  ]

  # Performance
  task_definition_cpu    = 512  # 0.5 vCPU
  task_definition_memory = 1024 # 1 GB

  # Service
  create_service                  = true
  desired_count                   = 1
  task_container_assign_public_ip = false
  force_new_deployment            = false
  enable_execute_command          = false
  capacity_provider_strategy = [
    {
      capacity_provider = "FARGATE_SPOT",
      weight            = 100
    }
  ]

  # Only for load balanced
  load_balanced = false

  # Containers
  container_exposed_port        = var.tunnel_port
  container_exposed_to_internet = false
  task_container_protocol       = "HTTP"

  container_definitions = [
    module.container_cloudflared.json_map_object,
    module.container_flask.json_map_object,
  ]
}


#
# Grant access to the SSM parameters
#
resource "aws_iam_role_policy" "access_ssm" {
  name   = "${var.service_name}-access-ssm"
  role   = module.task.execution_role_name
  policy = data.aws_iam_policy_document.access_ssm.json
}

data "aws_iam_policy_document" "access_ssm" {
  statement {
    sid    = "AllowECSRunTask"
    effect = "Allow"

    actions = [
      "ssm:GetParameters",
    ]

    resources = [
      aws_ssm_parameter.tunnel_credentials.arn,
    ]
  }
}

• Lines 12-18: provide the references for the ECS cluster to run in, as well as its VPC and subnets. If you want to go deeper into the networking side of ECS, the AWS blog has a helpful post: Task Networking in AWS Fargate.
• Lines 21-22: define the amount of CPU and memory to reserve for the Task. The AWS docs have a handy table with more information on the available combinations.
• Line 26: how many replicas of the service to maintain running. Here we are not load-balancing Flask, so 1 is sufficient.
• Line 27: the task_container_assign_public_ip must be true only if you don’t want to pay for Interface VPC Endpoints (as explained above). If you leave this set to false and don’t deploy the Endpoints, the Task then won’t be able to pull images from ECR [1, 2] nor secrets from SSM.
• Lines 41-48: reference the container definitions to use.
• Lines 55-74: ensure the Task has enough permissions to pull secrets from SSM Parameter Store.

And that’s it from the AWS side as well. Apply this Terraform configuration and push the images to ECR, and you should be good to go!

See it in Action

With the Tunnel running, you can first inspect the CloudWatch logs. You should see something like the following:

At the same time, you will see that the status of the Tunnel in the Cloudflare dashboard switched to HEALTHY:

Finally, you can access the endpoint https://flask.example.com: you will first get proxied to the Cloudflare Access authentication page, and, provided you can supply an email which has been allow-listed, you will be able to access the Flask endpoint.

Conclusions

In this article, I explain how to use Cloudflare Tunnel to securely access a Flask webapp running in a private subnet in ECS on Fargate, without exposing the app to the public Internet, while keeping costs at bay.

As mentioned at the beginning of this post, the code described can be found on GitHub at: ⌨️ marco-lancini/utils

I hope you found this post valuable and interesting, and I’m keen to get feedback on it! If you find the information shared helpful, if something is missing, or if you have ideas on improving it, please let me know on 🐣 Twitter or at 📢 feedback.marcolancini.it.

Thank you! 🙇‍♂️