Reading time ~11 minutes
Remotely Access your Kubernetes Lab with Cloudflare Tunnel
- The Environment: Kubernetes Lab on Baremetal
- Access the Host
- Access Kubernetes Services
- Automate with Terraform
- Conclusions
Back in April, I saw a post on the Cloudflare tech blog which explained how Auditable Terminal gives you a fully featured SSH client in your browser: you authenticate using Cloudflare Access, and can log into a computer - and get a terminal - just using a browser.
The post made me curious, but at the time I didn’t have capacity to look into it. Until now, when I decided to give it a go for my home lab.
The Environment: Kubernetes Lab on Baremetal
In Kubernetes Lab on Baremetal, I detailed the steps I took to deploy my own Kubernetes Lab on baremetal, and on an Intel NUC in particular.
With this post, I want to take the setup a step further, using Cloudflare Tunnel to access the cluster remotely.
The end goal is to use Cloudflare Tunnel to connect my Intel NUC to the Cloudflare network, and to use Auditable Terminal to connect to it using nothing more than a browser.
The remainder of this post will assume you have a server running Fedora CoreOS (or Linux in general) as operating system, as outlined in Kubernetes Lab on Baremetal.
Access the Host
From the Cloudflare’s documentation:
Cloudflare Tunnel runs a lightweight daemon (
cloudflared) in your infrastructure that establishes outbound connections (Tunnels) between your service and the Cloudflare edge. When Cloudflare receives a request for your chosen hostname, it proxies the request through those connections tocloudflared. In turn,cloudflaredproxies the request to your applications.
Basically, Cloudflare Tunnel requires the installation of a lightweight
(and open source) server-side daemon (called cloudflared),
to connect your infrastructure to Cloudflare.
Let’s see how to use it to access the Intel NUC remotely.
Create a Cloudflare Tunnel
Let’s start by SSH-ing into the host and
downloading cloudflared for Linux:
[[email protected] ~]$ curl -L https://github.com/cloudflare/cloudflared/releases/latest/download/cloudflared-linux-amd64 -o cloudflared
[[email protected] ~]$ chmod +x cloudflared
[[email protected] ~]$ sudo mv cloudflared /usr/local/bin/Once downloaded, we need to authenticate the cloudflared daemon:
issue the cloudflared tunnel login command and follow the instructions
to authenticate against your Cloudflare account.
After logging into your account, select the domain you want to use:
[[email protected] ~]$ cloudflared tunnel login
Please open the following URL and log in with your Cloudflare account:
https://dash.cloudflare.com/argotunnel?callback=<redacted>
Leave cloudflared running to download the cert automatically.
2021-08-25T19:46:03Z INF Waiting for login...
2021-08-25T19:46:56Z INF Waiting for login...
You have successfully logged in.
If you wish to copy your credentials to a server, they have been saved to:
/var/home/core/.cloudflared/cert.pem
Selecting a domain will automatically generate a cert.pem file
in the default .cloudflared directory,
which contains account-wide credentials.
We are now ready to create a tunnel (named nuc in this case):
[[email protected] ~]$ cloudflared tunnel create nuc
Tunnel credentials written to /var/home/core/.cloudflared/1111-2222-3333.json.
cloudflared chose this file based on where your origin certificate was found.
Keep this file secret. To revoke these credentials, delete the tunnel.
Created tunnel nuc with id 1111-2222-3333This command will generate a credentials file and a tunnel, by establishing a persistent relationship between the name we chose (i.e., nuc) and a UUID (i.e., 1111-2222-3333) for the tunnel.
At this point, no connection is active within the tunnel yet.
The last component missing to instaurate a connection is a configuration file
(.cloudflared/config.yml),
which will instruct the tunnel to route traffic from a given origin to the chosen hostname:
# [[email protected] ~]$ cat .cloudflared/config.yml
tunnel: 1111-2222-3333.json # the UUID of the tunnel
credentials-file: /var/home/core/.cloudflared/1111-2222-3333.json # the credentials file generated
ingress:
- hostname: nuc.marcolancini.it # the hostname to use
service: ssh://localhost:22
- service: http_status:404
# Catch-all rule, which responds with 404 if traffic doesn't match any of
# the earlier rulesTo start routing traffic, assign a CNAME record that points traffic to the tunnel’s subdomain:
[[email protected] ~]$ cloudflared tunnel route dns 1111-2222-3333 nuc.marcolancini.it
2021-08-25T19:58:10Z INF Added CNAME nuc.marcolancini.it which will route to this tunnel tunnelID=1111-2222-3333You can now finally run cloudflared
to proxy incoming traffic from the tunnel to any number of services running locally:
[[email protected] ~]$ cloudflared tunnel --config ~/.cloudflared/config.yml run nuc
2021-08-25T20:01:14Z INF Starting tunnel tunnelID=1111-2222-3333
2021-08-25T20:01:14Z INF Version 2021.8.3
2021-08-25T20:01:14Z INF GOOS: linux, GOVersion: devel +11087322f8 Fri Nov 13 03:04:52 2020 +0100, GoArch: amd64
2021-08-25T20:01:14Z INF Settings: map[config:/var/home/core/.cloudflared/config.yml cred-file:/var/home/core/.cloudflared/1111-2222-3333.json credentials-file:/var/home/core/.cloudflared/1111-2222-3333.json]
2021-08-25T20:01:14Z INF Generated Connector ID: <redacted>
2021-08-25T20:01:14Z INF cloudflared will not automatically update when run from the shell. To enable auto-updates, run cloudflared as a service: https://developers.cloudflare.com/argo-tunnel/reference/service/
2021-08-25T20:01:14Z INF Initial protocol http2
2021-08-25T20:01:14Z INF Starting metrics server on 127.0.0.1:36423/metrics
2021-08-25T20:01:16Z INF Connection <redacted> registered connIndex=0 location=AMS
2021-08-25T20:01:16Z INF Connection registered connIndex=1 location=LHR
2021-08-25T20:01:17Z INF Connection registered connIndex=2 location=AMS
2021-08-25T20:01:18Z INF Connection registered connIndex=3 location=LHR
[...]You can also check the status of the tunnel:
[[email protected] ~]$ cloudflared tunnel list
ID NAME CREATED CONNECTIONS
1111-2222-3333 nuc 2021-08-25T19:51:49Z 2xAMS, 2xLHRCreate a Zero Trust Policy
We do have a connection instaurated, but how can we connect to the host remotely then? The answer is Cloudflare Access.
If you don’t already have a Cloudflare for Teams account, you can visit https://dash.teams.cloudflare.com/ and follow the setup guide. The free plan is sufficient.
Once you have an account, navigate to the Cloudflare for Teams dashboard to create a new application.
Select the Applications page from the sidebar, and then Add application:
On the next page, choose Self-hosted as application type:
Give the application a name and specify the subdomain that will become the hostname where the application will be accessible via.
Note that it will have to match the hostname specified in the .cloudflared/config.yml file:
As a next step, we need to create rules that control who should be allowed access to the application
(in a Zero-Trust style).
In this case, we will add an email filter,
specifying that only people with a @marcolancini.it email address will be able to authenticate:
Finally, let’s enable “Browser Rendering”: Cloudflare can render an SSH client in your browser without the need for client software or end user configuration changes. Once enabled, when users authenticate and visit the URL of the application, Cloudflare will render a terminal in their browser.
To enable it, navigate to the application page of the Access section in the Cloudflare for Teams dashboard.
Click Edit and select the Settings tab.
In the cloudflared settings card, select SSH from the “Browser Rendering” drop-down menu:
If we try to connect now, we will see that password-based authentication is disabled (gladly!), in favour of key-based authentication. You’ll be presented with a text form where to enter the host’s private key:
If, like me, you don’t like to copy-paste keys around, it’s time to setup short-lived certificates.
Configure Short-Lived Certificates
Cloudflare Access can replace traditional SSH key models with short-lived certificates issued to users based on the token generated by their Access login. The SSH server can then use that certificate to start the session.
Let’s generate a short-lived certificate public key.
On the Teams dashboard, navigate to Access > Service Auth, then,
in the drop-down, choose your application (NUC in this case):
After clicking on Generate certificate,
a row will appear with a public key scoped to the application.
Save this key as a new .pub file in the host’s filesystem (e.g., /etc/ssh/ca.pub).
Next, Cloudflare Access requires two changes to the /etc/ssh/sshd_config file used on the target host:
PubkeyAuthenticationshould not be commented out, and should be set toyes(e.g.,PubkeyAuthentication yes).- Add a new line which will trust the newly created key:
TrustedUserCAKeys /etc/ssh/ca.pub.
After these changes, restart the SSH server:
[[email protected] ~]$ systemctl restart sshdEnsure Unix usernames match user SSO identities
It is important to note that the Cloudflare Access user should match a Unix username: Cloudflare Access will take the identity from a token and, using short-lived certificates, authorize the user on the host. Access matches based on the identity that precedes an email domain. Hence, Unix usernames must match the identity preceding the email domain.
For example, if the user’s identity is [email protected],
then Cloudflare Access will look to match that identity to the Unix user marco.
If you don’t have such a user, create one (e.g., sudo adduser marco).
If you try to connect and authenticate now,
Cloudflare will render a terminal in your browser
(notice the user marco).
Run cloudflared as a Service
Finally, to avoid having to manually launch the agent, Cloudflare Tunnel can install itself as a system service:
[[email protected] ~]$ sudo cloudflared service installThe configuration files in ~/.cloudflared/ will then automatically
be copied to /etc/cloudflared/.
Subscribe to CloudSecList
Access Kubernetes Services
The interesting fact is that Cloudflare Tunnel can also be used to route traffic to Kubernetes clusters. Let’s see how.
First of all, let’s deploy a sample service which we want to access remotely:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: httpbin-deployment
namespace: test
spec:
selector:
matchLabels:
app: httpbin
replicas: 1
template:
metadata:
labels:
app: httpbin
spec:
containers:
- name: httpbin
image: kennethreitz/httpbin:latest
ports:
- containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
name: web-service
namespace: test
spec:
selector:
app: httpbin
ports:
- protocol: TCP
port: 80➜ kubectl apply -f app.yaml
deployment.apps/httpbin-deployment created
service/web-service createdThen, let’s create a new tunnel dedicated to the test deployment, similarly as we did before:
[[email protected] ~]$ cloudflared tunnel create test-k8s
Tunnel credentials written to /var/home/core/.cloudflared/4444-5555-6666.json.
cloudflared chose this file based on where your origin certificate was found.
Keep this file secret. To revoke these credentials, delete the tunnel.
Created tunnel test-k8s with id 4444-5555-6666With the tunnel created, we now need to share with the Kubernetes cluster the file containing the credentials of the tunnel. We can do so by creating a Secret:
[[email protected] .cloudflared]$ kubectl -n test create secret generic tunnel-credentials --from-file=credentials.json=/var/home/core/.cloudflared/4444-5555-6666.json
secret/tunnel-credentials created
[[email protected] .cloudflared]$ kubectl -n test get secret
NAME TYPE DATA AGE
default-token-72zr2 kubernetes.io/service-account-token 3 5m51s
tunnel-credentials Opaque 1 10sNext, we need to associate the tunnel with a DNS record:
[[email protected] .cloudflared]$ cloudflared tunnel route dns 4444-5555-6666 k8s.marcolancini.it
2021-08-26T20:27:06Z INF Added CNAME k8s.marcolancini.it which will route to this tunnel tunnelID=4444-5555-6666Finally, we need to deploy cloudflared in the cluster.
The manifest below will generate a Deployment for cloudflared,
alongside with a ConfigMap for its config:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: cloudflared
namespace: test
spec:
selector:
matchLabels:
app: cloudflared
replicas: 1
template:
metadata:
labels:
app: cloudflared
spec:
containers:
- name: cloudflared
image: cloudflare/cloudflared:2021.8.3
args:
- tunnel
- --config
- /etc/cloudflared/config/config.yaml
- run
volumeMounts:
- name: config
mountPath: /etc/cloudflared/config
readOnly: true
- name: creds
mountPath: /etc/cloudflared/creds
readOnly: true
volumes:
- name: creds
secret:
secretName: tunnel-credentials
- name: config
configMap:
name: cloudflared
items:
- key: config.yaml
path: config.yaml
---
apiVersion: v1
kind: ConfigMap
metadata:
name: cloudflared
namespace: test
data:
config.yaml: |
tunnel: test-k8s
credentials-file: /etc/cloudflared/creds/credentials.json
# metrics: 0.0.0.0:2000
no-autoupdate: true
ingress:
- hostname: k8s.marcolancini.it
service: http://web-service:80
- service: http_status:404➜ kubectl apply -f cloudflared.yaml
deployment.apps/cloudflared created
configmap/cloudflared created
➜ kubectl get pods
NAME READY STATUS RESTARTS AGE
cloudflared-6fd757bfcd-lwcgv 1/1 Running 0 3m16s
httpbin-deployment-67f749774f-l587w 1/1 Running 0 27m
➜ kubectl logs $(kubectl get pod -l app=cloudflared -o jsonpath="{.items[0].metadata.name}")
2021-08-26T20:49:01Z INF Starting tunnel tunnelID=4444-5555-6666
2021-08-26T20:49:01Z INF Version
2021-08-26T20:49:01Z INF GOOS: linux, GOVersion: go1.16.4, GoArch: amd64
2021-08-26T20:49:01Z INF Settings: map[config:/etc/cloudflared/config/config.yaml cred-file:/etc/cloudflared/creds/credentials.json credentials-file:/etc/cloudflared/creds/credentials.json metrics:0.0.0.0:2000 no-autoupdate:true]
2021-08-26T20:49:01Z INF Generated Connector ID: b47e1cc3-96df-4f9f-b0b4-2b491f69990a
2021-08-26T20:49:01Z INF Initial protocol http2
2021-08-26T20:49:01Z INF Starting metrics server on [::]:2000/metrics
2021-08-26T20:49:01Z INF Starting Hello World server at 127.0.0.1:33855
2021-08-26T20:49:03Z INF Connection 519adf31-8b54-4c58-9aba-956e35548003 registered connIndex=0 location=AMS
{"level":"debug","time":"2021-08-26T20:49:03Z","message":"edgediscovery - GetDifferentAddr: Giving connection its new address"}
2021-08-26T20:49:03Z INF Connection 8964c25c-ab8e-41a8-ba2f-55d3eabd3035 registered connIndex=1 location=LHR
{"level":"debug","time":"2021-08-26T20:49:04Z","message":"edgediscovery - GetDifferentAddr: Giving connection its new address"}
2021-08-26T20:49:04Z INF Connection 50d3d0f7-98fc-4174-b127-8fdf7bb428b7 registered connIndex=2 location=AMS
{"level":"debug","time":"2021-08-26T20:49:05Z","message":"edgediscovery - GetDifferentAddr: Giving connection its new address"}
2021-08-26T20:49:06Z INF Connection b58d388f-f9d9-4162-a51c-d2104149cb35 registered connIndex=3 location=LHRWhen Cloudflare receives traffic for the hostname you configured in the previous step,
it will send that traffic to the cloudflared instances running in this deployment.
Then, those cloudflared instances will proxy the request to the application’s Service.
Automate with Terraform
Let’s take this a step further, and let’s automate it with Terraform. Cloudflare has a decent Terraform provider which can be used to automate the creation of the application and DNS bindings:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
# ==============================================================================
# VARIABLES
# ==============================================================================
variable "domain" {
default = "example.com"
}
variable "zone_id" {
default = <CLOUDFLARE_ZONE_ID>
}
# ==============================================================================
# APPLICATION
# ==============================================================================
resource "cloudflare_access_application" "cf_app_k8s" {
zone_id = var.zone_id
name = "K8s Example"
domain = "k8s.${var.domain}"
session_duration = "24h"
}
resource "cloudflare_access_policy" "cf_policy" {
application_id = cloudflare_access_application.cf_app.id
zone_id = var.zone_id
name = "Email Filter"
precedence = "1"
decision = "allow"
include {
email = ["[email protected]${var.domain}"]
}
}
resource "cloudflare_record" "cf_app_k8s_cname" {
zone_id = var.zone_id
name = "k8s"
value = "<redacted>.cfargotunnel.com"
type = "CNAME"
proxied = "true"
ttl = 1
}
Conclusions
In this blog post, part of the “Kubernetes Primer for Security Professionals” series, I described how to use Cloudflare Tunnel to connect my Intel NUC to the Cloudflare network, and Auditable Terminal to connect to it using nothing more than a browser.
I hope you found this post useful and interesting, and I’m keen to get feedback on it! If you find the information shared was useful, if something is missing, or if you have ideas on how to improve it, please let me know on Twitter.
