Cloudflare in front of the EKS demo (optional Phase 8)

The existing domain eksgitops.click moves behind Cloudflare: DNS, edge TLS, DDoS protection and the Free plan's managed rules, Tunnel and Access with Keycloak as the identity provider. Design, mental models and migration runbook. Prepared 2026-10-16; updated 2026-10-17 to what infra/cloudflare builds (real Service names, the Access policy, the flip mechanism).

Colour key:

1. Decision

2. Today vs target

Cluster-side changes:

• Added: a cloudflared Deployment and a NetworkPolicy.
• Retired (after the fallback window): the Ingresses, so the ALB disappears.
• Changed in the apps: only which client-IP header they trust.

3. Mental models

Client → edge → connector → origin

4. Identity and tokens

Who checks which token

Login flow: one password, several gates

• Front channel vs back channel: redirects travel through the browser; the code-for-token exchange is a direct server call from Cloudflare to Keycloak's /token endpoint. Both need auth.eksgitops.click reachable: through the tunnel, proxied but not gated by Access.
• The SPA is unaffected: the SPA's own PKCE flow gets its API token the same silent way, reusing the Keycloak session cookie, so the user types one password.
• Never put Keycloak's login pages behind Access: that's where the Access login itself happens, so it would loop. Only Keycloak's admin console stays hidden (it's already not routed publicly).
• What Keycloak needs: no code.
  • KC_HOSTNAME and KC_PROXY_HEADERS=xforwarded, already needed behind any proxy. cloudflared sets X-Forwarded-For/-Proto, so brute-force detection sees the real client.
  • One confidential client cloudflare-access in realm.yaml (standard flow, redirect https://eksgitops.cloudflareaccess.com/cdn-cgi/access/callback, the email scope), only because Access uses Keycloak as its IdP. Its secret is generated by Terraform and read from Secrets Manager demo/keycloak/cloudflare-access (JSON client_id, client_secret).
  • The GraphQL API changes only its trusted client-IP header.

5. Full vs partial vs subdomain setup

That's why the whole zone moves: every existing record must be recreated in Cloudflare before the name servers switch.

6. Prerequisites (one-time; done 2026-10-17)

• Cloudflare account (free) at dash.cloudflare.com, email verified, 2FA on. The Account ID (not secret) is the account_id variable of infra/cloudflare.
• The zone is not added in the dashboard: Terraform creates eksgitops.click (Free plan, full setup, no record scan) and every record.
• Zero Trust organization created once through the API (POST /accounts/<id>/access/organizations): team eksgitops, so eksgitops.cloudflareaccess.com. Free plan (up to 50 users). Terraform leaves it unmanaged: nothing in it differs from the defaults.
• A custom API token with: Account · Cloudflare Tunnel · Edit; Account · Access: Apps and Policies · Edit; Account · Access: Organizations, Identity Providers, and Groups · Edit; Account · Account Settings · Edit; Zone · Zone · Edit; Zone · Zone Settings · Edit; Zone · DNS · Edit (all zones; zone creation needs Zone · Edit). It expires 2026-12-17. Narrower would be an account-owned token scoped to the one zone.
• Store the token from a separate terminal, never in chat or Git: aws secretsmanager create-secret --region ap-southeast-2 --name demo/cloudflare/api-token --secret-string '…'. Only tf-apply on main reads it (masked); pull requests never do.

7. Migration runbook

you manual · Terraform infra/cloudflare · Argo gitops repo · CLI one-off command. Every step has a check.

A. Prepare (no user impact; one merge of infra/cloudflare)

1. Terraform Zone (pending until the switch) and its settings: SSL Full (strict), minimum TLS 1.2, Always Use HTTPS. Check: the tf-apply summary shows infra/cloudflare applied; the next plan shows no changes.
2. Terraform Mirror today's DNS in Cloudflare, so the name-server switch changes nothing:
   • app./auth./argocd./grafana. → DNS-only CNAMEs to the ALB, TTL 60 s. The ALB is found by the load balancer controller's tags (elbv2.k8s.aws/cluster, ingress.k8s.aws/stack=demo), so a rebuilt cluster needs no edit;
   • every other Route 53 record, read live and copied as-is: the ACM validation CNAME (kept for the fallback window) and external-dns's ownership TXT records. The apex NS/SOA are Cloudflare's own. The plan fails rather than drop a record it can't copy (an alias or an unusual type).
   Check: dig @<cf-ns> app.eksgitops.click returns a CNAME to the ALB name that Route 53's alias resolves to; the ACM CNAME is identical.
3. Terraform DNS ownership: Terraform-managed records (four static hosts, host_modes). external-dns keeps writing Route 53 until the ALB is retired; external-dns with the Cloudflare provider remains an option for dynamic hosts.

B. Build the tunnel side (still no user impact)

4. Terraform Tunnel + routes for all four hosts: app. → http://demo-api.demo.svc.cluster.local:8082, auth. → http://keycloak.keycloak.svc.cluster.local:8080 for /realms/demo and /resources only (the ALB routed every realm), with any dot-segment path (.., %2e%2e) refused first, so the admin console, its API and the master realm stay off the internet, argocd. → http://argocd-server.argocd.svc.cluster.local:80, grafana. → http://grafana.observability.svc.cluster.local:80, catch-all http_status:404. The token goes to Secrets Manager demo/cloudflare/tunnel-token. Check: the tunnel status is healthy once the connector runs.
5. Argo cloudflared Deployment: 2 replicas, PDB, readiness on /ready, token from demo/cloudflare/tunnel-token → ESO (its read grant is in infra/eks). NetworkPolicies allow cloudflared → apps (and the ALB, until it is retired). Check: 2/2 Ready, 4 connections each in the logs; a test pod in another namespace can't reach the apps.
6. Argo Apps trust CF-Connecting-IP from cloudflared only (rate limiter; Keycloak brute-force via forwarded headers), switched on after the ALB is retired (step 11). Until then, tunnelled requests reach the rate limiter from the cloudflared pods' addresses, so tunnel users share those buckets. Check: a unit test and a log line show the real client IP.
7. Terraform Access identity provider = Keycloak (OIDC, PKCE, scopes openid email profile, email claim email). Terraform generates the client secret and writes demo/keycloak/cloudflare-access; the cloudflare-access client in realm.yaml reads it Argo. Check: "Test" in the Access identity-provider settings logs a realm user in and shows their email.
8. Terraform Access applications for argocd./grafana. only, session 8 h, Keycloak as the only identity provider (no one-time PIN) with auto-redirect to its login page. One reusable policy: allow anyone who signs in through Keycloak; Argo CD and Grafana then apply their own roles (demo.guest is read-only in both). Never on auth. (the login happens there) or app.. Cloudflare accepts Access applications only on an active zone, so each one is created by its host's flip (step 10), just before the record turns proxied. grafana. also gets a path-scoped Bypass application for its externally shared dashboards (/public-dashboards/*, /api/public/*, /public/*, checked against Grafana 12.3.1's routes), so the public dashboard link keeps working without a login. Optional later: cloudflared's access origin check.

C. Name-server cutover (no behaviour change: the records still point at the ALB)

9. CLI DNSSEC is off (checked 2026-10-17: no DS record at the registrar, the hosted zone isn't signing). Switch the name servers to the two that terraform output nameservers prints:

   aws route53domains update-domain-nameservers --region us-east-1 --domain-name eksgitops.click \
     --nameservers Name=<a>.ns.cloudflare.com Name=<b>.ns.cloudflare.com

   • Both providers serve identical records during propagation.
   • Check: dig NS eksgitops.click @1.1.1.1 and @8.8.8.8 show Cloudflare; the zone is active in the API, and its Universal SSL certificate is active before any host is proxied.
   • Cloudflare deletes a zone that stays pending for 28 days, so switch within that window of the merge.

D. Flip host by host to the tunnel (per-record rollback)

10. Terraform One pull request per host sets its host_modes entry to tunnel: the record becomes a proxied CNAME to <tunnel-id>.cfargotunnel.com. Order: app. → auth. (no Access; proves the tunnel end to end), then argocd. → grafana. (each flip creates its Access application first).
    • Check after each flip: curl -sI https://<host> shows server: cloudflare, and the host works end to end.
    • Rollback: revert that pull request (the host goes back to alb). Resolvers pick it up within minutes, with no name-server change.

E. Retire (after a few stable days)

11. Argo Delete the Ingresses (the Argo CD one is in infra/argocd), so the LB controller removes the ALB, and stop external-dns writing Route 53 (remove the add-on). Then retire ACM-on-ALB in infra/bootstrap; the DNS mirror drops the validation record by itself. Then, in the gitops repo, drop the ALB from the NetworkPolicies and switch the apps to trusting CF-Connecting-IP from cloudflared. Check: aws elbv2 describe-load-balancers shows no demo ALB; all four hosts still answer through the tunnel.
12. CLI After a few more days, delete the Route 53 hosted zone, after removing what reads it (the mirror and the route53_nameservers output in infra/cloudflare, the zone lookup in infra/bootstrap). Before then, the last-resort rollback is name servers back to the four Route 53 awsdns servers.

8. Acceptance checks

• dig NS eksgitops.click shows Cloudflare name servers; all four hosts resolve to Cloudflare anycast IPs and return server: cloudflare.
• app./auth. work end to end through the tunnel: PKCE login, GraphQL queries, playground. No Access prompt on these hosts.
• argocd./grafana. show Cloudflare Access → Keycloak login → app SSO, with one password typed.
• The demo guest gets through Access and is read-only in Argo CD and Grafana. Someone without a realm account can't get past the Access login (Keycloak is the only identity provider; no one-time PIN), and the cloudflared/Argo logs show the request never arrived.
• Grafana's public dashboard link opens without any login through the tunnel, while the rest of grafana. asks for the Access login.
• Per-record rollback works: flipping one record back to the ALB restores that host during the fallback window.
• After retirement: no demo ALB exists, and nothing in AWS accepts inbound traffic for the apps (no public LB, no NodePort).
• After the ALB is retired, the rate limiter and Keycloak brute-force log real client IPs (CF-Connecting-IP). A forged header from a non-cloudflared pod is ignored or blocked.
• Last-resort rollback (name servers back to Route 53) is written down until the hosted zone is deleted.

9. Gotchas

Cost: Cloudflare Free + Zero Trust Free = US$0; the domain is unchanged. Retiring the ALB saves about US$20/month, and the hosted zone another US$0.50/month.

10. Config snippets (Terraform excerpts from infra/cloudflare, provider v5.28; the Kubernetes parts are sketches for the gitops repo)

apiVersion: apps/v1
kind: Deployment
metadata: { name: cloudflared, namespace: cloudflare }
spec:
  replicas: 2
  selector: { matchLabels: { app: cloudflared } }
  template:
    metadata: { labels: { app: cloudflared } }
    spec:
      containers:
        - name: cloudflared
          image: cloudflare/cloudflared:<pinned>
          args: ["tunnel", "--no-autoupdate", "--metrics", "0.0.0.0:2000", "run"]
          env:
            - name: TUNNEL_TOKEN
              valueFrom: { secretKeyRef: { name: cloudflared-token, key: token } }   # ESO from demo/cloudflare/tunnel-token (plain string)
          readinessProbe: { httpGet: { path: /ready, port: 2000 } }
          securityContext: { runAsNonRoot: true, readOnlyRootFilesystem: true, allowPrivilegeEscalation: false }

locals {
  hosts = {
    app     = { service = "http://demo-api.demo.svc.cluster.local:8082", path = null, access = false, public_paths = [] }
    auth    = { service = "http://keycloak.keycloak.svc.cluster.local:8080", path = "^/(realms/demo|resources)(/|$)", access = false, public_paths = [] }
    argocd  = { service = "http://argocd-server.argocd.svc.cluster.local:80", path = null, access = true, public_paths = [] }
    grafana = { service = "http://grafana.observability.svc.cluster.local:80", path = null, access = true,
                public_paths = ["/public-dashboards/*", "/api/public/*", "/public/*"] }   # Bypass application
  }
}

# variables.tf: host_modes = { app = "alb", auth = "alb", argocd = "alb", grafana = "alb" }
resource "cloudflare_dns_record" "host" {
  for_each = local.hosts
  zone_id  = cloudflare_zone.this.id
  name     = "${each.key}.${var.domain}"
  type     = "CNAME"
  content  = var.host_modes[each.key] == "tunnel" ? local.tunnel_cname : local.alb_dns_name
  proxied  = var.host_modes[each.key] == "tunnel"
  ttl      = var.host_modes[each.key] == "tunnel" ? 1 : 60
  depends_on = [cloudflare_zero_trust_access_application.this]   # Access before proxied
}

resource "cloudflare_zero_trust_tunnel_cloudflared_config" "eks" {
  account_id = var.account_id
  tunnel_id  = cloudflare_zero_trust_tunnel_cloudflared.eks.id
  config = {
    ingress = concat(
      flatten([for host, cfg in local.hosts : concat(
        cfg.path == null ? [] : [{ hostname = "${host}.${var.domain}", path = "\\.\\.", service = "http_status:404" }],   # dot segments
        [{ hostname = "${host}.${var.domain}", path = cfg.path, service = cfg.service }],
      )]),
      [{ hostname = null, path = null, service = "http_status:404" }],   # catch-all
    )
  }
}

resource "cloudflare_zero_trust_access_identity_provider" "keycloak" {
  account_id = var.account_id
  name       = "Keycloak"
  type       = "oidc"
  config = {
    client_id        = "cloudflare-access"
    client_secret    = random_password.access_client_secret.result   # also written to demo/keycloak/cloudflare-access
    auth_url         = "${local.issuer}/protocol/openid-connect/auth"
    token_url        = "${local.issuer}/protocol/openid-connect/token"
    certs_url        = "${local.issuer}/protocol/openid-connect/certs"
    scopes           = ["openid", "email", "profile"]
    email_claim_name = "email"
    pkce_enabled     = true
  }
}

resource "cloudflare_zero_trust_access_policy" "keycloak_users" {
  account_id = var.account_id
  name       = "Signed in with Keycloak"
  decision   = "allow"
  include    = [{ login_method = { id = cloudflare_zero_trust_access_identity_provider.keycloak.id } }]
}

resource "cloudflare_zero_trust_access_application" "this" {
  for_each                  = { for host, cfg in local.hosts : host => cfg if cfg.access && var.host_modes[host] == "tunnel" }
  account_id                = var.account_id
  type                      = "self_hosted"
  domain                    = "${each.key}.${var.domain}"
  destinations              = [{ type = "public", uri = "${each.key}.${var.domain}" }]
  session_duration          = "8h"
  allowed_idps              = [cloudflare_zero_trust_access_identity_provider.keycloak.id]
  auto_redirect_to_identity = true
  policies                  = [{ id = cloudflare_zero_trust_access_policy.keycloak_users.id, precedence = 1 }]
}

# The more specific application wins: grafana.'s externally shared dashboards stay public.
resource "cloudflare_zero_trust_access_policy" "public_paths" {
  account_id = var.account_id
  name       = "Public paths (bypass)"
  decision   = "bypass"
  include    = [{ everyone = {} }]
}

resource "cloudflare_zero_trust_access_application" "public" {
  for_each     = { for host, cfg in local.hosts : host => cfg if length(cfg.public_paths) > 0 && var.host_modes[host] == "tunnel" }
  account_id   = var.account_id
  type         = "self_hosted"
  domain       = "${each.key}.${var.domain}${each.value.public_paths[0]}"
  destinations = [for path in each.value.public_paths : { type = "public", uri = "${each.key}.${var.domain}${path}" }]
  policies     = [{ id = cloudflare_zero_trust_access_policy.public_paths.id, precedence = 1 }]
}

clients:
  - clientId: cloudflare-access
    protocol: openid-connect
    publicClient: false
    secret: "$(env:CLOUDFLARE_ACCESS_CLIENT_SECRET)"     # keycloak-config-cli substitution; ESO from demo/keycloak/cloudflare-access (client_secret)
    standardFlowEnabled: true
    redirectUris: ["https://eksgitops.cloudflareaccess.com/cdn-cgi/access/callback"]
    defaultClientScopes: [basic, email, profile]          # Access needs the email claim

provider: { name: cloudflare }
env:
  - name: CF_API_TOKEN
    valueFrom: { secretKeyRef: { name: cloudflare-api-token, key: token } }
domainFilters: ["eksgitops.click"]
txtOwnerId: eks-gitops-demo
extraArgs: ["--cloudflare-dns-records-per-page=5000"]   # proxied flag per record via annotation
                                                       # external-dns.alpha.kubernetes.io/cloudflare-proxied