Delivering a Secure On-Prem RKE2 Infrastructure And CI-CD Rollout
By Or Hazout · May 22, 2025 · 5 min read
Delivering a modern CI-CD and observability platform on strictly secure, on-premises infrastructure
Delivering a modern CI-CD and observability platform on strictly secure, on-premises infrastructure is never straightforward. Over the past five and a half months, our team built exactly that: two RKE2 Kubernetes clusters, end-to-end GitOps workflows with ArgoCD and Argo Workflows, Terraform-driven infrastructure, and fully automated pipelines in Azure DevOps - all governed by rigid proxy, firewall, and policy constraints. Here’s how we navigated each hurdle, turned roadblocks into launchpads, and emerged with a resilient, repeatable framework for future deployments.
Getting Terraform to Run
Before we start, we had to solve a basic problem: where to store the Terraform state. In a normal project, you'd just let Terraform create a new S3 bucket. We couldn't. Policy forbade creating public cloud resources on the fly. To fix this we had to use a set of pre-approved S3 buckets and DynamoDB tables. We configured our Terraform backend to use these existing resources before any terraform apply was run.
The next problem was the corporate proxy. Nothing gets to the outside world without it. We had to configure all of our Terraform processes to use the HTTP(S)_PROXY and NO_PROXY environment variables just to reach the AWS APIs. This became a standard part of our wrapper scripts for every pipeline.
Building RKE2 Clusters on a Zero-Trust Network
After we configured the terraform state, we moved on to the RKE2 clusters. We used Terraform to provision the nodes, which gave us a consistent, repeatable build. The real headache was the firewall. By default, our network policy blocked everything. Kubernetes is chatty, and we had to justify opening every port it needed - the API server, etcd, flannel, and more.
How? we sat with the network team and used packet captures to map out and prove exactly what traffic was necessary. We got our list of essential ports approved and then immediately locked things down further with costume Kubernetes NetworkPolicies inside the cluster. To make sure a firewall rule change didn't silently break our clusters overnight, we added scripted connectivity checks to our CI jobs. If a node can't reach the API server, we know about it instantly.
The Internal CA and SSO Nightmare
Our next step is deploying our GitOps tools - ArgoCD and Argo Workflows using Helm Charts.
The initial installs failed immediately. in one word? Certificates. Nothing inside our new clusters trusted our company's internal Certificate Authority (CA). Every attempt to pull an image or connect to an internal service failed with a TLS error.
This was a two-part solution:
Node-Level Trust: We built a DaemonSet that automatically injects our internal CA bundle into every node's OS trust store (/etc/ssl/certs/).
Provider-Level Trust: We configured our Terraform providers (like Helm and AWS) to reference a custom ca_file.
ArgoCD Can't Clone from Git
We had a classic on-prem problem: our ArgoCD instances couldn't reach our internal corporate Git servers. The firewall blocked both SSH and HTTPS traffic from the cluster's subnet. The security team (rightfully) refused to open broad port ranges.
To fix this we used SSH tunnels . We set up a stable SSH tunnel from the cluster to our Git servers via a bastion host. We only had to whitelist the specific IPs for that tunnel. We then updated ArgoCD's known_hosts config to trust our internal Git FQDN, and the clone errors vanished. This kept security happy and let Argo do its job.
Choosing the Right CI Tool
We initially tried using Argo Workflows for our CI builds. It seemed logical to keep CI-CD in the same cluster. It was a struggle. The proxy support was difficult to configure for every build step, and jobs would randomly fail pulling packages from our Azure DevOps artifact feed. Long-running jobs stalled when ECR credentials expired.
So we tried a different direction, we ran a quick test with Azure DevOps Pipelines. And it worked smoothly. Its proxy handling is native, and its integration with its own artifact feeds is seamless.
We made the call to migrate all CI pipelines to Azure DevOps, leaving ArgoCD to handle what it does best: continuous delivery (CD). This move and our flexibility cut our pipeline maintenance a lot. To solve the AWS ECR’s 12-hour credential expiry, we simply added a task to the start of every pipeline's build job to run aws ecr get-login-password. This dynamic login solved the 12-hour expiry problem for good.
Lighting Up Observability
With pipelines running, we deployed the standard Prometheus, Grafana, and Alertmanager stack using Terraform. Just like every other tool, we had to carefully configure all of them to use the corporate proxy and trust our internal CA. With that done, we had real-time dashboards for cluster health, pipeline durations, and application metrics, with critical alerts piped directly into our team's Slack channel.
Lessons Learned from the Trenches
This project reaffirmed that you can run a modern, automated platform in a high-security environment. It just takes persistence.
Solve Certificates First. Don't even try to deploy apps until your internal CAs are trusted by everything. It will save you days of debugging.
Automate Your Compliance. Don't fight security policy. We built our bootstrap scripts to enforce policy (like using pre-approved S3 buckets) from the start. This turned our security team into allies.
Be Open-Minded About Your Tools. We love Argo, but for CI in a heavy proxy environment, Azure DevOps was the clear winner. Use the right tool for the job, even if it means splitting your stack.
Secure Tunnels Are Your Friend. When you're walled off, a well-managed SSH tunnel is a secure and reliable way to connect essential services (like Argo to Git) without compromising your security posture.
The result is a platform that's secure, observable, and fully automated, proving that even under the tightest controls, modern DevOps isn't just possible, it's essential.
Topics: RKE2, Docker, Kubernetes, CI-CD